Kraken: A Direct Event/Frame-Based Multi-sensor Fusion SoC for Ultra-Efficient Visual Processing in Nano-UAVs

Abstract

Kraken is an ultra-low-power, heterogeneous System-on-Chip (SoC) fabricated in 22nm FDX technology designed to enable complex, autonomous visual tasks for Nano-UAVs under tight power constraints. The chip achieves high-speed, multi-functional visual processing by efficiently fusing data directly from both standard frame-based and novel event-based (DVS) sensors. Its architecture integrates three specialized accelerators, including an 8-core RISC-V cluster (1.8 TOPS/W) and a neuromorphic engine capable of sub-uJ/inference SNN processing.

Report

Key Highlights

• Purpose: Enables complex, autonomous visual processing (e.g., navigation, tracking, obstacle avoidance) fully onboard Nano-UAVs.
• Sensor Fusion Innovation: Demonstrates a direct multi-sensor fusion capability, combining output from high-speed event-based DVS sensors and traditional frame-based (BW/RGB) imagers on a single low-power chip.
• Fabrication & Efficiency: Built using energy-efficient 22nm FDX technology.
• Heterogeneous Architecture: Integrates an 8-core RISC-V cluster, a TNN accelerator, and a dedicated neuromorphic accelerator for specialized workload partitioning.
• Extreme Energy Efficiency: The dedicated neuromorphic engine allows for highly sparse, event-driven SNN inference at ultra-low power levels (sub-uJ/inference).

Technical Details

• Process Technology: 22nm FDX (Fully Depleted Silicon On Insulator).
• Processing Cluster: Features an 8-cores RISC-V processor cluster specifically designed for frame-based inference.
• RISC-V Performance Metric: The RISC-V cluster includes mixed-precision Deep Neural Network (DNN) extensions, achieving an efficiency of 1.8 TOPS/W (Tera Operations per Second per Watt).
• TNN Accelerator: Includes a Tensor Neural Network (TNN) accelerator boasting a remarkable efficiency of 1036 TOPS/W.
• Neuromorphic Computing: Handles Spiking Neural Network (SNN) inference using an energy-proportional accelerator optimized for event-driven, highly sparse data.
• Interfacing: The SoC includes a vast set of peripherals to efficiently manage both standard frame-based sensors and novel event-based DVS interfaces.

Implications

• RISC-V Domination at the Edge: Kraken underscores the growing adoption of RISC-V as the core instruction set for highly efficient, specialized, heterogeneous processing in constrained environments. The inclusion of specialized DNN extensions within the RISC-V cluster confirms the trend of customizing the ISA for AI acceleration.
• Enabling True Nano-UAV Autonomy: By integrating high-performance computation and low-power multi-sensor fusion onto a single chip, Kraken addresses the fundamental challenge of power and payload limitations, making true autonomy feasible for nano- and pico-sized UAVs for critical applications (e.g., search and rescue).
• Advancement in Sensor Modality: This architecture validates the practical use of fusing data streams from asynchronous event cameras with synchronous frame cameras, pushing the envelope for robust computer vision algorithms in dynamic, low-latency applications.
• Neuromorphic Hardware Maturation: The successful deployment of a dedicated sub-uJ/inf neuromorphic accelerator shows the viability of dedicated hardware for SNNs, promising massive efficiency gains for specific visual tasks where data sparsity is high.