XpulpNN: Enabling Energy Efficient and Flexible Inference of Quantized Neural Network on RISC-V based IoT End Nodes

Abstract

XpulpNN introduces lightweight RISC-V ISA extensions supporting 4-bit and 2-bit SIMD instructions to accelerate heavily Quantized Neural Network (QNN) inference on IoT end nodes. The architecture utilizes a parallel cluster of 8 extended processors and a custom execution paradigm fusing dot product and load operations. This solution achieves efficiencies of 2.22 TOPs/s/W, resulting in 6x to 8x speedups over 8-bit SIMD baselines and delivering performance three orders of magnitude better than state-of-the-art ARM Cortex-M MCUs.

Report

Key Highlights

• Target: Enabling energy-efficient and flexible inference of heavily Quantized Neural Networks (QNN) on RISC-V based microcontroller-class IoT end nodes.
• Core Innovation: Introduction of lightweight RISC-V ISA extensions specifically for nibble (4-bit) and crumb (2-bit) Single Instruction, Multiple Data (SIMD) operations.
• Performance: Convolution kernels run 6x faster (4-bit operands) and 8x faster (2-bit operands) compared to a baseline processing cluster supporting only 8-bit SIMD instructions.
• Efficiency Benchmark: Achieves a peak efficiency of 2.22 TOPs/s/W, comparable to dedicated DNN inference accelerators.
• Competitive Advantage: Performance is up to three orders of magnitude better than state-of-the-art ARM Cortex-M based microcontroller systems (e.g., STM32L4 and STM32H7).

Technical Details

• ISA Extensions: The RISC-V Instruction Set Architecture is extended with specialized SIMD instructions tailored for 4-bit and 2-bit integer arithmetic, providing near-linear speedup for key QNN computation kernels.
• Architectural Implementation: The extended RISC-V core is integrated into a parallel cluster comprising 8 processors, yielding a near-linear performance improvement over a single-core architecture.
• Execution Paradigm: A novel execution paradigm is proposed for SIMD sum-of-dot-product operations, which fuses the dot product computation with a load operation. This fusion results in an improved peak efficiency of up to 1.64x MAC/cycle.
• Fabrication Technology: The processor cluster implementing the proposed extensions was fully implemented using GF22FDX technology.

Implications

• RISC-V Ecosystem Maturation: XpulpNN demonstrates the capability of RISC-V ISA customization to efficiently tackle highly specialized workloads like low-bit QNN inference, solidifying RISC-V's relevance in high-performance embedded AI.
• Democratization of Edge AI: By achieving accelerator-level energy efficiency (2.22 TOPs/s/W) on a flexible microcontroller architecture, this solution lowers the barrier for deploying complex, high-speed neural network models directly on energy-constrained IoT devices.
• Challenging ARM: The finding that XpulpNN is three orders of magnitude more efficient than current high-end ARM Cortex-M MCUs provides a compelling competitive advantage for RISC-V in the deeply embedded AI market, pushing the envelope for ultra-low-power computing.