Streaming Speech Recognition on a Microcontroller: a 65.7M-Parameter Transducer in 2 and 4 Bits on the ESP32-P4

AI Analysis

Abstract

Key Highlights

• Achieved streaming speech recognition on an ESP32-P4 microcontroller with a 65.7M-parameter Zipformer2 transducer (Kroko).
• Implemented a training-free 2/4-bit weight quantization method, reducing model size from 71 MB to 26.65 MB.
• Utilized a C engine with hand-written PIE SIMD kernels for efficient processing on dual 400 MHz RISC-V cores.
• Processed 1.28 seconds of audio in 0.90 seconds, enabling faster-than-real-time streaming.
• Demonstrated 4.34% WER on LibriSpeech test-clean and 10.26% on test-other, with minimal accuracy loss from integer arithmetic.
• The system is open-source, including the inference engine, firmware, and 2/4-bit weights.

Technical Details

• The model is Kroko, a 65.7M-parameter streaming Zipformer2 transducer for English.
• The hardware platform is an ESP32-P4 microcontroller with two 400 MHz RISC-V cores, a 128-bit SIMD extension (PIE), and 32 MB of PSRAM.
• A training-free method quantizes the encoder's 290 matrices to 2 or 4 bits per weight, using Hadamard-rotated groups (Cactus Quants layout).
• Bit width for each matrix is determined by its measured sensitivity, followed by correction towards the original model's output, GPTQ rounding, and group norm optimization.
• The 71 MB of original int8 weights are reduced to a 26.65 MB file.
• Activations are int16 between matrix products and int8 into them, with dynamic scales for products.
• A C engine with hand-written PIE SIMD kernels is used, which is bit-exact with its PC build for accurate WER measurement on full test sets.
• The processing time for a 1.28-second audio chunk is 0.90 seconds (0.70 times real time for the live transcriber, 0.59 for the engine alone).
• Word Error Rates (WER) are 4.34% on LibriSpeech test-clean and 10.26% on test-other (with beam 4 search).
• The integer arithmetic introduces no measurable accuracy loss on LibriSpeech, FLEURS, or Common Voice.
• The system uses approximately 26.65 MB for weights, ~4 MB for engine state/workspace, and 1.2 MB for the screen frame buffer in PSRAM, leaving about 0.5 MB free.
• The search is k2's modified beam search (beam of 4), with token commitment rules and phrase biasing capabilities.

Implications

• Enables complex, open-vocabulary streaming speech recognition on resource-constrained microcontrollers without dedicated neural accelerators.
• Demonstrates the effectiveness of advanced post-training quantization techniques (2/4-bit, GPTQ, Hadamard rotation) for large models on edge devices.
• Highlights the importance of hardware-specific SIMD optimizations (PIE kernels) and careful integer arithmetic design for achieving real-time performance on microcontrollers.
• Provides an open-source solution that can serve as a foundation for further development in embedded Automatic Speech Recognition (ASR).
• Pushes the boundaries of what is considered feasible for speech recognition on low-power, low-cost hardware.

Limitations

• Energy and power consumption were not measured due to lack of instrumentation.
• The system is English-only.
• Evaluations were conducted on read or prompted speech (LibriSpeech, FLEURS, Common Voice, text-to-speech); performance on noisy, far-field, or conversational speech was not measured.
• Word Error Rates (WERs) are derived from a PC build, with bit-exactness checked on subsets of data, not on full test sets directly on the chip.
• The 2/4-bit weights introduce an accuracy gap compared to the original int8 model (0.5 to 2.0 points WER depending on the dataset).
• The quantized model is more sensitive to very quiet input and tends to drop initial words more often than the released model.
• The firmware requires ESP32-P4X (chip revision v3) with 32 MB of PSRAM, and the weights necessitate a microSD card.
• The quantization method was developed on a single model (Kroko), and its generalizability to other models has not been demonstrated.
• An attempt to reach a 16 MB memory target for the model stopped at 19.63 MB, indicating the target was not met.