AdaptViT: Runtime-Adaptive Vision Transformer Deployment on Custom RISC-V
AI Analysis Research Summary Deploying Vision Transformers (ViTs) on low-power edge devices faces challenges due to high computational demands and the storage overhead of conventional pruning methods. This paper presents AdaptViT, an end-to-end deployment pipeline that…
AI Analysis
Research Summary
Deploying Vision Transformers (ViTs) on low-power edge devices faces challenges due to high computational demands and the storage overhead of conventional pruning methods. This paper presents AdaptViT, an end-to-end deployment pipeline that transforms pretrained ViTs into a single runtime-configurable binary for embedded RISC-V CPUs. The system achieves dynamic compute-budget switching by restructuring C kernels with modified loop bounds and binary-mask control logic, coupled with a hardware-aligned block pruning strategy for Multi-Layer Perceptron (MLP) layers. Additionally, a custom MAC2 ISA extension is introduced to exploit input-reuse patterns in linear projection kernels. Evaluated on a Synopsys TRV32P3FX RISC-V processor, AdaptViT reduces on-device storage by up to 4.86x, achieves up to 2.8x speedup at 65% MLP and 50% attention-head pruning for ViT-Base, and the MAC2 ISA extension alone provides 1.56x speedup and 33% lower inference energy.
Research Question and Contribution
- Runtime-adaptive ViT inference from a single compiled binary on RISC-V.
- Up to 4.86x reduction in on-device storage compared to multi-binary deployment.
- Achieves up to 2.8x inference speedup for ViT-Base with combined pruning and ISA extension.
- Custom MAC2 ISA extension provides 1.56x speedup and 33% lower inference energy.
- Hardware-aligned block pruning strategy for MLP layers improves pruning efficiency.
Methods and Evidence
- The deployment pipeline converts pretrained ViTs into a single runtime-configurable binary for embedded RISC-V CPUs.
- C kernels are restructured with modified loop bounds and binary-mask control logic, enabling dynamic switching across discrete sparsity levels via compact external configuration files.
- A hardware-aligned block pruning strategy is applied to Multi-Layer Perceptron (MLP) layers, grouping neurons into blocks of eight and pruning at the block level.
- Attention heads are pruned per-head, and the corresponding Q/K/V slices are skipped in QKV projections and output projections.
- Batch matrix multiplication kernels for attention score computation and value aggregation exploit head-level sparsity by skipping pruned heads and zero-initializing their output regions.
- Partial INT8 quantization is applied to linear projection layers using TORCHAO, while nonlinear operations remain in floating point.
- A custom MAC2 instruction is added to the RISC-V ISA, performing two parallel MAC operations with a shared input activation, reducing innermost loop instructions from 16 to 4 in dense_pack kernels.
- Evaluation was conducted on the Oxford-IIIT Pet dataset using a Synopsys TRV32P3FX RISC-V processor, a 32-bit, 3-stage in-order core with single-precision floating-point support.
- Latency was measured via instruction-accurate (IA) simulation on both base and ISA-extended cores, with results validated against PyTorch quantized inference.
- ASIC synthesis (TSMC 28 nm) showed the MAC2 extension incurred 24.7% area and 5.1% power overhead, but achieved a lower normalized Area-Delay Product (ADP) than the baseline.
- The runtime control overhead for switching configurations is less than 5%, and classifier handling overhead is approximately 57K/114K cycles for ViT-Small/Base, respectively, which is less than 10^-5 times a single inference cycle.
What the Results Mean
- Enables efficient deployment of computationally intensive Vision Transformers on resource-constrained edge devices by reducing storage and improving inference speed and energy efficiency.
- The runtime-adaptive nature allows dynamic adjustment of computational budgets based on changing environmental conditions (e.g., battery levels, workload, user demands) without requiring model reloads or recompilation.
- The proposed methodology of combining structured pruning with C code transformation and custom ISA extensions offers a generalizable approach for optimizing deep learning models on embedded processors.
- The significant reduction in on-device storage makes ViT deployment more feasible for microcontrollers with limited flash memory.
- The energy-accuracy trade-off provided by multiple operating points allows developers to select optimal configurations based on specific application requirements.
Limitations and Open Questions
- The group-of-8 structured pruning introduces an average accuracy drop of 2.12% over baseline FP32 per-neuron pruning.
- INT8 quantization further leads to an additional 5.07% accuracy drop relative to FP32.
- The current framework's classifier handling involves copying weights and bias during runtime switching, which, while negligible, is identified as an area for future optimization to fully eliminate overhead.
- The proposed MAC2 ISA extension, while efficient, is tailored to specific input-reuse patterns in dominant projection kernels and may not generalize to all types of operations or models as broadly as general-purpose vector (RVV) or packed-SIMD (RVP) extensions.
- Future work is needed to extend mask-based skipping to nonlinear layers and explore heterogeneous acceleration using vector or matrix compute engines, indicating current scope limitations.
- ViT-Small exhibits larger accuracy degradation compared to ViT-Base due to lower redundancy, suggesting the effectiveness of pruning may vary with model size.
Source: https://arxiv.org/abs/2610.02288