RISC-V LinuxBetaOPEN SILICON. SHARED KNOWLEDGE.

AdaptViT: Runtime-Adaptive Vision Transformer Deployment on Custom RISC-V

AI Analysis Research Summary Deploying Vision Transformers (ViTs) on low-power edge devices faces challenges due to high computational demands and the storage overhead of conventional pruning methods. This paper presents AdaptViT, an end-to-end deployment pipeline that…

Fig. 1 : MLP Pruning Method.
Source: arxiv.org

AI Analysis

Research Summary

Deploying Vision Transformers (ViTs) on low-power edge devices faces challenges due to high computational demands and the storage overhead of conventional pruning methods. This paper presents AdaptViT, an end-to-end deployment pipeline that transforms pretrained ViTs into a single runtime-configurable binary for embedded RISC-V CPUs. The system achieves dynamic compute-budget switching by restructuring C kernels with modified loop bounds and binary-mask control logic, coupled with a hardware-aligned block pruning strategy for Multi-Layer Perceptron (MLP) layers. Additionally, a custom MAC2 ISA extension is introduced to exploit input-reuse patterns in linear projection kernels. Evaluated on a Synopsys TRV32P3FX RISC-V processor, AdaptViT reduces on-device storage by up to 4.86x, achieves up to 2.8x speedup at 65% MLP and 50% attention-head pruning for ViT-Base, and the MAC2 ISA extension alone provides 1.56x speedup and 33% lower inference energy.

Research Question and Contribution

  • Runtime-adaptive ViT inference from a single compiled binary on RISC-V.
  • Up to 4.86x reduction in on-device storage compared to multi-binary deployment.
  • Achieves up to 2.8x inference speedup for ViT-Base with combined pruning and ISA extension.
  • Custom MAC2 ISA extension provides 1.56x speedup and 33% lower inference energy.
  • Hardware-aligned block pruning strategy for MLP layers improves pruning efficiency.

Methods and Evidence

  • The deployment pipeline converts pretrained ViTs into a single runtime-configurable binary for embedded RISC-V CPUs.
  • C kernels are restructured with modified loop bounds and binary-mask control logic, enabling dynamic switching across discrete sparsity levels via compact external configuration files.
  • A hardware-aligned block pruning strategy is applied to Multi-Layer Perceptron (MLP) layers, grouping neurons into blocks of eight and pruning at the block level.
  • Attention heads are pruned per-head, and the corresponding Q/K/V slices are skipped in QKV projections and output projections.
  • Batch matrix multiplication kernels for attention score computation and value aggregation exploit head-level sparsity by skipping pruned heads and zero-initializing their output regions.
  • Partial INT8 quantization is applied to linear projection layers using TORCHAO, while nonlinear operations remain in floating point.
  • A custom MAC2 instruction is added to the RISC-V ISA, performing two parallel MAC operations with a shared input activation, reducing innermost loop instructions from 16 to 4 in dense_pack kernels.
  • Evaluation was conducted on the Oxford-IIIT Pet dataset using a Synopsys TRV32P3FX RISC-V processor, a 32-bit, 3-stage in-order core with single-precision floating-point support.
  • Latency was measured via instruction-accurate (IA) simulation on both base and ISA-extended cores, with results validated against PyTorch quantized inference.
  • ASIC synthesis (TSMC 28 nm) showed the MAC2 extension incurred 24.7% area and 5.1% power overhead, but achieved a lower normalized Area-Delay Product (ADP) than the baseline.
  • The runtime control overhead for switching configurations is less than 5%, and classifier handling overhead is approximately 57K/114K cycles for ViT-Small/Base, respectively, which is less than 10^-5 times a single inference cycle.

What the Results Mean

  • Enables efficient deployment of computationally intensive Vision Transformers on resource-constrained edge devices by reducing storage and improving inference speed and energy efficiency.
  • The runtime-adaptive nature allows dynamic adjustment of computational budgets based on changing environmental conditions (e.g., battery levels, workload, user demands) without requiring model reloads or recompilation.
  • The proposed methodology of combining structured pruning with C code transformation and custom ISA extensions offers a generalizable approach for optimizing deep learning models on embedded processors.
  • The significant reduction in on-device storage makes ViT deployment more feasible for microcontrollers with limited flash memory.
  • The energy-accuracy trade-off provided by multiple operating points allows developers to select optimal configurations based on specific application requirements.

Limitations and Open Questions

  • The group-of-8 structured pruning introduces an average accuracy drop of 2.12% over baseline FP32 per-neuron pruning.
  • INT8 quantization further leads to an additional 5.07% accuracy drop relative to FP32.
  • The current framework's classifier handling involves copying weights and bias during runtime switching, which, while negligible, is identified as an area for future optimization to fully eliminate overhead.
  • The proposed MAC2 ISA extension, while efficient, is tailored to specific input-reuse patterns in dominant projection kernels and may not generalize to all types of operations or models as broadly as general-purpose vector (RVV) or packed-SIMD (RVP) extensions.
  • Future work is needed to extend mask-based skipping to nonlinear layers and explore heterogeneous acceleration using vector or matrix compute engines, indicating current scope limitations.
  • ViT-Small exhibits larger accuracy degradation compared to ViT-Base due to lower redundancy, suggesting the effectiveness of pruning may vary with model size.

Read next

Explore all articles →