V Questions with… DAMO Academy’s Dr. Jianyi Meng

AI Analysis

Abstract

Key Highlights

• Achieved over 30 tokens per second (approximately 20–22 words per second) running the Qwen3.8-27B LLM on a 64-core RISC-V C950 processor.
• The AI workload was executed natively on the CPU without a separate GPU or NPU for acceleration.
• The C950 processor adheres to the RVA23 profile and incorporates a proprietary matrix acceleration extension.
• The proprietary matrix extension is being advanced through the RISC-V standards process as the AME (Attached Matrix Extension).
• Dedicated matrix registers are utilized to prevent contention with general-purpose vector operations and support mixed-precision accumulation.

Technical Details

• The Qwen3.8-27B is a large language model with 27 billion parameters.
• The C950 is a 64-core RISC-V-based processor, designed as a server-class 64-bit superscalar CPU.
• Achieving 30 tokens per second requires a memory architecture capable of moving approximately 400 gigabytes of data per second.
• The C950 meets the RVA23 profile, which mandates the RISC-V Vector extension (RVV 1.0).
• XuanTie developed a custom proprietary extension to accelerate matrix math, which is critical for LLMs.
• The AME task group was initiated to refine and standardize this matrix extension.
• The biggest technical challenge was co-designing quantization, matrix instructions, on-chip memory, memory bandwidth, and kernel-level optimization as a single system.
• Dedicated tile and accumulator registers are used for matrix operations to avoid increasing vector register pressure and to support mixed-precision accumulation (e.g., INT4/INT8 into INT32, FP4/FP8/FP16/BF16 into FP32).
• The matrix extension design allows compilers and the operating system to enable matrix state on demand, reducing context-switching overhead.

Implications

• Demonstrates RISC-V's growing capability to handle demanding AI workloads, such as large language models, directly on the CPU.
• The C950 architecture provides both immediate performance benefits and a pathway into a more unified RISC-V software ecosystem.
• The standardization of the AME will provide a unified software interface for matrix operations, fostering broader industry adoption and collaboration.
• Encourages further development of high-performance data center ecosystems for RISC-V, including complex software stacks and runtime adaptations like Java.
• Highlights the necessity for a complete software chain, including optimized operator libraries and updated compilers, to fully leverage RISC-V's potential as an xPU.
• RISC-V's open ISA offers unique opportunities for architectural innovation, enabling the exploration of new extensions and compute paradigms beyond existing architectures.

Limitations

• Software optimization for high-performance and AI workloads on RISC-V is still in a maturing phase.
• Full RISC-V adoption in high-performance data center ecosystems still requires significant development.
• The initial achievement relied on a proprietary matrix extension, though efforts are underway for its standardization.
• Raw compute power alone is insufficient; the challenge lies in balancing computation, data movement, and software optimization.
• A comprehensive standard for matrix extensions must address complex aspects like matrix dimensions, data layout, rounding, saturation, data movement, context management, exceptions, virtualization, and compiler/ABI mapping, which is an ongoing effort.
• The rapid evolution of capabilities like FP8, INT4, and sparsity, alongside algorithms, requires a standard that offers a unified software interface while allowing for innovation and microarchitectural evolution.