DataMaestro: A Versatile and Efficient Data Streaming Engine Bringing Decoupled Memory Access To Dataflow Accelerators

Abstract

DataMaestro is a novel data streaming engine that applies a decoupled access/execute architecture to Deep Neural Network (DNN) dataflow accelerators to mitigate performance bottlenecks caused by data movement. It features programmable access patterns, fine-grained prefetching to avoid conflicts, and on-the-fly data manipulation for efficiency. Integrated into a RISC-V host system, DataMaestro enabled the GeMM core to achieve nearly 100% utilization, showing 1.05x to 21.39x better performance than state-of-the-art solutions while adding minimal area and energy overhead.

Report

Key Highlights

• Core Innovation: DataMaestro introduces a decoupled access/execute architecture specifically tailored for DNN dataflow accelerators to resolve severe data movement bottlenecks.
• Performance Gain: The engine boosts utilization of the integrated General Matrix Multiplication (GeMM) core to nearly 100%.
• Benchmark Results: DataMaestro achieves substantial speedups, performing 1.05x to 21.39x better than other state-of-the-art solutions.
• Efficiency: Despite the performance uplift, the hardware overhead is minimal, accounting for only 6.43% of the total system area and 15.06% of the total energy consumption.

Technical Details

• Architecture: DataMaestro functions as a data streaming unit, leveraging the Decoupled Access/Execute (DAE) model to separate memory operations from compute operations.
• Flexibility: It supports flexible and programmable memory access patterns, allowing it to accommodate a diverse range of DNN workload types and dataflows.
• Bank Conflict Mitigation: The design incorporates fine-grained prefetch mechanisms and dynamic addressing mode switching to effectively alleviate memory bank conflicts.
• Data Manipulation: It enables customizable on-the-fly data manipulation, a technique used to reduce both memory footprints and the required access counts.
• Evaluation Setup: Five DataMaestro units were integrated alongside a Tensor Core-like GeMM accelerator and a dedicated Quantization accelerator within a RISC-V host system for comprehensive evaluation.
• Implementation: The system was validated using both an FPGA prototype and standard VLSI synthesis results.

Implications

• Addressing the Memory Wall: DataMaestro offers a high-impact solution to the memory bandwidth and movement limitations that plague modern AI hardware, ensuring that expensive computational units (like GeMM cores) are saturated with data.
• Advancing RISC-V Acceleration: The successful integration and superior performance results within a RISC-V host system validate RISC-V as a robust platform for implementing highly sophisticated, domain-specific accelerators (DSAs) focused on streaming data.
• Architectural Blueprint: This work establishes the DAE model as a critical architectural component for future energy-efficient, high-performance dataflow accelerators, potentially influencing standard interfaces and hardware design practices in the broader tech ecosystem.
• Versatility for AI: The streaming unit’s ability to handle diverse and programmable dataflows suggests that it can be a foundational component for generalized, yet highly efficient, AI accelerators capable of managing complex, varied DNN models.