A RISC-V Simulator for the Tenstorrent Tensix Architecture

AI Analysis

Abstract

Key Highlights

• Wolfpine is an open-source RISC-V simulator for Tenstorrent Tensix (Wormhole and Blackhole) architectures.
• Developed exclusively from publicly available documentation, open-source software, and hardware measurements.
• Integrates seamlessly with the TT-Metal software stack, allowing unmodified TT-Metal programs to run.
• Written in pure Python, prioritizing readability, ease of modification, and extensibility.
• Features an opt-in, provenance-tracked cost model for incremental timing fidelity.
• Provides extensive observability and diagnostics through typed event streams, per-component tracing, and timeline export.
• Supports multi-architecture configurations via profiles, demonstrated with Wormhole and Blackhole generations.

Technical Details

• Wolfpine's architecture is based on active (clock-driven) and passive (on-demand) IP blocks, composed into tiles and devices.
• Memory accesses are handled by a lookup arbitrator supporting layered address spaces, making remoteness invisible to processing elements.
• The simulator is cycle-driven, with state advancing at cycle granularity and deterministic ordering, but timing fidelity is added incrementally via an opt-in cost model.
• Integration with TT-Metal is achieved by reverse-engineering its socket-based wire protocol, enabling transparent execution of TT-Metal programs.
• Hardware probes (perfbench) were developed to characterize undocumented hardware aspects and calibrate the simulator's timing model.
• Performance optimizations include tile idle gating (55x speedup), firmware-loop parking (484x speedup), and deadline striding.
• The cost model uses declarative tables of per-unit cycle costs, accounting for pipeline stalls and NoC latency/bandwidth, with stall costs attributed to their documented or measured stall locations.
• Observability is provided through a typed publish-subscribe event bus, tracking events like instruction retirement, Tensix dispatches, memory accesses, and NoC transactions.
• Diagnostic outputs include line-delimited JSON, Perfetto-compatible timelines, Spike-compatible commit logs, Parquet datasets, Cachegrind profiles, and LCOV source-level attribution.
• Multi-architecture support is implemented using ArchProfile objects, which parameterize hardware details such as tile grid coordinates, memory map ranges, NoC parameters, and ISA extensions.
• Tensix compute tiles comprise L1 SRAM, five 32-bit RISC-V 'baby cores' (BRISC, NCRISC, TRISC), and a Tensix co-processor with specialized units (unpacker, matrix unit, vector unit, packer, THCON, synchronization unit).
• Programming of the Tensix co-processor is done via memory-mapped ranges and a bespoke RISC-V .ttinsn instruction extension.

Implications

• Enables offline code development and experimentation for Tenstorrent Tensix architectures, reducing reliance on physical hardware access.
• Provides advanced introspection and debugging capabilities for performance tuning, offering insights beyond those available on physical hardware or the vendor's simulator.
• Serves as an independent reference simulator, validating the completeness and openness of Tenstorrent's documentation and software stack.
• Its extensible, modular design and Python implementation lower the barrier for community contributions and future support for additional RISC-V accelerators.
• The methodology of constructing a simulator solely from open sources demonstrates a repeatable approach for other RISC-V architectures where documentation and source code are available.

Limitations

• Wolfpine, being written in Python, is significantly slower than Tenstorrent's C++ ttsim (4x to 20x slower) and physical hardware (3.5x to 192x slower).
• The cost model for the RISC-V integer division (divu) operation currently assumes the low end of its variable cycle range (6 cycles), leading to an 82% error for this specific operation.
• Detailed hardware traces are not publicly available, limiting the ability to validate Tensix per-instruction costs at a fine granularity.
• The current evaluation does not cover multi-chip configurations, as ethernet tiles are not yet modeled.
• The deadline striding optimization is less effective when many tiles are concurrently active, as it can only skip cycles when every tile is idle.
• The simulator does not model hardware jitter, which can lead to fractional cycle counts in hardware measurements that are not reflected in Wolfpine's integer cycle predictions.
• The simulator's performance can be impacted by modeling packet flight, flit-level link occupancy, and contention in NoC transactions, which Tenstorrent's ttsim handles by immediate data copying.