SPEC CPU2026 Under the Microscope: Programs, Phases, and Samples

AI Analysis

Abstract

Key Highlights

• Program-level analysis shows integer benchmarks are often frontend bound, while floating-point benchmarks are frequently memory bound, with compiler versions capable of halving or nearly doubling individual benchmark IPC.
• Phase-level characterization reveals that over half of the SPECrate inputs exhibit time-varying behavior, with IPC varying by more than an order of magnitude within a single input run.
• SimPoint, a common sampled simulation tool, is accurate for steady inputs but shows errors up to 43% on multi-phase and bursty inputs, and its accuracy can vary by 58 percentage points due to random seed changes.
• The root cause of SimPoint's inaccuracy is identified as the selection of representatives, not the clustering, as an oracle selection method reduces error to within 2.6% for all inputs.
• Practical recommendations are provided for using SimPoint with CPU2026, including pre-classifying inputs by phase behavior and running multiple random seeds for non-steady workloads.

Technical Details

• The study characterized SPEC CPU2026 on three platforms: Intel Sapphire Rapids (x86), Graviton4 (Arm), and SpacemiT K3 (RISC-V), using GCC 15.2.0 and GCC 13.3.1 compilers.
• Hardware counters were measured at whole-run, 10ms, and 100 million instruction intervals to capture program behavior at different granularities.
• For the program-level analysis, the authors reported whole-run metrics for all 52 benchmarks and classified them by dominant top-down bottlenecks (frontend, bad speculation, memory, core, or balanced).
• Workload scaling analysis showed that 'train' workloads, while smaller, often do not accurately represent 'ref' workloads for cache and branch behavior, despite similar average IPCs.
• Cross-platform comparisons revealed that Arm and RISC-V platforms executed 25% more instructions on average than x86 for floating-point benchmarks, making per-instruction metrics less comparable.
• Phase-level characterization of 90 SPECrate inputs on x86 classified them as steady (38 inputs), multi-phase (35 inputs), or bursty (17 inputs) based on IPC variation and slow interval time share.
• Slow intervals (IPC below 0.3) were found to take significantly more run time than their instruction share, with ratios ranging from 3.0 to 59 for some inputs.
• SimPoint 3.2 was validated against per-interval hardware CPI measurements, with default settings (K up to 30 clusters) and across five random seeds.
• An oracle experiment, replacing SimPoint's representatives with the cluster member closest to the cluster's cycle-weighted CPI, reduced the maximum error to 2.6%.

Implications

• Computer architecture researchers should be cautious when relying solely on whole-run average metrics for SPEC CPU2026, as they can mask significant time-varying behavior critical for accurate evaluation.
• Using smaller 'test' or 'train' workloads as proxies for 'ref' in detailed simulation studies, especially for memory hierarchy or branch prediction, can lead to inaccurate conclusions due to differing behaviors.
• The choice of compiler significantly impacts benchmark performance and instruction count, meaning IPC values from studies using different compilers are not directly comparable.
• SimPoint's unreliability for multi-phase and bursty benchmarks necessitates a more robust sampling strategy, as a single run with default settings can yield highly variable and inaccurate results.
• The findings highlight the need for improved representative selection mechanisms in sampled simulation tools like SimPoint, potentially by incorporating performance metrics alongside code signatures during clustering.

Limitations

• The cross-platform comparison of instruction count and IPC does not differentiate between differences arising from the Instruction Set Architecture (ISA) and those from the microarchitecture.
• The phase classification thresholds (e.g., 1.3x IPC ratio for steady, 2% slow interval time share for bursty) are heuristic, although sensitivity analysis indicated robustness.
• The sampling study focused exclusively on single-threaded SPECrate benchmarks and SimPoint, thus its conclusions may not extend to multi-threaded applications or other sampled simulation methods.
• The validation of SimPoint used hardware measurements as ground truth to isolate sampling error from simulator modeling error, but did not involve full cycle-level simulations.
• One benchmark, 766.femflow_r, failed to compile on the RISC-V platform, resulting in incomplete coverage for RISC-V results.
• The study's findings are specific to SPEC CPU2026 and the tested hardware/compiler configurations, and may not generalize to other benchmark suites or system setups.