Just TestIt! An SBST Approach To Automate System-Integration Testing

Abstract

This paper introduces TestIt, an open-source Python package designed to automate full-system integration testing using a Software-Based Self-Test (SBST) approach. TestIt dynamically generates test vectors and golden references, significantly reducing development complexity and enabling CI/CD methodologies for open-source RTL. A case study on the X-HEEP RISC-V microcontroller demonstrated TestIt's ability to detect faults and achieve an 11x speed-up when run on an FPGA compared to RTL simulation.

Report

Key Highlights

• TestIt Framework: An open-source Python package dedicated to automating system-integration testing.
• Methodology: Employs a Software-Based Self-Test (SBST) approach to automate the generation of test vectors and golden reference outputs.
• Efficiency Gains: Successfully demonstrated an 11x speed-up in testing time when utilizing the PYNQ-Z2 FPGA development board compared to traditional Register-Transfer Level (RTL) simulations.
• Fault Detection: The system proved capable of identifying hardware and software faults within integrated systems that traditional formal verification methods might fail to catch.
• CI/CD Enablement: Designed specifically to facilitate the widespread adoption of Continuous Integration/Continuous Deployment (CI/CD) practices in open-source RTL development.

Technical Details

• Tool Architecture: TestIt is implemented as an open-source Python package, ensuring accessibility and ease of integration into existing development workflows.
• Core Function: Automates the testing process by dynamically generating the necessary input stimuli (test vectors) and calculating the expected output (golden references).
• Supported Environments: The framework is flexible, supporting both traditional software simulation environments and rapid prototyping on hardware, specifically FPGAs.
• Case Study Target: The effectiveness of TestIt was validated using the X-HEEP RISC-V microcontroller (MCU), a system featuring a main CPU core integrated with a custom accelerator.
• Hardware Platform: Testing was accelerated using the PYNQ-Z2 FPGA development board for high-speed execution compared to software-based simulation.

Implications

• Accelerating RISC-V Validation: By providing a structured, automated SBST framework, TestIt significantly reduces the validation cycle time for complex RISC-V SoCs and MCUs (like X-HEEP), potentially accelerating time-to-market.
• Democratizing CI/CD for Hardware: The open-source nature and automated capabilities of TestIt directly enable smaller teams and open-source projects in the RISC-V ecosystem to implement robust, modern CI/CD pipelines, improving code quality and stability.
• Integration Quality: The focus on system-integration testing addresses a critical challenge in modular hardware development. By detecting faults missed by formal methods at the integration layer, TestIt ensures higher overall system reliability for custom RISC-V extensions and accelerators.
• Performance Characterization: The framework's ability to characterize system performance with minimal effort aids developers in optimizing new RISC-V architectures and custom instruction sets rapidly during the development phase.