A green regression needs a complete report

Run three pinned RTL projects and reject incomplete, stale, or inconsistent evidence before reporting success.

Checks passed

Public demonstration

This is not a client result. The recorded command and check status appear below.

Recorded outcome

All three pinned regressions return their complete expected results. Fourteen infrastructure tests confirm that incomplete or invalid evidence is rejected. A reported pass depends on both the process outcome and the checked report.

What was checked

Problem: A command can exit successfully while its report is incomplete or belongs to an older source revision. Counting only the available passing rows can hide tests that never ran.
Scope: One local, sequential Python runner orchestrating the public buffer, reset, and arbiter examples at immutable commits. The complete expected matrices contain 42 outcomes, including 14 expected failures from deliberately introduced RTL faults.
Approach: The runner fetches each pinned revision into a new directory, executes its tests, and validates the exact result matrix and required source hashes. Separate fixtures test missing executables, timeouts, malformed or stale reports, duplicate or missing cases, and incorrect source hashes.
Evidence: Each invocation keeps a manifest with revisions, commands, durations, validation decisions, and per-project logs. The public repository includes an independently recorded run. Unit fixtures check the infrastructure separately from the RTL mutants.

Scope and limits

What these checks do not establish.

The runner checks finite evidence from the pinned examples. It does not establish a formal proof or independently prove the upstream testbenches correct.
Separate working directories are not a security sandbox for untrusted code.
Process group limits apply to each launched command. Nested tools that start their own sessions rely on their own timeouts.
This example is a local sequential runner, not a hosted queue, access control system, or shared compute service. Timings include network and tool work and are not client estimates.

Reproduce and inspect

Recorded verification status

Rerun from public GitHub revision 84cdd86 on 15 September 2026. All three projects reproduce their complete 8, 18, and 16 outcome matrices. Fourteen infrastructure tests pass. Linux GitHub Actions also passes at this revision.

Recorded commandverification-automation

git clone https://github.com/Rivoryxa-Technologies/verification-automation
cd verification-automation
# Requires Git, Python 3, make, Icarus Verilog, and network access.
make test

Filestools/run.pypinned execution and report validationtools/test_runner.pyinfrastructure failure fixturesrecorded/2026-09-15/manifest.jsonindependently recorded resultrecorded/2026-09-15/jobs/upstream summaries and raw logs.github/workflows/verification.ymlLinux regression

Next steps

Service: Verification automation Other examples

Want this method on your design?

Share the problem in a few lines. We will discuss the scope before any confidential project material changes hands.

Discuss a verification problem