Making a Regression Reproducible

Part II · Module 25 · Chapter 25.9 · Making a Regression Reproducible

1. "It Failed Last Tuesday"

Every verification team has had this conversation. A test failed in last Tuesday's nightly regression. Someone wants to debug it on Friday. They run the test — and it passes. Or it fails differently. Or they cannot even work out which version of the design Tuesday's regression used.

Chapter 25.8 assumed that a rerun with the same seed reproduces the run. That is true only if everything else is the same too. A run's result depends on much more than its seed, and unless the runner writes all of it down at the time, Friday's rerun is a guess.

The goal of this chapter is a simple promise: for any run in any report, there is a record that says exactly what it depended on, and a script that checks today's checkout against that record before rerunning it. If everything matches, the rerun reproduces the run. If something does not, the script says what, instead of producing a confusing new result.

2. What a Run Depends On

The registry's example for this chapter records four of these:

printf {$meta} "commit=%s\nsim=%s\nseed=%d\ncmd=%s\n",
    $commit, $sim_version, $seed, "@cmd";

That is a good start, and it is far better than recording nothing. The next section measures what it misses.

3. The One-Line Record, Measured

Here is a scenario that happens all the time. On Tuesday, rand with seed 2 runs on a clean checkout and fails — it hits the adder's bug. On Wednesday, an engineer has a fix in their working tree that they have not committed yet, and runs the same test from that tree. It passes. Both runs write the one-line record. The full demonstration is in Section 9; here is the part that matters:

=== the one-line record (commit, sim, seed, cmd) of each run
  commit=61385f6c41 sim=Icarus Verilog version 13.0 (stable) (v13_0) seed=2 cmd=bin/simv +test=rand +seed=2  =>  FAIL
  commit=61385f6c41 sim=Icarus Verilog version 13.0 (stable) (v13_0) seed=2 cmd=bin/simv +test=rand +seed=2  =>  PASS

Two identical records. Opposite verdicts. Same commit, same simulator, same seed, same command. Anyone reading these on Friday has two contradictory facts and no way to choose between them. The natural conclusions — "the test is flaky", "the simulator is non-deterministic", "it depends on the machine" — are all wrong. The real difference is one character in rtl/sat_add.v, sitting in Wednesday's working tree, recorded nowhere.

4. A Commit Is Not the Design

A commit id names a snapshot that was committed. A run uses whatever is in the working tree when it builds. Most of the time those are the same, and that is exactly why the difference is so dangerous when they are not.

The record needs two more things:

• Whether the tree was clean, and if not, which files had changed. git status --porcelain prints one line per changed or untracked file and nothing at all for a clean tree, which makes it easy to test in Perl.
• The uncommitted change itself. Knowing that rtl/sat_add.v was modified is not enough to reproduce the run; you need what the modification was. git diff HEAD writes it out, and the recorder saves it next to the run as uncommitted.diff. With the commit and the diff, Wednesday's design can be rebuilt exactly.

Many teams add a policy on top: a regression whose results will be published or used for sign-off must run on a clean tree, and the runner refuses to start otherwise. Debug runs from a dirty tree are normal and useful — but their records must say they were dirty.

The repository needs one setting for this to work: the runner's own outputs — the build directory and the run areas — must be ignored by version control (here, with a two-line .gitignore). Otherwise every regression makes the tree "dirty" with its own results, and the dirty flag means nothing.

5. Hash the Inputs, Not the Artifact

A commit and a diff cover files under version control. A build often reads files that are not: generated headers, files copied from another project, anything listed in .gitignore. To be sure, the recorder also stores a content hash of every input the build reads — here, SHA-256 from Digest::SHA, which ships with Perl. Two files with the same hash have the same content, whatever their timestamps say (Chapter 17.9 explains why timestamps are a poor witness).

It is tempting to go one step further and hash the compiled result — "if the compiled simulation is identical, the run is identical". Measure that first:

#!/usr/bin/perl
# artifact_hash.pl -- build the same sources twice; is the compiled file the same?
use strict;
use warnings;
use Digest::SHA qw(sha256_hex);

sub build_and_read {
    unlink "build/sim.vvp";
    system("bin/build") == 0 or die "build failed\n";
    open my $fh, "<", "build/sim.vvp" or die; local $/; return <$fh>;
}
my $one = build_and_read();
my $two = build_and_read();
print "same sources, two builds; compiled files identical: ",
      (sha256_hex($one) eq sha256_hex($two) ? "yes" : "no"), "\n";
my @a = split /\n/, $one;
my @b = split /\n/, $two;
my ($first) = grep { $a[$_] ne $b[$_] } 0 .. $#a;
(my $line = $a[$first]) =~ s/0x[0-9a-f]+/0x<address>/g;
printf "first difference at line %d, a line of the form: %s\n", $first + 1, $line;
printf "every line that differs contains a 0x address: %s\n",
       (grep { $a[$_] ne $b[$_] && $a[$_] !~ /0x[0-9a-f]+/ } 0 .. $#a) ? "no" : "yes";

same sources, two builds; compiled files identical: no
first difference at line 10, a line of the form: S_0x<address> .scope module, "tb" "tb" 2 5;
every line that differs contains a 0x address: yes

The same sources, built twice, gave different compiled files. Every line that differs contains a memory address: the compiler writes labels such as S_0x... that come from where its data structures happened to sit in memory during that build. A hash of the compiled file changes on every build, so it can never confirm that two builds are the same, and comparing it would report a difference every time.

So the record identifies a build by what went into it — the input hashes, the tool version and the build command — never by the bytes that came out. This is the same discipline as earlier chapters' rule against publishing raw addresses or times: anything that changes between identical runs cannot be part of an identity.

6. Tool and Environment

Chapter 26.8 covered this part in depth: resolve the tool by path once, record its absolute path and the version it reports, and record the environment variables the run actually uses. This chapter's recorder keeps the version line the compiler prints, because that is what the reproduce check compares; a production recorder should keep everything Chapter 26.8's record_run.pl keeps.

One practical point from the reproduce side: compare the version string as reported, not a number parsed out of it. Two builds of "13.0" from different sources can report different strings, and that difference is exactly what you want to see.

7. The Recording Library

# recordlib.pl -- run one test and record everything needed to run it again.
use strict;
use warnings;
use Digest::SHA qw(sha256_hex);
use File::Path qw(make_path);

our @INPUTS = qw(rtl/sat_add.v tb/tb.v bin/build bin/simv);   # what the build reads

sub sha_of {                               # content hash of one file
    my ($f) = @_;
    open my $fh, "<:raw", $f or return "MISSING";
    local $/;
    return sha256_hex(<$fh>);
}
sub git { my $out = `git @_ 2>/dev/null`; chomp $out; return $out }

sub shell_quote {                          # Chapter 25.4's rule
    my ($w) = @_;
    return $w if $w =~ m{^[A-Za-z0-9_+=.,/:\@%-]+$};
    $w =~ s/'/'\\''/g;
    return "'$w'";
}

# Everything about the design and the tools that this run depends on.
sub snapshot {
    my %s;
    $s{commit} = git("rev-parse", "HEAD") || "NONE";
    my @dirty  = grep { length } split /\n/, git("status", "--porcelain");
    $s{dirty}  = @dirty ? join(",", map { (split ' ', $_, 2)[1] } @dirty) : "no";
    $s{tool}   = (`iverilog -V 2>&1` =~ /^(Icarus Verilog version [^\n]*)/m)[0] // "UNKNOWN";
    $s{"input.$_"} = sha_of($_) for @INPUTS;
    return %s;
}

# run_recorded(AREA, TEST, SEED): build, run, classify, and write AREA/ID/meta.
sub run_recorded {
    my ($area, $test, $seed) = @_;
    my $id  = "${test}_s$seed";
    my $dir = "$area/$id";
    make_path($dir);
    my %m = snapshot();
    if ($m{dirty} ne "no") {               # keep the uncommitted change itself
        system("git diff HEAD > $dir/uncommitted.diff");
        $m{diff} = "$dir/uncommitted.diff";
    }
    my @cmd = ("bin/simv", "+test=$test", "+seed=$seed");
    $m{cmd}  = join " ", map { shell_quote($_) } @cmd;
    $m{seed} = $seed;
    $m{test} = $test;

    system("bin/build") == 0 or die "build failed\n";
    my $log = `$m{cmd} 2>&1`;
    open my $lf, ">", "$dir/sim.log" or die; print {$lf} $log; close $lf;
    $m{verdict}   = $log =~ /^TEST PASSED$/m ? "PASS" : $log =~ /^TEST FAILED/m ? "FAIL" : "NO_VERDICT";
    ($m{signature}) = $log =~ /^((?:ERROR|FATAL):.*)$/m;
    $m{signature} //= "-";

    open my $mf, ">", "$dir/meta" or die "$dir/meta: $!\n";
    print {$mf} "$_=$m{$_}\n" for sort keys %m;
    close $mf or die "$dir/meta: $!\n";
    return ($dir, \%m);
}

sub read_meta {
    my ($file) = @_;
    open my $fh, "<", $file or die "$file: $!\n";
    return map { chomp; /^([^=]+)=(.*)$/ ? ($1 => $2) : () } <$fh>;
}

1;

What each part does:

• snapshot collects everything that describes the design and the tools right now: the commit, the dirty paths (or no), the tool's version line, and one input.<path> entry per input file with its SHA-256. Both the recorder and the reproduce check call it, so they can never disagree about what is being compared.
• run_recorded takes the snapshot before building, saves the uncommitted diff if there is one, writes the command shell-quoted with Chapter 25.4's rule, builds, runs, and classifies. It then writes all of it to meta, one key=value per line, keys sorted so the file is the same for the same run.
• The verdict and the first error line are part of the record. The reproduce check compares against them, so "reproduced" has a precise meaning: same verdict, same first error.
• read_meta reads the file back. Values may contain spaces and =, so the pattern takes the key up to the first = and everything after it as the value.

The format is deliberately plain text. A record that needs a special tool to read will not be read during a Friday-afternoon debug session. key=value lines can be read with cat, compared with diff, and searched with grep.

8. Check Before You Rerun

The reproduce script does the opposite of what people usually do. Instead of rerunning first and puzzling over the result, it checks first, and reruns only when every recorded dependency matches:

#!/usr/bin/perl
# reproduce.pl -- check that this checkout matches everything a run recorded,
# and only then run it again and compare.   Usage: perl reproduce.pl RUN_DIR
#   exit 0 reproduced, 1 ran but did not reproduce, 2 refused to run
use strict;
use warnings;
require "./recordlib.pl";

my $dir = shift // die "usage: reproduce.pl RUN_DIR\n";
my %was = read_meta("$dir/meta");
my %now = snapshot();
sub short { my ($v) = @_; $v //= ""; return $v =~ /^[0-9a-f]{40,}$/ ? substr($v, 0, 10) : $v }

my @different;
for my $k (sort grep { /^(?:commit|tool|input\.)/ } keys %was) {
    my $same = ($now{$k} // "") eq $was{$k};
    printf "  %-20s %s\n", $k, $same ? "same"
        : "DIFFERENT (recorded " . short($was{$k}) . ", now " . short($now{$k}) . ")";
    push @different, $k unless $same;
}
if (@different) {
    print "  refused: ", scalar(@different), " recorded input(s) differ from this checkout\n";
    print "  the run used uncommitted changes to $was{dirty}: apply $was{diff} first\n"
        if $was{dirty} ne "no";
    exit 2;
}
system("bin/build") == 0 or die "build failed\n";
my $log = `$was{cmd} 2>&1`;
my $verdict = $log =~ /^TEST PASSED$/m ? "PASS" : $log =~ /^TEST FAILED/m ? "FAIL" : "NO_VERDICT";
my ($sig)   = $log =~ /^((?:ERROR|FATAL):.*)$/m;
$sig //= "-";
my $same = $verdict eq $was{verdict} && $sig eq $was{signature};
printf "  ran: %s   recorded %s, now %s; first error %s\n", $was{cmd}, $was{verdict}, $verdict,
       $sig eq $was{signature} ? "identical" : "DIFFERENT";
print $same ? "  REPRODUCED\n" : "  NOT REPRODUCED\n";
exit($same ? 0 : 1);

• Every recorded commit, tool and input.* value is compared with today's snapshot, and each one is printed as same or DIFFERENT, with both values.
• If anything differs, it refuses — exit status 2 — and if the run had uncommitted changes, it names the saved diff that would restore them. A refusal is a useful answer: it says the rerun would not be a reproduction, and why.
• Only if everything matches does it build and run the recorded command, then compare the verdict and the first error with the record. Exit 0 means reproduced; exit 1 means everything matched and the result was still different — which, on a deterministic simulator, points at something the record does not cover yet (Section 10).

9. The Scenario, Run

The whole story, in one script. It creates a throwaway git repository from the example's own files (with fixed names and dates, so the commit id is the same every time the demonstration runs), and then plays out the week:

#!/usr/bin/perl
# repro_demo.pl -- "it passed on Wednesday": two runs of the same test, seed
# and commit, and what it takes to reproduce each of them on Friday.
use strict;
use warnings;
require "./recordlib.pl";

$| = 1;
sub step { print "\n=== $_[0]\n" }
sub sh   { system(@_) == 0 or die "failed: @_\n" }
sub reproduce { system($^X, "reproduce.pl", $_[0]); print "  (reproduce.pl exit ", $? >> 8, ")\n" }

# A throwaway repository holding this page's files. Fixed names and dates
# make the commit id the same on every run of this demonstration.
@ENV{qw(GIT_AUTHOR_NAME GIT_COMMITTER_NAME)}   = ("demo") x 2;
@ENV{qw(GIT_AUTHOR_EMAIL GIT_COMMITTER_EMAIL)} = ('demo@example.com') x 2;
@ENV{qw(GIT_AUTHOR_DATE GIT_COMMITTER_DATE)}   = ("2026-10-06T09:00:00Z") x 2;
open my $gi, ">", ".gitignore" or die; print {$gi} "build/\nruns_*/\n"; close $gi;
sh "git init -q && git add -A && git commit -q -m 'sat_add, with the 256 bug'";

step "Tuesday: a clean checkout";
my ($tue, $t) = run_recorded("runs_tue", "rand", 2);
print "  rand seed 2: $t->{verdict}\n";

step "Wednesday: a fix sits in the working tree, not yet committed";
{ local @ARGV = ("rtl/sat_add.v"); local $^I = ""; while (<>) { s/9'd256/9'd255/; print } }
my ($wed, $w) = run_recorded("runs_wed", "rand", 2);
print "  rand seed 2: $w->{verdict}\n";

step "the one-line record (commit, sim, seed, cmd) of each run";
for my $m ($t, $w) {
    printf "  commit=%.10s sim=%s seed=%d cmd=%s  =>  %s\n",
           $m->{commit}, $m->{tool}, $m->{seed}, $m->{cmd}, $m->{verdict};
}

step "the full record: what differs between the two runs";
my %tm = read_meta("$tue/meta");
my %wm = read_meta("$wed/meta");
sub brief { my ($v) = @_; return $v =~ /^[0-9a-f]{64}$/ ? substr($v, 0, 12) . " (sha256)" : $v }
for my $k (sort keys %tm) {
    next if $tm{$k} eq ($wm{$k} // "");
    printf "  %s\n      tuesday:   %s\n      wednesday: %s\n", $k, brief($tm{$k}), brief($wm{$k} // "-");
}

step "Friday, clean checkout: reproduce Wednesday's PASS";
sh "git checkout -q -- .";
reproduce($wed);

step "apply the change Wednesday's run saved, and try again";
sh "git apply $w->{diff}";
reproduce($wed);

step "back to the commit: reproduce Tuesday's FAIL";
sh "git checkout -q -- .";
reproduce($tue);

step "prove the tool check can fail: a record from another tool version";
sh "mkdir -p runs_x && cp $tue/meta runs_x/meta";
{ local @ARGV = ("runs_x/meta"); local $^I = ""; while (<>) { s/^tool=.*/tool=Icarus Verilog version 12.0 (stable)/; print } }
reproduce("runs_x");

=== Tuesday: a clean checkout
  rand seed 2: FAIL

=== Wednesday: a fix sits in the working tree, not yet committed
  rand seed 2: PASS

=== the one-line record (commit, sim, seed, cmd) of each run
  commit=61385f6c41 sim=Icarus Verilog version 13.0 (stable) (v13_0) seed=2 cmd=bin/simv +test=rand +seed=2  =>  FAIL
  commit=61385f6c41 sim=Icarus Verilog version 13.0 (stable) (v13_0) seed=2 cmd=bin/simv +test=rand +seed=2  =>  PASS

=== the full record: what differs between the two runs
  dirty
      tuesday:   no
      wednesday: rtl/sat_add.v
  input.rtl/sat_add.v
      tuesday:   5eb3a02660c0 (sha256)
      wednesday: e37fe4a2652b (sha256)
  signature
      tuesday:   ERROR: a=64 b=192 y=0 expected=255
      wednesday: -
  verdict
      tuesday:   FAIL
      wednesday: PASS

=== Friday, clean checkout: reproduce Wednesday's PASS
  commit               same
  input.bin/build      same
  input.bin/simv       same
  input.rtl/sat_add.v  DIFFERENT (recorded e37fe4a265, now 5eb3a02660)
  input.tb/tb.v        same
  tool                 same
  refused: 1 recorded input(s) differ from this checkout
  the run used uncommitted changes to rtl/sat_add.v: apply runs_wed/rand_s2/uncommitted.diff first
  (reproduce.pl exit 2)

=== apply the change Wednesday's run saved, and try again
  commit               same
  input.bin/build      same
  input.bin/simv       same
  input.rtl/sat_add.v  same
  input.tb/tb.v        same
  tool                 same
  ran: bin/simv +test=rand +seed=2   recorded PASS, now PASS; first error identical
  REPRODUCED
  (reproduce.pl exit 0)

=== back to the commit: reproduce Tuesday's FAIL
  commit               same
  input.bin/build      same
  input.bin/simv       same
  input.rtl/sat_add.v  same
  input.tb/tb.v        same
  tool                 same
  ran: bin/simv +test=rand +seed=2   recorded FAIL, now FAIL; first error identical
  REPRODUCED
  (reproduce.pl exit 0)

=== prove the tool check can fail: a record from another tool version
  commit               same
  input.bin/build      same
  input.bin/simv       same
  input.rtl/sat_add.v  same
  input.tb/tb.v        same
  tool                 DIFFERENT (recorded Icarus Verilog version 12.0 (stable), now Icarus Verilog version 13.0 (stable) (v13_0))
  refused: 1 recorded input(s) differ from this checkout
  (reproduce.pl exit 2)

Read it day by day:

• Tuesday fails, as it should: seed 2 hits the bug.
• Wednesday passes, from a working tree with the uncommitted fix.
• The one-line records are identical — the problem from Section 3.
• The full records differ in exactly the right places: dirty (no / rtl/sat_add.v), the hash of rtl/sat_add.v, and the results. The explanation is in the record; nobody has to remember anything.
• Friday, on a clean checkout, reproduce.pl refuses to rerun Wednesday's run: one input differs, and the run used uncommitted changes, saved in uncommitted.diff. Rerunning without that change would have produced a FAIL and a very confused engineer.
• After applying the saved diff, every input matches, and the rerun gives PASS with the same (absent) first error: reproduced.
• Back on the commit, Tuesday's run reproduces too: FAIL, with the identical first error line.
• The last step proves the tool check can fail. The record is copied and its tool line edited to say version 12.0, as if it had been made on a machine with an older simulator. reproduce.pl refuses. A check you have only ever seen pass has not been tested.

10. What This Record Still Cannot Capture

Be honest about the limits, because they are where the next confusing result will come from:

The rule for extending the record is simple: every time a rerun surprises you, find the dependency that explains it and add it to the snapshot. The record improves exactly where the flow has actually been bitten.

11. The Module, Assembled

Chapter 25.1 began with six obligations. Each chapter since has implemented one of them properly, and each one closed a measured failure:

And this chapter adds the property that makes the whole report useful later: every row can be reproduced, or the reason it cannot be is written down.

12. Exercises

1. Refuse a dirty tree. Add a --signoff option to a runner built on recordlib.pl that refuses to start if snapshot() reports a dirty tree, printing the changed paths. Check that .gitignore is what keeps the runner's own outputs from tripping it.

2. Derive the inputs. Replace the hand-written @INPUTS with a list derived from the build itself: parse the file names out of bin/build, or better, change the build to read a filelist and hash every file it names. Then add a file to the build without adding it to anything else, and confirm the record picks it up.

3. An environment dependency. Do Exercise 5 of Chapter 25.8 (a wrapper that reads NVEC). Run a test with and without the variable and show that reproduce.pl reports REPRODUCED for both records — wrongly. Add the variable to snapshot so that the check catches it.

4. Reproduce from a report row. Write repro_row.pl RESULTS.tsv RUN_ID, which finds the run's directory from a Chapter 25.7-style results.tsv and calls reproduce.pl on it. What has to be in results.tsv for this to work after the run directories have been archived?

5. A determinism check. Before trusting reruns at all, a team should know that the same run, twice, gives the same result. Write determinism.pl, which runs one recorded run twice in a row and compares verdicts, first errors and normalised logs. Run it on this fixture, then on a bench of your own.

13. What To Remember

• A rerun reproduces a run only if everything the run depended on is the same. Record it at the time; nobody can reconstruct it later.
• Measured: two runs with the same commit, simulator, seed and command gave FAIL and PASS. The one-line record cannot explain that; the full record can.
• A commit is not the design. Record whether the tree was dirty, and save the uncommitted diff itself.
• Hash every input the build reads. Measured: the compiled file differed between two builds of identical sources, because it contains memory addresses — never use an artifact's hash as its identity.
• Record the tool version as reported, and the environment, as Chapter 26.8 does.
• Check before rerunning. Compare every recorded dependency, refuse with a reason if any differs, and only then rerun and compare verdict and first error.
• Prove each check can fail — here, an edited tool line was refused.
• When a rerun surprises you, add the dependency that explains it to the record.

That completes Module 25: a runner that selects, launches, bounds, collects, reports and exits honestly — and whose every result can be run again. Module 26 places that runner inside a real flow, where Bash calls it, Tcl drives the tools around it, and every hand-off between languages has to preserve the same honesty about exit status and evidence.