Re-Running Only the Failures

Part II · Module 25 · Chapter 25.8 · Re-Running Only the Failures

1. The Debug Loop

A regression report lists what failed. What an engineer does next is almost always the same loop:

1. Rerun the failures exactly as they ran — same test, same seed. If they fail again in the same way, the failure is real and repeatable, and debugging can start.
2. Debug and fix.
3. Rerun the same failures again, on the fixed design. If they pass, the fix addresses at least those cases.
4. Run the whole regression before declaring victory, because a fix can break something else.

Steps 1 and 3 are this chapter. They need one thing the earlier chapters made sure of: a report that records every run with its seed. Chapter 25.7's results.tsv has one line per planned run, with its test, seed, verdict and category. Everything below starts from that file.

2. A Compact Runner for This Chapter

To show the loop end to end, this chapter needs to run several small regressions in a row. Here is a compact local runner that does in one file what Chapters 25.1 to 25.7 did step by step: it reads a plan, runs each row with a time limit, and writes results.tsv. It also records each failing run's signature — its first ERROR or FATAL line — so that a rerun can be compared with the original:

#!/usr/bin/perl
# regress.pl -- a compact local runner: Chapters 25.1 to 25.7 in one file.
# Usage: perl regress.pl REGRESSION_ID PLAN.tsv
# Runs every row of the plan (run_id, test, seed) with a time limit and writes
# runs_<id>/results.tsv with a verdict, a category and the first error line.
use strict;
use warnings;
use POSIX qw(WNOHANG);
use File::Path qw(make_path remove_tree);
use Time::HiRes qw(time sleep);

my ($REGR, $PLAN) = @ARGV;
die "usage: regress.pl REGRESSION_ID PLAN.tsv\n" unless $REGR && $PLAN;
my $LIMIT = 1;                                    # seconds per run
my $area  = "runs_$REGR";

open my $p, "<", $PLAN or die "$PLAN: $!\n";
<$p>;
my @plan = map { chomp; my ($id, $t, $s) = split /\t/; { id => $id, test => $t, seed => $s } } <$p>;

system("bin/build") == 0 or die "build failed\n";
system("bin/simv +test=smoke > /dev/null");      # warm-up: the first launch is the slowest
remove_tree($area);
for my $r (@plan) {
    my $dir = "$area/$r->{id}";
    make_path($dir);
    my $pid = fork // die "fork: $!\n";
    if ($pid == 0) {
        open STDOUT, ">", "$dir/sim.log" or exit 126;
        open STDERR, ">&", \*STDOUT      or exit 126;
        exec "bin/simv", "+test=$r->{test}", "+seed=$r->{seed}";
        exit 127;
    }
    my $deadline = time + $LIMIT;
    until (waitpid($pid, WNOHANG) == $pid) {
        if (time >= $deadline && !$r->{timed_out}) { $r->{timed_out} = 1; kill "TERM", $pid }
        sleep 0.02;
    }
    my ($pass, $first, $errors) = (0, "", 0);
    open my $log, "<", "$dir/sim.log" or die;
    while (<$log>) {
        chomp;
        $pass = 1 if /^TEST PASSED$/;
        if (/^(?:ERROR|FATAL):/) { $errors++; $first ||= $_ }
    }
    @$r{qw(verdict cat sig)} =
          $r->{timed_out} ? ("TIMEOUT", "NO VERDICT", "-")
        : $errors         ? ($first =~ /^FATAL/ ? "FATAL" : "FAIL", "FAIL", $first)
        : $pass           ? ("PASS", "PASS", "-")
        :                   ("NO_VERDICT", "NO VERDICT", "-");
}

open my $out, ">", "$area/results.tsv" or die;
print {$out} "run_id\ttest\tseed\tverdict\tcategory\tsignature\n";
print {$out} join("\t", @$_{qw(id test seed verdict cat sig)}), "\n" for @plan;
close $out;
my %n; $n{ $_->{cat} }++ for @plan;
printf "%s: %d runs  PASS %d  FAIL %d  NO VERDICT %d  -> %s/results.tsv\n", $REGR, scalar @plan,
       map({ $n{$_} // 0 } "PASS", "FAIL", "NO VERDICT"), $area;

This runner is deliberately simple — local, sequential, a one-second limit, no farm — so that the chapter can focus on what happens between regressions. The plan for last night's regression is a plain table:

run_id	test	seed
smoke_s1	smoke	1
corner_s1	corner	1
rand_s1	rand	1
rand_s2	rand	2
rand_s3	rand	3
rand_s4	rand	4
rand_s5	rand	5
rand_s6	rand	6
stall_s1	stall	1

3. The Retry List Keyed by Test, Measured

The registry's example for this chapter rebuilds the retry list from a hash of previous verdicts:

#!/usr/bin/perl
# naive_retry.pl -- the registry's retry idiom: previous results keyed by TEST.
use strict;
use warnings;

my (%prev, %seed_of);
open my $fh, "<", "runs_n1/results.tsv" or die "runs_n1/results.tsv: $!\n";
<$fh>;
while (<$fh>) {
    my ($id, $test, $seed, $verdict) = split /\t/;
    $prev{$test}    = $verdict;
    $seed_of{$test} = $seed;
}
my @runs;
my @retry = grep { $prev{$_} ne 'PASS' } sort keys %prev;
push @runs, { test => $_, seed => $seed_of{$_} } for @retry;
print "retry list: ", join(", ", map { "$_->{test} seed $_->{seed}" } @runs), "\n";

Its result, after last night's regression, is the second step of the full demonstration in Section 9:

retry list: corner seed 1, stall seed 1

Last night, three runs failed — corner_s1, rand_s2 and rand_s4 — and one timed out. The retry list has two entries. Both failures of the random test are missing.

The cause is the key. %prev and %seed_of are keyed by test name, and rand ran six times. Each row for rand overwrote the previous one, so after the loop the hash holds only the last rand row: seed 6, which passed. rand is not "not PASS", so it is not retried at all. Seeds 2 and 4 — the two runs that actually found the bug — left no trace in the retry list, and nothing printed a warning.

It is worth being precise about how bad this is. If the last row for rand had happened to fail, the list would have contained rand with that seed only, and the person debugging would have no idea that other seeds failed too. Either way, a hash keyed by test can hold one result per test, and a regression holds many results per test. The data does not fit the structure, so the structure silently keeps the last one.

4. Key by Run

The fix is to key by the thing that is unique: the run. Chapter 25.3 gave every run an id that includes its seed (rand_s2), and the report has one line per run id. A retry plan is then just: every run whose category is not PASS, with its id, test and original seed:

#!/usr/bin/perl
# retry_plan.pl -- build a plan of every run that did not pass, keyed by RUN,
# with its original seed.   Usage: perl retry_plan.pl RESULTS.tsv OUT.tsv
use strict;
use warnings;

my ($in, $out) = @ARGV;
open my $fh, "<", $in or die "$in: $!\n";
my @col = split /\t/, scalar(<$fh>) =~ s/\s+\z//r;           # columns by name
my %at  = map { $col[$_] => $_ } 0 .. $#col;
for my $need (qw(run_id test seed verdict category)) {
    die "$in: no '$need' column\n" unless exists $at{$need};
}
my (@retry, $total);
while (my $l = <$fh>) {
    chomp $l;
    my @f = split /\t/, $l;
    $total++;
    next if $f[ $at{category} ] eq "PASS";
    push @retry, [ @f[ @at{qw(run_id test seed verdict)} ] ];
}
open my $o, ">", $out or die "$out: $!\n";
print {$o} "run_id\ttest\tseed\n";
print {$o} join("\t", @$_[0 .. 2]), "\n" for @retry;
close $o or die "$out: $!\n";
printf "%d of %d runs did not pass; plan written to %s:\n", scalar @retry, $total, $out;
printf "  %-10s seed %-3s (was %s)\n", @$_[0, 2, 3] for @retry;

It writes the retry plan in exactly the same format as the original plan (run_id, test, seed), so the same runner can run it. The runs keep their original ids, which is what makes the next step possible: comparing each rerun with its first result, line by line.

The plan includes runs whose category is NO VERDICT as well as FAIL. A timed-out run has no verdict yet, and rerunning it is how you get one — or confirm that it hangs every time. The plan prints the previous verdict next to each run, so the person reading it knows why each one is there.

5. Reading the Previous Report Safely

retry_plan.pl reads results.tsv, a file written by another program, possibly last week, possibly by an older version of the runner. Three habits keep that safe:

• Find columns by name from the header line, not by position. If a later version of the collector adds a column in the middle, a positional reader silently reads the wrong field — for example, taking the reason text as the seed. retry_plan.pl builds %at from the header and refuses to run if any column it needs is missing.
• Remove the line ending before splitting (s/\s+\z//r on the header, chomp on each row), so that the last column does not carry a newline into a comparison.
• Never invent a seed. If the report has no seed for a run, that run cannot be retried as it ran. The right response is to say so, not to substitute a new one. (Section 7 shows why.)

6. What a Rerun Can Show

Running the retry plan produces a second results.tsv with the same run ids. compare.pl puts the two side by side and gives each run one of five outcomes:

#!/usr/bin/perl
# compare.pl -- what happened to each rerun, against its first result.
# Usage: perl compare.pl BEFORE/results.tsv AFTER/results.tsv
use strict;
use warnings;

sub load {                                  # run_id => { column => value }
    my ($file) = @_;
    open my $fh, "<", $file or die "$file: $!\n";
    my @col = split /\t/, scalar(<$fh>) =~ s/\s+\z//r;
    my %row;
    while (<$fh>) {
        chomp;
        my %r; @r{@col} = split /\t/;
        $row{ $r{run_id} } = \%r;
    }
    return \%row;
}
my ($was, $now) = map { load($_) } @ARGV;

my %n;
for my $id (sort keys %$now) {
    my ($a0, $b0) = ($was->{$id}, $now->{$id});
    die "$id is in the rerun but not in the first results\n" unless $a0;
    die "$id: seed changed from $a0->{seed} to $b0->{seed}\n" if $a0->{seed} ne $b0->{seed};
    my $outcome =
        $b0->{category} eq "PASS"                          ? "NOW PASSES"
      : $a0->{category} eq "FAIL" && $b0->{category} eq "FAIL"
                                                           ? ($a0->{signature} eq $b0->{signature}
                                                              ? "REPRODUCED" : "FAILS DIFFERENTLY")
      : $a0->{verdict} eq $b0->{verdict}                   ? "SAME AGAIN"
      :                                                      "CHANGED";
    printf "  %-10s %-8s -> %-8s %s\n", $id, $a0->{verdict}, $b0->{verdict}, $outcome;
    $n{$outcome}++;
}
print "  ", join("   ", map { "$_ $n{$_}" } sort keys %n), "\n";

Two safety checks are built in. compare.pl refuses a run that is in the rerun but not in the original results, and it refuses a run whose seed changed — because comparing two different seeds is not a rerun, and calling it one would turn the comparison into noise.

Comparing first-error lines exactly works here because the simulator is deterministic and the error message contains no time stamps. With real simulators, compare a normalised signature — the message with times, paths and instance numbers removed — which is what Chapter 21.9 builds.

A note on NOW PASSES without a design change: with a deterministic simulator and the same seed, the same design gives the same result, so a run that "now passes" on an unchanged design means something else changed — the tool version, a file the bench reads, an environment variable, a different machine. Chapter 25.9 records exactly those things, so that this situation can be explained rather than shrugged at.

7. New Seeds Are Not a Retry

A tempting shortcut is to "retry" failures by simply running the same tests again, letting the runner pick seeds as usual. Section 9's step 4 does that, on the unchanged, still-broken design:

x1: 3 runs  PASS 2  FAIL 1  NO VERDICT 0  -> runs_x1/results.tsv
  corner_s9  FAIL
  rand_s9    PASS
  rand_s10   PASS

Both rand runs passed. The design had not changed. The bug was still there. With new seeds, the random test simply did not happen to produce a sum of exactly 256 — just as seeds 1, 3, 5 and 6 did not the night before.

A person looking at this result could easily conclude that the rand failures were "flaky", "transient", or "fixed by someone else", and close the bug. All three conclusions are wrong. The only correct statement is: these two new seeds did not reach the bug. Chapter 25.3 measured the same effect across nights: a night with no failures does not mean the bug is fixed.

corner failed with a new seed too — because corner does not use the seed. Its stimulus is fixed, so it fails every time. That contrast is useful: a failure that survives a seed change is deterministic; a failure that disappears with a new seed is seed-dependent, and only its original seed reproduces it. Neither is "flaky" on a deterministic simulator.

So the retry must use the original seed. New seeds belong in the next regression, where they look for new failures — not in the retry, where the job is to look at the old ones again.

8. Confirming a Fix

The bug in the fixture is one character: the adder saturates only when the sum is greater than 256, so a sum of exactly 256 wraps to 0. Here is the fix:

// fix/sat_add.v -- 8-bit saturating adder: y = min(a + b, 255).
// The fix for the bug the Module 25 regressions found.
module sat_add (
    input  wire [7:0] a,
    input  wire [7:0] b,
    output wire [7:0] y
);
    wire [8:0] sum = a + b;
    // FIXED: saturate whenever the sum does not fit in 8 bits.
    assign y = (sum > 9'd255) ? 8'hFF : sum[7:0];
endmodule

After the fix, the same retry plan, with the same seeds, is rerun and compared with the original results (Section 9, step 5): all three failures report NOW PASSES, and the hung test reports SAME AGAIN.

That is good news, and it proves exactly this much: the three runs that failed now pass. It does not prove the bug is gone for every input, and it does not prove the fix broke nothing else. That is what step 6 is for: the whole regression, on the fixed design. Here it gives 8 passes and one no-verdict — stall, which hangs by design in this module's bench and would need its own fix.

In a real project, step 6 would also use new seeds, so that the fixed design is tested on inputs it has not seen; and the bug's exact condition (a sum of exactly 256) would deserve a directed test of its own, so that it is checked every time without relying on luck. In this fixture, corner already is that directed test.

9. The Whole Loop, Run

Here is everything above as one script — last night's regression, the two retry lists, the rerun, the new-seeds trap, the fix and the full regression:

#!/usr/bin/perl
# rerun_demo.pl -- a nightly regression, then the debug loop that follows it.
use strict;
use warnings;
use File::Copy qw(copy);

$| = 1;
sub step { print "\n=== $_[0]\n" }
sub run  { system($^X, @_) == 0 or die "$_[0] failed\n" }
sub show {                                  # print one results file's rows
    open my $fh, "<", $_[0] or die "$_[0]: $!\n";
    <$fh>;
    while (<$fh>) { my ($id, $t, $s, $v) = split /\t/; printf "  %-10s %s\n", $id, $v }
}

step "1. last night's regression";
run "regress.pl", "n1", "nightly.tsv";

step "2. the retry list, keyed by test";
run "naive_retry.pl";

step "3. the retry plan, keyed by run, original seeds";
run "retry_plan.pl", "runs_n1/results.tsv", "rerun.tsv";
run "regress.pl", "r1", "rerun.tsv";
run "compare.pl", "runs_n1/results.tsv", "runs_r1/results.tsv";

step "4. 'retrying' with new seeds instead (design unchanged)";
open my $ns, ">", "newseeds.tsv" or die;
print {$ns} "run_id\ttest\tseed\ncorner_s9\tcorner\t9\nrand_s9\trand\t9\nrand_s10\trand\t10\n";
close $ns;
run "regress.pl", "x1", "newseeds.tsv";
show "runs_x1/results.tsv";

step "5. apply the one-line fix, rerun the same plan";
copy("fix/sat_add.v", "rtl/sat_add.v") or die "copy: $!\n";
run "regress.pl", "r2", "rerun.tsv";
run "compare.pl", "runs_n1/results.tsv", "runs_r2/results.tsv";

step "6. the whole nightly again, on the fixed design";
run "regress.pl", "n2", "nightly.tsv";

=== 1. last night's regression
n1: 9 runs  PASS 5  FAIL 3  NO VERDICT 1  -> runs_n1/results.tsv

=== 2. the retry list, keyed by test
retry list: corner seed 1, stall seed 1

=== 3. the retry plan, keyed by run, original seeds
4 of 9 runs did not pass; plan written to rerun.tsv:
  corner_s1  seed 1   (was FAIL)
  rand_s2    seed 2   (was FAIL)
  rand_s4    seed 4   (was FAIL)
  stall_s1   seed 1   (was TIMEOUT)
r1: 4 runs  PASS 0  FAIL 3  NO VERDICT 1  -> runs_r1/results.tsv
  corner_s1  FAIL     -> FAIL     REPRODUCED
  rand_s2    FAIL     -> FAIL     REPRODUCED
  rand_s4    FAIL     -> FAIL     REPRODUCED
  stall_s1   TIMEOUT  -> TIMEOUT  SAME AGAIN
  REPRODUCED 3   SAME AGAIN 1

=== 4. 'retrying' with new seeds instead (design unchanged)
x1: 3 runs  PASS 2  FAIL 1  NO VERDICT 0  -> runs_x1/results.tsv
  corner_s9  FAIL
  rand_s9    PASS
  rand_s10   PASS

=== 5. apply the one-line fix, rerun the same plan
r2: 4 runs  PASS 3  FAIL 0  NO VERDICT 1  -> runs_r2/results.tsv
  corner_s1  FAIL     -> PASS     NOW PASSES
  rand_s2    FAIL     -> PASS     NOW PASSES
  rand_s4    FAIL     -> PASS     NOW PASSES
  stall_s1   TIMEOUT  -> TIMEOUT  SAME AGAIN
  NOW PASSES 3   SAME AGAIN 1

=== 6. the whole nightly again, on the fixed design
n2: 9 runs  PASS 8  FAIL 0  NO VERDICT 1  -> runs_n2/results.tsv

Read it as a debug session:

• Step 1: three failures and a timeout.
• Step 2: the test-keyed retry list has lost both rand failures.
• Step 3: the run-keyed plan has all four. Rerun with the same seeds, every failure is REPRODUCED — same first error, character for character — and the timeout happens again. The failures are real; debugging can start with any of them, using its cmd.sh (Chapter 25.4).
• Step 4: new seeds make the random failures disappear on an unchanged design — the trap from Section 7.
• Step 5: after the one-line fix, the same plan gives NOW PASSES three times.
• Step 6: the whole regression passes on the fixed design, except the test that hangs by design.

10. Practical Rules

Rerun into a new area. Each rerun writes runs_r1, runs_r2, … and never touches runs_n1. The original evidence is what every comparison is against; overwriting it destroys the baseline. Chapter 25.4's rule — refuse to launch into an existing run directory — applies to reruns as well.

Retry NO VERDICT runs differently from FAIL runs. A failure is rerun to confirm and debug. A timeout is rerun to learn whether it is repeatable — and it may deserve a longer limit on the rerun, recorded in the rerun's plan, so that a slow-but-correct run is not mistaken for a hang twice.

Keep the rerun small. The point of the loop is speed. If the retry plan has hundreds of runs, the failures probably share a cause; group them by signature first (Chapter 21.9) and rerun one or two from each group.

Never mix the rerun's results into the original report. The nightly report says what happened last night. The rerun report says what happened this morning. A report that quietly replaces last night's FAIL with this morning's PASS makes last night's regression look clean, and hides the fact that the design was broken when it ran.

11. Exercises

1. One result per test, measured. Change the order of the rows in nightly.tsv so that the last rand row is a failing seed. Run naive_retry.pl again. What does the retry list contain now, and what is still missing?

2. A longer limit for reruns. Add an optional limit column to the plan format, have retry_plan.pl write double the original limit for runs that were TIMEOUT, and make regress.pl honour it. Does stall now get a verdict? Why not?

3. Prove compare.pl can fail. Edit a copy of runs_r1/results.tsv by hand three times: change one seed, change one signature, and add a run that is not in the original. Run compare.pl against each copy and check that it reports or refuses each one as described in Section 6.

4. Normalised signatures. Change regress.pl so that the signature replaces every number in the first error line with N (ERROR: a=N b=N y=N expected=N). Rerun steps 1 and 3. What do rand_s2 and rand_s4 have in common now that they did not before, and what is that useful for?

5. A failure that does not reproduce. Change bin/simv so that it adds +n=$NVEC to the simulator's arguments whenever the environment variable NVEC is set. Rerun one failing run with NVEC=5. compare.pl will report NOW PASSES or FAILS DIFFERENTLY on an unchanged design. Which record, kept at the time of the first run, would have explained the difference? (Chapter 25.9 answers this.)

12. What To Remember

• The debug loop: rerun the failures with their seeds, fix, rerun the same failures, then run everything.
• Measured: a retry list keyed by test lost both rand failures, because six rand rows shared one hash key and the last one had passed.
• Key by run id, and keep the original seed. One line per run in, one line per run out.
• Read the previous report by column name, and never invent a seed.
• Compare each rerun with its first result: REPRODUCED, FAILS DIFFERENTLY, NOW PASSES, SAME AGAIN, CHANGED. Refuse a changed seed.
• Measured: on an unchanged, broken design, a "retry" with new seeds passed both rand runs. New seeds are a new test, not a retry.
• A failure that survives a seed change is deterministic; one that disappears is seed-dependent. On a deterministic simulator, neither is "flaky".
• NOW PASSES after a fix proves those runs pass — run the whole regression before calling it fixed.
• Rerun into a new area, and never overwrite the original report.

Chapter 25.9 closes the module with the record that makes all of this possible weeks later: the commit, the filelist, the tool version, the seed and the exact command for every run — so that "it failed last Tuesday" becomes a command someone can paste.