Linux & Bash · Part II · Module 32 · Regression Automation End to End

Re-Running Only What Failed

Re-running fifteen failures out of a thousand runs is the difference between a four-minute debug cycle and a four-hour one. This chapter builds that from the previous regression's own recorded data, and measures the guard that stopped a reproduction which would have silently measured something else — after restoring the source files had failed to restore the thing that mattered.

Advanced25 min readLinuxPart IIRegressionReproducibilityDebugging

Part II · Module 32 · Chapter 32.6 · Re-Running Only What Failed

Chapter 32.5's report named fifteen runs that are not a pass out of twenty-four. On a real block the proportions are kinder and the arithmetic is worse: fifteen failures out of two thousand runs, and re-running the regression to look at them again costs the whole night.

So you re-run the fifteen. The question this chapter answers is what has to be true for that to be the same experiment.

1. A Reproduction Has Three Parts

A run's identity, established across Module 31 and this one:

part	recorded in	if it differs
the test and the seed	`results.tsv`'s `spec`	a different stimulus — not a reproduction
the compiled image	`env.txt`'s `image_md5`	a different design or testbench
the invocation — limits, plusargs	`env.txt` and `one.sh`	a different experiment

The first is obvious and is why Chapter 31.3 insisted on recording seeds rather than generating them from the clock. The third is Chapter 27.9's subject.

The second is the one that gets lost, and the rest of this chapter is largely about it.

2. The Re-Runner

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# rerun RUNDIR [CLASS...] -- re-run the runs of a previous regression that are
# not a PASS, using that regression's own recorded seeds and image.
# With no CLASS arguments it reruns everything that is not PASS.
set -u
root=$(cd "$(dirname "$0")/.." && pwd)
prev=${1:-}
[ -n "$prev" ] || { echo "rerun: need a previous regression directory" >&2; exit 2; }
res=$prev/results.tsv
[ -r "$res" ]      || { echo "rerun: no results.tsv in '$prev'" >&2; exit 2; }
[ -r "$prev/env.txt" ] || { echo "rerun: no env.txt in '$prev'" >&2; exit 2; }
shift
classes=$*

# The image is part of a run's identity. Re-running a failure against a
# different image does not reproduce it -- it measures something else.
was=$(sed -n 's/^image_md5=//p' "$prev/env.txt")
now=$(md5 -q "$root/logs/sim.vvp" 2>/dev/null || md5sum "$root/logs/sim.vvp" | cut -d' ' -f1)
if [ "$was" != "$now" ]; then
  echo "rerun: the compiled image has changed since that regression" >&2
  echo "       then: $was" >&2
  echo "       now:  $now" >&2
  [ "${ALLOW_NEW_IMAGE:-0}" = 1 ] || {
    echo "       refusing -- set ALLOW_NEW_IMAGE=1 to rerun anyway (this is a new experiment, not a reproduction)" >&2
    exit 3; }
  echo "       continuing because ALLOW_NEW_IMAGE=1" >&2
fi

sel() {
  if [ -z "$classes" ]; then awk -F'\t' 'NR>1 && $4!="PASS" {print $1}' "$res"
  else
    for c in $classes; do awk -F'\t' -v k="$c" 'NR>1 && $4==k {print $1}' "$res"; done
  fi
}
out=$prev/rerun-$(date -u +%H%M%SZ)
mkdir -p "$out/runs" || exit 2
sel | sort -u > "$out/specs.txt"
n=$(grep -c . "$out/specs.txt")
[ "$n" -gt 0 ] || { echo "rerun: nothing selected -- every run was a PASS" >&2; exit 0; }
echo "rerun: $n spec(s) from $(basename "$prev")${classes:+ (classes: $classes)}" >&2
limit=$(sed -n 's/^limit_s=//p' "$prev/env.txt"); limit=${limit:-5}
cp "$prev/env.txt" "$out/env.txt"
cat > "$out/one.sh" <<INNER
#!/bin/bash
set -u
spec=\$1; t=\${spec%%:*}; s=\${spec#*:}
d="$out/runs/\${t}_s\${s}"
ROOT="$root" "$root/bin/runsim" "\$t" "\$s" "\$d" "$limit" || exit 2
printf '%s\n' "\$spec" > "\$d/spec"
v=\$("$root/bin/verdict" "\$d"); c=\$?
printf '%s %s\n' "\$v" "\$c" > "\$d/verdict.tmp" && mv "\$d/verdict.tmp" "\$d/verdict"
INNER
chmod +x "$out/one.sh"
xargs -P "${SLOTS:-8}" -I{} "$out/one.sh" {} < "$out/specs.txt"
"$root/bin/gather" "$out/specs.txt" "$out/runs" > "$out/results.tsv"
echo
printf '  %-18s %-14s %-14s %s\n' SPEC 'was' 'now' ''
same=0; diff=0
while IFS= read -r sp; do
  w=$(awk -F'\t' -v s="$sp" '$1==s {print $4}' "$res" | head -1)
  nw=$(awk -F'\t' -v s="$sp" '$1==s {print $4}' "$out/results.tsv" | head -1)
  if [ "$w" = "$nw" ]; then same=$((same+1)); m="reproduced"
  else diff=$((diff+1)); m="DID NOT REPRODUCE"; fi
  printf '  %-18s %-14s %-14s %s\n' "$sp" "$w" "$nw" "$m"
done < "$out/specs.txt"
printf '\n  %d reproduced, %d did not\n' "$same" "$diff"
echo "  $out"
[ "$diff" -eq 0 ] || exit 1

It reads the previous regression, not the current configuration — which is the whole point. The seeds come from results.tsv, the limit from env.txt, and neither is taken from cfg/:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

limit=$(sed -n 's/^limit_s=//p' "$prev/env.txt"); limit=${limit:-5}

If somebody has since changed cfg/seeds.txt or the default limit, the rerun is unaffected. A reproduction that reads today's configuration is not a reproduction.

And the output goes inside the previous regression:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

out=$prev/rerun-$(date -u +%H%M%SZ)

So the reruns live with the thing they are reruns of, and ls regr/20261009T093929Z shows rerun-094210Z beside the original runs/. A rerun that writes to a fresh top-level directory loses the connection within a week.

3. The Image Guard, Which Fired For Real

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

was=$(sed -n 's/^image_md5=//p' "$prev/env.txt")
now=$(md5 -q "$root/logs/sim.vvp" 2>/dev/null || md5sum "$root/logs/sim.vvp" | cut -d' ' -f1)
if [ "$was" != "$now" ]; then
  ...
  [ "${ALLOW_NEW_IMAGE:-0}" = 1 ] || {
    echo "       refusing -- set ALLOW_NEW_IMAGE=1 to rerun anyway (this is a new experiment, not a reproduction)" >&2
    exit 3; }

Here is what happened when I first ran it. I had changed the arbiter for Chapter 32.5's comparison, so before re-running I restored the saved copies of both source files and recompiled:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ cp tb/tb_arb_v1.v tb/tb_arb.v
$ cp rtl/arb_v1.v rtl/arb.v
$ iverilog -o logs/sim.vvp tb/tb_arb.v rtl/arb.v
$ rerun regr/20261009T093929Z
rerun: the compiled image has changed since that regression
       then: 8671d52fb85a569afed2321d6a929acf
       now:  ea9ca3fe2025e41e137b831bda98c4a6
       refusing -- set ALLOW_NEW_IMAGE=1 to rerun anyway (this is a new experiment, not a reproduction)

I had restored the files and the image was still different, and the guard was right.

The reason is ordinary and that is what makes it important: between that regression and this moment, two tests had been added to the testbench — vacuous and latecrash, built in Chapter 32.3 because two verdict rules turned out to be dead code. tb_arb_v1.v was saved after those edits, so "v1" was never the file that regression ran. My mental model of which sources were in play was wrong by two edits, and nothing in the filesystem would have told me.

The path is not the identity; the digest is

Suppose env.txt had recorded only this:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

image=/…/m32/logs/sim.vvp

The rerun would have checked that the path exists, found it, run fourteen simulations against a different binary, and reported its results under the previous regression's name. Some would have "reproduced" and some would not, and the ones that did not would have looked like flakiness.

That is the measured consequence, and it is not hypothetical. The same fourteen specs, re-run with ALLOW_NEW_IMAGE=1 against a changed image:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  SPEC               was            now
  crash:1            CRASH          CRASH          reproduced
  crash:3            CRASH          CRASH          reproduced
  hang:1             TIMEOUT        TIMEOUT        reproduced
  hang:3             TIMEOUT        TIMEOUT        reproduced
  inconsistent:1     INCONSISTENT   PASS           DID NOT REPRODUCE
  inconsistent:3     INCONSISTENT   PASS           DID NOT REPRODUCE
  latecrash:1        INCONSISTENT   INCONSISTENT   reproduced
  latecrash:3        INCONSISTENT   INCONSISTENT   reproduced
  prio0:1            FAIL           PASS           DID NOT REPRODUCE
  prio0:3            FAIL           PASS           DID NOT REPRODUCE
  silent:1           NO_VERDICT     NO_VERDICT     reproduced
  silent:3           NO_VERDICT     NO_VERDICT     reproduced
  vacuous:1          VACUOUS        VACUOUS        reproduced
  vacuous:3          VACUOUS        VACUOUS        reproduced

  10 reproduced, 4 did not

Ten of fourteen reproduced, which is exactly the pattern that gets written off as an intermittent problem. The four that did not are the four the design change affected, and without the digest nothing in that output would have said so — the specs are right, the seeds are right, and the table reads like four flaky tests.

The digest costs one line in env.txt and one comparison in the rerun. It is the single highest-value thing a regression records about itself, because it is the one piece of a run's identity that a filesystem will silently replace.

And the override is deliberately phrased as what it is: this is a new experiment, not a reproduction. Re-running old failures against a new image is a perfectly reasonable thing to want — it is how you check whether a fix worked — and it should be spelled differently from reproducing them.

Against the matching image

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ md5 -q logs/sim.vvp
ea9ca3fe2025e41e137b831bda98c4a6
$ for d in regr/2026*; do
>   printf '  %-22s %s\n' "$(basename $d)" "$(sed -n 's/^image_md5=//p' $d/env.txt)"
> done
  20261009T093929Z       8671d52fb85a569afed2321d6a929acf
  20261009T094027Z       2c57f558b5a293551fb3485f4127796c
  20261009T094111Z       ea9ca3fe2025e41e137b831bda98c4a6   <-- MATCHES

Three regressions, three different images — and the digest is what identifies which one the current build corresponds to. Against that one:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

rerun: 14 spec(s) from 20261009T094111Z

  SPEC               was            now
  crash:1            CRASH          CRASH          reproduced
  crash:3            CRASH          CRASH          reproduced
  hang:1             TIMEOUT        TIMEOUT        reproduced
  hang:3             TIMEOUT        TIMEOUT        reproduced
  inconsistent:1     INCONSISTENT   INCONSISTENT   reproduced
  inconsistent:3     INCONSISTENT   INCONSISTENT   reproduced
  latecrash:1        INCONSISTENT   INCONSISTENT   reproduced
  latecrash:3        INCONSISTENT   INCONSISTENT   reproduced
  prio0:1            FAIL           FAIL           reproduced
  prio0:3            FAIL           FAIL           reproduced
  silent:1           NO_VERDICT     NO_VERDICT     reproduced
  silent:3           NO_VERDICT     NO_VERDICT     reproduced
  vacuous:1          VACUOUS        VACUOUS        reproduced
  vacuous:3          VACUOUS        VACUOUS        reproduced

  14 reproduced, 0 did not

Fourteen for fourteen, including the timeouts and the crashes. A reproduction rate is a measurement of your regression's determinism, and it is worth printing every time: a rerun that reports 12 of 14 against an identical image has found something more interesting than the original failures.

4. Selecting What to Re-Run

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

sel() {
  if [ -z "$classes" ]; then awk -F'\t' 'NR>1 && $4!="PASS" {print $1}' "$res"
  else
    for c in $classes; do awk -F'\t' -v k="$c" 'NR>1 && $4==k {print $1}' "$res"; done
  fi
}

Two modes, and the class filter is the one that gets used in practice, because Chapter 32.3's classes map onto different people's mornings:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ rerun regr/20261009T094111Z TIMEOUT CRASH      # is the machine healthy again?
$ rerun regr/20261009T094111Z FAIL               # did the RTL fix work?
$ rerun regr/20261009T094111Z INCONSISTENT VACUOUS   # are the testbenches fixed?

That is the payoff for having seven classes rather than two. "Re-run the failures" is a vague instruction when four of the seven classes are not failures; "re-run the timeouts" is a precise one, and it is the one worth running first because a machine problem invalidates everything else.

sort -u on the selection matters more than it looks:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

sel | sort -u > "$out/specs.txt"

Without -u, rerun … FAIL FAIL runs everything twice, and two runs writing to the same directory is Chapter 31.6's collision — which there produced one log and four verdicts read out of it, two of them wrong.

And the selection is written to a file before anything runs, so specs.txt beside the rerun's results answers "what was re-run" without inference — the same argument Chapter 31.4 makes for the matrix.

5. The Comparison Is the Output

A rerun whose output is just another results.tsv has missed the point. What somebody wants is the diff:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

while IFS= read -r sp; do
  w=$(awk -F'\t' -v s="$sp" '$1==s {print $4}' "$res" | head -1)
  nw=$(awk -F'\t' -v s="$sp" '$1==s {print $4}' "$out/results.tsv" | head -1)
  if [ "$w" = "$nw" ]; then same=$((same+1)); m="reproduced"
  else diff=$((diff+1)); m="DID NOT REPRODUCE"; fi
  printf '  %-18s %-14s %-14s %s\n' "$sp" "$w" "$nw" "$m"
done < "$out/specs.txt"

Three details.

It iterates specs.txt, so a spec that was selected and then produced nothing still gets a line. Iterating the new results would silently drop it — Chapter 32.4's rule, applied to a smaller matrix.

head -1 on both lookups because $1==s could match more than once if a spec were duplicated, and a comparison that silently used the second occurrence would be unexplainable.

And the exit status is the reproduction rate:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[ "$diff" -eq 0 ] || exit 1

Non-zero means "this is not the same experiment", which is a different statement from "these tests failed". The tests failing is already known — that is why they were selected.

What a rerun cannot tell you

Three limits worth stating plainly, because a rerun looks more authoritative than it is.

A reproduction does not confirm a diagnosis. prio0 reproduces perfectly on every seed and every image where the reset is unchanged. That tells you the failure is deterministic; it tells you nothing about why.

A non-reproduction does not mean flakiness. It means something differed, and the image is the first suspect — Section 3 measured four "flaky" tests that were a design change. Only after the digest matches is "intermittent" a reasonable word.

And a rerun cannot re-create a timeout honestly. hang:1 reproduced as TIMEOUT, which is correct, but a run that timed out at four seconds on a loaded machine and completes in three on an idle one is not reproduced — it is a different measurement of a machine, and the verdict changing from TIMEOUT to PASS would be recorded as a fix. The honest handling is to treat a TIMEOUT that disappears as a question rather than an answer, and the material fact is Chapter 32.2's: a shell time limit is approximate — LIMIT_S=4 produced five-second runs — so the boundary is fuzzy by design.

What a rerun is authoritative about is the thing it prints: how many of these specs, against this image, give the same verdict they gave before.

6. What It Must Refuse

The guards, and the failure each one prevents:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[ -n "$prev" ] || { echo "rerun: need a previous regression directory" >&2; exit 2; }
[ -r "$res" ]      || { echo "rerun: no results.tsv in '$prev'" >&2; exit 2; }
[ -r "$prev/env.txt" ] || { echo "rerun: no env.txt in '$prev'" >&2; exit 2; }

No env.txt means no image digest, and without the digest the rerun cannot say whether it is reproducing anything. Refusing is the only honest option — running anyway and omitting the check would produce a table that looks identical to a real reproduction.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[ "$n" -gt 0 ] || { echo "rerun: nothing selected -- every run was a PASS" >&2; exit 0; }

An empty selection exits 0, and this is the one place in the module where "nothing to do" is genuinely success. Every other empty set in this module is a refusal — Chapter 31.2's empty test list, Chapter 32.2's empty matrix, Chapter 32.5's zero-row results file — because in those cases the emptiness means something failed to be produced. Here it means the regression passed, which is the outcome you wanted, and the message says which of the two it is rather than leaving the reader to infer it from a silent exit.

The distinction is the one worth carrying away: an empty set is a refusal when it should have been populated, and a success when it genuinely represents "none". Deciding which requires knowing what populated it — the previous regression's results, here, which is why the message names the reason.

Practice

Name the three parts of a run's identity and where each is recorded.
Write a re-runner that reads the previous regression and nothing from today's configuration.
Change cfg/seeds.txt after a regression, then rerun it, and confirm the rerun is unaffected.
Put the rerun's output inside the previous regression's directory and explain why.
Record an image digest in your own flow's run metadata.
Change one source file, recompile, and compare the digest against a previous run's.
Restore the source file, recompile, and compare again — and find out whether it matches.
Rerun against a deliberately different image with the override on, and count the "non-reproductions".
Explain how that table would read to somebody who did not know the image had changed.
Rerun against the matching image and record the reproduction rate.
Re-run by class — timeouts, then failures, then the testbench classes — and say why that order.
Drop the sort -u and pass a class twice; find the collision.
Write the selection to a file before running, and say what question that answers later.
Iterate the selection rather than the new results in the comparison, and construct the case that proves it matters.
Make the exit status the reproduction rate and explain how that differs from "the tests failed".
Remove env.txt and confirm the rerun refuses rather than guessing.
Rerun a regression in which everything passed and check the status and the message.
List every empty-set case in your flow and classify each as a refusal or a success.
Find a timeout that does not reproduce and decide what to call it.

Check Your Understanding

Why re-run a subset rather than the regression?
What are the three parts of a run's identity?
Which one gets lost, and why?
Why must the rerun read the previous env.txt rather than today's config?
Why does the rerun write inside the previous regression's directory?
I restored both source files and recompiled. Why did the digest still differ?
What would have happened if env.txt recorded only the image's path?
Of fourteen specs re-run against a changed image, how many "reproduced", and what were the other four?
Why is that pattern dangerous?
How is the override phrased, and why does the phrasing matter?
What does a reproduction rate measure?
What is more interesting than the original failures?
Why re-run by class, and which class first?
What does sort -u prevent?
Why write the selection to a file before running?
Why does the comparison iterate the selection rather than the new results?
Why head -1 on the verdict lookups?
What does the rerun's non-zero status mean, and what does it not mean?
What does a successful reproduction not confirm?
What does a non-reproduction not prove?
Why can a rerun not honestly re-create a timeout?
Why does the rerun refuse without env.txt?
Why is an empty selection a success here when every other empty set in this module is a refusal?

Answers

Because re-running two thousand runs to look again at fifteen costs the whole night, while the fifteen cost minutes.
The test and seed, the compiled image, and the invocation — limits and plusargs.
The image. The others are text in files; the image is a path that gets silently overwritten by the next build.
Because a reproduction that reads today's configuration is not a reproduction — somebody may have changed the seeds or the limit since.
So the reruns live beside the thing they are reruns of; a fresh top-level directory loses that connection within a week.
Because two tests had been added to the testbench in between, and the saved "v1" copy was taken after those edits — so v1 was never the file that regression ran.
The rerun would have checked the path exists, run fourteen simulations against a different binary, and reported the results under the previous regression's name.
Ten reproduced. The other four — inconsistent:1, inconsistent:3, prio0:1, prio0:3 — were the four the design change affected.
Because ten of fourteen is exactly the pattern written off as intermittent, and the specs and seeds are all correct, so nothing in the table says the image differed.
"This is a new experiment, not a reproduction." Re-running old failures against a new image is a reasonable thing to want — checking a fix — and it should be spelled differently from reproducing them.
Your regression's determinism. A rerun reporting 12 of 14 against an identical image has found something more interesting than the original failures.
Exactly that — a non-reproduction against an identical image.
Because the classes map onto different people's mornings. Timeouts and crashes first, because a machine problem invalidates everything else.
Running the same spec twice, which means two runs writing to one directory — the collision that produced one log and four verdicts, two of them wrong.
So that "what was re-run" is answerable later without inference.
So that a spec which was selected and then produced nothing still gets a line, instead of being silently dropped.
Because a duplicated spec would match twice, and a comparison that silently used the second occurrence would be unexplainable.
That this is not the same experiment. It does not mean the tests failed — that was already known, and is why they were selected.
A diagnosis. prio0 reproduces perfectly and that says only that the failure is deterministic.
Flakiness. It means something differed, and the image is the first suspect.
Because a run that timed out at four seconds on a loaded machine and finishes in three on an idle one is a different measurement of a machine, not a reproduction — and the limit itself is approximate, with LIMIT_S=4 producing five-second runs.
Because without the digest it cannot say whether it is reproducing anything, and a table omitting that check looks identical to a real reproduction.
Because an empty set is a refusal when it should have been populated and a success when it genuinely represents "none" — and here it means the previous regression passed, which is the wanted outcome. The message names which of the two it is.

7. Remember

Reproducing one of them by hand, outside the regression, is Chapter 33.2 — which measures one omitted plusarg turning a genuine failure into a clean pass, and the failure then being called intermittent.
A run's identity is the spec, the image and the invocation. Only the first is obvious.
The rerun reads the previous regression's recorded data, never today's configuration.
Record the image's digest, not its path. The path survives a rebuild; the identity does not.
Restoring the source files did not restore the image — two unrelated edits had landed in between, and only the digest knew.
Ten of fourteen "reproduced" against a changed image. That is what flakiness looks like when it is actually a design change.
Phrase the override as what it is: re-running against a new image is a new experiment, not a reproduction.
Print the reproduction rate every time. A non-reproduction against an identical image is more interesting than the failures were.
Re-run by class, timeouts first, because a machine problem invalidates the rest.
sort -u the selection, or two runs share one directory.
Iterate the selection, not the new results, so a spec that produced nothing still appears.
Non-zero means "not the same experiment", not "the tests failed".
A reproduction confirms determinism, not a diagnosis; a non-reproduction is not yet flakiness; and a timeout cannot be reproduced honestly at all.
An empty set is a refusal when it should have been populated and a success when it means "none". Say which.

8. What Is Next

Everything so far has produced output for a person to read. Chapter 32.7 gives the regression a status and a set of artifacts an automated caller can gate on — and measures the single most common way that status is destroyed, which takes exactly one character to introduce.

Where this fits

Part of the Linux & Bash curriculum.