Linux & Bash · Part II · Module 32 · Regression Automation End to End

The Driver Script: Structure and Responsibilities

A regression driver that is one long script can only be debugged by running it again, which for a four-hour regression means a four-hour debug cycle. This chapter builds it as five stages whose seams are files on disk, so a broken report can be fixed and re-run in a second against results that already exist.

Advanced25 min readLinuxPart IIRegressionScript StructureAutomation

Part II · Module 32 · Chapter 32.2 · The Driver Script

Chapter 32.1 stated four obligations. This chapter is about the shape of the thing that meets them, and the shape follows from one practical fact: a regression takes a long time, and the parts of it that break most often are the fast parts at the end.

1. Why Not One Script

The report is the code most likely to be wrong — it has the most formatting, the most arithmetic, and the least exercise. The runs are the part that takes four hours.

In a single-pass driver those are the same program, so fixing a report means running the regression again. That is the whole argument, and it generalises: the expensive stages and the error-prone stages should be separable, because they are not the same stages.

So the driver is five stages, and the seam between each pair is a file:

Figure 1 — the five stages, and the file each one leaves behind. The arrows are files on disk, not function calls, which is what lets any stage be re-run on its own: a broken report is fixed and re-run in a second against results that already exist, with no simulation repeated.

Each stage reads only the previous stage's output. Nothing reaches backwards, and nothing is held in a variable that only exists while the driver is running.

2. The Driver

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# regress [RUNDIR] -- the whole regression, in five stages.
#   setup -> expand -> dispatch -> collect -> report
# Every stage writes a file the next stage reads, so any one of them can be
# re-run on its own against the previous stage's output.
set -u

## ---- stage 1: setup -----------------------------------------------------
root=$(cd "$(dirname "$0")/.." && pwd)        # absolute, before any cd
stamp=$(date -u +%Y%m%dT%H%M%SZ)
out=${1:-$root/regr/$stamp}
slots=${SLOTS:-$(sysctl -n hw.ncpu 2>/dev/null || getconf _NPROCESSORS_ONLN 2>/dev/null || echo 4)}
limit=${LIMIT_S:-5}
[ -x "$root/bin/runsim" ]  || { echo "regress: bin/runsim missing" >&2; exit 2; }
[ -x "$root/bin/verdict" ] || { echo "regress: bin/verdict missing" >&2; exit 2; }
[ -r "$root/logs/sim.vvp" ]|| { echo "regress: no compiled image at logs/sim.vvp" >&2; exit 2; }
mkdir -p "$out/runs" || exit 2
cp "$root/cfg/tests.txt" "$out/tests.used"
cp "$root/cfg/seeds.txt" "$out/seeds.used"
{ echo "stamp=$stamp"; echo "slots=$slots"; echo "limit_s=$limit"
  echo "image=$root/logs/sim.vvp"
  echo "image_md5=$(md5 -q "$root/logs/sim.vvp" 2>/dev/null || md5sum "$root/logs/sim.vvp" | cut -d' ' -f1)"
  echo "host=$(hostname)"; echo "user=$(id -un)"
  echo "iverilog=$(iverilog -V 2>&1 | head -1)"
} > "$out/env.txt"
echo "regress: $out" >&2

## ---- stage 2: expand ----------------------------------------------------
strip() { sed -e 's/[[:space:]]*#.*$//' -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' "$1" | grep -v '^$'; }
: > "$out/specs.txt"
for t in $(strip "$out/tests.used"); do
  for s in $(strip "$out/seeds.used"); do printf '%s:%s\n' "$t" "$s" >> "$out/specs.txt"; done
done
n=$(grep -c . "$out/specs.txt")
[ "$n" -gt 0 ] || { echo "regress: the matrix is empty" >&2; exit 2; }
echo "regress: $n runs, $slots slots, ${limit}s limit each" >&2

## ---- stage 3: dispatch --------------------------------------------------
cat > "$out/one.sh" <<INNER
#!/bin/bash
# one spec -> one run directory with run.log, tool.rc, timedout, verdict
set -u
spec=\$1; t=\${spec%%:*}; s=\${spec#*:}
d="$out/runs/\${t}_s\${s}"
printf '%s\n' "\$spec" > /dev/null
ROOT="$root" CYCLES="\${CYCLES:-200}" "$root/bin/runsim" "\$t" "\$s" "\$d" "$limit" || exit 2
printf '%s\n' "\$spec" > "\$d/spec"
v=\$("$root/bin/verdict" "\$d"); code=\$?
printf '%s %s\n' "\$v" "\$code" > "\$d/verdict.tmp" && mv "\$d/verdict.tmp" "\$d/verdict"
exit 0
INNER
chmod +x "$out/one.sh"
"$root/bin/runsim" smoke 1 "$out/warmup" 5 >/dev/null 2>&1    # absorb the cold cache
rm -rf "$out/warmup"
t0=$(date +%s)
xargs -P "$slots" -I{} "$out/one.sh" {} < "$out/specs.txt"
xrc=$?
t1=$(date +%s)
echo "regress: dispatch finished in $((t1-t0))s (xargs rc=$xrc)" >&2
case $xrc in 126|127) echo "regress: one.sh not runnable -- nothing ran" >&2; exit 2;; esac

## ---- stage 4: collect ---------------------------------------------------
"$root/bin/gather" "$out/specs.txt" "$out/runs" > "$out/results.tsv" || true
echo "regress: $(grep -vc '^#' "$out/results.tsv") result rows" >&2

## ---- stage 5: report ----------------------------------------------------
"$root/bin/report" "$out" | tee "$out/report.txt"
exit $?

One run of it:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ SLOTS=8 LIMIT_S=4 regress
regress: /…/m32/regr/20261009T093929Z
regress: 24 runs, 8 slots, 4s limit each
regress: dispatch finished in 6s (xargs rc=0)
regress: 24 result rows
==============================================================
 regression 20261009T093929Z
==============================================================
 24 runs: 9 PASS  3 FAIL  2 TIMEOUT  2 CRASH
          2 NO_VERDICT  4 INCONSISTENT  2 VACUOUS
          0 MISSING  0 NOSTATUS

And what it left behind:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ls regr/20261009T093929Z
env.txt  one.sh  report.txt  results.tsv  runs  seeds.used  specs.txt  tests.used

Eight entries, and each one is a stage's output. Everything the regression knows about itself is in that directory.

3. Stage 1 — Setup Decides What This Run Is

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

root=$(cd "$(dirname "$0")/.." && pwd)        # absolute, before any cd

That is the first line of real work and it exists because of a mechanism this track has now met six times: a relative path that stops meaning what it meant once something changes directory. Chapter 30.1 measured a launcher that killed every job it started for exactly that reason, and Chapter 31.6 runs each simulation from inside its own directory, which guarantees the cd happens. Resolve the root once, at the top, and never pass a relative path downstream.

The rest of setup answers "what is this run" and writes the answer down:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ cat regr/20261009T093929Z/env.txt
stamp=20261009T093929Z
slots=8
limit_s=4
image=/…/m32/logs/sim.vvp
image_md5=8671d52fb85a569afed2321d6a929acf
host=MacBook-Air.local
user=azharuddinsaifi
iverilog=Icarus Verilog version 13.0 (stable) (v13_0)

The image_md5 line is the one that earns its keep, and Chapter 32.6 measures why: re-running a failure against a different compiled image does not reproduce it, and the only reliable statement about which image a run used is a digest of the image. The path is not enough, because the path gets overwritten.

The test and seed lists are copied, not referenced:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

cp "$root/cfg/tests.txt" "$out/tests.used"
cp "$root/cfg/seeds.txt" "$out/seeds.used"

cfg/tests.txt is a file somebody edits. By the time anybody reads the report, it may say something different — and a report reconciled against a different list than the one that ran is worse than no report. The same argument Chapter 27.9 makes for the command line and Chapter 31.2 makes for the test list.

Setup also refuses to continue when the pieces are missing:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[ -x "$root/bin/runsim" ]  || { echo "regress: bin/runsim missing" >&2; exit 2; }
[ -x "$root/bin/verdict" ] || { echo "regress: bin/verdict missing" >&2; exit 2; }
[ -r "$root/logs/sim.vvp" ]|| { echo "regress: no compiled image at logs/sim.vvp" >&2; exit 2; }

Three checks that cost nothing and convert "two thousand runs that all failed identically" into one line of output before anything starts.

4. Stage 2 — Expand Makes the Set Explicit

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

strip() { sed -e 's/[[:space:]]*#.*$//' -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' "$1" | grep -v '^$'; }
: > "$out/specs.txt"
for t in $(strip "$out/tests.used"); do
  for s in $(strip "$out/seeds.used"); do printf '%s:%s
' "$t" "$s" >> "$out/specs.txt"; done
done
n=$(grep -c . "$out/specs.txt")
[ "$n" -gt 0 ] || { echo "regress: the matrix is empty" >&2; exit 2; }

This is Chapter 31.4's generator, reduced to the two axes this fixture has, with two details carried over.

It expands the copies, tests.used and seeds.used, not the originals. So specs.txt is consistent with the lists recorded beside it, by construction rather than by care.

An empty matrix is a refusal. Zero runs that report success is the false green Chapter 31.2 measured, and it is one line to prevent.

And the result is a file, which is the thing Chapter 31.4 argued for: inspectable before launch, filterable into a valid subset, and still on disk at reporting time.

5. Stage 3 — Dispatch Does Nothing But Run Things

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

"$root/bin/runsim" smoke 1 "$out/warmup" 5 >/dev/null 2>&1    # absorb the cold cache
rm -rf "$out/warmup"
t0=$(date +%s)
xargs -P "$slots" -I{} "$out/one.sh" {} < "$out/specs.txt"
xrc=$?
t1=$(date +%s)
echo "regress: dispatch finished in $((t1-t0))s (xargs rc=$xrc)" >&2
case $xrc in 126|127) echo "regress: one.sh not runnable -- nothing ran" >&2; exit 2;; esac

Four things here, all of them Module 31's conclusions applied.

A warm-up run, discarded. Chapter 31.1 measured the first run of anything at roughly three times the warm cost, and Chapter 31.5 published a timing table that was backwards because of it. One throwaway run moves that cost outside the measured set.

xargs -P with a slot count from sysctl -n hw.ncpu, which Chapter 31.5 measured as the right default for real simulation work.

xargs's status is branched on for 126 and 127 only. Those two mean nothing ran. Every other value — including the 1 that covers both "one run failed" and "the regression was abandoned after three" — is logged and otherwise ignored, because the verdict comes from the status files. That is Chapter 31.7's conclusion, and dispatch is where it gets obeyed.

And the unit of work is a generated script, one.sh, written into the regression's own directory:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

cat > "$out/one.sh" <<INNER
#!/bin/bash
set -u
spec=\$1; t=\${spec%%:*}; s=\${spec#*:}
d="$out/runs/\${t}_s\${s}"
ROOT="$root" CYCLES="\${CYCLES:-200}" "$root/bin/runsim" "\$t" "\$s" "\$d" "$limit" || exit 2
printf '%s\n' "\$spec" > "\$d/spec"
v=\$("$root/bin/verdict" "\$d"); code=\$?
printf '%s %s\n' "\$v" "\$code" > "\$d/verdict.tmp" && mv "\$d/verdict.tmp" "\$d/verdict"
exit 0
INNER

Why generate the worker instead of exporting variables

xargs runs a command, not a shell function, and the command needs to know four things the driver computed: the output directory, the repository root, the time limit, and where runsim and verdict live.

Three ways to get them there, and only one of them leaves evidence:

Exported environment variables. Works, and the values vanish when the shell exits. A run directory found three weeks later cannot say what limit it was given.

Arguments through xargs -I{}. Works for one argument; adding four makes the command line long and the quoting fragile, and Chapter 31.6 measured xargs refusing a command line that got too long — while a probe of mine reported "intact" on the zero runs that produced.

A generated script, written into the regression's directory. The values are baked in at setup time, as literal text, in a file that survives next to the results. cat regr/…/one.sh is a complete and honest answer to "what exactly was run", which is Chapter 27.9's whole point.

The cost is that the heredoc needs its expansions managed: $out, $root and $limit must expand now, while $1, $spec and $d must survive into the generated file as literal text. That is what every \$ in the heredoc is doing, and getting one wrong produces a worker script that silently uses an empty path. The defence is to read the generated file rather than the generator:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ head -4 regr/20261009T093929Z/one.sh
#!/bin/bash
set -u
spec=$1; t=${spec%%:*}; s=${spec#*:}
d="/…/regr/20261009T093929Z/runs/${t}_s${s}"

$out expanded; $1 and ${t} did not. That is the check, and it takes one command.

6. Stages 4 and 5 — Collect and Report Read Files, Not Memory

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

"$root/bin/gather" "$out/specs.txt" "$out/runs" > "$out/results.tsv" || true
echo "regress: $(grep -vc '^#' "$out/results.tsv") result rows" >&2
"$root/bin/report" "$out" | tee "$out/report.txt"
exit $?

gather takes the matrix and the run directory and produces one row per spec — Chapter 32.4. report takes nothing but the regression directory and reads only results.tsv — Chapter 32.5.

Which means both can be run again, by hand, in under a second:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ report regr/20261009T093929Z          # re-run the report, change nothing else
$ gather regr/20261009T093929Z/specs.txt regr/20261009T093929Z/runs > /tmp/again.tsv

That is the payoff for the whole structure. A formatting bug in the report costs one second to fix and verify, not six seconds here and four hours on a real block.

`| tee` on the last line destroys the exit status — and this driver has the bug

Look carefully at the final two lines:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

"$root/bin/report" "$out" | tee "$out/report.txt"
exit $?

$? after a pipeline is the status of the last command in it, which is tee. tee almost always succeeds. So this driver exits 0 whatever the report decided, and the four-value contract from Chapter 32.1 is destroyed on the last line of the script.

Measured, with the report's own status known to be 1:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  INVOCATION                                     rc
  rc=$? directly                                 1
  rc=$? after | tail                             0
  rc=$? after | tee (the CI idiom)               0
  PIPESTATUS[0] after | tee                      1
  with set -o pipefail                           1

Two fixes, both one line:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# either read the right element of PIPESTATUS, immediately
"$root/bin/report" "$out" | tee "$out/report.txt"
exit "${PIPESTATUS[0]}"

# or turn the pipeline's failure into the pipeline's status
set -o pipefail
"$root/bin/report" "$out" | tee "$out/report.txt"
exit $?

PIPESTATUS must be read immediately — the very next command overwrites it, including a : — and under zsh the array is pipestatus and is indexed from 1. Chapter 12.8 establishes the rule; this is what it costs to forget it in the one place that matters, and Chapter 32.7 treats the whole contract.

The driver is published here with the defect in place, because the defect is the chapter's most useful line of code: it is one character from correct, it is in every regression script written in a hurry, and nothing except reading the status will ever tell you it is there.

7. What Each Stage Must Not Do

The structure only pays if the stages stay in their lanes. Four rules, each with a failure behind it:

Setup must not run anything. If setup runs a simulation to "check the tool works", a tool problem becomes a setup problem reported in the wrong place, and the warm-up run above is deliberately outside setup for that reason.

Expand must not decide anything. It produces the set; it does not skip entries it thinks will fail. A skipped run must appear in results.tsv as a run that did not happen, which is Chapter 31.7's MISSING.

Dispatch must not interpret. It runs things and records what happened. The moment dispatch decides a verdict, that decision is inside a parallel job whose output is interleaved with seven others — and it cannot be re-run without re-simulating.

Report must not read the run directories. Only results.tsv. The moment the report walks runs/*/, it can no longer be run against an archive, it can no longer be compared against another night's results, and it inherits Chapter 31.7's measured false green — a disk scan that reported 7 of 7 PASS, exit 0 about a matrix of eight.

The time limit is approximate, and by how much

runsim is given a limit in seconds and implements it with a polling loop:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

n=0; lim=$(( limit * 10 ))
while kill -0 "$p" 2>/dev/null && [ "$n" -lt "$lim" ]; do n=$((n+1)); sleep 0.1; done

With LIMIT_S=4, the hang runs recorded 5 seconds, not 4:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  hang:1   TIMEOUT   tool_rc=0   timedout=1   seconds=5

Two reasons, both unavoidable in a shell. sleep 0.1 is at least 0.1 s and the loop adds two process spawns per iteration, so forty iterations take more than four seconds. And after the limit fires, the kill is given a grace period — TERM first, then KILL after two seconds — because a simulator that is asked politely may still flush its log.

So a limit is a bound on the order of magnitude, not a deadline. If you need a real deadline, timeout(1) is the tool — and it is not installed here, which is why Chapter 29.5 built one by hand and measured its edges. Size the limit so the imprecision does not matter: a limit of 4 behaving as 5 is fine, a limit of 7200 behaving as 7205 is fine, and a limit chosen to be just above the expected runtime is a source of false timeouts.

Practice

Take a single-pass regression script you have and identify which parts are slow and which are error-prone.
Name the five stages and the file each one should leave behind.
Resolve the root to an absolute path at the top, and then break it by adding a cd.
Write env.txt with the stamp, slots, limit, image digest, host, user and tool version.
Delete the image digest and say what you can no longer answer.
Copy the test and seed lists into the run directory, then edit the originals and re-read the report.
Add the three setup refusals and trigger each one.
Make the matrix empty and confirm the driver refuses.
Add a discarded warm-up run and compare the first real run's time.
Branch on xargs' 126 and 127 and leave 1 alone; justify that in one sentence.
Generate the worker script and then read the generated file to confirm which expansions happened.
Get one \$ wrong on purpose and find out what the worker does.
Re-run the report alone against a finished regression.
Re-run gather alone and diff its output against the stored results.tsv.
Read the driver's exit status after its | tee, then with PIPESTATUS[0], then under set -o pipefail.
Insert a : between the pipeline and the PIPESTATUS read and watch what happens.
Make the report walk runs/*/ instead of results.tsv, delete one run directory, and see what it says.
Set a limit just above your expected runtime and count the false timeouts.
Measure how long a 4-second limit actually takes and explain the difference.

Check Your Understanding

Why not write the driver as one script?
Which stage is most likely to be wrong, and which takes the longest?
What is the seam between stages, and why not a function call?
Why resolve the root to an absolute path in the first line of work?
What does env.txt record, and which line matters most?
Why is the image's digest better than its path?
Why copy the test and seed lists rather than reference them?
What do the three setup refusals buy?
Why does expand read the copies rather than the originals?
Why is an empty matrix an error?
Why is there a discarded warm-up run, and why is it outside setup?
Which xargs statuses are branched on, and why only those?
Give the three ways to pass setup's values to the worker, and the advantage of generating a script.
What is every \$ in the heredoc doing, and how do you check you got them right?
What can report be run against, because it reads only results.tsv?
What is wrong with the driver's last two lines?
Give two one-line fixes, and the two traps in using PIPESTATUS.
Why must expand not skip entries it expects to fail?
Why must dispatch not decide verdicts?
Why must report not read the run directories?
A 4-second limit produced a 5-second run. Why, in two parts?
When does a limit's imprecision matter?

Answers

Because the report is the most error-prone stage and the runs are the slowest, so a single script makes a report fix cost a full regression.
The report is most likely to be wrong — most formatting, most arithmetic, least exercise. Dispatch takes the longest.
A file on disk. A function call's values vanish when the process exits, so the stage cannot be re-run later against what it actually received.
Because every later stage changes directory — each simulation runs from inside its own run directory — and a relative path stops meaning what it meant. The same mechanism killed every job in Chapter 30.1.
The stamp, slot count, limit, image path and digest, host, user and tool version. The digest matters most.
Because the path gets overwritten. Only a digest says which bytes a run actually used.
Because the originals are files somebody edits, and a report reconciled against a different list than the one that ran is worse than no report.
They convert "two thousand runs that failed identically" into one line of output before anything starts.
So that specs.txt is consistent with the recorded lists by construction rather than by care.
Because zero runs reporting success is a false green — nothing failed because nothing ran.
To move the cold-cache cost outside the measured set; outside setup so that a tool problem is not reported as a setup problem.
126 and 127, because they mean nothing ran. Everything else, including 1, cannot distinguish one failure from an abandoned regression, so the verdict comes from the status files.
Exported variables (vanish with the shell), xargs arguments (fragile quoting, and a command line that can get too long), and a generated script — which bakes the values in as literal text in a file that survives beside the results.
Deferring expansion: $out, $root and $limit expand now; $1, $spec and $d must reach the generated file as text. Check by reading the generated file, not the generator.
Any finished regression, including an archived one, and against another night's results for comparison.
report … | tee … followed by exit $? exits with tee's status, which is almost always 0 — so the driver reports success whatever the report decided.
exit "${PIPESTATUS[0]}", or set -o pipefail before the pipeline. The traps: PIPESTATUS must be read by the very next command, and under zsh the array is pipestatus, indexed from 1.
Because a skipped run must still appear in the results as a run that did not happen — otherwise the denominator is unknown.
Because the decision would be inside a parallel job with interleaved output, and it could not be re-run without re-simulating.
Because it could then no longer run against an archive or compare two nights, and it inherits the disk-scan false green that reported 7 of 7 passing about a matrix of eight.
sleep 0.1 is at least 0.1 s and the loop spawns processes, so forty iterations exceed four seconds; and the kill is given a two-second grace period after TERM so the simulator can flush its log.
When the limit is chosen just above the expected runtime — then the imprecision produces false timeouts. A limit an order of magnitude above it is unaffected.

8. Remember

Split the driver because the expensive stage and the error-prone stage are not the same stage. The report is the likeliest thing to be wrong and the cheapest to re-run.
Five stages — setup, expand, dispatch, collect, report — and every seam is a file.
Resolve the root absolutely in the first line of work. Every later stage changes directory.
Record the image's digest, not its path. The path gets overwritten; only the digest identifies the bytes that ran.
Copy the test and seed lists into the run directory. The originals will have changed by the time anybody reads the report.
An empty matrix is a refusal, never a silent success.
One warm-up run, discarded, outside setup.
Branch on xargs' 126 and 127 and nothing else. The verdict lives in the status files.
Generate the worker script into the regression's directory, so what ran is readable three weeks later — then read the generated file to confirm the expansions.
report | tee followed by exit $? returns tee's status. Use ${PIPESTATUS[0]} immediately, or set -o pipefail.
Stages stay in their lanes: expand does not skip, dispatch does not interpret, report reads only results.tsv.
A shell time limit is approximate — 4 seconds behaved as 5. Size it so that does not matter.

9. What Is Next

The driver runs things and records what happened, and deliberately decides nothing. Chapter 32.3 is where the deciding happens — and measures a verdict policy built on the exit status alone getting nine of thirteen runs wrong, a policy built on the log alone getting five wrong, and the two together getting none.

Where this fits

Part of the Linux & Bash curriculum.