From One Run to Many

Part II · Module 31 · Chapter 31.1 · From One Run to Many

Module 30 ran jobs by hand and submitted one to a queue. This module turns one invocation into two thousand, and the single most useful thing to understand first is that the hard part is the single run. Everything that goes wrong at scale is already wrong at one; it is merely invisible.

1. The Unit of Work

A regression's atom is a script that runs exactly one simulation and reports exactly one result:

#!/bin/bash
# run_one.sh TEST[:SEED] — one simulation in its own directory.
set -u
[ $# -gt 0 ] || { echo "run_one.sh: need TEST or TEST:SEED" >&2; exit 2; }
spec=$1
test=${spec%%:*}; seed=${spec#*:}; [ "$seed" = "$spec" ] && seed=1
[ -n "${ROOT:-}" ] || { echo "run_one.sh: set ROOT" >&2; exit 2; }
root=$ROOT
[ -n "${RUNS:-}" ] || { echo "run_one.sh: set RUNS" >&2; exit 2; }
d="$RUNS/${test}_s${seed}"
mkdir -p "$d" || exit 2
( cd "$d" && exec vvp "$root/logs/sim.vvp" \
    +TESTNAME="$test" +SEED="$seed" +CYCLES="${CYCLES:-120000}" +STRICT ) > "$d/run.log" 2>&1
rc=$?
grep -q 'TEST PASSED' "$d/run.log" && exit 0
grep -q 'TEST FAILED' "$d/run.log" && exit 1
exit 3

Fifteen lines, and every one of them is a decision made in an earlier module.

It takes one argument, TEST or TEST:SEED. A single string is what a test list holds (Chapter 31.2) and what xargs substitutes (Chapter 31.5), so the unit's interface has to be one word.

${spec%%:*} and ${spec#*:} split it without cut, awk or a subshell — Chapter 8.3's parameter expansion. At two thousand invocations the saved process matters less than the saved failure mode: no pipeline means no $? to misread.

ROOT and RUNS are required, and the guards exit 2 rather than using ${VAR:?} — §4's callout is why. A missing one fails at the top of the first run rather than producing two thousand runs in the wrong place, which is Chapter 20.3's rule with one modification.

The simulation runs in its own directory with exec. Chapter 27.3 measured what sharing a directory does; the exec is Chapter 27.7's rule so a signal reaches the tool.

And the exit status comes from the log. That is the whole of Module 29 in three lines, and §3 is about the third one.

2. The Single Run, Timed

$ export ROOT="$PWD" RUNS="$PWD/runs"
$ for i in 1 2 3; do
>   rm -rf runs; mkdir -p runs
>   s=$(date +%s%N); bin/run_one.sh random:1 >/dev/null 2>&1; e=$(date +%s%N)
>   printf '  run %d: %d ms\n' "$i" "$(( (e-s)/1000000 ))"
> done
  run 1: 309 ms
  run 2: 326 ms
  run 3: 345 ms

About 320 ms. Worth writing down, because two numbers derived from it are the ones a regression is planned with: 2,000 runs is roughly 11 minutes of CPU, and the first-ever run of the same command took 967 ms — three times longer, entirely from a cold page cache.

3. Three States, Not Two

grep -q 'TEST PASSED' "$d/run.log" && exit 0
grep -q 'TEST FAILED' "$d/run.log" && exit 1
exit 3

$ bin/run_one.sh random:1; echo "rc=$?"
rc=0
$ bin/run_one.sh bogus:1; echo "rc=$?"
rc=1

The third state is the one that makes a regression trustworthy. exit 3 means the log contained no verdict at all — and Module 29 and Module 30 between them found five distinct ways to arrive there:

Five causes, one state, and all five exit 0 from the tool's point of view in at least one variant. A two-state classifier has to put every one of them in the pass column or the fail column, and either choice is a lie about two thousand runs.

Note also what the script does not do with rc — the tool's own status, captured on the line after the simulation:

rc=$?

It is captured and then not used for the verdict. That is deliberate: the status is worth recording (Chapter 27.9) and is not evidence of correctness. Chapter 31.7 puts both in the result row so a disagreement between them is visible rather than resolved.

4. The Loop

With the unit correct, the loop is three lines:

for t in smoke fairness random; do
  bin/run_one.sh "$t"
done

And it is already wrong in a way that only shows at scale: it stops telling you anything. Each run_one.sh exits with its own status, the loop discards all of them, and $? after the loop is the last run's. So:

#!/bin/bash
# runmany SPEC... — run each spec, collect every result.
set -u
root=${ROOT:?runmany: set ROOT}
runs=${RUNS:?runmany: set RUNS}
mkdir -p "$runs" || exit 2

pass=0 fail=0 noverdict=0 other=0
for spec in "$@"; do
  "$root/bin/run_one.sh" "$spec"
  case "$?" in
    (0) pass=$((pass+1));      v=PASS ;;
    (1) fail=$((fail+1));      v=FAIL ;;
    (3) noverdict=$((noverdict+1)); v=NO-VERDICT ;;
    (*) other=$((other+1));    v=ERROR ;;
  esac
  printf '%-16s %s\n' "$spec" "$v"
done

printf '\n%d run(s): %d passed, %d failed, %d no verdict, %d error\n' \
       "$#" "$pass" "$fail" "$noverdict" "$other"
[ $((fail + noverdict + other)) -eq 0 ]

$ ROOT=$PWD RUNS=$PWD/runs bin/runmany smoke fairness random bogus
smoke            PASS
fairness         PASS
random           PASS
bogus            FAIL

4 run(s): 3 passed, 1 failed, 0 no verdict, 0 error
$ echo $?
1

Four decisions, each of which is a thing that breaks later if omitted.

The status is read immediately after the call, into a case. Any command between the call and the $? — even an echo — replaces it.

ERROR is a fourth bucket. run_one.sh exits 2 for any harness failure — a missing variable, a missing argument, a mkdir that failed — so a status that is not 0, 1 or 3 means the harness broke rather than the test. Collapsing that into FAIL reports a design problem where there is an infrastructure problem.

The counters are printed as a line, not accumulated silently. Chapter 29.8 is about the report; this is the minimum version of it.

It exits non-zero if anything is not a pass, so the loop composes.

5. What the Unit Must Not Do

Three things belong outside run_one.sh, and putting them inside is the commonest way a regression becomes unmaintainable.

It must not decide what to run. The list is data (Chapter 31.2) and the matrix is generated (Chapter 31.4). A unit that contains a list of tests means changing the regression is a code change.

It must not aggregate. One run, one result. A unit that appends to a shared summary file is the defect Chapter 31.6 measures — 8 corrupt rows in 200 parallel appends.

It must not retry. A unit that silently re-runs a failure makes the failure rate unmeasurable, and a flaky test indistinguishable from a solid one. Retrying is a decision for whatever reads the results (Chapter 32.6), with the first result kept.

And one thing it must do that is easy to get wrong: it must be safe to run twice. mkdir -p and a truncating > redirection give that for free here, and the thing that does not is a tool writing into a directory that already has output in it — which is Chapter 30.5's stale-artifact measurement, where a failed build reported success because the previous run's file was still present.

6. Measuring the Unit Before Scaling It

Two numbers are worth having before any loop is written, and both are one command.

How long does one run take, warm? §2: about 320 ms here. Multiply by the matrix size from Chapter 31.4 to get the sequential cost, which is the number that decides whether parallelism is needed at all.

How much memory does one run use? And this is where the measurement goes wrong if taken naively:

$ bin/run_heavy.sh 99 >/dev/null 2>&1 & w=$!
$ ps -o comm= -p $w
/bin/bash
$ pgrep -P $w
7832
$ # sample both until the wrapper exits
$ printf '  peak RSS of the WRAPPER: %s KB\n  peak RSS of VVP itself:  %s KB\n' "$mw" "$mv"
  peak RSS of the WRAPPER: 1184 KB
  peak RSS of VVP itself:  36656 KB

1,184 KB against 36,656 KB — the wrapper is a thirty-first of the work. $! is the shell that run_one.sh runs in, not the simulator it launched, so sampling $! reports about 1 MB for a 35 MB job.

That is the same $!-is-the-wrapper mechanism as Chapter 27.7's orphaned simulator and Chapter 30.4's misdirected signal, appearing a third time in resource measurement. The fix is the same shape — find the child:

# the process doing the work, not the shell that started it
vp=$(pgrep -P "$w" | head -1)
ps -o pid,rss,pcpu -p "${vp:-$w}"

And the reason it matters here specifically: the memory figure is what sets the parallelism limit in Chapter 31.5. A 35× underestimate turns "sixteen jobs will fit" into a machine that swaps.

Practice

1. Write run_one.sh taking one TEST:SEED argument, and confirm it creates its own directory.
2. Split the argument with parameter expansion rather than cut, and say what failure mode that removes.
3. Omit ROOT and confirm it fails on the first run rather than the last.
4. Time one run three times, warm, and record the spread.
5. Time the very first run on a cold cache and compare.
6. Make the test fail and confirm the status is 1 rather than the tool's 0.
7. Produce a run with no verdict — kill it, or break its dump path — and confirm the status is 3.
8. List the five ways to reach a no-verdict state in your own flow.
9. Write the loop that discards every status and then check $? after it.
10. Write runmany with four buckets, and confirm ERROR is distinct from FAIL.
11. Put an echo between the call and the $? and watch the status change.
12. Sample $!'s RSS and its child's, and compute the ratio.
13. Multiply your warm single-run time by your matrix size, and decide whether you need parallelism at all.

Check Your Understanding

1. Why is the hard part of a regression the single run?
2. Why does the unit take one string argument rather than two?
3. What does ${spec%%:*} avoid, beyond a process?
4. Where does the exit status come from, and why not from the tool?
5. What does exit 3 mean, and how many causes does it have?
6. Why can a two-state classifier not be correct?
7. Why is rc=$? captured and then not used for the verdict?
8. What does the three-line for loop discard?
9. Why is ERROR a separate bucket from FAIL?
10. Why must the status be read immediately after the call?
11. Name the three things the unit must not do, and why each matters.
12. What must the unit be safe to do, and what breaks it?
13. What did sampling $!'s RSS report, and what was the truth?
14. Why does that specific error matter in this module?

7. Remember

• The hard part is the single run. Everything wrong at scale is already wrong at one.
• One string argument — TEST:SEED — because that is what a list holds and what xargs substitutes.
• Require ROOT and RUNS with :?, so a mistake costs the first run and not all of them.
• Own directory, exec, and the verdict from the log.
• Three states, not two. exit 3 has five measured causes, and they all exit 0 from the tool.
• Capture $? immediately and keep a fourth ERROR bucket for harness failures.
• Warm single run ≈ 320 ms; the first ever took 967 ms. The first run of anything measures the filesystem.
• If a performance result confirms your expectation, measure it again.
• The unit must not choose what runs, aggregate, or retry — and must be safe to run twice.
• Sampling $! measured 1 MB of a 35 MB job. Find the child before measuring resources.

8. What Is Next

The unit is correct and the loop collects every result. Chapter 31.2 moves the list of runs out of the script and into a file — where the obvious way to read it silently loses the last line, and a loop whose body reads standard input consumes the list after one iteration.