Building the Run Matrix

Part II · Module 31 · Chapter 31.4 · Building the Run Matrix

Chapter 31.2 made tests a list and Chapter 31.3 made seeds a list. A real regression has at least one more axis — a configuration, a parameter set, a target — and the thing that runs is the product.

1. Nested Loops Hide the Number

The obvious implementation:

for c in fast slow; do
  for t in smoke fairness random burst error_inject back2back; do
    for s in $(seq 1 20); do
      run_one.sh "$t:$s:$c"
    done
  done
done

It is correct and it has one serious property: nobody knows how many runs that is until it is running. Six times twenty times two is not hard arithmetic, and the question is never asked, because the loop does not ask it.

Generating the product as a list first asks it for you:

$ matrix tests6.txt seeds20.txt configs.txt > specs.txt
matrix: 6 tests x 20 seeds x 2 configs = 240 runs
matrix: ~320ms each -> 0h 1m 16s sequential
$ grep -c . specs.txt
240

Three numbers before anything starts: the product, its factors, and a cost. On this fixture 240 runs is 76 seconds and the decision is trivial. The same command on a real block — where Chapter 31.1's per-run time is minutes rather than milliseconds — turns a routine for loop into a visible choice about a day of compute.

2. The Generator

#!/bin/bash
# matrix TESTS SEEDS [CONFIGS] — expand the axes into one run spec per line.
# Prints the SIZE and the estimated cost to stderr, the specs to stdout.
set -u
tl=${1:-}; sl=${2:-}; cl=${3:-}
[ -n "$tl" ] && [ -n "$sl" ] || { echo "usage: matrix TESTS SEEDS [CONFIGS]" >&2; exit 2; }
for f in "$tl" "$sl" ${cl:+"$cl"}; do
  [ -r "$f" ] || { echo "matrix: cannot read $f" >&2; exit 2; }
done

strip() {
  sed -e 's/[[:space:]]*#.*$//' -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' "$1" | grep -v '^$'
}

nt=$(strip "$tl" | grep -c .)
ns=$(strip "$sl" | grep -c .)
nc=1; [ -n "$cl" ] && nc=$(strip "$cl" | grep -c .)
[ "$nt" -gt 0 ] || { echo "matrix: $tl has no tests" >&2; exit 2; }
[ "$ns" -gt 0 ] || { echo "matrix: $sl has no seeds" >&2; exit 2; }
[ "$nc" -gt 0 ] || { echo "matrix: $cl has no configs" >&2; exit 2; }

total=$(( nt * ns * nc ))
per=${PER_RUN_MS:-320}
secs=$(( total * per / 1000 ))
printf 'matrix: %d tests x %d seeds x %d configs = %d runs\n' "$nt" "$ns" "$nc" "$total" >&2
printf 'matrix: ~%dms each -> %dh %dm %ds sequential\n' \
       "$per" "$((secs/3600))" "$(((secs%3600)/60))" "$((secs%60))" >&2
[ -n "${SLOTS:-}" ] && printf 'matrix: at %s slots -> about %dm %ds wall clock\n' \
  "$SLOTS" "$(( (secs/SLOTS)/60 ))" "$(( (secs/SLOTS)%60 ))" >&2

# the expansion. Order matters: seeds innermost so one test's seeds are adjacent.
if [ -n "$cl" ]; then
  for c in $(strip "$cl"); do
    for t in $(strip "$tl"); do
      for s in $(strip "$sl"); do printf '%s:%s:%s\n' "$t" "$s" "$c"; done
    done
  done
else
  for t in $(strip "$tl"); do
    for s in $(strip "$sl"); do printf '%s:%s\n' "$t" "$s"; done
  done
fi

$ matrix tests.txt seeds4.txt > /dev/null
matrix: 3 tests x 4 seeds x 1 configs = 12 runs
matrix: ~320ms each -> 0h 0m 3s sequential
$ matrix tests.txt seeds4.txt | tr '\n' ' '
smoke:1 smoke:2 smoke:3 smoke:4 fairness:1 fairness:2 fairness:3 fairness:4 random:1 random:2 random:3 random:4

With a third axis and a slot count:

$ SLOTS=8 matrix tests.txt seeds4.txt configs.txt > specs.txt
matrix: 3 tests x 4 seeds x 2 configs = 24 runs
matrix: ~320ms each -> 0h 0m 7s sequential
matrix: at 8 slots -> about 0m 0s wall clock
$ head -5 specs.txt
smoke:1:fast
smoke:2:fast
smoke:3:fast
smoke:4:fast
fairness:1:fast

And a size worth pausing over:

$ SLOTS=8 matrix tests6.txt seeds20.txt configs.txt > specs.txt
matrix: 6 tests x 20 seeds x 2 configs = 240 runs
matrix: ~320ms each -> 0h 1m 16s sequential
matrix: at 8 slots -> about 0m 9s wall clock

Six decisions in that script.

The summary goes to standard error and the specs to standard output. That is what makes matrix … > specs.txt print the size to your terminal while writing the list to the file — the same separation Chapter 12.3 established, used here so one command both informs and produces.

It reuses Chapter 31.2's strip on every axis, so comments and blank lines work in a seed file and a config file too. An axis file is a list, and all lists get the same treatment.

Every axis is checked for being non-empty. An empty config file would make nc=0 and the product zero, and a matrix of zero runs that reports success is Chapter 31.2's false green again — so each axis fails explicitly.

nc defaults to 1 when there is no config axis, which keeps the arithmetic uniform instead of branching on it.

The cost estimate is PER_RUN_MS, defaulted and overridable. 320 ms comes from Chapter 31.1's warm measurement, and it is wrong for anybody else's design — so it is a variable with a documented origin rather than a constant.

SLOTS is optional. The sequential figure is always meaningful; the wall-clock one requires knowing the parallelism, which is Chapter 31.5's subject.

3. The Ordering Is a Decision

The expansion puts seeds innermost, tests next, configs outermost:

smoke:1:fast     smoke:2:fast  …  smoke:4:fast
fairness:1:fast  …
random:4:fast
smoke:1:slow     …

Three consequences, and this is the part that is easy to leave to chance.

One test's seeds are adjacent. So a failure pattern that is seed-dependent — one of twenty seeds failing — appears as a single line in a sorted result listing rather than scattered through it.

One config is contiguous. If slow is broken, the failures are a block, which is visible in a report at a glance. Interleaving the configs would spread the same information across the whole list.

And the first runs to execute are a complete config. With a sequential run, or a parallel run that feeds in order, the fast sweep finishes before slow starts — so if the regression is cut short you have one complete config rather than two half-done ones.

That last point is the argument for configs outermost, and it reverses if the goal is different. A regression you intend to interrupt wants complete tests first:

# tests outermost: you get complete tests, not complete configs
for t in $(strip "$tl"); do
  for c in $(strip "$cl"); do
    for s in $(strip "$sl"); do printf '%s:%s:%s\n' "$t" "$s" "$c"; done
  done
done

4. The Matrix as an Artifact

specs.txt is a file, and that is the point. Four things follow that nested loops cannot do.

It can be inspected before running. grep -c . is the size; head is a sanity check that the axes crossed the way you intended.

It can be filtered. A subset is a grep, and the subset is itself a valid matrix:

# just the slow config
grep ':slow$' specs.txt > specs_slow.txt
# just one test, all seeds and configs
grep '^burst:' specs.txt > specs_burst.txt
# the three that failed last night, from a results file
grep -F -f failed_specs.txt specs.txt > specs_rerun.txt

It can be recorded with the results. Chapter 31.2's testlist.used for the whole matrix — so a report can answer "what was supposed to run", which is a different question from "what ran" and the difference is the interesting part.

And it can be diffed against the previous night's. A regression whose pass rate changed is a different investigation depending on whether the matrix changed too:

$ diff <(sort last_night/specs.txt) <(sort tonight/specs.txt) | head -4
> error_inject:17:slow
> error_inject:18:slow
> error_inject:19:slow
> error_inject:20:slow

Four runs added. A pass rate that moved with four new runs in the matrix needs no further explanation; one that moved with an identical matrix is a design change.

5. Driving the Matrix

With the matrix as a file, the driver from Chapter 31.2 needs no changes at all — a spec is a spec:

$ ROOT=$PWD RUNS=$PWD/runs runlist specs.txt
runlist: 24 tests from specs.txt
smoke:1:fast     PASS
smoke:2:fast     PASS
…

Which is the payoff for Chapter 31.1's insistence on a single-token interface. run_one.sh splits TEST:SEED; extending it to TEST:SEED:CONFIG is two lines:

# TEST:SEED:CONFIG, with sensible defaults when the later fields are absent
spec=$1
test=${spec%%:*}
rest=${spec#*:}; [ "$rest" = "$spec" ] && rest=""
seed=${rest%%:*}; [ -z "$seed" ] && seed=1
cfg=${rest#*:};   [ "$cfg" = "$rest" ] && cfg=default

And the directory name grows to match, which keeps Chapter 31.6's isolation intact:

d="$RUNS/${test}_s${seed}_${cfg}"

6. Seeing the Cost Before Paying It

The estimate in §2 is the whole reason to generate rather than loop, and it is worth being precise about what it is and is not.

What it is: the product times a measured per-run time. On this fixture, 240 × 320 ms = 76 s sequential, and 9 s across 8 slots.

What it is not: a prediction. Three things it omits, all measured elsewhere in this track:

So the estimate is an order of magnitude, and an order of magnitude is what the decision needs. The question it answers is not "will this take 76 seconds" but "is this ten seconds, ten minutes, or ten hours" — and those three have different answers about whether to run it now, run a subset, or think again about the matrix.

The one number worth computing exactly is the subset:

# what would the slow config alone cost?
$ grep -c ':slow$' specs.txt
120
$ PER_RUN_MS=320 SLOTS=8 matrix tests6.txt seeds20.txt /dev/null 2>&1 >/dev/null | head -2

Being able to price a subset is what turns "the regression is too long" from a complaint into a choice between axes — fewer seeds, fewer configs, or fewer tests, each with a visible cost.

Practice

1. Write the three nested loops and then count, by hand, how many runs they produce.
2. Generate the same product as a list and compare the size against your count.
3. Write matrix with the summary on stderr and the specs on stdout, and confirm matrix … > specs.txt prints to your terminal.
4. Give it an empty config file and confirm it refuses rather than producing zero runs.
5. Set PER_RUN_MS from your own measured per-run time and re-read the estimate.
6. Add SLOTS and compare the two figures.
7. Inspect the ordering and say which axis is contiguous.
8. Reverse it to put tests outermost and say what that buys.
9. Interrupt a sequential run halfway and check whether you have a complete config or two half ones.
10. Add a seeded shuffle, then explain the two costs of shuffling.
11. Warm up with one throwaway run and compare the first real run's time.
12. Filter the matrix three ways — one config, one test, a list of specs — and confirm each subset is still a valid matrix.
13. Diff tonight's matrix against last night's and interpret a pass-rate change in the light of it.
14. Put a colon in a config name and watch the spec reparse.
15. Add the colon validation and confirm it refuses.
16. Price three different subsets and decide which axis to cut.

Check Your Understanding

1. What property do nested loops have that a generated list does not?
2. What three numbers does the generator print before anything runs?
3. Why does the summary go to standard error?
4. Why is an empty axis an explicit failure rather than a zero-run matrix?
5. Why is nc defaulted to 1 instead of branching?
6. Why is the per-run time a variable rather than a constant?
7. Which axis is innermost, and what does that buy?
8. Which is outermost, and what does that buy?
9. When would you reverse the ordering?
10. Give the two objections to shuffling by default.
11. What is the better fix for cold-start bias, and what does it cost?
12. Why awk with a recorded seed rather than sort -R?
13. Name four things a matrix file can do that nested loops cannot.
14. Why is diffing matrices useful when a pass rate moves?
15. What happens if a config name contains a colon?
16. State the general rule about joined fields.
17. What three things does the cost estimate omit, and what question does it actually answer?

7. Remember

• Nested loops never say how big the job is. Generate the product as a list and the size is printed before launch.
• Summary to stderr, specs to stdout, so one command informs and produces.
• Every axis is a list and gets the same comment-and-blank stripping; an empty axis is a refusal, not a zero-run matrix.
• The per-run time is a variable with a measured origin, not a constant.
• Ordering is a decision: seeds innermost makes a seed-dependent failure contiguous; configs outermost means an interrupted run leaves one complete config.
• Do not shuffle by default — it needs a third recorded seed and destroys adjacency. Warm up once instead.
• sort -R cannot be seeded. Use awk with a recorded one.
• The matrix is an artifact: inspectable, filterable into valid subsets, recordable with the results, and diffable against last night's.
• A colon in a config name silently reparses the spec — the fourth instance in this track of a joined format failing quietly.
• The estimate answers "seconds, minutes or hours", not "how long exactly". Pricing a subset is what turns a complaint into a choice.

8. What Is Next

The matrix exists as a list and its cost is known. Chapter 31.5 runs it in parallel — where a careful measurement of the speedup curve contradicted the advice this module was expecting to give, because the first attempt measured a cold page cache instead of the parallelism.