Linux & Bash · Part II · Module 31 · Batch Execution, Test Lists and Seeds
Seeds and Reproducible Randomness
A randomised test is a different test on every seed, which makes the seed part of the test's identity rather than a tuning knob. This chapter measures that the same seed reproduces exactly and different seeds do not, then builds a sweep whose seeds are chosen by the caller, recorded, and reusable a year later.
Intermediate21 min readLinuxPart IISeedsReproducibilityBatch
Part II · Module 31 · Chapter 31.3 · Seeds and Reproducible Randomness
Chapter 31.2 made the list of tests data. A randomised test is not one test, though — it is one test per seed, and the seed is the second axis of the matrix.
1. The Same Seed Repeats; Different Seeds Do Not
$ for s in 1 7 7 42 99; do
> rm -rf "runs/random_s$s"
> bin/run_one.sh "random:$s" >/dev/null 2>&1
> printf ' %-8s %-10s %s\n' "$s" \
> "$(sed -n 's/.*checks=\([0-9]*\).*/\1/p' "runs/random_s$s/run.log" | tail -1)" \
> "$(grep -oE 'TEST (PASSED|FAILED)' "runs/random_s$s/run.log" | tail -1)"
> done
SEED CHECKS VERDICT
1 29802 TEST PASSED
7 29855 TEST PASSED
7 29855 TEST PASSED
42 29947 TEST PASSED
99 29678 TEST PASSEDTwo facts in one table. Four seeds produced four different check counts — 29,802, 29,855, 29,947, 29,678 — so the seed genuinely changes what the test exercises. And seed 7, run twice, produced 29,855 both times — so the stimulus is a function of the seed and nothing else.
That second property is what makes a seed worth recording and the first is why it matters. A failure at seed 42 is not a failure of random; it is a failure of random at seed 42, and without the number there is no test to re-run.
2. A Sweep Is a Loop Over Seeds
The registry's form:
for s in $(seq 1 20); do run_one.sh "random:$s"; doneseq has four shapes worth knowing, and the last two exist for a reason:
$ seq 1 5
1 2 3 4 5
$ seq 1 2 9
1 3 5 7 9
$ seq -w 1 10
01 02 03 04 05 06 07 08 09 10
$ seq -f %05g 1 3
00001 00002 00003-w pads to equal width and -f takes a format. Both exist because unpadded numbers sort wrongly, and a run directory named random_s9 sorts after random_s10:
$ printf 'random_s1\nrandom_s10\nrandom_s2\nrandom_s9\n' | sort
random_s1
random_s10
random_s2
random_s9That is Chapter 13.3's lexical trap in a directory listing, and seq -w removes it at the source rather than requiring sort -V everywhere downstream.
3. Variety and Reproducibility Are Not in Conflict
The instinct for "run different seeds every night" is to derive one from the clock:
bin/run_one.sh "random:$(date +%s)"That does give variety. Here is what it leaves behind:
$ s=$(date +%s); bin/run_one.sh "random:$s" >/dev/null 2>&1
$ ls -d runs/random_s* | tail -1
runs/random_s1791532953
$ grep -m1 'seed=' runs/random_s1791532953/run.log
TB test=random seed=1791532953 cycles=120000 verbosity=LOWTwo traces: the directory name, and a line in the log. And the second one exists only because this testbench echoes its configuration — Chapter 27.4 argued for that echo and this is one of the things it buys. A testbench that seeds itself internally and does not print the value leaves nothing, and a failure found that way is unreproducible by anybody including the person who found it.
The way to have both is to generate the variety in the caller, where it can be written down:
# twenty seeds, varied but reproducible, from a recorded generator seed
gen=${GEN_SEED:-12345}
awk -v n=20 -v s="$gen" 'BEGIN{srand(s); for(i=0;i<n;i++) printf "%d\n", rand()*100000}' > seeds.txt
printf '%s\n' "$gen" > seeds.gen$ awk -v n=6 -v s=12345 'BEGIN{srand(s); for(i=0;i<n;i++) printf "%d\n", rand()*100000}' | tr '\n' ' '
17839 39967 16659 21212 6193 5414
$ awk -v n=6 -v s=12345 'BEGIN{srand(s); for(i=0;i<n;i++) printf "%d\n", rand()*100000}' | tr '\n' ' '
17839 39967 16659 21212 6193 5414
$ awk -v n=6 -v s=999 'BEGIN{srand(s); for(i=0;i<n;i++) printf "%d\n", rand()*100000}' | tr '\n' ' '
263 64418 82428 26562 7275 7894Identical across invocations for the same generator seed, and different for a different one. So seeds.gen holding 12345 is enough to regenerate the entire sweep, and seeds.txt is enough to re-run it without the generator at all.
4. The Sweep
#!/bin/bash
# sweep TEST [N] [GEN_SEED] — run one test over N recorded seeds.
set -u
test=${1:-}
[ -n "$test" ] || { echo "usage: sweep TEST [N] [GEN_SEED]" >&2; exit 2; }
n=${2:-20}
gen=${3:-${GEN_SEED:-12345}}
root=${ROOT:-}; [ -n "$root" ] || { echo "sweep: set ROOT" >&2; exit 2; }
runs=${RUNS:-}; [ -n "$runs" ] || { echo "sweep: set RUNS" >&2; exit 2; }
case "$n" in (''|*[!0-9]*) echo "sweep: N must be a number, got '$n'" >&2; exit 2 ;; esac
[ "$n" -gt 0 ] || { echo "sweep: N must be at least 1" >&2; exit 2; }
mkdir -p "$runs" || exit 2
# the seeds, as values, plus the recipe that made them
awk -v n="$n" -v s="$gen" 'BEGIN{srand(s); for(i=0;i<n;i++) printf "%d\n", rand()*100000}' \
> "$runs/seeds.txt"
{ printf 'test %s\ngen_seed %s\ncount %s\nawk %s\n' \
"$test" "$gen" "$n" "$(awk --version 2>/dev/null | head -1 || awk -W version 2>&1 | head -1)"
} > "$runs/seeds.gen"
got=$(grep -c . "$runs/seeds.txt")
[ "$got" -eq "$n" ] || { echo "sweep: asked for $n seeds, generated $got" >&2; exit 2; }
printf 'sweep: %s over %s seeds (generator %s)\n' "$test" "$n" "$gen"
pass=0 fail=0 noverdict=0 error=0
while IFS= read -r -u 9 s || [ -n "$s" ]; do
"$root/bin/run_one.sh" "$test:$s" < /dev/null
case "$?" in
(0) pass=$((pass+1)) ;;
(1) fail=$((fail+1)); printf 'FAIL %s:%s\n' "$test" "$s" ;;
(3) noverdict=$((noverdict+1)); printf 'NO-VERDICT %s:%s\n' "$test" "$s" ;;
(*) error=$((error+1)); printf 'ERROR %s:%s\n' "$test" "$s" ;;
esac
done 9< "$runs/seeds.txt"
printf '\n%s: %d/%d passed, %d failed, %d no verdict, %d error\n' \
"$test" "$pass" "$n" "$fail" "$noverdict" "$error"
printf 'seeds in %s ; recipe in %s\n' "$runs/seeds.txt" "$runs/seeds.gen"
[ $((fail + noverdict + error)) -eq 0 ]$ ROOT=$PWD RUNS=$PWD/runs sweep random 6
sweep: random over 6 seeds (generator 12345)
random: 6/6 passed, 0 failed, 0 no verdict, 0 error
seeds in /…/m31/runs/seeds.txt ; recipe in /…/m31/runs/seeds.gen
$ cat runs/seeds.gen
test random
gen_seed 12345
count 6
awk awk version 20200816
$ cat runs/seeds.txt | tr '\n' ' '
17839 39967 16659 21212 6193 5414Those are the same six values §3 generated, which is the point of recording them. And every refusal:
$ sweep
usage: sweep TEST [N] [GEN_SEED]
$ sweep random abc
sweep: N must be a number, got 'abc'
$ sweep random 0
sweep: N must be at least 1
$ ( unset ROOT; sweep random 2 )
sweep: set ROOT
$ sweep random abc >/dev/null 2>&1; echo $?
2Six decisions.
N is validated as a number. Without the case, sweep random abc reaches [ "$n" -gt 0 ] and bash reports integer expression expected — a message about the script rather than about the input. Chapter 20.3's rule, and the explicit case is needed because -gt is not a validator.
It checks that awk produced the number of seeds asked for. A generator that silently produces fewer is a sweep that silently runs fewer, which is Chapter 31.2's reconciliation applied to the other axis.
It records awk's identity in seeds.gen, because the sequence is implementation-specific — the callout above.
It prints only the failures, not every pass. Twenty lines of PASS is noise; the ones that matter are named with their seed, so the re-run command is visible in the output.
The seed appears in the failure line as test:seed — exactly the form run_one.sh takes, so reproducing a failure is copy-and-paste rather than reconstruction.
And the guards exit 2, so a bad N is not counted as twenty failing tests (Chapter 31.1 §4).
5. Which Seeds to Run
Three sets, with different purposes, and a flow usually needs all three.
A fixed set, every night. Reproducible night to night, so a new failure means the design changed rather than the stimulus. This is what seeds.gen with a constant generator gives, and it is the set a trend is measured against.
A rotating set. New seeds each night, to find what the fixed set never reaches. Generated with the date as the generator seed, which keeps it reproducible:
# varied per night, and regenerable from the date alone
GEN_SEED=$(date -u '+%Y%m%d') sweep random 20That is the honest version of date +%s as a seed: the clock chooses the set, the set is written down, and 20261009 regenerates it exactly.
The failing set. Last night's failures, re-run first. That needs the previous results, which is Chapter 32.6's subject; the part that belongs here is that it is only possible because the seeds were recorded as values.
6. 03 and 3 Are the Same Seed and Two Different Runs
Section 2's sweep used seq. Chapter 31.4 uses seq -w, which pads to a common width so the names sort correctly:
$ seq 1 4 | tr '\n' ' '
1 2 3 4
$ seq -w 1 12 | head -4 | tr '\n' ' '
01 02 03 04Both spellings name the same seeds. The question is whether anything downstream agrees.
Running seed 3 and seed 03:
=== running seed 3 and seed 03 ===
+SEED=3 -> dir random_s3 rc=0
+SEED=03 -> dir random_s03 rc=0
=== what the testbench reports for each ===
3 TB test=random seed=3 cycles=120000 verbosity=LOW
03 TB test=random seed=3 cycles=120000 verbosity=LOW
=== are the two runs the same run? ===
logs are IDENTICAL -- the same simulation, stored under two names
checks: s3=29746 s03=29746The tool normalises the padding; the filesystem does not. The testbench read %d and got 3 both times, so the two runs are the same simulation down to the check count — and they are sitting in two directories:
=== directories on disk ===
random_s03
random_s3Which matters as soon as the sweep is more than a handful:
=== and a 20-seed sweep, both spellings, counted ===
seq -w 1 20 distinct dir names: 20
seq 1 20 distinct dir names: 20
union of the two: 29 (expected 20 if the spellings agree)Twenty-nine names for twenty seeds. The nine extra are seeds 1 through 9, each of which now exists twice. A regression that generated its seed list one way last night and the other way tonight has not run 20 seeds and then 20 more; it has run 11 new seeds and repeated 9 old ones, while a results directory containing 29 entries says it did something larger than either.
This is Chapter 31.6's identity rule from the other side. There, two runs that differed shared one name. Here, one run has two names — and the consequence is the same kind of lie about how much was covered.
The normaliser, and the trap inside it
The fix is to pick one spelling and normalise at the boundary, before the name is built. The expression everybody reaches for is bash arithmetic:
seed=$(( seed )) # strip the paddingMeasured across the values a padded sweep actually produces:
seed $((s)) rc $((10#$s)) rc note
1 1 0 1 0 agree
03 3 0 3 0 agree
07 7 0 7 0 agree
08 value too 1 8 0 $(( )) ERRORS
09 value too 1 9 0 $(( )) ERRORS
012 10 0 12 0 OCTAL: 10 not 12
020 16 0 20 0 OCTAL: 16 not 20
20 20 0 20 0 agreeTwo distinct failures, and the second is the dangerous one.
08 and 09 are errors. A leading zero makes bash read the number as octal, and octal has no digits 8 or 9: value too great for base (error token is "08"), exit status 1. So a seq -w 1 20 sweep works for seeds 01 through 07, then fails on 08 — the eighth run of twenty, far enough in to look like something else.
012 and 020 are silently wrong. Those are valid octal, so there is no error: 012 becomes 10 and 020 becomes 16. A three-digit sweep from seq -w 1 100 therefore runs seed 10 while calling it seed 12, and seed 16 while calling it 20 — and records the wrong number in the artifact that exists to make the run reproducible, which is the one defect this chapter is about.
10# forces base ten and both columns agree:
# normalise once, at the point the seed enters the script
seed=$(( 10#$seed ))Note the $. Writing $(( 10#seed )) without it asks bash to read the letters seed as a base-ten number, which fails with the same value too great for base message against a different token — a mistake I made while measuring this table, and one whose error text points at the base rather than at the missing $.
Practice
- Run one test at four seeds and record the check counts. Confirm they differ.
- Run the same seed twice and confirm the counts are identical.
- Explain why that makes the seed part of the test's identity.
- Try all four
seqforms and say what-wis for. - Sort
random_s1 random_s10 random_s2 random_s9and find the lexical trap. - Write the sweep as a loop over a file rather than a range, with both of 31.2's guards.
- Seed a run from
date +%sand then try to reproduce it from the results alone. - Find the two traces it left, and say which one depends on the testbench.
- Generate six seeds from a fixed
srandtwice and confirm they match. - Change the generator seed and confirm the set changes.
- Explain the difference between a generator seed and a run seed, and what conflating them breaks.
- Record
awk's version with your seed set and say why. - Write
sweepand confirm a non-numericNgives a message about the input. - Make
awkproduce fewer seeds than asked and confirm the reconciliation catches it. - Measure the check-count spread across your own sweep and interpret it.
- Generate the same seeds with
seqandseq -wand compare the two lists. - Run seed 3 and seed 03 and diff the two logs.
- Count the directories both spellings produce for a 20-seed sweep, then count their union.
- Explain the number 29 without counting again.
- Normalise a padded seed with
$(( seed ))and find the first value that errors. - Find a padded value that is silently wrong rather than an error, and say what it becomes.
- Switch to
10#$seedand re-run the whole table. - Drop the
$from10#$seedand read the error message carefully. - Argue for using one spelling everywhere instead, and say what
seq -w's width does to that argument.
Check Your Understanding
- What two facts does the five-row seed table establish?
- Why is a failure "at seed 42" rather than "in test random"?
- What are
seq -wandseq -ffor? - Which earlier chapter's trap does an unpadded seed reintroduce?
- Why should the sweep iterate a file rather than
$(seq 1 20)? - Which two guards does the file-based sweep inherit, and why each?
- What exactly does
date +%sas a run seed leave behind? - Which of those traces depends on the testbench, and what happens without it?
- What is the difference between a generator seed and a run seed?
- Give the specific failure mode of recording only the generator seed.
- Why is
seeds.txtthe artifact that matters rather thanseeds.gen? - Why validate
Nwith acaserather than relying on-gt? - Why check the number of seeds generated?
- Why print only failures, and in
test:seedform? - What was the check-count spread across four seeds, and what does a tight spread suggest?
- Do
+SEED=3and+SEED=03run the same simulation? How do you know? - What does the filesystem do with those two spellings?
- Why does a 20-seed sweep in both spellings produce 29 directory names?
- What has a regression actually covered if it switched spelling between two nights?
- How is this the mirror image of Chapter 31.6's collision?
- What happens to
$(( 08 )), and why? - What happens to
$(( 012 )), and why is that worse than an error? - What does
10#do, and what breaks if you omit the$? - What is the simpler alternative to normalising, and what undermines it?
7. Remember
- Whether anything else in the flow varies between runs is worth measuring rather than assuming — Chapter 34.6 found this simulation byte-identical across five runs and the compiler not reproducible at all.
3and03are one seed to the tool and two directories on disk. The two spellings produced identical logs in separate run directories.- Twenty seeds in both spellings give 29 names, the nine extras being seeds 1 to 9 — a results directory that overstates what was covered.
$(( 08 ))is an error and$(( 012 ))is 10, because a leading zero means octal. The silent one is the dangerous one.- Use
$(( 10#$seed ))— and keep the$, or the error blames the base instead of the typo. - Better still, pick one spelling and use it everywhere — but take the width from the sweep's declared size, not from
seq -w's guess. - The same seed reproduced exactly (29,855 twice) and four seeds gave four different check counts. The seed is part of the test's identity.
seq -wandseq -fpad, becauserandom_s10sorts beforerandom_s2.- Iterate a seed file, not a range, with 31.2's two guards — the set is rarely 1 to 20.
date +%sas a run seed leaves two traces, one of which exists only because the testbench echoes its configuration.- Generate variety in the caller: a recorded generator seed gives a set that is both varied and regenerable.
- Two layers of seed. The generator chooses the set; each run seed chooses the stimulus. Record the values and the recipe.
srandis implementation-specific, so the values are the artifact and the recipe is a convenience.- Validate
Nwith acase—-gtreports on the script, not the input. - Reconcile the seed count, print only failures, and print them as
test:seed. - Measure your own check-count spread. Here it was 0.9%, which says more seeds of this test buy little.
8. What Is Next
Tests are a list and seeds are a list. Chapter 31.4 multiplies them — along with any configuration axis — into the concrete set of runs, and makes the size of that set visible before anything is launched.
Where this fits
Part of the Linux & Bash curriculum.
