Skip to content
VLSI Mentor

Linux & Bash · Part II · Module 31 · Batch Execution, Test Lists and Seeds

Seeds and Reproducible Randomness

A randomised test is a different test on every seed, which makes the seed part of the test's identity rather than a tuning knob. This chapter measures that the same seed reproduces exactly and different seeds do not, then builds a sweep whose seeds are chosen by the caller, recorded, and reusable a year later.

Intermediate21 min readLinuxPart IISeedsReproducibilityBatch

Part II · Module 31 · Chapter 31.3 · Seeds and Reproducible Randomness

Chapter 31.2 made the list of tests data. A randomised test is not one test, though — it is one test per seed, and the seed is the second axis of the matrix.

1. The Same Seed Repeats; Different Seeds Do Not

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ for s in 1 7 7 42 99; do
>   rm -rf "runs/random_s$s"
>   bin/run_one.sh "random:$s" >/dev/null 2>&1
>   printf '  %-8s %-10s %s\n' "$s" \
>     "$(sed -n 's/.*checks=\([0-9]*\).*/\1/p' "runs/random_s$s/run.log" | tail -1)" \
>     "$(grep -oE 'TEST (PASSED|FAILED)' "runs/random_s$s/run.log" | tail -1)"
> done
  SEED     CHECKS     VERDICT
  1        29802      TEST PASSED
  7        29855      TEST PASSED
  7        29855      TEST PASSED
  42       29947      TEST PASSED
  99       29678      TEST PASSED

Two facts in one table. Four seeds produced four different check counts — 29,802, 29,855, 29,947, 29,678 — so the seed genuinely changes what the test exercises. And seed 7, run twice, produced 29,855 both times — so the stimulus is a function of the seed and nothing else.

That second property is what makes a seed worth recording and the first is why it matters. A failure at seed 42 is not a failure of random; it is a failure of random at seed 42, and without the number there is no test to re-run.

2. A Sweep Is a Loop Over Seeds

The registry's form:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
for s in $(seq 1 20); do run_one.sh "random:$s"; done

seq has four shapes worth knowing, and the last two exist for a reason:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ seq 1 5
1 2 3 4 5
$ seq 1 2 9
1 3 5 7 9
$ seq -w 1 10
01 02 03 04 05 06 07 08 09 10
$ seq -f %05g 1 3
00001 00002 00003

-w pads to equal width and -f takes a format. Both exist because unpadded numbers sort wrongly, and a run directory named random_s9 sorts after random_s10:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ printf 'random_s1\nrandom_s10\nrandom_s2\nrandom_s9\n' | sort
random_s1
random_s10
random_s2
random_s9

That is Chapter 13.3's lexical trap in a directory listing, and seq -w removes it at the source rather than requiring sort -V everywhere downstream.

3. Variety and Reproducibility Are Not in Conflict

The instinct for "run different seeds every night" is to derive one from the clock:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
bin/run_one.sh "random:$(date +%s)"

That does give variety. Here is what it leaves behind:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ s=$(date +%s); bin/run_one.sh "random:$s" >/dev/null 2>&1
$ ls -d runs/random_s* | tail -1
runs/random_s1791532953
$ grep -m1 'seed=' runs/random_s1791532953/run.log
TB  test=random seed=1791532953 cycles=120000 verbosity=LOW

Two traces: the directory name, and a line in the log. And the second one exists only because this testbench echoes its configuration — Chapter 27.4 argued for that echo and this is one of the things it buys. A testbench that seeds itself internally and does not print the value leaves nothing, and a failure found that way is unreproducible by anybody including the person who found it.

The way to have both is to generate the variety in the caller, where it can be written down:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
# twenty seeds, varied but reproducible, from a recorded generator seed
gen=${GEN_SEED:-12345}
awk -v n=20 -v s="$gen" 'BEGIN{srand(s); for(i=0;i<n;i++) printf "%d\n", rand()*100000}' > seeds.txt
printf '%s\n' "$gen" > seeds.gen
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ awk -v n=6 -v s=12345 'BEGIN{srand(s); for(i=0;i<n;i++) printf "%d\n", rand()*100000}' | tr '\n' ' '
17839 39967 16659 21212 6193 5414
$ awk -v n=6 -v s=12345 'BEGIN{srand(s); for(i=0;i<n;i++) printf "%d\n", rand()*100000}' | tr '\n' ' '
17839 39967 16659 21212 6193 5414
$ awk -v n=6 -v s=999   'BEGIN{srand(s); for(i=0;i<n;i++) printf "%d\n", rand()*100000}' | tr '\n' ' '
263 64418 82428 26562 7275 7894

Identical across invocations for the same generator seed, and different for a different one. So seeds.gen holding 12345 is enough to regenerate the entire sweep, and seeds.txt is enough to re-run it without the generator at all.

4. The Sweep

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
#!/bin/bash
# sweep TEST [N] [GEN_SEED] — run one test over N recorded seeds.
set -u
test=${1:-}
[ -n "$test" ] || { echo "usage: sweep TEST [N] [GEN_SEED]" >&2; exit 2; }
n=${2:-20}
gen=${3:-${GEN_SEED:-12345}}
root=${ROOT:-}; [ -n "$root" ] || { echo "sweep: set ROOT" >&2; exit 2; }
runs=${RUNS:-}; [ -n "$runs" ] || { echo "sweep: set RUNS" >&2; exit 2; }
case "$n" in (''|*[!0-9]*) echo "sweep: N must be a number, got '$n'" >&2; exit 2 ;; esac
[ "$n" -gt 0 ] || { echo "sweep: N must be at least 1" >&2; exit 2; }
mkdir -p "$runs" || exit 2

# the seeds, as values, plus the recipe that made them
awk -v n="$n" -v s="$gen" 'BEGIN{srand(s); for(i=0;i<n;i++) printf "%d\n", rand()*100000}' \
  > "$runs/seeds.txt"
{ printf 'test %s\ngen_seed %s\ncount %s\nawk %s\n' \
    "$test" "$gen" "$n" "$(awk --version 2>/dev/null | head -1 || awk -W version 2>&1 | head -1)"
} > "$runs/seeds.gen"

got=$(grep -c . "$runs/seeds.txt")
[ "$got" -eq "$n" ] || { echo "sweep: asked for $n seeds, generated $got" >&2; exit 2; }
printf 'sweep: %s over %s seeds (generator %s)\n' "$test" "$n" "$gen"

pass=0 fail=0 noverdict=0 error=0
while IFS= read -r -u 9 s || [ -n "$s" ]; do
  "$root/bin/run_one.sh" "$test:$s" < /dev/null
  case "$?" in
    (0) pass=$((pass+1)) ;;
    (1) fail=$((fail+1));      printf 'FAIL       %s:%s\n' "$test" "$s" ;;
    (3) noverdict=$((noverdict+1)); printf 'NO-VERDICT %s:%s\n' "$test" "$s" ;;
    (*) error=$((error+1));    printf 'ERROR      %s:%s\n' "$test" "$s" ;;
  esac
done 9< "$runs/seeds.txt"

printf '\n%s: %d/%d passed, %d failed, %d no verdict, %d error\n' \
       "$test" "$pass" "$n" "$fail" "$noverdict" "$error"
printf 'seeds in %s ; recipe in %s\n' "$runs/seeds.txt" "$runs/seeds.gen"
[ $((fail + noverdict + error)) -eq 0 ]
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ ROOT=$PWD RUNS=$PWD/runs sweep random 6
sweep: random over 6 seeds (generator 12345)

random: 6/6 passed, 0 failed, 0 no verdict, 0 error
seeds in /…/m31/runs/seeds.txt ; recipe in /…/m31/runs/seeds.gen
$ cat runs/seeds.gen
test random
gen_seed 12345
count 6
awk awk version 20200816
$ cat runs/seeds.txt | tr '\n' ' '
17839 39967 16659 21212 6193 5414

Those are the same six values §3 generated, which is the point of recording them. And every refusal:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ sweep
usage: sweep TEST [N] [GEN_SEED]
$ sweep random abc
sweep: N must be a number, got 'abc'
$ sweep random 0
sweep: N must be at least 1
$ ( unset ROOT; sweep random 2 )
sweep: set ROOT
$ sweep random abc >/dev/null 2>&1; echo $?
2

Six decisions.

N is validated as a number. Without the case, sweep random abc reaches [ "$n" -gt 0 ] and bash reports integer expression expected — a message about the script rather than about the input. Chapter 20.3's rule, and the explicit case is needed because -gt is not a validator.

It checks that awk produced the number of seeds asked for. A generator that silently produces fewer is a sweep that silently runs fewer, which is Chapter 31.2's reconciliation applied to the other axis.

It records awk's identity in seeds.gen, because the sequence is implementation-specific — the callout above.

It prints only the failures, not every pass. Twenty lines of PASS is noise; the ones that matter are named with their seed, so the re-run command is visible in the output.

The seed appears in the failure line as test:seed — exactly the form run_one.sh takes, so reproducing a failure is copy-and-paste rather than reconstruction.

And the guards exit 2, so a bad N is not counted as twenty failing tests (Chapter 31.1 §4).

5. Which Seeds to Run

Three sets, with different purposes, and a flow usually needs all three.

A fixed set, every night. Reproducible night to night, so a new failure means the design changed rather than the stimulus. This is what seeds.gen with a constant generator gives, and it is the set a trend is measured against.

A rotating set. New seeds each night, to find what the fixed set never reaches. Generated with the date as the generator seed, which keeps it reproducible:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
# varied per night, and regenerable from the date alone
GEN_SEED=$(date -u '+%Y%m%d') sweep random 20

That is the honest version of date +%s as a seed: the clock chooses the set, the set is written down, and 20261009 regenerates it exactly.

The failing set. Last night's failures, re-run first. That needs the previous results, which is Chapter 32.6's subject; the part that belongs here is that it is only possible because the seeds were recorded as values.

6. 03 and 3 Are the Same Seed and Two Different Runs

Section 2's sweep used seq. Chapter 31.4 uses seq -w, which pads to a common width so the names sort correctly:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ seq 1 4 | tr '\n' ' '
1 2 3 4
$ seq -w 1 12 | head -4 | tr '\n' ' '
01 02 03 04

Both spellings name the same seeds. The question is whether anything downstream agrees.

Running seed 3 and seed 03:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
=== running seed 3 and seed 03 ===
  +SEED=3    -> dir random_s3          rc=0
  +SEED=03   -> dir random_s03         rc=0

=== what the testbench reports for each ===
  3    TB  test=random seed=3 cycles=120000 verbosity=LOW
  03   TB  test=random seed=3 cycles=120000 verbosity=LOW

=== are the two runs the same run? ===
  logs are IDENTICAL -- the same simulation, stored under two names
  checks: s3=29746  s03=29746

The tool normalises the padding; the filesystem does not. The testbench read %d and got 3 both times, so the two runs are the same simulation down to the check count — and they are sitting in two directories:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
=== directories on disk ===
    random_s03
    random_s3

Which matters as soon as the sweep is more than a handful:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
=== and a 20-seed sweep, both spellings, counted ===
  seq -w 1 20  distinct dir names: 20
  seq 1 20     distinct dir names: 20
  union of the two:                29  (expected 20 if the spellings agree)

Twenty-nine names for twenty seeds. The nine extra are seeds 1 through 9, each of which now exists twice. A regression that generated its seed list one way last night and the other way tonight has not run 20 seeds and then 20 more; it has run 11 new seeds and repeated 9 old ones, while a results directory containing 29 entries says it did something larger than either.

This is Chapter 31.6's identity rule from the other side. There, two runs that differed shared one name. Here, one run has two names — and the consequence is the same kind of lie about how much was covered.

The normaliser, and the trap inside it

The fix is to pick one spelling and normalise at the boundary, before the name is built. The expression everybody reaches for is bash arithmetic:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
seed=$(( seed ))        # strip the padding

Measured across the values a padded sweep actually produces:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  seed    $((s))     rc     $((10#$s))   rc     note
  1       1          0      1            0      agree
  03      3          0      3            0      agree
  07      7          0      7            0      agree
  08      value too  1      8            0      $(( )) ERRORS
  09      value too  1      9            0      $(( )) ERRORS
  012     10         0      12           0      OCTAL: 10 not 12
  020     16         0      20           0      OCTAL: 16 not 20
  20      20         0      20           0      agree

Two distinct failures, and the second is the dangerous one.

08 and 09 are errors. A leading zero makes bash read the number as octal, and octal has no digits 8 or 9: value too great for base (error token is "08"), exit status 1. So a seq -w 1 20 sweep works for seeds 01 through 07, then fails on 08 — the eighth run of twenty, far enough in to look like something else.

012 and 020 are silently wrong. Those are valid octal, so there is no error: 012 becomes 10 and 020 becomes 16. A three-digit sweep from seq -w 1 100 therefore runs seed 10 while calling it seed 12, and seed 16 while calling it 20 — and records the wrong number in the artifact that exists to make the run reproducible, which is the one defect this chapter is about.

10# forces base ten and both columns agree:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
# normalise once, at the point the seed enters the script
seed=$(( 10#$seed ))

Note the $. Writing $(( 10#seed )) without it asks bash to read the letters seed as a base-ten number, which fails with the same value too great for base message against a different token — a mistake I made while measuring this table, and one whose error text points at the base rather than at the missing $.

Practice

  1. Run one test at four seeds and record the check counts. Confirm they differ.
  2. Run the same seed twice and confirm the counts are identical.
  3. Explain why that makes the seed part of the test's identity.
  4. Try all four seq forms and say what -w is for.
  5. Sort random_s1 random_s10 random_s2 random_s9 and find the lexical trap.
  6. Write the sweep as a loop over a file rather than a range, with both of 31.2's guards.
  7. Seed a run from date +%s and then try to reproduce it from the results alone.
  8. Find the two traces it left, and say which one depends on the testbench.
  9. Generate six seeds from a fixed srand twice and confirm they match.
  10. Change the generator seed and confirm the set changes.
  11. Explain the difference between a generator seed and a run seed, and what conflating them breaks.
  12. Record awk's version with your seed set and say why.
  13. Write sweep and confirm a non-numeric N gives a message about the input.
  14. Make awk produce fewer seeds than asked and confirm the reconciliation catches it.
  15. Measure the check-count spread across your own sweep and interpret it.
  16. Generate the same seeds with seq and seq -w and compare the two lists.
  17. Run seed 3 and seed 03 and diff the two logs.
  18. Count the directories both spellings produce for a 20-seed sweep, then count their union.
  19. Explain the number 29 without counting again.
  20. Normalise a padded seed with $(( seed )) and find the first value that errors.
  21. Find a padded value that is silently wrong rather than an error, and say what it becomes.
  22. Switch to 10#$seed and re-run the whole table.
  23. Drop the $ from 10#$seed and read the error message carefully.
  24. Argue for using one spelling everywhere instead, and say what seq -w's width does to that argument.

Check Your Understanding

  1. What two facts does the five-row seed table establish?
  2. Why is a failure "at seed 42" rather than "in test random"?
  3. What are seq -w and seq -f for?
  4. Which earlier chapter's trap does an unpadded seed reintroduce?
  5. Why should the sweep iterate a file rather than $(seq 1 20)?
  6. Which two guards does the file-based sweep inherit, and why each?
  7. What exactly does date +%s as a run seed leave behind?
  8. Which of those traces depends on the testbench, and what happens without it?
  9. What is the difference between a generator seed and a run seed?
  10. Give the specific failure mode of recording only the generator seed.
  11. Why is seeds.txt the artifact that matters rather than seeds.gen?
  12. Why validate N with a case rather than relying on -gt?
  13. Why check the number of seeds generated?
  14. Why print only failures, and in test:seed form?
  15. What was the check-count spread across four seeds, and what does a tight spread suggest?
  16. Do +SEED=3 and +SEED=03 run the same simulation? How do you know?
  17. What does the filesystem do with those two spellings?
  18. Why does a 20-seed sweep in both spellings produce 29 directory names?
  19. What has a regression actually covered if it switched spelling between two nights?
  20. How is this the mirror image of Chapter 31.6's collision?
  21. What happens to $(( 08 )), and why?
  22. What happens to $(( 012 )), and why is that worse than an error?
  23. What does 10# do, and what breaks if you omit the $?
  24. What is the simpler alternative to normalising, and what undermines it?

7. Remember

  • Whether anything else in the flow varies between runs is worth measuring rather than assuming — Chapter 34.6 found this simulation byte-identical across five runs and the compiler not reproducible at all.
  • 3 and 03 are one seed to the tool and two directories on disk. The two spellings produced identical logs in separate run directories.
  • Twenty seeds in both spellings give 29 names, the nine extras being seeds 1 to 9 — a results directory that overstates what was covered.
  • $(( 08 )) is an error and $(( 012 )) is 10, because a leading zero means octal. The silent one is the dangerous one.
  • Use $(( 10#$seed )) — and keep the $, or the error blames the base instead of the typo.
  • Better still, pick one spelling and use it everywhere — but take the width from the sweep's declared size, not from seq -w's guess.
  • The same seed reproduced exactly (29,855 twice) and four seeds gave four different check counts. The seed is part of the test's identity.
  • seq -w and seq -f pad, because random_s10 sorts before random_s2.
  • Iterate a seed file, not a range, with 31.2's two guards — the set is rarely 1 to 20.
  • date +%s as a run seed leaves two traces, one of which exists only because the testbench echoes its configuration.
  • Generate variety in the caller: a recorded generator seed gives a set that is both varied and regenerable.
  • Two layers of seed. The generator chooses the set; each run seed chooses the stimulus. Record the values and the recipe.
  • srand is implementation-specific, so the values are the artifact and the recipe is a convenience.
  • Validate N with a case — -gt reports on the script, not the input.
  • Reconcile the seed count, print only failures, and print them as test:seed.
  • Measure your own check-count spread. Here it was 0.9%, which says more seeds of this test buy little.

8. What Is Next

Tests are a list and seeds are a list. Chapter 31.4 multiplies them — along with any configuration axis — into the concrete set of runs, and makes the size of that set visible before anything is launched.

Where this fits

Part of the Linux & Bash curriculum.