Per-Test Options, Seeds and Plusargs

Part II · Module 25 · Chapter 25.3 · Per-Test Options, Seeds and Plusargs

1. One Entry, Many Runs

Chapter 25.2 turned the test list into records. One record is one test:

{ name => "rand", opts => { count => 4 }, plusargs => ["+n=60"] }

That is not yet something you can launch. count => 4 means four runs, and each run needs its own seed, its own command line and its own name. So the runner needs one more step, called expansion: turn each test into the list of runs that will actually be launched.

A run has five things, and every one of them must be decided before launch:

The rest of this chapter is about getting the seed and the arguments right, because those are the two fields where a small mistake gives a wrong result with no error.

2. The Seed Is the Run's Identity

A random test does not choose new random values every time it runs. It computes them from a seed. Same bench, same seed, same values. In the fixture from Chapter 25.1, rand with seed 2 always hits the adder's bug at a=64 b=192, and seed 3 never does:

$ bin/simv +test=rand +seed=2
TEST rand SEED 2
ERROR: a=64 b=192 y=0 expected=255
TEST FAILED (1 errors)
tb/tb.v:60: $finish called at 40 (1s)

That is what makes a random failure useful: if you know the seed, you can run the failure again, as often as you like, while you debug. If you do not know the seed, all you know is that some run failed, once.

This leads to the rule for the whole chapter: the runner chooses every seed itself, before launch, and writes it in the report. A seed that the simulator picked by itself, from the clock, may or may not be printed somewhere in a log; a seed the runner chose is a fact the runner already has.

3. Who Chooses the Seed: Three Schemes

There are three common ways to number seeds. They look equally good on day one. The difference appears when someone edits the test list. Here, one new test, alu_new, is inserted at the top, and we look at what happens to the four seeds of rand:

#!/usr/bin/perl
# seed_schemes.pl -- three ways to assign seeds, before and after someone
# inserts one new test at the top of the list.
use strict;
use warnings;
use Digest::MD5 qw(md5);

my $BASE = 1000;
my @v1 = ([smoke => 1], [corner => 1], [rand => 4]);
my @v2 = ([alu_new => 2], @v1);                     # one test inserted

my %scheme = (
    "A global counter" => sub { my ($test, $i, $k) = @_; $BASE + $k },
    "B per-test index" => sub { my ($test, $i, $k) = @_; $BASE + $i },
    "C derived"        => sub { my ($test, $i, $k) = @_;
                                unpack("N", md5("$BASE/$test/$i")) % 2147483646 + 1 },
);

sub seeds_of {                                      # test => [seeds]
    my ($list, $f) = @_;
    my ($k, %s) = (0);
    for my $e (@$list) {
        my ($test, $count) = @$e;
        push @{ $s{$test} }, $f->($test, $_, ++$k) for 1 .. $count;
    }
    return \%s;
}

for my $name (sort keys %scheme) {
    my ($s1, $s2) = map { seeds_of($_, $scheme{$name}) } \@v1, \@v2;
    my @old = @{ $s1->{rand} };
    my @new = @{ $s2->{rand} };
    my $changed = grep { $old[$_] != $new[$_] } 0 .. $#old;
    my %uses;                                       # how many tests use each seed
    for my $t (keys %$s2) { $uses{$_}++ for @{ $s2->{$t} } }
    my $shared = grep { $_ > 1 } values %uses;
    print "$name\n";
    print "  rand before the edit: @old\n";
    print "  rand after the edit:  @new\n";
    print "  changed $changed of 4; seeds used by more than one test: $shared\n";
}

A global counter
  rand before the edit: 1003 1004 1005 1006
  rand after the edit:  1005 1006 1007 1008
  changed 4 of 4; seeds used by more than one test: 0
B per-test index
  rand before the edit: 1001 1002 1003 1004
  rand after the edit:  1001 1002 1003 1004
  changed 0 of 4; seeds used by more than one test: 2
C derived
  rand before the edit: 1711902916 1956614937 1855608764 154077147
  rand after the edit:  1711902916 1956614937 1855608764 154077147
  changed 0 of 4; seeds used by more than one test: 0

What each scheme does:

• A — global counter. Every run in the whole regression gets the next number. Simple, and every seed is different. But inserting one test moved every seed after it. Tonight's rand runs are different stimulus from last night's, although nobody touched rand. If one of them now fails, is it a new bug, or just a new seed? The report cannot tell you. Comparing tonight with last night (Chapter 25.8 and the trend reports teams build on top of it) becomes a comparison of different things.
• B — per-test index. Seed = base + 1, base + 2, … within each test. Inserting a test changes nothing else, which fixes A's problem. The cost: every test uses the same few seeds. alu_new, smoke, corner and rand all start at 1001. If two tests draw random values in the same order, they get the same values, and you are running one test twice.
• C — derived. The seed is computed from the base, the test's name and the run's index, here by hashing them with MD5 and taking the result modulo the seed range. Inserting a test changes nothing else, and no two tests share a seed. The seeds are large, unfriendly numbers, but nobody should be typing them anyway: they are copied from the report.

The scheme in the registry's example for this chapter, $base_seed + $i, is scheme B. It is fine for a small regression. This chapter's expander uses scheme C.

Why MD5? Not for security — any function that mixes its input well would do. MD5 is used because Digest::MD5 ships with Perl, gives the same answer on every machine and every Perl version, and is fast. Do not use Perl's rand with srand for this: its sequence is not guaranteed to be the same across Perl builds, so the same base could give different seeds on two machines.

4. The Same Seeds Every Night

Scheme C still needs a base. If the base is a constant, the nightly regression runs exactly the same seeds every night. After the first night, it can only re-find what it already found. Here is a measurement on the real bench: eight rand runs per night for three nights, once with a fixed set of seeds and once with seeds derived from the date:

#!/usr/bin/perl
# nightly_seeds.pl -- three nights of eight rand runs: fixed seeds against
# seeds derived from the date.
use strict;
use warnings;
use Digest::MD5 qw(md5);

system("bin/build") == 0 or die "build failed\n";

sub derive_seed {
    my ($base, $test, $i) = @_;
    return unpack("N", md5("$base/$test/$i")) % 2147483646 + 1;
}
sub fails {
    my ($seed) = @_;
    return `bin/simv +test=rand +seed=$seed` =~ /^TEST FAILED/m ? 1 : 0;
}

my (%fixed, %dated);
for my $night (1 .. 3) {
    my $date  = 20261010 + $night;                    # the base for this night
    my @f = grep { fails($_) } map { 1000 + $_ } 1 .. 8;
    my @d = grep { fails($_) } map { derive_seed($date, "rand", $_) } 1 .. 8;
    $fixed{$_}++ for @f;
    $dated{$_}++ for @d;
    printf "night %d  fixed: %-10s  dated (base %d): %s\n",
           $night, "@f", $date, @d ? "@d" : "(none)";
}
printf "distinct failing seeds over 3 nights: fixed %d, dated %d\n",
       scalar keys %fixed, scalar keys %dated;

night 1  fixed: 1003 1005   dated (base 20261011): 431336034 1703094151 125595610 227831471
night 2  fixed: 1003 1005   dated (base 20261012): 995150769
night 3  fixed: 1003 1005   dated (base 20261013): (none)
distinct failing seeds over 3 nights: fixed 2, dated 5

The fixed seeds found seeds 1003 and 1005 on night 1, and then found exactly the same two again on nights 2 and 3. Two nights of compute, zero new information. The dated seeds explored 24 different seeds and found five that fail.

Three honest notes on this result:

1. Five failing seeds is not five bugs. All of them hit the same line of RTL — the sum of exactly 256. Counting distinct failing seeds measures how often the stimulus reaches a bug, not how many bugs there are. Chapter 21.9 classifies failures by signature for exactly this reason.
2. Night 3 found nothing. With new seeds every night, a night with no failures does not mean the bug was fixed. It means these eight seeds missed it. A report that says "rand: PASS" on night 3 is true and misleading at the same time, and only the earlier nights' failing seeds — rerun with the same seeds, as Chapter 25.8 does — can show that the bug is still there.
3. The date base must be recorded. Changing seeds every night is only safe because the base is written at the top of the report. With the base and the test name, every seed of that night can be recomputed, and every failure rerun.

A common, practical choice is: a fixed base for merge-gating regressions (the same seeds every time, so a change in result means a change in the design), and a varying base for nightly regressions (new seeds every night, to find new failures). Both record the base.

5. The Seed's Legal Range

A seed is a number the runner passes to the simulator as text. The bench reads it with $value$plusargs("seed=%d", seed) into a Verilog integer, which is 32 bits, signed. So what happens to a seed that does not fit, or is not a number at all? Ask the real bench:

#!/usr/bin/perl
# seed_edges.pl -- what the bench actually does with unusual seed values.
use strict;
use warnings;

system("bin/build") == 0 or die "build failed\n";

sub run {                                  # header line, verdict, warning?
    my (@args) = @_;
    my @out = `bin/simv @args 2>&1`;
    my ($head) = grep { /^TEST \S+ SEED/ } @out;
    my ($verd) = grep { /^TEST (?:PASSED|FAILED)/ } @out;
    my $warn   = grep { /^WARNING:/ } @out;
    chomp($head, $verd);
    return sprintf "%-27s %s%s", $head, $verd, $warn ? "   (and a WARNING)" : "";
}

printf "%-18s -> %s\n", $_, run("+test=rand", $_)
    for qw(+seed=2 +seed=4294967298 +seed=2147483647 +seed=2147483648
           +seed=-1 +seed=007 +seed= +seed=abc +seed=2x);
print "\n";
printf "%-18s -> %s\n", "@$_", run("+test=rand", @$_)
    for ["+seed=2", "+seed=1"], ["+seed=1", "+seed=2"], ["+Seed=2"];

+seed=2            -> TEST rand SEED 2            TEST FAILED (1 errors)
+seed=4294967298   -> TEST rand SEED 2            TEST FAILED (1 errors)
+seed=2147483647   -> TEST rand SEED 2147483647   TEST PASSED
+seed=2147483648   -> TEST rand SEED -2147483648  TEST PASSED
+seed=-1           -> TEST rand SEED -1           TEST PASSED
+seed=007          -> TEST rand SEED 7            TEST PASSED
+seed=             -> TEST rand SEED 0            TEST PASSED
+seed=abc          -> TEST rand SEED x            TEST PASSED   (and a WARNING)
+seed=2x           -> TEST rand SEED x            TEST PASSED   (and a WARNING)

+seed=2 +seed=1    -> TEST rand SEED 2            TEST FAILED (1 errors)
+seed=1 +seed=2    -> TEST rand SEED 1            TEST PASSED
+Seed=2            -> TEST rand SEED 1            TEST PASSED

Read the first block row by row. Each row is a way for a report to say one thing while the simulator did another:

The last row is the worst result in this module so far: a green row for a run that did no checking at all. It is also why Chapter 25.1 insisted on comparing the log's own header with what the runner asked for — here the header says SEED x, which matches nothing the runner asked for.

The fix is not in the bench. It is in the runner: choose seeds only from a range the bench can hold, and refuse anything else before launch. This chapter's expander uses 1 to 2³¹ − 1: positive, so there is no sign to get wrong, and small enough to fit in a signed 32-bit integer. The derived seeds of Section 3 already land in that range, because they are computed modulo 2³¹ − 2, plus 1.

These measurements are for this bench on Icarus 13.0. A bench that reads the seed into a 64-bit variable, or a simulator that seeds its random generator itself, has different edges. The method transfers: pass each edge value to your own simulator once, read what the log says it used, and write the range down in the runner.

6. One Plusarg Name, One Value

The second block of the output above shows two more ways to lose an argument.

A repeated plusarg keeps the first value. With +seed=2 +seed=1, the bench used seed 2; with +seed=1 +seed=2, it used seed 1. Icarus searches the plusargs from the start of the command line and stops at the first match. Now picture the usual way a runner adds a command-line override: it appends it to the end. If the test list already contains +n=60 and the user runs the regression with +n=10 to make it faster, the command line becomes ... +n=60 +n=10, and the override is silently ignored. Other simulators may resolve this differently, so do not rely on either order. Make sure each name appears once.

A misspelled name matches nothing. +Seed=2 is not +seed, so the bench used its default seed, 1, and passed. Chapter 25.1 measured the same thing from the naive runner's side.

Both are fixed the same way: build the plusargs as a hash keyed by name, apply the sources in order of priority so that the later one replaces the earlier, and then print each name exactly once. In this chapter's expander:

sub plusarg_name { my ($p) = @_; return $p =~ /^\+([^=]+)/ ? $1 : $p }

my (%pa, @names);
for my $p (@{ $t->{plusargs} }, @override) {   # list first, then overrides
    my $n = plusarg_name($p);
    push @names, $n unless exists $pa{$n};     # remember first-seen order
    $pa{$n} = $p;                              # a later source replaces it
}
my @args = ("+test=$t->{name}", "+seed=$seed", map { $pa{$_} } @names);

@names keeps the order in which names were first seen, so the command line is the same every time for the same input (Chapter 17.8 explains why that matters). And two names are reserved for the runner: +test and +seed. If they were allowed in the list or on the command line, there would be two sources for the one value that identifies a run. The expander rejects them and points the user at the --seed and --only options instead.

The hash cannot catch a misspelled name like +Seed, because it does not know which names the bench reads. Only the bench knows that. The defence is the header check from Chapter 25.1, plus one practical rule: a bench should print a warning for any plusarg it does not recognise. That is a change to the bench, so it belongs to whoever owns the bench — but it is worth asking for.

7. The Expander

Here is the whole step. It takes the five records from Chapter 25.2's reader (written into the script, so this chapter stands on its own) and prints the exact runs a regression would launch:

#!/usr/bin/perl
# expand.pl -- turn test records into the exact runs a regression launches.
# Usage: perl expand.pl [--base N] [--only TEST] [--seed N] [+plusarg ...]
#   --base N     base seed for this regression (default: today's date)
#   --only TEST  expand just this test
#   --seed N     exactly one run per selected test, with this seed
#   +plusarg     overrides the same plusarg from the list
use strict;
use warnings;
use Digest::MD5 qw(md5);
use POSIX qw(strftime);

my $SEED_MAX = 2**31 - 1;                 # the bench's seed is a 32-bit integer

# The records Chapter 25.2's reader produced for tests.list.
my @tests = (
    { name => "smoke",    opts => {},               plusargs => [] },
    { name => "corner",   opts => {},               plusargs => [] },
    { name => "rand",     opts => { count => 4 },   plusargs => ["+n=60"] },
    { name => "stall",    opts => { timeout => 5 }, plusargs => [] },
    { name => "rand_big", opts => { count => 2 },   plusargs => ["+n=4000", "+msg_tag=RB#1"] },
);

my ($base, $only, $pin, @override) = (strftime("%Y%m%d", localtime));
while (@ARGV) {
    my $arg = shift;
    if    ($arg eq "--base") { $base = shift // die "--base needs a value\n" }
    elsif ($arg eq "--only") { $only = shift // die "--only needs a test\n" }
    elsif ($arg eq "--seed") { $pin  = shift // die "--seed needs a value\n" }
    elsif ($arg =~ /^\+/)    { push @override, $arg }
    else                     { die "unknown argument '$arg'\n" }
}

my @errors;
push @errors, "base seed '$base' must be a whole number" unless $base =~ /^\d+$/;
for my $s (grep { defined } $pin) {
    push @errors, "seed '$s' must be a whole number from 1 to $SEED_MAX"
        unless $s =~ /^\d+$/ && $s >= 1 && $s <= $SEED_MAX;
}
for my $p (@override) {
    push @errors, "$p: the runner sets +seed and +test itself; use --seed or --only"
        if $p =~ /^\+(?:seed|test)(?:=|$)/;
}
if ($only && !grep { $_->{name} eq $only } @tests) {
    push @errors, "--only $only: no such test in the list";
}
if (@errors) { print "error: $_\n" for @errors; exit 2 }

# A seed that depends only on (base, test, index): adding or removing other
# tests never changes it, and the same base always gives the same seeds.
sub derive_seed {
    my ($b, $test, $i) = @_;
    return unpack("N", md5("$b/$test/$i")) % ($SEED_MAX - 1) + 1;   # 1 .. 2**31-1
}

# Merge plusargs by NAME so that each one appears exactly once; later wins.
sub plusarg_name { my ($p) = @_; return $p =~ /^\+([^=]+)/ ? $1 : $p }

my (@runs, %seen_id);
for my $t (grep { !$only || $_->{name} eq $only } @tests) {
    my $fixed = $pin // $t->{opts}{seed};
    my $count = defined $pin ? 1 : $t->{opts}{count} // 1;
    if (defined $fixed && $count > 1) {
        push @errors, "$t->{name}: seed= and count= together are ambiguous";
        next;
    }
    my (%pa, @names);
    for my $p (@{ $t->{plusargs} }, @override) {
        my $n = plusarg_name($p);
        push @names, $n unless exists $pa{$n};
        $pa{$n} = $p;
    }
    for my $i (1 .. $count) {
        my $seed = $fixed // derive_seed($base, $t->{name}, $i);
        my $id   = "$t->{name}_s$seed";
        if ($seen_id{$id}++) { push @errors, "$id: two runs with the same id"; next }
        push @runs, {
            id   => $id, test => $t->{name}, seed => $seed,
            args => [ "+test=$t->{name}", "+seed=$seed", map { $pa{$_} } @names ],
            timeout => $t->{opts}{timeout} // 600,
        };
    }
}
if (@errors) { print "error: $_\n" for @errors; exit 2 }

print "# base seed $base, ", scalar @runs, " run(s)\n";
printf "%-22s %-5s %s\n", $_->{id}, "$_->{timeout}s", "@{ $_->{args} }" for @runs;

The parts worth slowing down for:

• Everything is checked before anything is expanded. The base, the pinned seed, the overrides and the --only name are validated first, and all problems are reported together with exit status 2 — the same rule as Chapter 25.2's reader.
• derive_seed is scheme C. unpack("N", md5(...)) takes the first four bytes of the MD5 digest as an unsigned 32-bit number; % ($SEED_MAX - 1) + 1 maps it into 1 … 2³¹ − 1, the range Section 5 established. The $b here is a subroutine argument, not the $b that sort uses — inside this small function there is no sort, but in a longer one you would pick another name, because sort owns $a and $b.
• $pin // $t->{opts}{seed} — a seed from the command line beats a seed from the list. --seed also forces exactly one run, because the person debugging wants one run with that seed, not four copies of it.
• The run id includes the seed (rand_s1019263315). That makes ids unique without a counter, and it means the id alone is enough to rerun the run. The %seen_id check would catch two runs with the same id, for example a pinned seed in the list that happens to equal a derived one.
• The base is printed in the header. It is the one number that, together with the test names, regenerates every seed of this regression.
• The default base is today's date, so a bare perl expand.pl gives new seeds each day (Section 4). Every example below passes --base explicitly, so the output on this page does not change from one day to the next.

8. Using It

#!/usr/bin/perl
# expand_demo.pl -- six invocations of expand.pl and what each one prints.
use strict;
use warnings;

open STDERR, ">&", \*STDOUT or die;
$| = 1;
for my $args (
    [qw(--base 20261013)],
    [qw(--base 20261013 --only rand +n=10)],
    [qw(--base 20261013 --only rand --seed 2)],
    [qw(--base 20261013 +seed=5)],
    [qw(--base 20261013 --only rand --seed 4294967298)],
    [qw(--base 20261013 --only rand_bg)],
) {
    print "\$ perl expand.pl @$args\n";
    system($^X, "expand.pl", @$args);
    print "(exit ", $? >> 8, ")\n\n";
}

$ perl expand.pl --base 20261013
# base seed 20261013, 9 run(s)
smoke_s1733883798      600s  +test=smoke +seed=1733883798
corner_s1387083561     600s  +test=corner +seed=1387083561
rand_s1019263315       600s  +test=rand +seed=1019263315 +n=60
rand_s1609017581       600s  +test=rand +seed=1609017581 +n=60
rand_s774307888        600s  +test=rand +seed=774307888 +n=60
rand_s1217399705       600s  +test=rand +seed=1217399705 +n=60
stall_s1659991771      5s    +test=stall +seed=1659991771
rand_big_s1793184522   600s  +test=rand_big +seed=1793184522 +n=4000 +msg_tag=RB#1
rand_big_s1498847207   600s  +test=rand_big +seed=1498847207 +n=4000 +msg_tag=RB#1
(exit 0)

$ perl expand.pl --base 20261013 --only rand +n=10
# base seed 20261013, 4 run(s)
rand_s1019263315       600s  +test=rand +seed=1019263315 +n=10
rand_s1609017581       600s  +test=rand +seed=1609017581 +n=10
rand_s774307888        600s  +test=rand +seed=774307888 +n=10
rand_s1217399705       600s  +test=rand +seed=1217399705 +n=10
(exit 0)

$ perl expand.pl --base 20261013 --only rand --seed 2
# base seed 20261013, 1 run(s)
rand_s2                600s  +test=rand +seed=2 +n=60
(exit 0)

$ perl expand.pl --base 20261013 +seed=5
error: +seed=5: the runner sets +seed and +test itself; use --seed or --only
(exit 2)

$ perl expand.pl --base 20261013 --only rand --seed 4294967298
error: seed '4294967298' must be a whole number from 1 to 2147483647
(exit 2)

$ perl expand.pl --base 20261013 --only rand_bg
error: --only rand_bg: no such test in the list
(exit 2)

Six invocations, six behaviours:

1. The full plan. Nine runs: four of rand, two of rand_big, one each of the others. stall keeps its own 5-second timeout from the list; everything else gets the default. Every row has its seed in its id and in its arguments.
2. An override. +n=10 replaced the list's +n=60 instead of being appended after it. The seeds did not change, because --only selects tests and does not take part in the seed calculation.
3. The debug case. --seed 2 gives exactly one run with seed 2 — the command a person types after reading a failure in the report.
4. A reserved plusarg. +seed=5 is refused with a message that names the right option.
5. An out-of-range seed. 4294967298 is refused before launch. Section 5 showed that the simulator would have run it as seed 2.
6. A misspelled test. Refused, with the name echoed back.

9. Checking the Plan Against the Simulator

The plan is a promise: "this run used this seed". The last step is to check that the simulator kept it. Launch the four rand runs and compare each log's header with the plan:

#!/usr/bin/perl
# verify_plan.pl -- run the planned rand runs and check each log's header
# against the plan: did every run use the seed the plan says it used?
use strict;
use warnings;

system("bin/build") == 0 or die "build failed\n";
my @plan = grep { /^rand_s/ } `$^X expand.pl --base 20261013 --only rand`;
my ($ok, $n) = (0, 0);
for my $row (@plan) {
    my ($id, $timeout, @args) = split ' ', $row;
    my ($seed) = map { /^\+seed=(\d+)$/ ? $1 : () } @args;
    my @log     = `bin/simv @args`;
    my ($head)    = grep { /^TEST \S+ SEED/ } @log;
    my ($verdict) = grep { /^TEST (?:PASSED|FAILED)/ } @log;
    chomp($head, $verdict);
    my $match = $head eq "TEST rand SEED $seed";
    printf "%-18s %-30s %-6s %s\n", $id, $head, ($match ? "match" : "MISMATCH"), $verdict;
    $n++; $ok++ if $match;
}
print "$ok of $n runs used the planned seed\n";

rand_s1019263315   TEST rand SEED 1019263315      match  TEST PASSED
rand_s1609017581   TEST rand SEED 1609017581      match  TEST PASSED
rand_s774307888    TEST rand SEED 774307888       match  TEST PASSED
rand_s1217399705   TEST rand SEED 1217399705      match  TEST PASSED
4 of 4 runs used the planned seed

All four runs used their planned seed, and all four passed: with 60 vectors each, none of these four seeds happened to produce a sum of exactly 256. That is a normal result for a seed-dependent bug, and it is why rand runs four seeds, not one.

This script builds its commands as strings for backticks. That is safe here only because every argument is a simple word with no spaces or shell characters. Chapter 25.4 shows what happens when that is not true, and launches runs in a way that does not depend on it.

10. Other Simulators' Seed Options

This module's bench reads its seed from +seed=N, because that is how the bench was written. Commercial simulators seed SystemVerilog's random generator through their own options. The commonly documented spellings are:

These have not been run on this machine. Confirm them in your own tool's manual, and run Section 5's edge test against your simulator before trusting any range. The design point holds whatever the spelling: the seed option belongs in one table in the runner, not scattered through the code:

my %SEED_ARG = (
    fixture => sub { ("+seed=$_[0]") },          # this module's bench
    vcs     => sub { ("+ntb_random_seed=$_[0]") },
    xcelium => sub { ("-svseed", $_[0]) },        # two words: option and value
    questa  => sub { ("-sv_seed", $_[0]) },
);
my @seed_args = $SEED_ARG{$sim}->($seed);

Each entry returns a list, because some options are one word and others are two. Returning a list keeps that difference inside the table, and the launcher in Chapter 25.4 receives a ready-made list of arguments.

11. Exercises

1. A seed scheme for your team. Some teams need to run "the same seeds as last Tuesday" with a design change applied. Using scheme C, what do they need to know about last Tuesday's regression, and where should the runner have written it? Write the command they would type.

2. The untested branch. The expander's message "seed= and count= together are ambiguous" is not reached by any example on this page. Add a record to @tests that reaches it, check the message, and decide whether it should be an error or whether seed=N count=4 should mean something useful.

3. Find the range of your own bench. Run seed_edges.pl against a bench you work with (change the plusarg name if yours differs). Which values wrap, which go negative, and does any value produce a PASS with an unknown seed? Write the safe range into a comment in your runner.

4. Names the bench knows. rand_big is in the list, but the fixture's bench has no test with that name: running it gives a $fatal, which Chapter 25.1's runner reports as FATAL. Add a check to the expander that compares every test name against the names the bench supports and refuses unknown ones before launch. Where should the list of supported names come from, so that it cannot drift from the bench?

5. Override precedence. Design the full precedence order for a real runner with four sources of plusargs: the bench defaults, the test list, a per-team configuration file, and the command line. Which should win, and how would the report show which source each final plusarg came from?

12. What To Remember

• Expansion turns each test into explicit runs: id, test, seed, arguments, timeout — all decided before launch.
• The seed is the run's identity. The runner chooses every seed and writes it in the report.
• Measured: inserting one test moved all four rand seeds under a global counter. A derived seed (hash of base, test, index) moved none and shared none.
• Measured: fixed seeds found the same two failures three nights running. Date-based seeds found five — all one bug — and one night found nothing. A night with no failures is not a fix.
• Record the base. With the base and the test names, every seed can be recomputed.
• Measured on this bench: 4294967298 ran as seed 2, and abc ran as seed x and passed having tested nothing. Choose seeds from 1 to 2³¹ − 1 and refuse the rest before launch.
• Measured: a repeated plusarg kept the first value. An appended override was ignored. Merge plusargs by name so each appears once.
• +test and +seed belong to the runner. Nothing else may set them.
• Keep simulator-specific seed options in one table that returns a list of arguments.

Chapter 25.4 launches these runs: each one in its own directory, with its output captured to a log, its exit status checked, and its exact command line saved so that anyone can run it again.