What a Regression Runner Has to Do

Part II · Module 25 · Chapter 25.1 · What a Regression Runner Has to Do

1. The Problem a Runner Solves

A verification team does not run one test. It runs every test in a list, many of them with several random seeds, every night and before every merge. That set of runs is a regression. The script that turns "this list of tests" into "this verdict, with this evidence" is the regression runner.

Without a runner, someone types commands and reads logs by hand. That stops working at about ten runs. With a bad runner, the situation is worse: the team trusts a summary line that is wrong, and nobody reads the logs any more because the summary says there is nothing to read.

So the question for this chapter is not "how do I loop over tests in Perl". You know how to loop. The question is: what must a runner guarantee, so that its summary can be trusted? We answer that first, as a list. Every later chapter in this module then implements one item of the list properly.

2. The Fixture Used Through This Module

Every runner in Module 25 drives a real simulator: Icarus Verilog 13.0, the compiler iverilog and the runtime vvp. The design and bench below are a small educational fixture written for this module. They are not from a real project, and nothing in this module is a claim about how VCS, Xcelium or Questa behave. The point is that every output on these pages comes from a real simulation, not from text typed to look like one.

The design is an 8-bit saturating adder: y is a + b, but never more than 255. It has a deliberate bug, so the regression has something to find.

// rtl/sat_add.v -- 8-bit saturating adder: y = min(a + b, 255).
// It carries a deliberate bug, so that a regression has something to find.
module sat_add (
    input  wire [7:0] a,
    input  wire [7:0] b,
    output wire [7:0] y
);
    wire [8:0] sum = a + b;
    // BUG: saturates only when sum > 256. A sum of exactly 256 wraps to 0.
    assign y = (sum > 9'd256) ? 8'hFF : sum[7:0];
endmodule

The comparison should be sum > 255. Because it says 256, a sum of exactly 256 gives y = 0 instead of 255. Every other input is correct. That makes the bug a good model of real ones: most tests pass, and only some inputs reach it.

The bench holds several tests. The runner chooses one with the plusarg +test=NAME, and passes a seed with +seed=N:

// tb/tb.v -- one self-checking bench, several tests, chosen by plusarg:
//   +test=NAME   smoke | rand | corner | stall      (anything else: $fatal)
//   +seed=N      seed for the random test (default 1)
//   +n=N         number of random vectors (default 40)
module tb;
    reg  [7:0] a, b;
    wire [7:0] y;
    reg  [8*16-1:0] test;
    integer seed, n, i, errors;
    reg clk;

    sat_add dut (.a(a), .b(b), .y(y));

    task check;
        reg [8:0] s;
        reg [7:0] want;
        begin
            #1;
            s    = a + b;
            want = s[8] ? 8'hFF : s[7:0];
            if (y !== want) begin
                errors = errors + 1;
                $display("ERROR: a=%0d b=%0d y=%0d expected=%0d", a, b, y, want);
            end
        end
    endtask

    initial begin
        errors = 0;
        if (!$value$plusargs("test=%s", test)) test = "smoke";
        if (!$value$plusargs("seed=%d", seed)) seed = 1;
        if (!$value$plusargs("n=%d", n))       n    = 40;
        $display("TEST %0s SEED %0d", test, seed);

        if (test == "smoke") begin
            a = 1;   b = 2;   check;
            a = 100; b = 100; check;
            a = 250; b = 10;  check;
        end else if (test == "rand") begin
            for (i = 0; i < n; i = i + 1) begin
                a = $random(seed);
                b = $random(seed);
                check;
            end
        end else if (test == "corner") begin
            a = 0;   b = 0;   check;
            a = 255; b = 255; check;
            a = 128; b = 128; check;
        end else if (test == "stall") begin
            // waits for a handshake that never arrives: the clock keeps
            // simulated time moving, so the run never ends on its own
            clk = 0;
            forever #5 clk = ~clk;
        end else begin
            $fatal(1, "unknown test '%0s'", test);
        end

        if (errors == 0) $display("TEST PASSED");
        else             $display("TEST FAILED (%0d errors)", errors);
        $finish;
    end
endmodule

The four tests give a runner one of each outcome it must handle:

Two small shell scripts complete the fixture. Like a commercial flow, it compiles once and then runs many times with different plusargs:

#!/bin/sh
# bin/build -- compile the bench once; every run reuses build/sim.vvp.
cd "$(dirname "$0")/.." || exit 2
mkdir -p build
exec iverilog -o build/sim.vvp rtl/sat_add.v tb/tb.v

#!/bin/sh
# bin/simv -- run the compiled bench. All arguments are plusargs.
here=$(cd "$(dirname "$0")/.." && pwd)
exec vvp -n "$here/build/sim.vvp" "$@"

The name simv is borrowed from the executable VCS produces, because that is the shape of a real flow: build an executable once, then launch it per test and per seed. Running it by hand looks like this:

$ bin/build
$ bin/simv +test=corner
TEST corner SEED 1
ERROR: a=128 b=128 y=0 expected=255
TEST FAILED (1 errors)
tb/tb.v:60: $finish called at 3 (1s)
$ echo $?
0

Look at the last line. The simulator printed TEST FAILED and exited 0. Chapter 26.7 measured this in detail for Icarus: vvp exits 0 after a printed failure, after $error, and even after being sent SIGTERM. Keep that fact in mind. It decides what the rest of this chapter has to do.

3. Six Obligations

Here is the specification. A runner must do six things, in this order:

Each obligation answers one question that a user of the runner will ask. Each one also has a typical way of being skipped, and the skip produces a specific wrong answer:

The order matters. A stage can only work on what the earlier stage gave it. If bound is missing, the runner may never reach collect. If collect only looks at logs that exist, report cannot mention the runs that wrote nothing. That is why the figure is a chain, not a checklist.

4. The Naive Runner, Measured

Most runners start as something like this. It is short, it reads well, and it is wrong in ways that are hard to see from the code. Its default list is smoke, corner, rand with seed 2, and rand with seed 2 again, except that the last one was typed as +Seed=2 with a capital S. That is the kind of typo that sits in a test list for months.

#!/usr/bin/perl
# naive_run.pl -- the runner most flows start with. Do not copy it.
use strict;
use warnings;

system("bin/build") == 0 or die "build failed\n";

my @tests = @ARGV ? @ARGV : ("smoke", "corner", "rand +seed=2", "rand +Seed=2");
my ($pass, $fail) = (0, 0);
for my $t (@tests) {
    my $rc = system("bin/simv +test=$t > /dev/null 2>&1");
    if ($rc == 0) { $pass++ } else { $fail++; print "FAIL: $t\n" }
}
print "$pass passed, $fail failed\n";
exit($fail ? 1 : 0);

4 passed, 0 failed

And here is the truth, from running the same four commands by hand and reading the first lines of each log:

Three of the four answers are wrong, and each one is wrong for a different reason:

1. It trusted the exit status. vvp exits 0 after TEST FAILED, so corner and rand +seed=2 both count as passes. This breaks collect: the verdict must come from evidence, and for this tool the exit status is not evidence of success.
2. It never checked that the run was the one it asked for. Plusarg names are case-sensitive. +Seed=2 matches nothing, so the bench used its default seed, 1, and seed 1 happens to pass. The runner reported a pass for "seed 2" when seed 2 was never simulated. This breaks launch and collect together: nothing compared what was asked with what ran.
3. It threw the output away (> /dev/null). Even a person who suspected the summary has nothing to read. This breaks launch.

And then it exited 0, so any Makefile or CI job above it would carry on as if the design were clean. One summary line, wrong four ways, and every stage after the first one failed to catch the stage before it.

5. The Naive Runner and a Test That Never Ends

The naive runner has one more problem that the list above did not show, because none of those four tests hangs. Give it stall, and watch from outside for three seconds:

#!/usr/bin/perl
# hang_demo.pl -- give naive_run.pl a test that never ends, watch for
# 3 seconds, then stop it and look for anything it left behind.
use strict;
use warnings;
use POSIX qw(WNOHANG);

my $pid = fork // die "fork: $!\n";
if ($pid == 0) {
    setpgrp(0, 0);                    # own process group, so we can find
    exec $^X, "naive_run.pl", "smoke", "stall";   # everything it starts
    die "exec: $!\n";
}

for my $s (1 .. 3) {
    sleep 1;
    if (waitpid($pid, WNOHANG) == $pid) { print "runner finished by itself\n"; exit }
    print "after ${s}s: runner still running\n";
}

kill "TERM", $pid;                    # stop the runner itself ...
waitpid($pid, 0);
sleep 1;
my $left = kill 0, -$pid;             # ... is anything left in its group?
print "runner stopped; processes still in its group: ", ($left ? "yes" : "no"), "\n";
for (`ps -A -o pgid= -o comm=`) {          # name what is left
    my ($g, $cmd) = /^\s*(\d+)\s+(.*)$/ or next;
    print "  left running: $cmd\n" if $g == $pid;
}
kill "KILL", -$pid if $left;          # clean up what the runner abandoned

after 1s: runner still running
after 2s: runner still running
after 3s: runner still running
runner stopped; processes still in its group: yes
  left running: sh
  left running: vvp

Two things happened, and the second one is the more expensive:

• The runner had no limit. It would have waited for stall forever. In a nightly regression, that means the morning report never appears, and nobody can tell which of the thousand tests is the one holding it open. This is the missing bound obligation.
• Stopping the runner did not stop the simulation. After the runner died, its shell and vvp were still running, now with no parent watching them. On a laptop that wastes a CPU core. On a compute farm, where each job holds a licence and a slot, an abandoned simulation keeps holding both until someone notices. A runner that bounds its runs must also clean up what it started — and the demonstration only found the leftovers because it had put the runner in its own process group first. Chapter 25.6 builds on exactly that.

6. A Minimal Runner That Meets All Six

Now the same job, written against the specification. It is deliberately small: one file, runs in sequence, no farm. Later chapters replace each stage with a stronger version. What matters here is that every obligation has a line of code you can point at.

#!/usr/bin/perl
# mini_run.pl -- the smallest runner that meets all six obligations.
# Usage: perl mini_run.pl [TEST[:SEED] ...]
use strict;
use warnings;
use POSIX qw(WNOHANG);
use File::Path qw(make_path remove_tree);
use Time::HiRes qw(time);                          # fractional seconds

my $TIMEOUT = 2;                                   # seconds per run

# 1. SELECT: turn the request into a list of runs, and refuse bad ones now.
my @want = @ARGV ? @ARGV : qw(smoke corner rand:2 rand:3 rand:4294967298 stall bogus);
my @runs;
for my $w (@want) {
    my ($test, $seed) = split /:/, $w;
    $seed //= 1;
    die "bad seed '$seed' in '$w'\n" unless $seed =~ /^\d+$/;
    push @runs, { test => $test, seed => $seed, id => "${test}_s$seed" };
}

system("bin/build") == 0 or do { print "build failed\n"; exit 2 };
remove_tree("runs");

for my $r (@runs) {
    my $dir = "runs/$r->{id}";
    make_path($dir);
    my @cmd = ("bin/simv", "+test=$r->{test}", "+seed=$r->{seed}");

    # 2. LAUNCH: own process, output captured to this run's log.
    my $pid = fork // die "fork: $!\n";
    if ($pid == 0) {
        open STDOUT, ">", "$dir/sim.log" or exit 126;
        open STDERR, ">&", \*STDOUT      or exit 126;
        exec { $cmd[0] } @cmd;
        exit 127;
    }

    # 3. BOUND: never wait longer than $TIMEOUT, and record that we stopped it.
    my $deadline = time + $TIMEOUT;
    $r->{timed_out} = 0;
    until (waitpid($pid, WNOHANG) == $pid) {
        if (time >= $deadline && !$r->{timed_out}) {
            $r->{timed_out} = 1;
            kill "TERM", $pid;
        }
        select(undef, undef, undef, 0.05);         # sleep 50 ms
    }

    # 4. COLLECT: a verdict for this run, from evidence, with a reason.
    @$r{qw(verdict why)} = classify("$dir/sim.log", $r);
}

# 5. REPORT: one row per launched run, in launch order, plus totals.
my %n;
printf "%-8s %10s  %-8s %s\n", "TEST", "SEED", "VERDICT", "REASON";
for my $r (@runs) {
    printf "%-8s %10s  %-8s %s\n", @$r{qw(test seed verdict why)};
    $n{ $r->{verdict} }++;
}
print "\n", join("  ", map { "$_=$n{$_}" } sort keys %n), "  (", scalar @runs, " runs)\n";

# 6. EXIT HONESTLY: 0 only if every run passed.
exit(($n{PASS} // 0) == @runs ? 0 : 1);

sub classify {
    my ($log, $r) = @_;
    return ("TIMEOUT", "stopped by the runner after ${TIMEOUT}s") if $r->{timed_out};
    open my $fh, "<", $log or return ("NO_LOG", "no log written");
    my ($header, $passed, $failed, $fatal, $errors) = ("", 0, 0, 0, 0);
    while (my $l = <$fh>) {
        $header = $1 if !$header && $l =~ /^(TEST \S+ SEED \S+)$/;
        $passed = 1  if $l =~ /^TEST PASSED$/;
        $failed = 1  if $l =~ /^TEST FAILED/;
        $fatal  = 1  if $l =~ /^FATAL:/;
        $errors++    if $l =~ /^ERROR:/;
    }
    my $asked = "TEST $r->{test} SEED $r->{seed}";
    return ("FATAL",   "the bench stopped itself")            if $fatal;
    return ("WRONG",   "log says '$header', asked '$asked'")  if $header ne $asked;
    return ("FAIL",    "$errors error line(s)")               if $failed || $errors;
    return ("PASS",    "TEST PASSED, 0 errors")               if $passed;
    return ("UNKNOWN", "log has no verdict line");
}

TEST           SEED  VERDICT  REASON
smoke             1  PASS     TEST PASSED, 0 errors
corner            1  FAIL     1 error line(s)
rand              2  FAIL     1 error line(s)
rand              3  PASS     TEST PASSED, 0 errors
rand     4294967298  WRONG    log says 'TEST rand SEED 2', asked 'TEST rand SEED 4294967298'
stall             1  TIMEOUT  stopped by the runner after 2s
bogus             1  FATAL    the bench stopped itself

FAIL=2  FATAL=1  PASS=2  TIMEOUT=1  WRONG=1  (7 runs)

The runner exited 1, which is the honest answer. Walk through the stages once, with the output in front of you.

Select. The request rand:2 becomes a record { test => "rand", seed => 2, id => "rand_s2" }. The check $seed =~ /^\d+$/ refuses anything that is not a whole number before any simulation starts. A runner should fail on a bad request in the first second, not after an hour of simulations. (This check is not strong enough, and the fifth row shows why. Hold that thought.)

Launch. fork makes a child; the child points its STDOUT and STDERR at runs/<id>/sim.log and then execs the simulator. The list form of exec passes each argument separately, so no shell splits or reinterprets anything (Chapter 15.10 explains why that matters). Each run has its own directory, so logs never mix.

Bound. The parent polls with waitpid(..., WNOHANG), which returns at once whether or not the child has finished. If the deadline passes, the runner sends TERM and — this is the important part — records that it did so in $r->{timed_out}. It has to keep its own record, because vvp exits 0 when it receives TERM; the exit status will never say "timeout". The stall row is reported as TIMEOUT only because of that one flag.

Collect. classify reads the log and returns a verdict and a reason. The reason is what makes a report useful at 9am: nobody has to open a log to learn why a row is red. The order of the checks is the verdict ladder from Chapter 21.8: a run the runner stopped is a TIMEOUT whatever its log says; a bench that stopped itself is FATAL; and so on down to UNKNOWN for a log with no verdict line at all.

Report. One row per launched run, in launch order, with the seed on every row. The totals line counts verdicts, and the number of rows always equals the number of runs.

Exit. Zero only if every run passed. Otherwise 1. Chapter 15.4 explains how to choose these numbers; the build failure exits 2, a separate status for "could not even start".

The row that only the header check caught

Look at the fifth row. The request was rand with seed 4294967298. That passed the select check — it is a string of digits. But the log says it ran with seed 2. Seeds in this bench are 32-bit Verilog integers, and 4294967298 is 2^32 + 2, so it wraps to 2. Chapter 25.3 measures this boundary properly. Here, notice which check caught it: not the select stage, which only checked the shape of the seed, but the collect stage, which compared the run's own first line with what the runner asked for.

That comparison is cheap, and it is the only defence against the whole family of "the run was not the one you asked for" problems: a misspelled plusarg, a seed out of range, a test name the bench truncated, a wrapper script that dropped an argument. It works here because the bench prints what it understood. If your bench does not echo the test name and the seed it actually used, add that line before you write the runner. It is the most useful line a testbench can print.

7. Proving the Runner Itself

A runner is a program that decides what other programs did. It needs testing like any other program, and the test is not "it printed a nice table". The test is: give it one run of each kind whose real outcome you already know, and check that it reports each one correctly.

That is why the fixture has a test for each outcome, and why mini_run.pl's default list contains one of each. The table above shows each one classified as expected: two passes, the always-failing corner, the failing seed of rand, the wrapped seed, the hang and the fatal.

Two branches of classify were not reached by any of these seven runs: NO_LOG (the log file could not be opened) and UNKNOWN (a log with no verdict line). A branch that has never run has not been tested, however obvious it looks. Exercise 2 asks you to reach both. Chapter 25.7 returns to runs with no log, because on a compute farm they are common, not exotic.

Keep the known-outcome list with the runner, and run it whenever the runner changes. A change that turns the corner row green has broken the runner, not fixed the design.

What this runner still does not do

mini_run.pl is a specification you can execute, not a production tool. Be clear about its limits:

8. Exercises

1. Break the runner on purpose. Change one line of classify so that it trusts the exit status (hint: classify does not currently receive it — pass $? in). Run the default list. Which rows change, and does the exit status of the runner still tell the truth?

2. Reach the two untested branches. Make one run produce no log at all (for example, make the log directory unwritable before launch, or point @cmd at a program that does not exist and look carefully at what the child does), and make another produce a log with no verdict line. Check that classify gives NO_LOG and UNKNOWN. If either verdict is wrong, fix the runner, not the test.

3. A second wrong-run case. The bench stores the test name in reg [8*16-1:0] test, which holds 16 characters. Predict what happens if the runner asks for a test whose name is 20 characters long. Then try it. Which stage of the runner should have caught it, and which stage actually did?

4. Write the specification for your own flow. Take a runner you use at work, or one you have seen. For each of the six obligations, write one sentence saying how that runner meets it, or write "does not". For each "does not", write down the wrong answer it can produce.

5. Exit codes for a regression. mini_run.pl exits 1 for any non-pass. Design a scheme that also lets a CI job tell "a test failed" from "the regression did not complete" (some runs timed out or produced no verdict). Justify each number using the conventions in Chapter 15.4.

9. What To Remember

• A regression runner is a program that decides what other programs did. Write its specification before its code.
• Six obligations, in order: select, launch, bound, collect, report, exit honestly. Each stage can only work on what the stage before it produced.
• Measured: a typical first runner reported 4 passed, 0 failed for a list where two runs failed and a third ran the wrong seed — and exited 0.
• The exit status is not a verdict. Icarus vvp exits 0 after printing TEST FAILED. The verdict must come from the log.
• Check that each run is the run you asked for. A misspelled plusarg ran the default seed and passed. Make the bench print the test name and seed it actually used, and compare.
• Every run needs a time limit, and the runner must record when it used it. The simulator's own exit status will not say "timeout".
• Stopping a runner does not stop what it started. Measured: the shell and vvp kept running after the runner was killed.
• Report one row per launched run, with its seed and a reason.
• Test the runner with runs whose outcome you already know — and say which branches have never been reached.

Chapter 25.2 starts at the top of the chain, with select: reading the list of tests from a file that people edit, with comments, per-test options and include files, without letting a stray character turn into a silent change in what the regression runs.