Running an EDA Tool and Reading Its Exit Status

Part II · Module 26 · Chapter 26.7 · Running an EDA Tool and Reading Its Exit Status

1. What $? Can Say

Chapter 14.3 decoded $? completely, and this chapter does not repeat it. In one paragraph: after system or waitpid, the high byte ($? >> 8) is the exit code the program chose; the low seven bits ($? & 127) are the number of the signal that killed it, if one did; bit 7 ($? & 128) says whether a core file was written; and system returning -1 means no program was started at all, with the reason in $!.

That is everything the operating system can tell a parent about how a child ended. The question this chapter answers is different: when the child is an EDA tool, what do those numbers actually mean about the design? The canonical example for the chapter ends with the right warning:

system(@cmd);
my ($code, $sig) = ($? >> 8, $? & 127);
die "killed by signal $sig\n" if $sig;
# exit 0 is NOT proof of success -- scan the log as well

Everything below measures why that last comment is true, using the one real EDA tool available on the machine these examples were run on: Icarus Verilog 13.0, a compiler (iverilog) and a simulator (vvp). Every number is that tool's real behaviour. None of it is a claim about any commercial simulator, whose behaviour belongs to its vendor — the point of the chapter is to show how to measure your own tool, not to replace that measurement.

2. A Runner With a Log, a Limit, and a Record

Running a tool from a flow needs three things that a bare system does not give: the tool's output captured to a log file (it is evidence, and Section 7 reads it), a time limit (a hung simulation must not hang the regression), and an honest record of what the runner itself did — because, as Section 4 will show, if the runner stops a tool, the tool's own exit status may not say so.

#!/usr/bin/perl
# run_tool.pl -- launch one tool command with its output captured to a log,
# a wall-clock limit, and a complete description of how it ended.
#
# This file is a library of one subroutine, run_tool(), which the drivers on
# this page load with `require`.
use strict;
use warnings;
use POSIX qw(WNOHANG);
use Time::HiRes qw(time sleep);

# run_tool(log => FILE, timeout => SECONDS, cmd => [PROGRAM, ARGS...])
# Returns a hash reference describing what happened -- never dies.
sub run_tool {
    my %o = @_;
    my %r = (launched => 0, exit => undef, signal => 0, core => 0, timed_out => 0);

    my $pid = fork;
    if (!defined $pid) { $r{error} = "fork failed: $!"; return \%r }
    if ($pid == 0) {
        open STDOUT, ">", $o{log} or exit 126;
        open STDERR, ">&", \*STDOUT  or exit 126;
        { exec { $o{cmd}[0] } @{ $o{cmd} } }      # block form: see perldoc -f exec
        print STDERR "run_tool: cannot exec $o{cmd}[0]: $!\n";
        exit 127;                         # the shell's convention for "not found"
    }

    my $deadline = time + $o{timeout};
    while (1) {
        my $done = waitpid($pid, WNOHANG);
        last if $done == $pid;
        if (!$r{timed_out} && time >= $deadline) {
            $r{timed_out} = 1;            # OUR record that we stopped it
            kill "TERM", $pid;
            $deadline = time + 2;         # grace period, then insist
        } elsif ($r{timed_out} && time >= $deadline) {
            kill "KILL", $pid;
        }
        sleep 0.05;
    }

    $r{launched} = 1;
    $r{signal}   = $? & 127;
    $r{core}     = ($? & 128) ? 1 : 0;
    $r{exit}     = $r{signal} ? undef : $? >> 8;
    $r{launched} = 0 if defined $r{exit} && $r{exit} == 127 && log_says($o{log}, qr/^run_tool: cannot exec/);
    return \%r;
}

sub log_says {
    my ($log, $re) = @_;
    open my $fh, "<", $log or return 0;
    while (<$fh>) { return 1 if $_ =~ $re }
    return 0;
}

sub describe {
    my ($r) = @_;
    return "not launched"                          unless $r->{launched};
    return "killed by signal $r->{signal}"         if $r->{signal};
    return "exit $r->{exit}";
}

1;

Four details carry the design:

fork and exec instead of system, because the parent has to keep control while the child runs — to watch the clock. Chapter 15.10 introduced exec; here it is used the classic way, in the child of a fork. The block form { exec ... } tells Perl that the statement after it is deliberate: it runs only if exec failed.

The child redirects its own output to the log before exec, so the tool writes straight to the file and the parent never has to buffer it.

waitpid with WNOHANG checks whether the child has finished without blocking, so the loop can also check the time. At the limit the runner sends TERM, waits a short grace period, and then sends KILL, which cannot be caught.

timed_out is the runner's own record. It is set by the code that sent the signal, not inferred from $?. That distinction is the whole of Section 4.

A launch failure needs care with fork: if exec fails, the failure happens in the child, and the parent only sees the child exit. The runner therefore writes a recognisable line into the log and exits 127 — the shell's convention for "command not found" — and the parent reports "not launched" only when it sees both.

3. Seven Testbenches, One Real Simulator

Each testbench below ends in a way a real regression produces every night. They are deliberately tiny, so that the only thing being measured is how the run ends.

module tb;
  initial begin
    $display("TEST PASSED");
    $finish;
  end
endmodule

module tb;
  initial begin
    $display("ERROR: data mismatch exp=5 got=4");
    $display("TEST FAILED");
    $finish;
  end
endmodule

module tb;
  initial begin
    $error("data mismatch exp=5 got=4");
    $finish;
  end
endmodule

module tb;
  initial begin
    $display("TEST FAILED");
    $finish(1);
  end
endmodule

module tb;
  initial begin
    $fatal(1, "scoreboard overflow");
  end
endmodule

module tb;
  reg clk = 0;
  always #5 clk = ~clk;      // no $finish anywhere: runs until stopped
  initial $display("waiting for a handshake that never comes");
endmodule

module tb;
  initial begin
    $display("missing semicolon")
  end
endmodule

The census compiles and runs each one through run_tool, and prints only what the exit statuses say, next to what a person reading the log would conclude:

use strict;
use warnings;
require "./run_tool.pl";

# Compile and run every testbench, and record ONLY what the exit status says.
# The last column is what a person reading the log would conclude.
my %TRUTH = (
    pass     => "passed",
    mismatch => "FAILED (checker said so)",
    sv_error => q{FAILED ($error)},
    finish_1 => q{FAILED (then $finish(1))},
    sv_fatal => q{FAILED ($fatal)},
    hang     => "never finished",
    syntax   => "did not compile",
);

printf "%-9s %-17s %-22s %s\n", "test", "iverilog", "vvp", "the truth";
for my $t (qw(pass mismatch sv_error finish_1 sv_fatal hang syntax)) {
    my $c = run_tool(log => "$t.compile.log", timeout => 30,
                     cmd => ["iverilog", "-g2012", "-o", "$t.vvp", "tb/$t.v"]);
    my $sim = "(not run)";
    if (defined $c->{exit} && $c->{exit} == 0) {
        my $s = run_tool(log => "$t.sim.log", timeout => 2,
                         cmd => ["vvp", "-n", "$t.vvp"]);
        $sim = describe($s) . ($s->{timed_out} ? " (we killed it)" : "");
    }
    printf "%-9s %-17s %-22s %s\n", $t, describe($c), $sim, $TRUTH{$t};
}

test      iverilog          vvp                    the truth
pass      exit 0            exit 0                 passed
mismatch  exit 0            exit 0                 FAILED (checker said so)
sv_error  exit 0            exit 0                 FAILED ($error)
finish_1  exit 0            exit 0                 FAILED (then $finish(1))
sv_fatal  exit 0            exit 1                 FAILED ($fatal)
hang      exit 0            exit 0 (we killed it)  never finished
syntax    exit 2            (not run)              did not compile

Six runs did not pass. The simulator exited 0 for four of them.

mismatch — the testbench's own checker printed an error and TEST FAILED, then called $finish normally. From the simulator's point of view, the simulation ran to completion exactly as written. Printing the word "FAILED" is something the testbench did; the simulator has no idea it means anything. This is the most common shape in real testbenches, and it is why every regression system in the industry scans logs.

sv_error — $error is SystemVerilog's built-in error-severity report. Icarus printed it with the file, line and scope, and the simulation continued to the $finish after it, as $error is meant to. The language standard defines what these severity tasks print and whether simulation continues; the process exit status is the simulator's own choice, and this one chose 0.

finish_1 — $finish(1) looks like "finish with status 1". It is not. The argument to $finish is a diagnostic level that controls how much the simulator prints about the run — the log shows $finish(1) called at 0. It has nothing to do with the exit status, which was 0.

sv_fatal — $fatal is the only one of the four reporting tasks that ends the simulation, and the only one that changed the exit status: 1.

hang — the runner stopped it at its two-second limit with TERM. The simulator then exited 0. Section 4 measures that separately, because it is the least expected row.

syntax — the compiler refused the source and exited 2, so the simulation never ran. Section 5 shows that this number is not a simple yes/no either.

4. Signals: The Queue's TERM Disappears

Job queues stop jobs with signals: usually TERM at a time or memory limit, often followed by KILL if the job does not leave. Chapter 26.1 showed a shell wrapper turning a signal death into exit code 143. This is a different and more surprising effect — the tool itself making the signal vanish.

use strict;
use warnings;
use Time::HiRes qw(sleep);

# Stop a running simulation with three different signals and decode $? each time.
system("iverilog", "-g2012", "-o", "hang.vvp", "tb/hang.v") == 0 or die "compile failed\n";

for my $sig (qw(TERM INT KILL)) {
    my $pid = fork // die "fork: $!\n";
    if ($pid == 0) {
        open STDOUT, ">", "/dev/null" or die;
        open STDERR, ">", "/dev/null" or die;
        exec "vvp", "-n", "hang.vvp" or exit 127;
    }
    sleep 0.5;                  # let it start simulating
    kill $sig, $pid;
    waitpid $pid, 0;
    printf "sent %-4s  raw \$? %-5d  exit code %-3d  signal %-2d  core %d\n",
        $sig, $?, $? >> 8, $? & 127, ($? & 128) ? 1 : 0;
}

sent TERM  raw $? 0      exit code 0    signal 0   core 0
sent INT   raw $? 0      exit code 0    signal 0   core 0
sent KILL  raw $? 9      exit code 0    signal 9   core 0

TERM and INT both produced a raw $? of 0 — no signal, exit code 0, indistinguishable from a clean run. vvp handles those signals itself: it catches them, stops the simulation, and exits normally. Only KILL, which no program can catch, appears as signal 9.

Two conclusions follow, and both apply far beyond Icarus:

A signal in $? means the tool did not handle it. Tools that catch signals — to flush logs, release licences, or print a summary, all good reasons — report the result as an ordinary exit. Whether a given tool exits 0, 1 or something else after catching a signal is that tool's choice.

So the evidence that a run was stopped must come from whoever stopped it. That is why run_tool records timed_out itself. When a queue does the stopping, the evidence is in the queue's accounting — Chapter 21.2's argument about outside evidence, now with a measured reason. The log helps too: the hang log has no $finish line, which is the absence Chapter 21.2 taught you to look for.

The core column was 0 in every row, including when a program dies from a crash signal. Whether a core file is written depends on the process's core-size limit, and on this machine that limit is 0 (ulimit -c prints 0), so no core is written and bit 7 is never set. On a machine with core dumps enabled, the same crash would set it. Treat bit 7 as "a core file exists for you to look at", not as "the program crashed" — the signal number is the crash.

5. The Compiler's Status Is an Error Count

The compiler's exit status for the one-error syntax testbench was 2. That number is not a fixed "failure" code. The script below generates sources with a chosen number of broken modules — each one missing a semicolon — and records the compiler's exit status, how many errors its log reports, and whether it wrote an output file.

use strict;
use warnings;

# A source file with N broken modules: each one is missing a semicolon.
sub broken_source {
    my ($file, $n) = @_;
    open my $fh, ">", $file or die "$file: $!\n";
    print {$fh} "module m$_; initial begin \$display(1) end endmodule\n" for 1 .. $n;
    close $fh;
}

print "broken  iverilog  syntax errors  output\n";
print "modules exit      in its log     written?\n";
for my $n (1, 2, 127, 128, 129) {
    broken_source("broken.v", $n);
    unlink "broken.vvp";
    system("iverilog -o broken.vvp broken.v > broken.log 2>&1");
    my $code = $? >> 8;
    my $errors = () = do { open my $l, "<", "broken.log" or die; local $/; <$l> } =~ /syntax error/g;
    printf "%-7d %-9d %-14d %s\n", $n, $code, $errors, -e "broken.vvp" ? "yes" : "no";
}

broken  iverilog  syntax errors  output
modules exit      in its log     written?
1       2         1              no
2       4         2              no
127     254       127            no
128     0         128            no
129     2         129            no

The status grows with the number of errors — 2, 4, … 254 — and then, at 128 broken modules, it is 0. The log reports all 128 syntax errors, and no output file was written, but the exit status says the compilation succeeded.

The reason is Chapter 15.4's measured fact that an exit status is one byte: a program that exits with 256 exits with 0. A compiler that reports its error count through its exit status is safe only while the count stays small, and a large refactor or a wrong include path can produce hundreds of errors in one go. The exact relationship between error count and status here is Icarus's own internal detail — the measurement shows the shape, not a formula to rely on — but the one-byte truncation is the operating system's and applies to every tool.

6. The Stale Artifact: A False PASS With Every Status 0

On its own, a compiler that exits 0 without writing its output would simply make the next step fail. What turns it into a silent false pass is something every incremental flow has: yesterday's output is still there.

use strict;
use warnings;

# Monday: the design compiles and the test passes. The flow keeps sim.vvp.
system("iverilog -g2012 -o sim.vvp tb/pass.v") == 0 or die "monday compile failed\n";
print "monday : compiled, sim.vvp written\n";

# Tuesday: someone's edit breaks 128 modules. The flow checks the exit status.
open my $fh, ">", "design.v" or die;
print {$fh} "module m$_; initial begin \$display(1) end endmodule\n" for 1 .. 128;
close $fh;
system("iverilog -g2012 -o sim.vvp design.v > compile.log 2>&1");
my $rc = $? >> 8;
print "tuesday: iverilog exit $rc -> ", ($rc == 0 ? "flow says compiled" : "flow stops"), "\n";

if ($rc == 0) {
    my $out = `vvp -n sim.vvp 2>&1`;
    printf "tuesday: vvp exit %d, log says: %s\n", $? >> 8, (split /\n/, $out)[0];
}

# The fix: remove the artifact first, then require a fresh one afterwards.
unlink "sim.vvp";
system("iverilog -g2012 -o sim.vvp design.v > compile.log 2>&1");
$rc = $? >> 8;
print "fixed  : iverilog exit $rc, sim.vvp ", (-e "sim.vvp" ? "exists" : "MISSING -> compile FAILED"), "\n";

monday : compiled, sim.vvp written
tuesday: iverilog exit 0 -> flow says compiled
tuesday: vvp exit 0, log says: TEST PASSED
fixed  : iverilog exit 0, sim.vvp MISSING -> compile FAILED

Read Tuesday's lines in order. The compiler found 128 errors and exited 0. The flow believed it. The simulator ran sim.vvp — Monday's sim.vvp, built from the old, working design, because the failed compile never overwrote it — and Monday's design passed. Every exit status was 0, the log says TEST PASSED, and the design that was actually checked in does not compile.

The fix is two lines and costs nothing: delete the artifact before running the tool, and require a fresh one afterwards. With that, the exit status no longer has to be right — the missing file says the compile failed. Chapter 17.9 approached the same problem through timestamps; deleting first is stronger, because it does not depend on clock resolution or on two builds landing in the same second.

7. Four Witnesses and a Verdict by Elimination

No single source of evidence is trustworthy on its own. The exit status lied in five different ways above. The log can be truncated or missing (Chapter 18.9). The artifact can be stale. So a verdict combines them, and each one covers a failure the others miss:

The judge below asks them in a fixed order and reaches PASS only when nothing else applied — Chapter 21.8's verdict ladder, extended with the witnesses this chapter found. The driver runs every testbench twice: once through a flow that believes the exit statuses, and once through the judged flow, which also deletes the artifact before compiling. Two cases are added: Section 6's broken design with Monday's build still present, and a compiler that is not installed.

use strict;
use warnings;
require "./run_tool.pl";

# Read the simulation log for the facts the exit status cannot carry.
sub scan_log {
    my ($file) = @_;
    my %f = (finish => 0, errors => 0, fatal => 0, passed => 0);
    open my $fh, "<", $file or return %f;
    while (my $l = <$fh>) {
        $f{finish}++ if $l =~ /\$finish(?:\(\d\))? called at/;
        $f{fatal}++  if $l =~ /^FATAL:/;
        $f{errors}++ if $l =~ /^(?:ERROR|FATAL):|^TEST FAILED/;
        $f{passed}++ if $l =~ /^TEST PASSED$/;
    }
    return %f;
}

# Four witnesses: the exit statuses, the log, the artifact, and our own
# record of whether we had to stop the run. PASS only by elimination.
sub judge {
    my (%w) = @_;
    my ($c, $s) = @w{qw(compile sim)};
    return ("NOT_STARTED",    "the compiler could not be launched") unless $c->{launched};
    return ("COMPILE_FAILED", "iverilog " . describe($c))           if $c->{signal} || $c->{exit};
    return ("COMPILE_FAILED", "exit 0 but no fresh $w{vvp}")          unless $w{vvp_fresh};
    return ("TIMEOUT",        "we stopped it at the limit")          if $s->{timed_out};
    return ("CRASHED",        "killed by signal $s->{signal}")       if $s->{signal};
    my %log = scan_log($w{log});
    return ("FAIL",           "\$fatal, vvp exit $s->{exit}")        if $log{fatal};
    return ("NO_FINISH",      "no \$finish line in the log")         unless $log{finish};
    return ("FAIL",           "$log{errors} failure line(s) in log") if $log{errors};
    return ("FAIL",           "vvp exit $s->{exit}")                 if $s->{exit};
    return ("NO_VERDICT",     "finished, but never said PASSED")     unless $log{passed};
    return ("PASS",           "every witness agrees");
}

# Monday's good build of the design, left behind for Tuesday's broken one.
system("iverilog -g2012 -o broken128.vvp tb/pass.v") == 0 or die;
open my $b, ">", "tb/broken128.v" or die;
print {$b} "module m$_; initial begin \$display(1) end endmodule\n" for 1 .. 128;
close $b;

my %TRUTH = (pass => "PASS", mismatch => "FAIL", sv_error => "FAIL", finish_1 => "FAIL",
             sv_fatal => "FAIL", hang => "TIMEOUT", syntax => "COMPILE_FAILED",
             broken128 => "COMPILE_FAILED", no_tool => "NOT_STARTED");
my @TESTS = qw(pass mismatch sv_error finish_1 sv_fatal hang syntax broken128 no_tool);

printf "%-10s %-12s %-15s %-36s %s\n", "test", "exit-only", "judged", "because", "truth";
my ($naive_wrong, $judge_wrong) = (0, 0);
for my $t (@TESTS) {
    my $compiler = $t eq "no_tool" ? "iverilog-not-installed" : "iverilog";
    my $src      = $t eq "no_tool" ? "tb/pass.v" : "tb/$t.v";

    # The exit-only flow: compile, run, and believe the numbers.
    my $c = run_tool(log => "$t.c.log", timeout => 30,
                     cmd => [$compiler, "-g2012", "-o", "$t.vvp", $src]);
    my $s = { launched => 0, exit => undef, signal => 0, timed_out => 0 };
    $s = run_tool(log => "$t.s.log", timeout => 2, cmd => ["vvp", "-n", "$t.vvp"])
        if $c->{launched} && defined $c->{exit} && $c->{exit} == 0;
    my $naive = (defined $c->{exit} && $c->{exit} == 0 && defined $s->{exit} && $s->{exit} == 0)
              ? "PASS" : "FAIL";

    # The judged flow: same tools, but delete the artifact first and ask every witness.
    unlink "$t.vvp";
    $c = run_tool(log => "$t.c.log", timeout => 30,
                  cmd => [$compiler, "-g2012", "-o", "$t.vvp", $src]);
    my $fresh = -e "$t.vvp";
    $s = { launched => 0, exit => undef, signal => 0, timed_out => 0 };
    $s = run_tool(log => "$t.s.log", timeout => 2, cmd => ["vvp", "-n", "$t.vvp"]) if $fresh;
    my ($verdict, $why) = judge(compile => $c, sim => $s, vvp => "$t.vvp",
                                vvp_fresh => $fresh, log => "$t.s.log");

    my $nbad = $naive eq "PASS" && $TRUTH{$t} ne "PASS";
    $naive_wrong++ if $nbad;
    $judge_wrong++ if $verdict ne $TRUTH{$t};
    printf "%-10s %-12s %-15s %-36s %s\n", $t,
        $naive . ($nbad ? " <--" : ""), $verdict, $why, $TRUTH{$t};
}
printf "\nexit-only flow: %d false PASS of %d.  judged flow: %d wrong of %d.\n",
    $naive_wrong, scalar @TESTS, $judge_wrong, scalar @TESTS;

test       exit-only    judged          because                              truth
pass       PASS         PASS            every witness agrees                 PASS
mismatch   PASS <--     FAIL            2 failure line(s) in log             FAIL
sv_error   PASS <--     FAIL            1 failure line(s) in log             FAIL
finish_1   PASS <--     FAIL            1 failure line(s) in log             FAIL
sv_fatal   FAIL         FAIL            $fatal, vvp exit 1                   FAIL
hang       PASS <--     TIMEOUT         we stopped it at the limit           TIMEOUT
syntax     FAIL         COMPILE_FAILED  iverilog exit 2                      COMPILE_FAILED
broken128  PASS <--     COMPILE_FAILED  exit 0 but no fresh broken128.vvp    COMPILE_FAILED
no_tool    FAIL         NOT_STARTED     the compiler could not be launched   NOT_STARTED

exit-only flow: 5 false PASS of 9.  judged flow: 0 wrong of 9.

The exit-status flow gave five false passes in nine runs. Four are the simulator's zero exits from Section 3 and Section 4; the fifth is Section 6's stale build. The judged flow agreed with the truth on all nine, and each verdict carries the reason that decided it — which is what lets someone act on a FAIL without opening the log first.

Two honest limits on that result:

The judge has only been shown to work on these nine cases. Its CRASHED, NO_FINISH and NO_VERDICT branches, and a FAIL decided by the exit status alone, were never reached by any of them. A branch that has never run has not been tested, whatever it looks like on the page. Exercise 2 asks you to build the inputs that reach each one.

The log patterns are this testbench family's. ^TEST PASSED$ and ^ERROR: are conventions of these testbenches; UVM logs have their own (Module 21), and every project has some of its own. The structure of the judge transfers; the patterns must come from the logs your own flow produces.

8. Exercises

1. Your own simulator's census. If you have access to another simulator, port exit_census.pl to it: same seven endings, same table. Record which rows differ from Icarus. Before running it, write down your predictions — the gap between prediction and measurement is the useful part.

2. Reach every branch. Construct an input for each judge branch that the nine cases never reached: a simulation that ends without $finish and without a timeout; one that finishes cleanly but never prints TEST PASSED; one that is killed by a signal the simulator does not catch; and one where the log is clean but the exit status is non-zero. Run them, and fix the judge if any verdict is wrong.

3. The queue did it. Change the driver so that the hang test is stopped by an outside process rather than by run_tool, so the runner's timed_out record is never set. What does the judge report, which witness saved (or failed to save) the verdict, and what would you add to the run directory so that the queue's action is recorded where the judge can see it?

4. Two artifacts. Real flows compile into a library directory, not a single file. Design the "delete first, require fresh" rule for a step whose output is a directory of many files, without deleting anything the step does not own. What does fresh mean for a directory?

5. A tool that exits 1 on warnings. Some tools return a non-zero status when they emitted only warnings. Sketch how the judge's exit-status witness should change for such a tool, and explain why the fix belongs in a per-tool table of "what each exit code means" rather than in the judge's logic.

9. What To Remember

• $? says how the process ended — not whether the design is right. Decoding is Chapter 14.3; meaning is the tool's.
• Run tools with a captured log, a time limit, and your own record of what you did to them.
• Measured on Icarus: vvp exited 0 after a printed TEST FAILED, after $error, and after $finish(1). Only $fatal exited 1.
• $finish(N)'s argument is a diagnostic level, not an exit code.
• TERM and INT made vvp exit 0, with no signal in $?. Only KILL showed as signal 9.
• A signal in $? means the tool did not handle it. Evidence that a run was stopped must come from whoever stopped it.
• The core bit depends on the core-size limit — 0 here, so never set. The signal number is the crash.
• The compiler's status grows with its error count and is one byte: 128 broken modules exited 0 with no output written.
• A stale artifact turns a failed step into a false PASS. Measured: exit 0, exit 0, TEST PASSED, for a design that does not compile.
• Delete the artifact before the tool runs; require a fresh one afterwards.
• Combine four witnesses — the runner's record, the exit status, a fresh artifact, the log — and reach PASS only by elimination.
• Measured: exit-status-only, 5 false PASS of 9; judged, 0 wrong of 9. Branches never reached are untested.
• Measure your own tool the same way. Nothing here is a claim about any other simulator.

Chapter 26.8 turns to the environment a tool runs in — PATH, licence variables, environment modules — and to recording exactly which tool and version produced a result, so that a failure like the ones in this chapter can be explained a week later.