Linux & Bash · Part II · Module 32 · Regression Automation End to End

Deciding Pass and Fail for One Run

A regression's every later number rests on one function: the thing that turns a finished run into a word. This chapter measures three ways of writing it against thirteen real run behaviours, finds that the exit status alone is wrong nine times and the log alone five, and then mutates the working version rule by rule to show that each rule is pinned by exactly one behaviour.

Advanced28 min readLinuxPart IIRegressionExit StatusVerdict

Part II · Module 32 · Chapter 32.3 · Deciding Pass and Fail

Chapter 32.2's dispatch stage runs things and decides nothing. This chapter is the deciding, and it is the module's most important twenty minutes: every count, every pass rate, every trend and every CI gate in the rest of this module is this one function's output, aggregated. Get it wrong and nothing downstream can be right.

1. The Two Obvious Answers, Measured

There are two things a finished run gives you: a number from the operating system, and a file full of text. Both are routinely used alone.

Thirteen real behaviours from one compiled image, with the ground truth established by reading each log by hand:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  TEST          truth         A: exit only B: log only    C: both
  smoke         PASS          PASS         PASS           PASS
  fairness      PASS          PASS         PASS           PASS
  random        PASS          PASS         PASS           PASS
  b2b           PASS          PASS         PASS           PASS
  prio0         FAIL          PASS *       FAIL           FAIL
  inconsistent  INCONSISTENT  PASS *       PASS *         INCONSISTENT
  crash         CRASH         FAIL *       NO_VERDICT *   CRASH
  silent        NO_VERDICT    PASS *       NO_VERDICT     NO_VERDICT
  flaky         FAIL          PASS *       FAIL           FAIL
  hang          TIMEOUT       PASS *       NO_VERDICT *   TIMEOUT
  bogus         FAIL          PASS *       FAIL           FAIL
  vacuous       VACUOUS       PASS *       PASS *         VACUOUS
  latecrash     INCONSISTENT  FAIL *       PASS *         INCONSISTENT

  wrong answers (of 13): exit-only 9, log-only 5, both 0

Policy A — trust the exit status — is wrong nine times out of thirteen, and the direction is what matters:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  exit-only calls these a pass: prio0 inconsistent silent flaky hang bogus vacuous

Seven runs that did not pass, reported as passes. A design-property failure, a self-contradicting testbench, a run that said nothing, a seed-dependent failure, a hung run killed by a timeout, a test that does not exist, and a test that checked nothing. In a nightly regression that is seven bugs shipped.

Policy B — trust the log's verdict line — is wrong five times:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  log-only mis-handles: inconsistent crash hang vacuous latecrash

Better, and still unusable. It believes inconsistent's and vacuous's claims about themselves, it cannot tell a crash from a quiet run, and it cannot see a timeout at all.

Policy C — both, plus the evidence beside them — is wrong zero times. The rest of this chapter is that function.

Why the exit status is so much worse than it looks

Nine of thirteen is startling until you see the mechanism. Of the thirteen runs, eleven exited zero, including the hang, the vacuous pass and the run with no verdict at all. That is not a quirk of this testbench — it follows from how simulators end.

$finish exits 0. A testbench that detects a failure, prints TEST FAILED, and calls $finish has reported a failure and told the shell it succeeded. Chapter 22.2 measured it directly, and it is the single most consequential fact about reading simulator status: the exit status of a simulation is a statement about the simulator, not about the design.

The simulator exits non-zero when the simulator had a problem — $fatal, a missing file, a licence failure. Those are the crash and latecrash rows, and notice that policy A gets both of them wrong too, in the other direction: it calls them FAIL when one is a crash and the other is a testbench that passed and then died while closing out. Different investigations, same number.

2. The Signal Table, Because a Timeout Is Not a Failure

One row above deserves its own measurement. hang waits forever; something has to kill it; and what the shell then sees depends entirely on which signal was used.

vvp as the direct child of the shell — the subshell execs into it, so $! is the simulator and not a wrapper:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== vvp is the direct child (the subshell execs into it) ===
  kill -TERM  -> process ended, wait rc=0    log 105 bytes
  kill -INT   -> STILL RUNNING after 2s (signal ignored or handled and resumed)
  kill -HUP   -> process ended, wait rc=0    log 105 bytes
  kill -QUIT  -> STILL RUNNING after 2s (signal ignored or handled and resumed)
  kill -KILL  -> process ended, wait rc=137  log 0 bytes

=== control: a plain sleep through the same harness ===
  sleep + kill -TERM  -> wait rc=143
  sleep + kill -KILL  -> wait rc=137

Four findings, and the control row is what makes them trustworthy — sleep gives the textbook 128 + signal answers through the identical harness, so the harness is not the thing distorting them.

kill -TERM on a simulation yields exit status 0. Not 143. The tool installs a handler, shuts down, and exits successfully. So a timeout wrapper that kills a hung run and then reads the exit status concludes that the run passed — which is why runsim records a separate timedout flag and why the verdict function checks it first.

kill -INT does not stop it at all. Ctrl-C on a regression leaves simulators running. Chapter 30.4 measured the same tool declining to die on INT, and Chapter 30.1 found it resetting SIGHUP's ignore request so that nohup did not protect it. A tool that manages its own signals manages them in whichever direction it chose, not the direction you expect.

kill -QUIT is also ignored, so a script escalating INT → QUIT → KILL wastes its first two steps.

And kill -9 leaves a zero-byte log. The buffered output is lost — Chapter 30.4 measured that cost at exactly 88 log lines. TERM left 105 bytes; KILL left nothing. So the hard kill destroys the only evidence about why the run needed killing, which is the argument for TERM first and a grace period, however much slower it is.

Figure 1 — three verdict policies against the same thirteen measured run behaviours. The exit status alone is wrong nine times, and seven of those nine are a non-passing run reported as a pass. The log's verdict line alone is wrong five times. Reading both, plus the evidence recorded beside them, is wrong none.

3. The Verdict Function

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# verdict RUNDIR -- classify one finished run from its exit status AND its log.
# Prints one word on stdout and exits with the code for that class.
#   PASS 0 | FAIL 1 | TIMEOUT 2 | CRASH 3 | NO_VERDICT 4 | INCONSISTENT 5 | VACUOUS 7
set -u
d=${1:-}
[ -n "$d" ] || { echo "verdict: need a run directory" >&2; exit 64; }
log=$d/run.log
[ -r "$log" ] || { echo NO_LOG; exit 6; }
rc=$(cat "$d/tool.rc" 2>/dev/null || echo 0)
to=$(cat "$d/timedout" 2>/dev/null || echo 0)

# 1. The limit fired. This outranks everything, because a killed run's log is
#    whatever happened to be flushed -- and TERM makes vvp exit 0.
[ "$to" = 1 ] && { echo TIMEOUT; exit 2; }

# 2. The tool itself failed to finish. Non-zero without a verdict is a crash.
passed=$(grep -c 'TEST PASSED' "$log")
failed=$(grep -c 'TEST FAILED' "$log")
if [ "$rc" -ne 0 ] && [ "$failed" -eq 0 ] && [ "$passed" -eq 0 ]; then
  echo CRASH; exit 3
fi

# 3. No verdict at all. A run that said nothing did not pass.
[ "$passed" -eq 0 ] && [ "$failed" -eq 0 ] && { echo NO_VERDICT; exit 4; }

# 4. Both verdicts, or a verdict contradicted by the evidence beside it.
errlines=$(grep -c '[[:space:]]ERROR' "$log")
if [ "$passed" -gt 0 ] && [ "$failed" -gt 0 ]; then echo INCONSISTENT; exit 5; fi
if [ "$passed" -gt 0 ] && [ "$errlines" -gt 0 ]; then echo INCONSISTENT; exit 5; fi

# 5. An explicit failure.
[ "$failed" -gt 0 ] && { echo FAIL; exit 1; }

# 6. PASSED, no error lines, and the tool agreed. Require a non-zero check count:
#    a run that checked nothing has not demonstrated anything.
checks=$(sed -n 's/.*checks=\([0-9]*\).*/\1/p' "$log" | tail -1)
[ -n "$checks" ] && [ "$checks" -eq 0 ] && { echo VACUOUS; exit 7; }
[ "$rc" -ne 0 ] && { echo INCONSISTENT; exit 5; }
echo PASS; exit 0

Seven classes, and the ordering of the rules is the whole design:

class	exit	means	the investigation
`PASS`	0	finished, said so, and checked something	none
`FAIL`	1	finished and said it failed	the design
`TIMEOUT`	2	the limit fired	the machine, or the test
`CRASH`	3	the tool failed without a verdict	the environment
`NO_VERDICT`	4	finished and said nothing	the testbench
`INCONSISTENT`	5	its own evidence contradicts its claim	the testbench
`VACUOUS`	7	claimed a pass after zero checks	the test

Three things about that table are worth more than the code.

Four of the seven are not design questions. TIMEOUT, CRASH, NO_VERDICT and VACUOUS all mean "the run did not produce a usable answer", and collapsing them into FAIL sends somebody to open a waveform that has nothing in it. Chapter 32.5 reports that split explicitly.

Six is reserved and deliberately skipped, because 6 is NO_LOG — a run directory with no log at all, which is a different thing again from a run that produced one. And 7 rather than 6 for VACUOUS keeps every value below the shell's reserved region, which is Chapter 21.4's rule: stay under 126 or collide with 128 + signal.

And the order is not arbitrary. The timeout check is first because TERM makes the exit status 0 and lets the log look merely truncated. The crash check precedes the no-verdict check because a tool that died non-zero is a different report from a testbench that forgot to speak. Reordering the rules changes the answers, which the next section measures.

4. Every Rule, Pinned by Exactly One Behaviour

A verdict function is a thing that only ever says reassuring words, so it is exactly the kind of code that is never known to work. The test is Chapter 21.8's: break it one rule at a time and see whether anything notices.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  MUTATION                                       wrong    which tests
  (baseline: the real verdict)                   0
  drop rule 1 (the timeout outranks all)         1        hang
  drop rule 2 (non-zero + no verdict = CRASH)    1        crash
  drop rule 3 (no verdict is not a pass)         1        silent
  drop the PASSED-with-ERROR cross-check         1        inconsistent
  drop the zero-checks guard                    1        vacuous
  drop the final rc cross-check                 1        latecrash

Six rules, six removals, and each one breaks exactly one behaviour and a different one each time. No rule is redundant, no rule is doing two jobs, and no behaviour is caught twice. That is as tight a result as this kind of test produces, and it is worth saying plainly that it did not start out that way.

Two of those six rules were dead code

The first version of this table had four rules firing and two doing nothing:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  drop the zero-checks guard                    0
  drop the final rc cross-check                 0

Removing them changed no answer, because the fixture had no behaviour that needed them. I had written both from reasoning — "surely a run that checks nothing should not pass", "surely a non-zero status under a PASSED line matters" — and reasoning is not evidence.

Two choices at that point, and the wrong one is tempting. Delete the rules as speculation. Or construct the behaviour and find out whether they were right.

They were right, and the behaviours are both ordinary:

vacuous drives nothing, checks nothing, and prints TEST PASSED ... checks=0. Every verification engineer has shipped one: a test whose stimulus was commented out, or whose checker was never connected. It differs from a genuine pass only in the check count.
latecrash reaches a real verdict and then dies, the way a tool fails while releasing a licence or closing a coverage database. It differs from a genuine pass only in the exit status.

Both now appear in the mutation table, and both earn their rule.

The general lesson is the one this track keeps arriving at from different directions. A rule that no test exercises is not a safe rule, it is an unmeasured one — it may be wrong, and nothing will tell you. When a mutation survives, the first question is not "is my test weak" but "what behaviour does this rule claim exists, and does it?" Sometimes, as here, the answer is that the behaviour is real and your fixture is thin. Sometimes the rule is simply wrong. You cannot tell which without building the case.

5. The Rules, One at a Time

The timeout outranks everything

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[ "$to" = 1 ] && { echo TIMEOUT; exit 2; }

First, before the log is even read. Section 2 is the reason: a TERMed simulation exits 0, and its log is whatever happened to be flushed — which can easily include a TEST PASSED from an earlier phase of a multi-phase test. A timeout that is checked after the log can be overruled by the log, and then a hung run is a pass.

The flag comes from the thing that did the killing, because nothing else knows:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

printf '%s\n' "$timedout" > "$d/timedout"

Non-zero plus no verdict is a crash

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

if [ "$rc" -ne 0 ] && [ "$failed" -eq 0 ] && [ "$passed" -eq 0 ]; then
  echo CRASH; exit 3
fi

Both conjuncts matter. Non-zero with a verdict is latecrash — the tool died after deciding — and that is INCONSISTENT, a testbench or tool question, not a crash. Non-zero without a verdict is a tool that never got there.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[ "$passed" -eq 0 ] && [ "$failed" -eq 0 ] && { echo NO_VERDICT; exit 4; }

The silent row. A run that exited 0 and said nothing has demonstrated nothing, and the default must be to say so rather than to assume. This is the same shape as Chapter 31.7's MISSING: absence of bad news is not good news.

A claim contradicted by its own evidence

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

errlines=$(grep -c '[[:space:]]ERROR' "$log")
if [ "$passed" -gt 0 ] && [ "$failed" -gt 0 ]; then echo INCONSISTENT; exit 5; fi
if [ "$passed" -gt 0 ] && [ "$errlines" -gt 0 ]; then echo INCONSISTENT; exit 5; fi

The inconsistent row: TEST PASSED ... errors=0 on one line and TB ERROR ... gnt=10 expected 01 three lines above it. The testbench's verdict is hardcoded; its error counter is real; and they disagree.

Note the pattern is '[[:space:]]ERROR', not 'ERROR'. A bare ERROR matches the word inside error_inject, ERRORS=0, and a signal called error_flag — the over-matching Chapter 24.1 measured, where a naive signal search returned mostly comments and strings. Requiring leading whitespace anchors it to the message-prefix position this tool actually uses.

This rule reports a disagreement; it does not resolve it. INCONSISTENT deliberately does not mean "really a failure" — it means the run cannot be believed in either direction, and the thing to fix is the testbench. Chapter 29.4 reached the same conclusion from the counting side: when two classifiers disagree, the disagreement is the finding.

A pass has to have proved something

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

checks=$(sed -n 's/.*checks=\([0-9]*\).*/\1/p' "$log" | tail -1)
[ -n "$checks" ] && [ "$checks" -eq 0 ] && { echo VACUOUS; exit 7; }

The vacuous row. tail -1 is doing real work: a verbose run prints a checks= line twice, and the final one is the summary. Taking the first would read a mid-run figure.

And the [ -n "$checks" ] guard matters as much as the comparison. A testbench that does not report a check count at all leaves $checks empty, and [ "" -eq 0 ] is a syntax error, not a false. A policy that cannot be evaluated must not be silently skipped and must not crash — here, an absent count means the rule does not apply, which is the honest reading.

And the tool has to agree

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[ "$rc" -ne 0 ] && { echo INCONSISTENT; exit 5; }
echo PASS; exit 0

The latecrash row, and the last gate before a PASS is issued. Everything else has been ruled out; the tool's own status is the final cross-check.

6. What This Function Must Not Do

It must not fix anything. INCONSISTENT is not silently converted to FAIL, and VACUOUS is not converted to PASS on the grounds that nothing went wrong. A classifier that repairs its input destroys the signal that something needs repairing.

It must not read anything but this run. No comparison against the previous night, no knowledge of which tests "usually fail". Those are Chapter 32.5's job, and mixing them in makes a single run's verdict depend on history — so the same run gets different verdicts on different days.

It must not be approximate. Every path ends in exactly one echo and one exit, so there is no default and no fall-through. A verdict function with a reachable end-of-script is a verdict function that can return the empty string, and the empty string will be counted as something.

And it must not be slow. This runs once per run, which on a real regression is thousands of times. It is four greps and a sed over one file — cheap, and worth keeping cheap. Reading a log twice because two rules each want their own grep is the easy way to make reporting take longer than simulating.

Practice

Collect at least six genuinely different run behaviours from your own flow, with their exit statuses and logs.
Establish the ground truth by reading each log by hand, before writing any classifier.
Classify them by exit status alone and count the wrong answers.
Count how many of those errors are in the dangerous direction — not-a-pass reported as a pass.
Classify them by the log's verdict line alone and count again.
Find a run your tool reports as TEST FAILED while exiting 0.
Kill a hung simulation with TERM and read the exit status.
Do the same with INT, QUIT and KILL and tabulate what the tool does.
Run the same harness against sleep to prove the harness is not distorting the numbers.
Compare the log sizes after TERM and after KILL.
Record a separate timedout flag and explain why the exit status cannot carry it.
Write the verdict function with one class per outcome and no fall-through.
Put the timeout check first and then move it last; find the run whose verdict changes.
Mutate each rule in turn and record which behaviour breaks.
Find a rule whose removal breaks nothing — then decide whether to delete it or build the behaviour.
Build that behaviour and re-run the mutation.
Use a bare ERROR pattern and find something it matches that it should not.
Leave the check count out of a testbench's summary and confirm your rule does not crash.
Make the function convert INCONSISTENT to FAIL and say what information is lost.
Count how many file reads your function performs per run.

Check Your Understanding

Why is this function the most important in the module?
Of thirteen behaviours, how many does the exit status alone get wrong?
In which direction are those errors, and why does the direction matter?
How many does the log's verdict line alone get wrong, and which?
Why do eleven of thirteen runs exit zero?
What is the exit status of a simulation a statement about?
What does kill -TERM on vvp produce, and why is that dangerous?
What do INT and QUIT do to it?
Why does the sleep control row matter?
What does kill -9 cost you?
Why is the timeout check first?
Where does the timedout flag come from, and why can the exit status not carry it?
Why does the crash rule need both conjuncts?
Why is NO_VERDICT not PASS?
Why '[[:space:]]ERROR' rather than 'ERROR'?
What does INCONSISTENT claim, and what does it deliberately not claim?
What is tail -1 doing in the check-count extraction?
Why the [ -n "$checks" ] guard?
Which two rules were dead code, and what were the two options?
Why was constructing the behaviour the right choice?
State the general rule about a rule no test exercises.
Why must the function not convert INCONSISTENT to FAIL?
Why must it not consult the previous night's results?
Why must it have no fall-through path?
Why does its speed matter?

Answers

Because every count, pass rate, trend and CI gate in the rest of the module is this function's output aggregated — nothing downstream can be right if it is wrong.
Nine of thirteen.
Seven are not-a-pass reported as a pass — prio0, inconsistent, silent, flaky, hang, bogus, vacuous. That direction ships bugs rather than wasting time.
Five: inconsistent, crash, hang, vacuous, latecrash. It believes two runs' claims about themselves, cannot tell a crash from a quiet run, and cannot see a timeout.
Because $finish exits 0. A testbench that detects a failure, prints TEST FAILED and calls $finish has told the shell it succeeded.
About the simulator, not the design. It goes non-zero when the simulator had a problem — $fatal, a missing file, a licence failure.
Exit status 0 and a partially written log. A timeout wrapper that then reads the exit status concludes the hung run passed.
Nothing — the tool keeps running. So Ctrl-C leaves simulators alive, and an INT → QUIT → KILL escalation wastes its first two steps.
Because sleep gives the textbook 143 and 137 through the identical harness, proving the harness is not what distorts the simulator's numbers.
The buffered log. TERM left 105 bytes; KILL left zero — so the hard kill destroys the only evidence about why the run needed killing.
Because a TERMed run exits 0 and its log can still contain a TEST PASSED from an earlier phase, so a later timeout check can be overruled by the log.
From the process that did the killing — nothing else knows. The exit status cannot carry it because the tool exits 0 on TERM.
Because non-zero with a verdict is a tool that died after deciding — INCONSISTENT, a different report from a tool that never got there.
Because a run that exited 0 and said nothing has demonstrated nothing. Absence of bad news is not good news.
Because a bare ERROR matches error_inject, ERRORS=0 and a signal named error_flag. Requiring leading whitespace anchors it to the message-prefix position.
It claims the run's own evidence contradicts its claim, so the run cannot be believed in either direction. It deliberately does not claim the run really failed.
Taking the final checks= line: a verbose run prints one mid-run and one in the summary, and the summary is the one that counts.
Because a testbench that reports no count leaves the variable empty, and [ "" -eq 0 ] is a syntax error rather than a false. An absent count means the rule does not apply.
The zero-checks guard and the final rc cross-check. Delete them as speculation, or construct the behaviour and find out.
Because both behaviours are ordinary — a test whose checker was never connected, and a tool that dies while releasing a licence — and both differ from a genuine pass in exactly one field.
A rule no test exercises is not a safe rule, it is an unmeasured one. When a mutation survives, ask what behaviour the rule claims exists and whether it does.
Because a classifier that repairs its input destroys the signal that the testbench needs repairing.
Because then a single run's verdict depends on history, and the same run gets different verdicts on different days.
Because a reachable end-of-script can return the empty string, and the empty string will be counted as something.
Because it runs once per run — thousands of times on a real regression — and re-reading the log per rule is how reporting comes to take longer than simulating.

7. Remember

This one function is what every later number is made of.
Exit status alone: 9 of 13 wrong, and seven of those call a non-passing run a pass.
Log verdict alone: 5 of 13 wrong. Both together: 0 of 13.
$finish exits 0, so eleven of thirteen runs exited zero. A simulation's exit status describes the simulator, not the design.
kill -TERM on the simulator gives exit status 0 — not 143 — so a timeout must be recorded as its own flag and checked first.
INT and QUIT do not stop it at all, and kill -9 leaves a zero-byte log.
Seven classes, and four of them are not design questions: TIMEOUT, CRASH, NO_VERDICT, VACUOUS.
Keep every class below 126 so none collides with 128 + signal.
Each of the six rules is pinned by exactly one behaviour — and two of them were dead code until the behaviour was built rather than deleted.
A rule no test exercises is unmeasured, not safe.
'[[:space:]]ERROR', never bare ERROR.
Report disagreement, do not resolve it. INCONSISTENT means the run cannot be believed either way.
No fall-through, no history, no repairs, and four file reads — not one per rule.

8. What Is Next

Every run now has a word and a number attached to it, written in its own directory. Chapter 32.4 gathers thousands of those into one tab-separated file with a fixed schema — the only thing the report, the rerun and the trend will ever read, and the file that decides whether a run that never happened can be noticed.

Where this fits

Part of the Linux & Bash curriculum.

1. The Two Obvious Answers, Measured

2. The Signal Table, Because a Timeout Is Not a Failure

3. The Verdict Function

4. Every Rule, Pinned by Exactly One Behaviour

5. The Rules, One at a Time

The timeout outranks everything

Non-zero plus no verdict is a crash

Silence is not consent

A claim contradicted by its own evidence

A pass has to have proved something

And the tool has to agree

6. What This Function Must Not Do

Practice

Check Your Understanding

7. Remember

8. What Is Next

Where this fits