Linux & Bash · Part II · Module 34 · Defensive Automation Around EDA Tools

Tools That Exit Zero on Failure

A tool's exit status is a promise it does not always keep. This chapter measures a real compiler whose status is its error count modulo 256 — so a compile with exactly 256 errors exits zero and produces nothing — follows the resulting failure one stage downstream where it is blamed on the wrong tool, and builds a wrapper that decides from evidence instead.

Advanced26 min readLinuxPart IIEDAExit StatusDefensive

Part II · Module 34 · Chapter 34.1 · Tools That Exit Zero on Failure

Module 33 was about diagnosing a flow after it broke. This module is about writing one that cannot break quietly, and it begins where every flow begins: a tool ran, and you need to know whether it worked.

The obvious answer is the exit status. Here is what that is worth.

1. A Compiler Whose Status Is a Count

Generate Verilog containing exactly N undefined identifiers, compile it, and record what the tool says against what it returns:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  N errs   reported     exit     note
  1        1            1
  2        2            2
  10       10           10
  100      100          100
  200      200          200
  255      255          255
  256      256          0        <<< EXIT 0 WITH 256 ERRORS
  257      257          1
  300      300          44
  511      511          255
  512      512          0        <<< EXIT 0 WITH 512 ERRORS

The exit status is the error count modulo 256. Not a status at all — a count, truncated by the one byte a status gets, which Chapter 32.7 measured as the reason exit 256 reads as 0.

Three readings of that table, in increasing order of consequence.

300 errors → 44 is already meaningless. A caller branching on 44 is branching on arithmetic.

256 errors → 0 is a lie with a specific shape: the single most common value for "it worked".

And nothing in the table is a bug in the tool as such. The tool printed 256 error(s) during elaboration. every time. It told the truth on standard error and lost it on the way out, because the exit status has nowhere to put a number larger than 255.

2. What the Flow Does Next

A 256-error compile produces no output file at all:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  1    errors -> rc=1    artifact: NONE    log tail: 1 error(s) during elaboration.
  256  errors -> rc=0    artifact: NONE    log tail: 256 error(s) during elaboration.

So the flow everybody writes first:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
set -u
iverilog -o "$1" "$2" > "$3" 2>&1 || { echo "  compile FAILED (rc=$?)" >&2; exit 2; }
echo "  compile OK -- proceeding to simulate"
vvp "$1" > "$4" 2>&1
echo "  vvp rc=$?"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  compile OK -- proceeding to simulate
  vvp rc=255
  the run log it produced: 40 bytes -- err/out.vvp: Unable to open input file.

The failure surfaced one stage late, blamed on the wrong tool

Read that output as somebody would at nine in the morning. The compile reported success. The simulator then failed with Unable to open input file, exit 255, and a 40-byte log.

Every instinct now points at the simulator, the filesystem, or the path. People check whether the directory exists, whether the disk is full, whether a parallel job deleted it — and Chapter 33.1's triage would say TOOL layer, correctly and uselessly, because the simulator genuinely did fail to open a file that genuinely is not there.

The actual answer is in the previous stage's log, in a line that was printed, captured and never read:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

256 error(s) during elaboration.

Three properties make this failure class worth a chapter of its own:

The evidence exists. Nothing was lost; the tool said exactly what happened. It was the status that was consulted instead.

The symptom appears downstream. By the time anything looks wrong, the stage that was wrong has finished successfully and its log is one directory up.

And it is silent at exactly the wrong scale. One compile of a toy file with 256 errors is implausible. One compile of a real design after somebody renames a widely-used package is routine — and a few hundred Unable to bind errors is the ordinary count for it.

3. The Simulator Does It Too, Constantly

The compile case is dramatic and rare. The run case is undramatic and happens every day:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  TEST           rc    verdict line   ERRORs   the truth
  smoke          0     TEST PASSED    0        it passed
  prio0          0     TEST FAILED    1        a design check failed
  bogus          0     TEST FAILED    1        the test does not exist
  inconsistent   0     TEST PASSED    1        errors counted, PASSED printed
  crash          1                    0        the tool died
  silent         0                    0        nothing was decided
  vacuous        0     TEST PASSED    0        nothing was checked
  latecrash      1     TEST PASSED    0        passed, then died

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  of 5 runs that did NOT pass, 5 exited 0

Every single non-passing run returned zero. $finish exits 0, and a testbench that detects a failure, prints TEST FAILED and calls $finish has reported the failure and told the shell it succeeded. Chapter 22.2 measured this directly and it is the foundational fact of the whole track's Part II.

The two runs that did return non-zero are the two where the tool had a problem, not the design — which is the rule underneath all of this:

A tool's exit status describes the tool, not the work.

Non-zero means the tool could not complete. Zero means the tool completed, and says nothing whatever about whether what it completed is any good.

Figure 1 — the measured relationship between what the compiler reported and what it returned. The status is the error count modulo 256, so the two values that mean 'it worked' to a caller are produced by a clean compile and by one with exactly 256 errors.

4. Deciding From Evidence

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# compile OUT SRC... -- compile, and decide success from EVIDENCE, not status.
#
# iverilog's exit status is its error count modulo 256, so a compile with
# exactly 256 errors exits 0 and produces no output file. Measured.
# Three independent conditions must all hold before this reports success.
set -u
out=${1:-}; shift || true
[ -n "$out" ] && [ $# -gt 0 ] || { echo "compile: need OUT and at least one source" >&2; exit 2; }
log=${COMPILE_LOG:-$out.log}
rm -f "$out"                       # never let a previous artifact answer for this run
iverilog -o "$out" "$@" > "$log" 2>&1
rc=$?

# 1. the tool's own count, which it prints whether or not the status survives
errs=$(sed -n 's/^\([0-9][0-9]*\) error(s) during elaboration\./\1/p' "$log" | tail -1)
[ -n "$errs" ] || errs=0
# 2. error lines, in case the summary line is absent (a different failure mode)
lines=$(/usr/bin/grep -c ': error:' "$log")
# 3. the artifact itself
if [ -r "$out" ]; then sz=$(wc -c < "$out" | tr -d ' '); else sz=0; fi

printf 'compile: rc=%s  summary_errors=%s  error_lines=%s  artifact=%s bytes\n' \
  "$rc" "$errs" "$lines" "$sz" >&2

[ "$errs" -eq 0 ] || { echo "compile: $errs error(s) reported by the tool" >&2; exit 3; }
[ "$lines" -eq 0 ] || { echo "compile: $lines error line(s) in the log" >&2; exit 3; }
[ "$sz" -gt 0 ]    || { echo "compile: no output produced at '$out'" >&2; exit 4; }
exit 0

Three independent conditions, and the independence is the design:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  N errs   iverilog   compile    verdict
  0        0          0          OK
  1        1          3          refused: errors
  255      255        3          refused: errors
  256      0          3          refused: errors
  257      1          3          refused: errors
  512      0          3          refused: errors

The tool's own count

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

errs=$(sed -n 's/^\([0-9][0-9]*\) error(s) during elaboration\./\1/p' "$log" | tail -1)

The number the tool printed, which survived the truncation that destroyed the status. Anchored on ^ and matching the full sentence, because Chapter 33.7 measured an unanchored pattern rewriting a field it was never meant to touch. tail -1 because a multi-stage compile can print the summary more than once.

The error lines

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

lines=$(/usr/bin/grep -c ': error:' "$log")

A second, independent count, for the case the summary line is absent — a tool killed mid-compile prints its errors and never reaches its summary. Two counts that disagree is itself informative, which is why both are printed.

Note /usr/bin/grep rather than grep: on this machine grep is a shell function dispatching to ugrep, which honours .gitignore and gives different answers in a repository. Pinning the tool is Chapter 23.5's rule, measured there as the same command returning two hits or three.

The artifact

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

if [ -r "$out" ]; then sz=$(wc -c < "$out" | tr -d ' '); else sz=0; fi
…
[ "$sz" -gt 0 ] || { echo "compile: no output produced at '$out'" >&2; exit 4; }

A compile that reports no errors and produces nothing has failed in a way neither count can see. And the preceding line matters as much:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

rm -f "$out"                       # never let a previous artifact answer for this run

Without that, yesterday's image passes today's check. Chapter 30.5 measured exactly this — a stale artifact making a failed build report success — and the fix is to delete first rather than to compare timestamps, because Chapter 33.1 measured test -nt missing a same-second edit.

Why three conditions rather than the best one

The three overlap heavily and that is deliberate. Each catches something the others cannot:

condition	catches	misses
the summary count	any number of errors, including 256	a tool killed before it printed the summary
the error lines	a partial compile with no summary	a tool whose errors use different wording
the artifact	a silent failure with no diagnostic at all	a stale artifact, unless you delete first

A single check has to be the right one for every failure mode the tool has, and you do not know that list. Three cheap independent checks have to be wrong together, which is a much smaller target — and when they disagree, the disagreement is the finding, which is the conclusion Chapter 29.4 reached about two classifiers and Chapter 32.3 built into a seven-way verdict.

The cost is that the wrapper is tied to one tool's output format. That is real and it is the right trade: the alternative is a generic wrapper that trusts a status, and this chapter is about what that is worth.

5. The Statuses the Wrapper Returns

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  0  the compile produced a clean artifact
  2  usage -- the wrapper was called wrongly
  3  the tool reported errors
  4  the tool reported none and produced nothing

Three decisions here, all from earlier measurements.

3 and 4 are different. "It failed and said so" and "it failed silently" need different responses: the first is a source problem, the second is a tool or environment problem. Collapsing them is the mistake Chapter 32.7 measured in a regression's own contract.

Nothing returns 1. The value is reserved for the thing Chapter 32.3 uses it for — a failing test — so a build problem cannot be counted as one.

And the wrapper never passes the tool's own status through. rc is printed for the record and never returned, because this chapter is five hundred words of evidence that it does not mean what its name says.

6. Applying This Beyond One Tool

The specific numbers are Icarus's. The shape is not, and the questions to ask of any tool in a flow are the same four:

Does it have a documented exit-status contract? Most EDA tools do not, and the ones that do often document a value they return in a case nobody hits.

Does it print a machine-readable summary? N error(s) during elaboration. is a gift — a stable, greppable, countable sentence. Chapter 29.3 found the same in UVM's report summary, and both are worth more than any status.

Does it produce an artifact, and does it remove a stale one on failure? Measured here: iverilog produces nothing on failure and does not remove a previous output — which is why the wrapper deletes first.

And how does it behave at the boundaries? The 256 case was found by looking for it. The general technique is to drive the tool to an extreme deliberately and watch the status: zero errors, one error, hundreds, and — the one that found this — exactly 256.

A probe bug worth repeating, because exec does not come back

Counting how many non-passing runs exited zero, I wrote:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

rc=$(cd "$d" && exec vvp ../../logs/sim.vvp +TESTNAME="$t" … >/dev/null 2>&1; echo $?)

and got:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

bin/probe_runzero.sh: line 28: [: : integer expression expected
  of 5 runs that did NOT pass, 0 exited 0

rc was empty, and the tally reported 0 instead of 5. exec replaces the shell with the named program, so the echo $? after it is never reached — there is no shell left to run it. The command substitution captured the simulator's output, which was redirected away, so it captured nothing.

The fix is to not use exec when you need the shell afterwards:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

( cd "$d" && vvp ../../logs/sim.vvp +TESTNAME="$t" … ) >/dev/null 2>&1
rc=$?

exec is right in Chapter 31.6's wrapper, where the subshell should become the simulator so that $! is the tool and not a shell. It is wrong anywhere the shell has more to do. And the failure is quiet in the worst way — an empty variable, a [ complaining about an integer, and a plausible-looking 0 in the report.

Practice

Generate a source file with exactly N errors for several values of N.
Record what your compiler reports and what it returns, and look for a relationship.
Find the value where the status becomes 0, if there is one.
Check whether a failing compile leaves an artifact behind.
Check whether it removes a previous one.
Write the naive flow that trusts $? and run it on a 256-error source.
Read the resulting error message and say which tool it blames.
Find the line in the previous stage's log that explains it.
Tabulate your simulator's status against the truth for eight different outcomes.
Count how many non-passing runs returned zero.
State what a tool's exit status actually describes.
Write a wrapper with three independent conditions.
Delete the output before compiling and explain what that prevents.
Anchor the summary pattern and say what an unanchored one would hit.
Pin grep to /usr/bin/grep and find out what your shell's grep actually is.
Give "errors reported" and "nothing produced" different statuses.
Avoid returning 1 and say what it is reserved for.
Make the wrapper print the tool's status without returning it.
Ask the four questions of a tool in your own flow.
Drive that tool to a boundary and watch its status.
Write exec where the shell still has work to do and find out what your variable contains.

Check Your Understanding

What is the relationship between the compiler's exit status and its error count?
What does a compile with exactly 256 errors return, and what does it produce?
Why is 300 errors → 44 already a problem?
In what sense is this not a bug in the tool?
What does the naive flow print, and what fails next?
Which tool gets blamed, and what does the triage say?
Where is the real explanation?
Give the three properties that make this failure class distinctive.
Why is the 256 case not as implausible as it sounds?
How many of the five non-passing runs returned zero?
Which two runs returned non-zero, and what does that tell you?
State the rule about what an exit status describes.
Name the three independent conditions and what each catches.
What does each one miss?
Why three rather than the best one?
What is the cost of that approach?
Why delete the output before compiling?
Why not compare timestamps instead?
Why /usr/bin/grep rather than grep?
Why are statuses 3 and 4 different?
Why does the wrapper never return 1?
Why does it print the tool's status but not return it?
Give the four questions to ask of any tool.
How was the 256 case found?
What did exec do to the tally, and why?
Where is exec the right choice?

Answers

The status is the error count modulo 256.
Exit 0, and no artifact at all.
Because a caller branching on 44 is branching on arithmetic — the number carries no meaning as a status.
The tool printed 256 error(s) during elaboration. every time. It told the truth on its output and lost it on the way out, because a status has nowhere to put a number above 255.
compile OK -- proceeding to simulate, then the simulator fails with Unable to open input file, exit 255, and a 40-byte log.
The simulator. The triage says TOOL layer — correctly, because the simulator really did fail to open a file that really is not there.
In the previous stage's log, in the line 256 error(s) during elaboration. that was printed, captured and never read.
The evidence exists and was not consulted; the symptom appears one stage downstream; and it is silent at exactly the scale where it is plausible.
Because a few hundred Unable to bind errors is the ordinary count after somebody renames a widely-used package in a real design.
All five.
crash and latecrash — the two where the tool had a problem rather than the design.
A tool's exit status describes the tool, not the work. Non-zero means it could not complete; zero says nothing about whether what it completed is any good.
The tool's own summary count (any number of errors, including 256); the error lines (a partial compile with no summary); and the artifact (a silent failure with no diagnostic).
The summary misses a tool killed before printing it; the error lines miss different wording; the artifact misses a stale one unless you delete first.
Because a single check must be right for every failure mode the tool has, and you do not know that list. Three cheap independent checks have to be wrong together.
The wrapper is tied to one tool's output format — which is the right trade against a generic wrapper that trusts a status.
So that a previous artifact cannot answer for this run — a stale artifact making a failed build report success was measured in Chapter 30.5.
Because test -nt uses whole seconds and missed a same-second edit, measured in Chapter 33.1.
Because on this machine grep is a shell function dispatching to ugrep, which honours .gitignore and gives different answers inside a repository.
Because "it failed and said so" is a source problem and "it failed silently" is a tool or environment problem, and they need different responses.
Because 1 is reserved for a failing test, so a build problem cannot be counted as one.
Because the status is recorded as evidence while the chapter is an argument that it does not mean what its name says.
Does it have a documented exit-status contract; does it print a machine-readable summary; does it produce an artifact and remove a stale one on failure; and how does it behave at the boundaries?
By deliberately driving the tool to extremes and watching the status — zero, one, hundreds, and exactly 256.
It made rc empty, so the tally said 0 instead of 5. exec replaces the shell, so the echo $? after it is never reached.
In a run wrapper where the subshell should become the simulator, so that $! is the tool rather than a shell.

7. Remember

A compiler's exit status was its error count modulo 256. 256 errors exits 0; 300 exits 44; 512 exits 0.
That compile produced no artifact, and the flow that trusted $? printed "compile OK" and went on to simulate.
The failure surfaced one stage downstream, blamed on the simulator, with the explanation sitting unread in the previous log.
Five of five non-passing simulations exited 0. $finish exits 0.
A tool's exit status describes the tool, not the work. Non-zero means it could not complete; zero means nothing more than that it finished.
Decide from three independent conditions: the tool's own summary count, the error lines, and the artifact.
Delete the output first, so a stale artifact cannot answer for this run — and never use timestamps for it.
Anchor the summary pattern, and pin grep to /usr/bin/grep.
Separate "reported errors" from "produced nothing", and never return 1 — that belongs to a failing test.
Print the tool's status as evidence; never return it.
Ask four questions of any tool: contract, machine-readable summary, artifact behaviour, boundary behaviour — and find the boundaries by driving it there on purpose.
exec never returns. A $? after it is unreachable, and the result is an empty variable and a plausible wrong number.

8. What Is Next

This chapter assumed the log exists and can be read. Chapter 34.2 removes that assumption — a run that produced no log at all, or half of one — and measures what a checker built on "no error line was found" concludes about a file that is not there.

Where this fits

Part of the Linux & Bash curriculum.