Linux & Bash · Part II · Module 34 · Defensive Automation Around EDA Tools
Failing Loudly, and Never Half-Succeeding
Every habit in this module reduces to one: stop at the first unrecoverable problem and say which one it was. This chapter measures two flows on a source that does not compile — one reports the problem and returns a status naming the stage, the other prints RESULT: PASS — and then works out which failures should stop a flow and which it must survive.
Advanced25 min readLinuxPart IIEDAFail FastDefensive
Part II · Module 34 · Chapter 34.7 · Failing Loudly
Six chapters of this module have been about detecting one thing each. This one is about what a flow does once it has detected something, and it is the chapter the other six exist to serve:
a flow that cannot fail is a flow whose success means nothing.
1. Two Flows, One Broken Source
The flow most flows are:
#!/bin/bash
# flow_quiet -- the flow that carries on, as most of them do
root=$(cd "$(dirname "$0")/.." && pwd)
iverilog -o "$root/fq.vvp" ${SRC:-tb/tb_arb.v rtl/arb.v} > "$root/fq.compile.log" 2>&1
rm -rf "$root/fq.run"; mkdir -p "$root/fq.run"
( cd "$root/fq.run" && vvp "$root/fq.vvp" +TESTNAME="${TEST:-smoke}" +SEED=1 +CYCLES=200 ) > "$root/fq.run/run.log" 2>&1
n=$(/usr/bin/grep -c 'ERROR' "$root/fq.run/run.log" 2>/dev/null)
[ "${n:-0}" -eq 0 ] 2>/dev/null && echo "RESULT: PASS" || echo "RESULT: FAIL"And a flow that stops:
#!/bin/bash
# flow_loud -- stops at the first unrecoverable problem and names it.
set -u
root=$(cd "$(dirname "$0")/.." && pwd)
die() { printf 'flow: %s\n' "$1" >&2; exit "$2"; }
step() { printf 'flow: %s\n' "$1" >&2; }
step "1/4 checking the environment"
command -v iverilog >/dev/null || die "iverilog is not on PATH" 3
step "2/4 compiling"
COMPILE_LOG=$root/lf.compile.log "$root/bin/compile" "$root/lf.vvp" ${SRC:-tb/tb_arb.v rtl/arb.v} >/dev/null 2>&1 \
|| die "the compile did not produce a usable image (see lf.compile.log)" 4
step "3/4 running"
rm -rf "$root/lf.run"; mkdir -p "$root/lf.run"
( cd "$root/lf.run" && exec vvp "$root/lf.vvp" +TESTNAME="${TEST:-smoke}" +SEED=1 +CYCLES=200 ) > "$root/lf.run/run.log" 2>&1
step "4/4 checking the log"
"$root/bin/logcheck" "$root/lf.run/run.log" || die "the run did not produce a usable result" 5
printf 'flow: OK\n' >&2A source with 256 compile errors — Chapter 34.1's case, where iverilog exits 0 — through both:
-- a 256-error source, through both flows --
loud : flow: the compile did not produce a usable image (see lf.compile.log)
rc=4
quiet: RESULT: PASS
rc=0
-- a good source --
loud : flow: OK
rc=0
quiet: RESULT: PASS2. What "Loudly" Consists Of
die() { printf 'flow: %s\n' "$1" >&2; exit "$2"; }
step() { printf 'flow: %s\n' "$1" >&2; }Four properties, each of which can be got wrong independently.
It stops. exit, not return, not a flag checked later. Chapter 20.2 covers set -e and its limits — and the limit that matters here is the one measured in Chapter 18.3: set -e is suspended for a whole function body when the function is called in a condition, with ||, or as v=$(f). So a flow that relies on set -e for its stopping has a stopping behaviour that depends on how its functions happen to be called. An explicit die does not.
It names the problem. the compile did not produce a usable image (see lf.compile.log) identifies the stage, the nature of the failure, and where the evidence is. Compare the common alternative:
Error: command failedwhich names none of the three and is the reason people grep flow source code at two in the morning.
It goes to standard error. Chapter 12.3's separation, and Chapter 33.4 measured what happens when a diagnostic lands in a run's log instead: it is attributed to the tool. step() also writes to stderr, deliberately, so that a flow's progress narration never contaminates a stage's output on stdout.
And it returns a status that says which stage. Not 1 for everything.
The progress narration earns its place
flow: 1/4 checking the environment
flow: 2/4 compiling
flow: 3/4 running
flow: 4/4 checking the log
flow: OKThat looks like decoration and is not. A flow that stops silently cannot be distinguished from a flow that hung, and on a long flow the last step line printed is the only thing that says where it was. Chapter 33.6's bash -x gives far more detail at 32× the volume; the n/4 narration is the version a human reads while it runs.
The n/4 form specifically — rather than bare labels — because it bounds the wait. "3/4" tells a reader there is one stage left; "running" tells them nothing about how much is ahead.
3. Which Failures Should Stop a Flow
Not all of them, and getting this wrong in either direction is expensive.
| failure | stop? | why |
|---|---|---|
a tool is not on PATH | stop | nothing downstream can work (33.4) |
| the environment names the wrong project | stop | every result would be about another design |
| the compile produced nothing | stop | there is nothing to run |
| a test failed | continue | that is the result, not an error |
| one run of two hundred timed out | continue | 199 results are still worth having |
| a run's log is truncated | continue | record it and carry on |
| the disk filled up | stop | every later run will fail the same way |
The organising principle: stop when continuing would produce results that mean nothing; continue when the failure is itself a result.
A failing test is a result. A missing simulator is not — it is the absence of the conditions under which results exist, and two hundred CRASH rows are not two hundred findings, as Chapter 32.8 measured when a cron environment had no vvp on its PATH.
4. Never Half-Succeeding
The failure this chapter's title names second is subtler than a false pass: a flow that partly works and reports as though it wholly did.
Three shapes, each measured earlier in this track.
A stage that produces some of its output. Chapter 34.2's truncated waveform keeps a valid header and $enddefinitions, so everything structural passes while the file is a third of its length. The defence is to validate the end, not the beginning.
A step that succeeds for a subset. A results file assembled from 200 runs where 7 directories are missing — Chapter 31.7 measured a disk scan reporting 7 runs found, 7 PASS, exit 0 on a matrix of eight. The defence is one row per spec, always.
And a flow that completes after a stage was skipped. The most dangerous, because nothing is wrong with any artifact:
# BAD: the || true that makes a flow "robust"
"$root/bin/checkenv" chip_a || true
"$root/bin/compile" "$out" $src || trueTwo || trues and the flow always reaches its report, which will be about whatever happened to be on disk. Chapter 33.1 measured the consequence: a stale image producing a byte-identical log to a real design failure.
|| true has exactly one legitimate use in a flow, which is a step whose failure genuinely does not affect the result:
"$root/bin/snapshot" "$d/machine" || echo "snapshot failed" >&2 # never fatalChapter 33.8's argument — a flow that refuses to run because it could not record the load average has turned a diagnostic into an outage. The distinction is whether the step's output is an input to the result or a note beside it.
5. The Library, and the Flow That Uses It
Four words, sourced rather than copied:
# flowlib.sh -- the four words every wrapper in this module needs.
# Source it; do not copy it. Nine copies of `die` diverge within a quarter.
# die MESSAGE STATUS -- stop, name the problem, and say which stage.
die() { printf '%s: %s\n' "${FLOW_NAME:-flow}" "$1" >&2; exit "${2:-1}"; }
# step N TOTAL MESSAGE -- progress a human can read while it runs.
# n/total rather than a bare label, so a reader knows how much is left.
step() { printf '%s: %s/%s %s\n' "${FLOW_NAME:-flow}" "$1" "$2" "$3" >&2; }
# note MESSAGE -- something worth recording that must never stop the flow.
note() { printf '%s: note: %s\n' "${FLOW_NAME:-flow}" "$1" >&2; }
# need TOOL... -- every tool must be reachable before any of them is used.
need() {
# `local t` matters: without it this loop clobbers a caller's $t, and a
# wrapper whose test name is in $t then runs a test called "vvp". Measured.
local miss="" t
for t in "$@"; do command -v "$t" >/dev/null 2>&1 || miss="$miss $t"; done
[ -z "$miss" ] || die "not on PATH:$miss" 3
}
# want FILE... -- every input must be readable before any work starts.
want() {
local f
for f in "$@"; do [ -r "$f" ] || die "cannot read '$f'" 3; done
}And the whole module composed:
#!/bin/bash
# flow TEST OUTDIR -- the whole of Module 34, composed.
set -u
root=$(cd "$(dirname "$0")/.." && pwd)
FLOW_NAME=flow; . "$root/bin/flowlib.sh"
t=${1:-smoke}; d=${2:-}
[ -n "$d" ] || die "usage: flow TEST OUTDIR" 2
# absolute BEFORE anything cds -- bounded runs the tool from inside $d/run,
# so a relative image path would resolve against that directory instead.
mkdir -p "$d" || die "cannot create '$d'" 3
d=$(cd "$d" && pwd)
step 1 5 "checking the environment" # 33.4
need iverilog vvp
want "$root/cfg/files.f" "$root/tb/tb_arb.v" "$root/rtl/arb.v"
step 2 5 "checking disk headroom" # 34.5
"$root/bin/spacecheck" "$d" "${NEED_MB:-10}" >/dev/null 2>&1 \
|| die "not enough space under '$d'" 3
step 3 5 "compiling" # 34.1
COMPILE_LOG=$d/compile.log "$root/bin/compile" "$d/sim.vvp" \
"$root/tb/tb_arb.v" "$root/rtl/arb.v" >/dev/null 2>&1 \
|| die "the compile produced no usable image (see $d/compile.log)" 4
step 4 5 "running" # 34.3 + 34.4 + 34.5
( ulimit -f "${MAX_RUN_BLOCKS:-5242880}"
"$root/bin/bounded" "${LIMIT_S:-30}" "$d/run" \
vvp "$d/sim.vvp" "+TESTNAME=$t" "+SEED=${SEED:-1}" "+CYCLES=${CYCLES:-200}" )
"$root/bin/spaceclass" "$d/run" >"$d/run/space.txt" 2>&1 || note "$(cat "$d/run/space.txt")"
step 5 5 "checking the log" # 34.2
"$root/bin/logcheck" "$d/run/run.log" || die "the run produced no usable result" 5
printf '%s: OK\n' "$FLOW_NAME" >&2Seven inputs through it:
CASE rc last line
a passing test 0 flow: OK
a failing test 5 flow: the run produced no usable result
a hanging test 5 flow: the run produced no usable result
a vacuous test 0 flow: OK
a missing tool 3 flow: not on PATH: iverilog vvp
no output dir 2 flow: usage: flow TEST OUTDIR
a 256-error source 4 flow: the compile produced no usable image (see …)Six of the seven are exactly right. The seventh is Section 6's subject and is not a bug in the flow.
Three properties of the library worth naming.
FLOW_NAME is a variable with a default, so every message is prefixed by the flow that produced it and a second flow sourcing the same library does not claim the first one's name.
need and want collect before reporting. not on PATH: iverilog vvp names both; a loop that died on the first would have required two runs to learn two facts — the same argument Chapter 26.5 makes for validating a filelist.
And note exists so that || true does not have to. A step whose failure must be recorded and must not stop anything gets a word of its own, which makes the two cases visibly different in the source:
"$root/bin/spaceclass" "$d/run" >"$d/run/space.txt" 2>&1 || note "$(cat "$d/run/space.txt")"6. The Checklist This Module Produces
Seven chapters, one wrapper each, in the order a flow executes them:
| stage | the check | chapter |
|---|---|---|
| before anything | the tools are on PATH; the project is the right one | 33.4 |
| before anything | there is disk headroom | 34.5 |
| build | three independent conditions, and delete the old artifact first | 34.1 |
| build | record the source digests, not the output's | 33.1 |
| each run | bound the time, and record that the bound fired | 34.4 |
| each run | bound the file size | 34.5 |
| each run | retry only what the tool's words call transient | 34.3 |
| each run | the log must exist, end properly, and state one verdict | 34.2 |
| each run | snapshot the machine, non-fatally | 33.8 |
| collect | one row per spec, always | 31.7 |
| report | three questions, and compare per spec | 32.5 |
| return | a status that says which stage | 32.7 |
Every row is a thing measured failing quietly. None is more than a few lines. And the reason the list is worth having as a list is that each check is individually easy to skip and the skips are invisible — a flow missing all twelve behaves identically to one with all twelve, on every input that works.
Practice
- Write the quiet flow and run it on a source that does not compile.
- Trace the five steps by which it arrives at a pass.
- Explain why each step is individually reasonable.
- Run both flows on a good source and compare the output.
- Explain why the quiet flow survives review.
- Write
dieandstepand give each a distinct status. - Replace an
exitwith a flag checked later and find the path that ignores it. - Call a
set -efunction with||and watch the-estop applying. - Write "Error: command failed" and then write the version that names three things.
- Send a diagnostic to stdout and see where it lands in a run's log.
- Add
n/4progress narration and then remove the counts. - Stop a flow silently and try to distinguish it from a hang.
- Classify seven failures into stop and continue.
- State the organising principle and test it on a case not in the table.
- Argue against "abort after N failures" using Chapter 34.6's measurement.
- Write the legitimate early-abort and give it a setup status.
- Find a truncated artifact that passes a structural check.
- Build a results file from a disk scan with runs missing, and read what it claims.
- Add
|| trueto two stages and work out what the report is then about. - Find the one step in your flow where
|| trueis correct, and say why. - Walk the twelve-row checklist against your own flow and count what is missing.
- Explain why a flow missing all twelve looks identical to one with all twelve.
- Count your vacuous passes and say why no check in this module finds them.
Check Your Understanding
- State this chapter's one-line thesis.
- What did each flow report on a 256-error source?
- What did each report on a good source, and why does that matter?
- Trace the five steps to
RESULT: PASS. - Why does step 4 find zero errors?
- What is the significance of rc=4 rather than rc=1?
- Name the four properties of failing loudly.
- Why is an explicit
diebetter than relying onset -e? - What three things should a message name?
- Why must diagnostics go to standard error?
- Why does
step()also write to stderr? - What does progress narration buy that a trace does not?
- Why
n/4rather than bare labels? - Which failures stop a flow, and which does it survive?
- State the organising principle.
- Why are two hundred
CRASHrows not two hundred findings? - What tension does fail-fast have with Chapter 34.6's finding?
- How does the stop/continue table resolve it?
- Why is "abort after N failures" the worst option?
- What is the one legitimate early abort, and what status does it get?
- Name the three shapes of half-succeeding.
- What makes the third the most dangerous?
- When is
|| truecorrect, and what is the test? - Why is the twelve-row checklist worth having as a list?
- What can this module not give you?
- What is the one failure a defensive flow will never find, and what should be printed anyway?
7. Remember
- A flow that cannot fail is a flow whose success means nothing.
- The quiet flow printed
RESULT: PASSfor a design that did not compile — through five individually reasonable steps. - It is identical to the loud flow on every working input, which is why it survives review and spreads.
- Failing loudly is four things: stop, name the problem, write to stderr, and return a status that says which stage.
- Use an explicit
die, notset -e—-eis suspended for a function called in a condition, with||, or in a substitution. - Name the stage, the failure and where the evidence is. "Error: command failed" names none of them.
- Narrate progress as
n/4on stderr. A silent stop is indistinguishable from a hang. - Stop when continuing would produce meaningless results; continue when the failure is the result. A failing test is a result; a missing simulator is not.
- Never "abort after N failures" — it yields a partial result set whose contents depend on scheduling.
- Half-succeeding has three shapes: partial output, a subset succeeded, and a skipped stage. The last is worst because every artifact looks fine.
|| trueis correct only for a note beside the result, never for an input to it.- Twelve checks, a few lines each — and a flow missing all of them looks identical to one with all of them.
- This module cannot tell you the verification is good. Print the count of passes with
checks=0anyway.
8. What Is Next
Module 34 has made a flow's answer trustworthy: it fails on what it should, survives what it should, bounds what could run forever, validates what it produced, and returns a status that names the stage.
Every wrapper in it is still a script one person wrote and now maintains, and there are twelve of them. Module 35, Reusable Flow Tooling and Hand-Off, is about that: a shared library instead of nine copies of the same function, configuration instead of hardcoded paths, honest boundaries with Make and Python, and the documentation and review that keep a flow alive after its author has moved to another project.
Where this fits
Part of the Linux & Bash curriculum.
