Skip to content
VLSI Mentor

Linux & Bash · Part II · Module 34 · Defensive Automation Around EDA Tools

Failing Loudly, and Never Half-Succeeding

Every habit in this module reduces to one: stop at the first unrecoverable problem and say which one it was. This chapter measures two flows on a source that does not compile — one reports the problem and returns a status naming the stage, the other prints RESULT: PASS — and then works out which failures should stop a flow and which it must survive.

Advanced25 min readLinuxPart IIEDAFail FastDefensive

Part II · Module 34 · Chapter 34.7 · Failing Loudly

Six chapters of this module have been about detecting one thing each. This one is about what a flow does once it has detected something, and it is the chapter the other six exist to serve:

a flow that cannot fail is a flow whose success means nothing.

1. Two Flows, One Broken Source

The flow most flows are:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
#!/bin/bash
# flow_quiet -- the flow that carries on, as most of them do
root=$(cd "$(dirname "$0")/.." && pwd)
iverilog -o "$root/fq.vvp" ${SRC:-tb/tb_arb.v rtl/arb.v} > "$root/fq.compile.log" 2>&1
rm -rf "$root/fq.run"; mkdir -p "$root/fq.run"
( cd "$root/fq.run" && vvp "$root/fq.vvp" +TESTNAME="${TEST:-smoke}" +SEED=1 +CYCLES=200 ) > "$root/fq.run/run.log" 2>&1
n=$(/usr/bin/grep -c 'ERROR' "$root/fq.run/run.log" 2>/dev/null)
[ "${n:-0}" -eq 0 ] 2>/dev/null && echo "RESULT: PASS" || echo "RESULT: FAIL"

And a flow that stops:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
#!/bin/bash
# flow_loud -- stops at the first unrecoverable problem and names it.
set -u
root=$(cd "$(dirname "$0")/.." && pwd)
die() { printf 'flow: %s\n' "$1" >&2; exit "$2"; }
step() { printf 'flow: %s\n' "$1" >&2; }
step "1/4 checking the environment"
command -v iverilog >/dev/null || die "iverilog is not on PATH" 3
step "2/4 compiling"
COMPILE_LOG=$root/lf.compile.log "$root/bin/compile" "$root/lf.vvp" ${SRC:-tb/tb_arb.v rtl/arb.v} >/dev/null 2>&1 \
  || die "the compile did not produce a usable image (see lf.compile.log)" 4
step "3/4 running"
rm -rf "$root/lf.run"; mkdir -p "$root/lf.run"
( cd "$root/lf.run" && exec vvp "$root/lf.vvp" +TESTNAME="${TEST:-smoke}" +SEED=1 +CYCLES=200 ) > "$root/lf.run/run.log" 2>&1
step "4/4 checking the log"
"$root/bin/logcheck" "$root/lf.run/run.log" || die "the run did not produce a usable result" 5
printf 'flow: OK\n' >&2

A source with 256 compile errors — Chapter 34.1's case, where iverilog exits 0 — through both:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  -- a 256-error source, through both flows --
    loud : flow: the compile did not produce a usable image (see lf.compile.log)
           rc=4
    quiet: RESULT: PASS
           rc=0
  -- a good source --
    loud : flow: OK
           rc=0
    quiet: RESULT: PASS
The chain to a false pass: the compiler exits zero with 256 errors, the flow does not check because the status said success, the simulator is handed a missing file and fails into a log whose status nobody reads, a grep for ERROR finds zero because the message says Unable to open input file, and zero errors is reported as a pass.256 errors, exit 0no artifactFlow does not checkthe status said OKSimulator failsinto an unread loggrep ERROR finds 0the message differsRESULT: PASSexit 0The loud flownames it, exit 412
Figure 1 — the five steps by which a flow that trusts exit statuses arrives at RESULT: PASS for a design that does not compile. Every step is individually reasonable, and the only unusual thing about the outcome is that somebody looked.

2. What "Loudly" Consists Of

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
die() { printf 'flow: %s\n' "$1" >&2; exit "$2"; }
step() { printf 'flow: %s\n' "$1" >&2; }

Four properties, each of which can be got wrong independently.

It stops. exit, not return, not a flag checked later. Chapter 20.2 covers set -e and its limits — and the limit that matters here is the one measured in Chapter 18.3: set -e is suspended for a whole function body when the function is called in a condition, with ||, or as v=$(f). So a flow that relies on set -e for its stopping has a stopping behaviour that depends on how its functions happen to be called. An explicit die does not.

It names the problem. the compile did not produce a usable image (see lf.compile.log) identifies the stage, the nature of the failure, and where the evidence is. Compare the common alternative:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
Error: command failed

which names none of the three and is the reason people grep flow source code at two in the morning.

It goes to standard error. Chapter 12.3's separation, and Chapter 33.4 measured what happens when a diagnostic lands in a run's log instead: it is attributed to the tool. step() also writes to stderr, deliberately, so that a flow's progress narration never contaminates a stage's output on stdout.

And it returns a status that says which stage. Not 1 for everything.

The progress narration earns its place

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
flow: 1/4 checking the environment
flow: 2/4 compiling
flow: 3/4 running
flow: 4/4 checking the log
flow: OK

That looks like decoration and is not. A flow that stops silently cannot be distinguished from a flow that hung, and on a long flow the last step line printed is the only thing that says where it was. Chapter 33.6's bash -x gives far more detail at 32× the volume; the n/4 narration is the version a human reads while it runs.

The n/4 form specifically — rather than bare labels — because it bounds the wait. "3/4" tells a reader there is one stage left; "running" tells them nothing about how much is ahead.

3. Which Failures Should Stop a Flow

Not all of them, and getting this wrong in either direction is expensive.

failurestop?why
a tool is not on PATHstopnothing downstream can work (33.4)
the environment names the wrong projectstopevery result would be about another design
the compile produced nothingstopthere is nothing to run
a test failedcontinuethat is the result, not an error
one run of two hundred timed outcontinue199 results are still worth having
a run's log is truncatedcontinuerecord it and carry on
the disk filled upstopevery later run will fail the same way

The organising principle: stop when continuing would produce results that mean nothing; continue when the failure is itself a result.

A failing test is a result. A missing simulator is not — it is the absence of the conditions under which results exist, and two hundred CRASH rows are not two hundred findings, as Chapter 32.8 measured when a cron environment had no vvp on its PATH.

4. Never Half-Succeeding

The failure this chapter's title names second is subtler than a false pass: a flow that partly works and reports as though it wholly did.

Three shapes, each measured earlier in this track.

A stage that produces some of its output. Chapter 34.2's truncated waveform keeps a valid header and $enddefinitions, so everything structural passes while the file is a third of its length. The defence is to validate the end, not the beginning.

A step that succeeds for a subset. A results file assembled from 200 runs where 7 directories are missing — Chapter 31.7 measured a disk scan reporting 7 runs found, 7 PASS, exit 0 on a matrix of eight. The defence is one row per spec, always.

And a flow that completes after a stage was skipped. The most dangerous, because nothing is wrong with any artifact:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
# BAD: the || true that makes a flow "robust"
"$root/bin/checkenv" chip_a || true
"$root/bin/compile" "$out" $src || true

Two || trues and the flow always reaches its report, which will be about whatever happened to be on disk. Chapter 33.1 measured the consequence: a stale image producing a byte-identical log to a real design failure.

|| true has exactly one legitimate use in a flow, which is a step whose failure genuinely does not affect the result:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
"$root/bin/snapshot" "$d/machine" || echo "snapshot failed" >&2   # never fatal

Chapter 33.8's argument — a flow that refuses to run because it could not record the load average has turned a diagnostic into an outage. The distinction is whether the step's output is an input to the result or a note beside it.

5. The Library, and the Flow That Uses It

Four words, sourced rather than copied:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
# flowlib.sh -- the four words every wrapper in this module needs.
# Source it; do not copy it. Nine copies of `die` diverge within a quarter.

# die MESSAGE STATUS -- stop, name the problem, and say which stage.
die()  { printf '%s: %s\n' "${FLOW_NAME:-flow}" "$1" >&2; exit "${2:-1}"; }

# step N TOTAL MESSAGE -- progress a human can read while it runs.
# n/total rather than a bare label, so a reader knows how much is left.
step() { printf '%s: %s/%s %s\n' "${FLOW_NAME:-flow}" "$1" "$2" "$3" >&2; }

# note MESSAGE -- something worth recording that must never stop the flow.
note() { printf '%s: note: %s\n' "${FLOW_NAME:-flow}" "$1" >&2; }

# need TOOL... -- every tool must be reachable before any of them is used.
need() {
  # `local t` matters: without it this loop clobbers a caller's $t, and a
  # wrapper whose test name is in $t then runs a test called "vvp". Measured.
  local miss="" t
  for t in "$@"; do command -v "$t" >/dev/null 2>&1 || miss="$miss $t"; done
  [ -z "$miss" ] || die "not on PATH:$miss" 3
}

# want FILE... -- every input must be readable before any work starts.
want() {
  local f
  for f in "$@"; do [ -r "$f" ] || die "cannot read '$f'" 3; done
}

And the whole module composed:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
#!/bin/bash
# flow TEST OUTDIR -- the whole of Module 34, composed.
set -u
root=$(cd "$(dirname "$0")/.." && pwd)
FLOW_NAME=flow; . "$root/bin/flowlib.sh"
t=${1:-smoke}; d=${2:-}
[ -n "$d" ] || die "usage: flow TEST OUTDIR" 2
# absolute BEFORE anything cds -- bounded runs the tool from inside $d/run,
# so a relative image path would resolve against that directory instead.
mkdir -p "$d" || die "cannot create '$d'" 3
d=$(cd "$d" && pwd)

step 1 5 "checking the environment"                                  # 33.4
need iverilog vvp
want "$root/cfg/files.f" "$root/tb/tb_arb.v" "$root/rtl/arb.v"

step 2 5 "checking disk headroom"                                    # 34.5
"$root/bin/spacecheck" "$d" "${NEED_MB:-10}" >/dev/null 2>&1 \
  || die "not enough space under '$d'" 3

step 3 5 "compiling"                                                 # 34.1
COMPILE_LOG=$d/compile.log "$root/bin/compile" "$d/sim.vvp" \
    "$root/tb/tb_arb.v" "$root/rtl/arb.v" >/dev/null 2>&1 \
  || die "the compile produced no usable image (see $d/compile.log)" 4

step 4 5 "running"                                                   # 34.3 + 34.4 + 34.5
( ulimit -f "${MAX_RUN_BLOCKS:-5242880}"
  "$root/bin/bounded" "${LIMIT_S:-30}" "$d/run" \
      vvp "$d/sim.vvp" "+TESTNAME=$t" "+SEED=${SEED:-1}" "+CYCLES=${CYCLES:-200}" )
"$root/bin/spaceclass" "$d/run" >"$d/run/space.txt" 2>&1 || note "$(cat "$d/run/space.txt")"

step 5 5 "checking the log"                                          # 34.2
"$root/bin/logcheck" "$d/run/run.log" || die "the run produced no usable result" 5
printf '%s: OK\n' "$FLOW_NAME" >&2

Seven inputs through it:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  CASE                     rc   last line
  a passing test           0    flow: OK
  a failing test           5    flow: the run produced no usable result
  a hanging test           5    flow: the run produced no usable result
  a vacuous test           0    flow: OK
  a missing tool           3    flow: not on PATH: iverilog vvp
  no output dir            2    flow: usage: flow TEST OUTDIR
  a 256-error source       4    flow: the compile produced no usable image (see …)

Six of the seven are exactly right. The seventh is Section 6's subject and is not a bug in the flow.

Three properties of the library worth naming.

FLOW_NAME is a variable with a default, so every message is prefixed by the flow that produced it and a second flow sourcing the same library does not claim the first one's name.

need and want collect before reporting. not on PATH: iverilog vvp names both; a loop that died on the first would have required two runs to learn two facts — the same argument Chapter 26.5 makes for validating a filelist.

And note exists so that || true does not have to. A step whose failure must be recorded and must not stop anything gets a word of its own, which makes the two cases visibly different in the source:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
"$root/bin/spaceclass" "$d/run" >"$d/run/space.txt" 2>&1 || note "$(cat "$d/run/space.txt")"

6. The Checklist This Module Produces

Seven chapters, one wrapper each, in the order a flow executes them:

stagethe checkchapter
before anythingthe tools are on PATH; the project is the right one33.4
before anythingthere is disk headroom34.5
buildthree independent conditions, and delete the old artifact first34.1
buildrecord the source digests, not the output's33.1
each runbound the time, and record that the bound fired34.4
each runbound the file size34.5
each runretry only what the tool's words call transient34.3
each runthe log must exist, end properly, and state one verdict34.2
each runsnapshot the machine, non-fatally33.8
collectone row per spec, always31.7
reportthree questions, and compare per spec32.5
returna status that says which stage32.7

Every row is a thing measured failing quietly. None is more than a few lines. And the reason the list is worth having as a list is that each check is individually easy to skip and the skips are invisible — a flow missing all twelve behaves identically to one with all twelve, on every input that works.

Practice

  1. Write the quiet flow and run it on a source that does not compile.
  2. Trace the five steps by which it arrives at a pass.
  3. Explain why each step is individually reasonable.
  4. Run both flows on a good source and compare the output.
  5. Explain why the quiet flow survives review.
  6. Write die and step and give each a distinct status.
  7. Replace an exit with a flag checked later and find the path that ignores it.
  8. Call a set -e function with || and watch the -e stop applying.
  9. Write "Error: command failed" and then write the version that names three things.
  10. Send a diagnostic to stdout and see where it lands in a run's log.
  11. Add n/4 progress narration and then remove the counts.
  12. Stop a flow silently and try to distinguish it from a hang.
  13. Classify seven failures into stop and continue.
  14. State the organising principle and test it on a case not in the table.
  15. Argue against "abort after N failures" using Chapter 34.6's measurement.
  16. Write the legitimate early-abort and give it a setup status.
  17. Find a truncated artifact that passes a structural check.
  18. Build a results file from a disk scan with runs missing, and read what it claims.
  19. Add || true to two stages and work out what the report is then about.
  20. Find the one step in your flow where || true is correct, and say why.
  21. Walk the twelve-row checklist against your own flow and count what is missing.
  22. Explain why a flow missing all twelve looks identical to one with all twelve.
  23. Count your vacuous passes and say why no check in this module finds them.

Check Your Understanding

  1. State this chapter's one-line thesis.
  2. What did each flow report on a 256-error source?
  3. What did each report on a good source, and why does that matter?
  4. Trace the five steps to RESULT: PASS.
  5. Why does step 4 find zero errors?
  6. What is the significance of rc=4 rather than rc=1?
  7. Name the four properties of failing loudly.
  8. Why is an explicit die better than relying on set -e?
  9. What three things should a message name?
  10. Why must diagnostics go to standard error?
  11. Why does step() also write to stderr?
  12. What does progress narration buy that a trace does not?
  13. Why n/4 rather than bare labels?
  14. Which failures stop a flow, and which does it survive?
  15. State the organising principle.
  16. Why are two hundred CRASH rows not two hundred findings?
  17. What tension does fail-fast have with Chapter 34.6's finding?
  18. How does the stop/continue table resolve it?
  19. Why is "abort after N failures" the worst option?
  20. What is the one legitimate early abort, and what status does it get?
  21. Name the three shapes of half-succeeding.
  22. What makes the third the most dangerous?
  23. When is || true correct, and what is the test?
  24. Why is the twelve-row checklist worth having as a list?
  25. What can this module not give you?
  26. What is the one failure a defensive flow will never find, and what should be printed anyway?

7. Remember

  • A flow that cannot fail is a flow whose success means nothing.
  • The quiet flow printed RESULT: PASS for a design that did not compile — through five individually reasonable steps.
  • It is identical to the loud flow on every working input, which is why it survives review and spreads.
  • Failing loudly is four things: stop, name the problem, write to stderr, and return a status that says which stage.
  • Use an explicit die, not set -e — -e is suspended for a function called in a condition, with ||, or in a substitution.
  • Name the stage, the failure and where the evidence is. "Error: command failed" names none of them.
  • Narrate progress as n/4 on stderr. A silent stop is indistinguishable from a hang.
  • Stop when continuing would produce meaningless results; continue when the failure is the result. A failing test is a result; a missing simulator is not.
  • Never "abort after N failures" — it yields a partial result set whose contents depend on scheduling.
  • Half-succeeding has three shapes: partial output, a subset succeeded, and a skipped stage. The last is worst because every artifact looks fine.
  • || true is correct only for a note beside the result, never for an input to it.
  • Twelve checks, a few lines each — and a flow missing all of them looks identical to one with all of them.
  • This module cannot tell you the verification is good. Print the count of passes with checks=0 anyway.

8. What Is Next

Module 34 has made a flow's answer trustworthy: it fails on what it should, survives what it should, bounds what could run forever, validates what it produced, and returns a status that names the stage.

Every wrapper in it is still a script one person wrote and now maintains, and there are twelve of them. Module 35, Reusable Flow Tooling and Hand-Off, is about that: a shared library instead of nine copies of the same function, configuration instead of hardcoded paths, honest boundaries with Make and Python, and the documentation and review that keep a flow alive after its author has moved to another project.

Where this fits

Part of the Linux & Bash curriculum.