Linux & Bash · Part II · Module 35 · Reusable Flow Tooling and Hand-Off
Reviewing Shell Automation
A review checklist read as a list teaches nothing, so this chapter supplies a thirty-line regression script that is clean under bash -n and reports three passes on a regression where one test does not compile and another prints an error. Ten defects, each proven by running it, and the checklist derived from them rather than asserted.
Advanced32 min readLinuxPart IICode ReviewChecklistsRegression
Part II · Module 35 · Chapter 35.7 · Reviewing Shell Automation
Here is a script submitted for review. It is thirty lines, it is clean under bash -n, and the defects in it are the ones this part has measured one at a time.
1. The Submission
#!/bin/bash
# submitted.sh -- run the regression. Submitted for review.
set -e
RUNDIR=runs
LOGDIR=$PWD/logs
TESTS=$(cat tests/list.txt)
run_one() {
t=$1
mkdir -p $RUNDIR/$t
cd $RUNDIR/$t
iverilog -o sim.vvp "$OLDPWD/rtl/$t.v" 2>&1 | tee $LOGDIR/$t.log
if [ $? -ne 0 ]; then
echo "$t: compile failed"
return 1
fi
vvp sim.vvp > $LOGDIR/$t.log 2>&1
n=$(grep -c FATAL $LOGDIR/$t.log)
if [ $n -gt 0 ]; then
echo "$t: FAIL ($n errors)"
return 1
fi
echo "$t: PASS"
cd "$OLDPWD"
}
rm -rf $RUNDIR/
for t in $TESTS; do
run_one $t || true
done
echo "regression complete"Three tests, and what they really do:
$ iverilog -o g.vvp rtl/alu_smoke.v && vvp g.vvp
alu_smoke runs: PASSED
$ iverilog -o g.vvp rtl/fifo_reset.v
fifo_reset COMPILE FAIL (rc 2)
rtl/fifo_reset.v:1: syntax error
I give up.
$ iverilog -o g.vvp rtl/axi_burst.v && vvp g.vvp
axi_burst runs: ERROR: burst length mismatchThe honest verdict is 1 PASS, 2 FAIL. What the script reports:
$ /bin/bash submitted.sh
alu_smoke: PASS
fifo_reset: PASS
axi_burst: PASS
regression complete
exit status: 0Three passes, including a test that does not compile. And:
$ /bin/bash -n submitted.sh
(clean)So the review has to find, by reading, what no syntax check and no run will tell you — because the run says everything is fine.
2. Ten Defects, Each One Measured
D1 — $? after a pipe
iverilog -o sim.vvp "$OLDPWD/rtl/$t.v" 2>&1 | tee $LOGDIR/$t.log
if [ $? -ne 0 ]; then after pipe: $?=0
without the pipe: $?=2$? is the last command's status, and the last command is tee, which succeeds whenever it can write its file. Chapter 12.8 measured this; here it is the single defect that turns a compile failure into a pass. Fix: drop the pipe and redirect, or read ${'{'}PIPESTATUS[0]{'}'} immediately.
D2 — set -e is inert
The script's first line is set -e, which reads as a safety net. It is not one:
$ /bin/bash -c 'set -e
f(){ false; echo "reached after false"; return 0; }
f || true'
reached after false (set -e did not fire)A function invoked on the left of || runs with set -e suspended for its entire body. The script's loop is run_one $t || true, so every line of run_one is unprotected. Chapter 20.2 measured the rule; the review point is that set -e plus || true is a contradiction a reader has to notice, because neither line is wrong on its own.
D3 — a check whose pattern never matches
n=$(grep -c FATAL $LOGDIR/$t.log) grep -c FATAL -> n=0 (so the FAIL branch is never taken)
grep -c ERROR -> n=1
files containing FATAL: 0 -> the branch is DEADThe failing test prints ERROR. Nothing in the entire regression prints FATAL, so the FAIL branch cannot execute. Chapter 32.3 built the pattern list; the review question is not "is the pattern right" but "has this branch ever fired?" — a question with a mechanical answer: grep the corpus for the pattern.
D4 — the run log overwrites the compile log
iverilog … | tee $LOGDIR/$t.log # compile output
vvp sim.vvp > $LOGDIR/$t.log 2>&1 # same file, truncating before: compile output here
after: nosuch.vvp: Unable to open input file.One name, two writers, > — so the only record of why compilation failed is destroyed by the next step, and what remains is a confusing message about a missing binary. Chapter 27.3 and Chapter 33.3 both depend on that log existing.
D5 and D6 — cd and $OLDPWD
cd $RUNDIR/$t
…
return 1 # <-- no cd back
…
cd "$OLDPWD" # only on the success path after bad cwd=/tmp/rv/bad
after next cwd=/tmp/rv/badThe failure path leaves the shell inside the run directory, so the next test's relative paths resolve from there. That is the ninth appearance of a relative path crossing a cd in this part.
And the attempted fix is itself wrong:
$ cd /tmp; cd /var; cd /usr
cwd=/usr OLDPWD=/var (not /tmp)$OLDPWD is the previous directory, not the starting one. After the failed test left the shell in runs/bad, the next test's cd "$OLDPWD" returned it to runs/bad — not to the root. A variable that is usually right because the script usually does exactly one cd.
D7 — rm -rf on an unquoted variable
rm -rf $RUNDIR/$ RUNDIR=runs ; echo rm -rf $RUNDIR/
+ echo rm -rf runs/
$ RUNDIR= ; echo rm -rf $RUNDIR/
+ echo rm -rf /An empty RUNDIR makes that line rm -rf /. The trailing slash, added for clarity, is what supplies the target. Chapter 3.3 measured the family; the review rule is narrow and absolute: a variable inside rm -rf must be quoted and must be refused when empty.
$ set -u; RUNDIR=; echo rm -rf "${'${RUNDIR:?RUNDIR must be set}'}/"
/bin/bash: RUNDIR: RUNDIR must be setD8 — word splitting on the test list
TESTS=$(cat tests/list.txt)
for t in $TESTS; do$ printf 'alu_smoke\nfifo reset\n' ; for t in $T; do echo "got [$t]"; done
got [alu_smoke]
got [fifo]
got [reset]Three iterations for two tests. Chapter 9.2 measured why; the fix is the loop Chapter 17.5 builds:
$ while IFS= read -r t; do echo "read [$t]"; done < list.txt
read [alu_smoke]
read [fifo reset]D9 — the trailing echo is the exit status
echo "regression complete" with trailing echo: rc=0
with rc captured: rc=1The fifth appearance of this mechanism in Part II, and the one that makes the other nine invisible to CI: whatever the loop concluded, the script's status is the status of the last echo.
D10 — a missing local
run_one() {
t=$1 inside run: t=upper_a
loop variable is now t=upper_at is global, so the function overwrites the caller's loop variable. Here it is assigned the same value, which is why it does no damage in this version — and Chapter 34.6 measured the version where it did, running a test named vvp.
3. The Checklist, Derived
Figure 35.7.1 — Where a flow script's status can be lost between the tool and CI
Each item below is a question with a mechanical answer, and each cites the measurement it came from rather than a principle.
Status
| # | ask | because |
|---|---|---|
| R1 | Is any status read after a pipe? | tee gave 0 where the tool gave 2 (D1) |
| R2 | Does set -e actually apply? | f || true suspends it for the whole body (D2) |
| R9 | Is the last command a report? | a trailing echo is the script's status (D9) |
| — | Does the script's exit encode the verdict? | [ "$fail" -eq 0 ] || exit 1 |
Checks
| # | ask | because |
|---|---|---|
| R3 | Has every branch of the verdict ever fired? | the FATAL branch was dead; 0 files contain it |
| R12 | Is there a positive verdict check? | absence of bad news is not success (31.7) |
| — | Does grep -c run under set -e? | it exits 1 on a clean log (20.2) — inverted behaviour |
| — | Is the empty case a refusal? | zero rows reports clean (32.7) |
Paths and isolation
| # | ask | because |
|---|---|---|
| R4 | Is any relative path used across a cd? | ninth occurrence in this part (D5) |
| R5 | Is there one absolute anchor, computed once? | $OLDPWD is the previous directory (D6) |
| — | Does each run own its directory and log? | one name, two writers, > (D4) |
| R11 | Is every path expansion quoted? | a space in a path splits the word |
Data and cleanup
| # | ask | because |
|---|---|---|
| R7 | Is any variable inside rm -rf unquoted or possibly empty? | rm -rf $RUNDIR/ became rm -rf / (D7) |
| R8 | Is a list iterated with for … in $VAR? | 3 iterations for 2 tests (D8) |
| R10 | Is every function variable local? | a global t ran a test named vvp (34.6) |
| — | Is anything deleted that cannot be rebuilt? | 28.1 |
And two questions about the review itself
| ask | because |
|---|---|
| Do any two findings cancel? | || true was hiding the grep -c abort — fixing one alone makes it worse |
| Does the script report a verdict a human can check against ground truth? | the submission said 3 PASS where the truth was 1 |
4. The Same Regression, Reviewed
#!/bin/bash
# reviewed.sh -- the same regression, after the review in Chapter 35.7.
# Each numbered comment is the review finding the line answers.
set -u
FLOW_NAME=regress
die() { printf '%s: %s\n' "$FLOW_NAME" "$1" >&2; exit 2; }
ROOT=$(cd "$(dirname "$0")" && pwd) # R5: an absolute anchor, once
RUNDIR=${RUNDIR:-$ROOT/runs}
LOGDIR=${LOGDIR:-$ROOT/logs}
LIST=${LIST:-$ROOT/tests/list.txt}
[ -r "$LIST" ] || die "cannot read test list: $LIST"
run_one() { # R10: every variable is local
local t=$1 rd lg crc rrc n
rd=$RUNDIR/$t; lg=$LOGDIR/$t
mkdir -p "$rd" "$lg" || return 2 # R11: quoted
iverilog -o "$rd/sim.vvp" "$ROOT/rtl/$t.v" > "$lg/compile.log" 2>&1
crc=$? # R1: no pipe, status captured
if [ "$crc" -ne 0 ]; then
printf '%s %-11s COMPILE_FAIL rc=%s %s\n' "$FLOW_NAME" "$t" "$crc" "$lg/compile.log"
return 1
fi
[ -s "$rd/sim.vvp" ] || { printf '%s %-11s NO_BINARY\n' "$FLOW_NAME" "$t"; return 1; }
( cd "$rd" && vvp sim.vvp ) > "$lg/run.log" 2>&1 # R4+R5: subshell, own log
rrc=$?
n=$(/usr/bin/grep -c -E 'ERROR|FATAL|UVM_(ERROR|FATAL)' "$lg/run.log" || true)
if [ "$rrc" -ne 0 ] || [ "$n" -gt 0 ]; then # R3: a pattern proven to match
printf '%s %-11s FAIL rc=%s matches=%s %s\n' "$FLOW_NAME" "$t" "$rrc" "$n" "$lg/run.log"
return 1
fi
if ! /usr/bin/grep -q PASSED "$lg/run.log"; then # R12: a positive verdict
printf '%s %-11s NO_VERDICT %s\n' "$FLOW_NAME" "$t" "$lg/run.log"
return 1
fi
printf '%s %-11s PASS %s\n' "$FLOW_NAME" "$t" "$lg/run.log"
return 0
}
case ${RUNDIR:-} in "") die "RUNDIR is empty";; esac # R7: never rm an empty var
rm -rf -- "$RUNDIR"
pass=0; fail=0
while IFS= read -r t; do # R8: no word splitting
[ -n "$t" ] || continue
if run_one "$t"; then pass=$((pass+1)); else fail=$((fail+1)); fi
done < "$LIST"
printf '%s: %s pass, %s fail\n' "$FLOW_NAME" "$pass" "$fail"
[ "$fail" -eq 0 ] || exit 1 # R9: the status is the verdict
exit 0$ /bin/bash reviewed.sh
regress alu_smoke PASS LOGDIR/alu_smoke/run.log
regress fifo_reset COMPILE_FAIL rc=2 LOGDIR/fifo_reset/compile.log
regress axi_burst FAIL rc=0 matches=1 LOGDIR/axi_burst/run.log
regress silent NO_VERDICT LOGDIR/silent/run.log
regress aborts FAIL rc=1 matches=1 LOGDIR/aborts/run.log
regress: 1 pass, 4 fail
exit status: 1| test | submitted.sh | reviewed.sh |
|---|---|---|
alu_smoke | PASS | PASS |
fifo_reset | PASS (does not compile) | COMPILE_FAIL rc=2 |
axi_burst | PASS (prints ERROR) | FAIL matches=1 |
| exit | 0 | 1 |
And the log the submission destroyed:
$ head -3 logs/fifo_reset/compile.log
rtl/fifo_reset.v:1: syntax error
I give up.5. How to Run the Review
Four passes, in this order, because each one finds things the next cannot.
Read the statuses first. Trace one failing path from the tool's exit to the script's exit and name every place the value could be replaced. On the submission that pass alone finds D1, D2 and D9 — the three defects that make the other seven invisible.
Then ask of every branch: has this fired? Not "is it correct" — fired. grep the corpus for the pattern, or add the fixture that provokes it. This found D3, the dead branch, and it is the only pass that finds a check which has never run.
Then run it against known ground truth. Three tests whose real verdicts you established by hand, including at least one that fails in each way the flow can fail. This is what turns "3 PASS" from plausible into provably wrong, and no amount of reading substitutes for it.
Then ask how the findings interact. || true was hiding the grep -c abort; removing it alone makes the script die on its passing tests. A list of findings is not a plan.
And what not to do: do not review for style. A script with inconsistent indentation and a correct status chain is a working script. Chapter 35.6's audit covers the documentation, bash -n covers the syntax, and the review's scarce attention belongs on the four passes above.
Practice
- Take a flow script you did not write and run
bash -non it, to confirm what that does not tell you. - Establish ground truth for three tests by hand, including one that cannot compile.
- Run the script and compare its verdict against the truth.
- Trace one failing path from the tool's exit to the script's exit.
- Find every status read after a pipe and measure one of them both ways.
- Check whether
set -eis suspended anywhere by||or&&. - Check whether the last command is a report.
- List every branch of the verdict and ask of each whether it has fired.
- Grep the corpus for each pattern the script searches for.
- Find a check that cannot match and prove it.
- Look for two writers of one log file.
- Find a relative path used across a
cd. - Check whether
$OLDPWDis used as "where I started", and measure what it actually holds. - Find every
rmwith a variable in it and test it with the variable empty, usingecho. - Find every
for … in $VARand test it with a value containing a space. - Check every function variable for
local. - Run
grep -con a clean log and read its status. - Put that inside
set -eand see which of pass and fail aborts. - Ask whether any two findings cancel.
- Write the fix for the pair, and say why fixing one alone is worse.
- Add a positive verdict check and see how many more tests fail.
- Add a fixture that compiles, runs and says nothing, and classify it.
- Add a fixture that calls
$fatal, and compare its signals with the one that printsERROR. - Provoke every branch of your reviewed script.
- Make the script's exit status the verdict.
- Keep a separate compile log and run log and confirm both survive a failure.
- Compute one absolute anchor and use it everywhere.
- Refuse an empty variable before any destructive command.
- Re-run against ground truth and confirm the verdict now matches.
- Write the four review passes in order, and say what each finds that the next cannot.
Check Your Understanding
- What was the submitted script's verdict, and what was the truth?
- Did
bash -nfind anything? - Why is
$?0 afteriverilog … | tee? - What does
iverilogreturn on a syntax error? - Why is the script's
set -einert? - Why is "
set -eplus|| true" hard to spot? - Why could the
FAILbranch never execute? - What is the mechanical test for a dead branch?
- What destroyed the compile log, and why does that matter?
- What does the failure path leave the shell in?
- What does
$OLDPWDactually hold? - Why was
cd "$OLDPWD"usually right? - What does
rm -rf $RUNDIR/do whenRUNDIRis empty? - What supplies the target in that expansion?
- What is the narrow rule for a variable inside
rm -rf? - How many iterations did two tests produce, and why?
- What is the correct loop?
- Why is the trailing
echothe defect that hides the others? - What does a missing
localdo, and why was it harmless here? - What does
grep -cexit with on a clean log? - Under
set -e, which outcome aborts the regression? - Why could that bug not fire in the submitted script?
- Why is a cancelling pair the most dangerous state?
- What is the difference between the two uses of
|| true? - Name the four status questions in the checklist.
- What is the positive-verdict question, and where does it come from?
- Which branch does
NO_VERDICTcatch, and why is it a failure? - Why did the reviewed run report 4 failures where ground truth said 2?
- What do
axi_burstandabortsshow about signals? - Give the four review passes in order.
- What does the first pass find that makes the rest possible?
- What should a review not spend attention on?
6. Remember
- Thirty lines, clean under
bash -n, reporting 3 PASS where the truth was 1 PASS and 2 FAIL — including a test that does not compile. $?after| teewas 0 where the tool gave 2. The pipe is the single defect that turns a compile failure into a pass.set -ewas inert, becauserun_one $t || truesuspends it for the entire function body.- The
FATALbranch was dead — nothing in the regression printsFATAL. Ask of every branch whether it has fired, and answer by grepping the corpus. - One log name, two writers,
>— the run log destroyed the compile log, and with it the only explanation of the failure. $OLDPWDis the previous directory, not the starting one, so the second test "returned" into the first test's failed directory.rm -rf $RUNDIR/with an emptyRUNDIRisrm -rf /, and the trailing slash supplies the target. Quote it and refuse an empty value.for t in $TESTSran three iterations for two tests. Usewhile IFS= read -r.- A trailing
echois the script's exit status — the fifth instance in Part II, and the one that hides the rest from CI. - Under
set -e,n=$(grep -c ERROR log)aborts on a CLEAN log and continues on a dirty one. Exactly inverted. - Two defects were cancelling. Fixing the
|| truealone makes the regression die on its passing tests — so ask how findings interact before recommending any of them. - Suppressing a status you measured is fine; suppressing a function's status so
set -estops applying is not. - The reviewed script reports
1 pass, 4 fail, exit 1, keeps both logs, and all six of its branches were made to fire — includingNO_VERDICTfor a test that compiles, runs, exits 0 and says nothing. - Four passes, in order: statuses, branch firing, ground truth, interaction. Not style.
7. What Is Next
A reviewed, documented, measured flow is a flow in its best possible state — on one day. Chapter 35.8 is about the two years afterwards: what breaks when the tool version, the directory layout and the design all change underneath it, how to tell a script that still works from one that merely still runs, and how to retire one that nobody should be using — which closes Module 35, Part II, and the Linux & Bash track.
Where this fits
Part of the Linux & Bash curriculum.
