Skip to content
VLSI Mentor

Linux & Bash · Part II · Module 35 · Reusable Flow Tooling and Hand-Off

Reviewing Shell Automation

A review checklist read as a list teaches nothing, so this chapter supplies a thirty-line regression script that is clean under bash -n and reports three passes on a regression where one test does not compile and another prints an error. Ten defects, each proven by running it, and the checklist derived from them rather than asserted.

Advanced32 min readLinuxPart IICode ReviewChecklistsRegression

Part II · Module 35 · Chapter 35.7 · Reviewing Shell Automation

Here is a script submitted for review. It is thirty lines, it is clean under bash -n, and the defects in it are the ones this part has measured one at a time.

1. The Submission

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
#!/bin/bash
# submitted.sh -- run the regression. Submitted for review.
set -e
RUNDIR=runs
LOGDIR=$PWD/logs
TESTS=$(cat tests/list.txt)

run_one() {
  t=$1
  mkdir -p $RUNDIR/$t
  cd $RUNDIR/$t
  iverilog -o sim.vvp "$OLDPWD/rtl/$t.v" 2>&1 | tee $LOGDIR/$t.log
  if [ $? -ne 0 ]; then
    echo "$t: compile failed"
    return 1
  fi
  vvp sim.vvp > $LOGDIR/$t.log 2>&1
  n=$(grep -c FATAL $LOGDIR/$t.log)
  if [ $n -gt 0 ]; then
    echo "$t: FAIL ($n errors)"
    return 1
  fi
  echo "$t: PASS"
  cd "$OLDPWD"
}

rm -rf $RUNDIR/
for t in $TESTS; do
  run_one $t || true
done
echo "regression complete"

Three tests, and what they really do:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ iverilog -o g.vvp rtl/alu_smoke.v  && vvp g.vvp
  alu_smoke   runs: PASSED

$ iverilog -o g.vvp rtl/fifo_reset.v
  fifo_reset  COMPILE FAIL (rc 2)
      rtl/fifo_reset.v:1: syntax error
      I give up.

$ iverilog -o g.vvp rtl/axi_burst.v  && vvp g.vvp
  axi_burst   runs: ERROR: burst length mismatch

The honest verdict is 1 PASS, 2 FAIL. What the script reports:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ /bin/bash submitted.sh
alu_smoke: PASS
fifo_reset: PASS
axi_burst: PASS
regression complete
  exit status: 0

Three passes, including a test that does not compile. And:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ /bin/bash -n submitted.sh
  (clean)

So the review has to find, by reading, what no syntax check and no run will tell you — because the run says everything is fine.

2. Ten Defects, Each One Measured

D1 — $? after a pipe

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
iverilog -o sim.vvp "$OLDPWD/rtl/$t.v" 2>&1 | tee $LOGDIR/$t.log
if [ $? -ne 0 ]; then
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  after pipe:        $?=0
  without the pipe:  $?=2

$? is the last command's status, and the last command is tee, which succeeds whenever it can write its file. Chapter 12.8 measured this; here it is the single defect that turns a compile failure into a pass. Fix: drop the pipe and redirect, or read ${'{'}PIPESTATUS[0]{'}'} immediately.

D2 — set -e is inert

The script's first line is set -e, which reads as a safety net. It is not one:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ /bin/bash -c 'set -e
f(){ false; echo "reached after false"; return 0; }
f || true'
  reached after false (set -e did not fire)

A function invoked on the left of || runs with set -e suspended for its entire body. The script's loop is run_one $t || true, so every line of run_one is unprotected. Chapter 20.2 measured the rule; the review point is that set -e plus || true is a contradiction a reader has to notice, because neither line is wrong on its own.

D3 — a check whose pattern never matches

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
n=$(grep -c FATAL $LOGDIR/$t.log)
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  grep -c FATAL  -> n=0   (so the FAIL branch is never taken)
  grep -c ERROR  -> n=1
  files containing FATAL: 0  -> the branch is DEAD

The failing test prints ERROR. Nothing in the entire regression prints FATAL, so the FAIL branch cannot execute. Chapter 32.3 built the pattern list; the review question is not "is the pattern right" but "has this branch ever fired?" — a question with a mechanical answer: grep the corpus for the pattern.

D4 — the run log overwrites the compile log

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
iverilog … | tee $LOGDIR/$t.log      # compile output
vvp sim.vvp > $LOGDIR/$t.log 2>&1    # same file, truncating
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  before: compile output here
  after:  nosuch.vvp: Unable to open input file.

One name, two writers, > — so the only record of why compilation failed is destroyed by the next step, and what remains is a confusing message about a missing binary. Chapter 27.3 and Chapter 33.3 both depend on that log existing.

D5 and D6 — cd and $OLDPWD

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
cd $RUNDIR/$t
…
  return 1          # <-- no cd back
…
cd "$OLDPWD"        # only on the success path
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  after bad  cwd=/tmp/rv/bad
  after next cwd=/tmp/rv/bad

The failure path leaves the shell inside the run directory, so the next test's relative paths resolve from there. That is the ninth appearance of a relative path crossing a cd in this part.

And the attempted fix is itself wrong:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ cd /tmp; cd /var; cd /usr
  cwd=/usr  OLDPWD=/var  (not /tmp)

$OLDPWD is the previous directory, not the starting one. After the failed test left the shell in runs/bad, the next test's cd "$OLDPWD" returned it to runs/bad — not to the root. A variable that is usually right because the script usually does exactly one cd.

D7 — rm -rf on an unquoted variable

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
rm -rf $RUNDIR/
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ RUNDIR=runs ; echo rm -rf $RUNDIR/
  + echo rm -rf runs/
$ RUNDIR=     ; echo rm -rf $RUNDIR/
  + echo rm -rf /

An empty RUNDIR makes that line rm -rf /. The trailing slash, added for clarity, is what supplies the target. Chapter 3.3 measured the family; the review rule is narrow and absolute: a variable inside rm -rf must be quoted and must be refused when empty.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ set -u; RUNDIR=; echo rm -rf "${'${RUNDIR:?RUNDIR must be set}'}/"
  /bin/bash: RUNDIR: RUNDIR must be set

D8 — word splitting on the test list

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
TESTS=$(cat tests/list.txt)
for t in $TESTS; do
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ printf 'alu_smoke\nfifo reset\n' ; for t in $T; do echo "got [$t]"; done
  got [alu_smoke]
  got [fifo]
  got [reset]

Three iterations for two tests. Chapter 9.2 measured why; the fix is the loop Chapter 17.5 builds:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ while IFS= read -r t; do echo "read [$t]"; done < list.txt
  read [alu_smoke]
  read [fifo reset]

D9 — the trailing echo is the exit status

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
echo "regression complete"
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  with trailing echo: rc=0
  with rc captured:   rc=1

The fifth appearance of this mechanism in Part II, and the one that makes the other nine invisible to CI: whatever the loop concluded, the script's status is the status of the last echo.

D10 — a missing local

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
run_one() {
  t=$1
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  inside run: t=upper_a
  loop variable is now t=upper_a

t is global, so the function overwrites the caller's loop variable. Here it is assigned the same value, which is why it does no damage in this version — and Chapter 34.6 measured the version where it did, running a test named vvp.

3. The Checklist, Derived

Figure 35.7.1 — Where a flow script's status can be lost between the tool and CI

Six stages between a tool's exit status and the verdict CI reads, and where the status is lost at each. The tool returns 2 on a syntax error; a pipe replaces it with tee's 0; a function called on the left of || true runs with set -e suspended; a check greps for a pattern that never matches, and grep -c itself exits 1 when the log is clean; a trailing echo becomes the script's status; and CI reads 0 and goes green on two failed tests.01THE TOOL EXITSiverilog returns 2 on a syntax error02A PIPE| tee replaces it with tee's 0 (D1)03A FUNCTION|| true suspends set -e in the body (D2)04A CHECKa pattern that never matches; grep -c exits 1 when clean05A REPORTa trailing echo becomes the status (D9)06CIreads 0 and goes green on 2 failed tests
Each arrow is a place Part II measured a status being discarded. The reviewed script removes the pipe, captures the status, and makes the final exit the verdict.

Each item below is a question with a mechanical answer, and each cites the measurement it came from rather than a principle.

Status

#askbecause
R1Is any status read after a pipe?tee gave 0 where the tool gave 2 (D1)
R2Does set -e actually apply?f || true suspends it for the whole body (D2)
R9Is the last command a report?a trailing echo is the script's status (D9)
—Does the script's exit encode the verdict?[ "$fail" -eq 0 ] || exit 1

Checks

#askbecause
R3Has every branch of the verdict ever fired?the FATAL branch was dead; 0 files contain it
R12Is there a positive verdict check?absence of bad news is not success (31.7)
—Does grep -c run under set -e?it exits 1 on a clean log (20.2) — inverted behaviour
—Is the empty case a refusal?zero rows reports clean (32.7)

Paths and isolation

#askbecause
R4Is any relative path used across a cd?ninth occurrence in this part (D5)
R5Is there one absolute anchor, computed once?$OLDPWD is the previous directory (D6)
—Does each run own its directory and log?one name, two writers, > (D4)
R11Is every path expansion quoted?a space in a path splits the word

Data and cleanup

#askbecause
R7Is any variable inside rm -rf unquoted or possibly empty?rm -rf $RUNDIR/ became rm -rf / (D7)
R8Is a list iterated with for … in $VAR?3 iterations for 2 tests (D8)
R10Is every function variable local?a global t ran a test named vvp (34.6)
—Is anything deleted that cannot be rebuilt?28.1

And two questions about the review itself

askbecause
Do any two findings cancel?|| true was hiding the grep -c abort — fixing one alone makes it worse
Does the script report a verdict a human can check against ground truth?the submission said 3 PASS where the truth was 1

4. The Same Regression, Reviewed

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
#!/bin/bash
# reviewed.sh -- the same regression, after the review in Chapter 35.7.
# Each numbered comment is the review finding the line answers.
set -u
FLOW_NAME=regress
die() { printf '%s: %s\n' "$FLOW_NAME" "$1" >&2; exit 2; }

ROOT=$(cd "$(dirname "$0")" && pwd)          # R5: an absolute anchor, once
RUNDIR=${RUNDIR:-$ROOT/runs}
LOGDIR=${LOGDIR:-$ROOT/logs}
LIST=${LIST:-$ROOT/tests/list.txt}
[ -r "$LIST" ] || die "cannot read test list: $LIST"

run_one() {                                   # R10: every variable is local
  local t=$1 rd lg crc rrc n
  rd=$RUNDIR/$t; lg=$LOGDIR/$t
  mkdir -p "$rd" "$lg" || return 2            # R11: quoted
  iverilog -o "$rd/sim.vvp" "$ROOT/rtl/$t.v" > "$lg/compile.log" 2>&1
  crc=$?                                      # R1: no pipe, status captured
  if [ "$crc" -ne 0 ]; then
    printf '%s %-11s COMPILE_FAIL rc=%s %s\n' "$FLOW_NAME" "$t" "$crc" "$lg/compile.log"
    return 1
  fi
  [ -s "$rd/sim.vvp" ] || { printf '%s %-11s NO_BINARY\n' "$FLOW_NAME" "$t"; return 1; }
  ( cd "$rd" && vvp sim.vvp ) > "$lg/run.log" 2>&1   # R4+R5: subshell, own log
  rrc=$?
  n=$(/usr/bin/grep -c -E 'ERROR|FATAL|UVM_(ERROR|FATAL)' "$lg/run.log" || true)
  if [ "$rrc" -ne 0 ] || [ "$n" -gt 0 ]; then # R3: a pattern proven to match
    printf '%s %-11s FAIL rc=%s matches=%s %s\n' "$FLOW_NAME" "$t" "$rrc" "$n" "$lg/run.log"
    return 1
  fi
  if ! /usr/bin/grep -q PASSED "$lg/run.log"; then   # R12: a positive verdict
    printf '%s %-11s NO_VERDICT %s\n' "$FLOW_NAME" "$t" "$lg/run.log"
    return 1
  fi
  printf '%s %-11s PASS %s\n' "$FLOW_NAME" "$t" "$lg/run.log"
  return 0
}

case ${RUNDIR:-} in "") die "RUNDIR is empty";; esac   # R7: never rm an empty var
rm -rf -- "$RUNDIR"
pass=0; fail=0
while IFS= read -r t; do                      # R8: no word splitting
  [ -n "$t" ] || continue
  if run_one "$t"; then pass=$((pass+1)); else fail=$((fail+1)); fi
done < "$LIST"
printf '%s: %s pass, %s fail\n' "$FLOW_NAME" "$pass" "$fail"
[ "$fail" -eq 0 ] || exit 1                   # R9: the status is the verdict
exit 0
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ /bin/bash reviewed.sh
regress alu_smoke   PASS         LOGDIR/alu_smoke/run.log
regress fifo_reset  COMPILE_FAIL rc=2 LOGDIR/fifo_reset/compile.log
regress axi_burst   FAIL rc=0 matches=1 LOGDIR/axi_burst/run.log
regress silent      NO_VERDICT   LOGDIR/silent/run.log
regress aborts      FAIL rc=1 matches=1 LOGDIR/aborts/run.log
regress: 1 pass, 4 fail
  exit status: 1
testsubmitted.shreviewed.sh
alu_smokePASSPASS
fifo_resetPASS (does not compile)COMPILE_FAIL rc=2
axi_burstPASS (prints ERROR)FAIL matches=1
exit01

And the log the submission destroyed:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
$ head -3 logs/fifo_reset/compile.log
rtl/fifo_reset.v:1: syntax error
I give up.

5. How to Run the Review

Four passes, in this order, because each one finds things the next cannot.

Read the statuses first. Trace one failing path from the tool's exit to the script's exit and name every place the value could be replaced. On the submission that pass alone finds D1, D2 and D9 — the three defects that make the other seven invisible.

Then ask of every branch: has this fired? Not "is it correct" — fired. grep the corpus for the pattern, or add the fixture that provokes it. This found D3, the dead branch, and it is the only pass that finds a check which has never run.

Then run it against known ground truth. Three tests whose real verdicts you established by hand, including at least one that fails in each way the flow can fail. This is what turns "3 PASS" from plausible into provably wrong, and no amount of reading substitutes for it.

Then ask how the findings interact. || true was hiding the grep -c abort; removing it alone makes the script die on its passing tests. A list of findings is not a plan.

And what not to do: do not review for style. A script with inconsistent indentation and a correct status chain is a working script. Chapter 35.6's audit covers the documentation, bash -n covers the syntax, and the review's scarce attention belongs on the four passes above.

Practice

  1. Take a flow script you did not write and run bash -n on it, to confirm what that does not tell you.
  2. Establish ground truth for three tests by hand, including one that cannot compile.
  3. Run the script and compare its verdict against the truth.
  4. Trace one failing path from the tool's exit to the script's exit.
  5. Find every status read after a pipe and measure one of them both ways.
  6. Check whether set -e is suspended anywhere by || or &&.
  7. Check whether the last command is a report.
  8. List every branch of the verdict and ask of each whether it has fired.
  9. Grep the corpus for each pattern the script searches for.
  10. Find a check that cannot match and prove it.
  11. Look for two writers of one log file.
  12. Find a relative path used across a cd.
  13. Check whether $OLDPWD is used as "where I started", and measure what it actually holds.
  14. Find every rm with a variable in it and test it with the variable empty, using echo.
  15. Find every for … in $VAR and test it with a value containing a space.
  16. Check every function variable for local.
  17. Run grep -c on a clean log and read its status.
  18. Put that inside set -e and see which of pass and fail aborts.
  19. Ask whether any two findings cancel.
  20. Write the fix for the pair, and say why fixing one alone is worse.
  21. Add a positive verdict check and see how many more tests fail.
  22. Add a fixture that compiles, runs and says nothing, and classify it.
  23. Add a fixture that calls $fatal, and compare its signals with the one that prints ERROR.
  24. Provoke every branch of your reviewed script.
  25. Make the script's exit status the verdict.
  26. Keep a separate compile log and run log and confirm both survive a failure.
  27. Compute one absolute anchor and use it everywhere.
  28. Refuse an empty variable before any destructive command.
  29. Re-run against ground truth and confirm the verdict now matches.
  30. Write the four review passes in order, and say what each finds that the next cannot.

Check Your Understanding

  1. What was the submitted script's verdict, and what was the truth?
  2. Did bash -n find anything?
  3. Why is $? 0 after iverilog … | tee?
  4. What does iverilog return on a syntax error?
  5. Why is the script's set -e inert?
  6. Why is "set -e plus || true" hard to spot?
  7. Why could the FAIL branch never execute?
  8. What is the mechanical test for a dead branch?
  9. What destroyed the compile log, and why does that matter?
  10. What does the failure path leave the shell in?
  11. What does $OLDPWD actually hold?
  12. Why was cd "$OLDPWD" usually right?
  13. What does rm -rf $RUNDIR/ do when RUNDIR is empty?
  14. What supplies the target in that expansion?
  15. What is the narrow rule for a variable inside rm -rf?
  16. How many iterations did two tests produce, and why?
  17. What is the correct loop?
  18. Why is the trailing echo the defect that hides the others?
  19. What does a missing local do, and why was it harmless here?
  20. What does grep -c exit with on a clean log?
  21. Under set -e, which outcome aborts the regression?
  22. Why could that bug not fire in the submitted script?
  23. Why is a cancelling pair the most dangerous state?
  24. What is the difference between the two uses of || true?
  25. Name the four status questions in the checklist.
  26. What is the positive-verdict question, and where does it come from?
  27. Which branch does NO_VERDICT catch, and why is it a failure?
  28. Why did the reviewed run report 4 failures where ground truth said 2?
  29. What do axi_burst and aborts show about signals?
  30. Give the four review passes in order.
  31. What does the first pass find that makes the rest possible?
  32. What should a review not spend attention on?

6. Remember

  • Thirty lines, clean under bash -n, reporting 3 PASS where the truth was 1 PASS and 2 FAIL — including a test that does not compile.
  • $? after | tee was 0 where the tool gave 2. The pipe is the single defect that turns a compile failure into a pass.
  • set -e was inert, because run_one $t || true suspends it for the entire function body.
  • The FATAL branch was dead — nothing in the regression prints FATAL. Ask of every branch whether it has fired, and answer by grepping the corpus.
  • One log name, two writers, > — the run log destroyed the compile log, and with it the only explanation of the failure.
  • $OLDPWD is the previous directory, not the starting one, so the second test "returned" into the first test's failed directory.
  • rm -rf $RUNDIR/ with an empty RUNDIR is rm -rf /, and the trailing slash supplies the target. Quote it and refuse an empty value.
  • for t in $TESTS ran three iterations for two tests. Use while IFS= read -r.
  • A trailing echo is the script's exit status — the fifth instance in Part II, and the one that hides the rest from CI.
  • Under set -e, n=$(grep -c ERROR log) aborts on a CLEAN log and continues on a dirty one. Exactly inverted.
  • Two defects were cancelling. Fixing the || true alone makes the regression die on its passing tests — so ask how findings interact before recommending any of them.
  • Suppressing a status you measured is fine; suppressing a function's status so set -e stops applying is not.
  • The reviewed script reports 1 pass, 4 fail, exit 1, keeps both logs, and all six of its branches were made to fire — including NO_VERDICT for a test that compiles, runs, exits 0 and says nothing.
  • Four passes, in order: statuses, branch firing, ground truth, interaction. Not style.

7. What Is Next

A reviewed, documented, measured flow is a flow in its best possible state — on one day. Chapter 35.8 is about the two years afterwards: what breaks when the tool version, the directory layout and the design all change underneath it, how to tell a script that still works from one that merely still runs, and how to retire one that nobody should be using — which closes Module 35, Part II, and the Linux & Bash track.

Where this fits

Part of the Linux & Bash curriculum.