Maintaining a Flow Over a Project's Life

Part II · Module 35 · Chapter 35.8 · Maintaining a Flow

Every chapter of Part II so far has been about a flow on the day it was written. This one is about the two years afterwards, when nobody changes the script and it stops being correct anyway.

1. Month One and Month Eight

A small regression driver, of the shape Chapter 32.2 built:

while IFS= read -r t; do
  [ -n "$t" ] || continue
  n=$((n+1))
  iverilog -o /tmp/m.$t.vvp "rtl/$t.v" >/dev/null 2>&1 || continue
  vvp /tmp/m.$t.vvp 2>&1 | /usr/bin/grep -q PASSED && p=$((p+1))
done < tests/list.txt
printf 'flow: %s of %s passed\n' "$p" "$n"
[ "$p" -eq "$n" ] || exit 1

Month one, three blocks in rtl/ and three in the list:

$ /bin/bash flow.sh
flow: 3 of 3 passed
  rc=0

Month eight. Two new blocks have landed in rtl/. Nobody touched the script, and nobody updated tests/list.txt:

$ /bin/bash flow.sh
flow: 3 of 3 passed
  rc=0
  rtl/*.v: 5   tests/list.txt: 3

The same line, the same status, and two of five blocks are no longer tested. The new blocks print ERROR: not implemented, so they would have failed — but they are not in the list, so the flow does not know they exist.

This is the decay that matters, and its signature is precise: the denominator moved and the summary did not say so. 3 of 3 is a true statement about the list and a worthless one about the design. A reader checking the nightly e-mail sees the same text they saw in month one, which is the strongest possible signal that nothing has changed.

2. Checking Scope, Not Status

The flow's status answers "did the tests I ran pass?". Nothing answers "am I running the right tests?". That needs a second script, comparing what exists against what the flow does:

#!/bin/bash
# scopecheck.sh -- is the flow still testing everything it is supposed to?
# Compares what exists against what the flow runs. Exits 1 on a gap.
set -u
have=$(ls rtl/*.v 2>/dev/null | sed 's|rtl/||; s|\.v$||' | sort)
run=$(/usr/bin/grep -v '^$' tests/list.txt | sort)
gap=$(comm -23 <(printf '%s\n' "$have") <(printf '%s\n' "$run"))
extra=$(comm -13 <(printf '%s\n' "$have") <(printf '%s\n' "$run"))
[ -z "$gap" ]   || { printf 'scopecheck: untested blocks:%s\n' "$(printf ' %s' $gap)" >&2; }
[ -z "$extra" ] || { printf 'scopecheck: listed but absent:%s\n' "$(printf ' %s' $extra)" >&2; }
[ -z "$gap$extra" ] || exit 1
printf 'scopecheck: %s blocks, all listed\n' "$(printf '%s\n' "$have" | /usr/bin/grep -c .)"

$ /bin/bash scopecheck.sh
scopecheck: untested blocks: ddr pcie
  rc=1

Proven able to fail in both directions, which matters because they have different causes:

comm -23 and comm -13 on two sorted lists (Chapter 13.3) give both halves from one comparison. The second direction is the one that catches a retired block still being listed — a test that silently passes because its file is gone and the || continue skips it, which is how a deleted block keeps contributing to a pass count.

The general rule, and it applies to anything a flow enumerates:

If a script reads a list, something must check the list against reality.

Filelists (Chapter 26.1), test lists, corner lists, library paths. The list is a statement about the world made at the moment somebody typed it.

3. Stamping the Environment

The second invisible change is the toolchain. Chapter 33.8 measured why a result cannot be compared across environments; the maintenance version is to record the environment with every run so the comparison is possible later:

#!/bin/bash
# envstamp.sh -- record the parts of the environment that change a result.
# Printed into every run directory; compared against the last known-good one.
set -u
printf 'tool.iverilog\t%s\n' "$(iverilog -V 2>&1 | head -1)"
printf 'tool.vvp\t%s\n'      "$(vvp -V 2>&1 | head -1)"
printf 'tool.tclsh\t%s\n'    "$(echo 'puts [info patchlevel]' | tclsh 2>&1)"
printf 'tool.bash\t%s\n'     "$BASH_VERSION"
printf 'path.iverilog\t%s\n' "$(command -v iverilog)"
printf 'rtl.digest\t%s\n'    "$(cat rtl/*.v 2>/dev/null | /sbin/md5 -q)"
printf 'list.digest\t%s\n'   "$(/sbin/md5 -q tests/list.txt)"

$ /bin/bash envstamp.sh
tool.iverilog   Icarus Verilog version 13.0 (stable) (v13_0)
tool.vvp        Icarus Verilog runtime version 13.0 (stable) (v13_0)
tool.tclsh      8.5.9
tool.bash       3.2.57(1)-release
path.iverilog   /opt/homebrew/bin/iverilog
rtl.digest      971f79a3ea7e130a0d654cd958ada589
list.digest     be4ebd03af26f5258252ebc4869f2f51

Against a stamp from an earlier run:

$ diff stamp.old stamp.now
< tool.iverilog   Icarus Verilog version 12.0 (stable)
> tool.iverilog   Icarus Verilog version 13.0 (stable) (v13_0)
< list.digest     0000000000000000deadbeefdeadbeef
> list.digest     be4ebd03af26f5258252ebc4869f2f51

Seven lines, and each one is there because something this track measured depends on it:

• both tool versions, because Chapter 27.1's iverilog -V goes to stdout while vvp -V goes to stderr, and they can be from different installs
• path.iverilog, because Chapter 10.5 measured PATH order deciding which of two tools runs
• bash's version, because this machine is 3.2.57 and a script developed on 5.x will use declare -A and fail here
• digests rather than timestamps, because Chapter 35.3 measured mtime at whole-second resolution missing a stale file

The point is not that the diff is actionable on its own. It is that when a result changes, the stamp turns "something must have changed" into a two-line answer — and when a result does not change after a tool upgrade, the stamp is the evidence that the upgrade was clean.

4. The Decay That Is Asymmetric

The third invisible change is a check that stops matching. Break the flow's verdict pattern two ways, and the two results are not symmetric.

Stricter — the pattern no longer matches anything:

$ sed 's/grep -q PASSED/grep -q PASSED_RENAMED/' flow.sh > flow_broken.sh
$ /bin/bash flow_broken.sh
flow: 0 of 5 passed
  rc=1

Laxer — the pattern matches too much:

$ sed 's/grep -q PASSED/grep -q ./' flow.sh > flow_lax.sh
$ /bin/bash flow_lax.sh
flow: 5 of 5 passed
  rc=0

0 of 5 and exit 1 is loud; 5 of 5 and exit 0 is silent — and two of those five tests print ERROR. A check that tightens fails immediately and gets fixed the same morning. A check that loosens reports a better result than before and gets fixed never.

So the maintenance question is not "are the checks correct?" but "can this flow still detect a failure at all?" — and that is answerable with one fixture:

#!/bin/bash
# canary.sh -- run the one test that is KNOWN to fail. If it passes, the flow
# has stopped detecting failures and every green result is worthless.
set -u
iverilog -o /tmp/can.vvp rtl/canary.v >/dev/null 2>&1 || { echo "canary: will not compile" >&2; exit 2; }
out=$(vvp /tmp/can.vvp 2>&1)
if printf '%s\n' "$out" | /usr/bin/grep -q 'ERROR'; then
  echo "canary: detected (the flow can still see a failure)"
  exit 0
fi
echo "canary: NOT DETECTED -- the flow's verdicts mean nothing" >&2
exit 1

$ /bin/bash canary.sh
canary: detected (the flow can still see a failure)
  rc=0

$ /bin/bash canary_broken.sh          # the detector looks for FATAL now
canary: NOT DETECTED -- the flow's verdicts mean nothing
  rc=1

A test that must fail, run every night, whose passing is the alarm. It is the only check in this chapter that fires in the silent direction, and it costs one file and a few seconds. Chapter 31.7 argued that a clean run needs positive evidence; this is the same argument applied to the checker instead of the design — and it is the shape this whole track has returned to most often, from Chapter 21.8 onward: a check nobody has seen fail is not known to work.

5. Retiring a Script

Eventually a script should stop being used — superseded, like submitted.sh was in Chapter 35.7. There are three options and only one of them tells anybody.

Delete it.

$ /bin/bash ci/nightly.sh
/…/run_nightly.sh: No such file or directory
  caller rc=127
$ /usr/bin/grep -rl run_nightly .
     ./ci/nightly.sh
     ./ci/crontab.txt

Status 127, no successor named, and both callers still exist — one of them a crontab line at 30 2 * * * that will now fail silently every night (Chapter 32.8 measured where that output goes). And 127 is ambiguous with a PATH problem, so the first hypothesis is the wrong one.

Leave it.

$ /bin/bash ci/nightly.sh
old nightly flow
flow: 3 of 5 passed
  caller rc=1

It runs. Nobody learns it is obsolete. This is what actually happens, every time.

Replace it with a stub that refuses and explains.

#!/bin/bash
# RETIRED 2026-10-10. Superseded by reviewed.sh (Chapter 35.7).
# Kept as a stub, not deleted, so that every remaining caller is told what to
# use instead and is recorded here. Remove this file when the list is empty.
#
# Known callers still pointing at this path:
#   ci/nightly.sh
#   ci/crontab.txt  (30 2 * * *)
cat >&2 <<'MSG'
run_nightly.sh is RETIRED (2026-10-10).

  Use:   ./reviewed.sh
  Why:   it read $? after a pipe, so a compile failure reported PASS
         (see Chapter 35.7)

Update your caller. This stub will be removed once ci/nightly.sh and
ci/crontab.txt no longer reference it.
MSG
exit 2

$ /bin/bash ci/nightly.sh
run_nightly.sh is RETIRED (2026-10-10).

  Use:   ./reviewed.sh
  Why:   it read $? after a pipe, so a compile failure reported PASS
         (see Chapter 35.7)

Update your caller. This stub will be removed once ci/nightly.sh and
ci/crontab.txt no longer reference it.

  caller rc=2

Status 2 — "the flow is broken", Chapter 35.5's value — never 0 and never 127. Four things that make this the right answer:

That last row is the one that gets missed. A retirement with no exit criterion is a stub that lives forever, so the list of callers goes in the file, and when grep -rl returns nothing the stub goes too. Chapter 28.1 measured the cost of deleting the wrong thing; this is the procedure that makes deleting the right thing safe.

6. What a Maintained Flow Has

Five checks, each a few lines, each answering a question the others cannot — and the fifth box is honest: none of them catches a block that was added to rtl/ and to the list with a test that cannot fail. Scope, environment, detection and documentation are the parts that can be mechanised; whether a test tests anything is a review question (Chapter 35.7), not a cron job.

The schedule that makes them worth having:

7. The End of the Track

That is Part II, and with it the Linux & Bash track: thirty-five modules and two hundred and thirty-nine chapters, from pwd to retiring a flow script.

The thread running through Part II is narrower than it looked at the start. Almost every measurement in fourteen modules was a version of one finding: a flow reports success it has not earned. A pipe replaced a status; wait returned 0 for a failed job; a retry wrapper trusted $?; grep -c exited 1 on a clean log; a denominator moved; a pattern stopped matching; a generated script was empty; an empty result set looked like a clean one. Eight mechanisms, one shape — and the defence in every case was the same move: ask what positive evidence exists that the thing worked, and if the answer is "nothing failed", that is not an answer.

The craft that follows from it is also narrower than it looked. Capture the status before anything else runs. Quote the expansion. Anchor the path once, absolutely. Give each run its own directory and its own log. Pin the environment and record it. Make the empty case a refusal. Prove every check able to fail before trusting one that passes. And when the thing is working, write down what it assumes — because that is the part that decays.

Nothing in that list is difficult. All of it is easy to skip, which is why fourteen modules of measurement were needed to show what skipping it costs.

Practice

1. Take a flow you own and compare what it enumerates against what exists.
2. Count the gap in both directions.
3. Check whether its summary line names the denominator.
4. Add two blocks to the design without updating the list, and confirm the output does not change.
5. Classify your flow's assumptions by whether their failure is loud or silent.
6. Write a scopecheck using comm on two sorted lists.
7. Prove it able to fail in both directions.
8. Say what the second direction catches that the first cannot.
9. Write an envstamp and justify every line in it.
10. Capture both tool versions, remembering which stream each writes to.
11. Record the tool's resolved path, not just its version.
12. Use digests rather than timestamps, and say why.
13. Diff two stamps and read the result.
14. Break one of your checks to be stricter and read the output.
15. Break the same check to be laxer and read the output.
16. State which direction is silent, and why it never gets fixed.
17. Write a canary — a test that must fail.
18. Break the detector and confirm the canary is the only thing that fires.
19. Schedule the canary on every run.
20. Delete a superseded script and measure what its callers get.
21. Find every remaining caller with grep -rl.
22. Check whether any of them is a crontab line, and where its output would go.
23. Say why status 127 sends you to the wrong hypothesis.
24. Leave a superseded script in place and say what happens.
25. Write a refusing stub that names its successor and the reason.
26. Give it status 2, and say why not 0, 1 or 127.
27. List the remaining callers inside the stub as its deletion criterion.
28. Build the five-check set and put each on the right schedule.
29. Name what none of them catches.
30. State the one shape behind Part II's measurements, and the one move that defends against it.

Check Your Understanding

1. What did month one and month eight print?
2. What had changed between them?
3. What is the signature of that decay?
4. Why is 3 of 3 worthless in month eight?
5. Which of the four kinds of decay is loud, and why is it the only one planned for?
6. What do the three silent ones have in common?
7. What question does the flow's status answer, and what does it not?
8. What does scopecheck compare, and with what tool?
9. What does its second direction catch?
10. How does a deleted block keep contributing to a pass count?
11. State the rule about lists.
12. Why record both iverilog -V and vvp -V?
13. Why record the tool's resolved path?
14. Why record bash's version?
15. Why digests rather than timestamps?
16. What does a stamp give you when a result changes? When it does not?
17. What did the stricter broken pattern report?
18. What did the laxer one report, and what were those tests printing?
19. Why does a loosening check never get fixed?
20. What is the right maintenance question about checks?
21. What is a canary, and what is its alarm condition?
22. Which direction does it catch, and what does it cost?
23. What is the general form of that argument in this track?
24. What do a deleted script's callers get, and why is the status misleading?
25. What happens if you simply leave it?
26. What four things does a refusing stub do?
27. Why status 2?
28. What is a retirement with no exit criterion?
29. Name the five checks and their schedules.
30. What does none of them catch?
31. What is the one shape behind Part II's measurements? Name four of the eight mechanisms.
32. What is the one defensive move?

8. Remember

• Month one and month eight printed flow: 3 of 3 passed and exit 0. Between them two of five blocks stopped being tested. The denominator moved and the summary did not say so.
• Three of the four kinds of decay are silent; only a moved directory is loud, and it is the only one anybody plans for.
• The flow's status answers "did the tests I ran pass?" and nothing else. Scope needs its own check.
• scopecheck must fail in both directions — untested blocks, and blocks listed but absent. The second catches a deleted block still contributing to a pass count.
• If a script reads a list, something must check the list against reality.
• Stamp the environment every run: both tool versions (different streams), the resolved path, bash's version, and digests rather than timestamps.
• Detector decay is asymmetric. A stricter broken check said 0 of 5, rc 1 — loud. A laxer one said 5 of 5, rc 0, while two tests printed ERROR — silent, and silently better, so nobody looks.
• Run a canary: a test that must fail, whose passing is the alarm. It is the only check that fires in the silent direction.
• Deleting a superseded script gives its callers 127, names no successor, and leaves every caller in place — including a crontab line that now fails quietly every night.
• Retire with a refusing stub: exit 2, name the successor and the reason, and list the remaining callers as the file's own deletion criterion.
• A retirement with no exit criterion is a stub that lives forever.
• Part II's measurements all have one shape — a flow reporting success it has not earned — and one defence: ask what positive evidence exists, because "nothing failed" is not an answer.

9. What Is Next

Nothing in this track — this is Module 35 of 35, and the two hundred and thirty-ninth chapter of Linux & Bash. There is no Module 36.

Part I covered the shell itself: navigation, files, permissions, text processing, scripting and the tools that make a terminal a working environment. Part II applied all of it to the one thing a chip engineer's shell is actually for — running tools, reading what they produce, and knowing whether to believe it.

Where to go next depends on which boundary of this track you reached first. Tcl is the language on the other side of Chapter 35.5, and every EDA tool's interface. Python is the other side of Chapter 35.4, where the reporting and data work belongs once grouping enters it. And the verification tracks — SystemVerilog and UVM — are what the flows in Part II exist to run.