Linux & Bash · Part II · Module 35 · Reusable Flow Tooling and Hand-Off
Documenting a Flow for the Next Engineer
Advice about documentation is usually unfalsifiable, so this chapter does something else: it writes a normal README for the flow built in this module, then runs every claim in it. Four of ten were false, including a --help that performed the action it was documented to describe — and the tool that found them has a measured blind spot of its own.
Advanced25 min readLinuxPart IIDocumentationHand-OffMaintenance
Part II · Module 35 · Chapter 35.6 · Documenting a Flow
The flow from the last five chapters now has a library, a configuration file, a Makefile, a Python reporter and a generated Tcl script. Documenting it is the point at which most writing on the subject becomes advice — "explain your assumptions", "keep it up to date" — which nobody can check.
So instead: here is the README I would actually have written, and here is what happened when every line of it was run.
1. A Normal README
# report flow
## Usage
$ ./drive.sh # defaults to chip_a
$ ./drive.sh ok chip_b reports # explicit top and stage
$ ./drive.sh --help # usage
## Exit status
0 reports were analysed
1 no reports found
2 the flow itself is broken
## Requirements
tclsh, md5
## Notes
The generated script is written to /tmp and cleaned up afterwards.
Set FLOW_NAME to change the log prefix.Nothing about that is unusual. It has a usage block, an exit-status table, a requirements line and two notes — more than most flow scripts get. Ten factual claims.
2. Running Them
$ ./drive.sh
rc=126
$ ./drive.sh ok chip_b reports
rc=126
$ ./drive.sh --help
rc=126126 three times, which is "found, but not executable":
$ ls -l drive.sh
mode -rw-r--r-- drive.shChapter 11.1's status table: 126 is a permission problem, 127 is not found. The README's own invocation form does not work, because the execute bit was never set — and it was never set because every test during development was bash drive.sh.
Invoking it the way the author actually does:
$ /bin/bash drive.sh --help
runtcl: note: running gen/report.tcl (ef2979f9) args: chip_a reports
top=chip_a stage=reports reports=3
hold.rpt
power.rpt
setup.rpt
runtcl: note: flow finished rc=0
rc=0--help ran the flow. The case had no branch for it, so --help fell through to the default, the script generated the report script and ran it, and exited 0.
This is the worst defect of the four, and the reason is not that the output is wrong. It is that the one command a stranger types first, to find out what the script does without doing anything, performs the action. On this flow that is harmless. On Chapter 3.3's cleanup script it is not.
The remaining claims:
$ /bin/bash drive.sh ok chip_a nostage ; echo $?
measured rc=3 (README says 1)
$ ls /tmp/flow.*.tcl | wc -l
0
$ ls -l gen/report.tcl
EXISTS, 486 bytes, not cleaned up| # | claim | measured | verdict |
|---|---|---|---|
| 1 | ./drive.sh runs it | rc 126, no execute bit | wrong |
| 2 | defaults to chip_a | top=chip_a | ok |
| 3 | ok chip_b reports works | top=chip_b reports=3 | ok |
| 4 | --help prints usage | ran the flow, rc 0 | wrong |
| 5 | 0 = reports analysed | rc 0 | ok |
| 6 | 1 = no reports found | rc 3 | wrong |
| 7 | 2 = the flow is broken | rc 2 | ok |
| 8 | needs tclsh, md5 | both present | ok |
| 9 | script goes to /tmp, cleaned up | gen/report.tcl, kept | wrong |
| 10 | FLOW_NAME changes the prefix | timing: note: … | ok |
Four of ten. And none of the four is a typo or a slip of attention:
- #1 is a difference between how the author runs it and how the document says to.
- #4 is a feature the document describes and the script does not have.
- #6 is a status the document remembers from an earlier design.
- #9 is a sentence that was true about the version Chapter 35.5 replaced.
3. Making the Claims Executable
A README's usage block is already almost a test. It needs one annotation — the status the author believes it produces:
#!/bin/bash
# doctest.sh <doc> -- run every command the documentation shows, and check that
# it exits the way the documentation says.
#
# A line of the form
# $ <command> #= <expected rc>
# is executed; a missing "#=" means 0 is expected. Anything else is prose.
#
# 0 every documented command behaved as documented
# 1 at least one did not
# 2 usage / unreadable
set -u
doc=${1:?usage: doctest.sh <doc>}
[ -r "$doc" ] || { echo "doctest: cannot read $doc" >&2; exit 2; }
pass=0; fail=0
while IFS= read -r line; do
case $line in
*'$ '*) ;;
*) continue ;;
esac
cmd=${line#*$ }
want=0
case $cmd in
*'#='*) want=${cmd##*#=}; cmd=${cmd%%#=*} ;;
esac
want=$(printf '%s' "$want" | tr -d ' ')
cmd=$(printf '%s' "$cmd" | sed 's/#.*$//; s/[[:space:]]*$//')
[ -n "$cmd" ] || continue
eval "$cmd" >/dev/null 2>&1
got=$?
if [ "$got" = "$want" ]; then
printf 'doctest: ok rc=%s %s\n' "$got" "$cmd"
pass=$((pass+1))
else
printf 'doctest: WRONG want rc=%s got rc=%s %s\n' "$want" "$got" "$cmd" >&2
fail=$((fail+1))
fi
done < "$doc"
printf 'doctest: %s ok, %s wrong\n' "$pass" "$fail"
[ "$fail" -eq 0 ] || exit 1Run against the README as written, every claim unannotated and therefore expected to succeed:
doctest: WRONG want rc=0 got rc=126 ./drive.sh
doctest: WRONG want rc=0 got rc=126 ./drive.sh ok chip_b reports
doctest: WRONG want rc=0 got rc=126 ./drive.sh --help
doctest: 0 ok, 3 wrong
rc=1And with the README's own claimed statuses written in:
doctest: ok rc=0 /bin/bash drive.sh
doctest: ok rc=0 /bin/bash drive.sh ok chip_b reports
doctest: WRONG want rc=2 got rc=0 /bin/bash drive.sh --help
doctest: WRONG want rc=1 got rc=3 /bin/bash drive.sh ok chip_a nostage
doctest: ok rc=2 /bin/bash drive.sh empty
doctest: 3 ok, 2 wrong
rc=1Proven able to fail on five separate claims across the two runs — which is the standard Chapter 21.8 sets for any checker, and the reason to show the red output before the green.
4. The Blind Spot
doctest checks commands. Most of a README is prose, and prose is not executable:
The flow reads its stage from the `CONF_STAGE` setting and requires a licence.
Reports older than 24 hours are regenerated automatically.
The generated script is deleted after a successful run.
$ ./drive.sh #= 0
$ ./drive.sh --help #= 2doctest: ok rc=0 ./drive.sh
doctest: ok rc=2 ./drive.sh --help
doctest: 2 ok, 0 wrong
rc=0Three false claims, zero complaints. Each is false in the strongest possible sense — the words do not appear in the code at all:
CONF_STAGE: drive.sh:0 runtcl.sh:0
licence: drive.sh:0 runtcl.sh:0
24 hours: drive.sh:0 runtcl.sh:0
deletion: 1 (only in the 'gone' simulation)This is the same shape as Chapter 35.5's info complete and bash -n: a check that verifies structure and cannot verify meaning. Stating the blind spot is part of shipping the tool, because a green doctest on a README full of invented prose is a stronger false assurance than no check at all.
One slice of the prose is mechanically checkable, though — the identifiers it names:
#!/bin/bash
# refcheck.sh <doc> <script>... -- every identifier the doc names in backticks
# must appear in at least one script. Catches documentation that describes a
# variable, function or flag the code does not have.
set -u
doc=${1:?usage: refcheck.sh <doc> <script>...}; shift
[ $# -ge 1 ] || { echo "refcheck: need at least one script" >&2; exit 2; }
miss=0
for id in $(/usr/bin/grep -o '`[A-Za-z_][A-Za-z_0-9]*`' "$doc" | tr -d '`' | sort -u); do
if /usr/bin/grep -qw -- "$id" "$@"; then
printf 'refcheck: ok %s\n' "$id"
else
printf 'refcheck: MISSING %s (named in %s, in no script)\n' "$id" "$doc" >&2
miss=$((miss+1))
fi
done
printf 'refcheck: %s missing\n' "$miss"
[ "$miss" -eq 0 ] || exit 1refcheck: MISSING CONF_STAGE (named in README4.md, in no script)
refcheck: ok FLOW_NAME
refcheck: ok run_tcl
refcheck: ok tclcheck
refcheck: 1 missing
rc=1A document that names a variable, function or flag the code does not have is wrong in a way a machine can see — which covers the commonest drift of all: a setting that was renamed.
5. What the Document Should Contain
With both checks in place, the question "what goes in it" has a sharper answer than "the assumptions": everything a stranger cannot get by running --help, plus everything --help says, in a form the audit can check.
# report flow
Generates a Tcl report script, checks it, runs it, and keeps it.
## Run it
$ ./drive.sh #= 0
$ ./drive.sh ok chip_b reports #= 0
$ ./drive.sh --help #= 2
## Exit status
0 the script ran and exited 0
2 the flow is broken -- the interpreter was NOT launched
3 the script ran and exited non-zero (no reports, or no stage directory)
Never 1. In this flow 1 means a failing test, and this step does not run tests.
$ ./drive.sh ok chip_a nostage #= 3
$ ./drive.sh empty #= 2
$ ./drive.sh nonsense #= 2
## What it writes, and what it keeps
`gen/report.tcl` is the generated script. It is **not** deleted: it is the only
record of what the tool was asked to do, and it can be re-run by hand with no
bash involved --
$ tclsh gen/report.tcl chip_a reports #= 0
Its md5 is logged on every run. The same inputs give the same digest, so
"did the tool get the script I think it got?" is a comparison, not an argument.
## Settings
`FLOW_NAME` prefixes this flow's own messages (default `runtcl`).
$ FLOW_NAME=timing ./drive.sh #= 0
## Requirements
`tclsh` (8.5 or later) and `/sbin/md5`, which `run_tcl` calls by absolute path
so that PATH cannot change the answer.
## Known failure modes
- A generation that writes nothing still leaves a file, because `>` truncates
before the command runs. `run_tcl` rejects an empty script rather than
running it, because an empty Tcl script exits 0.
- `tclcheck` checks structure, not names: it rejects an unbalanced brace and
accepts a call to a command that does not exist.
- The stage argument is not validated by the shell. `report.tcl` checks it and
exits 3.doctest: 8 ok, 0 wrong rc=0
refcheck: 0 missing rc=0Five sections, and each earns its place for a reason this module measured:
Run it — the commands, annotated, so doctest covers them. Three lines, because the authoritative argument list is in --help.
Exit status — with the reason 1 is absent. A table of numbers is copied wrong — claim #6 in §2 above; a table with "1 means a failing test and this step does not run tests" survives a rewrite, because the next engineer can regenerate it from the rule.
What it writes, and what it keeps — the non-obvious decision. gen/report.tcl surviving looks like an oversight until the sentence explains that it is the only record of what the tool was asked to do. Undocumented deliberate behaviour gets helpfully "fixed" by the next person, which is the failure mode Chapter 28.1 measured from the other direction.
Settings — one line per variable, each with its default and a runnable example, so refcheck can see the name and doctest the behaviour.
Known failure modes — the three that cost real debugging time this module: the zero-byte artefact from > truncating early, tclcheck checking structure and not names, and the stage argument being validated by Tcl rather than by the shell. This is the section nobody writes and the only one with a measurable payoff: each entry is a debugging session the next engineer does not repeat.
And what is deliberately not there: the argument list (it is in --help), the internal design (it is in the script's own CONTRACT block from Chapter 35.1), a change log (that is git log), and anything about why the flow exists, which belongs to the project and not to the script.
6. Where Each Kind of Fact Lives
Figure 35.6.1 — Four places a fact can live, and the test for each
The operative test is who has to change when the script changes. A fact in --help or a CONTRACT block changes in the same edit as the code, in the same file, under the same review. A fact in the README changes in a different file and is therefore the one that drifts — which is exactly why the README is the part that gets audited, and why it should hold as little as it can.
Practice
- Write the README you would normally write for a script you already have.
- Count its factual claims.
- Run every command in it exactly as written.
- Check whether your script is executable, and how you have been invoking it.
- Run your documented
--helpand confirm it does not act. - Check every exit status in your status table.
- Check every statement about files written or removed.
- Tally the defects and classify each by cause.
- Enumerate your script's
casebranches and compare against the document. - Check for a
*)branch, and add one if it is missing. - Say why a missing claim is harder to find than a false one.
- Write a
doctestthat runs annotated commands and checks statuses. - Show it failing before you show it passing.
- For each defect, decide which side is wrong and say why.
- Move the usage text into the script as a quoted heredoc.
- Add
*) usage ;;and re-run the audit. - Write three false prose claims and confirm
doctestpasses. - Grep the code for each one to show how false they are.
- State your
doctest's blind spot in its own documentation. - Write a
refcheckfor backticked identifiers. - Compare its output against an independent count of the identifiers in the document.
- If you used
sed s///, check how many matches per line it took. - Rewrite it with
grep -oand compare. - Write the five-section document and get both checks green.
- Give the reason behind your status table, not just the numbers.
- Document one deliberate behaviour that looks like a bug.
- Write a known-failure-modes section from this module's debugging.
- List what you deliberately left out, and where each of those facts lives.
- Apply the who-has-to-change test to every fact in your document.
Check Your Understanding
- How many claims did the normal README make, and how many were false?
- What does exit status 126 mean, and why did it appear three times?
- Why had the execute bit never been set?
- What did
--helpdo, and why is that the worst of the four? - What status did "no reports found" actually produce?
- Where was the generated script, and what had the README said?
- Classify each of the four defects by cause.
- How many
casebranches did the script have, and how many were documented? - What happened for an unrecognised argument, and why?
- Why is a missing claim harder to find than a false one?
- What single annotation makes a usage block testable?
- On how many claims was
doctestproven able to fail? - For each of the four defects, which side was wrong?
- Why would a document written from the behaviour have been worse?
- Why put the usage text inside the script as a quoted heredoc?
- What does
*) usage ;;contribute to documentation? - What did
doctestsay about three invented prose claims? - How false were they?
- Which earlier tools share that blind spot?
- Why must the blind spot be stated rather than left implicit?
- What slice of prose is mechanically checkable?
- What did the first
refcheckread, and what did it report? - Why did
sedcapture the last identifier on a line rather than the first? - Why did the bug look like a working check?
- What is the fix, and what is the general rule?
- How was the bug detected at all?
- Name the five sections of the model document and the reason for each.
- Why give the reason behind a status table?
- What happens to undocumented deliberate behaviour?
- Which section has the most measurable payoff?
- What is deliberately left out, and where does each of those facts live?
- State the who-has-to-change test and what it predicts.
7. Remember
- A normal README made ten claims and four were false — and not one was a typo.
./drive.shexited 126: the documented invocation form had never been run, because development usedbash drive.sh.- The documented
--helpran the flow. The command a stranger types to look without touching performed the action. - The status table said 1 for no data; the script said 3. The script was right — 1 means a failing test.
- "Written to /tmp and cleaned up" was true of the version 35.5 replaced. Documentation decays by being correct about the past.
- Four
casebranches, one documented, and no*)— a false claim is visible in the document, a missing one is not. - Annotate each usage line with its expected status and the usage block becomes a test suite.
doctestwas proven able to fail on five claims. - The audit says a claim is wrong, not which side to change — here two were the script's fault and two the document's.
- Put the usage text in the script as a quoted heredoc, and add
*) usage ;;so it cannot drift from the code. doctestpassed three completely invented prose claims. Say the blind spot out loud; a green check on invented prose is worse than no check.refcheckread two of four identifiers and still looked right.sed s///is one replacement per line, and a greedy.*lands the capture on the last match — usegrep -o.- Five sections: run it, exit status with its reason, what it keeps and why, settings, known failure modes. Leave out the argument list, the design, the history and the rationale.
- Document deliberate behaviour that looks like a bug, or the next person will fix it.
- Who has to change when the script changes? Facts that move with the code do not drift; the README is the part that does, so keep it small and audit it.
8. What Is Next
The flow is now built, bounded, measured and documented — which is the state it is in on the day it is handed over, and not the state it stays in. Chapter 35.7 turns the whole of Part II into the review a flow script should pass before anybody else depends on it, applied to real scripts with real defects rather than to a list of principles.
Where this fits
Part of the Linux & Bash curriculum.
