Python Part II · Module 34 · From Script to Maintainable Tool
Documenting a Tool Someone Else Will Inherit
The tool now runs on other people's machines, which means other people will own it. Documentation is where this module's work either survives or is lost, and the usual failure is not that nobody wrote any — it is that what was written drifted away from what the code does, silently, because prose is the one part of a repository that nothing checks. Most of it can be checked, and the check finds errors in both directions.
Intermediate22 min readPythonPart IIAutomationDocumentationHandover
Part II · Module 34 · Chapter 34.6 · Documenting a Tool Someone Else Will Inherit
1. What the Next Owner Asks
QUESTIONS = [
("What does it do, in one line?", "the docstring", "yes"),
("How do I run it once?", "a worked example", "yes"),
("What does the exit code mean?", "an exit-code table", "yes"),
("What can I configure?", "the config section", "yes"),
("What will it NOT do?", "a non-goals section", "almost never"),
("How is it implemented?", "the code", "no — do not duplicate it"),
("Why is line 94 like that?", "a comment at line 94", "no"),
]
print(f"{'what the new owner asks':<34} {'where it belongs':<24} usually present")
for q, where, present in QUESTIONS:
print(f"{q:<34} {where:<24} {present}")
print()
print("questions a handover doc must answer:", 5)
print("questions it must NOT answer :", 2)
print("the one almost always missing :", "what it will not do")what the new owner asks where it belongs usually present
What does it do, in one line? the docstring yes
How do I run it once? a worked example yes
What does the exit code mean? an exit-code table yes
What can I configure? the config section yes
What will it NOT do? a non-goals section almost never
How is it implemented? the code no — do not duplicate it
Why is line 94 like that? a comment at line 94 no
questions a handover doc must answer: 5
questions it must NOT answer : 2
the one almost always missing : what it will not doThe last two rows are the ones that make handover documents long and useless. A document describing how the code works is a second copy of the code that nothing keeps in step, and it is the part a new owner skips because the real code is right there and is definitely current.
2. The Annotation
"""run_regress.py - run a suite; exit 0 only if every job passed."""One line, and the choice of a docstring rather than a comment is the whole technique. A comment has exactly one reader — somebody with the file open. A docstring is a value:
import io
import sys
from pathlib import Path
Path("run_regress.py").write_text(
'"""run_regress.py - run a suite; exit 0 only if every job passed."""\n'
"\n"
"def main(argv=None):\n"
" return 0\n", encoding="utf-8")
sys.path.insert(0, ".")
import run_regress
print("the docstring as a value:")
print(" module.__doc__ :", repr(run_regress.__doc__))
print()
print("readers of that one string:")
READERS = [
("somebody with the file open", "it is line 1"),
("help(run_regress)", "python reads __doc__"),
("run_regress.py --help", "argparse description=__doc__ — §4"),
("a doc linter", "ast.get_docstring — §6"),
]
for who, how in READERS:
print(f" {who:<30} {how}")
print()
print("readers a COMMENT would have had:", 1)
print("readers a DOCSTRING has :", len(READERS))the docstring as a value:
module.__doc__ : 'run_regress.py - run a suite; exit 0 only if every job passed.'
readers of that one string:
somebody with the file open it is line 1
help(run_regress) python reads __doc__
run_regress.py --help argparse description=__doc__ — §4
a doc linter ast.get_docstring — §6
readers a COMMENT would have had: 1
readers a DOCSTRING has : 4Four readers of one string. That is the argument for the shape: there is no second place for the one-line description to live, so it cannot drift from itself. The clause after the dash is also chosen carefully — exit 0 only if every job passed is a statement about the exit code, which is the one promise Chapter 33.8 established that other scripts depend on.
3. The Five Sections
SECTIONS = [
("the one-line contract", 1, "what it does and what exit 0 means"),
("a worked example", 4, "one command that works, copy-pasteable"),
("exit codes", 6, "a table — §5 checks it"),
("configuration", 8, "env vars, the config file, the flags — 34.5"),
("non-goals", 4, "what it deliberately does not do — §7"),
]
total = sum(n for _, n, _ in SECTIONS)
print(f"{'section':<24} {'lines':>6} content")
for name, n, what in SECTIONS:
print(f"{name:<24} {n:>6} {what}")
print()
print("total lines of documentation:", total)
print("lines of code in the tool :", 154)
print("ratio :", f"1 line of docs per {154 // total} of code")
print()
print("what is NOT in the list:", ["how it works", "why line 94 is like that",
"a changelog", "the design history"])section lines content
the one-line contract 1 what it does and what exit 0 means
a worked example 4 one command that works, copy-pasteable
exit codes 6 a table — §5 checks it
configuration 8 env vars, the config file, the flags — 34.5
non-goals 4 what it deliberately does not do — §7
total lines of documentation: 23
lines of code in the tool : 154
ratio : 1 line of docs per 6 of code
what is NOT in the list: ['how it works', 'why line 94 is like that', 'a changelog', 'the design history']Twenty-three lines. That is the whole handover document, and its smallness is deliberate: a document that fits on one screen gets read, and every one of the five sections answers a question from Section 1 that the code itself cannot answer by being read.
4. description=__doc__ Needs One More Argument
import argparse
import contextlib
import io
DOC = """run_regress.py - run a suite; exit 0 only if every job passed.
Example:
run_regress.py --suite smoke --seed 4121
Exit codes:
0 every job passed
1 at least one job failed
2 the command line was wrong
"""
def help_text(formatter=None):
kwargs = {"prog": "run_regress.py", "description": DOC}
if formatter is not None:
kwargs["formatter_class"] = formatter
parser = argparse.ArgumentParser(**kwargs)
parser.add_argument("--suite", help="a suite name from flow.toml")
parser.add_argument("--seed", type=int, action="append",
help="repeatable; overrides default_seeds")
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
parser.print_help()
return buf.getvalue()
for label, fmt in (("default formatter", None),
("RawDescriptionHelpFormatter",
argparse.RawDescriptionHelpFormatter)):
out = help_text(fmt)
print(f"--- {label}")
print(" 'Exit codes:' starts a line :", "\nExit codes:" in out)
print(" the table keeps its layout :", " 0 every job passed" in out)
print(" reflowed into one paragraph :", "passed. Example:" in out)--- default formatter
'Exit codes:' starts a line : False
the table keeps its layout : False
reflowed into one paragraph : True
--- RawDescriptionHelpFormatter
'Exit codes:' starts a line : True
the table keeps its layout : True
reflowed into one paragraph : FalseThe default formatter reflows the description into one paragraph, so the
exit-code table — the part a new owner most needs — arrives as
... passed. Example: run_regress.py --suite smoke --seed 4121 Exit codes: 0 every job passed 1 at least one job failed ... on a single wrapped line.
Nothing warns about this. description=__doc__ looks like it worked, because
the description is there; what is lost is every newline in it. One extra
argument fixes it:
import argparse
import contextlib
import io
DOC = """run_regress.py - run a suite; exit 0 only if every job passed.
Exit codes:
0 every job passed
1 at least one job failed
"""
parser = argparse.ArgumentParser(
prog="run_regress.py",
description=DOC,
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="Configuration: FLOW_RUN_ROOT, FLOW_SIM, flow.toml. "
"See the module docstring.",
)
parser.add_argument("--suite", default="smoke",
help="a suite name from flow.toml (default: %(default)s)")
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
parser.print_help()
out = buf.getvalue()
print(out, end="")
print()
print("the default is shown in --help:", "(default: smoke)" in out)
print("via the placeholder :", "%(default)s")
print("so the default lives in :", "one place — the add_argument call")usage: run_regress.py [-h] [--suite SUITE]
run_regress.py - run a suite; exit 0 only if every job passed.
Exit codes:
0 every job passed
1 at least one job failed
options:
-h, --help show this help message and exit
--suite SUITE a suite name from flow.toml (default: smoke)
Configuration: FLOW_RUN_ROOT, FLOW_SIM, flow.toml. See the module docstring.
the default is shown in --help: True
via the placeholder : %(default)s
so the default lives in : one place — the add_argument callThe table survives and the default is printed from the real default.
%(default)s is the small habit that matters most here: writing
help="a suite name (default: smoke)" duplicates the value, and the day
somebody changes default= the help text becomes a lie that nothing detects.
The epilog is where the configuration summary goes, because --help is the
one piece of documentation a user is guaranteed to find.
5. Documented, Reachable, Neither
A docstring that lists exit codes is making a checkable claim. Here is a tool whose claim is wrong in two different ways:
import ast
import re
from pathlib import Path
TOOL = '''"""run_regress.py - run a suite; exit 0 only if every job passed.
Exit codes:
0 every job passed
1 at least one job failed
2 the command line was wrong
3 an input was missing or unreadable
5 the licence server was unreachable
"""
import sys
EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_INPUT, EXIT_ENV = 0, 1, 2, 3, 4
def main(argv=None):
if not argv:
return EXIT_USAGE
if argv[0] == "bad":
return EXIT_INPUT
if argv[0] == "env":
return EXIT_ENV
return EXIT_OK
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
'''
Path("tool.py").write_text(TOOL, encoding="utf-8")
tree = ast.parse(TOOL)
doc = ast.get_docstring(tree)
documented = {int(m.group(1))
for m in re.finditer(r'^\s{2}(\d+)\s{2}', doc, re.M)}
consts = {}
for node in tree.body:
if isinstance(node, ast.Assign) and isinstance(node.value, ast.Tuple) \
and isinstance(node.targets[0], ast.Tuple):
names = [t.id for t in node.targets[0].elts]
values = [e.value for e in node.value.elts]
consts.update(dict(zip(names, values)))
reachable = set()
for node in ast.walk(tree):
if isinstance(node, ast.Return) and node.value is not None:
v = node.value
if isinstance(v, ast.Name) and v.id in consts:
reachable.add(consts[v.id])
elif isinstance(v, ast.Constant) and isinstance(v.value, int):
reachable.add(v.value)
print("constants defined :", consts)
print()
print("documented in the docstring:", sorted(documented))
print("reachable from main() :", sorted(reachable))
print()
print("documented but UNREACHABLE :", sorted(documented - reachable))
print("reachable but UNDOCUMENTED :", sorted(reachable - documented))
print("the two agree :", documented == reachable)constants defined : {'EXIT_OK': 0, 'EXIT_FAILED': 1, 'EXIT_USAGE': 2, 'EXIT_INPUT': 3, 'EXIT_ENV': 4}
documented in the docstring: [0, 1, 2, 3, 5]
reachable from main() : [0, 2, 3, 4]
documented but UNREACHABLE : [1, 5]
reachable but UNDOCUMENTED : [4]
the two agree : FalseThree defects, found by comparing a docstring with an AST. Each is a different kind of wrong and they need different responses.
Code 1 is documented and unreachable, and this is the dangerous one. The
docstring says exit 0 only if every job passed and declares code 1 for the
failure case — but main never returns EXIT_FAILED, so a run with failing
jobs exits 0. Every wrapper that branches on $? has been told a promise the
tool does not keep, which is Chapter 34.1's missing exit code wearing a
docstring.
Code 5 is pure fiction. No constant has the value 5. Somebody documented an intention, or copied the table from a different tool, and nothing noticed. Chapter 33.8 Section 8's "every documented code is reachable" test is exactly this check, and here it fails on two of five rows.
Code 4 is reachable and undocumented. EXIT_ENV is returned and never
mentioned, so a wrapper meets an exit code that is not in the contract and has
no defined meaning — the caller's only options are to treat it as a generic
failure or to crash.
6. The Doc Linter
import ast
import io
import re
import unittest
def exit_codes(source):
"""Returns (documented, reachable) from one module's source."""
tree = ast.parse(source)
doc = ast.get_docstring(tree) or ""
documented = {int(m.group(1))
for m in re.finditer(r'^\s{2}(\d+)\s{2}', doc, re.M)}
consts = {}
for node in tree.body:
if isinstance(node, ast.Assign) and isinstance(node.value, ast.Tuple) \
and isinstance(node.targets[0], ast.Tuple):
consts.update(dict(zip([t.id for t in node.targets[0].elts],
[e.value for e in node.value.elts])))
elif isinstance(node, ast.Assign) and isinstance(node.value,
ast.Constant) \
and isinstance(node.targets[0], ast.Name):
consts[node.targets[0].id] = node.value.value
reachable = set()
for node in ast.walk(tree):
if isinstance(node, ast.Return) and node.value is not None:
v = node.value
if isinstance(v, ast.Name) and v.id in consts:
reachable.add(consts[v.id])
elif isinstance(v, ast.Constant) and isinstance(v.value, int):
reachable.add(v.value)
return documented, reachable
def check_docs(source):
documented, reachable = exit_codes(source)
problems = []
for code in sorted(documented - reachable):
problems.append(f"exit {code} is documented and unreachable")
for code in sorted(reachable - documented):
problems.append(f"exit {code} is reachable and undocumented")
if not ast.get_docstring(ast.parse(source)):
problems.append("the module has no docstring")
return problems
HEADER = '"""tool - one line.\n\nExit codes:\n'
BODY = ('"""\nEXIT_OK, EXIT_FAILED = 0, 1\n\n'
"def main(argv=None):\n"
" if not argv:\n"
" return EXIT_FAILED\n"
" return EXIT_OK\n")
CASES = {
"correct": HEADER + " 0 ok\n 1 failed\n" + BODY,
"documents a code it cannot return": HEADER + " 0 ok\n 1 failed\n"
" 5 licence\n" + BODY,
"omits a code it can return": HEADER + " 0 ok\n" + BODY,
"no docstring at all": "EXIT_OK = 0\n\ndef main(argv=None):\n"
" return EXIT_OK\n",
}
print(f"{'module':<36} {'clean':<7} problems")
for label, src in CASES.items():
probs = check_docs(src)
print(f"{label:<36} {str(not probs):<7} {probs or '-'}")
print()
print("cases tried :", len(CASES))
print("deliberately broken :", 3)
print("caught :", sum(1 for s in list(CASES.values())[1:]
if check_docs(s)))
print("false positives on correct:", len(check_docs(CASES["correct"])))module clean problems
correct True -
documents a code it cannot return False ['exit 5 is documented and unreachable']
omits a code it can return False ['exit 1 is reachable and undocumented']
no docstring at all False ['exit 0 is reachable and undocumented', 'the module has no docstring']
cases tried : 4
deliberately broken : 3
caught : 3
false positives on correct: 0Three broken modules caught, no false positive on the correct one. Both halves are necessary, which is Chapter 33.9's argument again: the three broken cases are mutations of the documentation and the detection rate is the kill rate.
Two honest limitations, stated because a check that overstates its reach is
worse than none. The linter reads return statements in the module, so a code
returned by a called function — return _classify(rows) — is invisible to
it; and it parses the docstring with a regex expecting two-space-indented
entries, so a differently formatted table reads as zero documented codes and
every reachable code looks undocumented. Both are fine for a convention a
project adopts, and both must be written down as the convention.
7. What the Tool Deliberately Does Not Do
NON_GOALS = [
("does not submit to a job scheduler",
"runs serially, locally", "somebody wraps it in a scheduler loop"),
("does not retry a failed run",
"a failure is reported, not hidden", "a flaky result is read as real"),
("does not clean the run root",
"33.13's scratch removal is per-run only", "the volume fills"),
("does not compare against a golden reference",
"it reports status, not correctness", "a PASS is read as sign-off"),
("does not read vendor-specific waveforms",
"it parses the text log only", "somebody asks why coverage is missing"),
]
print(f"{'non-goal':<42} {'instead':<40} if nobody says so")
for goal, instead, consequence in NON_GOALS:
print(f"{goal:<42} {instead:<40} {consequence}")
print()
print("non-goals listed :", len(NON_GOALS))
print("each one prevents :", "a wrong assumption being built on")
print("the one that matters most :", "'does not retry' — §7 of 33.14")
print()
print("why this section is omitted :", "it reads as an apology")
print("why it is not :", "it is the scope boundary")non-goal instead if nobody says so
does not submit to a job scheduler runs serially, locally somebody wraps it in a scheduler loop
does not retry a failed run a failure is reported, not hidden a flaky result is read as real
does not clean the run root 33.13's scratch removal is per-run only the volume fills
does not compare against a golden reference it reports status, not correctness a PASS is read as sign-off
does not read vendor-specific waveforms it parses the text log only somebody asks why coverage is missing
non-goals listed : 5
each one prevents : a wrong assumption being built on
the one that matters most : 'does not retry' — §7 of 33.14
why this section is omitted : it reads as an apology
why it is not : it is the scope boundaryEvery row's third column is a real defect somebody else builds. That is the argument for a section teams leave out because it reads like a list of shortcomings: a non-goal is not an apology, it is the scope boundary, and each omitted boundary becomes a wrong assumption that gets built on and is expensive to unwind later.
The fourth row is the one worth being most explicit about in a verification
flow. It reports status, not correctness — a PASS means the simulation ran
and logged no errors, which is not the same as the design being right, and
Chapter 29.6's sign-off criteria are a separate thing entirely. A tool whose
output is read as sign-off evidence when it was never that is the worst
outcome on this list.
And the second row connects directly to Chapter 33.14 Section 7's measurement: a retry that succeeded hides flakiness completely, so not retrying is a deliberate design choice and saying so stops somebody adding a retry loop as a helpful improvement.
8. The Handover Document, Assembled
import argparse
import contextlib
import io
import sys
from pathlib import Path
DOC = '''run_regress.py - run a regression suite; exit 0 only if every job passed.
Example:
export FLOW_RUN_ROOT=/scratch/$USER/runs
run_regress.py --suite smoke --seed 4121 --seed 7734
Exit codes:
0 every job passed
1 at least one job failed
2 the command line was wrong
3 an input was missing or unreadable
4 the environment is wrong (no simulator on PATH)
Configuration:
FLOW_RUN_ROOT where run directories are created (default: <root>/runs)
FLOW_SIM the simulator to invoke (default: iverilog)
LM_LICENSE_FILE read by the simulator, not by this tool
flow.toml in the RTL repo: project name, block, tests, default_seeds
Does NOT:
- submit to a job scheduler; it runs serially and locally
- retry a failed run; a failure is reported, never hidden
- clean the run root; only per-run scratch is removed
- judge correctness; a PASS means "ran and logged no errors"
'''
Path("run_regress.py").write_text(
f'"""{DOC}"""\n'
"import sys\n"
"\n"
"EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_INPUT, EXIT_ENV = 0, 1, 2, 3, 4\n"
"\n"
"\n"
"def main(argv=None):\n"
" if argv is None:\n"
" return EXIT_USAGE\n"
" if argv.get('no_sim'):\n"
" return EXIT_ENV\n"
" if argv.get('bad_input'):\n"
" return EXIT_INPUT\n"
" if argv.get('failed'):\n"
" return EXIT_FAILED\n"
" return EXIT_OK\n", encoding="utf-8")
import ast
import re
src = Path("run_regress.py").read_text(encoding="utf-8")
tree = ast.parse(src)
doc = ast.get_docstring(tree)
documented = {int(m.group(1))
for m in re.finditer(r'^\s{2}(\d+)\s{2}', doc, re.M)}
consts = {}
for node in tree.body:
if isinstance(node, ast.Assign) and isinstance(node.value, ast.Tuple):
consts.update(dict(zip([t.id for t in node.targets[0].elts],
[e.value for e in node.value.elts])))
reachable = {consts[n.value.id] for n in ast.walk(tree)
if isinstance(n, ast.Return) and isinstance(n.value, ast.Name)
and n.value.id in consts}
print("sections present:")
for name, marker in (("one-line contract", "run_regress.py - run"),
("worked example", "Example:"),
("exit codes", "Exit codes:"),
("configuration", "Configuration:"),
("non-goals", "Does NOT:")):
print(f" {name:<20} {marker in doc}")
print()
print("documented codes :", sorted(documented))
print("reachable codes :", sorted(reachable))
print("the linter says :", "clean" if documented == reachable else "DRIFT")
print()
print("docstring lines :", len(doc.strip().splitlines()))
print("code lines :", len([l for l in src.splitlines()
if l.strip() and not l.startswith('"""')])
- len(doc.splitlines()))sections present:
one-line contract True
worked example True
exit codes True
configuration True
non-goals True
documented codes : [0, 1, 2, 3, 4]
reachable codes : [0, 1, 2, 3, 4]
the linter says : clean
docstring lines : 24
code lines : 7Five sections, five exit codes documented and all five reachable, and the linter is clean. Note that the previously missing code 1 now has a branch — fixing the documentation forced fixing the code, which is the point of making the claim checkable rather than aspirational.
The worked example starts with export FLOW_RUN_ROOT=... deliberately. Chapter
34.5 Section 3 measured what happens when that variable is set but empty, so an
example that shows it being set correctly is the cheapest defence against the
wrapper that sets it to nothing.
9. The Worked Example, Executed
Section 5 made the exit-code table checkable. The worked example can be checked too, by the standard library, with no extra tooling:
import contextlib
import doctest
import importlib
import io
import sys
from pathlib import Path
MODULES = {
"good_doc": '"""Percentage helpers for the regression report."""\n'
"\n"
"\n"
"def pass_rate(passed, total):\n"
' """Percentage of jobs that passed.\n'
"\n"
" >>> pass_rate(9, 12)\n"
" 75.0\n"
" >>> pass_rate(0, 0)\n"
" 0.0\n"
' """\n'
" return 100.0 * passed / total if total else 0.0\n",
"bad_doc": '"""The same helpers, with the example written from memory."""\n'
"\n"
"\n"
"def pass_rate(passed, total):\n"
' """Percentage of jobs that passed.\n'
"\n"
" >>> pass_rate(9, 12)\n"
" 75\n"
" >>> pass_rate(1, 3)\n"
" 33.3\n"
' """\n'
" return 100.0 * passed / total if total else 0.0\n",
}
for name, src in MODULES.items():
Path(name + ".py").write_text(src, encoding="utf-8")
sys.path.insert(0, ".")
print(f"{'module':<12} {'examples':>9} {'failed':>7} verdict")
for name in MODULES:
mod = importlib.import_module(name)
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
result = doctest.testmod(mod, verbose=False, report=False)
print(f"{name:<12} {result.attempted:>9} {result.failed:>7} "
f"{'clean' if not result.failed else 'DRIFT'}")
print()
mod = importlib.import_module("bad_doc")
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
doctest.testmod(mod, verbose=False, report=True)
print("what doctest says about the wrong example:")
for line in buf.getvalue().splitlines():
s = line.strip()
if s.startswith(("Failed example", "Expected", "Got", "pass_rate",
"75", "33")):
print(" ", s[:46])module examples failed verdict
good_doc 2 0 clean
bad_doc 2 2 DRIFT
what doctest says about the wrong example:
Failed example:
pass_rate(9, 12)
Expected:
75
Got:
75.0
Failed example:
pass_rate(1, 3)
Expected:
33.3
Got:
33.333333333333336Both of the wrong module's examples failed, and both failures are the
arithmetic hazards Chapter 34.1 Section 7 measured. 75 against 75.0 is
the float-division surprise; 33.3 against 33.333333333333336 is the same
value printed in full. Somebody wrote those examples from memory, which is what
people do, and doctest caught them in one call.
That is the strongest form of the chapter's argument. The exit-code linter in
Section 6 is something you have to write; doctest is already installed, it
treats every >>> in every docstring as a test, and it fails the build when
prose disagrees with behaviour.
SUITABILITY = [
("pass_rate(9, 12) -> 75.0", "yes", "pure, tiny, deterministic"),
("classify(counts, text) -> 'FAIL'", "yes", "pure, literal inputs"),
("run_sim(...) -> (0, log)", "no", "launches a simulator"),
("a path under /scratch/$USER", "no", "machine-specific output"),
("a timestamped log line", "no", "33.10 §7 — unstable"),
("a dict with >3 keys", "careful", "key order is stable, width is not"),
]
print(f"{'what the example shows':<34} {'doctest':<9} why")
for what, ok, why in SUITABILITY:
print(f"{what:<34} {ok:<9} {why}")
print()
good = sum(1 for _, ok, _ in SUITABILITY if ok == "yes")
print("examples doctest should own :", good)
print("examples that must stay prose:", len(SUITABILITY) - good - 1)
print()
print("so the shell example in §8 :", "stays prose — it exports a variable")
print("and the pure helpers :", "get doctests")
print("the rule :",
"doctest what is deterministic; narrate the rest")what the example shows doctest why
pass_rate(9, 12) -> 75.0 yes pure, tiny, deterministic
classify(counts, text) -> 'FAIL' yes pure, literal inputs
run_sim(...) -> (0, log) no launches a simulator
a path under /scratch/$USER no machine-specific output
a timestamped log line no 33.10 §7 — unstable
a dict with >3 keys careful key order is stable, width is not
examples doctest should own : 2
examples that must stay prose: 3
so the shell example in §8 : stays prose — it exports a variable
and the pure helpers : get doctests
the rule : doctest what is deterministic; narrate the restTwo of six examples belong in a doctest and three must not be. The
disqualifiers are the same three Chapter 33.4 and Chapter 33.10 identified for
golden fixtures: a tool launch, a machine-specific path, and a timestamp. A
doctest is a byte-exact comparison, so anything Chapter 33.14 called an
undeclared input fails it on somebody else's machine and the team switches
doctests off — which is worse than not having them.
The careful row is the practical one. A dict's repr is stable because
insertion order is preserved, so {'errors': 1, 'status': 'FAIL'} is a
reliable expected value; what is not reliable is a long one wrapping differently
in an editor, so keep doctested returns small.
So the handover document ends up with two kinds of example: the shell
invocation in Section 8, which is prose because it exports an environment
variable and runs a simulator, and the per-function >>> examples, which are
tests. The first is checked by a human reading it; the second is checked by
python -m doctest.
10. Predict, Run, Observe
import argparse
import ast
import contextlib
import io
import re
SRC = '"""t - one line.\n\nExit codes:\n 0 ok\n 7 gone\n"""\n' \
"EXIT_OK = 0\n\ndef main():\n return EXIT_OK\n"
tree = ast.parse(SRC)
doc = ast.get_docstring(tree)
# A
print("A:", repr(doc.splitlines()[0]))
# B
documented = {int(m.group(1))
for m in re.finditer(r'^\s{2}(\d+)\s{2}', doc, re.M)}
print("B:", sorted(documented))
# C
reachable = {0}
print("C:", sorted(documented - reachable), sorted(reachable - documented))
# D
p = argparse.ArgumentParser(prog="t", description=doc)
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
p.print_help()
print("D:", "\nExit codes:" in buf.getvalue())The verified output:
A: 't - one line.'
B: [0, 7]
C: [7] []
D: FalseA — ast.get_docstring returns the string with the quotes removed, so the
first line is the one-line contract and nothing else.
B — the regex found both documented codes, including the fictional 7.
C — 7 is documented and unreachable; nothing is reachable and
undocumented. Section 5's two directions, one of which is populated.
D — and argparse with the default formatter did not preserve the
newline before Exit codes:. The table is in the help output as words and not
as a table, which is Section 4 in one boolean — the failure that looks like
success.
11. Exercises
Exercise 1 — the five questions. Answer each of Section 1's five for your own tool, in one line apiece.
Exercise 2 — the two to omit. Find a paragraph in your own docs that describes how the code works, and delete it.
Exercise 3 — docstring, not comment. Convert a leading comment in one of your scripts into a module docstring.
Exercise 4 — four readers. Confirm the same string reaches __doc__,
help(), --help and a linter.
Exercise 5 — the clause after the dash. Write the one-line contract for your tool, ending with what exit 0 means.
Exercise 6 — twenty-three lines. Draft the five sections and count the lines.
Exercise 7 — one screen. Check that your handover document fits on one.
Exercise 8 — reflowed. Pass your docstring to argparse without a
formatter and read the help output.
Exercise 9 — the lost table. Say which part of your documentation the default formatter destroys.
Exercise 10 — one extra argument. Add RawDescriptionHelpFormatter and
confirm the layout survives.
Exercise 11 — %(default)s. Replace a hard-coded default in a help string
with the placeholder.
Exercise 12 — the duplicated value. Explain what breaks when somebody
changes default= and not the help text.
Exercise 13 — the epilog. Put your configuration summary where a user is guaranteed to find it.
Exercise 14 — list your codes. Write your tool's exit-code table.
Exercise 15 — extract the documented set. Parse your own table with a regex.
Exercise 16 — extract the reachable set. Walk your AST for returned codes.
Exercise 17 — both directions. Compute both set differences and record them.
Exercise 18 — documented and unreachable. For any you find, decide whether to delete the row or wire the branch.
Exercise 19 — reachable and undocumented. Say what a wrapper does when it meets one.
Exercise 20 — three mutations. Break your documentation three ways and confirm the linter catches each.
Exercise 21 — no false positive. Confirm it stays quiet on the correct version.
Exercise 22 — what it cannot see. Name a code your linter would miss, and write the convention down.
Exercise 23 — five non-goals. List what your tool deliberately does not do.
Exercise 24 — the defect each prevents. For each, name what somebody would otherwise build.
Exercise 25 — status, not correctness. Write the sentence that stops your output being read as sign-off.
Exercise 26 — the retry. Say explicitly whether your tool retries, and why.
Exercise 27 — the example sets the variable. Make your worked example export the environment variable correctly.
Exercise 28 — fixing docs forced fixing code. Find a promise in your docs that the code does not keep.
Exercise 29 — add a doctest. Put a >>> example in one pure function's docstring and run python -m doctest.
Exercise 30 — write one from memory. Deliberately get the expected value slightly wrong and read the failure.
Exercise 31 — the two hazards. Explain why 75 and 33.3 both failed.
Exercise 32 — already installed. Say what doctest gives you that the Section 6 linter does not.
Exercise 33 — classify your examples. For six examples in your own docs, decide which could be doctests.
Exercise 34 — the three disqualifiers. Name them and the chapter that measured each.
Exercise 35 — switched off. Explain what happens to a doctest suite that fails on other machines.
Exercise 36 — keep returns small. Say why a long dict is a poor doctested value.
Exercise 37 — two kinds of example. Identify which of your examples is prose and which is a test.
12. Summary
- A handover document answers five questions and must not answer two. Describing how the code works is a second copy of the code that nothing keeps in step.
- A comment has one reader; a docstring has four — the file,
help(),--helpviadescription=__doc__, and a linter viaast.get_docstring. - The one-line contract ends with what exit 0 means, because that is the promise other scripts depend on.
- Twenty-three lines, one screen. Smallness is what gets it read.
description=__doc__silently destroys the exit-code table. The default formatter reflows the description into one paragraph; it takesRawDescriptionHelpFormatterto keep the newlines, and nothing warns.- Use
%(default)sso a default is never written twice. - A documented exit-code table is a checkable claim, and comparing it with the AST found three defects in one 24-line tool.
- Documented and unreachable is the dangerous direction: code 1 was declared for the failure case and never returned, so failing runs exited 0 and every wrapper was told a promise the tool does not keep.
- A documented code with no constant at all is pure fiction, copied or intended, and nothing notices.
- Reachable and undocumented leaves a wrapper with an exit code that has no defined meaning.
- Three broken modules caught, zero false positives — the doc linter is Chapter 33.9's kill rate applied to prose.
- State what the linter cannot see: a code returned by a called function, and any table that does not match its regex.
- Every non-goal prevents a defect somebody else would build. It is the scope boundary, not an apology.
- "It reports status, not correctness" is the most important sentence in a verification tool's documentation.
- Making the claim checkable forced fixing the code — code 1 acquired a
- The worked example is checkable by the standard library.
doctesttreats every>>>in every docstring as a test and needs no extra tooling. - Both examples written from memory failed —
75against75.0and33.3against33.333333333333336, the two arithmetic hazards Chapter 34.1 Section 7 measured. - Two of six example kinds belong in a doctest. The disqualifiers are a tool launch, a machine-specific path and a timestamp — Chapter 33.14's undeclared inputs, which would fail on somebody else's machine and get the suite switched off.
- Keep doctested return values small, since a dict's order is stable but a long repr is awkward.
- A handover document has two kinds of example: the shell invocation, which is prose a human checks, and the per-function
>>>examples, whichpython -m doctestchecks. branch because the linter would not accept the promise otherwise.
The last chapter of the module deals with what happens once other people's scripts depend on all of this: changing a flag, an output format or an exit code without breaking every wrapper around it.
Where this fits
Part of the Python curriculum.