Python Part II · Module 33 · Testing and Debugging Automation
Why Automation Scripts Need Tests
The case for testing automation is usually made with analogies, and it does not need to be. This book has spent thirteen modules finding real defects in real flow code — a regex that silently discarded every fully-covered group, a subtraction that promoted eleven failing jobs to passing, a predicate incapable of ever returning true — and every one of them would have been caught by a single assertion written in the time it takes to read this sentence. This chapter collects six of them and writes those assertions.
Intermediate20 min readPythonPart IIAutomationTestingReliability
Part II · Module 33 · Chapter 33.1 · Why Automation Scripts Need Tests
1. What the Parser Decides
AUTHORITIES = [
("the simulator", "did the simulation run", "reports an exit code"),
("the testbench", "did this check pass", "prints a line"),
("your parser", "did the regression pass", "nothing checks it"),
("the dashboard", "shows what the parser decided", "trusts the parser"),
("the engineer", "reads the dashboard", "trusts the dashboard"),
]
print(f"{'who':<16} {'decides':<28} and")
for who, decides, and_ in AUTHORITIES:
print(f"{who:<16} {decides:<28} {and_}")
print()
print("things that check the simulator:", "the testbench")
print("things that check the parser :", "nothing, by default")who decides and
the simulator did the simulation run reports an exit code
the testbench did this check pass prints a line
your parser did the regression pass nothing checks it
the dashboard shows what the parser decided trusts the parser
the engineer reads the dashboard trusts the dashboard
things that check the simulator: the testbench
things that check the parser : nothing, by defaultThat asymmetry is the canonical purpose in a table. A design has a testbench; a testbench has a reviewer; a parser has a chain of people downstream who trust it and no mechanism above it. Chapter 30.7's callout made the same point about a report: it is the one artefact in a flow with no independent oracle.
2. The Annotation
assert classify("UVM_FATAL: ...") == "FAIL" # the cheapest test you will writedef classify(text):
if not text.strip():
return "EMPTY_LOG"
if "UVM_FATAL" in text:
return "FAIL"
if "TEST DONE" not in text:
return "NO_FINISH"
if "UVM_ERROR" in text:
return "FAIL"
return "PASS"
assert classify("UVM_FATAL: null handle in write()") == "FAIL"
print("the assertion passed")
print()
print("characters of test code:", len('assert classify("UVM_FATAL: ...") == "FAIL"'))
print("dependencies :", 0)
print("licences :", 0)
print("seconds to run :", "under one, by inspection")the assertion passed
characters of test code: 43
dependencies : 0
licences : 0
seconds to run : under one, by inspectionForty-three characters, no imports, no framework, no licence. The comment is accurate: this is the cheapest test in the book, and it is a real test — it executes the real function and fails if the behaviour changes.
It is also enough to catch a defect this book actually found. The rest of the chapter does that five more times.
3. Six Real Defects, Six Assertions
Every defect below was measured earlier in this book. For each one, the assertion that would have failed.
import re
# 29.1 §5 — the pattern requires a decimal point, so `100%` rows vanish
PAT_BAD = re.compile(r"(?P<name>\S+)\s+(?P<pct>\d+\.\d+)%")
PAT_GOOD = re.compile(r"^(?P<name>\w+)\s+(?P<pct>\d+(?:\.\d+)?)%", re.M)
def pct_rows(pattern, text, anchored):
if anchored:
return [m.group("name") for m in pattern.finditer(text)]
return [m.group("name") for m in
(pattern.search(l) for l in text.splitlines()) if m]
REPORT = "cg_axi_burst 100%\ncg_axi_len 74.22%\ncg_reset 100%\n"
bad = pct_rows(PAT_BAD, REPORT, anchored=False)
good = pct_rows(PAT_GOOD, REPORT, anchored=True)
print("groups in the report :", len(REPORT.strip().splitlines()))
print("the shipped pattern parsed:", len(bad), bad)
print("the fixed pattern parsed :", len(good), good)
print()
try:
assert len(bad) == 3, f"parsed {len(bad)} of 3 rows"
except AssertionError as exc:
print("the assertion that would have caught it:")
print(" assert len(parse(REPORT)) == 3")
print(" AssertionError:", exc)groups in the report : 3
the shipped pattern parsed: 1 ['cg_axi_len']
the fixed pattern parsed : 3 ['cg_axi_burst', 'cg_axi_len', 'cg_reset']
the assertion that would have caught it:
assert len(parse(REPORT)) == 3
AssertionError: parsed 1 of 3 rowsOne row of three. Chapter 29.1 Section 5 measured this reporting 53.67% coverage against a true 76.84%, and a three-line fixture with an expected count of 3 catches it in milliseconds.
from collections import Counter
# 30.1 §4 — with five statuses, total - failed is not passed
STATUSES = Counter({"PASS": 380, "FAIL": 9, "TIMEOUT": 4, "NO_RUN": 6, "INFRA": 1})
total = sum(STATUSES.values())
def passed_wrong(counts):
return sum(counts.values()) - counts["FAIL"]
def passed_right(counts):
return counts["PASS"]
print("total :", total)
print("passed_wrong() :", passed_wrong(STATUSES))
print("passed_right() :", passed_right(STATUSES))
print()
try:
assert passed_wrong(STATUSES) == STATUSES["PASS"], (
f"{passed_wrong(STATUSES)} != {STATUSES['PASS']}")
except AssertionError as exc:
print("the assertion:")
print(' assert passed(counts) == counts["PASS"]')
print(" AssertionError:", exc)total : 400
passed_wrong() : 391
passed_right() : 380
the assertion:
assert passed(counts) == counts["PASS"]
AssertionError: 391 != 380Eleven jobs promoted from failing to passing, and one assertion over a five-status fixture says so.
from fractions import Fraction
# 30.3 §3 — the flaky predicate cannot fire on one run per test
def flaky_set(fails, runs):
rates = {t: Fraction(f, runs) for t, f in fails.items()}
return {t for t, r in rates.items() if 0 < r < 1}
ONE_RUN = {"a": 0, "b": 1}
TWENTY = {"a": 0, "b": 3}
print("runs=1 flaky set:", flaky_set(ONE_RUN, 1) or "set()")
print("runs=20 flaky set:", flaky_set(TWENTY, 20) or "set()")
print()
try:
assert flaky_set({"b": 1}, 2), "the predicate returned nothing on data that should fire"
print("the assertion that proves the predicate CAN fire: passed")
except AssertionError as exc:
print("AssertionError:", exc)
try:
assert flaky_set(ONE_RUN, 1), "no flaky test found with runs=1"
except AssertionError as exc:
print()
print("the assertion that exposes the vacuous case:")
print(" assert flaky_set(fails, runs=1)")
print(" AssertionError:", exc)runs=1 flaky set: set()
runs=20 flaky set: {'b'}
the assertion that proves the predicate CAN fire: passed
the assertion that exposes the vacuous case:
assert flaky_set(fails, runs=1)
AssertionError: no flaky test found with runs=1Two assertions, and the first is the one Chapter 30.3 Section 3's whole argument needed: a test proving the predicate is capable of returning something. That is Chapter 33.9's subject and it starts here.
from collections import Counter
# 30.7 §3 — the count can be right while the set is wrong
JOBS = [f"job{i:03d}" for i in range(5)]
SUMMARY_JOBS = JOBS[:4] + ["job000"] # one duplicated, one missing
print("len(summary) == len(jobs):", len(SUMMARY_JOBS) == len(JOBS))
print("Counter equal :", Counter(SUMMARY_JOBS) == Counter(JOBS))
print()
try:
assert Counter(SUMMARY_JOBS) == Counter(JOBS), (
f"dup={sorted(j for j, n in Counter(SUMMARY_JOBS).items() if n > 1)} "
f"missing={sorted(set(JOBS) - set(SUMMARY_JOBS))}")
except AssertionError as exc:
print("the assertion:")
print(" assert Counter(summary_jobs) == Counter(submitted)")
print(" AssertionError:", exc)len(summary) == len(jobs): True
Counter equal : False
the assertion:
assert Counter(summary_jobs) == Counter(submitted)
AssertionError: dup=['job000'] missing=['job004']The length check passes and the multiset check fails — the one-character difference between a test that protects you and a test that does not.
# 31.1 §4 — one character in a Python range, four double grants in hardware
def gen_arb(n, bug=False):
out = []
for i in range(n):
rng = range(i) if not bug else range(max(0, i - 1))
higher = " | ".join(f"req[{j}]" for j in rng)
out.append(f"gnt[{i}] = req[{i}]" + (f" & ~({higher})" if higher else ""))
return out
def grants(lines, req_bits):
"""Evaluate the generated expressions against a request vector."""
g = []
for i, line in enumerate(lines):
expr = line.split("=", 1)[1]
val = eval(expr.replace("req[", "r[").replace("~", "not ")
.replace("|", "or").replace("&", "and"),
{}, {"r": req_bits})
g.append(bool(val))
return g
good = gen_arb(4)
bad = gen_arb(4, bug=True)
REQ = [True, True, False, False]
print("requests :", REQ)
print("good generator:", grants(good, REQ))
print("bad generator :", grants(bad, REQ))
print()
try:
assert sum(grants(bad, REQ)) <= 1, f"{sum(grants(bad, REQ))} simultaneous grants"
except AssertionError as exc:
print("the assertion:")
print(" assert sum(grants) <= 1 # a priority arbiter grants at most one")
print(" AssertionError:", exc)requests : [True, True, False, False]
good generator: [True, False, False, False]
bad generator : [True, True, False, False]
the assertion:
assert sum(grants) <= 1 # a priority arbiter grants at most one
AssertionError: 2 simultaneous grantsThe generated arbiter granted two channels at once, and the assertion is a property of the circuit rather than of the string — which is Chapter 31.4's rule about testing the artefact.
# 32.7 §4 — the exit status is taken modulo 256
def exit_code_from(error_count):
return error_count # the natural-looking version
def exit_code_right(error_count):
return 0 if error_count == 0 else 1
print(f"{'errors':>8} {'as a count':>12} {'what a shell sees':>19} {'classified':>12}")
for n in (0, 1, 255, 256, 512):
print(f"{n:>8} {exit_code_from(n):>12} {exit_code_from(n) % 256:>19} "
f"{exit_code_right(n):>12}")
print()
try:
assert exit_code_from(256) % 256 != 0, "256 errors reported success"
except AssertionError as exc:
print("the assertion:")
print(" assert exit_code(errors) % 256 != 0 or errors == 0")
print(" AssertionError:", exc) errors as a count what a shell sees classified
0 0 0 0
1 1 1 1
255 255 255 1
256 256 0 1
512 512 0 1
the assertion:
assert exit_code(errors) % 256 != 0 or errors == 0
AssertionError: 256 errors reported success4. The Tally
DEFECTS = [
("29.1 §5", "a regex dropped every 100% row", "coverage 23 points low", 1),
("30.1 §4", "total - failed is not passed", "11 jobs promoted", 1),
("30.3 §3", "a predicate that cannot fire", "flaky tests never found", 1),
("30.7 §3", "right count, wrong set", "a job silently missing", 1),
("31.1 §4", "range(i) -> range(i-1)", "2 grants at once", 1),
("32.7 §4", "sys.exit(error_count)", "256 errors report PASS", 1),
]
print(f"{'where':<10} {'the defect':<34} {'consequence':<26} asserts")
for where, what, cost, n in DEFECTS:
print(f"{where:<10} {what:<34} {cost:<26} {n}")
print()
print("defects collected here :", len(DEFECTS))
print("assertions needed in total :", sum(d[3] for d in DEFECTS))
print("frameworks needed :", 0)
print("licences needed :", 0)
print("every one was found by :", "this book, measuring rather than reasoning")where the defect consequence asserts
29.1 §5 a regex dropped every 100% row coverage 23 points low 1
30.1 §4 total - failed is not passed 11 jobs promoted 1
30.3 §3 a predicate that cannot fire flaky tests never found 1
30.7 §3 right count, wrong set a job silently missing 1
31.1 §4 range(i) -> range(i-1) 2 grants at once 1
32.7 §4 sys.exit(error_count) 256 errors report PASS 1
defects collected here : 6
assertions needed in total : 6
frameworks needed : 0
licences needed : 0
every one was found by : this book, measuring rather than reasoningSix defects, six assertions. That is the whole argument, and it is worth noticing what kind of argument it is: not that bugs are possible in principle, but that these specific bugs occurred in code written carefully for this book, by an author who knew the subject and was looking for problems.
5. The Blast Radius
The canonical purpose says the parser decides whether a chip's regression passed. Here is what that multiplies out to:
JOBS_PER_NIGHT = 400
NIGHTS_BEFORE_NOTICED = 60
PARSER_LINES = 50
verdicts = JOBS_PER_NIGHT * NIGHTS_BEFORE_NOTICED
print(f"jobs per night : {JOBS_PER_NIGHT}")
print(f"nights before anyone looks: {NIGHTS_BEFORE_NOTICED}")
print(f"verdicts the parser issued: {verdicts:,}")
print(f"lines of python behind them: {PARSER_LINES}")
print(f"verdicts per line : {verdicts // PARSER_LINES:,}")
print()
COMPARE = [
("a wrong line of RTL", "one block", "caught by its own testbench"),
("a wrong line of testbench", "one test", "caught when the test is reviewed"),
("a wrong line of parser", "every job, every night", "caught by nothing"),
]
print(f"{'a defect in':<28} {'affects':<24} {'and is'}")
for what, scope, caught in COMPARE:
print(f"{what:<28} {scope:<24} {caught}")jobs per night : 400
nights before anyone looks: 60
verdicts the parser issued: 24,000
lines of python behind them: 50
verdicts per line : 480
a defect in affects and is
a wrong line of RTL one block caught by its own testbench
a wrong line of testbench one test caught when the test is reviewed
a wrong line of parser every job, every night caught by nothingTwenty-four thousand verdicts from fifty lines. The ratio is the argument: a line of flow code has a blast radius thousands of times larger than a line of RTL, and a smaller fraction of the project's review attention.
That is also why "it is only a small script" is the wrong axis. The relevant measure is not how much code there is but how many decisions it makes, and a fifty-line parser that runs on four hundred logs a night makes more decisions per day than most modules of RTL make in a project.
6. Three Objections
OBJECTIONS = [
("it is only fifty lines",
"fifty lines issuing 480 verdicts each — §5"),
("I would notice if it were wrong",
"29.1's defect under-reported coverage; nobody investigates a low number"),
("the regression itself is the test",
"the regression's verdict IS the parser's output — 30.7's no-oracle problem"),
("I will write tests when it stabilises",
"it stabilised; that is why nobody reads it any more"),
]
print(f"{'objection':<38} answer")
for obj, ans in OBJECTIONS:
print(f"{obj:<38} {ans}")
print()
print("objections:", len(OBJECTIONS))
print("the third is the serious one because:",
"it is circular, and the circle is invisible from inside")objection answer
it is only fifty lines fifty lines issuing 480 verdicts each — §5
I would notice if it were wrong 29.1's defect under-reported coverage; nobody investigates a low number
the regression itself is the test the regression's verdict IS the parser's output — 30.7's no-oracle problem
I will write tests when it stabilises it stabilised; that is why nobody reads it any more
objections: 4
the third is the serious one because: it is circular, and the circle is invisible from insideThe third objection is the one worth taking seriously, because it sounds like a complete answer. The regression is the test assumes the regression has an independent view of whether it passed — and it does not: the regression's verdict is literally the parser's return value. Chapter 30.7's callout named this as the report having no independent oracle, and the parser is the same artefact one layer earlier.
The second objection is answered by a specific property of the defects in Section 3. Five of the six fail in the direction nobody investigates: coverage too low, a flaky set that is empty, a job quietly missing, a count that looks plausible, a clean exit code. Only the arbiter's double grant is loud, and it is loud in hardware, which is the most expensive place to find out.
7. From One Assertion to a Suite
The annotation's single assert is already a test. The only thing a framework adds is bookkeeping:
import io
import unittest
def classify(text):
if not text.strip():
return "EMPTY_LOG"
if "UVM_FATAL" in text:
return "FAIL"
if "TEST DONE" not in text:
return "NO_FINISH"
if "UVM_ERROR" in text:
return "FAIL"
return "PASS"
class ClassifyTests(unittest.TestCase):
def test_fatal(self):
self.assertEqual(classify("UVM_FATAL boom\n"), "FAIL")
def test_error_with_banner(self):
self.assertEqual(classify("UVM_ERROR x\nTEST DONE\n"), "FAIL")
def test_clean(self):
self.assertEqual(classify("UVM_INFO ok\nTEST DONE\n"), "PASS")
def test_empty(self):
self.assertEqual(classify(""), "EMPTY_LOG")
def test_truncated(self):
self.assertEqual(classify("UVM_INFO started\n"), "NO_FINISH")
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"errors": len(res.errors), "ok": res.wasSuccessful()}
print("suite result:", run(ClassifyTests))
print()
print("assertions :", 5)
print("behaviours pinned:", ["FAIL via fatal", "FAIL via error", "PASS",
"EMPTY_LOG", "NO_FINISH"])suite result: {'run': 5, 'failures': 0, 'errors': 0, 'ok': True}
assertions : 5
behaviours pinned: ['FAIL via fatal', 'FAIL via error', 'PASS', 'EMPTY_LOG', 'NO_FINISH']Five assertions covering all five outcomes of the classifier, run programmatically so the result is a dictionary rather than a wall of text.
That runner is worth keeping. Every later chapter in this module uses it, and Chapter 33.9 depends on it — because asking how many tests failed is how you find out whether the tests were protecting anything.
8. Where They Live and When They Run
PLACEMENTS = [
("beside the code, in tests/", "found by anybody reading the module", "yes"),
("in the same file as the code", "fine for one function, not for ten", "sometimes"),
("in a separate repository", "drifts within a month", "no"),
("nowhere, run by hand", "not run", "no"),
]
print(f"{'where the tests live':<34} {'consequence':<42} keep")
for where, why, keep in PLACEMENTS:
print(f"{where:<34} {why:<42} {keep}")
print()
TRIGGERS = [
("before every commit", "a pre-commit hook", "bypassable"),
("in CI on every merge request", "cannot be skipped", "the minimum"),
("nightly, before the regression", "catches a tool upgrade — 33.4", "worth it"),
("only when something breaks", "too late by definition", "no"),
]
print(f"{'when they run':<34} {'mechanism':<32} verdict")
for when, how, verdict in TRIGGERS:
print(f"{when:<34} {how:<32} {verdict}")where the tests live consequence keep
beside the code, in tests/ found by anybody reading the module yes
in the same file as the code fine for one function, not for ten sometimes
in a separate repository drifts within a month no
nowhere, run by hand not run no
when they run mechanism verdict
before every commit a pre-commit hook bypassable
in CI on every merge request cannot be skipped the minimum
nightly, before the regression catches a tool upgrade — 33.4 worth it
only when something breaks too late by definition noThe third row of the second table is the one specific to this domain. A flow's tests are not only protecting against your changes — Chapter 33.4 is about a tool upgrade renaming a message, which changes nothing in your repository and breaks your parser. Running the suite nightly before the regression turns that into a failed test instead of a wrong verdict, which is the difference between a ten-minute fix and a week of mistrusted results.
9. Where the Expected Value Comes From
An assertion compares two things, and the second one has to come from somewhere. There are three places to get it, they look equally reasonable in a diff, and only two of them can ever fail:
import io
import unittest
LOG = ("UVM_INFO a\n"
"UVM_ERROR drv.sv(88) bad\n"
"UVM_ERROR mon.sv(12) bad\n"
"TEST DONE\n")
def good(text):
return sum(1 for l in text.splitlines() if l.startswith("UVM_ERROR"))
def doubles(text): # a plausible typo: 2 instead of 1
return sum(2 for l in text.splitlines() if l.startswith("UVM_ERROR"))
def uses_in(text): # `in` instead of startswith
return sum(1 for l in text.splitlines() if "UVM_ERROR" in l)
def build(fn, kind):
class T(unittest.TestCase):
def test_count(self):
if kind == "calls":
expected = fn(LOG) # ask the code
elif kind == "reimplements":
expected = sum(1 for l in LOG.splitlines()
if l.startswith("UVM_ERROR"))
else:
expected = 2 # counted by hand
self.assertEqual(fn(LOG), expected)
return T
def passes(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
return unittest.TextTestRunner(stream=io.StringIO(),
verbosity=0).run(suite).wasSuccessful()
PARSERS = [("correct", good), ("counts 2 per line", doubles),
("`in` not startswith", uses_in)]
KINDS = ["calls", "reimplements", "literal"]
print(f"{'parser under test':<22} {'calls fn':<10} {'reimplements':<14} literal 2")
for label, fn in PARSERS:
row = [passes(build(fn, k)) for k in KINDS]
print(f"{label:<22} {str(row[0]):<10} {str(row[1]):<14} {row[2]}")
print()
print("the 'calls fn' column is:", "all True — it cannot fail")
print("because it asserts :", "fn(LOG) == fn(LOG)")parser under test calls fn reimplements literal 2
correct True True True
counts 2 per line True False False
`in` not startswith True True True
the 'calls fn' column is: all True — it cannot fail
because it asserts : fn(LOG) == fn(LOG)The first column is True all the way down. Deriving the expected value by
calling the function under test produces a test that passes against a correct
parser, against a parser that doubles every count, and against any other
parser you can write — because the assertion reduces to fn(LOG) == fn(LOG).
It is not a weak test; it is not a test.
That shape is easy to write by accident. It usually arrives as a helper —
expected = classify(LOG)["errors"] at the top of a test file, reused by six
assertions — and in review it reads like careful setup.
The middle column is better and still not what you want. Reimplementing the counting logic inside the test killed the doubling mutant, because the two implementations disagree there. But it is a second copy of the same idea by the same author on the same afternoon, so it agrees with the original wherever the original is wrong for a conceptual reason rather than a typing one.
The third column is the one to write. expected = 2 because somebody read
four lines and counted two — a fact about the fixture, established outside the
code, which is exactly what Chapter 33.4's provenance comment is for.
Then look at the last row, which is the honest limit of this whole section:
LOG = ("UVM_INFO a\n"
"UVM_ERROR drv.sv(88) bad\n"
"UVM_ERROR mon.sv(12) bad\n"
"TEST DONE\n")
HOSTILE = LOG + "echo UVM_ERROR about to retry\n"
def good(text):
return sum(1 for l in text.splitlines() if l.startswith("UVM_ERROR"))
def uses_in(text):
return sum(1 for l in text.splitlines() if "UVM_ERROR" in l)
print(f"{'fixture':<12} {'good':>6} {'uses_in':>8} distinguishes")
for name, text in (("LOG", LOG), ("HOSTILE", HOSTILE)):
a, b = good(text), uses_in(text)
print(f"{name:<12} {a:>6} {b:>8} {a != b}")
print()
print("no expected-value style kills `uses_in` on LOG:", True)
print("because on LOG the two parsers :", "agree")
print("what it needs instead :", "a different FIXTURE")fixture good uses_in distinguishes
LOG 2 2 False
HOSTILE 2 3 True
no expected-value style kills `uses_in` on LOG: True
because on LOG the two parsers : agree
what it needs instead : a different FIXTUREChanging where the expected value comes from cannot help when the two
parsers agree on the input. uses_in counts the same 2 on this log, so all
three columns pass, and the only thing that separates them is a fixture
containing a line that mentions a severity without being one — which is
Chapter 33.5's whole subject, and the reason it is a chapter rather than a
paragraph.
So there are two independent questions, and they are easy to run together. Where does the expected value come from? decides whether the test can fail at all. What is in the fixture? decides which wrong answers it can see. Chapter 33.9's mutation score measures the two of them jointly, which is why a survivor there sends you to look at both.
10. Predict, Run, Observe
from collections import Counter
def classify(text):
if "UVM_FATAL" in text:
return "FAIL"
return "PASS"
# A
print("A:", classify("UVM_FATAL: null handle") == "FAIL")
# B
print("B:", classify("UVM_ERROR: mismatch\nTEST DONE\n") == "FAIL")
# C
print("C:", len(["a", "b", "a"]) == len(["a", "b", "c"]),
Counter(["a", "b", "a"]) == Counter(["a", "b", "c"]))
# D
print("D:", 256 % 256, 512 % 256)The verified output:
A: True
B: False
C: True False
D: 0 0A — the annotation's assertion, as a boolean. It passes.
B — and the same two-line classifier gets UVM_ERROR wrong, which is what a second assertion is for. One passing test does not mean a correct function.
C — Chapter 30.7's defect in one line: the length agrees and the multiset does not.
D — 256 and 512 errors both report success.
11. Exercises
Exercise 1 — who checks your parser. Draw the chain of trust from your parser to the engineer reading the dashboard, and mark every point with an independent check.
Exercise 2 — the cheapest test. Write the forty-character assertion for your own classifier and run it.
Exercise 3 — a second assertion. Find an input your classifier gets wrong that your first assertion does not cover.
Exercise 4 — your own 29.1. Check whether your coverage parser handles a row printed as 100%.
Exercise 5 — your own 30.1. Check whether anything in your flow computes passes by subtraction.
Exercise 6 — a predicate that cannot fire. Look for a check in your flow whose output has never varied.
Exercise 7 — count versus set. Replace one length comparison with a multiset comparison and report what it found.
Exercise 8 — a property, not a string. Write an assertion about a generated artefact's behaviour rather than its text.
Exercise 9 — the exit code. Search your flow for sys.exit applied to a count.
Exercise 10 — six for six. Pick the three defects above most likely to exist in your flow and write their assertions.
Exercise 11 — your blast radius. Compute verdicts-per-line for your own parser.
Exercise 12 — compare with RTL. Explain to a colleague why a flow line has a larger blast radius than an RTL line.
Exercise 13 — decisions, not lines. Count the decisions your smallest flow script makes in a day.
Exercise 14 — the circular objection. Write the two-sentence answer to "the regression is the test".
Exercise 15 — which direction. For each defect in your flow that you know of, say whether it failed loudly or quietly.
Exercise 16 — the quiet majority. Explain why a defect that makes results look worse survives longest.
Exercise 17 — one assert to a suite. Turn your single assertion into five covering every outcome of the function.
Exercise 18 — all outcomes. List your classifier's possible return values and confirm each has a test.
Exercise 19 — a programmatic runner. Write the runner that returns counts, and say why that matters for Chapter 33.9.
Exercise 20 — no durations. Find a place where your tooling prints a duration into something that gets compared.
Exercise 21 — where yours live. Say where your flow's tests live and whether a new reader would find them.
Exercise 22 — when yours run. Say when they run and what can bypass that.
Exercise 23 — the nightly run. Decide whether your suite should run before the regression, and justify it.
Exercise 24 — the tool upgrade. Describe what happens to your parser the next time a tool renames a message.
Exercise 25 — the three sources. For one of your tests, say which of the three places its expected value came from.
Exercise 26 — find a tautology. Search your suite for an expected value computed by calling the code under test.
Exercise 27 — reduce it. Write out what that assertion compares, and show it is the same expression twice.
Exercise 28 — how it got there. Explain why a shared expected = parse(FIXTURE) helper reads as good practice in review.
Exercise 29 — the second copy. Say why reimplementing the logic in the test is better than calling it and still not enough.
Exercise 30 — count by hand. Take one fixture, count a fact in it yourself, and write the literal.
Exercise 31 — two questions. For a test of yours that passes against a mutation, decide whether the problem is the expected value or the fixture.
12. Summary
- A design has a testbench and a parser has nothing above it — the parser decides whether the regression passed, and every layer downstream trusts it.
- The annotation is forty-two characters, needs no framework and no licence, and it is a real test of real behaviour.
- Six defects this book measured are each caught by one assertion — a regex that dropped every
100%row, a subtraction that promoted eleven jobs, a vacuously false predicate, a right count with a wrong set, a one-character range bug that double-granted, and an exit code taken modulo 256. - These are not invented examples. They occurred in code written carefully, by someone who knew the subject and was looking for problems.
- Twenty-four thousand verdicts came from fifty lines — 480 per line, which is a blast radius thousands of times larger than a line of RTL and a fraction of the review attention.
- "It is only a small script" measures the wrong thing. The relevant number is how many decisions it makes.
- "The regression is the test" is circular, because the regression's verdict is the parser's output.
- Five of the six defects fail in the direction nobody investigates — too low, empty, missing, plausible, clean — and the only loud one is loud in hardware.
- A framework adds bookkeeping, not testing. Five assertions covered all five outcomes of the classifier.
- Use a programmatic runner that reports counts, because
unittest's own output contains a duration this book will not publish — and a suite your flow parses should report counts and names for the same reason. - Run the suite nightly before the regression, because a tool upgrade breaks your parser without changing anything in your repository.
- An assertion's expected value has three possible sources, and deriving it by calling the code under test produces a test that cannot fail — it reduces to
fn(x) == fn(x). - Reimplementing the logic inside the test is better and still not independent: it agrees with the original wherever the original is conceptually wrong.
- A hand-counted literal is the one that works, and its justification is the provenance comment of Chapter 33.4.
- No expected-value style helps when two parsers agree on the fixture — that is a fixture problem, and Chapter 33.5's subject.
The next chapter draws the line between the part of a flow that can be tested in milliseconds and the thin layer that genuinely cannot.
Where this fits
Part of the Python curriculum.