Python Part II · Module 30 · Engineering Report Generation
Checking That a Report Is Correct
Six chapters have built a report, and nothing has yet asked whether it is true. That question is uncomfortable because a report is the one artefact in a flow with no independent oracle: a wrong simulation eventually disagrees with a waveform, but a wrong summary agrees with itself perfectly and is believed for months. The two assertions everybody writes are both correct and they catch, as this chapter measures, exactly a third of the ways a report goes wrong — and the property that matters most cannot be written as an assertion over numbers at all.
Advanced22 min readPythonPart IIAutomationReportingValidation
Part II · Module 30 · Chapter 30.7 · Checking That a Report Is Correct
1. Three Properties, Not One
The canonical purpose names three separate things to check, and they live at three different levels:
PROPERTIES = [
("totals add up", "the summary dict", "arithmetic over counts"),
("every job appears exactly once", "the summary vs the job set", "a multiset comparison"),
("a failing run never renders clean", "the rendered artefact", "a property of the output bytes"),
]
print(f"{'property':<36} {'about':<26} checked by")
for prop, about, how in PROPERTIES:
print(f"{prop:<36} {about:<26} {how}")
print()
print("properties expressible as arithmetic on counts:", 1)
print("properties needing the job set :", 1)
print("properties needing the rendered output :", 1)property about checked by
totals add up the summary dict arithmetic over counts
every job appears exactly once the summary vs the job set a multiset comparison
a failing run never renders clean the rendered artefact a property of the output bytes
properties expressible as arithmetic on counts: 1
properties needing the job set : 1
properties needing the rendered output : 1Three levels, and a validator that works at one of them cannot see the other two. That is the structure of the whole chapter.
2. The Annotation
assert summary["total"] == len(jobs)
assert summary["passed"] + summary["failed"] + summary["other"] == summary["total"]jobs = [f"job{i:03d}" for i in range(400)]
summary = {"total": 400, "passed": 360, "failed": 9, "other": 31}
def check(summary, jobs):
assert summary["total"] == len(jobs), "total disagrees with the job list"
assert (summary["passed"] + summary["failed"] + summary["other"]
== summary["total"]), "buckets do not sum to the total"
return "both assertions pass"
print(check(summary, jobs))
print()
print("total == len(jobs) :", summary["total"] == len(jobs))
print("passed + failed + other :", summary["passed"] + summary["failed"] + summary["other"])
print("which property is unchecked?:", "two of the three")both assertions pass
total == len(jobs) : True
passed + failed + other : 400
which property is unchecked?: two of the threeBoth assertions are correct and worth keeping. They are also cheap, which is why they are the ones that get written — and Sections 3 and 4 measure what they leave open.
3. The Count Can Be Right While the Set Is Wrong
The second property says every job appears exactly once, and a length comparison does not express that:
from collections import Counter
JOBS = [f"job{i:03d}" for i in range(400)]
# the summary counted job000 twice and never counted job399
counted = JOBS[:399] + ["job000"]
print("jobs submitted :", len(JOBS))
print("entries in summary :", len(counted))
print("total == len(jobs) :", len(counted) == len(JOBS))
print()
seen = Counter(counted)
duplicated = sorted(j for j, n in seen.items() if n > 1)
missing = sorted(set(JOBS) - set(counted))
print("duplicated:", duplicated)
print("missing :", missing)
print("exactly once for every job:", set(seen.values()) == {1} and not missing)jobs submitted : 400
entries in summary : 400
total == len(jobs) : True
duplicated: ['job000']
missing : ['job399']
exactly once for every job: Falsetotal == len(jobs) is True while one job is counted twice and another is not counted at all. A length is a single number and two errors that cancel produce the right one; job399 could be the night's only failure and the report would be arithmetically perfect without it.
The check that expresses the property is a multiset comparison, and it costs three lines:
from collections import Counter
def appears_exactly_once(summary_jobs, submitted):
seen = Counter(summary_jobs)
return {
"ok": seen == Counter(submitted),
"duplicated": sorted(j for j, n in seen.items() if n > 1),
"missing": sorted(set(submitted) - set(seen)),
"unexpected": sorted(set(seen) - set(submitted)),
}
JOBS = [f"job{i:03d}" for i in range(5)]
for label, counted in (
("correct", list(JOBS)),
("one duplicated, one missing", JOBS[:4] + ["job000"]),
("a job nobody submitted", JOBS[:4] + ["job999"]),
("the same jobs, different order", list(reversed(JOBS)))):
r = appears_exactly_once(counted, JOBS)
print(f"{label:<32} ok={str(r['ok']):<6} dup={r['duplicated']} "
f"missing={r['missing']} unexpected={r['unexpected']}")correct ok=True dup=[] missing=[] unexpected=[]
one duplicated, one missing ok=False dup=['job000'] missing=['job004'] unexpected=[]
a job nobody submitted ok=False dup=[] missing=['job004'] unexpected=['job999']
the same jobs, different order ok=True dup=[] missing=[] unexpected=[]The last row is why a Counter rather than a list comparison: order is not part of the property, and a validator that rejected a reordered summary would be wrong and would be switched off. Chapter 30.1 Section 3 did the accounting for the parse; this does it for the summary, and they are different steps that can disagree.
4. Six Ways to Break It, and What Survives
Arguing about coverage of a check is unproductive. Measuring it is not:
from collections import Counter
JOBS = [f"job{i:03d}" for i in range(20)]
TRUTH = {j: ("FAIL" if j in ("job003", "job007") else "PASS") for j in JOBS}
def summarise(statuses):
c = Counter(statuses.values())
return {"total": len(statuses),
"passed": c["PASS"], "failed": c["FAIL"],
"other": len(statuses) - c["PASS"] - c["FAIL"],
"jobs": list(statuses),
"failures": [j for j, s in statuses.items() if s == "FAIL"]}
def assert_1(s, jobs):
return s["total"] == len(jobs)
def assert_2(s, jobs):
return s["passed"] + s["failed"] + s["other"] == s["total"]
MUTATIONS = {}
m = summarise(TRUTH); m["total"] -= 1; m["passed"] -= 1
MUTATIONS["drop a job from the summary"] = m
m = summarise(TRUTH); m["failed"] += 1
MUTATIONS["count one job in two buckets"] = m
bad = dict(TRUTH); bad["job003"] = "PASS"
m = summarise(bad)
MUTATIONS["misclassify a FAIL as a PASS"] = m
st = dict(TRUTH)
m = summarise(st); m["jobs"] = m["jobs"][:-1] + [m["jobs"][0]]
MUTATIONS["duplicate one job, drop another"] = m
m = summarise(TRUTH); m["failures"] = []
MUTATIONS["empty the failure list, keep counts"] = m
m = summarise(TRUTH); m["rendered_rows"] = 0
MUTATIONS["render the table with no rows"] = m
print(f"{'mutation':<38} {'assert 1':>9} {'assert 2':>9} {'caught':>8}")
caught = 0
for name, s in MUTATIONS.items():
a1 = assert_1(s, JOBS)
a2 = assert_2(s, JOBS)
hit = not (a1 and a2)
caught += hit
print(f"{name:<38} {'pass' if a1 else 'FAIL':>9} "
f"{'pass' if a2 else 'FAIL':>9} {str(hit):>8}")
print()
print(f"mutations : {len(MUTATIONS)}")
print(f"caught by annotation: {caught}")
print(f"survived : {len(MUTATIONS) - caught}")mutation assert 1 assert 2 caught
drop a job from the summary FAIL pass True
count one job in two buckets pass FAIL True
misclassify a FAIL as a PASS pass pass False
duplicate one job, drop another pass pass False
empty the failure list, keep counts pass pass False
render the table with no rows pass pass False
mutations : 6
caught by annotation: 2
survived : 4Two of six. And the four survivors are not exotic:
Misclassifying a FAIL as a PASS is the single worst outcome a report can have, and both assertions pass because the arithmetic is internally consistent — 19 passes and 1 failure sums to 20 just as well as 18 and 2.
Duplicating one job while dropping another is Section 3, now measured.
Emptying the failure list while keeping the counts is a renderer that says "2 failures" above an empty table. The counts are right; the actionable content is gone.
Rendering no rows is the same thing one layer further out, and it is the one Chapter 30.5 Section 5 showed is indistinguishable from a clean night.
5. Checks for the Four Survivors
Three of the four need information the summary dict does not contain, which is why they were missed:
from collections import Counter
JOBS = [f"job{i:03d}" for i in range(20)]
TRUTH = {j: ("FAIL" if j in ("job003", "job007") else "PASS") for j in JOBS}
def validate(summary, submitted, statuses):
"""Each check names the information it needs beyond the summary."""
problems = []
if summary["total"] != len(submitted):
problems.append("total disagrees with the submitted list")
if (summary["passed"] + summary["failed"] + summary["other"]
!= summary["total"]):
problems.append("buckets do not sum to the total")
seen = Counter(summary["jobs"])
if seen != Counter(submitted):
problems.append(
f"job set mismatch: dup={sorted(j for j, n in seen.items() if n > 1)} "
f"missing={sorted(set(submitted) - set(seen))}")
recounted = Counter(statuses.values())
if recounted["FAIL"] != summary["failed"]:
problems.append(
f"failed count {summary['failed']} disagrees with the statuses "
f"({recounted['FAIL']})")
if summary["failed"] and not summary["failures"]:
problems.append(f"{summary['failed']} failures reported with an empty list")
if summary["failures"] and summary.get("rendered_rows") == 0:
problems.append("failure rows exist but none were rendered")
return problems
def summarise(statuses):
c = Counter(statuses.values())
return {"total": len(statuses), "passed": c["PASS"], "failed": c["FAIL"],
"other": len(statuses) - c["PASS"] - c["FAIL"],
"jobs": list(statuses),
"failures": [j for j, s in statuses.items() if s == "FAIL"],
"rendered_rows": c["FAIL"]}
good = summarise(TRUTH)
print("a correct report:", validate(good, JOBS, TRUTH) or "no problems")
print()
cases = {}
bad = dict(TRUTH); bad["job003"] = "PASS"
s = summarise(bad); cases["misclassify a FAIL as a PASS"] = (s, TRUTH)
s = summarise(TRUTH); s["jobs"] = s["jobs"][:-1] + [s["jobs"][0]]
cases["duplicate one job, drop another"] = (s, TRUTH)
s = summarise(TRUTH); s["failures"] = []
cases["empty the failure list"] = (s, TRUTH)
s = summarise(TRUTH); s["rendered_rows"] = 0
cases["render no rows"] = (s, TRUTH)
for name, (s, st) in cases.items():
probs = validate(s, JOBS, st)
print(f"{name:<34} {len(probs)} problem(s)")
for p in probs:
print(f" - {p}")a correct report: no problems
misclassify a FAIL as a PASS 1 problem(s)
- failed count 1 disagrees with the statuses (2)
duplicate one job, drop another 1 problem(s)
- job set mismatch: dup=['job000'] missing=['job019']
empty the failure list 1 problem(s)
- 2 failures reported with an empty list
render no rows 1 problem(s)
- failure rows exist but none were renderedAll four caught, and the interesting one is the first. Catching a misclassification required recounting the statuses independently of the summary — the summary cannot detect it, because the summary is what is wrong. That is the same principle as Chapter 30.6 Section 2's parser: a check must consult a source the thing under test did not produce.
6. A Failing Run Must Never Render Clean
The third property in the purpose is about the artefact, and it needs a different kind of test entirely:
def render(summary):
lines = [f"{summary['total']} jobs: {summary['passed']} passed, "
f"{summary['failed']} failed"]
if summary["failures"]:
lines.append("")
lines.append("failures:")
lines.extend(f" {f}" for f in summary["failures"])
else:
lines.append("No failures.")
return "\n".join(lines)
FAILING = {"total": 20, "passed": 18, "failed": 2, "other": 0,
"failures": ["job003", "job007"]}
BROKEN = {"total": 20, "passed": 18, "failed": 2, "other": 0, "failures": []}
for label, s in (("correct renderer", FAILING), ("renderer that lost the list", BROKEN)):
out = render(s)
print(f"--- {label}")
print(out)
print(f" mentions a failing job: {any(f in out for f in FAILING['failures'])}")
print(f" says 'No failures' : {'No failures' in out}")
print()--- correct renderer
20 jobs: 18 passed, 2 failed
failures:
job003
job007
mentions a failing job: True
says 'No failures' : False
--- renderer that lost the list
20 jobs: 18 passed, 2 failed
No failures.
mentions a failing job: False
says 'No failures' : TrueRead the second block carefully. It reports "2 failed" and "No failures." in the same four lines, which is a self-contradicting artefact that both of the annotation's assertions accept, because its counts are perfect.
The property is stated as a test over the rendered bytes:
def render(summary):
lines = [f"{summary['total']} jobs: {summary['passed']} passed, "
f"{summary['failed']} failed"]
if summary["failures"]:
lines.append("failures:")
lines.extend(f" {f}" for f in summary["failures"])
else:
lines.append("No failures.")
return "\n".join(lines)
def rendering_is_honest(summary, text):
"""A failing run must not render as clean, and must name its failures."""
problems = []
failing = summary["failed"] > 0
if failing and "No failures" in text:
problems.append("a failing run rendered the clean-night sentence")
if failing:
named = [f for f in summary["failures"] if f in text]
if not named:
problems.append("a failing run named none of its failures")
elif len(named) != len(summary["failures"]):
problems.append(f"named {len(named)} of {len(summary['failures'])} failures")
if not failing and "failures:" in text:
problems.append("a clean run rendered a failure section")
return problems
CASES = {
"correct": {"total": 20, "passed": 18, "failed": 2, "other": 0,
"failures": ["job003", "job007"]},
"list lost": {"total": 20, "passed": 18, "failed": 2, "other": 0,
"failures": []},
"half the list": {"total": 20, "passed": 18, "failed": 2, "other": 0,
"failures": ["job003", "job007"]},
"clean night": {"total": 20, "passed": 20, "failed": 0, "other": 0,
"failures": []},
}
for label, s in CASES.items():
text = render(s)
if label == "half the list":
text = text.replace(" job007\n", "").replace(" job007", "")
probs = rendering_is_honest(s, text)
print(f"{label:<16} {len(probs)} problem(s) {probs}")correct 0 problem(s) []
list lost 2 problem(s) ['a failing run rendered the clean-night sentence', 'a failing run named none of its failures']
half the list 1 problem(s) ['named 1 of 2 failures']
clean night 0 problem(s) []Four cases, and the half the list row is the one worth having: a renderer that drops some rows produces output that passes every count check, contains a failure table, and still hides a failure. Only a comparison between the summary's failure list and the rendered bytes finds it.
This is the module's method stated as a rule. Chapter 30.6 Section 2 parsed its own HTML rather than reading it; Chapter 30.5 Section 7 counted cells with an independent splitter. The generalisation is that a report's tests belong to its output, because the output is what somebody acts on and the functions that produced it can be individually correct while the artefact is wrong.
7. Prove Every Check Can Fail
A validator is code, and untested code that returns "no problems" is indistinguishable from a validator that cannot return anything else. Chapter 30.3 Section 3 found a predicate that was vacuously false; the same risk applies here:
from collections import Counter
JOBS = [f"job{i:03d}" for i in range(20)]
TRUTH = {j: ("FAIL" if j in ("job003", "job007") else "PASS") for j in JOBS}
def summarise(statuses):
c = Counter(statuses.values())
return {"total": len(statuses), "passed": c["PASS"], "failed": c["FAIL"],
"other": len(statuses) - c["PASS"] - c["FAIL"],
"jobs": list(statuses),
"failures": [j for j, s in statuses.items() if s == "FAIL"],
"rendered_rows": c["FAIL"]}
CHECKS = {
"total_matches": lambda s, sub, st: s["total"] == len(sub),
"buckets_sum": lambda s, sub, st: (s["passed"] + s["failed"] + s["other"]
== s["total"]),
"jobs_once": lambda s, sub, st: Counter(s["jobs"]) == Counter(sub),
"failed_recounted": lambda s, sub, st: (Counter(st.values())["FAIL"]
== s["failed"]),
"list_not_empty": lambda s, sub, st: not (s["failed"] and not s["failures"]),
"rows_rendered": lambda s, sub, st: not (s["failures"]
and s.get("rendered_rows") == 0),
"always_true": lambda s, sub, st: True,
}
def breakers():
s = summarise(TRUTH); s["total"] -= 1
yield "total_matches", s, TRUTH
s = summarise(TRUTH); s["failed"] += 1
yield "buckets_sum", s, TRUTH
s = summarise(TRUTH); s["jobs"] = s["jobs"][:-1] + [s["jobs"][0]]
yield "jobs_once", s, TRUTH
bad = dict(TRUTH); bad["job003"] = "PASS"
yield "failed_recounted", summarise(bad), TRUTH
s = summarise(TRUTH); s["failures"] = []
yield "list_not_empty", s, TRUTH
s = summarise(TRUTH); s["rendered_rows"] = 0
yield "rows_rendered", s, TRUTH
good = summarise(TRUTH)
print(f"{'check':<20} {'passes a good report':>21} {'rejects its breaker':>21}")
proven = {}
for name, s, st in breakers():
ok_good = CHECKS[name](good, JOBS, TRUTH)
ok_bad = CHECKS[name](s, JOBS, st)
proven[name] = ok_good and not ok_bad
print(f"{name:<20} {str(ok_good):>21} {str(not ok_bad):>21}")
print(f"{'always_true':<20} {str(CHECKS['always_true'](good, JOBS, TRUTH)):>21} "
f"{'no breaker exists':>21}")
print()
print("checks proven able to fail:", sorted(k for k, v in proven.items() if v))
print("checks with no breaker :", ["always_true"])
print("a check with no breaker is DEAD, regardless of what it returns.")check passes a good report rejects its breaker
total_matches True True
buckets_sum True True
jobs_once True True
failed_recounted True True
list_not_empty True True
rows_rendered True True
always_true True no breaker exists
checks proven able to fail: ['buckets_sum', 'failed_recounted', 'jobs_once', 'list_not_empty', 'rows_rendered', 'total_matches']
checks with no breaker : ['always_true']
a check with no breaker is DEAD, regardless of what it returns.Every check passes a correct report and rejects a report broken in its own specific way. The always_true row is in the table as a control: it passes the good report exactly like the others, and the only thing distinguishing it from a real check is that nobody can write input it rejects.
That is the discipline. A validator's test suite needs one deliberately broken report per check, and a check for which you cannot construct a breaker is not a check — it is a line that makes the output look thorough.
8. The Three Renderings Must Agree
Chapters 30.4 to 30.6 produced three artefacts from one summary, and each was tested in its own format's terms. A cross-format check catches what none of them can:
import csv
import io
import re
from collections import Counter
from html.parser import HTMLParser
SUMMARY = {"total": 20, "passed": 17, "failed": 3, "other": 0,
"failures": [("job003", 11), ("job007", 22), ("job012", 33)]}
def to_csv(s, drop=0):
buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=["test", "seed", "status"])
w.writeheader()
for t, seed in s["failures"][: len(s["failures"]) - drop]:
w.writerow({"test": t, "seed": seed, "status": "FAIL"})
return buf.getvalue()
def to_md(s, drop=0):
lines = ["| test | seed | status |", "|---|---|---|"]
for t, seed in s["failures"][: len(s["failures"]) - drop]:
lines.append(f"| {t} | {seed} | FAIL |")
return "\n".join(lines)
def to_html(s, drop=0):
rows = "".join(f"<tr><td>{t}</td><td>{seed}</td><td>FAIL</td></tr>"
for t, seed in s["failures"][: len(s["failures"]) - drop])
return f"<table>{rows}</table>"
class Rows(HTMLParser):
def __init__(self):
super().__init__()
self.rows = 0
def handle_starttag(self, tag, attrs):
if tag == "tr":
self.rows += 1
def count_csv(text):
return len(list(csv.DictReader(io.StringIO(text))))
def count_md(text):
return sum(1 for l in text.split("\n")
if l.startswith("| ") and not set(l) <= set("|-: "))- 1
def count_html(text):
p = Rows()
p.feed(text)
return p.rows
def cross_check(s, drop_html=0):
counts = {
"summary": s["failed"],
"csv": count_csv(to_csv(s)),
"markdown": count_md(to_md(s)),
"html": count_html(to_html(s, drop=drop_html)),
}
return counts, len(set(counts.values())) == 1
counts, agree = cross_check(SUMMARY)
print("all three renderers correct:")
for k, v in counts.items():
print(f" {k:<10} {v}")
print(" agree:", agree)
print()
counts, agree = cross_check(SUMMARY, drop_html=1)
print("html renderer drops one row:")
for k, v in counts.items():
print(f" {k:<10} {v}")
print(" agree:", agree)
print()
print("the html file is valid markup and its own tests pass.")
print("only the cross-format comparison sees the missing row.")all three renderers correct:
summary 3
csv 3
markdown 3
html 3
agree: True
html renderer drops one row:
summary 3
csv 3
markdown 3
html 2
agree: False
the html file is valid markup and its own tests pass.
only the cross-format comparison sees the missing row.The HTML with two rows is well-formed, self-contained, correctly escaped and passes every check in Chapter 30.6. It is also missing a failure, and the only thing that notices is the comparison against the other two renderings and the summary.
This is a cheap and unusually high-value check, because the three renderers are independent code paths over one data structure: the probability that all three drop the same row is small, so disagreement localises the bug immediately.
9. The Validator, Assembled
from collections import Counter
from dataclasses import dataclass, field
from typing import Dict, List, Tuple
@dataclass
class Validation:
problems: List[str] = field(default_factory=list)
checks_run: int = 0
@property
def ok(self):
return not self.problems
def record(self, name, passed, message):
self.checks_run += 1
if not passed:
self.problems.append(f"{name}: {message}")
def validate_report(summary, submitted, statuses, rendered: Dict[str, int],
text: str):
v = Validation()
v.record("total_matches", summary["total"] == len(submitted),
f"total {summary['total']} vs {len(submitted)} submitted")
v.record("buckets_sum",
summary["passed"] + summary["failed"] + summary["other"]
== summary["total"], "buckets do not sum to total")
seen = Counter(summary["jobs"])
v.record("jobs_once", seen == Counter(submitted),
f"dup={sorted(j for j, n in seen.items() if n > 1)} "
f"missing={sorted(set(submitted) - set(seen))}")
recounted = Counter(statuses.values())
v.record("failed_recounted", recounted["FAIL"] == summary["failed"],
f"summary says {summary['failed']}, statuses say {recounted['FAIL']}")
v.record("list_not_empty", not (summary["failed"] and not summary["failures"]),
f"{summary['failed']} failures with an empty list")
v.record("renderings_agree", len(set(rendered.values()) | {summary["failed"]}) == 1,
f"summary {summary['failed']} vs {rendered}")
failing = summary["failed"] > 0
v.record("no_clean_sentence", not (failing and "No failures" in text),
"a failing run rendered the clean-night sentence")
named = [f for f in summary["failures"] if str(f) in text]
v.record("failures_named",
(not failing) or len(named) == len(summary["failures"]),
f"named {len(named)} of {len(summary['failures'])}")
return v
JOBS = [f"job{i:03d}" for i in range(20)]
STATUSES = {j: ("FAIL" if j in ("job003", "job007") else "PASS") for j in JOBS}
c = Counter(STATUSES.values())
SUMMARY = {"total": 20, "passed": c["PASS"], "failed": c["FAIL"], "other": 0,
"jobs": list(STATUSES),
"failures": ["job003", "job007"]}
TEXT = "20 jobs: 18 passed, 2 failed\nfailures:\n job003\n job007"
good = validate_report(SUMMARY, JOBS, STATUSES,
{"csv": 2, "markdown": 2, "html": 2}, TEXT)
print(f"correct report : checks {good.checks_run} ok {good.ok} "
f"problems {good.problems}")
print()
print("one broken report per check:")
BREAK = [
("total_matches", {**SUMMARY, "total": 19}, STATUSES,
{"csv": 2, "markdown": 2, "html": 2}, TEXT),
("buckets_sum", {**SUMMARY, "other": 1}, STATUSES,
{"csv": 2, "markdown": 2, "html": 2}, TEXT),
("jobs_once", {**SUMMARY, "jobs": list(STATUSES)[:-1] + ["job000"]},
STATUSES, {"csv": 2, "markdown": 2, "html": 2}, TEXT),
("failed_recounted", SUMMARY, {**STATUSES, "job003": "PASS"},
{"csv": 2, "markdown": 2, "html": 2}, TEXT),
("list_not_empty", {**SUMMARY, "failures": []}, STATUSES,
{"csv": 2, "markdown": 2, "html": 2}, TEXT),
("renderings_agree", SUMMARY, STATUSES,
{"csv": 2, "markdown": 2, "html": 1}, TEXT),
("no_clean_sentence", SUMMARY, STATUSES,
{"csv": 2, "markdown": 2, "html": 2},
"20 jobs: 18 passed, 2 failed\nNo failures."),
("failures_named", SUMMARY, STATUSES,
{"csv": 2, "markdown": 2, "html": 2},
"20 jobs: 18 passed, 2 failed\nfailures:\n job003"),
]
rejected = 0
for target, s, st, rend, txt in BREAK:
v = validate_report(s, JOBS, st, rend, txt)
hit = any(p.startswith(target + ":") for p in v.problems)
rejected += hit
print(f" {target:<18} rejected by its own check: {hit}")
print()
print(f"checks : {good.checks_run}")
print(f"breakers : {len(BREAK)}")
print(f"each rejected : {rejected} of {len(BREAK)}")
print(f"every check proven able to fail: {rejected == len(BREAK)}")correct report : checks 8 ok True problems []
one broken report per check:
total_matches rejected by its own check: True
buckets_sum rejected by its own check: True
jobs_once rejected by its own check: True
failed_recounted rejected by its own check: True
list_not_empty rejected by its own check: True
renderings_agree rejected by its own check: True
no_clean_sentence rejected by its own check: True
failures_named rejected by its own check: True
checks : 8
breakers : 8
each rejected : 8 of 8
every check proven able to fail: TrueEight checks, eight breakers, each rejected by the check written for it. The last line is the one that makes the suite worth running: a report validator that has never rejected anything is a report validator with no evidence behind it, and the breakers are that evidence.
Note the checks span all three levels from Section 1 — arithmetic on counts, a multiset against the submitted set, an independent recount of the statuses, and two properties of the rendered text. The annotation's two assertions are the first two of eight, and keeping them is right; believing they were sufficient is what this chapter is about.
10. What the Flow Does When Validation Fails
Every check so far returns problems. Nothing has said what happens next, and that decision is the one that determines whether the validator has any effect at all.
RESPONSES = [
("log a warning and publish", "nobody reads it", "the wrong summary is believed"),
("publish with a banner", "visible, and still published", "a reader can discount it"),
("refuse to publish", "nothing to believe", "no report at all tonight"),
("publish the facts, suppress the verdict", "honest and partial", "needs a design"),
]
print(f"{'response':<42} {'property':<32} consequence")
for name, prop, cons in RESPONSES:
print(f"{name:<42} {prop:<32} {cons}")
print()
print("responses that leave a wrong number in circulation:",
sum(1 for r in RESPONSES if "believed" in r[2]))response property consequence
log a warning and publish nobody reads it the wrong summary is believed
publish with a banner visible, and still published a reader can discount it
refuse to publish nothing to believe no report at all tonight
publish the facts, suppress the verdict honest and partial needs a design
responses that leave a wrong number in circulation: 1The first response is the one most flows have, and it is the one this whole chapter argues against: a validator whose only effect is a line in a log that nobody opens has the same effect as no validator.
Refusing to publish is tempting and too blunt. On a night when the scheduler lost twenty jobs, the accounting check fires — and the nineteen real failures in the results that did arrive are exactly what somebody needs to see.
def publish(problems, facts):
"""Severity decides the response: a wrong number is fatal, a partial one is not."""
FATAL = {"jobs_once", "failed_recounted", "renderings_agree",
"no_clean_sentence", "failures_named"}
fatal = [p for p in problems if p.split(":")[0] in FATAL]
degraded = [p for p in problems if p.split(":")[0] not in FATAL]
if fatal:
return {"publish": False,
"reason": "the summary may be wrong, not merely incomplete",
"shown": fatal}
return {"publish": True,
"banner": degraded or None,
"shown": list(facts)}
FACTS = ["counts", "buckets", "new/fixed"]
for label, problems in (
("clean", []),
("incomplete run", ["total_matches: 380 of 400 accounted"]),
("summary may be wrong", ["failed_recounted: summary 1, statuses 2"]),
("both", ["total_matches: 380 of 400 accounted",
"jobs_once: dup=['job000']"])):
r = publish(problems, FACTS)
print(f"{label:<22} publish={str(r['publish']):<6} "
+ (f"banner={r.get('banner')}" if r["publish"] else f"reason={r['reason']}"))clean publish=True banner=None
incomplete run publish=True banner=['total_matches: 380 of 400 accounted']
summary may be wrong publish=False reason=the summary may be wrong, not merely incomplete
both publish=False reason=the summary may be wrong, not merely incompleteThe split is between incomplete and wrong, and it is the only distinction that matters here. An incomplete report is publishable with a banner, because the numbers it shows are true of the jobs it saw and the banner says how many it did not see. A report whose counts disagree with the statuses, or that renders a failing run without naming its failures, is not incomplete — it is false, and publishing it with a banner invites a reader to discount the banner and believe the number.
One more property is needed, and it is the reason this section is last:
HISTORY = [
("2026-02-16", True, []),
("2026-02-17", True, ["total_matches: 399 of 400"]),
("2026-02-18", False, ["failed_recounted: summary 1, statuses 2"]),
]
print(f"{'date':<12} {'published':>10} problems")
for date, pub, probs in HISTORY:
print(f"{date:<12} {str(pub):>10} {probs or 'none'}")
print()
suppressed = [h[0] for h in HISTORY if not h[1]]
print("nights with no report:", suppressed)
print("a reader who does not know that sees:", "the 17th as the latest run")
print("so a suppressed report must leave a visible marker, not a gap.")date published problems
2026-02-16 True none
2026-02-17 True ['total_matches: 399 of 400']
2026-02-18 False ['failed_recounted: summary 1, statuses 2']
nights with no report: ['2026-02-18']
a reader who does not know that sees: the 17th as the latest run
so a suppressed report must leave a visible marker, not a gap.A refusal to publish must itself be published. Suppressing a report leaves the previous night's as the most recent, which a reader reasonably interprets as the current state — the same reassuring-by-absence failure as Chapter 30.1 Section 6's empty baseline and Chapter 30.3 Section 3's vacuous predicate, arriving one last time at the level of the whole artefact.
11. Predict, Run, Observe
from collections import Counter
JOBS = ["a", "b", "c"]
# A
print("A:", len(["a", "b", "a"]) == len(JOBS))
# B
print("B:", Counter(["a", "b", "a"]) == Counter(JOBS))
# C
s = {"passed": 18, "failed": 2, "other": 0, "total": 20}
print("C:", s["passed"] + s["failed"] + s["other"] == s["total"])
s2 = {"passed": 19, "failed": 1, "other": 0, "total": 20}
print(" ", s2["passed"] + s2["failed"] + s2["other"] == s2["total"])
# D
print("D:", "No failures" in "20 jobs: 18 passed, 2 failed\nNo failures.")The verified output:
A: True
B: False
C: True
True
D: TrueA — the length is right. Section 3.
B — the multiset is not. The same three lines of code separate a correct summary from one that lost a job.
C — both sum correctly, and one of them has a FAIL counted as a PASS. Arithmetic consistency is not accuracy.
D — "2 failed" and "No failures." in one artefact, which only a check over the rendered text can see.
12. Exercises
Exercise 1 — three levels. For your own report, name the check you have at each of the three levels. Note any level with none.
Exercise 2 — the two assertions. Add them to your generator if they are absent, and run them on a real night.
Exercise 3 — right count, wrong set. Construct a summary where the total matches and a job is missing.
Exercise 4 — the multiset check. Implement it and run it over a week of your reports.
Exercise 5 — order is not the property. Explain why a list comparison would be wrong, and what happens to a validator that is too strict.
Exercise 6 — write six mutations. Break your own report six ways and record which of your checks fire.
Exercise 7 — your coverage fraction. Report how many of your mutations your current checks catch.
Exercise 8 — the worst one. Explain why misclassifying a FAIL as a PASS passes both assertions.
Exercise 9 — independent recount. Add a status recount that does not use the summary, and say what source it reads.
Exercise 10 — no oracle. Explain in three sentences why a wrong report survives longer than a wrong simulation.
Exercise 11 — counts without content. Construct a report that says "3 failures" above an empty table.
Exercise 12 — the self-contradicting artefact. Produce output containing both a failure count and the clean-night sentence.
Exercise 13 — a property of the bytes. Write the check that a failing run names its failures in the rendered text.
Exercise 14 — half the rows. Drop one failure row from your renderer and find the check that notices.
Exercise 15 — a breaker per check. Write one deliberately broken report for every check you have.
Exercise 16 — find a dead check. Look for a check in your flow for which you cannot construct a breaker.
Exercise 17 — the control. Add an always_true check to your table and explain what it demonstrates.
Exercise 18 — cross-format agreement. Count the failure rows in your CSV, Markdown and HTML and compare them with the summary.
Exercise 19 — independent code paths. Explain why disagreement between three renderers localises a bug quickly.
Exercise 20 — valid and wrong. Construct an HTML report that passes every Chapter 30.6 check and is missing a failure.
Exercise 21 — where the checks live. Decide whether your validator runs in the generator, in CI, or both, and justify it.
Exercise 22 — what it should do on failure. Decide what your flow does when the validator rejects a report, and say why that is better than publishing it.
Exercise 23 — the whole module. Take one real night and list, for each of Module 30's seven chapters, the defect that chapter would have caught.
Exercise 24 — the report you trust. Write down what evidence you have that your current report is correct, and what you would need to trust it.
13. Summary
- The purpose names three properties at three levels — arithmetic on counts, a comparison against the job set, and a property of the rendered bytes — and a validator at one level cannot see the other two.
- Both of the annotation's assertions are correct and worth keeping. They are also two of the eight checks the report needs.
total == len(jobs)is satisfied with one job duplicated and another missing. A length is one number and two errors cancel.- "Exactly once" is a multiset comparison, and
Counterrather than a list, because order is not part of the property. - Of six mutations the annotation caught two. The four survivors all render as plausible reports.
- Misclassifying a FAIL as a PASS passes both assertions, because 19 + 1 sums to 20 as well as 18 + 2 does.
- Catching it requires recounting the statuses independently — the summary cannot detect an error in the summary.
- A report has no independent oracle. A wrong summary is self-consistent, consulted by people with no other source, and believed.
- A renderer that loses the failure list produces "2 failed" and "No failures." in four lines, and every count check accepts it.
- A report's tests belong to its output, because that is what somebody acts on and the functions behind it can each be correct while the artefact is wrong.
- Dropping some rows passes every count check and is found only by comparing the summary's failure list against the rendered bytes.
- Every check needs a breaker. A check for which you cannot construct one is dead, and the
always_truecontrol passes a good report exactly like a real check does. - The three renderings must agree with the summary. Well-formed, correctly escaped HTML with a missing row passes all of Chapter 30.6's checks and fails this one.
- Three independent renderers over one structure localise a bug fast, because they are unlikely to drop the same row.
- A validator whose only effect is a log line has the same effect as no validator.
- Split incomplete from wrong. An incomplete report is publishable with a banner; one whose counts disagree with its statuses is false and is not.
- A refusal to publish must itself be published — a gap leaves last night's report as the apparent current state, which is reassuring-by-absence one last time.
Module 31 turns from reporting what a flow did to generating what it consumes — build inputs, register maps and repetitive structure — with the professional boundary stated before any code.
Where this fits
Part of the Python curriculum.