Python Part II · Module 26 · Regression Automation
Collecting Results From Many Jobs
Four hundred jobs have run and each was supposed to leave a structured result in its own directory. Collecting them looks like a glob and a loop, and the version everybody writes first produces a summary that is arithmetically consistent, internally plausible, and capable of counting a result from last night's regression as one of tonight's passes. The fix is one parameter, and what makes it work is that it compares against something the results themselves cannot influence.
Advanced19 min readPythonPart IIAutomationRegressionVerification Flow
Part II · Module 26 · Chapter 26.6 · Collecting Results From Many Jobs
1. Each Job Left a Record
import json
from pathlib import Path
JOBS = [("axi_smoke", 1), ("axi_reset", 2), ("axi_burst", 3),
("axi_hang", 4), ("axi_lic", 5), ("axi_gone", 6)]
RECORDS = [
{"test": "axi_smoke", "seed": 1, "status": "PASS"},
{"test": "axi_reset", "seed": 2, "status": "FAIL", "reason": "UVM_ERROR"},
{"test": "axi_burst", "seed": 3, "status": "FAIL", "reason": "UVM_ERROR"},
{"test": "axi_hang", "seed": 4, "status": "NO_FINISH", "reason": "TIMEOUT"},
{"test": "axi_lic", "seed": 5, "status": "ERROR", "reason": "no licence"},
None,
]
for (test, seed), record in zip(JOBS, RECORDS):
d = Path("runs") / f"{test}_{seed}"
d.mkdir(parents=True, exist_ok=True)
if record is not None:
(d / "record.json").write_text(json.dumps(record), encoding="utf-8")
print("job directories created:", len(JOBS))
print("records written :", sum(1 for r in RECORDS if r is not None))
print("axi_gone produced :", "nothing — the machine went away mid-run")job directories created: 6
records written : 5
axi_gone produced : nothing — the machine went away mid-runChapter 26.4 gave each job a directory and Chapter 25.6 gave each a record. Collecting them is now a matter of reading them back — and the directory is the only place the result exists, because Chapter 25.1 Section 8 prohibited a job from writing into a shared summary.
Two more things happened to this regression that are entirely ordinary and are not visible above.
2. Six Files, Six Jobs, Wrong Answer
import json
from pathlib import Path
JOBS = [("axi_smoke", 1), ("axi_reset", 2), ("axi_burst", 3),
("axi_hang", 4), ("axi_lic", 5), ("axi_gone", 6)]
RECORDS = [
{"test": "axi_smoke", "seed": 1, "status": "PASS"},
{"test": "axi_reset", "seed": 2, "status": "FAIL", "reason": "UVM_ERROR"},
{"test": "axi_burst", "seed": 3, "status": "FAIL", "reason": "UVM_ERROR"},
{"test": "axi_hang", "seed": 4, "status": "NO_FINISH", "reason": "TIMEOUT"},
{"test": "axi_lic", "seed": 5, "status": "ERROR", "reason": "no licence"},
None,
]
for (test, seed), record in zip(JOBS, RECORDS):
d = Path("runs") / f"{test}_{seed}"
d.mkdir(parents=True, exist_ok=True)
if record is not None:
(d / "record.json").write_text(json.dumps(record), encoding="utf-8")
# the job was killed while writing its record
(Path("runs/axi_burst_3") / "record.json").write_text(
'{"test":"axi_burst","seed":3,"stat', encoding="utf-8")
# a directory left behind by last night's regression
old = Path("runs/axi_old_99")
old.mkdir(parents=True, exist_ok=True)
(old / "record.json").write_text(
json.dumps({"test": "axi_old", "seed": 99, "status": "PASS"}), encoding="utf-8")
found = sorted(Path("runs").rglob("record.json"))
loaded, corrupt = [], []
for f in found:
try:
loaded.append(json.loads(f.read_text(encoding="utf-8")))
except json.JSONDecodeError:
corrupt.append(f.parent.name)
print("files found :", len(found))
print("jobs expected:", len(JOBS))
print("the counts match:", len(found) == len(JOBS))
print()
print("parsed :", len(loaded))
print("corrupt :", corrupt)
print("would report:", f"{sum(1 for r in loaded if r['status'] == 'PASS')} passed of {len(loaded)}")files found : 6
jobs expected: 6
the counts match: True
parsed : 5
corrupt : ['axi_burst_3']
would report: 2 passed of 5Six files, six jobs, and the counts match — which is exactly what makes this invisible.
Three separate things are wrong with "2 passed of 5".
axi_burst was killed while writing its record, leaving truncated JSON. The except json.JSONDecodeError handled it gracefully and the job vanished from the report, which is Chapter 26.1 Section 2's sixth way for a job to go missing.
axi_gone produced no record at all and is equally absent.
One of the two passes is axi_old, from last night. Its directory was still on disk, the glob found it, the JSON was valid, and it was counted. Chapter 26.1 Section 4 called this the too-optimistic direction, and here it arrives through the filesystem rather than through arithmetic.
The coincidence that one file was lost and one was gained is what made the totals agree. Count-matching proves nothing, because a glob can only ever produce the intersection of what you wanted and what is there, and the two differences are where the problems live.
3. Reconcile Against the Job List
summary = aggregate(results, expected=len(jobs))JOBS = [("axi_smoke", 1), ("axi_reset", 2), ("axi_burst", 3),
("axi_hang", 4), ("axi_lic", 5), ("axi_gone", 6)]
PARSED = [{"test": "axi_smoke", "seed": 1, "status": "PASS"},
{"test": "axi_reset", "seed": 2, "status": "FAIL"},
{"test": "axi_hang", "seed": 4, "status": "NO_FINISH"},
{"test": "axi_lic", "seed": 5, "status": "ERROR"},
{"test": "axi_old", "seed": 99, "status": "PASS"}]
by_job = {(r["test"], r["seed"]): r for r in PARSED}
wanted = set(JOBS)
matched = [j for j in JOBS if j in by_job]
missing = [j for j in JOBS if j not in by_job]
extra = sorted(k for k in by_job if k not in wanted)
print("expected:", len(JOBS))
print("matched :", len(matched))
print("missing :", missing)
print("extra :", extra)
print()
print("a glob gives you:", "matched, and nothing else")
print("the problems are in:", "missing and extra")expected: 6
matched : 4
missing : [('axi_burst', 3), ('axi_gone', 6)]
extra : [('axi_old', 99)]
a glob gives you: matched, and nothing else
the problems are in: missing and extraThree sets, not one count. expected= is what makes the other two computable, and it has one property that nothing on disk has: the job list cannot be influenced by the jobs. A record cannot add itself to it, a stale directory cannot appear in it, and a lost job cannot remove itself from it.
That is why the parameter is expected=len(jobs) and not something derived from the results. Chapter 26.1 Section 3 said total comes from the job list; this is the line where that matters.
4. The Summary Balances
from collections import Counter
JOBS = [("axi_smoke", 1), ("axi_reset", 2), ("axi_burst", 3),
("axi_hang", 4), ("axi_lic", 5), ("axi_gone", 6)]
MATCHED = {("axi_smoke", 1): "PASS", ("axi_reset", 2): "FAIL",
("axi_hang", 4): "NO_FINISH", ("axi_lic", 5): "ERROR"}
MISSING = [("axi_burst", 3), ("axi_gone", 6)]
counts = Counter(MATCHED.values())
not_run = len(MISSING)
print("status counts:", dict(sorted(counts.items())))
print("not_run :", not_run)
print()
accounted = sum(counts.values()) + not_run
print("accounted :", accounted)
print("total :", len(JOBS))
print("balanced :", accounted == len(JOBS))
print()
tested = len(JOBS) - counts["ERROR"] - not_run
print("tested :", tested)
print("pass rate :", f"{counts['PASS']}/{tested}")status counts: {'ERROR': 1, 'FAIL': 1, 'NO_FINISH': 1, 'PASS': 1}
not_run : 2
accounted : 6
total : 6
balanced : True
tested : 3
pass rate : 1/3Chapter 26.1's invariant, holding. Four results and two not_run account for all six jobs, and the honest pass rate is 1 of 3 rather than the naive glob's 2 of 5.
The two figures differ in both directions at once, which is worth noting: the glob was optimistic about the numerator (it counted a stale pass) and optimistic about the denominator (it excluded the lost jobs entirely by never knowing about them). Neither error is visible without expected=.
5. Two Kinds of Missing
from pathlib import Path
import json
for name, content in (("axi_burst_3", '{"test":"axi_burst","seed":3,"stat'),
("axi_gone_6", None)):
d = Path("runs2") / name
d.mkdir(parents=True, exist_ok=True)
if content is not None:
(d / "record.json").write_text(content, encoding="utf-8")
def why_missing(run_dir):
record = Path(run_dir) / "record.json"
if not record.exists():
log = Path(run_dir) / "sim.log"
return ("no_record", "the job never wrote one"
if not log.exists() else "a log exists but no record")
try:
json.loads(record.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
return ("corrupt_record", f"{exc.msg} char {exc.pos}")
return ("present", "")
for name in ("axi_burst_3", "axi_gone_6"):
kind, detail = why_missing(Path("runs2") / name)
print(f"{name:<14} {kind:<16} {detail}")
print()
print("both count as not_run :", True)
print("both leave evidence :", False)
print("which one has evidence:", "axi_burst_3 — a truncated record and possibly a log")axi_burst_3 corrupt_record Unterminated string starting at char 29
axi_gone_6 no_record the job never wrote one
both count as not_run : True
both leave evidence : False
which one has evidence: axi_burst_3 — a truncated record and possibly a logBoth are not_run for the invariant and they are not the same problem.
A corrupt record means the job ran, did work, and was interrupted while writing — so the directory contains a log, possibly a full one, and the failure is recoverable by hand. The exception even says where the JSON stopped.
No record at all means the job may never have started. Chapter 26.4's allocation created the directory before launch, so an empty directory and a directory with a log are themselves different stories, which is why why_missing looks for the log too.
Folding them into one count loses the only thing a debugger can act on. The invariant needs the number; the report needs the reason — the same split Chapter 25.6 Section 6 made between status and reason, one level up.
6. Extra Results Are Not Harmless
JOBS = [("axi_smoke", 1), ("axi_reset", 2)]
ON_DISK = [("axi_smoke", 1, "PASS"), ("axi_reset", 2, "FAIL"),
("axi_old", 99, "PASS"), ("axi_smoke", 7, "PASS")]
wanted = set(JOBS)
extra = [(t, s, st) for t, s, st in ON_DISK if (t, s) not in wanted]
print("jobs in this regression:", len(JOBS))
print("records on disk :", len(ON_DISK))
print("extra :", [(t, s) for t, s, _ in extra])
print()
naive_pass = sum(1 for *_, st in ON_DISK if st == "PASS")
real_pass = sum(1 for t, s, st in ON_DISK if (t, s) in wanted and st == "PASS")
print("passes counted naively :", naive_pass)
print("passes actually ours :", real_pass)
print("inflation :", naive_pass - real_pass)
print()
print("where the extras came from:")
print(" axi_old/99 a test removed from the suite last week")
print(" axi_smoke/7 a different seed from a previous sweep")jobs in this regression: 2
records on disk : 4
extra : [('axi_old', 99), ('axi_smoke', 7)]
passes counted naively : 3
passes actually ours : 1
inflation : 2
where the extras came from:
axi_old/99 a test removed from the suite last week
axi_smoke/7 a different seed from a previous sweepThree passes counted, one of them real. Both extras are mundane: a test deleted from the suite whose directory nobody cleaned, and a seed from a previous sweep in a shared root.
Chapter 26.4 Section 7 prevents this by refusing to run into a non-empty directory and by giving each regression its own root. This is the detection half, and both are worth having, because prevention covers the directories a regression creates and detection covers the ones it merely finds.
The right response to an extra is to report it, not delete it. An unexpected record is evidence that something is misconfigured — a shared root, an uncleaned previous run, a job list that changed mid-flight — and deleting it removes the only trace.
7. The Shape of Aggregation
8. Aggregating While It Runs
from collections import Counter
JOBS = [("t1", 1), ("t2", 2), ("t3", 3), ("t4", 4), ("t5", 5), ("t6", 6)]
ARRIVED = {("t1", 1): "PASS", ("t2", 2): "FAIL", ("t3", 3): "PASS"}
def partial(jobs, records):
counts = Counter(records.get(j) for j in jobs if j in records)
done = sum(counts.values())
return {"total": len(jobs), "done": done, "outstanding": len(jobs) - done,
"counts": dict(sorted(counts.items())),
"pass_rate_so_far": None if done == 0 else f"{counts['PASS']}/{done}"}
p = partial(JOBS, ARRIVED)
for k, v in p.items():
print(f"{k:<17} {v}")
print()
print("is 'outstanding' the same as 'not_run'?", False)
print("outstanding means :", "no result YET")
print("not_run means :", "no result, and we have stopped waiting")total 6
done 3
outstanding 3
counts {'FAIL': 1, 'PASS': 2}
pass_rate_so_far 2/3
is 'outstanding' the same as 'not_run'? False
outstanding means : no result YET
not_run means : no result, and we have stopped waitingA regression that takes eight hours should not keep its results to itself for eight hours. Reading the records as they appear costs nothing — they are already on disk, one per directory — and it turns a long silence into a progress figure.
The distinction in the last three lines is the one that must not be collapsed. outstanding and not_run are the same observation at different times: during the run a missing record means the job has not finished, and after the run it means the job produced nothing. Reporting a partial summary as though it were final would count every still-running job as a loss.
So a partial summary deliberately does not balance:
from collections import Counter
JOBS = [("t1", 1), ("t2", 2), ("t3", 3), ("t4", 4)]
ARRIVED = {("t1", 1): "PASS", ("t2", 2): "FAIL"}
counts = Counter(ARRIVED.values())
done = sum(counts.values())
print("during the run:")
print(" accounted for:", done, "of", len(JOBS))
print(" balances :", done == len(JOBS))
print(" is that a bug:", False, "- the rest have not finished")
print()
print("after the run, the same shortfall means something else:")
print(" accounted for:", done, "of", len(JOBS))
print(" not_run :", len(JOBS) - done)
print(" balances :", done + (len(JOBS) - done) == len(JOBS))during the run:
accounted for: 2 of 4
balances : False
is that a bug: False - the rest have not finished
after the run, the same shortfall means something else:
accounted for: 2 of 4
not_run : 2
balances : TrueThe same numbers, two readings, separated only by whether the regression has stopped.
That is why Chapter 26.1's Summary refuses to construct unbalanced and a partial report is a different object. Trying to reuse one type for both would force the constructor's guard to be optional, and a guard that can be switched off is the guard that is off on the night it mattered.
Two practical consequences.
A progress report needs a denominator too. "127 passed" says nothing; "127 of 400, 31 outstanding" says the regression is three-quarters through and roughly on track. The job list supplies it exactly as it does for the final summary.
Watch the rate, not just the count. A regression whose done figure has not moved in an hour has a problem that no individual job's status will reveal — the licence server, the farm, the filesystem — and it is visible from the aggregator long before any timeout fires.
9. The Aggregator
import json
from collections import Counter
from dataclasses import dataclass, field
from pathlib import Path
SUMMARY_FIELDS = ("passed", "failed", "timeout", "infra_error", "not_run")
@dataclass(frozen=True)
class Summary:
total: int
passed: int = 0
failed: int = 0
timeout: int = 0
infra_error: int = 0
not_run: int = 0
missing: tuple = ()
extra: tuple = ()
unreadable: tuple = ()
def __post_init__(self):
if self.accounted != self.total:
raise ValueError(f"does not balance: {self.accounted} != {self.total}")
@property
def accounted(self):
return sum(getattr(self, f) for f in SUMMARY_FIELDS)
@property
def tested(self):
return self.total - self.infra_error - self.not_run
def to_field(status, reason=None):
if status == "NO_FINISH":
return "timeout" if reason == "TIMEOUT" else "failed"
mapping = {"PASS": "passed", "FAIL": "failed",
"ERROR": "infra_error", "COMPILE_ERROR": "infra_error"}
if status not in mapping:
raise ValueError(f"no summary column for status {status!r}")
return mapping[status]
def read_records(root):
"""Return {(test, seed): record} and the directories whose record would not parse."""
found, unreadable = {}, []
for path in sorted(Path(root).rglob("record.json")):
try:
r = json.loads(path.read_text(encoding="utf-8"))
found[(r["test"], r["seed"])] = r
except (json.JSONDecodeError, KeyError):
unreadable.append(path.parent.name)
return found, unreadable
def aggregate(records, jobs, unreadable=()):
wanted = list(dict.fromkeys(jobs))
counts = Counter()
missing = []
for job in wanted:
r = records.get(job)
if r is None:
missing.append(job)
continue
counts[to_field(r["status"], r.get("reason"))] += 1
extra = tuple(sorted(k for k in records if k not in set(wanted)))
return Summary(total=len(wanted), not_run=len(missing),
missing=tuple(missing), extra=extra,
unreadable=tuple(unreadable),
**{f: counts[f] for f in SUMMARY_FIELDS if f != "not_run"})
JOBS = [("axi_smoke", 1), ("axi_reset", 2), ("axi_burst", 3),
("axi_hang", 4), ("axi_lic", 5), ("axi_gone", 6)]
RECORDS = {("axi_smoke", 1): {"test": "axi_smoke", "seed": 1, "status": "PASS"},
("axi_reset", 2): {"test": "axi_reset", "seed": 2, "status": "FAIL",
"reason": "UVM_ERROR"},
("axi_hang", 4): {"test": "axi_hang", "seed": 4, "status": "NO_FINISH",
"reason": "TIMEOUT"},
("axi_lic", 5): {"test": "axi_lic", "seed": 5, "status": "ERROR",
"reason": "no licence"},
("axi_old", 99): {"test": "axi_old", "seed": 99, "status": "PASS"}}
s = aggregate(RECORDS, JOBS, unreadable=("axi_burst_3",))
print("total :", s.total)
print("passed :", s.passed)
print("failed :", s.failed)
print("timeout :", s.timeout)
print("infra_error:", s.infra_error)
print("not_run :", s.not_run)
print("balanced :", s.accounted == s.total)
print("pass rate :", f"{s.passed}/{s.tested}")
print()
print("missing :", list(s.missing))
print("extra :", list(s.extra))
print("unreadable :", list(s.unreadable))total : 6
passed : 1
failed : 1
timeout : 1
infra_error: 1
not_run : 2
balanced : True
pass rate : 1/3
missing : [('axi_burst', 3), ('axi_gone', 6)]
extra : [('axi_old', 99)]
unreadable : ['axi_burst_3']Five decisions, and the first is what makes the whole thing safe.
aggregate takes the job list, not a count. The annotation's expected=len(jobs) catches a total mismatch; taking the list itself also produces missing and extra by name, which is the difference between "one job is missing" and "axi_burst seed 3 is missing".
missing, extra and unreadable are carried in the summary. Chapter 24.4 Section 6's rule: a step that made a judgement reports what it decided. A summary that knows two jobs are missing and does not say which is barely better than one that does not know.
unreadable is separate from missing even though the job appears in both. axi_burst_3 is not_run for the invariant and has a corrupt record somebody can look at — Section 5's distinction, preserved all the way into the summary.
dict.fromkeys(jobs) deduplicates the job list before counting, so Chapter 26.2 Section 6's duplicate cannot inflate total and break the balance.
Summary.__post_init__ still refuses to construct. Everything above is machinery for getting the counts right; the constructor is what guarantees nobody ships a summary that does not add up.
10. Predict, Run, Observe
import json
from collections import Counter
JOBS = [("a", 1), ("b", 2), ("c", 3)]
RECS = {("a", 1): "PASS", ("b", 2): "FAIL", ("z", 9): "PASS"}
# A
print("A:", len(JOBS), len(RECS))
# B
print("B:", [j for j in JOBS if j in RECS])
# C
print("C:", [j for j in JOBS if j not in RECS])
# D
print("D:", sorted(k for k in RECS if k not in set(JOBS)))
# E
print("E:", sum(1 for v in RECS.values() if v == "PASS"))
# F
print("F:", sum(1 for j in JOBS if RECS.get(j) == "PASS"))
# G
counts = Counter(RECS[j] for j in JOBS if j in RECS)
print("G:", dict(sorted(counts.items())), sum(counts.values()) + 1 == len(JOBS))
# H
try:
json.loads('{"a": 1')
except json.JSONDecodeError as exc:
print("H:", type(exc).__name__, exc.pos)The verified output:
A: 3 3
B: [('a', 1), ('b', 2)]
C: [('c', 3)]
D: [('z', 9)]
E: 2
F: 1
G: {'FAIL': 1, 'PASS': 1} True
H: JSONDecodeError 7A — three jobs, three records, and the counts match. Section 2's trap in miniature.
B — only two actually correspond.
C — one job has no record: not_run.
D — one record belongs to no job: extra.
E — counting every record gives two passes.
F — counting only ours gives one. The inflation, in two lines.
G — the matched counts plus one not_run account for all three jobs.
H — a truncated record raises with the character position, which is worth keeping in the report.
11. Exercises
Exercise 1 — where results live. Say why each job writes its own record rather than appending to a shared file, citing Chapter 25.1.
Exercise 2 — matching counts. Construct a case where the file count equals the job count and the summary is wrong.
Exercise 3 — three problems. For that case, name all three and say which the glob could detect.
Exercise 4 — the intersection. Explain in one sentence why a glob can only produce matched results.
Exercise 5 — three sets. Compute matched, missing and extra for a regression of your own.
Exercise 6 — the authority. Say what property the job list has that nothing on disk has.
Exercise 7 — balance it. Show the invariant holding with your not_run count included.
Exercise 8 — both directions. Say how the naive glob was optimistic about both the numerator and the denominator.
Exercise 9 — two kinds of missing. Produce a corrupt record and an absent one, and give each its own classification.
Exercise 10 — the evidence. Say what a corrupt record leaves that an absent one does not.
Exercise 11 — look for the log. Extend why_missing to distinguish "never started" from "ran and wrote no record".
Exercise 12 — the silent except. Find the except json.JSONDecodeError in your own flow and say what makes it survivable or not.
Exercise 13 — extras. Put a stale record in your results root and show it being counted.
Exercise 14 — provenance. Name two mundane ways an extra record appears.
Exercise 15 — report, do not delete. Argue for reporting an extra rather than removing it.
Exercise 16 — prevention and detection. Say which part Chapter 26.4 handles and which part this chapter handles.
Exercise 17 — names, not counts. Say what expected=len(jobs) cannot tell you that the job list can.
Exercise 18 — carry the sets. Add missing, extra and unreadable to your summary and print them.
Exercise 19 — deduplicate first. Show a duplicated job inflating total and breaking the balance, then fix it.
Exercise 20 — a partial summary. Produce one mid-run with done, outstanding and a rate so far.
Exercise 21 — the same shortfall. Explain how the identical numbers mean different things before and after the regression stops.
Exercise 22 — two types. Argue against reusing Summary for a progress report.
Exercise 23 — the rate. Say what a done figure that has not moved in an hour tells you that no job status can.
Exercise 24 — refuse to construct. Confirm your summary raises rather than reporting an unbalanced total.
12. Summary
- Each job writes its own record in its own directory, because Chapter 25.1 Section 8 forbade writing into a shared summary.
- A matching file count proves nothing. Six files for six jobs hid one corrupt record, one absent job, and one stale record from a previous run.
- A glob can only ever produce the intersection of what you wanted and what is there.
- Reconcile against the job list — it is the only authority the results cannot influence.
expected=is what makesmissingandextracomputable, and passing the list rather than the count makes them nameable.- The invariant then balances: matched results plus
not_runequal the total. - The naive answer was optimistic twice over — a stale pass in the numerator and lost jobs excluded from the denominator.
- A corrupt record and an absent one are both
not_runand are not the same problem; one leaves evidence and the exception even says where the JSON stopped. except json.JSONDecodeError: continueis Chapter 14.9's silent-except, survivable only becauseexpected=reinstates the job.- Extra records are counted as passes unless something excludes them, which is Chapter 26.1's too-optimistic direction arriving through the filesystem.
- Report an extra rather than deleting it — an unexpected record is evidence that something is misconfigured.
- Chapter 26.4 prevents collisions and this chapter detects them; prevention covers directories you create, detection covers ones you find.
- Carry
missing,extraandunreadablein the summary, by name. - Deduplicate the job list before counting, or Chapter 26.2's duplicate inflates the total.
- Aggregate while it runs — the records are already on disk, and a long regression should report progress rather than silence.
outstandingandnot_runare the same observation at different times, and collapsing them counts running jobs as losses.- A partial summary is a different type, because Chapter 26.1's balance guard must never become optional.
- A progress report needs the denominator too, and the job list supplies it.
- Watch the rate: a
donecount that stops moving reveals a systemic problem long before any timeout fires. - Let the summary's constructor refuse to balance, so no unbalanced number can be shipped.
The next chapter takes the failures this summary counted and groups them by cause, so a morning review reads a handful of buckets instead of a list.
Where this fits
Part of the Python curriculum.