Python Part II · Module 30 · Engineering Report Generation
Turning Thousands of Results Into a Few Facts
Module 26 ran the regression and Module 29 measured what it covered. What arrives at a review is four hundred lines, three hundred and eighty of which say PASS, and the job of a report is to delete those three hundred and eighty without deleting anything that matters. That reduction is arithmetic, which makes it easy to get wrong quietly: every mistake in this chapter produces a number that looks plausible, sits in a dashboard for months, and is wrong in the direction that makes a regression look smaller than it is.
Advanced21 min readPythonPart IIAutomationReportingRegression
Part II · Module 30 · Chapter 30.1 · Turning Thousands of Results Into a Few Facts
1. What a Review Reads
A regression review does not read results. It reads four things and then opens one log:
WANTED = [
("counts", "how many ran, how many passed, how many did not"),
("buckets", "the failures grouped by cause, not listed"),
("a representative", "one log to open per bucket"),
("what changed", "which failures are new since last night"),
]
print(f"{'fact':<18} what it answers")
for name, what in WANTED:
print(f"{name:<18} {what}")
print()
print("facts a review needs :", len(WANTED))
print("result lines produced:", 400)
print("ratio :", f"{400 // len(WANTED)}:1 before any reduction")fact what it answers
counts how many ran, how many passed, how many did not
buckets the failures grouped by cause, not listed
a representative one log to open per bucket
what changed which failures are new since last night
facts a review needs : 4
result lines produced: 400
ratio : 100:1 before any reductionThose four are exactly what the canonical annotation names, which is why it is a good starting point rather than a strawman. The chapter is about the fact that each of its four values is computed in a way that is wrong, and that none of the four wrongness modes raises anything.
2. The Annotation
facts = {"total": n, "failed": len(fails), "buckets": counts, "new": new_fails}RESULTS = (
[{"test": f"t{i}", "seed": i, "status": "PASS"} for i in range(380)]
+ [{"test": f"f{i}", "seed": i, "status": "FAIL"} for i in range(9)]
+ [{"test": f"to{i}", "seed": i, "status": "TIMEOUT"} for i in range(4)]
+ [{"test": f"nr{i}", "seed": i, "status": "NO_RUN"} for i in range(6)]
+ [{"test": "infra0", "seed": 0, "status": "INFRA"}]
)
n = len(RESULTS)
fails = [r for r in RESULTS if r["status"] == "FAIL"]
counts = {"handshake": 5, "timeout": 4}
new_fails = set()
facts = {"total": n, "failed": len(fails), "buckets": counts, "new": new_fails}
for k, v in facts.items():
print(f"{k:<9} {v}")
print()
print("implied passes:", facts["total"] - facts["failed"])total 400
failed 9
buckets {'handshake': 5, 'timeout': 4}
new set()
implied passes: 391Four keys, four values, and a dashboard could be built on this tomorrow. Sections 3 to 6 take the four in order.
3. total Counts What Parsed, Not What Ran
The first value is the number of results the parser produced, and a job that never ran produces nothing to parse:
SUBMITTED = [f"job{i:03d}" for i in range(400)]
# twenty jobs produced no result file at all: the machine died, the
# scheduler never started them, the log was never written
PARSED = {j: "PASS" for j in SUBMITTED[:360]}
PARSED.update({j: "FAIL" for j in SUBMITTED[360:380]})
n = len(PARSED)
passed = sum(1 for s in PARSED.values() if s == "PASS")
print(f"jobs submitted : {len(SUBMITTED)}")
print(f"results parsed : {n}")
print(f"missing : {len(SUBMITTED) - n}")
print()
print(f"pass rate over results parsed : {100.0 * passed / n:.2f}%")
print(f"pass rate over jobs submitted : {100.0 * passed / len(SUBMITTED):.2f}%")
print(f"difference : {100.0*passed/n - 100.0*passed/len(SUBMITTED):+.2f} points")jobs submitted : 400
results parsed : 380
missing : 20
pass rate over results parsed : 94.74%
pass rate over jobs submitted : 90.00%
difference : +4.74 points94.74% against an honest 90.00%. The twenty jobs that produced nothing are invisible to a denominator built from what arrived, and the error always points the same way: a missing result can only inflate the rate, never deflate it, because the jobs that vanish are removed from the denominator while the passes stay in the numerator.
This is Chapter 29.3 Section 9's missing block in a new setting, and the defence is the same one Chapter 25.5 argued for: iterate the submitted set.
SUBMITTED = [f"job{i:03d}" for i in range(400)]
PARSED = {j: "PASS" for j in SUBMITTED[:360]}
PARSED.update({j: "FAIL" for j in SUBMITTED[360:380]})
def account(submitted, parsed):
missing = [j for j in submitted if j not in parsed]
extra = [j for j in parsed if j not in submitted]
return {
"submitted": len(submitted),
"accounted": len(parsed),
"missing": len(missing),
"unexpected": len(extra),
"complete": not missing and not extra,
"first_missing": missing[0] if missing else None,
}
a = account(SUBMITTED, PARSED)
for k, v in a.items():
print(f" {k:<14} {v}")
print()
print("a report over an incomplete set must say so:", not a["complete"]) submitted 400
accounted 380
missing 20
unexpected 0
complete False
first_missing job380
a report over an incomplete set must say so: Truefirst_missing is there for the same reason a bucket needs a representative: a count of twenty tells you to look, and a job name tells you where. The unexpected count catches the opposite error — a result for a job nobody submitted, which happens when a stale run directory is swept into the parse.
4. With Five Statuses, total - failed Is Not passed
The second value counts one status and implies the rest:
from collections import Counter
STATUSES = Counter({"PASS": 380, "FAIL": 9, "TIMEOUT": 4, "NO_RUN": 6, "INFRA": 1})
n = sum(STATUSES.values())
failed = STATUSES["FAIL"]
print(f"{'status':<9} {'count':>6}")
for s, c in sorted(STATUSES.items()):
print(f"{s:<9} {c:>6}")
print()
print(f"total : {n}")
print(f"len(fails) : {failed}")
print(f"total - failed : {n - failed} <- rendered as 'passed'")
print(f"actual PASS : {STATUSES['PASS']}")
print(f"overstated by : {(n - failed) - STATUSES['PASS']}")
print(f" which is : {STATUSES['TIMEOUT']} TIMEOUT + {STATUSES['NO_RUN']} NO_RUN + {STATUSES['INFRA']} INFRA")status count
FAIL 9
INFRA 1
NO_RUN 6
PASS 380
TIMEOUT 4
total : 400
len(fails) : 9
total - failed : 391 <- rendered as 'passed'
actual PASS : 380
overstated by : 11
which is : 4 TIMEOUT + 6 NO_RUN + 1 INFRAEleven jobs reported as passing that did not pass. A timeout is not a pass, a job that never started is not a pass, and an infrastructure failure is certainly not a pass — but subtracting one status from the total silently assigns all of them to the other side.
Chapter 25.6 built a classifier with exactly these five outcomes for exactly this reason. The report must carry the whole distribution, and the check is that it sums:
from collections import Counter
STATUSES = Counter({"PASS": 380, "FAIL": 9, "TIMEOUT": 4, "NO_RUN": 6, "INFRA": 1})
KNOWN = ("PASS", "FAIL", "TIMEOUT", "NO_RUN", "INFRA")
counts = {s: STATUSES.get(s, 0) for s in KNOWN}
unknown = {s: c for s, c in STATUSES.items() if s not in KNOWN}
total = sum(STATUSES.values())
print("counts:", counts)
print("unknown statuses:", unknown or "none")
print()
print(f"sum of known statuses : {sum(counts.values())}")
print(f"total : {total}")
print(f"accounts for everything: {sum(counts.values()) + sum(unknown.values()) == total}")
print()
not_passing = total - counts["PASS"]
print(f"jobs that did not pass: {not_passing}")
print(f"of which actual FAIL : {counts['FAIL']} ({100.0*counts['FAIL']/not_passing:.1f}%)")
print(f"of which not run/infra: {counts['NO_RUN'] + counts['INFRA']}")counts: {'PASS': 380, 'FAIL': 9, 'TIMEOUT': 4, 'NO_RUN': 6, 'INFRA': 1}
unknown statuses: none
sum of known statuses : 400
total : 400
accounts for everything: True
jobs that did not pass: 20
of which actual FAIL : 9 (45.0%)
of which not run/infra: 7Twenty jobs did not pass and fewer than half of them are design failures. That split is the most useful line in the whole report, because it tells a reviewer whether tonight is a design problem or a machine problem — and total - failed erases it completely.
5. A Bucket Count Is Not a Bucket
The third value groups the failures and throws away the ability to act on them:
FAILURES = [
{"test": "axi_burst", "seed": 7731, "bucket": "handshake",
"log": "runs/axi_burst_7731/sim.log"},
{"test": "axi_wrap", "seed": 1190, "bucket": "handshake",
"log": "runs/axi_wrap_1190/sim.log"},
{"test": "axi_narrow", "seed": 5512, "bucket": "handshake",
"log": "runs/axi_narrow_5512/sim.log"},
{"test": "dma_long", "seed": 2048, "bucket": "timeout",
"log": "runs/dma_long_2048/sim.log"},
{"test": "dma_chain", "seed": 9004, "bucket": "timeout",
"log": "runs/dma_chain_9004/sim.log"},
]
counts = {}
for f in FAILURES:
counts[f["bucket"]] = counts.get(f["bucket"], 0) + 1
print("what the annotation keeps:", counts)
print()
print("what a reviewer does with it:")
print(" reads 'handshake: 3'")
print(" asks 'which log do I open?'")
print(" answer: not in the report")what the annotation keeps: {'handshake': 3, 'timeout': 2}
what a reviewer does with it:
reads 'handshake: 3'
asks 'which log do I open?'
answer: not in the reportChapter 26.7 Section 8 made this argument when it built the buckets: a bucket exists so that one log can stand for many, and a count is the bucket with its one useful property removed. Keep a representative:
FAILURES = [
{"test": "axi_burst", "seed": 7731, "bucket": "handshake", "log": "runs/axi_burst_7731/sim.log"},
{"test": "axi_wrap", "seed": 1190, "bucket": "handshake", "log": "runs/axi_wrap_1190/sim.log"},
{"test": "axi_narrow", "seed": 5512, "bucket": "handshake", "log": "runs/axi_narrow_5512/sim.log"},
{"test": "dma_long", "seed": 2048, "bucket": "timeout", "log": "runs/dma_long_2048/sim.log"},
{"test": "dma_chain", "seed": 9004, "bucket": "timeout", "log": "runs/dma_chain_9004/sim.log"},
]
def bucket_facts(failures):
out = {}
for f in sorted(failures, key=lambda f: (f["bucket"], f["test"])):
b = out.setdefault(f["bucket"], {"count": 0, "tests": [], "representative": None})
b["count"] += 1
b["tests"].append(f["test"])
if b["representative"] is None:
b["representative"] = {"test": f["test"], "seed": f["seed"], "log": f["log"]}
return out
for name, b in bucket_facts(FAILURES).items():
rep = b["representative"]
print(f"{name:<11} {b['count']:>2} failures open: {rep['log']}")
print(f"{'':<11} tests: {', '.join(b['tests'])}")
print()
print("logs a reviewer must open to see every distinct cause:", 2)
print("failures those two logs stand for :", 5)handshake 3 failures open: runs/axi_burst_7731/sim.log
tests: axi_burst, axi_narrow, axi_wrap
timeout 2 failures open: runs/dma_chain_9004/sim.log
tests: dma_chain, dma_long
logs a reviewer must open to see every distinct cause: 2
failures those two logs stand for : 5Note the representative is chosen by a stated rule — first in sorted order — and not by whichever record happened to arrive first. A representative that depends on scheduler order changes between runs of an identical regression, and a reviewer comparing two mornings will see the example move and conclude something changed. Module 27's determinism discipline — never publish a value that depends on completion order — applies to reports as much as to runs.
6. On the First Run, new Is Not Empty — It Is Unknown
The fourth value compares against the previous run. Sometimes there is no previous run:
def new_failures(today, yesterday):
"""The annotation's shape: a set difference."""
return today - yesterday
TODAY = {"axi_burst", "dma_long"}
print("with a previous run:")
print(" ", sorted(new_failures(TODAY, {"dma_long"})))
print()
print("first ever run, or the previous results were pruned:")
print(" ", sorted(new_failures(TODAY, set())))
print()
print("rendered in a report, an empty set reads as:", "no new failures")
print("what it actually means here :", "there was nothing to compare against")with a previous run:
['axi_burst']
first ever run, or the previous results were pruned:
['axi_burst', 'dma_long']
rendered in a report, an empty set reads as: no new failures
what it actually means here : there was nothing to compare againstWith an empty baseline the set difference returns every failure as new, which overstates rather than understates — the loud direction, and therefore the survivable one. The dangerous case is the opposite: a baseline that loaded but is empty, or a comparison that silently skipped.
def changed(today, yesterday):
"""None for yesterday means: no baseline. Not the same as an empty baseline."""
if yesterday is None:
return {"comparable": False, "new": None, "fixed": None,
"reason": "no previous run to compare against"}
return {"comparable": True,
"new": sorted(today - yesterday),
"fixed": sorted(yesterday - today),
"reason": ""}
TODAY = {"axi_burst", "dma_long"}
for label, baseline in (("no baseline", None),
("empty baseline", set()),
("real baseline", {"dma_long", "old_test"})):
c = changed(TODAY, baseline)
print(f"{label:<16} comparable={str(c['comparable']):<6} "
f"new={c['new']} fixed={c['fixed']}")
print()
print("the three cases render differently, which is the point.")no baseline comparable=False new=None fixed=None
empty baseline comparable=True new=['axi_burst', 'dma_long'] fixed=[]
real baseline comparable=True new=['axi_burst'] fixed=['old_test']
the three cases render differently, which is the point.Three baselines, three different renderings, where a plain set difference gives two of them the same shape. None is Chapter 28.5 Section 4's unavailable-metric pattern and Chapter 28.6's UNKNOWN verdict: a report that cannot answer a question must say so, because the one thing it must never do is answer it reassuringly.
Note also that the annotation computes new and not fixed. Chapter 30.3 is about both, and about the third category neither of them covers.
7. The Reduction, as a Flow
The order of these stages is not arbitrary: accounting happens before counting, because a count over an incomplete set is wrong rather than approximate, and the comparison happens last because it needs the buckets to be stable.
From a regression's results to a review's facts
Stage 4's "and the unknown ones" is the guard from Section 4's callout. Stage 6's "or report that you cannot" is Section 6. Chapter 30.2 decides what of this reaches the page.
8. The Facts, Assembled
from collections import Counter
from dataclasses import dataclass, field
from typing import Dict, List, Optional
@dataclass
class Facts:
submitted: int
accounted: int
missing: List[str]
unexpected: List[str]
counts: Dict[str, int]
unknown_statuses: Dict[str, int]
buckets: Dict[str, dict]
comparable: bool
new: Optional[List[str]]
fixed: Optional[List[str]]
@property
def complete(self):
return not self.missing and not self.unexpected
@property
def passed(self):
return self.counts.get("PASS", 0)
@property
def not_passing(self):
return self.submitted - self.passed
KNOWN = ("PASS", "FAIL", "TIMEOUT", "NO_RUN", "INFRA")
def build_facts(submitted, results, failures, baseline):
parsed = {r["job"]: r for r in results}
missing = [j for j in submitted if j not in parsed]
unexpected = [j for j in parsed if j not in submitted]
seen = Counter(r["status"] for r in results)
counts = {s: seen.get(s, 0) for s in KNOWN}
unknown = {s: c for s, c in seen.items() if s not in KNOWN}
buckets = {}
for f in sorted(failures, key=lambda f: (f["bucket"], f["test"])):
b = buckets.setdefault(f["bucket"], {"count": 0, "representative": None})
b["count"] += 1
if b["representative"] is None:
b["representative"] = f["log"]
today = {f["test"] for f in failures}
if baseline is None:
comparable, new, fixed = False, None, None
else:
comparable = True
new = sorted(today - baseline)
fixed = sorted(baseline - today)
return Facts(len(submitted), len(parsed), missing, unexpected, counts,
unknown, buckets, comparable, new, fixed)
SUBMITTED = [f"job{i:03d}" for i in range(400)]
RESULTS = ([{"job": SUBMITTED[i], "status": "PASS"} for i in range(360)]
+ [{"job": SUBMITTED[360 + i], "status": "FAIL"} for i in range(9)]
+ [{"job": SUBMITTED[369 + i], "status": "TIMEOUT"} for i in range(4)]
+ [{"job": SUBMITTED[373 + i], "status": "NO_RUN"} for i in range(6)]
+ [{"job": SUBMITTED[379], "status": "INFRA"}])
FAILURES = [
{"test": "axi_burst", "bucket": "handshake", "log": "runs/axi_burst_7731/sim.log"},
{"test": "axi_wrap", "bucket": "handshake", "log": "runs/axi_wrap_1190/sim.log"},
{"test": "axi_narrow", "bucket": "handshake", "log": "runs/axi_narrow_5512/sim.log"},
{"test": "dma_long", "bucket": "timeout", "log": "runs/dma_long_2048/sim.log"},
{"test": "dma_chain", "bucket": "timeout", "log": "runs/dma_chain_9004/sim.log"},
]
f = build_facts(SUBMITTED, RESULTS, FAILURES, {"dma_long", "old_test"})
print(f"submitted {f.submitted} accounted {f.accounted} complete {f.complete}")
print(f"missing {len(f.missing)} (first: {f.missing[0] if f.missing else None})")
print(f"unexpected {len(f.unexpected)} unknown statuses {f.unknown_statuses or 'none'}")
print()
print("counts:", f.counts)
print(f"did not pass: {f.not_passing} of which design failures: {f.counts['FAIL']}")
print()
for name, b in f.buckets.items():
print(f" {name:<11} {b['count']:>2} open: {b['representative']}")
print()
print(f"comparable {f.comparable} new {f.new} fixed {f.fixed}")submitted 400 accounted 380 complete False
missing 20 (first: job380)
unexpected 0 unknown statuses none
counts: {'PASS': 360, 'FAIL': 9, 'TIMEOUT': 4, 'NO_RUN': 6, 'INFRA': 1}
did not pass: 40 of which design failures: 9
handshake 3 open: runs/axi_burst_7731/sim.log
timeout 2 open: runs/dma_chain_9004/sim.log
comparable True new ['axi_burst', 'axi_narrow', 'axi_wrap', 'dma_chain'] fixed ['old_test']Eleven lines. Now the ratio that is this chapter's thesis:
RESULT_LINES = 400
FACT_LINES = 11
print(f"result lines in : {RESULT_LINES}")
print(f"fact lines out : {FACT_LINES}")
print(f"reduction : {RESULT_LINES // FACT_LINES}:1")
print(f"lines dropped : {RESULT_LINES - FACT_LINES}")
print()
print("of the dropped lines, how many would a reviewer have read?")
print(" PASS lines that needed reading:", 0)
print(" but only because stage 3 proved every job was accounted for.")result lines in : 400
fact lines out : 11
reduction : 36:1
lines dropped : 389
of the dropped lines, how many would a reviewer have read?
PASS lines that needed reading: 0
but only because stage 3 proved every job was accounted for.Thirty-six to one, and the last line is the licence to do it. Deleting three hundred and eighty-nine lines of PASS is only safe because the accounting step proved they were all there; without it the reduction is not a summary but a sample, and a sample of a regression is not evidence of anything.
9. The First Failure Is a Fact, Not a Log
Section 5 gave each bucket a log to open. Chapter 30.2's purpose asks for something cheaper: the first failure line, in the report, so a reviewer knows what a bucket is before deciding whether to open anything.
LOG = """UVM_INFO tb.sv(42) @ 0: reset released
UVM_INFO axi_seq.sv(88) @ 1200: starting burst 0
UVM_ERROR axi_drv.sv(188) @ 141900: WLAST on beat 3 of 4
UVM_INFO axi_seq.sv(88) @ 142000: starting burst 1
UVM_ERROR axi_drv.sv(188) @ 158200: WLAST on beat 2 of 4
UVM_FATAL tb.sv(90) @ 158300: too many errors
"""
SEVERITIES = ("UVM_FATAL", "UVM_ERROR")
lines = LOG.splitlines()
hits = [(i, l) for i, l in enumerate(lines)
if any(l.startswith(s) for s in SEVERITIES)]
print("log lines :", len(lines))
print("severity lines :", len(hits))
print()
print("first :", hits[0][1])
print("last :", hits[-1][1])
print("worst :", [l for _, l in hits if l.startswith("UVM_FATAL")][0])log lines : 6
severity lines : 3
first : UVM_ERROR axi_drv.sv(188) @ 141900: WLAST on beat 3 of 4
last : UVM_FATAL tb.sv(90) @ 158300: too many errors
worst : UVM_FATAL tb.sv(90) @ 158300: too many errorsThree defensible choices and they are different lines. First is the one to publish, and the reason is causal rather than aesthetic: a UVM_FATAL at the end is usually a consequence — the testbench giving up — and the last error is whatever happened as the run unwound. The first severity line is the closest thing a log has to a cause, which is Chapter 20.7's argument.
Publishing it changes what the bucket row is worth:
BUCKETS = [
("handshake", 3, "UVM_ERROR axi_drv.sv(188) @ 141900: WLAST on beat 3 of 4"),
("timeout", 2, "no TEST DONE banner after 1800 s"),
("null_deref", 4, "UVM_FATAL reg_blk.sv(55) @ 9100: null handle in write()"),
]
print("without the first-failure line:")
for name, n, _ in BUCKETS:
print(f" {name:<11} {n}")
print()
print("with it:")
for name, n, first in BUCKETS:
print(f" {name:<11} {n} {first}")
print()
print("buckets a reviewer can triage without opening a log:",
sum(1 for _, _, f in BUCKETS if f))without the first-failure line:
handshake 3
timeout 2
null_deref 4
with it:
handshake 3 UVM_ERROR axi_drv.sv(188) @ 141900: WLAST on beat 3 of 4
timeout 2 no TEST DONE banner after 1800 s
null_deref 4 UVM_FATAL reg_blk.sv(55) @ 9100: null handle in write()
buckets a reviewer can triage without opening a log: 3The second table lets a reviewer assign all three buckets to owners without opening anything, which is the difference between a five-minute standup and three log sessions. The bucket name says which group a failure belongs to; the first line says what it is.
One constraint comes with it. The line is free-form log text, so it carries every hazard Chapters 30.4 to 30.6 are about — a comma, a pipe, an angle bracket — and it must be stored raw in the facts and escaped per format at render time, never escaped once at extraction. Escaping at extraction means the CSV contains Markdown escapes.
10. Predict, Run, Observe
from collections import Counter
S = Counter({"PASS": 380, "FAIL": 9, "TIMEOUT": 4, "NO_RUN": 6, "INFRA": 1})
SUBMITTED, PARSED = 400, 380
# A
print("A:", sum(S.values()) - S["FAIL"], S["PASS"])
# B
print("B:", f"{100.0*360/PARSED:.2f}", f"{100.0*360/SUBMITTED:.2f}")
# C
print("C:", sorted({"a", "b"} - set()), sorted({"a", "b"} - {"b"}))
# D
print("D:", {"handshake": 3}.get("handshake"), {"handshake": 3}.get("log", "absent"))The verified output:
A: 391 380
B: 94.74 90.00
C: ['a', 'b'] ['a']
D: 3 absentA — 391 implied passes against 380 real ones. Eleven jobs promoted by subtraction.
B — the same numerator over two denominators. The missing results inflate the rate by 4.74 points, and always upward.
C — an empty baseline makes every failure new; a real baseline makes one of them new. Both are set differences and only one of them is a comparison.
D — a bucket count answers 3 and has nothing to say about where to look. Section 5.
11. Exercises
Exercise 1 — four facts. For your last regression review, write down which of the four facts your report actually carried.
Exercise 2 — submitted versus parsed. Count the jobs your flow submits and the results it parses for one night, and report the difference.
Exercise 3 — the inflation is one-directional. Explain in two sentences why a missing result can only raise the pass rate.
Exercise 4 — name the missing. Add first_missing to your accounting and run it on a night when the machine died.
Exercise 5 — unexpected results. Construct a case where a result arrives for a job nobody submitted, and say how it happens.
Exercise 6 — your status set. List every status your runner can produce and check whether your report sums them.
Exercise 7 — the promotion. Compute, for one night, how many jobs total - failed would report as passing that did not pass.
Exercise 8 — design or machine. Split your non-passing jobs into design failures and infrastructure, and say which dominated last week.
Exercise 9 — the sixth status. Add a new status to your runner and say what your current report does with it.
Exercise 10 — count the unknown. Add an unknown-status counter and put its value in the report output.
Exercise 11 — a bucket with no log. Take a bucket count from your dashboard and time how long it takes to find a log to open.
Exercise 12 — a stated rule. Write the rule your representative is chosen by, and check it is stable across two runs of the same regression.
Exercise 13 — a moving example. Explain what a reviewer concludes when the representative changes between two identical runs.
Exercise 14 — no baseline. Say what your flow reports on the first run of a new branch.
Exercise 15 — three baselines. Implement the None / empty / real distinction and show all three renderings.
Exercise 16 — reassuring by accident. Give another example in your flow of a missing input rendering as good news.
Exercise 17 — your reduction ratio. Compute result lines in and fact lines out for your report.
Exercise 18 — sample or summary. Explain why the accounting step is what makes the reduction a summary rather than a sample.
Exercise 19 — the order of stages. Say what breaks if counting happens before accounting.
Exercise 20 — fixed as well as new. Add fixed to your facts and say what a reviewer does with it.
Exercise 21 — eleven lines. Write your own eleven-line fact block for one real night.
Exercise 22 — one log. For last night's failures, compute how many logs a reviewer must open to see every distinct cause.
Exercise 23 — the four wrongness modes. For each of the annotation's four values, state its defect in one sentence.
Exercise 24 — a plausible wrong number. Find a number in your current dashboard that you cannot derive from first principles, and derive it.
12. Summary
- A review reads four facts and then opens one log — counts, buckets, a representative, and what changed.
- The annotation names the right four and computes all four wrongly, and not one of its four defects raises anything.
totalcounts what parsed, not what ran. 400 submitted and 380 parsed reported a 94.74% pass rate where the honest figure was 90.00%.- A missing result can only inflate the rate, because the vanished jobs leave the denominator while the passes stay in the numerator.
- Iterate the submitted set and report
missing,unexpectedand a name to look at — a count says to look, a name says where. - With five statuses,
total - failedis notpassed— it promoted eleven jobs that timed out, never started or hit infrastructure. - Carry the whole distribution and check that it sums, including a counter for statuses you do not recognise, because the sixth status will arrive.
- Fewer than half of the non-passing jobs were design failures, which is the line that tells a reviewer whether tonight is a design or a machine problem.
- A bucket count is a bucket with its one useful property removed. Keep a representative log per bucket.
- Choose the representative by a stated rule, not by arrival order, or the example moves between identical runs.
- On the first run,
newis UNKNOWN, not empty. An empty set is the most reassuring possible rendering of having nothing to compare against. - Three baselines — absent, empty and real — must render as three different things, and a plain set difference gives two of them the same shape.
- The reduction was 36:1, and the accounting step is the licence for it: without it a summary is only a sample.
- Publish the first severity line per bucket, not just a log path. The bucket name says which group a failure is in; the first line says what it is.
- The first severity line is the closest a log has to a cause — a trailing
UVM_FATALis usually the testbench giving up. - Store the first-failure line raw and escape it per format at render time. Escaping at extraction puts Markdown escapes in the CSV.
The next chapter decides which of these facts reaches the page, and what a report has no business containing.
Where this fits
Part of the Python curriculum.