Python Part II · Module 30 · Engineering Report Generation
What Belongs in a Regression Report
Chapter 30.1 produced more facts than a report should show, which is the normal situation: the hard part of a summary is not computing it but deciding what to leave out. Most teams settle that by argument, and the argument is unwinnable because “useful” and “clutter” are opinions. This chapter replaces the argument with a test — three questions, taken from the canonical annotation, that every candidate item either answers or does not — and then measures what each item costs in the only currency a report has, which is lines on one screen.
Intermediate19 min readPythonPart IIAutomationReportingReview
Part II · Module 30 · Chapter 30.2 · What Belongs in a Regression Report
1. The Annotation Is a Specification
Most annotations in Part II are a line of code with a defect in it. This one is a comment, and it is right:
# one screen: what broke, what changed, and where to look nextSPEC = [
("one screen", "a size budget — §4 turns it into a line count"),
("what broke", "the failures, grouped"),
("what changed", "the difference from last night"),
("where to look next", "a path a reviewer can open"),
]
print(f"{'clause':<20} what it constrains")
for clause, what in SPEC:
print(f"{clause:<20} {what}")
print()
print("clauses that are questions a reader asks:", 3)
print("clauses that are a budget :", 1)
print("so the annotation is :", "a test, once it is written as one")clause what it constrains
one screen a size budget — §4 turns it into a line count
what broke the failures, grouped
what changed the difference from last night
where to look next a path a reviewer can open
clauses that are questions a reader asks: 3
clauses that are a budget : 1
so the annotation is : a test, once it is written as oneThree questions and a budget. Everything in this chapter follows from applying them literally.
2. The Three Questions as a Test
def answers(item_questions):
"""An item belongs if it answers at least one of the three questions."""
return len(item_questions) > 0
CASES = [
("new failures since last night", {"changed"}),
("a link to each failing log", {"where"}),
("the failure buckets", {"broke"}),
("the project logo", set()),
]
print(f"{'item':<32} {'answers':<22} belongs")
for name, qs in CASES:
print(f"{name:<32} {str(sorted(qs)) if qs else '[]':<22} {answers(qs)}")
print()
print("the test is not an opinion:", "an item either answers a question or it does not")item answers belongs
new failures since last night ['changed'] True
a link to each failing log ['where'] True
the failure buckets ['broke'] True
the project logo [] False
the test is not an opinion: an item either answers a question or it does notThe logo case is deliberately easy. The next section runs the test on the items people actually argue about.
3. Scoring Thirteen Candidates
CANDIDATES = [
("job counts by status", {"broke"}, 2),
("failure buckets with counts", {"broke"}, 5),
("one representative log per bucket", {"broke", "where"}, 5),
("new failures", {"changed"}, 3),
("fixed failures", {"changed"}, 2),
("first failure line per bucket", {"broke", "where"}, 5),
("runtime outliers", {"changed", "broke"}, 3),
("coverage delta", {"changed"}, 2),
("link to run directory and wave", {"where"}, 0),
("pass-rate pie chart", set(), 8),
("a sparkline of pass rate over 30 days", set(), 4),
("the project logo", set(), 3),
("every job's runtime", set(), 400),
]
print(f"{'candidate':<38} {'answers':<24} {'lines':>6}")
for name, qs, lines in CANDIDATES:
print(f"{name:<38} {','.join(sorted(qs)) or '—':<24} {lines:>6}")
belongs = [c for c in CANDIDATES if c[1]]
decorative = [c for c in CANDIDATES if not c[1]]
print()
print(f"candidates : {len(CANDIDATES)}")
print(f"answer >= 1 question : {len(belongs)}")
print(f"answer none : {len(decorative)}")
print(f"lines spent on items that answer nothing: {sum(c[2] for c in decorative)}")
print(f"lines spent on items that answer something: {sum(c[2] for c in belongs)}")candidate answers lines
job counts by status broke 2
failure buckets with counts broke 5
one representative log per bucket broke,where 5
new failures changed 3
fixed failures changed 2
first failure line per bucket broke,where 5
runtime outliers broke,changed 3
coverage delta changed 2
link to run directory and wave where 0
pass-rate pie chart — 8
a sparkline of pass rate over 30 days — 4
the project logo — 3
every job's runtime — 400
candidates : 13
answer >= 1 question : 9
answer none : 4
lines spent on items that answer nothing: 415
lines spent on items that answer something: 27The nine items that answer something are exactly the eight the canonical purpose names, plus the status counts. Four items answer nothing and cost 415 lines between them — more than fourteen times what the entire useful content costs.
Three of those four are worth defending, because each is defended regularly:
A pass-rate pie chart shows that 95% of jobs passed, which is the one fact nobody needs: a reviewer is there because of the 5%. It answers "what broke" only in the sense that it declines to say.
A thirty-day sparkline is genuinely informative and belongs in a trend view, not tonight's report. Chapter 29.4 Section 7 argued that a trend needs N runs; the corollary is that a trend is a different document with a different reader, and merging the two produces a page that serves neither.
Every job's runtime is the subject of Section 8, and it is the interesting case because a close relative of it does belong.
4. "One Screen" Is a Line Count
The budget clause is the only part of the annotation that can be checked mechanically:
BUDGET = 40 # a terminal screen, or a merge-request comment
SUBMITTED = 400
NOT_PASSING = 20
BUCKETS = 5
layouts = [
("one row per job", SUBMITTED + 1),
("one row per non-passing job", NOT_PASSING + 1),
("one row per bucket", BUCKETS + 1),
("one row per bucket + a facts header", BUCKETS + 1 + 8),
]
print(f"{'layout':<36} {'lines':>6} {'fits in ' + str(BUDGET):>12}")
for name, lines in layouts:
print(f"{name:<36} {lines:>6} {str(lines <= BUDGET):>12}")
print()
over = [l for l in layouts if l[1] > BUDGET]
print("layouts over budget:", [l[0] for l in over])
print(f"the per-job layout is {layouts[0][1] // BUDGET}x the budget")layout lines fits in 40
one row per job 401 False
one row per non-passing job 21 True
one row per bucket 6 True
one row per bucket + a facts header 14 True
layouts over budget: ['one row per job']
the per-job layout is 10x the budgetThe per-job table is ten times the budget and the per-bucket table is a seventh of it. That is the whole argument for the bucket being the unit of a report — and note that the per-non-passing-job layout fits comfortably at 21 lines, which means a report can afford to list every failing job on a normal night and will stop fitting on a bad one.
That is an uncomfortable property: the report degrades exactly when it matters most. The fix is to make the budget explicit in the code rather than discovering it:
BUDGET = 40
def render(buckets, failures, header_lines=8, detail_threshold=20):
"""Per-failure detail while it fits; per-bucket when it does not."""
per_failure = header_lines + len(failures) + 1
if len(failures) <= detail_threshold and per_failure <= BUDGET:
return {"mode": "per-failure", "lines": per_failure, "fits": True}
per_bucket = header_lines + len(buckets) + 1
return {"mode": "per-bucket", "lines": per_bucket,
"fits": per_bucket <= BUDGET, "elided": len(failures) - len(buckets)}
for label, nf, nb in (("quiet night", 5, 2), ("normal night", 20, 5), ("bad night", 180, 7)):
r = render(range(nb), range(nf))
print(f"{label:<14} failures {nf:>3} mode {r['mode']:<12} "
f"lines {r['lines']:>3} fits {r['fits']}"
+ (f" elided {r['elided']}" if "elided" in r else ""))quiet night failures 5 mode per-failure lines 14 fits True
normal night failures 20 mode per-failure lines 29 fits True
bad night failures 180 mode per-bucket lines 16 fits True elided 173The report stays inside the budget on all three nights, and on the bad night it says how many failures it elided. A report that silently truncates is worse than one that aggregates, because truncation looks like aggregation and is not.
5. What Broke, What Changed, Where to Look
The three questions are not three sections of equal size. They have a natural order and a natural shape:
The three questions a regression report answers
SECTIONS = [
("what broke", ["status counts", "failure buckets with counts"], 7),
("what changed", ["new failures", "fixed failures", "runtime outliers",
"coverage delta"], 10),
("where to look", ["representative log", "run directory", "wave"], 0),
]
print(f"{'section':<16} {'items':>6} {'lines':>6} contents")
for name, items, lines in SECTIONS:
print(f"{name:<16} {len(items):>6} {lines:>6} {', '.join(items)}")
print()
print("total items:", sum(len(s[1]) for s in SECTIONS))
print("total lines:", sum(s[2] for s in SECTIONS))
print("items the canonical purpose names:", 8)section items lines contents
what broke 2 7 status counts, failure buckets with counts
what changed 4 10 new failures, fixed failures, runtime outliers, coverage delta
where to look 3 0 representative log, run directory, wave
total items: 9
total lines: 17
items the canonical purpose names: 8Seventeen lines for nine items, against a forty-line budget. The report is not short because anything was sacrificed; it is short because the items that cost lines are the ones that do not earn them.
6. Nothing Decorative, Stated as a Rule
"Nothing decorative" is in the canonical purpose, and Section 3 makes it operational. It is worth stating the rule in the form that survives an argument:
def belongs(name, answers, lines, budget_remaining):
if not answers:
return False, "answers none of the three questions"
if lines > budget_remaining:
return False, f"costs {lines} lines, {budget_remaining} remain"
return True, f"answers {','.join(sorted(answers))}"
CASES = [
("failure buckets", {"broke"}, 5, 30),
("pass-rate pie chart", set(), 8, 30),
("every job's runtime", set(), 400, 30),
("per-failure detail on a bad night", {"broke"}, 180, 30),
]
for name, qs, lines, left in CASES:
ok, why = belongs(name, qs, lines, left)
print(f"{name:<36} {'IN ' if ok else 'OUT':<4} {why}")
print()
print("rejections for answering nothing:", 2)
print("rejections for cost :", 1)
print("both are reasons a reviewer can check.")failure buckets IN answers broke
pass-rate pie chart OUT answers none of the three questions
every job's runtime OUT answers none of the three questions
per-failure detail on a bad night OUT costs 180 lines, 30 remain
rejections for answering nothing: 2
rejections for cost : 1
both are reasons a reviewer can check.Two different rejection reasons, and the distinction matters in a review. An item rejected for answering nothing stays out permanently. An item rejected for cost is welcome the moment there is room — which is why the per-failure detail of Section 4 appears on a quiet night and not on a bad one, from the same report generator.
7. Runtime Outliers Belong; Runtimes Do Not
The canonical purpose names "runtime outliers" and not "runtimes", and the difference is the clearest illustration of the whole chapter:
RUNTIMES = [120 + (i * 7) % 40 for i in range(397)] + [1180, 1240, 1310]
n = len(RUNTIMES)
mean = sum(RUNTIMES) / n
var = sum((x - mean) ** 2 for x in RUNTIMES) / (n - 1)
sd = var ** 0.5
limit = mean + 3 * sd
outliers = sorted((x for x in RUNTIMES if x > limit), reverse=True)
print(f"jobs : {n}")
print(f"mean runtime : {mean:.1f} s")
print(f"stdev : {sd:.1f} s")
print(f"3-sigma limit : {limit:.1f} s")
print(f"outliers : {outliers}")
print()
print(f"lines to report every runtime : {n}")
print(f"lines to report the outliers : {len(outliers)}")
print(f"ratio : {n // len(outliers)}:1")jobs : 400
mean runtime : 147.7 s
stdev : 96.2 s
3-sigma limit : 436.2 s
outliers : [1310, 1240, 1180]
lines to report every runtime : 400
lines to report the outliers : 3
ratio : 133:1Three lines instead of four hundred, and the three lines are the informative ones. A job that took 1310 seconds against a 147-second mean is either a new hang, a machine problem or a test that grew — all of which are "what changed" — while the other 397 runtimes answer nothing at all.
This is Chapter 28.6 Section 3's method reused for a different purpose: there, the measured spread set a threshold; here, it defines which values are worth printing. Both rest on the same idea, that a distribution tells you which of its members are interesting.
RUNTIMES = {"dma_long": 1310, "dma_chain": 1240, "axi_sweep": 1180}
YESTERDAY = {"dma_long": 1295, "dma_chain": 160, "axi_sweep": None}
print(f"{'test':<12} {'tonight':>8} {'last night':>11} reading")
for test, tonight in RUNTIMES.items():
last = YESTERDAY[test]
if last is None:
reading = "no baseline — new test or did not run"
elif tonight > 3 * last:
reading = f"{tonight / last:.1f}x slower — investigate"
else:
reading = "slow, but slow yesterday too"
print(f"{test:<12} {tonight:>8} {str(last):>11} {reading}")
print()
print("outliers that are NEW tonight:",
[t for t, v in RUNTIMES.items()
if YESTERDAY[t] is not None and v > 3 * YESTERDAY[t]])test tonight last night reading
dma_long 1310 1295 slow, but slow yesterday too
dma_chain 1240 160 7.8x slower — investigate
axi_sweep 1180 None no baseline — new test or did not run
outliers that are NEW tonight: ['dma_chain']An outlier that was an outlier yesterday is not news. One of the three slowest jobs tonight is worth a sentence in the report, and it is not the slowest one — dma_long has always been slow, while dma_chain went from 160 seconds to 1240. The report that lists the three slowest jobs puts the wrong one first; the report that compares against a baseline names the right one.
8. The Report, Assembled
from dataclasses import dataclass
from typing import Dict, List, Optional
BUDGET = 40
@dataclass
class Report:
counts: Dict[str, int]
buckets: Dict[str, dict]
new: Optional[List[str]]
fixed: Optional[List[str]]
slow: List[tuple]
coverage_delta: Optional[float]
submitted: int
missing: int
def lines(self):
out = []
out.append(f"regression: {self.submitted} jobs submitted, "
f"{self.submitted - self.missing} accounted"
+ (f", {self.missing} MISSING" if self.missing else ""))
order = ("PASS", "FAIL", "TIMEOUT", "NO_RUN", "INFRA")
out.append(" " + " ".join(f"{s} {self.counts.get(s, 0)}" for s in order))
not_passing = self.submitted - self.counts.get("PASS", 0)
out.append(f" did not pass: {not_passing}"
f" (design {self.counts.get('FAIL', 0)},"
f" infra {self.counts.get('NO_RUN', 0) + self.counts.get('INFRA', 0)})")
out.append("")
out.append("what broke:")
for name, b in self.buckets.items():
out.append(f" {name:<12} {b['count']:>3} {b['log']}")
out.append("")
out.append("what changed:")
if self.new is None:
out.append(" no baseline — new and fixed are UNKNOWN")
else:
out.append(f" new {self.new or 'none'}")
out.append(f" fixed {self.fixed or 'none'}")
for test, tonight, last in self.slow:
out.append(f" slow {test} {tonight}s (was {last}s)")
if self.coverage_delta is not None:
out.append(f" coverage {self.coverage_delta:+.2f} points")
return out
r = Report(
counts={"PASS": 360, "FAIL": 9, "TIMEOUT": 4, "NO_RUN": 6, "INFRA": 1},
buckets={
"handshake": {"count": 3, "log": "runs/axi_burst_7731/sim.log"},
"timeout": {"count": 2, "log": "runs/dma_chain_9004/sim.log"},
"null_deref": {"count": 4, "log": "runs/reg_rw_4410/sim.log"},
},
new=["axi_burst", "reg_rw"],
fixed=["old_wrap"],
slow=[("dma_chain", 1240, 160)],
coverage_delta=-0.18,
submitted=400,
missing=20,
)
body = r.lines()
for line in body:
print(line)
print()
print(f"lines: {len(body)} budget: {BUDGET} fits: {len(body) <= BUDGET}")
print(f"jobs behind this report: {r.submitted}")
print(f"reduction: {r.submitted // len(body)}:1")regression: 400 jobs submitted, 380 accounted, 20 MISSING
PASS 360 FAIL 9 TIMEOUT 4 NO_RUN 6 INFRA 1
did not pass: 40 (design 9, infra 7)
what broke:
handshake 3 runs/axi_burst_7731/sim.log
timeout 2 runs/dma_chain_9004/sim.log
null_deref 4 runs/reg_rw_4410/sim.log
what changed:
new ['axi_burst', 'reg_rw']
fixed ['old_wrap']
slow dma_chain 1240s (was 160s)
coverage -0.18 points
lines: 14 budget: 40 fits: True
jobs behind this report: 400
reduction: 28:1Fourteen lines for four hundred jobs. Every line answers one of the three questions, the MISSING clause is Chapter 30.1 Section 3 refusing to be silent, and each bucket row carries the path that answers "where to look next" at a cost of zero extra lines.
Note what is not here: no pie chart, no sparkline, no per-job runtime, no logo, and no table of the 360 passes. The report is not minimal by taste — it is the result of applying a two-clause test to thirteen candidates.
9. The Second Screen
Section 6 rejected items for two different reasons, and the cost rejection is conditional: the item is wanted and does not fit. That leaves a gap the report can close without spending a line.
ITEMS = [
("every job's status", {"broke"}, 401, "csv — 30.4"),
("every job's runtime", set(), 400, "csv — 30.4"),
("full first-failure text per failure", {"broke"}, 20, "html — 30.6"),
("30-day pass-rate trend", {"changed"}, 4, "a trend view"),
("the failure buckets", {"broke"}, 5, "the report itself"),
]
print(f"{'item':<38} {'lines':>6} lives in")
for name, qs, lines, where in ITEMS:
print(f"{name:<38} {lines:>6} {where}")
print()
on_page = [i for i in ITEMS if i[3] == "the report itself"]
elsewhere = [i for i in ITEMS if i[3] != "the report itself"]
print("items on the page :", len(on_page))
print("items one link away :", len(elsewhere))
print("lines saved by linking :", sum(i[2] for i in elsewhere))item lines lives in
every job's status 401 csv — 30.4
every job's runtime 400 csv — 30.4
full first-failure text per failure 20 html — 30.6
30-day pass-rate trend 4 a trend view
the failure buckets 5 the report itself
items on the page : 1
items one link away : 4
lines saved by linking : 825Eight hundred and twenty-five lines displaced for the cost of four paths. The one-screen budget is not a limit on what the flow publishes; it is a limit on what the first screen shows, and the difference between those two readings is the whole art of the thing.
That reframes the earlier rejections honestly. every job's runtime still answers none of the three questions and still does not belong on the page — but it belongs in the CSV, where a consumer that wants it can compute whatever it likes, and the report does not have to decide in advance which aggregate anybody will need.
LINKS = [
("full results", "regression.csv", "every job, every column — 30.4"),
("failure detail", "report.html", "first failure text, waves — 30.6"),
("trend", "trend.html", "30 days of pass rate"),
]
print("footer:")
for label, path, what in LINKS:
print(f" {label:<16} {path:<18} {what}")
print()
print("footer lines:", len(LINKS))
print("budget spent :", f"{len(LINKS)} of 40")
print("the page now answers a fourth question:", "where is everything else?")footer:
full results regression.csv every job, every column — 30.4
failure detail report.html first failure text, waves — 30.6
trend trend.html 30 days of pass rate
footer lines: 3
budget spent : 3 of 40
the page now answers a fourth question: where is everything else?Three lines buy the entire rest of the flow's output. And note the fourth question is not a new one smuggled past the test — where is everything else is "where to look next" applied to the report rather than to a failure, which is why the footer belongs and a pie chart does not.
10. Predict, Run, Observe
# A
print("A:", 400 + 1 > 40, 5 + 1 > 40)
# B
print("B:", bool(set()), bool({"changed"}))
# C
RT = {"dma_long": (1310, 1295), "dma_chain": (1240, 160)}
print("C:", [t for t, (now, was) in RT.items() if now > 3 * was])
# D
print("D:", 400 // 14)The verified output:
A: True False
B: False True
C: ['dma_chain']
D: 28A — a per-job table busts the budget and a per-bucket table uses a seventh of it. Section 4.
B — the whole test, as a truth value: an empty answer set is falsy, which is all "decorative" means.
C — the slowest job is not the interesting one. dma_long was slow yesterday; dma_chain is 7.8x slower than it was.
D — 28 jobs per line of report.
11. Exercises
Exercise 1 — write the spec as a test. Take your own report and score each of its items against the three questions.
Exercise 2 — find an item that answers nothing. Name one thing in your current dashboard that answers none of the three, and its line cost.
Exercise 3 — the pie chart defence. Write the strongest case for a pass-rate chart, then answer it.
Exercise 4 — trend is a different document. Say who reads your trend view and who reads tonight's report, and whether they are the same person.
Exercise 5 — your budget. Decide the line budget for your report and say what sets it — a terminal, a merge-request comment, an email.
Exercise 6 — count your layouts. Compute the line count of a per-job, per-failing-job and per-bucket layout for one real night.
Exercise 7 — the uncomfortable property. Explain why a per-failing-job report degrades exactly on the nights it matters most.
Exercise 8 — degrade deliberately. Implement the per-failure/per-bucket switch and show it on a quiet and a bad night.
Exercise 9 — truncation versus aggregation. Say why silent truncation is worse than aggregation, in two sentences.
Exercise 10 — elided counts. Add the elided-failure count to your aggregated mode.
Exercise 11 — the free question. Confirm that adding a log path to an existing row costs no lines, and say why that question is still the one most often unanswered.
Exercise 12 — two rejection reasons. For three items you have excluded from a report, say whether each was excluded for answering nothing or for cost.
Exercise 13 — re-admit on a quiet night. Name an item you would include when there is room, and the condition you would gate it on.
Exercise 14 — outliers, not runtimes. Compute the 3-sigma runtime limit for your regression and list the jobs beyond it.
Exercise 15 — the ratio. Report how many lines every runtime would cost against how many the outliers cost.
Exercise 16 — an old outlier. Find a job in your flow that is always slow, and say why it should not be in tonight's report.
Exercise 17 — the wrong one first. Construct a night where the slowest job is not the one worth reporting.
Exercise 18 — no baseline for a runtime. Decide what to print for a test that did not run last night.
Exercise 19 — the MISSING clause. Add the missing-job count to your report header and say what it displaces.
Exercise 20 — design versus infra on one line. Write the one line that tells a reviewer whether tonight is a design or a machine problem.
Exercise 21 — your reduction. Compute jobs-per-line for your report and compare it with 28.
Exercise 22 — read it cold. Hand your report to somebody who was not on the project and ask them what broke, what changed and where to look.
Exercise 23 — the item you will be asked for. Predict the next item somebody will ask you to add, and score it in advance.
Exercise 24 — defend a deletion. Pick one thing to remove from your report and write the two-sentence justification using the test.
12. Summary
- The annotation is a specification, not a defect — three questions and a budget, and the chapter is what happens when you apply them literally.
- The three questions make "nothing decorative" checkable. An item either answers one of them or it does not; it is not a matter of taste.
- Of thirteen candidates, four answered nothing and cost 415 lines between them — more than fourteen times what the useful content costs.
- A pass-rate chart declines to say what broke. The reviewer is there because of the 5%.
- A thirty-day trend is a different document with a different reader. Merging it into tonight's report serves neither.
- "One screen" is a line count. A per-job table was 10x the budget; a per-bucket table was a seventh of it.
- A per-failing-job report degrades exactly when it matters, so make the budget explicit and switch to per-bucket deliberately.
- Silent truncation is worse than aggregation, because it looks like aggregation. Report the elided count.
- "Where to look next" costs zero lines — it is an attribute of a row that already exists, which is why it is both the cheapest question and the most often unanswered.
- There are two reasons to reject an item: it answers nothing, or it does not fit. The first is permanent and the second is conditional.
- Runtime outliers belong and runtimes do not — three lines instead of four hundred, at a ratio of 133:1.
- An outlier that was an outlier yesterday is not news. The slowest job tonight was not the one worth reporting; the one that went from 160s to 1240s was.
- Fourteen lines for four hundred jobs, with every line answering one of the three questions.
- The one-screen budget limits the first screen, not what the flow publishes. Four paths displaced 825 lines.
- An item rejected for cost belongs in another artefact — every job's runtime has no place on the page and a natural home in the CSV.
- A three-line footer buys the rest of the flow's output, and "where is everything else?" is "where to look next" applied to the report.
The next chapter takes the "what changed" section apart, because separating a new failure from yesterday's known one is harder than a set difference.
Where this fits
Part of the Python curriculum.