Python Part II · Module 29 · Coverage Automation
Per-Block Coverage and the Closure View
A closure meeting has one hour and a design with forty blocks, so somebody has to turn forty numbers into one. That reduction is where most coverage reporting goes wrong, and not because the arithmetic is hard — because there are two reasonable ways to do it, they disagree by twenty-three points on ordinary data, and they disagree about whether the design passes. The choice between them is not a matter of taste; one of them answers the question the meeting is actually asking.
Intermediate18 min readPythonPart IIAutomationCoverageReporting
Part II · Module 29 · Chapter 29.3 · Per-Block Coverage and the Closure View
1. What a Closure Meeting Asks
A coverage closure review is not browsing. It arrives with four questions and leaves when they are answered:
QUESTIONS = [
("Are we done?", "one number against one policy"),
("How much is left?", "uncovered bins, not percentage points"),
("Who owns it?", "a block name, and a person"),
("Is it moving?", "the same numbers from last week — 29.4"),
]
print(f"{'the question':<22} what answers it")
for q, a in QUESTIONS:
print(f"{q:<22} {a}")
print()
print("questions a single design-wide percentage answers:", 1)
print("questions it cannot answer :", 3)the question what answers it
Are we done? one number against one policy
How much is left? uncovered bins, not percentage points
Who owns it? a block name, and a person
Is it moving? the same numbers from last week — 29.4
questions a single design-wide percentage answers: 1
questions it cannot answer : 3The summary has to produce the one number and preserve the route to the other three. Averaging the per-block detail away satisfies the first question and destroys the rest, which is the failure this chapter is about.
2. The Annotation
The canonical annotation is a dict comprehension with a hole in it:
by_block = {b: weighted(cov[b]) for b in blocks}The comprehension is right — per block, keyed by block, which is Chapter 12.2's form and exactly the shape a closure table needs. weighted is undefined, and that is the whole chapter. Weighted by what?
CANDIDATES = [
("unweighted mean of block percentages", "every block counts equally"),
("bins covered / bins coverable", "every bin counts equally"),
("risk or area weights", "somebody's judgement, written down"),
]
for name, what in CANDIDATES:
print(f" {name:<38} {what}")
print()
print("candidates that need only the data:", 2)
print("candidates that need a policy :", 1) unweighted mean of block percentages every block counts equally
bins covered / bins coverable every bin counts equally
risk or area weights somebody's judgement, written down
candidates that need only the data: 2
candidates that need a policy : 1The first two look like two ways of computing the same thing. They are not, and the next section measures how far apart they are.
3. Two Summaries, One Data Set, Opposite Verdicts
Here is a design with six blocks. Every number below is derived from the same six pairs:
import statistics
BLOCKS = [
("alu", 196, 200),
("regfile", 498, 500),
("fifo", 135, 160),
("dma", 410, 1200),
("timer", 40, 40),
("gpio", 38, 40),
]
print(f"{'block':<9} {'covered':>8} {'bins':>6} {'pct':>8}")
for name, hits, total in BLOCKS:
print(f"{name:<9} {hits:>8} {total:>6} {100.0*hits/total:>7.2f}%")
pcts = [100.0 * h / t for _, h, t in BLOCKS]
H = sum(h for _, h, _ in BLOCKS)
T = sum(t for _, _, t in BLOCKS)
print()
print(f"unweighted mean of block percentages : {statistics.mean(pcts):.2f}%")
print(f"bins covered / bins coverable : {100.0*H/T:.2f}% ({H}/{T})")
print(f"difference : {statistics.mean(pcts) - 100.0*H/T:+.2f} points")block covered bins pct
alu 196 200 98.00%
regfile 498 500 99.60%
fifo 135 160 84.38%
dma 410 1200 34.17%
timer 40 40 100.00%
gpio 38 40 95.00%
unweighted mean of block percentages : 85.19%
bins covered / bins coverable : 61.54% (1317/2140)
difference : +23.65 pointsTwenty-three and a half points apart, from identical data. Neither is a mistake; they are answers to different questions. And the gap is not academic, because a threshold sits between them:
import statistics
BLOCKS = [("alu", 196, 200), ("regfile", 498, 500), ("fifo", 135, 160),
("dma", 410, 1200), ("timer", 40, 40), ("gpio", 38, 40)]
pcts = [100.0 * h / t for _, h, t in BLOCKS]
H = sum(h for _, h, _ in BLOCKS)
T = sum(t for _, _, t in BLOCKS)
FLOOR = 80.0
for label, value in (("unweighted mean", statistics.mean(pcts)),
("bins covered/coverable", 100.0 * H / T)):
print(f"{label:<24} {value:>6.2f} floor {FLOOR} "
f"{'PASS' if value >= FLOOR else 'FAIL'}")
print()
print("the design either passes or fails depending on which average was chosen.")unweighted mean 85.19 floor 80.0 PASS
bins covered/coverable 61.54 floor 80.0 FAIL
the design either passes or fails depending on which average was chosen.One data set, two defensible summaries, opposite decisions. This is the situation a reporting layer exists to prevent, and it cannot be prevented by being careful with the arithmetic — both calculations are correct. It is prevented by deciding, once and in writing, which question the headline number answers.
4. One Block Owns Almost All of the Gap
The reason the two numbers diverge is visible as soon as you stop looking at percentages:
BLOCKS = [("alu", 196, 200), ("regfile", 498, 500), ("fifo", 135, 160),
("dma", 410, 1200), ("timer", 40, 40), ("gpio", 38, 40)]
gaps = sorted(((t - h, name, h, t) for name, h, t in BLOCKS), reverse=True)
total_gap = sum(g for g, _, _, _ in gaps)
print(f"{'block':<9} {'uncovered':>10} {'share of all':>13} {'pct':>8}")
for gap, name, h, t in gaps:
print(f"{name:<9} {gap:>10} {100.0*gap/total_gap:>12.1f}% {100.0*h/t:>7.2f}%")
print()
print("uncovered bins in the design:", total_gap)
print("blocks:", len(BLOCKS))
print("a block's share of the unweighted mean: %.1f%%" % (100.0 / len(BLOCKS)))block uncovered share of all pct
dma 790 96.0% 34.17%
fifo 25 3.0% 84.38%
alu 4 0.5% 98.00%
regfile 2 0.2% 99.60%
gpio 2 0.2% 95.00%
timer 0 0.0% 100.00%
uncovered bins in the design: 823
blocks: 6
a block's share of the unweighted mean: 16.7%dma owns 96.0% of every uncovered bin in the design and gets 16.7% of the unweighted mean, because the mean counts blocks and the work lives in bins. That is the entire 23.65-point discrepancy in one sentence.
Note what this makes of the closure conversation. The honest agenda for this design is one item long — dma — and the unweighted mean actively conceals that by reporting a comfortable 85% with five green blocks around one red one.
5. A Lower Percentage Can Mean Less Work
The gap table above contains a result worth isolating, because it inverts the instinct that drives most coverage queues:
PAIR = [("gpio", 38, 40), ("alu", 196, 200)]
for name, hits, total in PAIR:
print(f"{name:<6} {100.0*hits/total:>6.2f}% uncovered bins: {total - hits}")
print()
print("ranked by percentage (worst first):",
[n for n, h, t in sorted(PAIR, key=lambda b: 100.0*b[1]/b[2])])
print("ranked by work remaining (most first):",
[n for n, h, t in sorted(PAIR, key=lambda b: -(b[2]-b[1]))])
print()
print("the block with the LOWER percentage has HALF the work.")gpio 95.00% uncovered bins: 2
alu 98.00% uncovered bins: 4
ranked by percentage (worst first): ['gpio', 'alu']
ranked by work remaining (most first): ['alu', 'gpio']
the block with the LOWER percentage has HALF the work.gpio at 95% has two uncovered bins; alu at 98% has four. A queue ordered by percentage puts gpio first and sends somebody to close two bins while four sit behind them. The two orderings are exact reverses of each other on this pair.
The reason is the denominator: gpio has forty bins, so one bin is worth 2.5 points, while alu has two hundred, so one bin is worth 0.5. A percentage point is not a unit of work, and it is worth a different amount in every block. Rank by uncovered bins.
BLOCKS = [("alu", 196, 200), ("regfile", 498, 500), ("fifo", 135, 160),
("dma", 410, 1200), ("timer", 40, 40), ("gpio", 38, 40)]
print(f"{'block':<9} {'bins':>6} {'one bin is worth':>18}")
for name, hits, total in sorted(BLOCKS, key=lambda b: b[2]):
print(f"{name:<9} {total:>6} {100.0/total:>17.2f} points")
print()
print("ratio between the most and least valuable point:",
f"{(100.0/40) / (100.0/1200):.0f}x")block bins one bin is worth
timer 40 2.50 points
gpio 40 2.50 points
fifo 160 0.62 points
alu 200 0.50 points
regfile 500 0.20 points
dma 1200 0.08 points
ratio between the most and least valuable point: 30xA bin in gpio moves its block's percentage thirty times as far as a bin in dma. Any process that rewards percentage movement will drain gpio and never touch dma, which is the behaviour the 96% figure in Section 4 predicts.
6. So What Is weighted?
With Sections 3 to 5 in hand, the annotation's missing function has an answer, and it is simpler than the name suggests:
BLOCKS = [("alu", 196, 200), ("regfile", 498, 500), ("fifo", 135, 160),
("dma", 410, 1200), ("timer", 40, 40), ("gpio", 38, 40)]
# Weighting each block's percentage by its share of the bins...
T = sum(t for _, _, t in BLOCKS)
weighted_mean = sum((100.0 * h / t) * (t / T) for _, h, t in BLOCKS)
# ...is the same arithmetic as the design total.
H = sum(h for _, h, _ in BLOCKS)
design_total = 100.0 * H / T
print(f"bin-weighted mean of percentages: {weighted_mean:.10f}")
print(f"sum(hits) / sum(total) : {design_total:.10f}")
print(f"identical: {abs(weighted_mean - design_total) < 1e-9}")
print()
print("so the honest `weighted` is not a weighted mean at all —")
print("it is the pair from 29.1 §6, summed.")bin-weighted mean of percentages: 61.5420560748
sum(hits) / sum(total) : 61.5420560748
identical: True
so the honest `weighted` is not a weighted mean at all —
it is the pair from 29.1 §6, summed.Weighting each block's percentage by its share of the bins is algebraically identical to summing the numerators and denominators, because the weight cancels the denominator it came from. So the implementation needs no weighting function, needs no policy, and needs exactly one thing the annotation does not have: the pair. A flow that kept only percentages cannot compute this at all, which is Chapter 29.1 Section 6 earning its keep for the second time.
Any other weighting — risk, area, ownership — is a different claim, and the rule from Chapter 29.2 Section 7 applies unchanged: it is a policy, so it is recorded with its reason:
WEIGHTS = {
"dma": (3.0, "customer-visible datapath; a hole here ships"),
"alu": (1.0, "default"),
"regfile": (1.0, "default"),
"fifo": (1.0, "default"),
"timer": (0.5, "legacy block, frozen RTL, no new features"),
"gpio": (0.5, "legacy block, frozen RTL, no new features"),
}
print(f"{'block':<9} {'weight':>7} reason")
for name, (w, why) in sorted(WEIGHTS.items()):
print(f"{name:<9} {w:>7.1f} {why}")
print()
print("weights that are not 1.0:", sorted(n for n, (w, _) in WEIGHTS.items() if w != 1.0))
print("weights with a recorded reason:", sum(1 for _, (_, why) in WEIGHTS.items() if why))
print("a weight without a reason is a number nobody can defend in review.")block weight reason
alu 1.0 default
dma 3.0 customer-visible datapath; a hole here ships
fifo 1.0 default
gpio 0.5 legacy block, frozen RTL, no new features
regfile 1.0 default
timer 0.5 legacy block, frozen RTL, no new features
weights that are not 1.0: ['dma', 'gpio', 'timer']
weights with a recorded reason: 6
a weight without a reason is a number nobody can defend in review.Risk weights are legitimate and they are not a summary — they are a prioritisation. Keep them out of the headline number and apply them to the queue, so the figure reported to a meeting stays a plain statement about bins and the judgement stays visible as judgement.
7. The Closure View
Now assemble the thing the meeting reads. One headline, one row per block, ordered by work remaining, with ownership:
BLOCKS = [
("alu", 196, 200, "verif-core"),
("regfile", 498, 500, "verif-core"),
("fifo", 135, 160, "verif-io"),
("dma", 410, 1200, "verif-io"),
("timer", 40, 40, "verif-core"),
("gpio", 38, 40, "verif-io"),
]
FLOOR = 80.0
H = sum(h for _, h, _, _ in BLOCKS)
T = sum(t for _, _, t, _ in BLOCKS)
overall = 100.0 * H / T
print(f"functional coverage {overall:.2f}% ({H}/{T} bins) "
f"floor {FLOOR} {'PASS' if overall >= FLOOR else 'FAIL'}")
print(f"uncovered bins {T - H}")
print()
print(f"{'block':<9} {'pct':>7} {'left':>6} {'share':>7} owner")
rows = sorted(BLOCKS, key=lambda b: -(b[2] - b[1]))
for name, hits, total, owner in rows:
gap = total - hits
print(f"{name:<9} {100.0*hits/total:>6.2f}% {gap:>6} "
f"{100.0*gap/(T-H):>6.1f}% {owner}")
print()
by_owner = {}
for name, hits, total, owner in BLOCKS:
by_owner[owner] = by_owner.get(owner, 0) + (total - hits)
for owner, gap in sorted(by_owner.items(), key=lambda kv: -kv[1]):
print(f"{owner:<12} owns {gap:>4} uncovered bins ({100.0*gap/(T-H):.1f}%)")functional coverage 61.54% (1317/2140 bins) floor 80.0 FAIL
uncovered bins 823
block pct left share owner
dma 34.17% 790 96.0% verif-io
fifo 84.38% 25 3.0% verif-io
alu 98.00% 4 0.5% verif-core
regfile 99.60% 2 0.2% verif-core
gpio 95.00% 2 0.2% verif-io
timer 100.00% 0 0.0% verif-core
verif-io owns 817 uncovered bins (99.3%)
verif-core owns 6 uncovered bins (0.7%)Four questions, four answers, in twelve lines. Are we done? No — 61.54% against a floor of 80. How much is left? 823 bins. Who owns it? verif-io, with 99.3% of the remaining work and dma as the single item. Is it moving? That is Chapter 29.4.
The ordering is doing real work here. By percentage, this table would open with dma and then fifo and then gpio — and a reader would spend attention on gpio's 95% without ever learning it represents two bins. Ordered by work remaining, the table is self-truncating: after the second row, the remaining four rows account for 0.9% of the problem and a reader can stop.
8. Keeping the Detail One Link Away
The canonical purpose asks for the per-block detail to be one link away rather than averaged out of existence, and that is a property of the record, not the table:
from dataclasses import dataclass
@dataclass(frozen=True)
class BlockCoverage:
block: str
hits: int
total: int
owner: str
report: str
@property
def pct(self):
return 100.0 * self.hits / self.total
@property
def gap(self):
return self.total - self.hits
rows = [
BlockCoverage("dma", 410, 1200, "verif-io", "cov/dma/func.rpt"),
BlockCoverage("fifo", 135, 160, "verif-io", "cov/fifo/func.rpt"),
]
for r in rows:
print(f"{r.block:<6} {r.pct:>6.2f}% {r.gap:>4} left -> {r.report}")
print()
print("headline recomputable from the rows:",
f"{100.0*sum(r.hits for r in rows)/sum(r.total for r in rows):.2f}%")
print("rows discarded to produce the headline:", 0)
print("clicks from the headline to a tool report:", 1)dma 34.17% 790 left -> cov/dma/func.rpt
fifo 84.38% 25 left -> cov/fifo/func.rpt
headline recomputable from the rows: 40.07%
rows discarded to produce the headline: 0
clicks from the headline to a tool report: 1The headline is derived from the rows rather than stored beside them, so it cannot drift out of agreement with its own detail — the redundancy Chapter 21.1 Section 3 removed from a result record, applied to a summary. And each row carries the path to the tool's own report, which is Chapter 28.1 Section 12's rule in its third setting: extract what a program compares, point at what a person reads.
Note the headline over these two rows is 40.07%, not the 61.54% from Section 7 — because two rows are not the design, and these happen to be the two worst of them. A derived headline is always the headline of the rows you have, which is a feature: it is impossible to publish a design-wide number from a partial set without noticing.
9. The Block That Is Not in the Report
Section 8 noted that a derived headline is always the headline of the rows you have. That is a safety property and it is only safe if somebody counts the rows:
EXPECTED_BLOCKS = {"alu", "regfile", "fifo", "dma", "timer", "gpio"}
REPORTED = [
("alu", 196, 200),
("regfile", 498, 500),
("fifo", 135, 160),
("timer", 40, 40),
("gpio", 38, 40),
]
present = {name for name, _, _ in REPORTED}
missing = sorted(EXPECTED_BLOCKS - present)
H = sum(h for _, h, _ in REPORTED)
T = sum(t for _, _, t in REPORTED)
print(f"blocks expected : {len(EXPECTED_BLOCKS)}")
print(f"blocks reported : {len(present)}")
print(f"missing : {missing}")
print()
print(f"headline over what was reported: {100.0*H/T:.2f}% ({H}/{T})")
print(f"headline with dma included : {100.0*(H+410)/(T+1200):.2f}% ({H+410}/{T+1200})")
print(f"difference : {100.0*H/T - 100.0*(H+410)/(T+1200):+.2f} points")blocks expected : 6
blocks reported : 5
missing : ['dma']
headline over what was reported: 96.49% (907/940)
headline with dma included : 61.54% (1317/2140)
difference : +34.95 pointsdma failed to elaborate, or its coverage database was not written, or the merge step dropped it. The report that came back is internally consistent and complete-looking, and it says 96.49% where the truth is 61.54% — the same 61.54% Section 7 published from the full set. A design that is 61% covered is reporting as very nearly closed.
This is the one failure mode in this chapter that a careful reader cannot catch, because nothing in the output is wrong — the five rows are accurate, the arithmetic over them is right, and the derived headline correctly summarises the rows it was given. The error is entirely in what is absent, and the only defence is an expected-set check:
EXPECTED = {"alu", "regfile", "fifo", "dma", "timer", "gpio"}
REPORTED = {"alu", "regfile", "fifo", "timer", "gpio"}
def summarise(expected, reported):
missing = sorted(expected - reported)
extra = sorted(reported - expected)
return {
"complete": not missing,
"missing": missing,
"unexpected": extra,
"coverage_of_the_design": f"{len(reported)}/{len(expected)} blocks",
}
for label, got in (("tonight", REPORTED), ("complete", EXPECTED)):
s = summarise(EXPECTED, got)
print(f"{label:<9} complete={str(s['complete']):<6} "
f"missing={str(s['missing']):<9} {s['coverage_of_the_design']}")
print()
print("a headline published from an incomplete set should be:", "labelled, or withheld")tonight complete=False missing=['dma'] 5/6 blocks
complete complete=True missing=[] 6/6 blocks
a headline published from an incomplete set should be: labelled, or withheldLoop over the expected set rather than the reported one. Chapter 25.5 made the same argument about a test missing from a regression's results: you cannot detect an absence by iterating over what arrived, because what is absent is not there to be iterated. The expected set is the only place the knowledge lives.
10. The Rollup, Assembled
from dataclasses import dataclass
@dataclass(frozen=True)
class BlockCoverage:
block: str
metric: str
hits: int
total: int
owner: str
report: str
def rollup(rows, floor, metric):
"""One headline per metric, derived from the rows; nothing averaged away."""
mine = [r for r in rows if r.metric == metric]
if not mine:
return None
H = sum(r.hits for r in mine)
T = sum(r.total for r in mine)
pct = 100.0 * H / T
return {
"metric": metric,
"pct": round(pct, 2),
"hits": H,
"total": T,
"gap": T - H,
"verdict": "PASS" if pct >= floor else "FAIL",
"blocks": len(mine),
"queue": [(r.block, r.total - r.hits, r.owner, r.report)
for r in sorted(mine, key=lambda r: -(r.total - r.hits))
if r.total > r.hits],
}
ROWS = [
BlockCoverage("alu", "functional", 196, 200, "verif-core", "cov/alu/f.rpt"),
BlockCoverage("regfile", "functional", 498, 500, "verif-core", "cov/rf/f.rpt"),
BlockCoverage("fifo", "functional", 135, 160, "verif-io", "cov/fifo/f.rpt"),
BlockCoverage("dma", "functional", 410, 1200, "verif-io", "cov/dma/f.rpt"),
BlockCoverage("timer", "functional", 40, 40, "verif-core", "cov/tmr/f.rpt"),
BlockCoverage("gpio", "functional", 38, 40, "verif-io", "cov/gpio/f.rpt"),
BlockCoverage("alu", "branch", 380, 400, "verif-core", "cov/alu/b.rpt"),
BlockCoverage("dma", "branch", 1100, 1300, "verif-io", "cov/dma/b.rpt"),
]
FLOORS = {"functional": 80.0, "branch": 90.0}
for metric in sorted(FLOORS):
s = rollup(ROWS, FLOORS[metric], metric)
print(f"{s['metric']:<11} {s['pct']:>6.2f}% {s['hits']}/{s['total']} "
f"gap {s['gap']:<4} blocks {s['blocks']} {s['verdict']}")
print()
s = rollup(ROWS, FLOORS["functional"], "functional")
print("functional queue, worst work first:")
for block, gap, owner, report in s["queue"]:
print(f" {block:<9} {gap:>4} bins {owner:<11} {report}")
print()
print("blocks with nothing left:", s["blocks"] - len(s["queue"]))
print("rows kept:", len(ROWS))
print("headline is derived, not stored:", True)branch 87.06% 1480/1700 gap 220 blocks 2 FAIL
functional 61.54% 1317/2140 gap 823 blocks 6 FAIL
functional queue, worst work first:
dma 790 bins verif-io cov/dma/f.rpt
fifo 25 bins verif-io cov/fifo/f.rpt
alu 4 bins verif-core cov/alu/f.rpt
regfile 2 bins verif-core cov/rf/f.rpt
gpio 2 bins verif-io cov/gpio/f.rpt
blocks with nothing left: 1
rows kept: 8
headline is derived, not stored: TrueOne function, one headline per metric, a queue ordered by work, and every row still present. Note that branch rolls up over two blocks and functional over six — the rollup reports blocks so a reader can see that the two headlines do not cover the same scope, which is Chapter 29.1 Section 7's warning about comparing figures produced differently.
11. Predict, Run, Observe
import statistics
BLOCKS = [("a", 9, 10), ("b", 100, 1000)]
# A
print("A:", f"{statistics.mean([100.0*h/t for _, h, t in BLOCKS]):.2f}")
# B
print("B:", f"{100.0*sum(h for _, h, _ in BLOCKS)/sum(t for _, _, t in BLOCKS):.2f}")
# C
print("C:", [n for n, h, t in sorted(BLOCKS, key=lambda b: -(b[2]-b[1]))])
# D
print("D:", (10 - 9), (1000 - 100))The verified output:
A: 50.00
B: 10.79
C: ['b', 'a']
D: 1 900A — the unweighted mean says 50%, treating a ten-bin block and a thousand-bin block as equals.
B — the bins say 10.79%. A 39-point gap on two blocks — larger than Section 3's 23.65 because the imbalance between the denominators is larger.
C — ranked by work, b first. Ranked by percentage it would be b too, here — the orderings only diverge when a small block has a lower percentage than a large one, as gpio and alu do.
D — one bin against nine hundred. The number that should drive the queue.
12. Exercises
Exercise 1 — four questions. Take your last closure review and say which of the four questions your summary answered.
Exercise 2 — both averages. Compute the unweighted mean and the bin-weighted total for your own design, and report the gap.
Exercise 3 — opposite verdicts. Find a floor for your data at which the two summaries disagree. If none exists, say what that tells you about your design's balance.
Exercise 4 — the dominant block. Rank your blocks by uncovered bins and give the top block's share of the total gap.
Exercise 5 — re-partition. Split your largest block into three in your data and recompute both summaries. Report which one moved.
Exercise 6 — the inversion. Find two blocks in your design where the lower percentage has less work, and give both pairs of numbers.
Exercise 7 — what a point is worth. Compute points-per-bin for every block and give the ratio between your largest and smallest.
Exercise 8 — prove the identity. Verify on your own data that the bin-weighted mean of percentages equals sum(hits)/sum(total), and explain in one sentence why.
Exercise 9 — risk weights. Write a weight for each of your blocks with a one-line reason, and say why the weighted figure should not be the headline.
Exercise 10 — the closure table. Produce the Section 7 table for your design, ordered by work remaining.
Exercise 11 — self-truncating. Say after how many rows your table accounts for 95% of the remaining work.
Exercise 12 — by owner. Aggregate your uncovered bins by owning team and report the split.
Exercise 13 — derive, do not store. Take a summary that stores its headline and change it to derive it. Say what class of bug that removes.
Exercise 14 — partial sets. Explain what a derived headline does when half the blocks are missing, and why that is better than the alternative.
Exercise 15 — one link away. Add a report path to each of your rows and confirm a reviewer can reach the tool's own output in one step.
Exercise 16 — loop the expected set. Add the expected-block check and run it against a night when a block failed to elaborate.
Exercise 17 — the direction is arbitrary. Construct two missing-block cases, one that raises the headline and one that lowers it.
Exercise 18 — label or withhold. Decide what your flow should publish when the block set is incomplete, and defend the choice.
Exercise 19 — block count beside the headline. Add (n/m blocks) to your summary and say which class of error it makes visible.
Exercise 20 — two headlines from one metric. Roll up one metric over two different block subsets and show both numbers.
Exercise 21 — scope in the rollup. Explain why reporting the block count per metric matters when two metrics cover different subsets.
Exercise 22 — the queue that finishes. Say after how many rows of your closure table a reader can stop, and why that is a property of the ordering.
Exercise 23 — points per bin. Compute points-per-bin for your largest and smallest blocks and say what it predicts about where effort goes.
Exercise 24 — re-partition twice. Split and then merge a block in your data and confirm the bin-weighted total is unchanged both times.
13. Summary
- A closure meeting asks four questions and a single design-wide percentage answers only the first.
- The annotation's
weightedis the whole chapter — weighted by blocks and weighted by bins are different questions, not different estimates. - The two summaries differ by 23.65 points on ordinary data, and at a floor of 80 they return opposite verdicts from identical numbers.
- One block owned 96.0% of the uncovered bins and 16.7% of the unweighted mean, because the mean counts blocks and the work lives in bins.
- An average over blocks depends on how the design is partitioned; splitting a block changes it without covering anything.
- A lower percentage can mean less work: 95% with two bins left against 98% with four.
- A percentage point is not a unit of work — one bin was worth 30x more in the smallest block than the largest.
- The bin-weighted mean is algebraically identical to
sum(hits)/sum(total), so the honest implementation needs no weighting function — only the pair. - Risk weights are a prioritisation, not a summary. Keep them out of the headline and record a reason for each.
- Order the closure table by work remaining, which makes it self-truncating.
- Derive the headline from the rows rather than storing it beside them, so it cannot disagree with its own detail.
- Carry a report path on every row — extract what a program compares, point at what a person reads.
- A missing block produces a report that is internally consistent and wrong — 96.49% where the truth was 61.54%, with every row accurate.
- Detect an absence by looping the expected set, never the reported one; what is absent is not there to be iterated.
- The direction of a missing-block error is arbitrary, set by which block vanished, so neither a high nor a low number is a reliable symptom.
- Publish the block count beside the headline —
61.54% (6/6 blocks)makes the whole class visible at a glance. - Report the block count per metric too, because two metrics in one package may cover different subsets of the design.
The next chapter asks what any of these numbers mean compared with last week's, which turns out to depend entirely on whether the denominator moved.
Where this fits
Part of the Python curriculum.