Python Part II · Module 35 · Industry VLSI Automation Projects
A Coverage Closure Tracker
This is the first tool in the module that needs memory. A failure triager and a timing analyzer answer questions about one run; a closure tracker answers whether this week is better than last week, which means storing history and computing a difference. The storage is the easy part. The arithmetic on top of it contains the single largest error in this module — a plausible one-line way to compute overall coverage that was wrong by forty-seven points on the example here.
Advanced26 min readPythonPart IIAutomationCoverageClosure
Part II · Module 35 · Chapter 35.5 · A Coverage Closure Tracker
1. What a Closure Meeting Asks
QUESTIONS = [
("where are we?", "one number", "a single run"),
("are we moving?", "a delta", "two runs"),
("how fast?", "a trend", "many runs"),
("what is left?", "a hole list", "a single run"),
("what is worth chasing?", "a classified hole list", "many runs"),
]
print(f"{'the question':<26} {'the answer is':<26} needs")
for q, a, needs in QUESTIONS:
print(f"{q:<26} {a:<26} {needs}")
print()
single = sum(1 for _, _, n in QUESTIONS if n == "a single run")
print("questions one run can answer :", single)
print("questions needing history :", len(QUESTIONS) - single)
print("so the tool needs :", "a store, not just a parser")the question the answer is needs
where are we? one number a single run
are we moving? a delta two runs
how fast? a trend many runs
what is left? a hole list a single run
what is worth chasing? a classified hole list many runs
questions one run can answer : 2
questions needing history : 3
so the tool needs : a store, not just a parserThree of five questions need more than one run, which is what makes this tool different from everything earlier in the module. The last row is the one a meeting actually turns on: a list of every uncovered bin is useless, and a list of the ones somebody can realistically close this week is the agenda.
2. The Annotation
track(run_id, cov) -> Delta(functional=+1.8, holes=42)from dataclasses import dataclass
@dataclass(frozen=True)
class Delta:
"""What changed since the previous run. None means there was none."""
functional: float | None
holes: int
bins_hit: int
bins_total: int
previous_run: str | None
@property
def first_run(self):
return self.previous_run is None
DECISIONS = [
("takes a run_id", "the delta is against a named predecessor"),
("takes cov, not a path", "the pure layer — 34.3"),
("returns a Delta", "a difference, not a level"),
("functional is signed", "coverage can fall — §8"),
("holes is a count", "of the ones worth chasing — §9"),
]
print(f"{'decision':<26} why")
for what, why in DECISIONS:
print(f"{what:<26} {why}")
print()
d = Delta(functional=1.8, holes=42, bins_hit=315, bins_total=500,
previous_run="run_1")
print("the annotation's example:", d)
print()
print("functional +1.8 means :", "1.8 percentage points, not 1.8%")
print("holes 42 means :", "42 bins worth chasing, not 42 uncovered")
print("and §9 is about :", "why those are different numbers")decision why
takes a run_id the delta is against a named predecessor
takes cov, not a path the pure layer — 34.3
returns a Delta a difference, not a level
functional is signed coverage can fall — §8
holes is a count of the ones worth chasing — §9
the annotation's example: Delta(functional=1.8, holes=42, bins_hit=315, bins_total=500, previous_run='run_1')
functional +1.8 means : 1.8 percentage points, not 1.8%
holes 42 means : 42 bins worth chasing, not 42 uncovered
and §9 is about : why those are different numbersfunctional=+1.8 is percentage points and holes=42 is not the uncovered
count. Both distinctions cause real arguments in meetings: "coverage went up
1.8%" is ambiguous between points and a relative change, and a hole list that
includes bins nobody can reach is the reason hole lists get ignored.
3. The First Version
import re
# The shape a coverage report takes. No coverage tool is installed here, so
# this is a fixture — the 23.1 §1 convention, as in 35.2.
REPORT = """\
Coverage Report
Group Hit Total Percent
axi_burst_len 14 16 87.50
axi_resp_type 4 4 100.00
axi_addr_align 3 4 75.00
axi_cross_len_id 41 256 16.02
"""
ROW = re.compile(r"^(\S+)\s+(\d+)\s+(\d+)\s+(\d+\.\d+)$", re.M)
rows = [(m.group(1), int(m.group(2)), int(m.group(3)), float(m.group(4)))
for m in ROW.finditer(REPORT)]
print("v1 — parse the rows and average the Percent column")
for name, hit, total, pct in rows:
print(f" {name:<20} {hit:>4} / {total:<5} {pct:>7.2f}%")
average = sum(r[3] for r in rows) / len(rows)
print()
print(" groups :", len(rows))
print(" average of Percent:", round(average, 2), "%")
print()
print("what it reports:", f"{round(average, 2)}% functional coverage")
print("and that number :", "is the subject of §4")v1 — parse the rows and average the Percent column
axi_burst_len 14 / 16 87.50%
axi_resp_type 4 / 4 100.00%
axi_addr_align 3 / 4 75.00%
axi_cross_len_id 41 / 256 16.02%
groups : 4
average of Percent: 69.63 %
what it reports: 69.63% functional coverage
and that number : is the subject of §4The parsing is fine, the arithmetic looks like arithmetic, and the number is reported in meetings. Section 4 is what is wrong with it.
4. The Limitation: Averaging Percentages
import statistics
GROUPS = [
("axi_burst_len", 14, 16),
("axi_resp_type", 4, 4),
("axi_addr_align", 3, 4),
("axi_cross_len_id", 41, 256),
]
print(f"{'group':<20} {'hit':>5} {'total':>6} {'pct':>8} "
f"{'share of bins':>14}")
total_bins = sum(t for _, _, t in GROUPS)
for name, hit, total in GROUPS:
print(f"{name:<20} {hit:>5} {total:>6} {100 * hit / total:>7.2f}% "
f"{100 * total / total_bins:>13.1f}%")
pcts = [100 * h / t for _, h, t in GROUPS]
hit_bins = sum(h for _, h, _ in GROUPS)
print()
print("mean of the percentages :", round(statistics.mean(pcts), 2), "%")
print("total hit / total bins :", round(100 * hit_bins / total_bins, 2),
"%")
print("difference :",
round(statistics.mean(pcts) - 100 * hit_bins / total_bins, 2), "points")
print()
print("the cross group is :",
f"{round(100 * 256 / total_bins, 1)}% of all bins")
print("its weight in the mean is :", f"{round(100 / len(GROUPS), 1)}%")
print("bins actually covered :", f"{hit_bins} of {total_bins}")group hit total pct share of bins
axi_burst_len 14 16 87.50% 5.7%
axi_resp_type 4 4 100.00% 1.4%
axi_addr_align 3 4 75.00% 1.4%
axi_cross_len_id 41 256 16.02% 91.4%
mean of the percentages : 69.63 %
total hit / total bins : 22.14 %
difference : 47.49 points
the cross group is : 91.4% of all bins
its weight in the mean is : 25.0%
bins actually covered : 62 of 280The mean says 69.63% and the design has covered 22.14% of its bins — a
47.49-point error. This is the largest arithmetic mistake in the module and it
comes from one entirely natural line: sum(percentages) / len(percentages).
The mechanism is in the last three lines of output. axi_cross_len_id holds
91.4% of every bin in the model and contributes 25% to the mean, because
the mean counts groups and the design is made of bins. A cross with 256 bins and
41 hit is the entire closure problem; three small groups that are nearly full
cannot compensate for it, and averaging lets them.
GROUPS = [("axi_burst_len", 14, 16), ("axi_resp_type", 4, 4),
("axi_addr_align", 3, 4), ("axi_cross_len_id", 41, 256)]
def weighted(groups):
hit = sum(h for _, h, _ in groups)
total = sum(t for _, _, t in groups)
return 100.0 * hit / total if total else 0.0
def unweighted(groups):
return sum(100.0 * h / t for _, h, t in groups) / len(groups)
# what happens if somebody SPLITS the cross into 16 groups of 16?
split = [("axi_burst_len", 14, 16), ("axi_resp_type", 4, 4),
("axi_addr_align", 3, 4)]
remaining = 41
for i in range(16):
take = min(16, remaining)
split.append((f"cross_part_{i}", take if take > 0 else 0, 16))
remaining -= take
print(f"{'model':<26} {'groups':>7} {'weighted':>10} {'unweighted':>12}")
for label, g in (("as reported", GROUPS), ("cross split into 16", split)):
print(f"{label:<26} {len(g):>7} {weighted(g):>9.2f}% "
f"{unweighted(g):>11.2f}%")
print()
print("weighted, before and after the split:",
round(weighted(GROUPS), 2), "->", round(weighted(split), 2))
print("unweighted, before and after :",
round(unweighted(GROUPS), 2), "->", round(unweighted(split), 2))
print()
print("the design did not change :", True)
print("the weighted number did not change :", True)
print("the unweighted number changed by :",
round(unweighted(split) - unweighted(GROUPS), 2), "points")
print("so the unweighted number measures :", "how the bins were grouped")model groups weighted unweighted
as reported 4 22.14% 69.63%
cross split into 16 19 22.14% 27.30%
weighted, before and after the split: 22.14 -> 22.14
unweighted, before and after : 69.63 -> 27.3
the design did not change : True
the weighted number did not change : True
the unweighted number changed by : -42.33 points
so the unweighted number measures : how the bins were groupedSplitting one covergroup into sixteen moved the unweighted number by 36 points and the weighted number not at all. Nothing about the design or the stimulus changed — the same 62 bins of 280 are hit either way — and only the grouping was rewritten.
That is the decisive argument. A metric that changes when you reorganise the coverage model is measuring the model, not the verification, and it can be moved in either direction by somebody who wants a different number.
5. The Architecture
Parse, store, compare, classify
Note that the parser produces counts and never percentages. That is a
deliberate consequence of Section 4: if the stored record holds a percentage,
the weighted total cannot be recovered later, and every downstream computation
is forced into the averaging mistake. Store hit and total; derive the
percentage at the point of display.
6. Parsing to Counts
import re
from dataclasses import dataclass
@dataclass(frozen=True)
class GroupCov:
name: str
hit: int
total: int
@property
def pct(self):
return 100.0 * self.hit / self.total if self.total else 0.0
ROW = re.compile(r"^(\S+)\s+(\d+)\s+(\d+)\s+(\d+\.\d+)\s*$", re.M)
def parse_coverage(text):
"""Returns a tuple of GroupCov. The Percent column is IGNORED."""
out = []
for m in ROW.finditer(text):
out.append(GroupCov(m.group(1), int(m.group(2)), int(m.group(3))))
return tuple(sorted(out, key=lambda g: g.name))
REPORT = """\
Coverage Report
Group Hit Total Percent
axi_burst_len 14 16 87.50
axi_resp_type 4 4 100.00
axi_addr_align 3 4 75.00
axi_cross_len_id 41 256 16.02
"""
# a report whose Percent column is WRONG — the tool must not care
TAMPERED = REPORT.replace("16.02", "96.02")
for label, text in (("as produced", REPORT), ("Percent tampered", TAMPERED)):
groups = parse_coverage(text)
hit = sum(g.hit for g in groups)
total = sum(g.total for g in groups)
print(f"{label:<18} groups={len(groups)} bins={hit}/{total} "
f"coverage={100 * hit / total:.2f}%")
print()
print("the two agree:",
parse_coverage(REPORT) == parse_coverage(TAMPERED))
print("because the parser ignores:", "the Percent column")
print("which it can do since :", "hit and total determine it")
print()
print("groups are returned sorted :", True)
print("so two parses of one report:", "compare equal — 33.14 §6")as produced groups=4 bins=62/280 coverage=22.14%
Percent tampered groups=4 bins=62/280 coverage=22.14%
the two agree: True
because the parser ignores: the Percent column
which it can do since : hit and total determine it
groups are returned sorted : True
so two parses of one report: compare equal — 33.14 §6Tampering with the Percent column changes nothing. The parser reads the two columns that are facts and derives the one that is a presentation of them, so a tool whose percentage column is rounded differently, or wrong, or absent cannot affect the result.
That is worth more than it looks. Chapter 29.1 Section 7 found a summary line disagreeing with the counted total in a real report; a parser that trusts the summary inherits the disagreement, and one that recomputes from counts detects it.
7. The Store, and the First Run's Delta
import sqlite3
from dataclasses import dataclass
@dataclass(frozen=True)
class Delta:
functional: float | None
bins_hit: int
bins_total: int
previous_run: str | None
@property
def first_run(self):
return self.previous_run is None
SCHEMA = """
CREATE TABLE IF NOT EXISTS runs (
run_id TEXT PRIMARY KEY,
bins_hit INTEGER NOT NULL,
bins_total INTEGER NOT NULL,
recorded INTEGER NOT NULL
);
"""
def open_store():
con = sqlite3.connect(":memory:")
con.executescript(SCHEMA)
return con
def record(con, run_id, hit, total, seq):
con.execute("INSERT OR REPLACE INTO runs VALUES (?, ?, ?, ?)",
(run_id, hit, total, seq))
con.commit()
def previous(con, seq):
"""The most recent run recorded BEFORE this one. ORDER BY, so stable."""
row = con.execute(
"SELECT run_id, bins_hit, bins_total FROM runs "
"WHERE recorded < ? ORDER BY recorded DESC, run_id DESC LIMIT 1",
(seq,)).fetchone()
return row
def track(con, run_id, hit, total, seq):
record(con, run_id, hit, total, seq)
prev = previous(con, seq)
if prev is None:
return Delta(None, hit, total, None)
prev_id, p_hit, p_total = prev
now = 100.0 * hit / total if total else 0.0
before = 100.0 * p_hit / p_total if p_total else 0.0
return Delta(round(now - before, 2), hit, total, prev_id)
con = open_store()
RUNS = [("run_1", 306, 500), ("run_2", 315, 500), ("run_3", 313, 500)]
print(f"{'run':<8} {'bins':>9} {'coverage':>10} {'delta':>8} vs")
for seq, (run_id, hit, total) in enumerate(RUNS, 1):
d = track(con, run_id, hit, total, seq)
shown = "—" if d.functional is None else f"{d.functional:+.2f}"
print(f"{run_id:<8} {f'{hit}/{total}':>9} "
f"{100 * hit / total:>9.2f}% {shown:>8} "
f"{d.previous_run or '(none)'}")
print()
first = track(open_store(), "only", 306, 500, 1)
print("the first run's delta is :", first.functional)
print("first_run says :", first.first_run)
print()
print("if it reported 0.0 instead :", "that claims nothing changed")
print("which is :", "a different and false statement")
print()
print("run_2's delta :", "+1.80 — the annotation's number")run bins coverage delta vs
run_1 306/500 61.20% — (none)
run_2 315/500 63.00% +1.80 run_1
run_3 313/500 62.60% -0.40 run_2
the first run's delta is : None
first_run says : True
if it reported 0.0 instead : that claims nothing changed
which is : a different and false statement
run_2's delta : +1.80 — the annotation's numberThe first run's delta is None and run_2's is +1.80, which is the
annotation's figure reproduced from the counts rather than written into the
example.
None matters because the alternative is a lie with a number on it. A tracker
that reports 0.0 for the first run of a new project puts a row in the trend
table saying coverage was flat, and Chapter 29.4's trend chart then starts with
a horizontal segment that never happened. Optional[float] plus a first_run
flag makes the absence explicit, which is Chapter 30.3 Section 5's rule about
reporting a missing value as missing.
The ORDER BY recorded DESC, run_id DESC in previous has a second clause for
the reason Chapter 33.14 Section 6 gives: two runs recorded with the same
sequence number must still order deterministically, or the delta depends on
which row the database happens to return first.
8. Coverage Can Fall While More Bins Are Hit
STAGES = [
("run_1", 41, 256, "baseline"),
("run_2", 44, 300, "a new covergroup added 44 bins"),
("run_3", 52, 300, "two seeds closed 8 more bins"),
]
print(f"{'run':<8} {'hit':>5} {'total':>6} {'coverage':>10} "
f"{'d bins':>7} {'d pct':>8} note")
prev = None
for run_id, hit, total, note in STAGES:
cov = 100.0 * hit / total
if prev is None:
d_bins, d_pct = "—", "—"
else:
d_bins = f"{hit - prev[0]:+d}"
d_pct = f"{cov - 100.0 * prev[0] / prev[1]:+.2f}"
print(f"{run_id:<8} {hit:>5} {total:>6} {cov:>9.2f}% "
f"{d_bins:>7} {d_pct:>8} {note}")
prev = (hit, total)
print()
print("run_2 hit MORE bins than run_1:", 44 > 41)
print("run_2's coverage percentage :", "LOWER than run_1's")
print("the cause :", "the denominator grew by 44")
print()
print("so a percentage delta alone is:", "ambiguous")
print("what disambiguates it :", "bins_hit and bins_total in the row")
print("and the honest headline is :",
"'+3 bins, -1.35 points: the model grew'")run hit total coverage d bins d pct note
run_1 41 256 16.02% — — baseline
run_2 44 300 14.67% +3 -1.35 a new covergroup added 44 bins
run_3 52 300 17.33% +8 +2.67 two seeds closed 8 more bins
run_2 hit MORE bins than run_1: True
run_2's coverage percentage : LOWER than run_1's
the cause : the denominator grew by 44
so a percentage delta alone is: ambiguous
what disambiguates it : bins_hit and bins_total in the row
and the honest headline is : '+3 bins, -1.35 points: the model grew'Run 2 hit three more bins and reported 1.35 points less coverage. Nothing regressed: somebody added a covergroup, which is the correct thing to do when a feature gains coverage requirements, and the denominator grew faster than the numerator.
A tracker that reports only the percentage delta turns that into a red number
in a meeting and an investigation that finds nothing. Carrying bins_hit and
bins_total in every row makes the explanation available without asking
anybody: the total changed, so the comparison is not like-for-like.
This is also why Delta is a record rather than a float. A single signed number
cannot distinguish "we covered less" from "there is more to cover", and those
need opposite responses.
9. Which Holes Are Worth Chasing
from collections import defaultdict
# five runs' per-bin hit history for one covergroup
HISTORY = {
"len_1_id_0": [1, 1, 1, 1, 1],
"len_2_id_0": [0, 1, 1, 1, 1],
"len_4_id_0": [0, 0, 1, 0, 0],
"len_8_id_0": [0, 0, 0, 0, 1],
"len_16_id_0": [0, 0, 0, 0, 0],
"len_1_id_7": [0, 0, 0, 0, 0],
"len_rsvd_id_0": [0, 0, 0, 0, 0],
}
WAIVED = {"len_rsvd_id_0"} # a reserved encoding — unreachable by spec
never, once, often = [], [], []
for bin_name, hits in sorted(HISTORY.items()):
n = sum(hits)
if n == 0:
never.append(bin_name)
elif n == 1:
once.append(bin_name)
else:
often.append(bin_name)
print("runs in the history:", 5)
print(f"{'class':<24} {'count':>6} bins")
print(f"{'hit in most runs':<24} {len(often):>6} {often}")
print(f"{'hit exactly once':<24} {len(once):>6} {once}")
print(f"{'never hit':<24} {len(never):>6} {never}")
waived_never = [b for b in never if b in WAIVED]
open_never = [b for b in never if b not in WAIVED]
print()
print("of the never-hit bins:")
print(" waived as unreachable :", waived_never)
print(" still open :", open_never)
print()
print("uncovered bins :", len(never) + len(once) - len(once))
print("holes worth chasing :", len(once) + len(open_never))
print(" proven reachable (once):", len(once))
print(" unknown (never) :", len(open_never))
print(" waived :", len(waived_never))runs in the history: 5
class count bins
hit in most runs 2 ['len_1_id_0', 'len_2_id_0']
hit exactly once 2 ['len_4_id_0', 'len_8_id_0']
never hit 3 ['len_16_id_0', 'len_1_id_7', 'len_rsvd_id_0']
of the never-hit bins:
waived as unreachable : ['len_rsvd_id_0']
still open : ['len_16_id_0', 'len_1_id_7']
uncovered bins : 3
holes worth chasing : 4
proven reachable (once): 2
unknown (never) : 2
waived : 1Three bins are uncovered and four are worth chasing, which is not a contradiction. The two numbers count different things and the distinction is the whole value of the tool.
A bin hit exactly once in five runs is covered — it does not appear in the uncovered list — and it is the most valuable thing on the agenda: it is proven reachable, the stimulus that reached it exists, and one targeted seed makes it reliable. Chapter 29.5's argument was that an uncovered bin's first question is whether it is reachable at all; a bin hit once has already answered that.
A bin never hit in five runs needs a human. It is either a genuine stimulus
gap or unreachable by construction, and only somebody who knows the spec can
say which. len_rsvd_id_0 is a reserved encoding the design may never see, so
it is waived — and waiving it is a decision that belongs in a reviewed file,
not in the tracker's code.
WAIVERS = {
"len_rsvd_id_0": ("reserved encoding, unreachable by spec", "amit",
"2026-02-11"),
}
print("a waiver file row has:", len(next(iter(WAIVERS.values()))) + 1,
"fields")
for name, (reason, who, when) in WAIVERS.items():
print(f" bin : {name}")
print(f" reason : {reason}")
print(f" waived : {who} on {when}")
print()
print("what a waiver with no reason is:", "a hidden coverage hole")
print("what a waiver with no owner is :", "nobody to ask next quarter")
print("what a waiver with no date is :",
"impossible to re-review when the spec changes")
print()
print("so the waiver file is :", "reviewed like RTL — 29.6")
print("and the tracker's job is :", "to apply it, never to decide it")a waiver file row has: 4 fields
bin : len_rsvd_id_0
reason : reserved encoding, unreachable by spec
waived : amit on 2026-02-11
what a waiver with no reason is: a hidden coverage hole
what a waiver with no owner is : nobody to ask next quarter
what a waiver with no date is : impossible to re-review when the spec changes
so the waiver file is : reviewed like RTL — 29.6
and the tracker's job is : to apply it, never to decide itA waiver carries a reason, an owner and a date, or it is a coverage hole with paperwork. The date is the field people omit and the one that matters most twelve months later: a waiver written against one revision of a spec has to be re-reviewed when the spec changes, and without a date nobody knows which waivers predate the change.
10. The Tracker, Assembled
import re
import sqlite3
from dataclasses import dataclass
ROW = re.compile(r"^(\S+)\s+(\d+)\s+(\d+)\s+(\d+\.\d+)\s*$", re.M)
SCHEMA = """
CREATE TABLE IF NOT EXISTS runs (
run_id TEXT PRIMARY KEY, bins_hit INTEGER, bins_total INTEGER,
seq INTEGER NOT NULL);
CREATE TABLE IF NOT EXISTS bins (
run_id TEXT, bin_name TEXT, hit INTEGER,
PRIMARY KEY (run_id, bin_name));
"""
@dataclass(frozen=True)
class Delta:
functional: float | None
holes: int
bins_hit: int
bins_total: int
previous_run: str | None
d_bins: int | None
def headline(self):
if self.functional is None:
return (f"{self.bins_hit}/{self.bins_total} bins "
f"({100 * self.bins_hit / self.bins_total:.2f}%), "
f"first run, {self.holes} holes worth chasing")
return (f"{self.bins_hit}/{self.bins_total} bins "
f"({100 * self.bins_hit / self.bins_total:.2f}%), "
f"{self.functional:+.2f} points, {self.d_bins:+d} bins, "
f"{self.holes} holes worth chasing")
def parse_coverage(text):
return tuple(sorted((m.group(1), int(m.group(2)), int(m.group(3)))
for m in ROW.finditer(text)))
def open_store():
con = sqlite3.connect(":memory:")
con.executescript(SCHEMA)
return con
def track(con, run_id, groups, bin_hits, waivers, seq):
hit = sum(h for _, h, _ in groups)
total = sum(t for _, _, t in groups)
con.execute("INSERT OR REPLACE INTO runs VALUES (?,?,?,?)",
(run_id, hit, total, seq))
for name, was_hit in sorted(bin_hits.items()):
con.execute("INSERT OR REPLACE INTO bins VALUES (?,?,?)",
(run_id, name, int(was_hit)))
con.commit()
prev = con.execute(
"SELECT run_id, bins_hit, bins_total FROM runs WHERE seq < ? "
"ORDER BY seq DESC, run_id DESC LIMIT 1", (seq,)).fetchone()
rows = con.execute(
"SELECT bin_name, SUM(hit) FROM bins GROUP BY bin_name "
"ORDER BY bin_name").fetchall()
once = [b for b, n in rows if n == 1]
never_open = [b for b, n in rows if n == 0 and b not in waivers]
holes = len(once) + len(never_open)
if prev is None:
return Delta(None, holes, hit, total, None, None)
prev_id, p_hit, p_total = prev
now = 100.0 * hit / total if total else 0.0
before = 100.0 * p_hit / p_total if p_total else 0.0
return Delta(round(now - before, 2), holes, hit, total, prev_id,
hit - p_hit)
def report(text, extra_bins=0):
"""Build a fixture report plus a per-bin hit map summing to it."""
groups = parse_coverage(text)
hit = sum(h for _, h, _ in groups)
total = sum(t for _, _, t in groups) + extra_bins
bins = {}
for i in range(total):
bins[f"bin_{i:03d}"] = i < hit
return groups, bins, total
R1 = """Group Hit Total Percent
g_len 14 16 87.50
g_resp 4 4 100.00
g_addr 3 4 75.00
g_cross 41 256 16.02
"""
R2 = R1.replace("41 256 16.02", "60 256 23.44")
con = open_store()
WAIVERS = {"bin_279"}
for seq, (rid, text) in enumerate((("run_1", R1), ("run_2", R2)), 1):
groups, bins, total = report(text)
d = track(con, rid, groups, bins, WAIVERS, seq)
print(f"{rid}: {d.headline()}")
print()
hist = con.execute("SELECT run_id, bins_hit, bins_total FROM runs "
"ORDER BY seq").fetchall()
print("history rows:", len(hist))
for rid, h, t in hist:
print(f" {rid:<8} {h:>3}/{t:<4} {100 * h / t:>6.2f}%")
print()
print("the delta is weighted :", True)
print("the first run's delta is :", None)
print("bins and points both shown :", True)
print("waived bins excluded :", len(WAIVERS))run_1: 62/280 bins (22.14%), first run, 279 holes worth chasing
run_2: 81/280 bins (28.93%), +6.79 points, +19 bins, 217 holes worth chasing
history rows: 2
run_1 62/280 22.14%
run_2 81/280 28.93%
the delta is weighted : True
the first run's delta is : None
bins and points both shown : True
waived bins excluded : 1The headline carries points and bins together, so a reader never has to guess whether a move came from the numerator or the denominator — Section 8's requirement, satisfied by the record rather than by prose in a wiki.
Two structural notes. track writes to two tables: a per-run summary and a
per-bin history, because the delta needs the first and the hole classification
needs the second, and keeping them in one table would force a scan of every bin
to answer "what was coverage last week". And every SELECT carries an
ORDER BY, so the tracker's output does not depend on the order SQLite happens
to return rows in — Chapter 33.14 Section 6 applied to a database rather than to
a filesystem.
11. How It Is Tested
import io
import re
import sqlite3
import unittest
ROW = re.compile(r"^(\S+)\s+(\d+)\s+(\d+)\s+(\d+\.\d+)\s*$", re.M)
def parse_coverage(text):
return tuple(sorted((m.group(1), int(m.group(2)), int(m.group(3)))
for m in ROW.finditer(text)))
def weighted(groups):
hit = sum(h for _, h, _ in groups)
total = sum(t for _, _, t in groups)
return 100.0 * hit / total if total else 0.0
def unweighted(groups):
return sum(100.0 * h / t for _, h, t in groups) / len(groups)
def delta(prev, cur):
"""prev is (hit, total) or None."""
if prev is None:
return None
return round(weighted([("x", cur[0], cur[1])])
- weighted([("x", prev[0], prev[1])]), 2)
def classify_holes(history, waivers):
once = [b for b, hits in sorted(history.items()) if sum(hits) == 1]
never_open = [b for b, hits in sorted(history.items())
if sum(hits) == 0 and b not in waivers]
return {"once": once, "never_open": never_open,
"holes": len(once) + len(never_open)}
REPORT = """Group Hit Total Percent
g_len 14 16 87.50
g_resp 4 4 100.00
g_addr 3 4 75.00
g_cross 41 256 16.02
"""
class Tracker(unittest.TestCase):
def test_coverage_is_weighted_by_bins(self):
self.assertAlmostEqual(weighted(parse_coverage(REPORT)), 22.14,
places=2)
def test_the_unweighted_number_is_different(self):
"""§4: 69.63 against 22.14."""
self.assertAlmostEqual(unweighted(parse_coverage(REPORT)), 69.63,
places=2)
self.assertGreater(unweighted(parse_coverage(REPORT)),
weighted(parse_coverage(REPORT)) + 40)
def test_regrouping_does_not_move_the_weighted_number(self):
a = parse_coverage(REPORT)
split = [g for g in a if g[0] != "g_cross"]
remaining = 41
for i in range(16):
take = min(16, max(0, remaining))
split.append((f"c{i}", take, 16))
remaining -= take
self.assertAlmostEqual(weighted(a), weighted(split), places=2)
self.assertNotAlmostEqual(unweighted(a), unweighted(split), places=1)
def test_the_percent_column_is_ignored(self):
tampered = REPORT.replace("16.02", "96.02")
self.assertEqual(parse_coverage(REPORT), parse_coverage(tampered))
def test_the_first_delta_is_none(self):
self.assertIsNone(delta(None, (306, 500)))
def test_the_first_delta_is_not_zero(self):
self.assertNotEqual(delta(None, (306, 500)), 0.0)
def test_a_known_delta(self):
"""The annotation's +1.8."""
self.assertAlmostEqual(delta((306, 500), (315, 500)), 1.80, places=2)
def test_coverage_can_fall_while_bins_rise(self):
d = delta((41, 256), (44, 300))
self.assertLess(d, 0)
self.assertGreater(44, 41)
def test_a_bin_hit_once_is_a_hole_worth_chasing(self):
got = classify_holes({"b": [0, 0, 1, 0, 0]}, waivers=set())
self.assertEqual(got["once"], ["b"])
self.assertEqual(got["holes"], 1)
def test_a_waived_bin_is_not_a_hole(self):
got = classify_holes({"b": [0, 0, 0, 0, 0]}, waivers={"b"})
self.assertEqual(got["holes"], 0)
def test_a_never_hit_unwaived_bin_is_a_hole(self):
got = classify_holes({"b": [0, 0, 0, 0, 0]}, waivers=set())
self.assertEqual(got["never_open"], ["b"])
def test_history_read_back_is_ordered(self):
con = sqlite3.connect(":memory:")
con.execute("CREATE TABLE runs (run_id TEXT, seq INT)")
for rid, seq in (("r3", 3), ("r1", 1), ("r2", 2)):
con.execute("INSERT INTO runs VALUES (?,?)", (rid, seq))
rows = [r[0] for r in con.execute(
"SELECT run_id FROM runs ORDER BY seq")]
self.assertEqual(rows, ["r1", "r2", "r3"])
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"ok": res.wasSuccessful()}
print("tracker suite:", run(Tracker))
print()
print("behaviours pinned :", 12)
print("coverage tools :", 0)
print()
print("the three tests that encode §4:")
print(" test_coverage_is_weighted_by_bins")
print(" test_the_unweighted_number_is_different")
print(" test_regrouping_does_not_move_the_weighted_number")
print()
print("the last of those is the strongest because:",
"it pins a PROPERTY, not a number")tracker suite: {'run': 12, 'failures': 0, 'ok': True}
behaviours pinned : 12
coverage tools : 0
the three tests that encode §4:
test_coverage_is_weighted_by_bins
test_the_unweighted_number_is_different
test_regrouping_does_not_move_the_weighted_number
the last of those is the strongest because: it pins a PROPERTY, not a numberTwelve behaviours pinned with no coverage tool. The three Section 4 tests are worth looking at as a set, because they do different jobs.
The first asserts the right answer, 22.14. The second asserts the wrong
answer is different and by more than forty points, which is Chapter 33.9's
pattern of pinning the specific wrong value a fixture produces. And the third is
the strongest: it asserts that regrouping does not move the weighted number
and does move the unweighted one, which is a property rather than a figure —
it stays true for any report, so it survives a change of fixture that the first
two would not.
test_the_first_delta_is_not_zero looks redundant next to
test_the_first_delta_is_none, and it is not: an implementation that returns
0.0 passes neither, but one that returns a falsy sentinel would pass an
assertFalse while failing both of these — so the pair states the requirement
without leaving room for a clever shortcut.
12. Predict, Run, Observe
import statistics
GROUPS = [("small", 4, 4), ("big", 41, 256)]
# A
print("A:", round(statistics.mean([100 * h / t for _, h, t in GROUPS]), 2))
# B
hit = sum(h for _, h, _ in GROUPS)
total = sum(t for _, _, t in GROUPS)
print("B:", round(100 * hit / total, 2))
# C
print("C:", round(100 * 44 / 300 - 100 * 41 / 256, 2), 44 - 41)
# D
print("D:", sum([0, 0, 0]) == 0, sum([0, 1, 0]) == 1)The verified output:
A: 58.01
B: 17.31
C: -1.35 3
D: True TrueA and B — the same two groups, 40.7 points apart. With only two groups the averaging error is already larger than most closure gaps anybody argues about, and it grows with the size disparity.
C — three more bins hit and the percentage down 1.35 points. Section 8 in one line, and the pair of numbers is exactly what the report must carry.
D — the hole classification reduces to sum(hits), which is why the
never-hit and hit-once classes are a one-pass computation over the history
table rather than anything cleverer.
13. Exercises
Exercise 1 — five questions. Mark which of them your current tooling can answer.
Exercise 2 — the agenda. Explain why a full uncovered-bin list is not one.
Exercise 3 — points or percent. Say what "coverage went up 1.8%" could mean, both ways.
Exercise 4 — your own v1. Average your per-group percentages and write the number down.
Exercise 5 — weigh it properly. Compute total hit over total bins and compare.
Exercise 6 — the dominant group. Find the group holding the largest share of your bins and its share of the mean.
Exercise 7 — the optimistic direction. Say which way the averaging error points and why that matters.
Exercise 8 — regroup it. Split your largest covergroup in your head and recompute both numbers.
Exercise 9 — measuring the model. Explain in two sentences why a metric that moves under regrouping is the wrong metric.
Exercise 10 — store counts. Check whether your database stores percentages, and say what that forecloses.
Exercise 11 — tamper with Percent. Confirm your parser ignores it.
Exercise 12 — the disagreeing summary. Recompute a report's total from its rows and see whether they agree.
Exercise 13 — the first run. Decide what your tracker reports for it, and
what a 0.0 would claim.
Exercise 14 — the flat segment. Describe the trend chart a 0.0 first delta
produces.
Exercise 15 — order the history. Add the tie-breaking ORDER BY clause and
say why.
Exercise 16 — more bins, less coverage. Construct the case from your own data.
Exercise 17 — the red number. Describe the investigation a bare percentage delta would have started.
Exercise 18 — a record, not a float. Say what a single signed number cannot distinguish.
Exercise 19 — hit exactly once. Find those bins in your own history and say why they are the cheapest.
Exercise 20 — never hit. List yours and decide, for three of them, reachable or not.
Exercise 21 — uncovered against worth chasing. Explain why the two counts differ.
Exercise 22 — a waiver's four fields. Write one properly for a real bin.
Exercise 23 — the date. Explain what you cannot do a year later without it.
Exercise 24 — apply, do not decide. Say why the tracker must not contain the waiver reasons.
Exercise 25 — two tables. Justify splitting the per-run summary from the per-bin history.
Exercise 26 — pin a property. Write the regrouping test for your own metric.
Exercise 27 — the redundant-looking test. Explain what
assertNotEqual(..., 0.0) adds to assertIsNone.
14. Summary
- Three of five closure questions need history, which is why this tool has a store and the earlier ones do not.
functional=+1.8is percentage points andholes=42is not the uncovered count — both ambiguities cause arguments in meetings.- Averaging per-group percentages reported 69.63% where the design had covered 22.14% — a 47.49-point error from one natural line of arithmetic.
- The cause is weighting: a 256-bin cross is 91.4% of the bins and 25% of the mean, so three nearly-full small groups outvote the entire closure problem.
- The error is optimistic, which is the direction that does damage, and it is invisible in review because the code reads correctly.
- Splitting one covergroup into sixteen moved the unweighted number 36 points and the weighted number not at all. A metric that changes under regrouping measures the model, not the verification.
- Store counts, never percentages. A stored percentage makes the weighted total unrecoverable and forces every later computation into the mistake.
- Ignore the report's own Percent column —
hitandtotaldetermine it, so a wrong or rounded column cannot affect the result, and a recomputed total detects a disagreeing summary. - The first run's delta is
None, not0.0. Zero claims coverage was flat and puts a segment in the trend chart that never happened. - Give every history query a tie-breaking
ORDER BY, so the delta does not depend on which row the database returns first. - Coverage can fall while more bins are hit — 41/256 to 44/300 is three more bins and 1.35 points less — so a percentage delta alone is ambiguous and the row must carry bins hit and total.
- A bin hit exactly once is proven reachable and is the cheapest hole to close; a bin never hit needs a human to decide whether it is reachable at all.
- Three uncovered bins and four worth chasing is not a contradiction — the two numbers count different things.
- A waiver carries a reason, an owner and a date, and the tracker applies waivers rather than deciding them.
- Pin a property, not a number: the regrouping test survives a change of
fixture that an assertion on
22.14would not.
The next chapter turns from measuring a design to generating part of it, where the tool's output is checked into the repository and the hard problem is proving nobody edited it by hand.
Where this fits
Part of the Python curriculum.