Python Part II · Module 29 · Coverage Automation
Which Coverage Holes Are Worth Chasing
Chapter 29.3 produced a queue ordered by uncovered bins and Chapter 29.4 counted them carefully, and both treated an uncovered bin as a unit of work. It is not. A bin reading zero may be waiting for a test somebody has not written, or it may be a combination the design makes impossible, or it may belong to a feature this chip does not have — and those three demand a test, a declaration, and nothing at all, respectively. Handing all of them to a verification engineer as a to-do list is how teams end up chasing a number that arithmetic forbids them from reaching.
Advanced20 min readPythonPart IIAutomationCoverageTriage
Part II · Module 29 · Chapter 29.5 · Which Coverage Holes Are Worth Chasing
1. Three Kinds of Zero
A bin at zero hits looks the same in every report. It has three completely different causes:
KINDS = [
("unreachable", "the design cannot produce it", "declare it illegal"),
("not fitted", "the feature is absent from this configuration", "exclude it, with a reason"),
("untargeted", "no test aims at it yet", "write the test — this is the work"),
]
print(f"{'kind':<13} {'why it reads zero':<44} what to do")
for kind, why, what in KINDS:
print(f"{kind:<13} {why:<44} {what}")
print()
print("kinds that are verification work:", sum(1 for k in KINDS if "write" in k[2]))
print("kinds that a report can distinguish on its own:", 0)kind why it reads zero what to do
unreachable the design cannot produce it declare it illegal
not fitted the feature is absent from this configuration exclude it, with a reason
untargeted no test aims at it yet write the test — this is the work
kinds that are verification work: 1
kinds that a report can distinguish on its own: 0The second number is the problem this chapter solves. A coverage report cannot tell you which kind a zero is, because the information is not in the design's behaviour — it is in somebody's knowledge of the design's intent. Triage is the act of writing that knowledge down.
2. The Annotation
holes = [b for b in bins if b.hits == 0 and not b.excluded]A list comprehension with two conditions, which produces a two-way split where the purpose names three kinds:
from dataclasses import dataclass
@dataclass(frozen=True)
class Bin:
name: str
hits: int
excluded: bool
bins = [
Bin("cg.mode_x_size.idle_b4", 0, False),
Bin("cg.err.ecc_double", 0, True),
Bin("cg.err.parity", 0, False),
Bin("cg.burst.len8", 14, False),
]
holes = [b for b in bins if b.hits == 0 and not b.excluded]
print("bins :", len(bins))
print("bins at zero :", sum(1 for b in bins if b.hits == 0))
print("annotation's holes:", [b.name for b in holes])
print()
print("categories the filter produces:", 2)
print("categories the purpose names :", 3)bins : 4
bins at zero : 3
annotation's holes: ['cg.mode_x_size.idle_b4', 'cg.err.parity']
categories the filter produces: 2
categories the purpose names : 3Two names come back, and only one of them is work. cg.mode_x_size.idle_b4 is the cross of an idle mode with a four-beat burst, which the design cannot produce — the filter cannot know that, so it hands a verification engineer a task that no test can complete. The next two sections are about how expensive that is.
3. An Unreachable Cross Caps the Percentage
A cross-coverage point multiplies two bin sets, and the product contains combinations the design forbids:
from itertools import product
MODES = ["idle", "burst", "single"]
SIZES = ["b1", "b4", "b8"]
cross = list(product(MODES, SIZES))
illegal = [(m, s) for m, s in cross if m == "idle" and s != "b1"]
reachable = [c for c in cross if c not in illegal]
print(f"{'mode':<8} {'size':<5} legal?")
for m, s in cross:
print(f"{m:<8} {s:<5} {'no — idle issues no burst' if (m, s) in illegal else 'yes'}")
print()
print(f"cross bins : {len(cross)}")
print(f"illegal : {len(illegal)}")
print(f"reachable : {len(reachable)}")
print(f"ceiling with illegal bins still counted: {100.0*len(reachable)/len(cross):.2f}%")mode size legal?
idle b1 yes
idle b4 no — idle issues no burst
idle b8 no — idle issues no burst
burst b1 yes
burst b4 yes
burst b8 yes
single b1 yes
single b4 yes
single b8 yes
cross bins : 9
illegal : 2
reachable : 7
ceiling with illegal bins still counted: 77.78%77.78% is this cross's maximum, and a perfect test suite reports exactly that. A gate at 90% fails it forever, and every sprint somebody re-asks why the number will not move.
Now scale it, because crosses are where coverage denominators get large:
print(f"{'cross':<10} {'bins':>6} {'legal':>6} {'ceiling':>9} chasing the gap means")
for n in (4, 8, 16):
total = n * n
legal = n + n - 1 # the diagonal plus one full row
ceiling = 100.0 * legal / total
print(f"{n}x{n:<7} {total:>6} {legal:>6} {ceiling:>8.2f}% "
f"covering {total - legal} impossible bins")cross bins legal ceiling chasing the gap means
4x4 16 7 43.75% covering 9 impossible bins
8x8 64 15 23.44% covering 49 impossible bins
16x16 256 31 12.11% covering 225 impossible binsA 16×16 cross whose legal combinations are a diagonal plus one row has a ceiling of 12.11%, and the other 225 bins cannot be hit by any stimulus whatsoever. A dashboard reporting "12.11% cross coverage" beside a 90% threshold is reporting a design that is fully verified as catastrophically broken.
The fix is not in the parser or the gate. It is illegal_bins in the covergroup — declaring the combinations the design forbids so the tool removes them from the denominator:
from itertools import product
MODES = ["idle", "burst", "single"]
SIZES = ["b1", "b4", "b8"]
ILLEGAL = {("idle", "b4"), ("idle", "b8")}
cross = list(product(MODES, SIZES))
declared = [c for c in cross if c not in ILLEGAL]
hit = {("idle", "b1"), ("burst", "b1"), ("burst", "b4"), ("burst", "b8"),
("single", "b1"), ("single", "b4"), ("single", "b8")}
print(f"undeclared: {len(hit)}/{len(cross)} = {100.0*len(hit)/len(cross):.2f}%")
print(f"declared : {len(hit)}/{len(declared)} = {100.0*len(hit)/len(declared):.2f}%")
print()
print("tests added between those two lines:", 0)
print("the second number is the honest one, and it is 100%.")undeclared: 7/9 = 77.78%
declared : 7/7 = 100.00%
tests added between those two lines: 0
the second number is the honest one, and it is 100%.The same stimulus reports 77.78% or 100.00% depending on whether somebody wrote down which combinations are illegal. That is not the exclusion abuse of Chapter 29.4 Section 6 — it is the opposite, and telling the two apart is the whole point of the next section.
4. An Exclusion Without a Reason Is Not an Exclusion
not b.excluded in the annotation treats the flag as truth. Here is what that costs:
from dataclasses import dataclass
@dataclass(frozen=True)
class Bin:
name: str
hits: int
excluded: bool
reason: str
bins = [
Bin("cg.err.ecc_double", 0, True, "ECC not fitted in this SKU"),
Bin("cg.err.crc_burst", 0, True, ""),
Bin("cg.err.parity", 0, False, ""),
]
print(f"{'bin':<20} {'hits':>5} {'excl':>6} reason")
for b in bins:
print(f"{b.name:<20} {b.hits:>5} {str(b.excluded):>6} {b.reason or '— none —'}")
annotation = [b.name for b in bins if b.hits == 0 and not b.excluded]
unjustified = [b.name for b in bins if b.excluded and not b.reason]
print()
print("the annotation reports as holes :", annotation)
print("exclusions carrying no reason :", unjustified)
print("bins removed from view with no justification:", len(unjustified))bin hits excl reason
cg.err.ecc_double 0 True ECC not fitted in this SKU
cg.err.crc_burst 0 True — none —
cg.err.parity 0 False — none —
the annotation reports as holes : ['cg.err.parity']
exclusions carrying no reason : ['cg.err.crc_burst']
bins removed from view with no justification: 1cg.err.crc_burst has disappeared from the queue, from the denominator and from the conversation, and nothing anywhere records why. It may be a legitimate exclusion whose author simply did not type a reason. It may be a bin somebody excluded on a Friday to make a gate pass. From the data, those two are the same event, and that is a property of the flow rather than of anybody's integrity.
The structural answer is to make an unjustified exclusion impossible to express:
from dataclasses import dataclass
@dataclass(frozen=True)
class Exclusion:
bin_name: str
reason: str
author: str
def __post_init__(self):
if not self.reason.strip():
raise ValueError(f"exclusion of {self.bin_name!r} needs a reason")
if not self.author.strip():
raise ValueError(f"exclusion of {self.bin_name!r} needs an author")
ok = Exclusion("cg.err.ecc_double", "ECC not fitted in this SKU", "a.patel")
print("accepted:", ok.bin_name, "—", ok.reason, f"({ok.author})")
for bad in (("cg.err.crc_burst", "", "a.patel"),
("cg.err.crc_burst", " ", "a.patel"),
("cg.err.crc_burst", "not needed", "")):
try:
Exclusion(*bad)
print("accepted (should not be):", bad)
except ValueError as exc:
print("refused:", exc)accepted: cg.err.ecc_double — ECC not fitted in this SKU (a.patel)
refused: exclusion of 'cg.err.crc_burst' needs a reason
refused: exclusion of 'cg.err.crc_burst' needs a reason
refused: exclusion of 'cg.err.crc_burst' needs an authorA whitespace-only reason is refused along with an empty one, which matters because if not self.reason alone would accept " ". This is a mechanism rather than a convention — the difference between a rule somebody can forget and a rule nothing can bypass. It is the same device as Chapter 21.7's configuration validation and the same reason Chapter 29.2 Section 7 put a reason beside every threshold.
5. The Bin No Test Targets
With unreachable and unfitted accounted for, what remains is the only category that is work:
from dataclasses import dataclass
from collections import Counter
@dataclass(frozen=True)
class Bin:
name: str
hits: int
kind: str
note: str
BINS = [
Bin("cg.mode_x_size.idle_b4", 0, "unreachable", "idle issues no burst"),
Bin("cg.mode_x_size.idle_b8", 0, "unreachable", "idle issues no burst"),
Bin("cg.err.ecc_double", 0, "not_fitted", "ECC absent in this SKU"),
Bin("cg.err.parity", 0, "untargeted", ""),
Bin("cg.burst.len16", 0, "untargeted", ""),
Bin("cg.burst.len8", 14, "covered", ""),
]
counts = Counter(b.kind for b in BINS)
zeros = [b for b in BINS if b.hits == 0]
work = [b for b in zeros if b.kind == "untargeted"]
print(f"{'bin':<24} {'hits':>5} {'kind':<12} note")
for b in BINS:
print(f"{b.name:<24} {b.hits:>5} {b.kind:<12} {b.note}")
print()
print("bins :", len(BINS))
print("bins at zero :", len(zeros))
print("of those, real work :", len(work), [b.name for b in work])
print("by kind :", dict(sorted(counts.items())))
print()
print(f"the annotation's queue length: {len(zeros) - 1}")
print(f"the real queue length : {len(work)}")bin hits kind note
cg.mode_x_size.idle_b4 0 unreachable idle issues no burst
cg.mode_x_size.idle_b8 0 unreachable idle issues no burst
cg.err.ecc_double 0 not_fitted ECC absent in this SKU
cg.err.parity 0 untargeted
cg.burst.len16 0 untargeted
cg.burst.len8 14 covered
bins : 6
bins at zero : 5
of those, real work : 2 ['cg.err.parity', 'cg.burst.len16']
by kind : {'covered': 1, 'not_fitted': 1, 'unreachable': 2, 'untargeted': 2}
the annotation's queue length: 4
the real queue length : 2Five bins at zero, two of them work. The annotation's queue has four items — it correctly drops the excluded one and keeps the two unreachable crosses — so it is twice as long as the real queue, and the two extra items cannot be completed by any test.
That ratio is what makes triage worth doing rather than a bureaucratic nicety. A verification engineer handed four tasks of which two are impossible will complete two, report blocked on two, and lose confidence in the queue. The second time it happens they stop reading it.
6. Classifying, Not Guessing
The kind field in Section 5 arrived pre-populated, which is the one thing a real flow cannot do. Classification has three sources and exactly one of them is automatic:
SOURCES = [
("illegal_bins in the covergroup", "automatic", "the tool removes them from the denominator"),
("an exclusion file with reasons", "reviewed", "written once per configuration — §4"),
("everything else", "untargeted by default", "the safe default: assume it is work"),
]
print(f"{'source':<32} {'how':<22} effect")
for src, how, eff in SOURCES:
print(f"{src:<32} {how:<22} {eff}")
print()
print("the default for an unclassified zero must be:", "untargeted")
print("because assuming unreachable would:", "silently shrink the queue")source how effect
illegal_bins in the covergroup automatic the tool removes them from the denominator
an exclusion file with reasons reviewed written once per configuration — §4
everything else untargeted by default the safe default: assume it is work
the default for an unclassified zero must be: untargeted
because assuming unreachable would: silently shrink the queueThe direction of the default is the whole design decision. An unclassified zero defaults to work, which produces a queue that is too long and visibly so; defaulting to unreachable would produce a queue that is too short and invisibly so. Chapter 29.1 Section 5 made the same argument about a parser that drops rows: prefer the error somebody will notice.
EXCLUSIONS = {
"cg.err.ecc_double": ("ECC not fitted in this SKU", "a.patel"),
}
ILLEGAL = {"cg.mode_x_size.idle_b4", "cg.mode_x_size.idle_b8"}
REPORT = {
"cg.mode_x_size.idle_b4": 0,
"cg.mode_x_size.idle_b8": 0,
"cg.err.ecc_double": 0,
"cg.err.parity": 0,
"cg.burst.len16": 0,
"cg.burst.len8": 14,
}
def classify(name, hits):
if hits > 0:
return "covered"
if name in ILLEGAL:
return "unreachable"
if name in EXCLUSIONS:
return "not_fitted"
return "untargeted"
for name, hits in REPORT.items():
kind = classify(name, hits)
why = EXCLUSIONS.get(name, ("", ""))[0] or ("declared illegal" if kind == "unreachable" else "")
print(f"{name:<24} {hits:>3} {kind:<12} {why}")
work = [n for n, h in REPORT.items() if classify(n, h) == "untargeted"]
print()
print("queue:", work)
print("queue length:", len(work))cg.mode_x_size.idle_b4 0 unreachable declared illegal
cg.mode_x_size.idle_b8 0 unreachable declared illegal
cg.err.ecc_double 0 not_fitted ECC not fitted in this SKU
cg.err.parity 0 untargeted
cg.burst.len16 0 untargeted
cg.burst.len8 14 covered
queue: ['cg.err.parity', 'cg.burst.len16']
queue length: 2Two inputs a human maintains, one function, and a queue of two. Note that classify checks hits > 0 first: a bin that is declared illegal and has hits is a contradiction worth knowing about — the design did something the specification says it cannot — and ordering the checks this way reports it as covered rather than hiding it under the declaration. Chapter 29.6 Section 5 turns that contradiction into an explicit check.
7. Ranking the Queue
Two bins is a short queue. A real one has hundreds, and a hole's size is not its value:
HOLES = [
("cg.err.parity", 1, "error handling", 3.0, "ships enabled; a hole here reaches customers"),
("cg.burst.len16", 1, "datapath", 2.0, "primary traffic shape"),
("cg.dbg.trace_lvl", 40, "debug", 0.5, "internal use only, never enabled in production"),
("cg.cfg.legacy_mode", 12, "legacy", 0.5, "frozen RTL, no new features"),
]
print(f"{'hole':<20} {'bins':>5} {'weight':>7} {'score':>7} why")
for name, bins, area, weight, why in sorted(
HOLES, key=lambda h: -(h[1] * h[3])):
print(f"{name:<20} {bins:>5} {weight:>7.1f} {bins*weight:>7.1f} {why}")
print()
print("ranked by bin count :",
[h[0] for h in sorted(HOLES, key=lambda h: -h[1])])
print("ranked by weighted :",
[h[0] for h in sorted(HOLES, key=lambda h: -(h[1] * h[3]))])hole bins weight score why
cg.dbg.trace_lvl 40 0.5 20.0 internal use only, never enabled in production
cg.cfg.legacy_mode 12 0.5 6.0 frozen RTL, no new features
cg.err.parity 1 3.0 3.0 ships enabled; a hole here reaches customers
cg.burst.len16 1 2.0 2.0 primary traffic shape
ranked by bin count : ['cg.dbg.trace_lvl', 'cg.cfg.legacy_mode', 'cg.err.parity', 'cg.burst.len16']
ranked by weighted : ['cg.dbg.trace_lvl', 'cg.cfg.legacy_mode', 'cg.err.parity', 'cg.burst.len16']Both orderings are identical, and that is a finding rather than a tidy result: a 3× risk weight does not overcome a 40× difference in bin count. Weighting a hole's size by its importance still ranks forty debug bins above one parity bin, because the arithmetic is dominated by the count.
If the intent is that a customer-visible hole outranks a debug hole regardless of size, the ranking has to say so structurally rather than numerically:
HOLES = [
("cg.err.parity", 1, "error handling", "ships enabled"),
("cg.burst.len16", 1, "datapath", "primary traffic shape"),
("cg.dbg.trace_lvl", 40, "debug", "never enabled in production"),
("cg.cfg.legacy_mode", 12, "legacy", "frozen RTL"),
]
TIERS = {"error handling": 0, "datapath": 0, "legacy": 1, "debug": 2}
TIER_REASON = {
0: "in the shipping configuration",
1: "present but frozen",
2: "not enabled in production",
}
ordered = sorted(HOLES, key=lambda h: (TIERS[h[2]], -h[1]))
print(f"{'tier':>4} {'hole':<20} {'bins':>5} {'area':<15} tier means")
for name, bins, area, why in ordered:
t = TIERS[area]
print(f"{t:>4} {name:<20} {bins:>5} {area:<15} {TIER_REASON[t]}")
print()
print("tier 0 bins to close:", sum(b for _, b, a, _ in HOLES if TIERS[a] == 0))
print("tier 2 bins ignored :", sum(b for _, b, a, _ in HOLES if TIERS[a] == 2))
print("every tier carries a recorded meaning:", len(TIER_REASON) == len(set(TIERS.values())))tier hole bins area tier means
0 cg.err.parity 1 error handling in the shipping configuration
0 cg.burst.len16 1 datapath in the shipping configuration
1 cg.cfg.legacy_mode 12 legacy present but frozen
2 cg.dbg.trace_lvl 40 debug not enabled in production
tier 0 bins to close: 2
tier 2 bins ignored : 40
every tier carries a recorded meaning: TrueSorting by tier first and size second puts two bins at the top of the queue and forty at the bottom, which is the ordering a release actually wants. A tier is a policy exactly like Chapter 29.2's thresholds and Chapter 29.3's weights, so it carries its meaning with it — and the last line is the check that no tier exists without one.
8. How Long Has It Been Open?
Section 7 ranked by tier and then by size. There is a third signal, and on a long project it discriminates better than either:
HOLES = [
("cg.err.parity", 1, "error handling", 94),
("cg.burst.len16", 1, "datapath", 2),
("cg.cfg.wide_mode", 6, "datapath", 61),
("cg.err.timeout", 3, "error handling", 1),
]
print(f"{'hole':<20} {'bins':>5} {'area':<15} {'days':>5} reading")
for name, bins, area, age in sorted(HOLES, key=lambda h: -h[3]):
if age > 60:
reading = "open for months — nobody is working on it"
elif age > 30:
reading = "stale"
else:
reading = "recent; may already be somebody's task"
print(f"{name:<20} {bins:>5} {area:<15} {age:>5} {reading}")
print()
old = [h for h in HOLES if h[3] > 60]
new = [h for h in HOLES if h[3] <= 7]
print("holes open longer than 60 days:", [h[0] for h in old])
print("holes younger than a week :", [h[0] for h in new])hole bins area days reading
cg.err.parity 1 error handling 94 open for months — nobody is working on it
cg.cfg.wide_mode 6 datapath 61 open for months — nobody is working on it
cg.burst.len16 1 datapath 2 recent; may already be somebody's task
cg.err.timeout 3 error handling 1 recent; may already be somebody's task
holes open longer than 60 days: ['cg.err.parity', 'cg.cfg.wide_mode']
holes younger than a week : ['cg.burst.len16', 'cg.err.timeout']Age splits the queue into two genuinely different kinds of problem, and the split is not about size at all.
A hole two days old is almost certainly fine. Somebody merged a feature on Tuesday, the covergroup grew, and a test is being written this week. It will close on its own, and putting it at the top of a queue wastes a conversation.
A hole ninety-four days old is a different object entirely. It has survived three months of a team that was looking at the coverage report, which means one of a small number of things is true, and every one of them is actionable in a way that writing a test is not:
EXPLANATIONS = [
("it is harder than it looks", "needs a plan, not a task"),
("it is unreachable and undeclared", "29.5 §3 — declare it and the queue shrinks"),
("it belongs to a feature nobody owns", "needs an owner before it needs a test"),
("everybody assumes somebody else has it", "needs naming in a meeting"),
]
print("why a hole survives 94 days of a team that reads the report:")
for why, what in EXPLANATIONS:
print(f" {why:<40} -> {what}")
print()
print("explanations resolved by writing a test:", 0)
print("explanations resolved by a decision :", len(EXPLANATIONS))why a hole survives 94 days of a team that reads the report:
it is harder than it looks -> needs a plan, not a task
it is unreachable and undeclared -> 29.5 §3 — declare it and the queue shrinks
it belongs to a feature nobody owns -> needs an owner before it needs a test
everybody assumes somebody else has it -> needs naming in a meeting
explanations resolved by writing a test: 0
explanations resolved by a decision : 4Not one of the four is fixed by writing a test, which is why age is the field that changes what you do rather than merely the order you do it in. An old hole is a question for a person; a new hole is a task for a queue.
Age costs one field and one comparison, and it needs the history Chapter 29.4 Section 7 argued for anyway:
HISTORY = {
"cg.err.parity": [0, 0, 0, 0, 0, 0],
"cg.burst.len16": [None, None, None, None, 0, 0],
"cg.err.timeout": [4, 6, 2, 0, 0, 0],
}
print(f"{'hole':<20} {'runs seen':>10} {'runs at zero':>13} status")
for name, series in sorted(HISTORY.items()):
seen = [v for v in series if v is not None]
trailing = 0
for v in reversed(seen):
if v == 0:
trailing += 1
else:
break
if len(seen) < len(series):
status = "new bin — did not exist in earlier runs"
elif trailing == len(seen):
status = "never once hit"
else:
status = f"was hit, then stopped {trailing} runs ago"
print(f"{name:<20} {len(seen):>10} {trailing:>13} {status}")hole runs seen runs at zero status
cg.burst.len16 2 2 new bin — did not exist in earlier runs
cg.err.parity 6 6 never once hit
cg.err.timeout 6 3 was hit, then stopped 3 runs agoThree bins, all at zero tonight, and three different stories. cg.err.timeout is the one to look at first and it is not visible in any single run: it was being hit and then stopped, which is a regression in the Chapter 29.6 sense rather than a hole at all. A bin that was covered and now is not means a test was removed, started failing before it reached that point, or the RTL path disappeared.
The None entries matter too. A bin that did not exist in earlier runs is not a bin that was uncovered then — Chapter 29.4 Section 5's distinction between an absent bin and an uncovered one, arriving in the triage.
9. The Triage, Assembled
from dataclasses import dataclass
from collections import Counter
@dataclass(frozen=True)
class Exclusion:
reason: str
author: str
def __post_init__(self):
if not self.reason.strip() or not self.author.strip():
raise ValueError("an exclusion needs a reason and an author")
ILLEGAL = {"cg.mx.idle_b4", "cg.mx.idle_b8"}
EXCLUSIONS = {"cg.err.ecc_double": Exclusion("ECC not fitted in this SKU", "a.patel")}
AREA = {
"cg.err.parity": "error handling", "cg.burst.len16": "datapath",
"cg.dbg.trace_lvl": "debug", "cg.mx.idle_b4": "datapath",
"cg.mx.idle_b8": "datapath", "cg.err.ecc_double": "error handling",
"cg.burst.len8": "datapath",
}
TIERS = {"error handling": 0, "datapath": 0, "legacy": 1, "debug": 2}
REPORT = {
"cg.mx.idle_b4": 0, "cg.mx.idle_b8": 0, "cg.err.ecc_double": 0,
"cg.err.parity": 0, "cg.burst.len16": 0, "cg.dbg.trace_lvl": 0,
"cg.burst.len8": 14,
}
def triage(report):
out, contradictions = [], []
for name, hits in report.items():
if hits > 0:
kind = "covered"
if name in ILLEGAL:
contradictions.append(name)
elif name in ILLEGAL:
kind = "unreachable"
elif name in EXCLUSIONS:
kind = "not_fitted"
else:
kind = "untargeted"
out.append((name, hits, kind, AREA[name]))
return out, contradictions
rows, contradictions = triage(REPORT)
counts = Counter(k for _, _, k, _ in rows)
print("by kind:", dict(sorted(counts.items())))
print("bins declared illegal that were hit:", contradictions or "none")
print()
queue = sorted((r for r in rows if r[2] == "untargeted"),
key=lambda r: (TIERS[r[3]], r[0]))
print(f"{'tier':>4} {'bin':<20} area")
for name, hits, kind, area in queue:
print(f"{TIERS[area]:>4} {name:<20} {area}")
print()
print("bins at zero :", sum(1 for _, h, _, _ in rows if h == 0))
print("queue length :", len(queue))
print("dropped as impossible:", counts["unreachable"])
print("dropped as not fitted:", counts["not_fitted"])
print("tier 0 in the queue :", sum(1 for r in queue if TIERS[r[3]] == 0))by kind: {'covered': 1, 'not_fitted': 1, 'unreachable': 2, 'untargeted': 3}
bins declared illegal that were hit: none
tier bin area
0 cg.burst.len16 datapath
0 cg.err.parity error handling
2 cg.dbg.trace_lvl debug
bins at zero : 6
queue length : 3
dropped as impossible: 2
dropped as not fitted: 1
tier 0 in the queue : 2Six bins at zero become a queue of three, of which two are in the shipping configuration and one is debug-only at the bottom. The annotation applied to the same report returns five names in no particular order, two of which no test can ever satisfy.
Nothing here is sophisticated. The entire mechanism is two human-maintained files, one function with its checks in a deliberate order, and a sort key with two terms — and the output is a list somebody can work through and finish.
10. Predict, Run, Observe
from itertools import product
ILLEGAL = {"cg.a"}
EXCL = {"cg.b"}
REPORT = {"cg.a": 0, "cg.b": 0, "cg.c": 0, "cg.d": 5}
# A
print("A:", [n for n, h in REPORT.items() if h == 0 and n not in EXCL])
# B
print("B:", [n for n, h in REPORT.items()
if h == 0 and n not in EXCL and n not in ILLEGAL])
# C
cross = list(product(range(16), range(16)))
legal = 16 + 16 - 1
print("C:", f"{100.0*legal/len(cross):.2f}")
# D
print("D:", f"{100.0*900/900:.2f}", f"{100.0*7/7:.2f}")The verified output:
A: ['cg.a', 'cg.c']
B: ['cg.c']
C: 12.11
D: 100.00 100.00A — the annotation's answer: two holes, one of which is unreachable.
B — one more condition, one real hole. The difference between a queue somebody can finish and one they cannot.
C — 12.11%. A fully verified 16×16 cross with an undeclared legality rule.
D — two identical numbers from opposite acts: excluding every uncovered bin (Chapter 29.4 Section 6) and declaring two combinations illegal (Section 3). The arithmetic cannot tell them apart; only the recorded reason can.
11. Exercises
Exercise 1 — three kinds. Take ten zero-hit bins from your own report and classify each by hand. Report the split.
Exercise 2 — the annotation's queue. Run the annotation's filter on your report and count how many of its results no test could satisfy.
Exercise 3 — a cross ceiling. Pick a real cross in your covergroups, count the illegal combinations, and compute its ceiling.
Exercise 4 — scale it. For your largest cross, give the bin count, the legal count and the ceiling.
Exercise 5 — declare them. Write the illegal_bins clause for one cross and state the percentage before and after.
Exercise 6 — illegal versus excluded. Explain in three sentences what each one claims and which a reviewer can check.
Exercise 7 — unjustified exclusions. Count the exclusions in your flow that carry no reason. Report the number without attributing motive.
Exercise 8 — refuse the blank. Implement the Exclusion guard and show it refusing a whitespace-only reason.
Exercise 9 — the default direction. Say what your flow currently does with an unclassified zero, and which error that choice prefers.
Exercise 10 — a contradiction. Find or construct a bin that is declared illegal and has hits, and say what it means about the specification.
Exercise 11 — size does not rank. Build a weighted ranking of your holes and check whether any weight overcomes the largest bin count.
Exercise 12 — tiers. Define three tiers for your design with a recorded meaning each, and re-sort your queue.
Exercise 13 — queue length. Report your queue length before and after triage, and the share that was impossible.
Exercise 14 — the second time. Describe what happens to a team's use of a coverage queue after they are handed impossible items twice.
Exercise 15 — two files. Say who owns your illegal_bins and who owns your exclusion file, and when each is reviewed.
Exercise 16 — age your holes. Add a first-seen date to your bins and report the oldest five.
Exercise 17 — two days versus ninety. Explain in two sentences why those two need different responses rather than different priorities.
Exercise 18 — the four explanations. For your oldest hole, say which of the four explanations applies and what decision it needs.
Exercise 19 — not a task. Say why none of the four explanations for a 94-day hole is resolved by writing a test.
Exercise 20 — trailing zeros. Implement the trailing-zero count over bin history and classify three of your own bins with it.
Exercise 21 — was hit, then stopped. Find a bin in your history that was covered and is not, and say what it means.
Exercise 22 — absent is not uncovered. Explain why a None in a bin's history is different from a zero, and what each implies.
Exercise 23 — a new bin at zero. Decide whether a bin that appeared in tonight's run belongs in tonight's queue, and justify it.
Exercise 24 — age against tier. Re-sort your queue by tier, then age, then size, and say which ordering you would publish.
12. Summary
- A zero has three causes — unreachable, not fitted, untargeted — and a coverage report cannot distinguish them, because the information is about intent rather than behaviour.
- The annotation produces a two-way split where the purpose names three kinds, so it hands impossible items to an engineer as work.
- An undeclared illegal combination caps the percentage: a 3×3 cross with two illegal pairs tops out at 77.78%, and a 16×16 cross with 31 legal pairs at 12.11%.
- No stimulus moves those numbers. A gate above the ceiling fails a fully verified design forever.
illegal_binsand an exclusion both shrink the denominator, and the difference is what is claimed: the specification forbids this, versus do not count this against me.- An exclusion with no reason is indistinguishable from a hidden bin. Refuse it as a mechanism — and refuse a whitespace-only reason too.
- Of five bins at zero, two were work. The annotation's queue was twice as long and its extra items could not be completed.
- Default an unclassified zero to untargeted, which makes the queue too long visibly rather than too short invisibly.
- Check
hits > 0before the legality rule, so a bin declared illegal that was hit is reported as a contradiction instead of hidden. - A 3× risk weight does not overcome a 40× bin count — if a customer-visible hole must outrank a debug hole, use tiers, not weights.
- Every tier, weight and threshold carries a recorded meaning, which is the same rule as Chapter 29.2's thresholds and Chapter 29.3's block weights.
- Age discriminates better than size on a long project. A two-day hole will probably close on its own; a ninety-four-day hole will not.
- None of the four reasons a hole survives three months is fixed by writing a test — each needs a plan, a declaration, an owner or a name in a meeting.
- An old hole is a question for a person; a new hole is a task for a queue.
- A bin that was hit and then stopped is a regression, not a hole, and it is invisible in any single run.
- A
Nonein a bin's history means the bin did not exist, which is not the same as having been uncovered — the same distinction as a renamed bin.
The last chapter of the module turns all of this into the evidence a signoff review needs, and asks what size of drop deserves to be called a regression.
Where this fits
Part of the Python curriculum.