Python Part II · Module 29 · Coverage Automation
Coverage Regressions and Signoff Evidence
This module has spent five chapters making coverage numbers trustworthy, and a signoff review asks two things of them: is this enough, and did it get worse. The second question needs a threshold, and the obvious move is to copy Chapter 28.6 and measure the flow's noise. That works, and it produces an answer that depends entirely on a decision nobody thinks of as part of the gate — whether the regression runs with pinned seeds or random ones. With one policy the right band is zero. With the other it is more than eight times the number most flows use.
Advanced21 min readPythonPart IIAutomationCoverageSignoff
Part II · Module 29 · Chapter 29.6 · Coverage Regressions and Signoff Evidence
1. A Drop Is Not Automatically a Regression
Chapter 29.4 established that four different events move a coverage number. Three of them can move it down without anything having got worse:
DROPS = [
("new coverable code merged", "no", "29.4 §4 — the denominator grew"),
("covergroups restructured", "no", "29.4 §5 — a bin was renamed"),
("different seeds this run", "no", "§5 — the spread is 4.25 points"),
("a test was removed or is failing", "yes", "this is the regression"),
("the RTL changed and a path is gone", "yes", "this is the regression"),
]
print(f"{'cause of a lower number':<34} {'regression?':<12} where it is handled")
for cause, is_regr, where in DROPS:
print(f"{cause:<34} {is_regr:<12} {where}")
print()
print("causes of a drop :", len(DROPS))
print("causes that are regressions:", sum(1 for d in DROPS if d[1] == "yes"))
print("so a bare drop is evidence of:", "nothing in particular")cause of a lower number regression? where it is handled
new coverable code merged no 29.4 §4 — the denominator grew
covergroups restructured no 29.4 §5 — a bin was renamed
different seeds this run no §5 — the spread is 4.25 points
a test was removed or is failing yes this is the regression
the RTL changed and a path is gone yes this is the regression
causes of a drop : 5
causes that are regressions: 2
so a bare drop is evidence of: nothing in particularTwo of five. The first two causes are handled by Chapter 29.4's decomposition, which means the comparison this chapter gates on should be run on the numerator effect, not on the headline. The third cause is what the threshold is for, and it is the one nobody measures.
2. The Annotation
if today["functional"] < last["functional"] - 0.5:
fail("coverage regression")The comment is right, the structure is right, and the number came from nowhere:
last = {"functional": 95.80}
today = {"functional": 95.10}
BAND = 0.5
def fail(msg):
return f"FAIL: {msg}"
verdict = (fail("coverage regression")
if today["functional"] < last["functional"] - BAND else "PASS")
print(f"last : {last['functional']:.2f}%")
print(f"today: {today['functional']:.2f}%")
print(f"drop : {last['functional'] - today['functional']:.2f} points")
print(f"band : {BAND}")
print(f"verdict: {verdict}")
print()
print("the question this chapter asks:", "where did 0.5 come from?")last : 95.80%
today: 95.10%
drop : 0.70 points
band : 0.5
verdict: FAIL: coverage regression
the question this chapter asks: where did 0.5 come from?Chapter 28.6 asked the same question of a timing threshold and answered it by measuring the flow's run-to-run spread, concluding that the annotation's 0.050 was safe and three times larger than it needed to be. The same method applied here reaches the opposite conclusion, twice over, and for a reason specific to coverage.
3. What Makes Coverage Vary Between Runs
Timing noise comes from the tool — threading, effort budgets, machine load. Coverage noise has exactly one dominant source:
SOURCES = [
("the seeds", "yes", "different random stimulus reaches different bins"),
("the testlist", "no", "fixed between runs by definition"),
("tool threading", "no", "coverage is collected, not optimised"),
("machine load", "no", "a slower run collects the same bins"),
]
print(f"{'source':<18} {'varies?':<9} why")
for src, varies, why in SOURCES:
print(f"{src:<18} {varies:<9} {why}")
print()
print("sources that vary:", [s[0] for s in SOURCES if s[1] == "yes"])
print("so the noise is a property of:", "the seed policy, which is a choice")source varies? why
the seeds yes different random stimulus reaches different bins
the testlist no fixed between runs by definition
tool threading no coverage is collected, not optimised
machine load no a slower run collects the same bins
sources that vary: ['the seeds']
so the noise is a property of: the seed policy, which is a choiceThe noise is a property of a choice, which is what separates this from Chapter 28.6. A place-and-route flow is non-deterministic whether you like it or not; a coverage regression is non-deterministic only if you chose random seeds. That makes the threshold question downstream of a policy question, and the next two sections measure both branches.
4. Pinned Seeds Make Coverage Deterministic
Take one branch of the policy question first:
import random
BINS = 400
DRAWS_PER_TEST = 60
def coverage(seeds):
"""Each seed drives one test; the test reaches DRAWS_PER_TEST bins."""
hit = set()
for s in seeds:
rnd = random.Random(s)
for _ in range(DRAWS_PER_TEST):
hit.add(rnd.randrange(BINS))
return 100.0 * len(hit) / BINS
PINNED = [11, 22, 33, 44]
values = [coverage(PINNED) for _ in range(5)]
print("pinned seed set:", PINNED)
print("five evaluations:", [f"{v:.2f}" for v in values])
print("distinct values :", len(set(values)))
print(f"spread : {max(values) - min(values):.2f} points")
print()
print("the right regression band for a pinned testlist:", 0.0)pinned seed set: [11, 22, 33, 44]
five evaluations: ['45.75', '45.75', '45.75', '45.75', '45.75']
distinct values : 1
spread : 0.00 points
the right regression band for a pinned testlist: 0.0One distinct value out of five runs. With the seeds pinned, the same testlist against the same design reaches the same bins every time, so the correct band is zero — any drop whatsoever is a real change, caused by the RTL, the testbench or the testlist.
Against this policy the annotation's 0.5 is too loose: it silently tolerates a real regression of up to half a point. On a design with 10,000 bins that is fifty bins of genuine coverage loss, passing without comment.
TOTAL = 10000
BAND = 0.5
print(f"bins in the design : {TOTAL}")
print(f"band : {BAND} points")
print(f"bins the band tolerates : {int(TOTAL * BAND / 100)}")
print(f"with pinned seeds that is : {int(TOTAL * BAND / 100)} bins of real loss, unreported")bins in the design : 10000
band : 0.5 points
bins the band tolerates : 50
with pinned seeds that is : 50 bins of real loss, unreported5. Random Seeds Span Four Points
Now the other branch. Same testlist, same design, same number of tests — only the seeds differ:
import random
import statistics
BINS = 400
DRAWS_PER_TEST = 60
def coverage(seeds):
hit = set()
for s in seeds:
rnd = random.Random(s)
for _ in range(DRAWS_PER_TEST):
hit.add(rnd.randrange(BINS))
return 100.0 * len(hit) / BINS
SETS = [[a, a + 1, a + 2, a + 3] for a in range(100, 900, 100)]
values = [coverage(s) for s in SETS]
for seeds, v in zip(SETS, values):
print(f" seeds {str(seeds):<22} -> {v:>6.2f}%")
print()
print(f"runs : {len(values)}")
print(f"min / max : {min(values):.2f} / {max(values):.2f}")
print(f"spread : {max(values) - min(values):.2f} points")
print(f"mean : {statistics.mean(values):.2f}")
print(f"stdev : {statistics.stdev(values):.3f}")
print(f"3 sigma : {3 * statistics.stdev(values):.2f}")
print(f"band = max(spread, 3 sigma) = {max(max(values) - min(values), 3 * statistics.stdev(values)):.2f}") seeds [100, 101, 102, 103] -> 45.25%
seeds [200, 201, 202, 203] -> 43.50%
seeds [300, 301, 302, 303] -> 45.00%
seeds [400, 401, 402, 403] -> 47.25%
seeds [500, 501, 502, 503] -> 44.00%
seeds [600, 601, 602, 603] -> 43.00%
seeds [700, 701, 702, 703] -> 44.50%
seeds [800, 801, 802, 803] -> 43.75%
runs : 8
min / max : 43.00 / 47.25
spread : 4.25 points
mean : 44.53
stdev : 1.333
3 sigma : 4.00
band = max(spread, 3 sigma) = 4.25Eight runs of an identical testlist against an identical design, spanning 43.00% to 47.25%. Nothing changed between them except the seeds, and the number moved 4.25 points.
Note max(spread, 3σ) chose the observed spread (4.25) over the computed 3σ (4.00), which is the same situation Chapter 28.6 met: with only eight samples the normal-distribution estimate can sit below a value you have already seen happen, and a band below an observed event is indefensible. Take the larger.
Against this policy the annotation's 0.5 is not slightly tight — it is eight and a half times too tight.
6. Pricing the Band in False Positives
A ratio is abstract. The cost of a wrong band is a measurable false-positive rate, and it is measurable without a single real regression — by asking how often the check fires on pairs of runs that are both fine:
import random
BINS = 400
DRAWS_PER_TEST = 60
def coverage(seeds):
hit = set()
for s in seeds:
rnd = random.Random(s)
for _ in range(DRAWS_PER_TEST):
hit.add(rnd.randrange(BINS))
return 100.0 * len(hit) / BINS
SETS = [[a, a + 1, a + 2, a + 3] for a in range(100, 900, 100)]
values = [coverage(s) for s in SETS]
pairs = [(a, b) for i, a in enumerate(values)
for j, b in enumerate(values) if i != j]
print(f"ordered pairs of runs of the IDENTICAL testlist: {len(pairs)}")
print()
print(f"{'band':>6} {'fires on':>10} {'rate':>7}")
for band in (0.5, 1.0, 2.0, 4.0, 4.25):
fires = sum(1 for a, b in pairs if b < a - band)
print(f"{band:>6.2f} {str(fires) + ' / ' + str(len(pairs)):>10} "
f"{100.0 * fires / len(pairs):>6.1f}%")ordered pairs of runs of the IDENTICAL testlist: 56
band fires on rate
0.50 21 / 56 37.5%
1.00 15 / 56 26.8%
2.00 7 / 56 12.5%
4.00 1 / 56 1.8%
4.25 0 / 56 0.0%The annotation's band fires on 21 of 56 pairs of identical testlists — 37.5% of the time. Not on a regression; on a pair of runs where nothing changed but the seed.
A check that cries wolf on more than a third of ordinary nights is worse than no check, because it gets routed to a folder within a fortnight, and then the real regression arrives and nobody reads that folder. This is Chapter 24.5's argument about a checker that complains on a good night being the fastest route to being switched off, arriving with a number attached.
So the module's closing idea:
POLICIES = [
("pinned seeds", 0.00, "any drop is real; investigate every one"),
("random seeds, measured band", 4.25, "0% false positives on 56 pairs"),
("random seeds, 0.5 band", 0.50, "37.5% false positives — the check dies"),
]
print(f"{'policy':<30} {'band':>6} consequence")
for name, band, what in POLICIES:
print(f"{name:<30} {band:>6.2f} {what}")
print()
print("the threshold is not a separate decision from the seed policy.")
print("pinned seeds buy sensitivity; random seeds buy stimulus diversity;")
print("and the band is the price of whichever you chose.")policy band consequence
pinned seeds 0.00 any drop is real; investigate every one
random seeds, measured band 4.25 0% false positives on 56 pairs
random seeds, 0.5 band 0.50 37.5% false positives — the check dies
the threshold is not a separate decision from the seed policy.
pinned seeds buy sensitivity; random seeds buy stimulus diversity;
and the band is the price of whichever you chose.7. A Gate Failure and a Regression Are Different Events
Chapter 29.2's gate asks whether coverage is high enough. This chapter's check asks whether it fell. Those are independent, and all four combinations happen:
FLOOR = 95.0
BAND = 0.5
CASES = [
("A", 96.5, 95.2),
("B", 94.9, 94.9),
("C", 96.0, 96.4),
("D", 95.4, 94.2),
]
print(f"{'case':<5} {'last':>6} {'today':>6} {'gate':<5} {'regression':<11} reading")
for name, last, today in CASES:
gate = "PASS" if today >= FLOOR else "FAIL"
regr = "YES" if today < last - BAND else "no"
reading = {
("PASS", "no"): "quiet night",
("PASS", "YES"): "fell, but still above the floor",
("FAIL", "no"): "below the floor and stable — a backlog, not news",
("FAIL", "YES"): "fell and is below the floor",
}[(gate, regr)]
print(f"{name:<5} {last:>6.1f} {today:>6.1f} {gate:<5} {regr:<11} {reading}")
print()
print("a gate verdict and a regression verdict are independent:", True)
print("so a run needs both, reported separately.")case last today gate regression reading
A 96.5 95.2 PASS YES fell, but still above the floor
B 94.9 94.9 FAIL no below the floor and stable — a backlog, not news
C 96.0 96.4 PASS no quiet night
D 95.4 94.2 FAIL YES fell and is below the floor
a gate verdict and a regression verdict are independent: True
so a run needs both, reported separately.Case A is the one a single verdict loses: coverage dropped 1.3 points and the gate says PASS, so a flow reporting only the gate reports a good night on the night something broke. Case B is its mirror — a red cell that is red every night and contains no new information, which is the cell teams learn to ignore.
Reporting them separately is what makes the dashboard legible: the gate is a statement about the project, and the regression check is a statement about last night.
8. A Bin Declared Illegal That Was Hit
Chapter 29.5 Section 6 ordered its classifier's checks so that a bin declared illegal with hits would be reported as covered rather than hidden, and promised that signoff is where the contradiction surfaces. Here it is:
ILLEGAL = {"cg.mx.idle_b4", "cg.mx.idle_b8"}
REPORT = {"cg.mx.idle_b1": 9, "cg.mx.idle_b4": 3, "cg.mx.burst_b8": 41}
hit_but_illegal = sorted(n for n, h in REPORT.items() if n in ILLEGAL and h > 0)
print("bins declared illegal:", sorted(ILLEGAL))
print("bins in the report :", sorted(REPORT))
print()
for name in hit_but_illegal:
print(f"CONTRADICTION: {name} is declared illegal and was hit {REPORT[name]} times")
print()
print("what it means:", "the design produced a combination the covergroup says cannot occur")
print("which of the two is wrong:", "unknown — and that is why it blocks signoff")bins declared illegal: ['cg.mx.idle_b4', 'cg.mx.idle_b8']
bins in the report : ['cg.mx.burst_b8', 'cg.mx.idle_b1', 'cg.mx.idle_b4']
CONTRADICTION: cg.mx.idle_b4 is declared illegal and was hit 3 times
what it means: the design produced a combination the covergroup says cannot occur
which of the two is wrong: unknown — and that is why it blocks signoffEither the design has a bug — it issued a four-beat burst while idle — or the declaration is wrong and a legitimate combination has been excluded from the denominator since the day somebody wrote it. Both possibilities are serious and the data cannot distinguish them, which is exactly the kind of finding a signoff review exists to catch. A real tool flags this itself; a flow that merges coverage from several sources has to check it, because the declaration and the hit can arrive from different runs.
9. The Evidence a Review Needs
The canonical purpose for this chapter has two halves, and the first is the one usually skipped: produce the coverage evidence a signoff review needs. Not a number. Evidence:
EVIDENCE = [
("per-metric hits and totals", "29.1 §6", "a percentage cannot be re-derived or compared"),
("thresholds with reasons", "29.2 §7", "a floor nobody can defend is not a policy"),
("the exclusion list", "29.5 §4", "with a reason and an author for each"),
("the illegal declarations", "§8", "and the contradiction check over them"),
("per-block detail", "29.3 §8", "so a reviewer can reach the tool's own report"),
("the testlist and seed policy", "§6", "the band means nothing without it"),
("the trend over N runs", "29.4 §7", "net, spread, steps up, steps down"),
("report paths", "29.1 §8", "point at what a person reads"),
]
print(f"{'evidence':<30} {'from':<9} why a review needs it")
for what, where, why in EVIDENCE:
print(f"{what:<30} {where:<9} {why}")
print()
print("items of evidence:", len(EVIDENCE))
print("items a single percentage provides:", 0)evidence from why a review needs it
per-metric hits and totals 29.1 §6 a percentage cannot be re-derived or compared
thresholds with reasons 29.2 §7 a floor nobody can defend is not a policy
the exclusion list 29.5 §4 with a reason and an author for each
the illegal declarations §8 and the contradiction check over them
per-block detail 29.3 §8 so a reviewer can reach the tool's own report
the testlist and seed policy §6 the band means nothing without it
the trend over N runs 29.4 §7 net, spread, steps up, steps down
report paths 29.1 §8 point at what a person reads
items of evidence: 8
items a single percentage provides: 0Eight items, every one of them produced by an earlier chapter of this module, and not one of them is derivable from a coverage percentage. That is the module in a table: the number at the top of a signoff slide is the least informative thing on it, and everything that makes it mean anything was decided and recorded elsewhere.
10. The Signoff Record, Assembled
from dataclasses import dataclass, field
from typing import Dict, List, Optional, Tuple
@dataclass(frozen=True)
class Exclusion:
reason: str
author: str
def __post_init__(self):
if not self.reason.strip() or not self.author.strip():
raise ValueError("an exclusion needs a reason and an author")
@dataclass
class SignoffRecord:
design: str
testlist: str
seed_policy: str
band: float
metrics: Dict[str, Tuple[int, int]]
thresholds: Dict[str, Optional[float]]
reasons: Dict[str, str]
exclusions: Dict[str, Exclusion]
illegal: set
hits_on_illegal: List[str]
history: List[float]
reports: Dict[str, str]
def pct(self, metric):
h, t = self.metrics[metric]
return 100.0 * h / t
def gate(self):
bad = []
ungoverned = sorted(set(self.metrics) - set(self.thresholds))
if ungoverned:
raise KeyError(f"no threshold for: {ungoverned}")
for m in sorted(self.metrics):
floor = self.thresholds[m]
if floor is not None and self.pct(m) < floor:
bad.append(m)
return bad
def regression(self):
if len(self.history) < 2:
return None
drop = self.history[-2] - self.history[-1]
return drop > self.band
def blockers(self):
out = []
if self.gate():
out.append(f"below threshold: {self.gate()}")
if self.regression():
out.append(f"regression beyond a band of {self.band}")
if self.hits_on_illegal:
out.append(f"illegal bins were hit: {self.hits_on_illegal}")
if self.seed_policy == "random" and self.band == 0.0:
out.append("a zero band with random seeds cannot be right")
return out
rec = SignoffRecord(
design="soc_top",
testlist="nightly.f",
seed_policy="pinned",
band=0.0,
metrics={"functional": (9496, 10000), "branch": (2742, 3000)},
thresholds={"functional": 95.0, "branch": 90.0},
reasons={"functional": "signoff policy", "branch": "agreed with design lead"},
exclusions={"cg.err.ecc_double": Exclusion("ECC not fitted in this SKU", "a.patel")},
illegal={"cg.mx.idle_b4", "cg.mx.idle_b8"},
hits_on_illegal=["cg.mx.idle_b4"],
history=[95.10, 95.02, 94.96],
reports={"functional": "cov/func.rpt", "branch": "cov/branch.rpt"},
)
print(f"design {rec.design} testlist {rec.testlist} seeds {rec.seed_policy} band {rec.band}")
print()
for m in sorted(rec.metrics):
h, t = rec.metrics[m]
floor = rec.thresholds[m]
print(f" {m:<11} {rec.pct(m):>6.2f}% {h}/{t} floor {floor} "
f"({rec.reasons[m]}) -> {rec.reports[m]}")
print()
print("exclusions:")
for name, ex in sorted(rec.exclusions.items()):
print(f" {name:<20} {ex.reason} ({ex.author})")
print()
print(f"history : {rec.history}")
print(f"last drop : {rec.history[-2] - rec.history[-1]:.2f} points")
print(f"regression : {rec.regression()}")
print(f"gate failures : {rec.gate()}")
print(f"illegal bins hit : {rec.hits_on_illegal}")
print()
print("BLOCKERS:")
for b in rec.blockers():
print(" -", b)
print()
print("signoff:", "BLOCKED" if rec.blockers() else "CLEAR")design soc_top testlist nightly.f seeds pinned band 0.0
branch 91.40% 2742/3000 floor 90.0 (agreed with design lead) -> cov/branch.rpt
functional 94.96% 9496/10000 floor 95.0 (signoff policy) -> cov/func.rpt
exclusions:
cg.err.ecc_double ECC not fitted in this SKU (a.patel)
history : [95.1, 95.02, 94.96]
last drop : 0.06 points
regression : True
gate failures : ['functional']
illegal bins hit : ['cg.mx.idle_b4']
BLOCKERS:
- below threshold: ['functional']
- regression beyond a band of 0.0
- illegal bins were hit: ['cg.mx.idle_b4']
signoff: BLOCKEDThree blockers from one record. The 0.06-point drop is a regression here and would not be one under any other policy in this chapter — because the seeds are pinned, the band is zero, and six bins of real coverage went missing. The annotation's 0.5 would have reported this night as clean while the gate failed for an unrelated reason, and the two facts would never have been connected.
Note the last blocker check: seed_policy == "random" and band == 0.0. The record validates its own policy for internal consistency, the way Chapter 21.7 validates a configuration before anything runs — because the one combination that is certainly wrong is a band that contradicts the seed policy it was chosen for.
11. Predict, Run, Observe
# A
last, today, BAND = 95.80, 95.10, 0.5
print("A:", today < last - BAND)
# B
print("B:", 95.02 - 94.96 > 0.0, 95.02 - 94.96 > 0.5)
# C
FLOOR = 95.0
print("C:", 95.2 >= FLOOR, 95.2 < 96.5 - 0.5)
# D
print("D:", int(10000 * 0.5 / 100))The verified output:
A: True
B: True False
C: True True
D: 50A — the annotation fires on a 0.70-point drop, which Section 6 showed happens on 37.5% of pairs of identical random-seed runs.
B — a 0.06-point drop is a regression under a pinned-seed band of zero and invisible under a band of 0.5. The same night, two verdicts, decided entirely by the seed policy.
C — PASS on the gate and YES on the regression, simultaneously. Section 7's case A, and the reason both have to be reported.
D — fifty bins. What a 0.5-point band tolerates on a 10,000-bin design.
12. Exercises
Exercise 1 — five causes. List the causes of a coverage drop in your flow and say which your current check can distinguish.
Exercise 2 — gate on the numerator. Change your regression check to compare numerator effects rather than headlines, and say what that fixes.
Exercise 3 — where did your number come from. Find your flow's coverage regression band and the person or commit that chose it.
Exercise 4 — pin and repeat. Run your regression twice with identical seeds and report whether the coverage numbers match exactly.
Exercise 5 — fifty bins. For your design's bin count, compute how many bins your current band tolerates.
Exercise 6 — measure the spread. Run the identical testlist with eight different seed sets and report min, max, spread, stdev and 3σ.
Exercise 7 — the larger of the two. Say which of spread and 3σ your data makes larger, and why you must take the larger.
Exercise 8 — price your band. Compute the false-positive rate of your current band over all ordered pairs of your eight runs.
Exercise 9 — the policy pair. State your seed policy and the band it implies, and say whether your flow's actual band matches.
Exercise 10 — two runs, two jobs. Describe how you would split a change-detecting pinned run from a bug-hunting random sweep in your flow.
Exercise 11 — four quadrants. Find a real night in your history for each of the four gate × regression combinations.
Exercise 12 — case A. Explain what your dashboard reported on a night when coverage fell and the gate passed.
Exercise 13 — the contradiction. Add the illegal-bins-were-hit check to your flow and run it over a merged coverage database.
Exercise 14 — eight items. Audit your last signoff package against Section 9's list and report which items were missing.
Exercise 15 — self-validating policy. Add a check that your band is consistent with your seed policy, and say what other inconsistency pairs are worth checking.
Exercise 16 — the whole module. Take one real coverage number from your flow and write down, for each of the six chapters of this module, what that chapter would ask about it.
Exercise 17 — the dominant source. List the things that vary between two runs of your regression and say which actually moves coverage.
Exercise 18 — policy before threshold. Explain why the band cannot be chosen before the seed policy is decided.
Exercise 19 — too loose. For your bin count, say how much real coverage loss a 0.5-point band hides under pinned seeds.
Exercise 20 — eight seed sets. Run the experiment and report the table, including which of spread and 3σ was larger.
Exercise 21 — the folder. Describe what happens operationally when a nightly check has a 37.5% false-positive rate.
Exercise 22 — price three candidates. Compute the false-positive rate of three candidate bands over your own ordered pairs.
Exercise 23 — two runs, two jobs. Write the one-paragraph proposal for splitting your pinned gate from your random sweep.
Exercise 24 — the stable red cell. Find a metric in your dashboard that has been red every night for a month and say what information it carries.
Exercise 25 — merged databases. Explain why the illegal-bins-were-hit check is needed when coverage is merged from several runs.
Exercise 26 — which is wrong. For a contradiction you find, write the two competing explanations and the experiment that would settle it.
Exercise 27 — the eight items. Produce a signoff package for one block containing all eight items of evidence.
Exercise 28 — more inconsistency pairs. Name two other policy pairs a record could validate about itself, besides band against seed policy.
13. Summary
- Five causes lower a coverage number and only two are regressions, so the check belongs on Chapter 29.4's numerator effect rather than on the headline.
- Coverage noise has one dominant source — the seeds — which makes it a property of a policy, unlike the tool noise Chapter 28.6 measured.
- Pinned seeds give the identical number five times out of five. The correct band is zero, and the annotation's 0.5 silently tolerates 50 bins of real loss on a 10,000-bin design.
- Eight different seed sets over one testlist spanned 4.25 points. Against that policy, 0.5 is eight and a half times too tight.
- Take
max(spread, 3σ)— with eight samples the normal estimate can fall below something you have already observed. - The annotation's band fires on 21 of 56 pairs of identical testlists: 37.5% false positives. A check that wrong gets filtered into a folder, and then the real regression arrives.
- The threshold and the seed policy are one decision. Pinned buys sensitivity, random buys stimulus diversity, and the band is the price of whichever you chose.
- A pinned change detector and a random bug hunter are two runs with two jobs. Gate the first; report the second as a trend.
- A gate failure and a regression are independent events — a run can fall 1.3 points and still pass the floor, and a red cell that is red every night carries no information.
- A bin declared illegal that was hit is a contradiction that blocks signoff, because either the design or the declaration is wrong and the data cannot say which.
- A signoff package is eight items of evidence, every one produced by an earlier chapter of this module, and not one of them derivable from a percentage.
- Let the record validate its own policy — a zero band with random seeds is certainly wrong, and a flow can say so before a reviewer has to.
- Coverage noise has one dominant source and it is a choice, which is what separates this threshold from the tool-noise threshold of Chapter 28.6.
- A 0.5-point band hides 50 bins of real loss on a 10,000-bin design when the seeds are pinned.
- Price a candidate band in false positives over ordered pairs of runs that are both fine — no real regression is needed to measure it.
- A check with a 37.5% false-positive rate gets filtered into a folder, and then the real regression arrives.
- A stable red cell carries no information. Report the gate and the regression separately so a new fall is distinguishable from a standing backlog.
- Let the record check its own policy for consistency — a zero band with random seeds is certainly wrong.
Module 30 turns the records this module and Module 28 produce into the reports a human actually reads — totals, buckets, new and fixed failures, rendered and checked.
Where this fits
Part of the Python curriculum.