Python Part II · Module 28 · Report Parsing
Detecting a QoR Regression
Chapter 28.5 produced a table of differences and left the hard question open: which of them matter. Turning that into an automatic verdict is one comparison against a threshold, and the threshold is the entire chapter — because a place-and-route flow does not produce the same answer twice, so some difference between two runs of the identical design is guaranteed. A number chosen rather than measured is either too tight, in which case the check fires on identical designs, or too loose, in which case it misses the regression it was written for.
Advanced19 min readPythonPart IIAutomationQoRReport Parsing
Part II · Module 28 · Chapter 28.6 · Detecting a QoR Regression
1. The Flow Is Not Deterministic
SOURCES = [
("multi-threaded placement", "work is split by thread, and threads race"),
("timing-driven optimisation", "it stops on a wall-clock or effort budget"),
("machine load", "a busier machine reaches a different stopping point"),
("tool version", "a patch release changes a heuristic"),
]
for name, why in SOURCES:
print(f"{name:<28} {why}")
print()
print("the design was identical in all four:", True)
print("the QoR numbers were :", "not identical")
print("so a difference of zero is :", "not the expected outcome")multi-threaded placement work is split by thread, and threads race
timing-driven optimisation it stops on a wall-clock or effort budget
machine load a busier machine reaches a different stopping point
tool version a patch release changes a heuristic
the design was identical in all four: True
the QoR numbers were : not identical
so a difference of zero is : not the expected outcomeTwo runs of the same design produce different numbers, which is the premise everything else rests on. A simulation regression can be driven to Chapter 25.7's zero non-determinism; a place-and-route flow cannot, because the non-determinism is in the tool's own search.
So a comparison always shows differences, and a verdict needs to know which size of difference is ordinary.
2. The Annotation
if delta["wns"] < -0.050:
fail("timing regression beyond noise threshold")The comment names exactly the right concept — beyond noise — and the chapter is about the number in front of it.
QUESTIONS = [
("where does 0.050 come from?", "measured, or chosen?"),
("what if wns is unavailable?", "Chapter 28.5 §4 produces that"),
("what about the other metrics?", "a verdict on one is not a verdict"),
("is an absolute threshold right?", "0.050 against a baseline of -0.010 or -2.000?"),
]
for q, note in QUESTIONS:
print(f"{q:<34} {note}")
print()
print("the comment is right:", True)
print("the number is :", "unjustified until somebody measures")where does 0.050 come from? measured, or chosen?
what if wns is unavailable? Chapter 28.5 §4 produces that
what about the other metrics? a verdict on one is not a verdict
is an absolute threshold right? 0.050 against a baseline of -0.010 or -2.000?
the comment is right: True
the number is : unjustified until somebody measures3. Measure the Noise
import statistics
SAME_DESIGN = [-0.045, -0.038, -0.052, -0.041, -0.049, -0.036, -0.047, -0.043]
print("eight runs, identical inputs:")
print(" ", SAME_DESIGN)
print()
print("mean :", round(statistics.mean(SAME_DESIGN), 4))
print("stdev :", round(statistics.stdev(SAME_DESIGN), 4))
print("min :", min(SAME_DESIGN))
print("max :", max(SAME_DESIGN))
print("observed spread:", round(max(SAME_DESIGN) - min(SAME_DESIGN), 4))
print("3 sigma :", round(3 * statistics.stdev(SAME_DESIGN), 4))eight runs, identical inputs:
[-0.045, -0.038, -0.052, -0.041, -0.049, -0.036, -0.047, -0.043]
mean : -0.0439
stdev : 0.0055
min : -0.052
max : -0.036
observed spread: 0.016
3 sigma : 0.0164The same design, eight times, spread across sixteen picoseconds. That is the number a threshold has to clear, and the only way to obtain it is the experiment above — Chapter 25.7's soak, pointed at an implementation flow instead of a simulation.
The sample is a constructed list here; in a real flow it is eight nightly runs of an unchanged commit, which costs nothing extra because the nightly runs anyway.
Now price every candidate threshold against it:
import statistics
SAME_DESIGN = [-0.045, -0.038, -0.052, -0.041, -0.049, -0.036, -0.047, -0.043]
pairs = [(a, b) for i, a in enumerate(SAME_DESIGN)
for j, b in enumerate(SAME_DESIGN) if i != j]
print(f"{'threshold':>10} {'fires on':>9} {'of':>5} {'false positives':>17}")
for t in (0.005, 0.010, 0.016, 0.020, 0.050):
fires = sum(1 for a, b in pairs if b - a < -t)
print(f"{t:>10.3f} {fires:>9} {len(pairs):>5} {100 * fires / len(pairs):>16.1f}%")
print()
print("observed spread :", round(max(SAME_DESIGN) - min(SAME_DESIGN), 4))
print("3 sigma :", round(3 * statistics.stdev(SAME_DESIGN), 4))
print("the annotation's:", 0.050) threshold fires on of false positives
0.005 16 56 28.6%
0.010 6 56 10.7%
0.016 0 56 0.0%
0.020 0 56 0.0%
0.050 0 56 0.0%
observed spread : 0.016
3 sigma : 0.0164
the annotation's: 0.05A threshold of 0.005 fires on 28.6% of pairs of identical runs.
Every one of those is a false alarm: the design did not change, the flow did, and the check reports a timing regression. At one nightly a day that is a spurious failure roughly twice a week, which is Chapter 25.7 Section 6's noise floor in a new place — and once a team has seen three of them, the fourth is ignored.
Three observations about the table.
The three-sigma figure and the observed spread agree, at 0.0164 and 0.016. That is a coincidence of this sample and a useful sanity check: if they disagree badly, the distribution is not what the statistics assume and the observed spread is the one to trust.
The annotation's 0.050 is safe and three times larger than it needs to be. It will never false-alarm on this flow and it will miss any regression between 16 and 50 picoseconds — which, on a design at −0.045, is most of the ones worth catching.
Neither of those facts is knowable without the experiment. A number chosen by intuition can be wrong in either direction and there is no way to tell from inside the flow.
4. Unavailable Is Not Pass
DELTA_OK = {"wns": -0.080, "total_area": 1200.0}
DELTA_MISSING = {"total_area": 1200.0}
for label, delta in (("wns present", DELTA_OK), ("wns unavailable", DELTA_MISSING)):
try:
verdict = "FAIL" if delta["wns"] < -0.050 else "PASS"
except KeyError as exc:
verdict = f"KeyError: {exc}"
print(f"{label:<17} {verdict}")
print()
print("where the missing wns comes from:", "Chapter 28.5 §4 — unavailable metric")
print("why that run matters :", "an unparseable report is itself a problem")
print()
for guard, name in ((lambda d: d.get("wns") is not None, "a .get() guard"),):
print(f"{name:<16} present: {guard(DELTA_OK)} missing: {guard(DELTA_MISSING)}")
print()
print("the honest verdict when it is missing:", "UNKNOWN")
print("not :", "PASS")wns present FAIL
wns unavailable KeyError: 'wns'
where the missing wns comes from: Chapter 28.5 §4 — unavailable metric
why that run matters : an unparseable report is itself a problem
a .get() guard present: True missing: False
the honest verdict when it is missing: UNKNOWN
not : PASSThe annotation raises on exactly the run a verdict matters for.
Chapter 28.1 Section 9 chose to report an unknown WNS as None rather than fabricating a zero, and Chapter 28.5 Section 4 carried that through as an unavailable metric. Both were right, and the consequence lands here: a check written as delta["wns"] crashes, and the obvious fix — delta.get("wns", 0.0) — is worse, because it says no change about a report nobody could read.
The verdict needs three outcomes rather than two:
def verdict(delta, threshold=-0.016):
if "wns" not in delta or delta["wns"] is None:
return "UNKNOWN", "wns unavailable — the timing report could not be read"
if delta["wns"] < threshold:
return "FAIL", f"wns regressed by {delta['wns']:+.4f}, beyond {threshold:+.4f}"
return "PASS", f"wns moved {delta['wns']:+.4f}, within noise"
CASES = [("clean", {"wns": 0.003}), ("regressed", {"wns": -0.080}),
("marginal", {"wns": -0.010}), ("unreadable", {"wns": None}),
("absent", {})]
for label, delta in CASES:
v, why = verdict(delta)
print(f"{label:<12} {v:<8} {why}")
print()
outcomes = sorted({verdict(d)[0] for _, d in CASES})
print("distinct outcomes:", outcomes)
print("UNKNOWN blocks a merge the same as FAIL:", "yes — it is not evidence of success")clean PASS wns moved +0.0030, within noise
regressed FAIL wns regressed by -0.0800, beyond -0.0160
marginal PASS wns moved -0.0100, within noise
unreadable UNKNOWN wns unavailable — the timing report could not be read
absent UNKNOWN wns unavailable — the timing report could not be read
distinct outcomes: ['FAIL', 'PASS', 'UNKNOWN']
UNKNOWN blocks a merge the same as FAIL: yes — it is not evidence of successThree outcomes, and UNKNOWN is not a pass. This is Chapter 26.1 Section 5's vocabulary argument: a gate with two states forces every uncertain case into one of them, and the comfortable direction is always the permissive one.
Note the marginal row. A WNS that dropped by 10 picoseconds is PASS under a measured threshold of 16, and that is the threshold working — not a near miss. Reporting it as "within noise" rather than silently is what lets somebody disagree with the calibration.
5. One Metric Is Not a Verdict
DELTA = {"wns": 0.010, "tns": 0.200, "total_area": 9000.0,
"cells": 2400.0, "power_w": 0.030}
annotation_verdict = "FAIL" if DELTA["wns"] < -0.050 else "PASS"
print("the annotation says:", annotation_verdict)
print()
print("what else happened:")
BASELINE = {"total_area": 128456.3, "power_w": 0.2343}
for k in ("total_area", "power_w"):
pct = DELTA[k] / BASELINE[k] * 100
print(f" {k:<12} {DELTA[k]:>+10.4f} {pct:+.2f}%")
print()
print("area grew by :", f"{DELTA['total_area'] / BASELINE['total_area'] * 100:.1f}%")
print("power grew by :", f"{DELTA['power_w'] / BASELINE['power_w'] * 100:.1f}%")
print("checked by the gate:", "neither")the annotation says: PASS
what else happened:
total_area +9000.0000 +7.01%
power_w +0.0300 +12.80%
area grew by : 7.0%
power grew by : 12.8%
checked by the gate: neitherA seven percent area growth and a thirteen percent power increase, reported as a pass.
A gate on one metric is a gate on one metric, and the one chosen is usually timing because timing is what blocks a tape-out. Area and power block one later and more expensively.
Every metric needs its own measured threshold, in its own units:
import statistics
NOISE_SAMPLES = {
"wns": [-0.045, -0.038, -0.052, -0.041, -0.049, -0.036, -0.047, -0.043],
"total_area": [128456.3, 128502.1, 128399.7, 128511.0, 128460.2,
128433.8, 128478.9, 128444.5],
"power_w": [0.2343, 0.2351, 0.2338, 0.2349, 0.2340, 0.2346, 0.2342, 0.2347],
}
def threshold_from(samples, sigmas=3):
"""A regression must exceed the observed spread AND the sigma estimate."""
spread = max(samples) - min(samples)
sigma = sigmas * statistics.stdev(samples)
return round(max(spread, sigma), 6)
for metric, samples in NOISE_SAMPLES.items():
spread = round(max(samples) - min(samples), 6)
sigma = round(3 * statistics.stdev(samples), 6)
print(f"{metric:<12} spread {spread:>12} 3-sigma {sigma:>12} "
f"threshold {threshold_from(samples):>12}")
print()
print("each metric has its own unit:", True)
print("a single threshold for all :", "meaningless — ns against um^2 against W")wns spread 0.016 3-sigma 0.016388 threshold 0.016388
total_area spread 111.3 3-sigma 109.35959 threshold 111.3
power_w spread 0.0013 3-sigma 0.001351 threshold 0.001351
each metric has its own unit: True
a single threshold for all : meaningless — ns against um^2 against WThree metrics, three thresholds, three units. Area's noise is 111 µm² — so the 9000 µm² growth in the previous example is eighty times the noise and unambiguously real.
Note which estimate won for each metric: wns and power_w took the three-sigma figure and total_area took the observed spread, because its sigma estimate (109.4) came out below what actually happened (111.3). That is exactly the case max(spread, sigma) exists for — a sigma estimate that under-predicts the real range is the one that would have produced false alarms.
max(spread, sigma) is deliberately the more conservative of the two estimates. The observed spread is what actually happened and the sigma figure extrapolates; taking the larger means a few extra missed marginal regressions and no false alarms, which is the right trade for a gate people have to trust.
6. Absolute or Relative
BASELINES = (-0.010, -0.500, -2.000)
ABSOLUTE = 0.050
print(f"{'baseline':>10} {'absolute T':>12} {'as a fraction of baseline':>27}")
for b in BASELINES:
print(f"{b:>10.3f} {ABSOLUTE:>12.3f} {abs(ABSOLUTE / b) * 100:>26.1f}%")
print()
print("the same threshold means:", "very different things")
print()
RELATIVE = 0.10
print(f"{'baseline':>10} {'relative T':>12} {'in ns':>10}")
for b in BASELINES:
print(f"{b:>10.3f} {RELATIVE:>11.0%} {abs(b) * RELATIVE:>10.4f}")
print()
print("the relative threshold on a design at -0.010:", f"{abs(-0.010) * RELATIVE:.4f} ns")
print("which is below the measured noise of :", 0.0164)
print("so it would fire constantly :", abs(-0.010) * RELATIVE < 0.0164) baseline absolute T as a fraction of baseline
-0.010 0.050 500.0%
-0.500 0.050 10.0%
-2.000 0.050 2.5%
the same threshold means: very different things
baseline relative T in ns
-0.010 10% 0.0010
-0.500 10% 0.0500
-2.000 10% 0.2000
the relative threshold on a design at -0.010: 0.0010 ns
which is below the measured noise of : 0.0164
so it would fire constantly : TrueNeither form works alone. An absolute threshold is 500% of one baseline and 2.5% of another; a relative one collapses below the noise floor as the design closes timing.
Chapter 28.5 Section 6 made the same point about reporting a change and the resolution here is the same shape — the floor comes from the noise and the ceiling from the proportion:
NOISE_T = 0.0164
RELATIVE = 0.10
def timing_threshold(baseline):
"""At least the noise floor; more for a design with a lot of slack to lose."""
return max(NOISE_T, abs(baseline) * RELATIVE)
for baseline in (-0.010, -0.100, -0.500, -2.000):
t = timing_threshold(baseline)
binding = "noise" if t == NOISE_T else "proportion"
print(f"baseline {baseline:>+7.3f} threshold {t:>7.4f} set by {binding}")
print()
print("below -0.164, the floor binds:", timing_threshold(-0.010) == NOISE_T)
print("above it, the proportion does:", timing_threshold(-2.000) > NOISE_T)
print("the crossover :", f"{NOISE_T / RELATIVE:.3f} ns of baseline slack")baseline -0.010 threshold 0.0164 set by noise
baseline -0.100 threshold 0.0164 set by noise
baseline -0.500 threshold 0.0500 set by proportion
baseline -2.000 threshold 0.2000 set by proportion
below -0.164, the floor binds: True
above it, the proportion does: True
the crossover : 0.164 ns of baseline slackA floor at the measured noise and a proportion above it, crossing over at 0.164 ns of baseline. A design close to meeting timing is gated on the noise floor, because every picosecond matters there; a design far out is gated proportionally, because a 10 picosecond move on a −2.0 ns path is not the problem.
7. The Gate
import statistics
NOISE_SAMPLES = {
"wns": [-0.045, -0.038, -0.052, -0.041, -0.049, -0.036, -0.047, -0.043],
"total_area": [128456.3, 128502.1, 128399.7, 128511.0, 128460.2,
128433.8, 128478.9, 128444.5],
"power_w": [0.2343, 0.2351, 0.2338, 0.2349, 0.2340, 0.2346, 0.2342, 0.2347],
}
DIRECTION = {"wns": "higher", "total_area": "lower", "power_w": "lower"}
UNITS = {"wns": "ns", "total_area": "um^2", "power_w": "W"}
def thresholds(samples, sigmas=3):
return {m: round(max(max(v) - min(v), sigmas * statistics.stdev(v)), 6)
for m, v in samples.items()}
def gate(before, after, limits):
findings, unknown = [], []
for metric, limit in sorted(limits.items()):
b, a = before.get(metric), after.get(metric)
if b is None or a is None:
unknown.append(metric)
continue
delta = a - b
worse = delta < 0 if DIRECTION[metric] == "higher" else delta > 0
if worse and abs(delta) > limit:
findings.append({"metric": metric, "delta": round(delta, 6),
"limit": limit, "factor": round(abs(delta) / limit, 1),
"unit": UNITS[metric]})
if unknown:
return "UNKNOWN", findings, unknown
return ("FAIL" if findings else "PASS"), findings, unknown
LIMITS = thresholds(NOISE_SAMPLES)
BEFORE = {"wns": -0.045, "total_area": 128456.3, "power_w": 0.2343}
for label, after in (
("clean", {"wns": -0.043, "total_area": 128470.0, "power_w": 0.2345}),
("timing regressed", {"wns": -0.120, "total_area": 128470.0, "power_w": 0.2345}),
("area regressed", {"wns": -0.043, "total_area": 137456.3, "power_w": 0.2345}),
("report unreadable", {"wns": None, "total_area": 128470.0, "power_w": 0.2345}),
):
v, findings, unknown = gate(BEFORE, after, LIMITS)
print(f"{label:<19} {v}")
for f in findings:
print(f" {f['metric']} {f['delta']:+.4f} {f['unit']} "
f"(limit {f['limit']:.4f}, {f['factor']}x)")
if unknown:
print(f" unavailable: {unknown}")
print()
print("thresholds in use:", {k: round(v, 4) for k, v in LIMITS.items()})clean PASS
timing regressed FAIL
wns -0.0750 ns (limit 0.0164, 4.6x)
area regressed FAIL
total_area +9000.0000 um^2 (limit 111.3000, 80.9x)
report unreadable UNKNOWN
unavailable: ['wns']
thresholds in use: {'wns': 0.0164, 'total_area': 111.3, 'power_w': 0.0014}Six decisions, and the fourth is what makes the verdict usable rather than merely correct.
Thresholds are derived from the noise samples, not written as literals, so recalibrating is a matter of adding runs to the sample rather than editing a constant.
Direction decides what "worse" means, per Chapter 28.5 Section 5 — so area growing and WNS shrinking are both regressions despite opposite signs.
UNKNOWN dominates. One unavailable metric makes the whole verdict unknown, rather than passing on the strength of the metrics that happened to parse.
Every finding reports the factor — 4.6× and 78.8× — which is the number that tells a reviewer whether this is marginal or catastrophic. A bare "exceeded the threshold" does not distinguish a 1.1× from a 78×.
The thresholds in use are printed, so the verdict can be argued with. Chapter 24.5's pattern: a decision that explains itself.
Findings are sorted by metric name, so two runs of the gate produce comparable output.
8. What the Gate Cannot Know
LIMITS = [
("a regression smaller than the noise", "invisible by construction",
"run more samples, or accept it"),
("a regression in a metric not measured", "congestion, hold timing, IR drop",
"add the metric"),
("a gradual drift", "each night within noise, a month out of spec",
"trend, not a pairwise gate"),
("a change that is correct but costly", "a deliberate area-for-timing trade",
"a human decides"),
]
print(f"{'what it misses':<40} {'why':<38} what helps")
for what, why, fix in LIMITS:
print(f"{what:<40} {why:<38} {fix}")
print()
print("misses that more samples fix :", 1)
print("misses that need a trend :", 1)
print("misses that need a person :", 1)what it misses why what helps
a regression smaller than the noise invisible by construction run more samples, or accept it
a regression in a metric not measured congestion, hold timing, IR drop add the metric
a gradual drift each night within noise, a month out of spec trend, not a pairwise gate
a change that is correct but costly a deliberate area-for-timing trade a human decides
misses that more samples fix : 1
misses that need a trend : 1
misses that need a person : 1The third row is the one a pairwise gate is structurally incapable of catching:
NIGHTLY = [-0.045, -0.050, -0.056, -0.061, -0.068, -0.074, -0.081, -0.088]
THRESHOLD = 0.0164
print("night-to-night deltas:")
fires = 0
for i in range(1, len(NIGHTLY)):
delta = NIGHTLY[i] - NIGHTLY[i - 1]
flagged = abs(delta) > THRESHOLD and delta < 0
fires += flagged
print(f" night {i}: {delta:+.4f} flagged: {flagged}")
total = NIGHTLY[-1] - NIGHTLY[0]
print()
print("nights flagged :", fires)
print("total drift :", f"{total:+.4f} ns")
print("as multiples of the threshold:", round(abs(total) / THRESHOLD, 1))
print()
print("every single step was within noise:", fires == 0)
print("the week was not :", abs(total) > THRESHOLD)night-to-night deltas:
night 1: -0.0050 flagged: False
night 2: -0.0060 flagged: False
night 3: -0.0050 flagged: False
night 4: -0.0070 flagged: False
night 5: -0.0060 flagged: False
night 6: -0.0070 flagged: False
night 7: -0.0070 flagged: False
nights flagged : 0
total drift : -0.0430 ns
as multiples of the threshold: 2.6
every single step was within noise: True
the week was not : TrueSeven nights, no flags, and 43 picoseconds lost.
Each step is within the noise and the week is 2.6 times the threshold. A pairwise gate comparing last night to tonight cannot see it, by construction, and the fix is not a tighter threshold — that would false-alarm at the rate Section 3 measured.
The fix is a second check against a fixed reference: last week's run, or the last known-good tag, so the accumulated drift is compared in one step. Chapter 26.7 Section 9 made the same argument about failure buckets — a night-to-night comparison answers a different question from a comparison against a baseline — and a QoR gate needs both.
9. Predict, Run, Observe
import statistics
S = [-0.045, -0.038, -0.052, -0.041, -0.049, -0.036, -0.047, -0.043]
pairs = [(a, b) for i, a in enumerate(S) for j, b in enumerate(S) if i != j]
# A
print("A:", round(statistics.stdev(S), 4))
# B
print("B:", round(max(S) - min(S), 4))
# C
print("C:", round(3 * statistics.stdev(S), 4))
# D
print("D:", sum(1 for a, b in pairs if b - a < -0.005), len(pairs))
# E
print("E:", sum(1 for a, b in pairs if b - a < -0.016))
# F
d = {"tns": -0.2}
print("F:", d.get("wns") is None)
# G
print("G:", max(0.0164, abs(-0.010) * 0.10), max(0.0164, abs(-2.000) * 0.10))
# H
seq = [-0.045, -0.050, -0.056, -0.061, -0.068, -0.074, -0.081, -0.088]
print("H:", all(abs(b - a) < 0.0164 for a, b in zip(seq, seq[1:])),
abs(seq[-1] - seq[0]) < 0.0164)The verified output:
A: 0.0055
B: 0.016
C: 0.0164
D: 16 56
E: 0
F: True
G: 0.0164 0.2
H: True FalseA — the run-to-run standard deviation.
B — and the observed spread, which is what a threshold must clear.
C — three sigma agrees with it closely on this sample.
D — a 0.005 threshold fires on sixteen of fifty-six identical-design pairs.
E — a measured threshold fires on none.
F — .get distinguishes unavailable from a value, which delta["wns"] cannot.
G — the floor binds near zero and the proportion binds far from it.
H — every step within noise and the total outside it. Section 8's drift, in one line.
10. Exercises
Exercise 1 — four sources. List the reasons your implementation flow is non-deterministic.
Exercise 2 — not zero. Say why a difference of zero is not the expected outcome.
Exercise 3 — four questions. Ask all four of your own gate and record the answers.
Exercise 4 — measure it. Run your flow eight times on an unchanged commit and report the spread.
Exercise 5 — the cost. Say why that experiment is nearly free in a flow that runs nightly.
Exercise 6 — price the thresholds. Compute the false-positive rate for four candidate thresholds.
Exercise 7 — too tight. Say what a 28% false-positive rate does to a team over a month.
Exercise 8 — too loose. Compute the range of regressions your current threshold would miss.
Exercise 9 — spread or sigma. Say what it means when the two disagree badly.
Exercise 10 — unavailable. Reproduce the KeyError and say which run produces it.
Exercise 11 — the worse fix. Explain why delta.get("wns", 0.0) is worse than crashing.
Exercise 12 — three outcomes. Implement UNKNOWN and say why it must block.
Exercise 13 — the marginal row. Say why reporting "within noise" beats silence.
Exercise 14 — one metric. Construct a run that passes a timing gate while area grows 7%.
Exercise 15 — a threshold each. Measure the noise for three of your metrics and report all three with units.
Exercise 16 — the conservative estimate. Justify max(spread, sigma) and say what it trades.
Exercise 17 — neither alone. Show an absolute threshold meaning 500% of one baseline, and a relative one below the noise floor.
Exercise 18 — the crossover. Compute where the floor stops binding for your flow.
Exercise 19 — the factor. Add it to your findings and say what it tells a reviewer.
Exercise 20 — the drift. Construct seven within-noise nights that lose more than the threshold in total.
Exercise 21 — two comparisons. Implement the fixed-reference check alongside the pairwise one, and say which question each answers.
11. Summary
- An implementation flow is not deterministic, so two runs of one design always differ and a verdict needs to know which size of difference is ordinary.
- The annotation's comment is right and its number is unjustified until somebody measures.
- Measure the spread by running the same design repeatedly — Chapter 25.7's soak, pointed at a P&R flow.
- A threshold of 0.005 fired on 28.6% of identical-run pairs, which is a false alarm roughly twice a week.
- A threshold below the observed spread fires on identical designs, at a measurable rate.
- The annotation's 0.050 is safe and three times larger than needed, missing every regression between 16 and 50 picoseconds.
- Check the three-sigma estimate against the observed spread; a bad disagreement means the distribution is not what the statistics assume.
- Re-measure after a tool upgrade or a thread-count change — a cluster of marginal regressions is usually a widened spread.
delta["wns"]raises on exactly the run a verdict matters for, because Chapter 28.1 refused to fabricate a WNS.delta.get("wns", 0.0)is worse than crashing — it claims no change about a report nobody could read.- A gate needs three outcomes, and
UNKNOWNblocks exactly asFAILdoes. - One metric is not a verdict: a 7% area growth and a 13% power increase passed a timing gate.
- Every metric needs its own measured threshold in its own unit; area's noise here is 111 µm².
- Take
max(spread, sigma)— the conservative estimate, trading a few missed marginal regressions for no false alarms. - Neither an absolute nor a relative threshold works alone; use a floor at the noise and a proportion above it.
- Report the factor by which a finding exceeds its threshold, which distinguishes 1.1× from 78×.
- A pairwise gate cannot see gradual drift — seven within-noise nights lost 2.6 thresholds' worth.
- Run two comparisons: last night for "did this commit regress", and a fixed reference for "is the design still where it should be".
That completes Module 28. Three kinds of implementation report now become records of numbers whose provenance travels with them, two records compare only when they describe the same experiment, and a difference becomes a verdict only when it exceeds a threshold somebody measured. Module 29 turns from timing and area to coverage.
Where this fits
Part of the Python curriculum.