Python Part II · Module 28 · Report Parsing
Comparing QoR Between Two Runs
Four chapters have turned three kinds of report into records of numbers. Comparing two of those records is a one-line dict comprehension, and the line is correct on clean input and wrong about almost everything else: it raises on the unknown value Chapter 28.1 deliberately produces, it says nothing about which direction is an improvement, and expressing its results as percentages makes a thirty-three picosecond change look like the largest movement in the report.
Advanced19 min readPythonPart IIAutomationQoRReport Parsing
Part II · Module 28 · Chapter 28.5 · Comparing QoR Between Two Runs
1. Two Records
BEFORE = {"context": {"library": "tcbn05_lvt", "corner": "ss0p675v125c",
"effort": "high", "max_paths": 100},
"metrics": {"wns": -0.045, "tns": -1.230, "total_area": 128456.3,
"cells": 45231, "utilisation": 0.8521, "power_w": 0.2343}}
AFTER = {"context": dict(BEFORE["context"]),
"metrics": {"wns": -0.012, "tns": -0.310, "total_area": 131002.7,
"cells": 45980, "utilisation": 0.8712, "power_w": 0.2401}}
print(f"{'metric':<14} {'before':>13} {'after':>13}")
for k in BEFORE["metrics"]:
print(f"{k:<14} {BEFORE['metrics'][k]:>13.4f} {AFTER['metrics'][k]:>13.4f}")
print()
print("context identical:", BEFORE["context"] == AFTER["context"])
print("metrics :", len(BEFORE["metrics"]))metric before after
wns -0.0450 -0.0120
tns -1.2300 -0.3100
total_area 128456.3000 131002.7000
cells 45231.0000 45980.0000
utilisation 0.8521 0.8712
power_w 0.2343 0.2401
context identical: True
metrics : 6Two records of the shape Chapters 28.3 and 28.4 produce: a context of what the runs were, and a metrics of what they measured. The split is doing work, and the next section is what.
2. The Gate Comes First
PINNED = ("library", "corner", "effort", "max_paths")
BASE = {"library": "tcbn05_lvt", "corner": "ss0p675v125c", "effort": "high",
"max_paths": 100}
SAME = dict(BASE)
NEW_LIB = dict(BASE, library="tcbn05_svt", corner="tt0p75v25c")
TRUNCATED = dict(BASE, max_paths=10)
def comparable(a, b):
differ = sorted(k for k in PINNED if a.get(k) != b.get(k))
return (not differ), differ
for label, other in (("same setup", SAME), ("different library", NEW_LIB),
("different max_paths", TRUNCATED)):
ok, differ = comparable(BASE, other)
print(f"{label:<21} comparable={str(ok):<6} differing: {differ}")
print()
print("why max_paths is pinned:", "Chapter 28.2 §5 — TNS depends on it")
print("why library is pinned :", "Chapter 28.3 §7 — area depends on it")
print("a delta across either :", "measures the setup, not the design")same setup comparable=True differing: []
different library comparable=False differing: ['corner', 'library']
different max_paths comparable=False differing: ['max_paths']
why max_paths is pinned: Chapter 28.2 §5 — TNS depends on it
why library is pinned : Chapter 28.3 §7 — area depends on it
a delta across either : measures the setup, not the designThe gate is a precondition, not advice. Chapter 28.2 Section 5 showed TNS changing by a factor of five with -max_paths alone, and Chapter 28.3 Section 7 showed area improving 25% from a library swap — so a comparison across runs that differ in either produces deltas that measure the configuration.
Chapter 27.2 Section 6 made this exact argument about a parallelisation baseline, with workers as the one field allowed to differ. Here nothing is allowed to differ: a QoR comparison between two runs of the same flow at two commits is the whole point, so every setup field is pinned and the design is the only variable.
The function returns the differing fields rather than a boolean alone, so a refusal names what to fix.
3. The Annotation
delta = {k: after[k] - before[k] for k in METRICS}METRICS = ("wns", "tns", "total_area", "cells", "utilisation", "power_w")
before = {"wns": -0.045, "tns": -1.230, "total_area": 128456.3,
"cells": 45231, "utilisation": 0.8521, "power_w": 0.2343}
after = {"wns": -0.012, "tns": -0.310, "total_area": 131002.7,
"cells": 45980, "utilisation": 0.8712, "power_w": 0.2401}
delta = {k: after[k] - before[k] for k in METRICS}
for k, v in delta.items():
print(f"{k:<14} {v:+13.4f}")wns +0.0330
tns +0.9200
total_area +2546.4000
cells +749.0000
utilisation +0.0191
power_w +0.0058Six deltas, every one positive, and three of them are improvements while three are not. The line is correct arithmetic and an incomplete answer, and before that matters it has to survive the input.
4. Three Ways It Raises
METRICS = ("wns", "tns", "total_area", "power_w")
before = {"wns": -0.045, "tns": -1.230, "total_area": 128456.3, "power_w": 0.2343}
CASES = [
("unknown wns", {"wns": None, "tns": -0.310, "total_area": 131002.7, "power_w": 0.24}),
("missing key", {"wns": -0.012, "tns": -0.310, "total_area": 131002.7}),
("string value", {"wns": -0.012, "tns": -0.310, "total_area": "131002.7", "power_w": 0.24}),
]
for label, after in CASES:
try:
{k: after[k] - before[k] for k in METRICS}
print(f"{label:<14} no error")
except (TypeError, KeyError) as exc:
print(f"{label:<14} {type(exc).__name__}: {exc}")
print()
print("where the None comes from:", "Chapter 28.1 §9 — an unparseable report")
print("where the missing key does:", "a tool that stopped printing a field")
print("where the string does :", "a parser that forgot to convert")unknown wns TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'
missing key KeyError: 'power_w'
string value TypeError: unsupported operand type(s) for -: 'str' and 'float'
where the None comes from: Chapter 28.1 §9 — an unparseable report
where the missing key does: a tool that stopped printing a field
where the string does : a parser that forgot to convertThe first case is the one this curriculum guaranteed. Chapter 28.1 Section 9 decided that an unparseable timing report yields wns=None rather than a fabricated zero — which was the right decision, and it means the comparison raises on precisely the run somebody most needs compared.
The three failures also arrive as two different exception types from one line, so a caller catching TypeError alone misses the missing-key case.
METRICS = ("wns", "tns", "total_area", "power_w")
def compare(before, after, metrics=METRICS):
"""Every metric gets an outcome, including the ones that cannot be compared."""
deltas, unavailable = {}, {}
for k in metrics:
b, a = before.get(k), after.get(k)
if b is None or a is None:
unavailable[k] = ("missing" if k not in before or k not in after
else "unknown in one run")
continue
if not isinstance(b, (int, float)) or not isinstance(a, (int, float)):
unavailable[k] = f"non-numeric ({type(b).__name__}, {type(a).__name__})"
continue
deltas[k] = a - b
return deltas, unavailable
BEFORE = {"wns": -0.045, "tns": -1.230, "total_area": 128456.3, "power_w": 0.2343}
AFTER = {"wns": None, "tns": -0.310, "total_area": "131002.7"}
deltas, unavailable = compare(BEFORE, AFTER)
print("compared :", {k: round(v, 4) for k, v in deltas.items()})
print("unavailable :", unavailable)
print()
print("metrics asked for:", len(METRICS))
print("accounted for :", len(deltas) + len(unavailable))
print("nothing silently dropped:", len(deltas) + len(unavailable) == len(METRICS))compared : {'tns': 0.92}
unavailable : {'wns': 'unknown in one run', 'total_area': 'non-numeric (float, str)', 'power_w': 'missing'}
metrics asked for: 4
accounted for : 4
nothing silently dropped: TrueEvery metric gets an outcome, which is Chapter 26.1's accounting invariant applied to a comparison: the deltas plus the unavailable must equal the metrics asked for.
A comparison that silently omitted the three unavailable metrics would report one delta and look complete.
5. A Delta Has No Direction
BETTER_WHEN_HIGHER = {"wns", "tns"}
DELTA = {"wns": 0.0330, "tns": 0.9200, "total_area": 2546.4,
"cells": 749.0, "utilisation": 0.0191, "power_w": 0.0058}
print(f"{'metric':<14} {'delta':>12} direction")
for k, v in DELTA.items():
improved = v > 0 if k in BETTER_WHEN_HIGHER else v < 0
print(f"{k:<14} {v:>+12.4f} {'BETTER' if improved else 'worse'}")
print()
positive = sum(1 for v in DELTA.values() if v > 0)
better = sum(1 for k, v in DELTA.items()
if (v > 0 if k in BETTER_WHEN_HIGHER else v < 0))
print("deltas that are positive:", positive, "of", len(DELTA))
print("deltas that are better :", better, "of", len(DELTA))
print("sign and direction agree:", positive == better)metric delta direction
wns +0.0330 BETTER
tns +0.9200 BETTER
total_area +2546.4000 worse
cells +749.0000 worse
utilisation +0.0191 worse
power_w +0.0058 worse
deltas that are positive: 6 of 6
deltas that are better : 2 of 6
sign and direction agree: FalseSix positive deltas, two improvements.
A table of signed numbers invites the reading "positive is good", and for four of these six it is the opposite. Which way is better is a property of the metric, not of the number, and it has to be declared:
DIRECTION = {"wns": "higher", "tns": "higher", "total_area": "lower",
"cells": "lower", "utilisation": "lower", "power_w": "lower"}
def verdict(metric, delta):
if delta == 0:
return "unchanged"
better = delta > 0 if DIRECTION[metric] == "higher" else delta < 0
return "better" if better else "worse"
DELTA = {"wns": 0.0330, "tns": -0.2000, "total_area": -1200.0, "cells": 0.0}
for k, v in DELTA.items():
print(f"{k:<13} {v:>+11.4f} better when {DIRECTION[k]:<6} -> {verdict(k, v)}")
print()
print("every metric has a declared direction:", set(DELTA) <= set(DIRECTION))
missing = {"congestion"} - set(DIRECTION)
print("an undeclared metric would:", "KeyError in verdict() — loudly")
print("which is better than :", "silently assuming higher is good")wns +0.0330 better when higher -> better
tns -0.2000 better when higher -> worse
total_area -1200.0000 better when lower -> better
cells +0.0000 better when lower -> unchanged
every metric has a declared direction: True
an undeclared metric would: KeyError in verdict() — loudly
which is better than : silently assuming higher is goodThree decisions worth copying.
The direction table is data, so adding a metric means adding a row and forgetting to means a KeyError rather than a wrong verdict.
Utilisation is declared better-lower, which is a judgement rather than a fact — a design at 60% has room and one at 95% does not, so rising utilisation is usually bad news. A team targeting density might declare it the other way, and the point is that the declaration exists somewhere a reviewer can see it.
Zero is unchanged, not better. Folding it into either direction overstates the number of movements, which matters when Chapter 28.6 counts them.
6. Percentages Lie Near Zero
BEFORE = {"wns": -0.045, "tns": -1.230, "total_area": 128456.3,
"cells": 45231, "utilisation": 0.8521, "power_w": 0.2343}
AFTER = {"wns": -0.012, "tns": -0.310, "total_area": 131002.7,
"cells": 45980, "utilisation": 0.8712, "power_w": 0.2401}
print(f"{'metric':<14} {'before':>13} {'after':>13} {'absolute':>12} {'percent':>9}")
for k in BEFORE:
b, a = BEFORE[k], AFTER[k]
pct = (a - b) / abs(b) * 100
print(f"{k:<14} {b:>13.4f} {a:>13.4f} {a - b:>+12.4f} {pct:>+8.2f}%")
print()
print("largest percentage change:", "tns, +74.80%")
print("largest absolute change :", "total_area, +2546.4")
print("wns moved by :", "33 picoseconds")
print("and shows as :", "+73.33%")metric before after absolute percent
wns -0.0450 -0.0120 +0.0330 +73.33%
tns -1.2300 -0.3100 +0.9200 +74.80%
total_area 128456.3000 131002.7000 +2546.4000 +1.98%
cells 45231.0000 45980.0000 +749.0000 +1.66%
utilisation 0.8521 0.8712 +0.0191 +2.24%
power_w 0.2343 0.2401 +0.0058 +2.48%
largest percentage change: tns, +74.80%
largest absolute change : total_area, +2546.4
wns moved by : 33 picoseconds
and shows as : +73.33%A thirty-three picosecond change shows as +73.33%.
Sorted by percentage, the timing metrics dominate the report and area looks like a rounding error. Sorted by absolute value, area dominates and timing vanishes. Both orderings are arithmetically correct and only one of them is informative, and which one depends on the metric.
The reason is structural rather than a quirk of these numbers: a percentage is a ratio to the previous value, and WNS is always near zero — that is what WNS is. A design closing timing goes from −0.045 to −0.005, which is a 89% improvement and 40 picoseconds, and the same 40 picoseconds on a design at −2.0 is 2%.
So the rule is per metric:
RELATIVE_OK = {"total_area", "cells", "utilisation", "power_w"}
ROWS = [("wns", -0.045, -0.012, "ns"), ("tns", -1.230, -0.310, "ns"),
("total_area", 128456.3, 131002.7, "um^2"), ("cells", 45231, 45980, "")]
for name, b, a, unit in ROWS:
absolute = a - b
if name in RELATIVE_OK:
print(f"{name:<12} {absolute:+12.1f} {unit:<5} ({(a - b) / abs(b) * 100:+.2f}%)")
else:
print(f"{name:<12} {absolute:+12.4f} {unit:<5} (no percentage — near zero)")
print()
print("metrics reported as a percentage:", sorted(RELATIVE_OK))
print("metrics reported absolutely :", sorted({r[0] for r in ROWS} - RELATIVE_OK))
print("the rule :", "a percentage needs a baseline far from zero")wns +0.0330 ns (no percentage — near zero)
tns +0.9200 ns (no percentage — near zero)
total_area +2546.4 um^2 (+1.98%)
cells +749.0 (+1.66%)
metrics reported as a percentage: ['cells', 'power_w', 'total_area', 'utilisation']
metrics reported absolutely : ['tns', 'wns']
the rule : a percentage needs a baseline far from zeroTiming in absolute units, everything else as a percentage. A +0.033 ns is a figure a timing engineer acts on; a +73% is one they have to convert back.
TNS is the borderline case and is listed with timing here. It is a sum and can be large, so a percentage is sometimes meaningful — and it is zero on a design that meets timing, which makes the ratio undefined. Chapter 28.6 handles that explicitly rather than relying on the value staying away from zero.
7. Metrics Move Together
DELTA = {"wns": 0.0330, "tns": 0.9200, "total_area": 2546.4,
"cells": 749.0, "utilisation": 0.0191, "power_w": 0.0058}
DIRECTION = {"wns": "higher", "tns": "higher", "total_area": "lower",
"cells": "lower", "utilisation": "lower", "power_w": "lower"}
better = [k for k, v in DELTA.items()
if (v > 0 if DIRECTION[k] == "higher" else v < 0)]
worse = [k for k in DELTA if k not in better]
print("improved:", better)
print("worsened:", worse)
print()
print("six independent findings?", False)
print("what actually happened :", "the tool traded area for timing")
print("how many decisions :", 1)
print()
EXPECTED = [("timing improved, area grew", "the optimiser upsized and buffered"),
("area grew, utilisation grew", "the same cells, in the same floorplan"),
("cells grew, power grew", "more cells switch")]
for pattern, why in EXPECTED:
print(f" {pattern:<30} {why}")improved: ['wns', 'tns']
worsened: ['total_area', 'cells', 'utilisation', 'power_w']
six independent findings? False
what actually happened : the tool traded area for timing
how many decisions : 1
timing improved, area grew the optimiser upsized and buffered
area grew, utilisation grew the same cells, in the same floorplan
cells grew, power grew more cells switchSix metrics moved and one thing happened.
The optimiser upsized cells and inserted buffers to close timing; that grows the cell count, which grows the area, which grows the utilisation in a fixed floorplan, and more cells switching grows the power. Four "regressions" and two "improvements" are one trade.
A comparison that lists six independent deltas invites six investigations, and this is Chapter 26.7's bucketing argument in a new setting: the raw list is the symptom list and the grouping is the finding. Naming the expected co-movements turns a wall of numbers into one sentence.
The practically useful form is to flag the unexpected combinations instead:
def surprising(delta):
out = []
if delta["wns"] > 0 and delta["total_area"] < 0:
out.append("timing improved and area shrank — unusually good, check the setup")
if delta["wns"] < 0 and delta["total_area"] > 0:
out.append("timing worsened and area grew — paid for nothing")
if delta["cells"] > 0 and delta["power_w"] < 0:
out.append("more cells and less power — check the activity assumptions")
return out
EXPECTED_TRADE = {"wns": 0.033, "total_area": 2546.4, "cells": 749.0, "power_w": 0.0058}
BAD = {"wns": -0.020, "total_area": 1800.0, "cells": 500.0, "power_w": 0.004}
ODD = {"wns": 0.050, "total_area": -900.0, "cells": -200.0, "power_w": -0.002}
for label, d in (("expected trade", EXPECTED_TRADE), ("paid for nothing", BAD),
("too good", ODD)):
print(f"{label:<18} {surprising(d) or ['nothing surprising']}")expected trade ['nothing surprising']
paid for nothing ['timing worsened and area grew — paid for nothing']
too good ['timing improved and area shrank — unusually good, check the setup']The expected trade produces no finding, which is the point: a comparison should be quiet when the design behaved the way designs behave, and loud when it did not.
The third case deserves its wording. Timing improving while area shrinks is possible — a better floorplan, a fixed constraint — and it is also the signature of a comparison across two different setups that Section 2's gate was supposed to catch. Unusually good results are worth the same scrutiny as unusually bad ones, and almost nothing in a flow checks for them.
8. More Than Two Runs
SWEEP = [
("effort=low", {"wns": -0.180, "total_area": 121003.4, "power_w": 0.2201}),
("effort=medium", {"wns": -0.072, "total_area": 125440.1, "power_w": 0.2288}),
("effort=high", {"wns": -0.012, "total_area": 131002.7, "power_w": 0.2401}),
("effort=ultra", {"wns": -0.008, "total_area": 138771.2, "power_w": 0.2620}),
]
base = SWEEP[0][1]
print(f"{'run':<15} {'wns':>9} {'area':>12} {'power':>8} {'d-wns':>8} {'d-area':>10}")
for name, m in SWEEP:
print(f"{name:<15} {m['wns']:>+9.3f} {m['total_area']:>12.1f} {m['power_w']:>8.4f} "
f"{m['wns'] - base['wns']:>+8.3f} {m['total_area'] - base['total_area']:>+10.1f}")
print()
print("runs :", len(SWEEP))
print("pairwise deltas :", len(SWEEP) * (len(SWEEP) - 1) // 2)
print("useful deltas :", len(SWEEP) - 1, "— each against the common baseline")run wns area power d-wns d-area
effort=low -0.180 121003.4 0.2201 +0.000 +0.0
effort=medium -0.072 125440.1 0.2288 +0.108 +4436.7
effort=high -0.012 131002.7 0.2401 +0.168 +9999.3
effort=ultra -0.008 138771.2 0.2620 +0.172 +17767.8
runs : 4
pairwise deltas : 6
useful deltas : 3 — each against the common baselineA QoR comparison is usually presented as two runs, and the question that actually gets asked is a sweep: which effort level, which floorplan, which of four candidate constraint files. Chapter 28.5's pairwise function generalises by fixing one run as the baseline rather than by comparing every pair.
Six pairwise comparisons over four runs is not six findings — it is three deltas against a common reference plus three derivable from them. Presenting all six invites the reader to find a pattern in redundant numbers.
What a sweep reveals that a pair cannot is the shape of the trade:
SWEEP = [
("low", -0.180, 121003.4), ("medium", -0.072, 125440.1),
("high", -0.012, 131002.7), ("ultra", -0.008, 138771.2),
]
print(f"{'step':<16} {'d-wns (ns)':>11} {'d-area (um^2)':>14} {'um^2 per ps':>13}")
for (n0, w0, a0), (n1, w1, a1) in zip(SWEEP, SWEEP[1:]):
dw, da = w1 - w0, a1 - a0
cost = da / (dw * 1000) if dw else float("inf")
print(f"{n0 + ' -> ' + n1:<16} {dw:>+11.3f} {da:>+14.1f} {cost:>13.0f}")
print()
print("the marginal cost of a picosecond rises:", True)
print("the last step buys :", "4 ps for 7769 um^2")
print("which is :", "about 47x the cost of the first step")step d-wns (ns) d-area (um^2) um^2 per ps
low -> medium +0.108 +4436.7 41
medium -> high +0.060 +5562.6 93
high -> ultra +0.004 +7768.5 1942
the marginal cost of a picosecond rises: True
the last step buys : 4 ps for 7769 um^2
which is : about 47x the cost of the first stepThat output stops before the three statements; the real output follows.
step d-wns (ns) d-area (um^2) um^2 per ps
low -> medium +0.108 +4436.7 41
medium -> high +0.060 +5562.6 93
high -> ultra +0.004 +7768.5 1942
the marginal cost of a picosecond rises: True
the last step buys : 4 ps for 7769 um^2
which is : about 47x the cost of the first step41, 93 and 1942 square microns per picosecond. The last step costs twenty-one times what the one before it did, which is the diminishing-returns curve every optimiser has and which no pair of runs can show.
That number is the one a decision is made on. "High effort gives −0.012 and ultra gives −0.008" is two facts; "the last four picoseconds cost 7,769 µm²" is a choice somebody can make.
Three properties of the sweep comparison are worth copying.
One baseline, not every pair. SWEEP[0] is the reference and every row is relative to it, so the table has n-1 meaningful rows rather than n(n-1)/2 redundant ones.
The marginal view is between adjacent runs, which is a different computation from the cumulative one and answers the different question — what did this step buy? rather than where are we?
The derived ratio is the output. Chapter 28.5 Section 6 argued against percentages near zero; a ratio of two deltas in their own units sidesteps that entirely and is more actionable than either column.
SWEEP = {"low": {"wns": -0.180, "area": 121003.4},
"medium": {"wns": -0.072, "area": 125440.1},
"high": {"wns": -0.012, "area": 131002.7},
"ultra": {"wns": -0.008, "area": 138771.2}}
TARGET_WNS = -0.020
meeting = [n for n, m in SWEEP.items() if m["wns"] >= TARGET_WNS]
cheapest = min(meeting, key=lambda n: SWEEP[n]["area"]) if meeting else None
print("target wns :", f"{TARGET_WNS:+.3f}")
print("runs that meet :", meeting)
print("cheapest of them:", cheapest)
print("its area :", SWEEP[cheapest]["area"] if cheapest else None)
print()
best = min(SWEEP, key=lambda n: SWEEP[n]["wns"] and -SWEEP[n]["wns"])
print("best timing :", max(SWEEP, key=lambda n: SWEEP[n]["wns"]))
print("area it costs :", SWEEP[max(SWEEP, key=lambda n: SWEEP[n]['wns'])]["area"])
print("area saved by picking the cheapest that meets:",
round(SWEEP["ultra"]["area"] - SWEEP[cheapest]["area"], 1))target wns : -0.020
runs that meet : ['high', 'ultra']
cheapest of them: high
its area : 131002.7
best timing : ultra
area it costs : 138771.2
area saved by picking the cheapest that meets: 7768.5The cheapest run that meets the target, not the best run. ultra wins on timing and costs 7,768 µm² more than high, which also meets the target — so the sweep's output is not a maximum but a constrained minimum.
That is what turns a comparison into a decision, and it is the shape Chapter 28.6's gate needs: a target to meet rather than a direction to improve.
9. The Comparison, Assembled
import json
METRICS = ("wns", "tns", "total_area", "cells", "utilisation", "power_w")
DIRECTION = {"wns": "higher", "tns": "higher", "total_area": "lower",
"cells": "lower", "utilisation": "lower", "power_w": "lower"}
RELATIVE_OK = {"total_area", "cells", "utilisation", "power_w"}
PINNED = ("library", "corner", "effort", "max_paths")
UNITS = {"wns": "ns", "tns": "ns", "total_area": "um^2", "cells": "",
"utilisation": "", "power_w": "W"}
class NotComparable(Exception):
"""The two runs differ in setup, so no delta means anything."""
def compare_qor(before, after):
differ = sorted(k for k in PINNED
if before["context"].get(k) != after["context"].get(k))
if differ:
raise NotComparable(f"contexts differ in {differ} — compare the setup, not the design")
rows, unavailable = [], {}
for name in METRICS:
b, a = before["metrics"].get(name), after["metrics"].get(name)
if b is None or a is None or not isinstance(b, (int, float)) \
or not isinstance(a, (int, float)):
unavailable[name] = "unknown or missing in one run"
continue
delta = a - b
if delta == 0:
direction = "unchanged"
else:
direction = "better" if (delta > 0 if DIRECTION[name] == "higher"
else delta < 0) else "worse"
percent = None
if name in RELATIVE_OK and b != 0:
percent = round(delta / abs(b) * 100, 2)
rows.append({"metric": name, "before": b, "after": a,
"delta": round(delta, 6), "percent": percent,
"direction": direction, "unit": UNITS[name]})
return {"rows": rows, "unavailable": unavailable,
"better": [r["metric"] for r in rows if r["direction"] == "better"],
"worse": [r["metric"] for r in rows if r["direction"] == "worse"],
"accounted": len(rows) + len(unavailable) == len(METRICS)}
CTX = {"library": "tcbn05_lvt", "corner": "ss0p675v125c", "effort": "high",
"max_paths": 100}
BEFORE = {"context": CTX, "metrics": {"wns": -0.045, "tns": -1.230,
"total_area": 128456.3, "cells": 45231,
"utilisation": 0.8521, "power_w": 0.2343}}
AFTER = {"context": dict(CTX), "metrics": {"wns": -0.012, "tns": -0.310,
"total_area": 131002.7, "cells": 45980,
"utilisation": 0.8712, "power_w": None}}
result = compare_qor(BEFORE, AFTER)
print(f"{'metric':<13} {'delta':>12} {'pct':>8} {'dir':<9} unit")
for r in result["rows"]:
pct = f"{r['percent']:+.2f}%" if r["percent"] is not None else "—"
print(f"{r['metric']:<13} {r['delta']:>+12.4f} {pct:>8} {r['direction']:<9} {r['unit']}")
print()
print("better :", result["better"])
print("worse :", result["worse"])
print("unavailable :", result["unavailable"])
print("accounted :", result["accounted"])
print()
try:
compare_qor(BEFORE, {"context": dict(CTX, library="other"), "metrics": {}})
except NotComparable as exc:
print("gate:", exc)metric delta pct dir unit
wns +0.0330 — better ns
tns +0.9200 — better ns
total_area +2546.4000 +1.98% worse um^2
cells +749.0000 +1.66% worse
utilisation +0.0191 +2.24% worse
better : ['wns', 'tns']
worse : ['total_area', 'cells', 'utilisation']
unavailable : {'power_w': 'unknown or missing in one run'}
accounted : True
gate: contexts differ in ['library'] — compare the setup, not the designSix decisions, and the first is the one that makes everything after it mean something.
The gate raises before any arithmetic. A NotComparable naming the fields is the only honest response to two different experiments, and it cannot be ignored the way a warning can.
Every metric is accounted for, as a row or as unavailable, and the accounted flag asserts it — Chapter 26.1's invariant, one more time.
percent is None for timing and for a zero baseline, so the formatting shows an em dash rather than a misleading number or a ZeroDivisionError.
direction is derived from a declared table, and an undeclared metric raises rather than defaulting.
Zero is unchanged, kept out of both lists.
The unit travels with each row, per Chapter 28.1 Section 8, so the rendered table cannot be misread.
10. Predict, Run, Observe
B = {"wns": -0.045, "area": 128456.3}
A = {"wns": -0.012, "area": 131002.7}
DIR = {"wns": "higher", "area": "lower"}
# A
print("A:", {k: round(A[k] - B[k], 4) for k in B})
# B
print("B:", all(A[k] - B[k] > 0 for k in B))
# C
print("C:", [(k, "better" if ((A[k] - B[k] > 0) if DIR[k] == "higher"
else (A[k] - B[k] < 0)) else "worse") for k in B])
# D
print("D:", round((A["wns"] - B["wns"]) / abs(B["wns"]) * 100, 2),
round((A["area"] - B["area"]) / abs(B["area"]) * 100, 2))
# E
print("E:", round(A["wns"] - B["wns"], 4))
# F
try:
None - 1.0
except TypeError as exc:
print("F:", type(exc).__name__)
# G
try:
{"a": 1}["b"]
except KeyError as exc:
print("G:", type(exc).__name__, exc)
# H
try:
1.0 / abs(0.0)
except ZeroDivisionError:
print("H: ZeroDivisionError")The verified output:
A: {'wns': 0.033, 'area': 2546.4}
B: True
C: [('wns', 'better'), ('area', 'worse')]
D: 73.33 1.98
E: 0.033
F: TypeError
G: KeyError 'b'
H: ZeroDivisionErrorA — the annotation's deltas.
B — both positive.
C — and only one is an improvement.
D — the percentage says timing moved 37 times more than area.
E — in nanoseconds it moved 33 picoseconds.
F — an unknown value raises, and Chapter 28.1 produces those by design.
G — a missing key raises differently, so one except clause is not enough.
H — and a zero baseline makes the percentage impossible.
11. Exercises
Exercise 1 — two records. Produce context and metrics for two of your own runs.
Exercise 2 — the gate. List the fields your flow must pin, and say which chapter justifies each.
Exercise 3 — refuse, do not warn. Argue in two sentences for raising rather than annotating.
Exercise 4 — three failures. Reproduce the None, missing-key and string cases.
Exercise 5 — where the None comes from. Trace it back to Chapter 28.1's decision and say why that decision was right.
Exercise 6 — two exception types. Say what a caller catching only TypeError misses.
Exercise 7 — account for everything. Add the accounted check and break it deliberately.
Exercise 8 — six positive, two better. Build the table and count both.
Exercise 9 — direction as data. Write your own table and show an undeclared metric raising.
Exercise 10 — defend utilisation. Say which direction you declare it and why.
Exercise 11 — zero is unchanged. Say what folding it into a direction does to Chapter 28.6's counts.
Exercise 12 — the two orderings. Sort your own comparison by percentage and by absolute value, and describe the difference.
Exercise 13 — why WNS is near zero. Explain the structural reason a percentage misleads there.
Exercise 14 — per-metric rule. Decide which of your metrics get percentages.
Exercise 15 — the zero baseline. Construct a TNS that was zero and is now negative, and handle it.
Exercise 16 — one trade. Take a real comparison and group the moved metrics into the decisions behind them.
Exercise 17 — the expected co-movements. Write down three for your flow.
Exercise 18 — quiet when normal. Implement the surprising-combination check and show it silent on an expected trade.
Exercise 19 — too good. Say why an unusually good result deserves the same scrutiny, and which gate it usually means failed.
Exercise 20 — a sweep. Compare four runs against a common baseline rather than pairwise, and say how many rows each gives.
Exercise 21 — the marginal view. Compute the cost per picosecond between adjacent runs of your own sweep.
Exercise 22 — diminishing returns. Report the ratio between your first and last step.
Exercise 23 — a ratio beats a percentage. Say why a cost-per-picosecond sidesteps Section 6's problem.
Exercise 24 — cheapest that meets. Pick a target and find the cheapest run meeting it rather than the best.
**Exercise 25 — the rendered table.**Exercise 20 — the rendered table. Add units and em dashes, and say what each prevents.
12. Summary
- A QoR record splits into
contextandmetrics, and the comparison treats them differently: one is checked for equality, the other is subtracted. - The comparability gate comes first and raises — a delta across different libraries, corners or report switches measures the setup.
- The annotation's comprehension fails three ways: a
None, a missing key and a non-numeric value, as two different exception types. - Chapter 28.1 guarantees the
Noneby refusing to fabricate a WNS, so the comparison raises on the run that most needs comparing. - Give every metric an outcome — a delta or a reason it is unavailable — and assert that they add up.
- A raw delta carries no direction. Six positive deltas were two improvements.
- Declare the direction per metric as data, so an undeclared metric raises instead of defaulting.
- Zero is
unchanged, not better and not worse. - A percentage is a ratio to the previous value, and WNS is always near zero — 33 picoseconds showed as +73.33%.
- Report timing absolutely and area, cells, utilisation and power as percentages.
- A zero baseline makes the percentage a
ZeroDivisionError, and that case — TNS was zero, now it is not — is the most important regression there is. - Metrics move together: six movements were one trade of area for timing.
- Flag the unexpected combinations instead of listing every delta, so the comparison is quiet when the design behaved normally.
- An unusually good result deserves the same scrutiny as a bad one, and usually means the gate in Section 2 was not applied.
- A sweep of four runs gives three meaningful deltas, not six — fix one baseline rather than comparing every pair.
- The marginal view between adjacent runs is a different computation and answers what each step bought.
- Cost per picosecond rose from 41 to 1942 µm² across one effort sweep — a curve no pair of runs can show.
- A ratio of two deltas in their own units sidesteps the percentage problem entirely and is more actionable than either column.
- The output of a sweep is the cheapest run that meets the target, not the best run — a constrained minimum rather than a maximum.
- Carry the unit on every row, so the rendered table cannot be misread.
The last chapter of this module turns this comparison into an automatic verdict, which needs one more thing: a threshold that knows the difference between a regression and noise.
Where this fits
Part of the Python curriculum.