Python Part II · Module 35 · Industry VLSI Automation Projects
A Synthesis QoR Comparator
The last tool in the module is the one a reviewer reads. Two synthesis runs, a list of what moved, and a verdict — which sounds like subtraction and is not, because the metrics disagree about which direction is good, one of them cannot be expressed as a percentage at all, and the most dangerous change a design can undergo makes every single number improve.
Advanced26 min readPythonPart IIAutomationSynthesisQoR
Part II · Module 35 · Chapter 35.9 · A Synthesis QoR Comparator
1. Two Runs, One Question
QUESTIONS = [
("did anything get worse?", "a reviewer", "block the merge"),
("did the thing I changed help?", "the author", "keep or revert"),
("is this ready for sign-off?", "a lead", "schedule"),
("what moved by how much?", "nobody", "it is a means, not an end"),
]
print(f"{'the question':<34} {'who asks':<12} what they do with it")
for q, who, action in QUESTIONS:
print(f"{q:<34} {who:<12} {action}")
print()
print("questions that need a verdict :", 3)
print("questions that need a table :", 1)
print("so the output is :", "a verdict, with the table under it")
print("and a tool that only tabulates:", "leaves the hard part to a human")the question who asks what they do with it
did anything get worse? a reviewer block the merge
did the thing I changed help? the author keep or revert
is this ready for sign-off? a lead schedule
what moved by how much? nobody it is a means, not an end
questions that need a verdict : 3
questions that need a table : 1
so the output is : a verdict, with the table under it
and a tool that only tabulates: leaves the hard part to a humanThree of four questions want a verdict and the fourth is nobody's. A comparator that produces a table of percentages has done the easy part and left the reviewer to decide which of twelve numbers matter and which way is good — which is exactly where review mistakes happen.
2. The Annotation
PARTS = [
("--before run_a/", "the reference run's directory", "a whole run"),
("--after run_b/", "the candidate run's directory", "a whole run"),
]
print(f"{'part':<20} {'meaning':<34} note")
for part, meaning, note in PARTS:
print(f"{part:<20} {meaning:<34} {note}")
print()
READS = [
("qor.rpt", "area, leakage, instances", "the QoR numbers"),
("timing.rpt", "WNS, TNS", "35.2 already parses this"),
("hierarchy.rpt", "the module list", "the structural check — §7"),
]
print(f"{'file in each run dir':<18} {'what it gives':<28} why")
for f, gives, why in READS:
print(f"{f:<18} {gives:<28} {why}")
print()
print("it takes DIRECTORIES, not files:", True)
print("because a QoR comparison needs :", "more than the QoR report — §7")
print("and the third file is the one :", "that catches the worst change")part meaning note
--before run_a/ the reference run's directory a whole run
--after run_b/ the candidate run's directory a whole run
file in each run dir what it gives why
qor.rpt area, leakage, instances the QoR numbers
timing.rpt WNS, TNS 35.2 already parses this
hierarchy.rpt the module list the structural check — §7
it takes DIRECTORIES, not files: True
because a QoR comparison needs : more than the QoR report — §7
and the third file is the one : that catches the worst changeThe annotation takes directories rather than report files, and that is the
design decision the chapter turns on. A comparator pointed at two qor.rpt
files can only compare QoR, and Section 7 shows that QoR alone cannot detect the
most serious thing that can happen between two runs.
3. The First Version
import re
BEFORE = """\
Report : qor
Design : axi_fifo
Total cell area 18432.0
Total leakage power 412.0
Number of instances 24110
Critical path slack -0.173
"""
AFTER = BEFORE.replace("18432.0", "16998.0") \
.replace("412.0", "381.0") \
.replace("24110", "21004") \
.replace("-0.173", "0.042")
ROW = re.compile(r"^\s{2}(.+?)\s{2,}(-?\d+\.?\d*)\s*$", re.M)
def parse(text):
return {m.group(1).strip(): float(m.group(2)) for m in ROW.finditer(text)}
before, after = parse(BEFORE), parse(AFTER)
print("v1 — subtract and print the percentage")
print(f" {'metric':<26} {'before':>10} {'after':>10} {'change':>10}")
for key in before:
b, a = before[key], after[key]
pct = (a - b) / abs(b) * 100 if b else float("nan")
print(f" {key:<26} {b:>10.3f} {a:>10.3f} {pct:>+9.2f}%")
print()
print("metrics compared:", len(before))
print("what the table says:", "four numbers changed")
print("what it does not say:", ["which way is good", "which changes matter",
"what the verdict is"])v1 — subtract and print the percentage
metric before after change
Total cell area 18432.000 16998.000 -7.78%
Total leakage power 412.000 381.000 -7.52%
Number of instances 24110.000 21004.000 -12.88%
Critical path slack -0.173 0.042 +124.28%
metrics compared: 4
what the table says: four numbers changed
what it does not say: ['which way is good', 'which changes matter', 'what the verdict is']Three of the four numbers are informative and the fourth is nonsense.
+124.28% for a slack that went from failing to passing is the subject of
Section 5; the missing direction column is Section 4.
4. The Limitation: Direction Is Per Metric
METRICS = [
# name, before, after, +1 if higher is better
("area_um2", 18432.0, 16998.0, -1),
("leakage_uw", 412.0, 381.0, -1),
("instances", 24110.0, 21004.0, -1),
("max_fanout", 64.0, 48.0, -1),
("wns_ns", -0.173, 0.042, +1),
("tns_ns", -2.410, 0.000, +1),
]
print(f"{'metric':<14} {'before':>10} {'after':>10} {'change':>10} "
f"{'wants':>7} better?")
for name, b, a, direction in METRICS:
pct = (a - b) / abs(b) * 100 if b else float("nan")
better = (a > b) if direction > 0 else (a < b)
wants = "higher" if direction > 0 else "lower"
print(f"{name:<14} {b:>10.3f} {a:>10.3f} {pct:>+9.2f}% {wants:>7} "
f"{better}")
lower = sum(1 for *_, d in METRICS if d < 0)
print()
print("metrics that want a LOWER value :", lower)
print("metrics that want a HIGHER value:", len(METRICS) - lower)
print()
print("so '-7.78%' is good for area :", True)
print("and would be bad for wns :", True)
print("the column a percentage table ")
print(" cannot supply :", "which way is good")metric before after change wants better?
area_um2 18432.000 16998.000 -7.78% lower True
leakage_uw 412.000 381.000 -7.52% lower True
instances 24110.000 21004.000 -12.88% lower True
max_fanout 64.000 48.000 -25.00% lower True
wns_ns -0.173 0.042 +124.28% higher True
tns_ns -2.410 0.000 +100.00% higher True
metrics that want a LOWER value : 4
metrics that want a HIGHER value: 2
so '-7.78%' is good for area : True
and would be bad for wns : True
the column a percentage table
cannot supply : which way is goodFour metrics want a lower number and two want a higher one, so a sign in
the change column carries no information on its own. The better? column is
the one a reviewer actually reads, and it can only be computed if the direction
is declared per metric in a table the tool owns.
That is the same move Chapter 35.5 made in storing counts rather than percentages: the raw fact is the pair of values, and every interpretation — direction, significance, verdict — is derived from a declaration the tool carries rather than from the number's shape.
5. A Percentage Is Meaningless Across Zero
CASES = [
("area improves", 18432.0, 16998.0),
("wns, fail to pass", -0.173, 0.042),
("wns, worse but still passing", 0.210, 0.042),
("wns, pass to fail", 0.042, -0.173),
("tns reaches zero", -2.410, 0.000),
("tns from zero", 0.000, -0.310),
]
print(f"{'case':<30} {'before':>9} {'after':>9} {'percent':>12} "
f"{'absolute':>10}")
for label, b, a in CASES:
pct = f"{(a - b) / abs(b) * 100:+.2f}%" if b else "undefined"
print(f"{label:<30} {b:>9.3f} {a:>9.3f} {pct:>12} {a - b:>+10.3f}")
print()
print("cases where the percentage is undefined:", 1)
print("cases where it crosses zero :", 3)
print("cases where it is interpretable :", 2)
print()
print("wns -0.173 -> +0.042 as a percentage:", "+124.28%")
print("what that would have to mean :", "124% of 0.173 ns")
print("what actually happened :", "it went from FAILING to PASSING")
print("the honest number :", "+0.215 ns")
print()
print("so slack is reported as :", "an absolute delta, in ns")
print("and area as :", "a percentage")case before after percent absolute
area improves 18432.000 16998.000 -7.78% -1434.000
wns, fail to pass -0.173 0.042 +124.28% +0.215
wns, worse but still passing 0.210 0.042 -80.00% -0.168
wns, pass to fail 0.042 -0.173 -511.90% -0.215
tns reaches zero -2.410 0.000 +100.00% +2.410
tns from zero 0.000 -0.310 undefined -0.310
cases where the percentage is undefined: 1
cases where it crosses zero : 3
cases where it is interpretable : 2
wns -0.173 -> +0.042 as a percentage: +124.28%
what that would have to mean : 124% of 0.173 ns
what actually happened : it went from FAILING to PASSING
the honest number : +0.215 ns
so slack is reported as : an absolute delta, in ns
and area as : a percentageFour of six percentages are undefined or uninterpretable. The last row
divides by zero; three others cross zero, and a percentage change across zero
has no meaning at all — -511.90% for a design that went from 42 ps of margin
to 173 ps of violation is a number that cannot be compared with anything.
The absolute column is right in every row. +0.215 ns and -0.215 ns are the
same magnitude in opposite directions, which is what a reviewer needs, and they
stay meaningful whether or not the sign flips.
So the rule is per metric again: a metric with a natural zero and a fixed sign takes a percentage; a signed metric takes an absolute delta. Area, leakage and instance counts are always positive and a 7.78% reduction is a sentence anybody can act on. Slack is signed, and 215 picoseconds is the only honest way to say what changed.
6. The Architecture
Parse three reports, declare each metric, then decide
Note that the verdict has four values and the fourth is SUSPECT. Better,
worse and mixed are all statements about QoR; SUSPECT is the structural
comparison overriding them, and Section 7 is why it has to be able to.
7. A Design That Lost Logic Improves Everywhere
METRICS = [
("area_um2", 18432.0, 12100.0, -1),
("leakage_uw", 412.0, 260.0, -1),
("instances", 24110.0, 14880.0, -1),
("wns_ns", -0.173, 0.880, +1),
("tns_ns", -2.410, 0.000, +1),
]
print(f"{'metric':<14} {'before':>10} {'after':>10} {'change':>10} better?")
improved = 0
for name, b, a, d in METRICS:
better = (a > b) if d > 0 else (a < b)
improved += better
pct = (a - b) / abs(b) * 100 if b else float("nan")
print(f"{name:<14} {b:>10.3f} {a:>10.3f} {pct:>+9.2f}% {better}")
print()
print("metrics improved :", improved, "of", len(METRICS))
print("metrics that got worse :", len(METRICS) - improved)
print("the QoR verdict would be :", "a large improvement")
print()
BEFORE_MODULES = {"axi_fifo", "fifo_ctrl", "fifo_mem", "axi_slave_if", "crc8"}
AFTER_MODULES = {"axi_fifo", "fifo_ctrl", "fifo_mem", "axi_slave_if"}
print("modules in the before netlist:", len(BEFORE_MODULES))
print("modules in the after netlist :", len(AFTER_MODULES))
print("missing :", sorted(BEFORE_MODULES - AFTER_MODULES))
print("appeared :", sorted(AFTER_MODULES - BEFORE_MODULES))
print()
print("the cause of every 'improvement':", "a deleted block")
print("QoR metrics that detect it :", 0)
print("the only detector :", "the module list")metric before after change better?
area_um2 18432.000 12100.000 -34.35% True
leakage_uw 412.000 260.000 -36.89% True
instances 24110.000 14880.000 -38.28% True
wns_ns -0.173 0.880 +608.67% True
tns_ns -2.410 0.000 +100.00% True
metrics improved : 5 of 5
metrics that got worse : 0
the QoR verdict would be : a large improvement
modules in the before netlist: 5
modules in the after netlist : 4
missing : ['crc8']
appeared : []
the cause of every 'improvement': a deleted block
QoR metrics that detect it : 0
the only detector : the module listFive of five metrics improved and the design lost a block. Area down 34%,
leakage down 37%, instances down 38%, timing from failing to 880 ps of margin —
every number moves the way everybody wants, and the reason is that crc8 is not
in the netlist any more.
This is the module's thesis arriving one last time, and in its purest form. Every chapter here has found a report that was confidently wrong; this one is a report that is confidently wrong in the direction people want to believe, which is the hardest kind to catch. A reviewer looking at that table has no reason to doubt it, and the author has every reason not to.
BEFORE = {"axi_fifo": 1, "fifo_ctrl": 1, "fifo_mem": 4, "axi_slave_if": 1,
"crc8": 2}
AFTER = {"axi_fifo": 1, "fifo_ctrl": 1, "fifo_mem": 4, "axi_slave_if": 1}
AFTER_RENAMED = {"axi_fifo": 1, "fifo_ctrl": 1, "fifo_mem": 4,
"axi_slave_if": 1, "crc8_v2": 2}
AFTER_FEWER = {"axi_fifo": 1, "fifo_ctrl": 1, "fifo_mem": 2,
"axi_slave_if": 1, "crc8": 2}
def compare_hierarchy(before, after):
missing = sorted(set(before) - set(after))
added = sorted(set(after) - set(before))
changed = sorted(m for m in set(before) & set(after)
if before[m] != after[m])
return {"missing": missing, "added": added, "count_changed": changed}
CASES = [("a module vanished", AFTER),
("a module was renamed", AFTER_RENAMED),
("fewer instances of one module", AFTER_FEWER)]
print(f"{'case':<32} {'missing':<12} {'added':<12} instance count changed")
for label, after in CASES:
r = compare_hierarchy(BEFORE, after)
print(f"{label:<32} {str(r['missing']):<12} {str(r['added']):<12} "
f"{r['count_changed']}")
print()
print("all three are structural changes:", True)
print("all three are invisible in QoR :", True)
print("the rename looks like :", "one missing and one added")
print("and that pairing is :", "exactly what a reviewer must see")case missing added instance count changed
a module vanished ['crc8'] [] []
a module was renamed ['crc8'] ['crc8_v2'] []
fewer instances of one module [] [] ['fifo_mem']
all three are structural changes: True
all three are invisible in QoR : True
the rename looks like : one missing and one added
and that pairing is : exactly what a reviewer must seeThree structural changes, three different signatures, none visible in QoR.
The third is the subtlest: every module is still present and fifo_mem is
instantiated twice instead of four times, so a check that only compares sets
of module names misses it entirely. Comparing instance counts catches it.
The rename case is reported as one missing and one added rather than as a
rename, which is the right behaviour: the tool cannot know that crc8_v2 is
the same logic as crc8, and guessing would let a genuine deletion hide behind
an unrelated addition.
8. Noise Is Per Metric Too
DECLARATIONS = [
("area_um2", "um2", -1, "percent", 0.5),
("leakage_uw", "uW", -1, "percent", 2.0),
("instances", "cells", -1, "percent", 0.5),
("wns_ns", "ns", +1, "absolute", 0.005),
("tns_ns", "ns", +1, "absolute", 0.010),
]
OBSERVED = [("area_um2", 18432.0, 18487.0), ("leakage_uw", 412.0, 416.5),
("instances", 24110.0, 24118.0), ("wns_ns", 0.210, 0.179),
("tns_ns", 0.000, 0.000)]
print(f"{'metric':<12} {'style':<9} {'change':>12} {'threshold':>10} signal?")
for name, b, a in OBSERVED:
decl = next(d for d in DECLARATIONS if d[0] == name)
_, unit, direction, style, noise = decl
if style == "percent":
change = (a - b) / abs(b) * 100 if b else 0.0
shown = f"{change:+.3f}%"
significant = abs(change) > noise
else:
change = a - b
shown = f"{change:+.3f} {unit}"
significant = abs(change) > noise
print(f"{name:<12} {style:<9} {shown:>12} {noise:>10.3f} {significant}")
print()
print("metrics observed :", len(OBSERVED))
print("changes above noise :",
sum(1 for name, b, a in OBSERVED
for d in [next(x for x in DECLARATIONS if x[0] == name)]
if (abs((a - b) / abs(b) * 100) > d[4] if d[3] == "percent" and b
else abs(a - b) > d[4])))
print()
print("0.298% of area is :", "noise")
print("0.031 ns of WNS is :", "signal")
print("a single global 1% ")
print(" threshold would :", "hide the WNS change and report the area one")metric style change threshold signal?
area_um2 percent +0.298% 0.500 False
leakage_uw percent +1.092% 2.000 False
instances percent +0.033% 0.500 False
wns_ns absolute -0.031 ns 0.005 True
tns_ns absolute +0.000 ns 0.010 False
metrics observed : 5
changes above noise : 1
0.298% of area is : noise
0.031 ns of WNS is : signal
a single global 1%
threshold would : hide the WNS change and report the area oneOne of five changes is signal, and a global threshold would have picked the wrong one. A single 1% rule reports the leakage move as significant and the WNS move as noise, which is backwards: 31 picoseconds off a 210 ps margin is the one change in that table a reviewer needs to see.
The thresholds differ because the metrics differ in how reproducible they are. Leakage estimates move by a per cent between runs of the same netlist with a different tool version; a WNS number from a fixed netlist and fixed constraints is stable to a few picoseconds. The noise floor is a property of the measurement, not a policy, and Chapter 29.2 Section 7's rule applies: it has to be written down.
9. The Verdict
from dataclasses import dataclass
@dataclass(frozen=True)
class Change:
metric: str
before: float
after: float
shown: str
better: bool
significant: bool
def verdict(changes, structural):
"""Four outcomes. SUSPECT overrides everything."""
if structural["missing"] or structural["count_changed"]:
return "SUSPECT"
moved = [c for c in changes if c.significant]
if not moved:
return "NO CHANGE"
better = [c for c in moved if c.better]
worse = [c for c in moved if not c.better]
if worse and better:
return "MIXED"
return "BETTER" if better else "WORSE"
CLEAN = {"missing": [], "added": [], "count_changed": []}
LOST = {"missing": ["crc8"], "added": [], "count_changed": []}
def C(name, better, significant):
return Change(name, 0.0, 0.0, "", better, significant)
SCENARIOS = [
("nothing moved", [C("area", True, False), C("wns", False, False)], CLEAN),
("area down, timing held", [C("area", True, True),
C("wns", True, False)], CLEAN),
("area down, timing worse", [C("area", True, True),
C("wns", False, True)], CLEAN),
("everything worse", [C("area", False, True),
C("wns", False, True)], CLEAN),
("everything better, a module gone", [C("area", True, True),
C("wns", True, True)], LOST),
]
print(f"{'scenario':<36} verdict")
for label, changes, structural in SCENARIOS:
print(f"{label:<36} {verdict(changes, structural)}")
print()
print("verdicts used :", 5)
print("MIXED is a real ")
print(" verdict :", "not a failure to decide")
print("and SUSPECT ")
print(" overrides :", "a table where every metric improved")scenario verdict
nothing moved NO CHANGE
area down, timing held BETTER
area down, timing worse MIXED
everything worse WORSE
everything better, a module gone SUSPECT
verdicts used : 5
MIXED is a real
verdict : not a failure to decide
and SUSPECT
overrides : a table where every metric improvedFive scenarios, five distinct verdicts, and the last one overrides a table in
which every metric improved. MIXED is the one worth defending: a comparator
that collapses "area down 8%, timing worse by 30 ps" into a single better-or-
worse answer has thrown away the trade-off that is the entire content of the
comparison.
NO CHANGE is also a real verdict rather than an absence of one. A run whose
every move is below its noise floor is a run that confirms a change was
neutral, which is often exactly what the author wanted to know.
10. The Comparator, Assembled
import re
from dataclasses import dataclass
from pathlib import Path
EXIT_OK, EXIT_WORSE, EXIT_SUSPECT = 0, 1, 3
@dataclass(frozen=True)
class Metric:
name: str
unit: str
direction: int # +1 higher is better, -1 lower is better
style: str # "percent" or "absolute"
noise: float
METRICS = {
"area_um2": Metric("area_um2", "um2", -1, "percent", 0.5),
"leakage_uw": Metric("leakage_uw", "uW", -1, "percent", 2.0),
"instances": Metric("instances", "cells", -1, "percent", 0.5),
"wns_ns": Metric("wns_ns", "ns", +1, "absolute", 0.005),
"tns_ns": Metric("tns_ns", "ns", +1, "absolute", 0.010),
}
LABELS = {"Total cell area": "area_um2",
"Total leakage power": "leakage_uw",
"Number of instances": "instances",
"Worst negative slack": "wns_ns",
"Total negative slack": "tns_ns"}
ROW = re.compile(r"^\s{2}(.+?)\s{2,}(-?\d+\.?\d*)\s*$", re.M)
HIER = re.compile(r"^\s{2}(\S+)\s+(\d+)\s*$", re.M)
@dataclass(frozen=True)
class Change:
metric: str
before: float
after: float
shown: str
better: bool
significant: bool
def parse_values(text):
out = {}
for m in ROW.finditer(text):
key = LABELS.get(m.group(1).strip())
if key:
out[key] = float(m.group(2))
return out
def parse_hierarchy(text):
return {m.group(1): int(m.group(2)) for m in HIER.finditer(text)}
def compute(before, after):
changes = []
for name in sorted(set(before) & set(after)):
decl = METRICS[name]
b, a = before[name], after[name]
if decl.style == "percent" and b:
value = (a - b) / abs(b) * 100
shown = f"{value:+.2f}%"
else:
value = a - b
shown = f"{value:+.3f} {decl.unit}"
better = (a > b) if decl.direction > 0 else (a < b)
changes.append(Change(name, b, a, shown, better,
abs(value) > decl.noise))
return changes
def compare_hierarchy(before, after):
return {"missing": sorted(set(before) - set(after)),
"added": sorted(set(after) - set(before)),
"count_changed": sorted(m for m in set(before) & set(after)
if before[m] != after[m])}
def verdict(changes, structural):
if structural["missing"] or structural["count_changed"]:
return "SUSPECT"
moved = [c for c in changes if c.significant]
if not moved:
return "NO CHANGE"
better = [c for c in moved if c.better]
worse = [c for c in moved if not c.better]
if better and worse:
return "MIXED"
return "BETTER" if better else "WORSE"
def report(changes, structural, v):
lines = [f"verdict: {v}", ""]
for c in changes:
mark = (" " if not c.significant else
"+ " if c.better else "- ")
note = "" if c.significant else " (within noise)"
lines.append(f"{mark}{c.metric:<12} {c.before:>10.3f} -> "
f"{c.after:>10.3f} {c.shown:>12}{note}")
if structural["missing"] or structural["added"] \
or structural["count_changed"]:
lines.append("")
lines.append("structural differences:")
for kind in ("missing", "added", "count_changed"):
if structural[kind]:
lines.append(f" {kind:<14} {structural[kind]}")
code = {"SUSPECT": EXIT_SUSPECT, "WORSE": EXIT_WORSE,
"MIXED": EXIT_WORSE}.get(v, EXIT_OK)
return "\n".join(lines), code
def write_run(path, area, leak, inst, wns, tns, modules):
d = Path(path)
d.mkdir(parents=True, exist_ok=True)
(d / "qor.rpt").write_text(
f"Report : qor\n Total cell area {area}\n"
f" Total leakage power {leak}\n"
f" Number of instances {inst}\n", encoding="utf-8")
(d / "timing.rpt").write_text(
f"Report : timing\n Worst negative slack {wns}\n"
f" Total negative slack {tns}\n", encoding="utf-8")
(d / "hierarchy.rpt").write_text(
"Report : hierarchy\n"
+ "".join(f" {m} {n}\n" for m, n in sorted(modules.items())),
encoding="utf-8")
def load(path):
d = Path(path)
values = parse_values((d / "qor.rpt").read_text(encoding="utf-8"))
values.update(parse_values((d / "timing.rpt").read_text(encoding="utf-8")))
hier = parse_hierarchy((d / "hierarchy.rpt").read_text(encoding="utf-8"))
return values, hier
MODULES = {"axi_fifo": 1, "fifo_ctrl": 1, "fifo_mem": 4, "axi_slave_if": 1,
"crc8": 2}
write_run("run_a", 18432.0, 412.0, 24110, -0.173, -2.410, MODULES)
write_run("run_b", 16998.0, 381.0, 21004, 0.042, 0.000, MODULES)
write_run("run_c", 12100.0, 260.0, 14880, 0.880, 0.000,
{k: v for k, v in MODULES.items() if k != "crc8"})
for label, after_dir in (("run_a -> run_b", "run_b"),
("run_a -> run_c", "run_c")):
before_v, before_h = load("run_a")
after_v, after_h = load(after_dir)
changes = compute(before_v, after_v)
structural = compare_hierarchy(before_h, after_h)
v = verdict(changes, structural)
text, code = report(changes, structural, v)
print(f"=== {label}")
for line in text.splitlines():
print(" ", line)
print(" exit:", code)
print()
print("metrics declared :", len(METRICS))
print("verdicts possible:", 5)
print("the SUSPECT case ")
print(" exits :", EXIT_SUSPECT)
print("and 33.8 reserved 3 for:", "an input problem — a netlist is an input")=== run_a -> run_b
verdict: BETTER
+ area_um2 18432.000 -> 16998.000 -7.78%
+ instances 24110.000 -> 21004.000 -12.88%
+ leakage_uw 412.000 -> 381.000 -7.52%
+ tns_ns -2.410 -> 0.000 +2.410 ns
+ wns_ns -0.173 -> 0.042 +0.215 ns
exit: 0
=== run_a -> run_c
verdict: SUSPECT
+ area_um2 18432.000 -> 12100.000 -34.35%
+ instances 24110.000 -> 14880.000 -38.28%
+ leakage_uw 412.000 -> 260.000 -36.89%
+ tns_ns -2.410 -> 0.000 +2.410 ns
+ wns_ns -0.173 -> 0.880 +1.053 ns
structural differences:
missing ['crc8']
exit: 3
metrics declared : 5
verdicts possible: 5
the SUSPECT case
exits : 3
and 33.8 reserved 3 for: an input problem — a netlist is an inputTwo comparisons with nearly identical tables and opposite verdicts. Both
show every metric improving; the second says SUSPECT and exits 3, and the only
difference between them is one line in hierarchy.rpt.
That is the clearest statement of what this module has been about. The numbers are the same shape, both reports are internally consistent, and the one piece of information that distinguishes a real improvement from a deleted block is in a file the obvious tool would never have opened.
Note the exit code: 3, which Chapter 33.8 reserved for an input problem. A
netlist missing a module is an input problem — the synthesis run was given
something other than the design — and giving it the same code as WORSE would
tell a CI job to treat a vanished block as a timing regression.
11. How It Is Tested
import io
import unittest
from dataclasses import dataclass
@dataclass(frozen=True)
class Metric:
name: str
unit: str
direction: int
style: str
noise: float
METRICS = {
"area_um2": Metric("area_um2", "um2", -1, "percent", 0.5),
"wns_ns": Metric("wns_ns", "ns", +1, "absolute", 0.005),
}
def change_of(name, before, after):
d = METRICS[name]
if d.style == "percent" and before:
value = (after - before) / abs(before) * 100
else:
value = after - before
better = (after > before) if d.direction > 0 else (after < before)
return value, better, abs(value) > d.noise
def compare_hierarchy(before, after):
return {"missing": sorted(set(before) - set(after)),
"added": sorted(set(after) - set(before)),
"count_changed": sorted(m for m in set(before) & set(after)
if before[m] != after[m])}
def verdict(flags, structural):
"""flags: list of (better, significant)."""
if structural["missing"] or structural["count_changed"]:
return "SUSPECT"
moved = [(b, s) for b, s in flags if s]
if not moved:
return "NO CHANGE"
better = [b for b, _ in moved if b]
worse = [b for b, _ in moved if not b]
if better and worse:
return "MIXED"
return "BETTER" if better else "WORSE"
CLEAN = {"missing": [], "added": [], "count_changed": []}
class Comparator(unittest.TestCase):
# --- direction is per metric
def test_area_down_is_better(self):
_, better, _ = change_of("area_um2", 18432.0, 16998.0)
self.assertTrue(better)
def test_area_up_is_worse(self):
_, better, _ = change_of("area_um2", 16998.0, 18432.0)
self.assertFalse(better)
def test_slack_up_is_better(self):
_, better, _ = change_of("wns_ns", -0.173, 0.042)
self.assertTrue(better)
def test_slack_down_is_worse(self):
_, better, _ = change_of("wns_ns", 0.042, -0.173)
self.assertFalse(better)
# --- percentage vs absolute
def test_slack_uses_an_absolute_delta(self):
value, _, _ = change_of("wns_ns", -0.173, 0.042)
self.assertAlmostEqual(value, 0.215, places=3)
def test_slack_is_not_a_percentage(self):
"""§5: the percentage would be +124.28, which is meaningless."""
value, _, _ = change_of("wns_ns", -0.173, 0.042)
self.assertNotAlmostEqual(value, 124.28, places=1)
def test_a_zero_baseline_does_not_divide(self):
value, _, _ = change_of("wns_ns", 0.0, -0.310)
self.assertAlmostEqual(value, -0.310, places=3)
# --- noise
def test_a_small_area_move_is_noise(self):
_, _, sig = change_of("area_um2", 18432.0, 18487.0)
self.assertFalse(sig)
def test_a_small_slack_move_is_signal(self):
_, _, sig = change_of("wns_ns", 0.210, 0.179)
self.assertTrue(sig)
# --- structural
def test_a_vanished_module_is_detected(self):
got = compare_hierarchy({"a": 1, "crc8": 2}, {"a": 1})
self.assertEqual(got["missing"], ["crc8"])
def test_a_renamed_module_is_one_missing_and_one_added(self):
got = compare_hierarchy({"crc8": 2}, {"crc8_v2": 2})
self.assertEqual((got["missing"], got["added"]),
(["crc8"], ["crc8_v2"]))
def test_fewer_instances_is_detected(self):
got = compare_hierarchy({"fifo_mem": 4}, {"fifo_mem": 2})
self.assertEqual(got["count_changed"], ["fifo_mem"])
def test_identical_hierarchies_are_clean(self):
got = compare_hierarchy({"a": 1}, {"a": 1})
self.assertEqual(got, CLEAN)
# --- the verdict
def test_all_better_is_better(self):
self.assertEqual(verdict([(True, True)], CLEAN), "BETTER")
def test_all_worse_is_worse(self):
self.assertEqual(verdict([(False, True)], CLEAN), "WORSE")
def test_both_is_mixed(self):
self.assertEqual(verdict([(True, True), (False, True)], CLEAN),
"MIXED")
def test_nothing_significant_is_no_change(self):
self.assertEqual(verdict([(True, False), (False, False)], CLEAN),
"NO CHANGE")
def test_a_missing_module_overrides_everything_better(self):
"""§7: every metric improved and the design lost a block."""
flags = [(True, True), (True, True), (True, True)]
self.assertEqual(verdict(flags, {"missing": ["crc8"], "added": [],
"count_changed": []}), "SUSPECT")
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"ok": res.wasSuccessful()}
print("comparator suite:", run(Comparator))
print()
print("direction tests :", 4)
print("expression tests :", 3)
print("noise tests :", 2)
print("structural tests :", 4)
print("verdict tests :", 5)
print()
print("the test that encodes the chapter:")
print(" test_a_missing_module_overrides_everything_better")
print("asserting that :",
"three improvements do not outvote one deletion")comparator suite: {'run': 18, 'failures': 0, 'ok': True}
direction tests : 4
expression tests : 3
noise tests : 2
structural tests : 4
verdict tests : 5
the test that encodes the chapter:
test_a_missing_module_overrides_everything_better
asserting that : three improvements do not outvote one deletionEighteen behaviours in five groups, and no synthesis tool. Every decision in the comparator takes numbers as arguments, so the whole of it is Chapter 34.3's testable middle layer — which is what a comparison tool should be, since its inputs are reports rather than tools.
test_slack_is_not_a_percentage is Chapter 33.9's pattern once more: it asserts
the specific wrong value 124.28 is absent, because a test checking only
that the answer is 0.215 would pass under an implementation that happened to
be right for this fixture and wrong in general.
test_a_missing_module_overrides_everything_better is the chapter in one
assertion. Three significant improvements and one missing module must produce
SUSPECT, and an implementation that counted votes would return BETTER.
12. Predict, Run, Observe
# A
b, a = -0.173, 0.042
print("A:", round((a - b) / abs(b) * 100, 2), round(a - b, 3))
# B
print("B:", (a < b), (a > b))
# C
try:
print("C:", (0.310 - 0.0) / abs(0.0) * 100)
except ZeroDivisionError:
print("C:", "ZeroDivisionError", round(0.310 - 0.0, 3))
# D
before = {"a": 1, "crc8": 2}
after = {"a": 1}
print("D:", sorted(set(before) - set(after)),
sorted(set(after) - set(before)))The verified output:
A: 124.28 0.215
B: False True
C: ZeroDivisionError 0.31
D: ['crc8'] []A — the same change expressed two ways: a meaningless 124.28% and an
honest +0.215 ns. Section 5 in one line.
B — and the direction, which is what the percentage's sign fails to carry: the slack went up, which for slack is better, and for area would be worse.
C — a zero baseline raises rather than producing a number, which is why the
style declaration exists: an absolute delta has no denominator and cannot
fail this way.
D — the structural comparison is two set differences, and that is the whole mechanism that catches the module's most dangerous report. Four lines of code against five metrics that all agreed the design had improved.
13. Exercises
Exercise 1 — four questions. Ask your own QoR report which of them it answers.
Exercise 2 — a verdict, not a table. Write the one-line verdict your last comparison should have produced.
Exercise 3 — directories, not files. Say what a comparator pointed at two
qor.rpt files cannot see.
Exercise 4 — your own v1. Subtract two reports and print percentages.
Exercise 5 — the nonsense cell. Find the value in your table that is not interpretable.
Exercise 6 — declare direction. Write the direction for every metric your flow reports.
Exercise 7 — the same sign, two meanings. Give an example from your own metrics.
Exercise 8 — across zero. Compute a percentage change for a slack that changed sign and say what it would mean.
Exercise 9 — the zero baseline. Find the metric in your set that can be exactly zero.
Exercise 10 — percent or absolute. Classify each of your metrics and justify the split.
Exercise 11 — five of five. Construct the lost-logic case for your own design and confirm every metric improves.
Exercise 12 — the four ordinary causes. For each, say what the RTL diff would look like.
Exercise 13 — compare the module lists. Add the structural check to your comparator.
Exercise 14 — instance counts too. Explain what a set-only comparison misses.
Exercise 15 — do not guess a rename. Say why reporting one missing and one added is better than inferring.
Exercise 16 — your noise floors. Measure each metric's run-to-run spread on an unchanged netlist.
Exercise 17 — a global threshold. Work out which of your changes it would misclassify.
Exercise 18 — noise is a measurement property. Explain why it is not a policy.
Exercise 19 — four verdicts. Implement BETTER, WORSE, MIXED and SUSPECT.
Exercise 20 — defend MIXED. Say what collapsing it loses.
Exercise 21 — NO CHANGE is a result. Describe when an author wants it.
Exercise 22 — SUSPECT overrides. Confirm it wins over a table of improvements.
Exercise 23 — the exit code. Decide what SUSPECT exits and why not 1.
Exercise 24 — assert the wrong value is absent. Add the
assertNotAlmostEqual for your percentage trap.
Exercise 25 — three improvements, one deletion. Write the verdict test and say what a vote-counting implementation would return.
Exercise 26 — the whole module. Pick one of the nine tools in this module and apply Chapter 33's testing and Chapter 34's structure to it end to end.
14. Summary
- Three of four questions a comparison must answer want a verdict, and a table of percentages leaves the hard part to the reviewer.
- The annotation takes directories, not report files, because the most serious difference between two runs is not in the QoR report at all.
- Direction is per metric: four of six want a lower number and two want a higher one, so the sign of a change carries no information on its own.
- A percentage is meaningless across zero. WNS from −0.173 to +0.042 computes as +124.28%, and a design going from 42 ps of margin to 173 ps of violation computes as −511.90%.
- Four of six percentage changes were undefined or uninterpretable, and the absolute delta was right in all six.
- So the expression style is declared per metric: a positive metric with a natural zero takes a percentage, a signed metric takes an absolute delta.
- A design that lost a module improved on all five metrics — area −34%, leakage −37%, instances −38%, timing from failing to 880 ps — and no QoR metric can detect it.
- It is confidently wrong in the direction people want to believe, which is
the hardest kind to catch, and the usual causes are ordinary: a tied-off
input, a false
generate, a renamed module, a commented-out line. - Compare module names and instance counts. Three structural changes have three different signatures, and a set-only comparison misses the one where every module is still present.
- Report a rename as one missing and one added, because inferring the pairing would let a genuine deletion hide behind an unrelated addition.
- Noise thresholds are per metric and are a property of the measurement. 0.298% of area is noise and 0.031 ns of WNS is signal; a global 1% rule gets both backwards.
- Four verdicts, and MIXED is a real one — collapsing "area down 8%, timing worse by 30 ps" into better-or-worse discards the trade-off that is the comparison's whole content.
- SUSPECT overrides every improvement and exits 3, because a netlist missing a module is an input problem rather than a regression.
- Two comparisons with nearly identical tables and opposite verdicts, separated by one line in a file the obvious tool would never have opened.
- Eighteen behaviours tested with no synthesis tool, because every decision takes numbers as arguments.
That closes Module 35. Nine tools, each built from a one-line version that was correct and unusable, and in every single case the limitation that forced the real design was the same shape: a number that was right about the wrong thing.
Where this fits
Part of the Python curriculum.