QoR Deltas Between Two Runs

Part II · Module 22 · Chapter 22.5 · QoR Deltas Between Two Runs

1. Two Runs, Three Movements

The two runs are real netlists differing by one buffer rewired, one inverter added, and two physical-only cells removed.

use strict;
use warnings;

# Two runs of the SAME design: counter_syn and counter_syn_v2, whose netlists
# differ by one buffer rewired, one inverter added and two physical cells
# removed. Both report sets are generated from those real netlists.
my @AREA_SPELL = (
  [ "Total cell area",    qr/^\s*Total cell area:\s+([\d.]+)/m        ],
  [ "Total Area (cells)", qr/^\s*Total Area \(cells\)\s*:\s*([\d.]+)/m ],
  [ "cell_area_total",    qr/^\s*cell_area_total\s*=\s*([\d.]+)/m      ],
);
sub slurp { open my $f, "<", $_[0] or die "$_[0]: $!"; local $/; my $t = <$f>; close $f; $t }

sub read_run {
    my ($tag) = @_;
    my %r = (tag => $tag);
    my $area = slurp("reports/$tag.area.rpt");
    for my $c (@AREA_SPELL) { my ($v) = $area =~ $c->[1]; if (defined $v) { $r{area} = $v; last } }
    ($r{cells}) = $area =~ /^\s*Number of cells:\s+(\d+)/m;
    my $cells = slurp("reports/$tag.cells.rpt");
    while ($cells =~ /^([A-Za-z][\w.\-]*)\s+\S+\s+([\d.]+)\s+(\d+)\s+([\d.]+)\s*$/mg) {
        $r{by_type}{$1} = $3;
    }
    return %r;
}

my %A = read_run("counter_syn");
my %B = read_run("counter_syn_v2");

printf "%-10s %-12s %-8s\n", "RUN", "AREA", "CELLS";
printf "%-10s %-12s %-8s\n", $_->{tag}, $_->{area}, $_->{cells} for (\%A, \%B);

my $d_area  = $B{area}  - $A{area};
my $d_cells = $B{cells} - $A{cells};
printf "\ndelta: area %+.4f (%+.2f%%), cells %+d\n",
    $d_area, 100 * $d_area / $A{area}, $d_cells;

# ---- per-type deltas, with a SIGNIFICANCE rule stated as data
my %all = map { $_ => 1 } (keys %{ $A{by_type} }, keys %{ $B{by_type} });
my @moved;
for my $t (sort keys %all) {
    my $was = $A{by_type}{$t} // 0;
    my $now = $B{by_type}{$t} // 0;
    next if $was == $now;
    push @moved, { type => $t, was => $was, now => $now, d => $now - $was };
}
printf "\n%d of %d cell types moved:\n", scalar @moved, scalar keys %all;
printf "%-12s %8s %8s %8s %s\n", "TYPE", "WAS", "NOW", "DELTA", "note";
for my $m (sort { abs($b->{d}) <=> abs($a->{d}) || $a->{type} cmp $b->{type} } @moved) {
    printf "%-12s %8d %8d %+8d %s\n", $m->{type}, $m->{was}, $m->{now}, $m->{d},
        ($m->{was} == 0 ? "APPEARED" : $m->{now} == 0 ? "DISAPPEARED" : "");
}

print "\nthe types that did NOT move are the majority, and reporting them\n";
printf "would make the list %d rows instead of %d:\n", scalar keys %all, scalar @moved;
printf "  unchanged: %s\n", join(", ", sort grep {
    (($A{by_type}{$_} // 0) == ($B{by_type}{$_} // 0)) } keys %all);

# ---- a threshold on AREA, and why a percentage alone is not enough
print "\nis the area movement significant?\n";
# Thresholds chosen to STRADDLE this data on purpose, so the disagreement
# is visible rather than asserted.
my @RULES = (
  [ "absolute, > 0.1 area units", sub { abs($d_area) > 0.1 } ],
  [ "relative, > 1%",             sub { abs(100 * $d_area / $A{area}) > 1 } ],
  [ "either of the two",          sub { abs($d_area) > 0.1 || abs(100 * $d_area / $A{area}) > 1 } ],
  [ "both of the two",            sub { abs($d_area) > 0.1 && abs(100 * $d_area / $A{area}) > 1 } ],
  [ "any cell type appeared/went",sub { scalar grep { $_->{was} == 0 || $_->{now} == 0 } @moved } ],
);
my @verdicts;
for my $r (@RULES) {
    my $v = $r->[1]->() ? "SIGNIFICANT" : "not significant";
    push @verdicts, $v;
    printf "  %-30s %s\n", $r->[0], $v;
}
my %distinct = map { $_ => 1 } @verdicts;
printf "  the %d rules produce %d DIFFERENT verdicts on the same numbers\n",
    scalar @RULES, scalar keys %distinct;
print "  -- so 'significant' is a policy, and it belongs in ONE named place\n";
print "  rather than inline at each comparison.\n";

# ---- what a delta CANNOT tell you
print "\nwhat the delta cannot tell you:\n";
my @a_types = sort keys %{ $A{by_type} };
my @b_types = sort keys %{ $B{by_type} };
printf "  the two runs used %d and %d distinct cell types\n", scalar @a_types, scalar @b_types;
print "  but a cell COUNT cannot distinguish a resynthesis that reached the\n";
print "  same structure from one that happened to use the same cell mix.\n";
print "  Equivalence is a formal question; a delta is a smoke test (20.7).\n";

RUN        AREA         CELLS   
counter_syn 34.580000    23      
counter_syn_v2 34.314000    22      

delta: area -0.2660 (-0.77%), cells -1

3 of 9 cell types moved:
TYPE              WAS      NOW    DELTA note
FILL1               1        0       -1 DISAPPEARED
INVD1               9       10       +1 
TIELO               1        0       -1 DISAPPEARED

the types that did NOT move are the majority, and reporting them
would make the list 9 rows instead of 3:
  unchanged: AN2D1, BUFFD1, CKBD1, CKLNQD1, DFCNQD1, XOR2D1

is the area movement significant?
  absolute, > 0.1 area units     SIGNIFICANT
  relative, > 1%                 not significant
  either of the two              SIGNIFICANT
  both of the two                not significant
  any cell type appeared/went    SIGNIFICANT
  the 5 rules produce 2 DIFFERENT verdicts on the same numbers
  -- so 'significant' is a policy, and it belongs in ONE named place
  rather than inline at each comparison.

what the delta cannot tell you:
  the two runs used 9 and 7 distinct cell types
  but a cell COUNT cannot distinguish a resynthesis that reached the
  same structure from one that happened to use the same cell mix.
  Equivalence is a formal question; a delta is a smoke test (20.7).

delta: area -0.2660 (-0.77%), cells -1

3 of 9 cell types moved:
FILL1   1 ->  0   -1  DISAPPEARED
INVD1   9 -> 10   +1
TIELO   1 ->  0   -1  DISAPPEARED

Three rows instead of nine. Six cell types did not move at all, and the output names them separately rather than mixing them in — because "unchanged" is information too, just not information that belongs in the same list.

2. Report What Moved, Not What Is

The structure is a hash comparison over the union of both runs' keys:

my %all = map { $_ => 1 } (keys %{ $A{by_type} }, keys %{ $B{by_type} });
for my $t (sort keys %all) {
    my $was = $A{by_type}{$t} // 0;
    my $now = $B{by_type}{$t} // 0;
    next if $was == $now;
    push @moved, { type => $t, was => $was, now => $now, d => $now - $was };
}

Three details that are easy to get wrong, and the first is the one that silently loses data:

Iterate the UNION, not either run's keys. Chapter 6.9's hash-as-set over both key lists. Looping over run A's types misses anything that appeared in B; looping over B's misses anything that disappeared. Both failures are silent, and both are exactly the movements most worth seeing.

// 0 for the missing side. A type absent from one run has a count of zero in that run, which is a real statement — unlike Chapter 22.2's area field, where absent meant unparseable. The distinction matters: here zero is the truth, there zero was a lie.

sort keys before iterating. Chapter 6.6 measured that hash order changes between runs, so without the sort two invocations of this script on the same reports produce the same rows in different orders — and a QoR delta is something people diff against yesterday's output.

And the same filter applies at the top level: the script reads Total cell area through Chapter 22.2's spellings table rather than one pattern, because the two runs can come from two tool versions. A QoR comparison is exactly the situation where that happens — somebody upgraded the tool, which is why you are comparing.

3. "Significant" Is A Policy

Now the measurement that is the point of the chapter. Five rules, all defensible, applied to the same -0.2660 (-0.77%):

Five rules, two verdicts, one pair of numbers.

So significance is not a property of the data. It is a decision, and the consequences of pretending otherwise are familiar from Chapter 20.7: a threshold that is too tight reports every run and nobody reads the report; one that is too loose reports nothing and the report is pointless.

What follows practically:

Put the rule in one named place. A significant() function, or a table of rules as Chapter 7.11's code references — not an inline if (abs($d) > 0.1) at each comparison, which is how three different thresholds end up in one script.

Report the number alongside the verdict. -0.2660 (-0.77%) [significant: absolute > 0.1] tells the reader both what happened and why it was flagged, so they can disagree with the policy without re-deriving the data.

Expect to tune it, and version it. The right threshold for a block being actively optimised is not the right threshold for a frozen one. Chapter 19.6's point about configurations applies: the varying part belongs in a description somebody can edit.

4. Appeared And Disappeared Are Their Own Categories

FILL1 1 -> 0 DISAPPEARED is flagged differently from INVD1 9 -> 10, and the distinction is not cosmetic.

A type going to zero means a category of cell left the design. Fillers disappearing between two runs usually means one report is pre-layout and the other post-layout — which makes the whole comparison invalid, not just that row. It is a signal about the inputs, not about the design.

A type appearing from zero is the stronger signal. A cell type that was not in the design last week is either a library change, a new construct in the RTL, or a different optimisation path. All three are worth a sentence.

A count changing is the ordinary case and usually the least interesting, which is the inverse of how a delta sorted by magnitude presents it.

So the appeared/disappeared flag earns its own column:

($m->{was} == 0 ? "APPEARED" : $m->{now} == 0 ? "DISAPPEARED" : "")

Chapter 21.4 found the same shape in warnings: the rare event is the informative one, and ranking by magnitude puts it last. Here, FILL1 -1 is the smallest delta in the table and the one that says "you may be comparing two different kinds of netlist".

5. What A Delta Cannot Tell You

the two runs used 9 and 7 distinct cell types
but a cell COUNT cannot distinguish a resynthesis that reached the
same structure from one that happened to use the same cell mix.

A delta of zero does not mean the designs are the same. Two netlists can have identical cell-type counts and completely different connectivity — and Chapter 20.1 established that net and instance names change on every resynthesis anyway, so even a textual comparison says nothing.

Three things a QoR delta is genuinely for:

A smoke test before the expensive checks. If area moved 40%, something is wrong with the inputs and formal equivalence will waste an hour confirming it.

A record of what changed, for later. Six months on, "the inverter count went up by one in week 14" is the kind of fact that answers a question nobody could have asked at the time.

A trigger for a human to look. Not a verdict. Chapter 20.7's rule — report direction and magnitude, never a judgement about acceptability, because acceptability belongs to whoever owns the design.

Equivalence is a formal-verification question, and a delta sits in front of it as a cheap filter. Saying that in the report is better than implying otherwise by silence.

6. What To Remember

• Nobody reviews a run by reading its reports — they compare it against the last one.
• Measured: 3 of 9 cell types moved; the six unchanged types are named separately, not mixed in.
• Iterate the UNION of both runs' keys (6.9) — looping over one run silently misses appearances or disappearances.
• // 0 for the missing side: here zero is the truth, unlike 22.2 where an absent field was unparseable.
• sort keys before iterating (6.6), or two runs of the script produce the same rows in a different order.
• Read the area through 22.2's spellings table — a QoR comparison is exactly when the tool version changed.
• Measured: 5 defensible significance rules give 2 different verdicts on one pair of numbers.
• Significance is a policy, not a property of the data.
• Put the rule in ONE named place (7.11), not inline at each comparison.
• Report the number alongside the verdict so a reader can disagree with the policy without re-deriving the data.
• My first version's thresholds made all four rules agree while the prose claimed they disagreed — check that a demonstration of disagreement actually disagrees.
• Flag APPEARED and DISAPPEARED separately from a count change.
• A type going to zero can invalidate the whole comparison — fillers vanishing usually means pre- versus post-layout reports.
• A type appearing is the stronger signal: a library change, new RTL, or a different optimisation path.
• The smallest delta can be the most informative (21.4's shape), and magnitude sorting puts it last.
• A delta of zero does not mean the designs match — identical counts, different connectivity.
• A delta is a smoke test, a record, and a trigger for a human — never a verdict on acceptability (20.7).

Chapter 22.6 closes the module with the one file this chapter has not touched. The log carries the tool's own complaints, and the useful operation is separating the ones that appear in every run from the ones that appeared today.