Perl Part II · Module 24 · Coverage and Regression Reporting
Parsing Simulator Coverage Output
Coverage output is easy to parse and easy to parse in a way that destroys the information you wanted. Three of the four mistakes in this chapter produce a clean table of plausible percentages — one of them by throwing away every unhit bin, which is to say every row the report exists to show. The fix in each case is the same: store what the tool measured, and derive the percentage yourself whenever you need one.
Intermediate11 min readPerlPart IIRegression Reporting
Part II · Module 24 · Chapter 24.3 · Parsing Simulator Coverage Output
1. The Block, And The Runs That Have None
Coverage arrives as a delimited section inside the log, so the parser is a state machine with one flag — Chapter 18.6's shape.
use strict;
use warnings;
# ---- the coverage block of one log: group, bin, hit count, denominator
sub read_coverage {
my ($path) = @_;
my (%cov, $grp, $inside);
open my $fh, "<", $path or return (undef, "cannot open");
while (my $l = <$fh>) {
if ($l =~ /^--- COVERAGE ---/) { $inside = 1; next }
if ($l =~ /^--- END COVERAGE ---/) { $inside = 0; next }
next unless $inside;
if ($l =~ /^COVERGROUP\s+(\S+)/) { $grp = $1; next }
# a hit count of 0 is a VALUE. The capture must not be used as a test.
if ($l =~ /^\s+BIN\s+(\S+)\s+(\d+)\s*$/) {
$cov{$grp}{$1} = $2 + 0;
}
}
close $fh;
return (\%cov, $inside ? "block never closed" : (%cov ? "ok" : "no block"));
}
my @logs = sort glob("logs/*.log");
my ($with, $without) = (0, 0);
my %state;
for my $p (@logs) {
my ($c, $why) = read_coverage($p);
$state{$why}++;
$c && %$c ? $with++ : $without++;
}
printf "logs: %d with a coverage block: %d without: %d\n",
scalar @logs, $with, $without;
printf " %-20s %d\n", $_, $state{$_} for sort keys %state;
print "the runs with no coverage block are the ones that never finished:\n";
for my $p (@logs) {
my ($c) = read_coverage($p);
next if $c && %$c;
(my $n = $p) =~ s{logs/|\.log}{}g;
printf " %s\n", $n;
}
# ---- 1. a zero hit count is data
print "\n==== a zero hit count is data, not absence ====\n";
my ($cov) = read_coverage("logs/arb_smoke.7.log");
my ($kept_truthy, $kept_defined) = (0, 0);
for my $g (sort keys %$cov) {
for my $b (sort keys %{ $cov->{$g} }) {
$kept_defined++;
$kept_truthy++ if $cov->{$g}{$b}; # the tempting test
}
}
printf "arb_smoke bins in the log : %d\n", $kept_defined;
printf "bins kept by 'if (\$hits)' : %d\n", $kept_truthy;
printf "bins DROPPED, every one of them a hole: %d\n", $kept_defined - $kept_truthy;
print "the first few that would vanish:\n";
my $shown = 0;
for my $g (sort keys %$cov) {
for my $b (sort keys %{ $cov->{$g} }) {
next if $cov->{$g}{$b};
printf " %-16s %-22s %d\n", $g, $b, $cov->{$g}{$b};
last if ++$shown >= 5;
}
last if $shown >= 5;
}
print "a parser that stores only non-zero bins cannot report a hole,\n";
print "because the hole is exactly what it discarded\n";
# ---- 2. keep the denominator
print "\n==== the denominator is not in the percentage ====\n";
printf "%-16s %-10s %-10s %s\n", "GROUP", "HIT/TOTAL", "PERCENT", "";
my @rows;
for my $g (sort keys %$cov) {
my @b = keys %{ $cov->{$g} };
my $h = grep { $cov->{$g}{$_} > 0 } @b;
push @rows, [$g, $h, scalar @b];
printf "%-16s %-10s %-10s\n", $g, "$h/" . scalar(@b),
sprintf("%.2f%%", 100 * $h / @b);
}
print "\ntwo groups at the same percentage, different denominators:\n";
my %bypct;
push @{ $bypct{ sprintf "%.2f", 100 * $_->[1] / $_->[2] } }, $_ for @rows;
for my $p (sort keys %bypct) {
next unless @{ $bypct{$p} } > 1;
printf " %s%% <- %s\n", $p,
join ", ", map { "$_->[0] $_->[1]/$_->[2]" } @{ $bypct{$p} };
}
printf "\n100%% of 4 bins and 100%% of 16 bins are the same number and not the\n";
printf "same amount of evidence, so the record keeps hit and total, never a rate\n";
# ---- 3. a percentage cannot be inverted
print "\n==== a percentage cannot be inverted ====\n";
my $pct = 93.75;
print "a vendor summary line reporting 93.75% is consistent with:\n";
for my $t (4, 8, 16, 32, 160, 1600) {
my $h = $pct / 100 * $t;
next if $h != int $h;
printf " %6d / %-6d = %.2f%%\n", $h, $t, 100 * $h / $t;
}
printf "\n15/16 and 30/32 are both 93.75%%: the same rate, %d bins more, and the\n", 32-16;
printf "number of HOLES went from %d to %d -- it doubled, invisibly\n", 16-15, 32-30;
# ---- 4. the same shape in a vendor summary format
print "\n==== the same numbers in a one-line-per-group format ====\n";
print "(a representative vendor shape, derived from the per-bin block above)\n";
my @sample = map { sprintf " %-18s %4d / %-4d %6.2f%%", $_->[0], $_->[1], $_->[2],
100 * $_->[1] / $_->[2] } @rows;
print "$_\n" for @sample;
print "\nparsed back:\n";
for my $l (@sample) {
if ($l =~ m{^\s*(\S+)\s+(\d+)\s*/\s*(\d+)\s+([\d.]+)%}) {
my ($g, $h, $t, $p) = ($1, $2, $3, $4);
my $recomputed = sprintf "%.2f", 100 * $h / $t;
printf " %-18s hit %-4d total %-4d printed %-7s recomputed %-7s %s\n",
$g, $h, $t, $p, $recomputed,
$p == $recomputed ? "agree" : "DISAGREE";
}
}
# ---- 5. two decimal places is not enough to be stable
print "\n==== rounding at two decimals ====\n";
printf "%-18s %-10s %-9s %-9s %s\n", "TEST", "HIT/TOTAL", "ROUNDED", "TRUNCATED", "";
my $differ = 0;
for my $p (sort glob("logs/*.7.log")) {
my ($c) = read_coverage($p);
next unless $c && %$c;
my ($h, $t) = (0, 0);
for my $g (keys %$c) {
$t += scalar keys %{ $c->{$g} };
$h += grep { $c->{$g}{$_} > 0 } keys %{ $c->{$g} };
}
my $exact = 100 * $h / $t;
my $round = sprintf "%.2f", $exact;
my $trunc = int($exact * 100) / 100;
$trunc = sprintf "%.2f", $trunc;
$differ++ if $round ne $trunc;
(my $n = $p) =~ s{logs/|\.7\.log}{}g;
printf "%-18s %-10s %-9s %-9s %s\n", $n, "$h/$t", $round, $trunc,
$round ne $trunc ? "DISAGREE" : "";
}
printf "\ntests where rounding and truncation print different numbers: %d\n", $differ;
print "so a trend diff comparing printed percentages can show a change\n";
print "on a night when nothing changed, and hide one when it did\n";logs: 36 with a coverage block: 29 without: 7
no block 7
ok 29
the runs with no coverage block are the ones that never finished:
arb_budget.1
arb_budget.1234
arb_budget.42
arb_budget.7
arb_trunc.1234
arb_trunc.42
arb_trunc.7
==== a zero hit count is data, not absence ====
arb_smoke bins in the log : 41
bins kept by 'if ($hits)' : 25
bins DROPPED, every one of them a hole: 16
the first few that would vanish:
cg_illegal grant_without_request 0
cg_req_pattern req_0011 0
cg_req_pattern req_0101 0
cg_req_pattern req_0110 0
cg_req_pattern req_0111 0
a parser that stores only non-zero bins cannot report a hole,
because the hole is exactly what it discarded
==== the denominator is not in the percentage ====
GROUP HIT/TOTAL PERCENT
cg_grant_id 4/4 100.00%
cg_illegal 0/1 0.00%
cg_ptr_x_grant 16/16 100.00%
cg_req_pattern 5/16 31.25%
cg_wait_ge3 0/4 0.00%
two groups at the same percentage, different denominators:
0.00% <- cg_illegal 0/1, cg_wait_ge3 0/4
100.00% <- cg_grant_id 4/4, cg_ptr_x_grant 16/16
100% of 4 bins and 100% of 16 bins are the same number and not the
same amount of evidence, so the record keeps hit and total, never a rate
==== a percentage cannot be inverted ====
a vendor summary line reporting 93.75% is consistent with:
15 / 16 = 93.75%
30 / 32 = 93.75%
150 / 160 = 93.75%
1500 / 1600 = 93.75%
15/16 and 30/32 are both 93.75%: the same rate, 16 bins more, and the
number of HOLES went from 1 to 2 -- it doubled, invisibly
==== the same numbers in a one-line-per-group format ====
(a representative vendor shape, derived from the per-bin block above)
cg_grant_id 4 / 4 100.00%
cg_illegal 0 / 1 0.00%
cg_ptr_x_grant 16 / 16 100.00%
cg_req_pattern 5 / 16 31.25%
cg_wait_ge3 0 / 4 0.00%
parsed back:
cg_grant_id hit 4 total 4 printed 100.00 recomputed 100.00 agree
cg_illegal hit 0 total 1 printed 0.00 recomputed 0.00 agree
cg_ptr_x_grant hit 16 total 16 printed 100.00 recomputed 100.00 agree
cg_req_pattern hit 5 total 16 printed 31.25 recomputed 31.25 agree
cg_wait_ge3 hit 0 total 4 printed 0.00 recomputed 0.00 agree
==== rounding at two decimals ====
TEST HIT/TOTAL ROUNDED TRUNCATED
arb_light 18/41 43.90 43.90
arb_random 40/41 97.56 97.56
arb_random_short 38/41 92.68 92.68
arb_single 25/41 60.98 60.97 DISAGREE
arb_smoke 25/41 60.98 60.97 DISAGREE
arb_soak 40/41 97.56 97.56
arb_stress 39/41 95.12 95.12
tests where rounding and truncation print different numbers: 2
so a trend diff comparing printed percentages can show a change
on a night when nothing changed, and hide one when it didThe first number is worth pausing on:
logs: 36 with a coverage block: 29 without: 7
the runs with no coverage block are the ones that never finished:
arb_budget.1 ... arb_trunc.7Coverage is missing exactly where a verdict is missing. The seven logs without a block are the four that hit the cycle budget and the three the farm cut short. That is not a coincidence — the block is printed at the end of a run, so any run that does not reach its end has none.
So a coverage reader needs the same three states a verdict reader needs (18.9): the block is present, the block is absent, or the block is absent because the run never got there. The third is the one that must not be silently merged in as zeros, because zero hits and no measurement are different facts.
2. A Zero Hit Count Is Data
This is the mistake that matters most, and it is the shortest to write:
arb_smoke bins in the log : 41
bins kept by 'if ($hits)' : 25
bins DROPPED, every one of them a hole: 16
cg_illegal grant_without_request 0
cg_req_pattern req_0011 0Sixteen of forty-one bins discarded, and every single one of them unhit. The filter is not biased, it is perfectly selective: if ($hits) keeps exactly the bins that need no attention and drops exactly the bins the coverage report exists to list.
The resulting hash is not wrong about anything it contains. It is 25 bins, all with real hit counts, and a roll-up over it will report 25 of 25 bins hit — 100%. Chapter 24.5 is about closure, and this is the single easiest way to report closure you do not have.
The fix is to test for presence, not truth:
$cov{$grp}{$1} = $2 + 0; # store it, whatever it isand later, when you want hits, say so: grep { $cov{$g}{$_} > 0 }. Chapter 6.4's distinction between exists, defined and truth is the whole chapter in one line — 0 is defined, it exists, and it is false.
The + 0 is deliberate too. Perl has no separate number and string type (2.6), so the stored value carries whichever it was last used as — and + 0 settles that at the point of capture rather than leaving it to the first comparison.
3. Keep The Denominator
0.00% <- cg_illegal 0/1, cg_wait_ge3 0/4
100.00% <- cg_grant_id 4/4, cg_ptr_x_grant 16/16Four bins at 100% and sixteen bins at 100% print the same number and are not the same amount of evidence. The same is true at the bottom: one unhit bin and four unhit bins are both "0%", and only one of them is a morning's work.
So the record keeps two integers:
$cov{$grp}{$bin} = $hits; # per bin
# and a rate, when a report wants one, is derived at the point of printingA stored rate is a derived value pretending to be a measurement, and it cannot be merged (24.4), cannot be summed (24.5), and cannot be compared across nights without the risk in the next two sections.
4. A Percentage Cannot Be Inverted
Vendor coverage summaries often print one line per group with a rate on it. Parse that line and keep only the rate, and this is what you have:
a vendor summary line reporting 93.75% is consistent with:
15 / 16 = 93.75%
30 / 32 = 93.75%
150 / 160 = 93.75%
1500 / 1600 = 93.75%
15/16 and 30/32 are both 93.75%: the same rate, 16 bins more, and the
number of HOLES went from 1 to 2 -- it doubled, invisiblyOne rate, many underlying pairs. And the pair that matters most for a trend is the one in the last line: somebody added a cover group, the number of unhit bins doubled, and the headline percentage did not move at all.
This is why the capture keeps all three fields when the line offers them:
if (m{^\s*(\S+)\s+(\d+)\s*/\s*(\d+)\s+([\d.]+)%}) {
$cov{$1} = { hit => $2, total => $3 }; # the rate is not stored
}The printed rate is still useful for one thing — checking that your arithmetic agrees with the tool's. The captured output does exactly that comparison, and on this data every group agrees. A disagreement would mean the tool is counting something you are not, which is a finding rather than a rounding detail.
5. Two Decimal Places Is Not A Stable Key
TEST HIT/TOTAL ROUNDED TRUNCATED
arb_single 25/41 60.98 60.97 DISAGREE
arb_smoke 25/41 60.98 60.97 DISAGREE
tests where rounding and truncation print different numbers: 225 of 41 is 60.9756…, so a tool that rounds prints 60.98 and a tool that truncates prints 60.97. Both are honest renderings of one number, and 2 of 7 tests in this regression land on such a value.
The consequence is in the trend report. Compare printed percentages night over night and a tool upgrade that changes the rounding mode makes every borderline group look like it moved, while a group that genuinely gained one bin in two thousand looks unchanged. Chapter 18.8 is the general version of this: a tool version difference you did not account for becomes data you believe.
Compare the integers. hit and total are exact, they diff cleanly, and the percentage is for the human reading the row — not for the code deciding whether something changed.
6. What To Remember
- Coverage is a delimited block, so parse it with one
insideflag (18.6) and ignore everything outside it. - Measured: 7 of 36 logs had no coverage block at all, and they are exactly the runs that never finished.
- Absent coverage is a third state (18.9) — do not merge "never measured" in as zeros.
- Measured:
if ($hits)dropped 16 of 41 bins, and all 16 were holes. - A hit count of 0 is a measurement, and it is the one the report exists to show. Store the value, then filter with
> 0when you want hits. - A hash of only non-zero bins rolls up to "100%" — the easiest way to report closure you do not have (24.5).
exists,definedand truth are three different tests (6.4);0is the value that separates them.- Store
$hits + 0so the value is settled as a number at capture time, not at the first comparison (2.6). - Measured: 0.00% was both 0/1 and 0/4; 100.00% was both 4/4 and 16/16. The denominator is not in the rate.
- Store hit and total; derive the rate at the point of printing.
- Measured: 93.75% fits 15/16, 30/32, 150/160 and 1500/1600 — a rate cannot be inverted.
- A constant rate hid the holes doubling from 1 to 2.
- Use the printed rate only to check your arithmetic against the tool's; a disagreement is a finding, not a rounding detail.
- Measured: 25/41 prints 60.98 rounded and 60.97 truncated, on 2 of 7 tests.
- Diff the integers, not the printed percentage — otherwise a rounding-mode change looks like a coverage change (18.8).
Chapter 24.4 takes these per-bin counts and merges them. Once several runs each contribute hits to the same bin, a bin that no run has ever hit becomes a question the structure can answer — and a bin that only one run hits becomes a different kind of warning.
Where this fits
Part of the Perl curriculum.