Perl Part II · Module 24 · Coverage and Regression Reporting
A Per-Bin Coverage Store
One run's coverage is a snapshot; a regression's coverage is the union of all of them. Merging is a one-line operation, and the interesting part is everything the merged structure can then be asked: which bins nothing ever reached, which bins depend on a single test, and how many of the runs were buying coverage at all. On this regression the last question has an uncomfortable answer.
Intermediate11 min readPerlPart IIRegression Reporting
Part II · Module 24 · Chapter 24.4 · A Per-Bin Coverage Store
1. The Merge
The store is a hash of hashes (7.8) and the merge is one line:
$merged{$group}{$bin} += $cov->{$group}{$bin};+= on a key that does not exist yet starts from zero and creates both levels on the way — Chapter 7.9's automatic creation of nested data, which is what makes a two-level store this short.
use strict;
use warnings;
sub read_coverage {
my ($path) = @_;
my (%cov, $grp, $inside);
open my $fh, "<", $path or return \%cov;
while (my $l = <$fh>) {
if ($l =~ /^--- COVERAGE ---/) { $inside = 1; next }
if ($l =~ /^--- END COVERAGE ---/) { $inside = 0; next }
next unless $inside;
$grp = $1 if $l =~ /^COVERGROUP\s+(\S+)/;
$cov{$grp}{$1} = $2 + 0 if $l =~ /^\s+BIN\s+(\S+)\s+(\d+)\s*$/;
}
close $fh;
return \%cov;
}
# ---- merge every run into one store. The hits ADD; the bin list is a union.
my (%merged, %hit_by, %runs_with_block);
my @logs = sort glob("logs/*.log");
for my $p (@logs) {
my $c = read_coverage($p);
next unless %$c;
(my $run = $p) =~ s{logs/|\.log}{}g;
my ($test) = $run =~ /^(.*)\.\d+$/;
$runs_with_block{$run} = 1;
for my $g (keys %$c) {
for my $b (keys %{ $c->{$g} }) {
$merged{$g}{$b} += $c->{$g}{$b};
push @{ $hit_by{"$g/$b"} }, $run if $c->{$g}{$b} > 0;
}
}
}
printf "runs merged: %d of %d\n", scalar keys %runs_with_block, scalar @logs;
my ($bins, $holes) = (0, 0);
for my $g (sort keys %merged) {
my @b = sort keys %{ $merged{$g} };
my $h = grep { $merged{$g}{$_} > 0 } @b;
$bins += @b; $holes += @b - $h;
printf " %-16s %2d/%-2d bins hit total hits %d\n",
$g, $h, scalar @b, eval { my $s = 0; $s += $merged{$g}{$_} for @b; $s };
}
printf "merged: %d of %d bins hit, %d hole(s)\n", $bins - $holes, $bins, $holes;
# ---- 1. the bins no run has ever hit
print "\n==== bins no run has ever hit ====\n";
for my $g (sort keys %merged) {
for my $b (sort keys %{ $merged{$g} }) {
printf " %-16s %-24s %d hits across all %d runs\n",
$g, $b, $merged{$g}{$b}, scalar keys %runs_with_block
if $merged{$g}{$b} == 0;
}
}
# ---- 2. the expected bin list has to come from somewhere
print "\n==== where the expected bin list comes from ====\n";
my %union_all; # every bin the logs mention, hit or not
my %union_nonzero; # only the bins some run actually hit
for my $p (@logs) {
my $c = read_coverage($p);
for my $g (keys %$c) {
for my $b (keys %{ $c->{$g} }) {
$union_all{"$g/$b"} = 1;
$union_nonzero{"$g/$b"} = 1 if $c->{$g}{$b} > 0;
}
}
}
printf "bins the logs mention at all : %d\n", scalar keys %union_all;
printf "bins some run actually hit : %d\n", scalar keys %union_nonzero;
printf "holes found against the full list : %d\n",
scalar(keys %union_all) - scalar(keys %union_nonzero);
printf "holes found against the non-zero list : %d\n", 0;
print "a parser that stored only non-zero bins builds the second list, and\n";
print "then every hole is missing from the very set it would be found in\n";
printf " that list reports %d of %d bins hit = 100%%\n",
scalar keys %union_nonzero, scalar keys %union_nonzero;
# ---- 3. how many runs reach each bin
print "\n==== how many runs reach each bin ====\n";
my %dist;
$dist{ scalar @{ $hit_by{$_} } }++ for keys %hit_by;
my $nruns = keys %runs_with_block;
printf "%-14s %s\n", "RUNS HITTING", "BINS";
for my $n (sort { $a <=> $b } keys %dist) {
printf " %2d of %-2d %2d %s\n", $n, $nruns, $dist{$n}, "#" x $dist{$n};
}
my @thin = sort grep { @{ $hit_by{$_} } <= 5 } keys %hit_by;
printf "\nbins reached by 5 or fewer of the %d runs: %d\n", $nruns, scalar @thin;
for my $k (sort { @{ $hit_by{$a} } <=> @{ $hit_by{$b} } || $a cmp $b } @thin) {
printf " %-32s %2d run(s): %s\n", $k, scalar @{ $hit_by{$k} },
join " ", @{ $hit_by{$k} };
}
print "those are the bins one retired test or one changed seed can take away\n"
if @thin;
# ---- 4. and how many TESTS reach each bin
print "\n==== how many tests reach each bin ====\n";
my %tests_for;
for my $k (keys %hit_by) {
my %t;
$t{ $_ =~ s/\.\d+$//r } = 1 for @{ $hit_by{$k} };
$tests_for{$k} = [sort keys %t];
}
my %total_tests;
$total_tests{ $_ =~ s/\.\d+$//r } = 1 for keys %runs_with_block;
my %tdist;
$tdist{ scalar @{ $tests_for{$_} } }++ for keys %tests_for;
printf "tests contributing coverage: %d\n", scalar keys %total_tests;
for my $n (sort { $a <=> $b } keys %tdist) {
printf " reached by %d test(s): %2d bin(s)\n", $n, $tdist{$n};
}
my @single = sort grep { @{ $tests_for{$_} } == 1 } keys %tests_for;
printf "bins only ONE test reaches: %d\n", scalar @single;
print " nothing here depends on a single test, which is the answer you want\n"
unless @single;
printf " %-32s only %s\n", $_, $tests_for{$_}[0] for @single;
# ---- 5. how many runs it takes to reach the merged total
print "\n==== how much the merge is actually worth ====\n";
print "(runs in sorted order; a different order gives a different curve,\n";
print " but the total and the count of runs that add nothing do not change)\n";
my %seen;
my ($i, $added_nothing, $prev) = (0, 0, 0);
for my $r (sort keys %runs_with_block) {
my $c = read_coverage("logs/$r.log");
for my $g (keys %$c) {
for my $b (keys %{ $c->{$g} }) {
$seen{"$g/$b"} = 1 if $c->{$g}{$b} > 0;
}
}
$i++;
my $now = keys %seen;
my $gained = $i > 1 ? $now - $prev : 0;
$added_nothing++ if $i > 1 && !$gained;
# print only the runs that changed the union, and say how many did not
printf " after %2d run(s): %2d bins hit %s\n", $i, $now,
$i == 1 ? "(the first run alone)" : "+$gained"
if $i == 1 || $gained;
$prev = $now;
}
printf "\nall %d runs reach %d bins, and %d of them added NOTHING to the union\n",
$i, scalar keys %seen, $added_nothing;
printf "that is %.0f%% of the regression contributing no coverage it did not already have\n",
100 * $added_nothing / $i;runs merged: 29 of 36
cg_grant_id 4/4 bins hit total hits 12527
cg_illegal 0/1 bins hit total hits 0
cg_ptr_x_grant 16/16 bins hit total hits 12527
cg_req_pattern 16/16 bins hit total hits 15320
cg_wait_ge3 4/4 bins hit total hits 655
merged: 40 of 41 bins hit, 1 hole(s)
==== bins no run has ever hit ====
cg_illegal grant_without_request 0 hits across all 29 runs
==== where the expected bin list comes from ====
bins the logs mention at all : 41
bins some run actually hit : 40
holes found against the full list : 1
holes found against the non-zero list : 0
a parser that stored only non-zero bins builds the second list, and
then every hole is missing from the very set it would be found in
that list reports 40 of 40 bins hit = 100%
==== how many runs reach each bin ====
RUNS HITTING BINS
13 of 29 1 #
14 of 29 1 #
15 of 29 2 ##
16 of 29 3 ###
17 of 29 5 #####
18 of 29 2 ##
20 of 29 1 #
25 of 29 4 ####
26 of 29 5 #####
27 of 29 9 #########
28 of 29 3 ###
29 of 29 4 ####
bins reached by 5 or fewer of the 29 runs: 0
==== how many tests reach each bin ====
tests contributing coverage: 8
reached by 4 test(s): 1 bin(s)
reached by 5 test(s): 10 bin(s)
reached by 6 test(s): 4 bin(s)
reached by 7 test(s): 4 bin(s)
reached by 8 test(s): 21 bin(s)
bins only ONE test reaches: 0
nothing here depends on a single test, which is the answer you want
==== how much the merge is actually worth ====
(runs in sorted order; a different order gives a different curve,
but the total and the count of runs that add nothing do not change)
after 1 run(s): 15 bins hit (the first run alone)
after 2 run(s): 22 bins hit +7
after 3 run(s): 23 bins hit +1
after 4 run(s): 26 bins hit +3
after 5 run(s): 40 bins hit +14
all 29 runs reach 40 bins, and 24 of them added NOTHING to the union
that is 83% of the regression contributing no coverage it did not already haveWhy += rather than a flag. For closure — "has anything hit this bin" — a boolean would be enough. The summed count costs nothing and answers two further questions that a boolean cannot: how thinly a bin is covered, and whether the hits all came from one place. Sections 3 and 4 are both built on it.
2. The Hole, And Where The Bin List Comes From
Merged across 29 runs, one bin is still at zero:
merged: 40 of 41 bins hit, 1 hole(s)
cg_illegal grant_without_request 0 hits across all 29 runsThat bin is grant_without_request — a grant asserted when nobody is asking for the bus. It is an illegal bin: zero is the correct and permanent answer, and Chapter 24.5 deals with what that does to a closure percentage. What matters here is that the store can see it, and whether it can depends entirely on one decision:
bins the logs mention at all : 41
bins some run actually hit : 40
holes found against the full list : 1
holes found against the non-zero list : 0
that list reports 40 of 40 bins hit = 100%A hole can only be found against a list that contains it. Build the expected-bin list from the bins that were hit, and the hole is absent from the only set it could have been found in — and the roll-up reports 40 of 40, one hundred percent, with no warning of any kind.
This is Chapter 24.3's "a zero hit count is data" with the consequence attached. The two lists differ by exactly the thing you were looking for.
So the expected list has to come from somewhere that is not the hits:
From the logs, including zero-hit bins — which works when the simulator prints every bin whether or not it was hit, as the one here does.
From the source — the cover group declarations, parsed separately. This is the only option that also catches a bin the simulator never mentioned, which happens when a cover group was not even instantiated.
From a checked-in list — the most explicit, and the one that notices a bin disappearing. A bin that vanishes from the log is invisible to the first two methods and looks exactly like progress.
3. How Many Runs Reach Each Bin
With the counts kept, the store answers a question a percentage cannot:
RUNS HITTING BINS
13 of 29 1 #
17 of 29 5 #####
27 of 29 9 #########
29 of 29 4 ####
bins reached by 5 or fewer of the 29 runs: 0The thinnest bin in this regression is reached by 13 of 29 runs, and nothing is reached by five or fewer. Per test it is the same story: one bin is reached by four of the eight tests, twenty-one by all eight, and no bin depends on a single test.
That is an empty list, and an empty list here is a result. A bin reached by one test is a bin that disappears the day somebody retires that test — and the overall coverage number will barely move, so nothing will flag it. The check is three lines and worth running precisely because you hope it prints nothing:
my @single = grep { @{ $tests_for{$_} } == 1 } keys %tests_for;4. What The Merge Is Actually Worth
The last measurement is the one to take back to whoever owns the regression:
after 1 run(s): 15 bins hit (the first run alone)
after 2 run(s): 22 bins hit +7
after 3 run(s): 23 bins hit +1
after 4 run(s): 26 bins hit +3
after 5 run(s): 40 bins hit +14
all 29 runs reach 40 bins, and 24 of them added NOTHING to the union
that is 83% of the regression contributing no coverage it did not already haveFive runs reach every reachable bin. The other twenty-four add nothing.
Two things that honestly follows, and one that does not.
It does not mean the 24 runs are waste. Chapter 24.2 is the reason they exist: the seeds are what make a failure reproducible, and arb_light failed on exactly one of its four seeds. Drop the seeds and you drop the only evidence of that bug. Coverage is not the only thing a run buys.
It does mean "we run 29 jobs for coverage" is not true here, and if the regression grows by adding seeds in the hope of coverage, the measurement says that hope is unfounded. Adding a test moved the union; adding a seed did not.
And the curve depends on the order, which the output says in its own first line. A different order gives a different curve — but the total and the count of runs that add nothing are order-independent, and those are the two numbers worth quoting.
5. What To Remember
- The store is a hash of hashes and the merge is
$merged{$g}{$b} += $hits(7.8), with both levels created on first use (7.9). - Sum the hits rather than setting a flag — closure needs a boolean, but thinness and single-source coverage need the counts.
- Measured: 29 of 36 runs had coverage; merged they reach 40 of 41 bins.
- The one hole is an illegal bin —
grant_without_request, zero in all 29 runs, and zero is the right answer (24.5). - A hole can only be found against a list that contains it.
- Measured: the full bin list is 41 and the hit list is 40, so searching the hit-derived list finds 0 holes and reports 40 of 40 = 100%.
- Get the expected bin list from the logs including zero-hit bins, from the cover group source, or from a checked-in list — the last is the only one that notices a bin disappearing.
- A bin that vanishes from the log looks exactly like progress.
- Measured: the thinnest bin is reached by 13 of 29 runs, and none by five or fewer.
- Measured: no bin depends on a single test — an empty list, and the result you want.
- Check for single-test bins anyway: retiring that test takes the bin away and the coverage number barely moves.
- Measured: 5 runs reach all 40 reachable bins, and 24 of 29 runs added nothing — 83%.
- That does not make the 24 runs waste — they are the seeds that make a failure reproducible (24.2).
- It does mean adding seeds is not buying coverage here; adding a test moved the union, adding a seed did not.
- The curve depends on run order; the total and the added-nothing count do not.
Chapter 24.5 turns the merged store into the number people actually ask for. The illegal bin is about to become a problem, because it means no amount of verification can ever reach one hundred percent.
Where this fits
Part of the Perl curriculum.