Perl Part II · Module 24 · Coverage and Regression Reporting
Coverage Closure Roll-Ups
Closure is the number everybody asks for and the number hardest to state honestly. There are two defensible ways to compute it that differ by seventeen points on the same data, there is a bin in this design that guarantees it can never reach a hundred percent, and the one output that lets anybody do something about it is not a percentage at all. This chapter computes all of it from the raw hit and total counts and shows what each number is actually claiming.
Intermediate11 min readPerlPart IIRegression Reporting
Part II · Module 24 · Chapter 24.5 · Coverage Closure Roll-Ups
1. Per-Group, Then Overall
Because 24.4 kept the counts rather than rates, the roll-up is a sum and one division.
use strict;
use warnings;
sub read_coverage {
my ($path) = @_;
my (%cov, $grp, $inside);
open my $fh, "<", $path or return \%cov;
while (my $l = <$fh>) {
if ($l =~ /^--- COVERAGE ---/) { $inside = 1; next }
if ($l =~ /^--- END COVERAGE ---/) { $inside = 0; next }
next unless $inside;
$grp = $1 if $l =~ /^COVERGROUP\s+(\S+)/;
$cov{$grp}{$1} = $2 + 0 if $l =~ /^\s+BIN\s+(\S+)\s+(\d+)\s*$/;
}
close $fh;
return \%cov;
}
# the merged store of 24.4
my %merged;
for my $p (sort glob("logs/*.log")) {
my $c = read_coverage($p);
for my $g (keys %$c) {
$merged{$g}{$_} += $c->{$g}{$_} for keys %{ $c->{$g} };
}
}
# ---- per-group closure from the raw counts
my %grp;
for my $g (sort keys %merged) {
my @b = keys %{ $merged{$g} };
$grp{$g} = { hit => scalar(grep { $merged{$g}{$_} > 0 } @b), total => scalar @b };
}
print "==== per-group closure ====\n";
printf "%-16s %-10s %s\n", "GROUP", "HIT/TOTAL", "CLOSURE";
printf "%-16s %-10s %6.2f%%\n", $_, "$grp{$_}{hit}/$grp{$_}{total}",
100 * $grp{$_}{hit} / $grp{$_}{total} for sort keys %grp;
my ($H, $T) = (0, 0);
$H += $grp{$_}{hit}, $T += $grp{$_}{total} for keys %grp;
printf "\noverall (hits over bins) : %d/%d = %.2f%%\n", $H, $T, 100 * $H / $T;
# ---- 1. the mean of the group rates is a different number
print "\n==== two ways to say 'overall' ====\n";
my $mean = 0;
$mean += 100 * $grp{$_}{hit} / $grp{$_}{total} for keys %grp;
$mean /= keys %grp;
printf "mean of the %d group rates : %.2f%%\n", scalar keys %grp, $mean;
printf "bins hit over bins total : %.2f%%\n", 100 * $H / $T;
printf "difference : %.2f points\n", abs($mean - 100 * $H / $T);
print "the mean gives a 1-bin group the same weight as a 16-bin group:\n";
printf " %-16s %2d bin(s) weight in the mean %.0f%% weight in the ratio %.1f%%\n",
$_, $grp{$_}{total}, 100 / keys(%grp), 100 * $grp{$_}{total} / $T
for sort { $grp{$a}{total} <=> $grp{$b}{total} or $a cmp $b } keys %grp;
# ---- 2. the illegal bin caps the number forever
print "\n==== the bin that must never be hit ====\n";
my @illegal = grep { /illegal/ } keys %grp;
printf "groups whose name marks them illegal: %s\n", join " ", @illegal;
my ($hi, $ti) = (0, 0);
$hi += $grp{$_}{hit}, $ti += $grp{$_}{total} for grep { !/illegal/ } keys %grp;
printf "with illegal bins in the denominator : %d/%d = %.2f%%\n", $H, $T, 100*$H/$T;
printf "with illegal bins excluded : %d/%d = %.2f%%\n", $hi, $ti, 100*$hi/$ti;
printf "the gap : %.2f points\n", 100*$hi/$ti - 100*$H/$T;
printf "\nthe illegal bin has %d hits in every run, which is CORRECT.\n",
$merged{cg_illegal}{grant_without_request};
print "So the first number can never reach 100%, and the last 2.44 points of\n";
print "'missing coverage' is a bin the design is required to leave alone.\n";
print "Chasing it means writing a test to break the thing it checks.\n";
# ---- 3. the holes, by name
print "\n==== the unhit bins, by name ====\n";
my $n = 0;
for my $g (sort keys %merged) {
for my $b (sort keys %{ $merged{$g} }) {
next if $merged{$g}{$b} > 0;
$n++;
printf " %-16s %-24s %s\n", $g, $b,
$g =~ /illegal/ ? "ILLEGAL -- must stay at zero"
: "a hole: nothing has reached it";
}
}
my ($n_illegal, $n_real) = (0, 0);
for my $g (keys %merged) {
for my $b (keys %{ $merged{$g} }) {
next if $merged{$g}{$b} > 0;
$g =~ /illegal/ ? $n_illegal++ : $n_real++;
}
}
printf "%d unhit bin(s): %d illegal, %d real hole(s)\n", $n, $n_illegal, $n_real;
print "a percentage cannot be acted on; a bin name can\n";
# ---- 4. per-test closure, and what a longer run bought
print "\n==== per-test closure ====\n";
my %bytest;
for my $p (sort glob("logs/*.log")) {
(my $run = $p) =~ s{logs/|\.log}{}g;
my ($t) = $run =~ /^(.*)\.\d+$/;
my $c = read_coverage($p);
next unless %$c;
for my $g (keys %$c) {
$bytest{$t}{$g}{$_} += $c->{$g}{$_} for keys %{ $c->{$g} };
}
}
printf "%-18s %-10s %-9s %s\n", "TEST", "HIT/TOTAL", "CLOSURE", "SIMTIME of one run";
for my $t (sort keys %bytest) {
my ($h, $tt) = (0, 0);
for my $g (keys %{ $bytest{$t} }) {
my @b = keys %{ $bytest{$t}{$g} };
$tt += @b;
$h += grep { $bytest{$t}{$g}{$_} > 0 } @b;
}
my $st = "-";
if (open my $fh, "<", "logs/$t.7.log") {
while (<$fh>) { $st = $1 if /^SIMTIME:\s+(\d+)/ }
close $fh;
}
printf "%-18s %-10s %6.2f%% %s\n", $t, "$h/$tt", 100 * $h / $tt,
$st eq "-" ? "(run never finished)" : "$st ns";
}==== per-group closure ====
GROUP HIT/TOTAL CLOSURE
cg_grant_id 4/4 100.00%
cg_illegal 0/1 0.00%
cg_ptr_x_grant 16/16 100.00%
cg_req_pattern 16/16 100.00%
cg_wait_ge3 4/4 100.00%
overall (hits over bins) : 40/41 = 97.56%
==== two ways to say 'overall' ====
mean of the 5 group rates : 80.00%
bins hit over bins total : 97.56%
difference : 17.56 points
the mean gives a 1-bin group the same weight as a 16-bin group:
cg_illegal 1 bin(s) weight in the mean 20% weight in the ratio 2.4%
cg_grant_id 4 bin(s) weight in the mean 20% weight in the ratio 9.8%
cg_wait_ge3 4 bin(s) weight in the mean 20% weight in the ratio 9.8%
cg_ptr_x_grant 16 bin(s) weight in the mean 20% weight in the ratio 39.0%
cg_req_pattern 16 bin(s) weight in the mean 20% weight in the ratio 39.0%
==== the bin that must never be hit ====
groups whose name marks them illegal: cg_illegal
with illegal bins in the denominator : 40/41 = 97.56%
with illegal bins excluded : 40/40 = 100.00%
the gap : 2.44 points
the illegal bin has 0 hits in every run, which is CORRECT.
So the first number can never reach 100%, and the last 2.44 points of
'missing coverage' is a bin the design is required to leave alone.
Chasing it means writing a test to break the thing it checks.
==== the unhit bins, by name ====
cg_illegal grant_without_request ILLEGAL -- must stay at zero
1 unhit bin(s): 1 illegal, 0 real hole(s)
a percentage cannot be acted on; a bin name can
==== per-test closure ====
TEST HIT/TOTAL CLOSURE SIMTIME of one run
arb_light 26/41 63.41% 325 ns
arb_random 40/41 97.56% 4025 ns
arb_random_short 39/41 95.12% 1025 ns
arb_single 25/41 60.98% 20025 ns
arb_smoke 25/41 60.98% 4025 ns
arb_soak 40/41 97.56% 4025 ns
arb_stress 39/41 95.12% 4025 ns
arb_trunc 40/41 97.56% (run never finished)GROUP HIT/TOTAL CLOSURE
cg_grant_id 4/4 100.00%
cg_illegal 0/1 0.00%
cg_ptr_x_grant 16/16 100.00%
cg_req_pattern 16/16 100.00%
cg_wait_ge3 4/4 100.00%
overall (hits over bins) : 40/41 = 97.56%The registry's sketch for this chapter does the sum with a comma operator inside a statement modifier:
$h += $_->{hit}, $t += $_->{total} for values %cov;That works — the comma operator evaluates both sides — and it is one line. Two statements in a block are easier to read and equally short, so the choice is taste rather than correctness. What is not taste is the next section.
2. Two Ways To Say "Overall", Seventeen Points Apart
mean of the 5 group rates : 80.00%
bins hit over bins total : 97.56%
difference : 17.56 pointsBoth are computed correctly from the same data, and they are not the same number. The reason is in the weights:
cg_illegal 1 bin(s) weight in the mean 20% weight in the ratio 2.4%
cg_req_pattern 16 bin(s) weight in the mean 20% weight in the ratio 39.0%The mean of group rates gives every group one vote. A group with one bin and a group with sixteen count the same, so a single-bin group at 0% drags the headline down by a fifth. The ratio weights each group by how many bins it has, which is what "how much of the coverage model have we reached" means.
So: sum the counts, then divide once. Averaging rates is Chapter 21.4's shape again — the number you choose determines the conclusion, and the obvious one is not always the right one. If a per-group mean is genuinely wanted, call it that and never call it closure.
3. The Bin That Must Never Be Hit
Here is the awkward part of every real coverage model:
with illegal bins in the denominator : 40/41 = 97.56%
with illegal bins excluded : 40/40 = 100.00%
the gap : 2.44 points
the illegal bin has 0 hits in every run, which is CORRECT.cg_illegal counts a grant asserted when nobody is requesting the bus. It is supposed to stay at zero. It is a check, not a goal.
Leave it in the denominator and closure is capped at 97.56% for the life of the project. Someone will eventually be asked to close that last 2.44% — and the only way to do it is to write a test that makes the design break the property the bin exists to catch.
So an illegal bin needs to be out of the closure denominator and reported separately, as a pass or a fail rather than a percentage:
| kind of bin | belongs in closure | what a non-zero count means |
|---|---|---|
| a normal bin | yes | progress |
| an illegal bin | no | a failure, reported like any other |
| an unreachable bin | no, once proven | the proof needs reviewing |
The code needs a way to tell them apart, and a naming convention is the cheapest one that survives — cg_illegal_*, or a checked-in list of illegal bin names. Inferring it from "has always been zero" cannot work: that is also the description of a hole.
4. The Holes, By Name
cg_illegal grant_without_request ILLEGAL -- must stay at zero
1 unhit bin(s): 1 illegal, 0 real hole(s)
a percentage cannot be acted on; a bin name canThis merged regression has no real holes at all. Every legal bin has been reached by something. That is a better statement than "97.56%", and it is the statement the bin names make possible.
The general point is the one the registry's own description makes: a percentage does not tell anyone what to write next. cg_req_pattern/req_0011 does. So the roll-up prints three things, in this order of usefulness:
The unhit bin names, grouped and labelled by kind. This is the work list.
The per-group counts, so a group that is 2/16 is distinguishable from one that is 15/16.
The overall rate, last, for the slide.
5. What A Longer Run Bought
The per-test table answers a question nobody usually asks of a coverage report:
TEST HIT/TOTAL CLOSURE SIMTIME of one run
arb_light 26/41 63.41% 325 ns
arb_single 25/41 60.98% 20025 ns
arb_smoke 25/41 60.98% 4025 nsarb_single ran five times longer than arb_smoke and reached exactly the same 25 bins. The two tests use the same stimulus profile and differ only in length, so five times the simulation bought nothing whatsoever.
And the sharper version: arb_light ran for 325 ns — sixty-one times less than arb_single — and reached one bin more. The shortest test in the regression covers more of the model than the longest.
The conclusion is not that long tests are useless. It is that the stimulus profile is the limit, not the run length. A profile that drives one requester at a time will never reach a two-requester bin, and it will not reach it faster by running all night. Chapter 24.4's saturation curve said the same thing from the other direction: adding a test moved the union, adding a seed did not.
That is also why this table belongs in the nightly report. A row showing hours of simulation next to unchanged coverage is the cheapest argument for writing a new test that anybody will read.
6. What To Remember
- Roll up from the raw counts: sum
hit, sumtotal, divide once. - Measured: the mean of the group rates is 80.00% and bins-over-bins is 97.56% — 17.56 points apart on identical data.
- The mean gives a 1-bin group the same weight as a 16-bin group (20% against 2.4%).
- Averaging rates is not closure. If you want a per-group mean, call it that (21.4).
- Measured: an illegal bin caps closure at 97.56% forever — 40/41 with it, 40/40 without.
- An illegal bin is a check, not a goal; zero is the correct answer, and a non-zero count is a failure to report like any other.
- Closing that last 2.44% would mean writing a test to break the property the bin exists to catch.
- Keep illegal and unreachable bins out of the closure denominator, and mark them by naming convention or a checked-in list.
- You cannot infer "illegal" from "always zero" — that is also the description of a hole.
- Measured: 1 unhit bin, 1 illegal, 0 real holes — a better sentence than any percentage.
- Report the unhit bin names first, then the per-group counts, then the overall rate.
- Measured: arb_single ran 5x longer than arb_smoke for exactly the same 25 bins.
- Measured: arb_light ran 61x less than arb_single and reached one bin more.
- The stimulus profile is the limit, not the run length (and 24.4 said so from the other side).
- Put simulated time next to coverage in the nightly report — hours of simulation beside unchanged coverage is the cheapest argument for a new test.
Chapter 24.6 starts printing. Everything from here reads the structures already built, and the first job is the report an engineer opens at nine in the morning — which means sorted, aligned, and short enough to read before the coffee lands.
Where this fits
Part of the Perl curriculum.