Perl Part II · Module 24 · Coverage and Regression Reporting
Aggregating Across Tests and Seeds
A test that runs under four seeds has four results, and the moment it has four results there are two different questions to ask about it: did it ever fail, and how many of its seeds failed. They are not the same question, they do not have the same answer, and a regression report that answers only one of them will mislead somebody before the week is out. This chapter builds the structure that answers both — and measures three ways the obvious one-liner answers neither.
Intermediate11 min readPerlPart IIRegression Reporting
Part II · Module 24 · Chapter 24.2 · Aggregating Across Tests and Seeds
1. Keep Every Run
Chapter 24.1 ended with a structure that threw away twenty-seven of thirty-six runs. The fix is one character:
push @{ $runs{$test} }, $record; # not $runs{$test} = $recordThat is Chapter 7.7's hash of arrays, and push @{ ... } autovivifies the array on first use, so no initialisation is needed.
use strict;
use warnings;
# the per-run record of 24.1, as a loadable sub
do "./record.pl_lib" if 0;
sub read_log {
my ($path) = @_;
my %r = (log => $path, verdict => "UNKNOWN", errors => 0, complete => 0,
budget => 0, test => undef, seed => undef, simtime => undef);
open my $fh, "<", $path or return \%r;
while (my $l = <$fh>) {
$r{test} = $1 if $l =~ /^TEST:\s+(\S+)/;
$r{seed} = $1 if $l =~ /^SEED:\s+(\d+)/;
$r{simtime} = $1 if $l =~ /^SIMTIME:\s+(\d+)/;
$r{budget} = 1 if $l =~ /^UVM_FATAL\s.*\[BUDGET\]/;
$r{msg}++ if $l =~ /^UVM_ERROR\s+\S+\(\d+\)/;
$r{complete} = 1 if $l =~ /^\*\* Report counts by severity/;
$r{sum} = $1 if $l =~ /^UVM_ERROR\s*:\s*(\d+)/;
$r{verdict} = "PASS" if $l =~ /^TEST PASSED/;
$r{verdict} = "FAIL" if $l =~ /^TEST FAILED/;
}
close $fh;
$r{errors} = $r{complete} ? $r{sum} : ($r{msg} || 0);
$r{verdict} = "TIMEOUT" if $r{budget} && $r{verdict} eq "UNKNOWN";
return \%r;
}
# ---- keep every run: a list of runs under each test name
my %runs;
for my $p (sort glob("logs/*.log")) {
my $r = read_log($p);
next unless defined $r->{test};
push @{ $runs{ $r->{test} } }, $r;
}
my $total = 0;
$total += @{ $runs{$_} } for keys %runs;
printf "tests: %d runs kept: %d\n", scalar keys %runs, $total;
# ---- the two questions the registry names
print "\n==== the two questions ====\n";
printf "%-18s %-10s %-12s %s\n", "TEST", "EVER FAIL", "SEEDS FAILED", "the four seeds";
for my $t (sort keys %runs) {
my @r = @{ $runs{$t} };
my $ever = (grep { $_->{verdict} eq "FAIL" } @r) ? "yes" : "no";
my $nfail = grep { $_->{verdict} eq "FAIL" } @r;
printf "%-18s %-10s %-12s %s\n", $t, $ever, "$nfail of " . scalar(@r),
join " ", map { "$_->{seed}=$_->{verdict}" }
sort { $a->{seed} <=> $b->{seed} } @r;
}
# ---- the trap in "not a pass"
print "\n==== 'ne PASS' is not the same as 'FAIL' ====\n";
printf "%-18s %-12s %-12s %s\n", "TEST", "ne PASS", "eq FAIL", "difference is";
my $diffs = 0;
for my $t (sort keys %runs) {
my @r = @{ $runs{$t} };
my $np = grep { $_->{verdict} ne "PASS" } @r;
my $ef = grep { $_->{verdict} eq "FAIL" } @r;
next if $np == $ef;
$diffs++;
my %v; $v{ $_->{verdict} }++ for grep { $_->{verdict} !~ /^(PASS|FAIL)$/ } @r;
printf "%-18s %-12s %-12s %s\n", $t, "$np of " . scalar(@r),
"$ef of " . scalar(@r), join ",", map { "$_ x$v{$_}" } sort keys %v;
}
printf "tests where the two rules disagree: %d of %d\n", $diffs, scalar keys %runs;
# ---- grep in scalar context returns a COUNT, not the seeds
print "\n==== what grep gives you depends on the context ====\n";
my %fail_seeds;
for my $t (sort keys %runs) {
# the obvious line, and the name promises seeds
$fail_seeds{$t} = grep { $_->{verdict} eq "FAIL" } @{ $runs{$t} };
}
printf "%-18s %-14s %s\n", "TEST", "fail_seeds", "the seeds that actually failed";
for my $t (sort keys %runs) {
my @seeds = map { $_->{seed} }
grep { $_->{verdict} eq "FAIL" } @{ $runs{$t} };
printf "%-18s %-14s %s\n", $t, $fail_seeds{$t},
@seeds ? join(" ", sort { $a <=> $b } @seeds) : "(none)";
}
printf "\nthe seeds this regression ran: %s\n",
join " ", sort { $a <=> $b }
do { my %s; $s{ $_->{seed} } = 1 for map { @$_ } values %runs; keys %s };
printf "\narb_light stored %s, and %s IS one of those seeds -- but the seed that\n",
$fail_seeds{arb_light}, $fail_seeds{arb_light};
printf "actually failed is %s. A count and a seed are the same kind of number.\n",
join ",", map { $_->{seed} }
grep { $_->{verdict} eq "FAIL" } @{ $runs{arb_light} };
print "\nthe list form, which is what a rerun needs:\n";
for my $t (sort keys %runs) {
my @seeds = map { $_->{seed} }
grep { $_->{verdict} eq "FAIL" } @{ $runs{$t} };
next unless @seeds;
printf " %-18s seeds %s\n", $t, join(",", sort { $a <=> $b } @seeds);
}
# ---- runs against tests: both true, different stories
print "\n==== aggregate: runs against tests ====\n";
my %rv;
for my $t (keys %runs) { $rv{ $_->{verdict} }++ for @{ $runs{$t} } }
printf "runs : %s (total %d)\n",
join(" ", map { "$_=$rv{$_}" } sort keys %rv), $total;
my (@allpass, @anyfail, @noverdict);
for my $t (sort keys %runs) {
my @r = @{ $runs{$t} };
my $p = grep { $_->{verdict} eq "PASS" } @r;
my $f = grep { $_->{verdict} eq "FAIL" } @r;
if ($f) { push @anyfail, $t }
elsif ($p == @r) { push @allpass, $t }
else { push @noverdict, $t }
}
printf "tests : all seeds pass %d, at least one FAIL %d, no verdict at all %d\n",
scalar @allpass, scalar @anyfail, scalar @noverdict;
printf " all pass : %s\n", join " ", @allpass;
printf " at least 1 FAIL : %s\n", join " ", @anyfail;
printf " no verdict : %s\n", join " ", @noverdict;
printf "\n'%d of %d runs failed' = %.0f%% but '%d of %d tests failed' = %.0f%%\n",
$rv{FAIL} // 0, $total, 100 * ($rv{FAIL} // 0) / $total,
scalar @anyfail, scalar keys %runs, 100 * @anyfail / keys %runs;
# ---- a pass rate needs a stated denominator
print "\n==== a pass rate needs its denominator stated ====\n";
printf "%-18s %-12s %-14s %s\n", "TEST", "PASS/ALL", "PASS/DECIDED", "note";
for my $t (sort keys %runs) {
my @r = @{ $runs{$t} };
my $p = grep { $_->{verdict} eq "PASS" } @r;
my @dec = grep { $_->{verdict} =~ /^(PASS|FAIL)$/ } @r;
my $d = @dec ? sprintf("%3.0f%%", 100 * $p / @dec) : " n/a";
my $note = @dec == @r ? ""
: sprintf "%d run(s) never reached a verdict", @r - @dec;
printf "%-18s %3.0f%% %-14s %s\n",
$t, 100 * $p / @r, $d, $note;
}tests: 9 runs kept: 36
==== the two questions ====
TEST EVER FAIL SEEDS FAILED the four seeds
arb_budget no 0 of 4 1=TIMEOUT 7=TIMEOUT 42=TIMEOUT 1234=TIMEOUT
arb_light yes 1 of 4 1=PASS 7=FAIL 42=PASS 1234=PASS
arb_random yes 4 of 4 1=FAIL 7=FAIL 42=FAIL 1234=FAIL
arb_random_short yes 4 of 4 1=FAIL 7=FAIL 42=FAIL 1234=FAIL
arb_single no 0 of 4 1=PASS 7=PASS 42=PASS 1234=PASS
arb_smoke no 0 of 4 1=PASS 7=PASS 42=PASS 1234=PASS
arb_soak yes 4 of 4 1=FAIL 7=FAIL 42=FAIL 1234=FAIL
arb_stress yes 4 of 4 1=FAIL 7=FAIL 42=FAIL 1234=FAIL
arb_trunc yes 1 of 4 1=FAIL 7=UNKNOWN 42=UNKNOWN 1234=UNKNOWN
==== 'ne PASS' is not the same as 'FAIL' ====
TEST ne PASS eq FAIL difference is
arb_budget 4 of 4 0 of 4 TIMEOUT x4
arb_trunc 4 of 4 1 of 4 UNKNOWN x3
tests where the two rules disagree: 2 of 9
==== what grep gives you depends on the context ====
TEST fail_seeds the seeds that actually failed
arb_budget 0 (none)
arb_light 1 7
arb_random 4 1 7 42 1234
arb_random_short 4 1 7 42 1234
arb_single 0 (none)
arb_smoke 0 (none)
arb_soak 4 1 7 42 1234
arb_stress 4 1 7 42 1234
arb_trunc 1 1
the seeds this regression ran: 1 7 42 1234
arb_light stored 1, and 1 IS one of those seeds -- but the seed that
actually failed is 7. A count and a seed are the same kind of number.
the list form, which is what a rerun needs:
arb_light seeds 7
arb_random seeds 1,7,42,1234
arb_random_short seeds 1,7,42,1234
arb_soak seeds 1,7,42,1234
arb_stress seeds 1,7,42,1234
arb_trunc seeds 1
==== aggregate: runs against tests ====
runs : FAIL=18 PASS=11 TIMEOUT=4 UNKNOWN=3 (total 36)
tests : all seeds pass 2, at least one FAIL 6, no verdict at all 1
all pass : arb_single arb_smoke
at least 1 FAIL : arb_light arb_random arb_random_short arb_soak arb_stress arb_trunc
no verdict : arb_budget
'18 of 36 runs failed' = 50% but '6 of 9 tests failed' = 67%
==== a pass rate needs its denominator stated ====
TEST PASS/ALL PASS/DECIDED note
arb_budget 0% n/a 4 run(s) never reached a verdict
arb_light 75% 75%
arb_random 0% 0%
arb_random_short 0% 0%
arb_single 100% 100%
arb_smoke 100% 100%
arb_soak 0% 0%
arb_stress 0% 0%
arb_trunc 0% 0% 3 run(s) never reached a verdict2. The Two Questions
TEST EVER FAIL SEEDS FAILED the four seeds
arb_light yes 1 of 4 1=PASS 7=FAIL 42=PASS 1234=PASS
arb_random yes 4 of 4 1=FAIL 7=FAIL 42=FAIL 1234=FAILBoth rows say yes. They are not the same situation, and the difference is the whole reason both numbers belong in the structure.
arb_random fails on every seed. It is reproducible, it will reproduce for whoever picks it up, and it is the one to debug first.
arb_light fails on one seed in four. It is just as real — the same bug, caught less often — but it arrives wearing the costume of a flaky test, and "three of four passed" is the sentence that gets it closed without a fix.
So the aggregate keeps both:
my $ever = (grep { $_->{verdict} eq "FAIL" } @r) ? "yes" : "no";
my $nfail = grep { $_->{verdict} eq "FAIL" } @r;An ever-failed flag on its own over-reports — one bad seed looks the same as a test that never works. A pass rate on its own under-reports — 75% reads like success. The pair is informative in a way neither half is.
3. "Not A Pass" Is Not "FAIL"
Here is where the obvious filter goes wrong, and it goes wrong in the direction that wastes someone's time:
TEST ne PASS eq FAIL difference is
arb_budget 4 of 4 0 of 4 TIMEOUT x4
arb_trunc 4 of 4 1 of 4 UNKNOWN x3
tests where the two rules disagree: 2 of 9Under ne 'PASS', arb_budget has four failing seeds out of four — and not one of its runs ever reached a verdict. All four hit the cycle budget and stopped. The test did not fail; it did not finish. Reporting it as four failures sends someone to debug an arbiter when the thing to fix is a budget.
arb_trunc is the same mistake in smaller print: one genuine failure and three runs the farm cut short, reported as four failures.
This is Chapter 18.9's rule in the aggregation layer. Not-passing covers at least four states, and they call for different actions:
| verdict | what happened | who fixes it |
|---|---|---|
FAIL | the test ran and the design was wrong | design or verification |
TIMEOUT | the run was stopped before it finished | whoever sets the budget |
UNKNOWN | no verdict was recorded at all | whoever owns the farm |
PASS | the test ran and agreed | nobody |
So count the verdict you mean. grep { $_->{verdict} eq "FAIL" } is three characters longer than ne "PASS" and it is the difference between a bug report and a false accusation.
4. A Count And A Seed Are The Same Kind Of Number
The registry's own sketch for this chapter stores the failing seeds like this:
$fail_seeds{$test} = grep { $_->{verdict} ne 'PASS' } @{ $runs{$test} };and the measurement shows what that actually stores:
TEST fail_seeds the seeds that actually failed
arb_light 1 7
arb_random 4 1 7 42 1234
the seeds this regression ran: 1 7 42 1234
arb_light stored 1, and 1 IS one of those seeds -- but the seed that
actually failed is 7. A count and a seed are the same kind of number.grep in scalar context returns a count (5.10, and Chapter 2.9 on context itself). The variable is called fail_seeds, the stored value is 1, and 1 is a seed this regression really ran — so the obvious reading is "seed 1 failed". Seed 1 passed. The failing seed is 7.
Nothing warns you. The name says seeds, the value is a count, and the two are the same type. Perl will not help here and neither will a test that only checks the count is non-zero.
Two habits make it a non-issue:
Name the variable after what it holds — fail_count for a number, fail_seeds for a list. A plural name holding a scalar is the bug announcing itself.
Store what a rerun needs. The count tells you how bad it is; the seed list tells you what to type:
arb_light seeds 7
arb_random seeds 1,7,42,12345. Runs, Tests, And The Denominator
The same thirty-six logs support two headlines:
runs : FAIL=18 PASS=11 TIMEOUT=4 UNKNOWN=3 (total 36)
tests : all seeds pass 2, at least one FAIL 6, no verdict at all 1
'18 of 36 runs failed' = 50% but '6 of 9 tests failed' = 67%Both are true. Fifty percent and sixty-seven percent describe the same night. Counting runs weights a test by how many seeds it got; counting tests weights every test equally. Neither is wrong and a report that does not say which it used is. Chapter 21.4 met this exact shape counting error occurrences against logs containing errors, and Chapter 20.5 counting text instances against elaborated instances.
The denominator needs the same care:
TEST PASS/ALL PASS/DECIDED note
arb_budget 0% n/a 4 run(s) never reached a verdict
arb_trunc 0% 0% 3 run(s) never reached a verdictarb_budget is "0% passing", which reads as completely broken. What actually happened is that nothing it did reached a verdict at all. A rate whose numerator can only be PASS and whose denominator includes runs that could never be a PASS is not a rate — it is a mixture of two different facts.
So report the undecided count alongside, or drop undecided runs from the denominator and say you did. n/a is a better answer than 0% when there is nothing to take a percentage of.
6. What To Remember
push @{ $runs{$test} }, $record— a hash of arrays (7.7), autovivified, keeps every run.- A test with several seeds raises two questions: did it ever fail, and how many seeds failed. Store both.
- Measured:
arb_randomfailed 4 of 4 andarb_light1 of 4. Same bug, and only one of them looks like a bug. - An ever-failed flag alone over-reports; a pass rate alone invites the word "flaky."
- Measured:
ne 'PASS'andeq 'FAIL'disagree on 2 of 9 tests. arb_budgetreported 4 failing seeds of 4 while NOT ONE run reached a verdict — all four hit the cycle budget. That sends someone to debug the wrong thing.- Not-passing covers FAIL, TIMEOUT and UNKNOWN, and each has a different owner (18.9, 21.8).
- Count the verdict you mean —
eq "FAIL"is three characters more thanne "PASS". - Measured:
$fail_seeds{arb_light}stored1becausegrepwas in scalar context (5.10, 2.9) — and 1 is a real seed here, while the seed that failed is 7. - A count and a seed are the same kind of number, so nothing warns you.
- Name the variable after what it holds; a plural name holding a scalar is the bug announcing itself.
- Store what a rerun needs — the seed list, not only the count.
- Measured: 50% of runs failed and 67% of tests failed, from one night. Both true; say which you mean (21.4, 20.5).
- State the denominator.
arb_budgetat "0% passing" is really "nothing reached a verdict" —n/abeats0%.
Chapter 24.3 turns to the other half of a nightly report. The same logs carry coverage, and a coverage number has the same two problems in a new costume: a percentage whose denominator nobody checked, and bins that nothing ever hit.
Where this fits
Part of the Perl curriculum.