Perl Part II · Module 24 · Coverage and Regression Reporting
Text Summaries for Mail and Terminal
A mail body gets read in a preview pane and a chat message gets read in a notification, which means the digest has one line to be useful in and about ten to be complete in. That constraint turns out to be clarifying: it forces a decision about which denominator the headline means, and it rules out the pass percentage entirely — because on one of the subsets measured here, the percentage cannot be computed at all.
Intermediate10 min readPerlPart IIRegression Reporting
Part II · Module 24 · Chapter 24.7 · Text Summaries for Mail and Terminal
1. The Digest
use strict;
use warnings;
my %RANK = (FAIL => 0, TIMEOUT => 1, UNKNOWN => 2, PASS => 3);
sub read_log {
my ($path) = @_;
my %r = (verdict => "UNKNOWN", errors => 0, complete => 0, budget => 0);
open my $fh, "<", $path or return \%r;
while (my $l = <$fh>) {
$r{test} = $1 if $l =~ /^TEST:\s+(\S+)/;
$r{seed} = $1 if $l =~ /^SEED:\s+(\d+)/;
$r{budget} = 1 if $l =~ /^UVM_FATAL\s.*\[BUDGET\]/;
$r{msg}++ if $l =~ /^UVM_ERROR\s+\S+\(\d+\)/;
$r{complete} = 1 if $l =~ /^\*\* Report counts by severity/;
$r{sum} = $1 if $l =~ /^UVM_ERROR\s*:\s*(\d+)/;
$r{first} = $1 if !$r{first} && $l =~ /^UVM_ERROR\s+\S+\(\d+\).*\[(\w+)\]/;
$r{verdict} = "PASS" if $l =~ /^TEST PASSED/;
$r{verdict} = "FAIL" if $l =~ /^TEST FAILED/;
}
close $fh;
$r{errors} = $r{complete} ? $r{sum} : ($r{msg} || 0);
$r{verdict} = "TIMEOUT" if $r{budget} && $r{verdict} eq "UNKNOWN";
return \%r;
}
# one digest builder, used for the whole night and for subsets
sub digest {
my ($label, @recs) = @_;
my %byt;
push @{ $byt{ $_->{test} } }, $_ for @recs;
my (%rv, @bad);
$rv{ $_->{verdict} }++ for @recs;
for my $t (sort keys %byt) {
my ($w) = sort { $RANK{$a} <=> $RANK{$b} }
map { $_->{verdict} } @{ $byt{$t} };
next if $w eq "PASS";
my @seeds = map { $_->{seed} }
grep { $_->{verdict} eq $w } @{ $byt{$t} };
my ($first) = grep { defined }
map { $_->{first} } @{ $byt{$t} };
push @bad, sprintf "%-18s %-8s seed %-14s %s", $t, $w,
join(",", sort { $a <=> $b } @seeds), $first // "-";
}
my $decided = grep { $_->{verdict} =~ /^(PASS|FAIL)$/ } @recs;
my @lines;
# the headline has to survive being the only line anybody reads
push @lines, sprintf "REGRESSION %s: %d of %d tests PASS, %d of %d runs PASS",
$label, scalar(keys %byt) - @bad, scalar keys %byt,
$rv{PASS} // 0, scalar @recs;
push @lines, sprintf " %s", join " ",
map { "$_ $rv{$_}" } sort { $RANK{$a} <=> $RANK{$b} } keys %rv;
push @lines, sprintf " %d of %d runs reached a verdict",
$decided, scalar @recs;
push @lines, "", " NOT PASSING:" if @bad;
push @lines, " $_" for @bad;
return @lines;
}
my @all = grep { defined $_->{test} } map { read_log($_) } sort glob("logs/*.log");
print "==== the digest ====\n";
print "$_\n" for digest("2026-10-13", @all);
# ---- 1. which denominator does the headline mean
print "\n==== 'PASS' over what? ====\n";
my %byt;
push @{ $byt{ $_->{test} } }, $_ for @all;
my $pass_runs = grep { $_->{verdict} eq "PASS" } @all;
my $pass_tests = grep { my $t = $_;
!grep { $_->{verdict} ne "PASS" } @{ $byt{$t} } } keys %byt;
printf "runs passing : %2d of %-2d = %3.0f%%\n",
$pass_runs, scalar @all, 100 * $pass_runs / @all;
printf "tests passing : %2d of %-2d = %3.0f%%\n",
$pass_tests, scalar keys %byt, 100 * $pass_tests / keys %byt;
printf "a headline saying just '%d/%d PASS' could be either\n",
$pass_runs, scalar @all;
# ---- 2. the length of the thing people actually see
print "\n==== how long is it ====\n";
my @lines = digest("2026-10-13", @all);
printf "digest lines : %d\n", scalar @lines;
printf "headline characters : %d\n", length $lines[0];
printf "one row per RUN instead : %d lines\n", scalar(@all) + 3;
printf "the digest lists one row per non-passing TEST, not per run\n";
# ---- 3. the subset where nothing reached a verdict
print "\n==== the night nothing finished ====\n";
my @only_budget = grep { $_->{test} eq "arb_budget" } @all;
printf "a subset of %d run(s), none of which reached a verdict:\n",
scalar @only_budget;
print "$_\n" for digest("budget-only", @only_budget);
my $dec = grep { $_->{verdict} =~ /^(PASS|FAIL)$/ } @only_budget;
printf "\nruns that reached a verdict: %d\n", $dec;
printf "a pass RATE over decided runs would be %s\n",
$dec ? sprintf("%.0f%%", 0) : "a division by zero";
print "so the headline reports COUNTS, never a bare rate: 0 of 4 is honest\n";
print "and '0%' over an empty denominator is not a number at all\n";
# ---- 4. the same builder on last night, for comparison
print "\n==== last night, same builder ====\n";
my @prev = grep { defined $_->{test} } map { read_log($_) } sort glob("prev/*.log");
print "$_\n" for (digest("2026-10-12", @prev))[0 .. 2];
# ---- 5. the digest has to be stable, or a diff of digests is noise
print "\n==== is the digest reproducible? ====\n";
my @build1 = digest("2026-10-13", @all);
my @build2 = digest("2026-10-13", @all);
my $same = (join("\n", @build1) eq join("\n", @build2)) ? "identical" : "DIFFERENT";
printf "two builds from the same records: %s\n", $same;
# the same digest with a wall-clock stamp folded into the headline
use Time::HiRes qw(time);
sub digest_stamped {
my @l = digest(@_);
$l[0] .= sprintf " [generated %.6f]", time();
return @l;
}
my @stamp1 = digest_stamped("2026-10-13", @all);
my @stamp2 = digest_stamped("2026-10-13", @all);
my $stamped_same = (join("\n", @stamp1) eq join("\n", @stamp2)) ? "identical" : "DIFFERENT";
printf "with a wall-clock stamp folded in: %s\n", $stamped_same;
# the values are masked here so this page is reproducible; what matters is
# how many lines changed, not what they changed to
my $changed = 0;
for my $i (0 .. $#stamp1) { $changed++ if $stamp1[$i] ne $stamp2[$i] }
printf "lines that differ between two builds of the SAME data: %d of %d\n",
$changed, scalar @stamp1;
printf "and all %d of them are the headline, which also carries the result\n",
$changed;
print "so a diff of two digests reports a change in the regression when the\n";
print "only thing that changed was the clock\n";
print "volatile data goes on its OWN line, so the result lines stay comparable\n";
# which lines change when the DATA changes, as it should
my @prev2 = grep { defined $_->{test} } map { read_log($_) } sort glob("prev/*.log");
my @last_night = digest("2026-10-12", @prev2);
my $n = 0;
for my $i (0 .. $#build1) {
$n++ if !defined $last_night[$i] || $build1[$i] ne $last_night[$i];
}
printf "\ntonight against last night: %d of %d lines differ\n", $n, scalar @build1;
printf "which is the signal 24.11 is built on\n";==== the digest ====
REGRESSION 2026-10-13: 2 of 9 tests PASS, 11 of 36 runs PASS
FAIL 18 TIMEOUT 4 UNKNOWN 3 PASS 11
29 of 36 runs reached a verdict
NOT PASSING:
arb_budget TIMEOUT seed 1,7,42,1234 GNT_VEC
arb_light FAIL seed 7 GNT_VEC
arb_random FAIL seed 1,7,42,1234 GNT_VEC
arb_random_short FAIL seed 1,7,42,1234 GNT_VEC
arb_soak FAIL seed 1,7,42,1234 GNT_VEC
arb_stress FAIL seed 1,7,42,1234 GNT_VEC
arb_trunc FAIL seed 1 GNT_VEC
==== 'PASS' over what? ====
runs passing : 11 of 36 = 31%
tests passing : 2 of 9 = 22%
a headline saying just '11/36 PASS' could be either
==== how long is it ====
digest lines : 12
headline characters : 60
one row per RUN instead : 39 lines
the digest lists one row per non-passing TEST, not per run
==== the night nothing finished ====
a subset of 4 run(s), none of which reached a verdict:
REGRESSION budget-only: 0 of 1 tests PASS, 0 of 4 runs PASS
TIMEOUT 4
0 of 4 runs reached a verdict
NOT PASSING:
arb_budget TIMEOUT seed 1,7,42,1234 GNT_VEC
runs that reached a verdict: 0
a pass RATE over decided runs would be a division by zero
so the headline reports COUNTS, never a bare rate: 0 of 4 is honest
and '0%' over an empty denominator is not a number at all
==== last night, same builder ====
REGRESSION 2026-10-12: 1 of 9 tests PASS, 4 of 36 runs PASS
FAIL 25 TIMEOUT 4 UNKNOWN 3 PASS 4
29 of 36 runs reached a verdict
==== is the digest reproducible? ====
two builds from the same records: identical
with a wall-clock stamp folded in: DIFFERENT
lines that differ between two builds of the SAME data: 1 of 12
and all 1 of them are the headline, which also carries the result
so a diff of two digests reports a change in the regression when the
only thing that changed was the clock
volatile data goes on its OWN line, so the result lines stay comparable
tonight against last night: 9 of 12 lines differ
which is the signal 24.11 is built onThe whole digest is this:
REGRESSION 2026-10-13: 2 of 9 tests PASS, 11 of 36 runs PASS
FAIL 18 TIMEOUT 4 UNKNOWN 3 PASS 11
29 of 36 runs reached a verdict
NOT PASSING:
arb_budget TIMEOUT seed 1,7,42,1234 GNT_VEC
arb_light FAIL seed 7 GNT_VECdigest() returns a list of lines rather than printing them. That is the one structural decision worth making here: the same function then fills a mail body, a chat message, a terminal and a file, and section 5 reuses it on a different night's data without a change. A sub that prints can only ever be used one way (10.1, 10.6).
2. "PASS" Over What?
The registry's sketch for this chapter has the headline as $pass/$total PASS. Here is the problem with that:
runs passing : 11 of 36 = 31%
tests passing : 2 of 9 = 22%
a headline saying just '11/36 PASS' could be eitherTwo true answers, nine points apart, and "11/36" does not say which it is. Worse, the two carry different meanings: 31% of runs passing sounds survivable, and 2 of 9 tests passing does not.
So the headline names both and labels each:
REGRESSION 2026-10-13: 2 of 9 tests PASS, 11 of 36 runs PASSThat line is 60 characters, which fits a mail subject and a chat preview with room to spare. It is worth measuring, because the headline is the only part of the digest most recipients will ever see — and a headline truncated mid-number is worse than no headline.
This is Chapter 24.6's filtered-totals finding one step further out. There the denominator went missing; here there are two denominators and the reader cannot tell which one you used.
3. How Long It Is
digest lines : 12
headline characters : 60
one row per RUN instead : 39 linesTwelve lines against thirty-nine. The saving comes entirely from listing one row per non-passing test rather than per run, and nothing actionable is lost in the process:
arb_random FAIL seed 1,7,42,1234 GNT_VECThe verdict, every failing seed, and the first error id, on one line. Chapter 24.2's rule about storing what a rerun needs is why that seed list exists to be printed; Chapter 21.10's triage context is why the error id is worth the column.
And the passing tests are not listed at all — only counted. A digest that lists what went well is a digest nobody finishes reading.
4. The Night Nothing Finished
This is the measurement that decides the headline's format. Take the subset of runs that hit the cycle budget:
a subset of 4 run(s), none of which reached a verdict:
REGRESSION budget-only: 0 of 1 tests PASS, 0 of 4 runs PASS
TIMEOUT 4
0 of 4 runs reached a verdict
runs that reached a verdict: 0
a pass RATE over decided runs would be a division by zeroZero runs reached a verdict, so a pass rate over decided runs has nothing to divide by. Perl will die with Illegal division by zero and take the nightly mail with it — and the fix of defaulting to 0% is worse, because 0% is a claim and there is nothing to make a claim about.
Counts do not have this problem. 0 of 4 runs PASS is true, informative, and computable on every input including an empty one. The second line, 0 of 4 runs reached a verdict, is what tells the reader that this is a dead regression rather than a broken design.
The general rule, and it applies to every report in this module: the headline carries counts; a rate is for a chart. A rate needs a guard, a denominator, and a sentence explaining it, and a headline has room for none of the three.
5. One Builder, Both Nights
REGRESSION 2026-10-12: 1 of 9 tests PASS, 4 of 36 runs PASS
FAIL 25 TIMEOUT 4 UNKNOWN 3 PASS 4Same function, last night's logs. One of nine tests passed then and two pass now, and twenty-five runs failed then against eighteen now.
Something clearly improved, and the digest cannot say what — that is Chapter 24.11's job, and it needs exactly this: two nights rendered by the same code so that any difference between them is a difference in the data. A digest built by hand for tonight and adapted for the comparison would make every change ambiguous between the regression and the report.
6. The Digest Has To Be Stable
One last property, and it is the one that makes Chapter 24.11 possible at all.
two builds from the same records: identical
with a wall-clock stamp folded in: DIFFERENT
lines that differ between two builds of the SAME data: 1 of 12
and all 1 of them are the headline, which also carries the resultTwo builds from the same records are byte-identical — because every loop in digest() runs over a sorted list, so hash order never reaches the output (6.6, 17.8).
Then the second measurement: fold a generation timestamp into the headline and one line of twelve differs between two builds of identical data — and it is the headline, which is also the line carrying the result. A diff of two digests now reports a change in the regression when the only thing that changed was the clock.
The fix is placement, not omission. A timestamp is genuinely useful; it just cannot share a line with a number anyone will compare:
REGRESSION 2026-10-13: 2 of 9 tests PASS, 11 of 36 runs PASS
generated 03:14 on build07, pid 44121Volatile data goes on its own line. Then a comparison skips that line and every remaining line is pure result data. The same discipline applies to a host name, a process id, a run directory and an absolute path — all useful, none comparable.
And when the data really does change, the digest says so loudly:
tonight against last night: 9 of 12 lines differNine of twelve lines moved between the two nights. That is a real signal, and it is only legible because the other three lines did not move for any reason other than the data.
7. What To Remember
- Return lines, do not print them. One
digest()then fills a mail body, a chat message, a terminal and a file (10.1, 10.6). - Measured: 31% of runs passed and 22% of tests passed — a bare
11/36 PASSis ambiguous between two true answers. - Name both denominators in the headline and label each:
2 of 9 tests PASS, 11 of 36 runs PASS. - Measured: that headline is 60 characters — it fits a mail subject and a chat preview, and the headline is the only line most recipients read.
- Measured: 12 lines against 39 for one row per run.
- List one row per non-passing TEST, with its verdict, every failing seed, and the first error id (24.2, 21.10).
- Do not list the passing tests — count them. A digest that lists good news is one nobody finishes.
- Measured: on a subset where 0 of 4 runs reached a verdict, a pass rate is a division by zero.
- Defaulting that to
0%is worse than dying — 0% is a claim about nothing. - The headline carries counts; a rate is for a chart. Counts are computable on every input, including an empty one.
- Add the line
N of M runs reached a verdict— it distinguishes a dead regression from a broken design. - Build both nights with the same function, so any difference between them is a difference in the data (24.11).
- Measured: two builds from the same records are byte-identical, because every loop runs over a sorted list (6.6, 17.8).
- Measured: folding a timestamp into the headline made 1 of 12 lines differ between two builds of the same data — and it was the line carrying the result.
- Volatile data goes on its own line: timestamp, host, pid, run directory, absolute path. Useful, never comparable.
- Measured: 9 of 12 lines differ between the two nights — a real signal, legible only because nothing else moves.
Chapter 24.8 writes the same records out for a machine instead of a person. That sounds easier and is not: a comma inside a field, a quote inside a field, and the difference between an empty value and a missing one each have exactly one correct handling, and the obvious code gets all three wrong.
Where this fits
Part of the Perl curriculum.