Perl Part II · Module 24 · Coverage and Regression Reporting
The Nightly Regression Report Table
This is the artefact the regression publishes — the thing linked in a mail, pasted into a channel and compared against yesterday. That last use is what sets the requirements: a row order that cannot move, totals that check themselves, and one column per verdict so no reader has to guess what a single word meant. All three are measurable, and this chapter measures them.
Intermediate11 min readPerlPart IIRegression Reporting
Part II · Module 24 · Chapter 24.9 · The Nightly Regression Report Table
1. One Row Per Test, One Column Per Verdict
use strict;
use warnings;
my @VERDICTS = qw(PASS FAIL TIMEOUT UNKNOWN);
my %RANK = (FAIL => 0, TIMEOUT => 1, UNKNOWN => 2, PASS => 3);
sub read_log {
my ($path) = @_;
my %r = (log => $path, verdict => "UNKNOWN", errors => 0,
complete => 0, budget => 0);
open my $fh, "<", $path or return \%r;
while (my $l = <$fh>) {
$r{test} = $1 if $l =~ /^TEST:\s+(\S+)/;
$r{seed} = $1 if $l =~ /^SEED:\s+(\d+)/;
$r{simtime} = $1 if $l =~ /^SIMTIME:\s+(\d+)/;
$r{budget} = 1 if $l =~ /^UVM_FATAL\s.*\[BUDGET\]/;
if ($l =~ /^UVM_ERROR\s+\S+\(\d+\)/) {
$r{msg}++;
$r{first_id} = $1 if !defined $r{first_id} && $l =~ /\[(\w+)\]/;
}
$r{complete} = 1 if $l =~ /^\*\* Report counts by severity/;
$r{sum} = $1 if $l =~ /^UVM_ERROR\s*:\s*(\d+)/;
$r{verdict} = "PASS" if $l =~ /^TEST PASSED/;
$r{verdict} = "FAIL" if $l =~ /^TEST FAILED/;
}
close $fh;
$r{errors} = $r{complete} ? $r{sum} : ($r{msg} || 0);
$r{verdict} = "TIMEOUT" if $r{budget} && $r{verdict} eq "UNKNOWN";
return \%r;
}
# The table builder. It takes the records in ANY order and must produce the
# same bytes, so every loop inside it sorts.
sub build_table {
my ($label, @recs) = @_;
my %by;
push @{ $by{ $_->{test} } }, $_ for @recs;
my $w = 4;
for (keys %by) { $w = length if length > $w }
my @out;
push @out, sprintf "NIGHTLY REGRESSION %s", $label;
push @out, sprintf "%-*s %7s %7s %7s %7s %6s %-8s %s",
$w, "TEST", @VERDICTS, "RUNS", "VERDICT", "FIRST ERROR";
push @out, "-" x ($w + 55);
my %col;
my $runs = 0;
for my $t (sort keys %by) {
my @r = @{ $by{$t} };
my %n;
$n{ $_->{verdict} }++ for @r;
$col{$_} += $n{$_} // 0 for @VERDICTS;
$runs += @r;
my ($worst) = sort { $RANK{$a} <=> $RANK{$b} }
map { $_->{verdict} } @r;
# the first error of a run that FAILED, not of any run: a passing
# run has no first error and an UNKNOWN run's may be incomplete
my ($fid) = grep { defined }
map { $_->{first_id} }
sort { $a->{seed} <=> $b->{seed} }
grep { $_->{verdict} eq "FAIL" } @r;
$fid ||= (grep { defined }
map { $_->{first_id} }
sort { $a->{seed} <=> $b->{seed} } @r)[0];
push @out, sprintf "%-*s %7d %7d %7d %7d %6d %-8s %s",
$w, $t, (map { $n{$_} // 0 } @VERDICTS), scalar @r, $worst,
$fid // "-";
}
push @out, "-" x ($w + 55);
push @out, sprintf "%-*s %7d %7d %7d %7d %6d",
$w, "TOTAL", (map { $col{$_} } @VERDICTS), $runs;
# the table checks itself
my $colsum = 0;
$colsum += $col{$_} for @VERDICTS;
push @out, sprintf "check: columns sum to %d, runs %d -- %s",
$colsum, $runs, $colsum == $runs ? "agree" : "DISAGREE";
my @odd = grep { my @r = @{ $by{$_} }; @r != 4 } sort keys %by;
push @out, sprintf "check: tests not having exactly 4 seeds: %d%s",
scalar @odd, @odd ? " (" . join(" ", @odd) . ")" : "";
return @out;
}
my @sorted = grep { defined $_->{test} } map { read_log($_) } sort glob("logs/*.log");
print "$_\n" for build_table("2026-10-13", @sorted);
# ---- 1. the table must not depend on the order the logs were read in
print "\n==== order independence ====\n";
my @reversed = reverse @sorted;
my @shuffled = @sorted[ map { ($_ * 7) % scalar @sorted } 0 .. $#sorted ];
my $t_sorted = join "\n", build_table("x", @sorted);
my $t_rev = join "\n", build_table("x", @reversed);
my $t_strided = join "\n", build_table("x", @shuffled);
printf "records read sorted vs reversed : %s\n", $t_sorted eq $t_rev ? "identical" : "DIFFER";
printf "records read sorted vs strided : %s\n", $t_sorted eq $t_strided ? "identical" : "DIFFER";
printf "distinct inputs compared : %d orderings\n", 3;
print "every loop in build_table sorts, so the input order cannot reach the\n";
print "output -- which is what makes a night-over-night diff meaningful\n";
# ---- 2. where the first error comes from
print "\n==== the first error id ====\n";
my %by;
push @{ $by{ $_->{test} } }, $_ for @sorted;
printf "%-18s %-28s %s\n", "TEST", "FIRST ID PER SEED", "REPORTED";
for my $t (sort keys %by) {
my @r = sort { $a->{seed} <=> $b->{seed} } @{ $by{$t} };
my ($fid) = grep { defined } map { $_->{first_id} }
grep { $_->{verdict} eq "FAIL" } @r;
printf "%-18s %-28s %s\n", $t,
join(" ", map { ($_->{first_id} // "-") } @r), $fid // "-";
}
print "a PASSING run has no first error, so taking 'the first defined id over\n";
print "all seeds' would report an id for a test whose failing seed is not the\n";
print "one it came from. The id is taken from a FAILING run.\n";
print "\narb_budget is the exception and shows why the rule has two steps: it\n";
print "has NO failing run, so the strict rule reports '-', while the table\n";
print "above shows GNT_VEC -- the errors it had already found when the cycle\n";
print "budget stopped it. Both are right, and the VERDICT column is what tells\n";
print "a reader which case the id came from.\n";
# ---- 3. the id names which bug, which is the point of the column
print "\n==== the same column, last night ====\n";
my @prev = grep { defined $_->{test} } map { read_log($_) } sort glob("prev/*.log");
my %pid;
for my $r (grep { $_->{verdict} eq "FAIL" } @prev) {
$pid{ $r->{first_id} // "-" }++;
}
my %tid;
for my $r (grep { $_->{verdict} eq "FAIL" } @sorted) {
$tid{ $r->{first_id} // "-" }++;
}
printf "tonight, first error ids over failing runs : %s\n",
join " ", map { "$_ x$tid{$_}" } sort keys %tid;
printf "last night, first error ids over failing runs: %s\n",
join " ", map { "$_ x$pid{$_}" } sort keys %pid;
print "two nights, two different ids, and neither night has both -- so the\n";
print "column says WHICH failure, not merely that there was one\n";NIGHTLY REGRESSION 2026-10-13
TEST PASS FAIL TIMEOUT UNKNOWN RUNS VERDICT FIRST ERROR
-----------------------------------------------------------------------
arb_budget 0 0 4 0 4 TIMEOUT GNT_VEC
arb_light 3 1 0 0 4 FAIL GNT_VEC
arb_random 0 4 0 0 4 FAIL GNT_VEC
arb_random_short 0 4 0 0 4 FAIL GNT_VEC
arb_single 4 0 0 0 4 PASS -
arb_smoke 4 0 0 0 4 PASS -
arb_soak 0 4 0 0 4 FAIL GNT_VEC
arb_stress 0 4 0 0 4 FAIL GNT_VEC
arb_trunc 0 1 0 3 4 FAIL GNT_VEC
-----------------------------------------------------------------------
TOTAL 11 18 4 3 36
check: columns sum to 36, runs 36 -- agree
check: tests not having exactly 4 seeds: 0
==== order independence ====
records read sorted vs reversed : identical
records read sorted vs strided : identical
distinct inputs compared : 3 orderings
every loop in build_table sorts, so the input order cannot reach the
output -- which is what makes a night-over-night diff meaningful
==== the first error id ====
TEST FIRST ID PER SEED REPORTED
arb_budget GNT_VEC GNT_VEC GNT_VEC GNT_VEC -
arb_light - GNT_VEC - - GNT_VEC
arb_random GNT_VEC GNT_VEC GNT_VEC GNT_VEC GNT_VEC
arb_random_short GNT_VEC GNT_VEC GNT_VEC GNT_VEC GNT_VEC
arb_single - - - - -
arb_smoke - - - - -
arb_soak GNT_VEC GNT_VEC GNT_VEC GNT_VEC GNT_VEC
arb_stress GNT_VEC GNT_VEC GNT_VEC GNT_VEC GNT_VEC
arb_trunc GNT_VEC GNT_VEC GNT_VEC GNT_VEC GNT_VEC
a PASSING run has no first error, so taking 'the first defined id over
all seeds' would report an id for a test whose failing seed is not the
one it came from. The id is taken from a FAILING run.
arb_budget is the exception and shows why the rule has two steps: it
has NO failing run, so the strict rule reports '-', while the table
above shows GNT_VEC -- the errors it had already found when the cycle
budget stopped it. Both are right, and the VERDICT column is what tells
a reader which case the id came from.
==== the same column, last night ====
tonight, first error ids over failing runs : GNT_VEC x18
last night, first error ids over failing runs: GNT_VALID x25
two nights, two different ids, and neither night has both -- so the
column says WHICH failure, not merely that there was oneTEST PASS FAIL TIMEOUT UNKNOWN RUNS VERDICT FIRST ERROR
arb_budget 0 0 4 0 4 TIMEOUT GNT_VEC
arb_light 3 1 0 0 4 FAIL GNT_VEC
arb_single 4 0 0 0 4 PASS -
arb_trunc 0 1 0 3 4 FAIL GNT_VEC
TOTAL 11 18 4 3 36Chapter 24.6 printed one row per run for somebody scanning a terminal. This is the published version, and the difference that matters is a count in every verdict column rather than a single word.
Look at arb_light: PASS 3 FAIL 1. That is Chapter 24.2's pair — it failed, and three quarters of it passed — on one row, in a form nobody can misread. A VERDICT column alone would say FAIL and invite an argument; a pass rate alone would say 75% and invite a shrug.
And arb_trunc reads FAIL 1 UNKNOWN 3, which says in four characters that this test has an infrastructure problem as well as a design one.
2. The Table Checks Itself
check: columns sum to 36, runs 36 -- agree
check: tests not having exactly 4 seeds: 0Two lines, and they need no input the table does not already have.
The first adds the four verdict columns and compares the sum against the run total. They can only disagree if a run was counted under no verdict or under two — which is what happens when a new verdict string appears and the column list was not updated.
The second asserts the shape of the matrix: every test should have run under every seed. A missing log, a job the farm never started, a duplicated file — all of them show up here as a row with three seeds or five, and nowhere else. Chapter 23.9 found the same hazard in a per-corner store, where one missing cell turned a violation into a pass.
Put both checks in the published output, not in a test. A check that lives in a test suite tells you the code is right; a check printed in the artefact tells every reader that this particular table is right, including the reader who finds it in a mail from three weeks ago.
3. The Table Cannot Depend On Input Order
records read sorted vs reversed : identical
records read sorted vs strided : identical
distinct inputs compared : 3 orderingsThree different input orderings, one output. The builder sorts inside every loop, so the order the logs arrived in cannot reach the page.
This is not a nicety. Chapter 24.11 diffs tonight's table against last night's, and a diff is only informative if every difference means something changed in the regression. One unsorted keys %hash anywhere in the builder and the diff fills with rows that moved for no reason — and the real change hides among them (6.6, 17.8).
Worth noting what the test actually does: it builds the table three times from the same records in different orders and compares the bytes. Comparing the data structures would prove less, because the defect being hunted lives in the printing.
4. Where The First Error Id Comes From
TEST FIRST ID PER SEED REPORTED
arb_light - GNT_VEC - - GNT_VEC
arb_random GNT_VEC GNT_VEC GNT_VEC GNT_VEC GNT_VEC
arb_single - - - - -arb_light's four seeds give - GNT_VEC - -. Three of them passed, and a passing run has no first error. So "the first defined id across the seeds" happens to give the right answer here — and only because the one failing seed is the only one with an id at all.
The rule has to be explicit: take the id from a run whose verdict is FAIL, in seed order so the choice is deterministic. Otherwise the first test whose seeds fail differently reports an id belonging to a run other than the one the row is about.
Then there is the case that needs a second step:
arb_budget is the exception and shows why the rule has two steps: it
has NO failing run, so the strict rule reports '-', while the table
above shows GNT_VEC -- the errors it had already found when the cycle
budget stopped it.arb_budget has no failing run at all — all four hit the budget — and it had still found errors before being stopped. Discarding them because no run reached a verdict would throw away the most useful thing on the row. So: a failing run's id if there is one, otherwise any run's, and the VERDICT column tells the reader which case they are looking at.
5. The Id Names Which Bug
tonight, first error ids over failing runs : GNT_VEC x18
last night, first error ids over failing runs: GNT_VALID x25Two nights, two ids, and neither night has both. Tonight's eighteen failures all report GNT_VEC — the arbiter granted the bus to the wrong requester. Last night's twenty-five all reported GNT_VALID — it asserted a grant when nobody was asking.
That is the column earning its width. Without it, both nights read "a lot of tests failed" and the obvious conclusion is that nothing has changed. With it, the table says the failures are a different failure, which is the difference between "still broken" and "one bug fixed, another introduced" — and Chapter 24.11 is about to need exactly that distinction.
Chapter 21.10's point applies here too: the id is a label, not a diagnosis. It tells you which of your checks fired first, which is enough to route the row to the right person and not enough to tell them what to change.
6. What To Remember
- One row per test, and a count in every verdict column — not a single verdict word.
- Measured:
arb_lightreads PASS 3 / FAIL 1, which states both halves of 24.2's pair and cannot be misread. arb_truncreading FAIL 1 / UNKNOWN 3 says design problem and infrastructure problem on one row.- The table checks itself in two lines: the verdict columns must sum to the run total, and every test must have the expected seed count.
- Measured: columns sum to 36 against 36 runs, and 0 tests have the wrong seed count.
- Print the checks in the artefact, not in a test suite — then a reader three weeks later knows this table is sound (23.9).
- Measured: the table is byte-identical built from records sorted, reversed and strided.
- Sort inside every loop in the builder — otherwise 24.11's diff fills with rows that moved for no reason (6.6, 17.8).
- Compare the bytes, not the data structures, when testing a printer.
- Measured:
arb_light's per-seed ids are- GNT_VEC - -— a passing run has no first error. - Take the first error id from a run whose verdict is FAIL, in seed order so the choice is deterministic.
- Fall back to any run's id when no run failed —
arb_budgetfound real errors before the budget stopped it, and the VERDICT column says which case the id came from. - Measured: tonight GNT_VEC x18, last night GNT_VALID x25, and neither night has both.
- The id distinguishes two different failures — the difference between "still broken" and "one bug fixed, another introduced" (24.11).
- An error id is a label, not a diagnosis (21.10).
Chapter 24.10 puts the same table on a web page, with no framework and no build step. The interesting part is not the HTML — it is that every field on the page came out of a log, and a log can contain anything.
Where this fits
Part of the Perl curriculum.