Perl Part II · Module 24 · Coverage and Regression Reporting
Human-Readable Summaries From a Hash
This is where the module starts printing. The structures are built, the numbers are decided, and all that remains is a table somebody reads with one hand on a coffee cup — which is harder than it sounds, because three of the four defects in this chapter produce a table that looks right. One of them is in the sort, one in the column widths, and the last one quietly removes the denominator from the totals line.
Intermediate11 min readPerlPart IIRegression Reporting
Part II · Module 24 · Chapter 24.6 · Human-Readable Summaries From a Hash
1. The Seeds Sort Wrongly
Start with the defect that is already present in this regression's own data.
use strict;
use warnings;
sub read_log {
my ($path) = @_;
my %r = (verdict => "UNKNOWN", errors => 0, complete => 0, budget => 0);
open my $fh, "<", $path or return \%r;
while (my $l = <$fh>) {
$r{test} = $1 if $l =~ /^TEST:\s+(\S+)/;
$r{seed} = $1 if $l =~ /^SEED:\s+(\d+)/;
$r{simtime} = $1 if $l =~ /^SIMTIME:\s+(\d+)/;
$r{budget} = 1 if $l =~ /^UVM_FATAL\s.*\[BUDGET\]/;
$r{msg}++ if $l =~ /^UVM_ERROR\s+\S+\(\d+\)/;
$r{complete} = 1 if $l =~ /^\*\* Report counts by severity/;
$r{sum} = $1 if $l =~ /^UVM_ERROR\s*:\s*(\d+)/;
$r{verdict} = "PASS" if $l =~ /^TEST PASSED/;
$r{verdict} = "FAIL" if $l =~ /^TEST FAILED/;
}
close $fh;
$r{errors} = $r{complete} ? $r{sum} : ($r{msg} || 0);
$r{verdict} = "TIMEOUT" if $r{budget} && $r{verdict} eq "UNKNOWN";
return \%r;
}
my @recs = grep { defined $_->{test} } map { read_log($_) } sort glob("logs/*.log");
# ---- 1. the seeds are numbers, and sort is a string sort
print "==== sorting the seeds ====\n";
my %seeds; $seeds{ $_->{seed} } = 1 for @recs;
printf "sort keys : %s\n", join " ", sort keys %seeds;
printf "sort { \$a <=> \$b } : %s\n", join " ", sort { $a <=> $b } keys %seeds;
my @str = sort keys %seeds;
my @num = sort { $a <=> $b } keys %seeds;
printf "the two orders %s\n",
("@str" eq "@num") ? "agree on this data" : "DISAGREE -- 1234 sorts before 42";
# ---- 2. the column width has to come from the data
print "\n==== the column width ====\n";
my $longest = 0;
for (@recs) { $longest = length $_->{test} if length $_->{test} > $longest }
printf "longest test name: %d characters (%s)\n", $longest,
(sort { length($b->{test}) <=> length($a->{test})
or $a->{test} cmp $b->{test} } @recs)[0]->{test};
print "\nwith a hardcoded %-12s:\n";
for my $r (@recs[0, 8, 12]) {
printf " %-12s %-8s %6d\n", $r->{test}, $r->{verdict}, $r->{errors};
}
printf "\nwith a width computed from the data (%%-%ds):\n", $longest;
for my $r (@recs[0, 8, 12]) {
printf " %-*s %-8s %6d\n", $longest, $r->{test}, $r->{verdict}, $r->{errors};
}
print "a name longer than the field does not truncate -- it pushes every\n";
print "later column right, on that row only, which is what breaks a table\n";
# ---- 3. verdict order: alphabetical puts PASS in the middle
print "\n==== the row order ====\n";
my %RANK = (FAIL => 0, TIMEOUT => 1, UNKNOWN => 2, PASS => 3);
printf "alphabetical : %s\n", join " ", sort keys %RANK;
printf "by severity : %s\n", join " ", sort { $RANK{$a} <=> $RANK{$b} } keys %RANK;
# ---- 4. the report an engineer reads
print "\n==== the 9am summary ====\n";
my @ordered = sort {
$RANK{ $a->{verdict} } <=> $RANK{ $b->{verdict} }
or $a->{test} cmp $b->{test}
or $a->{seed} <=> $b->{seed}
} @recs;
printf " %-*s %-6s %-8s %6s %s\n", $longest, "TEST", "SEED", "VERDICT",
"ERRORS", "SIMTIME";
printf " %s\n", "-" x ($longest + 32);
for my $r (@ordered) {
printf " %-*s %-6s %-8s %6d %s\n", $longest, $r->{test}, $r->{seed},
$r->{verdict}, $r->{errors},
defined $r->{simtime} ? "$r->{simtime} ns" : "-";
}
# ---- 5. the totals line
print "\n==== the totals line ====\n";
my %by;
$by{ $_->{verdict} }++ for @recs;
my $decided = grep { $_->{verdict} =~ /^(PASS|FAIL)$/ } @recs;
printf " %s\n", join " ", map { "$_ $by{$_}" }
sort { $RANK{$a} <=> $RANK{$b} } keys %by;
printf " %d run(s) total, %d reached a verdict, %d did not\n",
scalar @recs, $decided, @recs - $decided;
# ---- 6. the filtered report, and the totals that must not be filtered with it
print "\n==== showing only what is wrong ====\n";
my @bad = grep { $_->{verdict} ne "PASS" } @ordered;
printf "rows shown: %d of %d\n", scalar @bad, scalar @recs;
my %badby; $badby{ $_->{verdict} }++ for @bad;
printf "totals computed over the SHOWN rows : %s (total %d)\n",
join(" ", map { "$_=$badby{$_}" } sort { $RANK{$a} <=> $RANK{$b} } keys %badby),
scalar @bad;
printf "totals computed over ALL rows : %s (total %d)\n",
join(" ", map { "$_=$by{$_}" } sort { $RANK{$a} <=> $RANK{$b} } keys %by),
scalar @recs;
printf "\nthe filtered totals omit %d PASS run(s), so the report reads as\n",
$by{PASS} // 0;
printf "%d of %d failing when the truth is %d of %d\n",
scalar(@bad), scalar(@bad), scalar(@bad), scalar @recs;
print "a filtered report must still total the whole regression\n";
# ---- 7. one line per test instead of one line per run
print "\n==== the same night, one line per test ====\n";
my %byt;
push @{ $byt{ $_->{test} } }, $_ for @recs;
my $tw = 0;
for (keys %byt) { $tw = length if length > $tw }
printf " %-*s %-28s %-10s %s\n", $tw, "TEST", "SEEDS", "VERDICT", "WHICH SEEDS FAILED";
printf " %s\n", "-" x ($tw + 62);
my @trows = sort {
$RANK{ worst($byt{$a}) } <=> $RANK{ worst($byt{$b}) } or $a cmp $b
} keys %byt;
sub worst {
my ($rs) = @_;
my ($w) = sort { $RANK{$a} <=> $RANK{$b} } map { $_->{verdict} } @$rs;
return $w;
}
for my $t (@trows) {
my @rs = @{ $byt{$t} };
my %v; $v{ $_->{verdict} }++ for @rs;
my @bad = map { $_->{seed} }
grep { $_->{verdict} eq "FAIL" } @rs;
printf " %-*s %-28s %-10s %s\n", $tw, $t,
join(" ", map { "$_=$v{$_}" } sort { $RANK{$a} <=> $RANK{$b} } keys %v),
worst(\@rs),
@bad ? join(",", sort { $a <=> $b } @bad) : "-";
}
printf "\n%d row(s) instead of %d, and the failing seed is still on the row\n",
scalar @trows, scalar @recs;
printf "the per-test verdict is the WORST of its seeds, not the most common:\n";
for my $t (@trows) {
my @rs = @{ $byt{$t} };
my %v; $v{ $_->{verdict} }++ for @rs;
next unless keys %v > 1;
my ($common) = sort { $v{$b} <=> $v{$a} || $RANK{$a} <=> $RANK{$b} } keys %v;
printf " %-18s worst %-8s most common %-8s %s\n", $t, worst(\@rs), $common,
worst(\@rs) ne $common ? "<- they DISAGREE" : "";
}==== sorting the seeds ====
sort keys : 1 1234 42 7
sort { $a <=> $b } : 1 7 42 1234
the two orders DISAGREE -- 1234 sorts before 42
==== the column width ====
longest test name: 16 characters (arb_random_short)
with a hardcoded %-12s:
arb_budget TIMEOUT 777
arb_random FAIL 55
arb_random_short FAIL 17
with a width computed from the data (%-16s):
arb_budget TIMEOUT 777
arb_random FAIL 55
arb_random_short FAIL 17
a name longer than the field does not truncate -- it pushes every
later column right, on that row only, which is what breaks a table
==== the row order ====
alphabetical : FAIL PASS TIMEOUT UNKNOWN
by severity : FAIL TIMEOUT UNKNOWN PASS
==== the 9am summary ====
TEST SEED VERDICT ERRORS SIMTIME
------------------------------------------------
arb_light 7 FAIL 1 325 ns
arb_random 1 FAIL 55 4025 ns
arb_random 7 FAIL 62 4025 ns
arb_random 42 FAIL 62 4025 ns
arb_random 1234 FAIL 52 4025 ns
arb_random_short 1 FAIL 17 1025 ns
arb_random_short 7 FAIL 12 1025 ns
arb_random_short 42 FAIL 14 1025 ns
arb_random_short 1234 FAIL 12 1025 ns
arb_soak 1 FAIL 162 4025 ns
arb_soak 7 FAIL 161 4025 ns
arb_soak 42 FAIL 159 4025 ns
arb_soak 1234 FAIL 164 4025 ns
arb_stress 1 FAIL 251 4025 ns
arb_stress 7 FAIL 260 4025 ns
arb_stress 42 FAIL 264 4025 ns
arb_stress 1234 FAIL 263 4025 ns
arb_trunc 1 FAIL 55 4025 ns
arb_budget 1 TIMEOUT 777 12005 ns
arb_budget 7 TIMEOUT 782 12005 ns
arb_budget 42 TIMEOUT 788 12005 ns
arb_budget 1234 TIMEOUT 781 12005 ns
arb_trunc 7 UNKNOWN 8 -
arb_trunc 42 UNKNOWN 8 -
arb_trunc 1234 UNKNOWN 8 -
arb_light 1 PASS 0 325 ns
arb_light 42 PASS 0 325 ns
arb_light 1234 PASS 0 325 ns
arb_single 1 PASS 0 20025 ns
arb_single 7 PASS 0 20025 ns
arb_single 42 PASS 0 20025 ns
arb_single 1234 PASS 0 20025 ns
arb_smoke 1 PASS 0 4025 ns
arb_smoke 7 PASS 0 4025 ns
arb_smoke 42 PASS 0 4025 ns
arb_smoke 1234 PASS 0 4025 ns
==== the totals line ====
FAIL 18 TIMEOUT 4 UNKNOWN 3 PASS 11
36 run(s) total, 29 reached a verdict, 7 did not
==== showing only what is wrong ====
rows shown: 25 of 36
totals computed over the SHOWN rows : FAIL=18 TIMEOUT=4 UNKNOWN=3 (total 25)
totals computed over ALL rows : FAIL=18 TIMEOUT=4 UNKNOWN=3 PASS=11 (total 36)
the filtered totals omit 11 PASS run(s), so the report reads as
25 of 25 failing when the truth is 25 of 36
a filtered report must still total the whole regression
==== the same night, one line per test ====
TEST SEEDS VERDICT WHICH SEEDS FAILED
------------------------------------------------------------------------------
arb_light FAIL=1 PASS=3 FAIL 7
arb_random FAIL=4 FAIL 1,7,42,1234
arb_random_short FAIL=4 FAIL 1,7,42,1234
arb_soak FAIL=4 FAIL 1,7,42,1234
arb_stress FAIL=4 FAIL 1,7,42,1234
arb_trunc FAIL=1 UNKNOWN=3 FAIL 1
arb_budget TIMEOUT=4 TIMEOUT -
arb_single PASS=4 PASS -
arb_smoke PASS=4 PASS -
9 row(s) instead of 36, and the failing seed is still on the row
the per-test verdict is the WORST of its seeds, not the most common:
arb_light worst FAIL most common PASS <- they DISAGREE
arb_trunc worst FAIL most common UNKNOWN <- they DISAGREEsort keys : 1 1234 42 7
sort { $a <=> $b } : 1 7 42 1234
the two orders DISAGREE -- 1234 sorts before 42sort compares as strings by default, so "1234" comes before "42" because 1 is less than 4. These are the four seeds the regression actually ran — 1, 7, 42 and 1234 — so this is not a contrived example waiting for someone else's data.
sort { $a <=> $b } fixes it (5.9, and 3.7 on <=> itself). The habit worth forming is to decide, every time you write sort, whether the keys are text or numbers — because the string sort is right for test names and wrong for seeds, and both appear in the same report.
2. The Column Width Comes From The Data
longest test name: 16 characters (arb_random_short)
with a hardcoded %-12s:
arb_budget TIMEOUT 777
arb_random FAIL 55
arb_random_short FAIL 17Look at the third row. A name longer than the field does not get truncated — it pushes every later column to the right, on that row only. One long name and the table stops being a table.
printf takes the width as an argument when you give it *:
printf " %-*s %-8s %6d\n", $longest, $r->{test}, $r->{verdict}, $r->{errors};and $longest costs one pass over the records:
$longest = length $_->{test} for grep { length $_->{test} > $longest } @recs;This matters more over time than it looks. The report is written when the longest name is twelve characters, works for a year, and breaks the week somebody adds arb_random_short. A measured width never has that week.
3. Failures First
alphabetical : FAIL PASS TIMEOUT UNKNOWN
by severity : FAIL TIMEOUT UNKNOWN PASSSorting verdicts alphabetically puts PASS second, so eleven passing rows sit between the failures and the timeouts. The fix is a rank table:
my %RANK = (FAIL => 0, TIMEOUT => 1, UNKNOWN => 2, PASS => 3);and a comparator with three keys, each one breaking the previous tie:
sort { $RANK{$a->{verdict}} <=> $RANK{$b->{verdict}}
or $a->{test} cmp $b->{test}
or $a->{seed} <=> $b->{seed} } @recsNote the three different comparisons in one sort — numeric on the rank, string on the name, numeric on the seed. That is section 1's point again, and it is also what makes the order total: no two rows compare equal, so the report comes out identically on every run regardless of hash order (6.6, 17.8).
4. The Report
The table in the captured output is the whole point of the chapter, and three things about it are deliberate.
Failures are at the top and the eye stops at the first PASS. Everything below that line is confirmation, not information.
The error count is right-aligned with %6d, so 777 and 12 line up on their last digit and a glance ranks them.
A run with no simulated time prints -, not 0. arb_trunc seeds 7, 42 and 1234 never reached the line that reports the time, and zero would be a claim about a run that made none.
5. The Totals Line
FAIL 18 TIMEOUT 4 UNKNOWN 3 PASS 11
36 run(s) total, 29 reached a verdict, 7 did notTwo lines, and the second is the one most reports leave out. Seven runs never reached a verdict, and a reader who knows that reads the first line differently.
Now the trap, which appears the moment somebody asks for a shorter report:
rows shown: 25 of 36
totals computed over the SHOWN rows : FAIL=18 TIMEOUT=4 UNKNOWN=3 (total 25)
totals computed over ALL rows : the same plus PASS=11 (total 36)
the filtered totals omit 11 PASS run(s), so the report reads as
25 of 25 failing when the truth is 25 of 36Filtering the rows and then totalling the rows gives a report that says 25 of 25 failed. Every number in it is correct. The denominator is simply gone, and with it the only thing that made the numbers mean anything.
So the rule for every filtered report in this module, and the reason the structure of 24.2 keeps all thirty-six runs:
Filter the rows. Never filter the totals. Say what was hidden, in the report:
showing 25 non-passing of 36 runsChapter 24.3 found the same shape in a coverage percentage whose denominator was discarded, and Chapter 24.5 in a closure number that averaged rates instead of summing counts. A number without its denominator is the most reliable way to publish something true and misleading at once.
6. One Line Per Test
Thirty-six rows is a lot for a glance. The structure of 24.2 already groups the runs by test, so a one-line-per-test view costs nothing extra:
TEST SEEDS VERDICT WHICH SEEDS FAILED
------------------------------------------------------------------------------
arb_light FAIL=1 PASS=3 FAIL 7
arb_random FAIL=4 FAIL 1,7,42,1234
arb_trunc FAIL=1 UNKNOWN=3 FAIL 1
arb_budget TIMEOUT=4 TIMEOUT -
arb_smoke PASS=4 PASS -
9 row(s) instead of 36, and the failing seed is still on the rowNine rows, and nothing a reader needs has been dropped. The seed breakdown is in a column rather than in four rows, and the failing seed is still there — which is 24.2's rule about storing what a rerun needs, surviving into the printed report.
The one real decision is what verdict a test gets when its seeds disagree, and it has a measured answer:
the per-test verdict is the WORST of its seeds, not the most common:
arb_light worst FAIL most common PASS <- they DISAGREE
arb_trunc worst FAIL most common UNKNOWN <- they DISAGREEOn two of the nine tests, the worst verdict and the most common verdict are different. For arb_light the worst is FAIL and the most common is PASS — and PASS is the wrong answer, because a bug that reproduces one time in four is still a bug. Taking the majority would hide it under a green row.
So the per-test verdict is FAIL if any seed failed, and the seed count beside it is what stops that from being alarmist. The same rank table that ordered the rows does the reduction:
my ($worst) = sort { $RANK{$a} <=> $RANK{$b} } map { $_->{verdict} } @runs;A majority rule, an average, or a "mostly passing" column would each be defensible-sounding and each would hide the one bug this regression found intermittently.
7. What To Remember
sortis a string sort. Measured: the real seeds 1, 7, 42, 1234 come out 1 1234 42 7.- Decide every time whether the keys are text or numbers — a report usually has both (5.9, 3.7).
- Measured: the longest test name is 16 characters, and a hardcoded
%-12spushes every later column right on that row. - A too-narrow field does not truncate, it shifts — which is what destroys a table.
- Use
%-*swith a width measured from the data; one pass, and it never has the week somebody adds a longer name. - Measured: alphabetical verdict order puts PASS second. Use a rank table and put failures first.
- Make the sort total — rank, then name, then seed — so the report is identical on every run whatever the hash order (6.6, 17.8).
- Right-align counts with
%6dso a glance ranks them. - Print
-, not0, for a measurement a run never made. - Print a second totals line: how many runs reached a verdict and how many did not. Measured here: 29 and 7.
- Measured: filtering rows and then totalling them turned "25 of 36 failing" into "25 of 25 failing".
- Filter the rows; never filter the totals — and say in the report what was hidden.
- A number without its denominator is true and misleading at once (24.3, 24.5).
- A one-line-per-test view is 9 rows instead of 36 and drops nothing a reader needs.
- Keep the failing seed in a column, so the printed report still says what to rerun (24.2).
- Measured: the worst verdict and the most common verdict disagree on 2 of 9 tests.
- A test's verdict is the WORST of its seeds, not the majority —
arb_lightis FAIL with three passes, and a bug that reproduces one time in four is still a bug. - The seed count beside the verdict is what keeps "worst wins" from being alarmist.
Chapter 24.7 makes this shorter. A mail body or a chat message gets a few lines, which forces a decision about what the first line says — and the shortest useful digest turns out to be the one that leaves the passing tests out entirely while still counting them.
Where this fits
Part of the Perl curriculum.