Linux & Bash · Part II · Module 32 · Regression Automation End to End

The Regression Report

A report that lists what happened makes the reader do the comparing, and the comparing is the part that matters. This chapter builds a report that answers three questions in a fixed order — did it finish, what is not a pass, did anything get worse — and measures a night whose pass count improved while not one thing about the design had changed.

Advanced26 min readLinuxPart IIRegressionReportingComparison

Part II · Module 32 · Chapter 32.5 · The Regression Report

Chapter 32.4 produced a file that can answer any question about one regression. A report is not that file printed nicely. A report is an answer to the question somebody actually has at nine in the morning, which is never "what happened" and almost always "is this worse than yesterday, and if so, what do I look at first?"

1. Three Questions, In Order

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

==============================================================
 regression 20261009T093929Z
==============================================================
 24 runs: 9 PASS  3 FAIL  2 TIMEOUT  2 CRASH
          2 NO_VERDICT  4 INCONSISTENT  2 VACUOUS
          0 MISSING  0 NOSTATUS

 not a pass, by class:
     prio0:1            FAIL          tool_rc=0   checks=1    errors=1
     prio0:2            FAIL          tool_rc=0   checks=1    errors=1
     inconsistent:1     INCONSISTENT  tool_rc=0   checks=1    errors=0
     inconsistent:2     INCONSISTENT  tool_rc=0   checks=1    errors=0
     crash:1            CRASH         tool_rc=1   checks=-    errors=-
     crash:2            CRASH         tool_rc=1   checks=-    errors=-
     silent:1           NO_VERDICT    tool_rc=0   checks=-    errors=-
     silent:2           NO_VERDICT    tool_rc=0   checks=-    errors=-
     flaky:2            FAIL          tool_rc=0   checks=1    errors=1
     hang:1             TIMEOUT       tool_rc=0   checks=-    errors=-
     hang:2             TIMEOUT       tool_rc=0   checks=-    errors=-
     vacuous:1          VACUOUS       tool_rc=0   checks=0    errors=0
     vacuous:2          VACUOUS       tool_rc=0   checks=0    errors=0
     latecrash:1        INCONSISTENT  tool_rc=1   checks=1    errors=0
     latecrash:2        INCONSISTENT  tool_rc=1   checks=1    errors=0

 4 of those are environment, not design (TIMEOUT/CRASH/NOSTATUS)

The order of those sections is the chapter's main claim, so it is worth being explicit about why.

Question 1: did the regression finish? 0 MISSING 0 NOSTATUS. Until that line reads zero, every other number in the report is over an unknown denominator and means nothing. A pass rate of 90% across 100 runs when the matrix asked for 500 is not a pass rate; it is a fragment. Chapter 31.7 measured a reporter that answered the other questions first and reported 7 of 7 PASS, exit 0 about a matrix of eight.

Question 2: what is not a pass, and what evidence is beside it? Not "what failed" — Chapter 32.3 established seven classes and only one of them is FAIL. The evidence columns are there so the reader can triage without opening a log: checks=0 says the run proved nothing, tool_rc=1 beside a TEST PASSED says the tool died afterwards, checks=- says there is no count to read.

Question 3: what changed? Which is Section 3, and the only question that distinguishes a report from a listing.

And one line that is neither a count nor a comparison:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

 4 of those are environment, not design (TIMEOUT/CRASH/NOSTATUS)

Fifteen runs are not a pass; four of them are not design questions. Two timeouts and two crashes belong to whoever owns the machines, and sending them to a verification engineer is how a morning gets spent on a waveform with nothing in it.

2. The Report

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# report RUNDIR [BASELINE_RESULTS] -- the regression's one answer.
# THE EXIT-CODE CONTRACT, and nothing else may be returned:
#   0  every run passed
#   1  the regression finished; something is not a pass  -> a DESIGN question
#   2  the regression ran but did not finish             -> an INFRASTRUCTURE question
#   3  there is nothing to report on                     -> a SETUP question
# Answers three questions in order: did it finish, what is not a pass,
# and did anything get WORSE than the baseline.
set -u
out=${1:-}; base=${2:-}
res=$out/results.tsv
[ -r "$res" ] || { echo "report: no results.tsv in '$out' -- nothing to report on" >&2; exit 3; }
rowcount=$(grep -vc '^#' "$res")
[ "$rowcount" -gt 0 ] || { echo "report: results.tsv has no rows -- nothing ran" >&2; exit 3; }
rows=$(grep -vc '^#' "$res")
cnt() { awk -F'\t' -v k="$1" '$4==k' "$res" | grep -c . ; }
P=$(cnt PASS); F=$(cnt FAIL); T=$(cnt TIMEOUT); C=$(cnt CRASH)
N=$(cnt NO_VERDICT); I=$(cnt INCONSISTENT); V=$(cnt VACUOUS)
M=$(cnt MISSING); S=$(cnt NOSTATUS)
echo "=============================================================="
echo " regression $(basename "$out")"
echo "=============================================================="
printf ' %d runs: %d PASS  %d FAIL  %d TIMEOUT  %d CRASH\n' "$rows" "$P" "$F" "$T" "$C"
printf '          %d NO_VERDICT  %d INCONSISTENT  %d VACUOUS\n' "$N" "$I" "$V"
printf '          %d MISSING  %d NOSTATUS\n' "$M" "$S"
incomplete=$(( M + S ))
echo
if [ "$incomplete" -gt 0 ]; then
  echo " *** DID NOT FINISH: $incomplete run(s) produced no verdict ***"
  awk -F'\t' '$4=="MISSING"||$4=="NOSTATUS" {printf "     %-18s %s\n",$1,$4}' "$res"
  echo
fi
notpass=$(( F + T + C + N + I + V ))
if [ "$notpass" -gt 0 ]; then
  echo " not a pass, by class:"
  awk -F'\t' 'NR>1 && $4!="PASS" && $4!="MISSING" && $4!="NOSTATUS" \
    {printf "     %-18s %-13s tool_rc=%-3s checks=%-4s errors=%s\n",$1,$4,$6,$8,$9}' "$res"
  echo
fi
# the environment-vs-design split: these are not design questions
envclass=$(( T + C + S ))
[ "$envclass" -gt 0 ] && printf ' %d of those are environment, not design (TIMEOUT/CRASH/NOSTATUS)\n\n' "$envclass"
if [ -n "$base" ] && [ -r "$base" ]; then
  echo " against the baseline $(basename "$(dirname "$base")"):"
  join -t"$(printf '\t')" -j1 \
    <(awk -F'\t' 'NR>1{print $1"\t"$4}' "$base" | sort) \
    <(awk -F'\t' 'NR>1{print $1"\t"$4}' "$res"  | sort) \
    | awk -F'\t' '$2!=$3 {printf "     %-18s %s -> %s\n",$1,$2,$3; n++}
                  END{ if(!n) print "     every run has the same verdict as the baseline" }'
  # a spec present in one and not the other is a matrix change, not a result change
  comm -23 <(awk -F'\t' 'NR>1{print $1}' "$base" | sort) <(awk -F'\t' 'NR>1{print $1}' "$res" | sort) \
    | sed 's/^/     GONE from the matrix: /'
  comm -13 <(awk -F'\t' 'NR>1{print $1}' "$base" | sort) <(awk -F'\t' 'NR>1{print $1}' "$res" | sort) \
    | sed 's/^/     NEW in the matrix:    /'
  echo
fi
[ "$incomplete" -gt 0 ] && exit 2
[ "$notpass" -gt 0 ] && exit 1
exit 0

Three details are worth pulling out.

cnt() counts rows, not matches.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

cnt() { awk -F'\t' -v k="$1" '$4==k' "$res" | grep -c . ; }

awk '$4==k' requires the whole field to equal the class, so PASS does not match NO_VERDICT's substring — and grep -c . counts non-empty lines rather than relying on awk's own record count. Chapter 29.4 measured a severity count that invented a UVM_FATAL out of a summary line; an anchored field comparison is what prevents that whole family of error.

The class list is explicit, not discovered.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

P=$(cnt PASS); F=$(cnt FAIL); T=$(cnt TIMEOUT); C=$(cnt CRASH)
N=$(cnt NO_VERDICT); I=$(cnt INCONSISTENT); V=$(cnt VACUOUS)
M=$(cnt MISSING); S=$(cnt NOSTATUS)

Nine named classes. A report built by iterating whatever classes appear in the file would print a shorter report on a better night — and a class silently vanishing from a report is how 0 MISSING stops being reassuring and starts being absent. Print the zeros. Looping over the data tells you what is there; looping over the expected set tells you what is missing, which is the discipline Chapter 31.7 applied to runs and this applies to classes.

And the totals are derived, never re-counted.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

incomplete=$(( M + S ))
notpass=$(( F + T + C + N + I + V ))
envclass=$(( T + C + S ))

notpass is a sum of the classes, not a separate awk that asks for $4!="PASS". If a new class is added to Chapter 32.3 and not added to that sum, the report's own arithmetic stops adding up — rows will not equal P + notpass + incomplete — which is a loud failure. A !="PASS" filter would silently absorb the new class and report nothing wrong.

3. The Comparison, and the Night That Looked Better

Here is the whole argument for Question 3, in three measured regressions on the same fixture.

Run A is the report above: 24 runs, 9 PASS, 3 FAIL.

Run B followed a change to the arbiter — its reset was flipped so that agent 0 wins the first contention:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

 24 runs: 13 PASS  1 FAIL  2 TIMEOUT  2 CRASH
          2 NO_VERDICT  2 INCONSISTENT  2 VACUOUS

Nine passes became thirteen. The comparison says exactly which:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

 against the baseline 20261009T093929Z:
     inconsistent:1     INCONSISTENT -> PASS
     inconsistent:2     INCONSISTENT -> PASS
     prio0:1            FAIL -> PASS
     prio0:2            FAIL -> PASS

Four specs, named, with their before and after. Two of them were not the point of the change. The commit was aimed at prio0; it also silenced inconsistent, whose check happened to be looking at the same first grant. That is the single most useful thing a comparison does — it tells you what else moved — and a report that printed only "9 → 13 passes" would have hidden it completely.

And now the one that matters

Run C changed no RTL and no testbench. It ran the identical test list with seeds 1 and 3 instead of 1 and 2:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

 24 runs: 10 PASS  2 FAIL  2 TIMEOUT  2 CRASH
          2 NO_VERDICT  4 INCONSISTENT  2 VACUOUS

Against run A: one more pass, one fewer failure. By every headline number, a better night.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

 against the baseline 20261009T093929Z:
     every run has the same verdict as the baseline
     GONE from the matrix: b2b:2
     GONE from the matrix: crash:2
     GONE from the matrix: fairness:2
     GONE from the matrix: flaky:2
     GONE from the matrix: hang:2
     GONE from the matrix: inconsistent:2
     GONE from the matrix: latecrash:2
     GONE from the matrix: prio0:2
     GONE from the matrix: random:2
     GONE from the matrix: silent:2
     GONE from the matrix: smoke:2
     GONE from the matrix: vacuous:2
     NEW in the matrix:    b2b:3
     NEW in the matrix:    crash:3
     …

Every run that exists in both has an identical verdict. Nothing improved. Twelve specs left the matrix and twelve arrived, and one of the departures was flaky:2 — the single seed on which that test fails. The pass count went up because the failing run was removed from the regression.

A pass count is not comparable unless the matrix is identical

This is the defect that makes regression reporting hard, and it is not exotic. Seeds get regenerated. A test gets renamed. Somebody adds four seeds to one test and the total moves. Every one of those changes the denominator, and a report that compares counts attributes the change to the design.

The fix has two parts and the second is the one people skip.

Compare per spec, by joining on the spec name. Not the count, not the pass rate — the verdict of each named run against the verdict of the same named run last night:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

join -t"$(printf '\t')" -j1 \
  <(awk -F'\t' 'NR>1{print $1"\t"$4}' "$base" | sort) \
  <(awk -F'\t' 'NR>1{print $1"\t"$4}' "$res"  | sort) \
  | awk -F'\t' '$2!=$3 {printf "     %-18s %s -> %s\n",$1,$2,$3}'

And report the specs that are in one and not the other, because join is silent about them:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

comm -23 <(… "$base" …) <(… "$res" …) | sed 's/^/     GONE from the matrix: /'
comm -13 <(… "$base" …) <(… "$res" …) | sed 's/^/     NEW in the matrix:    /'

Without those two lines, run C's comparison would have printed every run has the same verdict as the baseline and nothing else — which is true, and which a reader would take as "nothing changed" rather than "half the matrix was replaced". A join that silently drops non-matching keys is the same hazard Chapter 29.7 met when correlating two stages' logs, and the same one Module 31 met when a results comparison joined on the run name and reported "same set" after one run was fixed and another broken.

sort before join, every time. join requires sorted input and does not check; given unsorted input it produces a partial result with no error, which is the quietest possible way to lose half a comparison.

Figure 1 — the same two regressions, compared two ways. By the headline counts the second night is better: one more pass, one fewer failure. Compared per specification, every run present in both nights has an identical verdict — and twelve specifications left the matrix while twelve arrived, one of the departures being the only seed on which that test fails.

4. What a Report Must Not Do

It must not print a pass rate as the headline. 9/24 = 37.5% is a true statement about this fixture and it answers nothing. The counts by class, plus what changed, is what somebody acts on. A percentage is the right thing to put on a trend chart (Chapter 32.8) and the wrong thing to put at the top of a report.

It must not sort failures by name. They come out in matrix order here, which groups a test's seeds together — Chapter 31.4's ordering decision paying off, because prio0:1 and prio0:2 adjacent reads as one problem rather than two.

It must not hide a class because it is empty. See Section 2.

It must not open a run directory. Chapter 32.4's rule, and it is what lets this same report run against an archive from three months ago.

And it must not be the only thing that returns a status. The report's exit status is the regression's exit status, which is Chapter 32.7's subject — and Chapter 32.2 measured the driver destroying it on its last line with a | tee.

The four statuses this report returns, measured

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  a regression with failures                     1
  a regression that did not finish               2
  a regression that could not start              3
  an all-pass regression                         0

Those came out of the script in Section 2, and the third one came out wrong first. The original guard was:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[ -r "$res" ] || { echo "report: no results.tsv in '$out'" >&2; exit 2; }

— which gave 2 both for "the regression ran and lost some runs" and for "there is no results file at all". Two genuinely different situations: one is an infrastructure problem inside a regression that happened, the other is a setup problem where nothing happened. Merging them sends somebody to look for lost runs in a regression that never started.

The corrected version reserves 3, and also catches the case where the file exists and is empty:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[ -r "$res" ] || { echo "report: no results.tsv in '$out' -- nothing to report on" >&2; exit 3; }
rowcount=$(grep -vc '^#' "$res")
[ "$rowcount" -gt 0 ] || { echo "report: results.tsv has no rows -- nothing ran" >&2; exit 3; }

Both zero-row cases were then measured explicitly, because a report over zero rows is the most dangerous input there is — every count is zero, nothing is "not a pass", and the naive arithmetic returns success:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  header only, no rows -> rc=3
  completely empty file -> rc=3

That is the fourth time this module has met the same shape. A condition written as the absence of bad news returns success on no news at all.

5. Where the Report Should Be Terse

A report that prints everything is a report nobody reads, and the three sections above are already at the limit. Two things are deliberately not in it:

The passing runs. Nine of them, by name, would double the report's length and change no decision. They are in results.tsv for anybody who wants them.

The log excerpts. Tempting, and wrong at this level: fifteen non-passing runs with three lines of log each is forty-five lines of detail nobody asked for yet. The report names the run; Chapter 29.2 is how you then find the first real error in that one log; and Chapter 32.6 is how you re-run it.

The test for whether a report is the right length is whether every line changes a decision. The class counts decide whether to escalate. The not-a-pass list decides who to send it to. The environment line decides whether that is a verification person at all. The comparison decides whether it is new. A log excerpt decides nothing, because by the time you want it you are already in the log.

Practice

Write out the three questions your report must answer, in order, and justify the order.
Put the pass rate at the top, then try to act on it, and explain what is missing.
Report over a matrix with a missing run and check that Question 1 fires before anything else.
Add the evidence columns to the not-a-pass list and triage one run without opening its log.
Compute the environment-versus-design split and say who each half goes to.
Count with $4==k, then with a substring match, and find a class that absorbs another.
Build the class list by iterating the data, then have a good night, and see which class disappears.
Derive notpass as a sum of classes, then add a new class without updating the sum, and watch the arithmetic break loudly.
Replace that sum with $4!="PASS" and watch the same mistake pass silently.
Run two regressions across one real change and read the per-spec comparison.
Find a spec that changed which you were not expecting and explain it.
Change only the seed list and compare the headline counts against the baseline.
Compare per spec instead and find that nothing changed.
Add the comm -23 and comm -13 lines and watch the matrix change appear.
Remove them and explain exactly what a reader would wrongly conclude.
Feed join unsorted input and count how many rows you lose.
Report over an empty results file and check the status.
Report over a header-only file and check it again.
Add log excerpts to the report, live with it for a week, and then decide.

Check Your Understanding

What question does somebody actually have at nine in the morning?
Why must "did it finish" come first?
Why is the second question "what is not a pass" rather than "what failed"?
What do the evidence columns let a reader do?
What does the environment-versus-design line decide?
Why awk '$4==k' rather than a substring match?
Why is the class list explicit rather than discovered?
Why print the zeros?
Why is notpass a sum of classes rather than a !="PASS" filter?
In run B, four specs changed. Why are two of them the most valuable part of the report?
Run C has one more pass and one fewer failure than run A. What changed?
What did the per-spec comparison say about run C?
What would run C's comparison have printed without the comm lines, and why is that dangerous?
Name three ordinary events that change a regression's denominator.
What does join do with a key present in only one input?
What does join do with unsorted input?
Why should a pass rate not head a report?
Why not sort the failures by name?
Why must the report not open a run directory?
Which status was wrong first, and what two situations was it merging?
Why is a zero-row results file the most dangerous input?
Give two things deliberately left out of the report, and the test for the right length.

Answers

Whether this is worse than yesterday, and if so what to look at first — not "what happened".
Because until MISSING and NOSTATUS are zero, every other number is over an unknown denominator.
Because there are seven verdict classes and only one of them is FAIL. The others need different people and different investigations.
Triage without opening a log: checks=0 means the run proved nothing, tool_rc=1 beside a PASSED line means the tool died afterwards, checks=- means there is no count.
Whether the item goes to a verification engineer at all, or to whoever owns the machines.
Because a whole-field comparison cannot match another class's substring, which is how a naive count once invented a UVM_FATAL out of a summary line.
Because a report that iterates whatever classes appear prints a shorter report on a better night, and a class that silently disappears is how 0 MISSING stops being reassuring.
So that the absence of a problem is visible as a zero rather than as a missing line.
Because if a new class is added and not added to the sum, the report's own arithmetic stops adding up — a loud failure. A !="PASS" filter absorbs the new class silently.
Because they were not the point of the change: the commit targeted prio0 and also silenced inconsistent. Knowing what else moved is the comparison's main value.
The seed list — seeds 1 and 3 instead of 1 and 2. No RTL and no testbench changed.
That every run present in both has an identical verdict, and that twelve specs left the matrix and twelve arrived — one of the departures being the only seed on which flaky fails.
Just every run has the same verdict as the baseline, which is true and reads as "nothing changed" rather than "half the matrix was replaced".
Seeds being regenerated, a test being renamed, and seeds being added to one test — any three that move the denominator.
Drops it silently, with no error. That is why the two comm lines exist.
Produces a partial result with no error at all, which is the quietest way to lose half a comparison. Always sort first.
Because a percentage answers nothing actionable. Counts by class plus what changed is what somebody acts on; the percentage belongs on a trend chart.
Because matrix order groups a test's seeds together, so prio0:1 and prio0:2 read as one problem.
Because that is what lets the same report run against an archive, and prevents it from disagreeing with the results file.
The "could not start" status. It returned 2, merging "the regression ran and lost runs" with "there is no results file at all" — an infrastructure problem inside a real regression against a setup problem where nothing happened.
Because every count is zero, nothing is "not a pass", and arithmetic written as the absence of bad news returns success.
The passing runs and the log excerpts. The test is whether every line changes a decision — a log excerpt decides nothing, because by the time you want it you are already in the log.

6. Remember

A report answers "is this worse than yesterday", not "what happened".
Three questions in order: did it finish, what is not a pass, what changed. The first gates the other two.
Report the environment-versus-design split. Four of fifteen non-passes here were not design questions at all.
Count with a whole-field comparison, never a substring.
Name the classes explicitly and print the zeros. A class that vanishes on a good night is a class that vanishes on a bad one.
Derive totals as sums of classes, so a new class breaks the arithmetic loudly instead of being absorbed.
Compare per spec by joining on the name — never counts, never a pass rate.
Report what left and joined the matrix. join is silent about keys present in only one side, and a comparison that says "no verdict changed" while half the matrix was replaced is worse than no comparison.
sort before join. Unsorted input gives a partial answer and no error.
A pass count is not comparable unless the matrix is identical. Run C gained a pass by dropping the one seed that fails.
Reserve a distinct status for "nothing to report on", and measure the zero-row cases — arithmetic over no rows returns success.
Leave out the passing runs and the log excerpts. Every line in a report should change a decision.

7. What Is Next

The report names fifteen runs that are not a pass. Chapter 32.6 re-runs exactly those, from the data the regression recorded about itself — and finds that restoring the source files you thought you changed does not restore the thing a reproduction actually depends on.

Where this fits

Part of the Linux & Bash curriculum.