From Logs to a Summary Table

Part II · Module 29 · Chapter 29.8 · From Logs to a Summary Table

Seven chapters of log reading produce answers one run at a time. A team needs one screen, and the screen has to answer three questions in order: is it worse than last time, what is failing, and what should I read. Anything else on it is decoration.

1. Three Questions, In That Order

The order is not arbitrary.

Is it worse? This is the only question that decides whether anybody acts tonight. A regression at 94% that was at 94% yesterday needs no action; the same number after a 99% yesterday needs somebody now. A report that leads with a pass rate and not a delta makes the reader do the comparison from memory.

What is failing? Grouped, with counts — Chapter 29.4's signatures. Forty failures in one class is one bug and a short conversation; forty in thirty classes is a different kind of day.

What should I read? A path to a log. Not a summary of the failure, a path — because the next action is always to open one.

Everything else a regression report typically carries — total runtime, machine names, coverage deltas — belongs on a second screen.

2. The Report

$ regress-summary regress
== regress: 12 runs ==
   4 passed, 5 failed, 2 no verdict, 1 inconsistent

-- failure signatures --
   5  TB ERROR @T dout=V expected V
        e.g. TB ERROR @35 dout=00 expected a5
   1  TB ERROR dout=V expected V
        e.g. TB ERROR dout=a4 expected a5

-- read these --
   FAIL           stuck_s1      regress/stuck_s1/run.log
   FAIL           stuck_s2      regress/stuck_s2/run.log
   FAIL           stuck_s3      regress/stuck_s3/run.log
   FAIL           xor_s1        regress/xor_s1/run.log
   FAIL           xor_s2        regress/xor_s2/run.log
   INCONSISTENT   liar_s1       regress/liar_s1/run.log
   NO-VERDICT     badcompile    regress/badcompile/run.log
   NO-VERDICT     killed        regress/killed/run.log
$ echo $?
1

Twelve runs on one screen. Four things about that output are deliberate.

Four states, not two. 4 passed, 5 failed, 2 no verdict, 1 inconsistent — because Chapter 29.4 measured that collapsing them puts a failed compile and a killed run into whichever column the classifier happens to favour. The two unusual states are small numbers and they are the ones worth looking at first.

The signature carries a representative line. TB ERROR @T dout=V expected V groups five runs; e.g. dout=00 expected a5 tells you it is a stuck-at-zero rather than a bit flip. Chapter 29.4 §4 is why both are needed.

The last section is paths. Eight of them, sorted by state so the unusual ones are adjacent.

It exits 1. A report that always succeeds cannot gate anything.

3. The Comparison That Lied

Now the part that matters, and the part I got wrong. Save a baseline, then change the regression in two directions at once — fix one failing run, break one passing run:

$ regress-summary regress > /dev/null && cp summary.tsv baseline.tsv
# ... rebuild stuck_s1 with the good design, and pass_s3 with a mutant ...
$ grep -oE 'TEST (PASSED|FAILED)' regress/stuck_s1/run.log | tail -1
TEST PASSED
$ grep -oE 'TEST (PASSED|FAILED)' regress/pass_s3/run.log | tail -1
TEST FAILED

And the first version of the comparison reported this:

-- against baseline.tsv: 8 bad then, 8 bad now (+0) --
   same set of runs and verdicts

The (+0) line is still worth printing, with its real meaning: the total did not move. It just must not be the only thing printed.

4. The Whole Tool

#!/bin/bash
# regress-summary RUNS [BASELINE] — one screen, from the raw logs.
# Answers three questions in order: is it worse, what is failing, what to read.
set -u
base=${1:?usage: regress-summary RUNS [BASELINE]}
prev=${2:-}
[ -d "$base" ] || { echo "regress-summary: no such directory: $base" >&2; exit 2; }

norm='s/@[0-9][0-9]*/@T/g; s/=[0-9a-fA-F][0-9a-fA-F]*/=V/g; s/ [0-9a-fA-F][0-9a-fA-F]*$/ V/; s/  */ /g'
tmp=$(mktemp -t regsum.XXXXXX) || exit 2
trap 'rm -f "$tmp" "$tmp.sig"' EXIT

pass=0 fail=0 noverdict=0 incons=0 total=0
: > "$tmp.sig"
for d in "$base"/*/; do
  [ -d "$d" ] || continue
  log="$d/run.log"; n=$(basename "$d"); total=$((total + 1))
  if [ ! -r "$log" ]; then noverdict=$((noverdict + 1)); printf '%s\tNO-LOG\t-\n' "$n" >> "$tmp"; continue; fi

  if   grep -q 'TEST PASSED' "$log"; then v=PASS
  elif grep -q 'TEST FAILED' "$log"; then v=FAIL
  else v=NO-VERDICT; fi
  e=$(grep -cE '(TB +ERROR|UVM_ERROR|UVM_FATAL)' "$log")
  byerr=$( [ "$e" -gt 0 ] && echo FAIL || echo PASS )

  sig=$(grep -h -E '(TB +ERROR|UVM_ERROR)' "$log" 2>/dev/null | sed "$norm" | sort -u | head -1)
  ex=$(grep -h -E '(TB +ERROR|UVM_ERROR)' "$log" 2>/dev/null | head -1 | sed 's/  */ /g')

  case "$v" in
    (PASS) if [ "$byerr" = FAIL ]; then incons=$((incons+1)); v=INCONSISTENT; else pass=$((pass+1)); fi ;;
    (FAIL) fail=$((fail+1)) ;;
    (*)    noverdict=$((noverdict+1)) ;;
  esac
  printf '%s\t%s\t%s\n' "$n" "$v" "${sig:--}" >> "$tmp"
  [ -n "$sig" ] && printf '%s\t%s\n' "$sig" "$ex" >> "$tmp.sig"
done

printf '== %s: %d runs ==\n' "$base" "$total"
printf '   %d passed, %d failed, %d no verdict, %d inconsistent\n' \
       "$pass" "$fail" "$noverdict" "$incons"

# 1. is it worse than the baseline?
if [ -n "$prev" ] && [ -r "$prev" ]; then
  pfail=$(awk -F'\t' '$2=="FAIL"||$2=="NO-VERDICT"||$2=="INCONSISTENT"{n++} END{print n+0}' "$prev")
  nfail=$((fail + noverdict + incons))
  printf '\n-- against %s: %d bad then, %d bad now (%+d) --\n' "$prev" "$pfail" "$nfail" "$((nfail - pfail))"
  # Compare the run NAME AND ITS VERDICT. Joining on the name alone finds
  # nothing when a run flips, because the name is present in both files.
  comm -13 <(cut -f1,2 "$prev" | sort) <(cut -f1,2 "$tmp" | sort) \
    | awk -F'\t' '$2!="PASS"{printf "   NOW BAD   %-14s %s\n", $1, $2}'
  comm -13 <(cut -f1,2 "$prev" | sort) <(cut -f1,2 "$tmp" | sort) \
    | awk -F'\t' '$2=="PASS"{printf "   NOW GOOD  %-14s %s\n", $1, $2}'
  if [ -z "$(comm -3 <(cut -f1,2 "$prev" | sort) <(cut -f1,2 "$tmp" | sort))" ]; then
    printf '   every run has the same verdict as the baseline\n'
  fi
fi

# 2. what is failing, grouped
printf '\n-- failure signatures --\n'
if [ -s "$tmp.sig" ]; then
  cut -f1 "$tmp.sig" | sort | uniq -c | sort -rn | while read -r c sig; do
    ex=$(awk -F'\t' -v s="$sig" '$1==s {print $2; exit}' "$tmp.sig")
    printf '%4d  %s\n        e.g. %s\n' "$c" "$sig" "$ex"
  done
else
  printf '   none\n'
fi

# 3. what to read first
printf '\n-- read these --\n'
awk -F'\t' '$2!="PASS"{printf "   %-14s %-13s %s/%s/run.log\n", $2, $1, "'"$base"'", $1}' "$tmp" | sort
cp "$tmp" "$base/../summary.tsv" 2>/dev/null
[ $((fail + noverdict + incons)) -eq 0 ]

Six decisions, each one a lesson from an earlier chapter.

The verdict and the error count are both consulted, and a disagreement becomes its own state. Chapter 29.4 §5: liar_s1 printed an error and then TEST PASSED, and neither classifier alone is right.

mktemp with a trap. Two temporary files, removed by an EXIT trap — Chapter 20.6's rule, and it matters here because the tool runs from cron. Measured on which signals the trap actually survives:

  signal  temporaries afterwards
  INT     gone
  HUP     gone
  TERM    gone
  KILL    STILL THERE

SIGKILL cannot be trapped, so a kill -9 leaves both files behind. That is not a flaw to fix — there is no fix — and it is the reason a long-lived flow needs the cleanup of Chapter 28.4 as well as a trap: the trap handles every ordinary exit, and something else has to sweep up after the one case it cannot.

The signatures are normalised positionally. @T, =V, a trailing field, and a whitespace collapse — Chapter 29.4 §3, including the reason a hex character class cannot be used.

The comparison is keyed on name and verdict, per §3.

It writes summary.tsv beside the runs, which is what the next invocation uses as its baseline. A report that cannot be compared against is a report that answers two of the three questions.

It exits non-zero if anything is not a pass — including inconsistent and no-verdict, not only failures.

5. Keeping the Table Honest as It Grows

Three things break a summary tool over a year, and all three are shape problems rather than bugs.

A new failure state appears. A scheduler starts evicting jobs, or a licence check starts timing out, and the new shape lands in whichever bucket the classifier reaches last. The defence is the one Chapter 28.2 used for retention: an explicit "I do not know" state that is printed. In this tool the NO-VERDICT bucket plays that role, and the honest version of growth is to split it when it gets large rather than to leave it as a mixed bag.

The signature normalisation stops matching. Somebody rewords a $display, and one class becomes two. Visible immediately as a count that halves and a new signature appearing — which is a good argument for printing signatures rather than mapping them to curated names, because a curated name silently absorbs a reworded message into the wrong class.

The baseline drifts. If every run updates the baseline, a slow degradation is never visible: each night is compared against a slightly worse yesterday. The usual answer is two baselines — last night for the delta, and a pinned known-good for the absolute — and the pinned one has to be updated deliberately by a human:

# the nightly delta, and the distance from a pinned reference
regress-summary runs last_night.tsv
regress-summary runs known_good.tsv

That is a policy rather than a feature, and it is the one that catches the failure mode nobody notices: a regression that gets 1% worse every week for ten weeks, with every single night reporting +0 or +1.

6. What the Summary Cannot Say

Module 29 closes where Chapter 29.6 §6 left it, because a summary inherits every limit of the logs it is built from.

It reports that 4 runs passed. Chapter 29.4 measured that one of those four — noreset_s1 — passes with a design that has no reset, because the test never exercised it. The summary is correct and the regression is not.

It reports 1 inconsistent run, which is a defect in a testbench rather than in a design, and the only reason it is visible is that two independent signals were compared.

It reports 2 runs with no verdict, and cannot distinguish a compile failure from a killed job without opening them — which is why the last section prints paths.

So the honest statement of what this screen is: a reliable account of what the checks said, and a complete silence about what they did not check. The number that would close that gap — how much of the design was exercised — is not in any log, and producing it is a different discipline. A summary that presents a pass rate as a measure of design quality is overstating what it has; one that presents it as a measure of what the suite looked at and found is exactly right.

That is the last and most important thing to carry out of Module 29. Everything in it makes logs trustworthy. None of it makes a test suite sufficient.

Practice

1. Write down the three questions your current regression report answers, in the order it answers them.
2. Find the pass rate on it and decide whether you can tell from the report alone whether that is better or worse than yesterday.
3. Count the states your report distinguishes. If it is two, find a killed run and work out which column it lands in.
4. Add a representative line to each failure signature and check whether two previously-merged classes are now distinguishable.
5. Build a baseline file, then fix one failing run and break one passing run.
6. Compare on the run name only and observe what the comparison reports.
7. Explain why join -v2 found nothing, in terms of what the key contained.
8. Rekey on name and verdict with comm -13 and confirm both changes appear.
9. Explain why the (+0) line made the wrong answer more convincing, not less.
10. Write regress-summary with mktemp and a trap, and confirm the temporaries are gone after an interrupt.
11. Make it write its own baseline and run it twice in a row with no changes.
12. Introduce a new failure shape — a run whose log is empty — and find which bucket it lands in.
13. Reword one $display in a testbench and watch a signature split.
14. Set up two baselines, a nightly and a pinned one, and describe the degradation each would catch.
15. Find the run in your own regression that passes with a broken design, by mutating something your tests do not look at.

Check Your Understanding

1. What are the three questions, and why is "is it worse" first?
2. Why is a pass rate without a delta insufficient?
3. Why does the report distinguish four states rather than two?
4. Why does each signature carry a representative line?
5. Why is the last section paths rather than descriptions?
6. What did the first comparison report after one run was fixed and one broken?
7. Why was the count genuinely unchanged?
8. Why did join -v2 find nothing?
9. What is the correct key, and which command compares it?
10. Why did the correct (+0) make the incorrect same set harder to catch?
11. Why does the tool write summary.tsv?
12. Why must it exit non-zero for an inconsistent run?
13. What are the three ways a summary tool breaks over a year?
14. Why are two baselines better than one?
15. What can this screen reliably say, and what is it silent about?

7. Remember

• Three questions, in order: is it worse, what is failing, what should I read. Everything else is a second screen.
• A pass rate without a delta makes the reader compare from memory.
• Four states, not two. A failed compile and a killed run otherwise land wherever the classifier happens to put them.
• Each signature needs a representative line, because normalisation merges distinct bugs.
• The last section is paths, because the next action is to open a log.
• My own comparison reported same set while two runs had flipped — the key was the run name, present in both files, so join -v2 matched everything.
• Compare the set with comm -13 on name and verdict. A count is a projection and two opposite changes share one.
• A correct number beside a wrong statement is harder to catch than either alone.
• Write the baseline you will be compared against, and keep a pinned known-good one as well as last night's.
• Exit non-zero for anything that is not a pass, inconsistency included.
• The screen is a reliable account of what the checks said, and silent about what they did not check. Logs can be made trustworthy; that does not make a suite sufficient.

8. What Is Next

Module 29 leaves every log in the flow readable, countable and comparable, with a one-screen report that gates on the result. Chapter 31.7 extends that report to a whole regression, where the runs that produced no log at all turn out to be the ones a summary most easily loses. Module 30 steps back from the artifacts to the machines that produce them: long jobs on shared compute, what changes when a scheduler runs your command instead of you, and why a command that works at your terminal fails when it is submitted.