Collecting Results Into One Place

Part II · Module 32 · Chapter 32.4 · Collecting Results

Chapter 32.3 gave every run a word. Those words are scattered across one directory per run, which is right for isolation (Chapter 31.6) and useless for answering a question about the regression.

This chapter makes one file. The reason it gets a chapter of its own is that every later stage reads it and nothing else — the report, the rerun, the trend and the CI gate — so its shape determines what those stages are capable of.

1. The Schema

#spec           test          seed  verdict       code  tool_rc  timedout  checks  errors  seconds  logbytes
smoke:1         smoke         1     PASS          0     0        0         3       0       0        136
smoke:2         smoke         2     PASS          0     0        0         3       0       0        136
fairness:1      fairness      1     PASS          0     0        0         6       0       0        143
fairness:2      fairness      2     PASS          0     0        0         6       0       0        143
random:1        random        1     PASS          0     0        0         52      0       0        141
random:2        random        2     PASS          0     0        0         56      0       0        141
b2b:1           b2b           1     PASS          0     0        0         8       0       0        133
b2b:2           b2b           2     PASS          0     0        0         8       0       0        133
prio0:1         prio0         1     FAIL          1     0        0         1       1       0        170
prio0:2         prio0         2     FAIL          1     0        0         1       1       0        170
inconsistent:1  inconsistent  1     INCONSISTENT  5     0        0         1       0       0        205
inconsistent:2  inconsistent  2     INCONSISTENT  5     0        0         1       0       0        205
crash:1         crash         1     CRASH         3     1        0         -       -       0        146
crash:2         crash         2     CRASH         3     1        0         -       -       0        146
silent:1        silent        1     NO_VERDICT    4     0        0         -       -       0        137
silent:2        silent        2     NO_VERDICT    4     0        0         -       -       0        137
flaky:1         flaky         1     PASS          0     0        0         1       0       0        136
flaky:2         flaky         2     FAIL          1     0        0         1       1       1        188
hang:1          hang          1     TIMEOUT       2     0        1         -       -       5        106
hang:2          hang          2     TIMEOUT       2     0        1         -       -       5        106
vacuous:1       vacuous       1     VACUOUS       7     0        0         0       0       1        140
vacuous:2       vacuous       2     VACUOUS       7     0        0         0       0       1        140
latecrash:1     latecrash     1     INCONSISTENT  5     1        0         1       0       1        207
latecrash:2     latecrash     2     INCONSISTENT  5     1        0         1       0       1        207

Eleven columns. Each one is there because some question needs it, and four of them are there because of a specific measured behaviour.

Read the three bold rows together. vacuous:1, latecrash:1 and smoke:1 all have verdict columns that a careless schema would reduce to the same thing — and they differ in exactly one column each. The schema is what makes Chapter 32.3's distinctions survive into the report. Dropping checks to save a column does not lose a column; it loses a class.

2. The Collector

#!/bin/bash
# gather SPECS RUNSDIR -- one tab-separated row per spec, in the matrix's order.
# A spec with nothing on disk still gets a row. This file is the only thing the
# report, the rerun and the trend ever read.
set -u
specs=${1:-}; runs=${2:-}
[ -r "$specs" ] || { echo "gather: cannot read '$specs'" >&2; exit 2; }
[ -d "$runs" ]  || { echo "gather: no run directory '$runs'" >&2; exit 2; }
printf '#spec\ttest\tseed\tverdict\tcode\ttool_rc\ttimedout\tchecks\terrors\tseconds\tlogbytes\n'
field() { [ -r "$1" ] && tr -d ' \n' < "$1" || printf '%s' "$2"; }
while IFS= read -r spec; do
  case $spec in ''|\#*) continue;; esac
  t=${spec%%:*}; s=${spec#*:}; [ "$s" = "$spec" ] && s=1
  d="$runs/${t}_s${s}"
  if [ ! -d "$d" ]; then
    printf '%s\t%s\t%s\tMISSING\t8\t-\t-\t-\t-\t-\t-\n' "$spec" "$t" "$s"; continue
  fi
  if [ ! -r "$d/verdict" ]; then
    printf '%s\t%s\t%s\tNOSTATUS\t9\t%s\t%s\t-\t-\t%s\t%s\n' "$spec" "$t" "$s" \
      "$(field "$d/tool.rc" -)" "$(field "$d/timedout" -)" "$(field "$d/seconds" -)" \
      "$( [ -r "$d/run.log" ] && wc -c < "$d/run.log" | tr -d ' ' || echo - )"
    continue
  fi
  read -r v code < "$d/verdict"
  ck=$(sed -n 's/.*checks=\([0-9]*\).*/\1/p' "$d/run.log" 2>/dev/null | tail -1)
  er=$(sed -n 's/.*errors=\([0-9]*\).*/\1/p' "$d/run.log" 2>/dev/null | tail -1)
  printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
    "$spec" "$t" "$s" "$v" "$code" "$(field "$d/tool.rc" -)" "$(field "$d/timedout" -)" \
    "${ck:--}" "${er:--}" "$(field "$d/seconds" -)" \
    "$( [ -r "$d/run.log" ] && wc -c < "$d/run.log" | tr -d ' ' || echo - )"
done < "$specs"

Four decisions, and the first is the one the whole module rests on.

It iterates the matrix, not the disk

while IFS= read -r spec; do
  ...
done < "$specs"

Not for d in "$runs"/*/. Chapter 31.7 measured what a disk scan does: on a matrix of eight with one run never started, it reported 7 runs found, 7 PASS, exit 0 — a green report on an incomplete regression. A directory scan enumerates what exists, and a run that never started does not exist, so it cannot appear.

Iterating specs.txt means the number of rows equals the number of runs that were supposed to happen, by construction. That one property is what lets the report answer "did the regression finish" at all.

Every spec gets a row, including the ones with nothing behind them

if [ ! -d "$d" ]; then
  printf '%s\t%s\t%s\tMISSING\t8\t-\t-\t-\t-\t-\t-\n' "$spec" "$t" "$s"; continue
fi
if [ ! -r "$d/verdict" ]; then
  printf '%s\t%s\t%s\tNOSTATUS\t9\t...' ...
  continue
fi

Two distinct classes, and the distinction is operational:

• MISSING — no directory at all. The run never started. The launcher was interrupted, or the matrix grew after dispatch.
• NOSTATUS — a directory exists but holds no verdict. The run started and its wrapper did not finish, which Chapter 31.7 produced by killing the launcher and found left four directories, zero status files and zero-byte logs.

Those lead to different questions — "why did dispatch stop" against "what killed the wrapper" — and both lead somewhere different from "a test failed".

Codes 8 and 9 are deliberately outside Chapter 32.3's 0–7 range, because they are not verdicts. No run produced them; gather did, by noticing an absence. Keeping them numerically separate means a query can ask "is this a verdict or a gap" with a comparison rather than a list of names.

It reads only files, and tolerates their absence

field() { [ -r "$1" ] && tr -d ' \n' < "$1" || printf '%s' "$2"; }

Every per-run fact comes from a small file that the run itself wrote — verdict, tool.rc, timedout, seconds — and each read has a default. A missing seconds file produces -, not an empty column and not an error.

The tr -d ' \n' is not decoration. These files end in a newline, and a newline inside a tab-separated field ends the row — turning one result into two malformed ones. Chapter 31.6 measured what malformed rows look like when they arrive by a different route, and the salient property there applies here too: the row count still comes out right, so a count-based check cannot see it.

It extracts counts from the log with tail -1

ck=$(sed -n 's/.*checks=\([0-9]*\).*/\1/p' "$d/run.log" 2>/dev/null | tail -1)

A verbose run prints checks= twice — once mid-run, once in the summary — and the last one is the total. Chapter 29.3 measured the same hazard from the other side: a summary line that also matches the pattern you are counting, which turned one warning class into eight.

3. What the File Can Answer

With a fixed schema and one row per spec, the questions become one-liners. All of these run against the file above, with no access to the run directories:

# how many of each class?
$ awk -F'\t' 'NR>1 {c[$4]++} END {for (k in c) printf "%d\t%s\n", c[k], k}' results.tsv \
    | sort -k1,1rn -k2,2 | awk -F'\t' '{printf "%-13s %d\n",$2,$1}'
PASS          9
INCONSISTENT  4
FAIL          3
CRASH         2
NO_VERDICT    2
TIMEOUT       2
VACUOUS       2

# which runs are not a pass, with the evidence beside them?
$ awk -F'\t' 'NR>1 && $4!="PASS" {printf "%-16s %-13s rc=%s checks=%s\n", $1,$4,$6,$8}' results.tsv

# which tests behaved DIFFERENTLY across their seeds?
$ awk -F'\t' 'NR>1 { if (!(($2) in first)) first[$2]=$4
                      else if ($4!=first[$2]) differs[$2]=first[$2]" / "$4 }
               END { n=0; for (t in differs) { printf "  %-14s %s\n", t, differs[t]; n++ }
                     if (!n) print "  (every test gave the same verdict on every seed)" }' results.tsv
  flaky          PASS / FAIL

# anything that took suspiciously long?
$ awk -F'\t' 'NR>1 && $10!="-" && $10+0 >= 5 {print $1, $10"s"}' results.tsv
hang:1 5s
hang:2 5s

# where did the time go?
$ awk -F'\t' 'NR>1 && $10!="-" {t+=$10; if ($4!="PASS") w+=$10}
               END {printf "  %ds total, %ds of it on runs that did not pass\n", t, w}' results.tsv
  15s total, 15s of it on runs that did not pass
$ awk -F'\t' 'NR>1 && $10!="-" {s[$4]+=$10} END {for (k in s) printf "%d\t%s\n",s[k],k}' results.tsv \
    | sort -k1,1rn -k2,2 | awk -F'\t' '$1>0{printf "    %-13s %ds\n",$2,$1}'
    TIMEOUT       10s
    INCONSISTENT  2s
    VACUOUS       2s
    FAIL          1s

The third query is the one worth noticing. flaky PASS / FAIL is a seed-dependent test identified by one awk program, from a file, after the regression finished — which is Chapter 31.3's whole argument arriving as a report rather than as a theory.

And the last pair quantifies something uncomfortable: of 15 seconds of simulation, all 15 were spent on runs that did not pass — every genuine pass completed inside a second — and two thirds of that went to the two hung runs waiting out their limit. On this fixture that is an artifact of thirteen deliberately-broken tests. On a real regression it is the number that decides whether to fix the testbenches before adding tests.

4. What It Costs

A regression's run directories are many small files, and that has a price worth knowing before deciding what to keep:

$ find regr/20261009T093929Z -type f | wc -l
     151
$ find regr/20261009T093929Z -type f -print0 | xargs -0 wc -c | tail -1
    8585 total
$ du -sk regr/20261009T093929Z
604     regr/20261009T093929Z

151 files, 8,585 bytes of content, 604 KB on disk — a factor of 72. Twenty-four run.log files account for 3.6 KB of that content; the rest is verdict, tool.rc, timedout, seconds and spec, five tiny files per run, each of which occupies a whole filesystem block.

That is the argument for the results file in its most concrete form. results.tsv is 1,085 bytes and answers every question in Section 3. The 604 KB of run directories exist to be opened when one specific run needs investigating, and Chapter 32.8 measures what to do with them after that.

It is also an argument against adding more per-run files casually. A sixth file per run costs another block per run, and on a ten-thousand-run regression that is 40 MB of nothing. When a new fact is needed per run, the cheap place for it is another column in the file the run already writes — which is why verdict holds a word and a number on one line rather than living in two files.

Practice

1. Write out the schema your own flow would need, one column per question.
2. For each column, name the behaviour that would be indistinguishable without it.
3. Find two outcomes in your flow that differ in exactly one field, then delete that field and see what collapses.
4. Choose tabs over commas and find a real log message that justifies it.
5. Write a # header line and the one-liner that strips it.
6. Align the file with spaces and then watch cut -f fail.
7. Put an empty field in a row and read it with read -r a b c; then read it with awk -F'\t'.
8. Write the collector to iterate the matrix, then rewrite it to scan the disk and compare on an incomplete regression.
9. Delete one run directory and confirm the row still appears.
10. Remove a run's verdict file and confirm you get a different class from a deleted directory.
11. Justify the two gap codes being outside the verdict range.
12. Forget the tr -d '\n' and find the malformed row — then confirm the row count is still right.
13. Make a run print its check count twice and check which one you extract.
14. Write the five queries from Section 3 against your own file.
15. Find a seed-dependent test with one awk line.
16. Compute how much of your regression's time was spent on runs that proved nothing.
17. Add a column, re-run the collector against existing run directories, and diff against the stored file.
18. Count your files, their content and their disk usage, and compute the ratio.
19. Work out what a sixth per-run file would cost across your real run count.

Check Your Understanding

1. Why does the results file get a chapter of its own?
2. Name the three columns that each rescue one otherwise-invisible class.
3. What is lost by dropping the checks column to save space?
4. Why tabs rather than commas?
5. Why a single # header line?
6. Why must the file not be space-aligned?
7. What does an empty field do to read -r a b c, and why write - instead?
8. Why iterate the matrix rather than the run directory?
9. What did a disk-scanning collector report about a matrix of eight?
10. What property does iterating the matrix guarantee?
11. What is the difference between MISSING and NOSTATUS, and what different question does each raise?
12. Why are their codes outside the 0–7 range?
13. What does tr -d ' \n' prevent, and why would a row count not catch it?
14. Why tail -1 on the check-count extraction?
15. How do you find a seed-dependent test from the file alone?
16. Name three capabilities that depend on the report reading only this file.
17. What goes wrong when a report re-reads the run directories?
18. When the report needs a fact the schema lacks, what is the right move?
19. How do you check that the collector reproduces its own output?
20. 151 files, 8,585 bytes of content — how much disk, and why?
21. What does that ratio argue against?

5. Remember

• The results file is the interface. Every later stage reads it and nothing else, so its schema bounds what they can do.
• tool_rc, timedout and checks each rescue one class — latecrash, a TERMed hang, and vacuous. Dropping a column drops a class.
• Tabs, one # header, no alignment. Tabs need no quoting; alignment breaks cut -f.
• Write -, never an empty field, because read collapses whitespace and shifts every later column.
• Iterate the matrix, not the disk. A scan reported 7 of 7 PASS, exit 0 on a matrix of eight.
• One row per spec, always — so the row count is the number of runs that should have happened.
• MISSING and NOSTATUS are different questions, and their codes sit outside the verdict range because gather invented them rather than a run reporting them.
• Strip the trailing newline from every per-run file; a newline inside a field splits the row and the row count still looks right.
• tail -1 for a count that appears twice.
• The report must never reach past the file — that is what makes archives, comparisons and self-consistency possible. Need a new fact? Add a column and re-run the collector.
• 151 files, 8.5 KB of content, 604 KB on disk — 72×. The 1 KB results file answers every aggregate question; the directories exist for the one run you investigate.

6. What Is Next

One file now holds everything the regression knows. Chapter 32.5 turns it into a report that answers did anything get worse — and measures a regression whose pass count went up by one while nothing about the design improved at all.