Collecting Exit Status From Many Jobs

Part II · Module 31 · Chapter 31.7 · Collecting Exit Status From Many Jobs

Chapter 31.6 gave every run a directory of its own. What remains is to turn 240 runs into one answer — and the first thing to establish is how little the launcher's own exit status can contribute to it.

1. One Status Cannot Describe Two Hundred Runs

Chapter 31.5 measured that xargs collapses every child failure to 1. That is a loss of detail. This is worse:

  one child among 8      xargs rc   completed  xargs message
  exits 0                0          7 of 8     (none)
  exits 7                1          7 of 8     (none)
  exits 255              1          3 of 8     xargs: bash: exited with status 255; aborting
  killed by signal 9     1          3 of 8     xargs: bash: terminated with signal 9; aborting

Read the last three rows together. rc=1 describes both "seven of eight ran and one failed" and "three of eight ran and the regression was abandoned." Those are opposite reports — one is a bug to debug, the other is a regression that produced almost no data — and the exit status is identical.

The manual says why:

     If a command line cannot be assembled, or cannot be invoked, or if an
     invocation of utility is terminated by a signal, or an invocation of
     utility exits with a value of 255, the xargs utility stops processing
     input and exits after all invocations of utility finish processing.

So two ordinary events truncate a regression silently:

A tool killed by a signal. A simulator that segfaults, a run killed by the out-of-memory killer — Chapter 11.1 found 137 to mean that almost always — or a job evicted at a resource cap, which Chapter 30.6 measured arriving as 128 + SIGXFSZ. Any one of them stops xargs feeding the remaining specs.

A tool exiting 255. Which is not an exotic value: it is what a tool returns when it exits -1, and Chapter 21.4's rule — keep your own statuses under 126, because above that you collide with the shell's reserved values and with 128 + signal — is a rule tools break.

The only thing that distinguishes the two cases is the message on standard error — aborting — and a launcher that redirects xargs output to a log nobody reads, or to /dev/null, discards it.

2. Each Run Records Its Own Verdict

The run is the only process that knows what happened to it, so the run writes it down:

#!/bin/bash
# run_one4.sh SPEC -- like run_one.sh, but records its own status on disk.
set -u
[ $# -gt 0 ] || { echo "run_one4.sh: need TEST or TEST:SEED" >&2; exit 2; }
spec=$1
test=${spec%%:*}; seed=${spec#*:}; [ "$seed" = "$spec" ] && seed=1
[ -n "${ROOT:-}" ] || { echo "run_one4.sh: set ROOT" >&2; exit 2; }
[ -n "${RUNS:-}" ] || { echo "run_one4.sh: set RUNS" >&2; exit 2; }
root=$ROOT
d="$RUNS/${test}_s${seed}"
mkdir -p "$d" || exit 2
printf '%s\n' "$spec" > "$d/spec"
( cd "$d" && exec vvp "$root/logs/sim.vvp" \
    +TESTNAME="$test" +SEED="$seed" +CYCLES="${CYCLES:-120000}" +STRICT ) > "$d/run.log" 2>&1
rc=$?
if grep -q 'TEST PASSED' "$d/run.log"; then v=PASS; s=0
elif grep -q 'TEST FAILED' "$d/run.log"; then v=FAIL; s=1
else v=UNKNOWN; s=3
fi
# status last, and in one write, so a partial file cannot be read as a verdict
printf '%s %s %s\n' "$v" "$s" "$rc" > "$d/status.tmp" && mv "$d/status.tmp" "$d/status"
exit "$s"

Four decisions in the last four lines.

The verdict and a status code are both recorded, because PASS/FAIL/UNKNOWN is what a human reads and the code is what a script branches on — Chapter 31.1's three-way status, preserved past the launcher.

The tool's own exit status is recorded separately as the third field. The verdict comes from the log; $rc comes from the simulator. When those two disagree, something interesting has happened, and §5 contains an example.

The status file is written last, after the log has been read. A status file that exists is therefore a promise that the run got to the end of the wrapper.

And it is written atomically, via a temporary name and mv:

printf '%s %s %s\n' "$v" "$s" "$rc" > "$d/status.tmp" && mv "$d/status.tmp" "$d/status"

A rename within one filesystem is atomic, so a reader either sees no status file or sees a complete one — never a half-written line that parses as a verdict. Chapter 31.6 measured what unprotected writes do under parallelism; this is the same problem with a different fix, and the fix is available here because the file has exactly one writer.

The spec file beside it records what this directory was for, which makes the directory self-describing when somebody finds it three weeks later.

3. The Reporter Everybody Writes First

#!/bin/bash
# collect_naive RUNSDIR -- the reporter everybody writes first: scan the disk.
set -u
runs=${1:?need a runs dir}
np=0; nf=0
for d in "$runs"/*/; do
  [ -f "$d/status" ] || continue
  read -r v code rc < "$d/status"
  case $v in PASS) np=$((np+1));; *) nf=$((nf+1));; esac
done
printf 'naive: %d runs found, %d PASS, %d not-PASS\n' "$((np+nf))" "$np" "$nf"
[ "$nf" -eq 0 ] && exit 0 || exit 1

It is short, it is obvious, and it is wrong in a specific way. Here it is on a matrix of eight runs, every one of which passes when left alone, after one run directory was removed — standing in for a run the launcher never started:

=== A. the naive reporter ===
naive: 7 runs found, 7 PASS, 0 not-PASS
  exit status: 0

Seven of seven passed, exit 0, green. The matrix had eight.

The defect is in the first line of the loop: for d in "$runs"/*/. A directory scan enumerates what exists, and a missing run does not exist, so it cannot appear in the output. The reporter is not failing to notice an error — there is no error in its input to notice. It is answering a different question from the one that was asked: "did everything I can see pass" rather than "did everything I asked for pass".

4. Reconciling Against the Matrix

#!/bin/bash
# collect SPECS RUNSDIR -- reconcile the matrix against the status files on disk.
# Every spec gets exactly one row. A spec with no status file is a row too.
set -u
specs=${1:-}; runs=${2:-}
[ -r "$specs" ] || { echo "collect: cannot read specs file '$specs'" >&2; exit 2; }
[ -d "$runs" ]  || { echo "collect: no run directory '$runs'" >&2; exit 2; }
np=0; nf=0; nu=0; nm=0; nb=0
while IFS= read -r spec; do
  case $spec in ''|\#*) continue;; esac
  t=${spec%%:*}; s=${spec#*:}; [ "$s" = "$spec" ] && s=1
  d="$runs/${t}_s${s}"
  if [ ! -f "$d/status" ]; then
    if [ -d "$d" ]; then printf '%-20s %-8s %s\n' "$spec" "NOSTATUS" "started, no verdict recorded"; nb=$((nb+1))
    else                 printf '%-20s %-8s %s\n' "$spec" "MISSING"  "no run directory at all"; nm=$((nm+1)); fi
    continue
  fi
  read -r v code rc < "$d/status"
  case $v in
    PASS) np=$((np+1));;
    FAIL) nf=$((nf+1));;
    *)    nu=$((nu+1));;
  esac
  printf '%-20s %-8s %s\n' "$spec" "$v" "exit=$code tool=$rc"
done < "$specs"
total=$(( np + nf + nu + nm + nb ))
printf -- '----\n'
printf 'collect: %d specs: %d PASS, %d FAIL, %d UNKNOWN, %d NOSTATUS, %d MISSING\n' \
  "$total" "$np" "$nf" "$nu" "$nb" "$nm"
# any row that is not a clean PASS makes the regression non-green
[ $(( nf + nu + nb + nm )) -eq 0 ] && exit 0 || exit 1

The loop is over specs.txt, not over the directory, and that one change produces all three of the categories the naive version cannot represent:

On the same eight-spec matrix:

=== B. reconciled against the matrix ===
fairness:1           PASS     exit=0 tool=0
fairness:2           PASS     exit=0 tool=0
fairness:3           PASS     exit=0 tool=0
fairness:4           PASS     exit=0 tool=0
smoke:1              PASS     exit=0 tool=0
smoke:2              PASS     exit=0 tool=0
random:1             PASS     exit=0 tool=0
random:2             MISSING  no run directory at all
----
collect: 8 specs: 7 PASS, 0 FAIL, 0 UNKNOWN, 0 NOSTATUS, 1 MISSING
  exit status: 1

Eight rows for eight specs, the missing one named, and a red exit status. Against the naive reporter's 7 PASS, exit 0 on identical input.

Two details in that script are worth more than they look.

Every spec produces exactly one row, including the ones with nothing to report. The report's length equals the matrix's length, which makes a truncated regression visible at a glance rather than requiring arithmetic.

And the exit status counts everything that is not a clean PASS:

[ $(( nf + nu + nb + nm )) -eq 0 ] && exit 0 || exit 1

Not [ "$nf" -eq 0 ]. A regression with a MISSING run has not passed; it has not finished. Treating "no data" as "no failures" is the mistake that makes the whole exercise pointless. Chapter 29.8 reached the same conclusion from the other direction, reporting 4 passed, 5 failed, 2 no verdict, 1 inconsistent rather than collapsing four states into two.

5. A Run Whose Simulator Was Killed

Eight long runs, one of their simulators killed mid-flight with kill -9:

=== 8 long runs at -P 8, all of which pass if left alone ===
  kill -9 29706   (one of the eight simulators)
  xargs exit status: 1

=== A. the naive reporter ===
naive: 8 runs found, 7 PASS, 1 not-PASS
  exit status: 1

=== B. reconciled against the matrix ===
heavy:1              PASS     exit=0 tool=0
...
heavy:6              UNKNOWN  exit=3 tool=137
heavy:7              PASS     exit=0 tool=0
heavy:8              PASS     exit=0 tool=0
----
collect: 8 specs: 7 PASS, 0 FAIL, 1 UNKNOWN, 0 NOSTATUS, 0 MISSING
  exit status: 1

Both reporters go red, so this is not a false green. The difference is what they say.

The naive reporter calls it 1 not-PASS — the same words it would use for a design bug.

collect calls it UNKNOWN exit=3 tool=137, and 137 is the whole story: it is 128 + 9, the shell's encoding of killed by signal 9, which Chapter 11.1 established and Chapter 30.4 used to distinguish a cancelled job from a failed one. A verdict of UNKNOWN next to a tool status of 137 says: this run did not fail, it was killed.

That distinction decides the next hour's work. A failing run means reading a waveform. A killed run means asking who killed it — a memory limit, a scheduler's wall-clock limit, a colleague tidying up, an operating system under pressure — and then running it again. Those investigations have nothing in common, and the naive reporter's single phrase sends you down the wrong one.

The wrapper deserves credit for the row existing at all: it recorded UNKNOWN because its own grep of the log found no verdict, which is Chapter 31.1's third status earning its place. A wrapper written as grep -q 'TEST PASSED' && exit 0 || exit 1 would have recorded this killed run as FAIL.

6. The Regression Where Nothing Ran

The last case is the one that matters most, and I produced it by accident.

Four long runs at four slots. I intended to kill one wrapper to see what a wrapper dying would leave behind, and I selected the victim like this:

victim=$(pgrep -f 'run_heavy4.sh' | head -1)
kill -9 "$victim"

The result:

  kill -9 29939  (a wrapper script)
  xargs exit status: 137

=== what is on disk ===
    heavy_s1     NO status file; run.log is 0 bytes
    heavy_s2     NO status file; run.log is 0 bytes
    heavy_s3     NO status file; run.log is 0 bytes
    heavy_s4     NO status file; run.log is 0 bytes

=== A. the naive reporter ===
naive: 0 runs found, 0 PASS, 0 not-PASS
  exit status: 0

=== B. reconciled against the matrix ===
heavy:1              NOSTATUS started, no verdict recorded
heavy:2              NOSTATUS started, no verdict recorded
heavy:3              NOSTATUS started, no verdict recorded
heavy:4              NOSTATUS started, no verdict recorded
----
collect: 4 specs: 0 PASS, 0 FAIL, 0 UNKNOWN, 4 NOSTATUS, 0 MISSING
  exit status: 1

naive: 0 runs found, 0 PASS, 0 not-PASS — exit status 0. A regression in which not one run completed, reported as success.

The mechanism is [ "$nf" -eq 0 ] applied to an empty loop. Zero failures, because zero runs. It is the same false green as Chapter 31.2's empty test list and Chapter 31.6's empty probe — a condition expressed as the absence of bad news instead of the presence of the expected results. Three times in one module, in three different scripts, and all three were mine.

collect cannot produce that answer, because its loop is over the four specs and each one produces a row whether or not anything exists on disk.

7. The Report

A reconciled report is a few lines more than a pass count, and every line is a decision somebody needs:

#!/bin/bash
# report SPECS RUNSDIR -- the regression's one answer.
set -u
specs=${1:-}; runs=${2:-}
[ -r "$specs" ] || { echo "report: cannot read '$specs'" >&2; exit 2; }

out=$(collect "$specs" "$runs"); rc=$?
printf '%s\n' "$out" > "$runs/report.txt"

# the headline, derived from the reconciliation and nothing else
printf '%s\n' "$out" | tail -2

# the three categories that are not a design question, called out separately
printf '%s\n' "$out" | awk '$2=="MISSING" || $2=="NOSTATUS"' > "$runs/incomplete.txt"
if [ -s "$runs/incomplete.txt" ]; then
  printf '\n*** %s run(s) produced no verdict -- the regression did not finish ***\n' \
    "$(grep -c . "$runs/incomplete.txt")"
  sed 's/^/    /' "$runs/incomplete.txt"
fi
printf '%s\n' "$out" | awk '$4=="tool=137" || $4=="tool=139"' > "$runs/killed.txt"
[ -s "$runs/killed.txt" ] && {
  printf '\n*** %s run(s) were killed, not failed ***\n' "$(grep -c . "$runs/killed.txt")"
  sed 's/^/    /' "$runs/killed.txt"
}
exit $rc

The three sections answer three different questions, in the order they have to be answered:

Did the regression finish? MISSING and NOSTATUS say no, and until that is zero the pass rate is not a measurement of the design. Chapter 29.4 measured the end of that road: a counter of error lines called both a failed compile and a killed run PASS, because a run that never happened complains about nothing.

Was anything killed? 137 and 139 (128 + 11, segmentation fault) are environment problems, not design problems. They go to whoever owns the machines.

And then: what failed? Which is the only part most reports contain.

8. Putting the Module Together

The whole module, as one sequence of files:

# 31.2, 31.3, 31.4 -- the matrix, with its size printed before anything runs
SLOTS=8 matrix tests.txt seeds.txt configs.txt > specs.txt

# 31.4 -- record what was supposed to run, next to the results
cp specs.txt "$RUNS/specs.used"

# 31.5, 31.6 -- bounded slots, every run in its own directory
xargs -P "${SLOTS:-8}" -I{} "$BIN/run_one4.sh" {} < specs.txt
echo "launcher rc=$? (informative only)" >&2

# 31.7 -- the answer
report "$RUNS/specs.used" "$RUNS"

Five lines, and each one exists because of something measured rather than assumed: the matrix because nested loops never state their size, the slot bound because oversubscription cost 7% on heavy jobs, the per-run directories because four runs sharing one produced one log and two false failures, and the reconciliation because a disk scan reported success on a regression in which nothing ran.

Note the last line takes specs.used, the copy, not specs.txt. The matrix file in the working tree can change between launch and report — somebody edits a test list, somebody regenerates with another axis — and a report reconciled against a different matrix than the one that ran is worse than no report. Chapter 27.9 made the same argument for the command line.

Practice

1. Tabulate xargs' behaviour for a child exiting 0, 7, 255, and dying on a signal; record the exit status and how many jobs completed.
2. Find the two rows with the same status and different outcomes and say why that matters.
3. Read the manual's sentence about signals and 255, and name two realistic causes of each.
4. Send xargs' stderr to /dev/null and establish what you can no longer tell.
5. Have a run write its own status file with a verdict, a code and the tool's own status.
6. Write it atomically and explain what a reader could otherwise see.
7. Write the naive disk-scanning reporter.
8. Delete one run directory from a matrix of eight all-passing runs and run it.
9. Explain why there is no error in its input to notice.
10. Write the reconciling reporter with one row per spec.
11. Run both on identical input and compare the exit statuses.
12. Make the exit status count MISSING and NOSTATUS as failures and defend that.
13. Kill one simulator mid-regression and find its row.
14. Decode tool=137 and say what investigation it calls for.
15. Predict what a two-way wrapper would have recorded for that run.
16. Run pgrep -f on your own wrapper's name and list everything it matches.
17. Kill the first match and see what happens to the regression.
18. Record the simulator's pid instead and kill that.
19. Produce the "nothing ran" case and confirm the naive reporter reports success.
20. Write the three-section report and explain the order of the sections.
21. Reconcile against a copy of the matrix, and construct the case where reconciling against the original misleads.

Check Your Understanding

1. What two different outcomes does xargs rc=1 cover?
2. What two events make xargs stop reading its input?
3. What is the only thing in the output that distinguishes an abort from a failure?
4. Why is 255 not an exotic exit status?
5. Why does the reporter compare against the matrix rather than the disk?
6. Why does the run itself record the verdict?
7. Why record the tool's exit status separately from the verdict?
8. Why is the status file written last?
9. Why write it through a temporary name and mv?
10. Why does the atomic-rename fix work here but not for Chapter 31.6's results file?
11. What is the defect in for d in "$runs"/*/?
12. What did the naive reporter say about a matrix of eight with one run missing?
13. What question is it actually answering?
14. Name the five categories and what each means.
15. Why must the exit status count MISSING and NOSTATUS?
16. What did the two reporters each say about the killed simulator?
17. What does tool=137 mean and how is it encoded?
18. Why is "killed" a different investigation from "failed"?
19. What would a two-way wrapper have recorded for it?
20. In the final case, what did the naive reporter report, and why?
21. Why did killing one "wrapper" stop all four runs?
22. How many processes did pgrep -f 'run_heavy4.sh' match, and which was first?
23. What is the general rule about broad patterns?
24. What three questions does the report answer, and in what order?
25. Why does the report read specs.used rather than specs.txt?

9. Remember

• xargs rc=1 cannot distinguish one failure from an aborted regression. The same status covered "7 of 8 ran" and "3 of 8 ran".
• A signalled child, or one exiting 255, stops xargs reading its input — and the only trace is the word aborting on standard error.
• The run records its own verdict, with a code and the tool's own status, written last and written atomically through mv.
• A directory scan cannot see a run that never started. It reported 7 PASS, exit 0 on a matrix of eight.
• Reconcile against the matrix, one row per spec, so the report's length equals the matrix's length.
• Five categories, not two: PASS, FAIL, UNKNOWN, NOSTATUS, MISSING.
• MISSING and NOSTATUS make the regression red. "No data" is not "no failures".
• tool=137 is 128 + 9: killed, not failed — a completely different investigation, and a two-way wrapper would have recorded it as FAIL.
• A disk-scanning reporter reported success on a regression in which nothing ran. The third false green of this module, all three from a condition written as the absence of bad news.
• pgrep -f <script> matches the launcher too, and it is the oldest, so head -1 picks it. Record the pid instead.
• Answer three questions in order: did it finish, was anything killed, what failed.
• Reconcile against a copy of the matrix taken at launch, not the file in the working tree.

10. What Is Next

Module 31 has turned a single run into a bounded, isolated, reconciled matrix of runs. Every piece of it is a script invoked by hand.

Module 32, Regression Automation End to End, makes it a system: one entry point, configuration separated from code, a regression that runs unattended on a schedule, results that accumulate across nights, and a report somebody who was asleep can read in the morning.