Returning a Status CI Can Act On

Part II · Module 32 · Chapter 32.7 · Returning a Status CI Can Act On

Everything so far produces text for a person. A nightly job, a CI pipeline or a merge gate reads something much narrower: one number between 0 and 255, plus whatever files it was told to look for.

This chapter is about getting that number right, which turns out to be mostly about the things that take it away from you.

1. The Contract

  0  every run passed
  1  the regression finished; something is not a pass   -> a DESIGN question
  2  the regression ran but did not finish              -> an INFRASTRUCTURE question
  3  there is nothing to report on                      -> a SETUP question

Four values, measured end to end:

  a regression with failures                     1
  a regression that did not finish               2
  a regression that could not start              3
  an all-pass regression                         0

The division is by who acts, not by how bad it is. That is the design principle, and it is why 2 and 3 are separate from 1: a non-passing run goes to whoever owns the design or the testbench, a regression that lost runs goes to whoever owns the machines, and a regression with no results at all goes to whoever broke the setup. Three different people, three different mornings.

Chapter 21.4 established the general rule this follows: a script's exit statuses are an interface, they need documenting, and they need to stay below 126. The documenting is literal — the contract lives in a comment at the top of the script that implements it:

# THE EXIT-CODE CONTRACT, and nothing else may be returned:
#   0  every run passed
#   1  the regression finished; something is not a pass  -> a DESIGN question
#   2  the regression ran but did not finish             -> an INFRASTRUCTURE question
#   3  there is nothing to report on                     -> a SETUP question

2. The Pipe, Which Is How It Actually Gets Lost

The report's status is 1. Here is what five different callers see:

  INVOCATION                                     rc
  rc=$? directly                                 1
  rc=$? after | tail                             0
  rc=$? after | tee (the CI idiom)               0
  PIPESTATUS[0] after | tee                      1
  with set -o pipefail                           1

$? after a pipeline is the last command's status, and in a pipeline ending in tee the last command is tee, which essentially always succeeds. So this:

regress | tee regression.log
exit $?

— exits 0 whatever the regression decided. And that is not a contrived example; it is the single most common last line in a CI script, because somebody wanted the output in a file and on the console.

Chapter 12.8 establishes the rule. What makes it worth a second visit here is where the mistake lands: the one place in the whole system where the status is the only output that matters. Every other stage can be re-run and re-read; this byte is consumed by a machine that then merges or does not merge.

Two fixes, both one line.

# read the right element, IMMEDIATELY
regress | tee regression.log
exit "${PIPESTATUS[0]}"

# or make the pipeline adopt its own failure
set -o pipefail
regress | tee regression.log
exit $?

And set -o pipefail is available here, which is worth confirming rather than assuming given this machine's bash:

$ /bin/bash -c 'set -o pipefail; false | true; echo "rc=$?"'
rc=1
$ /bin/bash -c 'false | true; echo "rc=$?"'
rc=0

3. The Numbers You Cannot Use

A status is one byte, and the shell does not warn you when your value does not fit:

  script exits       caller sees
  exit 0             0
  exit 1             1
  exit 3             3
  exit 125           125
  exit 126           126
  exit 127           127
  exit 128           128
  exit 137           137
  exit 255           255
  exit 256           0
  exit 257           1
  exit 300           44
  exit -1            255

exit 256 reads as 0. The value is taken modulo 256, silently, so a regression that computes its status from a count —

exit "$failures"            # DO NOT

— reports success on exactly 256 failures, and reports 1 on 257. There is no error and no warning. A 512-run regression in which everything failed is green.

That is the strongest possible argument for a small fixed set of statuses rather than a derived number. The count belongs in the report and in results.tsv; the status is a classification with four possible values.

Three further regions to stay out of, per Chapter 21.4:

126 and 127 belong to the shell. 127 is "command not found" and 126 is "found but not executable" — and xargs uses exactly those, which Chapter 32.2's dispatch branches on. A regression that returns 127 of its own accord is indistinguishable from a regression whose driver was not installed.

128 and above belong to signals. 137 is 128 + 9, and Chapter 32.3 measured the simulator's own 137 after a kill -9. A regression returning 130 says "I was interrupted" whether or not it was.

And negative numbers wrap to the top of that range. exit -1 is 255, which looks like a tool's generic failure.

So: 0 to 125, and in practice four of them.

4. Artifacts Are Part of the Interface

A status tells a caller whether; the artifacts tell a person what. Both are part of what a regression returns, and the artifacts need the same discipline as the status — stable paths, decided in advance.

regr/20261009T093929Z/
  report.txt      the human answer          <- publish this
  results.tsv     the machine answer        <- publish this
  env.txt         what this run was         <- publish this
  specs.txt       what was supposed to run
  tests.used      the list, as it was
  seeds.used
  one.sh          exactly what was invoked
  runs/           one directory per run, logs and all

Three of those are small and answer almost every question; the fourth is large and answers the remaining one. A CI job should collect the first three unconditionally — they total about 1.5 KB here — and the run directories only when something is not a pass.

Two rules make this work in practice.

The paths must not depend on the outcome. A regression that writes report.txt only on failure forces the caller to handle its absence, and "absent" then means both "everything passed" and "the regression died before reporting". Write it always.

And the timestamped directory needs a stable alias, because a caller cannot guess 20261009T093929Z:

ln -sfn "$out" "$root/regr/latest"

A symlink updated at the end of a successful report, so regr/latest/report.txt is a fixed path a pipeline can reference. Updated at the end, so latest never points at a regression still in progress — and ln -sfn rather than ln -sf, because without -n a second ln -sf to an existing symlink-to-a-directory creates the link inside the target directory instead of replacing it.

5. Gating On More Than the Status

A merge gate usually wants something finer than "did anything fail", and the results file supports it without touching the status:

# gate 1: the regression must have finished
[ "$rc" -eq 2 ] && { echo "incomplete regression -- not gating on it" >&2; exit 1; }

# gate 2: no NEW failures against the baseline (a known-failures list)
comm -13 <(sort known-failures.txt) \
         <(awk -F'\t' 'NR>1 && $4!="PASS" {print $1}' regr/latest/results.tsv | sort) \
  > /tmp/new-failures.txt
if [ -s /tmp/new-failures.txt ]; then
  echo "NEW failures not in the known list:" >&2; sed 's/^/  /' /tmp/new-failures.txt >&2; exit 1
fi

# gate 3: nothing in the environment classes, which are not the author's fault
awk -F'\t' 'NR>1 && ($4=="TIMEOUT"||$4=="CRASH"||$4=="NOSTATUS") {print $1}' \
  regr/latest/results.tsv > /tmp/env-problems.txt
[ -s /tmp/env-problems.txt ] && echo "warning: environment problems, not gating:" >&2

Three things about this.

Gate 1 refuses to gate rather than failing the author. A regression that did not finish has not demonstrated anything about the change, and failing the merge on it teaches people to re-run until it passes.

Gate 2 is the one that makes a known-failures list honest. comm -13 is new-failures-only; the complementary comm -23 finds entries in the known list that now pass, which is how a known-failures list gets shorter instead of growing forever:

$ comm -23 <(sort known-failures.txt) <(awk … | sort)
prio0:2        # fixed -- remove it from the known list

Both comm directions require sorted input and comm does not check — the same hazard Chapter 32.5 met with join, where unsorted input gives a partial answer and no error.

And gate 3 warns without blocking, because a timeout is not a statement about the submitted change. It is still printed, because a pipeline that silently tolerates environment problems accumulates them.

6. What the Status Must Not Be

Not derived from a count. Section 3: exit 256 is 0.

Not the number of failures even when it fits. exit 3 meaning "three failures" collides with the contract's "nothing to report on", and a caller cannot tell which. The count has a file to live in.

Not a boolean. Collapsing 2 and 3 into 1 is the mistake Chapter 32.1 names: a regression that never started reads as a regression that found bugs.

Not produced by two different places. Chapter 32.2's driver exits with the report's status and nothing else decides. A driver that also exits non-zero when xargs returned 1 would conflate "some child failed" — which Chapter 31.7 measured as meaning anything from one failure to an abandoned regression — with the report's considered answer.

And not the last thing the script happens to run. A script without an explicit final exit returns the status of its last command, so adding a cleanup line at the bottom silently changes the contract. Chapter 15.6 makes the general case; in a regression driver it means the last line is always an explicit exit, and a trap-based cleanup must not end in a command whose success overwrites it.

Practice

1. Write your contract down as a comment in the script that implements it, one line per value.
2. Divide your statuses by who acts, not by severity, and check that each value names a different person.
3. Return the same status for two different situations, then try to write the caller's case statement.
4. Feed your reporter an empty results file and check the status.
5. Feed it a header-only file and check again.
6. Pipe your regression into tee and read $?.
7. Fix it with PIPESTATUS[0], then insert a : before the read.
8. Run the same script under zsh and find out what ${PIPESTATUS[0]} gives you.
9. Fix it with set -o pipefail instead and confirm it works in your bash.
10. Exit 256, 257, 300 and -1 from a script and record what the caller sees.
11. Write exit "$failures" and construct the failure count that reports success.
12. Return 127 deliberately and explain why a caller cannot interpret it.
13. List your artifacts and decide which three a pipeline collects unconditionally.
14. Write report.txt only on failure, then write the caller that handles its absence.
15. Add a latest symlink updated at the end, and explain why not at the start.
16. Use ln -sf without -n twice and find where the second link went.
17. Write the caller's case with a * branch, and justify what that branch does.
18. Introduce a syntax error in the driver and see which contract value it collides with.
19. Add bash -n to CI before the real run.
20. Build a known-failures gate with comm -13, then use comm -23 to find what to remove from the list.
21. Feed comm unsorted input and count what you lose.
22. Delete the final exit from a driver and add a cleanup line; check the status.

Check Your Understanding

1. What does an automated caller actually read?
2. State the four-value contract.
3. What is the division principle, and why not severity?
4. Which two values were originally one, and what two situations did that merge?
5. Why is a zero-row results file the most dangerous input?
6. Why measure both zero-row cases rather than reasoning about them?
7. Why does regress | tee log; exit $? exit 0?
8. Why does this mistake matter more here than anywhere else?
9. Give the two one-line fixes.
10. What destroys PIPESTATUS, and how little does it take?
11. What happens to ${PIPESTATUS[0]} under zsh, and what does the resulting exit do?
12. Why is set -o pipefail the better fix for a final status?
13. What does exit 256 give the caller?
14. What is wrong with exit "$failures", and at exactly what count does it break?
15. Why can a regression not use 126 or 127?
16. Why not 128 and above?
17. What does exit -1 become?
18. What range is actually usable?
19. Why must artifact paths not depend on the outcome?
20. Why update the latest symlink at the end, and why -n?
21. What should the caller's * branch do, and why?
22. Which contract value does a bash syntax error collide with, and what removes the ambiguity?
23. Why does gate 1 refuse to gate rather than fail the author?
24. What does comm -23 find that comm -13 does not?
25. Why warn rather than block on the environment classes?
26. Give four things the status must not be.
27. What happens if a driver has no explicit final exit?

7. Remember

• The caller reads one byte and some files. Both are the interface.
• Four values, divided by who acts: 0 clean, 1 design, 2 infrastructure, 3 setup. Document them in the script that returns them.
• Reserve a distinct value for "nothing to report on" — and measure the zero-row cases, because arithmetic over an empty set returns success.
• regress | tee log; exit $? exits 0. The last command in the pipeline is tee.
• PIPESTATUS is destroyed by the very next command, a bare : included — and under zsh the array is pipestatus, 1-based, so ${PIPESTATUS[0]} is empty and exit "" is green.
• Prefer set -o pipefail for a final status. It works in bash 3.2 and has neither trap.
• exit 256 is 0, exit 300 is 44, exit -1 is 255. Never derive a status from a count.
• Stay in 0–125. 126 and 127 belong to the shell, 128 and up to signals.
• Write the artifacts unconditionally and give the timestamped directory a latest symlink, updated at the end, with ln -sfn.
• Give the caller a * branch that treats an unknown status as a setup problem — and run bash -n first, because a syntax error returns 2.
• Gate on the results file, not the status, for anything finer: refuse to gate on an unfinished regression, block on new failures only, warn on environment classes. sort before comm, in both directions.
• One explicit exit at the end, or the last command written becomes the contract.

8. What Is Next

The regression now runs, decides, records, reports and returns a status an automated caller can act on. Chapter 32.8 runs it every night — and measures why the instinct to delete the passing runs' logs saves 6% while compressing the whole thing saves 97.7%.