Exit Codes: How the Shell, CI and LSF Learn You Failed

Part II · Module 22 · Chapter 22.4 · Exit Codes: How the Shell, CI and LSF Learn You Failed

1. The Green Nightly

import sys

failed = 3
total = 40

if failed:
    print(f"REGRESSION FAILED: {failed} of {total} tests failed")
else:
    print(f"REGRESSION PASSED: {total} tests")

import subprocess
import sys

done = subprocess.run([sys.executable, "regress.py"], capture_output=True, text=True)
print("what it printed :", done.stdout.strip())
print("what it returned:", done.returncode)

shell = subprocess.run(
    ["/bin/sh", "-c", f"{sys.executable} regress.py && echo 'nightly: step succeeded'"],
    capture_output=True, text=True)
print()
print("$ regress.py && echo 'nightly: step succeeded'")
for line in shell.stdout.strip().splitlines():
    print("   " + line)

what it printed : REGRESSION FAILED: 3 of 40 tests failed
what it returned: 0

$ regress.py && echo 'nightly: step succeeded'
   REGRESSION FAILED: 3 of 40 tests failed
   nightly: step succeeded

The shell continued. && means run the next thing if this one succeeded, the script said it succeeded, and the nightly moved on to the next step with three failing tests behind it.

Nothing is wrong with the message. The message is excellent — it names the count and the total. It is simply not the channel anything upstream reads. The shell, make, a CI runner and a job scheduler all ask a process exactly one question when it finishes, and it is not what did you print.

A script that falls off the end of its last line exits 0. Success is the default, and a script has to go out of its way to report failure.

2. One Number, Everything Upstream

import subprocess
import sys

SCRIPT = (
    "import sys\n"
    "mode = sys.argv[1]\n"
    "if mode == 'ok':\n"
    "    print('PASS')\n"
    "elif mode == 'fail':\n"
    "    print('FAIL'); sys.exit(1)\n"
)

with open("tiny.py", "w", encoding="utf-8") as handle:
    handle.write(SCRIPT)


def sh(command):
    done = subprocess.run(["/bin/sh", "-c", command], capture_output=True, text=True)
    return done.stdout.strip()


PY = sys.executable
print("$ tiny.py fail && echo CONTINUED")
print("   ", sh(f"{PY} tiny.py fail && echo CONTINUED").replace("\n", " | "))
print("$ tiny.py fail || echo CAUGHT")
print("   ", sh(f"{PY} tiny.py fail || echo CAUGHT").replace("\n", " | "))
print("$ tiny.py ok && echo CONTINUED")
print("   ", sh(f"{PY} tiny.py ok && echo CONTINUED").replace("\n", " | "))
print('$ tiny.py fail ; echo "status was $?"')
print("   ", sh(f'{PY} tiny.py fail ; echo "status was $?"').replace("\n", " | "))

$ tiny.py fail && echo CONTINUED
    FAIL
$ tiny.py fail || echo CAUGHT
    FAIL | CAUGHT
$ tiny.py ok && echo CONTINUED
    PASS | CONTINUED
$ tiny.py fail ; echo "status was $?"
    FAIL | status was 1

Four shell constructs, one number. && runs the next command only on zero; || only on non-zero; $? is the number itself.

Everything upstream is built on those semantics:

None of them reads stdout. The exit status is the only thing a script says to a machine.

3. Three Codes

The canonical annotation, with its comments:

sys.exit(0)   # everything passed
sys.exit(1)   # tests failed
sys.exit(2)   # could not even run

The distinction between 1 and 2 is the one worth insisting on, and it is the same distinction Chapter 20.11 made between FAIL and NO_RUN:

Exit 1 means the tool worked. It read the log, counted the errors, and the answer was bad. A person should look at the design or the testbench.

Exit 2 means the tool did not work. The arguments were wrong, the log was missing, the simulator was not on the path. A person should look at the command or the environment.

A flow that collapses them sends an engineer to debug RTL because somebody mistyped a flag. Chapter 20.11 Section 2 made exactly this argument about a status string; this is the same argument about the number, and the number is the one the scheduler reads.

import sys

MODES = {"pass": (0, "PASS  0 errors"), "fail": (1, "FAIL  3 errors"),
         "nolog": (2, "cannot read sim.log: no such file")}

mode = sys.argv[1]
code, message = MODES[mode]
if code == 2:
    print(message, file=sys.stderr)
else:
    print(message)
sys.exit(code)

import subprocess
import sys

for mode in ["pass", "fail", "nolog"]:
    done = subprocess.run([sys.executable, "check.py", mode], capture_output=True, text=True)
    print(f"$ check.py {mode:<6} rc={done.returncode}  "
          f"out={done.stdout.strip()!r}  err={done.stderr.strip()!r}")

$ check.py pass   rc=0  out='PASS  0 errors'  err=''
$ check.py fail   rc=1  out='FAIL  3 errors'  err=''
$ check.py nolog  rc=2  out=''  err='cannot read sim.log: no such file'

Note where each message went. The two verdicts are results and go to stdout; the failure to run is a diagnostic and goes to stderr, leaving stdout empty. That is Chapter 22.5's discipline, and it pairs with the exit status: a caller can tell from the number whether reading stdout is worth doing.

4. Why 1 Is Overloaded

import sys

mode = sys.argv[1]
if mode == "fail":
    print("FAIL")
    sys.exit(1)
if mode == "crash":
    raise RuntimeError("boom")
if mode == "usage":
    print("usage: ec.py MODE", file=sys.stderr)
    sys.exit(2)
print("done")

import subprocess
import sys

for mode in ["fail", "crash", "usage", "done"]:
    done = subprocess.run([sys.executable, "ec.py", mode], capture_output=True, text=True)
    last = (done.stdout.strip() or done.stderr.strip().splitlines()[-1])
    print(f"$ ec.py {mode:<6} rc={done.returncode}  {last}")

$ ec.py fail   rc=1  FAIL
$ ec.py crash  rc=1  RuntimeError: boom
$ ec.py usage  rc=2  usage: ec.py MODE
$ ec.py done   rc=0  done

A deliberate sys.exit(1) and an uncaught exception produce the same number. Python exits 1 for any unhandled exception, so "the tests failed" and "the tool crashed with a traceback" are indistinguishable to a caller reading only the status.

That is the reason a third code exists. Reserving 2 for could not run gives a flow one number that cannot be produced by accident — nothing in Python exits 2 on its own, so a 2 is always something your code decided.

It is also why argparse exits 2 for every usage error, which Chapter 22.2 Section 5 observed and Chapter 22.3 Section 3 relied on. A tool using 2 for its own could-not-run cases is consistent with its own argument parser for free.

5. The Status Is Eight Bits

Chapter 17.8 Section 6 established this and it is worth seeing again, because the tempting mistake is specifically a regression-flow mistake:

import sys

n = int(sys.argv[1])
print(f"{n} errors")
sys.exit(n)

import subprocess
import sys

for n in ["0", "1", "3", "255", "256", "300"]:
    done = subprocess.run([sys.executable, "count.py", n], capture_output=True, text=True)
    print(f"$ count.py {n:<4} printed {done.stdout.strip():<12} exit status {done.returncode}")

$ count.py 0    printed 0 errors     exit status 0
$ count.py 1    printed 1 errors     exit status 1
$ count.py 3    printed 3 errors     exit status 3
$ count.py 255  printed 255 errors   exit status 255
$ count.py 256  printed 256 errors   exit status 0
$ count.py 300  printed 300 errors   exit status 44

sys.exit(error_count) looks like a neat way to report how bad it was. At exactly 256 errors it reports complete success, and at 300 it reports 44 — a number that means nothing to anybody.

The status is one byte. Values above 255 wrap, and the one value that wraps to zero is a perfectly plausible error count for a large regression.

The same chapter's other finding is worth re-running, because it is the opposite of what reads naturally:

import sys

ok = sys.argv[1] == "true"
print("ok =", ok)
sys.exit(ok)

import subprocess
import sys

for value in ["true", "false"]:
    done = subprocess.run([sys.executable, "bools.py", value], capture_output=True, text=True)
    print(f"$ bools.py {value:<6} {done.stdout.strip():<12} exit status {done.returncode}")

$ bools.py true   ok = True    exit status 1
$ bools.py false  ok = False   exit status 0

sys.exit(True) is 1 and sys.exit(False) is 0 — because True is 1 as an integer, and 1 means failure. sys.exit(everything_passed) has it exactly backwards, and the symptom is a nightly that fails on every clean run and passes on every broken one.

Write the mapping out:

def status_for(passed, could_run=True):
    """0 passed, 1 failed, 2 could not run."""
    if not could_run:
        return 2
    return 0 if passed else 1


for passed, could_run in [(True, True), (False, True), (False, False), (True, False)]:
    print(f"passed={str(passed):<5} could_run={str(could_run):<5} -> exit {status_for(passed, could_run)}")

passed=True  could_run=True  -> exit 0
passed=False could_run=True  -> exit 1
passed=False could_run=False -> exit 2
passed=True  could_run=False -> exit 2

Four lines, no booleans passed to sys.exit, and the fourth row handled — a tool that could not run has not passed, whatever a stale variable says.

6. The Pipeline That Hides It

A correct exit status can still be thrown away before anything sees it:

import sys

print("FAIL: 3 errors")
sys.exit(1)

import subprocess
import sys


def sh(command):
    done = subprocess.run(["/bin/sh", "-c", command], capture_output=True, text=True)
    return done.stdout.strip().replace("\n", " | ")


PY = sys.executable
print('$ tiny.py ; echo "status $?"')
print("   ", sh(f'{PY} tiny.py ; echo "status $?"'))
print('$ tiny.py | grep -c FAIL ; echo "status $?"')
print("   ", sh(f'{PY} tiny.py | grep -c FAIL ; echo "status $?"'))
print('$ tiny.py | tee out.log ; echo "status $?"')
print("   ", sh(f'{PY} tiny.py | tee out.log ; echo "status $?"'))

$ tiny.py ; echo "status $?"
    FAIL: 3 errors | status 1
$ tiny.py | grep -c FAIL ; echo "status $?"
    1 | status 0
$ tiny.py | tee out.log ; echo "status $?"
    FAIL: 3 errors | status 0

The script exited 1 and the shell reports 0, twice. In a pipeline, $? is the status of the last command — grep found a match and succeeded, tee wrote a file and succeeded, and the failing script's status went nowhere.

That second form is the one that appears in real flows, because | tee run.log is how everybody keeps a copy of the output. A wrapper doing run_sim.py --test $T | tee $LOG has silently disabled every exit status in the tool it is calling.

Three ways out, in order of portability:

import sys

print("FAIL: 3 errors")
sys.exit(1)

import subprocess
import sys


def sh(command):
    done = subprocess.run(["/bin/sh", "-c", command], capture_output=True, text=True)
    return done.stdout.strip().replace("\n", " | ")


PY = sys.executable
print('1. no pipeline      :', sh(f'{PY} tiny.py > out.log ; echo "status $?"'))
print('2. check it first   :',
      sh(f'if {PY} tiny.py > out.log ; then echo "status 0"; else echo "status $?"; fi'))
print('3. let python do it :',
      sh(f'{PY} -c "import subprocess,sys; '
         f'd=subprocess.run([sys.executable,\'tiny.py\'],capture_output=True,text=True); '
         f'open(\'out.log\',\'w\').write(d.stdout); print(\'status\', d.returncode)"'))

1. no pipeline      : status 1
2. check it first   : status 1
3. let python do it : status 1

Redirect instead of piping when all you want is a copy of the output — > out.log keeps the script's own status.

Test it explicitly with if ... then ... else, which works in any POSIX shell.

Do the capture in Python, which is where this flow is heading anyway — Chapter 23.2's subprocess.run gives you .returncode and the output together, with no shell to lose either.

bash and zsh also offer set -o pipefail, which makes a pipeline return the first non-zero status. It is the right answer where you control the shell and can rely on it; it is not POSIX sh, so a script that must run under /bin/sh on a farm image cannot assume it.

7. The Status of a Sweep

A tool that runs many things has to reduce many results to one number:

RUNS = [
    {"test": "axi_smoke", "status": "PASS"},
    {"test": "axi_reset", "status": "PASS"},
    {"test": "axi_burst", "status": "FAIL"},
    {"test": "axi_wrap", "status": "NO_FINISH"},
]

BROKEN = {"NO_RUN", "NO_FINISH"}


def sweep_status(runs, could_run=True):
    """0 if every run passed, 1 if any failed, 2 if the sweep itself broke."""
    if not could_run or not runs:
        return 2
    if any(run["status"] in BROKEN for run in runs):
        return 1
    return 0 if all(run["status"] == "PASS" for run in runs) else 1


print("mixed sweep :", sweep_status(RUNS))
print("all passed  :", sweep_status([{"test": "a", "status": "PASS"}]))
print("empty sweep :", sweep_status([]))
print("could not   :", sweep_status(RUNS, could_run=False))

mixed sweep : 1
all passed  : 0
empty sweep : 2
could not   : 2

Three decisions in that function, and each is a judgment a flow has to make explicitly.

A NO_FINISH run counts as a failure, not as a could-not-run. The sweep itself worked; one job inside it did not finish. Exit 2 is reserved for the sweep being unable to do its job at all.

An empty sweep is 2, not 0. Zero tests all passing is technically true and is never what anybody meant — it is Chapter 19.12 Section 8's NO DATA rule and Chapter 20.11's empty-log rule, now deciding a nightly's colour. A filelist that matched nothing, a suite name that was misspelled, a directory that was already cleaned: all of them produce an empty sweep, and all of them should be loud.

The reduction loses information on purpose. One number cannot say which four tests failed, and it does not need to — that is what the results table from Chapter 21.3 is for. The status answers should anybody look, and the table answers at what.

8. Predict, Run, Observe

import sys

mode = sys.argv[1]
if mode == "fallthrough":
    print("done")
elif mode == "zero":
    sys.exit(0)
elif mode == "true":
    sys.exit(True)
elif mode == "false":
    sys.exit(False)
elif mode == "big":
    sys.exit(256)
elif mode == "neg":
    sys.exit(-1)
elif mode == "text":
    sys.exit("something went wrong")
elif mode == "raise":
    raise ValueError("nope")

import subprocess
import sys

for mode in ["fallthrough", "zero", "true", "false", "big", "neg", "text", "raise"]:
    done = subprocess.run([sys.executable, "po4.py", mode], capture_output=True, text=True)
    last = (done.stdout.strip() or (done.stderr.strip().splitlines() or [""])[-1])
    print(f"{mode:<12} rc={done.returncode:<4} {last[:40]}")

The verified output:

fallthrough  rc=0    done
zero         rc=0    
true         rc=1    
false        rc=0    
big          rc=0    
neg          rc=255  
text         rc=1    something went wrong
raise        rc=1    ValueError: nope

fallthrough — reaching the end of the script is success. The default is 0 and a script must act to say otherwise.

true and false — inverted from how they read. sys.exit(everything_passed) fails every passing run.

big — 256 becomes 0. An error count is not an exit status.

neg — -1 becomes 255, because the byte wraps the other way.

text — a string argument is printed to stderr and the status is 1. Occasionally convenient, and it gives up the ability to distinguish 1 from 2.

raise — an uncaught exception is also 1, which is why Section 4 argues for reserving 2.

9. Exercises

Exercise 1 — the green nightly. Write a script that prints a failure and exits 0. Run it with && echo CONTINUED and record what happens.

Exercise 2 — the four constructs. For a failing script, show &&, ||, $? and a successful run with &&.

Exercise 3 — who reads it. For your own flow, name the thing that consumes your tool's exit status and say what it does with a non-zero.

Exercise 4 — three codes. Add 0, 1 and 2 to a tool of your own. Write one sentence per code saying where it sends a person.

Exercise 5 — stdout or stderr. For each of your three codes, decide which stream the message goes to and justify it.

Exercise 6 — the overloaded 1. Make your tool fail deliberately and crash accidentally. Confirm both exit 1 and say what a caller cannot tell.

Exercise 7 — 256 errors. Write sys.exit(error_count) and run it with 255, 256 and 300. Record all three statuses.

Exercise 8 — the inverted boolean. Run sys.exit(all_passed) on a passing run. Record the status and say what a nightly would report.

Exercise 9 — the mapping function. Write status_for(passed, could_run) and test all four combinations, including the contradictory one.

Exercise 10 — tee. Run a failing tool through | tee out.log and print $?. Record the status and explain where the real one went.

Exercise 11 — three ways out. Fix Exercise 10 by redirecting, by testing explicitly, and by capturing in Python. Say which your flow can rely on.

Exercise 12 — the sweep. Reduce four mixed results to one status. Decide what an empty sweep returns and defend it in two sentences.

10. Summary

• A script printing REGRESSION FAILED exits 0 and a && chain continues. The message is for a person; the status is for everything else.
• Falling off the end of a script is success. A script must act to report failure.
• The shell, make, CI and a scheduler all read the number and none reads stdout.
• && runs on zero, || on non-zero, $? is the number.
• Three codes: 0 ran and passed, 1 ran and something failed, 2 could not run at all.
• 1 versus 2 is the same distinction as Chapter 20.11's FAIL versus NO_RUN — the design, or the command and the environment.
• An uncaught exception also exits 1, so 1 cannot distinguish a failed run from a crash. Reserving 2 gives a flow a number nothing produces by accident.
• argparse already exits 2 for usage errors, so a tool using 2 is consistent with its own parser for free.
• The status is eight bits. sys.exit(256) reports success and sys.exit(300) reports 44.
• sys.exit(True) is 1 and sys.exit(False) is 0 — exactly backwards from how it reads.
• Map explicitly with a small function rather than passing a boolean or a count.
• In a pipeline, $? is the last command's status, so | tee run.log discards the status of everything before it.
• Redirect instead of piping, test explicitly, or capture in Python. pipefail works where you control the shell and is not POSIX sh.
• A sweep reduces many results to one number: any failure is 1, the sweep itself breaking is 2, and an empty sweep is 2, not 0.
• The reduction loses detail on purpose — the status says whether to look, the results table says at what.

The next chapter is about the two streams those messages went to, and why the split is what lets a tool be piped at all.