Return Codes: What the Tool Actually Told You

Part II · Module 23 · Chapter 23.4 · Return Codes: What the Tool Actually Told You

1. The Number, and the Decision

import sys

mode = sys.argv[1]
CODES = {"pass": 0, "fail": 1, "compile_error": 2, "licence": 7}

print(f"FakeSim: running in {mode} mode")
if mode == "compile_error":
    print("FakeSim: Error-[SE] Syntax error in alu.sv line 42", file=sys.stderr)
sys.exit(CODES[mode])

import subprocess
import sys

for mode in ["pass", "fail", "compile_error", "licence"]:
    cp = subprocess.run([sys.executable, "fake_sim.py", mode],
                        capture_output=True, text=True)
    print(f"{mode:<14} rc={cp.returncode}  stdout={cp.stdout.strip()[:34]!r}")

pass           rc=0  stdout='FakeSim: running in pass mode'
fail           rc=1  stdout='FakeSim: running in fail mode'
compile_error  rc=2  stdout='FakeSim: running in compile_error '
licence        rc=7  stdout='FakeSim: running in licence mode'

Four runs, four numbers, and subprocess.run returned normally for all four — Chapter 23.1 Section 5's point. Deciding what the numbers mean is your job, and this chapter is about where that decision goes.

Note that the codes are the tool's, not Module 22's. A simulator's 2 is whatever its documentation says it is, which is why Chapter 23.11 keeps the mapping in one adapter per tool rather than scattered through a flow.

2. What check=True Does

import sys

print("FakeSim: compiling")
print("FakeSim: Error-[SE] Syntax error in alu.sv line 42", file=sys.stderr)
sys.exit(2)

import subprocess
import sys

try:
    subprocess.run([sys.executable, "fake_sim.py"], check=True,
                   capture_output=True, text=True)
    print("never reached")
except subprocess.CalledProcessError as exc:
    print("raised     :", type(exc).__name__)
    print("returncode :", exc.returncode)
    print("cmd        :", [a.split("/")[-1] for a in exc.cmd])
    print("stdout     :", repr(exc.output))
    print("stderr     :", repr(exc.stderr))

raised     : CalledProcessError
returncode : 2
cmd        : ['python3', 'fake_sim.py']
stdout     : 'FakeSim: compiling\n'
stderr     : 'FakeSim: Error-[SE] Syntax error in alu.sv line 42\n'

check=True raises CalledProcessError on any non-zero status, and the exception carries everything: the code, the command as run, and both captured streams.

So the usual argument against it is wrong, and this chapter is not going to make it. check=True does not throw the output away. What it throws away is the next statement.

3. What It Actually Costs

import sys

print("FakeSim: compiling")
sys.exit(int(sys.argv[1]))

import subprocess
import sys
from pathlib import Path

TESTS = ["axi_smoke", "axi_reset", "axi_burst"]
CODES = {"axi_smoke": 2, "axi_reset": 0, "axi_burst": 1}


def with_check(tests):
    """check=True: the save-and-classify has to live in an except block."""
    results = []
    for test in tests:
        log = Path(f"{test}.log")
        try:
            cp = subprocess.run([sys.executable, "fake_sim.py", str(CODES[test])],
                                check=True, capture_output=True, text=True)
            log.write_text(cp.stdout, encoding="utf-8")
            results.append({"test": test, "status": "PASS", "rc": 0})
        except subprocess.CalledProcessError as exc:
            log.write_text(exc.output, encoding="utf-8")
            results.append({"test": test, "status": "FAIL", "rc": exc.returncode})
    return results


def with_inspection(tests):
    """No check: one path, and the classify is an expression."""
    results = []
    for test in tests:
        cp = subprocess.run([sys.executable, "fake_sim.py", str(CODES[test])],
                            capture_output=True, text=True)
        Path(f"{test}.log").write_text(cp.stdout, encoding="utf-8")
        results.append({"test": test,
                        "status": "PASS" if cp.returncode == 0 else "FAIL",
                        "rc": cp.returncode})
    return results


for label, fn in [("check=True", with_check), ("inspected", with_inspection)]:
    print(label)
    for r in fn(TESTS):
        print(f"   {r['test']:<11} {r['status']:<5} rc={r['rc']}")

check=True
   axi_smoke   FAIL  rc=2
   axi_reset   PASS  rc=0
   axi_burst   FAIL  rc=1
inspected
   axi_smoke   FAIL  rc=2
   axi_reset   PASS  rc=0
   axi_burst   FAIL  rc=1

Identical results, and one of them is twelve lines and the other six.

The duplication in with_check is the cost, and it is structural rather than stylistic: writing the log and appending the result have to happen whatever the tool did, so with check=True they appear twice — once in the try and once in the except. Two copies of the same four lines, which drift.

And the try/except is mandatory, not optional. A sweep over four hundred tests that lets CalledProcessError escape stops at the first failing test — which is the one thing a regression must never do.

So the rule the canonical purpose states:

if cp.returncode != 0:
    return Result(status="COMPILE_ERROR", rc=cp.returncode, log=log_path)

Inspect the number. A non-zero status from a simulator is data your flow was built to collect, and if is the right construct for data.

4. Classify, Then Decide

import sys

print("FakeSim: output")
sys.exit(int(sys.argv[1]))

import subprocess
import sys

# What this tool's documentation says its codes mean.
XRUN_CODES = {0: "PASS", 1: "FAIL", 2: "COMPILE_ERROR", 7: "NO_LICENCE"}


def classify(returncode):
    """The tool's number as this flow's status."""
    if returncode < 0:
        return "KILLED"
    return XRUN_CODES.get(returncode, "UNKNOWN")


def run_one(code):
    cp = subprocess.run([sys.executable, "fake_sim.py", str(code)],
                        capture_output=True, text=True)
    return {"rc": cp.returncode, "status": classify(cp.returncode),
            "log": "sim.log", "output_bytes": len(cp.stdout)}


for code in [0, 1, 2, 7, 99]:
    record = run_one(code)
    print(f"rc={record['rc']:<3} status={record['status']:<14} log={record['log']}")

rc=0   status=PASS           log=sim.log
rc=1   status=FAIL           log=sim.log
rc=2   status=COMPILE_ERROR  log=sim.log
rc=7   status=NO_LICENCE     log=sim.log
rc=99  status=UNKNOWN        log=sim.log

Three properties of that classify are worth copying.

The mapping is a dict, declared once. It is the tool's documentation written as code, and Chapter 23.11 moves it into the per-tool adapter so the rest of the flow never sees a raw number.

UNKNOWN for anything unmapped, rather than a KeyError — a tool release that adds a code should produce a result saying so, not an exception in the middle of a sweep. Chapter 20.6's always-present rule, applied to a status.

The record carries the raw rc as well as the status. The status is for logic and the number is for the bug report, and discarding it means a flow that cannot tell you what it actually saw.

NO_LICENCE is the row that earns the whole arrangement. A licence failure is not a design bug — it is Chapter 20.11's NO_RUN, and a flow that reports it as FAIL sends somebody to debug RTL for a queue that was full.

5. A Negative Number Is a Signal

import signal
import subprocess
import sys

KILLER = "import os, signal; os.kill(os.getpid(), signal.SIGTERM)"

cp = subprocess.run([sys.executable, "-c", KILLER], capture_output=True, text=True)

print("python returncode :", cp.returncode)
print("is negative       :", cp.returncode < 0)
print("the signal        :", signal.Signals(-cp.returncode).name)

shell = subprocess.run(
    ["/bin/sh", "-c", f'{sys.executable} -c "{KILLER}"; echo "shell saw $?"'],
    capture_output=True, text=True)
print("shell reports     :", shell.stdout.strip())
print("128 + 15          :", 128 + 15)

python returncode : -15
is negative       : True
the signal        : SIGTERM
shell reports     : shell saw 143
128 + 15          : 143

One event, two numbers. The child was killed by signal 15; Python reports -15 and the shell reports 143, which is 128 + 15.

This catches people for a specific reason: tool documentation and flow wikis are written from the shell's point of view, so they say the simulator returns 143 when the scheduler kills it. A Python flow checking if rc == 143 never matches, and one checking if rc > 0 misses signals entirely, because a negative number is not greater than zero.

The test is rc < 0, and it matters in exactly the situation a regression cares about most:

import signal

CASES = [0, 1, 2, -15, -9, -6]

for rc in CASES:
    if rc < 0:
        name = signal.Signals(-rc).name
        meaning = {"SIGTERM": "something asked it to stop — usually a scheduler timeout",
                   "SIGKILL": "killed outright — usually out of memory",
                   "SIGABRT": "the tool aborted itself"}.get(name, "a signal")
        print(f"rc={rc:<4} signal {name:<8} {meaning}")
    else:
        print(f"rc={rc:<4} {'exited normally' if rc == 0 else 'the tool exited with a failure'}")

rc=0    exited normally
rc=1    the tool exited with a failure
rc=2    the tool exited with a failure
rc=-15  signal SIGTERM  something asked it to stop — usually a scheduler timeout
rc=-9   signal SIGKILL  killed outright — usually out of memory
rc=-6   signal SIGABRT  the tool aborted itself

Those three signals are the ones a verification flow actually sees. SIGTERM is a scheduler enforcing a time limit, which is Chapter 20.11's NO_FINISH rather than a failure. SIGKILL is almost always the out-of-memory killer, which on a large gate-level simulation is a capacity problem and not a design one. SIGABRT is the tool deciding it cannot continue.

A flow that reports all three as FAIL is reporting three infrastructure problems as design bugs — and it does so silently, because rc != 0 is true for every one of them.

6. When check=True Is Right

import sys

sys.exit(int(sys.argv[1]))

import subprocess
import sys


def prepare(code):
    """A setup step whose failure makes everything after it meaningless."""
    subprocess.run([sys.executable, "setup_step.py", str(code)], check=True)
    return "ready"


print("setup succeeds :", prepare(0))

try:
    prepare(1)
except subprocess.CalledProcessError as exc:
    print("setup fails    : CalledProcessError, rc =", exc.returncode)
    print("                 and nothing after it ran")

setup succeeds : ready
setup fails    : CalledProcessError, rc = 1
                 and nothing after it ran

check=True is right where a failure makes the following code meaningless and there is nothing useful to record:

A setup step. Creating a run directory, copying a filelist, generating a config. If it fails, running the simulator is pointless and there is no simulation log to classify.

A tool you are only asking a question. xrun -version to check what is installed — either it answers or the flow cannot continue.

A one-off script. Chapter 22.1's threshold again: for something you run by hand today, a traceback is an adequate error report.

What those share is that there is no result to record. The moment a tool invocation produces evidence a flow wants — a log, a status, a row in a results table — check=True is the wrong tool, because the recording has to happen on both paths.

7. One Invocation, Completely

Everything together, as a flow would write it:

import sys

print("FakeSim: elaborating")
print("FakeSim: 1842 instances")
if sys.argv[1] == "2":
    print("FakeSim: Error-[SE] Syntax error in alu.sv line 42", file=sys.stderr)
sys.exit(int(sys.argv[1]))

import signal
import subprocess
import sys
from pathlib import Path

XRUN_CODES = {0: "PASS", 1: "FAIL", 2: "COMPILE_ERROR", 7: "NO_LICENCE"}


def classify(returncode):
    if returncode < 0:
        return {signal.SIGTERM: "TIMEOUT_KILLED",
                signal.SIGKILL: "OUT_OF_MEMORY"}.get(-returncode, "KILLED")
    return XRUN_CODES.get(returncode, "UNKNOWN")


def run_sim(test, code, run_dir):
    """Run one simulation. Always returns a record; never raises on tool failure."""
    log_path = Path(run_dir) / f"{test}.log"
    log_path.parent.mkdir(parents=True, exist_ok=True)

    cp = subprocess.run([sys.executable, "fake_sim.py", str(code)],
                        capture_output=True, text=True)

    log_path.write_text(cp.stdout + cp.stderr, encoding="utf-8")

    return {"test": test, "rc": cp.returncode, "status": classify(cp.returncode),
            "log": str(log_path), "log_lines": len(log_path.read_text(encoding="utf-8").splitlines())}


for test, code in [("axi_smoke", 0), ("axi_reset", 1), ("axi_burst", 2), ("axi_wrap", 7)]:
    r = run_sim(test, code, "runs")
    print(f"{r['test']:<11} rc={r['rc']:<3} {r['status']:<14} "
          f"{r['log']:<22} {r['log_lines']} lines")

axi_smoke   rc=0   PASS           runs/axi_smoke.log     2 lines
axi_reset   rc=1   FAIL           runs/axi_reset.log     2 lines
axi_burst   rc=2   COMPILE_ERROR  runs/axi_burst.log     3 lines
axi_wrap    rc=7   NO_LICENCE     runs/axi_wrap.log      2 lines

Four invocations, four records, no exceptions and no special cases. The log was written for every one of them, including the licence failure where there was barely anything to write — and that empty-ish log is itself evidence, because Chapter 20.11 Section 7 said a log with no test banner is NO_RUN.

The function's docstring states the contract in one line: always returns a record; never raises on tool failure. That is Chapter 20.6's promise, now covering a subprocess instead of a parser, and it is what lets a four-hundred-test sweep be a plain for loop.

Three things it does not do, each a later chapter. It captures the whole log into memory, which Chapter 23.5 fixes for a gigabyte. It runs in whatever directory the parent was in, which Chapter 23.7 fixes. And it waits forever, which Chapter 23.8 fixes.

8. Predict, Run, Observe

import signal
import subprocess
import sys

SCRIPT = "import sys; sys.exit(int(sys.argv[1]))"
KILLER = "import os, signal; os.kill(os.getpid(), signal.SIGTERM)"

# A
print("A:", subprocess.run([sys.executable, "-c", SCRIPT, "0"]).returncode)

# B
print("B:", subprocess.run([sys.executable, "-c", SCRIPT, "3"]).returncode)

# C
try:
    subprocess.run([sys.executable, "-c", SCRIPT, "3"], check=True)
except subprocess.CalledProcessError as exc:
    print("C:", type(exc).__name__, exc.returncode)

# D
subprocess.run([sys.executable, "-c", SCRIPT, "0"], check=True)
print("D: check=True on success returned normally")

# E
rc = subprocess.run([sys.executable, "-c", KILLER]).returncode
print("E:", rc, rc < 0, signal.Signals(-rc).name)

# F
print("F:", rc > 0, rc != 0)

# G
print("G:", {0: "PASS", 1: "FAIL"}.get(7, "UNKNOWN"))

The verified output:

A: 0
B: 3
C: CalledProcessError 3
D: check=True on success returned normally
E: -15 True SIGTERM
F: False True
G: UNKNOWN

A and B — without check, both return normally and the number is just data.

C — with check=True, a non-zero status raises and the exception carries the code.

D — check=True does nothing on success, which is why it reads as harmless.

E — a signal gives a negative number, and signal.Signals names it.

F — rc > 0 is False for a killed process and rc != 0 is True. A flow filtering failures with > 0 silently drops every signal.

G — .get with a default turns an unmapped code into a status instead of a KeyError.

9. Exercises

Exercise 1 — four codes. Run a stand-in tool exiting 0, 1, 2 and 7. Confirm subprocess.run returns normally for all four.

Exercise 2 — whose numbers. Look up what exit 2 means for a tool in your own flow, from its documentation. Say where that fact should live in your code.

Exercise 3 — what the exception carries. Trigger CalledProcessError with capture_output=True and print all four of its attributes.

Exercise 4 — the real cost. Write the same save-and-classify loop with and without check=True. Count the lines and the number of places the log is written.

Exercise 5 — the sweep. Let CalledProcessError escape inside a loop over four tests. Record how many tests ran.

Exercise 6 — classify. Write a classify returning a status for every code, including an unmapped one. Say why UNKNOWN beats a KeyError.

Exercise 7 — keep the number. Say what a bug report loses if the record keeps only the status and not the raw rc.

Exercise 8 — the licence row. Explain in two sentences why NO_LICENCE must not be reported as FAIL, using Chapter 20.11's vocabulary.

Exercise 9 — two numbers, one event. Kill a child with SIGTERM and record what Python reports and what a shell reports. Reconcile them.

Exercise 10 — the filter that misses. Filter a list of return codes with rc > 0 and with rc != 0, including a negative one. Say which failures the first one drops.

Exercise 11 — three signals. For SIGTERM, SIGKILL and SIGABRT, say what each usually means in a verification flow and which of Chapter 20.11's statuses it maps to.

Exercise 12 — when to check. Name one call in your flow that should use check=True and one that must not. Justify both in a sentence.

Exercise 13 — the contract. Write run_sim with a docstring promising it always returns a record. Confirm a four-test loop needs no try.

10. Summary

• A non-zero exit status is data, and subprocess.run returns normally for it — deciding what it means is your job.
• The numbers are the tool's, not Module 22's contract, so the mapping belongs in one place per tool.
• check=True raises CalledProcessError, which carries returncode, cmd, output and stderr — it does not throw the output away.
• What it throws away is the next statement. Saving the log and recording the result have to happen on both paths, so they end up written twice and drift.
• An escaping CalledProcessError stops a sweep at the first failing test, which is the one thing a regression must not do.
• Inspect the number with an if — it is a condition, not an exception, which is Chapter 14.4's distinction.
• Classify with a dict and a default, so an unmapped code becomes UNKNOWN rather than a KeyError mid-sweep.
• Keep the raw rc alongside the status. The status is for logic, the number is for the bug report.
• A negative returncode means the child was killed by a signal, and -rc is the signal number.
• The same event is 143 in a shell and -15 in Python, which is why flow documentation written from the shell's view does not match.
• rc > 0 misses every signal. The tests are rc != 0 for failure and rc < 0 for killed.
• SIGTERM is usually a scheduler timeout, SIGKILL usually out of memory, SIGABRT the tool giving up — three infrastructure problems that a naive flow reports as design bugs.
• check=True is right when a failure makes the following code meaningless and there is no result to record — a setup step, a version query, a one-off script.
• A run function that always returns a record and never raises is what lets a four-hundred-test sweep be a plain for loop.

The next chapter decides where all that output should go, which for a simulator is not into memory.