Saying PASS or FAIL So Other Tools Can Tell

Part II · Module 17 · Chapter 17.8 · Saying PASS or FAIL So Other Tools Can Tell

1. The Engineering Problem

A regression wrapper launches four hundred simulations overnight. Each one runs a check script that reads a log and decides whether the test passed. In the morning the dashboard is green.

It is green because every one of those check scripts printed its verdict and then finished normally. Eleven of them printed FAIL. The wrapper never saw a single one of them, because the wrapper does not read output — it reads a number, and every script handed it the same number, which was zero, which means success.

Nothing was broken. No exception was raised, no log was malformed, no tool misbehaved. Four hundred scripts did exactly what they were written to do, and the eleven real failures were invisible for the length of time it took somebody to open a log by hand.

2. The Line

print("FAIL")
sys.exit(1)   # this is what the shell actually sees

Two lines with two audiences. The first is for a person. The second is for everything else.

Every program that finishes hands its parent a single integer called the exit status. The convention is older than Python and is universal across Unix tooling:

• zero means the program succeeded;
• any non-zero value means it did not.

A Python script that ends without saying otherwise exits zero. That is the right default for most programs and exactly the wrong one for a script whose entire job is to report a verdict.

3. Measuring the Gap

Two scripts that differ by one line:

print("FAIL")

FAIL

And the same thing with the status set:

import sys

print("FAIL")
sys.exit(1)

FAIL

The visible output is identical. Both print FAIL and nothing else. To see the difference you have to ask the shell, which reports the status of the last command in $?:

$ python3 quiet.py
FAIL
$ echo $?
0

$ python3 loud.py
FAIL
$ echo $?
1

One reports success while printing failure. That is the entire chapter, and the consequences are worth seeing in the three places a flow actually tests it.

Chaining with &&, which runs the next step only on success:

$ python3 quiet.py && echo "shell ran the next step"
FAIL
shell ran the next step

$ python3 loud.py && echo "not reached"
FAIL

The first one published a result from a failing run to whatever came next.

Falling back with ||, which runs only on failure:

$ python3 loud.py || echo "fallback branch taken"
FAIL
fallback branch taken

And a shell if, which is what a wrapper script uses:

$ if python3 quiet.py > /dev/null; then echo "treated as success"; fi
treated as success

$ if python3 loud.py > /dev/null; then echo "not reached"; else echo "treated as failure"; fi
treated as failure

Three different mechanisms, one question underneath all of them, and the script that only printed is invisible to every one.

4. Where the Exit Belongs

sys.exit() works by raising SystemExit, which propagates like any other exception (Chapter 14.4). Two consequences follow.

It belongs at the end, in the outermost function. A helper that reads a file should return a value or raise; it should not decide to terminate the program. Burying sys.exit() three calls deep makes a function that cannot be reused, cannot be tested, and ends the process from somewhere a reader will not look.

The shape to aim for is a main() that collects a verdict and exits once:

import sys

END_MARKER = "Simulation complete"
LOG = ["UVM_INFO ok", "UVM_ERROR drv.sv(88)", "Simulation complete"]

EXIT_FOR = {"PASS": 0, "FAIL": 1, "NO_FINISH": 1, "EMPTY_LOG": 2, "NO_LOG": 2}


def classify(lines):
    """Pure: returns a verdict, never exits."""
    if not lines:
        return "EMPTY_LOG"
    errors = 0
    finished = False
    for line in lines:
        if "UVM_ERROR" in line:
            errors += 1
        if END_MARKER in line:
            finished = True
    if not finished:
        return "NO_FINISH"
    return "PASS" if errors == 0 else "FAIL"


def main(lines):
    verdict = classify(lines)
    print(verdict)
    return EXIT_FOR[verdict]


status = main(LOG)
print("would exit with:", status)
sys.exit(status)

FAIL
would exit with: 1

classify is testable because it returns rather than exits. main owns the one decision about status. The mapping lives in a dictionary where a reviewer can read the whole policy at once — which is Chapter 1.8's table, now executable.

And a bare except will swallow it. SystemExit inherits from BaseException, not Exception, so except Exception: correctly lets it through and a bare except: does not. A script with except: pass in a parsing loop can catch its own request to terminate and carry on, which is Chapter 14.9 Section 4's point arriving with a concrete victim.

5. The First Trap: sys.exit(True)

The natural-looking line is wrong, and wrong in the dangerous direction:

import sys

ok = False          # the run FAILED
print("passing ok =", ok, "to sys.exit")
sys.exit(ok)

passing ok = False to sys.exit

Asking the shell what that produced:

$ python3 boolexit.py
passing ok = False to sys.exit
$ echo $?
0

A failing run exited zero. sys.exit(False) is sys.exit(0), because False is 0 — and zero means success. The inverse is just as wrong: sys.exit(True) is sys.exit(1), so a passing run would report failure.

$ python3 -c "import sys; sys.exit(True)";  echo "True  -> $?"
True  -> 1
$ python3 -c "import sys; sys.exit(False)"; echo "False -> $?"
False -> 0

The confusion is structural rather than careless. In Python, True is truthy and good; in shell conventions, zero is good and non-zero is bad. The two scales run in opposite directions, so passing a boolean straight through inverts the meaning.

Write the mapping explicitly — sys.exit(0 if ok else 1) — or use the dictionary from Section 4. Never hand a boolean to sys.exit.

6. The Second Trap: the Status Is Eight Bits

A tempting idea is to exit with the number of errors, so the caller learns the count for free. It does not work, and the way it fails is almost designed to escape testing:

import sys

error_count = 256
print("exiting with the error count:", error_count)
sys.exit(error_count)

exiting with the error count: 256

What the shell receives:

$ python3 countexit.py
exiting with the error count: 256
$ echo $?
0

256 errors reported as success. The exit status is a single byte, so the value is taken modulo 256 and 256 becomes 0. Across the boundary:

$ for n in 1 255 256 257 512; do python3 -c "import sys; sys.exit($n)"; echo "  sys.exit($n) -> $?"; done
  sys.exit(1) -> 1
  sys.exit(255) -> 255
  sys.exit(256) -> 0
  sys.exit(257) -> 1
  sys.exit(512) -> 0

A script exiting with its error count is correct for 1 to 255 errors, reports success at exactly 256 and 512, and reports one error at 257. No test suite anybody writes happens to use 256 failing assertions, so the defect lives until a genuinely terrible night produces exactly the wrong number.

The status is a category, not a measurement. Counts belong in the printed output or a results file; the status says only which of a handful of outcomes occurred.

7. Three Codes, One Convention

Chapter 1.8 argued for distinguishing a bad result from no result. The exit status is where that distinction becomes real:

The three-way split is a convention rather than a language rule — Python only distinguishes zero from non-zero — and it is worth the team agreeing it explicitly, because a wrapper can then route a failure to the right people without knowing what any individual script does.

import sys

RESULTS = [("smoke", 0), ("cdc", 1), ("burst", 2), ("arb", 0), ("reset", 2)]

passed = failed = broken = 0
for name, status in RESULTS:
    if status == 0:
        passed += 1
    elif status == 1:
        failed += 1
    else:
        broken += 1

print(f"real results : {passed + failed} of {len(RESULTS)}  ({passed} pass, {failed} fail)")
print(f"never ran    : {broken}")
overall = 1 if failed else (2 if broken else 0)
print("wrapper exits:", overall)
sys.exit(overall)

real results : 3 of 5  (2 pass, 1 fail)
never ran    : 2
wrapper exits: 1

The wrapper aggregates five statuses into one, and its own exit status propagates the same convention upward. Module 22 Chapter 22.6 formalises this across every tool in a flow; the point here is that one line in each script is what makes the whole arrangement possible.

8. What the Status Cannot Carry, and the Third Channel

The exit status is one small integer, which is its strength and its limit. Three things it cannot do, and what to use instead.

It cannot carry detail. Section 6 showed why exiting with a count is wrong, and the general version is that the status answers which category and nothing else. A caller that needs the error count, the failing test names or the runtime must get them somewhere else — printed output it parses, or better, a results file. Trying to encode more than a category into eight bits is how teams end up with a convention where 37 means something.

It cannot be read by a human. echo $? is not something anybody does while watching a terminal. This is why the printed verdict is not redundant: the two outputs have genuinely different audiences, and a script that sets the status but prints nothing is as unhelpful in interactive use as one that prints but exits zero is in a pipeline.

It cannot explain itself. exit 2 says the question could not be answered; it does not say the log was missing. That explanation has to be printed — and where it is printed is the third channel most scripts ignore.

A finished program writes to two separate streams. Standard output is for the result: the verdict, the summary, the thing a caller might capture and parse. Standard error is for everything about how the run went: warnings, progress, the sentence explaining why the exit status is 2.

The separation matters the moment somebody redirects. A caller writing python3 check.py > results.txt gets a clean results file only if the diagnostics went to standard error; if everything was printed to standard output, the file now contains a progress message in the middle of the data. Conversely, an error explanation written to standard output disappears into a file nobody reads, exactly when it was needed.

In Python, print() writes to standard output, and sys.exit("message") — the form from Chapter 14.4 — writes its message to standard error and exits 1, which is a convenient pairing for the fatal case.

So the full shape of a script's reporting is three channels with three audiences: the status tells automation which category, standard output carries the result a caller might parse, and standard error carries the explanation a human needs when the status is not zero. Most flow scripts use one of the three, which is why their failures are either invisible or unreadable.

9. Exercises

Reasoning, with short programs and a shell.

Exercise 1 — Measure your own

Take a script your team runs and check its exit status after a failing input. If it is zero, describe what a wrapper around it currently concludes.

Exercise 2 — The three mechanisms

Run a failing script three ways: with &&, with ||, and inside a shell if. Record what each does before and after you add the exit line.

Exercise 3 — Invert a verdict

Write sys.exit(ok) where ok is a boolean, and check the status for both values. State the rule in one sentence.

Exercise 4 — Find the wrap

Exit with 255, 256 and 257 and record the status for each. Say why no realistic test would have caught this.

Exercise 5 — Move the exit

Find a script with sys.exit() inside a helper. Refactor so the helper returns and main() exits once, and say what became testable.

Exercise 6 — Swallow the exit

Put sys.exit(1) inside a try with a bare except: and observe what happens. Then change it to except Exception: and observe again.

Exercise 7 — Agree the codes

Write the three-code table for your flow and check it against one tool your team already depends on. Note whether that tool distinguishes a failure from an infrastructure problem.

10. Related Tutorials

What reads the status:

• GLS in Regression — the wrapper layer whose entire view of a run is the number this chapter is about.
• Automating GLS Triage — triage that routes by category, which only works if the categories reach it.

Why the distinction matters downstream:

• Coverage Closure and Signoff — signoff evidence that must distinguish "measured and failed" from "never measured".
• First-Mismatch Triage — what the design team does with an exit 1, and why an exit 2 should never reach them.

11. Summary

• Every program hands its parent an integer: zero means success, any non-zero means failure.
• A Python script that says nothing exits zero, so printing FAIL reports success.
• The printed output is identical either way — only $? shows the difference.
• &&, || and shell if all read the status, so a script without one is invisible to every flow mechanism.
• sys.exit() belongs in main(), once; helpers should return or raise so they stay testable.
• sys.exit() raises SystemExit, which a bare except: will swallow — the script catching its own request to stop.
• Never pass a boolean: sys.exit(False) is 0 and sys.exit(True) is 1, because the Python and shell scales run in opposite directions.
• The status is eight bits, so exiting with an error count reports success at exactly 256 and 512.
• The status is a category, not a measurement — counts belong in the output.
• The status carries a category and nothing else — detail belongs in the output or a results file.
• There are three channels, not two: the status for automation, standard output for the result, standard error for the explanation.
• The separation matters the moment somebody redirects, because diagnostics on standard output land in the data file.
• Three codes carry the distinction that matters: 0 good result, 1 bad result, 2 no result at all.

That closes the design half of this module. The next chapter is about the structural choice that decides whether any of these decisions can be tested later.