Failures Before the Tool Even Starts

Part II · Module 23 · Chapter 23.9 · Failures Before the Tool Even Starts

1. No Return Code at All

import subprocess

try:
    cp = subprocess.run(["xrun", "-64bit", "top.sv"], capture_output=True)
    print("returncode:", cp.returncode)
except FileNotFoundError as exc:
    print("raised   :", type(exc).__name__)
    print("errno    :", exc.errno)
    print("strerror :", exc.strerror)
    print("filename :", exc.filename)

raised   : FileNotFoundError
errno    : 2
strerror : No such file or directory
filename : xrun

There is no returncode, because cp was never assigned — the call raised instead of returning. Chapter 23.4's whole apparatus for reading exit status does not apply, because nothing exited.

That is the right behaviour. A return code is a tool's report about its own work; a missing tool has no work and no report. Python does not invent a number for it.

And exc.filename is xrun — the exception names the thing that was not found, which is what makes a useful message writable rather than guessable.

2. shell=True Destroys the Distinction

import subprocess

cp = subprocess.run("xrun -64bit top.sv", shell=True, capture_output=True, text=True)
print("returncode:", cp.returncode)
print("stderr    :", repr(cp.stderr.strip()))
print("exception :", "none — it returned normally")

returncode: 127
stderr    : '/bin/sh: xrun: command not found'
exception : none — it returned normally

The same command, the same missing tool, and no exception at all. The shell ran, could not find xrun, printed its own complaint and exited 127 — and subprocess.run reports that as a perfectly ordinary return code.

This matters more than it looks. Chapter 23.3 argued against shell=True on quoting and injection grounds. Here is a third argument that is arguably worse: it erases the difference between a tool that never started and a tool that ran and failed. With shell=False, "no such tool" is an exception and "the design has errors" is a return code — two different kinds of thing, impossible to confuse. With shell=True they are both integers, and telling them apart means knowing that 127 is the shell's convention for not-found.

import subprocess

OUTCOMES = {0: "the tool ran and passed", 1: "the tool ran and reported errors",
            126: "found but not executable (shell convention)",
            127: "not found (shell convention)"}

for rc in [0, 1, 126, 127]:
    print(f"  rc {rc:<4} {OUTCOMES[rc]}")
print()
print("…and 127 is a convention, not a guarantee: a tool is free to exit 127")
print("for its own reasons, at which point the two cases are identical.")

  rc 0    the tool ran and passed
  rc 1    the tool ran and reported errors
  rc 126  found but not executable (shell convention)
  rc 127  not found (shell convention)

Those are real shell conventions and worth knowing when you read somebody else's flow. They are not a reason to rely on them: 126 and 127 are the shell's choices, and a tool may exit with either for its own reasons. With a list and shell=False, there is no ambiguity to resolve.

3. The Same Exception for a Different Problem

import subprocess
import sys

for label, kwargs, cmd in (
    ("missing tool  ", {}, ["definitely_not_a_tool_xyz", "-run"]),
    ("missing cwd=  ", {"cwd": "runs/no_such_dir"}, [sys.executable, "-c", "pass"]),
):
    try:
        subprocess.run(cmd, **kwargs, capture_output=True)
    except FileNotFoundError as exc:
        print(f"{label} {type(exc).__name__}  errno={exc.errno}  filename={exc.filename!r}")

missing tool   FileNotFoundError  errno=2  filename='definitely_not_a_tool_xyz'
missing cwd=   FileNotFoundError  errno=2  filename='runs/no_such_dir'

Two completely different problems, one exception type, one errno.

A handler written as except FileNotFoundError: raise ToolError("tool not found") is wrong half the time — and wrong in the expensive direction, because it sends somebody to check their tool setup when the actual fault is a directory the script was supposed to create. Chapter 23.7's mkdir(parents=True, exist_ok=True) is the fix for the second case; being told to look at PATH guarantees nobody finds it.

exc.filename is the only thing that separates them, which makes it the one attribute a launch handler must not discard:

import subprocess
import sys


def classify_launch_failure(exc, tool, run_dir):
    """Which of the two errno-2 problems is this?"""
    if exc.filename == tool:
        return f"the tool '{tool}' is not on PATH — check the tool setup"
    if exc.filename == run_dir:
        return f"the run directory '{run_dir}' does not exist — create it before running"
    return f"something at '{exc.filename}' was not found"


for tool, run_dir, cmd, kwargs in (
    ("nope_tool_xyz", "runs/a", ["nope_tool_xyz"], {}),
    ("python3", "runs/no_such", [sys.executable, "-c", "pass"], {"cwd": "runs/no_such"}),
):
    try:
        subprocess.run(cmd, **kwargs, capture_output=True)
    except FileNotFoundError as exc:
        print(classify_launch_failure(exc, tool, run_dir))

the tool 'nope_tool_xyz' is not on PATH — check the tool setup
the run directory 'runs/no_such' does not exist — create it before running

Two messages, each naming the fix. That is the difference between a flow somebody can debug and one that generates tickets.

4. Check First with shutil.which

Catching the exception works, and checking first is better, because a pre-flight check can report every missing tool at once instead of stopping at the first:

import shutil

print("xrun      :", shutil.which("xrun"))
print("python3 found:", shutil.which("python3") is not None)

xrun      : None
python3 found: True

shutil.which searches PATH exactly as the shell would and returns the full path, or None. The real path for python3 differs on every machine, so only the boolean is printed here — but in a flow it is the None you test, not the path.

That is the canonical annotation:

if shutil.which("xrun") is None:
    raise ToolError("xrun not on PATH — check the tool setup")

And which checks more than existence:

import os
import shutil
from pathlib import Path

script = Path("wrapper.sh")
script.write_text("#!/bin/sh\necho hi\n", encoding="utf-8")

os.chmod(script, 0o644)
print("mode 644 — exists:", script.exists(), "| which:", shutil.which("./wrapper.sh"))

os.chmod(script, 0o755)
print("mode 755 — exists:", script.exists(), "| which found it:",
      shutil.which("./wrapper.sh") is not None)

mode 644 — exists: True | which: None
mode 755 — exists: True | which found it: True

The file existed in both cases. exists() says True either way; which says None until the execute bit is set.

So shutil.which answers the question you actually have — could I run this? — where os.path.exists answers one that does not decide anything. A wrapper script checked into a repository without its execute bit is a real and recurring failure, and exists() passes it straight through.

5. Not Executable, and Not a File

import os
import subprocess
from pathlib import Path

Path("wrapper.sh").write_text("#!/bin/sh\necho hi\n", encoding="utf-8")
os.chmod("wrapper.sh", 0o644)
Path("a_directory").mkdir(exist_ok=True)

for label, cmd in (("a file without +x", ["./wrapper.sh"]),
                   ("a directory      ", ["./a_directory"])):
    try:
        subprocess.run(cmd, capture_output=True)
    except OSError as exc:
        print(f"{label}  {type(exc).__name__}  errno={exc.errno}  {exc.strerror}")

a file without +x  PermissionError  errno=13  Permission denied
a directory        PermissionError  errno=13  Permission denied

Both PermissionError, errno 13 — and again two different causes behind one class. FileNotFoundError and PermissionError are both OSError subclasses, which is why catching OSError catches the whole family:

print("FileNotFoundError is an OSError:", issubclass(FileNotFoundError, OSError))
print("PermissionError   is an OSError:", issubclass(PermissionError, OSError))

FileNotFoundError is an OSError: True
PermissionError   is an OSError: True

Catching OSError is the right breadth for a launch handler: there are several ways a launch can fail at the operating-system level and you want all of them in the same place. The classification then happens inside, on errno and filename, rather than by writing one except clause per cause.

6. One Launch Handler

import shutil
import subprocess
import sys
from pathlib import Path


class ToolError(Exception):
    """The tool could not be launched. Not a tool failure — a setup failure."""


def preflight(tools):
    """Report every missing tool at once, before any work starts."""
    missing = [t for t in tools if shutil.which(t) is None]
    if missing:
        raise ToolError(
            f"not on PATH: {', '.join(missing)} — "
            f"is the tool module loaded? (checked {len(tools)} tools)")


def launch(cmd, run_dir):
    """Run a tool, converting every launch failure into a ToolError."""
    tool = cmd[0]
    try:
        return subprocess.run(cmd, cwd=run_dir, capture_output=True, text=True)
    except FileNotFoundError as exc:
        if exc.filename == str(run_dir):
            raise ToolError(f"run directory '{run_dir}' does not exist — "
                            f"create it before launching the tool") from exc
        raise ToolError(f"'{tool}' not on PATH — check the tool setup") from exc
    except PermissionError as exc:
        raise ToolError(f"'{tool}' is not executable — check its mode bits, "
                        f"and that it is a file and not a directory") from exc


Path("runs/ok").mkdir(parents=True, exist_ok=True)

for label, cmd, run_dir in (
    ("good        ", [sys.executable, "-c", "print('tool ran')"], "runs/ok"),
    ("missing tool", ["nope_tool_xyz", "-run"], "runs/ok"),
    ("missing dir ", [sys.executable, "-c", "pass"], "runs/absent"),
):
    try:
        cp = launch(cmd, run_dir)
        print(f"{label}  ok: {cp.stdout.strip()}")
    except ToolError as exc:
        print(f"{label}  ToolError: {exc}")

good          ok: tool ran
missing tool  ToolError: 'nope_tool_xyz' not on PATH — check the tool setup
missing dir   ToolError: run directory 'runs/absent' does not exist — create it before launching the tool

Four properties of that handler are worth copying, and the third is the one people skip.

Every launch failure becomes one exception type. Calling code catches ToolError and knows the meaning without knowing which errno produced it — Chapter 14.8's argument for a custom error type, which is that the type is part of the interface — and that chapter's canonical example is a ToolError for exactly this situation.

Each message names the fix, not the symptom. "Check the tool setup" and "create it before launching" are instructions; "No such file or directory" is an observation.

from exc is on every raise. This form has not appeared before, so it is worth one sentence: raise NewError(...) from original sets the new exception's cause, and the traceback then prints the original underneath a line reading "The above exception was the direct cause of the following exception". A developer reading it gets both the friendly message and the system-level FileNotFoundError that produced it. Without from exc the wrapper replaces the cause, and dropping it is the most common way a helpful error message becomes an unhelpful one.

preflight and launch both exist, and they are not redundant. preflight gives the good message before any work; launch catches what preflight cannot know — a tool that vanished between the check and the call, a run directory deleted by another job, a mode bit changed mid-sweep.

7. A Launch Failure Is Not a Test Result

RESULTS = [
    {"test": "axi_smoke", "kind": "ran", "rc": 0},
    {"test": "axi_reset", "kind": "ran", "rc": 1},
    {"test": "axi_burst", "kind": "launch_failed",
     "error": "'xrun' not on PATH — check the tool setup"},
    {"test": "axi_soak", "kind": "timed_out", "limit_s": 3600},
]


def status_of(record):
    if record["kind"] == "launch_failed":
        return "ERROR"
    if record["kind"] == "timed_out":
        return "NO_FINISH"
    return "PASS" if record["rc"] == 0 else "FAIL"


counts = {}
for record in RESULTS:
    status = status_of(record)
    counts[status] = counts.get(status, 0) + 1
    print(f"{record['test']:<11} {status}")

print()
print("summary   :", dict(sorted(counts.items())))
print("tests run :", sum(1 for r in RESULTS if r["kind"] != "launch_failed"))
print("conclusion:", "the regression is not a verdict on the design" if counts.get("ERROR") else "all tests ran")

axi_smoke   PASS
axi_reset   FAIL
axi_burst   ERROR
axi_soak    NO_FINISH

summary   : {'ERROR': 1, 'FAIL': 1, 'NO_FINISH': 1, 'PASS': 1}
tests run : 3
conclusion: the regression is not a verdict on the design

tests run is 3, not 4 — and that is the point of the whole section. A launch failure did not test anything, so counting it as a test that happened inflates the denominator of every pass rate in the report.

ERROR is a fifth status alongside Chapter 20.11's four, and it belongs to the infrastructure, not the design. A nightly with one ERROR and a hundred passes has not verified a hundred-test design; it has verified ninety-nine and one thing went wrong in the harness. Reporting it as FAIL sends a designer looking for a bug that does not exist; reporting it as PASS is worse.

8. Predict, Run, Observe

import os
import shutil
import subprocess
import sys
from pathlib import Path

Path("w.sh").write_text("#!/bin/sh\necho hi\n", encoding="utf-8")
os.chmod("w.sh", 0o644)

# A
try:
    subprocess.run(["nope_xyz"], capture_output=True)
except OSError as exc:
    print("A:", type(exc).__name__, exc.errno, repr(exc.filename))

# B
cp = subprocess.run("nope_xyz", shell=True, capture_output=True)
print("B:", cp.returncode)

# C
print("C:", shutil.which("nope_xyz"))

# D
print("D:", Path("w.sh").exists(), shutil.which("./w.sh"))

# E
try:
    subprocess.run(["./w.sh"], capture_output=True)
except OSError as exc:
    print("E:", type(exc).__name__, exc.errno)

# F
try:
    subprocess.run([sys.executable, "-c", "pass"], cwd="gone")
except OSError as exc:
    print("F:", type(exc).__name__, exc.errno, repr(exc.filename))

# G
print("G:", issubclass(FileNotFoundError, OSError), issubclass(PermissionError, OSError))

The verified output:

A: FileNotFoundError 2 'nope_xyz'
B: 127
C: None
D: True None
E: PermissionError 13
F: FileNotFoundError 2 'gone'
G: True True

A — a missing executable raises, and filename names it.

B — the same thing under shell=True is return code 127, with no exception.

C — which reports None rather than raising, which is what makes it usable in a pre-flight check.

D — the file exists and which still says None: no execute bit. Section 4.

E — launching it raises PermissionError, not FileNotFoundError.

F — a missing cwd= raises the same FileNotFoundError and errno as A. Only filename differs. Section 3.

G — both are OSError, which is the clause a launch handler should catch.

9. Exercises

Exercise 1 — no code. Run a command that does not exist and show there is no returncode to read. Say why that is correct.

Exercise 2 — the name. Print exc.filename and say what it lets you write that strerror does not.

Exercise 3 — the shell erases it. Run the same missing command with shell=True. Record the return code and say what information was lost.

Exercise 4 — 126 and 127. Say why relying on those two numbers is weaker than catching an exception.

Exercise 5 — the ambiguity. Produce a missing-tool failure and a missing-cwd= failure and show the type and errno are identical.

Exercise 6 — tell them apart. Write a classifier using exc.filename that gives each its own message naming its own fix.

Exercise 7 — the wrong message. Describe what a person does after being told "tool not found" when the real cause was a missing run directory.

Exercise 8 — pre-flight. Write a preflight that reports all missing tools in one message. Say why that beats failing on the first.

Exercise 9 — exists is not enough. Create a file without its execute bit and compare exists() with shutil.which. Say which question each answers.

Exercise 10 — the repository case. Explain why a wrapper script committed without +x is a realistic version of Exercise 9.

Exercise 11 — permission, twice. Produce PermissionError from a non-executable file and from a directory. Say why one except clause handles both.

Exercise 12 — one type out. Write a ToolError wrapper over a real call, with from exc on every raise. Say what the chain gives a developer reading the traceback.

Exercise 13 — both layers. Say what launch catches that preflight cannot, with two concrete examples.

Exercise 14 — the denominator. Take a 100-test sweep with 4 launch failures and compute the pass rate with and without them in the denominator. Say which is honest.

Exercise 15 — forty failures. Describe how you would tell a bad commit from a launch failure given only a summary table, and name the tell.

10. Summary

• A missing executable raises FileNotFoundError; there is no return code, because no process ever existed.
• exc.filename names what was not found, which is what makes a message that identifies the fix possible.
• shell=True turns a launch failure into return code 127 with no exception — a third argument against it, beyond Chapter 23.3's quoting and injection.
• 126 and 127 are shell conventions, not guarantees; a tool may exit with either for its own reasons.
• A missing cwd= directory raises the same exception and errno as a missing tool. Only filename tells them apart.
• A handler that reports "tool not found" for both sends people to PATH when the fix is Chapter 23.7's mkdir.
• shutil.which searches PATH as the shell would and returns None rather than raising, so it works as a pre-flight check.
• which also checks the execute bit — a file can exists() as True and still be unrunnable, which is the committed-without-+x wrapper script.
• A non-executable file and a directory both raise PermissionError, errno 13.
• Both classes are OSError subclasses, so catch OSError once and classify inside on errno and filename.
• Convert every launch failure into one custom exception type, with a message that names the fix and from exc on every raise.
• Keep both preflight and launch — one gives the good message before any work, the other catches what changed in between.
• A launch failure is ERROR, not FAIL — it belongs to the infrastructure, and it did not test anything.
• It must not count in the denominator of a pass rate, or every number in the report is wrong.
• Forty launch failures look exactly like a bad commit, and the tell is that none of them produced a log.

The next chapter makes a run repeatable by recording the command that produced it.