Python Part II · Module 33 · Testing and Debugging Automation
Partial Failure, Cleanup and Leaving Evidence Behind
A run has three endings, not two. It finishes, it fails, or something outside it decides it is over — and the third is the one that leaves a scratch directory the size of a disk and a results table with a hole in it. The canonical fix is a `finally` block that deletes the scratch and keeps the evidence, which is the right instinct and rests on an assumption worth measuring: that `finally` runs. On four of the seven ways a run can end it does, and the three where it does not include the one a job scheduler uses.
Advanced23 min readPythonPart IIAutomationCleanupSignals
Part II · Module 33 · Chapter 33.13 · Partial Failure, Cleanup and Leaving Evidence Behind
1. Three Endings
ENDINGS = [
("finished", "results written, scratch removable",
"the result", "no"),
("failed", "results missing, logs valuable",
"why it failed", "no"),
("killed from outside", "mid-write, scratch enormous",
"that it was killed", "yes"),
]
print(f"{'ending':<22} {'what the run dir holds':<36} {'what you need':<22} orphans scratch")
for end, holds, need, orphan in ENDINGS:
print(f"{end:<22} {holds:<36} {need:<22} {orphan}")
print()
print("endings a flow usually handles:", 2)
print("endings that actually happen :", 3)
print("the third is sent by :", "a scheduler, a timeout, or a person")ending what the run dir holds what you need orphans scratch
finished results written, scratch removable the result no
failed results missing, logs valuable why it failed no
killed from outside mid-write, scratch enormous that it was killed yes
endings a flow usually handles: 2
endings that actually happen : 3
the third is sent by : a scheduler, a timeout, or a personThe three endings want two different things from cleanup, and they conflict. A run that finished wants its scratch gone, because a 4 GiB intermediate directory per run fills a volume in an afternoon. A run that failed wants its scratch kept, because the thing you need is in there.
The canonical annotation resolves that conflict by classifying the files rather than the outcome, which is Section 7. First, the mechanism.
2. The Annotation
finally:
remove_scratch(run_dir) # keep logs, waves and cmd.shimport shutil
from pathlib import Path
SCRATCH = {"obj", "work", "csrc", "simv.daidir", "xcelium.d"}
KEEP_SUFFIX = {".log", ".vcd", ".fsdb", ".sh", ".json", ".csv"}
def build_run_dir(d):
"""A run directory as a flow actually leaves it."""
p = Path(d)
for sub in ("obj", "work", "csrc"):
(p / sub).mkdir(parents=True, exist_ok=True)
(p / sub / "big.tmp").write_text("x" * 64, encoding="utf-8")
p.mkdir(parents=True, exist_ok=True)
for name in ("sim.log", "compile.log", "waves.vcd", "cmd.sh",
"results.json", "scratch.tmp"):
(p / name).write_text("data\n", encoding="utf-8")
return p
def remove_scratch(run_dir):
"""Delete the regenerable parts; keep anything that is evidence."""
p = Path(run_dir)
removed = []
for child in sorted(p.iterdir()):
if child.is_dir() and child.name in SCRATCH:
shutil.rmtree(child)
removed.append(child.name + "/")
elif child.is_file() and child.suffix not in KEEP_SUFFIX:
child.unlink()
removed.append(child.name)
return removed
run_dir = build_run_dir("run_0001")
before = sorted(c.name + ("/" if c.is_dir() else "") for c in run_dir.iterdir())
print("before:", before)
try:
raise RuntimeError("the simulator exited 1")
except RuntimeError as exc:
print("failure:", exc)
finally:
removed = remove_scratch(run_dir)
after = sorted(c.name + ("/" if c.is_dir() else "") for c in run_dir.iterdir())
print("removed:", removed)
print("kept :", after)
print()
print("entries before:", len(before), "| removed:", len(removed),
"| kept:", len(after))before: ['cmd.sh', 'compile.log', 'csrc/', 'obj/', 'results.json', 'scratch.tmp', 'sim.log', 'waves.vcd', 'work/']
failure: the simulator exited 1
removed: ['csrc/', 'obj/', 'scratch.tmp', 'work/']
kept : ['cmd.sh', 'compile.log', 'results.json', 'sim.log', 'waves.vcd']
entries before: 9 | removed: 4 | kept: 5Four entries removed, five kept, and the run failed. That is the behaviour the annotation describes: the cleanup is unconditional — it is in a finally — but what it removes is decided by classification, not by whether the run succeeded.
The comment is doing real work. # keep logs, waves and cmd.sh is the policy, and without it a reader would reasonably assume remove_scratch removes the run directory.
Now: does the finally run?
3. When finally Actually Runs
import subprocess
import sys
from pathlib import Path
Path("t.py").write_text(
'import atexit\n'
'import os\n'
'import signal\n'
'import sys\n'
'\n'
'mode = sys.argv[1]\n'
'atexit.register(lambda: print("ATEXIT", flush=True))\n'
'\n'
'try:\n'
' print("STARTED", flush=True)\n'
' if mode == "raise": raise RuntimeError("tool crashed")\n'
' if mode == "exit": sys.exit(7)\n'
' if mode == "osexit": os._exit(9)\n'
' if mode == "sigint": os.kill(os.getpid(), signal.SIGINT)\n'
' if mode == "sigterm": os.kill(os.getpid(), signal.SIGTERM)\n'
' if mode == "sigkill": os.kill(os.getpid(), signal.SIGKILL)\n'
'finally:\n'
' print("FINALLY", flush=True)\n', encoding="utf-8")
MODES = ["ok", "raise", "exit", "osexit", "sigint", "sigterm", "sigkill"]
print(f"{'how the run ended':<20} {'rc':>4} {'finally':<8} atexit")
for mode in MODES:
r = subprocess.run([sys.executable, "t.py", mode],
stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
text=True)
out = r.stdout
print(f"{mode:<20} {r.returncode:>4} "
f"{str('FINALLY' in out):<8} {'ATEXIT' in out}")how the run ended rc finally atexit
ok 0 True True
raise 1 True True
exit 7 True True
osexit 9 False False
sigint -2 True True
sigterm -15 False False
sigkill -9 False Falsefinally runs on four of these seven endings and not on three. Read the rows that matter.
sys.exit(7) runs the cleanup. sys.exit raises SystemExit, which is an exception, so every finally on the way out executes. Chapter 33.11's sys.exit(f"...") therefore cleans up properly, which is not obvious from the name.
os._exit(9) does not. It is the raw syscall: no unwinding, no finally, no atexit, and no flush of buffered output. It exists for the child of a fork and almost nothing else.
SIGINT runs the cleanup and SIGTERM does not. That is the asymmetry worth knowing, and it is not a quirk — Python installs a handler for SIGINT that raises KeyboardInterrupt, so the interpreter unwinds normally. SIGTERM has no Python-level handler, so the default disposition applies and the process is terminated by the operating system with the interpreter never regaining control.
Note also that subprocess reports a signal death as a negative number — -2 for SIGINT, -15 for SIGTERM, -9 for SIGKILL — where a shell's $? shows 130, 143 and 137 for the same three events. Chapter 32.7 Section 5 covered that 128-plus-signal convention; returncode has a sign bit available where a shell's single byte does not, so the two spellings carry identical information in different notation. A sweep that records returncode directly and a wrapper that records $? will therefore disagree on every signal death unless one of them normalises, which is a real reconciliation problem rather than a curiosity — and the fix is to store the signal number, not either encoding of it.
4. Making SIGTERM Behave Like SIGINT
import subprocess
import sys
from pathlib import Path
HANDLER = (
'import signal\n'
'\n'
'def _on_term(signum, frame):\n'
' raise KeyboardInterrupt(f"terminated by signal {signum}")\n'
'\n'
'signal.signal(signal.SIGTERM, _on_term)\n'
)
BODY = (
'import os\n'
'import signal\n'
'from pathlib import Path\n'
'\n'
'try:\n'
' Path("scratch.tmp").write_text("x", encoding="utf-8")\n'
' os.kill(os.getpid(), signal.SIGTERM)\n'
'except KeyboardInterrupt:\n'
' print("INTERRUPTED", flush=True)\n'
'%s'
'finally:\n'
' p = Path("scratch.tmp")\n'
' if p.exists():\n'
' p.unlink()\n'
' print("CLEANED", flush=True)\n'
)
RERAISE = ' sys.exit(143)\n'
VARIANTS = [
("no handler", "", BODY % ""),
("handler, caught", HANDLER, BODY % ""),
("handler + exit 143", HANDLER, "import sys\n" + (BODY % RERAISE)),
]
print(f"{'variant':<22} {'rc':>5} {'cleaned':<9} {'scratch left':<14} rc says")
for label, prelude, body in VARIANTS:
Path("s.py").write_text(prelude + body, encoding="utf-8")
r = subprocess.run([sys.executable, "s.py"], stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL, text=True)
left = Path("scratch.tmp").exists()
says = {0: "SUCCESS", 143: "killed by signal 15"}.get(r.returncode,
"killed, uncleaned")
print(f"{label:<22} {r.returncode:>5} "
f"{str('CLEANED' in r.stdout):<9} {str(left):<14} {says}")
if left:
Path("scratch.tmp").unlink()variant rc cleaned scratch left rc says
no handler -15 False True killed, uncleaned
handler, caught 0 True False SUCCESS
handler + exit 143 143 True False killed by signal 15Three variants, and the middle one is the trap. Installing a SIGTERM handler that raises does exactly what Section 3 predicted — the interpreter unwinds, every finally runs, and the scratch is removed. And then the process exits 0, because the exception was caught and handled and nothing afterwards failed.
So the obvious fix trades one bug for a worse one. Variant one orphans its scratch and at least reports that it was killed; variant two cleans up perfectly and tells the sweep the run succeeded. A results table built from exit codes would record a PASS for a run the scheduler terminated at hour six — which is the confident wrong value Chapter 30.7 Section 3 called a report's worst defect.
The third variant is the one to copy. Catch the interrupt, let the finally blocks run, and then exit deliberately with a code that says what happened — 143 being the shell's spelling of "terminated by signal 15", so a wrapper reading $? sees the same value it would have seen without the handler. Cleanup and an honest status.
Two things the handler does not fix, and both should be said plainly. SIGKILL cannot be handled — that is its definition — so a run killed with -9 always orphans its scratch, and the only defence is a sweep-level sweeper that removes stale run directories on the next start. And a handler that raises mid-write still leaves a partially written file, which is why Chapter 30.7's write-to-temp-then-rename pattern matters more than any cleanup: a file that is either absent or complete needs no cleanup decision at all.
5. A return in finally Swallows the Exception
def cleanup_with_return():
try:
raise RuntimeError("the simulator exited 1")
finally:
return "scratch removed" # <- swallows it
def cleanup_without_return():
try:
raise RuntimeError("the simulator exited 1")
finally:
print(" scratch removed")
print("cleanup_with_return() returns:", repr(cleanup_with_return()))
caught = None
try:
cleanup_with_return()
except Exception as exc:
caught = type(exc).__name__
print("exception seen by the caller :", caught)
print()
print("cleanup_without_return():")
try:
cleanup_without_return()
except RuntimeError as exc:
print(" propagated:", type(exc).__name__, "-", exc)
print()
print("the run failed :", True)
print("the first version reported:", repr("success"))
print("so the sweep records :", "a PASS for a run that crashed")cleanup_with_return() returns: 'scratch removed'
exception seen by the caller : None
cleanup_without_return():
scratch removed
propagated: RuntimeError - the simulator exited 1
the run failed : True
the first version reported: 'success'
so the sweep records : a PASS for a run that crashedThe exception is gone. A return inside finally discards whatever was propagating — the caller receives the cleanup's return value and no error at all, and a run that crashed is recorded as having returned normally.
This is a silent, total loss of a failure, and it reaches the results table. Chapter 30.7 Section 3 argued that a report's worst defect is a wrong value presented confidently; this is how one gets there. The same applies to break and continue inside a finally in a loop.
The rule is simple and absolute: a finally block does cleanup and nothing else. No return, no break, no continue. If the cleanup has something to say, log it.
6. An Exception in finally Replaces the Original
import shutil
def run_with_fragile_cleanup():
try:
raise RuntimeError("the simulator exited 1 — THIS is the real failure")
finally:
shutil.rmtree("scratch_that_was_never_created")
try:
run_with_fragile_cleanup()
except Exception as exc:
print("the caller sees :", type(exc).__name__)
print(" message :", str(exc)[:52])
print("the real failure is:", "demoted to exc.__context__")
print(" type :", type(exc.__context__).__name__)
print(" message :", exc.__context__)
print()
print("what a log shows by default:", "the cleanup error")
print("what the engineer needs :", "the simulator error")the caller sees : FileNotFoundError
message : [Errno 2] No such file or directory: 'scratch_that_w
the real failure is: demoted to exc.__context__
type : RuntimeError
message : the simulator exited 1 — THIS is the real failure
what a log shows by default: the cleanup error
what the engineer needs : the simulator errorThe cleanup's failure replaced the crash it was cleaning up after. The FileNotFoundError is what propagates, what gets logged, and what ends up in the results row; the actual simulator failure survives only as __context__, which nothing prints unless you ask.
That is a worse outcome than Section 5 in one respect: at least a swallowed exception is silent. This one is loud about the wrong thing, and it sends whoever reads the report to investigate the cleanup code.
So the cleanup itself has to be unfailing:
import shutil
from pathlib import Path
def remove_scratch_safely(run_dir, scratch_names, log):
"""Cleanup must never be the reason a run's error is lost."""
p = Path(run_dir)
removed, problems = [], []
for name in scratch_names:
child = p / name
try:
if child.is_dir():
shutil.rmtree(child)
removed.append(name + "/")
elif child.exists():
child.unlink()
removed.append(name)
except OSError as exc:
problems.append(f"{name}: {type(exc).__name__}")
if problems:
log.append("cleanup could not remove: " + "; ".join(problems))
return removed
run = Path("run_0002")
(run / "obj").mkdir(parents=True, exist_ok=True)
(run / "obj" / "a.o").write_text("x", encoding="utf-8")
(run / "sim.log").write_text("UVM_ERROR\n", encoding="utf-8")
log = []
caught = None
try:
try:
raise RuntimeError("the simulator exited 1")
finally:
removed = remove_scratch_safely(
run, ["obj", "work", "nosuch", "sim.log.tmp"], log)
except RuntimeError as exc:
caught = str(exc)
print("removed :", removed)
print("cleanup problems :", log)
print("exception that ")
print(" reached the caller:", caught)
print()
print("the real failure survived:", caught is not None)
print("the cleanup's trouble is :", "logged, not raised")
print("sim.log still present :", (run / "sim.log").is_file())removed : ['obj/']
cleanup problems : []
exception that
reached the caller: the simulator exited 1
the real failure survived: True
the cleanup's trouble is : logged, not raised
sim.log still present : TrueThe real failure reached the caller. Three absent entries — work, nosuch, sim.log.tmp — were simply skipped rather than raising, so there was nothing for the cleanup to report; and had a removal genuinely failed, it would have been appended to a log rather than thrown.
The pattern generalises past cleanup: code that runs during failure handling must not be able to fail. Chapter 33.10 Section 8 found the logging module making the same decision for the same reason, and swallowing errors there has the same cost — it has to be paid somewhere, and a line in the log is the cheapest place.
7. Evidence or Scratch?
ARTEFACTS = [
("sim.log", "evidence", "no", "the only record of what happened"),
("compile.log", "evidence", "no", "a compile error lives only here"),
("cmd.sh", "evidence", "no", "regenerating it is the hard part — §8"),
("results.json", "evidence", "no", "the run's answer"),
("seed.txt", "evidence", "no", "without it nothing reproduces"),
("waves.vcd", "evidence*", "yes, slowly", "huge; keep on failure only"),
("obj/", "scratch", "yes", "a recompile regenerates it"),
("work/", "scratch", "yes", "the simulator rebuilds it"),
("csrc/", "scratch", "yes", "generated C"),
("*.tmp", "scratch", "yes", "by definition"),
]
print(f"{'artefact':<16} {'class':<11} {'regenerable':<13} why")
for name, cls, regen, why in ARTEFACTS:
print(f"{name:<16} {cls:<11} {regen:<13} {why}")
ev = sum(1 for _, c, _, _ in ARTEFACTS if c.startswith("evidence"))
print()
print("evidence:", ev, "| scratch:", len(ARTEFACTS) - ev)
print()
print("the test:", "could this be regenerated from what remains?")
print(" no -> evidence, keep it")
print(" yes -> scratch, delete it")
print("waves are the awkward case:", "regenerable, but only by a rerun")artefact class regenerable why
sim.log evidence no the only record of what happened
compile.log evidence no a compile error lives only here
cmd.sh evidence no regenerating it is the hard part — §8
results.json evidence no the run's answer
seed.txt evidence no without it nothing reproduces
waves.vcd evidence* yes, slowly huge; keep on failure only
obj/ scratch yes a recompile regenerates it
work/ scratch yes the simulator rebuilds it
csrc/ scratch yes generated C
*.tmp scratch yes by definition
evidence: 6 | scratch: 4
the test: could this be regenerated from what remains?
no -> evidence, keep it
yes -> scratch, delete it
waves are the awkward case: regenerable, but only by a rerunSix evidence, four scratch, and one row with an asterisk. Waveforms are technically regenerable — rerun with the same seed and you get them back — so by the test they are scratch, and by volume they must be, because keeping a gigabyte of VCD for every passing run is how a volume fills.
But "rerun with the same seed" assumes the failure is deterministic, and the failures that most need a waveform are exactly the ones that are not. Chapter 33.14 is about those. So waves get the one outcome-dependent rule in the policy: delete on success, keep on failure.
import shutil
from pathlib import Path
SCRATCH_DIRS = {"obj", "work", "csrc"}
EVIDENCE_SUFFIX = {".log", ".sh", ".json", ".txt"}
BULKY_SUFFIX = {".vcd", ".fsdb", ".shm"}
def cleanup(run_dir, failed):
"""Delete scratch always; keep bulky evidence only when it failed."""
p = Path(run_dir)
removed, kept = [], []
for child in sorted(p.iterdir()):
if child.is_dir():
if child.name in SCRATCH_DIRS:
shutil.rmtree(child)
removed.append(child.name + "/")
else:
kept.append(child.name + "/")
elif child.suffix in EVIDENCE_SUFFIX:
kept.append(child.name)
elif child.suffix in BULKY_SUFFIX:
if failed:
kept.append(child.name)
else:
child.unlink()
removed.append(child.name)
else:
child.unlink()
removed.append(child.name)
return removed, kept
def build(d):
p = Path(d)
(p / "obj").mkdir(parents=True, exist_ok=True)
(p / "obj" / "a.o").write_text("x", encoding="utf-8")
for n in ("sim.log", "cmd.sh", "results.json", "seed.txt",
"waves.vcd", "core.tmp"):
(p / n).write_text("x", encoding="utf-8")
return p
for failed in (False, True):
d = build(f"run_{'fail' if failed else 'pass'}")
removed, kept = cleanup(d, failed)
print(f"--- run {'FAILED' if failed else 'passed'}")
print(" removed:", removed)
print(" kept :", kept)
print()
print("the only outcome-dependent rule:", "bulky waveforms")
print("everything else :", "classified by what it is")--- run passed
removed: ['core.tmp', 'obj/', 'waves.vcd']
kept : ['cmd.sh', 'results.json', 'seed.txt', 'sim.log']
--- run FAILED
removed: ['core.tmp', 'obj/']
kept : ['cmd.sh', 'results.json', 'seed.txt', 'sim.log', 'waves.vcd']
the only outcome-dependent rule: bulky waveforms
everything else : classified by what it isThe passing run deleted its waveform; the failing run kept it. Everything else is classified by what the file is rather than by how the run ended, which keeps the policy readable: one rule depends on the outcome and nine do not.
8. cmd.sh Is the One That Matters Most
import shlex
import sys
from pathlib import Path
def write_cmd_sh(run_dir, argv, env_keys, cwd):
"""Everything needed to reproduce this run by hand."""
p = Path(run_dir) / "cmd.sh"
lines = ["#!/bin/sh",
"# written by the flow — reproduces this run exactly",
f"cd {shlex.quote(str(cwd))}"]
for k, v in env_keys:
lines.append(f"export {k}={shlex.quote(v)}")
lines.append(" ".join(shlex.quote(a) for a in argv))
p.write_text("\n".join(lines) + "\n", encoding="utf-8")
p.chmod(0o755)
return p
Path("run_0003").mkdir(parents=True, exist_ok=True)
cmd = write_cmd_sh(
"run_0003",
["vvp", "sim.vvp", "+seed=4121", "+UVM_TESTNAME=axi_burst_test",
"+dump=waves.vcd"],
[("UVM_HOME", "/opt/uvm-1.2"), ("SIM_ARGS", "-q --timeout 3600")],
Path("run_0003").name)
print(cmd.read_text(encoding="utf-8"), end="")
print()
print("executable :", oct(cmd.stat().st_mode & 0o777))
print("quoting handled by :", "shlex.quote — 23.1 §6")
print("why a FILE, not a ")
print(" log line :", "it can be run, not just read")#!/bin/sh
# written by the flow — reproduces this run exactly
cd run_0003
export UVM_HOME=/opt/uvm-1.2
export SIM_ARGS='-q --timeout 3600'
vvp sim.vvp +seed=4121 +UVM_TESTNAME=axi_burst_test +dump=waves.vcd
executable : 0o755
quoting handled by : shlex.quote — 23.1 §6
why a FILE, not a
log line : it can be run, not just readThat file is the difference between a ten-minute debug and a two-hour one. It records the working directory, the environment variables the tool reads, and the exact argument vector — and because shlex.quote did the quoting, SIM_ARGS with its embedded space survives as one value.
Three properties make it worth the eight lines. It is executable, so the next engineer runs it rather than reconstructing it. It is inside the run directory, so it cannot be separated from the log it explains. And it is written before the tool is launched, not after, so it exists even for the run that was killed at hour six.
The annotation names it explicitly — # keep logs, waves and cmd.sh — and of those three, this is the one that is irreplaceable. A log can sometimes be regenerated by rerunning; the command line that produced it cannot be, because it is the input you have lost.
9. The Run, Assembled
import shlex
import shutil
import signal
import sys
from pathlib import Path
SCRATCH_DIRS = {"obj", "work", "csrc"}
EVIDENCE_SUFFIX = {".log", ".sh", ".json", ".txt"}
BULKY_SUFFIX = {".vcd", ".fsdb"}
class Terminated(Exception):
"""SIGTERM, converted so that every finally block runs — §4."""
def install_term_handler(log):
def _on_term(signum, frame):
log.append(f"received signal {signum}; unwinding")
raise Terminated(f"signal {signum}")
try:
signal.signal(signal.SIGTERM, _on_term)
return True
except ValueError:
return False # not on the main thread
def remove_scratch(run_dir, failed, log):
"""Unfailing. Deletes scratch; keeps evidence; waves only on failure."""
p = Path(run_dir)
removed = []
for child in sorted(p.iterdir()):
try:
if child.is_dir():
if child.name in SCRATCH_DIRS:
shutil.rmtree(child)
removed.append(child.name + "/")
elif child.suffix in EVIDENCE_SUFFIX:
pass
elif child.suffix in BULKY_SUFFIX:
if not failed:
child.unlink()
removed.append(child.name)
else:
child.unlink()
removed.append(child.name)
except OSError as exc:
log.append(f"cleanup: {child.name}: {type(exc).__name__}")
return removed
def one_run(run_dir, argv, should_fail):
p = Path(run_dir)
(p / "obj").mkdir(parents=True, exist_ok=True)
(p / "obj" / "a.o").write_text("x", encoding="utf-8")
log = []
install_term_handler(log)
# cmd.sh FIRST — before anything can go wrong
(p / "cmd.sh").write_text(
"#!/bin/sh\n" + " ".join(shlex.quote(a) for a in argv) + "\n",
encoding="utf-8")
failed, reason = False, ""
try:
(p / "sim.log").write_text("UVM_INFO started\n", encoding="utf-8")
(p / "waves.vcd").write_text("$date\n", encoding="utf-8")
(p / "core.tmp").write_text("x", encoding="utf-8")
if should_fail:
raise RuntimeError("the simulator exited 1")
(p / "results.json").write_text('{"status": "PASS"}\n',
encoding="utf-8")
except (RuntimeError, Terminated) as exc:
failed, reason = True, f"{type(exc).__name__}: {exc}"
finally:
removed = remove_scratch(p, failed, log)
kept = sorted(c.name + ("/" if c.is_dir() else "") for c in p.iterdir())
return {"failed": failed, "reason": reason, "removed": removed,
"kept": kept, "log": log}
for should_fail in (False, True):
r = one_run(f"run_{'b' if should_fail else 'a'}",
["vvp", "sim.vvp", "+seed=4121"], should_fail)
print(f"--- {'FAILED' if r['failed'] else 'passed'}")
print(" reason :", r["reason"] or "—")
print(" removed:", r["removed"])
print(" kept :", r["kept"])
print(" log :", r["log"] or "—")
print()
print("cmd.sh kept in both cases :", True)
print("waves kept only on failure :", True)
print("cleanup ran in both cases :", True)
print("the exception survived :", "as `reason`, recorded not raised")--- passed
reason : —
removed: ['core.tmp', 'obj/', 'waves.vcd']
kept : ['cmd.sh', 'results.json', 'sim.log']
log : —
--- FAILED
reason : RuntimeError: the simulator exited 1
removed: ['core.tmp', 'obj/']
kept : ['cmd.sh', 'sim.log', 'waves.vcd']
log : —
cmd.sh kept in both cases : True
waves kept only on failure : True
cleanup ran in both cases : True
the exception survived : as `reason`, recorded not raisedTwo runs, two correct outcomes, and the failing one kept its waveform. The failing run has no results.json — it never got that far — which is the hole in the results table that Chapter 30.3 Section 5 said must be reported as missing rather than inferred.
Note the shape of the error handling. The exception is caught and recorded as reason, not allowed to propagate out of one_run, because a sweep must survive one bad run — and the exit-code decision is made once at the top from the collected results, which is Chapter 33.8's contract and Chapter 33.11 Section 7's collect-then-decide applied to runs instead of arguments.
10. Predict, Run, Observe
import shutil
import subprocess
import sys
from pathlib import Path
Path("f.py").write_text(
'import os\n'
'import signal\n'
'import sys\n'
'mode = sys.argv[1]\n'
'try:\n'
' if mode == "exit": sys.exit(4)\n'
' if mode == "osexit": os._exit(4)\n'
' if mode == "sigterm": os.kill(os.getpid(), signal.SIGTERM)\n'
'finally:\n'
' print("CLEANED", flush=True)\n', encoding="utf-8")
def probe(mode):
r = subprocess.run([sys.executable, "f.py", mode], stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL, text=True)
return r.returncode, "CLEANED" in r.stdout
# A
print("A:", probe("exit"))
# B
print("B:", probe("osexit"))
# C
print("C:", probe("sigterm"))
# D
def d():
try:
raise RuntimeError("x")
finally:
return "cleaned"
print("D:", d())The verified output:
A: (4, True)
B: (4, False)
C: (-15, False)
D: cleanedA — sys.exit(4) raises SystemExit, so the cleanup runs and the code is still 4.
B — os._exit(4) produces the same exit code with no cleanup. The two are indistinguishable from outside and completely different inside.
C — SIGTERM: no cleanup, and returncode is negative.
D — the return in finally, and the RuntimeError is simply gone. No output about it, no traceback, and a caller that believes the run succeeded.
A and B are the pair worth remembering: the exit code tells you nothing about whether cleanup happened.
11. Exercises
Exercise 1 — three endings. For your own flow, write down what the run directory holds after each.
Exercise 2 — the conflict. Explain why a finished run and a failed run want opposite things from cleanup.
Exercise 3 — the annotation's comment. Say what a reader would assume remove_scratch(run_dir) does without it.
Exercise 4 — eight endings. Reproduce Section 3's table on your own machine.
Exercise 5 — sys.exit cleans up. Confirm it, and say why that is not obvious from the name.
Exercise 6 — os._exit. Explain the one situation it is for.
Exercise 7 — the asymmetry. Say why SIGINT runs finally and SIGTERM does not.
Exercise 8 — what you test by hand. Explain why interactive testing gives the wrong answer about cleanup.
Exercise 9 — negative return codes. Reconcile -15 from subprocess with 143 from a shell.
Exercise 10 — the handler. Install a SIGTERM handler that raises and confirm cleanup runs.
Exercise 11 — the cost of 130. Say what distinction you lose, and log the signal to recover it.
Exercise 12 — what it cannot fix. Name two things the handler does not help with.
Exercise 13 — return in finally. Reproduce the swallowed exception.
Exercise 14 — the results row. Say what the sweep records for that run.
Exercise 15 — the absolute rule. State it, and check your own finally blocks against it.
Exercise 16 — a raising cleanup. Reproduce the replaced exception and find the original in __context__.
Exercise 17 — worse than silent. Explain why this is worse than Exercise 13 in one respect and better in another.
Exercise 18 — unfailing cleanup. Rewrite one of your cleanups so it cannot raise.
Exercise 19 — log, do not raise. Say where the cleanup's own problems should go.
Exercise 20 — classify your artefacts. Apply the regenerable test to everything your flow writes.
Exercise 21 — the asterisk. Decide your own policy for waveforms and justify it by volume.
Exercise 22 — one outcome-dependent rule. Explain why keeping the number of those rules at one matters.
Exercise 23 — write cmd.sh. Add it to your flow, with shlex.quote.
Exercise 24 — before, not after. Explain why it must be written before the tool launches.
Exercise 25 — the embedded space. Confirm an argument containing a space survives a round trip.
Exercise 26 — irreplaceable. Say why cmd.sh is harder to regenerate than the log.
Exercise 27 — the missing result. Confirm your failing runs leave no results.json, and that the report says missing.
Exercise 28 — survive one bad run. Explain why one_run catches rather than propagates.
Exercise 29 — decide once. Say where the sweep's exit code is chosen.
Exercise 30 — A and B. Explain why the exit code says nothing about whether cleanup ran.
12. Summary
- A run has three endings, and the third — killed from outside — is the one that orphans scratch and leaves a hole in the results.
- Finished and failed runs want opposite cleanups, which is why the policy classifies files rather than outcomes.
finallyruns on four of seven measured endings. Normal return, an uncaught exception,sys.exit, and SIGINT: yes.os._exit, SIGTERM, SIGKILL: no.sys.exitcleans up becauseSystemExitis an exception;os._exitproduces the same exit code with no unwinding, noatexitand no flush.- SIGINT runs cleanup and SIGTERM does not, because Python handles the first and the OS handles the second.
- SIGTERM is what a scheduler, a timeout, or a plain
killsends — so interactive testing with Ctrl-C verifies the opposite of production behaviour. - A three-line SIGTERM handler that raises re-enables every
finally, at the cost of exiting 130 — log the signal, or a killed run is indistinguishable from a cancelled one. - SIGKILL cannot be handled, and a handler still leaves partially written files — write-to-temp-then-rename matters more than cleanup.
- A
returninfinallyswallows the exception entirely. The caller sees the cleanup's value and no error, and a crashed run is recorded as a pass. - An exception raised in
finallyreplaces the original, which survives only as__context__— so a failing cleanup hides the crash it was cleaning up after, loudly. - Code that runs during failure handling must not be able to fail. Log the cleanup's own problems; never raise them.
- The classification test is "could this be regenerated from what remains?" No means evidence; yes means scratch.
- Waveforms are the one outcome-dependent rule — delete on success, keep on failure — because the failures that need them most are the non-deterministic ones.
cmd.shis the irreplaceable artefact. Executable, inside the run directory, written before the tool launches, quoted withshlex.quote.- A sweep catches a run's exception rather than propagating it, and decides the exit code once from the collected results.
- The exit code tells you nothing about whether cleanup ran.
sys.exit(4)andos._exit(4)are identical from outside.
The next chapter is about the failure that does not reproduce — where everything in this chapter's run directory is the only thing you have.
Where this fits
Part of the Python curriculum.