Python Part II · Module 33 · Testing and Debugging Automation
Testing the Failure Paths, Not Just the Happy One
Every flow has code that runs only when something has already gone wrong — the branch for a tool that is not on PATH, the branch for a run that timed out, the branch for a log that stops halfway. By construction nobody exercises those by hand, which makes them the least tested code in a flow and the code most likely to run on the worst night. Testing them needs one construct, and that construct has a defect that lets a test pass without ever calling the thing it claims to test.
Advanced21 min readPythonPart IIAutomationTestingError Handling
Part II · Module 33 · Chapter 33.6 · Testing the Failure Paths, Not Just the Happy One
1. The Code Nobody Runs
PATHS = [
("the happy path", "every night", "exercised constantly"),
("a missing tool", "the night a module file changed", "never by hand"),
("a non-zero return code", "a compile error", "sometimes"),
("a timeout", "a hang, or a slow machine", "never by hand"),
("partial output", "the job was killed", "never by hand"),
("the cleanup after a failure", "only after a failure", "never by hand"),
]
print(f"{'path':<30} {'when it runs':<34} exercised")
for path, when, how in PATHS:
print(f"{path:<30} {when:<34} {how}")
print()
never = [p for p in PATHS if p[2] == "never by hand"]
print("paths never exercised by hand:", len(never))
print("and they all run:", "on the night something is already wrong")path when it runs exercised
the happy path every night exercised constantly
a missing tool the night a module file changed never by hand
a non-zero return code a compile error sometimes
a timeout a hang, or a slow machine never by hand
partial output the job was killed never by hand
the cleanup after a failure only after a failure never by hand
paths never exercised by hand: 4
and they all run: on the night something is already wrongFour of six paths are never exercised by hand, and the compounding is the problem: they run when something else has already failed, so a bug in them arrives on top of an existing incident and gets attributed to it.
2. The Annotation
with pytest.raises(ToolError):
run_test("smoke", 1, sim="nosuchtool")That is the right shape and it uses a framework this environment does not have, so here it is in the standard library, which has the same construct under a different name:
import unittest
class ToolError(Exception):
"""Raised when the flow's own expectations about a tool are violated."""
def run_test(test, seed, sim="xrun"):
import shutil
if shutil.which(sim) is None:
raise ToolError(f"simulator not on PATH: {sim}")
return {"test": test, "seed": seed, "status": "PASS"}
class FailurePathTests(unittest.TestCase):
def test_missing_simulator_raises(self):
with self.assertRaises(ToolError):
run_test("smoke", 1, sim="nosuchtool")
import io
suite = unittest.TestLoader().loadTestsFromTestCase(FailurePathTests)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
print("result:", {"run": res.testsRun, "failures": len(res.failures),
"ok": res.wasSuccessful()})
print()
print("pytest.raises and unittest.assertRaises are:", "the same construct")
print("installed here :", "assertRaises only")result: {'run': 1, 'failures': 0, 'ok': True}
pytest.raises and unittest.assertRaises are: the same construct
installed here : assertRaises onlyThe test passes, and it is a real test of a real failure path: run_test with a simulator that does not exist raises, and the test asserts that it does.
It also has a hole, and the hole is in the construct rather than in this use of it.
3. assertRaises Passes If Anything in the Block Raises
import io
import unittest
class ToolError(Exception):
pass
def subject(sim):
"""The thing under test. Note it is NOT called in the first test."""
raise ToolError(f"simulator not on PATH: {sim}")
class TooBroad(unittest.TestCase):
def test_looks_like_a_failure_path_test(self):
with self.assertRaises(ToolError):
cfg = {"sim": "nosuchtool"}
raise ToolError("the test's own setup blew up")
subject(cfg["simulator"]) # never reached — and misspelt
def test_the_subjects_name_is_misspelt(self):
with self.assertRaises(NameError):
undefined_helper() # a typo in the TEST
subject("nosuchtool")
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"ok": res.wasSuccessful()}
print("two tests that never call the subject:", run(TooBroad))
print()
print("test 1 raised ToolError from:", "its own setup line")
print("test 2 raised NameError from:", "a typo in the test itself")
print("the subject was invoked :", 0, "times")
print("the suite reports :", "two passing failure-path tests")two tests that never call the subject: {'run': 2, 'failures': 0, 'ok': True}
test 1 raised ToolError from: its own setup line
test 2 raised NameError from: a typo in the test itself
the subject was invoked : 0 times
the suite reports : two passing failure-path testsTwo passing tests, and the function under test was never called.
assertRaises as a context manager checks only that something inside the block raised the expected type. It has no idea which statement raised, so a test whose own setup fails with the same exception class is indistinguishable from a test whose subject behaved correctly.
That is a worse failure than a test that is simply missing. A missing test is visible in a coverage report; this one appears as a green assertion about a failure path, so nobody looks at it again.
Two habits close it:
import io
import unittest
class ToolError(Exception):
pass
def run_test(test, seed, sim="xrun"):
import shutil
if shutil.which(sim) is None:
raise ToolError(f"simulator not on PATH: {sim}")
return {"test": test, "seed": seed, "status": "PASS"}
class Narrow(unittest.TestCase):
def test_one_call_inside_the_block(self):
# everything the test needs is built OUTSIDE the block
args = ("smoke", 1)
sim = "nosuchtool"
with self.assertRaises(ToolError):
run_test(*args, sim=sim)
def test_and_the_message_names_the_tool(self):
with self.assertRaises(ToolError) as cm:
run_test("smoke", 1, sim="nosuchtool")
self.assertIn("nosuchtool", str(cm.exception))
self.assertIn("not on PATH", str(cm.exception))
def test_the_happy_path_does_not_raise(self):
import sys
from pathlib import Path
r = run_test("smoke", 1, sim=Path(sys.executable).name)
self.assertEqual(r["status"], "PASS")
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"ok": res.wasSuccessful()}
print("narrowed suite:", run(Narrow))
print()
HABITS = [
("build the inputs outside the block", "a setup bug cannot satisfy it"),
("one call inside the block", "only the subject can raise"),
("assert on cm.exception", "the right error for the right reason"),
("keep a happy-path test", "a function that always raises cannot pass"),
]
for h, why in HABITS:
print(f" {h:<36} {why}")narrowed suite: {'run': 3, 'failures': 0, 'ok': True}
build the inputs outside the block a setup bug cannot satisfy it
one call inside the block only the subject can raise
assert on cm.exception the right error for the right reason
keep a happy-path test a function that always raises cannot passThree tests, four habits, and the third is the one that turns a type check into a real assertion: ToolError could be raised for any reason, and assertIn("nosuchtool", ...) pins that it was raised for this one.
4. Four Paths, Four Shapes
The canonical purpose names four, and they do not all raise:
import subprocess
import sys
from pathlib import Path
Path("ok.py").write_text("print('fine')\n", encoding="utf-8")
Path("bad.py").write_text("import sys\nprint('failing')\nsys.exit(3)\n",
encoding="utf-8")
Path("slow.py").write_text("import time\ntime.sleep(30)\n", encoding="utf-8")
print("1. a missing tool")
try:
subprocess.run(["nosuchtool_xyz"])
except FileNotFoundError as exc:
print(f" raises {type(exc).__name__}, errno {exc.errno}")
print("2. a non-zero return code")
r = subprocess.run([sys.executable, "bad.py"], stdout=subprocess.DEVNULL)
print(f" raises nothing; returncode = {r.returncode}")
try:
subprocess.run([sys.executable, "bad.py"], stdout=subprocess.DEVNULL,
check=True)
except subprocess.CalledProcessError as exc:
print(f" with check=True: {type(exc).__name__}, rc {exc.returncode}")
print("3. a timeout")
try:
subprocess.run([sys.executable, "slow.py"], timeout=1)
except subprocess.TimeoutExpired as exc:
print(f" raises {type(exc).__name__}, timeout {exc.timeout}")
print("4. partial output")
PARTIAL = "UVM_INFO started\nUVM_INFO still going\n"
print(f" raises nothing; the text parses, and has no banner: "
f"{'TEST DONE' not in PARTIAL}")1. a missing tool
raises FileNotFoundError, errno 2
2. a non-zero return code
raises nothing; returncode = 3
with check=True: CalledProcessError, rc 3
3. a timeout
raises TimeoutExpired, timeout 1
4. partial output
raises nothing; the text parses, and has no banner: TrueTwo of the four raise and two do not, which means a failure-path suite written entirely with assertRaises tests half of what the purpose names.
The second row is the one that catches people. subprocess.run with a non-zero return code is a successful call — the process ran and reported a failure, which is not an error in Python's terms — so a test written as with assertRaises(ToolError): run_step() never fires if run_step forwards the code instead of raising. Chapter 23.4 established the behaviour; here is the test consequence.
So the four paths need two different test shapes:
SHAPES = [
("a missing tool", "assertRaises", "FileNotFoundError or your ToolError"),
("a timeout", "assertRaises", "subprocess.TimeoutExpired"),
("a non-zero return code", "assertEqual", "the status your classifier returns"),
("partial output", "assertEqual", "NO_FINISH from the log — 33.5"),
]
print(f"{'path':<26} {'test shape':<16} what you assert")
for path, shape, what in SHAPES:
print(f"{path:<26} {shape:<16} {what}")
print()
print("paths tested with assertRaises:", 2)
print("paths tested with assertEqual :", 2)
print("suites that use only one shape:", "cover half the purpose")path test shape what you assert
a missing tool assertRaises FileNotFoundError or your ToolError
a timeout assertRaises subprocess.TimeoutExpired
a non-zero return code assertEqual the status your classifier returns
partial output assertEqual NO_FINISH from the log — 33.5
paths tested with assertRaises: 2
paths tested with assertEqual : 2
suites that use only one shape: cover half the purpose5. Testing a Timeout Without Waiting
A timeout test that waits for a real timeout is a test nobody runs:
import subprocess
import sys
import time
from pathlib import Path
Path("slow.py").write_text("import time\ntime.sleep(30)\n", encoding="utf-8")
start = time.monotonic()
try:
subprocess.run([sys.executable, "slow.py"], timeout=0.3)
except subprocess.TimeoutExpired:
elapsed = time.monotonic() - start
print("TimeoutExpired raised:", True)
print("the child slept for :", "30 s, by construction")
print("the test waited under 5 s:", elapsed < 5.0)
print("the test waited under 1 s:", elapsed < 1.0)
print()
print("(the duration is reported as a bound, not a number — 27's rule)")
print("the trick:", "a short timeout against a long sleep")
print("what it does NOT test:", "that your real timeout value is sensible")TimeoutExpired raised: True
the child slept for : 30 s, by construction
the test waited under 5 s: True
the test waited under 1 s: True
(the duration is reported as a bound, not a number — 27's rule)
the trick: a short timeout against a long sleep
what it does NOT test: that your real timeout value is sensibleA 0.3-second timeout against a 30-second sleep, so the test exercises the real TimeoutExpired path and finishes immediately.
The last line is the honest limit. This test proves the flow handles a timeout; it says nothing about whether 1800 seconds is the right limit for a nightly regression, which is a policy question of the kind Chapter 29.2 Section 7 said to record with a reason rather than test.
The same technique parameterises the production timeout so the test can shrink it:
import subprocess
import sys
from pathlib import Path
Path("slow.py").write_text("import time\ntime.sleep(30)\n", encoding="utf-8")
DEFAULT_TIMEOUT_S = 1800
def run_step(cmd, timeout_s=DEFAULT_TIMEOUT_S):
"""The timeout is a parameter, so a test can pass 0.3."""
try:
p = subprocess.run(cmd, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, text=True,
timeout=timeout_s)
return {"status": "DONE", "returncode": p.returncode}
except subprocess.TimeoutExpired:
return {"status": "TIMEOUT", "returncode": None}
print("production default:", DEFAULT_TIMEOUT_S, "s")
print("the test calls it with:", 0.3, "s")
print()
print("result:", run_step([sys.executable, "slow.py"], timeout_s=0.3))
print()
print("hardcoded timeout -> untestable:", True)
print("parameterised -> testable :", True)
print("this is 33.2 §5's seam :", "move the value out, leave the logic in")production default: 1800 s
the test calls it with: 0.3 s
result: {'status': 'TIMEOUT', 'returncode': None}
hardcoded timeout -> untestable: True
parameterised -> testable : True
this is 33.2 §5's seam : move the value out, leave the logic inreturncode: None is the detail worth keeping. Chapter 32.7 Section 9 showed that None distinguishes a timeout from an exit code, and here it is the value the timeout branch produces — so the status and the returncode together say timed out rather than exited with nothing.
6. Testing Partial Output and a Non-Zero Code Together
The two non-raising paths combine, and the combination is what a real bad night looks like:
import io
import subprocess
import sys
import unittest
from pathlib import Path
Path("partial.py").write_text(
"import sys\n"
"print('UVM_INFO tb.sv(42) @ 0: reset released')\n"
"print('UVM_INFO axi_seq.sv(88) @ 1200: starting burst 0')\n"
"sys.exit(139)\n", encoding="utf-8")
def classify(text, returncode, timed_out):
if timed_out:
return "TIMEOUT"
if returncode is not None and returncode < 0:
return "KILLED"
if not text.strip():
return "EMPTY_LOG"
if any(l.startswith("UVM_FATAL") for l in text.splitlines()):
return "FAIL"
if "TEST DONE" not in text:
return "NO_FINISH"
if any(l.startswith("UVM_ERROR") for l in text.splitlines()):
return "FAIL"
return "PASS"
def run_step(cmd, timeout_s=10):
try:
p = subprocess.run(cmd, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, text=True,
timeout=timeout_s)
return p.stdout, p.returncode, False
except subprocess.TimeoutExpired as exc:
return (exc.stdout or b"").decode(errors="replace"), None, True
class CombinedPaths(unittest.TestCase):
def test_partial_output_with_a_crash_code(self):
text, rc, to = run_step([sys.executable, "partial.py"])
self.assertEqual(rc, 139)
self.assertNotIn("TEST DONE", text)
self.assertEqual(classify(text, rc, to), "NO_FINISH")
def test_the_log_is_still_kept(self):
text, rc, to = run_step([sys.executable, "partial.py"])
self.assertGreater(len(text.splitlines()), 0,
"a partial log still has lines worth keeping")
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"ok": res.wasSuccessful()}
text, rc, to = run_step([sys.executable, "partial.py"])
print("the child printed", len(text.splitlines()), "lines and exited", rc)
print("the classifier says:", classify(text, rc, to))
print("suite:", run(CombinedPaths))the child printed 2 lines and exited 139
the classifier says: NO_FINISH
suite: {'run': 2, 'failures': 0, 'ok': True}Two lines of log and exit code 139, which is what a segfaulting simulator produces via a shell. Neither the return code nor the partial log raises, so both facts have to be asserted with assertEqual.
The second test is the one that matters most operationally: a partial log is evidence, and a failure path that discards it leaves the next morning with nothing. Chapter 33.13 is about that decision; this is the test that pins it.
7. A Domain Error Type
The annotation raises ToolError, not FileNotFoundError, and the difference is worth a section:
import shutil
import subprocess
class ToolError(Exception):
"""The flow's own expectations about a tool were violated."""
def run_raw(sim, args):
"""Lets the library's exception escape."""
return subprocess.run([sim] + args)
def run_wrapped(sim, args):
"""Translates into the flow's own vocabulary."""
if shutil.which(sim) is None:
raise ToolError(f"simulator not on PATH: {sim}. "
f"Check the module load in run.sh")
return subprocess.run([sim] + args)
for label, fn in (("raw", run_raw), ("wrapped", run_wrapped)):
try:
fn("nosuchtool_xyz", [])
print(f"{label:<10} no exception")
except Exception as exc:
print(f"{label:<10} {type(exc).__name__}: {str(exc)[:54]}")raw FileNotFoundError: [Errno 2] No such file or directory: 'nosuchtool_xyz'
wrapped ToolError: simulator not on PATH: nosuchtool_xyz. Check the modulTwo things change. The type distinguishes the flow's own faults from the library's, so a caller can catch ToolError without also catching a FileNotFoundError from an unrelated missing log — which is Chapter 14.8's argument for custom error types, arriving at a boundary.
And the message says what to do. [Errno 2] No such file or directory: 'nosuchtool_xyz' is accurate and tells a reader nothing about the module load that was supposed to put it on PATH.
class ToolError(Exception):
pass
class ConfigError(Exception):
pass
FAULTS = [
("the simulator is not on PATH", "ToolError", "the environment — 32.2 §9"),
("the filelist does not exist", "ConfigError", "the spec — 33.11"),
("the log could not be decoded", "ToolError", "the tool's output — 33.5"),
("a dict key is missing", "KeyError", "a bug in the flow — let it escape"),
("a path was built wrongly", "IsADirectoryError", "a bug in the flow"),
]
print(f"{'fault':<34} {'raise':<20} whose fault")
for fault, exc, whose in FAULTS:
print(f"{fault:<34} {exc:<20} {whose}")
print()
print("faults translated into the flow's vocabulary:", 3)
print("faults left to escape as themselves :", 2)
print("the rule:", "translate what the world does; let your own bugs escape")fault raise whose fault
the simulator is not on PATH ToolError the environment — 32.2 §9
the filelist does not exist ConfigError the spec — 33.11
the log could not be decoded ToolError the tool's output — 33.5
a dict key is missing KeyError a bug in the flow — let it escape
a path was built wrongly IsADirectoryError a bug in the flow
faults translated into the flow's vocabulary: 3
faults left to escape as themselves : 2
the rule: translate what the world does; let your own bugs escapeTranslate what the world does; let your own bugs escape. A KeyError wrapped in ToolError becomes indistinguishable from a genuine environment problem, which is the same mistake Chapter 33.5 Section 4 identified in absorbing an IsADirectoryError into a status.
8. The Failure-Path Suite, Assembled
import io
import shutil
import subprocess
import sys
import unittest
from pathlib import Path
class ToolError(Exception):
pass
DEFAULT_TIMEOUT_S = 1800
def classify(text, returncode, timed_out):
if timed_out:
return "TIMEOUT"
if returncode is not None and returncode < 0:
return "KILLED"
if not text.strip():
return "EMPTY_LOG"
if any(l.startswith("UVM_FATAL") for l in text.splitlines()):
return "FAIL"
if "TEST DONE" not in text:
return "NO_FINISH"
if any(l.startswith("UVM_ERROR") for l in text.splitlines()):
return "FAIL"
return "PASS"
def run_step(sim, args, timeout_s=DEFAULT_TIMEOUT_S):
if shutil.which(sim) is None:
raise ToolError(f"simulator not on PATH: {sim}")
try:
p = subprocess.run([sim] + args, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, text=True,
timeout=timeout_s)
return {"text": p.stdout, "returncode": p.returncode,
"timed_out": False}
except subprocess.TimeoutExpired as exc:
return {"text": (exc.stdout or b"").decode(errors="replace"),
"returncode": None, "timed_out": True}
PY = Path(sys.executable).name
Path("ok.py").write_text(
"print('UVM_INFO started')\nprint('TEST DONE')\n", encoding="utf-8")
Path("partial.py").write_text(
"import sys\nprint('UVM_INFO started')\nsys.exit(139)\n", encoding="utf-8")
Path("slow.py").write_text("import time\ntime.sleep(30)\n", encoding="utf-8")
class FailurePaths(unittest.TestCase):
# raising paths — one call per block, inputs built outside, message asserted
def test_missing_tool_raises_tool_error(self):
sim = "nosuchtool_xyz"
with self.assertRaises(ToolError) as cm:
run_step(sim, [])
self.assertIn(sim, str(cm.exception))
self.assertIn("not on PATH", str(cm.exception))
# non-raising paths — asserted with assertEqual
def test_non_zero_return_code_is_forwarded(self):
r = run_step(PY, ["partial.py"], timeout_s=10)
self.assertEqual(r["returncode"], 139)
self.assertFalse(r["timed_out"])
def test_partial_output_classifies_no_finish(self):
r = run_step(PY, ["partial.py"], timeout_s=10)
self.assertEqual(classify(**r), "NO_FINISH")
def test_partial_output_is_still_kept(self):
r = run_step(PY, ["partial.py"], timeout_s=10)
self.assertGreater(len(r["text"].splitlines()), 0)
def test_timeout_sets_status_and_none_returncode(self):
r = run_step(PY, ["slow.py"], timeout_s=0.3)
self.assertTrue(r["timed_out"])
self.assertIsNone(r["returncode"])
self.assertEqual(classify(**r), "TIMEOUT")
# the positive case, which stops an always-raising subject passing
def test_the_happy_path_still_works(self):
r = run_step(PY, ["ok.py"], timeout_s=10)
self.assertEqual(r["returncode"], 0)
self.assertEqual(classify(**r), "PASS")
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"errors": len(res.errors), "ok": res.wasSuccessful()}
print("failure-path suite:", run(FailurePaths))
print()
print("paths covered :", 4)
print("raising paths :", 1)
print("non-raising paths :", 3)
print("positive cases kept :", 1)
print("the suite waits at most :", "a fraction of a second, by construction")failure-path suite: {'run': 6, 'failures': 0, 'errors': 0, 'ok': True}
paths covered : 4
raising paths : 1
non-raising paths : 3
positive cases kept : 1
the suite waits at most : a fraction of a second, by constructionSix tests covering all four paths the canonical purpose names, with every habit from Section 3 applied: inputs built outside the block, one call inside it, the message asserted, and a positive case that an always-raising implementation would fail.
Note the timeout test uses timeout_s=0.3 against a thirty-second sleep. The production default of 1800 is never exercised and does not need to be — what the test pins is the branch, and Chapter 29.2 Section 7's rule covers the number.
9. The Type Is Not the Cause
assertRaises(ValueError) asserts that a ValueError happened. A parser
that raises ValueError for six different reasons therefore has six failure
paths and one test that passes for all of them:
import io
import unittest
def parse(text):
if not text:
raise ValueError("empty log")
if "TEST DONE" not in text:
raise ValueError("truncated log: no TEST DONE")
if not text.startswith("UVM_INFO"):
raise ValueError("not a UVM log: unexpected first line")
return "ok"
class TypeOnly(unittest.TestCase):
"""Intended to test the TRUNCATED path. Given the EMPTY input."""
def test_truncated_log_raises(self):
with self.assertRaises(ValueError):
parse("") # the wrong input entirely
class WithRegex(unittest.TestCase):
"""The same mistake, with the cause asserted."""
def test_truncated_log_raises(self):
with self.assertRaisesRegex(ValueError, "truncated"):
parse("")
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"ok": res.wasSuccessful()}
print("assertRaises , wrong input:", run(TypeOnly))
print("assertRaisesRegex , wrong input:", run(WithRegex))
print()
print("paths that raise ValueError:", 3)
print("the type-only test passes for:", "all 3")
print("so it pins :", "nothing about which path ran")assertRaises , wrong input: {'run': 1, 'failures': 0, 'ok': True}
assertRaisesRegex , wrong input: {'run': 1, 'failures': 1, 'ok': False}
paths that raise ValueError: 3
the type-only test passes for: all 3
so it pins : nothing about which path ranThe type-only test passes against an input it was not written for. Its
name says test_truncated_log_raises and it is exercising the empty-log path,
so the truncation branch could be deleted entirely and this test would stay
green — which is precisely the survivor Chapter 33.9's mutation run would
report.
assertRaisesRegex fails on the same input, for the right reason: the message
does not mention truncation. One extra argument turns "something went wrong"
into "the thing I meant went wrong".
There is a trade to state, because the regex couples the test to wording:
TRADES = [
("assertRaises(ValueError)", "no", "any of 3 paths", "none"),
("assertRaisesRegex(V, 'truncated')", "yes", "1 path", "the word 'truncated'"),
("assertRaisesRegex(V, 'no TEST DONE')", "yes", "1 path", "a whole phrase"),
("a distinct exception class", "yes", "1 path", "none — §7"),
]
print(f"{'what you assert':<38} {'pins the cause':<16} {'matches':<16} couples to")
for what, pins, matches, couples in TRADES:
print(f"{what:<38} {pins:<16} {matches:<16} {couples}")
print()
print("the best of the four:", "a distinct exception class")
print("the cheapest :", "a regex on a SHORT, stable substring")
print("the rule :", "match the one word the message cannot lose")what you assert pins the cause matches couples to
assertRaises(ValueError) no any of 3 paths none
assertRaisesRegex(V, 'truncated') yes 1 path the word 'truncated'
assertRaisesRegex(V, 'no TEST DONE') yes 1 path a whole phrase
a distinct exception class yes 1 path none — §7
the best of the four: a distinct exception class
the cheapest : a regex on a SHORT, stable substring
the rule : match the one word the message cannot loseRow four is the right answer and rows two and three are the cheap one. A
distinct exception class per failure mode — which the next section builds — pins
the cause with no dependence on wording at all. Until you have that, a regex
on one short word is the compromise: "truncated" survives every rewording
that keeps the message meaningful, where "truncated log: no TEST DONE" breaks
the moment somebody improves the sentence.
That is Chapter 33.8 Section 7's rule arriving inside a test suite. A message's exact wording is explicitly not part of the contract, so a test must depend on the smallest part of it that carries the meaning.
import io
import re
import unittest
def parse(text):
if not text:
raise ValueError("empty log")
if "TEST DONE" not in text:
raise ValueError("truncated log: no TEST DONE")
if not text.startswith("UVM_INFO"):
raise ValueError("not a UVM log: unexpected first line")
return "ok"
CASES = [("", "empty"), ("UVM_INFO a\n", "truncated"),
("garbage\nTEST DONE\n", "not a UVM log")]
class EveryPathIsDistinguished(unittest.TestCase):
def test_each_input_raises_for_its_own_reason(self):
for text, expected in CASES:
with self.subTest(expected=expected):
with self.assertRaisesRegex(ValueError, expected):
parse(text)
def test_the_three_messages_are_distinct(self):
msgs = []
for text, _ in CASES:
try:
parse(text)
except ValueError as exc:
msgs.append(str(exc))
self.assertEqual(len(set(msgs)), len(CASES))
def test_no_message_matches_another_cases_pattern(self):
"""The patterns must not overlap, or §2's test would pass by luck."""
msgs = {}
for text, expected in CASES:
try:
parse(text)
except ValueError as exc:
msgs[expected] = str(exc)
for pattern in msgs:
matching = [k for k, m in msgs.items()
if re.search(pattern, m)]
self.assertEqual(matching, [pattern])
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"ok": res.wasSuccessful()}
print("per-cause suite:", run(EveryPathIsDistinguished))
print()
print("failure paths :", len(CASES))
print("each pinned to its cause:", True)
print("patterns proven disjoint:", True)per-cause suite: {'run': 3, 'failures': 0, 'ok': True}
failure paths : 3
each pinned to its cause: True
patterns proven disjoint: TrueThree failure paths, each pinned to its own cause, and a third test doing something suites almost never do: proving the patterns do not overlap.
Without it, two causes whose messages share the matched word would each satisfy the other's assertion, and the suite would report three green tests while pinning only one distinction. That is the same shape as Chapter 33.9's equivalent mutant — a test that cannot distinguish the thing it claims to — and here one assertion rules it out.
10. Predict, Run, Observe
import subprocess
import sys
import unittest
from pathlib import Path
Path("bad.py").write_text("import sys\nsys.exit(3)\n", encoding="utf-8")
class T(unittest.TestCase):
def test_a(self):
with self.assertRaises(ValueError):
int("x")
raise RuntimeError("never reached")
# A
import io
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(
unittest.TestLoader().loadTestsFromTestCase(T))
print("A:", res.wasSuccessful())
# B
r = subprocess.run([sys.executable, "bad.py"])
print("B:", r.returncode)
# C
try:
subprocess.run([sys.executable, "bad.py"], check=True)
print("C: no exception")
except subprocess.CalledProcessError as exc:
print("C:", type(exc).__name__, exc.returncode)
# D
try:
subprocess.run(["nosuchtool_xyz"])
except FileNotFoundError as exc:
print("D:", exc.errno)The verified output:
A: True
B: 3
C: CalledProcessError 3
D: 2A — a passing test whose subject was never called, because int("x") satisfied the assertRaises.
B — subprocess.run returns 3 and raises nothing. A failure-path test written with assertRaises around this never fires.
C — check=True turns the same event into an exception, which is a choice about which test shape you want.
D — errno 2 for a missing executable, which is the one failure path that does raise by default.
11. Exercises
Exercise 1 — six paths. List your flow's failure paths and mark which are ever exercised by hand.
Exercise 2 — the compounding. Explain why a bug in a failure path is harder to attribute than one in the happy path.
Exercise 3 — the construct. Write the assertRaises test for one of your own error paths.
Exercise 4 — the hole. Add a deliberate setup bug inside an assertRaises block and confirm the test still passes.
Exercise 5 — never invoked. Count how many of your assertRaises blocks contain more than one call.
Exercise 6 — inputs outside. Move the setup out of one block and say what that rules out.
Exercise 7 — one call inside. Apply the rule to every assertRaises in your suite.
Exercise 8 — assert the message. Add a cm.exception check to one test and say what it pins.
Exercise 9 — the positive case. Confirm your negative suite has one, and break the subject to prove it matters.
Exercise 10 — four paths, two shapes. Classify your own failure paths into raising and non-raising.
Exercise 11 — run does not raise. Demonstrate that subprocess.run returns a non-zero code without raising.
Exercise 12 — check=True. Decide whether your flow wants it, and justify the choice.
Exercise 13 — half the purpose. Find a failure-path suite that uses only assertRaises and say what it misses.
Exercise 14 — a fast timeout test. Write one with a short timeout against a long sleep.
Exercise 15 — a bound, not a number. Report your test's duration as a bound and say why.
Exercise 16 — what it does not test. State what a timeout test says about your production timeout value.
Exercise 17 — parameterise the timeout. Make yours an argument with a production default.
Exercise 18 — None means timed out. Confirm your timeout branch produces a None return code.
Exercise 19 — 139 and two lines. Construct a partial-output-plus-crash-code case for your flow.
Exercise 20 — keep the evidence. Add the assertion that a partial log is retained.
Exercise 21 — a domain error type. Define one for your flow and translate two library exceptions into it.
Exercise 22 — the actionable message. Rewrite one exception message to say what to check.
Exercise 23 — let your bugs escape. Find a try in your flow that wraps a KeyError, and say what it hides.
Exercise 24 — six tests, four paths. Build the Section 8 suite for your own runner.
Exercise 25 — count the paths. Count how many places in one parser raise the same exception type.
Exercise 26 — the wrong input. Give a type-only test an input from a different failure path and watch it pass.
Exercise 27 — the name lies. Explain what the test's name claims and what it actually exercises.
Exercise 28 — the survivor. Say what Chapter 33.9's mutation run would report for that test.
Exercise 29 — add the regex. Convert it to assertRaisesRegex and confirm it now fails.
Exercise 30 — one short word. Choose the smallest substring of each message that carries its meaning.
Exercise 31 — the whole phrase. Reword one message and say which of your tests break.
Exercise 32 — prove them disjoint. Add the test asserting no message matches another case's pattern.
12. Summary
- Four of six failure paths are never exercised by hand, and they run when something has already failed — so a bug in them gets attributed to the incident.
pytest.raisesandunittest.assertRaisesare the same construct; only the second is installed here, and the annotation is carried as quoted text.assertRaisespasses if anything in its block raises that type — two tests passed with the subject never invoked, one satisfied by the test's own setup and one by a typo in the test.- That is worse than a missing test, because it appears as a green assertion about a failure path.
- Four habits close it: build inputs outside the block, keep one call inside it, assert on
cm.exception, and keep a happy-path test so an always-raising subject cannot pass. - Two of the four paths the purpose names do not raise at all.
subprocess.runwith a non-zero code is a successful call, so a suite written only withassertRaisescovers half the purpose. - Use
assertRaisesfor a missing tool and a timeout,assertEqualfor a return code and partial output. - A 0.3-second timeout against a 30-second sleep exercises the real
TimeoutExpiredbranch and finishes immediately — and says nothing about whether the production value is sensible. - Parameterise the timeout so a test can shrink it; a hardcoded value is untestable, which is Chapter 33.2's seam again.
returncode: Nonedistinguishes a timeout from an exit code, so status and returncode together say timed out.- A partial log is evidence, and the test that pins it is the assertion that it was kept.
- Translate what the world does into your own error type; let your own bugs escape — a
KeyErrorwrapped inToolErroris indistinguishable from an environment problem. - The message should say what to check, not merely what failed.
assertRaises(ValueError)passes for every path that raisesValueError, so a test named for one cause can be exercising another and stay green when its branch is deleted.assertRaisesRegexpins the cause and fails on the same wrong input.- A distinct exception class per failure mode is the right answer; a regex on one short, stable word is the cheap one.
- Depend on the smallest part of a message that carries the meaning — exact wording is not part of the contract (Chapter 33.8 Section 7).
- Prove the patterns are disjoint. Overlapping patterns let two causes satisfy each other's assertion, and the suite reports green while pinning one distinction.
The next chapter makes every test run in its own throwaway directory, so none of this touches a real run area.
Where this fits
Part of the Python curriculum.