Python Part II · Module 33 · Testing and Debugging Automation
What Can Be Tested, and What Needs the Tool
The usual objection to testing flow code is that it needs a simulator, a licence and forty minutes, and for one part of a flow that is true. The useful question is how large that part is — and it is measurable rather than arguable. This chapter takes a complete runner of the kind Module 25 built, sorts every function by what it reads, and counts the lines on each side of the line. The answer here is a third, concentrated in a single function, and the reason it is only a third comes down to one parameter.
Intermediate20 min readPythonPart IIAutomationTestingDesign
Part II · Module 33 · Chapter 33.2 · What Can Be Tested, and What Needs the Tool
1. The Question, Made Answerable
QUESTIONS = [
("does this function read the filesystem?", "no", "testable"),
("does it launch a process?", "no", "testable"),
("does it read the clock or the environment?", "no", "testable"),
("does it do any of those?", "yes", "needs a seam or a tool"),
]
print(f"{'ask of every function':<44} {'answer':<8} verdict")
for q, a, v in QUESTIONS:
print(f"{q:<44} {a:<8} {v}")
print()
print("the test is mechanical:", "what does this function READ?")
print("which is also 32.5's test:", "applied to a Python function instead of a tool command")ask of every function answer verdict
does this function read the filesystem? no testable
does it launch a process? no testable
does it read the clock or the environment? no testable
does it do any of those? yes needs a seam or a tool
the test is mechanical: what does this function READ?
which is also 32.5's test: applied to a Python function instead of a tool commandChapter 32.5 asked what does this work read to decide whether it belonged inside a tool. The same question sorts Python functions into testable and not, and the symmetry is not a coincidence: both are asking whether a piece of work can be given its inputs by hand.
2. The Annotation
# testable: parse_log_text(text) -> dict
# not testable in isolation: subprocess.run(["vcs", ...])def parse_log_text(text):
"""Pure: text in, dict out."""
errors = sum(1 for l in text.splitlines() if l.startswith("UVM_ERROR"))
fatals = sum(1 for l in text.splitlines() if l.startswith("UVM_FATAL"))
finished = "TEST DONE" in text
return {"errors": errors, "fatals": fatals, "finished": finished}
SAMPLE = ("UVM_INFO started\n"
"UVM_ERROR drv.sv(88): bad handshake\n"
"UVM_ERROR drv.sv(91): bad handshake\n"
"TEST DONE\n")
print("parse_log_text(SAMPLE) =", parse_log_text(SAMPLE))
print()
print("inputs it needs :", "one string")
print("files it touches :", 0)
print("processes it launches:", 0)
print("licences it needs :", 0)parse_log_text(SAMPLE) = {'errors': 2, 'fatals': 0, 'finished': True}
inputs it needs : one string
files it touches : 0
processes it launches: 0
licences it needs : 0The annotation's two comments are both right, and the interesting word is in the second one: "in isolation". subprocess.run(["vcs", ...]) is not untestable in principle — Section 6 shows three things its tests can assert — but it cannot be exercised end to end without the thing it launches.
3. The Word "Thin" Is a Claim
CLAIMS = [
("pure functions can be tested in milliseconds", "measured in §7"),
("only the tool-calling layer cannot", "measured in §4"),
("that layer is thin", "a number, not an adjective — §4"),
]
for c, how in CLAIMS:
print(f" {c:<46} {how}")
print()
print("claims in the canonical purpose:", len(CLAIMS))
print("claims that are checkable :", len(CLAIMS)) pure functions can be tested in milliseconds measured in §7
only the tool-calling layer cannot measured in §4
that layer is thin a number, not an adjective — §4
claims in the canonical purpose: 3
claims that are checkable : 3"Thin" is the load-bearing word, and it is a design property rather than a fact of nature — a flow can easily be written so that most of it needs a tool. The next section measures a well-shaped one and Section 5 shows what makes the difference.
4. Counting a Real Runner
import inspect
import subprocess
from pathlib import Path
def build_command(sim, test, seed, run_dir):
return [sim, "-f", "files.f", f"+UVM_TESTNAME={test}", f"+seed={seed}",
"-l", str(Path(run_dir) / "sim.log")]
def run_dir_for(test, seed):
return Path("runs") / f"{test}_{seed}"
def parse_log_text(text):
errors = sum(1 for l in text.splitlines() if l.startswith("UVM_ERROR"))
fatals = sum(1 for l in text.splitlines() if l.startswith("UVM_FATAL"))
return {"errors": errors, "fatals": fatals, "finished": "TEST DONE" in text}
def classify(parsed, returncode, timed_out):
if timed_out:
return "TIMEOUT"
if returncode is not None and returncode < 0:
return "KILLED"
if parsed["fatals"] or parsed["errors"]:
return "FAIL"
if not parsed["finished"]:
return "NO_FINISH"
return "PASS"
def aggregate(results):
out = {}
for r in results:
out[r["status"]] = out.get(r["status"], 0) + 1
return out
def render_row(r):
return f"| {r['test']} | {r['seed']} | {r['status']} | {r['errors']} |"
def exit_code_for(summary):
if summary.get("KILLED") or summary.get("TIMEOUT"):
return 4
return 1 if summary.get("FAIL") else 0
def execute(sim, test, seed):
"""The impure layer: builds, launches, writes, reads back."""
rd = run_dir_for(test, seed)
rd.mkdir(parents=True, exist_ok=True)
cmd = build_command(sim, test, seed, rd)
(rd / "cmd.txt").write_text(" ".join(cmd) + "\n", encoding="utf-8")
try:
p = subprocess.run(cmd, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, text=True, timeout=5)
out, rc, to = p.stdout, p.returncode, False
except FileNotFoundError:
return {"test": test, "seed": seed, "status": "INFRA", "errors": 0}
except subprocess.TimeoutExpired:
out, rc, to = "", None, True
(rd / "sim.log").write_text(out, encoding="utf-8")
parsed = parse_log_text(out)
return {"test": test, "seed": seed,
"status": classify(parsed, rc, to), "errors": parsed["errors"]}
FUNCS = [build_command, run_dir_for, parse_log_text, classify, aggregate,
render_row, exit_code_for, execute]
IMPURE = {"execute"}
print(f"{'function':<18} {'lines':>6} {'pure?':<7} reads")
total = pure_lines = 0
for f in FUNCS:
n = len(inspect.getsource(f).rstrip().splitlines())
total += n
impure = f.__name__ in IMPURE
if not impure:
pure_lines += n
reads = "the filesystem and a process" if impure else "its arguments"
print(f"{f.__name__:<18} {n:>6} {str(not impure):<7} {reads}")
print()
print(f"functions : {len(FUNCS)} pure {len(FUNCS) - len(IMPURE)}")
print(f"lines : {total} pure {pure_lines}")
print(f"pure fraction : {100.0 * pure_lines / total:.0f}%")
print(f"needs a tool : {total - pure_lines} lines, in {len(IMPURE)} function")function lines pure? reads
build_command 3 True its arguments
run_dir_for 2 True its arguments
parse_log_text 4 True its arguments
classify 10 True its arguments
aggregate 5 True its arguments
render_row 2 True its arguments
exit_code_for 4 True its arguments
execute 18 False the filesystem and a process
functions : 8 pure 7
lines : 48 pure 30
pure fraction : 62%
needs a tool : 18 lines, in 1 functionSeven of eight functions and 62% of the lines are pure, and everything needing a tool is in one function.
The shape is the point rather than the exact percentage. execute is the only function that touches the world, and it does four things — makes a directory, builds a command, launches it, writes a file — none of which involves a decision. Every decision in the runner (which directory, what command, what the log means, what the status is, what the exit code should be) lives in a function that takes its inputs as arguments.
5. The Seam Is One Parameter
What makes that split achievable is a habit visible in two function signatures:
from pathlib import Path
def parse_log(path):
"""Untestable without a filesystem."""
return parse_log_text(Path(path).read_text(encoding="utf-8"))
def parse_log_text(text):
"""Testable with a string literal."""
return {"errors": sum(1 for l in text.splitlines()
if l.startswith("UVM_ERROR"))}
SAMPLE = "UVM_ERROR a\nUVM_ERROR b\nTEST DONE\n"
print("parse_log_text needs:", "a str")
print("result :", parse_log_text(SAMPLE))
print()
print("parse_log needs :", "a path that exists")
try:
parse_log("no/such/file.log")
except FileNotFoundError as exc:
print("calling it with a missing path:", type(exc).__name__)
print()
print("difference between them:", "one line, and one parameter type")
print("tests the text version needs:", "a string literal")
print("tests the path version needs:", "a temporary directory — 33.7")parse_log_text needs: a str
result : {'errors': 2}
parse_log needs : a path that exists
calling it with a missing path: FileNotFoundError
difference between them: one line, and one parameter type
tests the text version needs: a string literal
tests the path version needs: a temporary directory — 33.7One parameter moves a function across the line. parse_log(path) cannot be tested without a filesystem; parse_log_text(text) is tested with a literal, and parse_log becomes a two-line wrapper whose only behaviour is reading a file.
The same transformation applies to every impure input:
SEAMS = [
("a path", "the file's text", "parse_log_text(text)"),
("the current time", "a timestamp argument", "stamp(now)"),
("os.environ", "a dict argument", "resolve(env)"),
("a subprocess result", "returncode and output", "classify(parsed, rc, to)"),
("a directory listing", "a list of names", "pick(names)"),
]
print(f"{'instead of reading':<24} {'take':<28} example")
for impure, pure, ex in SEAMS:
print(f"{impure:<24} {pure:<28} {ex}")
print()
print("seams listed:", len(SEAMS))
print("what they all do:", "move the reading out and leave the deciding in")instead of reading take example
a path the file's text parse_log_text(text)
the current time a timestamp argument stamp(now)
os.environ a dict argument resolve(env)
a subprocess result returncode and output classify(parsed, rc, to)
a directory listing a list of names pick(names)
seams listed: 5
what they all do: move the reading out and leave the deciding inMove the reading out and leave the deciding in. That one sentence produces the 60% from Section 4, and the fourth row is the one that matters most here — classify(parsed, returncode, timed_out) takes the results of a subprocess call rather than making one, so the whole status-classification logic of Chapter 25.6 is testable with three literals.
6. What the Impure Layer's Tests Can Still Assert
"Not testable in isolation" does not mean "no tests". Three things are checkable without the tool:
import io
import unittest
from pathlib import Path
def build_command(sim, test, seed, run_dir):
return [sim, "-f", "files.f", f"+UVM_TESTNAME={test}", f"+seed={seed}",
"-l", str(Path(run_dir) / "sim.log")]
def run_dir_for(test, seed):
return Path("runs") / f"{test}_{seed}"
def classify(parsed, returncode, timed_out):
if timed_out:
return "TIMEOUT"
if returncode is not None and returncode < 0:
return "KILLED"
if parsed["fatals"] or parsed["errors"]:
return "FAIL"
if not parsed["finished"]:
return "NO_FINISH"
return "PASS"
class ImpureLayerTests(unittest.TestCase):
def test_command_shape(self):
cmd = build_command("xrun", "axi_burst", 48210, "runs/x")
self.assertEqual(cmd[0], "xrun")
self.assertIn("+UVM_TESTNAME=axi_burst", cmd)
self.assertIn("+seed=48210", cmd)
def test_command_is_a_list(self):
cmd = build_command("xrun", "t", 1, "runs/t")
self.assertIsInstance(cmd, list)
self.assertTrue(all(isinstance(a, str) for a in cmd))
def test_run_dir_is_per_run(self):
a = run_dir_for("t", 1)
b = run_dir_for("t", 2)
self.assertNotEqual(a, b)
def test_status_from_a_given_log(self):
parsed = {"errors": 2, "fatals": 0, "finished": True}
self.assertEqual(classify(parsed, 0, False), "FAIL")
def test_killed_beats_errors(self):
parsed = {"errors": 0, "fatals": 0, "finished": False}
self.assertEqual(classify(parsed, -9, False), "KILLED")
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"errors": len(res.errors), "ok": res.wasSuccessful()}
print("tests of the tool-calling layer, with no tool:", run(ImpureLayerTests))
print()
COVERED = ["the command it would build", "that the command is a list of str",
"that run directories are per-run", "the status for a given log",
"the precedence between KILLED and FAIL"]
NOT_COVERED = ["whether the tool accepts those arguments",
"whether the tool writes the log where we said",
"whether the design actually passes"]
print("covered without a tool:")
for c in COVERED:
print(" +", c)
print("not covered:")
for c in NOT_COVERED:
print(" -", c)tests of the tool-calling layer, with no tool: {'run': 5, 'failures': 0, 'errors': 0, 'ok': True}
covered without a tool:
+ the command it would build
+ that the command is a list of str
+ that run directories are per-run
+ the status for a given log
+ the precedence between KILLED and FAIL
not covered:
- whether the tool accepts those arguments
- whether the tool writes the log where we said
- whether the design actually passesFive assertions about the tool-calling layer, no tool involved. The pattern is that execute was decomposed in Section 4 into small pure helpers (build_command, run_dir_for, classify) which are testable — so the genuinely untestable residue is the three lines that call subprocess.run and write two files.
This is exactly Chapter 32.5 Section 9's proc-stub harness in Python: the structure is checkable without the executor, and what the executor accepts is not. The honesty is in listing the second column.
7. The Speed Claim, as a Boolean
import io
import time
import unittest
def classify(text):
if "UVM_FATAL" in text:
return "FAIL"
if "TEST DONE" not in text:
return "NO_FINISH"
return "PASS"
class Suite(unittest.TestCase):
pass
for i in range(200):
def make(n):
def t(self):
self.assertEqual(classify("UVM_INFO x\nTEST DONE\n"), "PASS")
return t
setattr(Suite, f"test_{i:03d}", make(i))
start = time.monotonic()
suite = unittest.TestLoader().loadTestsFromTestCase(Suite)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
elapsed = time.monotonic() - start
print("tests run :", res.testsRun)
print("all passed :", res.wasSuccessful())
print("under 1 second :", elapsed < 1.0)
print("under 10 seconds :", elapsed < 10.0)
print()
print("a single tool invocation is under 1 second:", False)
print("the ratio is therefore at least :", "three orders of magnitude")
print("(the duration itself is not published — 27's determinism rule)")tests run : 200
all passed : True
under 1 second : True
under 10 seconds : True
a single tool invocation is under 1 second: False
the ratio is therefore at least : three orders of magnitude
(the duration itself is not published — 27's determinism rule)Two hundred assertions inside a second, reported as a boolean against a bound rather than as a measured duration — which is Chapter 27's rule and also makes this example reproducible on a machine ten times slower.
The comparison that matters is not the absolute speed but what it permits. A suite that runs in under a second runs on every save; a suite that needs a forty-minute tool invocation runs when somebody remembers. The pure fraction is therefore the fraction of a flow that can be protected continuously, and the rest is protected by review and by the structural tests of Section 6.
8. The Two Layers
Where the line falls in a flow
9. A Flow Shaped the Other Way
The 60% is a property of how the runner was written, and it is worth seeing the alternative to know what to avoid:
import inspect
MONOLITH = '''
def run_test(sim, test, seed):
import subprocess
from pathlib import Path
rd = Path("runs") / f"{test}_{seed}"
rd.mkdir(parents=True, exist_ok=True)
cmd = [sim, "-f", "files.f", f"+UVM_TESTNAME={test}", f"+seed={seed}"]
p = subprocess.run(cmd, stdout=subprocess.PIPE, text=True)
(rd / "sim.log").write_text(p.stdout, encoding="utf-8")
errors = sum(1 for l in p.stdout.splitlines() if l.startswith("UVM_ERROR"))
if p.returncode < 0:
status = "KILLED"
elif errors:
status = "FAIL"
elif "TEST DONE" not in p.stdout:
status = "NO_FINISH"
else:
status = "PASS"
print(f"| {test} | {seed} | {status} | {errors} |")
return 1 if status != "PASS" else 0
'''
lines = [l for l in MONOLITH.strip().splitlines()]
print("one function, doing everything:")
print(" lines :", len(lines))
print(" functions :", 1)
print(" pure functions :", 0)
print(" pure lines :", 0)
print(" pure fraction :", "0%")
print()
DECISIONS = ["which directory", "what command", "what the log means",
"what the status is", "how to render it", "what to exit with"]
print("decisions inside it:", len(DECISIONS))
for d in DECISIONS:
print(" -", d)
print()
print("tests possible without a tool:", 0)
print("the same code, same behaviour, 0% testable instead of 62%")one function, doing everything:
lines : 19
functions : 1
pure functions : 0
pure lines : 0
pure fraction : 0%
decisions inside it: 6
- which directory
- what command
- what the log means
- what the status is
- how to render it
- what to exit with
tests possible without a tool: 0
the same code, same behaviour, 0% testable instead of 62%Identical behaviour, zero per cent testable. Six decisions are present in both versions; in this one every single one of them is reachable only by launching a simulator.
That is the shape most flow code starts in, and it is why the canonical purpose's "thin" is an instruction rather than an observation. The refactor from this version to Section 4's is mechanical — extract each decision as a function over its inputs — and it does not change what the program does.
10. What “Pure” Does Not Mean
The split in this chapter rests on a word, and the word is slipperier than it
looks. A function with no I/O, no subprocess and no clock can still return
different answers for the same argument:
THRESHOLD = 100
def over_budget(cycles):
"""No I/O, no clock, no randomness. Is it pure?"""
return cycles > THRESHOLD
print("over_budget(150) :", over_budget(150))
THRESHOLD = 200 # somebody else's module-level constant
print("after THRESHOLD is rebound:")
print("over_budget(150) :", over_budget(150))
print()
print("same argument, different answer :", True)
print("arguments the function declares :", 1)
print("inputs the function actually has :", 2)
print("the second one is :", "a module global — invisible at the call site")over_budget(150) : True
after THRESHOLD is rebound:
over_budget(150) : False
same argument, different answer : True
arguments the function declares : 1
inputs the function actually has : 2
the second one is : a module global — invisible at the call siteOne declared argument, two actual inputs. Nothing here touches a file or a process, so the function passes every informal test for purity, and it still fails the only test that matters: the same input did not produce the same output.
That is not a contrived example. A threshold read from a module-level constant is how almost every flow starts, and the constant gets set from a config file at import time — so the function's answer depends on which config was loaded, which is exactly the kind of dependency a test needs to control and cannot see.
The working definition is therefore narrower than "no side effects":
CRITERIA = [
("returns the same output for the same arguments", "yes", "the real test"),
("writes no file, spawns no process", "yes", "necessary, not sufficient"),
("reads no module-level mutable state", "yes", "the one people miss"),
("calls no clock, no random, no environ", "yes", "all hidden inputs"),
("is short", "no", "length is unrelated"),
("has no loops", "no", "unrelated"),
("takes only primitives", "no", "a dataclass argument is fine"),
]
print(f"{'criterion':<48} {'required':<10} note")
for c, req, note in CRITERIA:
print(f"{c:<48} {req:<10} {note}")
required = sum(1 for _, r, _ in CRITERIA if r == "yes")
print()
print("required criteria:", required, "| irrelevant:", len(CRITERIA) - required)
print("the single question:", "same arguments -> same result, always?")criterion required note
returns the same output for the same arguments yes the real test
writes no file, spawns no process yes necessary, not sufficient
reads no module-level mutable state yes the one people miss
calls no clock, no random, no environ yes all hidden inputs
is short no length is unrelated
has no loops no unrelated
takes only primitives no a dataclass argument is fine
required criteria: 4 | irrelevant: 3
the single question: same arguments -> same result, always?Four criteria that matter and three that do not. The bottom three are worth listing because they are the ones people use as proxies: a long function with loops over a list of dataclasses can be perfectly pure, and a one-line function reading a global is not.
The fix is the one this chapter already argued for, applied one level down — make the hidden input an argument:
from dataclasses import dataclass
@dataclass(frozen=True)
class Budget:
cycles: int = 100
def over_budget(cycles, budget):
"""Both inputs are now visible at the call site."""
return cycles > budget.cycles
print("over_budget(150, Budget(100)):", over_budget(150, Budget(100)))
print("over_budget(150, Budget(200)):", over_budget(150, Budget(200)))
print()
print("same arguments -> same result:", True)
print("testable without any setup :", True)
print("the config is now :", "an argument the caller supplies")
print("and the default lives :", "in one frozen place")over_budget(150, Budget(100)): True
over_budget(150, Budget(200)): False
same arguments -> same result: True
testable without any setup : True
the config is now : an argument the caller supplies
and the default lives : in one frozen placeTwo calls, two answers, and the function is pure — because the thing that
changed is now written at the call site rather than hidden in a module. A test
constructs the Budget it wants and needs no setUp, no monkeypatching and
no import-order care, which is the same argument this chapter made about the
runner's seam arriving at the smallest possible scale.
frozen=True is doing a specific job in that snippet. A mutable config object
passed as an argument is still a shared input, so a test that modifies one
leaks into the next — Chapter 33.7's isolation problem in miniature.
11. Predict, Run, Observe
from pathlib import Path
def parse_text(text):
return len([l for l in text.splitlines() if l.startswith("UVM_ERROR")])
def parse_path(path):
return parse_text(Path(path).read_text(encoding="utf-8"))
# A
print("A:", parse_text("UVM_ERROR a\nUVM_ERROR b\n"))
# B
try:
parse_path("missing.log")
print("B: no error")
except FileNotFoundError:
print("B: FileNotFoundError")
# C
print("C:", 30, 48, f"{100 * 30 // 48}%", f"{100 * 30 / 48:.1f}%")
# D
print("D:", "what does this function read?")The verified output:
A: 2
B: FileNotFoundError
C: 30 48 62% 62.5%
D: what does this function read?A — the text version is tested with a literal and no filesystem.
B — the path version cannot be called at all without a file.
C — the pure fraction from Section 4, recomputed two ways. Integer division reports 62 and the float 62.5, which is a reminder to say which you used when a number reaches a report.
D — the one question that sorts every function.
12. Exercises
Exercise 1 — ask the question. For ten functions in your flow, answer "what does this read?" and sort them.
Exercise 2 — the symmetry. Explain why the same question works for Chapter 32.5's tool commands and for Python functions.
Exercise 3 — your annotation. Write the two-comment version of the annotation for your own runner.
Exercise 4 — "in isolation". Say what that phrase adds to "not testable".
Exercise 5 — count your functions. Report pure and impure counts for one module of your flow.
Exercise 6 — count your lines. Report the pure fraction, and say whether you used integer or float division.
Exercise 7 — where are the decisions? List the decisions in your runner and say which function each lives in.
Exercise 8 — the four things. Confirm that your impure function only makes directories, builds commands, launches and writes.
Exercise 9 — one parameter. Convert one f(path) into f_text(text) plus a two-line wrapper.
Exercise 10 — five seams. Apply the Section 5 table to five impure inputs in your own code.
Exercise 11 — the clock. Find a function that reads the current time and give it a timestamp parameter.
Exercise 12 — the environment. Do the same for os.environ.
Exercise 13 — the subprocess seam. Confirm your classifier takes a returncode rather than making a call.
Exercise 14 — hard to test, hard to find. Pick a decision in your flow that is hard to locate and say whether it is also hard to test.
Exercise 15 — five assertions, no tool. Write them for your own tool-calling layer.
Exercise 16 — the residue. Count the lines in your flow that genuinely cannot be tested without a tool.
Exercise 17 — list what is not covered. Write the honest second column for your own structural tests.
Exercise 18 — the Tcl parallel. Explain how Section 6 relates to Chapter 32.5 Section 9's proc stubs.
Exercise 19 — a boolean, not a duration. Time your suite and report it as a bound rather than a number.
Exercise 20 — what the speed permits. Say what your team would do differently if the suite ran on every save.
Exercise 21 — the monolith. Find a function in your flow that does all six things, and count its decisions.
Exercise 22 — refactor one. Extract one decision from it and write the test that is now possible.
Exercise 23 — zero to sixty. Explain why the refactor does not change behaviour.
Exercise 24 — an instruction, not an observation. Write the one-sentence rule you would put in your project's style guide.
Exercise 25 — the slippery word. Write your own one-sentence definition of pure, and check it against the module-global example.
Exercise 26 — one argument, two inputs. Find a function of yours that reads a module-level constant.
Exercise 27 — the config at import time. Say how that constant gets its value in your flow, and what a test would have to do to control it.
Exercise 28 — four and three. Classify three of your functions using the seven criteria.
Exercise 29 — a long pure function. Find or write one, and say why length was never the question.
Exercise 30 — make it an argument. Convert one hidden input into a parameter with a frozen default object.
Exercise 31 — why frozen. Explain what goes wrong if the config object passed between tests is mutable.
13. Summary
- One question sorts every function: what does it read? Its arguments means testable; the filesystem, a process, the clock or the environment means not — the same test Chapter 32.5 applied to tool commands.
- "Not testable in isolation" is narrower than "untestable" — Section 6 asserts five things about the tool-calling layer with no tool.
- "Thin" is a claim with a number. On the measured runner: 7 of 8 functions and 62% of the lines are pure, and all 18 lines needing a tool are in one function.
- The impure function makes no decisions — it creates a directory, builds a command, launches it and writes a file. Every decision lives in a function over plain data.
- One parameter moves a function across the line.
parse_log(path)needs a filesystem;parse_log_text(text)needs a string literal. - Move the reading out and leave the deciding in, and do it for paths, the clock, the environment, subprocess results and directory listings.
- A flow that is hard to test is usually one whose decisions are hard to find, and the two problems have one fix.
- Two hundred assertions ran inside a second, reported as a boolean against a wide bound rather than a duration, per Chapter 27's rule.
- The pure fraction is the fraction of a flow that can be protected continuously — a sub-second suite runs on every save; a forty-minute one runs when somebody remembers.
- The same six decisions in a monolithic function give a 0% pure fraction with identical behaviour, which is why "thin" is an instruction rather than an observation.
- The refactor is mechanical and changes nothing the program does.
- “Pure” does not mean “no I/O”. A function with no files, no processes and no clock can still return different answers for the same argument if it reads a module-level constant.
- Four criteria matter and three common proxies do not — length, loops and primitive-only arguments are all unrelated to purity.
- The only real test is: same arguments, same result, always.
- Make the hidden input an argument, with a
frozen=Truedefault object so a test cannot leak config into the next one.
The next chapter writes the test the whole module is built on: a fixed input, an expected result, an assertion.
Where this fits
Part of the Python curriculum.