Python Part II · Module 34 · From Script to Maintainable Tool
The Script That Outgrew Itself
Every team has one: the script that runs the regression, that one person wrote, that nobody will touch. It started at twenty lines and it works — which is the problem, because working is the only evidence anyone has that it is correct. This chapter reads one in full and measures it rather than describing it, and the measurements turn up something nobody on that team knows: it throws away every result it collects, writes an empty report, and reports success.
Intermediate24 min readPythonPart IIAutomationRefactoringCode Structure
Part II · Module 34 · Chapter 34.1 · The Script That Outgrew Itself
1. The Script Everyone Avoids
SIGNS = [
("it works", "nobody wants to change it", "working is the only evidence"),
("one person wrote it", "only they can change it", "no shared model of it"),
("it has no tests", "every change is a gamble", "33.1's argument, ignored"),
("it prints its results", "no other tool can use them", "§8"),
("it always exits 0", "CI cannot tell it failed", "§8"),
("paths are hard-coded", "it runs on one machine", "§5"),
]
print(f"{'what is true of it':<24} {'what follows':<34} why")
for a, b, c in SIGNS:
print(f"{a:<24} {b:<34} {c}")
print()
print("signs that are about the CODE :", 3)
print("signs that are about the TEAM :", 3)
print("and the two reinforce each other:", True)what is true of it what follows why
it works nobody wants to change it working is the only evidence
one person wrote it only they can change it no shared model of it
it has no tests every change is a gamble 33.1's argument, ignored
it prints its results no other tool can use them §8
it always exits 0 CI cannot tell it failed §8
paths are hard-coded it runs on one machine §5
signs that are about the CODE : 3
signs that are about the TEAM : 3
and the two reinforce each other: TrueThe first row is the one that traps a team. A script that produces plausible output every morning has accumulated months of evidence that it is fine, and none of that evidence is about its correctness — it is about its output looking the way yesterday's did. Section 6 is what that costs.
2. The Annotation
# hard-coded paths, one function, printed results, no exit code, no testsFive defects, named in advance. The rest of this chapter checks the specimen against that list and finds all five, plus four the annotation does not mention.
Here is the script. It is the module's running example, and Chapters 34.2 through 34.7 take it apart in that order:
#!/usr/bin/env python
# run_regress.py - run the axi_fifo regression
# azhar, march 2024. do not delete, dv flow depends on this
import os, sys, re, subprocess, glob, time
PROJ = "/proj/chip_a/users/azhar/axi_fifo"
RUNS = "/proj/chip_a/users/azhar/axi_fifo/runs"
RTL = "/proj/chip_a/users/azhar/axi_fifo/rtl"
TB = "/proj/chip_a/users/azhar/axi_fifo/tb"
FILELIST = "/proj/chip_a/users/azhar/axi_fifo/rtl/files.f"
SIM = "/tools/cadence/xcelium/19.09/bin/xrun"
LICENSE = "5280@lic-server-3"
TESTS = ["axi_burst_test", "axi_lite_test", "axi_wrap_test", "axi_err_test"]
SEEDS = [4121, 7734, 9001]
UVM_HOME = "/tools/cadence/xcelium/19.09/tools/uvm/uvm-1.2"
print("=== axi_fifo regression ===")
os.environ["LM_LICENSE_FILE"] = LICENSE
os.environ["UVM_HOME"] = UVM_HOME
def run_everything():
total = 0
passed = 0
failed = 0
for t in TESTS:
for s in SEEDS:
total = total + 1
rundir = RUNS + "/" + t + "_" + str(s)
if not os.path.isdir(rundir):
os.makedirs(rundir)
log = rundir + "/sim.log"
cmd = SIM + " -f " + FILELIST + " +UVM_TESTNAME=" + t + \
" +ntb_random_seed=" + str(s) + " -l " + log
print("running " + t + " seed " + str(s))
p = subprocess.Popen(cmd, shell=True, cwd=rundir,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out, err = p.communicate()
if not os.path.exists(log):
print(" no log!")
failed = failed + 1
continue
f = open(log)
text = f.read()
f.close()
errs = 0
fatals = 0
warns = 0
tname = ""
seed = 0
for line in text.split("\n"):
if line[0:9] == "UVM_ERROR":
if line.find("UVM_ERROR :") == -1:
errs = errs + 1
if line[0:9] == "UVM_FATAL":
if line.find("UVM_FATAL :") == -1:
fatals = fatals + 1
if line[0:11] == "UVM_WARNING":
warns = warns + 1
m = re.search("UVM_TESTNAME=(\\S+)", line)
if m:
tname = m.group(1)
m2 = re.search("ntb_random_seed=(\\d+)", line)
if m2:
seed = int(m2.group(1))
if text.find("TEST DONE") == -1:
print(" TIMEOUT or crash")
failed = failed + 1
elif fatals > 0:
print(" FATAL")
failed = failed + 1
elif errs > 0:
print(" FAIL with " + str(errs) + " errors")
failed = failed + 1
else:
print(" PASS")
passed = passed + 1
print("")
print("total " + str(total))
print("passed " + str(passed))
print("failed " + str(failed))
print("pass rate " + str(100 * passed / total) + "%")
for d in glob.glob(RUNS + "/*"):
for junk in glob.glob(d + "/*.tmp"):
os.remove(junk)
run_everything()
print("=== done ===")
RESULTS = []
def report(rows, out=[]):
out.append("test,seed,status,errors")
for r in rows:
out.append(r["test"] + "," + str(r["seed"]) + "," +
r["status"] + "," + str(r["errors"]))
f = open(RUNS + "/summary.csv", "w")
f.write("\n".join(out))
f.close()
print("wrote " + RUNS + "/summary.csv")
html = "<html><body><table>"
for r in rows:
if r["status"] == "PASS":
html = html + "<tr bgcolor=green>"
else:
html = html + "<tr bgcolor=red>"
html = html + "<td>" + r["test"] + "</td><td>" + str(r["seed"]) + \
"</td><td>" + r["status"] + "</td></tr>"
html = html + "</table></body></html>"
f2 = open(RUNS + "/summary.html", "w")
f2.write(html)
f2.close()
print("wrote " + RUNS + "/summary.html")
try:
subprocess.call("mail -s 'axi_fifo regression' dv-team@example.com < " +
RUNS + "/summary.csv", shell=True)
except:
pass
worst = None
for r in rows:
if r["status"] != "PASS":
if worst is None:
worst = r
elif r["errors"] > worst["errors"]:
worst = r
if worst:
print("worst: " + worst["test"] + " seed " + str(worst["seed"]))
log = RUNS + "/" + worst["test"] + "_" + str(worst["seed"]) + "/sim.log"
f3 = open(log)
t = f3.read()
f3.close()
n = 0
for line in t.split("\n"):
if line[0:9] == "UVM_ERROR":
if line.find("UVM_ERROR :") == -1:
n = n + 1
print(" " + line[0:100])
if n > 5:
break
def check_regressions():
old = RUNS + "/summary_last.csv"
if not os.path.exists(old):
print("no baseline")
return
f = open(old)
prev = {}
for line in f.read().split("\n")[1:]:
if line.strip() == "":
continue
p = line.split(",")
prev[p[0] + "_" + p[1]] = p[2]
f.close()
for r in RESULTS:
k = r["test"] + "_" + str(r["seed"])
if k in prev:
if prev[k] == "PASS" and r["status"] != "PASS":
print("NEW FAILURE: " + k)
if prev[k] != "PASS" and r["status"] == "PASS":
print("FIXED: " + k)
os.rename(RUNS + "/summary.csv", old)
report(RESULTS)
check_regressions()
print("regression finished at " + time.ctime())Read it before going on. It is not a strawman — it compiles, it has a sensible
structure in outline (run, report, compare), its parsing handles the
UVM_ERROR : summary line that Chapter 24.3 warned about, and somebody
competent wrote it under time pressure. That is what makes it worth measuring.
3. Measured, Not Described
Calling a script "messy" invites disagreement. ast replaces the adjective
with counts: it parses Python into a tree you can walk, it ships with the
interpreter, and it never runs the code it reads — which matters here, because
the specimen calls xrun, emails the DV team, and writes under /proj.
Everything the rest of this chapter claims about run_regress.py is measured
by this one program, reading the file above:
import ast
import re
from pathlib import Path
src = Path("run_regress.py").read_text(encoding="utf-8")
tree = ast.parse(src)
lines = src.split("\n")
funcs = [n for n in tree.body if isinstance(n, ast.FunctionDef)]
WORLD = {"open", "makedirs", "remove", "rename", "Popen", "call", "system",
"isdir", "exists", "glob", "ctime", "communicate", "write", "read",
"close", "print"}
def world_calls(node):
n = 0
for sub in ast.walk(node):
if isinstance(sub, ast.Call):
name = (sub.func.attr if isinstance(sub.func, ast.Attribute)
else sub.func.id if isinstance(sub.func, ast.Name)
else "?")
if name in WORLD:
n += 1
return n
def branches(node):
return sum(1 for sub in ast.walk(node)
if isinstance(sub, (ast.If, ast.For, ast.While, ast.Try)))
print("== size and shape")
print(" lines :", len(lines))
print(" code lines :", sum(1 for l in lines if l.strip()
and not l.strip().startswith("#")))
print(" functions :", len(funcs))
print(" classes :", sum(1 for n in ast.walk(tree)
if isinstance(n, ast.ClassDef)))
print(" branch/loop statements :", branches(tree))
print()
print("== the annotation's five defects")
# A naive "looks absolute" regex also catches concatenation FRAGMENTS like
# "/sim.log", so report both and keep only strings with a real root segment.
naive = re.findall(r'"(/[A-Za-z0-9/._@-]+)"', src)
rooted = [x for x in naive if x.count("/") > 1]
print(" path-like literals :", len(naive))
print(" distinct, naively :", len(set(naive)))
print(" distinct with a real root :", len(set(rooted)))
print(" fragments the naive count :",
sorted(set(naive) - set(rooted)))
print(" print() calls :", src.count("print("))
print(" sys.exit calls :", src.count("sys.exit"))
print(" `if __name__` guard :", "__name__" in src)
print(" module-level calls :", sum(1 for n in tree.body
if isinstance(n, ast.Expr)
and isinstance(n.value, ast.Call)))
print()
print("== four it does not name")
print(" mutable default arguments :", sum(1 for f in funcs
for d in f.args.defaults
if isinstance(d, (ast.List,
ast.Dict,
ast.Set))))
print(" bare `except:` :", len(re.findall(r'except:\s*$', src,
re.M)))
print(" open() outside a with-block:", src.count("open(")
- src.count("with open("))
print(" shell=True subprocesses :", src.count("shell=True"))
print(" UPPER-CASE module globals :", len([n for n in tree.body
if isinstance(n, ast.Assign)
and isinstance(n.targets[0],
ast.Name)
and n.targets[0].id.isupper()]))
print(" UVM_ERROR parse written :",
src.count('line[0:9] == "UVM_ERROR"'), "times")
print()
print("== per function")
print(" %-20s %5s %9s %5s %8s %6s %9s"
% ("name", "lines", "branches", "args", "returns", "world", "testable"))
for f in funcs:
span = f.end_lineno - f.lineno + 1
rets = sum(1 for n in ast.walk(f)
if isinstance(n, ast.Return) and n.value)
w = world_calls(f)
print(" %-20s %5d %9d %5d %8d %6d %9s"
% (f.name, span, branches(f), len(f.args.args), rets, w,
rets > 0 and w == 0))
print(" functions callable in a test:",
sum(1 for f in funcs if world_calls(f) == 0
and any(isinstance(n, ast.Return) and n.value
for n in ast.walk(f))))
print(" lines inside functions :",
sum(f.end_lineno - f.lineno + 1 for f in funcs))
print()
print("== the RESULTS global")
stores = [n.lineno for n in ast.walk(tree) if isinstance(n, ast.Name)
and n.id == "RESULTS" and isinstance(n.ctx, ast.Store)]
loads = [n.lineno for n in ast.walk(tree) if isinstance(n, ast.Name)
and n.id == "RESULTS" and isinstance(n.ctx, ast.Load)]
muts = [(n.attr, n.lineno) for n in ast.walk(tree)
if isinstance(n, ast.Attribute) and isinstance(n.value, ast.Name)
and n.value.id == "RESULTS"]
print(" assigned at lines :", stores)
print(" read at lines :", loads)
print(" appended to / mutated :", muts)
print()
print("== distinct rooted paths, categorised")
for pth in sorted(set(rooted)):
kind = ("one engineer's scratch directory"
if pth.startswith("/proj/chip_a/users/") else
"one tool install, pinned version")
print(" %-50s %s" % (pth, kind))== size and shape
lines : 166
code lines : 154
functions : 3
classes : 0
branch/loop statements : 37
== the annotation's five defects
path-like literals : 16
distinct, naively : 11
distinct with a real root : 7
fragments the naive count : ['/sim.log', '/summary.csv', '/summary.html', '/summary_last.csv']
print() calls : 21
sys.exit calls : 0
`if __name__` guard : False
module-level calls : 6
== four it does not name
mutable default arguments : 1
bare `except:` : 1
open() outside a with-block: 6
shell=True subprocesses : 2
UPPER-CASE module globals : 11
UVM_ERROR parse written : 2 times
== per function
name lines branches args returns world testable
run_everything 65 17 0 0 22 False
report 48 13 2 0 14 False
check_regressions 21 7 0 0 8 False
functions callable in a test: 0
lines inside functions : 134
== the RESULTS global
assigned at lines : [90]
read at lines : [154, 163]
appended to / mutated : []
== distinct rooted paths, categorised
/proj/chip_a/users/azhar/axi_fifo one engineer's scratch directory
/proj/chip_a/users/azhar/axi_fifo/rtl one engineer's scratch directory
/proj/chip_a/users/azhar/axi_fifo/rtl/files.f one engineer's scratch directory
/proj/chip_a/users/azhar/axi_fifo/runs one engineer's scratch directory
/proj/chip_a/users/azhar/axi_fifo/tb one engineer's scratch directory
/tools/cadence/xcelium/19.09/bin/xrun one tool install, pinned version
/tools/cadence/xcelium/19.09/tools/uvm/uvm-1.2 one tool install, pinned versionThat is the whole chapter's evidence in one run, and the annotation's five
defects are all in it: sixteen path-like literals, three functions whose shape
Section 4 examines, 21 print calls, sys.exit appearing 0 times, and —
Section 10 — zero functions a test could call.
The path count is reported three ways on purpose, because the obvious
measurement is wrong. A regex for "a quoted string starting with a slash"
finds eleven distinct values, four of which are not paths at all:
/sim.log, /summary.csv, /summary.html and /summary_last.csv are
fragments the script concatenates onto a root. Requiring a second slash
leaves seven genuine rooted paths. The naive number would have overstated
the problem by more than half, and this is the module's own lesson applied to
its own instruments: a count is only better than an adjective if the count is
right.
Four more it does not name are there too: a mutable default argument, a bare
except, six open calls outside a with, and two shell=True
invocations. And one it could not have named is the RESULTS block, which is
Section 6.
4. Three Functions, Thirty-Seven Branches
The per-function table in Section 3 is the one to read twice:
name lines branches args returns world testable
run_everything 65 17 0 0 22 False
report 48 13 2 0 14 False
check_regressions 21 7 0 0 8 FalseThe annotation says "one function" and three is close enough, because the count that matters is not how many functions exist but how many jobs each one owns:
JOBS = {
"run_everything": ["build a run-directory path", "create the directory",
"build the simulator command line", "launch the tool",
"wait for it", "check the log exists", "read the log",
"count severities", "extract the test name and seed",
"classify the run", "tally the counters",
"print each verdict", "compute the pass rate",
"delete temporary files"],
"report": ["build CSV text", "write the CSV", "build HTML",
"write the HTML", "send mail", "find the worst failure",
"re-read that run's log", "re-count its errors",
"print the first few"],
"check_regressions": ["find the baseline", "parse the baseline CSV",
"compare each row", "print new failures",
"print fixes", "rotate the baseline file"],
}
print(f"{'function':<20} {'jobs':>5} {'lines':>6} {'lines per job':>14}")
LINES = {"run_everything": 65, "report": 48, "check_regressions": 21}
for name, jobs in JOBS.items():
print(f"{name:<20} {len(jobs):>5} {LINES[name]:>6} "
f"{LINES[name] / len(jobs):>14.1f}")
print()
print("total distinct jobs in 134 lines:", sum(len(j) for j in JOBS.values()))
print("deepest nesting in run_everything:", 6)
print()
print("to change the severity-counting rule you must edit a function that:")
for j in JOBS["run_everything"][:5]:
print(" -", j)function jobs lines lines per job
run_everything 14 65 4.6
report 9 48 5.3
check_regressions 6 21 3.5
total distinct jobs in 134 lines: 29
deepest nesting in run_everything: 6
to change the severity-counting rule you must edit a function that:
- build a run-directory path
- create the directory
- build the simulator command line
- launch the tool
- wait for itTwenty-nine distinct jobs across three functions, and run_everything
owns fourteen of them nested six deep. That is what makes it unchangeable
rather than merely long: the severity-counting rule lives inside a function
that also launches a simulator, so there is no way to exercise a change to it
without a licence, a design, and twenty minutes.
Chapter 33.2's split exists precisely to break that coupling, and Chapter 34.3 applies it here — three layers, one direction of dependency, with the counting in the middle one where a test can reach it.
The args column is the quiet one. Two of the three functions take no
arguments, so they cannot be called with different inputs; they read module
globals instead, which is Chapter 33.2 Section 10's hidden-input problem at the
scale of a whole program. And the returns column is zero across the board —
Section 6 is what that costs.
5. Seven Machine Facts, Sixteen Edits
Section 3 reported both numbers, and they say different things:
LITERALS, DISTINCT = 16, 7
FACTS = [
("a username", "azhar", "nobody else can run it, including a teammate"),
("a project", "chip_a", "a second project cannot use it"),
("a block", "axi_fifo", "a second block cannot use it"),
("a scratch root", "/proj/.../users", "no shared location at all"),
("a tool vendor", "cadence", "one simulator only"),
("a tool version", "19.09", "breaks the day the site upgrades"),
("a licence server", "5280@lic-server-3", "one site only"),
]
print(f"{'what is baked in':<18} {'value':<22} consequence")
for what, val, why in FACTS:
print(f"{what:<18} {val:<22} {why}")
print()
print("rooted paths (facts about one machine):", DISTINCT)
print("path-like literals (edits to make) :", LITERALS)
print("the gap :", LITERALS - DISTINCT)
print("the gap exists because :",
"PROJ is spelled out again instead of composed from")
print()
print("a second team can use this script :", False)
print("the author on another machine can :", False)
print("the failure when the site upgrades :",
"looks like the simulator vanished")what is baked in value consequence
a username azhar nobody else can run it, including a teammate
a project chip_a a second project cannot use it
a block axi_fifo a second block cannot use it
a scratch root /proj/.../users no shared location at all
a tool vendor cadence one simulator only
a tool version 19.09 breaks the day the site upgrades
a licence server 5280@lic-server-3 one site only
rooted paths (facts about one machine): 7
path-like literals (edits to make) : 16
the gap : 9
the gap exists because : PROJ is spelled out again instead of composed from
a second team can use this script : False
the author on another machine can : False
the failure when the site upgrades : looks like the simulator vanishedSeven facts and sixteen edits. Seven is how much of one machine is baked
in; sixteen is how many string literals mention a path at all, and the nine-edit gap exists purely
because PROJ is written out in full five more times rather than being
composed from — RUNS = PROJ + "/runs" was available and was not used.
The username is the detail that makes this unfixable by goodwill.
/proj/chip_a/users/azhar is not a shared location, so the author's own
teammate cannot run it, and the licence server pins it to one site. Chapter
34.5 is entirely about moving these seven facts into configuration, and its
annotation is the shape they move into:
RUN_ROOT = Path(os.environ.get("FLOW_RUN_ROOT", ROOT / "runs"))Note what the tool paths pin that the others do not: a version.
xcelium/19.09 means the script breaks on upgrade day, and the symptom —
a path that no longer exists — reads as the simulator having vanished rather
than as a stale constant.
6. The Bug the Structure Hid
Everything so far is a style judgement a reasonable engineer could argue with. This is not. From Section 3:
== the RESULTS global
assigned at lines : [90]
read at lines : [154, 163]
appended to / mutated : []RESULTS is assigned an empty list and never appended to, and it is the
list both report and check_regressions work from. Trace the consequences:
RESULTS = [] # exactly what the specimen has at line 90
rows = RESULTS # exactly what report(RESULTS) receives
csv_lines = ["test,seed,status,errors"]
for r in rows:
csv_lines.append("...")
html_rows = [r for r in rows]
worst = None
for r in rows:
if r.get("status") != "PASS":
worst = r
print("simulations the script runs :", 4 * 3)
print("rows reaching report() :", len(rows))
print("lines in summary.csv :", len(csv_lines), "(the header)")
print("rows in the HTML table :", len(html_rows))
print("the 'worst failure' it reports :", worst)
print("baseline comparisons it performs :", len(rows))
print("NEW FAILURE lines it can print :", 0)
print()
print("what the terminal shows :", "a correct per-run summary")
print("because those counters are :", "local to run_everything")
print("and the exit code is :", 0)simulations the script runs : 12
rows reaching report() : 0
lines in summary.csv : 1 (the header)
rows in the HTML table : 0
the 'worst failure' it reports : None
baseline comparisons it performs : 0
NEW FAILURE lines it can print : 0
what the terminal shows : a correct per-run summary
because those counters are : local to run_everything
and the exit code is : 0Twelve simulations run and zero results recorded. The script emails a CSV
containing one header line, writes an HTML file with an empty table, never
identifies a worst failure, and compares nothing against the baseline — so
Chapter 30.3's new-versus-fixed detection, which is the whole reason
check_regressions exists, has never once produced a line of output.
And nobody noticed, for a specific and instructive reason: run_everything
keeps its tallies in local variables and prints them. The terminal output is
correct. total 12, passed 9, failed 3, and a per-run PASS/FAIL for
each — all accurate, all derived from counters that are discarded when the
function returns. The structured outputs, which nobody reads because the
terminal already told them what they wanted, are empty.
The structural cause is worth naming precisely, because it is what Chapter 34.2
fixes. run_everything returns nothing — its signature is
def run_everything(): with no return — so the only way for it to
communicate is a side effect, and the side effect it was supposed to perform
(appending to RESULTS) was never written. A function that returned its rows
could not have had this bug: the caller would have had nothing to pass on.
7. Two Defects Visible From the Signature
def report(rows, out=[]):
"""The specimen's signature, exactly. `out` is the defect."""
out.append("test,seed,status,errors")
for r in rows:
out.append(r["test"])
return out
FIRST = [{"test": "axi_burst_test"}]
SECOND = [{"test": "axi_lite_test"}]
print("call 1:", report(FIRST))
print("call 2:", report(SECOND))
print("call 3:", report([]))
print()
final = report([])
print("lines the 4th call would write:", len(final))
print("header lines among them :", final.count("test,seed,status,errors"))
print("the default list is created :", "once, when def runs — not per call")
print("so a second regression in one process appends to the first's report:",
True)call 1: ['test,seed,status,errors', 'axi_burst_test']
call 2: ['test,seed,status,errors', 'axi_burst_test', 'test,seed,status,errors', 'axi_lite_test']
call 3: ['test,seed,status,errors', 'axi_burst_test', 'test,seed,status,errors', 'axi_lite_test', 'test,seed,status,errors']
lines the 4th call would write: 6
header lines among them : 4
the default list is created : once, when def runs — not per call
so a second regression in one process appends to the first's report: TrueFour header lines in one CSV. A default argument is evaluated once, when
the def statement executes, so every call that does not pass out shares the
same list. In the specimen this is dormant because report is called once —
which is the worst kind of latent defect, waiting for the day somebody imports
this module and calls report twice.
The second signature-level defect is arithmetic:
for passed, total in ((9, 12), (0, 12), (12, 12), (1, 3)):
print(f" passed={passed:>3} total={total:>3} "
f"printed as: pass rate {100 * passed / total}%")
print()
print(" the type of 100 * 9 / 12:", type(100 * 9 / 12).__name__)
print(" so a clean run prints :", "pass rate 100.0%")
print(" and 1 of 3 prints :", f"pass rate {100 * 1 / 3}%")
print()
try:
print(100 * 0 / 0)
except ZeroDivisionError as exc:
print(" an empty TESTS list :", type(exc).__name__, "-", exc)
print(" which happens when :", "a filter leaves no tests to run") passed= 9 total= 12 printed as: pass rate 75.0%
passed= 0 total= 12 printed as: pass rate 0.0%
passed= 12 total= 12 printed as: pass rate 100.0%
passed= 1 total= 3 printed as: pass rate 33.333333333333336%
the type of 100 * 9 / 12: float
so a clean run prints : pass rate 100.0%
and 1 of 3 prints : pass rate 33.333333333333336%
an empty TESTS list : ZeroDivisionError - division by zero
which happens when : a filter leaves no tests to runpass rate 33.333333333333336% is what a human reads in the morning, and
Chapter 7.4's formatting is one :.1f away. The ZeroDivisionError is the
sharper problem: an empty test list is a perfectly ordinary thing to produce
with a filter, and the script's response is a traceback after it has already
run the simulations.
8. Printed, Not Returned, and Always Zero
OUTPUTS = [
("the per-run verdict", "print", "a human watching", "no", "correct"),
("the totals", "print", "a human watching", "no", "correct"),
("the pass rate", "print", "a human watching", "no", "correct"),
("summary.csv", "a file", "another script", "yes", "header only"),
("summary.html", "a file", "a browser", "yes", "empty table"),
("the exit code", "none exists", "CI", "yes", "always 0"),
]
print(f"{'what it produces':<22} {'how':<13} {'read by':<18} "
f"{'machine-readable':<17} state")
for what, how, who, mr, state in OUTPUTS:
print(f"{what:<22} {how:<13} {who:<18} {mr:<17} {state}")
correct_and_machine = [o for o in OUTPUTS if o[3] == "yes" and o[4] == "correct"]
print()
print("outputs a machine can consume :", sum(1 for o in OUTPUTS if o[3] == "yes"))
print("of those, ones that are correct :", len(correct_and_machine))
print("outputs that are correct :", sum(1 for o in OUTPUTS if o[4] == "correct"))
print("of those, ones a machine can use:", 0)
print()
print("sys.exit calls in the specimen :", 0, "— measured in §3")
print("so a CI job wrapping it sees :", "success, on every possible outcome")what it produces how read by machine-readable state
the per-run verdict print a human watching no correct
the totals print a human watching no correct
the pass rate print a human watching no correct
summary.csv a file another script yes header only
summary.html a file a browser yes empty table
the exit code none exists CI yes always 0
outputs a machine can consume : 3
of those, ones that are correct : 0
outputs that are correct : 3
of those, ones a machine can use: 0
sys.exit calls in the specimen : 0 — measured in §3
so a CI job wrapping it sees : success, on every possible outcomeThe two sets do not intersect. Three outputs are machine-readable and none of them is correct; three are correct and none of them is machine-readable. That is a complete account of why this script works in use and is unusable in automation, and it is not a coincidence — the correct ones are the ones a human checks every morning, and the broken ones are the ones nobody looks at.
The exit code is the sharpest case because it does not exist at all. Section 3
measured sys.exit appearing zero times, so every outcome — twelve clean runs,
twelve failures, a ZeroDivisionError from Section 7, a missing filelist —
leaves the process exiting 0. A CI job that wraps this script and branches on
$? is branching on a constant.
One small, telling detail sits in the specimen's imports: sys is imported on
line 4 and never used anywhere in 166 lines. Somebody intended to exit with a
status — that import is the fossil of the intention — and the line was never
written. Chapter 33.8's contract is exactly what is missing, and Chapter 34.6's
annotation states it as a promise the next owner can rely on:
"""run_regress.py - run a suite; exit 0 only if every job passed."""9. Importing It Runs It
The specimen has no if __name__ == "__main__": guard, and six calls at module
level. That has one consequence that matters more than all the style points:
import contextlib
import io
from pathlib import Path
Path("unguarded.py").write_text(
'print("=== regression starting ===")\n'
'TESTS = ["axi_burst_test"]\n'
'\n'
'def classify(text):\n'
' return "FAIL" if "UVM_ERROR" in text else "PASS"\n'
'\n'
'print("running", TESTS[0])\n'
'print("=== done ===")\n', encoding="utf-8")
Path("guarded.py").write_text(
'TESTS = ["axi_burst_test"]\n'
'\n'
'def classify(text):\n'
' return "FAIL" if "UVM_ERROR" in text else "PASS"\n'
'\n'
'def main():\n'
' print("=== regression starting ===")\n'
' print("running", TESTS[0])\n'
' print("=== done ===")\n'
' return 0\n'
'\n'
'if __name__ == "__main__":\n'
' import sys\n'
' sys.exit(main())\n', encoding="utf-8")
import sys
sys.path.insert(0, ".")
for name in ("unguarded", "guarded"):
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
mod = __import__(name)
printed = len(buf.getvalue().strip().splitlines())
print(f" import {name:<12} printed {printed} line(s), "
f"classify usable = {mod.classify('UVM_ERROR x') == 'FAIL'}")
print()
print("the guard costs :", "3 lines")
print("what it buys :", "the file becomes importable")
print("and 34.2's first line :", "from flow.parse import parse_sim_log")
print("which is impossible without:", "the guard") import unguarded printed 3 line(s), classify usable = True
import guarded printed 0 line(s), classify usable = True
the guard costs : 3 lines
what it buys : the file becomes importable
and 34.2's first line : from flow.parse import parse_sim_log
which is impossible without: the guardBoth files expose classify; only one can be imported without running the
regression. In the specimen, import run_regress would set two environment
variables, create directories under /proj, launch twelve simulations, and
email the DV team — as a side effect of wanting to reuse one parsing function.
That is why "no tests" in the annotation is not a matter of discipline. A test file begins by importing the module under test, and here the import is the regression. The guard is three lines and it is the entire difference between a script and something another file can use, which is Chapter 13.6's point arriving as a precondition for the whole module.
10. The Testable Surface Is Empty
The last column of Section 3's table is the one this whole module exists to change:
name lines branches args returns world testable
run_everything 65 17 0 0 22 False
report 48 13 2 0 14 False
check_regressions 21 7 0 0 8 False
functions callable in a test: 0CRITERIA = [
("returns a value", "0 of 3", "nothing to assert on"),
("free of world calls", "0 of 3", "calling it does something"),
("takes arguments", "1 of 3", "cannot be driven with test inputs"),
]
print(f"{'what a test needs':<22} {'specimen':<9} why it fails")
for what, got, why in CRITERIA:
print(f"{what:<22} {got:<9} {why}")
print()
print("functions a test could call :", 0)
print("lines inside functions :", 134)
print()
print("33.2 measured a real runner's pure fraction:", "62%")
print("the specimen's :", "0%")
print()
PURE_LOGIC = ["counting UVM_ERROR lines", "excluding the UVM_ERROR : summary",
"classifying FATAL / FAIL / TIMEOUT / PASS",
"computing the pass rate", "choosing the worst failure",
"diffing against the baseline"]
print("pure computations trapped inside impure functions:", len(PURE_LOGIC))
for x in PURE_LOGIC:
print(" -", x)what a test needs specimen why it fails
returns a value 0 of 3 nothing to assert on
free of world calls 0 of 3 calling it does something
takes arguments 1 of 3 cannot be driven with test inputs
functions a test could call : 0
lines inside functions : 134
33.2 measured a real runner's pure fraction: 62%
the specimen's : 0%
pure computations trapped inside impure functions: 6
- counting UVM_ERROR lines
- excluding the UVM_ERROR : summary
- classifying FATAL / FAIL / TIMEOUT / PASS
- computing the pass rate
- choosing the worst failure
- diffing against the baselineZero of three functions can be called from a test, and the reason is not that the logic is inherently impure. Six genuinely pure computations are listed there — the severity counting, the summary-line exclusion that Chapter 24.3 warned about, the four-way classification, the arithmetic, the worst-failure selection, and the baseline diff. Every one of them is a function of its inputs. Every one of them is written inside a function that also launches a simulator or writes a file.
That is the entire thesis of this module. The logic is fine. Its arrangement makes it untestable, unreusable, and — as Section 6 showed — capable of hiding a total failure behind a correct-looking print.
11. The Road Out
One script, six refactorings
Step 1 is the one that is usually skipped, and skipping it is why refactoring arguments go nowhere. "This script is a mess" invites disagreement; "this script has sixteen hard-coded paths, zero testable functions, and a global that is read twice and never written" does not.
Step 4's caption is deliberately conditional. Chapter 34.4's own purpose warns that a premature class costs more than it saves, so that stage can legitimately be skipped — it is the only one of the six that might be.
12. Predict, Run, Observe
import ast
SRC = (
"RESULTS = []\n"
"def collect():\n"
" rows = []\n"
" rows.append({'test': 'axi_burst_test'})\n"
"def emit(rows, out=[]):\n"
" out.append('header')\n"
" return out\n"
"collect()\n"
"emit(RESULTS)\n"
)
tree = ast.parse(SRC)
# A
stores = [n.lineno for n in ast.walk(tree)
if isinstance(n, ast.Name) and n.id == "RESULTS"
and isinstance(n.ctx, ast.Store)]
loads = [n.lineno for n in ast.walk(tree)
if isinstance(n, ast.Name) and n.id == "RESULTS"
and isinstance(n.ctx, ast.Load)]
print("A:", stores, loads)
# B
ns = {}
exec(compile(SRC, "m.py", "exec"), ns)
print("B:", ns["RESULTS"])
# C
print("C:", ns["emit"]([]), ns["emit"]([]))
# D
funcs = [n for n in tree.body if isinstance(n, ast.FunctionDef)]
print("D:", [(f.name, sum(1 for n in ast.walk(f)
if isinstance(n, ast.Return) and n.value))
for f in funcs])The verified output:
A: [1] [9]
B: []
C: ['header', 'header', 'header'] ['header', 'header', 'header']
D: [('collect', 0), ('emit', 1)]A — assigned on line 1, read on line 9, never appended to. Section 6's measurement in four lines, on a nine-line program.
B — and so it is still empty after the program has run. collect built its
rows in a local named rows and dropped them, which is the specimen's bug
exactly.
C — the mutable default, and note it is worse than Section 7 showed: the
module-level emit(RESULTS) call already appended once, so by the time these
two calls run the list has grown to three, and both printed values are the
same object. A reader seeing two identical lists might conclude the function
is deterministic; it is the opposite.
D — collect returns nothing, emit returns something. The only function
a test could assert on is the one that does not do the work.
13. Exercises
Exercise 1 — your own specimen. Find the script on your team that nobody wants to touch, and copy it somewhere safe to work on.
Exercise 2 — working is not evidence. Write down, in one sentence, what months of plausible output actually prove about it.
Exercise 3 — code versus team. Sort Section 1's six signs for your script into the ones about the code and the ones about the team.
Exercise 4 — the five defects. Check your specimen against the annotation's list and mark which of the five it has.
Exercise 5 — measure it. Run Section 3's measure function on your script
and record all fifteen numbers.
Exercise 6 — adjectives into counts. Rewrite one complaint about it as a number.
Exercise 7 — jobs per function. For its longest function, list the distinct jobs it owns.
Exercise 8 — nesting depth. Find the deepest nesting and say what it would take to test the innermost branch.
Exercise 9 — no arguments. Count its functions that take no arguments, and say what they read instead.
Exercise 10 — occurrences versus facts. Count both the distinct absolute paths and their total occurrences, and explain why the two numbers differ.
Exercise 11 — the username. Find a path containing a personal directory and say who can and cannot run the script.
Exercise 12 — a pinned version. Find a tool path with a version in it and describe the failure when the site upgrades.
Exercise 13 — a dead global. Walk its AST for a module-level name that is read and never written.
Exercise 14 — trace the consequence. If you find one, work out exactly which outputs are empty.
Exercise 15 — printed versus structured. For your script, say which output is maintained and which is merely present.
Exercise 16 — the returning function. Explain why a function that returned its rows could not have Section 6's bug.
Exercise 17 — a mutable default. Search for =[] or ={} in a signature
and call that function twice.
Exercise 18 — dormant today. Say what change would wake it up.
Exercise 19 — the pass rate. Find a percentage your script prints and check how many digits it shows.
Exercise 20 — the empty list. Work out what it does when a filter leaves nothing to run.
Exercise 21 — machine-readable outputs. List what another script could consume from yours, and whether any of it is correct.
Exercise 22 — the exit code. Count its sys.exit calls, and say what a CI
job wrapping it concludes.
Exercise 23 — a fossil import. Look for an import that is never used, and guess what it was for.
Exercise 24 — import it. Try importing your script in a Python shell and record what happens.
Exercise 25 — the guard. Add the three lines and confirm the import goes quiet while the functions stay reachable.
Exercise 26 — why no tests. Explain in two sentences why "no tests" was not a discipline problem.
Exercise 27 — the testable surface. Run Section 10's measurement and record how many of your functions a test could call.
Exercise 28 — pure logic in an impure place. Name one genuinely pure computation that is trapped inside a function that touches the world.
Exercise 29 — the arrangement, not the logic. Say which of your defects are about the logic being wrong and which are about where it sits.
Exercise 30 — the road out. Decide which of the six stages your script needs, and in what order.
Exercise 31 — the skippable one. Say whether your script needs a class at all, and why.
14. Summary
- "It works" is evidence that the output looks like yesterday's, not that the script is correct — and Section 6 is what that costs.
- Measure instead of describing.
astturns "messy" into fifteen counts, and a count is not arguable. - The specimen is 166 lines: 3 functions of 65, 48 and 21 lines, 37
branches, 16 path occurrences across 7 distinct paths, 11 globals, 21
printcalls, 0sys.exit, no__name__guard. - The annotation's five defects are all present, plus a mutable default, a
bare
except, six unguardedopencalls, and twoshell=Trueinvocations. - What makes a function unchangeable is jobs, not lines.
run_everythingowns ten, so the severity rule cannot be changed without a licence and a design. - Two of three functions take no arguments, so they cannot be called with different inputs — they read globals instead.
- Seven machine facts, sixteen edits. Distinct paths say how much is baked in; occurrences say what it costs to change, and the gap is the absence of composition.
RESULTSis assigned[], read twice, and never appended to. Twelve simulations run, zero rows recorded: a header-only CSV, an empty HTML table, no worst failure, no baseline comparison — and exit 0.- Nobody noticed because the printed summary is correct. The tallies live in locals and are printed; the structured outputs nobody reads are the empty ones.
- A function that returned its rows could not have had that bug — the caller would have had nothing to pass on.
- A default argument is evaluated once, at
deftime, soout=[]is shared across calls; four calls produce four header lines in one file. 100 * passed / totalis a float and prints33.333333333333336%; an empty test list raisesZeroDivisionErrorafter the simulations have run.- Every machine-readable output is empty or absent and every correct one goes to a terminal, which is why the script works in use and is unusable in automation.
sysis imported and never used — the fossil of an exit code somebody meant to add.- Importing the script runs the regression. The guard costs three lines and is the entire difference between a script and something another file can import, so "no tests" was a structural impossibility, not a lapse.
- 0 of 3 functions are callable from a test — none returns a value, all touch the world — against the 62% pure fraction Chapter 33.2 measured on a real runner.
- The logic is fine; the arrangement is the defect. That is what the next six chapters change.
The next chapter does the first and most consequential of those changes: making the file something another file can import.
Where this fits
Part of the Python curriculum.