Python Part II · Module 33 · Testing and Debugging Automation
Debugging an Automation Failure That Only Sometimes Happens
A script that fails every time is a pleasant afternoon's work. A script that fails once in twenty is the one that costs a week, because every technique in this module assumes you can run the thing again and watch it break. The way out is not cleverness — it is to stop debugging the failure and start debugging the difference between the run that failed and the run that did not, and then to make that difference something you control. Python contributes two sources of its own, and one of them changes on every process start.
Advanced24 min readPythonPart IIAutomationDebuggingDeterminism
Part II · Module 33 · Chapter 33.14 · Debugging an Automation Failure That Only Sometimes Happens
1. The Failure That Will Not Hold Still
SHAPES = [
("fails every time", "run it again", "an hour"),
("fails on one machine", "compare the two", "a morning"),
("fails for one user", "compare the environments", "a morning"),
("fails 1 run in 20", "nothing works", "a week"),
("stopped failing", "nothing works, and nobody believes you", "forever"),
]
print(f"{'the report says':<24} {'the technique that works':<40} typical cost")
for shape, tech, cost in SHAPES:
print(f"{shape:<24} {tech:<40} {cost}")
print()
print("what the last two have in common:",
"you cannot run the experiment")
print("so the first job is :",
"not to fix it — to make it reproduce")
print("and every technique in 33.1-33.13 assumes:", "it reproduces")the report says the technique that works typical cost
fails every time run it again an hour
fails on one machine compare the two a morning
fails for one user compare the environments a morning
fails 1 run in 20 nothing works a week
stopped failing nothing works, and nobody believes you forever
what the last two have in common: you cannot run the experiment
so the first job is : not to fix it — to make it reproduce
and every technique in 33.1-33.13 assumes: it reproducesThat last line is worth sitting with. Chapter 33.9's mutation testing needs a suite you can run repeatedly. Chapter 33.4's golden fixtures need byte-stable output. Chapter 33.8's contract tests need an exit code that means the same thing twice. An intermittent failure invalidates the premise of the entire module, and that is why this chapter is last rather than first.
2. The Annotation
# re-run cmd.sh by hand in the same run_dir
print(repr(os.environ.get("PATH")))Those two lines are the first two moves, in that order, and they are chosen to split the problem rather than to attack it.
MOVES = [
("re-run cmd.sh in the same run_dir", "does it fail again?",
"yes -> deterministic, given that state", "no -> the state differed"),
("print repr(os.environ.get('PATH'))", "what did the tool actually see?",
"a difference -> the environment", "identical -> look elsewhere"),
]
print(f"{'the move':<36} {'the question':<32} ")
for move, q, yes, no in MOVES:
print(f"{move:<36} {q:<32}")
print(f"{'':<36} {yes}")
print(f"{'':<36} {no}")
print()
print("what move 1 needs from 33.13:", "cmd.sh, written BEFORE the tool ran")
print("what it would cost without it:", "reconstructing the command line")
print("and that reconstruction is :", "the thing you cannot verify")the move the question
re-run cmd.sh in the same run_dir does it fail again?
yes -> deterministic, given that state
no -> the state differed
print repr(os.environ.get('PATH')) what did the tool actually see?
a difference -> the environment
identical -> look elsewhere
what move 1 needs from 33.13: cmd.sh, written BEFORE the tool ran
what it would cost without it: reconstructing the command line
and that reconstruction is : the thing you cannot verifyMove one is only possible because of the previous chapter. cmd.sh records the working directory, the environment variables and the exact argument vector before the tool launches, so it survives the run that was killed. Without it, "re-run it by hand" means reconstructing a command line from a log — and a reconstruction that fails to reproduce the bug tells you nothing, because you cannot distinguish "the bug is intermittent" from "I rebuilt the command wrong".
That is the whole value of Chapter 33.13 Section 8 arriving a chapter later: the cheapest artefact in the run directory is the one that makes debugging possible at all.
3. Why repr
PATHS = [
("/opt/tools/bin:/usr/bin", "normal"),
("/opt/tools/bin:/usr/bin:", "trailing colon"),
("/opt/tools/bin::/usr/bin", "empty element"),
("/opt/tools/bin :/usr/bin", "trailing space on an element"),
("/opt/tools/bin:/usr/bin\n", "trailing newline"),
]
print("constructed PATH values — the machine's real PATH is not published\n")
for value, label in PATHS:
parts = value.split(":")
cwd_on_path = any(p == "" or p == "." for p in parts)
print(f" {label}")
print(f" print() {value}")
print(f" print(repr()) {value!r}")
print(f" elements {len(parts)} "
f"current dir on PATH: {cwd_on_path}")constructed PATH values — the machine's real PATH is not published
normal
print() /opt/tools/bin:/usr/bin
print(repr()) '/opt/tools/bin:/usr/bin'
elements 2 current dir on PATH: False
trailing colon
print() /opt/tools/bin:/usr/bin:
print(repr()) '/opt/tools/bin:/usr/bin:'
elements 3 current dir on PATH: True
empty element
print() /opt/tools/bin::/usr/bin
print(repr()) '/opt/tools/bin::/usr/bin'
elements 3 current dir on PATH: True
trailing space on an element
print() /opt/tools/bin :/usr/bin
print(repr()) '/opt/tools/bin :/usr/bin'
elements 2 current dir on PATH: False
trailing newline
print() /opt/tools/bin:/usr/bin
print(repr()) '/opt/tools/bin:/usr/bin\n'
elements 2 current dir on PATH: FalseTwo of these five put the current directory on PATH, and a trailing newline is invisible without repr. A PATH ending in a colon has an empty final element, and an empty element means "here" — so a run started from a directory containing a file called iverilog picks that up instead of the real tool, and only when started from that directory.
That is an intermittent failure with a completely deterministic cause: it depends on the working directory, and the working directory depends on how the job was launched.
The trailing-space case is the one that produces the most confusing symptom. /opt/tools/bin with a space is a directory that does not exist, so the tool is simply not found on that element — and the error message says the tool is missing while echo $PATH appears to show it present. Chapter 25.5 Section 3's point about an invisible character applies to every environment variable a flow reads, which is why repr and not print is in the annotation.
4. Debug the Difference, Not the Failure
DIFFERENCES = [
("the environment", "PATH, tool version vars, locale", "cmd.sh exports"),
("the working directory", "relative paths resolve elsewhere", "cmd.sh cd"),
("the inputs", "a filelist that changed under you", "hash the inputs"),
("the seed", "different stimulus", "record it — 33.13"),
("the order of work", "iteration order, completion order", "§5, §6"),
("the clock", "a date-named dir, a timeout", "record the start time"),
("concurrency", "two runs sharing a path", "a unique run_dir"),
("leaked state", "a previous run's file still there", "a fresh dir — 33.7"),
]
print(f"{'what can differ':<24} {'how it shows up':<38} what pins it")
for what, how, pin in DIFFERENCES:
print(f"{what:<24} {how:<38} {pin}")
print()
print("candidate differences:", len(DIFFERENCES))
print("the method :", "make each one identical, one at a time")
print("when they all match :", "the failure is in the code, and reproduces")what can differ how it shows up what pins it
the environment PATH, tool version vars, locale cmd.sh exports
the working directory relative paths resolve elsewhere cmd.sh cd
the inputs a filelist that changed under you hash the inputs
the seed different stimulus record it — 33.13
the order of work iteration order, completion order §5, §6
the clock a date-named dir, a timeout record the start time
concurrency two runs sharing a path a unique run_dir
leaked state a previous run's file still there a fresh dir — 33.7
candidate differences: 8
the method : make each one identical, one at a time
when they all match : the failure is in the code, and reproducesEight candidates, and the reframing is the whole technique: you are not looking for a bug that happens sometimes, you are looking for an input you did not know you had. A failure that occurs on one run in twenty is perfectly deterministic with respect to something; the job is to find the something and put it in cmd.sh.
Row five is the one Python supplies for free, and it is the next two sections.
5. String Hashing Is Randomised Per Process
import subprocess
import sys
from pathlib import Path
Path("order.py").write_text(
'severities = {"UVM_ERROR", "UVM_FATAL", "UVM_WARNING", "UVM_INFO"}\n'
'print(",".join(severities))\n', encoding="utf-8")
RUNS = 20
orders = set()
for _ in range(RUNS):
r = subprocess.run([sys.executable, "order.py"], stdout=subprocess.PIPE,
text=True)
orders.add(r.stdout.strip())
print("the same program, run as", RUNS, "separate processes")
print("distinct iteration orders observed >= 2:", len(orders) >= 2)
print("so set order is stable within a process:", True)
print("and stable between processes :", False)
print()
print("hash randomisation is on by default:",
bool(sys.flags.hash_randomization))the same program, run as 20 separate processes
distinct iteration orders observed >= 2: True
so set order is stable within a process: True
and stable between processes : False
hash randomisation is on by default: TrueThe same program printed different orders on different runs. Python randomises the hash of str and bytes at interpreter start — a deliberate defence against hash-collision denial of service — so any code whose behaviour depends on set iteration order behaves differently from one process to the next.
The specific orders are not published here deliberately: they are different every time this page's examples are verified, which is exactly the property under discussion. What is stable, and therefore worth publishing, is that at least two distinct orders appear in twenty runs.
Dictionaries are not affected:
import subprocess
import sys
from pathlib import Path
Path("dicts.py").write_text(
'd = {"UVM_ERROR": 1, "UVM_FATAL": 2, "UVM_WARNING": 3, "UVM_INFO": 4}\n'
's = {"UVM_ERROR", "UVM_FATAL", "UVM_WARNING", "UVM_INFO"}\n'
'print("dict:", ",".join(d))\n'
'print("set :", ",".join(s))\n', encoding="utf-8")
def collect(n, env=None):
import os
e = dict(os.environ)
if env:
e.update(env)
dicts, sets = set(), set()
for _ in range(n):
r = subprocess.run([sys.executable, "dicts.py"], stdout=subprocess.PIPE,
text=True, env=e)
for line in r.stdout.strip().splitlines():
(dicts if line.startswith("dict:") else sets).add(line)
return dicts, sets
d1, s1 = collect(12)
print("default hash seed, 12 runs")
print(" distinct dict orders :", len(d1))
print(" distinct set orders >= 2 :", len(s1) >= 2)
d2, s2 = collect(12, {"PYTHONHASHSEED": "0"})
print()
print("PYTHONHASHSEED=0, 12 runs")
print(" distinct dict orders :", len(d2))
print(" distinct set orders :", len(s2))
print()
print("a dict preserves insertion order:", "always — 15.4 §2")
print("a set has no order to preserve :", True)
print("PYTHONHASHSEED=0 makes a set :", "reproducible, not sorted")default hash seed, 12 runs
distinct dict orders : 1
distinct set orders >= 2 : True
PYTHONHASHSEED=0, 12 runs
distinct dict orders : 1
distinct set orders : 1
a dict preserves insertion order: always — 15.4 §2
a set has no order to preserve : True
PYTHONHASHSEED=0 makes a set : reproducible, not sortedOne dict order always; one set order only with the seed pinned. That gives two separate practical conclusions and they should not be conflated.
PYTHONHASHSEED=0 is a diagnostic tool. Set it, run the failing job twenty times, and if the failure becomes consistently present or consistently absent you have found the cause. It is not a fix: shipping a flow that depends on it means shipping a flow whose correctness depends on an environment variable nobody will remember.
The fix is to not depend on set order at all. sorted(severities) where an order is needed, a dict or a list where insertion order is meaningful, and a set only where the question really is membership. Chapter 20.4's advice to sort before writing a report is the same rule: if the output has an order, choose it explicitly.
severities = {"UVM_ERROR", "UVM_FATAL", "UVM_WARNING", "UVM_INFO"}
print("a set, iterated :", "order varies per process")
print("a set, sorted :", ",".join(sorted(severities)))
print("a list :", ",".join(["UVM_FATAL", "UVM_ERROR",
"UVM_WARNING", "UVM_INFO"]))
print()
print("which of the three is reproducible:", "the last two")
print("which carries MEANING in its order:", "the list — severity rank")
print("so for a report you want :", "the list, not sorted()")
print("because alphabetical ranks :", ",".join(sorted(severities)))a set, iterated : order varies per process
a set, sorted : UVM_ERROR,UVM_FATAL,UVM_INFO,UVM_WARNING
a list : UVM_FATAL,UVM_ERROR,UVM_WARNING,UVM_INFO
which of the three is reproducible: the last two
which carries MEANING in its order: the list — severity rank
so for a report you want : the list, not sorted()
because alphabetical ranks : UVM_ERROR,UVM_FATAL,UVM_INFO,UVM_WARNINGThe last line is the small trap inside the fix. sorted() makes the order reproducible and alphabetical, which puts UVM_ERROR before UVM_FATAL and UVM_INFO before UVM_WARNING — a severity ordering that is wrong in both pairs. Reproducible and correct are different properties, and reaching for sorted() only buys the first.
6. Directory Order Is Not Sorted Order
import glob
import os
from pathlib import Path
for name in ["c_test", "b_test", "z_test", "a_test", "m_test"]:
Path(name + ".log").write_text("UVM_INFO\n", encoding="utf-8")
listed = [p for p in os.listdir(".") if p.endswith(".log")]
globbed = [os.path.basename(p) for p in glob.glob("*.log")]
print("files created in a deliberately non-alphabetical order")
print()
print("os.listdir is sorted :", listed == sorted(listed))
print("glob.glob is sorted :", globbed == sorted(globbed))
print("listdir and glob agree :", listed == globbed)
print("same SET of names :", set(listed) == set(globbed))
print("sorted(listdir) == sorted(glob) :", sorted(listed) == sorted(globbed))
print()
print("the actual order is not published because it is:",
"filesystem-specific")
print("which is exactly why you must not depend on it:", True)files created in a deliberately non-alphabetical order
os.listdir is sorted : False
glob.glob is sorted : False
listdir and glob agree : True
same SET of names : True
sorted(listdir) == sorted(glob) : True
the actual order is not published because it is: filesystem-specific
which is exactly why you must not depend on it: TrueNeither is sorted here, and — the stronger statement — neither is specified either way. The order depends on the filesystem, the creation history, and sometimes on how many entries a directory has ever contained, so two machines with identical files can enumerate them differently, and the same machine can return sorted order for one directory and not for another. Section 10 shows a three-file case that came back sorted, which makes the trap concrete: a sortedness check is not a test, because passing it proves nothing about the next directory.
That turns into an intermittent failure through a specific and common shape: a sweep that writes summary.csv by iterating glob("*.log") produces rows in a different order on a different machine, and Chapter 30.3's new-versus-fixed comparison — which diffs two summaries — reports every row as changed. The results are identical and the report says everything moved.
The fix is one call, and it belongs at the point of enumeration rather than later:
import glob
from pathlib import Path
for name in ["c_test", "b_test", "z_test", "a_test", "m_test"]:
Path(name + ".log").write_text("UVM_INFO\n", encoding="utf-8")
logs = sorted(Path(p) for p in glob.glob("*.log"))
print("sorted at enumeration:", [p.name for p in logs])
print()
print("rows in the report :", len(logs))
print("reproducible :", True)
print("on any filesystem :", True)
print()
print("and for a NUMERIC suffix, sorted() is still not enough:")
names = ["run_1.log", "run_2.log", "run_10.log", "run_20.log"]
print(" sorted() :", sorted(names))
print(" natural order :", sorted(names, key=lambda s: int(s.split('_')[1]
.split('.')[0])))sorted at enumeration: ['a_test.log', 'b_test.log', 'c_test.log', 'm_test.log', 'z_test.log']
rows in the report : 5
reproducible : True
on any filesystem : True
and for a NUMERIC suffix, sorted() is still not enough:
sorted() : ['run_1.log', 'run_10.log', 'run_2.log', 'run_20.log']
natural order : ['run_1.log', 'run_2.log', 'run_10.log', 'run_20.log']sorted() fixes reproducibility and introduces the lexicographic-versus-numeric problem Chapter 24.6 measured on sort -k2,2 — run_10 before run_2 — so a run index needs a key function or a zero-padded name. Reproducible first, then correct, and they are two separate pieces of work.
7. The Other Six Sources
SOURCES = [
("completion order", "a pool writes results as runs finish",
"collect, sort, then write", "high"),
("a shared path", "two runs write the same scratch file",
"a unique run_dir per run", "high"),
("leaked state", "a file from the previous run is still there",
"a fresh directory — 33.7", "high"),
("the clock", "a date-named dir collides at midnight",
"include the run index", "low"),
("locale", "a decimal comma in a parsed number",
"pin LC_ALL in cmd.sh", "low"),
("a retry that succeeded", "the run passed on attempt 2",
"log the retry — 33.10 §4", "medium"),
]
print(f"{'source':<20} {'how it bites':<44} {'frequency':<10} fix")
for src, how, fix, freq in SOURCES:
print(f"{src:<20} {how:<44} {freq:<10} {fix}")
print()
print("the top three share one property:", "they are all about SHARING")
print("the single best defence :", "one fresh directory per run")
print("which 33.7 argued for on :", "test-isolation grounds")
print("and is here required for :", "reproducibility")source how it bites frequency fix
completion order a pool writes results as runs finish high collect, sort, then write
a shared path two runs write the same scratch file high a unique run_dir per run
leaked state a file from the previous run is still there high a fresh directory — 33.7
the clock a date-named dir collides at midnight low include the run index
locale a decimal comma in a parsed number low pin LC_ALL in cmd.sh
a retry that succeeded the run passed on attempt 2 medium log the retry — 33.10 §4
the top three share one property: they are all about SHARING
the single best defence : one fresh directory per run
which 33.7 argued for on : test-isolation grounds
and is here required for : reproducibilityThe last row is the one that produces the report nobody can act on. A run that failed, retried, and passed is recorded as a pass — so the sweep looks clean and the underlying flakiness is invisible. That is precisely why Chapter 33.10 Section 4 defined WARNING as the run continued but not the way you asked: without that line in the log, there is no evidence the retry ever happened, and the one-in-twenty failure has been converted into a zero-in-twenty failure that is still there.
8. The Procedure
Turning an intermittent failure into a deterministic one
Step 5 is the one people skip, and skipping it is how a week gets lost. A one-in-twenty failure that does not appear in three runs after a change has told you almost nothing — the probability of that under no change at all is better than eight in ten. An intermittent bug can only be measured as a rate, so a candidate fix needs enough runs to distinguish the rate you had from the rate you have.
P_FAIL = 0.05 # one run in twenty
print("a 1-in-20 failure, and you run N times after a 'fix' that did nothing")
print()
print(f"{'runs':>6} {'chance of seeing zero failures':>32}")
for n in (1, 3, 5, 10, 20, 40, 60, 100):
p_none = (1 - P_FAIL) ** n
print(f"{n:>6} {p_none:>31.1%}")
print()
print("runs needed for 'no failures' to mean something:", "about 60")
print("runs most people do :", 3)
print("conclusion drawn from 3 clean runs :", "fixed")
print("probability that conclusion is wrong :", "about 86%")a 1-in-20 failure, and you run N times after a 'fix' that did nothing
runs chance of seeing zero failures
1 95.0%
3 85.7%
5 77.4%
10 59.9%
20 35.8%
40 12.9%
60 4.6%
100 0.6%
runs needed for 'no failures' to mean something: about 60
runs most people do : 3
conclusion drawn from 3 clean runs : fixed
probability that conclusion is wrong : about 86%Three clean runs leave an 86% chance that nothing was fixed. Sixty runs bring that below 5%, which is the number worth remembering: for a one-in-twenty failure, about sixty clean runs is the evidence threshold, and three is theatre.
That arithmetic is the same one Chapter 29.2 applied to coverage thresholds — a number is only evidence against the sample size that produced it — and it is why step 5 says measure the rate.
9. Once It Reproduces, It Is an Ordinary Bug
import re
SEVERITY_RANK = ["UVM_FATAL", "UVM_ERROR", "UVM_WARNING", "UVM_INFO"]
def worst_severity_buggy(lines):
"""Iterates a SET — the order decides the answer. The §5 bug."""
found = {s for line in lines for s in SEVERITY_RANK if line.startswith(s)}
for s in found: # <- set iteration order
return s
return "NONE"
def worst_severity_fixed(lines):
"""Iterates the RANK — the order is the specification."""
found = {s for line in lines for s in SEVERITY_RANK if line.startswith(s)}
for s in SEVERITY_RANK: # <- explicit, meaningful order
if s in found:
return s
return "NONE"
LOG = ["UVM_INFO top.sv(1) start",
"UVM_WARNING drv.sv(9) late",
"UVM_ERROR drv.sv(88) bad",
"TEST DONE"]
buggy = worst_severity_buggy(LOG)
fixed = worst_severity_fixed(LOG)
print("the log contains :", ["UVM_INFO", "UVM_WARNING", "UVM_ERROR"])
print("correct answer :", "UVM_ERROR")
print()
print("buggy version returned something in the set:", buggy in
{"UVM_INFO", "UVM_WARNING", "UVM_ERROR"})
print("buggy version is always correct :", False)
print("fixed version returned :", fixed)
print("fixed version depends on iteration order :", False)
print()
print("the test that pins it:")
print(" assert worst_severity(LOG) == 'UVM_ERROR' # every run")the log contains : ['UVM_INFO', 'UVM_WARNING', 'UVM_ERROR']
correct answer : UVM_ERROR
buggy version returned something in the set: True
buggy version is always correct : False
fixed version returned : UVM_ERROR
fixed version depends on iteration order : False
the test that pins it:
assert worst_severity(LOG) == 'UVM_ERROR' # every runThere is the one-in-twenty failure, as four lines of ordinary code. worst_severity_buggy returns whichever severity the set happened to yield first, so it is right whenever that is UVM_ERROR and wrong otherwise — a classifier that reports UVM_INFO for a log containing an error, occasionally, for reasons invisible in the code.
Note that the buggy version is not random in a way a test would catch by accident. Within one process it is perfectly consistent, so a test suite run once passes or fails and then keeps doing the same thing all day. Chapter 33.9's mutation experiment would not find it either, because it is not a missing check — it is a dependency on something nobody declared as an input.
And the fix is not defensive coding. It is replacing an implicit order with the explicit one that was always the specification: severity rank is a list, written down, and iterating it is both correct and reproducible.
PINS = [
("sorted() at every enumeration", "listdir, glob, set", "reproducible"),
("an explicit rank list", "anywhere order means something", "correct"),
("a unique run_dir per run", "no sharing", "isolated"),
("record seed, env, argv in cmd.sh", "every input written down", "replayable"),
("log retries at WARNING", "flakiness stays visible", "measurable"),
("collect, then write", "not in completion order", "reproducible"),
]
print(f"{'what you pin':<36} {'where':<36} what it buys")
for pin, where, buys in PINS:
print(f"{pin:<36} {where:<36} {buys}")
print()
print("none of these is a debugging technique:", True)
print("all of them are :", "ways not to need one")
print("the module's thesis :",
"an intermittent failure is an undeclared input")what you pin where what it buys
sorted() at every enumeration listdir, glob, set reproducible
an explicit rank list anywhere order means something correct
a unique run_dir per run no sharing isolated
record seed, env, argv in cmd.sh every input written down replayable
log retries at WARNING flakiness stays visible measurable
collect, then write not in completion order reproducible
none of these is a debugging technique: True
all of them are : ways not to need one
the module's thesis : an intermittent failure is an undeclared input10. Predict, Run, Observe
import glob
import os
import subprocess
import sys
from pathlib import Path
Path("o.py").write_text(
'print(",".join({"a", "b", "c", "d", "e", "f"}))\n', encoding="utf-8")
def distinct(n, env=None):
e = dict(os.environ)
if env:
e.update(env)
seen = set()
for _ in range(n):
seen.add(subprocess.run([sys.executable, "o.py"],
stdout=subprocess.PIPE, text=True,
env=e).stdout.strip())
return len(seen)
# A
print("A:", distinct(20) >= 2)
# B
print("B:", distinct(20, {"PYTHONHASHSEED": "0"}))
# C
for n in ["z.log", "a.log", "m.log"]:
Path(n).write_text("x", encoding="utf-8")
g = glob.glob("*.log")
print("C:", set(g) == {"z.log", "a.log", "m.log"}, len(g))
# D
d = {"z": 1, "a": 2, "m": 3}
print("D:", list(d), list(d) == sorted(d))The verified output:
A: True
B: 1
C: True 3
D: ['z', 'a', 'm'] FalseA — twenty processes, more than one set order. The failure's source, as one boolean.
B — with the seed pinned, exactly one order in twenty runs. That is the bisection tool, not the fix.
C — the set of names and the count are guaranteed; the order is not, which is why neither value here is about ordering. When these three files were enumerated they happened to come back in sorted order, where Section 6's five files did not — and that is the hazard stated precisely. An unspecified order is not reliably wrong, it is reliably unreliable, so a check asserting g == sorted(g) can pass on your machine, pass in review, and fail in the sweep. The only safe move is to sort and never ask.
D — the dict keeps insertion order, which is z, a, m and therefore not sorted. Reproducible and unsorted at the same time, which is the distinction Section 5 ended on: a dict solves the determinism problem and says nothing about whether the order is the one you wanted.
11. Exercises
Exercise 1 — five shapes. Classify the last intermittent failure you saw.
Exercise 2 — the invalidated premise. Explain why an intermittent failure breaks Chapter 33.9's method.
Exercise 3 — move one. Re-run a failing job's cmd.sh by hand and record whether it fails again.
Exercise 4 — without cmd.sh. Say why a reconstruction that does not reproduce proves nothing.
Exercise 5 — repr your PATH. Check your own for a trailing colon, an empty element and trailing spaces.
Exercise 6 — the current directory. Explain how an empty PATH element makes a failure depend on where the job started.
Exercise 7 — the invisible space. Describe the symptom of /opt/tools/bin and why echo $PATH hides it.
Exercise 8 — record, do not inspect. Add the environment variables your flow depends on to cmd.sh.
Exercise 9 — eight candidates. For your own flow, say which of Section 4's eight could differ between two runs.
Exercise 10 — an undeclared input. Restate your failure as "it depends on X, and X is not in cmd.sh".
Exercise 11 — twenty processes. Reproduce the set-order measurement.
Exercise 12 — why randomised. Say in one sentence why Python randomises string hashes.
Exercise 13 — dicts. Confirm dict order is stable across processes and explain why.
Exercise 14 — the diagnostic. Use PYTHONHASHSEED=0 on a flaky job and say what each outcome would tell you.
Exercise 15 — not a fix. Explain why shipping a flow that needs it is a bad idea.
Exercise 16 — audit your sets. Find a place where you iterate a set and decide what order it should be.
Exercise 17 — sorted() is not rank. Show that alphabetical order gets UVM severities wrong.
Exercise 18 — listdir. Confirm os.listdir is not sorted on your filesystem.
Exercise 19 — every row changed. Explain how enumeration order makes Chapter 30.3's diff useless.
Exercise 20 — sort at enumeration. Fix one place, and say why there rather than later.
Exercise 21 — run_10 before run_2. Add a key function or zero-pad your run names.
Exercise 22 — the top three. Explain what completion order, shared paths and leaked state have in common.
Exercise 23 — the retry that passed. Check whether your flow logs retries, and what the report says if it does not.
Exercise 24 — three runs prove nothing. Compute the probability of three clean runs for your own failure rate.
Exercise 25 — how many runs. Work out the number you would need for 95% confidence.
Exercise 26 — measure the rate. Re-run a candidate fix enough times to compare rates, and write both down.
Exercise 27 — the buggy classifier. Write worst_severity_buggy and explain why one process is consistent.
Exercise 28 — why mutation misses it. Say why Chapter 33.9's method would not find this bug.
Exercise 29 — explicit over defensive. Rewrite one implicit order as an explicit one.
Exercise 30 — pin six things. Apply Section 9's list to your own flow and say which you already had.
Exercise 31 — the test that would have caught it. Write it, and confirm it fails against the buggy version under at least one hash seed.
Exercise 32 — the module, applied. Pick one script and run every technique from 33.1 to 33.14 against it. Write down what you found.
12. Summary
- An intermittent failure invalidates the premise of this whole module, because every other technique assumes you can run the thing again and watch it break.
- The first job is not to fix it — it is to make it reproduce.
- Move one is only possible because of
cmd.sh. Without a record written before the tool launched, "re-run it by hand" means reconstructing a command line you cannot verify. repryour environment values. A trailing colon or an empty element puts the current directory onPATH; a trailing space makes a directory that does not exist; a trailing newline is invisible.- Record the environment into the run directory, rather than inspecting it later in a shell you have just configured.
- Debug the difference, not the failure. A one-in-twenty failure is deterministic with respect to something — you are looking for an input you did not know you had.
- String hashing is randomised per process, so set iteration order changes between runs of the same program. Twenty runs produced more than one order.
- Dictionaries are insertion-ordered and stable across processes.
PYTHONHASHSEED=0is a diagnostic, not a fix. Pin it to find the cause; do not ship a flow that depends on it.sorted()buys reproducibility, not correctness — alphabetical putsUVM_ERRORbeforeUVM_FATAL, which is the wrong severity rank.os.listdirandglob.globreturn filesystem order, which is not specified either way — so two machines with identical files can enumerate them differently, a report diff then shows every row as changed, and a check asserting the order is sorted can pass by luck.- Sort at the point of enumeration, and remember that
sorted()putsrun_10beforerun_2. - The three highest-frequency sources are all about sharing — completion order, a shared path, leaked state — and one fresh directory per run defends against all three.
- A retry that succeeded hides the flakiness entirely unless it is logged at
WARNING. - Three clean runs leave an 86% chance nothing was fixed. For a one-in-twenty failure, about sixty clean runs is the evidence threshold.
- An intermittent bug can only be measured as a rate, so a candidate fix needs enough runs to distinguish the old rate from the new one.
- Once it reproduces, it is an ordinary bug — and the fix is usually replacing an implicit order with the explicit one that was always the specification.
- An intermittent failure is an undeclared input. Every item on the pin list is a way of not needing a debugging technique at all.
That closes Module 33. The scripts you write from here are tested, logged, validated before they start, honest about what they leave behind when they stop, and reproducible when they misbehave — which is the difference between automation somebody maintains and automation somebody rewrites.
Where this fits
Part of the Python curriculum.