Python Part II · Module 25 · Simulation Automation
Making One Simulation Reliable Before Scaling
Six chapters have produced a runner that works. Working once and working four hundred times unattended are different claims, and the gap between them is where flows actually fail — on a stale artefact from a previous run, a variable that happened to be set in the shell you tested from, a log read before the tool finished flushing it. All of those are cheap to find by running one test a hundred times and impossible to find by running a hundred tests once, because in the second case every failure has a plausible alternative explanation.
Advanced19 min readPythonPart IIAutomationSimulationVerification Flow
Part II · Module 25 · Chapter 25.7 · Making One Simulation Reliable Before Scaling
1. One Test a Hundred Times
EXPERIMENTS = [
("100 tests, once each", "a failure could be the design, the test, or the flow",
"three explanations"),
("1 test, 100 times, same seed", "a varying outcome can only be the flow",
"one explanation"),
]
for what, why, how_many in EXPERIMENTS:
print(f"{what:<30} {why}")
print(f"{'':<30} -> {how_many}")
print()
print("the second experiment is also cheaper:")
print(" one compile, reused 100 times — Chapter 25.3 §7")
print(" one test's runtime, not a suite's")
print(" runnable in an afternoon, before anybody depends on the flow")100 tests, once each a failure could be the design, the test, or the flow
-> three explanations
1 test, 100 times, same seed a varying outcome can only be the flow
-> one explanation
the second experiment is also cheaper:
one compile, reused 100 times — Chapter 25.3 §7
one test's runtime, not a suite's
runnable in an afternoon, before anybody depends on the flowThe value of holding the test and the seed fixed is that it removes the alternative explanations. If the same inputs produce different outputs, the difference is in the machinery, and no amount of arguing about the design can make it otherwise.
That is the experiment the canonical annotation describes. Its instrument is the problem.
2. The assert Sees 41 of 100
for _ in range(100):
assert run_test("smoke", 1, sim=sim).status == "PASS"OUTCOMES = (["PASS"] * 40 + ["FAIL"] + ["PASS"] * 30 + ["NO_FINISH"] + ["PASS"] * 28)
def with_assert(outcomes):
for index, outcome in enumerate(outcomes, start=1):
if outcome != "PASS":
return index, outcome
return len(outcomes), None
stopped_at, what = with_assert(OUTCOMES)
print("runs in the experiment :", len(OUTCOMES))
print("runs actually observed :", stopped_at)
print("stopped by :", what)
print()
print("non-PASS outcomes present:", [o for o in OUTCOMES if o != "PASS"])
print("non-PASS outcomes seen :", [what])
print("never observed :", OUTCOMES[71])runs in the experiment : 100
runs actually observed : 41
stopped by : FAIL
non-PASS outcomes present: ['FAIL', 'NO_FINISH']
non-PASS outcomes seen : ['FAIL']
never observed : NO_FINISHThe assertion stopped at run 41 and the hang at run 72 was never observed.
That matters more than the lost runs. The experiment's purpose is to characterise how the flow misbehaves, and a FAIL and a NO_FINISH are two different machinery problems with two different fixes — Chapter 25.1 Section 6's table. An instrument that reports the first one and discards the rest tells you that something is wrong, which you could have guessed.
Collect instead:
from collections import Counter
OUTCOMES = (["PASS"] * 40 + ["FAIL"] + ["PASS"] * 30 + ["NO_FINISH"] + ["PASS"] * 28)
counts = Counter(OUTCOMES)
print("distribution :", dict(sorted(counts.items())))
print("runs :", sum(counts.values()))
print("pass rate :", f"{counts['PASS']}/{sum(counts.values())}")
print("distinct :", len(counts))
print()
print("failing run indices:", [i for i, o in enumerate(OUTCOMES, 1) if o != "PASS"])distribution : {'FAIL': 1, 'NO_FINISH': 1, 'PASS': 98}
runs : 100
pass rate : 98/100
distinct : 3
failing run indices: [41, 72]Both problems, their rate, and exactly which runs to go and look at. Chapter 24.5 Section 2 made this argument for a pre-flight check and the reason is identical: no later run depends on an earlier one's result, so there is nothing to be gained by stopping and a complete picture to be gained by continuing.
Keep the assertion, and put it at the end where it belongs:
from collections import Counter
OUTCOMES = (["PASS"] * 40 + ["FAIL"] + ["PASS"] * 30 + ["NO_FINISH"] + ["PASS"] * 28)
def soak(outcomes, *, require_all_pass=True):
counts = Counter(outcomes)
bad = {k: v for k, v in counts.items() if k != "PASS"}
report = {"runs": len(outcomes), "counts": dict(sorted(counts.items())),
"distinct": len(counts),
"indices": [i for i, o in enumerate(outcomes, 1) if o != "PASS"]}
if require_all_pass and bad:
raise AssertionError(f"{sum(bad.values())} of {len(outcomes)} runs were not PASS: "
f"{dict(sorted(bad.items()))} at runs {report['indices']}")
return report
try:
soak(OUTCOMES)
except AssertionError as exc:
print("AssertionError:", exc)
print()
print("report still available:", soak(OUTCOMES, require_all_pass=False))AssertionError: 2 of 100 runs were not PASS: {'FAIL': 1, 'NO_FINISH': 1} at runs [41, 72]
report still available: {'runs': 100, 'counts': {'FAIL': 1, 'NO_FINISH': 1, 'PASS': 98}, 'distinct': 3, 'indices': [41, 72]}One assertion, raised after all hundred runs, whose message contains the whole distribution. That is the shape: run everything, then decide. The annotation's intent is preserved — the soak still fails loudly — and the evidence survives.
3. One Distinct Result
from dataclasses import dataclass, field
from typing import Optional
@dataclass(frozen=True)
class RunResult:
test: str
seed: int
status: str
reason: Optional[str] = None
runtime_s: float = field(default=0.0, compare=False)
deterministic = [RunResult("smoke", 1, "PASS", runtime_s=180.0 + i * 0.37)
for i in range(100)]
print("100 runs, same test, same seed, varying runtimes")
print("distinct results :", len(set(deterministic)))
print("runtimes differ :", deterministic[0].runtime_s != deterministic[99].runtime_s)
print()
noisy = deterministic[:99] + [RunResult("smoke", 1, "FAIL", "UVM_ERROR", runtime_s=175.2)]
print("the same, with one run differing")
print("distinct results :", len(set(noisy)))
print("what they are :", sorted({(r.status, r.reason) for r in noisy}))100 runs, same test, same seed, varying runtimes
distinct results : 1
runtimes differ : True
the same, with one run differing
distinct results : 2
what they are : [('FAIL', 'UVM_ERROR'), ('PASS', None)]len(set(results)) == 1 is the definition of a deterministic test, and it is one expression.
This is the chapter Chapter 25.1 Section 7 was built for. Without field(compare=False) on runtime_s, those hundred identical outcomes would be a hundred distinct results, because no two runs take exactly the same time — and the check would report maximum non-determinism on a perfectly deterministic flow. One keyword argument, chosen six chapters ago, is what makes the headline measurement of this chapter a single call.
frozen=True is the other half: a non-frozen dataclass is unhashable and set() raises.
4. What a Hundred Runs Actually Finds
FINDINGS = [
("a stale artefact from the previous run", "run 2 onwards", "the run directory was reused"),
("a variable set in your shell, not in the flow", "every run on a clean machine", "it worked for you"),
("a log read before the tool flushed it", "intermittently", "a race between writer and reader"),
("two runs sharing a scratch path", "intermittently", "the path had no seed in it"),
("a licence held by a previous run", "early runs only", "cleanup did not release it"),
("disk filling up", "late runs only", "nothing deleted anything"),
]
print(f"{'what breaks':<44} {'when':<28} cause")
for what, when, cause in FINDINGS:
print(f"{what:<44} {when:<28} {cause}")
print()
print("how many of these a 100-test single-pass regression would catch:", "some, confusingly")
print("how many a 1-test 100-run soak catches:", "all of them, unambiguously")what breaks when cause
a stale artefact from the previous run run 2 onwards the run directory was reused
a variable set in your shell, not in the flow every run on a clean machine it worked for you
a log read before the tool flushed it intermittently a race between writer and reader
two runs sharing a scratch path intermittently the path had no seed in it
a licence held by a previous run early runs only cleanup did not release it
disk filling up late runs only nothing deleted anything
how many of these a 100-test single-pass regression would catch: some, confusingly
how many a 1-test 100-run soak catches: all of them, unambiguouslyThe when column is the diagnostic, and it is only readable because every run was recorded. "Run 2 onwards" is a reused directory — Chapter 23.7's whole subject. "Intermittently" is a race. "Late runs only" is a resource leak, which Chapter 24.4's retention policy addresses.
Two of those deserve naming because they are the ones that survive review.
A variable set in your shell. The flow works perfectly for the person who wrote it and fails on every machine that is not theirs. Chapter 24.5's pre-flight catches it if the variable is in the required list, and the soak catches it when run somewhere clean — which is why a soak on a fresh machine is worth more than a soak on yours.
A log read before it is flushed. Chapter 23.6 Section 5 warned that the last lines a tool flushed are not necessarily the last thing it did. In a runner that reads the log immediately after the process exits, this is usually fine and occasionally is not, and the symptom is a NO_FINISH on a run that finished — an end banner that was written but not yet visible. It appears in perhaps one run in fifty, which is exactly the rate at which a single pass never sees it.
5. Same Seed or Different Seeds
from collections import Counter
SAME_SEED = ["PASS"] * 97 + ["FAIL", "PASS", "FAIL"]
DIFF_SEEDS = ["PASS"] * 97 + ["FAIL", "PASS", "FAIL"]
for label, outcomes, varied in (("same seed, 100 runs", SAME_SEED, "nothing"),
("100 different seeds", DIFF_SEEDS, "the stimulus")):
counts = Counter(outcomes)
print(f"{label:<22} {dict(sorted(counts.items()))} varied: {varied}")
print()
print("identical numbers, opposite meanings:")
print(" same seed varying -> NON-DETERMINISM in the machinery")
print(" different seeds varying -> the testbench found something")
print()
print("only the first is a reliability problem")
print("the second is the testbench doing its job")same seed, 100 runs {'FAIL': 2, 'PASS': 98} varied: nothing
100 different seeds {'FAIL': 2, 'PASS': 98} varied: the stimulus
identical numbers, opposite meanings:
same seed varying -> NON-DETERMINISM in the machinery
different seeds varying -> the testbench found something
only the first is a reliability problem
the second is the testbench doing its jobThe same two numbers mean opposite things, and which one you have depends entirely on an experimental choice nobody records in the summary table.
This is why flaky is such a damaging word. It is used for both, and the two have nothing in common:
-
Same seed, varying outcome. The inputs were identical and the results were not. Something in the machinery is non-deterministic — a shared path, a race, an uninitialised value, a reused directory. The design is not implicated at all, and this must be fixed before scaling, because at four hundred runs it becomes a background noise floor that hides real failures.
-
Different seeds, varying outcome. Different stimulus produced different results, which is the entire point of constrained-random verification. Two failures in a hundred seeds is a finding, and the right response is to reproduce seed 95 — Chapter 25.5's one command — not to retry it until it passes.
def interpret(same_seed, distinct_outcomes):
if distinct_outcomes == 1:
return "deterministic" if same_seed else "no seed sensitivity found"
return "NON-DETERMINISM — fix before scaling" if same_seed else \
"seed-dependent failures — reproduce and debug"
for same_seed in (True, False):
for distinct in (1, 3):
label = "same seed " if same_seed else "diff seeds"
print(f"{label} distinct={distinct} -> {interpret(same_seed, distinct)}")same seed distinct=1 -> deterministic
same seed distinct=3 -> NON-DETERMINISM — fix before scaling
diff seeds distinct=1 -> no seed sensitivity found
diff seeds distinct=3 -> seed-dependent failures — reproduce and debugRun both experiments, in that order. The same-seed soak answers is the machinery trustworthy? and until the answer is yes, the different-seed sweep cannot be interpreted — because every failure in it has the non-determinism as an alternative explanation, and you are back to Section 1's three-explanation problem.
6. ERROR Is Not Flakiness
from collections import Counter
OUTCOMES = ["PASS"] * 94 + ["FAIL", "FAIL", "NO_FINISH", "ERROR", "ERROR", "PASS"]
counts = Counter(OUTCOMES)
total = len(OUTCOMES)
infra = counts["ERROR"]
tested = total - infra
passed = counts["PASS"]
print("distribution :", dict(sorted(counts.items())))
print()
print("naive pass rate :", f"{passed}/{total}", f"= {100 * passed / total:.0f}%")
print("honest pass rate :", f"{passed}/{tested}", f"= {100 * passed / tested:.0f}%")
print()
print("runs that tested nothing:", infra)
print("what ERROR means here :", "the soak found an INFRASTRUCTURE problem")
print("is that a success? :", "yes — that is what the soak is for")distribution : {'ERROR': 2, 'FAIL': 2, 'NO_FINISH': 1, 'PASS': 95}
naive pass rate : 95/100 = 95%
honest pass rate : 95/98 = 97%
runs that tested nothing: 2
what ERROR means here : the soak found an INFRASTRUCTURE problem
is that a success? : yes — that is what the soak is forChapter 23.9 Section 7's denominator rule, applied to the one number this whole experiment produces. Two ERROR rows are not failures of the design and not non-determinism in the ordinary sense; they are the soak doing exactly what it was run for.
And they are the most valuable rows in the table, because an infrastructure failure that shows up twice in a hundred runs will show up eight times in a four-hundred-test regression — every night, inconsistently, in a different test each time. That is the pattern that trains a team to ignore failures.
7. The Soak
from collections import Counter
from dataclasses import dataclass, field
from typing import Optional
TERMINAL = ("PASS", "FAIL", "COMPILE_ERROR", "NO_FINISH", "ERROR")
NOT_TESTED = ("ERROR", "COMPILE_ERROR")
@dataclass(frozen=True)
class RunResult:
test: str
seed: int
status: str
reason: Optional[str] = None
runtime_s: float = field(default=0.0, compare=False)
def fake_run(test, seed, index):
"""Stand in for run_test. Deterministic given the index."""
if index == 41:
return RunResult(test, seed, "FAIL", "UVM_ERROR", runtime_s=170.0)
if index == 72:
return RunResult(test, seed, "NO_FINISH", "NO_END_MARKER", runtime_s=3600.0)
if index == 88:
return RunResult(test, seed, "ERROR", "licence unavailable", runtime_s=0.4)
return RunResult(test, seed, "PASS", None, runtime_s=180.0 + index * 0.1)
def soak(test, seed, runs=100):
results = [fake_run(test, seed, i) for i in range(1, runs + 1)]
counts = Counter(r.status for r in results)
tested = sum(n for s, n in counts.items() if s not in NOT_TESTED)
outcomes = {(r.status, r.reason) for r in results}
return {
"runs": len(results),
"distinct_results": len(set(results)),
"outcomes": sorted(outcomes),
"counts": dict(sorted(counts.items())),
"pass_rate": f"{counts['PASS']}/{tested}",
"non_deterministic": len(set(results)) > 1,
"offending_runs": [i for i, r in enumerate(results, 1) if r.status != "PASS"],
}
report = soak("smoke", 1)
for key in ("runs", "distinct_results", "counts", "pass_rate",
"non_deterministic", "offending_runs"):
print(f"{key:<18} {report[key]}")
print()
for outcome in report["outcomes"]:
print(" outcome:", outcome)runs 100
distinct_results 4
counts {'ERROR': 1, 'FAIL': 1, 'NO_FINISH': 1, 'PASS': 97}
pass_rate 97/99
non_deterministic True
offending_runs [41, 72, 88]
outcome: ('ERROR', 'licence unavailable')
outcome: ('FAIL', 'UVM_ERROR')
outcome: ('NO_FINISH', 'NO_END_MARKER')
outcome: ('PASS', None)Four distinct outcomes from one test at one seed, and the report names all four plus the runs to look at. Four properties of it are worth copying.
distinct_results is the headline. One means the flow is deterministic; anything else is the number of different things that happened when nothing was supposed to vary. It is a single integer, and it is the one figure to put at the top of a reliability report.
non_deterministic is derived, not measured separately. There is no second code path that could disagree with the count, which is the kind of redundancy that drifts.
offending_runs gives indices, not just counts. Chapter 24.4 Section 6 made the same argument about a sweep plan: a number tells you how bad, a list tells you where to go.
The pass rate excludes the ERROR run, 97 of 99. Section 6's rule, implemented rather than remembered.
What a real soak replaces is fake_run — and nothing else. The function above takes a test and a seed and returns a RunResult, which is precisely Chapter 25.1's contract, and that substitutability is what the whole module has been for.
8. When Is It Ready
CRITERIA = [
("distinct results at one seed", 1, "any other value is non-determinism"),
("ERROR runs", 0, "infrastructure must be clean before scaling"),
("runs completed", 100, "a shorter soak cannot see a 1-in-50 race"),
("run on a clean machine", 1, "your shell is not the environment"),
("run with a cold cache", 1, "a warm compile cache hides a cache-key bug"),
]
print(f"{'criterion':<32} {'required':>9} why")
for name, required, why in CRITERIA:
print(f"{name:<32} {required:>9} {why}")
print()
print("all five, or the regression in Module 26 inherits the problem")criterion required why
distinct results at one seed 1 any other value is non-determinism
ERROR runs 0 infrastructure must be clean before scaling
runs completed 100 a shorter soak cannot see a 1-in-50 race
run on a clean machine 1 your shell is not the environment
run with a cold cache 1 a warm compile cache hides a cache-key bugFive criteria, and the last two are the ones people skip.
A clean machine is what turns Section 4's shell-variable problem from invisible into obvious. A soak on the author's workstation tests the author's workstation.
A cold cache is what exercises Chapter 25.3 Section 7's compile key and Chapter 25.5 Section 4's waves omission. A hundred runs that all reuse one cached binary have tested the cache exactly once, and the stale-binary bug — the worst failure in this module — is invisible to every one of them.
The number 100 is a convention rather than a theorem, and it is worth being honest about what it buys: a failure occurring in one run in fifty has a good chance of appearing, and one occurring in one in a thousand does not. If your flow has a rare failure you have already seen, the soak needs to be longer than the rate you are chasing — and if you have never seen one, a hundred is where to start.
9. Predict, Run, Observe
from collections import Counter
from dataclasses import dataclass, field
@dataclass(frozen=True)
class R:
s: str
r: str = None
t: float = field(default=0.0, compare=False)
OUT = ["PASS"] * 3 + ["FAIL"] + ["PASS"] + ["ERROR"]
# A
print("A:", dict(sorted(Counter(OUT).items())))
# B
print("B:", [i for i, o in enumerate(OUT, 1) if o != "PASS"])
# C
print("C:", len({R("PASS", None, float(i)) for i in range(50)}))
# D
print("D:", len({R("PASS"), R("FAIL", "UVM_ERROR")}))
# E
total = len(OUT)
tested = total - Counter(OUT)["ERROR"]
print("E:", f"{Counter(OUT)['PASS']}/{total}", f"{Counter(OUT)['PASS']}/{tested}")
# F
def stops(outs):
for i, o in enumerate(outs, 1):
if o != "PASS":
return i
return len(outs)
print("F:", stops(OUT), "of", len(OUT))
# G
print("G:", sorted({(R("PASS").s, R("PASS").r), (R("FAIL", "X").s, R("FAIL", "X").r)}))
# H
print("H:", len(set(OUT)) == 1)The verified output:
A: {'ERROR': 1, 'FAIL': 1, 'PASS': 4}
B: [4, 6]
C: 1
D: 2
E: 4/6 4/5
F: 4 of 6
G: [('FAIL', 'X'), ('PASS', None)]
H: FalseA — the whole distribution, not the first problem.
B — and the runs to go and look at.
C — fifty results differing only in runtime are one result. Chapter 25.1 Section 7.
D — a status difference still separates them.
E — the naive and honest pass rates differ because of one ERROR.
F — an assertion would have seen four of six runs.
G — (status, reason) pairs are the useful granularity for "what happened".
H — more than one distinct outcome, so this flow is not yet ready to scale.
10. Exercises
Exercise 1 — three explanations. Say why a failure in a hundred-test single pass is ambiguous and the same failure in a one-test soak is not.
Exercise 2 — the cost. Work out what a hundred-run soak of your fastest test costs, with the compile reused.
Exercise 3 — the assert. Run the annotation's form over a sequence with two different failures. Report how many runs it observed.
Exercise 4 — what it missed. Say why missing the second kind of failure matters more than missing the runs.
Exercise 5 — collect. Produce the distribution, the rate and the offending indices instead.
Exercise 6 — assert at the end. Keep the loud failure and put the whole distribution in its message.
Exercise 7 — one distinct result. Build a hundred results differing only in runtime and show len(set(...)) is 1.
Exercise 8 — without compare=False. Remove it and report what the same measurement gives.
Exercise 9 — unhashable. Drop frozen=True and record the error.
Exercise 10 — the when column. For each of Section 4's six findings, say what pattern of run indices it would produce.
Exercise 11 — your shell. Name a variable your flow depends on that is set in your shell. Say how a soak on a clean machine would reveal it.
Exercise 12 — the flush race. Explain how a log read too early produces a NO_FINISH on a run that finished.
Exercise 13 — same or different. Take a summary showing two failures in a hundred and say what else you need to know to interpret it.
Exercise 14 — opposite meanings. Write one sentence each for the two meanings of a varying outcome.
Exercise 15 — order. Say why the same-seed soak must come before the different-seed sweep.
Exercise 16 — the word flaky. Argue in three sentences for retiring it, and propose two replacements.
Exercise 17 — the denominator. Compute both pass rates over a soak containing ERROR runs.
Exercise 18 — the noise floor. Describe what a 2% non-determinism rate does to a four-hundred-test nightly.
Exercise 19 — the retry wrapper. Explain why adding one before fixing the non-determinism is the worst available move.
Exercise 20 — write the soak. Produce a report with distinct results, counts, honest pass rate and offending indices.
Exercise 21 — substitutability. Say what you have to change in your soak to point it at the real runner, and why it is only one function.
Exercise 22 — the last two criteria. Run your soak on a clean machine with a cold cache. Report what changed.
Exercise 23 — how long. Given a failure rate you have actually observed, work out how many runs a soak needs to be likely to see it.
11. Summary
- Hold the test and the seed fixed so a varying outcome has only one possible explanation: the machinery.
- The soak is also cheap — one compile reused, one test's runtime, an afternoon.
- The annotation's
assertstopped at run 41 and never observed theNO_FINISHat run 72. - Two different failure kinds mean two different fixes, so an instrument that reports only the first tells you nothing you could not guess.
- Collect every run, then assert once at the end, with the whole distribution in the message.
- No later run depends on an earlier one — Chapter 24.5's argument for a pre-flight check, in a new setting.
len(set(results)) == 1is the definition of a deterministic test, and it is one expression.- That measurement exists only because of
field(compare=False)onruntime_s— without it, a hundred identical outcomes are a hundred distinct results. frozen=Trueis the other half; a non-frozen dataclass is unhashable.- A soak finds stale artefacts, shell-only variables, flush races, shared scratch paths, held licences and filling disks — and the pattern of run indices identifies which.
- Same seed varying is non-determinism; different seeds varying is the testbench working. The same two numbers, opposite meanings.
- Run the same-seed soak first, or every failure in the seed sweep has an alternative explanation.
- "Flaky" conflates the two and should be replaced by whichever one you actually mean.
ERRORruns are not failures and not non-determinism — exclude them from the denominator, and treat finding them as a success.- A 2% non-determinism rate is worse than a broken flow, because it gets a retry wrapper instead of a fix.
- Report
distinct_resultsas the headline, derive the non-determinism flag from it, and list the offending run indices. - Require a clean machine and a cold cache — a warm cache has tested the compile key exactly once, and the stale-binary bug is invisible to every run.
- A hundred is a convention: it catches a one-in-fifty race and not a one-in-a-thousand one.
That completes Module 25. One test, one seed, one simulator, one directory it owns, one structured result — and a measurement that says whether any of it can be trusted at scale. Module 26 is where it is run four hundred times.
Where this fits
Part of the Python curriculum.