Python Part II · Module 30 · Engineering Report Generation
New Failures, Fixed Failures and Flaky Tests
The “what changed” line of a regression report is the one engineers act on, and it is produced by two expressions that look like they need no explanation. One of them manufactures regressions when somebody adds a test, reports a deleted test as repaired, and cannot see a failure that moved from one seed to another. The other is worse: on the data a nightly regression actually produces, it is incapable of returning anything at all, and reports a clean bill of health every night by being vacuously false.
Advanced22 min readPythonPart IIAutomationReportingFlaky Tests
Part II · Module 30 · Chapter 30.3 · New Failures, Fixed Failures and Flaky Tests
1. Three Categories, Not Two
A failing test tonight is one of three things, and only one of them is a regression:
KINDS = [
("new", "passed last night, fails tonight", "this is the regression"),
("known", "failed last night too", "already triaged; not news"),
("flaky", "fails some runs and not others", "a different kind of bug"),
]
print(f"{'kind':<8} {'definition':<36} what to do")
for kind, defn, what in KINDS:
print(f"{kind:<8} {defn:<36} {what}")
print()
print("kinds a two-way new/known split can express:", 2)
print("kinds that exist :", len(KINDS))
print("and the third is the one that wastes the most time.")kind definition what to do
new passed last night, fails tonight this is the regression
known failed last night too already triaged; not news
flaky fails some runs and not others a different kind of bug
kinds a two-way new/known split can express: 2
kinds that exist : 3
and the third is the one that wastes the most time.A flaky test is the expensive category because it costs attention repeatedly: it appears as new, gets triaged, passes on the re-run, gets closed, and appears as new again a week later. Separating it out is why this chapter's interview importance is Critical.
2. The Annotation
new = today_fails - yesterday_fails
flaky = {t for t in tests if 0 < fail_rate(t) < 1}Two expressions, both idiomatic, both the thing a competent engineer writes first:
yesterday_fails = {"dma_long", "reg_rw"}
today_fails = {"dma_long", "axi_burst"}
new = today_fails - yesterday_fails
fixed = yesterday_fails - today_fails
known = today_fails & yesterday_fails
print("new :", sorted(new))
print("fixed:", sorted(fixed))
print("known:", sorted(known))
print()
print("on this data the answers are correct.")new : ['axi_burst']
fixed: ['reg_rw']
known: ['dma_long']
on this data the answers are correct.On this data it is right, which is why the expression survives review. Sections 3 to 6 are the data on which it is not.
3. The Flaky Predicate Cannot Fire
Start with the second line, because its defect is total rather than partial:
from fractions import Fraction
def achievable_rates(runs):
"""Every fail rate observable from `runs` runs of one test."""
return [Fraction(f, runs) for f in range(runs + 1)]
print(f"{'runs':>5} {'achievable rates':>18} {'strictly between 0 and 1':>26}")
for runs in (1, 2, 5, 20):
rates = achievable_rates(runs)
strict = [r for r in rates if 0 < r < 1]
print(f"{runs:>5} {len(rates):>18} {len(strict):>26}")
print()
print("with runs = 1 the achievable rates are:", achievable_rates(1))
print("values satisfying 0 < r < 1 :",
[r for r in achievable_rates(1) if 0 < r < 1]) runs achievable rates strictly between 0 and 1
1 2 0
2 3 1
5 6 4
20 21 19
with runs = 1 the achievable rates are: [Fraction(0, 1), Fraction(1, 1)]
values satisfying 0 < r < 1 : []With one run per test, 0 < fail_rate(t) < 1 has no satisfying value. The predicate is not merely insensitive; it is vacuously false, and flaky is the empty set on every night of a nightly regression that runs each test once.
Now watch what that does in a report:
TESTS = ["axi_burst", "dma_long", "reg_rw", "fifo_wrap"]
TONIGHT = {"axi_burst": 1, "dma_long": 0, "reg_rw": 1, "fifo_wrap": 0} # failures
RUNS = {t: 1 for t in TESTS}
def fail_rate(t):
return TONIGHT[t] / RUNS[t]
flaky = {t for t in TESTS if 0 < fail_rate(t) < 1}
print("fail rates tonight:", {t: fail_rate(t) for t in TESTS})
print("flaky :", flaky or "set()")
print()
print("rendered in a report:", "Flaky tests: none")
print("what it means :", "the predicate could not have said otherwise")fail rates tonight: {'axi_burst': 1.0, 'dma_long': 0.0, 'reg_rw': 1.0, 'fifo_wrap': 0.0}
flaky : set()
rendered in a report: Flaky tests: none
what it means : the predicate could not have said otherwiseFlaky tests: none every night, for months, from a check that is structurally incapable of saying anything else. This is the same class of defect as Chapter 24.5's dead check and Chapter 30.1 Section 6's empty baseline: a result that would be identical if the mechanism were absent entirely.
4. A Set Difference on Names Loses the Seed
Now the first line. A regression runs tests at seeds, and the annotation's sets are of names:
YESTERDAY = {("axi_smoke", 1839), ("axi_burst", 7731)}
TODAY = {("axi_smoke", 2048), ("axi_burst", 7731)}
print("by (test, seed):")
print(" new :", sorted(TODAY - YESTERDAY))
print(" fixed:", sorted(YESTERDAY - TODAY))
y_names = {t for t, _ in YESTERDAY}
t_names = {t for t, _ in TODAY}
print("by name only:")
print(" new :", sorted(t_names - y_names))
print(" fixed:", sorted(y_names - t_names))
print()
print("the report by name says:", "nothing changed")
print("what actually happened :", "axi_smoke failed at a different seed")by (test, seed):
new : [('axi_smoke', 2048)]
fixed: [('axi_smoke', 1839)]
by name only:
new : []
fixed: []
the report by name says: nothing changed
what actually happened : axi_smoke failed at a different seedBoth views are defensible and they answer different questions. By name asks "is this test still broken?", which is the right question for a triage queue. By (test, seed) asks "is this the same failure?", which is the right question for deciding whether yesterday's debug still applies — and a failure that moved seeds is a strong hint of exactly the flakiness Section 3 could not detect.
The practical answer is to compute both and report the name view while keeping the pair view for the flakiness signal:
YESTERDAY = {("axi_smoke", 1839), ("axi_burst", 7731)}
TODAY = {("axi_smoke", 2048), ("axi_burst", 7731)}
def compare(today, yesterday):
by_name = ({t for t, _ in today}, {t for t, _ in yesterday})
moved = sorted(
t for t in by_name[0] & by_name[1]
if {s for n, s in today if n == t} != {s for n, s in yesterday if n == t})
return {
"new_tests": sorted(by_name[0] - by_name[1]),
"fixed_tests": sorted(by_name[1] - by_name[0]),
"still_failing": sorted(by_name[0] & by_name[1]),
"same_seed": sorted(today & yesterday),
"moved_seed": moved,
}
for k, v in compare(TODAY, YESTERDAY).items():
print(f" {k:<14} {v}")
print()
print("a test still failing but at a different seed is the flakiness hint.") new_tests []
fixed_tests []
still_failing ['axi_burst', 'axi_smoke']
same_seed [('axi_burst', 7731)]
moved_seed ['axi_smoke']
a test still failing but at a different seed is the flakiness hint.axi_burst failed at the same seed both nights, which is a reproducible bug somebody can debug. axi_smoke moved, which means either the seed set changed or the failure is seed-sensitive — and one line of output distinguishes two situations that a name-only comparison reports identically.
5. A Test That Did Not Run Is Not a New Failure
The set difference assumes every test in today's results was also in yesterday's. Adding a test breaks that:
YESTERDAY_RAN = {"axi_smoke", "axi_burst", "dma_long"}
YESTERDAY_FAILED = {"axi_burst"}
TODAY_FAILED = {"axi_burst", "reg_rw"} # reg_rw is brand new to the testlist
naive_new = TODAY_FAILED - YESTERDAY_FAILED
honest_new = (TODAY_FAILED - YESTERDAY_FAILED) & YESTERDAY_RAN
not_comparable = (TODAY_FAILED - YESTERDAY_FAILED) - YESTERDAY_RAN
print("naive new :", sorted(naive_new))
print("ran yesterday? :", {t: t in YESTERDAY_RAN for t in sorted(naive_new)})
print()
print("honest new :", sorted(honest_new) or "none")
print("not comparable :", sorted(not_comparable))
print()
print("adding one test to the testlist manufactured", len(not_comparable), "regression(s)")naive new : ['reg_rw']
ran yesterday? : {'reg_rw': False}
honest new : none
not comparable : ['reg_rw']
adding one test to the testlist manufactured 1 regression(s)reg_rw is failing and that matters — but it is not a regression, because nothing regressed. It is a new test that does not pass yet, which is a different conversation with a different owner, and reporting it under "new failures" sends somebody looking for the commit that broke it.
The fix is to intersect with what actually ran, and to report the third category explicitly rather than folding it into either side.
6. A Deleted Test Looks Fixed
The mirror image is worse, because it points the reassuring way:
YESTERDAY_FAILED = {"axi_burst", "flaky_mess"}
TODAY_RAN = {"axi_burst", "dma_long"} # flaky_mess was removed from the testlist
TODAY_FAILED = {"axi_burst"}
naive_fixed = YESTERDAY_FAILED - TODAY_FAILED
honest_fixed = (YESTERDAY_FAILED - TODAY_FAILED) & TODAY_RAN
disappeared = (YESTERDAY_FAILED - TODAY_FAILED) - TODAY_RAN
print("naive fixed :", sorted(naive_fixed))
print("ran tonight? :", {t: t in TODAY_RAN for t in sorted(naive_fixed)})
print()
print("honest fixed :", sorted(honest_fixed) or "none")
print("disappeared :", sorted(disappeared))
print()
print("the report says one failure was fixed. zero were.")naive fixed : ['flaky_mess']
ran tonight? : {'flaky_mess': False}
honest fixed : none
disappeared : ['flaky_mess']
the report says one failure was fixed. zero were.Deleting a failing test from the testlist is indistinguishable from repairing it, and it is the cheapest possible way to make a regression report go green. As with Chapter 29.4 Section 6's coverage exclusions, the point is not that anybody does this deliberately — a test gets commented out for an afternoon and the comment stays — but that the flow cannot tell the difference, and therefore has to report the count:
def changed(y_ran, y_failed, t_ran, t_failed):
return {
"new": sorted((t_failed - y_failed) & y_ran),
"fixed": sorted((y_failed - t_failed) & t_ran),
"still_failing": sorted(t_failed & y_failed),
"new_tests_failing": sorted((t_failed - y_failed) - y_ran),
"vanished_while_failing": sorted((y_failed - t_failed) - t_ran),
"testlist_grew_by": len(t_ran - y_ran),
"testlist_shrank_by": len(y_ran - t_ran),
}
r = changed(
y_ran={"axi_smoke", "axi_burst", "flaky_mess", "dma_long"},
y_failed={"axi_burst", "flaky_mess"},
t_ran={"axi_smoke", "axi_burst", "dma_long", "reg_rw"},
t_failed={"axi_burst", "reg_rw"},
)
for k, v in r.items():
print(f" {k:<24} {v}")
print()
print("real regressions tonight:", len(r["new"]))
print("real fixes tonight :", len(r["fixed"]))
print("things needing a different conversation:",
len(r["new_tests_failing"]) + len(r["vanished_while_failing"])) new []
fixed []
still_failing ['axi_burst']
new_tests_failing ['reg_rw']
vanished_while_failing ['flaky_mess']
testlist_grew_by 1
testlist_shrank_by 1
real regressions tonight: 0
real fixes tonight : 0
things needing a different conversation: 2The annotation on this night reports one new failure and one fix. The honest answer is zero and zero, with two separate matters that are not regressions at all — and note the testlist counts, which are what make the night interpretable: the list both grew and shrank by one, so a reader knows to expect exactly this.
7. Flaky Is Not Seed-Dependent
The canonical purpose says "a test that fails one seed in twenty", and that phrase contains an ambiguity worth resolving, because the two things it could mean need different responses:
SEEDS = list(range(20))
def seed_dependent(seed):
"""Deterministic: this test fails if and only if seed % 20 == 7."""
return seed % 20 == 7
results = {s: seed_dependent(s) for s in SEEDS}
failing = [s for s, bad in results.items() if bad]
print("failing seeds:", failing)
print(f"fail rate over the seed set: {len(failing)}/{len(SEEDS)}")
print()
print("re-running seed 7 five times:", [seed_dependent(7) for _ in range(5)])
print("re-running seed 3 five times:", [seed_dependent(3) for _ in range(5)])
print()
print("is this flaky?", "no — it is perfectly reproducible at every seed")failing seeds: [7]
fail rate over the seed set: 1/20
re-running seed 7 five times: [True, True, True, True, True]
re-running seed 3 five times: [False, False, False, False, False]
is this flaky? no — it is perfectly reproducible at every seedOne seed in twenty fails, and the test is not flaky at all. It is a test that found a real bug reachable only by the stimulus seed 7 produces, which is the best possible outcome of a random regression — and calling it flaky and quarantining it throws away a reproducible failure.
Flakiness is non-determinism at a fixed seed:
def truly_flaky(seed, attempt):
"""Fails on alternate attempts regardless of seed: a race, an uninitialised
read, a timeout near the limit."""
return attempt % 2 == 1
print("same seed, five attempts:")
print(" seed 3:", [truly_flaky(3, a) for a in range(5)])
print(" seed 7:", [truly_flaky(7, a) for a in range(5)])
print()
DISCRIMINATOR = [
("re-run the SAME seed", "same result every time", "seed-dependent — debug it"),
("re-run the SAME seed", "different results", "flaky — fix the test or the race"),
]
for what, observed, verdict in DISCRIMINATOR:
print(f" {what} -> {observed:<24} {verdict}")same seed, five attempts:
seed 3: [False, True, False, True, False]
seed 7: [False, True, False, True, False]
re-run the SAME seed -> same result every time seed-dependent — debug it
re-run the SAME seed -> different results flaky — fix the test or the raceThe discriminator is one experiment: re-run the identical seed. This is exactly Chapter 29.6 Section 4's argument in a different setting — pinning the seed converts a question about noise into a question with a deterministic answer — and it means a flakiness detector needs repeated runs at a fixed seed, not a sweep over seeds. A sweep measures the design's sensitivity to stimulus; repetition measures the test's reliability.
8. How Many Runs Before You Call It Flaky
Section 3 showed one run cannot detect flakiness. The next question is how many can, and it is answerable exactly:
def miss_probability(p, n):
"""P(a test with true fail rate p shows zero failures in n runs)."""
return (1 - p) ** n
print(f"{'true rate':>10} {'runs':>6} {'P(miss entirely)':>18}")
for p in (0.10, 0.05, 0.02):
for n in (1, 5, 20):
print(f"{p:>10.2f} {n:>6} {miss_probability(p, n):>17.2%}")
print()
print("so 20 runs of a 10%-flaky test miss it",
f"{miss_probability(0.10, 20):.2%} of the time")
print("and 20 runs of a 5%-flaky test miss it",
f"{miss_probability(0.05, 20):.2%} of the time") true rate runs P(miss entirely)
0.10 1 90.00%
0.10 5 59.05%
0.10 20 12.16%
0.05 1 95.00%
0.05 5 77.38%
0.05 20 35.85%
0.02 1 98.00%
0.02 5 90.39%
0.02 20 66.76%
so 20 runs of a 10%-flaky test miss it 12.16% of the time
and 20 runs of a 5%-flaky test miss it 35.85% of the timeTwenty runs miss a one-in-twenty flaky test more than a third of the time. That is the number which explains why flaky tests survive for months: the detection campaign that "found nothing" had a coin-flip's chance of finding nothing regardless.
Inverting it gives the sample size a policy should ask for:
import math
def runs_needed(p, confidence=0.95):
"""Runs required so P(missing a test with rate p) <= 1 - confidence."""
return math.ceil(math.log(1 - confidence) / math.log(1 - p))
print(f"{'true rate':>10} {'runs for 95% confidence':>26}")
for p in (0.20, 0.10, 0.05, 0.02):
print(f"{p:>10.2f} {runs_needed(p):>26}")
print()
print("detecting a 1-in-20 flake with 95% confidence needs", runs_needed(0.05), "runs")
print("which for a 400-job regression is", 59 * 1, "runs of ONE test, not of the suite") true rate runs for 95% confidence
0.20 14
0.10 29
0.05 59
0.02 149
detecting a 1-in-20 flake with 95% confidence needs 59 runs
which for a 400-job regression is 59 runs of ONE test, not of the suiteFifty-nine runs for a one-in-twenty flake, and the last line of the program is the point that makes it affordable: that is fifty-nine runs of one suspect test, not fifty-nine runs of the regression. Flakiness investigation is a targeted campaign against a short list, which is why Section 4's moved_seed signal matters — it is what produces the short list from a nightly run that cannot itself measure flakiness at all.
9. The Comparison, Assembled
from dataclasses import dataclass
from typing import Dict, List, Optional, Set, Tuple
@dataclass
class Night:
ran: Set[str]
failed_pairs: Set[Tuple[str, int]]
@property
def failed(self):
return {t for t, _ in self.failed_pairs}
def compare_nights(today: Night, yesterday: Optional[Night]) -> dict:
if yesterday is None:
return {"comparable": False,
"reason": "no previous run",
"failing_now": sorted(today.failed)}
new = sorted((today.failed - yesterday.failed) & yesterday.ran)
fixed = sorted((yesterday.failed - today.failed) & today.ran)
still = sorted(today.failed & yesterday.failed)
moved = sorted(
t for t in still
if {s for n, s in today.failed_pairs if n == t}
!= {s for n, s in yesterday.failed_pairs if n == t})
return {
"comparable": True,
"new": new,
"fixed": fixed,
"still_failing": still,
"same_seed": sorted(today.failed_pairs & yesterday.failed_pairs),
"moved_seed": moved,
"new_tests_failing": sorted((today.failed - yesterday.failed) - yesterday.ran),
"vanished_while_failing": sorted((yesterday.failed - today.failed) - today.ran),
"testlist_grew_by": len(today.ran - yesterday.ran),
"testlist_shrank_by": len(yesterday.ran - today.ran),
"flaky_candidates": moved,
"flakiness_measurable_from_this_run": False,
}
YESTERDAY = Night(
ran={"axi_smoke", "axi_burst", "dma_long", "flaky_mess"},
failed_pairs={("axi_burst", 7731), ("axi_smoke", 1839), ("flaky_mess", 11)},
)
TODAY = Night(
ran={"axi_smoke", "axi_burst", "dma_long", "reg_rw"},
failed_pairs={("axi_burst", 7731), ("axi_smoke", 2048), ("reg_rw", 4410)},
)
r = compare_nights(TODAY, YESTERDAY)
for k in ("new", "fixed", "still_failing", "same_seed", "moved_seed",
"new_tests_failing", "vanished_while_failing",
"testlist_grew_by", "testlist_shrank_by",
"flaky_candidates", "flakiness_measurable_from_this_run"):
print(f" {k:<36} {r[k]}")
print()
naive_new = sorted(TODAY.failed - YESTERDAY.failed)
naive_fixed = sorted(YESTERDAY.failed - TODAY.failed)
print("the annotation would report new :", naive_new)
print("the annotation would report fixed:", naive_fixed)
print("this comparator reports new :", r["new"])
print("this comparator reports fixed :", r["fixed"])
print()
print("no baseline:", compare_nights(TODAY, None)["reason"]) new []
fixed []
still_failing ['axi_burst', 'axi_smoke']
same_seed [('axi_burst', 7731)]
moved_seed ['axi_smoke']
new_tests_failing ['reg_rw']
vanished_while_failing ['flaky_mess']
testlist_grew_by 1
testlist_shrank_by 1
flaky_candidates ['axi_smoke']
flakiness_measurable_from_this_run False
the annotation would report new : ['reg_rw']
the annotation would report fixed: ['flaky_mess']
this comparator reports new : []
this comparator reports fixed : []
no baseline: no previous runRead the two pairs of lines at the bottom together. The annotation reports one regression and one fix on a night that had neither — reg_rw is a new test that does not pass yet and flaky_mess was removed from the testlist. A team acting on the annotation's output spends the morning looking for a commit that broke reg_rw and congratulating somebody for fixing flaky_mess.
The explicit flakiness_measurable_from_this_run: False is there because Section 3's lesson deserves to be in the data rather than in a comment: this run cannot measure flakiness, it offers a candidate list, and the campaign of Section 8 is a separate job.
10. Quarantine Is a Decision With an Expiry
Section 8 priced the campaign that confirms a test is flaky. What happens next is a policy question, and getting it wrong is how a flaky test becomes permanent.
OPTIONS = [
("leave it in", "the queue has a permanent false entry", "triage cost every night"),
("delete it", "30.6's problem: looks fixed", "coverage silently lost"),
("quarantine with no expiry", "quiet, and forgotten", "the bug ships"),
("quarantine with an expiry", "quiet, and comes back", "somebody must act by a date"),
]
print(f"{'option':<28} {'effect':<34} cost")
for name, effect, cost in OPTIONS:
print(f"{name:<28} {effect:<34} {cost}")
print()
print("options that keep the signal and stop the noise:",
[o[0] for o in OPTIONS if "expiry" in o[0]])option effect cost
leave it in the queue has a permanent false entry triage cost every night
delete it 30.6's problem: looks fixed coverage silently lost
quarantine with no expiry quiet, and forgotten the bug ships
quarantine with an expiry quiet, and comes back somebody must act by a date
options that keep the signal and stop the noise: ['quarantine with no expiry', 'quarantine with an expiry']Only the fourth option has a mechanism behind it rather than an intention. A quarantine entry carries an owner and a date, and the report enforces the date:
QUARANTINE = {
"flaky_mess": {"owner": "a.patel", "until": "2026-03-01",
"evidence": "7 failures in 59 runs at seed 11",
"rate": 7 / 59},
"timing_race": {"owner": "r.kaur", "until": "2026-02-10",
"evidence": "3 failures in 29 runs at seed 4",
"rate": 3 / 29},
}
TODAY = "2026-02-18"
def status(entry, today):
return "EXPIRED" if entry["until"] < today else "active"
print(f"{'test':<14} {'owner':<10} {'until':<12} {'rate':>7} state")
for test, e in sorted(QUARANTINE.items()):
print(f"{test:<14} {e['owner']:<10} {e['until']:<12} "
f"{e['rate']:>6.1%} {status(e, TODAY)}")
expired = [t for t, e in QUARANTINE.items() if status(e, TODAY) == "EXPIRED"]
print()
print("expired entries:", expired)
print("these must appear in the report as failures again, not as quarantine.")
print("entries with no evidence field would be refused:", 0)test owner until rate state
flaky_mess a.patel 2026-03-01 11.9% active
timing_race r.kaur 2026-02-10 10.3% EXPIRED
expired entries: ['timing_race']
these must appear in the report as failures again, not as quarantine.
entries with no evidence field would be refused: 0An expired quarantine entry is a report item, and that single rule is what separates quarantine from deletion. The test is still running, its result is still recorded, and the only thing suppressed is its contribution to the new-failure list — until the date passes, at which point the suppression lifts itself.
The evidence field is there for the same reason Chapter 29.5 Section 4 demanded a reason on every coverage exclusion: 7 failures in 59 runs is a claim somebody can check, and "it's flaky" is not. Quarantining on a hunch and quarantining on a measured rate look identical a month later unless the number is stored.
11. Predict, Run, Observe
from fractions import Fraction
# A
print("A:", [r for r in (Fraction(0, 1), Fraction(1, 1)) if 0 < r < 1])
# B
Y, T = {("a", 1)}, {("a", 2)}
print("B:", sorted(T - Y), sorted({n for n, _ in T} - {n for n, _ in Y}))
# C
print("C:", sorted({"x", "new"} - {"x"}), sorted(({"x", "new"} - {"x"}) & {"x", "old"}))
# D
print("D:", f"{(1 - 0.05) ** 20:.4f}")The verified output:
A: []
B: [('a', 2)] []
C: ['new'] []
D: 0.3585A — the flaky predicate's satisfying set on one run per test. Empty, always.
B — the pair view sees the failure move and the name view sees nothing.
C — a naive new set contains a test that never ran yesterday; intersecting with what ran empties it.
D — a 5%-flaky test shows nothing in twenty runs 35.85% of the time.
12. Exercises
Exercise 1 — three categories. Classify last night's failures into new, known and flaky-candidate, and report the counts.
Exercise 2 — the expensive category. Estimate how many engineer-hours a single flaky test cost your team last quarter.
Exercise 3 — your flaky set. Look at your flow's flaky-test output over the last thirty runs and say whether it has ever been non-empty.
Exercise 4 — prove the predicate can fire. Construct input on which your flakiness check returns something, and say how many runs per test it needed.
Exercise 5 — achievable rates. For your runs-per-test, list every fail rate a single test can show.
Exercise 6 — a check with no variance. Name another check in your flow whose output has never changed, and decide whether it is healthy or dead.
Exercise 7 — name or pair. Say which of the two comparison views your report uses, and which question your reviewers are actually asking.
Exercise 8 — a failure that moved. Find a test that failed on different seeds on two nights, and say what your report said about it.
Exercise 9 — the moved-seed signal. Add moved_seed to your comparison and report its size for a week.
Exercise 10 — same seed twice. Find a failure that reproduced at the identical seed, and say why that is the good case.
Exercise 11 — add a test. Add a failing test to your testlist and record exactly what your report claims about it.
Exercise 12 — the honest new set. Implement the intersection with yesterday's ran-set and show the difference on real data.
Exercise 13 — a new test is a different conversation. Write the two sentences you would send about a new test that does not pass yet.
Exercise 14 — delete a failing test. On a scratch branch, remove a failing test and observe your report. Report what it said.
Exercise 15 — the cheapest green. Explain in two sentences why deleting a test is indistinguishable from fixing it, and what makes it visible.
Exercise 16 — testlist deltas. Add grew-by and shrank-by counts to your report and say what they let a reader expect.
Exercise 17 — seed-dependent or flaky. Pick a test that fails rarely and run the discriminating experiment. Report the verdict.
Exercise 18 — one seed in twenty. Explain why a test failing at exactly one seed out of twenty may be the best result of the night.
Exercise 19 — what a sweep measures. Say what a seed sweep measures and what repetition at a fixed seed measures, in one sentence each.
Exercise 20 — your miss probability. For your current re-run policy, compute the probability of missing a 10% flake.
Exercise 21 — the campaign size. Compute the runs needed to catch your worst suspected flake rate at 95% confidence.
Exercise 22 — targeted, not global. Explain why 59 runs is affordable, with reference to what is being re-run.
Exercise 23 — found nothing. Take a past investigation that concluded a test was not flaky and compute what its miss probability was.
Exercise 24 — the morning after. Write what a team does on the morning the annotation reports one new failure and one fix on a night that had neither.
13. Summary
- A failing test is one of three things and only one is a regression; the flaky one is expensive because it costs attention repeatedly.
0 < fail_rate(t) < 1cannot fire on one run per test. The achievable rates are 0 and 1, so the predicate is vacuously false andflakyis empty every night.- A check whose output never varies should be assumed broken. A flaky set empty for six months is evidence of a predicate nobody exercised.
- A set difference on names loses the seed. A failure that moved from seed 1839 to 2048 reports as "nothing changed".
- Compute both views: the name view for the triage queue, the pair view for whether yesterday's debug still applies — and
moved_seedis the flakiness candidate list. - A test that did not run yesterday is not a new failure. Adding one test to the testlist manufactured a regression.
- A deleted test looks fixed, which is the cheapest possible way to make a report go green — so report the count, not the suspicion.
- Intersect with what ran on both sides, and report
new_tests_failingandvanished_while_failingas their own categories. - Report the testlist deltas. A list that both grew and shrank by one is what makes the night interpretable.
- Flaky is not seed-dependent. A test that always fails at seed 7 is perfectly reproducible, and calling it flaky throws away a real bug.
- The discriminator is one experiment: re-run the identical seed. A sweep measures the design's sensitivity; repetition measures the test's reliability.
- Twenty runs miss a 1-in-20 flake 35.85% of the time, which is why flaky tests survive investigations that "found nothing".
- Catching a 5% flake at 95% confidence needs 59 runs — of one suspect test, not of the suite, which is what makes the campaign affordable.
- On the worked night the annotation reported one regression and one fix where there were none.
- Quarantine needs an owner and an expiry, or it is deletion with extra steps.
- An expired quarantine entry is a report item. The suppression lifts itself, which is what separates quarantine from deletion.
- Store the evidence — "7 failures in 59 runs" — not the conclusion. A hunch and a measured rate look identical a month later.
The next chapter writes these facts to a file another program reads, where a different class of mistake lives.
Where this fits
Part of the Python curriculum.