Python Part II · Module 33 · Testing and Debugging Automation
Golden Log Snippets as Test Data
Chapter 33.3's fixture was invented, which is fine for pinning logic and useless for the failure this chapter is about: a tool upgrade that renames a message, changing nothing in your repository and breaking your parser overnight. Real extracts kept beside the tests turn that into a failed test instead of a wrong verdict. The canonical annotation sets them up in two lines, and one of those lines can make an entire fixture suite pass without executing a single assertion.
Intermediate20 min readPythonPart IIAutomationTestingFixtures
Part II · Module 33 · Chapter 33.4 · Golden Log Snippets as Test Data
1. What an Invented Fixture Cannot Catch
FAILURES = [
("a logic error in the parser", "yes", "33.3's invented fixture is enough"),
("an unhandled status", "yes", "a table of cases — 33.3 §6"),
("malformed input", "yes", "33.5, with constructed input"),
("a tool renaming a message", "NO", "needs a real extract — §5"),
("a tool adding a field", "NO", "needs a real extract"),
("a second tool's different format", "NO", "needs that tool's extract"),
]
print(f"{'the failure':<34} {'invented fixture catches it':<28} why")
for what, catches, why in FAILURES:
print(f"{what:<34} {catches:<28} {why}")
print()
print("failures an invented fixture catches:", 3)
print("failures needing a real extract :", 3)
print("and the three it misses all come from:", "outside your repository")the failure invented fixture catches it why
a logic error in the parser yes 33.3's invented fixture is enough
an unhandled status yes a table of cases — 33.3 §6
malformed input yes 33.5, with constructed input
a tool renaming a message NO needs a real extract — §5
a tool adding a field NO needs a real extract
a second tool's different format NO needs that tool's extract
failures an invented fixture catches: 3
failures needing a real extract : 3
and the three it misses all come from: outside your repositoryThe three it misses all originate outside your repository, which is why they are the ones nothing else protects against. A logic error arrives in a commit somebody reviews; a renamed message arrives in a tool release nobody on the project has read.
2. The Annotation
for sample in Path("tests/logs").glob("*.log"):
check(sample)from pathlib import Path
LOGS = Path("tests/logs")
LOGS.mkdir(parents=True, exist_ok=True)
(LOGS / "xrun_2025.03_two_errors.log").write_text(
"UVM_INFO tb.sv(42) @ 0: reset released\n"
"UVM_ERROR drv.sv(88) @ 141900: bad handshake\n"
"UVM_ERROR drv.sv(91) @ 158200: bad handshake\n"
"TEST DONE\n", encoding="utf-8")
(LOGS / "xrun_2025.03_clean.log").write_text(
"UVM_INFO tb.sv(42) @ 0: reset released\n"
"TEST DONE\n", encoding="utf-8")
def check(path):
text = path.read_text(encoding="utf-8")
assert text, f"{path.name} is empty"
return path.name
checked = []
for sample in sorted(LOGS.glob("*.log")):
checked.append(check(sample))
print("fixtures found:", len(checked))
for name in checked:
print(" ", name)
print()
print("assertions executed:", len(checked))fixtures found: 2
xrun_2025.03_clean.log
xrun_2025.03_two_errors.log
assertions executed: 2Two fixtures, two checks, and the structure is right: real extracts in a directory, discovered rather than listed, so adding a fixture needs no change to the test. The next two sections are about the two words doing the least work — glob and check.
3. An Empty Glob Is a Green Test
import io
import unittest
from pathlib import Path
def check(path):
text = path.read_text(encoding="utf-8")
assert "TEST DONE" in text, f"{path.name}: no end banner"
class FixtureTests(unittest.TestCase):
def test_all_fixtures(self):
for sample in sorted(Path("tests/logs").glob("*.log")):
check(sample)
def run():
suite = unittest.TestLoader().loadTestsFromTestCase(FixtureTests)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"ok": res.wasSuccessful()}
SCENARIOS = [
("two real fixtures", ["a.log", "b.log"]),
("directory empty", []),
("files named .txt", ["a.txt", "b.txt"]),
]
for label, names in SCENARIOS:
import shutil
shutil.rmtree("tests", ignore_errors=True)
Path("tests/logs").mkdir(parents=True, exist_ok=True)
for n in names:
(Path("tests/logs") / n).write_text("UVM_INFO x\nTEST DONE\n",
encoding="utf-8")
found = len(list(Path("tests/logs").glob("*.log")))
print(f"{label:<22} files={len(names)} matched={found} "
f"suite={run()}")
import shutil
shutil.rmtree("tests", ignore_errors=True)
print(f"{'directory missing':<22} files=0 matched=0 suite={run()}")two real fixtures files=2 matched=2 suite={'run': 1, 'failures': 0, 'ok': True}
directory empty files=0 matched=0 suite={'run': 1, 'failures': 0, 'ok': True}
files named .txt files=2 matched=0 suite={'run': 1, 'failures': 0, 'ok': True}
directory missing files=0 matched=0 suite={'run': 1, 'failures': 0, 'ok': True}All four scenarios pass, and three of them tested nothing at all.
An empty directory, a directory of files with the wrong extension, and a directory that does not exist all produce zero iterations — so the loop body never runs, no assertion is evaluated, and unittest reports a passing test because the method returned without raising.
This is Chapter 30.3 Section 3's vacuous predicate in a new form, and it is worse here for a specific reason: a directory called tests/logs is evidence that somebody intended fixtures. A reader seeing a green fixture suite reasonably concludes the fixtures ran. The empty flaky-test set at least looked like an absence of findings; this looks like coverage.
The fix is one assertion, and it belongs before the loop:
import io
import shutil
import unittest
from pathlib import Path
MIN_FIXTURES = 2
class FixtureTests(unittest.TestCase):
def test_fixtures_exist(self):
found = sorted(Path("tests/logs").glob("*.log"))
self.assertGreaterEqual(
len(found), MIN_FIXTURES,
f"expected at least {MIN_FIXTURES} fixtures, found {len(found)}")
def test_all_fixtures_have_a_banner(self):
found = sorted(Path("tests/logs").glob("*.log"))
self.assertTrue(found, "no fixtures matched tests/logs/*.log")
for sample in found:
with self.subTest(fixture=sample.name):
self.assertIn("TEST DONE",
sample.read_text(encoding="utf-8"))
def run():
suite = unittest.TestLoader().loadTestsFromTestCase(FixtureTests)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
names = [t.id().split(".")[-1] for t, _ in res.failures]
return {"run": res.testsRun, "failures": len(res.failures), "which": names}
for label, names in (("two fixtures", ["a.log", "b.log"]), ("none", [])):
shutil.rmtree("tests", ignore_errors=True)
Path("tests/logs").mkdir(parents=True, exist_ok=True)
for n in names:
(Path("tests/logs") / n).write_text("UVM_INFO x\nTEST DONE\n",
encoding="utf-8")
print(f"{label:<14} {run()}")two fixtures {'run': 2, 'failures': 0, 'which': []}
none {'run': 2, 'failures': 2, 'which': ['test_all_fixtures_have_a_banner', 'test_fixtures_exist']}Two failures where there were none. A count assertion with a floor, plus a non-empty assertion inside the iterating test, and the empty directory now reports what it is.
4. check(sample) Needs an Expected Value
The second defect is what check can possibly assert with only a log in hand:
from pathlib import Path
def parse(text):
lines = text.splitlines()
return {"errors": sum(1 for l in lines if l.startswith("UVM_ERROR")),
"fatals": sum(1 for l in lines if l.startswith("UVM_FATAL")),
"finished": "TEST DONE" in text}
def parse_completely_wrong(text):
"""Returns a plausible record that is never right."""
return {"errors": 0, "fatals": 0, "finished": True}
SAMPLE = ("UVM_INFO started\n"
"UVM_ERROR drv.sv(88): bad handshake\n"
"UVM_ERROR drv.sv(91): bad handshake\n"
"TEST DONE\n")
def check_no_crash(parse_fn, text):
"""What a fixture with no expected value can assert."""
r = parse_fn(text)
assert isinstance(r, dict)
assert "errors" in r
return True
for name, fn in (("the real parser", parse),
("a parser that always says 0 errors", parse_completely_wrong)):
ok = check_no_crash(fn, SAMPLE)
print(f"{name:<36} did-not-crash check: {ok} record={fn(SAMPLE)}")the real parser did-not-crash check: True record={'errors': 2, 'fatals': 0, 'finished': True}
a parser that always says 0 errors did-not-crash check: True record={'errors': 0, 'fatals': 0, 'finished': True}A parser that always reports zero errors passes the did-not-crash check. That is all a fixture without an expected value can test, and it is a property every wrong parser also has.
So each fixture needs a paired expectation, and a sibling file is the simplest form:
import json
import shutil
from pathlib import Path
shutil.rmtree("tests", ignore_errors=True)
LOGS = Path("tests/logs")
LOGS.mkdir(parents=True, exist_ok=True)
FIXTURES = {
"xrun_2025.03_two_errors": (
("UVM_INFO tb.sv(42) @ 0: reset released\n"
"UVM_ERROR drv.sv(88) @ 141900: bad handshake\n"
"UVM_ERROR drv.sv(91) @ 158200: bad handshake\n"
"TEST DONE\n"),
{"errors": 2, "fatals": 0, "finished": True, "status": "FAIL"}),
"xrun_2025.03_clean": (
("UVM_INFO tb.sv(42) @ 0: reset released\nTEST DONE\n"),
{"errors": 0, "fatals": 0, "finished": True, "status": "PASS"}),
"xrun_2025.03_truncated": (
("UVM_INFO tb.sv(42) @ 0: reset released\n"),
{"errors": 0, "fatals": 0, "finished": False, "status": "NO_FINISH"}),
}
for stem, (text, expected) in FIXTURES.items():
(LOGS / f"{stem}.log").write_text(text, encoding="utf-8")
(LOGS / f"{stem}.json").write_text(json.dumps(expected, indent=2) + "\n",
encoding="utf-8")
print("files in tests/logs:")
for p in sorted(LOGS.iterdir()):
print(f" {p.name:<34} {p.stat().st_size:>4} bytes")
logs = sorted(LOGS.glob("*.log"))
jsons = sorted(LOGS.glob("*.json"))
print()
print("logs :", len(logs))
print("jsons:", len(jsons))
print("every log has an expectation:",
{p.stem for p in logs} == {p.stem for p in jsons})files in tests/logs:
xrun_2025.03_clean.json 73 bytes
xrun_2025.03_clean.log 49 bytes
xrun_2025.03_truncated.json 79 bytes
xrun_2025.03_truncated.log 39 bytes
xrun_2025.03_two_errors.json 73 bytes
xrun_2025.03_two_errors.log 139 bytes
logs : 3
jsons: 3
every log has an expectation: TrueThree pairs, and the last line is itself a test worth keeping: a .log without a .json is a fixture nobody wrote an expectation for, which is the single-file version of Section 3's empty directory.
5. The Tool Upgrade
This is the failure the chapter exists for. A tool release renames a message, nothing in the repository changes, and the parser stops seeing errors:
import json
import shutil
from pathlib import Path
def parse(text):
"""Written against version 2024.09, which said UVM_ERROR."""
lines = text.splitlines()
return {"errors": sum(1 for l in lines if l.startswith("UVM_ERROR")),
"finished": "TEST DONE" in text,
"status": "FAIL" if any(l.startswith("UVM_ERROR") for l in lines)
else ("PASS" if "TEST DONE" in text else "NO_FINISH")}
OLD = ("UVM_INFO tb.sv(42) @ 0: reset released\n"
"UVM_ERROR drv.sv(88) @ 141900: bad handshake\n"
"TEST DONE\n")
NEW = ("UVM_INFO tb.sv(42) @ 0: reset released\n"
"UVM_ERR drv.sv(88) @ 141900: bad handshake\n"
"TEST DONE\n")
EXPECTED = {"errors": 1, "finished": True, "status": "FAIL"}
for label, text in (("2024.09 log", OLD), ("2025.03 log", NEW)):
got = parse(text)
agrees = got == EXPECTED
print(f"{label:<14} parsed={got}")
print(f"{'':<14} matches the expectation: {agrees}")2024.09 log parsed={'errors': 1, 'finished': True, 'status': 'FAIL'}
matches the expectation: True
2025.03 log parsed={'errors': 0, 'finished': True, 'status': 'PASS'}
matches the expectation: FalseThe same parser reports FAIL on the old log and PASS on the new one, and the only difference is that UVM_ERROR became UVM_ERR.
Now the point of keeping a fixture per version:
import io
import json
import shutil
import unittest
from pathlib import Path
def parse(text):
lines = text.splitlines()
errors = sum(1 for l in lines if l.startswith("UVM_ERROR"))
return {"errors": errors, "finished": "TEST DONE" in text,
"status": "FAIL" if errors
else ("PASS" if "TEST DONE" in text else "NO_FINISH")}
shutil.rmtree("tests", ignore_errors=True)
LOGS = Path("tests/logs")
LOGS.mkdir(parents=True, exist_ok=True)
PER_VERSION = {
"xrun_2024.09_one_error": (
"UVM_INFO started\nUVM_ERROR drv.sv(88): bad handshake\nTEST DONE\n",
{"errors": 1, "finished": True, "status": "FAIL"}),
"xrun_2025.03_one_error": (
"UVM_INFO started\nUVM_ERR drv.sv(88): bad handshake\nTEST DONE\n",
{"errors": 1, "finished": True, "status": "FAIL"}),
}
for stem, (text, exp) in PER_VERSION.items():
(LOGS / f"{stem}.log").write_text(text, encoding="utf-8")
(LOGS / f"{stem}.json").write_text(json.dumps(exp) + "\n", encoding="utf-8")
class GoldenFixtures(unittest.TestCase):
def test_fixtures_exist(self):
self.assertGreaterEqual(len(list(LOGS.glob("*.log"))), 2)
def test_every_fixture_parses_as_expected(self):
logs = sorted(LOGS.glob("*.log"))
self.assertTrue(logs, "no fixtures found")
for log in logs:
exp_path = log.with_suffix(".json")
with self.subTest(fixture=log.name):
self.assertTrue(exp_path.is_file(),
f"{log.name} has no .json expectation")
expected = json.loads(exp_path.read_text(encoding="utf-8"))
self.assertEqual(parse(log.read_text(encoding="utf-8")),
expected)
suite = unittest.TestLoader().loadTestsFromTestCase(GoldenFixtures)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
print("tests run:", res.testsRun, " failures:", len(res.failures))
for test, trace in res.failures:
bad = [l for l in trace.splitlines() if "SubTest" in l or "AssertionError" in l]
print(" failed:", bad[0].strip()[:72] if bad else "?")
print()
print("the 2024.09 fixture :", "passes — the parser handles it")
print("the 2025.03 fixture :", "fails — the parser does not")
print("what the regression would have reported for a 2025.03 log:", "PASS")tests run: 2 failures: 1
failed: AssertionError: {'errors': 0, 'finished': True, 'status': 'PASS'} != {'e
the 2024.09 fixture : passes — the parser handles it
the 2025.03 fixture : fails — the parser does not
what the regression would have reported for a 2025.03 log: PASSOne failed test instead of a night of wrong verdicts. That is the whole argument of the canonical purpose, and the mechanism is unremarkable: keep the extract from each version you run against, and the day the tool changes, the suite says so.
Note the ordering this implies for a flow. Chapter 33.1 Section 8 said to run the suite nightly before the regression; this is why. The fixture test fails in seconds, before four hundred jobs are launched against a parser that no longer understands their logs.
6. Why Ten Lines
SIZES = [
(400, "a captured log, unedited", "no", "nobody reads it in a diff"),
(40, "trimmed to one scenario", "maybe", "still too long for review"),
(10, "the minimum that exercises the case", "yes", "readable in a failure"),
(1, "one line", "no", "cannot show a multi-line banner or context"),
]
print(f"{'lines':>6} {'what it is':<36} {'keep':<7} why")
for n, what, keep, why in SIZES:
print(f"{n:>6} {what:<36} {keep:<7} {why}")
print()
REASONS = [
("read by a human during a failure", "a failing fixture is opened, not skimmed"),
("diffed in review", "a 400-line fixture change is unreviewable"),
("committed to a shared repository", "design names must not leak — see the callout"),
]
for what, why in REASONS:
print(f" {what:<36} {why}") lines what it is keep why
400 a captured log, unedited no nobody reads it in a diff
40 trimmed to one scenario maybe still too long for review
10 the minimum that exercises the case yes readable in a failure
1 one line no cannot show a multi-line banner or context
read by a human during a failure a failing fixture is opened, not skimmed
diffed in review a 400-line fixture change is unreviewable
committed to a shared repository design names must not leak — see the calloutTen lines is the canonical purpose's number and it is the right order of magnitude for all three reasons. The fourth row matters too: a single line cannot exercise a parser that tracks state across lines, which is what Chapter 20.15's section-spanning parsing needed.
7. Provenance on a Fixture
A failing fixture test has two possible causes and the fixture must say which is likelier:
import json
import shutil
from pathlib import Path
shutil.rmtree("tests", ignore_errors=True)
LOGS = Path("tests/logs")
LOGS.mkdir(parents=True, exist_ok=True)
META = {
"tool": "xrun",
"tool_version": "2025.03",
"captured_from": "runs/axi_burst_48210/sim.log",
"trimmed": True,
"renamed_identifiers": True,
"why": "one error, one info, and the end banner",
}
EXPECTED = {"errors": 1, "finished": True, "status": "FAIL"}
(LOGS / "xrun_2025.03_one_error.log").write_text(
"UVM_INFO tb.sv(42) @ 0: reset released\n"
"UVM_ERROR dut.sv(88) @ 141900: bad handshake on din\n"
"TEST DONE\n", encoding="utf-8")
(LOGS / "xrun_2025.03_one_error.json").write_text(
json.dumps({"meta": META, "expected": EXPECTED}, indent=2) + "\n",
encoding="utf-8")
doc = json.loads((LOGS / "xrun_2025.03_one_error.json")
.read_text(encoding="utf-8"))
print("expectation:", doc["expected"])
print()
print("provenance:")
for k, v in doc["meta"].items():
print(f" {k:<22} {v!r}")
print()
QUESTIONS = [
("is the parser wrong?", "compare against a newer fixture"),
("is the fixture stale?", "meta.tool_version says which release it is from"),
("where did it come from?", "meta.captured_from"),
("has it been edited?", "meta.trimmed and meta.renamed_identifiers"),
]
for q, a in QUESTIONS:
print(f" {q:<26} {a}")expectation: {'errors': 1, 'finished': True, 'status': 'FAIL'}
provenance:
tool 'xrun'
tool_version '2025.03'
captured_from 'runs/axi_burst_48210/sim.log'
trimmed True
renamed_identifiers True
why 'one error, one info, and the end banner'
is the parser wrong? compare against a newer fixture
is the fixture stale? meta.tool_version says which release it is from
where did it come from? meta.captured_from
has it been edited? meta.trimmed and meta.renamed_identifiersSix metadata fields, and each answers a question a reader of a failing test asks. This is Chapter 29.1 Section 8's layout field applied to test data: when a number looks wrong the first question is which parsing path produced it, and when a fixture fails the first question is which tool release it represents.
The why field is the one that ages best. Six months later, nobody remembers whether a fixture exists to pin the error count, the banner text or the timestamp format, and a trimmed fixture gives no hint — so a reader tempted to "fix" a failing fixture by editing it has no way to know what it was protecting.
8. The Fixture Suite, Assembled
import io
import json
import shutil
import unittest
from pathlib import Path
MIN_FIXTURES = 3
LOGS = Path("tests/logs")
def parse(text):
lines = text.splitlines()
errors = sum(1 for l in lines if l.startswith("UVM_ERROR"))
fatals = sum(1 for l in lines if l.startswith("UVM_FATAL"))
finished = "TEST DONE" in text
known = ("UVM_INFO", "UVM_ERROR", "UVM_FATAL", "TEST DONE")
unparsed = sum(1 for l in lines if l.strip() and not l.startswith(known))
if not text.strip():
status = "EMPTY_LOG"
elif errors or fatals:
status = "FAIL"
elif not finished:
status = "NO_FINISH"
else:
status = "PASS"
return {"errors": errors, "fatals": fatals, "finished": finished,
"unparsed": unparsed, "status": status}
def write_fixture(stem, text, expected, meta):
(LOGS / f"{stem}.log").write_text(text, encoding="utf-8")
(LOGS / f"{stem}.json").write_text(
json.dumps({"meta": meta, "expected": expected}, indent=2) + "\n",
encoding="utf-8")
shutil.rmtree("tests", ignore_errors=True)
LOGS.mkdir(parents=True, exist_ok=True)
BASE_META = {"tool": "xrun", "tool_version": "2025.03",
"captured_from": "runs/axi_burst_48210/sim.log",
"trimmed": True, "renamed_identifiers": True}
write_fixture("xrun_2025.03_one_error",
"UVM_INFO tb.sv(42) @ 0: reset released\n"
"UVM_ERROR dut.sv(88) @ 141900: bad handshake on din\n"
"TEST DONE\n",
{"errors": 1, "fatals": 0, "finished": True,
"unparsed": 0, "status": "FAIL"},
{**BASE_META, "why": "the error prefix and the banner"})
write_fixture("xrun_2025.03_clean",
"UVM_INFO tb.sv(42) @ 0: reset released\nTEST DONE\n",
{"errors": 0, "fatals": 0, "finished": True,
"unparsed": 0, "status": "PASS"},
{**BASE_META, "why": "the PASS path"})
write_fixture("xrun_2025.03_unknown_prefix",
"UVM_INFO tb.sv(42) @ 0: reset released\n"
"Simulation complete via $finish\n"
"TEST DONE\n",
{"errors": 0, "fatals": 0, "finished": True,
"unparsed": 1, "status": "PASS"},
{**BASE_META, "why": "the unparsed counter — 20.6"})
class GoldenFixtures(unittest.TestCase):
def test_enough_fixtures_exist(self):
found = sorted(LOGS.glob("*.log"))
self.assertGreaterEqual(len(found), MIN_FIXTURES,
f"found {len(found)} fixtures")
def test_every_log_has_an_expectation(self):
logs = sorted(LOGS.glob("*.log"))
self.assertTrue(logs, "no fixtures matched")
for log in logs:
with self.subTest(fixture=log.name):
self.assertTrue(log.with_suffix(".json").is_file())
def test_every_fixture_has_provenance(self):
jsons = sorted(LOGS.glob("*.json"))
self.assertTrue(jsons, "no expectations matched")
for j in jsons:
with self.subTest(fixture=j.name):
doc = json.loads(j.read_text(encoding="utf-8"))
for field in ("tool", "tool_version", "why"):
self.assertIn(field, doc["meta"])
def test_every_fixture_parses_as_expected(self):
logs = sorted(LOGS.glob("*.log"))
self.assertTrue(logs, "no fixtures matched")
for log in logs:
with self.subTest(fixture=log.name):
doc = json.loads(log.with_suffix(".json")
.read_text(encoding="utf-8"))
self.assertEqual(parse(log.read_text(encoding="utf-8")),
doc["expected"])
suite = unittest.TestLoader().loadTestsFromTestCase(GoldenFixtures)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
print("tests run :", res.testsRun)
print("failures :", len(res.failures))
print("ok :", res.wasSuccessful())
print()
print("fixtures :", len(list(LOGS.glob('*.log'))))
print("facts asserted per fixture:", 5)
print("guards against an empty directory:", 3)tests run : 4
failures : 0
ok : True
fixtures : 3
facts asserted per fixture: 5
guards against an empty directory: 3Four tests, three fixtures, five facts each, and three separate guards against the empty-directory failure — a minimum count, a non-empty assertion inside each iterating test, and the pairing check that a .log without a .json fails.
The third test is the one that would be easiest to omit and hardest to add later. Checking that every fixture carries tool, tool_version and why costs four lines now; retrofitting provenance onto thirty undocumented fixtures is an afternoon of archaeology.
9. Where to Cut a Real Log
Section 6 argued for ten lines. A real log is forty thousand, so the practical question is which ten — and the answer is not "the first ten", because the facts a fixture pins depend entirely on where you cut:
FULL = ("UVM_INFO @0: top start\n" * 3
+ "UVM_ERROR @100: drv.sv(88) bad\n"
+ "UVM_INFO @200: mid\n" * 4
+ "UVM_ERROR @300: mon.sv(12) bad\n"
+ "TEST DONE\n"
+ "UVM_INFO Summary: 2 errors\n")
def facts(text):
return {"errors": sum(1 for l in text.splitlines()
if l.startswith("UVM_ERROR")),
"done": "TEST DONE" in text,
"summary": "Summary:" in text}
print(f"{'kept':<12} {'lines':>6} facts still pinned")
for n in (None, 10, 6, 4):
text = FULL if n is None else "\n".join(FULL.splitlines()[:n]) + "\n"
label = "all" if n is None else f"first {n}"
print(f"{label:<12} {len(text.splitlines()):>6} {facts(text)}")
print()
print("the full log pins :", 3, "facts")
print("the first 10 lines pin :", 2, "— the summary is gone")
print("the first 6 lines pin :", 1, "— and the error count is now WRONG")
print("so a head -6 fixture :", "silently changes the expected value")kept lines facts still pinned
all 11 {'errors': 2, 'done': True, 'summary': True}
first 10 10 {'errors': 2, 'done': True, 'summary': False}
first 6 6 {'errors': 1, 'done': False, 'summary': False}
first 4 4 {'errors': 1, 'done': False, 'summary': False}
the full log pins : 3 facts
the first 10 lines pin : 2 — the summary is gone
the first 6 lines pin : 1 — and the error count is now WRONG
so a head -6 fixture : silently changes the expected valueCutting at six lines changes the error count from 2 to 1. Not "loses a
test" — changes the right answer. A fixture built with head -6 and an
expected value copied from the full log's summary produces a test that fails
for a reason that has nothing to do with the parser, and an engineer who then
"fixes" the expected value to 1 has pinned the truncation rather than the
behaviour.
Cutting at ten is better and still drops the Summary: line, so any assertion
about summary reconciliation — Chapter 29.1's cross-check between counted and
reported totals — is now untestable with this fixture, silently.
The method that avoids both is to cut by structure, not by line count:
FULL = ("UVM_INFO @0: top start\n" * 3
+ "UVM_ERROR @100: drv.sv(88) bad\n"
+ "UVM_INFO @200: mid\n" * 4
+ "UVM_ERROR @300: mon.sv(12) bad\n"
+ "TEST DONE\n"
+ "UVM_INFO Summary: 2 errors\n")
KEEP_PREFIX = ("UVM_ERROR", "UVM_FATAL", "TEST DONE")
KEEP_CONTAINS = ("Summary:",)
def trim(text, context=0):
"""Keep every structurally significant line; drop filler."""
lines = text.splitlines()
keep = set()
for i, l in enumerate(lines):
if l.startswith(KEEP_PREFIX) or any(c in l for c in KEEP_CONTAINS):
for j in range(max(0, i - context), min(len(lines), i + 1)):
keep.add(j)
keep.add(0) # the first line: the header
return "\n".join(lines[i] for i in sorted(keep)) + "\n"
def facts(text):
return {"errors": sum(1 for l in text.splitlines()
if l.startswith("UVM_ERROR")),
"done": "TEST DONE" in text,
"summary": "Summary:" in text}
trimmed = trim(FULL)
print("original lines:", len(FULL.splitlines()))
print("trimmed lines :", len(trimmed.splitlines()))
print()
print(trimmed, end="")
print()
print("facts in the original:", facts(FULL))
print("facts in the trimmed :", facts(trimmed))
print("identical :", facts(FULL) == facts(trimmed))
print()
print("filler lines dropped :", len(FULL.splitlines()) - len(trimmed.splitlines()))
print("significant lines :", "all kept, by construction")original lines: 11
trimmed lines : 5
UVM_INFO @0: top start
UVM_ERROR @100: drv.sv(88) bad
UVM_ERROR @300: mon.sv(12) bad
TEST DONE
UVM_INFO Summary: 2 errors
facts in the original: {'errors': 2, 'done': True, 'summary': True}
facts in the trimmed : {'errors': 2, 'done': True, 'summary': True}
identical : True
filler lines dropped : 6
significant lines : all kept, by constructionEleven lines to five, and all three facts survive. The trim is driven by what the parser looks for rather than by a line budget, so the fixture shrinks to exactly the lines that can change an answer — and the check that it worked is in the output: the facts before and after are identical.
That last comparison is the step to keep. A trimmed fixture is a new input, so run the parser on the full log and on the trim and assert they agree before committing the small one. It costs one assertion and it is the only thing standing between a tidy fixture and a fixture that quietly pins a different answer.
Two lines survive the trim for reasons worth noting. The first line is kept
unconditionally because a header carries the test name and seed that
Chapter 33.4's provenance comment refers to. And the Summary: line is kept
by a contains rule rather than a prefix rule, because it is a UVM_INFO
line — a reminder that structural significance is about content, not severity.
10. Predict, Run, Observe
import shutil
from pathlib import Path
shutil.rmtree("tests", ignore_errors=True)
Path("tests/logs").mkdir(parents=True, exist_ok=True)
# A
print("A:", len(list(Path("tests/logs").glob("*.log"))))
# B
count = 0
for p in Path("tests/logs").glob("*.log"):
count += 1
assert False, "this never runs"
print("B:", count, "iterations, no assertion evaluated")
# C
print("C:", len(list(Path("no/such/dir").glob("*.log"))))
# D
(Path("tests/logs") / "a.txt").write_text("x\n", encoding="utf-8")
print("D:", len(list(Path("tests/logs").glob("*.log"))),
len(list(Path("tests/logs").glob("*.txt"))))The verified output:
A: 0
B: 0 iterations, no assertion evaluated
C: 0
D: 0 1A — an empty directory matches nothing.
B — the loop body with assert False in it never raises, which is the whole defect in four lines.
C — a missing directory also yields zero rather than an error, so a typo in the path is indistinguishable from an empty directory.
D — the right files with the wrong extension match nothing either.
11. Exercises
Exercise 1 — three and three. For your own parser, list which failures an invented fixture catches and which need a real extract.
Exercise 2 — outside your repository. Explain why the three it misses are the ones nothing else protects.
Exercise 3 — your glob. Find every glob, iterdir or comprehension inside your test suite.
Exercise 4 — empty it. Move your fixture directory aside and run the suite. Report the result.
Exercise 5 — four scenarios. Reproduce Section 3's table for your own suite.
Exercise 6 — wrong extension. Rename one fixture's suffix and confirm the suite still passes.
Exercise 7 — the misleading directory. Explain why this is worse than Chapter 30.3's empty flaky set.
Exercise 8 — add the floor. Put a minimum-count assertion in front of your fixture loop.
Exercise 9 — non-empty inside. Add the assertion inside the iterating test too, and say why both are needed.
Exercise 10 — every loop. Audit your suite for loops without a non-empty assertion.
Exercise 11 — did-not-crash. Write a deliberately wrong parser and confirm it passes your fixture check.
Exercise 12 — pair them. Give one fixture a .json expectation and assert against it.
Exercise 13 — orphan fixtures. Add the check that every .log has a .json.
Exercise 14 — rename a message. Edit a fixture to simulate a tool renaming a prefix, and confirm a test fails.
Exercise 15 — what the regression would say. For that same change, say what your flow would have reported.
Exercise 16 — one fixture per version. List the tool versions your flow runs against and say which have fixtures.
Exercise 17 — before the regression. Confirm your suite runs before jobs are launched, and say what that buys.
Exercise 18 — trim one. Take a 400-line log down to ten and list what you removed.
Exercise 19 — the privacy pass. Rename every identifier in one fixture and confirm the test still passes.
Exercise 20 — what the shape is. Say which parts of a fixture carry its value and which are incidental.
Exercise 21 — state across lines. Construct a fixture that a one-line sample could not replace.
Exercise 22 — six fields. Add provenance to one fixture and say which question each field answers.
Exercise 23 — the why field. Write it for three of your fixtures without looking at the parser.
Exercise 24 — retrofitting. Estimate the cost of adding provenance to all your existing fixtures.
Exercise 25 — cut at six. Take a real log, keep the first six lines, and compare the error count to the full log's.
Exercise 26 — the silent fix. Describe what happens when an engineer adjusts the expected value to match the truncation.
Exercise 27 — the lost assertion. Name a fact your current fixture cannot pin because of where it was cut.
Exercise 28 — trim by structure. Write a trim function driven by the prefixes your parser looks for.
Exercise 29 — assert they agree. Add the full-versus-trimmed comparison before committing a shrunk fixture.
Exercise 30 — the kept header. Say why the first line survives unconditionally.
Exercise 31 — severity is not significance. Find a line in your logs that matters and is only UVM_INFO.
12. Summary
- An invented fixture catches logic errors and misses everything that originates outside your repository — a renamed message, a new field, another tool's format.
- An empty glob makes the test pass with zero assertions. An empty directory, wrong extensions and a missing directory all yield zero iterations and a green test.
- That is worse than Chapter 30.3's vacuous predicate, because a directory named
tests/logslooks like evidence that fixtures ran. - Every loop in a test suite needs a non-empty assertion — globs, comprehensions,
iterdir, queries — and that assertion is what makes the others meaningful. - A fixture with no expected value can only assert that the parser did not crash, which a parser that always reports zero errors also manages.
- Pair every
.logwith a.json, and assert the pairing: an orphan log is a fixture nobody wrote an expectation for. - The same parser reported FAIL on a 2024.09 log and PASS on a 2025.03 one, differing only in
UVM_ERRORbecomingUVM_ERR. - Keeping a fixture per tool version turns that into one failed test instead of a night of wrong verdicts — which is why the suite runs before the regression launches.
- Ten lines is the right size for three reasons: a failing fixture is read, a fixture change is reviewed, and a captured log carries your design's names.
- Trimming is a privacy measure as much as a readability one. A fixture's value is its shape, not its identifiers.
- Put provenance on every fixture — tool, version, where it came from, whether it was edited, and why it exists — because a reader of a failing test must know whether the parser or the fixture is stale.
- The
whyfield ages best and is the one that stops somebody "fixing" a fixture by editing away the thing it was protecting. - Three guards against the empty directory: a minimum count, a non-empty assertion per iterating test, and the pairing check.
- Where you cut a log decides which facts the fixture pins — and cutting at six lines changed the error count from 2 to 1, so the right answer changed rather than a test being lost.
- An engineer who then “fixes” the expected value has pinned the truncation, not the behaviour.
- Trim by structure, not by line count: keep every line the parser looks for, drop the filler. Eleven lines became five with all three facts intact.
- Assert that the full log and the trim agree before committing the small fixture — one assertion, and it is the only guard against a tidy fixture pinning a different answer.
- Structural significance is about content, not severity. A
Summary:line isUVM_INFOand must survive the trim.
The next chapter feeds the parser the inputs a bad night produces: empty, truncated, and the other tool's format.
Where this fits
Part of the Python curriculum.