Python Part II · Module 33 · Testing and Debugging Automation
Known Input, Expected Output: Testing a Parser
The technique this module is built on fits in three lines and the canonical annotation is those three lines. It is also, as written, a test that tells you less than it could when it fails — and since a test's entire value is realised at the moment it fails, that is worth more attention than it usually gets. This chapter measures what three different spellings of the same assertion report when the parser breaks, and then looks at how many facts a single five-line fixture is actually capable of pinning down.
Intermediate20 min readPythonPart IIAutomationTestingParsers
Part II · Module 33 · Chapter 33.3 · Known Input, Expected Output: Testing a Parser
1. Three Lines
STEPS = [
("a fixed input", "SAMPLE", "a string literal, checked into the repo"),
("the result you expect", "errors == 2, status == FAIL", "written by a human"),
("an assertion", "assert", "fails loudly when the behaviour changes"),
]
print(f"{'step':<24} {'in the annotation':<30} what it is")
for step, where, what in STEPS:
print(f"{step:<24} {where:<30} {what}")
print()
print("lines of test code:", 3)
print("the second step is the only one a machine cannot supply:", True)step in the annotation what it is
a fixed input SAMPLE a string literal, checked into the repo
the result you expect errors == 2, status == FAIL written by a human
an assertion assert fails loudly when the behaviour changes
lines of test code: 3
the second step is the only one a machine cannot supply: TrueThe middle step is the whole value. A fixed input is cheap and an assertion is syntax; the expected result is a human claim about correct behaviour, and that claim is what the test preserves after everybody who understood the parser has moved on.
2. The Annotation
def test_counts():
r = parse_sim_log_text(SAMPLE)
assert r["errors"] == 2 and r["status"] == "FAIL"def parse_sim_log_text(text):
lines = text.splitlines()
errors = sum(1 for l in lines if l.startswith("UVM_ERROR"))
fatals = sum(1 for l in lines if l.startswith("UVM_FATAL"))
finished = "TEST DONE" in text
first = next((l for l in lines if l.startswith(("UVM_ERROR", "UVM_FATAL"))), None)
if fatals or errors:
status = "FAIL"
elif not finished:
status = "NO_FINISH"
else:
status = "PASS"
return {"errors": errors, "fatals": fatals, "finished": finished,
"first_failure": first, "status": status}
SAMPLE = ("UVM_INFO tb.sv(42) @ 0: reset released\n"
"UVM_ERROR drv.sv(88) @ 141900: bad handshake\n"
"UVM_ERROR drv.sv(91) @ 158200: bad handshake\n"
"UVM_INFO tb.sv(90) @ 158300: finishing\n"
"TEST DONE\n")
def test_counts():
r = parse_sim_log_text(SAMPLE)
assert r["errors"] == 2 and r["status"] == "FAIL"
test_counts()
print("test_counts() passed")
print()
print("the parsed record:")
for k, v in parse_sim_log_text(SAMPLE).items():
print(f" {k:<14} {v!r}")test_counts() passed
the parsed record:
errors 2
fatals 0
finished True
first_failure 'UVM_ERROR drv.sv(88) @ 141900: bad handshake'
status 'FAIL'It passes, and it is a genuine test — a real parser over a real log shape with a human-written expectation. The next section breaks the parser two different ways and watches what the test says.
3. assert A and B Reports Neither Side
def make_parser(error_prefix="UVM_ERROR", fail_on_errors=True):
"""A parser with two independently breakable parts."""
def parse(text):
lines = text.splitlines()
errors = sum(1 for l in lines if l.startswith(error_prefix))
fatals = sum(1 for l in lines if l.startswith("UVM_FATAL"))
finished = "TEST DONE" in text
if fatals or (errors and fail_on_errors):
status = "FAIL"
elif not finished:
status = "NO_FINISH"
else:
status = "PASS"
return {"errors": errors, "status": status}
return parse
SAMPLE = ("UVM_INFO started\n"
"UVM_ERROR drv.sv(88): bad handshake\n"
"UVM_ERROR drv.sv(91): bad handshake\n"
"TEST DONE\n")
BREAKAGES = {
"correct": make_parser(),
"counts with the wrong prefix": make_parser(error_prefix="UVM_"),
"errors no longer mean FAIL": make_parser(fail_on_errors=False),
}
for label, parse in BREAKAGES.items():
r = parse(SAMPLE)
try:
assert r["errors"] == 2 and r["status"] == "FAIL"
print(f"{label:<30} PASS")
except AssertionError as exc:
print(f"{label:<30} FAIL message={str(exc)!r} "
f"(actual errors={r['errors']}, status={r['status']!r})")correct PASS
counts with the wrong prefix FAIL message='' (actual errors=3, status='FAIL')
errors no longer mean FAIL FAIL message='' (actual errors=2, status='PASS')Two completely different defects and an identical, empty failure message.
One breakage counted three errors because the prefix matched UVM_INFO too; the other stopped treating errors as failures. A reader who sees AssertionError with no message has to go and run the parser by hand to find out which — and the information was available at the moment the test failed.
The and is the cause. Python evaluates it to a single boolean before assert ever sees it, so there is nothing left to report:
r = {"errors": 3, "status": "FAIL"}
combined = r["errors"] == 2 and r["status"] == "FAIL"
print("what assert receives:", combined)
print("its type :", type(combined).__name__)
print("information about which side failed:", "none — it was discarded")
print()
print("the two sides separately:")
print(" errors == 2 :", r["errors"] == 2)
print(" status == FAIL:", r["status"] == "FAIL")what assert receives: False
its type : bool
information about which side failed: none — it was discarded
the two sides separately:
errors == 2 : False
status == FAIL: Trueassert is handed False. It cannot report what it was not given.
4. Three Spellings, Measured
import io
import unittest
def make_parser(error_prefix="UVM_ERROR", fail_on_errors=True):
def parse(text):
lines = text.splitlines()
errors = sum(1 for l in lines if l.startswith(error_prefix))
fatals = sum(1 for l in lines if l.startswith("UVM_FATAL"))
finished = "TEST DONE" in text
if fatals or (errors and fail_on_errors):
status = "FAIL"
elif not finished:
status = "NO_FINISH"
else:
status = "PASS"
return {"errors": errors, "status": status}
return parse
SAMPLE = ("UVM_INFO started\nUVM_ERROR a\nUVM_ERROR b\nTEST DONE\n")
BROKEN = make_parser(error_prefix="UVM_")
def form_and(parse):
r = parse(SAMPLE)
assert r["errors"] == 2 and r["status"] == "FAIL"
def form_two_asserts(parse):
r = parse(SAMPLE)
assert r["errors"] == 2, f"errors: expected 2, got {r['errors']}"
assert r["status"] == "FAIL", f"status: expected FAIL, got {r['status']}"
class WithAssertEqual(unittest.TestCase):
parse = staticmethod(BROKEN)
def test_errors(self):
self.assertEqual(self.parse(SAMPLE)["errors"], 2)
def test_status(self):
self.assertEqual(self.parse(SAMPLE)["status"], "FAIL")
print("--- form 1: one assert with `and`")
try:
form_and(BROKEN)
except AssertionError as exc:
print(" AssertionError:", repr(str(exc)))
print("--- form 2: two asserts with messages")
try:
form_two_asserts(BROKEN)
except AssertionError as exc:
print(" AssertionError:", str(exc))
print("--- form 3: assertEqual")
suite = unittest.TestLoader().loadTestsFromTestCase(WithAssertEqual)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
print(" tests run:", res.testsRun, " failures:", len(res.failures))
for test, trace in res.failures:
print(" ", test.id().split(".")[-1], "->",
[l for l in trace.strip().splitlines() if "AssertionError" in l][0].strip())--- form 1: one assert with `and`
AssertionError: ''
--- form 2: two asserts with messages
AssertionError: errors: expected 2, got 3
--- form 3: assertEqual
tests run: 2 failures: 1
test_errors -> AssertionError: 3 != 2Three spellings of the same claim, and three different amounts of information on failure.
Form 1 says a test failed. Form 2 names the field and both values, because the message was written by hand. Form 3 names the field by the test's name, gives both values automatically, and — the part the first two cannot do — keeps running: test_status still executed and still passed, so the failure is localised to one of the two facts rather than stopping the method.
That last property is the argument for one fact per test rather than one test per function:
FORMS = [
("assert A and B", 1, "nothing", "no"),
("assert A, msg; assert B, msg", 1, "the field and values", "no"),
("one assertEqual per test method", 2, "the field and values", "yes"),
]
print(f"{'form':<34} {'tests':>6} {'reports':<24} other facts still checked")
for form, n, reports, keeps in FORMS:
print(f"{form:<34} {n:>6} {reports:<24} {keeps}")
print()
print("the cost of form 3:", "one method per fact")
print("the benefit :", "a failing suite names every broken fact, not the first")form tests reports other facts still checked
assert A and B 1 nothing no
assert A, msg; assert B, msg 1 the field and values no
one assertEqual per test method 2 the field and values yes
the cost of form 3: one method per fact
the benefit : a failing suite names every broken fact, not the first5. How Many Facts a Fixture Pins
The annotation asserts two things. The fixture supports more:
def parse_sim_log_text(text):
lines = text.splitlines()
errors = sum(1 for l in lines if l.startswith("UVM_ERROR"))
fatals = sum(1 for l in lines if l.startswith("UVM_FATAL"))
finished = "TEST DONE" in text
first = next((l for l in lines if l.startswith(("UVM_ERROR", "UVM_FATAL"))), None)
known = ("UVM_INFO", "UVM_ERROR", "UVM_FATAL", "TEST DONE")
unparsed = sum(1 for l in lines if l.strip() and not l.startswith(known))
status = "FAIL" if (errors or fatals) else ("PASS" if finished else "NO_FINISH")
return {"errors": errors, "fatals": fatals, "finished": finished,
"first_failure": first, "unparsed": unparsed, "status": status}
SAMPLE = ("UVM_INFO tb.sv(42) @ 0: reset released\n"
"UVM_ERROR drv.sv(88) @ 141900: bad handshake\n"
"UVM_ERROR drv.sv(91) @ 158200: bad handshake\n"
"Simulation complete via $finish\n"
"TEST DONE\n")
r = parse_sim_log_text(SAMPLE)
FACTS = [
("errors", 2, "the count"),
("fatals", 0, "that fatals and errors are counted separately"),
("finished", True, "that the end banner was seen"),
("status", "FAIL", "the classification derived from the above"),
("first_failure", SAMPLE.splitlines()[1], "which line is reported first"),
("unparsed", 1, "that one line matched no known prefix"),
]
print(f"{'field':<15} {'expected':<46} pins")
for field, expected, why in FACTS:
ok = r[field] == expected
shown = repr(expected)
print(f"{field:<15} {shown[:44]:<46} {why if ok else 'MISMATCH'}")
print()
print("facts this one fixture can pin :", len(FACTS))
print("facts the annotation asserts :", 2)
print("lines in the fixture :", len(SAMPLE.strip().splitlines()))field expected pins
errors 2 the count
fatals 0 that fatals and errors are counted separately
finished True that the end banner was seen
status 'FAIL' the classification derived from the above
first_failure 'UVM_ERROR drv.sv(88) @ 141900: bad handshak which line is reported first
unparsed 1 that one line matched no known prefix
facts this one fixture can pin : 6
facts the annotation asserts : 2
lines in the fixture : 5Six facts from a five-line fixture, and the annotation uses two. The other four are free — the fixture is already written, the parser already computes them, and each assertion is one line.
Two of them are worth singling out. fatals == 0 pins that errors and fatals are counted separately, which is the distinction Chapter 20.7 established and which a single combined counter would silently lose. And unparsed == 1 is Chapter 20.6's counter, which is the only assertion here that would catch a tool upgrade introducing a new message prefix.
6. One Table, Many Cases
Writing a method per case gets tedious before it gets thorough, and the stdlib has an answer:
import io
import unittest
def classify(text):
if not text.strip():
return "EMPTY_LOG"
if "UVM_FATAL" in text:
return "FAIL"
if "TEST DONE" not in text:
return "NO_FINISH"
if "UVM_ERROR" in text:
return "FAIL"
return "PASS"
CASES = [
("clean", "UVM_INFO ok\nTEST DONE\n", "PASS"),
("one error", "UVM_ERROR x\nTEST DONE\n", "FAIL"),
("fatal, no banner", "UVM_FATAL boom\n", "FAIL"),
("truncated", "UVM_INFO started\n", "NO_FINISH"),
("empty", "", "EMPTY_LOG"),
("whitespace only", " \n\n", "EMPTY_LOG"),
]
class BareLoop(unittest.TestCase):
def test_all(self):
for name, text, expected in CASES:
self.assertEqual(classify(text), expected, name)
class WithSubTest(unittest.TestCase):
def test_all(self):
for name, text, expected in CASES:
with self.subTest(case=name):
self.assertEqual(classify(text), expected)
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return res
print("with the correct classifier:")
for cls in (BareLoop, WithSubTest):
res = run(cls)
print(f" {cls.__name__:<12} run={res.testsRun} failures={len(res.failures)}")with the correct classifier:
BareLoop run=1 failures=0
WithSubTest run=1 failures=0Both pass. Now break the classifier so that three of the six cases are wrong:
import io
import unittest
def classify(text):
"""Broken: the empty and truncated cases are gone."""
if "UVM_FATAL" in text:
return "FAIL"
if "UVM_ERROR" in text:
return "FAIL"
return "PASS"
CASES = [
("clean", "UVM_INFO ok\nTEST DONE\n", "PASS"),
("one error", "UVM_ERROR x\nTEST DONE\n", "FAIL"),
("fatal, no banner", "UVM_FATAL boom\n", "FAIL"),
("truncated", "UVM_INFO started\n", "NO_FINISH"),
("empty", "", "EMPTY_LOG"),
("whitespace only", " \n\n", "EMPTY_LOG"),
]
class BareLoop(unittest.TestCase):
def test_all(self):
for name, text, expected in CASES:
self.assertEqual(classify(text), expected, name)
class WithSubTest(unittest.TestCase):
def test_all(self):
for name, text, expected in CASES:
with self.subTest(case=name):
self.assertEqual(classify(text), expected)
def failures_of(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
names = []
for test, trace in res.failures:
tag = [l for l in trace.splitlines() if "AssertionError" in l]
names.append(tag[0].strip() if tag else "?")
return len(res.failures), names
n1, m1 = failures_of(BareLoop)
n2, m2 = failures_of(WithSubTest)
print("cases that should fail:", 3)
print()
print(f"BareLoop reported {n1} failure(s)")
for m in m1:
print(" ", m[:58])
print(f"WithSubTest reported {n2} failure(s)")
for m in m2:
print(" ", m[:58])cases that should fail: 3
BareLoop reported 1 failure(s)
AssertionError: 'PASS' != 'NO_FINISH'
WithSubTest reported 3 failure(s)
AssertionError: 'PASS' != 'NO_FINISH'
AssertionError: 'PASS' != 'EMPTY_LOG'
AssertionError: 'PASS' != 'EMPTY_LOG'One reported against three. The bare loop stops at the first mismatch, so a reader fixes truncated, re-runs, discovers empty, fixes that, re-runs, discovers whitespace only — three cycles for information that was available in the first run.
subTest makes each row an independently reported case while still being one method, which is the same property Section 4 wanted from one-assertion-per-method, without a method per row.
7. A Test's Name Is Its First Diagnostic
NAMES = [
("test_counts", "names what it calls", "poor"),
("test_parse", "names the function", "poor"),
("test_two_errors_gives_fail", "names the behaviour", "good"),
("test_truncated_log_is_no_finish", "names the behaviour", "good"),
("test_unparsed_line_is_counted", "names the behaviour", "good"),
]
print(f"{'name':<36} {'what it tells a reader':<26} verdict")
for name, tells, verdict in NAMES:
print(f"{name:<36} {tells:<26} {verdict}")
print()
print("when a suite fails, the reader sees:", "the name, before anything else")
print("so the name should state:", "the expectation, not the call")name what it tells a reader verdict
test_counts names what it calls poor
test_parse names the function poor
test_two_errors_gives_fail names the behaviour good
test_truncated_log_is_no_finish names the behaviour good
test_unparsed_line_is_counted names the behaviour good
when a suite fails, the reader sees: the name, before anything else
so the name should state: the expectation, not the callThe annotation's test_counts is the common case and it is the weaker choice: it says which function is involved and nothing about what should be true. test_two_errors_gives_fail fails with a name that is already half the diagnosis — and combined with assertEqual's 3 != 2, the reader knows what broke before opening the file.
8. The Suite, Assembled
import io
import unittest
def parse_sim_log_text(text):
lines = text.splitlines()
errors = sum(1 for l in lines if l.startswith("UVM_ERROR"))
fatals = sum(1 for l in lines if l.startswith("UVM_FATAL"))
finished = "TEST DONE" in text
first = next((l for l in lines
if l.startswith(("UVM_ERROR", "UVM_FATAL"))), None)
known = ("UVM_INFO", "UVM_ERROR", "UVM_FATAL", "TEST DONE")
unparsed = sum(1 for l in lines if l.strip() and not l.startswith(known))
if not text.strip():
status = "EMPTY_LOG"
elif errors or fatals:
status = "FAIL"
elif not finished:
status = "NO_FINISH"
else:
status = "PASS"
return {"errors": errors, "fatals": fatals, "finished": finished,
"first_failure": first, "unparsed": unparsed, "status": status}
TWO_ERRORS = ("UVM_INFO tb.sv(42) @ 0: reset released\n"
"UVM_ERROR drv.sv(88) @ 141900: bad handshake\n"
"UVM_ERROR drv.sv(91) @ 158200: bad handshake\n"
"Simulation complete via $finish\n"
"TEST DONE\n")
class TwoErrorLog(unittest.TestCase):
"""Every fact the TWO_ERRORS fixture pins down."""
def setUp(self):
self.r = parse_sim_log_text(TWO_ERRORS)
def test_two_errors_are_counted(self):
self.assertEqual(self.r["errors"], 2)
def test_fatals_counted_separately(self):
self.assertEqual(self.r["fatals"], 0)
def test_end_banner_seen(self):
self.assertTrue(self.r["finished"])
def test_two_errors_gives_fail(self):
self.assertEqual(self.r["status"], "FAIL")
def test_first_failure_is_the_earliest(self):
self.assertEqual(self.r["first_failure"],
"UVM_ERROR drv.sv(88) @ 141900: bad handshake")
def test_unknown_line_is_counted_unparsed(self):
self.assertEqual(self.r["unparsed"], 1)
STATUS_CASES = [
("clean", "UVM_INFO ok\nTEST DONE\n", "PASS"),
("one error", "UVM_ERROR x\nTEST DONE\n", "FAIL"),
("fatal without a banner", "UVM_FATAL boom\n", "FAIL"),
("truncated", "UVM_INFO started\n", "NO_FINISH"),
("empty", "", "EMPTY_LOG"),
("whitespace only", " \n\n", "EMPTY_LOG"),
]
class StatusTable(unittest.TestCase):
def test_every_status(self):
for name, text, expected in STATUS_CASES:
with self.subTest(case=name):
self.assertEqual(parse_sim_log_text(text)["status"], expected)
def run(*cases):
loader = unittest.TestLoader()
suite = unittest.TestSuite(loader.loadTestsFromTestCase(c) for c in cases)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"errors": len(res.errors), "ok": res.wasSuccessful()}
print("suite:", run(TwoErrorLog, StatusTable))
print()
print("test methods :", 7)
print("facts asserted :", 6 + len(STATUS_CASES))
print("fixtures :", 1 + len(STATUS_CASES))
print("lines of fixture :", len(TWO_ERRORS.strip().splitlines()))
print("frameworks installed:", "the standard library only")suite: {'run': 7, 'failures': 0, 'errors': 0, 'ok': True}
test methods : 7
facts asserted : 12
fixtures : 7
lines of fixture : 5
frameworks installed: the standard library onlySeven methods asserting twelve facts, with every status outcome covered and every field of the record pinned. setUp parses the fixture once per test, which is the stdlib's way of keeping the three-line technique visible: fixed input, expected result, assertion.
Note what this suite does not yet do. It uses invented log text rather than real extracts, it never feeds the parser something malformed, and nothing has checked that the suite is capable of failing. Those are Chapters 33.4, 33.5 and 33.9, in that order.
9. Asserting Too Much
Section 5 counted how many facts a fixture pins. The opposite failure is pinning facts nobody asked for, and it has a measurable cost the first time the parser's return value grows a field:
import io
import unittest
def classify_v1(text):
errors = sum(1 for l in text.splitlines() if l.startswith("UVM_ERROR"))
return {"errors": errors, "status": "FAIL" if errors else "PASS"}
def classify_v2(text):
"""Identical answers, plus one field a later chapter wanted."""
d = classify_v1(text)
d["parser_version"] = 2
return d
LOG = "UVM_ERROR a.sv(1) x\n"
def suites(fn):
class WholeDict(unittest.TestCase):
def test_failing_log(self):
self.assertEqual(fn(LOG), {"errors": 1, "status": "FAIL"})
def test_clean_log(self):
self.assertEqual(fn(""), {"errors": 0, "status": "PASS"})
class PerField(unittest.TestCase):
def test_error_count(self):
self.assertEqual(fn(LOG)["errors"], 1)
def test_status(self):
self.assertEqual(fn(LOG)["status"], "FAIL")
def test_clean_status(self):
self.assertEqual(fn("")["status"], "PASS")
return WholeDict, PerField
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return f"{res.testsRun} run, {len(res.failures)} failed"
for label, fn in (("v1", classify_v1), ("v2 — one field added", classify_v2)):
whole, field = suites(fn)
print(f" {label:<22} whole-dict: {run(whole):<16} per-field: {run(field)}")
print()
print("behaviour that changed between v1 and v2:", "none")
print("whole-dict tests that broke :", 2)
print("per-field tests that broke :", 0) v1 whole-dict: 2 run, 0 failed per-field: 3 run, 0 failed
v2 — one field added whole-dict: 2 run, 2 failed per-field: 3 run, 0 failed
behaviour that changed between v1 and v2: none
whole-dict tests that broke : 2
per-field tests that broke : 0Every whole-dict test broke and no behaviour changed. classify_v2 returns
the same error count and the same status for every input; it adds a field. The
whole-dict assertions fail because assertEqual on dictionaries is exact, so
they were never asserting "the count is 1 and the status is FAIL" — they were
asserting "the return value has exactly these two keys and nothing else."
That is a promise about the function's shape, made accidentally, by a test whose author was thinking about its values. And the cost is paid later, by whoever adds the field: a one-line change that breaks a dozen tests teaches a team that tests are an obstacle.
The nuance is that over-specification is sometimes exactly right:
CASES = [
("the count for a known log", "one field", "assertEqual(r['errors'], 1)"),
("the status for a known log", "one field", "assertEqual(r['status'], 'FAIL')"),
("nothing is reported for a clean log", "whole dict",
"assertEqual(r, {...}) — absence IS the claim"),
("the exact keys the API promises", "whole dict",
"assertEqual(set(r), {...}) — one test, deliberately"),
("every field, in every test", "whole dict", "no — this is the mistake"),
]
print(f"{'what you are claiming':<38} {'assert':<12} how")
for claim, kind, how in CASES:
print(f"{claim:<38} {kind:<12} {how}")
print()
print("whole-dict assertions that are right:", 2)
print("the test for which to use :",
"is the ABSENCE of other fields part of the claim?")what you are claiming assert how
the count for a known log one field assertEqual(r['errors'], 1)
the status for a known log one field assertEqual(r['status'], 'FAIL')
nothing is reported for a clean log whole dict assertEqual(r, {...}) — absence IS the claim
the exact keys the API promises whole dict assertEqual(set(r), {...}) — one test, deliberately
every field, in every test whole dict no — this is the mistake
whole-dict assertions that are right: 2
the test for which to use : is the ABSENCE of other fields part of the claim?Two of five rows justify a whole-dict assertion, and both have the same character: the absence of something is part of what is being claimed. A clean log must report no errors and no fatals and no warnings, so the complete dictionary is the claim — which is why Chapter 33.5's negative tests use that form deliberately.
Row four is the useful pattern that most suites lack. One test asserting
set(result) == {"errors", "fatals", "warnings", "status"} pins the API's
shape in exactly one place, so adding a field breaks one test whose name says
it is about the shape — and whoever adds the field updates it, reads the
promise, and moves on. The alternative is the same promise scattered across
twelve value tests that break for a reason their names do not mention.
10. Predict, Run, Observe
r = {"errors": 3, "status": "FAIL"}
# A
try:
assert r["errors"] == 2 and r["status"] == "FAIL"
except AssertionError as exc:
print("A:", repr(str(exc)))
# B
try:
assert r["errors"] == 2, f"expected 2, got {r['errors']}"
except AssertionError as exc:
print("B:", str(exc))
# C
print("C:", r["errors"] == 2 and r["status"] == "FAIL")
# D
print("D:", bool("") , bool(" ".strip()))The verified output:
A: ''
B: expected 2, got 3
C: False
D: False FalseA — the annotation's form, on a broken parser. An empty message.
B — one extra argument, and the failure names both values.
C — what assert actually received: a bare False.
D — why if not text.strip() rather than if not text is the right empty check: a whitespace-only log is empty in every sense that matters, and Chapter 33.5 is about inputs like it.
11. Exercises
Exercise 1 — three lines. Write the fixed input, expected result and assertion for one of your own parsers.
Exercise 2 — the human claim. Say which of the three steps a machine could generate and which it could not.
Exercise 3 — break it two ways. Make two different changes to your parser and check whether your test distinguishes them.
Exercise 4 — the empty message. Run an assert A and B against a broken parser and record what it prints.
Exercise 5 — what assert receives. Explain in one sentence why the and discards the information.
Exercise 6 — add the message. Convert one bare assert to assert cond, msg and show the improved failure.
Exercise 7 — assertEqual. Convert the same check and compare the failure text.
Exercise 8 — independence. Demonstrate that one failing assertion in a method hides the next.
Exercise 9 — two jobs. Say where you would use assert msg and where one-assertion-per-method, and why.
Exercise 10 — count your facts. For one existing fixture, list every fact it could pin.
Exercise 11 — the free four. Add the assertions your fixture supports but your tests omit.
Exercise 12 — separate counters. Confirm your parser counts errors and fatals separately, with a test.
Exercise 13 — the unparsed counter. Add it and the assertion that would catch a new message prefix.
Exercise 14 — a table. Convert four of your tests into one table.
Exercise 15 — the bare loop. Break three rows and confirm your loop reports one.
Exercise 16 — subTest. Wrap the loop body and confirm all three are reported.
Exercise 17 — three cycles into one. Explain the cost of the bare loop in terms of edit-run cycles.
Exercise 18 — rename your tests. Rewrite five test names to state the expectation rather than the call.
Exercise 19 — the first diagnostic. Run a failing suite and say how much you learned from the name alone.
Exercise 20 — setUp. Use it to parse one fixture once per test, and say what it keeps visible.
Exercise 21 — every outcome. List your classifier's outcomes and confirm the table covers each.
Exercise 22 — twelve facts, seven methods. Report those two numbers for your own suite.
Exercise 23 — what is still missing. Say which of real fixtures, malformed input and a self-check your suite lacks.
Exercise 24 — empty versus whitespace. Check whether your empty-log test uses not text or not text.strip().
Exercise 25 — add a field. Add a field to one of your parser's return values and count the tests that break.
Exercise 26 — shape or values. For each broken test, say whether its name is about the shape or the values.
Exercise 27 — the exact-keys assertion. Read one of your assertEqual(r, {...}) tests aloud as a promise about shape.
Exercise 28 — the obstacle. Describe what a team learns when a one-line change breaks a dozen tests.
Exercise 29 — when absence is the claim. Find a test of yours where the complete dictionary is genuinely right.
Exercise 30 — one place for the shape. Add a single set(result) == {...} test and loosen the value tests.
Exercise 31 — the name says why. Name that test so the next person understands why it broke.
12. Summary
- The technique is three lines — a fixed input, an expected result, an assertion — and the middle one is the only part a machine cannot supply.
assert A and Breports neither side. Two completely different parser defects produced an identical, emptyAssertionError.- Python evaluates the
andto a bareFalsebeforeassertsees it, so there is nothing left to report. - Three spellings give three amounts of information: nothing, the field and values, or the field and values plus the other facts still being checked.
assert cond, msgis a good test and belongs inside the flow too — Chapter 33.11 uses it as a pre-flight check; one assertion per method belongs in the suite.- A five-line fixture pinned six facts and the annotation asserted two. The other four are one line each and already computed.
fatals == 0pins that errors and fatals are counted separately, andunparsed == 1is the only assertion that would catch a tool adding a new message prefix.- A bare loop over a table reported one failure where three existed, costing three edit-run cycles for information available in the first.
subTestmakes each row independently reported while staying one method.- A test's name is its first diagnostic.
test_countsnames the call;test_two_errors_gives_failnames the expectation. - Seven methods asserted twelve facts over one real record and every status outcome, using only the standard library.
- Use
not text.strip(), notnot text— a whitespace-only log is empty in every sense that matters. assertEqualon a dictionary is exact, so a whole-dict assertion promises the return value's shape, not only its values.- Adding a field broke every whole-dict test and no behaviour changed — the cost is paid by whoever extends the parser.
- The test is whether the absence of other fields is part of the claim. For a clean log it is; for a count it is not.
- Pin the API's shape in exactly one test with
set(result) == {...}, so a new field breaks one test whose name explains why.
The next chapter replaces the invented fixture with real extracts, and finds the way a fixture directory can make a suite pass while testing nothing.
Where this fits
Part of the Python curriculum.