Python Part II · Module 33 · Testing and Debugging Automation
Test the Test: Break the Parser on Purpose
Every chapter so far has written tests and watched them pass. A passing suite is the least informative thing a test run can produce: it is equally consistent with a suite that checks everything and a suite that checks almost nothing, and nothing in the output distinguishes them. This chapter gets a number instead — break the parser on purpose, count how many breakages the suite notices, and read what the survivors say. Two of ours survived a six-test suite, and the two had completely different causes.
Advanced24 min readPythonPart IIAutomationTestingMutation Testing
Part II · Module 33 · Chapter 33.9 · Test the Test: Break the Parser on Purpose
1. A Green Suite Is Not Evidence
OBSERVATIONS = [
("the suite passes", "the code is right", "or the tests check nothing"),
("the suite passes", "the tests are good", "or they assert trivia"),
("8 tests pass", "8 behaviours are pinned", "or 8 tests pin the same one"),
("coverage is 100%", "every line ran", "lines ran — nothing was checked"),
]
print(f"{'what you see':<20} {'what it suggests':<26} what it is also consistent with")
for seen, suggests, also in OBSERVATIONS:
print(f"{seen:<20} {suggests:<26} {also}")
print()
print("distinguishable from the output of a run?", False)
print("what would distinguish them:", "a run where the code is WRONG")what you see what it suggests what it is also consistent with
the suite passes the code is right or the tests check nothing
the suite passes the tests are good or they assert trivia
8 tests pass 8 behaviours are pinned or 8 tests pin the same one
coverage is 100% every line ran lines ran — nothing was checkedThe last row is Chapter 29.1 Section 5's point arriving in a test suite: coverage records that a line executed, and a line can execute inside a test that asserts nothing about it.
So a green run carries no information about the suite. The only experiment that does is the one where the code is broken and you watch what happens.
2. The Annotation
# remove the UVM_FATAL branch, then run the suite
# expected: at least one test fails — if none does, fix the testsHere is the parser the whole chapter measures, and the smallest suite someone might plausibly write for it:
import io
import unittest
PARSER_SRC = '''
import re
PAT = re.compile(r"^(UVM_ERROR|UVM_FATAL|UVM_WARNING)\\s+(\\S+)\\((\\d+)\\)")
def classify(text):
errors = fatals = warnings = 0
for line in text.splitlines():
m = PAT.match(line)
if not m:
continue
sev = m.group(1)
if sev == "UVM_ERROR":
errors += 1
elif sev == "UVM_FATAL":
fatals += 1
elif sev == "UVM_WARNING":
warnings += 1
if fatals:
status = "FATAL"
elif errors:
status = "FAIL"
else:
status = "PASS"
return {"errors": errors, "fatals": fatals,
"warnings": warnings, "status": status}
'''
CLEAN = "UVM_INFO top.sv(10) start\nTEST DONE\n"
ONE_ERR = "UVM_INFO a\nUVM_ERROR drv.sv(88) bad\nTEST DONE\n"
def load(src):
"""Exec a parser source string and hand back its namespace."""
ns = {}
exec(compile(src, "parser.py", "exec"), ns)
return ns
def weak_suite(mod):
class WeakSuite(unittest.TestCase):
def test_clean_log_passes(self):
self.assertEqual(mod["classify"](CLEAN)["status"], "PASS")
def test_one_error_fails(self):
r = mod["classify"](ONE_ERR)
self.assertEqual(r["status"], "FAIL")
self.assertEqual(r["errors"], 1)
return WeakSuite
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"errors": len(res.errors), "ok": res.wasSuccessful()}
print("baseline:", run(weak_suite(load(PARSER_SRC))))
# the annotation's mutation, applied literally
BRANCH = ' elif sev == "UVM_FATAL":\n fatals += 1\n'
assert BRANCH in PARSER_SRC, "the mutation target must exist"
mutated = PARSER_SRC.replace(BRANCH, "", 1)
print("mutated :", run(weak_suite(load(mutated))))
print()
print("the annotation expected:", "at least one test fails")
print("what happened :", "no test failed")
print("so the annotation says :", "fix the tests")baseline: {'run': 2, 'failures': 0, 'errors': 0, 'ok': True}
mutated : {'run': 2, 'failures': 0, 'errors': 0, 'ok': True}
the annotation expected: at least one test fails
what happened : no test failed
so the annotation says : fix the testsThe UVM_FATAL branch is gone and the suite is still green. The parser now counts a fatal as nothing at all — the one severity that most reliably means the run is worthless — and two tests pass exactly as before.
That is not a contrived failure. Those two tests are what a careful engineer writes first: one clean log, one log with an error, status and count asserted on both. They are good tests. They just do not touch the branch.
3. The Loop
One mutation, one measurement
Step 6 is the one that distinguishes a useful mutation run from a tedious one, and Section 7 is about nothing else. Step 1 is the one that silently invalidates the whole run when skipped, which is Section 9.
4. Eight Mutations
MUTATIONS = [
("drop the UVM_FATAL branch", "the annotation's own"),
("fatals no longer force FATAL", "status logic"),
("off-by-one in the error count", "arithmetic"),
("anchor removed from the pattern", "regex"),
("warnings counted as errors", "severity mapping"),
("line number group made optional", "regex"),
("PASS and FAIL swapped", "status logic"),
("whitespace made optional", "regex"),
]
print(f"{'#':>2} {'mutation':<34} kind")
for n, (name, kind) in enumerate(MUTATIONS, 1):
print(f"{n:>2} {name:<34} {kind}")
print()
from collections import Counter
for kind, n in sorted(Counter(k for _, k in MUTATIONS).items()):
print(f" {kind:<18} {n}")
print()
print("chosen to be:", "the mistakes a tired engineer actually makes")
print("NOT chosen to be:", "random character edits") # mutation kind
1 drop the UVM_FATAL branch the annotation's own
2 fatals no longer force FATAL status logic
3 off-by-one in the error count arithmetic
4 anchor removed from the pattern regex
5 warnings counted as errors severity mapping
6 line number group made optional regex
7 PASS and FAIL swapped status logic
8 whitespace made optional regex
arithmetic 1
regex 3
severity mapping 1
status logic 2
the annotation's own 1
chosen to be: the mistakes a tired engineer actually makes
NOT chosen to be: random character editsThat last distinction matters more than it sounds. A tool that mutates random characters produces mostly syntax errors, which every suite "kills" trivially and which teach nothing. These eight are plausible edits — a dropped branch, an off-by-one, a loosened quantifier — so a survivor among them is a gap that a real change could walk through.
5. The Harness
import io
import unittest
PARSER_SRC = '''
import re
PAT = re.compile(r"^(UVM_ERROR|UVM_FATAL|UVM_WARNING)\\s+(\\S+)\\((\\d+)\\)")
def classify(text):
errors = fatals = warnings = 0
for line in text.splitlines():
m = PAT.match(line)
if not m:
continue
sev = m.group(1)
if sev == "UVM_ERROR":
errors += 1
elif sev == "UVM_FATAL":
fatals += 1
elif sev == "UVM_WARNING":
warnings += 1
if fatals:
status = "FATAL"
elif errors:
status = "FAIL"
else:
status = "PASS"
return {"errors": errors, "fatals": fatals,
"warnings": warnings, "status": status}
'''
MUTATIONS = [
("drop the UVM_FATAL branch",
' elif sev == "UVM_FATAL":\n fatals += 1\n', ''),
("fatals no longer force FATAL",
' if fatals:\n status = "FATAL"\n elif errors:',
' if errors:'),
("off-by-one in the error count", 'errors += 1', 'errors += 2'),
("anchor removed from the pattern", 'r"^(UVM_ERROR', 'r"(UVM_ERROR'),
("warnings counted as errors",
' elif sev == "UVM_WARNING":\n warnings += 1',
' elif sev == "UVM_WARNING":\n errors += 1'),
("line number group made optional", r'\((\d+)\)', r'(\((\d+)\))?'),
("PASS and FAIL swapped",
' status = "FAIL"\n else:\n status = "PASS"',
' status = "PASS"\n else:\n status = "FAIL"'),
("whitespace made optional", r'\s+(\S+)', r'\s*(\S+)'),
]
print(f"{'#':>2} {'mutation':<34} target found one change only")
for n, (name, old, new) in enumerate(MUTATIONS, 1):
found = old in PARSER_SRC
count = PARSER_SRC.count(old)
print(f"{n:>2} {name:<34} {str(found):<12} {count == 1}")
print()
print("every target string present:",
all(old in PARSER_SRC for _, old, _ in MUTATIONS))
print("every target unique :",
all(PARSER_SRC.count(old) == 1 for _, old, _ in MUTATIONS)) # mutation target found one change only
1 drop the UVM_FATAL branch True True
2 fatals no longer force FATAL True True
3 off-by-one in the error count True True
4 anchor removed from the pattern True True
5 warnings counted as errors True True
6 line number group made optional True True
7 PASS and FAIL swapped True True
8 whitespace made optional True True
every target string present: True
every target unique : TrueBoth columns must be True before any number from this harness means anything, and that is not ceremony. A mutation whose target string is absent changes nothing, so the suite passes, so it reports as a survivor — a mutation that was never applied is indistinguishable in the results from one the tests failed to catch. Checking that old in src is the difference between measuring a suite and measuring a typo.
The uniqueness column matters for the same reason in reverse. str.replace with no count replaces every occurrence, so a target appearing twice makes two changes at once and a kill no longer tells you which one the suite caught. Passing 1 as the count and asserting count == 1 keeps one mutation to one edit.
6. Two Suites, The Same Eight Mutations
import io
import unittest
PARSER_SRC = '''
import re
PAT = re.compile(r"^(UVM_ERROR|UVM_FATAL|UVM_WARNING)\\s+(\\S+)\\((\\d+)\\)")
def classify(text):
errors = fatals = warnings = 0
for line in text.splitlines():
m = PAT.match(line)
if not m:
continue
sev = m.group(1)
if sev == "UVM_ERROR":
errors += 1
elif sev == "UVM_FATAL":
fatals += 1
elif sev == "UVM_WARNING":
warnings += 1
if fatals:
status = "FATAL"
elif errors:
status = "FAIL"
else:
status = "PASS"
return {"errors": errors, "fatals": fatals,
"warnings": warnings, "status": status}
'''
MUTATIONS = [
("drop the UVM_FATAL branch",
' elif sev == "UVM_FATAL":\n fatals += 1\n', ''),
("fatals no longer force FATAL",
' if fatals:\n status = "FATAL"\n elif errors:',
' if errors:'),
("off-by-one in the error count", 'errors += 1', 'errors += 2'),
("anchor removed from the pattern", 'r"^(UVM_ERROR', 'r"(UVM_ERROR'),
("warnings counted as errors",
' elif sev == "UVM_WARNING":\n warnings += 1',
' elif sev == "UVM_WARNING":\n errors += 1'),
("line number group made optional", r'\((\d+)\)', r'(\((\d+)\))?'),
("PASS and FAIL swapped",
' status = "FAIL"\n else:\n status = "PASS"',
' status = "PASS"\n else:\n status = "FAIL"'),
("whitespace made optional", r'\s+(\S+)', r'\s*(\S+)'),
]
CLEAN = "UVM_INFO top.sv(10) start\nTEST DONE\n"
ONE_ERR = "UVM_INFO a\nUVM_ERROR drv.sv(88) bad\nTEST DONE\n"
def load(src):
ns = {}
exec(compile(src, "parser.py", "exec"), ns)
return ns
def build(mod, strong):
class Weak(unittest.TestCase):
def test_clean_log_passes(self):
self.assertEqual(mod["classify"](CLEAN)["status"], "PASS")
def test_one_error_fails(self):
r = mod["classify"](ONE_ERR)
self.assertEqual(r["status"], "FAIL")
self.assertEqual(r["errors"], 1)
if not strong:
return Weak
class Strong(Weak):
def test_fatal_is_its_own_status(self):
r = mod["classify"]("UVM_FATAL env.sv(3) boom\n")
self.assertEqual(r["status"], "FATAL")
self.assertEqual(r["fatals"], 1)
def test_warning_is_not_an_error(self):
r = mod["classify"]("UVM_WARNING a.sv(1) w\n")
self.assertEqual(r, {"errors": 0, "fatals": 0,
"warnings": 1, "status": "PASS"})
def test_severity_mid_line_is_not_matched(self):
r = mod["classify"]("echo UVM_ERROR drv.sv(1) x\n")
self.assertEqual(r["errors"], 0)
def test_line_number_is_required(self):
r = mod["classify"]("UVM_ERROR drv.sv bad\n")
self.assertEqual(r["errors"], 0)
return Strong
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return res.wasSuccessful(), res.testsRun
def score(strong):
base_ok, n = run(build(load(PARSER_SRC), strong))
assert base_ok, "the baseline must pass — see §9"
verdicts = []
for name, old, new in MUTATIONS:
assert PARSER_SRC.count(old) == 1, name
src = PARSER_SRC.replace(old, new, 1)
try:
ok, _ = run(build(load(src), strong))
except Exception:
ok = False # it did not even import: killed
verdicts.append((name, "KILLED" if not ok else "SURVIVED"))
return n, verdicts
nw, weak = score(False)
ns, strong = score(True)
print(f"{'mutation':<34} {'WEAK (%d tests)':<16} STRONG (%d tests)"
% (nw, ns) if False else
f"{'mutation':<34} {'WEAK':<10} STRONG")
for (name, w), (_, s) in zip(weak, strong):
print(f"{name:<34} {w:<10} {s}")
kw = sum(1 for _, v in weak if v == "KILLED")
ks = sum(1 for _, v in strong if v == "KILLED")
print()
print(f"weak suite : {nw} tests, {kw}/{len(MUTATIONS)} killed "
f"= {100 * kw // len(MUTATIONS)}%")
print(f"strong suite: {ns} tests, {ks}/{len(MUTATIONS)} killed "
f"= {100 * ks // len(MUTATIONS)}%")
print()
print("both baselines were green:", True)
print("tests added:", ns - nw, "| mutations newly caught:", ks - kw)mutation WEAK STRONG
drop the UVM_FATAL branch SURVIVED KILLED
fatals no longer force FATAL SURVIVED KILLED
off-by-one in the error count KILLED KILLED
anchor removed from the pattern SURVIVED SURVIVED
warnings counted as errors SURVIVED KILLED
line number group made optional SURVIVED KILLED
PASS and FAIL swapped KILLED KILLED
whitespace made optional SURVIVED SURVIVED
weak suite : 2 tests, 2/8 killed = 25%
strong suite: 6 tests, 6/8 killed = 75%
both baselines were green: True
tests added: 4 | mutations newly caught: 425% against 75%, and both baselines were green. Nothing in a passing test run distinguishes the two-test suite from the six-test suite; the mutation column does, immediately and numerically.
Three details in that table are worth reading carefully.
The weak suite's two kills are the two mutations that change a value it already asserts. errors += 2 breaks assertEqual(r["errors"], 1); swapping PASS and FAIL breaks both status assertions. Everything else lives in code those two logs never reach, or changes a value they never look at.
Four added tests caught exactly four more mutations — a one-to-one return, which is unusual and happens here because each added test was written against a specific unexercised behaviour rather than to raise a coverage number.
And two mutations survived both suites. Those are the interesting ones.
7. A Survivor Is a Question
The reflex is to treat every survivor as a missing test. Measured, our two survivors need two completely different responses:
import re
STRICT = re.compile(r"^(UVM_ERROR|UVM_FATAL)\s+(\S+)\((\d+)\)")
LOOSE = re.compile(r"(UVM_ERROR|UVM_FATAL)\s+(\S+)\((\d+)\)")
CASES = [
"UVM_ERROR drv.sv(88) bad",
"echo UVM_ERROR drv.sv(1) x",
" UVM_ERROR drv.sv(1) x",
"xUVM_ERROR drv.sv(1) x",
"UVM_FATAL env.sv(3) boom",
]
print("survivor 4 — `^` removed, and the code calls .match()")
print(f" {'input':<30} {'with ^':<8} {'without':<8} differ")
for c in CASES:
a, b = bool(STRICT.match(c)), bool(LOOSE.match(c))
print(f" {c!r:<30} {str(a):<8} {str(b):<8} {a != b}")
print()
print(" any input distinguishing them under .match():",
any(bool(STRICT.match(c)) != bool(LOOSE.match(c)) for c in CASES))
print(" any input distinguishing them under .search():",
any(bool(STRICT.search(c)) != bool(LOOSE.search(c)) for c in CASES))survivor 4 — `^` removed, and the code calls .match()
input with ^ without differ
'UVM_ERROR drv.sv(88) bad' True True False
'echo UVM_ERROR drv.sv(1) x' False False False
' UVM_ERROR drv.sv(1) x' False False False
'xUVM_ERROR drv.sv(1) x' False False False
'UVM_FATAL env.sv(3) boom' True True False
any input distinguishing them under .match(): False
any input distinguishing them under .search(): TrueNo test can kill that mutation, because there is no input on which the two patterns behave differently. re.match anchors at the start of the string by definition — Chapter 19.2 Section 4's distinction between match and search — so a leading ^ in a pattern used only with .match() is redundant. The mutant is semantically identical to the original: an equivalent mutant, and writing a test to catch it is impossible, not merely difficult.
The right response is a decision about the code rather than the tests. Either delete the ^, since it asserts something the call already guarantees; or keep it deliberately, knowing it becomes load-bearing the instant somebody changes .match to .search — and the last line of that output is exactly how much it would then matter. A redundant guard against a future edit is defensible; what is not defensible is believing a test could have caught it.
The other survivor is the opposite:
import re
STRICT = re.compile(r"^(UVM_ERROR|UVM_FATAL|UVM_WARNING)\s+(\S+)\((\d+)\)")
LOOSE = re.compile(r"^(UVM_ERROR|UVM_FATAL|UVM_WARNING)\s*(\S+)\((\d+)\)")
CASES = [
"UVM_ERROR drv.sv(88) bad",
"UVM_ERROR drv.sv(88) bad",
"UVM_ERRORdrv.sv(88) x",
"UVM_ERROR_drv.sv(88) x",
"UVM_ERROR_COUNT drv.sv(1) x",
"UVM_ERROR_SUMMARY(0) : done",
"UVM_FATALITY(0)",
]
print("survivor 8 — `\\s+` loosened to `\\s*`")
print(f" {'input':<32} {'strict':<8} {'loose':<8} {'differ':<7} loose field 2")
for s in CASES:
a, b = STRICT.match(s), LOOSE.match(s)
print(f" {s!r:<32} {str(bool(a)):<8} {str(bool(b)):<8} "
f"{str(bool(a) != bool(b)):<7} "
f"{repr(b.group(2)) if b else '-'}")
print()
print(" inputs that distinguish them:",
sum(1 for s in CASES if bool(STRICT.match(s)) != bool(LOOSE.match(s))))
print(" did any test feed one? :", False)
print(" so this survivor is :", "a real gap")survivor 8 — `\s+` loosened to `\s*`
input strict loose differ loose field 2
'UVM_ERROR drv.sv(88) bad' True True False 'drv.sv'
'UVM_ERROR drv.sv(88) bad' True True False 'drv.sv'
'UVM_ERRORdrv.sv(88) x' False True True 'drv.sv'
'UVM_ERROR_drv.sv(88) x' False True True '_drv.sv'
'UVM_ERROR_COUNT drv.sv(1) x' False False False -
'UVM_ERROR_SUMMARY(0) : done' False True True '_SUMMARY'
'UVM_FATALITY(0)' False True True 'ITY'
inputs that distinguish them: 4
did any test feed one? : False
so this survivor is : a real gapFour inputs distinguish them and no test fed any of them. This survivor is a genuine hole, and the right-hand column shows its consequence is worse than the mutation itself suggests.
Read the last two rows. UVM_ERROR_SUMMARY(0) : done matches the loose pattern: \s* consumes nothing, (\S+) starts at the underscore and captures _SUMMARY, and (0) satisfies the line-number group. So the loose parser reads a summary line reporting zero errors as one error — and UVM_FATALITY(0) is read as a fatal, which forces the whole run's status to FATAL. The severity field no longer means a severity; it means a prefix.
The fifth row is the boundary, and it is the part worth understanding rather than memorising. UVM_ERROR_COUNT drv.sv(1) x does not match the loose pattern, even though it begins with the same prefix. (\S+) cannot contain a space, so after capturing _COUNT the pattern needs ( immediately and finds a space instead — and there is no way to backtrack across it. The loose pattern swallows a longer identifier only when a parenthesised number follows it with no space, which is why UVM_ERROR_SUMMARY(0) matches and UVM_ERROR_COUNT drv.sv(1) does not.
That shape is the Chapter 24.3 severity-prefix problem rediscovered from the opposite direction: there, naive counting invented a UVM_FATAL by matching a summary line; here a mutation experiment on a parser's tests pointed at the same class of bug, in code that was still correct.
SURVIVORS = [
("anchor removed", "no input distinguishes them",
"equivalent mutant", "decide about the CODE"),
("whitespace loosened", "'UVM_ERROR_SUMMARY(0)' does",
"test gap", "add the negative test"),
]
print(f"{'survivor':<22} {'distinguishing input':<32} {'kind':<20} response")
for s, inp, kind, resp in SURVIVORS:
print(f"{s:<22} {inp:<32} {kind:<20} {resp}")
print()
print("the test that separates the two kinds:")
print(" can you name an input on which original and mutant differ?")
print(" yes -> a test gap, and you have just written the test")
print(" no -> an equivalent mutant, and no test will ever kill it")survivor distinguishing input kind response
anchor removed no input distinguishes them equivalent mutant decide about the CODE
whitespace loosened 'UVM_ERROR_SUMMARY(0)' does test gap add the negative test
the test that separates the two kinds:
can you name an input on which original and mutant differ?
yes -> a test gap, and you have just written the test
no -> an equivalent mutant, and no test will ever kill itThat question is the whole of the classification, and it is self-rewarding: answering "yes" requires naming the input, and naming the input is writing the missing test. Chapter 29.5 Section 6 reached the same shape for an uncovered coverage bin — the useful question is not "how do I cover it" but "is it reachable at all", and the two answers lead to opposite work.
8. Closing the Real Gap
import io
import re
import unittest
def classify(text):
pat = re.compile(r"^(UVM_ERROR|UVM_FATAL|UVM_WARNING)\s+(\S+)\((\d+)\)")
errors = fatals = warnings = 0
for line in text.splitlines():
m = pat.match(line)
if not m:
continue
sev = m.group(1)
if sev == "UVM_ERROR":
errors += 1
elif sev == "UVM_FATAL":
fatals += 1
else:
warnings += 1
status = "FATAL" if fatals else ("FAIL" if errors else "PASS")
return {"errors": errors, "fatals": fatals,
"warnings": warnings, "status": status}
class SeverityIsAWholeWord(unittest.TestCase):
"""The test that kills survivor 8 — written from §7's input."""
def test_no_space_after_severity_is_not_a_match(self):
self.assertEqual(classify("UVM_ERRORdrv.sv(88) x\n")["errors"], 0)
def test_a_longer_identifier_is_not_a_severity(self):
for line in ("UVM_ERROR_SUMMARY(0) : done\n",
"UVM_ERRORS_FOUND(3)\n",
"UVM_FATALITY(0)\n"):
with self.subTest(line=line.strip()):
r = classify(line)
self.assertEqual(r["status"], "PASS")
self.assertEqual((r["errors"], r["fatals"]), (0, 0))
def test_the_real_shape_still_matches(self):
r = classify("UVM_ERROR drv.sv(88) bad\n")
self.assertEqual((r["errors"], r["status"]), (1, "FAIL"))
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"errors": len(res.errors), "ok": res.wasSuccessful()}
print("gap-closing suite:", run(SeverityIsAWholeWord))
print()
# and prove it kills the mutant it was written for
loose = re.compile(r"^(UVM_ERROR|UVM_FATAL|UVM_WARNING)\s*(\S+)\((\d+)\)")
print("against the MUTANT pattern:")
LINES = ["UVM_ERRORdrv.sv(88) x", "UVM_ERROR_SUMMARY(0) : done",
"UVM_ERRORS_FOUND(3)", "UVM_FATALITY(0)"]
for line in LINES:
print(f" {line!r:<32} matches = {bool(loose.match(line))}")
print()
print("new-test inputs the mutant mis-matches:",
sum(1 for l in LINES if loose.match(l)))
print("so the new tests fail on the mutant :", True)
print("which is what 'killed' means :", "the suite noticed")gap-closing suite: {'run': 3, 'failures': 0, 'errors': 0, 'ok': True}
against the MUTANT pattern:
'UVM_ERRORdrv.sv(88) x' matches = True
'UVM_ERROR_SUMMARY(0) : done' matches = True
'UVM_ERRORS_FOUND(3)' matches = True
'UVM_FATALITY(0)' matches = True
new-test inputs the mutant mis-matches: 4
so the new tests fail on the mutant : True
which is what 'killed' means : the suite noticedThree tests, written directly from the distinguishing inputs Section 7 named, and the second is now the more valuable of them: a line like UVM_ERROR_SUMMARY(0) is the shape a real UVM log uses to report that there were no errors, so that test defends against a parser that would read a clean summary as a failure — a mistake nobody has made yet, in code one loosened quantifier away from making it.
That is what a mutation run buys. It did not find a bug in the parser — the parser was right. It found that the suite could not tell the right parser from a wrong one, and the test written to fix that now guards a line the original author never considered.
9. The Baseline Rule
import io
import unittest
GOOD = "def width():\n return 32\n"
MUTATIONS = [("32 -> 64", "32", "64"), ("32 -> 0", "32", "0")]
def load(src):
ns = {}
exec(compile(src, "m.py", "exec"), ns)
return ns
def suite_for(mod, expected):
class S(unittest.TestCase):
def test_width(self):
self.assertEqual(mod["width"](), expected)
return S
def run(case):
s = unittest.TestLoader().loadTestsFromTestCase(case)
return unittest.TextTestRunner(stream=io.StringIO(),
verbosity=0).run(s).wasSuccessful()
for label, expected in (("a CORRECT baseline", 32), ("a BROKEN baseline", 16)):
base = run(suite_for(load(GOOD), expected))
killed = 0
for name, old, new in MUTATIONS:
if not run(suite_for(load(GOOD.replace(old, new, 1)), expected)):
killed += 1
print(f"{label:<22} baseline ok={str(base):<6} "
f"killed {killed}/{len(MUTATIONS)} = {100 * killed // len(MUTATIONS)}%")
print()
print("the broken baseline scored:", "100%")
print("what it measured :", "nothing")
print("why :", "the suite was already failing")
print("the rule :", "assert the baseline passes, then mutate")a CORRECT baseline baseline ok=True killed 2/2 = 100%
a BROKEN baseline baseline ok=False killed 2/2 = 100%A broken baseline scores a perfect 100%. Every mutant "fails the suite" because the suite was failing before any mutation was applied, and a kill rate computed that way is a measurement of nothing — the best-looking number this chapter produces is the one that means least.
That is why step 1 of the Section 3 loop is an assert, not a note, and why the Section 6 harness carries assert base_ok inside score(). A mutation run without that line reports a number whose most likely cause is that the experiment never ran.
10. Predict, Run, Observe
import io
import re
import unittest
SRC = 'import re\nPAT = re.compile(r"^UVM_ERROR\\s+(\\S+)")\n' \
'def n(t):\n return sum(1 for l in t.splitlines() if PAT.match(l))\n'
LOG = "UVM_ERROR a.sv x\nUVM_INFO b\nUVM_ERROR c.sv y\n"
def load(src):
ns = {}
exec(compile(src, "m.py", "exec"), ns)
return ns
def suite(mod):
class S(unittest.TestCase):
def test_count(self):
self.assertEqual(mod["n"](LOG), 2)
return S
def killed(src):
s = unittest.TestLoader().loadTestsFromTestCase(suite(load(src)))
r = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(s)
return not r.wasSuccessful()
# A
print("A:", killed(SRC))
# B
print("B:", killed(SRC.replace("sum(1", "sum(2", 1)))
# C
print("C:", killed(SRC.replace('r"^UVM_ERROR', 'r"UVM_ERROR', 1)))
# D
print("D:", killed(SRC.replace("NOT_PRESENT", "x", 1)))The verified output:
A: False
B: True
C: False
D: FalseA — the unmutated source: not killed, which is the baseline being green. Section 9's assert is exactly this value.
B — counting 2 per line doubles the result, and the one assertion catches it.
C — the anchor again, surviving again for Section 7's reason: .match() anchors anyway.
D — and here is the trap. The target string does not exist, so replace changed nothing, so the suite passed, so it reports False — reading as a survivor. C and D are indistinguishable in this output, and they mean opposite things: one is an equivalent mutant worth thinking about, the other is a typo in the harness. Only Section 5's assert old in src separates them.
11. Exercises
Exercise 1 — what green means. Write down three things a passing suite is consistent with besides correct code.
Exercise 2 — coverage is not checking. Write a test that executes a function and asserts nothing about its result, and confirm coverage counts the lines.
Exercise 3 — the annotation, literally. Remove one branch from your own parser and run your suite. Record whether anything failed.
Exercise 4 — if none does. If your suite stayed green, write the test that would have caught it.
Exercise 5 — eight plausible mutations. List eight single edits a tired engineer might make to one of your parsers.
Exercise 6 — plausible, not random. Explain in two sentences why random character edits make a poor mutation set.
Exercise 7 — assert the target exists. Add the old in src check to a mutation script and say what it protects you from.
Exercise 8 — one change only. Explain why str.replace without a count can make a kill uninformative.
Exercise 9 — score a weak suite. Run your eight mutations against your current suite and compute the kill rate.
Exercise 10 — add four tests. Write tests for four unexercised behaviours and re-score.
Exercise 11 — one-to-one? Compare tests added against mutations newly caught, and say what a ratio below one would mean.
Exercise 12 — which two died first. For your weak suite, say why its kills were the ones they were.
Exercise 13 — name the input. For each of your survivors, try to name an input on which original and mutant differ.
Exercise 14 — classify. Sort your survivors into test gaps and equivalent mutants.
Exercise 15 — an equivalent mutant of your own. Find a redundant guard in your code whose removal no test could detect.
Exercise 16 — delete or keep. For that guard, decide whether to remove it or document why it stays.
Exercise 17 — match versus search. Audit your patterns for a ^ that only .match() ever sees.
Exercise 18 — the worse consequence. For one real gap, work out what the mutant would do to a real log — not just to a test.
Exercise 19 — the severity prefix. Check whether your parser counts UVM_ERROR_SUMMARY : 0 as an error.
Exercise 20 — close one gap. Write the three-test group that kills one survivor, including a positive test proving the real shape still matches.
Exercise 21 — the broken baseline. Deliberately run a mutation set against a failing suite and record the kill rate.
Exercise 22 — 100% means nothing. Explain in one sentence why that number was perfect.
Exercise 23 — assert it. Add the baseline assertion to your mutation runner.
Exercise 24 — survivor versus typo. Explain how Section 10's C and D produced the same value for different reasons.
Exercise 25 — restore every time. Say what goes wrong if a mutation is left in place before the next one is applied.
Exercise 26 — a kill rate in review. Decide what kill rate you would require before trusting a parser's suite, and justify the number.
12. Summary
- A passing suite carries no information about the suite. It is equally consistent with tests that check everything and tests that assert trivia.
- Coverage records that a line executed, which is not the same as a line being checked.
- The annotation's own mutation survived the smallest plausible suite. Removing the UVM_FATAL branch left two good tests green, so "if none does, fix the tests" is the normal outcome, not a formality.
- Two suites, the same eight mutations: 25% against 75% — and both baselines were green, so only the kill rate distinguished them.
- The weak suite killed exactly the mutations that changed a value it already asserted, and nothing else.
- Four added tests caught four more mutations, a one-to-one return that comes from writing each test against an unexercised behaviour rather than for a coverage number.
- Mutations must be plausible edits, not random characters — random edits produce syntax errors every suite kills for free.
- Assert that the target string exists and is unique. An unapplied mutation reports as a survivor, indistinguishable from a real gap.
- A survivor is a question, not a missing test. Ours split two ways.
- An equivalent mutant cannot be killed by any test. A leading
^in a pattern used only with.match()is redundant — no input distinguishes them — so the decision is about the code. - A real gap names its own test. Four inputs distinguish
\s+from\s*, and naming one is writing the missing test. - The real gap's consequence was worse than the mutation. The loose pattern reads
UVM_ERROR_SUMMARY(0)as an error andUVM_FATALITY(0)as a fatal — Chapter 24.3's severity-prefix bug arrived at from the other side. \S+cannot cross a space, which is why the loose pattern swallowsUVM_ERROR_SUMMARY(0)but notUVM_ERROR_COUNT drv.sv(1)— the boundary is worth understanding, not memorising.- A broken baseline scores 100%. The best-looking number in this chapter measures nothing, which is why step 1 of the loop is an assertion.
The next chapter turns from proving a script works to recording what it did while it ran.
Where this fits
Part of the Python curriculum.