Skip to content
VLSI Mentor

Python Part II · Module 36 · Python for VLSI Design Review

Reviewing a Parser: Can Malformed Input Silently Pass?

This is the most important question in the module, and the reason is asymmetry: a parser that reports a failure that did not happen wastes an afternoon, and a parser that reports a pass that did not happen ships the bug. The review is a battery of malformed inputs fed in during the review itself — and the first thing the battery teaches is that counting how many come back PASS gives the wrong answer, because some of them should.

Advanced26 min readPythonPart IIAutomationCode ReviewParsing

Part II · Module 36 · Chapter 36.2 · Reviewing a Parser: Can Malformed Input Silently Pass?

1. The Asymmetry That Makes This Critical

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
OUTCOMES = [
    ("a failure reported as a failure", "correct", "—"),
    ("a pass reported as a pass", "correct", "—"),
    ("a pass reported as a failure", "a false alarm", "an afternoon"),
    ("a failure reported as a pass", "a missed bug", "a tape-out"),
]

print(f"{'what happened -> what was reported':<38} {'class':<16} costs")
for what, cls, cost in OUTCOMES:
    print(f"{what:<38} {cls:<16} {cost}")

print()
print("the two errors are                :", "not symmetric")
print("a false alarm is                  :", "self-correcting — somebody looks")
print("a missed bug is                   :", "silent — nobody looks")
print()
print("so the review's question is        :",
      "can anything bad come back PASS?")
print("and not                            :", "is the parser correct?")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
what happened -> what was reported     class            costs
a failure reported as a failure        correct          —
a pass reported as a pass              correct          —
a pass reported as a failure           a false alarm    an afternoon
a failure reported as a pass           a missed bug     a tape-out

the two errors are                : not symmetric
a false alarm is                  : self-correcting — somebody looks
a missed bug is                   : silent — nobody looks

so the review's question is        : can anything bad come back PASS?
and not                            : is the parser correct?

The two error directions are not symmetric. A false alarm is self-correcting because somebody investigates and finds nothing; a missed bug is silent because a green result is nobody's action item. That asymmetry is why this chapter is the module's critical one and why the review question is narrower than "is the parser correct".

2. The Annotation

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
# during the review itself: feed it "" and a truncated log

During the review itself is the operative phrase. The alternative is to read the parser and reason about what it would do, which is slower and worse:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
APPROACHES = [
    ("read the regex and reason", "minutes each", "you reason about what you see"),
    ("ask the author", "minutes", "they reason about what they meant"),
    ("check the test suite", "minutes", "it tests what they thought of"),
    ("feed it the input", "seconds each", "it does what it does"),
]

print(f"{'approach':<30} {'cost':<16} what you learn")
for how, cost, learn in APPROACHES:
    print(f"{how:<30} {cost:<16} {learn}")

print()
print("the three reading approaches share:", "the author's blind spots")
print("the fourth shares                 :", "nothing")
print()
print("the two inputs the annotation names:", ['""', "a truncated log"])
print("why those two first                :",
      "they need no knowledge of the format")
print("and a parser that fails them       :",
      "fails before the format matters")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
approach                       cost             what you learn
read the regex and reason      minutes each     you reason about what you see
ask the author                 minutes          they reason about what they meant
check the test suite           minutes          it tests what they thought of
feed it the input              seconds each     it does what it does

the three reading approaches share: the author's blind spots
the fourth shares                 : nothing

the two inputs the annotation names: ['""', 'a truncated log']
why those two first                : they need no knowledge of the format
and a parser that fails them       : fails before the format matters

The three reading approaches all inherit the author's blind spots. The test suite is the clearest case: it covers the inputs the author thought of, so reading it tells you what they considered and not what they missed — which is Chapter 33.9's argument for mutation testing, applied to a review.

3. The Battery Needs an Expected Verdict

Here is the specimen, and the mistake a reviewer makes on the first attempt:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
def classify(text):
    """The specimen. This is a plausible parser that passes code review."""
    errors = sum(1 for l in text.splitlines()
                 if l.startswith("UVM_ERROR") and "UVM_ERROR :" not in l)
    fatals = sum(1 for l in text.splitlines()
                 if l.startswith("UVM_FATAL") and "UVM_FATAL :" not in l)
    if fatals:
        return "FATAL"
    if errors:
        return "FAIL"
    return "PASS"


INPUTS = [
    ("empty string", ""),
    ("whitespace only", "\n\n   \n"),
    ("truncated mid-line", "UVM_INFO a\nUVM_ERROR drv.sv(88) data mis"),
    ("no TEST DONE", "UVM_INFO a\nUVM_INFO b\n"),
    ("another tool's prefix",
     "Error-[SE] a.sv, 88\n  data mismatch\nTEST DONE\n"),
    ("a renamed message", "ERROR: drv.sv(88) data mismatch\nTEST DONE\n"),
    ("lowercase severity", "uvm_error drv.sv(88) x\nTEST DONE\n"),
    ("an indented log", "  UVM_ERROR drv.sv(88) x\nTEST DONE\n"),
    ("severity mid-line", "echo UVM_ERROR about to retry\nTEST DONE\n"),
    ("summary line only", "UVM_ERROR :    3\nTEST DONE\n"),
]

passes = [label for label, text in INPUTS if classify(text) == "PASS"]

print("the naive review: feed it everything and count the PASSes")
print()
for label, text in INPUTS:
    print(f"  {label:<24} {classify(text)}")
print()
print("malformed inputs tried:", len(INPUTS))
print("that came back PASS   :", len(passes))
print()
print("the obvious conclusion:", f"{len(passes)} defects")
print("and it is wrong because:", "some of those PASSes are CORRECT")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
the naive review: feed it everything and count the PASSes

  empty string             PASS
  whitespace only          PASS
  truncated mid-line       FAIL
  no TEST DONE             PASS
  another tool's prefix    PASS
  a renamed message        PASS
  lowercase severity       PASS
  an indented log          PASS
  severity mid-line        PASS
  summary line only        PASS

malformed inputs tried: 10
that came back PASS   : 9

the obvious conclusion: 9 defects
and it is wrong because: some of those PASSes are CORRECT

Eight PASSes, and that is not eight defects. Two of those eight are the behaviour you want: a line that mentions UVM_ERROR without being one must not count — Chapter 35.1 Section 6 established that — and a UVM_ERROR : summary line with zero real errors is a genuine pass.

So the battery is not a list of inputs. It is a list of (input, expected verdict) pairs, and building it forces the reviewer to decide what each input should produce before looking at what it does:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
BATTERY = [
    # label, text, expected, why
    ("a real failing log",
     "UVM_INFO a\nUVM_ERROR drv.sv(88) x\nUVM_ERROR :    1\nTEST DONE\n",
     "FAIL", "control"),
    ("a real passing log", "UVM_INFO a\nTEST DONE\n", "PASS", "control"),
    ("empty string", "", "UNUSABLE", "nothing ran"),
    ("whitespace only", "\n\n   \n", "UNUSABLE", "nothing ran"),
    ("truncated mid-line", "UVM_INFO a\nUVM_ERROR drv.sv(88) data mis",
     "UNUSABLE", "no TEST DONE"),
    ("no TEST DONE", "UVM_INFO a\nUVM_INFO b\n", "UNUSABLE",
     "did not finish"),
    ("another tool's prefix",
     "Error-[SE] a.sv, 88\n  data mismatch\nTEST DONE\n", "FAIL",
     "a real error"),
    ("a renamed message", "ERROR: drv.sv(88) data mismatch\nTEST DONE\n",
     "FAIL", "a real error"),
    ("lowercase severity", "uvm_error drv.sv(88) x\nTEST DONE\n", "FAIL",
     "a real error"),
    ("an indented log", "  UVM_ERROR drv.sv(88) x\nTEST DONE\n", "FAIL",
     "a real error"),
    ("severity mid-line", "echo UVM_ERROR about to retry\nTEST DONE\n",
     "PASS", "NOT an error — 35.1"),
    ("summary line only", "UVM_ERROR :    3\nTEST DONE\n", "PASS",
     "0 real errors"),
    ("a 200k info line", "UVM_INFO " + "x" * 200000 + "\nTEST DONE\n",
     "PASS", "just long"),
]

expected_pass = [b for b in BATTERY if b[2] == "PASS"]
print("inputs in the battery        :", len(BATTERY))
print("whose EXPECTED verdict is PASS:", len(expected_pass))
for label, _, _, why in expected_pass:
    print(f"   {label:<22} {why}")
print()
print("so a PASS is a defect only when:", "the expected verdict was not PASS")
print("and the battery must carry      :", "the expectation, not just the input")
print()
print("verdicts the battery needs      :",
      sorted({b[2] for b in BATTERY}))
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
inputs in the battery        : 13
whose EXPECTED verdict is PASS: 4
   a real passing log     control
   severity mid-line      NOT an error — 35.1
   summary line only      0 real errors
   a 200k info line       just long

so a PASS is a defect only when: the expected verdict was not PASS
and the battery must carry      : the expectation, not just the input

verdicts the battery needs      : ['FAIL', 'PASS', 'UNUSABLE']

Four of the thirteen inputs should produce PASS, three of them malformed in some way and correct to pass anyway. The last line is the other finding: the battery needs a verdict the parser cannot produce.

4. Thirteen Inputs, Eight Wrong

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
def classify(text):
    errors = sum(1 for l in text.splitlines()
                 if l.startswith("UVM_ERROR") and "UVM_ERROR :" not in l)
    fatals = sum(1 for l in text.splitlines()
                 if l.startswith("UVM_FATAL") and "UVM_FATAL :" not in l)
    if fatals:
        return "FATAL"
    if errors:
        return "FAIL"
    return "PASS"


BATTERY = [
    ("a real failing log",
     "UVM_INFO a\nUVM_ERROR drv.sv(88) x\nUVM_ERROR :    1\nTEST DONE\n",
     "FAIL", "control"),
    ("a real passing log", "UVM_INFO a\nTEST DONE\n", "PASS", "control"),
    ("empty string", "", "UNUSABLE", "nothing ran"),
    ("whitespace only", "\n\n   \n", "UNUSABLE", "nothing ran"),
    ("truncated mid-line", "UVM_INFO a\nUVM_ERROR drv.sv(88) data mis",
     "UNUSABLE", "no TEST DONE"),
    ("no TEST DONE", "UVM_INFO a\nUVM_INFO b\n", "UNUSABLE", "did not finish"),
    ("another tool's prefix",
     "Error-[SE] a.sv, 88\n  data mismatch\nTEST DONE\n", "FAIL",
     "a real error"),
    ("a renamed message", "ERROR: drv.sv(88) data mismatch\nTEST DONE\n",
     "FAIL", "a real error"),
    ("lowercase severity", "uvm_error drv.sv(88) x\nTEST DONE\n", "FAIL",
     "a real error"),
    ("an indented log", "  UVM_ERROR drv.sv(88) x\nTEST DONE\n", "FAIL",
     "a real error"),
    ("severity mid-line", "echo UVM_ERROR about to retry\nTEST DONE\n",
     "PASS", "NOT an error — 35.1"),
    ("summary line only", "UVM_ERROR :    3\nTEST DONE\n", "PASS",
     "0 real errors"),
    ("a 200k info line", "UVM_INFO " + "x" * 200000 + "\nTEST DONE\n",
     "PASS", "just long"),
]

print(f"{'input':<22} {'expected':<10} {'got':<8} {'ok':<6} why")
wrong = []
for label, text, expected, why in BATTERY:
    got = classify(text)
    ok = got == expected
    if not ok:
        wrong.append((label, expected, got, why))
    print(f"{label:<22} {expected:<10} {got:<8} {str(ok):<6} {why}")

print()
print("inputs            :", len(BATTERY))
print("correct verdicts  :", len(BATTERY) - len(wrong))
print("WRONG verdicts    :", len(wrong))
print()
vocab = [w for w in wrong if w[1] == "UNUSABLE"]
matcher = [w for w in wrong if w[1] == "FAIL"]
print("wrong because the parser has no UNUSABLE verdict:", len(vocab))
print("wrong because the matcher missed a real error   :", len(matcher))
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
input                  expected   got      ok     why
a real failing log     FAIL       FAIL     True   control
a real passing log     PASS       PASS     True   control
empty string           UNUSABLE   PASS     False  nothing ran
whitespace only        UNUSABLE   PASS     False  nothing ran
truncated mid-line     UNUSABLE   FAIL     False  no TEST DONE
no TEST DONE           UNUSABLE   PASS     False  did not finish
another tool's prefix  FAIL       PASS     False  a real error
a renamed message      FAIL       PASS     False  a real error
lowercase severity     FAIL       PASS     False  a real error
an indented log        FAIL       PASS     False  a real error
severity mid-line      PASS       PASS     True   NOT an error — 35.1
summary line only      PASS       PASS     True   0 real errors
a 200k info line       PASS       PASS     True   just long

inputs            : 13
correct verdicts  : 5
WRONG verdicts    : 8

wrong because the parser has no UNUSABLE verdict: 4
wrong because the matcher missed a real error   : 4

Thirteen inputs, five correct, eight wrong — and the eight split evenly into two defect classes with different fixes. That split is the finding to write in the review, because "the parser is wrong about eight inputs" is one comment and these are two.

5. Defect Class One: No Vocabulary for Unusable

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
VERDICTS = [
    ("PASS", "the run finished and found nothing", "yes"),
    ("FAIL", "the run finished and found errors", "yes"),
    ("FATAL", "the run died", "yes"),
    ("UNUSABLE", "the log cannot be read at all", "NO"),
]

print(f"{'verdict':<12} {'means':<38} the parser can emit it")
for v, means, can in VERDICTS:
    print(f"{v:<12} {means:<38} {can}")

print()
UNREADABLE = [
    ("empty string", "PASS", "a run that produced nothing is 'clean'"),
    ("whitespace only", "PASS", "same"),
    ("no TEST DONE", "PASS", "a hung run is 'clean'"),
    ("truncated mid-line", "FAIL", "accidentally right, for the wrong reason"),
]
print(f"{'unreadable input':<22} {'reported as':<12} what that claims")
for what, got, claims in UNREADABLE:
    print(f"{what:<22} {got:<12} {claims}")

print()
print("inputs that cannot be read  :", len(UNREADABLE))
print("reported as a clean run     :", 3)
print("accidentally reported FAIL  :", 1)
print()
print("the truncated one is the worst:", "it is right by accident")
print("because                       :",
      "the fragment happened to start with UVM_ERROR")
print("so a fix to the matcher would :", "turn it into a PASS")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
verdict      means                                  the parser can emit it
PASS         the run finished and found nothing     yes
FAIL         the run finished and found errors      yes
FATAL        the run died                           yes
UNUSABLE     the log cannot be read at all          NO

unreadable input       reported as  what that claims
empty string           PASS         a run that produced nothing is 'clean'
whitespace only        PASS         same
no TEST DONE           PASS         a hung run is 'clean'
truncated mid-line     FAIL         accidentally right, for the wrong reason

inputs that cannot be read  : 4
reported as a clean run     : 3
accidentally reported FAIL  : 1

the truncated one is the worst: it is right by accident
because                       : the fragment happened to start with UVM_ERROR
so a fix to the matcher would : turn it into a PASS

The parser has three verdicts and the battery needs four. Three unreadable logs come back as a clean run, and the fourth is right by accident — which is the most dangerous row in the table, because fixing the matcher without adding the verdict would turn that accidental FAIL into a PASS.

That is Chapter 35.8 Section 4's three-valued logic arriving in a parser. There, a diagnostic needed UNKNOWN because a check could not be performed; here a parser needs UNUSABLE because the input cannot be read, and in both cases a missing state is reported as the state that requires no action.

6. Defect Class Two: The Matcher Is Too Narrow

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
LINES = [
    ("UVM_ERROR drv.sv(88) x", True, "the canonical form"),
    ("  UVM_ERROR drv.sv(88) x", True, "indented by a wrapper"),
    ("uvm_error drv.sv(88) x", True, "lowercase from a different tool"),
    ("ERROR: drv.sv(88) x", True, "a renamed message"),
    ("Error-[SE] a.sv, 88", True, "another vendor's format"),
    ("echo UVM_ERROR about to retry", False, "a mention, not an error"),
    ("UVM_ERROR :    3", False, "the summary line"),
    ("UVM_INFO ERROR handling test", False, "the word in a test name"),
]


def narrow(line):
    """The specimen's matcher."""
    return line.startswith("UVM_ERROR") and "UVM_ERROR :" not in line


print(f"{'line':<34} {'should match':<13} {'does':<6} ok")
misses, false_pos = [], []
for text, should, why in LINES:
    does = narrow(text)
    ok = does == should
    if should and not does:
        misses.append(text)
    if does and not should:
        false_pos.append(text)
    print(f"{text[:34]:<34} {str(should):<13} {str(does):<6} {ok}")

print()
print("lines that SHOULD match     :", sum(1 for _, s, _ in LINES if s))
print("that the matcher misses     :", len(misses))
print("false positives             :", len(false_pos))
print()
print("so the matcher is           :", "too narrow and not too broad")
print("which is the easier direction:", "to fix wrongly — §7")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
line                               should match  does   ok
UVM_ERROR drv.sv(88) x             True          True   True
  UVM_ERROR drv.sv(88) x           True          False  False
uvm_error drv.sv(88) x             True          False  False
ERROR: drv.sv(88) x                True          False  False
Error-[SE] a.sv, 88                True          False  False
echo UVM_ERROR about to retry      False         False  True
UVM_ERROR :    3                   False         False  True
UVM_INFO ERROR handling test       False         False  True

lines that SHOULD match     : 5
that the matcher misses     : 4
false positives             : 0

so the matcher is           : too narrow and not too broad
which is the easier direction: to fix wrongly — §7

Four misses and zero false positives. The matcher is too narrow, which is the direction that loses errors, and the indented case is the one worth flagging hardest because it needs no exotic tool: a wrapper that prefixes every line — sed 's/^/ /', a log aggregator, a tee with a run label — turns every severity line invisible at once.

The three correct rejections are not luck and must survive any fix. echo UVM_ERROR is Chapter 35.1's mid-line mention, UVM_ERROR : is Chapter 24.3's summary line, and UVM_INFO ERROR handling test is a test name containing the word — three different reasons to reject, all of which a broadened matcher can break.

7. The Fix Has Two Halves and People Do One

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import re


def narrow(line):
    return line.startswith("UVM_ERROR") and "UVM_ERROR :" not in line


def too_broad(line):
    """The tempting fix: look for the word anywhere, case-insensitively."""
    return "error" in line.lower()


SEVERITY = re.compile(
    r"^(?:UVM_ERROR|UVM_FATAL|ERROR|FATAL|Error-\[\w+\])\b(?!\s+:)",
    re.IGNORECASE)


def correct(line):
    """Anchored after stripping indentation; the summary form excluded."""
    return SEVERITY.match(line.lstrip()) is not None


LINES = [
    ("UVM_ERROR drv.sv(88) x", True),
    ("  UVM_ERROR drv.sv(88) x", True),
    ("uvm_error drv.sv(88) x", True),
    ("ERROR: drv.sv(88) x", True),
    ("Error-[SE] a.sv, 88", True),
    ("echo UVM_ERROR about to retry", False),
    ("UVM_ERROR :    3", False),
    ("UVM_INFO ERROR handling test", False),
]

print(f"{'line':<32} {'want':<6} {'narrow':<8} {'broad':<7} correct")
for text, want in LINES:
    print(f"{text[:32]:<32} {str(want):<6} {str(narrow(text)):<8} "
          f"{str(too_broad(text)):<7} {correct(text)}")

for name, fn in (("narrow", narrow), ("too_broad", too_broad),
                 ("correct", correct)):
    miss = sum(1 for t, w in LINES if w and not fn(t))
    fp = sum(1 for t, w in LINES if not w and fn(t))
    print()
    print(f"{name:<10} misses={miss} false positives={fp}")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
line                             want   narrow   broad   correct
UVM_ERROR drv.sv(88) x           True   True     True    True
  UVM_ERROR drv.sv(88) x         True   False    True    True
uvm_error drv.sv(88) x           True   False    True    True
ERROR: drv.sv(88) x              True   False    True    True
Error-[SE] a.sv, 88              True   False    True    True
echo UVM_ERROR about to retry    False  False    True    False
UVM_ERROR :    3                 False  False    True    False
UVM_INFO ERROR handling test     False  False    True    False

narrow     misses=4 false positives=0

too_broad  misses=0 false positives=3

correct    misses=0 false positives=0

The tempting fix trades four misses for three false positives. "error" in line.lower() catches everything the narrow matcher missed and also matches the mid-line mention, the summary line and a test called ERROR handling — so a clean run reports three errors and the team stops trusting the tool.

The working version keeps both halves: it strips indentation before anchoring, which handles the wrapper case and the canonical case with one rule; it is case-insensitive, which handles the other tool; it lists the alternative prefixes explicitly; and it uses a negative lookahead to exclude the summary form rather than a substring test, so the exclusion travels with the pattern instead of being a separate condition somebody can delete.

That lookahead is also where this chapter's own first attempt was wrong, and the mistake is worth having. Written as (?!\s*:) — "not followed by optional whitespace and a colon" — it correctly rejects UVM_ERROR : 3 and also rejects ERROR: drv.sv(88) x, because the renamed form legitimately ends its prefix with a colon. Measured, that version had one miss:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
line                             want   (?!\s*:)  (?!\s+:)
UVM_ERROR drv.sv(88) x           True   True      True
ERROR: drv.sv(88) x              True   FALSE     True
UVM_ERROR :    3                 False  False     False

The distinguishing feature is whitespace before the colon: the summary form has at least one space, a prefix ending in a colon has none. (?!\s+:) takes the first and keeps the second. And the way the mistake surfaced is exactly Section 7's own advice — it was not visible in the misses, which all matched; it appeared when the battery in Section 8 was re-run and the renamed row came back PASS.

8. The Fixed Parser, Re-measured

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import re
from dataclasses import dataclass

SEVERITY = re.compile(
    r"^(?:UVM_FATAL|FATAL|UVM_ERROR|ERROR|Error-\[\w+\])\b(?!\s+:)",
    re.IGNORECASE)
FATAL = re.compile(r"^(?:UVM_FATAL|FATAL)\b(?!\s+:)", re.IGNORECASE)
DONE_MARKER = "TEST DONE"


@dataclass(frozen=True)
class Verdict:
    status: str
    reason: str


def classify(text):
    """Four verdicts. UNUSABLE is checked first and is not a fall-through."""
    if not text.strip():
        return Verdict("UNUSABLE", "the log is empty")
    lines = text.splitlines()
    if not any(l.strip() == DONE_MARKER for l in lines):
        return Verdict("UNUSABLE", f"no {DONE_MARKER!r} line: "
                                   "the run did not finish")
    fatals = [l for l in lines if FATAL.match(l.lstrip())]
    errors = [l for l in lines if SEVERITY.match(l.lstrip())
              and not FATAL.match(l.lstrip())]
    if fatals:
        return Verdict("FATAL", f"{len(fatals)} fatal line(s)")
    if errors:
        return Verdict("FAIL", f"{len(errors)} error line(s)")
    return Verdict("PASS", "finished with no severity lines")


BATTERY = [
    ("a real failing log",
     "UVM_INFO a\nUVM_ERROR drv.sv(88) x\nUVM_ERROR :    1\nTEST DONE\n",
     "FAIL"),
    ("a real passing log", "UVM_INFO a\nTEST DONE\n", "PASS"),
    ("empty string", "", "UNUSABLE"),
    ("whitespace only", "\n\n   \n", "UNUSABLE"),
    ("truncated mid-line", "UVM_INFO a\nUVM_ERROR drv.sv(88) data mis",
     "UNUSABLE"),
    ("no TEST DONE", "UVM_INFO a\nUVM_INFO b\n", "UNUSABLE"),
    ("another tool's prefix",
     "Error-[SE] a.sv, 88\n  data mismatch\nTEST DONE\n", "FAIL"),
    ("a renamed message", "ERROR: drv.sv(88) data mismatch\nTEST DONE\n",
     "FAIL"),
    ("lowercase severity", "uvm_error drv.sv(88) x\nTEST DONE\n", "FAIL"),
    ("an indented log", "  UVM_ERROR drv.sv(88) x\nTEST DONE\n", "FAIL"),
    ("severity mid-line", "echo UVM_ERROR about to retry\nTEST DONE\n",
     "PASS"),
    ("summary line only", "UVM_ERROR :    3\nTEST DONE\n", "PASS"),
    ("a 200k info line", "UVM_INFO " + "x" * 200000 + "\nTEST DONE\n",
     "PASS"),
]

print(f"{'input':<22} {'expected':<10} {'got':<10} {'ok':<6} reason")
wrong = 0
for label, text, expected in BATTERY:
    v = classify(text)
    ok = v.status == expected
    wrong += not ok
    print(f"{label:<22} {expected:<10} {v.status:<10} {str(ok):<6} "
          f"{v.reason[:30]}")

print()
print("inputs          :", len(BATTERY))
print("correct verdicts:", len(BATTERY) - wrong)
print("wrong verdicts  :", wrong)
print()
print("before the fix  :", "5 of 13 correct")
print("after the fix   :", f"{len(BATTERY) - wrong} of {len(BATTERY)} correct")
print()
print("the four correct PASSes still PASS:",
      all(classify(t).status == "PASS" for l, t, e in BATTERY
          if e == "PASS"))
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
input                  expected   got        ok     reason
a real failing log     FAIL       FAIL       True   1 error line(s)
a real passing log     PASS       PASS       True   finished with no severity line
empty string           UNUSABLE   UNUSABLE   True   the log is empty
whitespace only        UNUSABLE   UNUSABLE   True   the log is empty
truncated mid-line     UNUSABLE   UNUSABLE   True   no 'TEST DONE' line: the run d
no TEST DONE           UNUSABLE   UNUSABLE   True   no 'TEST DONE' line: the run d
another tool's prefix  FAIL       FAIL       True   1 error line(s)
a renamed message      FAIL       FAIL       True   1 error line(s)
lowercase severity     FAIL       FAIL       True   1 error line(s)
an indented log        FAIL       FAIL       True   1 error line(s)
severity mid-line      PASS       PASS       True   finished with no severity line
summary line only      PASS       PASS       True   finished with no severity line
a 200k info line       PASS       PASS       True   finished with no severity line

inputs          : 13
correct verdicts: 13
wrong verdicts  : 0

before the fix  : 5 of 13 correct
after the fix   : 13 of 13 correct

the four correct PASSes still PASS: True

Thirteen of thirteen, and the four correct PASSes are still PASS. That last line is the assertion the review's second pass exists for: the fix added UNUSABLE, stripped indentation, became case-insensitive and learned three more prefixes, and did not break a single one of the rejections.

Two structural points about the fixed version are worth a reviewer's attention. UNUSABLE is checked first and explicitly, so it is not a fall-through — and PASS is now the only fall-through, which is correct because "finished and found nothing" genuinely is the remaining case once unreadability has been excluded. And the Verdict carries a reason, so the four UNUSABLE rows distinguish an empty log from an unfinished one without the caller parsing a string.

9. The Battery as a Review Artefact

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import io
import re
import unittest
from dataclasses import dataclass

SEVERITY = re.compile(
    r"^(?:UVM_FATAL|FATAL|UVM_ERROR|ERROR|Error-\[\w+\])\b(?!\s+:)",
    re.IGNORECASE)
FATAL = re.compile(r"^(?:UVM_FATAL|FATAL)\b(?!\s+:)", re.IGNORECASE)


@dataclass(frozen=True)
class Verdict:
    status: str
    reason: str


def classify(text):
    if not text.strip():
        return Verdict("UNUSABLE", "the log is empty")
    lines = text.splitlines()
    if not any(l.strip() == "TEST DONE" for l in lines):
        return Verdict("UNUSABLE", "no 'TEST DONE' line")
    fatals = [l for l in lines if FATAL.match(l.lstrip())]
    errors = [l for l in lines if SEVERITY.match(l.lstrip())
              and not FATAL.match(l.lstrip())]
    if fatals:
        return Verdict("FATAL", f"{len(fatals)} fatal line(s)")
    if errors:
        return Verdict("FAIL", f"{len(errors)} error line(s)")
    return Verdict("PASS", "finished with no severity lines")


# The battery, checked into the repository next to the parser.
BATTERY = [
    ("control_fail", "UVM_INFO a\nUVM_ERROR d.sv(1) x\nTEST DONE\n", "FAIL"),
    ("control_pass", "UVM_INFO a\nTEST DONE\n", "PASS"),
    ("empty", "", "UNUSABLE"),
    ("whitespace", "   \n\n", "UNUSABLE"),
    ("truncated", "UVM_INFO a\nUVM_ERROR d.sv(1) data mis", "UNUSABLE"),
    ("no_done", "UVM_INFO a\n", "UNUSABLE"),
    ("other_vendor", "Error-[SE] a.sv, 1\nTEST DONE\n", "FAIL"),
    ("renamed", "ERROR: d.sv(1) x\nTEST DONE\n", "FAIL"),
    ("lowercase", "uvm_error d.sv(1) x\nTEST DONE\n", "FAIL"),
    ("indented", "  UVM_ERROR d.sv(1) x\nTEST DONE\n", "FAIL"),
    ("mention", "echo UVM_ERROR retrying\nTEST DONE\n", "PASS"),
    ("summary_only", "UVM_ERROR :    3\nTEST DONE\n", "PASS"),
    ("fatal", "UVM_FATAL e.sv(1) x\nTEST DONE\n", "FATAL"),
]


class Battery(unittest.TestCase):
    """One test per battery row, named by the row."""

    def test_every_row(self):
        for name, text, expected in BATTERY:
            with self.subTest(row=name):
                self.assertEqual(classify(text).status, expected)

    def test_unusable_rows_say_why(self):
        for name, text, expected in BATTERY:
            if expected == "UNUSABLE":
                with self.subTest(row=name):
                    self.assertTrue(classify(text).reason)

    def test_the_battery_covers_every_verdict(self):
        self.assertEqual(sorted({e for _, _, e in BATTERY}),
                         ["FAIL", "FATAL", "PASS", "UNUSABLE"])

    def test_the_battery_has_both_controls(self):
        names = {n for n, _, _ in BATTERY}
        self.assertTrue({"control_pass", "control_fail"} <= names)

    def test_rejections_are_represented(self):
        """The §7 lesson: the lines that must NOT match are in the battery."""
        rejections = {"mention", "summary_only"}
        self.assertTrue(rejections <= {n for n, _, _ in BATTERY})


def run(case):
    suite = unittest.TestLoader().loadTestsFromTestCase(case)
    res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
    return {"run": res.testsRun, "failures": len(res.failures),
            "ok": res.wasSuccessful()}


print("battery suite:", run(Battery))
print()
print("battery rows            :", len(BATTERY))
print("verdicts covered        :", sorted({e for _, _, e in BATTERY}))
print("rows expecting PASS     :", sum(1 for _, _, e in BATTERY if e == "PASS"))
print("rows expecting UNUSABLE :",
      sum(1 for _, _, e in BATTERY if e == "UNUSABLE"))
print()
print("the battery lives       :", "next to the parser, in the repository")
print("so the next reviewer     :", "adds a row instead of repeating the work")
print("and a new format         :", "is a row, not an argument")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
battery suite: {'run': 5, 'failures': 0, 'ok': True}

battery rows            : 13
verdicts covered        : ['FAIL', 'FATAL', 'PASS', 'UNUSABLE']
rows expecting PASS     : 3
rows expecting UNUSABLE : 4

the battery lives       : next to the parser, in the repository
so the next reviewer     : adds a row instead of repeating the work
and a new format         : is a row, not an argument

Five tests over thirteen rows, with the battery itself checked in. That is the artefact this chapter produces: a review that only leaves comments is work somebody repeats next quarter, and a review that leaves a battery makes the next reviewer's job adding a row.

Three of the five tests are about the battery rather than about the parser. test_the_battery_covers_every_verdict fails if a verdict has no row; test_the_battery_has_both_controls fails if somebody deletes the known-good and known-bad cases, which is how a battery degrades into a list of edge cases with nothing to calibrate it; and test_rejections_are_represented encodes Section 7's lesson so a future broadening of the matcher cannot quietly drop the lines that must not match.

10. Which of the Thirteen Are Load-Bearing?

Section 9 argued the battery is the review artefact worth keeping. That argument is only as good as the battery, and there is a way to score it that does not rely on anybody's judgement: break the fixed parser one capability at a time and count which inputs notice. An input that notices nothing is carrying no weight.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import re

DONE = "TEST DONE"


def make_parser(lstrip=True, icase=True, lookahead=True, empty=True,
                done=True, prefixes=True):
    """The Section 8 parser, with each capability switchable off."""
    pref = (r"UVM_FATAL|FATAL|UVM_ERROR|ERROR|Error-\[\w+\]" if prefixes
            else r"UVM_FATAL|UVM_ERROR")
    la = r"(?!\s+:)" if lookahead else ""
    flags = re.IGNORECASE if icase else 0
    SEV = re.compile(rf"^(?:{pref})\b{la}", flags)
    FAT = re.compile(rf"^(?:UVM_FATAL|FATAL)\b{la}", flags)

    def classify(text):
        if empty and not text.strip():
            return "UNUSABLE"
        lines = text.splitlines()
        if done and not any(l.strip() == DONE for l in lines):
            return "UNUSABLE"
        prep = (lambda l: l.lstrip()) if lstrip else (lambda l: l)
        fatals = [l for l in lines if FAT.match(prep(l))]
        errors = [l for l in lines if SEV.match(prep(l))
                  and not FAT.match(prep(l))]
        if fatals:
            return "FATAL"
        if errors:
            return "FAIL"
        return "PASS"

    return classify


BATTERY = [
    ("a real failing log", "UVM_INFO a\nUVM_ERROR drv.sv(88) x\n"
                           "UVM_ERROR :    1\nTEST DONE\n", "FAIL"),
    ("a real passing log", "UVM_INFO a\nTEST DONE\n", "PASS"),
    ("empty string", "", "UNUSABLE"),
    ("whitespace only", "\n\n   \n", "UNUSABLE"),
    ("truncated mid-line", "UVM_INFO a\nUVM_ERROR drv.sv(88) data mis",
     "UNUSABLE"),
    ("no TEST DONE", "UVM_INFO a\nUVM_INFO b\n", "UNUSABLE"),
    ("another tool's prefix", "Error-[SE] a.sv, 88\n  data mismatch\n"
                              "TEST DONE\n", "FAIL"),
    ("a renamed message", "ERROR: drv.sv(88) data mismatch\nTEST DONE\n",
     "FAIL"),
    ("lowercase severity", "uvm_error drv.sv(88) x\nTEST DONE\n", "FAIL"),
    ("an indented log", "  UVM_ERROR drv.sv(88) x\nTEST DONE\n", "FAIL"),
    ("severity mid-line", "echo UVM_ERROR about to retry\nTEST DONE\n",
     "PASS"),
    ("summary line only", "UVM_ERROR :    3\nTEST DONE\n", "PASS"),
    ("a 200k info line", "UVM_INFO " + "x" * 200000 + "\nTEST DONE\n",
     "PASS"),
]

CAPABILITIES = [
    ("strip indentation", "lstrip"),
    ("case-insensitive", "icase"),
    ("the (?!\\s+:) lookahead", "lookahead"),
    ("the empty-log check", "empty"),
    ("the TEST DONE check", "done"),
    ("the extra prefixes", "prefixes"),
]

print("the fixed parser is clean on all thirteen:",
      all(make_parser()(t) == e for _, t, e in BATTERY))
print()

noticed = {label: [] for label, _, _ in BATTERY}
print(f"{'capability removed':<26} {'noticed by':>10}  which")
for name, flag in CAPABILITIES:
    broken = make_parser(**{flag: False})
    catchers = [label for label, text, exp in BATTERY
                if broken(text) != exp]
    for label in catchers:
        noticed[label].append(name)
    print(f"{name:<26} {len(catchers):>10}  {catchers}")

print()
print(f"{'input':<22} {'breaks it notices':>18}  irreplaceable for")
for label, _, _ in BATTERY:
    unique = [n for n in noticed[label]
              if sum(1 for v in noticed.values() if n in v) == 1]
    print(f"{label:<22} {len(noticed[label]):>18}  {unique}")

print()
print("inputs noticing nothing  :",
      sum(1 for v in noticed.values() if not v))
print("inputs that are the ONLY ")
print("  witness to some break  :",
      sum(1 for label, _, _ in BATTERY
          if any(sum(1 for v in noticed.values() if n in v) == 1
                 for n in noticed[label])))
print("capabilities no input    ")
print("  can justify            :",
      sum(1 for name, flag in CAPABILITIES
          if all(make_parser(**{flag: False})(t) == e for _, t, e in BATTERY)))
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
the fixed parser is clean on all thirteen: True

capability removed         noticed by  which
strip indentation                   1  ['an indented log']
case-insensitive                    2  ["another tool's prefix", 'lowercase severity']
the (?!\s+:) lookahead              1  ['summary line only']
the empty-log check                 0  []
the TEST DONE check                 2  ['truncated mid-line', 'no TEST DONE']
the extra prefixes                  2  ["another tool's prefix", 'a renamed message']

input                   breaks it notices  irreplaceable for
a real failing log                      0  []
a real passing log                      0  []
empty string                            0  []
whitespace only                         0  []
truncated mid-line                      1  []
no TEST DONE                            1  []
another tool's prefix                   2  []
a renamed message                       1  []
lowercase severity                      1  []
an indented log                         1  ['strip indentation']
severity mid-line                       0  []
summary line only                       1  ['the (?!\\s+:) lookahead']
a 200k info line                        0  []

inputs noticing nothing  : 6
inputs that are the ONLY 
  witness to some break  : 2
capabilities no input    
  can justify            : 1

Six of the thirteen inputs notice nothing, two are irreplaceable, and one capability of the parser cannot be justified by any input in the battery.

Take the last one first, because it is a finding about this chapter's own fixed parser. Removing the empty-log check changes no verdict at all: an empty string has no TEST DONE line either, so the second check catches it and returns UNUSABLE anyway. By the only measure the battery can apply, that branch is dead weight.

It is not dead, and the reason is the half the battery does not look at:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import re

DONE = "TEST DONE"
SEV = re.compile(
    r"^(?:UVM_FATAL|FATAL|UVM_ERROR|ERROR|Error-\[\w+\])\b(?!\s+:)", re.I)
FAT = re.compile(r"^(?:UVM_FATAL|FATAL)\b(?!\s+:)", re.I)


def classify(text, empty_check=True):
    if empty_check and not text.strip():
        return "UNUSABLE", "the log is empty"
    lines = text.splitlines()
    if not any(l.strip() == DONE for l in lines):
        return "UNUSABLE", f"no {DONE!r} line: the run did not finish"
    fatals = [l for l in lines if FAT.match(l.lstrip())]
    errors = [l for l in lines if SEV.match(l.lstrip())
              and not FAT.match(l.lstrip())]
    if fatals:
        return "FATAL", f"{len(fatals)} fatal line(s)"
    if errors:
        return "FAIL", f"{len(errors)} error line(s)"
    return "PASS", "finished with no severity lines"


for label, text in (("empty string", ""), ("whitespace only", "\n\n   \n")):
    kept = classify(text, True)
    dropped = classify(text, False)
    print(f"{label}")
    print(f"  with the check : {kept[0]:<9} {kept[1]!r}")
    print(f"  without it     : {dropped[0]:<9} {dropped[1]!r}")
    print(f"  same status    : {kept[0] == dropped[0]}")
    print(f"  same reason    : {kept[1] == dropped[1]}")
    print()

print("so the empty-log check changes:", "the reason, never the status")
print("and the battery compares      :", "statuses only")
print("which makes the branch        :", "unjustifiable by this battery")
print("and the review finding        :", "assert the reason, not just the status")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
empty string
  with the check : UNUSABLE  'the log is empty'
  without it     : UNUSABLE  "no 'TEST DONE' line: the run did not finish"
  same status    : True
  same reason    : False

whitespace only
  with the check : UNUSABLE  'the log is empty'
  without it     : UNUSABLE  "no 'TEST DONE' line: the run did not finish"
  same status    : True
  same reason    : False

so the empty-log check changes: the reason, never the status
and the battery compares      : statuses only
which makes the branch        : unjustifiable by this battery
and the review finding        : assert the reason, not just the status

The branch changes the reason and never the status. Without it, an empty log is reported as "no 'TEST DONE' line: the run did not finish" — which is wrong about an empty file, and wrong in the direction that sends an engineer to look at a simulation that never produced a byte.

So the finding is not "delete the empty check". It is the battery is checking half of what the parser returns. Section 8 added a reason to the Verdict precisely so a caller could tell an empty log from an unfinished one, and then the battery compares statuses — so the field the fix was built for is the field nothing tests. That is Chapter 33.9's survivor in a different costume: the capability is real, and the suite cannot tell whether it is there.

The two other columns are worth a reviewer's time for a different reason:

  • The two controls notice nothing. A real failing log and a real passing log are the first inputs anybody writes, and neither detects any of the six breaks. They are not useless — they are the inputs that catch a parser which has stopped working altogether — but they contribute nothing to the eight defects this chapter is about.
  • an indented log and summary line only are irreplaceable. Each is the sole witness to one capability. Delete either and the battery goes quiet about a defect that Section 6 and Section 7 spend their length on — the indentation strip and the lookahead that Section 7 got wrong on its first attempt.

11. Predict, Run, Observe

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import re

SEVERITY = re.compile(
    r"^(?:UVM_FATAL|FATAL|UVM_ERROR|ERROR|Error-\[\w+\])\b(?!\s+:)",
    re.IGNORECASE)


def narrow(line):
    return line.startswith("UVM_ERROR") and "UVM_ERROR :" not in line


# A
print("A:", narrow("UVM_ERROR x"), narrow("  UVM_ERROR x"))

# B
print("B:", bool(SEVERITY.match("  UVM_ERROR x".lstrip())),
      bool(SEVERITY.match("UVM_ERROR :    3".lstrip())))

# C
print("C:", "error" in "UVM_INFO ERROR handling test".lower())

# D
def fallthrough(text):
    return "FAIL" if any(narrow(l) for l in text.splitlines()) else "PASS"


print("D:", fallthrough(""), fallthrough("UVM_INFO a\n"))

The verified output:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
A: True False
B: True False
C: True
D: PASS PASS

A — the canonical line matches and the indented one does not. One sed away from invisible.

B — the fixed pattern matches the indented line after lstrip() and still rejects the summary line, which is both halves of Section 7 in one line.

C — the tempting broad fix matches a test name containing the word error. That is the false positive that gets a tool switched off.

D — and the fall-through: an empty log and a log with no result both come back PASS. Two completely different situations, the same answer, and the answer is the one that requires nobody to act.

12. Exercises

Exercise 1 — the asymmetry. Write down, for your own flow, the cost of a false alarm and the cost of a missed bug.

Exercise 2 — the narrower question. Restate "is the parser correct" as the question a review can answer.

Exercise 3 — during the review. Feed "" to a parser your team owns, right now, and record the verdict.

Exercise 4 — the author's blind spots. Explain why reading the test suite inherits them.

Exercise 5 — count the PASSes. Run a ten-input battery and count them.

Exercise 6 — now say which are defects. Decide the expected verdict for each, and recount.

Exercise 7 — the correct PASSes. Identify the malformed inputs that should pass anyway.

Exercise 8 — expected verdicts first. Explain why the expectation must be written before running anything.

Exercise 9 — two defect classes. Split your own wrong verdicts into vocabulary and matcher failures.

Exercise 10 — count your verdicts. List the statuses your parser can emit and the ones your battery needs.

Exercise 11 — the fall-through. Find a return "PASS" as the last line of a classifier and say what it conflates.

Exercise 12 — right by accident. Find a case your parser gets right for the wrong reason.

Exercise 13 — fixing one half. Explain what happens to that case if the matcher is fixed and the verdict is not.

Exercise 14 — the indented log. Pipe a log through sed 's/^/ /' and re-run your parser.

Exercise 15 — four misses, zero false positives. Characterise your matcher's error direction.

Exercise 16 — the three rejections. List the lines that must not match, and why each one must not.

Exercise 17 — the tempting fix. Implement the broad matcher and count its false positives.

Exercise 18 — strip then anchor. Rewrite your matcher to handle indentation and the canonical form with one rule.

Exercise 19 — the negative lookahead. Move a summary-line exclusion into the pattern and say what that buys.

Exercise 20 — review the rejections. On a second pass, feed only the lines that must not match.

Exercise 21 — re-measure. Run the whole battery against the fixed parser and confirm nothing regressed.

Exercise 22 — the only fall-through. Justify PASS being the last line once UNUSABLE is explicit.

Exercise 23 — a reason per verdict. Add one and say which rows need it most.

Exercise 24 — check the battery in. Put yours next to the parser.

Exercise 25 — tests about the battery. Add the coverage, controls and rejection tests.

Exercise 26 — a new format is a row. Describe what happens next time a tool changes its message prefix.

Exercise 27 — ablate one capability. Switch off one feature of your own parser and count which battery inputs notice.

Exercise 28 — score every input. Build the full table and find the inputs that notice nothing.

Exercise 29 — the irreplaceable ones. Identify the inputs that are the sole witness to some break.

Exercise 30 — the unjustifiable branch. Find a capability no input in your battery can justify.

Exercise 31 — dead or untested. Decide which of the two it is, and say how you told them apart.

Exercise 32 — status against reason. Assert the reason as well as the status, and re-run the ablation.

Exercise 33 — the controls. Explain what the two real logs are for, given that they notice none of the breaks.

Exercise 34 — not a deletion list. Say why a 200k info line should stay even though it scores zero.

Exercise 35 — the survivor, again. State this finding in Chapter 33.9's vocabulary.

13. Summary

  • The two error directions are not symmetric. A false alarm is self-correcting; a missed bug is silent, which is why the review question is "can anything bad come back PASS?" rather than "is the parser correct?"
  • Feed it the input during the review. Reading the regex, asking the author and reading the test suite all inherit the author's blind spots.
  • Counting PASSes gives the wrong number. Eight of ten malformed inputs came back PASS and two of those eight were correct.
  • The battery is a list of (input, expected verdict) pairs, and writing the expectation first is what turns "it returned PASS" into a defect or a confirmation.
  • Thirteen inputs, five correct verdicts, eight wrong, splitting evenly into two classes with different fixes.
  • Four wrong verdicts are a vocabulary failure. The parser emits PASS, FAIL and FATAL; the battery needs UNUSABLE, so three unreadable logs come back as a clean run.
  • One unreadable log was right by accident, which makes it the most dangerous row: fixing the matcher alone turns that FAIL into a PASS.
  • A bare return "PASS" as a classifier's last line conflates "found nothing" with "could not look", and is worth a review comment every time.
  • Four wrong verdicts are a matcher failure — another vendor's prefix, a renamed message, a lowercase severity and an indented log, which one sed in a wrapper is enough to produce.
  • The matcher was too narrow with zero false positives, and the tempting fix trades four misses for three false positives by matching a mid-line mention, a summary line and a test name.
  • The working fix has four parts: strip indentation before anchoring, be case-insensitive, list the alternative prefixes, and exclude the summary form with a lookahead inside the pattern.
  • Review the fix against the rejections, not the misses. The author will verify the four misses; nobody re-checks the three lines that must not match.
  • Thirteen of thirteen after the fix, with every correct PASS preserved, and UNUSABLE checked explicitly so PASS is the only fall-through left.
  • Check the battery in next to the parser, and add tests about the battery
  • Ablating one capability at a time scores the battery without anybody's judgement: six of thirteen inputs notice nothing, two are the sole witness to a break, and one capability no input can justify.
  • Removing the empty-log check changes no verdict — an empty log has no TEST DONE line either, so the next check returns UNUSABLE anyway.
  • It changes the reason, never the status, and reports an empty log as unfinished — which sends an engineer to look at a run that never produced a byte.
  • So the finding is that the battery tests half of what the parser returns. Section 8 added reason so a caller could tell those apart, and nothing asserts it — Chapter 33.9's survivor in a different costume.
  • The two real logs notice none of the six breaks. They catch a parser that has stopped working altogether and contribute nothing to the eight defects.
  • an indented log and summary line only are irreplaceable — each is the only witness to one capability, including the lookahead Section 7 got wrong at the first attempt.
  • An input scoring zero is not an input to delete: the ablation knows only these six breaks, and a 200k info line pins a claim none of them touches. — every verdict covered, both controls present, the rejections represented — so the next reviewer adds a row instead of repeating the work.

The next chapter reviews the layer this one took for granted: the code that runs the tool, where the question is whether a reviewer can reproduce the failure at all.

Where this fits

Part of the Python curriculum.