Python Part II · Module 33 · Testing and Debugging Automation
Negative Tests: Malformed, Empty and Truncated Input
Chapter 33.3 and 33.4 pinned what the parser does with logs that make sense. A bad night does not produce those. It produces a log the simulator was killed halfway through writing, a zero-byte file from a job that never started, a report from a different tool because somebody fixed a path wrongly, and — the one nobody tests — last night's log still sitting in a directory that got reused. This chapter feeds a parser eight of them and sorts the answers into the ones that name the problem and the ones that guess.
Intermediate21 min readPythonPart IIAutomationTestingRobustness
Part II · Module 33 · Chapter 33.5 · Negative Tests: Malformed, Empty and Truncated Input
1. What a Bad Night Produces
INPUTS = [
("a complete log", "the happy path", "33.3"),
("an empty file", "the job never started", "the annotation"),
("a truncated log", "the simulator was killed", "the annotation"),
("another tool's format", "a path was fixed wrongly", "§6"),
("last night's log", "a reused run directory", "§7"),
("non-UTF-8 bytes", "a corrupt write, or a core file", "§5"),
("a directory, not a file", "a glob matched the wrong thing", "§5"),
("a missing file", "the tool wrote it elsewhere", "§5"),
]
print(f"{'input':<26} {'how it arises':<34} covered in")
for what, why, where in INPUTS:
print(f"{what:<26} {why:<34} {where}")
print()
print("inputs :", len(INPUTS))
print("the annotation covers:", 2)
print("every one of these:", "arrives from the world, not from your code")input how it arises covered in
a complete log the happy path 33.3
an empty file the job never started the annotation
a truncated log the simulator was killed the annotation
another tool's format a path was fixed wrongly §6
last night's log a reused run directory §7
non-UTF-8 bytes a corrupt write, or a core file §5
a directory, not a file a glob matched the wrong thing §5
a missing file the tool wrote it elsewhere §5
inputs : 8
the annotation covers: 2
every one of these: arrives from the world, not from your codeEight inputs, and the last line is why they need tests rather than care: none of them is produced by your code, so none of them appears in a code review and all of them appear on the night something else goes wrong.
2. The Annotation
assert classify("") == "EMPTY_LOG"
assert classify(TRUNCATED) == "NO_FINISH"def classify(text):
if not text.strip():
return "EMPTY_LOG"
if "UVM_FATAL" in text:
return "FAIL"
if "TEST DONE" not in text:
return "NO_FINISH"
if "UVM_ERROR" in text:
return "FAIL"
return "PASS"
TRUNCATED = ("UVM_INFO tb.sv(42) @ 0: reset released\n"
"UVM_INFO axi_seq.sv(88) @ 1200: starting burst 0\n"
"UVM_INFO axi_seq.sv(88) @ 1400: start")
assert classify("") == "EMPTY_LOG"
assert classify(TRUNCATED) == "NO_FINISH"
print("both annotation assertions pass")
print()
print("the truncated sample ends mid-word:", repr(TRUNCATED[-12:]))
print("it has no end banner :", "TEST DONE" not in TRUNCATED)
print("so it classifies as :", classify(TRUNCATED))both annotation assertions pass
the truncated sample ends mid-word: ' 1400: start'
it has no end banner : True
so it classifies as : NO_FINISHBoth lines of the annotation are correct, which is worth saying plainly because it is unusual in this book. EMPTY_LOG and NO_FINISH are the right answers and the classifier produces them.
So this chapter extends rather than corrects. Six inputs remain, and the question for each is the purpose's: does the parser say so, or does it guess?
3. Honest and Guessed
def classify(text):
if not text.strip():
return "EMPTY_LOG"
if "UVM_FATAL" in text:
return "FAIL"
if "TEST DONE" not in text:
return "NO_FINISH"
if "UVM_ERROR" in text:
return "FAIL"
return "PASS"
QUESTA = ("# Loading work.tb\n"
"# ** Error: (vsim-3601) drv.sv(88): bad handshake\n"
"# ** Error: (vsim-3601) drv.sv(91): bad handshake\n"
"# End time: 03:14:15 on Feb 18,2026\n")
CRLF = "UVM_INFO started\r\nUVM_ERROR drv.sv(88): bad\r\nTEST DONE\r\n"
NULLS = "UVM_INFO started\x00\x00\x00\nTEST DONE\n"
STALE = ("UVM_INFO tb.sv(42) @ 0: reset released\n"
"UVM_INFO tb.sv(90) @ 158300: finishing\n"
"TEST DONE\n")
CASES = [
("empty", "", "EMPTY_LOG", "honest"),
("whitespace only", " \n\n", "EMPTY_LOG", "honest"),
("truncated", "UVM_INFO started\n", "NO_FINISH", "honest"),
("another tool's format", QUESTA, "?", "?"),
("CRLF line endings", CRLF, "?", "?"),
("embedded null bytes", NULLS, "?", "?"),
("last night's clean log", STALE, "?", "?"),
]
print(f"{'input':<26} {'classify() says':<14} verdict")
for name, text, _, _ in CASES:
got = classify(text)
honest = got in ("EMPTY_LOG", "NO_FINISH") or (
got == "FAIL" and "UVM_ERROR" in text)
print(f"{name:<26} {got:<14} {'honest' if honest else 'GUESSED'}")input classify() says verdict
empty EMPTY_LOG honest
whitespace only EMPTY_LOG honest
truncated NO_FINISH honest
another tool's format NO_FINISH honest
CRLF line endings FAIL honest
embedded null bytes PASS GUESSED
last night's clean log PASS GUESSEDFive honest answers and two guesses, and the surprises are in both columns.
CRLF is handled — "TEST DONE" in text is a substring test and splitlines() strips \r, so a Windows-written log works by luck rather than design. That is worth a test to pin, because a parser using text.split("\n") instead would see "TEST DONE\r" and fail to match.
Another tool's log classified as NO_FINISH, which is honest by accident: the Questa log has no TEST DONE banner, so the parser says it did not finish. Section 6 shows how fragile that accident is.
The two guesses are the subject of Sections 6 and 7.
4. Three Inputs Never Reach the Parser
The remaining three fail before classify is called, which makes them a different test shape:
from pathlib import Path
def read_log(path):
return Path(path).read_text(encoding="utf-8")
Path("bin.log").write_bytes(b"\x00\x01\xff\xfe binary junk\n")
Path("adir.log").mkdir(exist_ok=True)
CASES = [
("non-UTF-8 bytes", "bin.log"),
("a directory", "adir.log"),
("a missing file", "nosuch.log"),
]
print(f"{'input':<22} {'raises':<24} errno")
for name, path in CASES:
try:
read_log(path)
print(f"{name:<22} {'nothing':<24} -")
except (UnicodeDecodeError, IsADirectoryError, FileNotFoundError) as exc:
errno = getattr(exc, "errno", None)
print(f"{name:<22} {type(exc).__name__:<24} {errno}")input raises errno
non-UTF-8 bytes UnicodeDecodeError None
a directory IsADirectoryError 21
a missing file FileNotFoundError 2Three different exceptions, and each one is more informative than any status string a parser could return — which is the point. An input that cannot be read is not a classification problem, and a try that swallowed these into return "FAIL" would turn three distinguishable infrastructure faults into one design verdict.
Chapter 33.6 is about testing that these raise rather than being absorbed. What belongs here is the decision about which of them the flow should tolerate:
POLICY = [
("a missing log", "the job may not have started",
"classify as NO_RUN, keep going"),
("a directory where a log should be", "the flow built the wrong path",
"raise — this is a bug in the flow"),
("non-UTF-8 bytes", "a corrupt or truncated write",
"classify as CORRUPT, keep the file"),
]
print(f"{'input':<36} {'what it means':<34} policy")
for what, means, policy in POLICY:
print(f"{what:<36} {means:<34} {policy}")
print()
print("inputs the flow should tolerate:", 2)
print("inputs that are a bug in the flow:", 1)
print("the distinction is:", "could the world have produced this?")input what it means policy
a missing log the job may not have started classify as NO_RUN, keep going
a directory where a log should be the flow built the wrong path raise — this is a bug in the flow
non-UTF-8 bytes a corrupt or truncated write classify as CORRUPT, keep the file
inputs the flow should tolerate: 2
inputs that are a bug in the flow: 1
the distinction is: could the world have produced this?"Could the world have produced this?" is the test. A missing log and a corrupt log happen on a bad night; a directory where a filename belongs happens because the flow built a path wrongly, and absorbing it into a status hides a bug in your own code.
5. Another Tool's Format
Section 3 found that a Questa log classifies as NO_FINISH, which looked honest. It is not robust:
def classify(text):
if not text.strip():
return "EMPTY_LOG"
if "UVM_FATAL" in text:
return "FAIL"
if "TEST DONE" not in text:
return "NO_FINISH"
if "UVM_ERROR" in text:
return "FAIL"
return "PASS"
QUESTA_NO_BANNER = ("# ** Error: (vsim-3601) drv.sv(88): bad handshake\n"
"# End time: 03:14:15 on Feb 18,2026\n")
QUESTA_WITH_BANNER = ("# ** Error: (vsim-3601) drv.sv(88): bad handshake\n"
"# TEST DONE\n"
"# End time: 03:14:15 on Feb 18,2026\n")
for label, text in (("no banner text", QUESTA_NO_BANNER),
("the string TEST DONE appears", QUESTA_WITH_BANNER)):
print(f"{label:<32} classify -> {classify(text)}")
print(f"{'':<32} errors in the file: 1")
print(f"{'':<32} UVM_ERROR lines : "
f"{sum(1 for l in text.splitlines() if l.startswith('UVM_ERROR'))}")no banner text classify -> NO_FINISH
errors in the file: 1
UVM_ERROR lines : 0
the string TEST DONE appears classify -> PASS
errors in the file: 1
UVM_ERROR lines : 0The second one reports PASS over a log containing an error. The NO_FINISH of Section 3 was luck: it depended on the other tool's log happening not to contain the substring TEST DONE, and one comment line changes the answer to a clean pass.
The parser is not wrong about the text. It is wrong about what text it was given, and nothing in a prefix-counting design can notice that — except the one field Chapter 20.6 insisted on:
KNOWN_PREFIXES = ("UVM_INFO", "UVM_WARNING", "UVM_ERROR", "UVM_FATAL",
"TEST DONE", "Simulation complete")
def classify_with_unparsed(text):
lines = [l for l in text.splitlines() if l.strip()]
if not lines:
return {"status": "EMPTY_LOG", "unparsed": 0, "lines": 0}
unparsed = sum(1 for l in lines if not l.startswith(KNOWN_PREFIXES))
errors = sum(1 for l in lines if l.startswith("UVM_ERROR"))
fatals = sum(1 for l in lines if l.startswith("UVM_FATAL"))
if unparsed == len(lines):
status = "UNRECOGNISED_FORMAT"
elif fatals or errors:
status = "FAIL"
elif "TEST DONE" not in text:
status = "NO_FINISH"
else:
status = "PASS"
return {"status": status, "unparsed": unparsed, "lines": len(lines)}
XRUN = ("UVM_INFO tb.sv(42) @ 0: reset released\n"
"UVM_ERROR drv.sv(88) @ 141900: bad handshake\n"
"TEST DONE\n")
QUESTA = ("# ** Error: (vsim-3601) drv.sv(88): bad handshake\n"
"# TEST DONE\n"
"# End time: 03:14:15 on Feb 18,2026\n")
for label, text in (("an xrun log", XRUN), ("a questa log", QUESTA)):
r = classify_with_unparsed(text)
print(f"{label:<14} {r['status']:<22} unparsed {r['unparsed']}/{r['lines']}")
print()
print("the discriminator:", "unparsed == every line means the format is not ours")
print("the status that says so:", "UNRECOGNISED_FORMAT")an xrun log FAIL unparsed 0/3
a questa log UNRECOGNISED_FORMAT unparsed 3/3
the discriminator: unparsed == every line means the format is not ours
the status that says so: UNRECOGNISED_FORMATunparsed == len(lines) is the whole detector. If not one line in the file matched a prefix the parser knows, the parser is reading something it was not written for, and saying so is strictly better than any verdict it could compute.
That is also the generalisation of Chapter 20.6's counter. There, counting unparsed lines turned a silent format change into a visible number; here, the ratio turns a wrong-file mistake into an explicit status.
6. Last Night's Log
The input nobody writes a test for parses perfectly:
import os
import time
from pathlib import Path
def classify(text):
if not text.strip():
return "EMPTY_LOG"
if "UVM_ERROR" in text or "UVM_FATAL" in text:
return "FAIL"
if "TEST DONE" not in text:
return "NO_FINISH"
return "PASS"
RUN_DIR = Path("runs/axi_burst")
RUN_DIR.mkdir(parents=True, exist_ok=True)
LOG = RUN_DIR / "sim.log"
# last night's run, which passed
LOG.write_text("UVM_INFO tb.sv(42) @ 0: reset released\n"
"UVM_INFO tb.sv(90) @ 158300: finishing\n"
"TEST DONE\n", encoding="utf-8")
old_mtime = time.time() - 26 * 3600
os.utime(LOG, (old_mtime, old_mtime))
print("tonight the job failed to launch, so nothing wrote sim.log")
print()
text = LOG.read_text(encoding="utf-8")
print("the parser reads :", LOG)
print("it classifies as :", classify(text))
print("the file is :", "26 hours old")
print("the verdict is about:", "last night's run")
print("anything in the text that says so:", "nothing")tonight the job failed to launch, so nothing wrote sim.log
the parser reads : runs/axi_burst/sim.log
it classifies as : PASS
the file is : 26 hours old
the verdict is about: last night's run
anything in the text that says so: nothingA confident PASS about a run that did not happen. The log is well-formed, complete and truthful; it is simply about yesterday, and a parser reading text has no way to know.
Two defences, and they work at different levels:
import os
import time
from pathlib import Path
RUN_DIR = Path("runs/axi_burst")
RUN_DIR.mkdir(parents=True, exist_ok=True)
LOG = RUN_DIR / "sim.log"
# last night's log, 26 hours old
LOG.write_text("UVM_INFO tb.sv(42) @ 0: reset released\nTEST DONE\n",
encoding="utf-8")
_old = time.time() - 26 * 3600
os.utime(LOG, (_old, _old))
def classify_with_age(path, started_at, max_age_s=3600):
"""Reject a log older than the run that supposedly produced it."""
if not Path(path).is_file():
return {"status": "NO_RUN", "reason": "no log"}
mtime = Path(path).stat().st_mtime
if mtime < started_at:
return {"status": "STALE_LOG",
"reason": f"log predates the run by "
f"{int(started_at - mtime)} s"}
return {"status": "PASS", "reason": ""}
run_started = time.time()
print("structural fix (23.7): a per-run directory")
print(" last night :", "runs/axi_burst_48210_20260217/")
print(" tonight :", "runs/axi_burst_48210_20260218/")
print(" a stale log is in a different directory and cannot be read by mistake")
print()
print("detector (for a flow that reuses directories):")
r = classify_with_age(LOG, run_started)
print(" 26-hour-old log, run started now ->", r["status"], "|", r["reason"])
Path(LOG).touch()
r = classify_with_age(LOG, run_started)
print(" freshly written log ->", r["status"])structural fix (23.7): a per-run directory
last night : runs/axi_burst_48210_20260217/
tonight : runs/axi_burst_48210_20260218/
a stale log is in a different directory and cannot be read by mistake
detector (for a flow that reuses directories):
26-hour-old log, run started now -> STALE_LOG | log predates the run by 93600 s
freshly written log -> PASSThe structural fix is Chapter 23.7's per-run directory, and it is the better answer: a log from another run is not merely detected but unreachable, because it is in a path the current run never constructs.
The detector matters for flows that cannot do that — a vendor tool that insists on a fixed output path, or a shared build area. Comparing the log's mtime against the run's start time is four lines, and the message it produces names the actual problem rather than a verdict.
Note the one thing the detector must not do: delete the stale log. Chapter 33.13 is about what to keep when things go wrong, and a stale log is evidence that the directory was reused — which is the bug worth fixing.
7. The Negative Suite
import io
import os
import time
import unittest
from pathlib import Path
KNOWN_PREFIXES = ("UVM_INFO", "UVM_WARNING", "UVM_ERROR", "UVM_FATAL",
"TEST DONE", "Simulation complete")
def classify(text):
lines = [l for l in text.splitlines() if l.strip()]
if not lines:
return "EMPTY_LOG"
unparsed = sum(1 for l in lines if not l.startswith(KNOWN_PREFIXES))
if unparsed == len(lines):
return "UNRECOGNISED_FORMAT"
if any(l.startswith("UVM_FATAL") for l in lines):
return "FAIL"
if "TEST DONE" not in text:
return "NO_FINISH"
if any(l.startswith("UVM_ERROR") for l in lines):
return "FAIL"
return "PASS"
NEGATIVE_CASES = [
("empty string", "", "EMPTY_LOG"),
("whitespace only", " \n\t\n", "EMPTY_LOG"),
("single newline", "\n", "EMPTY_LOG"),
("truncated mid-line", "UVM_INFO tb.sv(42) @ 0: reset rele", "NO_FINISH"),
("truncated after info", "UVM_INFO started\n", "NO_FINISH"),
("another tool, no banner",
"# ** Error: (vsim-3601) drv.sv(88): bad\n# End time: 03:14\n",
"UNRECOGNISED_FORMAT"),
("another tool, banner text present",
"# ** Error: (vsim-3601) drv.sv(88): bad\n# TEST DONE\n# End time: 03:14\n",
"UNRECOGNISED_FORMAT"),
("crlf line endings",
"UVM_INFO started\r\nUVM_ERROR drv.sv(88): bad\r\nTEST DONE\r\n", "FAIL"),
("fatal without a banner", "UVM_FATAL boom\n", "FAIL"),
]
class NegativeTests(unittest.TestCase):
def test_every_malformed_input(self):
self.assertTrue(NEGATIVE_CASES, "no negative cases defined")
for name, text, expected in NEGATIVE_CASES:
with self.subTest(case=name):
self.assertEqual(classify(text), expected)
def test_no_malformed_input_reports_pass(self):
for name, text, expected in NEGATIVE_CASES:
with self.subTest(case=name):
self.assertNotEqual(classify(text), "PASS")
def test_a_good_log_still_passes(self):
good = "UVM_INFO started\nTEST DONE\n"
self.assertEqual(classify(good), "PASS")
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"ok": res.wasSuccessful()}
print("negative suite:", run(NegativeTests))
print()
print("malformed inputs covered:", len(NEGATIVE_CASES))
print("inputs that report PASS :",
sum(1 for _, t, _ in NEGATIVE_CASES if classify(t) == "PASS"))
print("statuses used :",
sorted({classify(t) for _, t, _ in NEGATIVE_CASES}))negative suite: {'run': 3, 'failures': 0, 'ok': True}
malformed inputs covered: 9
inputs that report PASS : 0
statuses used : ['EMPTY_LOG', 'FAIL', 'NO_FINISH', 'UNRECOGNISED_FORMAT']Nine malformed inputs, none reporting PASS, and four distinct statuses used.
The second test method is the one worth copying verbatim into any flow. test_no_malformed_input_reports_pass asserts a property over the whole table rather than a value per row: whatever the classifier decides about a malformed input, it must not be the one answer that lets a bad night through. That single assertion would have caught both of Section 3's guesses without anybody predicting them.
And the third method exists because the first two can be satisfied by a classifier that returns FAIL for everything. Chapter 33.9 is about systematically finding that kind of hole; the cheap version is to keep one positive case in the negative suite.
8. A Log That Stops Mid-Line
Every bad input so far has been a complete file with wrong contents. The one a long regression actually produces is different: a file the simulator was still writing when it died, which ends partway through a line.
CASES = [
("newline-terminated", "UVM_INFO a\nUVM_ERROR drv.sv(88) bad\nTEST DONE\n"),
("truncated after a line", "UVM_INFO a\nUVM_ERROR drv.sv(88) bad\n"),
("truncated mid-line", "UVM_INFO a\nUVM_ERROR drv.sv(88) bad"),
("truncated mid-record", "UVM_INFO a\nUVM_ERROR drv.sv(8"),
]
print(f"{'input':<24} {'splitlines':>11} {'ends with \\n':>13} last line")
for label, text in CASES:
lines = text.splitlines()
print(f"{label:<24} {len(lines):>11} {str(text.endswith(chr(10))):>13} "
f"{lines[-1]!r}")
print()
a = "UVM_INFO a\nUVM_ERROR drv.sv(88) bad\n".splitlines()
b = "UVM_INFO a\nUVM_ERROR drv.sv(88) bad".splitlines()
print("complete file, line count :", len(a))
print("truncated file, line count :", len(b))
print("splitlines can tell them apart:", len(a) != len(b))input splitlines ends with \n last line
newline-terminated 3 True 'TEST DONE'
truncated after a line 2 True 'UVM_ERROR drv.sv(88) bad'
truncated mid-line 2 False 'UVM_ERROR drv.sv(88) bad'
truncated mid-record 2 False 'UVM_ERROR drv.sv(8'
complete file, line count : 2
truncated file, line count : 2
splitlines can tell them apart: Falsesplitlines() returns the same count for a complete file and a truncated
one. It does not require a trailing newline and it does not report whether
there was one, so every line-oriented parser in this module reads a
half-written log as though it were finished — and the only difference is a
final record that may be a fragment.
The fragment is the interesting part, because of how a well-written parser treats it:
import re
PAT = re.compile(r"^(UVM_ERROR|UVM_FATAL)\s+(\S+)\((\d+)\)")
FRAGMENTS = ["UVM_ERROR drv.sv(88) bad",
"UVM_ERROR drv.sv(8",
"UVM_ERROR drv.s",
"UVM_ERR"]
print(f"{'last line of the file':<28} {'matches':<9} what the parser does")
for f in FRAGMENTS:
m = PAT.match(f)
print(f"{f!r:<28} {str(bool(m)):<9} "
f"{'counted' if m else 'skipped by `continue`'}")
print()
print("fragments silently skipped:", 3)
print("errors the parser reports :", 1)
print("errors the log contains :", "1, plus one unknown")
print("the difference :", "invisible — no warning, no error")last line of the file matches what the parser does
'UVM_ERROR drv.sv(88) bad' True counted
'UVM_ERROR drv.sv(8' False skipped by `continue`
'UVM_ERROR drv.s' False skipped by `continue`
'UVM_ERR' False skipped by `continue`
fragments silently skipped: 3
errors the parser reports : 1
errors the log contains : 1, plus one unknown
the difference : invisible — no warning, no errorThree of four fragments are skipped by the if not m: continue that every
parser in this module uses, and that branch exists for a good reason — most
lines in a UVM log are not severity lines. So the parser's most defensible
line of code is the one that discards the evidence that the file is
incomplete.
The resulting failure mode is the worst kind. A simulation that died mid-write
after logging four errors reports three, the run is classified FAIL either
way, and the count in the regression report is quietly wrong — Chapter 30.7's
confidently-wrong value, produced by correct code on a damaged input.
The negative test is a trailing-newline check, and it belongs with the
TEST DONE check rather than in the parser's loop:
import io
import re
import unittest
PAT = re.compile(r"^(UVM_ERROR|UVM_FATAL)\s+(\S+)\((\d+)\)")
def count_loose(text):
"""startswith: a fragment still begins with the severity, so it COUNTS."""
return sum(1 for l in text.splitlines() if l.startswith("UVM_ERROR"))
def count_strict(text):
"""the full pattern: a fragment fails it, so it is SKIPPED."""
return sum(1 for l in text.splitlines() if PAT.match(l))
def read_log(text):
"""Returns (facts, warnings). A truncated file is a WARNING, not an error."""
warnings = []
if text and not text.endswith("\n"):
warnings.append("file does not end in a newline: "
"the simulator was still writing it")
if "TEST DONE" not in text:
warnings.append("no TEST DONE line: the run did not finish")
loose, strict = count_loose(text), count_strict(text)
if loose != strict:
warnings.append(f"{loose - strict} severity line(s) did not parse: "
"the last record may be incomplete")
return {"errors": strict, "unparsed": loose - strict,
"complete": not warnings}, warnings
COMPLETE = ("UVM_INFO a\nUVM_ERROR drv.sv(88) bad\n"
"UVM_ERROR mon.sv(12) bad\nTEST DONE\n")
TRUNCATED = ("UVM_INFO a\nUVM_ERROR drv.sv(88) bad\n"
"UVM_ERROR mon.sv(1")
class TruncatedLogIsDetected(unittest.TestCase):
def test_complete_log_has_no_warnings(self):
facts, warnings = read_log(COMPLETE)
self.assertEqual(warnings, [])
self.assertTrue(facts["complete"])
self.assertEqual(facts["errors"], 2)
def test_truncated_log_raises_three_warnings(self):
facts, warnings = read_log(TRUNCATED)
self.assertEqual(len(warnings), 3)
self.assertFalse(facts["complete"])
def test_the_two_counters_disagree_on_a_fragment(self):
self.assertEqual(count_loose(TRUNCATED), 2)
self.assertEqual(count_strict(TRUNCATED), 1)
def test_they_agree_on_a_complete_log(self):
self.assertEqual(count_loose(COMPLETE), count_strict(COMPLETE))
def test_an_empty_file_is_not_called_truncated(self):
facts, warnings = read_log("")
self.assertNotIn("newline", " ".join(warnings))
def run(case):
suite = unittest.TestLoader().loadTestsFromTestCase(case)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
return {"run": res.testsRun, "failures": len(res.failures),
"ok": res.wasSuccessful()}
print("truncation suite:", run(TruncatedLogIsDetected))
print()
facts, warnings = read_log(TRUNCATED)
for w in warnings:
print(" warning:", w)
print()
print("loose count (startswith):", count_loose(TRUNCATED))
print("strict count (full regex):", count_strict(TRUNCATED))
print("parsed errors reported :", facts["errors"])
print("unparsed severity lines :", facts["unparsed"])
print("marked incomplete :", not facts["complete"])truncation suite: {'run': 5, 'failures': 0, 'ok': True}
warning: file does not end in a newline: the simulator was still writing it
warning: no TEST DONE line: the run did not finish
warning: 1 severity line(s) did not parse: the last record may be incomplete
loose count (startswith): 2
strict count (full regex): 1
parsed errors reported : 1
unparsed severity lines : 1
marked incomplete : TrueFive tests, three warnings, and the two counters disagree by exactly one.
The third warning is the one worth having: a startswith counter sees two
severity lines because the fragment still begins with UVM_ERROR, while the
full pattern parses only one — and the difference between a loose and a
strict count is a truncation detector that needs no newline check at all.
That is a second, independent signal, which matters because the first one is weak. A trailing newline is a convention, not a guarantee: some tools do not write one even on a clean exit, so a newline check alone produces false positives. A line that looks like a severity line and fails to parse is much harder to explain innocently.
The fourth test is what keeps the detector honest. On a complete log the two counters must agree — otherwise the parser has a pattern bug and every run would report phantom truncation, which is Chapter 33.9's point that a check needs a case where it stays quiet as much as one where it fires.
The honest output is the partial count plus an explicit flag — a truncated log is not useless, it is incomplete — and never a count presented as final.
The last test is the one that needs the most care. An empty file does not
end in a newline either, so a naive truncation check calls a zero-byte log
"truncated" — and a zero-byte log usually means the tool never started, which
is a completely different diagnosis. Guarding with if text and ... separates
them, and Chapter 33.6's distinction between the tool failing to launch and the
tool failing is exactly the distinction being preserved.
9. Predict, Run, Observe
def classify(text):
if not text.strip():
return "EMPTY_LOG"
if "TEST DONE" not in text:
return "NO_FINISH"
if "UVM_ERROR" in text:
return "FAIL"
return "PASS"
# A
print("A:", classify(""), classify(" \n"), classify("\n"))
# B
print("B:", classify("# ** Error: vsim-3601 bad\n# TEST DONE\n"))
# C
print("C:", "TEST DONE" in "TEST DONE\r\n",
"TEST DONE" == "TEST DONE\r\n".split("\n")[0])
# D
print("D:", classify("UVM_INFO ok\nTEST DONE\n"))The verified output:
A: EMPTY_LOG EMPTY_LOG EMPTY_LOG
B: PASS
C: True False
D: PASSA — all three empty shapes classify the same, because the test is not text.strip().
B — another tool's log with an error reports PASS. Section 6's defect in one line.
C — why CRLF works by luck: the substring test succeeds and a split-based comparison fails. A parser written the second way breaks on a Windows-written log.
D — the positive case, which the negative suite must keep so that returning FAIL for everything cannot pass.
10. Exercises
Exercise 1 — eight inputs. List the malformed inputs your own flow can receive, and how each arises.
Exercise 2 — from the world. Explain why none of them appears in a code review.
Exercise 3 — the two assertions. Confirm your classifier handles empty and truncated, and run the assertions.
Exercise 4 — whitespace only. Check whether your empty test uses not text or not text.strip().
Exercise 5 — honest or guessed. Feed your parser six malformed inputs and sort the answers.
Exercise 6 — CRLF by luck. Find out whether your parser uses a substring test or a split, and which logs that affects.
Exercise 7 — pin the luck. Add a CRLF fixture so the behaviour is asserted rather than accidental.
Exercise 8 — three exceptions. Reproduce Section 4's table on your own platform.
Exercise 9 — could the world produce this? Apply that test to five inputs and decide tolerate-or-raise for each.
Exercise 10 — do not absorb a bug. Find a try in your flow that turns a path error into a status.
Exercise 11 — another tool's log. Feed your parser a log from a different simulator and record the verdict.
Exercise 12 — one comment line. Construct the case where the other tool's log reports PASS.
Exercise 13 — the unparsed ratio. Add unparsed == len(lines) and the UNRECOGNISED_FORMAT status.
Exercise 14 — a status about the parser. Explain why this is the only status in the set that is not about the design.
Exercise 15 — last night's log. Reuse a run directory deliberately and record what your flow reports.
Exercise 16 — per-run directories. Confirm yours are unique per test and seed, and say what that makes unreachable.
Exercise 17 — the age check. Implement the mtime comparison for one fixed-output-path tool.
Exercise 18 — do not delete it. Say why a stale log is evidence worth keeping.
Exercise 19 — the table property. Add test_no_malformed_input_reports_pass to your own suite.
Exercise 20 — unpredicted holes. Explain how that one assertion catches guesses nobody anticipated.
Exercise 21 — always-FAIL. Write a classifier that returns FAIL for everything and see which of your tests still pass.
Exercise 22 — keep a positive. Add the positive case to your negative suite and say what it rules out.
Exercise 23 — four statuses. List the statuses your negative inputs produce and check none is PASS.
Exercise 24 — the night it matters. Describe the most recent night your flow met one of these inputs, and what it reported.
Exercise 25 — the same count. Confirm that splitlines() returns the same length for a complete and a truncated file.
Exercise 26 — the fragment. Truncate a real log mid-record and check whether your parser's pattern matches the last line.
Exercise 27 — the defensible line. Explain why if not m: continue is both correct and the reason the truncation is invisible.
Exercise 28 — the wrong count. Describe the report a mid-write death produces, and why the status is still FAIL.
Exercise 29 — check the newline. Add a trailing-newline check outside the parsing loop.
Exercise 30 — warning, not error. Say why truncation is a warning and what the report should print.
Exercise 31 — the empty file. Confirm your check does not call a zero-byte log truncated, and say what a zero-byte log means instead.
11. Summary
- Eight malformed inputs, none produced by your code — so none appears in a review and all appear on a bad night.
- Both annotation assertions are correct, which is unusual in this book; the chapter extends rather than corrects.
- Five of seven text inputs got honest answers and two were guesses. The question the purpose asks is whether the parser says so.
- CRLF works by luck — a substring test succeeds where a split-based comparison fails — so it needs a fixture to pin it.
- Three inputs never reach the parser:
UnicodeDecodeError,IsADirectoryErrorandFileNotFoundError, each more informative than any status. - "Could the world have produced this?" decides tolerate-or-raise. A missing log is a bad night; a directory where a filename belongs is a bug in the flow.
- Another tool's log reported
NO_FINISHby accident andPASSafter one comment line changed, because the parser was wrong about what file it was given rather than about the text. unparsed == len(lines)is the whole detector, andUNRECOGNISED_FORMATis the only status that is about the parser rather than the design.- Last night's log parses perfectly and reports a verdict about a run that did not happen. Per-run directories make it unreachable; an mtime check detects it where they cannot.
- Do not delete a stale log — it is evidence that a directory was reused.
- Assert a property over the whole table: no malformed input may report PASS. That one line catches guesses nobody predicted.
- Keep one positive case in the negative suite, or a classifier returning FAIL for everything passes it.
splitlines()returns the same count for a complete file and a truncated one — it neither requires nor reports a trailing newline.- A mid-record fragment fails the severity pattern and is skipped by the
if not m: continueevery parser needs, so the most defensible line in the loop is the one that discards the evidence of truncation. - The result is a confidently wrong count produced by correct code on a damaged input.
- Check the trailing newline outside the loop, alongside the
TEST DONEcheck, and report truncation as a warning with the partial count — never a count presented as final. - An empty file does not end in a newline either, so guard the check: a zero-byte log means the tool never started, which is a different diagnosis.
The next chapter tests the paths that only run when something has already gone wrong.
Where this fits
Part of the Python curriculum.