What a Parser Owes Its Caller

Part II · Module 20 · Chapter 20.6 · What a Parser Owes Its Caller

1. Decide Before You Pattern

A parser's contract is the set of promises the calling code can rely on without reading the parser. There are four worth settling in advance, and all four are cheaper to decide now than to change after six chapters of patterns depend on them:

2. How Many Passes

Chapter 20.3 Section 6 established the fact this turns on: a file object is an iterator, consumed once. So the decision is not really "stream or load" — it is how many passes the contract needs, and streaming is what you get when the answer is one.

For a simulation-log parser the honest answer is one pass, and it is worth knowing why rather than assuming: every fact this module extracts — the severity counts, the first failure, the test name, the end-of-simulation banner — can be decided by looking at each line once and keeping a few variables. Chapter 15.7 Section 5 called that carrying a bounded summary.

The one thing a single pass makes awkward is context before a line you have not read yet, which Chapter 20.10 needs for the lines around the first failure. It solves it by keeping the last few lines deliberately, in a small buffer — not by loading the file.

3. What Comes Back

The canonical annotation answers this in a docstring:

def parse_sim_log(path) -> dict:
    """errors, fatals, first_failure, finished, status"""

Five keys. The important promise is not the list — it is a word that does not appear in it: always. Every one of those keys is in the returned dict on every log, whatever the log contained.

Here is the alternative, which is what a parser grows into if nobody decides:

CLEAN = ["UVM_INFO @ 0: reporter [RNTST] Running test axi_reset_test...",
         "UVM_INFO scb.sv(120) @ 160000: 96 transactions compared",
         "TEST DONE"]

DIRTY = ["UVM_INFO @ 0: reporter [RNTST] Running test axi_smoke_test...",
         "UVM_ERROR drv.sv(88) @ 141900: bad handshake",
         "TEST DONE"]


def sparse(lines):
    """Adds a key when it finds something."""
    out = {}
    for line in lines:
        if line.startswith("UVM_ERROR"):
            out["errors"] = out.get("errors", 0) + 1
            if "first_failure" not in out:
                out["first_failure"] = line
    return out


print("clean:", sparse(CLEAN))
print("dirty:", sparse(DIRTY))

clean: {}
dirty: {'errors': 1, 'first_failure': 'UVM_ERROR drv.sv(88) @ 141900: bad handshake'}

Both results are true. The clean log has no errors and no first failure, so a dict with no keys is an accurate description of it.

It is also unusable, and the way it fails is this module's recurring shape:

CLEAN = ["UVM_INFO @ 0: reporter [RNTST] Running test axi_reset_test...", "TEST DONE"]
DIRTY = ["UVM_ERROR drv.sv(88) @ 141900: bad handshake", "TEST DONE"]


def sparse(lines):
    out = {}
    for line in lines:
        if line.startswith("UVM_ERROR"):
            out["errors"] = out.get("errors", 0) + 1
    return out


for name, lines in [("smoke", DIRTY), ("reset", CLEAN)]:
    record = sparse(lines)
    print(f"{name:<6} errors={record['errors']}")

smoke  errors=1
Traceback (most recent call last):
  File "report.py", line 15, in <module>
    print(f"{name:<6} errors={record['errors']}")
                              ~~~~~~^^^^^^^^^^
KeyError: 'errors'

The report works for the failing log and crashes on the passing one. That is the worst possible ordering: every log you test against during development has errors in it, because that is what you were debugging, and the first clean night is when the nightly job dies.

The fix is one line at the top of the parser:

CLEAN = ["UVM_INFO @ 0: reporter [RNTST] Running test axi_reset_test...",
         "UVM_INFO scb.sv(120) @ 160000: 96 transactions compared", "TEST DONE"]

DIRTY = ["UVM_INFO @ 0: reporter [RNTST] Running test axi_smoke_test...",
         "UVM_ERROR drv.sv(88) @ 141900: bad handshake", "TEST DONE"]


def parse(lines):
    """Every key is present on every log."""
    record = {"errors": 0, "fatals": 0, "first_failure": None, "finished": False}
    for line in lines:
        if line.startswith("UVM_ERROR"):
            record["errors"] += 1
            if record["first_failure"] is None:
                record["first_failure"] = line.rstrip()
        elif line.startswith("UVM_FATAL"):
            record["fatals"] += 1
        if line.startswith("TEST DONE"):
            record["finished"] = True
    return record


for name, lines in [("smoke", DIRTY), ("reset", CLEAN)]:
    record = parse(lines)
    print(f"{name:<6} errors={record['errors']} fatals={record['fatals']} "
          f"finished={record['finished']} first={record['first_failure'] is not None}")

smoke  errors=1 fatals=0 finished=True first=True
reset  errors=0 fatals=0 finished=True first=False

Two rows with the same columns, which is the only thing a report, a CSV writer or a comparison between two nights can actually consume.

Note that first_failure is None and not "" on the clean log — Chapter 19.10 Section 6's distinction, now a contract term: None means there was no first failure; an empty string would mean there was one and it was blank.

4. What It Does With a Line It Does Not Recognise

This is the decision almost nobody makes, and the one that costs the most.

Every parser skips lines — a simulation log is mostly text no checker cares about. The question is whether skipped and not understood are the same thing. Three answers:

• Ignore silently. Anything not matched is assumed irrelevant.
• Count them. Anything not matched increments unparsed, which comes back in the record.
• Fail. An unrecognised line raises.

"Fail" is wrong for a simulation log — there are thousands of legitimately irrelevant lines. The real choice is between the first two, and here is what it is worth. A tool release starts prefixing every line with a thread id:

OLD = ["UVM_INFO @ 0: reporter [RNTST] starting",
       "UVM_ERROR drv.sv(88) @ 141900: bad handshake",
       "UVM_ERROR drv.sv(104) @ 180400: timeout",
       "TEST DONE"]

NEW = ["[0] UVM_INFO @ 0: reporter [RNTST] starting",
       "[0] UVM_ERROR drv.sv(88) @ 141900: bad handshake",
       "[0] UVM_ERROR drv.sv(104) @ 180400: timeout",
       "[0] TEST DONE"]


def parse(lines):
    record = {"errors": 0, "finished": False, "unparsed": 0}
    for line in lines:
        if line.startswith("UVM_ERROR"):
            record["errors"] += 1
        elif line.startswith("UVM_"):
            pass
        elif line.startswith("TEST DONE"):
            record["finished"] = True
        else:
            record["unparsed"] += 1
    return record


header = f"{'log':<12} {'errors':>6} {'finished':>9} {'unparsed':>9}   {'errors only':<12}{'+finished':<11}+unparsed"
print(header)
for name, lines in [("old format", OLD), ("new format", NEW)]:
    r = parse(lines)
    a = "PASS" if r["errors"] == 0 else "FAIL"
    b = "PASS" if r["errors"] == 0 and r["finished"] else "FAIL"
    c = "SUSPECT" if r["unparsed"] else b
    print(f"{name:<12} {r['errors']:>6} {str(r['finished']):>9} {r['unparsed']:>9}   {a:<12}{b:<11}{c}")

log          errors  finished  unparsed   errors only +finished  +unparsed
old format        2      True         0   FAIL        FAIL       FAIL
new format        0     False         4   PASS        FAIL       SUSPECT

Read the new format row across. The log contains two errors and the parser found none, because none of its tests match any more.

errors only reports PASS. A failing run, reported as clean, with total confidence. This is the catastrophic column.

+finished reports FAIL — the right answer for the wrong reason. It says the test failed; what actually happened is that the parser failed. Somebody will spend an hour looking at RTL.

+unparsed reports SUSPECT, because four lines went unrecognised in a four-line log. That is the only column that points at the parser, and it costs one integer in the record and one else in the loop.

5. What Happens When the File Is Not There

The fourth decision, which the canonical annotation leaves to you. Two defensible answers:

Let it raise. open() raises FileNotFoundError, the caller sees exactly which path was missing, and the parser's contract says nothing about missing files because it never returns for one. This is right when a missing log means the caller made a mistake — Chapter 14.4's argument for letting an exception escape.

Return a record saying so. The parser always returns a dict, with a status of NO_LOG and the counts at zero. This is right when the caller is sweeping a thousand run directories and a missing log is a result, not an accident — a crash there loses the 999 other answers.

The two are not compatible and a parser must pick one:

def parse_strict(path):
    """Raises FileNotFoundError if the log is not there."""
    record = {"errors": 0, "status": "UNKNOWN"}
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            if line.startswith("UVM_ERROR"):
                record["errors"] += 1
    record["status"] = "FAIL" if record["errors"] else "PASS"
    return record


def parse_tolerant(path):
    """Always returns a record; a missing log is status NO_LOG."""
    try:
        return parse_strict(path)
    except FileNotFoundError:
        return {"errors": 0, "status": "NO_LOG"}


print("tolerant, missing:", parse_tolerant("nowhere.log"))

try:
    parse_strict("nowhere.log")
except FileNotFoundError as exc:
    print("strict, missing  :", type(exc).__name__, exc.filename)

tolerant, missing: {'errors': 0, 'status': 'NO_LOG'}
strict, missing  : FileNotFoundError nowhere.log

Note the shape: the tolerant version is a three-line wrapper around the strict one. Write the parser strict, and let the sweeping code decide whether it can afford to continue. That keeps the policy where Chapter 20.1 Section 4 said it belongs and means there is only one parser.

And note status: NO_LOG rather than status: PASS with zero errors. A log that does not exist is not a log with no errors in it. That is the same rule Chapter 19.12 Section 8 applied to an unreadable report, and Chapter 20.11 makes it one of four outcomes.

6. Writing the Contract Down

All four decisions, in a signature and a docstring:

def parse_sim_log(path) -> dict:
    """Parse one UVM simulation log in a single pass.

    Returns a dict with these keys, always, whatever the log contained:
        errors        int   -- count of UVM_ERROR messages
        fatals        int   -- count of UVM_FATAL messages
        first_failure str or None -- the first failing line, stripped
        finished      bool  -- an end-of-simulation banner was seen
        unparsed      int   -- lines matching nothing the parser knows
        status        str   -- PASS, FAIL, or NO_FINISH

    Raises FileNotFoundError if the log does not exist; a caller sweeping
    many run directories should catch that and record NO_LOG itself.
    """
    record = {"errors": 0, "fatals": 0, "first_failure": None,
              "finished": False, "unparsed": 0, "status": "UNKNOWN"}
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            if line.startswith("UVM_ERROR"):
                record["errors"] += 1
                if record["first_failure"] is None:
                    record["first_failure"] = line.rstrip()
            elif line.startswith("UVM_FATAL"):
                record["fatals"] += 1
                if record["first_failure"] is None:
                    record["first_failure"] = line.rstrip()
            elif line.startswith("UVM_"):
                pass
            elif line.startswith("TEST DONE"):
                record["finished"] = True
            else:
                record["unparsed"] += 1

    if not record["finished"]:
        record["status"] = "NO_FINISH"
    elif record["errors"] or record["fatals"]:
        record["status"] = "FAIL"
    else:
        record["status"] = "PASS"
    return record


print(sorted(parse_sim_log.__doc__.split("\n")[0:1]))
print("keys promised:", 6)

['Parse one UVM simulation log in a single pass.']
keys promised: 6

Four things that docstring does which a comment would not.

-> dict is a type hint — Chapter 11.12 — and it is the weakest of the four promises, because dict says nothing about the keys. The docstring carries the real contract.

Every key is listed with its type and its meaning, including that first_failure may be None. A caller can write the report without reading the loop.

The exception is named. A contract that only describes success is half a contract.

status has an enumerated set of values — PASS, FAIL, NO_FINISH — not "a string". A caller can write an exhaustive comparison, and Chapter 20.11 adds the fourth value deliberately rather than by accident.

Notice that the body still contains no regex. Every test is startswith, which Chapter 20.2 proved is not good enough — and the contract does not change when the patterns do. That is the point of settling it first: Chapter 20.7 replaces the severity tests entirely, and not one line of this docstring needs editing.

7. Predict, Run, Observe

def parse(lines):
    record = {"errors": 0, "first_failure": None, "finished": False, "unparsed": 0}
    for line in lines:
        if line.startswith("UVM_ERROR"):
            record["errors"] += 1
            if record["first_failure"] is None:
                record["first_failure"] = line
        elif line.startswith("UVM_"):
            pass
        elif line.startswith("TEST DONE"):
            record["finished"] = True
        else:
            record["unparsed"] += 1
    return record


CLEAN = ["UVM_INFO a", "TEST DONE"]
TWO = ["UVM_ERROR first", "UVM_INFO a", "UVM_ERROR second", "TEST DONE"]
ODD = ["xcelium> run", "UVM_ERROR first", "TEST DONE"]

# A
print("A:", parse(CLEAN))

# B
print("B:", parse(TWO)["first_failure"])

# C
print("C:", parse(TWO)["errors"], parse(CLEAN)["errors"])

# D
print("D:", parse(ODD)["unparsed"])

# E
print("E:", sorted(parse(CLEAN)) == sorted(parse(TWO)))

# F
print("F:", parse([])["finished"], parse([])["errors"])

The verified output:

A: {'errors': 0, 'first_failure': None, 'finished': True, 'unparsed': 0}
B: UVM_ERROR first
C: 2 0
D: 1
E: True
F: False 0

A — every key present on a log with nothing in it. The contract, working.

B — the first, not the last. The is None guard is what makes it first.

C — two and zero, both from the same function and both usable.

D — the xcelium> run line matched nothing and was counted rather than ignored.

E — the two records have the same keys. That is the property a report depends on, and it is checkable in one line.

F — an empty log gives finished=False and zero errors, which by Section 6's rules is NO_FINISH rather than PASS. An empty log is not a clean run.

8. Exercises

Exercise 1 — count the passes. List every fact your parser must extract and say, for each, whether it needs anything the single pass has already gone past.

Exercise 2 — the sparse parser. Write a parser that adds keys only when it finds things. Run it on a clean log and on a failing one and print both dicts.

Exercise 3 — the clean-run crash. Write a report loop over both records from Exercise 2. Record which log crashes it and explain why that is the worst one to crash on.

Exercise 4 — always present. Fix the parser with an initialised record. Confirm both logs produce the same keys with one sorted() comparison.

Exercise 5 — None, not empty. Say what first_failure="" would mean that first_failure=None does not, and which your report should test for.

Exercise 6 — the format change. Take a log and prefix every line with something. Run your parser on both versions and record the error count for each.

Exercise 7 — three policies. Build Section 4's three-column table for your own parser. Confirm the errors-only column reports a pass on the changed format.

Exercise 8 — read the unparsed. Turn the unparsed count into a verdict. Choose a threshold and say why a handful of unrecognised lines is not a problem and most of them is.

Exercise 9 — strict or tolerant. Write both versions for a missing file. Say which one a single-log checker should ship and which one a thousand-directory sweep should use.

Exercise 10 — NO_LOG is not PASS. Say in two sentences what goes wrong if a missing log returns zero errors and a status of PASS.

Exercise 11 — write it down. Write the docstring for your parser listing every key, its type, its meaning, and the exception it raises. Then hand it to someone and ask them to write the report without reading the body.

9. Summary

• A parser's contract is what the caller can rely on without reading the parser, and it is cheaper to decide before the patterns than after.
• Four decisions: how many passes, what comes back, what a missing file does, and what an unrecognised line means.
• A file object is consumed once, so "stream or load" is really "how many passes does the contract need" — and for this parser it is one.
• One pass buys bounded memory and costs the ability to look backwards, which Chapter 20.10 solves with a small deliberate buffer rather than by loading the file.
• Every key, every time. A parser that adds keys only when it finds something returns {} for a passing log.
• That fails on the clean runs, which is the worst ordering: everything you develop against has errors in it.
• None means absent; "" would mean present and blank. The distinction is a contract term, not a detail.
• Skipped and not understood are different things. The third policy counts unrecognised lines instead of ignoring them.
• On a tool format change, an errors-only verdict reports PASS, an errors-plus-finished verdict reports FAIL for the wrong reason, and only an unparsed count points at the parser.
• An unparsed count nobody reads is the same as not having one — turn it into a verdict.
• A missing file either raises or returns a record, and the tolerant version is best written as a three-line wrapper around the strict one.
• A missing log is NO_LOG, never PASS with zero errors.
• -> dict is the weakest promise in the contract; the docstring listing every key, its type and the exception is the real one.
• The contract does not change when the patterns do — Chapter 20.7 replaces every test in the body and edits none of the docstring.

The next chapter replaces those startswith tests with a pattern that can tell a UVM message from a line that merely counts them — and checks its own answer against the tool's.