Stateful Parsing: Report Sections That Span Lines

Part II · Module 20 · Chapter 20.15 · Stateful Parsing: Report Sections That Span Lines

1. The Report a Pattern Cannot Read

Timing Report: setup, corner ss_125c
========================================
Startpoint: u_core/u_alu/acc_reg[7] (rising edge-triggered flip-flop clocked by clk)
Endpoint: u_core/u_fifo/wptr_reg[3] (rising edge-triggered flip-flop clocked by clk)
  Path Group: clk
  Path Type: max
  data required time          2.431
  data arrival time           2.573
  slack (VIOLATED)           -0.142

Startpoint: u_core/u_ctl/state_reg[1] (rising edge-triggered flip-flop clocked by clk)
Endpoint: u_core/u_ctl/done_reg (rising edge-triggered flip-flop clocked by clk)
  Path Group: clk
  Path Type: max
  slack (MET)                 0.031

Startpoint: u_core/u_dp/mul_reg[2] (rising edge-triggered flip-flop clocked by clk)
Endpoint: u_core/u_dp/acc_reg[0] (rising edge-triggered flip-flop clocked by clk)
  slack (VIOLATED)           -1

import re

SLACK = re.compile(r"slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+(?:\.\d+)?)")
START = re.compile(r"^Startpoint: (\S+)", re.MULTILINE)

with open("timing.rpt", encoding="utf-8") as handle:
    text = handle.read()

print("slacks     :", SLACK.findall(text))
print("startpoints:", START.findall(text))

slacks     : ['-0.142', '0.031', '-1']
startpoints: ['u_core/u_alu/acc_reg[7]', 'u_core/u_ctl/state_reg[1]', 'u_core/u_dp/mul_reg[2]']

Two lists of three. Everything needed for a report is there, and nothing in the data connects them — the association is "the third slack belongs to the third startpoint", which is true here and is an assumption about the file's regularity rather than a fact the file states.

Chapter 19.19 Section 5 named the reason: a regex is one expression evaluated at a position, with no variables, so it cannot remember which block it is inside. Pairing by index works until a path omits a field, a header appears mid-report, or a path reports both setup and hold slack — and then the lists are different lengths and every pair after the irregularity is wrong.

2. One Record, Several Lines

Two lines carry the structure. Startpoint: means a new record begins here, and the slack line means this record is complete. Everything between them is a field.

That is the canonical annotation:

in_path = False
for line in f:
    if line.startswith("Startpoint:"):
        in_path, cur = True, {}
    elif in_path and "slack" in line:
        cur["slack"] = parse_slack(line)
        paths.append(cur)
        in_path = False

3. The Parser

Timing Report: setup, corner ss_125c
========================================
Startpoint: u_core/u_alu/acc_reg[7] (rising edge-triggered flip-flop clocked by clk)
Endpoint: u_core/u_fifo/wptr_reg[3] (rising edge-triggered flip-flop clocked by clk)
  Path Group: clk
  Path Type: max
  data required time          2.431
  data arrival time           2.573
  slack (VIOLATED)           -0.142

Startpoint: u_core/u_ctl/state_reg[1] (rising edge-triggered flip-flop clocked by clk)
Endpoint: u_core/u_ctl/done_reg (rising edge-triggered flip-flop clocked by clk)
  Path Group: clk
  Path Type: max
  slack (MET)                 0.031

Startpoint: u_core/u_dp/mul_reg[2] (rising edge-triggered flip-flop clocked by clk)
Endpoint: u_core/u_dp/acc_reg[0] (rising edge-triggered flip-flop clocked by clk)
  slack (VIOLATED)           -1

import re

START = re.compile(r"^Startpoint: (\S+)")
END = re.compile(r"^Endpoint: (\S+)")
GROUP = re.compile(r"^\s+Path Group: (\S+)")
SLACK = re.compile(r"slack\s+\((?P<state>VIOLATED|MET)\)\s+(?P<value>-?\d+(?:\.\d+)?)")


def paths(path):
    """One record per timing path, yielded as each completes."""
    current = None
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            m = START.match(line)
            if m is not None:
                current = {"start": m.group(1), "end": None,
                           "group": None, "slack": None, "state": None}
                continue
            if current is None:
                continue
            m = END.match(line)
            if m is not None:
                current["end"] = m.group(1)
                continue
            m = GROUP.match(line)
            if m is not None:
                current["group"] = m.group(1)
                continue
            m = SLACK.search(line)
            if m is not None:
                current["slack"] = float(m["value"])
                current["state"] = m["state"]
                yield current
                current = None


print(f"{'slack':>8}  {'state':<9} {'group':<5} path")
for record in paths("timing.rpt"):
    print(f"{record['slack']:>8.3f}  {record['state']:<9} {str(record['group']):<5} "
          f"{record['start']} -> {record['end']}")

   slack  state     group path
  -0.142  VIOLATED  clk   u_core/u_alu/acc_reg[7] -> u_core/u_fifo/wptr_reg[3]
   0.031  MET       clk   u_core/u_ctl/state_reg[1] -> u_core/u_ctl/done_reg
  -1.000  VIOLATED  None  u_core/u_dp/mul_reg[2] -> u_core/u_dp/acc_reg[0]

Three complete records, each carrying the fields that belong together — which the two findall lists in Section 1 could not produce at any price.

Four things in that loop are load-bearing.

current = None means "between records". Section 4 is about why that is better than the canonical boolean.

if current is None: continue right after the start check. Everything before the first Startpoint: — the report header, the row of equals signs — is skipped by this one line, and so is anything after a record closes.

The order of the tests. START is checked first and unconditionally, so a Startpoint: arriving while a record is open opens a new one rather than being swallowed. That is what makes the parser recover from a malformed block instead of merging it into the previous one.

yield rather than append. Chapter 20.13's form, so a report over a thousand timing reports holds one record at a time.

4. None-or-Dict Beats a Boolean

The canonical annotation keeps two things: in_path and cur. That is one more than necessary:

# Two names for one fact
in_path = False
cur = None

in_path, cur = True, {"start": "a"}
print("open   :", in_path, cur)

# nothing stops these drifting apart
in_path = False
print("drifted:", in_path, cur)
print("the dict is still here, and the flag says we are not in a path")

open   : True {'start': 'a'}
drifted: False {'start': 'a'}
the dict is still here, and the flag says we are not in a path

in_path is derivable from cur: if there is a partial record, you are inside one. Keeping both means there are states where they disagree, and nothing prevents it.

current = None or a dict carries the same information with no redundancy — current is not None is in_path, and it cannot get out of step with itself. The dict is both the flag and the data.

This is a genuine improvement on the canonical sketch rather than a stylistic preference, and the sketch is still worth reading as written: in_path names the idea clearly, which is what a sketch is for.

5. The Bug: Forgetting to Reset

Startpoint: path_one_start
Endpoint: path_one_end
  slack (VIOLATED)           -0.142
some unrelated report text
  slack (MET)                 0.031

import re

START = re.compile(r"^Startpoint: (\S+)")
END = re.compile(r"^Endpoint: (\S+)")
SLACK = re.compile(r"slack\s+\((?P<state>VIOLATED|MET)\)\s+(?P<value>-?\d+(?:\.\d+)?)")


def parse(path, reset):
    current = None
    out = []
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            m = START.match(line)
            if m is not None:
                current = {"start": m.group(1), "end": None, "slack": None}
                continue
            if current is None:
                continue
            m = END.match(line)
            if m is not None:
                current["end"] = m.group(1)
                continue
            m = SLACK.search(line)
            if m is not None:
                current["slack"] = float(m["value"])
                out.append(current)
                if reset:
                    current = None
    return out


for reset in (True, False):
    print(f"reset={reset}")
    for record in parse("timing.rpt", reset):
        print("   ", record)

reset=True
    {'start': 'path_one_start', 'end': 'path_one_end', 'slack': -0.142}
reset=False
    {'start': 'path_one_start', 'end': 'path_one_end', 'slack': 0.031}
    {'start': 'path_one_start', 'end': 'path_one_end', 'slack': 0.031}

Without the reset, two records, both reporting the second slack value against the first path's endpoints — and both identical, because out holds two references to the same dict that was mutated again.

Every number in that output is real and every association is wrong. A timing report built from it says path path_one_start → path_one_end has a met slack of 0.031, when in fact it has a violated slack of −0.142. Nothing raised; the record shape is perfect.

Two separate mistakes combine there and both are worth naming. The missing reset lets a later slack line attach to a stale record. The shared dict means appending without copying leaves every stored record pointing at the same object, so a later mutation rewrites history — Chapter 5.8's aliasing, inside a parser.

The reset fixes both at once, because a fresh Startpoint: builds a new dict.

6. Fields That Are Sometimes Absent

The third path in Section 3's report has no Path Group: line, and the output showed group: None rather than omitting the key or borrowing the previous path's value.

That is Chapter 20.6's always-present rule, and in a stateful parser it has teeth: the dict is built complete, with every key, at the moment the record opens. If the keys were added as the lines arrived, a record missing a field would be missing a key, and the report would hit Chapter 20.6 Section 3's KeyError — on the irregular records, which are the interesting ones.

def opened_complete(start):
    return {"start": start, "end": None, "group": None, "slack": None, "state": None}


def opened_empty(start):
    return {"start": start}


complete = opened_complete("a")
empty = opened_empty("a")

print("complete:", sorted(complete))
print("empty   :", sorted(empty))
print("group on complete:", complete["group"])
try:
    print(empty["group"])
except KeyError as exc:
    print("group on empty   : KeyError", exc)

complete: ['end', 'group', 'slack', 'start', 'state']
empty   : ['start']
group on complete: None
group on empty   : KeyError 'group'

Both records describe the same path. One of them can be written to a CSV; the other cannot.

7. The Record That Never Closes

A report truncated mid-path — a tool killed, a disk filled, Chapter 20.12's territory — leaves a record open at the end of the file:

Startpoint: path_one_start
Endpoint: path_one_end
  slack (VIOLATED)           -0.142

Startpoint: path_two_start
Endpoint: path_two_end
  Path Group: clk

import re

START = re.compile(r"^Startpoint: (\S+)")
END = re.compile(r"^Endpoint: (\S+)")
SLACK = re.compile(r"slack\s+\((?P<state>VIOLATED|MET)\)\s+(?P<value>-?\d+(?:\.\d+)?)")


def parse(path):
    current = None
    complete = []
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            m = START.match(line)
            if m is not None:
                current = {"start": m.group(1), "end": None, "slack": None}
                continue
            if current is None:
                continue
            m = END.match(line)
            if m is not None:
                current["end"] = m.group(1)
                continue
            m = SLACK.search(line)
            if m is not None:
                current["slack"] = float(m["value"])
                complete.append(current)
                current = None
    return complete, current


done, dangling = parse("cut.rpt")
print("complete paths:", len(done))
print("dangling      :", dangling)
print("report is      :", "TRUNCATED" if dangling is not None else "whole")

complete paths: 1
dangling      : {'start': 'path_two_start', 'end': 'path_two_end', 'slack': None}
report is      : TRUNCATED

The second path was opened and never closed. A parser that just returns complete silently reports one path where the file described two, and the missing one is the one at the end — which in a timing report sorted by slack is often the worst.

current after the loop is the evidence, and it should be used rather than discarded. Three defensible policies, and the choice is Chapter 17.7's:

• Report it. Return the dangling record alongside the complete ones, and let the caller decide. That is what the code above does.
• Flag the report. Set a truncated: True field on the result, which is the same shape as Chapter 20.12's reason.
• Raise. Right when a truncated timing report must never be silently signed off, and wrong when a sweep is reading hundreds of them.

What is not defensible is ignoring it, and the giveaway in somebody else's parser is a loop whose state variable is never examined after the loop ends.

8. The Same Shape, Other Reports

The technique is not about timing. Any report where a record spans lines has the same two structural lines:

That last row is different and worth noticing: some records have no terminator at all. A lint report that lists a filename and then its warnings until the next filename closes each record by opening the next one — so the emit happens in the Startpoint: branch rather than in a slack branch, and the final record is emitted after the loop:

LINES = [
    "=== rtl/fifo.sv",
    "  warning: unused signal wr_ptr_d1",
    "  warning: width mismatch at line 88",
    "=== rtl/arb.sv",
    "  warning: latch inferred for grant_q",
]

records = []
current = None

for line in LINES:
    if line.startswith("=== "):
        if current is not None:
            records.append(current)
        current = {"file": line[4:], "warnings": 0}
        continue
    if current is None:
        continue
    if line.strip().startswith("warning:"):
        current["warnings"] += 1

if current is not None:
    records.append(current)

for record in records:
    print(f"{record['file']:<14} {record['warnings']}")

rtl/fifo.sv    2
rtl/arb.sv     1

Two emits — one in the opening branch for the previous record, one after the loop for the last. Forgetting the second is the classic version of this bug, and it always loses exactly one record: the last one.

That is also why the dangling check in Section 7 is not optional bookkeeping. In a terminator-less format, the dangling record is not an error case — it is every record's normal path out.

9. Predict, Run, Observe

LINES = ["Startpoint: a", "Endpoint: b", "slack (MET) 1.0",
         "junk", "Startpoint: c", "slack (MET) 2.0"]


def parse(lines, reset=True):
    current = None
    out = []
    for line in lines:
        if line.startswith("Startpoint:"):
            current = {"start": line.split()[1], "end": None, "slack": None}
            continue
        if current is None:
            continue
        if line.startswith("Endpoint:"):
            current["end"] = line.split()[1]
            continue
        if "slack" in line:
            current["slack"] = float(line.split()[-1])
            out.append(current)
            if reset:
                current = None
    return out, current


# A
done, dangling = parse(LINES)
print("A:", len(done), dangling)

# B
print("B:", done[0])

# C
print("C:", done[1])

# D
done2, _ = parse(LINES, reset=False)
print("D:", len(done2))

# E
done3, dangling3 = parse(["Startpoint: a", "Endpoint: b"])
print("E:", len(done3), dangling3)

# F
done4, _ = parse(["slack (MET) 9.0"])
print("F:", len(done4))

The verified output:

A: 2 None
B: {'start': 'a', 'end': 'b', 'slack': 1.0}
C: {'start': 'c', 'end': None, 'slack': 2.0}
D: 2
E: 0 {'start': 'a', 'end': 'b', 'slack': None}
F: 0

A — two complete records and nothing dangling.

B — the first record's fields belong together, which is the whole point.

C — the second path has no Endpoint: line, so end is None. The key is present; Section 6.

D — without the reset this input still gives two, because every slack line here follows a Startpoint:. The bug does not reproduce on well-formed input, which is exactly why it survives review.

E — a record opened and never closed: nothing complete, and the dangling record available to inspect.

F — a slack line with no open record is ignored, because current is None. Header junk cannot invent a path.

10. Exercises

Exercise 1 — the two lists. findall the slacks and the startpoints from a timing report. Say what connects them and why that is an assumption.

Exercise 2 — break the assumption. Add a path whose slack line is missing. Re-run both findalls and say which pairs are now wrong.

Exercise 3 — the two structural lines. For a report format of your own, name the line that opens a record and the line that closes it.

Exercise 4 — the parser. Write the stateful loop for your format and print one complete record per line of output.

Exercise 5 — None-or-dict. Write the version with a separate boolean, then remove it. Say what state became impossible.

Exercise 6 — forget the reset. Remove the reset and run it on a report with stray text after a record. Record the wrong output and say why both stored records are identical.

Exercise 7 — the shared dict. Explain in two sentences why appending without resetting gives two references to one object, and which chapter covered that.

Exercise 8 — complete at open. Build the record empty and add keys as lines arrive. Run a report loop over a record that is missing a field and record the exception.

Exercise 9 — the dangling record. Truncate a report mid-record. Report how many complete records you got and what the state variable held afterwards.

Exercise 10 — three policies. For your own flow, choose between reporting, flagging and raising on a dangling record. Justify it in two sentences.

Exercise 11 — no terminator. Write a parser for a format where the next header closes the previous record. Confirm the last record is emitted, then delete the post-loop emit and say exactly what you lose.

Exercise 12 — named states. Sketch the state variable for a report with three sections. Say why a string beats inferring the section from which keys are filled.

11. Summary

• A record spread over several lines cannot be parsed by a pattern, because the association between its fields is positional and the data never states it.
• Two separate findall lists pair correctly only while the file is regular, and the pairing breaks silently from the first irregular block onwards.
• Two lines carry the structure: one opens a record, one closes it. Everything between is a field.
• current = None or a dict carries the state and the partial record in one name, so they cannot disagree — better than the canonical boolean plus dict.
• Check for the opening line first and unconditionally, so a new record displaces a malformed open one instead of merging into it.
• if current is None: continue skips the report header and everything between records in one line.
• Build the record complete, with every key, when it opens. Adding keys as lines arrive gives a KeyError on exactly the irregular records.
• Forgetting the reset attaches a later value to a stale record, and because the dict is shared, both stored records become identical.
• The reset bug does not reproduce on well-formed input, which is why it survives review.
• A record open at the end of the file means the report was truncated — the same inference as Chapter 20.12's missing banner.
• Examine the state variable after the loop. Report it, flag it, or raise; ignoring it silently loses the last record.
• The same shape fits synthesis, coverage and lint reports — only the two structural lines change.
• Some formats have no terminator, and there the record is emitted when the next one opens, plus once after the loop. Forgetting the second loses exactly one record: the last.
• Yield the records so a sweep over a thousand reports holds one at a time.

The last chapter of this module puts every one of these facts into a single record — the value every later module consumes.