Tcl · Module 24

Reviewing a Report Parser

A parser is tested by the reports it will receive when something has gone wrong. Three stages of defence, five specimens, and the principle that makes the third stage work.

Advanced21 min readtcledapart-iireportstestingreview

Part II · Module 24 · Chapter 24.7 · Reviewing a Report Parser

Every parser in this module worked on the report it was written for. That is the easy case and it is not the one that matters.

The case that matters is the report that arrives when something else has already gone wrong — and a parser's behaviour then decides whether the flow stops or reports success.

1. The specimens

Four degenerate reports, each produced from the real one by a single mechanical operation, and each modelling a failure that actually happens:

Specimen	How it was made	What it models
`empty.rpt`	zero bytes	the tool died before writing anything, or the disk filled
`truncated.rpt`	first 20 lines of the real report	the run was killed partway through
`renamed.rpt`	`slack (VIOLATED)` → `worst slack (VIOLATED)`	a tool upgrade changed a label
`nosection.rpt`	every `slack` line deleted	the section is absent because the analysis did not run

Three things about that table are deliberate.

Each is one operation. Truncation, substitution, deletion. Building a specimen suite does not require inventing realistic-looking broken files — it requires damaging a real one in one identifiable way, so that when a test fails you know exactly which property the parser depended on.

None of them is corrupt in a way that would crash. All four are valid text files a parser will read happily to the end. That is the point: the interesting failures are the ones that produce a result.

The real report stays in the suite. A suite of only broken inputs cannot tell a parser that refuses everything from a parser that works — the lesson from Chapter 22.9 and from every checker that got switched off for crying wolf.

2. Stage 1: the naive gate

Here is the gate almost everyone writes, and it is defensible on its face — parse the report, count the violations, pass if there are none:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

proc gate_naive {path} {
    set paths [parse_timing $path]
    set v 0
    foreach p $paths { if {[dict get $p status] eq "VIOLATED"} { incr v } }
    if {$v == 0} { return "PASS" }
    return "FAIL ($v violated)"
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== the naive gate on five inputs ===
    timing.rpt     paths 3   verdict FAIL (2 violated)  the real report
    empty.rpt      paths 0   verdict PASS              an empty file
    truncated.rpt  paths 0   verdict PASS              a run killed partway through
    renamed.rpt    paths 1   verdict PASS              the field renamed by a tool upgrade
    nosection.rpt  paths 0   verdict PASS              the slack section simply absent

This is the failure shape every chapter in this module has hit from a different angle. 24.1 §7 returned a truncated number with ok=1. 24.3 §2 inverted a conclusion with three successful matches. 24.6 §3 wrote a corrupt row with a correct-looking library call. Here a gate passes a design on an empty file. In every case the operation succeeded.

3. Stage 2: make the parser assert what it expected

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

proc parse_timing_checked {path} {
    if {![file isfile $path]} { error "no such report: $path" {} {RPT NOFILE} }
    if {[file size $path] == 0} { error "report is empty: $path" {} {RPT EMPTY} }
    set paths [parse_timing $path]
    if {[llength $paths] == 0} {
        error "no timing paths found in $path — wrong format, or the field was renamed" \
              {} {RPT NOPATHS}
    }
    # every path must be complete: a slack with no endpoint is a parse failure
    set i 0
    foreach p $paths {
        incr i
        foreach k {start end group status slack} {
            if {[dict get $p $k] eq ""} {
                error "path $i parsed with an empty '$k'" {} [list RPT INCOMPLETE $k]
            }
        }
    }
    return $paths
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== the checked parser on the same five ===
    timing.rpt     accepted 3 path(s)
    empty.rpt      REFUSED  report is empty: reports/empty.rpt
    truncated.rpt  REFUSED  no timing paths found in reports/truncated.rpt — wrong format, or the field was renamed
    renamed.rpt    accepted 1 path(s)
    nosection.rpt  REFUSED  no timing paths found in reports/nosection.rpt — wrong format, or the field was renamed

Three of four caught. Note what the checks are doing:

file isfile and file size before parsing at all. Cheap, and they name the real problem instead of letting it become "0 paths".
Zero paths is an error, not a result. This is the Chapter 23.6 §6 argument — an ECO script that made no changes ran perfectly and did nothing — and Chapter 21.2's empty file list. A parser that found nothing has either been given the wrong file or has stopped matching.
Every field of every row must be non-empty. This catches the partial-block case directly rather than inferring it: a slack line that committed a row with no endpoint means a pattern stopped matching mid-block.
Each error carries a machine-readable errorCode — RPT EMPTY, RPT NOPATHS, RPT INCOMPLETE start — so a caller can distinguish "no report" from "wrong format" and act differently, per Chapter 14.2.

And then there is renamed.rpt.

4. The specimen that defeats stage 2

renamed.rpt was accepted. It parsed 1 path, that path was MET, so the gate says PASS.

The substitution changed slack (VIOLATED) to worst slack (VIOLATED), so the anchored pattern ^\s*slack \(…\) no longer matches those two lines. The slack (MET) line was not renamed, so it still matches — and one complete, valid, fully-populated path survives.

Every stage-2 check is satisfied. The file exists, it is non-empty, at least one path was found, and that path has all five fields. The parser is behaving exactly as designed.

Two of three paths vanished, both of them the violating ones, and the design reports clean.

This is also the most likely of the four specimens to happen for real. A tool upgrade changing a label is routine; a zero-byte report is rare. The failure that defeats your checks is the one that happens most often.

More checks of the same kind do not help, and that is the key observation. Every stage-2 check is written in terms of what the parser found, and the parser found one perfectly good path. You cannot detect a pattern that stopped matching by asking the pattern what it matched.

5. Stage 3: measure the file a second, independent way

The way out is to count something about the file that does not go through the pattern under test:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

proc count_lines_matching {path re} {
    set fh [open $path r] ; set n 0
    while {[gets $fh line] >= 0} { if {[regexp $re $line]} { incr n } }
    close $fh
    return $n
}

proc parse_timing_crosschecked {path} {
    if {![file isfile $path]} { error "no such report: $path" {} {RPT NOFILE} }
    if {[file size $path] == 0} { error "report is empty: $path" {} {RPT EMPTY} }
    set paths [parse_timing $path]
    # Every path block carries exactly one Startpoint and exactly one slack line.
    # Counting the blocks independently of the parse catches a pattern that
    # stopped matching.
    set blocks [count_lines_matching $path {^\s*Startpoint:}]
    if {$blocks == 0} { error "no path blocks in $path" {} {RPT NOBLOCKS} }
    if {[llength $paths] != $blocks} {
        error "parsed [llength $paths] path(s) from $blocks block(s) — a field has changed" \
              {} [list RPT COUNTMISMATCH [llength $paths] $blocks]
    }
    return $paths
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== blocks counted independently of the parse ===
    FILE           BLOCKS   PARSED   VERDICT
    timing.rpt     3        3        accepted
    empty.rpt      0        0        REFUSED
    truncated.rpt  1        0        REFUSED
    renamed.rpt    3        1        REFUSED
    nosection.rpt  3        0        REFUSED

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== the renamed file, in full ===
    catch 1
    parsed 1 path(s) from 3 block(s) — a field has changed
    errorcode: RPT COUNTMISMATCH

All four degenerate inputs refused, the real one accepted. And the error message names the actual problem — a field has changed — rather than a symptom.

The truncated.rpt row is worth a second look: 1 block, 0 parsed. The truncation cut the file in the middle of the first block, after its Startpoint: and before its slack line. The count catches that too, for the same reason.

The principle, and it is what Module 24 has been building toward.

A parser cannot validate itself with more of its own patterns, because the thing that failed is a pattern. It needs a second, independent measurement of the same file.

That is why every cross-check in this module worked, and they are four instances of one idea:

Chapter	First measurement	Independent second measurement
24.2 §4	the component areas	they sum to the stated total, to the last digit
24.3 §4	three fields in two units	the normalised total reproduces the report's `14.3316 mW`
24.4 §6	5 parsed rows	the sections' own `Total N (VIOLATED)` lines sum to 5
24.7 §5	3 parsed paths	3 `Startpoint:` lines counted separately

"Independent" is the load-bearing word. Counting Startpoint: lines with the same pattern the parser uses for them would prove nothing. The counts are independent because the two measurements depend on different parts of the file — one on the slack lines, one on the block headers — so a change to either is visible as a disagreement.

Reports hand these assertions over for free, because they almost always state a total beside their parts. Take them.

6. What a cross-check still cannot catch

Honest limits, because stage 3 now looks like a solution.

A change that affects both measurements equally. If a release renamed Startpoint: to Start Point: and changed the slack line, both counts would go to zero together and RPT NOBLOCKS would fire — which is the right outcome by luck rather than by design. A format change large enough to break everything is caught; one that breaks both counts by the same amount is not.

A value that parsed wrongly but completely. 24.1 §7's -0.0 is a fully populated path whose slack is wrong by a factor of 73. Three blocks, three paths, counts agree, every field non-empty — and a number that is wrong. Count cross-checks verify structure, not values; the defence against a wrong value is the arithmetic cross-check of 24.2 §4 and 24.3 §4, which is why both kinds are worth having.

A report that is internally consistent and describes the wrong design. No amount of parsing catches a correct report on a stale netlist. That is Chapter 22.9's subject and it lives upstream of everything here.

The specimens you did not think of. The suite has four entries because four failures were imagined. A fifth will happen. The mitigation is that stage 3's cross-check is not specimen-specific — it catches any disagreement between the two counts, including from a cause nobody anticipated, which is why a structural invariant is worth more than a longer list of tests.

7. The review

What to check before trusting a report parser:

Does it refuse a missing file and a zero-byte file by name, rather than returning zero rows?
Is zero rows an error? A parser that found nothing has the wrong file or a stale pattern.
Is every field of every row asserted non-empty, so a half-parsed block cannot become a row?
Is there a second, independent count — block headers against parsed rows, or the report's own total against yours?
Is there an arithmetic cross-check where the report states a total beside its parts?
Are units captured with values, so a comparison cannot be silently wrong (Chapter 24.3)?
Are provenance fields captured — design, library, tool version, corner — so two runs can be refused as incomparable (Chapter 24.2 §6)?
Does every error carry a machine-readable code, so a caller can tell "no report" from "wrong format"?
Does the suite contain the good report too, so a parser that refuses everything fails the suite?
Is the exit status non-zero when any of this fails, so the flow stops rather than continuing on a result nobody checked?

That last one closes the module where Chapter 15.4 began: a script whose checks all failed and whose exit status is 0 has told the flow that everything is fine.

8. Exercises

Exercise 1. Why is each specimen made by a single mechanical operation on the real report?

Exercise 2. Why must the real report stay in the specimen suite?

Exercise 3. The naive gate reports PASS on a zero-byte file. Name the inference that is wrong.

Exercise 4. Why is "zero rows parsed" an error rather than a result?

Exercise 5. renamed.rpt passes every stage-2 check. Explain how, including why it parsed 1 path rather than 0.

Exercise 6. Why can't more checks of the stage-2 kind catch the renamed field?

Exercise 7. State the closing principle in one sentence.

Exercise 8. Why would counting Startpoint: lines with the parser's own pattern prove nothing?

Exercise 9. truncated.rpt shows 1 block and 0 parsed paths. What does that tell you about where the file was cut?

Exercise 10. Name one thing a count cross-check cannot catch, and what does catch it.

Answers.

1. So that when a test fails you know exactly which property the parser depended on. A realistic-looking broken file that fails for several reasons at once tells you much less.

2. Because a suite of only broken inputs cannot distinguish a working parser from one that refuses everything — and a checker that refuses everything gets switched off.

3. "No violations found, therefore no violations." It is valid only if the report was actually read, and the gate never established that. Absence of evidence became evidence of absence.

4. Because a parser that found nothing has either been given the wrong file or has stopped matching. Both are failures, and neither is "the design is clean".

5. The substitution broke the two slack (VIOLATED) lines but left slack (MET) untouched, so exactly one complete path survives. The file exists, is non-empty, yielded at least one path, and that path has all five fields — so every check passes, and because the surviving path is MET the gate reports PASS. Two violations vanished.

6. Because every stage-2 check is expressed in terms of what the parser found, and the parser found one perfectly good path. You cannot detect a pattern that stopped matching by asking the pattern what it matched.

7. A parser cannot validate itself with more of its own patterns, because the thing that failed is a pattern — it needs a second, independent measurement of the same file.

8. Because the two measurements would no longer be independent. They must depend on different parts of the file — the slack lines and the block headers — so that a change to either shows up as a disagreement.

9. It was cut inside the first block, after its Startpoint: line and before its slack line — so the block header exists and no row was ever committed.

10. A value that parsed completely but wrongly — -0.0 instead of -0.0730, where all counts agree. The defence against that is an arithmetic cross-check against a total the report states.

Remember.

Build specimens by one mechanical operation each — truncate, substitute, delete — and keep the good report in the suite.
The interesting failures do not crash. All four degenerate reports are valid text files that produce a result.
A naive gate reports PASS on four of five wrong inputs, a zero-byte file included. Absence of evidence is not evidence of absence.
Refuse a missing or empty file by name, make zero rows an error, and assert every field of every row non-empty.
A renamed field defeats all of that — one untouched line yields one complete valid path, and the gate passes. It is also the likeliest failure in practice.
A parser cannot validate itself with more of its own patterns. It needs a second, independent measurement of the same file.
Independent means depending on a different part of the file — block headers against parsed rows, not the same pattern counted twice.
Reports hand you these assertions for free by stating a total beside their parts. Four chapters of this module used one.
Counts verify structure; arithmetic verifies values. Have both — -0.0 passes every count.
Exit non-zero when a check fails, or the flow has been told everything is fine.

Where this fits

Part of the Tcl curriculum.