Pre-Merge Review: Parser Robustness

Part II · Module 27 · Chapter 27.9 · Pre-Merge Review: Parser Robustness

1. The Checklist

A parser passes its author's test because its author tested it with the input they had. Production sends it every input that exists. The robustness review closes that gap with one question per kind of input:

Questions 1 and 5 overlap on purpose: supporting the versions you know is question 1; behaving safely on a version you do not know is question 5.

2. The Inputs, Produced by Running the Tool

The review is only as good as its inputs, and Chapter 27.2's rule applies: produce them, do not type them. There is no timing tool on the machine these examples were run on, so the reports come from Chapter 26.4's teaching mock — plain Tcl that writes this module's synthetic report format. One more corner data file is needed, a test mode in which no paths are constrained:

# synthetic path data for the teaching mock: no paths constrained in this mode

#!/bin/bash
# make_inputs.sh -- every input the review needs, produced by running the
# 26.4 teaching mock (plus ONE constructed variant, labelled below).
set -euo pipefail
mkdir -p in

one_corner() {            # one_corner CORNER OUTFILE : one report in one file
    printf 'read_design chip_top\nset_corner %s\nreport_timing -max_paths 2 > %s\n' "$1" "$2" > one.tcl
    tclsh mock_sta.tcl -f one.tcl > /dev/null
}
one_corner ss_0p72v_125c       in/ss.rpt          # an ordinary report
one_corner tt_0p80v_25c_scan   in/scan_empty.rpt  # a corner with no paths at all

# several corners reported to stdout in one session -- one aggregate log
printf 'read_design chip_top\n' > all.tcl
for c in ss_0p72v_125c tt_0p80v_25c; do printf 'set_corner %s\nreport_timing -max_paths 2\n' "$c" >> all.tcl; done
tclsh mock_sta.tcl -f all.tcl > in/aggregate.log

head -n 7 in/ss.rpt > in/truncated.rpt                 # cut mid-report
printf 'Startpoint: a\nEndpoint  : b\n  slack is fine\n' > in/not_a_report.txt

# CONSTRUCTED: no second format version exists here. This imitates a version
# that prints the status AFTER the value -- a field that moved.
sed -E 's/slack \((MET|VIOLATED)\) +(-?[0-9.]+)/slack \2 (\1)/' in/ss.rpt > in/ss_v2.rpt
# CONSTRUCTED: a layout NOBODY anticipated, to test that it is refused loudly.
sed -E 's/slack \((MET|VIOLATED)\) +(-?[0-9.]+)/slack: \2 ns \1/' in/ss.rpt > in/ss_v3.rpt
ls in

Five of the inputs are real runs of the mock: an ordinary report, a corner with no paths, two corners reported to standard output in one session (an aggregate log, banner line included), a report cut off after its first path, and a file that is not a report at all. Two are constructed with sed, and labelled in the script, because no second format version exists here: ss_v2.rpt imitates a later version that prints the status after the value, and ss_v3.rpt imitates a layout nobody anticipated.

#!/bin/bash
bash make_inputs.sh > /dev/null
echo "--- in/scan_empty.rpt"; sed 's/^/  | /' in/scan_empty.rpt
echo "--- in/aggregate.log (first 5 lines)"; head -n 5 in/aggregate.log | sed 's/^/  | /'
echo "--- the slack lines in each version"
grep -h slack in/ss.rpt in/ss_v2.rpt in/ss_v3.rpt | sed 's/^/  | /'

--- in/scan_empty.rpt
  | Report : timing (synthetic teaching format)
  | Design : chip_top
  | Corner : tt_0p80v_25c_scan
  | 
  | End of report
--- in/aggregate.log (first 5 lines)
  | mock_sta: design chip_top loaded
  | Report : timing (synthetic teaching format)
  | Design : chip_top
  | Corner : ss_0p72v_125c
  | 
--- the slack lines in each version
  |   slack (VIOLATED)   -0.043
  |   slack (MET)         0.120
  |   slack -0.043 (VIOLATED)
  |   slack 0.120 (MET)
  |   slack: -0.043 ns VIOLATED
  |   slack: 0.120 ns MET

3. The Parser as Submitted, and the Review

This is the parser submitted for review. It reads one report, collects the slack values, and prints the corner, the number of paths and the worst slack. On the report its author tested — an ordinary one — it is correct.

#!/usr/bin/perl
# parse_report_submitted.pl -- as submitted for review. Works on the report
# its author tested it with.
use strict;
use warnings;

my ($corner, @slack);
while (my $line = <>) {
    $corner //= $1 if $line =~ /^Corner\s*:\s*(\S+)/;
    push @slack, $1 if $line =~ /^\s+slack \((?:MET|VIOLATED)\)\s+(-?\d+\.\d+)/;
}
my ($worst) = sort { $a <=> $b } @slack;
printf "%s paths=%d worst=%s\n", $corner // "?", scalar @slack, $worst // "n/a";

The review runs it on every input and writes down, next to each, what a person reading that input would conclude. It runs the reviewed version (section 9) alongside, so each failure can be compared with what the right answer looks like.

use strict;
use warnings;

# Each checklist question, as one input, with the answer a reader of that
# input would give.
my @CASES = (
    [ "ordinary report",          "in/ss.rpt",           "ss: 2 paths, worst -0.043" ],
    [ "empty section",            "in/scan_empty.rpt",   "scan corner: 0 paths, and that is real" ],
    [ "truncated file",           "in/truncated.rpt",    "incomplete -- do not trust" ],
    [ "multi-corner aggregate",   "in/aggregate.log",    "ss 2 paths, tt 2 paths, separately" ],
    [ "a field that moved (v2)",  "in/ss_v2.rpt",        "ss: 2 paths, worst -0.043" ],
    [ "a layout nobody expected", "in/ss_v3.rpt",        "an error: the slack lines are unreadable" ],
    [ "not a report at all",      "in/not_a_report.txt", "an error, loudly" ],
);

sub run {
    my ($parser, $input) = @_;
    my $out = `perl $parser $input 2>&1`;
    my $rc = $? >> 8;
    $out =~ s/\n(?=.)/ | /g;
    chomp $out;
    return "$out  (exit $rc)";
}

for my $c (@CASES) {
    my ($question, $input, $truth) = @$c;
    print "$question  [$input]\n";
    print "  truth:     $truth\n";
    print "  submitted: ", run("parse_report_submitted.pl", $input), "\n";
    print "  reviewed:  ", run("parse_report_reviewed.pl",  $input), "\n";
}

#!/bin/bash
bash make_inputs.sh > /dev/null
perl robustness_review.pl

ordinary report  [in/ss.rpt]
  truth:     ss: 2 paths, worst -0.043
  submitted: ss_0p72v_125c paths=2 worst=-0.043  (exit 0)
  reviewed:  ss_0p72v_125c paths=2 worst=-0.043 OK  (exit 0)
empty section  [in/scan_empty.rpt]
  truth:     scan corner: 0 paths, and that is real
  submitted: tt_0p80v_25c_scan paths=0 worst=n/a  (exit 0)
  reviewed:  tt_0p80v_25c_scan paths=0 worst=n/a OK  (exit 0)
truncated file  [in/truncated.rpt]
  truth:     incomplete -- do not trust
  submitted: ss_0p72v_125c paths=1 worst=-0.043  (exit 0)
  reviewed:  ss_0p72v_125c paths=1 worst=-0.043 INCOMPLETE  (exit 2)
multi-corner aggregate  [in/aggregate.log]
  truth:     ss 2 paths, tt 2 paths, separately
  submitted: ss_0p72v_125c paths=4 worst=-0.043  (exit 0)
  reviewed:  ss_0p72v_125c paths=2 worst=-0.043 OK | tt_0p80v_25c paths=2 worst=0.212 OK  (exit 0)
a field that moved (v2)  [in/ss_v2.rpt]
  truth:     ss: 2 paths, worst -0.043
  submitted: ss_0p72v_125c paths=0 worst=n/a  (exit 0)
  reviewed:  ss_0p72v_125c paths=2 worst=-0.043 OK  (exit 0)
a layout nobody expected  [in/ss_v3.rpt]
  truth:     an error: the slack lines are unreadable
  submitted: ss_0p72v_125c paths=0 worst=n/a  (exit 0)
  reviewed:  ss_0p72v_125c paths=0 worst=n/a UNREADABLE_SLACK_LINES=2  (exit 2)
not a report at all  [in/not_a_report.txt]
  truth:     an error, loudly
  submitted: ? paths=0 worst=n/a  (exit 0)
  reviewed:  parse_report: no 'Corner :' line -- not a timing report  (exit 2)

The submitted parser is right on two of seven inputs — the ordinary report it was written for, and the empty corner. On the other five it is wrong, and it exits 0 every time, so nothing downstream can tell. The reviewed parser is right on all seven, and exits 2 on the three inputs that must not be trusted.

The worst pattern in the table is not any single wrong row. It is that four different situations produce the same output from the submitted parser: a genuinely empty corner, a moved field, a layout nobody anticipated, and a file that is not a report all print paths=0 worst=n/a with exit 0. Only one of the four means "no paths". A reader of the summary cannot tell them apart, which means the parser has destroyed the information that matters most.

4. Questions 1 and 5: Versions, Known and Unknown

The moved field made the submitted parser report zero paths. Its slack pattern requires (STATUS) value; the v2 layout is value (STATUS), so every slack line simply failed to match and was skipped. Nothing was wrong as far as the parser could tell — it found a corner with no paths. A violating corner, read with the wrong version's pattern, looks like an unconstrained one.

The reviewed parser handles the known v2 layout with a second pattern, $SLACK_V2, named for the version it serves (Chapter 18.8's approach to version tolerance). That answers question 1 for the versions in use.

Question 5 asks the harder thing: what happens with a version nobody has seen yet? The reviewed parser has one more branch — any line that starts like a slack line (/^\s+slack\b/) but matches neither known layout is counted, and a non-zero count makes the corner UNREADABLE_SLACK_LINES and the exit status 2. The v3 row shows it working on a layout written for this test: two unreadable slack lines, reported, exit 2. The parser does not need to understand a new format to notice that it does not understand it.

5. Question 2: A Real Zero Must Be Distinguishable

Both parsers say the scan corner has zero paths, and that is correct — the corner data has none. The question is not whether the parser can report zero; it is whether zero is the only thing that ever produces zero.

For the submitted parser it is not, as section 3 showed. For the reviewed parser, a real empty corner is paths=0 ... OK with exit 0, while every failure that also found no paths carries a state and exit 2. Chapter 26.3 made the same point about results files — empty and zero are different claims — and this is where a parser either keeps that difference or loses it.

6. Question 3: A Truncated File Must Not Pass

The submitted parser reported the truncated report as one path, exit 0. The report was cut after its first path, which is the violating one, so the summary even looks plausible. Chapter 18.9 measured the same failure for tool logs and Chapter 26.5 for report directories: the evidence of truncation is a missing last line, and only a parser that looks for it will notice.

The reviewed parser marks each report complete only when it sees End of report, and reports INCOMPLETE — with exit 2 — when the file ends first.

7. Question 4: A Multi-Corner Aggregate

The submitted parser read the aggregate log as one corner with four paths. It takes the corner name from the first Corner : line (//= keeps the first value) and collects every slack in the file into one list. The typical corner's two paths were silently credited to the slow corner. In this case the worst slack is unchanged, so even that number looks right; in an aggregate where the second corner is the worse one, the slow corner would be blamed for its violation.

The reviewed parser starts a new record at every Corner : line — the state-machine shape Chapter 18.7 built for sectioned reports — and closes it at End of report. Each corner's paths stay with that corner. Lines before the first report, such as the tool's banner, are skipped because there is no current record yet.

8. Question 6: Something That Is Not a Report

The submitted parser read a file with no report header as corner ?, zero paths, exit 0. A wrong path in a flow script — the log instead of the report, a directory listing, an old file of a different kind — produces exactly this, and the summary shows a corner called ? that nobody will investigate.

The reviewed parser requires at least one Corner : line, and otherwise prints a message naming the problem on standard error and exits 2. Chapter 27.4's negative tests made the same demand of the simulation-log parser: input of the wrong kind must be refused, not interpreted.

9. The Reviewed Parser

#!/usr/bin/perl
# parse_report_reviewed.pl -- the same job, after the robustness review.
# Output: one line per corner found.  Exit: 0 every report complete and
# understood; 2 anything incomplete, unrecognised, or no report at all.
use strict;
use warnings;

# Both known layouts of the slack line, each named for the version it is from.
my $SLACK_V1 = qr/^\s+slack \((MET|VIOLATED)\)\s+(-?\d+\.\d+)\s*$/;   # status, then value
my $SLACK_V2 = qr/^\s+slack (-?\d+\.\d+) \((MET|VIOLATED)\)\s*$/;     # value, then status

my (@reports, $cur);
while (my $line = <>) {
    if ($line =~ /^Corner\s*:\s*(\S+)/) {             # each report starts a new record
        $cur = { corner => $1, slack => [], complete => 0, unknown => 0 };
        push @reports, $cur;
        next;
    }
    next unless $cur;                                  # banner lines before any report
    if    ($line =~ $SLACK_V1)        { push @{ $cur->{slack} }, $2 }
    elsif ($line =~ $SLACK_V2)        { push @{ $cur->{slack} }, $1 }
    elsif ($line =~ /^\s+slack\b/)    { $cur->{unknown}++ }       # a slack line we cannot read
    elsif ($line =~ /^End of report$/) { $cur->{complete} = 1; undef $cur }
}

unless (@reports) {
    print STDERR "parse_report: no 'Corner :' line -- not a timing report\n";
    exit 2;
}
my $bad = 0;
for my $r (@reports) {
    my ($worst) = sort { $a <=> $b } @{ $r->{slack} };
    my $state = !$r->{complete} ? "INCOMPLETE"
              : $r->{unknown}   ? "UNREADABLE_SLACK_LINES=$r->{unknown}"
              :                   "OK";
    $bad++ if $state ne "OK";
    printf "%s paths=%d worst=%s %s\n", $r->{corner}, scalar @{ $r->{slack} }, $worst // "n/a", $state;
}
exit($bad ? 2 : 0);

It is about twice the length of the submitted version, and every added line answers one question in the checklist. That is the useful way to read a review's outcome: not "the code got longer", but "each line now has a reason that is written down as a test input".

Two further steps finish the job, and both come from earlier chapters. The seven inputs belong in the parser's test suite as golden fixtures, with Chapter 27.4's negative tests asserting the exit status for the three that must fail. And the two slack patterns belong in Chapter 27.5's table, one row per real line from each version, including the lines that must not match.

10. Exercises

1. A third known version. Suppose the v3 layout turns out to be real and in use. Add support for it the way Chapter 27.5 prescribes — row first, then the pattern — and confirm that the v1 and v2 rows still pass and that a fourth, different layout is still refused.

2. The worse aggregate. Build an aggregate log in which the second corner has the worst slack, and show what the submitted parser blames on the first corner. Then write the one-line negative test that would have caught this before review.

3. Banner lines. The aggregate log starts with the mock's banner. Find a banner line that would confuse the reviewed parser — think about a line that starts with Corner — and decide whether to fix the parser or to call the input malformed.

4. Unknown versus empty. Construct an input in which a report has some readable slack lines and one unreadable one. What does the reviewed parser report, and is exit 2 the right answer when most of the corner was read correctly? Argue both sides.

5. Review a real parser. For a parser you maintain, produce one real input for each of the six questions. Record which ones you could not produce from your tools, and what that tells you about the risk that remains.

11. What To Remember

• Review a parser by running it on every kind of input production will send, not by reading it.
• Six questions: every tool version, an empty section, a truncated file, a multi-corner aggregate, a moved field, an unrecognised file.
• Produce the inputs by running the tool (here, the teaching mock), and label any you had to construct.
• Measured: the submitted parser was wrong on 5 of 7 inputs and exited 0 every time.
• Four different situations printed the same paths=0 line. A parser that maps failures onto a real value destroys the information that matters.
• A field that moved must not vanish. Support the known layouts by name, and count slack-like lines that match none of them.
• A truncated report must fail: require the end marker (18.9).
• An aggregate needs a new record per section (18.7), or one corner's paths are credited to another.
• A file that is not a report must be refused, on stderr, with a non-zero exit.
• Every line a review adds should be backed by a test input — fixtures (27.2), negative tests (27.4), a regex table (27.5).

Chapter 27.10 moves from whether a parser is right today to whether the next engineer can keep it right: the maintainability review.