Testing a Regex Against Every Tool Version

Part II · Module 27 · Chapter 27.5 · Testing a Regex Against Every Tool Version

1. The Pattern Is the Parser's Riskiest Line

Look back at the shared module from Chapter 27.1. Almost everything in it is ordinary control flow. The two lines that decide whether it is right are the patterns:

my $FINISH = qr/^\S+:\d+: \$finish(?:\(\d\))? called at /;
my $SLACK  = qr/^\s*slack \((?:MET|VIOLATED)\)\s+(-?\d+\.\d+)\s*$/;

Each is one line, and each encodes several decisions: what comes before the marker, which spellings are accepted, what is captured, where the line must end. Patterns are also the lines most likely to be edited, because they are the ones that "obviously" need simplifying or extending — Chapter 27.3's suite caught exactly such an edit.

Whole-log tests check patterns only indirectly: a pattern mistake shows up only if some golden log happens to contain a line that exercises it. A regex regression table checks the pattern directly. It is a list of lines, each with the result the pattern must give, including lines that must not match. Every line comes from real output, every row records which tool version it came from, and the whole table runs on every change.

2. Capture Every Way a Run Can End

The first table tests $FINISH. To build it, capture every way the simulator can print the end of a run. The script below writes one tiny testbench per ending — $finish with each diagnostic level, and $stop — and each testbench first prints an INFO line that mentions the task it is about to call, because real testbenches say things like that and a pattern must not mistake the mention for the event.

#!/bin/bash
# capture_endings.sh -- one real log per way a simulation can end, from the
# simulator in use. Each testbench first prints an INFO line that MENTIONS the
# system task, so every log also holds a line that must NOT count as the end.
set -euo pipefail
mkdir -p tb t/data/endings
version=$(iverilog -V 2>/dev/null | sed -n 1p)
for task in 'finish' 'finish(0)' 'finish(1)' 'finish(2)' 'stop'; do
    name=end_$(echo "$task" | tr -d '()')
    printf 'module tb;\n  initial begin\n    $display("INFO: calling $%s next");\n    #3 $%s;\n  end\nendmodule\n' \
        "$task" "$task" > "tb/$name.v"
    iverilog -g2012 -o "$name.vvp" "tb/$name.v"
    vvp -n "$name.vvp" > "t/data/endings/$name.log" 2>&1
    echo "$name.log  \$$task  ($version)"
done

#!/bin/bash
bash capture_endings.sh
for f in t/data/endings/*.log; do
    echo "--- $f"
    sed 's/^/  | /' "$f"
done

end_finish.log  $finish  (Icarus Verilog version 13.0 (stable) (v13_0))
end_finish0.log  $finish(0)  (Icarus Verilog version 13.0 (stable) (v13_0))
end_finish1.log  $finish(1)  (Icarus Verilog version 13.0 (stable) (v13_0))
end_finish2.log  $finish(2)  (Icarus Verilog version 13.0 (stable) (v13_0))
end_stop.log  $stop  (Icarus Verilog version 13.0 (stable) (v13_0))
--- t/data/endings/end_finish.log
  | INFO: calling $finish next
  | tb/end_finish.v:4: $finish called at 3 (1s)
--- t/data/endings/end_finish0.log
  | INFO: calling $finish(0) next
--- t/data/endings/end_finish1.log
  | INFO: calling $finish(1) next
  | tb/end_finish1.v:4: $finish(1) called at 3 (1s)
--- t/data/endings/end_finish2.log
  | INFO: calling $finish(2) next
  | tb/end_finish2.v:4: $finish(2) called at 3 (1s)
--- t/data/endings/end_stop.log
  | INFO: calling $stop next
  | tb/end_stop.v:4: $stop called at 3 (1s)

Read the end_finish0.log block. $finish(0) printed no end-of-run line at all. The simulation finished normally — the process exited 0 — but the only line in the log is the testbench's own INFO. The argument to $finish is a diagnostic level (Chapter 26.7), and level 0 means "print nothing".

That is a finding about the module from Chapter 27.1, and it was found only because the table was built from the tool rather than from memory. For a testbench that ends with $finish(0), parse_log cannot see a finish, so it reports NO_FINISH for a run that finished. No pattern can fix that: there is no line to match. Section 3 records what to do about it.

The other endings are as expected: $finish, $finish(1) and $finish(2) each print a file:line: end line — on this version, the same kind of line for all three — and $stop, which in batch mode (-n) ends the run, prints its own line naming $stop.

3. The Endings Table

Each captured log becomes one row, and each row is a decision with its reason written beside it:

use strict;
use warnings;
use Test::More;
use FindBin qw($Bin);
use lib "$Bin/../lib";
use VLSI::Report qw(parse_log);

# One row per real way a run can end, captured by capture_endings.sh.
# "finished" is what the parser must report. Each row is a DECISION, and the
# reason is written next to it, because the next person will want to change it.
my @CASES = (
    # log                tool version   finished  why
    [ "end_finish",     "icarus 13.0", 1, "the plain end marker" ],
    [ "end_finish1",    "icarus 13.0", 1, "\$finish(1): Icarus prints the same kind of line" ],
    [ "end_finish2",    "icarus 13.0", 1, "\$finish(2): Icarus prints the same kind of line" ],
    [ "end_finish0",    "icarus 13.0", 0, "\$finish(0) prints NO line: nothing to see" ],
    [ "end_stop",       "icarus 13.0", 0, "\$stop is for interactive debugging: flag it" ],
);

for my $c (@CASES) {
    my ($log, $version, $finished, $why) = @$c;
    is parse_log("$Bin/data/endings/$log.log")->{finished}, $finished,
        "$log ($version): finished=$finished -- $why";
}

done_testing;

Two rows deserve their reasons in full, because the next engineer will want to change them.

end_finish0 expects finished=0. That is not a claim that the run failed to finish; it records what the log can show. Three ways to deal with the gap, in order of preference:

• A project rule that testbenches end with $finish or $finish(1), never $finish(0). It costs nothing, and Chapter 27.10's review checklist is where such a rule is enforced.
• A testbench convention that prints its own explicit end line — TEST DONE — just before $finish, so the log always records the end whatever the diagnostic level.
• Outside evidence, from the runner that launched the simulator (Chapter 26.7's runner record). Note that the exit status cannot help: a $finish(0) run and a run killed with TERM both exited 0 there.

What the parser must not do is treat "no end line, but a TEST PASSED" as finished. Chapter 27.4's killed-after-PASSED log looks exactly the same, and calling it a pass is the dangerous mistake. The current behaviour — a false NO_FINISH — is a false alarm, and a false alarm is the safe direction to be wrong in.

end_stop expects finished=0. In batch mode $stop does end the run, so finished=1 would be defensible. The table records the other choice and why: $stop is meant for pausing an interactive debugging session, so a testbench that relies on it in batch is almost always a leftover from debugging, and flagging the run is more useful than passing it. Either choice is reasonable. What matters is that it is written down as a row, so changing it is a visible edit to a test rather than an invisible edit to a pattern.

The tool version column says icarus 13.0 on every row because that is the only version on the machine these examples were run on. In a flow that runs two versions, each version gets its own captured rows, and the table holds all of them.

4. Three Plausible Rewrites, Each Caught by One Row

A table earns its place by catching edits that look reasonable. Here are three rewrites of $FINISH that someone might well propose, run against the endings table and against Chapter 27.3's whole-log suite:

use strict;
use warnings;

# Three rewrites of the end-of-run pattern that someone might well propose,
# each run against the endings table and the golden-log suite.
my $MOD = "lib/VLSI/Report.pm";
my $src = do { open my $f, "<", $MOD or die "$MOD: $!\n"; local $/; <$f> };
my ($line) = $src =~ /^(my \$FINISH = qr.*)$/m or die "no \$FINISH line\n";

my @PROPOSALS = (
    [ "current pattern",           undef ],
    [ "simpler: any \$finish",       q{my $FINISH = qr/\$finish/;} ],
    [ "stricter: only what we saw", q{my $FINISH = qr/^\S+:\d+: \$finish(?:\(1\))? called at /;} ],
    [ "also count \$stop",           q{my $FINISH = qr/^\S+:\d+: \$(?:finish|stop)(?:\(\d\))? called at /;} ],
);

for my $p (@PROPOSALS) {
    my ($name, $new) = @$p;
    (my $edited = $src) =~ s/^\Q$line\E$/$new/m if defined $new;
    open my $w, ">", $MOD or die; print {$w} ($edited // $src); close $w;
    my @failed = map { /^not ok \d+ - (.*)/ ? $1 : () }
                 `perl t/endings.t 2>/dev/null; perl t/parse_log.t 2>/dev/null`;
    printf "%-28s %d failing\n", $name, scalar @failed;
    print "    $_\n" for @failed;
}
open my $w, ">", $MOD or die; print {$w} $src; close $w;

#!/bin/bash
bash capture_endings.sh > /dev/null
perl try_patterns.pl

current pattern              0 failing
simpler: any $finish         1 failing
    end_finish0 (icarus 13.0): finished=0 -- $finish(0) prints NO line: nothing to see
stricter: only what we saw   1 failing
    end_finish2 (icarus 13.0): finished=1 -- $finish(2): Icarus prints the same kind of line
also count $stop             1 failing
    end_stop (icarus 13.0): finished=0 -- $stop is for interactive debugging: flag it

Each rewrite fails exactly one row, and the failing row's reason says why it matters.

"Simpler: any $finish" is the most instructive. Every golden log in Chapter 27.3's suite still passes, so a suite of whole logs would have approved this change. It fails one row in the endings table — end_finish0 — because the unanchored pattern now matches the testbench's own sentence INFO: calling $finish(0) next, and reports a run with no end line as finished. That is precisely the "must not match" case, and only a table that contains such lines can catch it. The anchor and the file:line: prefix in the current pattern are not decoration; they are what make the pattern match the simulator's event rather than any text that mentions it (Chapter 18.4's argument about severity tags).

"Stricter: only what we saw" fails end_finish2. It matches $finish and $finish(1) — the two spellings in the golden logs — and nothing else. Somebody writing a pattern from the examples in front of them produces exactly this, and the first testbench to use $finish(2) then reads as unfinished.

"Also count $stop" fails end_stop. Here the failure is not a bug report; it is the table telling you that this change reverses a recorded decision. If the decision should change, change the row and its reason in the same commit, so that the history shows it was deliberate.

5. The Slack Table, in a Data File

The second pattern extracts slack values. Its table lives in a data file rather than in Perl, so that adding a line from a new tool version is a one-line edit that needs no programming:

# expected | line, verbatim from the module's synthetic report format (26.4)
-0.043 |  slack (VIOLATED)   -0.043
0.120  |  slack (MET)         0.120
0.000  |  slack (MET)         0.000
undef  |expected slack: none
undef  |Total negative slack: -1.234
undef  |  slack (MET)

Each row is EXPECTED | LINE, with the line copied verbatim — leading spaces included — from the module's synthetic report format (the format Chapter 26.4's teaching mock writes; no real timing tool is available here). undef means "must not match".

The rows are chosen, not collected. The first three are the three kinds of value: negative, positive, and exactly zero. The last three are lines that must not produce a number, each for a different reason: a line about slack that is not a path's slack, a summary line containing the word slack and a negative number, and a slack line with its value missing. Chapter 23.2 extracted slack from real-tool report formats; whatever the format, the table needs both halves.

use strict;
use warnings;
use Test::More;
use FindBin qw($Bin);
use lib "$Bin/../lib";
use VLSI::Report qw(extract_slack);

# The table lives in a data file, so adding a line from a new tool version is
# a one-line change that needs no Perl. Format: EXPECTED | LINE (LINE verbatim).
open my $fh, "<", "$Bin/data/slack_lines.txt" or die "slack_lines.txt: $!\n";
my @rows = map { chomp; [ split /\s*\|/, $_, 2 ] } grep { !/^#/ && /\|/ } <$fh>;
close $fh;

my $mode = $ENV{COMPARE} // "numeric";
for my $row (@rows) {
    my ($want, $line) = @$row;
    my $got = extract_slack($line);
    my $name = "[$line]";
    if ($want eq "undef") {
        ok !defined $got, "$name must not match";
    } elsif ($mode eq "string") {
        is $got, $want, "$name -> $want (compared as text)";
    } else {
        ok defined $got && $got == $want, "$name -> $want"
            or diag "got: " . ($got // "undef");
    }
}
done_testing;

The test reads the rows, then checks each one. A must-not-match row checks that the result is undefined — not false, not zero. Section 6 explains the COMPARE switch.

6. Comparing a Number: Text or Value?

There are two ways to compare an extracted slack with the table's expected value, and only one is right.

#!/bin/bash
echo "--- compared as numbers"
perl t/slack_table.t 2>&1 | grep -E '^(ok|not ok)'
echo "--- compared as text, with is()"
COMPARE=string perl t/slack_table.t 2>&1 | grep -E '^(not ok|#  +(got|expected))'
echo "--- and a zero slack, tested for truth"
perl -Ilib -MVLSI::Report=extract_slack -e \
  'my $s = extract_slack("  slack (MET)         0.000");
   printf "  defined: %s   true: %s\n", defined $s ? "yes" : "no", $s ? "yes" : "NO"'

--- compared as numbers
ok 1 - [  slack (VIOLATED)   -0.043] -> -0.043
ok 2 - [  slack (MET)         0.120] -> 0.120
ok 3 - [  slack (MET)         0.000] -> 0.000
ok 4 - [expected slack: none] must not match
ok 5 - [Total negative slack: -1.234] must not match
ok 6 - [  slack (MET)] must not match
--- compared as text, with is()
not ok 2 - [  slack (MET)         0.120] -> 0.120 (compared as text)
#          got: '0.12'
#     expected: '0.120'
not ok 3 - [  slack (MET)         0.000] -> 0.000 (compared as text)
#          got: '0'
#     expected: '0.000'
--- and a zero slack, tested for truth
  defined: yes   true: NO

Compared as text, a correct function fails two rows. is compares strings (Chapter 27.3). The table holds the text 0.120, copied from the report; extract_slack returns the number 0.12, as its contract promises, and the number 0.12 prints as 0.12. The two are equal as values and different as text. The same happens with 0.000 and 0. Chapter 3.5 covered the difference between == and eq; a test table is a place where it bites without warning, because the expected value looks like a number on the page and is a string in the program.

So for a numeric capture, compare values — $got == $want, after checking $got is defined — and write a diagnostic that shows what came back. For a text capture, such as a pin name, is is right.

A zero slack is defined, and false. That is the third line of the output, and it is the trap in every caller that writes if (my $s = extract_slack($line)). A path with exactly zero slack is a real, met path — and that test throws it away, silently. The contract's promise of "a number, or undef" exists so callers can test defined, and the table's zero row exists so the function's own tests do.

7. A New Tool Version: Add the Row, Watch It Fail, Change on Purpose

The table's main job comes the day a tool upgrade changes a line. Since only one tool version is available here, this section uses a constructed line to show the workflow: suppose a new version of the report format appends the unit, -0.043 ns. The procedure has three steps, and their order matters.

First, add the new line to the table, with the result it should give. Then run the table, before touching the pattern:

use strict;
use warnings;

# The deliberate pattern change for the new format: allow an optional "ns".
my $MOD = "lib/VLSI/Report.pm";
my $src = do { open my $f, "<", $MOD or die "$MOD: $!\n"; local $/; <$f> };
my $old = q{(-?\d+\.\d+)\s*$/;};
my $new = q{(-?\d+\.\d+)(?:\s*ns)?\s*$/;};
my $at = index($src, $old);
die "pattern not found\n" if $at < 0;
substr($src, $at, length $old) = $new;
open my $w, ">", $MOD or die "$MOD: $!\n";
print {$w} $src;
close $w;
print "changed: ", ($src =~ /^(my \$SLACK.*)$/m)[0], "\n";

#!/bin/bash
echo "1. add the new version's line to the table, and run it"
printf '%s\n' '-0.043 |  slack (VIOLATED)   -0.043 ns' >> t/data/slack_lines.txt
perl t/slack_table.t 2>&1 | grep -E '^not ok|^# got|Looks like' | sed 's/^/   /'

echo "2. change the pattern on purpose"
perl allow_units.pl | sed 's/^/   /'

echo "3. run every table again: the new row AND every old one"
prove t/slack_table.t t/parse_log.t 2>&1 | grep -E '^(t/|Files|Result)' \
    | sed -E -e 's/[0-9]+ wallclock secs.*/(timing removed)/' -e 's/^/   /'

1. add the new version's line to the table, and run it
   not ok 7 - [  slack (VIOLATED)   -0.043 ns] -> -0.043
   # got: undef
   # Looks like you failed 1 test of 7.
2. change the pattern on purpose
   changed: my $SLACK  = qr/^\s*slack \((?:MET|VIOLATED)\)\s+(-?\d+\.\d+)(?:\s*ns)?\s*$/;
3. run every table again: the new row AND every old one
   t/slack_table.t .. ok
   t/parse_log.t .... ok
   Files=2, Tests=22,  (timing removed)
   Result: PASS

Step 1 failed, and that is the point. The current pattern ends with \s*$ — nothing may follow the number — so the new line returned undef. In a flow, that would not raise an error: every path from the new tool version would simply have no slack, and the violation count would quietly fall to zero. The table turns that silent loss into a failing test before the new version is used.

Step 2 changes the pattern deliberately, allowing an optional ns, and nothing else.

Step 3 runs everything again. The new row passes, and so do all six old rows and all fifteen golden-log tests — which is the evidence that the change made room for the new version without breaking the old one. Chapter 18.8 argued that a flow must keep handling every tool version still in use; this is how you know it does.

The order is the discipline. Changing the pattern first and adding the row afterwards proves only that the new pattern matches the new line, which you already believed. Adding the row first proves the table can detect the problem, and then proves the change fixed it.

8. Exercises

1. Grow the endings table. Add rows for two more real endings — a run that ends because it runs out of events (Chapter 27.4 met one), and a run that ends with $fatal. Capture them with a script, decide what finished should be for each, and write the reason. Does either change what parse_log must do?

2. A pattern that passes the table but is wrong. Write a rewrite of $FINISH that passes every row of the endings table and every golden-log test, yet is wrong for some line the simulator could print. Then add the row that catches it. What does this say about the size a table needs to be?

3. Text captures. Write a third table for a pattern that captures a name — for example, the startpoint in the synthetic timing report. Include a bus bit, a name with an escaped character, and a line that mentions Startpoint but is not one. Which comparison function is right for these rows, and why is it a different answer from section 6's?

4. Two versions at once. Add a column to slack_lines.txt for the format version, and make slack_table.t print a separate count of passes per version. Design what should happen when a row exists for a version the flow no longer uses: delete it, or keep it?

5. The $finish(0) decision. Choose one of section 3's three remedies for the $finish(0) gap and implement it fully: the testbench change, the parser change if any, and the table rows that prove it. Then explain why the remedy you did not choose would have been worse for your flow.

9. What To Remember

• Patterns are a parser's riskiest lines — short, dense, and most often "simplified".
• A regex table lists real lines and the result each must give, including lines that must not match, with the tool version each came from.
• Build the table from the tool. Measured: $finish(0) prints no end line at all, so a run that finished reads as NO_FINISH.
• No pattern can match a line that is not there. Fix it with a project rule, an explicit end line, or the runner's own record — never by treating TEST PASSED as an end.
• Write each row as a decision with its reason, so changing it is a visible edit.
• Measured: three plausible rewrites each failed exactly one row. "Any $finish" passed every golden log and still failed a must-not-match row.
• Keep the table in a data file when non-programmers will add rows.
• Compare numbers as numbers. Measured: is failed a correct function on 0.12 vs 0.120 and 0 vs 0.000.
• A zero slack is defined and false. Test defined, never truth.
• For a new tool version: add the row, watch it fail, change the pattern on purpose, re-run every table. Measured: the new row failed with undef; after the change, all 7 slack rows and all 15 golden-log tests pass.

Chapter 27.6 moves from testing a parser to changing a script safely — how to restructure a working flow script without changing a single byte of what it produces, by capturing its output first and comparing after every step.