Perl Part II · Module 18 · Log Parsing and Text Processing
What Pattern Matching Cannot Do
Ten chapters of this module have been about getting a number out of a log, and patterns are the right instrument for that job. This last chapter draws the line on the other side. Asked to count the blocks in a module, find where one ends, or report how wide a signal is, a pattern does not fail loudly — it returns a number that looks reasonable and is wrong. The useful skill is recognising which kind of question you have been handed before you write the regex, because one kind has no regex answer at all, however clever.
Advanced19 min readPerlPart IILog Parsing
Part II · Module 18 · Chapter 18.11 · What Pattern Matching Cannot Do
1. Four Ways To Count A Keyword
A simple-sounding request: how many begin blocks are in this file? Here is a fragment with exactly five, surrounded by text that merely spells the word.
use strict;
use warnings;
# A SystemVerilog fragment with exactly FOUR real `begin` keywords.
# Every other spelling of "begin" is a decoy, and the decoys are chosen so
# that each method below fails on a different one.
my $src = <<'SV';
module arbiter (input logic clk, input logic rst_n, output logic grant);
localparam int begin_addr = 32'h1000; // an identifier containing the word
// the reset branch should begin here one day
/* an older version used to begin with a
two-state machine, kept for reference */
always_ff @(posedge clk) begin
if (!rst_n) begin
grant <= 1'b0;
end
else begin
grant <= 1'b1;
end
end
always_comb begin : named_block
if (grant) $display("note: see doc // section 2"); if (grant) begin
grant_dbg = grant;
end
end
SV
# note the line above: a real `begin` opens the block, and a LATER line holds
# a string containing "//" -- a regex comment-stripper deletes the rest of it.
$src .= "endmodule\n";
my $TRUTH = 5;
my $a = () = $src =~ /begin/g;
my $b = () = $src =~ /\bbegin\b/g;
my $stripped = $src;
$stripped =~ s{/\*.*?\*/}{}gs;
$stripped =~ s{//[^\n]*}{}g;
my $c = () = $stripped =~ /\bbegin\b/g;
sub scan {
my ($text) = @_;
my ($state, $out) = ("code", "");
my @ch = split //, $text;
for (my $i = 0; $i < @ch; $i++) {
my $two = ($ch[$i] // "") . ($ch[$i+1] // "");
if ($state eq "code") {
if ($two eq "//") { $state = "line_comment"; $i++ }
elsif ($two eq "/*") { $state = "block_comment"; $i++ }
elsif ($ch[$i] eq '"') { $state = "string" }
else { $out .= $ch[$i] }
}
elsif ($state eq "line_comment") { $state = "code" if $ch[$i] eq "\n" }
elsif ($state eq "block_comment") { if ($two eq "*/") { $state = "code"; $i++ } }
elsif ($state eq "string") { $state = "code" if $ch[$i] eq '"' }
}
my $n = () = $out =~ /\bbegin\b/g;
return $n;
}
my $d = scan($src);
printf "the fragment contains exactly %d real `begin` keywords\n\n", $TRUTH;
printf "%-40s %-7s %-11s %s\n", "METHOD", "COUNT", "ERROR", "verdict";
for my $m (["A /begin/ substring", $a],
["B /\\bbegin\\b/ word boundaries", $b],
["C regex comment-strip, then B", $c],
["D four-state character scanner", $d]) {
my ($label, $n) = @$m;
printf "%-40s %-7d %-11s %s\n", $label, $n,
($n == $TRUTH ? "-" : sprintf("%+d", $n - $TRUTH)),
($n == $TRUTH ? "correct" : $n > $TRUTH ? "WRONG (over)" : "WRONG (under)");
}
# ---- which decoy breaks which method, measured one decoy at a time
print "\nmeasured one hazard at a time (count of real keywords is 1 in each):\n";
my @CASES = (
[ "plain ", "always_comb begin\nend\n", 1 ],
[ "identifier begin_addr ", "int begin_addr;\nalways_comb begin\nend\n", 1 ],
[ "begin in a // comment ", "// should begin soon\nalways_comb begin\nend\n", 1 ],
[ "begin in a /* */ comment ", "/* used to begin here */\nalways_comb begin\nend\n", 1 ],
[ "begin in a \"string\" ", "\$display(\"begin\");\nalways_comb begin\nend\n", 1 ],
[ "real begin after \"//\" on ONE line", "\$display(\"see // doc\"); if (x) begin\nend\n", 1 ],
);
printf " %-32s %5s %5s %5s %5s\n", "HAZARD", "A", "B", "C", "D";
for my $case (@CASES) {
my ($label, $txt, $truth) = @$case;
my $ca = () = $txt =~ /begin/g;
my $cb = () = $txt =~ /\bbegin\b/g;
my $st = $txt; $st =~ s{/\*.*?\*/}{}gs; $st =~ s{//[^\n]*}{}g;
my $cc = () = $st =~ /\bbegin\b/g;
my $cd = scan($txt);
printf " %-32s %5s %5s %5s %5s\n", $label,
map { $_ == $truth ? "ok" : "BAD" } ($ca, $cb, $cc, $cd);
}the fragment contains exactly 5 real `begin` keywords
METHOD COUNT ERROR verdict
A /begin/ substring 8 +3 WRONG (over)
B /\bbegin\b/ word boundaries 7 +2 WRONG (over)
C regex comment-strip, then B 4 -1 WRONG (under)
D four-state character scanner 5 - correct
measured one hazard at a time (count of real keywords is 1 in each):
HAZARD A B C D
plain ok ok ok ok
identifier begin_addr BAD ok ok ok
begin in a // comment BAD BAD ok ok
begin in a /* */ comment BAD BAD ok ok
begin in a "string" BAD BAD BAD ok
real begin after "//" on ONE line ok ok BAD okFour methods, four different answers, and only one of them right.
A, the substring, counts 8. It matches begin_addr, and the word inside both comments, and the word inside the string.
B, with word boundaries, counts 7. \b fixes the identifier and nothing else — a comment is still text, and so is a string.
C, strip the comments first, counts 4 — too few. This is the interesting one, and section 2 explains it.
D, a character scanner, counts 5. Correct.
Read the second table, which runs each hazard on its own. Each method is defeated by a different one, so the sequence A → B → C → D is not "the same idea, written more carefully". Each step is a different idea, and the last one is not a pattern at all.
2. The Boundary
Figure 18.11.1 — Which instrument answers which question
The first row is where this module has lived. The second row is where Chapter 18.7's state machine already took you — and notice that method D in section 1 is that state machine, applied to source instead of a report. Four states, code, line_comment, block_comment and string, and the transitions between them.
That is worth saying plainly: the correct way to count a keyword in source is to write a scanner. Not a better regex. The ladder in section 1 does not converge on a clever pattern; it converges on parsing, and it gets there by row two.
3. Nesting Is Not A Pattern Problem
Rows three and below are a different kind of impossible. Not "hard to get right" — not expressible.
use strict;
use warnings;
# TWO sibling blocks, the first of which has a nested block inside it.
# Lines are numbered in the comment so the truth is checkable by eye.
my $rtl = <<'SV';
always_ff @(posedge clk) begin
if (!rst_n) begin
grant <= 1'b0;
end
end
always_comb begin
dbg = grant;
end
SV
# 1 always_ff ... begin <- the FIRST block opens
# 2 if (!rst_n) begin <- a nested block opens
# 3 grant <= 1'b0;
# 4 end <- the nested block closes
# 5 end <- the FIRST block closes (its body: lines 2-4)
# 6 always_comb begin <- a SECOND, sibling block
# 7 dbg = grant;
# 8 end
my $TRUTH = "lines 2-4";
sub extent { # describe a captured string as a line range
my ($cap) = @_;
return "no match" unless defined $cap;
my @l = grep { /\S/ } split /\n/, $cap;
return "no lines" unless @l;
my ($first, $last) = ($l[0], $l[-1]);
s/^\s+|\s+$//g for $first, $last;
return "'$first' .. '$last'";
}
my ($lazy) = $rtl =~ /begin(.*?)end/s;
my ($greedy) = $rtl =~ /begin(.*)end/s;
# the counter: track depth, which is what a parser does
my ($depth, $open_line, $close_line, $ln) = (0, 0, 0, 0);
for my $line (split /\n/, $rtl) {
$ln++;
my $o = () = $line =~ /\bbegin\b/g;
my $c = () = $line =~ /\bend\b/g;
$open_line ||= $ln if $o && $depth == 0;
$depth += $o - $c;
$close_line = $ln if $depth == 0 && $open_line && !$close_line;
}
print "truth: the first block's body is $TRUTH\n\n";
printf "%-32s %s\n", "A /begin(.*?)end/s lazy", extent($lazy);
printf "%-32s %s\n", "B /begin(.*)end/s greedy", extent($greedy);
printf "%-32s lines %d-%d\n", "C depth counter", $open_line + 1, $close_line - 1;
print "\nverdicts:\n";
printf " lazy captured only the NESTED block's body? %s\n",
($lazy =~ /grant <= 1'b0/ && $lazy !~ /dbg = grant/ && $lazy =~ /begin/
? "YES -- it stopped at the inner end" : "no");
printf " greedy ran past the first block into the 2nd? %s\n",
($greedy =~ /always_comb/ ? "YES -- it swallowed the sibling block" : "no");
printf " the depth counter got %s? %s\n", $TRUTH,
($open_line + 1 == 2 && $close_line - 1 == 4 ? "YES" : "no");
print "\nlazy and greedy are the ONLY two choices a regex offers.\n";
print "Neither is right here, and there is no third setting to try:\n";
print "balanced nesting to arbitrary depth is not a regular language.\n";
print "The counter is not a cleverer pattern -- it is a parser.\n";
# ---- part 2: a macro is expanded BEFORE the text you are matching exists
print "\n", "-" x 62, "\n";
my $src2 = <<'SV2';
`define WIDTH 8
module fifo (input logic clk);
logic [`WIDTH-1:0] data;
logic [7:0] shadow;
endmodule
SV2
my @eight = $src2 =~ /logic\s*\[7:0\]/g;
my @macro = $src2 =~ /logic\s*\[`(\w+)-1:0\]/g;
print "declarations that are 8 bits wide, in truth: 2\n";
printf "found by /logic\\s*\\[7:0\\]/: %d\n", scalar @eight;
printf "found only as a macro reference: %d (%s)\n",
scalar @macro, $macro[0];
print "a width question cannot be answered without expanding the macro,\n";
print "and expanding macros means resolving include files and their order.\n";
# ---- part 3: so what IS a pattern good for
print "\n", "-" x 62, "\n";
my $report = " slack (VIOLATED) -0.142\n slack (MET) 0.071\n";
my @viol = $report =~ /^\s+slack \(VIOLATED\)\s+(-?\d+\.\d+)/mg;
printf "extracting a known field from a known line shape: %s\n",
(@viol == 1 && $viol[0] == -0.142 ? "WORKS (-0.142)" : "FAILED");
print "one line, one shape, one field -- the job patterns do well.\n";truth: the first block's body is lines 2-4
A /begin(.*?)end/s lazy 'if (!rst_n) begin' .. 'grant <= 1'b0;'
B /begin(.*)end/s greedy 'if (!rst_n) begin' .. 'dbg = grant;'
C depth counter lines 2-4
verdicts:
lazy captured only the NESTED block's body? YES -- it stopped at the inner end
greedy ran past the first block into the 2nd? YES -- it swallowed the sibling block
the depth counter got lines 2-4? YES
lazy and greedy are the ONLY two choices a regex offers.
Neither is right here, and there is no third setting to try:
balanced nesting to arbitrary depth is not a regular language.
The counter is not a cleverer pattern -- it is a parser.
--------------------------------------------------------------
declarations that are 8 bits wide, in truth: 2
found by /logic\s*\[7:0\]/: 1
found only as a macro reference: 1 (WIDTH)
a width question cannot be answered without expanding the macro,
and expanding macros means resolving include files and their order.
--------------------------------------------------------------
extracting a known field from a known line shape: WORKS (-0.142)
one line, one shape, one field -- the job patterns do well.The task is to find the body of the first block, which has another block nested inside it and a sibling after it.
Lazy stops at the inner end. It finds the shortest match, which is the nested block's close.
Greedy swallows the sibling. It finds the longest match, which runs to the last end in the file — through the first block's close and into the next block entirely.
Those are the only two settings a regex has. Not "the two I tried": lazy and greedy are the complete set of choices about how much a quantifier consumes. Neither is the answer, and there is no third option to reach for, because matching balanced pairs to arbitrary depth is not something a regular expression can express at all. Perl's regex engine has recursion ((?R), (?1)) which can do it, and the moment you use that you have written a parser with unusually poor syntax.
The depth counter gets it right in five lines. It is not a cleverer pattern — it is a different instrument: it keeps state across lines, which is the one thing a pattern does not do.
4. A Macro Is Expanded Before Your Text Exists
The second measurement in the output above is the one that ends the argument.
Two declarations in that fragment are eight bits wide. A pattern for logic [7:0] finds one. The other is written logic [`WIDTH-1:0], and the text 7:0 does not appear anywhere in the file.
No regex fixes this, and neither does the scanner, because the information is not in the file:
`define lives in another file, reached through an `include that may itself be conditional.
The include path depends on the tool invocation — the +incdir+ arguments, which live in a filelist or a Makefile, not in the source.
Two configurations can give the same line two different widths. The question "how wide is data" has no answer that is a property of this file alone.
So a width question is not a parsing problem either. It is an elaboration problem: it needs the preprocessor, the parser, and the configuration that selects between them. That is three tools you are not going to write.
The same applies to every question of that family: is this identifier a net or a variable, is this instance in the final netlist, which always_ff drives this signal, does this parameter get overridden. All of them need the whole design resolved, and resolving it is what the EDA tool does.
5. So Ask The Tool
Here is the practical rule the module ends on.
When the question is about the design, do not parse the source. Get the tool to answer it and parse the answer.
Every tool in the flow can be made to emit a report, and a report is exactly what the previous ten chapters were about:
For structure, have the simulator or synthesis tool write a hierarchy or elaboration report. One line per instance, fixed columns, no nesting to track.
For widths and connectivity, read the elaborated netlist the tool emits, not the RTL it read.
For a list of what was compiled, take the tool's file list or dependency output rather than inferring it from `include lines.
For anything the tool has a dedicated query for, use that: most tools expose a Tcl or command-line query and will print exactly the fact you want, one line, one shape.
This is a better answer for three reasons that have nothing to do with difficulty:
It is authoritative. The tool's view of the design is the one that matters. Your parser's view, even if correct, is a second opinion.
It survives the language. A new SystemVerilog construct breaks your parser; the tool's report keeps the same columns.
It turns a parsing job into an extraction job — which is the job patterns do well, and which this module has now covered end to end.
The third measurement in the output above is that job, in one line: a slack value pulled out of a timing report by a pattern that names the line shape it expects. One line, one shape, one field. That is where a regex belongs, and the fact that it is unglamorous is the point.
6. What To Remember
- Extraction and parsing are different jobs. Patterns do the first one well and the second one not at all.
- Measured: four ways to count
begingive 8, 7, 4 and 5 — the truth is 5. - Each method is defeated by a different hazard, so the ladder is not "the same idea done more carefully".
- A substring match counts identifiers like
begin_addr;\bfixes only that. - A regex comment-stripper deletes real code after a string containing
//, so it under-counts. - A method that is accidentally right on your fixture is the most dangerous one — my first draft shipped C as correct because its fixture removed a decoy.
- The correct keyword count needs a four-state scanner: code, line comment, block comment, string. That is Chapter 18.7's state machine applied to source.
- The ladder converges on parsing, not on a cleverer pattern, and it gets there by the second rung.
- Measured: lazy stops at the inner
end; greedy swallows the sibling block. Those are the only two settings a regex has. - Balanced nesting to arbitrary depth is not expressible as a regular expression. Recursive patterns are a parser with worse syntax.
- A depth counter keeps state across lines — the one thing a pattern does not do.
- A real counter also needs
fork/join,case/endcase, named blocks — the slope ends at maintaining a SystemVerilog parser. - Measured: two declarations are 8 bits wide, a
[7:0]pattern finds one. The other says`WIDTH. - A width is not in the file. It needs the macro, the include path, and the configuration — elaboration, not parsing.
- When the question is about the design, read the tool's report, not the source. Authoritative, survives language changes, and turns parsing back into extraction.
- If you must search source, say so in the output — "lower bound, not a census".
That closes Module 18. You can take a tool log apart: scan it a line at a time, anchor a severity match so the log's own summary does not inflate it, pull values out with named captures, hold multi-line records, track sections with a state machine, accept several spellings of a field across tool versions, tell a truncated log from a clean one, and stream a file larger than memory — and you can tell when the question in front of you is not a text-processing question at all.
Where this fits
Part of the Perl curriculum.