Perl Part II · Module 18 · Log Parsing and Text Processing

The Line-by-Line Scan Pattern

Every parser in the rest of this curriculum is the same loop with a different pattern in the middle, so it is worth assembling that loop once and carefully. It has seven steps and none of them is difficult — the reason this chapter is rated Critical is that six of the seven are easy to leave out, and leaving one out does not produce an error. It produces a number. This chapter measures the two most common omissions, and both of them make a broken input report zero errors and a clean pass, which is the worst answer a flow script can give.

Foundation13 min readPerlPart IILog Parsing

Part II · Module 18 · Chapter 18.2 · The Line-by-Line Scan Pattern

1. The Seven Steps

Figure 18.2.1 — The scan every parser reuses

Seven steps in a fixed order. The two checks at the start are what separate a real answer from a plausible one: without them a file that cannot be opened and a file with nothing in it both fall straight through the loop and report zero matches. The three steps in the middle are the parsing, and the two at the end are what make the result usable by a flow rather than only readable by a person.

Steps 3 to 6 are the part people think of as the parser. Steps 1, 2 and 7 are the part that decides whether the answer can be trusted, and they are the ones that get left out.

2. What Each Omission Costs

Azvya Education Pvt. Ltd.VLSI Mentor

seven-steps.pl

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;

my $dir = tempdir(CLEANUP => 1);
my $p   = sub { File::Spec->catfile($dir, shift) };

open my $w, ">", $p->("good.log") or die "cannot write: $!\n";
print $w "UVM_ERROR a.sv(1) @ 10: one\nUVM_ERROR b.sv(2) @ 20: two\n";
close $w;
open my $e, ">", $p->("empty.log") or die "cannot write: $!\n";
close $e;                                     # deliberately empty

my $PAT = qr/^UVM_ERROR\s+\S+\(\d+\)/;        # the bounded pattern from 18.4

# ---- the full seven-step scan
sub scan_full {
    my ($f) = @_;
    open my $fh, "<", $f or return (undef, "cannot read: $!");       # 1
    my ($n, $lines) = (0, 0);
    while (my $l = <$fh>) { $lines++; chomp $l; $n++ if $l =~ $PAT } # 3,4,5,6
    close $fh;
    return (undef, "empty log — nothing was scanned") unless $lines; # 2
    return ($n, "ok");
}

# ---- omit step 1: the open is not checked
sub no_checked_open {
    my ($f) = @_;
    open my $fh, "<", $f;                     # no "or"
    my $n = 0;
    { no warnings;                            # a flow discards stderr anyway
      while (my $l = <$fh>) { chomp $l; $n++ if $l =~ $PAT } }
    return ($n, "ok");
}

# ---- omit step 2: no empty-input guard
sub no_empty_guard {
    my ($f) = @_;
    open my $fh, "<", $f or return (undef, "cannot read: $!");
    my $n = 0;
    while (my $l = <$fh>) { chomp $l; $n++ if $l =~ $PAT }
    close $fh;
    return ($n, "ok");
}

printf "%-18s %-12s %-7s %s\n", "SCAN", "INPUT", "COUNT", "VERDICT";
for my $f (qw(good.log empty.log missing.log)) {
    for my $pair (["full seven steps", \&scan_full],
                  ["no checked open",  \&no_checked_open],
                  ["no empty guard",   \&no_empty_guard]) {
        my ($n, $why) = $pair->[1]->($p->($f));
        $why =~ s/: .*/: <reason>/ if $why =~ /cannot read/;   # keep it stable
        printf "%-18s %-12s %-7s %s\n", $pair->[0], $f,
            (defined $n ? $n : "-"), $why;
    }
    print "\n";
}
print "two different ways to report a clean pass on broken input\n";

Azvya Education Pvt. Ltd.VLSI Mentor

Captured output

SCAN               INPUT        COUNT   VERDICT
full seven steps   good.log     2       ok
no checked open    good.log     2       ok
no empty guard     good.log     2       ok

full seven steps   empty.log    -       empty log — nothing was scanned
no checked open    empty.log    0       ok
no empty guard     empty.log    0       ok

full seven steps   missing.log  -       cannot read: <reason>
no checked open    missing.log  0       ok
no empty guard     missing.log  -       cannot read: <reason>

two different ways to report a clean pass on broken input

Read the missing.log and empty.log blocks.

With no checked open, a log that does not exist reports 0 and ok. The open failed, the filehandle is unusable, the while loop ran zero times, and the count is zero. Zero errors is a pass. A regression step whose log was never written reports success.

With no empty guard, a log with nothing in it reports 0 and ok. This is the case where the tool started, wrote nothing, and died — or where the disk filled. Again: zero errors, pass.

The full scan distinguishes all three. A count for the good log, and a distinct refusal for each broken input.

Perl does emit readline() on closed filehandle for the first case. That warning goes to standard error, which a flow routinely discards — Chapter 14.2 measured that script.pl > log does not capture it at all. So the only thing the flow sees is 0, and 0 means pass.

The general shape of this failure is worth naming, because it recurs in every parsing chapter that follows.

A parser that cannot read its input must not return a count. Zero is a measurement, and it is only available when something was measured. When nothing was measured the answer is not zero — it is "unknown", and Chapter 15.4 gave it its own exit status for exactly this reason.

So a scan has three possible outcomes, not two:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

my ($count, $why) = scan($log);
if (!defined $count) {
    print STDERR "cannot scan $log: $why\n";
    exit 3;                        # could not reach a verdict
}
exit($count ? 1 : 0);              # a real verdict: found, or clean

Returning undef rather than 0 for "could not tell" is the whole discipline. Chapter 18.9 takes the four distinct bad inputs apart one at a time.

3. The Reference Scan

This is the loop to copy. Every parser in the rest of Part II is this with a different step 5 and 6.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

sub scan_log {
    my ($path) = @_;

    open my $fh, "<", $path                              # 1
        or return { ok => 0, why => "cannot read $path: $!" };

    my ($lines, @found) = (0);
    while (my $line = <$fh>) {                           # 3
        $lines++;
        chomp $line;                                     # 4
        next unless $line =~ $PATTERN;                   # 5
        push @found, { line => $lines, text => $line };  # 6
    }
    close $fh
        or return { ok => 0, why => "read error on $path: $!" };

    return { ok => 0, why => "empty log: $path" } unless $lines;   # 2

    return { ok => 1, lines => $lines, found => \@found };         # 7
}

Six details in those lines are deliberate.

open ... or return, with $! in the message. Not or die — a function that dies cannot be reused by a caller who wants to scan twenty logs and report on all of them. Chapter 14.9's argument about who decides.

The three-argument open. Chapter 13.1: a filename beginning with > or containing a leading space is a filename, not a mode.

chomp before matching. Without it the last field of every line ends in a newline, so $line =~ /(\S+)\z/ captures text with \n attached and a later eq comparison fails. Chapter 13.2 measured this.

next unless rather than if. The guard clause keeps the body shallow, and in a loop with four conditions that is the difference between readable and not.

close is checked. On a read this catches an I/O error mid-file — rare, and the one case where a truncated read looks like a short file. Chapter 13.5.

The empty guard comes after the loop, because you need the line count to know. Putting a -s $path test before the open is also fine and catches it earlier; doing both is better, since a file can be non-empty and still yield no lines through a read error.

Compile the pattern once. A pattern written as a string and interpolated inside the loop is recompiled on every line.

Azvya Education Pvt. Ltd.VLSI Mentor

compile-once.pl

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;

my $dir = tempdir(CLEANUP => 1);
my $log = File::Spec->catfile($dir, "s.log");
open my $w, ">", $log or die "cannot write: $!\n";
print $w "UVM_ERROR f.sv(1) \@ 10: one\n"
       . "XUVM_ERRORY f.sv(2) \@ 20: two\n"
       . "UVM.ERROR f.sv(3) \@ 30: three\n";
close $w;

# A severity name read from a config file could contain a metacharacter.
# Interpolating it raw turns that character into pattern syntax.
printf "%-12s %-26s %s\n", "SEVERITY", "raw /^\$sev\\s/ matches", "\\Q quoted matches";
for my $sev ("UVM_ERROR", "UVM.ERROR") {
    my $raw   = qr/^$sev\s/;
    my $safe  = qr/^\Q$sev\E\s/;
    open my $fh, "<", $log or die "cannot read: $!\n";
    my (@r, @q);
    while (my $l = <$fh>) {
        chomp $l;
        push @r, $. if $l =~ $raw;
        push @q, $. if $l =~ $safe;
    }
    close $fh;
    printf "%-12s %-26s %s\n", "'$sev'",
        (@r ? "lines " . join(",", @r) : "none"),
        (@q ? "lines " . join(",", @q) : "none");
}
print "\nline 1 is UVM_ERROR, line 2 is XUVM_ERRORY, line 3 is UVM.ERROR\n";

Azvya Education Pvt. Ltd.VLSI Mentor

Captured output

SEVERITY     raw /^$sev\s/ matches      \Q quoted matches
'UVM_ERROR'  lines 1                    lines 1
'UVM.ERROR'  lines 1,3                  lines 3

line 1 is UVM_ERROR, line 2 is XUVM_ERRORY, line 3 is UVM.ERROR

With the severity UVM.ERROR, the raw interpolation matched lines 1 and 3 — the . became "any character", so the pattern also matched the UVM_ERROR line. The \Q form matched line 3 only.

That is the reason to compile with qr// and \Q, and it is not speed. I measured the speed too and did not publish it: the difference between an interpolated pattern and a pre-compiled one was not reliably detectable across runs, because Perl already caches an interpolated pattern when the interpolated value has not changed. A number I cannot reproduce is not a finding — Chapter 16.1 made the same call about interpreter startup.

The three reasons that do hold:

It names the pattern. $RX at the top of the file is documentation; a regex buried in a loop condition is not.

\Q...\E makes the interpolation safe. $sev here is a plain word, but a severity read from a configuration file could contain a . or a +, and without \Q that becomes a pattern. Chapter 11.4's metacharacters.

It is reusable. The same qr// can be handed to a function, stored in a table of patterns per tool version — which is exactly what Chapter 18.8 needs. Chapter 11.18 covers compiled patterns in full.

Report to standard error, data to standard output. The second refinement, and Chapter 15.12 measured why: a progress line printed to standard output gets sorted in with the data by the next command in the pipeline.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

printf STDERR "scanned %d lines of %s\n", $r->{lines}, $path;      # message
printf "%s\t%d\n", $_->{text}, $_->{line} for @{ $r->{found} };    # data

5. What To Remember

Every parser in Part II is this one loop with a different match and accumulate step.
Seven steps: checked open, empty guard, line loop, chomp, bounded match, accumulate, report.
Steps 1, 2 and 7 are the ones that get left out, and they are what make the answer trustworthy.
Measured: with no checked open, a MISSING log reports 0 and ok.
Measured: with no empty guard, an EMPTY log reports 0 and ok.
Zero errors is a pass, so both omissions turn a broken run into a green one.
Perl's readline() on closed filehandle warning goes to stderr, which a flow discards — the flow sees only the 0.
A parser that cannot read its input must return undef, not 0. Zero is a measurement and requires that something was measured.
Three outcomes, not two: a count, a clean verdict, and "could not tell" with its own exit status.
open ... or return, not or die — a dying function cannot scan twenty logs and report on all of them.
chomp before matching, or a captured last field carries a newline.
Check close on a read — it catches an I/O error that otherwise looks like a short file.
Put the empty guard after the loop (you need the count), and a -s test before the open as well.
Return a hash reference once there is more than one result; a mis-ordered list assignment gives no error.
Compile the pattern with qr// and \Q outside the loop. Measured: with a severity of UVM.ERROR, raw interpolation matched lines 1 and 3 because . became any-character, where \Q matched line 3 only.
The reason is not speed. I measured that too and did not publish it — Perl already caches an interpolated pattern whose value has not changed, so the difference was not reproducible.
Messages to standard error, data to standard output.

Chapter 18.3 handles the step that has to happen before any of this works on real input. The same message from three different simulators does not look the same, and an anchored pattern matches only one of the three.

Where this fits

Part of the Perl curriculum.

1. The Seven Steps

2. What Each Omission Costs

3. The Reference Scan

4. Two Refinements That Matter at Scale

5. What To Remember

Where this fits