Perl Part II · Module 18 · Log Parsing and Text Processing

Multi-Line Records and Section Parsing

Some facts in a tool report do not fit on one line. A timing path is a block: a startpoint on one line, an endpoint on the next, the clock group and the slack further down. A UVM message with a long payload wraps onto continuation lines. A line-at-a-time loop can count the pieces of these perfectly and still be unable to answer any useful question, because the pieces are only meaningful together. This chapter measures that gap, builds the accumulate loop that closes it, and ends on the bug that makes an accumulating parser silently lose exactly one record — the last one.

Advanced13 min readPerlPart IILog Parsing

Part II · Module 18 · Chapter 18.6 · Multi-Line Records and Section Parsing

1. Counting Is Not Associating

Azvya Education Pvt. Ltd.VLSI Mentor

accumulate.pl

use strict;
use warnings;

# a SAMPLE report in the shape an STA path report has: one block per path,
# blocks separated by a blank line, fields indented under their headers
my @log = split /\n/, <<'LOG';
Startpoint: u_alu/A (rising edge-triggered flip-flop clocked by clk_core)
Endpoint: u_fifo/D (rising edge-triggered flip-flop clocked by clk_core)
  Path Group: clk_core
  Path Type: max
  slack (VIOLATED)     -0.034

Startpoint: u_regf/Q (rising edge-triggered flip-flop clocked by clk_ddr)
Endpoint: u_pad/D (rising edge-triggered flip-flop clocked by clk_ddr)
  Path Group: clk_ddr
  Path Type: max
  slack (MET)           0.121
LOG

# ---- A: count the pieces, a line at a time
my ($starts, $slacks) = (0, 0);
for my $l (@log) {
    $starts++ if $l =~ /^Startpoint:/;
    $slacks++ if $l =~ /^\s+slack\s+\(/;
}
printf "A. counting:   %d startpoint(s), %d slack line(s)\n", $starts, $slacks;
print  "   both numbers are correct, and neither says which slack belongs\n";
print  "   to which path -- the only question a flow ever asks.\n\n";

# ---- B: accumulate a record until its terminator
#      NOTE the appended "" sentinel: without it the LAST record is never
#      pushed, because nothing after it closes the block. See section 4.
my (@rec, %cur);
for my $l (@log, "") {
    if ($l =~ /^Startpoint:\s*(\S+)/) {
        push @rec, { %cur } if %cur;          # a new header closes the previous
        %cur = (start => $1);
        next;
    }
    next unless %cur;                          # ignore anything before the first
    $cur{end}   = $1 if $l =~ /^Endpoint:\s*(\S+)/;
    $cur{group} = $1 if $l =~ /^\s+Path Group:\s*(\S+)/;
    if ($l =~ /^\s+slack\s+\((VIOLATED|MET)\)\s+(-?[\d.]+)/) {
        $cur{status} = $1;
        $cur{slack}  = $2;
    }
    if ($l eq "") {                            # a blank line closes the block
        push @rec, { %cur } if %cur;
        %cur = ();
    }
}

printf "B. accumulated %d complete record(s):\n", scalar @rec;
for my $r (@rec) {
    printf "   %-9s %-9s -> %-9s slack %-8s %s\n",
        $r->{group} // "?", $r->{start} // "?", $r->{end} // "?",
        $r->{slack} // "?", $r->{status} // "?";
}

my @viol = sort { $a->{slack} <=> $b->{slack} }
           grep { ($_->{status} // "") eq "VIOLATED" } @rec;
printf "\n   %d violating path(s); worst %s from %s to %s in %s\n",
    scalar @viol,
    (@viol ? $viol[0]{slack} : "-"), (@viol ? $viol[0]{start} : "-"),
    (@viol ? $viol[0]{end}   : "-"), (@viol ? $viol[0]{group} : "-");

Azvya Education Pvt. Ltd.VLSI Mentor

Captured output

A. counting:   2 startpoint(s), 2 slack line(s)
   both numbers are correct, and neither says which slack belongs
   to which path -- the only question a flow ever asks.

B. accumulated 2 complete record(s):
   clk_core  u_alu/A   -> u_fifo/D  slack -0.034   VIOLATED
   clk_ddr   u_regf/Q  -> u_pad/D   slack 0.121    MET

   1 violating path(s); worst -0.034 from u_alu/A to u_fifo/D in clk_core

Both of A's numbers are correct and neither is useful. Two startpoints, two slack lines — and no way to pair them. A flow never asks "how many lines mentioned slack"; it asks which path violated and by how much, and that question needs the pieces joined.

B produces two complete records and can therefore answer it: the worst violating path is -0.034 from u_alu/A to u_fifo/D in clk_core.

Note the sort on $a->{slack} <=> $b->{slack} — numeric, because Chapter 17.8 measured that a string sort puts -1.5 after -0.2.

2. The Accumulate Loop

The shape is four rules, and they are the same for every multi-line format:

A header line starts a record. ^Startpoint: here. When one arrives, the record in progress is finished — so push it before starting the new one. That single push ... if %cur handles a format with no explicit terminator at all.

Field lines add to the record in progress. Guarded by next unless %cur, so anything before the first header is ignored rather than accumulated into a phantom record.

A terminator closes it. A blank line here.

Push a copy, not the hash.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

push @rec, { %cur };          # a COPY
push @rec, \%cur;             # WRONG: every element is the same hash

\%cur pushes a reference to the one hash the loop keeps reusing, so after %cur = () every record in @rec is empty — or, if you forget the reset, every record is the last one. { %cur } builds a new anonymous hash from the current contents. Chapter 7.13 measured what gets shared when nested data is copied, and this is the simplest case of it.

Use a hash of arrays when a record has repeating fields, which an STA path block does — it has one line per element of the path.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

push @{ $cur{elements} }, { pin => $1, delay => $2 }
    if $l =~ /^\s+(\S+)\s+\(\S+\)\s+([\d.]+)/;

push @{ $cur{elements} }, ... autovivifies the array on first use — Chapter 7.9 — so there is no initialisation step, and a record that has no such lines simply has no elements key. Test with exists rather than assuming an empty array.

3. The Four Terminators

Formats end a record in one of four ways, and recognising which you have is the whole design.

A blank line. The commonest, and the easiest. $l eq "" after chomp, or $l =~ /^\s*$/ if trailing whitespace is possible — and it usually is.

The next header. No explicit terminator; a record ends when the next one begins. Handled by the push ... if %cur on the header branch, and this is why that line comes before %cur = (...).

An explicit end marker. --- end of path ---, or a line of dashes. The cleanest to parse, and the rarest.

A change of indentation. A record's fields are indented and the next unindented line ends it. This is why Chapter 18.3 warned against stripping leading whitespace: for this format the indentation is the terminator.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

if ($l =~ /^\S/ && %cur) { push @rec, { %cur }; %cur = () }

A format often has two of these and you should handle both, because a tool that normally separates with a blank line will omit it at the end of a section.

4. The Last Record, and Nesting

Two bugs account for most broken multi-line parsers.

The last record is never pushed. When the input ends, the record in progress is still in %cur and nothing closes it. A report with five paths yields four.

The measured loop avoids it with an explicit sentinel:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

for my $l (@log, "") { ... }        # an extra empty line at the end

Or, reading a file, by flushing after the loop:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

while (my $l = <$fh>) { ... }
push @rec, { %cur } if %cur;        # the flush

The sentinel is tidier because the closing logic exists in one place. Either works; omitting both is the bug.

And it is worth being precise about why this one survives review: it loses exactly one record, the last. In an STA report sorted worst-first, the last path is the least bad — so the parser drops the least important row and every total is slightly low. The output looks right, and the error is in the safe direction, which is the worst possible combination for being noticed. Chapter 14.1's third kind of failure again.

Nested records are not records. An STA path block contains per-element lines that look like fields. If you treat the element lines as record fields, the last one wins and you report one delay per path instead of a path's worth. If you treat them as records, you get a hundred "paths" with no startpoint.

The test for which you have is simple: does the inner thing repeat within one outer thing? If yes it is a list inside the record, per section 2's push @{ $cur{elements} }. If it never repeats, it is a field.

Resist the temptation to solve all of this by slurping the file and splitting on the record separator:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

local $/ = "";                       # paragraph mode
while (my $block = <$fh>) { ... }    # one record per read

Perl's paragraph mode genuinely works and is elegant for a small file. Two reasons it is the wrong default here:

It needs the terminator to be a blank line. Three of the four terminators above are not.

It holds a whole record in memory, and a record can be large. An STA report with a thousand-element path, or a UVM message with a packed payload dump, makes one "paragraph" of unbounded size. Chapter 18.10 is about the log that does not fit in memory, and paragraph mode is the first thing that breaks there.

The accumulate loop holds one record at a time by construction, and works for all four terminators. Use paragraph mode when you have measured that the records are small and the terminator is blank lines — not before.

5. What To Remember

Some facts span several lines: a timing path block, a wrapped UVM message, a sectioned report.
Measured: a counting parser reported 2 startpoints and 2 slack lines and could not pair them.
A flow never asks "how many lines" — it asks which path, and how bad.
Measured: accumulating gave 2 complete records and named the worst violating path.
Sort slack numerically — <=>, per Chapter 17.8's string-sort measurement.
Four rules: a header starts a record, field lines add to it, a terminator closes it, push a copy.
Push { %cur }, never \%cur — a reference means every record is the same hash.
Guard field lines with next unless %cur, so nothing before the first header accumulates.
Use push @{ $cur{list} } for repeating fields — autovivification means no initialisation.
Four terminators: a blank line, the next header, an explicit marker, a change of indentation.
Indentation as a terminator is why 18.3 said not to strip leading whitespace.
Handle two terminators when a format has them, because the last section often omits the blank line.
The last record is never pushed unless you append a sentinel or flush after the loop.
That bug loses exactly one record — the last — which in a worst-first report is the least bad one, so the output looks right and totals are slightly low.
Nested repeating lines are a list inside the record, not records. Test whether the inner thing repeats within one outer thing.
Paragraph mode (local $/ = "") needs a blank-line terminator and holds a whole record in memory — fine for small records, and the first thing to break in Chapter 18.10.

Chapter 18.7 handles the report where a pattern is only valid in one part of the file. Chapter 18.1 measured a summary line being counted as a message; the fix is an explicit state variable, and once there are more than two states it is worth drawing.

Where this fits

Part of the Perl curriculum.