Scanning a String Match by Match: pos and \G

Part I · Module 11 · Chapter 11.20 · Scanning a String Match by Match: pos and \G

1. Reading the Bookmark

use strict;
use warnings;

my $s = "u_alu/A u_fifo/B u_regf/C";

while ($s =~ m{(\w+)/(\w+)}g) {
    printf "  %-9s at pos %2d\n", "$1/$2", pos($s);
}
printf "after the loop : pos = %s\n", (defined pos($s) ? pos($s) : "undef");

  u_alu/A   at pos  7
  u_fifo/B  at pos 16
  u_regf/C  at pos 25
after the loop : pos = undef

Three matches at positions 7, 16 and 25 — each the offset just past the end of that match. Then undef, because the failing match that ended the loop reset the bookmark, which is Chapter 11.12's finding.

Note what that reset costs you here: you cannot read pos after the loop to find out how far the scan got. It is always undef at that point. Section four shows the way round it.

2. Writing the Bookmark, and What \G Adds

pos is not read-only.

use strict;
use warnings;

my $s = "aaa BBB ccc";

# pos() is writable: start the scan part-way in
pos($s) = 4;
if ($s =~ /\G(\w+)/g) {
    printf "starting at pos 4, \\G captured : %s\n", $1;
}

# Without \G the pattern is free to search forward from pos
pos($s) = 4;
if ($s =~ /(c\w+)/g) {
    printf "without \\G, searched forward  : %s\n", $1;
}

starting at pos 4, \G captured : BBB
without \G, searched forward  : ccc

Setting pos($s) = 4 moved the bookmark to just before BBB, and \G(\w+) captured BBB — the match was pinned to the bookmark.

The second half is the contrast that defines \G. The bookmark was again set to 4, but the pattern has no \G, so the engine was free to search forward from there and found ccc at the end of the string. Without \G, the bookmark is a starting point for a search; with \G, it is a requirement.

That distinction is the whole chapter. \G means "the match must begin exactly here", which is what a parser needs and a searcher does not.

3. A Sequential Parse

use strict;
use warnings;

# \G anchors the match to exactly where the last one stopped,
# which is what makes a sequential parse possible.
my $record = "NAME=u_alu;TYPE=cell;PINS=3;";

my %field;
my $reached = 0;
while ($record =~ /\G(\w+)=([^;]*);/g) {
    $field{$1} = $2;
    $reached = pos($record);     # capture it INSIDE the loop
}

for my $k (sort keys %field) {
    printf "  %-5s => %s\n", $k, $field{$k};
}

printf "\nconsumed %d of %d characters\n", $reached, length $record;
printf "fully consumed : %s\n", ($reached == length $record ? "yes" : "NO");
printf "pos after loop : %s   <- the failing match reset it\n",
    (defined pos($record) ? pos($record) : "undef");

  NAME  => u_alu
  PINS  => 3
  TYPE  => cell

consumed 28 of 28 characters
fully consumed : yes
pos after loop : undef   <- the failing match reset it

Three fields parsed, and then the part that makes this a parse rather than a search: 28 of 28 characters consumed, so the record was fully understood.

The $reached = pos($record) inside the loop body is there because of section one. My first version of this script printed pos after the loop and got undef; a completeness check has to capture the position while the loop is still running. The last line of output shows the undef for comparison, so the reason is visible rather than asserted.

That completeness test is the deliverable. $reached == length $record is a real verification that the whole record matched the expected grammar, in order, with nothing left over. No field-by-field search can tell you that.

4. Why It Matters: \G Notices Corruption

use strict;
use warnings;

# The point of \G: it REFUSES to skip over junk.
my $good = "A=1;B=2;";
my $bad  = "A=1;??;B=2;";

for my $rec ($good, $bad) {
    my @got;
    while ($rec =~ /\G(\w+)=(\w+);/g) { push @got, "$1=$2" }
    printf "%-12s with \\G : %d field(s)  %s\n",
        $rec, scalar @got, (@got ? join(",", @got) : "(none)");

    my @loose = $rec =~ /(\w+)=(\w+);/g;
    printf "%-12s no  \\G : %d item(s)   %s\n",
        "", scalar @loose, join(",", @loose);
}

A=1;B=2;     with \G : 2 field(s)  A=1,B=2
             no  \G : 4 item(s)   A,1,B,2
A=1;??;B=2;  with \G : 1 field(s)  A=1
             no  \G : 4 item(s)   A,1,B,2

Read the two records against each other, because this is the measurement the chapter is for.

On the clean record, the \G parse found two fields and the plain /g scan found four items — the same information in the flat-list shape Chapter 11.10 measured. Both work.

On the corrupt record, with ??; wedged in the middle, the two diverge completely. The \G parse found one field and stopped, because the junk is not a name=value; and \G would not let the engine step over it. The plain /g scan found four items — exactly what it found for the clean record. It skipped the corruption silently and reported both fields as though nothing were wrong.

The practical form combines this with section three: parse with \G, then compare the position you reached against the length. If they differ, you know both that the record was malformed and where, which is the makings of a usable error message.

5. When To Use This

The curriculum rates this chapter Low, and that is the right rating. It is worth being plain about why rather than overselling a construct because it is clever.

Most of the text you will parse in this field is line-oriented. A report has one row per line, a constraint file has one command per line, a log has one event per line. For all of that, reading a line at a time and applying a pattern to it is clearer, easier to debug, and gives you an obvious place to put a per-line error message. split is often better still. Chapter 11.22 makes that argument properly.

\G earns its place on one specific job: a single string that is a sequence, where the order and the completeness matter. A packed record like the one in section three. An attribute string where fields must appear in a fixed order. A small embedded format inside one field of a larger line. In those cases nothing else gives you the completeness check.

It is also worth knowing because it explains behaviour you might otherwise find baffling — in particular that pos survives an early last out of a /g loop, so a second loop over the same variable resumes mid-string. That is Chapter 11.12's warning, and pos($s) = 0 is the fix.

6. Reporting Where the Parse Stopped

Combine the completeness check with substr and the parser can say where the input went wrong, which is the difference between a usable error and a shrug.

use strict;
use warnings;

sub parse {
    my ($rec) = @_;
    my %f;
    my $reached = 0;
    while ($rec =~ /\G(\w+)=([^;]*);/g) {
        $f{$1} = $2;
        $reached = pos($rec);
    }
    return (\%f, $reached);
}

for my $rec ("A=1;B=2;C=3;", "A=1;B=2;??;C=3;") {
    my ($f, $reached) = parse($rec);
    my $len = length $rec;
    printf "record : %s\n", $rec;
    printf "  fields   : %d (%s)\n", scalar keys %$f,
        join(",", map { "$_=$f->{$_}" } sort keys %$f);
    printf "  consumed : %d of %d\n", $reached, $len;
    if ($reached != $len) {
        printf "  STOPPED AT: [%s]\n", substr($rec, $reached, 6);
    }
    else {
        print  "  complete\n";
    }
}

record : A=1;B=2;C=3;
  fields   : 3 (A=1,B=2,C=3)
  consumed : 12 of 12
  complete
record : A=1;B=2;??;C=3;
  fields   : 2 (A=1,B=2)
  consumed : 8 of 15
  STOPPED AT: [??;C=3]

The clean record reports three fields and complete. The corrupt one reports two fields, consumed 8 of 15, and STOPPED AT: [??;C=3] — the exact text where the grammar broke.

That last line is the whole argument for this chapter. A parser that knows its position can quote the offending input, and quoting the offending input is what turns a bug report into a fix. Note also that the corrupt record's C=3 is genuinely lost rather than silently collected — which is correct, because once the structure is broken the parser has no basis for trusting what follows it.

The substr($rec, $reached, 6) is a deliberately short window. In real use you would want the line number too, which means parsing per line and tracking it yourself — a reminder that \G is a tool for one string, and the surrounding bookkeeping is still yours.

7. What To Remember

• pos($str) reports the /g bookmark, the offset just past the last match.
• You cannot read pos after a /g loop — the failing match that ends the loop resets it to undef. Capture it inside the loop body.
• pos is writable, so a scan can start part-way into a string.
• Without \G the bookmark is a starting point for a search; with \G it is a requirement. Measured: from pos = 4, \G(\w+) took BBB while (c\w+) searched forward and took ccc.
• \G plus a position check is a real parse. Comparing the position reached against length verifies the whole record matched, in order, with nothing left over.
• A plain /g scan cannot detect corruption. On a record with junk in the middle it returned the same four items as the clean record; the \G parse stopped at the junk and reported one field.
• It is rated Low honestly. Line-oriented input is better served by a per-line pattern or split. \G is for a single string that is a sequence whose completeness matters.
• pos($s) = 0 resets a bookmark left behind by an early last.

Chapter 11.21 turns to what to do when a pattern does not work. There are four common reasons, they need different diagnoses, and the chapter builds a procedure for telling them apart — including the one that no amount of staring at the pattern will reveal, where the input contains a tab and the pattern expects spaces.