Lookahead and Lookbehind

Part I · Module 11 · Chapter 11.19 · Lookahead and Lookbehind

1. Looking Ahead

(?=...) succeeds if what follows matches, and consumes nothing.

use strict;
use warnings;

my @subjects = ("0.42 ns", "0.42 ps", "0.42");

# (?=...) asserts what FOLLOWS without consuming it
printf "%-10s %-14s %s\n", "SUBJECT", "\\d+(?=\\s*ns)", "captured";
for my $s (@subjects) {
    if (my ($n) = $s =~ /([\d.]+)(?=\s*ns)/) {
        printf "%-10s %-14s %s\n", $s, "match", $n;
    }
    else {
        printf "%-10s %-14s -\n", $s, "-";
    }
}

SUBJECT    \d+(?=\s*ns)   captured
0.42 ns    match          0.42
0.42 ps    -              -
0.42       -              -

The pattern takes a number only when nanoseconds follow it. 0.42 ps has a perfectly good number and was rejected; so was the bare 0.42 with no unit.

2. Zero-Width Is the Whole Point

use strict;
use warnings;

my $s = "0.42 ns";

# The lookahead is NOT part of the match
if ($s =~ /([\d.]+)(?=\s*ns)/) {
    printf "whole match \$& : [%s]   <- 'ns' is not in it\n", $&;
    printf "capture \$1      : [%s]\n", $1;
}

# Compare with simply including it
if ($s =~ /([\d.]+)\s*ns/) {
    printf "including it    : \$& = [%s]\n", $&;
}

whole match $& : [0.42]   <- 'ns' is not in it
capture $1      : [0.42]
including it    : $& = [0.42 ns]

$& — the whole match from Chapter 11.10 — is 0.42. The ns constrained the match and did not become part of it. Written the ordinary way, $& is 0.42 ns.

Like ^, $ and \b from Chapter 11.8, a lookaround is a zero-width assertion: it matches a position, by asking a question about what is there, and advances nothing.

That property is what makes it useful in exactly two situations. The first is substitution: s/([\d.]+)(?=\s*ns)/<$1>/ wraps the number and leaves the unit alone, because the unit was never matched and so cannot be replaced. Written with the unit inside the match you would have to put it back by hand in the replacement. The second is when the context must be available to a later part of the pattern — a lookahead can inspect text that a subsequent component will then match properly.

3. Looking Ahead Negatively

(?!...) succeeds when what follows does not match.

use strict;
use warnings;

my @pins = ("u_alu/A", "u_alu/A_reg", "u_alu/A_bar", "u_fifo/B");

# (?!...) asserts what does NOT follow
printf "%-14s %s\n", "PIN", "/u_alu\\/A(?!_)/";
for my $p (@pins) {
    printf "%-14s %s\n", $p, ($p =~ m{u_alu/A(?!_)} ? "match" : "-");
}

PIN            /u_alu\/A(?!_)/
u_alu/A        match
u_alu/A_reg    -
u_alu/A_bar    -
u_fifo/B       -

Only the bare u_alu/A matched. The two suffixed names were rejected because an underscore follows the A.

This is the form that is hardest to replace with a capture group, and therefore the one most worth knowing. "Match this, except where it is followed by that" has no positive equivalent — you cannot capture the absence of something.

Compare it with the alternative from Chapter 11.8. m{\bu_alu/A\b} would also reject u_alu/A_reg, because the underscore is a word character so there is no boundary. The difference is specificity: \b rejects any word character after the A, while (?!_) rejects only an underscore and would accept u_alu/Ax. Which you want depends on your naming convention, and the lookahead is the one that lets you say precisely which suffix separator you mean.

4. Looking Behind

(?<=...) asserts what precedes the match.

use strict;
use warnings;

my @subjects = ("slack=-0.12", "setup=-0.12", "slack=0.44");

# (?<=...) asserts what PRECEDES, without consuming it
printf "%-14s %s\n", "SUBJECT", "(?<=slack=)(\\S+)";
for my $s (@subjects) {
    if (my ($v) = $s =~ /(?<=slack=)(\S+)/) {
        printf "%-14s %s\n", $s, $v;
    }
    else {
        printf "%-14s -\n", $s;
    }
}

SUBJECT        (?<=slack=)(\S+)
slack=-0.12    -0.12
setup=-0.12    -
slack=0.44     0.44

The two slack= lines gave up their values; the setup= line was refused.

Again a capture group would do: /slack=(\S+)/ captures the same thing. The lookbehind version's advantage is that $& contains only the value, which matters for a substitution that must not disturb the key.

5. Looking Behind Negatively

use strict;
use warnings;

my @rows = ("value 42", "novalue 42", "myvalue 42");

# (?<!...) asserts what does NOT precede
printf "%-14s %s\n", "ROW", "(?<!no)value";
for my $r (@rows) {
    printf "%-14s %s\n", $r, ($r =~ /(?<!no)value/ ? "match" : "-");
}

ROW            (?<!no)value
value 42       match
novalue 42     -
myvalue 42     match

value matched, novalue was excluded — and myvalue matched.

That third row is the one to think about. (?<!no) excludes exactly the two characters no immediately before the match. The characters before value in myvalue are my, which is not no, so the assertion succeeds.

A negative lookbehind is therefore not a "this must be a whole word" test, even though it is often reached for as one. If what you mean is "the word value, not part of a longer word", the tool is \bvalue\b from Chapter 11.8, which rejects both novalue and myvalue. Use the negative lookbehind when you mean one specific prefix is forbidden.

6. The Width Restriction, Measured

Most references state that a lookbehind must be a fixed width. On this Perl the behaviour is more layered than that, and the measurement is worth having because the error message is confusing.

use strict;
use warnings;

# A lookbehind must be a FIXED width in Perl 5.34. A variable-width one
# is a compile error, so it has to be built at run time to be caught.
for my $src ('(?<=ab)c', '(?<=a|ab)c', '(?<=a+)c') {
    my $ok = eval { my $re = qr/$src/; "ok" } ;
    my $err = $@;
    $err =~ s/\s+at .*//s if $err;
    printf "%-12s : %s\n", $src, ($ok ? "compiled" : "REJECTED - $err");
}

(?<=ab)c     : compiled
(?<=a|ab)c   : compiled
(?<=a+)c     : REJECTED - Lookbehind longer than 255 not implemented in regex m/(?<=a+)c/

Three cases, three different outcomes.

(?<=ab)c — fixed width, two characters. Compiles, as expected.

(?<=a|ab)c — variable width, bounded. It compiles, which contradicts the simple rule. It is not silent though: on stderr, at compile time, Perl says Variable length lookbehind is experimental in regex. Compiling is not the same as being safe to depend on — an experimental feature can change between Perl versions, so this is a thing to know exists rather than a thing to build a production parser on.

(?<=a+)c — variable width, unbounded. Rejected, with:

Lookbehind longer than 255 not implemented in regex m/(?<=a+)c/

That message is worth quoting because it does not say what you would expect. It mentions 255 rather than variability, because a+ has no upper bound and the engine's limit is 255 characters. Someone meeting that error while debugging a pattern with no numbers in it will not immediately connect it to their +.

Note how the script had to be written to catch any of this. These are compile-time errors, so a literal bad pattern in the source would stop the program before it printed anything — which is why the patterns are built from strings at run time inside an eval. That is a useful technique in its own right for probing what the engine will accept, and Chapter 11.21 uses the same idea.

The practical rule: keep a lookbehind a fixed width. Use a plain alternation of equal-length pieces if you need a choice, and if you genuinely need variable-length context, restructure to use a capture group and check the captured text in code instead. Lookahead has no such restriction — (?=a+) is fine — so an asymmetric-looking pattern is sometimes the honest consequence of this limitation.

7. What To Remember

• A lookaround asserts something about the surrounding text and consumes nothing. $& after ([\d.]+)(?=\s*ns) is 0.42, not 0.42 ns.
• Four forms: (?=...) ahead, (?!...) not ahead, (?<=...) behind, (?<!...) not behind.
• Prefer a capture group where one works. /([\d.]+)\s*ns/ accepts the same lines as the lookahead version and reads better — which is why this chapter is rated Medium.
• Lookaround earns its place in two cases: a substitution that must not replace the context, and a pattern whose later part needs to match the text the assertion inspected.
• Negative lookahead has no positive equivalent — "except where followed by" cannot be written as a capture.
• A negative lookbehind excludes one exact prefix, not a word boundary. (?<!no)value still matched myvalue; \bvalue\b is the word test.
• Lookbehind width is restricted, in layers. Fixed width compiles; bounded variable width compiles but warns Variable length lookbehind is experimental; unbounded is refused with Lookbehind longer than 255 not implemented, a message that never mentions variability.
• Build patterns from strings inside eval when you want to find out what the engine accepts, since these are compile-time failures.

Chapter 11.20 returns to the bookmark that Chapter 11.12 introduced. pos can be read and written, and \G anchors a match to exactly where the last one stopped — which turns /g from a search into a sequential parse that notices when the input stops making sense.