Anchors and Word Boundaries

Part I · Module 11 · Chapter 11.8 · Anchors and Word Boundaries

1. The Wrong Count, Measured

Start with the bug, on a log extract of the kind any flow produces.

use strict;
use warnings;

my @lines = (
    "ERROR: setup violation",
    "NO_ERROR",
    "WARNING: unconstrained",
    "ERROR: hold violation",
    "ERRORS_TOTAL 2",
    "fixed_the_error",
);

my $loose  = grep { /ERROR/     } @lines;
my $word   = grep { /\bERROR\b/ } @lines;
my $start  = grep { /^ERROR\b/  } @lines;

printf "lines            : %d\n", scalar @lines;
printf "/ERROR/          : %d\n", $loose;
printf "/\\bERROR\\b/      : %d\n", $word;
printf "/^ERROR\\b/       : %d\n", $start;
printf "\nreal error lines : 2\n";

lines            : 6
/ERROR/          : 4
/\bERROR\b/      : 2
/^ERROR\b/       : 2

real error lines : 2

Two lines in that list are errors. The plain pattern reported four — double the truth.

The two false positives are NO_ERROR, which is a line saying explicitly that there was no error, and ERRORS_TOTAL 2, which is a summary header. Both contain the letters ERROR, and a pattern with no anchors asks only whether the letters are present somewhere. The sixth line, fixed_the_error, was not counted, but only because it is lowercase — add /i to that loose pattern and the count would reach five.

What makes this expensive is that 4 is a believable number. Nothing crashes, nothing warns, and a report that says "4 errors" looks like a report. The failure is detectable only by someone who already knows the right answer.

Both corrected patterns report 2. The rest of the chapter is about which one to reach for and why.

2. ^ and $: Tying a Match to the Ends

use strict;
use warnings;

my @subjects = ("ERROR", "ERROR: bad", "pre-ERROR", "x ERROR y");

printf "%-13s %-9s %-10s %-10s %s\n",
    "SUBJECT", "/ERROR/", "/^ERROR/", "/ERROR\$/", "/^ERROR\$/";
for my $s (@subjects) {
    printf "%-13s %-9s %-10s %-10s %s\n", $s,
        ($s =~ /ERROR/   ? "yes" : "-"),
        ($s =~ /^ERROR/  ? "yes" : "-"),
        ($s =~ /ERROR$/  ? "yes" : "-"),
        ($s =~ /^ERROR$/ ? "yes" : "-");
}

SUBJECT       /ERROR/   /^ERROR/   /ERROR$/   /^ERROR$/
ERROR         yes       yes        yes        yes
ERROR: bad    yes       yes        -          -
pre-ERROR     yes       -          yes        -
x ERROR y     yes       -          -          -

The four columns narrow progressively. /ERROR/ accepts all four subjects. /^ERROR/ requires the match to begin at the start, so it keeps ERROR: bad and drops pre-ERROR. /ERROR$/ requires it to end at the end, so it does the reverse. /^ERROR$/ demands both and accepts only the subject that is exactly ERROR.

An anchor is worth thinking of as a zero-width assertion: it matches a position, not a character. ^ consumes nothing and succeeds only at the start of the string; $ consumes nothing and succeeds at the end. That is why /^ERROR$/ is five characters long and matches a five-character string rather than a seven-character one.

There is a sharper point hiding in the /^PASS$/ idiom, and it is worth making against my own tools:

use strict;
use warnings;

# ^...$ is how you say "the whole field is exactly this"
my @status = ("PASS", "PASSED", "pass", " PASS");

printf "%-9s %-11s %s\n", "VALUE", "/^PASS\$/", "eq 'PASS'";
for my $s (@status) {
    printf "%-9s %-11s %s\n", "[$s]",
        ($s =~ /^PASS$/ ? "yes" : "-"),
        ($s eq "PASS"   ? "yes" : "-");
}
print "\nfor a fixed whole-string test, eq is clearer than a pattern\n";

VALUE     /^PASS$/    eq 'PASS'
[PASS]    yes         yes
[PASSED]  -           -
[pass]    -           -
[ PASS]   -           -

for a fixed whole-string test, eq is clearer than a pattern

The two columns are identical on all four values. When the thing you are testing is a fixed string and you want an exact match, eq says so more plainly than /^PASS$/, and it carries none of the questions about trailing newlines that the next section raises. Reach for the anchored pattern when part of the field varies — /^PASS(?:ED)?$/ — and for eq when nothing does. Chapter 11.1's rule about index applies here in a different form: the pattern is the right tool when there is a rule to express.

3. \b: The Edge of a Word

A word boundary is the position between a word character and a non-word character. It is zero-width, like the anchors.

use strict;
use warnings;

my @subjects = ("ERROR", "NO_ERROR", "NO-ERROR", "ERROR:", "XERROR", "ERRORS");

printf "%-11s %s\n", "SUBJECT", "/\\bERROR\\b/";
for my $s (@subjects) {
    printf "%-11s %s\n", $s, ($s =~ /\bERROR\b/ ? "yes" : "no");
}
print "\n";
print "_ is a word character, so NO_ERROR is ONE word: no boundary before ERROR\n";
print "- is not a word character, so NO-ERROR has a boundary and matches\n";

SUBJECT     /\bERROR\b/
ERROR       yes
NO_ERROR    no
NO-ERROR    yes
ERROR:      yes
XERROR      no
ERRORS      no

_ is a word character, so NO_ERROR is ONE word: no boundary before ERROR
- is not a word character, so NO-ERROR has a boundary and matches

Read NO_ERROR and NO-ERROR together — they are the pair that makes \b comprehensible.

NO_ERROR does not match. The underscore is a word character, as Chapter 11.5 measured, so the whole token is one unbroken word and there is no boundary in front of ERROR. NO-ERROR does match, because a hyphen is not a word character, so there is a boundary on each side of ERROR.

The other rows follow the same single rule. ERROR: matches because a colon is not a word character. XERROR and ERRORS do not, because the adjacent X and S are word characters and so there is no edge where \b needs one.

This is exactly the behaviour you want from a log scanner, and it is why \bERROR\b gave the right count in section one. It also happens to be a lucky alignment: it works because tools name things with underscores. A tool that wrote NO-ERROR would defeat it, and the honest fix there is ^ERROR\b, which says what you actually mean — the line starts with the word.

The complement exists too, and is occasionally useful:

use strict;
use warnings;

# \B is the complement: a position that is NOT a word boundary
my @subjects = ("ERROR", "NO_ERROR", "XERROR");

printf "%-11s %-12s %s\n", "SUBJECT", "\\bERROR", "\\BERROR";
for my $s (@subjects) {
    printf "%-11s %-12s %s\n", $s,
        ($s =~ /\bERROR/ ? "yes" : "-"),
        ($s =~ /\BERROR/ ? "yes" : "-");
}

SUBJECT     \bERROR      \BERROR
ERROR       yes          -
NO_ERROR    -            yes
XERROR      -            yes

\B matches precisely where \b does not. In practice you reach for it rarely — most often to find a substring inside a token, as in "which signal names have clk embedded in the middle rather than at the start".

4. $ Is Not the End of the String

This is the detail worth the price of the chapter.

use strict;
use warnings;

my $with    = "42\n";
my $without = "42";

printf "%-14s %-10s %-10s %s\n", "SUBJECT", "/^\\d+\$/", "/^\\d+\\z/", "/^\\d+\\Z/";
printf "%-14s %-10s %-10s %s\n", "'42'",
    ($without =~ /^\d+$/  ? "match" : "-"),
    ($without =~ /^\d+\z/ ? "match" : "-"),
    ($without =~ /^\d+\Z/ ? "match" : "-");
printf "%-14s %-10s %-10s %s\n", "'42\\n'",
    ($with =~ /^\d+$/  ? "match" : "-"),
    ($with =~ /^\d+\z/ ? "match" : "-"),
    ($with =~ /^\d+\Z/ ? "match" : "-");

SUBJECT        /^\d+$/    /^\d+\z/   /^\d+\Z/
'42'           match      match      match
'42\n'         match      -          match

/^\d+$/ matched "42\n". The string contains a character that is not a digit, and the pattern claimed the whole string was digits.

That is by design. $ means "at the end, or just before a newline at the end". Perl makes this choice because line-oriented input arrives with its newline attached, and it saves you from a chomp in the common case. There are three anchors in this family:

• $ — the end, or immediately before a final newline.
• \z — the absolute end, with nothing after it at all.
• \Z — like $, permitting one final newline. (\A is the counterpart of \z at the start, and unlike ^ it is never affected by /m.)

So \z is the strict one. If you are validating that a field contains only digits and nothing else, \z is the anchor that means it.

5. Where That Bites: Lines You Forgot to chomp

The forgiving $ is convenient until you mix it with .*.

use strict;
use warnings;

# Lines as they arrive from a filehandle: each still has its newline.
my @raw = ("ERROR: setup\n", "NO_ERROR\n", "ERROR: hold\n");

my $with_dollar = grep { /^ERROR\b.*$/  } @raw;
my $with_z      = grep { /^ERROR\b.*\z/ } @raw;
my $exact_z     = grep { /^(\w+)\z/     } @raw;

printf "unchomped lines     : %d\n", scalar @raw;
printf "/^ERROR\\b.*\$/       : %d\n", $with_dollar;
printf "/^ERROR\\b.*\\z/      : %d\n", $with_z;
printf "/^(\\w+)\\z/ on them  : %d   <- the newline defeats \\z\n", $exact_z;

my @clean = map { my $c = $_; chomp $c; $c } @raw;
my $after = grep { /^(\w+)\z/ } @clean;
printf "same after chomp    : %d\n", $after;

unchomped lines     : 3
/^ERROR\b.*$/       : 2
/^ERROR\b.*\z/      : 0
/^(\w+)\z/ on them  : 0   <- the newline defeats \z
same after chomp    : 1

The $ version found both error lines. The \z version found zero.

The reason is a combination of two facts already established. \z demands the absolute end of the string, and the dot does not match a newline (Chapter 11.5). So .* runs up to the newline and stops, \z is not satisfied there because a newline follows, and there is no way for the pattern to reach the real end. Nothing matches — on input where two lines plainly are errors.

The third count makes the same point for a whole-token test: /^(\w+)\z/ found nothing on the unchomped lines, and found one after chomp. (One rather than three, because only NO_ERROR is entirely word characters; the two error lines contain a colon and a space.)

6. Anchoring a Field Inside a Longer Line

Anchors tie to the whole string, so for a name buried in a line, \b is the tool.

use strict;
use warnings;

# A boundary is what makes a WORD search safe inside a longer line.
my @lines = (
    "set_load 0.4 [get_pins u_alu/A]",
    "set_load 0.4 [get_pins u_alu/AB]",
    "set_load 0.4 [get_pins pre_u_alu/A]",
);

for my $l (@lines) {
    printf "%-38s plain:%-4s bounded:%s\n", $l,
        ($l =~ /u_alu\/A/    ? "yes" : "no"),
        ($l =~ /\bu_alu\/A\b/ ? "yes" : "no");
}

set_load 0.4 [get_pins u_alu/A]        plain:yes  bounded:yes
set_load 0.4 [get_pins u_alu/AB]       plain:yes  bounded:no
set_load 0.4 [get_pins pre_u_alu/A]    plain:yes  bounded:no

The plain pattern matched all three lines; only the first is the pin you asked for.

u_alu/AB is a different pin whose name merely starts the same way, and the trailing \b rejects it because B is a word character sitting where a boundary was required. pre_u_alu/A is a different instance, and the leading \b rejects it because the underscore before u_alu is a word character — the same fact that made NO_ERROR fail to match, now doing useful work.

This pairs with Chapter 11.4's lesson. Matching a name from a variable wants both treatments: \Q so its own characters stay literal, and \b so it does not match inside a longer name. Together that is /\b\Q$pin\E\b/, which is close to the shortest correct way to ask "does this line mention exactly this pin".

7. What To Remember

• Without anchors a pattern matches anywhere. /ERROR/ counted 4 lines where 2 were errors — NO_ERROR and ERRORS_TOTAL were the false positives.
• ^ and $ are zero-width: they match a position, not a character.
• \b is the edge of a word, and "word" means \w — so NO_ERROR has no boundary before ERROR while NO-ERROR does.
• $ permits one trailing newline. /^\d+$/ matched "42\n".
• \z is the absolute end; \Z behaves like $; \A is the strict start and is immune to /m.
• \z plus .* finds nothing on unchomped lines — the dot cannot cross the newline, so the pattern can never reach the end. A count of zero, not an error.
• chomp first. Then the distinction stops mattering.
• For an exact fixed string, eq is clearer than /^...$/ and has no newline subtlety. The pattern earns its place when part of the field varies.
• /\b\Q$name\E\b/ is the idiom for "mentions exactly this name" — \Q for the characters, \b for the edges.
• ^ and $ change meaning under /m. Everything here is the default, single-line reading.

Chapter 11.9 adds the two constructs a pattern needs to express a choice: a vertical bar for alternatives, and parentheses to group part of a pattern so a quantifier or an alternation applies to the right piece. It also measures the precedence mistake that makes /^ERROR|FATAL$/ mean something other than what almost everyone reads it as.