Metacharacters and Escaping

Part I · Module 11 · Chapter 11.4 · Metacharacters and Escaping

1. The Dot, and Why This Chapter Exists

Start with the character that causes the most quiet damage.

use strict;
use warnings;

my @subjects = ("v1.2", "v1X2", "v142");

printf "%-8s %-12s %s\n", "SUBJECT", "/v1.2/", "/v1\\.2/";
for my $s (@subjects) {
    printf "%-8s %-12s %s\n", $s,
        ($s =~ /v1.2/  ? "match" : "-"),
        ($s =~ /v1\.2/ ? "match" : "-");
}

SUBJECT  /v1.2/       /v1\.2/
v1.2     match        match
v1X2     match        -
v142     match        -

The unescaped pattern matched all three subjects. The escaped one matched only the real version string.

A dot in a pattern means "any one character". So /v1.2/ asks for a v, a 1, anything at all, then a 2 — which v1X2 and v142 both satisfy. Nothing warns you, because nothing is wrong: you wrote a pattern and Perl used it.

This is the shape of every metacharacter bug. The pattern still looks like the text you meant, the match still succeeds, and the acceptance is simply wider than you intended. In a release-version gate that means approving the wrong build; in a log filter it means counting lines you meant to exclude.

2. The Complete Set

There are fourteen of them, and it is worth seeing the list produced rather than recited.

use strict;
use warnings;

# The characters that mean something other than themselves in a pattern
my @meta = split //, q{.^$*+?()[]{}|\\};

printf "metacharacters : %s\n", join(" ", @meta);
printf "count          : %d\n", scalar @meta;
print  "everything else in a pattern matches itself\n";

metacharacters : . ^ $ * + ? ( ) [ ] { } | \
count          : 14
everything else in a pattern matches itself

That final line is the useful half of the rule. Letters, digits, spaces, underscores, hyphens, colons, slashes, commas and the rest are all literal. You do not need to escape a slash or a hyphen or an @-free email address — only these fourteen.

Each of them gets its own treatment later in the module: the character classes and [ ] in 11.5, the quantifiers * + ? and { } in 11.6, the anchors ^ and $ in 11.8, and alternation | with grouping ( ) in 11.9. For now the only fact you need is that they are instructions, and that a backslash demotes any one of them back to plain text.

3. A Backslash for One Character

use strict;
use warnings;

my $line = "set_load 0.4 [get_pins u_alu/A]";

# Unescaped, these are all special and the match fails or misleads
printf "literal bracket, escaped   : %s\n", ($line =~ /\[get_pins/ ? "match" : "no match");
printf "literal dot, escaped       : %s\n", ($line =~ /0\.4/       ? "match" : "no match");
printf "literal plus (not present) : %s\n", ($line =~ /0\+4/       ? "match" : "no match");

# A dot matches the dot AND anything else
printf "unescaped dot on '0.4'     : %s\n", ($line =~ /0.4/        ? "match" : "no match");
printf "unescaped dot on '0x4'     : %s\n", ("0x4"  =~ /0.4/       ? "match" : "no match");

literal bracket, escaped   : match
literal dot, escaped       : match
literal plus (not present) : no match
unescaped dot on '0.4'     : match
unescaped dot on '0x4'     : match

\[ matched a real bracket, \. matched a real dot, and \+ correctly found nothing because the line has no plus sign. The last two lines show the dot's breadth once more: 0.4 and 0x4 are both accepted by /0.4/.

A note on escaping things that do not need it. A backslash before a letter is not harmless, because many letters already have meanings — \d, \w, \s, \b are all coming in the next chapters. Escaping a letter with no meaning gets you a warning:

use strict;
use warnings;

my $s = "qxy";

# \q has no meaning as an escape. Perl warns at COMPILE time (on stderr)
# and then treats it as a plain q.
my $hit = ($s =~ /\qxy/);

printf "/\\qxy/ on 'qxy' : %s\n", ($hit ? "match" : "no match");
printf "/qxy/  on 'qxy' : %s\n", ($s =~ /qxy/ ? "match" : "no match");

/\qxy/ on 'qxy' : match
/qxy/  on 'qxy' : match

Both match, so the stray backslash did no damage here. But the run is not silent — on stderr, at compile time, Perl says:

Unrecognized escape \q passed through in regex; marked by <-- HERE in m/\q <-- HERE xy/

The output block above shows only stdout, which is why the warning does not appear in it. It is worth knowing this is a compile-time warning: you cannot catch it with a $SIG{__WARN__} handler, because the handler is installed when the program runs and the warning was issued before that. Treat it as a signal that you escaped something that did not need it — the fix is to delete the backslash, not to silence the warning.

4. quotemeta Shows You What Escaping Means

Before the operator form, the function form, because its output makes the rule concrete.

use strict;
use warnings;

my @raw = ("v1.2 (beta)", "a+b", "u_alu[0]", "/proj/lib");

for my $r (@raw) {
    printf "%-14s -> %s\n", $r, quotemeta($r);
}

v1.2 (beta)    -> v1\.2\ \(beta\)
a+b            -> a\+b
u_alu[0]       -> u_alu\[0\]
/proj/lib      -> \/proj\/lib

Two details are more interesting than the obvious ones.

quotemeta escaped the space in v1.2 (beta). A space is not a metacharacter, but an escaped space is still just a space, so this is harmless — and it matters if the pattern is later used under /x mode, where unescaped whitespace is ignored. That is Chapter 11.17, and \Q getting this right is why a quoted string keeps working there.

It also escaped the slashes in /proj/lib, which are not metacharacters either. quotemeta does not consult a list of fourteen; it puts a backslash before every character that is not a word character. That is a broader rule than strictly necessary, and deliberately so — it is correct under every delimiter and every mode.

5. \Q...\E for a Whole Variable

This is the section that changes how you write patterns.

use strict;
use warnings;

my $line = "release v1.2.3 (beta) ok";
my $want = "v1.2.3 (beta)";

printf "subject : %s\n", $line;
printf "wanted  : %s\n\n", $want;

printf "/\$want/       : %s\n", ($line =~ /$want/     ? "match" : "no match");
printf "/\\Q\$want\\E/   : %s\n", ($line =~ /\Q$want\E/ ? "match" : "no match");

my $plus = "a+b";
my $txt  = "x a+b y";
print "\n";
printf "subject : %s\n", $txt;
printf "wanted  : %s\n", $plus;
printf "/\$plus/       : %s\n", ($txt =~ /$plus/     ? "match" : "no match");
printf "/\\Q\$plus\\E/   : %s\n", ($txt =~ /\Q$plus\E/ ? "match" : "no match");

subject : release v1.2.3 (beta) ok
wanted  : v1.2.3 (beta)

/$want/       : no match
/\Q$want\E/   : match

subject : x a+b y
wanted  : a+b
/$plus/       : no match
/\Q$plus\E/   : match

Both plain interpolations failed on text that is literally present in the subject. That is the failure mode worth internalising, because it is the opposite of the dot problem in section one: there the pattern was too permissive, here it is simply broken.

The reasons differ. In v1.2.3 (beta) the parentheses became a capture group, so the pattern asked for v1.2.3 followed immediately by beta with no parentheses around it. In a+b the plus became a quantifier attached to a, so the pattern asked for one-or-more a characters followed by b — it would have matched ab or aaab, but never a+b.

\Q begins a quoted region and \E ends it. Everything between them is treated as literal text, with the same rule quotemeta uses.

You can also quote part of a pattern. /^\Q$prefix\E\d+$/ matches a literal prefix followed by digits — the \Q region ends at \E, so the \d+ and the anchor still work normally.

6. Inside Square Brackets, Most of Them Relax

A character class has its own, much smaller set of special characters.

use strict;
use warnings;

# Inside [] most metacharacters lose their power
printf "a[.]b on 'a.b' : %s\n", ("a.b" =~ /a[.]b/ ? "match" : "no match");
printf "a[.]b on 'aXb' : %s\n", ("aXb" =~ /a[.]b/ ? "match" : "no match");
printf "[+] on 'a+b'   : %s\n", ("a+b" =~ /a[+]b/ ? "match" : "no match");

print "\n";
# But - and ^ and ] still matter inside brackets
printf "'b' in [a-c]   : %s\n", ("b" =~ /^[a-c]$/ ? "match" : "no match");
printf "'-' in [a-c]   : %s\n", ("-" =~ /^[a-c]$/ ? "match" : "no match");
printf "'-' in [ac-]   : %s\n", ("-" =~ /^[ac-]$/ ? "match" : "no match");
printf "'-' in [a\\-c]  : %s\n", ("-" =~ /^[a\-c]$/ ? "match" : "no match");

a[.]b on 'a.b' : match
a[.]b on 'aXb' : no match
[+] on 'a+b'   : match

'b' in [a-c]   : match
'-' in [a-c]   : no match
'-' in [ac-]   : match
'-' in [a\-c]  : match

[.] matched a real dot and rejected aXb, so inside brackets the dot is literal. The same goes for +.

This gives you a second way to escape a single character: [.] and \. do the same job. Some engineers prefer the bracket form on the grounds that it is harder to overlook than a backslash. Either is fine; be consistent.

The hyphen is the one to watch. [a-c] is a range and does not contain a hyphen — the measurement confirms - is not in it. Put the hyphen last, as in [ac-], or escape it, as in [a\-c], and it becomes a literal. A ^ is special only as the first character inside the brackets, where it negates the class; anywhere else it is literal. Chapter 11.5 covers classes properly.

7. The Dollar Sign Is Sometimes a Variable

Here is the trap the standard fourteen-character list does not prepare you for. A pattern is interpolated like a double-quoted string before it is compiled, so $ and @ have two jobs each.

use strict;
use warnings;

my $subject = "a1b";
my $digit   = "ZZZ";

# $ at the END of a pattern is an anchor
printf "/b\$/ on 'a1b'    : %s   <- \$ is an anchor here\n",
    ($subject =~ /b$/ ? "match" : "no match");

# $ followed by a NAME is a variable, not an anchor
printf "/a\$digit/        : %s   <- became /aZZZ/, not an anchor\n",
    ($subject =~ /a$digit/ ? "match" : "no match");

# To match a literal dollar sign, escape it
my $cost = 'cost $5';
printf "/\\\$5/ on 'cost \$5' : %s\n", ($cost =~ /\$5/ ? "match" : "no match");

/b$/ on 'a1b'    : match   <- $ is an anchor here
/a$digit/        : no match   <- became /aZZZ/, not an anchor
/\$5/ on 'cost $5' : match

Perl decides between the two readings by what follows the $. At the end of the pattern, or before a ) or a |, it is the end-of-string anchor. Followed by an identifier, it is a variable, and the variable's contents are spliced in. So /a$digit/ became /aZZZ/ and found nothing in a1b.

The @ case is worse, and worth a separate warning because it is a compile error rather than a wrong answer. A pattern containing a bare @ followed by a name — matching an email address, say, or a Verilog-style @(posedge clk) — will try to interpolate an array. Under use strict that stops the program:

Global symbol "@host" requires explicit package name (did you forget to declare "my @host"?)

with Possible unintended interpolation of @host in string alongside it. The fix is \@, or \Q if the text came from a variable.

8. A Worked Case: Both Kinds of Wrong at Once

The two failure modes in this chapter — a pattern that is too wide and a pattern that is simply broken — can both appear in one script, from one missing \Q. Here is a pin list as a tool emits it, filtered by a watch list of the kind you would read from a config file.

use strict;
use warnings;

# A pin list as a tool would emit it, and a watch list read from a config file.
my @pins = (
    "u_core/u_alu[0]/A",
    "u_core/u_alu[0]/B",
    "u_core/u_alu[1]/A",
    "u_core/u_alu0/A",       # a DIFFERENT instance: no brackets
    "u_core/u_alu1/A",
    "u_core/u_fifo.wr/A",
    "u_core/u_fifoXwr/A",    # a DIFFERENT instance: X where the dot was
);

my @watch = ("u_alu[0]", "u_fifo.wr");

for my $w (@watch) {
    my @loose = grep { $_ =~ /$w/      } @pins;
    my @exact = grep { $_ =~ /\Q$w\E/  } @pins;

    printf "watch %-12s plain: %d match(es)   \\Q..\\E: %d match(es)\n",
        $w, scalar @loose, scalar @exact;
    printf "    plain  -> %s\n", (@loose ? join(", ", @loose) : "(none)");
    printf "    quoted -> %s\n", (@exact ? join(", ", @exact) : "(none)");
}

watch u_alu[0]     plain: 1 match(es)   \Q..\E: 2 match(es)
    plain  -> u_core/u_alu0/A
    quoted -> u_core/u_alu[0]/A, u_core/u_alu[0]/B
watch u_fifo.wr    plain: 2 match(es)   \Q..\E: 1 match(es)
    plain  -> u_core/u_fifo.wr/A, u_core/u_fifoXwr/A
    quoted -> u_core/u_fifo.wr/A

Read the two watch terms separately, because they fail in opposite directions.

u_alu[0] found one pin, and it was the wrong one. Without quoting, [0] is a character class containing the single character 0, so the pattern means u_alu followed by a 0 — which describes u_alu0, a genuinely different instance. The two pins the engineer was actually asking about were not reported at all. The quoted version found both.

u_fifo.wr found two pins, and one of them was wrong. Here the dot matched the X in u_fifoXwr, so an unrelated instance was pulled into the result. The quoted version found the one real pin.

So the same omission produced an under-count that hid real pins and an over-count that invented one. Neither run errored, neither warned, and both printed a plausible number. If this were a script applying set_load to a watch list, the first term would have loaded the wrong instance and skipped the intended ones.

9. What To Remember

• Fourteen characters are special: . ^ $ * + ? ( ) [ ] { } | \ — everything else in a pattern matches itself.
• The dot is the expensive one. /v1.2/ matched v1.2, v1X2 and v142. It makes a pattern wider than it looks, silently.
• A backslash makes one metacharacter literal. [.] does the same job if you find it easier to see.
• A backslash on a letter with no meaning warns — Unrecognized escape \q passed through in regex, at compile time, on stderr. Delete the backslash.
• \Q...\E makes a whole region literal, and is what you want for any text that came from a variable. Without it, both measured interpolations failed on text that was literally present.
• quotemeta is the function form, and it escapes every non-word character — spaces and slashes included.
• Inside [ ] most metacharacters are literal, but the hyphen makes a range unless it is last or escaped, and ^ negates when it is first.
• A $ followed by an identifier is a variable, not an anchor, and a bare @ followed by a name is an array — a compile error under use strict.

Chapter 11.5 moves from literal text to kinds of characters: \d for a digit, \w for a word character, \s for whitespace, and your own sets in square brackets — including the measured surprise about what \w actually contains.