Why Regex Is Built Into the Language

Part I · Module 11 · Chapter 11.1 · Why Regex Is Built Into the Language

1. Two Different Questions

Almost every line-processing task you will write is one of two questions, and they deserve different tools.

The first is yes or no. Does this log contain the word ERROR? Did the tool print a banner? Is this line a comment? The answer is a single bit, and you already know the exact text you are looking for.

The second is give me the value. What slack did this path report? Which pin does this constraint apply to? What number followed the word scored? Here you do not know the text in advance — that is the whole point. You know its shape, and you want the part that varies.

Perl answers the first question perfectly well without any pattern at all. The curriculum's own framing for this chapter says so: a plain text search is enough for yes or no. Learning where that line falls is more useful than learning another twelve metacharacters, so we will start there and work toward the case that genuinely needs a pattern.

2. Yes or No Needs No Pattern

index searches for a fixed substring and returns where it found it.

use strict;
use warnings;

my $line = "Ravi scored 72";

my $at = index($line, "scored");
printf "index(line, 'scored') = %d\n", $at;
printf "verdict               = %s\n", ($at >= 0 ? "present" : "absent");

my $miss = index($line, "failed");
printf "index(line, 'failed') = %d\n", $miss;
printf "verdict               = %s\n", ($miss >= 0 ? "present" : "absent");

index(line, 'scored') = 5
verdict               = present
index(line, 'failed') = -1
verdict               = absent

Two details in that output matter more than they look.

index returned 5, not a true/false value — it tells you the position, which is often exactly what you wanted next. And a failure returned -1, not 0. That is deliberate: position 0 is a perfectly good hit at the very start of the string. This is why the comparison is >= 0 and never a bare truth test. Writing if (index($line, "ERROR")) is a real bug — it silently reports "absent" for a line that begins with ERROR, and reports "present" for -1, because -1 is true in Perl.

For a fixed word, index is the right call. It is faster, it cannot be thrown off by a stray metacharacter in your search term, and a reader does not have to parse a pattern to see what it looks for.

3. Where a Pattern Starts To Earn Its Cost

Now change the question from is the word there to what number follows it.

use strict;
use warnings;

for my $line ("Ravi scored 72", "Ravi scored 72 of 100") {
    print "subject: $line\n";

    my ($marks) = $line =~ /scored (\d+)/;
    printf "  pattern     : [%s]\n", $marks;

    my $key = "scored ";
    my $pos = index($line, $key);
    my $tail = ($pos >= 0) ? substr($line, $pos + length($key)) : "";
    printf "  index+substr: [%s]\n", $tail;
}

subject: Ravi scored 72
  pattern     : [72]
  index+substr: [72]
subject: Ravi scored 72 of 100
  pattern     : [72]
  index+substr: [72 of 100]

On the first line both approaches work and the index version looks like a reasonable alternative. On the second line it returns 72 of 100.

It is worth being precise about why, because the reason is structural rather than a slip. substr needs an end position, and index can only find text you already know. To make the hand-written version correct you would have to find where the number stops — which means asking "is this character a digit?", one character at a time, until the answer is no. You would be writing a small number parser. The pattern scored (\d+) says the same thing in nine characters because \d+ is that loop, expressed as a shape.

That is the trade in one sentence: you reach for a pattern when the thing you are looking for is described by a rule rather than spelled out.

4. A Pattern States a Shape; a Loop States an Alphabet

Here is the argument that matters, and the one that is usually left out of regex tutorials in favour of counting characters saved.

A timing report line is supposed to look like <pin containing a slash>, then a decimal number, then the unit ns. Below, that one rule is implemented twice. Neither implementation is a straw man — the hand-written one checks the field count, checks the slash, checks the unit, rejects an empty value, and validates every character of the number.

use strict;
use warnings;

# One rule, two implementations:
#   "<pin containing a slash>  <decimal number>  ns"

sub ok_by_hand {
    my ($line) = @_;
    my @f = split " ", $line;
    return 0 unless @f == 3;
    my ($pin, $val, $unit) = @f;
    return 0 unless index($pin, "/") > 0;
    return 0 unless $unit eq "ns";
    return 0 if $val eq "";
    for my $c (split //, $val) {
        return 0 unless $c =~ /[0-9.]/;      # every character is legal...
    }
    return 1;                                 # ...but is the WHOLE thing a number?
}

sub ok_by_pattern {
    my ($line) = @_;
    return $line =~ m{^ \S+/\S+ \s+ \d+(?:\.\d+)? \s+ ns $}x ? 1 : 0;
}

my @subjects = (
    "u_alu/A 0.42 ns",
    "u_alu/A 42 ns",
    "u_alu/A .... ns",
    "u_alu/A 0.4.2 ns",
    "u_alu/A -0.42 ns",
    "/A 0.42 ns",
    "u_alu/A 0.42 ps",
);

printf "%-22s %-8s %-8s %s\n", "SUBJECT", "BY HAND", "PATTERN", "AGREE?";
my $disagree = 0;
for my $s (@subjects) {
    my ($h, $p) = (ok_by_hand($s), ok_by_pattern($s));
    $disagree++ if $h != $p;
    printf "%-22s %-8d %-8d %s\n", $s, $h, $p, ($h == $p ? "yes" : "NO");
}
printf "\ndisagreements: %d of %d\n", $disagree, scalar @subjects;

SUBJECT                BY HAND  PATTERN  AGREE?
u_alu/A 0.42 ns        1        1        yes
u_alu/A 42 ns          1        1        yes
u_alu/A .... ns        1        0        NO
u_alu/A 0.4.2 ns       1        0        NO
u_alu/A -0.42 ns       0        0        yes
/A 0.42 ns             0        0        yes
u_alu/A 0.42 ps        0        0        yes

disagreements: 2 of 7

Four of the seven subjects are rejected by both, which tells you the hand-written version is not simply broken — it gets the negative cases right. It disagrees on exactly two, and both are the same mistake:

.... and 0.4.2 are accepted as numbers.

The loop asks, of each character, are you a member of the set [0-9.]. Every character of .... is. Every character of 0.4.2 is. What the loop never asks is whether the characters are arranged in the order a number requires — some digits, then at most one dot, then some more digits. It validates an alphabet. The rule is a shape.

A note on honesty: the same discipline applies to patterns. \d+(?:\.\d+)? is itself a choice about what counts as a number here — it rejects -0.42, and the output proves it. On a real timing report that would be a serious bug, since slack is negative exactly when you care. The lesson is not that patterns are automatically correct. It is that a pattern puts the whole rule in one readable place where a reviewer can see what it accepts, instead of spreading it across six return 0 unless lines and a loop.

5. An Operator, Not a Function Call

The reason a Perl match is short is that matching is built into the grammar. There is no object to construct and no method to call, and the operators return useful values rather than a result object you then have to interrogate.

use strict;
use warnings;

my $s = "slack -0.12";

my $subs = ($s =~ s/-/minus /);
printf "s/// returned  : %s\n", $subs;
printf "string is now  : [%s]\n", $s;

my $lower = ($s =~ tr/a-z//);
printf "tr/// counted  : %d lowercase letters\n", $lower;

my $hit = ($s =~ /minus/);
printf "m// returned   : [%s]\n", $hit;

my $no = ($s =~ /ERROR/);
printf "a failed m//   : [%s] (defined: %s)\n", $no, (defined $no ? "yes" : "no");

s/// returned  : 1
string is now  : [slack minus 0.12]
tr/// counted  : 10 lowercase letters
m// returned   : [1]
a failed m//   : [] (defined: yes)

Three facts to carry forward from that run.

s/// returns a count. It replaced one dash and returned 1. This makes my $n = ($text =~ s/old/new/g); a one-line way to both edit a string and learn how many edits happened — a pattern you will use constantly in report rewriting.

tr/// counts too. With an empty replacement list it changes nothing and reports how many characters matched. Here the string was slack minus 0.12, and it found 10 lowercase letters: the five in slack plus the five in minus. The spaces, the digits and the dot are not in the a-z range, so they were not counted.

A failed match returns the empty string, and it is defined. The last line is the one worth memorising. A successful match gave 1; the failure gave "", and defined reported yes. So a match result is a clean boolean — true or false — but it is never undef. Testing if (defined $line =~ /x/) is always true and tests nothing. Chapter 11.2 goes into what m// returns in each context in detail.

6. A Pattern Is a Value

Patterns are not only syntax. qr// compiles one into a value you can store, pass to a subroutine, and keep in a configuration table.

use strict;
use warnings;

my $pat = qr/^ERROR\b/;

printf "ref(\$pat)   : %s\n", ref($pat);
printf "as a string : %s\n", "$pat";

my @lines = ("ERROR: setup violation", "WARNING: unconstrained", "ERRORS_TOTAL 2");
my @hits  = grep { $_ =~ $pat } @lines;

printf "lines       : %d\n", scalar @lines;
printf "matched     : %d\n", scalar @hits;
printf "which       : %s\n", join(", ", @hits);

ref($pat)   : Regexp
as a string : (?^:^ERROR\b)
lines       : 3
matched     : 1
which       : ERROR: setup violation

ref reports Regexp, so this really is an object in the ordinary Perl sense. Stringifying it shows (?^:^ERROR\b) — Perl wrapped your pattern in a group recording which flags it was compiled with, which is why a qr// keeps its own modifiers even when interpolated into a larger pattern later.

Note also which line matched. ERRORS_TOTAL 2 starts with the letters ERROR but was not counted, because \b demands a word boundary after it and S is a word character. That single boundary is the difference between a log summary that says 1 error and one that says 2. Chapter 11.8 is devoted to it; qr// gets its own treatment at 11.18.

7. Choosing the Delimiter

A pattern does not have to be written between slashes, and for anything involving file paths it should not be.

use strict;
use warnings;

my $path = "/proj/lib/cells/and2.lib";

printf "slashes : %s\n", ($path =~ /^\/proj\/lib\//  ? "match" : "no match");
printf "m{}     : %s\n", ($path =~ m{^/proj/lib/}    ? "match" : "no match");
printf "m!!     : %s\n", ($path =~ m!^/proj/lib/!    ? "match" : "no match");

slashes : match
m{}     : match
m!!     : match

All three match the same path. The first is the version you will meet in old scripts, and the backslashes exist only to protect the delimiter — they say nothing about the pattern. Writing m{...} or m!...! lets the slashes be slashes. m{} is the usual house choice because it pairs, so it nests cleanly and most editors highlight it correctly. You will see this used throughout the module.

8. What It Costs

Patterns are not free, and it is worth having a real number rather than a feeling.

use strict;
use warnings;
use Time::HiRes qw(time);

my @lines = map { "u_alu/bit$_ 0.42 ns" } 1 .. 200000;

my $t0 = time;
my $c1 = 0;
for my $l (@lines) { $c1++ if index($l, "u_alu") >= 0 }
my $t1 = time;

my $c2 = 0;
for my $l (@lines) { $c2++ if $l =~ /u_alu/ }
my $t2 = time;

printf "lines        : %d\n", scalar @lines;
printf "index found  : %d\n", $c1;
printf "pattern found: %d\n", $c2;
printf "both agree   : %s\n", ($c1 == $c2 ? "yes" : "no");

lines        : 200000
index found  : 200000
pattern found: 200000
both agree   : yes

The script prints only the counts, because a wall-clock time is not a reproducible output. Running the timed version five times on this machine gave index 0.017–0.024 s and the literal pattern 0.024–0.032 s for the same 200,000 lines — a ratio between 1.30x and 1.47x.

So the pattern is measurably slower, and the honest conclusion is still that it does not matter: 0.03 seconds for 200,000 lines is not the bottleneck in any flow you will write. Prefer index for a fixed-string yes/no because it states your intent more plainly, not because of the microseconds. Save the performance argument for the cases that deserve it — a pattern rebuilt inside a loop over a million lines, which is what 11.18 is about.

9. What To Remember

• Yes/no on a known word: use index. Compare with >= 0, never as a bare truth test — -1 is true and position 0 is a real hit.
• Extracting a value: use a pattern. index plus substr cannot express "stop at the end of the number" without you writing a parser by hand.
• A pattern states a shape; a character loop states an alphabet. That is why the hand-written check in section four accepted .... and 0.4.2 as numbers while the pattern rejected them — 2 disagreements out of 7 subjects.
• A pattern is not automatically right. The one in section four also rejected -0.42. Putting the whole rule in one line makes it reviewable, not correct.
• m// returns 1 on success and "" on failure — false, but always defined.
• s/// and tr/// return counts, so you can edit and measure in one statement.
• qr// makes a pattern a value of class Regexp that carries its own flags.
• Use m{...} for anything with slashes in it. Backslashed delimiters obscure the pattern.

Chapter 11.2 takes the match operator itself — what m// gives back in scalar and list context, and the fact that a bare pattern with no variable attached quietly tests $_.