Applying a Pattern to a Variable: =~ and !~

Part I · Module 11 · Chapter 11.3 · Applying a Pattern to a Variable: =~ and !~

1. The Operator That Joins a Pattern to a String

=~ reads as "is matched against". It takes a string expression on its left and a pattern on its right, and gives back the verdict. !~ is the same thing with the answer inverted.

use strict;
use warnings;

my $line = "NO_ERROR reported";

printf "\$line =~ /ERROR/  : %s\n", ($line =~ /ERROR/  ? "true" : "false");
printf "\$line !~ /ERROR/  : %s\n", ($line !~ /ERROR/  ? "true" : "false");
printf "\$line =~ /^ERROR/ : %s\n", ($line =~ /^ERROR/ ? "true" : "false");
printf "\$line !~ /^ERROR/ : %s\n", ($line !~ /^ERROR/ ? "true" : "false");

$line =~ /ERROR/  : true
$line !~ /ERROR/  : false
$line =~ /^ERROR/ : false
$line !~ /^ERROR/ : true

The four results are two pairs, each pair exactly opposite. NO_ERROR reported does contain ERROR — in the middle of NO_ERROR — so the first pair is true/false. It does not start with ERROR, so the second pair is false/true.

That first pair is the wrong-count bug in miniature, and it is why ^ appears in the second. Chapter 11.8 is about fixing it properly.

!~ exists because unless and negated conditions read badly once nested. Both of these say the same thing:

• if ($line !~ /ERROR/) { ... }
• unless ($line =~ /ERROR/) { ... }

Prefer !~ when the test sits inside a larger condition, and unless when the whole statement is the negation. What you should not write is the third form, and section six explains why.

2. Leave the Binding Out and You Test $_

Chapter 11.2 showed a bare pattern testing $_ while $_ was unset, which produced a warning. The more dangerous case is when $_ holds something.

use strict;
use warnings;

my $v = "slack -0.12";

# $_ is set to something ELSE entirely
$_ = "a totally different string";

printf "\$v                 = [%s]\n", $v;
printf "\$_                 = [%s]\n", $_;
printf "bare /slack/       = %s   <- tested \$_, not \$v\n", (/slack/      ? "true" : "false");
printf "bare /different/   = %s   <- also \$_\n",            (/different/  ? "true" : "false");
printf "\$v =~ /slack/      = %s   <- tested \$v\n",          ($v =~ /slack/ ? "true" : "false");

$v                 = [slack -0.12]
$_                 = [a totally different string]
bare /slack/       = false   <- tested $_, not $v
bare /different/   = true   <- also $_
$v =~ /slack/      = true   <- tested $v

The bare /slack/ reported false about a variable that plainly contains slack, and the run produced no warning of any kind. Nothing was undefined, nothing was invalid — the pattern simply matched a different string than the author meant.

In a real script $_ gets set by all the constructs that make Perl convenient: for (@lines), grep, map, and while (my $l = <$fh>) variants that assign to $_. So the usual way this bug arrives is a pattern that was correct inside a loop, copied out of the loop during a refactor, where it keeps compiling and keeps returning a verdict about whatever $_ held last.

3. What =~ Will Bind To

The left side does not have to be a simple scalar. It can be any expression that produces a string.

use strict;
use warnings;

my %slack = (setup => "-0.12", hold => "0.44");
my @pins  = ("u_alu/A", "u_fifo/B");

# =~ binds to any expression that yields a string
printf "hash element  : %s\n", ($slack{setup} =~ /^-/     ? "negative" : "positive");
printf "array element : %s\n", ($pins[1]      =~ /fifo/   ? "fifo pin" : "other");
printf "a function     : %s\n", (uc("error")  =~ /ERROR/  ? "match"    : "no match");
printf "a literal      : %s\n", ("u_alu/A"    =~ m{/}     ? "has slash": "no slash");

hash element  : negative
array element : fifo pin
a function     : match
a literal      : has slash

Hash elements, array elements, function results and literals all work. This is what makes conditions like next unless $row{status} =~ /^(?:ok|pass)$/ idiomatic — there is no need to copy the field into a temporary first.

There is one restriction, and it only applies to the operators that modify. A match can read anything, but s/// and tr/// have to write back, so their left side must be something assignable. Binding a substitution to a literal or to a function's return value is a compile-time error, and binding it to a read-only value dies at run time — which Chapter 9.2 measured and Chapter 11.13 revisits.

4. The Negated Form Still Sets Your Captures

This is the finding of the chapter, and it is not in most references.

use strict;
use warnings;

my $line = "slack -0.12";

# A SUCCESSFUL =~ sets $1
if ($line =~ /slack\s+(\S+)/) {
    printf "after =~ success : \$1 = [%s]\n", $1;
}

# Does !~ set captures when the pattern DOES match the text?
my $neg = ($line !~ /slack\s+(\S+)/);
printf "!~ returned      : [%s] (false, because the pattern DID match)\n", $neg;
printf "after !~         : \$1 = [%s]\n", (defined $1 ? $1 : "undef");

after =~ success : $1 = [-0.12]
!~ returned      : [] (false, because the pattern DID match)
after !~         : $1 = [-0.12]

Look at the last two lines together. !~ returned the empty string — false — and $1 is nevertheless -0.12.

Once you see the mechanism it is obvious. !~ is not a different kind of matching. The regex engine runs the pattern exactly as =~ would, stores whatever the capture groups caught, and then the ! part flips the true/false answer on its way out. The captures are a side effect of running the pattern, and running the pattern is not optional.

The consequence is a plausible-looking piece of code that is wrong:

use strict;
use warnings;

my $line = "clean line";

# !~ is TRUE here (no match). What is $1?
my $neg = ($line !~ /slack\s+(\S+)/);
printf "!~ returned : [%s]\n", $neg;
printf "\$1          : %s\n", (defined $1 ? "[$1]" : "undef");
print "there is nothing to capture when the pattern did not match\n";

!~ returned : [1]
$1          : undef
there is nothing to capture when the pattern did not match

Here the pattern did not match, !~ is true, and $1 is undef. Be careful about what that proves: it is undef because no match had ever succeeded in this program, not because !~ cleared anything. Nothing clears $1. In a loop where an earlier line matched, this same code would report a stale value from that earlier line. Chapter 11.10 measures that directly, and it is the reason the guard taught there uses a list assignment rather than $1.

So: never read a capture variable after a negated match. If you want both "did it match" and "what did it capture", you want =~ with a list assignment, and the negative case is the else.

5. !~ Is a Boolean Even in List Context

Chapter 11.2 established that =~ in list context returns captures. The negated form does not.

use strict;
use warnings;

my $line = "slack -0.12";

my @pos = ($line =~ /slack\s+(\S+)/);
printf "=~ in list context : %d item(s) [%s]\n", scalar @pos, join("|", @pos);

my @neg = ($line !~ /slack\s+(\S+)/);
printf "!~ in list context : %d item(s) [%s]\n", scalar @neg, join("|", @neg);
print "!~ is a BOOLEAN operator -- it never returns captures\n";

=~ in list context : 1 item(s) [-0.12]
!~ in list context : 1 item(s) []
!~ is a BOOLEAN operator -- it never returns captures

Both returned one item, but they are different things. The =~ version returned the capture -0.12. The !~ version returned its boolean — here the empty string, because the pattern did match and the negation is therefore false.

The trap is that both arrays have length 1, so a length test cannot tell them apart. my ($x) = ($line !~ /(\S+)/); compiles, runs, and assigns a boolean to a variable named as though it holds captured text.

6. Precedence: Why a Match Needs No Parentheses

=~ binds very tightly — more tightly than assignment, the logical operators, and the ternary.

use strict;
use warnings;

my $line = "ERROR: setup";

# =~ binds TIGHTER than || && ! ?: and =
my $verdict = $line =~ /ERROR/ ? "fail" : "pass";
printf "in a ternary, no parens  : %s\n", $verdict;

my $flag = $line =~ /ERROR/;
printf "in an assignment         : [%s]\n", $flag;

my $both = $line =~ /ERROR/ && $line =~ /setup/;
printf "with &&                  : %s\n", ($both ? "true" : "false");

my $negated = !($line =~ /ERROR/);
printf "!(\$line =~ /ERROR/)      : %s\n", ($negated ? "true" : "false");

# This is the one that surprises people: ! binds LOOSER than =~,
# so !$line =~ /x/ means (!$line) =~ /x/

in a ternary, no parens  : fail
in an assignment         : [1]
with &&                  : true
!($line =~ /ERROR/)      : false

All four behave as written. my $verdict = $line =~ /ERROR/ ? "fail" : "pass"; needs no parentheses because the match is evaluated before the ternary sees it. This is why Perl conditions involving patterns read as cleanly as they do.

There is one exception to "binds tighter", and it is the one that costs people an afternoon.

use strict;
use warnings;

# The line does NOT contain ERROR, so "does not contain" should be TRUE.
my $line = "clean: 0 violations";

printf "subject: %s\n\n", $line;

# Two correct ways to say "does not contain ERROR"
printf "\$line !~ /ERROR/    : %s\n", ($line !~ /ERROR/    ? "true" : "false");
printf "!(\$line =~ /ERROR/) : %s\n", (!($line =~ /ERROR/) ? "true" : "false");

# WRONG: ! binds tighter than =~, so this reads (!$line) =~ /ERROR/
my $wrong = (!$line =~ /ERROR/);
printf "!\$line =~ /ERROR/   : %s   <- WRONG\n", ($wrong ? "true" : "false");

printf "\n(!\$line) is the empty string [%s]\n", (!$line);
print "the empty string does not contain ERROR, so the broken form says false\n";

subject: clean: 0 violations

$line !~ /ERROR/    : true
!($line =~ /ERROR/) : true
!$line =~ /ERROR/   : false   <- WRONG

(!$line) is the empty string []
the empty string does not contain ERROR, so the broken form says false

The two correct forms say true. The broken form says false, and the broken form is one character away from a correct one.

! is a unary operator and binds to $line before =~ is considered, so !$line =~ /ERROR/ parses as (!$line) =~ /ERROR/. $line is a non-empty string, so !$line is the empty string, and the empty string is then matched against /ERROR/. The verdict has nothing to do with your data.

This produced no warning under use warnings. I checked that separately with perl -we, and the run is silent. There is nothing invalid happening — a string got matched against a pattern, which is exactly what you asked for.

7. A String on the Right Is Compiled as a Pattern

The right side can be interpolated, which is how you drive a match from configuration.

use strict;
use warnings;

my $line = "ERROR: setup violation";

# A plain string can be used as a pattern -- it is interpolated and compiled
my $what = "setup";
printf "interpolated string : %s\n", ($line =~ /$what/ ? "match" : "no match");

# ...which means metacharacters in the STRING are still metacharacters
my $dotted = "ERROR.";
printf "pattern [%s] on the line : %s\n", $dotted,
    ($line =~ /$dotted/ ? "match" : "no match");
print "  the dot matched the ':' -- see Chapter 11.4 for \\Q\n";

interpolated string : match
pattern [ERROR.] on the line : match
  the dot matched the ':' -- see Chapter 11.4 for \Q

The first match is the useful case: a pattern held in a variable, perhaps read from a config file, applied to each line.

The second shows the catch. The string ERROR. was interpolated into the pattern and then compiled as a pattern, so its dot is a metacharacter meaning "any character" — and it matched the colon. If that string had come from a user, a search for a literal ERROR. would quietly match ERROR:, ERRORS, and ERROR as well. Chapter 11.4 is about exactly this, and gives you \Q to switch it off.

8. What To Remember

• =~ names the string the pattern applies to; !~ runs the same match and inverts the verdict.
• Omit the binding and you test $_. If $_ is set, this is completely silent — no error, no warning, and a confident wrong answer.
• =~ binds to any string expression: hash and array elements, function results, literals. Only the modifying operators s/// and tr/// require something assignable.
• A false !~ leaves $1 populated, because the engine ran the pattern and stored the captures before the negation flipped the answer. Never read a capture after a negated match.
• $1 being undef after a failed match proves nothing about clearing — nothing clears it. See Chapter 11.10.
• !~ returns a boolean even in list context, so a one-element list is not evidence you captured something.
• =~ binds tighter than ?:, =, && and || — so a match drops into a larger expression without parentheses.
• But ! binds tighter still. !$line =~ /ERROR/ means (!$line) =~ /ERROR/, tests the empty string, and does not warn.
• An interpolated string is compiled as a pattern, metacharacters included.

Chapter 11.4 takes up that last point: which characters are special, what a backslash does to them, and how \Q...\E escapes a whole variable's worth of text so a path or a signal name can be matched literally.