Misconception: "Regex-Heavy Automation Needs Perl"

Part II · Module 16 · Chapter 16.8 · Misconception: "Regex-Heavy Automation Needs Perl"

1. A Claim That Can Be Settled

"You need Perl for the regex" is the argument most often used to keep a script in Perl, and it is the easiest one to check. Both interpreters are on this machine. The same pattern can be handed to both engines and the result compared.

So this chapter does not reason about it. It runs it.

The experiment is deliberately simple: sixteen features, one subject string each, one pattern each. For every feature, does the pattern compile, and does it match the subject it is supposed to match? A feature that fails to compile is unsupported; a feature that compiles and matches works.

2. The Table

use strict;
use warnings;
use File::Temp qw(tempdir);
use File::Spec;

# label, subject, pattern, needs /x ?
my @feature = (
 [ "named capture",      'U1/Q 0.44', '(?<pin>\S+)\s+(?<slack>\S+)', 0 ],
 [ "non-greedy",         'aXbXc',     'a(.*?)X',                     0 ],
 [ "lookahead",          'clk_a x',   'clk_\w+(?=\s)',               0 ],
 [ "lookbehind, fixed",  'x=0.44',    '(?<==)[\d.]+',                0 ],
 [ "lookbehind, varying",'xy=0.44',   '(?<=x{1,3}y=)[\d.]+',         0 ],
 [ "backreference",      'abab',      '(ab)\1',                      0 ],
 [ "/x whitespace mode", 'U1/Q 0.44', '(\S+) \s+ ([\d.]+)',          1 ],
 [ "inline (?i)",        'ERROR',     '(?i)error',                   0 ],
 [ "alternation",        'hold',      '(setup|hold)',                0 ],
 [ "anchors A and z",    '42',        '\A\d+\z',                     0 ],
 [ "word boundary",      'clk_a x',   '\bclk_a\b',                   0 ],
 [ "unicode \\p{Lu}",    'A',         '\p{Lu}',                      0 ],
 [ "possessive a++",     'aaab',      'a++b',                        0 ],
 # the anchors must stay OUTSIDE the recursion, or (?R) recurses them too
 [ "recursion (?R)",     '(())',      '\((?:[^()]|(?R))*\)',         0 ],
 [ "code block (?{})",   'x',         'x(?{ 1 })',                   0 ],
 [ "\\K keep",           'pin=U1',    'pin=\KU\d',                   0 ],
);

# ---- run them in PERL
my %perl;
for my $f (@feature) {
    my ($label, $subj, $pat, $x) = @$f;
    my $re;
    { local $@; $re = eval { $x ? qr/$pat/x : qr/$pat/ } }
    if (!defined $re) { $perl{$label} = "unsupported"; next }
    my $m = eval { $subj =~ $re ? 1 : 0 };
    $perl{$label} = !defined $m ? "error" : $m ? "works" : "no match";
}

# ---- run the SAME features in PYTHON, translating only the spellings
#      that differ by name: (?<name>..) -> (?P<name>..), \z -> \Z
my $dir = tempdir(CLEANUP => 1);
my $py = File::Spec->catfile($dir, "parity.py");
open my $w, ">", $py or die "cannot write: $!\n";
print $w "import re, sys\n";
for my $f (@feature) {
    my ($label, $subj, $pat, $x) = @$f;
    (my $ppat = $pat) =~ s/\(\?<(\w+)>/(?P<$1>/g;
    $ppat =~ s/\\z/\\Z/g;
    for ($ppat, $subj, $label) { s/\\/\\\\/g; s/'/\\'/g }
    printf $w "try:\n    rx = re.compile('%s', re.X if %d else 0)\n"
            . "    print('%s\\t' + ('works' if rx.search('%s') else 'no match'))\n"
            . "except re.error:\n    print('%s\\tunsupported')\n",
        $ppat, $x, $label, $subj, $label;
}
close $w;
my %python = map { my ($k, $v) = split /\t/; ($k => $v) }
             grep { /\t/ } split /\n/, `python3 '$py' 2>&1`;

# ---- merge
my ($same, $perl_only, $neither) = (0, 0, 0);
printf "%-22s %-12s %-12s %s\n", "FEATURE", "PERL 5.34", "PYTHON 3.13", "VERDICT";
for my $f (@feature) {
    my $label = $f->[0];
    my ($p, $y) = ($perl{$label} // "?", $python{$label} // "?");
    my $verdict = $p eq $y                              ? do { $p eq "works" ? $same++ : $neither++; "same" }
                : $p eq "works" && $y eq "unsupported"  ? do { $perl_only++; "PERL ONLY" }
                : $y eq "works" && $p eq "unsupported"  ? "python only"
                :                                          "differs";
    printf "%-22s %-12s %-12s %s\n", $label, $p, $y, $verdict;
}
printf "\n  identical in both : %d of %d\n", $same, scalar @feature;
printf "  Perl only         : %d\n", $perl_only;
printf "  neither, as given : %d\n", $neither;

FEATURE                PERL 5.34    PYTHON 3.13  VERDICT
named capture          works        works        same
non-greedy             works        works        same
lookahead              works        works        same
lookbehind, fixed      works        works        same
lookbehind, varying    works        unsupported  PERL ONLY
backreference          works        works        same
/x whitespace mode     works        works        same
inline (?i)            works        works        same
alternation            works        works        same
anchors A and z        works        works        same
word boundary          works        works        same
unicode \p{Lu}         works        unsupported  PERL ONLY
possessive a++         works        works        same
recursion (?R)         works        unsupported  PERL ONLY
code block (?{})       unsupported  unsupported  same
\K keep                works        unsupported  PERL ONLY

  identical in both : 11 of 16
  Perl only         : 4
  neither, as given : 1

Eleven of sixteen behave identically. Named captures, non-greedy quantifiers, lookahead, fixed-width lookbehind, backreferences, the whitespace mode, inline flags, alternation, anchors and word boundaries all work the same way in both engines.

And one row deserves singling out: possessive quantifiers work in Python. a++b compiles and matches. These are widely described as a Perl feature, and they were added to Python in 3.11. The folklore is simply out of date, which is the risk with any claim of this kind that has not been re-checked.

3. The Four Gaps, and Whether They Matter

Four features work in Perl and not in Python. For each, the honest question is whether a flow script would use it.

Variable-length lookbehind — (?<=x{1,3}y=). Perl accepts it and warns that it is experimental; Python rejects it outright with "look-behind requires fixed-width pattern". In practice: rarely needed, and the standard workaround is to capture what precedes instead of asserting it. A pattern relying on an experimental Perl feature is also a pattern that may change behaviour on a Perl upgrade, so leaning on it is a cost in Perl too.

\p{Lu} and the unicode properties. Genuinely useful for natural-language text, essentially never needed for a timing report whose alphabet is ASCII identifiers and digits. Python's re has no \p{...}; the third-party regex module does, which reintroduces Chapter 12.10's dependency problem on the Python side.

Recursion, (?R). Matches balanced structures — nested parentheses, nested braces. This is the one with a real use case in this field: a netlist or constraint file with nested brackets. But the right answer there is almost never a recursive regex; it is a small parser with a counter, which is clearer, debuggable and works in both languages. A regex that matches nested structures is a regex nobody will be able to modify.

\K. Drops everything matched so far from the result, so pin=\KU\d matches only the U1. A convenience, and the workaround is a capture group — which is what you would write in Perl anyway if the pattern were going to be read by anybody.

And one feature works in neither as written: (?{ code }), which embeds Perl code in a pattern. Perl requires use re 'eval' to enable it, and Python has no equivalent. It is a feature to recognise if you meet it in a legacy script, not one to use.

So the gap is four features, of which one is experimental even in Perl, one is for text in natural languages, one has a better non-regex answer, and one is a convenience with a one-step workaround.

4. What the Claim Gets Right

The engines being comparable does not make the two languages equally pleasant for this work, and it is worth being precise about the difference, because this is the part of the misconception that is pointing at something real.

In Perl a pattern is syntax; in Python it is a function call. Chapter 16.1 measured the smallest version of this — Python raising NameError: name 're' is not defined with nothing imported. The larger version is in the operators:

# Perl: the pattern is part of the statement
next unless /^UVM_ERROR/;
s/\s+$//;
my ($pin, $slack) = /(\S+)\s+(\S+)/;

# Python: the pattern is an argument to a function
if not re.match(r'^UVM_ERROR', line): continue
line = re.sub(r'\s+$', '', line)
m = re.search(r'(\S+)\s+(\S+)', line)
pin, slack = m.group(1), m.group(2)

Both are clear. The Perl is shorter, and in a loop that applies six patterns per line the difference accumulates — which is the measured 55-against-121 result from Chapter 16.4, seen from the inside.

Perl's $1, $_ and the implicit match target remove a lot of bookkeeping — and also create the hazards Part I measured, including Chapter 11.10's unguarded $1 holding a stale value after a failed match. The convenience and the trap are the same feature.

So the honest version of the claim is: Perl is more comfortable for dense line-by-line pattern work, and its engine is not meaningfully more capable. The first half is a reason to enjoy writing a one-liner in Perl. It is not a reason a two-hundred-line script cannot be migrated.

5. What To Remember

• This is the only one of the four misconceptions that is a technical claim, so it can be settled by running it.
• Measured: 11 of 16 features behave identically in Perl 5.34 and Python 3.13.
• Possessive quantifiers work in Python — added in 3.11. The folklore is out of date.
• Four features are Perl-only: variable-length lookbehind, \p{...}, (?R) recursion, and \K.
• One works in neither as written: (?{ code }) needs use re 'eval' in Perl and has no Python equivalent.
• Variable-length lookbehind is experimental in Perl too, so relying on it costs something in both languages.
• \p{...} matters for natural language, not for ASCII identifiers in a timing report.
• (?R) has a real use case and a better answer — a small parser with a counter, readable in either language.
• \K is a convenience; a capture group is the workaround and is more readable anyway.
• Translate a spelling, never a mechanism, when comparing: (?P<name>) and \Z are fair; anything else measures your transcription.
• The regex engine is almost never the reason a script stays in Perl. Look at the patterns — they will be the eleven identical rows.
• The real reasons are Chapter 16.2's: environment variables, external tools, exit conventions, no tests.
• What the claim gets right: in Perl a pattern is syntax, with its own operators and implicit target, which is genuinely more comfortable for dense line work.
• The convenience and the trap are the same feature — $1 surviving a failed match is Chapter 11.10's measured hazard.

Chapter 16.9 takes the last misconception, and the one with the most direct consequence for the reader. If new Perl is rarely written, what exactly is the value of learning it? The answer follows from Chapter 16.7's measurement rather than from any opinion about the language.