Compiling and Reusing Patterns

Part II · Module 19 · Chapter 19.16 · Compiling and Reusing Patterns

1. One Object Instead of a String

re.compile turns a pattern into a reusable object:

import re

UVM = re.compile(r"UVM_(?P<sev>ERROR|FATAL)")

print("type   :", type(UVM).__name__)
print("pattern:", UVM.pattern)
print("groups :", UVM.groups)
print("names  :", UVM.groupindex)

type   : Pattern
pattern: UVM_(?P<sev>ERROR|FATAL)
groups : 1
names  : {'sev': 1}

A Pattern object remembers the pattern text, how many groups it has and what their names are — UVM.groups and UVM.groupindex are the two facts Chapter 19.9 had to be careful about, now readable from the object rather than counted by eye.

And the canonical shape, which is how this is almost always written:

UVM = re.compile(r"UVM_(ERROR|FATAL)")
for line in f:
    if UVM.search(line):
        ...

A module-level constant, compiled once, used in a loop.

2. Every Function Becomes a Method

The five entry points from Chapter 19.11 are all available, with the pattern argument gone:

import re

UVM = re.compile(r"UVM_(?P<sev>ERROR|FATAL)")

TEXT = "UVM_ERROR a\nUVM_FATAL b\nUVM_ERROR c\n"

print("search   :", UVM.search(TEXT).group("sev"))
print("findall  :", UVM.findall(TEXT))
print("finditer :", len(list(UVM.finditer(TEXT))))
print("match    :", bool(UVM.match(TEXT)), bool(UVM.match("x UVM_ERROR")))
print("fullmatch:", bool(UVM.fullmatch("UVM_ERROR")))

search   : ERROR
findall  : ['ERROR', 'FATAL', 'ERROR']
finditer : 3
match    : True False
fullmatch: True

re.search(pattern, text) becomes PAT.search(text). Nothing else changes — same semantics, same return types, same None for the first three and same collection for the last two. Everything Chapters 19.2 to 19.15 established about patterns applies unchanged, because the pattern is the same pattern.

3. The Speed Claim, Honestly

The usual reason given for compiling is speed, and it is the weakest of the four.

re keeps an internal cache of patterns it has compiled. When you call re.search(r"UVM_ERROR", line) in a loop, the pattern string is compiled on the first call and looked up on every call after that — so the work you are supposedly avoiding is largely already avoided for you. What remains is a dictionary lookup per call, which is real but small, and re.purge() exists precisely because that cache is a cache.

This chapter publishes no measured number, deliberately. A timing figure here would depend on the machine, the Python build, the pattern, the line length and what else the box is doing, and a number that changes between two runs of the same script is not a fact a tutorial should assert. What is worth saying:

• For a pattern used a few times, the difference is not measurable in any way that matters.
• For a pattern used millions of times in a tight loop, compiling removes a per-call lookup and the effect is real but modest — single-digit percent territory on a scan whose cost is dominated by reading the file and by the match itself.
• The cache holds a bounded number of patterns. A program that generates many distinct pattern strings — one per instance path, say — can evict its own entries and then genuinely recompile on every call. That is the one case where compiling converts a real cost into no cost, and it is also a sign you should be building one pattern with alternation instead of many patterns.

If you want the number for your own workload, Exercise 4 shows how to measure it. On your machine, on your data, it means something.

4. Benefit One: The Pattern Is Validated Once, Early

This is the first argument that is fully observable. A pattern string with a syntax error is inert until something tries to use it:

import re

BROKEN = r"UVM_(?P<sev>ERROR|FATAL"

print("string built, nothing checked")

try:
    re.compile(BROKEN)
except re.PatternError as exc:
    print("re.PatternError:", exc)

string built, nothing checked
re.PatternError: missing ), unterminated subpattern at position 4

With re.compile at module level, that error happens when the module is imported — before the script opens a file, before it processes a line, at a moment when the traceback points straight at the pattern.

Pass the same string around and call re.search with it deep inside a loop, and the error happens the first time that branch runs. On a parser where one pattern only applies to fatal messages, that can be the first fatal of the quarter — Chapter 14.1's distinction between a syntax error and a runtime error, applied to a pattern: re.compile converts a late runtime failure into an early one.

The same logic covers Chapter 19.15's fixed-width lookbehind and Chapter 19.9's duplicate group name. All three are compile-time errors, and compiling at module level is what makes "compile time" mean "import time" rather than "whenever we next hit that line".

5. Benefit Two: The Flags Cannot Be Forgotten

Chapter 19.14 Section 3 showed the worst bug in that chapter: ^warning needs both IGNORECASE and MULTILINE, and each flag alone returns one plausible, wrong result. With patterns passed as strings, every call site has to remember both:

import re

TEXT = "Warning: a\nWARNING: b\nwarning: c\n"
PAT = r"^warning"


def count_careful(text):
    return len(re.findall(PAT, text, re.IGNORECASE | re.MULTILINE))


def count_forgot(text):
    return len(re.findall(PAT, text, re.MULTILINE))


WARN = re.compile(PAT, re.IGNORECASE | re.MULTILINE)


def count_compiled(text):
    return len(WARN.findall(text))


print("careful :", count_careful(TEXT))
print("forgot  :", count_forgot(TEXT))
print("compiled:", count_compiled(TEXT))

careful : 3
forgot  : 1
compiled: 3

Three functions, one shared pattern string, two different answers. count_forgot is a single omitted flag and it reports one warning in a report containing three — and it is exactly the kind of difference that appears when a second function is added to a module six months later by someone copying the call next to it.

A compiled pattern carries its flags. WARN.findall(text) cannot be called with the wrong flags because there is nowhere to put them, so the pattern and its flags travel as one thing. That is a structural fix rather than a discipline, and on a module with four patterns and ten call sites it is the strongest argument in this chapter.

You can ask a compiled pattern what it carries:

import re

WARN = re.compile(r"^warning", re.IGNORECASE | re.MULTILINE)
PLAIN = re.compile(r"^warning")

print("WARN  ignorecase:", bool(WARN.flags & re.IGNORECASE))
print("WARN  multiline :", bool(WARN.flags & re.MULTILINE))
print("PLAIN ignorecase:", bool(PLAIN.flags & re.IGNORECASE))

WARN  ignorecase: True
WARN  multiline : True
PLAIN ignorecase: False

.flags & re.IGNORECASE is Chapter 3.7's bitwise-and used as a membership test, and it is the readable way to check — printing .flags directly gives you a number whose spelling is an implementation detail.

6. Benefit Three: pos and endpos — and the Hazard They Bring

Compiled methods take two extra arguments that the module-level functions do not have: a start and an end position to search within.

import re

UVM = re.compile(r"UVM_(ERROR|FATAL)")

TEXT = "UVM_ERROR a\nUVM_FATAL b\nUVM_ERROR c\n"

print("whole subject :", UVM.search(TEXT).span())
print("from char 12  :", UVM.search(TEXT, 12).span())
print("up to char 11 :", UVM.search(TEXT, 0, 11).span())
print("first 5 chars :", UVM.search(TEXT, 0, 5))

whole subject : (0, 9)
from char 12  : (12, 21)
up to char 11 : (0, 9)
first 5 chars : None

pos and endpos restrict the search to a slice without copying the string, which is how you scan a window of a large buffer repeatedly. They are also not equivalent to slicing: UVM.search(TEXT[12:]) would give you a match whose span is relative to the slice, while pos keeps the spans in the original coordinates — which is what you want when those spans are going into a report.

Now the hazard, which exists because pos exists:

import re

WARN = re.compile(r"^warning", re.IGNORECASE | re.MULTILINE)

TEXT = "Warning: a\nWARNING: b\nwarning: c\n"

print("correct       :", WARN.findall(TEXT))
print("flag passed in:", WARN.findall(TEXT, re.MULTILINE))
print("int value of M:", int(re.MULTILINE))
print("same as pos=8 :", WARN.findall(TEXT, 8))

correct       : ['Warning', 'WARNING', 'warning']
flag passed in: ['WARNING', 'warning']
int value of M: 8
same as pos=8 : ['WARNING', 'warning']

WARN.findall(TEXT, re.MULTILINE) does not raise. A flag is an integer — re.MULTILINE is 8 — and the second argument of a compiled method is pos, so the call means start searching at character 8. The first eight characters are skipped, the first match is lost, and the result is a shorter list that looks entirely normal.

This is the most dangerous single line in the chapter, and it is the direct consequence of converting a string call to a method call without deleting the flags argument:

re.findall(PAT, text, re.MULTILINE)   # flags  — correct
PAT.findall(text, re.MULTILINE)       # pos=8  — silently wrong

The migration rule: when you move a pattern into re.compile, the flags move with it and the call sites lose their third argument. If a compiled-method call has a re. constant in it, that is a bug.

7. The Real Win: A Named Pattern at the Top of the File

The argument the canonical purpose puts first is readability, and this is what it looks like in a module:

import re

SEVERITY = re.compile(r"UVM_(?P<sev>ERROR|FATAL|WARNING)")
LOCATION = re.compile(r"(?P<file>\w+\.\w+)\((?P<line>\d+)\)")
CYCLE = re.compile(r"@ (?P<cycle>\d+):")

LOG = """UVM_INFO top.sv(142) @ 0: [CFG] test=smoke_rw
UVM_ERROR drv.sv(88) @ 12500: [WR] bad handshake
UVM_WARNING mon.sv(51) @ 99300: [SMP] late sample
UVM_ERROR drv.sv(104) @ 141900: [WR] timeout
"""


def issues(text):
    """Every non-info UVM message, as (severity, file, line, cycle) tuples."""
    rows = []
    for line in text.splitlines():
        sev = SEVERITY.search(line)
        if sev is None:
            continue
        where = LOCATION.search(line)
        when = CYCLE.search(line)
        if where is None or when is None:
            continue
        rows.append((sev["sev"], where["file"], int(where["line"]), int(when["cycle"])))
    return rows


for severity, src, num, cycle in issues(LOG):
    print(f"{severity:<8} {src:<8} {num:>4} {cycle:>7}")

ERROR    drv.sv     88   12500
WARNING  mon.sv     51   99300
ERROR    drv.sv    104  141900

Three named patterns at the top, three uses in one function, and a reader who opens this file sees immediately what it knows how to recognise. Compare the same function with three raw strings inline: the reader has to read the regex to learn the field's name, and the names SEVERITY, LOCATION and CYCLE would exist only as comments if at all.

Four properties of that arrangement are worth naming.

The patterns are a declared interface. A new tool version that renames a field changes one line near the top, not a line buried in a loop.

Each has one job. Three small patterns are easier to test individually than one twelve-group pattern, which is what Chapter 19.18 builds on.

The flags, if any, are attached to the pattern they belong to, not repeated at each call.

A syntax error in any of them stops the module at import. Section 4, in its useful form.

8. Predict, Run, Observe

import re

PAT = re.compile(r"^err", re.IGNORECASE | re.MULTILINE)
TEXT = "ERR one\nmid\nerr two\n"

# A
print("A:", PAT.findall(TEXT))

# B
print("B:", PAT.findall(TEXT, 4))

# C
print("C:", PAT.findall(TEXT, re.MULTILINE))

# D
print("D:", PAT.pattern, PAT.groups)

# E
print("E:", bool(PAT.flags & re.IGNORECASE), bool(PAT.flags & re.DOTALL))

# F
print("F:", PAT.search(TEXT).span(), PAT.search(TEXT, 4).span())

# G
try:
    re.compile(r"(?P<a>x)(?P<a>y)")
    print("G: compiled")
except re.PatternError as exc:
    print("G:", exc)

The verified output:

A: ['ERR', 'err']
B: ['err']
C: ['err']
D: ^err 0
E: True False
F: (0, 3) (12, 15)
G: redefinition of group name 'a' as group 2; was group 1 at position 12

A — both flags are carried by the object, so both spellings on their own lines are found.

B — pos=4 skips the first line entirely.

C — identical to B, because re.MULTILINE is the integer 8 and became pos=8. Section 6's hazard: no error, one match gone.

D — the object remembers its text and that it has no groups.

E — the flags it carries, tested with &.

F — pos keeps spans in the original string's coordinates, so the second match reports 12 rather than 8.

G — a compile-time error, raised by re.compile with no data involved at all. Section 4.

9. Exercises

Exercise 1 — compile and inspect. Compile a two-group named pattern and print .pattern, .groups and .groupindex. Say which of the three you would have had to count by hand.

Exercise 2 — every method. Call all five methods on one compiled pattern and confirm each matches the behaviour Chapter 19.11 described for its function form.

Exercise 3 — not inside the loop. Write a loop that compiles its pattern on every iteration, then move the compile out. Say what the first version paid for that the second does not.

Exercise 4 — measure it yourself. Using time.perf_counter, compare a compiled and an uncompiled scan over at least a million generated lines, on your machine. Report your two numbers and the ratio — and then say why this chapter did not print a number of its own.

Exercise 5 — early failure. Build a pattern string with an unbalanced parenthesis. Show that constructing the string is silent and that re.compile is not. Say where in a real script you would want that error to appear.

Exercise 6 — the forgotten flag. Reproduce Section 5's three functions. Then write one sentence explaining how the compiled version makes count_forgot impossible to write.

Exercise 7 — ask the object. Check with & whether a compiled pattern carries IGNORECASE, and explain why you would not print .flags directly in a report.

Exercise 8 — pos is not slicing. Search with pos=n and search in text[n:]. Print both spans and say which one you would want in an error message that quotes a character offset.

Exercise 9 — the silent hazard. Convert an re.findall(PAT, text, re.MULTILINE) call to the compiled form and leave the flag argument in place. Record the result, the integer value of the flag, and the equivalent explicit pos call.

Exercise 10 — a parser module. Rewrite one of your own scripts so every pattern is a named module-level constant. List what a new reader learns from the top of the file that they could not learn before.

Exercise 11 — when not to. Find a pattern in your own code that should not be compiled, and justify it in two sentences.

10. Summary

• re.compile returns a Pattern object that remembers the pattern text, its group count and its group names.
• Every function becomes a method with the pattern argument dropped: re.search(p, t) → PAT.search(t). Semantics are unchanged.
• The speed argument is the weakest one. re caches compiled patterns internally, so the saving for a looped pattern is a per-call lookup.
• No timing number is published here, because it would be machine-specific — measure it on your own workload if you need it.
• Never compile inside the loop. That form pays for the object construction on every iteration.
• Benefit one: the pattern is validated once, early. With a module-level constant, a bad pattern fails at import rather than the first time a rare branch runs.
• Benefit two: the flags cannot be forgotten. A compiled pattern carries them, which makes Chapter 19.14's two-single-flag bug structurally impossible.
• Check carried flags with PAT.flags & re.IGNORECASE, not by printing .flags.
• Benefit three: pos and endpos restrict the search without copying, and keep spans in the original coordinates.
• The hazard that comes with them: PAT.findall(text, re.MULTILINE) does not raise — the flag's integer value becomes a start position and a match disappears.
• When you move a pattern into re.compile, the flags move with it and every call site loses its third argument.
• The real win is readability: named patterns at the top of a module are a declared interface, each with one job and its own flags.
• Do not compile a pattern used once inside the function that owns it. Compile when it is shared, needs flags, or deserves a name.

The next chapter is about what to do when a pattern compiles, runs, returns a result and is still wrong.