When Not to Use Regex

Part II · Module 19 · Chapter 19.19 · When Not to Use Regex

1. Match the Tool to the Shape of the Text

The question to ask before writing a pattern is not can a regex do this — it almost always can — but what shape is this text already in:

Everything below is one box of that figure, with the run that justifies it.

2. str.split: One Fixed Separator

A regression summary row, parsed two ways:

import re

ROW = "smoke_rw,1839274,PASS,4210,0"

by_split = ROW.split(",")
by_regex = re.search(r"^([^,]+),([^,]+),([^,]+),([^,]+),([^,]+)$", ROW).groups()

print("split:", by_split)
print("regex:", list(by_regex))
print("same :", by_split == list(by_regex))

split: ['smoke_rw', '1839274', 'PASS', '4210', '0']
regex: ['smoke_rw', '1839274', 'PASS', '4210', '0']
same : True

Identical. One is eleven characters and one is thirty-nine, and the thirty-nine-character version has five numbered groups that Chapter 19.8 Section 5 warned about, a count that must match the file format, and nothing whatever to show for it.

split also tells you something the regex does not:

ROW = "smoke_rw,1839274,PASS,4210,0"
SHORT = "smoke_rw,1839274,PASS"

print("fields:", len(ROW.split(",")))
print("short :", len(SHORT.split(",")))
print("limit :", ROW.split(",", 2))

fields: 5
short : 3
limit : ['smoke_rw', '1839274', 'PASS,4210,0']

A field count is a check — if len(fields) != 5: ... catches a truncated row immediately, where a five-group regex just returns None and leaves you with Chapter 19.17's four-cause triage. And split(sep, maxsplit) handles the first two fields and the rest, which is Chapter 19.13's greedy-versus-lazy question answered by a second argument.

Use a regex here only if the separator itself varies — a file that mixes commas and semicolons, or pads them with whitespace, is a real reason to reach for a pattern. Re-describing a comma as r"([^,]+)" is not.

3. str.partition: One Key and One Value

A key=value line where the value may itself contain the separator:

import re

LINE = "note=a=b=c"

print("partition:", LINE.partition("="))
print("split 1  :", LINE.split("=", 1))
print("regex    :", re.search(r"(\w+)=(.*)", LINE).groups())
print("split all:", LINE.split("="))

partition: ('note', '=', 'a=b=c')
split 1  : ['note', 'a=b=c']
regex    : ('note', 'a=b=c')
split all: ['note', 'a', 'b', 'c']

Three of those four are correct and one is the bug. partition splits at the first occurrence and returns three parts — before, the separator itself, and everything after — so the value arrives intact. split("=", 1) does the same in two parts. The regex does the same in nine more characters. And split("=") with no limit is the version that destroys the value, which is the mistake partition exists to make impossible.

rpartition is the same from the right, which is Chapter 19.13's instance-path split with no quantifier at all:

PATH = "uvm_test_top.env.agt.drv"

print("rpartition:", PATH.rpartition("."))
print("partition :", PATH.partition("."))

rpartition: ('uvm_test_top.env.agt', '.', 'drv')
partition : ('uvm_test_top', '.', 'env.agt.drv')

Those are exactly the two splits that Chapter 19.13 Section 6 produced with ^(.*)\.(\w+)$ and ^(.*?)\.(.*)$ — the parent-and-leaf split and the root-and-rest split. Here the choice between them is the name of the method, which is a thing a reader cannot misread, where the choice between .* and .*? is a thing that takes a paragraph to explain.

4. csv and json: Already Structured

The canonical annotation for this chapter is a rule and two lines:

# already structured — do not regex it
row = next(csv.DictReader(f))
cfg = json.load(f)

For CSV, this is not a readability argument. It is a correctness argument, and it is the strongest one in the chapter — because a CSV field may be quoted and contain the separator:

test,seed,note,status
smoke_rw,1839274,"timeout, retry pending",FAIL
burst_wr,771002,clean,PASS

import csv
import re

with open("regress.csv", encoding="utf-8") as handle:
    lines = handle.read().splitlines()

tricky = lines[1]

print("raw      :", tricky)
print("split    :", tricky.split(","))
print("regex    :", re.findall(r"([^,]+)", tricky))

raw      : smoke_rw,1839274,"timeout, retry pending",FAIL
split    : ['smoke_rw', '1839274', '"timeout', ' retry pending"', 'FAIL']
regex    : ['smoke_rw', '1839274', '"timeout', ' retry pending"', 'FAIL']

The regex and the naive split return the same five wrong fields. That is the whole point of this section: a regex is not a step up from split here, it is the same mistake written with more syntax. Both broke the quoted note in half, both produced a stray " at each end, and both turned a four-field row into five — so a script using either would read FAIL into the wrong column or fail a field-count check for the wrong reason.

The module that knows the format gets it right:

test,seed,note,status
smoke_rw,1839274,"timeout, retry pending",FAIL
burst_wr,771002,clean,PASS

import csv

with open("regress.csv", newline="", encoding="utf-8") as handle:
    for row in csv.DictReader(handle):
        print(f"{row['test']:<10} {row['status']:<5} {row['note']!r}")

smoke_rw   FAIL  'timeout, retry pending'
burst_wr   PASS  'clean'

Four fields, the note intact, and the fields reachable by name from the header row — which is Chapter 19.9's argument for named groups, obtained for free because the file already carried its own field names.

JSON is the same argument with a sharper edge, because a key can appear at more than one level:

{
  "test": "smoke_rw",
  "timeout": 7200,
  "stages": {
    "compile": {"timeout": 600},
    "sim": {"timeout": 14400}
  }
}

import json
import re

with open("flow.json", encoding="utf-8") as handle:
    text = handle.read()

cfg = json.loads(text)

print("json  top   :", cfg["timeout"], type(cfg["timeout"]).__name__)
print("json  sim   :", cfg["stages"]["sim"]["timeout"])
print("regex first :", re.search(r'"timeout":\s*(\d+)', text).group(1))
print("regex all   :", re.findall(r'"timeout":\s*(\d+)', text))

json  top   : 7200 int
json  sim   : 14400
regex first : 7200
regex all   : ['7200', '600', '14400']

Three timeout values in one file at three nesting levels. re.search returns the first one, which happens to be the right answer here and would be the wrong one if the file's key order changed — and nothing in a regex can express the timeout inside stages.sim, because nesting is not something a pattern can see. json.loads gives a nested dictionary where cfg["stages"]["sim"]["timeout"] says precisely which value it wants.

And the types: json gives int, a regex gives str. Chapter 19.8 Section 3's conversion is work that the right tool simply does not create.

5. A Stateful Parser: What Regex Cannot Do

Everything so far has been regex works but something else is better. This section is different.

An STA report where the slack value belongs to the path whose startpoint and endpoint precede it:

import re

REPORT = """Startpoint: u_core/u_alu/acc_reg[7]
Endpoint: u_core/u_fifo/wptr_reg[3]
  slack (VIOLATED)          -0.142
Startpoint: u_core/u_ctl/state_reg[1]
Endpoint: u_core/u_ctl/done_reg
  slack (MET)                0.031
Startpoint: u_core/u_dp/mul_reg[2]
Endpoint: u_core/u_dp/acc_reg[0]
  slack (VIOLATED)          -1
"""

print("slacks     :", re.findall(r"slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+(?:\.\d+)?)", REPORT))
print("startpoints:", re.findall(r"^Startpoint: (\S+)", REPORT, re.MULTILINE))

slacks     : ['-0.142', '0.031', '-1']
startpoints: ['u_core/u_alu/acc_reg[7]', 'u_core/u_ctl/state_reg[1]', 'u_core/u_dp/mul_reg[2]']

Two lists, three items each, in the right order — and that is all a regex can give you. To report the worst path you need the slack and the startpoint associated, and the only thing joining them is their position in the file. Pairing the lists by index works until one path omits its endpoint, or a header line appears mid-report, or a path has two slack lines for setup and hold, and then the two lists are different lengths and every pairing after that point is silently wrong.

The reason is structural, not a gap in your pattern-writing: a regex has no memory of which block it is inside. It is one expression evaluated at a position, with no variables.

A loop has variables:

import re

REPORT = """Startpoint: u_core/u_alu/acc_reg[7]
Endpoint: u_core/u_fifo/wptr_reg[3]
  slack (VIOLATED)          -0.142
Startpoint: u_core/u_ctl/state_reg[1]
Endpoint: u_core/u_ctl/done_reg
  slack (MET)                0.031
Startpoint: u_core/u_dp/mul_reg[2]
Endpoint: u_core/u_dp/acc_reg[0]
  slack (VIOLATED)          -1
"""

START = r"^Startpoint: (\S+)"
END = r"^Endpoint: (\S+)"
SLACK = r"slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+(?:\.\d+)?)"

paths = []
start = None
end = None

for line in REPORT.splitlines():
    m = re.match(START, line)
    if m is not None:
        start = m.group(1)
        continue
    m = re.match(END, line)
    if m is not None:
        end = m.group(1)
        continue
    m = re.search(SLACK, line)
    if m is not None:
        paths.append((start, end, float(m.group(1))))
        start = None
        end = None

for start, end, value in paths:
    print(f"{value:>8.3f}  {start} -> {end}")

worst = paths[0]
for path in paths:
    if path[2] < worst[2]:
        worst = path
print(f"worst   : {worst[2]:.3f} on {worst[0]}")

  -0.142  u_core/u_alu/acc_reg[7] -> u_core/u_fifo/wptr_reg[3]
   0.031  u_core/u_ctl/state_reg[1] -> u_core/u_ctl/done_reg
  -1.000  u_core/u_dp/mul_reg[2] -> u_core/u_dp/acc_reg[0]
worst   : -1.000 on u_core/u_dp/mul_reg[2]

Three patterns, nine lines of loop, and the answer a timing engineer actually wants — the worst slack and the path it is on. Note that this is not an argument against regex: the three patterns are doing exactly what Chapters 19.7 and 19.12 built them for. The regex recognises the lines; the loop remembers the context. Those are two jobs, and giving the second one to a pattern is what produces the thousand-character monsters that Chapter 19.12's symbol audit cannot explain.

The start = None reset after each path matters too. Without it, a path whose startpoint line is missing silently inherits the previous path's startpoint, which is the index-pairing bug moved inside the loop.

6. Nesting Is Not Something a Pattern Can Count

One hard limit, worth seeing once because it explains the shape of the previous section:

import re

EXPR = "(a + (b * (c - d)))"

print("greedy   :", re.search(r"\(.*\)", EXPR).group(0))
print("innermost:", re.search(r"\(([^()]*)\)", EXPR).group(1))

greedy   : (a + (b * (c - d)))
innermost: c - d

A pattern can find the outermost pair of brackets and it can find a pair with nothing inside it. It cannot find the contents of the second level, and no amount of quantifier tuning changes that — matching balanced brackets requires counting, and counting requires memory, which is the same thing Section 5 was missing.

This comes up in real work: a constraint block in a SystemVerilog file, a nested Tcl command, a {} group in a tool script. The answer is the same every time — a loop with a depth counter, or a parser for that language. Not a longer pattern.

7. Where Regex Is Right

The module would be dishonest without this section. Three jobs regex is genuinely the best tool for:

import re

LINES = [
    "UVM_ERROR drv.sv(88) @ 12500: [WR] bad handshake",
    "slack (VIOLATED)          -0.142",
    "test=smoke_rw seed=1839274 host=farm_07",
]

# 1. a value embedded in free text, at no fixed position
print("1:", re.search(r"seed=(\d+)", LINES[2]).group(1))

# 2. one pattern accepting several spellings
print("2:", re.search(r"UVM_(ERROR|FATAL|WARNING)", LINES[0]).group(1))

# 3. a field whose shape is a rule, not a position
print("3:", re.search(r"(-?\d+(?:\.\d+)?)", LINES[1]).group(1))

1: 1839274
2: ERROR
3: -0.142

A value embedded in free text has no index to slice and no separator to split on — seed= could be anywhere on the line, and that is the case str methods cannot reach.

Several accepted spellings in one pass is Chapter 19.10, and the alternative is three in tests and a chain of elif.

A field defined by its shape — signed, optional fraction — is a rule rather than a location, and \d and ? state rules.

And below all of them sit the three cheapest tools, which are worth reaching for first:

PATH = "runs/smoke_1234/logs/sim.log"
LINE = "UVM_ERROR drv.sv(88) @ 12500: bad handshake"

print("endswith:", PATH.endswith(".log"))
print("in      :", "UVM_ERROR" in LINE)
print("startswith:", LINE.startswith("UVM_"))

endswith: True
in      : True
startswith: True

re.search(r"\.log$", path) and path.endswith(".log") answer the same question, and one of them needs no knowledge of anchors, escaping or raw strings to read. Chapter 19.1 opened this module by saying a substring test is the right first move; nineteen chapters later that is still true, and it is worth more now that you know what the alternative costs.

8. Predict, Run, Observe

import re

ROW = "burst_wr,771002,clean,PASS"
KV = "args=-x -y --flag=2"
PATH = "uvm_test_top.env.agt.drv"

# A
print("A:", ROW.split(",") == list(re.findall(r"[^,]+", ROW)))

# B
print("B:", KV.partition("="))

# C
print("C:", KV.split("="))

# D
print("D:", PATH.rpartition(".")[2], PATH.partition(".")[0])

# E
print("E:", "no-equals-here".partition("="))

# F
print("F:", re.search(r"\(.*\)", "(a (b) c)").group(0))

# G
print("G:", "sim.log".endswith(".log"), bool(re.search(r"\.log$", "sim.log")))

The verified output:

A: True
B: ('args', '=', '-x -y --flag=2')
C: ['args', '-x -y --flag', '2']
D: drv uvm_test_top
E: ('no-equals-here', '', '')
F: (a (b) c)
G: True True

A — the regex and split agree on a clean delimited row, so the regex bought nothing.

B — partition splits once and the value keeps its own =.

C — split with no limit destroys the value. The defect partition prevents.

D — leaf from the right, root from the left, with no quantifier involved.

E — no separator is not an error: the whole string and two empty strings, so if sep == "" is the test.

F — the greedy match takes the outermost brackets; a pattern cannot count nesting.

G — identical answers, and one of the two needs no regex knowledge to read.

9. Exercises

Exercise 1 — the regex that bought nothing. Parse a five-field delimited row with split and with a five-group anchored pattern. Confirm they agree and count the characters in each.

Exercise 2 — the field count check. Add a truncated row to Exercise 1. Show what split tells you and what the regex tells you, and say which is easier to act on.

Exercise 3 — maxsplit. Use split(sep, 1) to take the first field and the rest. Say which chapter's greedy-versus-lazy decision you just avoided.

Exercise 4 — partition the awkward value. Parse a key=value line whose value contains =, three ways: partition, split with a limit, and split without one. Say which is the bug.

Exercise 5 — both ends. Split a four-level instance path with rpartition and partition. Name the Chapter 19.13 patterns these replace.

Exercise 6 — no separator. Call partition on a line without the separator and record all three parts. Write the one-line test for absence.

Exercise 7 — the quoted comma. Build a CSV row whose field contains the separator inside quotes. Parse it with split, with r"([^,]+)" and with csv.reader. Report all three field counts and say what the first two have in common.

Exercise 8 — by name. Read the same file with csv.DictReader and access two fields by name. Say what this gives you that Chapter 19.9's named groups gave you, and where the names came from.

Exercise 9 — the repeated key. Write a JSON config with the same key at two nesting levels. Show re.findall returning both and say how you would express "the inner one" as a pattern.

Exercise 10 — types. For that config, print the type of a numeric value from json.loads and from a regex group. Say where the conversion goes in each case.

Exercise 11 — the association. Take a three-path STA extract and extract the slacks and the startpoints as two lists. Then break the extract by deleting one endpoint line, and say what index-pairing would now report.

Exercise 12 — a stateful parser. Write the loop that associates them, including the reset. Then remove the reset, re-run on your broken extract, and record the wrong answer.

Exercise 13 — nesting. Try to extract the middle level of a three-deep bracketed expression with a pattern. Say what you would write instead.

Exercise 14 — the three right cases. For a line of your own, write one extraction a regex is genuinely best for and one that endswith or in answers better. Justify both in a sentence.

10. Summary

• Ask what shape the text is already in before asking what pattern would match it.
• str.split beats a regex on a fixed separator — the same fields, a third of the characters, and a field count you can check.
• split(sep, maxsplit) answers Chapter 19.13's greedy question with an argument instead of a quantifier.
• str.partition splits once and keeps the rest intact, which is what a key=value line with = in the value needs; split with no limit is the bug it prevents.
• rpartition and partition are the parent-and-leaf and root-and-rest splits from Chapter 19.13, chosen by method name rather than by quantifier.
• partition never raises — a missing separator gives the whole string and two empties.
• On a quoted CSV field containing the separator, a regex returns the same wrong fields as a naive split. It is not a step up; it is the same mistake with more syntax.
• csv.DictReader gives fields by name, which is Chapter 19.9's benefit obtained from the file's own header.
• A JSON key can appear at several nesting levels, and no pattern can say which one it wants — json.load gives a nested dict and the right types.
• A regex has no memory of which block it is inside, so it cannot associate a value with the header above it. Two lists paired by index break the moment a block is irregular.
• Let the patterns recognise the lines and let a loop remember the context — and reset the context variables after each record.
• Nesting cannot be counted by a pattern. Balanced brackets need a depth counter or a real parser.
• Regex is right for three jobs: a value embedded in free text at no fixed position, several accepted spellings in one pass, and a field whose shape is a rule rather than a location.
• Below all of it: in, startswith, endswith. Chapter 19.1 said a substring test is the right first move, and knowing the alternative makes that advice worth more, not less.

That closes Module 19. You can now read a pattern someone else wrote, build one that survives a tool upgrade, test it before it reaches a nightly, debug it when the report comes back strange — and decide not to write one at all. The next module puts all of it to work on the file every verification engineer reads most: a simulation log, parsed from a five-line checker into a tool that streams a multi-gigabyte file and never reports a broken run as a pass.