Named Groups for Parsers Others Can Read

Part II · Module 19 · Chapter 19.9 · Named Groups for Parsers Others Can Read

1. The Failure This Chapter Removes

A regression summary line, and a six-group pattern that reads it:

import re

ROW = "smoke_rw 1839274 PASS 4210 0 drv.sv"

PATTERN = r"(\w+) (\d+) (\w+) (\d+) (\d+) (\S+)"

m = re.search(PATTERN, ROW)
print("verdict:", m.group(3))
print("errors :", m.group(5))

verdict: PASS
errors : 0

Correct. Now the flow team prefixes every row with the run directory, so the input gains one field on the left and the pattern gains one group to read it:

import re

ROW = "r7 smoke_rw 1839274 PASS 4210 0 drv.sv"

PATTERN = r"(\S+) (\w+) (\d+) (\w+) (\d+) (\d+) (\S+)"

m = re.search(PATTERN, ROW)
print("verdict:", m.group(3))
print("errors :", m.group(5))

verdict: 1839274
errors : 4210

Nothing raised. Nothing warned. The pattern matched, both group() calls succeeded, and the script now reports a seed as a verdict and a cycle count as an error count. A nightly summary built on this would print 1839274 in the verdict column and an engineer would read past it, because a wrong number in a report looks exactly like a right one.

The mistake was not in the pattern — the new pattern is correct. It was that the two group() calls further down the file were written against a numbering that the pattern no longer has, and numbers are the only thing tying them together. Chapter 19.8 Section 5 named this; here is what it costs.

2. (?P<name>...)

The syntax is a prefix inside the opening parenthesis:

import re

LINE = "UVM_INFO top.sv(142) @ 0: [CFG] test=smoke_rw seed=1839274"

m = re.search(r"test=(?P<test>\w+) seed=(?P<seed>\d+)", LINE)

print("test:", m["test"])
print("seed:", int(m["seed"]) + 1)

test: smoke_rw
seed: 1839275

(?P<test>\w+) is an ordinary group with a label attached. The ?P<...> part is pure bookkeeping — it matches nothing, consumes nothing and changes nothing about which lines the pattern accepts. Everything inside after the > is the pattern, exactly as before.

Three ways to read a named group, all equivalent:

import re

m = re.search(r"(?P<key>\w+)=(?P<val>\d+)", "seed=1839274")

print("brackets :", m["key"])
print("group()  :", m.group("key"))
print("by number:", m.group(1))
print("groups() :", m.groups())
print("span     :", m.span("val"))

brackets : seed
group()  : seed
by number: seed
groups() : ('seed', '1839274')
span     : (5, 12)

m["key"] is the short form and the one to prefer. m.group("key") is identical and reads better when the name is in a variable. And crucially, the numbers still work — a named group is a numbered group that also has a name, which means naming groups is never a breaking change to code that already indexes them.

span() accepts a name too, which is how you locate a named field in the original line.

Here is the canonical two-line idiom from Chapter 19.8, rewritten:

m = re.search(r"(?P<test>\w+) seed=(?P<seed>\d+)", line)
m["test"], int(m["seed"])

The int() has not gone anywhere. A named group is still a string — the name changes how you reach the value, never what the value is.

3. Names Are Immune to Renumbering

Section 1's second pattern, with names:

import re

ROW = "r7 smoke_rw 1839274 PASS 4210 0 drv.sv"

PATTERN = (
    r"(?P<run>\S+) (?P<test>\w+) (?P<seed>\d+) "
    r"(?P<verdict>\w+) (?P<cycles>\d+) (?P<errors>\d+) (?P<src>\S+)"
)

m = re.search(PATTERN, ROW)
print("verdict:", m["verdict"])
print("errors :", int(m["errors"]))

verdict: PASS
errors : 0

The run group was added on the left, exactly as before, and m["verdict"] is still the verdict. There is no numbering to keep in sync, so there is nothing to get out of sync. The defect from Section 1 is not merely less likely here — it is unrepresentable.

Note the pattern itself: two adjacent string literals in parentheses, which Python joins into one. Python joins two string literals that sit next to each other with nothing but whitespace between them, which is the cleanest way to break a long pattern across lines — a seven-field pattern on one line is unreadable at any width, and \ continuations are worse. The trailing space at the end of the first fragment is load-bearing — a seven-field pattern split across two lines has to keep its separator.

4. groupdict(): The Whole Row at Once

Reaching for fields one at a time is fine for two. For six it is noise, and there is a better move:

import re

ROW = "r7 smoke_rw 1839274 PASS 4210 0 drv.sv"

PATTERN = (
    r"(?P<run>\S+) (?P<test>\w+) (?P<seed>\d+) "
    r"(?P<verdict>\w+) (?P<cycles>\d+) (?P<errors>\d+) (?P<src>\S+)"
)

m = re.search(PATTERN, ROW)
fields = m.groupdict()

print(type(fields).__name__)
for key, value in fields.items():
    print(f"  {key:<8} {value!r}")

dict
  run      'r7'
  test     'smoke_rw'
  seed     '1839274'
  verdict  'PASS'
  cycles   '4210'
  errors   '0'
  src      'drv.sv'

m.groupdict() returns a plain dictionary mapping each name to what it captured — every tool Module 8 built for dictionaries applies to it immediately. Note the iteration order: insertion order, which for a groupdict() is left-to-right pattern order, so the dict reads like the line it came from.

Two details worth noticing. It contains only the named groups — an unnamed group in the same pattern is reachable by number and absent from the dict. And every value is still a string, including '0', which is why the next section converts.

5. From a Match to a Structured Result

Chapter 16.14 argued that a function returning six loose values is harder to use than one returning a structured result, and Chapter 16.15 gave the tool. A named pattern plus a dataclass is where those two chapters meet a real log:

import re
from dataclasses import dataclass


@dataclass
class Issue:
    sev: str
    file: str
    line: int
    cycle: int


ISSUE = r"UVM_(?P<sev>ERROR|WARNING) (?P<file>\w+\.sv)\((?P<line>\d+)\) @ (?P<cycle>\d+):"

LOG = """UVM_INFO top.sv(142) @ 0: [CFG] test=smoke_rw seed=1839274
UVM_ERROR drv.sv(88) @ 12500: [WR] bad handshake
UVM_WARNING mon.sv(51) @ 99300: [SMP] late sample
UVM_ERROR drv.sv(104) @ 141900: [WR] timeout
"""

issues = []
for line in LOG.splitlines():
    m = re.search(ISSUE, line)
    if m is None:
        continue
    f = m.groupdict()
    issues.append(Issue(sev=f["sev"], file=f["file"], line=int(f["line"]), cycle=int(f["cycle"])))

for issue in issues:
    print(issue)

errors = 0
for issue in issues:
    if issue.sev == "ERROR":
        errors += 1
print(errors, "errors")

Issue(sev='ERROR', file='drv.sv', line=88, cycle=12500)
Issue(sev='WARNING', file='mon.sv', line=51, cycle=99300)
Issue(sev='ERROR', file='drv.sv', line=104, cycle=141900)
2 errors

That is a parser. Four lines of log in, three typed objects out, and the conversion from text to int happens exactly once, at the boundary where the text stops being text. Everything downstream — the count, a sort by cycle, a filter by severity — works on real integers.

The pattern borrows (ERROR|WARNING) from Chapter 19.10, which covers it properly; read it here as either word. And note that it is a named alternation: (?P<sev>ERROR|WARNING) captures whichever side matched under the name sev, so the choice and the label cost one group between them.

When every name in the pattern matches a field name, the construction collapses further:

import re
from dataclasses import dataclass


@dataclass
class Config:
    test: str
    seed: str
    host: str


m = re.search(r"test=(?P<test>\w+) seed=(?P<seed>\d+) host=(?P<host>\S+)",
              "[CFG] test=smoke_rw seed=1839274 host=farm_07")

cfg = Config(**m.groupdict())
print(cfg)
print(cfg.test, int(cfg.seed) % 1000)

Config(test='smoke_rw', seed='1839274', host='farm_07')
smoke_rw 274

Config(**m.groupdict()) is Chapter 11.6's keyword unpacking applied to the dict: each key becomes a keyword argument. It is elegant and it has one real cost, visible in the output — seed is still a string, because nothing in this form can convert anything. Use it when every field is genuinely text; use the explicit form from the previous example the moment one field is a number, which in VLSI reports it almost always is.

6. What Counts as a Name

The rules are narrow and worth knowing before a pattern refuses to compile:

import re

CANDIDATES = [r"(?P<sev>x)", r"(?P<slack_ns>x)", r"(?P<class>x)",
              r"(?P<1st>x)", r"(?P<test-name>x)", r"(?P<>x)"]

for pattern in CANDIDATES:
    try:
        re.compile(pattern)
        print(f"OK      {pattern}")
    except re.PatternError as exc:
        print(f"REJECT  {pattern:<18} {exc}")

OK      (?P<sev>x)
OK      (?P<slack_ns>x)
OK      (?P<class>x)
REJECT  (?P<1st>x)         bad character in group name '1st' at position 4
REJECT  (?P<test-name>x)   bad character in group name 'test-name' at position 4
REJECT  (?P<>x)            missing group name at position 4

A group name must be a valid Python identifier: letters, digits and underscores, not starting with a digit. So slack_ns is fine, 1st is not, and test-name is not — the hyphen that is perfectly normal in a tool's own field names has to become an underscore.

(?P<class>x) being accepted is the interesting one. class is a reserved keyword and class = 1 is a SyntaxError, but a group name never becomes a variable — it is a dictionary key and a lookup string, so the keyword rules do not reach it. You can still only read it as m["class"], never as an attribute.

Two names that collide are rejected outright, and this one is worth seeing because it fails at compile time rather than at match time:

import re

try:
    re.search(r"(?P<name>\w+)=(\d+) (?P<name>\w+)=(\d+)", "seed=1 iter=2")
except re.PatternError as exc:
    print(type(exc).__name__)
    print(exc)

PatternError
redefinition of group name 'name' as group 3; was group 1 at position 24

Reused names are a re.PatternError the moment the pattern is compiled, before it looks at a single character of data. That is the right behaviour — m["name"] could not possibly mean both — and it is a reminder that a regex is a small program with its own syntax errors, which is the point Chapter 19.3 made about escaping.

Asking for a name the pattern does not have is a different failure, and it is the same one Chapter 19.8 Section 4 produced for a bad number:

import re

m = re.search(r"(?P<key>\w+)=(?P<val>\d+)", "seed=1839274")

try:
    print(m["seed"])
except IndexError as exc:
    print("IndexError:", exc)

IndexError: no such group

m["seed"] looks reasonable and is wrong — seed is the value this pattern captured, and key and val are its group names. IndexError on a string subscript reads oddly until you remember m[...] is m.group(...) in disguise, not a dictionary lookup. For a real dictionary lookup, use m.groupdict() and then .get().

7. When Not to Bother

Named groups are not free: (?P<seed>\d+) is nine characters of overhead on a four-character pattern, and a line of regex that is mostly bookkeeping is harder to read, not easier. The honest rule is about distance and count.

Skip the name when the pattern has one group and the extraction is immediate — two lines apart, one obvious field:

import re

LINE = "Info: slack (VIOLATED)   -0.142"

m = re.search(r"(-?\d+\.\d+)", LINE)
if m is not None:
    print(float(m.group(1)))

-0.142

Nothing is clearer for being called m["slack"] here. The group(1) sits three characters from the pattern that defines it.

Use the name from two groups upward, and always when the pattern and its use are separated by anything — a function boundary, a module-level constant, a different file. Section 1's defect needs exactly two conditions: more than one group, and distance between the pattern and the group(n) call. Names remove the second condition's consequences entirely.

One more case, which is the one the canonical purpose names: a pattern someone else will maintain. m["slack"] tells a reader what the field is. m.group(3) tells them to go count parentheses, and the count they do today may not be the count that was true when the line was written.

8. Predict, Run, Observe

import re

LINE = "r7 smoke_rw seed=1839274 PASS"

m = re.search(r"(?P<run>\S+) (?P<test>\w+) seed=(?P<seed>\d+) (\w+)", LINE)

# A
print("A:", m["test"], m.group(2))

# B
print("B:", m.groupdict())

# C
print("C:", m.groups())

# D
print("D:", m.group(4), type(m["seed"]).__name__)

# E
try:
    print("E:", m["verdict"])
except IndexError as exc:
    print("E:", "IndexError:", exc)

The verified output:

A: smoke_rw smoke_rw
B: {'run': 'r7', 'test': 'smoke_rw', 'seed': '1839274'}
C: ('r7', 'smoke_rw', '1839274', 'PASS')
D: PASS str
E: IndexError: no such group

A — a named group is also a numbered group. m["test"] and m.group(2) are the same group reached two ways.

B — groupdict() has three entries, not four. The unnamed (\w+) is not in it.

C — groups() has four entries. It does not care whether a group has a name.

D — the unnamed group is reachable only by number, and seed is still a string. Section 2's rule.

E — verdict is a name the pattern never defined, so IndexError. The last group is unnamed; there is no way to reach it except as group 4.

9. Exercises

Exercise 1 — reproduce the defect. Build Section 1's six-group pattern and its two group() calls, confirm the right answers, then add a group on the left and record the two wrong answers. Write one sentence explaining why nothing raised.

Exercise 2 — fix it with names. Convert your Exercise 1 pattern to named groups and add the leading group again. Confirm both fields are still correct.

Exercise 3 — three ways in. For a two-group named pattern, print one field via m["x"], m.group("x") and m.group(n), and say why all three work.

Exercise 4 — the whole row. Write a five-field named pattern for a line of your own invention and print m.groupdict() one key per line, aligned.

Exercise 5 — named and unnamed together. Write a pattern with two named groups and one unnamed one. Predict the lengths of groups() and groupdict() before running, then check.

Exercise 6 — typed result. Define a dataclass with two str and two int fields, and populate it from a named match with the conversions in the right place. Print the object.

Exercise 7 — the ** form and its cost. Build a dataclass from Config(**m.groupdict()), then show that a numeric field arrives as a string. Say what you would change for a field a nightly flow compares numerically.

Exercise 8 — legal names. Test five candidate names including one starting with a digit and one containing a hyphen. Record which compile, and rewrite the two rejects as legal names.

Exercise 9 — duplicate name. Write a pattern that reuses a group name, catch the re.PatternError and print the message. Say at what moment the error occurred relative to the data being matched.

Exercise 10 — judgment. For each, say named or numbered and why: (a) a one-group pattern used on the next line; (b) a four-group pattern defined as a module constant and used in two functions; (c) a two-group pattern inside a one-off throwaway probe; (d) a pattern in a script three other engineers run nightly.

10. Summary

• (?P<name>...) is an ordinary group with a label, and the label matches nothing and changes nothing about what the pattern accepts.
• Read it as m["name"] or, equivalently, m.group("name"). span("name") works too.
• A named group is also a numbered group, so adding names never breaks code that indexes by number.
• A named group is still a string — int() and float() stay exactly where they were.
• Numbers shift when a group is added on the left; names do not. That is the whole argument, and Section 1 is what the shift costs: a complete, plausible, wrong report with no error.
• m.groupdict() is a plain dict of named groups only, in pattern order — every Module 8 tool applies.
• groups() includes unnamed groups; groupdict() does not.
• A named pattern plus a dataclass is a parser: text in, typed objects out, converted once at the boundary.
• Type(**m.groupdict()) is neat and cannot convert anything — use the explicit form when a field is numeric.
• Names must be Python identifiers — no leading digit, no hyphen — but keywords like class are allowed, because a group name never becomes a variable.
• A reused name is a re.PatternError at compile time, before any data is examined.
• An unknown name raises IndexError: no such group, because m[...] is group(...), not a dict lookup.
• Skip names for one obvious group used immediately; use them from two groups up, and always when the pattern and its use are separated.

The next chapter covers the construct this one borrowed twice: one pattern that accepts either of two words, and tolerates a field a tool prints only sometimes.