Alternation and Optional Parts

Part II · Module 19 · Chapter 19.10 · Alternation and Optional Parts

1. Either This or That

| inside a group means one of these:

import re

LINES = [
    "UVM_ERROR drv.sv(88) @ 12500: bad handshake",
    "UVM_FATAL env.sv(12) @ 0: no agent built",
    "UVM_WARNING mon.sv(51) @ 99300: late sample",
    "UVM_INFO top.sv(142) @ 0: starting test",
]

for line in LINES:
    m = re.search(r"UVM_(ERROR|FATAL)", line)
    if m is None:
        print("skip   ", line[:20])
    else:
        print("counted", m.group(1))

counted ERROR
counted FATAL
skip    UVM_WARNING mon.sv(5
skip    UVM_INFO top.sv(142)

(ERROR|FATAL) matches ERROR or FATAL and nothing else. The pipe separates alternatives, and the group around them does two jobs at once: it bounds which part of the pattern the choice applies to, and it captures whichever alternative won — which is why m.group(1) can tell you which severity was found, not merely that one was.

There is no limit on the number of alternatives:

import re

for line in ["UVM_ERROR drv.sv", "UVM_FATAL env.sv", "UVM_WARNING mon.sv", "UVM_INFO top.sv"]:
    m = re.search(r"UVM_(ERROR|FATAL|WARNING)", line)
    print(f"{line:<20} {m.group(1) if m else '-'}")

UVM_ERROR drv.sv     ERROR
UVM_FATAL env.sv     FATAL
UVM_WARNING mon.sv   WARNING
UVM_INFO top.sv      -

One pass over the log, three severities recognised, each one identifiable from the group. Before this chapter that needed three separate searches or three in tests.

2. Alternatives Can Be Patterns, Not Just Words

Each alternative is a full sub-pattern, so everything from Chapters 19.4 to 19.7 is available inside one:

import re

FIELDS = ["0x40001000", "1839274", "-0.142", "smoke_rw"]

NUMBER = r"(0x[0-9a-fA-F]+|-?\d+\.\d+|\d+)"

for field in FIELDS:
    m = re.search(r"^" + NUMBER + r"$", field)
    print(f"{field:<12} {m.group(1) if m else 'not a number'}")

0x40001000   0x40001000
1839274      1839274
-0.142       -0.142
smoke_rw     not a number

Three shapes of number — hex, signed decimal, plain integer — accepted by one pattern, each written with the character classes and quantifiers already covered. Note that NUMBER is an ordinary Python string being concatenated with anchors, which is Chapter 19.3's point that a pattern is just text until re compiles it.

The order of those three alternatives is not arbitrary. Section 4 is about why.

3. Alternation Binds Loosest of Everything

This is the first of the chapter's two traps, and it is the one that catches people on their very first alternation:

import re

m = re.search(r"UVM_ERROR|FATAL", "UVM_FATAL mon.sv(9) @ 400: no driver")

print("matched:", repr(m.group(0)))
print("span   :", m.span())

matched: 'FATAL'
span   : (4, 9)

The pattern looks like UVM_ followed by either ERROR or FATAL. It is not. | has the lowest precedence of any regex construct, lower than concatenation, so UVM_ERROR|FATAL means:

(UVM_ERROR)  or  (FATAL)

The second alternative is the bare word FATAL, which is present inside UVM_FATAL starting at offset 4 — and the span proves it, because a match of UVM_FATAL would start at 0. The pattern matched, reported success, and skipped the four characters that identified it as a UVM message.

On this line the mistake is harmless. On a line that merely mentions the word in prose it is not:

import re

PROSE = "note: a FATAL in the previous run was demoted to a warning"

print("loose :", re.search(r"UVM_ERROR|FATAL", PROSE))
print("fixed :", re.search(r"UVM_(ERROR|FATAL)", PROSE))

loose : <re.Match object; span=(8, 13), match='FATAL'>
fixed : None

The loose pattern finds a FATAL in a sentence explaining that there was not one, and a severity counter built on it over-reports every night. The fixed pattern requires the UVM_ prefix on both sides of the choice, which is what was meant.

Parentheses around an alternation are not optional punctuation. They are what limits the choice to the part you intended, and the rule to carry is simple: if a pattern contains a |, the alternation extends as far left and as far right as it possibly can, until it hits a parenthesis or the end of the pattern.

4. Order Matters: The First Alternative That Fits Wins

The second trap produces no visible symptom at all. A regression tool that reports three statuses:

import re

ROWS = ["status=PASS", "status=FAIL", "status=FAIL_TIMEOUT"]

for row in ROWS:
    m = re.search(r"status=(PASS|FAIL|FAIL_TIMEOUT)", row)
    print(f"{row:<20} -> {m.group(1)}")

status=PASS          -> PASS
status=FAIL          -> FAIL
status=FAIL_TIMEOUT  -> FAIL

Two of three correct. The third row says FAIL_TIMEOUT and the parser reports FAIL, because the regex engine tries alternatives strictly left to right and stops at the first one that fits. FAIL fits the start of FAIL_TIMEOUT, so the engine never looks at the third alternative.

The consequence is specific and bad: a timeout is a different failure from a check failure. One means the design is wrong, the other usually means the testbench hung. A triage script that collapses them has lost the distinction it existed to make — and it reported FAIL, which is true enough to never look suspicious.

There are two fixes and they are not the same fix.

Order the alternatives longest first:

import re

for row in ["status=PASS", "status=FAIL", "status=FAIL_TIMEOUT"]:
    m = re.search(r"status=(FAIL_TIMEOUT|FAIL|PASS)", row)
    print(f"{row:<20} -> {m.group(1)}")

status=PASS          -> PASS
status=FAIL          -> FAIL
status=FAIL_TIMEOUT  -> FAIL_TIMEOUT

All three correct. Where one alternative is a prefix of another, the longer one must come first, because the engine's preference is positional rather than greedy across alternatives.

Or require a boundary, using Chapter 19.7's \b:

import re

for row in ["status=PASS", "status=FAIL", "status=FAIL_TIMEOUT"]:
    m = re.search(r"status=(PASS|FAIL|FAIL_TIMEOUT)\b", row)
    print(f"{row:<20} -> {m.group(1)}")

status=PASS          -> PASS
status=FAIL          -> FAIL
status=FAIL_TIMEOUT  -> FAIL_TIMEOUT

The alternatives are in the original, wrong order and the answers are right. This is worth understanding rather than memorising. On the third row the engine tries FAIL, then checks \b and finds _ — a word character on both sides, so there is no boundary, so the whole attempt fails. Rather than giving up, it backtracks and tries the next alternative, and FAIL_TIMEOUT\b succeeds at the end of the string.

Two mechanisms, same result here. The difference shows when they disagree: ordering tells the engine your preference, while \b states a fact about the data — this token ends here. Prefer \b when the fact is true, because it keeps working when someone adds a fourth status in the wrong place.

5. The Optional Part

? after something means zero or one of it — Chapter 19.6 introduced it as a quantifier, and this is the use it exists for:

import re

for field in ["color", "colour"]:
    print(field, "->", bool(re.search(r"^colou?r$", field)))

for field in ["1839274", "-42", "+7"]:
    m = re.search(r"^[-+]?\d+$", field)
    print(field, "->", bool(m))

color -> True
colour -> True
1839274 -> True
-42 -> True
+7 -> True

Applied to a single character, ? tolerates a spelling variant or an optional sign. Applied to a group, it tolerates a whole missing field, which is what the canonical annotation does:

import re

for row in ["slack (VIOLATED)   -0.142", "slack   0.031"]:
    m = re.search(r"slack( \(VIOLATED\))?", row)
    print(f"{row!r:<30} group(0)={m.group(0)!r}")

'slack (VIOLATED)   -0.142'    group(0)='slack (VIOLATED)'
'slack   0.031'                group(0)='slack'

One pattern, both lines. When the (VIOLATED) annotation is present the group consumes it; when it is absent the group matches nothing and the overall pattern still succeeds. Without the ? the second line would not match at all, and a timing script would report that it could not read a report it reads perfectly well.

Note what the ? applies to: the group immediately before it, including the leading space inside the parentheses. Putting the space outside — slack ( \(VIOLATED\))? — would require a space that the second line's format does not guarantee, which is the kind of one-character decision that makes an optional field work on one tool version and fail on the next.

6. A Group That Did Not Participate Is None

An optional group that matched nothing is not an empty string:

import re

for row in ["slack (VIOLATED)   -0.142", "slack   0.031"]:
    m = re.search(r"slack(?P<flag> \(VIOLATED\))?", row)
    flag = m.group("flag")
    print(f"{row!r:<30} flag={flag!r} is_none={flag is None}")

'slack (VIOLATED)   -0.142'    flag=' (VIOLATED)' is_none=False
'slack   0.031'                flag=None is_none=True

None means this group never took part in the match, which is a genuinely different fact from this group matched the empty string. The distinction matters because None does not behave like text:

import re

m = re.search(r"slack(?P<flag> \(VIOLATED\))?", "slack   0.031")

print("flag:", m["flag"])
print("stripped:", m["flag"].strip())

flag: None
Traceback (most recent call last):
  File "none_flag.py", line 6, in <module>
    print("stripped:", m["flag"].strip())
                       ^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'strip'

That is Chapter 19.8 Section 6's AttributeError arriving from a different direction. There the whole match was None; here the match succeeded and only one group is None. Both crash on the same access, and only one of them is fixed by checking if m is None.

Three ways to handle it, in increasing order of how much they tell the reader:

import re

PATTERN = r"slack(?P<flag> \(VIOLATED\))?\s+(?P<value>-?\d+\.\d+)"

for row in ["slack (VIOLATED)   -0.142", "slack   0.031"]:
    m = re.search(PATTERN, row)

    a = m["flag"] or "(none)"
    b = m.groupdict("")["flag"]
    violated = m["flag"] is not None

    print(f"{float(m['value']):>7.3f}  or={a!r:<14} default={b!r:<13} violated={violated}")

 -0.142  or=' (VIOLATED)'  default=' (VIOLATED)' violated=True
  0.031  or='(none)'       default=''            violated=False

m["flag"] or "(none)" substitutes a placeholder, which is fine for display and dishonest for data, because an absent field and a field whose value happens to be falsy become indistinguishable — Chapter 9.5's truthiness warning, in a new costume.

m.groupdict("") sets the default for every non-participating group at once, which is the right tool when you are about to write the whole row to a CSV and empty cells are what you want.

m["flag"] is not None converts presence into a boolean, and it is the one to prefer when the question is was this field there. That is almost always the question an optional group is really asking.

7. One Pattern, Two Tool Versions

Both constructs together, on the problem that motivates them. A simulator printed the instance scope in one release and omitted it in an earlier one, and the same parser has to read both logs:

import re

LINES = [
    "UVM_ERROR drv.sv(88) @ 12500: [WR] bad handshake",
    "UVM_ERROR drv.sv(88) @ 12500: uvm_test_top.env.agt.drv [WR] bad handshake",
    "UVM_FATAL env.sv(12) @ 0: uvm_test_top.env [BLD] no agent",
]

ISSUE = (
    r"UVM_(?P<sev>ERROR|FATAL|WARNING) (?P<file>\w+\.\w+)\((?P<line>\d+)\)"
    r" @ (?P<cycle>\d+): (?P<scope>uvm_\S+ )?\[(?P<id>\w+)\]"
)

for line in LINES:
    m = re.search(ISSUE, line)
    if m is None:
        print("UNPARSED:", line[:40])
        continue
    f = m.groupdict()
    scope = f["scope"].strip() if f["scope"] is not None else "-"
    print(f"{f['sev']:<8} {f['file']:<8} {int(f['line']):>4} {int(f['cycle']):>7}  {f['id']:<4} {scope}")

ERROR    drv.sv     88   12500  WR   -
ERROR    drv.sv     88   12500  WR   uvm_test_top.env.agt.drv
FATAL    env.sv     12       0  BLD  uvm_test_top.env

One pattern, two formats, six named fields, no duplicated parser. Three things in it are worth naming explicitly.

(?P<sev>ERROR|FATAL|WARNING) is a named alternation — the choice and the label share one group, so m["sev"] is whichever severity matched. No alternative here is a prefix of another, so Section 4's ordering rule does not apply, and writing them in severity order is purely for the reader.

(?P<scope>uvm_\S+ )? is the optional field, and its uvm_ prefix is doing real work: without it, \S+ would happily match the [WR] that follows on the first line, and the pattern would then fail to find a [ and report UNPARSED on a line it should read. An optional group should be specific enough that its absence is unambiguous.

The f["scope"] is not None check is Section 6's rule, not defensive clutter. .strip() on a missing scope is an AttributeError on the first line of the log.

8. Predict, Run, Observe

import re

# A
print("A:", re.search(r"UVM_WARN|ERROR", "UVM_ERROR x").span())

# B
print("B:", re.search(r"UVM_(WARN|ERROR)", "UVM_ERROR x").span())

# C
print("C:", re.search(r"(WARN|WARNING)", "WARNING: late").group(1))

# D
m = re.search(r"test=(?P<name>\w+)(?P<rand> rand)?", "test=smoke_rw seed=7")
print("D:", m["name"], repr(m["rand"]))

# E
print("E:", re.search(r"^(PASS|FAIL)$", "FAIL_TIMEOUT"))

The verified output:

A: (4, 9)
B: (0, 9)
C: WARN
D: smoke_rw None
E: None

A — alternation binds loosest, so the pattern is (UVM_WARN)|(ERROR) and the bare ERROR at offset 4 wins. Section 3.

B — with parentheses the choice applies to the suffix only, and the match starts at 0 and covers UVM_ERROR.

C — WARN is a prefix of WARNING and comes first, so it wins. Section 4's defect at its smallest. (WARNING|WARN) or (WARN|WARNING)\b both fix it.

D — the optional group did not participate, so it is None and not ''. One .strip() from a crash.

E — anchored and parenthesised, so neither alternative can match a longer string. FAIL is rejected because $ is not satisfied — which is the behaviour you want from a verdict check.

9. Exercises

Exercise 1 — one pass, three severities. Write a pattern that recognises UVM_ERROR, UVM_FATAL and UVM_WARNING and reports which it found, and run it over a five-line extract that includes one UVM_INFO.

Exercise 2 — alternatives as patterns. Write one pattern that accepts a hex literal, a signed decimal and a plain integer, and test it against four fields including one that is none of them.

Exercise 3 — reproduce the precedence trap. Run r"UVM_ERROR|FATAL" against a UVM_FATAL line, print span(), and say in one sentence why the span does not start at 0.

Exercise 4 — the prose false positive. Find or invent a log line that mentions a severity word in prose, and show the loose pattern matching it and the parenthesised pattern not.

Exercise 5 — the prefix defect. Build a three-alternative status pattern where one alternative is a prefix of another, and record the wrong answer. Then fix it twice — once by ordering and once with \b — and confirm both.

Exercise 6 — which fix. For your Exercise 5 pattern, say which of the two fixes survives someone adding a fourth status at the end of the list, and why.

Exercise 7 — optional character. Write one pattern that accepts WARN, WARNING, warn is not required — just the two uppercase spellings — using ? rather than alternation. Say which of the two reads better.

Exercise 8 — optional group. Write a pattern for a line whose middle field is sometimes absent, and run it against both forms. Print group(0) for each and state which characters the optional group consumed.

Exercise 9 — None, not empty. Take your Exercise 8 pattern and call .strip() on the optional group unguarded. Record the error, then fix it three ways: or, groupdict(""), and an is not None boolean. Say which you would ship and why.

Exercise 10 — two formats, one parser. Write a single named pattern with four fields, one of them optional, that reads two different formats of the same report line. Print an aligned table and a - where the optional field was absent.

10. Summary

• | means one of these, and the group around it both bounds the choice and captures whichever alternative won.
• An alternative is a full sub-pattern, so classes, quantifiers and anchors all work inside one.
• | has the lowest precedence of any regex construct — UVM_ERROR|FATAL is (UVM_ERROR) or (FATAL), not UVM_ plus a choice.
• Parentheses around an alternation are mandatory in practice, and ^a|b$ is two unrelated tests rather than one.
• Alternatives are tried left to right and the first that fits wins, so an alternative that is a prefix of a later one shadows it.
• Two fixes for that: order the longer alternative first, or require \b, which makes the engine backtrack into the longer one.
• \b states a fact about the data; ordering states a preference — prefer \b when the fact holds.
• ? means zero or one of the single character or the group immediately before it.
• An optional group tolerates a field a tool prints only sometimes, which is what lets one parser read two tool versions.
• A group that did not participate is None, not "" — and None.strip() is an AttributeError even though the match succeeded.
• Handle it with is not None when the question is presence, groupdict("") when writing a table, or only for display.
• An optional group should be specific, or its absence becomes ambiguous and the pattern fails on a line it should read.
• Every ? weakens the pattern as a check. Add one for a field that is genuinely absent, never to stop a pattern failing.

The next chapter steps back from pattern syntax to the five functions that run a pattern, including the one that changes what it returns the moment your pattern contains a group.