Lookahead and Lookbehind

Part II · Module 19 · Chapter 19.15 · Lookahead and Lookbehind

1. Four Constructs, One Idea

All four are zero width: they test the text at the current position and consume none of it, exactly as ^, $ and \b do in Chapter 19.7. The difference is that those three test a fixed property of the position, and these four test a pattern of your choosing.

The zero-width claim is checkable:

import re

m = re.search(r"\d+(?= ns)", "delay 420 ns")

print("group:", m.group(0))
print("span :", m.span())
print("width:", len(m.group(0)))

group: 420
span : (6, 9)
width: 3

The pattern required ns to follow and the match is three characters long. The unit was checked and left in the subject — which is the whole proposition, and in the one case where it is clearly right:

import re

for field in ["delay 420 ns", "delay 420 ps", "count 420 errors"]:
    m = re.search(r"\d+(?= ns)", field)
    print(f"{field:<20} {m.group(0) if m else None}")

delay 420 ns         420
delay 420 ps         None
count 420 errors     None

A number only where it is a nanosecond value. The picosecond row and the prose row are both rejected, and the result is the number alone with no unit to strip.

2. Lookbehind, and the Canonical Annotation

The canonical annotation for this chapter puts the context on the left:

re.search(r"(?<=slack )-?\d+\.\d+", line)

On a simple line it does exactly what it says:

import re

LINE = "slack -0.142"

m = re.search(r"(?<=slack )-?\d+\.\d+", LINE)
print("group:", m.group(0))
print("span :", m.span())

group: -0.142
span : (6, 12)

Six characters consumed, starting at offset 6 — the slack before it was asserted, not captured. No group, no group(1), no numbering to maintain. For a one-field extraction that is genuinely tidy.

Now run it on the line a timing tool actually prints:

import re

SIMPLE = "slack -0.142"
REAL = "  slack (VIOLATED)   -0.142"

print("simple:", re.search(r"(?<=slack )-?\d+\.\d+", SIMPLE))
print("real  :", re.search(r"(?<=slack )-?\d+\.\d+", REAL))

simple: <re.Match object; span=(6, 12), match='-0.142'>
real  : None

None on the real line, for the reason Chapter 19.12 Stage 3 predicted: the tool aligns its columns, so what precedes the number is not slack but (VIOLATED) and a run of spaces. The lookbehind describes a context that does not exist.

The obvious fix is the one Chapter 19.12 used — \s+ instead of a literal space. It does not work:

import re

try:
    re.compile(r"(?<=slack\s+)-?\d+\.\d+")
    print("compiled")
except re.PatternError as exc:
    print("re.PatternError:", exc)

re.PatternError: look-behind requires fixed-width pattern

A lookbehind must be fixed width. The engine implements it by stepping back a known number of characters and matching forward from there, so it has to know how far back to step — and \s+ could be one character or forty. (?<=slack ) is seven characters and legal; (?<=slack\s+) is not, and neither is (?<=slack\s{1,8}), (?<=(slack|setup) ), or anything else of variable length. Alternatives of equal length are accepted, which is a narrow escape hatch and not a general one.

This is the single most important fact in the chapter, and it has a direct consequence: a lookbehind cannot skip a tool's column alignment, which is present in essentially every report this module parses. That is why Chapter 19.12's shipped pattern uses \s+ and a capturing group, and why the canonical annotation above is best read as the idea, with the aligned-column reality handled the way that chapter handled it.

Lookahead has no such restriction — it matches forward from the current position, so (?=\s+-?\d) is perfectly legal.

3. Negative Lookahead and the Second Annotation

The chapter's other canonical annotation asks for a test name that is not a randomised variant:

re.search(r"\w+(?!_rand)\b", name)

Run it on five names:

import re

NAMES = ["smoke_rw", "smoke_rand", "burst_rand_wr", "rand_smoke", "smoke_randx"]

for name in NAMES:
    m = re.search(r"\w+(?!_rand)\b", name)
    print(f"{name:<14} -> {m.group(0) if m else None}")

smoke_rw       -> smoke_rw
smoke_rand     -> smoke_rand
burst_rand_wr  -> burst_rand_wr
rand_smoke     -> rand_smoke
smoke_randx    -> smoke_randx

Every name matched, including smoke_rand. The pattern reads like a word not followed by _rand and behaves like a word, and the reason is Chapter 19.13's greedy rule:

\w+ takes the entire name, because _ is a word character and nothing stops it. The engine is now at the end of the string. It evaluates (?!_rand) there — is the text after this position not _rand? There is no text after this position, so yes, trivially. Then \b succeeds at the end of a word. The assertion was satisfied by being asked in a place where it could not fail.

This is the characteristic lookaround bug: an assertion placed where it cannot be false. It does not raise, it does not warn, and the pattern looks like it is doing a check.

A working version asks the question about the whole name, before consuming anything:

import re

NAMES = ["smoke_rw", "smoke_rand", "burst_rand_wr", "rand_smoke", "smoke_randx"]

for name in NAMES:
    m = re.fullmatch(r"(?!.*_rand\b)\w+", name)
    print(f"{name:<14} -> {m.group(0) if m else None}")

smoke_rw       -> smoke_rw
smoke_rand     -> None
burst_rand_wr  -> burst_rand_wr
rand_smoke     -> rand_smoke
smoke_randx    -> smoke_randx

Three changes, each load-bear. fullmatch instead of search, so the pattern has to account for the whole name — Chapter 19.11's validator. The lookahead moved to the front, where it is evaluated at position 0 and can actually fail. And .*_rand\b inside it, so the assertion is this name does not contain _rand as a complete trailing token anywhere — which correctly keeps burst_rand_wr (where _rand is followed by more name), keeps rand_smoke (no underscore before rand) and keeps smoke_randx (randx, not rand).

4. Two Lookarounds That Are Redundant

Recognising a lookaround you do not need is worth as much as writing one you do, and Chapter 19.7 already solved the problem people reach for (?<!...) to solve:

import re

LINES = ["UVM_ERROR drv.sv(88)", "checker reports NO_ERROR", "status: NO-ERROR", "an ERROR was found"]

for line in LINES:
    plain = re.search(r"\bERROR\b", line)
    guarded = re.search(r"(?<!NO_)\bERROR\b", line)
    print(f"{line:<28} plain={bool(plain)}  lookbehind={bool(guarded)}")

UVM_ERROR drv.sv(88)         plain=False  lookbehind=False
checker reports NO_ERROR     plain=False  lookbehind=False
status: NO-ERROR             plain=True  lookbehind=True
an ERROR was found           plain=True  lookbehind=True

The two columns are identical. (?<!NO_) contributed nothing, because \b had already rejected NO_ERROR — an underscore is a word character, so there is no boundary there at all, which is exactly what Chapter 19.7 Section 4 demonstrated.

The same shape with a lookahead:

import re

BUS = "wr_en rd_en wr_en_d1 rd_data"

print("plain    :", re.findall(r"\b\w+_en\b", BUS))
print("lookahead:", re.findall(r"\b\w+_en\b(?!_)", BUS))

plain    : ['wr_en', 'rd_en']
lookahead: ['wr_en', 'rd_en']

Identical again, and for the same reason: \b after _en already means the token ends here, so (?!_) is asserting something \b has made impossible to violate.

Where a lookbehind does earn its place is the gap Chapter 19.7 left open — NO-ERROR, with a hyphen, which \b accepts because a hyphen is not a word character:

import re

LINES = ["UVM_ERROR drv.sv(88)", "checker reports NO_ERROR", "status: NO-ERROR", "an ERROR was found"]

for line in LINES:
    plain = re.search(r"\bERROR\b", line)
    guarded = re.search(r"(?<![-_])\bERROR\b", line)
    print(f"{line:<28} plain={bool(plain)}  guarded={bool(guarded)}")

UVM_ERROR drv.sv(88)         plain=False  guarded=False
checker reports NO_ERROR     plain=False  guarded=False
status: NO-ERROR             plain=True  guarded=False
an ERROR was found           plain=True  guarded=True

One column differs on one row, and it is the row Chapter 19.7 flagged as a false positive. (?<![-_]) is one character wide, so the fixed-width rule is satisfied, and it rejects a severity token glued to a prefix by either separator while keeping the standalone word.

That is what a well-placed lookaround looks like: it changes exactly one answer, and you can point at which one.

5. Where Lookaround Is the Only Tool

Two cases, and the distinction between them and the rest is worth making precisely.

A negative condition has no capturing equivalent. You cannot write "a signal name ending in _en that does not start with rd_" by capturing something, because there is nothing to capture — the requirement is about absence:

import re

BUS = "wr_en rd_en wr_data_en rd_valid_en"

print("all     :", re.findall(r"\b\w+_en\b", BUS))
print("not rd_ :", re.findall(r"\b(?!rd_)\w+_en\b", BUS))

all     : ['wr_en', 'rd_en', 'wr_data_en', 'rd_valid_en']
not rd_ : ['wr_en', 'wr_data_en']

The lookahead sits right after \b, where the next characters are the start of a token — a position at which (?!rd_) genuinely can be false, which is Section 3's rule applied correctly. Two of four names are rejected, and no arrangement of groups would have done it.

Overlapping matches need a zero-width pattern. Matches from findall and finditer never overlap, because each one consumes its characters and the next search starts after them. A lookahead consumes nothing, so a pattern made of one finds every position:

import re

PATH = "a.b.c.d"

print("consuming:", re.findall(r"\w+\.\w+", PATH))
print("overlapped:", re.findall(r"(?=(\w+\.\w+))", PATH))

consuming: ['a.b', 'c.d']
overlapped: ['a.b', 'b.c', 'c.d']

The consuming pattern found a.b, then resumed after the b and found c.d — the pair b.c was unreachable because b had already been eaten. The lookahead version matches at every position where the pattern could start, captures inside the assertion, and moves on by one character, so all three adjacent pairs appear.

That idiom — (?=(...)) with the real pattern inside a group inside the lookahead — is the standard way to get overlapping matches in Python, and it is the one thing in this chapter with no alternative at all.

6. The Honest Ranking

Positive lookaround is usually replaceable, and often should be replaced:

import re

FIELD = "delay 420 ns"

look = re.search(r"\d+(?= ns)", FIELD)
group = re.search(r"(\d+) ns", FIELD)

print("lookahead:", look.group(0), look.span())
print("group    :", group.group(1), group.span(1))
print("same text:", look.group(0) == group.group(1))

lookahead: 420 (6, 9)
group    : 420 (6, 9)
same text: True

Same number, same span, two constructs. The capturing version is understood by more readers and has no fixed-width restriction; the lookaround version keeps group(0) equal to just the number, which occasionally matters when you are about to use span() to edit the line.

So the ranking that holds up in a flow script:

• Negative lookahead and lookbehind are a capability. "Not preceded by", "not followed by" and "does not contain" have no capturing form. Use them, place them where they can fail, and test a case that must be rejected.
• (?=(...)) for overlapping matches is a capability, and the only way to get them.
• Positive lookaround is a convenience. Prefer a capturing group unless you specifically want the context excluded from group(0).
• A lookbehind cannot be variable width, so it cannot cross a tool's column alignment — which rules it out of most report parsing.
• A lookaround that duplicates \b is decoration. Check whether Chapter 19.7's boundary already does the job before adding one.

7. Predict, Run, Observe

import re

# A
print("A:", re.search(r"\d+(?= ns)", "420 ns").span())

# B
print("B:", re.search(r"\d+(?= ns)", "420 ps"))

# C
print("C:", re.search(r"(?<=0x)[0-9a-f]+", "addr 0xdeadbeef").group(0))

# D
print("D:", re.search(r"(?<=0x)[0-9a-f]+", "addr deadbeef"))

# E
try:
    re.compile(r"(?<=addr\s+)0x[0-9a-f]+")
    print("E: compiled")
except re.PatternError as exc:
    print("E:", exc)

# F
print("F:", re.search(r"\w+(?!_d1)\b", "wr_en_d1").group(0))

# G
print("G:", re.fullmatch(r"(?!.*_d1\b)\w+", "wr_en_d1"))

# H
print("H:", re.findall(r"(?=(\d\d))", "1234"))

The verified output:

A: (0, 3)
B: None
C: deadbeef
D: None
E: look-behind requires fixed-width pattern
F: wr_en_d1
G: None
H: ['12', '23', '34']

A — three characters consumed; the unit was asserted, not taken.

B — the assertion failed, so the whole match failed, even though digits are present.

C — a positive lookbehind of fixed width, excluding the 0x from the group. Tidy and legal.

D — no 0x, so no match. The lookbehind is a real requirement, not a hint.

E — \s+ inside a lookbehind is a compile error, every time. Section 2.

F — the assertion is evaluated at the end of the string where nothing follows, so it cannot fail. Section 3's defect exactly.

G — the assertion moved to the front and the name is correctly rejected.

H — overlapping pairs, which only a zero-width pattern can produce. Section 5.

8. Exercises

Exercise 1 — zero width. Extract a number that must be followed by a unit, and print span() and len(group(0)) to show the unit was not consumed.

Exercise 2 — the unit that is not there. Run your Exercise 1 pattern against the same number with a different unit. Say why the match failed rather than returning the digits.

Exercise 3 — fixed-width lookbehind. Extract a hex value that must follow 0x, and confirm the 0x is absent from group(0).

Exercise 4 — the fixed-width rule. Try to write a lookbehind containing \s+. Record the exact error text and say at what moment it was raised.

Exercise 5 — the aligned column. Run (?<=slack ) against a real aligned timing line and against a simplified one. Say what Chapter 19.12 did instead, and why.

Exercise 6 — the assertion that cannot fail. Reproduce Section 3's annotation on a name that should be rejected. Explain in two sentences where the engine was standing when it evaluated the assertion.

Exercise 7 — move it to the front. Fix Exercise 6 with a front-anchored lookahead and fullmatch. Test five names, at least two of which must be rejected for different reasons.

Exercise 8 — find a redundant lookaround. Write a lookaround that duplicates \b and show the two result columns are identical. Then delete it.

Exercise 9 — one answer changed. Write a lookbehind that changes exactly one row of a four-row table, and name the row.

Exercise 10 — negative only. Extract every signal name ending in _valid that does not begin with a given prefix. Say why no arrangement of capturing groups achieves this.

Exercise 11 — overlapping. Use (?=(...)) to find every overlapping pair of dot-separated components in a four-level instance path. Report both counts and say which component the consuming version skipped.

Exercise 12 — which would you ship. Take one positive lookaround from this chapter, rewrite it with a capturing group, and argue for one of the two in three sentences.

9. Summary

• Four constructs: (?=...), (?!...), (?<=...), (?<!...) — follows, does not follow, precedes, does not precede.
• All four are zero width. They test the text at a position and consume none of it, so the context never appears in group(0).
• A lookbehind must be fixed width. (?<=slack\s+) raises re.PatternError: look-behind requires fixed-width pattern at compile time.
• That rules a lookbehind out of most report parsing, because tools align their columns and a lookbehind cannot cross a variable gap. Lookahead has no such limit.
• An assertion after a greedy quantifier is usually unfalsifiable. \w+(?!_rand) matches smoke_rand, because \w+ eats the name and the assertion is then asked at the end of the string.
• Put the assertion where it can fail — at the front with fullmatch, or between two things the subject actually has — and always test an input that must be rejected.
• A lookaround that duplicates \b changes nothing. (?<!NO_)\bERROR\b and \bERROR\b give identical answers on every line.
• (?<![-_])\bERROR\b is the useful version — it closes the NO-ERROR gap that Chapter 19.7 identified, and changes exactly one row.
• Negative lookaround is a capability: "not preceded by", "not followed by" and "does not contain" have no capturing equivalent.
• (?=(...)) is the only way to get overlapping matches, because a zero-width match advances by one character instead of consuming.
• Positive lookaround is a convenience — \d+(?= ns) and (\d+) ns reach the same number with the same span.
• The readability cost is real. If you cannot say why the assertion sits where it does, a capturing group and an if is better engineering.

The next chapter is about compiling a pattern once instead of on every line — and about which of the two benefits is the one that actually matters.