Building a Real Pattern Step by Step

Part II · Module 19 · Chapter 19.12 · Building a Real Pattern Step by Step

1. The Report, and the Question

Here is an extract from a static timing report. It is the shape every STA tool prints, with the fields the pattern will have to survive:

REPORT = """Startpoint: u_core/u_alu/acc_reg[7] (rising edge-triggered flip-flop)
Endpoint: u_core/u_fifo/wptr_reg[3] (rising edge-triggered flip-flop)
Info: computing slack for 1842 endpoints
  data required time          2.431
  data arrival time          2.573
  slack (VIOLATED)          -0.142
Startpoint: u_core/u_ctl/state_reg[1] (rising edge-triggered flip-flop)
  slack (MET)                0.031
  note: slack margin policy is 0.000 for this corner
  slack (VIOLATED)          -1
"""

print(len(REPORT.splitlines()), "lines")
print(REPORT.count("slack"), "mentions of the word slack")

10 lines
5 mentions of the word slack

Ten lines, and the word slack appears five times. Three of those are slack results. The other two are an informational count and a note about policy, and telling them apart is the entire job.

The question the script has to answer is the one a timing-closure flow asks every night: what is the worst slack in this report, and did anything violate? That needs a number, not a yes-or-no — so by Chapter 19.8's distinction this is an extraction, and by Chapter 19.11's it wants finditer or a per-line search.

2. Stage 1 — "slack" in line

Start with no regex at all. Chapter 19.1 made the case that a substring test is the right first move, and it is still right here:

REPORT = """Startpoint: u_core/u_alu/acc_reg[7] (rising edge-triggered flip-flop)
Endpoint: u_core/u_fifo/wptr_reg[3] (rising edge-triggered flip-flop)
Info: computing slack for 1842 endpoints
  data required time          2.431
  data arrival time          2.573
  slack (VIOLATED)          -0.142
Startpoint: u_core/u_ctl/state_reg[1] (rising edge-triggered flip-flop)
  slack (MET)                0.031
  note: slack margin policy is 0.000 for this corner
  slack (VIOLATED)          -1
"""

hits = []
for line in REPORT.splitlines():
    if "slack" in line:
        hits.append(line)

print(len(hits), "lines matched")
for line in hits:
    print("   ", repr(line))

5 lines matched
    'Info: computing slack for 1842 endpoints'
    '  slack (VIOLATED)          -0.142'
    '  slack (MET)                0.031'
    '  note: slack margin policy is 0.000 for this corner'
    '  slack (VIOLATED)          -1'

Five lines, three of them wanted. The two false positives are a progress message and a policy note, and both would end up in a report as slack results if nothing else changed.

What this stage is worth: it took ten seconds to write and it told us something real — which lines contain the word, and therefore what the false positives actually look like. You cannot design a pattern against false positives you have not seen. Starting here is not naivety; it is reconnaissance.

What it cannot do: it cannot extract. There is no number in True.

3. Stage 2 — r"slack"

Translate the test into a pattern that does exactly the same thing:

import re

REPORT = """Info: computing slack for 1842 endpoints
  data required time          2.431
  slack (VIOLATED)          -0.142
  slack (MET)                0.031
  note: slack margin policy is 0.000 for this corner
  slack (VIOLATED)          -1
"""

substring = []
pattern = []
for line in REPORT.splitlines():
    if "slack" in line:
        substring.append(line)
    if re.search(r"slack", line) is not None:
        pattern.append(line)

print("substring:", len(substring))
print("pattern  :", len(pattern))
print("identical:", substring == pattern)

substring: 5
pattern  : 5
identical: True

This stage buys nothing, and it is worth doing anyway. r"slack" contains no metacharacters, so it is a literal search and gives byte-for-byte the same answer as in — Chapter 19.2 made that point and here it is as a provable equality rather than a claim.

What the stage buys is a place to grow. The next three stages are edits to this string, and each one is a few characters. If you skip this translation and try to write stage 5 directly, you are debugging five constructs at once with no working version to fall back to.

4. Stage 3 — r"slack\s+\(VIOLATED\)"

The false positives share a property: neither is followed by a parenthesised tag. So require one:

import re

REPORT = """Info: computing slack for 1842 endpoints
  data required time          2.431
  slack (VIOLATED)          -0.142
  slack (MET)                0.031
  note: slack margin policy is 0.000 for this corner
  slack (VIOLATED)          -1
"""

for line in REPORT.splitlines():
    m = re.search(r"slack\s+\(VIOLATED\)", line)
    print(f"{'HIT ' if m else '    '} {line!r}")

     'Info: computing slack for 1842 endpoints'
     '  data required time          2.431'
HIT  '  slack (VIOLATED)          -0.142'
     '  slack (MET)                0.031'
     '  note: slack margin policy is 0.000 for this corner'
HIT  '  slack (VIOLATED)          -1'

Both false positives are gone. Three symbols did it, and each has a stated reason:

\s+ — one or more whitespace characters, from Chapter 19.4. Not a literal space, because the tool aligns its columns and the gap between slack and (VIOLATED) is whatever the alignment produced. Writing " " here would work on this report and fail on the first one that used two spaces or a tab, and it would fail by matching nothing — a silent zero.

\( and \) — escaped parentheses, from Chapter 19.3. Unescaped, (VIOLATED) would be a capturing group matching the bare word with no brackets, and would then happily match the note: line's prose if it ever said VIOLATED. The backslashes are what make these literal characters in the data rather than regex syntax.

What this stage cannot do: the slack (MET) line is gone too, and that is a real loss. A timing script that only sees violations cannot report the worst slack in a clean run, cannot show margin, and cannot tell "no violations" apart from "could not read the report".

5. Stage 4 — r"slack\s+\((?:VIOLATED|MET)\)"

Accept either tag. Chapter 19.10 gave the construct, with one new piece of syntax:

import re

REPORT = """Info: computing slack for 1842 endpoints
  slack (VIOLATED)          -0.142
  slack (MET)                0.031
  note: slack margin policy is 0.000 for this corner
  slack (VIOLATED)          -1
  slack (UNCONSTRAINED)
"""

for line in REPORT.splitlines():
    m = re.search(r"slack\s+\((?:VIOLATED|MET)\)", line)
    print(f"{'HIT ' if m else '    '} {line.strip()!r}")

     'Info: computing slack for 1842 endpoints'
HIT  'slack (VIOLATED)          -0.142'
HIT  'slack (MET)                0.031'
     'note: slack margin policy is 0.000 for this corner'
HIT  'slack (VIOLATED)          -1'
     'slack (UNCONSTRAINED)'

Three slack results, both false positives still excluded, and an UNCONSTRAINED line — which real tools do print — correctly left out, because the alternation lists the two tags that carry a number and no others.

(?: opens a non-capturing group. It does everything an ordinary group does for structure — it bounds the alternation so the choice applies to VIOLATED or MET and not to the whole pattern, which is Chapter 19.10 Section 3's precedence trap — and it does not create a group you can read back:

import re

LINE = "  slack (VIOLATED)          -0.142"

capturing = re.search(r"slack\s+\((VIOLATED|MET)\)", LINE)
non_capturing = re.search(r"slack\s+\((?:VIOLATED|MET)\)", LINE)

print("capturing    :", capturing.groups())
print("non-capturing:", non_capturing.groups())
print("same match   :", capturing.group(0) == non_capturing.group(0))

capturing    : ('VIOLATED',)
non-capturing: ()
same match   : True

Identical match, different bookkeeping. (?:...) says group these characters, do not number them, and the reason to use it is Chapter 19.8 Section 5: every capturing group you do not need is a number that shifts when someone edits the pattern. A pattern whose only group is the one you actually read is a pattern where group(1) cannot drift.

Here it is a judgment call rather than a rule. The tag is interesting — knowing a path was VIOLATED rather than MET saves a comparison — so Section 8 captures it deliberately, with a name. The canonical pattern uses (?:...) because at this stage the tag is only being used as a filter, and that is exactly when a group should not be numbered.

6. Stage 5 — r"slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+(?:\.\d+)?)"

Finally, the number:

import re

REPORT = """Info: computing slack for 1842 endpoints
  slack (VIOLATED)          -0.142
  slack (MET)                0.031
  note: slack margin policy is 0.000 for this corner
  slack (VIOLATED)          -1
  slack (UNCONSTRAINED)
"""

SLACK = r"slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+(?:\.\d+)?)"

for line in REPORT.splitlines():
    m = re.search(SLACK, line)
    if m is None:
        continue
    print(f"{m.group(0)!r:<40} -> {float(m.group(1)):>8.3f}")

'slack (VIOLATED)          -0.142'       ->   -0.142
'slack (MET)                0.031'       ->    0.031
'slack (VIOLATED)          -1'           ->   -1.000

Three numbers out of ten lines. Four symbols did that, and the last one is the interesting one.

\s+ again — the alignment gap between the tag and the number, same reason as stage 3.

( and ) unescaped — a capturing group this time, because this is the one thing the script needs to read. Exactly one numbered group in the whole pattern.

-? — an optional minus sign, from Chapter 19.10 Section 5. A violated slack is negative and a met slack is positive, so a pattern that cannot accept a sign reads 0.142 out of -0.142 and reports a 142-picosecond margin on a failing path. This is the single worst defect this pattern could have, and it is one character away.

\d+ — one or more digits, the integer part. + and not *, by Chapter 19.6's rule: zero digits is not a number.

(?:\.\d+)? — the fractional part, optional, and grouped so that the ? applies to all three symbols at once. This is the second use of (?:...) and the one that is not about capturing at all: without the group, \.\d+? would make only the last digit optional, which is not a thing anyone wants.

That last construct earns its place on one line of the report, and it is worth seeing what happens without it:

import re

ROWS = [
    "  slack (VIOLATED)          -0.142",
    "  slack (MET)                0.031",
    "  slack (VIOLATED)          -1",
]

NAIVE = r"slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+\.\d+)"
FINAL = r"slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+(?:\.\d+)?)"

for row in ROWS:
    a = re.search(NAIVE, row)
    b = re.search(FINAL, row)
    print(f"{row.strip():<34} naive={a.group(1) if a else 'NO MATCH':<9} final={b.group(1)}")

slack (VIOLATED)          -0.142   naive=-0.142    final=-0.142
slack (MET)                0.031   naive=0.031     final=0.031
slack (VIOLATED)          -1       naive=NO MATCH  final=-1

Read the third row and then read it again. The naive pattern does not match the worst path in the report. It does not crash, it does not warn, it returns None — and a loop with if m is None: continue in it, which is the correct guard from Chapter 19.8, skips that line and reports the worst slack as -0.142.

A tool that prints -1 instead of -1.000 for a whole-picosecond value is not unusual, and the failure it causes here is the worst class there is: the script reports a better number than the truth, and reports it confidently. That is why this chapter adds constructs against real output rather than against an idea of what the output looks like.

7. The Symbol Audit

The canonical standard for this chapter is that no symbol in the pattern is unexplained. Here is the pattern with one row per symbol:

slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+(?:\.\d+)?)

Ten rows, nine distinct constructs, and not one of them is there for completeness. If you cannot write this table for a pattern in your own script, you cannot safely change that pattern when the tool's next release moves a column — and you will not be able to tell whether a symbol you deleted was load-bearing.

8. The Finished Parser

The pattern as a flow script would actually ship it: both fields named, the tag captured because the report wants it, and the verdict separated from the parsing in the way Chapter 17.10 argued for.

import re

SLACK = r"slack\s+\((?P<state>VIOLATED|MET)\)\s+(?P<value>-?\d+(?:\.\d+)?)"

REPORT = """Startpoint: u_core/u_alu/acc_reg[7] (rising edge-triggered flip-flop)
Info: computing slack for 1842 endpoints
  data arrival time          2.573
  slack (VIOLATED)          -0.142
  slack (MET)                0.031
  note: slack margin policy is 0.000 for this corner
  slack (VIOLATED)          -1
  slack (UNCONSTRAINED)
"""


def slack_values(text):
    """Every slack result in a timing report, as (state, value) pairs."""
    results = []
    for m in re.finditer(SLACK, text):
        results.append((m["state"], float(m["value"])))
    return results


def verdict(results):
    """FAIL if anything violated, NO DATA if the report had no slack results."""
    if not results:
        return "NO DATA"
    worst = results[0][1]
    for _, value in results:
        if value < worst:
            worst = value
    return "FAIL" if worst < 0 else "PASS"


found = slack_values(REPORT)
for state, value in found:
    print(f"{state:<9} {value:>8.3f}")
print("paths   :", len(found))
print("verdict :", verdict(found))
print("empty   :", verdict([]))

VIOLATED    -0.142
MET          0.031
VIOLATED    -1.000
paths   : 3
verdict : FAIL
empty   : NO DATA

slack_values extracts and converts; verdict decides. The split matters because the two change for different reasons — the pattern changes when the tool changes, and the rule changes when the project's policy does.

And verdict([]) returning NO DATA rather than PASS is Chapter 17.7's definition of failure applied to a parser: an unreadable report is not a clean report. A verdict that returned PASS for an empty list would turn every one of this chapter's parsing failures into a green nightly.

Finally, the check that makes the pattern maintainable — a list of what it must match and what it must not:

import re

SLACK = r"slack\s+\((?P<state>VIOLATED|MET)\)\s+(?P<value>-?\d+(?:\.\d+)?)"

MUST_MATCH = [
    ("  slack (VIOLATED)          -0.142", -0.142),
    ("  slack (MET)   0.031", 0.031),
    ("slack (VIOLATED) -1", -1.0),
    ("\tslack\t(MET)\t12.5", 12.5),
]

MUST_NOT_MATCH = [
    "Info: computing slack for 1842 endpoints",
    "  note: slack margin policy is 0.000 for this corner",
    "  slack (UNCONSTRAINED)",
    "  data arrival time          2.573",
]

failures = 0

for text, want in MUST_MATCH:
    m = re.search(SLACK, text)
    got = float(m["value"]) if m is not None else None
    ok = got == want
    failures += 0 if ok else 1
    print(f"{'ok  ' if ok else 'FAIL'} must match     {text!r:<40} -> {got}")

for text in MUST_NOT_MATCH:
    m = re.search(SLACK, text)
    ok = m is None
    failures += 0 if ok else 1
    print(f"{'ok  ' if ok else 'FAIL'} must not match {text!r:<40} -> {m.group(0) if m else None}")

print("failures:", failures)

ok   must match     '  slack (VIOLATED)          -0.142'     -> -0.142
ok   must match     '  slack (MET)   0.031'                  -> 0.031
ok   must match     'slack (VIOLATED) -1'                    -> -1.0
ok   must match     '\tslack\t(MET)\t12.5'                   -> 12.5
ok   must not match 'Info: computing slack for 1842 endpoints' -> None
ok   must not match '  note: slack margin policy is 0.000 for this corner' -> None
ok   must not match '  slack (UNCONSTRAINED)'                -> None
ok   must not match '  data arrival time          2.573'     -> None
failures: 0

Eight cases, four of each kind. The tab-separated row passes because \s+ was chosen over a literal space in stage 3, which is the first time that decision has been tested rather than merely argued. Chapter 19.18 turns this list into something a regression can run.

9. Predict, Run, Observe

import re

LINE = "  slack (VIOLATED)          -1"

# A
print("A:", re.search(r"slack \(VIOLATED\)", LINE))

# B
print("B:", re.search(r"slack\s+\(VIOLATED\)", LINE).group(0))

# C
print("C:", re.search(r"slack\s+\((?:VIOLATED|MET)\)", LINE).groups())

# D
print("D:", re.search(r"slack\s+\((VIOLATED|MET)\)", LINE).groups())

# E
print("E:", re.search(r"(-?\d+\.\d+)", LINE))

# F
print("F:", re.search(r"(-?\d+(?:\.\d+)?)", LINE).group(1))

# G
print("G:", re.search(r"(-?\d+(\.\d+)?)", LINE).groups())

The verified output:

A: <re.Match object; span=(2, 18), match='slack (VIOLATED)'>
B: slack (VIOLATED)
C: ()
D: ('VIOLATED',)
E: None
F: -1
G: ('-1', None)

A — a single literal space happens to be right on this line, which is exactly what makes it a bad idea. One tab in the next release and this returns None.

B — \s+ matches the same single space here, and would also match a tab or six spaces.

C — the non-capturing group contributes nothing to groups(). Stage 4.

D — remove the ?: and the tag is group 1.

E — the naive decimal pattern finds nothing on an integer slack. Section 6's defect, isolated.

F — with the fraction optional, -1 is captured. Note it is the string '-1'; float() is still your job.

G — a capturing fractional group would work for the match but adds a second group that is None here. (?:...) is what keeps the pattern to exactly one readable group.

10. Exercises

Exercise 1 — reconnaissance. Take a report extract of your own and run the substring test for your keyword. List every false positive and describe what each one has in common with the others.

Exercise 2 — the free translation. Convert your substring test to re.search with a literal pattern and prove the two give identical results on the same subject.

Exercise 3 — one construct, one reason. Add exactly one construct to exclude your first false positive. State in one sentence which line it was for.

Exercise 4 — the literal space defect. Take a working pattern that uses a literal space and run it against the same line with the space replaced by a tab. Record the result and fix it with \s+.

Exercise 5 — escape or group. Write (VIOLATED) unescaped in a pattern and print groups(). Explain the result and then escape it.

Exercise 6 — capturing or not. Write the same alternation twice, once as (A|B) and once as (?:A|B). Show that the match is identical and groups() is not, and say which you would ship for a filter.

Exercise 7 — the sign. Remove the -? from the finished pattern and run it on the violated line. Record what number the script now reports and what a reviewer reading a nightly summary would conclude.

Exercise 8 — the optional fraction. Reproduce Section 6's three-row comparison with your own data, including one integer value. Say why the naive version's failure is worse than a crash would be.

Exercise 9 — the audit. Write the symbol table for a pattern you did not write — one from this module's earlier chapters is fine. Flag any symbol you cannot justify and say what you would test to find out whether it is load-bearing.

Exercise 10 — must and must not. Write eight cases for your own pattern, four of each kind, including one with tabs and one near-miss that must be rejected. Run them and report the failure count.

Exercise 11 — empty is not clean. Write a verdict function for your parser that distinguishes "nothing violated" from "nothing parsed". Say what the nightly would have reported if you had collapsed the two.

11. Summary

• Start with the dumbest test that could work. A substring test is reconnaissance: it shows you the false positives you have to design against.
• Translate it to a literal pattern before adding anything. The stage buys no behaviour and provides the working baseline every later stage is checked against.
• Add one construct per stage, for a stated reason — a specific line of real tool output that the previous stage got wrong.
• \s+, never a literal space, wherever the tool aligns columns. A literal space fails by matching nothing.
• Escape literal brackets, or (VIOLATED) becomes a capturing group matching a bare word.
• (?:...) groups without numbering — use it when the parentheses are for structure only, and keep your numbered groups down to the ones you read.
• (?:...) is also how a quantifier applies to several symbols at once, which is what makes (?:\.\d+)? an optional fraction rather than an optional last digit.
• -? is one character and the whole sign. Without it, a violated path reports a positive margin.
• A tool that prints -1 instead of -1.000 makes a naive decimal pattern drop the worst path in the report — silently, with the correct None guard in place.
• The failure to fear is not the crash. It is the pattern that reports a better number than the truth.
• No symbol unexplained — if you cannot write the symbol table, you cannot safely edit the pattern.
• Separate extraction from the verdict, because the pattern changes when the tool changes and the rule changes when the policy does.
• An empty result is NO DATA, not PASS. An unreadable report is not a clean report.
• Keep a must-match and must-not-match list. It is what makes a choice like \s+ over " " a tested decision rather than an argument.

The next chapter is about the one construct this pattern deliberately avoided, and what happens to a field when .* is allowed to take as much as it likes.