Testing Your Patterns Against Real Output

Part II · Module 19 · Chapter 19.18 · Testing Your Patterns Against Real Output

1. Two Lists

A pattern's test is two lists and a function. The function is the parser from Chapter 19.12:

import re

SLACK = r"slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+(?:\.\d+)?)"


def parse_slack(text, pattern=SLACK):
    """The slack value on one report line, or None if the line has none."""
    m = re.search(pattern, text)
    return float(m.group(1)) if m is not None else None


MUST_MATCH = [
    ("  slack (VIOLATED)   -0.043", -0.043),
    ("  slack (MET)          0.031", 0.031),
    ("slack (VIOLATED) -1", -1.0),
    ("\tslack\t(MET)\t12.5", 12.5),
    ("  slack (VIOLATED)   -0.142  ", -0.142),
]

MUST_NOT_MATCH = [
    "Info: computing slack for 1842 endpoints",
    "  note: slack margin policy is 0.000 for this corner",
    "  slack (UNCONSTRAINED)",
    "  data arrival time          2.573",
]

print("must match    :", len(MUST_MATCH))
print("must not match:", len(MUST_NOT_MATCH))
print("first case    :", parse_slack(MUST_MATCH[0][0]), MUST_MATCH[0][1])

must match    : 5
must not match: 4
first case    : -0.043 -0.043

The shape of those two lists is the whole design.

MUST_MATCH holds pairs, not just lines — the text and the value it has to produce. A list of lines that merely match is a much weaker test, because Chapter 19.17 Section 5 showed a pattern can match and return the wrong number. The expected value is what makes a case a test.

MUST_NOT_MATCH holds lines alone, because there is nothing to expect. These are the false positives: the progress message, the policy note, the unconstrained path, and a line from elsewhere in the report that contains a number in a similar position.

Both lists matter, and the second one is the one people skip. A pattern tested only against lines it should match is tested against half its job. Every false positive in this module — the prose FATAL, the policy note read as a measurement, the 1842 endpoints — would have been caught by a MUST_NOT_MATCH entry.

And parse_slack takes its pattern as a defaulted argument. That looks like over-engineering for one caller and it is what makes Section 6 possible.

2. A Runner That Reports Everything

The runner collects failures rather than stopping at the first one:

import re

SLACK = r"slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+(?:\.\d+)?)"


def parse_slack(text, pattern=SLACK):
    m = re.search(pattern, text)
    return float(m.group(1)) if m is not None else None


MUST_MATCH = [
    ("  slack (VIOLATED)   -0.043", -0.043),
    ("  slack (MET)          0.031", 0.031),
    ("slack (VIOLATED) -1", -1.0),
    ("\tslack\t(MET)\t12.5", 12.5),
    ("  slack (VIOLATED)   -0.142  ", -0.142),
]

MUST_NOT_MATCH = [
    "Info: computing slack for 1842 endpoints",
    "  note: slack margin policy is 0.000 for this corner",
    "  slack (UNCONSTRAINED)",
    "  data arrival time          2.573",
]


def check(pattern):
    """Every failing case for one pattern, as (kind, text, want, got) tuples."""
    failures = []
    for text, want in MUST_MATCH:
        got = parse_slack(text, pattern)
        if got != want:
            failures.append(("must match", text, want, got))
    for text in MUST_NOT_MATCH:
        got = parse_slack(text, pattern)
        if got is not None:
            failures.append(("must not match", text, None, got))
    return failures


problems = check(SLACK)
print("cases   :", len(MUST_MATCH) + len(MUST_NOT_MATCH))
print("failures:", len(problems))
for kind, text, want, got in problems:
    print(f"  {kind:<14} {text.strip()[:30]!r:<34} want={want} got={got}")

cases   : 9
failures: 0

Nine cases, no failures, and the loop over problems printed nothing because there is nothing to print.

The failures list rather than a break is a deliberate choice. A suite that stops at the first mismatch turns five diagnosable problems into one, and the one it shows you is whichever happens to be first in the list — not the informative one. Chapter 19.17's whole method depends on seeing which cases failed, because the pattern of failures is the diagnosis.

Note also that check returns data and the printing happens outside it. That is Chapter 17.10's separation of pure logic from output, and it is what lets Section 6 call check in a loop without drowning in text.

3. Where the Samples Come From

The canonical purpose is specific about this: short real extracts from each tool and version. Each word is doing work.

Real. A line you typed from memory tests your memory. The samples in this chapter all came from the probes in Chapters 19.12 and 19.17, which came from the shapes real STA tools print — including -1 instead of -1.000, which nobody would invent.

Short. Three lines of context, not a 40-megabyte report. A sample has to live in the repository next to the code, be readable in a diff, and be quotable in a bug report.

From each tool and version. This is the part that pays off a year later:

import re

SLACK = r"slack(?:_ns)?\s+\((?:VIOLATED|MET)\)\s+(-?\d+(?:\.\d+)?)"


def parse_slack(text):
    m = re.search(SLACK, text)
    return float(m.group(1)) if m is not None else None


SAMPLES = {
    "sta_v3_violated": ("  slack (VIOLATED)          -0.142", -0.142),
    "sta_v3_met": ("  slack (MET)                0.031", 0.031),
    "sta_v4_renamed": ("  slack_ns (VIOLATED)       -0.142", -0.142),
    "sta_v4_integer": ("  slack_ns (VIOLATED)       -1", -1.0),
}

for name in sorted(SAMPLES):
    text, want = SAMPLES[name]
    got = parse_slack(text)
    print(f"{'ok  ' if got == want else 'FAIL'} {name:<16} want={want:>8} got={got}")

ok   sta_v3_met       want=   0.031 got=0.031
ok   sta_v3_violated  want=  -0.142 got=-0.142
ok   sta_v4_integer   want=    -1.0 got=-1.0
ok   sta_v4_renamed   want=  -0.142 got=-0.142

Naming each sample by tool and version makes the suite a record of what the parser is claimed to support. When a fifth tool arrives, you add two lines; when support for the third version is dropped, you delete two and find out immediately whether anything else depended on them. A flat list of anonymous strings supports none of that.

4. The assert Form

The canonical annotation is the compact version:

SAMPLES = [("slack (VIOLATED)   -0.043", -0.043)]
for text, want in SAMPLES:
    assert parse_slack(text) == want

And it works exactly as written:

import re

SLACK = r"slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+(?:\.\d+)?)"


def parse_slack(text):
    m = re.search(SLACK, text)
    return float(m.group(1)) if m is not None else None


SAMPLES = [("slack (VIOLATED)   -0.043", -0.043)]

for text, want in SAMPLES:
    assert parse_slack(text) == want

print("all", len(SAMPLES), "cases passed")

all 1 cases passed

Add a message and a failure becomes readable:

import re

SLACK = r"slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+(?:\.\d+)?)"


def parse_slack(text):
    m = re.search(SLACK, text)
    return float(m.group(1)) if m is not None else None


WRONG_EXPECTATION = [("slack (VIOLATED)   -0.043", -0.042)]

try:
    for text, want in WRONG_EXPECTATION:
        assert parse_slack(text) == want, f"{text!r}: want {want}, got {parse_slack(text)}"
except AssertionError as exc:
    print("AssertionError:", exc)

AssertionError: 'slack (VIOLATED)   -0.043': want -0.042, got -0.043

The message carries the case, the expectation and the reality, which is everything you need. An assert with no message tells you a line number and nothing else, and on a nine-case loop a line number identifies the loop rather than the case.

And this is the one place assert belongs. Chapter 14.7 established the rule — never use assert for a runtime safety check — and a test is not a runtime safety check. It is a statement about code that is being verified, run by a developer or a CI job, where stopping with a clear message is exactly the desired behaviour. The next section is why that distinction has to be stated precisely rather than assumed.

5. What -O Does to a Suite

Chapter 14.7 proved that python3 -O removes assert statements. For a suite, the consequence is specific and severe:

import re
import subprocess
import sys

CHILD = '''
import re
GOOD = r"slack\\s+\\((?:VIOLATED|MET)\\)\\s+(-?\\d+(?:\\.\\d+)?)"
def parse_slack(text):
    m = re.search(GOOD, text)
    return float(m.group(1)) if m else None
WRONG = [("slack (VIOLATED)   -0.043", -0.042)]
for text, want in WRONG:
    assert parse_slack(text) == want, f"{text!r}: want {want}, got {parse_slack(text)}"
print("suite reported: all cases passed")
'''

for flags in ([], ["-O"]):
    run = subprocess.run([sys.executable] + flags + ["-c", CHILD], capture_output=True, text=True)
    label = "python3 -O" if flags else "python3   "
    last = run.stderr.strip().splitlines()[-1] if run.stderr.strip() else "(no error)"
    print(f"{label}  rc={run.returncode}  stdout={run.stdout.strip()!r}")
    print(f"            {last}")

python3     rc=1  stdout=''
            AssertionError: 'slack (VIOLATED)   -0.043': want -0.042, got -0.043
python3 -O  rc=0  stdout='suite reported: all cases passed'

That is the same source, with the same deliberately wrong expectation, run twice. subprocess.run starts a child interpreter — sys.executable is the Python running this script — and captures what it printed; the only difference between the two runs is the -O flag.

Plain: return code 1, an AssertionError naming the case. Under -O: return code 0 and the words "all cases passed", from a suite whose single case is wrong. Every assert was removed before the loop ran, so the loop did nothing and the final print was reached.

A CI job that runs python3 -O on a test file reports a green build for a suite that tests nothing. The consequences are worth separating:

• assert in a test is correct, and remains correct — provided the test is never run with -O.
• assert in shipping code is Chapter 14.7's rule, and the reason is this exact mechanism.
• If you cannot be certain how a suite will be invoked, use the Section 2 runner. if got != want: failures.append(...) is not an assert and cannot be optimised away.

6. A Suite That Has Never Failed Is Not Known to Work

Nine passing cases say the pattern satisfies nine cases. They say nothing about whether the suite would notice if the pattern were wrong — and that is a separate, checkable question. Break the pattern deliberately, five ways, and see:

import re

SLACK = r"slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+(?:\.\d+)?)"

MUST_MATCH = [
    ("  slack (VIOLATED)   -0.043", -0.043),
    ("  slack (MET)          0.031", 0.031),
    ("slack (VIOLATED) -1", -1.0),
    ("\tslack\t(MET)\t12.5", 12.5),
    ("  slack (VIOLATED)   -0.142  ", -0.142),
]

MUST_NOT_MATCH = [
    "Info: computing slack for 1842 endpoints",
    "  note: slack margin policy is 0.000 for this corner",
    "  slack (UNCONSTRAINED)",
    "  data arrival time          2.573",
]


def parse_slack(text, pattern):
    m = re.search(pattern, text)
    return float(m.group(1)) if m is not None else None


def check(pattern):
    failures = []
    for text, want in MUST_MATCH:
        got = parse_slack(text, pattern)
        if got != want:
            failures.append(("must match", text, want, got))
    for text in MUST_NOT_MATCH:
        got = parse_slack(text, pattern)
        if got is not None:
            failures.append(("must not match", text, None, got))
    return failures


MUTANTS = [
    ("sign removed", r"slack\s+\((?:VIOLATED|MET)\)\s+(\d+(?:\.\d+)?)"),
    ("literal space", r"slack \((?:VIOLATED|MET)\) +(-?\d+(?:\.\d+)?)"),
    ("fraction required", r"slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+\.\d+)"),
    ("tag dropped", r"slack\D*(-?\d+(?:\.\d+)?)"),
    ("MET only", r"slack\s+\(MET\)\s+(-?\d+(?:\.\d+)?)"),
]

print("shipped pattern failures:", len(check(SLACK)))
print()
for name, mutant in MUTANTS:
    failures = check(mutant)
    kind, text, want, got = failures[0]
    print(f"{name:<18} caught by {len(failures)} case(s)")
    print(f"                   first: {kind}  {text.strip()[:28]!r}  want={want} got={got}")

shipped pattern failures: 0

sign removed       caught by 3 case(s)
                   first: must match  'slack (VIOLATED)   -0.043'  want=-0.043 got=None
literal space      caught by 1 case(s)
                   first: must match  'slack\t(MET)\t12.5'  want=12.5 got=None
fraction required  caught by 1 case(s)
                   first: must match  'slack (VIOLATED) -1'  want=-1.0 got=None
tag dropped        caught by 5 case(s)
                   first: must match  'slack (VIOLATED)   -0.043'  want=-0.043 got=0.043
MET only           caught by 3 case(s)
                   first: must match  'slack (VIOLATED)   -0.043'  want=-0.043 got=None

Five mutations, five caught, and the shipped pattern still at zero failures. That is the evidence the suite works, and it is evidence of a different kind from the nine passing cases — those show the pattern is right, this shows the suite can tell.

Each mutation is one of this module's defects, deliberately reintroduced:

sign removed drops the -? from Chapter 19.12 Section 6 — three cases fail because three samples are negative.

literal space is Chapter 19.12 Stage 3's mistake, and it is caught by one case: the tab sample. That number matters and Section 7 is about it.

fraction required is the naive \d+\.\d+, caught only by the integer-slack sample.

tag dropped is the interesting one. It is caught by all five must-match cases and by none of the must-not-match ones — and look at the value: want=-0.043 got=0.043. The \D* consumed the minus sign along with the brackets, so the mutant matches and returns a positive number for a violated path. A must-match list that only checked whether a line matched would have passed this mutant completely. The expected value in each pair is what caught it.

MET only removes an alternative, and the three VIOLATED samples fail.

7. Which Sample Is Load-Bearing?

Section 6 produced two mutants caught by exactly one case each, which raises an obvious and answerable question: what happens to the suite if a reviewer deletes a sample that looks redundant? Drop each must-match case in turn and re-run every mutant:

import re

SLACK = r"slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+(?:\.\d+)?)"

MUST_MATCH = [
    ("  slack (VIOLATED)   -0.043", -0.043),
    ("  slack (MET)          0.031", 0.031),
    ("slack (VIOLATED) -1", -1.0),
    ("\tslack\t(MET)\t12.5", 12.5),
]

MUST_NOT_MATCH = [
    "Info: computing slack for 1842 endpoints",
    "  slack (UNCONSTRAINED)",
]

MUTANTS = [
    ("sign removed", r"slack\s+\((?:VIOLATED|MET)\)\s+(\d+(?:\.\d+)?)"),
    ("literal space", r"slack \((?:VIOLATED|MET)\) +(-?\d+(?:\.\d+)?)"),
    ("fraction required", r"slack\s+\((?:VIOLATED|MET)\)\s+(-?\d+\.\d+)"),
    ("tag dropped", r"slack\D*(-?\d+(?:\.\d+)?)"),
]


def parse_slack(text, pattern):
    m = re.search(pattern, text)
    return float(m.group(1)) if m is not None else None


def caught(pattern, must_match, must_not_match):
    """True if this pattern fails at least one case."""
    for text, want in must_match:
        if parse_slack(text, pattern) != want:
            return True
    for text in must_not_match:
        if parse_slack(text, pattern) is not None:
            return True
    return False


print("full suite catches all mutants:",
      all(caught(mutant, MUST_MATCH, MUST_NOT_MATCH) for _, mutant in MUTANTS))
print()
for dropped in range(len(MUST_MATCH)):
    reduced = []
    for index, case in enumerate(MUST_MATCH):
        if index != dropped:
            reduced.append(case)
    survivors = []
    for name, mutant in MUTANTS:
        if not caught(mutant, reduced, MUST_NOT_MATCH):
            survivors.append(name)
    label = MUST_MATCH[dropped][0].strip()[:22]
    print(f"without {label!r:<26} survivors: {survivors}")

full suite catches all mutants: True

without 'slack (VIOLATED)   -0.'   survivors: []
without 'slack (MET)          0'   survivors: []
without 'slack (VIOLATED) -1'      survivors: ['fraction required']
without 'slack\t(MET)\t12.5'       survivors: ['literal space']

Two of the four samples are the sole witness for one symbol each.

Delete the -1 sample and the suite stops noticing that (?:\.\d+)? was replaced by a required fraction — the exact defect from Chapter 19.12 Section 6, which silently drops the worst path in a report.

Delete the tab sample and the suite stops noticing that \s+ was replaced by a literal space — the defect from Chapter 19.12 Stage 3.

The first two samples are each caught by other cases, so dropping either leaves the suite fully effective. That is not an argument for deleting them — they document the two normal shapes of the data, and a suite is documentation as well as a check — but it is useful to know which cases are carrying the weight.

And it is the answer to a review comment. "Why is there a sample with tabs in it, the tool uses spaces" has a precise reply: because it is the only case that fails when \s+ becomes a space, and here is the run that shows it.

8. Predict, Run, Observe

import re

PAT = r"seed=(\d+)"


def parse_seed(text, pattern=PAT):
    m = re.search(pattern, text)
    return int(m.group(1)) if m is not None else None


CASES = [("seed=1839274", 1839274), ("seed=0", 0), ("seed=00", 0)]
REJECT = ["seed=", "seed=abc", "no seed here"]

# A
print("A:", [parse_seed(t) == w for t, w in CASES])

# B
print("B:", [parse_seed(t) for t in REJECT])

# C
print("C:", parse_seed("seed=12", r"seed=(\d)"))

# D
print("D:", [parse_seed(t, r"seed=(\d)") == w for t, w in CASES])

# E
print("E:", parse_seed("x seed=5 y", r"^seed=(\d+)"))

# F
print("F:", parse_seed("note: seed policy for 12 corners", r"seed\D*(\d+)"))

The verified output:

A: [True, True, True]
B: [None, None, None]
C: 1
D: [False, True, True]
E: None
F: 12

A — all three must-match cases pass, including seed=00, which int() turns into 0.

B — all three rejections return None. seed= has no digits, so \d+ fails.

C — a mutant taking one digit returns 1 from seed=12. Wrong, and plausible.

D — the suite does catch that mutant, but only just: one False out of three. The two single-digit cases pass under the broken pattern, because seed=0 and seed=00 both still reach 0. If the first case were absent, a one-digit mutant would survive a suite that looks perfectly reasonable — which is Section 7's drop-one analysis arriving in miniature.

E — an anchored mutant fails on a mid-line seed, which a must-match case would catch.

F — a loosened mutant reads 12 out of prose and would need a must-not-match case to catch it, since it returns a value rather than None.

9. Exercises

Exercise 1 — two lists. For a pattern of your own, write four must-match pairs and four must-not-match lines. Say what each of the four rejections represents in the real data.

Exercise 2 — pairs, not lines. Take one must-match case and change the expected value to a wrong number. Confirm your runner reports it, and say what a match-only test would have reported.

Exercise 3 — report everything. Write the Section 2 runner, then rewrite it with a break on the first failure. Introduce three failures and compare the two outputs.

Exercise 4 — real and short. Find or recall one real tool line whose shape you would not have invented. Add it with its expected value and say what it tests that the others do not.

Exercise 5 — name by version. Convert a flat sample list into a dictionary keyed by tool and version. Say what the suite now documents that it did not before.

Exercise 6 — the assert form. Write the canonical two-line form for three cases, then give the assertion a message that names the case, the expectation and the reality.

Exercise 7 — the -O demonstration. Run a deliberately wrong assertion in a child interpreter twice, with and without -O, and report both return codes. Say what a CI job using -O would have reported.

Exercise 8 — which form to ship. For each, choose assert or the collected-failures runner and justify it: (a) a scratch check while writing a pattern; (b) a suite run by a nightly job you do not control; (c) a module-level sanity check inside shipping code.

Exercise 9 — five mutations. Mutate your own pattern five ways, each reintroducing a real defect from this module. Report how many cases catch each.

Exercise 10 — the mutant that survives. Keep mutating until you find one your suite does not catch. Add the sample that catches it, and say what that sample tests.

Exercise 11 — value, not presence. Build a mutant that matches every must-match line and returns a wrong value. Confirm only the expected values catch it.

Exercise 12 — drop one. Run Section 7's drop-one analysis on your suite. Name every sample that is a sole witness, and write the one-sentence review reply you would give for each.

10. Summary

• A pattern's test is two lists: must-match pairs of text and expected value, and must-not-match lines.
• Pairs, not lines. A match-only test cannot catch a pattern that matches and returns the wrong number — which is the defect that reaches reports.
• The must-not-match list is the one people skip, and every false positive in this module would have been caught by an entry in it.
• Collect failures; never break on the first. The pattern of failures is the diagnosis.
• Keep the parser's pattern as a defaulted argument so the suite can run it against a deliberately broken pattern.
• Samples should be real, short, and named by tool and version — which makes the suite a record of what the parser claims to support.
• The line that exposed a bug is the most valuable sample you will get. Add it, watch it fail, then fix the pattern.
• assert is correct in a test — Chapter 14.7's rule is about runtime safety checks, and a test is not one.
• Give every assertion a message naming the case, the expectation and the reality; a line number identifies the loop, not the case.
• python3 -O strips every assert, so a suite run that way reports return code 0 and "all cases passed" with a deliberately wrong expectation.
• If you cannot control how the suite is invoked, use the collected-failures runner — an if cannot be optimised away.
• A suite that has never failed is not known to work. Break the pattern five ways and confirm the suite goes red.
• A mutant can be caught by value rather than by absence — slack\D* returns a positive number for a violated path, which only the expected values reveal.
• Drop one sample at a time to find the sole witnesses. The awkward-looking sample with tabs in it is often the only case standing between you and a real defect.

The last chapter of this module is the one that argues against the whole module in the places where something else is better.