Python Part II · Module 26 · Regression Automation

Classifying Failures Into Buckets

A regression that reports twelve failures gets read line by line. One that reports four hundred gets skimmed, and then gets a rule in somebody's mail client. The difference is not that four hundred failures are harder to understand — it is usually one bug thirty-eight times over, which is easier — but that nothing in the report says so. Grouping them is two lines of code, and the interesting part is the field those two lines depend on, which does not exist yet.

Advanced19 min readPythonPart IIAutomationRegressionVerification Flow

Part II · Module 26 · Chapter 26.7 · Classifying Failures Into Buckets

1. Four Hundred Lines Nobody Reads

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

FAILURES = 400

print("a report that lists every failure:")
print("   lines              :", FAILURES)
print("   read in the morning :", "the first ten")
print("   acted on            :", "whichever one somebody recognised")
print()
print("a report that groups them:")
print("   lines              :", 5)
print("   read in the morning :", "all of them")
print("   acted on            :", "the largest, first")
print()
print("the failures are the same failures:", True)
print("what changed:", "whether the report says how many distinct problems there are")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

a report that lists every failure:
   lines              : 400
   read in the morning : the first ten
   acted on            : whichever one somebody recognised

a report that groups them:
   lines              : 5
   read in the morning : all of them
   acted on            : the largest, first

the failures are the same failures: True
what changed: whether the report says how many distinct problems there are

The regression already knows everything needed to produce the second report. Chapter 25.6 gave each result a status and a reason, Chapter 20.10 gave each failing log a first error, and Chapter 26.6 collected them all. What is missing is a category — and the annotation for this chapter assumes it already exists.

2. The Coarse Level Is Nearly Free

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

RESULTS = [("axi_smoke", "FAIL", "UVM_ERROR"), ("axi_reset", "FAIL", "UVM_ERROR"),
           ("axi_burst", "FAIL", "UVM_FATAL"), ("axi_hang", "NO_FINISH", "TIMEOUT"),
           ("axi_lic", "ERROR", "no licence"), ("axi_bad", "COMPILE_ERROR", None)]


def coarse(status, reason):
    if status == "COMPILE_ERROR":
        return "compile"
    if status == "ERROR":
        return "infrastructure"
    if status == "NO_FINISH":
        return "timeout" if reason == "TIMEOUT" else "no-end-marker"
    if status == "FAIL":
        return {"UVM_FATAL": "fatal", "ASSERT": "assertion"}.get(reason, "uvm-error")
    return "pass"


for test, status, reason in RESULTS:
    print(f"{test:<11} {status:<14} {str(reason):<12} -> {coarse(status, reason)}")

print()
print("distinct coarse categories:", len({coarse(s, r) for _, s, r in RESULTS}))

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

axi_smoke   FAIL           UVM_ERROR    -> uvm-error
axi_reset   FAIL           UVM_ERROR    -> uvm-error
axi_burst   FAIL           UVM_FATAL    -> fatal
axi_hang    NO_FINISH      TIMEOUT      -> timeout
axi_lic     ERROR          no licence   -> infrastructure
axi_bad     COMPILE_ERROR  None         -> compile

distinct coarse categories: 5

That is the canonical purpose's list — compile error, fatal, assertion, timeout, infrastructure — and it is almost entirely derived from fields Chapter 25.6 already produced. It costs nothing and it is worth having.

It is also not what a morning review needs, because the first two rows are in one bucket and may be two completely different bugs. A category that puts every UVM_ERROR together has grouped 380 of the 400 failures into one undifferentiated pile, which is the problem restated rather than solved.

3. Six Messages, Three Bugs

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

ERRORS = [
    "UVM_ERROR drv.sv(88) @ 141900: response mismatch exp=0x12 got=0x34",
    "UVM_ERROR drv.sv(88) @ 203450: response mismatch exp=0xab got=0xcd",
    "UVM_ERROR drv.sv(88) @ 99120: response mismatch exp=0x00 got=0xff",
    "UVM_ERROR mon.sv(31) @ 120500: illegal transition IDLE->DATA",
    "UVM_FATAL env.sv(12) @ 0: config object not found",
    "UVM_ERROR drv.sv(88) @ 77: response mismatch exp=0x01 got=0x02",
]

print("raw error lines   :", len(ERRORS))
print("distinct as strings:", len(set(ERRORS)))
print()
print("how many distinct bugs a person would say these are:", 3)
print("what makes four of them the same bug:", "the same file, line and message")
print("what makes them look different:", "the simulation time and the data values")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

raw error lines   : 6
distinct as strings: 6

how many distinct bugs a person would say these are: 3
what makes four of them the same bug: the same file, line and message
what makes them look different: the simulation time and the data values

Six lines, six distinct strings, three bugs. Grouping by the raw message gives six buckets of one, which is the same useless report with extra ceremony.

The four drv.sv(88) lines differ in exactly two kinds of thing: the simulation time and the data values. Both vary between runs of the same bug — a different seed hits the mismatch at a different time with different data — and neither identifies it.

4. The Signature

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

buckets = defaultdict(list)
buckets[res.category].append(res)

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import re
from collections import defaultdict

TIME = re.compile(r"@\s*\d+")
HEXV = re.compile(r"0x[0-9a-fA-F]+")

ERRORS = [
    "UVM_ERROR drv.sv(88) @ 141900: response mismatch exp=0x12 got=0x34",
    "UVM_ERROR drv.sv(88) @ 203450: response mismatch exp=0xab got=0xcd",
    "UVM_ERROR drv.sv(88) @ 99120: response mismatch exp=0x00 got=0xff",
    "UVM_ERROR mon.sv(31) @ 120500: illegal transition IDLE->DATA",
    "UVM_FATAL env.sv(12) @ 0: config object not found",
    "UVM_ERROR drv.sv(88) @ 77: response mismatch exp=0x01 got=0x02",
]


def signature(line):
    """Replace what varies between runs of one bug. Nothing else."""
    return HEXV.sub("0xV", TIME.sub("@T", line))


buckets = defaultdict(list)
for line in ERRORS:
    buckets[signature(line)].append(line)

print("distinct signatures:", len(buckets), "from", len(ERRORS), "lines")
print()
for sig, members in sorted(buckets.items(), key=lambda kv: -len(kv[1])):
    print(f"{len(members):>2}  {sig}")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

distinct signatures: 3 from 6 lines

 4  UVM_ERROR drv.sv(88) @T: response mismatch exp=0xV got=0xV
 1  UVM_ERROR mon.sv(31) @T: illegal transition IDLE->DATA
 1  UVM_FATAL env.sv(12) @T: config object not found

Three signatures from six lines, and the four-member bucket is the one to look at first.

The annotation is those two lines — a defaultdict(list) and an append — and they are genuinely all the bucketing needs. The work is signature, and it is built on a single rule:

Normalise the values that vary between runs of one bug. Nothing else.

A simulation time varies; a file and line do not. A data value varies; a message text does not. Each substitution has to be justified by that test, and the next section is what happens when one is not.

5. One Rule, Applied Twice More

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import re

TIME = re.compile(r"@\s*\d+")
HEXV = re.compile(r"0x[0-9a-fA-F]+")
SEED = re.compile(r"\bseed[= ]\d+", re.IGNORECASE)
ADDR = re.compile(r"\b[0-9a-f]{8,}\b", re.IGNORECASE)

CANDIDATES = [
    ("the simulation time", "@ 141900", True, "a different seed hits it at a different time"),
    ("a hex data value", "0x12", True, "the payload differs run to run"),
    ("the seed", "seed=1839274", True, "every job has a different one"),
    ("a long hex address", "deadbeefcafe", True, "an allocated address varies"),
    ("the file and line", "drv.sv(88)", False, "this is what identifies the bug"),
    ("the message text", "response mismatch", False, "likewise"),
    ("a small integer", "(88)", False, "it is the line number"),
]

print(f"{'value':<24} {'example':<18} {'normalise?':<11} why")
for name, example, yes, why in CANDIDATES:
    print(f"{name:<24} {example:<18} {str(yes):<11} {why}")

print()
print("normalise:", sum(1 for *_, y, _ in CANDIDATES if y))
print("keep     :", sum(1 for *_, y, _ in CANDIDATES if not y))

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

value                    example            normalise?  why
the simulation time      @ 141900           True        a different seed hits it at a different time
a hex data value         0x12               True        the payload differs run to run
the seed                 seed=1839274       True        every job has a different one
a long hex address       deadbeefcafe       True        an allocated address varies
the file and line        drv.sv(88)         False       this is what identifies the bug
the message text         response mismatch  False       likewise
a small integer          (88)               False       it is the line number

normalise: 4
keep     : 3

Four substitutions, three things kept, and every row decided by the same question. The last two rows are the interesting pair: 88 and 141900 are both bare integers, and one identifies the bug while the other varies with every run of it.

That is why the patterns are anchored to context — @\s*\d+ matches a time because of the @, and 0x... matches a value because of the prefix. A pattern that matched bare digits could not tell them apart.

6. Over-Normalising Merges Two Bugs

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import re

TIME = re.compile(r"@\s*\d+")
HEXV = re.compile(r"0x[0-9a-fA-F]+")
ANYNUM = re.compile(r"\b\d+\b")

ERRORS = [
    "UVM_ERROR drv.sv(88) @ 141900: response mismatch exp=0x12 got=0x34",
    "UVM_ERROR drv.sv(204) @ 9000: fifo overflow depth=0x10",
    "UVM_ERROR mon.sv(31) @ 120500: illegal transition IDLE->DATA",
]


def careful(line):
    return HEXV.sub("0xV", TIME.sub("@T", line))


def greedy(line):
    return ANYNUM.sub("N", careful(line))


print("careful:")
for sig in sorted({careful(e) for e in ERRORS}):
    print("  ", sig)
print("   distinct:", len({careful(e) for e in ERRORS}))

print()
print("greedy (bare integers replaced too):")
for sig in sorted({greedy(e) for e in ERRORS}):
    print("  ", sig)
print("   distinct:", len({greedy(e) for e in ERRORS}))
print()
print("two different bugs in drv.sv are now:",
      len({greedy(e) for e in ERRORS if "drv.sv" in e}), "signature(s)")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

careful:
   UVM_ERROR drv.sv(204) @T: fifo overflow depth=0xV
   UVM_ERROR drv.sv(88) @T: response mismatch exp=0xV got=0xV
   UVM_ERROR mon.sv(31) @T: illegal transition IDLE->DATA
   distinct: 3

greedy (bare integers replaced too):
   UVM_ERROR drv.sv(N) @T: fifo overflow depth=0xV
   UVM_ERROR drv.sv(N) @T: response mismatch exp=0xV got=0xV
   UVM_ERROR mon.sv(N) @T: illegal transition IDLE->DATA
   distinct: 3

two different bugs in drv.sv are now: 2 signature(s)

Here the greedy version happens to survive, because the two drv.sv bugs have different message text. The line numbers merged and the messages kept them apart.

That is luck, not correctness. Two bugs on different lines of one file reporting the same message — two instances of an overflow check, two identical assertions in a generated block — collapse into one bucket under the greedy rule, and the report then says one bug where there are two. The failure is silent and in the direction that matters: under-reporting distinct problems, which is Chapter 26.1 Section 4's optimistic error.

7. The Shape of a Morning

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

from collections import Counter

BUCKETS = (["drv.sv(88) response mismatch"] * 380
           + ["mon.sv(31) illegal transition"] * 12
           + ["timeout"] * 5
           + ["env.sv(12) config not found"] * 2
           + ["infrastructure: no licence"] * 1)

counts = Counter(BUCKETS)
total = len(BUCKETS)

print("failures        :", total)
print("distinct buckets:", len(counts))
print()
for name, n in counts.most_common():
    print(f"{n:>4}  {100 * n / total:>5.1f}%  {name}")

print()
top, top_n = counts.most_common(1)[0]
print("largest bucket  :", f"{100 * top_n / total:.0f}% of all failures")
print("distinct problems:", len(counts))
print("where to start  :", top)

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

failures        : 400
distinct buckets: 5

 380   95.0%  drv.sv(88) response mismatch
  12    3.0%  mon.sv(31) illegal transition
   5    1.2%  timeout
   2    0.5%  env.sv(12) config not found
   1    0.2%  infrastructure: no licence

largest bucket  : 95% of all failures
distinct problems: 5
where to start  : drv.sv(88) response mismatch

Four hundred failures, five problems, and one of them is ninety-five percent of the noise.

That is the report this chapter exists to produce, and the thing worth noticing is how different it feels from the same data as a list. A list of four hundred failures reads as a catastrophe; this reads as one bug, one secondary bug, and three stragglers — which is both more accurate and more actionable.

Sorting by size is not cosmetic. most_common() puts the 380 first, so the single most valuable piece of information in the report is on its first line. A report sorted by test name buries it at whatever position drv happens to fall.

The small buckets are worth keeping visible too. The single infrastructure failure is one ERROR — Chapter 26.6 counted it and excluded it from the denominator, and here it is a line somebody can act on rather than a number.

8. A Representative per Bucket

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

from collections import defaultdict
from dataclasses import dataclass


@dataclass(frozen=True)
class Failure:
    test: str
    seed: int
    signature: str
    run_dir: str

    @property
    def reproduce(self):
        return f"flow run --test {self.test} --seed {self.seed} --waves"


FAILURES = [
    Failure("axi_burst", 1839274, "drv.sv(88) response mismatch", "runs/axi_burst_1839274"),
    Failure("axi_burst", 1839275, "drv.sv(88) response mismatch", "runs/axi_burst_1839275"),
    Failure("axi_smoke", 1839280, "drv.sv(88) response mismatch", "runs/axi_smoke_1839280"),
    Failure("axi_mon", 1839274, "mon.sv(31) illegal transition", "runs/axi_mon_1839274"),
]

buckets = defaultdict(list)
for f in FAILURES:
    buckets[f.signature].append(f)

for sig, members in sorted(buckets.items(), key=lambda kv: -len(kv[1])):
    rep = min(members, key=lambda f: (f.test, f.seed))
    tests = sorted({f.test for f in members})
    print(f"{len(members):>2}  {sig}")
    print(f"      tests affected: {tests}")
    print(f"      debug this one: {rep.reproduce}")
    print(f"      its evidence  : {rep.run_dir}")
    print()

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

 3  drv.sv(88) response mismatch
      tests affected: ['axi_burst', 'axi_smoke']
      debug this one: flow run --test axi_burst --seed 1839274 --waves
      its evidence  : runs/axi_burst_1839274

 1  mon.sv(31) illegal transition
      tests affected: ['axi_mon']
      debug this one: flow run --test axi_mon --seed 1839274 --waves
      its evidence  : runs/axi_mon_1839274

Three things per bucket, and each answers a different question.

The count says how much of the regression this one problem accounts for.

The tests affected says whether it is a design bug or a test bug. The first bucket appears under two different tests, which makes a shared-logic problem far more likely than a testbench one — a distinction that is invisible when the failures are listed individually and obvious once they are grouped.

The representative is the one job somebody actually debugs, with Chapter 25.5 Section 5's reproduce command and Chapter 26.4's directory. Choosing it by min(...) on (test, seed) rather than by position makes the report stable: the same failures produce the same representative tomorrow, so a bucket somebody is already working on does not silently switch to a different example.

9. New, Known, and Gone

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

YESTERDAY = {"drv.sv(88) response mismatch": 380,
             "mon.sv(31) illegal transition": 12,
             "env.sv(12) config not found": 2}
TODAY = {"drv.sv(88) response mismatch": 12,
         "mon.sv(31) illegal transition": 12,
         "fifo.sv(44) overflow": 7}

appeared = sorted(set(TODAY) - set(YESTERDAY))
gone = sorted(set(YESTERDAY) - set(TODAY))
shared = sorted(set(TODAY) & set(YESTERDAY))

print("NEW today:")
for s in appeared:
    print(f"   {TODAY[s]:>4}  {s}")
print("GONE since yesterday:")
for s in gone:
    print(f"   {YESTERDAY[s]:>4}  {s}")
print("still here:")
for s in shared:
    print(f"   {TODAY[s]:>4}  {s}  ({TODAY[s] - YESTERDAY[s]:+d})")

print()
print("total failures yesterday:", sum(YESTERDAY.values()))
print("total failures today    :", sum(TODAY.values()))
print("the headline moved by   :", sum(TODAY.values()) - sum(YESTERDAY.values()))

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

NEW today:
      7  fifo.sv(44) overflow
GONE since yesterday:
      2  env.sv(12) config not found
still here:
     12  drv.sv(88) response mismatch  (-368)
     12  mon.sv(31) illegal transition  (+0)

total failures yesterday: 394
total failures today    : 31
the headline moved by   : -363

The headline improved by 363 failures and there is a brand-new bug in it.

This is what a morning review actually does, and no single night's report can show it. The three sets each answer a question today's numbers alone cannot.

New is the only thing that is genuinely urgent. fifo.sv(44) overflow did not exist yesterday, so something that merged since then caused it — and seven failures is small enough to sit at the bottom of a 31-failure report sorted by size, which is exactly where Section 7's ordering would bury it.

Gone is the quiet one and it is not always good news. A bucket disappearing usually means a fix landed, and it sometimes means the test that found it stopped running — Chapter 26.1's not_run, or a suite edit. A disappearance is worth one glance at whether the job count also changed.

Still here, with a delta. drv.sv(88) went from 380 to 12: a fix working on most paths and not all. That count is a far better progress measure than the headline, which moved for a different reason entirely.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

YESTERDAY = {"a": 380, "b": 12}
TODAY = {"a": 12, "b": 12, "c": 7}

print(f"{'bucket':<8} {'yday':>6} {'today':>6} {'delta':>7}  status")
for name in sorted(set(YESTERDAY) | set(TODAY)):
    y, t = YESTERDAY.get(name), TODAY.get(name)
    if y is None:
        print(f"{name:<8} {'-':>6} {t:>6} {'NEW':>7}  investigate first")
    elif t is None:
        print(f"{name:<8} {y:>6} {'-':>6} {'GONE':>7}  fixed, or no longer tested")
    else:
        print(f"{name:<8} {y:>6} {t:>6} {t - y:>+7}  ongoing")

print()
print("sorted by size   :", sorted(TODAY, key=lambda k: -TODAY[k]))
print("sorted by newness:", sorted(TODAY, key=lambda k: (k in YESTERDAY, -TODAY[k])))

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

bucket     yday  today   delta  status
a           380     12    -368  ongoing
b            12     12      +0  ongoing
c             -      7     NEW  investigate first

sorted by size   : ['a', 'b', 'c']
sorted by newness: ['c', 'a', 'b']

Size-ordering and newness-ordering disagree, and the new bucket is last under one and first under the other. Section 7's argument for sorting by size holds within a single night; across nights, new beats big.

What makes any of this work is that a signature is stable. Section 5's rule — normalise only what varies between runs of one bug — is also what makes the same bug produce the same key tomorrow. A signature containing a timestamp would make every bucket new every night and the comparison would report nothing but noise.

10. The Classifier, Assembled

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import re
from collections import Counter, defaultdict
from dataclasses import dataclass
from typing import Optional

TIME = re.compile(r"@\s*\d+")
HEXV = re.compile(r"0x[0-9a-fA-F]+")
SEED = re.compile(r"\bseed[= ]\d+", re.IGNORECASE)
LONGHEX = re.compile(r"\b[0-9a-f]{8,}\b", re.IGNORECASE)


@dataclass(frozen=True)
class Result:
    test: str
    seed: int
    status: str
    reason: Optional[str] = None
    first_error: Optional[str] = None
    run_dir: Optional[str] = None


def signature(result):
    """Coarse category, refined by a normalised first error where one exists."""
    if result.status == "COMPILE_ERROR":
        return "compile"
    if result.status == "ERROR":
        return f"infrastructure: {result.reason}"
    if result.status == "NO_FINISH":
        return f"no-finish: {result.reason}"
    if result.status != "FAIL":
        return "pass"
    if not result.first_error:
        return "fail: no error line found"
    s = result.first_error
    for pattern, replacement in ((TIME, "@T"), (LONGHEX, "HEX"),
                                 (HEXV, "0xV"), (SEED, "seed=S")):
        s = pattern.sub(replacement, s)
    return s


def classify(results):
    buckets = defaultdict(list)
    for r in results:
        if r.status == "PASS":
            continue
        buckets[signature(r)].append(r)
    return buckets


RESULTS = [
    Result("axi_burst", 1, "FAIL", "UVM_ERROR",
           "UVM_ERROR drv.sv(88) @ 141900: mismatch exp=0x12", "runs/axi_burst_1"),
    Result("axi_burst", 2, "FAIL", "UVM_ERROR",
           "UVM_ERROR drv.sv(88) @ 9000: mismatch exp=0xff", "runs/axi_burst_2"),
    Result("axi_smoke", 3, "FAIL", "UVM_ERROR",
           "UVM_ERROR drv.sv(88) @ 77: mismatch exp=0x01", "runs/axi_smoke_3"),
    Result("axi_mon", 4, "FAIL", "UVM_ERROR",
           "UVM_ERROR mon.sv(31) @ 120500: illegal transition", "runs/axi_mon_4"),
    Result("axi_hang", 5, "NO_FINISH", "TIMEOUT", None, "runs/axi_hang_5"),
    Result("axi_lic", 6, "ERROR", "no licence", None, "runs/axi_lic_6"),
    Result("axi_odd", 7, "FAIL", "UVM_ERROR", None, "runs/axi_odd_7"),
    Result("axi_ok", 8, "PASS", None, None, "runs/axi_ok_8"),
]

buckets = classify(RESULTS)
failures = sum(len(v) for v in buckets.values())

print("results        :", len(RESULTS))
print("non-passing    :", failures)
print("distinct buckets:", len(buckets))
print()
for sig, members in sorted(buckets.items(), key=lambda kv: (-len(kv[1]), kv[0])):
    rep = min(members, key=lambda r: (r.test, r.seed))
    print(f"{len(members):>2}  {sig}")
    print(f"      tests: {sorted({m.test for m in members})}  start: {rep.run_dir}")

print()
print("every non-passing result is in exactly one bucket:",
      failures == sum(1 for r in RESULTS if r.status != "PASS"))

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

results        : 8
non-passing    : 7
distinct buckets: 5

 3  UVM_ERROR drv.sv(88) @T: mismatch exp=0xV
      tests: ['axi_burst', 'axi_smoke']  start: runs/axi_burst_1
 1  UVM_ERROR mon.sv(31) @T: illegal transition
      tests: ['axi_mon']  start: runs/axi_mon_4
 1  fail: no error line found
      tests: ['axi_odd']  start: runs/axi_odd_7
 1  infrastructure: no licence
      tests: ['axi_lic']  start: runs/axi_lic_6
 1  no-finish: TIMEOUT
      tests: ['axi_hang']  start: runs/axi_hang_5

Five decisions, and the third is the one that keeps the report honest.

The coarse category handles everything that is not a FAIL, so a timeout and a licence error get readable buckets without needing a log line at all.

The fine signature only applies to FAIL, which is the only status where the status alone is insufficient.

A FAIL with no first error gets its own bucket, rather than being dropped or merged into another. axi_odd failed and no error line was found — which is itself a finding, probably about the parser rather than the design — and a classifier that silently discarded it would hide a gap in Module 20's patterns.

The substitution order matters. LONGHEX runs before HEXV because 0xdeadbeefcafe would otherwise become 0xV and lose the information that it was an address; running the long-hex pattern first is the same precedence reasoning Chapter 25.6 Section 4 applied to statuses.

Every non-passing result lands in exactly one bucket, asserted at the end. A classifier that drops a result is Chapter 26.1's missing job in a different costume, and the check costs one line.

11. Predict, Run, Observe

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import re
from collections import Counter, defaultdict

T = re.compile(r"@\s*\d+")
H = re.compile(r"0x[0-9a-fA-F]+")
N = re.compile(r"\b\d+\b")

E = ["UVM_ERROR drv.sv(88) @ 100: m exp=0x1",
     "UVM_ERROR drv.sv(88) @ 200: m exp=0x2",
     "UVM_ERROR drv.sv(99) @ 300: m exp=0x3"]

# A
print("A:", len(set(E)))

# B
sig = lambda s: H.sub("0xV", T.sub("@T", s))
print("B:", len({sig(e) for e in E}))

# C
print("C:", sig(E[0]))

# D
greedy = lambda s: N.sub("N", sig(s))
print("D:", len({greedy(e) for e in E}))

# E
b = defaultdict(list)
for e in E:
    b[sig(e)].append(e)
print("E:", sorted(len(v) for v in b.values()))

# F
print("F:", Counter(sig(e) for e in E).most_common(1)[0][1])

# G
print("G:", min([("b", 2), ("a", 9), ("a", 1)]))

# H
print("H:", sorted(b, key=lambda k: -len(b[k]))[0][:28])

The verified output:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

A: 3
B: 2
C: UVM_ERROR drv.sv(88) @T: m exp=0xV
D: 1
E: [1, 2]
F: 2
G: ('a', 1)
H: UVM_ERROR drv.sv(88) @T: m

A — three distinct raw messages.

B — two signatures: the two drv.sv(88) lines merged.

C — the time and the hex value are what got replaced.

D — the greedy version collapses the line numbers too, giving one signature for two different lines. Section 6's hazard, in one number.

E — the careful buckets are sized two and one.

F — the largest has two members.

G — min on a tuple gives a stable representative.

H — sorting by size descending puts the biggest bucket first.

12. Exercises

Exercise 1 — the two reports. Take a real regression's failures and write both the list and the grouped form. Say which you would read.

Exercise 2 — already known. Name the three things the regression already has that a category is derived from.

Exercise 3 — the coarse level. Implement it from status and reason alone. Report how many categories you get.

Exercise 4 — why it is not enough. Say what fraction of your failures land in one coarse bucket.

Exercise 5 — six and three. Take six real error lines from one bug and count distinct strings.

Exercise 6 — what varies. For those lines, list exactly what differs between them.

Exercise 7 — the rule. State the normalisation rule in one sentence.

Exercise 8 — justify each. For every substitution in your signature function, say which value it removes and why that value varies.

Exercise 9 — anchored patterns. Explain why @\s*\d+ and \b\d+\b must be treated differently.

Exercise 10 — over-normalise. Add a bare-integer substitution and find a pair of bugs it merges.

Exercise 11 — the asymmetry. Argue in three sentences why too loose is worse than too tight.

Exercise 12 — your morning. Bucket your last regression's failures and report the largest bucket's percentage.

Exercise 13 — sort by size. Say what sorting by test name instead would bury.

Exercise 14 — keep the small ones. Give a reason to show a one-member bucket in the report.

Exercise 15 — tests affected. Find a bucket spanning two tests and say what that suggests.

Exercise 16 — a stable representative. Say why choosing by position rather than by min is a problem across two days.

Exercise 17 — the empty first error. Produce a FAIL with no parsed error line and say what its bucket tells you.

Exercise 18 — substitution order. Construct a case where applying a narrow pattern before a broad one gives the wrong signature.

Exercise 19 — three sets. Compare two nights of your own buckets and report new, gone and still-here.

Exercise 20 — new beats big. Find a night where the new bucket was not the largest, and say where size-ordering put it.

Exercise 21 — gone is not always good. Give two reasons a bucket can disappear and say how to tell them apart.

Exercise 22 — the delta as progress. Say why a bucket's change beats the headline total.

Exercise 23 — stability. Explain why a timestamp in a signature destroys the night-to-night comparison.

Exercise 24 — nothing dropped. Add the assertion that every non-passing result is bucketed, and say which earlier chapter's bug it prevents.

13. Summary

Four hundred failures are usually a handful of problems, and nothing in a list says so.
The coarse category is nearly free — compile, fatal, assertion, timeout, infrastructure all follow from Chapter 25.6's status and reason.
It is also insufficient, because every UVM_ERROR lands in one bucket and that is most of the failures.
Six distinct error strings were three bugs. What differed was the simulation time and the data values.
Normalise the values that vary between runs of one bug, and nothing else.
Anchor the patterns to context — @\s*\d+ is a time and (88) is a line number, and only the surrounding text distinguishes them.
Over-normalising merges two bugs in one file, and it survives on many specimens by luck rather than correctness.
Too loose is worse than too tight: an extra bucket is self-correcting, a merged one gets the second bug closed as a duplicate.
Sort buckets by size — the 95% bucket belongs on the first line.
Keep the one-member buckets visible; a single infrastructure failure is actionable as a line and invisible as a number.
Report the count, the tests affected and a representative for each bucket.
A bucket spanning two tests suggests shared logic rather than a testbench bug, which only grouping makes visible.
Choose the representative with min on a stable key, so a bucket somebody is working on does not change example tomorrow.
Give a FAIL with no parsed error line its own bucket — that is a finding about the parser.
Apply the narrower pattern before the broader one, the same precedence reasoning as Chapter 25.6's classifier.
Compare tonight's buckets with last night's — new, gone and still-here-with-a-delta each answer a question one night cannot.
New beats big across nights: a seven-failure bucket that did not exist yesterday outranks a 380-failure one that did.
A bucket disappearing is not always a fix — the test may have stopped running.
A bucket's delta is a better progress measure than the headline, which moves for unrelated reasons.
The comparison works only because signatures are stable; a timestamp in one makes every bucket new every night.
Persist the bucket counts beside the summary — a dozen lines of JSON, and the highest-value artefact in the module.
Assert that every non-passing result is bucketed exactly once, or a dropped failure is Chapter 26.1's missing job again.

The next chapter makes a regression that died at eighty percent pick up where it stopped.

Where this fits

Part of the Python curriculum.