Grouping Errors for Triage

Part III · Module 9 · Chapter 9.4 · Grouping Errors for Triage

1. Why This Matters

A failing test produces a log, and the first thing anybody does with it is scroll. Hundreds of error lines go past, almost all of them the same check firing on different data, and somewhere among them is a second error that is a different problem entirely.

Grouping is what makes that second error visible. It is the single highest-value transformation in this module, because it changes a log from something you read to something you can act on.

It is rated critical for a reason that is not about the code: the grouping key decides how many problems you appear to have, and the number you report is the one a lead uses to allocate a day of somebody's time.

2. A Log Where the Choice Matters

UVM_INFO @ 0: reporter [RNTST] Running test mixed_test...
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1000: uvm_test_top.env.sb0 [MISMATCH] exp=1 act=2
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1100: uvm_test_top.env.sb0 [MISMATCH] exp=3 act=4
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1200: uvm_test_top.env.sb1 [MISMATCH] exp=5 act=6
UVM_ERROR ../tb/env/alu_sb.sv(201) @ 1300: uvm_test_top.env.sb0 [MISMATCH] exp=7 act=8
UVM_ERROR ../tb/env/fifo_sb.sv(88) @ 1400: uvm_test_top.env.fsb [MISMATCH] exp=9 act=10
UVM_ERROR ../tb/env/alu_cov.sv(77) @ 2000: uvm_test_top.env.cov [NO_HIT] bin never hit
--- UVM Report Summary ---
UVM_ERROR :    6

Six error records. Five of them carry the id MISMATCH, which is the point — a single id is reused by every scoreboard in a testbench, because MISMATCH is the obvious thing to call a mismatch.

Look at what differs between them: two different source files, two different line numbers within one of those files, and three different component instances. Each of those is a candidate grouping key.

3. Three Keys, Three Answers

awk '/^--- UVM Report Summary ---/ { s = 1 } s { next } /^UVM_ERROR (\.|\/|@)/ { id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); h = ""; for (i = 1; i <= NF; i++) if ($i == "@") { h = $(i + 2); break } a[id]++; b[id SUBSEP $2]++; c[id SUBSEP $2 SUBSEP h]++; recs++ } END { na = 0; nb = 0; nc = 0; for (k in a) na++; for (k in b) nb++; for (k in c) nc++; printf "%d record(s): by id=%d, by (id,loc)=%d, by (id,loc,hier)=%d\n", recs, na, nb, nc }' triage.log

6 record(s): by id=2, by (id,loc)=4, by (id,loc,hier)=5

Two, four or five distinct failures from the same six records.

4. By Id Alone — Too Coarse

awk '/^--- UVM Report Summary ---/ { s = 1 } s { next } /^UVM_ERROR (\.|\/|@)/ { id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); n[id]++ } END { for (k in n) printf "%-12s %d record(s)\n", k, n[k] | "sort -k2 -rn" }' triage.log

MISMATCH     5 record(s)
NO_HIT       1 record(s)

Two rows, and the table says there are two problems.

That is wrong in a specific way. The five MISMATCH records come from two different source files — alu_sb.sv and fifo_sb.sv — which are the ALU scoreboard and the FIFO scoreboard. Those are different blocks checked by different code, so they are at least two separate failures merged into one row.

The cause is that MISMATCH is a generic id. A verification engineer writing a scoreboard calls a mismatch MISMATCH, and so does every other engineer on the project. The id names a kind of failure, not an instance of one — which is exactly what Chapter 6.2 section 4 meant by choosing the key carefully.

So grouping by id alone hides work: a lead reading this assigns one investigation where three are needed.

5. By Id and Location — The Default

awk '/^--- UVM Report Summary ---/ { s = 1 } s { next } /^UVM_ERROR (\.|\/|@)/ { id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); n[id SUBSEP $2]++ } END { for (k in n) { split(k, p, SUBSEP); printf "%-12s %-28s %d record(s)\n", p[1], p[2], n[k] } }' triage.log | sort -k3,3rn -k1,1 -k2,2

MISMATCH     ../tb/env/alu_sb.sv(145)     3 record(s)
MISMATCH     ../tb/env/alu_sb.sv(201)     1 record(s)
MISMATCH     ../tb/env/fifo_sb.sv(88)     1 record(s)
NO_HIT       ../tb/env/alu_cov.sv(77)     1 record(s)

The secondary sort keys (-k1,1 -k2,2) matter: four rows share a count of 1, and sort gives no defined order among ties, so without them the table's row order varies between runs and a diff against yesterday's is useless — Chapter 7.7's point, in the place it actually bites.

Four rows, and each is a distinct check that fired. The two scoreboards are now separate, and so are the two different checks inside alu_sb.sv — line 145 and line 201 are different comparisons, probably of different fields.

This is the recommended default, and the reason is that file(line) identifies the code that detected the problem. Two records from the same line are the same check seeing bad data twice; two records from different lines are different checks, which usually means different symptoms and possibly different causes.

It is also stable in a useful way: the location changes only when somebody edits the testbench, so the same failure produces the same signature across runs — which is what Chapter 9.6 needs to tell a new failure from a known one.

6. Adding the Hierarchy — Usually Too Fine

awk '/^--- UVM Report Summary ---/ { s = 1 } s { next } /^UVM_ERROR (\.|\/|@)/ { id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); h = ""; for (i = 1; i <= NF; i++) if ($i == "@") { h = $(i + 2); break } n[id SUBSEP $2 SUBSEP h]++ } END { for (k in n) { split(k, p, SUBSEP); printf "%-10s %-26s %-24s %d\n", p[1], p[2], p[3], n[k] } }' triage.log | sort -k4,4rn -k1,1 -k2,2 -k3,3

MISMATCH   ../tb/env/alu_sb.sv(145)   uvm_test_top.env.sb0     2
MISMATCH   ../tb/env/alu_sb.sv(145)   uvm_test_top.env.sb1     1
MISMATCH   ../tb/env/alu_sb.sv(201)   uvm_test_top.env.sb0     1
MISMATCH   ../tb/env/fifo_sb.sv(88)   uvm_test_top.env.fsb     1
NO_HIT     ../tb/env/alu_cov.sv(77)   uvm_test_top.env.cov     1

Five rows. The first group from section 5 has split into sb0 and sb1.

Those are two instances of the same scoreboard, both running the same check at alu_sb.sv(145), both seeing a mismatch. On a design with two identical ALU channels that is almost certainly one bug — the same RTL defect appearing on both channels — and reporting it as two failures invents work.

The hierarchy extraction here uses the field scan from Chapter 9.2 section 6: find the field that is exactly @, then take the field two further on. The + 2 skips the time field.

7. Collapsing a Large Log

The value of grouping scales with the log. Chapter 9.3's fixture, where one bug dominates:

UVM_INFO @ 0: reporter [RNTST] Running test alu_test...
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1000: uvm_test_top.env.sb [MISMATCH] exp!=act
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1100: uvm_test_top.env.sb [MISMATCH] exp!=act
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1200: uvm_test_top.env.sb [MISMATCH] exp!=act
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1300: uvm_test_top.env.sb [MISMATCH] exp!=act
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1400: uvm_test_top.env.sb [MISMATCH] exp!=act
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1500: uvm_test_top.env.sb [MISMATCH] exp!=act
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1600: uvm_test_top.env.sb [MISMATCH] exp!=act
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 1700: uvm_test_top.env.sb [MISMATCH] exp!=act
UVM_ERROR ../tb/env/alu_cov.sv(77) @ 2000: uvm_test_top.env.cov [NO_HIT] bin never hit
--- UVM Report Summary ---
UVM_ERROR :    9

awk '/^--- UVM Report Summary ---/ { s = 1 } s { next } /^UVM_ERROR (\.|\/|@)/ { id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); loc = $2; sub(/\(/, ":", loc); sub(/\)/, "", loc); n[id SUBSEP loc]++; recs++ } END { c = 0; for (k in n) c++; printf "%d record(s) -> %d row(s):\n", recs, c; for (k in n) { split(k, p, SUBSEP); printf "  %-10s %-26s %d\n", p[1], p[2], n[k] } }' logs/alu.log | sort -k3 -rn

  MISMATCH   ../tb/env/alu_sb.sv:145    8
  NO_HIT     ../tb/env/alu_cov.sv:77    1
9 record(s) -> 2 row(s):

Eight records in one row, and the NO_HIT is now a line of two instead of a line of nine. That is the whole point of the chapter in one output — a reader scanning the ungrouped log would very likely stop after the eighth identical MISMATCH.

Note the sort -k3 -rn puts the summary line last because it has no third column; in a real script the header would be printed from END after a close(), which Chapter 7.7 section 4 covers.

8. Both Granularities at Once

Since the choice is a judgement, produce the coarser grouping and report the finer one as a column:

awk '/^--- UVM Report Summary ---/ { s = 1 } s { next } /^UVM_ERROR (\.|\/|@)/ { id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); h = ""; for (i = 1; i <= NF; i++) if ($i == "@") { h = $(i + 2); break } chk[id SUBSEP $2]++; inst[id SUBSEP $2 SUBSEP h] = 1 } END { for (k in chk) { split(k, p, SUBSEP); ni = 0; for (j in inst) { split(j, q, SUBSEP); if (q[1] == p[1] && q[2] == p[2]) ni++ } printf "%-10s %-26s %d record(s) across %d instance(s)%s\n", p[1], p[2], chk[k], ni, (ni > 1 ? "  <- same check, several instances" : "") } }' triage.log | sort -k3,3rn -k1,1 -k2,2

MISMATCH   ../tb/env/alu_sb.sv(145)   3 record(s) across 2 instance(s)  <- same check, several instances
MISMATCH   ../tb/env/alu_sb.sv(201)   1 record(s) across 1 instance(s)
MISMATCH   ../tb/env/fifo_sb.sv(88)   1 record(s) across 1 instance(s)
NO_HIT     ../tb/env/alu_cov.sv(77)   1 record(s) across 1 instance(s)

Four rows — the section 5 granularity — with the instance count as a column rather than as a split. The flagged row says the same check fired in two places, which is the information section 6's finer grouping provided, without inventing a fifth row.

That is usually the right answer: group at the coarser level and report the finer level as an attribute. A reader deciding whether sb0 and sb1 are one bug has the fact in front of them, and the row count still reflects the number of checks that failed.

The inst[] array is a set keyed on the full triple, and the inner loop counts the members whose first two parts match the current check — which is the only way to ask "how many instances does this group cover?" when the array is flat. Chapter 6.1 section 11 noted that limitation of SUBSEP keys, and this is it in practice: there is no way to query part of a composite key without walking everything.

9. Which Row to Look At First

Chapter 6.2 section 6 established that the most frequent row is rarely the most important. Applied here, with severity as the primary key:

awk '/^--- UVM Report Summary ---/ { s = 1 } s { next } /^UVM_(ERROR|FATAL) (\.|\/|@)/ { id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); rank = ($1 == "UVM_FATAL" ? 1 : 2); n[rank SUBSEP $1 SUBSEP id SUBSEP $2]++ } END { for (k in n) { split(k, p, SUBSEP); printf "%s|%-10s %-10s %-26s %d\n", p[1], p[2], p[3], p[4], n[k] } }' sim.log | sort -t'|' -k1,1n

UVM_ERROR ../tb/env/alu_sb.sv(145) @ 20500: uvm_test_top.env.sb [MISMATCH] exp=32'hdead act=32'hbeef
UVM_ERROR ../tb/env/alu_sb.sv(145) @ 21000: uvm_test_top.env.sb [MISMATCH] exp=32'hcafe act=32'h0000
UVM_FATAL ../tb/env/alu_env.sv(300) @ 30500: uvm_test_top.env [DEADLOCK] no response for 500ns
--- UVM Report Summary ---
UVM_ERROR :    2
UVM_FATAL :    1

1|UVM_FATAL  DEADLOCK   ../tb/env/alu_env.sv(300)  1
2|UVM_ERROR  MISMATCH   ../tb/env/alu_sb.sv(145)   2

The fatal is first despite having fewer records. A UVM_FATAL stopped the simulation, so everything after it is missing — which makes it both the most severe finding and the one that invalidates the rest of the log.

The leading rank column exists only to give sort a key, as in Chapter 7.7 section 6, and would be stripped in a real report.

10. Common Mistakes

Grouping by id alone. A generic id like MISMATCH merges different blocks — section 4.

Adding the hierarchy by default. It splits one design bug across instances — section 6.

Not stating the grouping key in the output. A row then cannot be interpreted.

Omitting the record count. A row from one record and one from eight hundred look identical.

Ranking by record count. The loudest failure is rarely the most important — Chapter 6.2 §6.

Grouping on the message text. It contains the data values, so every record is unique and nothing groups.

Using $6 for the id. The reporter record shifts the fields — Chapter 9.2 §3.

Counting the report summary as records. Chapter 9.3 §3.

Treating file(line) as the bug's location. It is where the check lives — Chapter 9.2 §7.

11. Exercises

Exercise 1 — Count the groups three ways

awk '/^--- UVM Report Summary ---/ { s = 1 } s { next } /^UVM_ERROR (\.|\/|@)/ { id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); h = ""; for (i = 1; i <= NF; i++) if ($i == "@") { h = $(i + 2); break } a[id]++; b[id SUBSEP $2]++; c[id SUBSEP $2 SUBSEP h]++; recs++ } END { na = 0; nb = 0; nc = 0; for (k in a) na++; for (k in b) nb++; for (k in c) nc++; printf "%d record(s): id=%d, (id,loc)=%d, (id,loc,hier)=%d\n", recs, na, nb, nc }' triage.log

6 record(s): id=2, (id,loc)=4, (id,loc,hier)=5

Exercise 2 — The recommended grouping

awk '/^--- UVM Report Summary ---/ { s = 1 } s { next } /^UVM_ERROR (\.|\/|@)/ { id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); n[id SUBSEP $2]++ } END { for (k in n) { split(k, p, SUBSEP); printf "%-12s %-28s %d\n", p[1], p[2], n[k] } }' triage.log | sort -k3,3rn -k1,1 -k2,2

MISMATCH     ../tb/env/alu_sb.sv(145)     3
MISMATCH     ../tb/env/alu_sb.sv(201)     1
MISMATCH     ../tb/env/fifo_sb.sv(88)     1
NO_HIT       ../tb/env/alu_cov.sv(77)     1

Exercise 3 — Debug an over-merged table

A triage table reports two problems and three engineers find three different bugs.

awk '/^--- UVM Report Summary ---/ { s = 1 } s { next } /^UVM_ERROR (\.|\/|@)/ { id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); n[id]++ } END { for (k in n) printf "%-12s %d\n", k, n[k] | "sort -k2 -rn" }' triage.log

MISMATCH     5
NO_HIT       1

The five MISMATCH records come from three different checks in two different files. Grouping by the id alone merged them because MISMATCH is a generic name every scoreboard uses.

awk '/^--- UVM Report Summary ---/ { s = 1 } s { next } /^UVM_ERROR (\.|\/|@)/ { id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); n[id SUBSEP $2]++ } END { c = 0; for (k in n) c++; printf "%d distinct check(s)\n", c }' triage.log

4 distinct check(s)

Exercise 4 — Debug an over-split table

The opposite failure: a table reports five problems where there are four checks.

awk '/^--- UVM Report Summary ---/ { s = 1 } s { next } /^UVM_ERROR (\.|\/|@)/ { id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); h = ""; for (i = 1; i <= NF; i++) if ($i == "@") { h = $(i + 2); break } n[id SUBSEP $2 SUBSEP h]++ } END { for (k in n) { split(k, p, SUBSEP); printf "%-10s %-26s %-24s %d\n", p[1], p[2], p[3], n[k] } }' triage.log | sort -k4,4rn -k1,1 -k2,2 -k3,3

MISMATCH   ../tb/env/alu_sb.sv(145)   uvm_test_top.env.sb0     2
MISMATCH   ../tb/env/alu_sb.sv(145)   uvm_test_top.env.sb1     1
MISMATCH   ../tb/env/alu_sb.sv(201)   uvm_test_top.env.sb0     1
MISMATCH   ../tb/env/fifo_sb.sv(88)   uvm_test_top.env.fsb     1
NO_HIT     ../tb/env/alu_cov.sv(77)   uvm_test_top.env.cov     1

Rows one and two are the same check at alu_sb.sv(145), reported by two instances of the scoreboard — sb0 and sb1. On identical channels that is one RTL bug, so this table has one row too many.

Section 8's form keeps four rows and reports the instance count as a column instead.

Exercise 5 — Collapse a dominated log

awk '/^--- UVM Report Summary ---/ { s = 1 } s { next } /^UVM_ERROR (\.|\/|@)/ { id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id); loc = $2; sub(/\(/, ":", loc); sub(/\)/, "", loc); n[id SUBSEP loc]++; recs++ } END { c = 0; for (k in n) c++; printf "collapsed %d record(s) into %d row(s)\n", recs, c }' logs/alu.log

collapsed 9 record(s) into 2 row(s)

Exercise 6 — Engineering application

Produce a triage table a lead can act on: severity-ordered rows at the check granularity, each with its record count and instance count, the grouping key stated, and a non-zero status if anything failed.

/^--- UVM Report Summary ---/ { s = 1 } s { next }
/^UVM_(ERROR|FATAL) (\.|\/|@)/ {
    id = $0; sub(/^[^[]*\[/, "", id); sub(/\].*$/, "", id)
    h = ""; for (i = 1; i <= NF; i++) if ($i == "@") { h = $(i + 2); break }
    rank = ($1 == "UVM_FATAL" ? 1 : 2)
    key = rank SUBSEP $1 SUBSEP id SUBSEP $2
    cnt[key]++; inst[key SUBSEP h] = 1; recs++
}
END {
    if (!recs) { print "no error or fatal records"; exit 0 }
    nchk = 0
    for (k in cnt) {
        nchk++
        split(k, p, SUBSEP)
        ni = 0
        for (j in inst) { split(j, q, SUBSEP); if (q[1] == p[1] && q[2] == p[2] && q[3] == p[3] && q[4] == p[4]) ni++ }
        printf "%s|%-10s %-11s %-27s %2d rec  %d inst\n", p[1], p[2], p[3], p[4], cnt[k], ni
    }
    printf "(%d record(s) in %d check(s); grouped by severity, id and file(line))\n", recs, nchk > "/dev/stderr"
    exit 1
}

awk -f triage6.awk triage.log 2>/dev/null | sort -t'|' -k1,1n -k2,2 -k4,4

2|UVM_ERROR  MISMATCH    ../tb/env/alu_sb.sv(145)     3 rec  2 inst
2|UVM_ERROR  MISMATCH    ../tb/env/alu_sb.sv(201)     1 rec  1 inst
2|UVM_ERROR  MISMATCH    ../tb/env/fifo_sb.sv(88)     1 rec  1 inst
2|UVM_ERROR  NO_HIT      ../tb/env/alu_cov.sv(77)     1 rec  1 inst

awk -f triage6.awk triage.log 2>&1 >/dev/null

(6 record(s) in 4 check(s); grouped by severity, id and file(line))

awk -f triage6.awk triage.log >/dev/null 2>&1
echo "status: $?"

status: 1

Four actionable rows, the instance count beside each, the grouping key stated, and a status a flow can branch on.

Three deliberate choices. The trailer goes to stderr so that it cannot be sorted into the middle of the rows — the rows pass through sort and the trailer does not, which Chapter 6.2 section 10's buffering problem makes necessary. The sort keys are fully specified (-k1,1n -k2,2 -k4,4), so the three rows that tie at one record come out in a defined order and the table is diffable. And the row with 2 inst is the one needing a judgement — the same check failing in two scoreboard instances is probably one design bug, and the table reports the fact without deciding.

The engineering caveat: 4 check(s) counts checks that fired, not defects. One RTL bug can trip several checks — a wrong value fails a data comparison and a protocol assertion — so the number is an upper bound on independent problems.

Exercise 7 — Challenge: show that the message text cannot be a grouping key

A tempting alternative is to group on the whole message. Demonstrate why it fails.

awk '/^--- UVM Report Summary ---/ { s = 1 } s { next } /^UVM_ERROR (\.|\/|@)/ { msg = $0; sub(/^[^]]*\] */, "", msg); n[msg]++ } END { c = 0; for (k in n) c++; printf "%d distinct message text(s) from %d record(s)\n", c, NR - 2 }' triage.log

6 distinct message text(s) from 7 record(s)

Six distinct texts from six error records — nothing grouped at all, because each message contains its own data values (exp=1 act=2, exp=3 act=4, and so on).

On the dominated log it is worse, because there the messages happen to be identical:

awk '/^--- UVM Report Summary ---/ { s = 1 } s { next } /^UVM_ERROR (\.|\/|@)/ { msg = $0; sub(/^[^]]*\] */, "", msg); n[msg]++; recs++ } END { c = 0; for (k in n) c++; printf "%d distinct message text(s) from %d record(s)\n", c, recs }' logs/alu.log

2 distinct message text(s) from 9 record(s)

Two groups — the right answer, by luck. The fixture's messages are exp!=act with no values, so identical text happened to mean identical failure.

That is the trap: grouping on message text works on a log whose messages omit the data and fails on one that includes it — and including the data is the normal, more useful thing for a scoreboard to do. A script tested on the first kind of log breaks silently on the second, producing one row per record.

The general rule, which Chapter 6.3 section 8 stated for keys in general: group on a field that identifies the category, not one that carries the instance. The id and the source location are chosen by the engineer and are stable; the message text is generated per occurrence and is not.

12. Summary

• Grouping makes the second failure visible — the highest-value transformation in this module.
• The same six records give 2, 4 or 5 distinct failures depending on the key.
• By id alone is too coarse: a generic id like MISMATCH merges different blocks and hides work.
• By (id, file(line)) is the default: one row per check that fired, stable across runs.
• Adding the hierarchy is usually too fine: it splits one design bug across instances and invents work.
• Both wrong directions produce a plausible table, with no error of any kind.
• Group at the coarser level and report the finer one as a column — section 8.
• State the grouping key in the output, or a row cannot be interpreted.
• Order by severity, not by record count. A fatal with one record outranks a mismatch with eight hundred.
• Never group on the message text. It carries the data values, so it works only on logs whose messages omit them.
• A count of checks is an upper bound on defects, since one bug can trip several checks.

One log is now a table. Chapter 9.5 — Per-Test Summaries scales it to a directory: one row per test across a whole regression, including the distinction between a test that failed and a test that produced no log at all.