Linux & Bash · Part II · Module 33 · Debugging Failed Jobs and Broken Flows

Classifying Failures Across a Regression

Thirty-six failures are not thirty-six problems, and the useful question is how many problems they are. This chapter groups them by signature, measures a message-only grouping merging four distinct bugs into two groups of ten, and finds the narrow window between normalising so little that nothing groups and so much that different failures look identical.

Advanced25 min readLinuxPart IIDebuggingRegressionClassification

Part II · Module 33 · Chapter 33.7 · Classifying Failures Across a Regression

Chapter 33.6 worked on one run. This chapter is the opposite scale, and it starts from a number nobody can act on:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

 50 runs: 14 PASS  11 FAIL  0 TIMEOUT  5 CRASH
          5 NO_VERDICT  10 INCONSISTENT  5 VACUOUS

Thirty-six runs are not a pass. The question that decides the next week is not how many failed; it is how many problems that is, because the answer determines whether anybody can finish by Friday.

1. Thirty-Six Runs, Eight Problems

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# classify REGRDIR -- group a regression's non-passing runs by SIGNATURE, so
# one fix can be matched to every run it explains.
#
# A signature is (verdict class, normalised first diagnostic). The class is in
# it because two different bugs can print the identical message: a design
# property violation and a testbench that prints PASSED anyway both emit
# "ERROR t=35 gnt=10 expected 01".
set -u
R=${1:-}
[ -n "$R" ] || { echo "classify: need a regression directory" >&2; exit 2; }
res=$R/results.tsv
[ -r "$res" ] || { echo "classify: no results.tsv in '$R'" >&2; exit 2; }

# Normalise by POSITION, not by alphabet. [0-9a-f] matches ordinary letters and
# turns "dout expected" into "Nout Nxpected" -- measured in 29.4.
# And anchor each rule: an unanchored s/t=[0-9]*/t=T/ also rewrites gnt=10,
# merging three genuinely different grant values into one signature (measured).
norm() {
  sed -e 's/\([^a-zA-Z]\)t=[0-9][0-9]*/\1t=T/g' \
      -e 's/cycle [0-9][0-9]*/cycle N/g' \
      -e 's/(draw=[0-9][0-9]*)/(draw=N)/g' \
      -e 's/:[0-9][0-9]*:/:L:/g' \
      -e 's/[[:space:]][[:space:]]*/ /g' \
      -e 's/^ //' -e 's/ $//'
}

tmp=$(mktemp) || exit 2
trap 'rm -f "$tmp"' EXIT INT TERM
awk -F'\t' 'NR>1 && $4!="PASS" {print $1"\t"$2"\t"$3"\t"$4}' "$res" | while IFS="$(printf '\t')" read -r spec t s cls; do
  log=$R/runs/${t}_s${s}/run.log
  d=$(/usr/bin/grep -m1 -E 'ERROR|FATAL' "$log" 2>/dev/null | norm)
  [ -n "$d" ] || d="(no diagnostic line)"
  printf '%s\t%s\t%s\n' "$cls" "$d" "$spec"
done > "$tmp"

total=$(grep -c . "$tmp")
groups=$(cut -f1,2 "$tmp" | sort -u | grep -c .)
printf 'classify: %d non-passing run(s) in %d signature group(s)\n\n' "$total" "$groups"
cut -f1,2 "$tmp" | sort | uniq -c | sort -k1,1rn -k2,2 | while read -r n rest; do
  cls=${rest%%	*}; msg=${rest#*	}
  printf '  %3d  %-13s %s\n' "$n" "$cls" "$msg"
  # name a representative and the full span, because a count alone is not actionable
  members=$(awk -F'\t' -v c="$cls" -v m="$msg" '$1==c && $2==m {printf "%s ", $3}' "$tmp")
  printf '       %s\n' "$(printf '%s' "$members" | cut -c1-70)"
done

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

classify: 36 non-passing run(s) in 8 signature group(s)

    5  CRASH         FATAL: tb/tb_arb.v:L: deliberate fatal
       crash:1 crash:2 crash:3 crash:4 crash:5
    5  FAIL          TB ERROR t=T gnt=10 expected 01
       prio0:1 prio0:2 prio0:3 prio0:4 prio0:5
    5  FAIL          TB ERROR t=T late check failed at cycle N
       latefail:1 latefail:2 latefail:3 latefail:4 latefail:5
    5  INCONSISTENT  FATAL: tb/tb_arb.v:L: failed while closing out
       latecrash:1 latecrash:2 latecrash:3 latecrash:4 latecrash:5
    5  INCONSISTENT  TB ERROR t=T gnt=10 expected 01
       inconsistent:1 inconsistent:2 inconsistent:3 inconsistent:4 inconsiste
    5  NO_VERDICT    (no diagnostic line)
       silent:1 silent:2 silent:3 silent:4 silent:5
    5  VACUOUS       (no diagnostic line)
       vacuous:1 vacuous:2 vacuous:3 vacuous:4 vacuous:5
    1  FAIL          TB ERROR t=T seed-dependent check failed (draw=N)
       flaky:2

Thirty-six down to eight, and the eight are actionable in a way the thirty-six are not: seven of them are "fix one thing, clear five runs", and one of them is a single seed-dependent failure that needs Chapter 31.3's treatment.

The member list matters as much as the count. A group that says 5 tells you how much a fix is worth; a group that names prio0:1 … prio0:5 tells you what to re-run to prove it, which is Chapter 32.6's selection list arriving ready-made.

2. The Message Alone Is Not a Signature

Group on the diagnostic and nothing else — which is what every first version of this does:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

    10 (no diagnostic line)
    10 TB ERROR t=T gnt=10 expected 01
     5 FATAL: tb/tb_arb.v:L: deliberate fatal
     5 FATAL: tb/tb_arb.v:L: failed while closing out
     5 TB ERROR t=T late check failed at cycle N
     1 TB ERROR t=T seed-dependent check failed (draw=N)

Six groups instead of eight, and the two biggest are each two different bugs.

Two groups of ten, four distinct causes

10 TB ERROR t=T gnt=10 expected 01 is prio0 and inconsistent together. They print the identical line because they check the identical property at the identical moment. They are not the same problem:

prio0 is a design-property failure. The arbiter grants agent 1 first and the test says so. Fix the RTL.
inconsistent is a testbench defect. It counts the same error and then prints TEST PASSED ... errors=0 anyway. Fix the testbench — and note that fixing the RTL would make this one pass while leaving the defect in place, so the next real failure it sees will also be reported as a pass.

10 (no diagnostic line) is silent and vacuous:

silent exits 0 having never stated an outcome.
vacuous states TEST PASSED after making zero checks.

Both are testbench problems and they are different ones: one forgot to report, the other reported without testing.

So the message-only grouping says "one fix explains ten runs" twice, and is wrong both times. The cost is specific and familiar: somebody fixes the arbiter, re-runs, sees five of the ten clear, and now has a group of five that "did not get fixed" — which is the start of a hunt for a second RTL bug that does not exist.

The class is what separates them, because the class is derived from evidence the message does not carry — Chapter 32.3's seven-way reading of the exit status, the verdict line, the error count and the check count together. A signature has to include everything that distinguishes the causes, and the message is only one of those things.

Figure 1 — thirty-six non-passing runs, grouped two ways. Keying on the diagnostic alone gives six groups, two of which are ten runs each and four distinct bugs. Adding the verdict class gives eight groups that each correspond to one problem.

3. Normalising: the Window Is Narrow in Both Directions

Raw messages do not group at all, because every one of them carries a time:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  three runs of the SAME bug at different times -> 3 group(s) unnormalised
  with the time normalised                       -> 1 group(s)

So some normalisation is mandatory. And my first rule normalised too much:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

sed -e 's/t=[0-9][0-9]*/t=T/g'

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  three genuinely different failures:
    TB  ERROR t=35 gnt=10 expected 01
    TB  ERROR t=35 gnt=11 expected 01
    TB  ERROR t=35 gnt=00 expected 01
  normalised with  s/t=[0-9]*/t=T/g :
    TB  ERROR t=T gnt=T expected 01
    -> 1 distinct signature(s) from 3 distinct failures

gnt=10 became gnt=T, because t= occurs inside gnt=. Three different wrong grant values — three potentially different bugs — collapsed into one signature. Anchoring the rule so it cannot start mid-word:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

sed -e 's/\([^a-zA-Z]\)t=[0-9][0-9]*/\1t=T/g'

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

    TB  ERROR t=T gnt=00 expected 01
    TB  ERROR t=T gnt=10 expected 01
    TB  ERROR t=T gnt=11 expected 01
    -> 3 distinct signature(s)

And the real regression still groups to eight, now with the grant value preserved:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

    5  FAIL          TB ERROR t=T gnt=10 expected 01

	what it does	the cost
too little	every timestamp makes its own group	36 groups for 8 problems — no grouping at all
too much	different values look identical	3 bugs in 1 group; a fix credited with runs it did not touch

The rule that keeps you inside the window is the one Chapter 29.4 arrived at from the counting side: normalise by position, not by alphabet. Rewrite what follows a known marker — t=, cycle , draw=, a :line: — and never rewrite a character class. That chapter measured s/[0-9a-f]*/N/g turning dout=a4 expected a5 into Nout=N NxpNtN N, which still grouped, which is what makes it dangerous.

Report a signature and a representative, never a signature alone

A normalised signature is lossy by construction, so a group's output needs an unnormalised example:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

    5  FAIL          TB ERROR t=T gnt=10 expected 01
       prio0:1 prio0:2 prio0:3 prio0:4 prio0:5

The member list is that example — naming prio0:1 lets somebody open one real log and see t=35 rather than t=T. Without it the reader has a pattern and no way back to the evidence, and the first thing they will do is grep the regression for the pattern by hand, which does not match because the pattern is normalised.

Three further things worth printing that this version does not:

The group's span across tests. Five runs of one test is a different problem from five runs of five tests: the first is one bug, the second may be something environmental. cut -f1 -d: over the member list answers it.

The group's span across seeds, which separates a deterministic failure from a seed-dependent one — prio0 on all five seeds against flaky on one.

And whether the group is new. A group that appeared tonight is the one to look at, and that needs last night's classification to compare against — the same join Chapter 32.5 built, applied to signatures rather than to specs.

4. Choosing the Diagnostic Line

classify takes the first ERROR or FATAL line:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

d=$(/usr/bin/grep -m1 -E 'ERROR|FATAL' "$log" 2>/dev/null | norm)

Three decisions in that line.

First, not last. Chapter 29.2 measured why: a failure cascades, so the last error is usually a consequence and the first is usually the cause. A classifier keyed on the last line groups by symptom.

-m1 rather than head -1, so grep stops reading. On a regression of two thousand runs with hundred-megabyte logs that is the difference between a report and a lunch break.

And a fallback that is not empty. A run with no diagnostic at all gets the literal string (no diagnostic line) rather than an empty signature:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[ -n "$d" ] || d="(no diagnostic line)"

An empty signature would group with every other empty one and sort oddly, and — worse — an empty field in a tab-separated line shifts every later column, which is Chapter 32.4's measured hazard. The (no diagnostic line) group is also informative in its own right: ten runs reached it here, and splitting them by class is what revealed two separate testbench defects.

What this grouping cannot tell you

A signature group is a hypothesis, not a diagnosis. Five runs printing one message is evidence that one fix may clear five runs; it is not evidence that there is one bug. Two independent defects can produce the same message, which is exactly what Section 2 measured — and the class separated those two only because the class happened to differ. Two design bugs that both violate the same check at the same cycle would still be one group.

The group sizes are not severities. The largest group here is five, and so are six others. A single-run group can be the most important thing in the regression: flaky:2 is one run, and a seed-dependent failure in a block about to be signed off matters more than five runs of a test somebody disabled last week.

And the counts move when the matrix moves. Chapter 32.5 measured a pass count improving purely because a seed list changed. Group sizes have the same property, more strongly: a group of five across five seeds becomes a group of twenty when somebody adds fifteen seeds, with nothing about the design different. Compare signatures between nights, never sizes.

The honest statement of what this chapter produces is narrow and still worth a great deal: a list of distinct observed failure shapes, with the runs that exhibit each one. Turning that into a count of bugs requires reading one log per group, which is now eight logs instead of thirty-six.

5. The Workflow It Enables

The point of classification is the order it imposes on a morning:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# 1. how many problems, not how many failures
$ classify regr/latest

# 2. the classes that are not design questions, first
$ classify regr/latest | grep -E 'CRASH|TIMEOUT|NOSTATUS'

# 3. one log per group, not one per run
$ repro regr/latest/runs/prio0_s1          # 33.2
$ triage regr/latest/runs/prio0_s1         # 33.1

# 4. prove a fix against the group's own member list
$ rerun regr/latest FAIL                   # 32.6

Step 2 is the one most often skipped and it is where Chapter 32.5's environment-versus-design split pays off. Of the eight groups here, two are CRASH and INCONSISTENT-from-a-late-death — tool and environment questions that will waste a verification engineer's day if they are handed one.

Step 3 is where this module's earlier chapters attach: one representative per group, triaged to a layer before it is debugged. Eight representatives, eight triages, and Chapter 33.1 measured that two of its three layer answers need no log reading at all.

And step 4 closes the loop with a number: if a fix was supposed to clear a group of five and clears four, the group was two problems, and you have learned that from a rerun rather than from a week.

Practice

Run a regression with enough failures that the count is not actionable, and write down how many problems you think it is.
Group them by the first diagnostic line and count the groups.
Group them by class and message together and count again.
Find a group that split and explain what distinguishes the two causes.
Say what happens when somebody fixes one of the two and re-runs.
Group the raw messages with no normalisation and count.
Normalise the timestamps and count again.
Write an unanchored s/t=[0-9]*/t=T/ and apply it to three different values of a second field.
Anchor the rule and re-apply.
State the two failure directions of normalisation and what each costs.
Apply s/[0-9a-f]*/N/g to a message with hex values and read the result.
Print a representative with every group and explain why the pattern alone is not enough.
Add the group's span across tests and find a group that spans several.
Add the span across seeds and separate the deterministic from the seed-dependent.
Compare tonight's signatures against last night's and find the new group.
Take the first diagnostic rather than the last, then swap them and see how the grouping changes.
Use -m1 and explain what it saves on a large log.
Give a run with no diagnostic a literal fallback, then make it empty and watch the columns shift.
Find a single-run group that matters more than a five-run one.
Add fifteen seeds and watch a group size quadruple with nothing fixed.
Work the four-step morning on your own regression.

Check Your Understanding

What is the question that matters about thirty-six failures?
How many groups did thirty-six non-passing runs reduce to?
Why does the member list matter as much as the count?
How many groups does grouping on the message alone give?
Which two groups merge distinct causes, and what are the four causes?
Why would fixing the RTL leave the inconsistent defect in place?
What is the specific cost of the message-only grouping?
What does the class carry that the message does not?
What happens with no normalisation at all?
What did the unanchored t= rule do, and why?
What does the anchored version give?
State the cost of normalising too little, and of too much.
What is the rule that keeps you in the window?
What did s/[0-9a-f]*/N/g do in Chapter 29.4, and why was it dangerous?
Why must a group report a representative?
What happens if a reader greps the regression for a normalised pattern?
Name three further things worth printing per group.
Why take the first diagnostic line and not the last?
Why -m1 rather than head -1?
Why must the no-diagnostic fallback be a literal string?
Why is a signature group a hypothesis rather than a diagnosis?
Why are group sizes not severities?
What happens to group sizes when the matrix changes?
What should be compared between nights?
What exactly does this chapter produce, stated narrowly?
Give the four steps of the morning, and which is most often skipped.

Answers

Not how many runs failed, but how many problems that is — which decides whether the work can be finished.
Eight.
Because the count tells you what a fix is worth and the member list tells you what to re-run to prove it.
Six.
10 TB ERROR t=T gnt=10 expected 01 is prio0 (a design-property failure) plus inconsistent (a testbench that prints PASSED anyway); 10 (no diagnostic line) is silent (never stated an outcome) plus vacuous (reported a pass after zero checks).
Because the testbench's verdict is hardcoded. Fixing the RTL makes that run pass while the defect remains, so the next real failure it sees will also be reported as a pass.
Somebody fixes the arbiter, sees five of ten clear, and starts hunting a second RTL bug that does not exist.
Evidence the message does not carry — the exit status, the verdict line, the error count and the check count read together.
Nothing groups: every timestamp makes its own group, giving one group per run.
It rewrote gnt=10 to gnt=T, because t= occurs inside gnt=. Three different grant values collapsed to one signature.
Three distinct signatures, with the grant value preserved.
Too little: 36 groups for 8 problems, which is no grouping. Too much: different failures look identical, and a fix is credited with runs it did not touch.
Normalise by position, not by alphabet — rewrite what follows a known marker, never a character class.
Turned dout=a4 expected a5 into Nout=N NxpNtN N, and it still grouped, which is what made it dangerous.
Because a normalised signature is lossy, so the reader needs a way back to a real log.
It does not match, because the pattern is normalised and the logs are not.
The group's span across tests, its span across seeds, and whether the group is new compared with last night.
Because a failure cascades, so the last error is usually a consequence and the first is usually the cause. Keying on the last groups by symptom.
Because grep stops reading at the first match, which on hundred-megabyte logs is the difference between a report and a lunch break.
Because an empty signature groups with every other empty one and, in a tab-separated line, shifts every later column.
Because two independent defects can produce the same message — which was measured — so one message is evidence that one fix may clear several runs, not that there is one bug.
Because a single-run group can be the most important thing in the regression, and seven groups here are all the same size.
They move with the matrix: a group of five across five seeds becomes twenty when fifteen seeds are added, with nothing about the design different.
Signatures, never sizes.
A list of distinct observed failure shapes, with the runs that exhibit each one. Turning that into a count of bugs still needs one log read per group — eight instead of thirty-six.
Classify; look at the non-design classes first; one representative per group through reproduce and triage; then prove the fix against the group's member list. Step 2 is most often skipped.

6. Remember

The useful question is how many problems, not how many failures. Thirty-six non-passing runs were eight groups.
Print the member list, not just the count — it is the rerun selection, ready-made.
A signature is the class plus the message. Message alone gave six groups, and two of them were four different bugs.
prio0 and inconsistent print the identical line and need opposite fixes; fixing the RTL would hide the testbench defect rather than remove it.
No normalisation means no grouping; too much means different failures look identical. Three grant values collapsed to one signature because t= matched inside gnt=.
Normalise by position, not by alphabet, and anchor every rule so it cannot start mid-word.
Always report a representative. A normalised pattern cannot be grepped for in the logs it came from.
Take the first diagnostic, with -m1, and give a no-diagnostic run a literal fallback string.
A group is a hypothesis. Two independent defects can share a message; group sizes are not severities; and a one-run group can outrank a five-run one.
Group sizes move with the matrix. Compare signatures between nights, never sizes.
The morning is: classify, non-design classes first, one representative per group, then prove the fix against the member list.

7. What Is Next

Every technique so far has assumed you can run the thing again where you are sitting. Chapter 33.8 handles the case where that assumption fails — a failure that happens only on a compute node, under a scheduler, in an environment you were never in — which is where this module's reproduce-and-compare discipline has to work at a distance.

Where this fits

Part of the Linux & Bash curriculum.