Perl Part II · Module 21 · Simulation and UVM Log Analysis

Classifying Failures by Signature

A regression with a hundred failures almost never has a hundred problems, but the list it hands you looks like it does — because each failure's message carries an address, a data value and a cycle number, which makes every one of them textually unique. The grouping information is in the shape of the message rather than its contents. This chapter normalises the values out, measures the collapse, and then does the part that is usually skipped: checks in both directions that the grouping is neither too coarse nor too fine.

Advanced12 min readPerlPart IIUVM Logs

Part II · Module 21 · Chapter 21.9 · Classifying Failures by Signature

1. Twenty-Two Failures, Twenty-Two Texts

Azvya Education Pvt. Ltd.VLSI Mentor

signature.pl

use strict;
use warnings;

my @FILES = sort glob("logs/batch/*.log");
my $COUNT_LINE = qr/^(?:#\s*)?UVM_\w+\s*:\s*\d+\s*$/;
my $MSG = qr/^(?:\#\s*)?(UVM_\w+)\s+(?:(\S+?)\((\d+)\)\s+)?\@\s*(\d+)\s*:\s*(\S*)\s*\[(\w+)\]\s*(.*)$/;

# ---- collect each failing test's FIRST error (21.5)
my @fail;
for my $f (@FILES) {
    open my $fh, "<", $f or die "$f: $!";
    my $first;
    while (my $l = <$fh>) {
        next if $l =~ $COUNT_LINE;
        next unless $l =~ $MSG;
        next unless $1 eq "UVM_ERROR";
        $first = { id => $6, text => $7, time => $4 };
        last;
    }
    close $fh;
    next unless $first;
    my ($short) = $f =~ m{([^/]+)\.log$};
    push @fail, { test => $short, %$first };
}

printf "%d failing tests\n", scalar @fail;
my %raw;
$raw{ $_->{text} }++ for @fail;
printf "distinct first-error TEXTS, unnormalised: %d\n", scalar keys %raw;
printf "  so grouping on the raw text gives %d 'causes' for %d failures\n\n",
    scalar keys %raw, scalar @fail;

# ---- normalise: replace every VALUE with a placeholder, keep the SHAPE.
# Order matters: the widest patterns first, or a later rule eats a digit that
# an earlier rule needed to recognise its own shape.
sub signature {
    my ($id, $text) = @_;
    my $s = $text;
    $s =~ s/0x[0-9a-fA-F]+/0xADDR/g;        # hex literals, with or without 0x
    $s =~ s/\b[0-9a-fA-F]{4,}\b/HEX/g;      # bare hex-looking runs
    $s =~ s/\b\d+\b/N/g;                    # any remaining number
    $s =~ s/\[N\]/[IDX]/g;                  # a bit index reads better named
    $s =~ s/\s+/ /g;
    $s =~ s/^ | $//g;
    return "$id: $s";
}

my (%sig, %sig_tests);
for my $t (@fail) {
    my $s = signature($t->{id}, $t->{text});
    $sig{$s}++;
    push @{ $sig_tests{$s} }, $t->{test};
}

printf "distinct SIGNATURES after normalising: %d\n\n", scalar keys %sig;
printf "%-6s %-54s %s\n", "COUNT", "SIGNATURE", "tests";
for my $s (sort { $sig{$b} <=> $sig{$a} || $a cmp $b } keys %sig) {
    my @t = @{ $sig_tests{$s} };
    printf "%-6d %-54s %s%s\n", $sig{$s}, $s,
        join(",", @t[0 .. ($#t > 2 ? 2 : $#t)]),
        ($#t > 2 ? " +" . ($#t - 2) . " more" : "");
}

printf "\n%d failures collapsed to %d causes (%.1fx)\n",
    scalar @fail, scalar keys %sig, (scalar @fail) / (scalar keys %sig);

# ---- is the normalisation too aggressive? check it does not merge DIFFERENT ids
print "\ndoes any signature mix two different IDs?\n";
my $mixed = 0;
for my $s (keys %sig) {
    my ($id) = $s =~ /^(\w+):/;
    $mixed++ unless defined $id;
}
printf "  signatures without a single owning ID: %d\n", $mixed;
print "  the ID is part of the signature by construction, so two subsystems\n";
print "  reporting similar text never merge.\n";

# ---- and is it too WEAK? show one signature's raw members
print "\none signature's raw texts, to check they really are one cause:\n";
my ($biggest) = sort { $sig{$b} <=> $sig{$a} } keys %sig;
my @members = grep { signature($_->{id}, $_->{text}) eq $biggest } @fail;
printf "  %s\n", $_->{text} for @members[0 .. ($#members > 3 ? 3 : $#members)];
printf "  (%d members; the values differ, the shape does not)\n", scalar @members;

Azvya Education Pvt. Ltd.VLSI Mentor

Captured output

22 failing tests
distinct first-error TEXTS, unnormalised: 22
  so grouping on the raw text gives 22 'causes' for 22 failures

distinct SIGNATURES after normalising: 4

COUNT  SIGNATURE                                              tests
10     SCBD: data mismatch exp=HEX got=N addr=0xADDR          test_04,test_08,test_12 +7 more
6      PROTO: req asserted while busy at cycle N              test_05,test_10,test_15 +3 more
3      AXI: burst crossed a 4KB boundary at addr=0xADDR       test_09,test_18,test_27
3      TIMING: setup violation on data[IDX] at N ps           test_13,test_26,test_39

22 failures collapsed to 4 causes (5.5x)

does any signature mix two different IDs?
  signatures without a single owning ID: 0
  the ID is part of the signature by construction, so two subsystems
  reporting similar text never merge.

one signature's raw texts, to check they really are one cause:
  data mismatch exp=0000001c got=00 addr=0x00000040
  data mismatch exp=00000038 got=00 addr=0x00000080
  data mismatch exp=00000054 got=00 addr=0x000000c0
  data mismatch exp=00000070 got=00 addr=0x00000100
  (10 members; the values differ, the shape does not)

The first number is the problem:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

22 failing tests
distinct first-error TEXTS, unnormalised: 22

Grouping on the raw text produces twenty-two groups of one. Every failure is textually unique, because every message carries values — exp=0000001c, addr=0x00000040, at cycle 15 — and no two runs produce the same ones.

That is not a quirk of this fixture. It is what a real error message looks like, and it is the reason a failure list from a regression arrives as a hundred apparently unrelated lines.

2. Normalise The Values, Keep The Shape

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

distinct SIGNATURES after normalising: 4
22 failures collapsed to 4 causes (5.5x)

Three substitutions do it, and the order is load-bearing:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$s =~ s/0x[0-9a-fA-F]+/0xADDR/g;      # hex literals with a 0x prefix
$s =~ s/\b[0-9a-fA-F]{4,}\b/HEX/g;    # bare hex-looking runs
$s =~ s/\b\d+\b/N/g;                  # any remaining number

Widest first. The 0x rule must run before the bare-hex rule, or the bare rule eats the digits after 0x and leaves 0xHEX sometimes and 0xADDR others. The plain-number rule runs last because it would otherwise consume the digit runs the hex rules need to recognise their own shape. Chapter 20.5's ^CKLN before ^CK is the same ordering principle: when one pattern can match a subset of another's input, the specific one goes first.

The result is four signatures that read as causes rather than as incidents:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

10    SCBD: data mismatch exp=HEX got=N addr=0xADDR
6     PROTO: req asserted while busy at cycle N
3     AXI: burst crossed a 4KB boundary at addr=0xADDR
3     TIMING: setup violation on data[IDX] at N ps

Four lines instead of twenty-two, with the test names attached so each one is a starting point.

One imperfection in that output, left visible rather than tidied away.

The SCBD signature reads exp=HEX got=N. Both are the same kind of field — a data value — and they normalised differently: exp=0000001c is eight hex digits and the {4,} rule caught it, while got=00 is two digits and fell through to the plain-number rule.

Normalisation is width-sensitive, and a narrow value and a wide value of the same field can land in different buckets. It did no harm here, because both forms are consistent across all ten members, so the signature is still stable. It would do harm if the width varied between runs of the same cause — then one cause would produce two signatures.

The fix, when it matters, is to normalise by field name rather than by value shape:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$s =~ s/\b(exp|got|addr|data)=\S+/$1=V/g;    # name-driven, width-blind

That is more work and needs knowledge of the message formats. Worth doing for the handful of IDs that dominate your regression, and not worth doing for the long tail.

3. Verified In Both Directions

A collapse ratio on its own proves nothing. s/.*/SAME/ would collapse 22 failures to 1 signature, a 22x improvement by that measure and completely useless. So the script checks both failure modes.

Too aggressive? Would two different causes merge into one signature?

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

signatures without a single owning ID: 0

They cannot, because the ID is part of the signature. Section 4 is about why that matters more than it looks.

Too weak? Are the members of a signature really one cause?

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

one signature's raw texts, to check they really are one cause:
  data mismatch exp=0000001c got=00 addr=0x00000040
  data mismatch exp=00000038 got=00 addr=0x00000080
  data mismatch exp=00000054 got=00 addr=0x000000c0
  data mismatch exp=00000070 got=00 addr=0x00000100
  (10 members; the values differ, the shape does not)

Printing the raw members of a group is the cheapest possible audit and it is the one that catches over-merging. If you cannot look at four members of a group and agree they are one problem, the normalisation is wrong — and you will only find out by looking.

This is Chapter 20.7's lesson in a different setting: the output of a classifier has to be audited line by line the first time, because its entire purpose is to be trusted later without being read.

4. The ID Belongs In The Signature

"$id: $s" rather than just $s, and it is not cosmetic.

Two subsystems can produce messages that normalise to the same shape. data mismatch exp=HEX got=N addr=0xADDR from a scoreboard and the same sentence from a memory model are different problems with different owners, and without the ID they merge into one signature with twice the count — which looks like one bigger problem and sends one engineer to investigate two.

The ID is also what makes the signature routable. SCBD and AXI map to people, and a triage list grouped by signature is already grouped by who should look at it.

And it is cheap insurance against the normalisation being slightly wrong: if a substitution is too aggressive, the ID limits the damage to within one subsystem rather than across the whole regression.

What a signature is not for.

It is not a fingerprint of a bug. Two tests with the same signature can have different root causes that happen to surface the same way — a scoreboard mismatch is a symptom, and several bugs produce it. The signature groups symptoms, which is the right first cut and not a conclusion.

It is not stable across releases. Change a message's wording and its signature changes, so "this signature appeared last week too" is only true while nobody edits the testbench. If you want to track a cause over time, track the ID plus a short human label you assign once, with the signature as the evidence.

It is a pattern over text (Chapter 18.11, again). It knows nothing about the design. A signature count of 10 means ten tests printed similarly-shaped messages, which is a strong hint and not a measurement of anything in the hardware.

Said plainly: a signature turns an unreadable list into a readable one. That is a large and real improvement, and it is the whole claim.

5. What To Remember

A regression with a hundred failures rarely has a hundred problems — but its failure list looks like it does.
Measured: 22 failing tests, 22 distinct first-error texts. Grouping on raw text gives 22 groups of one.
Every message carries values — an address, a data value, a cycle — so no two runs match textually.
The grouping information is the SHAPE, and normalising the values out is what exposes it.
Measured: 22 failures collapsed to 4 signatures, 5.5x.
Order the substitutions widest-first: 0x… before bare hex before plain numbers, or an earlier rule eats what a later one needs (20.5's principle).
Normalisation is width-sensitive — exp=0000001c became HEX and got=00 became N. Harmless when consistent, harmful when the width varies between runs of one cause.
Normalise by field NAME when it matters: s/\b(exp|got|addr)=\S+/$1=V/g. Worth it for the dominant IDs, not for the tail.
A collapse ratio proves nothing on its own — s/.*/SAME/ gives 22 to 1.
Check too-aggressive: no signature should mix two IDs.
Check too-weak: print a group's raw members and agree they are one problem.
If you cannot look at four members and call them one cause, the normalisation is wrong — and only looking finds it (20.7).
Put the ID in the signature. Two subsystems can produce the same shape, and merging them sends one engineer to investigate two problems.
The ID makes the signature routable — a list grouped by signature is grouped by owner.
A signature groups SYMPTOMS, not root causes — several bugs produce one scoreboard mismatch.
It is not stable across releases: edit a message and its signature changes. Track an ID plus a human label for continuity.
It is a pattern over text (18.11) and knows nothing about the design.

Chapter 21.10 makes the signature actionable. Having grouped the failures, each group still needs one concrete place to start — a short, bounded excerpt around the first error, with its line number and simulation time, so that reading it is faster than opening the log.

Where this fits

Part of the Perl curriculum.