Perl Part II · Module 21 · Simulation and UVM Log Analysis

The UVM Report Summary Block

UVM ends a run by printing how many messages it issued at each severity. That block is worth parsing for its own sake, and it is worth far more as a check on everything else: you have just counted the same things by scanning lines, so you have two independent numbers for one quantity. When they agree, the parser is probably right. When they disagree, you get to find out why — and this chapter's measurement disagrees four times out of sixteen, for a reason that turns out to make both numbers correct.

Intermediate12 min readPerlPart IIUVM Logs

Part II · Module 21 · Chapter 21.7 · The UVM Report Summary Block

1. Five Logs, Sixteen Comparisons

Azvya Education Pvt. Ltd.VLSI Mentor

summary.pl

use strict;
use warnings;

my $MSG = qr/^(?:\#\s*)?(UVM_\w+)\s+(?:(\S+?)\((\d+)\)\s+)?\@\s*(\d+)\s*:\s*(\S*)\s*\[(\w+)\]\s*(.*)$/;
# the summary's own count line: "UVM_ERROR :    3"
my $COUNT_LINE = qr/^(?:\#\s*)?(UVM_\w+)\s*:\s*(\d+)\s*$/;
my $SUMMARY_HDR = qr/^(?:\#\s*)?---\s*UVM Report Summary\s*---/;

sub analyse {
    my ($file) = @_;
    open my $fh, "<", $file or die "$file: $!";
    my (%mine, %stated, $in_summary, $saw_hdr);
    while (my $l = <$fh>) {
        if ($l =~ $SUMMARY_HDR) { $in_summary = 1; $saw_hdr = 1; next }
        if ($l =~ $COUNT_LINE)  { $stated{$1} = $2; next }   # skipped from counting (18.4)
        next unless $l =~ $MSG;
        $mine{$1}++;
    }
    close $fh;
    return (\%mine, \%stated, $saw_hdr);
}

my @LOGS = ("logs/pass.log", "logs/errors.log", "logs/fatal.log",
            "logs/verbose.log", "logs/crash.log");
my @SEV = qw(UVM_INFO UVM_WARNING UVM_ERROR UVM_FATAL);

printf "%-14s %-9s %s\n", "LOG", "SUMMARY?", "severity: mine vs stated";
my ($agree, $disagree, $nosummary) = (0, 0, 0);
my %diff_by_sev;
for my $f (@LOGS) {
    my ($mine, $stated, $hdr) = analyse($f);
    my ($short) = $f =~ m{([^/]+)\.log$};
    unless ($hdr) {
        printf "%-14s %-9s -- no report summary block, so nothing to check\n",
            $short, "no";
        $nosummary++;
        next;
    }
    my @cells;
    for my $s (@SEV) {
        my $m = $mine->{$s}   // 0;
        my $t = $stated->{$s} // 0;
        push @cells, sprintf("%s %d/%d%s", ($s =~ s/^UVM_//r), $m, $t,
                             ($m == $t ? "" : " <>"));
        if ($m == $t) { $agree++ } else { $disagree++; $diff_by_sev{$s} += $m - $t }
    }
    printf "%-14s %-9s %s\n", $short, "yes", join("  ", @cells);
}

printf "\n%d severity comparisons agreed, %d disagreed, %d log(s) had no summary\n",
    $agree, $disagree, $nosummary;

print "\nwhere the disagreements are:\n";
for my $s (sort keys %diff_by_sev) {
    printf "  %-13s mine exceeds stated by %+d in total\n", $s, $diff_by_sev{$s};
}

print "\nthe cause, found by looking rather than assuming:\n";
open my $fh, "<", "logs/pass.log" or die $!;
my @info;
while (my $l = <$fh>) {
    next if $l =~ $COUNT_LINE;
    push @info, $l if $l =~ $MSG && $1 eq "UVM_INFO";
}
close $fh;
printf "  pass.log has %d UVM_INFO message lines; the summary says %d.\n",
    scalar @info, 4;
print "  the extra one is the FIRST:\n";
printf "    %s", $info[0];
print "  it has no file(line) field, because it was printed directly rather\n";
print "  than through the report task that keeps the counters. So BOTH numbers\n";
print "  are correct -- they count different things.\n";

print "\nso a disagreement means one of three things:\n";
print "  1. your pattern matches something that is not a report message\n";
print "  2. your pattern MISSES report messages (the dangerous one)\n";
print "  3. the log contains severity-shaped lines not issued by the report\n";
print "     server -- which is this case, and is not a bug in either number\n";

Azvya Education Pvt. Ltd.VLSI Mentor

Captured output

LOG            SUMMARY?  severity: mine vs stated
pass           yes       INFO 5/4 <>  WARNING 1/1  ERROR 0/0  FATAL 0/0
errors         yes       INFO 5/4 <>  WARNING 2/2  ERROR 3/3  FATAL 0/0
fatal          yes       INFO 4/3 <>  WARNING 2/2  ERROR 3/3  FATAL 1/1
verbose        yes       INFO 137/136 <>  WARNING 2/2  ERROR 0/0  FATAL 0/0
crash          no        -- no report summary block, so nothing to check

12 severity comparisons agreed, 4 disagreed, 1 log(s) had no summary

where the disagreements are:
  UVM_INFO      mine exceeds stated by +4 in total

the cause, found by looking rather than assuming:
  pass.log has 5 UVM_INFO message lines; the summary says 4.
  the extra one is the FIRST:
    UVM_INFO @ 0: reporter [RNTST] Running test my_test...
  it has no file(line) field, because it was printed directly rather
  than through the report task that keeps the counters. So BOTH numbers
  are correct -- they count different things.

so a disagreement means one of three things:
  1. your pattern matches something that is not a report message
  2. your pattern MISSES report messages (the dangerous one)
  3. the log contains severity-shaped lines not issued by the report
     server -- which is this case, and is not a bug in either number

Twelve agreed, four disagreed, and one log had no summary to compare against.

Note what the four disagreements are: all UVM_INFO, and all by exactly one. WARNING, ERROR and FATAL agree in every log — which is the part that matters, because those three decide the verdict.

2. What The Disagreement Actually Was

The script does not stop at reporting the difference; it looks:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

pass.log has 5 UVM_INFO message lines; the summary says 4.
the extra one is the FIRST:
  UVM_INFO @ 0: reporter [RNTST] Running test my_test...

That line has no file(line) field — Chapter 21.3's regex makes the group optional precisely so it parses — because it is printed directly rather than through the report task that maintains the counters.

So the two numbers count different populations:

My count counts severity-shaped lines in the log.

The summary counts messages issued through the report server.

Both are correct. This is the honest outcome of the cross-check, and it is worth sitting with, because the instinct when a check fires is to find a bug. Sometimes there is no bug and the two quantities were never the same quantity — and knowing that is itself the result. It also tells you which number to publish: for "how many messages did the testbench issue", the summary; for "how many lines will my parser process", your own.

Parsing the count line needs a pattern that is nearly identical to the one that skips it:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

my $COUNT_LINE = qr/^(?:\#\s*)?(UVM_\w+)\s*:\s*(\d+)\s*$/;

Chapter 18.4's skip pattern and this capture pattern are the same shape, which is the point — the line you must not count as a message is exactly the line you want to read the count from. Writing them as one pattern used two ways, rather than two patterns that drift apart, is worth doing:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

if ($l =~ $COUNT_LINE) { $stated{$1} = $2; next }   # read it, then skip it

One statement reads the stated count and prevents the double-count in the same breath. Two separate patterns is how the skip gets updated for the Questa prefix and the capture does not.

3. The Three Things A Disagreement Can Mean

The script ends by enumerating them, because which one you have determines what to do:

Your pattern matches something that is not a report message. Your count is too high. The classic case is Chapter 18.4's: the summary's own count lines, matched as messages.

Your pattern misses report messages. Your count is too low, and this is the dangerous one — a missed UVM_ERROR turns a failing test into a passing one. Chapter 21.1 measured exactly how this happens in practice: a Questa transcript's # prefix makes an anchored pattern match nothing at all.

The log contains severity-shaped lines not issued by the report server. Neither number is wrong. This is the measured case here.

The direction of the difference tells you which you are in, and it is worth putting in the output:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

mine exceeds stated by +4 in total

Mine higher points at case one or three. Mine lower points at case two, and should be loud — it is the only one of the three that can flip a verdict. Chapter 18.8's rule applies: a parser that notices it cannot account for something must say so rather than carry on.

4. The Log With No Summary

The crash row:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

crash          no        -- no report summary block, so nothing to check

A truncated run has no summary, so the cross-check cannot run — and the correct response is to say so, not to compare against zero.

This is Chapter 18.9's three-state rule and Chapter 21.2's NO_FINISH, appearing once more in a different guise. The three states for the summary block are:

Present and parsed — compare, and report agreement or difference.

Absent — the run did not reach the end. Report no summary and let Chapter 21.2's completion check own the verdict. Comparing your counts against an absent summary means comparing against 0, which would report "mine exceeds stated by 3" on a crashed failing test and look like a parser bug.

Present but partial — the header printed and the counts did not, because the log was cut between them. Rarer, and the reason the script tracks the header separately from the counts rather than inferring one from the other.

That last distinction is why $saw_hdr exists as its own variable. Without it, "no counts found" and "no summary block" are the same condition, and they have different causes.

5. What To Remember

UVM prints its own severity counts; you have just counted the same things. Two independent numbers for one quantity.
Measured: 12 of 16 comparisons agreed, 4 disagreed, 1 log had no summary.
All four disagreements were UVM_INFO, by exactly one.
WARNING, ERROR and FATAL agreed in every log — the three that decide the verdict.
The cause: a banner line printed directly rather than through the report task. It has no file(line) field, which is why 21.3's regex makes that group optional.
Both numbers are correct — they count different populations: lines in the log versus messages issued through the report server.
When a check fires, the answer is sometimes "these were never the same quantity" — and that is a result, not a dead end.
Read the count line and skip it in one statement, with one shared pattern (18.4), or the skip and the capture drift apart.
A disagreement means one of three things: you match non-messages, you miss messages, or the log has severity-shaped lines from elsewhere.
"Mine lower" is the dangerous direction — a missed UVM_ERROR turns a failure into a pass. Make it loud (18.8).
Report the direction of the difference, not just that there is one.
The check is free — the log is already read and the severities already counted.
It is the only INDEPENDENT check a log parser has. UVM's counters are not maintained by your regex (20.4's argument).
No summary block means the run did not finish — say so rather than comparing against zero, which looks like a parser bug.
Track the header separately from the counts, so "cut off between them" is distinguishable from "no summary at all".

Chapter 21.8 brings the evidence together. The finish marker, the severity counts and the wall-clock record each answer part of the question, and the verdict has four possible values — which makes the order and the explicitness of the rule the whole design.

Where this fits

Part of the Perl curriculum.