Skip to content
VLSI Mentor

Perl Part II · Module 21 · Simulation and UVM Log Analysis

Deciding PASS, FAIL, TIMEOUT or UNKNOWN

Everything the previous chapters extracted now has to collapse into one word per test, and the hard part is not the logic — it is that the logic is almost always written as a chain of nested conditionals where the order is accidental and invisible. This chapter writes the rule as an ordered table instead, which makes three things possible: reading the precedence at a glance, having each verdict explain itself, and proving that every rule can actually fire. The first attempt had three rules that could not.

Intermediate12 min readPerlPart IIUVM Logs

Part II · Module 21 · Chapter 21.8 · Deciding PASS, FAIL, TIMEOUT or UNKNOWN

1. Eight Inputs, Seven Rules

Azvya Education Pvt. Ltd.VLSI Mentor
verdict.pl
use strict;
use warnings;

my $MSG = qr/^(?:\#\s*)?(UVM_\w+)\s+(?:(\S+?)\((\d+)\)\s+)?\@\s*(\d+)\s*:\s*(\S*)\s*\[(\w+)\]\s*(.*)$/;
my $COUNT_LINE = qr/^(?:\#\s*)?(UVM_\w+)\s*:\s*(\d+)\s*$/;
my @FINISH = (
  qr/^(?:#\s*)?\$finish\b/, qr/^\$finish at simulation time/,
  qr/^Simulation complete via \$finish/, qr/^#\s*\*\* Note: \$finish/,
);

# ---- gather the EVIDENCE. No verdict here: just facts.
sub evidence {
    my ($name) = @_;
    my $log = "logs/$name.log";
    my %e = (test => $name);
    unless (-e $log)  { $e{no_log}  = 1; return %e }
    if     (-z $log)  { $e{empty}   = 1; return %e }
    open my $fh, "<", $log or do { $e{unreadable} = 1; return %e };
    $e{$_} = 0 for qw(err fatal warn finish summary);
    while (my $l = <$fh>) {
        $e{summary} = 1 if $l =~ /^(?:#\s*)?---\s*UVM Report Summary\s*---/;
        next if $l =~ $COUNT_LINE;
        for my $f (@FINISH) { $e{finish} = 1 if $l =~ $f }
        next unless $l =~ $MSG;
        $e{err}++   if $1 eq "UVM_ERROR";
        $e{fatal}++ if $1 eq "UVM_FATAL";
        $e{warn}++  if $1 eq "UVM_WARNING";
    }
    close $fh;
    # evidence from OUTSIDE the log (21.2): the job's own runtime record
    if (open my $ri, "<", "logs/$name.runinfo") {
        while (my $l = <$ri>) { $e{$1} = $2 if $l =~ /^(\w+)=(\S+)$/ }
        close $ri;
    }
    return %e;
}

# ---- the RULE, written as an ordered table rather than nested ifs.
# Each row: a name, a test on the evidence, and why it comes where it does.
my @RULE = (
  [ "NO_LOG",   sub { $_[0]{no_log} },     "the job never produced a log"        ],
  [ "EMPTY",    sub { $_[0]{empty} },      "zero bytes: the job died at startup" ],
  [ "UNREAD",   sub { $_[0]{unreadable} }, "it exists and cannot be opened"      ],
  [ "TIMEOUT",  sub { my $e = shift;
                      !$e->{finish} && defined $e->{wall_seconds}
                      && defined $e->{limit_seconds}
                      && $e->{wall_seconds} >= $e->{limit_seconds} },
                                            "no finish AND it reached its limit"  ],
  [ "CRASH",    sub { !$_[0]{finish} },     "no finish, and not at its limit"     ],
  [ "FAIL",     sub { $_[0]{fatal} || $_[0]{err} }, "finished, with errors"       ],
  [ "PASS",     sub { 1 },                  "finished, nothing to report"         ],
);

sub verdict {
    my (%e) = @_;
    for my $r (@RULE) { return $r->[0] if $r->[1]->(\%e) }
    return "UNKNOWN";                        # unreachable, and kept deliberately
}

# Build the three degenerate inputs so every rule row can be exercised.
# A rule that cannot fire is dead code until it has been shown to fire.
open my $z, ">", "logs/zerobyte.log" or die $!; close $z;          # 0 bytes
open my $u, ">", "logs/locked.log"   or die $!; print $u "x\n"; close $u;
chmod 0000, "logs/locked.log";
# logs/absent.log is deliberately never created

my @TESTS = qw(pass errors fatal crash timeout absent zerobyte locked);
printf "%-9s %-7s %-7s %-8s %-9s %-9s %-11s %s\n",
    "TEST", "errors", "fatals", "finish?", "summary?", "wall/lim", "VERDICT", "why";
my %got;
for my $t (@TESTS) {
    my %e = evidence($t);
    my $v = verdict(%e);
    $got{$t} = $v;
    my ($why) = map { $_->[2] } grep { $_->[0] eq $v } @RULE;
    printf "%-9s %-7s %-7s %-8s %-9s %-9s %-11s %s\n", $t,
        ($e{err} // "-"), ($e{fatal} // "-"),
        (defined $e{finish} ? ($e{finish} ? "yes" : "NO") : "-"),
        (defined $e{summary} ? ($e{summary} ? "yes" : "no") : "-"),
        (defined $e{wall_seconds} ? "$e{wall_seconds}/$e{limit_seconds}" : "-"),
        $v, ($why // "");
}

chmod 0644, "logs/locked.log";

print "\nis every rule row reachable? (a rule that cannot fire is dead code)\n";
my %fired = map { $_ => 1 } values %got;
for my $r (@RULE) {
    printf "  %-9s %s\n", $r->[0],
        ($fired{$r->[0]} ? "fired on this set" : "NOT exercised here");
}

print "\nwhy TIMEOUT must be tested BEFORE CRASH:\n";
print "  both require 'no finish marker'. CRASH is the weaker condition, so a\n";
print "  CRASH row placed first would swallow every timeout. Order IS the rule.\n";
my %e = evidence("timeout");
printf "  timeout.log: finish=%s wall=%s limit=%s -> %s\n",
    ($e{finish} ? "yes" : "no"), $e{wall_seconds}, $e{limit_seconds}, verdict(%e);

print "\nand what happens with no runinfo at all:\n";
my %t2 = evidence("timeout");
delete $t2{wall_seconds};
printf "  same log, runtime record missing -> %s\n", verdict(%t2);
print "  the verdict degrades to CRASH, which is honest: without outside\n";
print "  evidence a timeout is indistinguishable from a crash (21.2).\n";
Azvya Education Pvt. Ltd.VLSI Mentor
Captured output
TEST      errors  fatals  finish?  summary?  wall/lim  VERDICT     why
pass      0       0       yes      yes       12/7200   PASS        finished, nothing to report
errors    3       0       yes      yes       14/7200   FAIL        finished, with errors
fatal     3       1       yes      yes       15/7200   FAIL        finished, with errors
crash     0       0       NO       no        9/7200    CRASH       no finish, and not at its limit
timeout   0       0       NO       no        7201/7200 TIMEOUT     no finish AND it reached its limit
absent    -       -       -        -         -         NO_LOG      the job never produced a log
zerobyte  -       -       -        -         -         EMPTY       zero bytes: the job died at startup
locked    -       -       -        -         -         UNREAD      it exists and cannot be opened

is every rule row reachable? (a rule that cannot fire is dead code)
  NO_LOG    fired on this set
  EMPTY     fired on this set
  UNREAD    fired on this set
  TIMEOUT   fired on this set
  CRASH     fired on this set
  FAIL      fired on this set
  PASS      fired on this set

why TIMEOUT must be tested BEFORE CRASH:
  both require 'no finish marker'. CRASH is the weaker condition, so a
  CRASH row placed first would swallow every timeout. Order IS the rule.
  timeout.log: finish=no wall=7201 limit=7200 -> TIMEOUT

and what happens with no runinfo at all:
  same log, runtime record missing -> CRASH
  the verdict degrades to CRASH, which is honest: without outside
  evidence a timeout is indistinguishable from a crash (21.2).

Eight inputs, eight verdicts, and every one of the seven rule rows fired.

That last part took work. The first version of this measurement used only the five real logs, and NO_LOG, EMPTY and UNREAD never fired — three rules out of seven that were dead code as far as any evidence went. The three degenerate inputs exist because of it: a log that was never created, a zero-byte file, and a file with its permissions removed.

2. Evidence And Rule Are Separate

The script has two functions and the split is the design:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
sub evidence { ... }      # returns FACTS: counts, flags, runtime record. No verdict.
sub verdict  { ... }      # returns ONE WORD from those facts. Reads no files.

Three things that buys:

The rule is testable without logs. verdict() takes a hash, so a unit test is a hash literal — which is how section 5's "what if the runtime record is missing" case is measured, by deleting a key rather than by manufacturing a log.

The evidence is reusable. Chapter 21.11's per-test summary needs the same counts for its columns, and reports them beside the verdict rather than re-deriving them.

Neither function can quietly do the other's job. A verdict function that reads a file ends up re-reading it for a special case, and an evidence function that returns a verdict ends up with the rule spread across both.

Note what evidence() returns for a missing log: a hash with no_log set and no counts at all. Not zeroes. Chapter 2.7's distinction — a missing log has no error count, and 0 would be a claim about a run that did not happen.

3. The Rule As An Ordered Table

Figure 21.8.1 — The verdict ladder, in the one order that works

A seven-gate verdict ladder: no log, empty log, unreadable log, timeout when there is no finish marker and the job reached its wall-clock limit, crash when there is no finish marker, fail when errors or fatals are present, and pass by elimination.Seven gates, one word1No log?the job produced nothing2Zero bytes?died at startup3Unreadable?exists, cannot open4No finish + at limit?TIMEOUT5No finish?CRASH6Errors or fatals?FAIL7PASSby elimination only
Each gate is only meaningful once every gate above it has declined. The three existence checks come first because the counts do not exist yet. The completion check comes before the severity check because PASS and FAIL are both statements about a run that happened. TIMEOUT precedes CRASH because both require a missing finish marker and CRASH is the weaker test, so reversing them makes every timeout a crash. PASS is last and is reached only by elimination, never by satisfying a condition.

In code the ladder is data:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
my @RULE = (
  [ "NO_LOG",  sub { $_[0]{no_log} },     "the job never produced a log"       ],
  [ "EMPTY",   sub { $_[0]{empty} },      "zero bytes: the job died at startup" ],
  ...
  [ "PASS",    sub { 1 },                 "finished, nothing to report"         ],
);

sub verdict {
    my (%e) = @_;
    for my $r (@RULE) { return $r->[0] if $r->[1]->(\%e) }
    return "UNKNOWN";
}

Each row is a name, a test (Chapter 7.11's code reference), and a reason. Four properties follow:

The precedence is visible. Reading seven rows top to bottom is the whole rule. A nested if/elsif chain encodes the same precedence in indentation, where it is correct by accident and changed by accident.

Every verdict explains itself. The reason string is printed in the output — no finish AND it reached its limit — so the report does not need a reader who knows the rule.

Reachability is testable, because the rows are enumerable. The loop that asserts every row fired is four lines and it is what found the three dead rules.

PASS is sub { 1 } at the bottom. Not a condition — the absence of any other. Chapter 21.2 measured what happens when PASS is written as a condition instead: a crashed log satisfies $err == 0.

4. Why TIMEOUT Must Precede CRASH

Both rules require the same thing — no finish marker — and differ only in that TIMEOUT additionally requires the wall-clock evidence:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
[ "TIMEOUT", sub { ... !$e->{finish} && $e->{wall_seconds} >= $e->{limit_seconds} } ],
[ "CRASH",   sub { !$_[0]{finish} } ],

CRASH is the weaker test, so it matches everything TIMEOUT matches. Place it first and every timeout is reported as a crash, and no amount of correct code in the TIMEOUT row will ever run.

The measurement states it rather than leaving it to be noticed:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
timeout.log: finish=no wall=7201 limit=7200 -> TIMEOUT

This is the general shape of every classifier in this track: when one condition implies another, the more specific one goes first. Chapter 19.5's dependency order, Chapter 20.5's ^CKLN before ^CK, Chapter 18.8's spelling table ordered by expected frequency — the same rule each time, and in every case the failure is silent, because the wrong answer is still one of the legal answers.

5. Degrading Honestly

The last measurement removes the runtime record from the timeout case and re-runs the rule:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
same log, runtime record missing -> CRASH

The verdict degrades to CRASH, which is the honest answer. Chapter 21.2 measured that a crash and a timeout are byte-identical in the log; the distinction lives entirely in the wall-clock evidence. Without it, "crashed" is what you actually know.

The defined guards are what make that work rather than crash the script:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
!$e->{finish} && defined $e->{wall_seconds} && defined $e->{limit_seconds}
  && $e->{wall_seconds} >= $e->{limit_seconds}

Both defined tests come before the comparison, so a missing record makes the row decline rather than producing an uninitialized-value warning on stderr — Chapter 20.3's lesson about a warning nobody reads while stdout looks right.

And the right thing to do about the degradation is to say it in the report, not to hide it: CRASH (no runtime record, so a timeout cannot be ruled out). One clause, and nobody re-investigates a distinction the data could not support.

6. What To Remember

  • Measured: 8 inputs, 7 rule rows, every row fired.
  • The first attempt had 3 rules that never fired — NO_LOG, EMPTY, UNREAD — until degenerate inputs were built for them. A rule that cannot fire is dead code.
  • Separate evidence() from verdict(): one reads files and returns facts, the other reads facts and returns a word.
  • A testable rule takes a hash, so a test case is a hash literal, not a manufactured log.
  • A missing log has NO counts, not zero counts (2.7) — 0 is a claim about a run that did not happen.
  • Write the rule as an ordered table of [name, test, reason], not nested conditionals.
  • The precedence becomes visible data instead of being encoded in indentation.
  • Each row carries its reason, so the verdict explains itself in the output.
  • Reachability becomes testable, because the rows are enumerable.
  • PASS is sub { 1 } at the bottom — reached by elimination, never satisfied as a condition (21.2).
  • Keep the unreachable terminal UNKNOWN — it documents the invariant that the last row must be a catch-all.
  • TIMEOUT must precede CRASH: both need "no finish", CRASH is weaker, and reversing them makes every timeout a crash.
  • When one condition implies another, the specific one goes first — the same rule as 19.5's ordering, 20.5's ^CKLN before ^CK, and 18.8's spelling table.
  • The failure is always silent, because the wrong answer is still a legal answer.
  • Measured: remove the runtime record and TIMEOUT degrades to CRASH — the honest answer, since the log alone cannot tell them apart (21.2).
  • Put the defined guards before the comparison (20.3), or a missing record warns on stderr.
  • Say the degradation in the report: "CRASH (no runtime record, so a timeout cannot be ruled out)".

Chapter 21.9 takes the verdict for granted and asks the next question. A hundred failing tests are rarely a hundred problems, and turning each first error into a normalised signature is what collapses them into the handful of causes somebody can actually work through.

Where this fits

Part of the Perl curriculum.