Python Part II · Module 25 · Simulation Automation

Classifying the Run and Returning a Structured Result

Four steps have produced three separate pieces of evidence about what just happened: an exit status from the simulator, a log full of what the testbench said, and a measurement of how long it took. The job now is to turn those into one word and one object. That sounds like bookkeeping and it is not — the three sources disagree in most of the cases they can produce, and the order in which you consult them decides whether a hung simulation is reported as a hang or as a pass.

Intermediate20 min readPythonPart IIAutomationSimulationVerification Flow

Part II · Module 25 · Chapter 25.6 · Classifying the Run and Returning a Structured Result

1. Three Witnesses

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

EVIDENCE = [
    ("the return code", "the simulator's own claim about how it exited", "Chapter 23.4"),
    ("the log", "what the testbench actually reported while running", "Chapters 20.7–20.12"),
    ("the wall clock", "whether it finished inside the time it was given", "Chapter 23.8"),
]

for name, what, where in EVIDENCE:
    print(f"{name:<18} {what:<48} {where}")

print()
print("none of the three is authoritative on its own:")
print("   a return code is a claim, not evidence — Chapter 23.4 §4")
print("   a log can be truncated by a kill — Chapter 23.8 §7")
print("   a clock says nothing about correctness")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

the return code    the simulator's own claim about how it exited    Chapter 23.4
the log            what the testbench actually reported while running Chapters 20.7–20.12
the wall clock     whether it finished inside the time it was given Chapter 23.8

none of the three is authoritative on its own:
   a return code is a claim, not evidence — Chapter 23.4 §4
   a log can be truncated by a kill — Chapter 23.8 §7
   a clock says nothing about correctness

Three witnesses, none of them reliable alone. Classification is what you do when witnesses disagree, and the first question is how often that actually happens.

2. The Annotation

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

return RunResult(test, seed, status, errors, runtime_s, log_path)

Six values, and they come from three different places: test and seed are the caller's, status and errors are derived, runtime_s is measured, and log_path is where the evidence lives. Chapter 25.1 defined the type; this chapter is about where status comes from.

3. How Often the Witnesses Disagree

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import itertools


def honest(rc_ok, log_clean, finished):
    if not finished:
        return "NO_FINISH"
    if not log_clean:
        return "FAIL"
    if not rc_ok:
        return "FAIL"
    return "PASS"


print(f"{'rc==0':<7} {'log clean':<11} {'finished':<10} status")
rows = []
for rc_ok, log_clean, finished in itertools.product([True, False], repeat=3):
    status = honest(rc_ok, log_clean, finished)
    rows.append((rc_ok, log_clean, finished, status))
    print(f"{str(rc_ok):<7} {str(log_clean):<11} {str(finished):<10} {status}")

unanimous = sum(1 for a, b, c, _ in rows if a == b == c)
print()
print("combinations            :", len(rows))
print("all three agree         :", unanimous)
print("need a precedence rule  :", len(rows) - unanimous)

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

rc==0   log clean   finished   status
True    True        True       PASS
True    True        False      NO_FINISH
True    False       True       FAIL
True    False       False      NO_FINISH
False   True        True       FAIL
False   True        False      NO_FINISH
False   False       True       FAIL
False   False       False      NO_FINISH

combinations            : 8
all three agree         : 2
need a precedence rule  : 6

Six of eight combinations have the witnesses in conflict. Only all-good and all-bad are unanimous; everything between them is a decision somebody has to make deliberately.

That is why this chapter exists as more than a constructor call. Two of those conflicts are worth naming, because both are real and both have a right answer.

rc == 0 and the log reports errors. The log wins. A UVM testbench that reports UVM_ERROR and never calls $fatal finishes normally and the simulator exits zero — it ran the program it was given and the program ended. Chapter 23.4 Section 4 made this point generally; here it is the specific case that matters most, because a flow trusting the return code alone would report a failing test as a pass.

rc != 0 and the log is clean. Still FAIL, but for a different reason: the tool died before it could report anything. A segmentation fault in the simulator, a licence lost mid-run, a SIGKILL from the scheduler. The status is the same and the cause is not, which is what the reason field is for.

4. The Order of the Checks Is the Design

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

def correct(rc, errors, finished):
    if not finished:
        return "NO_FINISH"
    if errors:
        return "FAIL"
    if rc != 0:
        return "FAIL"
    return "PASS"


def happy_path_first(rc, errors, finished):
    """The natural way to write it, and it is wrong."""
    if rc == 0 and not errors:
        return "PASS"
    if not finished:
        return "NO_FINISH"
    return "FAIL"


CASES = [(0, 0, True, "a clean run"),
         (0, 2, True, "errors in the log, tool exited 0"),
         (1, 0, True, "tool crashed, log clean"),
         (0, 0, False, "hung — killed at the timeout, nothing logged")]

print(f"{'rc':<4} {'errors':<8} {'finished':<10} {'correct':<11} {'happy-first':<12} agree")
for rc, errors, finished, _ in CASES:
    a, b = correct(rc, errors, finished), happy_path_first(rc, errors, finished)
    print(f"{rc:<4} {errors:<8} {str(finished):<10} {a:<11} {b:<12} {a == b}")

print()
bad = [d for rc, e, f, d in CASES if correct(rc, e, f) != happy_path_first(rc, e, f)]
print("cases where they differ:", bad)

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

rc   errors   finished   correct     happy-first  agree
0    0        True       PASS        PASS         True
0    2        True       FAIL        FAIL         True
1    0        True       FAIL        FAIL         True
0    0        False      NO_FINISH   PASS         False

cases where they differ: ['hung — killed at the timeout, nothing logged']

The natural way to write it reports PASS for a hung simulation.

Look at the last row. The simulator was killed at its timeout, so it logged nothing unusual and the partial log is clean; the return code variable still holds 0 because the run never set one. rc == 0 and not errors is satisfied, and the function returns PASS before it ever consults finished.

That is the worst mistake this module can make. A FAIL reported as PASS loses one test; a NO_FINISH reported as PASS loses a class of failure permanently, because a hang is exactly the thing nobody notices is missing. Chapter 20.12 spent a whole chapter on detecting a run that never finished, and one misordered if throws that detection away.

The rule that prevents it:

Check the ways a run can be invalid before you check whether it passed. A run that did not complete has no verdict to report, whatever its other evidence says.

not finished first, then evidence of failure, then — and only then — success. PASS is the value you fall through to, never the one you test for.

5. Where `finished` Comes From

Azvya Education Pvt. Ltd.VLSI Mentor

sim.log

UVM_INFO @ 0: reporter [RNTST] Running test axi_burst...
UVM_INFO drv.sv(60) @ 141700: issuing write
UVM_ERROR drv.sv(88) @ 141900: response mismatch
UVM_INFO @ 200000: reporter [TEST_DONE] Test finished
--- UVM Report Summary ---
UVM_ERROR : 1
TEST DONE

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import re
from pathlib import Path

END_MARKERS = ("TEST DONE", "--- UVM Report Summary ---", "$finish")
ERROR_LINE = re.compile(r"^(UVM_ERROR|UVM_FATAL)\b")


def read_log(path):
    text = Path(path).read_text(encoding="utf-8", errors="replace")
    lines = text.splitlines()
    errors = tuple(l for l in lines if ERROR_LINE.match(l))
    finished = any(marker in text for marker in END_MARKERS)
    return errors, finished


errors, finished = read_log("sim.log")
print("error lines :", len(errors))
print("first error :", errors[0] if errors else None)
print("finished    :", finished)
print("marker found:", next((m for m in END_MARKERS if m in Path('sim.log').read_text(encoding='utf-8')), None))

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

error lines : 2
first error : UVM_ERROR drv.sv(88) @ 141900: response mismatch
finished    : True
marker found: TEST DONE

Two error lines, from a log containing one error. The specimen's UVM report summary ends with UVM_ERROR : 1 — UVM's own tally — and ^UVM_ERROR\b matches that too. The count is now double, and on a real run with a hundred errors it would be a hundred and one, which looks exactly like an off-by-one nobody can find.

The tally line is distinguishable because what follows the tag is a colon and a number rather than a source location:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import re

LINES = ["UVM_ERROR drv.sv(88) @ 141900: response mismatch",
         "UVM_ERROR : 1",
         "UVM_FATAL tb.sv(12) @ 0: no config"]

NAIVE = re.compile(r"^(UVM_ERROR|UVM_FATAL)\b")
TALLY = re.compile(r"^(UVM_ERROR|UVM_FATAL)\s*:\s*\d+\s*$")

for line in LINES:
    print(f"{line:<52} naive={bool(NAIVE.match(line))} tally={bool(TALLY.match(line))}")

real = [l for l in LINES if NAIVE.match(l) and not TALLY.match(l)]
print()
print("counted as errors:", len(real))

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

UVM_ERROR drv.sv(88) @ 141900: response mismatch     naive=True tally=False
UVM_ERROR : 1                                        naive=True tally=True
UVM_FATAL tb.sv(12) @ 0: no config                   naive=True tally=False

counted as errors: 2

Excluding the tally gives two real findings from three matching lines. This is Chapter 20.7's territory and it is worth re-stating here because the consequence lands in a number somebody reports: a summary line counted as an error inflates every error count in the regression by exactly one per run, uniformly, which is the hardest kind of wrong number to notice.

finished is not a property of the process — it is a property of the log, and specifically of whether the testbench got far enough to say so. Chapter 20.12 built this; two details carry forward.

Several markers, any of which counts. Different testbenches end differently, and a list with any() tolerates that where a single hard-coded string would report every run from a differently-written bench as a hang.

errors=() is a tuple, matching Chapter 25.1 Section 4's result type, so it can be stored in a frozen dataclass and the whole result stays hashable.

The timeout path is the one case where finished is decided without the log at all:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

EVIDENCE = [
    ("normal exit, banner present", False, True, "the log decides"),
    ("normal exit, banner absent", False, False, "the log decides — a hang that died quietly"),
    ("killed at the timeout", True, False, "the timeout decides, regardless of the log"),
]

for what, timed_out, banner, who in EVIDENCE:
    finished = (not timed_out) and banner
    print(f"{what:<30} timed_out={str(timed_out):<6} banner={str(banner):<6} "
          f"finished={str(finished):<6} {who}")

print()
print("a timed-out run is never 'finished', even if a stale banner is in the log")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

normal exit, banner present    timed_out=False  banner=True   finished=True   the log decides
normal exit, banner absent     timed_out=False  banner=False  finished=False  the log decides — a hang that died quietly
killed at the timeout          timed_out=True   banner=False  finished=False  the timeout decides, regardless of the log

The second row is Chapter 20.12's inferred hang — the process exited, nothing was logged at the end, and nobody knows why. The third is Chapter 23.8's imposed one, which is the certain case because the parent set the limit.

6. The Classifier

Figure 1 — the three witnesses of Section 1 — the return code, the log and the clock — reach one classifier, which consults them in a fixed order: did it finish, did it report errors, did it exit cleanly. None is authoritative alone, which is why the order is part of the design rather than an implementation detail, and why a single RunResult comes out the other side.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

def classify(*, timed_out, banner, errors, returncode, launch_error=None,
             compile_failed=False):
    """Fixed precedence: invalid, then failed, then passed."""
    if launch_error is not None:
        return "ERROR", launch_error
    if compile_failed:
        return "COMPILE_ERROR", None
    if timed_out:
        return "NO_FINISH", "TIMEOUT"
    if not banner:
        return "NO_FINISH", "NO_END_MARKER"
    if errors:
        return "FAIL", "UVM_ERROR"
    if returncode != 0:
        return "FAIL", f"EXIT_{returncode}"
    return "PASS", None


CASES = [
    dict(timed_out=False, banner=True, errors=0, returncode=0),
    dict(timed_out=False, banner=True, errors=3, returncode=0),
    dict(timed_out=False, banner=True, errors=0, returncode=139),
    dict(timed_out=False, banner=False, errors=0, returncode=0),
    dict(timed_out=True, banner=False, errors=0, returncode=None),
    dict(timed_out=False, banner=False, errors=0, returncode=0, compile_failed=True),
    dict(timed_out=False, banner=False, errors=0, returncode=None,
         launch_error="xrun not on PATH"),
]

for case in CASES:
    status, reason = classify(**case)
    print(f"{status:<14} reason={str(reason)}")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

PASS           reason=None
FAIL           reason=UVM_ERROR
FAIL           reason=EXIT_139
NO_FINISH      reason=NO_END_MARKER
NO_FINISH      reason=TIMEOUT
COMPILE_ERROR  reason=None
ERROR          reason=xrun not on PATH

Seven cases, five statuses, and every branch tested. Four properties of that function are worth copying.

The order runs from least-valid to most-valid. ERROR (nothing ran), COMPILE_ERROR (nothing was tested), NO_FINISH (no verdict), FAIL (a verdict, bad), PASS (a verdict, good). Each branch rules out a world in which the later questions are meaningless.

Every parameter is keyword-only. Six booleans and integers in a row is Chapter 25.1 Section 3's argument in its purest form — a positional call here would be unreadable and silently reorderable.

reason distinguishes causes within a status. Two NO_FINISH rows have different reasons and two FAIL rows do too, which keeps the vocabulary at Chapter 25.1's five while losing nothing. EXIT_139 is the segmentation-fault case, and Chapter 23.4 Section 5's negative codes would appear here the same way.

PASS is the fall-through. There is no if that produces it, which is the structural version of Section 4's rule: you cannot accidentally reach PASS by failing to ask a question, because every question is asked before it.

7. Measuring Without Publishing

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import time

start = time.monotonic()
total = sum(i * i for i in range(200000))
elapsed = time.monotonic() - start

print("measured something :", total > 0)
print("elapsed is a float :", isinstance(elapsed, float))
print("non-negative       :", elapsed >= 0)
print("monotonic never goes backwards:", time.monotonic() >= start)
print()
print("rounded for the record:", isinstance(round(elapsed, 2), float))
print("the value itself is NOT printed — it differs on every machine")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

measured something : True
elapsed is a float : True
non-negative       : True
monotonic never goes backwards: True

rounded for the record: True
the value itself is NOT printed — it differs on every machine

Two decisions in that measurement.

time.monotonic, not time.time. A wall clock can be adjusted — by NTP, by a daylight-saving change, by an administrator — and a run straddling an adjustment produces a duration that is wrong or negative. monotonic only ever increases, which is the property a duration needs. It is meaningless as an absolute time, which is fine, because a duration is a difference.

The value is recorded and never published here. Every assertion above is a bound or a type. This curriculum does not print timings, and a runner nevertheless must record one — the two are compatible because the record is for a trend report, not for a reader.

And the value's place in the result was decided in Chapter 25.1 Section 7:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

from dataclasses import dataclass, field, fields


@dataclass(frozen=True)
class RunResult:
    test: str
    seed: int
    status: str
    runtime_s: float = field(default=0.0, compare=False)


a = RunResult("axi_smoke", 1839274, "PASS", runtime_s=182.4)
b = RunResult("axi_smoke", 1839274, "PASS", runtime_s=206.9)

print("two runs of the same test, different durations")
print("equal      :", a == b)
print("in the record:", a.runtime_s, b.runtime_s)
print("compares on:", [f.name for f in fields(a) if f.compare])

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

two runs of the same test, different durations
equal      : True
in the record: 182.4 206.9
compares on: ['test', 'seed', 'status']

Measured, stored, and excluded from identity. A trend report reads runtime_s; a "did anything change?" comparison does not.

8. One Result, Assembled

Azvya Education Pvt. Ltd.VLSI Mentor

sim.log

UVM_INFO @ 0: reporter [RNTST] Running test axi_burst...
UVM_INFO drv.sv(60) @ 141700: issuing write
UVM_ERROR drv.sv(88) @ 141900: response mismatch
UVM_ERROR drv.sv(95) @ 142100: response mismatch
--- UVM Report Summary ---
TEST DONE

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import re
import time
from dataclasses import dataclass, field, asdict
from pathlib import Path
from typing import Optional
import json

STATUSES = ("PASS", "FAIL", "COMPILE_ERROR", "NO_FINISH", "ERROR")
END_MARKERS = ("TEST DONE", "--- UVM Report Summary ---", "$finish")
ERROR_LINE = re.compile(r"^(UVM_ERROR|UVM_FATAL)\b")
TALLY_LINE = re.compile(r"^(UVM_ERROR|UVM_FATAL)\s*:\s*\d+\s*$")
MAX_ERRORS = 10


@dataclass(frozen=True)
class RunResult:
    test: str
    seed: int
    sim: str
    status: str
    errors: tuple = ()
    error_count: int = 0
    reason: Optional[str] = None
    returncode: Optional[int] = None
    run_dir: Optional[str] = None
    log_path: Optional[str] = None
    runtime_s: float = field(default=0.0, compare=False)

    def __post_init__(self):
        if self.status not in STATUSES:
            raise ValueError(f"unknown status {self.status!r}")


def parse_log(path):
    text = Path(path).read_text(encoding="utf-8", errors="replace")
    lines = [l for l in text.splitlines()
             if ERROR_LINE.match(l) and not TALLY_LINE.match(l)]
    return tuple(lines), any(m in text for m in END_MARKERS)


def classify(*, timed_out, banner, errors, returncode):
    if timed_out:
        return "NO_FINISH", "TIMEOUT"
    if not banner:
        return "NO_FINISH", "NO_END_MARKER"
    if errors:
        return "FAIL", "UVM_ERROR"
    if returncode != 0:
        return "FAIL", f"EXIT_{returncode}"
    return "PASS", None


def build_result(test, seed, sim, run_dir, log_path, returncode, timed_out, elapsed):
    errors, banner = parse_log(log_path) if Path(log_path).exists() else ((), False)
    status, reason = classify(timed_out=timed_out, banner=banner,
                              errors=len(errors), returncode=returncode)
    return RunResult(test=test, seed=seed, sim=sim, status=status,
                     errors=errors[:MAX_ERRORS], error_count=len(errors),
                     reason=reason, returncode=returncode,
                     run_dir=run_dir, log_path=log_path,
                     runtime_s=round(elapsed, 2))


start = time.monotonic()
result = build_result("axi_burst", 1839274, "vcs", "runs/axi_burst_1839274",
                      "sim.log", returncode=0, timed_out=False,
                      elapsed=time.monotonic() - start)

print("status      :", result.status)
print("reason      :", result.reason)
print("error_count :", result.error_count)
print("errors kept :", len(result.errors))
print("first error :", result.errors[0])
print("runtime is a float:", isinstance(result.runtime_s, float))
print()
record = {k: v for k, v in asdict(result).items() if k != "runtime_s"}
print(json.dumps(record, indent=2, sort_keys=True)[:320])

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

status      : FAIL
reason      : UVM_ERROR
error_count : 2
errors kept : 2
first error : UVM_ERROR drv.sv(88) @ 141900: response mismatch
runtime is a float: True

{
  "error_count": 2,
  "errors": [
    "UVM_ERROR drv.sv(88) @ 141900: response mismatch",
    "UVM_ERROR drv.sv(95) @ 142100: response mismatch"
  ],
  "log_path": "sim.log",
  "reason": "UVM_ERROR",
  "returncode": 0,
  "run_dir": "runs/axi_burst_1839274",

Read the returncode in that record: zero, and the status is FAIL. Section 3's first disagreement, resolved in favour of the log, and the record keeps the return code so a reader can see the conflict rather than having it hidden.

Three more decisions.

errors is truncated and error_count is not. Chapter 20.16's point: a run with four thousand errors must not put four thousand strings in a record that four hundred runs' worth of tooling will load. Ten lines is enough to recognise the failure, and the count is the number a report sums.

runtime_s is excluded from the JSON shown — only from the display, to keep this page's output identical on every machine. In a real record it is written; Chapter 24.6's fingerprint would exclude it from the digest for the same reason compare=False excludes it from equality.

Nothing raises. A missing log gives ((), False) and the classifier turns that into NO_FINISH with NO_END_MARKER, which is the honest answer when the evidence is absent. Chapter 25.1 Section 8's rule: a failing test is a result.

9. The Value Everything Else Consumes

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

RESULTS = [("axi_smoke", "PASS", None), ("axi_reset", "FAIL", "UVM_ERROR"),
           ("axi_burst", "FAIL", "EXIT_139"), ("axi_hang", "NO_FINISH", "TIMEOUT"),
           ("axi_cfg", "PASS", None), ("axi_lic", "ERROR", "no licence")]

counts = {}
for _, status, _ in RESULTS:
    counts[status] = counts.get(status, 0) + 1

print("summary   :", dict(sorted(counts.items())))
tested = sum(n for s, n in counts.items() if s not in ("ERROR", "COMPILE_ERROR"))
passed = counts.get("PASS", 0)
print("runs       :", len(RESULTS))
print("tested     :", tested)
print("pass rate  :", f"{passed}/{tested}")
print()
print("who is paged, by reason:")
for _, status, reason in RESULTS:
    if status == "PASS":
        continue
    who = {"UVM_ERROR": "design or testbench", "EXIT_139": "the tool, or a testbench crash",
           "TIMEOUT": "the end condition", "no licence": "infrastructure"}[reason]
    print(f"   {reason:<12} {who}")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

summary   : {'ERROR': 1, 'FAIL': 2, 'NO_FINISH': 1, 'PASS': 2}
runs       : 6
tested     : 5
pass rate  : 2/5

who is paged, by reason:
   UVM_ERROR    design or testbench
   EXIT_139     the tool, or a testbench crash
   TIMEOUT      the end condition
   no licence   infrastructure

That is what the rest of this curriculum reads. Module 26 counts them and sorts them into buckets, Module 27 produces them in parallel, and Module 30 turns them into the report somebody actually reads — and every one of those is easy exactly to the extent that this object is precise.

The pass rate is 2 of 5, not 2 of 6. Chapter 23.9 Section 7's rule, finally operating on real data: the ERROR row did not test anything, so it is excluded from the denominator. A flow that reported 2/6 would be understating the design's quality because of a licence problem.

The reason column is what makes the summary actionable. Four non-passing runs and four different people, which no amount of status counting alone could produce.

10. Predict, Run, Observe

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

from dataclasses import dataclass, field


def classify(timed_out, banner, errors, rc):
    if timed_out:
        return "NO_FINISH"
    if not banner:
        return "NO_FINISH"
    if errors:
        return "FAIL"
    if rc != 0:
        return "FAIL"
    return "PASS"


def naive(timed_out, banner, errors, rc):
    if rc == 0 and not errors:
        return "PASS"
    return "FAIL"


@dataclass(frozen=True)
class R:
    s: str
    t: float = field(default=0.0, compare=False)


# A
print("A:", classify(False, True, 0, 0))

# B
print("B:", classify(False, True, 2, 0))

# C
print("C:", classify(False, True, 0, 139))

# D
print("D:", classify(False, False, 0, 0))

# E
print("E:", naive(False, False, 0, 0))

# F
print("F:", classify(True, True, 0, 0))

# G
print("G:", R("PASS", 1.0) == R("PASS", 9.0))

# H
print("H:", len({R("PASS", 1.0), R("PASS", 9.0), R("FAIL", 1.0)}))

The verified output:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

A: PASS
B: FAIL
C: FAIL
D: NO_FINISH
E: PASS
F: NO_FINISH
G: True
H: 2

A — all three witnesses agree.

B — the log wins over a zero return code.

C — a crash with a clean log is still FAIL.

D — no end banner is NO_FINISH, even with a zero code and no errors.

E — and the naive order calls that same run PASS. Section 4.

F — a timeout overrides even a banner, because the parent imposed the limit.

G — compare=False keeps the runtime out of identity.

H — so three results are two distinct outcomes.

11. Exercises

Exercise 1 — three witnesses. For your flow, say what each of the three can and cannot tell you.

Exercise 2 — the truth table. Enumerate all eight combinations and assign a status to each.

Exercise 3 — count the conflicts. Report how many of the eight have all three agreeing.

Exercise 4 — the log wins. Construct a run that exits zero with errors in the log. Say what a return-code-only flow reports.

Exercise 5 — the clean crash. Construct the opposite and say what the reason should be.

Exercise 6 — write it naively. Implement happy-path-first and find the case where it differs.

Exercise 7 — the worst error. Explain in three sentences why a hang reported as a pass is worse than a fail reported as a pass.

Exercise 8 — state the rule. Write the precedence rule in one sentence.

Exercise 9 — fall through. Say what it buys to have no if that returns PASS.

Exercise 10 — several markers. List every end-of-simulation marker your benches emit and implement any().

Exercise 11 — two kinds of unfinished. Distinguish an inferred hang from an imposed timeout, and say which is certain.

Exercise 12 — a stale banner. Say why a timed-out run is never finished even if a banner appears in the partial log.

Exercise 13 — all five. Extend your classifier with ERROR and COMPILE_ERROR and exercise every branch.

Exercise 14 — keyword-only. Convert your classifier's parameters and say what that prevents.

Exercise 15 — reason, not status. Give two causes that share a status and distinguish them with a reason.

Exercise 16 — monotonic. Say what time.time would do to a run straddling a clock adjustment.

Exercise 17 — measured, not published. Record a runtime and assert on it without printing it.

Exercise 18 — truncate the errors. Cap the stored error lines and keep the count. Say what a 4,000-error run would otherwise do.

Exercise 19 — nothing raises. Feed your builder a missing log and confirm it returns a result.

Exercise 20 — the denominator. Compute a pass rate over a set containing one ERROR. Report both the honest and the naive figure.

Exercise 21 — who is paged. Produce a summary that maps each reason to a person.

12. Summary

Three witnesses — the return code, the log and the clock — and none is authoritative alone.
Six of the eight evidence combinations are in conflict, so classification is a decision procedure, not bookkeeping.
When the return code says zero and the log reports errors, the log wins. A UVM bench that does not $fatal exits cleanly having failed.
When the code is non-zero and the log is clean, it is still FAIL — the tool died before it could report.
The order of the checks is the design. Happy-path-first reports PASS for a run that was killed at its timeout.
Check the ways a run can be invalid before checking whether it passed, and let PASS be the fall-through with no if of its own.
finished is a property of the log, decided by any of several end markers — and overridden by a timeout, which is the certain case.
Order the branches least-valid to most-valid: ERROR, COMPILE_ERROR, NO_FINISH, FAIL, PASS.
Make every classifier parameter keyword-only; six positional booleans is unreadable and silently reorderable.
Use reason to distinguish causes within a status, keeping the vocabulary at five.
Measure with time.monotonic, which cannot go backwards, and record the duration without publishing it.
runtime_s is stored and excluded from identity via field(compare=False).
Truncate the stored error lines and keep the full count, or a four-thousand-error run poisons every tool that loads the record.
Nothing in the builder raises. A missing log is evidence of NO_FINISH, not an exception.
This object is what Modules 26 to 31 consume, and each of them is easy exactly to the extent that it is precise.
Exclude ERROR and COMPILE_ERROR from the pass-rate denominator — 2 of 5, never 2 of 6.

The last chapter of this module runs one test a hundred times before anybody runs a hundred tests once.

Where this fits

Part of the Python curriculum.

1. Three Witnesses

2. The Annotation

3. How Often the Witnesses Disagree

4. The Order of the Checks Is the Design

5. Where finished Comes From

6. The Classifier

7. Measuring Without Publishing

8. One Result, Assembled

9. The Value Everything Else Consumes

10. Predict, Run, Observe

11. Exercises

12. Summary

Where this fits

5. Where `finished` Comes From