Classifying a Run: Pass, Fail, Timeout, Never Started

Part II · Module 20 · Chapter 20.11 · Classifying a Run: Pass, Fail, Timeout, Never Started

1. Four Logs

UVM_INFO @ 0: reporter [RNTST] Running test axi_reset_test...
UVM_INFO scb.sv(120) @ 160000: uvm_test_top.env.scb [CHK] 96 compared
--- UVM Report Summary ---
UVM_ERROR :    0
UVM_FATAL :    0
TEST DONE
Simulation complete via $finish(1) at time 200000 NS

UVM_INFO @ 0: reporter [RNTST] Running test axi_smoke_test...
UVM_ERROR drv.sv(88) @ 141900: uvm_test_top.env.agt.drv [WR] bad handshake
--- UVM Report Summary ---
UVM_ERROR :    1
UVM_FATAL :    0
TEST DONE
Simulation complete via $finish(1) at time 200000 NS

UVM_INFO @ 0: reporter [RNTST] Running test axi_burst_test...
UVM_INFO drv.sv(60) @ 141700: uvm_test_top.env.agt.drv [WR] issuing write
UVM_INFO drv.sv(60) @ 141800: uvm_test_top.env.agt.drv [WR] waiting for resp

xcelium> run
xmelab: *E,CUVWSP: module worklib.tb_top not found
xcelium: *F,NOELAB: elaboration failed

import re

UVM_FAIL = re.compile(r"^UVM_(ERROR|FATAL)\b.*@\s+\d+:")


def count_failures(path):
    errors = fatals = 0
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            m = UVM_FAIL.match(line)
            if m is None:
                continue
            if m.group(1) == "ERROR":
                errors += 1
            else:
                fatals += 1
    return errors, fatals


print(f"{'log':<12} {'errors':>7} {'fatals':>7}   two-outcome verdict")
for path in ["pass.log", "fail.log", "hang.log", "norun.log"]:
    errors, fatals = count_failures(path)
    verdict = "PASS" if errors == 0 and fatals == 0 else "FAIL"
    print(f"{path:<12} {errors:>7} {fatals:>7}   {verdict}")

log           errors  fatals   two-outcome verdict
pass.log           0       0   PASS
fail.log           1       0   FAIL
hang.log           0       0   PASS
norun.log          0       0   PASS

Three passes out of four, and one of these runs passed.

hang.log is a test that stopped producing output and was killed by the job scheduler hours later. norun.log is a simulation that never started — elaboration failed because a module was missing, so the testbench was never built, let alone run.

Both have zero errors. Both satisfy "no errors means pass" perfectly, and they satisfy it for the same reason: a run that produced no errors because it produced nothing at all is indistinguishable, by error count, from a run that produced no errors because everything worked.

2. Four Outcomes

The two on the bottom row are not shades of failure. They are different questions, and the reason to keep them apart is that they send a person somewhere different:

• FAIL — the design or the testbench is wrong. Open the first failure and its context.
• NO_FINISH — the test hung, or the scheduler killed it. Look at the last thing it was doing, and at the timeout.
• NO_RUN — nothing ran. Look at the compile and elaborate step; the log you are reading is not a simulation log at all.

Collapsing NO_RUN into FAIL sends an engineer to debug RTL for a missing file in a filelist. Collapsing either into PASS means nobody goes anywhere.

3. The Evidence for Each

Each outcome rests on something present or absent in the log:

Started is the UVM report server's RNTST banner — the line that says Running test <name>. It is written once the testbench has been built and the test object created, so its presence means a test genuinely began.

Finished is an end-of-simulation marker. There are several spellings — TEST DONE, Simulation complete, $finish — and like the seed in Chapter 20.8 they are tool- and testbench-specific, so a list is the right shape.

Note what is not in that table: the error count plays no part in distinguishing the bottom two rows. A hung test may well have logged errors before it hung, and it is still NO_FINISH.

4. The Classifier

UVM_INFO @ 0: reporter [RNTST] Running test axi_reset_test...
UVM_INFO scb.sv(120) @ 160000: uvm_test_top.env.scb [CHK] 96 compared
TEST DONE
Simulation complete via $finish(1) at time 200000 NS

UVM_INFO @ 0: reporter [RNTST] Running test axi_smoke_test...
UVM_ERROR drv.sv(88) @ 141900: uvm_test_top.env.agt.drv [WR] bad handshake
TEST DONE
Simulation complete via $finish(1) at time 200000 NS

UVM_INFO @ 0: reporter [RNTST] Running test axi_burst_test...
UVM_INFO drv.sv(60) @ 141700: uvm_test_top.env.agt.drv [WR] issuing write
UVM_INFO drv.sv(60) @ 141800: uvm_test_top.env.agt.drv [WR] waiting for resp

xcelium> run
xmelab: *E,CUVWSP: module worklib.tb_top not found
xcelium: *F,NOELAB: elaboration failed

import re

UVM_FAIL = re.compile(r"^UVM_(?P<sev>ERROR|FATAL)\b.*@\s+\d+:")
STARTED = re.compile(r"\[RNTST\] Running test (?P<test>\w+)")
FINISHED = re.compile(r"TEST DONE|Simulation complete|\$finish")


def scan(path):
    """One pass. Facts only — no verdict."""
    facts = {"errors": 0, "fatals": 0, "started": False,
             "finished": False, "test": None}
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            m = UVM_FAIL.match(line)
            if m is not None:
                facts["errors" if m["sev"] == "ERROR" else "fatals"] += 1
            s = STARTED.search(line)
            if s is not None:
                facts["started"] = True
                facts["test"] = s["test"]
            if FINISHED.search(line):
                facts["finished"] = True
    return facts


def classify(facts):
    """Policy. One of PASS, FAIL, NO_FINISH, NO_RUN."""
    if not facts["started"]:
        return "NO_RUN"
    if not facts["finished"]:
        return "NO_FINISH"
    return "FAIL" if (facts["errors"] or facts["fatals"]) else "PASS"


print(f"{'log':<12} {'status':<10} {'err':>4} {'fat':>4} {'started':>8} {'finished':>9}  test")
for path in ["pass.log", "fail.log", "hang.log", "norun.log"]:
    facts = scan(path)
    print(f"{path:<12} {classify(facts):<10} {facts['errors']:>4} {facts['fatals']:>4} "
          f"{str(facts['started']):>8} {str(facts['finished']):>9}  {facts['test']}")

log          status      err  fat  started  finished  test
pass.log     PASS          0    0     True      True  axi_reset_test
fail.log     FAIL          1    0     True      True  axi_smoke_test
hang.log     NO_FINISH     0    0     True     False  axi_burst_test
norun.log    NO_RUN        0    0    False     False  None

Four logs, four different answers, and each one tells somebody where to look.

The split between scan and classify is Chapter 20.1 Section 4's, and here it earns something specific: classify takes facts, not a path. It never opens a file, so the four-row table above is also its entire test suite — four dicts and four expected strings, no logs required. Chapter 19.18's sample-list testing applies directly.

And classify contains the canonical annotation almost verbatim:

if not finished:
    return "NO_FINISH"
return "PASS" if not (errors or fatals) else "FAIL"

with NO_RUN added in front of it. That addition is this chapter's one departure from the canonical sketch, and it is deliberate — the sketch covers three of the four outcomes, and the fourth is the one that produces the worst report, because a log from a simulation that never started is not a simulation log.

5. The Order of the Checks

started, then finished, then failures. That order is not stylistic:

FACTS = {"errors": 3, "fatals": 1, "started": False, "finished": False}


def right_order(f):
    if not f["started"]:
        return "NO_RUN"
    if not f["finished"]:
        return "NO_FINISH"
    return "FAIL" if (f["errors"] or f["fatals"]) else "PASS"


def wrong_order(f):
    if f["errors"] or f["fatals"]:
        return "FAIL"
    if not f["started"]:
        return "NO_RUN"
    if not f["finished"]:
        return "NO_FINISH"
    return "PASS"


print("right order:", right_order(FACTS))
print("wrong order:", wrong_order(FACTS))

right order: NO_RUN
wrong order: FAIL

A log that never started a test, with three errors in it — which is exactly what an elaboration failure looks like, because the tool's own *E, messages can be mistaken for errors by a loose pattern.

wrong_order reports FAIL, which sounds harmless and is not: it says the test failed, so somebody opens the waveform for a test that was never built. right_order reports NO_RUN, and they go and look at the filelist.

Each earlier question makes the later ones meaningless rather than merely false. "How many errors did it have" is not a slightly-wrong question to ask of a run that never began; it is a question with no answer, and answering it anyway is how a report acquires numbers that mean nothing.

6. Four Statuses, One Summary

Over a sweep, the four outcomes are what makes the summary honest:

RUNS = [
    {"test": "axi_reset_test", "status": "PASS"},
    {"test": "axi_smoke_test", "status": "FAIL"},
    {"test": "axi_burst_test", "status": "NO_FINISH"},
    {"test": "axi_wrap_test", "status": "PASS"},
    {"test": None, "status": "NO_RUN"},
    {"test": "axi_excl_test", "status": "FAIL"},
]

ORDER = ["PASS", "FAIL", "NO_FINISH", "NO_RUN"]

counts = dict.fromkeys(ORDER, 0)
for run in RUNS:
    counts[run["status"]] += 1

total = len(RUNS)
for status in ORDER:
    print(f"{status:<10} {counts[status]:>3}")
print(f"{'total':<10} {total:>3}")
print()
print("clean          :", counts["PASS"] == total)
print("needs RTL debug:", counts["FAIL"])
print("needs flow work:", counts["NO_FINISH"] + counts["NO_RUN"])

PASS         2
FAIL         2
NO_FINISH    1
NO_RUN       1
total        6

clean          : False
needs RTL debug: 2
needs flow work: 2

Two rows at the bottom that a two-outcome report cannot produce. needs flow work: 2 is the number that gets a regression fixed, and in a two-outcome world those two runs are indistinguishable from the two genuine passes — the summary would read "4 passed, 2 failed" and the four would include a hang and a build failure.

dict.fromkeys(ORDER, 0) again, so a sweep with no hangs prints NO_FINISH 0 rather than omitting the row. A missing row in a summary reads as "not applicable"; a zero reads as "checked, none found".

7. What the Log Cannot Tell You

Honest limits, because three of the four outcomes are inferred from absence, and absence has more than one cause:

A log with no end-of-simulation banner might be a hang, a test killed by the scheduler's time limit, a job killed for memory, a full disk, an NFS write that never landed, or a file somebody copied while it was still being written. All of them look identical.

A log with no test banner might be an elaboration failure, or a simulation whose output was redirected somewhere else, or an empty file created by a wrapper script that then died.

An empty file is NO_RUN by these rules, and that is the right default — but it tells you nothing about why.

The parser should not guess between those, and the fix is not a cleverer pattern. It is more evidence, from outside the log: the runner's own exit status, the wall-clock time the job took, and whether the scheduler reported a kill. A test that ran for exactly the timeout is a hang; one that ran for nine seconds and stopped is something else. That evidence belongs to whatever launched the simulation, and Chapter 20.12 is about combining it with what the log says.

8. Predict, Run, Observe

def classify(f):
    if not f["started"]:
        return "NO_RUN"
    if not f["finished"]:
        return "NO_FINISH"
    return "FAIL" if (f["errors"] or f["fatals"]) else "PASS"


def facts(started, finished, errors=0, fatals=0):
    return {"started": started, "finished": finished, "errors": errors, "fatals": fatals}


# A
print("A:", classify(facts(True, True)))

# B
print("B:", classify(facts(True, True, errors=1)))

# C
print("C:", classify(facts(True, False)))

# D
print("D:", classify(facts(True, False, errors=40)))

# E
print("E:", classify(facts(False, False, errors=3)))

# F
print("F:", classify(facts(False, True)))

# G
print("G:", classify(facts(True, True, fatals=1)))

The verified output:

A: PASS
B: FAIL
C: NO_FINISH
D: NO_FINISH
E: NO_RUN
F: NO_RUN
G: FAIL

A and B — the two outcomes a simple verdict gets right.

C and D — a hang is NO_FINISH whether it logged forty errors first or none. The status says where to look; the count is still in the record.

E — never started, with three errors. NO_RUN beats FAIL, which is Section 5's ordering.

F — finished but never started is a contradiction the data can produce — an end-of-sim marker matched in a tool banner, say — and the classifier handles it rather than being surprised. NO_RUN is the right call: without a test banner there is no test.

G — a fatal is a failure like an error, which is why the test is errors or fatals and not just errors.

9. Exercises

Exercise 1 — three passes. Build four logs matching Section 1 and run a two-outcome verdict over them. Record how many it calls PASS.

Exercise 2 — why both. Explain in one sentence the property the hang and the never-started run share that makes a two-outcome verdict pass them both.

Exercise 3 — where each one sends you. For each of the four outcomes, write the one-line instruction you would put next to it in a summary.

Exercise 4 — the evidence table. For a log of your own, identify the started marker and every end-of-simulation spelling your flow produces.

Exercise 5 — facts and policy. Write scan and classify separately. Confirm classify never opens a file.

Exercise 6 — the test suite you got free. Test classify with eight dicts and no logs at all. Include a case for each outcome and one contradiction.

Exercise 7 — the wrong order. Put the failure check first and run it on facts with started=False and three errors. Record the status and say who gets sent where.

Exercise 8 — keep the counts. Produce two NO_FINISH records, one with 40 errors and one with none. Say what each suggests.

Exercise 9 — the summary. Count a sweep by status with dict.fromkeys. Print a zero row and say why a zero reads differently from a missing row.

Exercise 10 — the flow number. Add needs flow work to your summary and say what a two-outcome report would have shown instead.

Exercise 11 — absence has causes. List four distinct things that produce a log with no end-of-simulation banner. Say which piece of evidence outside the log would separate them.

Exercise 12 — observation, not conclusion. Argue in three sentences for or against renaming NO_FINISH to TIMEOUT.

10. Summary

• Two outcomes are not enough. Over four logs with one clean run, a two-outcome verdict reports three passes.
• A hang and a never-started run both have zero errors, because a run that produced nothing produced no errors.
• Four outcomes: PASS, FAIL, NO_FINISH, NO_RUN.
• They send people to different places — the design, the testbench and timeout, or the compile and elaborate step.
• Started is the UVM RNTST banner; finished is an end-of-simulation marker with several tool-specific spellings, so a list.
• The error count plays no part in distinguishing the bottom two rows. A hung test that logged errors first is still NO_FINISH.
• classify takes facts, not a path, so a handful of dicts is its whole test suite.
• The canonical sketch covers three outcomes; NO_RUN is added in front because a log from a simulation that never started is not a simulation log.
• Order the checks started, finished, failures. Each earlier state makes the later questions meaningless rather than merely false.
• With the order reversed, an elaboration failure reports FAIL and somebody opens a waveform for a test that was never built.
• Keep the counts on every record, including NO_RUN and NO_FINISH — the status says where to look, the counts say what you will find.
• needs flow work is the number a two-outcome report cannot produce, and it is the one that gets a regression fixed.
• Three of the four outcomes are inferred from absence, and absence is also a full disk, an NFS stall, a killed job or a part-copied file.
• The fix is more evidence, not a cleverer pattern — the runner's exit status and wall clock.
• NO_FINISH says what was observed; TIMEOUT is a conclusion and is wrong for the full-disk case.

The next chapter brings in that outside evidence, and works out how much of a hang a log can diagnose on its own.