Python Part II · Module 35 · Industry VLSI Automation Projects

A Simulation Runner

This is the tool every other tool in the module depends on, which is why it is the one marked critical: a regression manager is only as honest as the single-run command underneath it. The command itself is four lines of shell. What makes it a tool rather than a shortcut is that both of the signals a shell one-liner naturally trusts turn out to be wrong, and both are measured here against a real simulator rather than argued about.

Advanced27 min readPythonPart IIAutomationSimulationExit Codes

Part II · Module 35 · Chapter 35.3 · A Simulation Runner

1. One Command, Five Outcomes

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

OUTCOMES = [
    ("the design is fine", "PASS", "0", "nothing to do"),
    ("the design is wrong", "FAIL", "1", "a design bug"),
    ("the testbench died", "FATAL", "1", "a design or env bug"),
    ("it never finished", "TIMEOUT", "1", "a design or env bug"),
    ("it never compiled", "BUILD", "3", "a source or filelist bug"),
]

print(f"{'what happened':<24} {'status':<9} {'exit':<6} who it belongs to")
for what, status, code, who in OUTCOMES:
    print(f"{what:<24} {status:<9} {code:<6} {who}")

print()
print("outcomes a runner must distinguish:", len(OUTCOMES))
print("outcomes `vvp` distinguishes       :", 2)
print("so the classification comes from   :", "the log, not the tool")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

what happened            status    exit   who it belongs to
the design is fine       PASS      0      nothing to do
the design is wrong      FAIL      1      a design bug
the testbench died       FATAL     1      a design or env bug
it never finished        TIMEOUT   1      a design or env bug
it never compiled        BUILD     3      a source or filelist bug

outcomes a runner must distinguish: 5
outcomes `vvp` distinguishes       : 2
so the classification comes from   : the log, not the tool

Five outcomes, and the last column is why they must be distinguished rather than collapsed into pass and fail. BUILD is somebody's broken commit, TIMEOUT is usually a hang in the design, and Chapter 30.1 Section 4's design-versus- infrastructure split needs all five to survive into the report.

2. The Annotation

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

run_sim.py --test axi_burst --seed 48210 --waves

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

FLAGS = [
    ("--test axi_burst", "which test", "required", "names the run directory"),
    ("--seed 48210", "which stimulus", "required", "names the run directory"),
    ("--waves", "dump a VCD", "optional", "off by default — §9"),
]

print(f"{'flag':<20} {'decides':<16} {'required':<10} note")
for flag, decides, req, note in FLAGS:
    print(f"{flag:<20} {decides:<16} {req:<10} {note}")

print()
print("what the command prints  :", "one line, the verdict")
print("what the command returns :", "an exit code the flow branches on")
print("what it writes            :", "run_<test>_<seed>/ with log, cmd.sh, vcd")
print()
print("why --seed is required and not defaulted:",
      "a default seed makes every run the same run")
print("why --waves is off by default           :",
      "a VCD per run fills a volume — 33.13 §7")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

flag                 decides          required   note
--test axi_burst     which test       required   names the run directory
--seed 48210         which stimulus   required   names the run directory
--waves              dump a VCD       optional   off by default — §9

what the command prints  : one line, the verdict
what the command returns : an exit code the flow branches on
what it writes            : run_<test>_<seed>/ with log, cmd.sh, vcd

why --seed is required and not defaulted: a default seed makes every run the same run
why --waves is off by default           : a VCD per run fills a volume — 33.13 §7

--seed has no default on purpose. A runner that picks a seed when none is given produces runs that cannot be reproduced from the command line alone, which is Chapter 33.14's undeclared input; a runner that defaults to a fixed seed produces four hundred identical runs in a regression, which is worse because it looks like coverage.

3. The First Version

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import shutil
import subprocess
from pathlib import Path

print("iverilog present:", shutil.which("iverilog") is not None)
print("vvp present     :", shutil.which("vvp") is not None)
print()

Path("tb.v").write_text(
    "module tb;\n"
    "  initial begin\n"
    '    $display("UVM_INFO @0: tb.sv(1) start");\n'
    '    $display("TEST DONE");\n'
    "    $finish;\n"
    "  end\n"
    "endmodule\n", encoding="utf-8")

# v1, as a shell pipeline would do it
build = subprocess.run(["iverilog", "-o", "sim.vvp", "tb.v"],
                       stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                       text=True)
run = subprocess.run(["vvp", "sim.vvp"], stdout=subprocess.PIPE,
                     stderr=subprocess.STDOUT, text=True)
Path("sim.log").write_text(run.stdout, encoding="utf-8")

print("v1 — iverilog && vvp && grep")
print("  iverilog rc :", build.returncode)
print("  vvp rc      :", run.returncode)
print("  log lines   :", len(run.stdout.strip().splitlines()))
print("  verdict     :", "PASS" if run.returncode == 0 else "FAIL")
print()
print("on a passing test this is:", "correct")
print("and that is the problem   :", "it is correct for the wrong reason")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

iverilog present: True
vvp present     : True

v1 — iverilog && vvp && grep
  iverilog rc : 0
  vvp rc      : 0
  log lines   : 3
  verdict     : PASS

on a passing test this is: correct
and that is the problem   : it is correct for the wrong reason

The verdict is right and the reasoning is not. vvp returned 0 and the test passed, so the exit code looks like the verdict — and a one-liner built on that observation works perfectly until the first failing run.

4. A Failing Test Exits Zero

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import subprocess
from pathlib import Path

CASES = {
    "pass": ('    $display("UVM_INFO @0: tb.sv(1) start");\n'
             '    $display("TEST DONE");\n'
             "    $finish;\n"),
    "fail": ('    $display("UVM_INFO @0: tb.sv(1) start");\n'
             '    $display("UVM_ERROR @10: drv.sv(88) data mismatch");\n'
             '    $display("TEST FAILED");\n'
             "    $finish;\n"),
    "fatal": ('    $display("UVM_INFO @0: tb.sv(1) start");\n'
              '    $display("UVM_FATAL @5: env.sv(3) no interface");\n'
              '    $fatal(2, "giving up");\n'),
    "timeout": ('    $display("UVM_INFO @0: tb.sv(1) start");\n'
                "    #1000;\n"
                '    $display("no TEST DONE line here");\n'
                "    $finish;\n"),
}

for name, body in CASES.items():
    Path(f"tb_{name}.v").write_text(
        f"module tb;\n  initial begin\n{body}  end\nendmodule\n",
        encoding="utf-8")

Path("tb_syntax.v").write_text(
    'module tb;\n  initial begin\n    $display("oops"\n  end\nendmodule\n',
    encoding="utf-8")

print(f"{'testbench':<10} {'iverilog':>9} {'built':>6} {'vvp rc':>7} "
      f"{'errors':>7} {'fatals':>7}")
for name in list(CASES) + ["syntax"]:
    b = subprocess.run(["iverilog", "-o", f"{name}.vvp", f"tb_{name}.v"],
                       stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                       text=True)
    built = b.returncode == 0
    if not built:
        print(f"{name:<10} {b.returncode:>9} {str(built):>6} {'-':>7} "
              f"{'-':>7} {'-':>7}")
        continue
    r = subprocess.run(["vvp", f"{name}.vvp"], stdout=subprocess.PIPE,
                       stderr=subprocess.STDOUT, text=True)
    errs = sum(1 for l in r.stdout.splitlines() if l.startswith("UVM_ERROR"))
    fat = sum(1 for l in r.stdout.splitlines() if l.startswith("UVM_FATAL"))
    print(f"{name:<10} {b.returncode:>9} {str(built):>6} {r.returncode:>7} "
          f"{errs:>7} {fat:>7}")

print()
print("the FAIL row's vvp rc is:", 0)
print("and it printed           :", "UVM_ERROR and TEST FAILED")
print("so the exit code is      :", "not the verdict — 32.6")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

testbench   iverilog  built  vvp rc  errors  fatals
pass               0   True       0       0       0
fail               0   True       0       1       0
fatal              0   True       1       0       1
timeout            0   True       0       0       0
syntax             2  False       -       -       -

the FAIL row's vvp rc is: 0
and it printed           : UVM_ERROR and TEST FAILED
so the exit code is      : not the verdict — 32.6

The fail row returns 0 from a run that printed UVM_ERROR and TEST FAILED. That is not a quirk of this testbench: $finish is the normal end of a simulation, and a Verilog testbench that detects a mismatch and then calls $finish has, from the simulator's point of view, completed successfully. Chapter 32.6 argued this abstractly; here it is one row of a table.

Three other rows carry information a runner needs:

$fatal returns 1. So a non-zero vvp status does mean something went wrong — it just does not mean the design is wrong, and a zero status does not mean it is right. The exit code is a crash detector, not a verdict.

A syntax error returns 2 from iverilog and builds no artifact. That has to be a distinct outcome because nothing ran: there is no log to classify, and reporting it as a design failure sends it to the wrong person.

The timeout row looks exactly like pass — rc 0, no errors, no fatals — and Section 5 is how you tell them apart.

5. A Substring Says PASS

The timeout and pass runs are distinguished only by whether the log contains the completion marker, and the obvious way to check that is wrong:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

LINES = [
    "TEST DONE",
    "no TEST DONE line here",
    "  TEST DONE  ",
    "TEST DONE_EARLY",
    "the TEST DONE marker was not printed",
]

print(f"{'line':<40} {'substring':>10} {'whole line':>11}  agree")
for line in LINES:
    sub = "TEST DONE" in line
    whole = line.strip() == "TEST DONE"
    print(f"{line!r:<40} {str(sub):>10} {str(whole):>11}  {sub == whole}")

disagree = sum(1 for l in LINES
               if ("TEST DONE" in l) != (l.strip() == "TEST DONE"))
print()
print("candidate lines              :", len(LINES))
print("where the two checks disagree:", disagree)
print("and the substring check is   :", "wrong on every one of them")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

line                                      substring  whole line  agree
'TEST DONE'                                    True        True  True
'no TEST DONE line here'                       True       False  False
'  TEST DONE  '                                True        True  True
'TEST DONE_EARLY'                              True       False  False
'the TEST DONE marker was not printed'         True       False  False

candidate lines              : 5
where the two checks disagree: 3
and the substring check is   : wrong on every one of them

Three of five lines fool a substring check, and the second one is not hypothetical — it is the line the timeout testbench in Section 4 actually prints. A log saying "no TEST DONE line here" contains the string TEST DONE, so if "TEST DONE" in text reports that the test completed.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import subprocess
from pathlib import Path

Path("tb_timeout.v").write_text(
    "module tb;\n  initial begin\n"
    '    $display("UVM_INFO @0: tb.sv(1) start");\n'
    "    #1000;\n"
    '    $display("no TEST DONE line here");\n'
    "    $finish;\n  end\nendmodule\n", encoding="utf-8")
Path("tb_pass.v").write_text(
    "module tb;\n  initial begin\n"
    '    $display("UVM_INFO @0: tb.sv(1) start");\n'
    '    $display("TEST DONE");\n'
    "    $finish;\n  end\nendmodule\n", encoding="utf-8")


def sim(name):
    subprocess.run(["iverilog", "-o", f"{name}.vvp", f"tb_{name}.v"],
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    r = subprocess.run(["vvp", f"{name}.vvp"], stdout=subprocess.PIPE,
                       stderr=subprocess.STDOUT, text=True)
    return r.returncode, r.stdout


print(f"{'run':<10} {'rc':>3} {'substring -> status':<22} anchored -> status")
for name in ("pass", "timeout"):
    rc, text = sim(name)
    sub = "TEST DONE" in text
    anchored = any(l.strip() == "TEST DONE" for l in text.splitlines())
    print(f"{name:<10} {rc:>3} {('PASS' if sub else 'TIMEOUT'):<22} "
          f"{'PASS' if anchored else 'TIMEOUT'}")

print()
print("the timeout run, by substring:", "PASS")
print("the timeout run, anchored    :", "TIMEOUT")
print("both runs' exit codes        :", 0)
print("so the completion check is   :", "the only thing separating them")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

run         rc substring -> status    anchored -> status
pass         0 PASS                   PASS
timeout      0 PASS                   TIMEOUT

the timeout run, by substring: PASS
the timeout run, anchored    : TIMEOUT
both runs' exit codes        : 0
so the completion check is   : the only thing separating them

The timeout run is reported as PASS by the substring check. Two runs with identical exit codes, identical error counts and opposite verdicts, and the only thing that tells them apart is whether the completion marker is matched as a whole line.

That is Chapter 24.3's severity-prefix hazard moved to the completion marker, and it is worse here because there is no second signal to catch it: the exit code says 0 for both, so a wrong completion check is the end of the chain.

6. The Architecture

Seven stages, and the verdict comes from the log

Stages 2 and 4 are the only ones that can produce a non-zero status from the tool itself, and neither of them is the verdict: a build failure means nothing ran, and a non-zero simulation status means the process died rather than that the design is wrong. Stage 5 is where the verdict is decided, from the log text.

Step 2 is placed before step 3 deliberately. Chapter 33.13 Section 8 measured that cmd.sh is the irreplaceable artefact precisely because it survives the run that was killed, and a compile failure is the most likely way for a run to end before anything else is written.

7. Compile and Run Are Separate Outcomes

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import subprocess
from pathlib import Path
from dataclasses import dataclass

EXIT = {"PASS": 0, "FAIL": 1, "FATAL": 1, "TIMEOUT": 1, "BUILD": 3}


@dataclass(frozen=True)
class Result:
    test: str
    seed: int
    status: str
    errors: int
    fatals: int
    built: bool

    @property
    def exit_code(self):
        return EXIT[self.status]


def compile_sources(sources, out, cwd):
    """Returns (ok, text). A failure here means NOTHING ran."""
    cp = subprocess.run(["iverilog", "-o", out, *sources], cwd=cwd,
                        stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                        text=True)
    return cp.returncode == 0, cp.stdout


Path("good.v").write_text(
    'module tb;\n  initial begin\n    $display("TEST DONE");\n'
    "    $finish;\n  end\nendmodule\n", encoding="utf-8")
Path("bad.v").write_text(
    'module tb;\n  initial begin\n    $display("oops"\n  end\nendmodule\n',
    encoding="utf-8")

for label, src in (("a clean source", "good.v"), ("a syntax error", "bad.v")):
    ok, text = compile_sources([src], "x.vvp", ".")
    first = text.strip().splitlines()[0] if text.strip() else "(no output)"
    print(f"{label:<16} built={str(ok):<6} first line of output: {first}")

print()
print("a build failure gives a log to classify:", False)
print("so BUILD must be                       :", "its own status")
print("and its exit code is                   :", EXIT["BUILD"])
print("which 33.8 reserved for                :", "an input problem")
print()
print("reporting a build failure as FAIL would:",
      "send a filelist bug to a designer")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

a clean source   built=True   first line of output: (no output)
a syntax error   built=False  first line of output: bad.v:4: syntax error

a build failure gives a log to classify: False
so BUILD must be                       : its own status
and its exit code is                   : 3
which 33.8 reserved for                : an input problem

reporting a build failure as FAIL would: send a filelist bug to a designer

A clean compile prints nothing and a syntax error names the line. The important asymmetry is in the next block of output: a build failure leaves no log to classify, so it cannot be squeezed into the same code path as a simulation result, and Chapter 33.8's code 3 — an input problem — is where it belongs.

Note iverilog prints its diagnostics and then I give up., so the first line of output is the useful one and the last is noise. A runner should keep the whole compile log in the run directory and quote only the first line in its one-line verdict.

8. Classifying From the Log

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

DONE_MARKER = "TEST DONE"


def classify(log_text, sim_rc, built):
    """The verdict. Reads the LOG; uses rc only to detect a crash."""
    if not built:
        return "BUILD"
    lines = log_text.splitlines()
    errors = sum(1 for l in lines if l.startswith("UVM_ERROR")
                 and "UVM_ERROR :" not in l)
    fatals = sum(1 for l in lines if l.startswith("UVM_FATAL")
                 and "UVM_FATAL :" not in l)
    completed = any(l.strip() == DONE_MARKER for l in lines)
    if fatals or sim_rc != 0:
        return "FATAL"
    if not completed:
        return "TIMEOUT"
    if errors:
        return "FAIL"
    return "PASS"


CASES = [
    ("clean", "UVM_INFO start\nTEST DONE", 0, True, "PASS"),
    ("errors then done", "UVM_ERROR x\nTEST DONE", 0, True, "FAIL"),
    ("fatal", "UVM_FATAL y", 1, True, "FATAL"),
    ("no marker", "UVM_INFO start\nno TEST DONE line here", 0, True,
     "TIMEOUT"),
    ("marker as substring", "the TEST DONE marker was not printed", 0, True,
     "TIMEOUT"),
    ("did not compile", "", None, False, "BUILD"),
    ("summary line only", "UVM_ERROR :    0\nTEST DONE", 0, True, "PASS"),
    ("crash, rc non-zero", "UVM_INFO start\nTEST DONE", 1, True, "FATAL"),
]

print(f"{'case':<22} {'rc':>5} {'built':<6} {'got':<9} expected  ok")
ok_all = True
for label, text, rc, built, expected in CASES:
    got = classify(text, rc if rc is not None else 0, built)
    ok = got == expected
    ok_all &= ok
    print(f"{label:<22} {str(rc):>5} {str(built):<6} {got:<9} {expected:<9} "
          f"{ok}")

print()
print("cases            :", len(CASES))
print("all as expected  :", ok_all)
print()
print("the two that need the anchored check:", 2)
print("the one that needs the 24.3 guard   :", 1)
print("the one where rc overrides the log  :", 1)

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

case                      rc built  got       expected  ok
clean                      0 True   PASS      PASS      True
errors then done           0 True   FAIL      FAIL      True
fatal                      1 True   FATAL     FATAL     True
no marker                  0 True   TIMEOUT   TIMEOUT   True
marker as substring        0 True   TIMEOUT   TIMEOUT   True
did not compile         None False  BUILD     BUILD     True
summary line only          0 True   PASS      PASS      True
crash, rc non-zero         1 True   FATAL     FATAL     True

cases            : 8
all as expected  : True

the two that need the anchored check: 2
the one that needs the 24.3 guard   : 1
the one where rc overrides the log  : 1

Eight cases, all classified as intended. Three of the rows exist because of earlier chapters and are worth naming.

marker as substring is Section 5: the log says the marker was not printed, and only the anchored check gets TIMEOUT.

summary line only is Chapter 24.3: UVM_ERROR : 0 is a severity line by prefix and must not be counted, or a clean run reports FAIL.

crash, rc non-zero is the one place the exit code wins. A log that looks complete with a non-zero status means the process died after printing the marker — a segfault in a PLI, a disk error on close — and calling that PASS would be trusting a log the simulator did not finish writing. The order of the checks encodes that: fatals or sim_rc != 0 is tested first.

9. Waves, Seeds and the Run Directory

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import subprocess
from pathlib import Path

TB = """module tb;
  reg clk = 0;
  reg [7:0] count = 0;
  initial begin
`ifdef WAVES
    $dumpfile("waves.vcd");
    $dumpvars(0, tb);
`endif
    $display("UVM_INFO @0: tb.sv(1) start seed=%0d", `SEED);
    repeat (8) begin #5 clk = ~clk; end
    $display("TEST DONE");
    $finish;
  end
  always @(posedge clk) count <= count + 1;
endmodule
"""


def run_one(test, seed, waves):
    """One isolated run directory, named from the test and the seed."""
    run_dir = Path(f"run_{test}_{seed}")
    run_dir.mkdir(parents=True, exist_ok=True)
    (run_dir / "tb.v").write_text(TB, encoding="utf-8")

    argv = ["iverilog", f"-DSEED={seed}"]
    if waves:
        argv.append("-DWAVES")
    argv += ["-o", "sim.vvp", "tb.v"]

    # cmd.sh FIRST — 33.13 §8
    (run_dir / "cmd.sh").write_text(
        "#!/bin/sh\n" + " ".join(argv) + "\nvvp sim.vvp\n", encoding="utf-8")

    b = subprocess.run(argv, cwd=run_dir, stdout=subprocess.PIPE,
                       stderr=subprocess.STDOUT, text=True)
    if b.returncode != 0:
        return run_dir, "BUILD", b.stdout
    r = subprocess.run(["vvp", "sim.vvp"], cwd=run_dir,
                       stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                       text=True)
    (run_dir / "sim.log").write_text(r.stdout, encoding="utf-8")
    return run_dir, "ran", r.stdout


for waves in (False, True):
    run_dir, state, log = run_one("axi_burst", 48210, waves)
    vcd = run_dir / "waves.vcd"
    print(f"--- --waves {'on ' if waves else 'off'}")
    print("    state           :", state)
    print("    seed in the log :", "seed=48210" in log)
    print("    files kept      :",
          sorted(p.name for p in run_dir.iterdir() if p.is_file()))
    print("    vcd exists      :", vcd.is_file())
    if vcd.is_file():
        data = vcd.read_bytes()
        print("    vcd > 100 bytes :", len(data) > 100)
        print("    has $enddefinitions:", b"$enddefinitions" in data)
        print("    has a $date header :", b"$date" in data)

print()
print("the run directory name encodes:", "the test and the seed")
print("so two seeds of one test       :", "cannot collide — 33.14 §7")
print("the VCD's bytes are not shown  :",
      "it carries a $date header, so it is not reproducible")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

--- --waves off
    state           : ran
    seed in the log : True
    files kept      : ['cmd.sh', 'sim.log', 'sim.vvp', 'tb.v']
    vcd exists      : False
--- --waves on 
    state           : ran
    seed in the log : True
    files kept      : ['cmd.sh', 'sim.log', 'sim.vvp', 'tb.v', 'waves.vcd']
    vcd exists      : True
    vcd > 100 bytes : True
    has $enddefinitions: True
    has a $date header : True

the run directory name encodes: the test and the seed
so two seeds of one test       : cannot collide — 33.14 §7
the VCD's bytes are not shown  : it carries a $date header, so it is not reproducible

The VCD appears only with --waves, and the seed reaches the log. Two details are worth keeping.

The seed is passed as a compile-time define here because that is what Icarus offers; a commercial simulator takes it as a plus-arg at run time, which is better because it means one compile serves every seed. Either way the seed must appear in the log, so that a log found later can be traced back to the command that produced it without consulting a database.

The last line is the determinism rule from Chapter 33.14 applied to this chapter's own output. A VCD contains a $date header, so two runs of the same simulation produce files that differ — which means a VCD can never be a golden fixture, and only facts about it (does it exist, is it non-trivial, does it have an $enddefinitions section) are safe to assert.

10. The Runner, Assembled

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path

EXIT = {"PASS": 0, "FAIL": 1, "FATAL": 1, "TIMEOUT": 1, "BUILD": 3,
        "USAGE": 2}
DONE_MARKER = "TEST DONE"

TB = """module tb;
  reg clk = 0;
  initial begin
`ifdef WAVES
    $dumpfile("waves.vcd");
    $dumpvars(0, tb);
`endif
    $display("UVM_INFO @0: tb.sv(1) start seed=%0d", `SEED);
`ifdef INJECT_ERROR
    $display("UVM_ERROR @10: drv.sv(88) data mismatch");
`endif
    repeat (4) begin #5 clk = ~clk; end
`ifndef INJECT_TIMEOUT
    $display("TEST DONE");
`else
    $display("no TEST DONE line here");
`endif
    $finish;
  end
endmodule
"""


@dataclass(frozen=True)
class Result:
    test: str
    seed: int
    status: str
    errors: int
    fatals: int

    @property
    def exit_code(self):
        return EXIT[self.status]

    def one_line(self):
        return (f"{self.test} seed={self.seed} -> {self.status} "
                f"(errors={self.errors} fatals={self.fatals})")


def classify(text, sim_rc, built):
    if not built:
        return "BUILD", 0, 0
    lines = text.splitlines()
    errors = sum(1 for l in lines
                 if l.startswith("UVM_ERROR") and "UVM_ERROR :" not in l)
    fatals = sum(1 for l in lines
                 if l.startswith("UVM_FATAL") and "UVM_FATAL :" not in l)
    completed = any(l.strip() == DONE_MARKER for l in lines)
    if fatals or sim_rc != 0:
        return "FATAL", errors, fatals
    if not completed:
        return "TIMEOUT", errors, fatals
    if errors:
        return "FAIL", errors, fatals
    return "PASS", errors, fatals


def run_sim(test, seed, waves=False, defines=()):
    """One command, one run directory, one Result."""
    run_dir = Path(f"run_{test}_{seed}")
    run_dir.mkdir(parents=True, exist_ok=True)
    (run_dir / "tb.v").write_text(TB, encoding="utf-8")

    argv = ["iverilog", f"-DSEED={seed}"]
    argv += [f"-D{d}" for d in defines]
    if waves:
        argv.append("-DWAVES")
    argv += ["-o", "sim.vvp", "tb.v"]
    (run_dir / "cmd.sh").write_text(
        "#!/bin/sh\n" + " ".join(argv) + "\nvvp sim.vvp\n", encoding="utf-8")

    b = subprocess.run(argv, cwd=run_dir, stdout=subprocess.PIPE,
                       stderr=subprocess.STDOUT, text=True)
    (run_dir / "compile.log").write_text(b.stdout, encoding="utf-8")
    if b.returncode != 0:
        status, e, f = classify("", 0, built=False)
        return Result(test, seed, status, e, f)

    r = subprocess.run(["vvp", "sim.vvp"], cwd=run_dir,
                       stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                       text=True)
    (run_dir / "sim.log").write_text(r.stdout, encoding="utf-8")
    status, e, f = classify(r.stdout, r.returncode, built=True)
    return Result(test, seed, status, e, f)


RUNS = [
    ("axi_burst", 48210, False, ()),
    ("axi_burst", 48211, False, ("INJECT_ERROR",)),
    ("axi_lite", 7734, False, ("INJECT_TIMEOUT",)),
    ("axi_wrap", 9001, True, ()),
]

results = [run_sim(*spec) for spec in RUNS]

for res in results:
    print(f"  {res.one_line():<58} exit={res.exit_code}")

print()
print("runs            :", len(results))
print("statuses seen   :", sorted({r.status for r in results}))
print("exit 0 count    :", sum(1 for r in results if r.exit_code == 0))
print("waves dumped for:",
      [r.test for r, spec in zip(results, RUNS) if spec[2]])
print()
print("the flow's overall exit code:",
      0 if all(r.status == "PASS" for r in results) else 1)
print("which is 34.6's promise     :", "exit 0 only if every job passed")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  axi_burst seed=48210 -> PASS (errors=0 fatals=0)           exit=0
  axi_burst seed=48211 -> FAIL (errors=1 fatals=0)           exit=1
  axi_lite seed=7734 -> TIMEOUT (errors=0 fatals=0)          exit=1
  axi_wrap seed=9001 -> PASS (errors=0 fatals=0)             exit=0

runs            : 4
statuses seen   : ['FAIL', 'PASS', 'TIMEOUT']
exit 0 count    : 2
waves dumped for: ['axi_wrap']

the flow's overall exit code: 1
which is 34.6's promise     : exit 0 only if every job passed

Four real simulations, three distinct statuses, and the two failing runs exit 1 where vvp returned 0 for both. The TIMEOUT run is the one that proves the whole chapter: vvp was happy, no errors were printed, and the only evidence is a missing anchored marker.

Note that run_sim returns a Result rather than calling sys.exit. That is Chapter 33.8 Section 5's testable entry point and it is what lets Chapter 35.4's regression manager call this function four hundred times — a runner that exits cannot be a library.

11. How It Is Tested

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import io
import unittest

EXIT = {"PASS": 0, "FAIL": 1, "FATAL": 1, "TIMEOUT": 1, "BUILD": 3}
DONE_MARKER = "TEST DONE"


def classify(text, sim_rc, built):
    if not built:
        return "BUILD"
    lines = text.splitlines()
    errors = sum(1 for l in lines
                 if l.startswith("UVM_ERROR") and "UVM_ERROR :" not in l)
    fatals = sum(1 for l in lines
                 if l.startswith("UVM_FATAL") and "UVM_FATAL :" not in l)
    completed = any(l.strip() == DONE_MARKER for l in lines)
    if fatals or sim_rc != 0:
        return "FATAL"
    if not completed:
        return "TIMEOUT"
    if errors:
        return "FAIL"
    return "PASS"


class Classifier(unittest.TestCase):
    """Pure: no simulator needed. §10's run_sim is tested separately."""

    def test_clean_run_passes(self):
        self.assertEqual(classify("UVM_INFO a\nTEST DONE", 0, True), "PASS")

    def test_errors_fail(self):
        self.assertEqual(classify("UVM_ERROR x\nTEST DONE", 0, True), "FAIL")

    def test_a_failing_run_is_not_rescued_by_rc_zero(self):
        """§4: vvp returns 0 for a failing test."""
        self.assertEqual(classify("UVM_ERROR x\nTEST FAILED\nTEST DONE",
                                  0, True), "FAIL")

    def test_fatal_beats_errors(self):
        self.assertEqual(classify("UVM_ERROR x\nUVM_FATAL y", 1, True),
                         "FATAL")

    def test_missing_marker_is_a_timeout(self):
        self.assertEqual(classify("UVM_INFO a", 0, True), "TIMEOUT")

    def test_the_substring_trap(self):
        """§5: the log says the marker was NOT printed."""
        self.assertEqual(
            classify("UVM_INFO a\nno TEST DONE line here", 0, True),
            "TIMEOUT")

    def test_marker_with_a_suffix_is_not_the_marker(self):
        self.assertEqual(classify("TEST DONE_EARLY", 0, True), "TIMEOUT")

    def test_marker_with_surrounding_space_counts(self):
        self.assertEqual(classify("  TEST DONE  ", 0, True), "PASS")

    def test_the_uvm_summary_line_is_not_an_error(self):
        """24.3: UVM_ERROR : 0 must not make a clean run FAIL."""
        self.assertEqual(classify("UVM_ERROR :    0\nTEST DONE", 0, True),
                         "PASS")

    def test_a_nonzero_rc_overrides_a_complete_log(self):
        self.assertEqual(classify("TEST DONE", 1, True), "FATAL")

    def test_a_build_failure_is_not_a_design_failure(self):
        self.assertEqual(classify("", 0, False), "BUILD")

    def test_every_status_has_an_exit_code(self):
        statuses = {classify(*c) for c in
                    (("TEST DONE", 0, True), ("UVM_ERROR x\nTEST DONE", 0, True),
                     ("UVM_FATAL y", 1, True), ("UVM_INFO a", 0, True),
                     ("", 0, False))}
        self.assertEqual(statuses, {"PASS", "FAIL", "FATAL", "TIMEOUT",
                                    "BUILD"})
        for s in statuses:
            self.assertIn(s, EXIT)

    def test_only_pass_exits_zero(self):
        zero = [s for s, c in EXIT.items() if c == 0]
        self.assertEqual(zero, ["PASS"])


def run(case):
    suite = unittest.TestLoader().loadTestsFromTestCase(case)
    res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
    return {"run": res.testsRun, "failures": len(res.failures),
            "ok": res.wasSuccessful()}


print("classifier suite:", run(Classifier))
print()
print("simulator invocations in this suite:", 0)
print("statuses proven reachable          :", 5)
print("statuses that exit 0               :", 1)
print()
print("the classifier is pure, so the suite is:", "fast and runs anywhere")
print("and the subprocess part is tested      :", "by §10, which really ran")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

classifier suite: {'run': 13, 'failures': 0, 'ok': True}

simulator invocations in this suite: 0
statuses proven reachable          : 5
statuses that exit 0               : 1

the classifier is pure, so the suite is: fast and runs anywhere
and the subprocess part is tested      : by §10, which really ran

Thirteen behaviours pinned with no simulator at all. classify takes text, a return code and a boolean, so the entire verdict logic — the part that can be subtly wrong — is Chapter 34.3's testable middle layer, and the four real simulations in Section 10 exercise the impure shell around it.

Two tests deserve their place for structural reasons rather than for a specific bug. test_every_status_has_an_exit_code asserts that all five statuses are reachable and that every one has a code, which is Chapter 33.8 Section 8's check applied to a status enum rather than to a docstring. And test_only_pass_exits_zero pins the single most important property of the whole tool in one line: if a second status ever maps to 0, every wrapper in the flow starts silently accepting failures.

12. Predict, Run, Observe

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import subprocess
from pathlib import Path

Path("tb.v").write_text(
    "module tb;\n  initial begin\n"
    '    $display("UVM_INFO @0: start");\n'
    '    $display("UVM_ERROR @10: drv.sv(88) mismatch");\n'
    '    $display("TEST FAILED");\n'
    "    $finish;\n  end\nendmodule\n", encoding="utf-8")
subprocess.run(["iverilog", "-o", "sim.vvp", "tb.v"],
               stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
r = subprocess.run(["vvp", "sim.vvp"], stdout=subprocess.PIPE,
                   stderr=subprocess.STDOUT, text=True)

# A
print("A:", r.returncode)

# B
print("B:", sum(1 for l in r.stdout.splitlines()
                if l.startswith("UVM_ERROR")))

# C
print("C:", "TEST DONE" in r.stdout,
      any(l.strip() == "TEST DONE" for l in r.stdout.splitlines()))

# D
Path("bad.v").write_text('module tb;\n  initial $display("x"\nendmodule\n',
                         encoding="utf-8")
b = subprocess.run(["iverilog", "-o", "bad.vvp", "bad.v"],
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
print("D:", b.returncode, Path("bad.vvp").exists())

The verified output:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

A: 0
B: 1
C: False False
D: 2 False

A — the whole chapter in one value. The simulation printed an error and TEST FAILED, and vvp returned 0.

B — one error in the log, which is where the verdict actually lives.

C — both completion checks agree here, because this log contains TEST FAILED and not TEST DONE at all. The checks only diverge on a log that mentions the marker while denying it — Section 5's case.

D — iverilog returns 2 on a syntax error and leaves no artifact. Both halves matter: the code distinguishes a build failure from a simulation result, and the missing file means there is nothing to run even if a caller ignored the code.

13. Exercises

Exercise 1 — five outcomes. For your own flow, write down who each of the five belongs to.

Exercise 2 — two from the tool. Say which of the five your simulator can distinguish by itself.

Exercise 3 — no default seed. Explain what a fixed default seed does to a four-hundred-run regression.

Exercise 4 — waves off. Justify the default from Chapter 33.13's volume argument.

Exercise 5 — your own v1. Write the shell one-liner and say what it gets right.

Exercise 6 — correct for the wrong reason. Explain why a passing run hides the defect.

Exercise 7 — a failing test's exit code. Measure it with your own simulator.

Exercise 8 — $finish is a normal end. Explain in two sentences why the exit code is 0.

Exercise 9 — $fatal's argument. Confirm it is a verbosity level, not a status.

Exercise 10 — the build failure. Record your compiler's exit code and whether it leaves an artifact.

Exercise 11 — five candidate lines. Run Section 5's comparison on your own completion marker.

Exercise 12 — the denying line. Construct a log line that contains your marker and says it is absent.

Exercise 13 — anchored. Switch to a whole-line check and re-classify your last regression.

Exercise 14 — no second signal. Explain why a wrong completion check is not caught downstream.

Exercise 15 — cmd.sh before compile. Say which failure mode that ordering protects against.

Exercise 16 — BUILD is its own status. Explain what reporting it as FAIL costs.

Exercise 17 — first line, not last. Decide what your one-line verdict quotes from a compile log.

Exercise 18 — the check order. Justify testing fatals or rc != 0 before the completion check.

Exercise 19 — the summary-line guard. Confirm UVM_ERROR : does not make your clean runs fail.

Exercise 20 — the seed in the log. Make your runner put the seed in the log and say why a database is not enough.

Exercise 21 — compile-time or run-time seed. Say which your simulator supports and what it costs.

Exercise 22 — the VCD is not a fixture. Explain why, and list three facts about it you can assert.

Exercise 23 — return, do not exit. Explain why run_sim must not call sys.exit.

Exercise 24 — thirteen tests, no simulator. Build the pure classifier suite for your own statuses.

Exercise 25 — every status reachable. Add the test and say which chapter it comes from.

Exercise 26 — only PASS exits 0. Add that test and describe what breaks if it ever fails.

Exercise 27 — the TIMEOUT run. Construct one and confirm your runner does not call it a pass.

14. Summary

Five outcomes, not two: PASS, FAIL, FATAL, TIMEOUT and BUILD, and the last two belong to different people than the middle one.
--seed has no default. A chosen seed is an undeclared input; a fixed default makes four hundred identical runs look like coverage.
A failing test exits 0. Measured on real Icarus: a testbench that printed UVM_ERROR and TEST FAILED returned vvp status 0, because $finish is a normal end of simulation.
$fatal returns 1 and its first argument is a verbosity level, not a status — so a non-zero status is a crash detector, never a verdict.
A syntax error returns 2 from iverilog and builds no artifact, so BUILD cannot share a code path with a simulation result.
A substring check for TEST DONE calls a timeout a PASS. Three of five candidate lines fool it, and the one that matters is a log saying the marker was not printed.
Two runs with identical exit codes and opposite verdicts are separated only by an anchored whole-line completion check — there is no second signal to catch a mistake here.
Write cmd.sh before compiling, because a build failure is the most likely way for a run to end before anything else is written.
Classify from the log; use the return code only to detect a crash — and test fatals or rc != 0 first, so a complete-looking log with a non-zero status is not called a pass.
The seed goes in the log, so a log found later traces back to its command without a database.
A VCD is never a golden fixture — it carries a $date header — so assert only that it exists, is non-trivial, and has an $enddefinitions section.
run_sim returns a Result and never exits, which is what lets a regression manager call it four hundred times.
Thirteen behaviours pinned with no simulator, because the verdict logic takes text, a code and a boolean.
Assert that every status is reachable and that only PASS exits 0 — the second is the single most important property of the tool.

The next chapter calls this runner many times at once, which introduces every problem that comes with concurrency: isolation, bounded parallelism, resume, and one honest exit code for the whole sweep.

Where this fits

Part of the Python curriculum.