Skip to content
VLSI Mentor

Python Part II · Module 32 · Python, Tcl and Bash in EDA Flows

Python Driving the Simulator Directly

Chapters 32.4 and 32.5 kept a tool's own interpreter in the picture because a synthesis or timing tool has a design database that cannot leave it. A simulator used from a regression is different: it takes a command line, writes a log, and that is the whole interface — so Python can own the entire flow. There is a real simulator on this machine, which means this chapter is not a sketch of that flow but a working one, and the first thing it establishes is that the simulator will not tell you whether your test passed.

Intermediate21 min readPythonPart IIAutomationSimulationExit Codes

Part II · Module 32 · Chapter 32.6 · Python Driving the Simulator Directly

1. Why This Boundary Is Different

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
BOUNDARIES = [
    ("synthesis / STA", "a design database", "a Tcl script", "32.4, 32.5"),
    ("place and route", "a design database", "a Tcl script", "32.4, 32.5"),
    ("a simulator in a regression", "none that outlives the run",
     "a command line and a log", "this chapter"),
]

print(f"{'tool':<28} {'state Python cannot hold':<28} {'interface':<26} where")
for tool, state, iface, where in BOUNDARIES:
    print(f"{tool:<28} {state:<28} {iface:<26} {where}")

print()
print("tools needing a generated script:", 2)
print("tools Python can drive outright :", 1)
print("the difference is:", "whether anything has to stay loaded between commands")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
tool                         state Python cannot hold     interface                  where
synthesis / STA              a design database            a Tcl script               32.4, 32.5
place and route              a design database            a Tcl script               32.4, 32.5
a simulator in a regression  none that outlives the run   a command line and a log   this chapter

tools needing a generated script: 2
tools Python can drive outright : 1
the difference is: whether anything has to stay loaded between commands

A simulator does have a Tcl interface and interactive state, and a debug session uses them. A regression run does not: it compiles, runs to completion and exits, so the only things crossing the boundary are a command line, a log and a return code — three of Chapter 32.2's four channels, with no fifth thing that cannot leave.

That is why Chapter 32.5's test says "outside": the inputs are a test name, a seed and a filelist, none of which is a design database.

2. The Annotation

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
res = run_test("axi_burst", 48210, sim="xrun")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
def run_test(test, seed, sim="xrun"):
    """The annotation's signature, with nothing behind it yet."""
    return {"test": test, "seed": seed, "sim": sim}


res = run_test("axi_burst", 48210, sim="xrun")
for k, v in res.items():
    print(f"  {k:<6} {v!r}")

print()
GAPS = ["the version of sim", "a run directory", "a timeout",
        "a status derived from the log"]
print("things the signature does not mention:")
for g in GAPS:
    print(f"  - {g}")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  test   'axi_burst'
  seed   48210
  sim    'xrun'

things the signature does not mention:
  - the version of sim
  - a run directory
  - a timeout
  - a status derived from the log

Three arguments and four omissions. The signature itself is good — a test, a seed and a simulator are exactly the three things that identify a run — and the fourth omission is the one that makes the chapter, because Section 4 shows nothing else will supply it.

3. The Real Thing, Compiled and Run

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import subprocess
from pathlib import Path

Path("dut.v").write_text("""module dut(input clk, input [7:0] din, output [7:0] dout);
  reg [7:0] r;
  always @(posedge clk) r <= din ^ 8'h0F;
  assign dout = r;
endmodule
""", encoding="utf-8")

Path("tb.v").write_text("""module tb;
  reg clk = 0; reg [7:0] din; wire [7:0] dout;
  integer seed, errors, i;
  dut u(.clk(clk), .din(din), .dout(dout));
  always #1 clk = ~clk;
  initial begin
    errors = 0;
    if (!$value$plusargs("seed=%d", seed)) begin
      $display("UVM_FATAL tb.v(8) @ 0: no +seed given");
      $finish;
    end
    $display("UVM_INFO tb.v(11) @ 0: seed=%0d", seed);
    for (i = 0; i < 4; i = i + 1) begin
      din = $random(seed);
      @(posedge clk); @(negedge clk);
      if (dout !== (din ^ 8'h0F)) begin
        errors = errors + 1;
        $display("UVM_ERROR tb.v(18) @ %0t: din=%02h dout=%02h", $time, din, dout);
      end
    end
    $display("UVM_INFO tb.v(22) @ %0t: errors=%0d", $time, errors);
    $display("TEST DONE");
    $finish;
  end
endmodule
""", encoding="utf-8")

c = subprocess.run(["iverilog", "-g2005", "-o", "sim.vvp", "dut.v", "tb.v"],
                   stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
print("compile returncode:", c.returncode)

r = subprocess.run(["vvp", "sim.vvp", "+seed=48210"], stdout=subprocess.PIPE,
                   stderr=subprocess.STDOUT, text=True)
print("run returncode    :", r.returncode)
for line in r.stdout.splitlines():
    if line.startswith(("UVM_", "TEST")):
        print("  ", line)
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
compile returncode: 0
run returncode    : 0
   UVM_INFO tb.v(11) @ 0: seed=48210
   UVM_INFO tb.v(22) @ 8: errors=0
   TEST DONE

A real compile, a real run, a seed passed on the command line, and a log in the shape Module 20 taught. Everything from here uses this as the thing being driven.

4. The Simulator Will Not Tell You If the Test Passed

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import subprocess
from pathlib import Path

Path("dut.v").write_text("""module dut(input clk, input [7:0] din, output [7:0] dout);
  reg [7:0] r;
  always @(posedge clk) r <= din ^ 8'h0E;
  assign dout = r;
endmodule
""", encoding="utf-8")

Path("tb.v").write_text("""module tb;
  reg clk = 0; reg [7:0] din; wire [7:0] dout;
  integer seed, errors, i;
  dut u(.clk(clk), .din(din), .dout(dout));
  always #1 clk = ~clk;
  initial begin
    errors = 0;
    if (!$value$plusargs("seed=%d", seed)) begin
      $display("UVM_FATAL tb.v(8) @ 0: no +seed given");
      $finish;
    end
    for (i = 0; i < 4; i = i + 1) begin
      din = $random(seed);
      @(posedge clk); @(negedge clk);
      if (dout !== (din ^ 8'h0F)) begin
        errors = errors + 1;
        $display("UVM_ERROR tb.v(18) @ %0t: mismatch", $time);
      end
    end
    $display("UVM_INFO tb.v(22) @ %0t: errors=%0d", $time, errors);
    $display("TEST DONE");
    $finish;
  end
endmodule
""", encoding="utf-8")

subprocess.run(["iverilog", "-g2005", "-o", "sim.vvp", "dut.v", "tb.v"],
               stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

CASES = [("a broken DUT, seeded", ["+seed=48210"]),
         ("no seed at all (UVM_FATAL)", [])]

for label, extra in CASES:
    r = subprocess.run(["vvp", "sim.vvp"] + extra, stdout=subprocess.PIPE,
                       stderr=subprocess.STDOUT, text=True)
    errs = sum(1 for l in r.stdout.splitlines() if l.startswith("UVM_ERROR"))
    fatals = sum(1 for l in r.stdout.splitlines() if l.startswith("UVM_FATAL"))
    done = "TEST DONE" in r.stdout
    print(f"  {label:<28} rc={r.returncode}  "
          f"UVM_ERROR={errs} UVM_FATAL={fatals} end_banner={done}")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  a broken DUT, seeded         rc=0  UVM_ERROR=4 UVM_FATAL=0 end_banner=True
  no seed at all (UVM_FATAL)   rc=0  UVM_ERROR=0 UVM_FATAL=1 end_banner=False

Both runs returned zero. Four UVM_ERRORs returned zero. A UVM_FATAL returned zero.

That is the finding the last clause of the canonical purpose exists for. A simulator's exit code answers did the simulator work — did it find its files, elaborate, and reach $finish — and not did the test pass. The two questions have different answers and only one of them is in the return code.

So a flow that tests returncode == 0 reports a green regression over a design that failed every check, which is exactly the shape of defect Chapter 30.7 Section 6 named: a failing run rendering as a clean summary.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
QUESTIONS = [
    ("did the simulator start?", "returncode", "yes, reliably"),
    ("did it elaborate?", "returncode", "yes, reliably"),
    ("did it reach $finish?", "the end banner", "the log"),
    ("did the test pass?", "UVM_ERROR / UVM_FATAL counts", "the log"),
    ("was it the seed I asked for?", "the seed echo", "the log"),
]

print(f"{'question':<32} {'answered by':<32} source")
for q, by, src in QUESTIONS:
    print(f"{q:<32} {by:<32} {src}")

print()
print("questions the returncode answers:", 2)
print("questions only the log answers   :", 3)
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
question                         answered by                      source
did the simulator start?         returncode                       yes, reliably
did it elaborate?                returncode                       yes, reliably
did it reach $finish?            the end banner                   the log
did the test pass?               UVM_ERROR / UVM_FATAL counts     the log
was it the seed I asked for?     the seed echo                    the log

questions the returncode answers: 2
questions only the log answers   : 3

5. The Other Three Omissions

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import os
import subprocess
from pathlib import Path

print("=== 1. the simulator version is free to capture, once")
r = subprocess.run(["iverilog", "-V"], stdout=subprocess.PIPE,
                   stderr=subprocess.STDOUT, text=True)
version = r.stdout.splitlines()[0]
print("  sim='iverilog' tells you:", "which tool")
print("  this line tells you      :", version)
print("  cost of capturing it     :", "one subprocess call per run")

print()
print("=== 2. without a run directory, two runs collide")
for d in ("runs/axi_burst_48210", "runs/axi_burst_777"):
    Path(d).mkdir(parents=True, exist_ok=True)
    (Path(d) / "sim.log").write_text(f"log for {d}\n", encoding="utf-8")

flat = Path("sim.log")
flat.write_text("first run\n", encoding="utf-8")
flat.write_text("second run overwrote it\n", encoding="utf-8")
print("  one shared sim.log holds:", flat.read_text(encoding="utf-8").strip())
print("  per-run directories      :", sorted(p.name for p in Path("runs").iterdir()))

print()
print("=== 3. a timeout is the difference between a slow run and a hung flow")
import sys
Path("hang.py").write_text("import time\ntime.sleep(30)\n", encoding="utf-8")
try:
    subprocess.run([sys.executable, "hang.py"], timeout=1)
    print("  no timeout: the flow waits")
except subprocess.TimeoutExpired:
    print("  timeout=1 : TimeoutExpired raised, the flow continues")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
=== 1. the simulator version is free to capture, once
  sim='iverilog' tells you: which tool
  this line tells you      : Icarus Verilog version 13.0 (stable) (v13_0)
  cost of capturing it     : one subprocess call per run

=== 2. without a run directory, two runs collide
  one shared sim.log holds: second run overwrote it
  per-run directories      : ['axi_burst_48210', 'axi_burst_777']

=== 3. a timeout is the difference between a slow run and a hung flow
  timeout=1 : TimeoutExpired raised, the flow continues

Three omissions, three one-line fixes, and the middle one is the only one that silently produces wrong data rather than a visible failure: a shared log file means the second run's log is the only one that exists, and the first run's result record points at a file describing a different run.

Chapter 23.7 and Chapter 26.4 both argued for per-run directories; this is the same argument at the point where the directory is created.

6. The Seed Contract

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import subprocess
from pathlib import Path

Path("dut.v").write_text("""module dut(input clk, input [7:0] din, output [7:0] dout);
  reg [7:0] r;
  always @(posedge clk) r <= din ^ 8'h0F;
  assign dout = r;
endmodule
""", encoding="utf-8")
Path("tb.v").write_text("""module tb;
  reg clk = 0; reg [7:0] din; wire [7:0] dout;
  integer seed, i;
  dut u(.clk(clk), .din(din), .dout(dout));
  always #1 clk = ~clk;
  initial begin
    if (!$value$plusargs("seed=%d", seed)) begin
      $display("UVM_FATAL tb.v(8) @ 0: no +seed given"); $finish;
    end
    $display("UVM_INFO tb.v(11) @ 0: seed=%0d", seed);
    for (i = 0; i < 3; i = i + 1) begin
      din = $random(seed);
      @(posedge clk); @(negedge clk);
      $display("UVM_INFO tb.v(15) @ %0t: din=%02h", $time, din);
    end
    $display("TEST DONE"); $finish;
  end
endmodule
""", encoding="utf-8")
subprocess.run(["iverilog", "-g2005", "-o", "sim.vvp", "dut.v", "tb.v"],
               stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)


def stimulus(seed):
    r = subprocess.run(["vvp", "sim.vvp", f"+seed={seed}"],
                       stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
    return [l.split("din=")[1] for l in r.stdout.splitlines() if "din=" in l]


a = stimulus(48210)
b = stimulus(48210)
c = stimulus(777)

print("seed 48210, first run :", a)
print("seed 48210, again     :", b)
print("seed 777              :", c)
print()
print("same seed reproduces  :", a == b)
print("different seed differs:", a != c)
print("the seed crossed as   :", "+seed=48210 on the command line — 32.2 §4's argv channel")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
seed 48210, first run : ['8c', '16', 'c4']
seed 48210, again     : ['8c', '16', 'c4']
seed 777              : ['06', '10', 'f4']

same seed reproduces  : True
different seed differs: True
the seed crossed as   : +seed=48210 on the command line — 32.2 §4's argv channel

Both halves of the contract hold, and both are worth checking rather than assuming.

The same seed reproduces the same stimulus exactly — 8c 16 c4 twice — which is what makes a failing run debuggable and what Chapter 26.3 required of a regression. And a different seed produces different stimulus, 06 10 f4, which is what makes a sweep worth running.

The second half is the one nobody verifies, and Chapter 30.3 Section 3 is why it matters: a predicate whose output never varies is vacuously false, and a seed whose output never varies is the same defect in a different place. A sweep over a seed that the testbench accepted and ignored would run one scenario under twenty labels and report broad coverage of it — and the symptom would be a coverage number that refuses to move, which is the hardest kind of problem to attribute.

So the check is two runs and a comparison, it costs one line, and it belongs in the flow's own test suite because nothing else in the stack performs it. Here it passes; the point is that it was run.

7. The Flow, as a Flow

Python owning a simulation run end to end

A seven-stage flow for one simulation run: validate the inputs, create the run directory, build the command and record it, run with a timeout, parse the log, classify the result, and return a record with an exit code.01VALIDATE INPUTStest, seed, filelist exist02CREATE THE RUN DIRECTORYruns/test_seed03BUILD THE COMMANDa list, never a string04RECORD ITcmd.txt, tool version05RUN WITH A TIMEOUTcapture the log06PARSE THE LOGerrors, banner, seed echo07CLASSIFY AND RETURNstatus, then the exit code
Seven stages, and the two shaded as checks are the ones the annotation's one-liner has no room for. Stage 6 is where the status comes from, because — as Section 4 measured — stage 5's return code cannot supply it.

Stage 3's caption is worth one sentence. Chapter 23.3 argued for a list rather than a string because a string has to be re-parsed by a shell, and the seed makes it concrete: f"vvp sim.vvp +seed={seed}" is a shell command, and a list ["vvp", "sim.vvp", f"+seed={seed}"] is an argument vector — which is Chapter 32.3 Section 2's quoting problem, avoided by not creating it.

8. The Flow, Assembled

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import subprocess
import sys
from dataclasses import dataclass, field
from pathlib import Path
from typing import List, Optional

Path("dut.v").write_text("""module dut(input clk, input [7:0] din, output [7:0] dout);
  reg [7:0] r;
  always @(posedge clk) r <= din ^ 8'h0F;
  assign dout = r;
endmodule
""", encoding="utf-8")
Path("tb.v").write_text("""module tb;
  reg clk = 0; reg [7:0] din; wire [7:0] dout;
  integer seed, errors, i;
  dut u(.clk(clk), .din(din), .dout(dout));
  always #1 clk = ~clk;
  initial begin
    errors = 0;
    if (!$value$plusargs("seed=%d", seed)) begin
      $display("UVM_FATAL tb.v(8) @ 0: no +seed given"); $finish;
    end
    $display("UVM_INFO tb.v(11) @ 0: seed=%0d", seed);
    for (i = 0; i < 4; i = i + 1) begin
      din = $random(seed);
      @(posedge clk); @(negedge clk);
      if (dout !== (din ^ 8'h0F)) begin
        errors = errors + 1;
        $display("UVM_ERROR tb.v(18) @ %0t: mismatch", $time);
      end
    end
    $display("UVM_INFO tb.v(22) @ %0t: errors=%0d", $time, errors);
    $display("TEST DONE"); $finish;
  end
endmodule
""", encoding="utf-8")


@dataclass
class RunResult:
    test: str
    seed: int
    sim: str
    sim_version: str
    status: str
    errors: int
    fatals: int
    returncode: Optional[int]
    timed_out: bool
    seed_echoed: Optional[int]
    run_dir: str
    log_path: str

    @property
    def exit_code(self):
        return {"PASS": 0, "FAIL": 1, "NO_FINISH": 2,
                "TIMEOUT": 3, "INFRA": 4}[self.status]


def sim_version(sim):
    r = subprocess.run([sim, "-V"], stdout=subprocess.PIPE,
                       stderr=subprocess.STDOUT, text=True)
    return r.stdout.splitlines()[0] if r.returncode == 0 else "unknown"


def run_test(test, seed, srcs, sim="iverilog", runner="vvp", timeout=30):
    missing = [s for s in srcs if not Path(s).is_file()]
    if missing:
        return RunResult(test, seed, sim, "unknown", "INFRA", 0, 0,
                         None, False, None, "", "")

    run_dir = Path("runs") / f"{test}_{seed}"
    run_dir.mkdir(parents=True, exist_ok=True)
    vvp_file = run_dir / "sim.vvp"
    log_path = run_dir / "sim.log"

    build = [sim, "-g2005", "-o", str(vvp_file)] + list(srcs)
    cmd = [runner, str(vvp_file), f"+seed={seed}"]
    (run_dir / "cmd.txt").write_text(
        " ".join(build) + "\n" + " ".join(cmd) + "\n", encoding="utf-8")

    c = subprocess.run(build, stdout=subprocess.PIPE,
                       stderr=subprocess.STDOUT, text=True)
    if c.returncode != 0:
        log_path.write_text(c.stdout, encoding="utf-8")
        return RunResult(test, seed, sim, sim_version(sim), "INFRA", 0, 0,
                         c.returncode, False, None, str(run_dir), str(log_path))

    timed_out = False
    try:
        r = subprocess.run(cmd, stdout=subprocess.PIPE,
                           stderr=subprocess.STDOUT, text=True, timeout=timeout)
        out, rc = r.stdout, r.returncode
    except subprocess.TimeoutExpired as exc:
        out, rc, timed_out = (exc.stdout or b"").decode(errors="replace"), None, True

    log_path.write_text(out, encoding="utf-8")
    errors = sum(1 for l in out.splitlines() if l.startswith("UVM_ERROR"))
    fatals = sum(1 for l in out.splitlines() if l.startswith("UVM_FATAL"))
    finished = "TEST DONE" in out
    echoed = None
    for l in out.splitlines():
        if "seed=" in l and l.startswith("UVM_INFO"):
            echoed = int(l.split("seed=")[1].split()[0])
            break

    if timed_out:
        status = "TIMEOUT"
    elif fatals or errors:
        status = "FAIL"
    elif not finished:
        status = "NO_FINISH"
    else:
        status = "PASS"

    return RunResult(test, seed, sim, sim_version(sim), status, errors, fatals,
                     rc, timed_out, echoed, str(run_dir), str(log_path))


res = run_test("axi_burst", 48210, ["dut.v", "tb.v"])
print(f"test        {res.test}")
print(f"seed        {res.seed}  echoed back as {res.seed_echoed}")
print(f"sim         {res.sim}")
print(f"status      {res.status}")
print(f"errors      {res.errors}   fatals {res.fatals}")
print(f"returncode  {res.returncode}   timed_out {res.timed_out}")
print(f"run_dir     {res.run_dir}")
print(f"exit_code   {res.exit_code}")
print()
print("files in the run directory:",
      sorted(p.name for p in Path(res.run_dir).iterdir()))
print("seed confirmed by the log :", res.seed_echoed == res.seed)
print("version captured          :", res.sim_version.startswith("Icarus"))

bad = run_test("axi_burst", 48210, ["dut.v", "nonexistent.v"])
print()
print("a missing source file ->", bad.status, "exit_code", bad.exit_code)
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
test        axi_burst
seed        48210  echoed back as 48210
sim         iverilog
status      PASS
errors      0   fatals 0
returncode  0   timed_out False
run_dir     runs/axi_burst_48210
exit_code   0

files in the run directory: ['cmd.txt', 'sim.log', 'sim.vvp']
seed confirmed by the log : True
version captured          : True

a missing source file -> INFRA exit_code 4

Every clause of the canonical purpose, executed: the command was built as a list, the run directory is owned and holds cmd.txt alongside the log, the log was read, the result is a record, and the exit code is derived from the status rather than from the simulator's return code.

Two details carry most of the value. seed_echoed compares what the log says against what was asked for, so a plusarg the testbench ignored is caught rather than assumed — which is Chapter 30.7's "check the artefact" applied to a command-line argument. And INFRA with exit code 4 separates the flow is broken from the design is broken, which Chapter 30.1 Section 4 showed is the most useful line in a regression report.

9. Two Streams, One Log

The log is the only evidence a simulation leaves, and a simulator writes to two file descriptors:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import subprocess
import sys
from pathlib import Path

Path("two.py").write_text(
    "import sys\n"
    "print('to stdout: UVM_INFO starting', flush=True)\n"
    "print('to stderr: Error-[SE] syntax error', file=sys.stderr, flush=True)\n"
    "print('to stdout: TEST DONE', flush=True)\n", encoding="utf-8")

print("--- stdout only (stderr goes to the terminal, not the log)")
r = subprocess.run([sys.executable, "two.py"], stdout=subprocess.PIPE,
                   stderr=subprocess.DEVNULL, text=True)
print("captured lines:", len(r.stdout.strip().splitlines()))
for l in r.stdout.strip().splitlines():
    print("   ", l)

print()
print("--- both streams into one")
r = subprocess.run([sys.executable, "two.py"], stdout=subprocess.PIPE,
                   stderr=subprocess.STDOUT, text=True)
print("captured lines:", len(r.stdout.strip().splitlines()))
for l in r.stdout.strip().splitlines():
    print("   ", l)
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
--- stdout only (stderr goes to the terminal, not the log)
captured lines: 2
    to stdout: UVM_INFO starting
    to stdout: TEST DONE

--- both streams into one
captured lines: 3
    to stdout: UVM_INFO starting
    to stderr: Error-[SE] syntax error
    to stdout: TEST DONE

The stdout-only capture lost the error line entirely. A log that holds two of three lines is worse than no log, because it looks complete — and the missing line is the one naming the problem, since tools write elaboration and syntax errors to stderr while writing their normal progress to stdout.

Chapter 32.6 Section 4 established that the returncode is not the verdict and the log is. This is the corollary: if the log is the verdict, it has to contain everything.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import subprocess
import sys
from pathlib import Path

Path("two.py").write_text(
    "import sys\n"
    "print('out 1', flush=True)\n"
    "print('err 1', file=sys.stderr, flush=True)\n"
    "print('out 2', flush=True)\n", encoding="utf-8")

OPTIONS = [
    ("stderr=DEVNULL", {"stdout": subprocess.PIPE, "stderr": subprocess.DEVNULL},
     "the error is gone"),
    ("stderr=PIPE (separate)", {"stdout": subprocess.PIPE, "stderr": subprocess.PIPE},
     "two logs, order between them unknown"),
    ("stderr=STDOUT (merged)", {"stdout": subprocess.PIPE, "stderr": subprocess.STDOUT},
     "one log, one order"),
]

for label, kw, note in OPTIONS:
    r = subprocess.run([sys.executable, "two.py"], text=True, **kw)
    out = len(r.stdout.strip().splitlines()) if r.stdout else 0
    err = len(r.stderr.strip().splitlines()) if r.stderr else 0
    print(f"  {label:<24} stdout={out} stderr={err}  {note}")

print()
print("the option a flow wants:", "stderr=STDOUT")
print("because a log is read by:", "a parser that needs one ordered stream")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  stderr=DEVNULL           stdout=2 stderr=0  the error is gone
  stderr=PIPE (separate)   stdout=2 stderr=1  two logs, order between them unknown
  stderr=STDOUT (merged)   stdout=3 stderr=0  one log, one order

The middle option is the one that looks most careful and causes the most trouble. Keeping the streams separate preserves which descriptor each line came from and destroys the interleaving — so a parser cannot tell whether the error arrived before or after the last progress message, which is exactly what Chapter 20.9's first-failure logic needs.

Merging loses the stream label and keeps the order. For a log that a parser reads, order is the information worth keeping; if the stream matters, the tool usually says so in the line's own prefix anyway.

One caveat belongs here rather than in a footnote. Merging means the two streams share a pipe, and a process that writes a large amount to one of them while the parent is not reading can fill the buffer and block — which is why Chapter 23.6 preferred subprocess.run with capture over a hand-rolled Popen read loop. run reads until the child exits, so the deadlock cannot occur, and the one real cost is that nothing is visible until the run finishes.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
TRADES = [
    ("subprocess.run, merged", "no deadlock, full log", "nothing visible live"),
    ("Popen with a read loop", "live output", "deadlock if you read one pipe"),
    ("run with a file handle", "no deadlock, live via tail -f", "two opens"),
]

print(f"{'approach':<28} {'gains':<26} costs")
for name, gain, cost in TRADES:
    print(f"{name:<28} {gain:<26} {cost}")

print()
print("the one to default to:", "run with a file handle for the log")
print("why:", "the flow gets a complete log and an engineer can tail it")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
approach                     gains                      costs
subprocess.run, merged       no deadlock, full log      nothing visible live
Popen with a read loop       live output                deadlock if you read one pipe
run with a file handle       no deadlock, live via tail -f two opens

Writing straight to a file handle is the default worth having: the child writes into sim.log as it goes, so an engineer can tail -f a long run, and Python reads the file afterwards rather than holding the whole log in memory — which for a UVM log at full verbosity is the difference between a few kilobytes of Python and a few hundred megabytes.

10. Predict, Run, Observe

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import subprocess
from pathlib import Path

Path("t.v").write_text("""module tb;
  initial begin
    $display("UVM_FATAL t.v(3) @ 0: giving up");
    $finish;
  end
endmodule
""", encoding="utf-8")
subprocess.run(["iverilog", "-g2005", "-o", "t.vvp", "t.v"],
               stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
r = subprocess.run(["vvp", "t.vvp"], stdout=subprocess.PIPE,
                   stderr=subprocess.STDOUT, text=True)

# A
print("A:", r.returncode)

# B
print("B:", sum(1 for l in r.stdout.splitlines() if l.startswith("UVM_FATAL")))

# C
print("C:", "TEST DONE" in r.stdout)

# D
print("D:", r.returncode == 0 and "UVM_FATAL" in r.stdout)

The verified output:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
A: 0
B: 1
C: False
D: True

A — a simulation that gave up returned zero.

B — the log says exactly what happened.

C — and the end banner is absent, which is the other thing only the log knows.

D — the two together: returncode zero and a fatal. Any flow that reads only the left-hand side of that and reports this run as a pass.

11. Exercises

Exercise 1 — why this boundary differs. Say what state a simulator holds during a regression run that must outlive it.

Exercise 2 — apply 32.5's test. Name the inputs of a simulation run and confirm they place it outside.

Exercise 3 — the signature. Write the signature your own run_test has, and list what it omits.

Exercise 4 — run one for real. Compile and run a small design with a seed on the command line.

Exercise 5 — a failing test's returncode. Make your DUT wrong, run it, and report the returncode.

Exercise 6 — a fatal's returncode. Make the testbench call a fatal and report the returncode.

Exercise 7 — five questions. For your simulator, say which of Section 4's five the returncode answers.

Exercise 8 — green over broken. Describe what your flow would report today for a run with four UVM_ERRORs and rc=0.

Exercise 9 — two fields, not one. Keep returncode and status separately in one result and say what that distinguishes.

Exercise 10 — capture the version. Add the simulator version to your result record and say what it costs.

Exercise 11 — the shared log. Run two tests writing to one log path and report which log survives.

Exercise 12 — silently wrong versus visibly broken. Explain why the shared-log omission is the worst of the three.

Exercise 13 — add a timeout. Put one on your simulation call and say what value you chose and why.

Exercise 14 — reproduce a seed. Run the same seed twice and diff the stimulus.

Exercise 15 — does your seed do anything? Run two different seeds and confirm the stimulus actually differs.

Exercise 16 — a knob that does nothing. Explain why an unvarying output means the knob is broken, with reference to Chapter 30.3.

Exercise 17 — the sweep that was not. Describe what a seed sweep reports if the seed has no effect.

Exercise 18 — a list, not a string. Convert one command built by string formatting into a list, and say what quoting problem disappears.

Exercise 19 — seven stages. Map your own runner onto Section 7's stages and name the ones it skips.

Exercise 20 — echo the seed. Make your testbench print the seed it used and compare it in Python.

Exercise 21 — the plusarg that was ignored. Say how you would detect a testbench that does not read +seed.

Exercise 22 — INFRA versus FAIL. Give your runner a distinct status and exit code for a missing source file.

Exercise 23 — derive the exit code. Map every status to an exit code and justify each number.

Exercise 24 — cmd.txt. Add the recorded command to your run directory and paste it back to reproduce a run.

12. Summary

  • A simulator in a regression holds no state that outlives the run, so Python can own the whole flow — unlike the tools of Chapters 32.4 and 32.5.
  • The annotation's three arguments are the right three and it omits four things: the version, a run directory, a timeout, and a status.
  • A simulation with four UVM_ERRORs returned 0, and one with a UVM_FATAL returned 0. The exit code reports whether the simulator worked, not whether the test passed.
  • Two of five questions come from the returncode and three only from the log; keep both as separate fields so an infrastructure failure is distinguishable from a failing design.
  • The version is one subprocess call and sim="xrun" names a tool without it.
  • A shared log path is the only omission that silently produces wrong data — the second run's log is the one that survives, and the first result record points at it.
  • A timeout is what separates a slow run from a hung flow.
  • The same seed reproduced the same stimulus exactly, which is what makes a failure debuggable.
  • A different seed produced different stimulus, which is the half nobody verifies — a seed the testbench accepted and ignored would run one scenario under twenty labels and show up only as a coverage number that will not move.
  • Build the command as a list, not a string, which removes Chapter 32.3's quoting problem by never creating it.
  • Compare the seed the log echoes against the seed you asked for, so a plusarg the testbench ignored is caught rather than assumed.
  • Give INFRA its own status and exit code, because the flow is broken and the design is broken are the split a reviewer most needs.
  • Derive the exit code from the status, never from the simulator's return code.
  • A simulator writes to two file descriptors, and a stdout-only capture lost the error line while looking complete.
  • If the log is the verdict, it must contain everything — tools write syntax and elaboration errors to stderr and progress to stdout.
  • Keeping the streams separate destroys the interleaving, which is exactly what first-failure logic needs; merging loses the stream label and keeps the order.
  • Write straight to a file handle: the child fills sim.log as it goes so an engineer can tail -f it, and Python reads the file afterwards instead of holding a UVM log in memory.

The last chapter of the module follows a failure through every hop of the chain and finds where it disappears.

Where this fits

Part of the Python curriculum.