What the Simulation Runner Must Do

Part II · Module 25 · Chapter 25.1 · What the Simulation Runner Must Do

1. Four Questions Before Any Code

CONTRACT = [
    ("inputs", "what must the caller supply, and what is optional"),
    ("ownership", "which directory does the runner create, own and write to"),
    ("result", "what comes back — always, including when the run failed"),
    ("failure modes", "every distinct way one run can fail, enumerated"),
]

for name, question in CONTRACT:
    print(f"{name:<14} {question}")

print()
print("decided now  : cheap — one function has no callers yet")
print("decided later: expensive — Modules 26 to 31 all call this")

inputs         what must the caller supply, and what is optional
ownership      which directory does the runner create, own and write to
result         what comes back — always, including when the run failed
failure modes  every distinct way one run can fail, enumerated

decided now  : cheap — one function has no callers yet
decided later: expensive — Modules 26 to 31 all call this

The fourth question is the one that gets skipped, and it is the one that decides the shape of the other three. A runner written for the case where the simulator runs and reports errors turns out, later, to need a different answer for a compile that never produced a binary, a tool that was never on PATH, and a run that was still going when the clock ran out — and by then it has callers.

2. The Signature

def run_test(test, seed, *, sim, waves=False) -> RunResult: ...

Five decisions in one line, and each is worth naming.

test and seed are positional, because they are what identifies a run — Chapter 23.7 Section 2 named the run directory after exactly this pair, and Chapter 20.8 made the seed the reproducibility handle.

sim is keyword-only and has no default. Chapter 23.11's adapter table means the runner can drive several simulators; which one is in use is never a thing to guess.

waves is keyword-only with a default, because Chapter 25.5 will show waves cost runtime and disk, so off is the right default and on must be asked for.

It returns a RunResult, not a boolean, a tuple or a return code.

It does not raise for a failing test. A test that fails is a result.

3. Why Keyword-Only

import inspect


def run_test(test, seed, *, sim, waves=False):
    return {"test": test, "seed": seed, "sim": sim, "waves": waves}


print("signature:", inspect.signature(run_test))
print()
print(run_test("axi_smoke", 1839274, sim="xrun"))
print(run_test("axi_smoke", 1839274, sim="vcs", waves=True))
print()
try:
    run_test("axi_smoke", 1839274, "xrun")
except TypeError as exc:
    print("positional sim ->", type(exc).__name__ + ":", exc)

signature: (test, seed, *, sim, waves=False)

{'test': 'axi_smoke', 'seed': 1839274, 'sim': 'xrun', 'waves': False}
{'test': 'axi_smoke', 'seed': 1839274, 'sim': 'vcs', 'waves': True}

positional sim -> TypeError: run_test() takes 2 positional arguments but 3 were given

The bare * in the parameter list means everything after this must be passed by name.

What that buys is not style. Consider the call this prevents:

run_test("axi_smoke", 1839274, "xrun", True)

Readable only if you remember the order. Now imagine a sixth parameter arrives — a timeout, say — and somebody inserts it in a sensible-looking place. Every positional call site silently shifts by one, and waves starts receiving a number of seconds. Keyword-only arguments make that failure impossible, because a name cannot be in the wrong position.

The TypeError arrives at the call site, immediately, naming the arity. Compare the alternative, where the wrong value travels into the function and produces something strange several frames deeper.

4. The Result Type

from dataclasses import dataclass, asdict, fields
from typing import Optional
import json


@dataclass(frozen=True)
class RunResult:
    test: str
    seed: int
    status: str
    errors: tuple = ()
    runtime_s: float = 0.0
    log_path: Optional[str] = None


result = RunResult("axi_smoke", 1839274, "PASS",
                   runtime_s=182.4, log_path="runs/axi_smoke_1839274/sim.log")

print(result)
print()
print("fields :", [f.name for f in fields(result)])
print("as JSON:", json.dumps(asdict(result), sort_keys=True))

RunResult(test='axi_smoke', seed=1839274, status='PASS', errors=(), runtime_s=182.4, log_path='runs/axi_smoke_1839274/sim.log')

fields : ['test', 'seed', 'status', 'errors', 'runtime_s', 'log_path']
as JSON: {"errors": [], "log_path": "runs/axi_smoke_1839274/sim.log", "runtime_s": 182.4, "seed": 1839274, "status": "PASS", "test": "axi_smoke"}

The 182.4 is a constant written into the example, not something measured — this curriculum does not publish timings, and Chapter 25.6 explains why a runner nevertheless records one.

Three properties of that declaration matter.

frozen=True makes it immutable. A result describes something that already happened, and code that adjusts a status after the fact is code nobody can reason about:

from dataclasses import dataclass, FrozenInstanceError


@dataclass(frozen=True)
class RunResult:
    test: str
    seed: int
    status: str


result = RunResult("axi_smoke", 1839274, "FAIL")
try:
    result.status = "PASS"
except FrozenInstanceError as exc:
    print("cannot rewrite history:", type(exc).__name__)
print("still:", result.status)

cannot rewrite history: FrozenInstanceError
still: FAIL

asdict makes it JSON-ready, which is Chapter 21.4's requirement for a record a program reads later. Note the tuple became a JSON list — tuples have no JSON counterpart, and that asymmetry matters when the file is read back.

errors defaults to a tuple, not a list, and that is not a preference:

from dataclasses import dataclass


try:
    @dataclass
    class Bad:
        errors: list = []
except ValueError as exc:
    print("ValueError:", exc)

print()


def bad_function(errors=[]):
    errors.append("x")
    return errors


print("but a function accepts it silently:", bad_function(), bad_function())

ValueError: mutable default <class 'list'> for field errors is not allowed: use default_factory

but a function accepts it silently: ['x', 'x'] ['x', 'x']

The dataclass refuses at class-creation time, with the fix in the message. The function accepts the same mistake silently and shows it only at runtime — Chapter 11.5's trap.

And look closely at what it printed: both calls show two entries. Python evaluates every argument before print runs, so the second call has already appended by the time the first value is rendered — and they render identically because they are not two lists, they are two references to the one list that is the function's default. The dataclass refuses to let you create that situation at all.

Use a tuple for a fixed collection, or field(default_factory=list) when it must be a list.

5. The Shape of One Run

Read it as the module's map: Chapter 25.2 is step two, 25.3 is step three, 25.4 is step four, 25.5 is step five, and 25.6 is step six. Step one was Chapter 24.5.

6. Every Way One Run Can Fail

FAILURE_MODES = [
    ("the tool is not on PATH", "ERROR", "infrastructure", "Chapter 23.9"),
    ("the run directory cannot be created", "ERROR", "infrastructure", "Chapter 24.5"),
    ("a required variable is unset", "ERROR", "infrastructure", "Chapter 24.2"),
    ("no sources matched the filelist pattern", "ERROR", "infrastructure", "Chapter 25.2"),
    ("the design does not compile", "COMPILE_ERROR", "the RTL", "Chapter 25.3"),
    ("the testbench reports UVM_ERROR", "FAIL", "design or testbench", "Chapter 20.7"),
    ("an assertion fired", "FAIL", "design or testbench", "Chapter 20.9"),
    ("the simulator crashed", "FAIL", "the tool, or the testbench", "Chapter 23.4"),
    ("the clock ran out", "NO_FINISH", "the end condition", "Chapter 23.8"),
    ("the log has no end-of-simulation banner", "NO_FINISH", "the end condition", "Chapter 20.12"),
    ("everything worked", "PASS", "nobody", "—"),
]

print(f"{'what happened':<42} {'status':<14} {'who looks at it'}")
for what, status, who, _ in FAILURE_MODES:
    print(f"{what:<42} {status:<14} {who}")

statuses = sorted({s for _, s, _, _ in FAILURE_MODES})
print()
print("distinct statuses:", statuses)
print("count            :", len(statuses))

what happened                              status         who looks at it
the tool is not on PATH                    ERROR          infrastructure
the run directory cannot be created        ERROR          infrastructure
a required variable is unset               ERROR          infrastructure
no sources matched the filelist pattern    ERROR          infrastructure
the design does not compile                COMPILE_ERROR  the RTL
the testbench reports UVM_ERROR            FAIL           design or testbench
an assertion fired                         FAIL           design or testbench
the simulator crashed                      FAIL           the tool, or the testbench
the clock ran out                          NO_FINISH      the end condition
the log has no end-of-simulation banner    NO_FINISH      the end condition
everything worked                          PASS           nobody

distinct statuses: ['COMPILE_ERROR', 'ERROR', 'FAIL', 'NO_FINISH', 'PASS']
count            : 5

Eleven ways for one run to end, and five statuses — which is the number that matters, because the status is what a report counts and what decides who gets told.

The grouping is the design. Chapter 20.11 argued that FAIL and NO_FINISH are different because they send people to different places, and the third column here is that argument made explicit for every row. Two rows that send the same person to the same place share a status; two that do not, do not.

ERROR is the one that must not be folded in. Chapter 23.9 Section 7 showed why: those four rows did not test anything, so counting them as FAIL inflates the denominator of every pass rate and makes an infrastructure problem look like a design regression.

The honest way to use this table is to treat it as incomplete. A real flow discovers a twelfth row eventually — a licence queue that blocks for an hour, a disk that fills mid-run — and the question each time is not "what new status?" but "does this send somebody somewhere new?" Usually it does not, and a reason field alongside the status is the right place for it, exactly as Chapter 23.8 Section 5 recorded NO_FINISH with reason="TIMEOUT".

7. The Field That Breaks Equality

from dataclasses import dataclass


@dataclass(frozen=True)
class RunResult:
    test: str
    seed: int
    status: str
    runtime_s: float = 0.0


monday = RunResult("axi_smoke", 1839274, "PASS", runtime_s=182.4)
tuesday = RunResult("axi_smoke", 1839274, "PASS", runtime_s=182.5)

print("same test, same seed, same verdict")
print("monday == tuesday:", monday == tuesday)
print("they differ only in:", monday.runtime_s, "vs", tuesday.runtime_s)

same test, same seed, same verdict
monday == tuesday: False
they differ only in: 182.4 vs 182.5

Two runs that agree about everything anybody cares about compare unequal, because one took a tenth of a second longer.

A dataclass generates __eq__ from every field, and runtime_s is the one field guaranteed to differ between two runs. So every "did anything change since last week?" comparison — the question Chapter 24.6 built a whole record format around — reports a change, every single time, and the comparison becomes useless.

The fix is one argument:

from dataclasses import dataclass, field, fields, asdict


@dataclass(frozen=True)
class RunResult:
    test: str
    seed: int
    status: str
    runtime_s: float = field(default=0.0, compare=False)


monday = RunResult("axi_smoke", 1839274, "PASS", runtime_s=182.4)
tuesday = RunResult("axi_smoke", 1839274, "PASS", runtime_s=999.9)
failed = RunResult("axi_smoke", 1839274, "FAIL", runtime_s=182.4)

print("monday == tuesday :", monday == tuesday, " (runtimes differ wildly)")
print("monday == failed  :", monday == failed, " (status differs)")
print()
print("still stored      :", monday.runtime_s, tuesday.runtime_s)
print("still serialised  :", "runtime_s" in asdict(monday))
print("compares on       :", [f.name for f in fields(monday) if f.compare])

monday == tuesday : True  (runtimes differ wildly)
monday == failed  : False  (status differs)

still stored      : 182.4 999.9
still serialised  : True
compares on       : ['test', 'seed', 'status']

field(default=0.0, compare=False) keeps the value stored, printed and serialised and removes it from __eq__ alone. The value is still there for a trend report; it just stops being part of what makes two results the same result.

And because frozen=True also supplies __hash__, that decision reaches further than equality:

from dataclasses import dataclass, field


@dataclass(frozen=True)
class RunResult:
    test: str
    seed: int
    status: str
    runtime_s: float = field(default=0.0, compare=False)


runs = [RunResult("axi_smoke", 1839274, "PASS", runtime_s=182.4),
        RunResult("axi_smoke", 1839274, "PASS", runtime_s=201.7),
        RunResult("axi_smoke", 1839274, "FAIL", runtime_s=182.4)]

print("three results, distinct outcomes:", len(set(runs)))
print("the two PASS rows collapsed     :", runs[0] == runs[1])


@dataclass
class Mutable:
    test: str


try:
    {Mutable("x")}
except TypeError as exc:
    print()
    print("a non-frozen dataclass:", type(exc).__name__ + ":", exc)

three results, distinct outcomes: 2
the two PASS rows collapsed     : True

a non-frozen dataclass: TypeError: unhashable type: 'Mutable'

Three results, two distinct outcomes, with no key function written anywhere. "How many different things happened across these hundred runs?" becomes len(set(results)), which Chapter 25.7 uses directly to answer whether one test is stable.

A non-frozen dataclass is unhashable, so none of that works. frozen=True is doing three jobs at once: it prevents rewriting a verdict, it makes the result safe to pass around, and it makes results set-comparable.

8. What the Runner Owns, and What It Must Not Do

OWNS = [
    "creating its own run directory, named for the test and seed",
    "writing the filelist, compile log, sim log, cmd.sh and env.json into it",
    "deciding the status from the return code and the log",
    "returning a RunResult, always",
]
MUST_NOT = [
    ("print a summary", "the caller decides what the user sees — Chapter 22.5"),
    ("decide a process exit code", "one run is not a program — Chapter 22.4"),
    ("retry", "a retry policy belongs to the regression — Module 26"),
    ("raise for a failing test", "a failing test is a result, not an exception"),
    ("write outside its run directory", "two concurrent runs must not interact"),
]

print("the runner owns:")
for item in OWNS:
    print("   " + item)
print()
print("the runner must not:")
for what, why in MUST_NOT:
    print(f"   {what:<26} {why}")

the runner owns:
   creating its own run directory, named for the test and seed
   writing the filelist, compile log, sim log, cmd.sh and env.json into it
   deciding the status from the return code and the log
   returning a RunResult, always

the runner must not:
   print a summary            the caller decides what the user sees — Chapter 22.5
   decide a process exit code one run is not a program — Chapter 22.4
   retry                      a retry policy belongs to the regression — Module 26
   raise for a failing test   a failing test is a result, not an exception
   write outside its run directory two concurrent runs must not interact

The second list is the useful one, and the last two entries carry most of the weight.

A failing test is a result, not an exception. A runner that raises on FAIL forces every caller into a try/except whose job is to convert the exception back into a result — and a regression loop that forgets one loses a test silently. Reserve exceptions for the cases where there is no result to return: Chapter 23.9's ToolError, where no tool ever ran.

Writing only inside its own run directory is what makes Module 27's parallel regression possible at all. A runner that writes a shared latest.log works perfectly until two run at once, and then fails in a way that reproduces for nobody.

9. The Contract, Written Down

from dataclasses import dataclass, field
from typing import Optional

STATUSES = ("PASS", "FAIL", "COMPILE_ERROR", "NO_FINISH", "ERROR")


@dataclass(frozen=True)
class RunResult:
    """What one simulation produced. Immutable; runtime is not part of identity."""
    test: str
    seed: int
    sim: str
    status: str
    errors: tuple = ()
    reason: Optional[str] = None
    returncode: Optional[int] = None
    run_dir: Optional[str] = None
    log_path: Optional[str] = None
    runtime_s: float = field(default=0.0, compare=False)

    def __post_init__(self):
        if self.status not in STATUSES:
            raise ValueError(f"unknown status {self.status!r} — expected one of {STATUSES}")


def run_test(test, seed, *, sim, waves=False) -> RunResult:
    """Run one test once. Never raises for a failing test; always returns a RunResult."""
    raise NotImplementedError("Chapters 25.2 to 25.6")


ok = RunResult("axi_smoke", 1839274, "xrun", "PASS",
               run_dir="runs/axi_smoke_1839274",
               log_path="runs/axi_smoke_1839274/sim.log", runtime_s=182.4)
bad = RunResult("axi_reset", 771002, "xrun", "NO_FINISH", reason="TIMEOUT",
                run_dir="runs/axi_reset_771002", runtime_s=3600.0)

for r in (ok, bad):
    print(f"{r.test:<10} {r.status:<10} reason={str(r.reason):<8} rc={r.returncode} dir={r.run_dir}")

try:
    RunResult("axi_smoke", 1, "xrun", "BROKEN")
except ValueError as exc:
    print()
    print("guard:", exc)

axi_smoke  PASS       reason=None     rc=None dir=runs/axi_smoke_1839274
axi_reset  NO_FINISH  reason=TIMEOUT  rc=None dir=runs/axi_reset_771002

guard: unknown status 'BROKEN' — expected one of ('PASS', 'FAIL', 'COMPILE_ERROR', 'NO_FINISH', 'ERROR')

Four things in that declaration are deliberate.

__post_init__ validates the status against a fixed tuple. A status is a vocabulary, not a free string, and a typo like "Pass" would otherwise flow silently into a report that counts zero passes. Chapter 21.7 made the same argument about configuration — validate at the boundary, where the message can name the field.

reason is optional and separate from status. Chapter 23.8 Section 5's shape: the five statuses stay fixed so a summary never changes, and a new way of not finishing adds a reason rather than a sixth column.

run_dir sits alongside log_path, because Chapter 23.7 Section 7 found that a record naming only the log sends somebody hunting for the waveform.

run_test raises NotImplementedError naming the chapters that fill it in. The contract is real and testable now — the result type can be constructed, validated and compared before a single simulator is involved, which is what makes the next five chapters implementations rather than design sessions.

10. Predict, Run, Observe

from dataclasses import dataclass, field, fields, asdict, FrozenInstanceError
import json


@dataclass(frozen=True)
class R:
    test: str
    status: str
    runtime_s: float = field(default=0.0, compare=False)


def f(a, b, *, c, d=1):
    return (a, b, c, d)


a = R("t", "PASS", 1.0)
b = R("t", "PASS", 9.0)
c = R("t", "FAIL", 1.0)

# A
print("A:", a == b)

# B
print("B:", a == c)

# C
print("C:", len({a, b, c}))

# D
print("D:", [x.name for x in fields(a) if x.compare])

# E
print("E:", json.dumps(asdict(a), sort_keys=True))

# F
try:
    a.status = "FAIL"
except FrozenInstanceError:
    print("F: FrozenInstanceError")

# G
try:
    f(1, 2, 3)
except TypeError as exc:
    print("G:", type(exc).__name__, "- raised at the call site")

# H
print("H:", f(1, 2, c=3))

The verified output:

A: True
B: False
C: 2
D: ['test', 'status']
E: {"runtime_s": 1.0, "status": "PASS", "test": "t"}
F: FrozenInstanceError
G: TypeError - raised at the call site
H: (1, 2, 3, 1)

A — compare=False keeps a runtime difference from making two results different.

B — a status difference still does.

C — so three results are two distinct outcomes, via frozen's __hash__.

D — exactly the comparing fields, and runtime_s is not among them.

E — it is still serialised, so a trend report can read it.

F — a verdict cannot be rewritten.

G — a positional keyword-only argument fails at the call site.

H — and the named form works, with the default filled in.

11. Exercises

Exercise 1 — four questions. Answer all four for a runner in a flow you know. Write them down before looking at the code.

Exercise 2 — the skipped one. Say why enumerating failure modes first changes the other three answers.

Exercise 3 — read the signature. For run_test(test, seed, *, sim, waves=False), name each of the five decisions it encodes.

Exercise 4 — the shift. Insert a timeout_s parameter into a positional signature and describe what happens to existing call sites.

Exercise 5 — keyword-only. Make sim keyword-only and record the TypeError from a positional call. Say where it is raised.

Exercise 6 — the split. Classify six parameters of your own as identity or configuration, and justify two.

Exercise 7 — frozen. Build a frozen result and attempt to change its status. Say what the immutability prevents.

Exercise 8 — tuple, not list. Reproduce the dataclass ValueError and contrast it with Chapter 11.5's function default.

Exercise 9 — JSON shape. Serialise a result with a tuple field and say what it becomes. Note what that means for reading it back.

Exercise 10 — enumerate. List every way one run of your flow can fail. Aim for at least ten rows.

Exercise 11 — group them. Assign each a status, justified by who looks at it. Report how many distinct statuses you needed.

Exercise 12 — the twelfth row. Invent a new failure and decide whether it needs a status or a reason.

Exercise 13 — the equality trap. Build a result with runtime_s in __eq__ and show two identical runs comparing unequal.

Exercise 14 — what breaks. Say which comparison in Chapter 24.6 that would make useless.

Exercise 15 — compare=False. Apply it and confirm the value is still stored and serialised.

Exercise 16 — the set. Count distinct outcomes across five results with varying runtimes. Say why frozen=True is required.

Exercise 17 — must not. For each of the five prohibitions, describe a concrete failure caused by violating it.

Exercise 18 — the ERROR boundary. Explain why ToolError is right inside the runner and wrong escaping from it.

Exercise 19 — validate the vocabulary. Add __post_init__ and show it rejecting a typo'd status.

Exercise 20 — stub it. Write your contract with a NotImplementedError body and a test that exercises the result type alone.

12. Summary

• Answer four questions before writing a runner: inputs, ownership, result, failure modes — and the fourth decides the other three.
• The cost of deciding later is every caller, and Modules 26 to 31 all call this one function.
• test and seed are positional because they identify the run; sim and waves are keyword-only because they configure it.
• A bare * makes everything after it keyword-only, which makes a silently shifted argument impossible when a parameter is added.
• The TypeError lands at the call site, naming the arity, instead of travelling into the function.
• Return a RunResult, not a boolean, a tuple or a return code.
• frozen=True prevents rewriting a verdict, makes the result safe to pass around, and supplies __hash__.
• asdict makes it JSON-ready, and a tuple field serialises as a list.
• A mutable dataclass default is rejected at class creation with the fix in the message — unlike Chapter 11.5's function default, which Python accepts silently.
• Enumerate every way one run can fail, then group by who looks at it; eleven failure modes collapsed to five statuses.
• ERROR must not be folded into FAIL — those runs tested nothing and must not count in a denominator.
• A new failure usually needs a reason, not a sixth status.
• A dataclass compares every field, so runtime_s makes two identical runs unequal — and every "did anything change?" comparison then reports a change on every run.
• field(compare=False) keeps the value stored, printed and serialised while removing it from __eq__.
• len(set(results)) then counts distinct outcomes with no key function, which Chapter 25.7 uses directly.
• A non-frozen dataclass is unhashable, so none of that works without frozen=True.
• The runner must not print, exit, retry, raise for a failing test, or write outside its run directory.
• A failing test is a result; reserve exceptions for when there is no result — and catch ToolError inside the runner, returning status="ERROR".
• Validate the status vocabulary in __post_init__, because a typo'd status silently counts as zero of everything.

The next chapter builds step two of the figure: finding the sources and writing a filelist that is the same on every run of the same commit.