Tcl · Module 25

Headless Batch Simulation

In batch there is nobody to read the log, so the exit status is the whole report. It does not say what you think it says, and this chapter measures exactly what it does say.

Intermediate21 min readtcledapart-iisimulationregressionexit-status

Part II · Module 25 · Chapter 25.4 · Headless Batch Simulation

Interactively, a failing test is obvious: you read the transcript. In batch — a nightly regression, a merge gate, a thousand runs on a farm — nobody reads anything. The exit status is the entire report.

So the question is what the exit status actually means. It is measurable, and the answer is not what most people assume.

1. Running without a GUI

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

vsim -c -do run.do work.fifo_tb        # Questa family: -c is "console", no GUI
vvp sim/fifo_tb.vvp +test=smoke        # Icarus: there is no GUI to suppress

The first line is the registry's own sample and is not executed anywhere in this module — no Questa exists on the machine these chapters were verified on (Chapter 25.1 §1). The second line is real, and everything measured below came from it.

Three things change when the GUI goes away, and all three are the reason this chapter exists:

Nothing is interactive. A command that would have opened a dialog or waited at a prompt either fails or hangs, and a hang in a batch farm is worse than a failure because it consumes a licence until a timeout kills it.
The log is the only record. Whatever was not printed did not happen, as far as anyone will ever know.
The exit status is the only thing the caller sees. A regression driver running 500 simulations reads 500 integers.

2. What the exit status reports

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== what exit status does each ending give? ===
    MODE     EXIT  LAST LINE OF OUTPUT
    finish   0     TEST FAILED
    fatal    1     TEST FAILED
    fatal1   1     TEST FAILED
    fatal2   1     TEST FAILED

3. The same testbench, two designs

The abstract version is unconvincing, so here is a real one. One testbench, one injected defect — the FIFO's full asserts one slot early:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

assign full  = (count == DEPTH - 1);   // DEFECT: asserts one slot early

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== the same testbench against two DUTs ===
  --- good DUT ---
    test=fill_drain seed=1
    fill_drain: 9 checks, 0 errors
    TEST PASSED
    tb/fifo_tb.v:68: $finish called at 170 (1s)
    vvp exit status = 0
  --- bad DUT ---
    test=fill_drain seed=1
    FAIL 90 count after 4 pushes: got 3 expected 4
    FAIL 170 fifo order: got 18 expected 19
    fill_drain: 9 checks, 2 errors
    TEST FAILED
    tb/fifo_tb.v:68: $finish called at 170 (1s)
    vvp exit status = 0

Two errors found, the defect correctly diagnosed, TEST FAILED printed — and exit status 0. The testbench did its job perfectly. The only thing wrong is the channel the verdict travelled on.

One detail in that output is worth noticing, because it is a nice piece of design-understanding: full after 4 pushes did not fail, only count did. With full asserting at 3, the fourth push is blocked, so count stays at 3 — and full is indeed 1 at that moment, so the full check passes. A single defect can satisfy one of your checks while violating another, which is why the count of checks matters as much as the count of failures.

4. `exec` cannot tell the difference either

The natural Tcl reflex is to let catch around exec decide:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

proc run_sim {binary args} {
    set cmd [list vvp $binary]
    foreach a $args { lappend cmd $a }
    set code [catch {exec {*}$cmd} out]
    return [list $code $out]
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== what exec reports for a passing and a failing run ===
    passing  catch code 0   output lines 4
    failing  catch code 0   output lines 6
    both report catch code 0 — the simulator ran, which is all it said

Identical. exec reports the child's exit status, and the child said 0 both times.

5. Read the verdict out of the log

If the status cannot carry the verdict, the log must:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

proc verdict {out} {
    set pass 0 ; set fail 0
    foreach line [split $out "\n"] {
        if {[string match "TEST PASSED*" [string trim $line]]} { incr pass }
        if {[string match "TEST FAILED*" [string trim $line]]} { incr fail }
    }
    if {$fail} { return FAILED }
    if {$pass} { return PASSED }
    return NO-VERDICT
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== so read the verdict out of the log ===
    passing  verdict PASSED
    failing  verdict FAILED

Three decisions in nine lines.

FAILED is checked first. A log containing both — a test that printed a pass line for one phase and a fail for another — must come out FAILED. Ordering the tests so the bad news wins is the only safe precedence.

string match with a trailing *, because the verdict line may carry a suffix, and string match is anchored at both ends by definition (Chapter 5.5) so the * is required rather than decorative.

The third outcome exists. NO-VERDICT is not an error case bolted on; it is what the function returns when neither line is present, and §6 is about why that matters more than either of the other two.

6. The third outcome

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== and a log with no verdict at all ===
    verdict of an empty log      : NO-VERDICT
    verdict of a log that cut off: NO-VERDICT

An empty log and a truncated log both produce NO-VERDICT rather than PASSED.

A log with no TEST FAILED in it is not a log of a passing test.

This is exactly Chapter 24.7 §2's argument, arriving from the simulation side. There, a parser reported PASS on a zero-byte timing report. Here, a verdict function written as "FAILED if the fail line is present, otherwise PASSED" reports a pass for:

a simulation killed by the farm's timeout,
a simulation killed by the out-of-memory reaper,
a licence checkout that failed after the log was created,
a testbench that hit $fatal in a monitor before reaching its summary,
a disk that filled while the log was being written.

Every one of those is a run that produced no verdict, and every one of them is far more likely in a 500-run nightly than in your interactive testing — which is why this defect reliably survives until it is found by a bug escaping to silicon.

Absence of evidence is not evidence of absence. Three outcomes, not two.

7. The gate

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

proc gate {binary args} {
    set r [run_sim $binary {*}$args]
    set ran [lindex $r 0] ; set out [lindex $r 1]
    if {$ran != 0} { return [list 2 "the simulator itself failed"] }
    switch -- [verdict $out] {
        PASSED     { return [list 0 "test passed"] }
        FAILED     { return [list 1 "test failed"] }
        NO-VERDICT { return [list 3 "no verdict in the log — the run did not finish"] }
    }
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== the gate: three outcomes, three exit statuses ===
    passing  exit 0  test passed
    failing  exit 1  test failed
    missing  exit 2  the simulator itself failed

Four distinct statuses for four distinct situations, which is Chapter 15.4's convention applied here:

Status	Meaning	What a driver should do
0	the test passed	continue
1	the test failed	report the design, keep the log
2	the simulator or its inputs failed	report the flow, not the design
3	no verdict — the run did not finish	retry once, then report the infrastructure

1 and 2 must be different, because they go to different people. A failing test is the design team's; a missing binary, a licence timeout or a compile error is the flow's. A regression that returns the same status for both sends every failure to the wrong queue.

3 deserves its own status because it is the one that is usually retried rather than reported. A run that vanished is often infrastructure, and a driver that retries a 3 once and reports a 1 immediately is doing the right thing with both.

8. The two ways to make the status right

The gate above fixes the status outside the simulation. The other option is to fix it inside, and they are not exclusive.

In the testbench, end on $fatal rather than $finish when the test failed:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

if (errors == 0) begin
    $display("TEST PASSED");
    $finish;
end else begin
    $display("TEST FAILED: %0d of %0d checks", errors, checks);
    $fatal(1, "test failed");
end

Now the exit status is 1 and the wrapper's job gets easier. Two caveats, both measured: $fatal gives 1 regardless of its argument (§2), so it cannot encode which failure; and $fatal stops the simulation immediately, so anything the testbench would have done afterwards — a coverage dump, a final report, closing a log file — does not happen. If the end-of-test bookkeeping matters, print it before $fatal, not after.

In the do file, end on quit -code:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

if {[verdict $transcript] eq "PASSED"} { quit -code 0 } else { quit -code 1 }

That is Chapter 25.2 §6's mechanism, and in the Questa family it is the natural place for the decision because the do file can see the transcript.

9. What else a batch run needs

A timeout. A hung simulation holds a licence until something kills it. The driver owns this, not the testbench.
A log per run, named for the run. Chapter 25.5 is about what "named for the run" has to mean, and it is less obvious than it sounds.
No $stop. It drops to an interactive prompt, which in batch means a hung job. Ban it from testbench code rather than hoping.
No GUI command on a batch path. view wave and relatives fail or hang headless; guard them behind the same flag that chose batch mode.
The plusargs recorded. +seed=7 is the difference between a reproducible failure and a ghost, and it belongs in the log and the manifest, not only in the command line that has scrolled away.

10. Exercises

Exercise 1. What three things change when the GUI goes away?

Exercise 2. What does $finish mean, and what does its exit status therefore report?

Exercise 3. What exit status does $fatal(2, "msg") give, and why is the 2 misleading?

Exercise 4. The bad DUT failed count after 4 pushes but passed full after 4 pushes. Explain.

Exercise 5. Why can catch {exec vvp ...} not distinguish a passing from a failing run?

Exercise 6. vvp writes nothing to stderr. Why is that worth knowing, and what would a simulator that did write to stderr do to catch?

Exercise 7. Why is FAILED tested before PASSED in verdict?

Exercise 8. Name three real situations that produce a log with no verdict.

Exercise 9. Why must exit 1 and exit 2 be different statuses?

Exercise 10. Give two drawbacks of ending a failing test with $fatal.

Exercise 11. Why keep the wrapper's log parse even when the testbench already calls $fatal?

Answers.

1. Nothing is interactive, so a prompt becomes a hang; the log becomes the only record; and the exit status becomes the only thing the caller sees.

2. That the simulation reached its end — which it did. Its status reports that the simulator completed, not that the design was correct.

3. Status 1. The first argument is the LRM's diagnostic verbosity, not an exit code, so it cannot be used to encode which failure occurred.

4. With full asserting at count 3, the fourth push is blocked, so count stays 3 — and full genuinely is 1 at that point, so the full check is satisfied. One defect can satisfy one check while violating another.

5. Because exec reports the child's exit status, and $finish returns 0 in both cases.

6. Because exec raises when a command succeeds but writes to stderr, so people expect that to catch the failure — and it never fires here. A simulator that wrote its summary to stderr would make catch return 1 on a passing test: the opposite false signal, equally wrong.

7. Because a log containing both lines must come out FAILED. Letting the bad news win is the only safe precedence.

8. Any three of: a farm timeout kill, an out-of-memory kill, a licence failure after the log was created, a $fatal in a monitor before the summary, or a full disk.

9. Because they go to different people. A failing test belongs to the design team; a missing binary or a licence failure belongs to the flow. One status for both routes every failure to the wrong queue.

10. It gives 1 regardless of its argument, so it cannot say which failure occurred; and it stops immediately, so any end-of-test bookkeeping after it does not run.

11. Because if the testbench died before reaching its own verdict logic, the mechanism that was supposed to report the failure is the mechanism that failed. Only something outside the run can notice, which is what NO-VERDICT is for.

Remember.

In batch the exit status is the entire report, and $finish exits 0 after printing TEST FAILED.
$fatal exits 1 — all forms. $fatal(2, …) is not exit 2; the argument is diagnostic verbosity.
A real defect, a correct diagnosis, and exit 0. The testbench was right; the channel was wrong.
catch {exec …} cannot tell the runs apart, and the stderr-raises behaviour never fires because vvp writes only to stdout.
Read the verdict from the log, test FAILED before PASSED, and use string match with a trailing *.
There are three outcomes, not two. An empty or truncated log is NO-VERDICT, never a pass.
Absence of evidence is not evidence of absence — the same argument as an empty timing report.
Four statuses for four situations: pass, test failed, tool failed, no verdict. 1 and 2 go to different people; 3 is usually retried.
Fix the status inside and outside — $fatal in the bench, quit -code in the do file, and a log parse in the wrapper that trusts neither.
A run cannot validate itself with the mechanism that failed.
Add a timeout, ban $stop, guard GUI commands, and record the plusargs.

Where this fits

Part of the Tcl curriculum.

1. Running without a GUI

2. What the exit status reports

3. The same testbench, two designs

4. exec cannot tell the difference either

5. Read the verdict out of the log

6. The third outcome

7. The gate

8. The two ways to make the status right

9. What else a batch run needs

10. Exercises

Where this fits

4. `exec` cannot tell the difference either