Tcl · Module 25

Regression Bookkeeping in Tcl

A regression is believable when you can go back to any single run and see what it did. Two defects in this chapter's own code show how easily that stops being true.

Advanced23 min readtcledapart-iisimulationregressionverification

Part II · Module 25 · Chapter 25.5 · Regression Bookkeeping in Tcl

Chapter 25.4 made one run report its verdict honestly. A regression is hundreds of those, and the new problem is not running them — it is being able to believe the result afterwards.

Everything in this chapter ran for real: Icarus Verilog driving the Module 25 FIFO, with Tcl as the regression layer. Both of the defects it discusses are defects I found in my own code while writing it, which is why they are here rather than a list of good practices.

1. What a run's record must contain

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

proc run_one {binary test seed logdir} {
    set log [file join $logdir "[run_id $binary $test $seed].log"]
    set t0 [clock milliseconds]
    set ran [catch {exec vvp $binary +test=$test +seed=$seed} out]
    set ms [expr {[clock milliseconds] - $t0}]
    set fh [open $log w] ; puts $fh $out ; close $fh
    if {$ran != 0} {
        return [dict create test $test seed $seed binary $binary \
                    verdict SIM-ERROR ms $ms log $log]
    }
    return [dict create test $test seed $seed binary $binary \
                verdict [verdict $out] ms $ms log $log]
}

Six fields, and each one answers a question somebody asks later:

Field	The question it answers
`test`	which test was this
`seed`	how do I reproduce it
`binary`	which build — which RTL, which compile options
`verdict`	what did it conclude
`ms`	is it getting slower, and did it time out
`log`	where is the evidence

seed is the one people omit and the one that matters most. A randomised test that fails without a recorded seed is a failure nobody can reproduce, which makes it a failure nobody can fix — and the plusarg mechanism from Chapter 25.4 exists precisely so one binary can serve every seed.

verdict comes from the three-outcome function of Chapter 25.4 §5, and SIM-ERROR is a fourth value for "the simulator itself failed" — because Chapter 25.4 §7's argument about routing failures to different people applies to a row in a table as much as to an exit status.

ms is recorded in every row but is left out of the printed tables in this chapter, because it differs on every run — which is itself the point of recording it. A duration that drifts upward across nightlies is how a test that has started timing out announces itself before it starts failing, and a duration that collapses to near zero is how you notice a run that died immediately. It belongs in the manifest and in a trend, not in a table you compare by eye.

2. The regression

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

set PLAN {
    {sim/rand_good.vvp  rand   {1 2 3 4}}
    {sim/good.vvp       smoke  {1}}
    {sim/rand_bad.vvp   rand   {1 2 3 4}}
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== running the regression ===
    TEST    SEED  VERDICT      LOG
    rand    1     PASSED       rand_s1.log
    rand    2     PASSED       rand_s2.log
    rand    3     PASSED       rand_s3.log
    rand    4     PASSED       rand_s4.log
    smoke   1     PASSED       smoke_s1.log
    rand    1     FAILED       rand_s1.log
    rand    2     FAILED       rand_s2.log
    rand    3     FAILED       rand_s3.log
    rand    4     FAILED       rand_s4.log

=== the summary ===
    PASSED        5 of 9
    FAILED        4 of 9
    NO-VERDICT    0 of 9
    SIM-ERROR     0 of 9

The summary reports all four categories, including the two that are zero — for Chapter 25.3 §5's reason. A regression report that lists only the categories with members cannot tell you that NO-VERDICT has stopped being checked.

And every line carries its denominator, of 9, which is the rule from Chapter 22.7 that has now appeared in every module of Part II.

Now look at the LOG column.

3. Nine rows, five logs

rand_s1.log appears twice. So do rand_s2, rand_s3 and rand_s4.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== rows against distinct log files ===
    rows in the manifest : 9
    distinct log files   : 5

=== which names collide ===
    rand_s1.log written by 2 runs:
        sim/rand_good.vvp
        sim/rand_bad.vvp
    rand_s2.log written by 2 runs:
        sim/rand_good.vvp
        sim/rand_bad.vvp
    rand_s3.log written by 2 runs:
        sim/rand_good.vvp
        sim/rand_bad.vvp
    rand_s4.log written by 2 runs:
        sim/rand_good.vvp
        sim/rand_bad.vvp

I built the log name from test and seed. The identity of a run is test, seed and binary.

Four logs were overwritten by the later run of the same test and seed against a different binary. The manifest still has nine rows, each pointing confidently at a file — four of them at files whose contents belong to a different run.

Then the consequence:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== so what does the surviving log say? ===
    TEST FAILED
    but two of the manifest rows that point here say PASSED

A row that says PASSED points at a log that says TEST FAILED. The bookkeeping contradicts itself, and nothing complained at any point: the simulations ran correctly, the verdicts were read correctly, the manifest was written correctly. The only error was a filename built from fewer fields than the thing it names.

This is the worst kind of regression defect, because the regression still works. 5 of 9 passed is the true count. What is destroyed is the ability to go back and look — which is the entire reason the logs exist.

4. The fix, and a check that catches it

Name the file after everything that distinguishes the run:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# The identity of a run is test + seed + BINARY. A name built from fewer
# fields than that collides, and the collision overwrites evidence silently.
proc run_id {binary test seed} {
    return "${test}_[file rootname [file tail $binary]]_s${seed}"
}

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== with the run identity in the name ===
    rows in the manifest : 9
    distinct log files   : 9

    TEST    SEED  BINARY             VERDICT     LOG
    rand    1     rand_good.vvp      PASSED      rand_rand_good_s1.log
    rand    2     rand_good.vvp      PASSED      rand_rand_good_s2.log
    rand    3     rand_good.vvp      PASSED      rand_rand_good_s3.log
    rand    4     rand_good.vvp      PASSED      rand_rand_good_s4.log
    smoke   1     good.vvp           PASSED      smoke_good_s1.log
    rand    1     rand_bad.vvp       FAILED      rand_rand_bad_s1.log
    rand    2     rand_bad.vvp       FAILED      rand_rand_bad_s2.log
    rand    3     rand_bad.vvp       FAILED      rand_rand_bad_s3.log
    rand    4     rand_bad.vvp       FAILED      rand_rand_bad_s4.log

But a fix is not the lesson. The lesson is that nothing was checking, and the check is three lines:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# An independent count: one log per row, or evidence was overwritten.
set logs {}
foreach r $results { lappend logs [dict get $r log] }
if {[llength [lsort -unique $logs]] != [llength $results]} {
    error "[llength $results] run(s) wrote only [llength [lsort -unique $logs]] log file(s) — log names collide" \
          {} {REG LOGCOLLISION}
}

Proven able to fail, by putting the weak run_id back:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== and the self-check fires when the name is too weak ===
    1 / 9 run(s) wrote only 5 log file(s) — log names collide

This is Chapter 24.7 §5's principle applied to a regression's own output.

The runner cannot detect the collision by looking harder at any single run — each run is individually perfect. It needs a second, independent measurement of the same thing: the number of rows, and the number of distinct files those rows point at. Two counts of one population, which must agree.

lsort -unique is doing the work, and it is worth noting why that is the right tool rather than a dict of seen names: the question is "how many distinct values", not "which ones", and lsort -unique answers it in one command (Chapter 6.10).

The same shape has now appeared four times across Modules 24 and 25 — component areas against a stated total, parsed rows against a report's own count, parsed paths against counted blocks, and now manifest rows against distinct logs. It is the most reusable idea in Part II.

5. The second defect: a test that could not fail

The regression above is green on four seeds of rand against the good design and red on four against the bad one, which is what a regression should look like.

An earlier version of that same testbench was green on all eight.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== the broken DUT against two benches that differ in ONE line ===
    SEED   stimulus gated on DUT full/empty gated on the model's count
    1      TEST PASSED                TEST FAILED
    2      TEST PASSED                TEST FAILED
    3      TEST PASSED                TEST FAILED
    4      TEST PASSED                TEST FAILED
    5      TEST PASSED                TEST FAILED
    6      TEST PASSED                TEST FAILED
    7      TEST PASSED                TEST FAILED
    8      TEST PASSED                TEST FAILED
    defect caught on 0 of 8 seeds / 8 of 8 seeds

The one line:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

push = do_push && !full;            // gated on a DUT OUTPUT
push = do_push && (mcount < 4);     // gated on the MODEL's occupancy

The testbench used the DUT's own full output to decide whether to push. That makes the check a tautology.

Follow it through. The injected defect makes full assert one slot early. The bench reads full, so it does not push. The reference model is updated from the same gated push signal, so the model does not record a push either. Model and DUT agree perfectly — about the wrong behaviour.

The design never has to hold four entries, because the bench never asks it to. The defect is unobservable, on every seed, for ever.

And the control case is what makes this precise:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== and both benches pass the correct DUT ===
    SEED   DUT-gated bench            model-gated bench
    1      TEST PASSED                TEST PASSED
    2      TEST PASSED                TEST PASSED
    3      TEST PASSED                TEST PASSED

Both benches pass the correct design. So the first bench is not broken, not misconfigured, and not obviously wrong. It is a working testbench that cannot express this failure — and it would have sat in a regression suite reporting green for years.

The rule that falls out of it:

A testbench must decide its stimulus and compute its expected values from its own model, never from a DUT output. The moment a DUT output appears in either, the DUT is being compared against itself.

That is not always easy — flow control exists precisely so a producer can respond to a consumer — and the honest resolution is that a bench which must respond to a DUT output should check that output against the model independently:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

// respond to the DUT, but also check it
check("full matches the model", full, (mcount == 4));
push = do_push && (mcount < 4);

Now the handshake signal is both used and verified, and a wrong full is caught by the check rather than silently steering the stimulus.

6. What this does to the number "0 of 8"

Worth drawing out, because it changes how to read any regression result.

0 of 8 seeds caught the defect looks like a coverage problem — run more seeds, run longer, widen the constraints. It is not. No number of seeds would have caught it, because the defect was not reachable through that bench at all.

So a green regression has two possible explanations and the result alone cannot distinguish them:

The design is correct.
The tests cannot see the bug.

The way to tell them apart is to make the regression fail on purpose: inject a defect you know about and confirm the suite reports it. If it does not, you have measured your suite rather than your design. That is what the rand_bad.vvp entry in §2's plan is doing in a regression that also runs rand_good.vvp — a deliberately broken build whose job is to stay red.

The same argument appeared in Chapter 24.7 §1: a specimen suite of only broken inputs cannot distinguish a working parser from one that refuses everything, so the good report stays in the suite. Here it is the mirror image — the broken build stays in the suite, so a suite that has stopped catching anything is visible.

7. The manifest

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== the manifest ===
test,seed,binary,verdict,log
rand,1,sim/rand_good.vvp,PASSED,logs/pre/rand_s1.log
rand,2,sim/rand_good.vvp,PASSED,logs/pre/rand_s2.log
rand,3,sim/rand_good.vvp,PASSED,logs/pre/rand_s3.log
rand,4,sim/rand_good.vvp,PASSED,logs/pre/rand_s4.log
smoke,1,sim/good.vvp,PASSED,logs/pre/smoke_s1.log
rand,1,sim/rand_bad.vvp,FAILED,logs/pre/rand_s1.log
rand,2,sim/rand_bad.vvp,FAILED,logs/pre/rand_s2.log
rand,3,sim/rand_bad.vvp,FAILED,logs/pre/rand_s3.log
rand,4,sim/rand_bad.vvp,FAILED,logs/pre/rand_s4.log

That is the pre-fix manifest, kept deliberately, because it is the artefact in which the §3 defect is visible: rand_s1.log on two rows with different verdicts. A manifest makes a collision findable — which is the argument for writing one even when a summary line would do.

Written with the quoting rules of Chapter 24.6, since an ECO-style reason field or a test name with a comma would otherwise corrupt the row. And the units belong in the column names for Chapter 24.6 §6's reason — ms is already named for its unit, which is why that column is safe and a column called duration would not be.

What a regression manifest should also carry, and this one does not:

The git revision of the RTL and of the testbench. Without it, "which build" is a filename rather than a fact. This is Chapter 24.2 §6's provenance argument and Chapter 23.6's netlist-and-log pairing, arriving a third time.
The tool version. A regression result belongs to a simulator release.
A start timestamp, so two nightlies can be told apart when both are in the same directory.
The plusargs in full, not just the seed, since a test's behaviour may depend on others.

8. The review

Before believing a regression result:

Is the seed recorded for every run? A failure without one is not reproducible.
Is the binary recorded, so "which RTL" has an answer?
Does every row point at a distinct log, and is something checking that?
Are all verdict categories reported, including the zeros — especially NO-VERDICT?
Does the suite contain a deliberately broken build that must stay red?
Does any testbench decide stimulus or expected values from a DUT output?
Is the provenance there — RTL revision, testbench revision, tool version, timestamp?
Does the driver's own exit status reflect the regression's outcome, with the Chapter 15.4 caveat that a count of failures is not a usable exit status?

That last point closes Module 25 where Chapter 25.4 started: the mechanism that reports the result is the mechanism most likely to be wrong about it, and the only defence is a second measurement that does not share its assumptions.

9. Exercises

Exercise 1. Name the six fields of a run's record and the question each answers.

Exercise 2. Why is the seed the field people omit and the one that matters most?

Exercise 3. Nine rows produced five log files. What was wrong with the name?

Exercise 4. Why is a PASSED row pointing at a FAILED log the worst kind of regression defect?

Exercise 5. Why can the runner not detect the collision by examining individual runs?

Exercise 6. Why lsort -unique rather than a dict of seen names?

Exercise 7. The bench gated push on the DUT's full. Trace why that made the defect unobservable.

Exercise 8. Both benches pass the correct DUT. Why does that matter to the argument?

Exercise 9. Why would more seeds not have caught the defect?

Exercise 10. What are the two explanations for a green regression, and how do you tell them apart?

Exercise 11. A bench genuinely must respond to a DUT handshake output. How do you keep that from becoming a tautology?

Exercise 12. Why keep the pre-fix manifest in this chapter?

Answers.

1. test (which test), seed (how to reproduce), binary (which build), verdict (what it concluded), ms (is it slowing down, did it time out), log (where the evidence is).

2. Because a randomised failure without its seed cannot be reproduced, and a failure nobody can reproduce is one nobody can fix.

3. It was built from test and seed, but a run's identity is test, seed and binary. Two binaries running the same test and seed wrote to the same file, and the later one overwrote the earlier.

4. Because the regression still works — the simulations, the verdicts and the counts are all correct. What is destroyed is the ability to go back and look, which is the only reason the logs exist, and nothing reports it.

5. Because each run is individually perfect. The defect exists only in the relationship between rows, so it needs a second, independent measurement — the row count against the distinct-file count.

6. Because the question is "how many distinct values", not "which ones", and lsort -unique answers exactly that in one command.

7. The defect makes full assert one slot early; the bench reads full and does not push; the model is updated from the same gated push, so it does not record a push either. Model and DUT agree about the wrong behaviour, and the design is never asked to hold four entries.

8. Because it shows the first bench is not broken or misconfigured. It is a working bench that cannot express this particular failure — which is why it would survive in a suite indefinitely.

9. Because the defect was not reachable through that bench at all. Seeds vary the stimulus within what the bench can generate, and this bench could never generate the condition.

10. Either the design is correct, or the tests cannot see the bug. Tell them apart by injecting a known defect and confirming the suite reports it — a deliberately broken build that must stay red.

11. Use the output to respond, but also check it against the model independently in the same cycle. Then a wrong handshake is caught by a check rather than silently steering the stimulus.

12. Because it is the artefact in which the collision is visible — the same log filename on two rows with different verdicts. It is the argument for writing a manifest rather than only a summary.

Remember.

Six fields per run: test, seed, binary, verdict, duration, log. The seed is the one that makes a failure fixable.
A run's identity includes the binary. A name built from fewer fields collides and overwrites evidence silently.
Nine rows, five logs, and a PASSED row pointing at a FAILED log — with every individual step correct.
Check row count against distinct-log count. The runner cannot see the defect by examining single runs; it needs a second, independent measurement.
That principle has now appeared four times across Modules 24 and 25. It is Part II's most reusable idea.
Report every verdict category including the zeros, especially NO-VERDICT.
Never decide stimulus or expected values from a DUT output. One such line took a defect from 8-of-8 caught to 0-of-8.
Both benches passed the good design — so the tautological one is a working bench that cannot see the bug, not a broken one.
More seeds would not have helped. The defect was unreachable, not unlikely.
A green regression means the design is right or the tests are blind. Keep a deliberately broken build that must stay red.
If a bench must respond to a handshake, check that handshake against the model too.
Carry provenance: RTL revision, testbench revision, tool version, timestamp, and the full plusargs.

Where this fits

Part of the Tcl curriculum.