One Wrapper, Several Simulators

Part II · Module 27 · Chapter 27.8 · One Wrapper, Several Simulators

A project acquires a second simulator for ordinary reasons: a licence shortage, a formal tool that needs a different front end, an open-source tool for continuous integration, a vendor evaluation. The flow then forks, and the fork is permanent — every subsequent change happens twice, and the two copies drift in exactly the way Chapter 26.6 measured for filelists.

This chapter is unusual in Part II in that it is entirely measurable here, because two real simulators are installed and they disagree about almost everything.

1. How Far Apart Two Tools Actually Are

Not one row of that is a cosmetic difference. The stage count changes the control flow; the test-name mechanism changes when the choice is made, which — section 4 — changes the shape of a regression; and the waveform rows change what Chapter 27.7's checks can do.

The instinct is to abstract the command. That fails immediately, because one tool has two commands and the other has three. The thing to abstract is the job: get this test running and tell me what happened.

2. The Interface First

Decide what the flow asks for, in the flow's own vocabulary, before looking at any tool:

runsim -s <simulator> [-t TEST] [-S SEED] [-d RUNDIR] [-D]

Six decisions are already made by that line.

The simulator is an argument, not an environment variable. It appears in the command that gets recorded (Chapter 27.9), and it cannot be set by accident in a shell profile.

The test and seed are the flow's concepts. How they reach the tool is the tool's problem, which is the entire point.

There is a run directory, defaulting to one derived from the other arguments — Chapter 27.3.

Dumping is one boolean, because that is what the flow knows. Which plusarg or option it becomes is per-tool.

There is no option for "stage". The wrapper runs all of them, because a partial run is not a thing the flow wants.

The output is one line. A wrapper whose output format depends on which tool ran has not abstracted anything.

3. The Implementation, With Its Two Real Bugs

#!/bin/bash
# runsim — one interface over two simulators that do not agree on anything:
#   icarus: compile(+elaborate) then run, Verilog, plusargs at run time
#   nvc:    analyse, elaborate, run as three commands, VHDL, generics
# Usage: runsim -s icarus|nvc [-t TEST] [-S SEED] [-d RUNDIR] [-D]
set -u
sim=icarus test=smoke seed=1 rundir="" dump=0
while getopts "s:t:S:d:D" o; do
  case "$o" in
    (s) sim=$OPTARG ;; (t) test=$OPTARG ;; (S) seed=$OPTARG ;;
    (d) rundir=$OPTARG ;; (D) dump=1 ;;
    (*) echo "runsim: bad option" >&2; exit 2 ;;
  esac
done
root=$(cd "$(dirname "$0")/.." && pwd)
: "${rundir:=$root/runs/${sim}_${test}_s${seed}}"
mkdir -p "$rundir" || exit 2

case "$sim" in
  (icarus)
    stages="compile run"
    # Module 26's rule: source paths inside a filelist resolve against the CWD,
    # so the compile runs at $root and only the OUTPUT goes to the run directory.
    ( cd "$root" && iverilog -o "$rundir/sim.vvp" -f "$root/filelists/sim.f" ) \
        > "$rundir/compile.log" 2>&1
    crc=$?
    echo "compile rc=$crc" >> "$rundir/stages.txt"
    [ $crc -ne 0 ] && { echo "runsim: compile failed, see $rundir/compile.log" >&2; exit $crc; }
    args=(+TESTNAME="$test" +SEED="$seed" +STRICT)
    [ "$dump" = 1 ] && args+=(+DUMP +DUMP_FILE=dump.vcd)
    ( cd "$rundir" && vvp sim.vvp "${args[@]}" ) > "$rundir/run.log" 2>&1
    rrc=$?
    echo "run rc=$rrc" >> "$rundir/stages.txt"
    ;;
  (nvc)
    stages="analyse elaborate run"
    ( cd "$rundir" && nvc --work=work -a "$root/vhdl/arb.vhd" "$root/vhdl/tb_arb.vhd" ) \
        > "$rundir/analyse.log" 2>&1
    arc=$?; echo "analyse rc=$arc" >> "$rundir/stages.txt"
    [ $arc -ne 0 ] && { echo "runsim: analyse failed, see $rundir/analyse.log" >&2; exit $arc; }
    # VHDL has no plusargs. The equivalent is a GENERIC, and a generic is set at
    # ELABORATION — so changing the test name costs a re-elaboration here, where
    # the Icarus side pays nothing. Same -t option, different stage.
    ( cd "$rundir" && nvc --work=work -e -gTESTNAME="$test" -gSEED="$seed" tb_arb ) \
        > "$rundir/elaborate.log" 2>&1
    erc=$?; echo "elaborate rc=$erc" >> "$rundir/stages.txt"
    [ $erc -ne 0 ] && { echo "runsim: elaborate failed, see $rundir/elaborate.log" >&2; exit $erc; }
    nvcargs=()
    [ "$dump" = 1 ] && nvcargs+=(--wave=dump.fst)
    # ${a[@]+"${a[@]}"} — the only expansion that is BOTH safe under set -u with
    # an empty array in bash 3.2 AND passes zero arguments rather than one empty.
    ( cd "$rundir" && nvc --work=work -r ${nvcargs[@]+"${nvcargs[@]}"} tb_arb ) \
        > "$rundir/run.log" 2>&1
    rrc=$?; echo "run rc=$rrc" >> "$rundir/stages.txt"
    ;;
  (*) echo "runsim: unknown simulator '$sim' (icarus|nvc)" >&2; exit 2 ;;
esac

# One verdict rule for both tools: the log decides, never the exit status alone.
if   grep -q 'TEST PASSED' "$rundir/run.log"; then verdict=PASS
elif grep -q 'TEST FAILED' "$rundir/run.log"; then verdict=FAIL
else verdict=NO-VERDICT; fi
printf '%-7s %-9s seed=%-4s stages=[%s] run_rc=%-3s %s\n' \
       "$sim" "$test" "$seed" "$stages" "$rrc" "$verdict"
[ "$verdict" = PASS ] || exit 1

Running it across both tools and three tests:

$ for s in icarus nvc; do for t in smoke fairness bogus; do runsim -s $s -t $t -D; done; done
icarus  smoke     seed=1    stages=[compile run] run_rc=0   PASS
icarus  fairness  seed=1    stages=[compile run] run_rc=0   PASS
icarus  bogus     seed=1    stages=[compile run] run_rc=0   FAIL
nvc     smoke     seed=1    stages=[analyse elaborate run] run_rc=0   PASS
nvc     fairness  seed=1    stages=[analyse elaborate run] run_rc=0   PASS
nvc     bogus     seed=1    stages=[analyse elaborate run] run_rc=0   FAIL

Six runs, one output format, and the verdicts agree across tools because the designs agree — which is the only reason worth having two tools in the first place.

4. The Disagreement That Changes the Flow's Shape

The test name is a run-time plusarg on one tool and an elaboration generic on the other. The wrapper hides that, and it cannot hide its consequence.

# icarus: the test name is a RUNTIME plusarg -> compile once, run N times
compile            57 ms  (once)
run smoke          22 ms
run fairness       16 ms
run smoke          11 ms
run fairness       15 ms
TOTAL for 4 tests  121 ms

# nvc: the test name is an ELABORATION generic -> analyse once, elaborate+run N times
analyse            139 ms  (once)
elaborate smoke    41 ms + run 41 ms
elaborate fairness 23 ms + run 76 ms
elaborate smoke    27 ms + run 28 ms
elaborate fairness 23 ms + run 64 ms
TOTAL for 4 tests  462 ms

The practical consequence for the wrapper is that it should expose batch as a concept, so the per-tool back end can amortise whatever that tool amortises:

# the shape the next module needs: the back end decides what is per-batch
#   icarus: compile once, then loop runs
#   nvc:    analyse once, then loop (elaborate + run)
runsim_batch() {
  local sim=$1; shift
  case "$sim" in
    (icarus) compile_once || return $?; for t in "$@"; do run_one "$t"; done ;;
    (nvc)    analyse_once || return $?; for t in "$@"; do elab_and_run "$t"; done ;;
  esac
}

Chapter 31.1 builds this out properly. The point here is that the abstraction has to be at the right level: a wrapper that only knows how to run one test forces both tools into the worse of the two shapes.

5. Making a Dropped Option Impossible

The silent -t was not caught by any exit status, so the defence cannot be a status check. Three mechanisms, in increasing strength.

Declare what each back end supports, and fail on what it does not.

# the honest version of "simulator agnostic": say what is not supported
case "$sim:$dump" in
  (nvc:1) : ;;                 # supported, via --wave
  (icarus:1) : ;;              # supported, via a plusarg the testbench reads
esac
case "$sim" in
  (nvc) [ -n "${coverage:-}" ] && { echo "runsim: -c (coverage) is not supported for nvc" >&2; exit 2; } ;;
esac

An unsupported option must be an error, never a silent no-op. This is the same rule as Chapter 27.4's misspelled plusarg, arriving one layer up.

Make each back end echo what it passed on. The wrapper writes down its translation, so the mapping is in the record rather than in the reader's memory:

printf 'sim %s\ntest %s\nseed %s\nmechanism %s\n' \
  "$sim" "$test" "$seed" \
  "$(case $sim in (icarus) echo "+TESTNAME plusarg at run" ;; (nvc) echo "-gTESTNAME generic at elaborate" ;; esac)" \
  > "$rundir/config.txt"

And the one that actually caught the bug: compare the artifacts across back ends. Two tools running the same two designs should produce different results for different tests. If they do not, an option was dropped:

#!/bin/bash
# xcheck RUNS — do different tests actually produce different runs?
#
# A back end that silently ignores -t makes every test identical. No exit
# status reveals that: the runs pass, including ones that should fail.
# The signature is what the run EXAMINED, which differs per test by design.
set -u
base=${1:?usage: xcheck RUNS}
rc=0
for sim in icarus nvc; do
  n=0; missing=0; sigs=""
  for d in "$base/${sim}_"*/; do
    [ -d "$d" ] || continue
    n=$((n+1))
    c=$(sed -n 's/.*checks=\([0-9][0-9]*\).*/\1/p' "$d/run.log" 2>/dev/null | tail -1)
    if [ -z "$c" ]; then
      # no check count at all: the run did not get far enough to have one.
      # That is a signature too, but an absent one -- count it separately
      # rather than letting an empty string pose as a distinct value.
      missing=$((missing+1))
      sigs="$sigs
<none> $(basename "$d")"
    else
      sigs="$sigs
$c $(basename "$d")"
    fi
  done
  [ "$n" = 0 ] && { printf '%-7s no runs found\n' "$sim"; continue; }
  u=$(printf '%s\n' "$sigs" | grep -v '^$' | awk '{print $1}' | sort -u | grep -c .)
  printf '%-7s %s runs, %s distinct signatures (%s with no check count)\n' "$sim" "$n" "$u" "$missing"
  if [ "$n" -gt 1 ] && [ "$u" -eq 1 ]; then
    printf '  FAIL every %s run examined the same thing — an option is being ignored\n' "$sim"
    printf '%s\n' "$sigs" | grep -v '^$' | sed 's/^/    /'
    rc=1
  fi
done
exit $rc

$ xcheck runs
icarus  3 runs, 3 distinct signatures (0 with no check count)
nvc     3 runs, 3 distinct signatures (1 with no check count)
$ echo $?
0

And on the broken version — -gTESTNAME removed from the nvc back end, nothing else changed:

$ xcheck runs2
icarus  3 runs, 3 distinct signatures (0 with no check count)
nvc     3 runs, 1 distinct signatures (0 with no check count)
  FAIL every nvc run examined the same thing — an option is being ignored
    3 nvc_bogus_s1
    3 nvc_fairness_s1
    3 nvc_smoke_s1
$ echo $?
1

Look at what the broken wrapper had reported for those three runs:

$ for d in runs2/nvc_*/; do
>   printf '%-22s %s\n' "$(basename $d)" "$(grep -ohE 'TEST (PASSED|FAILED).*' $d/run.log | head -1)"
> done
nvc_bogus_s1           TEST PASSED checks=3
nvc_fairness_s1        TEST PASSED checks=3
nvc_smoke_s1           TEST PASSED checks=3

bogus is a test that must fail, and it passed — because the generic was never set, so the testbench ran smoke three times under three names. Three green verdicts, three identical 549-byte dumps, and a test designed to fail reporting success.

That is a cross-check in the sense of Chapter 26.5: a second, independent measurement of something the first measurement had no way to question. The verdicts said PASS three times; the signatures said the three runs were one run.

6. What Belongs Behind the Abstraction, and What Does Not

The boundary is easy to get wrong in both directions.

Behind it go the stage count, the command names, the option spellings, the intermediate artifact's shape, the waveform format and its enabling mechanism, and the per-tool amortisation shape from section 4. These are all answers to "how", and they are the whole reason the abstraction exists.

In front of it stay the test name, the seed, the run directory, the dumping decision, and the verdict rule. These are the flow's own vocabulary.

And some things must not be abstracted at all. The clearest case is the verdict's evidence. Both back ends in runsim end at the same three lines:

if   grep -q 'TEST PASSED' "$rundir/run.log"; then verdict=PASS
elif grep -q 'TEST FAILED' "$rundir/run.log"; then verdict=FAIL
else verdict=NO-VERDICT; fi

That is shared deliberately, and it only works because both testbenches were written to print the same verdict line. The portability is in the testbenches, not in the wrapper — the wrapper can only translate options, and it cannot make two tools agree about what a pass looks like. Pushing a per-tool verdict rule into the back ends is the usual mistake, and it produces a flow where the two tools' pass rates are not comparable, which removes the only benefit of having both.

A second thing not to abstract: the per-tool logs. stages.txt names the stages by the names that tool uses — compile run for one and analyse elaborate run for the other — because when a stage fails, the engineer needs the tool's own vocabulary to look it up in the tool's own documentation. A wrapper that renames analyse to compile for uniformity has made its logs harder to act on in exchange for a cosmetic consistency nobody asked for.

Practice

1. Tabulate two tools you have across stage count, option spellings and intermediate form. Count the rows that differ.
2. Write down the interface first — the arguments your flow wants — without looking at either tool's manual.
3. Explain why abstracting the command fails when one tool has two commands and another has three.
4. Implement one back end and confirm the output line does not mention the tool's own vocabulary except in the stage list.
5. Reproduce the filelist base-directory bug by compiling from inside the run directory, then fix it.
6. Reproduce the empty-array bug by making the optional waveform array empty under set -u, then fix it with the + expansion.
7. Remove the generic from the nvc back end so -t is ignored, and confirm every run still reports PASS.
8. Then write xcheck and confirm it catches that, where no status could.
9. Measure the per-test cost both ways: one fixed stage plus N runs, against one fixed stage plus N elaborate-and-run pairs.
10. Explain why the ratio you measured should not be quoted, and what should be quoted instead.
11. Try abstracting the verdict rule per tool, then ask whether the two tools' pass rates can still be compared.
12. Rename a stage for uniformity and then try to look that stage up in the tool's documentation.

Check Your Understanding

1. How many stages do the two measured tools have, and why does that defeat abstracting the command?
2. What should be abstracted instead?
3. Why is the simulator an argument rather than an environment variable?
4. Which of the three defects in the author's wrapper produced no error at all?
5. How was that defect found?
6. Why does the test name arriving as a generic rather than a plusarg change the shape of a regression?
7. Why should the measured 121 ms against 462 ms not be quoted as a ratio?
8. Why will the structural difference get worse rather than better as a design grows?
9. What must happen when a back end does not support an option the interface offers?
10. Why is the verdict rule shared between back ends rather than per-tool?
11. Why should stages.txt keep each tool's own stage names?

7. Remember

• Two real tools disagreed on stage count, language, option spelling, intermediate form and waveform format — every row of the comparison is behavioural.
• Abstract the job, not the command, because the number of commands is itself one of the differences.
• Define the interface in the flow's vocabulary first, then implement back ends against it.
• A uniform interface will happily accept an option a back end drops. That defect has no exit status: three runs passed and two tests were one test.
• Catch it by comparing artifacts across runs — distinct tests must produce distinct evidence.
• An unsupported option must be an error, not a no-op.
• Where an option takes effect changes the flow's shape: a run-time plusarg amortises one compile over N tests; an elaboration generic costs N elaborations.
• Quote the structure, not the ratio — and expect the structural gap to widen as designs grow.
• Share the verdict rule and keep each tool's own stage names. Portability lives in the testbenches; the wrapper only translates options.

8. What Is Next

The wrapper works over two tools and the verdict is comparable between them. Chapter 27.9 closes the module with the habit that makes any of it reproducible a month later — writing down the exact command, environment and tool identity, and proving the record is good enough to replay.