Linux & Bash · Part II · Module 27 · Invoking Simulators and EDA Tools from the Shell

Running a UVM Test from the Command Line

A UVM command line has three audiences — the simulator, the UVM library and your testbench — and they are not distinguishable by looking at it. This chapter sorts the arguments by who consumes them, and measures what each layer does with one it does not recognise. The answer is the reason wrong invocations survive into regression results.

Intermediate22 min readLinuxPart IIUVMPlusargsEDA Tools

Part II · Module 27 · Chapter 27.5 · Running a UVM Test from the Command Line

The command everybody learns first looks like this:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# FORM, not a transcript -- see the note below
./simv +UVM_TESTNAME=axi_burst_test +UVM_VERBOSITY=UVM_LOW +ntb_random_seed=4210

Four arguments, and they go to four different places. One is the executable, one is read by the UVM library, one is read by the UVM library's reporting subsystem, and one is consumed by the simulator's own randomisation. Nothing about the syntax tells you which is which, and when one of them is wrong, what happens depends entirely on which layer was supposed to read it.

What is and is not measured in this chapter

There is no UVM on this machine, and no simulator here can run it. Both halves of that are measured, not assumed:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# no UVM library anywhere
$ find /opt/homebrew/share /usr/local/share /opt/uvm "$HOME" \
>      -maxdepth 4 -iname 'uvm_pkg.sv' 2>/dev/null
$                                   # nothing

# and a UVM driver cannot compile: a virtual interface in a class
$ iverilog -g2012 -o /tmp/uvmish.vvp /tmp/uvmish.sv
/tmp/uvmish.sv:3: syntax error
/tmp/uvmish.sv:3: error: Invalid class item.
I give up.
$ echo $?
4

Worse, a class that does compile here dispatches wrongly and says nothing:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ vvp /tmp/vdisp.vvp
dispatched to: base  (should be derived)

A base-class handle holding a derived object called the base method. Clean compile, wrong answer, no warning — so even a hand-written UVM-shaped library would produce silently incorrect results on this machine.

So every UVM command line in this chapter is a form, written from the documented plusarg interface, and is never presented as output. What is measured is the chapter's actual subject — how each layer treats an argument it does not recognise — using the tools that do exist here. That mechanism is identical; only the plusarg names differ.

1. Four Audiences in One Line

Take the command apart by consumer.

part	consumed by	when
`./simv`	the shell	immediately — it is the program
`+UVM_TESTNAME=axi_burst_test`	the UVM library, at run time	after elaboration, in the build phase
`+UVM_VERBOSITY=UVM_LOW`	the UVM report server, at run time	continuously, as messages are issued
`+ntb_random_seed=4210`	the simulator, not UVM	before any user code runs

The last row is the one that surprises people. The seed is a simulator argument that happens to be spelled like a plusarg, and its name is vendor-specific — a different simulator spells the same concept differently. So a command line that is portable in three of its four parts is not portable in the fourth.

There is a second, invisible division. Some arguments go to the compile, some to the run:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# FORM. The UVM library is a source that must be compiled in.
#   compile: the library, the include path, the language version
vcs -sverilog -ntb_opts uvm -f filelists/sim.f -o simv
#   run:     the test name, the verbosity, the seed
./simv +UVM_TESTNAME=axi_burst_test +UVM_VERBOSITY=UVM_LOW +ntb_random_seed=4210

Getting that split wrong is the most common first failure, and the next section measures what the two stages actually do about it.

2. What Each Layer Does With an Argument It Does Not Know

Seven invocations on the tools that exist here, each a plausible mistake, with the exit status and what happened:

invocation	rc	result
`iverilog … +TESTNAME=smoke`	1	`+TESTNAME=smoke: No such file or directory`
`vvp build.vvp -g2012`	0	ran normally; the compile option was ignored
`vvp … +TESTNAME=smoke +UVM_VERBOSITY=UVM_HIGH`	0	`TEST PASSED`; second plusarg ignored
`vvp -l run.log build.vvp …`	0	ran; `-l` consumed by `vvp` — and it tees
`vvp build.vvp -l run.log …`	0	ran; `-l` not consumed, passed to the simulation
`vvp … +TEST_NAME=fairness`	0	ran `smoke` — the silent default
`vvp … +UVM_TESTNAME=fairness`	0	ran `smoke` — the silent default

One of seven failed loudly. Six ran to completion and exited 0.

Each of those six is worth a sentence, because they fail for different reasons and only one of them is harmless.

A plusarg given to the compiler is a filename. Row one. The compiler's argument grammar has no +TESTNAME in it, so the word falls through to the position where filenames live, and the error you get names a file you never mentioned. This is the good case: it is loud, and the message, once you know to read it this way, tells you exactly what you did.

A compile option given to the runtime is ignored. Row two. -g2012 selects a language version, which is meaningless after compilation, and vvp does not object. Nothing in the output hints that an argument was discarded.

An unread plusarg is ignored. Row three. +UVM_VERBOSITY means nothing to this testbench, and — as Chapter 27.4 established — nothing is positioned to say so.

Option position matters, and getting it wrong is silent. Rows four and five are the same -l in two places. Before the executable, vvp consumes it and writes a log. After the executable, vvp has already stopped parsing its own options and hands the word to the simulation, which ignores it. Two commands, differing only in word order, one of which writes a log file and one of which does not — with the same exit status and the same output.

And rows six and seven are the silent-default case, which is 27.4's finding showing up in its most likely real form: a flow ported from a UVM environment, using UVM spellings, against a testbench that reads its own.

3. The Argument That Must Not Be Portable

Of the four parts in section 1, the seed is the one that cannot be written once and reused. The concept is universal and the spelling is per-vendor:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# FORMS, from each vendor's documented interface -- not run here
./simv +ntb_random_seed=4210              # one vendor's runtime seed
xrun … -svseed 4210                       # another's, and it is a DASH option
vsim … -sv_seed 4210                      # a third's

Note the second and third: those are not plusargs at all. They are ordinary options, which means they are parsed by the tool's own argument grammar — so a wrong spelling there fails loudly, unlike a wrong plusarg. The same concept has opposite failure behaviour depending on which syntax a vendor chose for it.

This is exactly the situation Chapter 27.8 exists to contain: one concept, one interface in your flow, and one place per tool where it is translated.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# the shape of the answer: your flow has ONE spelling, translated once
seed_args() {
  case "$1" in
    (icarus) printf '%s\n' "+SEED=$2" ;;      # a plusarg this testbench reads
    (nvc)    printf '%s\n' "-gSEED=$2" ;;     # an elaboration generic
    (*)      echo "seed_args: unknown tool '$1'" >&2; return 2 ;;
  esac
}

Measured on the two tools that are here:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ seed_args icarus 4210
+SEED=4210
$ seed_args nvc 4210
-gSEED=4210
$ seed_args vcs 4210
seed_args: unknown tool 'vcs'
$ echo $?
2

The third line is the important one. An unknown tool is an error, not a silently empty result — because an empty seed argument is precisely the defect that makes a run unreproducible while looking fine.

4. Verbosity, and Why It Is Not Just Noise

A UVM run's verbosity decides which messages the report server emits:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# FORM
./simv +UVM_TESTNAME=axi_burst_test +UVM_VERBOSITY=UVM_HIGH

Two practical consequences, neither of which is about readability.

Verbosity changes the size of the artifact, so it changes what the flow can afford. A high-verbosity run of a long test produces a log that is large enough to matter on a shared filesystem, and Module 28 is about what that does to a disk. The normal arrangement is low verbosity by default and high verbosity only on a rerun of something that already failed.

Verbosity must not change the verdict. This is the subtle one. If a flow decides pass or fail by searching for message text, then raising the verbosity can change the answer — because at UVM_HIGH there are lines matching ERROR that were not emitted at UVM_LOW. A verdict rule that depends on verbosity is not a verdict rule.

The defence is to count severities from the summary rather than grepping the body:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# the report summary is emitted at every verbosity -- count from there,
# not from the body, so the verdict does not move when verbosity does
sed -n '/UVM Report Summary/,$p' "$rundir/run.log" > "$rundir/summary.txt"

The fixture's testbench does the same thing in miniature — one checks=N errors=M line at the end, emitted regardless of its verbosity setting:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ vvp build.vvp +TESTNAME=smoke +VERBOSITY=LOW
TB  test=smoke seed=1 cycles=40 verbosity=LOW
TB  TEST PASSED  test=smoke checks=3 errors=0

$ vvp build.vvp +TESTNAME=smoke +VERBOSITY=HIGH
TB  test=smoke seed=1 cycles=40 verbosity=HIGH
TB  checks=3 errors=0
TB  TEST PASSED  test=smoke checks=3 errors=0

High verbosity added a line. It did not change the verdict line, and the verdict line is what a flow reads. Module 29 builds the real version of this.

5. A Wrapper for One Test

Putting the chapter together, here is the shape of a single-test launcher. The UVM-specific parts are marked, because on this machine they are forms rather than measurements:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# runtest TEST [SEED] — launch one test into its own run directory.
set -u
test=${1:?runtest: need a test name}
seed=${2:-1}
root=$(cd "$(dirname "$0")/.." && pwd)
rundir="$root/runs/${test}_s${seed}"
mkdir -p "$rundir" || exit 2

# compile-time arguments: the library, the language, the sources.
# These do not vary per run, which is why the compile is done once.
cargs=(-o "$rundir/sim.vvp" -f "$root/filelists/sim.f")
( cd "$root" && iverilog "${cargs[@]}" ) > "$rundir/compile.log" 2>&1 || {
  echo "runtest: compile failed, see $rundir/compile.log" >&2; exit 1; }

# run-time arguments: everything that identifies THIS run.
#   On a UVM flow the same list would read
#     +UVM_TESTNAME="$test" +UVM_VERBOSITY=UVM_LOW <vendor seed option>
#   and the structure below is unchanged.
rargs=(+TESTNAME="$test" +SEED="$seed" +VERBOSITY="${VERBOSITY:-LOW}" +STRICT)

# write down what we asked for, so it can be compared with what happened
{ printf 'test %s\nseed %s\n' "$test" "$seed"
  printf 'rargs %s\n' "$(printf '%s ' ${rargs[@]+"${rargs[@]}"})"
} > "$rundir/config.txt"

( cd "$rundir" && exec vvp sim.vvp ${rargs[@]+"${rargs[@]}"} ) > "$rundir/run.log" 2>&1
rc=$?

# the verdict comes from the log; the status is recorded beside it, not trusted
want=$test
got=$(sed -n 's/^TB  test=\([^ ]*\).*/\1/p' "$rundir/run.log" | head -1)
if [ "$want" != "$got" ]; then
  printf 'runtest: MISMATCH asked for %s, ran %s\n' "$want" "${got:-<nothing>}" >&2
  exit 1
fi
if   grep -q 'TEST PASSED' "$rundir/run.log"; then verdict=PASS
elif grep -q 'TEST FAILED' "$rundir/run.log"; then verdict=FAIL
else verdict=NO-VERDICT; fi
printf '%-10s seed=%-5s rc=%-3s %s\n' "$test" "$seed" "$rc" "$verdict"
[ "$verdict" = PASS ]

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ runtest smoke
smoke      seed=1     rc=0   PASS
$ runtest fairness 7
fairness   seed=7     rc=0   PASS
$ runtest bogus
bogus      seed=1     rc=0   FAIL
$ echo $?
1

The last pair is the point of the whole design: rc=0 from the tool, FAIL from the log, and 1 from the wrapper. The wrapper's status is the one a caller can gate on, and it was derived from the log rather than inherited from the tool.

Practice

Take a UVM command line apart by consumer, and say which of its arguments go to the compile and which to the run.
Give a plusarg to the compiler and read the error. Explain why it names a file.
Give a compile option to the runtime and confirm it is ignored with status 0.
Put -l before and after the executable and compare the two runs. Check whether a log file appeared in each case.
Confirm -l tees by counting the lines on standard output and in the log file.
Count how many of seven wrong invocations produce a non-zero status. Then decide which of the remaining six you could detect from a log alone.
Write seed_args for the two tools you have and confirm an unknown tool returns 2 rather than an empty string.
Run the same test at two verbosities and confirm the verdict line is identical while the line count is not.
Build a verdict rule by grepping the body for ERROR, then run at high verbosity and see whether the verdict moves.
Write runtest and confirm it exits 1 on a failing test whose tool status is 0.
Break the plusarg spelling inside runtest and confirm the mismatch check catches it before the verdict is computed.

Check Your Understanding

Name the four consumers of ./simv +UVM_TESTNAME=t +UVM_VERBOSITY=UVM_LOW +ntb_random_seed=42.
Which of those four is not portable between simulators, and why is that surprising given its syntax?
Of seven wrong invocations measured here, how many failed loudly?
Why does a plusarg handed to the compiler produce a complaint about a file?
What is the difference between vvp -l log a.vvp and vvp a.vvp -l log?
Why does -l teeing rather than redirecting lead to a wrong number in a report?
Why do some vendors' seed arguments fail loudly when misspelled while others do not?
Why must seed_args treat an unknown tool as an error rather than returning nothing?
How can raising the verbosity change a verdict, and what rule prevents it?
In the final example the tool exited 0 and the wrapper exited 1. Where did the 1 come from?

Answers

The shell runs ./simv; the UVM library reads +UVM_TESTNAME; the UVM report server reads +UVM_VERBOSITY; the simulator consumes +ntb_random_seed.
The seed. It is spelled like a plusarg, which suggests it is read by the testbench and therefore portable, but it is a vendor-specific simulator argument.
One. The other six ran to completion and exited 0.
Because the compiler's grammar has no such option, so the word falls through to the position where filenames are expected, and the error names it as a missing file.
Before the executable, vvp consumes -l and writes a log; after it, vvp has stopped parsing its own options and passes the word to the simulation, which ignores it. Same status, same output, no log file.
Because the output then exists in two places, and a count of matching lines "across the logs" counts every line twice.
Because a dash option is parsed by the tool's own argument grammar, which rejects what it does not recognise, while a plusarg is held and offered to the design, which is under no obligation to ask for it.
Because an empty seed argument produces a run that looks fine and is not reproducible — the exact defect the function exists to prevent.
If the verdict is decided by searching the message body, a higher verbosity emits lines that match. Counting severities from the end-of-run report summary, which is emitted at every verbosity, prevents it.
From the log. The wrapper classified TEST FAILED as FAIL and returned non-zero itself, rather than passing on the tool's status.

6. Remember

One command line, four consumers: the shell, the UVM library, its report server, and the simulator itself. The syntax does not distinguish them.
The seed is a simulator argument wearing plusarg syntax, and its spelling is per-vendor.
Of seven wrong invocations, one failed loudly and six exited 0. Only the loud one stops your flow.
A plusarg given to the compiler becomes a filename; a compile option given to the runtime is discarded in silence.
Option position matters silently: the same -l before or after the executable either writes a log or does nothing.
vvp -l tees rather than redirects, which double-counts if you also redirect.
A dash option fails loudly when misspelled; a plusarg does not. Same concept, opposite behaviour.
A verdict must not move when verbosity moves — count from the end-of-run summary, never from the message body.
Translate per-tool spellings in one place, and make an unknown tool an error rather than an empty string.

7. What Is Next

The test runs and the verdict is sound. Chapter 27.6 turns on the one option that changes a run's cost rather than its result — waveform dumping — and measures what it actually costs, including which of the two obvious ways to reduce it works.

Where this fits

Part of the Linux & Bash curriculum.