Makefile · Part II · Module 6 · UVM and Regression Automation
UVM Test Selection
Module 5 built a flow that compiles once and runs a test. A real verification project has dozens of tests sharing one compiled image, and an engineer picks one by name from the command line. This chapter builds that path end to end — and then adds the four-line check that stops a typo from costing you a full compile and run.
Intermediate25 min readMakefileGNU MakePart IIUVMTest Selection
A verification environment is built once and used many times. The design, the testbench, the drivers, the monitors and the scoreboard all compile into a single image, and the thing that changes between runs is small: which test should run. One test writes a register and reads it back. Another hammers the bus with random traffic. Another deliberately provokes an error response.
They share everything except the few lines that decide the stimulus. So they must share the compiled image too — otherwise every test change costs a full compile, which on a real block is the difference between a two-minute answer and a twenty-minute one.
This chapter builds the path from make run TEST=axi_smoke to the simulator, and gives each test somewhere of its own to put its results.
1. How a Simulator Learns Which Test to Run
The mechanism is a plusarg: an argument starting with + that is passed to the simulation, not to the compiler. The compiler never sees it, which is exactly why it costs no rebuild.
In real UVM, the testbench's top module calls run_test() with no argument, and UVM asks the command line which test class to build:
// Each test registers itself with the UVM factory.
class axi_smoke_test extends uvm_test;
`uvm_component_utils(axi_smoke_test)
function new(string name, uvm_component parent);
super.new(name, parent);
endfunction
task run_phase(uvm_phase phase);
// ... drive the smoke sequence ...
endtask
endclass
module tb_top;
initial run_test(); // <- the test name comes from +UVM_TESTNAME
endmodule./simv +UVM_TESTNAME=axi_smoke_test +UVM_VERBOSITY=UVM_MEDIUM # VCS
xmsim tb_top +UVM_TESTNAME=axi_smoke_test # Xcelium
vsim -c tb_top +UVM_TESTNAME=axi_smoke_test -do "run -all" # QuestaNotice what is the same in all three: +UVM_TESTNAME=<name>. The command that launches the simulation differs, the plusarg does not. That is the seam a Makefile should sit on.
The stand-in used for the measurements
Since UVM cannot run here, the measured transcripts come from a plain-Verilog testbench that reads the same plusarg and dispatches on it. It is not UVM and does not pretend to be, but it behaves identically from the Makefile's point of view:
reg [8*48-1:0] testname;
integer seed;
initial begin
if (!$value$plusargs("UVM_TESTNAME=%s", testname)) testname = "axi_smoke";
if (!$value$plusargs("ntb_random_seed=%d", seed)) seed = 1;
$display("UVM_INFO @ 0: reporter [RNTST] Running test %0s...", testname);
case (testname)
"axi_smoke": begin /* write reg0, read it back */ end
"axi_burst": begin /* write all four, read all four */ end
"axi_random": begin /* random data, self-checking */ end
"axi_decode_err": begin /* poke an undecoded address */ end
default: $display("UVM_FATAL @ %0t: reporter [INVTST] no test named %0s in the factory",
$time, testname);
endcase
endThe case on a string is doing real work, and it is worth knowing it is reliable: a packed vector filled by $value$plusargs("%s", ...) compares correctly against string literals both with == and inside a case. Measured on three names, including one that does not exist:
+UVM_TESTNAME=axi_smoke -> case -> smoke
+UVM_TESTNAME=axi_burst -> case -> burst
+UVM_TESTNAME=axi_nope -> case -> default
(no plusarg at all) -> case -> smoke2. The Makefile Side
The Makefile's job is small and entirely mechanical: take a variable from the command line, put it in the plusarg, and put it in the path.
BUILD := build
FILELIST := filelists/all.f
SIMV := $(BUILD)/simv
TEST ?= axi_smoke
SEED ?= 1
RUNDIR := $(BUILD)/$(TEST)/$(SEED)
RUNLOG := $(RUNDIR)/run.log
STATUS := $(RUNDIR)/passed.stamp
.PHONY: all compile run log clean
all: run
compile: $(SIMV)
run: $(STATUS)
$(SIMV): $(FILELIST)
@mkdir -p $(@D)
@echo " [compile] -> $@"
@iverilog -g2012 -f $(FILELIST) -o $@
$(STATUS): $(SIMV)
@mkdir -p $(@D)
@echo " [run] $(TEST) seed=$(SEED) -> $(RUNLOG)"
@vvp $(SIMV) +UVM_TESTNAME=$(TEST) +ntb_random_seed=$(SEED) > $(RUNLOG)
@grep -q "^UVM_ERROR : 0" $(RUNLOG) || { echo " [FAIL] $(TEST) seed=$(SEED)"; exit 1; }
@touch $@
@echo " [PASS] $(TEST) seed=$(SEED)"
log:
@test -f $(RUNLOG) || { echo "no log for $(TEST) seed $(SEED)"; exit 1; }
@cat $(RUNLOG)
clean:
@rm -rf $(BUILD)Everything here was built in Module 5 and is being reused rather than re-explained: ?= so the command line wins (Chapter 3.2), the run directory carrying the identity of the run (Chapter 5.3), and passed.stamp as the target rather than the log (Chapter 5.4).
The only new line is the one that matters:
@vvp $(SIMV) +UVM_TESTNAME=$(TEST) +ntb_random_seed=$(SEED) > $(RUNLOG)One variable, used twice: once in the plusarg, once — via RUNDIR — in the path.
3. Running It
Six invocations in order, on a clean tree:
$ make run
[compile] -> build/simv
[run] axi_smoke seed=1 -> build/axi_smoke/1/run.log
[PASS] axi_smoke seed=1
$ make run TEST=axi_burst
[run] axi_burst seed=1 -> build/axi_burst/1/run.log
[PASS] axi_burst seed=1
$ make run TEST=axi_burst SEED=42
[run] axi_burst seed=42 -> build/axi_burst/42/run.log
[PASS] axi_burst seed=42
$ make run TEST=axi_burst SEED=42
make: Nothing to be done for `run'.
$ make run TEST=axi_flaky SEED=5
[run] axi_flaky seed=5 -> build/axi_flaky/5/run.log
[FAIL] axi_flaky seed=5
make: *** [build/axi_flaky/5/passed.stamp] Error 1
$ make run TEST=axi_nosuch
[run] axi_nosuch seed=1 -> build/axi_nosuch/1/run.log
[FAIL] axi_nosuch seed=1
make: *** [build/axi_nosuch/1/passed.stamp] Error 1The second invocation is the one to dwell on. There is no [compile] line. A completely different test ran against the image built for the first one, because the test name never reached the compiler. That is the whole point of a plusarg, and it is the behaviour that makes an interactive debug session bearable.
The resulting tree keeps every run apart:
build/axi_burst/1/passed.stamp
build/axi_burst/1/run.log
build/axi_burst/42/passed.stamp
build/axi_burst/42/run.log
build/axi_flaky/5/run.log
build/axi_nosuch/1/run.log
build/axi_smoke/1/passed.stamp
build/axi_smoke/1/run.logNote the two directories with a log and no stamp: those are the two failures. The log is there to read, and nothing claims they passed. That is 5.4's design doing its job.
Proving the image really is untouched
"The image is reused" is the chapter's central claim, and the absence of a [compile] line is only indirect evidence — a recipe could in principle rebuild quietly. So check the file itself. Run three different tests and take a digest of the image after each:
make run TEST=axi_smoke > /dev/null; md5 -q build/simv
make run TEST=axi_burst > /dev/null; md5 -q build/simv
make run TEST=axi_random SEED=7 > /dev/null; md5 -q build/simv after axi_smoke : d91b5051a91f6629bb8813d1f51520e7
after axi_burst : d91b5051a91f6629bb8813d1f51520e7
after axi_random : d91b5051a91f6629bb8813d1f51520e7
IDENTICAL - one image served three testsByte-identical, and the modification time never moved either. Three tests, two seeds, one compile. That is the number to quote when somebody proposes putting the test name in a +define+.
4. When the Test Name Is Wrong
Look again at the last invocation above. make run TEST=axi_nosuch did something unhelpful: it ran the simulator, which dutifully reported that no such test exists, and only then did Make fail.
The log explains itself clearly enough:
UVM_INFO @ 0: reporter [RNTST] Running test axi_nosuch...
UVM_INFO @ 0: reporter [SEED] seed=1
UVM_FATAL @ 12000: reporter [INVTST] no test named axi_nosuch in the factory
--- UVM Report Summary ---
** Report counts by severity
UVM_INFO : 2
UVM_WARNING : 0
UVM_ERROR : 1
UVM_FATAL : 0
tb/tb_top.v:123: $finish called at 32000 (1ps)That is a real UVM behaviour too: a name that is not in the factory produces a fatal at the start of the run. Here it cost almost nothing. On a real environment it costs the full elaboration and start-up before telling you that you typed axi_smok.
It also leaves rubbish behind. Measured:
unguarded typo: 55 ms (simulator ran, then Make failed)
unguarded also left build/axi_nope/ behindA directory named after a test that does not exist, which will sit in build/ confusing the next person who lists it.
5. Checking the Name First
The fix is to tell the Makefile which test names are legal. Keep them in a file, because Chapter 6.3 is going to need exactly the same list to drive a whole regression:
# One test per line. Comments and blank lines are ignored.
axi_smoke
axi_burst
axi_random
axi_decode_err
axi_flakyRead it once with :=, then refuse to proceed if TEST is not in it:
TESTLIST := tests/testlist.f
# Read the list once, dropping comments and blank lines.
TESTS := $(shell grep -vE '^[[:space:]]*(\#|$$)' $(TESTLIST))
# Fail while the Makefile is being READ, before anything is built or run.
ifneq ($(TEST),$(filter $(TEST),$(TESTS)))
$(error unknown test "$(TEST)" -- known tests are: $(TESTS))
endif
.PHONY: tests
tests:
@echo "$(TESTS)" | tr ' ' '\n' | sed 's/^/ /'Three pieces, all from Part I. $(shell ...) assigned with := runs the command once (Chapter 3.4). $(filter $(TEST),$(TESTS)) returns $(TEST) if it is in the list and nothing otherwise, so comparing the two is a membership test. ifneq with $(error) aborts at parse time (Chapter 4.2 and Chapter 4.4 §9).
What Make read:
axi_smoke
axi_burst
axi_random
axi_decode_err
axi_flakyFive names: the comment and the blank line are gone. And now the typo:
Makefile.guard:17: *** unknown test "axi_smok" -- known tests are:
axi_smoke axi_burst axi_random axi_decode_err axi_flaky. Stop.It names what you typed, lists what you could have meant, and nothing was compiled or run. Measured against the unguarded version:
guarded typo: 22 ms (no simulator invoked)
unguarded typo: 55 ms (simulator ran, then Make failed)The guard has a hole, and it is worth seeing
That guard looks complete. It is not. Try an empty test name — which is what a wrapper script produces when the variable it meant to pass was never set:
[run] seed=1 -> build//1/run.log
[FAIL] seed=1
make: *** [build/1/passed.stamp] Error 1It ran. The guard did nothing, and the run landed in a directory called build/1/ because the empty name collapsed the path. The exit status was 2, so it failed — but it failed after the run, which is exactly what the guard was supposed to prevent.
The reason is worth understanding, because it is a general trap with $(filter):
TEST=[]
filter result=[]
are they equal? YESWith TEST empty, $(filter $(TEST),$(TESTS)) is empty too — and ifneq is comparing empty against empty, which are equal, so the condition does not fire. A membership test written as a comparison against $(filter) always passes for the empty string. (Writing it the other way round — $(if $(filter $(TEST),$(TESTS)),,$(error ...)), which tests whether the filter result is non-empty — does catch it. Chapter 7.2 §2 measures both forms side by side.)
One more line fixes it, and it has to come first:
$(if $(strip $(TEST)),,$(error TEST is empty -- known tests are: $(TESTS)))
ifneq ($(TEST),$(filter $(TEST),$(TESTS)))
$(error unknown test "$(TEST)" -- known tests are: $(TESTS))
endif$ make run TEST=
Makefile.guard:16: *** TEST is empty -- known tests are:
axi_smoke axi_burst axi_random axi_decode_err axi_flaky. Stop.
$ make run TEST=axi_smok
Makefile.guard:18: *** unknown test "axi_smok" -- known tests are:
axi_smoke axi_burst axi_random axi_decode_err axi_flaky. Stop.
$ make run TEST=axi_smoke
[run] axi_smoke seed=1 -> build/axi_smoke/1/run.log
[PASS] axi_smoke seed=1Empty rejected, typo rejected, real name accepted. And while we are here, two cases the $(filter) test does handle correctly on its own — a name with a space in it, and a name that is merely a prefix of a real test:
$ make run TEST="axi smoke"
*** unknown test "axi smoke" -- known tests are: axi_smoke axi_burst ...
$ make run TEST=axi
*** unknown test "axi" -- known tests are: axi_smoke axi_burst ...$(filter) matches whole words, so axi is not treated as a shortening of axi_smoke. That is the behaviour you want, and it is worth having checked rather than assumed.
6. The # That Breaks $(shell ...)
That \# in the grep pattern is not decoration, and getting it wrong produces one of Make's most confusing errors. Write the pattern the way you would in a shell:
TESTS := $(shell grep -vE '^[[:space:]]*(#|$$)' $(TESTLIST))and Make says:
Makefile:7: *** unterminated call to function `shell': missing `)'. Stop.There is nothing wrong with your brackets. A # starts a comment in a Makefile, even inside quotes, even inside a function call. Make threw away everything from the # onwards, including the closing ), and then complained about the bracket you did write.
Three ways out, all measured:
# BROKEN - the # begins a comment and eats the rest of the line
A := $(shell grep -vE '^[[:space:]]*(#|$$)' $(L))
# WORKS - escape it
B := $(shell grep -vE '^[[:space:]]*(\#|$$)' $(L))
# WORKS - keep the character in a variable
HASH := \#
C := $(shell grep -vE "^[[:space:]]*($(HASH)|$$)" $(L))--- unescaped # ---
t.mk:2: *** unterminated call to function `shell': missing `)'. Stop.
--- escaped \# ---
B=[axi_smoke axi_burst axi_random axi_decode_err axi_flaky]
--- # held in a variable ---
C=[axi_smoke axi_burst axi_random axi_decode_err axi_flaky]And to be sure it is the character rather than anything about the regular expression, the same bare # inside a completely different command fails in exactly the same way:
D := $(shell sed -e 's/[[:space:]]*#.*//' -e '/^[[:space:]]*$$/d' $(L))t4.mk:2: *** unterminated call to function `shell': missing `)'. Stop.Same error, sed instead of grep. The lesson is about Make, not about either tool: a literal # anywhere on a Makefile line must be written \#.
7. Why the List Lives in a File
It would be shorter to write TESTS := axi_smoke axi_burst axi_random directly in the Makefile. The file is better for three reasons that all show up later:
The simulator may need it too. Some flows generate a UVM test package, or a documentation page, or a CI job matrix from the same list. One file, many readers — the same argument Chapter 5.2 made for the source filelist.
It changes more often than the Makefile. Adding a test should not require editing build logic, and a diff that touches only tests/testlist.f is obviously safe to approve.
Comments and grouping become possible. A list file can be organised:
# ---- smoke: must pass before anything else ----
axi_smoke
axi_burst
# ---- random: self-checking, seed-sensitive ----
axi_random
axi_flaky
# ---- error injection ----
axi_decode_errThe grep drops every comment and blank line, so the structure costs nothing at build time. Chapter 6.3 builds the regression on top of exactly this file.
8. Giving the Flow a Front Door
Two small phony targets make the flow usable by somebody who has never read the Makefile:
.PHONY: help tests
help:
@echo "targets:"
@echo " make tests list the tests that can be run"
@echo " make run TEST=<name> [SEED=<n>] run one test"
@echo " make log TEST=<name> [SEED=<n>] print that run's log"
@echo " make clean remove $(BUILD)/"
@echo ""
@echo "tests: $(TESTS)"targets:
make tests list the tests that can be run
make run TEST=<name> [SEED=<n>] run one test
make log TEST=<name> [SEED=<n>] print that run's log
make clean remove build/
tests: axi_smoke axi_burst axi_random axi_decode_err axi_flakyThis is not decoration. A flow that cannot tell you what it can do gets used for one target and ignored for the rest, and the engineer who needed make log re-runs the whole test instead because they did not know it existed. Both help and tests are phony and have no prerequisites, so neither can ever start a simulation — the same rule 5.4 §7 set for make log.
9. Common Mistakes
Putting the test name in a +define+. A define is read at compile time, so every test change becomes a rebuild — Chapter 5.2 §2. The test name is a run-time value and belongs in a plusarg.
One shared output directory for every test. Then the second test overwrites the first one's log. The test name belongs in the path (§2).
No validation of the test name. A typo costs a full compile and run, and leaves a directory named after a test that does not exist (§4).
Comparing a variable against its own $(filter) result. $(filter ,$(TESTS)) is empty and so is $(TEST), so the comparison succeeds and an unset variable sails through — measured, it ran and wrote to build/1/ (§5).
Writing a literal # in a $(shell ...). Make treats it as a comment and reports an unterminated function call that has nothing to do with your brackets (§6).
Assigning $(shell ...) with =. It re-runs the command at every expansion. Use := (Chapter 3.1), and Chapter 6.2 §8 shows what that mistake does to a seed.
Giving help or tests prerequisites. Asking what the flow can do should never launch a simulation (§8).
Trusting the simulator's exit status for a bad test name. The run reported UVM_FATAL in its log and the tool still exited 0 — the verdict comes from reading the log (Chapter 5.4 §5).
10. Check Your Understanding
Question 1. An engineer runs make run TEST=axi_smoke, then make run TEST=axi_burst. The second command prints no [compile] line. Explain why that is correct rather than a sign the flow is broken.
Question 2. A Makefile contains TESTS := $(shell grep -v '^#' tests/testlist.f) and Make reports unterminated call to function 'shell': missing ')'. The brackets are balanced. What is wrong, and what are two ways to fix it?
Question 3. Why should the test-name check use $(error) inside ifneq rather than a test-and-exit 1 line at the top of the run recipe?
Question 4. The test list is read with := rather than =. What would go wrong with = on a list of 200 tests?
11. Exercise
Build the flow in §2, add the guard from §5, and confirm each behaviour. Every transcript below is real output.
Step 1. List what can be run:
make tests axi_smoke
axi_burst
axi_random
axi_decode_err
axi_flakyStep 2. Run one test, then a different one. Confirm the second has no [compile] line:
make run TEST=axi_smoke
make run TEST=axi_burst [compile] -> build/simv
[run] axi_smoke seed=1 -> build/axi_smoke/1/run.log
[PASS] axi_smoke seed=1
[run] axi_burst seed=1 -> build/axi_burst/1/run.log
[PASS] axi_burst seed=1Step 3. Mistype a name. Nothing should be compiled or run:
make run TEST=axi_smokMakefile.guard:17: *** unknown test "axi_smok" -- known tests are:
axi_smoke axi_burst axi_random axi_decode_err axi_flaky. Stop.Step 4. Run a test that genuinely fails, and read its log without typing the path:
make run TEST=axi_flaky SEED=5
make log TEST=axi_flaky SEED=5 [run] axi_flaky seed=5 -> build/axi_flaky/5/run.log
[FAIL] axi_flaky seed=5
make: *** [build/axi_flaky/5/passed.stamp] Error 1
UVM_INFO @ 0: reporter [RNTST] Running test axi_flaky...
UVM_INFO @ 0: reporter [SEED] seed=5
UVM_ERROR @ 55000: uvm_test_top [TEST] addr 4 undecoded - data lostThe UVM_ERROR line is the one the Makefile's grep reacts to; the report summary below it reads UVM_ERROR : 1, which is why no stamp was written.
Step 5. Confirm the failed run left a log and no stamp:
ls build/axi_flaky/5/run.logThen change two things and predict before you look. First, remove the \ from \# in the TESTS line and run make tests — you should get the unterminated-function error, and you should be able to say why without reading §6 again. Second, change TESTS := to TESTS = and add $(warning reading the list) just above it; run make run TEST=axi_smoke and count how many times the warning appears. That count is the number of times the grep ran, and it is the argument for := in one number.
12. Remember
- A verification environment compiles once and runs many tests. The only thing that changes between runs is which test, so that must not be a compile-time value.
+UVM_TESTNAME=<name>is the seam. VCS, Xcelium and Questa launch differently but all read the same plusarg, so a Makefile written around it ports between them.- A plusarg never reaches the compiler, which is exactly why a different test needs no rebuild — measured: no
[compile]line on the second run. - The test name belongs in the output path too, so two tests cannot overwrite each other's results.
- Validate the name before you spend a run.
$(filter $(TEST),$(TESTS))plusifneqand$(error)rejects a typo at parse time, with no tool work at all. - An unvalidated typo costs a full compile and run and leaves a directory named after a test that does not exist.
- Guard the empty string separately. A membership test written as
ifneq ($(TEST),$(filter $(TEST),$(TESTS)))passes for an emptyTEST, because empty equals empty — measured, the run proceeded intobuild/1/. - Measured: one image served three tests byte-identically (same md5, unchanged mtime), which is the evidence behind "a plusarg costs no rebuild".
- A literal
#in a Makefile line starts a comment, even inside$(shell ...)— write it\#, or hold it in a variable. The error you get names the wrong problem. - Keep the test list in a file, not in the Makefile: other tools read it, it changes more often than build logic, and it can carry comments.
- Read that file with
:=, so thegrepruns once rather than once per expansion. - Give the flow a
helpand ateststarget, both phony and both without prerequisites, so asking what it does can never start a simulation. - Be honest about tool limits. UVM is not installed here, so real UVM appears as labelled reference syntax and the measurements come from a stand-in that reads the same plusargs.
One test at a time, chosen by name, with its own directory and its own verdict. The next knob is the one that decides what the test actually does: in constrained-random verification the same test at two different seeds is two different stimulus streams and two genuinely different results. Chapter 6.2 makes the seed a first-class part of the flow — and measures three ways a seed can look right while reproducing nothing.
Where this fits
Part of the Makefile curriculum.