Regression Targets and Test Lists

One test at one seed is a debugging session. A regression is the other thing a verification flow has to do: take every test, run each at several seeds, and come back with one answer. On a real project that is hundreds of runs launched by one command and read as a single number the next morning.

The temptation is to write it as a shell loop. Resist it for one chapter, because Make already knows how to do the hard parts — skipping work that is already done, running independent jobs at the same time, and tracking which outputs exist — and a loop throws all three away.

1. The Test List, Reused

Chapter 6.1 §5 put the legal test names in a file so a typo could be caught. That same file is the regression's input:

# One test per line. Comments and blank lines are ignored.
axi_smoke
axi_burst
axi_random
axi_decode_err

axi_flaky

TESTLIST := tests/testlist.f
TESTS    := $(shell grep -vE '^[[:space:]]*(\#|$$)' $(TESTLIST))
SEEDS    ?= 1

Three things carried over from earlier chapters: := so the grep runs once (Chapter 3.1), the \# escape because a bare # would be a Makefile comment (Chapter 6.1 §6), and ?= on SEEDS so the command line can override it (Chapter 3.2).

SEEDS is a list, which is the new idea. SEEDS=1 is one seed; SEEDS="1 2 3 4 5 6 7 8" is eight.

2. Guarding the List First

Before building anything on top of that list, check it is real. Two failure modes, both measured, both quiet.

A missing list file: grep writes to standard error and TESTS is simply empty. Make does not stop.

grep: tests/nosuchlist.f: No such file or directory
tests: 
seeds: 1
runs : 0

An empty list — a file where every line is a comment — gives the same zero. And the naive report built on it fails with a message about a file nobody mentioned:

/bin/sh: build/report.txt: No such file or directory
make[1]: *** [report-only] Error 1
make: *** [report] Error 2

Neither message says "your test list is empty," which is the actual problem. Two lines fix both:

# A missing list file makes grep write to stderr and leaves TESTS EMPTY -
# Make does not stop, so check the file exists before trusting the list.
$(if $(wildcard $(TESTLIST)),,$(error test list "$(TESTLIST)" does not exist))
TESTS := $(shell grep -vE '^[[:space:]]*(\#|$$)' $(TESTLIST))
$(if $(strip $(TESTS)),,$(error test list "$(TESTLIST)" contains no tests))

$ make report TESTLIST=tests/nosuchlist.f
Makefile.final:8: *** test list "tests/nosuchlist.f" does not exist.  Stop.

$ make report TESTLIST=tests/empty.f
Makefile.final:10: *** test list "tests/empty.f" contains no tests.  Stop.

Both fire at parse time, name the file, and say what is wrong with it. $(if ...) with $(error) is Chapter 4.4 §9's mechanism; $(wildcard ...) as an existence test is Chapter 3.4's.

3. The Cross Product

A regression is every test crossed with every seed. $(foreach) nested twice builds exactly that list of filenames:

RESULTS := $(foreach t,$(TESTS),$(foreach s,$(SEEDS),$(BUILD)/$(t)/$(s)/result.txt))

.PHONY: list
list:
	@echo "tests: $(TESTS)"
	@echo "seeds: $(SEEDS)"
	@echo "runs : $(words $(RESULTS))"

tests: axi_smoke axi_burst axi_random axi_decode_err axi_flaky
seeds: 1
runs : 5

tests: axi_smoke axi_burst axi_random axi_decode_err axi_flaky
seeds: 1 2 3 4 5 6 7 8
runs : 40

Five tests, eight seeds, forty runs — and $(words) counting them is worth having as a target of its own. "How many runs is this about to launch?" is a question you want answered before you start an overnight job, and Chapter 3.4 introduced both functions for exactly this kind of list arithmetic.

The outer $(foreach) walks tests, the inner one walks seeds, and the result is a list of file paths. Those paths are the whole trick: because each one is a distinct file, Make can treat each run as an independent target and apply everything it knows about building files.

4. One Pattern Rule for Every Run

Forty targets do not need forty rules. They have the same shape, so Chapter 4.1's pattern rule covers all of them:

$(BUILD)/%/passed.stamp: $(SIMV)
	@mkdir -p $(@D)
	@t=$(patsubst %/,%,$(dir $*)); s=$(notdir $*); \
	 echo "  [run] $$t seed=$$s"; \
	 vvp $(SIMV) +UVM_TESTNAME=$$t +ntb_random_seed=$$s > $(@D)/run.log; \
	 if grep -q "^UVM_ERROR :  0" $(@D)/run.log; then touch $@; \
	 else echo "  [FAIL] $$t seed=$$s"; exit 1; fi

The clever part is recovering the test name and seed from the target path. The stem % matches whatever sits between build/ and /passed.stamp, so for build/axi_burst/42/passed.stamp the stem is axi_burst/42. From that:

• $(dir $*) is axi_burst/, and $(patsubst %/,%,...) strips the trailing slash → axi_burst
• $(notdir $*) is 42

Both are plain text functions on a string, so they cost nothing and work for any test name. Check it without running anything, using Chapter 4.4 §2's make -n:

make -n build/axi_burst/42/passed.stamp

mkdir -p build/axi_burst/42
t=axi_burst; s=42; \
	 echo "  [run] $t seed=$s"; \
	 vvp build/simv +UVM_TESTNAME=$t +ntb_random_seed=$s > build/axi_burst/42/run.log; \
	 if grep -q "^UVM_ERROR :  0" build/axi_burst/42/run.log; then touch build/axi_burst/42/passed.stamp; else echo "  [FAIL] $t seed=$s"; exit 1; fi

t=axi_burst and s=42, recovered from the filename. One rule now serves every test and every seed, including tests added to the list next month.

Note that the shell variables are $$t and $$s in the Makefile: the doubled dollar leaves a single one for the shell, which is Chapter 1.4 §8's rule. And the whole recipe is one line continued with backslashes, because each recipe line is its own shell — Chapter 4.3 §3 — so t= set on one line would be gone by the next.

Now the regression target is just a list of prerequisites:

.PHONY: regress
regress: $(STAMPS)
	@echo "  [regress] $(words $(STAMPS)) run(s) up to date"

  [compile] -> build/simv
  [run] axi_smoke seed=1
  [run] axi_burst seed=1
  [run] axi_random seed=1
  [run] axi_decode_err seed=1
  [run] axi_flaky seed=1
  [regress] 5 run(s) up to date

One compile, five runs, in one command.

5. The First Failure Stops Everything

Now run the same regression at a seed where one test fails. The test list is reordered so the failing test is second of five, because that is the case that shows the behaviour:

axi_smoke
axi_flaky
axi_burst
axi_random
axi_decode_err

  [run] axi_smoke seed=5
  [run] axi_flaky seed=5
  [FAIL] axi_flaky seed=5
make: *** [build/axi_flaky/5/passed.stamp] Error 1
  make rc=2
  runs attempted : 2 of 5
  stamps         : 1

Two of five. Make did what it always does: a recipe failed, so it stopped. axi_burst, axi_random and axi_decode_err were never attempted at all.

Think about what that means for an overnight regression. You arrive in the morning to one known failure and no information whatsoever about three quarters of your test suite. They are not passing and not failing — they did not run. And the longer the suite, the worse it gets: one broken test in the first ten minutes wastes the whole night.

6. make -k, and Why It Is Not Enough

Chapter 4.3 §8 introduced the flag for exactly this: -k tells Make to keep going after a failed target, doing everything that does not depend on the failure.

  [run] axi_smoke seed=5
  [run] axi_flaky seed=5
  [FAIL] axi_flaky seed=5
make: *** [build/axi_flaky/5/passed.stamp] Error 1
  [run] axi_burst seed=5
  [run] axi_random seed=5
  [run] axi_decode_err seed=5
make: Target `regress' not remade because of errors.
  make rc=2
  runs attempted : 5 of 5
  stamps         : 4

Five of five attempted, four passed, one failed, and Make still exits non-zero so no script mistakes it for success. That is the behaviour we wanted.

It has one real weakness: it depends on the person remembering the flag. make regress and make -k regress do very different things, the difference is invisible until something fails, and the version without the flag is the one people type. A flow whose correctness rests on a command-line option will eventually be run without it.

You can put it in the Makefile with MAKEFLAGS += -k, but that applies -k to everything, including the compile — and a compile should absolutely stop at its first error. So the honest answer is that -k is the right tool used in the wrong place. The better fix is to make the failure not be a failure in Make's eyes at all.

7. A Run That Always Succeeds

Turn it around. A run's job is not to succeed; it is to produce a verdict. Let the recipe always exit zero, and write PASS or FAIL into a file:

RESULTS := $(foreach t,$(TESTS),$(foreach s,$(SEEDS),$(BUILD)/$(t)/$(s)/result.txt))

# A run ALWAYS succeeds. Its verdict goes in result.txt, so one failing test
# never stops the sweep and `-k` is not needed anywhere.
$(BUILD)/%/result.txt: $(SIMV)
	@mkdir -p $(@D)
	@t=$(patsubst %/,%,$(dir $*)); s=$(notdir $*); \
	 vvp $(SIMV) +UVM_TESTNAME=$$t +ntb_random_seed=$$s > $(@D)/run.log 2>&1; \
	 if grep -q "^UVM_ERROR :  0" $(@D)/run.log && ! grep -q "UVM_FATAL @" $(@D)/run.log; \
	 then echo PASS > $@; echo "  [PASS] $$t seed=$$s"; \
	 else echo FAIL > $@; echo "  [FAIL] $$t seed=$$s  ($(@D)/run.log)"; fi

.PHONY: regress
regress: $(RESULTS)

  [compile] -> build/simv
  [PASS] axi_smoke seed=5
  [PASS] axi_burst seed=5
  [PASS] axi_random seed=5
  [PASS] axi_decode_err seed=5
  [FAIL] axi_flaky seed=5  (build/axi_flaky/5/run.log)

Every run attempted, the failing one clearly marked, no flag required, and make exited 0 — because nothing went wrong with the build. A test failing is a result, not a build error.

That last point is the one that deserves thought, because it looks like it contradicts Chapter 5.4, which insisted a failing test must make Make exit non-zero. Both are right, at different levels:

• A single interactive run should fail loudly. make run TEST=x is a question with a yes-or-no answer, and the exit status is that answer.
• A regression should complete and then report. The pass/fail answer belongs to the report, not to each run.

So the exit status moves up one level. Chapter 6.4 puts it on the report target, where it covers all forty runs at once.

8. Running in Parallel

Now the payoff for all that insistence on per-run directories. Every run writes only inside build/<test>/<seed>/, so no two runs touch the same file. That means Make can run them at the same time with -j, and nothing has to change.

Forty runs — five tests at eight seeds — with the image already built, three repetitions:

rep 1: serial 1642 ms | -j8  342 ms | 4.80x
rep 2: serial 1513 ms | -j8  294 ms | 5.15x
rep 3: serial 1439 ms | -j8  281 ms | 5.12x

Roughly five times faster on eight cores, consistently. On a real regression where each run is minutes rather than milliseconds, that is the difference between a sweep that fits in a lunch break and one that does not.

Speed is worthless if the answer changes, so check that too. The report file produced by each:

serial : bb96e0174e73a90651f2c95aaf1aa3aa
-j8    : bb96e0174e73a90651f2c95aaf1aa3aa

Byte-identical. Forty runs in arbitrary order, across eight processes, and the same result.

That is a property of the design, not luck. It holds because:

• each run writes only into its own directory, so there is nothing to race over;
• the report is generated after all runs finish, by walking the test and seed lists in a fixed order, so it does not depend on completion order;
• the runs are genuinely independent — no run reads another run's output.

Break any of those three and -j will start producing results that differ between invocations, which is the hardest class of bug to chase. Module 8 covers parallel-build reliability properly, including the shared resources that do break it.

9. Common Mistakes

Writing the regression as a shell for loop. It throws away skipping, parallelism and dependency tracking — the three things Make is for (§4).

Letting the default stop-on-error behaviour stand. Measured: a five-test regression attempted two tests, and three were never run at all (§5).

Relying on somebody typing -k. The flag works, but the command without it is the one people type, and the difference is invisible until something fails (§6).

Putting -k in MAKEFLAGS. It then applies to the compile too, which must stop at its first error (§6).

Trusting a regression that ran zero tests. A missing or empty list leaves TESTS empty, Make does not stop, and nothing fails (§2).

Writing one rule per test. Forty near-identical rules go stale the moment a test is added. One pattern rule covers them all (§4).

Using $* directly as the test name. The stem is <test>/<seed>; the name needs $(patsubst %/,%,$(dir $*)) and the seed needs $(notdir $*) (§4).

Setting a shell variable on one recipe line and using it on the next. Each line is its own shell; continue with backslashes (Chapter 4.3 §3).

make -j with no number. Unlimited parallelism launches every run at once and will exhaust memory on a real design (§8).

Sharing anything between runs. A shared log, a shared waveform, a shared scratch file — any of them turns -j into a source of results that change between runs (§8).

10. Check Your Understanding

Question 1. A 200-run regression is launched with make regress. One test fails after four minutes. In the morning, how many of the 200 results do you have, and why?

Question 2. make -k regress attempted every test and still exited 2. Is that a bug?

Question 3. A pattern rule is written $(BUILD)/%/result.txt: and the recipe uses $* as the test name. The targets are build/axi_smoke/1/result.txt and friends. What does the simulator actually receive as +UVM_TESTNAME, and what is the fix?

Question 4. Serial and -j8 runs of the same regression produce byte-identical reports. Name the three properties of the design that make that true rather than lucky.

11. Exercise

Build the regression from §7 and confirm each behaviour. Every transcript is real output.

Step 1. Ask how big the job is before starting it:

make list
make list SEEDS="1 2 3 4 5 6 7 8"

tests: axi_smoke axi_burst axi_random axi_decode_err axi_flaky
seeds: 1
runs : 5
tests: axi_smoke axi_burst axi_random axi_decode_err axi_flaky
seeds: 1 2 3 4 5 6 7 8
runs : 40

Step 2. Check the stem decomposition without running anything:

make -n build/axi_burst/42/result.txt

mkdir -p build/axi_burst/42
t=axi_burst; s=42; \
	 vvp build/simv +UVM_TESTNAME=$t +ntb_random_seed=$s > build/axi_burst/42/run.log 2>&1; \
	 if grep -q "^UVM_ERROR :  0" build/axi_burst/42/run.log && ! grep -q "UVM_FATAL @" build/axi_burst/42/run.log; \
	 then echo PASS > build/axi_burst/42/result.txt; echo "  [PASS] $t seed=$s"; \
	 else echo FAIL > build/axi_burst/42/result.txt; echo "  [FAIL] $t seed=$s  (build/axi_burst/42/run.log)"; fi

Step 3. Run the regression at a seed where one test fails. Confirm all five are attempted with no -k:

make regress SEEDS=5

  [compile] -> build/simv
  [PASS] axi_smoke seed=5
  [PASS] axi_burst seed=5
  [PASS] axi_random seed=5
  [PASS] axi_decode_err seed=5
  [FAIL] axi_flaky seed=5  (build/axi_flaky/5/run.log)

Step 4. Run the eight-seed sweep in parallel and count the verdicts:

make clean
make -j8 regress SEEDS="1 2 3 4 5 6 7 8"
grep -l FAIL build/*/*/result.txt | wc -l

4

Step 5. Confirm parallel and serial agree. Run both and compare the verdict files:

make clean && make     regress SEEDS="1 2 3 4 5 6 7 8" > /dev/null && cat build/*/*/result.txt | md5 -q
make clean && make -j8 regress SEEDS="1 2 3 4 5 6 7 8" > /dev/null && cat build/*/*/result.txt | md5 -q

6528ee11872b59c9ba537f10f7f30027
6528ee11872b59c9ba537f10f7f30027

Then reproduce §5's stop-at-first-failure measurement yourself, and pay attention to the ordering trap. Copy tests/testlist.f to tests/order.f, move axi_flaky to the second line, and run make regress SEEDS=5 with the stamp-based rule from §4 (the one that exits 1 on failure). Count the run.log files afterwards: you should find two, not five. Then move axi_flaky to the last line and repeat — now all five run, and the transcript looks like nothing is wrong. That is the experiment that fools you if you only do it once.

12. Remember

• A regression is the cross product of tests and seeds, built with nested $(foreach) into a list of file paths, and counted with $(words) before you launch it.
• Guard the test list. A missing file leaves TESTS empty and Make does not stop; a regression that ran zero tests must never look like one that passed.
• One pattern rule covers every run. The stem is <test>/<seed>, so the test name is $(patsubst %/,%,$(dir $*)) and the seed is $(notdir $*).
• Verify the stem decomposition with make -n before trusting it — measured as t=axi_burst; s=42.
• Make's default is to stop at the first failure, and for a regression that is wrong. Measured: two of five tests attempted, three never run.
• Measure that honestly. With the failing test last in the list, all five run and the transcript appears to show no problem.
• -k fixes it but depends on being typed, and putting it in MAKEFLAGS would wrongly apply it to the compile.
• Better: let a run always succeed and write its verdict to a file. The sweep then always completes, needs no flag, and the pass/fail decision moves up to the report.
• Keep both rules. A failing exit status is right for one interactive run and wrong for one run inside a sweep.
• Per-run directories make -j safe, because nothing is shared for two runs to race over.
• Measured: about 5× faster on eight cores (1439–1642 ms serial against 281–342 ms at -j8) with a byte-identical report.
• Always give -j a number. Bare -j is unlimited and will exhaust memory with real simulators.

Forty runs, finished, each with a verdict in a file. What is missing is the one thing the person who launched it actually wants: a single answer. Reading forty result.txt files by hand is not a report, and "did the regression pass?" needs to be a question with an exit status. Chapter 6.4 collects the verdicts, distinguishes the four states a run can really be in — not two — and makes the regression target itself fail when anything failed.