Complete APB/AXI Verification Makefile

This chapter introduces nothing. That is deliberate: everything below has a chapter behind it, and §8 is the table that names which. What is new is seeing it whole, and seeing every target actually run.

The flow is a verification environment for a small AXI-Lite register slave, with a test list that spans APB-style and AXI-style tests. It runs on Icarus Verilog, which is free, so every transcript here is real — and it is structured so that the three commercial simulators of Chapter 7.2 drop in as one extra file each.

1. The Shape of It

Makefile                  six include lines, nothing else
mk/config.mk              variables, guards, the test list
mk/icarus.mk              the only tool-specific file
mk/sim.mk                 compile, run, waves, log, rerun
mk/regress.mk             many runs, and the report
mk/clean.mk               clean, distclean
mk/help.mk                help, sims, tests
filelists/all.f           the one source list every tool reads
tests/testlist.f          the tests, in groups
scripts/verdict.sh        reads one log, prints PASS or FAIL
scripts/report.py         walks the expected runs, sets the exit status
rtl/ tb/                  the design and the testbench

Eleven files, of which six are Makefile fragments owned by different concerns — the division Chapter 8.1 §7 argued for.

It is worth asking why this is not one file, because it could be: the whole flow is about 120 lines of Make, which fits comfortably in one Makefile and would be slightly easier to read top to bottom. The answer is not length but change. Over a year, mk/config.mk is edited when a knob is added, mk/icarus.mk when the site changes tool, mk/regress.mk when the report gains a column, and tests/testlist.f weekly. Those are four different people and four different review conversations, and in one file they are four sets of conflicting edits to the same lines. Splitting by who changes it and why costs one include line each and makes every diff local to its subject.

The top-level Makefile contains no rules at all:

# VLSI Mentor - complete verification flow (Makefile track capstone).
include mk/config.mk
include mk/$(SIM).mk
include mk/sim.mk
include mk/regress.mk
include mk/clean.mk
include mk/help.mk

config.mk is first because everything else uses its variables and relies on its guards having run.

2. config.mk — What the Flow Is Told, and What It Refuses

BUILD     ?= build
FILELIST  ?= filelists/all.f
TESTLIST  ?= tests/testlist.f
TOP       ?= tb_top
SIM       ?= icarus
SIM_LIST  := icarus vcs xrun questa
TEST      ?= axi_smoke
SEED      ?= 1
SEEDS     ?= 1

$(if $(strip $(SIM)),,$(error SIM is empty -- choose one of: $(SIM_LIST)))
$(if $(filter $(SIM),$(SIM_LIST)),,$(error unknown SIM "$(SIM)" -- choose one of: $(SIM_LIST)))
$(if $(wildcard mk/$(SIM).mk),,$(error SIM "$(SIM)" is listed but mk/$(SIM).mk is missing))
$(if $(wildcard $(FILELIST)),,$(error filelist "$(FILELIST)" does not exist))
$(if $(wildcard $(TESTLIST)),,$(error test list "$(TESTLIST)" does not exist))

TESTS := $(shell grep -vE '^[[:space:]]*(\#|$$)' $(TESTLIST))
$(if $(strip $(TESTS)),,$(error test list "$(TESTLIST)" contains no tests))

RUNDIR  = $(BUILD)/$(SIM)/$(TEST)/$(SEED)
RESULTS = $(foreach t,$(TESTS),$(foreach s,$(SEEDS),$(BUILD)/$(SIM)/$(t)/$(s)/result.txt))
REPORT  = $(BUILD)/$(SIM)/report.txt

Half of this file is guards, and that proportion is right. Every one of the six refuses a specific mistake at parse time, before a directory is created or a license is checked out:

$ make report SIM=vsim
mk/config.mk:14: *** unknown SIM "vsim" -- choose one of: icarus vcs xrun questa.  Stop.

$ make report TESTLIST=tests/nope.f
mk/config.mk:17: *** test list "tests/nope.f" does not exist.  Stop.

Three details that are each a chapter:

• Every knob is ?=, so the command line wins (Chapter 3.2).
• TESTS uses := with \# — := so the grep runs once, and the escape because a bare # would be a Makefile comment that swallows the closing bracket (Chapter 6.1 §6).
• RUNDIR and RESULTS use =, not :=, because they must re-expand after SIM is known and whenever TEST, SEED or SEEDS is overridden (Chapter 7.2 §4).

3. icarus.mk — The Only Tool-Specific File

IVERILOG  ?= iverilog
VVP       ?= vvp
SIM_LABEL := Icarus Verilog
SIMV       = $(BUILD)/$(SIM)/simv
DEPFILE    = $(BUILD)/$(SIM)/simv.d

define SIM_COMPILE
$(IVERILOG) -g2012 -f $(FILELIST) -M $(BUILD)/$(SIM)/files.raw -o $(SIMV)
printf '%s: ' "$(SIMV)" > $(DEPFILE)
tr '\n' ' ' < $(BUILD)/$(SIM)/files.raw >> $(DEPFILE)
echo "" >> $(DEPFILE)
endef

# $1 = test, $2 = seed, $3 = run dir, $4 = extra plusargs
define SIM_RUN
$(VVP) $(SIMV) +UVM_TESTNAME=$(1) +ntb_random_seed=$(2) $(4)
endef

Everything that differs between simulators is in this file and nowhere else. Swapping to VCS means writing mk/vcs.mk with the same two define blocks and running make SIM=vcs — the rest of the flow does not change, which is Chapter 7.2 §4's whole claim.

The -M flag and the three lines after it build a dependency file listing every source the compiler actually read — Chapter 5.2 §7's solution to the oldest trap in the track. Measured:

build/icarus/simv: rtl/axi_lite_slave.v tb/tb_top.v

SIM_RUN takes arguments rather than reading globals, so one definition serves the interactive run, the waveform run and the regression — each passing its own test, seed, directory and extra plusargs.

4. sim.mk — Compile, Run, Waves, Log

.PHONY: compile run waves log rerun
compile: $(SIMV)

$(SIMV): $(FILELIST)
	@mkdir -p $(@D)
	@echo "  [compile] $(SIM_LABEL) -> $@"
	@$(SIM_COMPILE)

-include $(DEPFILE)

# One interactive run: fails loudly, so `make run` answers a yes/no question.
run: $(RUNDIR)/passed.stamp
$(BUILD)/$(SIM)/%/passed.stamp: $(SIMV)
	@mkdir -p $(@D)
	@t=$(patsubst %/,%,$(dir $*)); s=$(notdir $*); \
	 echo "  [run] $$t seed=$$s -> $(@D)/run.log"; \
	 $(call SIM_RUN,$$t,$$s,$(@D)) > $(@D)/run.log 2>&1; \
	 if sh scripts/verdict.sh $(@D)/run.log > $(@D)/result.txt; \
	 then touch $@; echo "  [PASS] $$t seed=$$s"; \
	 else echo "  [FAIL] $$t seed=$$s  (see $(@D)/run.log)"; exit 1; fi

waves: $(RUNDIR)/dump.vcd
$(BUILD)/$(SIM)/%/dump.vcd: $(SIMV)
	@mkdir -p $(@D)
	@t=$(patsubst %/,%,$(dir $*)); s=$(notdir $*); \
	 echo "  [waves] $$t seed=$$s -> $@"; \
	 $(call SIM_RUN,$$t,$$s,$(@D),+VCD=$@) > $(@D)/run.log 2>&1
	@echo "  [waves] $$(wc -c < $@) bytes, $$(grep -c '^\$$var' $@) signals"

log:
	@test -f $(RUNDIR)/run.log || { echo "no log for $(TEST) seed $(SEED)"; exit 1; }
	@cat $(RUNDIR)/run.log

rerun:
	@rm -rf $(RUNDIR)
	@$(MAKE) --no-print-directory -f $(firstword $(MAKEFILE_LIST)) run TEST=$(TEST) SEED=$(SEED)

The densest file in the flow, and every line has a reason:

-include $(DEPFILE) with the dash, so a clean tree is not an error, and Make's build-and-restart handles the first build (Chapter 4.2 §9).

The target is passed.stamp, not the log. A failed run keeps its log for reading and writes no stamp, so it can never be mistaken for up to date (Chapter 5.4 §6).

touch $@ is last, after the verdict, so a failure cannot record itself as a pass.

One pattern rule for every test and seed, with the stem <test>/<seed> decomposed by $(patsubst %/,%,$(dir $*)) and $(notdir $*) (Chapter 6.3 §4).

The whole recipe is one continued line, because each recipe line gets its own shell and t= would not survive to the next (Chapter 4.3 §3).

Waveforms are opt-in, enabled by a run-time plusarg rather than a second image, because dumping was measured at 2.6–3.5× slower and 101 MB from nine signals (Chapter 5.4 §2).

log has no prerequisites, so asking to read a log can never start a simulation.

rerun uses $(MAKE) and -f $(firstword $(MAKEFILE_LIST)), because a bare make loses the jobserver (Chapter 8.2 §6) and -f is not inherited (Chapter 6.4 §7).

5. regress.mk — Many Runs, One Answer

.PHONY: regress report report-only
# A regression run NEVER fails: it records a verdict and moves on.
$(BUILD)/$(SIM)/%/result.txt: $(SIMV)
	@mkdir -p $(@D)
	@t=$(patsubst %/,%,$(dir $*)); s=$(notdir $*); \
	 $(call SIM_RUN,$$t,$$s,$(@D)) > $(@D)/run.log 2>&1; \
	 if sh scripts/verdict.sh $(@D)/run.log > $@; \
	 then echo "  [PASS] $$t seed=$$s"; else echo "  [FAIL] $$t seed=$$s"; fi

regress: $(RESULTS)
report: regress
	@$(MAKE) --no-print-directory -f $(firstword $(MAKEFILE_LIST)) report-only

# The ONE place that decides the overall answer.
report-only:
	@mkdir -p $(dir $(REPORT))
	@python3 scripts/report.py $(BUILD)/$(SIM) "$(TESTS)" "$(SEEDS)" > $(REPORT); st=$$?; \
	 sed 's/^/  /' $(REPORT); exit $$st

Two run rules, deliberately. The passed.stamp rule in §4 fails — right for make run, which is a question with a yes-or-no answer. The result.txt rule here never fails — right for a regression, which must finish and then report (Chapter 6.3 §7). They write into the same run directory and do not interfere.

report-only is where the pipe is carefully avoided: the exit status is captured into st before anything is printed, because a | sed would have replaced it with sed's (Chapter 7.4 §5 measured exactly that bug in my own gate).

The report script walks the expected cross product rather than listing files that exist, so a run that never happened gets a row instead of vanishing (Chapter 6.4 §2), and classifies each into four states:

if (d / "result.txt").exists():
    st = (d / "result.txt").read_text().strip() or "INCOMPLETE"
elif (d / "run.log").exists():
    st = "INCOMPLETE"
else:
    st = "NOTRUN"
...
bad = tot - n["PASS"]
print("RESULT: " + ("PASSED" if bad == 0 else "FAILED"))
sys.exit(1 if bad else 0)

Anything that is not a pass fails the regression, including a run whose outcome is unknown.

6. clean.mk and help.mk

.PHONY: clean distclean
clean:
	@test -n "$(BUILD)" || { echo "  [clean] BUILD is empty - refusing"; exit 1; }
	@test "$(BUILD)" != "/" || { echo "  [clean] BUILD is / - refusing"; exit 1; }
	@rm -rf $(BUILD)
	@echo "  [clean] removed $(BUILD)/"
distclean: clean
	@rm -f $(BUILD).tar.gz
	@echo "  [distclean] done"

The two guards are there because rm -rf $(BUILD) is one unset variable away from something much worse, and rm -rf $(BUILD)/ with BUILD empty expands to rm -rf / (Chapter 5.4 §8). Measured:

  [clean] BUILD is empty - refusing
make: *** [clean] Error 1

.PHONY: help sims tests
.DEFAULT_GOAL := help
help:
	@echo "targets:"
	@echo "  make compile                        build the image"
	@echo "  make run    TEST=<t> [SEED=<n>]     one test, fails on failure"
	@echo "  make waves  TEST=<t> [SEED=<n>]     the same run, with a VCD"
	@echo "  make log    TEST=<t> [SEED=<n>]     print that run's log"
	@echo "  make rerun  TEST=<t> [SEED=<n>]     force a completed run again"
	@echo "  make report [SEEDS='1 2 3']         whole regression + exit status"
	@echo "  make clean | distclean              remove generated output"
	@echo ""
	@echo "  SIM=$(SIM) ($(SIM_LABEL))   sims: $(SIM_LIST)"
	@echo "  tests ($(words $(TESTS))): $(TESTS)"

.DEFAULT_GOAL := help makes a bare make safe and useful rather than launching something (Chapter 2.4):

targets:
  make compile                        build the image
  make run    TEST=<t> [SEED=<n>]     one test, fails on failure
  make waves  TEST=<t> [SEED=<n>]     the same run, with a VCD
  make log    TEST=<t> [SEED=<n>]     print that run's log
  make rerun  TEST=<t> [SEED=<n>]     force a completed run again
  make report [SEEDS='1 2 3']         whole regression + exit status
  make clean | distclean              remove generated output

  SIM=icarus (Icarus Verilog)   sims: icarus vcs xrun questa
  tests (7): axi_smoke apb_smoke axi_burst axi_decode_err apb_decode_err axi_random axi_flaky

7. Running It

Every target, in order, on a clean tree.

$ make compile
  [compile] Icarus Verilog -> build/icarus/simv

$ make run TEST=axi_smoke SEED=1
  [run] axi_smoke seed=1 -> build/icarus/axi_smoke/1/run.log
  [PASS] axi_smoke seed=1

$ make run TEST=axi_smoke SEED=1
make: Nothing to be done for `run'.

$ make run TEST=apb_smoke SEED=1
  [run] apb_smoke seed=1 -> build/icarus/apb_smoke/1/run.log
  [PASS] apb_smoke seed=1

$ make run TEST=axi_flaky SEED=5
  [run] axi_flaky seed=5 -> build/icarus/axi_flaky/5/run.log
  [FAIL] axi_flaky seed=5  (see build/icarus/axi_flaky/5/run.log)
make: *** [build/icarus/axi_flaky/5/passed.stamp] Error 1
  make rc=2

Three properties in that transcript. The second axi_smoke run did nothing, because a passed test against an unchanged image has nothing to redo. The apb_smoke run has no [compile] line — a different test on the same image. And the failure exited 2 while keeping its log.

  UVM_INFO @ 0: reporter [RNTST] Running test axi_flaky...
  UVM_INFO @ 0: reporter [SEED] seed=5
  UVM_ERROR @ 55000: uvm_test_top [TEST] addr 4 undecoded - data lost
  
  --- UVM Report Summary ---
  
  ** Report counts by severity
  UVM_INFO :  2
  UVM_WARNING :  0
  UVM_ERROR :  1
  UVM_FATAL :  0
  tb/tb_top.v:142: $finish called at 75000 (1ps)

$ make waves TEST=axi_smoke SEED=1
  [waves] axi_smoke seed=1 -> build/icarus/axi_smoke/1/dump.vcd
  [waves]     2255 bytes, 34 signals

$ make rerun TEST=axi_smoke SEED=1
  [run] axi_smoke seed=1 -> build/icarus/axi_smoke/1/run.log
  [PASS] axi_smoke seed=1

Then the regression. Seven tests at one seed:

  [compile] Icarus Verilog -> build/icarus/simv
  [PASS] axi_smoke seed=1
  [PASS] apb_smoke seed=1
  [PASS] axi_burst seed=1
  [PASS] axi_decode_err seed=1
  [PASS] apb_decode_err seed=1
  [PASS] axi_random seed=1
  [PASS] axi_flaky seed=1
  ==================================
  axi_smoke           1  PASS
  apb_smoke           1  PASS
  axi_burst           1  PASS
  axi_decode_err      1  PASS
  apb_decode_err      1  PASS
  axi_random          1  PASS
  axi_flaky           1  PASS
  ==================================
  7 passed  0 failed  0 incomplete  0 not run   (of 7)
  RESULT: PASSED
  make rc=0

And the same seven tests across four seeds, in parallel — 28 runs:

  [PASS] axi_flaky seed=1
  [FAIL] axi_flaky seed=5
  [PASS] axi_flaky seed=2
  [PASS] axi_flaky seed=3
  ==================================
  ...
  axi_flaky           1  PASS
  axi_flaky           2  PASS
  axi_flaky           3  PASS
  axi_flaky           5  FAIL
  ==================================
  27 passed  1 failed  0 incomplete  0 not run   (of 28)
  RESULT: FAILED
make[1]: *** [report-only] Error 1
make: *** [report] Error 2
  make rc=2

Three separate properties are visible in that one transcript, and each was a chapter.

The sweep completed. axi_flaky failed at seed 5 and the other 27 runs still happened, because a regression run records a verdict instead of failing (Chapter 6.3 §7). Had it used the interactive rule, Make would have stopped and the morning's information would have been one failure and 27 unknowns.

The failure is localised to one test at one seed, which makes it reproducible as a single command — make run TEST=axi_flaky SEED=5 — because the seed is both applied and recorded (Chapter 6.2).

The overall answer is non-zero. 27 of 28 passing is a failing regression, and nothing in the flow is tempted to round that up. The exit status is what a merge gate reads (Chapter 6.4 §8).

And it does not matter how the work was scheduled:

  serial md5 : b9f57c817ad24d467cef73c48dcf3c23
  -j8    md5 : b9f57c817ad24d467cef73c48dcf3c23
  IDENTICAL

Finally, clean leaves exactly the hand-maintained inputs:

  [clean] removed build/
    Makefile
    filelists
    mk
    rtl
    scripts
    tb
    tests

Seven entries, all of them things a person wrote. That is the test of a good clean.

8. Every Line Accounted For

The point of a capstone is that nothing in it needs new explanation. Here is the whole flow mapped back:

Adapting it to a real project

If you take this flow as a starting point, the order to change things in matters, because each step is testable before the next.

First, replace the tool file. Write mk/<yoursim>.mk with the two define blocks and add the name to SIM_LIST. Nothing else moves. Confirm with make compile SIM=<yoursim> and, if you want certainty about what the Makefile composed, the argv wrapper from Chapter 7.1 §7.

Second, point it at your sources. Replace filelists/all.f. If your flow already has a filelist, this step is a path change — and if it has several, that is worth resolving before anything else, because Chapter 5.2 §7's argument about one list with many readers is the difference between synthesising what you verified and synthesising something else.

Third, fix the verdict. scripts/verdict.sh greps for UVM_ERROR : 0, which is a convention of this testbench, not a law. Change it to whatever your environment actually prints on success — and then do the thing Chapter 7.3 §4 insists on: run it against a known failing log and a known passing log and confirm two different exit statuses. A verdict script is the single highest-leverage line in the flow, because every other number is derived from it.

Fourth, put your tests in the list, and add the check target from Chapter 8.4 §5 so the list and the testbench cannot drift apart.

Only then turn on -j, and run make clean && make -j8 report three times. If all three agree, the graph is telling the truth. If they do not, you have found a missing prerequisite or a shared file, and Chapter 8.2 has both.

That order is deliberate: each step leaves the flow runnable, so a problem introduced at step three cannot be confused with one introduced at step one.

9. Common Mistakes

Putting a rule in the top-level Makefile. Then it has no owner. The top level is six include lines (Chapter 8.1 §7).

One run rule instead of two. make run should fail; a run inside a regression should record and continue (§5).

:= on RUNDIR. It must re-expand after SIM is known and when TEST or SEED is overridden (§2).

A | sed between the report and its exit status. It becomes sed's status (§5).

touch $@ before the verdict. A failure then records itself as a pass (§4).

Prerequisites on log or help. Asking what the flow does, or to read a log, must never start a simulation (§4, §6).

An unguarded rm -rf $(BUILD). One unset variable from disaster (§6).

A second image for waveforms when the tool accepts a run-time dump path (§4).

Omitting the depfile. Then editing a source the filelist names rebuilds nothing — the oldest trap in the track (§3).

10. Check Your Understanding

Question 1. make run TEST=apb_smoke printed no [compile] line immediately after make run TEST=axi_smoke. Why is that correct?

Question 2. Why does the flow have both a passed.stamp rule and a result.txt rule when both run a simulation?

Question 3. The serial and -j8 reports are byte-identical. Name the two properties of the design that make that true.

Question 4. make report TESTLIST=tests/nope.f stops at mk/config.mk:17. What would have happened without that guard?

Question 5. Adding synthesis to this flow should require no edit to any existing fragment. Why?

11. Exercise

Build the flow and run every target. All transcripts above are real; these are the ones to reproduce.

Step 1. Confirm the default goal is safe:

make

It must print help and run nothing.

Step 2. Compile once, then run two different tests. Confirm the second has no [compile] line.

Step 3. Run the failing test and check what is on disk:

make run TEST=axi_flaky SEED=5 ; echo "rc=$?"
ls build/icarus/axi_flaky/5/

rc=2
result.txt
run.log

A log and a verdict — and no passed.stamp. The verdict file holds FAIL, so the regression report will classify this run correctly even though the interactive target failed.

Step 4. Run the whole regression across four seeds in parallel, then serially, and compare:

make clean && make -j8 report SEEDS="1 2 3 5" ; echo "rc=$?"
md5 -q build/icarus/report.txt

  27 passed  1 failed  0 incomplete  0 not run   (of 28)
  RESULT: FAILED
rc=2
b9f57c817ad24d467cef73c48dcf3c23

Step 5. Try all four ways of asking for something impossible:

make report SIM=vsim
make report SIM=
make report TESTLIST=tests/nope.f
make clean BUILD=

Each must stop with one clear line naming the problem.

Then extend it, which is the real exercise. Add mk/synth.mk holding Chapter 7.4's synthesis target and slack gate, and add one include line to the top-level Makefile. You should not need to touch config.mk, sim.mk, regress.mk, clean.mk or help.mk — except to add one line to the help text. If you find yourself editing anything else, work out why: either a value belongs in config.mk and is not there yet, or something flow-wide has leaked into a fragment that should not own it.

12. Remember

• A capstone should introduce nothing. Every construct here has an earlier chapter behind it, and §8 names which one.
• The top-level Makefile is six include lines and no rules, so every rule has an owner.
• config.mk comes first, and half of it is guards that refuse a specific mistake at parse time.
• One file per tool, holding two define blocks — swapping simulator is one new file plus SIM=.
• Two run rules, on purpose: one that fails for make run, one that records and continues for the regression.
• The stamp is the target and touch is last, so a failed run keeps its log and cannot look up to date.
• The verdict comes from the log, through a script, because no simulator's exit status reports a failed test.
• One place decides the overall answer, and it captures the status before printing anything.
• The report walks the expected runs, so a run that never happened gets a row instead of vanishing.
• Nothing is shared between runs, which is why 28 runs at -j8 produced a byte-identical report.
• .DEFAULT_GOAL := help, and help and log have no prerequisites.
• clean is guarded and leaves exactly the files a person wrote — measured, seven entries.
• A new flow is a new fragment, not an edit. That is the test of whether the structure is right.

Reading a flow that works is one thing. The last chapter is about the day it does not: a result that is quietly stale, a variable that expanded to nothing, a tool that exited zero after an error. Chapter 8.4 is the diagnostic toolkit and the review habits, and it closes the track by naming the one argument every chapter in it has been making.