Source Filelists, Includes and Defines

Part II · Module 5 · Chapter 5.2 · Source Filelists, Includes and Defines

1. Why a Filelist Exists at All

Chapter 5.1 named the sources on the compile line and that was fine for two files. A real design block has several hundred, and three things about them do not fit on a command line:

• Order can matter. Packages and interfaces must be read before the modules that use them.
• Include paths and compile-time options travel with the list. Where to find headers, which features to enable.
• Several tools need the same list. The simulator, the linter, the synthesis script — and if each keeps its own copy they drift.

So projects keep a filelist: one file naming the sources and the options, which every tool is pointed at. This is the one from Chapter 5.1, now with its full syntax:

// include search path
+incdir+./rtl/include

// design
rtl/counter.v

// testbench
tb/tb_counter.v

Three kinds of line, and all three were verified against Icarus Verilog:

• A source path — compiled, in the order given.
• +incdir+<dir> — adds a directory to the include search path.
• // comments and blank lines — ignored, so the list can be organised.

iverilog -g2012 -f filelists/all.f -o build/simv
vvp build/simv

TB: test=smoke seed=1
TB: count=10
TB: PASSED

The option forms deserve a careful note, because Icarus splits them in a way that catches people out. Each form works in exactly one of the two places:

Both refusals were measured. The dash-forms inside a command file stop the parse:

Error: unable to parse line 1 in filelists/withinc.f.
iverilog: parsing failed in base command file filelists/withinc.f.

And the plus-forms on the command line fail in a more confusing way:

+incdir+rtl/include: No such file or directory
No top level modules, and no -s option.

That second message is the giveaway. On the command line Icarus does not handle + options at all, so it read the whole thing as a filename, failed to open it, and then found no modules to compile. Nothing says "wrong place for this option" — you get a missing-file error for a file you never mentioned.

So the plus-forms are the ones a filelist must use, which is also what VCS and Xcelium accept, making a plus-form filelist the portable choice. Keep the split in mind, though: it decides where the options in §8 have to live.

2. Compile-Time Defines

A filelist can also carry +define+, which sets a Verilog macro for the whole compile. That is how a flow enables assertions, selects a configuration, or turns on debug code.

Here is a testbench fragment that reacts to one:

module tb_defs;
  initial begin
`ifdef EXTRA_DEBUG
    $display("TB: EXTRA_DEBUG is set");
`else
    $display("TB: EXTRA_DEBUG is NOT set");
`endif
    $finish;
  end
endmodule

With the define in the filelist and then without it:

+define+EXTRA_DEBUG
tb/tb_defs.v

TB: EXTRA_DEBUG is set
TB: EXTRA_DEBUG is NOT set

The define genuinely reached the source. In a Makefile this is usually a variable so it can be composed:

INCDIRS := +incdir+rtl/include
DEFINES := +define+SIM +define+EXTRA_DEBUG

3. The Problem, Reproduced With a Real Compiler

Now the trap. The obvious Makefile names the filelist as the prerequisite — the filelist is the input, after all:

FILELIST := filelists/all.f

build/simv: $(FILELIST)
	@mkdir -p $(@D)
	@iverilog -g2012 -f $(FILELIST) -o $@
	@echo "  [compile] -> $@"

Build it, then edit rtl/counter.v — a file the filelist names — and build again:

make
# add a line to rtl/counter.v, then:
make

  [compile] -> build/simv
make: `build/simv' is up to date.

The design changed and Make did nothing. The next simulation runs the old image.

The reason is exactly what Chapter 1.2 said: filelists/all.f contains names, and editing a file it names does not modify the list. Make compared the one prerequisite it was given, found it unchanged, and stopped. It cannot look inside the filelist — it does not know the file contains filenames rather than prose.

This is the single most consequential Makefile bug in verification, because the symptom is a passing test on a design that no longer exists.

4. Stage One: Expand the Filelist Into Prerequisites

If Make cannot read the filelist, read it for Make. $(shell ...) from Chapter 3.4 runs a command while the Makefile is being read, and its output becomes a variable:

FILELIST := filelists/all.f
SOURCES  := $(shell grep -vE '^[[:space:]]*(//|\+|$$)' $(FILELIST))

build/simv: $(FILELIST) $(SOURCES)
	@mkdir -p $(@D)
	@iverilog -g2012 -f $(FILELIST) -o $@
	@echo "  [compile] $(words $(SOURCES)) sources -> $@"

print-%:
	@echo "$* = [$($*)]"

The grep -v throws away the three line kinds that are not sources: comments, plus-options, and blank lines. Check what it produced, using the print-% rule from Chapter 4.4:

make print-SOURCES

SOURCES = [rtl/counter.v tb/tb_counter.v]

The filelist is now the single source of truth for both the compiler and Make. Watch the trap close:

make
make
# edit rtl/counter.v, then:
make

  [compile] 2 sources -> build/simv
make: `build/simv' is up to date.
  [compile] 2 sources -> build/simv

Build, skip, rebuild-on-edit. Adding a source to the filelist now updates both lists at once, because there is only one list.

Two details that matter. $(shell ...) is assigned with := — with = it would re-run the grep on every use, which Chapter 3.4 measured. And $$ in the pattern is a literal $ for the shell, which is Chapter 1.4's escaping rule.

5. Stage Two: The Header Make Still Cannot See

Stage one fixed the listed files. Now edit the header that rtl/counter.v includes — a file no filelist mentions:

# add a line to rtl/include/defines.vh, then:
make

make: `build/simv' is up to date.

Still stale. The filelist names two sources and an include directory; it does not name defines.vh, because the design reaches that file through `include "defines.vh" at compile time.

This is the boundary Chapter 2.3 called out: the compiler follows the include and knows the dependency; Make never opens a source file and cannot. A filelist cannot close the gap because the information is not in the filelist either.

The quick fix is to glob the headers and add them:

INCDIR  := rtl/include
HEADERS := $(wildcard $(INCDIR)/*.vh)

build/simv: $(FILELIST) $(SOURCES) $(HEADERS)
	@mkdir -p $(@D)
	@iverilog -g2012 -f $(FILELIST) -o $@
	@echo "  [compile] $(words $(SOURCES)) sources + $(words $(HEADERS)) headers -> $@"

make
# edit rtl/include/defines.vh, then:
make
make --debug=b build/simv

  [compile] 2 sources + 1 headers -> build/simv
  [compile] 2 sources + 1 headers -> build/simv
 Prerequisite `rtl/include/defines.vh' is newer than target `build/simv'.
Must remake target `build/simv'.

It works, and --debug=b confirms why. But be honest about what it is: a guess. It assumes every header lives in rtl/include and ends in .vh. A header elsewhere, with a different suffix, or included by another header, is still invisible. And it over-declares in the other direction — a header nothing includes still triggers rebuilds, which is Chapter 2.3's over-declaration.

For many projects that trade is acceptable, and it is far better than nothing. But there is a correct answer.

6. Stage Three: Ask the Compiler What It Read

The compiler opened every file. It knows the real list. Most simulators can be asked to write it down, and Icarus does it with -M:

iverilog -g2012 -M build/files.raw -f filelists/all.f -o build/simv
cat build/files.raw

rtl/counter.v
./rtl/include/defines.vh
tb/tb_counter.v

Three files — and the middle one is the header no filelist named and no wildcard guessed. The compiler reports it because it genuinely opened it.

That list is the exact set of prerequisites the image has. All that remains is to give it to Make, and Chapter 4.2 already supplied the mechanism: write the list as a Makefile fragment and pull it in with -include.

7. Stage Four: The Complete Solution

Here is the whole thing, and every line was executed:

.PHONY: all clean
.DEFAULT_GOAL := all
.DELETE_ON_ERROR:

BUILD    := build
FILELIST := filelists/all.f
SIMV     := $(BUILD)/simv
DEPFILE  := $(BUILD)/simv.d

-include $(DEPFILE)

all: $(SIMV)

$(SIMV): $(FILELIST)
	@mkdir -p $(@D)
	@iverilog -g2012 -M $(BUILD)/files.raw -f $(FILELIST) -o $@
	@printf '%s: ' "$@" > $(DEPFILE)
	@tr '\n' ' ' < $(BUILD)/files.raw >> $(DEPFILE)
	@echo "" >> $(DEPFILE)
	@echo "  [compile] -> $@"

clean:
	@rm -rf $(BUILD)

The three printf/tr/echo lines turn the compiler's bare file list into one Makefile rule. After a build, the generated fragment reads:

build/simv: rtl/counter.v ./rtl/include/defines.vh tb/tb_counter.v

Now watch it work from a clean tree:

make
cat build/simv.d
make
# edit rtl/include/defines.vh — the header no filelist names — then:
make

  [compile] -> build/simv
build/simv: rtl/counter.v ./rtl/include/defines.vh tb/tb_counter.v 
make: `build/simv' is up to date.
  [compile] -> build/simv

The header edit rebuilt the image. Confirm the reason rather than assuming it:

touch rtl/include/defines.vh
make --debug=b

 Prerequisite `rtl/include/defines.vh' is newer than target `build/simv'.
Must remake target `build/simv'.

Make named the include file as the cause. Nothing in the Makefile mentions defines.vh, no wildcard guessed at it, and no filelist lists it — the compiler reported it and Make believed it.

And the image still runs:

vvp build/simv

TB: test=smoke seed=1
TB: count=10
TB: PASSED

Why this works on the very first build

The ordering looks impossible: the depfile is produced by the compile, so on a clean tree it does not exist when -include runs. Chapter 4.2 measured what happens — Make builds a missing included file if it has a rule for it, then restarts itself and re-reads everything.

Here the dash in -include makes the missing file harmless on the first pass, the compile runs for its own sake, and the depfile is in place for every later invocation. The first build is slightly under-specified and every build after it is exact, which is the normal and accepted shape of this pattern.

The same mechanism is how C projects have handled header dependencies for decades (gcc -MMD). Commercial simulators offer equivalents under different names, and the Makefile shape is identical — only the flag that produces the list changes.

8. Putting the Options in a File

With the sources and headers tracked, the last loose end is the options. Start by composing them as variables so each one appears exactly once:

INCDIRS := +incdir+rtl/include
DEFINES := +define+SIM
DEFINES += +define+EXTRA_DEBUG

+= accumulating defines is Chapter 3.1; ?= would let a user override a whole group from the command line, as Chapter 3.2 showed.

Now, where do those options go? §1's split already answers it: the plus-forms cannot sit on the iverilog command line, so they have to reach the tool through a -f file. Icarus accepts more than one -f in a single command, so a flow can generate a small options file and pass it alongside the hand-maintained source list.

That turns out to fix a second problem at the same time. A variable is not a file, and Make only ever compares files. If the options lived only in DEFINES, editing them would change nothing Make can see: make would report everything up to date while the image on disk still carried the old macros. Writing them to a file puts the options into the dependency graph, where they belong.

OPTFILE := $(BUILD)/opts.f

INCDIRS := +incdir+rtl/include
DEFINES := +define+SIM
DEFINES += +define+EXTRA_DEBUG

# Runs every time, but rewrites the file only when the contents differ,
# so the timestamp moves on a real option change and never otherwise.
$(OPTFILE): FORCE
	@mkdir -p $(@D)
	@printf '%s\n' $(INCDIRS) $(DEFINES) > $@.tmp
	@if cmp -s $@.tmp $@; then rm -f $@.tmp; else mv $@.tmp $@; echo "  [opts] changed"; fi

$(SIMV): $(FILELIST) $(OPTFILE)
	@mkdir -p $(@D)
	@iverilog -g2012 -f $(OPTFILE) -f $(FILELIST) -M $(BUILD)/files.raw -o $@
	@printf '%s: ' "$@" > $(DEPFILE)
	@tr '\n' ' ' < $(BUILD)/files.raw >> $(DEPFILE)
	@echo "" >> $(DEPFILE)
	@echo "  [compile] -> $@"

.PHONY: FORCE
FORCE:

The FORCE prerequisite makes the options rule run on every invocation — that is Chapter 2.2's phony-as-a-prerequisite idea doing real work. But the recipe writes to a temporary file and only replaces opts.f when cmp says the contents actually differ. The rule runs constantly; the timestamp moves rarely. Make's file comparison then does exactly the right thing.

Five invocations, measured in order:

$ make                                 # 1. clean tree
  [opts] changed
  [compile] -> build/simv

$ make                                 # 2. nothing touched
                                       #    (no output at all)

$ touch rtl/include/defines.vh         # 3. the included header
$ make
  [compile] -> build/simv

$ # edit the Makefile: EXTRA_DEBUG -> QUIET_MODE
$ make                                 # 4. an option changed
  [opts] changed
  [compile] -> build/simv

$ make                                 # 5. nothing touched
                                       #    (no output at all)

Run 3 is the depfile from §7 doing its job. Run 4 is the new part: an option edit, and nothing else, produced a rebuild. And the rebuilt image genuinely differs — the testbench's `ifdef takes the other branch now:

TB: EXTRA_DEBUG is NOT set

Run 5 confirms the flow still settles. Both generated files are worth looking at:

+incdir+rtl/include
+define+SIM
+define+QUIET_MODE

build/simv: rtl/counter.v rtl/include/defines.vh tb/tb_counter.v

Two generated files, and between them every input the compile actually consumes — sources, headers and options — is now a prerequisite Make can see. That is the whole goal of the chapter.

Giving each configuration its own build directory is the other common answer to the same problem, and Chapter 5.3 adopts a directory-per-run for a closely related reason.

9. Common Mistakes

Naming only the filelist as a prerequisite. The oldest trap in the track: editing a listed source changes nothing (§3).

Keeping two lists. A filelist for the tool and a hand-written SOURCES for Make will drift. Derive one from the other (§4).

Assuming a wildcard covers the headers. It covers the headers you guessed the location and suffix of (§5).

Assigning $(shell ...) with =. It re-runs on every expansion (Chapter 3.4).

Forgetting the dash on -include. A plain include of a depfile that does not exist yet is fatal on a clean checkout (Chapter 4.2).

Leaving the options only in variables. A variable is not a file, so an option edit rebuilds nothing — write them to a file and depend on it (§8).

Putting a test name in a define. Every test change becomes a recompile — see §2's callout.

Mixing up the two option forms. -I and -DNAME work only on the command line; +incdir+ and +define+ work only inside a -f file, and each is rejected in the other place (§1).

10. Check Your Understanding

Question 1. A compile rule reads build/simv: filelists/all.f. An engineer edits a source named inside that filelist and re-runs make. State what happens and why, in one sentence each.

Question 2. SOURCES := $(shell grep -vE '^[[:space:]]*(//|\+|$$)' $(FILELIST)) — explain what each of the three alternatives in that pattern throws away, and why $$ is doubled.

Question 3. After stage one is in place, editing rtl/include/defines.vh still does not rebuild. Why can no filelist fix this, and what does the compiler know that Make does not?

Question 4. On a completely clean tree, -include build/simv.d refers to a file that does not exist. Explain why the first make nevertheless produces a correct build, naming the mechanism.

11. Exercise

Start from the project in Chapter 5.1 §2 and this deliberately naive Makefile.

BUILD    := build
FILELIST := filelists/all.f

all: $(BUILD)/simv

$(BUILD)/simv: $(FILELIST)
	@mkdir -p $(@D)
	@iverilog -g2012 -f $(FILELIST) -o $@
	@echo "  [compile] -> $@"

Carry out these steps and record the output of each:

1. make, then make again.
2. Add a comment line to rtl/counter.v, then make.
3. Add a line to rtl/include/defines.vh, then make.
4. Rewrite the Makefile so steps 2 and 3 both rebuild, using the compiler's own dependency output. Prove each fix with make --debug=b.

12. Remember

• A filelist is the single list every tool should read — sources in order, plus +incdir+ and +define+ options, with // comments allowed.
• Use the plus-form options inside a filelist, and only there. Icarus rejects -I/-DNAME inside a -f file and rejects +incdir+/+define+ on the command line — each form works in exactly one place. Commercial tools accept the plus-forms, so a plus-form filelist is the portable one.
• Naming only the filelist as a prerequisite is the track's oldest trap. Editing a listed source changes nothing and the next run simulates the old design.
• Expand the filelist with $(shell grep ...) assigned with :=, so one list serves both the compiler and Make.
• That still misses `included headers, because no filelist names them and Make never opens a source file.
• Globbing the include directory is a working guess, not a guarantee — it depends on every header matching the pattern.
• The complete fix is to ask the compiler. iverilog -M writes every file it actually read, including the headers; convert that to a depfile and -include it.
• It works on a clean tree because Make rebuilds a missing include and restarts itself — the first build is slightly under-specified, every later one is exact.
• A define is compile-time and a plusarg is run-time. Putting a test name in a define costs a recompile per test.
• Changing an options variable does not rebuild anything. Variables are not files; put the options in a file or use a per-configuration directory.

The filelist is now honest, and the longest-running problem in this track is closed. What is still one step is the compile itself: Icarus produces a runnable image in a single command, while commercial tools separate analysis from elaboration, and a flow has to let a run happen without rebuilding anything. Chapter 5.3 splits the stages and gets the run-time knobs right.