Wildcards and Essential Make Functions

Part I · Module 3 · Chapter 3.4 · Wildcards and Essential Make Functions

1. The Last List Written by Hand

Chapter 3.3 removed every filename from inside the recipes. This is what is left:

SOURCES := rtl/defines.vh rtl/alu.v rtl/tb_top.v

build/sim.img: $(SOURCES)
	@mkdir -p $(@D)
	@cat $^ > $@

For three files that is honest and readable. For a real design block it is not: a few hundred files, added and renamed by several people, and a list that somebody has to remember to edit. A file missing from the list is not a loud failure — it is Chapter 2.3's stale output, silently.

A function is how Make computes a value instead of you typing it. The syntax looks like a variable with arguments:

$(function-name argument,argument)

A space separates the name from the first argument; commas separate the arguments after that. That irregularity is unusual and worth noticing once — $(patsubst %.v,%.o,$(SRCS)) has a space after patsubst and commas thereafter.

This chapter covers the handful that do almost all the work in real Makefiles.

2. $(wildcard ...) — Ask the Directory

wildcard takes one or more shell-style patterns and returns the files that exist and match:

V_FILES := $(wildcard rtl/*.v)
ALL_RTL := $(wildcard rtl/*)
BOTH    := $(wildcard rtl/*.v tb/*.v)
MISSING := $(wildcard rtl/*.sv)

show:
	@echo "rtl/*.v        = [$(V_FILES)]"
	@echo "rtl/*          = [$(ALL_RTL)]"
	@echo "two patterns   = [$(BOTH)]"
	@echo "no matches     = [$(MISSING)]"

With rtl/ holding alu.v, regfile.v, defines.vh and README.md, and tb/ holding tb_top.v:

rtl/*.v        = [rtl/alu.v rtl/regfile.v]
rtl/*          = [rtl/README.md rtl/alu.v rtl/defines.vh rtl/regfile.v]
two patterns   = [rtl/alu.v rtl/regfile.v]
no matches     = []

Four things to read from that. rtl/*.v matched only the Verilog files. rtl/* matched everything including the README.md, which is a reminder that a pattern is not a filter for "source code". Two patterns can be given in one call. And no matches gives an empty list, not an error — which §4 shows is the important difference from a bare glob.

3. What a Wildcard Cannot See

This is the trap, and it follows directly from Chapter 3.1: := takes a snapshot when the line is read.

wildcard asks the filesystem at the moment it is expanded. With :=, that moment is while Make is reading the Makefile — before any recipe has run. So it cannot see a file your own build is about to create:

GENERATED := $(wildcard build/*.img)

.PHONY: all
all: build/sim.img
	@echo "  wildcard build/*.img = [$(GENERATED)]"

build/sim.img: rtl/alu.v
	@mkdir -p $(@D)
	@cp $< $@
	@echo "  built $@"

Run it twice, changing nothing:

  built build/sim.img
  wildcard build/*.img = []

  wildcard build/*.img = [build/sim.img]

The same Makefile reports two different answers. On the first run build/ did not exist when the file was read, so the snapshot was empty — even though by the time the message printed, the file was there. On the second run the file already existed at read time, so the snapshot found it.

That is the signature of this bug: a Makefile that behaves differently on a clean checkout than on a machine that has already built once. It is why a clean target can appear to break a build, and why a colleague's "works for me" is not evidence.

The rule to take away: use wildcard for files a human wrote, never for files the build produces. Generated files should be named by the rules that make them, or derived from the source list with patsubst — which is the next section.

4. A Bare * Is Not the Same Thing

You can also write a glob directly in a prerequisite list, with no wildcard call. When the files exist, the two look identical:

glob: *.txt
	@echo "  bare glob matched: [$^]"

func: $(wildcard *.txt)
	@echo "  wildcard matched:  [$^]"

  bare glob matched: [one.txt two.txt]
  wildcard matched:  [one.txt two.txt]

The difference appears when nothing matches:

nomatch: *.nope
	@echo "  should not get here"

nomatch2: $(wildcard *.nope)
	@echo "  wildcard with no match: [$^] (rule still runs)"

make: *** No rule to make target '*.nope', needed by 'nomatch'.  Stop.
  wildcard with no match: [] (rule still runs)

A bare glob that matches nothing is passed through literally, so Make looks for a file whose name is the five characters *.nope, cannot find it, and stops — the No rule to make target answer from Chapter 2.1. $(wildcard) returns nothing and the rule proceeds with no prerequisites.

Neither behaviour is universally right, and that is the point of knowing both:

For a directory that must contain sources, the bare glob's loud failure is arguably better than silently compiling nothing. For an optional set of files, $(wildcard) is correct. What you should not do is use one and expect the other's behaviour.

5. $(patsubst ...) — Turn One List Into Another

patsubst rewrites every item in a list that matches a pattern. The % stands for "the part that varies":

SRCS := rtl/alu.v rtl/regfile.v

show:
	@echo "to .o        = [$(patsubst %.v,%.o,$(SRCS))]"
	@echo "to build/    = [$(patsubst rtl/%.v,build/%.img,$(SRCS))]"

to .o        = [rtl/alu.o rtl/regfile.o]
to build/    = [build/alu.img build/regfile.img]

Read the second one carefully, because it is the useful shape. The pattern rtl/%.v captures alu and regfile in the %, and the replacement build/%.img puts each back in a new path. One line turned a list of sources into the list of artifacts they produce.

That is how a Makefile derives generated names instead of globbing for them — the correct answer to §3's trap. The sources are discovered with wildcard because a human wrote them; the artifacts are computed with patsubst because the build will write them.

6. Four More You Will Actually Use

SRCS  := rtl/alu.v rtl/regfile.v
MIXED := rtl/alu.v rtl/defines.vh rtl/README.md

show:
	@echo "subst rtl->src = [$(subst rtl,src,$(SRCS))]"
	@echo "filter %.v     = [$(filter %.v,$(MIXED))]"
	@echo "filter-out %.md= [$(filter-out %.md,$(MIXED))]"
	@echo "sort+dedupe    = [$(sort $(SRCS) $(SRCS))]"

subst rtl->src = [src/alu.v src/regfile.v]
filter %.v     = [rtl/alu.v]
filter-out %.md= [rtl/alu.v rtl/defines.vh]
sort+dedupe    = [rtl/alu.v rtl/regfile.v]

• subst from,to,text replaces plain text anywhere it appears. No patterns, no % — it would happily rewrite rtl inside a longer word, so it is blunter than patsubst.
• filter pattern,list keeps the items that match. $(filter %.v,...) is how you take just the Verilog out of a mixed list.
• filter-out pattern,list keeps the items that do not. Useful for excluding a file from a globbed set — a work-in-progress module, or a testbench you compile separately.
• sort list sorts and removes duplicates. Worth reaching for whenever a list is written into a generated file.

Together with wildcard these four cover most of what a Makefile needs: discover a set, narrow it, make it reproducible.

7. Looking at a List

A few functions answer questions about a list rather than transforming it. They are mostly useful for messages and for sanity checks:

V_FILES := rtl/alu.v rtl/regfile.v

show:
	@echo "words     = [$(words $(V_FILES))]"
	@echo "firstword = [$(firstword $(V_FILES))]"
	@echo "notdir    = [$(notdir $(V_FILES))]"
	@echo "basename  = [$(basename $(V_FILES))]"
	@echo "dir       = [$(dir $(V_FILES))]"

words     = [2]
firstword = [rtl/alu.v]
notdir    = [alu.v regfile.v]
basename  = [rtl/alu rtl/regfile]
dir       = [rtl/ rtl/]

$(words ...) is the one to remember, because printing a count is the cheapest guard against §3's empty-list bug: a compile that reports compiling 0 files tells you immediately what a silent success would not.

Note basename strips the suffix, not the directory — the opposite of the shell command of the same name. notdir is what strips the directory.

8. $(foreach ...) — Do Something Per Item

foreach walks a list and builds a new piece of text for each item. It takes a variable name, the list, and the text to produce:

SRCS := rtl/alu.v rtl/regfile.v

show:
	@echo "[$(foreach f,$(SRCS),<$(f)>)]"

[<rtl/alu.v> <rtl/regfile.v>]

Inside the third argument, $(f) is the current item. Note that f exists only there — it is not a normal variable and it does not survive the call.

Most of what foreach can do, patsubst and addprefix do more clearly, so reach for it when the per-item text is more complicated than a rename. Its common real use is building a list of command-line options from a list of paths:

INCDIRS := rtl/include tb/include

show:
	@echo "[$(foreach d,$(INCDIRS),+incdir+$(d))]"

[+incdir+rtl/include +incdir+tb/include]

That is a genuinely common shape in simulator command lines, and $(addprefix +incdir+,$(INCDIRS)) produces the same thing more briefly.

9. $(shell ...) — and Why It Needs :=

shell runs a command and returns its output as the value:

HELLO := $(shell echo hello from the shell)
NFILES := $(shell ls rtl | wc -l | tr -d ' ')

show:
	@echo "[$(HELLO)]"
	@echo "[$(NFILES)]"

[hello from the shell]
[4]

This is powerful and the easiest function to misuse, for two reasons.

It runs while Make is reading the file, not when the recipe runs. So it sees the directory as it was before the build — the same timing as §3. A $(shell ls build) is almost always empty on a clean checkout.

With = it runs again on every single use. That is not a theoretical concern; here it is counted. Each invocation appends a line to a file, so the line count is the number of times the command actually ran:

LAZY   = $(shell echo run >> $(COUNT_FILE); echo value)
EAGER := $(shell echo run >> $(COUNT_FILE); echo value)

probe:
	@echo "lazy  1: $(LAZY)"
	@echo "lazy  2: $(LAZY)"
	@echo "lazy  3: $(LAZY)"
	@echo "eager 1: $(EAGER)"
	@echo "eager 2: $(EAGER)"
	@echo "total shell invocations recorded: $(shell wc -l < $(COUNT_FILE) | tr -d ' ')"

lazy  1: value
lazy  2: value
lazy  3: value
eager 1: value
eager 2: value
total shell invocations recorded: 4

Four invocations for five uses. EAGER ran once, at the line it was written, and its stored result was reused both times. LAZY ran three times — once per use. Every value printed is identical, so nothing looks wrong; the only symptom is time.

Put a git describe, a date, or a directory scan behind = in a large Makefile and it can run dozens of times per build, for the same answer. Always assign $(shell ...) with :=.

10. An RTL Flow That Discovers Its Own Sources

Now the whole module together: sources discovered, artifacts computed, recipes written with automatic variables, knobs overridable. This version is portable and was executed, standing in for a simulator:

.PHONY: all clean
.DEFAULT_GOAL := all

RTL_DIR := rtl
BUILD   := build

SOURCES := $(sort $(wildcard $(RTL_DIR)/*.v) $(wildcard $(RTL_DIR)/*.vh))
STAMPS  := $(patsubst $(RTL_DIR)/%,$(BUILD)/%.checked,$(SOURCES))

all: $(BUILD)/sim.img
	@echo "  [all] $(words $(SOURCES)) source files"

$(BUILD)/sim.img: $(SOURCES)
	@echo "  [compile] $(words $^) files -> $@"
	@mkdir -p $(@D)
	@cat $^ > $@

clean:
	@rm -rf $(BUILD)

Four things in that file are the module's whole content:

• SOURCES is discovered, from two patterns, and passed through sort so the list is reproducible (§2).
• STAMPS is computed from SOURCES with patsubst rather than globbed, because those files do not exist yet (§3, §5).
• $(words ...) reports the count in two places, so an empty list is visible immediately rather than silently succeeding (§7).
• The recipes name no files, using $^, $@ and $(@D) from Chapter 3.3.

The simulator version is illustrative — vcs is not installed here and was not run:

.PHONY: all clean
.DEFAULT_GOAL := all

RTL_DIR := rtl
BUILD   := build
SIM     ?= vcs
TOP     ?= tb_top

SOURCES := $(sort $(wildcard $(RTL_DIR)/*.v) $(wildcard $(RTL_DIR)/*.sv))
INCDIRS := $(RTL_DIR)/include tb/include
INCOPTS := $(addprefix +incdir+,$(INCDIRS))

all: $(BUILD)/simv

$(BUILD)/simv: $(SOURCES)
	mkdir -p $(@D)
	$(SIM) -full64 -sverilog $(INCOPTS) -top $(TOP) $^ -o $@

clean:
	rm -rf $(BUILD)

$(addprefix +incdir+,$(INCDIRS)) is §8's shape doing real work: a list of directories becomes a list of simulator options, with the option spelled once.

One honest limit, because this is the natural place to over-claim. A globbed $(wildcard rtl/*.v) compiles whatever happens to be in the directory — including a half-written file somebody left there, and in the wrong order for tools that care about compilation order. Real projects therefore keep an explicit filelist, and generating and consuming one correctly is Chapter 5.2's subject. What this chapter gives you is the machinery; which sources a flow should compile remains an engineering decision.

11. Common Mistakes

Using wildcard for generated files. It is a snapshot taken before the build, so it is empty on a clean checkout and correct afterwards (§3). Compute those names with patsubst.

Assigning $(shell ...) with =. The command re-runs on every expansion — measured at three runs for three uses in §9. Use :=.

Expecting a bare unmatched glob to be empty. It is passed through literally and becomes a hard No rule to make target error (§4).

Relying on wildcard's order. Wrap it in $(sort ...) when the order affects a generated file or a command line.

Using subst where patsubst was meant. subst replaces text anywhere, including inside a longer word; patsubst matches whole items against a pattern.

Forgetting the space-then-commas syntax. $(patsubst %.v,%.o,$(SRCS)) — space after the function name, commas between arguments, and no spaces after the commas unless you want them in the result.

Letting an empty list succeed silently. A glob that matches nothing produces a compile of zero files and a cheerful exit. Print $(words ...) so the number is in the log.

12. Check Your Understanding

Question 1. OBJS := $(wildcard build/*.o) sits at the top of a Makefile whose rules create those .o files. What does OBJS contain on a clean checkout, and what does it contain if you run make twice? Which function should have been used instead?

Question 2. A rule reads sim: $(wildcard rtl/*.sv) and the directory contains only .v files. What happens? What would happen with sim: rtl/*.sv?

Question 3. A Makefile defines VERSION = $(shell git describe --tags) and uses $(VERSION) in six recipes. How many times does git describe run, and what single character fixes it?

Question 4. Write one line that produces the list build/alu.img build/regfile.img from SOURCES := rtl/alu.v rtl/regfile.v.

13. Exercise

Create rtl/ containing defines.vh, alu.v, regfile.v and a README.md. Then this Makefile, which is meant to compile every RTL source and report how many there were.

.PHONY: all clean
.DEFAULT_GOAL := all

BUILD   := build
SOURCES := $(wildcard rtl/*)
ARTIFACTS := $(wildcard $(BUILD)/*.img)
STAMP = $(shell date +%s)

all: $(BUILD)/sim.img
	@echo "  [all] built from $(words $(SOURCES)) files, artifacts now [$(ARTIFACTS)]"

$(BUILD)/sim.img: $(SOURCES)
	@mkdir -p $(@D)
	@cat $^ > $@

clean:
	@rm -rf $(BUILD)

Tasks:

1. SOURCES has a bug that is about content, not timing. Find it.
2. ARTIFACTS has the timing bug from §3. Describe what it prints on a clean checkout versus a second run.
3. STAMP has the §9 bug. Say what is wrong even though it is never used.
4. Give the corrected Makefile.

14. Remember

• A function computes a value: $(name arg,arg) — space after the name, commas between arguments.
• $(wildcard pattern) asks the filesystem and returns the files that exist. No matches gives an empty list, not an error.
• A wildcard is a snapshot, not a live view. With := it is taken before any recipe runs, so it cannot see files the build is about to create — the classic clean-checkout bug.
• Use wildcard for authored files and patsubst for generated ones. Discover what a human wrote; compute what the build will write.
• An unmatched bare glob is a hard error; an unmatched $(wildcard) is an empty list. Choose deliberately.
• patsubst's % is string surgery, not a pattern rule. The rule form is Chapter 4.1.
• filter, filter-out, subst and sort narrow, exclude, replace and make a list reproducible. sort also removes duplicates.
• $(words ...) is the cheapest guard against a silently empty list.
• Always assign $(shell ...) with :=. With = it re-runs on every use — three uses, three invocations, measured.
• Wrap wildcard in $(sort ...) when the order of the list affects a generated file or a command line.

That completes Module 3 and Part I's work on variables. You can now write a Makefile that discovers its own sources, computes the names it will produce, parameterises itself for whoever runs it, and repeats no filename anywhere. Part I has one module left: Module 4 turns the single rules you have been writing into reusable ones — pattern rules that build a whole class of files, conditionals and includes that let one Makefile serve several configurations, the real behaviour of recipes when a command fails, and the flags that show you what Make is doing when none of it works.