Makefile Debugging and Engineering Best Practices

Chapter 4.4 covered the flags. This chapter is about using them on a project, which is a different problem: at that scale the failures you care about are not errors. An error tells you where it is. The failures below produce a well-formed artifact, a sensible-looking log and an exit status of zero, and they can sit in a flow for months.

There are three of them worth knowing by sight.

1. A Stale Artifact Nobody Noticed

Here is a flow with one missing prerequisite. The image depends on the source; the result depends on nothing.

all: $(BUILD)/image $(BUILD)/result.txt
$(BUILD)/image: src.v
	@mkdir -p $(@D); echo "image-from-$$(cat src.v)" > $@; echo "  [compile] $@"
# BUG: result depends on nothing, so it is never redone
$(BUILD)/result.txt:
	@mkdir -p $(@D); printf '  [run] using '; cat $(BUILD)/image; cp $(BUILD)/image $@

  [compile] build/image
  [run] using image-from-v1

Correct. Now edit the source and rebuild:

  [compile] build/image

  result.txt now holds: image-from-v1

The image was rebuilt from v2. The result still reports v1. Nothing failed, nothing warned, and make exited 0.

Read that as a verification engineer. The design was changed, the flow reported a successful build, and the result on disk describes the previous design. If that result is what the regression report reads, the report is about a design that no longer exists — and it will keep being about that design until somebody deletes build/ for an unrelated reason, at which point the problem vanishes without ever being understood.

The symptom to recognise: a make that rebuilds something but not the thing downstream of it. The second transcript above has one line where it should have two, and that missing line is the entire bug.

2. The Variable That Expanded to Nothing

Make treats an undefined variable as the empty string, with no complaint at all. A one-character typo:

FILELIST := filelists/all.f
SIMV     := $(BUILD)/simv
run:
	@echo "  compiling from $(FILELSIT) into $(SIMV)"

  compiling from  into build/simv

The only evidence is the double space between "from" and "into". In a real recipe that empty expansion becomes iverilog -g2012 -f -o build/simv, and the error you get is from the tool, about its own command line, naming an option rather than a variable.

One flag turns it into a diagnostic:

  w.mk:5: warning: undefined variable `FILELSIT'
    compiling from  into build/simv

File, line, name. This is Chapter 4.4 §4's flag, and the reason it reappears here is that at project scale it is worth running always, not only when something is wrong:

MAKEFLAGS += --warn-undefined-variables

3. Why Did This Rebuild, or Not

When a target rebuilds and you do not know why — or worse, does not rebuild and you do not know why — --debug=b answers exactly that question and nothing else:

  Reading makefiles...
  Updating goal targets....
   Prerequisite `src/in.txt' is newer than target `build/out.txt'.
  Must remake target `build/out.txt'.
  Successfully remade target file `build/out.txt'.

It names the specific prerequisite. And in the other direction:

  make: `build/out.txt' is up to date.

That second line is the one to use against §1's stale artifact. Ask for the target that should have been redone and was not, and Make will tell you it considers it up to date — which means the thing you changed is not among its prerequisites.

There are heavier settings, and they are usually the wrong choice:

  --debug=b : 13 lines
  --debug=v : 40 lines

Three times the output for the same answer on a trivial example; on a real flow --debug=v is thousands of lines and --debug=a is worse. Reach for b first, because the question is almost always "why this target" rather than "what is Make doing in general".

The complementary flag is make -n, which prints recipes without running them, and make -W <file>, which answers "what would rebuild if I touched this?" without touching it:

make -n regress                 # what would run
make -W rtl/axi_lite_slave.v -n # what would a change to this source trigger

-W is the one worth remembering for a flow you have inherited. It is the fastest way to find out whether the dependency graph believes what you believe.

4. Reading the Database Without Drowning

Chapter 4.4 §7 introduced make -p and measured it at 1,313 lines on a nine-line Makefile, with the conclusion that it is a thing you grep rather than read. That conclusion does not change at project scale; what changes is that you now have a reason to reach for it, so it is worth knowing the two questions it answers that nothing else does.

First, the scale, re-measured here on an even smaller input to make the point unmissable — a two-line Makefile:

  lines in -p for a 2-line Makefile : 1254
  with -r (no built-in rules)       : 408

Twelve hundred lines for two lines of input, almost all of it Make's built-in rules for C, Fortran, Pascal, RCS and SCCS. (4.4's nine-line Makefile gave 1,313 — slightly more, because it defined more variables of its own. The built-in bulk is the constant; your Makefile is the rounding error.) So it is only ever useful filtered:

# What is this variable, and where was it set?
make -p | grep -A2 '^TESTS'

# Which rule will Make use for this target?
make -p | grep -B2 -A6 'build/simv:'

# Drop the built-in noise entirely
make -rp | grep ...

-r is worth knowing for a second reason: it disables the built-in rules, which on a large project is a measurable speed-up and, more importantly, stops Make inventing a rule you did not write. A flow that accidentally relies on the built-in %.o: %.c rule is a flow that behaves differently on a machine with a different Make.

The two questions worth -p are the ones 4.4 §7 named: where did this value come from, and which built-in rule might be matching my target. For everything else the lighter alternative, and the one to use day to day, is the print-% rule from Chapter 4.4 §3:

print-%:
	@echo '$* = [$($*)]  (origin: $(origin $*))'

make print-TESTS
make print-RUNDIR TEST=axi_burst SEED=42

Adding $(origin ...) to it is the upgrade this chapter recommends: it tells you whether the value came from the Makefile, the environment, the command line or a default, which is the difference between a bug in your file and somebody's shell profile. That is the same provenance -p reports in the comment line after each variable (4.4 §7) — the point of putting it in print-% is that you get it for one variable in one command, instead of grepping twelve hundred lines for it.

5. A Flow That Checks Itself

Every diagnostic so far is something a person runs after noticing a problem. The better investment is a target that notices for them.

.PHONY: check
check:
	@fail=0; \
	 for f in $(FILELIST) $(TESTLIST); do \
	   test -f $$f || { echo "  MISSING: $$f"; fail=1; }; \
	 done; \
	 for t in $(TESTS); do \
	   grep -q "\"$$t\"" tb/tb_top.v || { echo "  test '$$t' is in the list but not in the testbench"; fail=1; }; \
	 done; \
	 test -n "$(TESTS)" || { echo "  the test list is empty"; fail=1; }; \
	 [ $$fail -eq 0 ] && echo "  check: ok ($(words $(TESTS)) tests)" || exit 1

That second loop is the valuable one: it cross-checks the test list against the testbench, so a test added to one and not the other is caught by make check rather than by a UVM_FATAL twenty minutes into a regression. It is the kind of check only the project can write, because only the project knows the two things should agree.

  check: ok (7 tests)

  test 'axi_nonexistent' is in the list but not in the testbench
make: *** [check] Error 1

Two of its three branches are dead code

Then I tried to break the other two, and found something worth more than the check itself. Asking for a missing filelist:

mk/config.mk:16: *** filelist "filelists/nope.f" does not exist.  Stop.

mk/config.mk:20: *** test list "/tmp/empty.f" contains no tests.  Stop.

Neither message came from check. Both came from Chapter 8.3 §2's guards, which run while the Makefile is being read — long before any recipe. So by the time check executes, the filelist and the test list are already known to exist and to be non-empty, and the two branches testing for that cannot fire.

They are not broken. Removing the guards proves they work:

  MISSING: filelists/nope.f
make: *** [check] Error 1

  the test list is empty
make: *** [check] Error 1

They are simply unreachable, which makes them dead code — and the only reason I know that is that I tried to break them. A reviewer reading check would have counted three checks; the flow has one.

The useful response is not to delete them. It is to notice that the cross-check against the testbench is the only part that earns its place, because it is the only thing config.mk cannot express — a guard can test whether a file exists, but not whether two files agree with each other. That is the shape of a check worth writing: it asserts something no earlier layer already asserts.

6. The Review Checklist

These are not style preferences. Every line is a failure measured somewhere in this track, with the chapter that measured it.

Correctness

• Does every recipe that reads a file name that file as a prerequisite? (2.4 §8 and 8.2 §2 — correct serially, wrong on every parallel attempt in both.)
• Do generated dependencies come from the tool, not from a guess? (5.2 §7 — a globbed header pattern is right only by convention.)
• Is the target the thing whose existence means success, rather than a by-product? (5.4 §6 — with the log as the target, a failed run's second invocation exits 0.)
• Does anything two targets write live in two places? (8.2 §4 — 2 transitions serially, 17 under -j2, exit 0 both times.)

Status and verdicts

• Is the pass criterion read from the log, never from the tool's exit status? (5.1 §4, 7.4 §3 — both exit 0 on failure.)
• Is there a pipe between a check and the status you act on? (7.4 §5 — printed VIOLATED, exited 0.)
• Any || in a recipe that could be swallowing a real failure? (8.2 §2.)
• Does one place decide the overall answer? (6.4 §8.)
• Does the report walk the runs you expected, so a run that never happened gets a row? (6.4 §2 — "36 passed, 0 failed", exit 0, with four runs missing.)

Variables

• := for anything derived from $(shell ...)? (6.2 §7 — one assignment, three different seeds.)
• ?= on every knob a caller should be able to override? (3.2.)
• Is --warn-undefined-variables on? (§2 here.)
• Are fragment variables prefixed? (8.1 §6 — a silent collision no flag reports.)

Recursion and tools

• $(MAKE) rather than make in every recipe? (8.2 §6 — a bare one silently serialises.)
• -f $(firstword $(MAKEFILE_LIST)) on every sub-make? (6.4 §7 — -f is not inherited.)
• Is every tool named in a variable, never found on PATH? (7.1 §6 — the same export worked for two tools and failed for a third.)

Safety and usability

• Is every rm -rf guarded against an empty variable? (5.4 §8 — rm -rf $(BUILD)/ is rm -rf /.)
• Is the default goal safe? (2.4 — help, not something that runs.)
• Do help and log have no prerequisites? (5.4 §7.)
• Does make clean leave exactly the files a person wrote? (8.3 §7 — measured, seven entries.)

And the two that catch the rest

• Run make clean && make -j8 three times. Does it pass every time?
• Break each check on purpose. Does each one fail? (§5.)

7. A Debugging Session, in Order

When a flow misbehaves, this order gets to the answer fastest.

1. Reproduce it, and write down what you expected. "I edited the design and the result did not change" is a usable report. "The build is broken" is not.

2. Ask what Make thinks it will do, before letting it do anything:

make -n <the-target>

If the recipe you expected is absent, the problem is in the graph, not the recipe. Stop here and go to step 3.

3. Ask why, about the specific target:

make --debug=b <the-target>

Either it names the prerequisite that triggered the rebuild, or it says the target is up to date — and "up to date" when you expected a rebuild means the file you changed is not a prerequisite. That is the answer.

4. Ask what a value actually is, if the recipe ran but with the wrong arguments:

make print-FILELIST
make print-RUNDIR TEST=axi_burst SEED=42

Including $(origin ...), which distinguishes your Makefile from somebody's environment.

5. Ask what the tool actually received, if the values look right and the tool still complains. A wrapper that records argv (Chapter 7.1 §7) answers a question no log can.

6. Only then read the tool's log. It is the biggest artifact and the last place the answer usually is, because by this point you have already established whether the right command ran with the right inputs.

Steps 2 and 3 resolve most of it, and both are faster than opening a log.

8. What This Track Has Been Arguing

Thirty-two chapters, one idea.

A Makefile is a claim about dependencies, and Make will act on that claim without ever checking whether it is true. Everything else follows from that.

Every failure measured in this track is the same failure wearing different clothes — a claim that looked true and was not, with nothing complaining:

• A filelist named as a prerequisite while the sources inside it were not (5.2).
• A log used as a target, so a failed run looked finished (5.4).
• A test name in a variable Make cannot see, so a different test reused a cached result (5.3).
• A seed that was read and never applied, so two hundred runs tested one case (6.2).
• A report counting the files it found instead of the runs it expected (6.4).
• Two sub-makes with no way to state the dependency between them (8.1).
• A dependency satisfied by the order prerequisites happened to be listed in (8.2).

And the corollary, which is why Part II spent so much of its length on exit statuses: no tool will tell you. Icarus exits 0 on a failed test. Design Compiler exits 0 on a timing violation. A pipe replaces a failure with a success. || converts one into the other. .ONESHELL is ignored without a word. An undefined variable becomes an empty string. A $(filter) guard passes the empty string. Silence is the default behaviour of every layer in the stack.

So the engineering practice that matters is narrow and specific: state the dependency you actually have, make the target the thing whose existence means success, read the verdict from the artifact rather than the status, and prove every check can fail. Four habits. Everything in this track is one of those four, applied to a different part of a flow.

The reward is a flow somebody else can trust: one command, an answer in ten seconds, and an exit status a merge gate can act on — which is the whole difference between a Makefile that works on your machine and one a team depends on.

9. Common Mistakes

Treating "it works after a clean build" as a resolution. It is a bug report about a missing prerequisite (§1).

Running without --warn-undefined-variables. Measured: a typo'd variable expanded to nothing and the only evidence was a double space (§2).

Reaching for --debug=v or --debug=a first. Measured 40 lines against 13 for the same answer; b is the one that answers "why this target" (§3).

Reading make -p unfiltered. 1254 lines for a two-line Makefile (§4).

Relying on built-in rules. Use -r, so Make cannot invent a rule you did not write (§4).

Writing a check and never breaking it. Four separate defects in this track were checks that had never been proven able to fail (§5).

Reading the tool log first. It is the biggest artifact and the last place the answer usually is (§7).

Debugging a recipe when the problem is the graph. make -n separates those two in one command (§7).

10. Check Your Understanding

Question 1. A colleague reports that the flow "only works after make clean". What is your first question, and which command answers it?

Question 2. A recipe runs but the tool complains about a missing argument to -f. Name two commands that will find the cause faster than reading the tool log.

Question 3. Why is --debug=b the right first choice over --debug=v?

Question 4. A check target has passed every day for a year. What do you know about it?

Question 5. In one sentence, what is the single claim a Makefile makes, and what checks it?

11. Exercise

Step 1. Reproduce the stale artifact. Build §1's Makefile, run it, edit src.v, run it again:

  [compile] build/image

  result.txt now holds: image-from-v1

One line where there should be two, and the result is one version behind. Now ask Make why:

make --debug=b build/result.txt

It will tell you the target is up to date. Add $(BUILD)/image as a prerequisite and confirm both lines return.

Step 2. Introduce a one-character typo into a variable name in any Makefile you have, and run it twice:

  compiling from  into build/simv
  w.mk:5: warning: undefined variable `FILELSIT'

The second line is what --warn-undefined-variables adds. Then add MAKEFLAGS += --warn-undefined-variables near the top of the real flow from Chapter 8.3 and fix whatever it reports.

Step 3. Measure the database:

make -p | wc -l
make -rp | wc -l

  1254
  408

Step 4. Write the check target from §5 for the 8.3 flow, confirm it passes, then break each thing it checks in turn: rename the filelist, empty the test list, and add a test name to tests/testlist.f that is not in the testbench. All three must fail make check. If any passes, that branch of the check is decoration.

Step 5. The whole-flow test:

make clean && make -j8 report SEEDS="1 2 3 5"
make clean && make -j8 report SEEDS="1 2 3 5"
make clean && make -j8 report SEEDS="1 2 3 5"

Three identical results, or you have a race. Then grep the flow for the two places a parallel failure hides:

grep -rn '||' mk/ Makefile
grep -rn '[^(]make ' mk/ Makefile

Then do the audit on something real. Take the oldest Makefile you have access to and work down §6's checklist. Predict, before you start, how many items it will fail — and pay particular attention to the last two, because a flow that has never been run with -j and has never had its checks broken is a flow whose correctness has never actually been tested.

12. Remember

• The failures that matter at scale are not errors. They produce a plausible artifact, a sensible log and exit status 0.
• A stale artifact is a missing prerequisite. Measured: the image rebuilt from v2 while the result still reported v1, with nothing failing.
• "It works after a clean build" is a bug report, and the question it raises is which target did not know it was out of date.
• An undefined variable expands to the empty string, silently. Measured: the only evidence was a double space in a message.
• Run --warn-undefined-variables always, via MAKEFLAGS, and give genuinely optional variables a ?= default.
• No flag catches two fragments assigning the same variable — only prefixing does.
• --debug=b names the prerequisite, or says "up to date", which is the answer when you expected a rebuild. 13 lines against --debug=v's 40.
• make -W <file> -n answers "what would a change to this trigger" without touching anything — the fastest way to test an inherited graph.
• make -p is 1254 lines for a two-line Makefile, so filter it; -r removes the built-ins and stops Make inventing rules.
• Add $(origin ...) to the print-% rule, to tell your Makefile apart from somebody's environment.
• A check that has never failed is not known to work. Four separate defects in this track were exactly that.
• Debug in order: expectation, make -n, --debug=b, print-%, the argv, and the tool log last.
• A Makefile is a claim about dependencies, and nothing checks it. Every failure in this track was that claim being false while every layer stayed silent.
• Four habits carry all of it: state the dependency you actually have; make the target the thing whose existence means success; read the verdict from the artifact, not the status; and prove every check can fail.

That is the end of the track. Thirty-two chapters, from a three-line rule in Chapter 1.2 to a complete verification flow running twenty-eight simulations in parallel and answering with one exit status.

The flow in Chapter 8.3 is a reasonable starting point for a real project: clone the shape, replace mk/icarus.mk with your site's tool, put your filelist and test list in, and work down §6's checklist. What will not transfer is anything you take on trust — so when you adapt it, break each piece once and watch it fail. That is the only part of this track that cannot be read.