Organizing Makefiles for Large Projects

Every Makefile in this track so far has been one file, because every flow so far has been one subject. A real project is not like that. There are several blocks, each with its own sources and tests; there is a shared set of rules nobody wants to copy; there are flows for simulation, synthesis and timing that overlap but are not the same; and there are people who own one part and should not have to read the others.

There are two ways to split it, and they are not equally good. One of them has been the conventional answer for thirty years and is wrong for reasons that are easy to demonstrate.

1. The Two Structures

Recursive make gives every directory its own Makefile. The top level loops over the subdirectories and invokes make in each. It is the structure most projects reach for first, because it mirrors the directory tree and because each Makefile stays small.

SUBDIRS := liba libb
.PHONY: all $(SUBDIRS)
all: $(SUBDIRS)
$(SUBDIRS):
	@$(MAKE) --no-print-directory -C $@ all

Non-recursive make keeps one Make process with one dependency graph, and splits the text across files with include. Each directory contributes a fragment describing its own targets, and the top-level Makefile is little more than a list of fragments.

include liba/rules.mk
include libb/rules.mk
all: $(ALL_LIBS)

Both produce small, per-directory files. The difference is whether Make ends up with one graph or several, and that difference is measurable.

2. What Recursive Make Cannot Express

Here is the situation every real project has: libb needs something liba produced.

all: ../out/liba.lib
../out/liba.lib:
	@mkdir -p ../out; sleep 0.4; echo "liba" > ../out/liba.lib; echo "    [liba] built"

all: ../out/libb.lib
# libb NEEDS liba.lib, but this Makefile cannot express that - it is in another directory
../out/libb.lib:
	@mkdir -p ../out
	@printf '    [libb] needs liba.lib -> '; cat ../out/liba.lib 2>&1 || true
	@echo "libb" > ../out/libb.lib

Run it the ordinary way and everything is fine:

    [liba] built
    [libb] needs liba.lib -> liba

It works, and it works for a bad reason: liba comes first in SUBDIRS, so it happens to run first. Nothing in either Makefile says it must. Ask for two jobs at once:

  run 1:
      [libb] needs liba.lib -> cat: ../out/liba.lib: No such file or directory
      [liba] built
  run 2:
      [libb] needs liba.lib -> cat: ../out/liba.lib: No such file or directory
      [liba] built
  run 3:
      [libb] needs liba.lib -> cat: ../out/liba.lib: No such file or directory
      [liba] built

Three for three. And notice it is not even subtle here — libb ran first every time, because liba has a deliberate sleep in it. On a real project the two are closer in duration and the failure becomes intermittent, which is far worse.

The reason is structural rather than a mistake somebody made. libb/Makefile cannot name liba's output as a prerequisite in any way that works: the file lives outside its directory, and the rule that builds it lives in a Makefile this Make process will never read. Each sub-make has a complete graph of its own directory and no knowledge of any other. The real dependency exists in the code and has nowhere to live in the build system.

3. The Usual Fix, and What It Costs

The conventional repair is to state the ordering at the top level, where both directories are visible:

SUBDIRS := liba libb
all: $(SUBDIRS)
libb: liba          # hand-written cross-directory ordering
$(SUBDIRS):
	@$(MAKE) --no-print-directory -C $@ all

  -j2 run 1:  [liba] built [libb] needs liba.lib -> liba
  -j2 run 2:  [liba] built [libb] needs liba.lib -> liba
  -j2 run 3:  [liba] built [libb] needs liba.lib -> liba

Correct, three for three. And it has three costs worth naming.

It puts a fact about the code into the build system, by hand. That libb: liba line is a restatement of something the source already implies. When somebody adds a dependency from libb on a new libc, they must remember to come here and say so. Nothing checks that they did — the penalty for forgetting is an intermittent parallel failure months later.

It is coarser than the truth. The real dependency is one file on one file. What is written is one directory on another directory, so every target in libb now waits for every target in liba, including the ones that have nothing to do with each other.

It serialises what could overlap. With one real dependency between two files out of hundreds, almost all of the work in those two directories is independent, and this structure gives up all of it.

4. One Graph, Many Files

The alternative keeps one Make process. Each directory contributes a fragment that describes its own targets — and, crucially, can name anything in the project as a prerequisite, because there is only one graph.

LIBA := out/liba.lib
$(LIBA): liba/src.txt | out
	@sleep 0.4; cp $< $@; echo "    [liba] built $@"
ALL_LIBS += $(LIBA)

LIBB := out/libb.lib
# The real dependency, expressed where it belongs.
$(LIBB): libb/src.txt $(LIBA) | out
	@printf '    [libb] needs $(LIBA) -> '; cat $(LIBA)
	@cp $< $@; echo "    [libb] built $@"
ALL_LIBS += $(LIBB)

ALL_LIBS :=
include liba/rules.mk
include libb/rules.mk

.PHONY: all
all: $(ALL_LIBS)
	@echo "    [all] $(words $(ALL_LIBS)) lib(s): $(ALL_LIBS)"
out:
	@mkdir -p out

  run 1:
      [liba] built out/liba.lib
      [libb] needs out/liba.lib -> liba-content
      [libb] built out/libb.lib
      [all] 2 lib(s): out/liba.lib out/libb.lib
  run 2: (identical)
  run 3: (identical)

      [all] 2 lib(s): out/liba.lib out/libb.lib

Three for three under -j4, and nothing rebuilt on the second call. The ordering is now a consequence of a stated dependency rather than of the order of a list, so it cannot be got wrong by someone editing a different file.

Two details in those fragments are doing real work. Paths are relative to the top level, not to the fragment — there is one Make process with one working directory, so out/liba.lib means the same thing everywhere. And ALL_LIBS += lets each fragment add itself to a list the top level consumes, so adding a directory means adding one include line and nothing else.

The | out is an order-only prerequisite, explained in full in Chapter 4.2 §10: everything after the | must exist before the target is built, its timestamp is never compared, and it is excluded from $^. Here it means "make sure the directory exists first, but do not rebuild me when it changes". Chapter 8.2 §5 revisits it for the one thing Part I had no reason to raise — how it behaves when several jobs want the directory at once.

5. A Fragment That Knows Where It Lives

Writing top-level-relative paths by hand in every fragment is tedious and breaks when a directory moves. Make can tell a fragment its own location, because MAKEFILE_LIST grows as files are read and its last entry is the file currently being read:

THIS_MK  := $(lastword $(MAKEFILE_LIST))
THIS_DIR := $(patsubst %/,%,$(dir $(THIS_MK)))
$(info   a/rules.mk: THIS_MK=$(THIS_MK)  THIS_DIR=$(THIS_DIR))

a/rules.mk: THIS_MK=a/rules.mk  THIS_DIR=a
b/rules.mk: THIS_MK=b/rules.mk  THIS_DIR=b

Each fragment computed its own directory correctly. Note := rather than =: the value must be captured at read time, while that file is still the last word. With = it would be re-expanded later and give whichever file was read last.

Which is exactly what happens to the variable itself:

  after both includes, THIS_DIR = b   <-- the LAST one wins
  MAKEFILE_LIST =  Makefile a/rules.mk b/rules.mk

THIS_DIR is a single variable, so the second fragment overwrote the first. That is fine as long as each fragment uses it immediately, on the lines right after setting it, and does not expect it to survive. If a fragment needs its directory later — in a recipe, or in a list the top level reads — it must save it under its own name.

Also worth noticing: MAKEFILE_LIST has a leading space. That is why Chapter 6.4 §7 insisted on $(firstword $(MAKEFILE_LIST)) for naming the top-level file, and why $(lastword ...) is the right accessor here.

6. The Collision That Does Not Warn

One Make process means one namespace. Two fragments that happen to pick the same variable name do not get an error, a warning, or any indication at all.

SRCS := a/one.v a/two.v
ALL  += $(SRCS)

SRCS := b/three.v
ALL  += $(SRCS)

  SRCS (the last writer wins) = b/three.v
  ALL  (3) = a/one.v a/two.v b/three.v

SRCS holds only b/three.v. The a fragment's two files are gone. ALL happens to be right, with all three files — and the reason it survived is worth understanding: ALL += $(SRCS) on a :=-assigned variable is expanded at read time, so a's contribution was baked in before b overwrote SRCS.

So one of the two variables survived the collision and the other did not, depending only on when each was expanded. Nothing reported anything.

Two habits prevent it:

A_DIR  := $(patsubst %/,%,$(dir $(lastword $(MAKEFILE_LIST))))
A_SRCS := $(A_DIR)/one.v $(A_DIR)/two.v
ALL_SRCS += $(A_SRCS)

  A_DIR=a  A_SRCS=a/one.v a/two.v
  B_DIR=b  B_SRCS=b/three.v
  ALL_SRCS (3) = a/one.v a/two.v b/three.v

Prefix every variable a fragment owns with the fragment's name — A_SRCS, LIBB_OBJS, SIM_LABEL. And accumulate into one agreed shared list with +=, which is the only variable the top level needs to know about.

7. Who Owns Which Fragment

With the mechanism settled, the useful question is how to divide the files. The division that works is by owner and by subject, not by Make feature.

One fragment per thing that has an owner. A block, a tool, a flow. liba/rules.mk belongs to whoever owns liba; mk/vcs.mk belongs to whoever owns the VCS integration (Chapter 7.2 §3). Two engineers adding two simulators then never touch the same lines.

One fragment for configuration, read first, holding the variables and the guards. Everything else can then assume BUILD, SIM and TESTS are valid, because the flow refused to start otherwise.

One fragment per flow: simulation, regression, synthesis, cleaning, help. These are the files that hold rules rather than values, and they are where a reader goes to find out what make run does.

Nothing in the top-level Makefile but includes. If the top level grows a rule, that rule has no owner.

Applied to the flow this track has built, that is six files:

include mk/config.mk       # variables, guards, the test list
include mk/$(SIM).mk       # the only tool-specific file
include mk/sim.mk          # compile, run, waves, log
include mk/regress.mk      # many runs, and the report
include mk/clean.mk        # clean, distclean
include mk/help.mk         # help, sims, tests

Six lines, and a reader who wants to know how a waveform is produced goes straight to mk/sim.mk. Chapter 8.3 walks through all six in full.

8. When Recursion Is Still Right

The argument above is about a project whose parts depend on each other. Recursive make is not always wrong, and it is worth being clear about where it is fine.

A genuinely separate project — a vendor IP block with its own build system, a third-party library — is not part of your graph and should not be. Invoke its build as a single opaque step, declare its output as a prerequisite, and let it own its internals.

A sub-make for one isolated job, as the rerun target in Chapter 5.3 §8 and the sweep in Chapter 7.4 §8 do, where the point is to run one target with different variable values. That is recursion as a loop, not as a structure.

In both cases the rules from earlier chapters apply: use $(MAKE) and not make, so the jobserver and flags travel (Chapter 8.2 §6), and pass -f $(firstword $(MAKEFILE_LIST)) because -f is not inherited (Chapter 6.4 §7).

What is not fine is using recursion as the primary structure for parts of one project that depend on each other, because then the dependency graph has holes and §2's measurement is what the holes look like.

9. Common Mistakes

Recursive make for parts of one project. Measured: correct serially, wrong three times out of three under -j2, because no sub-make can name another directory's file (§2).

Treating serial success as proof. The ordering that makes it work is an accident of the subdirectory list, and the first person to add -j gets blamed (§2).

Ordering subdirectories by hand instead. It works, and it writes a fact about the code into the build system where nothing checks it, at directory granularity (§3).

Writing paths relative to the fragment. There is one Make process with one working directory; fragment paths are top-level-relative (§4).

THIS_DIR = with = instead of :=. It must be captured while that file is still the last word of MAKEFILE_LIST (§5).

Expecting THIS_DIR to survive the next include. It is one variable and the last writer wins — use it immediately or save it under a prefixed name (§5).

Unprefixed variables in a fragment. Measured: two fragments both using SRCS, and one fragment's files silently vanished with no warning of any kind (§6).

A rule in the top-level Makefile. Then it has no owner; the top level is a list of includes (§7).

config.mk not first. Every other fragment depends on its variables and its guards (§7).

10. Check Your Understanding

Question 1. A recursive build has worked for three years. A new engineer adds -j16 to the nightly job and the build starts failing intermittently. Who introduced the bug?

Question 2. Why can libb/Makefile not simply name ../out/liba.lib as a prerequisite?

Question 3. THIS_DIR := $(patsubst %/,%,$(dir $(lastword $(MAKEFILE_LIST)))) appears in three fragments. After all three are read, what does $(THIS_DIR) hold, and why is that usually harmless?

Question 4. Two fragments both set SRCS and both do ALL += $(SRCS). Measured, SRCS ended up with one fragment's value but ALL had everything. Explain the difference.

Question 5. When is a sub-make the right answer rather than an include?

11. Exercise

Build both structures and compare them. Every transcript is real output.

Step 1. Create the recursive version from §2 and run it serially:

make -C rec all

    [liba] built
    [libb] needs liba.lib -> liba

Step 2. Run the same thing with two jobs, three times:

for i in 1 2 3; do rm -rf rec/out; make -j2 -C rec all; done

    [libb] needs liba.lib -> cat: ../out/liba.lib: No such file or directory
    [liba] built

Step 3. Add libb: liba to the top-level Makefile and repeat step 2 — all three runs now correct. Then ask yourself what happens when somebody adds libc that liba needs, and does not edit this file.

Step 4. Build the non-recursive version from §4 and run it under -j4:

for i in 1 2 3; do rm -rf inc/out; make -j4 -C inc all; done

    [liba] built out/liba.lib
    [libb] needs out/liba.lib -> liba-content
    [libb] built out/libb.lib
    [all] 2 lib(s): out/liba.lib out/libb.lib

Step 5. Prove the collision. Put SRCS := a/one.v a/two.v in one fragment and SRCS := b/three.v in another, both with ALL += $(SRCS), and print both:

  SRCS (the last writer wins) = b/three.v
  ALL  (3) = a/one.v a/two.v b/three.v

Then try to get Make to tell you about it. Run the same thing with --warn-undefined-variables and with --debug=v. Neither reports the collision, because nothing is undefined and nothing is ambiguous — one file simply assigned a variable another file had already assigned. Predict that before you run it, then add the A_/B_ prefixes and confirm both fragments' values survive.

12. Remember

• Recursive make gives one graph per directory, and no graph can name a file whose rule lives in another.
• Measured: correct serially, wrong 3 of 3 under -j2 — and on a real project the failure is intermittent rather than consistent.
• Serial success proves nothing. The ordering is an accident of the subdirectory list, and it is load-bearing without being written down.
• Hand-ordering subdirectories works and costs three things: a fact about the code restated in the build system, directory-coarse granularity, and all the parallelism between those directories.
• Non-recursive make keeps one process and one graph, with the text split by include — measured correct 3 of 3 under -j4.
• Fragment paths are top-level-relative, because there is one Make process with one working directory.
• Each fragment adds itself to a shared list with +=, so a new directory costs one include line.
• $(lastword $(MAKEFILE_LIST)) is the file being read, so a fragment can find its own directory — with :=, and used immediately.
• MAKEFILE_LIST has a leading space; use $(firstword ...) and $(lastword ...), never the bare variable.
• One process means one namespace, and collisions are silent. Measured: one fragment's SRCS vanished entirely, while ALL survived only because += expanded at read time.
• Prefix every variable a fragment owns, and accumulate into one agreed list. No Make flag will warn you.
• Divide fragments by owner and subject — a block, a tool, a flow — with configuration first and nothing but includes at the top level.
• Recursion is still right for a genuinely separate project, and as a loop to re-run one target with different values.

One graph, split across files that each have an owner. That graph is now complete enough to be worth running in parallel — which is the thing the whole chapter has been pointing at, because a complete graph is exactly the precondition for -j being safe. Chapter 8.2 measures what -j buys, and the two specific ways a flow that works serially can still be wrong when it runs concurrently.