Bash and Make: Steps vs Dependencies

Part II · Module 35 · Chapter 35.3 · Bash and Make

The flow built across Module 32 is a sequence: set up, expand, dispatch, collect, report. It runs all five every time, which is correct and sometimes wasteful.

A build is not a sequence. It is a set of things that depend on other things, and the useful question for any flow is which of the two it actually is.

1. A Flow as a Graph

$ command -v make; make --version | head -1
/usr/bin/make
GNU Make 3.81

# A flow as a dependency graph. The image depends on the sources; each run
# depends on the image.
ROOT    := $(shell cd .. && pwd)
SRCS    := $(ROOT)/proj/chip_a/tb/tb_arb.v $(ROOT)/proj/chip_a/rtl/arb.v
IMAGE   := build/sim.vvp
TESTS   := smoke fairness b2b
LOGS    := $(TESTS:%=build/%/run.log)

.PHONY: all clean
all: $(LOGS)

$(IMAGE): $(SRCS)
	@mkdir -p $(dir $@)
	iverilog -o $@ $(SRCS)
	@echo "   [compiled]"

build/%/run.log: $(IMAGE)
	@mkdir -p $(dir $@)
	@cd $(dir $@) && vvp $(ROOT)/mk/$(IMAGE) +TESTNAME=$* +SEED=1 +CYCLES=200 > run.log 2>&1
	@echo "   [ran $*]"

clean:
	rm -rf build

Three rules: the image depends on the sources, each test's log depends on the image, and all depends on every log. Nothing says when to do anything.

2. What That Buys, Counted

Timing it on this fixture is useless — everything takes under a second. So count the work Make chooses, which make -n reports without doing it:

=== how much work does make choose to do? (counted with make -n) ===
  from clean                             4 command(s)
  immediately after a full build         0
  after touching one source              4
  after deleting ONE test's log          1

Four readings, and the last is the argument.

From clean: 4. One compile and three runs — the whole graph.

After a full build: 0. make: Nothing to be done for 'all'. A script would have run all four again.

After touching one source: 4. Correct and worth noticing: the image changed, so every log that depends on it is stale. Make is not being clever here — it is being right, and a hand-written script that recompiled and then re-ran only the test somebody was interested in would have been wrong.

After deleting one test's log: 1. This is what a dependency graph is for. Make rebuilt exactly the missing artifact and left the other two alone. A script cannot do this without reimplementing staleness, and reimplementing staleness is what the rest of this chapter is about.

3. Make Compares Whole Seconds

Chapter 33.1 measured test -nt missing a source edit that landed in the same whole second as the build. Make rests on the same comparison, and the consequence is the same — reproduced here on the first attempt:

=== build and edit inside ONE whole second ===
  attempt 1: SAME second (1791648058)
    make wants 0 command(s)  -> THE EDIT IS INVISIBLE TO MAKE
    sub-second: image=1791648058.417882981 source=1791648058.544244023

The source is 127 milliseconds newer than the image, and Make believes the build is current. The filesystem knows; stat -f %Fm prints both to nanoseconds. Make 3.81 does not use them.

The 127 ms is just what this run happened to produce — repeating the probe gives a different gap each time (213 ms on the next attempt). The reproducible finding is not the number but the threshold: any gap that does not cross a whole-second boundary is invisible, whether it is one millisecond or nine hundred. That is why the fix is a digest rather than a tighter timestamp — make is comparing truncated seconds, so there is no gap small enough to be safe and no gap large enough to rely on.

This is not a reason to avoid Make. It is a reason to know which of its two uses you are making.

So the rule is the one that chapter arrived at from the other direction: use Make to decide what to do, and a recorded source digest to decide what a result means. They are different questions and only one of them tolerates a one-second blind spot.

In practice that means a Makefile can drive the build and the manifest from Chapter 33.1 is still mandatory:

$(IMAGE): $(SRCS)
	@mkdir -p $(dir $@)
	iverilog -o $@ $(SRCS)
	@$(ROOT)/bin/manifest $@ $(SRCS) > $@.manifest

Make decides whether to run that rule; the manifest records what it ran on. A flow with only the first has a fast build and unverifiable results.

4. Six Ways a Recipe Is Not a Shell Script

A Makefile recipe looks like shell and is not. Each of these was measured.

Spaces instead of a tab.

  Makefile.spaces:2: *** missing separator.  Stop.

The commonest first encounter with Make, and the error names neither tabs nor the line's actual problem.

Each line is its own shell.

  cd /tmp
  pwd
  /…/m35/mk/haz          <-- the ORIGINAL directory
  -- joined with && on one line --
  cd /tmp && pwd
  /tmp

cd on one line has no effect on the next. This is the same mechanism as Chapter 33.3's working-directory difference, except that here it resets between lines of what looks like one script.

A variable set in one line is gone in the next.

  x=5
  echo "x=[$x]"
  x=[]

Which follows from the previous one, and catches people who have just learned to join cd with && and assume the fix is general.

A target named like an existing file is "up to date".

  make: `clean' is up to date.

A file called clean existed, so the clean target had nothing to do. .PHONY: clean fixes it:

  echo cleaning
  cleaning

The recipe shell is /bin/sh.

  echo $0; echo "${BASH_VERSION:-not bash}"
  /bin/sh
  3.2.57(1)-release

An interesting pair: $0 is /bin/sh and BASH_VERSION is set, because on this machine /bin/sh is bash — invoked under a name that puts it in POSIX mode. So a recipe gets bash's binary with sh's rules, and a [[ ]] or an array that works in a script may or may not work in a recipe depending on which POSIX restriction it touches. SHELL := /bin/bash in the Makefile removes the ambiguity, and is worth setting explicitly rather than discovering.

A failing line stops the target; a failing pipe does not.

  false
  make: *** [a] Error 1

  false | cat
  echo reached
  reached

false | cat succeeded, so Make carried on. This is Chapter 12.8's rule — a pipeline's status is its last command's — arriving in a place where the usual fix is unavailable: set -o pipefail is a shell option, and a recipe line is a fresh sh. The remedies are to set SHELL := /bin/bash and .SHELLFLAGS := -o pipefail -c, or to avoid pipes in recipes and put them in a script the recipe calls.

That last option is the chapter's actual recommendation, for a reason beyond pipelines.

5. Where the Boundary Goes

The useful division is not "Make for builds, Bash for everything else". It is:

Make expresses the dependency graph. Bash expresses each step. A recipe should be one command.

build/%/run.log: $(IMAGE)
	@mkdir -p $(dir $@)
	@$(ROOT)/bin/runone $* $@

rather than eight recipe lines. Four reasons, each from a measurement in this track.

A script can be tested. Chapter 35.1's suite runs against bin/runone; nothing runs against a recipe body. A recipe is the one part of a flow with no test harness.

A script can be run by hand. Reproducing a failure means running the step — Chapter 33.2's whole subject — and make build/smoke/run.log rebuilds dependencies you did not want rebuilt, while bin/runone smoke out/ does exactly one thing.

A script keeps the shell you tested in. Section 4's /bin/sh-in-POSIX-mode ambiguity disappears if the recipe's only job is to invoke a #!/bin/bash script.

And a script has the guards. Everything Module 34 built — the bound, the size cap, the log check, the three-condition compile — lives in bin/. A recipe that calls iverilog directly has none of it, and the measured consequence is Chapter 34.1's: a 256-error compile exits 0, and Make will cheerfully treat the missing image as built because the rule succeeded.

6. Reading a Makefile Somebody Else Wrote

Four commands, in the order that makes an unfamiliar Makefile tractable.

$ make -n all              # what would it do?
$ make -p 2>/dev/null | grep -E '^[a-zA-Z][^ ]*:' | head -40   # every target
$ make --debug=b all       # why does it want to do that?
$ make -t all              # mark everything up to date, changing nothing

make -n first, always, because a Makefile's clean may not be the clean you expect — and because a target named after an existing file does nothing at all, which Section 4 measured and which -n makes visible in one line.

make -p dumps the database including Make's built-in rules, which is why the grep is needed: GNU Make ships implicit rules for C, Fortran, Pascal and more, and a .v file that happens to match one will be compiled by a rule nobody wrote. Deleting the built-ins is one line and worth it in a flow Makefile:

MAKEFLAGS += --no-builtin-rules --no-builtin-variables

And --debug=b answers the question that actually comes up — why is it rebuilding something that looks current — which on this machine is often Section 3's one-second window.

Practice

1. Express a small flow as three rules and run it from clean.
2. Run it again unchanged and read what Make says.
3. Count the work with make -n rather than timing it, and say why.
4. Delete one artifact and count again.
5. Touch one source and explain why everything downstream is rebuilt.
6. Build, then edit a source within the same second, and ask Make what it wants to do.
7. Print both mtimes to nanoseconds and confirm the source is newer.
8. Decide which of Make's two uses tolerates that window.
9. Add a manifest step to the build rule and say which question each mechanism answers.
10. Indent a recipe with spaces and read the error.
11. Put cd and pwd on separate recipe lines, then join them.
12. Set a variable on one line and read it on the next.
13. Create a file named after a target and run that target.
14. Add .PHONY and run it again.
15. Print $$0 and $$BASH_VERSION from a recipe.
16. Set SHELL := /bin/bash and explain what ambiguity that removes.
17. Put false | cat in a recipe and see whether the next line runs.
18. Fix it with .SHELLFLAGS, then argue for the other fix.
19. Reduce a multi-line recipe to one script invocation, and give four reasons.
20. Express a regression as one target per run, then list three things that breaks.
21. Explain why a simulation log is not like an object file.
22. Draw the Make/Bash boundary for your own flow.
23. Read an unfamiliar Makefile with -n, -p, --debug=b.
24. Disable the built-in rules and say what that prevents.

Check Your Understanding

1. What is the difference between a sequence and a dependency graph?
2. Give the four measured work counts.
3. Which one is the argument for Make, and why can a script not do it?
4. Why is counting commands better than timing them?
5. Why does the count need a grep?
6. What did Make do with an edit made 127 ms after the build?
7. Did the filesystem know? How do you show it?
8. Which of Make's two uses tolerates the one-second window, and which does not?
9. State the rule that follows.
10. What must the build rule still do, and why?
11. What happens if a recipe is indented with spaces?
12. What does cd on one recipe line do to the next?
13. What happens to a variable set in a recipe line?
14. What does a target named after an existing file do?
15. Which shell runs a recipe here, and what is odd about it?
16. Why set SHELL := /bin/bash explicitly?
17. What did false | cat do, and why is the usual fix unavailable?
18. Give the two remedies, and which this chapter prefers.
19. State the boundary rule in one sentence.
20. Give the four reasons a recipe should be one command.
21. What does a recipe that calls iverilog directly lack?
22. Why is a regression matrix not a dependency graph — give three reasons.
23. Why is a simulation log unlike an object file?
24. What default does a matrix want, and what is Make's?
25. Give the Make/Bash division for a verification flow.
26. What are the four commands for reading an unfamiliar Makefile, and why is -n first?
27. Why does make -p need a filter, and what should a flow Makefile disable?

7. Remember

• A sequence runs everything; a graph runs what is stale. Measured: 4 commands from clean, 0 when current, 1 after deleting one artifact.
• Count commands with make -n, do not time them — and filter, because -n prints mkdir and echo too.
• Make compares whole seconds. An edit 127 ms after the build was invisible, while stat -f %Fm showed the source newer.
• Use Make to decide what to do, and a digest to decide what a result means. Only the first tolerates a one-second blind spot.
• A recipe is not a shell script: spaces give missing separator; every line is a fresh shell so cd and variables do not carry; a target named after a file is "up to date" without .PHONY; the shell is /bin/sh, which here is bash in POSIX mode.
• A failing recipe line stops the target; false | cat does not. Set SHELL/.SHELLFLAGS, or keep pipes out of recipes.
• A recipe should be one command — a script. Testable, runnable by hand, in the shell you tested, and carrying Module 34's guards.
• A regression matrix is not a dependency graph. A log is irreplaceable, existence is not success, and a matrix needs every spec visible.
• Make for the build, Bash for the regression.
• Read an unfamiliar Makefile with -n first, filter -p, and disable the built-in rules in your own.

8. What Is Next

Make takes over the part of a flow that is a graph. Chapter 35.4 takes over a different part — and identifies the point at which a shell pipeline stops being the right tool for reading a log, measured against the pipelines this track has already built.