Parallel Execution and Build Reliability

Make knows which targets are independent, because the dependency graph says so. -jN tells it to run up to N of them at once, and on a regression of independent simulations that is close to free speed.

It is free only if the graph is true. Everything Make does in parallel it does on the strength of what you declared, so -j is best understood as a test of your prerequisites rather than a feature. Two things fail that test, and this chapter measures both.

Only one of the two is new here. Chapter 2.4 §7–§9 already met the first one in Part I, with three independent targets whose order moved under -j4 and a setup/build pair that passed serially and failed on all four parallel attempts. This chapter revisits it for two reasons — it is the failure that a project-scale flow actually hits, and there is a way of writing recipes that hides it even from -j — and then goes on to the second failure, which Part I never touched.

1. What -j Buys

Start with the number, so the rest has a reason. Seven tests across eight seeds — 56 independent runs — with the image already built, three paired repetitions on an eight-core machine:

  rep 1: serial  9202 ms | -j8  2001 ms | 4.60x
  rep 2: serial  8176 ms | -j8  2223 ms | 3.68x
  rep 3: serial  9958 ms | -j8  2272 ms | 4.38x

Between 3.7 and 4.6 times faster. On a real regression where each run is minutes rather than milliseconds, that is the difference between a sweep that fits in a coffee break and one that does not.

The whole speed-up rests on Make knowing those 56 runs are independent, which it knows because Chapter 6.3 gave each one its own directory and its own result file. That is the precondition. Without it, -j is not faster, it is wrong.

2. Failure One: a Dependency You Did Not Declare

This is Chapter 2.4 §8's bug, in its smallest form. Two targets, where b reads a file that a produces and never says so:

all: a b
a:
	@sleep 0.3; echo "generated-by-a" > shared.txt; echo "  [a] wrote shared.txt"
b:
	@printf '  [b] reads shared.txt -> '; cat shared.txt

Serially it is perfect, three times out of three:

  run 1:
    [a] wrote shared.txt
    [b] reads shared.txt -> generated-by-a
  run 2: (identical)
  run 3: (identical)

It works because all: a b lists a first, and Make processes prerequisites left to right when it has only one job. That is an accident of the order the names were typed.

Now two jobs:

  run 1:
    [b] reads shared.txt -> cat: shared.txt: No such file or directory
    [a] wrote shared.txt
  run 2: (identical)
  run 3: (identical)

  make rc=2
    [b] reads shared.txt -> cat: shared.txt: No such file or directory
  make: *** [b] Error 1
  make: *** Waiting for unfinished jobs....
    [a] wrote shared.txt

Three for three, and the Waiting for unfinished jobs.... line is Make telling you it has stopped starting new work and is letting a finish.

The bug was always there. b has always depended on a and has never said so. Serial execution was hiding it behind a coincidence, and -j4 removed the coincidence. If this had been in a flow for three years, the person who added -j did not break it.

That much is 2.4 §8's result reproduced — the same shape, a different filename, and three runs instead of four. Nothing about it is new, and that is the point worth making: a failure Part I demonstrated on four lines of Makefile is the same failure that takes down a regression, which is why the fix in §3 is also unchanged. What is new is the next paragraph.

3. The Fix Is One Word

all: b
b: a
	@printf '  [b] reads shared.txt -> '; cat shared.txt
a:
	@sleep 0.3; echo "generated-by-a" > shared.txt; echo "  [a] wrote shared.txt"

  -j4 run 1:   [a] wrote shared.txt|  [b] reads shared.txt -> generated-by-a|
  -j4 run 2:   [a] wrote shared.txt|  [b] reads shared.txt -> generated-by-a|
  -j4 run 3:   [a] wrote shared.txt|  [b] reads shared.txt -> generated-by-a|

b: a. That is the entire change, and it is not a parallelism fix — it is a correctness fix that the parallelism happened to force. Chapter 2.4 §9 made the same repair the slightly better way, by naming the file rather than the target: build: stamp.txt, with a rule that produces stamp.txt. Prefer that form when the producer has a real output, because it also lets Make decide whether the step needs redoing at all.

The habit that prevents the whole class: when a recipe reads a file, that file is a prerequisite. Not "the target that makes it" where you can avoid it, but the file itself, so Make can also decide whether it is out of date. That is why Chapter 5.2 went to the trouble of asking the compiler which files it read — a dependency you have to remember is a dependency you will eventually forget, and -j is what tells you.

4. Failure Two: Two Targets Writing One File

The second failure is not about ordering. Two independent targets both write to the same file, and they are genuinely independent — the graph is not lying.

all: w1 w2
w1:
	@for i in 1 2 3 4 5 6 7 8 9 0; do printf 'AAAAAAAAAA' >> out.txt; sleep 0.01; done
w2:
	@for i in 1 2 3 4 5 6 7 8 9 0; do printf 'BBBBBBBBBB' >> out.txt; sleep 0.01; done

Serially, the file has two clean halves:

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
  transitions A<->B: 2

Under two jobs, it is shredded:

AAAAAAAAAABBBBBBBBBBAAAAAAAAAABBBBBBBBBBBBBBBBBBBB
AAAAAAAAAAAAAAAAAAAABBBBBBBBBBAAAAAAAAAABBBBBBBBBB
AAAAAAAAAABBBBBBBBBBAAAAAAAAAABBBBBBBBBBAAAAAAAAAA
BBBBBBBBBBAAAAAAAAAABBBBBBBBBBBBBBBBBBBBAAAAAAAAAA
  transitions A<->B: 17

Two transitions serially, seventeen under -j2. Both runs "succeeded" — nothing failed, no error, exit status 0 — and the file is nonsense.

In a verification flow this is a shared log, a shared coverage database, a shared summary file, or a build/ scratch file two runs both use for a temporary. The symptom is not a crash; it is a report with lines from two tests mixed together, which somebody then reads as a single test behaving strangely.

There is no prerequisite that fixes this, because the problem is not ordering. The fix is structural: give each writer its own file.

all: out1.txt out2.txt
out1.txt:
	@for i in 1 2 3; do printf 'AAAAAAAAAA' >> $@; done
out2.txt:
	@for i in 1 2 3; do printf 'BBBBBBBBBB' >> $@; done

Which is exactly what Chapter 5.3 §6 did with run directories, and why Chapter 6.3 §8's 56 parallel runs produced a byte-identical report: nothing is shared, so there is nothing to race over. If two targets must contribute to one artifact, let each write its own piece and add a third target that depends on both and combines them — which is what a report target is.

5. Order-Only Prerequisites, and Directories

A directory is the one shared thing almost every target needs. Chapter 4.2 §10 covered the mechanism and the reason for it: a directory's timestamp changes whenever anything inside it changes, so as an ordinary prerequisite it makes every target rebuild whenever any sibling is written, and the fix is to list it after a | where its timestamp is never compared.

What Part I had no reason to ask is how that behaves when several jobs want the directory at the same time, which is the only part this section adds.

all: build/a.txt build/b.txt
build/a.txt: | build
	@echo a > $@
build/b.txt: | build
	@echo b > $@
build:
	@mkdir -p $@

Make 3.81 supports order-only prerequisites, and says so when asked:

  MAKE_VERSION = 3.81
  .FEATURES    = target-specific order-only second-expansion else-if archives jobserver check-symlink

The alternative that most flows use — @mkdir -p $(@D) as the first line of every recipe — also works, and is what this track has mostly done, because mkdir -p succeeds when the directory already exists and is therefore safe to run from many jobs at once. Two reasonable options:

mkdir -p $(@D) in the recipe is simplest, needs no extra rule, and is robust under -j because of -p. Use it when the directory is an incidental.

An order-only prerequisite is better when the directory is itself produced by something non-trivial, or when you want it to appear in the graph — and under -j it has a second advantage, because Make builds the directory target once no matter how many jobs depend on it.

Either way, the thing to avoid is plain mkdir without -p inside a recipe: that is a race under -j, because two jobs can both find the directory missing and the second one fails. -p is what makes the simple form safe, and it is why every recipe in Modules 5 to 8 writes mkdir -p $(@D) rather than mkdir $(@D).

6. The Jobserver: $(MAKE) or Silence

When Make runs sub-makes, the parallel budget has to be shared — otherwise -j8 at the top with eight sub-makes each doing -j8 is 64 jobs. GNU Make solves it with a jobserver: a pair of file descriptors the sub-makes use to claim a slot from one shared pool.

A sub-make joins that pool only if it is invoked as $(MAKE). Measured, with the same four-target sub-Makefile:

      [sub] s2
      [sub] s4
      [sub] s3
      [sub] s1

      [sub] s1
      [sub] s2
      [sub] s3
      [sub] s4

The first ran genuinely in parallel — the out-of-order completion proves it. The second ran strictly serially, and printed no warning of any kind. It simply lost the parallelism.

The reason is visible in MAKEFLAGS:

make      -> MAKEFLAGS = []
make -j4  -> MAKEFLAGS = [ --jobserver-fds=3,4 -j]
make -j4 -k -> MAKEFLAGS = [k --jobserver-fds=3,4 -j]

Two details worth keeping. -j appears with no number — the limit lives in the jobserver pipe, not in the flag, which is why a sub-make that does not inherit the descriptors cannot know the budget and defaults to one job. And -k appears as a bare k, with no dash, which is how Make encodes single-letter flags there — and a reminder from Chapter 6.4 §7 that -f never appears at all.

7. Interleaved Output, and 3.81's Missing Flag

Parallel output arrives interleaved, because several recipes are printing at once:

  [t1] start
  [t1] middle
  [t1] end
  [t2] start
  ...

  [t1] start
  [t2] start
  [t3] start
  [t1] middle
  [t2] middle
  [t3] middle
  [t1] end
  [t2] end
  [t3] end

Readable here with three targets printing three lines each; unreadable with fifty simulations printing thousands.

Newer GNU Make has --output-sync to group each target's output. This machine does not:

make: unrecognized option `--output-sync=target'

So on 3.81 the flow has to do it. Capture each target's output to its own file and print it when that target finishes:

all: t1 t2 t3
t1 t2 t3:
	@{ echo "  [$@] start"; sleep 0.1; echo "  [$@] end"; } > .$@.out 2>&1
	@cat .$@.out && rm -f .$@.out

  [t1] start
  [t1] middle
  [t1] end
  [t2] start
  [t2] middle
  [t2] end
  [t3] start
  [t3] middle
  [t3] end

Grouped again. And note this is the same move the flow already makes for a different reason: every run in Modules 5 to 7 writes its log to its own file in its own directory, so the terminal only ever carries one short status line per run. A flow built that way needs --output-sync much less.

8. Choosing a Number

-j with no number means unlimited, which on a 56-run regression starts 56 simulators at once. With a real simulator needing gigabytes each, that is how a machine starts swapping and everything takes longer than serial would have.

So always give a number. Measured on eight cores, 56 runs, image pre-built:

  -j1    8917 ms
  -j2    5063 ms
  -j4    2149 ms
  -j8    1922 ms
  -j16   1643 ms

The shape is the useful part. From -j1 to -j4 the time drops by a factor of four — almost ideal. From -j4 to -j8 it barely moves, and -j16 is only a little better still, because these runs are short enough that process startup and file I/O dominate rather than computation.

Do not read a general rule out of those last two rows. These are millisecond simulations of a tiny design. On a real flow each run holds a large memory image, and past the point where the working set exceeds RAM the curve does not flatten — it reverses, sharply. The number to pick is bounded by whichever of these is smallest:

• the cores you have, or were given on a shared machine;
• memory divided by what one run needs, which is usually the real limit for simulation and almost always for synthesis;
• the licenses available, which is a hard limit and not a performance question at all (Chapter 7.4 §8).

# A sane default the caller can override: make -j$(JOBS) or make JOBS=4
JOBS ?= 4
.PHONY: regress-p
regress-p:
	@$(MAKE) --no-print-directory -f $(firstword $(MAKEFILE_LIST)) -j$(JOBS) regress

.NOTPARALLEL, and when to reach for it

.NOTPARALLEL tells Make to ignore -j for this Makefile entirely:

.NOTPARALLEL:

  made dir
  a
  b

Serial, despite -j4. It is a legitimate tool in two situations: a flow that genuinely cannot be parallel — a tool that insists on a single shared work directory, a license-limited stage — and as a temporary measure when a parallel build is failing and you need the nightly job green while you find the missing dependency.

It is not a fix. Reaching for it permanently converts "we have races we have not found" into a policy, and the races are still there for anyone who runs a target directly. The honest version is to scope it: put the serial requirement on the stage that needs it, and leave the regression parallel.

9. Common Mistakes

Treating a serial pass as evidence. Measured: correct 3 of 3 serially, wrong 3 of 3 under -j4, with the bug present the whole time (§2) — and already measured on four attempts in Chapter 2.4 §8.

Blaming whoever added -j. The missing prerequisite was always there; -j removed the coincidence that hid it (§2).

|| in a recipe that reads a file. Measured: a race and exit status 0, because the failing command became the left side of a successful one (§2).

Depending on the target rather than the file. Name the file the recipe reads, so Make can also tell when it is out of date (§3).

Two targets appending to one file. Measured 2 transitions serially against 17 under -j2, with exit status 0 both times — no prerequisite fixes this, only separate files (§4).

mkdir without -p. Two jobs can both find the directory missing and the second one fails (§5); the order-only form is Chapter 4.2 §10's.

A bare make in a recipe. Measured: strictly serial, no warning, four times slower than it looked (§6).

Depending on .ONESHELL. Silently ignored on 3.81 — the line was echoed and the variable came back empty (§7).

Expecting --output-sync. Not present on 3.81; buffer per target instead (§7).

-j with no number. Unlimited parallelism starts every run at once and will exhaust memory on a real design (§8).

.NOTPARALLEL as a permanent fix. It hides races rather than removing them, and only for people who run the top-level target (§8).

10. Check Your Understanding

Question 1. A build has been green for three years. Somebody adds -j8 and it fails intermittently. What changed, and what did not?

Question 2. Two targets append to one log file. Adding a prerequisite between them makes the output clean again. Why is that still the wrong fix?

Question 3. A nightly job with a recursive sweep inside it takes all night while the machine is mostly idle, and nothing fails. What would you check first?

Question 4. Why does -j appear in MAKEFLAGS with no number?

Question 5. -j4 was four times faster than -j1, and -j8 was barely faster than -j4. Does that mean four jobs is the right setting for a real regression?

11. Exercise

Every transcript below is real output.

Step 1. Build the undeclared-dependency Makefile from §2 and run it serially three times — three passes. Then:

for i in 1 2 3; do rm -f shared.txt; make -j4 all; done

  [b] reads shared.txt -> cat: shared.txt: No such file or directory
  [a] wrote shared.txt

Step 2. Add || echo "FAILED" to b's cat and run it again under -j4. Check the exit status:

  make rc=0

The race is still there and Make now thinks everything is fine.

Step 3. Remove the ||, add b: a, and run three more times under -j4 — three passes.

Step 4. Measure the shared-file corruption. Build §4's two writers, run serially and then under -j2, counting character-run transitions:

sed 's/\(.\)\1*/\1/g' out.txt | wc -c

  serial: 2
  -j2   : 17

Step 5. Prove the jobserver point. Give a sub-Makefile four sleep targets, then call it both ways under make -j4:

      [sub] s2
      [sub] s4
      [sub] s3
      [sub] s1

      [sub] s1
      [sub] s2
      [sub] s3
      [sub] s4

Step 6. Find your own -j curve on the real regression from Chapter 6.3:

for j in 1 2 4 8 16; do make clean >/dev/null; make compile >/dev/null; \
  /usr/bin/time -p make -j$j regress SEEDS="1 2 3 5 7 11 13 17" >/dev/null; done

Then do the thing that actually matters. Take a Makefile you already rely on, run make clean && make -j8, and then run it twice more. If all three succeed, grep its recipes for || and for bare make, because those are the two places a parallel failure can be hiding behind a success. Predict what you will find before you look — and if the first -j8 run fails, you have just found a dependency that has been missing all along.

12. Remember

• -j is a test of your prerequisites, not a feature. Everything it does concurrently it does on the strength of what you declared.
• Measured: 3.7–4.6× on eight cores for 56 independent runs — which is available only because each run owns its own directory and result file.
• Failure one is a dependency you never declared — and it is not new: Chapter 2.4 §8 measured it in Part I. Correct 3 of 3 serially, wrong 3 of 3 under -j4, with the bug present all along.
• Serial execution satisfies missing dependencies by accident, in the order prerequisites happen to be listed.
• || in a recipe hides the failure as well. Measured: the race happened and make exited 0.
• When a recipe reads a file, that file is a prerequisite — not the target that makes it.
• Failure two is two targets writing one file, and no prerequisite fixes it. Measured 2 transitions serially against 17 under -j2, exit status 0 both times.
• One file per writer, plus a target that combines them. That is what a report target is.
• mkdir -p is safe under -j; plain mkdir is a race. Order-only prerequisites (|) are the alternative — the mechanism is Chapter 4.2 §10's, and 3.81 supports them.
• Use $(MAKE), never a bare make. Measured: a bare one ran strictly serially with no warning and the right exit status.
• -j in MAKEFLAGS carries no number, because the budget lives in the jobserver's descriptors — and -f is never there at all.
• 3.81 has no --output-sync and silently ignores .ONESHELL. Buffer each target's output to a file, and use backslash continuations.
• Always give -j a number, bounded by cores, by memory divided by one run's footprint, and by licenses.
• .NOTPARALLEL is a scope-limited tool or a temporary measure, never the fix.

A complete graph, split into files with owners, running correctly in parallel. Every piece this track has built now exists — and they have been introduced one at a time, each in its own chapter, against its own small example. Chapter 8.3 puts the whole thing in one place: a complete verification flow, every target exercised, with every line in it traced back to the chapter that explained why it is written that way.