Shell Execution and Error Handling

Part I · Module 4 · Chapter 4.3 · Shell Execution and Error Handling

1. Why This Chapter Was Deferred Four Times

Earlier chapters kept running into the same wall. Chapter 1.1 noted that a failing compile stops the simulation. Chapter 1.4 showed a failing recipe line stopping a rule. Chapter 2.1 found a recipe that printed a shell error and still reported success. Chapter 2.3 found a failed build leaving a corrupt file that the next run accepted.

All four are the same question: how does Make know whether a command worked?

The answer is a single number. Every command a shell runs finishes with an exit status — zero means success, anything else means failure. Make looks at that number and nothing else. It does not read output, it does not look for the word "error", and it does not inspect the files produced.

So the whole of error handling in Make comes down to keeping that number honest.

2. The Default: Stop on a Non-Zero Status

By default, a non-zero status stops the rule immediately:

.PHONY: plain
plain:
	@false
	@echo "  plain: never reached"

false is a command whose only job is to exit 1.

make plain

make: *** [plain] Error 1

The second line never printed, and make itself exited non-zero. This is the behaviour you want, and it is why a Makefile is safer than a shell script written without care — a simulation cannot run after a compile that failed.

3. Failure Is Per Line, Because Each Line Is a Shell

Chapter 1.4 established that Make hands each recipe line to its own shell. That has a direct consequence for errors: the status Make checks is the status of each line as a whole.

So a line's internal structure matters. These two recipes look similar and behave differently:

	cd results && rm -f *.log     # one line, one status: the && chain's
	cd results; rm -f *.log       # one line, one status: only rm's

With ; the line's status is the last command's, so a failed cd is invisible and the rm runs in the wrong directory — the destructive accident Chapter 1.4 warned about. With && the second command runs only if the first succeeded, and the line fails if either does.

Prefer && when joining commands that depend on each other. It is not a style preference; it is the difference between a failure Make can see and one it cannot.

4. A Pipeline Hides a Failure

Here is the single most useful fact in this chapter. In a pipeline, the shell reports the status of the last command only. Everything upstream is discarded.

.PHONY: piped plain
piped:
	@false | cat
	@echo "  piped: make thinks this succeeded"

plain:
	@false
	@echo "  plain: never reached"

make piped
echo "exit: $?"
make plain

  piped: make thinks this succeeded
exit: 0
make: *** [plain] Error 1

false failed in both. In piped its failure vanished because cat succeeded, so the line's status was zero and Make carried on — and make returned success to whatever ran it.

This is not a contrived example. Real recipes are full of pipelines:

	vcs -full64 -f filelist.f 2>&1 | tee compile.log
	./simv | grep -v "^UVM_INFO"
	wc -l < $< | tr -d ' ' > $@

In every one of those, the tool's failure is hidden behind the success of tee, grep or tr. A compile can fail and the Makefile will happily proceed to run a simulation. Chapter 2.1's exercise hit exactly this with wc -l < missing | tr -d ' ' — a visible shell error and exit status zero.

5. Fixing a Pipeline: set -o pipefail on the Same Line

pipefail is a shell option that makes a pipeline fail if any command in it fails. Because each recipe line is its own shell (Chapter 1.4), the option has to be set on the same line as the pipeline — a set on the previous line would die with its own shell.

SHELL := /bin/bash

.PHONY: guarded unguarded
unguarded:
	@false | cat
	@echo "  unguarded: reported success"

guarded:
	@set -o pipefail; false | cat
	@echo "  guarded: never reached"

make unguarded
make guarded

  unguarded: reported success
make: *** [guarded] Error 1

The guarded version failed, as it should. Two details matter:

• SHELL := /bin/bash is there because pipefail is not in the POSIX shell standard. Many systems' /bin/sh supports it anyway — it does on the machine these transcripts were recorded on — but naming bash explicitly is what makes the Makefile portable.
• The ; after set -o pipefail is required, and it is the one place a ; is right: you want the pipeline to run regardless, having changed the shell's behaviour first.

For a flow whose recipes pipe tool output into tee, this one habit converts a class of silent failures into real ones.

6. .SHELLFLAGS and .ONESHELL Need a Newer Make

You will find two tidier-looking solutions recommended elsewhere. Both are real GNU Make features — and both arrived in Make 3.82, so on an older Make they are silently ignored. That silence is the problem, so here is the measurement.

.SHELLFLAGS is meant to set the options every recipe shell starts with:

SHELL := /bin/bash
.SHELLFLAGS := -o pipefail -c

.PHONY: piped
piped:
	@false | cat
	@echo "  never reached"

make piped
echo "exit: $?"

  never reached
exit: 0

It did nothing. Asking the shell which options are actually active confirms why:

SHELL := /bin/bash
.SHELLFLAGS := -o pipefail -c

show:
	@echo "flags in effect? $$-"

flags in effect? hBc

The active flags are h, B and c — no pipefail. The directive was parsed as an ordinary variable assignment and never consulted.

.ONESHELL is meant to run a whole recipe in one shell, which would make cd persist between lines:

.ONESHELL:

.PHONY: where
where:
	cd sub
	pwd

cd sub
pwd
/work/chip/make-intro

The pwd printed the parent directory, so the two lines still ran in separate shells. .ONESHELL was inert too.

The lesson is not "avoid these features" — on Make 3.82 or later they work and are genuinely cleaner. It is that a Make directive your version does not know is not an error, it is a no-op. If a mechanism seems to have no effect, check make --version before anything else, and prefer the version-independent form in a Makefile that other people will run on machines you do not control.

make --version

GNU Make 3.81

7. Ignoring a Failure on Purpose: the - Prefix

Sometimes a command is allowed to fail — removing a file that may not exist, or a check whose result you only want reported. A - at the start of a recipe line tells Make to ignore that line's status:

.PHONY: tolerant
tolerant:
	-@false
	@echo "  tolerant: carried on"

make tolerant
echo "exit: $?"

make: [tolerant] Error 1 (ignored)
  tolerant: carried on
exit: 0

Note that Make still reports the failure — and marks it (ignored) — so it stays visible in a log while not stopping the build. The - goes before any @, and the two combine as -@command.

Use it narrowly. - on a compile or a simulation defeats the whole point of the exit status; rm -f already tolerates a missing file without it. The honest use is a command whose failure genuinely carries no information.

8. Carrying On Across Targets: make -k

The - prefix tolerates one line. make -k ("keep going") tolerates a whole failed target and continues with the others:

.PHONY: all one two
all: one two
one:
	@echo "  one fails"; false
two:
	@echo "  two ran"

make -k all
echo "exit: $?"

  one fails
make: *** [one] Error 1
  two ran
make: Target `all' not remade because of errors.
exit: 2

Three things happened, and all three are the point. one failed and was reported. two ran anyway. And make still exited 2 — -k changes how much work is attempted, never whether the run is called a success.

That makes it the right flag for a regression: you want every test attempted and a truthful verdict at the end, not a stop at the first failure. It is the wrong flag for a compile-then-run chain, where continuing is pointless.

9. Not Leaving Half a File Behind

Chapter 2.3 showed the worst consequence of a failure: a recipe that dies partway can leave a partially written target which is newer than its prerequisites, so the next make reports it up to date and exits 0.

The one-line guard is .DELETE_ON_ERROR:, which makes Make remove a target whose recipe failed. It belongs at the top of any Makefile whose recipes write their targets by redirection, and it is part of the error-handling strategy rather than a detail — a flow that can hand you a truncated artifact and call it current is worse than one that fails loudly.

10. The Hard Case: a Tool That Fails and Exits Zero

Everything so far assumed the tool tells the truth. Many do not. Simulators in particular routinely finish a run that found errors and exit 0, because from the tool's point of view the simulation succeeded — it ran to completion and reported what it found.

Here is a three-line stand-in that behaves exactly that way:

echo "UVM_ERROR : 3 errors detected"
echo "simulation finished"
exit 0

A naive recipe believes it:

.PHONY: naive
naive:
	@sh fake_sim.sh > run.log
	@echo "  naive: make says the run passed"

make naive
echo "exit: $?"

  naive: make says the run passed
exit: 0

Three errors in the log, and a green build. No Make feature can fix this, because Make is reading the number the tool gave it and the number says success. The check has to be yours.

.PHONY: checked
checked:
	@sh fake_sim.sh > run.log
	@if grep -q "UVM_ERROR : [1-9]" run.log; then \
	   echo "  checked: FAILED - errors in run.log"; exit 1; \
	 else echo "  checked: clean"; fi

make checked
echo "exit: $?"

  checked: FAILED - errors in run.log
make: *** [checked] Error 1
exit: 2

Now the build fails. The recipe reads the artifact the tool produced, decides for itself, and exit 1 converts that decision into the one number Make understands.

Two details in that recipe are deliberate. The pattern is UVM_ERROR : [1-9] rather than just UVM_ERROR, so a line reporting zero errors does not trip it. And the whole if is one recipe line, continued with backslashes — because each line is its own shell, an if split across lines would not work at all.

This is the single most valuable habit in a verification Makefile: decide what "passed" means, and check it yourself. Collecting those verdicts across a whole regression is Chapter 6.4's subject; the mechanism is here.

11. An RTL Flow That Checks Its Own Log

Putting the chapter together. This version is portable and was executed — the "simulator" is the stand-in script above:

.PHONY: all run clean
.DEFAULT_GOAL := all
.DELETE_ON_ERROR:

SHELL   := /bin/bash
BUILD   := build
SOURCES := $(sort $(wildcard rtl/*.v))

all: $(BUILD)/run.log

$(BUILD)/sim.img: $(SOURCES) | $(BUILD)
	@echo "  [compile] $(words $^) files -> $@"
	@set -o pipefail; cat $^ 2>&1 | tee $(BUILD)/compile.log > $@

$(BUILD)/run.log: $(BUILD)/sim.img
	@echo "  [run] $@"
	@sh fake_sim.sh > $@
	@if grep -q "UVM_ERROR : [1-9]" $@; then \
	   echo "  [run] FAILED - see $@"; exit 1; \
	 else echo "  [run] passed"; fi

$(BUILD):
	@mkdir -p $(BUILD)

clean:
	@rm -rf $(BUILD)

Five deliberate choices, each one this chapter's:

• .DELETE_ON_ERROR: so a failed step cannot leave a half-written artifact that looks current (§9).
• SHELL := /bin/bash so pipefail is available by name rather than by luck (§5).
• set -o pipefail; on the pipeline, so a compile failure is not hidden by tee (§4, §5).
• An explicit log check with exit 1, so a tool that exits zero cannot report a pass (§10).
• No - prefix anywhere, because nothing in this flow is allowed to fail quietly (§7).

Running it against a stand-in that reports errors:

make
echo "exit: $?"
ls build/

  [compile] 2 files -> build/sim.img
  [run] build/run.log
  [run] FAILED - see build/run.log
make: *** [build/run.log] Error 1
make: *** Deleting file `build/run.log'
exit: 2
compile.log	sim.img

Every mechanism is visible in those seven lines. The compile succeeded and was kept. The run completed, the log check rejected it, and the recipe's exit 1 became Make's Error 1. .DELETE_ON_ERROR: then removed build/run.log — confirmed by the ls, which shows it is gone — so the next make will genuinely retry rather than accept a rejected log as current. And make exited 2, so a script or CI job wrapping this sees the failure.

The simulator version is illustrative — no commercial tool was run:

.DELETE_ON_ERROR:
SHELL := /bin/bash

$(BUILD)/simv: $(SOURCES) | $(BUILD)
	set -o pipefail; vcs -full64 -sverilog $^ -o $@ 2>&1 | tee $(BUILD)/compile.log

$(BUILD)/run.log: $(BUILD)/simv
	set -o pipefail; ./$< -l $@ 2>&1 | tee $(BUILD)/run.tee
	@if grep -qE "UVM_(ERROR|FATAL) *: *[1-9]" $@; then \
	   echo "FAILED - see $@"; exit 1; fi

Both recipes pipe into tee so the output is both visible and saved, and both guard the pipeline. The log check is a regular expression over the simulator's own summary lines — which is a real pattern, though the exact text differs per tool and per UVM version, so it is one of the things a flow has to be told rather than something to copy blindly.

12. Common Mistakes

Assuming a green make means everything worked. It means every recipe line returned zero. A pipeline, a lenient tool or a - prefix can all produce that without success (§4, §7, §10).

Piping tool output into tee without pipefail. This is the most common real instance of §4, and it hides exactly the failures you most want to see.

Putting set -o pipefail on its own line. It dies with that line's shell. It must precede the pipeline on the same line (§5).

Trusting .SHELLFLAGS or .ONESHELL without checking the version. On Make 3.81 both are silently inert (§6).

Joining commands with ; when the second depends on the first. The line's status becomes the last command's, so the failure disappears (§3).

Sprinkling - prefixes to make a build "work". It makes the build quiet, not correct.

Using make -k for a compile-then-run chain. Continuing after a failed compile just produces a meaningless run. It is for regressions, where every test should be attempted.

Expecting Make to notice errors in a log. It reads one number. If "passed" means something more than that, the Makefile has to say so (§10).

13. Check Your Understanding

Question 1. A recipe line reads vcs -f filelist.f 2>&1 | tee compile.log. The compile fails. What status does Make see, what does it do next, and what is the one-line fix?

Question 2. An engineer adds .ONESHELL: so a recipe's cd will persist, and the recipe still runs in the wrong directory. Give two possible explanations and the command you would run first.

Question 3. What is the difference between -@rm -f stale.log and @rm -f stale.log? Which is redundant, and why?

Question 4. A nightly regression runs 200 tests. The simulator exits 0 even for tests that report UVM_ERROR. Describe what the Makefile must do so that make regress fails when any test fails, and say which flag you would also want.

14. Exercise

Create rtl/ with two .v files, plus fake_sim.sh from §10, and this Makefile.

.PHONY: all clean
.DEFAULT_GOAL := all

BUILD   := build
SOURCES := $(sort $(wildcard rtl/*.v))

all: $(BUILD)/run.log

$(BUILD)/sim.img: $(SOURCES) | $(BUILD)
	@echo "  [compile] -> $@"
	@cat $^ nosuchfile.v 2>&1 | tee $(BUILD)/compile.log > $@

$(BUILD)/run.log: $(BUILD)/sim.img
	@echo "  [run] -> $@"
	@sh fake_sim.sh > $@
	@echo "  [run] done"

$(BUILD):
	@mkdir -p $(BUILD)

clean:
	@rm -rf $(BUILD)

The compile reads a file that does not exist, and the simulation reports errors. Tasks:

1. Run make and record what it prints and its exit status. Is anything reported as failing?
2. Name the two separate reasons the failures are invisible.
3. Fix both, and add the guard that stops a failed step leaving a partial artifact.
4. Give the corrected Makefile.

15. Remember

• Make judges success by one number, the exit status of each recipe line. It never reads output and never inspects the files produced.
• A non-zero status stops the rule and anything depending on it, and make exits non-zero too.
• In a pipeline the shell reports only the last command's status, so tool | tee log hides the tool's failure completely.
• set -o pipefail; must be on the same recipe line as the pipeline, because each line gets its own shell. Name SHELL := /bin/bash so it is available portably.
• .SHELLFLAGS and .ONESHELL need Make 3.82 or later. On an older Make they are silently inert — measured, not assumed.
• && joins commands so a failure is visible; ; hides it by making the line's status the last command's.
• A - prefix ignores one line's failure and still reports it as (ignored). Use it only where failure carries no information.
• make -k attempts every target and still exits non-zero. Right for a regression, wrong for a compile-then-run chain.
• .DELETE_ON_ERROR: stops a failed recipe leaving a partial target that the next run would call current.
• A tool that exits zero after finding errors cannot be fixed by any Make feature. Decide what "passed" means, check the artifact yourself, and exit 1.

Part I has one chapter left. You can now write a Makefile that is correct, reusable and honest about failure — and when one of those things is not true, the next question is how to find out why. Chapter 4.4 covers the flags that show what Make intends to run, what a variable actually expanded to, and why it believes a target is up to date.