Linux & Bash · Part I · Module 20 · Defensive Scripting

set -e, set -u, set -o pipefail — and Their Limits

This is the line every guide tells you to put at the top of a script, usually without qualification. It is worth putting there. It is also surrounded by exemptions that make it inert in several of the exact cases Chapter 20.1 catalogued — including, measurably, the entire body of a function you called inside an if. This chapter is that table, run rather than quoted.

Advanced24 min readLinuxPart IBashset -epipefailDefensive ScriptingScripting

Part I · Module 20 · Chapter 20.2 · set -e, set -u, set -o pipefail — and Their Limits

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/usr/bin/env bash
set -euo pipefail

Put it at the top of your scripts. Then read the rest of this chapter, because the gap between what people believe that line does and what it measurably does is the single most common source of misplaced confidence in shell scripting.

1. What Each One Is

Option	Long form	Effect
`-e`	`set -o errexit`	exit the script when a command fails
`-u`	`set -o nounset`	treat an unset variable as an error
`-o pipefail`	—	a pipeline fails if any stage failed, not just the last

Written out, which is what a script read by other people should use:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

set -o errexit
set -o nounset
set -o pipefail

Three lines instead of one, each greppable and each independently removable while debugging. The compact set -euo pipefail is fine and is what you will see most often; nothing in this chapter depends on which spelling you choose.

2. Where set -e Acts

Measured. These abort the script:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ bash -c 'set -e; false; echo REACHED'
$ echo "rc=$?"
rc=1

$ bash -c 'set -e; ls /nope; echo REACHED'
ls: /nope: No such file or directory
$ echo "rc=$?"
rc=1

$ bash -c 'set -e; f() { false; }; f; echo REACHED'
$ echo "rc=$?"
rc=1

REACHED never printed in any of the three. A plain failing step — including a bare call to a function that fails — stops the script. That is the behaviour you wanted, and it covers most of what people assume the option does.

Two more that abort, and the first surprises almost everybody:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ bash -c 'set -e; v=$(false); echo "REACHED v=[$v]"'
$ echo "rc=$?"
rc=1

$ bash -c 'set -e; (false); echo REACHED'
$ echo "rc=$?"
rc=1

A plain assignment from a failing command substitution does abort. v=$(false) stops the script. Hold on to that, because §4 shows the one-word change that makes it not.

3. Where set -e Does Nothing

Each of these prints REACHED:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ bash -c 'set -e; if false; then :; fi;      echo REACHED'
REACHED
$ bash -c 'set -e; false && echo X;           echo REACHED'
REACHED
$ bash -c 'set -e; false || echo X;           echo REACHED'
X
REACHED
$ bash -c 'set -e; ! true;                    echo REACHED'
REACHED
$ bash -c 'set -e; while false; do :; done;   echo REACHED'
REACHED
$ bash -c 'set -e; false | true;              echo REACHED'
REACHED
$ bash -c 'set -e; i=0; (( i++ ));            echo REACHED'
REACHED
$ bash -c 'set -e; (( 0 ));                   echo REACHED'
REACHED

There is one rule behind all of them: set -e steps aside wherever the shell can see that you are examining the status yourself. A condition, either side of && or ||, a negation, a non-final pipeline stage — in each case the status is being consumed by the construct, so aborting would make the construct impossible to write.

That rule is reasonable. Its consequence is not.

4. The Exemption That Matters

Where the abort-on-failure option acts, and where it steps aside

The option stops the script when a command fails as a plain step, with nothing examining its status. It is suspended for any command whose status is being consumed — a condition, either side of a chain, a negation — and that suspension applies to the whole body of a function called in such a position, so the function keeps running past its own failure and then reports whatever its last line returned.

Here is the measurement. One function, three call sites:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ f() { echo "  line1"; false; echo "  line2 AFTER FALSE"; }

$ bash -c 'set -e; f() { echo "  line1"; false; echo "  line2 AFTER FALSE"; }; f; echo "  REACHED"'
  line1

$ bash -c 'set -e; f() { echo "  line1"; false; echo "  line2 AFTER FALSE"; }; f || true; echo "  REACHED"'
  line1
  line2 AFTER FALSE
  REACHED

$ bash -c 'set -e; f() { echo "  line1"; false; echo "  line2 AFTER FALSE"; }; if f; then echo "  then"; else echo "  else"; fi; echo "  REACHED"'
  line1
  line2 AFTER FALSE
  then
  REACHED

Read those three.

Called bare, set -e works: the function stops at false and the script stops with it.

Called with || true, the suspension applies to the whole body. line2 AFTER FALSE ran. The function did not stop at its failure; it carried on doing work on a false premise — which is exactly Chapter 20.1's shape one, inside a script that has set -e at the top.

Called in an if, the same — and worse. The then branch ran. The function's last command was a successful echo, so the function returned 0, so if f took the true branch. The failure is now invisible twice over: once because set -e was suspended, and once because Chapter 18.3 §3's tail-echo masked the status.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ cat t1.sh
set -e
f() { false; echo "  tail echo ran"; }
if f; then echo "  f SUCCEEDED — the tail echo masked the failure"; else echo "  f failed"; fi

$ bash t1.sh
  tail echo ran
  f SUCCEEDED — the tail echo masked the failure

What this means for how you write functions

This is the practical takeaway of the chapter, and it is not "stop using if f".

set -e cannot be the error handling for a function you call in a condition. The construct if validate_inputs; then is good style — Chapter 16.7 argued for it — and inside that function set -e is off. So the function must handle its own failures explicitly:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# set -e will NOT save this function when it is called in a condition
check_tree() {
  cd "$1"                      # if this fails, the next lines still run
  [ -f Makefile ]
}

# it handles its own failures, so the call site can be anything
check_tree() {
  cd -- "$1" || return 1
  [ -f Makefile ] || return 1
}

The rule: a function that might be called in a tested position must return explicitly on every failure path it cares about. That is Chapter 18.3's "a function reports, the top level decides", and set -e's exemption is the reason it is not optional.

And the one-word version of the same thing, which Chapter 18.4 §5 measured from the other side:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ bash -c 'set -e; v=$(false);                          echo REACHED'
$ echo "plain assignment  rc=$?"
plain assignment  rc=1

$ bash -c 'set -e; f() { local v=$(false); echo "  IN FN"; }; f; echo "  REACHED"'
  IN FN
  REACHED

$ bash -c 'set -e; f() { local v; v=$(false); echo "  IN FN"; }; f; echo "  REACHED"'
$ echo "split form  rc=$?"
split form  rc=1

v=$(false) aborts. local v=$(false) does not. local v; v=$(false) aborts again. The middle one is exempt because the command whose status set -e sees is local, and local succeeded. Declare on one line and assign on the next — the same rule Chapter 18.4 gave for detecting errors at all, now with a second reason.

And one more exemption, which completes the picture in an uncomfortable way. A function called inside a command substitution is also in a tested position:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ printf 'TEST PASSED\n' > clean.log

$ cat a1.sh
set -e
f() { local n; n=$(grep -c UVM_ERROR clean.log); printf '%s\n' "$(( n ))"; }
v=$(f)
echo "REACHED v=[$v]"
$ bash a1.sh
REACHED v=[0]
$ echo "rc=$?"
rc=0

$ cat a2.sh
set -e
f() { local n; n=$(grep -c UVM_ERROR clean.log); printf 'count=%s\n' "$(( n ))"; }
f
echo "REACHED"
$ bash a2.sh
$ echo "rc=$?"
rc=1

The same function. Called via $(f) it runs past a failing assignment; called bare it aborts. (grep -c returns 1 when the count is zero — measured — so that assignment genuinely fails.)

Which means something worth stating plainly: whether set -e protects a function's body depends on its call site, and the call site is not visible from inside the function. You cannot read a function and know whether its unguarded steps are protected. That is the final argument for Chapter 18.3's rule — a function handles its own failures — not as style, but because set -e provably cannot be relied on to do it for you.

5. pipefail, and the Abort That Depends on Your Data

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ bash -c 'false | true; echo "no pipefail: $?"'
no pipefail: 0
$ bash -c 'set -o pipefail; false | true; echo "pipefail:    $?"'
pipefail:    1

That is the fix for Chapter 20.1's shape three, and it is the reason pipefail belongs in the line. With set -e as well, such a pipeline aborts the script.

Two details before you trust it.

It reports the last failing stage, not the first:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ bash -c "set -o pipefail; bash -c 'exit 3' | bash -c 'exit 4' | true; echo \"status=\$? PIPESTATUS=\${PIPESTATUS[*]}\""
status=4 PIPESTATUS=3 4 0

Status 4, not 3. PIPESTATUS (Chapter 12.8, an array per Chapter 19.2) has all three. If you need to know which stage broke, read the array.

And now the one that costs people real time. A consumer that exits before reading all its input makes the producer die of SIGPIPE, and pipefail counts that as a failure:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ bash -c 'set -o pipefail; yes | head -1; echo "rc=$?"'
y
rc=141

141 is 128 + 13, Chapter 7.6's signal arithmetic: SIGPIPE. Under set -e that aborts the script, in a pipeline that did exactly what it was supposed to do.

It gets worse, because whether it happens depends on how much data there is:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ for n in 10 1000 100000; do
>   seq 1 $n > f$n.txt
>   printf '  %6d lines: ' "$n"
>   bash -c "set -o pipefail; cat f$n.txt | head -1 > /dev/null; echo rc=\$?"
> done
      10 lines: rc=0
    1000 lines: rc=0
  100000 lines: rc=141

The same line of code, rc 0 on a small file and rc 141 on a large one. cat finishes writing before head exits when the file is small enough to fit in the pipe buffer, and gets killed when it is not. So a script with set -eo pipefail and a | head in it works all through development and dies in production on a bigger input.

What to do about it

Do not drop pipefail — it catches a real class of bug. Handle the specific pipelines that can legitimately end early:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# an early-exiting consumer: allow 141, or avoid the pipe
head -1 big.log                       # no pipe at all, when possible
grep -m1 PATTERN big.log              # grep's own "stop after one"
cat big.log | head -1 || [ $? -eq 141 ]   # explicit, and documents why

The first two are better: they remove the pipeline rather than excusing it. Measured on a 500,000-line file where every line matches:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ bash -c 'set -eo pipefail; grep PATTERN many.log | head -1 > /dev/null; echo REACHED'
$ echo "rc=$?"
rc=141

$ bash -c 'set -eo pipefail; grep -m1 PATTERN many.log > /dev/null; echo REACHED'
REACHED

The pipeline aborted the script; grep -m1 did the same job and returned 0. It is the right answer whenever you were about to write grep … | head -1, and it was available all along — Chapter 14.5's option list.

The general habit: when set -eo pipefail aborts on a pipeline you believe is correct, check for a consumer that stops early before you blame the option.

6. set -u, and What It Does Not Cover

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ bash -c 'set -u; echo "[$UNDEFINED_VAR]"'
bash: UNDEFINED_VAR: unbound variable

$ bash -c 'set -u; echo "[${UNDEFINED_VAR:-default}]"'
[default]

An unset variable is an error; a default supplies a value and is fine. That is Chapter 8.2's ${var:-word}, and it is why defensive scripts are full of ${1:-}.

It covers positional parameters and empty arrays too:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ cat t6.sh
set -u
echo "[$1]"
$ bash t6.sh
t6.sh: line 2: $1: unbound variable
$ echo "rc=$?"
rc=1

$ bash -c 'set -u; a=(); echo "${a[@]}"'
bash: a[@]: unbound variable

The array case is Chapter 19.2 §5, and the precise table there matters: "${a[@]}", "${a[*]}" and "${a[0]}" abort, while "${!a[@]}" and ${#a[@]} do not.

Two gaps worth knowing:

$@ and $* with zero arguments are fine:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ cat t7.sh
set -u
echo "count=$#"
echo "star=[$*]"
echo "at=[$@]"
echo REACHED
$ bash t7.sh
count=0
star=[]
at=[]
REACHED

So $1 is an error when there are no arguments and $@ is not — which is consistent (an empty argument list is a legitimate state; a missing first argument is a question about something that is not there) and worth knowing before you write a guard that relies on the wrong one.

And set -u is about unset, not empty:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ bash -c 'set -u; dir=""; echo rm -rf "$dir/"'
rm -rf /

dir is set. It is set to nothing. set -u has no objection, and Chapter 20.1's shape four is untouched by it. That is Chapter 20.4's entire subject, and it is the clearest example of why this one line is not a defence on its own.

7. Where to Put It, and When Not To

Put it in a script, after the shebang, before anything else:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/usr/bin/env bash
set -euo pipefail

Do not put it in a library you expect others to source. Chapter 18.6 §2 measured that sourcing shares shell options, so set -euo pipefail in lib.sh silently changes the error behaviour of every script that sources it — including scripts written years earlier that depend on the old defaults. A library sets no options; the script that uses it decides.

Do not rely on it in a #!/bin/sh script. set -e and set -u are POSIX; pipefail is not, and dash rejects it:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ dash -c 'set -o pipefail; echo ok'
dash: 1: set: Illegal option -o pipefail

And do not add it to a large existing script without testing. A script written under the old defaults may well depend on them — a grep -q that is expected to return 1 sometimes, a pipeline with a head in it. Turning the options on is a behaviour change, not a safety net you can apply retroactively for free.

8. A Complete Script

The options turned on, plus the explicit handling they do not provide:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/usr/bin/env bash
# summarise.sh — count errors across a results tree
# usage: summarise.sh <dir>
#
# exit: 0 no errors · 1 errors found · 2 unusable arguments
#
# note: set -e is suspended inside any function called in a condition,
#       so every function here returns explicitly on failure.

set -o errexit
set -o nounset
set -o pipefail

note() { printf '%s: %s\n' "${0##*/}" "$*" >&2; }
die()  { note "$*"; exit 2; }

# usable <dir> -> 0 if we can work with it. called in a condition, so
# set -e is OFF in here: every failure path returns explicitly.
usable() {
  [ $# -eq 1 ]   || return 2
  [ -d "$1" ]    || return 1
  [ -r "$1" ]    || return 1
  [ -x "$1" ]    || return 1          # needed to resolve names inside it
  return 0
}

# count_in <file> -> prints the count. no pipeline, so no pipefail risk.
count_in() {
  [ $# -eq 1 ] || return 2
  local n
  n=$(grep -c 'UVM_ERROR' "$1") || true   # grep returns 1 on zero matches
  printf '%s\n' "$(( n ))"
}

dir=${1:-}
[ -n "$dir" ] || die "usage: ${0##*/} <dir>"

if ! usable "$dir"; then
  die "$dir is not a usable directory"
fi

# -print0 and a read loop: no pipe, so the array survives (19.4)
declare -a logs=()
while IFS= read -r -d '' f; do
  logs+=("$f")
done < <(find "$dir" -type f -name '*.log' -print0)

if [ "${#logs[@]}" -eq 0 ]; then
  note "no logs under $dir"
  exit 0
fi

total=0
worst_n=0
worst=""

for f in "${logs[@]}"; do
  n=$(count_in "$f")
  total=$(( total + n ))
  if (( n > worst_n )); then
    worst_n=$n
    worst=$f
  fi
done

printf 'logs    %d\n' "${#logs[@]}"
printf 'errors  %d\n' "$total"
[ -z "$worst" ] || printf 'worst   %s (%d)\n' "$worst" "$worst_n"

[ "$total" -eq 0 ]

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ mkdir -p tree/sub
$ printf 'UVM_ERROR : a\nUVM_ERROR : b\n' > tree/one.log
$ printf 'TEST PASSED\n'                  > tree/two.log
$ printf 'UVM_ERROR : c\n'                > tree/sub/three.log

$ ./summarise.sh tree
logs    3
errors  3
worst   tree/one.log (2)
$ echo $?
1

$ rm tree/one.log tree/sub/three.log
$ ./summarise.sh tree
logs    1
errors  0
$ echo $?
0

$ mkdir bare && ./summarise.sh bare
summarise.sh: no logs under bare
$ echo $?
0

$ ./summarise.sh /nope
summarise.sh: /nope is not a usable directory
$ echo $?
2

Five decisions that exist because of this chapter's measurements:

The header comment says set -e is suspended in a tested function, and usable is written accordingly — four explicit || return lines rather than four bare tests. Without them, a usable whose first test failed would run the remaining three anyway.

n=$(grep -c ... ) || true. grep -c returns 1 when the count is zero — measured — which is not a failure in this context, and §2 showed that a plain assignment from a failing substitution does abort under set -e.

As it happens, removing the || true from this script changes nothing, because count_in is only ever called as n=$(count_in "$f") and §4's last measurement showed that position is exempt. Which is exactly why it stays: the guard documents the function's own contract rather than relying on a property of its one current call site. Move that call to a bare count_in "$f" > /dev/null and the unguarded version dies on the first clean log.

local n then n=$(...) inside count_in — §4's split form, so the assignment's status is visible rather than masked by local.

No pipeline anywhere. find ... -print0 feeds a read loop through process substitution (Chapter 19.4), and grep -c writes to a capture. There is nothing for pipefail to be surprised by — which is a design choice, not an accident.

The empty-tree case exits 0 with a message on stderr, not 2. Chapter 17.1 §5's question answered deliberately: a results directory with no logs in it has not failed, it has nothing to report, and the distinction is the script's to make.

Practice

Reproduce the §2 list: false, ls /nope, a bare failing function call, v=$(false), and (false). Confirm all five abort and that REACHED never prints.
Reproduce the §3 list. For each one, say in your own words which construct is consuming the status.
Run the three call sites of f() { echo line1; false; echo line2; } under set -e — bare, with || true, and in an if. Write down which print line2.
Add a trailing echo to a function that fails, call it in an if under set -e, and confirm the then branch runs. Name the two separate mechanisms that hid the failure.
Rewrite check_tree from the §4 callout so that it is correct when called in a condition. How many || return lines did it need?
Measure v=$(false), local v=$(false) and local v; v=$(false) under set -e. Explain the middle one.
Run false | true with and without pipefail. Then run a three-stage pipeline where two stages fail and print both $? and "${PIPESTATUS[*]}".
Reproduce the seq/head measurement at 10, 1,000 and 100,000 lines under pipefail. Find the smallest line count on your machine that produces 141.
Replace a grep PATTERN big.log | head -1 with grep -m1. Confirm the status under set -eo pipefail for both.
Under set -u, test $UNDEFINED, ${UNDEFINED:-d}, $1 with no arguments, $@ with no arguments, and dir="". Which two are not errors, and why is the last one the reason Chapter 20.4 exists?
Put set -euo pipefail in a library, source it from a script that relies on a grep -q returning 1, and watch what happens. Then state the rule.
Take summarise.sh, remove the || true from the grep -c line, and run it on a tree whose logs contain no errors. It still works — explain why using §4's last measurement. Then change the call site to a bare count_in "$f" > /dev/null and run it again.

Check Your Understanding

What are the three options, and what does each do?
Name three places set -e does not abort.
What is the single rule behind all of those exemptions?
A function fails on its second line and is called with || true. What happens to the rest of its body? What if it is called as v=$(f)?
Why does local v=$(false) not abort when v=$(false) does?
pipefail reports status 4 for a pipeline whose first stage exited 3. Why, and where is the 3?
Why can the same | head -1 pipeline abort on one file and not another?
Does set -u catch dir=""?

Answers

1. -e exits the script when a command fails; -u makes an unset variable an error; -o pipefail makes a pipeline fail if any stage failed rather than only the last. 2. Any three of: an if or while condition, either side of && or ||, a negated command, a non-final pipeline stage, $( ) inside a test, local v=$(cmd), (( i++ )). 3. set -e steps aside wherever the shell can see that you are examining the status yourself — because aborting would make those constructs impossible to write. 4. It keeps running — the suspension applies to the whole body, so the function continues past its own failure and returns whatever its last command gave. v=$(f) is the same: a command substitution is a tested position too. So whether a body is protected depends on the call site, which you cannot see from inside the function. 5. Because the command whose status set -e observes is local, and local succeeded. The substitution's failure is discarded before set -e can see it. Split the declaration from the assignment. 6. Because pipefail reports the last stage that failed, not the first. The 3 is in PIPESTATUS[0]. 7. Because a consumer that exits early kills the producer with SIGPIPE (status 141) only if the producer is still writing — which depends on whether the data fit in the pipe buffer. Measured: rc 0 at 1,000 lines, rc 141 at 100,000. 8. No. set -u is about unset, not empty. dir="" is set, and rm -rf "$dir/" still becomes rm -rf /.

9. Remember

Chapter 34.7 is why an explicit die beats relying on set -e in a flow: it measures two flows on a source that does not compile, one returning 4 with a message and the other printing RESULT: PASS.
set -euo pipefail belongs at the top of a script. It is worth having and it is not a defence on its own.
set -e acts on a plain failing step, including a bare call to a failing function, v=$(false), and (false).
It steps aside wherever you examine the status yourself — conditions, &&/||, !, non-final pipeline stages.
That suspension covers a whole function body. A function called with || true, in an if, or as v=$(f) runs past its own failures — so whether set -e protects a body depends on the call site, which is invisible from inside the function.
So a function that may be called in a condition must || return on every failure path it cares about.
local v=$(cmd) is exempt; v=$(cmd) is not. Declare, then assign.
pipefail reports the last failing stage. PIPESTATUS has them all.
pipefail plus an early-exiting consumer gives 141, and whether it happens depends on the size of the input. Prefer grep -m1 or no pipe at all.
set -u covers unset variables, unset $1, and an empty array's "${a[@]}" — but not $@/$* with no arguments, and not an empty variable.
Never put these options in a sourced library. Sourcing shares options; the script decides.
pipefail is not POSIX and dash rejects it.

Chapter 20.3 handles what set -u cannot: stating, once and at the top, what the script requires — so that a missing input is reported by name instead of becoming an empty expansion four steps later.

Where this fits

Part of the Linux & Bash curriculum.