trap: Cleanup That Runs Even on Failure

Part I · Module 20 · Chapter 20.5 · trap: Cleanup That Runs Even on Failure

Chapter 20.2 turned on set -e, which means a script can now stop at any line. Chapter 20.3 put the validation before anything is created, which helps. But a script that creates a scratch directory on line 20 and fails on line 40 still leaves it there.

1. One Handler, Every Ordinary Exit

#!/usr/bin/env bash
tmp=$(mktemp -d)
trap 'echo "  cleanup: removing $tmp"; rm -rf -- "$tmp"' EXIT
echo "  working in $tmp"
touch "$tmp/scratch"
exit 0

$ bash tr1.sh
  working in /…/tmp.rk4SVw2vCe
  cleanup: removing /…/tmp.rk4SVw2vCe

trap COMMANDS EXIT registers a command to run when the shell exits — Chapter 7.6's signals, with EXIT as a pseudo-signal meaning "on the way out, however that happens".

And it really is however that happens. On an explicit non-zero exit:

$ cat tr2.sh
tmp=$(mktemp -d)
trap 'echo "  cleanup ran"; rm -rf -- "$tmp"' EXIT
echo "  about to fail"
false
exit 7

$ bash tr2.sh
  about to fail
  cleanup ran
$ echo $?
7

And on a set -e abort, which is the case that matters most:

$ cat tr3.sh
set -e
trap 'echo "  cleanup ran (rc was $?)"' EXIT
echo "  about to fail"
false
echo "NOT REACHED"

$ bash tr3.sh
  about to fail
  cleanup ran (rc was 1)
$ echo $?
1

The handler ran, $? inside it was the failing command's status, and the script's exit status was preserved. That is the whole value of EXIT: one place, every path, and you do not have to remember to clean up at each of the fifteen die calls Chapter 20.3 added.

2. Register It Immediately After Creating

The order of those first two lines is not arbitrary:

tmp=$(mktemp -d)                      # created
trap 'rm -rf -- "$tmp"' EXIT          # registered — one line later

Anything between them is a window in which a failure leaves debris. And it must be after, because a handler registered first would run with $tmp unset — which under set -u is an error inside the handler, and without it is rm -rf -- on nothing.

Use a function once the handler is more than one command, and use single quotes so $tmp is read when the handler runs rather than when it is registered:

cleanup() {
  rm -rf -- "${tmp:-}"
  [ -z "${lock:-}" ] || rm -f -- "$lock"
}
trap cleanup EXIT

${tmp:-} with the default matters: the handler can fire before tmp is assigned if something fails very early, and under set -u an unset variable in the handler turns a tidy exit into a confusing second error.

3. A Trap Can Change the Exit Status

Chapter 15.6 measured this and it belongs here:

$ cat tr4.sh
trap 'echo "  handler"; exit 9' EXIT
exit 3

$ bash tr4.sh
  handler
$ echo $?
9

The script asked for 3 and exited 9. An exit inside an EXIT handler overrides the status the script was leaving with — so a handler that ends in exit 0, or in a command that happens to be the last thing it runs, can silently convert a failure into a success.

The fix is to capture the real status on the handler's first line:

$ cat tr5.sh
cleanup() { local st=$?; echo "  handler saw rc=$st"; exit "$st"; }
trap cleanup EXIT
exit 3

$ bash tr5.sh
  handler saw rc=3
$ echo $?
3

local st=$? must be first, for Chapter 11.1's reason — anything before it overwrites $?. (And note this is one of the few places local st=$? is correct rather than the Chapter 18.4 trap: $? is not a command substitution, so there is no status being masked.)

In practice, a handler that only removes files needs none of this: rm -rf succeeding leaves $? from the rm, which is 0, and that becomes the status. So:

# quietly converts every failure into a success
trap 'rm -rf -- "$tmp"' EXIT

is a real bug in a script whose exit status anyone reads. Either capture and re-exit, or make the handler's last command harmless:

cleanup() { local st=$?; rm -rf -- "${tmp:-}"; return "$st"; }
trap cleanup EXIT

return "$st" from the handler leaves the status alone, which is what you want in the common case.

4. ERR, and Where It Does Not Fire

ERR runs the handler when a command fails — the same trigger as set -e:

$ cat tr6.sh
set -e
trap 'echo "  ERR at line $LINENO (rc $?)"' ERR
echo "  line 3"
false
echo "NOT REACHED"

$ bash tr6.sh
  line 3
  ERR at line 4 (rc 1)
$ echo $?
1

A line number and a status, which is a genuinely useful error report and is how a script gives you a backtrace. With Chapter 18.2's FUNCNAME and BASH_SOURCE arrays you can produce a real one:

on_err() {
  local st=$? line=$1
  printf '%s: error at %s:%s (status %d) in %s\n' \
    "${0##*/}" "${BASH_SOURCE[1]##*/}" "$line" "$st" "${FUNCNAME[1]:-main}" >&2
}
trap 'on_err "$LINENO"' ERR

And ERR has exactly set -e's exemptions, which is the thing to know:

$ cat tr7.sh
set -e
trap 'echo "  ERR fired"' ERR
if false; then :; fi
false || true
echo "  REACHED, no ERR"

$ bash tr7.sh
  REACHED, no ERR

Two failing commands, no ERR. Chapter 20.2 §3's rule again: a status being examined is not an error. So ERR is a reporting mechanism for the failures set -e would abort on, and it cannot see the ones set -e misses — including, measurably, everything inside a function called in a condition.

5. The Interrupt, Measured Three Ways

This is the part people get wrong, and it needs three scripts to see.

Each one signals itself, so a wrapper can read the real exit status — sending the signal from outside and then calling wait reports the wrapper's status, not the script's, which is how this gets measured wrongly.

# n1.sh — a trap, no re-raise
tmp=$(mktemp -d)
trap 'echo "  [n1] cleanup"; rm -rf -- "$tmp"' EXIT INT
echo "  [n1] running"
kill -INT $$
sleep 2
echo "  [n1] NOT REACHED"

# n2.sh — a trap that clears itself and re-raises
tmp=$(mktemp -d)
cleanup() { echo "  [n2] cleanup"; rm -rf -- "$tmp"; }
trap cleanup EXIT
trap 'cleanup; trap - INT EXIT; kill -INT $$' INT
echo "  [n2] running"
kill -INT $$
sleep 2
echo "  [n2] NOT REACHED"

# n3.sh — no trap at all
echo "  [n3] running (no trap)"
kill -INT $$
sleep 2
echo "  [n3] NOT REACHED"

$ for s in n1 n2 n3; do ./$s.sh; printf '  -> %s exit=%s\n' "$s" "$?"; done
  [n1] running
  [n1] cleanup
  [n1] NOT REACHED
  [n1] cleanup
  -> n1 exit=0
  [n2] running
  [n2] cleanup
  -> n2 exit=130
  [n3] running (no trap)
  -> n3 exit=130

Read n1 carefully, because it is the version most people write.

The cleanup ran. Then NOT REACHED printed. Then the cleanup ran again. And the script exited 0.

Four things wrong in one run. Trapping INT without re-raising means the handler returns and execution resumes at the next line — so the script finished the work it was interrupted out of, cleaned up twice (once for INT, once for EXIT), and told its caller everything was fine. A nightly job interrupted by a queue timeout would report success.

n3 gets the status right — 130, which is Chapter 7.6's 128 + 2 for SIGINT — and leaves the temporary directory behind, because there is no handler at all.

n2 is the correct shape, and it has three parts:

trap 'cleanup; trap - INT EXIT; kill -INT $$' INT
#     ^clean    ^remove both    ^re-raise

• Clean up, because that is why you trapped it.
• Remove the handlers, so the re-raised signal is not caught again and the EXIT handler does not clean up a second time.
• Re-raise with kill -INT $$, so the shell dies of the signal and the caller sees 130.

exit 130 instead of the re-raise is almost equivalent and is what you will often see. The difference matters when the script is itself a child: a caller using wait can distinguish "killed by SIGINT" from "exited with 130", and a make or a job scheduler may act on that. Re-raising is the honest version.

6. A Complete Script

Cleanup on every path, correct statuses, and an interrupt that reports itself:

#!/usr/bin/env bash
# build-with-cleanup.sh — build in a scratch dir, cleaning up whatever happens
# usage: build-with-cleanup.sh [-k] <srcdir>
#        -k  keep the scratch directory (for debugging)
#
# exit: 0 built · 1 build failed · 2 unusable arguments · 130 interrupted

set -o errexit
set -o nounset
set -o pipefail

note() { printf '%s: %s\n' "${0##*/}" "$*" >&2; }
die()  { note "$*"; exit 2; }

keep=0
case ${1:-} in
  -k) keep=1; shift ;;
esac

src=${1:-}
[ -n "$src" ] || die "usage: ${0##*/} [-k] <srcdir>"
[ -d "$src" ] || die "not a directory: $src"

# ---- the scratch directory, and its handler one line later --------------

tmp=
lock=

cleanup() {
  local st=$?                               # FIRST line, always
  if [ "$keep" -eq 1 ] && [ -n "$tmp" ]; then
    note "keeping scratch directory: $tmp"
  else
    rm -rf -- "${tmp:-}"
  fi
  [ -z "${lock:-}" ] || rm -f -- "$lock"
  return "$st"                              # leave the status alone
}
trap cleanup EXIT
trap 'cleanup; trap - INT EXIT; kill -INT $$' INT
trap 'cleanup; trap - TERM EXIT; kill -TERM $$' TERM

tmp=$(mktemp -d)
lock="$tmp/.lock"
: > "$lock"

note "scratch: $tmp"

# ---- the work ------------------------------------------------------------

cp -R -- "$src/." "$tmp/" || die "cannot copy $src"

cd -- "$tmp"

if [ -f build.sh ]; then
  if ! bash build.sh > build.log 2>&1; then
    note "build failed; last 3 lines:"
    tail -3 build.log >&2
    exit 1
  fi
  note "build succeeded"
else
  note "no build.sh in $src — nothing to do"
fi

printf 'ok\n'

$ mkdir -p proj && printf '#!/bin/bash\necho compiling\n' > proj/build.sh

$ ./build-with-cleanup.sh proj
build-with-cleanup.sh: scratch: /…/tmp.AbCdEf
build-with-cleanup.sh: build succeeded
ok
$ echo $?
0

$ printf '#!/bin/bash\necho compiling\nexit 4\n' > proj/build.sh
$ ./build-with-cleanup.sh proj
build-with-cleanup.sh: scratch: /…/tmp.GhIjKl
build-with-cleanup.sh: build failed; last 3 lines:
compiling
$ echo $?
1

$ ./build-with-cleanup.sh -k proj
build-with-cleanup.sh: scratch: /…/tmp.MnOpQr
build-with-cleanup.sh: build failed; last 3 lines:
compiling
build-with-cleanup.sh: keeping scratch directory: /…/tmp.MnOpQr
$ echo $?
1

$ ./build-with-cleanup.sh /nope
build-with-cleanup.sh: not a directory: /nope
$ echo $?
2

Six decisions:

tmp= and lock= are initialised to empty before the traps are registered. Under set -u, a handler that fires between registration and assignment would otherwise die on an unset variable — turning a tidy failure into two confusing errors. The ${tmp:-} in the handler is belt and braces for the same reason.

local st=$? is the handler's first line, and it returns rather than exits. So a build failure stays 1, a bad argument stays 2, and an interrupt stays 130 — §3's measurement applied.

Three traps, and the two signal handlers each clear both handlers before re-raising. Without trap - INT EXIT, the re-raised signal would be caught again and the EXIT handler would run a second time, which §5 measured in n1.

-k keeps the scratch directory and says where it is. A cleanup you cannot switch off is a cleanup that destroys the evidence of the failure you are trying to debug — the single most common complaint about scripts that use mktemp well.

cd -- "$tmp" happens after the copy, and nothing after it uses a relative path from the original directory. Chapter 20.1's shape one is still a hazard inside a script that cleans up correctly.

exit 1 on a build failure, with the last three log lines on stderr. The cleanup still runs, because that is what EXIT is for, and the caller gets both a usable status and enough context to decide whether to look at the full log.

Practice

1. Write the §1 script and confirm the handler runs on exit 0. Then change it to exit 7 and confirm both the handler and the status.
2. Add set -e and a false, and confirm the handler runs and reports rc was 1.
3. Write trap 'echo handler; exit 9' EXIT with exit 3. Note the script's status. Then fix it with local st=$? and confirm 3.
4. Write a handler that ends in rm -rf -- "$tmp" and have the script exit 1. What is the script's actual status, and why?
5. Register two EXIT traps and confirm only the second runs. What does that imply for two libraries that each register one?
6. Register trap 'rm -rf -- "$tmp"' EXIT before assigning tmp, under set -u. What error do you get, and where?
7. Compare trap 'rm -rf "$tmp"' EXIT with trap "rm -rf \"$tmp\"" EXIT where tmp is assigned afterwards. Which one deletes the right directory?
8. Write the three INT scripts from §5 and run all three, recording the output and exit status of each. Which one printed NOT REACHED, and which one left a directory behind?
9. In n2.sh, remove trap - INT EXIT from the INT handler. How many times does the cleanup run?
10. Add an ERR trap that prints $LINENO. Then trigger a failure inside an if condition and confirm it does not fire. Which chapter explains that?
11. Run build-with-cleanup.sh with -k on a failing build and confirm the directory survives. Then argue why -k is worth the extra branch.

Check Your Understanding

1. What does trap COMMANDS EXIT cover?
2. Why must the trap be registered immediately after creating the thing it cleans up?
3. Why single quotes rather than double?
4. A handler ends with a successful rm. What happens to the script's exit status?
5. Where must local st=$? go, and why?
6. Does an ERR trap fire for a command inside an if condition?
7. Trapping INT without re-raising: name the three things that go wrong.

7. Remember

• trap COMMANDS EXIT covers every ordinary way out — normal end, explicit exit, and a set -e abort — in one place.
• Register it on the line after you create the thing, and initialise the variable to empty first so a very early failure does not break the handler.
• Single quotes. Double quotes expand at registration time, not when the trap fires.
• A handler whose last command succeeds sets the status to 0. Capture local st=$? on the first line and return "$st".
• trap replaces rather than adds. Two registrations for one signal leave only the last — put everything in one function.
• trap - SIG removes a handler; trap -p prints what is set.
• ERR gives you a line number and a status, and has exactly set -e's exemptions — it cannot see what set -e cannot see.
• An INT handler without a re-raise is the worst of the three options: execution resumes, the cleanup runs twice, and the script exits 0.
• No handler at all gets the status right (130) and leaves the debris.
• The correct shape is clean up, clear the handlers, re-raise: trap 'cleanup; trap - INT EXIT; kill -INT $$' INT.
• Give yourself a way to keep the scratch directory. Cleanup that cannot be switched off destroys the evidence.

Chapter 20.6 is the other half of every trap in this chapter: creating that scratch directory in a way that two copies of the same script, running at the same time, cannot collide over.