Idempotence and Dry Runs

Part I · Module 21 · Chapter 21.5 · Idempotence and Dry Runs

A regression wrapper got halfway through and died on a licence timeout. Do you run it again?

If you cannot answer that from reading the script, the honest thing is to go and clean up by hand first — which is what people actually do, and it is why a script that is merely correct still does not get used.

1. Perform an Action, or Reach a State

Idempotent means running it twice has the same effect as running it once. The useful reformulation is a change of grammar: stop writing do this and start writing make sure this is so.

Every row on the right can be repeated. Every row on the left cannot, and the measurements are one option apart.

2. The Measurements

Each pair, run twice:

$ mkdir d
$ mkdir d
mkdir: d: File exists
$ echo "rc=$?"
rc=1

$ rm -rf d
$ mkdir -p d && mkdir -p d && echo "rc=0 — idempotent"
rc=0 — idempotent

$ rm nope.txt
rm: nope.txt: No such file or directory
$ echo "rc=$?"
rc=1

$ rm -f nope.txt; echo "rc=$?"
rc=0

$ touch target
$ ln -s target link
$ ln -s target link
ln: link: File exists
$ echo "rc=$?"
rc=1

$ rm -f link
$ ln -s target link && ln -sfn target link && echo "rc=0 — idempotent"
rc=0 — idempotent

$ rm -f out.txt; for i in 1 2; do echo "line" >> out.txt; done
$ wc -l < out.txt
       2
$ rm -f out.txt; for i in 1 2; do echo "line" > out.txt; done
$ wc -l < out.txt
       1

Four pairs. -p, -f, -sfn, and > instead of >> — that is the entire difference between a script you can retry and one you cannot.

A summary worth keeping:

3. Where Idempotence Comes From

Three patterns cover almost everything.

Use the option that means "already fine is fine". -p, -f, -sfn. This is the first thing to reach for because it is one character and the command does the checking atomically — Chapter 20.6's point about mkdir being a lock.

Write the whole file, do not append to it. Appending is the single most common source of non-idempotence, and it is invisible until the second run:

# not idempotent: a filelist that doubles
for f in ./*.sv; do echo "$f" >> files.f; done

# idempotent: build it, then write it once
declare -a srcs=(./*.sv)
printf '%s\n' "${srcs[@]}" > files.f

Chapter 19.4's array, used to make the write a single statement. The second version also fails cleanly if the directory is unreadable, rather than leaving a half-written list.

And make the step conditional on the state, not on a marker you maintain. The tempting version is a flag file:

# fragile: the marker and the reality can disagree
[ -f .setup-done ] || { do_setup; touch .setup-done; }

That breaks the moment anything removes the setup without removing the marker — which includes a partially-failed previous run. Test the actual condition instead:

[ -d "$outdir/db" ] || create_db "$outdir/db"

The marker is a cache of a fact you can just look up. Chapter 20.1's shape four in a new costume: the script believed a premise instead of checking it.

4. The Dry Run

The second property, and it is the one that gets a tool its first use.

if [ "$dry_run" -eq 1 ]; then
  printf 'WOULD RUN (%d args):\n' "${#cmd[@]}"
  printf '  [%s]\n' "${cmd[@]}"
  exit 0
fi
"${cmd[@]}"

$ ./dr.sh -n
WOULD RUN (5 args):
  [rm]
  [-rf]
  [--]
  [run dir/a.log]
  [run dir/b.log]
$ echo $?
0

Four rules make the difference between a dry run that is useful and one that is decoration.

Print the array, one element per line. Chapter 19.5 §5's rule: run dir/a.log is one argument, and a joined string would show rm -rf -- run dir/a.log run dir/b.log, which hides exactly the boundary error you are previewing to catch.

Build the plan once and use it for both paths. A dry run that assembles the command differently from the real run can lie — Chapter 20.4 §6's point. One array, two consumers.

Exit 0 and do nothing else. A dry run that creates the output directory "because it needs to anyway" is not a dry run. The test is: after -n, is the filesystem byte-identical?

And make it reachable before the expensive checks. A dry run should work on a system that could not actually do the work — that is often why you are running it.

5. A Complete Script

A results-directory cleaner: idempotent, with a dry run, and a lock for the part that is not:

#!/usr/bin/env bash
# rotate-results — keep the newest N result directories, remove the rest
#
# usage: rotate-results [-n] [-k KEEP] <resultsdir>
#        -n        dry run: print what would happen, change nothing
#        -k KEEP   how many to keep (default 3)
#
# Safe to re-run: every step is idempotent. A second run with the same
# arguments prints "nothing to do" and exits 0.
#
# exit: 0 ok (including nothing to do) · 1 a removal failed
#       2 bad invocation · 3 another run is in progress

set -o errexit
set -o nounset
set -o pipefail
shopt -s nullglob

readonly EX_OK=0
readonly EX_FAIL=1
readonly EX_USAGE=2
readonly EX_LOCKED=3

note()  { printf '%s: %s\n' "${0##*/}" "$*" >&2; }
usage() { printf 'usage: %s [-n] [-k KEEP] <resultsdir>\n' "${0##*/}" >&2; }
die()   { note "$*"; usage; exit "$EX_USAGE"; }

dry_run=0
keep=3

while getopts ':nk:' opt; do
  case $opt in
    n)  dry_run=1 ;;
    k)  keep=$OPTARG ;;
    \?) die "unknown option: -$OPTARG" ;;
    :)  die "option -$OPTARG needs a value" ;;
  esac
done
shift $(( OPTIND - 1 ))

[ $# -eq 1 ] || { usage; exit "$EX_USAGE"; }
root=$1

[ "$keep" -ge 1 ] 2>/dev/null || die "keep must be a positive integer, got: $keep"
[ -d "$root" ]                || die "not a directory: $root"
root=$(cd -- "$root" && pwd)  || die "cannot enter: $1"

# ---- work out the plan, once --------------------------------------------

# a glob is already sorted ([Chapter 3.5](/linux-bash/wildcards-and-filename-expansion)) and forks nothing — no find,
# no `sort -z`, and nothing to misparse
declare -a all=("$root"/run_*)

declare -a doomed=()
if [ "${#all[@]}" -gt "$keep" ]; then
  doomed=("${all[@]:0:$(( ${#all[@]} - keep ))}")
fi

printf 'found   %d run directory(ies)\n' "${#all[@]}"
printf 'keeping %d\n' "$keep"

if [ "${#doomed[@]}" -eq 0 ]; then
  printf 'nothing to do\n'
  exit "$EX_OK"
fi

# ---- the dry run exits before anything is created or removed ------------

if [ "$dry_run" -eq 1 ]; then
  printf 'WOULD REMOVE %d:\n' "${#doomed[@]}"
  printf '  %s\n' "${doomed[@]}"
  exit "$EX_OK"
fi

# ---- the lock: removal is repeatable, but two runs at once are not ------

lock="$root/.rotating"
if ! mkdir "$lock" 2>/dev/null; then
  note "another run is in progress ($lock)"
  exit "$EX_LOCKED"
fi
trap 'rmdir -- "$lock" 2>/dev/null || true' EXIT

failed=0
for d in "${doomed[@]}"; do
  # rm -rf is idempotent: a second run finds nothing and returns 0
  if rm -rf -- "${d:?}"; then
    printf 'removed %s\n' "$d"
  else
    note "could not remove $d"
    failed=$(( failed + 1 ))
  fi
done

printf -- '-- %d removed, %d failed\n' "$(( ${#doomed[@]} - failed ))" "$failed"
[ "$failed" -eq 0 ] || exit "$EX_FAIL"
exit "$EX_OK"

$ mkdir -p res
$ for i in 1 2 3 4 5; do mkdir -p "res/run_$i"; done

$ ./rotate-results -n res
found   5 run directory(ies)
keeping 3
WOULD REMOVE 2:
  /…/res/run_1
  /…/res/run_2
$ ls res | tr '\n' ' '
run_1 run_2 run_3 run_4 run_5

The dry run changed nothing — that listing is the test. Now for real, twice:

$ ./rotate-results res
found   5 run directory(ies)
keeping 3
removed /…/res/run_1
removed /…/res/run_2
-- 2 removed, 0 failed
$ echo $?
0

$ ./rotate-results res
found   3 run directory(ies)
keeping 3
nothing to do
$ echo $?
0

The second run is a no-op that exits 0, which is the whole property. A caller can put this in a cron job, run it twice by accident, or retry it after an interruption, and none of those does anything surprising.

$ ./rotate-results -k 1 res
found   3 run directory(ies)
keeping 1
removed /…/res/run_3
removed /…/res/run_4
-- 2 removed, 0 failed

$ for i in 6 7 8 9; do mkdir -p "res/run_$i"; done
$ mkdir res/.rotating
$ ./rotate-results res
found   5 run directory(ies)
keeping 3
rotate-results: another run is in progress (/…/res/.rotating)
$ echo $?
3

$ ./rotate-results -k 9 res
found   5 run directory(ies)
keeping 9
nothing to do
$ rmdir res/.rotating

Note the second of those: the lock was still held, and the script did not care — because the plan was empty and it exited before taking it. The lock guards the removal, not the inspection, so asking what would happen never blocks and two people can both do it at once.

And the dry run really is dry, which §4 named as the test:

$ find res | sort > before
$ ./rotate-results -n res > /dev/null
$ find res | sort > after
$ diff before after && echo "byte-identical"
byte-identical

$ ./rotate-results
usage: rotate-results [-n] [-k KEEP] <resultsdir>
$ echo $?
2

$ ./rotate-results -k 0 res
rotate-results: keep must be a positive integer, got: 0
usage: rotate-results [-n] [-k KEEP] <resultsdir>
$ echo $?
2

Six decisions:

"Safe to re-run" is in the header comment, in those words, with the one exception named. That sentence is what a colleague is looking for, and it is cheaper to write than to make them read the body.

The plan is one array, used by both the dry run and the real run. doomed is computed once; -n prints it and the real path removes it. There is no second code path to drift.

-n exits before the lock and before any mkdir. The filesystem after a dry run is byte-identical, which §4 named as the test.

rm -rf -- "${d:?}" is idempotent and guarded — Chapter 20.4's :? on the dangerous line, even though d came from an array that cannot contain an empty element. The guard costs nothing and survives a refactor.

The lock is mkdir, not touch. Atomic, so two simultaneous runs cannot both take it — Chapter 20.6's argument, and the trap releases it on every path.

And "nothing to do" is exit 0, not 1. Chapter 17.1 §5's question, answered: a rotation with nothing to rotate has succeeded. A caller that wants to know whether anything happened reads the output, which says so.

Practice

1. Run each of the four pairs in §2 twice and record both statuses. Then write the one-character fix for each from memory.
2. Measure cp -n on an existing destination. Note the status and whether the file changed. Then explain why that combination is dangerous under set -e.
3. Write a loop that appends to a filelist and run it twice. Count the lines. Then rewrite it with an array and a single >.
4. Write the [ -f .setup-done ] || marker version and the [ -d "$outdir/db" ] || state version. Delete the thing the marker claims exists and run both. Which one recovers?
5. Add a -n flag to a script of your own that prints "${cmd[@]}" one element per line. Then compare the output to a joined echo "${cmd[*]}" on a path containing a space.
6. Verify a dry run is really dry: snapshot the directory with find . | sort > before, run with -n, snapshot again, and diff them.
7. Take a script whose dry run builds its plan separately from the real path. Find an input where the two disagree.
8. Implement the mkdir lock with a trap release. Then implement it with touch and [ -f ] and describe the race you just created.
9. Run rotate-results twice and confirm the second is a no-op with status 0. Then run it with -k 0 and with no arguments, and check both statuses against the header comment.
10. Add a non-idempotent step — appending a line to a log outside the results tree — to rotate-results. Then decide: make it idempotent, or document it. Write the sentence you would add to the header.

Check Your Understanding

1. What does idempotent mean, and what is the grammatical reformulation that makes it easier to write?
2. Give the idempotent form of mkdir, rm, ln -s and >>.
3. cp -n leaves the destination alone on a second run. Why is it still a problem?
4. Why is appending the most common source of non-idempotence?
5. What is wrong with a .setup-done marker file?
6. Name the four rules for a useful dry run.
7. Why is the lock in rotate-results taken after the dry-run exit rather than before?

6. Remember

• Write steps that reach a state, not steps that perform an action. "Make sure this exists" is repeatable; "create this" is not.
• mkdir -p, rm -f, ln -sfn, and > instead of >> — four options, measured, each turning a second-run failure into a no-op.
• An idempotent step must be idempotent in its status too. cp -n returns 1 on the second run and will abort a set -e script.
• Appending is the commonest non-idempotence, and it is silent. Build the content in an array and write it once.
• Test the real condition, not a marker file. A marker is a cache that can disagree with reality.
• Some things cannot be idempotent. Take an atomic mkdir lock, or document which half is safe to re-run.
• A dry run prints the array, one element per line — a joined string hides the boundary error you are checking for.
• Build the plan once and use it for both paths, or the preview can lie.
• A dry run must leave the filesystem byte-identical. find . | sort before and after is the test.
• Make -n reachable before the expensive checks, because you often run it on a machine that could not do the work.
• Put "safe to re-run" in the header comment, in those words, and name the exception.

Chapter 21.6 is the other half of "it works on my machine": which of the constructs this course has taught are Bash-only, and which option spellings differ between the system you wrote on and the one your script will meet.