Cleaning Old Runs Safely

Part II · Module 28 · Chapter 28.4 · Cleaning Old Runs Safely

Chapter 28.3 found the space and it is in old runs. Deleting them is the only irreversible thing in this entire track, and the usual implementation is one line in a crontab that nobody has read since it was written.

1. The Age Filter Has an Off-By-One

Start with the part that looks trivial. Ten directories are created at exact ages in days, using touch -t, and then find is asked for the ones older than a fortnight:

$ for d in 0 1 6 7 8 13 14 15 20 30; do
>   mkdir -p "runs/run_age${d}"; echo log > "runs/run_age${d}/run.log"
>   t=$(date -v-${d}d '+%Y%m%d%H%M')
>   touch -t "$t" "runs/run_age${d}/run.log" "runs/run_age${d}"
> done

$ find runs -maxdepth 1 -mindepth 1 -type d -mtime +14
    age 15     age 20     age 30
$ find runs -maxdepth 1 -mindepth 1 -type d -mtime 14
    age 14
$ find runs -maxdepth 1 -mindepth 1 -type d -mtime +13
    age 14     age 15     age 20     age 30

-mtime +14 does not match the 14-day-old directory. The + means strictly greater than, and find measures age in whole 24-hour periods, so +14 means "at least 15 periods old". To mean "a fortnight or more" you write +13.

The same at a week:

$ find runs -maxdepth 1 -mindepth 1 -type d -mtime +7
    age 8     age 13     age 14     age 15     age 20     age 30
$ find runs -maxdepth 1 -mindepth 1 -type d -mtime +6
    age 7     age 8     age 13     age 14     age 15     age 20     age 30

One further caution on -mtime for directories: a directory's mtime changes when an entry is added or removed inside it, not when a file deep below it is written. A run directory whose log was appended to yesterday may still have a directory mtime of three weeks ago. If the policy is about when the run last did something, test a file:

# age of the run, judged by its log rather than by its directory entry
find "$RUNS" -mindepth 2 -maxdepth 2 -name run.log -mtime +13 -print

2. The Root Can Expand to the Whole Filesystem

Now the part that is not an off-by-one. Here is the canonical cleanup line, with echo in front of it so the expansion is visible:

$ RUNS=/proj/runs
$ echo rm -rf "$RUNS"/*
rm -rf /proj/runs/*

Correct. Now the same line with the variable unset — because a typo in the name, a set -u that was never enabled, or a sourced setup script that failed earlier:

$ unset RUNS
$ echo rm -rf "$RUNS"/*
rm -rf /Applications /Library /System /Users /Volumes /bin /cores /dev /etc /home /opt /private /sbin /tmp /usr /var

Sixteen top-level directories. "$RUNS" becomes the empty string, ""/* becomes /*, and the glob expands to the root of the filesystem. An empty value does exactly the same:

$ RUNS=""
$ echo rm -rf "$RUNS"/*
rm -rf /Applications /Library /System /Users /Volumes /bin /cores /dev /etc /home /opt /private /sbin /tmp /usr /var

And RUNS=/ — which somebody writes when they mean "the root of the runs area" — produces the same set with a doubled slash:

$ RUNS=/
$ echo rm -rf "$RUNS"/*
rm -rf //Applications //Library //System //Users //Volumes //bin //cores //dev //etc //home //opt //private //sbin //tmp //usr //var

Note what the quoting did and did not do. "$RUNS" is correctly quoted — it would survive a space in the path — and quoting has no bearing on this failure at all. The problem is not splitting; it is that an empty root plus a glob is a valid, catastrophic command.

3. The Guard, and the One That Looks Like It

$ for case in unset empty set; do
>   ( case $case in (unset) unset RUNS ;; (empty) RUNS="" ;; (set) RUNS=/proj/runs ;; esac
>     : "${RUNS:?cleanup: RUNS must be set}" 2>/dev/null; printf '  RUNS %-6s rc=%d\n' "$case" "$?" )
> done
  RUNS unset  rc=1
  RUNS empty  rc=1
  RUNS set    rc=0

${RUNS:?message} fails for both unset and empty, prints the message to standard error, and returns 1. The : before it is the null command — it does nothing except force the expansion to happen, which is exactly what you want at the top of a script.

The form that looks almost identical does not guard anything:

$ unset RUNS
$ echo rm -rf "${RUNS:-}"/*
rm -rf /Applications /Library /System /Users /Volumes /bin /cores /dev /etc /home /opt /private /sbin /tmp /usr /var

:- substitutes a default — here an empty one — and the command proceeds. One character apart, and one of them is a guard while the other is a loaded gun. Chapter 8.2 has the full family; the two that matter here are :? (fail) and :- (substitute).

4. find Can Match Its Own Starting Point

The third hazard is the quietest:

$ find runs -maxdepth 1 -type d | wc -l
      11
$ find runs -maxdepth 1 -mindepth 1 -type d | wc -l
      10
$ find runs -maxdepth 1 -type d | head -1
runs

The first match is runs itself. With -delete, or piped into xargs rm -rf, that removes the root of the area along with everything in it — and if the age test happens to match the root's own mtime, it will. -mindepth 1 is what excludes the starting point, and it belongs in every find that is going to delete.

5. A Cleanup Script That Earns Its Trust

Six guards, each answering one of the failures above or in earlier chapters. It defaults to a dry run, and the destructive branch has to be asked for by name.

#!/bin/bash
# clean-runs — delete run directories older than a retention window.
# DRY RUN BY DEFAULT. Pass --delete to act.
set -u

days=14
apply=0
root=${RUNS:-}

while [ $# -gt 0 ]; do
  case "$1" in
    (--days)   days=${2:?--days needs a number}; shift 2 ;;
    (--root)   root=${2:?--root needs a path};   shift 2 ;;
    (--delete) apply=1; shift ;;
    (*) echo "clean-runs: unknown argument '$1'" >&2; exit 2 ;;
  esac
done

# GUARD 1 — the root must be set and non-empty. Not ${root:-}: that proceeds.
: "${root:?clean-runs: set RUNS or pass --root}"

# GUARD 2 — it must be an absolute path, so the result cannot depend on the cwd
case "$root" in
  (/*) : ;;
  (*) echo "clean-runs: --root must be absolute, got '$root'" >&2; exit 2 ;;
esac

# GUARD 3 — it must not be a filesystem root or a short path near one
case "$root" in
  (/|//|/*/|/[a-z]*) depth=$(printf '%s' "${root#/}" | tr -cd '/' | wc -c | tr -d ' ')
     if [ "$depth" -lt 2 ]; then
       echo "clean-runs: refusing to operate on '$root' -- too close to the root" >&2
       exit 2
     fi ;;
esac

# GUARD 4 — it must exist, be a directory, and carry a marker file. The marker
# is the strongest guard here: it cannot be satisfied by accident.
[ -d "$root" ] || { echo "clean-runs: not a directory: $root" >&2; exit 2; }
[ -f "$root/.runs-area" ] || {
  echo "clean-runs: $root has no .runs-area marker -- refusing" >&2
  echo "  if this really is the runs area:  touch '$root/.runs-area'" >&2
  exit 2; }

# GUARD 5 — the age expression. +$((days-1)) so "14" means "14 days or older".
age=$((days - 1))
[ "$age" -ge 0 ] || { echo "clean-runs: --days must be at least 1" >&2; exit 2; }

# GUARD 6 — never touch a run that is still being written to
still_running() {
  # lsof +D exits 1 whether or not it found anything, so the COUNT decides.
  # And if lsof is unavailable we cannot know -- so report busy, not idle.
  command -v lsof >/dev/null 2>&1 || return 0
  [ "$(lsof +D "$1" 2>/dev/null | grep -c .)" -gt 1 ]
}

n=0; skipped=0
while IFS= read -r d; do
  if still_running "$d"; then
    printf 'SKIP   %s  (a process still has files open in it)\n' "$d"
    skipped=$((skipped + 1)); continue
  fi
  n=$((n + 1))
  if [ "$apply" = 1 ]; then
    printf 'DELETE %s\n' "$d"
    rm -rf -- "$d"
  else
    printf 'WOULD  %s\n' "$d"
  fi
done < <(find "$root" -mindepth 1 -maxdepth 1 -type d -mtime +"$age" -print | sort)

if [ "$apply" = 1 ]; then verb=deleted; else verb='would delete'; fi
printf '%s %d directories older than %d days under %s\n' "$verb" "$n" "$days" "$root"
if [ "$skipped" -gt 0 ]; then
  printf '%d skipped because a process still had files open in them\n' "$skipped"
fi
[ "$apply" = 1 ] || printf 'dry run -- pass --delete to act\n'

Running it against the aged fixture — ten directories at exact ages — dry by default:

$ RUNS=…/m28/runs clean-runs --days 14
clean-runs: …/m28/runs has no .runs-area marker -- refusing
  if this really is the runs area:  touch '…/m28/runs/.runs-area'
$ echo $?
2

$ touch …/m28/runs/.runs-area
$ RUNS=…/m28/runs clean-runs --days 14
WOULD  …/runs/run_age14
WOULD  …/runs/run_age15
WOULD  …/runs/run_age20
WOULD  …/runs/run_age30
would delete 4 directories older than 14 days under …/runs
dry run -- pass --delete to act

Four directories, including the 14-day-old one — which is §1's off-by-one fixed, visible in the output. A -mtime +14 would have listed three.

The other guards, each refusing:

$ ( unset RUNS; clean-runs --days 14 )
clean-runs: line 20: root: clean-runs: set RUNS or pass --root
$ clean-runs --root runs --days 14
clean-runs: --root must be absolute, got 'runs'
$ clean-runs --root /tmp --days 14
clean-runs: refusing to operate on '/tmp' -- too close to the root
$ clean-runs --root /usr/local --days 14
clean-runs: refusing to operate on '/usr/local' -- too close to the root

And the destructive branch, on throwaway copies:

$ ls del
fresh  old_20  old_30
$ RUNS=…/m28/del clean-runs --days 14 --delete
DELETE …/del/old_20
DELETE …/del/old_30
deleted 2 directories older than 14 days under …/del
$ ls del
fresh

Three of the four guards are refusals rather than corrections, and that is deliberate. A cleanup script that fixes a bad root is a script that proceeds on a guess.

6. The Two Guards That Are About Time, Not Paths

Guard 6 is the one most cleanup scripts lack, and Chapter 28.3 measured why it matters: deleting a directory while a job still has files open in it does not free the space, it just removes the name — df stays full, du reports the area as clean, and the job keeps writing into a file nobody can find.

The obvious implementation is wrong, and I shipped it that way first:

# WRONG -- this never skips anything
still_running() {
  command -v lsof >/dev/null 2>&1 || return 1
  lsof +D "$1" >/dev/null 2>&1
}

$ ( exec 9< runs/run_age30/run.log; sleep 20 ) &
$ RUNS=…/runs clean-runs --days 14
WOULD  …/runs/run_age14
WOULD  …/runs/run_age15
WOULD  …/runs/run_age20
WOULD  …/runs/run_age30        # <- held open, and not skipped

lsof +D descends the whole directory rather than listing just its entries, which is slower than +d and is the right choice because a simulator's open dump may be several levels down.

And a second time-shaped hazard worth naming because it has no code fix: the retention window and the archive step have to be ordered. If cleanup runs before Chapter 28.5's archiving, the failures you wanted to keep are gone. The ordering lives in whatever schedules them, not in either script, so it is exactly the kind of thing that is correct on the day it is set up and silently wrong a year later when somebody changes a cron time.

Practice

1. Create directories at exact ages with touch -t and find which ones -mtime +14 matches. Confirm the 14-day one is absent.
2. Work out the expression that means "fourteen days or older" and verify it.
3. Test a directory's mtime against a file's inside it, after appending to the file. Note which one changed.
4. Print — do not run — rm -rf "$RUNS"/* with RUNS unset, and count the arguments.
5. Do the same with RUNS="" and RUNS=/ and compare the expansions.
6. Confirm that correct quoting does not help, and explain which failure quoting does prevent.
7. Measure ${RUNS:?}'s exit status for unset, empty and set.
8. Print the expansion with ${RUNS:-} instead and note that the command proceeds.
9. Count find's matches with and without -mindepth 1, and identify the extra one.
10. Write clean-runs with all six guards and confirm it refuses a missing marker with status 2.
11. Point it at an absolute but wrong directory that has no marker, and confirm guards 1 to 3 pass while guard 4 saves you.
12. Start a job holding a file open in an old run directory, then run the cleanup and confirm that directory is skipped — and that it is not skipped once the holder exits.
13. Compare lsof +D's exit status against its line count on a busy and an idle directory, then decide which your guard should test.
14. Remove lsof from PATH and check whether your guard now claims the directory is idle.

Check Your Understanding

1. What does the + in -mtime +14 mean, and which directory does it therefore skip?
2. Write the expression that means "fourteen days old or more".
3. Why can a directory's mtime be much older than the files inside it?
4. With RUNS unset, what does rm -rf "$RUNS"/* expand to?
5. Does correct quoting prevent that? What does quoting prevent?
6. What is the difference between ${RUNS:?} and ${RUNS:-}?
7. Why is there a : before "${root:?…}"?
8. What extra directory does find runs -maxdepth 1 -type d match?
9. Why is a marker file a stronger guard than any test on the path string?
10. Why does the marker live with the data rather than in the script?
11. Why must deleting a run that is still being written to be avoided, in terms of what it actually achieves?
12. lsof +D printed 3 lines on a busy directory and 0 on an idle one, and exited 1 for both. Which signal must the guard use, and what must it do when lsof is missing?
13. What ordering hazard has no fix inside either script?

7. Remember

• Print the expansion before running it. Nothing in this chapter deleted anything, and all three hazards were still fully visible.
• -mtime +14 skips the 14-day-old run. + is strictly greater in whole days; write +$((days - 1)) and keep the off-by-one in one visible place.
• The registry's own hint contains that off-by-one, which is a lesson about auditing rules rather than about find.
• A directory's mtime is not its contents' mtime. Test a file if the policy is about activity.
• An unset or empty root makes rm -rf "$RUNS"/* expand to sixteen top-level directories. Quoting is irrelevant to this failure.
• ${RUNS:?} fails for unset and empty; ${RUNS:-} proceeds. One character apart.
• find matches its own starting point without -mindepth 1, so a -delete removes the area.
• Require a marker file in the data. It is the only guard that cannot be satisfied by accident.
• Refuse rather than correct. A script that fixes a bad root is a script that guesses.
• Never delete a run that is still open — it frees nothing and hides the space.
• lsof +D exits 1 whether or not it finds anything. Test the line count; my first guard branched on the status and could never fire.
• A guard that cannot run must report busy. My fallback returned "idle", the opposite of what the prose beside it claimed.
• Prove a guard fires. A safety check never observed to trigger is indistinguishable from one that cannot.
• Archive before you clean, and know that the ordering lives in the scheduler where nothing checks it.

8. What Is Next

The cleanup is safe and it is about to delete a directory containing a failure somebody still needs. Chapter 28.5 runs before it and keeps exactly enough — measured at 4 KB out of 60 MB, verified by deleting the original and reproducing the failure from the archive.