When Expansion Produces Nothing, or Too Much

Part I · Module 9 · Chapter 9.6 · When Expansion Produces Nothing, or Too Much

The pipeline has been described as though every stage produces something sensible. The last chapter of the module is about the edges — and they are where the expensive mistakes live.

1. Nothing, From a Variable

Chapter 9.2 showed that an unquoted empty variable produces no argument. Here is the version that has destroyed real machines.

$ unset BUILDDIR

$ echo "would run: rm -rf ${BUILDDIR}/*"
would run: rm -rf /*

Two stages, both working exactly as specified. Stage three replaced ${BUILDDIR} with nothing, leaving /*. Stage five then matched that against every entry in the root directory.

The command that was written to clean one build directory is now a command to delete the system. Quoting does not help here — "${BUILDDIR}"/* is still ""/* which is still /*. The value is the problem, not the splitting.

2. Nothing, From a Glob

The second case is quieter, and almost everybody meets it in a loop.

$ cd empty-dir

$ echo *.log
*.log

$ ls *.log
ls: *.log: No such file or directory

A pattern that matches nothing is passed through unchanged. Bash does not report it, does not substitute nothing, and does not warn. The command receives the pattern itself, as a literal file name — and ls then reports that no file called *.log exists, which is true and completely misleading.

The damaging version:

$ n=0
$ for f in *.log; do n=$((n+1)); echo "  saw [$f]"; done
  saw [*.log]
$ echo "the loop ran $n time(s)"
the loop ran 1 time(s)

One iteration, over a file that does not exist. A loop written to process logs just processed a filename made of a star. Whatever is in the body — copying, deleting, parsing — ran once on garbage.

Three ways to handle it

nullglob — an unmatched pattern disappears:

$ shopt -s nullglob
$ echo "[$(echo *.log)]"
[]
$ n=0; for f in *.log; do n=$((n+1)); done; echo "loop ran $n times"
loop ran 0 times

This is almost always the behaviour you want for a loop. Set it once at the top of a script.

failglob — an unmatched pattern is an error:

$ shopt -s failglob
$ echo *.log
bash: no match: *.log

Right when a missing file means the script's assumptions are wrong and continuing is worse than stopping.

A guard, with no shell option at all:

for f in *.log; do
  [ -e "$f" ] || continue
  ...
done

Portable, local, and visible at the point of use. [ -e "$f" ] fails for the literal pattern because no such file exists. This is the version to use in a script you are not sure you control the options of.

3. Something the Command Reads as Syntax

A variant of the same shape: the expansion produced a real file name, and the command misreads it.

$ ls
-notes.txt   -rf

$ ls -notes.txt
ls: invalid option -- .
usage: ls [-@ABCFGHILOPRSTUWXabcdefghiklmnopqrstuvwxy1%,] [--color=when] ...

A file whose name begins with a hyphen arrives looking like an option. With rm * in a directory containing a file called -rf, the consequences are obvious and bad.

Two fixes, both worth knowing:

$ ls -- -notes.txt
-notes.txt

$ ls ./-notes.txt
-notes.txt

• -- means "no more options" — everything after it is an operand, whatever it starts with. Nearly every GNU and BSD tool honours it.
• ./ in front makes the name start with a dot, so it cannot be read as an option. This works even with the few tools that do not understand --.

The habit worth forming is -- before any expansion that reaches a destructive command:

rm -- "$file"
rm -- "$dir"/*.log

It costs three characters and removes an entire category of accident — including the malicious kind, where a file name is chosen by somebody else.

4. Too Much

The third case is the only one that announces itself.

$ getconf ARG_MAX
1048576

There is a maximum total size for a command's arguments and environment. A little over a megabyte here; the exact figure varies by system. A glob matching a few thousand files is nowhere near it. A glob matching a few hundred thousand can be.

$ /bin/echo $(seq 1 2000000) > /dev/null
bash: /bin/echo: Argument list too long

The two answers

find with -exec, which never builds a giant list:

find . -maxdepth 1 -name '*.log' -delete
find . -name '*.tmp' -exec rm -- {} +

The + at the end batches as many names as fit per invocation, running rm a handful of times instead of once per file. (\; runs it once per file — correct, much slower.)

xargs, which splits the list for you:

find . -name '*.log' -print0 | xargs -0 rm --

-print0 and -0 separate names with a null byte instead of whitespace, which is the only separator that cannot appear in a file name — so this handles spaces, newlines and quotes correctly. Use the null-separated pair every time; plain find | xargs breaks on exactly the file names everything else in this module has been about.

$ printf 'f1\nf2\nf3\nf4\nf5\n' | xargs -n 2 echo "pair:"
pair: f1 f2
pair: f3 f4
pair: f5

-n 2 shows the batching directly: xargs ran the command three times rather than once.

(Module 14 covers find and xargs properly. The part that belongs here is why you reach for them.)

5. The Three Together

A cleanup script, written with every guard from this chapter:

#!/bin/bash
set -euo pipefail
shopt -s nullglob

# Case 1: a required value, named in the error if it is missing.
logdir="${LOG_DIR:?set LOG_DIR to the directory to clean}"

# ...and checked, because set -u does not catch an empty string.
[ -d "$logdir" ] || { echo "not a directory: $logdir" >&2; exit 1; }

# Case 2: nullglob means this loop runs zero times when there is nothing to do.
removed=0
for f in "$logdir"/*.log; do
  rm -- "$f"
  removed=$(( removed + 1 ))
done

# Case 3: the tmp tree may hold a hundred thousand files, so never build one list.
find "$logdir" -name '*.tmp' -print0 | xargs -0 --no-run-if-empty rm --

echo "removed $removed log file(s)"

Six defences in twenty lines:

• :? names the missing variable instead of expanding to nothing.
• [ -d ] catches the set-but-empty and set-but-wrong cases that set -u does not.
• nullglob makes the loop run zero times rather than once over a literal *.log.
• "$logdir"/*.log quotes the variable part and leaves the pattern bare — the variable must not be split, the pattern must still match. That split is worth noticing: it is Chapter 9.1's two stages, used deliberately.
• -- on every rm, so a file called -rf is a file.
• -print0 | xargs -0 for the unbounded case, with null separation so awkward names survive.

--no-run-if-empty is a GNU xargs option that stops it running the command at all when there is no input; BSD xargs does this by default. Without it on GNU, an empty list runs rm -- once with no arguments, which is harmless but reports an error.

Practice

mkdir -p /tmp/ee/empty && cd /tmp/ee

1 — Watch a path collapse, safely.

unset BUILDDIR
echo "would run: rm -rf ${BUILDDIR}/*"
( set -u; echo "rm -rf ${BUILDDIR}/*" )
( echo "rm -rf ${BUILDDIR:?set BUILDDIR first}/*" )

Every one uses echo. Never test this one for real.

2 — Watch an unmatched glob be handed over.

cd /tmp/ee/empty
echo *.log
ls *.log
n=0; for f in *.log; do n=$((n+1)); echo "  saw [$f]"; done; echo "ran $n time(s)"

3 — Fix it three ways.

( shopt -s nullglob; n=0; for f in *.log; do n=$((n+1)); done; echo "nullglob: $n" )
( shopt -s failglob; echo *.log )
n=0; for f in *.log; do [ -e "$f" ] || continue; n=$((n+1)); done; echo "guarded: $n"

4 — Meet the hyphen file.

touch ./-notes.txt
ls -notes.txt
ls -- -notes.txt
ls ./-notes.txt
rm -- ./-notes.txt

5 — Find your limit.

getconf ARG_MAX
echo $(seq 1 2000000) > /dev/null
/bin/echo $(seq 1 2000000) > /dev/null

The builtin succeeds and the external command does not.

6 — Batch with xargs.

printf 'f%s\n' 1 2 3 4 5 | xargs -n 2 echo "pair:"

7 — Use the null-safe pair.

cd /tmp/ee && touch "a.tmp" "two words.tmp"
find . -name '*.tmp' -print0 | xargs -0 -n 1 echo "deleting:"
find . -name '*.tmp' -print0 | xargs -0 rm --

8 — Clean up. cd /tmp && rm -rf /tmp/ee

Check Your Understanding

1. rm -rf "$BUILDDIR"/* with BUILDDIR unset. Does quoting save you, and what does?

2. A loop over *.log in a directory with no logs runs once. What was $f, and what are two fixes?

3. What is the difference between nullglob and failglob, and when would you choose each?

4. Why does echo * work in a directory where rm * fails with "argument list too long"?

5. A directory contains a file called -rf. Why is rm * dangerous there, and what prevents it?

6. Why is find … -print0 | xargs -0 better than find … | xargs?

6. Remember

• Three failure modes: nothing from a variable, nothing from a glob, too much from a glob. Only the third announces itself.
• An unset variable in a path collapses it — rm -rf ${X}/* becomes rm -rf /*, and quoting does not help.
• Guard with set -u, with ${VAR:?message}, and with an explicit check. set -u catches unset but not empty; :? catches both and names the variable.
• Put ${var:?} on the dangerous command itself, where an editor will see it.
• An unmatched glob is passed through literally, so a loop runs once over a file that does not exist.
• shopt -s nullglob makes it vanish; shopt -s failglob makes it an error; [ -e "$f" ] || continue needs no option at all.
• shopt changes the whole shell. Set it at the top of a script; use a subshell to experiment.
• A file name starting with a hyphen is read as an option. -- ends option parsing and ./ makes the name safe.
• ARG_MAX is a real, finite limit — about a megabyte — imposed when an external program is executed.
• A builtin has no such limit, which is why echo * can succeed where rm * fails.
• find -exec … + and xargs batch the work instead of building one enormous line.
• Always -print0 | xargs -0. Null is the only separator a file name cannot contain.
• Quote the variable, leave the pattern bare: "$dir"/*.log. Two stages, used on purpose.
• In a loop, an unmatched pattern runs the body once on the pattern itself. Chapter 17.2 measures it, and Chapter 17.6 is the [ -e "$f" ] || continue guard that handles it where nullglob cannot.

That completes Module 9, and with it the expansion arc of Part I. You can now read any command line the way Bash reads it: six stages in a fixed order, each seeing only what the one before produced, with quoting as the switch that turns individual stages off. The advice to put quotes around an expansion is no longer a rule to remember — it is a consequence you could derive.

Module 10 turns to the question of where a setting actually lives: the environment a program inherits, how PATH decides which copy of a command you are really running, which startup file runs in which kind of shell, and the sourcing rule that every environment-setup script you will ever be handed depends on.