Looping Over Files Safely

Part I · Module 17 · Chapter 17.2 · Looping Over Files Safely

Chapter 17.1 ended with a loop over *.log, which is the most common loop in shell scripting. It is also where most scripts are quietly wrong, and the reason is always the same: a file name is not a word.

1. Two Files, Three Iterations

Two files. One of them has a space in its name, which is completely legal and which you did not choose — it arrived in a tarball, or a GUI made it:

$ ls -1
clean.log
my report.log

Now loop over them the way people write it:

$ for f in $(ls); do echo "[$f]"; done
[clean.log]
[my]
[report.log]

Three iterations for two files. And the second and third are not files at all:

$ ls -l my
ls: my: No such file or directory

Nothing exotic happened. $(ls) is command substitution (Chapter 8.4), which produces one block of text. Unquoted, that text goes through word-splitting (Chapter 9.2), which splits on spaces because that is what word-splitting does. Bash never saw "two file names"; it saw characters.

A pattern does not have this problem:

$ for f in *.log; do echo "[$f]"; done
[clean.log]
[my report.log]

Two iterations, the space intact. The difference is where the words come from. Globbing is done by the shell, which knows where each name ends, and it produces separate words directly — they are never a block of text, so they are never re-split.

2. The File That Disappears

The space problem is well known. This one is not, and it is about the loop's body rather than its list.

Four entries in a directory, one of them named -n:

$ ls -1
-n
clean.log
my report.log
notes.txt

A correct glob, and a loop body that looks unimpeachable:

$ for f in *; do echo "$f"; done
clean.log
my report.log
notes.txt

Three lines for four files. -n is gone.

The glob was right — it produced four words. What happened is in the body: echo "$f" became echo "-n", and -n is echo's suppress-the-newline flag. echo did not print a line containing -n; it printed nothing and suppressed the newline, so the file vanished from the output without a trace.

$ for f in *; do printf '%s\n' "$f"; done
-n
clean.log
my report.log
notes.txt

printf has no such flag — its first argument is a format string, not an option — so all four appear. This is why printf '%s\n' is preferred over echo for anything whose value you did not choose, a rule Chapter 4.1 introduced and that this is the sharpest case of.

Now make it dangerous. Replace echo with a command that has destructive flags:

$ ls -1
-rf
keep.txt

$ for f in *; do printf 'ARG: %s\n' "$f"; done
ARG: -rf
ARG: keep.txt

A loop body of rm "$f" would call rm -rf — with the keep.txt of the next iteration arriving as its operand. The quotes around "$f" are correct and do not help: quoting controls word-splitting, not whether the receiving program interprets a leading dash as an option.

The fix is to make the name not start with a dash:

$ for f in ./*; do printf '%s\n' "$f"; done
./-rf
./keep.txt

./* is the same glob with two characters prepended to every result. ./-rf begins with a dot, so no program can read it as an option, and it names exactly the same file. This is the habit: write ./* rather than * in any loop whose body passes the name to a command. It is the same defence Chapter 14.7 used for find, and Chapter 1.5's -- convention is the other half of it:

for f in ./*; do rm -- "$f"; done        # belt and braces

3. The Name With a Newline In It

The worst case, and the one that separates "mostly works" from "correct". A file name may contain any byte except / and NUL — including a newline:

$ ls -1
ok.log
two
lines.log

Three display lines, two files. The second is named two⏎lines.log.

A glob gets it right, because globbing never serialises anything:

$ for f in *.log; do printf '[%s]\n' "$f"; done
[ok.log]
[two
lines.log]

Two iterations. The second value contains a real newline, which is why it prints across two lines — but it is one word and one file.

Anything that goes through a text stream does not:

$ ls *.log | while IFS= read -r f; do printf '[%s]\n' "$f"; done
[ok.log]
[two]
[lines.log]

Three iterations, two of them on files that do not exist. And note that this loop did everything "right" — IFS=, -r, quoted expansion. It still fails, because the damage was done by ls before read ever saw the data. A newline-delimited stream cannot represent a name containing a newline. No amount of careful reading fixes an ambiguous format.

4. When You Need find

A glob handles one directory. For a tree you need find (Chapter 14.6) — and the same ambiguity arrives with it:

$ find . -name '*.log'
./nl
name.log
./sp ace.log
./sub/deep.log

Three files, four lines. Read that with a loop and you get four items:

$ find . -name '*.log' | while IFS= read -r f; do printf '[%s]\n' "$f"; done
[./nl]
[name.log]
[./sp ace.log]
[./sub/deep.log]

The answer is to change the delimiter to the one byte a file name cannot contain:

$ find . -name '*.log' -print0 | while IFS= read -r -d '' f; do printf '[%s]\n' "$f"; done
[./nl
name.log]
[./sp ace.log]
[./sub/deep.log]

Three items. -print0 terminates each name with a NUL instead of a newline, and read -d '' reads up to a NUL instead of a newline. The format is now unambiguous, so the loop is correct for every possible name.

Two notes on that form, both of which matter:

-d '' means NUL. An empty delimiter string is how you say "the zero byte" in Bash, because you cannot put a literal NUL in an argument. It looks like a typo and is not.

A pipe into a loop loses the loop's variables. Count inside that loop and the count is zero afterwards — Chapter 17.7 is entirely about why. The fix is process substitution:

n=0
while IFS= read -r -d '' f; do
  n=$(( n + 1 ))
done < <(find . -name '*.log' -print0)
echo "$n"          # 3, not 0

5. The Unmatched Pattern, Again

Chapter 17.1 raised it; here is what to do about it. A glob that matches nothing expands to itself, so the body runs once on a file that does not exist:

$ for f in ./*.nope; do printf '[%s]\n' "$f"; done
[./*.nope]

Two fixes, and they are for different situations.

shopt -s nullglob makes an unmatched pattern vanish, so the list is empty and the body never runs:

$ shopt -s nullglob
$ for f in ./*.nope; do printf '[%s]\n' "$f"; done
$

Set it once near the top of a script. The caveat is that it is global for the rest of the script, and it applies to every glob — including ones where you wanted the literal, and including arguments to commands. In a short, self-contained script that is fine and it is the cleanest answer.

A guard inside the body works everywhere, including under #!/bin/sh where shopt does not exist:

for f in ./*.log; do
  [ -e "$f" ] || continue
  process "$f"
done

[ -e "$f" ] || continue is Chapter 16.4's existence test plus Chapter 17.6's continue, and it is the line you will see in careful scripts everywhere. It also handles a second case nullglob does not: a file that was deleted between the glob expanding and the body reaching it.

6. A Complete Script

Every rule from this chapter, in a script that survives a hostile directory:

#!/usr/bin/env bash
# archive-logs.sh — compress every log in a directory, safely
# usage: archive-logs.sh <dir>
#
# exit: 0 archived (possibly zero) · 1 at least one failure · 2 bad usage

set -o nounset

dir=${1:-}

if [ -z "$dir" ]; then
  echo "usage: ${0##*/} <dir>" >&2
  exit 2
fi
if ! [ -d "$dir" ] || ! [ -r "$dir" ]; then
  echo "${0##*/}: $dir is not a readable directory" >&2
  exit 2
fi

archived=0
failed=0

# "$dir"/* — quoted variable, unquoted glob. Both deliberate.
for f in "$dir"/*.log; do

  # the glob may not have matched, or the file may have gone
  [ -e "$f" ] || continue

  # a directory named something.log is not a log
  [ -f "$f" ] || continue

  if gzip -- "$f" 2>/dev/null; then
    printf 'archived  %s\n' "$f"
    archived=$(( archived + 1 ))
  else
    printf 'FAILED    %s\n' "$f" >&2
    failed=$(( failed + 1 ))
  fi
done

printf -- '-- %d archived, %d failed\n' "$archived" "$failed"
[ "$failed" -eq 0 ]

$ mkdir hostile && cd hostile
$ touch 'my report.log' clean.log notes.txt -- '-n.log'
$ mkdir adir.log
$ ls -1
-n.log
adir.log
clean.log
my report.log
notes.txt

$ ../archive-logs.sh .
archived  ./-n.log
archived  ./clean.log
archived  ./my report.log
-- 3 archived, 0 failed
$ echo $?
0

$ ../archive-logs.sh .
-- 0 archived, 0 failed
$ echo $?
0

$ ../archive-logs.sh /nope
archive-logs.sh: /nope is not a readable directory
$ echo $?
2

Four things in there are the chapter:

"$dir"/*.log — the variable is quoted and the glob is not. Those are two different pieces of the same word and they need opposite treatment: "$dir" must not be split (the directory could be my results), and *.log must be expanded. Writing "$dir/*.log" quotes the asterisk too and gives you one iteration on a file named with a literal star.

./-n.log was archived, and the leading ./ came from $dir being .. Pass an absolute path and names start with /; either way they do not start with a dash. Combined with gzip --, a file called -n.log is handled rather than misread.

The second run archived nothing and still returned 0. The .log files became .log.gz, so the glob matched nothing, [ -e "$f" ] || continue fired, and the loop did nothing. Reporting success for "there was nothing to do" is the right call here — and Chapter 17.1 §5 is the discussion of when it is not.

adir.log was skipped by [ -f "$f" ] || continue. A directory can be named anything, and handing one to gzip is an error you can avoid with one line.

Practice

1. Create clean.log and 'my report.log', then run for f in $(ls), for f in "$(ls)" and for f in *.log, printing [$f]. Count the iterations in each. Explain why quoting makes the first one worse rather than better.
2. Create a file named -n (touch -- -n). Run for f in *; do echo "$f"; done and count the output lines against ls -1. Then swap in printf '%s\n'.
3. Create a file named -rf in a scratch directory. Write a loop body of printf 'WOULD RUN: rm %s\n' "$f" and look at what the command line would have been. Fix it two ways.
4. Create a file whose name contains a newline: touch "$(printf 'two\nlines.log')". Loop over *.log with a glob, then over ls *.log with while read. Which one is right, and at what point was the information lost?
5. Build a two-level tree with a space in one name and a newline in another. Count the files with find | while read and then with find -print0 | while read -r -d ''. Explain the difference in counts.
6. Run the -print0 loop with a counter and print it after the loop. Then make the counter survive, and name the chapter that explains why you had to.
7. In an empty directory, run for f in ./*.log; do echo "[$f]"; done. Fix it with nullglob, then fix it again with a guard, then name one thing the guard catches that nullglob does not.
8. Run archive-logs.sh on a directory containing a file literally named *.log. Does the [ -e "$f" ] guard behave correctly? What would the [ "$f" = './*.log' ] version have done?

Check Your Understanding

1. Two files, one with a space. How many iterations does for f in $(ls) run, and what are the extra ones?
2. Why can't quoting fix $(ls)?
3. A loop over * prints three lines for four files. The glob was correct — what went wrong?
4. What does ./* buy you, and what does it not protect against?
5. Why does ls *.log | while IFS= read -r f fail on a name containing a newline even though the read is careful?
6. What does -d '' mean in read -r -d ''?
7. Name the two ways to handle a glob that matches nothing, and one thing the second catches that the first does not.

7. Remember

• for f in $(ls) is broken, not unfashionable. Two files with one space produce three iterations, two of them on files that do not exist.
• Quoting $(ls) makes it worse — one iteration containing everything. The boundaries were lost before quoting could matter.
• Loop over a glob. The shell produces separate words directly, so spaces and newlines in names survive.
• printf '%s\n' not echo. A file named -n disappears entirely from echo output; printf has no option that a value can impersonate.
• Write ./*, not *, in any loop whose body passes the name to a command — and -- before the operand as well.
• Quotes stop word-splitting, not option parsing. "$f" is necessary and not sufficient.
• A newline-delimited stream cannot represent a name containing a newline. No careful reading fixes an ambiguous format.
• For a tree, use find -print0 with read -r -d '' — NUL is the one byte a name cannot contain.
• An unmatched glob runs the body once on the pattern. Use shopt -s nullglob, or [ -e "$f" ] || continue, which also catches a file that vanished mid-loop.
• "$dir"/*.log — quote the variable, leave the glob bare. "$dir/*.log" quotes the star and breaks.

Chapter 17.3 turns to the loop that has no list at all. When what you need is an index — run ten seeds, retry three times, step through a range — building a list of words to count through is the wrong shape, and Bash has arithmetic for it.