for: Iterating Over a List of Words

Part I · Module 17 · Chapter 17.1 · for: Iterating Over a List of Words

Module 16 gave a script the ability to choose. It still runs once. A regression has four hundred tests, a filelist has nine hundred files, and a results directory has a log per run — and none of that is reachable with a conditional.

1. The Shape

$ for t in smoke burst random; do echo "$t"; done
smoke
burst
random

Three parts: a variable name, a list of words, and a body.

for  t  in  smoke burst random
do
  echo "$t"
done

The semicolons in the one-line form are separators exactly as in Chapter 16.1 — do is a word that has to be distinguishable from the end of the list, and a newline does the same job. done closes it.

t is an ordinary shell variable. It is not scoped to the loop, it is not special, and it keeps its last value afterwards:

$ for t in smoke burst random; do :; done
$ echo "$t"
random

2. The List Is Finished Before the Loop Starts

This is the chapter. Everything else follows from it.

Bash does not hold an expression and re-evaluate it per iteration. It takes the whole line, runs every expansion in it — Chapter 9.1's pipeline, in Chapter 9.1's order — and what comes out is a flat list of plain words. Only then does the loop begin, and all it does is walk that list.

Here is the proof, and it is worth typing:

$ list="a b c"
$ for x in $list; do list="CHANGED"; printf '%s ' "$x"; done; echo "| list=$list"
a b c | list=CHANGED

The loop reassigned list on its very first pass. It still printed a b c. By the time the body ran once, $list had already been expanded, split into three words, and handed over; the variable it came from was no longer connected to anything.

3. Where the List Comes From

Any expansion at all, because by step 2 they are indistinguishable. Literals:

$ for t in smoke burst random; do printf '%s ' "$t"; done; echo
smoke burst random

Brace expansion (Chapter 9.5) — a counted range with no external process:

$ for i in {1..5}; do printf '%s ' "$i"; done; echo
1 2 3 4 5

Braces cross-multiply, which is occasionally exactly what you want:

$ for i in {1..3}{a,b}; do printf '%s ' "$i"; done; echo
1a 1b 2a 2b 3a 3b

A glob (Chapter 3.5) — the most common loop in real scripts:

$ for f in *.log; do printf '%s ' "$f"; done; echo
clean.log run.log

Command substitution (Chapter 8.4) — and the one to be careful with:

$ for i in $(seq 1 5); do printf '%s ' "$i"; done; echo
1 2 3 4 5

That works because seq emits numbers, and numbers have no spaces in them. The moment the words could contain a space, this form is broken — which is Chapter 17.2, and is the single most common bug in shell scripts.

A variable, unquoted on purpose:

$ tests="smoke burst random"
$ for t in $tests; do printf '%s ' "$t"; done; echo
smoke burst random

$tests is deliberately unquoted here — the whole point is for word-splitting to break it into three words. Quote it and you get one iteration with all three words in it:

$ for t in "$tests"; do printf '[%s] ' "$t"; done; echo
[smoke burst random]

This is the one place in Bash where leaving a variable unquoted is the correct choice, and it is why Chapter 19.1 exists: a string pretending to be a list works only until one element contains a space.

4. The Brace Range That Does Not Work

A trap that follows directly from Section 2, and that everyone hits once:

$ n=5
$ for i in {1..$n}; do printf '%s ' "$i"; done; echo
{1..5}

One iteration, on the literal text {1..5}.

Chapter 9.1's ordering is the whole explanation: brace expansion happens before parameter expansion. When Bash reached the braces, $n was still $n — not a number — and {1..$n} is not a valid range, so brace expansion left it alone. Then $n expanded, producing the five characters you see.

Three things do work:

$ for i in $(seq 1 $n);      do printf '%s ' "$i"; done; echo
1 2 3 4 5
$ for (( i=1; i<=n; i++ ));  do printf '%s ' "$i"; done; echo
1 2 3 4 5
$ for i in $(eval echo {1..$n}); do printf '%s ' "$i"; done; echo
1 2 3 4 5

The second is the right answer and is Chapter 17.3. The first costs a process. The third works and you should not write it — eval on anything a user can influence is a command-injection hole, and there is no reason to accept that risk for a counting loop.

5. An Empty List Runs Zero Times

$ for x in; do echo "BODY"; done; echo "status=$?"
status=0

$ empty=""
$ for x in $empty; do echo "BODY"; done; echo "status=$?"
status=0

Both run the body no times and report success. The second is the interesting one: $empty is unquoted and empty, so word-splitting removed it entirely and the list was genuinely empty — exactly the mechanism from Chapter 16.2 that broke [ $v = "" ], here doing something benign.

Reporting 0 for "nothing to do" is correct, and it is also a place to be careful. A loop that was supposed to process a hundred files and silently processed none is indistinguishable, to the caller, from a loop that succeeded. If "the list was empty" is a problem, you have to say so:

shopt -s nullglob
logs=( *.log )
if [ ${#logs[@]} -eq 0 ]; then
  echo "${0##*/}: no logs to process" >&2
  exit 2
fi

That uses an array, which is Module 19. Chapter 17.2 shows how to do the same check without one.

6. A Glob That Matches Nothing

Related, and worse, because it is not empty:

$ for f in *.nope; do echo "[$f]"; done
[*.nope]

One iteration, on a file that does not exist. Chapter 9.6 established the rule: an unmatched glob expands to itself. So the loop body runs once, with $f set to the pattern, and whatever the body does — wc -l "$f", grep x "$f" — fails with "no such file".

The fix is a shell option (Chapter 10.6):

$ shopt -s nullglob
$ for f in *.nope; do echo "[$f]"; done
$

With nullglob set, an unmatched pattern expands to nothing, the list is empty, and the body never runs. Chapter 17.2 covers when to use it and the guard to write when you cannot.

7. for With No List

Omit in entirely and the loop iterates the script's arguments:

$ set -- p q r
$ for x; do printf '%s ' "$x"; done; echo
p q r

for x; do is exactly for x in "$@"; do — with the quotes, which matters. Chapter 15.4 measured the difference and it shows up here directly:

$ set -- "a b" c
$ for x in "$@"; do echo "[$x]"; done
[a b]
[c]

$ for x in $@; do echo "[$x]"; done
[a]
[b]
[c]

Two arguments became three words. "$@" is the only form that preserves the argument boundaries your caller set, and the bare for x; do shorthand uses it — which makes the shorthand not just shorter but safer than the thing people write by hand.

8. The Loop's Exit Status

A for loop reports the status of the last body iteration:

$ for x in a b; do true;  done; echo "status=$?"
status=0
$ for x in a b; do false; done; echo "status=$?"
status=1

Note what that does not give you: if nine iterations fail and the tenth succeeds, the loop reports success. A loop that processes a list and must report whether everything worked needs to accumulate the answer itself:

failed=0
for log in *.log; do
  if ! check "$log"; then
    failed=$(( failed + 1 ))
  fi
done
[ "$failed" -eq 0 ]

That closing bare test is Chapter 16.7's ending, and it is the whole return value — zero when nothing failed.

9. A Complete Script

A regression driver. Every list in it comes from a different expansion, on purpose:

#!/usr/bin/env bash
# run-suite.sh — run each test in a suite at each of several seeds
# usage: run-suite.sh <outdir> <test> [test...]
#
# exit: 0 everything passed · 1 at least one failure · 2 bad usage

set -o nounset

outdir=${1:-}
shift || true

if [ -z "$outdir" ] || [ $# -eq 0 ]; then
  echo "usage: ${0##*/} <outdir> <test> [test...]" >&2
  exit 2
fi

mkdir -p "$outdir" || exit 2

seeds="1 2 3"
failed=0
ran=0

# "$@" — the tests the caller named, boundaries intact
for test in "$@"; do

  # an unquoted variable, split on purpose into three words
  for seed in $seeds; do

    log="$outdir/${test}_s${seed}.log"
    ran=$(( ran + 1 ))

    # stand-in for the simulator: seed 2 of burst "fails"
    if [ "$test" = "burst" ] && [ "$seed" = "2" ]; then
      printf 'UVM_ERROR : mismatch\n' > "$log"
    else
      printf 'TEST PASSED\n' > "$log"
    fi

    if grep -q "TEST PASSED" "$log"; then
      printf '  pass   %-12s seed %s\n' "$test" "$seed"
    else
      printf '  FAIL   %-12s seed %s\n' "$test" "$seed"
      failed=$(( failed + 1 ))
    fi
  done
done

printf -- '-- %d runs, %d failures\n' "$ran" "$failed"
[ "$failed" -eq 0 ]

$ ./run-suite.sh results smoke burst
  pass   smoke        seed 1
  pass   smoke        seed 2
  pass   smoke        seed 3
  pass   burst        seed 1
  FAIL   burst        seed 2
  pass   burst        seed 3
-- 6 runs, 1 failures
$ echo $?
1

$ ./run-suite.sh results smoke
  pass   smoke        seed 1
  pass   smoke        seed 2
  pass   smoke        seed 3
-- 3 runs, 0 failures
$ echo $?
0

$ ./run-suite.sh results
usage: run-suite.sh <outdir> <test> [test...]
$ echo $?
2

Four decisions in that script are this chapter's content:

for test in "$@" is quoted; for seed in $seeds is not. The first must preserve the caller's argument boundaries, because a test could be named my test. The second must break a string into three words, because that is the only thing a string-as-a-list can do. Both are correct, for opposite reasons, three lines apart.

The loops nest, and the inner list is re-expanded each time — not because the loop re-evaluates anything, but because the inner for is a new statement on every outer pass, and a new statement expands its own list afresh. Section 2 says a loop cannot change its own list; it says nothing about a loop that is started again.

failed accumulates. Section 8's point: the loop's own status is only the last iteration's, so the script counts and tests at the end.

shift || true guards against shift on an empty argument list, which Chapter 15.4 measured as a failure — and under set -o nounset plus a future set -e that would abort the script before it could print its usage message.

Practice

1. Run the Section 2 proof yourself, then extend it: reassign the loop variable x inside the body instead of list. Does the next iteration see your value? Explain both results with one sentence.
2. Predict then run: for i in {1..3}{a,b}, then for i in {a,b}{1..3}. Why is the output order what it is?
3. Set n=4 and run all four of {1..$n}, $(seq 1 $n), for (( i=1; i<=n; i++ )), and {1..4}. Name the expansion-order rule that explains the first.
4. In an empty directory, run for f in *.log; do echo "[$f]"; done. Explain the output, then fix it two ways — with shopt -s nullglob and with a guard inside the body.
5. Run set -- "a b" c then compare for x in "$@", for x in $@, for x in "$*" and bare for x. Which two agree, and which is the one to write?
6. Write a loop over a b c whose last iteration fails but where the loop still reports 0. Then rewrite it so the script reports 1.
7. Create files one.log and two.log, then run for f in *.log; do cp "$f" "$f.bak"; done. How many files exist afterwards, and why is it not an infinite loop?
8. Take run-suite.sh and quote $seeds. What changes, and what does the output tell you about how many words the list had?

Check Your Understanding

1. When does Bash expand the list in a for loop?
2. list="a b c"; for x in $list; do list=CHANGED; done prints a b c. Why?
3. Why does {1..$n} not work, and what does it produce instead?
4. for f in *.nope runs the body once. What is $f, and why?
5. Why is $seeds left unquoted in for seed in $seeds when every other chapter says to quote?
6. Nine of ten iterations fail and the tenth succeeds. What does the loop report?
7. What is for x; do shorthand for?

10. Remember

• The list is finished before iteration one. Every expansion runs first, in Chapter 9.1's order, and the loop walks a flat list of plain words.
• A loop cannot change its own list. Reassigning the variable it came from does nothing — and that is what makes a rename-in-place loop terminate.
• The loop variable is an ordinary variable and keeps its last value after done.
• {1..$n} does not work — brace expansion precedes parameter expansion. Use for (( )), or seq, but never eval.
• An empty list runs the body zero times and returns 0. If "nothing to do" is an error, you must say so yourself.
• An unmatched glob runs the body once on the pattern itself. shopt -s nullglob is the fix.
• for x in $var unquoted is correct when you want the string split — the one sanctioned exception to quoting, and the reason arrays exist.
• for x; do means for x in "$@"; do, quotes included, so it preserves argument boundaries.
• The loop's exit status is its last iteration's. Accumulate a failure count and test it after done.

Chapter 17.2 takes the most common loop in all of shell scripting — one over files — and shows why for f in $(ls) is not merely unfashionable but actually broken, with a file name that makes it process something that does not exist.