Word Splitting, and Why an Unquoted Variable Is a Bug

Part I · Module 9 · Chapter 9.2 · Word Splitting

Chapter 9.1 put six stages in order and flagged stage four in red. This chapter is that stage.

1. What the Stage Does

After variables, command output and arithmetic have been substituted, Bash looks at the text those substitutions produced and breaks it into separate words.

$ d="my dir"

$ ls -d $d
ls: my: No such file or directory
ls: dir: No such file or directory

$ ls -d "$d"
my dir

Two errors, naming two things you never asked about. ls was given two arguments because the shell made two arguments — and from ls's point of view, it was asked about my and about dir, and neither exists.

This is not a malfunction. The stage is doing precisely its job. The problem is that the stage cannot tell the difference between "a list of three file names separated by spaces" and "one file name that contains a space", because by the time it runs, both are just text.

2. What It Splits On

The splitting characters come from a variable, IFS — the internal field separator. By default it holds space, tab and newline, which is why spaces are what you notice.

$ v="a:b:c"

$ for x in $v; do echo "  [$x]"; done
  [a:b:c]

One item — there are no spaces in it, so nothing split. Change IFS and the same value splits differently:

$ v="a:b:c"
$ IFS=:
$ for x in $v; do echo "  [$x]"; done
  [a]
  [b]
  [c]

Whitespace separators behave differently from other ones

This is a genuine wrinkle and worth knowing, because it explains two behaviours that look inconsistent.

With the default whitespace IFS, runs collapse and the ends are trimmed:

$ v="   a    b   "
$ printf '[%s] ' $v; echo
[a] [b]

Three leading spaces, four in the middle, three trailing — and the result is exactly two fields with nothing empty.

With a non-whitespace IFS, every separator counts:

$ v="a::b"
$ IFS=:
$ printf '[%s] ' $v; echo
[a] [] [b]

Three fields, the middle one empty. The two colons were not collapsed into one.

That difference is what you want in both cases — consecutive spaces in prose are noise, whereas consecutive colons in a::b mean an empty field — but it does mean "splitting on IFS" is two slightly different behaviours wearing one name.

3. The Failure That Is Worse Than Splitting

Splitting one argument into two is visible — you get an error naming words you did not type. The other failure is silent.

An unquoted expansion that is empty produces no argument at all.

$ unset u

$ set -- a $u b
$ printf 'argc=%s: ' "$#"; printf '[%s] ' "$@"; echo
argc=2: [a] [b] 

$ set -- a "$u" b
$ printf 'argc=%s: ' "$#"; printf '[%s] ' "$@"; echo
argc=3: [a] [] [b]

Unquoted, the command received two arguments where the line clearly had three. Quoted, it received three, the middle one empty.

Nothing reports this. The command runs, with the wrong number of arguments, and whether it notices depends entirely on the command. A tool expecting --output <file> and receiving only --output will consume the next argument as its filename, and now something else entirely is being overwritten.

4. The Rule

The practical form, carried from Chapter 8.2:

cp "$src" "$dst"
mkdir -p "$logdir"
grep "$pattern" "$file"
cd "$project/rtl" || exit 1
echo "count: $(wc -l < "$file")"

Note the last one: quotes nest correctly inside $( ), because the substitution's contents are parsed as a fresh command line. Chapter 9.4 covers why.

5. The Idioms That Are Broken

Three patterns you will see in real scripts, all broken by stage four, all with a direct fix.

Iterating over ls

$ for f in $(ls *.txt); do echo "  [$f]"; done
  [a.txt]
  [b.txt]
  [two]
  [words.txt]

$ for f in *.txt; do echo "  [$f]"; done
  [a.txt]
  [b.txt]
  [two words.txt]

Four items versus three, for the reason Chapter 9.1 established: a glob's results are produced at stage five, after splitting, so they are never split. Use the glob directly. There is no situation in which $(ls) is better.

Reading a file by substituting it

for line in $(cat list.txt); do ...   # splits on every space, not every line

The file's whole contents arrive as one piece of text and are then split on spaces and newlines, so a line with two words becomes two iterations. Use the loop from Chapter 8.6:

while IFS= read -r line; do ...; done < list.txt

Building a command in a string

cmd="ls -l 'my dir'"
$cmd                    # does not work

The quotes inside the value are data by the time $cmd is expanded — Chapter 9.1 §5 — so splitting cuts at the space inside them and ls receives 'my and dir', quote characters and all.

Use an array (Chapter 19.3 is the quoting rule that makes it work):

cmd=(ls -l "my dir")
"${cmd[@]}"

"${cmd[@]}" expands to exactly the elements, each one argument, regardless of content. It is the only correct way to hold a command in a variable.

6. When You Actually Want Splitting

The rule has edges. Four cases where leaving an expansion unquoted is correct — and in every one of them, it is a deliberate choice you could defend in review.

Turning a configured string into separate arguments.

EXTRA_FLAGS="-v --seed 7"
./run_sim.sh $EXTRA_FLAGS

You want three arguments. Quoting would pass one twelve-character argument. (An array is still better if you control the source.)

Splitting a known-format field.

IFS=: read -r user _ uid _ _ home _ <<< "$passwd_line"

Here the splitting is scoped and the separator is specified — the safe version of the same idea.

A value you have just produced and know the shape of.

pids=$(pgrep -f run_sim)
kill $pids

PIDs cannot contain spaces, so splitting them into separate arguments is exactly right. Note the dependence on knowing that — which is why the next point matters.

Optional flags from Chapter 8.2.

./run_sim.sh ${SEED:+--seed "$SEED"}

The whole construct is unquoted so that it can disappear entirely when SEED is unset.

7. "$@" Is the Same Rule

When a script or function passes its own arguments along, the same stage applies, and the three forms give three different answers:

$ set -- "two words.txt" "b.txt"

$ printf 'original argc=%s: ' "$#"; printf '[%s] ' "$@"; echo
original argc=2: [two words.txt] [b.txt]

"$@" is the only form that preserves the original arguments exactly. It is special-cased by the shell: it expands to one word per argument, whatever those arguments contain.

"$*" joins them into a single string, separated by the first character of IFS. That is occasionally what you want for a message; it is never what you want for passing arguments on.

Bare $@ has no legitimate use that "$@" does not do better.

#!/bin/bash
# a wrapper that adds one flag and passes everything else through
exec ./real_tool --verbose "$@"

(Module 15 covers positional parameters properly; the quoting is this chapter's.)

8. Finding the Bugs You Already Have

Two practical techniques.

set -x shows the finished command, after every stage:

$ d="my dir"
$ set -x
$ ls -d $d
+ ls -d my dir
$ ls -d "$d"
+ ls -d 'my dir'

Bash prints the trace with its own quoting, so a multi-word argument appears quoted. Two bare words versus one quoted word is the bug, visible directly.

Test with a hostile name. The fastest way to find every splitting bug in a script is to run it over a directory containing a file with a space in the name. Most scripts have never seen one.

mkdir -p /tmp/hostile && cd /tmp/hostile
touch "normal.txt" "two words.txt" "it's.txt" "-leading-hyphen.txt"

That last one belongs to Chapter 9.6, and it will break a different set of things.

Practice

mkdir -p /tmp/ws && cd /tmp/ws
touch a.txt b.txt "two words.txt"
mkdir -p "my dir"

1 — Feel the split.

d="my dir"
ls -d $d
ls -d "$d"

2 — Watch the argument count change.

unset u
set -- a $u b;   printf 'argc=%s: ' "$#"; printf '[%s] ' "$@"; echo
set -- a "$u" b; printf 'argc=%s: ' "$#"; printf '[%s] ' "$@"; echo

3 — See whitespace collapse but colons not.

v="   a    b   "; printf '[%s] ' $v; echo
( v="a::b"; IFS=:; printf '[%s] ' $v; echo )

The parentheses run it in a subshell so your IFS is not changed — which is itself the lesson from §2.

4 — Break and fix the ls loop.

for f in $(ls *.txt); do echo "  [$f]"; done
for f in *.txt;       do echo "  [$f]"; done

5 — Try to run a command from a string, then do it properly.

cmd="ls -d 'my dir'"
$cmd
cmd=(ls -d "my dir")
"${cmd[@]}"

6 — Compare the three argument forms.

set -- "two words.txt" "b.txt"
f() { printf '  argc=%s: ' "$#"; printf '[%s] ' "$@"; echo; }
f "$@"
f $@
f "$*"

7 — Read the trace.

d="my dir"; set -x; ls -d $d; ls -d "$d"; set +x

8 — Clean up. cd /tmp && rm -rf /tmp/ws

Check Your Understanding

1. src="Q3 report.txt" and cp $src backup/ fails with two errors. Which stage, and what is the fix?

2. Why does an unquoted empty variable change how many arguments a command receives?

3. v="a b" with the default IFS gives two fields. v="a:::b" with IFS=: gives four. Why the difference?

4. Why is for f in *.txt correct where for f in $(ls *.txt) is not?

5. When is leaving an expansion unquoted the right choice, and what should accompany it?

6. A wrapper script passes its arguments on with $@. What breaks, and what should it be?

9. Remember

• Chapter 33.5 is the catalogue of what this does in a real flow — including an empty variable and a space together producing the correct argument count with every value wrong.
• Word splitting is stage four: it cuts the text an expansion produced into words.
• It cannot tell a list from a single value containing a space. By the time it runs, both are text.
• IFS holds the separators — space, tab and newline by default.
• Whitespace separators collapse and trim; other separators do not. a::b on IFS=: is three fields.
• Set IFS for one command, not globally. IFS=: read -r … cannot leak.
• An unquoted value with a space becomes too many arguments.
• An unquoted empty value becomes no argument at all — and that one is silent.
• Quote every expansion as a reflex: "$var", "$(cmd)", "${arr[@]}".
• Use globs, not $(ls). A glob's results are produced after splitting and are never split.
• Read files with while IFS= read -r line, not by substituting the whole file.
• A command held in a string cannot work — the quotes inside it are data. Use an array and "${cmd[@]}".
• Deliberate splitting is legitimate for flag strings, PID lists and ${VAR:+…} — write a comment saying so.
• "$@" is the only correct way to pass arguments on. $@ splits them; "$*" joins them into one.
• set -x prints the finished command, and shellcheck finds unquoted expansions for you — Chapter 21.7 runs it properly and reads its findings.

You know what splitting does and that quotes stop it. The next chapter is about the quotes themselves — what each of the three switches off, and how to pick deliberately rather than by trial and error.