Why a String Is Not a List

Part I · Module 19 · Chapter 19.1 · Why a String Is Not a List

Every module since Module 8 has used a string as a list, and four of them admitted it was wrong.

1. Four Promises, Now Due

Chapter 17.1 §3 had you write for t in $tests with the variable deliberately unquoted, and said: "a string pretending to be a list works only until one element contains a space."

Chapter 17.2 measured that — two files, one with a space, and for f in $(ls) ran three iterations, two of them on files that do not exist.

Chapter 18.4 §7 hit it again from the other end: a function printing three values, captured into one string and re-split, produced four.

Chapter 18.5 hit it a fourth time without naming it. audit.sh called count_lines twice on every file — once in report and once in total — because there was nowhere to keep the per-file counts between the two. The Permission denied appearing twice in that transcript was this problem wearing a different hat.

All four are one defect. Here it is, isolated:

$ count() { echo "  $# arguments:"; printf '    [%s]\n' "$@"; }

$ tests="smoke two words burst"
$ count $tests
  4 arguments:
    [smoke]
    [two]
    [words]
    [burst]

Four arguments. The author meant three — smoke, two words, burst — and there is nothing in $tests that says so.

2. Quoting Cannot Fix It

The instinct, trained by every other chapter in this course, is to add quotes:

$ count "$tests"
  1 arguments:
    [smoke two words burst]

One argument. Quoted is as wrong as unquoted, in the opposite direction.

That is the proof. Everywhere else in Bash, quoting is the fix — Chapter 9.2's rule, Chapter 16.2's rule, Chapter 17.5's rule. Here neither form is right, because the information needed to produce the right answer is not in the string:

"smoke two words burst"
      ^   ^     ^
      which of these three spaces separates items,
      and which is part of an item?

Nothing can answer that. Word-splitting guesses "all of them" and gets four; quoting guesses "none of them" and gets one. The item boundaries were discarded at the moment the string was built, and no reader of that string — not the shell, not awk, not you — can recover them.

3. The Same Defect in Three More Costumes

From a command's output (Chapter 18.4's version):

$ a=($(printf 'x\ny z\n'))
$ echo "count=${#a[@]}"
count=3

Two records in, three elements out. The newline and the space were both treated as separators because IFS contains both.

From a glob through a string:

$ ls -1
clean.log
my report.log
$ files=$(ls)
$ count $files
  3 arguments:
    [clean.log]
    [my]
    [report.log]

And from a function's return value, which is Chapter 18.4's whole §7.

Four appearances, one cause: at some point the items were written into a single stream with a separator, and the separator is also a legal character inside an item.

4. The Workarounds, and What Each Costs

People do get this to work, and it is worth knowing why the standard tricks are not good enough — because each one narrows the hole rather than closing it.

Change IFS to something rarer. A newline is a less likely character inside a filename than a space:

$ files=$(ls)
$ IFS=$'\n'
$ count $files
  2 arguments:
    [clean.log]
    [my report.log]

Correct — for this data. It fails the moment a name contains a newline, which Chapter 17.2 measured as entirely legal:

$ ls -1
ok.log
two
lines.log
$ files=$(ls); IFS=$'\n'; count $files
  3 arguments:
    [ok.log]
    [two]
    [lines.log]

And it has a second cost: IFS is global, so changing it changes word-splitting for everything else in the script until you change it back. Chapter 17.5's IFS= prefix exists precisely so that you do not have to.

Use a separator that cannot appear in the data. There is exactly one — the NUL byte — and Chapter 17.2 measured it working. But a Bash variable cannot contain a NUL:

$ v=$(printf 'a\0b')
$ printf '%s' "$v" | xxd -p
6162

The NUL is gone — a and b are now adjacent with nothing between them. So NUL works as a stream separator between two processes, which is why find -print0 | while read -d '' is correct, and cannot work as an in-variable separator. You can pass a NUL-delimited list through a pipe; you cannot hold one in a string.

Pick a separator you believe the data will not contain — a colon, a pipe, a control character. This is what PATH does, and it is why a directory with a colon in its name cannot go on your PATH. It is a documented restriction on the data rather than a solution, and it is only acceptable when you control the data and can state the restriction.

Every one of these makes the failure rarer. None makes it impossible, because they are all the same move: encode structure in a stream and hope the encoding is unambiguous.

5. The Fix Is to Not Join Them

An array keeps each item in its own slot, so there is no separator and nothing to parse:

$ tests=(smoke "two words" burst)
$ count "${tests[@]}"
  3 arguments:
    [smoke]
    [two words]
    [burst]

Three items in, three arguments out. Not because the quoting is cleverer — because the space inside two words was never asked to double as a boundary.

That is the whole idea, and the rest of this module is its mechanics: how to build one (19.2), the one expansion form that preserves the boundaries (19.3), how to fill one from a command (19.4), how to use one as a command's arguments (19.5), and how to key one by name instead of number (19.6).

6. A Scalar Is Already an Array of One

A bridge into Chapter 19.2, and a genuine surprise:

$ x=hello
$ echo "${x[0]}"
hello
$ echo "${#x[@]}"
1

An ordinary variable answers to [0] and reports a length of 1. There is no separate "scalar" type in Bash — every variable is an array, and a plain assignment sets element zero.

Which explains a trap in the other direction:

$ a=(p q r)
$ echo "$a"
p
$ echo "${a[0]}"
p

$a on an array is element 0, not the whole array. It is not an error and produces no warning; it silently gives you the first item. Any time you see $myarray without a subscript in real code, it is either a deliberate "first element" or — far more often — a bug.

7. When a String-as-List Is Still Fine

This chapter is not an instruction to use arrays everywhere. A string is a perfectly good list when you control every element and can guarantee none contains a separator:

seeds="1 2 3"
for seed in $seeds; do ...          # fine: integers cannot contain spaces

Chapter 17.1's run-suite.sh did exactly that, deliberately and with a comment. The things that make it safe are specific: the values are literals in the script, they are numbers, and nobody outside the file can change them.

The moment any element comes from outside — a filename, a user argument, a line from a log, a tool's output — those guarantees are gone and you want an array. The test is simple: can I name the character that cannot appear in any element? If not, do not use a string.

8. A Complete Comparison

The same job twice, so the difference is a diff rather than an argument. A script that collects a list of logs and reports on each:

#!/usr/bin/env bash
# both-ways.sh — the same work with a string list and with an array
# usage: both-ways.sh <dir>

set -o nounset

dir=${1:?usage: ${0##*/} <dir>}

echo "=== as a string ==="
files=""
for f in "$dir"/*.log; do
  [ -e "$f" ] || continue
  files="$files $f"                 # joining: the boundaries go here
done

n=0
for f in $files; do                 # splitting: and cannot come back
  n=$(( n + 1 ))
  printf '  %-30s %s\n' "$f" "$( [ -f "$f" ] && echo ok || echo MISSING )"
done
echo "  counted $n"

echo "=== as an array ==="
afiles=()
for f in "$dir"/*.log; do
  [ -e "$f" ] || continue
  afiles+=("$f")                    # appending: each item keeps its slot
done

n=0
for f in "${afiles[@]}"; do         # expanding: one argument per item
  n=$(( n + 1 ))
  printf '  %-30s %s\n' "$f" "$( [ -f "$f" ] && echo ok || echo MISSING )"
done
echo "  counted $n"

$ mkdir logs
$ touch logs/clean.log 'logs/my report.log'
$ ls -1 logs
clean.log
my report.log

$ ./both-ways.sh logs
=== as a string ===
  logs/clean.log                 ok
  logs/my                        MISSING
  report.log                     MISSING
  counted 3
=== as an array ===
  logs/clean.log                 ok
  logs/my report.log             ok
  counted 2

Three "files", two of which do not exist, against two that do. The two halves of that script differ by four characters: files="$files $f" versus afiles+=("$f"), and $files versus "${afiles[@]}".

Note what the string version got right: the glob was correct, "$f" was quoted everywhere, [ -e "$f" ] || continue was in place. Every lesson from Chapter 17.2 was applied, and it still reported two files that do not exist — because the damage happened in the files="$files $f" line, where correct quoting cannot help.

Practice

1. Write the count helper and run it on tests="smoke two words burst" both quoted and unquoted. Record both argument counts, then state in one sentence why neither is 3.
2. Point at the exact character in "smoke two words burst" that is ambiguous, and say what information would have to be added to the string to resolve it.
3. Build the Section 4 IFS=$'\n' workaround and show it working. Then create a file whose name contains a newline and show it failing. What did the workaround actually buy?
4. Run v=$(printf 'a\0b'); printf '%s' "$v" | xxd -p. Explain what happened to the NUL, and why that rules out the one separator that would be safe.
5. Set x=hello and print ${x[0]} and ${#x[@]}. Then set a=(p q r) and print $a. Explain both results with one sentence about Bash's type system.
6. Take both-ways.sh and run it on a directory with three logs, one containing a space and one containing a newline. Report both counts.
7. In both-ways.sh, add quotes to for f in $files and re-run. How many iterations now, and is it better or worse?
8. Find a list in a script of your own that is held in a string. Decide whether it is safe by naming the character that cannot appear in any element. If you cannot name one, note it for conversion.

Check Your Understanding

1. tests="smoke two words burst" is passed unquoted to a command. How many arguments, and how many did the author mean?
2. Quoted, how many? Why does that make the point rather than fixing it?
3. Where exactly was the information lost?
4. Why is changing IFS to a newline not a solution?
5. Why can't a string hold a NUL-separated list?
6. a=(p q r). What does echo "$a" print?
7. When is a string-as-list acceptable?

9. Remember

• A space-separated string cannot represent a list whose items may contain spaces. The boundaries are destroyed when the string is built.
• Quoting cannot fix it. Unquoted gives too many items, quoted gives one — and the right answer is neither.
• This is the one place where "did I quote it?" is the wrong question. Ask where the boundaries went.
• Four appearances, one cause — for f in $(ls), a capture re-split, an IFS loop, and nowhere to keep per-item state.
• Changing IFS narrows the hole and makes it global; a newline is still legal in a filename.
• A variable cannot hold a NUL, so the one unambiguous separator works only in a stream.
• An array keeps each item in its own slot, so there is no separator to misparse.
• An array is not a cure for bad quoting — ${a[@]} unquoted re-splits everything, which is Chapter 19.3.
• Every Bash variable is already an array of one. ${x[0]} works on a scalar, and $a on an array is element 0.
• A string list is fine when you can name the forbidden character. If you cannot, use an array.

Chapter 19.2 is the mechanics: creating an array, reading one element or all of them, appending, counting, and the one thing that goes wrong when you delete an element from the middle.