Building an Array from Command Output

Part I · Module 19 · Chapter 19.4 · Building an Array from Command Output

Three chapters of arrays, and every array so far was written out by hand or came from a glob. Real lists come from find, from a testlist file, from a tool's output.

1. The One-Liner Is the Bug

Three files, with the awkward names Chapter 17.2 established are legal:

$ ls -1
nl
name.sv
sp ace.sv
$ ls -1 sub
deep.sv

Three .sv files: sp ace.sv, nl⏎name.sv (one file whose name contains a newline), and sub/deep.sv.

Now the line everybody writes:

$ a=($(find . -name '*.sv'))
$ echo "count=${#a[@]}"
count=5
$ printf '  [%s]\n' "${a[@]}"
  [./sub/deep.sv]
  [./nl]
  [name.sv]
  [./sp]
  [ace.sv]

Five elements for three files. ./nl, name.sv, ./sp and ace.sv are four names, and none of them is a file.

This is Chapter 19.1's defect exactly. $(find ...) produces one block of text; the assignment is unquoted, so word-splitting breaks it on every space and every newline in IFS; and the array faithfully stores whatever came out. The array is not at fault — it never received the right items.

And quoting it does not help, for Chapter 19.1 §2's reason:

$ a=("$(find . -name '*.sv')")
$ echo "count=${#a[@]}"
count=1

One element containing all three names with newlines in the middle. Neither form is right, because the boundaries were destroyed by find's output format before the assignment saw it.

2. Read One Record, Append One Element

The portable fix is a loop — Chapter 17.5's loop, with += instead of a body:

$ a=()
$ while IFS= read -r -d '' f; do
>   a+=("$f")
> done < <(find . -name '*.sv' -print0)
$ echo "count=${#a[@]}"
count=3
$ printf '  [%s]\n' "${a[@]}"
  [./sub/deep.sv]
  [./nl
name.sv]
  [./sp ace.sv]

Three elements, exactly right, including the one whose name spans two display lines because it genuinely contains a newline.

Every token in that loop is load-bearing and every one of them you already have:

• -print0 / -d '' — the NUL separator, because it is the one byte a filename cannot contain (Chapter 17.2 §4).
• IFS= — so leading and trailing whitespace in a name survives (Chapter 17.5 §2).
• -r — so a backslash in a name is a backslash (Chapter 17.5 §3).
• a+=("$f") — quoted, one element per record (Chapter 19.2 §3).
• < <(...) — process substitution, not a pipe, so the array still exists after the loop (Chapter 17.7).

That last one is the easiest to get wrong, and it fails in the most confusing way:

$ a=(); find . -name '*.sv' -print0 | while IFS= read -r -d '' f; do a+=("$f"); done
$ echo "count=${#a[@]}"
count=0

Zero. The loop ran, appended three elements, and did it in a subshell that then exited. Chapter 17.7 measured this for counters; an array is the same thing with more at stake.

3. A Glob Straight Into an Array

When the list is "the files in one directory", you do not need a command at all:

$ shopt -s nullglob
$ a=(./*.sv)
$ echo "count=${#a[@]}"
count=2
$ printf '  [%s]\n' "${a[@]}"
  [./nl
name.sv]
  [./sp ace.sv]

Two, correctly — this directory has two .sv files and sub/deep.sv is not in it. Globbing never serialises anything, so there is nothing to misparse, and Chapter 17.2's whole argument applies.

Prefer a glob whenever it can do the job. It is shorter, it forks nothing, it cannot be wrong, and shopt -s nullglob gives you an honest zero. Reach for find and a read loop when you need to recurse, or to filter on type, age or size.

4. mapfile, and Why This Chapter Cannot Demonstrate It

Bash 4 added a builtin that does exactly this job in one line:

mapfile -t logs < <(find . -name '*.log')     # bash 4.0+

-t strips the trailing newline from each line, and each line becomes one element. readarray is the same builtin under a second name. It is genuinely the right tool when you have it.

You do not have it here:

$ mapfile -t a < <(printf 'x\n')
bash: mapfile: command not found
$ readarray -t a < <(printf 'x\n')
bash: readarray: command not found
$ bash --version | head -1
GNU bash, version 3.2.57(1)-release (arm64-apple-darwin25)

Every other transcript in this chapter was run and its output pasted. That one is quoted from the documentation, because macOS ships Bash 3.2 and mapfile arrived in Bash 4.0 — so nothing in this section is measured, and you should treat it accordingly.

Two consequences for what you write:

mapfile is newline-delimited by default. mapfile -t a < <(find ...) has exactly the §2 callout's limitation — it is wrong for a name containing a newline. Bash 4.4 added -d '' for NUL, so the fully correct mapfile form needs 4.4, not 4.0.

A script that must run on macOS cannot use it. If you want to use it where available and still work elsewhere, test for it rather than for a version number:

if type -t mapfile > /dev/null; then
  mapfile -t logs < <(find . -name '*.log')
else
  logs=()
  while IFS= read -r l; do logs+=("$l"); done < <(find . -name '*.log')
fi

$ type -t mapfile > /dev/null && echo "available" || echo "not available"
not available

Honestly: for three lines of fallback you may as well write the loop unconditionally. The loop is correct everywhere, it is the form a reader of any Bash version understands, and it is where you have to put the -d '' anyway if the records are file names.

5. Other Sources

A file, line by line — the same loop with a redirect:

$ printf 'smoke\nburst 7\n# comment\n\ncache coherency\n' > tests.txt
$ a=()
$ while IFS= read -r l || [ -n "$l" ]; do
>   case $l in ''|\#*) continue ;; esac
>   a+=("$l")
> done < tests.txt
$ echo "count=${#a[@]}"
count=3
$ printf '  [%s]\n' "${a[@]}"
  [smoke]
  [burst 7]
  [cache coherency]

The || [ -n "$l" ] guard is Chapter 17.5 §4's, and the case filter is Chapter 17.6 §2's. Nothing new — the only change is a+=("$l") instead of doing the work inline, which is what lets you count the list before processing it.

Splitting one string on a known separator. When you genuinely have a delimited string — a PATH, a config value — set IFS for one assignment:

$ IFS=: read -r -a parts <<< "/usr/bin:/bin:/opt/tool bin"
$ echo "count=${#parts[@]}"
count=3
$ printf '  [%s]\n' "${parts[@]}"
  [/usr/bin]
  [/bin]
  [/opt/tool bin]

read -a NAME reads words into an array, and IFS=: makes the colon the only separator — so the space in /opt/tool bin survives. That is the correct way to split a PATH-style string, and it is a one-liner because the separator here is genuinely guaranteed absent from the elements (Chapter 19.1 §4's "name the forbidden character" test, passed).

Another array — copying needs the same quoting as everything else:

$ a=(p "q r" s)
$ b=("${a[@]}")
$ echo "${#b[@]}"
3
$ c=(${a[@]})
$ echo "${#c[@]}"
4

Chapter 19.3, one more time.

6. A Complete Script

A regression driver that builds its work list from a tree and its options from a file:

#!/usr/bin/env bash
# gather.sh — build a work list from a tree, report it, and prove it survived
# usage: gather.sh <dir> [testlist]
#
# exit: 0 found work · 2 bad usage · 3 nothing to do

set -o nounset
shopt -s nullglob

note() { printf '%s: %s\n' "${0##*/}" "$*" >&2; }

dir=${1:-}
list=${2:-}

[ -n "$dir" ] || { note "usage: ${0##*/} <dir> [testlist]"; exit 2; }
[ -d "$dir" ] || { note "$dir is not a directory"; exit 2; }

# ---- sources (every name survives, whatever is in it) --------------------

# 1. a tree, NUL-delimited, via process substitution so the array survives
declare -a sources=()
while IFS= read -r -d '' f; do
  sources+=("$f")
done < <(find "$dir" -type f -name '*.sv' -print0)

# 2. one directory, straight from a glob — no process, cannot be wrong
declare -a top_level=("$dir"/*.sv)

# 3. a text file, newline-delimited, with comments and blanks dropped
declare -a tests=()
if [ -n "$list" ]; then
  if ! [ -r "$list" ]; then
    note "$list is not readable"
    exit 2
  fi
  while IFS= read -r l || [ -n "$l" ]; do
    case $l in ''|\#*) continue ;; esac
    tests+=("$l")
  done < "$list"
fi

# ---- report --------------------------------------------------------------

printf 'sources    %d (recursive)\n' "${#sources[@]}"
printf 'top level  %d (this directory only)\n' "${#top_level[@]}"
printf 'tests      %d\n' "${#tests[@]}"

if [ "${#sources[@]}" -eq 0 ]; then
  note "no .sv files under $dir"
  exit 3
fi

echo '-- sources:'
for s in "${sources[@]}"; do
  printf '  [%s]\n' "$s"
done

if [ "${#tests[@]}" -gt 0 ]; then
  echo '-- tests:'
  for t in "${tests[@]}"; do
    printf '  [%s]\n' "$t"
  done
fi

$ mkdir -p tree/sub
$ touch 'tree/sp ace.sv' tree/sub/deep.sv
$ touch "tree/$(printf 'nl\nname.sv')"
$ printf 'smoke\nburst 7\n# a comment\n\ncache coherency\n' > tests.txt

$ ./gather.sh tree tests.txt
sources    3 (recursive)
top level  2 (this directory only)
tests      3
-- sources:
  [tree/sub/deep.sv]
  [tree/nl
name.sv]
  [tree/sp ace.sv]
-- tests:
  [smoke]
  [burst 7]
  [cache coherency]

$ ./gather.sh tree
sources    3 (recursive)
top level  2 (this directory only)
tests      0
-- sources:
  [tree/sub/deep.sv]
  [tree/nl
name.sv]
  [tree/sp ace.sv]

$ mkdir bare && ./gather.sh bare
sources    0 (recursive)
top level  0 (this directory only)
tests      0
gather.sh: no .sv files under bare
$ echo $?
3

Now replace just the process substitution with a pipe and run the identical tree:

$ sed 's|done < <(find "$dir" -type f -name .\*.sv. -print0)|done|; s|^while IFS= read -r -d .. f; do|find "$dir" -type f -name "*.sv" -print0 \| while IFS= read -r -d "" f; do|' gather.sh > piped.sh
$ chmod +x piped.sh && ./piped.sh tree tests.txt
sources    0 (recursive)
top level  2 (this directory only)
tests      3
piped.sh: no .sv files under tree
$ echo $?
3

sources is 0 and the script reports there is nothing to do — on a tree with three files in it. The other two arrays are fine, because neither was built through a pipe. That is the single most confusing failure in this chapter: two thirds of the script works and the third is silently empty.

Four decisions:

Three sources, three different techniques, each matched to the data. NUL-delimited find for a tree of names nobody vetted; a plain glob for one directory, because it forks nothing and cannot be wrong; a newline loop for a text file whose records are lines by definition.

< <(find ...) not find ... |. The only difference between the working script and piped.sh.

Every array is count-guarded before expansion, which is Chapter 19.2 §5 — without it the bare run would abort under set -u instead of printing a clean message.

declare -a on all three even though name=() would do, because a reader meeting sources+=("$f") fifteen lines later should not have to work out what sources is.

Practice

1. Build the three-file fixture from §1, including the name with a newline. Run a=($(find . -name '*.sv')) and print the count and the elements. Account for every element that is not a file.
2. Quote it — a=("$(find ...)") — and print the count. Explain why neither form is right using Chapter 19.1.
3. Write the NUL read loop and confirm the count is 3. Then remove -print0 and -d '' and report the new count. Which name broke, and which survived?
4. Change < <(find ...) to find ... | and print the count after the loop. Name the chapter that explains it.
5. Use a glob instead of find on the same directory. Why is the count 2 rather than 3, and why is that correct?
6. Run mapfile -t a < <(printf 'x\n') on your machine. If it fails, write the type -t guard and the loop fallback, then argue for or against bothering with the guard.
7. Split "/usr/bin:/bin:/opt/tool bin" into an array with IFS=: read -r -a. Confirm the space survives. Then apply Chapter 19.1 §4's test: name the character that cannot appear in a PATH element.
8. Copy an array two ways — b=("${a[@]}") and c=(${a[@]}) — where one element contains a space. Print both counts.
9. Read a testlist with comments, blanks, and a final line with no newline. Confirm the count, then remove the || [ -n "$l" ] guard and confirm which test disappears.
10. Run piped.sh from §6 on a populated tree. Which arrays are correct, which is empty, and what makes this harder to debug than a script that is wrong everywhere?

Check Your Understanding

1. a=($(find ...)) on three files gives five elements. Where did the extra two come from?
2. Does quoting the command substitution fix it?
3. Name the five parts of the correct read loop and what each one defends against.
4. Why < <(cmd) rather than cmd |?
5. When is the newline-delimited loop good enough?
6. When should you use a glob instead of find?
7. Why does this chapter not show mapfile working?

7. Remember

• a=($(cmd)) is the bug. Three files measured as five elements, two of which name nothing.
• Quoting it gives one element. Neither form is right; the boundaries were gone before the assignment.
• The portable fix is a read loop plus +=, with all five parts: -print0/-d '', IFS=, -r, a+=("$f"), and < <(...).
• A pipe into the loop leaves the array empty. Process substitution, every time.
• Newline-delimited reading is enough when records cannot contain a newline — testlists, config files, tool reports.
• Use NUL for file names you did not create, where a newline is legal and being wrong is expensive.
• Prefer a glob when the list is one directory. No process, nothing to misparse, and nullglob for an honest zero.
• mapfile is Bash 4.0 and absent on macOS — and it is newline-delimited, so the fully correct form needs 4.4. Write the loop.
• IFS=: read -r -a parts <<< "$str" splits a genuinely delimited string in one line.
• Copy an array with b=("${a[@]}"). Unquoted re-splits it.

Chapter 19.5 is the payoff. You now have a correct list; the next chapter is how to hand it to a command as that command's arguments — which is the answer to "I built my command line in a string and it broke", and includes a measured command injection.