Indexed Arrays: Creating, Reading, Appending

Part I · Module 19 · Chapter 19.2 · Indexed Arrays: Creating, Reading, Appending

Chapter 19.1 argued for arrays. This is how to make one.

1. Creating

The literal form, which is what you will write nearly always:

$ tests=(smoke burst random)
$ echo "${#tests[@]}"
3

Parentheses, items separated by whitespace, no commas. Each item follows the ordinary quoting rules, so anything containing a space gets quoted exactly as it would anywhere else:

$ tests=(smoke "two words" burst)
$ echo "${#tests[@]}"
3

An explicit declaration does the same thing and says so:

$ declare -a tests=(one two)
$ echo "${tests[*]}"
one two

declare -a is optional for an indexed array — a plain name=(...) creates one — but it is worth writing at the top of a script for the same reason local is: it tells a reader the intent before they meet the usage. Inside a function use local -a, which Chapter 18.5's rule requires anyway:

$ f() { local -a a=(x y); echo "${a[*]}"; }; f
x y

An empty array, which is how you start one you will append to:

$ files=()
$ echo "${#files[@]}"
0

By index, which creates the array if it does not exist:

$ a[0]=x
$ a[1]=y
$ echo "${a[*]}"
x y

And from any expansion, since the contents of ( ) go through the usual expansions:

$ a=({1..5})
$ echo "${a[*]}"
1 2 3 4 5
$ a=(./*.log)
$ echo "${#a[@]}"
2

2. Reading

$ a=(p q r s)
$ echo "${a[0]}  ${a[2]}"
p  r
$ echo "${#a[@]}"
4
$ echo "${!a[@]}"
0 1 2 3
$ echo "${a[@]:1:2}"
q r

Two of those rows catch people.

${#a[0]} is a length, not a count. # means "length of" and a[0] is a string, so you get the length of the first element. ${#a[@]} is the count. One character apart:

$ a=(hello world)
$ echo "count=${#a[@]}  length of first=${#a[0]}"
count=2  length of first=5

The last element needs care on this shell. Bash 4.3 added negative subscripts; Bash 3.2 rejects them:

$ a=(p q r)
$ echo "${a[-1]}"
bash: a: bad array subscript

Two portable forms, both measured:

$ echo "${a[@]: -1}"
r
$ echo "${a[${#a[@]}-1]}"
r

The space in ${a[@]: -1} is mandatory. Without it, :- is Chapter 8.2's default-value operator and the whole thing means something else entirely:

$ a=(p q r)
$ echo "with space:    [${a[@]: -1}]"
with space:    [r]
$ echo "without space: [${a[@]:-1}]"
without space: [p q r]

${a[@]:-1} reads as "a[@], or the literal 1 if it is unset" — and since it is set, you get the entire array. Silently, with no hint that you asked for the last element. That is the kind of detail that makes the second form, ugly as it is, the safer one to write.

3. Appending

$ tests=(smoke burst)
$ tests+=(random)
$ echo "${tests[*]}"
smoke burst random
$ tests+=("two words" more)
$ echo "${#tests[@]}"
5

+=( ... ) appends, and it takes as many items as you like. This is the workhorse of the module — the standard way to build a list is an empty array plus += in a loop:

logs=()
for f in ./*.log; do
  [ -f "$f" ] || continue
  logs+=("$f")
done

4. An Array Can Have Holes

This is the behaviour that breaks code written by people who know other languages.

$ a=(p q r s)
$ unset "a[1]"
$ echo "indices=${!a[@]}"
indices=0 2 3
$ echo "count=${#a[@]}"
count=3
$ echo "values=${a[*]}"
values=p r s

unset removes the element without renumbering. Index 1 is now unset, indices 2 and 3 keep their old numbers, and ${#a[@]} is 3 — the number of elements that are set, not the highest index plus one.

Those two facts together break the loop everybody writes first:

$ a=(p q r s); unset "a[1]"
$ for (( i=0; i<${#a[@]}; i++ )); do printf '[%s]' "${a[$i]}"; done; echo
[p][][r]

Three iterations over indices 0, 1, 2. Index 1 is the hole, so it expands to nothing and prints []. Index 3 holds s and is never visited at all. The loop silently dropped a real element and invented an empty one.

Two correct loops:

$ for v in "${a[@]}"; do printf '[%s]' "$v"; done; echo
[p][r][s]
$ for i in "${!a[@]}"; do printf '[%s]' "${a[$i]}"; done; echo
[p][r][s]

Iterate the values with "${a[@]}" when you only need the items — which is nearly always. Iterate the indices with "${!a[@]}" when you need the index too, for instance to walk two parallel arrays together (Chapter 19.6 uses exactly that).

Never use for (( i=0; i<${#a[@]}; i++ )) on an array. It is correct only for a dense array, and nothing in the language tells you whether yours is dense.

Holes also arrive without unset:

$ a=(p)
$ a[5]=z
$ echo "indices=${!a[@]}  count=${#a[@]}  values=[${a[*]}]"
indices=0 5  count=2  values=[p z]

Assigning to a high index creates the element and leaves everything between unset. There is no "length" in the sense other languages mean — an array is a set of index/value pairs that happens to be usually contiguous.

5. An Empty Array Can Abort Your Script

The second trap, and it is the one that bites in production because it only fires on the empty case:

$ bash -c 'set -u; a=(); printf "[%s]\n" "${a[@]}"; echo reached'
bash: a[@]: unbound variable

set -u treats "${a[@]}" on an empty array as an unset variable on Bash 3.2, and the script dies.

Not every form does, and the difference is worth knowing exactly:

$ bash -c 'set -u; a=(); echo "count=${#a[@]}"'
count=0
$ bash -c 'set -u; a=(); for i in "${!a[@]}"; do :; done; echo "indices loop survived"'
indices loop survived
$ bash -c 'set -u; a=(); for v in "${a[@]}"; do :; done; echo unreachable'
bash: a[@]: unbound variable

So the count and the index loop are safe and the value loop is not — which means a script can walk an empty array by index without noticing it never had the guard it needed for the value loop.

So a script that works all week dies the first day there are no logs to process — which is exactly the day you are least likely to be watching.

Three ways round it. Use argument count to tell them apart, because printing is misleading here (Chapter 15.4 measured that printf runs its format once even with zero arguments):

$ count() { echo "  got $# args"; }

$ bash -c 'count() { echo "  got $# args"; }; set -u; a=(); count ${a[@]+"${a[@]}"}'
  got 0 args

$ bash -c 'count() { echo "  got $# args"; }; set -u; a=(); count "${a[@]:-}"'
  got 1 args

${a[@]+"${a[@]}"} gives zero arguments — correct. It reads as "if a[@] is set at all, substitute "${a[@]}"", using Chapter 8.2's ${var+word} form, and the outer expansion is deliberately unquoted because the inner one carries the quotes.

"${a[@]:-}" gives one argument, an empty string — almost certainly wrong. It is the form people reach for because it looks like the familiar default, and it quietly passes an empty argument to whatever you were calling.

And on a non-empty array both are correct, which is why the bug hides:

$ bash -c 'count() { echo "  got $# args: $*"; }; set -u; a=(p "q r"); count ${a[@]+"${a[@]}"}; count "${a[@]:-}"'
  got 2 args: p q r
  got 2 args: p q r

The third way is the one to prefer in a script, because it is readable:

if [ "${#files[@]}" -gt 0 ]; then
  process "${files[@]}"
fi

A count test says what you mean, needs no obscure expansion form, and gives you somewhere to put the "nothing to do" message that Chapter 17.1 §5 argued for.

6. Inspecting an Array

declare -p prints an array's full definition, holes and all:

$ a=(x y)
$ declare -p a
declare -a a='([0]="x" [1]="y")'
$ unset "a[0]"
$ declare -p a
declare -a a='([1]="y")'
$ b=()
$ declare -p b
declare -a b='()'

That is the single most useful debugging line in this module. It shows the indices, which ${a[*]} hides, so it answers "is my array sparse?" and "is element 3 empty or missing?" — two questions you cannot distinguish any other way.

7. A Complete Script

Every form in this chapter, in a script that builds a work list and reports on it:

#!/usr/bin/env bash
# worklist.sh — build a list of logs, drop the unusable ones, report
# usage: worklist.sh <dir>
#
# exit: 0 all usable · 1 some were dropped · 2 bad usage · 3 nothing to do

set -o nounset
shopt -s nullglob          # an unmatched glob must give zero elements

note() { printf '%s: %s\n' "${0##*/}" "$*" >&2; }

dir=${1:-}
[ -n "$dir" ] || { note "usage: ${0##*/} <dir>"; exit 2; }
[ -d "$dir" ] || { note "$dir is not a directory"; exit 2; }

# collect — nullglob means this is empty rather than literal when nothing matches
declare -a candidates=("$dir"/*.log)

if [ "${#candidates[@]}" -eq 0 ]; then
  note "no logs in $dir"
  exit 3
fi

# filter into a second array rather than unsetting from the first,
# which would leave holes — §4
declare -a usable=()
declare -a dropped=()

for f in "${candidates[@]}"; do
  if   ! [ -f "$f" ]; then dropped+=("$f (not a regular file)")
  elif ! [ -r "$f" ]; then dropped+=("$f (not readable)")
  elif ! [ -s "$f" ]; then dropped+=("$f (empty)")
  else                     usable+=("$f")
  fi
done

printf 'found    %d\n' "${#candidates[@]}"
printf 'usable   %d\n' "${#usable[@]}"
printf 'dropped  %d\n' "${#dropped[@]}"

# index loop, because the position is part of the output
if [ "${#usable[@]}" -gt 0 ]; then
  echo "-- usable:"
  for i in "${!usable[@]}"; do
    printf '  %2d  %-28s %5s lines\n' \
      "$(( i + 1 ))" "${usable[$i]##*/}" "$(( $(wc -l < "${usable[$i]}") ))"
  done
  echo "-- last one is ${usable[${#usable[@]}-1]##*/}"
fi

# value loop, because only the message matters
if [ "${#dropped[@]}" -gt 0 ]; then
  echo "-- dropped:" >&2
  for d in "${dropped[@]}"; do
    printf '      %s\n' "$d" >&2
  done
fi

[ "${#dropped[@]}" -eq 0 ]

$ mkdir logs
$ printf 'a\nb\n'              > logs/clean.log
$ printf 'c\n'                 > 'logs/my report.log'
$ : >                            logs/empty.log
$ mkdir                          logs/adir.log
$ printf 'z\n' > logs/locked.log && chmod 000 logs/locked.log

$ ./worklist.sh logs
found    5
usable   2
dropped  3
-- usable:
   1  clean.log                        2 lines
   2  my report.log                    1 lines
-- last one is my report.log
-- dropped:
      logs/adir.log (not a regular file)
      logs/empty.log (empty)
      logs/locked.log (not readable)
$ echo $?
1

$ rm -rf logs/adir.log logs/empty.log logs/locked.log
$ ./worklist.sh logs
found    2
usable   2
dropped  0
-- usable:
   1  clean.log                        2 lines
   2  my report.log                    1 lines
-- last one is my report.log
$ echo $?
0

$ mkdir empty-dir && ./worklist.sh empty-dir
worklist.sh: no logs in empty-dir
$ echo $?
3

Seven decisions from this chapter:

shopt -s nullglob is on line 2. Delete it and run the same empty directory:

$ sed '/nullglob/d' worklist.sh > no-nullglob.sh && chmod +x no-nullglob.sh
$ ./no-nullglob.sh empty-dir
found    1
usable   0
dropped  1
-- dropped:
      empty-dir/*.log (not a regular file)
$ echo $?
1

It "found" one log, named empty-dir/*.log, dropped it as not a regular file, and exited 1. The exit 3 path — the one that says there is nothing to do — never fires, and the caller is told the run failed rather than that it was empty.

my report.log survived with its space — the whole point of the module, and nothing special was needed for it.

Two arrays instead of one with holes. Filtering by appending to usable and dropped keeps both dense; unset-ing rejects from candidates would have left exactly the sparse array §4 warns about.

Every expansion is count-guarded before it is used, so set -u cannot abort on an empty usable or dropped. That is §5, applied three times.

"${!usable[@]}" for the numbered list, "${dropped[@]}" for the messages. Indices where the position is part of the output, values where only the item matters.

${usable[${#usable[@]}-1]} for the last element, rather than [-1] which this shell rejects.

$(( $(wc -l < ...) )) strips BSD wc's padding, which Chapter 16.4 measured.

Practice

1. Create a=(one "two words" three) and print ${#a[@]}. Then create it with commas between the items and explain what you get.
2. Set a=(hello world) and print both ${#a[@]} and ${#a[0]}. State the rule in one sentence.
3. Run a=(*.nope); echo ${#a[@]} in a directory with no matches. Then do it again after shopt -s nullglob. Which count would break a script, and how?
4. Append to an array two ways: a+=(s) and a+=s. Print ${#a[@]} and ${a[*]} after each. Explain the second using Chapter 19.1 §6.
5. Build a=(p q r s), unset "a[1]", then print ${!a[@]}, ${#a[@]} and ${a[*]}. Predict each before you run it.
6. On that same sparse array, run all three loops from §4. Note which element the counting loop loses and which one it invents.
7. Run a=(p); a[5]=z; declare -p a. How many elements, and what are the indices?
8. Run bash -c 'set -u; a=(); printf "[%s]\n" "${a[@]}"'. Then try all three workarounds, measuring with a function that prints $# rather than with printf. Explain why printf is a bad instrument here.
9. Get the last element of a=(p q r) three ways: ${a[-1]}, ${a[@]: -1} and ${a[${#a[@]}-1]}. Then remove the space from the second one and explain the result.
10. In worklist.sh, delete shopt -s nullglob and run it on an empty directory. What does it report, and which exit path never fires?
11. In worklist.sh, replace the filter with unset on candidates and the index loop with a counting loop. Find an input where the output is wrong.

Check Your Understanding

1. What separates items in an array literal?
2. What is the difference between ${#a[@]} and ${#a[0]}?
3. a=(*.nope) in a directory with no matches. What is ${#a[@]}?
4. a=(p q r); a+=s. What is ${#a[@]} and why?
5. After unset "a[1]" on a four-element array, what are the indices and the count?
6. Why is for (( i=0; i<${#a[@]}; i++ )) wrong for an array?
7. Under set -u, what does "${a[@]}" do when a is empty, and what is the readable fix?

8. Remember

• a=(one "two words" three) — whitespace-separated, no commas, ordinary quoting rules per item.
• declare -a at script level and local -a in a function, for the same reason as local.
• A glob matching nothing gives one element holding the pattern. shopt -s nullglob when you build arrays from globs.
• ${#a[@]} is the count; ${#a[0]} is the length of element 0.
• a+=(x) appends to the array; a+=x appends to element 0. Always write the parentheses.
• unset leaves a hole. Indices are not renumbered and the count is of set elements.
• Never for (( i=0; i<${#a[@]}; i++ )). Use "${a[@]}" for values, "${!a[@]}" for indices.
• Assigning a high index creates a sparse array — there is no length, only index/value pairs.
• set -u plus an empty array aborts on "${a[@]}" on Bash 3.2. Guard with [ "${#a[@]}" -gt 0 ].
• "${a[@]:-}" is not the fix — it produces one empty argument. ${a[@]+"${a[@]}"} produces none.
• ${a[-1]} is a Bash 4.3 feature. Use ${a[@]: -1} with the space, or ${a[${#a[@]}-1]}.
• declare -p a shows the indices, which is the only way to see a hole.

Chapter 19.3 is the one expansion form that matters. You have now written "${a[@]}" a dozen times; the next chapter measures exactly what happens when you write any of the three alternatives instead, and why only one of them is ever a list.