Linux & Bash · Part I · Module 19 · Arrays and Lists of Values

Arrays as Command Arguments

This is what the module has been building towards. A command line is a list of arguments, so assembling one in a string hits Chapter 19.1's defect in the place where it does the most damage — and the standard escape, evaluating the string, turns a quoting problem into a security problem. Both are measured here, including a value that executes.

Intermediate20 min readLinuxPart IBashArraysevalSecurityScripting

Part I · Module 19 · Chapter 19.5 · Arrays as Command Arguments

Every wrapper script eventually needs to build a command line — add -v when verbose, add a define when debugging, pass the output path the user chose. The obvious way to do that is to accumulate a string.

1. The Measurement

A stand-in tool that reports exactly what it received:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# tool.sh
echo "  tool got $# args:"
printf '    [%s]\n' "$@"

Four options, one of whose values contains a space — an output path under a directory called out dir:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ./tool.sh $(echo "-o out dir/run.log -D SIM=fast")
  tool got 5 args:
    [-o]
    [out]
    [dir/run.log]
    [-D]
    [SIM=fast]

Five. The path split in two, so -o was given the value out and dir/run.log became a stray positional argument. Most tools will either fail confusingly or write to a file called out.

Quote the string instead:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ opts="-o out dir/run.log -D SIM=fast"
$ ./tool.sh "$opts"
  tool got 1 args:
    [-o out dir/run.log -D SIM=fast]

One. The entire command line arrived as a single argument, which no tool will understand.

And the array:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ opts=(-o "out dir/run.log" -D "SIM=fast")
$ ./tool.sh "${opts[@]}"
  tool got 4 args:
    [-o]
    [out dir/run.log]
    [-D]
    [SIM=fast]

Four, correct. Chapter 19.1's three answers — too many, one, or right — in the place where being wrong is most expensive.

A command line built in a string, and in an array

A string has to be split to become arguments, and splitting cannot distinguish a space between two options from a space inside one option's value — so the usual escape is to have the shell evaluate the string, which will run whatever the data happens to contain. An array is already a list of arguments, so there is nothing to split and nothing to evaluate.

2. eval Works, and Is the Wrong Answer

Faced with the string version, the standard move is to put quotes inside the string and have the shell parse it again:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ opts="-o \"out dir/run.log\" -D SIM=fast"
$ eval ./tool.sh $opts
  tool got 4 args:
    [-o]
    [out dir/run.log]
    [-D]
    [SIM=fast]

Four arguments. It works. That is the problem — it works well enough that people ship it.

eval takes its arguments, joins them, and runs the result as shell source code. So the string is parsed a second time, which is how the embedded quotes come to mean something. And a second parse means every shell metacharacter in your data is now syntax:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ name="x; echo PWNED"
$ opts="-o $name"
$ eval ./tool.sh $opts
  tool got 2 args:
    [-o]
    [x]
PWNED

PWNED was printed by a command that is not in the script. The semicolon in $name ended the tool.sh command and started a new one, which eval duly ran.

Here $name was a literal in the transcript. In a real script it is a test name from a testlist, a path from $1, a value out of a config file, a branch name from CI — and anyone who can influence any of those can run commands as whoever runs the script.

The array version is simply immune, because there is no second parse:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ name="x; echo PWNED"
$ opts=(-o "$name")
$ ./tool.sh "${opts[@]}"
  tool got 2 args:
    [-o]
    [x; echo PWNED]

Two arguments, the second of which is the literal text including the semicolon. Nothing was executed, because nothing was ever re-parsed.

printf %q, and why you still do not need it

There is a way to make a string safe for eval: printf '%q' escapes a value so it survives a re-parse.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ name="x; echo PWNED"
$ printf '%q\n' "$name"
x\;\ echo\ PWNED

So opts="-o $(printf '%q' "$name")"; eval ./tool.sh $opts is correct. It is also longer than the array, requires you to remember %q at every single point where data enters the string, and fails silently if you forget once. The array has no such obligation. Know that %q exists — you will meet it in code that generates shell scripts — and do not build command lines this way.

3. Building Conditionally

The real use. Options accumulate as the script works out what it needs:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ opts=()
$ verbose=1
$ debug=1
$ [ "$verbose" -eq 1 ] && opts+=(-v)
$ [ "$debug"   -eq 1 ] && opts+=(+define+DEBUG "with space")
$ printf 'arg: [%s]\n' "${opts[@]}"
arg: [-v]
arg: [+define+DEBUG]
arg: [with space]
$ echo "count=${#opts[@]}"
count=3

Each += adds exactly as many arguments as you wrote, and the quoting at the append site is the only quoting there is. Compare the string version, where you must get the spacing right at every append and hope no value contains a space:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

opts=""
[ "$verbose" -eq 1 ] && opts="$opts -v"          # leading space? trailing?
[ "$debug" -eq 1 ]   && opts="$opts +define+DEBUG"

The array version also handles the empty case cleanly, which the string version does not — an empty $opts unquoted disappears, which happens to be right, but an empty "$opts" quoted becomes one empty argument, which is wrong. With an array:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ opts=()
$ ./tool.sh "${opts[@]}" input.sv
  tool got 1 args:
    [input.sv]

Zero options plus one file is one argument. Under set -u that line needs Chapter 19.2 §5's guard, which is the one wrinkle:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

if [ "${#opts[@]}" -gt 0 ]; then
  ./tool.sh "${opts[@]}" input.sv
else
  ./tool.sh input.sv
fi

Or, more neatly, put the fixed arguments in the array too so it is never empty:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

cmd=(./tool.sh)
[ "$verbose" -eq 1 ] && cmd+=(-v)
cmd+=(input.sv)
"${cmd[@]}"

4. The Whole Command in an Array

That last form generalises: there is no reason the command's name cannot be element zero.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ cmd=(printf '%s\n' "one two" three)
$ "${cmd[@]}"
one two
three

This is the idiom for a script that has to decide what to run as well as how:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

if [ "$dry_run" -eq 1 ]; then
  runner=(echo "WOULD RUN:")
else
  runner=()
fi
"${runner[@]}" vcs "${opts[@]}" "$top"

And it makes a dry-run mode almost free — which matters, because Chapter 14.7's "preview, then act" rule needs the preview to be the same command line that will run, not a re-rendering of it.

An array cannot hold a pipeline or a redirection

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ cmd=(ls / "|" head -3)
$ "${cmd[@]}"
ls: -3: No such file or directory
ls: head: No such file or directory
ls: |: No such file or directory

The | is an argument here, not a pipe, because by the time "${cmd[@]}" expands, the shell has already decided what the command's structure is. Expansion produces words; it cannot produce syntax.

The same applies to >, <, &&, ; and &. If your "command" needs any of those, it is not one command — write the pipeline out, or put it in a function and put the function's name in the array. Trying to smuggle shell syntax through a variable is the exact instinct that leads back to eval.

5. A Complete Script

A simulator wrapper that assembles everything it runs:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/usr/bin/env bash
# sim.sh — build and run a simulator command line
# usage: sim.sh [-v] [-d] [-n] [-o OUTDIR] <top> [test...]
#
# exit: the tool's status · 2 bad usage

set -o nounset

note() { printf '%s: %s\n' "${0##*/}" "$*" >&2; }
die()  { note "$*"; exit 2; }

verbose=0
debug=0
dry_run=0
outdir="out dir"          # a default with a space, on purpose

while [ $# -gt 0 ]; do
  case $1 in
    -v) verbose=1; shift ;;
    -d) debug=1;   shift ;;
    -n) dry_run=1; shift ;;
    -o) [ $# -ge 2 ] || die "-o needs a directory"; outdir=$2; shift 2 ;;
    --) shift; break ;;
    -*) die "unknown option: $1" ;;
    *)  break ;;
  esac
done

[ $# -ge 1 ] || die "usage: ${0##*/} [-v] [-d] [-n] [-o OUTDIR] <top> [test...]"

top=$1; shift

declare -a tests=("$@")

mkdir -p -- "$outdir" || die "cannot create $outdir"

# ---- assemble ------------------------------------------------------------

declare -a cmd=(./tool.sh)

[ "$verbose" -eq 1 ] && cmd+=(-v)
[ "$debug"   -eq 1 ] && cmd+=(+define+DEBUG)

cmd+=(-o "$outdir/run.log")
cmd+=(-top "$top")

# one -test flag per test, each value kept whole.
# the count guard is not decoration: see the note below.
if [ "${#tests[@]}" -gt 0 ]; then
  for t in "${tests[@]}"; do
    cmd+=(-test "$t")
  done
fi

# ---- preview or run ------------------------------------------------------

if [ "$dry_run" -eq 1 ]; then
  printf 'WOULD RUN (%d args):\n' "${#cmd[@]}"
  printf '  [%s]\n' "${cmd[@]}"
  exit 0
fi

"${cmd[@]}"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ./sim.sh -n -v -d top_tb smoke "cache coherency"
WOULD RUN (11 args):
  [./tool.sh]
  [-v]
  [+define+DEBUG]
  [-o]
  [out dir/run.log]
  [-top]
  [top_tb]
  [-test]
  [smoke]
  [-test]
  [cache coherency]

$ ./sim.sh -v top_tb smoke "cache coherency"
  tool got 9 args:
    [-v]
    [-o]
    [out dir/run.log]
    [-top]
    [top_tb]
    [-test]
    [smoke]
    [-test]
    [cache coherency]

$ ./sim.sh -n top_tb 'x; echo PWNED'
WOULD RUN (7 args):
  [./tool.sh]
  [-o]
  [out dir/run.log]
  [-top]
  [top_tb]
  [-test]
  [x; echo PWNED]

$ ./sim.sh
sim.sh: usage: sim.sh [-v] [-d] [-n] [-o OUTDIR] <top> [test...]
$ echo $?
2

$ ./sim.sh -o
sim.sh: -o needs a directory
$ echo $?
2

$ ./sim.sh -n top_tb
WOULD RUN (5 args):
  [./tool.sh]
  [-o]
  [out dir/run.log]
  [-top]
  [top_tb]
$ echo $?
0

That last case is a bug I shipped in this very script

The first draft of sim.sh had no count guard around the -test loop:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

for t in "${tests[@]}"; do cmd+=(-test "$t"); done

Run with a top module and no tests, under the set -o nounset on line 2:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ./sim.sh -n top_tb
./sim.sh: line 27: tests[@]: unbound variable
$ echo $?
1

Chapter 19.2 §5's trap, in the chapter after the one that warns about it. It is worth seeing why it slipped through: every test I ran had at least one test name, because that is the interesting case. The empty case is the boring one, which is exactly why nobody exercises it until a user does.

if [ "${#tests[@]}" -gt 0 ] is the fix, and the general habit is: wherever you expand an array you did not write out yourself, ask what happens when it is empty.

Now break one character — ${cmd[@]} without the quotes on the last line — and run the same command:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ./bad.sh -v top_tb "cache coherency"
  tool got 9 args:
    [-v]
    [-o]
    [out]
    [dir/run.log]
    [-top]
    [top_tb]
    [-test]
    [cache]
    [coherency]

Nine arguments both ways. The count is identical to the correct run, and the command line is completely different: -o now receives out, dir/run.log is a stray, -test receives cache, and coherency is another stray. An argument count is not a check — only looking at the boundaries is.

Five decisions:

out dir/run.log survives as one argument through the default value, the mkdir -p --, and the -o flag. Nothing anywhere had to escape it.

x; echo PWNED arrives as one literal argument and nothing runs. The dry run shows exactly that, which is the point of a dry run.

The dry run prints the array, not a reconstruction. printf ' [%s]\n' "${cmd[@]}" shows the real argument boundaries — a reconstructed string would hide precisely the mistake you are looking for. Chapter 14.7's preview rule, done properly.

cmd=(./tool.sh) starts non-empty, so "${cmd[@]}" can never trip set -u, and -n becomes four lines instead of a parallel code path.

One -test flag per test, appended in a loop. That is the shape a real tool wants and it is trivial with an array; building it in a string means tracking separators and quoting per iteration.

Practice

Write tool.sh and run the three forms from §1 — unquoted string, quoted string, array — with a value containing a space. Record the three argument counts.
Reproduce the eval injection with name="x; echo PWNED". Confirm PWNED prints. Then do the same with the array form and confirm it does not.
Explain in one sentence what eval does that makes the injection possible.
Escape the hostile value with printf '%q' and show the eval version is now correct. Then say why you would still not ship it.
Build an options array conditionally from three flags. Run it with all three on, then none on. Under set -u, what happens in the second case, and what are the two fixes?
Put a command's name in element zero and run "${cmd[@]}". Then add "|" and head to the array and run it again. Explain the error without using the word "quoting".
Add a -n dry-run mode to a wrapper of your own that prints "${cmd[@]}" one argument per line. Find a value for which this preview tells you something a joined string would have hidden.
In sim.sh, change "${cmd[@]}" to ${cmd[@]} and re-run with a two-word test. The argument count is unchanged at 9 — list what is actually different, and say what that tells you about using a count as a check.
Remove the [ "${#tests[@]}" -gt 0 ] guard from sim.sh and run ./sim.sh -n top_tb with no test names. Then explain why this case is the one that escapes testing.
In sim.sh, replace the array with a string and eval. Then run it with -o 'out; echo PWNED' and report what happens.

Check Your Understanding

Three forms, one option value containing a space. What are the three argument counts?
Why does eval make the string version work?
What else does that second parse enable?
Is the array form vulnerable to the same injection?
What is printf '%q' for, and why is it not the answer here?
Can an array element be a | or a >?
Why should a dry run print the array rather than a joined string?

Answers

1. Unquoted string: 5 — the value with the space splits. Quoted string: 1 — the whole command line becomes one argument. Array: 4 — correct. 2. Because eval runs its argument as shell source code, so the string is parsed a second time and the quotes you embedded in it finally mean something. 3. Every other shell metacharacter in the data: ;, &, |, $( ), backticks. A value containing ; echo PWNED ends your command and starts another, which eval runs — measured. 4. No. There is no second parse, so a semicolon in a value is just a character in an argument. 5. It escapes a value so it survives a re-parse, which does make the eval version correct. It is not the answer because you must remember it at every point data enters the string and it fails silently if you forget once — where the array has no such obligation. 6. Only as a literal argument. Expansion produces words, not syntax, so the command's structure was decided before "${cmd[@]}" expanded. Write the pipeline out or wrap it in a function. 7. Because the array shows the real argument boundaries, and a joined string hides exactly the mistake — a value that should be one argument and is about to be two — that you are previewing in order to catch.

6. Remember

A command line is a list, so building one in a string hits Chapter 19.1's defect where it costs most.
Measured: 5 arguments unquoted, 1 quoted, 4 correct with an array.
eval makes the string version work by parsing it twice — and the second parse turns every metacharacter in your data into syntax.
That is a command injection, measured. name="x; echo PWNED" ran echo PWNED.
If you are reaching for eval to build a command line, you want an array. No trade-off to weigh.
printf '%q' makes a string eval-safe and obliges you to remember it everywhere. Know it; do not build command lines with it.
Build conditionally with +=, quoting only at the append site.
Put the command name in element zero so the array is never empty and set -u cannot trip.
An array holds arguments, not syntax. A | in an element is a literal pipe character.
A dry run should print "${cmd[@]}" one element per line, because a joined string hides the boundary error you are looking for.

Chapter 19.6 closes the module with the other kind of array — keyed by name rather than number. It is also the only chapter in this course whose main feature cannot be run on the machine everything else was measured on, and it says so rather than pretending.

Where this fits

Part of the Linux & Bash curriculum.