Building a Tool Command Line Safely

Part II · Module 27 · Chapter 27.2 · Building a Tool Command Line Safely

A simulator command line starts at four arguments and ends at thirty. Some are constant, some come from a variable, some are present only when a flag was given, and several contain paths you do not control. Chapter 27.1 got the stages separated; this chapter is about what you hand them, and it is a quoting problem wearing a tool-flow costume.

1. Why a String Is the Wrong Container

The instinct is to build the command as text:

# the shape almost everybody writes first
cmd="iverilog -o $out -f $flist"
[ "$dump" = 1 ] && cmd="$cmd +define+DUMP"
$cmd

This works until one of those values contains a space, and then it does not fail — it does something else. The reason is the order of operations from Chapter 9.1: the shell expands $cmd and then splits the result on whitespace, so a path with a space becomes two arguments and the tool receives a filename that does not exist plus an argument it does not recognise.

An array holds the arguments as arguments, with no round trip through text:

args=(iverilog -o "$out" -f "$flist")
[ "$dump" = 1 ] && args+=(+define+DUMP)
"${args[@]}"

Each element is one argument for its whole life. "${args[@]}" expands to exactly as many words as there are elements, regardless of what is inside them. That is the whole technique, and the rest of this chapter is about the three ways it goes wrong.

2. Four Expansions, Measured by Counting Arguments

The honest test of an expansion is not what it prints but how many arguments the callee received, so the probe is a function that reports $#:

#!/bin/bash
set -u
count() { echo "argc=$#$( [ $# -gt 0 ] && printf ' argv=[%s]' "$@" )"; }
e=()          # empty
n=(a b)       # filled

Run against both arrays on this machine — bash 3.2.57, which is the version that ships with macOS and is still what a great many engineering servers run:

Three things in that table deserve separate attention.

The first row is a real trap, and it is version-specific. Under set -u, expanding an empty array with "${a[@]}" is an unbound-variable error in bash 3.2. The script dies on a line that looks unimpeachable. Newer bash (4.4 and later) treats it as zero words and does not complain — so this defect appears only on the older machine, which is usually the server rather than the laptop.

The second row is worse than the first. "${a[@]:-}" is the fix people reach for, because it makes the error go away. It does so by substituting one empty argument. The script survives and the tool now receives an argument you never intended, and nothing says so.

The third row is the answer. ${a[@]+"${a[@]}"} reads as "if the array is set, expand it quoted; otherwise expand to nothing", and it is the only form that is both safe under set -u and correct for an empty array. It also preserves spaces, which is the thing the whole exercise is for:

$ sp=(-o "my dir/sim.vvp")
$ count ${sp[@]+"${sp[@]}"}
argc=2 argv=[-o] argv=[my dir/sim.vvp]

3. What a Tool Does With an Empty Argument

The second row above injects one empty argument. Whether that is harmless depends entirely on the tool, so it is worth measuring rather than assuming:

$ iverilog -o /tmp/e.vvp -f filelists/sim.f ""
$ echo $?
0                                   # accepted silently
$ vvp build.vvp +TESTNAME=smoke ""
TB  TEST PASSED  test=smoke checks=3 errors=0
$ echo $?
0                                   # accepted silently
$ nvc --work=/tmp/w_e -a vhdl/arb.vhd ""
** Fatal: opening : No such file or directory
$ echo $?
1                                   # rejected

Two of the three tools took the empty argument without comment. One rejected it, and its message — opening : No such file or directory, with nothing between the colon and the word No — is exactly the message you will spend twenty minutes failing to understand, because the filename it is complaining about is invisible.

So the :- idiom produces a defect whose symptom depends on which tool you are driving: silence from two, and a cryptic message from the third. That is the worst possible distribution, because the silence trains you to believe the idiom is fine.

4. The Reverse Mistake, and Why It Is Quiet

The array protects you from under-splitting. The matching error is over-collapsing: putting several arguments into one. In bash the way to do that accidentally is [*] instead of [@].

$ args=(+TESTNAME=random +SEED=7 +CYCLES=20)
$ vvp build.vvp "${args[@]}"
TB  test=random seed=7 cycles=20 verbosity=LOW
TB  TEST PASSED  test=random checks=6 errors=0

$ vvp build.vvp "${args[*]}"
TB  test=random +SEED=7 +CYCLES=20 seed=1 cycles=40 verbosity=LOW
TB  TEST FAILED  test=random +SEED=7 +CYCLES=20 checks=0 errors=1

Read the second one carefully, because it is the most instructive failure in this chapter.

"${args[*]}" joined the three elements into one argument. The testbench reads its test name with $value$plusargs("TESTNAME=%s", testname), and %s in a plusarg match is greedy to the end of the argument — so testname became the string random +SEED=7 +CYCLES=20. That matched no known test, so the testbench ran nothing: checks=0. The seed and cycle count were never seen either, so they fell back to their defaults, which the TB line reports as seed=1 cycles=40 — printed right next to the test name that swallowed them.

And the exit status was 0.

5. A Builder You Can Inspect Before You Trust It

Two habits make the array approach reliable in practice: build the arguments in one place, and be able to print what you are about to run without running it.

#!/bin/bash
# simargs — build a simulator command line in an array, then show it or run it.
# Every value that could contain a space stays one array element end to end.
set -u
root=$(cd "$(dirname "$0")/.." && pwd)
test=smoke seed=1 dump=0 depth=0 dryrun=0 out=""
while getopts "t:S:Dd:o:n" o; do
  case "$o" in
    (t) test=$OPTARG ;; (S) seed=$OPTARG ;; (D) dump=1 ;;
    (d) depth=$OPTARG ;; (o) out=$OPTARG ;; (n) dryrun=1 ;;
    (*) echo "simargs: bad option" >&2; exit 2 ;;
  esac
done

cargs=(-o "${out:-$root/build.vvp}" -f "$root/filelists/sim.f")
rargs=(+TESTNAME="$test" +SEED="$seed" +STRICT)
if [ "$dump" = 1 ]; then
  rargs+=(+DUMP +DUMP_DEPTH="$depth" +DUMP_FILE="dump.vcd")
fi

if [ "$dryrun" = 1 ]; then
  # printf %q is for the human; the array is what the tool gets
  printf 'compile: iverilog '; printf '%q ' "${cargs[@]}"; printf '\n'
  printf 'run    : vvp %q ' "${out:-$root/build.vvp}"
  printf '%q ' ${rargs[@]+"${rargs[@]}"}; printf '\n'
  printf 'argc   : compile %d, run %d\n' "${#cargs[@]}" "${#rargs[@]}"
  exit 0
fi
iverilog "${cargs[@]}" || exit $?
exec vvp "${out:-$root/build.vvp}" ${rargs[@]+"${rargs[@]}"}

$ simargs -n -t fairness -S 9 -D
compile: iverilog -o /…/m27/build.vvp -f /…/m27/filelists/sim.f
run    : vvp /…/m27/build.vvp +TESTNAME=fairness +SEED=9 +STRICT +DUMP +DUMP_DEPTH=0 +DUMP_FILE=dump.vcd
argc   : compile 4, run 6

$ simargs -n -o "/tmp/my dir/sim.vvp" -t smoke
compile: iverilog -o /tmp/my\ dir/sim.vvp -f /…/m27/filelists/sim.f
run    : vvp /tmp/my\ dir/sim.vvp +TESTNAME=smoke +SEED=1 +STRICT
argc   : compile 4, run 3

Four details in that script are load-bearing.

printf '%q ' for display, the array for execution. %q prints a form you can paste back into a shell — notice my\ dir — so the dry run is both readable and unambiguous about where an argument ends. It is a rendering of the arguments, not the arguments themselves, and it is important not to confuse the two: Chapter 27.9 shows what happens when a flow tries to replay from the %q line instead of from the array.

argc is printed. The count is the only thing that distinguishes correct splitting from either failure mode, so the dry run reports it.

The parenthesised case patterns. (t) rather than t). On bash 3.2 a case arm inside a command substitution or a process substitution fails to parse without the opening parenthesis — a fact from Chapter 25.1 — and writing them this way everywhere costs nothing and removes the failure mode entirely. Note the matching discipline: do not then write ((t), which is a syntax error that makes every invocation return 2.

exec vvp. The wrapper has nothing left to do, so it replaces itself with the tool. The benefit is not speed: it means the tool's exit status is the script's exit status with no code in between that could lose it, and it means a signal sent to the script reaches the tool. Chapter 27.7 measures what happens when exec is missing and a kill lands on the wrapper instead.

6. Conditional Arguments Without Conditional Text

The reason a command line grows is that most of its arguments are optional. Three patterns cover almost everything, and all three keep the array shape:

# 1. a flag that is simply present or absent
[ "$dump" = 1 ] && rargs+=(+DUMP)

# 2. an option with a value, only when the value exists
[ -n "${seed:-}" ] && rargs+=(+SEED="$seed")

# 3. a repeated option over a list — one element per item, never one joined string
for d in "${incdirs[@]}"; do cargs+=(-I"$d"); done

The third is the one to be careful about. Tools differ on whether an option and its value are one argument (-I/path) or two (-I /path), and the array makes the choice explicit rather than accidental. Build it the way the tool documents, and if you are unsure, use the dry run to look at argc — the difference between four arguments and five is visible there and invisible in a log.

Two further guards worth having in any builder:

# reject an option value that is empty rather than passing an empty argument
: "${test:?simargs: -t requires a test name}"

# refuse to build a command containing a newline, which no tool handles and
# which turns a one-line log entry into two
case "$test" in (*$'\n'*) echo "simargs: test name contains a newline" >&2; exit 2 ;; esac

The first uses the ${var:?message} form from Chapter 8.2, which fails loudly at the point of use. The second exists because a value with a newline in it is the one case where even the array technique produces something a log cannot represent faithfully.

Practice

1. Write the count probe and run all four expansions against an empty and a filled array. Record argc for each of the eight cases.
2. Confirm the version dependence. Run the "${e[@]}" case under set -u and note the error; then check your bash version with echo "$BASH_VERSION".
3. Show that :- injects an argument rather than fixing anything, by reading the argc it produces.
4. Feed one empty argument to three different tools and record which accept it silently and which complain.
5. Build an array with a path containing a space and prove with count that it arrives as one argument.
6. Reproduce the [*] failure. Run a simulation with "${args[*]}" and find checks=0 in the output. Confirm the exit status.
7. Then explain the seed=1 cycles=40 that appears in the same line as the swallowed arguments.
8. Write simargs with a dry-run mode that prints argc, and use it to tell a four-argument compile from a five-argument one.
9. Add the two guards — a ${var:?} for a required value, and the newline rejection — and prove each one fires.
10. Remove exec from the last line, run the script in the background, kill it, and see whether the tool is still running.
11. Convert one real invocation from your own work into an array, dry-run it, and compare argc against the number of arguments you intended.

Check Your Understanding

1. Why does building a command as a string break on a path containing a space, and at which step?
2. What does "${a[@]}" do with an empty array under set -u in bash 3.2, and in bash 4.4?
3. Why is "${a[@]:-}" worse than the error it suppresses?
4. Write the expansion that is correct for both an empty and a filled array.
5. Two of three tools accepted an empty argument. Why does that make the :- idiom more dangerous rather than less?
6. What did "${args[*]}" make the testbench's test name, and why?
7. What was checks in that run, and what was the exit status?
8. What is printf '%q' for, and what is it not for?
9. Why does a dry run print the argument count?
10. What two things does exec preserve that a trailing plain invocation does not?

7. Remember

• Build arguments in an array, never in a string. A string is re-split after expansion; an array is not.
• "${a[@]}" on an empty array dies under set -u in bash 3.2 and is silent in bash 4.4 — so this defect appears on servers and not on laptops.
• "${a[@]:-}" is not the fix: it passes one empty argument. ${a[@]+"${a[@]}"} is the fix.
• An empty argument is accepted silently by some tools and rejected with an unreadable message by others.
• "${args[*]}" collapses the list into one argument, and a greedy %s plusarg match then swallows the rest of it: checks=0, exit status 0.
• checks=0 is the anomaly to watch for — a run that completed and examined nothing.
• Print the command with printf '%q ' and the argument count, and keep the printed form strictly for humans.
• End the wrapper with exec so the tool owns the exit status and the signals.
• A tool script generated by the shell is a command line too. Chapter 35.5 measures what one forgotten backslash does to it: a report stage that globbed the root directory, found nothing, and exited 0.

8. What Is Next

The command is assembled correctly. Chapter 27.3 decides where it runs, which turns out to be the difference between four usable waveform databases and one corrupt file that matches none of them.