Validating Inputs and Required Variables

Part I · Module 20 · Chapter 20.3 · Validating Inputs and Required Variables

Chapter 20.2 ended on the gap: set -u is about unset, not empty, and dir="" sails straight through it. Chapter 20.1's shape four — an empty variable turning "$out/result.txt" into /result.txt — is still open.

1. State the Requirement, Don't Infer It

The problem with relying on set -u is not only the empty case. It is when and how it reports:

$ cat late.sh
#!/usr/bin/env bash
set -u
build_dir=$(mktemp -d)
cp -r src "$build_dir/"
cd "$build_dir"
make > "$LOGDIR/build.log"

$ ./late.sh
./late.sh: line 6: LOGDIR: unbound variable

Line 6. By then a temporary directory exists, a source tree has been copied into it, and the shell has changed directory. The script did five lines of work before discovering it could never have finished, and the message names an expansion rather than a requirement.

The alternative is one line, before anything happens:

$ cat early.sh
#!/usr/bin/env bash
set -euo pipefail
: "${LOGDIR:?must be set — export LOGDIR=/path/to/logs}"
build_dir=$(mktemp -d)
...

$ ./early.sh
./early.sh: line 3: LOGDIR: must be set — export LOGDIR=/path/to/logs
$ echo $?
1

Nothing was created, nothing was copied, and the message says what the script needs and how to give it to it.

2. The : "${VAR:?message}" Idiom

Two pieces, and both are worth understanding rather than copying.

${VAR:?message} is Chapter 8.2's family: if VAR is unset or empty, print message to stderr and exit a non-interactive shell. The shell supplies the prefix — script name, line number, variable name — and you supply the rest:

$ cat v1.sh
#!/usr/bin/env bash
set -euo pipefail
: "${PROJECT_ROOT:?must be set (export PROJECT_ROOT=/path/to/project)}"
: "${SIM:?must be set}"
echo "ok: PROJECT_ROOT=$PROJECT_ROOT SIM=$SIM"

$ ./v1.sh
./v1.sh: line 3: PROJECT_ROOT: must be set (export PROJECT_ROOT=/path/to/project)
$ echo $?
1

$ PROJECT_ROOT=/p ./v1.sh
./v1.sh: line 4: SIM: must be set
$ echo $?
1

$ PROJECT_ROOT=/p SIM=vcs ./v1.sh
ok: PROJECT_ROOT=/p SIM=vcs
$ echo $?
0

Note the second run: the first requirement passed and the next one reported. Requirements are checked in order and the first failure stops the script, so a user fixing them one at a time gets one clear message per attempt.

And empty is rejected too:

$ PROJECT_ROOT= SIM=vcs ./v1.sh
./v1.sh: line 3: PROJECT_ROOT: must be set (export PROJECT_ROOT=/path/to/project)
$ echo $?
1

That is the gap set -u left, closed. The colon is what does it:

$ bash -c 'dir=""; echo "[${dir?unset only}]"'
[]
$ bash -c 'dir=""; echo "[${dir:?empty or unset}]"'
bash: dir: empty or unset

Always write the colon. PROJECT_ROOT= is a far more common mistake than forgetting the variable entirely — it is what an unset variable in a wrapper script, a blank line in a config file, or a failed $(...) produces.

The leading : is the shell's do-nothing builtin. It exists so that you can have an expansion evaluated for its side effect without doing anything with the result:

$ type :
: is a shell builtin
$ bash -c ': "$(echo "the arg WAS expanded" >&2)"'
the arg WAS expanded

: ignores its arguments but the shell still expands them, which is exactly what you want: the ${VAR:?} either passes silently or kills the script, and nothing is printed either way on success. Writing echo "${VAR:?...}" > /dev/null would work and is noisier; [ -n "${VAR:?...}" ] would work and implies a test that is not happening.

3. Arguments: Count, Then Shape

:? is for the environment. Arguments need the same treatment and have more to check.

Count first, because every later check assumes the argument exists:

usage() { printf 'usage: %s <seeds> <outdir>\n' "${0##*/}" >&2; }
[ $# -eq 2 ] || { usage; exit 2; }

-eq rather than -ge: a script that silently ignores a third argument is a script that silently ignores your typo. Chapter 16.6's run-one.sh used -ge 1 and -le 2 precisely because its second argument was optional — be deliberate about which.

Then shape, one check per requirement, each naming the value it rejected:

seeds=$1
outdir=$2

[ "$seeds" -ge 1 ] 2>/dev/null || {
  printf '%s: seeds must be a positive integer, got: %s\n' "${0##*/}" "$seeds" >&2
  exit 2
}

case $outdir in
  /*) ;;
  *)  printf '%s: outdir must be absolute, got: %s\n' "${0##*/}" "$outdir" >&2
      exit 2 ;;
esac

$ ./v2.sh
usage: v2.sh <seeds> <outdir>
$ echo $?
2

$ ./v2.sh abc /tmp/x
v2.sh: seeds must be a positive integer, got: abc
$ echo $?
2

$ ./v2.sh 3 rel/path
v2.sh: outdir must be absolute, got: rel/path
$ echo $?
2

$ ./v2.sh 3 /tmp/x
ok: 3 seeds into /tmp/x
$ echo $?
0

Three things in that numeric check are deliberate.

[ "$seeds" -ge 1 ] tests numeric-ness and range at once. Chapter 16.3 measured that [ reports a non-numeric operand with status 2 and a message on stderr; 2>/dev/null discards the message and the || catches the status, turning the error into a clean validation. It is also why (( seeds >= 1 )) is the wrong tool here — Chapter 16.3 §6 measured that arithmetic silently treats abc as 0, which would pass a >= 0 test and fail a >= 1 one for the wrong reason.

The message quotes the value back. got: abc is the difference between a user fixing their command line in one attempt and reading the script to work out what it wanted.

Every failure is exit 2. Chapter 15.6's convention: 2 means you called it wrongly, which is distinct from 1 meaning the work failed. A caller can tell "fix your invocation" from "the build is broken".

4. What to Check, and What Not To

The useful question for each candidate check is what goes wrong without it, and would I rather find out here or later. A few that usually earn their place:

And some that usually do not:

Re-checking what you just created. mkdir -p "$d" || die is the check; a following [ -d "$d" ] adds nothing, because mkdir -p already told you.

Checking a file you are about to open. Chapter 16.4 §9's point: a test is a statement about a moment, and the window between it and the open is real. For a file you are going to read immediately, attempting the read and handling the failure is both shorter and more correct — n=$(wc -l < "$f") || die "cannot read $f" beats [ -r "$f" ] || die ... followed by an unguarded read.

Validating something the tool validates better. A simulator will reject a malformed +define+ with a better message than you can write. Pass it through.

The line to draw: validate the things that let your script do damage or waste time, and let the tools validate their own inputs.

5. A Complete Script

Everything checked, in order, before anything is created:

#!/usr/bin/env bash
# launch.sh — validate everything, then launch a regression
# usage: launch.sh <seeds> <outdir>
#
# environment: PROJECT_ROOT must be set and be a directory
#
# exit: 0 launched · 2 unusable invocation or environment
#       4 a required tool is missing

set -o errexit
set -o nounset
set -o pipefail

note()  { printf '%s: %s\n' "${0##*/}" "$*" >&2; }
die()   { note "$*"; exit 2; }
die4()  { note "$*"; exit 4; }
usage() { printf 'usage: %s <seeds> <outdir>\n' "${0##*/}" >&2; }
have()  { command -v "$1" > /dev/null 2>&1; }

# ---- 1. requirements -----------------------------------------------------

: "${PROJECT_ROOT:?must be set — export PROJECT_ROOT=/path/to/project}"

# ---- 2. arguments --------------------------------------------------------

[ $# -eq 2 ] || { usage; exit 2; }

seeds=$1
outdir=$2

[ "$seeds" -ge 1 ] 2>/dev/null \
  || die "seeds must be a positive integer, got: $seeds"
[ "$seeds" -le 100 ] 2>/dev/null \
  || die "seeds must be 100 or fewer, got: $seeds"

case $outdir in
  /*) ;;
  *)  die "outdir must be an absolute path, got: $outdir" ;;
esac

# ---- 3. the world --------------------------------------------------------

[ -d "$PROJECT_ROOT" ] || die "PROJECT_ROOT is not a directory: $PROJECT_ROOT"
[ -r "$PROJECT_ROOT" ] || die "PROJECT_ROOT is not readable: $PROJECT_ROOT"

testlist="$PROJECT_ROOT/tests.txt"
[ -f "$testlist" ] || die "no testlist at $testlist"
[ -s "$testlist" ] || die "testlist is empty: $testlist"

have printf || die4 "printf is not available — something is very wrong"

# ---- 4. only now does anything change ------------------------------------

mkdir -p -- "$outdir" || die "cannot create outdir: $outdir"
[ -w "$outdir" ]      || die "outdir is not writable: $outdir"

declare -a tests=()
while IFS= read -r line || [ -n "$line" ]; do
  case $line in ''|\#*) continue ;; esac
  tests+=("$line")
done < "$testlist"

[ "${#tests[@]}" -gt 0 ] || die "testlist has no tests: $testlist"

printf 'launching %d test(s) x %d seed(s) into %s\n' \
  "${#tests[@]}" "$seeds" "$outdir"
for t in "${tests[@]}"; do
  for (( s = 1; s <= seeds; s++ )); do
    printf '  %s seed %d\n' "$t" "$s"
  done
done

$ mkdir -p proj && printf '# the suite\n\nsmoke\ncache coherency\n' > proj/tests.txt

$ env -u PROJECT_ROOT ./launch.sh 2 /tmp/out
./launch.sh: line 22: PROJECT_ROOT: must be set — export PROJECT_ROOT=/path/to/project
$ echo $?
1

$ PROJECT_ROOT=$PWD/proj ./launch.sh
usage: launch.sh <seeds> <outdir>
$ echo $?
2

$ PROJECT_ROOT=$PWD/proj ./launch.sh abc /tmp/out
launch.sh: seeds must be a positive integer, got: abc
$ echo $?
2

$ PROJECT_ROOT=$PWD/proj ./launch.sh 500 /tmp/out
launch.sh: seeds must be 100 or fewer, got: 500
$ echo $?
2

$ PROJECT_ROOT=$PWD/proj ./launch.sh 2 relative/out
launch.sh: outdir must be an absolute path, got: relative/out
$ echo $?
2

$ PROJECT_ROOT=/nope ./launch.sh 2 /tmp/out
launch.sh: PROJECT_ROOT is not a directory: /nope
$ echo $?
2

$ PROJECT_ROOT=$PWD/proj ./launch.sh 2 /tmp/m20out
launching 2 test(s) x 2 seed(s) into /tmp/m20out
  smoke seed 1
  smoke seed 2
  cache coherency seed 1
  cache coherency seed 2
$ echo $?
0

Six decisions:

Four numbered sections, in dependency order. Requirements, then arguments, then the world, then changes. Nothing in section 4 can be reached by an invocation that section 1, 2 or 3 would have rejected.

mkdir -p is the first thing that changes the system, and it is on line 4-something of a forty-line script. Every rejected invocation leaves the filesystem exactly as it found it.

Two :?-style exits and a separate die4. A missing PROJECT_ROOT exits 1 (the shell's choice, via :?); a bad argument exits 2; a missing tool exits 4. Three categories a caller can act on differently.

[ "$seeds" -le 100 ] is an upper bound, which is the check people skip. launch.sh 100000 /tmp/out is a typo that would otherwise queue a hundred thousand jobs, and no later step would question it.

cache coherency survives with its space through the array, the loop and the output — Chapter 19.3, and the reason the testlist is read into an array rather than split.

[ "${#tests[@]}" -gt 0 ] after reading the file, even though [ -s "$testlist" ] already passed. Those are different claims: the file is non-empty, versus the file contains at least one line that is not a comment. A testlist of nothing but comments passes the first and fails the second — and Chapter 17.1 §5 is why running zero tests must not look like success.

Practice

1. Write late.sh from §1 with an unset LOGDIR and run it. Count what it created before failing. Then add the :? line and run it again.
2. Run a script with two :? requirements and nothing set. Fix the first, run again, and note which message you get. Why is that better than reporting both?
3. Test ${VAR:?msg} and ${VAR?msg} with the variable unset, then set to empty. Fill in the §2 table from your own output, then say which mistake is more common in practice.
4. Run bash -c ': "$(echo expanded >&2)"'. What does that prove about :?
5. Write the three-way argument validator from §3 and exercise all four paths. Confirm every failure is exit 2.
6. Replace [ "$seeds" -ge 1 ] 2>/dev/null with (( seeds >= 1 )) and pass abc. What happens, and which chapter measured why?
7. Remove the 2>/dev/null from the numeric check and pass abc. What extra line appears, and whose message is it?
8. In launch.sh, move mkdir -p -- "$outdir" to the top of the script. Then run it with a bad seeds value and look at /tmp. Describe the problem you just created.
9. Give launch.sh a testlist containing only comments and blank lines. Which check catches it, and why was [ -s ] not enough?
10. Add a check to launch.sh that the outdir is not inside PROJECT_ROOT. Decide whether it earns its place using §4's question.

Check Your Understanding

1. set -u reports an unbound variable on line 6. What is wrong with relying on that?
2. What does ${VAR:?message} do, and where does the message go?
3. What is the difference between ${VAR:?msg} and ${VAR?msg}?
4. Why the leading :?
5. Why [ "$n" -ge 1 ] 2>/dev/null rather than (( n >= 1 ))?
6. Why should every argument-validation failure exit 2 rather than 1?
7. Name two checks that usually do not earn their place.

6. Remember

• State requirements at the top, before anything is created. set -u fires at the point of use, which may be five lines of real work later.
• : "${VAR:?message}" is one line per requirement. The shell supplies the script name, line and variable; you supply the fix.
• Write the colon. ${VAR:?} rejects empty as well as unset; ${VAR?} lets an empty value through.
• : is a no-op that still expands its arguments — which is why it is the right carrier for the check.
• :? always exits 1. Write the check out longhand if you need exit 2.
• Check the argument count before anything that assumes an argument, and use -eq unless an argument really is optional.
• Quote the rejected value back in the message. got: abc saves a round trip.
• [ "$n" -ge 1 ] 2>/dev/null tests numeric-ness and range together. (( )) treats a word as 0.
• Bound the upper end too. 100000 seeds is a typo nothing downstream will question.
• Validate in dependency order: requirements, arguments, shape, world, then change things. A rejected invocation should leave nothing behind.
• Non-empty is not the same as having content. A testlist of comments passes -s and has no tests.

Chapter 20.4 takes the single worst consequence of an unvalidated value. An empty variable in a path is harmless until the command is rm -rf — at which point, measurably, the target becomes the root of the filesystem.