Shell Options: set and shopt

Part I · Module 10 · Chapter 10.6 · Shell Options: set and shopt

set -u appeared in Chapter 8.1. shopt -s nullglob in Chapter 9.6. shopt -s expand_aliases and shopt -s extdebug in Chapter 10.5. And set -euo pipefail at the top of half the example scripts. This chapter is the explanation those uses have been borrowing against.

1. Two Commands, Two Lists

$ set -o | head -8
allexport      	off
braceexpand    	on
emacs          	off
errexit        	off
errtrace       	off
functrace      	off
hashall        	on
histexpand     	off

$ set -o | wc -l
27

$ shopt | head -6
cdable_vars    	off
cdspell        	off
checkhash      	off
checkwinsize   	off
cmdhist        	on
compat31       	off

$ shopt | wc -l
34

Twenty-seven options in one list and thirty-four in the other, and they do not overlap. This is not two views of the same thing.

The split is historical rather than principled:

• set controls options that come from the POSIX shell — the ones most shells have in common. Most have a single-letter form.
• shopt — short for shell options — controls Bash's own additions. These have names only.

There is no way to predict which list an option is in. You look it up, or you try one and then the other.

set also has a long form that avoids the problem entirely:

set -o nounset        # same as set -u
set +o nounset        # same as set +u

Prefer the long form in scripts. set -o nounset says what it means to a reader who has not memorised the letters; set -u does not.

2. The set Options Worth Knowing

Three of those have already earned their place in this course.

-u is the one that prevents disasters. Chapter 9.6 showed rm -rf ${BUILDDIR}/* becoming rm -rf /*; set -u turns that into an error before anything runs.

-x is the best debugging tool in Bash. Chapter 9.1 used it to show the finished command after every expansion stage:

$ d="my dir"
$ set -x
$ ls -d $d
+ ls -d my dir
$ ls -d "$d"
+ ls -d 'my dir'
$ set +x

You can also enable it for one run without editing anything:

bash -x ./script.sh

-C is worth knowing about because of how it fails:

$ echo one > nc.txt
$ set -o noclobber
$ echo two > nc.txt
bash: nc.txt: cannot overwrite existing file

$ echo three >| nc.txt
$ cat nc.txt
three

>| is the override — "yes, I do mean to overwrite." Useful at an interactive prompt where a mistyped > can destroy a file; less useful in scripts, where it tends to break things that deliberately rewrite a file.

3. What set -euo pipefail Switches On

You have seen this line repeatedly. It is three options:

set -euo pipefail

pipefail matters because by default a pipeline reports only its last command's status:

grep ERROR huge.log | head -5

Without pipefail, that succeeds whenever head succeeds — which is always — even if grep could not open the file. (Module 12 covers pipelines and their status.)

The three together are a reasonable default for a new script. They are not a substitute for checking things, and the -e part specifically will not catch everything you expect.

4. The shopt Options Worth Knowing

Three of these are worth a demonstration.

globstar gives you a recursive pattern:

shopt -s globstar
ls **/*.sv                 # every .sv file at any depth
grep -l ERROR **/*.log     # every log, anywhere below here

Without it, ** behaves exactly like * — one level only, silently. This is a Bash 4 feature, so it is absent on macOS's Bash 3.2 along with the brace forms from Chapter 9.5.

dotglob changes what * means:

$ ls -a
.  ..  .hidden  visible.txt

$ echo *
visible.txt

$ shopt -s dotglob
$ echo *
.hidden visible.txt

Useful for a script that must copy a directory completely. Dangerous in a script that then deletes what it matched — rm * with dotglob on reaches configuration files you did not mean to include.

nullglob and failglob are Chapter 9.6's, and the reason that chapter's loop ran once over a literal *.log.

5. Reading the State

set -o                      # every set option, with on/off
shopt                       # every shopt option, with on/off
shopt -s                    # only the ones that are ON
shopt -u                    # only the ones that are OFF

shopt -s with no name is a genuinely useful one-liner: it tells you what somebody has customised, without you reading past thirty defaults.

And for a single option, quietly, in a script:

shopt -q nullglob && echo "nullglob is on"
set -o | grep -q 'errexit.*on' && echo "errexit is on"

shopt -q sets the exit status and prints nothing, which is what a conditional wants.

$- is the short answer

$ bash -c 'echo "non-interactive: $-"'
non-interactive: hBc

$ bash -ic 'echo "interactive:     $-"'
interactive:     hiBHc

$- holds the single-letter options currently enabled. The i in the second line is the interactivity test Chapter 10.3 used:

case "$-" in *i*) echo interactive ;; *) echo not ;; esac

It is also the quickest way to confirm that -x or -e is on somewhere you did not expect.

6. Scope: Options Are Per-Shell

An option set in your shell applies to that shell, and like everything else in Chapter 10.1, it is not inherited by child processes:

$ shopt -s nullglob
$ bash -c 'shopt nullglob'
nullglob       	off

Three consequences:

• A script must set its own options. It cannot rely on yours, and should not.
• An option set interactively lasts until that shell exits. Which is why Chapter 9.6 suggested experimenting inside ( … ) — a subshell takes a copy and discards it.
• Options you want every time belong in ~/.bashrc (Chapter 10.3), because that is the file every interactive shell reads.

A reasonable interactive set:

# ~/.bashrc
shopt -s histappend      # do not lose history from other terminals
shopt -s checkwinsize    # keep LINES and COLUMNS right after a resize
shopt -s cdspell         # tolerate a typo in a cd
shopt -s globstar        # ** across directories (Bash 4+)

Note that set -e and set -u are deliberately absent. They are right for scripts and actively unpleasant at a prompt — set -u closes your terminal on a typo in a variable name, and set -e closes it whenever any command returns non-zero, which at a prompt includes a grep that found nothing.

7. The Module, Assembled

A script header using everything from Module 10:

#!/bin/bash
set -o errexit
set -o nounset
set -o pipefail
shopt -s nullglob

# 10.4 — find our own directory, so sourcing a sibling is reliable
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)

# 10.4 — the tool environment must be SOURCED, never executed
. "$script_dir/tool_env.sh"

# 10.1 — required from outside, named if missing
run_dir="${RUN_DIR:?set RUN_DIR to the directory to process}"

# 10.2 — prepend, build on the old value, and do not duplicate
case ":$PATH:" in
  *":$script_dir/bin:"*) ;;
  *) export PATH="$script_dir/bin:$PATH" ;;
esac

# 10.2 — prove which tool we are about to use, and record it
echo "simulator : $(command -v vcs || echo 'NOT FOUND')"
echo "path head : $(echo "$PATH" | cut -d: -f1-2)"

# 9.6 + nullglob — zero iterations when there is nothing to do
for log in "$run_dir"/*.log; do
  echo "processing ${log##*/}"
done

Every line is a chapter: set/shopt from here, sourcing and BASH_SOURCE from 10.4, PATH hygiene from 10.2, the required-variable form from 10.1 and 8.2, command -v from 10.5, and the quoted-variable-bare-pattern split from 9.6.

Practice

1 — Compare the two lists.

set -o | wc -l
shopt | wc -l
comm -12 <(set -o | awk '{print $1}' | sort) <(shopt | awk '{print $1}' | sort)

The third command prints nothing — there is no overlap.

2 — Feel the opposite syntaxes.

( set -u;  set -o | grep nounset )
( set +u;  set -o | grep nounset )
( shopt -s nullglob; shopt nullglob )
( shopt -u nullglob; shopt nullglob )

3 — Watch -u catch a typo.

( echo "without -u: [${TYPOED_NAME}]" )
( set -u; echo "with -u: [${TYPOED_NAME}]" )

4 — Use -x on a real script.

printf 'd="my dir"\nls -d $d\nls -d "$d"\n' > /tmp/t.sh
bash -x /tmp/t.sh 2>&1 | head

5 — Syntax-check without running.

printf 'if [ 1 -eq 1 ]\n  echo broken\nfi\n' > /tmp/bad.sh
bash -n /tmp/bad.sh

6 — Try noclobber.

cd /tmp && echo one > nc.txt
( set -o noclobber; echo two > nc.txt )
( set -o noclobber; echo three >| nc.txt && cat nc.txt )
rm -f nc.txt

7 — See dotglob change what * means.

mkdir -p /tmp/dg && cd /tmp/dg && touch .hidden visible.txt
echo *
( shopt -s dotglob; echo * )
cd /tmp && rm -rf /tmp/dg

8 — Try globstar, if your Bash is new enough.

bash --version | head -1
( shopt -s globstar 2>/dev/null && echo "globstar available" || echo "needs Bash 4" )

9 — Read your own state.

echo "flags: $-"
shopt -s | head

10 — Save and restore.

old=$(shopt -p nullglob); echo "saved as: $old"
shopt -s nullglob; shopt nullglob
eval "$old"; shopt nullglob

Check Your Understanding

1. Why are there two commands, and how do you know which list an option is in?

2. What does set +u do?

3. What are the three options in set -euo pipefail?

4. Why would you put shopt -s histappend in .bashrc but not set -u?

5. You set shopt -s nullglob and a script you run still gets the literal pattern. Why?

6. How do you restore an option to whatever it was before you changed it?

8. Remember

• Two commands, two non-overlapping lists. set holds the POSIX options, shopt holds Bash's own.
• The syntaxes are opposite. set -u enables and set +u disables; shopt -s sets and shopt -u unsets.
• Prefer the long form in scripts: set -o nounset reads, set -u does not.
• -u prevents the empty-variable disasters of Chapter 9.6.
• -x prints each command after expansion and is the best debugging tool Bash has; bash -x script.sh needs no edit.
• -n parses without running — a free syntax check.
• -C makes > refuse to overwrite, and >| overrides it.
• set -euo pipefail is errexit, nounset and pipefail, and -e is weaker than it looks — Chapter 20.2.
• nullglob, failglob, globstar, dotglob, extglob are the shopt options that change what a pattern means.
• globstar makes ** recursive and needs Bash 4.
• dotglob makes * match hidden files — useful for copying, dangerous for deleting.
• shopt -s with no name lists what is switched on — the fastest read of somebody's customisation.
• $- holds the single-letter flags, and i among them means interactive.
• Options are per-shell and not inherited. A script sets its own; experiment inside ( … ).
• shopt -p name prints the restoring command, which is how you put an option back.

That completes Module 10. You can now say where any setting lives, which process will see it, which file established it, why running a script achieves nothing and sourcing it achieves everything, what a name resolves to before the filesystem is consulted, and what every switch at the top of a script actually turns on.

Module 11 turns to the thing every command quietly returns and almost nobody looks at: its exit status. It is the basis of &&, of if, of set -e, and of knowing whether the regression that just finished actually passed.