Portability: bash vs sh, GNU vs BSD

Part I · Module 21 · Chapter 21.6 · Portability: bash vs sh, GNU vs BSD

"It works on my machine" has two distinct causes, and they need different answers.

The shell is different. Your #!/bin/bash script is run as sh, or runs on a system where /bin/sh is dash and something sources it. Half the syntax disappears.

The tools are different. The shell is fine and sed -i means something else, because GNU coreutils and BSD userland diverged decades ago and macOS ships the BSD side.

1. First, a Note on Measuring It

Before the tables, the method — because the first version of this chapter's probe was wrong in a way worth showing.

The obvious probe runs a snippet under dash and checks whether it failed:

probe() { printf '%-28s ' "$1"; dash -c "$2" >/dev/null 2>&1 && echo "OK" || echo "FAILS"; }
probe '$RANDOM'    'echo $RANDOM'
probe '+= append'  'v=a; v+=b; echo "$v"'

$RANDOM                      OK
+= append                    OK

Both wrong. dash has no $RANDOM, so echo $RANDOM printed an empty line — and exited 0, because echoing nothing is a success. v+=b likewise did not error; it did something else.

The probe has to check the value:

chk() {
  printf '  %-26s ' "$1"
  got=$(dash -c "$2" 2>/dev/null || echo '<<ERR>>')
  [ "$got" = "$3" ] && echo "sh: OK" \
    || printf 'sh: NO  (got [%s], wanted [%s])\n' "$got" "$3"
}
chk '$RANDOM'   'echo "${RANDOM:+set}"'   'set'
chk '+= append' 'v=a; v+=b; echo "$v"'    'ab'

  $RANDOM                    sh: NO  (got [], wanted [set])
  += append                  sh: NO  (got [a], wanted [ab])

v+=b gave a. Not an error, not ab — dash parsed it as something that left v unchanged. A status-only probe would have let that into a script.

That is the same lesson as Chapter 20.1's: a status tells you whether a command ran, not whether it did what you meant. A portability probe must assert the result. Everything below was measured that way.

2. bash vs sh

Measured against dash, which is what /bin/sh is on Debian and Ubuntu and is the strictest thing you are likely to meet:

Does not work under sh:

Works under sh:

Two things to read off that.

The expensive losses are arrays and [[ ]]. A #!/bin/sh script cannot hold a list correctly — Chapter 19.1's defect has no fix in POSIX shell beyond "$@" and set --. That alone is usually the reason to require Bash.

And the cheap wins are larger than people expect. getopts, trap EXIT, ${v:?}, read -r, printf and $(( )) are all portable, which means Chapters 20.3, 20.5 and 21.2 — validation, cleanup and option parsing — survive intact. The parts of this course that do not travel are narrower than "half of it".

3. Bash Versions

Separate from sh, and the one that bites on macOS. Everything in this course was measured on Bash 3.2.57, which is what Apple ships, and these are the features it does not have:

The case-conversion pair is worth separating, because the two forms fail differently — and only one of them is silent:

$ v=ab; echo "${v^^}"
bash: ${v^^}: bad substitution
$ echo "rc=$?"
rc=1

$ a=(ab cd); echo "${a[@]^^}"
ab cd
$ echo "rc=$?"
rc=0

The scalar form is a loud syntax error — unpleasant but findable. The array form returns the original values with no error and status 0, which is the only silent failure in the whole list. A script that upper-cases a list of tool names before comparing them will compare the wrong things on every Mac and nothing will say so.

Chapter 13.5's tr '[:lower:]' '[:upper:]' is the portable form for both.

And the version check is two lines, which Chapter 19.6 §2 argued is cheaper than discovering the problem in production:

if (( BASH_VERSINFO[0] < 4 )); then
  die "needs bash 4 or newer (found $BASH_VERSION)"
fi

$ echo "$BASH_VERSION  major=${BASH_VERSINFO[0]}"
3.2.57(1)-release  major=3

4. GNU vs BSD

The second cause, and the more common one in practice because the shell is usually fine. Each of these was run on this machine:

Two of those deserve a note.

sed -i is the one that corrupts data. GNU takes no argument; BSD requires one, and an empty string means "no backup":

$ sed -i 's/a/b/' f.txt
sed: 1: "f.txt

BSD read s/a/b/ as the backup suffix and f.txt as the script. On a bad day the arguments line up differently and you get a file renamed or a mangled edit rather than an error. sed -i '' 's/a/b/' f works on BSD and fails on GNU — there is no spelling that works on both, which is why portable scripts write to a temporary file instead:

tmp=$(mktemp) && sed 's/a/b/' f > "$tmp" && mv -- "$tmp" f

readlink -f works here, which contradicts the usual advice. It is worth stating precisely rather than repeating folklore: this macOS has it, older macOS did not, and the portable idiom remains the one Chapter 18.6 §4 used:

dir=$(cd -- "$(dirname -- "$1")" && pwd)

That needs no readlink at all and works everywhere, which is why it is still the right thing to write even on a system where readlink -f happens to exist.

And \w does work in BSD ERE:

$ echo abc1 | /usr/bin/grep -E '\w+'
abc1

But grep -P does not exist at all, so a PCRE pattern — lookahead, \d, non-greedy — has no BSD equivalent. Chapter 14.3's advice to stay in ERE is the portable one.

5. Deciding What to Target

Three positions, and the choice is about who runs the script rather than about purity.

#!/usr/bin/env bash, Bash 3.2 features only. The default for this course, and the right answer for anything that might touch a Mac. You keep arrays, [[ ]], (( )) and local; you give up declare -A, mapfile, ${v^^} and negative subscripts. Chapter 19.6's substitutes cover the gap.

#!/usr/bin/env bash with a version gate. Right when a Bash 4 feature genuinely pays for itself — a large associative array, say — and the gate turns "mysteriously wrong on a Mac" into a clear message:

(( BASH_VERSINFO[0] >= 4 )) || die "needs bash 4+ (found $BASH_VERSION)"

#!/bin/sh, POSIX only. Right for something that must run in a minimal container, in a Makefile recipe, or as an init script. Expensive: no arrays means Chapter 19.1's defect is unavoidable, so keep such scripts short and single-purpose.

And #!/usr/bin/env bash rather than #!/bin/bash, which Chapter 15.2 argued: on a Mac with Homebrew Bash 5 installed, env finds the newer one on PATH while the absolute path pins you to 3.2.

Two habits that cost nothing:

Say what you need, in the header. One line, and it saves the next person the investigation:

# requires: bash 4+ (associative arrays), GNU sed (-i without an argument)

And prefer the construct that works everywhere when it is not worse. printf over echo -e, $(cd … && pwd) over readlink -f, tr over ${v^^}, ERE over PCRE, a temporary file over sed -i. None of those is a sacrifice — several are better anyway.

6. A Complete Script

A portability probe you can run on a new machine before trusting your scripts to it:

#!/usr/bin/env bash
# probe-shell — report which constructs and tool options this host supports
#
# usage: probe-shell [-q]
#        -q  only report what is MISSING
#
# exit: 0 everything probed is available · 1 something is missing
#       2 bad invocation

set -o nounset

note()  { printf '%s: %s\n' "${0##*/}" "$*" >&2; }
usage() { printf 'usage: %s [-q]\n' "${0##*/}" >&2; }

quiet=0
while getopts ':q' opt; do
  case $opt in
    q)  quiet=1 ;;
    \?) note "unknown option: -$OPTARG"; usage; exit 2 ;;
  esac
done
shift $(( OPTIND - 1 ))
[ $# -eq 0 ] || { usage; exit 2; }

missing=0

# chk <label> <snippet> <expected>   — asserts the VALUE, not the status
chk() {
  local label=$1 snippet=$2 want=$3 got
  got=$(eval "$snippet" 2>/dev/null) || got='<<error>>'
  if [ "$got" = "$want" ]; then
    [ "$quiet" -eq 1 ] || printf '  ok      %s\n' "$label"
  else
    printf '  MISSING %-26s (got [%s], wanted [%s])\n' "$label" "$got" "$want"
    missing=$(( missing + 1 ))
  fi
}

printf 'shell: %s\n' "${BASH_VERSION:-not bash}"

printf -- '-- bash features\n'
chk 'arrays'              'a=(x y); echo "${a[1]}"'                 'y'
chk '[[ ]]'               '[[ -n x ]] && echo yes'                  'yes'
chk '(( ))'               '(( 1 < 2 )) && echo yes'                 'yes'
chk 'local'               'f() { local v=1; echo "$v"; }; f'        '1'
chk '+= append'           'v=a; v+=b; echo "$v"'                    'ab'
chk 'here-string'         'cat <<< hi'                              'hi'
chk 'process subst'       'read -r v < <(echo hi); echo "$v"'       'hi'
chk '${v//a/b}'           'v=aa; echo "${v//a/b}"'                  'bb'
chk 'pipefail'            '(set -o pipefail) && echo yes'           'yes'

printf -- '-- bash 4+ features\n'
chk 'declare -A'          'declare -A m 2>/dev/null && m[k]=v && echo "${m[k]}"'  'v'
chk 'mapfile'             'mapfile -t a < <(echo x) && echo "${a[0]}"'            'x'
chk 'case conv (scalar)'  'v=ab; echo "${v^^}"'                                   'AB'
chk 'case conv (array)'   'a=(ab); echo "${a[@]^^}"'                              'AB'
chk 'negative subscript'  'a=(p q); echo "${a[-1]}"'                              'q'

printf -- '-- tool options\n'
chk 'sed -i (GNU form)'   'printf "a\n" > /tmp/ps.$$; sed -i "s/a/b/" /tmp/ps.$$ && cat /tmp/ps.$$'  'b'
chk 'stat -c (GNU)'       'printf "ab" > /tmp/ps.$$; stat -c "%s" /tmp/ps.$$'     '2'
chk 'stat -f (BSD)'       'printf "ab" > /tmp/ps.$$; stat -f "%z" /tmp/ps.$$'     '2'
chk 'date -d (GNU)'       'date -u -d @0 "+%Y"'                                   '1970'
chk 'date -r (BSD)'       'date -u -r 0 "+%Y"'                                    '1970'
chk 'grep -P'             'echo abc | grep -P "a\w+"'                             'abc'
chk 'readlink -f'         'cd /tmp && ln -sf ps.$$ psl.$$; readlink -f psl.$$ >/dev/null && echo yes'  'yes'
chk 'seq'                 'seq 1 3 | tr "\n" " " | sed "s/ $//"'                  '1 2 3'

rm -f /tmp/ps.$$ /tmp/psl.$$

printf -- '-- %d missing\n' "$missing"
[ "$missing" -eq 0 ]

$ ./probe-shell -q
shell: 3.2.57(1)-release
-- bash features
-- bash 4+ features
  MISSING declare -A                 (got [<<error>>], wanted [v])
  MISSING mapfile                    (got [<<error>>], wanted [x])
  MISSING case conv (scalar)         (got [<<error>>], wanted [AB])
  MISSING case conv (array)          (got [ab], wanted [AB])
  MISSING negative subscript         (got [<<error>>], wanted [q])
-- tool options
  MISSING sed -i (GNU form)          (got [<<error>>], wanted [b])
  MISSING stat -c (GNU)              (got [<<error>>], wanted [2])
  MISSING date -d (GNU)              (got [<<error>>], wanted [1970])
  MISSING grep -P                    (got [<<error>>], wanted [abc])
-- 9 missing
$ echo $?
1

Nine findings, every one of them something this chapter claimed. And note the two case-conversion rows: the scalar form reports <<error>> while the array form reports got [ab] — the silent no-op from §3. A status-only probe would have flagged the first and passed the second, which is the whole argument of §1 appearing in the tool's own output.

$ ./probe-shell | head -12
shell: 3.2.57(1)-release
-- bash features
  ok      arrays
  ok      [[ ]]
  ok      (( ))
  ok      local
  ok      += append
  ok      here-string
  ok      process subst
  ok      ${v//a/b}
  ok      pipefail
-- bash 4+ features

Five decisions:

chk asserts the value and never the status. §1's whole point, and the case conversion row is the measurable payoff — a status-only probe reports it as working.

got='<<error>>' on failure so an error is distinguishable from an empty result. Without it, a missing feature and a feature returning nothing look the same, which is exactly the confusion §1 described.

Both the GNU and BSD spelling of stat and date are probed, and the output tells you which family you are on rather than asserting one is correct. A probe that only knew about GNU would report a healthy BSD box as broken.

-q prints only what is missing, which is what you want in a setup script: ./probe-shell -q || die "this host cannot run our tools".

And it exits 1 when anything is missing, so it is usable as a gate rather than only as a report — Chapter 21.4's contract applied to a diagnostic tool.

Practice

1. Write the status-only probe from §1 and run it for $RANDOM and v+=b under dash. Then write the value-asserting version and compare. Explain why the first one lied.
2. Run dash -c 'v=a; v+=b; echo "$v"' and note the value. Then explain why no error was reported.
3. Take the §2 "does not work" table and verify any five rows yourself against dash.
4. Take the "works" table and verify that getopts, trap EXIT and ${v:?} all work under dash. Which chapters does that make portable?
5. Write a library that requires Bash and source it from dash. Then add the [ -z "${BASH_VERSION:-}" ] guard and source it again.
6. Run echo "${a[@]^^}" on an array of lowercase words. If it returns them unchanged, say what that would do to a script comparing an upper-cased tool name.
7. Run sed -i 's/a/b/' f and then sed -i '' 's/a/b/' f on your machine. Which works? Then write the temporary-file form that works on both.
8. Run type -a grep sed stat on your machine and read the first line of each. Then run bash --noprofile --norc -c 'command -v grep sed stat' and compare. If the two disagree, you have found a shadow in your shell configuration — explain what that means for a script you share with a colleague, and for the measurements you took by hand.
9. If type -a grep shows a function or alias first, run grep -P 'a\w+' and /usr/bin/grep -P 'a\w+' on the same file and compare both the output and $?. Which one is a script going to get?
10. Replace a readlink -f in a script of your own with $(cd -- "$(dirname -- "$1")" && pwd). Confirm it still works, then argue for keeping the change even though readlink -f works here.
11. Run probe-shell -q on this machine and on any other you have access to. Diff the two outputs.
12. Add a probe for one more construct to probe-shell — and make sure it would report a silent failure, not just an error.

Check Your Understanding

1. What is wrong with a portability probe that checks only the exit status?
2. dash -c 'v=a; v+=b; echo "$v"' prints what?
3. Name the two most expensive things a #!/bin/sh script gives up.
4. Name three things from this course that are POSIX and so survive under sh.
5. Why does a shebang not protect a library?
6. Which Bash 4 feature fails silently on Bash 3.2, and what should you use instead?
7. Is there a spelling of sed -i that works on both GNU and BSD?

7. Remember

• The sharpest consequence is Chapter 33.3: the same line built two arguments at a zsh prompt and four in a bash script, and the prompt's result was the dangerous one because it ran.
• A portability probe must assert the value, not the status. echo $RANDOM under dash prints nothing and exits 0.
• v+=b under dash leaves v as a — silently, which is why the method matters more than the table.
• sh loses arrays, [[ ]], (( )), <<<, < <( ), source, ${v//a/b}, ${v^^}, declare, pipefail, $RANDOM, $SECONDS and $'\t'.
• sh keeps getopts, trap EXIT, ${v:?}, read -r, printf, $(( )) and ${v%s} — so validation, cleanup and option parsing are portable.
• Arrays are usually the reason to require Bash. Without them Chapter 19.1's defect is unavoidable.
• A shebang does not protect a sourced library. Test $BASH_VERSION and return 1.
• Bash 3.2 is what macOS ships: no declare -A, mapfile, lastpipe, ${a[-1]} or local -n.
• ${v^^} is a silent no-op on 3.2 — the only quiet failure in the list. Use tr.
• (( BASH_VERSINFO[0] >= 4 )) || die is two lines and converts a mystery into a message.
• GNU/BSD differences that bite: sed -i, stat -c vs -f, date -d vs -r, grep -P absent, wc padding.
• There is no portable sed -i. Write to a temporary file and mv.
• type -a the tool before blaming the system — a function, an alias or a Homebrew gnubin on PATH explains most one-machine bugs.
• Read the first line of type -a. It lists every match in resolution order; the later lines are what is being shadowed. Measured here: grep was a shell function dispatching to ugrep, which supports -P, so the portability answer inverted.
• command -v printing a bare name rather than a path means a function or builtin is in the way. bash --noprofile --norc -c 'command -v x' is the same machine without the startup files.
• State what you require in the header, and prefer the construct that works everywhere when it is not worse.

Chapter 21.7 turns the defects this course has catalogued into something a machine checks for you — and is honest about the fact that the standard tool for it is not installed on the machine everything here was measured on.