Splitting a Script, and Function Libraries You Source

Part I · Module 18 · Chapter 18.6 · Splitting a Script, and Function Libraries You Source

Five chapters of functions, and every one of them lived inside the script that used it. Three scripts needing the same note, die and have is the point at which that stops being reasonable.

1. Sourcing Runs It Here; Executing Runs It There

Chapter 10.4 made this distinction for setup scripts. It is the whole mechanism for libraries:

# lib.sh
LIB_VERSION=1.2
log()  { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
die()  { log "FATAL: $*"; exit 1; }

#!/usr/bin/env bash
# use.sh
source ./lib.sh
echo "LIB_VERSION=$LIB_VERSION"
type log | head -1

$ ./use.sh
LIB_VERSION=1.2
log is a function

Now execute the library instead of sourcing it:

#!/usr/bin/env bash
# use2.sh
./lib.sh
echo "LIB_VERSION=[$LIB_VERSION]"
type log 2>&1 | head -1

$ ./use2.sh
LIB_VERSION=[]
log is /usr/bin/log

Nothing was defined, and log resolved to Apple's logging tool. Executing the library started a child shell, which dutifully set LIB_VERSION and defined log and then exited, taking both with it — Chapter 17.7's rule yet again. The parent got nothing.

And then type log found /usr/bin/log, the real 800-kilobyte system command from Chapter 18.1's callout. That is what a failed library load looks like in practice: not an error, but a script that quietly calls something else entirely.

2. It Shares Everything, and Isolates Nothing

This is the fact to carry away. A sourced file is your script's code, running in your shell, from line one. So it can do anything your script can do — including things you did not ask for:

# bad-lib.sh — every line of this is a mistake
cd /tmp
shopt -s nullglob
set -u
x=CLOBBERED

#!/usr/bin/env bash
x=mine
echo "before: pwd=$PWD x=$x nullglob=$(shopt -q nullglob && echo on || echo off)"
source ./bad-lib.sh
echo "after:  pwd=$PWD x=$x nullglob=$(shopt -q nullglob && echo on || echo off)"

$ ./u3.sh
before: pwd=/home/engineer/work x=mine nullglob=off
after:  pwd=/tmp x=CLOBBERED nullglob=on

Working directory changed, a variable overwritten, a shell option flipped. None of it announced. A real library would not cd /tmp on purpose, but it is entirely plausible for one to set x as a module-level default, or to turn on nullglob because its own globbing needs it — and both of those reach into every script that sources it.

Compare what other languages give you and what Bash does not:

Every "no" in that column is something you have to replace with a convention.

3. The Conventions That Replace a Module System

Prefix everything. There is one namespace, so a library's names must not be plausible names for anything else:

# lib.sh
VM_LIB_VERSION=1.2                 # not LIB_VERSION
vm_log()  { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
vm_die()  { vm_log "FATAL: $*"; exit 1; }
vm_have() { command -v "$1" > /dev/null 2>&1; }

Verbose, and it is the single highest-value habit for shared shell code. It removes the entire class of problem where two sourced libraries both define log, and the second silently wins (Chapter 18.1 §4 measured that redefinition is silent). It also removes the /usr/bin/log collision.

Guard against being sourced twice. Chapter 18.3 §5 measured that return at the top of a sourced file hands control back to the caller:

# lib.sh
[ -n "${_VM_LIB_LOADED:-}" ] && return 0
_VM_LIB_LOADED=1

# ... definitions ...

$ cat u7.sh
#!/usr/bin/env bash
source ./guarded.sh
source ./guarded.sh
source ./guarded.sh
echo "  done"

$ ./u7.sh
  guarded.sh body ran
  done

Three source lines, one execution of the body. Without the guard, a library sourced by both your script and another library it loads would run twice — harmless for function definitions, not harmless for anything that appends to a variable or increments a counter.

Note the ${_VM_LIB_LOADED:-} with the default: it has to survive set -u, which Chapter 16.3's callout covered and which a library cannot assume is off.

Define only; do not act. A library should contain definitions and nothing else. No cd, no set, no shopt, no output, and no work. If it genuinely needs setup, make that an explicit function the caller invokes:

# lib.sh
vm_init() {
  [ -n "${VM_OUT_DIR:-}" ] || VM_OUT_DIR=./out
  mkdir -p "$VM_OUT_DIR"
}

Now the caller decides when and whether, and a script that just wants vm_log is not surprised by a directory appearing.

Declare your locals, for the caller's sake as much as your own. Chapter 18.5's rule applies with extra force: a library function that assigns an undeclared n corrupts the variables of every script that ever sources it, in a file its author will never read.

4. Finding the Library

The path problem is real, because a script does not run from a known directory.

A relative path is relative to the caller's working directory, not the script's:

$ cd /tmp && /home/engineer/bin/use.sh
/home/engineer/bin/use.sh: line 2: ./lib.sh: No such file or directory

Chapter 15.3 raised this and gave the idiom; here it is for a library:

_VM_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
source "$_VM_DIR/lib.sh"

$BASH_SOURCE, not $0 — and the difference is measurable:

# who.sh, sourced by u4.sh
echo "  \$0=$0"
echo "  \$BASH_SOURCE=$BASH_SOURCE"
echo "  dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

$ ./u4.sh
  $0=./u4.sh
  $BASH_SOURCE=./who.sh
  dir=/home/engineer/work

$0 is the script the user ran; $BASH_SOURCE is the file the line is in. Inside a library, $0 tells you about the caller — useful for a usage message (Chapter 18.2 §2) and useless for finding yourself.

5. . or source

$ bash -c '. ./lib.sh && echo ok'
ok
$ dash -c '. ./lib.sh && echo "dash: . works"'
dash: . works
$ dash -c 'source ./lib.sh'
dash: 1: source: not found

. is POSIX; source is a Bash and ksh alias for it. They are identical in Bash.

Use source in a #!/usr/bin/env bash script — it is far more legible, and . is easy to miss when scanning. Use . in anything with #!/bin/sh, where source does not exist. This is the same trade as [[ ]] versus [ ] from Chapter 16.2, with the same answer.

6. return and exit, From a Library

Both measured in Chapter 18.3 §5 and worth restating in this context, because the consequences are the library author's responsibility:

$ printf 'echo "  lib line 1"\nreturn 0\necho "  NOT REACHED"\n' > early.sh
$ bash -c 'source ./early.sh; echo "  caller continues, status=$?"'
  lib line 1
  caller continues, status=0

$ printf 'echo "  lib line 1"\nexit 5\n' > fatal.sh
$ bash -c 'source ./fatal.sh; echo "  NEVER PRINTED"'
  lib line 1
$ echo $?
5

return is how a library declines; exit is how it kills its caller. A library that exits because a prerequisite is missing has decided, on behalf of every script that sources it, that the situation is fatal. Sometimes that is right. Usually the library should return non-zero and let the script decide:

# lib.sh
if ! command -v vcs > /dev/null 2>&1; then
  printf 'lib.sh: vcs is not on PATH\n' >&2
  return 1                          # not exit
fi

# the caller decides
if ! source "$_VM_DIR/lib.sh"; then
  note "falling back to the open-source simulator"
  source "$_VM_DIR/lib-oss.sh" || exit 4
fi

That is Chapter 18.3's "a function reports, the top level decides", applied one level up.

7. A Complete Pair

A library and a script that uses it properly:

# vmlib.sh — shared helpers. Source me; do not execute me.
# every public name is prefixed vm_ / VM_.

[ -n "${_VM_LIB_LOADED:-}" ] && return 0
_VM_LIB_LOADED=1

VM_LIB_VERSION=1.0

# vm_log <message>...   — timestamped line on stderr
vm_log() {
  printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*" >&2
}

# vm_die <message>...   — log and end the CALLER's script
vm_die() {
  vm_log "FATAL: $*"
  exit 1
}

# vm_have <command>     — 0 if the command exists
vm_have() {
  command -v "$1" > /dev/null 2>&1
}

# vm_count_lines <file> — prints the count; 1 if unreadable
vm_count_lines() {
  local f=$1
  local n
  n=$(wc -l < "$f") || return 1
  printf '%s\n' "$(( n ))"
}

# vm_require_version <min> — 0 if this library is at least <min>
vm_require_version() {
  local want=$1
  [ "${VM_LIB_VERSION%%.*}" -ge "${want%%.*}" ]
}

#!/usr/bin/env bash
# summarise.sh — uses vmlib.sh
# usage: summarise.sh <file> [file...]
#
# exit: 0 all readable · 1 at least one was not · 2 bad usage
#       4 the library did not load

set -o nounset

# find the library next to this script, whatever the caller's cwd
_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)

if ! source "$_DIR/vmlib.sh"; then
  printf '%s: cannot source %s/vmlib.sh\n' "${0##*/}" "$_DIR" >&2
  exit 4
fi
if ! declare -F vm_log > /dev/null; then
  printf '%s: vmlib.sh loaded but defines no vm_log\n' "${0##*/}" >&2
  exit 4
fi
if ! vm_require_version 1; then
  printf '%s: need vmlib 1.x, found %s\n' "${0##*/}" "$VM_LIB_VERSION" >&2
  exit 4
fi

[ $# -ge 1 ] || { printf 'usage: %s <file> [file...]\n' "${0##*/}" >&2; exit 2; }

vm_have wc || vm_die "wc is not on PATH"

bad=0
total=0

for f in "$@"; do
  if ! n=$(vm_count_lines "$f"); then
    vm_log "cannot read $f"
    bad=$(( bad + 1 ))
    continue
  fi
  printf '%-20s %5s\n' "${f##*/}" "$n"
  total=$(( total + n ))
done

printf '%-20s %5s\n' TOTAL "$total"
[ "$bad" -eq 0 ]

$ printf 'a\nb\n' > x.txt
$ printf 'c\n'    > y.txt
$ ./summarise.sh x.txt y.txt
x.txt                    2
y.txt                    1
TOTAL                    3
$ echo $?
0

$ cd /tmp && ~/work/summarise.sh ~/work/x.txt
x.txt                    2
TOTAL                    2
$ echo $?
0

$ cd ~/work
$ printf 'z\n' > locked.txt && chmod 000 locked.txt
$ ./summarise.sh x.txt locked.txt
x.txt                    2
/home/engineer/work/vmlib.sh: line 29: locked.txt: Permission denied
[18:15:40] cannot read locked.txt
TOTAL                    2
$ echo $?
1

Note whose line number that is: vmlib.sh: line 29, not summarise.sh. The wc that failed is inside vm_count_lines, which lives in the library — so the shell's own diagnostic names the library's file and line even though the script is what you ran. That is sourcing being exactly what it claims: the library's lines are running, as lines of code, with their own identity. It is also a small gift when debugging, because the message points at the real location rather than at the call site.

Now break the load, two different ways:

$ mv vmlib.sh vmlib.sh.bak
$ ./summarise.sh x.txt
./summarise.sh: line 13: /home/engineer/work/vmlib.sh: No such file or directory
summarise.sh: cannot source /home/engineer/work/vmlib.sh
$ echo $?
4

$ grep -v '^vm_log' vmlib.sh.bak > vmlib.sh
$ ./summarise.sh x.txt
summarise.sh: vmlib.sh loaded but defines no vm_log
$ echo $?
4

Two distinct messages for two distinct failures, and the second is the one source alone cannot catch — the file was perfectly readable and simply did not contain what the script needed.

Six decisions this chapter is responsible for:

Every library name is prefixed. vm_log, vm_die, VM_LIB_VERSION. No chance of colliding with /usr/bin/log, with the script's own helpers, or with a second library.

The double-source guard is the first line, with ${_VM_LIB_LOADED:-} so it survives set -u in the caller.

The library defines and does not act. No cd, no set, no shopt, no output at load time. The only thing it does is set two variables, both prefixed.

The script finds the library with $BASH_SOURCE, which is why the second transcript — run from /tmp with an absolute path — works at all.

Three separate load checks, three distinct failures: unreadable file, loaded-but-wrong, and too old. All exit 4, which is distinct from the script's own 1 and 2 so a caller can tell "your environment is broken" from "your input was".

vm_die exits and is named for it. The script's own vm_have wc || vm_die is the one place a library function is allowed to end things, and the name is the contract — Chapter 18.3's rule.

Practice

1. Write lib.sh defining a function and setting a variable. Source it from one script and execute it from another, printing the variable and type of the function in both. Explain both results.
2. On a Mac, check type log before defining anything. Then write a script that intends to define log() but whose source line has a typo, and describe exactly what happens when it calls log "starting".
3. Write bad-lib.sh that does cd /tmp, shopt -s nullglob and x=CLOBBERED. Source it and print all three before and after. Which of the three would you be least likely to notice in a real script?
4. Add a double-source guard to a library, source it three times, and confirm the body runs once. Then remove the :- from ${_LOADED:-} and source it under set -u.
5. Put return 0 in a sourced file after one echo, then exit 5, and show the difference to the sourcing script. Which should a library use for a missing prerequisite?
6. Print $0 and $BASH_SOURCE inside a sourced file. Then write the one-line idiom that gives the library's own directory and explain why $0 cannot.
7. Create pdir/mylib.sh on PATH and a different mylib.sh in the current directory. Run source mylib.sh and determine which one loaded. State the rule you will follow from now on.
8. Run . ./lib.sh and source ./lib.sh in bash, then both in dash. Which spelling goes in a #!/bin/sh script?
9. Take summarise.sh, mv the library away, and run it. Then make the library readable but delete vm_log from it and run again. Confirm the two different exit-4 messages.
10. Add n=5 at the top level of vmlib.sh with no prefix. Find a way to make summarise.sh produce a wrong total because of it, and then state the convention that prevents it.

Check Your Understanding

1. What is the difference between sourcing a library and executing it?
2. You executed the library by mistake and the script still ran. Why, and what did log resolve to?
3. Name three things a sourced file can change in your shell without saying so.
4. Why prefix every name in a library?
5. Why does the double-source guard use ${_LOADED:-} rather than $_LOADED?
6. Inside a sourced file, what is $0 and what is $BASH_SOURCE?
7. Why should source lib.sh never be written without a slash?

8. Remember

• Sourcing runs the file in your shell; executing it runs it in a child that then exits with everything it defined.
• A failed load is silent, and on macOS log then resolves to /usr/bin/log. Check with source … || exit and declare -F.
• Sourcing shares everything and isolates nothing — variables, functions, working directory, shell options, traps.
• There is no namespace, so prefix every name a library defines. It is the highest-value habit in shared shell code.
• Guard against double-sourcing with [ -n "${_X_LOADED:-}" ] && return 0, and keep the :- so it survives set -u.
• A library defines; it does not act. No cd, set, shopt or output at load time — put setup in a function the caller calls.
• Declare locals in library functions, or you corrupt every script that ever sources you.
• Find the library with $BASH_SOURCE, not $0. $0 is the caller.
• Always source a path with a slash. A bare name is searched on PATH first.
• . is POSIX, source is Bash. Use source under a bash shebang, . under #!/bin/sh.
• A library should return to decline, not exit, and let the script decide what a missing prerequisite means.

9. The Module, in One Page

Six chapters, and they reduce to one sentence and its consequences.

A Bash function is a command you named — not a procedure, not a value-returning subroutine. Everything else follows:

• So it is called without parentheses, it has an exit status, and it beats builtins and PATH (18.1). f("x") is a syntax error; log is a real program.
• So data goes in as positional parameters (18.2) — its own $1, $# and $@, restored on return, but not its own $0, which still names the script.
• So "how it went" is a number, and return hands it back while exit ends everything (18.3). A function reports, the top level decides — and a die inside $( ) does not die.
• So a real value comes out as output or through an agreed variable (18.4), and local v=$(cmd) silently throws the status away.
• And because there is one namespace, every variable a function assigns is shared unless declared (18.5) — with dynamic scope on top, so a callee can read and change its caller's locals.
• Moving the functions to a file shares all of that with every script that sources it (18.6), which is why libraries need prefixes, guards, and a rule against acting at load time.

A script can now decide, repeat, and be decomposed into named pieces with their own arguments, statuses and scope. One thing has been deferred in every module since Module 8, and worked around three times in the last two: a list.

"$@" is a list the shell hands you and you cannot build. A string split on IFS breaks the moment an element contains a space — Chapter 17.1 §3 admitted it, Chapter 17.2 measured it, Chapter 18.4 §7 ran into it again, and Chapter 18.5's double Permission denied was it a fourth time.

Module 19 is arrays: a real list you can build, append to, index, pass to a command with its boundaries intact, and key by name instead of number. It is the last piece of Bash the rest of Part I has been writing around.