Debugging a Script: set -x, bash -n and PS4

Part I · Module 20 · Chapter 20.8 · Debugging a Script: set -x, bash -n and PS4

Chapter 20.1 opened with a script that reported success having done nothing. Everything since has been about making that impossible. This chapter is about the other half: when a script has gone wrong, finding out what it did.

1. Check the Syntax Without Running Anything

$ bash -n d.sh
$ echo $?
0

Silence means it parses. bash -n reads the whole script, builds it, and does not execute a single command — so it is safe on a script you do not trust and instant on a script that takes an hour.

$ cat broken.sh
#!/bin/bash
if true; then
  echo hi

$ bash -n broken.sh
broken.sh: line 4: syntax error: unexpected end of file
$ echo $?
2

An unterminated if, caught without creating anything. Put bash -n in front of any script you are about to run against something you care about, and in your editor's save hook if it has one. It catches the whole class of error that Chapter 18.1 §1 described — a missing fi, a missing ; before }, an unbalanced quote — and those are exactly the errors that otherwise surface as unexpected end of file after the script has already done half its work.

What it cannot catch is anything about values: an empty variable, a wrong path, a command that does not exist. It is a parser, not a checker.

2. Tracing Shows the Expansion, Which Is the Point

#!/usr/bin/env bash
# d.sh
name=${1:-run}
dir="out dir"
file="$dir/$name.log"
mkdir -p -- "$dir"
printf 'hello\n' > "$file"
wc -l < "$file"

$ bash -x d.sh
+ name=run
+ dir='out dir'
+ file='out dir/run.log'
+ mkdir -p -- 'out dir'
+ printf 'hello\n'
+ wc -l
       1

Look at what the trace is telling you that the source cannot.

+ dir='out dir' — the value is shown, and it is quoted. Bash adds quotes around any traced word containing a space, which means the trace shows you argument boundaries. That is the single most useful property of set -x, and it is why it finds the Module 19 class of bug:

$ bash -c 'set -x; dir="out dir"; f="$dir/a.log"; printf "%s\n" "$f" >/dev/null' 2>&1 | grep '^+'
+ dir='out dir'
+ f='out dir/a.log'
+ printf '%s\n' 'out dir/a.log'

One argument, quoted. Drop the quotes in the script and the trace changes visibly:

$ bash -c 'set -x; dir="out dir"; f=$dir/a.log; printf "%s\n" $f >/dev/null' 2>&1 | grep '^+'
+ dir='out dir'
+ f='out dir/a.log'
+ printf '%s\n' out dir/a.log

Two arguments, no quotes. printf received out and dir/a.log. Reading the source would not have shown you that; the trace shows it immediately.

+ printf 'hello\n' with no redirection shown, and + wc -l with no input file. Redirections are not traced — only the command and its arguments. That is a real limitation: a trace cannot tell you which file a > went to. For that you need the redirection's target in a variable you can see, or set -v.

3. A PS4 Worth Setting

The + prefix comes from PS4, and the default is minimally useful on anything longer than ten lines:

$ PS4='+ ${BASH_SOURCE##*/}:${LINENO}:${FUNCNAME[0]:-main}: ' bash -x d.sh
+ d.sh:2:main: name=run
+ d.sh:3:main: dir='out dir'
+ d.sh:4:main: file='out dir/run.log'
+ d.sh:5:main: mkdir -p -- 'out dir'
+ d.sh:6:main: printf 'hello\n'
+ d.sh:7:main: wc -l
       1

File, line, and function on every line. BASH_SOURCE and FUNCNAME are Chapter 18.2's arrays, and ${FUNCNAME[0]:-main} gives main at the top level where the array is empty. With that set, a trace through a script that sources two libraries tells you which file each command came from.

Set it in your environment rather than in the script:

export PS4='+ ${BASH_SOURCE##*/}:${LINENO}:${FUNCNAME[0]:-main}: '

Then bash -x anyscript is immediately readable without editing anything.

4. Trace a Region, Not the Whole Script

set -x and set +x turn tracing on and off, so you can narrow to the part you are debugging:

$ cat d2.sh
#!/usr/bin/env bash
echo "quiet part"
set -x
v=$(( 2 + 2 ))
set +x
echo "quiet again, v=$v"

$ ./d2.sh
quiet part
+ v=4
+ set +x
quiet again, v=4

Note + set +x is itself traced — the set +x is the last thing the trace shows, which is a small confirmation that the region ended where you meant.

For a script you cannot edit, the same thing from outside:

bash -x script.sh 2> trace.log       # the whole thing, to a file

Tracing goes to stderr, which is Chapter 20.7's rule applied to the shell itself — so 2> trace.log separates the trace from the script's own output, and | grep '^+' isolates the trace lines when they are interleaved.

And one command without touching the script at all:

$ bash -c 'set -x; dir=""; echo rm -rf "$dir/"' 2>&1 | grep '^+'
+ dir=
+ echo rm -rf /

That is Chapter 20.4's method, and it is the thing to reach for whenever you are unsure what an expansion produces.

5. Reading Nested Substitutions

The number of + signs is the nesting depth, which matters when a line contains a substitution:

$ env -u RESULTS_ROOT bash -x ./nightly.sh 2>&1 | grep '^+'
+ cd /nightly
+ rm -rf 'old/*'
++ cat tests.txt
+ testlist=
++ grep -lc UVM_ERROR './*.log'
++ wc -l
+ failed='       0'
+ echo 'nightly complete:        0 failures'

That is Chapter 20.1's lying script, and the trace is the whole diagnosis in eight lines.

+ cd /nightly — the variable vanished, so the path is wrong. One +.

++ cat tests.txt — two + signs, because cat ran inside a command substitution. Then + testlist= at depth one shows the result: empty.

+ rm -rf 'old/*' — quoted by the trace, which tells you the glob did not match and stayed literal.

+ failed=' 0' — the padding from BSD wc is visible, which is where the gap in the output line came from.

Every one of Chapter 20.1's claims about that script is read directly off its trace. When a script has done something inexplicable, trace it before theorising — it is faster than reading, and it cannot be wrong about what happened.

6. The Rest of the Toolbox

bash -v shows the source instead of the expansion:

$ bash -v -c 'x=1
echo "$x"'
x=1
echo "$x"
1

Useful for seeing which lines ran — a long case, an if chain — where -x tells you what they became. The two combine: bash -xv gives source then expansion for each line, which is verbose and occasionally exactly right.

set -u and set -e are debugging tools too. Chapter 20.2's options turn a silent wrong answer into a stopped script with a line number, which is often the whole investigation. Adding them temporarily to a misbehaving script is a legitimate first move.

trap … ERR with a line number from Chapter 20.5 §4 turns a set -e abort into a report:

trap 'printf "%s: failed at line %s (status %d)\n" "${0##*/}" "$LINENO" "$?" >&2' ERR

declare -p prints a variable's real value including an array's indices, which Chapter 19.2 §6 showed is the only way to see a hole.

And shellcheck, which is not part of Bash and is worth knowing exists. It is a static analyser that finds a large fraction of what Modules 9, 17, 19 and 20 have been teaching — unquoted expansions, $(ls) in a loop, local x=$(cmd) masking a status, rm -rf $dir without a guard. If you write shell regularly, run it on everything. It is not installed on this machine, so nothing in this course was checked with it; treat that as a gap in the course rather than an endorsement of skipping it.

7. A Complete Session

Not a script this time — the sequence you actually follow. A nightly run reported success and produced no logs:

$ ./nightly.sh
nightly complete:        0 failures
$ echo $?
0

One: syntax, because it is free.

$ bash -n nightly.sh
$ echo $?
0

Parses. Not a typo, then.

Two: turn on the options, because they may answer it immediately.

$ bash -c 'set -euo pipefail; . ./nightly.sh'
./nightly.sh: line 4: RESULTS_ROOT: unbound variable

There it is — and that is often where the investigation ends. Chapter 20.2's set -u found in one run what reading would not have.

Three: trace, to see the consequences.

$ env -u RESULTS_ROOT bash -x ./nightly.sh 2>&1 | grep '^+'
+ cd /nightly
+ rm -rf 'old/*'
++ cat tests.txt
+ testlist=
++ grep -lc UVM_ERROR './*.log'
++ wc -l
+ failed='       0'
+ echo 'nightly complete:        0 failures'

Now you know not just the cause but the whole chain: wrong directory, empty testlist, zero iterations, a count of nothing reported as a count of zero failures.

Four: fix the cause, and the reporting. The unset variable is one line (Chapter 20.3). The lie is a separate bug and needs its own fix — a count of zero tests must not be reported as zero failures (Chapter 17.1 §5), and the summary must not be the thing that sets the exit status (Chapter 20.7).

Five: re-trace to confirm. A fix you have not seen in a trace is a fix you are assuming.

That order — parse, options, trace, fix, re-trace — is worth keeping. Each step is cheaper than the next and any of them may finish the job.

Practice

1. Run bash -n on a working script, then on one with a missing fi. Record both exit statuses and confirm nothing ran in the second case.
2. Write d.sh from §2 and run it under bash -x. Identify every place the trace added quotes, and say what each one tells you.
3. Remove the quotes from "$file" in d.sh and re-trace. How many arguments does printf receive now, and which line of the trace shows it?
4. Set the §3 PS4 and re-run. Then add a function to the script, call it, and confirm the function's name appears.
5. Set PS4='+ $(date +%s) ' and trace a loop of a thousand iterations. Time it. Then switch to ${SECONDS} and time it again.
6. Use set -x / set +x to trace exactly one block of a script. Confirm that + set +x is the last traced line.
7. Trace a script whose line contains a command substitution. Count the + signs and explain what the extra one means.
8. Reproduce the §5 nightly.sh trace. Then annotate each of the eight lines with what it proves.
9. Run a script under bash -v, then bash -x, then bash -xv. Describe what each is good for in one sentence.
10. Trace a command with a redirection and confirm the redirection does not appear. Then restructure the line so the trace shows you the target.
11. Find out whether shellcheck is available to you. If it is, run it on the longest script you have written in this course and count the findings.

Check Your Understanding

1. What does bash -n do, and what can it not catch?
2. What is the single most useful property of set -x output?
3. A traced line reads + printf '%s\n' out dir/a.log. What is wrong, and how do you know?
4. What does the number of + signs mean?
5. Why is PS4='+ $(date) ' a bad idea?
6. Name two things a trace does not show.
7. In what order would you apply the tools in this chapter?

8. Remember

• Chapter 33.6 takes this to a whole flow, and measures two things that surprise everybody: bash -x traces none of a script's children, and switching tracing on reclassified a passing run as vacuous.
• bash -n parses without running. Free, safe, instant — use it before any script you are about to point at something you care about.
• It cannot see values. It is a parser, not a checker.
• set -x prints each command after expansion, which is the whole point: the source cannot show you a value that is about to split.
• The trace quotes any word containing a space, so it shows real argument boundaries.
• Redirections are not traced — only the command and its arguments.
• Set a PS4 with file, line and function. '+ ${BASH_SOURCE##*/}:${LINENO}:${FUNCNAME[0]:-main}: ', exported once.
• Never put a command substitution in PS4 — it forks per traced line. Use ${SECONDS}.
• The number of + signs is the nesting depth. Two means inside a substitution or subshell.
• set -x / set +x narrows to a region, and + set +x is itself traced.
• Tracing goes to stderr, so 2> trace.log separates it from the script's output.
• set -u and set -e are debugging tools, not just production ones. Adding them temporarily is a legitimate first move.
• declare -p is the only way to see an array's holes.
• The order is: parse, options, trace, fix, re-trace. A fix you have not seen in a trace is an assumption.

9. The Module, in One Page

Eight chapters, and they answer one question: what should a script do when something is wrong?

• By default, nothing (20.1). A failed step does not stop the script, the status is the last command's, and seven measured shapes of that let a script do the wrong thing and report success.
• set -euo pipefail fixes most of it and is not a defence (20.2). It steps aside wherever a status is being examined — and that covers the whole body of a function called in an if, with || true, or as v=$(f). pipefail plus an early-exiting consumer gives 141, on inputs above a size you did not choose.
• State requirements before doing work (20.3), because set -u fires at the point of use and says nothing about what was needed.
• Guard the commands that cannot be undone (20.4). An empty variable makes rm -rf "$dir/" into rm -rf /, and "$dir"/* into every top-level directory — measured, and invisible to set -u.
• Clean up on every path (20.5). One EXIT handler covers finishing, failing and being aborted; an interrupt needs its own handler and a re-raise, or the script runs on past the signal and exits 0.
• Create scratch space atomically (20.6), because a predictable name in a shared directory is a collision bug and a security hole at once.
• Answers on stdout, everything else on stderr (20.7), or the script cannot be piped, cannot be captured, and cannot have its log separated.
• And when it has already gone wrong, trace it (20.8) — because the errors of the last four modules live in expansions, and only a trace shows those.

10. What This Module Leaves You With

Twenty modules in, a script you write can hold data, decide, repeat, be decomposed into named pieces with their own arguments and scope, build a correct command line out of values it did not choose, fail loudly, clean up after itself, and tell the truth in its exit status. That is the language, and it is enough to automate your own work reliably.

It is not yet enough to hand the script to someone else.

Everything in this module has been about a script's behaviour when something goes wrong internally. None of it is about the script's interface — what it takes, how a caller passes options, where a setting comes from when three places disagree, what each of its exit codes promises, whether it is safe to run twice, or whether it will work on a machine you did not configure.

Those are the properties that decide whether a colleague is willing to run your script at all, and they are Module 21, which closes Part I:

• Interface first: deciding what a script takes before writing what it does, and a usage message that fires before any work.
• getopts: flags and option arguments the conventional way, rather than positional parameters whose order nobody remembers.
• Configuration precedence: defaults, then environment, then flags — stated once so the same script behaves predictably for everyone.
• Exit codes as a contract: defining what each non-zero status means, because that number is the only thing an automated caller can react to. Chapter 15.6 and this module have been using 1, 2 and 4 by convention; Module 21 makes it a documented interface.
• Idempotence and dry runs: safe to run twice, and able to say what it would do. Chapters 19.5 and 20.4 built dry runs ad hoc; this is the general version.
• Portability: which constructs are Bash-only and which option spellings differ between GNU and BSD — a question this course has answered a dozen times in passing (mapfile, declare -A, pipefail, ${a[-1]}, mktemp templates) and that deserves one place.
• shellcheck: the static checker this chapter named and could not run, applied properly — and how to read a review of shell code.
• And testing a shell script: proving it behaves correctly, including that it fails when it should, which is the half people skip and the half that matters.

After that, Part II is the same language applied to the job — RTL repositories, EDA environments, simulator invocation, logs, regressions, and the specific ways real tools misbehave. Nothing in it re-teaches Part I.