Linux & Bash · Part II · Module 33 · Debugging Failed Jobs and Broken Flows

A Command That Fails Only Inside the Script

The command works when you paste it and fails when the script runs it, which feels like magic and is always one of four things. This chapter measures each of them on this machine — starting with the one nobody checks, that the shell you tested in is not the shell that ran it, and that the same line built two arguments at the prompt and four in the script.

Advanced26 min readLinuxPart IIDebuggingShellQuoting

Part II · Module 33 · Chapter 33.3 · A Command That Fails Only Inside the Script

Chapter 33.2 replayed a recorded argv and got the same result. This chapter is the inverse situation: the command is right, you have tested it, and it behaves differently when a script runs it.

There are four differences between your prompt and a script's execution, and the first one is the one nobody checks.

1. It Is Not the Same Shell

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ echo $SHELL
/bin/zsh
$ dscl . -read ~/ UserShell | awk '{print $2}'
/bin/zsh
$ printf '#!/bin/bash\necho $BASH_VERSION\n' > /tmp/w.sh; chmod +x /tmp/w.sh; /tmp/w.sh
3.2.57(1)-release

The prompt is zsh. The script is bash 3.2.57. Those are different languages that share a syntax, and a line tested in one is not a line tested in the other. This is the default on every current macOS, and it is extremely common on Linux too, where the prompt is often bash and #!/bin/sh scripts run dash.

Here is a line from a flow wrapper. The image lives under a directory whose name contains a space, which is the kind of thing a vendor drop or a shared area inflicts on you:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

IMAGE="$ROOT/sim builds/sim.vvp"
OPTS="+TESTNAME=smoke +SEED=1"
vvp $IMAGE $OPTS

Pasted at the prompt:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

TB  test=smoke +SEED=1 seed=1 cycles=40 verbosity=LOW
TB  ERROR unknown test 'smoke +SEED=1'
TB  TEST FAILED  test=smoke +SEED=1 checks=0 errors=1
tb/tb_arb.v:165: $finish called at 25 (1s)
  rc=0

Run by the script:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

/…/m33/sim: Unable to open input file.
  rc=255

Two shells, one line, two completely different failures — and the reason is visible the moment you ask each shell how many arguments it built:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  zsh:
    argc=2
      [1] /…/m33/sim builds/sim.vvp
      [2] +TESTNAME=smoke +SEED=1
  bash:
    argc=4
      [1] /…/m33/sim
      [2] builds/sim.vvp
      [3] +TESTNAME=smoke
      [4] +SEED=1

zsh does not word-split an unquoted parameter expansion. bash does. So zsh kept the path intact — and also kept both plusargs glued into one argument, which the testbench read as a test named smoke +SEED=1. bash split the path at its space and passed the first half as the image.

The prompt's result was the more dangerous of the two

The bash failure is loud: exit 255 and a message naming a file that does not exist. Ten seconds to diagnose.

The zsh result ran a simulation, produced a log, reported a verdict, and exited 0. In a flow that reads the log for a verdict — which is every flow in Module 32 — that is a FAIL row attributed to a test called smoke +SEED=1, and the only clue is a space inside a test name in a report nobody is reading character by character.

So the usual framing of this chapter's problem is backwards. It is not "the prompt is right and the script is wrong". Both were wrong, differently, and the prompt's wrongness is the kind that gets committed — because it ran.

The fix is not to pick a shell. It is to quote:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

vvp "$IMAGE" +TESTNAME=smoke +SEED=1

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  zsh        TB  test=smoke seed=1 cycles=40 verbosity=LOW
  bash       TB  test=smoke seed=1 cycles=40 verbosity=LOW

Identical, because a quoted expansion has no splitting to differ about. Chapter 9.2 is the rule; this is what it costs when the two shells disagree about it, and Chapter 33.5 is the full catalogue.

The other four disagreements worth knowing

Every one of these was run in both shells on this machine:

construct	zsh	bash
`a="x y z"; set -- $a; echo $#`	1	3
`a=(p q r); echo ${a[1]}`	p	q
`echo /nonexistent/*.xyz`	error: no matches found	prints the pattern
`false \| true; echo ${PIPESTATUS[0]}`	empty	1
`$0` inside `-c`	`zsh`	`/bin/bash`

The second row is the one that silently corrupts data: ${a[1]} is the first element in zsh and the second in bash. A loop tested at a zsh prompt that indexes from 1 reads one element earlier in a script, and arrays of filenames do not announce an off-by-one.

The third row is the direction people do not expect: zsh refuses a glob that matches nothing, while bash hands the tool a literal *. So the interactive test is the safe one and the script is where a pattern reaches a tool as a filename — the shape Chapter 3.5 warns about and Chapter 24.5 measured in a filelist search.

The fourth row is why Chapter 32.7's PIPESTATUS fix must be paired with a #!/bin/bash line: under zsh the array is pipestatus, indexed from 1, so ${PIPESTATUS[0]} is empty and exit "" is exit 0.

The practical rule: test the line in the shell that will run it.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ /bin/bash -c 'the exact line here'

Not pasted at your prompt. One extra wrapper, and it removes this entire category.

Figure 1 — the four differences between your prompt and a script's execution, in the order they are worth checking. The first two are reproductions rather than inspections — one command each makes the script's conditions happen at your prompt — which is why they settle the question instead of narrowing it.

2. The Environment Is Not the Same Environment

A flow script that needs a project root:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
echo "  PROJ_ROOT=${PROJ_ROOT:-<unset>}"
[ -n "${PROJ_ROOT:-}" ] || { echo "  needsroot: PROJ_ROOT is not set" >&2; exit 3; }

Three ways to give it one, and only two of them work:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

=== a variable SET but not EXPORTED ===
  PROJ_ROOT=<unset>
  needsroot: PROJ_ROOT is not set
  rc=3
=== the same variable EXPORTED ===
  PROJ_ROOT=/…/m33
  found the image
  rc=0
=== set on the command line of the invocation (one-shot) ===
  PROJ_ROOT=/…/m33
  found the image
  rc=0

A variable you set at the prompt is a shell variable; a child process inherits only environment variables. Chapter 10.1 is the distinction, and the reason it shows up as "works for me" is that the person who tested it had also exported it at some point weeks ago, in a shell that is still running.

How big is the gap, really

It is worth measuring rather than assuming, because the answer is surprising in both directions:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  this shell          : 62 variables
  bash -lc            : 61
  env -i bash -c      : 3

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ comm -23 <(env | cut -d= -f1 | sort) <(/bin/bash -lc 'env' | cut -d= -f1 | sort)
OLDPWD

A login shell loses almost nothing — one variable, and it is OLDPWD. So the folk explanation that "a script gets a different environment" is not true for a script you launch yourself from a terminal.

The gap appears when something else launches it:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  PATH entries: interactive 15 / bash -lc 15 / env -i 5
  env -i PATH : /usr/gnu/bin:/usr/local/bin:/bin:/usr/bin:.
  is iverilog reachable? interactive=yes login=yes clean=NO

Three variables and five PATH entries, and the simulator is not on it. That is what a cron job, a CI agent or a batch submission hands your script, and Chapter 32.8 measured the consequence — twenty-four runs that all crashed with exec: vvp: not found, and an empty image digest reported only on standard error.

So the diagnosis splits in two:

If it fails when you run the script, the environment is almost certainly not the difference. Look at Sections 1, 3 and 4 first.

If it fails only under a scheduler or a timer, the environment is the first suspect, and env -i reproduces it exactly, locally, for free.

3. The Working Directory Is Not the Same Directory

Chapter 33.2 measured the shape of this: the working directory does not change what a simulation computes, it changes where everything it writes lands. The verdict was identical from two directories; the waveform moved to the project root; and two runs from the same place overwrote each other's dump.

What makes it a script problem rather than a run problem is that scripts change directory and prompts rarely do. Chapter 31.6 runs every simulation from inside its own run directory precisely so the tool's self-named files are isolated — which means every relative path in the argv is resolved against a directory you were not in when you tested the line.

This is the mechanism that has now appeared in this track more than any other:

chapter	what broke
26.1	a filelist's relative source paths, resolved against the cwd
27.8	a tool invocation's relative arguments
30.1	every job a launcher started, with `logs/long.vvp: Unable to open input file`
30.5	a relative output path, on a read-only directory
31.6	the argv, invalidated by the `cd` that provides isolation
33.1	the image path, had it not been resolved first

One fix, applied once, at the top, before anything moves:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

root=$(cd "$(dirname "$0")/.." && pwd)        # absolute from here on

And the diagnostic, when a script fails on a path that exists: print the working directory next to the path, because the path is almost never the thing that is wrong.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

printf 'about to read %s from %s\n' "$f" "$PWD" >&2

4. The Options Are Not the Same Options

A non-interactive bash starts with none of the safety you may believe is on:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ /bin/bash -c 'set -o | grep -E "^(errexit|nounset|pipefail|xtrace)"'
errexit         off
nounset         off
pipefail        off
xtrace          off

All four off. So a script that does not set them has no error exit, no unset-variable check, no pipeline status and no trace — and Chapter 20.2 covers both what they buy and the significant things they do not catch.

And one option is off in a way that changes what commands mean:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ zsh -ic 'alias rs="vvp /dev/null"; type rs'
rs is an alias for vvp /dev/null
$ /bin/bash -c 'alias rs="vvp /dev/null"; type rs'
rs is /usr/bin/rs
$ /bin/bash -c 'shopt expand_aliases'
expand_aliases  off

expand_aliases is off in a non-interactive shell, so the alias definition was accepted and then ignored. And the name resolved to something else entirely — /usr/bin/rs is a real program, a BSD array-reshaping tool nobody has used on purpose in thirty years.

That is the worst possible outcome of the alias difference: not "command not found", but a different program, found and executed. Chapter 10.5 establishes the lookup order and that aliases are checked first and only interactively; the consequence for debugging is that an alias is not part of a command's meaning inside a script, and a convenience you defined for yourself can be a real binary on somebody else's machine.

The four-question diagnostic, in order of how often it is the answer

When a command works at your prompt and fails in a script, ask these in this order. Each is one command, and the order is by measured frequency rather than by elegance.

1. Is it the same shell? Re-run the exact line with /bin/bash -c '…'. If it fails there too, the script is innocent and you have just reproduced the bug in one line.

2. Is it the same options? Add set -x to the script, or run the line under bash -x -c, and compare what the shell says it is executing against what you think it is. Chapter 33.6 is this question in full.

3. Is it the same directory? pwd on both sides. Then check every relative path in the command.

4. Is it the same environment? env > /tmp/mine at the prompt, the same inside the script, and diff. If the script is launched by a scheduler, skip straight to env -i as the reproduction.

Questions 1 and 2 together cover the great majority, and they share a property worth naming: both are answered by making the script's conditions happen at your prompt, rather than by reasoning about what the script might be doing. /bin/bash -c and env -i are reproductions, not inspections, which is why they settle the question instead of narrowing it.

5. The Debugging Harness

Putting the four questions in one script, so the comparison is mechanical:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# whydiff 'COMMAND LINE' -- run one line four ways and show what differs.
set -u
line=${1:-}
[ -n "$line" ] || { echo "whydiff: give me a command line, quoted" >&2; exit 2; }
run() {   # run LABEL SHELL-AND-ARGS...
  local label=$1; shift
  local out rc
  out=$("$@" 2>&1); rc=$?
  printf '  %-26s rc=%-4s %s
' "$label" "$rc" "$(printf '%s' "$out" | head -1 | cut -c1-60)"
}
printf 'whydiff: %s
' "$line"
printf '  cwd=%s
' "$PWD"
run "your prompt's shell"   "${SHELL:-/bin/sh}" -c "$line"
run "bash (what scripts use)" /bin/bash          -c "$line"
run "bash, traced"            /bin/bash -x       -c "$line"
run "bash, clean environment" env -i /bin/bash   -c "$line"

Four lines of output, and whichever one differs from the others names the category:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ export IMAGE="$PWD/sim builds/sim.vvp" OPTS="+TESTNAME=smoke +SEED=1"
$ whydiff 'vvp $IMAGE $OPTS'
whydiff: vvp $IMAGE $OPTS
  cwd=/tmp/m33demo
  your prompt's shell        rc=0    TB  test=smoke +SEED=1 seed=1 cycles=40 verbosity=LOW
  bash (what scripts use)    rc=255  /tmp/m33demo/sim: Unable to open input file.
  bash, traced               rc=255  + vvp /tmp/m33demo/sim builds/sim.vvp +TESTNAME=smoke +SEED=
  bash, clean environment    rc=127  /bin/bash: vvp: command not found

Each row is a hypothesis tested rather than considered, and four different exit statuses from one line is the point: 0, 255, 255, 127.

The same harness on the quoted version, which is how you confirm a fix rather than hope for one:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ whydiff 'vvp "$IMAGE" +TESTNAME=smoke +SEED=1'
whydiff: vvp "$IMAGE" +TESTNAME=smoke +SEED=1
  cwd=/tmp/m33demo
  your prompt's shell        rc=0    TB  test=smoke seed=1 cycles=40 verbosity=LOW
  bash (what scripts use)    rc=0    TB  test=smoke seed=1 cycles=40 verbosity=LOW
  bash, traced               rc=0    + vvp '/tmp/m33demo/sim builds/sim.vvp' +TESTNAME=smoke +SEE
  bash, clean environment    rc=127  /bin/bash: vvp: command not found

Three rows agree and the fourth is the clean environment, which is a separate and known problem.

The traced row is the most informative of the four, and the quoted run shows why: bash's trace prints

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

+ vvp '/tmp/m33demo/sim builds/sim.vvp' +TESTNAME=smoke +SEED=1

— with quotes around the argument that contains a space. The trace is not an echo of your source line; it is the shell telling you the argv it built, quoted so the boundaries are unambiguous. Comparing that against the broken run's unquoted + vvp /tmp/m33demo/sim builds/sim.vvp … is the whole diagnosis in one character. Chapter 33.6 is about reading exactly that.

Two ways this harness lies, and the guard for each

$SHELL is not necessarily the shell you are typing into. It is the shell listed in your user record, which is usually the same thing and is not when you have started a different one. The honest way to ask is to look at the parent process:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ ps -o comm= -p $PPID

If whydiff's first row says zsh and you are actually in bash, it compares bash against bash and reports "no difference" for a line that does differ.

And env -i erases too much to be a scheduler. Three variables is cleaner than any real scheduler: a batch system typically keeps HOME, USER, LOGNAME and a minimal PATH, and Chapter 30.5 measured which. So rc=127 vvp: command not found on that row is a true statement about an environment nobody will actually give you, and the useful version is to reproduce the environment you were given:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

run "as submitted" env -i HOME="$HOME" USER="$USER" PATH=/usr/bin:/bin /bin/bash -c "$line"

Both failures have the same shape, and it is the one to watch for in any diagnostic tool: a comparison is only as good as the fidelity of the thing it compares against. A row that is too clean, or a row that is secretly the same as another row, produces a confident answer to a question you did not ask.

Practice

Find out what shell your prompt is, from $SHELL and from $PPID, and see whether they agree.
Find out what shell your scripts run by printing $BASH_VERSION from one.
Take a line with an unquoted expansion and run it in both shells, printing the argument count.
Put a space in a path and compare what each shell builds.
Explain which of the two results is more dangerous, and why.
Quote the expansion and confirm both shells agree.
Index an array from 1 in both shells and compare.
Glob something that does not exist in both shells.
Read ${PIPESTATUS[0]} in both shells.
Set a variable without exporting it and run a script that needs it.
Export it and run again; then use the one-shot form.
Count your environment variables at the prompt, in bash -lc, and under env -i.
Diff the variable names between your shell and a login shell.
Check whether your tools are reachable in all three.
Decide, from those numbers, when the environment is a plausible suspect.
Print pwd next to a failing relative path and see which one is wrong.
Resolve your script's root to an absolute path on the first line of work.
Check the four shell options in a non-interactive bash.
Define an alias in a script and then ask type what the name means.
Find out whether that name is a real program on your machine.
Write the four-way harness and run it on a line that misbehaves.
Make the harness compare bash against bash by faking $SHELL, and watch it report no difference.
Replace env -i with an approximation of what your scheduler actually passes.

Check Your Understanding

What shell is the prompt on this machine, and what runs the scripts?
What argument counts did the two shells build from the same line?
Why did zsh keep the path intact?
What did the testbench think the test was called?
Which of the two results is more dangerous, and why?
What is the fix, and why does it make the shells agree?
What is ${a[1]} in each shell?
What does each shell do with a glob that matches nothing, and which direction is surprising?
What is ${PIPESTATUS[0]} under zsh, and what does exit then do?
What is the practical rule for testing a line?
What is the difference between a shell variable and an environment variable?
How many variables did a login shell lose, and which?
What does that tell you about the folk explanation?
How many variables and PATH entries does env -i leave, and is the simulator reachable?
When is the environment a plausible suspect, and when is it not?
Why is the working directory a script problem rather than a run problem?
How many times has the relative-path mechanism appeared in this track?
What is the one-line fix, and what is the diagnostic?
Which four shell options are off in a non-interactive bash?
What happens to an alias defined inside a script?
What did the name rs resolve to, and why is that the worst outcome?
Give the four diagnostic questions in order.
What property do questions 1 and 2 share?
Give the two ways whydiff can lie, and the guard for each.
State the general rule about comparisons.

Answers

The prompt is zsh — both $SHELL and the user record say /bin/zsh. Scripts run bash 3.2.57.
zsh built 2 arguments; bash built 4.
Because zsh does not word-split an unquoted parameter expansion, so the space in the path survived.
smoke +SEED=1, because zsh also kept both plusargs in one argument.
The zsh one. It ran a simulation, produced a log, reported a verdict and exited 0 — so a flow reading the log records a FAIL for a test name containing a space. The bash failure is exit 255 with a clear message.
Quote the expansion — vvp "$IMAGE" +TESTNAME=smoke +SEED=1. A quoted expansion has no splitting to differ about, so both shells produce identical output.
p in zsh — the first element — and q in bash, the second.
zsh refuses with "no matches found"; bash passes the literal pattern through. The surprise is the direction: the interactive test is the safe one and the script is where a pattern reaches a tool as a filename.
Empty, because zsh's array is pipestatus and is 1-based. exit "" is exit 0.
Test the line in the shell that will run it: /bin/bash -c 'the exact line', not pasted at your prompt.
A child process inherits only environment variables. A variable set at the prompt is a shell variable until it is exported.
One — OLDPWD. 62 against 61.
That "a script gets a different environment" is not true for a script you launch yourself from a terminal.
Three variables, five PATH entries, and the simulator is not reachable.
Only when something else launches the script — a scheduler, a timer, a CI agent. If it fails when you run it yourself, look at the shell, the options and the directory first.
Because scripts change directory and prompts rarely do, so every relative path in the argv resolves against a directory you were not in when you tested the line.
Six times, across Chapters 26.1, 27.8, 30.1, 30.5, 31.6 and 33.1.
Resolve the root once at the top with root=$(cd "$(dirname "$0")/.." && pwd). The diagnostic is to print $PWD next to the failing path.
errexit, nounset, pipefail and xtrace — all off.
It is accepted and ignored, because expand_aliases is off in a non-interactive shell.
/usr/bin/rs, a real BSD array-reshaping program. It is the worst outcome because the result is not "command not found" but a different program, found and executed.
Is it the same shell; the same options; the same directory; the same environment.
Both are answered by making the script's conditions happen at your prompt — they are reproductions rather than inspections, so they settle the question instead of narrowing it.
$SHELL may not be the shell you are in, so the first row can secretly be bash and report no difference — check ps -o comm= -p $PPID. And env -i erases more than any real scheduler, so its row answers a question nobody asked — reproduce the environment you were actually given.
A comparison is only as good as the fidelity of the thing it compares against. A row that is too clean, or secretly identical to another row, gives a confident answer to the wrong question.

6. Remember

The prompt is zsh and the script is bash 3.2.57. A line tested in one is not a line tested in the other.
The same line built 2 arguments in zsh and 4 in bash. zsh does not word-split an unquoted expansion.
The prompt's result was the dangerous one, because it ran, produced a verdict and exited 0 under a test name containing a space.
Quote the expansion and both shells agree exactly.
${a[1]} is the first element in zsh and the second in bash — an off-by-one that arrays of filenames do not announce.
zsh refuses a glob that matches nothing; bash hands the tool a literal *. The interactive test is the safer one.
${PIPESTATUS[0]} is empty under zsh, so a correct status fix goes green under the wrong shell.
Test the line with /bin/bash -c, not at your prompt.
A child inherits only exported variables — and a login shell loses exactly one, OLDPWD, so the environment is a weak suspect until a scheduler is involved. Then it is three variables, five PATH entries, and no simulator.
Scripts change directory and prompts do not. Resolve the root absolutely on the first line; print $PWD beside any failing path.
errexit, nounset, pipefail and xtrace are all off, and expand_aliases is off too — so an alias is accepted, ignored, and the name may resolve to a real program instead.
Four questions in order: shell, options, directory, environment — and the first two are reproductions rather than inspections.
A comparison is only as good as what it compares against. env -i is cleaner than any real scheduler.

7. What Is Next

Three of the four questions are now one command each. Chapter 33.4 takes the fourth apart properly — the failures caused by a variable that is unset, stale, or set to something that was true last month — and builds the check that would have caught each one before any work started.

Where this fits

Part of the Linux & Bash curriculum.