Linux & Bash · Part II · Module 33 · Debugging Failed Jobs and Broken Flows

Missing, Stale or Wrong Environment

A variable can be unset, it can point at something that has gone, or it can point at something real that is not what you meant — and those three go wrong in increasing order of damage and decreasing order of noise. This chapter measures all three against two real project areas, then builds the check that separates them before any tool starts.

Advanced25 min readLinuxPart IIDebuggingEnvironmentValidation

Part II · Module 33 · Chapter 33.4 · Missing, Stale or Wrong Environment

Chapter 33.3 listed the environment as the fourth of four differences and found it the weakest suspect for a script you run yourself. That is true for whether a variable arrives. It is not true for what the variable says, which is this chapter, and which produces the quietest failure in the module.

1. Three Ways, Not One

A flow wrapper that trusts a project root — the shape of almost every real one:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
set -u
test=${1:-smoke}; d=${2:?need an outdir}
mkdir -p "$d"
( cd "$d" && exec vvp "$PROJ_ROOT/logs/sim.vvp" +TESTNAME="$test" +SEED=1 +CYCLES=200 ) > "$d/run.log" 2>&1
printf '%s\n' "$?" > "$d/rc"

The fixture has two complete project areas. proj_a is chip_a, whose arbiter grants the wrong agent first; proj_b is chip_b, whose arbiter is fixed. Each has its own filelist, its own built image, and a PROJECT file naming itself.

Running the same test, prio0, four ways:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  case      script rc  vvp rc  log
  missing   0          1       flow2.sh: line 6: PROJ_ROOT: unbound variable
  stale     0          255     <R>/proj_deleted/logs/sim.vvp: Unable to open input file.
  wrong_a   0          0       TB  test=prio0 seed=1 cycles=200 verbosity=LOW   (TEST FAILED)
  wrong_b   0          0       TB  test=prio0 seed=1 cycles=200 verbosity=LOW   (TEST PASSED)

Read the first column before anything else. The script exited 0 in all four cases, including the two where no simulation ran at all.

`set -u` fired, and it did not help

The missing row is the one that looks handled. set -u is on, the variable is unbound, and bash said so:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

flow2.sh: line 6: PROJ_ROOT: unbound variable

Three things went wrong with that, and all three are structural rather than careless.

The message went into run.log. The failing command is inside ( … ) > "$d/run.log" 2>&1, so the shell's own diagnostic was captured along with the tool's output. A log reader — Chapter 29.1's subject — now sees a line in the simulator's log that was never written by the simulator, and Chapter 32.3's classifier sees a log with no verdict and calls it NO_VERDICT: a testbench problem.

The subshell died and the script continued. set -u exits the shell it is set in, and the shell it was set in was the subshell. The parent carried on, wrote 1 into $d/rc, and returned 0.

And rc=1 is indistinguishable from a failing test. Chapter 32.3 measured this: a simulation that detects a failure and calls $finish exits 0, while a tool problem exits non-zero — so 1 here lands in the same bucket as a genuine tool error, which is the right bucket but for entirely the wrong reason.

So set -u is necessary and nowhere near sufficient. Chapter 20.2 says as much about its limits; the specific lesson here is that a guard whose diagnostic is redirected into an artifact is a guard whose diagnostic is lost — and the redirection is there for good reasons that nobody is going to remove.

2. Severity Is Inverse to Loudness

The three kinds, ordered by how much damage they do:

Missing — the loudest, the least damaging. Something says unbound variable or a path appears as /logs/sim.vvp with an obviously absent prefix. Nothing ran, and it is clear that nothing ran. Minutes.

Stale — louder still, and still not damaging. The variable points at an area that has been archived, renamed or deleted:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

<R>/proj_deleted/logs/sim.vvp: Unable to open input file.

Exit 255, a full path in the message, and the path is visibly wrong once you read it. This is the best failure of the three, and it is worth noticing that it is better than the "handled" case in Section 1 — because the tool's own error message names the file it could not open, while set -u's message names only a variable.

Wrong — silent, and the only one that matters. The variable points at a real project area with a real filelist and a real built image. Everything works:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  -- intended: chip_a, whose arbiter is broken --
    PROJ_ROOT=<R>/proj_a
    ERROR t=35 gnt=10 expected 01   TEST FAILED
  -- actual:   chip_b, whose arbiter is fixed --
    PROJ_ROOT=<R>/proj_b
    TEST PASSED

The same command, two valid roots, opposite verdicts. The second run produced a PASS for a design that was never tested, there is no error anywhere, the exit status is 0, the log is well-formed, and the only evidence is a path in a line nobody compares against anything.

	how it announces itself	what it costs	who notices
missing	`unbound variable`, or a path with a gap in it	minutes	the first run
stale	`Unable to open input file`, with the path	minutes	the first run
wrong	nothing at all	a signed-off block	nobody

This is the same inversion Chapter 33.1 found between a genuinely broken design and a stale image: the failure modes that produce a clean, confident, well-formed result are the expensive ones, and they are expensive precisely because every check downstream passes.

3. The Check

Three failures need three statuses, because they need three different people:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
# checkenv EXPECTED_PROJECT -- assert the environment before any work starts.
# Three distinct failures, three distinct statuses, because they need three
# different people:
#   3  a required variable is not set          -> whoever launched this
#   4  it points at something that is not there -> whoever moved it
#   5  it points at the WRONG project           -> the dangerous one
set -u
want=${1:-}
[ -n "$want" ] || { echo "checkenv: name the project you expect" >&2; exit 2; }
fail() { printf 'checkenv: %s\n' "$1" >&2; exit "$2"; }

# 1. set, and not empty. ${X:?} would exit 1, which collides with a test failure.
[ -n "${PROJ_ROOT:-}" ] || fail "PROJ_ROOT is not set" 3

# 2. it resolves to a directory that exists
[ -d "$PROJ_ROOT" ] || fail "PROJ_ROOT points at '$PROJ_ROOT', which is not a directory" 4
for f in cfg/files.f logs/sim.vvp PROJECT; do
  [ -r "$PROJ_ROOT/$f" ] || fail "PROJ_ROOT has no $f -- this does not look like a project area" 4
done

# 3. it is the project you meant. A valid-but-wrong root is the silent failure:
#    it produces a real run, a real log and a real verdict about another design.
got=$(tr -d ' \n' < "$PROJ_ROOT/PROJECT")
[ "$got" = "$want" ] || fail "PROJ_ROOT is the '$got' project; this flow expects '$want'" 5

printf 'checkenv: %s at %s\n' "$got" "$PROJ_ROOT" >&2

Every branch, exercised:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  CASE                         rc     MESSAGE
  missing                      3      checkenv: PROJ_ROOT is not set
  empty string                 3      checkenv: PROJ_ROOT is not set
  stale (gone)                 4      checkenv: PROJ_ROOT points at '<R>/proj_deleted', which is…
  a dir, not a project         4      checkenv: PROJ_ROOT has no cfg/files.f -- this does not lo…
  WRONG project                5      checkenv: PROJ_ROOT is the 'chip_b' project; this flow exp…
  correct                      0      checkenv: chip_a at <R>/proj_a

Four decisions in that script are worth more than the rest.

It checks the identity, not just the existence

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

got=$(tr -d ' \n' < "$PROJ_ROOT/PROJECT")
[ "$got" = "$want" ] || fail "PROJ_ROOT is the '$got' project; this flow expects '$want'" 5

Existence checks catch missing and stale, which were already loud. Only an identity check catches wrong, and the identity has to come from inside the area — a PROJECT file, a manifest, a version — rather than from the path, because the path is the thing that is wrong.

A check written against the path instead, case $PROJ_ROOT in */chip_a) ok;;, fails in both directions: a legitimate area reached through a symlink or an automounter has a different path, and a copied area has the right path shape with the wrong contents inside.

Empty is treated as missing

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[ -n "${PROJ_ROOT:-}" ] || fail "PROJ_ROOT is not set" 3

PROJ_ROOT="" passes set -u — it is set. And then "$PROJ_ROOT/logs/sim.vvp" is /logs/sim.vvp, an absolute path into the root of the filesystem, which on an unlucky machine exists. Chapter 20.3 makes the general case; [ -n ] rather than a bare set check is what covers it, and the measured row empty string → 3 is the proof that the branch works.

The statuses avoid 1

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#   3  a required variable is not set          -> whoever launched this
#   4  it points at something that is not there -> whoever moved it
#   5  it points at the WRONG project           -> the dangerous one

Not ${PROJ_ROOT:?message}, which is the idiomatic one-liner and exits 1 — the value Chapter 32.3 reserved for a failing test. A setup problem reported as 1 is a setup problem that a regression driver counts as a design failure, which is Module 31's measured collision between ${VAR:?} and a FAIL status.

And all of them stay under 126, per Chapter 21.4.

It refuses when it is not told what to expect

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

[ -n "$want" ] || { echo "checkenv: name the project you expect" >&2; exit 2; }

A check called with no expectation cannot do the thing it exists for. Defaulting to "whatever is there" would make the dangerous case pass silently — the check would validate the environment against itself.

4. Where the Check Goes

At the top of the flow, before anything expensive, and before the run directory is created:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

#!/bin/bash
set -u
root=$(cd "$(dirname "$0")/.." && pwd)
"$root/bin/checkenv" chip_a || exit $?        # pass the status through unchanged

Three properties of that line.

|| exit $? rather than || exit 1. The whole point of three statuses is that they reach the caller. Collapsing them at the first boundary is the mistake Chapter 32.7 spent a chapter on.

Before the run directory exists. A flow that creates runs/smoke_s1/ and then discovers the environment is wrong has left a directory that Chapter 32.4's collector will classify as NOSTATUS — a run that started and produced no verdict. The environment failure is now disguised as a different, more worrying problem.

And its message goes to standard error, un-redirected. Section 1's whole lesson: a diagnostic captured into a run's log is a diagnostic attributed to the tool.

What to check, and the trap in checking too much

The variables worth asserting are the ones whose wrongness is silent, which is a much shorter list than the ones a flow reads:

A project or design root — the wrong case above.

A tool version or path, because Chapter 25.7 measured the same sources through two tool versions being two different experiments, and nothing in a log says which one ran.

A library or technology path — the equivalent of PROJ_ROOT for the back end, with the same property: a valid-but-wrong one produces real numbers about the wrong technology.

And a licence or server variable, though this one fails loudly and belongs on the list only because Chapter 25.4 measured how confusing its loud failure is.

The trap is the opposite excess. A check that asserts thirty variables becomes a check that is wrong whenever the flow legitimately changes, and a check that is wrong gets commented out — which is Chapter 29.4's observation about checkers needing false-positive tests, arriving as an organisational failure rather than a technical one.

The test for whether a variable belongs in the check is a single question: if this were wrong but plausible, would anything else notice? For PROJ_ROOT the answer is no, and that is why it is worth five lines. For a verbosity setting the answer is that nobody cares, and asserting it only creates a reason to delete the script.

5. Diagnosing One That Already Happened

A failure has arrived and you suspect the environment. In order:

Is the variable in the run's record at all? Chapter 32.2's env.txt and Chapter 33.1's cmd.args exist for this. If the flow recorded nothing about its environment, that is the first thing to fix, because every other step is now guesswork.

Does the recorded path still mean the same thing? This is the stale-and-wrong question, and it needs the identity rather than the path:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ grep '^PROJ_ROOT=' regr/…/env.txt
PROJ_ROOT=/proj/chip_a
$ cat /proj/chip_a/PROJECT
chip_b

Which happens more often than it sounds, because areas get re-pointed, symlinks get re-aimed, and a mount gets remapped — and none of those changes the path recorded in last night's env.txt.

Compare the whole environment, not the variable you suspect. Chapter 33.3's harness, applied to the record:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

$ diff <(sort regr/last-good/env.txt) <(sort regr/last-night/env.txt)

Which is the one step that finds the variable you were not thinking about — and it only works if both runs recorded it, which is Section 4's argument for recording more than you expect to read.

And when there is no record, reproduce the environment rather than reasoning about it. env -i for a scheduled job (Chapter 32.8 measured five PATH entries and no simulator), and /bin/bash -c for a shell difference.

Practice

Write a flow wrapper that trusts a root variable, and run it with the variable unset.
Check the script's exit status in that case, and then the tool's.
Find where set -u's message went, and say who gets blamed for it.
Explain why the subshell dying did not stop the script.
Point the variable at a directory that does not exist and compare the message.
Build two project areas whose designs differ, each with its own image.
Run the same test against both and compare the verdicts.
Say what evidence in the second run's output reveals the mistake.
Rank the three kinds by damage, and then by loudness.
Write the check with one status per kind.
Give it an empty string and confirm it is treated as missing.
Give it a directory that is not a project and confirm a different status.
Point it at the wrong project and confirm the identity check fires.
Check the identity from the path instead, and construct the two cases where that fails.
Use ${VAR:?} instead and work out which status it returns and what that collides with.
Call the check with no expectation and confirm it refuses.
Put the check before the run directory is created, then move it after, and see what the collector reports.
Pass the status through with || exit $?, then collapse it to exit 1 and say what is lost.
List the variables in your flow and apply the "would anything else notice" test to each.
Find a recorded PROJ_ROOT from an old run and check whether it still means the same thing.
Diff two runs' whole environments and find a variable you were not suspecting.

Check Your Understanding

What did the script's exit status say in all four cases?
set -u fired on the missing variable. Name the three things that went wrong anyway.
Why does the shell's diagnostic end up attributed to the tool?
Which shell did set -u exit, and what did the parent do next?
Why is rc=1 there indistinguishable from a failing test?
Rank missing, stale and wrong by damage and by loudness.
Why is the stale failure better than the "handled" missing one?
What exactly did the two valid roots produce?
What is the only evidence that the wrong project was used?
What general inversion does this share with Chapter 33.1?
Why must the identity come from inside the area rather than from the path?
Give the two cases where a path-based identity check fails.
Why is an empty string treated as missing, and what does "$PROJ_ROOT/logs/sim.vvp" become?
Why avoid status 1, and what does ${VAR:?} return?
Why does the check refuse when given no expectation?
Why || exit $? rather than || exit 1?
Why must the check run before the run directory is created?
Why must its message not be redirected?
Which four kinds of variable are worth asserting, and what do they have in common?
What is the trap in checking too much?
State the single test for whether a variable belongs in the check.
What are the four diagnostic steps for a failure that already happened?
Why can a recorded path stop meaning the same thing?
Which diagnostic step finds the variable you were not thinking about, and what does it require?

Answers

Zero, in all four — including the two where no simulation ran at all.
Its message went into run.log; the subshell died while the parent continued; and the recorded rc=1 is indistinguishable from a tool error.
Because the failing command is inside ( … ) > run.log 2>&1, so the shell's own diagnostic is captured into the simulator's log.
The subshell. The parent carried on, wrote 1 into the rc file, and returned 0.
Because a simulation that detects a failure exits 0 while a tool problem exits non-zero, so 1 lands in the tool bucket — the right bucket for the wrong reason.
Damage: wrong ≫ stale ≈ missing. Loudness: stale > missing ≫ wrong. The order is inverted.
Because the tool's message names the file it could not open, while set -u's message names only a variable.
chip_a gave ERROR t=35 gnt=10 expected 01 and TEST FAILED; chip_b gave TEST PASSED — opposite verdicts from the same command.
A path in a line nobody compares against anything. There is no error, the status is 0, and the log is well-formed.
That the failure modes producing a clean, confident, well-formed result are the expensive ones — expensive because every downstream check passes.
Because the path is the thing that is wrong.
A legitimate area reached through a symlink or automounter has a different path; and a copied area has the right path shape with the wrong contents inside.
Because an empty variable passes set -u — it is set. "$PROJ_ROOT/logs/sim.vvp" becomes /logs/sim.vvp, an absolute path into the filesystem root.
Because 1 is the value reserved for a failing test, so a setup problem reported as 1 gets counted as a design failure. ${VAR:?} exits 1.
Because a check with no expectation would validate the environment against itself, letting the dangerous case pass.
Because the whole point of three statuses is that they reach the caller; collapsing them at the first boundary discards the distinction.
Because a created-then-abandoned run directory is classified as NOSTATUS — a run that started and produced no verdict — which disguises the environment failure as something more worrying.
Because a diagnostic captured into a run's log is a diagnostic attributed to the tool.
A project or design root, a tool version or path, a library or technology path, and a licence variable. The first three share the property that being wrong-but-plausible is silent.
A check asserting thirty variables is wrong whenever the flow legitimately changes, and a check that is wrong gets commented out.
If this were wrong but plausible, would anything else notice?
Is the variable recorded at all; does the recorded path still mean the same thing; diff the whole environment between a good run and a bad one; and when there is no record, reproduce the environment rather than reasoning about it.
Because areas get re-pointed, symlinks re-aimed and mounts remapped, and none of those changes the path written in last night's record.
Diffing two runs' whole environments — and it requires that both runs recorded it, which is the argument for recording more than you expect to read.

6. Remember

Three kinds, not one: missing, stale, and wrong. Severity is inverse to loudness.
The script exited 0 in all four measured cases, including the two where nothing ran.
set -u fired and did not help: its message went into run.log, the subshell died while the parent continued, and the recorded 1 looks like a tool error.
A guard whose diagnostic is redirected into an artifact is a guard whose diagnostic is lost.
The stale failure is the best of the three, because the tool names the file it could not open.
A valid-but-wrong root produced TEST PASSED for a design that was never tested. Same command, two roots, opposite verdicts, no error anywhere.
Only an identity check catches it — read a marker from inside the area, never infer it from the path.
Treat an empty string as missing. PROJ_ROOT="" passes set -u and makes the image path /logs/sim.vvp.
One status per kind, none of them 1, and ${VAR:?} exits 1.
Refuse when not told what to expect, or the check validates the environment against itself.
Run it before the run directory is created, pass the status through with || exit $?, and keep its message on un-redirected stderr.
Assert only the variables whose wrongness would be silent — a check that is wrong gets commented out.
To diagnose one after the fact: diff the whole recorded environment, which only works if it was recorded.

7. What Is Next

The environment is now asserted rather than assumed. Chapter 33.5 takes on the defect class behind more flow bugs than any other — and the one Chapter 33.3 already met twice, where one line built two arguments in one shell and four in another.

Where this fits

Part of the Linux & Bash curriculum.