Debugging on the Machine Where It Happened

Part II · Module 33 · Chapter 33.8 · Debugging on the Machine Where It Happened

Chapter 33.2 replayed a recorded argv. Chapter 33.6 traced one run. Both assume the run can happen again, here, now.

This chapter is the failure that cannot, because the thing that went wrong was not in the command — it was in the machine.

1. The Run That Passes Everywhere Except Where It Ran

A ulimit -f of 20 blocks, which is 20 KB — the kind of cap a scheduler applies to stop one job filling a shared filesystem:

$ ( ulimit -f 20; cd cap/cap1 && exec vvp …/logs/sim.vvp \
      +TESTNAME=random +SEED=1 +CYCLES=200 +DUMP ) > cap/cap1/run.log 2>&1
  rc=153  run.log=0 bytes  dump=10240 bytes
  verdict in the log:

The same command with no cap:

  rc=0  run.log=229 bytes  dump=18678 bytes  verdict=TEST PASSED

A passing run became exit 153 with a zero-byte log. The waveform is there and truncated at 10,240 bytes; the log is empty; there is no verdict, no error message, and no indication anywhere of what stopped it.

The status is the only clue, and it is a real one:

$ trap -l | tr ',' '\n' | grep XFSZ
 25) SIGXFSZ
$ echo $((128 + 25))
153

128 + 25 — killed by SIGXFSZ, the signal for exceeding a file-size limit, which Chapter 11.1 established as the shell's encoding and Chapter 30.6 measured arriving from a scheduler's own cap.

2. What Survives, and What Is Gone in Seconds

  FACT                   recoverable? why
  the test and seed      yes          in results.tsv
  the argv               yes          in cmd.args, if recorded
  the image identity     yes          a digest, if recorded
  which node ran it      NO           nothing else records it
  that node load         NO           gone within seconds
  the limits in force    NO           per-process, not logged
  free memory then       NO           gone
  the PATH it had        NO           not in the tool log

The top three are what Module 32 and Chapter 33.1 spent their effort on, and they are all properties of the request. They are written down because somebody chose to write them down, and they stay true.

The bottom five are properties of the moment. A load average is meaningless ten seconds later. A ulimit exists only inside one process tree. Free memory is a reading, not a fact. And none of them appears in any tool's log, because the tool does not know them either.

So the rule is blunt: capture the machine at the time, or never know. There is no later.

3. The Snapshot

#!/bin/bash
# snapshot OUTDIR -- record the machine-local facts an investigation cannot
# recover afterwards. Everything here is either gone or different by the time
# somebody reads the log: the node, its limits, its PATH, its free memory.
set -u
d=${1:-}
[ -n "$d" ] || { echo "snapshot: need an output directory" >&2; exit 2; }
mkdir -p "$d" || exit 2
{
  printf 'when\t%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  printf 'host\t%s\n' "$(hostname)"
  printf 'user\t%s\n' "$(id -un)"
  printf 'cwd\t%s\n' "$PWD"
  printf 'shell\t%s\n' "${BASH_VERSION:-not-bash}"
  printf 'cores\t%s\n' "$(/usr/sbin/sysctl -n hw.ncpu 2>/dev/null || getconf _NPROCESSORS_ONLN 2>/dev/null || echo ?)"
  # absolute paths: a snapshot that depends on PATH records nothing in the
  # environment where a snapshot matters most (measured -- loadavg came back
  # empty under env -i because sysctl was not reachable).
  printf 'loadavg\t%s\n' "$(/usr/sbin/sysctl -n vm.loadavg 2>/dev/null | tr -d '{}' | sed 's/^ *//')"
  for t in iverilog vvp; do
    p=$(command -v "$t" 2>/dev/null)
    printf 'tool.%s\t%s\n' "$t" "${p:-NOT-ON-PATH}"
  done
  printf 'iverilog.version\t%s\n' "$(iverilog -V 2>&1 | head -1)"
  printf 'path.entries\t%s\n' "$(printf '%s' "$PATH" | tr ':' '\n' | grep -c .)"
} > "$d/machine.tsv"
printf '%s\n' "$PATH" | tr ':' '\n' > "$d/path.txt"
env | sort > "$d/env.txt"
ulimit -a > "$d/ulimit.txt" 2>&1
/bin/df -k . > "$d/df.txt" 2>&1
/usr/bin/vm_stat 2>/dev/null | head -6 > "$d/vm_stat.txt"
printf 'snapshot: %s\n' "$d" >&2

Interactively:

  when              2026-10-09T12:19:56Z
  host              MacBook-Air.local
  user              azharuddinsaifi
  cwd               <R>
  shell             3.2.57(1)-release
  cores             8
  loadavg           41.61 37.37 30.57
  tool.iverilog     /opt/homebrew/bin/iverilog
  tool.vvp          /opt/homebrew/bin/vvp
  iverilog.version  Icarus Verilog version 13.0 (stable) (v13_0)
  path.entries      15

And from the environment a scheduled job actually gets:

  tool.iverilog     NOT-ON-PATH
  tool.vvp          NOT-ON-PATH
  iverilog.version  bin/snapshot: line 21: iverilog: command not found
  path.entries      5

NOT-ON-PATH for both tools and five PATH entries instead of fifteen — Chapter 32.8's measured gap, now recorded in the one place an investigator will look.

Note that iverilog.version captured the literal command not found text rather than an empty field. That is deliberate: an empty field says nothing, while the error message says exactly what happened. A snapshot's job is to record, including recording failures to record.

4. Where the Snapshot Goes

Inside the run directory, taken by the job, as its first action:

#!/bin/bash
set -u
root=$(cd "$(dirname "$0")/.." && pwd)
d=$1
mkdir -p "$d"
"$root/bin/snapshot" "$d/machine" || echo "snapshot failed" >&2   # never fatal
"$root/bin/checkenv" chip_a || exit $?                            # 33.4
… the run …

Four properties, each from something measured earlier.

By the job, not by the launcher. The launcher runs on the submitting host; the job runs on the node. Chapter 30.5 measured that submission changes the environment, the working directory and the limits — so a snapshot taken by the launcher describes a machine that did not run anything.

First, before checkenv. The environment check will exit on a bad environment, and a bad environment is precisely when you want the snapshot. Ordering them the other way loses the evidence for the failure most likely to occur.

Never fatal. || echo … >&2 and carry on. A flow that refuses to run because it could not record the load average has converted a diagnostic into an outage.

And inside the run directory, so Chapter 32.8's archive carries it and a run found three weeks later is self-describing. The cost is small — machine.tsv, path.txt, env.txt, ulimit.txt, df.txt and vm_stat.txt are a few kilobytes per run — and it compresses with everything else at that chapter's measured 2.3%.

5. Getting Onto the Node

Sometimes the snapshot is not enough and you need the machine. Three routes, in order of preference:

Reproduce the environment locally. env -i is a free, exact local reproduction of the PATH problem, and ulimit is a free local reproduction of the cap:

$ ( ulimit -f 20; … )           # reproduces exit 153 on your own desk

That one line is why Section 1's measurement was possible at all, and it settles a large fraction of node-only failures without a node.

Run an interactive job on the same node class. Every scheduler has a form of this — an interactive submission that gives you a shell on a node under the same limits. There is no scheduler on this machine (Chapter 30.5 measured their complete absence), so the honest statement is that the shape of the command is what matters: ask for a shell with the same resource request as the failing job, and then run the reproduction from Chapter 33.2 inside it.

And instrument the job rather than watching it. A node you can log into is a node whose state you are changing. Chapter 30.3 built the sampling; the point here is that a sampler writing into the run directory produces evidence that survives, while a person watching top produces a recollection.

6. The Module, Assembled

Module 33 is eight techniques and one order:

The order is not decorative. Steps 1 and 2 are cheap and eliminate the two most expensive mistakes in the module — investigating a stale image, and debugging something that does not reproduce. Step 7 belongs late in the chapter order and early in a real morning, because it decides how many times you will run steps 1 through 6.

And one theme runs through all eight, which is worth stating as the module's conclusion. Every chapter found the same shape: the failure modes that produce a clean, confident, well-formed result are the expensive ones. A byte-identical log from three causes. A reproduction that passes. A command that runs with a silently wrong test name. A variable pointing at a real project that is the wrong one. An argc of three with every value wrong. A trace that reclassified a pass. Ten failures that were four bugs. A zero-byte log from a cap nobody recorded.

None of those announced itself. All of them were found by comparing two things that should have agreed — two logs, two shells, two project areas, two argument counts, two nights' signatures, a run and its replay.

Practice

1. Apply a ulimit -f to a passing run and record the exit status and the log size.
2. Decode that status from trap -l.
3. Run your classifier and your triage on it and see which layer they name.
4. Explain why neither can see the cause.
5. List which facts about a run are recoverable later and which are gone.
6. Write a snapshot of the machine-local facts.
7. Run it interactively and under env -i and diff the two.
8. Find a field that came back empty in the clean environment.
9. Survey which of your snapshot's own helpers survive a clean PATH.
10. Name every helper absolutely and re-run.
11. Make a missing value loud rather than blank, and say why.
12. Take the snapshot by the job rather than the launcher, and explain the difference.
13. Put it before the environment check, then after, and say what the second order loses.
14. Make the snapshot non-fatal and justify it.
15. Put it in the run directory and confirm your archive carries it.
16. Reproduce a cap locally with ulimit and confirm you get the same status.
17. Reproduce a PATH problem locally with env -i.
18. Work out what a login on the node would and would not tell you.
19. List the questions a login is genuinely good for and add them to the snapshot.
20. Walk the eight steps on a failure from your own flow.
21. For each of this module's findings, name the two things that were compared.

Check Your Understanding

1. What did a 20 KB file-size cap do to a passing run?
2. What is 153, and how do you confirm it?
3. What did the classifier and the triage say, and were they wrong?
4. Why can neither name the cause?
5. What is the defining property of this class of failure?
6. Which three facts about a run survive, and why?
7. Which five do not, and what do they have in common?
8. State the rule that follows.
9. What did the snapshot record differently under env -i?
10. Why does it keep the command not found text instead of an empty field?
11. Which of the snapshot's own helpers was missing from a clean PATH, and where does it live?
12. Why is that failure circular?
13. How many times has that mechanism appeared across the last two modules?
14. What should a missing value look like?
15. Why must the job take the snapshot rather than the launcher?
16. Why must it come before the environment check?
17. Why must it never be fatal?
18. Why put it inside the run directory?
19. Give the three routes onto a node, in order of preference.
20. Give the three reasons a login is the worst of them.
21. What is a login genuinely good for?
22. Why is step 7 late in the chapters and early in a morning?
23. State the theme running through all eight chapters.
24. What found every one of those failures?

7. Remember

• A 20 KB file-size cap turned a passing run into exit 153 with a zero-byte log — and 153 is 128 + 25, SIGXFSZ.
• The classifier said CRASH and the triage said "tool layer". Both correct, both naming the wrong cause, because neither can see a limit nobody recorded.
• This class of failure destroys its own evidence, so the answer is prevention, not investigation.
• The request survives and the moment does not. Test, seed, argv and image digest are recoverable; node, load, limits, memory and PATH are gone in seconds.
• Capture the machine at the time, or never know.
• Record the failure to record: keep the command not found text, not an empty field, and make a missing value loud.
• Name every helper absolutely. sysctl lives in /usr/sbin and is off a clean PATH, so the snapshot blanked the one field that mattered — the third time that mechanism has struck in two modules.
• The job takes the snapshot, not the launcher; before the environment check; never fatally; and into the run directory so the archive carries it.
• env -i and ulimit are free, exact, local reproductions — and they settle most node-only failures without a node.
• A login on the node is the worst of three routes: wrong environment, uninherited limits, and a machine whose state has moved on. Use it for durable facts only.
• Eight steps, in order — layer, reproducibility, the four differences, the environment, the argv, the trace, the classification, the machine.
• Across all eight: the clean, confident, well-formed result is the expensive one, and every finding came from comparing two things that should have agreed.

8. What Is Next

Module 33 assumed the flow was already written and something had gone wrong with it. Module 34, Defensive Automation Around EDA Tools, inverts that: writing the wrapper so these failures cannot be silent in the first place — tools that exit zero on failure, output that has to be validated rather than trusted, and the checks that turn a quiet wrong answer into a loud refusal.