Recording the Exact Command That Ran

Part II · Module 27 · Chapter 27.9 · Recording the Exact Command That Ran

Module 27 began with three stages whose exit statuses all said success on a design with no DUT in it, and it ends here because every chapter since has added something that will be asked about later. Which test ran. Which seed. Which simulator, from which path, at which version. Whether the dump is complete. A result nobody can reproduce is an anecdote, and the difference between a result and an anecdote is about forty lines of shell.

1. What Has to Be Written Down

Not the whole environment — that is unreadable and most of it is irrelevant. Five things, each because omitting it has cost somebody a day:

The command, as arguments. Not as a sentence. Section 3 is about why.

Where it ran, and as whom. The working directory, because Chapter 26.6 measured that filelist paths resolve against it; the user and host, because a result that only reproduces for one person is a finding about the environment.

The tool's identity, resolved. Not vvp but /opt/homebrew/bin/vvp, plus its version, size and modification time. A version string alone is not an identity — two builds can report the same version, which section 4 demonstrates by building one.

The variables that change a tool's behaviour. PATH from Chapter 25.2, the library paths from Chapter 25.3, the licence variables from Chapter 25.4, and the project root. A short named list, not env in its entirety — which is Chapter 25.8's discipline, and its section on what not to capture is the reason this one is six names long.

The outcome. The exit status, and the time it started and finished.

2. The Recorder

#!/bin/bash
# record-run RUNDIR -- CMD...   Run CMD in RUNDIR and write down enough to repeat it.
set -u
rundir=${1:?usage: record-run RUNDIR -- CMD...}; shift
[ "${1:-}" = "--" ] && shift
[ $# -gt 0 ] || { echo "record-run: no command given" >&2; exit 2; }
mkdir -p "$rundir" || exit 2
rundir=$(cd "$rundir" && pwd)

# 1. the command, one argument per line — the only form that survives spaces
: > "$rundir/cmd.args"
for a in "$@"; do printf '%s\n' "$a" >> "$rundir/cmd.args"; done
# 2. a human-readable single line, quoted so it can be pasted back
{ printf '%q ' "$@"; printf '\n'; } > "$rundir/cmd.sh"
# 3. where it ran and as whom
{ printf 'cwd       %s\n' "$PWD"
  printf 'rundir    %s\n' "$rundir"
  printf 'user      %s\n' "$(id -un)"
  printf 'host      %s\n' "$(hostname -s)"
  printf 'started   %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
  printf 'shell     %s\n' "$BASH_VERSION"
} > "$rundir/context.txt"
# 4. the tool, resolved, with its version AND its identity on disk — a version
#    string alone is not enough: two builds can report the same version.
tool=$(command -v "$1" 2>/dev/null || echo "$1")
{ printf 'argv0     %s\n' "$1"
  printf 'resolved  %s\n' "$tool"
  printf 'mtime     %s\n' "$( [ -e "$tool" ] && date -u -r "$tool" '+%Y-%m-%dT%H:%M:%SZ' || echo unknown )"
  printf 'size      %s\n' "$( [ -e "$tool" ] && wc -c < "$tool" | tr -d ' ' || echo unknown )"
  printf 'version   %s\n' "$("$1" -V 2>&1 | head -1 || true)"
} > "$rundir/tool.txt"
# 5. only the variables that change a tool's behaviour, not the whole environment
for v in PATH LD_LIBRARY_PATH DYLD_LIBRARY_PATH LM_LICENSE_FILE PROJECT_ROOT LANG; do
  eval "val=\${$v:-}"
  printf '%s=%s\n' "$v" "$val"
done > "$rundir/env.txt"

( cd "$rundir" && "$@" ) > "$rundir/run.log" 2>&1
rc=$?
printf 'exit %d\n' "$rc" > "$rundir/status.txt"
printf 'finished  %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" >> "$rundir/context.txt"
exit $rc

$ record-run rec/r1 -- vvp "$PWD/build.vvp" +TESTNAME=random +SEED=11 +CYCLES=500 +STRICT
$ echo $?
0
$ ls rec/r1
cmd.args  cmd.sh  context.txt  env.txt  run.log  status.txt  tool.txt
$ cat rec/r1/cmd.sh
vvp /…/m27/build.vvp +TESTNAME=random +SEED=11 +CYCLES=500 +STRICT
$ grep -E 'resolved|version' rec/r1/tool.txt
resolved  /opt/homebrew/bin/vvp
version   Icarus Verilog runtime version 13.0 (stable) (v13_0)

Two details in the implementation are worth pointing out.

rundir=$(cd "$rundir" && pwd) turns a relative path into an absolute one after the directory exists. The record then names a location that is meaningful from anywhere, which matters because the record outlives the shell that made it.

$("$1" -V 2>&1 | head -1 || true) captures standard error as well as standard output. That is not defensive padding: Chapter 25.7 measured that vvp -V writes its version to stderr while iverilog -V writes to stdout, from the same package. Capturing only stdout records an empty version string for half the tools in one toolchain.

3. Why the Arguments Go One Per Line

cmd.sh is pleasant to read and dangerous to use. The record that matters is cmd.args, and here is the measurement that separates them.

$ record-run rec/r2 -- vvp "$PWD/build.vvp" +TESTNAME=smoke \
>     "+DUMP_FILE=my dump.vcd" +DUMP +STRICT
$ ls rec/r2/*.vcd
rec/r2/my dump.vcd

The run really did create a file whose name contains a space. Now the two records of that command:

$ n=0; while IFS= read -r a; do n=$((n+1)); printf '  [%d] %s\n' "$n" "$a"; done < rec/r2/cmd.args
  [1] vvp
  [2] /…/m27/build.vvp
  [3] +TESTNAME=smoke
  [4] +DUMP_FILE=my dump.vcd
  [5] +DUMP
  [6] +STRICT

Six arguments, with the fourth containing a space. And the human-readable line, re-split the way a script naturally would:

$ line=$(cat rec/r2/cmd.sh)
$ set -- $line
$ printf 're-split: %d words; arg4=%s arg5=%s\n' "$#" "$4" "$5"
re-split: 7 words; arg4=+DUMP_FILE=my\ arg5=dump.vcd

Seven words. The fourth argument split in two, every argument after it shifted by one position, and the trailing backslash from %q quoting is now part of a value. A replay built from cmd.sh would pass +DUMP_FILE=my\ and dump.vcd as separate arguments, and the second of those is a plusarg-shaped nothing that the simulator will silently ignore — Chapter 27.4's failure mode, reached by way of a record.

4. A Record Is Not a Record Until It Replays

Everything above is a claim about sufficiency, and a claim about sufficiency has to be tested by use.

#!/bin/bash
# replay RUNDIR — re-run exactly what record-run recorded, reporting any drift.
set -u
rundir=${1:?usage: replay RUNDIR}
[ -r "$rundir/cmd.args" ] || { echo "replay: $rundir has no cmd.args" >&2; exit 2; }

args=()
while IFS= read -r line; do args+=("$line"); done < "$rundir/cmd.args"
[ ${#args[@]} -gt 0 ] || { echo "replay: empty command" >&2; exit 2; }

drift=0
was=$(grep '^resolved ' "$rundir/tool.txt" | awk '{print $2}')
now=$(command -v "${args[0]}" 2>/dev/null || echo "${args[0]}")
[ "$was" = "$now" ] || { echo "DRIFT tool path: recorded $was, now $now"; drift=1; }
wasv=$(sed -n 's/^version   //p' "$rundir/tool.txt")
nowv=$("${args[0]}" -V 2>&1 | head -1 || true)
[ "$wasv" = "$nowv" ] || { echo "DRIFT tool version: recorded '$wasv', now '$nowv'"; drift=1; }
wasp=$(sed -n 's/^PATH=//p' "$rundir/env.txt")
[ "$wasp" = "${PATH:-}" ] || { echo "DRIFT PATH differs from the recorded one"; drift=1; }

out="$rundir/replay"
mkdir -p "$out"
( cd "$rundir" && "${args[@]}" ) > "$out/run.log" 2>&1
rc=$?
oldrc=$(sed -n 's/^exit //p' "$rundir/status.txt")
printf 'replayed: exit %s (recorded %s), drift=%d\n' "$rc" "$oldrc" "$drift"
if diff -q "$rundir/run.log" "$out/run.log" >/dev/null 2>&1; then
  echo "output identical to the recorded run"
else
  echo "output DIFFERS from the recorded run:"; diff "$rundir/run.log" "$out/run.log" | head -6 | sed 's/^/  /'
fi
[ "$rc" = "$oldrc" ] && [ "$drift" = 0 ]

$ replay rec/r1
replayed: exit 0 (recorded 0), drift=0
output identical to the recorded run
$ echo $?
0

Byte-identical output. Which proves nothing yet, because a check that has only ever agreed has not been shown capable of disagreeing.

5. Proving the Replay Can Fail

Three deliberate perturbations.

A different PATH:

$ PATH="/usr/local/bin:$PATH" replay rec/r1
DRIFT PATH differs from the recorded one
replayed: exit 0 (recorded 0), drift=1
output identical to the recorded run
$ echo $?
1

The output was still identical — the same tool was still found — and the replay still returned 1, because the environment it ran in was not the environment that was recorded. That is the right answer: the result reproduced, and the conditions were not the ones claimed.

A different tool build with the same name. This is the case a version string cannot catch, so here is one constructed on purpose:

$ cat shim/vvp
#!/bin/bash
# a wrapper that forwards to the real vvp but adds one line of output
echo "SHIM active"
exec /opt/homebrew/bin/vvp "$@"

$ PATH="$PWD/shim:$PATH" replay rec/r1
DRIFT tool path: recorded /opt/homebrew/bin/vvp, now /…/m27/shim/vvp
DRIFT tool version: recorded 'Icarus Verilog runtime version 13.0 (stable) (v13_0)', now 'SHIM active'
DRIFT PATH differs from the recorded one
replayed: exit 0 (recorded 0), drift=1
output DIFFERS from the recorded run:
  0a1
  > SHIM active
$ echo $?
1

Three drifts and an output difference. Note which check caught it first: the resolved path, before the version was even consulted. A shim that forwarded silently — without the extra line — would have produced identical output and an identical version string, and only the path check would have noticed. That is why record-run stores the resolved path, size and mtime rather than trusting the version banner, and it is the same lesson as Chapter 25.2's: the tool you ran is the one PATH found, not the one you think is installed.

And a defect in my own probe, worth more than the two above. The first time I ran these perturbations I piped the output through sed to indent it and read $? afterwards:

# WRONG -- $? is sed's status, not replay's
$ ( export PATH="$PWD/shim:$PATH"; replay rec/r1 ) | sed 's/^/  /'; echo "exit=$?"
…
exit=0

# RIGHT
$ out=$( PATH="$PWD/shim:$PATH" replay rec/r1 2>&1 ); echo "exit=$?"
exit=1

The drift detector was working perfectly and I nearly recorded it as broken, because a pipeline's $? is the last command's status. That is Chapter 12.8's rule, and it bites hardest in exactly this situation: while testing whether a check can fail, with a formatting pipe in the way.

6. Wiring It Into the Wrapper

In practice the recorder is not a separate command — the wrapper writes the record itself, because it already knows everything:

# in runsim, before the tool runs
{ printf 'sim %s\ntest %s\nseed %s\ndump %s\n' "$sim" "$test" "$seed" "$dump"
  printf 'rargs %s\n' "$(printf '%s ' ${args[@]+"${args[@]}"})"
} > "$rundir/config.txt"
: > "$rundir/cmd.args"
for a in vvp sim.vvp ${args[@]+"${args[@]}"}; do printf '%s\n' "$a" >> "$rundir/cmd.args"; done

# and after
{ printf 'verdict %s\n' "$verdict"
  printf 'run_rc %s\n' "$rrc"
  printf 'dump_bytes %s\n' "$( [ -f "$rundir/dump.vcd" ] && wc -c < "$rundir/dump.vcd" | tr -d ' ' || echo 0 )"
} >> "$rundir/config.txt"

The run directory then holds everything the module has produced, and every one of those files answers a question somebody will ask:

That table is the real output of Module 27. Not the wrapper — the run directory, which is self-describing, self-validating and replayable, and which the next five modules all consume.

Practice

1. Write record-run and confirm it produces all seven files for one invocation.
2. Check whether your tool writes its version to stdout or stderr, and confirm your recorder captures it either way.
3. Record a command with a space in one argument and confirm the run created the file with the space in its name.
4. Count the arguments in cmd.args, then re-split cmd.sh with set -- $line and count the words. Explain the difference.
5. Identify which argument shifted in the re-split version, and what the simulator would do with the fragment.
6. Write replay and confirm it reports no drift and identical output immediately after a recording.
7. Change PATH and replay. Confirm the output is still identical and the exit status is not 0.
8. Build a shim with the same name as your tool, put it earlier in PATH, and replay. Note which check fires first.
9. Make the shim silent — no extra output — and determine which checks still catch it.
10. Read $? after a pipe on a drifting replay and watch it report success. Then read it without the pipe.
11. Move the source tree between a recording and a replay, and confirm the replay reports no drift while running different RTL.
12. Write the run-directory table for your own flow, and find a question it cannot answer.

Check Your Understanding

1. Name the five categories a run record has to contain.
2. Why store the resolved tool path rather than just its name and version?
3. Why must a version probe capture standard error as well as standard output?
4. What does cmd.args do that cmd.sh cannot?
5. How many arguments were recorded in the measured example, and how many words did the one-liner re-split into?
6. What happens to the arguments after the one containing a space?
7. Why does a replay with only a PATH difference return non-zero even when the output is identical?
8. Which drift check caught the shim first, and why does that ordering matter?
9. Which checks would a silent shim defeat?
10. Why did the author's first perturbation test report success for a replay that had detected drift?
11. Name a change the record cannot detect, and say what would be needed to detect it.

7. Remember

• A result nobody can reproduce is an anecdote. The difference is about forty lines of shell.
• Record five things: the argument vector, the place and identity, the tool's resolved build, the behaviour-changing variables, and the outcome.
• Store the resolved tool path, size and mtime — a version string is a description, not an identity.
• Capture stderr when probing a version: within one toolchain, vvp -V writes to stderr and iverilog -V to stdout.
• One argument per line is the replayable record. The pasteable one-liner re-split from six arguments into seven words, shifting everything after the space.
• Two records, two purposes. Humans read cmd.sh; scripts read cmd.args. Reading the wrong one is the defect.
• A record is not a record until it replays, and a replay is not a check until it has been shown able to disagree.
• Report drift even when the output matches — the result reproduced, the conditions did not.
• Do not read $? through a pipe while testing whether a check can fail.
• The run directory is the module's real output: self-describing, self-validating, replayable.
• Know the limits: the record pins the tool, not the source contents, not the machine, and not the verdict.

8. What Is Next

Module 27 hands over a run directory that can be read, validated and replayed. Module 28 asks the next question about it: of everything in there, what is source, what is derived, what is safe to delete — and how to clean up a working area without destroying the evidence of the failure you were about to debug.