Reproducibility: Why the Same Command Gives Two Answers

Part II · Module 22 · Chapter 22.6 · Reproducibility: Why the Same Command Gives Two Answers

"It works for me" is not a claim about the command. It is a claim about everything the command did not say.

A command line is a few dozen characters. What actually runs is decided by that, plus the environment, plus the filesystem, plus which of several identically-named programs is first on PATH, plus what is left over from last time. Every one of those is invisible in the thing you pasted into chat.

1. The Six Causes, Cheapest First

Each section below is one box, and each ends with the command that settles it.

2. Which Program Actually Ran

The command line names a program. It does not say which program, and the answer is decided by PATH order, shell functions and aliases.

Here is that difference manufactured deliberately, so the mechanism is visible:

$ iverilog -V | head -1
Icarus Verilog version 13.0 (stable) (v13_0)
$ PATH="$PWD/fake:$PATH" iverilog -V | head -1
Icarus Verilog version 99.0 (FAKE)

Same command, two tools, and nothing in the command line distinguishes them. If one of those is on your PATH and the other on a colleague's, the two of you are running different software while believing you are running the same.

That is not a contrived example. Chapter 21.6 §4 measured a real instance on this very machine: grep is a shell function dispatching to ugrep, not /usr/bin/grep:

$ type -a grep
grep is a shell function from /Users/engineer/.../snapshot-zsh-….sh
grep is /usr/bin/grep
$ grep -P 'a\w+' g.txt ; echo "rc=$?"
abc1
rc=0
$ /usr/bin/grep -P 'a\w+' g.txt ; echo "rc=$?"
grep: invalid option -- P
rc=2

A script that greps with -P works in one engineer's interactive shell and fails in a batch job, because an interactive shell loads a function definition the batch shell does not. The command is identical; the program is not.

The four questions that identify a tool, in order:

$ type -a iverilog
iverilog is /opt/homebrew/bin/iverilog
$ command -v iverilog
/opt/homebrew/bin/iverilog
$ cd -- "$(dirname -- "$(command -v iverilog)")" && pwd -P
/opt/homebrew/bin
$ iverilog -V | head -1
Icarus Verilog version 13.0 (stable) (v13_0)

type -a first, because it is the only one that reveals a function or alias — and read its first line, which is the one that runs. command -v prints a path for a program and a bare name for a function, so a bare name there is itself the answer. Then resolve the directory physically, because a PATH entry can be a symlink into a versioned tree. And finally ask the tool, because two paths can hold two versions of the same name.

3. The Locale

The locale changes what text operations mean, which makes it the most surprising entry on the list — nobody expects sort to be configuration.

$ cat t.txt
test_B
test_a
test_C
test_b
$ LC_ALL=C sort t.txt
test_B
test_C
test_a
test_b
$ LC_ALL=en_US.UTF-8 sort t.txt
test_a
test_b
test_B
test_C

Under LC_ALL=C the order is by byte value, so all uppercase precedes all lowercase. Under a UTF-8 locale it is by collation, which interleaves case. A test list sorted on two machines is two different lists, and a regression that runs tests in file order therefore runs them in a different order — which matters the moment any test has a side effect on the next.

And the sharper version, from Chapter 22.1 §5, because it changes which files are selected rather than their order:

$ LC_ALL=C bash -c 'case B in [a-z]) echo yes;; *) echo no;; esac'
no
$ LC_ALL=en_US.UTF-8 bash -c 'case B in [a-z]) echo yes;; *) echo no;; esac'
yes

The current setting is worth knowing, and it is two variables rather than one:

$ echo "LANG=${LANG:-unset} LC_ALL=${LC_ALL:-unset}"
LANG=C.UTF-8 LC_ALL=unset
$ locale | head -2
LANG="C.UTF-8"
LC_COLLATE="C.UTF-8"

LC_ALL overrides everything when set; LANG is the fallback; and individual categories such as LC_COLLATE sit between them. Which is why checking only LANG is not enough — locale with no arguments shows what is actually in effect.

The rule for any script whose output is compared, diffed or sorted: set the locale yourself.

export LC_ALL=C      # byte order, everywhere, deterministically

That belongs near the top of any script that sorts a test list, diffs two reports, or uses a character range. It costs nothing and removes a whole category of divergence. The one caveat is that it also makes [[:upper:]]-style classes ASCII-only, which is what you want for tool output and not what you want for human text.

4. The Working Directory, and Whether It Is One Place

Covered in Chapter 22.3 §4 and recalled here because it belongs on this list: a path can resolve, be correct, and still not be where the other machine's files are.

$ pwd
/var/folders/7g/8jhn.../tmp.NTLIZ8kj4T/alias
$ pwd -P
/private/var/folders/7g/8jhn.../tmp.NTLIZ8kj4T/proj

The one-line check:

[ "$(pwd)" = "$(pwd -P)" ] || echo "NOTE: this path goes through a symbolic link"

And the related cause that is not about links at all: a relative path means a different thing depending on where the command was run from. A flow script that says -f files.f works from one directory and fails from another, and the error names a file rather than a directory. The fix is the idiom from Chapter 21.6 §4, resolved physically:

here=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)

5. The umask

The permissions on everything a run creates come from the umask of the shell that created it, and the umask is per-shell configuration that nobody thinks about.

$ for m in 022 077 002; do (umask $m; : > f; stat -f "umask %s -> %Sp" f); done
umask 022 -> -rw-r--r--
umask 077 -> -rw-------
umask 002 -> -rw-rw-r--
$ umask
0022

Chapter 5.6 is the mechanism. What Part II adds is the specific failure: an engineer with umask 077 runs a regression into a shared results area, and nobody else can read the results. Not the logs, not the waveforms, not the summary. The run succeeded; the output is unusable.

And the subtler form: directories created with umask 077 are mode 0700, so a colleague cannot even cd into them to find out what is wrong. The error is "Permission denied" on a directory whose contents would have been fine.

Two defences, and the second is the one for shared areas:

umask 022                      # at the top of a flow script: world-readable output
umask 002                      # for a shared results area: group-writable too

Setting it explicitly in the script is what makes the output independent of whoever ran it. Checking it is one word, and it belongs in §7's fingerprint for exactly that reason.

6. Stale Artifacts: Last Run Deciding This One

The last cause is the most expensive to find, because the evidence is not in the environment at all — it is on disk.

A flow that decides whether to rebuild by asking whether the output exists:

if [ -f build/top.vvp ]; then echo "compile: up to date"; exit 0; fi

Given a source file that has since been broken, measured:

$ ./compile.sh
compile: up to date
$ echo $?
0

The source does not compile and the flow reported success, because yesterday's output is still sitting there. This is Chapter 20.1's "exit zero wrongly" with the environment as the culprit, and it is why a colleague with a clean tree sees a failure you do not.

The fix is to compare timestamps rather than test existence:

if [ -f build/top.vvp ] && [ ! top.v -nt build/top.vvp ]; then
  echo "compile: up to date"; exit 0
fi
echo "compile: source is newer than the artifact -> must rebuild"; exit 3

$ ./compile2.sh
compile: source is newer than the artifact -> must rebuild
$ echo $?
3

-nt is "newer than", and the condition is written as a negated -nt rather than -ot deliberately: [ ! a -nt b ] is true when a is the same age or older, whereas [ a -ot b ] is false when the two timestamps are equal, which would rebuild on every run. Chapter 16.4 is the file-test vocabulary, including the comparison operators most people never reach for.

7. The Environment, and the One Diff That Finds It

When the five specific causes are ruled out, compare the whole environment. The technique is one line and it is the reason env exists:

$ diff <(env | sort) <(ssh other-host 'env | sort') | grep -E '^[<>]'
> FOO=bar
< LANG=C.UTF-8
< PATH=/opt/homebrew/bin:/opt/homebrew/sbin:/Users/engineer/.nvm/...
> PATH=/usr/bin:/bin

Four differing lines out of a full environment, each one a candidate. env | sort because env's order is not defined, and <( ) because process substitution lets a command's output appear where a file is expected — which is what makes this a one-liner rather than two temporary files. Chapter 8.6 §6 introduced the construct and Chapter 17.7 is where it earns its keep.

Two refinements that make it usable in practice:

# ignore the variables that always differ and never matter
env | sort | grep -vE '^(_|SHLVL|OLDPWD|PWD|TMPDIR|SSH_|TERM)' > /tmp/env.$$

and the PATH-specific version, because a single long PATH line diffs as one difference while containing twenty:

tr ':' '\n' <<< "$PATH" | nl        # one entry per line, numbered

Chapter 13.5 is tr; the nl is so that "entry 3 differs" is something you can say.

And the thing to do instead of all of this, which is the chapter's real recommendation: record the environment at the time of the run, so the comparison does not require both engineers to still have their shells open.

#!/usr/bin/env bash
# fingerprint — the facts that decided this run, for the record
set -o nounset
printf '%-16s %s\n' \
  "host"    "$(hostname)" \
  "user"    "$(id -un)" \
  "cwd"     "$(pwd -P)" \
  "shell"   "${BASH_VERSION:-unknown}" \
  "locale"  "LANG=${LANG:-unset} LC_ALL=${LC_ALL:-unset}" \
  "umask"   "$(umask)" \
  "tool"    "$(command -v iverilog || echo 'not found')" \
  "version" "$(iverilog -V 2>&1 | head -1)" \
  "git"     "$(git rev-parse --short HEAD 2>/dev/null || echo 'not a repo')"

$ ./fingerprint
host             compute-07
user             engineer
cwd              /scratch/engineer/run_4417
shell            3.2.57(1)-release
locale           LANG=C.UTF-8 LC_ALL=unset
umask            0022
tool             /opt/homebrew/bin/iverilog
version          Icarus Verilog version 13.0 (stable) (v13_0)
git              not a repo

Nine lines, written into every run directory, and the entire investigation becomes diff fingerprint.mine fingerprint.theirs. Four details are deliberate:

pwd -P, not $PWD — §4's reason, and the whole point is that this file will be read on another machine.

command -v and the version as separate lines, because the path can match while the versions differ (a symlink repointed at a new install) and the version can match while the paths differ.

|| echo 'not found' on both tool lines, so a missing tool is recorded as a fact rather than producing an empty line that reads like a bug in the script. Chapter 18.4 §5 is why a bare $( ) swallows the status.

And git rev-parse --short HEAD, because "which source" is as much a part of the answer as "which tool". A fingerprint without a revision is half a fingerprint.

8. The Ordered Procedure

Put together, the thing to actually do when two engineers disagree:

Seven of the eight take one command and no setup. The discipline is to do them in order rather than starting with step 8, because the environment diff always finds differences and most of them are irrelevant — which is how a one-hour problem becomes a one-day one.

Practice

1. Put a fake iverilog (or any tool you have) earlier on PATH and confirm -V reports the fake. Then find it with type -a and command -v.
2. Run type -a grep sed sort and read the first line of each. Is anything shadowed in your shell?
3. Run bash --noprofile --norc -c 'command -v grep' and compare. Explain what a scheduler-launched job would get.
4. Reproduce §3's sort experiment. Then sort a real test list under both locales and diff the two orders.
5. Run the case B in [a-z]) test under both locales. Then write a pattern that means the same thing in both.
6. Add export LC_ALL=C to a script that sorts, and explain to a colleague what it buys and what it costs.
7. Create a file under umask 077 in a directory a colleague can read, and have them try to read it. Then do the same for a directory, and note which error is more confusing.
8. Write §6's compile.sh and compile2.sh. Break the source, run both, and confirm you get 0 and 3.
9. Then make the artifact newer than the source with touch and confirm compile2.sh reports up to date. Explain why timestamps are not content.
10. Write fingerprint and run it in two different directories, one reached through a symlink. Diff the two outputs.
11. Add fingerprint to the start of a flow you have, writing into the run directory. Then deliberately change your locale and diff two runs.
12. Work through §8's eight steps on a real "works for me" report. Record which step found it.

Check Your Understanding

1. Why does a command line not determine which program runs?
2. Why should you read the first line of type -a?
3. What does command -v printing a bare name rather than a path tell you?
4. Why can a command work in your terminal and fail in a scheduled job?
5. Name two things the locale changes, and give the fix for a script whose output is compared.
6. Which of LANG and LC_ALL wins, and which command shows what is actually in effect?
7. What specific failure does umask 077 cause in a shared results area?
8. Why is [ ! a -nt b ] preferred over [ a -ot b ]?
9. Why should the fingerprint record both the tool's path and its version?
10. Why is the environment diff last rather than first?

9. Remember

• A command line does not determine what runs. PATH order, functions and aliases do.
• Measured: the same iverilog -V gave version 13.0 and version 99.0 on one machine, from PATH alone.
• Read the first line of type -a; the rest is what is being shadowed.
• command -v printing a bare name means a function or builtin — which a script will not have.
• bash --noprofile --norc -c '…' is how to see what a scheduled job will get.
• The locale changes sort order and the meaning of [a-z]. Set export LC_ALL=C in anything whose output is sorted, diffed or compared.
• LC_ALL beats LANG; locale shows the truth.
• pwd versus pwd -P finds a path through a symlink in one line.
• umask 077 makes a shared regression's output unreadable — files 0600, directories 0700. Set umask explicitly in flow scripts.
• Testing existence instead of timestamps reports success on a broken source. Measured: "compile: up to date", exit 0.
• [ ! src -nt out ], not [ src -ot out ], so equal timestamps do not force a rebuild.
• Timestamps are not content, and clock skew across a network filesystem produces the stale-artifact symptom for real.
• Record a fingerprint into every run directory — host, user, physical cwd, shell, locale, umask, tool path, tool version, source revision.
• Do the eight checks in order. The environment diff is last because it always finds something.

10. Module 22 in One Page

The module's whole argument, now that all six chapters are behind it:

• 22.1 — EDA tools assume a case-sensitive filesystem, POSIX processes, shared compute and network licences. The first assumption fails on macOS, and Top.v and top.v became one file.
• 22.2 — the shell assembles, sequences, decides and reports. A compile error exits 2; a failing test exits 0. That asymmetry is Part II's organising problem.
• 22.3 — workstation, compute servers, file server, one mount point. pwd and pwd -P disagreed in both the prefix and the last component, which is how a correct path points to the wrong place.
• 22.4 — home, project, scratch, archive. du reports blocks and ls reports bytes; -mtime +7 skips the seven-day-old file.
• 22.5 — divide the load by the cores, watch memory before CPU, nice the batch work, and leave nothing behind.
• 22.6 — six causes of divergence, checked in order, and a fingerprint written where the next engineer will find it.

Part II's remaining thirteen modules are each one of these pressures applied to a specific tool. Module 23 starts with the one you meet first: a design repository you did not create, and finding your way around it.