Making an Environment Reproducible

Part II · Module 24 · Chapter 24.6 · Making an Environment Reproducible

1. The Question a Record Has to Answer

CONVERSATION = [
    ("engineer", "axi_smoke has started failing."),
    ("you", "Did anything change?"),
    ("engineer", "No. It worked last week."),
    ("you", "Same RTL?"),
    ("engineer", "Same commit, yes. Same command — I checked cmd.sh."),
]

for who, line in CONVERSATION:
    print(f"{who:>9}: {line}")

print()
print("what cmd.sh proves  :", "the command was identical")
print("what it cannot prove:", "the command found the same binary")
print("the missing artefact:", "a record of the environment the run read")

 engineer: axi_smoke has started failing.
      you: Did anything change?
 engineer: No. It worked last week.
      you: Same RTL?
 engineer: Same commit, yes. Same command — I checked cmd.sh.

what cmd.sh proves  : the command was identical
what it cannot prove: the command found the same binary
the missing artefact: a record of the environment the run read

Chapter 23.10 Section 7 named this gap and Chapter 24.1 Section 1 demonstrated it — the same ["xrun"] running two different releases. The whole of Module 24 has been about closing it, and this is the artefact that does.

Note what the goal is not. Nothing here will let you resurrect last week's environment; the tools are installed where they are installed. The goal is narrower and far more useful: make the difference visible in seconds, so the conversation above ends at the second line.

2. The Annotation

(run_dir/"env.json").write_text(json.dumps({
    "tool_versions": versions, "path": os.environ["PATH"]}, indent=2))

Three things about that are right, and they are not trivial.

It is written into the run directory, next to sim.log, cmd.sh and record.json — so the evidence travels with the run, which is Chapter 23.7's whole argument about a run owning a directory.

It is JSON, so Chapter 21.4's point applies: a program can read four hundred of these and compare them, which a formatted text dump would not allow.

It records tool_versions, which is Chapter 24.2's contribution and the single field that answers Chapter 24.1's opening question.

Two things about it are wrong, and the second is the subject of this chapter.

3. The Subscript, and Then the Real Problem

import json
import os

os.environ.pop("PATH", None)

try:
    blob = json.dumps({"tool_versions": {}, "path": os.environ["PATH"]}, indent=2)
    print("wrote", len(blob), "bytes")
except KeyError as exc:
    print("KeyError:", exc, "— the record-writing line is what crashed")

print()
safe = json.dumps({"tool_versions": {}, "path": os.environ.get("PATH", "")}, indent=2)
print("with .get, the record is still written:", len(safe) > 0)
print("and it records the absence           :", json.loads(safe)["path"] == "")

KeyError: 'PATH' — the record-writing line is what crashed

with .get, the record is still written: True
and it records the absence           : True

Chapter 24.2 Section 1 established that the subscript raises. Here that has a particular sting: this is forensic code, and the one circumstance in which a missing PATH is most worth recording is the circumstance in which the subscript destroys the record instead of writing it.

A record-writing line should never be the thing that raises. "an empty PATH" is a finding; a KeyError in the reporting path is a mystery.

That is the easy fix. Here is the one that matters:

import os

last_week = "/tools/xcelium-24.09/bin:/usr/local/bin:/usr/bin:/bin"
today = "/tools/xcelium-23.12/bin:/usr/local/bin:/usr/bin:/bin"

print("as stored by the annotation — one string each:")
print("   equal:", last_week == today)
print("   that is the entire answer the record can give")
print()

before, after = last_week.split(os.pathsep), today.split(os.pathsep)
removed = [p for p in before if p not in after]
added = [p for p in after if p not in before]

print("stored as lists:")
print("   removed  :", removed)
print("   added    :", added)
print("   unchanged:", len([p for p in before if p in after]), "of", len(before))

as stored by the annotation — one string each:
   equal: False
   that is the entire answer the record can give

stored as lists:
   removed  : ['/tools/xcelium-24.09/bin']
   added    : ['/tools/xcelium-23.12/bin']
   unchanged: 3 of 4

One bit versus the answer.

"not equal" is true and tells nobody anything; a sixty-entry PATH that differs in one place compares exactly as unequal as one that was rewritten entirely. Split on os.pathsep and stored as a list, the same two records say the Xcelium 24.09 directory was replaced by 23.12, and nothing else moved — which is the diagnosis, delivered by the storage format rather than by anybody's cleverness.

A record is only worth what its diff is worth. Store the structure, not the serialisation.

This is the same decision Chapter 23.10 Section 6 made about cmd — store the list, derive the string — and for the same reason. A list round-trips through JSON exactly and can be compared element by element; a joined string can only be compared whole.

4. Diffing Two Records

def diff(before, after, prefix=""):
    """Every leaf field that differs, by dotted path."""
    out = []
    for key in sorted(set(before) | set(after)):
        a, b = before.get(key), after.get(key)
        if isinstance(a, dict) and isinstance(b, dict):
            out += diff(a, b, prefix + key + ".")
        elif a != b:
            out.append((prefix + key, a, b))
    return out


LAST_WEEK = {"tool_versions": {"xrun": "24.09-s001", "vcs": "U-2023.12"},
             "env": {"UVM_HOME": "/tools/uvm-1.2",
                     "XCELIUM_HOME": "/tools/xcelium-24.09"}}
TODAY = {"tool_versions": {"xrun": "23.12-s002", "vcs": "U-2023.12"},
         "env": {"UVM_HOME": "/tools/uvm-1.2",
                 "XCELIUM_HOME": "/tools/xcelium-23.12", "NEW_VAR": "x"}}

for field, before, after in diff(LAST_WEEK, TODAY):
    print(f"{field:<26} {str(before):<22} -> {after}")

print()
print("fields that differ:", len(diff(LAST_WEEK, TODAY)))
print("identical records :", diff(LAST_WEEK, LAST_WEEK))

env.NEW_VAR                None                   -> x
env.XCELIUM_HOME           /tools/xcelium-24.09   -> /tools/xcelium-23.12
tool_versions.xrun         24.09-s001             -> 23.12-s002

fields that differ: 3
identical records : []

Three fields, and the first line of the conversation in Section 1 is now answered. The simulator version moved from 24.09 to 23.12, its home directory moved to match, and a new variable appeared.

Three properties of that function are worth copying.

It recurses into nested dicts and reports a dotted path, so tool_versions.xrun says both what changed and where it lives. A flat comparison of the two top-level dicts would say "tool_versions differs" and leave the reader to find out how.

sorted(set(before) | set(after)) means a key present in only one record still appears — env.NEW_VAR went from None to x. A loop over before alone would silently miss every addition, which is the half of a diff people forget.

Identical records produce an empty list, not a message. That is what makes it usable in a condition: if diff(a, b): reads correctly, and a report of four hundred runs can list only the ones that moved.

5. The Allow-List, Again

import json

SENSITIVE_MARKERS = ("LICENSE", "LICENCE", "TOKEN", "SECRET", "KEY", "PASSWORD")
RECORDED_VARS = ("PATH", "LD_LIBRARY_PATH", "UVM_HOME", "XCELIUM_HOME",
                 "VCS_HOME", "SNPSLMD_LICENSE_FILE", "TMPDIR")

PRETEND = {"PATH": "/tools/xcelium-24.09/bin:/usr/bin",
           "UVM_HOME": "/tools/uvm-1.2",
           "SNPSLMD_LICENSE_FILE": "27000@licsrv-a.example.com",
           "ARTIFACTORY_TOKEN": "abcdef-not-real",
           "HOME": "/home/someone"}


def capture_env(source, names=RECORDED_VARS, pathsep=":"):
    out = {}
    for name in names:
        if name not in source:
            out[name] = None
            continue
        value = source[name]
        if any(marker in name.upper() for marker in SENSITIVE_MARKERS):
            out[name] = {"set": True, "length": len(value), "value": "<redacted>"}
        elif name.endswith("PATH") and pathsep in value:
            out[name] = value.split(pathsep)
        else:
            out[name] = value
    return out


captured = capture_env(PRETEND)
print(json.dumps(captured, indent=2, sort_keys=True))
print()
print("variables present but NOT recorded:",
      sorted(k for k in PRETEND if k not in RECORDED_VARS))

{
  "LD_LIBRARY_PATH": null,
  "PATH": [
    "/tools/xcelium-24.09/bin",
    "/usr/bin"
  ],
  "SNPSLMD_LICENSE_FILE": {
    "length": 26,
    "set": true,
    "value": "<redacted>"
  },
  "TMPDIR": null,
  "UVM_HOME": "/tools/uvm-1.2",
  "VCS_HOME": null,
  "XCELIUM_HOME": null
}

variables present but NOT recorded: ['ARTIFACTORY_TOKEN', 'HOME']

Four decisions, and the last line is the one that matters most.

ARTIFACTORY_TOKEN and HOME never reach the file. This record goes into a run directory that is shared, archived and attached to tickets — Chapter 24.1 Section 8's warning. An allow-list means a secret nobody anticipated is excluded by default rather than by having been thought about.

The licence variable is in the allow-list and still redacted. Being worth recording and being safe to print are different questions, and the set-plus-length form answers the first without conceding the second. The length distinguishes unset, empty and plausible, which covers every environment question you actually have.

A None means "we looked and it was not set", which is information. A key simply absent from the record would be ambiguous between not set and not checked, and the diff in Section 4 depends on the distinction.

Any variable whose name ends in PATH is split into a list, so Section 3's diff works for LD_LIBRARY_PATH too — a variable with exactly the same ordered-search semantics and exactly the same shadowing hazard.

6. A Fingerprint for the Fast Path

import hashlib
import json

A = {"tool_versions": {"xrun": "24.09", "vcs": "U-2023.12"}, "env": {"UVM_HOME": "/u"}}
B = {"env": {"UVM_HOME": "/u"}, "tool_versions": {"vcs": "U-2023.12", "xrun": "24.09"}}
C = {"tool_versions": {"xrun": "23.12", "vcs": "U-2023.12"}, "env": {"UVM_HOME": "/u"}}


def fingerprint(record):
    canonical = json.dumps(record, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()


print("A and B hold the same data in a different key order")
print("  fingerprint(A) == fingerprint(B):", fingerprint(A) == fingerprint(B))
print("  fingerprint(A) == fingerprint(C):", fingerprint(A) == fingerprint(C))
print("  A's first 12 hex digits         :", fingerprint(A)[:12])
print()
naive = lambda r: hashlib.sha256(json.dumps(r).encode("utf-8")).hexdigest()
print("without sort_keys, the same two records:")
print("  naive(A) == naive(B):", naive(A) == naive(B))

A and B hold the same data in a different key order
  fingerprint(A) == fingerprint(B): True
  fingerprint(A) == fingerprint(C): False
  A's first 12 hex digits         : cf902bfd4ce7

without sort_keys, the same two records:
  naive(A) == naive(B): False

sort_keys=True is what makes the fingerprint mean anything. Without it, two records holding identical data hash differently purely because of the order the keys were inserted — so every comparison reports a change and the fingerprint becomes noise.

separators=(",", ":") removes the whitespace that indent=2 adds, for the same reason: the digest must depend on the data, not on the formatting.

What the fingerprint buys is the fast path over a regression:

import hashlib
import json


def fingerprint(record):
    return hashlib.sha256(
        json.dumps(record, sort_keys=True, separators=(",", ":")).encode("utf-8")
    ).hexdigest()


BASE = {"tool_versions": {"xrun": "24.09"}, "env": {"UVM_HOME": "/u"}}
DRIFTED = {"tool_versions": {"xrun": "23.12"}, "env": {"UVM_HOME": "/u"}}

runs = [("axi_smoke", BASE), ("axi_reset", BASE), ("axi_burst", DRIFTED),
        ("axi_soak", BASE), ("axi_hang", DRIFTED)]

groups = {}
for test, record in runs:
    groups.setdefault(fingerprint(record)[:12], []).append(test)

print("distinct environments in this regression:", len(groups))
for digest, tests in sorted(groups.items(), key=lambda kv: -len(kv[1])):
    print(f"  {digest}  {len(tests)} run(s): {', '.join(tests)}")
print()
print("worth investigating:", [t for d, ts in groups.items() if len(ts) < 3 for t in ts])

distinct environments in this regression: 2
  195ab7e0c6ef  3 run(s): axi_smoke, axi_reset, axi_soak
  7eb5af02038e  2 run(s): axi_burst, axi_hang

worth investigating: ['axi_burst', 'axi_hang']

Two distinct environments in one regression — which should never happen, and which no amount of reading individual env.json files would have surfaced as quickly. The digest reduces "are these four hundred runs comparable?" to counting distinct values.

A regression split across two environments is not a design result; it is two half-regressions, and any comparison between the halves is meaningless. This is the cheapest possible detector for it.

7. The Record, Assembled

import hashlib
import json
import os
import shutil
from pathlib import Path

SENSITIVE_MARKERS = ("LICENSE", "LICENCE", "TOKEN", "SECRET", "KEY", "PASSWORD")
RECORDED_VARS = ("PATH", "LD_LIBRARY_PATH", "UVM_HOME", "XCELIUM_HOME", "TMPDIR")


def capture_env(source, names=RECORDED_VARS, pathsep=":"):
    out = {}
    for name in names:
        if name not in source:
            out[name] = None
        elif any(m in name.upper() for m in SENSITIVE_MARKERS):
            out[name] = {"set": True, "length": len(source[name]), "value": "<redacted>"}
        elif name.endswith("PATH") and pathsep in source[name]:
            out[name] = source[name].split(pathsep)
        else:
            out[name] = source[name]
    return out


def fingerprint(record):
    return hashlib.sha256(
        json.dumps(record, sort_keys=True, separators=(",", ":")).encode("utf-8")
    ).hexdigest()


def write_env_record(run_dir, tool_versions, resolved, source):
    """Everything about the environment this run read. Never raises on a missing key."""
    run_dir = Path(run_dir)
    run_dir.mkdir(parents=True, exist_ok=True)
    body = {"tool_versions": tool_versions,
            "resolved_tools": resolved,
            "env": capture_env(source)}
    record = {**body, "fingerprint": fingerprint(body)}
    (run_dir / "env.json").write_text(json.dumps(record, indent=2, sort_keys=True) + "\n",
                                      encoding="utf-8")
    return record


PRETEND_ENV = {"PATH": "/tools/xcelium-24.09/bin:/usr/bin",
               "UVM_HOME": "/tools/uvm-1.2",
               "ARTIFACTORY_TOKEN": "abcdef-not-real"}

record = write_env_record(
    "runs/axi_smoke_1839274",
    {"xrun": "TOOL: xrun 24.09-s001"},
    {"xrun": "/tools/xcelium-24.09/bin/xrun"},
    PRETEND_ENV)

print("keys written   :", sorted(record))
print("PATH is a list :", isinstance(record["env"]["PATH"], list),
      "of", len(record["env"]["PATH"]))
print("token excluded :", "ARTIFACTORY_TOKEN" not in record["env"])
print("fingerprint    :", record["fingerprint"][:12])
print()
reloaded = json.loads((Path("runs/axi_smoke_1839274") / "env.json").read_text(encoding="utf-8"))
body = {k: v for k, v in reloaded.items() if k != "fingerprint"}
print("round-trips    :", body == {k: v for k, v in record.items() if k != "fingerprint"})
print("hash still matches after a file round trip:", fingerprint(body) == reloaded["fingerprint"])

keys written   : ['env', 'fingerprint', 'resolved_tools', 'tool_versions']
PATH is a list : True of 2
token excluded : True
fingerprint    : 89d217409e45

round-trips    : True
hash still matches after a file round trip: True

The whole module is in that function.

tool_versions is Chapter 24.2's banner, kept raw. resolved_tools is Chapter 23.9's shutil.which answer — which binary, not which name, and the field that makes Chapter 24.1 Section 1's two-installations problem visible. env is the allow-listed, redacted, list-split capture. fingerprint is computed over the body and excluded from its own input, which is why the round-trip check at the end works.

That last pair of lines is not decoration. A record that does not survive a JSON round trip is a record that cannot be compared tomorrow, and the hash recomputed from the reloaded body matching the stored one proves both the serialisation and the canonicalisation.

8. What This Still Does Not Pin

CAPTURED = [
    ("the command", "cmd.sh", "Chapter 23.10", True),
    ("which binary the name resolved to", "resolved_tools", "this chapter", True),
    ("what the tool said its version was", "tool_versions", "Chapter 24.2", True),
    ("the variables that were read", "env", "this chapter", True),
    ("the bytes of that binary", "—", "would need a checksum", False),
    ("the contents of the tool install", "—", "unbounded", False),
    ("the RTL revision", "—", "belongs in version control", False),
    ("the filelist's targets", "—", "paths may be refactored", False),
]

print(f"{'fact about the run':<36} {'field':<16} {'pinned'}")
for fact, field, where, pinned in CAPTURED:
    print(f"{fact:<36} {field:<16} {pinned}")

print()
print("pinned    :", sum(1 for *_, p in CAPTURED if p), "of", len(CAPTURED))
print("the record answers:", "what changed between two runs")
print("it does not answer:", "whether either run can be recreated today")

fact about the run                   field            pinned
the command                          cmd.sh           True
which binary the name resolved to    resolved_tools   True
what the tool said its version was   tool_versions    True
the variables that were read         env              True
the bytes of that binary             —                False
the contents of the tool install     —                False
the RTL revision                     —                False
the filelist's targets               —                False

pinned    : 4 of 8
the record answers: what changed between two runs
it does not answer: whether either run can be recreated today

Half, and it is worth being exact about which half.

An environment record pins what was read, not what was true. resolved_tools says the name resolved to a particular path; it does not say that path holds the same bytes today. A reinstall that keeps the directory name is invisible to every field here, and tool_versions catches it only if the version string changed.

Nothing here version-controls the design. The RTL revision belongs in git and the record should carry the commit — but that is a property of the inputs, not of the environment, and conflating the two produces a record that claims more than it knows.

So the honest claim is the narrow one, and it is the one Section 1 asked for:

This record cannot recreate last week. It can tell you, in three lines, what is different from last week — which is what the conversation actually needed.

9. Predict, Run, Observe

import hashlib
import json
import os

A = {"b": 2, "a": 1}
B = {"a": 1, "b": 2}
canon = lambda r: json.dumps(r, sort_keys=True, separators=(",", ":"))
fp = lambda r: hashlib.sha256(canon(r).encode()).hexdigest()

# A
os.environ.pop("NOPE_VAR", None)
try:
    os.environ["NOPE_VAR"]
except KeyError:
    print("A: KeyError")

# B
print("B:", repr(os.environ.get("NOPE_VAR", "")))

# C
print("C:", "a:b:c".split(":"))

# D
print("D:", "/x:/y" == "/x:/z", [p for p in "/x:/y".split(":") if p not in "/x:/z".split(":")])

# E
print("E:", fp(A) == fp(B))

# F
print("F:", hashlib.sha256(json.dumps(A).encode()).hexdigest()
      == hashlib.sha256(json.dumps(B).encode()).hexdigest())

# G
def diff(x, y, pre=""):
    out = []
    for k in sorted(set(x) | set(y)):
        if x.get(k) != y.get(k):
            out.append(pre + k)
    return out
print("G:", diff({"a": 1}, {"a": 1, "b": 2}))

# H
print("H:", diff(A, B))

The verified output:

A: KeyError
B: ''
C: ['a', 'b', 'c']
D: False ['/y']
E: True
F: False
G: ['b']
H: []

A — the subscript raises, which is wrong for a line whose job is to record a failure.

B — .get with a default records the absence instead.

C — splitting on the separator is what turns a path variable into something comparable.

D — as strings, "not equal"; as lists, this one entry.

E — a canonical dump makes the fingerprint order-independent.

F — without sort_keys, the same data hashes differently.

G — a key present in only one side still shows up, because the union is iterated.

H — identical data produces an empty diff, which is what makes if diff(...) read correctly.

10. Exercises

Exercise 1 — the conversation. Write down the last "it worked last week" you encountered and say which artefact would have ended it.

Exercise 2 — what cmd.sh cannot prove. State it in one sentence, citing Chapter 24.1.

Exercise 3 — the narrow goal. Explain why "recreate the old environment" is the wrong target and "see the difference" is the right one.

Exercise 4 — the subscript. Reproduce the KeyError from the annotation's form and say why it is especially bad in this particular line.

Exercise 5 — record the absence. Rewrite so an unset variable becomes a recorded fact.

Exercise 6 — one bit. Compare two PATH strings differing in one entry. Report what the comparison tells you.

Exercise 7 — the answer. Split both and report added, removed and unchanged. Say which output you would want at 9am.

Exercise 8 — the general rule. State the storage rule in one sentence and name the earlier chapter that made the same call about cmd.

Exercise 9 — dotted paths. Write a recursive diff that reports tool_versions.xrun rather than tool_versions.

Exercise 10 — additions. Show that iterating only the first record misses a newly added key.

Exercise 11 — empty means equal. Say why returning [] rather than a message matters for a four-hundred-run report.

Exercise 12 — allow-list. Build yours. List the variables in your environment that it deliberately excludes.

Exercise 13 — recorded and redacted. Explain why a variable can be worth recording and unsafe to print, and what the length buys.

Exercise 14 — null means checked. Say what a null records that an absent key would not.

Exercise 15 — every PATH. Apply the list-splitting rule to LD_LIBRARY_PATH and say why it needs the same treatment.

Exercise 16 — canonical. Hash two equal records built in different key orders, with and without sort_keys. Report both.

Exercise 17 — separators too. Say why indent=2 must not reach the hashed string.

Exercise 18 — count the environments. Group a regression's runs by fingerprint. Say what more than one distinct value means for the results.

Exercise 19 — assemble. Write write_env_record with versions, resolved paths, the allow-listed environment and a fingerprint over the body.

Exercise 20 — round trip. Reload the file and recompute the hash. Say what failing that check would mean.

Exercise 21 — read not true. Give a change to your environment that every field in your record would miss.

Exercise 22 — the inputs. Say why the RTL revision belongs in the record but not in the environment record.

Exercise 23 — the checksum trade. Price hashing the simulator binary on every run, and propose the caching rule you would use instead.

11. Summary

• cmd.sh proves the command was identical; it cannot prove the command found the same binary. Chapter 24.1 Section 1 is the demonstration.
• The goal is not to recreate last week — it is to make the difference visible in seconds.
• Write the record into the run directory, as JSON, including the tool versions. The annotation is right about all three.
• Never use the subscript in record-writing code. A missing PATH is a finding; a KeyError while recording it is a mystery.
• A record is only worth what its diff is worth. Two PATH strings differing in one entry compare as "not equal" — one bit.
• Split path-like variables on os.pathsep and store lists, and the same comparison names the entry that moved and the three that did not.
• Store the structure, not the serialisation — the same call Chapter 23.10 made about cmd.
• Diff recursively and report dotted paths, so the field and its location arrive together.
• Iterate the union of both key sets, or every addition is invisible.
• Return an empty list for identical records, so if diff(...) reads correctly over four hundred runs.
• Record an allow-list, so a secret nobody anticipated is excluded by default — Chapter 24.1 Section 8.
• A variable can be worth recording and unsafe to print; set-plus-length answers the first without conceding the second.
• null records "we looked and it was unset", which an absent key could not distinguish from "we never checked".
• A fingerprint needs sort_keys=True and compact separators, or identical data hashes differently and the digest is noise.
• Group a regression by fingerprint: more than one distinct value means two half-regressions, and no comparison across them is meaningful.
• Record resolved_tools as well as tool_versions — which binary, not which name.
• Round-trip the file and recompute the hash; a record that does not survive serialisation cannot be compared tomorrow.
• The record pins what was read, not what was true. A reinstall under the same directory name is invisible unless the version string changed.
• The RTL revision belongs in the result record, not the environment record, and conflating them claims more than the data knows.

That completes Module 24. An environment is no longer something a script inherits and hopes about: it is chosen deliberately, its tools are identified rather than named, its directories are separated by what deserves to survive, its failures are caught before anything launches, and what it was is written down in a form that answers the only question anybody asks of it. Module 25 takes the single reliable run this module has made possible and defines the contract a simulation runner has to meet.