Environment Variables and Tool Versions

Part II · Module 24 · Chapter 24.2 · Environment Variables and Tool Versions

1. Three Ways to Read One Variable

import os

os.environ["UVM_HOME"] = "/tools/uvm-1.2"

print("os.environ[...]      :", repr(os.environ["UVM_HOME"]))
print("os.environ.get(...)  :", repr(os.environ.get("UVM_HOME")))
print("os.getenv(...)       :", repr(os.getenv("UVM_HOME")))
print()
print("the three agree when the variable is set:",
      os.environ["UVM_HOME"] == os.environ.get("UVM_HOME") == os.getenv("UVM_HOME"))

os.environ[...]      : '/tools/uvm-1.2'
os.environ.get(...)  : '/tools/uvm-1.2'
os.getenv(...)       : '/tools/uvm-1.2'

the three agree when the variable is set: True

They agree completely while the variable is set, which is why the difference between them is so easy to miss. It appears only when it is not:

import os

os.environ.pop("VCS_HOME", None)

try:
    print("subscript:", repr(os.environ["VCS_HOME"]))
except KeyError as exc:
    print("subscript:", type(exc).__name__, exc)

print("get      :", repr(os.environ.get("VCS_HOME")))
print("get+dflt :", repr(os.environ.get("VCS_HOME", "/tools/vcs")))
print("getenv   :", repr(os.getenv("VCS_HOME")))
print("getenv+d :", repr(os.getenv("VCS_HOME", "/tools/vcs")))

subscript: KeyError 'VCS_HOME'
get      : None
get+dflt : '/tools/vcs'
getenv   : None
getenv+d : '/tools/vcs'

os.getenv(name, default) and os.environ.get(name, default) are the same function in practice — os.getenv is a thin wrapper. Use either; be consistent.

The subscript form is the right one when the variable is mandatory: a KeyError naming the variable, at the moment you read it, is a better failure than a None that travels three functions before producing something incomprehensible. Chapter 24.5 turns that into a pre-flight check.

2. A Default Does Not Fire for an Empty Value

import os

os.environ["UVM_HOME"] = ""          # exported, but empty

print("is it present?        :", "UVM_HOME" in os.environ)
print("is it truthy?         :", bool(os.environ["UVM_HOME"]))
print()
print("get with a default    :", repr(os.environ.get("UVM_HOME", "/tools/uvm")))
print("getenv with a default :", repr(os.getenv("UVM_HOME", "/tools/uvm")))

is it present?        : True
is it truthy?         : False

get with a default    : ''
getenv with a default : ''

The default never fired. get(name, default) substitutes the default when the key is absent, and this key is present — it just happens to hold the empty string.

That is the sharpest edge in this chapter, because the state it describes is not hypothetical:

• a setup script that ran halfway and exported UVM_HOME= before failing;
• a scheduler that passes every variable through, converting unset to empty;
• a config template rendered with a missing field;
• a Makefile exporting UVM_HOME=$(UVM_ROOT) where UVM_ROOT was never set.

Every one of those produces a variable that in reports as present and that holds nothing. Your default is right there in the code, and it does not run.

The fix is one operator:

import os

os.environ["UVM_HOME"] = ""
os.environ.pop("VCS_HOME", None)
os.environ["XCELIUM_HOME"] = "/tools/xcelium-24.09"


def tool_home(name, default):
    """Treat 'absent' and 'present but empty' as the same thing."""
    return os.environ.get(name) or default


for name in ("UVM_HOME", "VCS_HOME", "XCELIUM_HOME"):
    naive = os.environ.get(name, f"<default for {name}>")
    fixed = tool_home(name, f"<default for {name}>")
    state = "empty" if name in os.environ and not os.environ[name] else (
        "set" if name in os.environ else "absent")
    print(f"{name:<14} {state:<7} naive={naive!r:<26} or-form={fixed!r}")

UVM_HOME       empty   naive=''                         or-form='<default for UVM_HOME>'
VCS_HOME       absent  naive='<default for VCS_HOME>'   or-form='<default for VCS_HOME>'
XCELIUM_HOME   set     naive='/tools/xcelium-24.09'     or-form='/tools/xcelium-24.09'

os.environ.get(name) or default falls back on anything falsy, which for a string means absent or empty. All three rows then behave the way a reader expects.

3. Setting One

import os
import subprocess
import sys

uvm_home = "/tools/uvm-1.2"

env = {**os.environ, "UVM_HOME": uvm_home}

CHECK = "import os; print('child UVM_HOME:', os.environ.get('UVM_HOME'))"
cp = subprocess.run([sys.executable, "-c", CHECK], env=env, capture_output=True, text=True)
print(cp.stdout.strip())
print("parent UVM_HOME:", os.environ.get("UVM_HOME"))

child UVM_HOME: /tools/uvm-1.2
parent UVM_HOME: None

That is the first half of the canonical annotation:

env = {**os.environ, "UVM_HOME": uvm_home}

Chapter 24.1 Section 4 argued for this form; here is the one thing it leaves open. Sometimes you do want the variable set for your own process — because a library you call reads it, or because every child from here on needs it:

import os
import subprocess
import sys

os.environ["UVM_HOME"] = "/tools/uvm-1.2"

CHECK = "import os; print(os.environ.get('UVM_HOME'))"
first = subprocess.run([sys.executable, "-c", CHECK], capture_output=True, text=True)
second = subprocess.run([sys.executable, "-c", CHECK], capture_output=True, text=True)

print("first child :", first.stdout.strip())
print("second child:", second.stdout.strip())
print("and ours    :", os.environ["UVM_HOME"])

first child : /tools/uvm-1.2
second child: /tools/uvm-1.2
and ours    : /tools/uvm-1.2

os.environ[name] = value changes this process, and therefore every child started afterwards. It is the os.chdir of environment handling: it works, it is global, and it is not undone if something raises.

The rule that follows is the same one Chapter 23.7 Section 5 reached about directories:

Per-run settings go in env=. Process-wide settings go in os.environ, early, once, and deliberately.

A loop giving each test a different UVM_HOME must use env=. A script that needs TMPDIR set for everything it does may set it in os.environ at startup — but near the top, where a reader finds it, not in the middle of a function.

4. Asking the Tool

import sys

print("TOOL:\txrun\t24.09-s001")
print("Copyright (c) 1996-2024 Cadence Design Systems, Inc.")

import subprocess
import sys
from pathlib import Path

tool = [sys.executable, str(Path("faketool.py").resolve())]

version = subprocess.run(tool + ["-version"], capture_output=True, text=True).stdout

print("raw:", repr(version))
print()
print("first line:", version.splitlines()[0])

raw: 'TOOL:\txrun\t24.09-s001\nCopyright (c) 1996-2024 Cadence Design Systems, Inc.\n'

first line: TOOL:	xrun	24.09-s001

That is the second half of the canonical annotation:

version = subprocess.run([tool, "-version"], capture_output=True, text=True).stdout

Simple, and correct for a tool that cooperates. The next three sections are the tools that do not.

The flag differs by vendor — -version, --version, -V, and Questa's vsim -version — so it belongs in Chapter 23.11's adapter alongside everything else vendor-specific.

5. Some Tools Answer on the Wrong Stream

import subprocess
import sys

OUT = "import sys; sys.stdout.write('ToolSim 1.2.3\\n')"
ERR = "import sys; sys.stderr.write('OtherSim 4.5.6\\n')"

for label, code in (("prints to stdout", OUT), ("prints to stderr", ERR)):
    cp = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True)
    print(f"{label}: stdout={cp.stdout.strip()!r:<16} stderr={cp.stderr.strip()!r}")

print()
print("merging the streams catches either:")
for label, code in (("stdout tool", OUT), ("stderr tool", ERR)):
    cp = subprocess.run([sys.executable, "-c", code], stdout=subprocess.PIPE,
                        stderr=subprocess.STDOUT, text=True)
    print(f"  {label}: {cp.stdout.strip()!r}")

prints to stdout: stdout='ToolSim 1.2.3'  stderr=''
prints to stderr: stdout=''               stderr='OtherSim 4.5.6'

merging the streams catches either:
  stdout tool: 'ToolSim 1.2.3'
  stderr tool: 'OtherSim 4.5.6'

The canonical annotation's .stdout returns the empty string for the second tool. Not an error, not an exception — an empty version string that flows straight into your run record and sits there looking like the tool has no version.

Chapter 23.5 Section 4 argued that stderr=subprocess.STDOUT is usually wrong for a run, because it interleaves a tool's diagnostics with its real output and you lose the distinction. For a version query the argument reverses: there is one line of output, you want it wherever it is, and the distinction between the streams carries nothing. This is the case the merge is for.

6. A Banner Is Evidence, Not a Format

Four tools, four shapes. These are written out by hand — no simulator is installed on this machine, per the convention from Chapter 23.1 Section 1 — and what is being demonstrated is the parser's behaviour over the variety, not any vendor's exact wording:

import re

BANNERS = {
    "xrun": "TOOL:\txrun\t24.09-s001\nCopyright (c) 1996-2024 Cadence Design Systems, Inc.\n",
    "vcs": "Synopsys VCS Version U-2023.12-SP1_Full64 -- Dec 15 2023\n",
    "vsim": "Questa Sim-64 vsim 2024.2 Simulator 2024.05 May 20 2024\n",
    "verilator": "Verilator 5.024 2024-04-05 rev v5.024\n",
}

NAIVE = re.compile(r"\b(\d+\.\d+[\w.\-]*)")

for tool, banner in BANNERS.items():
    first = banner.splitlines()[0]
    match = NAIVE.search(first)
    print(f"{tool:<10} -> {match.group(1) if match else None}")

xrun       -> 24.09-s001
vcs        -> 2023.12-SP1_Full64
vsim       -> 2024.2
verilator  -> 5.024

Every one of those is a number from the line, and two of them are arguably the wrong one.

vcs picked up -SP1_Full64, because the regex's [\w.\-]* happily runs through a build suffix. Trimming the regex to stop sooner would then truncate 24.09-s001, where the -s001 is part of the release.

vsim picked 2024.2 and the line also carries 2024.05. Which one identifies the simulator is a question about Questa, not about regular expressions.

So the honest design is not a better regex:

import re


def tool_version(banner):
    """Keep the evidence; parse only as a convenience."""
    lines = [line.strip() for line in banner.splitlines() if line.strip()]
    if not lines:
        return {"raw": "", "line": None, "parsed": None, "ok": False}
    first = lines[0]
    match = re.search(r"\b(\d+\.\d+[\w.\-]*)", first)
    return {"raw": banner, "line": first,
            "parsed": match.group(1) if match else None,
            "ok": True}


for banner in ("Synopsys VCS Version U-2023.12-SP1_Full64 -- Dec 15 2023\n",
               "a tool that says nothing numeric at all\n",
               "",):
    info = tool_version(banner)
    print(f"ok={info['ok']!s:<6} parsed={str(info['parsed']):<20} line={str(info['line'])[:44]!r}")

ok=True   parsed=2023.12-SP1_Full64   line='Synopsys VCS Version U-2023.12-SP1_Full64 --'
ok=True   parsed=None                 line='a tool that says nothing numeric at all'
ok=False  parsed=None                 line='None'

Three properties make that the right shape.

The raw banner is kept. It is the only field guaranteed to be correct, and six months from now it is what somebody will actually read. A parsed field that silently dropped -SP1 is worse than no parsed field.

parsed is allowed to be None, and the second row is a tool that simply does not put a number where the regex looks. Nothing broke; the record is still useful.

ok distinguishes "the tool said nothing" from "the tool said something unparseable." The first is a problem with the query — wrong flag, wrong stream, tool missing — and the second is not a problem at all.

7. Do Not Gate on the Return Code

import subprocess
import sys

CODE = "import sys; print('ToolSim 9.9.1'); sys.exit(1)"
cp = subprocess.run([sys.executable, "-c", CODE], capture_output=True, text=True)

print("returncode     :", cp.returncode)
print("stdout         :", repr(cp.stdout.strip()))
print("a rc-gated read :", repr(cp.stdout.strip() if cp.returncode == 0 else None))
print("a content read  :", repr(cp.stdout.strip() or None))

returncode     : 1
stdout         : 'ToolSim 9.9.1'
a rc-gated read : None
a content read  : 'ToolSim 9.9.1'

The tool answered correctly and exited 1. Several real tools do: -version is not a job, so some of them treat "I did not do any work" as a non-zero outcome, and some print the banner and then exit with whatever the argument parser felt about an unrecognised flag.

This is the one place in this curriculum where Chapter 23.4's rule — the return code is the tool's verdict and you must check it — does not apply, and it is worth being precise about why rather than treating it as an exception. Chapter 23.4's rule is about a tool that did a job. A version query is a question, the answer is the text, and the text is either there or it is not. Judge it by its output, not its status.

Record the status anyway. A version query that exits non-zero on a tool that normally exits zero is a real signal, even though it is not a reason to discard the answer.

8. Capture Once, Record With the Run

import sys

print("ToolSim 4.2.1 (build 20240915)")

import json
import shutil
import subprocess
import sys
from pathlib import Path

_CACHE = {}
CALLS = []


def probe_version(tool_cmd, key):
    """Ask a tool its version at most once per process."""
    if key in _CACHE:
        return _CACHE[key]
    CALLS.append(key)
    cp = subprocess.run(list(tool_cmd) + ["-version"], stdout=subprocess.PIPE,
                        stderr=subprocess.STDOUT, text=True)
    banner = cp.stdout
    first = next((l.strip() for l in banner.splitlines() if l.strip()), None)
    info = {"tool": key, "line": first, "rc": cp.returncode,
            "ok": bool(first), "resolved": shutil.which(key) is not None}
    _CACHE[key] = info
    return info


tool = [sys.executable, str(Path("faketool.py").resolve())]

for test in ("axi_smoke", "axi_reset", "axi_burst", "axi_soak"):
    info = probe_version(tool, "toolsim")

print("tests that needed the version:", 4)
print("times the tool was actually asked:", len(CALLS))
print()
record = {"test": "axi_smoke", "seed": 1839274, "status": "PASS",
          "tool_versions": {k: v["line"] for k, v in _CACHE.items()}}
print(json.dumps(record, indent=2))

tests that needed the version: 4
times the tool was actually asked: 1

{
  "test": "axi_smoke",
  "seed": 1839274,
  "status": "PASS",
  "tool_versions": {
    "toolsim": "ToolSim 4.2.1 (build 20240915)"
  }
}

Four tests, one version query. Asking per test would launch four hundred extra processes in a regression for an answer that cannot change mid-sweep — and on a licence-checked tool, a -version query can itself be slow or can touch the licence server.

The cache is a plain dict keyed by tool name, which is the whole mechanism. Two details make it honest:

CALLS exists so the saving is measured rather than assumed. A cache nobody checked is a cache that might be keyed wrongly and silently doing nothing.

The version goes into the run record, not just a log line. Chapter 21.1's record is what a program reads next month, and tool_versions is the field that answers Chapter 24.1's opening question — which binary was this? — without anybody needing to reconstruct a PATH.

9. Predict, Run, Observe

import os
import subprocess
import sys

os.environ["EMPTY"] = ""
os.environ.pop("ABSENT", None)
os.environ["SET"] = "value"

# A
print("A:", "EMPTY" in os.environ, bool(os.environ["EMPTY"]))

# B
print("B:", repr(os.environ.get("EMPTY", "dflt")))

# C
print("C:", repr(os.environ.get("ABSENT", "dflt")))

# D
print("D:", repr(os.environ.get("EMPTY") or "dflt"))

# E
try:
    os.environ["ABSENT"]
except KeyError as exc:
    print("E:", type(exc).__name__, exc)

# F
cp = subprocess.run([sys.executable, "-c", "import sys; sys.stderr.write('v1\\n')"],
                    capture_output=True, text=True)
print("F:", repr(cp.stdout), repr(cp.stderr.strip()))

# G
cp = subprocess.run([sys.executable, "-c", "import sys; sys.stderr.write('v1\\n')"],
                    stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
print("G:", repr(cp.stdout.strip()))

# H
cp = subprocess.run([sys.executable, "-c", "import sys; print('v2'); sys.exit(1)"],
                    capture_output=True, text=True)
print("H:", cp.returncode, repr(cp.stdout.strip()))

The verified output:

A: True False
B: ''
C: 'dflt'
D: 'dflt'
E: KeyError 'ABSENT'
F: '' 'v1'
G: 'v1'
H: 1 'v2'

A — present and falsy at the same time. That combination is the whole of Section 2.

B — the default did not fire, because the key is there.

C — it fires for a genuinely absent key.

D — or treats empty and absent alike.

E — the subscript raises, naming the variable, which is what you want for a mandatory one.

F — capture_output put the version in stderr and left .stdout empty.

G — merging the streams finds it either way.

H — a correct answer with a non-zero status. Judge the output, not the code.

10. Exercises

Exercise 1 — three readers. Read one set variable three ways and show they agree.

Exercise 2 — and when unset. Repeat for an unset variable and record all four behaviours.

Exercise 3 — mandatory. Say when the subscript form is the right choice and what its failure gives you that None does not.

Exercise 4 — the empty trap. Export a variable as the empty string and show the default not firing.

Exercise 5 — four ways to get there. Name four realistic ways a variable ends up present-but-empty in a flow you have seen.

Exercise 6 — the fix. Rewrite with or and show all three of absent, empty and set behaving sensibly.

Exercise 7 — presence or content. State in one sentence the difference between what get(k, d) asks and what or asks.

Exercise 8 — the exception. Describe a variable whose empty value is meaningful, and say how you would write and document that read.

Exercise 9 — per run or per process. Classify five settings in your flow as env= or os.environ, and justify the two hardest.

Exercise 10 — ask a tool. Capture a real tool's version banner and print it raw with repr.

Exercise 11 — which stream? Determine, for each tool in your flow, whether the version lands on stdout or stderr.

Exercise 12 — merge. Rewrite the query to catch either, and say why Chapter 23.5's argument against merging does not apply here.

Exercise 13 — four shapes. Collect banners from four tools and run one regex over all of them. Report what it got wrong.

Exercise 14 — keep the raw. Explain in two sentences why the raw line must be in the record even when parsing succeeded.

Exercise 15 — parsed may be None. Say what a tool with no number in its banner should produce, and what must not happen.

Exercise 16 — ok versus parsed. Distinguish the two failure modes those fields separate.

Exercise 17 — non-zero. Find or simulate a tool that exits non-zero for -version. Say why Chapter 23.4's rule does not apply.

Exercise 18 — measure the cache. Add a call counter and prove your cache is actually working.

Exercise 19 — not across runs. Argue in three sentences against persisting a version cache to disk.

11. Summary

• os.environ[name], .get(name) and os.getenv(name) agree while the variable is set and differ when it is not: KeyError, None, None.
• Use the subscript for a mandatory variable — a KeyError naming it beats a None that travels.
• A default does not fire for an empty value. get(name, default) asks about presence, and an exported-but-empty variable is present.
• That state is produced by ordinary things: a half-run setup script, a scheduler normalising unset to empty, a template with a missing field, a Makefile exporting an unset variable.
• os.environ.get(name) or default treats absent and empty alike, which is almost always what was meant.
• The same shape appears wherever a default meets a mapping — Chapter 21.7's JSON config key present with null.
• When an empty value is meaningful, test in explicitly and comment it, or every reader will read it as this bug.
• Per-run settings go in env=; process-wide settings go in os.environ early, once, and visibly.
• os.environ[name] = value is the os.chdir of environments: global, and not undone when something raises.
• A version query is a question, not a job. Judge it by its output.
• Some tools answer on stderr, so capture_output(...).stdout silently returns an empty version string — merge with stderr=subprocess.STDOUT.
• This is the one place Chapter 23.5's argument against merging reverses, because there is one line and the stream distinction carries nothing.
• Some tools exit non-zero and still answer correctly, so do not gate the read on the return code — but record the code anyway.
• A banner is evidence, not a format. One regex over four shapes over-captured a build suffix on one and faced two version numbers on another.
• Keep the raw line, let parsed be None, and use ok to separate "said nothing" from "said something unparseable."
• Query the version once per process and cache it — and count the calls, so the cache is measured rather than assumed.
• Do not persist the cache across runs; a reinstall makes a stored version confidently wrong.
• Put tool_versions in the run record, which is what answers Chapter 24.1's question without reconstructing anybody's PATH.

The next chapter gives each run the directories it needs: one it owns for its results, and a separate one for the large temporary data that should never be archived with them.