Python Part II · Module 24 · EDA Environment Engineering
Diagnosing a Broken Environment
Everything this module has described can go wrong, and when it does the failure arrives at the worst possible moment: four hundred tests into a nightly, as a wall of identical errors from a tool complaining about something three layers below anything you wrote. All of it is knowable beforehand. Whether a tool is on PATH, whether a variable is set, whether a directory can be written, whether the licence setting is even the right shape — these are questions a script can ask in under a second, before it launches anything, and answer in a message that names the fix.
Advanced19 min readPythonPart IIAutomationEDA ToolsEnvironment
Part II · Module 24 · Chapter 24.5 · Diagnosing a Broken Environment
1. The Failure This Replaces
TIMELINE = [
("22:00", "nightly starts", ""),
("22:00", "test 1 launches", "tool exits 1, log says 'Error-[LIC] cannot checkout'"),
("22:01", "test 2 launches", "same"),
("04:30", "test 400 launches", "same"),
("08:15", "somebody reads the summary", "400 FAIL"),
("08:40", "somebody opens a log", "the licence variable was never set"),
]
for when, what, detail in TIMELINE:
print(f"{when} {what:<28} {detail}")
print()
print("tests that produced information :", 0)
print("compute hours spent :", "all of them")
print("time to the actual diagnosis :", "10 hours 40 minutes after it was knowable")22:00 nightly starts
22:00 test 1 launches tool exits 1, log says 'Error-[LIC] cannot checkout'
22:01 test 2 launches same
04:30 test 400 launches same
08:15 somebody reads the summary 400 FAIL
08:40 somebody opens a log the licence variable was never set
tests that produced information : 0
compute hours spent : all of them
time to the actual diagnosis : 10 hours 40 minutes after it was knowableChapter 23.9 Section 7 made the classification point — those four hundred results are ERROR, not FAIL, and must not count in any denominator. This chapter makes the earlier point: the whole night was avoidable, because the question "is the licence variable set?" could have been asked at 21:59.
A pre-flight check is a cheap thing that converts a ten-hour failure into a one-line one.
2. Collect Every Problem
for tool in ("vcs", "verdi"):
if shutil.which(tool) is None:
problems.append(f"{tool} not on PATH")That is the canonical annotation, and the important word in it is append.
CHECKS = [("vcs on PATH", False), ("verdi on PATH", False),
("UVM_HOME set", False), ("run directory writable", True)]
def first_failure(checks):
for name, ok in checks:
if not ok:
raise RuntimeError(name)
def collect_all(checks):
return [name for name, ok in checks if not ok]
try:
first_failure(CHECKS)
except RuntimeError as exc:
print("stop-at-first reports:", exc)
problems = collect_all(CHECKS)
print("collect-all reports :", problems)
print()
print("fixes needed :", len(problems))
print("round trips, first-failure style:", len(problems))
print("round trips, collect-all style :", 1)stop-at-first reports: vcs on PATH
collect-all reports : ['vcs on PATH', 'verdi on PATH', 'UVM_HOME set']
fixes needed : 3
round trips, first-failure style: 3
round trips, collect-all style : 1A broken environment is almost never broken in one way. One tool module that failed to load takes out the tools, the variables and the licence setting together — so the three problems above are one cause, and a checker that reports them one at a time makes somebody fix, re-run, fix, re-run, fix.
This is the opposite of Chapter 23.4's rule about a failing run, and the difference is worth being precise about. A tool that fails should stop, because everything after it is built on what it did not produce. A check produces nothing that later checks depend on, so there is no reason to stop and every reason to keep going.
3. The Shape of the Check
4. Tools
import shutil
REQUIRED_TOOLS = ("vcs", "verdi", "python3")
def check_tools(tools):
problems = []
for tool in tools:
if shutil.which(tool) is None:
problems.append(f"{tool} not on PATH — is the tool module loaded?")
return problems
for problem in check_tools(REQUIRED_TOOLS):
print(problem)
print()
print("found :", [t for t in REQUIRED_TOOLS if shutil.which(t)])
print("missing :", [t for t in REQUIRED_TOOLS if not shutil.which(t)])vcs not on PATH — is the tool module loaded?
verdi not on PATH — is the tool module loaded?
found : ['python3']
missing : ['vcs', 'verdi']Chapter 23.9 Section 4 established that shutil.which checks the execute bit as well as existence, which is why it is the right question here rather than os.path.exists.
The message carries a hypothesis, not just an observation. "vcs not on PATH" is true and leaves somebody to work out what to do; "is the tool module loaded?" is the first thing they would have checked anyway, and naming it converts a report into a suggestion. When two or three tools are all missing at once, that hypothesis is almost certainly the right one.
5. Variables
import os
REQUIRED_VARS = ("UVM_HOME", "VCS_HOME")
os.environ["UVM_HOME"] = "" # the half-run setup script
os.environ.pop("VCS_HOME", None) # never set at all
os.environ["XCELIUM_HOME"] = "/tools/xcelium" # fine
def check_vars(names):
problems = []
for name in names:
if name not in os.environ:
problems.append(f"{name} is not set")
elif not os.environ[name]:
problems.append(f"{name} is set but empty — a setup script probably failed part-way")
return problems
for problem in check_vars(REQUIRED_VARS + ("XCELIUM_HOME",)):
print(problem)
print()
print("a single truthiness test would have reported:",
[n for n in REQUIRED_VARS if not os.environ.get(n)])
print("…which cannot tell the two causes apart")UVM_HOME is set but empty — a setup script probably failed part-way
VCS_HOME is not set
a single truthiness test would have reported: ['UVM_HOME', 'VCS_HOME']
…which cannot tell the two causes apartTwo states, two messages. Chapter 24.2 Section 2 argued that or should treat absent and empty alike when reading a value — and here, when diagnosing, the distinction is the most useful thing on the line.
"Not set" means nobody ran the setup. "Set but empty" means somebody ran it and it did not finish, which is a different person and a different fix. Collapsing them into "missing" throws away the only clue that separates them.
6. Directories
import os
import shutil
from pathlib import Path
for name in ("good_dir", "ro_dir"):
Path(name).mkdir(exist_ok=True)
os.chmod("ro_dir", 0o555)
Path("a_file").write_text("not a directory", encoding="utf-8")
def writable(path):
"""Ask the question we actually have, by doing the thing."""
probe = Path(path) / ".preflight_probe"
try:
probe.write_text("x", encoding="utf-8")
probe.unlink()
return True
except OSError:
return False
def check_dir(path, label):
path = Path(path)
if not path.exists():
return [f"{label}: {path} does not exist — create it before running"]
if not path.is_dir():
return [f"{label}: {path} exists but is not a directory"]
if not writable(path):
return [f"{label}: {path} is not writable — check its mode and ownership"]
return []
for path, label in (("good_dir", "run root"), ("ro_dir", "scratch"),
("a_file", "run root"), ("no_such_dir", "scratch")):
for problem in check_dir(path, label):
print(problem)
print()
print("os.access agrees here:", os.access("ro_dir", os.W_OK) == writable("ro_dir"))
os.chmod("ro_dir", 0o755)scratch: ro_dir is not writable — check its mode and ownership
run root: a_file exists but is not a directory
scratch: no_such_dir does not exist — create it before running
os.access agrees here: TrueThree distinct states, three distinct messages, and the fixes are genuinely different: create it, point somewhere else, change its mode.
On the choice of writability test: os.access(path, os.W_OK) and the write probe agreed in this example, and the last line measures that rather than asserting it. They are not the same question, though. os.access consults the permission bits; the probe performs the operation. The documented cases where the bits and the outcome diverge — a privileged user for whom the bits do not apply, a network filesystem whose server decides independently — are exactly the environments a compute farm runs on, which is why the probe is the one to ship.
The probe cleans up after itself, and the unlink is inside the try: a directory where the write succeeds and the delete fails is still effectively broken, and the except OSError catches both.
7. The Licence Setting, Without a Network Call
def licence_shape(value):
"""Classify a FlexLM-style setting. Never contacts anything."""
if not value:
return {"kind": "unset", "servers": 0, "ok": False}
parts = [p for p in value.split(":") if p]
servers = [p for p in parts if "@" in p]
if servers:
return {"kind": "server", "servers": len(servers), "ok": True}
return {"kind": "file", "servers": 0, "ok": True}
SETTINGS = ["27000@licsrv-a.example.com", "1717@lic1:1717@lic2",
"/tools/licenses/synopsys.dat", "", "licsrv-a"]
for value in SETTINGS:
info = licence_shape(value)
shown = "<unset>" if not value else f"{len(value)} chars"
print(f"{shown:<10} kind={info['kind']:<7} servers={info['servers']} ok={info['ok']}")26 chars kind=server servers=1 ok=True
19 chars kind=server servers=2 ok=True
28 chars kind=file servers=0 ok=True
<unset> kind=unset servers=0 ok=False
8 chars kind=file servers=0 ok=TrueThe setting's value is never printed — only its length and its shape, per Chapter 24.1 Section 8. A length plus a classification is enough to debug with: "unset", "one server", "two servers", "a file path".
Note the last row. licsrv-a has no @, so it is classified as a file path — and it is almost certainly a hostname somebody wrote without the port. The shape check cannot catch that, and pretending otherwise would be the worst kind of false confidence.
That is the honest boundary of an offline check:
Shape is checkable without a network. Reachability is not.
A pre-flight check that tried to contact the licence server would be slower, would fail for reasons unrelated to your flow, and would turn a one-second check into something with a timeout of its own. The right division is to verify offline what can be verified offline, and let the first real tool launch discover the rest — which Chapter 23.9's ToolError already classifies as ERROR rather than FAIL.
8. Headroom
import shutil
usage = shutil.disk_usage(".")
print("fields :", usage._fields)
print("total is positive:", usage.total > 0)
print("free <= total :", usage.free <= usage.total)
print("used + free <= total:", usage.used + usage.free <= usage.total)
NEEDED_GIB = 2.0
free_gib = usage.free / 2**30
print()
print(f"need {NEEDED_GIB} GiB, have enough:", free_gib >= NEEDED_GIB)fields : ('total', 'used', 'free')
total is positive: True
free <= total : True
used + free <= total: True
need 2.0 GiB, have enough: Trueshutil.disk_usage returns a named tuple of bytes. The actual numbers are a property of this machine, so what is published is the relationships — the same discipline Chapter 24.4 Section 5 applied to time.
The check that matters is a comparison against what a run needs, and that figure comes from measurement rather than guesswork: Chapter 24.4's dry run reported the byte total of what a sweep would free, and the same accounting over a completed run directory gives the per-run footprint to multiply by.
Headroom is the one check in this chapter that is usually a warning rather than a blocker, which is the next section.
9. Blockers and Warnings
def report(problems):
"""Split by severity. Only a blocker stops the run."""
blockers = [p for sev, p in problems if sev == "blocker"]
warnings = [p for sev, p in problems if sev == "warning"]
for p in blockers:
print("BLOCKER:", p)
for p in warnings:
print("warning:", p)
print()
print(f"{len(blockers)} blocker(s), {len(warnings)} warning(s)")
return 1 if blockers else 0
FOUND = [
("blocker", "vcs not on PATH — is the tool module loaded?"),
("blocker", "UVM_HOME is set but empty — a setup script probably failed part-way"),
("warning", "free space is below the 2.0 GiB a full run needs"),
("warning", "DISPLAY is not set — a GUI tool would fail, batch runs will not"),
]
code = report(FOUND)
print("exit code :", code)
print("would run tests:", code == 0)BLOCKER: vcs not on PATH — is the tool module loaded?
BLOCKER: UVM_HOME is set but empty — a setup script probably failed part-way
warning: free space is below the 2.0 GiB a full run needs
warning: DISPLAY is not set — a GUI tool would fail, batch runs will not
2 blocker(s), 2 warning(s)
exit code : 1
would run tests: FalseThe split exists because a checker with only one severity gets turned off. Every real environment has something slightly wrong with it, and if each of those stops the flow, somebody adds --skip-preflight and nobody ever runs it again.
A blocker is something that guarantees every test will fail. No vcs, no UVM_HOME, an unwritable run root.
A warning is something that might matter. Low disk, no DISPLAY, a tool version nobody has used before. Printed, counted, and not fatal.
Two mechanical points. The exit code is 1 when there is a blocker and 0 otherwise, which is Chapter 22.4's contract and is what lets a shell wrapper gate on it. And in a real script these lines go to stderr, per Chapter 22.5 — a diagnostic is not the program's output, and a wrapper piping stdout somewhere should still see them.
10. The Checker Must Be Able to Fail
import os
import shutil
from pathlib import Path
def writable(path):
probe = Path(path) / ".preflight_probe"
try:
probe.write_text("x", encoding="utf-8")
probe.unlink()
return True
except OSError:
return False
def preflight(tools, variables, directories):
problems = []
for tool in tools:
if shutil.which(tool) is None:
problems.append(("blocker", f"{tool} not on PATH"))
for name in variables:
if name not in os.environ:
problems.append(("blocker", f"{name} is not set"))
elif not os.environ[name]:
problems.append(("blocker", f"{name} is set but empty"))
for path, label in directories:
p = Path(path)
if not p.exists():
problems.append(("blocker", f"{label}: {p} does not exist"))
elif not p.is_dir():
problems.append(("blocker", f"{label}: {p} is not a directory"))
elif not writable(p):
problems.append(("blocker", f"{label}: {p} is not writable"))
return problems
Path("ok_dir").mkdir(exist_ok=True)
Path("bad_dir").mkdir(exist_ok=True)
os.chmod("bad_dir", 0o555)
os.environ["GOOD_VAR"] = "value"
os.environ["EMPTY_VAR"] = ""
os.environ.pop("ABSENT_VAR", None)
healthy = preflight(("python3",), ("GOOD_VAR",), (("ok_dir", "run root"),))
print("healthy environment ->", len(healthy), "problems")
for label, args in (
("missing tool ", (("definitely_not_a_tool_xyz",), (), ())),
("absent var ", ((), ("ABSENT_VAR",), ())),
("empty var ", ((), ("EMPTY_VAR",), ())),
("missing dir ", ((), (), (("no_such", "scratch"),))),
("not a dir ", ((), (), (("ok_dir/../a_file", "run root"),))),
("unwritable dir", ((), (), (("bad_dir", "scratch"),))),
):
Path("a_file").write_text("x", encoding="utf-8")
found = preflight(*args)
print(f"{label} -> {len(found)} problem(s): {found[0][1] if found else 'NONE — CHECK IS DEAD'}")
os.chmod("bad_dir", 0o755)healthy environment -> 0 problems
missing tool -> 1 problem(s): definitely_not_a_tool_xyz not on PATH
absent var -> 1 problem(s): ABSENT_VAR is not set
empty var -> 1 problem(s): EMPTY_VAR is set but empty
missing dir -> 1 problem(s): scratch: no_such does not exist
not a dir -> 1 problem(s): run root: ok_dir/../a_file is not a directory
unwritable dir -> 1 problem(s): scratch: bad_dir is not writableSix checks, six demonstrated failures, and a healthy environment reporting zero. Both halves are necessary.
A checker that has never been observed to fail is not known to work — it is known to be silent, which is also what a broken checker looks like. This is the same instinct as mutation testing: break the thing deliberately and confirm the detector notices. A pre-flight check is a safety device, and an untested safety device is decoration.
The NONE — CHECK IS DEAD branch is the part that makes this a test rather than a demonstration. If a check stops working — a refactor drops a condition, a pattern stops matching — that line prints, loudly, instead of a quiet zero that reads like success.
The healthy row matters just as much. A checker that reports problems in a correct environment is worse than none, because it is the fastest possible route to somebody disabling it.
11. Predict, Run, Observe
import os
import shutil
from pathlib import Path
Path("pro_ok").mkdir(exist_ok=True)
Path("pro_ro").mkdir(exist_ok=True)
os.chmod("pro_ro", 0o555)
Path("pro_file").write_text("x", encoding="utf-8")
os.environ["PRO_EMPTY"] = ""
os.environ.pop("PRO_ABSENT", None)
def writable(p):
probe = Path(p) / ".probe"
try:
probe.write_text("x", encoding="utf-8")
probe.unlink()
return True
except OSError:
return False
# A
print("A:", shutil.which("definitely_not_a_tool_xyz"))
# B
print("B:", "PRO_EMPTY" in os.environ, bool(os.environ["PRO_EMPTY"]))
# C
print("C:", "PRO_ABSENT" in os.environ)
# D
print("D:", Path("pro_file").exists(), Path("pro_file").is_dir())
# E
print("E:", writable("pro_ok"), writable("pro_ro"))
# F
print("F:", shutil.disk_usage(".").free <= shutil.disk_usage(".").total)
# G
problems = [("blocker", "x"), ("warning", "y"), ("warning", "z")]
print("G:", sum(1 for s, _ in problems if s == "blocker"),
1 if any(s == "blocker" for s, _ in problems) else 0)
os.chmod("pro_ro", 0o755)The verified output:
A: None
B: True False
C: False
D: True False
E: True False
F: True
G: 1 1A — a missing tool resolves to None rather than raising, which is what makes it usable in a loop that keeps going.
B — present and empty: the state that needs its own message.
C — absent: a different state, a different message.
D — a path can exist and not be a directory. Three checks, not one.
E — the write probe distinguishes the two directories.
F — only the relationship is published; the numbers are this machine's.
G — one blocker, exit code 1. Warnings do not change it.
12. Exercises
Exercise 1 — the timeline. Write out the timeline of the last environment failure your team had, and mark when it first became knowable.
Exercise 2 — ERROR not FAIL. Say what those runs should be classified as, citing Chapter 23.9.
Exercise 3 — append, not raise. Implement both styles over four failing checks and count the round trips each produces.
Exercise 4 — one cause, three symptoms. Give a single root cause in your environment that produces at least three separate check failures.
Exercise 5 — why checks differ from runs. Explain in two sentences why Chapter 23.4's stop-at-the-first-failure rule does not apply to a checker.
Exercise 6 — the hypothesis. Rewrite three of your error messages to carry a likely cause rather than only an observation.
Exercise 7 — two states. Produce both an absent and an empty variable and give each its own message. Say which person fixes each.
Exercise 8 — reading versus diagnosing. Reconcile this with Chapter 24.2's advice to use or.
Exercise 9 — three directory states. Produce a missing path, a non-directory and an unwritable directory. Give each its own fix.
Exercise 10 — probe or access. Implement both writability tests and compare them. Say which you would ship and why, without claiming a disagreement you did not observe.
Exercise 11 — clean up the probe. Say why the unlink belongs inside the try.
Exercise 12 — licence shape. Classify four licence settings offline. Print no values.
Exercise 13 — what shape cannot catch. Give a setting your shape check accepts that is wrong, and say what would catch it.
Exercise 14 — the boundary. Argue in three sentences against contacting the licence server from a pre-flight check.
Exercise 15 — headroom. Measure one run's footprint from a real run directory and turn it into a threshold.
Exercise 16 — severity. Classify ten checks in your flow as blocker or warning. Defend the two closest calls.
Exercise 17 — why the split exists. Describe what happens to a checker that makes everything fatal.
Exercise 18 — exit code and stream. Make your checker exit 1 only on a blocker and print to stderr. Say what each buys a wrapper.
Exercise 19 — warnings must be rare. Count how many warnings your checker emits on a healthy machine. If it is more than zero, justify each.
Exercise 20 — break every check. Exercise each of your checks against a deliberately broken environment and show it firing.
Exercise 21 — the dead check. Add the explicit NONE — CHECK IS DEAD branch and say what it protects against.
Exercise 22 — and the healthy case. Show your checker reporting zero problems on a good environment, and say why that test matters as much as the others.
13. Summary
- An environment failure is knowable before anything launches, and discovering it at 08:40 instead of 21:59 costs a whole night of compute and produces no information.
- Those runs are
ERROR, notFAIL— Chapter 23.9 Section 7 — and a pre-flight check prevents them rather than classifying them. - The annotation's
appendis the important part. Collect every problem; do not stop at the first. - A broken environment is rarely broken in one way, so one cause produces several findings and reporting them singly turns one fix into several round trips.
- Chapter 23.4's stop-on-failure rule does not apply to checks, because no check depends on an earlier one's output.
- Put a hypothesis in the message — "is the tool module loaded?" is a suggestion where "not on PATH" is only an observation.
shutil.whichis the right tool test, because it checks the execute bit as well as existence.- Absent and empty are two states with two different fixes — nobody ran the setup, versus it ran and failed part-way.
- This is the mirror of Chapter 24.2's advice:
orcollapses the distinction when reading, and diagnosing is where it is most useful. - A directory has three failure states — missing, not a directory, not writable — and three different fixes.
- A write probe asks the question you actually have;
os.accessasks about permission bits, and the two diverge exactly on privileged users and network filesystems. - Put the probe's
unlinkinside thetry: a directory you can write but not clean is still broken. - A licence setting's shape is checkable offline; its reachability is not, and a shape check cannot catch a hostname missing its port.
- Print the length and the classification, never the value.
shutil.disk_usagegives bytes; publish the comparison, not the number, and get the threshold from a measured run footprint.- Split blockers from warnings, or somebody adds a flag to skip the whole thing.
- Exit 1 only on a blocker, and print to stderr — Chapters 22.4 and 22.5.
- Warnings must be rare to be read; one that fires on every healthy run should be promoted or deleted.
- Exercise every check against a deliberately broken environment. A checker never observed to fail is not known to work.
- Make a dead check loud, and test the healthy case too — a checker that complains about a good environment is the fastest route to being switched off.
The last chapter of this module writes the environment a run actually used into the run record, so "it worked last week" becomes a question answerable with a diff.
Where this fits
Part of the Python curriculum.