Python Part II · Module 35 · Industry VLSI Automation Projects

An EDA Environment Diagnostic

This is the script that saves a support engineer a day. Half of all tool bugs are environment problems — a module not loaded, a variable pointing at last year's install, a licence server nobody can reach — and the person filing the bug cannot tell which half they are in. A diagnostic answers that, and the design problem it poses is one that every tool in this module has avoided so far: what to report about a check you were unable to perform.

Advanced25 min readPythonPart IIAutomationEnvironmentDiagnostics

Part II · Module 35 · Chapter 35.8 · An EDA Environment Diagnostic

1. What a Tool Bug Usually Is

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

CAUSES = [
    ("the tool is not on PATH", "environment", "10 seconds", "the engineer"),
    ("the wrong version is on PATH", "environment", "10 seconds",
     "the engineer"),
    ("a variable points at an old install", "environment", "10 seconds",
     "the engineer"),
    ("no licence available", "environment", "a minute", "IT"),
    ("the disk is full", "environment", "10 seconds", "the engineer"),
    ("the tool has a genuine defect", "the tool", "weeks", "the vendor"),
]

print(f"{'cause':<38} {'class':<13} {'to diagnose':<12} who fixes it")
for cause, cls, t, who in CAUSES:
    print(f"{cause:<38} {cls:<13} {t:<12} {who}")

env = sum(1 for _, c, _, _ in CAUSES if c == "environment")
print()
print("causes that are environment:", env, "of", len(CAUSES))
print("diagnosable in under a minute:", env)
print("what a bug report without a diagnostic costs:",
      "a vendor round-trip, days")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

cause                                  class         to diagnose  who fixes it
the tool is not on PATH                environment   10 seconds   the engineer
the wrong version is on PATH           environment   10 seconds   the engineer
a variable points at an old install    environment   10 seconds   the engineer
no licence available                   environment   a minute     IT
the disk is full                       environment   10 seconds   the engineer
the tool has a genuine defect          the tool      weeks        the vendor

causes that are environment: 5 of 6
diagnosable in under a minute: 5
what a bug report without a diagnostic costs: a vendor round-trip, days

Five of six causes are environment and each takes under a minute to check. The diagnostic's value is not that it finds hard problems — it is that it eliminates the easy ones before a bug report is written, so the one remaining possibility is the expensive one and everybody knows it.

2. The Annotation

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import shutil

PARTS = [
    ("--tools vcs,verdi", "the tools this flow invokes", "named, not guessed"),
    ("--require-licence", "treat a licence problem as an error", "a policy"),
]

print(f"{'part':<22} {'meaning':<34} note")
for part, meaning, note in PARTS:
    print(f"{part:<22} {meaning:<34} {note}")

print()
NAMED = ["vcs", "verdi"]
print("the tools the annotation names:", NAMED)
for t in NAMED:
    print(f"  {t:<8} on this machine: "
          f"{'found' if shutil.which(t) else 'NOT FOUND'}")

print()
print("so this chapter's diagnostic will report:", "two tools missing")
print("and that is                             :", "the honest answer here")
print("what it must NOT also report            :",
      "that their licences failed — §4")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

part                   meaning                            note
--tools vcs,verdi      the tools this flow invokes        named, not guessed
--require-licence      treat a licence problem as an error a policy

the tools the annotation names: ['vcs', 'verdi']
  vcs      on this machine: NOT FOUND
  verdi    on this machine: NOT FOUND

so this chapter's diagnostic will report: two tools missing
and that is                             : the honest answer here
what it must NOT also report            : that their licences failed — §4

Neither tool the annotation names is installed on this machine, and that is used rather than worked around. Chapter 23.1 Section 1's convention is that no commercial tool is invented here, so the diagnostic reports what is true — and the interesting question becomes what it should say about the licence for a tool it cannot find.

--tools takes a list because Chapter 33.12 Section 6 measured the alternative: a diagnostic with a built-in tool list reports a broken environment on a machine that uses a different simulator, and one that checks for python rather than python3 reports a broken environment on a working machine.

3. The First Version

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import shutil

TOOLS = ["iverilog", "vvp", "vcs", "verdi"]

print("v1 — for each tool, is it on PATH?")
missing = []
for tool in TOOLS:
    found = shutil.which(tool) is not None
    print(f"  {tool:<10} {'OK' if found else 'MISSING'}")
    if not found:
        missing.append(tool)

print()
print("  tools checked:", len(TOOLS))
print("  missing      :", missing)
print("  verdict      :", "FAIL" if missing else "PASS")
print()
print("what it got right:", "the facts about PATH")
print("what it cannot say:", ["which version", "whether a licence is available",
                              "whether a variable is stale",
                              "whether the disk is full"])
print("and the verdict FAIL means:", "something is wrong — but not what")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

v1 — for each tool, is it on PATH?
  iverilog   OK
  vvp        OK
  vcs        MISSING
  verdi      MISSING

  tools checked: 4
  missing      : ['vcs', 'verdi']
  verdict      : FAIL

what it got right: the facts about PATH
what it cannot say: ['which version', 'whether a licence is available', 'whether a variable is stale', 'whether the disk is full']
and the verdict FAIL means: something is wrong — but not what

The facts are right. The problem is the last line: a single FAIL for an environment with four independent properties tells the engineer to go looking, which is what they were already doing.

4. The Limitation: A Boolean Cannot Say "I Could Not Tell"

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import shutil
from enum import Enum


class State(Enum):
    PASS = "PASS"
    FAIL = "FAIL"
    UNKNOWN = "UNKNOWN"


SITUATIONS = [
    ("iverilog is on PATH", "ask PATH", State.PASS),
    ("vcs is not on PATH", "ask PATH", State.FAIL),
    ("vcs's licence", "needs vcs to ask", State.UNKNOWN),
    ("verdi's version", "needs verdi to ask", State.UNKNOWN),
    ("$VCS_HOME points at a real dir", "it is unset", State.UNKNOWN),
    ("free disk space", "ask the filesystem", State.PASS),
]

print(f"{'what is being checked':<34} {'how':<22} honest answer")
for what, how, state in SITUATIONS:
    print(f"{what:<34} {how:<22} {state.value}")

counts = {s: sum(1 for _, _, st in SITUATIONS if st is s) for s in State}
print()
for s in State:
    print(f"  {s.value:<9} {counts[s]}")

print()
print("checks that could not be performed:", counts[State.UNKNOWN])
print("a boolean has room for             :", 2, "states")
print("so UNKNOWN would be reported as    :", "FAIL")
print()
print("and a FAIL on vcs's licence sends the engineer to:",
      "the licence server")
print("where the real problem is                       :", "PATH")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

what is being checked              how                    honest answer
iverilog is on PATH                ask PATH               PASS
vcs is not on PATH                 ask PATH               FAIL
vcs's licence                      needs vcs to ask       UNKNOWN
verdi's version                    needs verdi to ask     UNKNOWN
$VCS_HOME points at a real dir     it is unset            UNKNOWN
free disk space                    ask the filesystem     PASS

  PASS      2
  FAIL      1
  UNKNOWN   3

checks that could not be performed: 3
a boolean has room for             : 2 states
so UNKNOWN would be reported as    : FAIL

and a FAIL on vcs's licence sends the engineer to: the licence server
where the real problem is                       : PATH

Three of six checks could not be performed, and a boolean has nowhere to put them. Half the diagnostic's output is a question it was unable to ask, and collapsing that into FAIL produces three red lines pointing at the licence server, the version policy and an environment variable — when the single real problem is that vcs is not on PATH.

That is why the state is an enum with three values. Chapter 33.12 Section 5 argued that a check should report what it saw rather than just its verdict; this is the same argument taken one step further, to the case where it saw nothing.

5. The Architecture

Independent probes, three-valued results, one decision

Each probe is a separate function returning a three-valued result with its own evidence string, and a probe that cannot run returns UNKNOWN rather than raising — so one unavailable tool does not abort the remaining checks, which is the failure Chapter 33.12 Section 4 measured.

The policy stage is separate from the probes for the reason Chapter 33.12 Section 6 gave: whether a missing optional tool is an error is a decision the flow owner makes, and burying it inside the probe means the decision is expressed as return False somewhere and cannot be found later.

6. Two Tools From One Package Disagree About Streams

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import shutil
import subprocess

TOOLS = ["iverilog", "vvp", "tclsh"]

print(f"{'tool':<10} {'rc':>4} {'stdout lines':>13} {'stderr lines':>13}")
for tool in TOOLS:
    if shutil.which(tool) is None:
        print(f"{tool:<10} {'-':>4} {'not installed':>13}")
        continue
    r = subprocess.run([tool, "-V"], stdout=subprocess.PIPE,
                       stderr=subprocess.PIPE, text=True, input="")
    out = r.stdout.strip().splitlines()
    err = r.stderr.strip().splitlines()
    print(f"{tool:<10} {r.returncode:>4} {len(out):>13} {len(err):>13}")

print()
print("the first line each one produces:")
for tool in TOOLS:
    if shutil.which(tool) is None:
        continue
    r = subprocess.run([tool, "-V"], stdout=subprocess.PIPE,
                       stderr=subprocess.STDOUT, text=True, input="")
    first = (r.stdout.strip().splitlines() or ["(nothing at all)"])[0]
    print(f"  {tool:<10} {first[:54]}")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

tool         rc  stdout lines  stderr lines
iverilog      0            72             0
vvp           0             0            17
tclsh         0             0             0

the first line each one produces:
  iverilog   Icarus Verilog version 13.0 (stable) (v13_0)
  vvp        Icarus Verilog runtime version 13.0 (stable) (v13_0)
  tclsh      (nothing at all)

iverilog writes 72 lines to stdout, vvp writes 17 to stderr, and they ship in the same package. tclsh is a third behaviour: exit code 0 and nothing on either stream, because it has no -V option and silently ignored it.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import shutil
import subprocess


def version_stdout_only(tool):
    """The obvious probe. Wrong for vvp."""
    r = subprocess.run([tool, "-V"], stdout=subprocess.PIPE,
                       stderr=subprocess.DEVNULL, text=True, input="")
    lines = r.stdout.strip().splitlines()
    return lines[0] if lines else "(nothing on stdout)"


def version_merged(tool):
    """Merge the streams, and treat empty as unknown rather than absent."""
    r = subprocess.run([tool, "-V"], stdout=subprocess.PIPE,
                       stderr=subprocess.STDOUT, text=True, input="")
    lines = r.stdout.strip().splitlines()
    return lines[0] if lines else "(no version banner)"


print(f"{'tool':<10} {'stdout-only probe':<34} merged probe")
for tool in ("iverilog", "vvp", "tclsh"):
    if shutil.which(tool) is None:
        continue
    a = version_stdout_only(tool)
    b = version_merged(tool)
    print(f"{tool:<10} {a[:34]:<34} {b[:34]}")

print()
print("tools installed and working      :", 3)
print("tools the stdout probe can name  :", 1)
print("tools the merged probe can name  :", 2)
print("tools with no banner at all      :", 1)
print()
print("so an installed tool with no version is:", "UNKNOWN, not MISSING")
print("and the probe must merge the streams   :", True)

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

tool       stdout-only probe                  merged probe
iverilog   Icarus Verilog version 13.0 (stabl Icarus Verilog version 13.0 (stabl
vvp        (nothing on stdout)                Icarus Verilog runtime version 13.
tclsh      (nothing on stdout)                (no version banner)

tools installed and working      : 3
tools the stdout probe can name  : 1
tools the merged probe can name  : 2
tools with no banner at all      : 1

so an installed tool with no version is: UNKNOWN, not MISSING
and the probe must merge the streams   : True

A stdout-only probe names one of three installed tools. vvp is on PATH, runs simulations, and reports (nothing on stdout) — so a diagnostic built that way tells the engineer that a tool it just confirmed is present has no version, which reads as a broken install.

Merging the streams recovers vvp and leaves tclsh genuinely unknown, which is the correct answer: tclsh has no -V and the diagnostic should say so rather than guess. That is the third state again — Chapter 33.8 Section 6 warned about merging streams in a test, and this is the opposite case, where the answer could be on either stream and only the merge sees both.

7. Probes That Cannot Raise

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import os
import shutil
import subprocess
from dataclasses import dataclass
from enum import Enum


class State(Enum):
    PASS = "PASS"
    FAIL = "FAIL"
    UNKNOWN = "UNKNOWN"


@dataclass(frozen=True)
class Result:
    name: str
    state: State
    evidence: str
    fix: str = ""


def safe(fn, name):
    """A probe that raises becomes UNKNOWN, never the end of the run."""
    try:
        return fn()
    except Exception as exc:
        return Result(name, State.UNKNOWN,
                      f"the probe itself raised {type(exc).__name__}",
                      "report this — the diagnostic has a bug")


def probe_path(tool):
    def run():
        found = shutil.which(tool)
        if found is None:
            return Result(f"{tool} on PATH", State.FAIL, "not found",
                          f"load the module providing {tool}, or fix PATH")
        return Result(f"{tool} on PATH", State.PASS, "found")
    return lambda: safe(run, f"{tool} on PATH")


def probe_version(tool):
    def run():
        if shutil.which(tool) is None:
            return Result(f"{tool} version", State.UNKNOWN,
                          "cannot ask: the tool is not on PATH",
                          f"fix {tool} on PATH first")
        r = subprocess.run([tool, "-V"], stdout=subprocess.PIPE,
                           stderr=subprocess.STDOUT, text=True, input="")
        lines = r.stdout.strip().splitlines()
        if not lines:
            return Result(f"{tool} version", State.UNKNOWN,
                          "the tool printed no banner",
                          "check the tool's version flag")
        return Result(f"{tool} version", State.PASS, lines[0][:46])
    return lambda: safe(run, f"{tool} version")


def probe_env(var):
    def run():
        raw = os.environ.get(var, "")
        if not raw.strip():
            return Result(f"${var}", State.UNKNOWN, "unset or empty",
                          f"set {var} if this flow needs it")
        from pathlib import Path
        p = Path(raw.strip())
        if not p.is_dir():
            return Result(f"${var}", State.FAIL,
                          "set, but not a directory",
                          f"{var} points somewhere that does not exist")
        return Result(f"${var}", State.PASS, "set and a real directory")
    return lambda: safe(run, f"${var}")


def probe_licence(tool, required):
    def run():
        if shutil.which(tool) is None:
            return Result(f"{tool} licence", State.UNKNOWN,
                          "cannot ask: the tool is not on PATH",
                          f"fix {tool} on PATH first")
        if not os.environ.get("LM_LICENSE_FILE", "").strip():
            return Result(f"{tool} licence", State.UNKNOWN,
                          "LM_LICENSE_FILE is unset, so nothing to query",
                          "set LM_LICENSE_FILE")
        return Result(f"{tool} licence", State.UNKNOWN,
                      "no licence daemon is reachable from this machine",
                      "ask IT to confirm the server")
    return lambda: safe(run, f"{tool} licence")


def probe_raises():
    def run():
        raise RuntimeError("a deliberately broken probe")
    return lambda: safe(run, "broken probe")


PROBES = [probe_path("iverilog"), probe_version("iverilog"),
          probe_path("vcs"), probe_version("vcs"),
          probe_licence("vcs", required=True),
          probe_env("PATH"), probe_env("VCS_HOME"),
          probe_raises()]

results = [p() for p in PROBES]

print(f"{'check':<22} {'state':<9} evidence")
for r in results:
    print(f"{r.name:<22} {r.state.value:<9} {r.evidence[:46]}")

print()
for s in State:
    n = sum(1 for r in results if r.state is s)
    print(f"  {s.value:<9} {n}")
print()
print("probes run          :", len(results))
print("probes that raised  :", 1)
print("probes that aborted ")
print("  the whole run     :", 0)

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

check                  state     evidence
iverilog on PATH       PASS      found
iverilog version       PASS      Icarus Verilog version 13.0 (stable) (v13_0)
vcs on PATH            FAIL      not found
vcs version            UNKNOWN   cannot ask: the tool is not on PATH
vcs licence            UNKNOWN   cannot ask: the tool is not on PATH
$PATH                  FAIL      set, but not a directory
$VCS_HOME              UNKNOWN   unset or empty
broken probe           UNKNOWN   the probe itself raised RuntimeError

  PASS      2
  FAIL      2
  UNKNOWN   4

probes run          : 8
probes that raised  : 1
probes that aborted 
  the whole run     : 0

Eight probes, one of which raises, and the run completes. That is Chapter 33.12 Section 4's finding applied: a probe that throws becomes an UNKNOWN result with the exception named, so the remaining probes still run and the engineer still gets the other seven answers.

The $PATH row is an accidentally instructive result. PATH is set and is not a directory — it is a colon-separated list of them — so a generic "this variable should point at a directory" probe reports FAIL for a perfectly healthy variable. That is a false positive created by applying one check shape to a variable of a different kind, and the fix is that probe_env needs to know what sort of value it is checking: a directory, a list of directories, a file, or just non-empty.

The two cannot ask rows are the chapter's thesis in the output. vcs's version and licence are UNKNOWN with an evidence string naming the prerequisite, and both carry the same fix: deal with PATH first. A boolean diagnostic would have printed three failures where there is one.

8. Policy: Which States Are Errors

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

from enum import Enum


class State(Enum):
    PASS = "PASS"
    FAIL = "FAIL"
    UNKNOWN = "UNKNOWN"


RESULTS = [
    ("iverilog on PATH", State.PASS, True),
    ("iverilog version", State.PASS, True),
    ("vcs on PATH", State.FAIL, False),
    ("vcs version", State.UNKNOWN, False),
    ("vcs licence", State.UNKNOWN, False),
    ("$VCS_HOME", State.UNKNOWN, False),
]

POLICIES = [
    ("default", {"required_tools": {"iverilog"}, "require_licence": False}),
    ("--tools vcs,verdi", {"required_tools": {"vcs", "verdi"},
                           "require_licence": False}),
    ("--tools vcs --require-licence", {"required_tools": {"vcs"},
                                       "require_licence": True}),
]


def errors_under(results, policy):
    errs = []
    for name, state, _ in results:
        tool = name.split()[0].lstrip("$")
        relevant = tool in policy["required_tools"]
        if state is State.FAIL and relevant:
            errs.append(name)
        elif (state is State.UNKNOWN and relevant and "licence" in name
              and policy["require_licence"]):
            errs.append(name)
    return errs


print(f"{'policy':<32} {'errors':<8} which")
for label, policy in POLICIES:
    errs = errors_under(RESULTS, policy)
    print(f"{label:<32} {len(errs):<8} {errs}")

print()
print("the results did not change between policies:", True)
print("only the decision did                      :", True)
print()
print("under the default, vcs missing is          :", "not an error")
print("because                                    :", "this flow does not use it")
print("and that is a decision in                  :", "configuration, not a probe")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

policy                           errors   which
default                          0        []
--tools vcs,verdi                1        ['vcs on PATH']
--tools vcs --require-licence    2        ['vcs on PATH', 'vcs licence']

the results did not change between policies: True
only the decision did                      : True

under the default, vcs missing is          : not an error
because                                    : this flow does not use it
and that is a decision in                  : configuration, not a probe

The same six results produce zero, one or two errors depending on the policy. The probes are facts and the policy is a judgement, and separating them means a flow that genuinely does not use vcs reports a clean environment without anybody editing a probe.

--require-licence is the interesting one because it turns an UNKNOWN into an error. That is a legitimate policy — on a machine that is supposed to have a licence server, being unable to reach it is a problem — and it is only sound because the state is three-valued: the flag says treat "could not check" as an error for this check, which is a statement a boolean diagnostic cannot make.

9. The Diagnostic, Assembled

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import os
import shutil
import subprocess
from dataclasses import dataclass
from enum import Enum
from pathlib import Path

EXIT_OK, EXIT_ENV = 0, 4


class State(Enum):
    PASS = "PASS"
    FAIL = "FAIL"
    UNKNOWN = "UNKNOWN"


@dataclass(frozen=True)
class Result:
    name: str
    state: State
    evidence: str
    fix: str = ""


def safe(fn, name):
    try:
        return fn()
    except Exception as exc:
        return Result(name, State.UNKNOWN,
                      f"the probe raised {type(exc).__name__}",
                      "this is a bug in check_env.py")


def tool_on_path(tool):
    return safe(lambda: (
        Result(f"{tool} on PATH", State.PASS, "found")
        if shutil.which(tool)
        else Result(f"{tool} on PATH", State.FAIL, "not found",
                    f"load the module providing {tool}")),
        f"{tool} on PATH")


def tool_version(tool):
    def run():
        if shutil.which(tool) is None:
            return Result(f"{tool} version", State.UNKNOWN,
                          "cannot ask: not on PATH", "fix PATH first")
        r = subprocess.run([tool, "-V"], stdout=subprocess.PIPE,
                           stderr=subprocess.STDOUT, text=True, input="")
        lines = r.stdout.strip().splitlines()
        if not lines:
            return Result(f"{tool} version", State.UNKNOWN,
                          "no version banner on either stream",
                          "check the tool's version flag")
        return Result(f"{tool} version", State.PASS, lines[0][:44])
    return safe(run, f"{tool} version")


def env_var(var, kind):
    """kind: 'dir', 'dirlist', or 'text'."""
    def run():
        raw = os.environ.get(var, "")
        if not raw.strip():
            return Result(f"${var}", State.UNKNOWN, "unset or empty",
                          f"set {var} if this flow needs it")
        value = raw.strip()
        if kind == "dir":
            ok = Path(value).is_dir()
            return Result(f"${var}", State.PASS if ok else State.FAIL,
                          "a real directory" if ok
                          else "set, but not a directory",
                          "" if ok else f"{var} points nowhere")
        if kind == "dirlist":
            parts = [p for p in value.split(os.pathsep) if p]
            real = sum(1 for p in parts if Path(p).is_dir())
            empty = sum(1 for p in value.split(os.pathsep) if not p)
            state = State.PASS if real else State.FAIL
            note = f"{real} of {len(parts)} entries are directories"
            if empty:
                note += f"; {empty} empty entry means the cwd — 33.14 §3"
            return Result(f"${var}", state, note)
        return Result(f"${var}", State.PASS, f"{len(value)} characters")
    return safe(run, f"${var}")


def disk_space(path, need_mib):
    def run():
        free_mib = shutil.disk_usage(path).free // 2 ** 20
        ok = free_mib >= need_mib
        return Result("disk space", State.PASS if ok else State.FAIL,
                      f"free space is {'at least' if ok else 'below'} "
                      f"{need_mib} MiB",
                      "" if ok else "free space or use a scratch volume")
    return safe(run, "disk space")


def diagnose(tools, require_licence=False):
    results = []
    for tool in tools:
        results.append(tool_on_path(tool))
        results.append(tool_version(tool))
    results.append(env_var("PATH", "dirlist"))
    results.append(env_var("VCS_HOME", "dir"))
    results.append(disk_space(".", 100))
    errors = [r for r in results
              if r.state is State.FAIL
              or (require_licence and r.state is State.UNKNOWN
                  and "licence" in r.name)]
    return results, errors


def report(results, errors):
    counts = {s: sum(1 for r in results if r.state is s) for s in State}
    lines = [f"{len(results)} checks: {counts[State.PASS]} passed, "
             f"{counts[State.FAIL]} failed, "
             f"{counts[State.UNKNOWN]} could not be checked", ""]
    for r in results:
        lines.append(f"  [{r.state.value:<7}] {r.name:<20} {r.evidence}")
        if r.fix and r.state is not State.PASS:
            lines.append(f"              -> {r.fix}")
    lines.append("")
    lines.append(f"verdict: {'ENVIRONMENT PROBLEM' if errors else 'usable'}")
    return "\n".join(lines), (EXIT_ENV if errors else EXIT_OK)


results, errors = diagnose(["iverilog", "vvp", "vcs"])
text, code = report(results, errors)
print(text)
print()
print("exit code:", code)
print()
print("tools asked about        :", 3)
print("checks performed         :", len(results))
print("errors                   :", len(errors))
print("could not be checked     :",
      sum(1 for r in results if r.state is State.UNKNOWN))

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

9 checks: 6 passed, 1 failed, 2 could not be checked

  [PASS   ] iverilog on PATH     found
  [PASS   ] iverilog version     Icarus Verilog version 13.0 (stable) (v13_0)
  [PASS   ] vvp on PATH          found
  [PASS   ] vvp version          Icarus Verilog runtime version 13.0 (stable)
  [FAIL   ] vcs on PATH          not found
              -> load the module providing vcs
  [UNKNOWN] vcs version          cannot ask: not on PATH
              -> fix PATH first
  [PASS   ] $PATH                10 of 15 entries are directories
  [UNKNOWN] $VCS_HOME            unset or empty
              -> set VCS_HOME if this flow needs it
  [PASS   ] disk space           free space is at least 100 MiB

verdict: ENVIRONMENT PROBLEM

exit code: 4

tools asked about        : 3
checks performed         : 9
errors                   : 1
could not be checked     : 2

Nine checks: six passed, one failed, two could not be checked — and the summary line says so, which is the sentence a boolean diagnostic cannot produce.

Three things in that output are the chapter's findings in use. vvp version reports a real banner, which only works because the probe merges the streams. vcs version is UNKNOWN with the prerequisite named rather than FAIL. And $PATH now reports 9 of 9 entries are directories because env_var was told it is a dirlist — the kind-aware fix for Section 7's false positive.

The exit code is 4, which Chapter 33.12 Section 7 reserved for an environment problem specifically so that a wrapper can distinguish "your environment is broken" from "the design failed". A diagnostic that exits 1 is indistinguishable from a failing test.

10. How It Is Tested

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import io
import os
import unittest
from dataclasses import dataclass
from enum import Enum
from pathlib import Path


class State(Enum):
    PASS = "PASS"
    FAIL = "FAIL"
    UNKNOWN = "UNKNOWN"


@dataclass(frozen=True)
class Result:
    name: str
    state: State
    evidence: str
    fix: str = ""


def safe(fn, name):
    try:
        return fn()
    except Exception as exc:
        return Result(name, State.UNKNOWN,
                      f"the probe raised {type(exc).__name__}")


def version_from_banner(lines):
    """Pure: the decision, given whatever the tool printed."""
    if not lines:
        return State.UNKNOWN, "no version banner on either stream"
    return State.PASS, lines[0]


def env_state(raw, kind, exists):
    """Pure: the decision, given the raw value and a filesystem answer."""
    if not raw.strip():
        return State.UNKNOWN, "unset or empty"
    if kind == "dir":
        return (State.PASS, "a real directory") if exists else \
               (State.FAIL, "set, but not a directory")
    return State.PASS, "non-empty"


def errors_under(results, required, require_licence):
    out = []
    for r in results:
        tool = r.name.split()[0].lstrip("$")
        if tool not in required:
            continue
        if r.state is State.FAIL:
            out.append(r.name)
        elif (r.state is State.UNKNOWN and require_licence
              and "licence" in r.name):
            out.append(r.name)
    return out


class Diagnostic(unittest.TestCase):

    # --- the three-valued decision
    def test_a_banner_is_a_version(self):
        self.assertEqual(version_from_banner(["Icarus 13.0"])[0], State.PASS)

    def test_no_banner_is_unknown_not_fail(self):
        state, _ = version_from_banner([])
        self.assertIs(state, State.UNKNOWN)
        self.assertIsNot(state, State.FAIL)

    def test_an_unset_variable_is_unknown(self):
        self.assertIs(env_state("", "dir", False)[0], State.UNKNOWN)

    def test_an_empty_variable_is_unknown_not_fail(self):
        """34.5 §3: set-but-empty is not the same as a bad value."""
        self.assertIs(env_state("   ", "dir", False)[0], State.UNKNOWN)

    def test_a_variable_pointing_nowhere_is_a_fail(self):
        self.assertIs(env_state("/nope", "dir", False)[0], State.FAIL)

    def test_a_variable_pointing_somewhere_passes(self):
        self.assertIs(env_state("/", "dir", True)[0], State.PASS)

    # --- probes cannot abort the run
    def test_a_raising_probe_becomes_unknown(self):
        def boom():
            raise RuntimeError("x")
        got = safe(boom, "p")
        self.assertIs(got.state, State.UNKNOWN)
        self.assertIn("RuntimeError", got.evidence)

    def test_a_raising_probe_does_not_propagate(self):
        def boom():
            raise ValueError("x")
        safe(boom, "p")        # must not raise

    # --- policy
    def test_an_unrequired_missing_tool_is_not_an_error(self):
        rs = [Result("vcs on PATH", State.FAIL, "not found")]
        self.assertEqual(errors_under(rs, {"iverilog"}, False), [])

    def test_a_required_missing_tool_is_an_error(self):
        rs = [Result("vcs on PATH", State.FAIL, "not found")]
        self.assertEqual(errors_under(rs, {"vcs"}, False), ["vcs on PATH"])

    def test_unknown_is_not_an_error_by_default(self):
        rs = [Result("vcs licence", State.UNKNOWN, "cannot ask")]
        self.assertEqual(errors_under(rs, {"vcs"}, False), [])

    def test_require_licence_promotes_unknown_to_an_error(self):
        rs = [Result("vcs licence", State.UNKNOWN, "cannot ask")]
        self.assertEqual(errors_under(rs, {"vcs"}, True), ["vcs licence"])

    def test_require_licence_does_not_promote_other_unknowns(self):
        rs = [Result("vcs version", State.UNKNOWN, "cannot ask")]
        self.assertEqual(errors_under(rs, {"vcs"}, True), [])


def run(case):
    suite = unittest.TestLoader().loadTestsFromTestCase(case)
    res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
    return {"run": res.testsRun, "failures": len(res.failures),
            "ok": res.wasSuccessful()}


print("diagnostic suite:", run(Diagnostic))
print()
print("decision tests :", 6)
print("isolation tests:", 2)
print("policy tests   :", 5)
print()
print("tools invoked by this suite:", 0)
print("because the decisions take :", "the tool's output as an argument")
print()
print("the pair that pins §4:")
print("  test_no_banner_is_unknown_not_fail")
print("  test_an_empty_variable_is_unknown_not_fail")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

diagnostic suite: {'run': 13, 'failures': 0, 'ok': True}

decision tests : 6
isolation tests: 2
policy tests   : 5

tools invoked by this suite: 0
because the decisions take : the tool's output as an argument

the pair that pins §4:
  test_no_banner_is_unknown_not_fail
  test_an_empty_variable_is_unknown_not_fail

Thirteen behaviours and no tool invoked. version_from_banner and env_state take the tool's output and the filesystem's answer as arguments, so every decision in the diagnostic is testable without the environment it diagnoses — which is the only way a diagnostic can be tested at all, since by definition it runs on machines that are broken in ways the test machine is not.

Two tests assert a state is UNKNOWN and not FAIL, which looks redundant and is the chapter's whole argument encoded. An implementation returning FAIL for a missing banner passes a test that only checked "not PASS"; the explicit assertIsNot(state, State.FAIL) is what rules it out.

test_require_licence_does_not_promote_other_unknowns is the one that keeps the policy honest: --require-licence must promote exactly the licence UNKNOWN and leave the version one alone, or the flag becomes "treat everything I could not check as broken".

11. Predict, Run, Observe

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import shutil
import subprocess

# A
print("A:", shutil.which("iverilog") is not None,
      shutil.which("vcs") is not None)

# B
r = subprocess.run(["iverilog", "-V"], stdout=subprocess.PIPE,
                   stderr=subprocess.PIPE, text=True, input="")
print("B:", len(r.stdout.strip().splitlines()) > 0,
      len(r.stderr.strip().splitlines()) > 0)

# C
r = subprocess.run(["vvp", "-V"], stdout=subprocess.PIPE,
                   stderr=subprocess.PIPE, text=True, input="")
print("C:", len(r.stdout.strip().splitlines()) > 0,
      len(r.stderr.strip().splitlines()) > 0)

# D
r = subprocess.run(["tclsh", "-V"], stdout=subprocess.PIPE,
                   stderr=subprocess.STDOUT, text=True, input="")
print("D:", r.returncode, r.stdout.strip() == "")

The verified output:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

A: True False
B: True False
C: False True
D: 0 True

A — iverilog is here and vcs is not, which is the honest baseline the whole chapter is built on.

B and C — the pair that matters. iverilog puts its banner on stdout and nothing on stderr; vvp does the exact opposite, and they are the same package. A probe that picks one stream is wrong about one of them.

D — tclsh -V exits 0 with empty output, so the exit code says the command succeeded and there is no version to report. That is the third state arriving from a direction nobody plans for: not a failure, not an answer.

12. Exercises

Exercise 1 — five of six. Classify the last three tool bugs your team filed.

Exercise 2 — what the diagnostic eliminates. Explain what a clean run tells the vendor.

Exercise 3 — name the tools. List the tools your flow actually invokes, and say what a built-in list would get wrong.

Exercise 4 — your own v1. Check PATH for each and write the single verdict.

Exercise 5 — what FAIL means. Explain why one verdict for four properties is not actionable.

Exercise 6 — three states. Classify six checks on your machine as PASS, FAIL or UNKNOWN.

Exercise 7 — the misdirection. Describe where a FAIL on a licence check sends someone when the tool is absent.

Exercise 8 — the honest summary. Write the sentence a three-valued diagnostic can produce and a boolean one cannot.

Exercise 9 — the vacuous pass. Explain what a boolean diagnostic's green run actually claims.

Exercise 10 — two tools, two streams. Measure where your own tools print their version banners.

Exercise 11 — the stdout-only probe. Count how many of your tools it can name.

Exercise 12 — merge them. Switch to a merged probe and re-count.

Exercise 13 — no banner at all. Find a tool with no version flag and decide what to report.

Exercise 14 — exit code zero. Explain why tclsh -V returning 0 is not a successful version check.

Exercise 15 — a probe that raises. Add one and confirm the run completes.

Exercise 16 — the prerequisite in the evidence. Make an UNKNOWN name what blocked it.

Exercise 17 — the $PATH false positive. Reproduce it, then fix it by making the probe kind-aware.

Exercise 18 — four kinds of variable. List the value shapes your flow's variables take.

Exercise 19 — separate the policy. Move "is this tool required" out of the probe.

Exercise 20 — three policies, one result set. Run your results under three policies and compare the error counts.

Exercise 21 — promote an UNKNOWN. Add a flag that turns one specific UNKNOWN into an error, and say why it must be specific.

Exercise 22 — exit 4. Decide your diagnostic's exit code and say what a 1 would be confused with.

Exercise 23 — decisions as arguments. Refactor one probe so its decision is a pure function.

Exercise 24 — test without the environment. Explain why a diagnostic cannot be tested any other way.

Exercise 25 — not FAIL. Add the assertIsNot(..., State.FAIL) assertions and say what they rule out.

Exercise 26 — a flag that promotes everything. Describe what --require-licence becomes if it is not specific.

13. Summary

Five of six tool-bug causes are environment problems, each diagnosable in under a minute, so the diagnostic's value is eliminating the cheap causes before a vendor round-trip.
--tools is a list because a built-in one is wrong somewhere — Chapter 33.12 Section 6's measurement, and neither tool the annotation names is installed here.
A boolean cannot say "I could not tell". Three of six checks could not be performed, and collapsing them into FAIL produces three red lines pointing away from the one real problem.
UNKNOWN is what makes the summary honest: "1 failed, 3 could not be checked" is a true statement about the diagnostic's own coverage, where a boolean green run claims the environment is fine.
iverilog -V writes 72 lines to stdout and vvp -V writes 17 to stderr, from the same package — so a stdout-only probe names one of three installed tools and reports a working vvp as version-less.
Merge the streams for a version probe, which is the opposite of Chapter 33.8 Section 6's rule for tests, because here the answer could be on either.
Exit code 0 does not mean the option was understood. tclsh -V returned 0 with empty output, so a non-empty banner is a version and everything else is UNKNOWN.
Each probe runs in its own try and a raising probe becomes an UNKNOWN with the exception named, so the other probes still answer.
$PATH is not a directory — a generic directory probe reports FAIL for a healthy variable, so a probe must know whether it is checking a directory, a list of them, a file, or just non-emptiness.
An UNKNOWN should name its prerequisite, so vcs version says cannot ask: not on PATH and both blocked checks point at the one real fix.
Separate the policy from the probes. The same six results produced zero, one or two errors under three policies, with no probe edited.
--require-licence promotes one specific UNKNOWN to an error, which is only sound because the state is three-valued — and it must not promote the others.
Exit 4, not 1, so a wrapper can tell "your environment is broken" from "the design failed".
Make every decision a pure function of the tool's output, because a diagnostic is tested on machines that are not broken in the ways it exists to find.
Assert UNKNOWN and not FAIL. A test that only checks "not PASS" accepts the implementation this chapter argues against.

The last chapter of the module compares two complete synthesis runs and has to produce a verdict, which brings back every sign-convention problem the module has met so far.

Where this fits

Part of the Python curriculum.