Python Part II · Module 23 · Running EDA Tools from Python

Working Directory, Generated Artefacts and Cleanup

EDA tools are famously untidy. A compile drops a simulator binary, an object-code tree, a library directory and a handful of lock files, and it drops them wherever the process happened to be standing when it started — which, as Chapter 23.1 established, is wherever your script was standing. Giving every run its own directory is one keyword argument. It also changes the meaning of every relative path in the command, which is why the fix frequently arrives together with a new failure that looks nothing like a directory problem.

Advanced17 min readPythonPart IIAutomationEDA ToolsVerification Flow

Part II · Module 23 · Chapter 23.7 · Working Directory, Generated Artefacts and Cleanup

1. Tools Write Where They Stand

Azvya Education Pvt. Ltd.VLSI Mentor

fake_compile.py

import os

open("simv", "w").write("a simulator binary\n")
os.makedirs("csrc", exist_ok=True)
open(os.path.join("csrc", "obj.o"), "w").write("object code\n")
os.makedirs("xcelium.d", exist_ok=True)
open(os.path.join("xcelium.d", "lib.lib"), "w").write("library\n")
print("compile finished")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import subprocess
import sys
from pathlib import Path

subprocess.run([sys.executable, "fake_compile.py"], capture_output=True)

print("beside the script now:")
for name in sorted(p.name for p in Path(".").iterdir()):
    print("   " + name)

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

beside the script now:
   csrc
   fake_compile.py
   main.py
   simv
   xcelium.d

One compile and the directory has three new entries it did not ask for. Run a second test and they are overwritten; run two tests at once and they collide.

This is not the tool misbehaving — it is Chapter 23.1 Section 3's inheritance working exactly as described. The child started in the parent's working directory, and an output path the tool did not qualify is relative to that.

The names are the real ones: simv is VCS's simulator binary, csrc its generated C sources, xcelium.d Xcelium's library directory, work/ Questa's. Every one of them is written beside your script by default.

2. One Directory per Run

Figure 1 — one run, one directory. Inputs come from outside it and must be named absolutely; outputs land inside it because the tool is standing there and nobody told it otherwise.

Azvya Education Pvt. Ltd.VLSI Mentor

fake_compile.py

import os

open("simv", "w").write("a simulator binary\n")
os.makedirs("csrc", exist_ok=True)
open(os.path.join("csrc", "obj.o"), "w").write("object code\n")
print("compile finished")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import subprocess
import sys
from pathlib import Path

script = Path("fake_compile.py").resolve()
run_dir = Path("runs/axi_smoke_1839274")
run_dir.mkdir(parents=True, exist_ok=True)

cp = subprocess.run([sys.executable, str(script)], cwd=run_dir, capture_output=True, text=True)

print("tool said      :", cp.stdout.strip())
print("in the run dir :", sorted(p.name for p in run_dir.iterdir()))
print("beside us      :", sorted(p.name for p in Path(".").iterdir() if p.name != "runs"))
print("our cwd moved  :", Path.cwd() != Path.cwd())

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

tool said      : compile finished
in the run dir : ['csrc', 'simv']
beside us      : ['fake_compile.py', 'main.py']
our cwd moved  : False

That is the canonical annotation:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

subprocess.run(cmd, cwd=run_dir)

The artefacts are in the run directory and the parent's directory is clean. And the parent itself never moved — Chapter 23.1 Section 4's one-way boundary means cwd= affects the child only, which is the property that makes this safe to use in a loop.

run_dir.mkdir(parents=True, exist_ok=True) first: cwd= requires the directory to exist, and a missing one raises FileNotFoundError naming the directory — which Chapter 23.9 classifies as a launch failure.

The directory name carries the test and the seed, which is Chapter 20.8's point about reproducibility arriving as a layout decision. Two runs of the same test with different seeds get different directories and cannot collide.

3. And Now the Filelist Is Not Found

Azvya Education Pvt. Ltd.VLSI Mentor

files.f

rtl/alu.sv
rtl/fifo.sv

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import subprocess
import sys
from pathlib import Path

run_dir = Path("runs/axi_smoke")
run_dir.mkdir(parents=True, exist_ok=True)

CHECK = "import os, sys; print('found:', os.path.exists(sys.argv[1]))"

relative = subprocess.run([sys.executable, "-c", CHECK, "files.f"],
                          cwd=run_dir, capture_output=True, text=True)
absolute = subprocess.run([sys.executable, "-c", CHECK, str(Path("files.f").resolve())],
                          cwd=run_dir, capture_output=True, text=True)
no_cwd = subprocess.run([sys.executable, "-c", CHECK, "files.f"],
                        capture_output=True, text=True)

print("with cwd=, relative path :", relative.stdout.strip())
print("with cwd=, absolute path :", absolute.stdout.strip())
print("without cwd=, relative   :", no_cwd.stdout.strip())

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

with cwd=, relative path : found: False
with cwd=, absolute path : found: True
without cwd=, relative   : found: True

The same command, the same file, and adding cwd= broke it.

files.f is relative, and a relative path is resolved against the child's working directory — which is now the run directory, where no such file exists. The file is still exactly where it was; the question "relative to what?" got a new answer.

This is why adopting cwd= is not a one-line change, and why the symptom is misleading. The tool reports something like cannot open files.f or Error-[CFF] Cannot find file, which points at the filelist. The filelist is fine. Somebody added a keyword argument in a different function.

The rule, and it is the whole chapter in a sentence:

Absolute paths for inputs; relative paths for outputs.

Inputs are resolved by the parent, which knows where they are. Outputs are left unqualified so the tool drops them where it is standing — which is the run directory by construction.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

from pathlib import Path


def compile_cmd(filelist, top, log_name="compile.log", out_name="simv"):
    """Inputs absolute, outputs relative."""
    return ["xrun", "-64bit",
            "-f", str(Path(filelist).resolve()),      # input: absolute
            "-top", top,
            "-l", log_name,                           # output: relative
            "-o", out_name]                           # output: relative


Path("files.f").touch()
for part in compile_cmd("files.f", "tb_top"):
    kind = ""
    if part.startswith("/"):
        kind = "  <- absolute (an input)"
    elif part.endswith((".log", "simv")):
        kind = "  <- relative (an output)"
    print(f"  {part.split('/')[-1] if part.startswith('/') else part}{kind}")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

  xrun
  -64bit
  -f
  files.f  <- absolute (an input)
  -top
  tb_top
  -l
  compile.log  <- relative (an output)
  -o
  simv  <- relative (an output)

The printing shows only the final component of the absolute path, because the real one differs on every machine — but the command itself carries the full path, which is what the tool needs.

Path(...).resolve() is the conversion, and it is worth applying to every input the moment it enters your code rather than at the point of use. A path that has been absolute since it was read cannot become ambiguous later.

4. `cwd=` Does Not Change the Parent

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import subprocess
import sys
from pathlib import Path

before = Path.cwd()
run_dir = Path("runs/a")
run_dir.mkdir(parents=True, exist_ok=True)

subprocess.run([sys.executable, "-c", "pass"], cwd=run_dir)

print("parent unchanged  :", Path.cwd() == before)
print("relative still ok :", Path("runs").is_dir())

for name in ["runs/a", "runs/b", "runs/c"]:
    Path(name).mkdir(parents=True, exist_ok=True)
    cp = subprocess.run([sys.executable, "-c",
                         "import os; print(os.path.basename(os.getcwd()))"],
                        cwd=name, capture_output=True, text=True)
    print(f"  ran in {name:<8} child reported: {cp.stdout.strip()}")
print("parent still      :", Path.cwd() == before)

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

parent unchanged  : True
relative still ok : True
  ran in runs/a   child reported: a
  ran in runs/b   child reported: b
  ran in runs/c   child reported: c
parent still      : True

Three children in three directories, and the parent never moved. Every relative path in the parent still means what it meant.

That is what makes cwd= usable in a sweep: there is no state to restore, because nothing changed. A loop over four hundred tests needs no care at all, and a tool that raises part-way leaves nothing to undo.

5. Why Not `os.chdir`

The alternative people reach for first:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import os
import subprocess
import sys
from pathlib import Path

root = Path.cwd()
Path("runs/d").mkdir(parents=True, exist_ok=True)
Path("marker.txt").write_text("beside the script\n", encoding="utf-8")

os.chdir("runs/d")
subprocess.run([sys.executable, "-c", "pass"])
print("after chdir, marker.txt visible:", Path("marker.txt").exists())
os.chdir(root)
print("after chdir back, visible      :", Path("marker.txt").exists())

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

after chdir, marker.txt visible: False
after chdir back, visible      : True

os.chdir works, and it has three problems cwd= does not.

It moves the parent, so every relative path in the rest of your script changes meaning — including ones in code you did not write, several functions away. The marker.txt above is a file the script could see before the chdir and not after.

It is not undone if anything raises. The os.chdir(root) on the last line only runs if nothing between threw, so an exception leaves the whole script in the wrong directory, and the next thing it does is wrong in a way that has nothing to do with the exception. A try/finally fixes that and is three more lines than cwd=.

It is process-wide. A script that ever runs two tools concurrently — Chapter 27.3's parallel regression — has one working directory shared between them, so the second chdir silently changes where the first tool's relative paths resolve. cwd= gives each child its own, which is the only version that survives concurrency.

cwd= has none of the three because it does not change anything: it tells the operating system where to start the child, which is a property of the launch rather than a mutation of the parent.

6. What a Run Leaves Behind

Azvya Education Pvt. Ltd.VLSI Mentor

fake_run.py

import os
import sys

open("sim.log", "w").write("UVM_INFO starting\nTEST DONE\n")
open("simv", "w").write("binary\n")
os.makedirs("csrc", exist_ok=True)
open(os.path.join("csrc", "obj.o"), "w").write("x" * 500)
open("waves.fsdb", "w").write("waveform\n")
sys.exit(int(sys.argv[1]))

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import subprocess
import sys
from pathlib import Path

run_dir = Path("runs/axi_smoke")
run_dir.mkdir(parents=True, exist_ok=True)
script = Path("fake_run.py").resolve()

cp = subprocess.run([sys.executable, str(script), "1"], cwd=run_dir, capture_output=True)

print("left behind:")
for item in sorted(run_dir.rglob("*")):
    kind = "dir " if item.is_dir() else "file"
    print(f"   {kind} {item.relative_to(run_dir)}")
print("returncode :", cp.returncode)

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

left behind:
   dir  csrc
   file csrc/obj.o
   file sim.log
   file simv
   file waves.fsdb

Four kinds of thing, and they are worth keeping for very different lengths of time:

Artefact	Worth keeping	Why
`sim.log`	always	the evidence — Chapter 21.1's `log_path`
`cmd.sh`	always	how to re-run it — Chapter 23.10
`waves.fsdb`	on failure	needed to debug, large, useless on a pass
`simv`, `csrc/`	rarely	regenerable, and the bulk of the disk

The decision is a function of the run's status, which is what makes it reviewable rather than a habit:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import shutil
from pathlib import Path

KEEP_ALWAYS = {"sim.log", "cmd.sh"}
KEEP_ON_FAILURE = {"waves.fsdb"}
BUILD_ARTEFACTS = {"simv", "csrc", "xcelium.d", "work"}


def cleanup(run_dir, status, keep_builds=False):
    """Delete what this run no longer needs. Returns what was removed."""
    run_dir = Path(run_dir)
    removed = []
    for item in sorted(run_dir.iterdir()):
        name = item.name
        if name in KEEP_ALWAYS:
            continue
        if name in KEEP_ON_FAILURE and status != "PASS":
            continue
        if name in BUILD_ARTEFACTS and keep_builds:
            continue
        if item.is_dir():
            shutil.rmtree(item)
        else:
            item.unlink()
        removed.append(name)
    return removed


for status in ["PASS", "FAIL"]:
    run_dir = Path(f"runs/{status.lower()}")
    run_dir.mkdir(parents=True, exist_ok=True)
    for name in ["sim.log", "cmd.sh", "waves.fsdb", "simv"]:
        (run_dir / name).write_text("x", encoding="utf-8")
    (run_dir / "csrc").mkdir(exist_ok=True)
    (run_dir / "csrc" / "obj.o").write_text("x", encoding="utf-8")

    removed = cleanup(run_dir, status)
    kept = sorted(p.name for p in run_dir.iterdir())
    print(f"{status}: removed {removed}")
    print(f"      kept    {kept}")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

PASS: removed ['csrc', 'simv', 'waves.fsdb']
      kept    ['cmd.sh', 'sim.log']
FAIL: removed ['csrc', 'simv']
      kept    ['cmd.sh', 'sim.log', 'waves.fsdb']

A passing run keeps two small files; a failing one keeps the waveform as well. Three properties of that function matter more than its exact policy.

It is driven by the status, so the policy is one table a team can argue about rather than a habit spread through a script.

It never deletes anything outside run_dir. Chapter 18.9's safe-cleanup rule: every path comes from iterating the directory, so a bug cannot produce a path somewhere else.

It returns what it removed, which is both a log line and the thing a test asserts on. A cleanup that reports nothing is one nobody can verify.

7. One Run, Assembled

Azvya Education Pvt. Ltd.VLSI Mentor

fake_run.py

import os
import sys

open("sim.log", "w").write("UVM_INFO starting\nUVM_ERROR drv.sv(88) @ 141900: bad\nTEST DONE\n")
open("simv", "w").write("binary\n")
os.makedirs("csrc", exist_ok=True)
open(os.path.join("csrc", "obj.o"), "w").write("obj\n")
sys.exit(int(sys.argv[1]))

Azvya Education Pvt. Ltd.VLSI Mentor

files.f

rtl/alu.sv

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import shutil
import subprocess
import sys
from pathlib import Path

BUILD_ARTEFACTS = {"simv", "csrc"}


def run_one(test, seed, filelist, code):
    """One isolated run: its own directory, absolute inputs, relative outputs."""
    run_dir = Path("runs") / f"{test}_{seed}"
    run_dir.mkdir(parents=True, exist_ok=True)

    cmd = [sys.executable, str(Path("fake_run.py").resolve()), str(code)]

    cp = subprocess.run(cmd, cwd=run_dir, capture_output=True, text=True)

    status = "PASS" if cp.returncode == 0 else "FAIL"

    if status == "PASS":
        for name in BUILD_ARTEFACTS:
            target = run_dir / name
            if target.is_dir():
                shutil.rmtree(target)
            elif target.exists():
                target.unlink()

    return {"test": test, "seed": seed, "status": status,
            "run_dir": str(run_dir), "log": str(run_dir / "sim.log"),
            "kept": sorted(p.name for p in run_dir.iterdir())}


Path("files.f").write_text("rtl/alu.sv\n", encoding="utf-8")

for test, seed, code in [("axi_smoke", 1839274, 1), ("axi_reset", 771002, 0)]:
    record = run_one(test, seed, "files.f", code)
    print(f"{record['test']:<11} {record['status']:<5} {record['run_dir']:<24} kept={record['kept']}")

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

axi_smoke   FAIL  runs/axi_smoke_1839274   kept=['csrc', 'sim.log', 'simv']
axi_reset   PASS  runs/axi_reset_771002    kept=['sim.log']

Two runs, two directories, no collision, and the passing one has already given its build tree back.

The record carries run_dir as well as log, which is Chapter 21.1 Section 4's point: the log says what happened and the run directory is where everything else is. A failing result whose record names only the log sends somebody looking for the waves.

Three things this still does not do, and they are the next three chapters. It waits forever (Chapter 23.8). It assumes the tool exists (Chapter 23.9). And it does not record the command, so nobody can repeat it (Chapter 23.10).

8. Predict, Run, Observe

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

import os
import subprocess
import sys
from pathlib import Path

root = Path.cwd()
Path("runs/x").mkdir(parents=True, exist_ok=True)
Path("here.txt").write_text("hi", encoding="utf-8")
EXISTS = "import os, sys; print(os.path.exists(sys.argv[1]))"

# A
print("A:", subprocess.run([sys.executable, "-c", EXISTS, "here.txt"],
                           capture_output=True, text=True).stdout.strip())

# B
print("B:", subprocess.run([sys.executable, "-c", EXISTS, "here.txt"], cwd="runs/x",
                           capture_output=True, text=True).stdout.strip())

# C
print("C:", subprocess.run([sys.executable, "-c", EXISTS, str(Path("here.txt").resolve())],
                           cwd="runs/x", capture_output=True, text=True).stdout.strip())

# D
subprocess.run([sys.executable, "-c", "open('made.txt','w').write('x')"], cwd="runs/x")
print("D:", Path("runs/x/made.txt").exists(), Path("made.txt").exists())

# E
print("E:", Path.cwd() == root)

# F
try:
    subprocess.run([sys.executable, "-c", "pass"], cwd="runs/nope")
except FileNotFoundError as exc:
    print("F:", type(exc).__name__)

# G
print("G:", Path("here.txt").resolve().is_absolute())

The verified output:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

A: True
B: False
C: True
D: True False
E: True
F: FileNotFoundError
G: True

A — without cwd=, the relative path resolves beside the parent and is found.

B — with cwd=, the same string resolves in the run directory and is not. Section 3's whole point.

C — resolved to an absolute path, it is found from anywhere.

D — the child's output landed in the run directory, not beside the parent.

E — the parent never moved.

F — a missing cwd= directory raises before the tool starts. Chapter 23.9's class of failure.

G — .resolve() is what makes a path independent of where anything is standing.

9. Exercises

Exercise 1 — the mess. Run a stand-in tool with no cwd= and list what appeared beside your script.

Exercise 2 — name them. For the simulator your flow uses, list the artefacts it drops in the current directory.

Exercise 3 — one per run. Add cwd= with a directory named for the test and seed. Confirm the artefacts moved and the parent is clean.

Exercise 4 — and now it breaks. Keep a relative input path in the command and run with cwd=. Record the tool's error and say what it points at.

Exercise 5 — the rule. Convert your command so inputs are absolute and outputs relative. Say which elements changed.

Exercise 6 — resolve early. Resolve every input path at the point it enters your code rather than at the point of use. Say what that prevents.

Exercise 7 — the absolute output. Name an output absolutely and run two tests concurrently in your head. Describe the collision.

Exercise 8 — three directories. Run three children in three directories in a loop and confirm the parent never moves.

Exercise 9 — chdir's three problems. Reproduce the os.chdir version and demonstrate at least one of: a changed relative path, a missing restore after an exception, or a concurrency hazard.

Exercise 10 — what a run leaves. List everything a real run of your flow leaves behind and classify each as always-keep, keep-on-failure or regenerable.

Exercise 11 — cleanup by status. Write a cleanup driven by status that never touches anything outside the run directory and returns what it removed.

Exercise 12 — the disk. Say which artefact in your flow is the largest, whether it is regenerable, and what reproduces it.

10. Summary

A tool writes its artefacts in whatever directory it was started in, which Chapter 23.1 established is the parent's.
simv, csrc/, xcelium.d/ and work/ are the real names, and they collide the moment two runs share a directory.
cwd=run_dir starts the child somewhere else, and mkdir(parents=True, exist_ok=True) first because a missing directory raises.
Name the run directory for the test and the seed, so two runs of one test cannot collide.
Adding cwd= changes how every relative path in the command resolves — a filelist beside the parent is suddenly not found, and the tool's error points at the filelist.
Absolute paths for inputs; relative paths for outputs. The parent knows where the inputs are; the outputs belong wherever the tool is standing.
.resolve() every input as it enters your code, not at the point of use.
An output named absolutely ignores the run directory, so it must carry something unique to the run.
cwd= does not move the parent, so there is no state to restore and a sweep needs no care.
os.chdir has three problems: it changes every relative path in the rest of the script, it is not undone when something raises, and it is process-wide so it breaks under concurrency.
Keep the log and the command always, the waves on failure, and treat the build trees as regenerable.
The build tree is where the disk goes, and deleting it on success is safe because cmd.sh reproduces it.
Drive cleanup from the run's status, never delete a path that did not come from iterating the run directory, and return what was removed.
Record run_dir alongside log — the log says what happened and the directory is where everything else is.

The next chapter bounds the one thing none of this has yet: how long the tool is allowed to take.

Where this fits

Part of the Python curriculum.

1. Tools Write Where They Stand

2. One Directory per Run

3. And Now the Filelist Is Not Found

4. cwd= Does Not Change the Parent

5. Why Not os.chdir

6. What a Run Leaves Behind

7. One Run, Assembled

8. Predict, Run, Observe

9. Exercises

10. Summary

Where this fits

4. `cwd=` Does Not Change the Parent

5. Why Not `os.chdir`