Run Directories and Scratch Space

Part II · Module 24 · Chapter 24.3 · Run Directories and Scratch Space

1. Two Kinds of File

PRODUCED = [
    ("sim.log", "small", "the evidence", "keep"),
    ("cmd.sh", "small", "how to repeat it", "keep"),
    ("record.json", "small", "what a program reads", "keep"),
    ("waves.fsdb", "large", "needed to debug a failure", "keep on failure"),
    ("csrc/", "very large", "generated C for the build", "discard"),
    ("xcelium.d/", "very large", "the compiled library", "discard"),
    ("vlog_sorted.tmp", "medium", "an intermediate nobody reads", "discard"),
]

print(f"{'artefact':<16} {'size':<11} {'why it exists':<28} {'fate'}")
for name, size, why, fate in PRODUCED:
    print(f"{name:<16} {size:<11} {why:<28} {fate}")

keep = sum(1 for _, _, _, f in PRODUCED if f.startswith("keep"))
print()
print(f"keep {keep} of {len(PRODUCED)} — and the discards are the large ones")

artefact         size        why it exists                fate
sim.log          small       the evidence                 keep
cmd.sh           small       how to repeat it             keep
record.json      small       what a program reads         keep
waves.fsdb       large       needed to debug a failure    keep on failure
csrc/            very large  generated C for the build    discard
xcelium.d/       very large  the compiled library         discard
vlog_sorted.tmp  medium      an intermediate nobody reads discard

keep 4 of 7 — and the discards are the large ones

Chapter 23.7 Section 6 made that classification and applied it as a cleanup afterwards. Cleanup works and has a weakness: between the tool finishing and the cleanup running, everything is in one place, so the peak disk usage is the sum of both kinds — and if the run is killed, the cleanup never happens at all.

The alternative is to never put them together: give the tool a different directory for the throwaway output in the first place. Then a killed run leaves no mess in the run directory, the archive step needs no filter, and the peak usage on the filesystem you care about is only ever the small set.

There is a second reason, and in a real environment it is usually the louder one. The two directories often want to be on different filesystems. Results live on a shared network volume that is backed up; intermediates want fast local disk that is not. Writing a multi-gigabyte build tree over NFS is slow in a way that shows up in every runtime number you report.

2. The Run Directory, Kept

from pathlib import Path

RUNS = Path("runs")


def run_dir_for(test, seed):
    d = RUNS / f"{test}_{seed}"
    d.mkdir(parents=True, exist_ok=True)
    return d


for test, seed in (("axi_smoke", 1839274), ("axi_smoke", 771002), ("axi_reset", 1839274)):
    d = run_dir_for(test, seed)
    print(f"{test:<10} seed {seed:<8} -> {d}")

print()
print("distinct directories:", len({str(run_dir_for(t, s))
                                   for t, s in (("axi_smoke", 1839274),
                                                ("axi_smoke", 771002),
                                                ("axi_reset", 1839274))}))

axi_smoke  seed 1839274  -> runs/axi_smoke_1839274
axi_smoke  seed 771002   -> runs/axi_smoke_771002
axi_reset  seed 1839274  -> runs/axi_reset_1839274

distinct directories: 3

This is Chapter 23.7 Section 2's directory, and the naming is the part worth restating: test and seed together, because either alone collides. Three runs, three directories, and the same test at two seeds does not overwrite itself.

What this chapter adds is that the run directory is now explicitly the small, kept, archived half — and everything else needs somewhere to go.

3. Scratch, Discarded

runs/axi_smoke_1839274/     <- small, kept, archived, often on a network volume
    sim.log
    cmd.sh
    record.json

<scratch>/axi_smoke_1839274/   <- large, discarded, local fast disk
    csrc/
    xcelium.d/
    vlog_sorted.tmp

The scratch location is not something a script should decide on its own, because it depends on the machine: a workstation, a compute farm node and a container each have a different right answer, and the site usually publishes one through an environment variable. SCRATCH is a common spelling; TMPDIR is the standard one.

So the shape is: take it from the environment, fall back to something safe. Which is exactly what the canonical annotation says, and exactly where the problem is.

4. The Annotation, and the Bug Inside It

run_dir = RUNS / f"{test}_{seed}"
scratch = Path(os.environ.get("SCRATCH", run_dir/"tmp"))

Read that second line against Chapter 24.2 Section 2 and the fault is already visible. Here it is measured:

import os
from pathlib import Path

RUNS = Path("runs")
run_dir = RUNS / "axi_smoke_1839274"

print("Path('') is:", repr(Path("")))
print("Path('') == Path('.'):", Path("") == Path("."))
print()

for label, setting in (("SCRATCH unset", None), ("SCRATCH empty", ""),
                       ("SCRATCH set", "/big/scratch")):
    os.environ.pop("SCRATCH", None)
    if setting is not None:
        os.environ["SCRATCH"] = setting
    scratch = Path(os.environ.get("SCRATCH", run_dir / "tmp"))
    print(f"{label:<14} scratch={str(scratch):<28} under run_dir: {str(scratch).startswith('runs')}")

os.environ.pop("SCRATCH", None)

Path('') is: PosixPath('.')
Path('') == Path('.'): True

SCRATCH unset  scratch=runs/axi_smoke_1839274/tmp   under run_dir: True
SCRATCH empty  scratch=.                            under run_dir: False
SCRATCH set    scratch=/big/scratch                 under run_dir: False

With SCRATCH exported empty, scratch becomes Path(".") — the current working directory.

Two things conspire. The default does not fire, because the key is present — Chapter 24.2 Section 2, which listed four ordinary ways a variable ends up exported-but-empty. And Path("") is not an error or a null; it is Path("."), a perfectly valid relative path meaning here.

The consequence depends on what here is, and both answers are bad:

• with Chapter 23.7's cwd=run_dir, the tool's scratch lands in the run directory — so the gigabytes are back in the archived half, which is the exact outcome this chapter exists to prevent;
• with no cwd=, it lands beside your script, which is where Chapter 23.7 Section 1 started.

And nothing fails. The run passes, the log is correct, and the only symptom is a disk filling up somewhere nobody is looking.

The fix is Chapter 24.2's operator:

import os
from pathlib import Path

RUNS = Path("runs")
run_dir = RUNS / "axi_smoke_1839274"

for label, setting in (("SCRATCH unset", None), ("SCRATCH empty", ""),
                       ("SCRATCH set", "/big/scratch")):
    os.environ.pop("SCRATCH", None)
    if setting is not None:
        os.environ["SCRATCH"] = setting
    naive = Path(os.environ.get("SCRATCH", run_dir / "tmp"))
    fixed = Path(os.environ.get("SCRATCH") or run_dir / "tmp")
    print(f"{label:<14} naive={str(naive):<28} or-form={fixed}")

os.environ.pop("SCRATCH", None)

SCRATCH unset  naive=runs/axi_smoke_1839274/tmp   or-form=runs/axi_smoke_1839274/tmp
SCRATCH empty  naive=.                            or-form=runs/axi_smoke_1839274/tmp
SCRATCH set    naive=/big/scratch                 or-form=/big/scratch

One character — , becomes or — and the empty case falls back to the safe default like the absent one.

5. Two Directories, One Run

6. tempfile, TMPDIR, and a Cache

Python's own temporary files obey the same environment, and with one trap of their own:

import os
import tempfile
from pathlib import Path

d = Path("myscratch").resolve()
d.mkdir(exist_ok=True)

os.environ["TMPDIR"] = str(d)
tempfile.tempdir = None          # force a fresh look at the environment

print("gettempdir is our directory:", tempfile.gettempdir() == str(d))

with tempfile.TemporaryDirectory() as td:
    print("TemporaryDirectory inside it:", Path(td).parent == d)

other = Path("elsewhere").resolve()
other.mkdir(exist_ok=True)
with tempfile.TemporaryDirectory(dir=other) as td:
    print("dir= overrides TMPDIR       :", Path(td).parent == other)

gettempdir is our directory: True
TemporaryDirectory inside it: True
dir= overrides TMPDIR       : True

The booleans rather than the paths are deliberate: the real temp directory differs on every machine, so what is published is the relationship.

Note the tempfile.tempdir = None line, because without it the first assertion can fail:

import os
import tempfile
from pathlib import Path

d = Path("latescratch").resolve()
d.mkdir(exist_ok=True)

first = tempfile.gettempdir()        # anything at all that uses tempfile
os.environ["TMPDIR"] = str(d)
second = tempfile.gettempdir()

print("TMPDIR set after first use took effect:", second == str(d))
print("the same value came back              :", first == second)

tempfile.tempdir = str(d)
print("assigning tempfile.tempdir works      :", tempfile.gettempdir() == str(d))

TMPDIR set after first use took effect: False
the same value came back              : True
assigning tempfile.tempdir works      : True

tempfile reads the environment once and caches the answer in tempfile.tempdir. Setting TMPDIR afterwards changes nothing, and the first use can be some library you did not know called it.

So there are two correct approaches and one that merely looks correct:

Set TMPDIR before anything runs — at the top of the script, Chapter 24.2 Section 3's early, once, and visibly.

Or pass dir= at every call site, which overrides unconditionally and needs no ordering argument at all. For a flow that already knows its scratch directory, this is the clearer one.

Setting os.environ["TMPDIR"] in the middle of a function is the one that looks correct, and whether it works depends on whether anything happened to touch tempfile first.

import tempfile
from pathlib import Path

scratch = Path("myscratch").resolve()
scratch.mkdir(exist_ok=True)

handle = tempfile.NamedTemporaryFile(dir=scratch, delete=False, suffix=".f")
name = Path(handle.name)
handle.write(b"rtl/alu.sv\n")
handle.close()

print("still there after close():", name.exists())
print("in our scratch directory :", name.parent == scratch)
print("suffix preserved         :", name.suffix)

name.unlink()
print("after unlink             :", name.exists())

still there after close(): True
in our scratch directory : True
suffix preserved         : .f
after unlink             : False

delete=False is needed whenever a tool has to read the file, because the tool is a separate process and the file must survive your close(). A generated filelist is the standard case. The cost is that deleting it is now yours, and a flow that forgets leaves one small file per run forever — which is why scratch gets deleted wholesale in Section 8 rather than file by file.

7. Telling the Tool Where Scratch Is

import os
import subprocess
import sys
from pathlib import Path


def run_paths(test, seed, runs_root="runs"):
    """The two directories for one run, both absolute, both created."""
    run_dir = (Path(runs_root) / f"{test}_{seed}").resolve()
    base = os.environ.get("SCRATCH") or (run_dir / "tmp")
    scratch = (Path(base) / f"{test}_{seed}").resolve()
    run_dir.mkdir(parents=True, exist_ok=True)
    scratch.mkdir(parents=True, exist_ok=True)
    return run_dir, scratch


run_dir, scratch = run_paths("axi_smoke", 1839274)

env = {**os.environ, "TMPDIR": str(scratch)}

SHOW = ("import os, sys; "
        "print('child TMPDIR is scratch:', os.environ['TMPDIR'] == sys.argv[1]); "
        "print('child cwd is run_dir   :', os.getcwd() == sys.argv[2])")

cp = subprocess.run([sys.executable, "-c", SHOW, str(scratch), str(run_dir)],
                    cwd=run_dir, env=env, capture_output=True, text=True)
print(cp.stdout.strip())
print("both absolute          :", run_dir.is_absolute() and scratch.is_absolute())
print("scratch under run_dir  :", scratch.is_relative_to(run_dir))

child TMPDIR is scratch: True
child cwd is run_dir   : True
both absolute          : True
scratch under run_dir  : True

Four decisions there, and the third is the one Chapter 23.7 made the hard way.

The tool is told through TMPDIR, not through a flag, because most tools honour it and the ones that do not have their own flag — vcs -Mdir, Verilator's --Mdir — which belongs in Chapter 23.11's adapter.

The scratch path carries the test and seed too. A shared SCRATCH volume is shared by every run on the machine, so a flat scratch directory collides exactly as a flat run directory would.

Both paths are absolute. Chapter 23.7 Section 3: once cwd= is in play, a relative path in the command or the environment resolves against the child's directory. A relative TMPDIR would quietly put scratch inside the run directory again.

Both are created before launch. A TMPDIR pointing at a directory that does not exist produces a failure deep inside the tool, far from the line that set it.

The last line shows scratch is under the run directory here, because SCRATCH is not set on this machine — the safe fallback, doing its job.

8. Deleting Scratch Without Touching Results

import shutil
from pathlib import Path


def make_run(test, seed):
    run_dir = Path("runs") / f"{test}_{seed}"
    scratch = run_dir / "tmp"
    scratch.mkdir(parents=True, exist_ok=True)
    for name in ("sim.log", "cmd.sh", "record.json"):
        (run_dir / name).write_text("x", encoding="utf-8")
    (scratch / "csrc").mkdir(exist_ok=True)
    (scratch / "csrc" / "obj.o").write_text("x" * 100, encoding="utf-8")
    (scratch / "vlog_sorted.tmp").write_text("x" * 50, encoding="utf-8")
    return run_dir, scratch


def drop_scratch(run_dir, scratch):
    """Delete scratch only, and only if it is where we think it is."""
    run_dir, scratch = Path(run_dir).resolve(), Path(scratch).resolve()
    if not scratch.exists():
        return "already gone"
    if scratch == run_dir:
        raise ValueError("refusing to delete: scratch is the run directory")
    shutil.rmtree(scratch)
    return "removed"


run_dir, scratch = make_run("axi_smoke", 1839274)
print("before:", sorted(p.name for p in run_dir.iterdir()))
print("drop  :", drop_scratch(run_dir, scratch))
print("after :", sorted(p.name for p in run_dir.iterdir()))

try:
    drop_scratch(run_dir, run_dir)
except ValueError as exc:
    print("guard :", exc)

before: ['cmd.sh', 'record.json', 'sim.log', 'tmp']
drop  : removed
after : ['cmd.sh', 'record.json', 'sim.log']
guard : refusing to delete: scratch is the run directory

One rmtree call, and the three kept files are untouched — which is the payoff for having separated them in the first place. Chapter 23.7's cleanup had to enumerate and classify the contents of one directory; this deletes a subtree without inspecting anything.

Three properties make that rmtree safe enough to run unattended, and Chapter 18.9 argued for all three:

It resolves both paths first, so the comparison is between real locations rather than between spellings — runs/x and runs/./x compare equal only after resolving.

It refuses when scratch equals the run directory. That is precisely the state Section 4's bug produces, and it is the one case where a correct-looking rmtree destroys the results. A guard that fires on the known failure mode is worth more than a general one.

It returns what it did rather than succeeding silently, so a caller can log it and a test can assert on it.

9. Predict, Run, Observe

import os
import tempfile
from pathlib import Path

run_dir = Path("runs") / "t_1"

# A
print("A:", repr(Path("")))

# B
print("B:", Path("") == Path("."))

# C
os.environ["SCRATCH"] = ""
print("C:", repr(str(Path(os.environ.get("SCRATCH", run_dir / "tmp")))))

# D
print("D:", repr(str(Path(os.environ.get("SCRATCH") or run_dir / "tmp"))))

# E
os.environ.pop("SCRATCH", None)
print("E:", repr(str(Path(os.environ.get("SCRATCH", run_dir / "tmp")))))

# F
d = Path("pro_scratch").resolve()
d.mkdir(exist_ok=True)
os.environ["TMPDIR"] = str(d)
tempfile.tempdir = None
print("F:", tempfile.gettempdir() == str(d))

# G
other = Path("pro_other").resolve()
other.mkdir(exist_ok=True)
with tempfile.TemporaryDirectory(dir=other) as td:
    print("G:", Path(td).parent == other)

# H
h = tempfile.NamedTemporaryFile(dir=other, delete=False)
n = Path(h.name)
h.close()
print("H:", n.exists())
n.unlink()

The verified output:

A: PosixPath('.')
B: True
C: '.'
D: 'runs/t_1/tmp'
E: 'runs/t_1/tmp'
F: True
G: True
H: True

A — the empty string is a valid path meaning here.

B — and it is exactly Path(".").

C — the annotation's form with SCRATCH empty: scratch is the current directory.

D — or gives the intended fallback.

E — with the variable genuinely absent, both forms agree. Which is why this is only ever found in production.

F — TMPDIR works when tempfile has not already cached something else.

G — dir= overrides regardless of any of that.

H — delete=False survives close(), and deleting it is now your job.

10. Exercises

Exercise 1 — classify. List everything one run of your flow produces and mark each keep or discard. Note which ones are large.

Exercise 2 — peak usage. Explain why cleaning up afterwards does not reduce peak disk, and what a killed run leaves behind.

Exercise 3 — two filesystems. Say why results and intermediates often want different volumes, and what writing a build tree over NFS costs.

Exercise 4 — collisions. Show that naming a run directory by test alone, or by seed alone, collides.

Exercise 5 — the empty path. Print Path("") and say why it is neither an error nor a null.

Exercise 6 — reproduce the bug. Export SCRATCH="" and show the annotation's form yielding the current directory.

Exercise 7 — both outcomes. Describe where scratch then lands with cwd=run_dir and without it, and why neither fails loudly.

Exercise 8 — one character. Fix it with or and show all three cases behaving.

Exercise 9 — why it survives review. Say why the unset case passing makes this bug hard to catch before production.

Exercise 10 — the cache. Call tempfile.gettempdir(), then set TMPDIR, then call it again. Record the result and explain it.

Exercise 11 — two correct ways. Implement both the early-TMPDIR and the dir= approaches, and say which suits your flow.

Exercise 12 — delete=False. Generate a filelist a tool must read, and say why delete=False is required and what it obliges you to do.

Exercise 13 — tell the tool. Pass scratch through TMPDIR and confirm the child received it. Find the explicit flag your simulator offers.

Exercise 14 — absolute. Say what a relative TMPDIR does once cwd= is in use.

Exercise 15 — scratch collides too. Explain why a shared SCRATCH needs the test and seed in the path.

Exercise 16 — one rmtree. Delete scratch and show the kept files untouched. Contrast with Chapter 23.7's enumerate-and-classify cleanup.

Exercise 17 — the guard. Add the scratch-equals-run-directory refusal and explain which bug it is aimed at.

Exercise 18 — retention. Write the policy for how long a failed run's scratch is kept, and say what happens without one.

11. Summary

• A run produces small things worth keeping and large things worth discarding, and the large ones are the build trees.
• Cleaning up afterwards still pays the peak disk cost, and a killed run never cleans up at all.
• Results and intermediates often belong on different filesystems — a backed-up share versus fast local disk.
• Name the run directory for the test and the seed; either alone collides.
• Take the scratch location from the environment with a safe fallback — the site knows, your script does not.
• Path("") is Path("."), a valid relative path meaning the current directory.
• So os.environ.get("SCRATCH", default) with SCRATCH exported empty yields Path(".") — Chapter 24.2's trap, with a filesystem consequence.
• Scratch then lands in the run directory under cwd=, or beside your script without it, and nothing fails.
• or instead of the default argument is the whole fix, and the unset case passing is why this is found in production.
• tempfile caches its directory on first use, so setting TMPDIR afterwards does nothing — the first use may be a library you did not call directly.
• Set TMPDIR early and once, or pass dir= at every call site; the middle option only looks correct.
• delete=False is required when a tool must read the file, and deleting it becomes your responsibility.
• Tell the tool through TMPDIR, with the vendor's own flag in Chapter 23.11's adapter where one exists.
• Make both paths absolute and create both before launch — a relative one resolves against the child once cwd= is in use.
• Put the test and seed in the scratch path too; a shared volume collides exactly as a flat run directory would.
• Deleting scratch is then one rmtree against Chapter 23.7's enumerate-and-classify, and the kept files are never inspected.
• Resolve before comparing, refuse when scratch equals the run directory, and return what was removed.

The next chapter turns the keep-or-discard column of Section 1 into a written policy with a retention horizon, applied the same way whether a run passed or failed.