Creating the Run-Directory Tree

Part II · Module 18 · Chapter 18.8 · Creating the Run-Directory Tree

1. The Shape to Build

A run needs a directory of its own, with somewhere for logs and somewhere for waves:

2. parents=True, and the Error Without It

from pathlib import Path

run = Path("runs/smoke_1234/logs")

try:
    run.mkdir()
except FileNotFoundError as e:
    print("asked to create :", run)
    print("error           :", type(e).__name__, "-", e.strerror)
    print("runs/ exists    :", Path("runs").exists())
    print("its parent too  :", run.parent.exists())

asked to create : runs/smoke_1234/logs
error           : FileNotFoundError - No such file or directory
runs/ exists    : False
its parent too  : False

The error names the path you asked for, which is not the one that is missing — the last two lines show that runs/ and runs/smoke_1234/ are both absent too. A reader checks whether logs exists, finds it does not, and is briefly confused about why that is an error when the whole point was to create it.

mkdir creates one directory and expects its parent to be there. parents=True creates the whole chain:

from pathlib import Path

run = Path("runs/smoke_1234/logs")
run.mkdir(parents=True)

print("created      :", run.is_dir())
print("and its parents:", Path("runs").is_dir(), Path("runs/smoke_1234").is_dir())

created      : True
and its parents: True True

3. exist_ok=True, and What It Actually Promises

Run the same script twice and the second run fails:

from pathlib import Path

run = Path("runs/smoke_1234")
run.mkdir(parents=True)
print("first time :", run.is_dir())

try:
    run.mkdir(parents=True)
except FileExistsError as e:
    print("second time:", type(e).__name__, "-", e.strerror)

first time : True
second time: FileExistsError - File exists

The canonical form adds exist_ok=True, which makes the call idempotent:

from pathlib import Path

run = Path("runs/smoke_1234")

for attempt in [1, 2, 3]:
    (run / "logs").mkdir(parents=True, exist_ok=True)
    (run / "waves").mkdir(parents=True, exist_ok=True)
    print(f"attempt {attempt}: no error, tree exists:", (run / "logs").is_dir())

print("tree:", sorted(str(p) for p in Path("runs").rglob("*")))

attempt 1: no error, tree exists: True
attempt 2: no error, tree exists: True
attempt 3: no error, tree exists: True
tree: ['runs/smoke_1234', 'runs/smoke_1234/logs', 'runs/smoke_1234/waves']

Three calls, one tree, no errors. That is what exist_ok=True promises, and it is worth being precise about what it does not.

4. It Does Not Clear Anything

The canonical purpose names two failure modes — a re-run that fails, and one that quietly overwrites last night's evidence. exist_ok=True fixes the first and has nothing to say about the second:

from pathlib import Path

run = Path("runs/smoke_1234")
(run / "logs").mkdir(parents=True, exist_ok=True)

(run / "logs" / "sim.log").write_text("LAST NIGHT: the run that found the bug\n")
print("before re-run:", (run / "logs" / "sim.log").read_text().strip())

(run / "logs").mkdir(parents=True, exist_ok=True)

print("after mkdir  :", (run / "logs" / "sim.log").read_text().strip())
print("file survived:", (run / "logs" / "sim.log").exists())

before re-run: LAST NIGHT: the run that found the bug
after mkdir  : LAST NIGHT: the run that found the bug
file survived: True

The directory was not touched. exist_ok=True means "do nothing if it is already there" — and nothing includes not emptying it.

So the second run starts with last night's log still in place. What happens next depends on the simulator: it may truncate sim.log and leave compile.log from last night beside it, or append, or write a file with a different name. In every case the directory now holds a mixture of two runs and no record of which artefact came from which.

That is worse than either failing or overwriting cleanly, because the result looks complete. A triage script reading runs/smoke_1234/logs/ gets a plausible set of files describing two different executions.

5. Three Policies, and You Must Choose One

An existing run directory has exactly three sensible responses, and exist_ok=True silently picks the first:

import shutil
from pathlib import Path

RUNS = Path("runs")


def prepare(run_dir, policy):
    """Return a message describing what happened."""
    if not run_dir.exists():
        run_dir.mkdir(parents=True)
        return "created fresh"
    if policy == "reuse":
        return "reused, previous contents still present"
    if policy == "refuse":
        return "REFUSED: directory already exists"
    if policy == "rotate":
        old = run_dir.with_name(run_dir.name + ".prev")
        if old.exists():
            shutil.rmtree(old)
        run_dir.rename(old)
        run_dir.mkdir(parents=True)
        return f"rotated previous to {old.name}, created fresh"
    raise ValueError(policy)


for policy in ["reuse", "refuse", "rotate"]:
    run = RUNS / f"smoke_{policy}"
    run.mkdir(parents=True)
    (run / "sim.log").write_text("last night\n")
    print(f"{policy:<8} -> {prepare(run, policy)}")

print()
print("after rotate, previous kept:", (RUNS / "smoke_rotate.prev" / "sim.log").read_text().strip())
print("after rotate, new is empty :", list((RUNS / "smoke_rotate").iterdir()) == [])

reuse    -> reused, previous contents still present
refuse   -> REFUSED: directory already exists
rotate   -> rotated previous to smoke_rotate.prev, created fresh

after rotate, previous kept: last night
after rotate, new is empty : True

Reuse is exist_ok=True. Correct when the run genuinely continues or when nothing persists between runs; wrong when artefacts accumulate.

Refuse is the safest and the most annoying. Right for a directory whose contents are signoff evidence, where silently mixing two runs is unacceptable and a human should decide.

Rotate keeps one generation and starts clean. The usual best answer for a nightly, and note the rename-aside pattern from Chapter 15.9 — with the guard, because rename would otherwise destroy an existing .prev.

This is Chapter 1.8's procedure applied to an output directory: enumerate the states, decide the response, and write it down. exist_ok=True is a decision, and the problem is only that it does not look like one.

6. The Better Fix Is a Name That Cannot Collide

Everything above handles a collision. The stronger move is to make collisions impossible, and it costs nothing:

from pathlib import Path

RUNS = Path("runs")


def run_dir(test, seed):
    return RUNS / f"{test}_{seed}"


for test, seed in [("smoke", 1234), ("smoke", 5678), ("cdc", 1234)]:
    d = run_dir(test, seed)
    d.mkdir(parents=True, exist_ok=True)
    print(f"{test:<6} seed {seed:<5} -> {d}")

print()
print("directories created:", len(sorted(RUNS.iterdir())))
print("same test, different seed collides:", run_dir("smoke", 1234) == run_dir("smoke", 5678))

smoke  seed 1234  -> runs/smoke_1234
smoke  seed 5678  -> runs/smoke_5678
cdc    seed 1234  -> runs/cdc_1234

directories created: 3
same test, different seed collides: False

Three runs, three directories, no policy needed — because the identity of the run is in its name. Two genuinely different runs never meet.

The only collision left is a re-run of the identical test and seed, which is a deliberate act and is exactly the case where a human should choose between reuse, refuse and rotate.

7. Predict, Run, Observe

from pathlib import Path

run = Path("runs/smoke_1234")
(run / "logs").mkdir(parents=True, exist_ok=True)
(run / "logs" / "sim.log").write_text("first\n")

print("A:", (run / "logs" / "sim.log").exists())

run.mkdir(parents=True, exist_ok=True)
print("B:", (run / "logs" / "sim.log").read_text().strip())

(run / "logs" / "sim.log").write_text("second\n")
print("C:", (run / "logs" / "sim.log").read_text().strip())

print("D:", sorted(p.name for p in run.iterdir()))
print("E:", (run / "waves").exists())
print("F:", run.parent.name)

A: True
B: first
C: second
D: ['logs']
E: False
F: runs

B — mkdir(exist_ok=True) on the run directory left everything beneath it untouched, which is Section 4 in one line.

C — the file write overwrote, because write_text truncates (Chapter 15.9). So the overwriting a re-run does comes from the writes, not from the directory creation, and the two are easy to conflate.

D and E together show a partially built tree: logs exists because it was created, waves does not because nothing asked for it. A run script that creates logs on one path and waves on another produces different trees depending on which options were set — worth creating both unconditionally so the shape is predictable.

F confirms the run directory sits directly under runs/, which is what makes Chapter 18.5's one-level pattern find it.

8. Exercises

Exercise 1 — the nested error. Create a three-level path without parents=True. Read the error and say which directory is actually missing.

Exercise 2 — idempotent. Call mkdir(parents=True, exist_ok=True) three times in a loop. Confirm no error and one tree.

Exercise 3 — it does not clear. Write a file into a run directory, re-run the mkdir, and confirm the file survived.

Exercise 4 — three policies. Implement reuse, refuse and rotate. Say which your flow should use and why.

Exercise 5 — rotate twice. Run the rotation three times and confirm only one previous generation is kept. Say what you would change to keep three.

Exercise 6 — names that cannot collide. List the fields that identify a run in your flow. Build a directory name from them and check two different runs differ.

Exercise 7 — the mixed directory. Deliberately produce a run directory containing artefacts from two runs. Describe how a triage script would be misled.

Exercise 8 — predictable shape. Make a run script that creates waves/ only when dumping is enabled, then make it unconditional. Say which you would ship.

9. Summary

• mkdir creates one directory and expects its parent to exist; the error names the path you asked for, not the one that is missing.
• parents=True creates the whole chain.
• exist_ok=True makes the call idempotent, so a re-run does not fail.
• It does not clear the directory — last night's artefacts are still there afterwards.
• A reused directory holds a mixture of two runs, which is worse than either failing or overwriting cleanly because it looks complete.
• Three policies exist: reuse, refuse, rotate — and exist_ok=True silently chooses reuse.
• Rotate keeps one generation using the rename-aside pattern, with a guard because rename would destroy an existing target.
• The better fix is a name that cannot collide: put the run's identity in its directory name.
• The only remaining collision is a deliberate re-run of the same test and seed, which is exactly where a human should choose.
• Create the whole tree unconditionally so its shape does not depend on which options were set.

The next chapter removes these directories, which is the operation where getting the path wrong is not recoverable.