One Run Directory per Job

Part II · Module 26 · Chapter 26.4 · One Run Directory per Job

1. What Sharing Costs

CONSEQUENCES = [
    ("two jobs append to one log", "both sets of lines survive, interleaved"),
    ("two jobs truncate one log", "one job's log is gone entirely"),
    ("two jobs write one waveform", "the file is a mixture, or corrupt"),
    ("two jobs share a build directory", "one compile's objects feed the other's link"),
    ("a retry reuses the first attempt's directory", "the failing evidence is overwritten"),
]

print(f"{'what happens':<44} {'consequence'}")
for what, consequence in CONSEQUENCES:
    print(f"{what:<44} {consequence}")

print()
print("how many produce an error message:", 0)
print("how many change a reported status :", "some — and unpredictably")

what happens                                 consequence
two jobs append to one log                   both sets of lines survive, interleaved
two jobs truncate one log                    one job's log is gone entirely
two jobs write one waveform                  the file is a mixture, or corrupt
two jobs share a build directory             one compile's objects feed the other's link
a retry reuses the first attempt's directory the failing evidence is overwritten

how many produce an error message: 0
how many change a reported status : some — and unpredictably

Five ways, no error messages, and the last column is what makes this chapter Critical rather than merely important. A collision does not make a regression fail — it makes a regression lie, and the lie is different each time because it depends on which job happened to be writing when.

2. Interleaving, Measured

import sys

tag, count = sys.argv[1], int(sys.argv[2])
with open("shared/sim.log", "a", encoding="utf-8") as handle:
    for i in range(count):
        handle.write(f"{tag} line {i}\n")
        handle.flush()

import subprocess
import sys
from pathlib import Path

Path("shared").mkdir(exist_ok=True)

procs = [subprocess.Popen([sys.executable, "writer.py", tag, "200"])
         for tag in ("A", "B")]
for p in procs:
    p.wait()

lines = Path("shared/sim.log").read_text(encoding="utf-8").splitlines()
blocks = 1 + sum(1 for a, b in zip(lines, lines[1:]) if a[0] != b[0])

print("total lines     :", len(lines))
print("from job A      :", sum(1 for l in lines if l.startswith("A")))
print("from job B      :", sum(1 for l in lines if l.startswith("B")))
print("nothing was lost:", len(lines) == 400)
print()
print("blocks is at least 2    :", blocks >= 2)
print("and how many it is      :", "depends on how the OS scheduled them")

total lines     : 400
from job A      : 200
from job B      : 200
nothing was lost: True

blocks is at least 2    : True
and how many it is      : depends on how the OS scheduled them

Nothing was lost. Both jobs' lines are all present — append mode with a flush per line is well behaved about that.

What is not published here is how many interleaved blocks resulted, and the reason is the point of the section. On the machine that verified this chapter the two processes happened not to overlap much; on a loaded machine, with longer logs, or with a simulator that writes for twenty minutes, they overlap thoroughly. The degree of interleaving is a property of the scheduler, not of your code — which means it varies between runs of the identical regression, and a bug that depends on it is the least reproducible kind there is.

So the right way to see the consequence is on a log that is definitely interleaved:

INTERLEAVED = [
    "A UVM_INFO @ 0: Running test axi_burst...",
    "B UVM_INFO @ 0: Running test axi_smoke...",
    "B UVM_ERROR mon.sv(31) @ 120500: bad response",
    "A UVM_INFO drv.sv(60) @ 141700: issuing write",
    "A UVM_ERROR drv.sv(88) @ 141900: response mismatch",
    "B TEST DONE",
]

errors = [l for l in INTERLEAVED if " UVM_ERROR" in l]
first = errors[0]
finished = any("TEST DONE" in l for l in INTERLEAVED)

print("what a Module 20 parser sees in this one file:")
print("   error lines  :", len(errors))
print("   first failure:", first)
print("   finished     :", finished)
print()
print("but job A's first failure is  :", [l for l in errors if l.startswith("A")][0])
print("and job A finished            :", any(l.startswith("A") and "TEST DONE" in l
                                             for l in INTERLEAVED))

what a Module 20 parser sees in this one file:
   error lines  : 2
   first failure: B UVM_ERROR mon.sv(31) @ 120500: bad response
   finished     : True

but job A's first failure is  : A UVM_ERROR drv.sv(88) @ 141900: response mismatch
and job A finished            : False

Every one of those three readings is wrong for job A. Its error count is one, not two; its first failure is the driver mismatch, not the monitor one; and it never finished at all — the TEST DONE belongs to job B.

That invalidates most of Module 20 at once. Chapter 20.10's first-failure logic takes the earliest error line, Chapter 20.15's stateful parsing accumulates across lines, and Chapter 25.6's end-banner check asks whether the log finished. Every one of them reads a sequence and assumes the sequence belongs to one run.

Worse, job A's status comes out as PASS: there is a TEST DONE, so Chapter 25.6's classifier sees a finished run, and if the two errors were both B's it would see no errors either. A hung job reported as a pass, caused by a directory name.

Truncating writers are simpler and worse:

import sys

with open("shared2/sim.log", "w", encoding="utf-8") as handle:
    handle.write(f"{sys.argv[1]} finished\n")

import subprocess
import sys
from pathlib import Path

Path("shared2").mkdir(exist_ok=True)
for tag in ("A", "B"):
    subprocess.run([sys.executable, "writer2.py", tag])

content = Path("shared2/sim.log").read_text(encoding="utf-8").strip()
print("the log contains:", repr(content))
print("jobs that wrote :", 2)
print("jobs with a log :", 1)
print("job A's evidence:", "gone")

the log contains: 'B finished'
jobs that wrote : 2
jobs with a log : 1
job A's evidence: gone

A tool opening its log with -l sim.log truncates, which is the normal case. Two jobs, one surviving log, and the job whose log vanished still produced a return code — so it gets a status, from evidence belonging to the other job.

3. The Retry That Erases Its Own Reason

from pathlib import Path

run_dir = Path("runs/axi_burst_1839274")
run_dir.mkdir(parents=True, exist_ok=True)
log = run_dir / "sim.log"

log.write_text("attempt 1: UVM_ERROR drv.sv(88) @ 141900: response mismatch\n",
               encoding="utf-8")
print("after attempt 1:", repr(log.read_text(encoding="utf-8").strip()))

log.write_text("attempt 2: TEST DONE\n", encoding="utf-8")
print("after attempt 2:", repr(log.read_text(encoding="utf-8").strip()))

print()
print("the failing evidence survives:", "UVM_ERROR" in log.read_text(encoding="utf-8"))
print("the reported status          :", "PASS")
print("what a reviewer can inspect  :", "a clean log from a job that failed once")

after attempt 1: 'attempt 1: UVM_ERROR drv.sv(88) @ 141900: response mismatch'
after attempt 2: 'attempt 2: TEST DONE'

the failing evidence survives: False
the reported status          : PASS
what a reviewer can inspect  : a clean log from a job that failed once

This is the one to be angry about. Chapter 26.5 will introduce retries, and a retry that reuses the directory destroys the only record of why the retry was needed.

The result is a job marked PASS with a clean log, and nothing anywhere showing that it failed first. Chapter 25.7 Section 6 argued that a retry wrapper hides the next real bug; this is the mechanism by which it does so — not a policy choice, a filesystem one. The failure is not merely unreported, it is unrecoverable: no amount of later analysis can reconstruct a log that was overwritten.

4. The Annotation

run_dir = ROOT / "runs" / f"{test}_{seed}"

from pathlib import Path

ROOT = Path("/proj/regr/nightly_2024_09_14")

for test, seed in (("axi_smoke", 1839274), ("axi_smoke", 1839275), ("axi_burst", 1839274)):
    run_dir = ROOT / "runs" / f"{test}_{seed}"
    print(str(run_dir).replace(str(ROOT), "<root>"))

names = {f"{t}_{s}" for t, s in (("axi_smoke", 1839274), ("axi_smoke", 1839275),
                                 ("axi_burst", 1839274))}
print()
print("distinct names:", len(names), "for 3 jobs")
print("collisions    :", 3 - len(names))

<root>/runs/axi_smoke_1839274
<root>/runs/axi_smoke_1839275
<root>/runs/axi_burst_1839274

distinct names: 3 for 3 jobs
collisions    : 0

Three jobs, three directories, no collisions — the annotation is correct for the jobs Chapter 26.2 produces, which vary only in test and seed.

ROOT is displayed as <root> because the real prefix differs per machine. What matters is that it exists: every regression gets its own root, so last night's axi_smoke_1839274 and tonight's do not collide either. A date or a run identifier in the root is what buys that.

5. Two Components Is One Short

JOBS = [("axi_burst", 1839274, "vcs", 1),
        ("axi_burst", 1839274, "vcs", 2),
        ("axi_burst", 1839274, "xrun", 1),
        ("axi_burst", 1839275, "vcs", 1)]

SCHEMES = [("test_seed", (0, 1)),
           ("test_seed_sim", (0, 1, 2)),
           ("test_seed_sim_attempt", (0, 1, 2, 3))]

for label, keys in SCHEMES:
    names = {"_".join(str(job[k]) for k in keys) for job in JOBS}
    print(f"{label:<24} {len(names)} name(s) for {len(JOBS)} jobs   collisions={len(JOBS) - len(names)}")

print()
print("the four jobs differ in:", "seed, simulator, and attempt number")
print("a name must carry every dimension the job list varies in")

test_seed                2 name(s) for 4 jobs   collisions=2
test_seed_sim            3 name(s) for 4 jobs   collisions=1
test_seed_sim_attempt    4 name(s) for 4 jobs   collisions=0

the four jobs differ in: seed, simulator, and attempt number
a name must carry every dimension the job list varies in

Two components give two names for four jobs. The annotation's f"{test}_{seed}" was sufficient in Module 25 because a runner ran one test, once, on one simulator. A regression changes both of those:

• Chapter 26.5 adds retries, so the same (test, seed) runs more than once;
• a cross-simulator regression adds a simulator axis, so the same (test, seed) runs under VCS and Xcelium.

Each is a dimension, and a dimension missing from the name is a collision. The rule is mechanical:

The directory name must contain every field that distinguishes two jobs. If two jobs in your list differ in a field, that field is in the name.

Which, stated the other way round, is the useful form: the directory name is the job's identity, and anything the name does not capture is something the filesystem cannot keep apart.

6. One Directory per Job

Read it left to right and the guarantee is structural. Two jobs with different identities get different names; two jobs with the same identity are the same job. A collision stops being unlikely and becomes impossible, which is a much better property than being careful.

7. Refusing a Collision

from pathlib import Path


class CollisionError(Exception):
    """Two jobs resolved to the same run directory."""


def job_dir(root, test, seed, sim, attempt):
    name = f"{test}_{seed}_{sim}" + (f"_a{attempt}" if attempt > 1 else "")
    return Path(root) / "runs" / name


def allocate(root, jobs):
    """Create one directory per job, refusing to reuse a non-empty one."""
    seen = {}
    for job in jobs:
        d = job_dir(root, *job)
        if d in seen:
            raise CollisionError(f"{job} and {seen[d]} both map to {d.name}")
        seen[d] = job
        if d.exists() and any(d.iterdir()):
            raise CollisionError(f"{d.name} already exists and is not empty")
        d.mkdir(parents=True, exist_ok=True)
    return seen


JOBS = [("axi_burst", 1839274, "vcs", 1), ("axi_burst", 1839274, "vcs", 2),
        ("axi_burst", 1839274, "xrun", 1), ("axi_burst", 1839275, "vcs", 1)]

allocated = allocate("root1", JOBS)
print("jobs      :", len(JOBS))
print("directories:", len(allocated))
for d in sorted(allocated, key=lambda p: p.name):
    print("   " + d.name)

print()
try:
    allocate("root1", [("axi_burst", 1839274, "vcs", 1), ("axi_burst", 1839274, "vcs", 1)])
except CollisionError as exc:
    print("duplicate job :", exc)

Path("root2/runs/axi_burst_1839274_vcs").mkdir(parents=True, exist_ok=True)
Path("root2/runs/axi_burst_1839274_vcs/sim.log").write_text("old\n", encoding="utf-8")
try:
    allocate("root2", [("axi_burst", 1839274, "vcs", 1)])
except CollisionError as exc:
    print("stale directory:", exc)

jobs      : 4
directories: 4
   axi_burst_1839274_vcs
   axi_burst_1839274_vcs_a2
   axi_burst_1839274_xrun
   axi_burst_1839275_vcs

duplicate job : ('axi_burst', 1839274, 'vcs', 1) and ('axi_burst', 1839274, 'vcs', 1) both map to axi_burst_1839274_vcs
stale directory: axi_burst_1839274_vcs already exists and is not empty

Four decisions, and the second is the one that catches a real class of bug.

Attempt 1 has no suffix. axi_burst_1839274_vcs rather than ..._a1, so the common case reads cleanly and a retry is visibly exceptional. The cost is that _a2 sorts after the unsuffixed name, which is what you want.

Two jobs mapping to one directory raises before anything runs. Chapter 26.2 Section 6's duplicate would otherwise reach the filesystem, and a duplicate detected at allocation names both jobs — a duplicate detected by its symptoms names neither.

A non-empty existing directory raises. This is the stale-artefact case Chapter 25.7 Section 4 found by soaking: a directory left over from a previous run contributes a log the current job did not write. Refusing is better than deleting, because a directory you did not expect is evidence of something and deleting it destroys the evidence.

Allocation happens up front, for every job. A collision found at job 273 of 400 has already wasted the night; found before anything runs, it costs nothing. This is the same argument Chapter 24.5 made for a pre-flight check.

8. Nothing Escapes the Directory

from pathlib import Path


def writes_outside(run_dir, paths):
    """Which of these would land outside the job's directory?"""
    run_dir = Path(run_dir).resolve()
    out = []
    for p in paths:
        full = (run_dir / p).resolve() if not Path(p).is_absolute() else Path(p).resolve()
        if not full.is_relative_to(run_dir):
            out.append(p)
    return out


Path("runs/axi_burst_1839274").mkdir(parents=True, exist_ok=True)

CANDIDATES = ["sim.log", "cmd.sh", "csrc/obj.o", "../latest.log",
              "/tmp/regression_summary.json", "waves/axi.fsdb"]

bad = writes_outside("runs/axi_burst_1839274", CANDIDATES)
for p in CANDIDATES:
    print(f"   {p:<34} {'OUTSIDE' if p in bad else 'inside'}")

print()
print("escapes:", bad)

   sim.log                            inside
   cmd.sh                             inside
   csrc/obj.o                         inside
   ../latest.log                      OUTSIDE
   /tmp/regression_summary.json       OUTSIDE
   waves/axi.fsdb                     inside

escapes: ['../latest.log', '/tmp/regression_summary.json']

A unique directory name guarantees nothing if the job writes somewhere else, and the two escapes above are both things people add deliberately.

../latest.log is a convenience symlink or copy — "so I can always tail the most recent one" — and it is a single shared path that every job in the regression overwrites. It works perfectly with one job.

/tmp/regression_summary.json is a job writing into the summary. Chapter 25.1 Section 8 prohibited it: a runner that reports is a runner two of which cannot run at once. Aggregation is Chapter 26.6's job and it reads results, it does not receive them.

Path.is_relative_to is the check, and it runs on resolved paths — Chapter 23.7 Section 3's reason: runs/x/../../etc is not inside runs/x however it is spelled, and only resolving reveals that.

9. Predict, Run, Observe

from pathlib import Path

JOBS = [("a", 1, "vcs", 1), ("a", 1, "vcs", 2), ("a", 1, "xrun", 1), ("a", 2, "vcs", 1)]


def name(test, seed, sim, attempt):
    return f"{test}_{seed}_{sim}" + (f"_a{attempt}" if attempt > 1 else "")


# A
print("A:", len({f"{j[0]}_{j[1]}" for j in JOBS}))

# B
print("B:", len({name(*j) for j in JOBS}))

# C
print("C:", name("a", 1, "vcs", 1), name("a", 1, "vcs", 2))

# D
print("D:", sorted({name(*j) for j in JOBS})[:2])

# E
root = Path("pro/runs/a_1_vcs").resolve()
print("E:", (root / "sim.log").resolve().is_relative_to(root))

# F
print("F:", (root / ".." / "latest.log").resolve().is_relative_to(root))

# G
Path("pro/runs/a_1_vcs").mkdir(parents=True, exist_ok=True)
print("G:", Path("pro/runs/a_1_vcs").exists(), any(Path("pro/runs/a_1_vcs").iterdir()))

# H
Path("pro/runs/a_1_vcs/sim.log").write_text("x", encoding="utf-8")
print("H:", any(Path("pro/runs/a_1_vcs").iterdir()))

The verified output:

A: 2
B: 4
C: a_1_vcs a_1_vcs_a2
D: ['a_1_vcs', 'a_1_vcs_a2']
E: True
F: False
G: True False
H: True

A — two components, two names for four jobs.

B — all four components, four names.

C — attempt 1 is unsuffixed and a retry is visibly different.

D — the retry sorts immediately after its original, which is what you want when reading a directory listing.

E — an ordinary artefact is inside.

F — .. escapes, and only resolving reveals it.

G — a freshly created directory exists and is empty, which is the state allocate requires.

H — one file makes it non-empty, which is what the guard refuses.

10. Exercises

Exercise 1 — five ways. List the ways two jobs could collide in your own flow, and say which produce an error.

Exercise 2 — lie, not fail. Explain in two sentences why a collision is worse than a crash.

Exercise 3 — interleave it. Run two appenders into one file and report the line counts and whether either is contiguous.

Exercise 4 — why a boolean. Say why the number of interleaved blocks must not be published as a measurement.

Exercise 5 — what breaks. Name three things from Module 20 that an interleaved log invalidates.

Exercise 6 — truncation. Show two truncating writers and say how many logs survive.

Exercise 7 — a status from somebody else's log. Describe how the job whose log vanished still gets classified.

Exercise 8 — the retry. Overwrite a failing log with a passing one and say what a reviewer can now establish.

Exercise 9 — unrecoverable. Say why this is worse than merely unreported.

Exercise 10 — count the collisions. Build a job list varying in four fields and count distinct names under each scheme.

Exercise 11 — the rule. State it in one sentence, both ways round.

Exercise 12 — your dimensions. List every field two jobs in your regression can differ in. Check your directory name carries all of them.

Exercise 13 — per-regression root. Say what a shared root costs when last night's run is still on disk.

Exercise 14 — attempt 1 unsuffixed. Argue for it, and say what it does to sort order.

Exercise 15 — refuse, do not delete. Explain why a non-empty directory should raise rather than be cleaned.

Exercise 16 — up front. Say what allocating at job 273 costs that allocating at job 0 does not.

Exercise 17 — the tolerant mkdir. Show exist_ok=True accepting a stale directory, then add the guard.

Exercise 18 — escapes. Run writes_outside over the paths your flow actually writes. Report any escapes.

Exercise 19 — latest.log. Describe why the convenience symlink works perfectly until it does not.

Exercise 20 — resolve first. Give a path that looks inside and is not.

11. Summary

• Five ways two jobs collide, none of which produces an error message, and a collision makes a regression lie rather than fail.
• Two concurrent appenders lose nothing and interleave everything, so neither job's log is contiguous.
• Publish that as a boolean: the block pattern depends on OS scheduling and is not a property of the code.
• An interleaved log invalidates first-failure logic, stateful parsing and the end-banner check.
• Two truncating writers leave one log, and the job whose log vanished still gets a status — from the other job's evidence.
• A retry into the same directory destroys the record of why the retry happened, leaving a PASS with a clean log and no trace of the failure.
• That is unrecoverable, and it is the filesystem mechanism behind Chapter 25.7's retry-hides-the-bug argument.
• f"{test}_{seed}" is correct for Module 25 and one component short here — four jobs collapsed to two names.
• The directory name must contain every field two jobs can differ in, which means the name is the job's identity.
• Give every regression its own root, or last night's directories collide with tonight's.
• Leave attempt 1 unsuffixed so a retry is visibly exceptional and sorts immediately after.
• Raise on two jobs mapping to one directory, before anything runs — the error can then name both jobs.
• Raise on a non-empty existing directory rather than cleaning it; an unexpected directory is evidence.
• mkdir(exist_ok=True) alone silently accepts a stale run; the emptiness check is what makes it a guard.
• Allocate every directory up front, because a collision found at job 273 has already cost the night.
• A unique name guarantees nothing if the job writes elsewhere — check with is_relative_to on resolved paths.
• latest.log and a job writing into the summary are the two deliberate escapes, and both work perfectly with one job.

The next chapter bounds how long each of those jobs may take, and decides which of them may be run again.