Paths That Survive Being Run From Anywhere

Part II · Module 18 · Chapter 18.10 · Paths That Survive Being Run From Anywhere

1. The Problem, Restated

Chapter 18.1 established that a relative path is resolved against the process's working directory, which belongs to whoever started the process. A shell, a Makefile, a cron job and an LSF submission start in four different places, so a script with relative paths has four different behaviours.

The fix is to stop using a variable the caller controls and start using one the caller cannot touch: the location of the script file itself.

2. The Anchor

ROOT = Path(__file__).resolve().parents[1]
TESTS = ROOT / "tb" / "tests"

__file__ is the path of the source file currently being executed. Three operations in sequence:

• Path(__file__) — the script's own path, as Python was given it.
• .resolve() — made absolute and canonical, collapsing .. and symlinks (Chapter 18.1 Section 6). This is the step that removes any dependence on cwd, and it must happen before anything else uses the path.
• .parents[1] — climb to the project root. For a script in project/bin/, index 0 is bin/ and index 1 is project/.

Every other path derives from ROOT, which Chapter 18.4 Section 5 built into a paths module.

3. The Proof

The same script, run from four directories, with no absolute path printed:

module alu; endmodule

class smoke; endclass

import os
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
RTL = ROOT / "rtl"
TESTS = ROOT / "tb" / "tests"

HERE = Path.cwd()
places = [".", "rtl", "tb/tests", "bin"]

print(f"{'started from':<12} {'anchored finds rtl':<20} {'relative finds rtl'}")
for place in places:
    os.chdir(HERE / place)
    anchored = (RTL / "alu.sv").is_file()
    relative = Path("rtl/alu.sv").is_file()
    print(f"{place:<12} {str(anchored):<20} {relative}")
os.chdir(HERE)

print()
print("ROOT was computed once and never changed:", ROOT == Path(__file__).resolve().parents[1])
print("every derived path is absolute          :", RTL.is_absolute(), TESTS.is_absolute())

started from anchored finds rtl   relative finds rtl
.            True                 True
rtl          True                 False
tb/tests     True                 False
bin          True                 False

ROOT was computed once and never changed: True
every derived path is absolute          : True True

Four starting directories, one answer from the anchored path and three failures from the relative one. That is the entire deliverable of this module, and the relative version works only in the one case where the caller happened to be standing in the right place.

Note that ROOT is computed once, at import time, before anything has a chance to chdir. A script that computes it later — after changing into a run directory, say — gets a different answer, which is Chapter 18.1 Section 6's warning about resolving at the wrong moment.

4. The Off-by-One, and How to Not Have It

parents[N] is the part that goes wrong, and it goes wrong quietly:

from pathlib import Path

script = Path("/proj/chipA/bin/check.py")

for n in range(4):
    print(f"parents[{n}] = {script.parents[n]}")

parents[0] = /proj/chipA/bin
parents[1] = /proj/chipA
parents[2] = /proj
parents[3] = /

parents[0] is the directory containing the script, not the script itself — the indexing starts at the immediate parent. So a script in bin/ needs parents[1], and one in bin/flow/ needs parents[2].

Getting it wrong does not raise. parents[2] gives /proj, every derived path becomes /proj/rtl instead of /proj/chipA/rtl, and the first failure is a FileNotFoundError naming a path that looks almost right.

The fix costs one line: verify the anchor against something that must exist at the root.

from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]

MARKER = "rtl"
if not (ROOT / MARKER).is_dir():
    raise RuntimeError(
        f"project root looks wrong: {ROOT.name!r} contains no {MARKER}/ -- "
        f"check the parents[] index for a script at {Path(__file__).parent.name}/"
    )

print("anchor verified :", (ROOT / MARKER).is_dir())
print("root is absolute:", ROOT.is_absolute())
print("script's dir    :", Path(__file__).parent.name)
print("marker is under it:", (ROOT / MARKER).parent == ROOT)

module alu; endmodule

anchor verified : True
root is absolute: True
script's dir    : bin
marker is under it: True

A single is_dir() against a directory the project must have, and a misanchor becomes an immediate error naming the likely cause rather than a confusing FileNotFoundError three functions later. This is Chapter 14.7's raise-at-the-boundary rule applied to the most important value in the script.

Good markers are things a project cannot be without: rtl/, a Makefile, a .git directory, a setup.cfg. Pick one that is genuinely never absent.

5. Three Ways It Breaks

The script moves. Somebody reorganises bin/ into bin/flow/ and parents[1] is now one short. The marker check catches it on the first run; without the check it produces wrong paths. This is the common case, and it is why the marker is worth the line.

bin/ is a symlink. resolve() follows symlinks, so a script invoked through a symlinked bin/ anchors to the real location rather than the apparent one. That is usually right — the project is where the files really are — and occasionally not, if the symlink was deliberately pointing a shared tool at a per-user tree. Where the apparent path is wanted, Path(__file__).absolute() makes it absolute without following links.

There is no __file__. In an interactive session, in python -c, and in some frozen or packaged forms, __file__ is absent or meaningless. For a flow script this rarely matters — it is run as a file — but a module imported into a notebook will raise NameError. Where that is a real risk, fall back to an explicit configuration value rather than guessing.

6. The Three Candidate Anchors

__file__ is not the only option, and the alternatives are legitimate choices with different trade-offs rather than mistakes:

import os
from pathlib import Path


def project_root(explicit=None):
    """Prefer an explicit value, then the environment, then the script's location."""
    if explicit:
        return Path(explicit).resolve(), "explicit argument"
    env = os.environ.get("PROJECT_ROOT")
    if env:
        return Path(env).resolve(), "PROJECT_ROOT"
    return Path(__file__).resolve().parents[1], "script location"


root, how = project_root()
print("resolved via :", how)
print("is absolute  :", root.is_absolute())

root2, how2 = project_root("/proj/other")
print("resolved via :", how2, "->", root2)

resolved via : script location
is absolute  : True
resolved via : explicit argument -> /proj/other

The ordering is the useful part: an explicit argument wins, then the environment, then the script's own location as a default that always works. A caller who needs to point the tool elsewhere can, and a caller who does nothing gets correct behaviour.

Note that the environment variable is a hidden input in Chapter 1.7's sense — it does not appear in the command line, differs per user, and is invisible in a pasted bug report. Returning how the root was resolved, as this function does, is what makes that visible when something goes wrong.

7. The Assembled Shape

Everything in this module, in the form it actually takes at the top of a flow script:

from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]

if not (ROOT / "rtl").is_dir():
    raise RuntimeError(f"project root looks wrong: {ROOT.name!r} has no rtl/")

RTL = ROOT / "rtl"
TB = ROOT / "tb"
TESTS = TB / "tests"
RUNS = ROOT / "runs"

SKIP = {"generated", "third_party", ".git"}


def sources():
    """Discovered, filtered and sorted -- 18.5, 18.6 and the ordering rule."""
    return sorted(p for p in RTL.rglob("*.sv") if not (SKIP & set(p.parts)))


def run_dir(test, seed):
    """A name that cannot collide -- 18.8."""
    return RUNS / f"{test}_{seed}"

module alu; endmodule

module regs; endmodule

import os
from paths import ROOT, RTL, RUNS, sources, run_dir

HERE = os.getcwd()
for place in [".", "rtl"]:
    os.chdir(ROOT / place)
    found = sources()
    print(f"from {place:<5} -> {len(found)} sources: {[str(p.relative_to(ROOT)) for p in found]}")
os.chdir(HERE)

d = run_dir("smoke", 1234)
(d / "logs").mkdir(parents=True, exist_ok=True)
print("run dir      :", d.relative_to(ROOT))
print("inside RUNS  :", d.is_relative_to(RUNS))
print("logs created :", (d / "logs").is_dir())

from .     -> 1 sources: ['rtl/alu.sv']
from rtl   -> 1 sources: ['rtl/alu.sv']
run dir      : runs/smoke_1234
inside RUNS  : True
logs created : True

Twenty lines carrying nine chapters: an anchor that does not depend on the caller, a verification of it, derived paths, filtered and sorted discovery that gives the same answer from two directories, a run directory named so it cannot collide, and a containment property the cleanup step will check before removing anything.

The generated source is excluded, the discovery is identical from both working directories, and nothing in flow.py mentions a path at all.

8. Predict, Run, Observe

from pathlib import Path

script = Path("/proj/chipA/bin/flow/check.py")

print("A:", script.parents[0].name)
print("B:", script.parents[1].name)
print("C:", script.parents[2].name)
print("D:", len(script.parents))
print("E:", script.parent == script.parents[0])
print("F:", script.parents[-1])

A: flow
B: bin
C: chipA
D: 5
E: True
F: /

A, B and C — a script two directories deep needs parents[2], not parents[1]. The index is the number of levels to climb, and counting it wrong is Section 4's silent failure.

E — parent and parents[0] are the same thing, which is the clearest way to remember that the sequence starts at the immediate parent rather than at the path itself.

D is 5, not 4 — the chain is flow, bin, chipA, proj, /, so a four-component path has five ancestors once the filesystem root is counted. Reading a parents length as "how deep am I" is off by one in the same direction as the indexing.

F is the filesystem root, and parents always ends there. An index larger than the depth raises IndexError rather than returning the root repeatedly, which is at least a loud failure — but only when the path is shallower than you assumed, which is the opposite of the common mistake.

9. Exercises

Exercise 1 — anchor one. Put a script in bin/ and compute ROOT. Run it from three directories and confirm the derived paths are identical.

Exercise 2 — break it with cwd. Add a relative path beside the anchored one and show which fails from a subdirectory.

Exercise 3 — count the levels. For a script in bin/flow/, determine the correct parents index by printing them all.

Exercise 4 — the marker. Add a verification against a directory your project must have. Deliberately use the wrong index and confirm the error is clear.

Exercise 5 — move the script. Move an anchored script one directory deeper without changing it. Record what happens with and without the marker check.

Exercise 6 — the three anchors. Implement the explicit/environment/__file__ fallback. Show which one is used in three scenarios.

Exercise 7 — the hidden input. Set PROJECT_ROOT to a wrong value and run a script that prefers it. Say how a colleague would diagnose it from a pasted command.

Exercise 8 — assemble. Write the paths.py for your own project with an anchor, a marker check, four derived paths and one discovery function.

10. Summary

• Stop using a variable the caller controls and anchor to the script's own location.
• Path(__file__).resolve().parents[N] — resolve first, because that is what removes the dependence on cwd.
• Compute the root once, at import time, before anything can change directory.
• parents[0] is the containing directory, so a script in bin/ needs parents[1] and one in bin/flow/ needs parents[2].
• A wrong index does not raise — it produces paths that look almost right.
• Verify the anchor against a marker that the project cannot be without; one is_dir() converts a silent misanchor into a clear error.
• Three things break it: the script moving, a symlinked directory that resolve() follows, and contexts where __file__ does not exist.
• The three candidate anchors are cwd, an environment variable and __file__ — the last is the right default, and an explicit argument should win over both.
• An environment-variable root is a hidden input, so report which anchor was used.
• Print the resolved root once — it is the exception to this module's rule about absolute paths, and it answers the first question anybody asks.

That closes Module 18. A script can now find the project from anywhere, discover and filter its sources, emit a filelist a tool will accept, create the directory a run writes into, and remove only what it owns — which is every path question a flow script asks before it does any real work.