Absolute Paths, Relative Paths and the Current Directory

Part II · Module 18 · Chapter 18.1 · Absolute Paths, Relative Paths and the Current Directory

1. Two Kinds of Path

from pathlib import Path

relative = Path("sim.log")
absolute = Path("/proj/runs/seed_41/sim.log")

print("relative is_absolute:", relative.is_absolute())
print("absolute is_absolute:", absolute.is_absolute())
print("absolute parts      :", absolute.parts)
print("relative parts      :", relative.parts)

relative is_absolute: False
absolute is_absolute: True
absolute parts      : ('/', 'proj', 'runs', 'seed_41', 'sim.log')
relative parts      : ('sim.log',)

An absolute path starts at the root — the leading / is the first element of parts — and names exactly one place on the machine. It means the same thing to every process.

A relative path does not, and the question this chapter answers is what it is relative to.

2. The Answer

from pathlib import Path
print(Path.cwd())
Path("sim.log")            # relative to cwd, whatever that is
Path("/proj/runs/sim.log") # absolute

A relative path is resolved against the process's current working directory: the directory the process was started in.

Three things about that sentence are worth separating, because each one causes a different bug.

It is the process's working directory, not the script's. The file your code lives in has no influence on it whatsoever.

It is where the process was started, which is decided by whoever started it — your shell, a Makefile, a scheduler. It is an input to your script that does not appear in its arguments (Chapter 1.7's hidden inputs, now with a mechanism).

And it can change while the process runs, which Section 4 demonstrates.

Path.cwd() returns it. Its value is not printed anywhere in this module, for the same reason it causes bugs: it is different on every machine and in every invocation.

3. The Same Script, Two Answers

Here is the entire problem, with no absolute path shown. The script writes a log, then reads it back from two different working directories:

import os
from pathlib import Path

Path("work").mkdir()
Path("sim.log").write_text("the log\n")


def try_relative(label):
    try:
        text = Path("sim.log").read_text().strip()
        print(f"  {label:<16} found {text!r}")
    except FileNotFoundError:
        print(f"  {label:<16} FileNotFoundError")


print("reading the relative path 'sim.log':")
try_relative("from the top")
os.chdir("work")
try_relative("from work/")
os.chdir("..")

reading the relative path 'sim.log':
  from the top     found 'the log'
  from work/       FileNotFoundError

The code is identical on both calls. Same function, same string, same file sitting on the disk the whole time. The only thing that changed is where the process was standing, and the answer to "where is sim.log?" changed with it.

Spell the path absolutely and the dependence disappears:

import os
from pathlib import Path

Path("work").mkdir()
Path("sim.log").write_text("the log\n")

ABSOLUTE = Path("sim.log").resolve()


def try_absolute(label):
    print(f"  {label:<16} found {ABSOLUTE.read_text().strip()!r}")


print("reading the same file by absolute path:")
try_absolute("from the top")
os.chdir("work")
try_absolute("from work/")
os.chdir("..")
print()
print("is the stored path absolute:", ABSOLUTE.is_absolute())
print("its final component       :", ABSOLUTE.name)

reading the same file by absolute path:
  from the top     found 'the log'
  from work/       found 'the log'

is the stored path absolute: True
its final component       : sim.log

Note what is printed about ABSOLUTE: that it is absolute, and its last component. Not its value — because its value is a different string on every machine, which is exactly the property that makes it work and makes it unpublishable.

4. This Is the Third Time

The same fact has already appeared twice under other names, and recognising it as one fact is worth more than the three individually.

Chapter 13.5 Section 6 found that import resolves a module name against the directory you ran Python from, so running project/run.py from the project root and from inside project/ import differently.

Chapter 15.2 Section 6 found that open("sim.log") looks in the working directory, and named it as the usual cause of FileNotFoundError.

This chapter is the general statement: every relative name a process resolves — imports, file reads, writes, subprocess invocations — is resolved against the same single directory, and that directory belongs to whoever started the process.

So a script with one relative path has one dependency on its caller. A script with fifteen has fifteen, and they all break together.

5. Why It Works for You and Fails Under LSF

The canonical purpose names the symptom, and the mechanism is now available to explain it.

When you test the script, you cd to the run directory and type the command. Your working directory is the run directory because you put it there, and every relative path resolves exactly as you intended.

When a scheduler runs it, the process starts in whatever directory the scheduler chose — frequently your home directory, sometimes a scratch area, occasionally a path that exists only on the execution host. Nobody made a decision about this; it is a default. Every relative path in the script now resolves against that directory, and sim.log is not there.

The same applies to a Makefile recipe (which runs in the directory of the make invocation, not the Makefile), a cron job (which typically starts in the home directory), and a CI runner (which starts in the checkout root). Four contexts, four different answers, one unchanged script.

Chapter 18.10 gives the fix — anchor to the project root rather than the working directory — and it is worth knowing now that a fix exists, because the instinct after this chapter is to hard-code an absolute path, which trades one problem for a worse one.

6. resolve() and What It Does Not Fix

resolve() turns a path into an absolute one, collapsing .. and following symlinks:

from pathlib import Path

Path("runs/seed_41").mkdir(parents=True)
messy = Path("runs/seed_41/../seed_41/./sim.log")

print("before resolve, absolute:", messy.is_absolute())
print("parts before            :", messy.parts)

clean = messy.resolve()
print("after resolve, absolute :", clean.is_absolute())
print("name                    :", clean.name)
print("parent's name           :", clean.parent.name)
print("no dot-dot remains      :", ".." not in clean.parts)

before resolve, absolute: False
parts before            : ('runs', 'seed_41', '..', 'seed_41', 'sim.log')
after resolve, absolute : True
name                    : sim.log
parent's name           : seed_41
no dot-dot remains      : True

Two details in that output. The . component is already gone before resolve() ran — Path discards a bare . at construction, because it means "here" and adds nothing. The .. survives, because it cannot be removed without knowing what it is relative to.

resolve() is useful for producing a canonical form: two paths that refer to the same file resolve to the same string, which == on the unresolved versions would not tell you.

But read what it did carefully: it resolved against the current working directory, because the input was relative and there was nothing else to resolve against. resolve() does not remove the dependency on cwd; it bakes it in. A path resolved at the wrong moment is now permanently wrong, and harder to debug because it looks absolute and authoritative.

The rule that follows:

resolve() is for canonicalising a path you already know is correct. It is not a fix for not knowing where you are.

7. Predict, Run, Observe

import os
from pathlib import Path

Path("runs/seed_41").mkdir(parents=True)
Path("runs/seed_41/sim.log").write_text("a\n")

rel = Path("runs/seed_41/sim.log")
abs_at_top = rel.resolve()

print("A:", rel.exists(), abs_at_top.exists())

os.chdir("runs")
print("B:", rel.exists(), abs_at_top.exists())
print("C:", Path("seed_41/sim.log").exists())
os.chdir("..")

print("D:", rel.exists())
print("E:", rel.is_absolute(), abs_at_top.is_absolute())
print("F:", abs_at_top.name, abs_at_top.parent.name)

A: True True
B: False True
C: True
D: True
E: False True
F: sim.log seed_41

A and B — the relative path stopped working and the absolute one did not. rel is the same object throughout; only the process moved.

C — from inside runs/, a different relative path finds the same file. Relative paths are not wrong, they are incomplete: they only mean something in combination with a working directory.

D — moving back restored it. The path was never damaged; the question simply had a different answer for a while.

F — name and parent.name are safe to print because they are the parts you chose, not the machine-specific prefix.

8. Exercises

Exercise 1 — two directories. Write a script that reads a relative path, and run it from its own directory and from the parent. Record both outcomes.

Exercise 2 — the invisible variable. Write down the command you ran in both cases from Exercise 1. Confirm the two commands are identical, and name the thing that differed.

Exercise 3 — absolute survives. Repeat Exercise 1 with the path resolved once at the top of the script. Explain why the second run now works.

Exercise 4 — count your dependencies. Take a real script and count its relative paths. That number is how many ways it depends on its caller.

Exercise 5 — the three appearances. Reproduce the import version from Chapter 13.5 and the open() version from Chapter 15.2. State in one sentence what all three have in common.

Exercise 6 — resolve at the wrong time. Resolve a relative path from the wrong directory and show the result is confidently incorrect. Say why it is harder to debug than the original error.

Exercise 7 — canonical form. Build two different spellings of one file's path and show they compare unequal before resolve() and equal after.

Exercise 8 — predict the scheduler. For a script in your flow, say what its working directory would be under a shell, a Makefile and a cron job. Check one of the three.

9. Summary

• An absolute path starts at the root and means the same thing to every process.
• A relative path is resolved against the process's current working directory.
• That directory belongs to whoever started the process, not to your script or its file.
• It is a hidden input — it does not appear in the command line, which is why the bug report says the script is broken.
• It can change while the process runs, so the same relative path can succeed and then fail.
• Every relative name resolves against it: imports, reads, writes and subprocess calls alike.
• A script with fifteen relative paths has fifteen dependencies on its caller, and they fail together.
• A shell, a Makefile, a cron job and a CI runner start in four different directories.
• resolve() canonicalises and collapses .. and symlinks, and resolves a relative path against cwd — baking the dependency in rather than removing it.
• resolve() is not a fix for not knowing where you are.

The next chapter is about the other half of the problem: building a path correctly once you know what it should be.