Paths Are Not Strings

Part II · Module 18 · Chapter 18.2 · Paths Are Not Strings

1. Why This Chapter Comes Before the Good One

Chapter 18.3 introduces pathlib, and its advice is simple: new flow code should use it. So a chapter about the older os.path style needs a reason to exist, and the reason is in the canonical purpose — you will read far more of it than you will ever write.

Every flow script written before about 2016 builds paths with os.path.join, and a large number written since do too, because they were copied from one that did. Inheriting a two-hundred-line regression wrapper means reading os.path, and reading it carelessly means missing the one behaviour in Section 4 that silently changes what a path means.

So: this chapter is about reading. The next one is about writing.

2. Three Ways Concatenation Fails

The instinct is to glue strings, and all three failures are silent:

run_dir = "runs/seed_41"
trailing = "runs/seed_41/"

print("A missing separator :", run_dir + "logs/sim.log")
print("B manual separator  :", run_dir + "/" + "logs/sim.log")
print("C doubled separator :", trailing + "/" + "logs/sim.log")

A missing separator : runs/seed_41logs/sim.log
B manual separator  : runs/seed_41/logs/sim.log
C doubled separator : runs/seed_41//logs/sim.log

A is wrong and nothing raised. seed_41logs is a directory that does not exist, and the error arrives later as a FileNotFoundError naming a path no human would have typed.

B is correct, and it is correct only because run_dir happened not to end in a slash.

C is the same code as B with a different input, and it produces a doubled separator. That one actually works — the operating system collapses // — which makes it worse rather than better, because the path now behaves correctly and compares unequal to the sensible spelling:

a = "runs/seed_41//logs/sim.log"
b = "runs/seed_41/logs/sim.log"

print("the same file:", a.replace("//", "/") == b)
print("equal as strings:", a == b)
print("in a set of two:", len({a, b}))

the same file: True
equal as strings: False
in a set of two: 2

One file, two spellings, and any code that deduplicates paths, uses them as dictionary keys or compares them against a filelist now sees two different things.

The underlying problem is that whether the concatenation is correct depends on a property of the input string that nothing checks. A function that takes run_dir and appends to it works for half its callers.

3. What os.path.join Actually Does

import os.path

run_dir = "runs/seed_41"
trailing = "runs/seed_41/"
test = "smoke"

print("A:", os.path.join(run_dir, "logs", f"{test}.log"))
print("B:", os.path.join(trailing, "logs", f"{test}.log"))
print("C:", os.path.join("runs", "seed_41", "logs"))
print("D separator in use:", repr(os.sep))

A: runs/seed_41/logs/smoke.log
B: runs/seed_41/logs/smoke.log
C: runs/seed_41/logs
D separator in use: '/'

A and B produce the same string from inputs that differ by a trailing slash. That is the whole value: join inserts exactly one separator where one is needed and none where it is not, so correctness no longer depends on how the caller happened to spell its argument.

It takes any number of components, which is why the canonical form passes three rather than nesting two calls.

4. The Rule That Surprises Everybody

One behaviour of join is not obvious and is worth knowing before you read inherited code:

import os.path

print("A normal      :", os.path.join("runs", "seed_41", "sim.log"))
print("B absolute mid:", os.path.join("runs", "/etc", "passwd"))
print("C absolute end:", os.path.join("runs", "seed_41", "/tmp/other.log"))

A normal      : runs/seed_41/sim.log
B absolute mid: /etc/passwd
C absolute end: /tmp/other.log

An absolute component discards everything before it. join treats a component starting with / as "start again from here", so runs vanishes in B and runs/seed_41 vanishes in C.

The logic is defensible — joining a relative prefix to an absolute path cannot sensibly mean anything else — and the consequence in a flow script is not.

import os.path

RUNS = "runs"

for configured in ["seed_41", "/scratch/seed_41", "seed_42"]:
    full = os.path.join(RUNS, configured, "sim.log")
    inside = full.startswith(RUNS + "/")
    print(f"{configured:<18} -> {full:<28} inside runs/: {inside}")

seed_41            -> runs/seed_41/sim.log         inside runs/: True
/scratch/seed_41   -> /scratch/seed_41/sim.log     inside runs/: False
seed_42            -> runs/seed_42/sim.log         inside runs/: True

A run directory name read from a config file, where one entry happens to be absolute, and the path a script believed was inside runs/ is not. Nothing raised, and the script will read — or in Chapter 18.9's case, delete — somewhere it was never supposed to reach.

Chapter 18.3's / operator has exactly the same rule, so this is not a reason to prefer one over the other — it is a fact about path joining that follows you into the modern API.

5. Reading the Inherited Vocabulary

The rest of os.path is a small set of functions you will meet constantly. Here they are on one path, with their pathlib equivalents beside them:

import os.path
from pathlib import Path

s = "runs/seed_41/sim.log"
p = Path(s)

print("dirname    :", os.path.dirname(s), "   | .parent :", p.parent)
print("basename   :", os.path.basename(s), "           | .name   :", p.name)
print("splitext[0]:", os.path.splitext(s)[0], " | .stem   :", p.stem)
print("splitext[1]:", os.path.splitext(s)[1], "            | .suffix :", p.suffix)
print("exists     :", os.path.exists(s), "        | .exists():", p.exists())

dirname    : runs/seed_41    | .parent : runs/seed_41
basename   : sim.log            | .name   : sim.log
splitext[0]: runs/seed_41/sim  | .stem   : sim
splitext[1]: .log             | .suffix : .log
exists     : False         | .exists(): False

The translation table, which is the thing worth keeping:

Two differences worth noting while reading. os.path functions take and return strings, so the result of one can be passed to any string operation — including the concatenation this chapter opened with, which is how the two styles get mixed in one file. And splitext returns a pair, so inherited code indexes it, which is why [0] and [1] appear so often and are so easy to transpose.

6. The Separator Is Not the Real Argument

The usual justification for join is portability: os.sep is / on Linux and \ on Windows, so joining manually with "/" breaks on Windows.

For a VLSI flow that is close to irrelevant. The tools run on Linux, the farm runs Linux, and a script that hard-codes / will never meet a backslash. Presenting portability as the reason is unconvincing to the audience that needs convincing.

The real arguments are the ones this chapter has shown:

• the separator logic, not the separator character — join knows whether one is needed, and a + does not;
• correctness that does not depend on the caller's spelling, so a trailing slash in a config value cannot change the result;
• one canonical form, so paths compare and deduplicate correctly;
• and, with pathlib, a type that is not a string, so the string operations that silently produce nonsense are not available in the first place.

That last one is the next chapter's subject, and it is the strongest of the four.

7. Predict, Run, Observe

import os.path

base = "runs/seed_41/"
name = "sim.log"

print("A:", base + name)
print("B:", os.path.join(base, name))
print("C:", os.path.join(base, "", name))
print("D:", os.path.join("", name))
print("E:", os.path.dirname("sim.log"))
print("F:", os.path.basename("runs/seed_41/"))
print("G:", os.path.splitext("archive.tar.gz"))

A: runs/seed_41/sim.log
B: runs/seed_41/sim.log
C: runs/seed_41/sim.log
D: sim.log
E: 
F: 
G: ('archive.tar', '.gz')

A and B agree, because this time the base happened to end in a slash. Change base to omit it and A breaks while B does not — the whole chapter in two lines.

E is an empty string, not .. A bare filename has no directory part, and code that does os.path.dirname(p) and then joins to it gets a relative path by accident rather than by intent.

F is also empty, which catches people: basename of a path ending in a separator is "", because the last component genuinely is empty. A loop reporting "processing {basename}" prints nothing for a directory written with a trailing slash.

G splits only the last extension. archive.tar.gz has stem archive.tar, which is correct by the rules and rarely what the author wanted. pathlib has suffixes for the full list, and the same trap for stem.

8. Exercises

Exercise 1 — three failures. Reproduce the missing, manual and doubled separator cases. For each, say whether it raises.

Exercise 2 — depends on the data. Write a function that appends a filename to a directory with +. Call it with and without a trailing slash and show one result is wrong.

Exercise 3 — two spellings. Build the doubled-separator path and the clean one. Show they open the same file and compare unequal.

Exercise 4 — the absolute component. Run os.path.join with an absolute middle component. Then write the containment check that would have caught it.

Exercise 5 — from a config. Build a path from a list of directory names where one is absolute, and report which results are inside the intended root.

Exercise 6 — translate. Take ten lines of inherited os.path code and rewrite them with pathlib using the Section 5 table. Confirm the outputs match.

Exercise 7 — the empty results. Find inputs for which dirname and basename return "". Say what a loop printing them would show.

Exercise 8 — double extension. Run splitext on a .tar.gz and on a file with no extension. Say which of the two surprises you more.

9. Summary

• Concatenating paths fails three ways, all silent: a missing separator, a manual one that depends on the input, and a doubled one.
• A doubled separator works and compares unequal, so one file acquires two spellings.
• Correctness of a + depends on a property of the input that nothing checks.
• os.path.join inserts exactly one separator where needed, so a trailing slash in a caller's value cannot change the result.
• An absolute component discards everything before it — join("runs", "/etc", "passwd") is /etc/passwd.
• That rule applies to pathlib's / too, and is the reason a cleanup script needs a containment check.
• os.path takes and returns strings, which is how the two styles get mixed in one file.
• The translation table is the thing to keep: dirname/parent, basename/name, splitext/stem and suffix.
• Portability is the weakest argument for a Linux-only flow; the separator logic, canonical form and a non-string type are the real ones.
• dirname and basename return "" for some inputs, and splitext splits only the last extension.

The next chapter is the API you should actually write, where a path stops being a string at all.