Skip to content
VLSI Mentor

Python Part II · Module 33 · Testing and Debugging Automation

Temporary Directories: Tests That Touch Nothing Real

A test that writes a file has to write it somewhere, and the obvious somewheres are all wrong: the project directory pollutes the repository, a fixed temporary path collides when two tests run at once, and a real run area is somebody's actual data. The standard library solves the placement in one line. What it does not solve, and what this chapter measures, is what happens to that directory when the test fails — because the cleanup most people write is the cleanup that does not run on exactly the runs that produce the mess.

Intermediate19 min readPythonPart IIAutomationTestingFilesystem

Part II · Module 33 · Chapter 33.7 · Temporary Directories: Tests That Touch Nothing Real

1. Four Places a Test Could Write

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
PLACES = [
    ("the project directory", "pollutes the repo, shows in git status", "no"),
    ("a fixed temp path", "two tests collide — §5", "no"),
    ("a real run area", "somebody's actual results", "never"),
    ("a fresh temp directory per test", "isolated, parallel-safe", "yes"),
]

print(f"{'where':<34} {'consequence':<40} use it")
for where, why, ok in PLACES:
    print(f"{where:<34} {why:<40} {ok}")

print()
print("places that work:", 1)
print("the purpose's three requirements:",
      ["runs anywhere", "runs in parallel", "touches nothing real"])
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
where                              consequence                              use it
the project directory              pollutes the repo, shows in git status   no
a fixed temp path                  two tests collide — §5                   no
a real run area                    somebody's actual results                never
a fresh temp directory per test    isolated, parallel-safe                  yes

places that work: 1
the purpose's three requirements: ['runs anywhere', 'runs in parallel', 'touches nothing real']

One of four, and the three requirements in the canonical purpose are independent — a suite can satisfy "touches nothing real" and still fail "in parallel", which is Section 5.

2. The Annotation

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
def test_writes_report(tmp_path):
    write_report(tmp_path / "r.csv", results)

tmp_path is pytest's fixture: it injects a fresh directory per test and removes it afterwards. pytest is not installed here, so the same thing in the standard library:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import csv
import io
import tempfile
import unittest
from pathlib import Path

RESULTS = [{"test": "axi_burst", "seed": 7731, "status": "FAIL"},
           {"test": "axi_wrap", "seed": 1190, "status": "PASS"}]
FIELDS = ["test", "seed", "status"]


def write_report(path, results):
    with open(path, "w", encoding="utf-8", newline="") as f:
        w = csv.DictWriter(f, fieldnames=FIELDS)
        w.writeheader()
        for r in results:
            w.writerow(r)
    return path


class ReportTests(unittest.TestCase):
    def setUp(self):
        self.tmp = tempfile.TemporaryDirectory()
        self.tmp_path = Path(self.tmp.name)
        self.addCleanup(self.tmp.cleanup)

    def test_writes_report(self):
        out = write_report(self.tmp_path / "r.csv", RESULTS)
        self.assertTrue(out.is_file())

    def test_report_has_a_header_and_two_rows(self):
        out = write_report(self.tmp_path / "r.csv", RESULTS)
        lines = out.read_text(encoding="utf-8").splitlines()
        self.assertEqual(lines[0], "test,seed,status")
        self.assertEqual(len(lines), 3)


def run(case):
    suite = unittest.TestLoader().loadTestsFromTestCase(case)
    res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
    return {"run": res.testsRun, "failures": len(res.failures),
            "ok": res.wasSuccessful()}


print("suite:", run(ReportTests))
print()
print("pytest's tmp_path      ->", "tempfile.TemporaryDirectory in setUp")
print("pytest's auto-cleanup  ->", "self.addCleanup(self.tmp.cleanup)")
print("a fresh directory per test:", True)
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
suite: {'run': 2, 'failures': 0, 'ok': True}

pytest's tmp_path      -> tempfile.TemporaryDirectory in setUp
pytest's auto-cleanup  -> self.addCleanup(self.tmp.cleanup)
a fresh directory per test: True

Four lines of setUp replace the fixture, and setUp runs before every test method, so the two tests above each got their own directory and neither saw the other's r.csv.

3. The Mapping, Stated Once

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
MAPPING = [
    ("tmp_path", "Path(tempfile.TemporaryDirectory().name)", "a fresh dir"),
    ("tmp_path_factory", "tempfile.mkdtemp()", "several dirs in one test"),
    ("automatic teardown", "self.addCleanup(...)", "runs even on failure — §4"),
    ("monkeypatch.chdir", "os.chdir + addCleanup(os.chdir, old)", "§6"),
    ("capsys", "contextlib.redirect_stdout(io.StringIO())", "capture output"),
]

print(f"{'pytest':<22} {'standard library':<46} what it gives")
for a, b, c in MAPPING:
    print(f"{a:<22} {b:<46} {c}")

print()
print("features needed by this module:", len(MAPPING))
print("features missing from the stdlib:", 0)
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
pytest                 standard library                               what it gives
tmp_path               Path(tempfile.TemporaryDirectory().name)       a fresh dir
tmp_path_factory       tempfile.mkdtemp()                             several dirs in one test
automatic teardown     self.addCleanup(...)                           runs even on failure — §4
monkeypatch.chdir      os.chdir + addCleanup(os.chdir, old)           §6
capsys                 contextlib.redirect_stdout(io.StringIO())      capture output

features needed by this module: 5
features missing from the stdlib: 0

Nothing this module needs is missing from the standard library. pytest's version is shorter and that is its whole advantage here; the mechanics are identical, and a flow that cannot install packages — which describes a great many EDA environments — loses nothing but brevity.

4. Cleanup After the Assertion Does Not Run

This is the chapter's main finding, and it is measured on two deliberately failing tests:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import io
import shutil
import tempfile
import unittest
from pathlib import Path

MADE = []


class CleanupComparison(unittest.TestCase):
    def test_cleanup_written_after_the_assertion(self):
        d = tempfile.mkdtemp()
        MADE.append(("inline", d))
        self.assertEqual(1, 2, "deliberate failure")
        shutil.rmtree(d)                       # never reached

    def test_cleanup_registered_with_addCleanup(self):
        d = tempfile.mkdtemp()
        MADE.append(("addCleanup", d))
        self.addCleanup(shutil.rmtree, d, True)
        self.assertEqual(1, 2, "deliberate failure")


suite = unittest.TestLoader().loadTestsFromTestCase(CleanupComparison)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)

print("tests run      :", res.testsRun)
print("failures       :", len(res.failures), "(both deliberate)")
print()
print("directories created:", len(MADE))
for kind, d in MADE:
    print(f"  {kind:<12} still exists after the suite: {Path(d).is_dir()}")

leaked = [d for kind, d in MADE if Path(d).is_dir()]
print()
print("directories leaked:", len(leaked))
for d in leaked:
    shutil.rmtree(d, ignore_errors=True)
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
tests run      : 2
failures       : 2 (both deliberate)

directories created: 2
  addCleanup   still exists after the suite: False
  inline       still exists after the suite: True

directories leaked: 1

One directory leaked and one did not, from two tests that both failed.

A failing assertion raises, so every statement after it in the method is skipped — including shutil.rmtree. That is the ordinary behaviour of an exception and it has an unpleasant consequence for cleanup: inline cleanup runs on the tests that pass and leaks on the tests that fail, which is the exact inverse of what you want. A suite that fails a hundred times during a debugging session leaves a hundred directories.

addCleanup registers the call with the test framework, which runs it in a finally-like phase regardless of outcome.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import io
import shutil
import tempfile
import unittest
from pathlib import Path

ORDER = []


class ThreeWays(unittest.TestCase):
    def test_context_manager(self):
        with tempfile.TemporaryDirectory() as d:
            ORDER.append(("ctx-enter", Path(d).is_dir()))
            try:
                self.assertEqual(1, 2)
            except AssertionError:
                pass
        ORDER.append(("ctx-after", Path(d).is_dir()))

    def test_tear_down_method(self):
        self.d = tempfile.mkdtemp()
        ORDER.append(("tearDown-during", Path(self.d).is_dir()))
        self.assertEqual(1, 2, "deliberate")

    def tearDown(self):
        d = getattr(self, "d", None)
        if d:
            shutil.rmtree(d, ignore_errors=True)
            ORDER.append(("tearDown-after", Path(d).is_dir()))


suite = unittest.TestLoader().loadTestsFromTestCase(ThreeWays)
res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)

print("failures:", len(res.failures), "(deliberate)")
print()
for what, exists in ORDER:
    print(f"  {what:<18} directory exists: {exists}")
print()
print("all three mechanisms clean up on failure:", True)
print("the one that does not:", "a bare rmtree after the assertion")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
failures: 1 (deliberate)

  ctx-enter          directory exists: True
  ctx-after          directory exists: False
  tearDown-during    directory exists: True
  tearDown-after     directory exists: False

all three mechanisms clean up on failure: True
the one that does not: a bare rmtree after the assertion

Three mechanisms work and one does not. A with block, a tearDown method and addCleanup all run on failure; a statement after the assertion does not.

addCleanup is the one to prefer for a specific reason: it is registered at the point the resource is created, so there is no way to create something in one place and forget the cleanup in another. tearDown has to guess what the test made, which is why the example above needs getattr(self, "d", None) — a test that failed before creating the directory would otherwise crash in teardown.

5. Two Tests, One Fixed Path

The parallel half of the purpose:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import tempfile
from pathlib import Path

FIXED = Path(tempfile.gettempdir()) / "flow_tests_report"
FIXED.mkdir(exist_ok=True)


def test_a_writes():
    (FIXED / "r.csv").write_text("test,seed,status\naxi_a,1,PASS\n",
                                 encoding="utf-8")
    return (FIXED / "r.csv").read_text(encoding="utf-8")


def test_b_writes():
    (FIXED / "r.csv").write_text("test,seed,status\naxi_b,2,FAIL\n",
                                 encoding="utf-8")
    return (FIXED / "r.csv").read_text(encoding="utf-8")


a = test_a_writes()
b = test_b_writes()
print("test A wrote:", a.splitlines()[1])
print("test B wrote:", b.splitlines()[1])
print("what is on disk now:",
      (FIXED / "r.csv").read_text(encoding="utf-8").splitlines()[1])
print()
print("A's assertion would now read B's data:", True)
print("the two tests are:", "order-dependent and not parallel-safe")

import shutil
shutil.rmtree(FIXED, ignore_errors=True)
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
test A wrote: axi_a,1,PASS
test B wrote: axi_b,2,FAIL
what is on disk now: axi_b,2,FAIL

A's assertion would now read B's data: True
the two tests are: order-dependent and not parallel-safe

B overwrote A's file. Run sequentially the tests happen to pass, because each writes immediately before reading; run in parallel, or interleaved by a framework that shuffles, and A reads B's row.

mkdtemp exists precisely to avoid inventing a name:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import tempfile
from pathlib import Path

paths = [tempfile.mkdtemp() for _ in range(4)]

print("directories requested:", len(paths))
print("distinct paths        :", len(set(paths)))
print("all exist             :", all(Path(p).is_dir() for p in paths))
print("any two equal         :", len(set(paths)) != len(paths))
print()
print("the names are random, so this chapter does not print them")
print("what is asserted instead:", "that they are distinct and exist")

import shutil
for p in paths:
    shutil.rmtree(p, ignore_errors=True)
print("after cleanup, any still exist:", any(Path(p).is_dir() for p in paths))
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
directories requested: 4
distinct paths        : 4
all exist             : True
any two equal         : False

the names are random, so this chapter does not print them
what is asserted instead: that they are distinct and exist
after cleanup, any still exist: False

Four requests, four distinct directories, and the assertions are about distinctness and existence rather than about any particular path — which is both the determinism rule and the only assertion that can hold on a second machine.

6. A chdir Leaks Into Every Later Test

The working directory is process-wide state and unittest does not isolate it:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import io
import os
import shutil
import tempfile
import unittest
from pathlib import Path

OBSERVED = []
START = os.getcwd()


class LeakyChdir(unittest.TestCase):
    def test_a_changes_directory(self):
        OBSERVED.append(("a", os.getcwd() == START))
        d = tempfile.mkdtemp()
        self.addCleanup(shutil.rmtree, d, True)
        os.chdir(d)                     # never restored

    def test_b_runs_afterwards(self):
        OBSERVED.append(("b", os.getcwd() == START))


suite = unittest.TestLoader().loadTestsFromTestCase(LeakyChdir)
unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)

print("each test saw the starting directory?")
for name, same in OBSERVED:
    print(f"  test_{name}: {same}")

print()
try:
    restored = os.getcwd() == START
    print("process cwd restored after the suite:", restored)
except FileNotFoundError as exc:
    print("os.getcwd() after the suite raised:", type(exc).__name__)
    print("because the directory the process is in:", "was deleted by addCleanup")
os.chdir(START)
print("restored by hand; cwd == start now:", os.getcwd() == START)
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
each test saw the starting directory?
  test_a: True

os.getcwd() after the suite raised: FileNotFoundError
because the directory the process is in: was deleted by addCleanup
restored by hand; cwd == start now: True

Read what happened to test_b: it never recorded an observation at all. The run is worse than "B saw the wrong directory" — addCleanup deleted A's temporary directory while the process was still inside it, so by the time B ran, the process's working directory did not exist. os.getcwd() then raises FileNotFoundError, which is why the final check had to be wrapped in a try.

That is a failure mode with no good diagnostic. B fails with an error about a missing directory it never mentioned, the traceback points into B, and the cause is a line in A.

That is the worst shape of test failure: a passing test and a failing one where the failure is caused by the passing one. The fix is to register the restore at the moment of the change:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import io
import os
import shutil
import tempfile
import unittest

OBSERVED = []
START = os.getcwd()


class IsolatedChdir(unittest.TestCase):
    def setUp(self):
        self.start = os.getcwd()
        self.addCleanup(os.chdir, self.start)
        d = tempfile.mkdtemp()
        self.addCleanup(shutil.rmtree, d, True)
        os.chdir(d)

    def test_a_runs_in_its_own_dir(self):
        OBSERVED.append(("a", os.getcwd() != self.start))

    def test_b_also_runs_in_its_own_dir(self):
        OBSERVED.append(("b", os.getcwd() != self.start))


suite = unittest.TestLoader().loadTestsFromTestCase(IsolatedChdir)
unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)

print("each test ran in a directory of its own?")
for name, own in OBSERVED:
    print(f"  test_{name}: {own}")
print()
print("process cwd restored after the suite:", os.getcwd() == START)
print()
print("the order addCleanup runs them:", "reverse of registration")
print("so the chdir-back happens:", "before the rmtree — which is required")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
each test ran in a directory of its own?
  test_a: True
  test_b: True

process cwd restored after the suite: True

the order addCleanup runs them: reverse of registration
so the chdir-back happens: before the rmtree — which is required

Both tests isolated and the process restored. The last two lines are the detail that makes it correct: addCleanup runs its callbacks in reverse registration order, so registering os.chdir first and rmtree second means the directory change is undone before the directory is removed. Registering them the other way round would try to remove the current working directory.

7. Not Writing at All Is Better

A temporary directory is the answer when a function must write. Chapter 33.2 Section 5's seam often means it does not have to:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import csv
import io
import tempfile
import unittest
from pathlib import Path

FIELDS = ["test", "seed", "status"]
RESULTS = [{"test": "axi_burst", "seed": 7731, "status": "FAIL"}]


def render_report(results):
    """Pure: returns the text. No filesystem."""
    buf = io.StringIO()
    w = csv.DictWriter(buf, fieldnames=FIELDS)
    w.writeheader()
    for r in results:
        w.writerow(r)
    return buf.getvalue()


def write_report(path, results):
    """Two lines, and the only part that needs a directory."""
    Path(path).write_text(render_report(results), encoding="utf-8", newline="")
    return path


class PureRender(unittest.TestCase):
    def test_header(self):
        self.assertEqual(render_report(RESULTS).splitlines()[0],
                         "test,seed,status")

    def test_one_row(self):
        self.assertEqual(len(render_report(RESULTS).splitlines()), 2)

    def test_the_row_contents(self):
        self.assertIn("axi_burst,7731,FAIL", render_report(RESULTS))


class ActuallyWrites(unittest.TestCase):
    def setUp(self):
        self.tmp = tempfile.TemporaryDirectory()
        self.addCleanup(self.tmp.cleanup)

    def test_the_file_appears(self):
        p = write_report(Path(self.tmp.name) / "r.csv", RESULTS)
        self.assertTrue(p.is_file())


def run(*cases):
    loader = unittest.TestLoader()
    suite = unittest.TestSuite(loader.loadTestsFromTestCase(c) for c in cases)
    res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
    return {"run": res.testsRun, "failures": len(res.failures)}


print("suite:", run(PureRender, ActuallyWrites))
print()
print("tests needing a temp directory:", 1)
print("tests needing nothing          :", 3)
print("facts about the CSV format asserted without touching a disk:", 3)
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
suite: {'run': 4, 'failures': 0}

tests needing a temp directory: 1
tests needing nothing          : 3
facts about the CSV format asserted without touching a disk: 3

Three of four tests need no directory at all. Splitting write_report into a pure render_report plus a two-line writer moves every assertion about the format — the header, the row count, the row contents — onto a string, and leaves exactly one test that needs a filesystem: the one asserting a file appears.

That is Chapter 33.2 Section 5's seam producing a second dividend. There it made the decision testable; here it makes three tests independent of the filesystem, which means they cannot leak, cannot collide and cannot be affected by a chdir.

8. The Isolated Suite, Assembled

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import csv
import io
import os
import shutil
import tempfile
import unittest
from pathlib import Path

FIELDS = ["run_id", "test", "seed", "status"]


def render_report(run_id, results):
    buf = io.StringIO()
    w = csv.DictWriter(buf, fieldnames=FIELDS, extrasaction="raise")
    w.writeheader()
    for r in results:
        w.writerow({"run_id": run_id, **r})
    return buf.getvalue()


def write_report(path, run_id, results):
    p = Path(path)
    p.parent.mkdir(parents=True, exist_ok=True)
    p.write_text(render_report(run_id, results), encoding="utf-8", newline="")
    return p


RESULTS = [{"test": "axi_burst", "seed": 7731, "status": "FAIL"},
           {"test": "axi_wrap", "seed": 1190, "status": "PASS"}]


class TempDirCase(unittest.TestCase):
    """Base: a fresh directory, cwd inside it, both undone in the right order."""

    def setUp(self):
        self.start = os.getcwd()
        self.addCleanup(os.chdir, self.start)
        self.tmp = Path(tempfile.mkdtemp())
        self.addCleanup(shutil.rmtree, self.tmp, True)
        os.chdir(self.tmp)

    def assertIsolated(self):
        self.assertTrue(self.tmp.is_dir())
        self.assertNotEqual(Path(os.getcwd()).resolve(),
                            Path(self.start).resolve())


class ReportWriting(TempDirCase):
    def test_runs_in_its_own_directory(self):
        self.assertIsolated()

    def test_writes_into_a_subdirectory_it_creates(self):
        p = write_report(Path("out") / "regression.csv", "r3", RESULTS)
        self.assertTrue(p.is_file())
        self.assertTrue(Path("out").is_dir())

    def test_relative_paths_resolve_inside_the_temp_dir(self):
        p = write_report("regression.csv", "r3", RESULTS)
        self.assertEqual(p.resolve().parent, self.tmp.resolve())

    def test_rows_carry_the_run_id(self):
        p = write_report("regression.csv", "r3", RESULTS)
        rows = list(csv.DictReader(p.read_text(encoding="utf-8").splitlines()))
        self.assertEqual({r["run_id"] for r in rows}, {"r3"})

    def test_a_failing_test_still_cleans_up(self):
        # recorded so the check below can see whether the dir survived
        type(self).LEAKED_CANDIDATE = self.tmp
        self.assertTrue(True)


def run(case):
    suite = unittest.TestLoader().loadTestsFromTestCase(case)
    res = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
    return {"run": res.testsRun, "failures": len(res.failures),
            "errors": len(res.errors), "ok": res.wasSuccessful()}


start_cwd = os.getcwd()
print("suite:", run(ReportWriting))
print()
print("cwd restored after the suite        :", os.getcwd() == start_cwd)
leaked = getattr(ReportWriting, "LEAKED_CANDIDATE", None)
print("the last test's temp dir removed    :",
      leaked is not None and not leaked.is_dir())
print("tests that needed a real path       :", 4)
print("cleanups registered per test        :", 2)
print("registration order                  :", "chdir-back first, rmtree second")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
suite: {'run': 5, 'failures': 0, 'errors': 0, 'ok': True}

cwd restored after the suite        : True
the last test's temp dir removed    : True
tests that needed a real path       : 4
cleanups registered per test        : 2
registration order                  : chdir-back first, rmtree second

Five tests, each in its own directory, with the working directory restored and nothing left behind.

The base class is the piece worth copying. Putting the four lines of setUp in a TempDirCase means every test class that needs a filesystem inherits the isolation rather than re-implementing it — and the two addCleanup registrations are in the one place where their order can be got right once.

Note test_relative_paths_resolve_inside_the_temp_dir. Because setUp also changes directory, a function under test that writes to a relative path — which is most flow code, by Chapter 23.7's run-directory convention — lands inside the temporary directory without the test having to pass an absolute path in. That is what makes it possible to test code that was not written with testing in mind.

9. Comparing Paths, and Sharing One Directory

Two mistakes survive everything this chapter has set up so far. The first is comparing path strings:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import os
import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as d:
    given = Path(d)                       # the path the test hands the tool
    resolved = given.resolve()            # the path the tool reports back
    log = given / "sim.log"
    log.write_text("UVM_INFO started\n", encoding="utf-8")

    print("on the machine these examples were verified on (macOS):")
    print("  the two paths are equal      :", given == resolved)
    print("  their child strings are equal:",
          str(log) == str(resolved / "sim.log"))
    print("  samefile() says it is one file:",
          log.samefile(resolved / "sim.log"))
    print("  the file exists under both    :",
          log.is_file() and (resolved / "sim.log").is_file())
    print()
    print("  one path is a suffix of the other:",
          str(resolved).endswith(str(given)))
    print("  the resolved one is longer       :",
          len(str(resolved)) > len(str(given)))
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
on the machine these examples were verified on (macOS):
  the two paths are equal      : False
  their child strings are equal: False
  samefile() says it is one file: True
  the file exists under both    : True

  one path is a suffix of the other: True
  the resolved one is longer       : True

Two unequal strings naming one file. macOS puts the temporary directory under a symlink, so the path tempfile hands you and the path resolve() returns differ by a prefix — and a test asserting assertEqual(result["log_path"], str(tmp / "sim.log")) fails, with a message showing two paths that look almost identical, for a tool that did exactly the right thing.

On Linux the two are usually equal, which is worse than if they never were: the test passes in CI and fails on a developer's laptop, or the reverse, and the diff shows nothing wrong.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import os
import tempfile
from pathlib import Path

CHOICES = [
    ("assertEqual(str(a), str(b))", "no", "fails across a symlinked tmpdir"),
    ("assertEqual(a, b)", "no", "Path equality is also string-based"),
    ("assertTrue(a.samefile(b))", "yes", "asks the filesystem"),
    ("assertEqual(a.resolve(), b.resolve())", "yes", "normalises both sides"),
    ("assertEqual(a.name, b.name)", "partly", "ignores the directory entirely"),
    ("assertTrue(a.is_relative_to(tmp))", "yes", "the usual real question"),
]

print(f"{'assertion':<42} {'robust':<9} why")
for how, robust, why in CHOICES:
    print(f"{how:<42} {robust:<9} {why}")

print()
print("robust choices:", 3)
print("the question most tests actually have:",
      "is this file inside the directory I gave it?")
print("which is answered by:", "is_relative_to, not equality")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
assertion                                  robust    why
assertEqual(str(a), str(b))                no        fails across a symlinked tmpdir
assertEqual(a, b)                          no        Path equality is also string-based
assertTrue(a.samefile(b))                  yes       asks the filesystem
assertEqual(a.resolve(), b.resolve())      yes       normalises both sides
assertEqual(a.name, b.name)                partly    ignores the directory entirely
assertTrue(a.is_relative_to(tmp))          yes       the usual real question

Three robust choices, and the last row is the one to reach for first. Most assertions about a path are not really about its exact spelling — they are asking did the tool write inside the directory I gave it, which is_relative_to answers directly and which no amount of string comparison answers at all.

Path.__eq__ is the trap worth naming explicitly. It compares the normalised string, not the file, so switching from str(a) == str(b) to a == b looks like an improvement and changes nothing.

The second mistake is sharing one directory across tests:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import io
import shutil
import tempfile
import unittest
from pathlib import Path


class SharedDirectory(unittest.TestCase):
    """setUpClass runs once, so every test writes into the same directory."""

    @classmethod
    def setUpClass(cls):
        cls.tmp = tempfile.mkdtemp()

    @classmethod
    def tearDownClass(cls):
        shutil.rmtree(cls.tmp, ignore_errors=True)

    def test_a_writes_a_file(self):
        Path(self.tmp, "state.txt").write_text("from a", encoding="utf-8")

    def test_b_should_see_nothing(self):
        SharedDirectory.leaked = Path(self.tmp, "state.txt").exists()


class FreshDirectory(unittest.TestCase):
    """setUp runs per test, so each one gets its own."""

    def setUp(self):
        self.tmp = tempfile.mkdtemp()
        self.addCleanup(shutil.rmtree, self.tmp, True)

    def test_a_writes_a_file(self):
        Path(self.tmp, "state.txt").write_text("from a", encoding="utf-8")

    def test_b_should_see_nothing(self):
        FreshDirectory.leaked = Path(self.tmp, "state.txt").exists()


for case in (SharedDirectory, FreshDirectory):
    suite = unittest.TestLoader().loadTestsFromTestCase(case)
    unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
    print(f"  {case.__name__:<18} test_b saw test_a's file: {case.leaked}")

print()
print("both suites passed:", True)
print("one of them is isolated:", "only FreshDirectory")
print("and the leak is invisible:", "until a test depends on it")
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
  SharedDirectory    test_b saw test_a's file: True
  FreshDirectory     test_b saw test_a's file: False

both suites passed: True
one of them is isolated: only FreshDirectory
and the leak is invisible: until a test depends on it

setUpClass gives every test in the class the same directory. test_b saw a file test_a wrote, and both suites reported success — so the leak costs nothing today and produces a test that passes only when run after its sibling, which is Chapter 33.14's order dependence arriving inside a suite.

setUpClass is not wrong; it is for expensive setup that is genuinely read-only — compiling a design once for twenty tests that only read the artefact. The moment a test writes into that directory, the sharing becomes a dependency between tests that nothing declares.

Note the addCleanup(shutil.rmtree, self.tmp, True) form in the second class. Passing the function and its arguments separately rather than wrapping them in a lambda means addCleanup captures the value of self.tmp now, which matters because a lambda would capture self and read self.tmp later — after a subsequent setUp has already replaced it.

10. Predict, Run, Observe

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import os
import shutil
import tempfile
from pathlib import Path

# A
d1, d2 = tempfile.mkdtemp(), tempfile.mkdtemp()
print("A:", d1 == d2, Path(d1).is_dir() and Path(d2).is_dir())
shutil.rmtree(d1, ignore_errors=True)
shutil.rmtree(d2, ignore_errors=True)

# B
with tempfile.TemporaryDirectory() as d:
    inside = Path(d).is_dir()
print("B:", inside, Path(d).is_dir())

# C
d3 = tempfile.mkdtemp()
try:
    raise AssertionError("boom")
    shutil.rmtree(d3)
except AssertionError:
    pass
print("C:", Path(d3).is_dir())
shutil.rmtree(d3, ignore_errors=True)

# D
start = os.getcwd()
d4 = tempfile.mkdtemp()
os.chdir(d4)
moved = os.getcwd() != start
os.chdir(start)
shutil.rmtree(d4, ignore_errors=True)
print("D:", moved, os.getcwd() == start)

The verified output:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
A: False True
B: True False
C: True
D: True True

A — two mkdtemp calls give two different directories, both existing. No name was invented and none printed.

B — the context manager's directory exists inside the block and is gone after it.

C — the rmtree after the raise never ran, so the directory survives. Section 4's finding in five lines.

D — chdir is process-wide, and restoring it is the caller's job.

11. Exercises

Exercise 1 — four places. Say where your own tests write today, and which of the four it is.

Exercise 2 — three requirements. Check your suite against runs-anywhere, runs-in-parallel and touches-nothing-real.

Exercise 3 — git status. Run your suite and check whether anything new appears in the repository.

Exercise 4 — the fixture, in the stdlib. Convert one pytest-style test to setUp plus addCleanup.

Exercise 5 — the mapping. For each pytest feature your suite uses, name the stdlib equivalent.

Exercise 6 — fail on purpose. Add a failing assertion before an inline cleanup and confirm the directory leaks.

Exercise 7 — count the leak. Run a failing suite ten times and count the directories left behind.

Exercise 8 — the inverse of what you want. Explain in one sentence why inline cleanup leaks exactly when it matters.

Exercise 9 — three mechanisms. Demonstrate that with, tearDown and addCleanup all survive a failure.

Exercise 10 — registered at creation. Say why addCleanup is preferable to tearDown for a resource made mid-test.

Exercise 11 — keep the evidence. For one test that drives a real tool, decide what to copy out before cleanup.

Exercise 12 — a fixed path. Make two tests share one temporary file and run them in both orders.

Exercise 13 — order dependence. Explain why they pass sequentially and fail in parallel.

Exercise 14 — distinct and existing. Write the two assertions that are safe to make about a temporary path.

Exercise 15 — never print the path. Find an assertion in your suite that mentions a specific temporary path.

Exercise 16 — the chdir leak. Write the two-test demonstration and confirm the second sees the wrong directory.

Exercise 17 — a failure caused by a passing test. Explain why this is the worst shape of test failure.

Exercise 18 — reverse order. Say what happens if rmtree is registered before chdir-back.

Exercise 19 — isolate the cwd. Add the restore to your own base class.

Exercise 20 — do not write at all. Split one writer into a pure renderer plus a two-line writer.

Exercise 21 — count the tests freed. Report how many of your tests stop needing a filesystem.

Exercise 22 — a base class. Write TempDirCase for your suite and inherit it twice.

Exercise 23 — relative paths. Confirm that code writing to a relative path lands inside your temporary directory.

Exercise 24 — code not written for testing. Say what the chdir-in-setUp trick makes testable that otherwise would not be.

Exercise 25 — two strings, one file. Reproduce the path comparison on your own machine and say whether they differ.

Exercise 26 — which platform. Say what the same test does on Linux, and why that is worse than always differing.

Exercise 27 — Path equality. Confirm that a == b is no more robust than comparing strings.

Exercise 28 — the real question. For three path assertions of yours, decide whether equality or is_relative_to is what you meant.

Exercise 29 — samefile. Rewrite one assertion with samefile and say what it now asks.

Exercise 30 — share a directory. Convert one setUp to setUpClass and check whether a later test sees an earlier one's file.

Exercise 31 — run one test alone. Run a single test by name and confirm it still passes.

Exercise 32 — when setUpClass is right. Name an expensive, read-only setup in your flow that belongs there.

Exercise 33 — the lambda trap. Explain why addCleanup(rmtree, self.tmp, True) is safer than wrapping it in a lambda.

12. Summary

  • One of four plausible places is correct: a fresh temporary directory per test. The project directory, a fixed temporary path and a real run area are all wrong.
  • Never print a temporary path. The names are random and machine-specific, so assert distinctness and existence instead — the same determinism rule that forbids publishing durations.
  • tmp_path maps to four lines of setUp, and nothing this module needs is missing from the standard library; pytest's advantage here is brevity alone.
  • Cleanup written after the assertion does not run when the test fails. Measured: one directory leaked, one did not, from two failing tests.
  • That is the inverse of what you want — inline cleanup tidies the passing runs and leaks on the failing ones, so a debugging session leaves a directory per failure.
  • with, tearDown and addCleanup all survive a failure; addCleanup is preferable because it is registered at the point the resource is created.
  • There is a real tension with keeping evidence. For a unit test, clean up; for a test that drove a tool, copy the log out first.
  • Two tests sharing a fixed path collide — B overwrote A — and they pass sequentially while failing in parallel.
  • A chdir leaks into every later test. unittest provides no working-directory isolation, and the process stayed in the wrong directory after the suite finished.
  • A failure caused by a passing test is the worst shape of failure, and the fix is registering the restore at the moment of the change.
  • addCleanup runs in reverse registration order, so register chdir-back before rmtree or the removal targets the current directory.
  • Three of four tests needed no directory at all once the writer was split into a pure renderer plus two lines — Chapter 33.2's seam paying a second dividend.
  • Put the isolation in a base class, where the registration order can be got right once.
  • chdir-in-setUp makes relative-path code testable without modifying code that was not written with tests in mind.
  • Two unequal path strings can name one file. macOS puts the temp directory under a symlink, so the path you pass in and the path resolve() returns differ by a prefix.
  • On Linux they are usually equal, which is worse: the test passes on one machine and fails on another with a diff that shows nothing wrong.
  • Path.__eq__ compares the normalised string, not the file, so a == b is no improvement over str(a) == str(b).
  • Most path assertions really ask “did it write inside the directory I gave it?” — which is is_relative_to, not equality. Use samefile when identity is the claim.
  • setUpClass shares one directory across every test in the class, and a test that writes there creates an undeclared dependency on run order — both suites still pass.
  • An order-dependent test fails alone and passes in the suite. Running one test by name is the cheapest check for the whole class of problem.
  • Pass addCleanup the function and its arguments, not a lambda, so the current value is captured rather than read back after the next setUp.

The next chapter tests the part of a script other programs actually depend on: its exit code.

Where this fits

Part of the Python curriculum.