with: Why Files Close Themselves

Part I · Module 15 · Chapter 15.5 · with: Why Files Close Themselves

1. An Open File Is a Borrowed Resource

open() does not copy a file into your program. It asks the operating system for a file descriptor — a numbered slot in a table the kernel keeps for your process. The number is small, the table is finite, and the slot stays taken until someone gives it back.

Closing is giving it back. Everything in this chapter follows from the fact that the handing-back must happen even when your code is in the middle of failing.

2. The Guarantee

with open(path) as f:
    ...
# closed here, even if the body raised

The comment is the entire specification. Proof:

Compile start

handle = None

try:
    with open("sim.log") as f:
        handle = f
        print("inside the block, closed =", f.closed)
        raise ValueError("the parser blew up")
except ValueError as e:
    print("caught:", e)

print("after the block, closed =", handle.closed)

inside the block, closed = False
caught: the parser blew up
after the block, closed = True

The body raised. The exception propagated normally and was caught outside. And the file was closed anyway, before the handler ran.

Compare the same code without with:

Compile start

handle = open("sim.log")

try:
    raise ValueError("the parser blew up")
except ValueError as e:
    print("caught:", e)

print("after the failure, closed =", handle.closed)
handle.close()

caught: the parser blew up
after the failure, closed = False

Same failure, same handler, and the descriptor is still held. The close() on the last line only runs because nothing stopped execution reaching it — which in a real script is exactly what an exception does.

3. On a Write, the Damage Is Data

For a read, a leaked handle is untidy. For a write it is worse, because writes are buffered: f.write() puts your text in memory and the operating system receives it later. Closing flushes the buffer. Not closing can mean the data never arrives:

import subprocess, sys

CHILD = '''
f = open("out.txt", "w")
f.write("line one\\n")
f.write("line two\\n")
# no close, and the process is about to be killed outright
import os
os._exit(0)
'''

with open("child.py", "w") as f:
    f.write(CHILD)

subprocess.run([sys.executable, "child.py"])

with open("out.txt") as f:
    print("file contains:", repr(f.read()))

file contains: ''

Two lines were written and the file is empty. The text was sitting in a buffer when the process was killed, and buffers do not survive the process.

A with block removes the possibility:

import subprocess, sys

CHILD = '''
with open("out.txt", "w") as f:
    f.write("line one\\n")
    f.write("line two\\n")
import os
os._exit(0)
'''

with open("child.py", "w") as f:
    f.write(CHILD)

subprocess.run([sys.executable, "child.py"])

with open("out.txt") as f:
    print("file contains:", repr(f.read()))

file contains: 'line one\nline two\n'

This is the Chapter 15.1 nightmare arriving from your own side: a file that is valid text and simply shorter than it should be. The next tool in the flow will read it without complaint.

(subprocess is used here only to get a second process that can be killed; Module 24 covers running programs properly.)

4. It Is try/finally With the Boilerplate Removed

Chapter 14.6 gave you a way to guarantee cleanup. with is that, specialised:

print("-- with --")
handle_a = None
try:
    with open("sim.log") as f:
        handle_a = f
        raise ValueError("boom")
except ValueError:
    pass
print("closed:", handle_a.closed)

print("-- try/finally --")
handle_b = open("sim.log")
try:
    raise ValueError("boom")
except ValueError:
    pass
finally:
    handle_b.close()
print("closed:", handle_b.closed)

Compile start

-- with --
closed: True
-- try/finally --
closed: True

Identical behaviour. The with version is three lines shorter, cannot be got wrong, and keeps the cleanup next to the acquisition instead of eight lines below it.

with is not a shortcut for close(). It is a shortcut for try/finally.

That distinction is why Chapter 14.6 Section 9 promised this chapter: once you have seen the general pattern, with stops being magic and becomes an abbreviation.

5. Leaks Accumulate

One leaked handle is harmless. Automation does not leak one:

import os

for i in range(200):
    with open(f"f{i}.log", "w") as f:
        f.write("x\n")

leaked = []


def scan_without_with(n):
    """Opens n files and dies before closing any of them."""
    for i in range(n):
        leaked.append(open(f"f{i}.log"))
        if i == 199:
            raise RuntimeError("parser failed on the last file")


try:
    scan_without_with(200)
except RuntimeError as e:
    print("caught:", e)

still_open = 0
for h in leaked:
    if not h.closed:
        still_open += 1

print("handles opened :", len(leaked))
print("still open     :", still_open)

still_open_after = 0
for h in leaked:
    h.close()
    if not h.closed:
        still_open_after += 1
print("after cleanup  :", still_open_after)

caught: parser failed on the last file
handles opened : 200
still open     : 200

Every handle the loop took is still held when the exception leaves the function. A regression sweep over ten thousand logs with this shape does not leak one descriptor — it leaks ten thousand.

6. Why a Shared Machine Makes This Real

Each process gets a limited number of descriptors, and the limit is a property of the machine, not of Python:

import resource

soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
print("this process may hold a finite number of descriptors")
print("soft limit is a positive integer:", soft > 0)
print("soft is at most hard:", soft <= hard or hard == resource.RLIM_INFINITY)
print("a 10,000-log sweep would need:", 10000, "if none were closed")

this process may hold a finite number of descriptors
soft limit is a positive integer: True
soft is at most hard: True
a 10,000-log sweep would need: 10000 if none were closed

The actual number differs between machines, which is why it is checked rather than printed — but it is typically in the low thousands for a user process, and a regression sweep can exceed it.

When it is exceeded you get OSError: [Errno 24] Too many open files, and the failure lands at an unrelated line — the next open() to run, which may be in somebody else's code. On a shared compute node the consequences are worse than a crash: a long-running process holding thousands of descriptors is consuming a resource other jobs need, and a file held open can block the directory it lives in from being cleaned up.

7. Why "Just Remember to Close It" Fails

The manual version is not wrong, it is unreliable, and it fails in a specific pattern:

def count_errors(path):
    f = open(path)
    total = 0
    for line in f:
        if "UVM_ERROR" in line:
            total += 1
    if total == 0:
        raise ValueError(f"no errors found in {path} -- suspicious")
    f.close()
    return total


print(count_errors("bad.log"))
print(count_errors("clean.log"))

UVM_ERROR drv.sv(88) bad handshake

UVM_INFO all good

1
Traceback (most recent call last):
  File "manual.py", line 14, in <module>
    print(count_errors("clean.log"))
          ~~~~~~~~~~~~^^^^^^^^^^^^^
  File "manual.py", line 8, in count_errors
    raise ValueError(f"no errors found in {path} -- suspicious")
ValueError: no errors found in clean.log -- suspicious

The close() is present, correct, and on the success path only. The author did remember — and the handle still leaks, because the raise on line 8 jumps over the close() on line 9.

Note the traceback's source line: it shows {path}, the f-string as written, not the interpolated text. Python prints the line from your file; only the message underneath is evaluated.

The one time you fail to close is the time something went wrong, which is precisely the time you are least able to reason about cleanup. That is the argument for making it structural rather than remembered.

8. Several Files at Once

One with can manage more than one resource — useful when reading a log and writing a summary in the same pass:

smoke PASS
cdc FAIL

with open("in.log") as src, open("out.log", "w") as dst:
    for line in src:
        if "FAIL" in line:
            dst.write(line)

with open("out.log") as f:
    print(repr(f.read()))

'cdc FAIL\n'

Both files are closed when the block ends, in reverse order, and both are closed if the body raises. Writing two nested with statements means the same thing.

9. Predict, Run, Observe

alpha
beta

with open("p.log") as f:
    first = f.readline()

print("A:", f.closed)
print("B:", repr(first))

try:
    print("C:", f.readline())
except ValueError as e:
    print("C: ValueError ->", e)

with open("p.log") as f2:
    pass
print("D:", f2.closed)

A: True
B: 'alpha\n'
C: ValueError -> I/O operation on closed file.

A — the name f still exists after the block. with closes the file; it does not delete the variable.

B — the data read inside the block is still perfectly usable outside it. Closing releases the descriptor, not your string.

C — reading a closed file raises ValueError, and the program stopped there, so D never printed. That is worth noticing: f.closed is how you ask, and touching a closed handle is an error, not an empty result.

The distinction matters because an exhausted handle (Chapter 15.4) returns '' silently while a closed one raises. Two different failures that look similar in a bug report.

10. Exercises

Exercise 1 — the guarantee. Raise an exception inside a with block and confirm closed is True afterwards. Then do it without with.

Exercise 2 — lost data. Write to a file in a child process that exits without closing, and read the result. Then add with and compare.

Exercise 3 — the equivalence. Write the try/finally version of a with block. Count the lines of each and name what the longer one can get wrong.

Exercise 4 — accumulate. Adapt Section 5 to open 50 files and report how many remain open after a failure. Then fix it with with and re-run.

Exercise 5 — the wrong line. Explain in two sentences why Too many open files usually appears somewhere other than the leaking loop.

Exercise 6 — the success path. Reproduce Section 7's manual.py. Move the close() so the handle is released on both paths without using with, and say which Chapter 14.6 clause you needed.

Exercise 7 — two at once. Open an input and an output in one with and copy only the lines containing FAIL.

Exercise 8 — closed versus exhausted. Produce both: a read that returns '' and a read that raises ValueError. Say what each tells you about the handle.

11. Summary

• An open file holds a file descriptor, a finite per-process resource owned by the operating system.
• with closes the file when the block ends, whether it finished normally or raised.
• with is try/finally, abbreviated — not a shortcut for close().
• Writes are buffered, so a missing close can mean the data never reaches the disk at all.
• A half-written file is valid text that stops early, and the next tool will read it happily.
• Leaks accumulate: a sweep over ten thousand logs leaks ten thousand handles.
• Exceeding the descriptor limit raises OSError: Too many open files at an unrelated line, far from the loop that caused it.
• "Remember to close" fails exactly when it matters, because the thing that skipped the close() was an exception.
• One with can hold several files, closed in reverse order.
• A closed handle raises on read; an exhausted one returns ''.

The next chapter writes files — where the difference between two mode letters is the difference between a report and an erased one.