Writing Files: Filelists, Reports and Generated Text

Part I · Module 15 · Chapter 15.6 · Writing Files: Filelists, Reports and Generated Text

1. Opening for Writing

with open("report.txt", "w") as f:
    f.write("smoke PASS 182.4\n")
    f.write("cdc FAIL 204.1\n")

with open("report.txt") as f:
    print(f.read(), end="")

smoke PASS 182.4
cdc FAIL 204.1

"w" means write text. The file is created if it does not exist, and the with block flushes and closes it — which Chapter 15.5 Section 3 showed is not optional.

2. "w" Empties the File Immediately

The detail that costs people a night's results:

with open("report.txt", "w") as f:
    f.write("last night's results\n")

with open("report.txt") as f:
    print("before:", repr(f.read()))

f = open("report.txt", "w")
with open("report.txt") as probe:
    print("after open(), before any write:", repr(probe.read()))
f.close()

before: "last night's results\n"
after open(), before any write: ''

The file was emptied by open() itself. No write call had run. The truncation is part of opening in "w" mode, so a script that opens its output file and then spends two minutes computing has already destroyed the previous contents — and if it crashes during those two minutes, that is the end state.

This is the single strongest argument for Section 7's pattern.

3. Append, and the Other Modes

with open("log.txt", "w") as f:
    f.write("first\n")

with open("log.txt", "w") as f:
    f.write("second\n")
with open("log.txt") as f:
    print("after second w:", repr(f.read()))

with open("log.txt", "a") as f:
    f.write("third\n")
with open("log.txt") as f:
    print("after a       :", repr(f.read()))

try:
    with open("log.txt", "x") as f:
        f.write("fourth\n")
except FileExistsError as e:
    print("x mode        :", type(e).__name__)

after second w: 'second\n'
after a       : 'second\nthird\n'
x mode        : FileExistsError

• "w" replaces. Use it for a report that is regenerated each run.
• "a" appends, creating the file if needed. Use it for an accumulating log.
• "x" creates but refuses to overwrite. Use it when overwriting would be a bug — a run directory that should be fresh.

"x" is underused and is the cheapest possible guard against clobbering a result you meant to keep.

4. write() Does Not Add a Newline

with open("a.txt", "w") as f:
    f.write("smoke")
    f.write("cdc")
    f.write("burst")

with open("a.txt") as f:
    print("A:", repr(f.read()))

with open("b.txt", "w") as f:
    f.write("smoke\n")
    f.write("cdc\n")

with open("b.txt") as f:
    print("B:", repr(f.read()))

A: 'smokecdcburst'
B: 'smoke\ncdc\n'

A is one line, not three. f.write() writes exactly the characters you give it — no separator, no conversion. Three names became one word, and nothing raised.

It also refuses non-strings, which catches the other half of the mistake:

with open("c.txt", "w") as f:
    f.write(182.4)

Traceback (most recent call last):
  File "nonstr.py", line 2, in <module>
    f.write(182.4)
    ~~~~~~~^^^^^^^
TypeError: write() argument must be str, not float

5. print(..., file=f) Is Usually What You Want

print already knows how to add a newline and convert values to text, and it accepts a destination:

rows = [("smoke", "PASS", 182.4), ("cdc", "FAIL", 204.1)]

with open("report.txt", "w") as f:
    print("test status runtime_s", file=f)
    for name, status, runtime in rows:
        print(name, status, runtime, file=f)

with open("report.txt") as f:
    print(f.read(), end="")

test status runtime_s
smoke PASS 182.4
cdc FAIL 204.1

Note 182.4 was written without a conversion — print called str() for you, where f.write would have raised.

Use print(..., file=f) for line-oriented output. Use f.write() when you need exact control over every character.

The canonical form for this chapter is the print one:

with open("files.f", "w") as f:
    for src in sources:
        print(src, file=f)

6. A Filelist, Written Deterministically

That canonical snippet is a compile filelist — the file a simulator reads to learn which sources to compile. It has a requirement most output does not:

SOURCES = {
    "rtl/alu.sv",
    "rtl/regfile.sv",
    "rtl/decoder.sv",
    "tb/env.sv",
}

with open("files.f", "w") as f:
    for src in sorted(SOURCES):
        print(src, file=f)

with open("files.f") as f:
    print(f.read(), end="")

rtl/alu.sv
rtl/decoder.sv
rtl/regfile.sv
tb/env.sv

sorted() is doing real work. SOURCES is a set, and Chapter 7.2 established that a set has no reliable order — so without sorted() two runs of the same script on the same commit can emit the same filenames in different orders.

That matters more than it looks:

• a tool's compile order can affect which definition of a macro wins;
• a diff between two runs becomes unreadable;
• a build cache keyed on the filelist's contents misses every time.

Deterministic output is a property you have to choose. Module 18 Chapter 18.7 covers filelists properly; the habit to take from here is sorted() around anything whose order is not already meaningful.

7. Do Not Open the Real Path Until You Have the Content

Section 2 showed that "w" destroys the old file at open(). Combine that with a script that can fail halfway and you get the canonical purpose's warning — a corrupt file the next tool reads happily:

ROWS = [("smoke", 182.4), ("cdc", 204.1), ("reset", None)]

with open("report.txt", "w") as f:
    print("test runtime_s", file=f)
    for name, runtime in ROWS:
        print(f"{name} {runtime:.1f}", file=f)

Traceback (most recent call last):
  File "corrupt.py", line 6, in <module>
    print(f"{name} {runtime:.1f}", file=f)
                   ^^^^^^^^^^^^^
TypeError: unsupported format string passed to NoneType.__format__

The script died on the third row. What is on disk now?

ROWS = [("smoke", 182.4), ("cdc", 204.1), ("reset", None)]

try:
    with open("report.txt", "w") as f:
        print("test runtime_s", file=f)
        for name, runtime in ROWS:
            print(f"{name} {runtime:.1f}", file=f)
except TypeError as e:
    print("failed:", e)

print("--- report.txt now contains ---")
with open("report.txt") as f:
    print(f.read(), end="")
print("--- end ---")

failed: unsupported format string passed to NoneType.__format__
--- report.txt now contains ---
test runtime_s
smoke 182.4
cdc 204.1
--- end ---

A header and two of three rows. It is valid, well-formed, and wrong — a reader has no way to tell it apart from a run that genuinely had two tests. The with block did its job perfectly: it flushed and closed, which is exactly what preserved the partial output.

The fix is to make the real path appear only when the content is complete:

import os

ROWS = [("smoke", 182.4), ("cdc", 204.1), ("reset", None)]


def write_report(path, rows):
    tmp = path + ".tmp"
    with open(tmp, "w") as f:
        print("test runtime_s", file=f)
        for name, runtime in rows:
            print(f"{name} {runtime:.1f}", file=f)
    os.replace(tmp, path)


with open("report.txt", "w") as f:
    print("last night's good report", file=f)

try:
    write_report("report.txt", ROWS)
except TypeError as e:
    print("failed:", e)

with open("report.txt") as f:
    print("report.txt still contains:", repr(f.read()))
print("tmp left behind:", os.path.exists("report.txt.tmp"))

failed: unsupported format string passed to NoneType.__format__
report.txt still contains: "last night's good report\n"
tmp left behind: True

The previous report survived. The failure happened while writing report.txt.tmp, and os.replace — the line that would have published it — never ran.

Three properties make this work:

1. the real path is never opened in "w" until the content is known good;
2. os.replace is atomic on the same filesystem, so a reader sees either the old file or the new one, never a half-written one;
3. a failure leaves a .tmp file, which is visible evidence rather than silent corruption.

8. The Trailing Newline Is Part of the Format

Two ways to write the same three names produce two different files, and the difference is one byte at the end:

rows = ["rtl/alu.sv", "rtl/decoder.sv"]

with open("a.f", "w") as f:
    f.write("\n".join(rows))

with open("b.f", "w") as f:
    for r in rows:
        print(r, file=f)

for name in ["a.f", "b.f"]:
    with open(name) as f:
        content = f.read()
    with open(name) as f:
        lines = f.readlines()
    print(f"{name}: {content!r}")
    print(f"      {len(lines)} lines, last = {lines[-1]!r}")

a.f: 'rtl/alu.sv\nrtl/decoder.sv'
      2 lines, last = 'rtl/decoder.sv'
b.f: 'rtl/alu.sv\nrtl/decoder.sv\n'
      2 lines, last = 'rtl/decoder.sv\n'

"\n".join(rows) puts a separator between items, so the last line is unterminated. print(..., file=f) terminates every line, including the last.

Both files have two lines by Python's reckoning, so readlines() and iteration agree. The disagreement is elsewhere:

• many Unix tools treat a file whose last line is unterminated as malformed, and some silently drop that line;
• appending to a.f later joins the new content onto rtl/decoder.sv, producing rtl/decoder.svrtl/next.sv;
• a diff between the two files shows a change on the last line even when the names are identical.

Chapter 15.3 Section 4 met this from the reading side — the final line of a file that does not end with a newline has a different shape from every other line. This is the same fact, now under your control.

Terminate every line, including the last. print(..., file=f) does it for you; "\n".join(...) does not.

print also takes sep, which makes a delimited file a one-liner:

with open("c.csv", "w") as f:
    print("test", "status", "runtime_s", sep=",", file=f)
    print("smoke", "PASS", 182.4, sep=",", file=f)

with open("c.csv") as f:
    print(repr(f.read()))

'test,status,runtime_s\nsmoke,PASS,182.4\n'

That is enough for a file you control. It is not enough for CSV in general, because a field containing a comma or a quote needs escaping — which is why Module 21 uses the csv module rather than this.

9. Predict, Run, Observe

with open("p.txt", "w") as f:
    f.write("a")
    print("b", file=f)
    f.write("c")
    print(42, file=f)

with open("p.txt") as f:
    print("A:", repr(f.read()))

with open("p.txt", "a") as f:
    print("d", file=f)

with open("p.txt") as f:
    print("B:", repr(f.read()))

with open("p.txt", "w") as f:
    pass

with open("p.txt") as f:
    print("C:", repr(f.read()))

A: 'ab\nc42\n'
B: 'ab\nc42\nd\n'
C: ''

A — write("a") added no separator, so print("b", file=f) continued the same line and then ended it. write("c") started the next line and print(42) appended to it, converting the int. Four calls, two lines.

B — "a" reopened at the end and added one line. Nothing was lost.

C is the one to remember. A with block that opens in "w" and writes nothing leaves an empty file. The truncation happened at open(); the empty body never undid it. That is Section 2 as a one-liner, and it is how a report becomes zero bytes.

10. Exercises

Exercise 1 — three rows. Write a three-line report with f.write() and then with print(..., file=f). Compare the code and say which you would keep.

Exercise 2 — the missing newline. Write three values with f.write() and no separators. Print the repr of the result and explain what a reader would see.

Exercise 3 — truncation timing. Open a non-empty file in "w", read it from a second handle before writing anything, and record what you find.

Exercise 4 — the three modes. For "w", "a" and "x", predict the result on a file that exists and one that does not. Verify all six.

Exercise 5 — determinism. Write a filelist from a set without sorted(). Run it several times and compare. Then add sorted() and repeat.

Exercise 6 — the partial report. Reproduce Section 7's corrupt.py and inspect the file afterwards. Say what a downstream tool would conclude.

Exercise 7 — atomic publish. Implement write_report with the temp-and- rename pattern. Prove the old file survives a failure.

Exercise 8 — the empty file. Explain in two sentences how a report can end up zero bytes without any write failing, and name the guard you would add.

11. Summary

• "w" truncates the file at open(), before any write — so opening early destroys the old contents even if you never write.
• "a" appends and "x" refuses to overwrite; "x" is a cheap guard against clobbering.
• f.write() adds nothing — no newline, no conversion — and raises on a non-string.
• print(..., file=f) adds the newline and converts, and is the right default for line-oriented output.
• Order your output deliberately — sorted() around anything from a set, or two runs of the same commit produce different files.
• A crash mid-write leaves a valid, incomplete file that the next tool reads without complaint.
• with preserves that partial output — closing correctly is what makes it readable, not what makes it right.
• Write to a temporary name and os.replace into place, so the real path only ever holds complete content.
• A failed atomic write leaves a .tmp file, which is evidence.
• An empty with block in "w" mode produces a zero-byte file.
• Terminate every line, including the last — "\n".join() separates where print(..., file=f) terminates, and an unterminated final line breaks appends, diffs and some Unix tools.
• print takes sep, which is enough for a delimited file you control but not for general CSV, where fields need escaping.

The next chapter returns to reading, with the one pattern that makes a multi-gigabyte log tractable.