Why Programs Use Files

Part I · Module 15 · Chapter 15.1 · Why Programs Use Files

1. Two Things a Variable Cannot Do

# a variable disappears when the script ends
# a file is how one program hands work to the next

Those two comments are the whole module in advance, and they are separate claims:

1. Persistence — a variable's lifetime is the run. A file's is not.
2. Handoff — a variable can only hold what this program computed. A file can hold what another program produced.

The first is the obvious one. The second is the one that matters in automation, because in a real flow your script almost never generates its own input.

2. A Counter That Forgets

Here is a script that counts how many times it has run, using a variable:

runs = 0


def record_run():
    global runs
    runs += 1
    return runs


print("this process, call 1:", record_run())
print("this process, call 2:", record_run())
print("this process, call 3:", record_run())

this process, call 1: 1
this process, call 2: 2
this process, call 3: 3

Within one process it works. Start the process again and runs is 0 again — the module-level assignment runs afresh, exactly as Chapter 13.1 Section 6 described for imports. Nothing a program holds in memory survives its own exit.

To count runs across invocations you need somewhere outside the process to put the number, and that somewhere is a file. Chapter 15.6 writes one.

3. Data Your Program Did Not Produce

The deeper point. Consider what a regression script actually works with:

LOG_LINES = [
    "Chronologic VCS simulator copyright 1991-2024",
    "UVM_INFO @ 0: reporter [RNTST] Running test smoke_test...",
    "UVM_ERROR drv.sv(88) @ 1420: uvm_test_top.env.agt.drv [DRV] bad handshake",
    "UVM_INFO @ 9180: reporter [FINISH] *** TEST FAILED ***",
]


def verdict(lines):
    for line in lines:
        if "TEST FAILED" in line:
            return "FAIL"
        if "TEST PASSED" in line:
            return "PASS"
    return "NO_VERDICT"


print(verdict(LOG_LINES))
print(len(LOG_LINES), "lines examined")

FAIL
4 lines examined

That function is the heart of a real regression summariser, and the list it reads is fake. In production those four strings are not in the script — they are in a 400 MB file that a simulator wrote forty minutes ago on a different machine. The logic is identical. The only missing piece is how the lines get from the disk into the list, which is Chapter 15.2.

4. The File Is the Interface

Read what the arrows do not say. There is no call. The simulator does not invoke your script and your script does not invoke the simulator. Neither knows the other exists. They are connected only by an agreement about what is in the file and what it means.

That agreement is an interface, and it has a property no function signature has: you did not design it, and you cannot change it.

5. Two Programs, One File

The handoff in miniature — a producer and a consumer that share only a format:

def write_report(results):
    """The producer. In a real flow this is a different program entirely."""
    lines = []
    for name, status, runtime in results:
        lines.append(f"{name} {status} {runtime}")
    return "\n".join(lines)


def read_report(text):
    """The consumer. It knows only the format, not the producer."""
    failures = []
    for line in text.split("\n"):
        fields = line.split()
        if fields[1] == "FAIL":
            failures.append(fields[0])
    return failures


REPORT = write_report([
    ("smoke", "PASS", "182.4"),
    ("cdc", "FAIL", "204.1"),
    ("burst", "PASS", "199.8"),
])
print(REPORT)
print("---")
print("failures:", read_report(REPORT))

smoke PASS 182.4
cdc FAIL 204.1
burst PASS 199.8
---
failures: ['cdc']

The string in the middle is standing in for the file. read_report knows nothing about write_report — not its name, not its arguments, not its language. It knows that field 0 is a name and field 1 is a status, and that is the whole contract.

Replace write_report with a commercial simulator and nothing about read_report changes. That substitution is what the rest of Part I is building toward.

6. You Are a Guest in Someone Else's Format

Because you did not design the format, three things are true that are not true of ordinary function calls:

It can change without warning. A tool upgrade can add a column, rename a message, or start emitting a banner. Your parser was not consulted.

It has no type checking. A function that expects an int gets an int or raises. A file gives you text, and "N/A" where a number should be is not an error until you make it one — Chapter 14.3's ValueError, arriving from outside.

It can be incomplete. A function either returns or raises. A file can be half-written, because the thing writing it was killed:

COMPLETE = "Compile start\nUVM_INFO running\nSimulation complete\n"
TRUNCATED = "Compile start\nUVM_INFO runn"


def finished(text):
    return text.endswith("Simulation complete\n")


for label, text in [("complete", COMPLETE), ("truncated", TRUNCATED)]:
    print(f"{label:<10} lines={len(text.splitlines()):<3} finished={finished(text)}")

complete   lines=3   finished=True
truncated  lines=2   finished=False

The truncated version is not corrupt in any way a computer can detect. It is valid text, it has lines, every line is readable — it simply stops. A parser that only looks for UVM_ERROR would call that run clean.

7. What a File Actually Is, For Now

For the rest of this module, a file is:

• a name — a path, which is text your operating system understands;
• a sequence of bytes on a disk that persists after your program exits;
• decoded into text when you read it, which is Chapter 15.3's subject and the source of a whole class of surprise.

Note what is not in that list: structure. A file is not a list of lines, not a table and not a dictionary. Lines are a convention — a particular byte, written between records, that everyone agreed to treat as a separator. Chapter 15.3 shows that byte directly, and it explains most of what beginners find strange about reading files.

8. Predict, Run, Observe

RAW = "smoke PASS 182.4\ncdc FAIL 204.1\n"

print("A:", len(RAW))
print("B:", RAW.splitlines())
print("C:", RAW.split("\n"))
print("D:", len(RAW.splitlines()), "vs", len(RAW.split("\n")))

A: 32
B: ['smoke PASS 182.4', 'cdc FAIL 204.1']
C: ['smoke PASS 182.4', 'cdc FAIL 204.1', '']
D: 2 vs 3

A — 32 characters for two lines of 16 and 14 printable characters. The extra two are the separators, and they are as real as the letters.

B and C differ by one empty string. splitlines() understands that a trailing separator ends the last line rather than starting a new one. split("\n") is a general-purpose string operation that knows nothing about files, so a trailing separator leaves an empty final field.

D is the bug you will write at some point: a loop over split("\n") that processes a phantom blank line and raises IndexError on fields[1] — Chapter 14.3's exact shape, now with a cause.

The rule: use splitlines() on file text, or iterate the file itself, which Chapter 15.7 shows is better still.

9. Exercises

Exercise 1 — the forgetting counter. Run Section 2's script twice in two separate processes. Record what the first printed line says each time and explain it in one sentence.

Exercise 2 — swap the source. Take Section 3's verdict function and change only the LOG_LINES list to describe a passing run. Confirm the function is untouched, and say what that tells you about where the real work will be.

Exercise 3 — the contract. Write down, in one sentence each, the three things read_report in Section 5 assumes about the text it receives. For each, say what a tool upgrade could do to break it.

Exercise 4 — break it. Add a header line to REPORT and run read_report again. Record the error and say which of your three assumptions it violated.

Exercise 5 — truncation. Write a finished() check for a format of your own invention. Then write a log that is truncated in a way your check does not catch, and say what you would add.

Exercise 6 — splitlines. Build a string with a trailing newline and one without. Run both through splitlines() and split("\n") and tabulate the four results.

Exercise 7 — the phantom line. Write a loop over split("\n") that raises IndexError on the trailing empty string. Fix it two ways: with splitlines(), and with a guard.

Exercise 8 — name the interfaces. List three files a VLSI flow passes between programs. For each, name the producer, the consumer, and one thing that would break the consumer if it changed.

10. Summary

• A variable's lifetime is the run; a file's is not.
• The more important property is handoff: a file holds data that another program produced, possibly in another language, by another vendor.
• The file format is the interface, and unlike a function signature you did not design it and cannot change it.
• The two programs never meet — no call, no import, no shared code; only an agreement about content.
• A format can change without warning, has no type checking, and can be incomplete.
• A truncated file is valid text that simply stops, which no language feature detects for you.
• A file is a name, a sequence of bytes, and a decoding — structure is a convention layered on top.
• Lines are a convention: a separator byte everyone agreed on.
• Use splitlines() on file text, not split("\n"), or you process a phantom final line.

The next chapter opens a real file, reads it, and closes it safely — three things that are one statement in modern Python.