Text Mode, Encoding and the Invisible Newline

Part I · Module 15 · Chapter 15.3 · Text Mode, Encoding and the Invisible Newline

1. Bytes In, Text Out

Chapter 15.2 called f.read() and got a str. A disk does not store str objects — it stores bytes. The conversion is called decoding, and open() does it by default, which is what "text mode" means.

You can see both sides:

with open("sim.log", "wb") as f:
    f.write(b"Compile start\n")

with open("sim.log") as f:
    as_text = f.read()

with open("sim.log", "rb") as f:
    as_bytes = f.read()

print("text :", type(as_text).__name__, repr(as_text))
print("bytes:", type(as_bytes).__name__, repr(as_bytes))

text : str 'Compile start\n'
bytes: bytes b'Compile start\n'

The b prefix is the difference. "rb" means read binary — no decoding, no newline handling, just the bytes. Text mode is the default and is what you want for logs, reports and filelists. Binary mode is for images, archives and waveform databases, which this module does not cover.

2. The Mode String

open(path, mode) takes a short string. The ones that matter in Part I:

open(path) and open(path, "r") are identical. Chapter 15.6 covers the writing modes, where the difference between "w" and "a" is the difference between a report and a disaster.

3. One Line, Under a Microscope

The canonical form for inspecting what you actually received:

Compile start
UVM_INFO top.sv(42) smoke begin

with open("sim.log", encoding="utf-8") as f:
    first = f.readline()

print(repr(first))

'Compile start\n'

Two things to take from that output.

readline() returns one line. It is one of three reading methods and Chapter 15.4 is about choosing between them; it appears here because this chapter needs exactly one line to examine.

repr() is how you see what is really there. print(first) would show Compile start and tell you nothing about the trailing character. In file code, reach for repr whenever something does not match.

4. The Newline Is Part of the String

This is the defect everyone writes once:

smoke PASS
cdc FAIL
burst PASS

with open("status.log") as f:
    lines = f.read().splitlines()

with open("status.log") as f:
    raw_first = f.readline()

print("A:", repr(raw_first))
print("B:", raw_first == "smoke PASS")
print("C:", raw_first.strip() == "smoke PASS")
print("D:", repr(lines[0]), lines[0] == "smoke PASS")

A: 'smoke PASS\n'
B: False
C: True
D: 'smoke PASS' True

B is False and that is the whole problem. raw_first is eleven characters; the literal is ten. The comparison is correct, the data is correct, and the answer is wrong because an invisible character is present.

Two fixes, both shown:

• .strip() removes leading and trailing whitespace including the newline;
• .splitlines() never includes it in the first place.

Every line from readline(), readlines() or iteration carries its newline. Lines from splitlines() do not.

The last line is the exception that catches the other half of people — if the file does not end with a newline, its final line has none:

with open("a.log", "w") as f:
    f.write("first\nsecond")

with open("a.log") as f:
    lines = f.readlines()

print(lines)

['first\n', 'second']

Two lines, two different shapes. Code that assumes line[:-1] strips the newline will silently remove the d from second. Use .strip(), or better .rstrip("\n"), which removes only what you meant.

5. Windows Line Endings Arrive Translated

A file written on Windows separates lines with two bytes, \r\n. Python hides this:

with open("crlf.log", "wb") as f:
    f.write(b"Compile start\r\nSimulation complete\r\n")

with open("crlf.log", "rb") as f:
    print("A raw bytes :", repr(f.read()))

with open("crlf.log") as f:
    print("B text mode :", repr(f.read()))

with open("crlf.log", newline="") as f:
    print("C untranslated:", repr(f.read()))

A raw bytes : b'Compile start\r\nSimulation complete\r\n'
B text mode : 'Compile start\nSimulation complete\n'
C untranslated: 'Compile start\r\nSimulation complete\r\n'

B is the useful default. Text mode translates \r\n to \n on the way in, so a log produced by a Windows tool and parsed on a Linux farm just works. This is called universal newlines.

C shows the translation is a choice. newline="" turns it off, which you want only when the exact bytes matter — rewriting a file without changing its line endings, for instance.

The practical consequence: if a stray \r ever appears in your parsed data, you read the file in binary mode or with newline="". In ordinary text mode it cannot reach you. And note that .strip() removes \r as well, which is why it is the more robust habit than .rstrip("\n") when the source is unknown.

6. When the Encoding Matters

The default encoding on a modern system is UTF-8, which handles almost everything. Almost:

with open("vendor.log", "wb") as f:
    f.write("Vref café ± 5mV\n".encode("latin-1"))

with open("vendor.log", encoding="utf-8") as f:
    print(f.read())

Traceback (most recent call last):
  File "decode.py", line 5, in <module>
    print(f.read())
          ~~~~~~^^
  File "<frozen codecs>", line 325, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 8: invalid continuation byte

A vendor banner containing one accented character, written by a tool that uses an older encoding, and the read fails on byte 8. Nothing about the surrounding log is wrong.

The error is specific and worth reading: byte 0xe9 at position 8 is not valid UTF-8.

Note the deepest frame: File "<frozen codecs>" — inside Python itself, not your file. This is exactly the shape Chapter 14.2 Section 6 described, and the rule there applies unchanged: the lowest frame you wrote is decode.py line 5, and that is where the decision to decode as UTF-8 was made.

Three responses, in order of preference:

with open("vendor.log", "wb") as f:
    f.write("Vref café ± 5mV\n".encode("latin-1"))

with open("vendor.log", encoding="latin-1") as f:
    print("A:", repr(f.read()))

with open("vendor.log", encoding="utf-8", errors="replace") as f:
    print("B:", repr(f.read()))

A: 'Vref café ± 5mV\n'
B: 'Vref caf� � 5mV\n'

A — name the right encoding. If you know the tool writes latin-1, say so. This is correct and lossless.

B — errors="replace" substitutes �, the replacement character, for anything undecodable. The file reads, and the damaged positions are visible. Use it when you only need to scan for ASCII markers like UVM_ERROR and do not care about a mangled banner.

The third response is to not guess: if the encoding is genuinely unknown and the content matters, read binary and decide explicitly.

7. Predict, Run, Observe

with open("p.log", "w") as f:
    f.write("alpha\nbeta")

with open("p.log") as f:
    lines = f.readlines()

print("A:", lines)
print("B:", [len(x) for x in lines])
print("C:", [x.strip() for x in lines])
print("D:", lines[0][:-1], "|", lines[1][:-1])
with open("p.log") as f:
    print("E:", f.read().splitlines())

A: ['alpha\n', 'beta']
B: [6, 4]
C: ['alpha', 'beta']
D: alpha | bet
E: ['alpha', 'beta']

A and B — the first line is six characters, the second four. The file has no trailing newline, so only one of the two lines carries a separator.

C — .strip() handles both correctly, because it removes a newline if there is one and does nothing if there is not.

D is the bug. [:-1] removes the last character unconditionally, which is the newline on line 1 and the letter a on line 2. bet is wrong and nothing raised.

E — splitlines() gives the same answer as .strip() applied to each line, in one call, which is why Chapter 15.1 recommended it.

8. Exercises

Exercise 1 — both sides. Read the same file in text mode and binary mode. Print the type and repr of each and state the one difference.

Exercise 2 — the invisible character. Read a line with readline() and compare it to the string literal you expect. Make the comparison fail, then fix it twice — with .strip() and with splitlines().

Exercise 3 — the last line. Create a file with no trailing newline. Print readlines() and account for why the two entries have different shapes.

Exercise 4 — [:-1] damage. Reproduce Section 7's D and say why no error was raised. Write the one-line fix.

Exercise 5 — CRLF. Write a file with \r\n in binary mode. Read it three ways — binary, text, and newline="" — and tabulate what you get.

Exercise 6 — decode failure. Write a latin-1 byte into a file and read it as UTF-8. Record the error's byte value and position, then read it correctly.

Exercise 7 — replace. Read the same file with errors="replace" and say when you would accept that result and when you would not.

Exercise 8 — the mode table. For each mode in Section 2, predict what happens when the file exists and when it does not. Verify the two you are least sure about.

9. Summary

• Text mode is a decode: open() returns str; "rb" returns bytes.
• The mode string chooses read, write, append or exclusive-create, and whether the result is text or bytes.
• repr() is the debugging tool for file text — print hides the separators.
• Every line from readline, readlines or iteration carries its newline; lines from splitlines() do not.
• A comparison against a literal fails when the newline is still attached, and nothing raises.
• The last line has no newline when the file does not end with one, so [:-1] silently eats a real character.
• .strip() is the robust habit; it also removes a stray \r.
• Text mode translates \r\n to \n, so Windows-produced logs just work; newline="" turns that off.
• The default encoding depends on the machine, so be explicit with encoding="utf-8" in anything shared.
• UnicodeDecodeError names the byte and position; errors="replace" lets you scan a file whose banner you do not care about.

The next chapter settles the choice this one borrowed from: read, readline and readlines, and which to use on a log you cannot fit in memory.