Reading Your Own Script Line by Line

Part II · Module 20 · Chapter 20.3 · Reading Your Own Script Line by Line

1. The Script, Numbered

UVM_INFO @ 0: reporter [RNTST] Running test axi_smoke_test...
UVM_ERROR drv.sv(88) @ 141900: uvm_test_top.env.agt.drv [WR] bad handshake
UVM_ERROR drv.sv(104) @ 180400: uvm_test_top.env.agt.drv [WR] timeout
UVM_INFO seq.sv(140) @ 200000: uvm_test_top.env.agt.seqr [SEQ] done

errors = 0                                        # 1
with open("run.log", encoding="utf-8") as handle:  # 2
    for line in handle:                            # 3
        if "UVM_ERROR" in line:                    # 4
            errors += 1                            # 5
print("FAIL" if errors else "PASS", errors)        # 6

FAIL 2

Six lines. Chapter 20.1's four parts map onto them: line 2 is input, lines 1 and 3 to 5 are processing, the if errors in line 6 is the decision and the print is the output. The rest of this chapter is what each line is actually doing.

2. Line 1 — errors = 0

It creates the name and gives it a starting value, and both halves matter.

LINES = ["UVM_ERROR drv.sv(88) @ 141900: bad handshake"]

for line in LINES:
    if "UVM_ERROR" in line:
        errors += 1

print(errors)

Traceback (most recent call last):
  File "no_init.py", line 5, in <module>
    errors += 1
    ^^^^^^
NameError: name 'errors' is not defined

errors += 1 means read errors, add one, store it back — so the name has to already exist. This is the one mutation in the chapter that raises, and it raises on the first error line rather than at the end, which makes it the easiest of the six to diagnose.

The starting value has to be 0 and not None or "", because line 5 adds to it. It is also what makes a clean log report zero rather than reporting nothing: a log with no errors never reaches line 5, and errors is still the 0 from line 1.

3. Line 2 — with open("run.log", encoding="utf-8") as handle

Three separate decisions on one line.

UVM_INFO @ 0: reporter [RNTST] Running test axi_smoke_test...
UVM_ERROR drv.sv(88) @ 141900: uvm_test_top.env.agt.drv [WR] bad handshake
UVM_ERROR drv.sv(104) @ 180400: uvm_test_top.env.agt.drv [WR] timeout
UVM_INFO seq.sv(140) @ 200000: uvm_test_top.env.agt.seqr [SEQ] done

with open("run.log", encoding="utf-8") as handle:
    print("type    :", type(handle).__name__)
    print("name    :", handle.name)
    print("mode    :", handle.mode)
    print("encoding:", handle.encoding)

type    : TextIOWrapper
name    : run.log
mode    : r
encoding: utf-8

open(...) returns a TextIOWrapper — not the text of the file, and not a list. It is an object that knows how to produce the file's content on demand, which is why line 3 can loop over it without the file ever being fully read.

mode defaulted to "r" — reading, in text mode. Chapter 15.3 covered the alternative.

encoding="utf-8" was a choice, and leaving it out does not fail here. It falls back to whatever the machine's locale says, which is the defect Chapter 15.3 described: the same script reads the same log differently on two machines, and the one that breaks is usually the compute farm rather than your laptop. Name the encoding.

as handle is the name, and it is the reason with works at all — the file object has somewhere to live for the duration of the block, so Python can close it at the end rather than waiting for the garbage collector.

UVM_INFO @ 0: reporter [RNTST] Running test axi_smoke_test...
UVM_ERROR drv.sv(88) @ 141900: uvm_test_top.env.agt.drv [WR] bad handshake
UVM_ERROR drv.sv(104) @ 180400: uvm_test_top.env.agt.drv [WR] timeout
UVM_INFO seq.sv(140) @ 200000: uvm_test_top.env.agt.seqr [SEQ] done

with open("run.log", encoding="utf-8") as handle:
    pass

print("closed:", handle.closed)

try:
    next(iter(handle))
except ValueError as exc:
    print("ValueError:", exc)

closed: True
ValueError: I/O operation on closed file.

The name outlives the block; the open file does not. That is with doing its one job.

4. Line 3 — for line in handle

The canonical annotation for this chapter is this line and its comment:

for line in open("sim.log"):   # one line at a time, not the whole file

What arrives in line each time is a string ending in a newline:

UVM_INFO @ 0: reporter [RNTST] Running test axi_smoke_test...
UVM_ERROR drv.sv(88) @ 141900: uvm_test_top.env.agt.drv [WR] bad handshake
UVM_ERROR drv.sv(104) @ 180400: uvm_test_top.env.agt.drv [WR] timeout
UVM_INFO seq.sv(140) @ 200000: uvm_test_top.env.agt.seqr [SEQ] done

with open("run.log", encoding="utf-8") as handle:
    for line in handle:
        print(repr(line))
        break

'UVM_INFO @ 0: reporter [RNTST] Running test axi_smoke_test...\n'

The \n is still attached. It does not matter to "UVM_ERROR" in line, and it matters enormously the moment you compare a line to something, print it, or use the $ anchor from Chapter 19.7 — which is why every later chapter in this module calls .rstrip() before storing a line.

And one line at a time is literal. The file is read in chunks as the loop asks for them; at no point does the whole log exist in memory. Chapter 15.7 established this; here it is the difference between a script that works on a 20 GB log and one that is killed by the job scheduler.

5. Lines 4 and 5 — the Test and the Count

if "UVM_ERROR" in line is Chapter 4.6's membership test: does this substring occur anywhere in this string. It is the cheapest possible check and Chapter 20.2 Section 6 showed exactly where it is too cheap.

errors += 1 runs only for the lines that pass the test, which is the entire mechanism by which the number at the end means anything.

Note what the loop does not keep. Each line exists for one iteration and is then replaced by the next one. Nothing accumulates except the integer:

UVM_INFO @ 0: reporter [RNTST] Running test axi_smoke_test...
UVM_ERROR drv.sv(88) @ 141900: uvm_test_top.env.agt.drv [WR] bad handshake
UVM_ERROR drv.sv(104) @ 180400: uvm_test_top.env.agt.drv [WR] timeout
UVM_INFO seq.sv(140) @ 200000: uvm_test_top.env.agt.seqr [SEQ] done

with open("run.log", encoding="utf-8") as handle:
    for line in handle:
        pass

print("after the loop:", repr(line))

after the loop: 'UVM_INFO seq.sv(140) @ 200000: uvm_test_top.env.agt.seqr [SEQ] done\n'

The loop variable survives, holding the last line — Chapter 10.12's rule. That is occasionally useful (Chapter 20.12 uses the tail of a log to detect a timeout) and more often a trap, because line after a loop looks like it means "the line we were interested in" and means "whichever one happened to be last".

6. A File Object Is Consumed Once

This is the fact the rest of the module depends on:

UVM_INFO @ 0: reporter [RNTST] Running test axi_smoke_test...
UVM_ERROR drv.sv(88) @ 141900: uvm_test_top.env.agt.drv [WR] bad handshake
UVM_ERROR drv.sv(104) @ 180400: uvm_test_top.env.agt.drv [WR] timeout
UVM_INFO seq.sv(140) @ 200000: uvm_test_top.env.agt.seqr [SEQ] done

with open("run.log", encoding="utf-8") as handle:
    first_pass = sum(1 for _ in handle)
    second_pass = sum(1 for _ in handle)

print("first pass :", first_pass)
print("second pass:", second_pass)

first pass : 4
second pass: 0

The second count is zero and nothing went wrong. A file object is an iterator: it has a position, the first loop ran it to the end, and there is nothing after the end. Opening the file again, or calling handle.seek(0), is what gives you a second pass.

That is the price of not holding the whole file in memory, and it is the right trade — but it means a script that needs two passes over a log must decide, deliberately, which one it is paying for. Chapter 20.13 builds generators on exactly this behaviour, and Chapter 20.14 is about when to stop and build a list instead.

7. Line 6 — The Verdict

errors = 2
print("FAIL" if errors else "PASS", errors)
print("FAIL" if errors != 0 else "PASS")
print(bool(0), bool(1), bool(2))

FAIL 2
FAIL
False True True

if errors rather than if errors != 0 is Chapter 9.5's truthiness: 0 is false and every other integer is true. It reads well here and it is worth knowing it is a shortcut — on a value that might be None rather than 0, the two spellings stop agreeing, which is Chapter 19.10 Section 6's distinction.

8. The Deletion Test

Now the chapter's actual claim. Five one-token changes to those six lines, each a thing a person really writes:

UVM_INFO @ 0: reporter [RNTST] Running test axi_smoke_test...
UVM_ERROR drv.sv(88) @ 141900: uvm_test_top.env.agt.drv [WR] bad handshake
UVM_ERROR drv.sv(104) @ 180400: uvm_test_top.env.agt.drv [WR] timeout
UVM_INFO seq.sv(140) @ 200000: uvm_test_top.env.agt.seqr [SEQ] done

def baseline():
    errors = 0
    with open("run.log", encoding="utf-8") as handle:
        for line in handle:
            if "UVM_ERROR" in line:
                errors += 1
    return errors


def no_test():
    """Line 4 deleted."""
    errors = 0
    with open("run.log", encoding="utf-8") as handle:
        for line in handle:
            errors += 1
    return errors


def assign_not_add():
    """Line 5 is `errors = 1` instead of `errors += 1`."""
    errors = 0
    with open("run.log", encoding="utf-8") as handle:
        for line in handle:
            if "UVM_ERROR" in line:
                errors = 1
    return errors


def read_not_lines():
    """Line 3 loops over handle.read() instead of handle."""
    errors = 0
    with open("run.log", encoding="utf-8") as handle:
        for piece in handle.read():
            if "UVM_ERROR" in piece:
                errors += 1
    return errors


def init_inside():
    """Line 1 moved inside the loop."""
    with open("run.log", encoding="utf-8") as handle:
        for line in handle:
            errors = 0
            if "UVM_ERROR" in line:
                errors += 1
    return errors


for name, fn in [
    ("baseline", baseline),
    ("no test (line 4 gone)", no_test),
    ("= 1 instead of += 1", assign_not_add),
    ("read() instead of lines", read_not_lines),
    ("init inside the loop", init_inside),
]:
    print(f"{name:<26} -> {fn()}")

baseline                   -> 2
no test (line 4 gone)      -> 4
= 1 instead of += 1        -> 1
read() instead of lines    -> 0
init inside the loop       -> 0

Four wrong answers. No exception, no warning, no traceback — four integers that a report would print without comment.

Each one is worth naming, because each is a real mistake rather than a contrivance:

no_test returns 4 — the number of lines in the log. A count that happens to equal the line count is the easiest wrong answer to miss, because on a big log it looks like a big number of errors.

assign_not_add returns 1 — a boolean wearing a number's clothes. It is 1 for any failing log and 0 for a clean one, so the verdict stays correct and only the count is wrong. This survives for years.

read_not_lines returns 0 — handle.read() returns one string, and iterating a string gives characters. No single character contains UVM_ERROR, so the count is zero and every log passes. Chapter 10.3 established what iterating a string does; this is what it costs in a parser.

init_inside returns 0 — the counter is reset on every iteration, so the result is whatever the last line contributed. The last line of this log is a UVM_INFO, so the answer is zero. Had the log ended on an error line it would be 1, and the bug would look like assign_not_add.

9. What the Script Holds

At any moment during the loop, the script's entire state is:

Nothing grows with the size of the log. That is the property that makes this six-line script the right starting point for a tool that will eventually read a multi-gigabyte file, and it is a property you can lose with one line — lines = handle.readlines() or an all_errors.append(line) that nobody prunes. Chapter 15.7 Section 5 called that carrying a bounded summary rather than a growing list, and every chapter in this module keeps to it.

10. Predict, Run, Observe

LINES = ["UVM_ERROR a\n", "UVM_INFO b\n", "UVM_ERROR c\n"]

# A
n = 0
for line in LINES:
    if "UVM_ERROR" in line:
        n += 1
print("A:", n)

# B
n = 0
for line in LINES:
    if "UVM_ERROR" in line:
        n = 1
print("B:", n)

# C
for line in LINES:
    pass
print("C:", repr(line))

# D
print("D:", sum(1 for ch in "UVM_ERROR a" if "UVM_ERROR" in ch))

# E
print("E:", "FAIL" if 0 else "PASS", "FAIL" if 3 else "PASS")

# F
it = iter(LINES)
print("F:", sum(1 for _ in it), sum(1 for _ in it))

The verified output:

A: 2
B: 1
C: 'UVM_ERROR c\n'
D: 0
E: PASS FAIL
F: 3 0

A — the baseline. Two error lines, two counted.

B — = 1 instead of += 1. The verdict would still be FAIL; the count is wrong.

C — the loop variable holds the last item, with its newline.

D — iterating a string gives characters, and no character contains the marker.

E — 0 is falsy and 3 is truthy, which is what line 6 relies on.

F — an iterator is consumed once. Three, then zero, with nothing raised. Section 6, with a list iterator instead of a file.

11. Exercises

Exercise 1 — number your own. Take the checker and write one sentence per line saying what it does and what it holds.

Exercise 2 — the one that raises. Delete the initialiser and record the error, the line it names, and why it appears on the first error line rather than at the end.

Exercise 3 — inspect the handle. Print type, name, mode and encoding for an open log. Say which of the four you chose and which Python chose for you.

Exercise 4 — the newline. Print repr() of the first line. Say which of the following would be affected by the trailing \n: a membership test, an equality comparison, a $-anchored pattern, and a print.

Exercise 5 — closed. Show that the handle is closed after the with and that using it raises. Name the exception.

Exercise 6 — consumed once. Count the lines of a log twice from one handle. Record both numbers and say what a parser doing this would report.

Exercise 7 — the deletion table. Reproduce all five rows of Section 8 on a log of your own. Confirm four wrong answers and no exceptions.

Exercise 8 — which survive a verdict test. For each of your four wrong rows, say whether a PASS/FAIL-only test would catch it. Say what that implies about what a test should assert.

Exercise 9 — the last-line dependency. Run the init_inside version against a log that ends on an error line. Record the new answer and explain the change.

Exercise 10 — bounded state. Add one line to the baseline that makes its memory grow with the log. Say what it now costs on a 20 GB file and what you would keep instead.

12. Summary

• Six lines, and Chapter 20.1's four parts map onto them — line 2 input, lines 1 and 3 to 5 processing, line 6 decision and output.
• errors = 0 must exist before += 1, which reads the name before writing it; without it the script raises NameError on the first error line.
• open() returns a TextIOWrapper, not the file's text — an object that produces content on demand.
• encoding="utf-8" is a choice you make; omitting it hands the decision to the machine's locale, which differs between your laptop and the farm.
• with gives the file a name for the duration of the block and closes it at the end; the name survives, the open file does not.
• Each line arrives with its \n attached, which is invisible to in and decisive for comparison, printing and $.
• The loop variable survives the loop, holding whichever line happened to be last.
• A file object is an iterator consumed once. A second pass over the same handle yields nothing, with no error.
• if errors is truthiness, and it stops agreeing with errors != 0 the moment the value might be None.
• Five one-token changes give 2, 4, 1, 0, 0 — and only the deleted initialiser raises.
• Iterating handle.read() iterates characters, so the count is zero and every log passes.
• Two of the wrong versions keep the verdict correct and only corrupt the count, which a PASS/FAIL-only test cannot catch.
• Nothing in the script grows with the log, and that property is one careless append away from being lost.

The next chapter takes the first step away from a script about one file: lifting the path and the marker out of the body.