Yielding Records Instead of Building One Big List

Part II · Module 20 · Chapter 20.13 · Yielding Records Instead of Building One Big List

1. The List at the Boundary

Here is the parser's shape so far, as a function returning failures:

UVM_INFO @ 0: reporter [RNTST] starting
UVM_ERROR drv.sv(88) @ 141900: uvm_test_top.env.agt.drv [WR] bad handshake
UVM_INFO scb.sv(120) @ 160000: uvm_test_top.env.scb [CHK] 160 compared
UVM_ERROR drv.sv(104) @ 180400: uvm_test_top.env.agt.drv [WR] timeout
UVM_ERROR scb.sv(204) @ 190100: uvm_test_top.env.scb [CHK] data mismatch

def failures_list(path):
    """Every failing line, as a list."""
    out = []
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            if line.startswith("UVM_ERROR"):
                out.append(line.rstrip())
    return out


found = failures_list("a.log")
print("count:", len(found))
print("first:", found[0][:46])

count: 3
first: UVM_ERROR drv.sv(88) @ 141900: uvm_test_top.en

Nothing wrong with that on one log. The loop inside is still bounded — it reads one line at a time — but what it hands back is not: out grows with the number of failures, and the caller holds the whole list.

On a log with three failures that is three strings. On a regression sweep it is the sum over every log, held simultaneously, because the caller is in a loop and the lists do not go away until it finishes. A test that logged forty thousand errors before giving up is forty thousand strings, and there is usually one of those in every sweep.

The irony is specific: Chapter 15.7 and Chapter 20.10 were careful to keep the parsing bounded, and then the return type undoes it.

2. One Keyword

UVM_INFO @ 0: reporter [RNTST] starting
UVM_ERROR drv.sv(88) @ 141900: uvm_test_top.env.agt.drv [WR] bad handshake
UVM_INFO scb.sv(120) @ 160000: uvm_test_top.env.scb [CHK] 160 compared
UVM_ERROR drv.sv(104) @ 180400: uvm_test_top.env.agt.drv [WR] timeout
UVM_ERROR scb.sv(204) @ 190100: uvm_test_top.env.scb [CHK] data mismatch

def failures(path):
    """Every failing line, one at a time."""
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            if line.startswith("UVM_ERROR"):
                yield line.rstrip()


for failure in failures("a.log"):
    print(failure[:46])

UVM_ERROR drv.sv(88) @ 141900: uvm_test_top.en
UVM_ERROR drv.sv(104) @ 180400: uvm_test_top.e
UVM_ERROR scb.sv(204) @ 190100: uvm_test_top.e

out = [], out.append(...) and return out became yield. The caller's loop is unchanged. At no point does a list of failures exist.

That is the canonical annotation:

def failures(path):
    with open(path) as f:
        for line in f:
            if UVM.search(line):
                yield line.strip()

A function containing yield anywhere in its body is a generator function, and calling it does something quite different from calling an ordinary one.

3. What Calling It Actually Does

That first step is checkable, and it is worth checking because nothing about the syntax suggests it:

UVM_INFO @ 0: reporter [RNTST] starting
UVM_ERROR drv.sv(88) @ 141900: bad handshake
UVM_ERROR drv.sv(104) @ 180400: timeout

opened = []


def failures(path):
    with open(path, encoding="utf-8") as handle:
        opened.append(handle)
        for line in handle:
            if line.startswith("UVM_ERROR"):
                yield line.rstrip()


gen = failures("a.log")
print("type               :", type(gen).__name__)
print("files opened so far:", len(opened))

first = next(gen)
print("after one request  :", first[:34])
print("files opened now   :", len(opened), " closed:", opened[0].closed)

rest = list(gen)
print("the rest           :", len(rest))
print("after exhaustion   :", " closed:", opened[0].closed)

type               : generator
files opened so far: 0
after one request  : UVM_ERROR drv.sv(88) @ 141900: bad
files opened now   : 1  closed: False
the rest           : 1
after exhaustion   :  closed: True

Read the second line. Calling failures("a.log") opened no file. The body had not run at all — not the with, not the loop, nothing. What came back is a generator object that knows how to run the body when asked.

next(gen) is the asking. It runs the body until the first yield, which opens the file, reads to the first match, hands the value over and pauses there — with the file still open and the loop's position preserved.

list(gen) asks repeatedly until the body ends. When it does, the with block exits normally and the file closes.

4. It Is Still an Iterator

Everything Chapter 20.3 Section 6 established about file objects applies, because a generator is the same kind of thing:

UVM_ERROR one
UVM_INFO x
UVM_ERROR two
UVM_ERROR three

def failures(path):
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            if line.startswith("UVM_ERROR"):
                yield line.rstrip()


gen = failures("a.log")
print("first pass :", len(list(gen)))
print("second pass:", len(list(gen)))

fresh = failures("a.log")
print("fresh pass :", len(list(fresh)))

print("len(gen)?  :", hasattr(gen, "__len__"))

first pass : 3
second pass: 0
fresh pass : 3
len(gen)?  : False

Consumed once, exactly like the file handle underneath it. The second pass is empty and nothing raises.

And a generator has no length. That is not an oversight: it cannot know how many values it will produce without producing them. Any code that wants a count must either consume it — sum(1 for _ in gen), which then leaves it empty — or keep a counter as it goes, which is what Chapter 20.16's record does.

That is the real cost of this shape, and it is worth naming plainly: you give up knowing how much there is in exchange for never holding it all. For a parser feeding a report that processes each record as it arrives, that is a good trade. For code that needs the data twice, or needs a total before it starts, it is not — which is Chapter 20.14's subject.

5. The File Stays Open If You Walk Away

The hazard, and it is specific to the with-inside-a-generator shape the canonical annotation uses:

UVM_ERROR one
UVM_ERROR two
UVM_ERROR three

opened = []


def failures(path):
    with open(path, encoding="utf-8") as handle:
        opened.append(handle)
        for line in handle:
            if line.startswith("UVM_ERROR"):
                yield line.rstrip()


gen = failures("a.log")
first = next(gen)
print("got        :", first)
print("file closed:", opened[0].closed)

gen.close()
print("after close:", opened[0].closed)

got        : UVM_ERROR one
file closed: False
after close: True

The caller took one value and walked away. The generator is paused inside the with block, so the block has not exited and the file is still open.

In an ordinary function with guarantees the file closes when the function returns. In a generator it guarantees the file closes when the body finishes — and the body does not finish if nobody asks for the rest. That is not a bug in with; it is what "paused mid-function" means.

It matters when a reference to the paused generator stays alive, and the shape that does that is a sweep collecting one generator per log:

UVM_ERROR one
UVM_ERROR two

opened = []


def failures(path):
    with open(path, encoding="utf-8") as handle:
        opened.append(handle)
        for line in handle:
            if line.startswith("UVM_ERROR"):
                yield line.rstrip()


kept = []
for _ in range(3):
    gen = failures("a.log")
    kept.append(gen)
    print("first failure:", next(gen))

print("files still open:", [not h.closed for h in opened])

for gen in kept:
    gen.close()
print("after closing   :", [not h.closed for h in opened])

first failure: UVM_ERROR one
first failure: UVM_ERROR one
first failure: UVM_ERROR one
files still open: [True, True, True]
after closing   : [False, False, False]

Three generators, three open files, none of them finished. Scale that to a thousand logs and the sweep hits the operating system's limit on open file descriptors — at which point it fails with OSError: Too many open files on a log that is perfectly fine.

Now the version that does not leak, which is worth knowing precisely because it looks equally careless:

UVM_ERROR one
UVM_ERROR two

opened = []


def failures(path):
    with open(path, encoding="utf-8") as handle:
        opened.append(handle)
        for line in handle:
            if line.startswith("UVM_ERROR"):
                yield line.rstrip()


def first_failure(path):
    for failure in failures(path):
        return failure
    return None


print("got   :", first_failure("a.log"))
print("closed:", opened[0].closed)

got   : UVM_ERROR one
closed: True

Returning from inside the loop abandons the generator — and the file is closed anyway. CPython reference-counts, so the moment first_failure returns, the last reference to the generator disappears, it is collected, closing it runs the with block's exit, and the file closes. There is nothing left holding it.

So the rule is narrower than "always close a generator", and worth stating exactly:

The file stays open for as long as something holds the paused generator. Abandon it with no references left and CPython closes it immediately; keep it in a list, a dict or a long-lived variable and it stays open until you close it or the name goes away.

Two consequences follow. First, the common shapes in this module — a plain for loop over a generator, or a helper that returns from inside one — are safe on CPython without any ceremony. Second, the shape that is not safe is the one that looks most like good engineering: building a collection of parsers up front and consuming them later.

def first_failure_explicit(gen):
    """Safe regardless of who holds the generator or which interpreter."""
    try:
        for failure in gen:
            return failure
        return None
    finally:
        gen.close()


def numbers():
    yield 1
    yield 2


print("value:", first_failure_explicit(numbers()))

value: 1

The point is the finally: it closes the generator whoever is holding it, and on an implementation that does not reference-count promptly it is the only thing that does.

6. What It Buys Across a Sweep

UVM_ERROR drv.sv(88) @ 141900: bad handshake
UVM_ERROR drv.sv(104) @ 180400: timeout

UVM_INFO @ 0: reporter [RNTST] starting

UVM_ERROR scb.sv(204) @ 90100: data mismatch

def failures(path):
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            if line.startswith("UVM_ERROR"):
                yield line.rstrip()


total = 0
worst_log = None
worst_count = -1
live_records = 0

for path in ["run1.log", "run2.log", "run3.log"]:
    count = 0
    for failure in failures(path):
        count += 1
        total += 1
        live_records = max(live_records, 1)
    if count > worst_count:
        worst_count, worst_log = count, path
    print(f"{path:<10} failures={count}")

print()
print("total failures      :", total)
print("worst log           :", worst_log, worst_count)
print("records held at once:", live_records)

run1.log   failures=2
run2.log   failures=0
run3.log   failures=1

total failures      : 3
worst log           : run1.log 2
records held at once: 1

One record held at a time, across any number of logs with any number of failures in them. The totals and the worst-log comparison are integers and a string, which do not grow.

That is the shape the canonical purpose describes: a report over a thousand logs never holds a thousand logs in memory at once. Note that the sweep still keeps a running total and a running maximum — the generator does not stop you accumulating, it stops you accumulating the raw data when all you wanted was a summary of it. Chapter 15.7 Section 5's bounded summary, now applied at the level of the whole regression rather than one file.

Each inner for loop also runs to exhaustion, so Section 5's hazard does not arise here.

7. Yielding Records, Not Lines

Nothing says the yielded value has to be a string. For a parser, a dict per failure is more useful:

UVM_INFO @ 0: reporter [RNTST] starting
UVM_ERROR drv.sv(88) @ 141900: uvm_test_top.env.agt.drv [WR] bad handshake
UVM_ERROR scb.sv(204) @ 190100: uvm_test_top.env.scb [CHK] data mismatch

import re

UVM_FAIL = re.compile(
    r"^UVM_(?P<sev>ERROR|FATAL)\s+(?P<file>\S+\.sv)\((?P<line>\d+)\)"
    r"\s+@\s+(?P<time>\d+):\s+(?P<scope>\S+)\s+\[(?P<id>[^\]]+)\]\s*(?P<msg>.*)$"
)


def failure_records(path):
    """One dict per failing message, in file order."""
    with open(path, encoding="utf-8") as handle:
        for number, line in enumerate(handle, 1):
            m = UVM_FAIL.match(line)
            if m is None:
                continue
            record = m.groupdict()
            record["time"] = int(record["time"])
            record["line_no"] = number
            yield record


for record in failure_records("a.log"):
    print(f"line {record['line_no']:>3}  t={record['time']:>7}  "
          f"{record['id']:<4} {record['scope']}")

line   2  t= 141900  WR   uvm_test_top.env.agt.drv
line   3  t= 190100  CHK  uvm_test_top.env.scb

Parsed, converted and numbered, one record at a time. enumerate(handle, 1) gives the line number for free — Chapter 10.9 — and it is worth carrying, because a report that says line 88 of the log lets somebody jump straight there.

This is the form Chapter 20.16 builds its record from, and the form Chapter 20.14 chains further transformations onto.

8. Predict, Run, Observe

log = []


def gen():
    log.append("started")
    yield 1
    log.append("middle")
    yield 2
    log.append("ended")


# A
g = gen()
print("A:", log)

# B
print("B:", next(g), log)

# C
print("C:", next(g), log)

# D
print("D:", list(gen()))

# E
g2 = gen()
print("E:", len(list(g2)), len(list(g2)))

# F
print("F:", hasattr(gen(), "__len__"), hasattr(gen(), "__next__"))

# G
log.clear()
g3 = gen()
next(g3)
g3.close()
print("G:", log)

The verified output:

A: []
B: 1 ['started']
C: 2 ['started', 'middle']
D: [1, 2]
E: 2 0
F: False True
G: ['started']

A — calling gen() ran nothing. The list is empty.

B — the first next ran the body up to the first yield and stopped.

C — the second resumed after that yield, not from the top.

D — list drives it to exhaustion, so both values come back.

E — consumed once: two, then zero.

F — no __len__, but it is an iterator.

G — close() on a generator paused at the first yield unwinds it without running the rest, so "middle" never appears. That is what lets the with exit early in Section 5.

9. Exercises

Exercise 1 — the list at the boundary. Write the list-returning version and say what a caller looping over a thousand logs holds.

Exercise 2 — one keyword. Convert it to yield. Confirm the caller's loop does not change.

Exercise 3 — nothing runs. Add a side effect to the first line of your generator's body and call the function without consuming it. Record what happened.

Exercise 4 — pause and resume. Instrument a generator with a log list and reproduce Section 8's A to C. Say in one sentence where the body resumes.

Exercise 5 — consumed once. Consume a generator twice and report both counts. Then get a second pass and say what it cost you.

Exercise 6 — no length. Try len() on a generator. Record the error, then count its records two different ways and say what each costs.

Exercise 7 — the open file. Reproduce Section 5: take one value and check whether the file is closed. Then close the generator and check again.

Exercise 8 — leaky and safe. Write both first_failure versions. Say which one you would ship and what the other one's symptom looks like.

Exercise 9 — the innocent log. Explain in two sentences why a descriptor leak reports its error on a file that is not the problem.

Exercise 10 — one at a time. Sweep three logs with a generator, keeping a running total and a worst-log comparison. Report how many records were live at once.

Exercise 11 — yield a record. Change your generator to yield a dict with the fields parsed, the time converted and the log line number included.

Exercise 12 — when not to. Name one thing your report needs that a generator cannot give it, and say what you would do about it.

10. Summary

• A function returning a list makes its caller hold everything, which undoes the care taken inside the parser.
• yield turns the function into a generator that hands values back one at a time; the caller's for loop is unchanged.
• Calling a generator function runs none of its body. It returns a generator object — the file in the canonical annotation is not even opened.
• The first request runs the body up to the first yield and pauses there, with locals intact and the file open.
• A generator is a paused function, not a collection. Laziness, bounded memory and single-pass all follow from that.
• It is consumed once, exactly like the file handle underneath it, and a second pass yields nothing with no error.
• It has no length, because it cannot know without producing the values.
• You trade knowing how much there is for never holding it all — a good trade for a streaming report, a bad one when you need the data twice.
• with inside a generator closes the file when the body finishes, not when the caller stops asking.
• The file stays open for as long as something holds the paused generator. Abandon it with no references left and CPython collects and closes it at once.
• The leaking shape is a collection of generators built up front — a list or dict of one parser per log — not the plain loop or the return-from-inside-a-loop helper, both of which are safe.
• The error appears on an innocent log — look for the generators somebody is still holding, not the file in the traceback.
• Fixes: exhaust it, or gen.close() in a finally, which works whoever holds it and on an implementation that does not reference-count promptly.
• Yield records, not lines — a dict with the fields parsed, the time converted and the line number from enumerate.
• The sweep still accumulates totals and maxima. A generator stops you holding the raw data, not the summary.

The next chapter chains generators into a pipeline — and is equally clear about the cases where materialising a list is simply the right answer.