Lazy Pipelines — and When a List Is Simply Better

Part II · Module 20 · Chapter 20.14 · Lazy Pipelines — and When a List Is Simply Better

1. A Parse Is Usually Several Steps

Reading failures out of a log is really three jobs stacked: drop what is not a line of content, keep what is a failure, turn it into a field. Chapter 20.13 did all three inside one generator, which works and grows awkward as stages are added.

The alternative is a stage each:

UVM_INFO @ 0: reporter [RNTST] starting

UVM_ERROR drv.sv(88) @ 141900: uvm_test_top.env.agt.drv [WR] bad handshake
   
UVM_ERROR scb.sv(204) @ 190100: uvm_test_top.env.scb [CHK] data mismatch
UVM_INFO seq.sv(140) @ 200000: done

import re

UVM = re.compile(r"^UVM_ERROR")

with open("a.log", encoding="utf-8") as handle:
    stripped = (line.rstrip() for line in handle)
    content = (line for line in stripped if line.strip())
    failures = (line for line in content if UVM.search(line))
    times = (int(m.group(1)) for m in
             (re.search(r"@ (\d+):", line) for line in failures) if m)

    found = list(times)

print("failure times:", found)

failure times: [141900, 190100]

Four stages, each one line, each doing one thing. The canonical annotation is the middle two:

lines = (l for l in f if l.strip())
fails = (l for l in lines if UVM.search(l))

2. A Generator Expression

(line for line in handle) is a generator expression — the same syntax as a list comprehension with round brackets instead of square ones, and an entirely different object:

LINES = ["UVM_ERROR a", "UVM_INFO b", "UVM_ERROR c"]

as_list = [line for line in LINES if line.startswith("UVM_ERROR")]
as_gen = (line for line in LINES if line.startswith("UVM_ERROR"))

print("list:", type(as_list).__name__, as_list)
print("gen :", type(as_gen).__name__, as_gen is not None)
print("consumed:", list(as_gen))
print("again   :", list(as_gen))
print("has len :", hasattr(as_list, "__len__"), hasattr(as_gen, "__len__"))

list: list ['UVM_ERROR a', 'UVM_ERROR c']
gen : generator True
consumed: ['UVM_ERROR a', 'UVM_ERROR c']
again   : []
has len : True False

Square brackets build the whole list now. Round brackets build nothing now and produce values when asked — Chapter 20.13's generator, in expression form rather than as a def. Everything from that chapter applies: consumed once, no length, laziness.

Chapter 12.1 introduced the list comprehension and Chapter 12.7 argued about when a loop reads better. The same judgment applies here, with one extra axis: the comprehension's cost is the list it builds, and that cost is what the round brackets remove.

3. Nothing Is Pushed; the Last Stage Pulls

The word "pipeline" suggests data being pushed along, and that is the wrong picture. Instrument each stage and the real mechanism is visible:

trace = []


def tracked(source, tag):
    """Pass values through, recording each time this stage produces one."""
    for item in source:
        trace.append(tag)
        yield item


LINES = ["", "UVM_ERROR one", "UVM_INFO x", "UVM_ERROR two"]

stage_a = tracked((line for line in LINES), "a")
stage_b = tracked((line for line in stage_a if line.strip()), "b")
stage_c = tracked((line for line in stage_b if line.startswith("UVM_ERROR")), "c")

print("after building:", trace)

first = next(stage_c)
print("after one value:", trace)
print("the value      :", first)

rest = list(stage_c)
print("after the rest :", trace)
print("the rest       :", rest)

after building: []
after one value: ['a', 'a', 'b', 'c']
the value      : UVM_ERROR one
after the rest : ['a', 'a', 'b', 'c', 'a', 'b', 'a', 'b', 'c']
the rest       : ['UVM_ERROR two']

Read the second line. To produce one value, stage a ran twice, stage b once and stage c once — because the first input line was blank, so a had to be asked twice before b had anything to pass on.

That is the pull model: list() asks stage_c, which asks stage_b, which asks stage_a, which asks the source. Each stage does the minimum work needed to produce one value, and then stops. Nothing moves until the end of the chain asks for it, which is why after building is empty.

The consequence that matters for a parser: a chain of ten stages over a twenty-million-line log holds ten values at a time, one per stage, not twenty million.

4. When a List Is Simply Better

Five situations, all common in reporting, where the right move is list(...) and being lazy is a mistake:

You need the data twice. A generator is consumed once, so a second pass gets nothing — and gets it silently:

def failures():
    yield {"test": "smoke", "errors": 2}
    yield {"test": "burst", "errors": 1}


lazy = failures()
print("count:", sum(1 for _ in lazy))
print("worst:", max((f["errors"] for f in lazy), default=None))

eager = list(failures())
print("count:", len(eager))
print("worst:", max(f["errors"] for f in eager))

count: 2
worst: None
count: 2
worst: 2

The lazy version reports None for the worst failure count on data that plainly has one, because counting consumed it. That is the single most common generator bug, and default=None is the only thing standing between it and a ValueError.

You need the length. len() does not work, and sum(1 for _ in gen) costs the whole stream and leaves it empty.

You need it sorted. Sorting is inherently eager — nothing can know the smallest element without seeing them all — so sorted(gen) materialises internally anyway. There is no laziness to preserve:

gen = (x for x in [3, 1, 2])
print("sorted:", sorted(gen))
print("type  :", type(sorted((x for x in [1]))).__name__)

sorted: [1, 2, 3]
type  : list

You need an index. records[0], records[-1] and slicing all need a sequence. For a report that wants the top five, the data has to be a list by then regardless.

The data must outlive its source. Which is Section 5.

5. The Trap: A Pipeline That Outlives Its File

This is the one that catches people, and it is specific to generator expressions:

UVM_ERROR one
UVM_INFO x
UVM_ERROR two

def build():
    with open("a.log", encoding="utf-8") as handle:
        return (line.rstrip() for line in handle if line.startswith("UVM_ERROR"))


pipeline = build()
print(list(pipeline))

Traceback (most recent call last):
  File "pipe.py", line 7, in <module>
    print(list(pipeline))
          ~~~~^^^^^^^^^^
  File "pipe.py", line 3, in <genexpr>
    return (line.rstrip() for line in handle if line.startswith("UVM_ERROR"))
                                      ^^^^^^
ValueError: I/O operation on closed file.

build() returns the generator expression, the with block exits, the file closes — and then the caller asks for values from a pipeline whose source is a closed file.

Chapter 20.13's def ... yield form does not have this problem:

UVM_ERROR one
UVM_INFO x
UVM_ERROR two

def build():
    with open("a.log", encoding="utf-8") as handle:
        for line in handle:
            if line.startswith("UVM_ERROR"):
                yield line.rstrip()


print(list(build()))

['UVM_ERROR one', 'UVM_ERROR two']

The difference is where the with is. In the def form the whole function body — including the with — is suspended inside the generator, so the block has not exited when the caller starts consuming. In the expression form the with is in the enclosing function, which returned.

That asymmetry gives a rule worth keeping:

Use def ... yield at a function boundary. Use generator expressions inside one.

A pipeline assembled and consumed within a single with block, as in Section 1, is correct and reads well. A pipeline returned from a function should be a generator function — or a list, if the caller genuinely needs one.

6. The Honest Position

Chained generators are a narrow win, and the honest summary of when they pay:

The first two rows are the case this module cares about: a sweep of a thousand logs, each read once, feeding a summary that accumulates as it goes. Everything else in the table is a report, and a report nearly always wants a list — it sorts, it counts, it shows a top five, it prints totals before details.

The practical shape that follows: stream the parsing and materialise the summary. One record per log flows lazily out of the parser; the few hundred summary dicts that come back go into a list, because a few hundred dicts is nothing and sorting them is the whole point. Chapter 20.16's record is exactly the boundary where that switch happens.

7. Predict, Run, Observe

trace = []


def tracked(source, tag):
    for item in source:
        trace.append(tag)
        yield item


data = [1, 2, 3, 4]
evens = tracked((n for n in data if n % 2 == 0), "e")

# A
print("A:", trace)

# B
print("B:", next(evens), trace)

# C
print("C:", list(evens))

# D
print("D:", list(evens))

# E
g = (n for n in [3, 1, 2])
print("E:", sorted(g), list(g))

# F
h = (n for n in [1, 2, 3])
print("F:", sum(1 for _ in h), max(h, default="gone"))

# G
print("G:", type((n for n in [1])).__name__, type([n for n in [1]]).__name__)

The verified output:

A: []
B: 2 ['e']
C: [4]
D: []
E: [1, 2, 3] []
F: 3 gone
G: generator list

A — building the pipeline ran nothing.

B — one value out; the stage ran once. The odd 1 was consumed by the filter without the stage recording it, because the filter sits inside the expression rather than in tracked.

C and D — the rest, then nothing. Consumed once.

E — sorted works and drains the generator, so the follow-up list is empty. Sorting is eager by nature.

F — the counting pass consumed it, so max has nothing and returns the default. This is the bug from Section 4, in two expressions on one line.

G — round brackets give a generator, square brackets give a list.

8. Exercises

Exercise 1 — four stages. Build a pipeline that strips, drops blanks, keeps failures and extracts a time. Confirm it works inside one with block.

Exercise 2 — brackets. Write the same filter as a list comprehension and a generator expression. Print both types and consume each twice.

Exercise 3 — trace the pull. Instrument three stages with a tracked wrapper and record the trace after building, after one value, and after exhaustion.

Exercise 4 — explain the trace. For your result, say why the first stage ran more times than the last.

Exercise 5 — break is free. Put a break after the second value and report how many times the first stage ran.

Exercise 6 — the silent empty. Count a generator and then take its maximum. Record what you get, then say what happens without default=.

Exercise 7 — sorting. Sort a generator and then list it. Explain the second result in one sentence.

Exercise 8 — the closed file. Return a generator expression from a function with a with block in it. Record the exception and the line the traceback blames.

Exercise 9 — the def form. Rewrite it with def ... yield and confirm it works. Say where the with block is in each case.

Exercise 10 — the rule. State in one sentence when to use a generator expression and when to use def ... yield.

Exercise 11 — your own table. For a report you have written, fill in Section 6's table. Say which rows applied and what you would change.

Exercise 12 — the boundary. Identify the point in your own parser where streaming should stop and a list should start. Justify it in two sentences.

9. Summary

• A parse is usually several stages, and each can be one generator expression.
• (x for x in y) is a generator expression — comprehension syntax, round brackets, and an entirely different object from a list.
• Square brackets build the list now; round brackets build nothing now.
• Nothing is pushed through a pipeline. The last stage pulls from the one before it, as far as needed, which is why building one runs no code at all.
• A chain of ten stages holds ten values, not the input.
• Stopping early really stops the work, at every stage, not just the loop you wrote.
• Use a list when you need the data twice — a second pass over a generator is silently empty.
• Counting a generator consumes it, so a following max returns the default or raises.
• Sorting is eager by nature. sorted(gen) materialises internally and returns a list.
• Indexing, slicing and a top-five report all need a sequence.
• A generator expression returned from a function whose with has exited raises ValueError: I/O operation on closed file.
• The def ... yield form does not, because the with is suspended inside the generator rather than already exited.
• Use def ... yield at a function boundary and generator expressions inside one.
• Stream the parsing and materialise the summary. A thousand logs flow; a few hundred summary dicts go in a list, because that is what sorting and reporting need.

The next chapter takes on the one parsing problem none of these tools solve: a record that is spread across several lines, where the meaning of a line depends on which section you are in.