Streaming a Multi-Gigabyte Simulation Log

Part I · Module 15 · Chapter 15.7 · Streaming a Multi-Gigabyte Simulation Log

1. The Log Is Biggest When It Matters Most

A clean run prints a banner, a few hundred status lines and a summary. A run where the DUT is misbehaving prints an error for every failing transaction, and a protocol checker firing once per clock edge produces gigabytes in minutes.

So the pathological input is not an edge case you can decline to handle. It is the failing run, which is the only one anybody is going to ask you about.

2. The Change

path = "sim.log"
with open(path, "w") as f:
    f.write("UVM_INFO ok\n")

# not on a big log: text = f.read()

with open(path) as f:
    for line in f:
        print("saw:", line.rstrip())

saw: UVM_INFO ok

Chapter 15.4 Section 6 measured what that buys: for a 50,000-line log, holding the file costs 27,228 times more memory than holding its largest line. The ratio grows with the file.

A file object is iterable, and iterating it yields one line at a time, newline included, reading ahead only far enough to find the next separator:

Compile start
UVM_INFO top.sv(42) smoke begin
UVM_ERROR drv.sv(88) bad handshake
Simulation complete

errors = 0
scanned = 0

with open("sim.log") as f:
    for line in f:
        scanned += 1
        if "UVM_ERROR" in line:
            errors += 1

print("lines scanned:", scanned)
print("errors found :", errors)

lines scanned: 4
errors found : 1

Four lines scanned, one match. The whole file was examined and at no point was more than one line held.

3. Make the Parser Show Its Work

When a count disagrees with what you believe is in the file, do not reason about it — print the decision for every line:

Compile start
UVM_INFO top.sv(42) smoke begin
UVM_ERROR drv.sv(88) bad handshake
Simulation complete

with open("sim.log") as f:
    for n, line in enumerate(f, start=1):
        hit = "UVM_ERROR" in line
        print(f"{n}: hit={str(hit):<5} {line.rstrip()}")

1: hit=False Compile start
2: hit=False UVM_INFO top.sv(42) smoke begin
3: hit=True  UVM_ERROR drv.sv(88) bad handshake
4: hit=False Simulation complete

One match, on line 3, and now you can see why each of the other three missed.

enumerate works on a file exactly as Chapter 10.9 described — it does not read ahead or materialise anything, it just counts as the stream goes past. This four-line loop is the fastest way to settle an argument with a log, and it costs nothing to write.

Two patterns that catch most disagreements:

• line.rstrip() in the output, so a trailing \r or stray whitespace becomes visible as a gap before the closing position;
• print the test result, not just the matches — a parser that finds nothing and a parser that is never reached look identical otherwise.

4. One Pass, No Going Back

Iteration gives you a stream: each line arrives once, in order, and is gone. Three things you cannot do:

with open("sim.log") as f:
    print(len(f))

Compile start

Traceback (most recent call last):
  File "nopass.py", line 2, in <module>
    print(len(f))
          ~~~^^^
TypeError: object of type '_io.TextIOWrapper' has no len()

• No len() — the number of lines is unknown until you have read them all.
• No indexing — f[17] is not a thing; you cannot jump.
• No second look — once the loop has moved on, that line is gone unless you kept it.

The last one is the real constraint, and it shapes everything below.

5. Carry a Bounded Summary, Not a Growing List

The discipline is: whatever you carry across lines must not grow with the file.

with open("big.log", "w") as f:
    for i in range(20000):
        kind = "UVM_ERROR" if i % 2500 == 0 else "UVM_INFO"
        print(f"{kind} seq.sv({i}) transaction {i}", file=f)

counts = {}
first_error = None
scanned = 0

with open("big.log") as f:
    for line in f:
        scanned += 1
        kind = line.split()[0]
        counts[kind] = counts.get(kind, 0) + 1
        if first_error is None and kind == "UVM_ERROR":
            first_error = line.rstrip()

print("scanned     :", scanned)
print("counts      :", counts)
print("first error :", first_error)
print("things held :", len(counts), "counter keys + 1 line")

scanned     : 20000
counts      : {'UVM_ERROR': 8, 'UVM_INFO': 19992}
first error : UVM_ERROR seq.sv(0) transaction 0
things held : 2 counter keys + 1 line

Twenty thousand lines produced a dictionary with two keys and one saved string. Make the file a thousand times larger and those numbers do not change — counts is bounded by the number of distinct message types, not by the length of the log.

That is the shape to aim for: counters, sums, flags, a first-match, a running maximum. All O(1) in the file size.

6. The One Place a List Is Fine

The real request is rarely "count the errors". It is "show me what was happening just before the first one":

with open("big.log", "w") as f:
    for i in range(20000):
        kind = "UVM_ERROR" if i == 12000 else "UVM_INFO"
        print(f"{kind} seq.sv({i}) transaction {i}", file=f)

WINDOW = 3
recent = []
context = None

with open("big.log") as f:
    for line in f:
        if "UVM_ERROR" in line and context is None:
            context = list(recent)
            context.append(line.rstrip())
        recent.append(line.rstrip())
        if len(recent) > WINDOW:
            recent.pop(0)

print("lines held at any moment: at most", WINDOW)
for entry in context:
    print("  ", entry)

lines held at any moment: at most 3
   UVM_INFO seq.sv(11997) transaction 11997
   UVM_INFO seq.sv(11998) transaction 11998
   UVM_INFO seq.sv(11999) transaction 11999
   UVM_ERROR seq.sv(12000) transaction 12000

recent is a list and it is appended to on every line — and it is safe, because pop(0) keeps it at three entries. The test is not whether you use a list:

Is the list's maximum size bounded by something other than the file length?

Here it is bounded by WINDOW. In Section 5's counter it is bounded by the number of message types. A list of every error line is bounded by the number of errors, which on a badly failing run is the file length — so that one is only safe if you also cap it.

7. Stop Reading When You Are Done

break does what it always did, and on a stream it means the rest of the file is never read:

with open("big.log", "w") as f:
    print("Chronologic VCS simulator", file=f)
    print("UVM_INFO version 2020.3", file=f)
    for i in range(50000):
        print(f"UVM_INFO seq.sv({i}) transaction {i}", file=f)

scanned = 0
version = None

with open("big.log") as f:
    for line in f:
        scanned += 1
        if line.startswith("UVM_INFO version"):
            version = line.rstrip()
            break

print("version :", version)
print("lines read:", scanned, "of 50002")

version : UVM_INFO version 2020.3
lines read: 2 of 50002

Two lines read from a file of fifty thousand. On a 20 GB log, finding a banner near the top costs essentially nothing — the remaining bytes are never touched.

This is why "scan the log for the tool version" is cheap and "count every warning" is not, and it is worth knowing which of your questions is which.

8. The Habits That Silently Undo It

This is the section that matters most, because each of these looks like streaming code:

import sys

with open("big.log", "w") as f:
    for i in range(20000):
        print(f"UVM_INFO seq.sv({i}) transaction {i}", file=f)

with open("big.log") as f:
    everything = list(f)
print("A list(f)        :", len(everything), "lines held")

with open("big.log") as f:
    in_order = sorted(f)
print("B sorted(f)      :", len(in_order), "lines held")

with open("big.log") as f:
    joined = "".join(f)
print("C ''.join(f)     :", sys.getsizeof(joined), "bytes held")

with open("big.log") as f:
    picked = [line for line in f if "seq.sv(1)" in line]
print("D comprehension  :", len(picked), "lines held (of 20000 scanned)")

with open("big.log") as f:
    count = 0
    for line in f:
        if "seq.sv(1)" in line:
            count += 1
print("E streamed count :", count, "lines held: 1")

A list(f)        : 20000 lines held
B sorted(f)      : 20000 lines held
C ''.join(f)     : 797821 bytes held
D comprehension  : 1 lines held (of 20000 scanned)
E streamed count : 1 lines held: 1

A, B and C hold the entire file. list(f) is readlines() spelled differently. sorted(f) must see every line before it can emit the first, so sorting a stream is inherently a materialisation. "".join(f) rebuilds read().

D is the subtle one. A list comprehension over a file streams the input but materialises the output — here only one line, which is fine. The danger is a filter that matches a large fraction: [line for line in f if "UVM_" in line] on a UVM log keeps almost everything.

E is the shape that stays bounded regardless of how many lines match.

9. Predict, Run, Observe

alpha
beta
gamma
delta

with open("p.log") as f:
    first_two = []
    for line in f:
        first_two.append(line.rstrip())
        if len(first_two) == 2:
            break
    rest = f.read()

print("A:", first_two)
print("B:", repr(rest))

with open("p.log") as f:
    for line in f:
        pass
    print("C:", repr(line))
    print("D:", repr(f.read()))

A: ['alpha', 'beta']
B: 'gamma\ndelta\n'
C: 'delta\n'
D: ''

A and B — break left the cursor part-way through, and f.read() picked up exactly where the loop stopped. Breaking out of a stream does not discard the rest; it just stops consuming it.

C — after the loop ends, line still holds the last value it was bound to. The loop variable outlives the loop, which Chapter 10.12 lists among the classic loop mistakes — and which is occasionally useful, as here, for "the final line of the log".

D — the handle is exhausted, so read() returns '' without raising. That is Chapter 15.4 Section 8's fact, and it is the reason a second loop over the same handle silently does nothing.

10. Exercises

Exercise 1 — the swap. Take a parser you wrote with f.read().splitlines() and convert it to for line in f. Confirm the output is identical.

Exercise 2 — show the decisions. Write the Section 3 debug loop for a pattern of your choice, printing hit/miss per line. Use it on a file where your count surprises you.

Exercise 3 — three impossibilities. Try len(f), f[0] and a second for loop over the same handle. Record all three results.

Exercise 4 — bounded summary. Stream a 100,000-line log and produce a per-message-type count. State what bounds the dictionary's size.

Exercise 5 — the window. Implement "the five lines before the first UVM_ERROR". Prove your list never exceeds five entries.

Exercise 6 — unbounded by accident. Write a version that collects every error line. Describe the input that makes it as expensive as read().

Exercise 7 — early exit. Find a marker on line 3 of a 50,000-line file and report how many lines were read. Then move the marker to the end and compare.

Exercise 8 — spot the materialisation. For each, say whether the whole file ends up in memory: (a) max(f) (b) sum(1 for _ in f) (c) set(f) (d) f.readline() (e) [l for l in f if l.startswith("UVM_FATAL")].

11. Summary

• The failing run produces the biggest log, so the pathological input is the one you most need to parse.
• for line in f reads one line at a time, newline included, and is the default for anything a tool produced.
• A stream has no len(), no indexing and no second look.
• Whatever you carry across lines must not grow with the file — counters, sums, flags and first-matches are all O(1).
• A list is fine when its maximum size is bounded by something other than the file length, such as a fixed context window.
• break stops reading, so finding something near the top of a huge file is almost free — and the cursor stays where you stopped.
• list(f), sorted(f) and "".join(f) all hold the whole file, however much they look like stream code.
• A comprehension streams its input but materialises its output, so a permissive filter is as expensive as readlines().
• Sorting cannot be streamed — recognise it early rather than optimising around it.
• A loop variable outlives its loop, which gives you the last line for free.

The next chapter asks questions about a file without opening it, which is how you find out whether there is anything to stream in the first place.