Memory Usage

Simulation Performance Optimization · Module 28 · Page 28.2

The Engineering Problem

The previous chapter made the environment fast. This one keeps it alive, because memory is a different kind of problem from runtime. Runtime makes a test slow — a slow test still finishes. Memory, when it grows without bound, makes a test die — the compute farm OOM-kills it the moment it crosses the memory ceiling. And the most dangerous memory problem is time-dependent: a structure that grows with simulation time runs fine for a while, then crosses the ceiling and dies — so the test passes in short runs and OOMs after hours in the long nightly regression, masking in development and striking in production. The trap is reading the OOM as an infrastructure or functional failure ("why did it die at hour four?") when it's a memory leak — and guessing at the cause instead of measuring what grew. In a garbage-collected language like SystemVerilog, a "leak" isn't a missing free — it's retaining a reference to an object you no longer need, so the garbage collector can't reclaim it, and the set of retained objects grows without bound: a scoreboard queue that never drains, a coverage database that grows, a history that's never pruned, unnecessary copies. The problem this chapter solves is memory usage: why memory is a hard ceiling and unbounded growth is fatal, why it hides in short tests, how to find the growing structure by profiling heap over time, and how to bound retention so the working set stays fixed and large environments fit.

Memory usage optimization is finding and bounding the structures that grow without bound, so the testbench retains a fixed-size working set regardless of how long it runs. The crucial difference from runtime: memory is a hard ceiling — unbounded growth doesn't slow the test, it kills it (OOM) — and it's time-dependent (fails only on long-enough runs, masking in short tests). The dominant cause is unbounded accumulation: retaining objects you no longer need (in a GC language, holding a reference prevents reclamation), so the retained set grows with simulation time. The common accumulators: transaction queues that never drain (a scoreboard or FIFO that pushes faster than it pops, or never pops unmatched entries — the canonical leak); coverage databases growing unbounded (fine-grained per-transaction bins); histories and logs retained "for debug" and never pruned; associative arrays keyed by something unbounded (every address, every ID); and unnecessary copies (cloning large transactions when a reference would do). The fix is almost always to bound the retention: free, pop, match-and-discard, or prune as you go, so the working set is the outstanding set (small and fixed), not the cumulative history (unbounded). The cardinal discipline: make memory bounded — a fixed working set that does not grow with simulation time — and find the leak by profiling heap over time (which structure grows monotonically?), not by guessing. This chapter is memory usage: the hard ceiling, unbounded accumulation, the time-dependence, heap profiling, and bounding retention.

Why is memory a hard ceiling that OOM-kills a test rather than just slowing it, why does unbounded accumulation hide in short tests and strike only on long runs, how do you find the growing structure by profiling heap over time, and how do you bound retention so the working set stays fixed?

Motivation — why memory is a hard ceiling and unbounded growth is fatal

Memory differs from runtime in ways that make unbounded growth a distinct, fatal, and sneaky failure. The reasons:

• Memory is a hard ceiling; runtime is a slope. A slow test is still correct and still finishes — runtime is a gradual cost. An out-of-memory test dies — memory has a hard limit (the farm's RAM), and crossing it is fatal, not gradual.
• Unbounded growth is time-dependent, so it hides. A structure that grows with simulation time is small in a short test (passes) and huge in a long one (OOMs) — so it hides in development (short runs) and strikes in regression (long runs). The failure correlates with run length, not with what the test does.
• The OOM looks like something else. A test that dies at hour four looks like an infrastructure problem or a functional hang — so the natural first guess is wrong, and you debug the wrong thing until you recognize it as a memory leak.
• In a GC language, leaks are retained references, not missing frees. SystemVerilog garbage-collects — there's no manual free. So a leak is holding a reference (in a queue, an array, a history) to something you no longer need, which prevents reclamation. The cause is accumulation, not forgetting to free.
• Bounded vs unbounded is a design property you can measure. A memory-safe testbench has a fixed working set regardless of run length; an unsafe one grows. The difference is measurable (does the heap grow monotonically?) — so you find it by profiling over time, not guessing.

The motivation, in one line: memory is a hard ceiling (OOM kills, doesn't slow), and unbounded accumulation is time-dependent (hides in short tests, strikes on long runs) and looks like an infrastructure failure — and in a GC language it's retained references, not missing frees — so you must recognize the time-correlated OOM as a leak, profile the heap over time to find what grows, and bound the retention to a fixed working set.

Mental Model

Hold memory usage as a warehouse that must ship goods as fast as it receives them — or inventory piles up to overflow:

A warehouse takes in shipments and sends them back out. As long as it ships goods out as fast as it takes them in, the amount of inventory sitting on the floor stays roughly constant — it's just the goods currently in transit, being processed and moved along. That's a healthy, bounded operation: no matter how many years it runs or how many millions of shipments pass through, the floor never holds more than the in-transit set at any moment, which is small relative to the total throughput. Now suppose something goes wrong with shipping out. Maybe a fraction of incoming shipments get misrouted to a corner and forgotten, or a paperwork mismatch means they can never be matched to an outgoing order, so they just sit there. Each one is small, and on any given day you'd barely notice — the floor looks fine. But they accumulate. Every day, a few more shipments arrive that will never leave, and the pile of stuck inventory grows, slowly and relentlessly, without any bound. For a while the warehouse has plenty of room and operates normally. Then, after enough time, the stuck inventory fills the floor to the rafters, there's no room to process new shipments, and the whole operation grinds to a halt. The tragedy is that a short inspection would never have caught it — early on there's loads of space — and the failure shows up only after the warehouse has been running long enough for the slow pile-up to reach the ceiling. The fix is never "buy a bigger warehouse," which only delays the halt; it's to make sure everything received is shipped back out, so inventory stays bounded at the in-transit set. A warehouse takes in shipments and sends them out. As long as it ships as fast as it receives, the inventory on the floor stays roughly constant — just the goods in transit. That's a healthy, bounded operation: no matter how many years it runs, the floor never holds more than the in-transit set, which is small relative to total throughput. Now suppose shipping out goes wrong: a fraction of incoming shipments get misrouted to a corner and forgotten, or a mismatch means they can never be matched to an outgoing order, so they just sit there. Each is small, and on any given day you'd barely notice — the floor looks fine. But they accumulate: every day, a few more arrive that will never leave, and the pile grows, slowly and relentlessly, without bound. For a while the warehouse has plenty of room. Then, after enough time, the stuck inventory fills the floor to the rafters, there's no room, and operations halt. The tragedy: a short inspection would never have caught it (early on there's loads of space), and the failure shows up only after running long enough for the slow pile-up to reach the ceiling. The fix is never "buy a bigger warehouse" (which only delays the halt); it's to ship everything received back out, so inventory stays bounded at the in-transit set.

So memory usage is a warehouse shipping as fast as it receives: transactions arriving are shipments received, matching-and-freeing them is shipping them out, and the heap is the inventory on the floor. A bounded testbench ships as fast as it receives — its working set is the in-transit (outstanding) set, small and fixed regardless of run length. An unbounded one receives without fully shipping — a fraction of transactions are retained and never freed (a queue that never drains, a history never pruned), so inventory piles up slowly and relentlessly until it fills to the rafters (crosses the memory ceiling) and operations halt (OOM). The time-dependence is the warehouse's: short runs have loads of room (the test passes), long runs fill to overflow (the test OOMs after hours). And the fix is the warehouse's: not a bigger warehouse (more RAM only delays the OOM) but ship everything received (bound the retention — free/match/prune as you go, so the working set stays the outstanding set). Diagnose memory like warehouse inventory: profile the heap over time to find what piles up (grows monotonically), and bound retention so everything received is shipped out — because a structure that grows with run length will eventually hit the ceiling and halt, and a bigger warehouse only delays it. Ship as fast as you receive, or the floor fills to the rafters.

Visual Explanation — bounded vs unbounded, and the common accumulators

The defining picture is the bounded-vs-unbounded distinction and the common accumulators that grow without bound.

The figure shows bounded versus unbounded and the common accumulators. Bounded working set (the success-colored — the goal): holds only the outstanding/in-transit set — a fixed size regardless of run length, does not grow with time. Unbounded: queues that never drain (the warning-colored — the canonical leak): a scoreboard or FIFO that pushes faster than it pops, or never removes unmatched entries. Unbounded: coverage + histories (brand-colored): coverage databases with fine-grained bins; histories and logs retained for debug and never pruned. Unbounded: unbounded keys + copies (default-colored): associative arrays keyed by every address/ID; unnecessary copies of large transactions. The crucial reading is the binary distinction between the success-colored bounded working set and the three unbounded categories: a bounded structure's size is a function of the outstanding work (which stays small in steady state), while an unbounded structure's size is a function of cumulative simulation time (which grows without limit). This is the property that determines whether a testbench survives a long run: bounded survives any run length; unbounded dies once the run is long enough to cross the ceiling. The common accumulators are all the same mistake in different forms — retaining something proportional to total throughput rather than outstanding work: a queue that keeps every transaction instead of only the unmatched; coverage that keeps a bin per distinct value when the value space is huge; a history that keeps every transaction "for debug"; an array keyed by every ID ever seen; copies kept when references would do. The fix for all of them is the same: bound the retention — retain only the outstanding set, free/prune/cap the rest. The diagram is the memory-safety distinction: bounded (fixed working set, survives) versus unbounded (grows with time, OOMs), with the common accumulators all being forms of retaining cumulative rather than outstanding state. Memory is safe when bounded — a fixed working set independent of run length — and dies when unbounded; the common accumulators all retain cumulative throughput instead of outstanding work, and the fix is to bound retention.

RTL / Simulation Perspective — the unbounded queue and bounding it

In code, the canonical leak is the queue that never fully drains, and the fix is bounding the retention. The example shows the leak and the bound.

// ✗ UNBOUNDED: expected transactions pushed, but unmatched ones are NEVER removed → grows with time
class leaky_scoreboard extends uvm_scoreboard;
  my_txn expected_q[$];                       // the queue — should hold only OUTSTANDING expected
  function void write_expected(my_txn t);
    expected_q.push_back(t);                   // push every expected transaction
  endfunction
  function void write_actual(my_txn a);
    foreach (expected_q[i]) if (expected_q[i].matches(a)) begin
      expected_q.delete(i); return;            // matched → removed (good)
    end
    // ✗ if NO match (a dropped/never-arriving response), the expected entry stays FOREVER
  endfunction                                   // → a few unmatched per million → queue grows unboundedly → OOM after hours
endclass
 
// ✓ BOUNDED: cap retention — age out stale unmatched entries (and flag them), keep the working set fixed
class bounded_scoreboard extends uvm_scoreboard;
  typedef struct { my_txn t; int unsigned age; } entry_t;
  entry_t expected_q[$];
  function void write_actual(my_txn a);
    foreach (expected_q[i]) if (expected_q[i].t.matches(a)) begin expected_q.delete(i); return; end
  endfunction
  function void age_and_prune();                // called periodically (e.g., per N transactions)
    foreach (expected_q[i]) expected_q[i].age++;
    for (int i = expected_q.size()-1; i >= 0; i--)
      if (expected_q[i].age > MAX_AGE) begin
        `uvm_error("SB", "expected transaction never matched (timed out)")  // flag the real issue
        expected_q.delete(i);                   // ✓ remove → the queue stays BOUNDED to the outstanding set
      end
  endfunction
endclass
 
// === MEASURE: profile heap over time to find WHICH structure grows monotonically ===
//   take heap snapshots at t1 and t2 (e.g., hour 1 and hour 3) → diff → the structure that GREW is the leak
//   the simulator's memory profiler / +mem reports rank objects by count — the growing one is the accumulator

The code shows the unbounded queue and the bound. Unbounded (leaky_scoreboard): expected_q pushes every expected transaction, and write_actual removes a matched one — but if no match (a dropped or never-arriving response), the expected entry stays forever. So a few unmatched per million make the queue grow unboundedly → OOM after hours. Bounded (bounded_scoreboard): cap the retention by aging out stale unmatched entries — age_and_prune (called periodically) increments each entry's age and removes (and flags with uvm_error) any older than MAX_AGE — so the queue stays bounded to the outstanding set, and the never-matched entries are both freed and reported (the prune surfaces the real issue instead of silently leaking). Measure: profile heap over time — take snapshots at two times (hour 1, hour 3), diff them, and the structure that grew is the leak; the simulator's memory profiler ranks objects by count, and the growing one is the accumulator. The shape to carry: the canonical leak is a queue whose unmatched entries are never removed, growing unboundedly with time; the fix is to bound retention — cap, age out, or prune so the working set is the outstanding set — and the pruning also flags the real problem (the never-matched transaction) rather than hiding it as a slow leak. Bound retention so queues hold only the outstanding set — age out or prune stale entries (flagging them) — and find the leak by profiling the heap over time to see which structure grows.

Verification Perspective — the leak hides in short tests

The defining danger is the time-dependence: an unbounded leak passes in short tests and OOMs only on long runs, so it escapes development and strikes regression. Seeing why clarifies why you test at scale and profile over time.

The figure shows the leak hiding in short tests and striking on long runs. An unbounded structure grows linearly with simulation time. In a short test (development, a quick smoke run), it grows only a little, stays well under the memory ceiling, and the test passes — the leak is invisible. In the long nightly regression (millions of transactions over hours), it grows large, crosses the memory ceiling, and OOM-kills the test. The verification insight is that the same leak is harmless at short run lengths and fatal at long ones — so the failure correlates with run length, not with what the test does. This inverts the usual debugging signal: normally a failure points at the test that triggers it, but a memory leak is triggered by duration, so it appears in whatever test runs longest (often a soak or regression test), regardless of which test contains the leaky code — the leaky scoreboard might be exercised by every test, but only the long one OOMs. The warning leak → default short → success pass path and the warning leak → default long → warning OOM path show the same structure producing opposite outcomes based purely on run length. This is why memory safety cannot be confirmed by short tests: a passing smoke test proves nothing about memory — the leak is there, just not yet large enough to kill. And it's why you profile heap growth over time (does it grow?) rather than checking peak memory once (is it under the limit right now?): a one-time peak check on a short run passes a leak that will OOM on a long run, while a growth-over-time check catches it (the slope is non-zero). The crucial discipline is to test memory at scale (long soak runs) and measure growth (not just peak), because the leak's defining property — growth with time — is only visible over time. The diagram is the time-dependence: a leak passes short and OOMs long, so confirm memory safety with long runs and growth measurement, not short tests and peak checks. A leak is harmless at short run lengths and fatal at long ones, so it hides in development and strikes regression — confirm memory safety by measuring heap growth over long runs, not peak memory once on a short test.

Runtime / Execution Flow — finding and bounding the leak

At run time, finding a leak follows a measure-then-bound discipline: recognize the time-correlated OOM, profile heap over time, find the growing structure, bound its retention. The flow shows it.

The flow shows finding and bounding a leak. Symptom (step 1): an OOM that correlates with run length — dies on long runs, passes on short — a leak, not an infrastructure or functional bug. Profile (step 2): snapshots at two points, or the memory profiler ranking objects by count — diff them. Find (step 3): the structure that grew monotonically — the accumulator retains cumulative state (every transaction/bin) instead of outstanding state. Bound → re-profile (step 4): free/pop/prune to the outstanding set; re-profile a long run to confirm the heap is flat over time. The runtime insight is that the diff between two heap snapshots is the key technique: a single snapshot tells you what's in memory now (which might be fine), but the diff between two times tells you what grew, and what grew is the leak. This is the memory analogue of profiling for runtime (28.1): don't guess which structure leaks — measure which one grows. The finding step (3) often reveals the cause directly: the growing structure's contents show what it's retaining (a queue full of old unmatched transactions, a coverage model with millions of bins), which points at the retention bug. The bounding step (4) is the fix — make the retention proportional to outstanding work, not cumulative throughput — and the re-profile is the confirmation: a bounded fix makes the heap flat over time (the slope goes to zero), which a long re-profile verifies. The crucial point is that the confirmation must be over time on a long run: a fixed leak and an unfixed leak both look fine in a short run, so only a long run with growth measurement proves the fix worked (the heap stopped growing). The brand (symptom) → success (profile/find) → warning (bound/confirm) flow shows measurement driving the fix and confirming it. The flow is the leak diagnostic: time-correlated OOM → profile heap over time → find what grew → bound retention → re-profile flat. Find a leak by recognizing the run-length-correlated OOM, profiling the heap over time to see which structure grew, bounding its retention to the outstanding set, and re-profiling a long run to confirm the heap is flat.

Waveform Perspective — the queue that only grows

The unbounded growth is visible as a retained count that only ratchets up — never returning to a baseline, unlike a bounded queue that oscillates around a steady level. The waveform shows the monotonic growth.

The waveform shows the queue that only grows. Transactions arrive (txn_in) and most are matched and freed (freed pulses), which would keep a healthy queue bounded. But a fraction are never matched, so each time, the retained queue depth ratchets up by one and never comes back down — queue_depth climbs monotonically (01, 02, 03, 04, 05) rather than oscillating around a steady level. The crucial reading is the shape of queue_depth: it only rises — never falls back — and that monotonic, ratcheting climb is the signature of unbounded accumulation. A bounded queue would rise and fall as transactions are matched and freed, hovering around the outstanding count (the in-transit set); this one only rises, because the unmatched entries are never removed. The picture to carry is the contrast between oscillating and ratcheting: a healthy working set oscillates (fills and drains, steady on average) — its depth is bounded; a leaking one ratchets (fills faster than it drains, or never drains some entries) — its depth grows without bound. Reading the queue depth over time — does it return toward a baseline, or only climb? — is the diagnostic: oscillating around steady → bounded → safe; climbing monotonically → unbounded → will OOM on a long enough run. Over the short window shown, the climb is small (01 to 05) and harmless — but extrapolated over millions of transactions and hours, that same slope crosses the memory ceiling and OOM-kills the test. This is why the signature is the slope, not the level: the level (5) is fine now, but the non-zero slope (always rising) guarantees a future OOM. A retained count that only ratchets up, never draining back toward a baseline, is the signature of an unbounded leak — a bounded queue oscillates around a steady level, while a leaking one climbs monotonically and will OOM on a long enough run; watch the slope, not the level.

DebugLab — the scoreboard queue that OOM-killed the regression after four hours

✗ UNBOUNDED — unmatched expected entries are NEVER removed:
  function void write_actual(my_txn a);
    foreach (expected_q[i]) if (expected_q[i].matches(a)) begin expected_q.delete(i); return; end
    // if NO match (a corner-case response that never arrives), the entry stays in expected_q FOREVER
  endfunction
  // a few unmatched per million transactions → expected_q grows ~linearly with run time
  // short test: queue small → PASSES.  4-hour soak: queue huge → crosses ceiling → OOM (DebugLab)
  // more memory just delays the OOM (bigger warehouse, same pile-up)
 
✓ BOUNDED — age out + flag stale unmatched entries → working set stays the OUTSTANDING set:
  function void age_and_prune();   // periodically
    for (int i = expected_q.size()-1; i >= 0; i--)
      if (++expected_q[i].age > MAX_AGE) begin
        `uvm_error("SB", "expected transaction never matched — the REAL bug, now surfaced")
        expected_q.delete(i);      // remove → queue bounded; AND the dropped-response bug is now reported
      end
  endfunction
 
  // FOUND BY: heap snapshots at hour 1 vs hour 3 → diff → expected_q grew monotonically → the leak

Common Mistakes

• Retaining cumulative state instead of outstanding state. A queue or history that keeps every transaction grows without bound; retain only the outstanding/in-transit set.
• Never removing unmatched entries. A scoreboard that only removes matched entries leaks the unmatched ones forever; age out and flag stale entries.
• Confirming memory safety with short tests. A leak passes every short run; confirm with long soak runs and heap-growth measurement, not peak memory once.
• Reading a time-correlated OOM as infrastructure. An OOM that depends on run length is a leak; don't blame the farm or chase a hang.
• Adding memory instead of bounding retention. More RAM only delays the OOM; the fix is to make the working set bounded, not the warehouse bigger.
• Unbounded keys and unnecessary copies. Associative arrays keyed by every ID, or copies kept where references would do, accumulate; bound the keys and prefer references.

Senior Design Review Notes

Interview Insights

Exercises

1. Bounded or unbounded. For each — a scoreboard holding unmatched expected transactions, a queue of in-flight requests drained on response, a per-address associative array, a transaction history capped at 1000 entries — say whether memory is bounded or unbounded and why.
2. Diagnose the OOM. Given a soak test that OOMs at hour four but passes when shortened, and dies later with more memory, explain why this is a leak and how you'd find it.
3. Bound the queue. Given a scoreboard that only removes matched entries, add an aging-and-pruning mechanism and explain how it bounds memory and surfaces the dropped-response bug.
4. Confirm the fix. Describe the measurement that confirms a memory fix worked, and why it must be over time on a long run.

Summary

• Memory usage optimization is finding and bounding the structures that grow without bound, so the testbench retains a fixed-size working set regardless of run length.
• Memory is a hard ceiling, unlike runtime: an unbounded structure OOM-kills the test (doesn't slow it), and it's time-dependent — it passes in short tests and strikes only on long runs, hiding in development and looking like infrastructure.
• The dominant cause is unbounded accumulation: in a GC language, a leak is a retained reference (the GC can't reclaim a referenced object), so retaining cumulative state (every transaction, every bin) grows with time — the canonical leak is a scoreboard queue that never removes unmatched entries.
• Find the leak by profiling the heap over time (diff two snapshots; the structure that grew monotonically is the accumulator), not by guessing; the signature is the slope (growth), not the level.
• The durable rule of thumb: make memory bounded — a fixed working set that retains only the outstanding (in-transit) set, not the cumulative history — because an unbounded structure that retains cumulative state grows with simulation time and OOM-kills the test on a long enough run while passing every short test, and more memory only delays the death; recognize a run-length-correlated OOM as a leak, profile the heap over time to find the structure that grew monotonically, and bound its retention (free, pop, cap, or age-out and flag what you prune, which also surfaces the underlying bug) — ship everything you receive, don't buy a bigger warehouse.

Next — Logging Optimization: logging sits at the intersection of runtime and memory — it costs CPU to format and write, and it accumulates on disk and in memory. The next chapter focuses on the reporting system as a performance concern: why verbose logging dominates both runtime and storage, how to control verbosity and message construction so logs cost nothing when disabled, how to log enough to debug without drowning the simulation, and how to keep the reporting system from becoming the bottleneck it so often is.