Outstanding Depth & Buffering

Chapters 13.1–13.2 established the two performance axes — bandwidth and latency — and the principle that latency is hidden by keeping transactions in flight. This chapter makes that quantitative: how much outstanding depth do you actually need, and how much buffering does it require? The answer is the bandwidth-latency product (BLP): depth must equal throughput × latency to keep the data channel busy through the latency. Size it right and you reach peak; too little and you're latency-bound; too much wastes area. This chapter covers the BLP, the throughput-vs-depth curve, the buffering it implies, and depth as a path-minimum property.

1. Required Depth = the Bandwidth-Latency Product

From Little's law (Chapter 8.1): throughput = outstanding ÷ latency, so to reach a target throughput you need:

Outstanding depth ≥ Throughput × Latency — the bandwidth-latency product (BLP).

In beat terms: to keep the data channel transferring a beat every cycle through a latency of L cycles, you need enough transactions in flight to cover L — roughly depth ≥ L / (beats per transaction) transactions, or L beats of data in flight. Concretely, a 20-cycle round-trip latency on a path that should transfer one beat/cycle needs ~20 beats in flight; with 16-beat bursts that's ~2 outstanding transactions minimum, but with single-beat transactions it's ~20 outstanding.

So the BLP tells you the minimum depth to be bandwidth-bound rather than latency-bound. Below it, the data channel idles waiting for responses (throughput = depth/L < peak); at or above it, the channel stays busy (throughput = peak). Sizing depth is fundamentally computing the BLP for the path's latency and target bandwidth.

2. Depth Fills the Bubbles

The mechanism, on the wire: issuing several transactions ahead lets responses stream back continuously, filling what would otherwise be idle cycles:

3. The Throughput-vs-Depth Curve

The relationship between depth and achieved throughput has a characteristic shape:

• Below the BLP: throughput rises linearly with depth (throughput = depth ÷ latency) — every added outstanding transaction fills more idle cycles. You're latency-bound.
• At the BLP: throughput reaches peak (the data channel is fully utilized). This is the knee of the curve.
• Above the BLP: throughput plateaus — extra depth adds no throughput (the channel is already saturated), only more buffering/area. Diminishing returns.

So the sizing target is the knee: enough depth to hit the BLP (peak throughput), no more. Below it you leave bandwidth on the table; above it you waste silicon on buffers that never help. Sweeping depth and finding where throughput plateaus is the practical way to size it (and validate the Little's-law model) — which is exactly the depth-vs-throughput verification sweep from Chapter 13.1.

4. Buffering and the Path-Minimum

Outstanding depth requires buffering to hold the in-flight transactions:

• Data buffers — the read-data and write-data in flight must be held (a FIFO sized roughly to depth × burst size of data, to absorb the data of outstanding transactions across the latency).
• Tracking structures — per-outstanding-transaction state (ID, address, routing) the manager/interconnect maintains to match responses — sized to the outstanding depth.

So sizing depth implies sizing buffers: more depth → more data FIFO and more tracking entries. This is the area cost of latency hiding, and it's why depth is sized to the knee (BLP), not higher — buffers past the BLP are pure waste.

And depth is a path-minimum property (Chapter 8.5): the effective outstanding depth is the smallest of the manager's issuing capability, the interconnect's per-path buffering, and the subordinate's acceptance capability. A deep manager behind a shallow interconnect achieves only the shallow depth. So sizing means ensuring every stage along the path can hold the BLP — the bottleneck stage caps the achievable depth, hence the throughput. Budget the buffering end-to-end, not just at the manager.

5. Common Misconceptions

6. Debugging Insight

7. Verification Insight

8. Interview Questions

9. Summary

Outstanding depth is sized from the bandwidth-latency product: by Little's law, depth ≥ throughput × latency to keep the data channel busy through the latency — enough transactions in flight to fill the cycles a single transaction's latency would otherwise leave idle. The throughput-vs-depth curve rises linearly below the BLP (latency-bound), reaches peak at the BLP knee, and plateaus above it (wasted buffering), so you size to the knee — enough for peak, no more. Depth requires buffering (data FIFOs ≈ depth × burst-size, plus tracking entries ≈ depth), so sizing depth is sizing buffers, and it's the area cost of latency hiding.

Critically, depth is a path-minimum (Chapter 8.5): the effective depth is the smallest of manager-issue, interconnect-buffering, and subordinate-acceptance, so the BLP must be provided at every stage or the shallowest caps throughput. The whole thing is a performance triangle — throughput, latency, and depth bound by Little's law — where depth is the lever to hide a given latency and reach a target throughput, and when depth is infeasible you must reduce latency instead. Verify with the depth-vs-throughput sweep against a latency-accurate slave to find the knee and confirm peak at the configured depth. Next: burst efficiency — how burst length and narrow transfers shape the achievable throughput within this framework.

10. What Comes Next

You've sized the depth that hides latency; next, the efficiency of the bursts themselves:

• 13.4 — Burst Efficiency (coming next) — how burst length and narrow transfers affect achievable efficiency, refining the bandwidth picture.
• 13.5 — Backpressure & Pipeline Effects (coming soon) — quantifying how stalls and pipelining shape throughput.

Previous: 13.2 — Latency Analysis. Related: 8.1 — Why Outstanding Transactions Exist for the Little's-law foundation, and 8.5 — Interconnect Implications for the path-minimum. For the broader protocol catalog, see the AMBA family overview doc.