AMBA AXI · Module 13

Outstanding Depth & Buffering

Sizing AXI outstanding depth and buffers to hide latency — the bandwidth-latency product, the throughput-vs-depth curve that saturates at the BLP, buffer structures, and depth as a path-minimum property.

Chapters 13.1–13.2 established the two performance axes — bandwidth and latency — and the principle that latency is hidden by keeping transactions in flight. This chapter makes that quantitative: how much outstanding depth do you actually need, and how much buffering does it require? The answer is the bandwidth-latency product (BLP): depth must equal throughput × latency to keep the data channel busy through the latency. Size it right and you reach peak; too little and you're latency-bound; too much wastes area. This chapter covers the BLP, the throughput-vs-depth curve, the buffering it implies, and depth as a path-minimum property.

1. Required Depth = the Bandwidth-Latency Product

From Little's law (Chapter 8.1): throughput = outstanding ÷ latency, so to reach a target throughput you need:

Outstanding depth ≥ Throughput × Latency — the bandwidth-latency product (BLP).

In beat terms: to keep the data channel transferring a beat every cycle through a latency of L cycles, you need enough transactions in flight to cover L — roughly depth ≥ L / (beats per transaction) transactions, or L beats of data in flight. Concretely, a 20-cycle round-trip latency on a path that should transfer one beat/cycle needs ~20 beats in flight; with 16-beat bursts that's ~2 outstanding transactions minimum, but with single-beat transactions it's ~20 outstanding.

So the BLP tells you the minimum depth to be bandwidth-bound rather than latency-bound. Below it, the data channel idles waiting for responses (throughput = depth/L < peak); at or above it, the channel stays busy (throughput = peak). Sizing depth is fundamentally computing the BLP for the path's latency and target bandwidth.

Figure 1 — required depth = the bandwidth-latency product. By Little's law, outstanding depth ≥ throughput × latency keeps the data channel busy through the latency. Below the BLP you're latency-bound (throughput = depth ÷ latency); at or above it, bandwidth-bound (throughput = peak). The BLP is the minimum depth to saturate the path.

2. Depth Fills the Bubbles

The mechanism, on the wire: issuing several transactions ahead lets responses stream back continuously, filling what would otherwise be idle cycles:

depth-vs-latency — issuing ahead keeps the data channel busy

8 cycles

Figure 2 — depth-vs-latency: issuing 3 transactions ahead. The addresses go out back-to-back; after the first-word latency, read data returns continuously (D0…D5) because there's always a next transaction's data ready — the depth fills the latency gap. At depth 1, the channel would idle for the full latency between each transaction; sufficient depth keeps it busy.

3. The Throughput-vs-Depth Curve

The relationship between depth and achieved throughput has a characteristic shape:

Below the BLP: throughput rises linearly with depth (throughput = depth ÷ latency) — every added outstanding transaction fills more idle cycles. You're latency-bound.
At the BLP: throughput reaches peak (the data channel is fully utilized). This is the knee of the curve.
Above the BLP: throughput plateaus — extra depth adds no throughput (the channel is already saturated), only more buffering/area. Diminishing returns.

So the sizing target is the knee: enough depth to hit the BLP (peak throughput), no more. Below it you leave bandwidth on the table; above it you waste silicon on buffers that never help. Sweeping depth and finding where throughput plateaus is the practical way to size it (and validate the Little's-law model) — which is exactly the depth-vs-throughput verification sweep from Chapter 13.1.

Figure 3 — the throughput-vs-depth curve. Below the BLP, throughput rises linearly with depth (latency-bound — each transaction fills bubbles). At the BLP (the knee), throughput hits peak (channel saturated). Above the BLP, it plateaus — extra depth only wastes buffering. Size to the knee: enough depth for peak, no more.

4. Buffering and the Path-Minimum

Outstanding depth requires buffering to hold the in-flight transactions:

Data buffers — the read-data and write-data in flight must be held (a FIFO sized roughly to depth × burst size of data, to absorb the data of outstanding transactions across the latency).
Tracking structures — per-outstanding-transaction state (ID, address, routing) the manager/interconnect maintains to match responses — sized to the outstanding depth.

So sizing depth implies sizing buffers: more depth → more data FIFO and more tracking entries. This is the area cost of latency hiding, and it's why depth is sized to the knee (BLP), not higher — buffers past the BLP are pure waste.

And depth is a path-minimum property (Chapter 8.5): the effective outstanding depth is the smallest of the manager's issuing capability, the interconnect's per-path buffering, and the subordinate's acceptance capability. A deep manager behind a shallow interconnect achieves only the shallow depth. So sizing means ensuring every stage along the path can hold the BLP — the bottleneck stage caps the achievable depth, hence the throughput. Budget the buffering end-to-end, not just at the manager.

Figure 4 — buffering and the path-minimum. Outstanding depth needs data buffers (≈ depth × burst size) and tracking entries (≈ depth) — the area cost of hiding latency, sized to the BLP knee. Effective depth is the minimum along the path (manager issue, interconnect buffering, subordinate acceptance); the shallowest stage caps it, so the BLP buffering must be provided at every stage.

5. Common Misconceptions

Myths to unlearn

"More outstanding depth always helps." Only up to the BLP (the knee). Beyond it, throughput plateaus and extra depth just wastes buffering area.
"Depth is set by the manager alone." It's the minimum along the path — manager issuing, interconnect buffering, and subordinate acceptance. The shallowest stage caps the effective depth (Chapter 8.5).
"The BLP depends only on latency." It's throughput × latency — both matter. Higher target throughput or higher latency raises the required depth.
"Depth and buffering are separate concerns." Depth requires buffering (data FIFOs + tracking entries) to hold the in-flight transactions — sizing depth is sizing buffers.
"Deeper buffers are always safer." Past the BLP they add area and (for some structures) latency without throughput benefit. Size to the knee, not to infinity.
"Long bursts don't change the depth needed." They do — with long bursts each transaction covers many beats, so fewer transactions are needed to hold the BLP's worth of beats in flight (depth in transactions ≈ BLP-beats ÷ burst-length).

6. Debugging Insight

🔍 Debugging insight — depth shortfalls and buffer bottlenecks

Throughput-vs-depth problems have clear signatures:

Throughput below peak, data channel idling (latency-bound)? Depth below the BLP. Compute throughput_target × latency; if the achieved depth is less, that's the cap. Raise depth at the limiting stage.
Manager configured deep but throughput still capped? A shallower stage along the path — interconnect per-path buffering or subordinate acceptance (Chapter 8.5). Measure achieved depth at each hop; the smallest is the bottleneck.
Throughput doesn't improve when you add depth? You're already at/above the BLP (the plateau) — the limiter is elsewhere (bandwidth ceiling, contention, narrow transfers). Adding depth past the knee does nothing.
Latency rose (e.g., added a CDC/converter) and throughput dropped? The BLP grew with the latency, so the previously-sufficient depth is now below it. Re-size depth to the new (larger) BLP.
Area/buffers large but throughput at peak? Possibly over-sized past the BLP — buffers beyond the knee are wasted; you can trim depth to the knee to save area.

The triage: idle-and-slow → depth below BLP (raise it at the limiting stage); deep-but-capped → path-minimum bottleneck; depth-doesn't-help → at the plateau (limiter elsewhere); latency-grew-slowed → BLP grew (re-size). Always compute the BLP and compare to achieved depth per stage.

7. Verification Insight

✅ Verification insight — the depth-vs-throughput sweep and per-stage depth

The defining check is the depth-vs-throughput sweep: vary the outstanding depth (and/or the latency) and measure throughput, confirming the curve rises linearly below the BLP and plateaus at/above it — this both validates the Little's-law model and locates the knee (the depth needed for the target bandwidth). Assert that at the design's configured depth, throughput meets the target under the path's real latency (using a latency-accurate slave model — a zero-latency testbench slave hides depth problems entirely). Verify the path-minimum: measure achieved outstanding depth at each stage (manager, interconnect, subordinate) and confirm none caps below the BLP. Check buffer sizing: the data FIFOs and tracking structures hold the configured depth without overflow (and back-pressure correctly when full, not drop).

Coverage should include: a depth sweep from 1 to beyond the BLP (find the knee); the design depth at realistic and worst-case latency (confirm peak); each stage's depth under load (path-minimum); and full-buffer conditions (back-pressure, no overflow/drop). The highest-value test is the depth sweep against a latency-accurate slave — it proves the design's depth actually achieves the target bandwidth at the real latency, which a fixed-depth functional test never shows. Since depth is sized from the BLP, verifying the depth-vs-throughput relationship under real latency is how you confirm the latency-hiding actually works — the difference between a path that hits peak and one that ships latency-bound.

8. Interview Questions

9. Summary

Outstanding depth is sized from the bandwidth-latency product: by Little's law, depth ≥ throughput × latency to keep the data channel busy through the latency — enough transactions in flight to fill the cycles a single transaction's latency would otherwise leave idle. The throughput-vs-depth curve rises linearly below the BLP (latency-bound), reaches peak at the BLP knee, and plateaus above it (wasted buffering), so you size to the knee — enough for peak, no more. Depth requires buffering (data FIFOs ≈ depth × burst-size, plus tracking entries ≈ depth), so sizing depth is sizing buffers, and it's the area cost of latency hiding.

Critically, depth is a path-minimum (Chapter 8.5): the effective depth is the smallest of manager-issue, interconnect-buffering, and subordinate-acceptance, so the BLP must be provided at every stage or the shallowest caps throughput. The whole thing is a performance triangle — throughput, latency, and depth bound by Little's law — where depth is the lever to hide a given latency and reach a target throughput, and when depth is infeasible you must reduce latency instead. Verify with the depth-vs-throughput sweep against a latency-accurate slave to find the knee and confirm peak at the configured depth. Next: burst efficiency — how burst length and narrow transfers shape the achievable throughput within this framework.

10. What Comes Next

You've sized the depth that hides latency; next, the efficiency of the bursts themselves:

13.4 — Burst Efficiency (coming next) — how burst length and narrow transfers affect achievable efficiency, refining the bandwidth picture.
13.5 — Backpressure & Pipeline Effects (coming soon) — quantifying how stalls and pipelining shape throughput.

Previous: 13.2 — Latency Analysis. Related: 8.1 — Why Outstanding Transactions Exist for the Little's-law foundation, and 8.5 — Interconnect Implications for the path-minimum. For the broader protocol catalog, see the AMBA family overview doc.