Buffering Strategies

24.1 sized a retry buffer. 24.2 sized a merge buffer and a replay buffer. 24.3 sized a scheduler queue. 24.4 sized an elastic buffer at every clock boundary. Every one of those was sized alone, and every one of them comes out of the same SRAM.

That is the chapter. A buffer is not a number in a parameter list — it is depth bought from a fixed budget and paid for twice, once in the area it occupies and once in the latency every request behind it inherits. Sizing each queue correctly and independently is how a device overspends its SRAM by fifty units while every individual number is right.

The failure is also the one that survives simulation, because a queue that never overflowed in a hundred million cycles of the traffic somebody happened to write is not a queue that was sized — which is section 14, and it is the whole chapter.

1. The Engineering Problem — Depth Is Bought And Paid For

A buffer that does not cover a round trip throttles the link. Sixty-four entries against a sixty-four-cycle round trip at one beat a cycle is exactly enough; thirty-two runs the link at half rate and no error is ever reported. Section 5.

A watermark asserted at the depth is asserted too late. Sixteen beats already in flight arriving at a queue with no headroom left is an overflow that the watermark existed to prevent. Section 6.

One buffer serialises what two overlap. A forty-cycle fill and a twenty-four-cycle drain cost sixty-four together and forty apart — a 37% saving for one more buffer. Section 7.

A burst is absorbed or it spills. Five hundred and twelve beats arriving four a cycle and draining one a cycle needs three hundred and eighty-four entries held, and a buffer of two hundred and fifty-six spills a hundred and twenty-eight. Section 8.

And every one of those numbers is drawn against the same budget. Two hundred, one hundred and fifty, three hundred and one hundred against seven hundred is fifty over — while the largest single queue, the one a designer sizing in isolation sees, is three hundred and fits comfortably. Section 13.

This chapter against 24.4, stated precisely. That one owns the boundaries between blocks. This one owns the storage that sits on them — which is why every model here converts a traffic property into a depth, and why section 14's weak definition is a simulation report rather than a sign-off.

2. The One-Sentence Model

A device's buffers are sized when every depth covers a round trip, every watermark leaves room for the flight, the burst profile fits, the latency the depth adds is inside the budget, and every queue is counted against one SRAM budget — and "no queue overflowed in simulation" is none of those five.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block that converts one traffic property into one depth, or one depth into one cost. A real buffer is a dual-port SRAM or a register file with pointer logic, ECC, a bypass path, clock gating and a BIST wrapper, and none of that is reproduced. What is reproduced is the arithmetic that decides how deep it is, and the shape of the mistake when the arithmetic is skipped.

Three simplifications are worth stating. Section 5 uses a fixed round trip where a real one varies with congestion. Section 12 uses a mean service time in place of a distribution, understating the tail that sets a latency budget. Section 11 prices control as a flat percentage of the array, where it really depends on port count. In each case the conclusion is the same and the model is abbreviated.

Each model is built twice — a correct build and a broken build selected by a parameter. Every broken build here is a buffer sized by capacity rather than by traffic: depth without a round trip, a watermark without a flight, a burst assumed smooth, a pool assumed private, storage without control, a queue without its neighbours. Each is the number a spreadsheet produces, which is why they reach a design review unchallenged.

Figure 1 — Four queues, four correct depths, four different traffic properties. The largest is three hundred and fits inside seven hundred, which is the number a designer sizing one queue at a time sees. The sum is section 13.

5. RTL 1 — A Buffer Covers A Round Trip Or The Link Runs Slower

// RTL 1 - the bandwidth-delay product. A buffer covers a round trip's worth of
// traffic or the link runs at the fraction it does cover.
module bdp_sizing #(parameter int IGNORE_ROUND_TRIP = 0) (
  input  logic clk, rst_n,
  input  logic        size_it,
  input  logic [15:0] rtt_cycles, beats_per_cycle, entries,
  output logic [15:0] needed, achievable_pct,
  output logic        sufficient,
  output logic [7:0]  n_sizings, n_short,
  output logic        bdp_ignored_err
);
  logic [31:0] n_q, a_q, r_q;
  assign n_q = {16'd0, rtt_cycles} * {16'd0, beats_per_cycle};
  assign needed = (n_q > 32'd65535) ? 16'hFFFF : n_q[15:0];
  // Sizing a buffer by capacity alone ignores how long a beat is in flight.
  assign r_q = (needed == 16'd0) ? 32'd100
             : (({16'd0, entries} * 32'd100) / {16'd0, needed});
  assign a_q = (IGNORE_ROUND_TRIP != 0) ? 32'd100
             : ((r_q > 32'd100) ? 32'd100 : r_q);
  assign achievable_pct = (a_q > 32'd65535) ? 16'hFFFF : a_q[15:0];
  assign sufficient = (achievable_pct >= 16'd100);
  // A buffer smaller than a round trip, reported as running at full rate.
  assign bdp_ignored_err = size_it && (entries < needed) && (achievable_pct == 16'd100);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_sizings <= 8'd0; n_short <= 8'd0;
    end else if (size_it) begin
      n_sizings <= n_sizings + 8'd1;
      if (!sufficient) n_short <= n_short + 8'd1;
    end
  end
endmodule

Six sizings. A sixty-four-cycle round trip unless stated.

Three short when the round trip is counted; none when it is not.

A buffer covers the traffic in flight or it does not. A sender may have as many beats outstanding as the buffer has entries; when they run out it waits for a credit that is still travelling back, and the credit takes a round trip. Sixty-four cycles at one beat a cycle is sixty-four beats in flight, so sixty-four entries keep the sender busy and thirty-two keep it busy half the time. This is the bandwidth-delay product that sizes a TCP window — arithmetic rather than protocol.

Row two is the whole failure mode. Half the entries gives half the rate, and nothing anywhere reports an error — the link is up, no packet is lost, no credit is violated, no timeout fires. The only symptom is a bandwidth number that is half what the datasheet says, and by the time anybody measures it the buffer is in silicon. The capacity-only build reports one hundred percent because one hundred percent is what a buffer that never overflows looks like from inside.

Row three is why oversizing has a ceiling. A hundred and twenty-eight entries against a sixty-four-beat round trip does not run the link at two hundred percent — the extra sixty-four entries buy nothing at all except area and, per section 12, latency. That ceiling is the reason "make it bigger" is not a strategy: it is free of risk and not free of cost, and past the round trip it is pure cost.

6. RTL 2 — A Watermark Has To Leave Room For What Is Already Sent

// RTL 2 - watermarks. Backpressure has to be asserted early enough that the
// traffic already in flight still fits when it arrives.
module watermark_margin #(parameter int ASSERT_WHEN_FULL = 0) (
  input  logic clk, rst_n,
  input  logic        check,
  input  logic [15:0] depth, in_flight, occupancy,
  output logic [15:0] watermark, headroom, worst_occupancy,
  output logic        safe, would_overflow,
  output logic [7:0]  n_checks, n_unsafe,
  output logic        late_watermark_err
);
  // The watermark must leave room for everything already sent.
  assign watermark = (ASSERT_WHEN_FULL != 0) ? depth
                   : ((depth > in_flight) ? (depth - in_flight) : 16'd0);
  assign headroom = (depth > watermark) ? (depth - watermark) : 16'd0;
  assign worst_occupancy = occupancy + in_flight;
  assign would_overflow = (occupancy >= watermark) && (worst_occupancy > depth);
  assign safe = (headroom >= in_flight);
  // A watermark that leaves no room for the traffic already in flight.
  assign late_watermark_err = check && (in_flight != 16'd0) && (depth != 16'd0)
                              && (watermark == depth);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_checks <= 8'd0; n_unsafe <= 8'd0;
    end else if (check) begin
      n_checks <= n_checks + 8'd1;
      if (!safe) n_unsafe <= n_unsafe + 8'd1;
    end
  end
endmodule

Eight checks. A sixty-four-entry queue unless stated.

Two unsafe with an early watermark; seven with a late one.

A watermark is not a full flag. By the time a queue is full, everything the sender already put on the wire is still coming, and it has nowhere to go. The watermark has to be asserted at depth minus the flight, so that the beats already committed still fit when they land. Sixteen in flight against a sixty-four-entry queue means backpressure at forty-eight, and the last sixteen entries are not capacity — they are the landing zone.

Row one is the pair. The same queue, the same traffic, two policies: assert at forty-eight and there are sixteen entries of headroom for sixteen beats in flight; assert at sixty-four and there are zero entries for the same sixteen beats. The second design is not marginal, it is guaranteed to overflow, and it overflows only when the queue actually fills — which is exactly the condition a light simulation never reaches.

Row four is the boundary the sizing rule is built on. Thirty-two in flight against a thirty-two-entry headroom is exactly safe and one more beat in flight is not. A margin computed as "half the depth" or "a comfortable amount" passes here by accident; a margin computed as the flight passes here by construction. The rule is an equality, and equalities are where sizing rules are tested.

Rows five, six and seven are the three relationships between occupancy and the watermark, and a single overflow bit conflates them. Sixty is past the watermark and overflows; forty-eight is exactly on it, with a worst case of exactly the depth, and is the last occupancy that does not; sixty-four is a queue already full. The middle one is the design margin.

7. RTL 3 — Two Buffers Overlap What One Serialises

// RTL 3 - ping-pong against a single buffer. Two buffers let a producer fill
// one while the consumer drains the other; one buffer serialises them.
module ping_pong #(parameter int SINGLE_BUFFER = 0) (
  input  logic clk, rst_n,
  input  logic        run,
  input  logic [15:0] fill_cycles, drain_cycles, transfers,
  output logic [15:0] per_transfer, total_cycles, overlap_pct,
  output logic        overlapped,
  output logic [7:0]  n_runs, n_serial,
  output logic        overlap_ignored_err
);
  logic [31:0] p_q, t_q, o_q;
  logic [15:0] slower;
  assign slower = (fill_cycles > drain_cycles) ? fill_cycles : drain_cycles;
  // With two buffers the pair costs the slower of the two, not the sum.
  assign p_q = (SINGLE_BUFFER != 0) ? ({16'd0, fill_cycles} + {16'd0, drain_cycles})
             : {16'd0, slower};
  assign per_transfer = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  assign t_q = {16'd0, per_transfer} * {16'd0, transfers};
  assign total_cycles = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  assign o_q = (({16'd0, fill_cycles} + {16'd0, drain_cycles}) == 32'd0) ? 32'd0
             : ((({16'd0, fill_cycles} + {16'd0, drain_cycles}
                  - {16'd0, per_transfer}) * 32'd100)
                / ({16'd0, fill_cycles} + {16'd0, drain_cycles}));
  assign overlap_pct = (o_q > 32'd65535) ? 16'hFFFF : o_q[15:0];
  assign overlapped = (per_transfer < (fill_cycles + drain_cycles));
  // Two stages that could have overlapped and were run in series.
  assign overlap_ignored_err = run && (fill_cycles != 16'd0)
                               && (drain_cycles != 16'd0) && !overlapped;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_runs <= 8'd0; n_serial <= 8'd0;
    end else if (run) begin
      n_runs <= n_runs + 8'd1;
      if (!overlapped) n_serial <= n_serial + 8'd1;
    end
  end
endmodule

Five runs. Eight transfers unless stated.

Two serial with ping-pong because one stage was free; all five with one buffer.

One buffer makes the producer wait for the consumer. Fill it, drain it, fill it again — the stages are strictly ordered because they touch the same storage, so the pair costs their sum. Two buffers break the dependency: the producer fills one while the consumer drains the other, and the pair costs the slower rather than the total. Forty and twenty-four is sixty-four in series and forty in parallel — 37% for one more buffer's worth of SRAM.

Row two is the ceiling on the technique. Equal stages overlap perfectly and save exactly half — fifty percent is the most ping-pong can ever return, and it is returned only when the two stages are balanced. Skewed stages return less, and a stage that is twice the other returns a third. That is the whole design rule: ping-pong pays best where the stages are already matched, which is the opposite of where a designer's intuition sends it.

8. RTL 4 — A Burst Is Absorbed Or It Spills

// RTL 4 - burst absorption. A buffer smooths a burst only while the burst is
// shorter than the buffer plus what drains during it.
module burst_absorption #(parameter int ASSUME_SMOOTH = 0) (
  input  logic clk, rst_n,
  input  logic        absorb,
  input  logic [15:0] burst_beats, burst_rate, drain_rate, entries,
  output logic [15:0] burst_cycles, drained, absorbed_needed, spill,
  output logic        absorbs,
  output logic [7:0]  n_bursts, n_spilling,
  output logic        burst_ignored_err
);
  logic [31:0] c_q, d_q, a_q;
  assign c_q = (burst_rate == 16'd0) ? 32'd0
             : ({16'd0, burst_beats} / {16'd0, burst_rate});
  assign burst_cycles = (c_q > 32'd65535) ? 16'hFFFF : c_q[15:0];
  assign d_q = {16'd0, burst_cycles} * {16'd0, drain_rate};
  assign drained = (d_q > 32'd65535) ? 16'hFFFF : d_q[15:0];
  // A burst that arrives faster than it drains has to be held meanwhile.
  assign a_q = (ASSUME_SMOOTH != 0) ? 32'd0
             : ((burst_beats > drained) ? ({16'd0, burst_beats} - {16'd0, drained})
                : 32'd0);
  assign absorbed_needed = (a_q > 32'd65535) ? 16'hFFFF : a_q[15:0];
  assign spill = (absorbed_needed > entries) ? (absorbed_needed - entries) : 16'd0;
  assign absorbs = (spill == 16'd0);
  // A burst above the drain rate reported as needing no buffer.
  assign burst_ignored_err = absorb && (burst_beats > drained)
                             && (absorbed_needed == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_bursts <= 8'd0; n_spilling <= 8'd0;
    end else if (absorb) begin
      n_bursts <= n_bursts + 8'd1;
      if (!absorbs) n_spilling <= n_spilling + 8'd1;
    end
  end
endmodule

Five bursts. Five hundred and twelve beats arriving four a cycle unless stated.

Two spilling when the burst is modelled; none when it is assumed smooth.

Average rate does not size a buffer; the gap between arrival and drain does. Five hundred and twelve beats at four a cycle occupy a hundred and twenty-eight cycles, during which a drain of one a cycle removes a hundred and twenty-eight. The other three hundred and eighty-four have to be somewhere, and that somewhere is the buffer. Dividing total beats by total time gives an average that fits in any depth and sizes the buffer at nothing.

Row one is that mistake spelled out. The smooth model reports zero entries needed for a burst that demands three hundred and eighty-four, and it does so without any arithmetic error — it is answering a different question, namely how much storage a perfectly paced stream needs, which is genuinely none. The two models disagree by the entire depth, and the disagreement is invisible unless somebody states which traffic shape the number assumes.

Row two is the boundary. Three hundred and eighty-four entries against a need of exactly three hundred and eighty-four absorbs, and three hundred and eighty-three does not. Sizing to the boundary is correct and sizing to the boundary is fragile — every parameter feeding it (burst length, arrival rate, drain rate) has to be a worst case rather than a typical one, because at the boundary a ten-percent error in any of them is a spill.

Row three removes the buffer entirely. A drain that matches the arrival rate drains the burst as it arrives, and the correct depth is zero — the honest version of the smooth assumption, right when the drain keeps up and wrong otherwise.

9. RTL 5 — A Shared Pool Is Smaller And One Class Can Starve Another

// RTL 5 - shared against dedicated buffering. One pool is smaller and lets one
// class consume another's space; per-class pools cannot.
module shared_pool #(parameter int SHARE_EVERYTHING = 0) (
  input  logic clk, rst_n,
  input  logic        request,
  input  logic [15:0] pool_entries, own_entries, others_holding, want,
  output logic [15:0] available, granted, refused,
  output logic        grantable,
  output logic [7:0]  n_requests, n_refused,
  output logic        cross_class_err
);
  logic [15:0] pool_free;
  assign pool_free = (pool_entries > others_holding)
                     ? (pool_entries - others_holding) : 16'd0;
  // A shared pool offers whatever nobody else is holding; a dedicated one
  // offers only this class's own reservation.
  assign available = (SHARE_EVERYTHING != 0) ? pool_free : own_entries;
  assign granted = (want > available) ? available : want;
  // granted is a minimum against want, so this cannot underflow.
  assign refused = want - granted;
  assign grantable = (refused == 16'd0);
  // A grant made against entries reserved for another class.
  assign cross_class_err = request && grantable && (want > own_entries);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_requests <= 8'd0; n_refused <= 8'd0;
    end else if (request) begin
      n_requests <= n_requests + 8'd1;
      if (!grantable) n_refused <= n_refused + 8'd1;
    end
  end
endmodule

Five requests. A 256-entry pool, 32 reserved to this class, 200 held elsewhere unless stated.

Three refused with dedicated pools; one with a shared pool.

A shared pool is a smaller pool that works more often, until it does not. Reserving thirty-two entries to each of eight classes costs two hundred and fifty-six entries and guarantees every class thirty-two. Sharing two hundred and fifty-six between them costs the same and gives any single class up to two hundred and fifty-six — but only while the others are idle. The saving is real when the classes are bursty and uncorrelated, and it evaporates when they burst together.

Row one is the trade in one line. The dedicated design refuses sixteen of a forty-eight-entry request while fifty-six entries sit free elsewhere — wasted capacity, visible and quantified. The shared design grants all forty-eight, twenty-four of which are somebody else's guarantee. Neither is wrong: CXL's own class separation exists because on some paths the answer must be "the other class still gets its entries."

Row four is where sharing becomes a deadlock argument. When other classes hold the whole pool the shared design has nothing to offer, while the dedicated design still has its thirty-two unconditionally. If the starved class is the one that returns credits or drains the pool, the device stops — a hang rather than a regression, and the reason a mostly-shared design still reserves a minimum per class.

10. RTL 6 — Store-And-Forward Adds The Packet To Every Hop

// RTL 6 - the cut-through decision. A buffer that stores a whole packet before
// forwarding adds the packet's own transmission time to every hop.
module cut_through #(parameter int STORE_AND_FORWARD = 0) (
  input  logic clk, rst_n,
  input  logic        forward,
  input  logic [15:0] packet_beats, header_beats, hops, budget_cycles,
  output logic [15:0] per_hop, total_cycles, added_cycles,
  output logic        within_budget,
  output logic [7:0]  n_forwards, n_over,
  output logic        store_cost_ignored_err
);
  logic [31:0] p_q, t_q;
  // Cut-through forwards once the header is decoded; store-and-forward waits
  // for the last beat of the packet.
  assign p_q = (STORE_AND_FORWARD != 0) ? {16'd0, packet_beats}
             : {16'd0, header_beats};
  assign per_hop = (p_q > 32'd65535) ? 16'hFFFF : p_q[15:0];
  assign t_q = {16'd0, per_hop} * {16'd0, hops};
  assign total_cycles = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  assign added_cycles = (per_hop > header_beats)
                        ? ((per_hop - header_beats) * hops) : 16'd0;
  assign within_budget = (total_cycles <= budget_cycles);
  // A store-and-forward hop costed as if it forwarded on the header.
  assign store_cost_ignored_err = forward && (packet_beats > header_beats)
                                  && (STORE_AND_FORWARD != 0)
                                  && (added_cycles == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_forwards <= 8'd0; n_over <= 8'd0;
    end else if (forward) begin
      n_forwards <= n_forwards + 8'd1;
      if (!within_budget) n_over <= n_over + 8'd1;
    end
  end
endmodule

Five forwards. A 64-beat packet with a 4-beat header against a 32-cycle budget unless stated.

None over budget with cut-through; three with store-and-forward.

Store-and-forward pays for the packet at every hop it passes. A switch that waits for the last beat before forwarding the first cannot start until the packet has entirely arrived, so each hop costs the packet's whole transmission time rather than its header's. Sixty-four beats across three hops is a hundred and ninety-two cycles against twelve — a factor of sixteen, and it is latency that no amount of bandwidth removes.

The buffering consequence is the point. Cut-through needs enough storage to hold a header and a little slack; store-and-forward needs enough to hold an entire maximum-size packet per port, which is a completely different area number. The forwarding policy is chosen and the buffer size follows, not the other way round, and a team that sizes buffers before choosing the policy has already chosen it.

Row two is where the distinction disappears. A packet no longer than its header costs the same either way — there is nothing to wait for. This matters more than it looks: a fabric carrying mostly small control packets sees almost no difference between the policies, and a fabric carrying bulk data sees a factor of sixteen. The right policy is a property of the traffic, and a device carrying both may want different policies on different virtual channels.

Row four is the boundary. Eight hops of cut-through is exactly thirty-two cycles, exactly the budget, and exactly inside it. A checker written with a strict comparison rejects the last legal topology in the fabric.

11. RTL 7 — A Buffer Is Not Only Its Array

// RTL 7 - what a buffer costs. Storage is an array and the control around it is
// not, so a deep narrow buffer and a shallow wide one price differently.
module buffer_area #(parameter int STORAGE_ONLY = 0) (
  input  logic clk, rst_n,
  input  logic        budget,
  input  logic [15:0] entries, width_bits, bits_per_area, control_pct,
  output logic [15:0] storage_area, control_area, total_area,
  output logic        efficient,
  output logic [7:0]  n_budgets, n_costly,
  output logic        control_ignored_err
);
  logic [31:0] s_q, c_q, t_q;
  assign s_q = (bits_per_area == 16'd0) ? 32'd0
             : (({16'd0, entries} * {16'd0, width_bits}) / {16'd0, bits_per_area});
  assign storage_area = (s_q > 32'd65535) ? 16'hFFFF : s_q[15:0];
  // Pointers, comparators, watermark logic and the read path do not scale with
  // the array and do not disappear when it is small.
  assign c_q = (STORAGE_ONLY != 0) ? 32'd0
             : (({16'd0, storage_area} * {16'd0, control_pct}) / 32'd100);
  assign control_area = (c_q > 32'd65535) ? 16'hFFFF : c_q[15:0];
  assign t_q = {16'd0, storage_area} + {16'd0, control_area};
  assign total_area = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  assign efficient = (control_area <= (storage_area / 16'd4));
  // Control logic that was budgeted at nothing.
  assign control_ignored_err = budget && (control_pct != 16'd0)
                               && (storage_area != 16'd0) && (control_area == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_budgets <= 8'd0; n_costly <= 8'd0;
    end else if (budget) begin
      n_budgets <= n_budgets + 8'd1;
      if (!efficient) n_costly <= n_costly + 8'd1;
    end
  end
endmodule

Six budgets. 256 entries of 64 bits at 128 bits per unit of area unless stated.

One costly when the control is counted; none when it is not.

An SRAM is not what a buffer costs. Around every array sit read and write pointers, a full/empty comparator, section 6's watermark logic, a bypass path, clock gating and, on protocol state, ECC. None of it scales with depth and none of it disappears when the array is small — which is why row one's 256-entry buffer carries twenty-five units of control on a hundred and twenty-eight of storage.

Row one is the twenty-five units a spreadsheet does not have a column for. The storage-only model is not wrong about the array, it is silent about everything else, and a floorplan built on it is short by the control of every buffer in the device. 24.4 §13 found eighteen percent of a device's area in glue nobody owned; this is where a share of that glue actually lives, and the two chapters are describing the same missing column from opposite ends.

Row four is that threshold at equality. Control at exactly a quarter of the storage is exactly efficient and one unit more is not — a policy line rather than a physical one, and the number matters less than having one.

Figure 3 — The left column is complete, correct and twenty-five units short. Every uncounted block is fixed cost, which is why the error is proportionally worst on exactly the small buffers a designer is least worried about.

12. RTL 8 — Depth Bought For Throughput Is Paid For In Latency

// RTL 8 - what a buffer costs in latency. An entry sitting in a queue is an
// entry waiting, so depth bought for throughput is paid for in latency.
module buffer_latency #(parameter int DEPTH_IS_FREE = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] occupancy, service_cycles, base_latency, budget_cycles,
  output logic [15:0] queue_latency, total_latency, headroom,
  output logic        within_budget,
  output logic [7:0]  n_measures, n_over,
  output logic        queue_latency_ignored_err
);
  logic [31:0] q_q, t_q;
  // Every entry ahead is serviced before this one is looked at.
  assign q_q = (DEPTH_IS_FREE != 0) ? 32'd0
             : ({16'd0, occupancy} * {16'd0, service_cycles});
  assign queue_latency = (q_q > 32'd65535) ? 16'hFFFF : q_q[15:0];
  assign t_q = {16'd0, base_latency} + {16'd0, queue_latency};
  assign total_latency = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  assign headroom = (budget_cycles > total_latency)
                    ? (budget_cycles - total_latency) : 16'd0;
  assign within_budget = (total_latency <= budget_cycles);
  // A queue standing in front of a request, costed at nothing.
  assign queue_latency_ignored_err = measure && (occupancy != 16'd0)
                                     && (service_cycles != 16'd0)
                                     && (queue_latency == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_over <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (!within_budget) n_over <= n_over + 8'd1;
    end
  end
endmodule

Five measurements. Two cycles of service per entry, twelve cycles of base latency, a 48-cycle budget.

Two over budget when the queue is counted; none when it is not.

Every entry ahead of a request is a request being served first. Twenty entries at two cycles each is forty cycles of waiting before the memory controller of 24.3 even looks at the request — and the twelve-cycle base latency that the datasheet quotes is now fifty-two. This is Little's law read backwards: depth and throughput and latency are three views of one thing, and buying any two fixes the third.

Row one is why "make the buffer deeper" is not free. A deeper buffer spills less, and every entry it adds is latency for whoever lands behind it. The free-depth model reports twelve cycles — the number a designer gets by measuring an idle machine, which is not when the budget matters.

Row three is the boundary the budget is written on. Eighteen entries is thirty-six cycles of queueing and forty-eight in total, exactly the budget and exactly inside it. Nineteen is not. That converts a latency budget directly into a maximum useful occupancy — and therefore into a watermark, which is section 6's number arrived at from the opposite direction. A queue deeper than its latency budget allows is a queue whose extra entries can never legally be used.

Row five is what a deep buffer does under sustained overload. Two hundred entries at two cycles each is four hundred cycles of queueing — eight times the budget — with every request serviced, in order, without error. This is bufferbloat inside a chip. The fix is not a deeper queue but section 6's backpressure, asserted early enough to keep occupancy inside the budget.

13. RTL 9 — Every Queue Draws From One SRAM Budget

// RTL 9 - the buffers compete. Every queue in the device is drawn from one SRAM
// budget, and sizing each in isolation overspends it.
module buffer_budget #(parameter int SIZE_IN_ISOLATION = 0) (
  input  logic clk, rst_n,
  input  logic        budget,
  input  logic [15:0] replay_area, merge_area, sched_area, elastic_area,
  input  logic [15:0] sram_budget,
  output logic [15:0] claimed_total, largest_claim, overrun,
  output logic        fits,
  output logic [7:0]  n_budgets, n_over,
  output logic        isolation_err
);
  logic [31:0] t_q;
  logic [15:0] a, b;
  assign a = (replay_area > merge_area) ? replay_area : merge_area;
  assign b = (sched_area > elastic_area) ? sched_area : elastic_area;
  assign largest_claim = (a > b) ? a : b;
  // Every queue is real at the same time; sizing one at a time counts the
  // largest and forgets the rest.
  assign t_q = (SIZE_IN_ISOLATION != 0) ? {16'd0, largest_claim}
             : ({16'd0, replay_area} + {16'd0, merge_area}
                + {16'd0, sched_area} + {16'd0, elastic_area});
  assign claimed_total = (t_q > 32'd65535) ? 16'hFFFF : t_q[15:0];
  assign overrun = (claimed_total > sram_budget)
                   ? (claimed_total - sram_budget) : 16'd0;
  assign fits = (claimed_total <= sram_budget);
  // Several queues, budgeted as one.
  assign isolation_err = budget && (claimed_total == largest_claim)
                         && ((replay_area + merge_area + sched_area
                              + elastic_area) > largest_claim);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_budgets <= 8'd0; n_over <= 8'd0;
    end else if (budget) begin
      n_budgets <= n_budgets + 8'd1;
      if (!fits) n_over <= n_over + 8'd1;
    end
  end
endmodule

Five budgets. Replay 200, merge 150, scheduler 300, elastic 100 unless stated.

One over budget when every queue is counted; none when only the largest is.

This is the chapter's thesis in one table. The replay buffer of 24.2 §13 is two hundred units against section 5's round trip; the merge buffer of 24.2 §9 is a hundred and fifty against a write burst; the scheduler queue of 24.3 §12 is three hundred against a reorder window; the elastic buffers of 24.4 §10 are a hundred against a clock ratio. Every number is correct, defensible and derived from real traffic. Their sum is seven hundred and fifty against seven hundred.

Row one is why nobody notices. Each queue is owned by a different engineer, sized in a different chapter, against a different traffic property, and reviewed in a different meeting. The largest single claim is three hundred and it fits inside seven hundred with room to spare — which is the number every one of those four reviews sees. The sum is not any individual's number, and it is the only number that matters to the floorplan.

Row three is the boundary and it is the realistic one. A claim of exactly seven hundred and fifty against exactly seven hundred and fifty fits — and there is nothing left. That is where most devices land after the negotiation, so every later depth increase from any team is a floorplan change. A budget met exactly cannot absorb a late discovery, and late discoveries are what sections 5 through 12 are made of.

14. RTL 10 — A Device's Buffering Assembled

// RTL 10 - a device's buffering assembled. Everything that must hold before a
// queue that never overflows is a queue that was sized rather than guessed.
module buffering_model #(parameter int NEVER_OVERFLOWS = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       never_overflows,     // no queue has been seen to overflow
  input  logic       bdp_covered,         // depth covers a round trip
  input  logic       watermark_early,     // backpressure leaves room for the flight
  input  logic       bursts_absorbed,     // the burst profile fits
  input  logic       latency_budgeted,    // the depth's latency is in the budget
  input  logic       sram_budgeted,       // every queue is in one budget
  output logic       sized,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_sized,
  output logic       false_sizing_err
);
  assign fail_mask[0] = ~never_overflows;
  assign fail_mask[1] = ~bdp_covered;
  assign fail_mask[2] = ~watermark_early;
  assign fail_mask[3] = ~bursts_absorbed;
  assign fail_mask[4] = ~latency_budgeted;
  assign fail_mask[5] = ~sram_budgeted;
  // The never-overflows build is what a simulation report says.
  assign sized = (NEVER_OVERFLOWS != 0) ? never_overflows : (fail_mask == 6'd0);
  assign false_sizing_err = evaluate && sized && (fail_mask != 6'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_sized <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (sized) n_sized <= n_sized + 8'd1;
    end
  end
endmodule

Seven configurations.

One sized under the full model; six under the simulation report.

"No queue overflowed" is the weak definition, and it is the one every project actually uses. It is what a regression reports, it is what a coverage summary shows, and it is a statement about the traffic somebody wrote rather than about the depths somebody chose. Five of the six properties below it are invisible to it, and the table's five false claims are five real devices.

Row two is the depth that throttles. A buffer shorter than the round trip never overflows — it is too small to overflow, and it runs the link at half rate instead. Simulation reports zero overflows and the sign-off passes. That is not a subtle interaction; it is the primary failure mode of section 5, and the weak definition is structurally incapable of seeing it because the symptom is the absence of the thing it measures.

Rows four and five are the two costs. A queue can be perfectly sized for throughput, never overflow, and still miss the latency budget by four hundred cycles or overspend the SRAM by fifty units. Neither is a functional failure, and both are found by a tool that is not a simulator.

Row six is the one the report catches, and it is worth being precise about what that means. An observed overflow is real evidence of a real defect — the weak definition's failures are all false negatives, never false positives. That is exactly what makes it dangerous: it is never wrong when it complains, so nobody questions it when it is silent.

Figure 4 — The throttle is asked first because it is the failure that produces no symptom at all, and the two cost questions are asked last because they are the two that a simulator was never going to answer. Every exit above is a device that passed its regression.

15. Quantitative Reasoning

The round trip. Sixty-four cycles at one beat a cycle needs sixty-four entries; thirty-two entries run the link at 50% and forty-eight at 75%, with no error reported anywhere.

Watermarks. Sixteen beats in flight against a sixty-four-entry queue means backpressure at forty-eight; asserting at sixty-four leaves zero headroom for sixteen beats that are already committed.

Ping-pong. A forty-cycle fill and a twenty-four-cycle drain cost sixty-four in series and forty in parallel — 37%; equal stages save 50%, which is the ceiling.

Burst absorption. Five hundred and twelve beats at four a cycle draining at one needs three hundred and eighty-four entries; a 256-entry buffer spills 128 and the smooth assumption asks for none.

Shared pools. Forty-eight wanted against a thirty-two-entry reservation is sixteen refused while fifty-six sit free; when others hold the whole pool, the shared design offers nothing and the dedicated design still offers thirty-two.

Cut-through. A sixty-four-beat packet across three hops is twelve cycles cut-through and a hundred and ninety-two stored — a factor of sixteen, and store-and-forward needs a whole packet of storage per port.

Buffer area. A 256-entry, 64-bit buffer is a hundred and twenty-eight units of array and twenty-five of control at twenty percent — a hundred and fifty-three against a sheet that reports a hundred and twenty-eight.

Queueing latency. Twenty entries at two cycles each is forty cycles of queueing on a twelve-cycle base against a forty-eight-cycle budget; two hundred entries is four hundred, eight times the budget.

The shared budget. Two hundred, one hundred and fifty, three hundred and one hundred is seven hundred and fifty against seven hundred — fifty over, while the largest single claim is three hundred and fits comfortably.

The assembled model. Six properties, seven configurations, one sized. The simulation report called six sized.

16. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Every inclusive threshold is driven at exactly equal, every ceiling both on and off its boundary, and every floor past it — the three rules this batch inherits, plus the one added in 24.4 §16: on every degenerate case, assert both builds.

The round trip. A buffer exactly equal to the round trip is driven, a buffer twice as large is asserted not to exceed full rate, and an unmeasured round trip is asserted on both builds.

chk(dGa == 16'd100, "a bigger buffer does not exceed full rate");
chk(dGs == 1'b1,    "and is sufficient");

Watermarks. Eight checks cover the four distinct relationships between occupancy and the watermark, including an occupancy exactly on it, where the worst case is exactly the depth and does not overflow.

chk(wGo == 16'd64, "the worst occupancy is exactly the depth");
chk(wGf == 1'b0,   "which is the last occupancy that does not overflow");

Burst absorption. A buffer exactly equal to the need is driven, a drain rate that keeps up is asserted to need nothing, and an unmeasured arrival rate is asserted to hold the whole burst.

Cut-through. A total exactly at the budget is driven, a packet that is entirely header is asserted to cost the same in both designs, and the store-and-forward added cost is asserted as a value rather than only through its error flag — which is what killed one of the two survivors.

chk(cBa == 16'd180,"180 of which the store step added");

The shared budget. A claim exactly equal to the budget is driven, and a single-queue device is asserted as not an isolation error, which is what proves the checker detects several queues rather than the isolated build.

The assembled model. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone so that no two bits can be swapped without a test noticing.

Totals: 275 checks across two testbenches, 144 on the front five models and 131 on the back five, all passing on the unmutated sources.

17. Mutation Testing

Ninety-two mutations were injected one at a time. 92 injected, 92 killed, after two survivors.

Survivor 1 — the overflow test at the depth rather than past it. Changing worst_occupancy > depth to >= survived, and the reason is arithmetic. The watermark is depth - in_flight, so an occupancy at the watermark gives a worst case of exactly the depth — the two conditions meet only at equality, and I had driven occupancy above and below the watermark but never on it. Adding §6 row seven killed it. The boundary was not merely undriven: it is the one point the model's own arithmetic makes reachable, which a floor or ceiling rule does not generate.

Survivor 2 — the added store cost scaled by the budget instead of the hops. (per_hop - header_beats) * hops became * budget_cycles and every test still passed, because added_cycles was never asserted as a value on the build where it is non-zero. It was observed only through store_cost_ignored_err, which tests it against zero — and eighteen hundred and eighty is as non-zero as a hundred and eighty. One line fixed it: asserting cBa == 180 on a case already being driven.

That is the same shape as 24.4's second survivor and 24.3's, three chapters running: an output that is computed, driven, and observed only through a flag. The rule generalises past "assert both builds" into something sharper — every output a model computes must be asserted as a value at least once, on the build where it is not zero. A flag that tests a value against zero proves the value is non-zero and nothing else about it.

18. Verification Strategy

What a testbench for a buffer-sizing model must cover.

Assert every computed output as a value, on the build where it is non-zero. Survivor 2, and the third instance of this shape in three chapters. A flag comparing a value against zero constrains one bit of it.

Find the boundary the model's own arithmetic pins. Survivor 1 was not a floor or a ceiling — it was the single point where two conditions can meet, produced by the relationship between the watermark and the worst case rather than by a threshold in the source. Reading the guards is not enough; the reachable equality has to be derived.

Drive the configuration the model is not named about. A round trip of zero in a round-trip model. A free drain stage in an overlap model. A drain rate that keeps up in a burst model. A single-queue device in a shared-budget model. Four such exemptions here, each one the case that keeps a check from firing on correct hardware.

Counters as a second signature. Ten models, ten pairs of totals, differing in all ten — three short against none, seven unsafe against two, five serial against two, two spilling against none, three refused against one, and six sized against one. This is the first chapter in the module where every pair differs, and it is a property of the subject: every broken build here under-reports a cost, and an under-reported cost always moves a count.

19. Synthesis and Implementation Reality

A buffer is an SRAM macro or a register file, and which one is a depth decision. Below roughly thirty-two entries, flops usually win on area and always on timing; above it, a macro wins on area and costs a cycle of read latency. Section 12's arithmetic changes across that boundary, and a depth chosen at the boundary should be chosen deliberately on one side of it.

Section 7's ping-pong is two buffers and a one-bit selector, and the cost is not the selector — it is that both buffers must be full depth, so ping-pong doubles the area to save 37% of the time. That trade is worth stating in exactly those terms at a review, because it is frequently presented as if the second buffer were free.

Section 13's budget is negotiated once and consumed continuously. The floorplan allocates an SRAM area early, every subsequent depth increase draws on it, and the team that discovers a needed depth increase last pays for everyone's. That is an organisational property of the budget rather than a technical one, and it is why the total belongs on a dashboard rather than in a spreadsheet.

20. Silicon Observability

"Grants refused per class, and entries free at the time" is the entry that settles arguments. The shared-against-dedicated debate is conducted in principle and decided by whichever engineer argues better; this counter turns it into a measurement — sixteen refusals while fifty-six entries sat free is a number, and so is zero refusals with two entries of margin. Neither policy is right in general and the counter says which is right here.

21. Debug Lab

Symptom. A CXL memory expander is in bring-up. It boots, enumerates, and passes every functional test. Then: read bandwidth is 51% of the projection, a single customer workload produces occasional dropped completions, tail latency at load is nine times the datasheet figure, and the floorplan has just come back fifty units of SRAM over budget.

Step 1 — the bandwidth. No overflow is reported anywhere, no credit violation, no retry. Section 5's shape exactly: a buffer too small to overflow. The replay buffer is a hundred and twenty-eight entries; the measured round trip under load is two hundred and fifty cycles at one beat a cycle. A hundred and twenty-eight against two hundred and fifty is 51%.

Step 2 — why the depth was wrong. The original sizing used a round trip of a hundred and twenty-eight cycles, measured on a bench link with no congestion. The number was correct when it was taken and the conditions it was taken under were not written down. Section 5, and the model's zero-round-trip row is the same defect with the measurement missing entirely rather than stale.

Step 3 — the dropped completions. They appear only under one customer workload, which issues four-kilobyte writes in bursts. The merge buffer absorbs up to its depth and spills past it — section 8, and the spill is silent because the drop is handled as a retry nobody counts. The burst histogram at the queue's input does not exist, so the sizing input cannot be recovered from the field.

Step 4 — the tail latency. Section 12. The scheduler queue is three hundred deep and runs at an occupancy near two hundred under load — four hundred cycles of queueing on a base of twelve. Nothing is broken: the depth that absorbs the burst is the depth that produces the tail.

Step 5 — the floorplan. Two hundred, one hundred and fifty, three hundred and a hundred against seven hundred. Section 13. Each depth was signed off in a different review and the sum was first computed by a floorplan tool. Fifty over, and no queue can give up fifty without one of steps 1, 3 or 4 getting worse.

Step 6 — why did none of this appear in verification? The regression reports zero overflows across every test. Section 14's weak definition: the throttle is invisible because the buffer is too small to overflow, the spill needs a burst nobody wrote, the tail latency is not a functional failure, and the SRAM total is not a simulator's output. Four of six properties failed and the report was clean.

The finding. Four defects, no functional bugs, and the four are coupled — every fix for one makes another worse. Sections 5, 8, 12 and 13 in that order, which is the order of measurability rather than of severity.

The fix. Deepen the replay buffer to two hundred and fifty-six against the round trip measured under load, and record the conditions next to the number. Deepen the merge buffer against the customer's burst histogram. Assert backpressure on the scheduler queue at an occupancy the latency budget allows — sections 6 and 12 meeting, capping the tail without removing the depth. And take the SRAM back from the elastic buffers, the only queue whose sizing input is fixed rather than traffic-dependent.

What made this hard. Nothing overflowed. Every queue behaved exactly as designed, every functional test passed, and the device delivered half its bandwidth, dropped a customer's completions, missed its latency by nine times and did not fit — on four depths that were each individually defensible.

22. Design Review

1. What is the round trip each depth was sized against, and under what load was it measured? A number taken on an uncongested bench ages into a throttle. Section 5.

2. How much traffic can be in flight when a watermark asserts, and does the headroom cover it? Depth minus flight, not depth. Section 6.

3. Which queues are ping-pong, and was the doubled area weighed against the 37%? Two full-depth buffers to save the difference between the sum and the slower. Section 7.

4. What burst length was each depth sized against, and where is the histogram? Average rate sizes a buffer at nothing. Section 8.

5. Which pools are shared, and what does each class hold when the pool is empty? A class that cannot make progress stops the device. Section 9.

6. Is forwarding cut-through or store-and-forward, and does the buffer size follow the policy? Store-and-forward needs a whole packet per port. Section 10.

7. Does the area number for each buffer include its control logic? Twenty percent of the array, and worst proportionally on the small ones. Section 11.

8. What latency does each queue's depth add at its expected occupancy? Forty cycles on a twelve-cycle base is not the datasheet figure. Section 12.

9. What is the sum of every queue in the device against the SRAM budget? The number no individual owns. Section 13.

10. Which of the six properties does "no overflow in simulation" imply? Section 14 exists because the answer is the first one only.

23. How This Appears In Real Engineering

A block designer sizes the queue inside their block against the traffic they can see at their ports, and every number in this chapter is computed correctly at that scope. Sections 5, 8 and 12 are properties of traffic that arrives from outside the block, and section 13 is a property of a resource shared with blocks the designer does not own.

A performance engineer meets sections 5 and 12 as the same measurement from opposite ends: bandwidth below projection with no errors is a depth below the round trip, and latency above budget with no errors is a depth above what the budget allows. The same queue produces both, and the fix for one is the cause of the other.

A verification engineer owns the weak definition of section 14 whether or not they chose it. The regression reports overflows because overflows are what a simulator can see, and the other five properties need a different kind of check — which is 25.1's subject, and it is the next chapter.

24. Common Misconceptions

"A deeper buffer is always safer." It is safer against overflow and strictly worse against latency (section 12) and area (sections 11 and 13). Past the round trip it buys no bandwidth at all (section 5 row three), so the extra depth is pure cost — and the cost lands on requests that arrive behind a full queue, which is to say under exactly the load where latency matters.

"The buffer never overflowed, so it is big enough." It is big enough for the traffic that was run. Section 14 row two is a buffer too small to overflow, running the link at half rate; row seven is a burst profile nobody wrote. The absence of an overflow is evidence about a workload, not about a depth.

"Average bandwidth sizes the buffer." Average bandwidth sizes nothing. The gap between arrival and drain over the burst sizes it (section 8), and a stream whose average fits comfortably can need three hundred and eighty-four entries.

"A watermark at ninety percent is a reasonable margin." Ninety percent of sixty-four is fifty-seven, and sixteen beats in flight need forty-eight. A percentage is a guess about the flight; the flight is a number that can be computed. Section 6 replaces the guess with the calculation, and they differ by nine entries in this example and by more on a longer link.

25. Interview Reasoning

"How deep should this buffer be?" The answer is a question: what is the round trip, what is the burst, and what is the latency budget? A depth is derived from three traffic properties and constrained by a fourth, the shared budget — a candidate who names a number has answered a different question, and one who names the four inputs has understood the problem.

"Bandwidth is half the projection and there are no errors anywhere. Where do you look?" Section 5. A buffer smaller than the round trip throttles silently — it is too small to overflow, so nothing is reported. The measurement that confirms it is cycles stalled on credit exhaustion, and the number that predicts it is the round trip under load rather than on a bench.

"Why not assert backpressure when the queue is full?" Because everything already in flight still arrives. Depth minus flight is the watermark, and the last entries are the landing zone rather than capacity. A follow-up worth asking back: what happens when the flight is deeper than the queue? — section 6 row three, where no watermark is safe and the link needs fewer outstanding beats instead.

"Would you use one buffer or two here?" Two buffers cost twice the area and save the difference between the sum and the slower stage — 37% for a forty-and-twenty-four split, 50% for a balanced one, and nothing at all if one stage is free. The answer depends on whether the stages are balanced, which is the opposite of where intuition sends the technique.

"Your four queues are each sized correctly and the device is over budget. What do you give up?" Name what each depth buys: the replay buffer buys bandwidth, the merge buffer burst tolerance, the scheduler queue reordering, and the elastic buffer a clock crossing. Only the last has a sizing input that is not traffic-dependent, so it is the one that can be re-derived rather than negotiated. Shaving all four equally misses that they are not the same kind of number.

"How would you know the buffering is sized, rather than just not overflowing?" Section 14's six properties. The distinction between a property and an observation is the thing being tested here, and it is the same distinction 25.1 is built on.

26. Exercises

1. Re-derive the depth when the round trip is a distribution. A link whose round trip is 64 cycles at the median and 250 at the 99th percentile. Compute both depths, say which one a buffer must be sized against, and price the difference with §11's model.

2. Turn a latency budget into a watermark. A 64-entry queue, 16 beats in flight, a 48-cycle budget at two cycles of service. §6 and §12 each produce a limit — compute both and say which binds.

3. Price ping-pong properly. One 40-entry buffer against two, at 64 bits and 20% control. Express the 37% saving as cycles per unit of area, then find the fill/drain split at which the trade stops paying.

4. Size a buffer from a histogram. A stream that is 90% single beats and 10% bursts of 512 beats at four a cycle, draining at one. Compute the depth that absorbs every burst and the depth that absorbs 99% of traffic. Which belongs in a datasheet?

5. Build the shared-pool starvation case. Construct the smallest configuration in which the starved class is the one that drains the pool, then find the minimum reservation that makes it impossible and price it.

6. Choose a forwarding policy per virtual channel. A fabric carrying 80% 4-beat control and 20% 64-beat data. Compute mean latency and per-port buffer area under each policy.

7. Extend the area model to port count. Replace §11's flat percentage with control scaling in read and write ports. At which port count does a 256-entry buffer fail the quarter rule?

8. Replace the mean service time with a distribution. Two cycles 90% of the time and forty 10% of the time, at an occupancy of twenty. Which of the mean and the 99th percentile must a budget be written against?

9. Solve the budget overrun. §13's four queues claim 750 against 700. Using §§5, 8 and 12, compute what fifty units costs at each queue in turn and recommend one, with the loss stated.

10. Add the seventh property. Propose a property none of §14's six implies, and construct the configuration where the six hold and it fails. A property that cannot fail alone is not a seventh property.

27. Summary

A buffer covers a round trip or the link runs at the fraction it covers. Thirty-two entries against a sixty-four-beat round trip is 50%, forty-eight is 75%, and nothing anywhere reports an error — the buffer is too small to overflow.

A watermark has to leave room for what is already sent. Sixteen in flight against a sixty-four-entry queue means backpressure at forty-eight, and asserting at sixty-four leaves zero headroom for sixteen committed beats. The margin is a calculation, not a percentage.

Two buffers overlap what one serialises. Forty and twenty-four is sixty-four in series and forty in parallel — 37%, with a ceiling of 50% at balanced stages, for twice the area.

Average rate sizes nothing. Five hundred and twelve beats at four a cycle draining at one needs three hundred and eighty-four entries held, where the smooth assumption asks for none — and a 256-entry buffer spills a hundred and twenty-eight.

A shared pool is smaller, grants more often, and can starve a class to a stop. Sixteen refused with fifty-six free is the dedicated design's waste; nothing available at all is the shared design's hang.

Store-and-forward pays for the packet at every hop. Sixty-four beats across three hops is a hundred and ninety-two cycles against twelve — and a whole packet of storage per port rather than a header's worth.

A buffer is not its array. A hundred and twenty-eight units of storage carries twenty-five of control, and the control does not shrink with the depth — worst proportionally on exactly the small buffers nobody reviews.

Depth bought for throughput is paid for in latency. Twenty entries at two cycles each is forty cycles on a twelve-cycle base, four times the datasheet figure, and two hundred entries is eight times the entire budget.

And every queue draws from one budget. Two hundred, one hundred and fifty, three hundred and one hundred is seven hundred and fifty against seven hundred — while the largest single claim is three hundred and fits with room to spare. Four correct numbers, fifty units over, and the sum is nobody's number.

Two mutations survived: one on a boundary the model's own arithmetic pins to a single point, and one on an output driven and never asserted as a value. That second shape is now three chapters running, and the rule is sharper than the one it replaces: every computed output must be asserted as a value at least once, on the build where it is not zero. A flag testing a value against zero constrains one bit of it.

"No queue overflowed" is one property of six. The simulation report called six of seven configurations sized when one was — and section 21 is a device that boots, enumerates, passes every functional test, and delivers half its bandwidth, drops a customer's completions, misses its latency by nine times and does not fit on the die.

Module 24 closes here. 24.1 built the link interface, 24.2 the transaction pipeline, 24.3 the memory controller, 24.4 the boundaries between them, and this chapter the storage that sits on those boundaries. Every one of the five ended on the same finding from a different direction: the thing that passes is not the thing that works.

25.1 — CXL Protocol Verification takes that finding and makes it the subject. Every chapter of this module produced a property that a simulation could not see — a throttle with no error, a burst nobody wrote, a latency that is not a failure, a sum that is not an output. The next chapter is about how a verification methodology is built to find them anyway.