SmartNIC — Average Margin Is Not the Question

29.3 sized a buffer with a product — outstanding requests times bytes each — and got an exact answer. This chapter cannot do that. The arrival process is external, and the honest output of the sizing exercise is a loss probability, not a depth.

1. Sources, Scope, and What This Chapter Refuses to Do

2. Two Domains, Neither of Which Will Negotiate

Three readings.

Row 2 is the structural difference. 29.3's card could stop asking. A NIC cannot tell a remote sender to pause, so the buffer is the only defence and its failure is silent loss.

Row 4 is the uncomfortable one. "How deep should this buffer be?" has no answer without a statement about the arrival process. A depth alone is not a specification — it becomes one only paired with a burst bound or a loss target.

And row 5 is why teams are surprised. The rate arithmetic genuinely balances. Average service rate exceeds average arrival rate, the card drops packets, and both facts are true simultaneously (§4).

3. The Boundary

4. Why Average Margin Is Not the Question

Define utilisation ρ = arrival_rate ÷ service_rate. For the model above, mean queue occupancy grows as:

mean_occupancy ≈ ρ ÷ (1 − ρ)

Four readings.

The margin column is linear and the occupancy column is not. Going from ρ = 0.9 to ρ = 0.95 halves the margin and doubles the occupancy. So a capacity plan expressed as "we have N % headroom" describes the wrong quantity — it is linear in a variable the buffer responds to hyperbolically.

Which means the interesting region is exactly where systems are operated. Nobody designs for ρ = 0.5; cost pressure pushes ρ toward 1, and that is precisely where occupancy becomes sensitive to small rate changes. A 3 % service-rate regression at ρ = 0.92 is a very different event than the same regression at ρ = 0.6.

And the service rate is not a constant — this is the PCIe-specific part. service_rate here is the DMA engine's drain rate, which depends on descriptor availability (26.4), payload size and the outstanding window (29.2 §5), and host memory behaviour. So ρ moves during operation without any change on the network side, and a system provisioned at ρ = 0.85 can find itself at ρ = 0.97 because software fell behind on descriptor replenishment.

The honest conclusion is not a number. It is that buffer sizing must be stated against a burst bound or a loss target (§5), and that average-rate margin should never be quoted as a safety statement (§6).

5. What Can Be Made Deterministic

Not everything here is statistical, and the deterministic part should be extracted first.

The deterministic burst bound. During a back-to-back burst the buffer grows at R_in − R_out, so:

required_depth ≥ B × (1 − R_out ÷ R_in)

Worked, illustratively: a 64 KB burst, R_in = 12 GB/s, R_out = 9 GB/s:

65536 × (1 − 9/12) = 16 384 bytes

Three readings.

This part is exactly like 29.3 §5 and should be done the same way — a bound, checked at elaboration, with no probability involved. A design that cannot survive its specified burst bound is broken deterministically, and that is worth catching before any statistical argument begins.

The two arguments answer different questions. The burst bound answers "can we survive the worst burst we claim to support?"; the utilisation argument answers "how often will we exceed it anyway?" A design needs both, and only the first has a clean answer.

And R_out in that formula is the instantaneous drain rate, not the average. It is itself the product of 29.2 §5's four factors, so a payload-size or tag-count change moves it. If descriptors run out mid-burst (26.4), R_out briefly becomes zero and the required depth becomes the full burst B. That is the case worth sizing against — and it makes descriptor replenishment a buffer-sizing parameter, which is not where teams look for it.

6. Wrong RTL — Sampling the Occupancy

The instrument is the bug this time — the buffer logic is fine.

// WRONG. ILLUSTRATIVE. A conventional, cheap statistics block: sample occupancy
// on a timer, accumulate for an average, and derive "rate margin" from byte
// counters. Every line is standard practice and the result is an instrument
// that cannot see the failure it exists to detect.
localparam int SAMPLE_PERIOD = 10000;             // cycles between samples
 
logic [15:0] occ_bytes_q;                          // live occupancy (correct)
logic [15:0] occ_sample_q;                         // BUG 1
logic [47:0] occ_sum_q;
logic [31:0] occ_n_q;
logic [15:0] samp_cnt_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    occ_sample_q <= '0; occ_sum_q <= '0; occ_n_q <= '0; samp_cnt_q <= '0;
  end else begin
    samp_cnt_q <= (samp_cnt_q == SAMPLE_PERIOD[15:0]) ? '0 : samp_cnt_q + 16'd1;
 
    // BUG 1: occupancy is SAMPLED on a timer. A burst that fills and drains
    //        between two samples is invisible — the instrument aliases it away
    //        completely. Bursts are exactly what the buffer exists to absorb.
    if (samp_cnt_q == SAMPLE_PERIOD[15:0]) begin
      occ_sample_q <= occ_bytes_q;
      occ_sum_q    <= occ_sum_q + 48'(occ_bytes_q);
      occ_n_q      <= occ_n_q + 32'd1;
    end
  end
end
 
// BUG 2: "margin" computed from AVERAGE rates. §4: this is linear in a variable
//        the buffer responds to hyperbolically, and it reads healthy at ρ = 0.95.
assign rate_margin_pct = 8'((100 * (rx_capacity - rx_bytes_q)) / rx_capacity);
 
// BUG 3: one drop counter, no attribution. A drop from a burst, a drop from
//        descriptor starvation and a drop from a service-rate regression are
//        three different problems reported as one number.
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n)            drops_q <= '0;
  else if (drop_fire)    drops_q <= drops_q + 32'd1;

Architecture. A periodic sampler, an averaging accumulator, a derived margin percentage, and a single drop counter.

State. occ_sample_q, occ_sum_q, occ_n_q, drops_q. The missing state is a high-water mark — one register that would have made the bursts visible, and which costs less than the averaging logic it sits beside.

Event. Sampling on a timer, uncorrelated with traffic. This is the defect: the sample times are chosen by the instrument and the interesting times are chosen by the network.

Contract. Whoever reads occ_sample_q and rate_margin_pct believes they describe how close the buffer is to overflow. They describe occupancy at arbitrary instants and a linear function of average rates.

Failure — the timeline. Bursty arrivals: 40 µs of near-idle, then a 5 µs burst, repeating. Sample period ≈ 50 µs of cycles.

First divergence: 40–45 µs — occupancy reached 31 KB of a 32 KB buffer and no sample was taken. By 50 µs the burst had drained and the instrument recorded 3 KB.

Root cause. Periodic sampling of a quantity whose interesting values are transient. The sampled occupancy is not a wrong measurement of the peak — it is a correct measurement of the wrong thing, which is harder to argue with.

BUG 2 corroborates the wrong story. 61 % margin agrees with 12 % occupancy, so two independent-looking metrics tell the same false story and confidence rises. Both are averages over a distribution whose tail is the entire problem.

And BUG 3 removes the last chance. The drop count is real and unattributed. "1 400 drops with 61 % margin and 12 % occupancy" is a self-contradictory report, and teams resolve it by disbelieving the drops — blaming the counter, the switch, or the sender.

DV/debug. The signature is exactly that contradiction. The discriminator is a high-water mark, and §7 adds it in one line.

7. Corrected RTL — Peaks, Attribution, and a Burst Detector

// CORRECT. ILLUSTRATIVE. Three additions, all cheap:
//   (a) a high-water mark — captures the peak regardless of sample timing;
//   (b) per-cause drop attribution — three counters instead of one;
//   (c) a near-full residency counter — how LONG the buffer spent in the
//       danger zone, which is the quantity that predicts the next overflow.
localparam int DEPTH_BYTES = 32768;
localparam int NEAR_FULL   = 24576;               // 75 % — a WATCH line, not a
                                                  // safety threshold (§8)
 
logic [15:0] occ_bytes_q, occ_high_water_q;
logic [31:0] near_full_cyc_q;
logic [31:0] drop_burst_q;                        // buffer full on arrival
logic [31:0] drop_nodesc_q;                       // no descriptor available
logic [31:0] drop_error_q;                        // malformed / filtered
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n || stat_clear) begin
    occ_high_water_q <= '0; near_full_cyc_q <= '0;
    drop_burst_q <= '0; drop_nodesc_q <= '0; drop_error_q <= '0;
  end else begin
    // (a) One comparison per cycle. Sample timing becomes irrelevant, because
    //     the register remembers the peak instead of a snapshot.
    if (occ_bytes_q > occ_high_water_q) occ_high_water_q <= occ_bytes_q;
 
    // (c) Residency, not an average. Time spent above the watch line is the
    //     leading indicator; the high-water mark is the lagging one.
    if (occ_bytes_q >= NEAR_FULL[15:0]) near_full_cyc_q <= near_full_cyc_q + 32'd1;
 
    // (b) Attribution at the point of decision, where the cause is still known.
    //     Downstream, all three look identical.
    if (drop_fire) begin
      unique case (drop_cause)
        DROP_FULL:   drop_burst_q  <= drop_burst_q  + 32'd1;
        DROP_NODESC: drop_nodesc_q <= drop_nodesc_q + 32'd1;
        default:     drop_error_q  <= drop_error_q  + 32'd1;
      endcase
    end
  end
end

Architecture. One extra comparator, one residency counter, and a three-way demultiplex of an existing drop event.

State. occ_high_water_q is the load-bearing addition — it converts a sampling problem into a non-problem, because the peak is accumulated in hardware and read at any convenient time.

Event. The high-water comparison runs every cycle; attribution happens at the drop decision, where drop_cause is still available.

Contract. stat_clear must be explicit and infrequent. A high-water mark cleared every sample period degenerates into §6's sampler — the value is entirely in the fact that it spans the whole observation window. This is the most likely way to reintroduce the bug while appearing to fix it.

Failure. NEAR_FULL set too high makes the residency counter useless, and set too low makes it always non-zero. It is a diagnostic threshold, not a control threshold — nothing in the datapath acts on it, which is why choosing it badly is a lost signal rather than a malfunction.

And attribution is what makes the report actionable. drop_burst high with drop_nodesc zero is a sizing problem (§5). drop_nodesc high is a software problem — descriptor replenishment (26.4), which §4's third reading identified as a hidden term in R_out. Same total, two entirely different fixes, and §6's single counter cannot distinguish them.

8. Checks and Assertions

// MANDATORY. English: the high-water mark never decreases except on an explicit
// statistics clear. This is what makes it a peak rather than a sample, and it
// catches the §7 contract failure of clearing it on a timer.
a_high_water_monotonic: assert property (
  @(posedge clk) disable iff (!rst_n || stat_clear)
    occ_high_water_q >= $past(occ_high_water_q)
);
 
// MANDATORY. English: if occupancy ever reached the depth, at least one
// burst-cause drop must be recorded. Ties the peak to the consequence, and
// catches a full buffer that silently discards without counting.
a_full_implies_drop: assert property (
  @(posedge clk) disable iff (!rst_n || stat_clear)
    (occ_bytes_q == 16'(DEPTH_BYTES)) ##1 arrival_fire |=> (drop_burst_q > $past(drop_burst_q))
);
 
// MANDATORY. English: every drop increments exactly one attribution counter.
// Catches a new drop cause added to the datapath without a counter — the
// failure that silently restores §6 BUG 3 one release later.
a_drop_attributed_once: assert property (
  @(posedge clk) disable iff (!rst_n || stat_clear)
    drop_fire |=> ((32'(drop_burst_q  != $past(drop_burst_q))
                  + 32'(drop_nodesc_q != $past(drop_nodesc_q))
                  + 32'(drop_error_q  != $past(drop_error_q))) == 32'd1)
);

Reading the three.

stat_clear appears in every disable term, because all three properties are statements about an accumulation window and a clear legitimately breaks each one. Omitting it produces three false failures on the first clear — and the usual response, deleting the assertions, is worse than never writing them.

The second is a two-cycle sequence for a reason. The full condition and the arrival that gets rejected are separate events; ##1 arrival_fire |=> says "full, then an arrival, then a counted drop." A same-cycle implication would be wrong — a full buffer with no arrival correctly drops nothing.

And the third is the maintenance assertion. It does not protect against today's bug; it protects against the fourth drop cause someone adds next quarter without a counter, which silently restores the unattributed report. That is the failure mode this chapter's whole argument is about, so it is worth an assertion.

9. Measured Behaviour

Four readings.

The ρ = 0.85 row is the most valuable one in this chapter. Zero drops, and the buffer reached 28 KB of 32 KB. §6's instruments report 3.1 KB occupancy and 18 % margin — a picture of comfortable operation with 12 % of the buffer actually remaining. This configuration is one small regression from loss and every conventional metric says it is fine.

And that is what the residency counter is for. 4.1 % near-full residency at ρ = 0.85 is a leading indicator with no drops behind it yet — the only metric in the table that moves before the failure rather than after.

The sampled column barely changes across the whole sweep. 1.8 KB → 4.6 KB while the high-water goes 11 KB → 32 KB. A metric that varies by 2.5× while the thing it claims to measure varies by 3× and saturates is not tracking it — the sampled value is dominated by the idle intervals, which are most of the time and none of the risk.

The drops rise 7× between ρ = 0.92 and ρ = 0.95 for a 3-point utilisation change, which is §4's shape appearing in the loss column. Near the knee, small service-rate changes have large consequences, and this is the region cost-optimised designs occupy.

10. What Can Actually Be Done — Four Levers, Honestly

Three readings.

Lever 3 is usually the best return and the last one considered. §5's third reading showed that a descriptor stall makes R_out momentarily zero, which makes the required depth the entire burst. Fixing replenishment can reduce the required buffer more than doubling the buffer does, and it costs no silicon.

Lever 2 benefits from §4's non-linearity in the good direction. Moving ρ from 0.95 to 0.90 halves the model's mean occupancy. The same 5-point improvement from 0.60 to 0.55 is nearly worthless — so service-rate work is worth much more at high utilisation than a linear reading suggests.

And lever 4 deserves its place because unbounded silent loss is the actual failure. A design that chooses what to drop and counts why has converted an outage into a degradation. The distinction between drop_error and drop_burst in §7 is exactly this distinction, which is why the attribution counters are an architectural feature rather than a debug convenience.

11. Executable Counterexamples

Case 2 requires bursts shorter than the sample period, which is a stimulus property, not a DUT property. A testbench with uniform arrivals cannot produce it at any rate — the bug is invisible to a well-behaved traffic generator, which is why it survives.

And case 3 produces zero drops in both designs. It is a test whose pass/fail is a metric comparison, not an error — the environment must assert that the reported occupancy is within tolerance of the true peak, or the near-miss stays invisible.

12. Verification

Three readings.

"Verify the instrument, not just the datapath" is the whole point. §6's buffer logic is correct and §6's report is wrong. A verification plan that only checks packets in versus packets out passes it, because the packets were right and the description of the buffer was not.

The coverage requirement "residency non-zero with drops zero" encodes the near-miss. It is a coverage point for a healthy-looking configuration, which is unusual — most coverage targets failures. This one targets the state just before one.

And the reset row is a genuine design question rather than a check. A statistics clear during a burst destroys the peak, and there is no way to avoid it. The answer is to make clears rare and to record that one happened — so a report can say "peak since last clear" honestly.

13. Debugging

Three readings.

The contradiction is the diagnostic, and it should be trusted in the direction of the drops. Counted drops are a direct observation; margin and mean occupancy are derived statistics. When they disagree, the statistic is wrong far more often than the counter — but the social gradient runs the other way, because the statistic is in a dashboard and the counter is in a register dump.

One register ends the investigation. High-water at depth: the buffer filled, and the sampled figure was an artefact. High-water well below depth: the drops are not from overflow at all, and drop_nodesc or drop_error will say so.

And the prevention line generalises past NICs. Any queue whose failure is a tail event is mis-described by its mean. The pattern — peak plus residency plus attribution, instead of average — is the reusable result of this chapter.

14. Misconceptions

"We have 50 % rate margin, so the buffer is fine." §4: margin is linear, occupancy is not. §9 shows 28 KB of a 32 KB buffer used at 18 % reported margin.

"Zero drops means healthy." §9 row 3: zero drops with 12 % of the buffer remaining. One regression away from loss, and every conventional metric reads comfortable.

"Average occupancy tells us how full the buffer gets." §6, §9: the sampled average moved 2.5× while the peak moved 3× and saturated. The average is dominated by idle intervals.

"Sampling faster fixes it." §6: faster sampling narrows the window, it does not close it — and a burst can still fall between samples. A high-water mark is exact at any read rate, and cheaper.

"A deeper buffer always helps." §10, and 29.3 §9 makes the sharper version of the point: not if ρ ≥ 1. Buffers absorb variance around a sustainable average; they cannot manufacture service rate.

"Drops are a network problem." §7, §13: drop_nodesc is a descriptor-replenishment problem (26.4) — host software — and it is indistinguishable from a network problem without attribution.

"The service rate is a fixed property of the link." §4, §5: it depends on payload size (22.4), the outstanding window (26.2 §5) and descriptor supply, so ρ moves during operation with no change on the network side.

"This is a queueing-theory result, so it's theoretical." §5: the burst bound is deterministic and belongs in an elaboration check. Only the loss-probability half is statistical — and §4 states its assumptions rather than hiding them.

15. Understanding Check

Q1. A card reports 61 % rate margin and 12 % mean buffer occupancy, and drops thousands of packets per second. Reconcile the three.

All three are correct measurements of different distributions (§6, §13). Margin is linear in average rates, occupancy is sampled on a timer uncorrelated with traffic, and drops are a direct observation of tail events. §6's timeline shows the mechanism: a burst drove occupancy from 2 KB to 31 KB and back entirely between two samples, so the recorded value was 3 KB. The sampled figure is not an inaccurate measurement of the peak — it is an accurate measurement of the wrong thing, which is why it is hard to argue with. The two metrics agreeing with each other raises confidence in the false picture. Trust the drops: counted events are observations, margins are derived statistics — the same precedence 29.3 §13 applied when a clean link contradicted a user report. The discriminator is one register — an occupancy high-water mark — and the follow-up question is drop_burst versus drop_nodesc, because those are a sizing problem and a software problem respectively.

Q2. Why is "we have N % headroom" the wrong way to state safety here?

Because it is linear in a variable the buffer responds to hyperbolically (§4). Under the stated single-server memoryless model, mean occupancy goes as ρ/(1−ρ): halving the margin from ρ = 0.90 to ρ = 0.95 doubles occupancy, and from 0.95 to 0.98 doubles it again. Real traffic burstier than memoryless is worse, not better. Two consequences follow. Cost pressure pushes systems toward ρ → 1, which is exactly the sensitive region — so a 3 % service-rate regression at ρ = 0.92 is a different event from the same regression at ρ = 0.60. And service_rate is not constant: it depends on payload size (22.4), the outstanding window (26.2 §5) and descriptor availability (26.4), so ρ drifts upward during operation with nothing changing on the network side. The defensible statements are a burst bound (§5) or a loss target — not a margin.

Q3. What part of this problem is deterministic, and what should you do with it?

The burst bound (§5). For a claimed back-to-back burst of B bytes at peak rate R_in drained at R_out, the buffer must satisfy depth ≥ B × (1 − R_out/R_in) — illustratively, a 64 KB burst at 12 GB/s in and 9 GB/s out needs 16 KB. Put it in an elaboration check, exactly as 29.3 §7 does for its own headroom inequality, so a configuration that cannot survive its own specified burst fails the build rather than the lab. And note the subtlety in R_out: it is the instantaneous drain rate. If descriptors run dry mid-burst, R_out momentarily becomes zero and the requirement becomes the full burst B. That makes descriptor replenishment a buffer-sizing parameter — which is why §10 ranks stabilising it above deepening the buffer: it can reduce the required depth by more than doubling the memory does, at no silicon cost.

Q4. Design verification that would catch the §6 instrument. Explain why an ordinary NIC testbench would not.

Verify the instrument, not only the datapath (§12). §6's buffer logic is correct — packets in match packets out — and only its report is wrong, so a plan that checks packet integrity passes it. The checker must therefore compare the DUT's reported high-water against a testbench model's independently-tracked true peak: a metric assertion rather than an error check.

The stimulus is the other half. The bug needs bursts shorter than the statistics period, which is a property of the traffic generator; uniform arrivals cannot expose it at any rate. So coverage bins burst length on both sides of the statistics period, plus ρ bins at 0.85, 0.92 and 0.95, plus the unusual point "near-full residency non-zero while drops are zero" — a coverage target for a healthy-looking state one step from failure (§9 row 3).

Two negative tests prove the checkers are alive. Stimulate a burst peaking at 90 % of depth with zero drops and assert the reported peak is within tolerance — that is §9's near-miss, and an environment that only fails on drops sees nothing. And clear statistics on a timer and confirm a_high_water_monotonic fires, since periodic clearing silently degrades the peak back into §6's sampler. Add the maintenance check a_drop_attributed_once, which fires when a future drop cause is added without a counter — the same forward-protection role 29.3 §8's a_committed_bounded plays for the opposite failure direction.

16. What Comes Next

Three chapters have now accounted for a device the host must explicitly move data to and from. The last one asks what changes when it does not have to.

29.5 closes the module on an AI accelerator, where the decisive question is the attach model — whether the host copies data explicitly or the device participates in a shared view of memory — and what the host software sees differently in each case. The buffer arithmetic of the last two chapters does not disappear; it moves to a place where the programmer can no longer see it.