A High-Rate Data-Centre NIC

Chapter 23.3 faced sixty-four ports sharing one buffer. A NIC faces one port sharing a host, and every number in it is set by two quantities: 6.72 nanoseconds and the width of a PCIe link.

Row four is the chapter's first result and it surprises people. A descriptor is sixteen octets and a minimum frame is sixty-four; fetching one descriptor and writing one back costs thirty-two octets per packet, which is half the frame and 38% of the link's payload bandwidth in DMA traffic that carries no data at all.

And it scales the wrong way.

At 400 Gb/s a NIC's descriptor traffic alone is 152 Gb/s, and payload plus descriptors exceed a Gen5 ×16 host interface by 46%. The descriptors are why, not the payload.

Then the structural finding, which is the opposite of a switch's.

Chapter 23.3 §4 found a switch that was 91% packet buffer. A NIC is 66% connection context and eleven times smaller in total, and the difference decides everything about how it fails.

Because context is a cache, and a cache has a cliff.

Row five is Chapter 23.2 §11's warning, derived. That chapter's cheapest remedy for a training cluster — stripe each ring link into 32 connections — puts twice the working set into the cache and pushes the miss rate past the cliff, producing a throughput collapse that looks exactly like the fabric problem striping was deployed to fix.

1. Scope — One Port, One Host, and 6.72 Nanoseconds

Scope: a fixed-function high-rate NIC — its per-packet budget, its host interface, its queues, and the state that dominates it.

Not in scope: programmable offload. What a NIC does when it can run arbitrary code is Chapter 23.5's. This chapter's part parses, classifies, checksums, steers and moves bytes, all in fixed logic, and the interesting constraints turn out to be about neither logic nor bytes.

Two numbers govern the chapter.

Number 1 is the same constraint Chapter 23.3 §6 met and resolved differently. A switch has 64 ports and made its pipeline 8 wide; a NIC has one port and 6.72 ns is a real per-packet budget rather than an aggregate one. The NIC cannot go wide — there is only one packet stream — so it goes deep, and its pipeline is a few tens of stages of fixed work.

Number 2 is the one this chapter is really about. A switch's constraint is on-die memory; a NIC's is the pipe to the host. Everything a NIC does that touches host memory — fetching a descriptor, writing one back, filling a context, evicting one, moving the payload — competes for the same 378 Gb/s, and Sections 4, 6 and 12 are three different ways of running out of it.

Three things the chapter establishes.

2. The Per-Packet Budget, and What Fits In It

Derive the budget and then list what has to happen inside it.

minimum frame plus preamble and gap = 84 octets = 672 bits

at 100 Gb/s: 672 ÷ 100 × 10⁹ = 6.72 ns

Row four is under two cycles, which is why a 400 Gb/s NIC's receive path is pipelined and partly replicated rather than iterative — the same answer Chapter 23.3 §6 reached for a switch, arrived at from the opposite direction.

What must happen in those 6.7 cycles, per packet.

Rows one to five are logic and fit comfortably. Chapter 18.7 §18 priced the whole offload set and it is a few thousand flops; at 6.72 ns per packet the work is pipelined and the throughput is one packet per cycle regardless of depth.

Rows six to eight are the problem, and they are not logic at all. Each is a host memory transaction with a latency of hundreds of nanoseconds to microseconds, against a budget of 6.72. They cannot be done inside the budget; they must be done ahead of it, which is Chapter 18.2 §3's prefetch store and Section 7's in-flight model.

Nothing a NIC does per packet takes longer than 6.72 nanoseconds. Everything it does that touches the host takes a hundred times that, and the entire design is about hiding the difference.

Which gives the chapter's organising split.

3. RTL 1 — The NIC Package and the Host Bandwidth Model

// ---------------------------------------------------------------------
// dcnic_pkg -- a 100 Gb/s fixed-function NIC, and the one resource
// everything in it competes for.
//
// Chapter 23.3's package held an area model because a switch's problem is
// on-die memory. This one holds a BANDWIDTH model, because a NIC's
// problem is the pipe to the host -- and Section 12 shows the two are
// connected through a cache.
// ---------------------------------------------------------------------
package dcnic_pkg;

  // ---- the wire ---------------------------------------------------------
  localparam int RATE_GBPS      = 100;
  localparam int MIN_WIRE_OCTETS= 84;          // frame + preamble + IFG
  localparam int PPS_MILLION    = 148;         // 148.81, floored
  localparam int NS_PER_PKT_X100= 672;         // 6.72 ns

  // ---- the host interface ------------------------------------------------
  // PCIe Gen5 x16: 32 GT/s x 128/130 x 16 lanes = 504.1 Gb/s raw.
  // TLP headers, ACKs and flow control take about a quarter.
  localparam int HOST_RAW_GBPS  = 504;
  localparam int TLP_EFF_PCT    = 75;
  localparam int HOST_USABLE_GBPS = (HOST_RAW_GBPS * TLP_EFF_PCT) / 100;  // 378

  // ---- descriptors -------------------------------------------------------
  // Chapter 18.2's ring. A fetch and a writeback per packet.
  localparam int DESC_OCTETS    = 16;
  localparam int DESC_PER_PKT   = 2;           // fetch + writeback

  // ---- connection context -------------------------------------------------
  // The state a NIC holds per connection: send and receive queue
  // pointers, a congestion-control context, a reassembly pointer.
  localparam int CTX_OCTETS     = 256;
  localparam int CTX_CACHE_ENTRIES = 16 * 1024;
  // A miss costs a fill AND an eviction writeback.
  localparam int CTX_MISS_OCTETS = CTX_OCTETS * 2;

  // ---- queues --------------------------------------------------------------
  localparam int QUEUES         = 2048;
  localparam int Q_STATE_BITS   = 512;

  // ---- the area unit, from Chapter 23.3 Section 2 --------------------------
  localparam int BCE_PER_FLOP   = 20;
  localparam int DATAPATH_BCE   = 283320;

  // ---- the arithmetic that decides the chapter ----------------------------
  function automatic int desc_gbps();
    // pps x 32 octets x 8 bits, in Gb/s
    return (PPS_MILLION * DESC_OCTETS * DESC_PER_PKT * 8) / 1000;
  endfunction

  function automatic int ctx_gbps_at_miss_pct(int miss_pct);
    return (PPS_MILLION * CTX_MISS_OCTETS * 8 * miss_pct) / 100000;
  endfunction

  function automatic int host_headroom_gbps();
    return HOST_USABLE_GBPS - RATE_GBPS - desc_gbps();
  endfunction

  // Section 12: the miss rate at which payload + descriptors + context
  // traffic exactly fills the host interface.
  function automatic int critical_miss_pct();
    automatic int full = (PPS_MILLION * CTX_MISS_OCTETS * 8) / 1000;
    if (full == 0) return 100;
    return (host_headroom_gbps() * 100) / full;
  endfunction

endpackage

Classification: a package whose central function returns a miss rate rather than a size.

What it teaches: that host_headroom_gbps() is what everything else in the NIC spends. After the payload's 100 Gb/s and the descriptors' 38, a Gen5 ×16 link has 239.9 Gb/s left, and that surplus is the budget for context fills, evictions, interrupt writes, doorbell reads and every other host transaction the NIC makes. A switch's headroom is measured in megabytes of buffer; a NIC's is measured in gigabits per second of a shared pipe.

And it teaches that CTX_MISS_OCTETS is twice CTX_OCTETS. A cache miss is not one transfer — it is a fill and, if the evicted line is dirty, a writeback, and a connection context is almost always dirty because the connection just advanced. That factor of two halves the sustainable miss rate, from 78.7% to 39.4%, and it is the difference between a cliff a workload can reach and one it cannot.

Deliberately simplified: TLP_EFF_PCT = 75 is a single number where real PCIe efficiency depends on payload size — a 64-octet write is far less efficient than a 512-octet one, and a NIC moving minimum frames gets worse efficiency exactly when it needs more. PPS_MILLION floors 148.81 to 148, a 0.5% error carried through every function. CTX_OCTETS = 256 is a plausible figure and a real context depends entirely on the transport. And DESC_PER_PKT = 2 ignores batching — Section 5's subject.

Production implication: the payload-size dependence of TLP_EFF_PCT is the simplification that most flatters the design, and it fails in the same direction as everything else in this chapter. A 64-octet payload write with a 24-octet TLP header is 73% efficient; a 512-octet write is 95%. So a NIC receiving minimum-size frames faces both the highest packet rate and the worst host-interface efficiency, and the two multiply. At 148.81 Mpps with 64-octet writes the effective usable bandwidth is nearer 368 Gb/s than 378, and the critical miss rate falls with it. Model the efficiency as a function of the transfer size, because the workload that stresses the packet rate is the same one that ruins the efficiency.

// ---------------------------------------------------------------------
// host_bandwidth_model -- everything that crosses the host interface,
// added up, against what the link provides.
//
// The block exists so that "the NIC does not reach line rate" becomes a
// bandwidth statement with named contributors rather than a mystery.
// ---------------------------------------------------------------------
module host_bandwidth_model
  import dcnic_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [15:0] line_rate_gbps,
  input  logic [15:0] pps_million,
  input  logic [7:0]  ctx_miss_pct,
  input  logic [7:0]  desc_batch,        // descriptors per DMA transaction
  input  logic [15:0] interrupt_rate_khz,

  output logic [15:0] payload_gbps,
  output logic [15:0] descriptor_gbps,
  output logic [15:0] context_gbps,
  output logic [15:0] interrupt_gbps,
  output logic [15:0] total_gbps,
  output logic [15:0] headroom_gbps,
  output logic        fits,
  output logic [7:0]  critical_miss_pct_o,
  output logic [15:0] largest_contributor
);
  always_comb begin
    payload_gbps = line_rate_gbps;

    // Batching amortises the DMA transaction over several descriptors
    // but does NOT reduce the bytes moved. Section 5.
    descriptor_gbps = 16'((32'(pps_million) * DESC_OCTETS
                           * DESC_PER_PKT * 8) / 1000);

    context_gbps = 16'((32'(pps_million) * CTX_MISS_OCTETS * 8
                        * 32'(ctx_miss_pct)) / 100000);

    // An interrupt is a small write plus the host's read of the ring.
    interrupt_gbps = 16'((32'(interrupt_rate_khz) * 64 * 8) / 1_000_000);

    total_gbps   = payload_gbps + descriptor_gbps + context_gbps
                 + interrupt_gbps;
    headroom_gbps = (total_gbps < 16'(HOST_USABLE_GBPS))
                  ? (16'(HOST_USABLE_GBPS) - total_gbps) : 16'd0;
    fits = (total_gbps <= 16'(HOST_USABLE_GBPS));

    critical_miss_pct_o = 8'(critical_miss_pct());

    largest_contributor = (context_gbps > descriptor_gbps)
                        ? context_gbps : descriptor_gbps;
  end
endmodule

Classification: a bandwidth budget with four named consumers, of which only one carries data.

What it teaches: that desc_batch is an input the block deliberately ignores in descriptor_gbps. Batching descriptors — fetching four in one 64-octet transaction — amortises the per-transaction overhead and does not reduce the bytes. Chapter 18.2 §14's cache line holds four descriptors, so batching improves TLP efficiency and leaves the 38.1% figure alone. That distinction is the one most often got wrong when somebody proposes batching as a fix.

And it teaches that largest_contributor is usually context_gbps rather than descriptor_gbps. At the critical miss rate of 39.4%, context traffic is 239.9 Gb/s against the descriptors' 38.1 — six times larger — which means a NIC in trouble is almost always in trouble about cache, not about rings.

Deliberately simplified: interrupt_gbps models an interrupt as a 64-octet write, ignoring the host's subsequent ring reads which are usually larger. Nothing here models the direction of traffic, and PCIe's two directions are independent — a receive-heavy NIC saturates one and leaves the other idle. ctx_miss_pct is an input rather than derived from a working set, which Section 11 supplies. And fits is a steady-state statement that says nothing about a burst.

Production implication: the missing directionality is a real 2× and it is worth exploiting rather than merely noting. PCIe is full duplex: a receive path writes payload to the host and reads descriptors from it, so the two flows use opposite directions. Payload writes are 100 Gb/s upstream; descriptor fetches are 19 Gb/s downstream and writebacks 19 upstream — so the upstream direction carries 119 and the downstream 19. A model that adds them into one 138 figure over-constrains the design by nearly a factor of two on the lighter direction, and a design that sizes both directions from the sum buys lanes it will never use. Model the two directions separately; the critical miss rate on the busier one is what matters.

4. The Descriptors Cost 38% of the Payload

A descriptor carries no data and it is 38% of the link's payload bandwidth. Derive it, and then check what does and does not reduce it.

The arithmetic.

per packet: one descriptor fetched (16 octets) + one written back (16) = 32 octets

at 148.81 Mpps: 148.81 × 10⁶ × 32 × 8 = 38.1 Gb/s

And it is worse than it looks, because the overhead is per packet and the payload is per octet.

A factor of ninety-five between the extremes, and the workload that produces minimum frames — an all-to-all, an RPC storm, a storage metadata burst — is exactly the one that also produces the highest packet rate. The descriptor overhead is maximal precisely when the NIC is busiest.

Four things that are proposed as fixes, and what each actually does.

Row one is the one that gets proposed and it does not reduce the 38.1%. Chapter 18.2 §14's cache line holds four 16-octet descriptors; fetching four in one transaction moves the same 64 octets and saves the per-transaction header. That is a real improvement in TLP efficiency and it leaves the byte count untouched.

Row three is the one that works and it changes the programming model. A receive path that draws from a pre-posted buffer pool rather than a per-packet descriptor eliminates the fetch entirely, halving the overhead to 19 Gb/s — at the cost of the flexibility a descriptor gives, since every buffer must then be the same size and the software must post them in advance.

Descriptors are the price of a flexible receive path, and at minimum frame size the price is 38% of the link.

5. RTL 2 — The Descriptor Rate Model

// ---------------------------------------------------------------------
// descriptor_rate_model -- Chapter 18.2's ring, at 148.81 million
// packets per second.
//
// The block's purpose is to separate two things that are constantly
// conflated: the BYTES a descriptor scheme moves, which batching does
// not change, and the TRANSACTIONS it makes, which batching does.
// ---------------------------------------------------------------------
module descriptor_rate_model
  import dcnic_pkg::*;
#(
  parameter int BATCH = 4          // descriptors per host transaction
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        pkt_arrives,
  input  logic [15:0] pps_million,
  input  logic [15:0] frame_octets,

  output logic [15:0] desc_octets_per_pkt,
  output logic [15:0] desc_gbps_o,
  output logic [15:0] desc_overhead_pct,
  output logic [31:0] transactions_per_second,
  output logic [15:0] tlp_efficiency_pct,
  output logic        batching_reduces_bytes,
  output logic [31:0] c_fetches,
  output logic [31:0] c_writebacks,
  output logic [31:0] c_prefetch_underruns
);
  logic [15:0] payload_gbps;

  always_comb begin
    desc_octets_per_pkt = 16'(DESC_OCTETS * DESC_PER_PKT);   // 32

    desc_gbps_o = 16'((32'(pps_million) * 32'(desc_octets_per_pkt) * 8)
                      / 1000);

    payload_gbps = 16'((32'(pps_million) * 32'(frame_octets) * 8) / 1000);
    desc_overhead_pct = (payload_gbps == 0) ? 16'd0
                      : (desc_gbps_o * 16'd100) / payload_gbps;

    // Batching changes THIS and not desc_gbps_o.
    transactions_per_second = (32'(pps_million) * 1_000_000 * DESC_PER_PKT)
                              / BATCH;

    // A 64-octet TLP carries 24 octets of header; a 256-octet one
    // carries the same 24.
    tlp_efficiency_pct = 16'((BATCH * DESC_OCTETS * 100)
                             / ((BATCH * DESC_OCTETS) + 24));

    // Always false, and it is the point.
    batching_reduces_bytes = 1'b0;
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      c_fetches <= '0; c_writebacks <= '0; c_prefetch_underruns <= '0;
    end else if (pkt_arrives) begin
      c_fetches    <= c_fetches + 32'd1;
      c_writebacks <= c_writebacks + 32'd1;
    end
  end
endmodule

Classification: a rate model whose most useful output is a hard-wired 1'b0.

What it teaches: that batching_reduces_bytes is false and tlp_efficiency_pct is what batching moves. Fetching four 16-octet descriptors in one transaction moves 64 octets instead of four lots of 16 — the same 64 — and pays one 24-octet TLP header instead of four. Efficiency goes from 40.0% to 72.7%; the descriptor bandwidth stays at 38.1 Gb/s. Proposing batching as a fix for the 38% is the commonest error in this area and the block exists to refuse it.

And it teaches that c_prefetch_underruns is the counter that matters and is never incremented here. Chapter 18.2 §3's prefetch store holds descriptors fetched ahead of need; an underrun means a packet arrived and no descriptor was ready, which at 6.72 ns per packet and a microsecond of host latency is a 149-packet hole. The model does not produce it because it has no latency; Section 7 supplies the missing half.

Deliberately simplified: BATCH is a compile-time constant where a real prefetcher batches opportunistically. tlp_efficiency_pct assumes a 24-octet header, which is right for a 64-bit-addressed memory write and wrong for other transaction types. desc_overhead_pct divides by payload rather than by the total, so it reads 38% rather than 27.6% — both are used in practice and the block should say which. And the two counters increment together, so a design that writes back in batches is not modelled.

Production implication: the prefetch store's depth is set by the same Little's-law argument as everything else at this boundary, and getting it wrong produces a failure that looks like a fabric problem. At 148.81 Mpps and 1 µs of host read latency the NIC needs 149 descriptors in flight to never underrun; Chapter 18.2 §3's store holds eight. A store sized for a 10 Gb/s part — where the requirement is 15 — underruns constantly at 100 Gb/s, and the symptom is receive drops at line rate with a host that is not busy and a ring that is not full. Size the prefetch store from rate × latency, and count the underruns.

6. At 400 Gb/s the Descriptors Break the Host Interface

Run Section 4's arithmetic forward through three generations of line rate and one of PCIe, and the result is a wall that is not where people expect it.

Against the host interface.

Crossing the two tables.

Row three is the finding. A 400 Gb/s NIC needs 552.4 Gb/s of host bandwidth and a Gen5 ×16 link provides 378 — it misses by 46% — and 152.4 of those gigabits are descriptors. Without the descriptor overhead the payload alone would fit with room to spare.

A 400 Gb/s NIC does not fit on PCIe Gen5 ×16 because of its descriptors, not because of its payload. The data needs 400 and the bookkeeping needs 152.

Three responses, and their costs.

Row two is what datasheets do and it is worth being explicit about. A 400 Gb/s NIC's specification is usually written against large frames, where the packet rate is 32.8 Mpps and the descriptor overhead is 8.4 Gb/s — 2.1%. That configuration fits Gen5 ×16 comfortably. The minimum-frame configuration does not fit at all, and both are the same part.

Even at maximum frame size a 400 Gb/s NIC is 8% over a Gen5 ×16 link, which is why 400 Gb/s parts are Gen5 ×16 with a caveat or Gen6, and why the caveat is usually a frame-size assumption buried in a footnote.

7. RTL 3 — The In-Flight Model

// ---------------------------------------------------------------------
// inflight_model -- Little's law at the host boundary, for the third
// time in the track.
//
// Chapter 19.6 Section 4 applied it to a memory interface and got 12 KiB
// at 100 Gb/s and 1 us. Chapter 21.8 Section 5 applied it to a throughput
// ceiling. Here it sets the depth of every prefetch store and every
// outstanding-request tracker on the part.
// ---------------------------------------------------------------------
module inflight_model
  import dcnic_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [15:0] pps_million,
  input  logic [15:0] host_latency_ns,
  input  logic [15:0] rate_gbps,
  input  logic [15:0] prefetch_depth,
  input  logic [15:0] outstanding_limit,

  output logic [15:0] descriptors_in_flight,
  output logic [31:0] payload_in_flight_bytes,
  output logic [15:0] prefetch_shortfall,
  output logic        underrun_possible,
  output logic [15:0] achievable_pps_million,
  output logic [15:0] achievable_gbps,
  output logic [31:0] c_underruns
);
  always_comb begin
    // Little's law: in flight = rate x latency.
    descriptors_in_flight = 16'((32'(pps_million) * 32'(host_latency_ns))
                                / 1000);

    payload_in_flight_bytes = (32'(rate_gbps) * 32'(host_latency_ns)) / 8;

    prefetch_shortfall = (descriptors_in_flight > prefetch_depth)
                       ? (descriptors_in_flight - prefetch_depth) : 16'd0;
    underrun_possible  = (prefetch_shortfall != 16'd0);

    // Chapter 21.8 Section 5's ceiling, at this boundary: if the design
    // cannot hold enough in flight, the rate it achieves is the count
    // divided by the latency -- and the latency is the HOST's.
    achievable_pps_million = (host_latency_ns == 0) ? pps_million
                           : 16'((32'(outstanding_limit) * 1000)
                                 / 32'(host_latency_ns));
    achievable_gbps = 16'((32'(achievable_pps_million) * 672) / 1000);
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n)                  c_underruns <= '0;
    else if (underrun_possible)  c_underruns <= c_underruns + 32'd1;
  end
endmodule

Classification: Little's law, for the third time in the track, at a boundary the design does not own either end of.

What it teaches: that achievable_gbps is Chapter 21.8 §5's ceiling arriving at a new interface. A NIC that can hold outstanding_limit descriptor requests in flight across a host latency of host_latency_ns achieves outstanding / latency packets per second, whatever its line rate is. With 64 outstanding and 1 µs of latency that is 64 Mpps — 43 Gb/s of a 100 Gb/s port, and the port is healthy throughout.

And it teaches that payload_in_flight_bytes reproduces Chapter 19.6 §4 exactly. 100 Gb/s across 1 µs is 12.5 kB = 12.2 KiB, and that chapter's reorder buffer was 12 KiB. The same arithmetic, the same answer, a different block — which is the point of noticing that Little's law appears at every boundary where a rate meets a latency the design does not own.

Deliberately simplified: host_latency_ns is a constant where real host latency has a long tail — a TLB miss, a memory-controller conflict or a power-state transition can multiply it — and a design sized for the mean underruns on the tail. achievable_pps_million assumes every outstanding slot is productive, ignoring the ones spent on retries. And nothing here models the variance, which is what actually sizes a prefetch store: a store sized for the mean latency underruns half the time.

Production implication: the latency tail is what turns a correct Little's-law calculation into an underrunning design, and it is the host's property rather than the NIC's. A prefetch store sized for a 1 µs mean underruns whenever the host delivers 3 µs, which happens on a TLB miss or a contended memory controller; at 148.81 Mpps a 2 µs excursion is a 298-descriptor hole. The defence is to size the store against a high percentile rather than the mean — 298 or 450 descriptors rather than 149 — which is 4.8 or 7.2 kB of on-die store — 38 144 to 57 600 BCE, 0.13 to 0.20 of a MAC datapath. It is the cheapest possible insurance against a failure whose symptom is receive drops at line rate with no visible cause.

8. Little's Law at the Host Boundary, Again

The same relation has now produced a design constraint at four different interfaces in this track. This section collects them, because the pattern is more useful than any one instance.

in flight = rate × latency

Row four is the same calculation with packets as the unit rather than octets, and it produces a number that is easy to get wrong by an order of magnitude: Chapter 18.2 §3's prefetch store holds eight.

A store sized for a gigabit part is short by nineteen times at a hundred, and the failure is not graceful: when the store is empty and a packet arrives, the packet is dropped, at line rate, with a host that is idle and a ring that is not full.

And the same relation gives the throughput ceiling when the store cannot be made large enough.

achievable packets/s = outstanding ÷ latency

Row one is Chapter 21.8's ceiling in its most extreme form — a 100 Gb/s port delivering 5.4% because of a prefetch store sized for a previous generation — and it is invisible to every counter on the link.

Every boundary in this track where a rate meets a latency the design does not own has produced the same calculation and the same failure: a structure sized for the previous generation, and a ceiling nobody computed.

One qualification that matters, and it is the one Section 7's production note raised. The latency in these tables is a mean, and the structure must be sized against a tail. A 3 µs excursion at 148.81 Mpps is a 446-descriptor hole, so the honest sizing is three times the mean's answer — and that is still only 7.2 kB of on-die store, 0.20 of a MAC receive datapath. The structure is cheap; the arithmetic is what gets skipped.

9. RTL 4 — The Connection Context Cache

// ---------------------------------------------------------------------
// context_cache -- the state a NIC holds per connection, and the
// reason a NIC's dominant structure is a CACHE where a switch's is a
// BUFFER.
//
// Chapter 23.3 Section 4 found a switch that is 91% packet buffer.
// Section 10 finds a NIC that is 66% connection context, and the
// difference is that a buffer holds what arrived while a cache holds
// what it guessed would be needed.
// ---------------------------------------------------------------------
module context_cache
  import dcnic_pkg::*;
#(
  parameter int ENTRIES = CTX_CACHE_ENTRIES,    // 16 384
  parameter int WAYS    = 8
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        lookup,
  input  logic [31:0] conn_id,
  input  logic [31:0] working_set,       // distinct connections in use

  output logic        hit,
  output logic        miss,
  output logic        evict_dirty,
  output logic [15:0] miss_pct,
  output logic [31:0] fill_octets,
  output logic [31:0] writeback_octets,
  output logic [15:0] miss_bandwidth_gbps,
  output logic        exceeds_capacity,
  output logic [31:0] c_lookups,
  output logic [31:0] c_misses,
  output logic [31:0] c_dirty_evictions
);
  // The model is a capacity model rather than a tag array: for a
  // working set W against a capacity C with a reasonably random access
  // pattern, the hit rate is about min(1, C/W). That approximation is
  // what makes Section 12's cliff computable.
  always_comb begin
    exceeds_capacity = (working_set > 32'(ENTRIES));

    miss_pct = exceeds_capacity
             ? 16'(((working_set - 32'(ENTRIES)) * 100) / working_set)
             : 16'd0;

    hit  = lookup && !exceeds_capacity;
    miss = lookup &&  exceeds_capacity;

    // A connection whose context was just used has almost certainly
    // been advanced, so the evicted line is dirty. That doubles the
    // cost of a miss and halves the sustainable miss rate.
    evict_dirty = miss;

    fill_octets      = miss ? 32'(CTX_OCTETS) : 32'd0;
    writeback_octets = evict_dirty ? 32'(CTX_OCTETS) : 32'd0;

    miss_bandwidth_gbps = 16'((32'(PPS_MILLION) * 32'(CTX_MISS_OCTETS)
                               * 8 * 32'(miss_pct)) / 100000);
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      c_lookups <= '0; c_misses <= '0; c_dirty_evictions <= '0;
    end else begin
      if (lookup)      c_lookups         <= c_lookups + 32'd1;
      if (miss)        c_misses          <= c_misses + 32'd1;
      if (evict_dirty) c_dirty_evictions <= c_dirty_evictions + 32'd1;
    end
  end
endmodule

Classification: a capacity model rather than a cache, because the chapter's subject is the cliff and not the tags.

What it teaches: that evict_dirty equals miss, and that equality is what makes the cliff reachable. A connection whose context was just fetched has been advanced — a sequence number moved, a window updated, a queue pointer incremented — so the line it evicts is dirty and must be written back. A miss therefore costs 512 octets rather than 256, and the sustainable miss rate halves from 78.7% to 39.4%.

And it teaches that exceeds_capacity is a step rather than a slope. While the working set fits, the miss rate is zero; the moment it does not, misses appear in proportion to the excess. Throughput is continuous in the NIC's parameters and discontinuous in the workload's, which is Section 20's refused class stated as a signal.

Deliberately simplified: the hit rate is min(1, C/W), which is right for a uniformly random access pattern and optimistic for a skewed one and pessimistic for a perfectly looping one. There are no tags, no ways and no replacement policy — WAYS is declared and unused — so conflict misses are absent and only capacity misses are modelled. miss_pct is recomputed combinationally from a working_set input the NIC cannot actually observe. And the model has no warm-up, so a cold cache reads as a perfectly warm one.

Production implication: the NIC's inability to observe its own working set is what makes this failure hard to diagnose in the field, and the fix is a counter rather than a mechanism. A NIC knows its miss count and its lookup count and therefore its miss rate; what it does not know is why, and the two candidates — a working set that outgrew the cache, and a skewed access pattern causing conflict misses — have different remedies. Publishing both the miss rate and the number of distinct connection identifiers seen in a window separates them: a high miss rate with a large distinct count is capacity, and a high miss rate with a small one is conflict. A 4 096-entry approximate distinct counter is a few kilobits — under 0.02 of a MAC datapath — and it converts the NIC is slow into the working set is 32 768 and the cache is 16 384.

10. A NIC's Dominant State Is Context, Not Buffer

Chapter 23.3 §4 accounted a switch in BCE and found it 91% packet buffer. The same accounting on a NIC gives a different answer, and the difference is the chapter.

Set against the switch.

Row four is the distinction that matters and it is not about size. A packet buffer holds things that have definitely arrived; running out of it produces a drop, which is Chapter 21.6's well-mapped territory with counters and a taxonomy. A context cache holds things the design predicted would be needed; running out of it produces a fetch across the host interface, which consumes the resource Section 3 showed everything else is already competing for.

A switch's dominant structure fails by filling. A NIC's fails by missing, and a miss does not drop a packet — it spends host bandwidth.

And the sizes scale differently with the thing they serve.

Row two is the useful one. A 400 Gb/s NIC serving the same application mix needs the same context cache as a 100 Gb/s one, because the working set is a property of the software. So the context cache's share of a faster NIC falls, and the descriptor and buffer structures grow around it — but the cliff does not move, because the cliff is set by the working set and the host bandwidth, both of which the NIC does not control.

11. RTL 5 — The Miss-Rate Collapse

// ---------------------------------------------------------------------
// context_miss_model -- where the cliff is, and what falls off it.
//
// Section 3's host interface has 239.9 Gb/s of headroom after payload
// and descriptors. A context miss costs 512 octets. At 148.81 million
// packets per second that headroom buys a miss rate of 39.4% and no
// more.
// ---------------------------------------------------------------------
module context_miss_model
  import dcnic_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [15:0] miss_pct,
  input  logic [15:0] line_rate_gbps,
  input  logic [15:0] pps_million,

  output logic [15:0] ctx_gbps,
  output logic [15:0] total_host_gbps,
  output logic [15:0] headroom_gbps,
  output logic [7:0]  critical_miss_pct_o,
  output logic        past_the_cliff,
  output logic [15:0] achieved_rate_gbps,
  output logic [15:0] achieved_pct,
  output logic [31:0] c_cliff_cycles
);
  logic [15:0] desc_gbps_l;
  logic [15:0] full_miss_gbps;

  always_comb begin
    desc_gbps_l = 16'((32'(pps_million) * DESC_OCTETS * DESC_PER_PKT * 8)
                      / 1000);

    full_miss_gbps = 16'((32'(pps_million) * CTX_MISS_OCTETS * 8) / 1000);

    ctx_gbps = 16'((32'(full_miss_gbps) * 32'(miss_pct)) / 100);

    total_host_gbps = line_rate_gbps + desc_gbps_l + ctx_gbps;

    headroom_gbps = (16'(HOST_USABLE_GBPS) > (line_rate_gbps + desc_gbps_l))
                  ? (16'(HOST_USABLE_GBPS) - line_rate_gbps - desc_gbps_l)
                  : 16'd0;

    critical_miss_pct_o = (full_miss_gbps == 0) ? 8'd100
                        : 8'((32'(headroom_gbps) * 100)
                             / 32'(full_miss_gbps));

    past_the_cliff = (total_host_gbps > 16'(HOST_USABLE_GBPS));

    // Past the cliff the NIC is host-bandwidth limited, and the rate it
    // achieves is what is left after the bookkeeping takes its share.
    achieved_rate_gbps = past_the_cliff
      ? ((16'(HOST_USABLE_GBPS) > (desc_gbps_l + ctx_gbps))
          ? (16'(HOST_USABLE_GBPS) - desc_gbps_l - ctx_gbps) : 16'd0)
      : line_rate_gbps;

    achieved_pct = (line_rate_gbps == 0) ? 16'd0
                 : (achieved_rate_gbps * 16'd100) / line_rate_gbps;
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n)              c_cliff_cycles <= '0;
    else if (past_the_cliff) c_cliff_cycles <= c_cliff_cycles + 32'd1;
  end
endmodule

Classification: a bandwidth budget with a threshold, and the threshold is the chapter's headline number.

What it teaches: that achieved_rate_gbps falls linearly past the cliff while miss_pct rises, which makes the collapse steep rather than gradual. At a 39.4% miss rate the NIC delivers 99.7 Gb/s; at 50% it delivers 35.1; at 55.7% it delivers nothing at all — the descriptors and context fills have taken the entire host interface. The whole collapse is sixteen percentage points wide, and the NIC is not broken at any point on it.

And it teaches that past_the_cliff is the only Boolean in the chapter that a design can act on. Everything else — miss rates, working sets, bandwidths — is a continuous quantity that a control loop could respond to. Crossing the cliff is a state, and the right response is not to tune but to shed: reduce the number of active connections, or accept a lower rate deliberately rather than by collapse.

Deliberately simplified: the model assumes host bandwidth is shared fairly between payload, descriptors and context fills, where a real interface arbitrates by transaction and a burst of context fills can starve the payload entirely. achieved_rate_gbps subtracts the full context traffic even past the cliff, where in reality fewer packets are processed so fewer misses occur — the model overstates the collapse's steepness and gets its location right. And nothing models the queueing delay a saturated host interface adds, which is what the application actually observes.

Production implication: the model's assumption of fair arbitration is the one that matters for the failure's shape, and reality is worse. PCIe does not arbitrate between a NIC's own transaction types — the NIC does, in its own request scheduler — and a naive scheduler that issues context fills as they are needed will starve payload writes, because a fill blocks a packet and a payload write does not. The result is that crossing the cliff produces a receive-buffer overflow rather than a graceful rate reduction, and the symptom is drops rather than slowness. Give payload writes priority over context fills and accept the stalls, so that the failure is a rate reduction the application can observe rather than a drop it cannot.

12. Thirty-Nine Per Cent, and Why Chapter 23.2's Remedy Breaks It

Chapter 23.2 §11's production note warned that striping past the NIC's context cache produces a collapse that looks like the fabric problem it was deployed to fix. This section derives it.

The cliff, from Section 11.

And the working set, from Chapter 23.2.

Row five is past the cliff. Chapter 23.2 §12 measured striping ×32 as taking a ring all-reduce from 29.6 Gb/s to 93.8 — the best fabric-side result in that chapter — and this chapter's arithmetic says the NIC cannot deliver it, because a 50% context miss rate needs 304.8 Gb/s of host bandwidth against 239.9 available.

Crossing the two chapters.

So the optimum is at ×16 and not ×32, and the reason is in a different chapter, in a different part, measured in a different unit. Striping ×8 delivers 77.1 Gb/s comfortably; ×16 delivers 87.9 with no margin; ×32 delivers 93.8 on paper and 35.1 in practice.

The fabric's best configuration and the NIC's are not the same configuration, and neither part can see the other's constraint.

Which gives the practical rule this chapter exists to state.

stripes ≤ context cache entries ÷ participants

And the rule has a consequence for Chapter 23.2's ranking that is worth stating plainly. That chapter ranked five remedies and put load-aware path selection and hierarchical reduction at 100 Gb/s, striping ×8 at 77.1. Striping's ceiling is not the efficiency curve — it is the NIC's context cache, and the two remedies that reach 100 Gb/s do not create connections at all. Hierarchical reduction sends one flow per leaf; load-aware selection places the flows that already existed. Neither touches the cache.

Rows one and two were already ranked first in Chapter 23.2 §12 on fabric grounds. This chapter adds a second, independent reason to prefer them, and it is the stronger one: they cost nothing at the host.

13. RTL 6 — The Queue Set and Interrupt Moderation

// ---------------------------------------------------------------------
// queue_set_model -- 2048 queues, one host, and the arithmetic of not
// interrupting it to death.
//
// Chapter 18.7 Section 11 built a queue set and Section 9 its indirection
// table. This block is what happens to both at 148.81 million packets
// per second, where an interrupt per packet is not merely wasteful --
// it is impossible.
// ---------------------------------------------------------------------
module queue_set_model
  import dcnic_pkg::*;
#(
  parameter int Q = QUEUES              // 2048
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        pkt_arrives,
  input  logic [10:0] pkt_queue,
  input  logic [15:0] moderation_us,
  input  logic [15:0] moderation_frames,
  input  logic [15:0] pps_million,

  output logic        interrupt,
  output logic [31:0] interrupts_per_second,
  output logic [15:0] packets_per_interrupt,
  output logic [31:0] added_latency_ns,
  output logic [15:0] host_cycles_per_interrupt,
  output logic [15:0] host_cores_needed_x10,
  output logic [31:0] c_interrupts,
  output logic [31:0] c_packets
);
  logic [15:0] pending;
  logic [31:0] timer_ns;

  always_comb begin
    // Two conditions, whichever fires first: a frame count or a timer.
    interrupt = (pending >= moderation_frames) ||
                ((timer_ns >= (32'(moderation_us) * 1000)) &&
                 (pending != 16'd0));

    interrupts_per_second = (moderation_us == 0) ? 32'hFFFF_FFFF
                          : (1_000_000 / 32'(moderation_us));

    packets_per_interrupt = (interrupts_per_second == 0) ? 16'd0
                          : 16'((32'(pps_million) * 1_000_000)
                                / interrupts_per_second);

    // The cost of moderation: a packet may wait a whole window.
    added_latency_ns = 32'(moderation_us) * 1000;

    // An interrupt costs the host an entry, a ring walk and a return.
    host_cycles_per_interrupt = 16'd2000;

    // At 3 GHz, what fraction of a core the interrupts alone consume.
    host_cores_needed_x10 = 16'((interrupts_per_second
                                 * 32'(host_cycles_per_interrupt) * 10)
                                / 3_000_000_000);
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      pending <= '0; timer_ns <= '0; c_interrupts <= '0; c_packets <= '0;
    end else begin
      if (pkt_arrives) begin
        pending  <= pending + 16'd1;
        c_packets <= c_packets + 32'd1;
      end
      timer_ns <= timer_ns + 32'd1;
      if (interrupt) begin
        pending      <= '0;
        timer_ns     <= '0;
        c_interrupts <= c_interrupts + 32'd1;
      end
    end
  end
endmodule

Classification: a moderation timer, and its outputs are a latency and a fraction of a CPU core.

What it teaches: that an interrupt per packet is arithmetically impossible at these rates. At 148.81 Mpps and 2 000 host cycles per interrupt, an unmoderated NIC would demand 2.98 × 10¹¹ cycles per second — ninety-nine 3 GHz cores doing nothing but entering and leaving interrupt handlers. Moderation is not an optimisation; it is the only way the arrangement works at all.

And it teaches the trade in one pair of numbers. A 50 µs moderation window costs 50 µs of added latency and reduces the interrupt rate to 20 000 per second — 0.013 of a core. A 5 µs window costs 5 µs and needs 0.13 of a core. The latency added is exactly the window, and the CPU saved is inversely proportional to it.

Deliberately simplified: host_cycles_per_interrupt = 2000 is a single constant covering entry, the ring walk, the packet's delivery and the return — it varies by more than an order of magnitude with the packets per interrupt. The timer increments once per clock as if the clock were 1 GHz. pending counts packets across all 2 048 queues where moderation is per queue, which is the block's largest omission: 2 048 queues each moderated at 50 µs produce 40.96 million interrupts per second, not 20 000.

Production implication: that per-queue omission inverts the block's conclusion and is the reason multi-queue NICs moderate the way they do. Moderation is per queue, so the interrupt rate is queues ÷ window — at 2 048 queues and a 50 µs window that is 40.96 M/s, which is 27 cores of pure interrupt overhead. Real designs avoid it in three ways: only queues with traffic generate interrupts (so the rate follows the active queue count, not the configured one); a single queue's window is adaptive, lengthening under load; and busy-polling drivers disable interrupts entirely above a rate threshold. The last is the one that matters at 100 Gb/s — a polled receive path has no interrupts at all — and it means the moderation arithmetic above describes the idle and lightly loaded regime, which is where latency matters and throughput does not.

14. What a NIC Must Never Do

Five prohibitions, each derived earlier.

One — never quote a line rate without a frame size.

A factor of eighteen in the overhead between two configurations of the same part.

Two — never propose batching as a fix for descriptor bandwidth.

Section 5: batching four 16-octet descriptors into one transaction moves the same 64 octets. It improves TLP efficiency from 40.0% to 72.7% and leaves the 38.1 Gb/s untouched. What reduces the bytes is a smaller descriptor or no descriptor at all.

Three — never size a prefetch store from the mean host latency.

A store sized for the mean underruns on the tail, and the symptom is receive drops at line rate with an idle host and a non-full ring — which looks like a fabric problem and is not.

Four — never let a context miss take priority over a payload write.

Section 11's production note. A context fill blocks a packet and a payload write does not, so a naive request scheduler issues fills first and starves the payload — turning a rate reduction into a receive-buffer overflow. Give payload writes priority and accept the stalls, so that crossing the cliff is observable as slowness rather than as loss.

Five — never let a fabric-side optimisation choose the connection count without checking the cache.

Section 12's rule: stripes ≤ context cache entries ÷ participants. It is one division, it belongs in the collective library's configuration, and nobody currently performs it.

15. RTL 7 — NIC Telemetry

// ---------------------------------------------------------------------
// nic_telemetry -- what the part knows about itself, and the two fields
// that separate the chapter's three failure modes.
//
// Those two are the context miss rate and the distinct-connection
// count. Together they say whether a slow NIC is short of cache, short
// of prefetch, or short of host bandwidth -- and no other pair does.
// ---------------------------------------------------------------------
module nic_telemetry
  import dcnic_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,
  input  logic        snapshot,

  input  logic [31:0] i_packets,
  input  logic [31:0] i_ctx_lookups,
  input  logic [31:0] i_ctx_misses,
  input  logic [31:0] i_distinct_conns,
  input  logic [31:0] i_prefetch_underruns,
  input  logic [31:0] i_desc_fetches,
  input  logic [15:0] i_host_gbps_used,
  input  logic [15:0] i_achieved_gbps,
  input  logic [15:0] i_line_rate_gbps,
  input  logic [31:0] i_interrupts,

  output logic [31:0] o_packets,
  output logic [15:0] o_ctx_miss_pct,
  output logic [31:0] o_distinct_conns,
  output logic [15:0] o_cache_pressure_x10,
  output logic [15:0] o_underrun_ppm,
  output logic [15:0] o_host_utilisation_pct,
  output logic [15:0] o_achieved_pct,
  output logic [31:0] o_packets_per_interrupt,
  output logic [2:0]  o_limiter,
  output logic [2:0]  o_caveats
);
  localparam int LIM_NONE     = 3'd0;
  localparam int LIM_CACHE    = 3'd1;
  localparam int LIM_PREFETCH = 3'd2;
  localparam int LIM_HOST     = 3'd3;

  localparam int C_PAST_CLIFF = 0;
  localparam int C_UNDERRUN   = 1;
  localparam int C_NO_DISTINCT= 2;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      o_packets <= '0; o_ctx_miss_pct <= '0; o_distinct_conns <= '0;
      o_cache_pressure_x10 <= '0; o_underrun_ppm <= '0;
      o_host_utilisation_pct <= '0; o_achieved_pct <= '0;
      o_packets_per_interrupt <= '0; o_limiter <= 3'd0; o_caveats <= '0;
    end else if (snapshot) begin
      o_packets        <= i_packets;
      o_distinct_conns <= i_distinct_conns;

      if (i_ctx_lookups != 0)
        o_ctx_miss_pct <= 16'((i_ctx_misses * 100) / i_ctx_lookups);
      if (i_packets != 0)
        o_underrun_ppm <= 16'((i_prefetch_underruns * 1_000_000)
                              / i_packets);
      if (i_interrupts != 0)
        o_packets_per_interrupt <= i_packets / i_interrupts;

      // The field that distinguishes a capacity miss from a conflict
      // miss: how many distinct connections the NIC actually saw.
      o_cache_pressure_x10 <= 16'((i_distinct_conns * 10)
                                  / 32'(CTX_CACHE_ENTRIES));

      o_host_utilisation_pct <= 16'((32'(i_host_gbps_used) * 100)
                                    / 32'(HOST_USABLE_GBPS));
      if (i_line_rate_gbps != 0)
        o_achieved_pct <= (i_achieved_gbps * 16'd100) / i_line_rate_gbps;

      // The single most useful output: which of three things is
      // limiting the part.
      if (i_achieved_gbps >= i_line_rate_gbps)          o_limiter <= 3'(LIM_NONE);
      else if (o_cache_pressure_x10 > 16'd10)           o_limiter <= 3'(LIM_CACHE);
      else if (i_prefetch_underruns != 0)               o_limiter <= 3'(LIM_PREFETCH);
      else                                              o_limiter <= 3'(LIM_HOST);

      o_caveats[C_PAST_CLIFF]  <= (o_ctx_miss_pct > 16'd39);
      o_caveats[C_UNDERRUN]    <= (i_prefetch_underruns != 0);
      o_caveats[C_NO_DISTINCT] <= (i_distinct_conns == 0);
    end
  end
endmodule

Classification: a snapshot bank whose last useful output names which of three constraints is binding.

What it teaches: that o_cache_pressure_x10 is the field that makes the miss rate actionable. A high miss rate has two causes with different remedies: a working set larger than the cache (reduce the connection count — Section 12's rule) and a skewed access pattern producing conflict misses (change the hash or the associativity). The distinct-connection count separates them in one comparison, and without it the miss rate alone is a symptom with two candidate diseases.

And it teaches that o_limiter is a three-way verdict the part can reach and an operator cannot. From outside, all three failures look identical: the NIC does not reach line rate. From inside, cache pressure above 1.0, a non-zero underrun count and a saturated host interface are three distinct, mutually exclusive observations — and reporting which one holds turns an investigation into a lookup.

Deliberately simplified: o_limiter is a priority ladder rather than a measurement, so two simultaneous constraints report only the first. i_distinct_conns requires an approximate distinct counter the block does not contain. No read-clear, and i_packets at 148.81 Mpps wraps a 32-bit counter in 28.9 seconds — worse than Chapter 23.3 §15's switch, because a NIC's counters are per port rather than aggregated. And o_host_utilisation_pct uses a nominal 378 Gb/s rather than a measured one.

Production implication: the 28.9-second wrap is shorter than most telemetry polling intervals and it makes the whole block's packet-derived fields useless if ignored. A 32-bit packet counter at 148.81 Mpps wraps in 28.9 seconds; a system polling every 30 seconds reads a wrapped value more often than not, and every rate it computes is wrong by an unknown multiple of 2³². A 48-bit counter wraps in 21.9 days and a 64-bit one in 3 900 years. Widening the fifty or so packet-derived counters on a NIC from 32 to 48 bits costs 800 flops — 16 000 BCE, 0.06 of a MAC receive datapath, 0.03% of the part. There is no defensible reason not to.

16. RTL 8 — The NIC Conformance Monitor

// ---------------------------------------------------------------------
// nic_conformance_monitor -- properties for a fixed-function NIC.
//
// Properties 1 to 10 hold at every point on Section 11's collapse
// curve, including the point where the part delivers 35 Gb/s of a
// 100 Gb/s port. Properties 11 and 12 are the two that do not.
// ---------------------------------------------------------------------
module nic_conformance_monitor
  import dcnic_pkg::*;
(
  input logic        clk,
  input logic        rst_n,

  input logic [15:0] payload_gbps,
  input logic [15:0] descriptor_gbps,
  input logic [15:0] context_gbps,
  input logic [15:0] total_gbps,
  input logic        fits,
  input logic        past_the_cliff,
  input logic [15:0] achieved_gbps,
  input logic [15:0] line_rate_gbps,
  input logic [15:0] miss_pct,
  input logic [7:0]  critical_miss_pct_i,
  input logic [31:0] working_set,
  input logic        exceeds_capacity,
  input logic        batching_reduces_bytes,
  input logic [15:0] descriptors_in_flight,
  input logic [15:0] prefetch_depth,
  input logic        underrun_possible,
  input logic        interrupt,
  input logic [15:0] pending_frames,
  input logic [15:0] moderation_frames,
  input logic        claim_line_rate
);
  // 1. The host budget is the sum of its named parts.
  p_nc_budget: assert property (@(posedge clk) disable iff (!rst_n)
    total_gbps == (payload_gbps + descriptor_gbps + context_gbps));

  // 2. Fitting means the total is within the link.
  p_nc_fits: assert property (@(posedge clk) disable iff (!rst_n)
    fits == (total_gbps <= 16'(HOST_USABLE_GBPS)));

  // 3. Past the cliff is the negation of fitting.
  p_nc_cliff: assert property (@(posedge clk) disable iff (!rst_n)
    past_the_cliff == !fits);

  // 4. Batching never reduces descriptor bytes.
  p_nc_batching: assert property (@(posedge clk) disable iff (!rst_n)
    !batching_reduces_bytes);

  // 5. A miss occurs only when the working set exceeds the cache.
  p_nc_miss_cause: assert property (@(posedge clk) disable iff (!rst_n)
    (miss_pct != 0) |-> exceeds_capacity);

  // 6. The critical miss rate is headroom over full-miss traffic.
  p_nc_critical: assert property (@(posedge clk) disable iff (!rst_n)
    (miss_pct <= 16'(critical_miss_pct_i)) |-> fits);

  // 7. In-flight descriptors follow Little's law.
  p_nc_little: assert property (@(posedge clk) disable iff (!rst_n)
    underrun_possible == (descriptors_in_flight > prefetch_depth));

  // 8. An interrupt fires only with packets pending.
  p_nc_int_pending: assert property (@(posedge clk) disable iff (!rst_n)
    interrupt |-> (pending_frames != 16'd0));

  // 9. Moderation is honoured: no interrupt below the frame threshold
  //    unless the timer expired.
  p_nc_moderation: assert property (@(posedge clk) disable iff (!rst_n)
    (interrupt && (pending_frames < moderation_frames)) |-> timer_expired);

  // 10. Achieved never exceeds the line rate.
  p_nc_achieved_bounded: assert property (@(posedge clk) disable iff (!rst_n)
    achieved_gbps <= line_rate_gbps);

  // 11. A LINE-RATE CLAIM requires the working set to fit. This is the
  //     property Section 20 is about.
  p_nc_claim_needs_fit: assert property (@(posedge clk) disable iff (!rst_n)
    claim_line_rate |-> !exceeds_capacity);

  // 12. And it requires the host budget to close.
  p_nc_claim_needs_budget: assert property (@(posedge clk) disable iff (!rst_n)
    claim_line_rate |-> fits);
endmodule

Classification: twelve properties, ten about mechanism and two that gate a performance claim on a workload property.

What it teaches: that properties 11 and 12 constrain a claim using a quantity the NIC does not control. exceeds_capacity is a function of the working set, which belongs to the software; fits is a function of the host interface, which belongs to the platform. A line-rate claim is therefore conditional on two things outside the part, and the honest specification says so rather than quoting a number.

And it teaches that properties 1 to 10 hold at 35 Gb/s. The budget sums correctly, the cliff is detected, batching is honest, misses are caused by capacity, Little's law holds, moderation is honoured and the achieved rate is bounded. Every mechanism in the part is working exactly as designed while it delivers a third of its line rate, which is Section 20's subject.

Deliberately simplified: property 9 references a timer_expired signal the port list does not declare. Property 5's implication is one-directional — it catches a miss without a cause and not a cause without a miss. Property 6 is a restatement of the model rather than an independent check. And property 10's bound is trivially true by construction in the model, where in a real part an achieved rate above the line rate would indicate a measurement error worth catching.

Production implication: property 11's shape is the one to carry into a datasheet, and it inverts how NIC performance is usually specified. A specification that says 100 Gb/s line rate is making an unconditional claim that the part cannot support; one that says 100 Gb/s line rate for working sets up to 16 384 connections on a host interface providing 378 Gb/s is making a claim that is true. The two extra clauses are the entire content of this chapter, they are both measurable, and a customer who reads them can check their own workload against them before buying.

17. A NIC Against a Switch, in BCE

Chapter 23.3 §2 defined the unit so that two parts with different dominant structures could be compared. This is the comparison.

Row four is the one that is not obvious. Per gigabit of line rate a NIC carries 5.8 times more on-die state than a switch does — 507 000 BCE per Gb/s against 87 900 — which inverts the intuition that a switch, being the larger part, must be the more state-hungry one.

The reason is in what each structure is for.

Row six is the deepest difference. A switch's buffer requirement comes from traffic — a quantity the network produces and a network engineer can reason about. A NIC's context requirement comes from the application's connection count, which is chosen by software written by somebody who has never seen the NIC. Chapter 23.2 §12's striping decision is exactly that: a collective library's configuration parameter that sets a NIC's cache pressure, with neither side aware of the other.

And the two parts' failure modes follow.

Row one is worth sitting with. A NIC past Section 11's cliff drops nothing. It processes fewer packets per second because each one waits for a context fill, so the receive path applies backpressure, the link's flow control engages or the switch upstream buffers, and the loss — if any — happens somewhere else entirely. Chapter 21.6's method looks for the drop at the point of loss; here the point of loss is a switch, and the cause is a cache two hops away.

A switch that is overloaded drops packets and says so. A NIC that is overloaded slows down and says nothing, and the drops appear on somebody else's counters.

One closing comparison, to place both against the rest of the track.

Rows two, three and four span four orders of magnitude and the unit handles all of them, which is what Chapter 23.3 §2 was for.

18. What the NIC Assumes

Seven premises. Four are about the host and three about the software, and the part owns none of them.

Rows five and six are the two that decide whether a part reaches its number, and both belong to software.

Row five's specific shape is worth restating because it is a cliff. The assumption is not the working set is small; it is the working set is under 16 384, and the difference between 16 000 and 20 000 connections is the difference between 100 Gb/s and 100 Gb/s — but the difference between 16 384 and 32 768 is 100 Gb/s and 35.1. The assumption has no gradient, which is Section 20's subject.

Row six is the one datasheets handle by omission. A 400 Gb/s NIC's specification is written against a frame-size distribution, usually IMIX or maximum-size, and the minimum-frame configuration of the same part does not fit its host interface at all — Section 6's table. The assumption is stated in a footnote if it is stated at all.

And one premise that is not on the list because the part cannot even express it.

That the connection count the software chooses has anything to do with the cache the hardware provides. Chapter 23.2 §12's striping parameter is chosen to optimise a fabric efficiency curve, by a collective library, with no input from the NIC and no interface through which the NIC could supply one. Section 12's rule — stripes ≤ cache entries ÷ participants — is one division that nobody currently performs, because the two quantities live in different products.

19. The Cost, Accounted — in BCE

Chapter 23.3 §2's unit, applied to this chapter's blocks and to the part around them.

The ratio of memory to logic is 4 495 : 1, which is nearly twice Chapter 23.3 §19's switch figure of 2 371. A NIC is even more memory-dominated than a switch, relative to the logic that manages it — because the cache's control logic is a few hundred flops and the cache is thirty-three megabits.

Which makes the instrumentation argument even more one-sided than it was in Chapter 23.3.

Five hundredths of one per cent of the part, and between them they supply the miss rate, the distinct-connection count, the underrun count and rates that do not wrap in 28.9 seconds — which is everything Section 22's debugging procedure needs.

And the comparison that closes the chapter's accounting.

Row two against row four is the chapter's practical conclusion. The hardware fix for Chapter 23.2's striping collapse is a 66% larger NIC; the software fix is a division. Chapter 23.2 §19 reached the same shape — 512 flops against 3.5× the fabric — and here the ratio is starker still, because the cheap side costs nothing at all.

20. Properties Worth Asserting, and One Worth Refusing

Six groups. All thirty-four hold at a 50% context miss rate, where the part delivers 35.1 Gb/s of a 100 Gb/s port.

Group A — the host budget.

// A1. The total is the sum of its named parts.
p_hb_sum: assert property (@(posedge clk) disable iff (!rst_n)
  total_gbps == (payload_gbps + descriptor_gbps + context_gbps
                 + interrupt_gbps));

// A2. Fitting means the total is within the link.
p_hb_fits: assert property (@(posedge clk) disable iff (!rst_n)
  fits == (total_gbps <= 16'(HOST_USABLE_GBPS)));

// A3. Past the cliff is the negation of fitting.
p_hb_cliff: assert property (@(posedge clk) disable iff (!rst_n)
  past_the_cliff == !fits);

// A4. Headroom is never negative.
p_hb_headroom: assert property (@(posedge clk) disable iff (!rst_n)
  headroom_gbps >= 16'd0);

// A5. Payload never exceeds the line rate.
p_hb_payload: assert property (@(posedge clk) disable iff (!rst_n)
  payload_gbps <= line_rate_gbps);

// A6. The critical miss rate is headroom over full-miss traffic.
p_hb_critical: assert property (@(posedge clk) disable iff (!rst_n)
  (miss_pct <= 16'(critical_miss_pct_i)) |-> fits);

Group B — descriptors.

// B1. Thirty-two octets per packet, fetch plus writeback.
p_ds_octets: assert property (@(posedge clk) disable iff (!rst_n)
  desc_octets_per_pkt == 16'(DESC_OCTETS * DESC_PER_PKT));

// B2. Batching never reduces bytes.
p_ds_batching: assert property (@(posedge clk) disable iff (!rst_n)
  !batching_reduces_bytes);

// B3. Batching does improve TLP efficiency.
p_ds_tlp: assert property (@(posedge clk) disable iff (!rst_n)
  tlp_efficiency_pct ==
    16'((BATCH * DESC_OCTETS * 100) / ((BATCH * DESC_OCTETS) + 24)));

// B4. Fetches and writebacks are one-for-one per packet.
p_ds_pair: assert property (@(posedge clk) disable iff (!rst_n)
  c_fetches == c_writebacks);

// B5. Descriptor bandwidth scales with the packet rate, not the octet
//     rate.
p_ds_scales: assert property (@(posedge clk) disable iff (!rst_n)
  desc_gbps_o == 16'((pps_million * desc_octets_per_pkt * 8) / 1000));

Group C — in flight.

// C1. Little's law: in flight is rate times latency.
p_if_little: assert property (@(posedge clk) disable iff (!rst_n)
  descriptors_in_flight == 16'((pps_million * host_latency_ns) / 1000));

// C2. An underrun is possible exactly when the store is short.
p_if_underrun: assert property (@(posedge clk) disable iff (!rst_n)
  underrun_possible == (descriptors_in_flight > prefetch_depth));

// C3. The achievable rate is outstanding over latency.
p_if_achievable: assert property (@(posedge clk) disable iff (!rst_n)
  (host_latency_ns != 0) |->
    (achievable_pps_million ==
      16'((outstanding_limit * 1000) / host_latency_ns)));

// C4. Payload in flight matches Chapter 19.6 Section 4's figure.
p_if_payload: assert property (@(posedge clk) disable iff (!rst_n)
  payload_in_flight_bytes == ((rate_gbps * host_latency_ns) / 8));

// C5. The shortfall is never negative.
p_if_shortfall: assert property (@(posedge clk) disable iff (!rst_n)
  prefetch_shortfall >= 16'd0);

Group D — the context cache.

// D1. A miss occurs only when the working set exceeds the cache.
p_cx_cause: assert property (@(posedge clk) disable iff (!rst_n)
  miss |-> exceeds_capacity);

// D2. Every miss evicts a dirty line.
p_cx_dirty: assert property (@(posedge clk) disable iff (!rst_n)
  miss |-> evict_dirty);

// D3. A miss costs a fill and a writeback.
p_cx_octets: assert property (@(posedge clk) disable iff (!rst_n)
  miss |-> ((fill_octets + writeback_octets) == 32'(CTX_MISS_OCTETS)));

// D4. Hit and miss are mutually exclusive.
p_cx_exclusive: assert property (@(posedge clk) disable iff (!rst_n)
  !(hit && miss));

// D5. The miss rate follows the capacity model.
p_cx_rate: assert property (@(posedge clk) disable iff (!rst_n)
  exceeds_capacity |->
    (miss_pct == 16'(((working_set - ENTRIES) * 100) / working_set)));

// D6. Dirty evictions never exceed misses.
p_cx_evictions: assert property (@(posedge clk) disable iff (!rst_n)
  c_dirty_evictions <= c_misses);

Group E — queues and moderation.

// E1. An interrupt fires only with packets pending.
p_qs_pending: assert property (@(posedge clk) disable iff (!rst_n)
  interrupt |-> (pending != 16'd0));

// E2. Moderation is honoured.
p_qs_moderation: assert property (@(posedge clk) disable iff (!rst_n)
  (interrupt && (pending < moderation_frames)) |-> timer_expired);

// E3. An interrupt clears the pending count.
p_qs_clears: assert property (@(posedge clk) disable iff (!rst_n)
  interrupt |=> (pending == 16'd0));

// E4. The added latency is the moderation window.
p_qs_latency: assert property (@(posedge clk) disable iff (!rst_n)
  added_latency_ns == (moderation_us * 1000));

// E5. A queue index is inside the queue set.
p_qs_index: assert property (@(posedge clk) disable iff (!rst_n)
  pkt_arrives |-> (pkt_queue < 11'(QUEUES)));

// E6. Packets counted equal packets arrived.
p_qs_count: assert property (@(posedge clk) disable iff (!rst_n)
  pkt_arrives |=> (c_packets == $past(c_packets) + 1));

Group F — the claim.

// F1. A line-rate claim requires the working set to fit.
p_cl_fit: assert property (@(posedge clk) disable iff (!rst_n)
  claim_line_rate |-> !exceeds_capacity);

// F2. And the host budget to close.
p_cl_budget: assert property (@(posedge clk) disable iff (!rst_n)
  claim_line_rate |-> fits);

// F3. And a frame size the claim was measured at.
p_cl_frame_size: assert property (@(posedge clk) disable iff (!rst_n)
  claim_line_rate |-> (claimed_frame_octets != 16'd0));

// F4. The achieved rate never exceeds the line rate.
p_cl_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  achieved_gbps <= line_rate_gbps);

// F5. The limiter verdict names a constraint that is actually binding.
p_cl_limiter: assert property (@(posedge clk) disable iff (!rst_n)
  (o_limiter == LIM_CACHE) |-> exceeds_capacity);

// F6. A past-the-cliff state is reported, not smoothed.
p_cl_reported: assert property (@(posedge clk) disable iff (!rst_n)
  past_the_cliff |-> caveat_past_cliff);

Thirty-four properties, and every one holds at both ends of Section 11's collapse.

21. Verification Scenarios

Fifty-eight scenarios. Group 4's require a workload rather than a stimulus, and they are the only ones that reach the cliff.

Group 1 — the per-packet budget (8).

Group 2 — descriptors and the host interface (12).

Group 3 — in flight (10).