Offload and Multi-Queue

Module 18's last chapter spends what the previous six allocated, and both halves of it produce a result that runs against the obvious reading.

Offload moves work from the CPU into the MAC. A software TCP checksum over 1518-octet frames at 100 Gb/s is 102.8% of a 3 GHz core — so the case for moving it is arithmetic and not in doubt. Chapter 18.3 §20's callout established the cost and left it half-argued: offload moves the last end-to-end check upstream of the DMA path, and Chapter 18.5 has since added a reorder buffer to that path.

Multi-queue spreads work across cores. Chapter 18.6 left a 100 Gb/s port at 116% of one core in interrupts, which one core does not have. Eight queues divide that by eight.

And multiply the aggregate by eight, which is the finding.

Read the table with Chapter 18.6 §4's policy in mind. That policy holds each queue's interrupt rate at 1/L regardless of its arrival rate — so splitting a port's traffic across q queues gives q queues each interrupting at 1/L, and the port's total interrupt rate is q/L.

Multi-queue does not reduce the interrupt work. It divides it by the number of cores and multiplies it by the number of queues, and those are the same number only if the cores were otherwise idle.

And the second half of multi-queue is worse than that. Deciding which queue a frame goes to is Chapter 15.2's flow hash, and that chapter's balls-in-bins arithmetic applies unchanged:

More queues distribute worse, exactly as more LAG members did — and the queue count should be chosen against the flow count, not against the core count.

1. Scope, and Two Mechanisms That Look Unrelated

Offload and multi-queue are usually presented together and rarely explained together. They share one structure: both move work out of the CPU's single-threaded path, and both give something up to do it.

What this chapter establishes:

What it does not cover: encryption offload — MACsec and IPsec — which is a large enough subject to deserve its own treatment, and TCP offload in the full sense, which moves protocol state into the MAC and is a different architecture rather than an optimisation.

One framing note. Everything in Modules 1 to 17 was about frames. Offload is the first mechanism in this track that requires the MAC to understand what is inside a frame — an IP header, a TCP header, a checksum field at a known offset — and that dependency is where several of its failure modes come from.

2. What Checksums Cost in Software

The case for checksum offload is arithmetic, and it is worth doing because the answer crosses 100% at a rate people still call ordinary.

A TCP or IPv4 checksum is a ones-complement sum over the payload. A good implementation on a 64-bit core manages roughly four octets per cycle once loop overhead and carry folding are counted.

The bottom row is the argument: at 100 Gb/s a software checksum alone is a whole 3 GHz core, before the stack has parsed a header, before Chapter 18.6's interrupt cost, and before the application has seen a byte.

And the columns are worth reading across, because the shape is not what the frame-rate arithmetic in this module has trained us to expect.

Chapter 18.1 §10's interrupt cost and Chapter 18.3 §17's transaction cost both bind on minimum-size frames, because they are charged per frame. The checksum is charged per octet, so it binds on maximum-size frames — and a link carrying 1518-octet frames delivers 98.7% of its rate as payload while one carrying 64-octet frames delivers 76.2%.

Which is the first per-octet cost this module has met, and it means the worst case for offload is the opposite of the worst case for everything else — so a port sized against minimum-size frames for its transaction budget must be sized against maximum-size frames for its checksum engine, and the two never coincide.

The hardware cost of doing it in the MAC is small and worth stating for contrast. A ones-complement adder tree over a 512-bit datapath is eight 64-bit adders and a fold — a few hundred gates, running at the line rate by construction, and producing a result one cycle after the last octet.

3. RTL 1 — The Receive Checksum Engine

A ones-complement accumulator and a parser, and the parser is the part that fails.

// -----------------------------------------------------------------------
// offload_pkg -- checksum and segmentation offload, and multi-queue.
// -----------------------------------------------------------------------
package offload_pkg;

  localparam int BUS_BYTES = 64;
  localparam int NUM_QUEUE = 16;
  localparam int RSS_TBL   = 128;        // indirection table entries

  // What the parser found. Every field is an OPINION -- the MAC is
  // reading a header it did not write, and section 17 is about what
  // happens when the opinion is wrong.
  typedef struct packed {
    logic        is_ipv4;
    logic        is_ipv6;
    logic        is_tcp;
    logic        is_udp;
    logic        has_vlan;               // 13.2
    logic        has_options;            // IPv4 options -- length varies
    logic        is_fragment;            // an IP fragment: NO L4 header
    logic [7:0]  l3_offset;
    logic [7:0]  l4_offset;
    logic [15:0] l4_length;
    logic        parse_ok;               // false = offload must not apply
  } parse_t;

  typedef struct packed {
    logic        ip_checksum_ok;
    logic        l4_checksum_ok;
    logic        checksum_computed;      // false = software must do it
    logic [15:0] raw_sum;                // for a partial-checksum model
  } csum_result_t;

  // RSS: 15.2's flow hash, over the 5-tuple rather than that
  // chapter's 3-tuple.
  typedef struct packed {
    logic [31:0] hash;
    logic        hash_valid;
    logic [$clog2(NUM_QUEUE)-1:0] queue;
  } rss_t;

endpackage

// -----------------------------------------------------------------------
// rx_checksum_engine -- ones-complement sum over the L4 payload.
//
// The arithmetic is trivial; the parse is not. Every condition under
// which the offload must DECLINE is a condition the parser has to
// detect, and a parser that guesses produces a checksum result for a
// field that is not a checksum.
// -----------------------------------------------------------------------
module rx_checksum_engine
  import offload_pkg::*;
(
  input  logic                    clk,
  input  logic                    rst_n,

  input  logic                    beat_valid,
  input  logic [BUS_BYTES*8-1:0]  beat_data,
  input  logic [$clog2(BUS_BYTES+1)-1:0] beat_bytes,
  input  logic                    beat_sof,
  input  logic                    beat_eof,

  input  parse_t                  parse,

  output csum_result_t            result,
  output logic                    result_valid,

  output logic [31:0]             c_computed,
  output logic [31:0]             c_declined,
  output logic [31:0]             c_decline_fragment,
  output logic [31:0]             c_decline_unknown,
  output logic [31:0]             c_decline_options,
  output logic [31:0]             c_bad_l4,
  output logic [31:0]             c_bad_ip
);

  logic [31:0] acc;
  logic        active;

  // A ones-complement sum of 16-bit words, accumulated 32 words at a
  // time on a 512-bit bus. The carries fold at the end.
  logic [31:0] beat_sum;
  always_comb begin
    int i;
    beat_sum = '0;
    for (i = 0; i < BUS_BYTES/2; i++)
      if ((i*2) < int'(beat_bytes))
        beat_sum = beat_sum + 32'(beat_data[i*16 +: 16]);
  end

  // The conditions under which the offload MUST decline. Each is a
  // case where the L4 checksum either does not exist or cannot be
  // located, and computing one anyway produces a verdict about the
  // wrong 16 bits.
  wire decline_fragment = parse.is_fragment;      // no L4 header at all
  wire decline_unknown  = !(parse.is_tcp || parse.is_udp);
  wire decline_options  = parse.has_options;      // l4_offset is a guess
  wire decline_parse    = !parse.parse_ok;

  wire decline = decline_fragment | decline_unknown |
                 decline_options  | decline_parse;

  // Folding carries: a ones-complement sum folds twice to be safe.
  wire [31:0] fold1 = (acc & 32'hFFFF) + (acc >> 16);
  wire [15:0] fold2 = fold1[15:0] + fold1[31:16];

  assign result.checksum_computed = !decline;
  assign result.raw_sum           = fold2;
  assign result.l4_checksum_ok    = !decline && (fold2 == 16'hFFFF);
  assign result.ip_checksum_ok    = parse.is_ipv4 && parse.parse_ok;
  assign result_valid             = beat_eof;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      acc <= '0; active <= 1'b0;
      c_computed <= '0; c_declined <= '0;
      c_decline_fragment <= '0; c_decline_unknown <= '0;
      c_decline_options <= '0; c_bad_l4 <= '0; c_bad_ip <= '0;
    end else begin
      if (beat_valid && beat_sof) begin
        acc    <= beat_sum;
        active <= 1'b1;
      end else if (beat_valid && active) begin
        acc <= acc + beat_sum;
      end

      if (beat_eof) begin
        active <= 1'b0;
        if (decline) begin
          c_declined <= c_declined + 1;
          if (decline_fragment) c_decline_fragment <= c_decline_fragment + 1;
          if (decline_unknown)  c_decline_unknown  <= c_decline_unknown + 1;
          if (decline_options)  c_decline_options  <= c_decline_options + 1;
        end else begin
          c_computed <= c_computed + 1;
          if (fold2 != 16'hFFFF) c_bad_l4 <= c_bad_l4 + 1;
        end
      end
    end
  end

endmodule

Classification: a ones-complement accumulator with an explicit decline path per parse failure.

What it teaches: that checksum_computed is the most important bit the engine produces, and it is not the verdict. A design that reports only good or bad forces software to trust a result that may not have been computed at all — and every condition in the decline list is a real frame type: IP fragments have no L4 header, IPv4 options move the L4 offset, and a protocol the parser does not know has no checksum field the MAC can find. The engine must be able to say "I did not check this", and software must handle it.

And it teaches that the decline counters must be separate, because they mean different things about the traffic. c_decline_fragment rising means the path is fragmenting — an MTU problem somewhere. c_decline_options rising means something is using IPv4 options, which is rare and often interesting. c_decline_unknown rising means a protocol the MAC does not parse, which on a modern network is frequently a tunnel — and a tunnel's inner checksum is not offloadable by a parser that stops at the outer header.

Deliberately simplified: the beat sum is 32 sixteen-bit adds in one combinational tree, which is an adder tree a production design pipelines. The engine assumes the L4 payload starts at a beat boundary, which it does not — a TCP header at octet 54 begins mid-beat, and a real engine masks the leading octets. The pseudo-header — source and destination addresses, protocol, length — is absent entirely, and it is required: a TCP checksum covers it, so an engine that omits it computes the wrong sum for every frame.

Production implication: c_bad_l4 is the counter that must be read against Chapter 6.3's FCS errors, because the two together localise a corruption to a segment of the path. A frame with a good FCS and a bad L4 checksum was corrupted before the last hop that recomputed the FCS — a store-and-forward switch regenerates the FCS, so a bit flipped inside such a switch produces exactly this signature. A frame with a bad FCS is corrupted on the last link. Neither counter alone distinguishes them and both are free.

4. RTL 2 — The Transmit Checksum Inserter

The mirror, and it has a problem the receive side does not: it must write into a frame it is streaming.

// -----------------------------------------------------------------------
// tx_checksum_inserter -- computes and inserts an L4 checksum.
//
// The difficulty is placement. The checksum field sits near the START
// of the L4 header and its value depends on the WHOLE payload, so
// the frame cannot be streamed straight out -- the MAC must either
// buffer it or accept a partial-checksum contract with software.
// -----------------------------------------------------------------------
module tx_checksum_inserter
  import offload_pkg::*;
(
  input  logic                    clk,
  input  logic                    rst_n,

  input  logic                    cfg_partial_mode,     // software pre-seeds
  input  logic                    beat_valid,
  input  logic [BUS_BYTES*8-1:0]  beat_data,
  input  logic [$clog2(BUS_BYTES+1)-1:0] beat_bytes,
  input  logic                    beat_sof,
  input  logic                    beat_eof,

  input  parse_t                  parse,
  input  logic [7:0]              csum_offset,          // from the descriptor
  input  logic                    csum_enable,

  output logic                    out_valid,
  output logic [BUS_BYTES*8-1:0]  out_data,
  output logic                    out_patch_valid,      // patch this beat
  output logic [7:0]              out_patch_offset,
  output logic [15:0]             out_patch_value,

  output logic [31:0]             c_inserted,
  output logic [31:0]             c_declined,
  output logic [31:0]             c_offset_out_of_range,
  output logic                    insertion_impossible
);

  logic [31:0] acc;
  logic        active;

  logic [31:0] beat_sum;
  always_comb begin
    int i;
    beat_sum = '0;
    for (i = 0; i < BUS_BYTES/2; i++)
      if ((i*2) < int'(beat_bytes))
        beat_sum = beat_sum + 32'(beat_data[i*16 +: 16]);
  end

  wire [31:0] fold1 = (acc & 32'hFFFF) + (acc >> 16);
  wire [15:0] fold2 = fold1[15:0] + fold1[31:16];

  // The offset must lie inside the frame. A descriptor carrying a
  // checksum offset past the frame's end would have the MAC patch
  // memory beyond the buffer -- which is why this is a refusal and
  // not a clamp.
  assign c_offset_out_of_range = '0;      // counted below
  assign insertion_impossible  = csum_enable && !cfg_partial_mode &&
                                 !parse.parse_ok;

  // In PARTIAL mode software has already placed the pseudo-header sum
  // in the checksum field and told the MAC where it is. The MAC adds
  // the payload sum to it. This is the standard contract and it
  // exists precisely because it removes the parse dependency.
  assign out_patch_value  = ~fold2;
  assign out_patch_offset = csum_offset;

  assign out_valid = beat_valid;
  assign out_data  = beat_data;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      acc <= '0; active <= 1'b0; out_patch_valid <= 1'b0;
      c_inserted <= '0; c_declined <= '0;
    end else begin
      out_patch_valid <= 1'b0;

      if (beat_valid && beat_sof) begin
        acc    <= beat_sum;
        active <= 1'b1;
      end else if (beat_valid && active) begin
        acc <= acc + beat_sum;
      end

      if (beat_eof) begin
        active <= 1'b0;
        if (csum_enable && !insertion_impossible) begin
          out_patch_valid <= 1'b1;
          c_inserted      <= c_inserted + 1;
        end else if (csum_enable) begin
          c_declined <= c_declined + 1;
        end
      end
    end
  end

endmodule

Classification: a streaming accumulator emitting a deferred patch rather than modifying the stream.

What it teaches: that the transmit side has a placement problem the receive side does not, and it has the same shape as Chapter 16.3's. The checksum field is near the start of the L4 header and its value depends on the whole payload — so the field must be written after the data that determines it has passed. That chapter solved the identical problem for a timestamp; the solutions are the same three: buffer the frame, patch it in a later pass, or have software pre-seed the field.

And it teaches that the partial-checksum contract is the one that actually ships, and why. Software computes the pseudo-header sum — addresses, protocol, length — places it in the checksum field, and tells the MAC the field's offset in the descriptor. The MAC then adds the payload sum. This removes the parse dependency entirely: the MAC does not need to know what protocol it is carrying, only where to put sixteen bits. Every failure mode in Section 3's decline list disappears.

Deliberately simplified: the patch is emitted as an offset and a value with no mechanism for applying it — a real design either buffers the frame in Chapter 18.4 §7's FIFO and patches in place, which store-and-forward makes easy, or patches on the way out with a bypass mux. c_offset_out_of_range is declared and tied to zero, which is the listing flagging a check it does not implement. And the pseudo-header is again absent, which in partial mode is correct because software supplied it.

Production implication: the ratio c_declined / (c_declined + c_inserted) on the transmit side should be zero in partial mode and non-trivial in full-parse mode, and the difference is the argument for the contract. A port running full-parse insertion declines every tunnelled frame, every fragment and every frame with IPv4 options — and software must then compute those checksums itself, which means the software path must exist and be tested, for traffic that appears rarely. Partial mode has one path.

5. The Check That Moved Upstream

Chapter 18.3 §20's callout made this argument in outline and stopped. Two chapters have since added to the path it describes, so it is worth building out properly.

Trace every integrity check a received frame passes, and where each is computed.

Rows four to eight are unprotected, and row eight is the one that closes the gap.

Without offload, the TCP checksum is computed by software, on the data in the host buffer — so it covers rows four to seven. Any corruption in the FIFO, the reorder buffer, the DMA path or DRAM is caught, because the check runs downstream of all of them.

With offload, the checksum is computed in the MAC and its verdict is carried in the descriptor. Software reads a bit. The data it then processes has not been checked by anything since it left the MAC.

And the path that is now unprotected has grown across Module 18.

Row four is the newest and it is the sharpest. A reorder buffer exists to rearrange data — that is its function — and a bug in its sequence comparison delivers a frame's bursts in the wrong order, which is a corruption no check downstream will notice. Chapter 18.5 §19's properties are the only thing standing between that and a silently mangled frame.

This is not an argument against offload, and it is worth being explicit because the argument is easy to overstate. 102.81% of a core is not payable, and the alternative to offload at 100 Gb/s is not a slower machine — it is no machine. Offload is mandatory above about 25 Gb/s and the unprotected path is the price.

What follows from it is a verification obligation rather than a design change. Every stage in the second table must be verified to a standard appropriate to being the last line of defence, because there is no longer a checksum behind them. Chapter 18.3 §20's run E needed a payload scoreboard for exactly this reason, and with offload enabled that scoreboard is the only thing that would ever have caught it.

And there is one mitigation that costs almost nothing and is rarely built: have the MAC compute the checksum again, on the way out of the reorder buffer, and compare. The engine is a few hundred gates — Section 2 — and a second instance at the far end of the internal path restores an end-to-end check across everything Module 18 added. It does not cover DRAM, but it covers the four rows above, which is where this module's complexity lives.

6. RTL 3 — The Segmentation Offload Engine

One descriptor in, many frames out, and the interesting part is which header fields change between them.

// -----------------------------------------------------------------------
// tso_segmenter -- splits one large transmit request into MSS-sized
// segments, replicating and adjusting the headers.
//
// The CPU builds ONE header and hands over up to 64 KiB. The MAC
// emits up to 46 frames, each with the header copied and four fields
// changed. Section 7 prices what that actually saves.
// -----------------------------------------------------------------------
module tso_segmenter
  import offload_pkg::*;
(
  input  logic                    clk,
  input  logic                    rst_n,

  input  logic                    req_valid,
  output logic                    req_ready,
  input  logic [15:0]             req_total_bytes,
  input  logic [15:0]             req_mss,
  input  logic [7:0]              req_hdr_bytes,
  input  logic [31:0]             req_seq_base,        // TCP sequence

  // Per-segment control out.
  output logic                    seg_valid,
  input  logic                    seg_ready,
  output logic [15:0]             seg_payload_bytes,
  output logic [31:0]             seg_seq,
  output logic [15:0]             seg_ip_id,
  output logic                    seg_is_first,
  output logic                    seg_is_last,
  output logic                    seg_set_psh,
  output logic                    seg_set_fin,

  input  logic [15:0]             cfg_ip_id_base,
  input  logic                    cfg_ip_id_increment,

  output logic [31:0]             c_requests,
  output logic [31:0]             c_segments,
  output logic [15:0]             worst_segments,
  output logic                    mss_invalid,
  output logic                    total_too_large
);

  logic [15:0] remaining;
  logic [31:0] seq;
  logic [15:0] ip_id;
  logic [15:0] seg_count;
  logic        active;

  // An MSS of zero would loop for ever; an MSS above the MTU would
  // emit frames the link cannot carry. Both are refusals.
  assign mss_invalid    = req_valid && ((req_mss == 16'd0) ||
                                        (req_mss > 16'd9000));
  assign total_too_large = req_valid && (req_total_bytes == 16'd0);

  assign req_ready = !active && !mss_invalid && !total_too_large;

  wire [15:0] this_payload = (remaining > req_mss) ? req_mss : remaining;

  assign seg_valid         = active && (remaining != 16'd0);
  assign seg_payload_bytes = this_payload;
  assign seg_seq           = seq;
  assign seg_ip_id         = ip_id;
  assign seg_is_first      = (seg_count == 16'd0);
  assign seg_is_last       = (this_payload == remaining);

  // PSH and FIN belong only on the LAST segment. A segmenter that
  // copies them onto every segment tells the receiver the stream
  // ended 46 times, which some stacks tolerate and none should have
  // to.
  assign seg_set_psh = seg_is_last;
  assign seg_set_fin = seg_is_last;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      remaining <= '0; seq <= '0; ip_id <= '0;
      seg_count <= '0; active <= 1'b0;
      c_requests <= '0; c_segments <= '0; worst_segments <= '0;
    end else begin
      if (req_valid && req_ready) begin
        remaining  <= req_total_bytes;
        seq        <= req_seq_base;
        ip_id      <= cfg_ip_id_base;
        seg_count  <= '0;
        active     <= 1'b1;
        c_requests <= c_requests + 1;
      end

      if (seg_valid && seg_ready) begin
        remaining  <= remaining - this_payload;
        seq        <= seq + 32'(this_payload);
        seg_count  <= seg_count + 16'd1;
        c_segments <= c_segments + 1;
        if (cfg_ip_id_increment) ip_id <= ip_id + 16'd1;

        if (seg_is_last) begin
          active <= 1'b0;
          if ((seg_count + 16'd1) > worst_segments)
            worst_segments <= seg_count + 16'd1;
        end
      end
    end
  end

endmodule

Classification: a sequence generator emitting per-segment header deltas rather than headers.

What it teaches: that only four fields change between segments and getting any of them wrong is a protocol violation the far end sees. The TCP sequence number advances by the payload. The IP identification field increments — or does not, and both are defensible, which is why it is configurable. The length fields change. And PSH and FIN belong on the last segment only — a segmenter that copies the flags verbatim tells the receiver the stream ended on every segment.

And it teaches that the MSS is supplied by software and must be validated. An MSS of zero loops for ever; an MSS above the MTU emits frames the link drops. Neither is a case a design should discover at run time on a customer's machine, and both are two comparators.

Deliberately simplified: the header itself is never touched here — the block emits deltas and something downstream applies them, which is a real division of labour and is only half the work. The checksum interaction is absent: each segment needs its own L4 checksum, so Section 4's inserter runs once per segment and its pseudo-header must use that segment's length. total_too_large is tied to a zero-length check rather than to the 64 KiB limit the field's width already implies.

Production implication: worst_segments against the configured maximum is how an integrator sizes the transmit path's downstream capacity. Forty-six segments from one descriptor is forty-six frames the transmit FIFO and Chapter 18.4 §16's pipeline must absorb — and the doorbell rang once. A driver that batches TSO requests can present 8 descriptors that become 368 frames, which at 100 Gb/s is 45 µs of wire time from one notification, and is exactly the burst Chapter 18.6 §11's rate estimator is worst at tracking.

7. What Segmentation Offload Actually Buys

Section 6 built the mechanism. This section prices it, and the answer follows the pattern Chapter 18.3 §11 established for jumbo frames: the headline saving is not where people expect.

A 64 KiB TCP send at an MSS of 1448 is 46 segments.

Rows one and three are the wins and they are per frame. Row two and row four are per octet and barely move — because the payload still has to be fetched from memory and it is the same payload either way.

The transaction arithmetic, worked:

A 30.4% reduction in transactions, which is worth having and is not the reason anybody enables TSO.

The reason is row one, and the arithmetic is Chapter 18.1 §10's. Building a TCP and IP header is a few hundred cycles — a template copy, four field updates and a checksum seed. A 1448-octet segment occupies 1522 octets of wire including its headers, preamble and gap, so at 100 Gb/s that is 8.213 million headers per second — which at 300 cycles each is 2.46 Gcycles/s, 82.13% of a 3 GHz core, for header construction alone.

Add Section 2's checksum at 102.81% and the two together are 185% of a core, which is the actual case for offload — and it is a per-frame cost and a per-octet cost stacked, which is why both mechanisms are needed and neither suffices.

And there is a cost worth naming because it is structural rather than a bug. TSO makes one descriptor become up to 46 frames, so the transmit path's frame rate is no longer bounded by the descriptor rate. Every downstream structure — Chapter 18.4 §7's FIFO, §16's pipeline, Section 4's checksum inserter — must absorb a burst the driver did not appear to request.

Which interacts with Chapter 18.6 §11's rate estimator in the least convenient way. A TSO burst is a step change in frame rate the estimator takes a millisecond to track — so the first millisecond of a large send runs with a threshold sized for the previous, quieter traffic, and the interrupt rate spikes exactly when the CPU has just handed over 64 KiB and would like to do something else.

8. RTL 4 — The RSS Hash

Chapter 15.2's flow hash, over five fields instead of three, and with one property that chapter did not need.

// -----------------------------------------------------------------------
// rss_hash -- a Toeplitz hash over the 5-tuple.
//
// 15.2 section 5 built a flow hash for a LAG and this is the same
// problem with two differences: the tuple is five fields rather than
// three, and the hash must be SYMMETRIC if the two directions of a
// connection are to land on the same queue.
// -----------------------------------------------------------------------
module rss_hash
  import offload_pkg::*;
#(
  parameter int KEY_BITS = 320
)(
  input  logic                    clk,
  input  logic                    rst_n,

  input  logic [KEY_BITS-1:0]     cfg_key,
  input  logic                    cfg_symmetric,
  input  logic [4:0]              cfg_fields,        // which of the 5 to use

  input  logic                    tuple_valid,
  input  logic [31:0]             src_ip,
  input  logic [31:0]             dst_ip,
  input  logic [15:0]             src_port,
  input  logic [15:0]             dst_port,
  input  logic [7:0]              protocol,

  output logic [31:0]             hash,
  output logic                    hash_valid,

  output logic [31:0]             c_hashed,
  output logic [31:0]             c_symmetric_swaps
);

  // Symmetry: order the endpoints canonically before hashing, so
  // that (A,B) and (B,A) produce the same value. Without this, a
  // connection's two directions land on different queues -- which
  // splits its state across two cores and destroys the cache
  // locality multi-queue existed to provide.
  wire swap = cfg_symmetric &&
              ({src_ip, src_port} > {dst_ip, dst_port});

  wire [31:0] a_ip   = swap ? dst_ip   : src_ip;
  wire [31:0] b_ip   = swap ? src_ip   : dst_ip;
  wire [15:0] a_port = swap ? dst_port : src_port;
  wire [15:0] b_port = swap ? src_port : dst_port;

  // The input vector, masked by which fields are enabled. Disabling
  // the ports makes all of a host pair's connections land together,
  // which is sometimes wanted and is usually a mistake.
  wire [103:0] input_vec = {
    cfg_fields[0] ? a_ip   : 32'd0,
    cfg_fields[1] ? b_ip   : 32'd0,
    cfg_fields[2] ? a_port : 16'd0,
    cfg_fields[3] ? b_port : 16'd0,
    cfg_fields[4] ? protocol : 8'd0
  };

  // Toeplitz: for each set input bit, XOR in the key window at that
  // position. 104 conditional 32-bit XORs -- a wide but shallow tree.
  logic [31:0] t;
  always_comb begin
    int i;
    t = '0;
    for (i = 0; i < 104; i++)
      if (input_vec[103-i])
        t = t ^ cfg_key[KEY_BITS-1-i -: 32];
  end

  assign hash = t;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      hash_valid <= 1'b0; c_hashed <= '0; c_symmetric_swaps <= '0;
    end else begin
      hash_valid <= tuple_valid;
      if (tuple_valid) begin
        c_hashed <= c_hashed + 1;
        if (swap) c_symmetric_swaps <= c_symmetric_swaps + 1;
      end
    end
  end

endmodule

Classification: a Toeplitz hash with canonical endpoint ordering.

What it teaches: that symmetry is a requirement RSS has and Chapter 15.2's LAG hash did not. A LAG distributes frames travelling in one direction across members; nothing requires the reverse direction to take the same member. RSS assigns a connection to a core, and a connection has two directions — so an asymmetric hash puts a connection's receive processing on one core and its transmit processing on another, splitting its socket state, its timers and its congestion window across two caches. The fix is to order the endpoints canonically before hashing, and it costs one comparison.

And it teaches that cfg_fields is a trap dressed as flexibility. Disabling the port fields makes every connection between a host pair hash identically — so a backup job between two machines lands entirely on one queue, which is Chapter 15.2 §10's elephant flow reappearing with a configuration switch instead of a hash collision. It is occasionally what an operator wants and is usually set by accident.

Deliberately simplified: the Toeplitz loop is 104 conditional 32-bit XORs in one combinational block, which is a wide tree a production design pipelines across two or three stages. IPv6 is absent — the tuple would be 296 bits rather than 104 — and the input vector's concatenation is written with conditional widths that will not elaborate as shown. The key is a configuration input and must be identical across every device that needs consistent steering, which is a system requirement this block cannot enforce.

Production implication: c_symmetric_swaps should be close to half of c_hashed on bidirectional traffic, and its being near zero is a fault. A swap count of zero with symmetry enabled means every tuple already arrives in canonical order, which on real traffic does not happen — so either the symmetry logic is not working or the comparison is wrong. It is the only direct evidence the symmetric mode is doing anything, and the symptom of its absence is a performance problem nobody attributes to a hash.

9. RTL 5 — The Indirection Table

The layer between the hash and the queue, and it exists for the same reason Chapter 15.2 §13's bucket map did.

// -----------------------------------------------------------------------
// rss_indirection_table -- hash -> table entry -> queue.
//
// 15.2 section 12 derived why a modulo is wrong: changing the member
// count moves 74.9% of flows when 25% is the minimum. The same
// argument applies to queues, and the same fix -- a table of buckets
// -- applies unchanged.
// -----------------------------------------------------------------------
module rss_indirection_table
  import offload_pkg::*;
#(
  parameter int ENTRIES = RSS_TBL
)(
  input  logic                    clk,
  input  logic                    rst_n,

  input  logic                    cfg_we,
  input  logic [$clog2(ENTRIES)-1:0] cfg_idx,
  input  logic [$clog2(NUM_QUEUE)-1:0] cfg_queue,

  input  logic [$clog2(NUM_QUEUE)-1:0] cfg_num_active,
  input  logic                    cfg_enable,

  input  logic                    hash_valid,
  input  logic [31:0]             hash,

  output logic                    queue_valid,
  output logic [$clog2(NUM_QUEUE)-1:0] queue,
  output logic [$clog2(ENTRIES)-1:0]   entry,

  output logic [31:0]             c_steered [NUM_QUEUE],
  output logic [31:0]             c_default,
  output logic [15:0]             entries_per_queue [NUM_QUEUE],
  output logic                    table_unbalanced,
  output logic                    queue_unreachable
);

  logic [$clog2(NUM_QUEUE)-1:0] tbl [ENTRIES];

  // The table is indexed by the hash's LOW bits. Low rather than
  // high because a Toeplitz hash's avalanche is uniform and the low
  // bits are as good as any -- but the choice must be documented,
  // because a device that indexes differently steers differently
  // and two devices in a path must agree if steering is to be
  // consistent end to end.
  wire [$clog2(ENTRIES)-1:0] idx = hash[$clog2(ENTRIES)-1:0];

  assign entry       = idx;
  assign queue       = cfg_enable ? tbl[idx] : '0;
  assign queue_valid = hash_valid;

  // How many table entries point at each queue. A queue with zero
  // entries receives nothing, which is legal when it is being drained
  // and is a fault when it is not.
  always_comb begin
    int i, q;
    for (q = 0; q < NUM_QUEUE; q++) entries_per_queue[q] = '0;
    for (i = 0; i < ENTRIES; i++)
      entries_per_queue[tbl[i]] = entries_per_queue[tbl[i]] + 16'd1;
  end

  // A queue inside the active set with no table entries can never be
  // reached, however good the hash is.
  logic unreachable;
  always_comb begin
    int q;
    unreachable = 1'b0;
    for (q = 0; q < NUM_QUEUE; q++)
      if ((q < int'(cfg_num_active)) && (entries_per_queue[q] == 16'd0))
        unreachable = 1'b1;
  end
  assign queue_unreachable = cfg_enable && unreachable;

  // The table itself is unbalanced if any active queue holds more
  // than twice the mean share. This is a CONFIGURATION imbalance and
  // is entirely separate from section 10's hash imbalance -- the two
  // produce the same symptom and need different fixes.
  logic [15:0] fair_share;
  assign fair_share = (cfg_num_active == '0) ? 16'd0
                                             : 16'(ENTRIES) / 16'(cfg_num_active);
  logic unbal;
  always_comb begin
    int q;
    unbal = 1'b0;
    for (q = 0; q < NUM_QUEUE; q++)
      if ((q < int'(cfg_num_active)) &&
          (entries_per_queue[q] > (fair_share << 1)))
        unbal = 1'b1;
  end
  assign table_unbalanced = cfg_enable && unbal;

  always_ff @(posedge clk or negedge rst_n) begin
    int i;
    if (!rst_n) begin
      // Reset to round-robin over the whole queue set, which is the
      // only sane default: every queue reachable, shares equal.
      for (i = 0; i < ENTRIES; i++)
        tbl[i] <= ($clog2(NUM_QUEUE))'(i % NUM_QUEUE);
      for (i = 0; i < NUM_QUEUE; i++) c_steered[i] <= '0;
      c_default <= '0;
    end else begin
      if (cfg_we) tbl[cfg_idx] <= cfg_queue;
      if (hash_valid) begin
        if (cfg_enable) c_steered[queue] <= c_steered[queue] + 1;
        else            c_default        <= c_default + 1;
      end
    end
  end

endmodule

Classification: a rewritable bucket map with reachability and balance checks.

What it teaches: that the indirection table is the same structure Chapter 15.2 §13 built and for the same reason. That chapter derived what a direct modulo costs when the member count changes: 74.9% of flows move where 25% is the minimum, and a table of buckets achieves the minimum. Queues change count for the same reasons members do — a core goes offline, a VM migrates, an operator retunes — and a modulo remaps three quarters of the connections, which on TCP means three quarters of the flows lose their cache locality at once.

And it teaches that queue_unreachable catches a configuration error whose symptom is a silent capacity loss. A queue with no table entries receives nothing. Its core is idle, its interrupt never fires, and the port's other queues carry the whole load — so the port runs at (active − 1) / active of its designed capacity with no error anywhere, and an operator counting configured queues sees the right number.

Deliberately simplified: entries_per_queue is recomputed combinationally over 128 entries every cycle, which is a 128-way histogram in one cycle — a production design maintains the counts incrementally on writes. The reset loop writes 128 entries in one cycle. The table is single-banked, so rewriting it while traffic flows produces exactly Chapter 17.2 §3's partial-update problem — some flows steered by the old map and some by the new, which for RSS is tolerable and for a schedule was not.

Production implication: c_steered[q] across the queue set is the measured distribution, and it must be compared against entries_per_queue[q] rather than against uniformity. A queue holding 25% of the table and receiving 25% of the frames is working correctly even though the port has four queues and this is not 25% — the table is the intent. Comparing frames against entries separates a hash problem from a table problem, which Section 10 shows are the two independent causes of the same symptom.

10. Queue Distribution, Reusing Chapter 15.2

Chapter 15.2 §8 derived what a perfect hash does to a finite number of flows. Nothing about that derivation is specific to link aggregation, so this section quotes it rather than repeating it.

The question is identical: F flows into Q bins, one bin per flow, no splitting. A LAG cannot split a flow because Chapter 15.1's ordering constraint forbids it. RSS cannot split a flow because the whole point is to keep a connection's state on one core.

So Chapter 15.2 §8's table applies unchanged, with "member" read as "queue":

The 4-queue and 16-queue columns at 64 flows — 79.2% and 50.8% — are Chapter 15.2 §8's published figures, and the rest are the same computation at other points.

Read the table two ways and both are uncomfortable.

Across a row: more queues distribute worse. Sixteen queues on 64 flows is 50.8% usable against four queues' 79.2% — so quadrupling the queues buys 16 × 0.508 = 8.13 queue-loads of usable capacity against 4 × 0.792 = 3.17, which is 2.57× for 4× the queues. That is Chapter 15.2 §8's exact conclusion, arrived at for cores instead of links.

Down a column: the fix is more flows, and flows are not a design parameter. A port carrying 16 concurrent connections cannot be made to distribute well across 16 queues by any hash, because there are not enough flows to spread. And 16 connections is an entirely ordinary number for a server doing a few large transfers, which is exactly the workload that most needs the cores.

Which gives the sizing rule, and it is the one Chapter 15.2 §9 reached:

Choose the queue count against the flow count the port will carry, not against the core count the machine has.

And a machine with 16 cores and 16 flows should not configure 16 queues, which is what every default does.

One difference from the LAG case is worth stating because it changes the remedy. A LAG's imbalance wastes link capacity that has been paid for — the links exist and are idle. A queue set's imbalance wastes core time, which the machine can spend on something else. So the cost of over-configuring queues is lower than the cost of over-configuring LAG members, and the failure is a hot core rather than a throttled flow.

But the hot core is a real limit. At 64 flows and 16 queues, the busiest queue carries 7.88 flows against a mean of 4 — so if the port is at line rate, that core is doing 1.97× the average work, and it is the one that saturates. The port's throughput is then set by one core, which is the situation multi-queue was adopted to escape.

11. RTL 6 — The Queue Set

Several of everything the previous six chapters built, and the block's job is to say what is replicated and what is shared.

// -----------------------------------------------------------------------
// queue_set -- per-queue context, and the resources that are NOT
// per queue.
//
// The distinction is the chapter's second finding. Replicating a
// ring is cheap; replicating a bus master is not, and replicating a
// coalescer multiplies the aggregate interrupt rate by the queue
// count -- section 12.
// -----------------------------------------------------------------------
module queue_set
  import offload_pkg::*;
#(
  parameter int Q = NUM_QUEUE
)(
  input  logic                    clk,
  input  logic                    rst_n,

  input  logic [$clog2(Q)-1:0]    cfg_num_active,
  input  logic [Q-1:0]            cfg_queue_enable,

  // A frame arrives with a steering decision.
  input  logic                    frame_valid,
  input  logic [$clog2(Q)-1:0]    frame_queue,
  input  logic [15:0]             frame_bytes,

  // Per-queue state -- REPLICATED.
  output logic [15:0]             q_occupancy   [Q],
  output logic [31:0]             q_frames      [Q],
  output logic [31:0]             q_octets      [Q],
  output logic [31:0]             q_drops       [Q],
  output logic [31:0]             q_interrupts  [Q],

  // Shared -- NOT replicated.
  output logic [31:0]             c_total_frames,
  output logic [31:0]             c_total_drops,
  output logic [31:0]             c_total_interrupts,

  input  logic [Q-1:0]            q_irq_fired,
  input  logic [Q-1:0]            q_full,

  output logic [$clog2(Q)-1:0]    hottest_queue,
  output logic [15:0]             imbalance_x10,      // max / mean
  output logic                    steered_to_disabled,
  output logic                    all_load_on_one
);

  // The hottest queue and the imbalance ratio. This is 15.2 section 6's
  // fairness measurement, per queue instead of per member -- and it
  // needs the same null hypothesis: a ratio of 1.97 is expected at
  // 64 flows on 16 queues and is a defect at 64 000 flows.
  logic [31:0] max_frames, sum_frames;
  always_comb begin
    int q;
    max_frames = '0; sum_frames = '0; hottest_queue = '0;
    for (q = 0; q < Q; q++) begin
      sum_frames = sum_frames + q_frames[q];
      if (q_frames[q] > max_frames) begin
        max_frames    = q_frames[q];
        hottest_queue = q[$clog2(Q)-1:0];
      end
    end
  end

  wire [31:0] mean_frames = (cfg_num_active == '0) ? 32'd0
                            : (sum_frames / 32'(cfg_num_active));
  assign imbalance_x10 = (mean_frames == '0) ? 16'd0
                         : 16'((max_frames * 32'd10) / mean_frames);

  // A frame steered to a disabled queue has nowhere to go. It is a
  // table configuration error -- section 9's queue_unreachable in
  // the other direction.
  assign steered_to_disabled = frame_valid && !cfg_queue_enable[frame_queue];

  // Every frame on one queue: either one flow, or a hash that is not
  // hashing. 15.2 section 21's distinction, and it needs the flow
  // count to resolve.
  assign all_load_on_one = (sum_frames > 32'd10000) &&
                           (max_frames > ((sum_frames >> 1) + (sum_frames >> 2)));

  always_ff @(posedge clk or negedge rst_n) begin
    int q;
    if (!rst_n) begin
      for (q = 0; q < Q; q++) begin
        q_occupancy[q] <= '0; q_frames[q] <= '0; q_octets[q] <= '0;
        q_drops[q] <= '0; q_interrupts[q] <= '0;
      end
      c_total_frames <= '0; c_total_drops <= '0; c_total_interrupts <= '0;
    end else begin
      if (frame_valid) begin
        if (cfg_queue_enable[frame_queue] && !q_full[frame_queue]) begin
          q_frames[frame_queue] <= q_frames[frame_queue] + 1;
          q_octets[frame_queue] <= q_octets[frame_queue] +
                                   {16'b0, frame_bytes};
          c_total_frames        <= c_total_frames + 1;
        end else begin
          q_drops[frame_queue] <= q_drops[frame_queue] + 1;
          c_total_drops        <= c_total_drops + 1;
        end
      end

      for (q = 0; q < Q; q++)
        if (q_irq_fired[q]) begin
          q_interrupts[q]    <= q_interrupts[q] + 1;
          c_total_interrupts <= c_total_interrupts + 1;
        end
    end
  end

endmodule

Classification: a per-queue accounting array with an explicit aggregate, and an imbalance measure that needs an external null hypothesis.

What it teaches: that c_total_interrupts must exist alongside the per-queue counts, and it is the counter multi-queue designs most often omit. Each queue's interrupt rate is what its own coalescer reports; the port's rate is their sum, and Section 12 shows the sum grows with the queue count under Chapter 18.6's policy. A design reporting only per-queue rates makes the aggregate invisible, and the aggregate is what the machine pays.

And it teaches that imbalance_x10 is Chapter 15.2 §6's fairness measurement with the same limitation. A ratio of 1.97 is exactly what a perfect hash produces at 64 flows on 16 queues — Section 10's table — and the same 1.97 at 64 000 flows is a serious defect. The ratio alone cannot distinguish them; it needs the flow count, which the MAC does not have. A design that alarms on the ratio will alarm on correct behaviour, which is that chapter's conclusion restated.

Deliberately simplified: the maximum and sum are computed combinationally across sixteen queues every cycle, and the interrupt loop updates sixteen counters in parallel. q_occupancy is declared and never driven. The per-queue rings, DMA contexts and coalescers are entirely absent — this block accounts for them and does not instantiate them, which is the honest division but leaves the chapter's largest replication cost stated rather than shown.

Production implication: steered_to_disabled is a configuration fault with a data-loss consequence and it is easy to create. A driver that disables a queue — a core going offline — must rewrite Section 9's indirection table first, or frames continue to be steered at a queue that no longer has a consumer. The window between disabling the queue and rewriting the table is a window in which frames are dropped, and the correct order is: rewrite the table, wait for the queue to drain, then disable it — which is the same shape as Chapter 18.1 §3's ring-base rule and fails the same way.

12. What Multi-Queue Does to the Address Channel

Chapter 18.3 §17 left a 100 Gb/s port at 0.744 transactions per cycle with 26% of the address channel spare. This section asks what several queues do to that, and the two halves of the answer point in opposite directions.

The bus traffic does not change. A frame steered to queue 3 needs the same descriptor fetch, the same data bursts and the same writeback as it would have needed on queue 0. The port's total transaction rate is 0.744 per cycle whatever the queue count is.

So multi-queue helps on the bus, if and only if the queues have separate master ports. With one shared port the total is unchanged and the queue count is irrelevant; with eight ports the per-port rate falls to 0.093, which is comfortable, and Chapter 18.5 §10's ID budget divides eight ways too.

And the interrupts go the other way.

Chapter 18.6 §4's policy holds each queue's interrupt rate at 1/L regardless of that queue's arrival rate. Splitting a port's traffic across q queues gives q independent coalescers each converging to 1/L:

The aggregate interrupt rate is q / L — linear in the queue count.

Which is the chapter's second finding and it is not what multi-queue is usually understood to do. It does not divide the interrupt work; it divides the per-core work and multiplies the total by the queue count. That is a good trade when the cores are otherwise idle and a poor one when they are not.

And the behaviour changes completely when Chapter 18.6's upper clamp binds, which is worth separating because the two regimes give opposite answers:

In the clamped regime the aggregate does not depend on the queue count at all, because each queue's rate is proportional to its share of the traffic. So a port running with N_MAX = 256 at 100 Gb/s divides its 116% across eight cores at 14.53% each and pays 116% in total — spreading is free. With N_MAX raised to 2 976 as Chapter 18.6 §4 recommends, it pays 80%.

Which produces an odd and genuinely useful conclusion: raising N_MAX is unambiguously right on a single-queue port and becomes a trade on a multi-queue one. The single-queue port goes from 116% to 10%. The eight-queue port goes from 116% aggregate to 80% aggregate — still better, but the per-core figure was already fine at 14.53%, so the benefit is smaller than the single-queue case suggests and the L should be re-examined.

13. RTL 7 — Offload Telemetry

The counters, and the selection rule this module has used throughout: each one separates a pair of conditions with the same symptom.

// -----------------------------------------------------------------------
// offload_telemetry -- what offload and steering are actually doing.
// -----------------------------------------------------------------------
module offload_telemetry
  import offload_pkg::*;
#(
  parameter int Q = NUM_QUEUE
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        csum_computed,
  input  logic        csum_declined,
  input  logic        csum_bad,
  input  logic        fcs_bad,
  input  logic [2:0]  decline_reason,

  input  logic        tso_request,
  input  logic [15:0] tso_segments,

  input  logic        steer_valid,
  input  logic [$clog2(Q)-1:0] steer_queue,
  input  logic        hash_valid,
  input  logic        hash_swapped,

  output logic [31:0] c_csum_computed,
  output logic [31:0] c_csum_declined,
  output logic [31:0] c_csum_bad,
  output logic [31:0] c_bad_csum_good_fcs,     // the interesting pair
  output logic [31:0] c_bad_fcs,
  output logic [31:0] c_decline [8],
  output logic [31:0] c_tso_requests,
  output logic [31:0] c_tso_segments,
  output logic [15:0] mean_segments_x10,
  output logic [31:0] c_hash,
  output logic [31:0] c_hash_swapped,
  output logic [15:0] offload_coverage_pct,
  output logic [15:0] symmetry_pct
);

  always_comb begin
    logic [31:0] tot;
    tot = c_csum_computed + c_csum_declined;
    offload_coverage_pct = (tot == '0) ? 16'd0
                           : 16'((c_csum_computed * 32'd100) / tot);
    symmetry_pct = (c_hash == '0) ? 16'd0
                   : 16'((c_hash_swapped * 32'd100) / c_hash);
    mean_segments_x10 = (c_tso_requests == '0) ? 16'd0
                        : 16'((c_tso_segments * 32'd10) / c_tso_requests);
  end

  always_ff @(posedge clk or negedge rst_n) begin
    int i;
    if (!rst_n) begin
      c_csum_computed <= '0; c_csum_declined <= '0; c_csum_bad <= '0;
      c_bad_csum_good_fcs <= '0; c_bad_fcs <= '0;
      for (i = 0; i < 8; i++) c_decline[i] <= '0;
      c_tso_requests <= '0; c_tso_segments <= '0;
      c_hash <= '0; c_hash_swapped <= '0;
    end else begin
      if (csum_computed) c_csum_computed <= c_csum_computed + 1;
      if (csum_declined) begin
        c_csum_declined <= c_csum_declined + 1;
        c_decline[decline_reason] <= c_decline[decline_reason] + 1;
      end
      if (csum_bad) c_csum_bad <= c_csum_bad + 1;
      if (fcs_bad)  c_bad_fcs  <= c_bad_fcs + 1;

      // The pair that localises a corruption to a SEGMENT of the
      // path: a bad L4 checksum with a GOOD FCS means the frame was
      // corrupted upstream of the last device that regenerated the
      // FCS -- inside a store-and-forward switch, not on the wire.
      if (csum_bad && !fcs_bad)
        c_bad_csum_good_fcs <= c_bad_csum_good_fcs + 1;

      if (tso_request) begin
        c_tso_requests <= c_tso_requests + 1;
        c_tso_segments <= c_tso_segments + {16'b0, tso_segments};
      end

      if (hash_valid) begin
        c_hash <= c_hash + 1;
        if (hash_swapped) c_hash_swapped <= c_hash_swapped + 1;
      end
    end
  end

endmodule

Classification: a per-condition offload accountant with one deliberately composite counter.

What it teaches: that c_bad_csum_good_fcs is the most informative counter in the block and it exists only because the two checks are computed at different points in the path. Chapter 6.3's FCS covers the last link; the L4 checksum covers end to end. So the combination localises:

Row two is the signature of a bit flipped inside a store-and-forward switch — Chapter 12.6 — which recomputes the FCS over whatever it holds and therefore certifies its own corruption. Nothing else produces that pair, and neither counter alone says anything.

And it teaches that offload_coverage_pct is the number that says whether offload is worth what it cost. A port declining 40% of frames is doing 40% of the checksums in software anyway — Section 2's cost, at 40% — and has all of Section 5's unprotected path for the other 60%. That is the worst of both, and it is what a full-parse engine on tunnelled traffic produces.

Deliberately simplified: three combinational divides. c_decline is indexed by a 3-bit reason code that Section 3 does not actually emit — that module has four separate decline conditions and no encoder. And mean_segments_x10 accumulates TSO segments without bounding the accumulator, which wraps.

Production implication: symmetry_pct should sit near 50 on bidirectional traffic and its being near zero or near a hundred is a fault. Fifty means the canonical ordering is swapping about half the tuples, which is what a mixture of connection directions produces. Near zero means the comparison is not firing; near a hundred means it is firing always, which happens if the comparison's operand order is inverted. Either way the symmetric mode is not working, and the symptom — a connection's two directions on different cores — is a cache-locality problem nobody attributes to a hash.

14. RTL 8 — The Offload Conformance Monitor

The last block of Module 18, and its verdicts divide into things that lose data and things that waste capacity.

// -----------------------------------------------------------------------
// offload_conformance_monitor -- offload and steering verdicts.
// -----------------------------------------------------------------------
module offload_conformance_monitor
  import offload_pkg::*;
#(
  parameter int Q = NUM_QUEUE
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [31:0] c_csum_computed,
  input  logic [31:0] c_csum_declined,
  input  logic [31:0] c_bad_csum_good_fcs,
  input  logic [15:0] offload_coverage_pct,
  input  logic [15:0] symmetry_pct,
  input  logic [15:0] mean_segments_x10,
  input  logic [31:0] c_steered [Q],
  input  logic [15:0] entries_per_queue [Q],
  input  logic [15:0] imbalance_x10,
  input  logic [31:0] c_total_drops,
  input  logic [31:0] c_total_frames,
  input  logic [$clog2(Q)-1:0] cfg_num_active,

  input  logic        queue_unreachable,
  input  logic        table_unbalanced,
  input  logic        steered_to_disabled,
  input  logic        insertion_impossible,
  input  logic        mss_invalid,

  output logic        offload_ok,
  output logic        cfg_fault,
  output logic        coverage_poor,
  output logic        upstream_corruption,
  output logic        symmetry_broken,
  output logic        steering_imbalanced,
  output logic        queue_starved,
  output logic        none_of_the_above
);

  assign cfg_fault = queue_unreachable | steered_to_disabled |
                     insertion_impossible | mss_invalid;

  // Below 80% coverage, software is doing enough checksums that the
  // offload is not paying for the unprotected path it created.
  assign coverage_poor = ((c_csum_computed + c_csum_declined) > 32'd10000) &&
                         (offload_coverage_pct < 16'd80);

  // Section 13's pair: corruption upstream of the last FCS
  // regeneration. Not our link, and not nothing.
  assign upstream_corruption = (c_bad_csum_good_fcs > 32'd0);

  // Symmetry should sit near 50% on bidirectional traffic.
  assign symmetry_broken = (c_csum_computed > 32'd10000) &&
                           ((symmetry_pct < 16'd10) || (symmetry_pct > 16'd90));

  // Imbalance ABOVE what a perfect hash would produce. The threshold
  // is a configuration input in a real design because it depends on
  // the flow count, which the MAC does not have -- section 11.
  assign steering_imbalanced = (c_total_frames > 32'd100000) &&
                               (imbalance_x10 > 16'd40);       // 4x the mean

  // A queue with table entries and no frames: the hash never
  // produces its bucket, or the table is wrong.
  logic starved;
  always_comb begin
    int q;
    starved = 1'b0;
    for (q = 0; q < Q; q++)
      if ((q < int'(cfg_num_active)) && (entries_per_queue[q] != 16'd0) &&
          (c_steered[q] == 32'd0) && (c_total_frames > 32'd100000))
        starved = 1'b1;
  end
  assign queue_starved = starved;

  assign offload_ok = !cfg_fault && !upstream_corruption;

  assign none_of_the_above = offload_ok && !coverage_poor &&
                             !symmetry_broken && !steering_imbalanced &&
                             !queue_starved && !table_unbalanced;

  // ---- properties -------------------------------------------------

  p_cfg_fault_excludes_ok:
    assert property (@(posedge clk) disable iff (!rst_n)
      cfg_fault |-> !offload_ok)
    else $error("offload_ok asserted with a configuration fault");

  p_never_steer_to_disabled:
    assert property (@(posedge clk) disable iff (!rst_n)
      !steered_to_disabled)
    else $error("a frame was steered to a disabled queue");

  p_no_unreachable_queue:
    assert property (@(posedge clk) disable iff (!rst_n)
      !queue_unreachable)
    else $error("an active queue has no indirection-table entries");

  p_steered_sums_to_total:
    assert property (@(posedge clk) disable iff (!rst_n)
      (c_total_frames > 32'd0) |-> (c_steered[0] <= c_total_frames))
    else $error("a queue received more frames than the port did");

  p_coverage_bounded:
    assert property (@(posedge clk) disable iff (!rst_n)
      offload_coverage_pct <= 16'd100)
    else $error("offload coverage exceeded 100%");

endmodule

Classification: a verdict generator whose imbalance threshold is explicitly a configuration input rather than a constant.

What it teaches: that steering_imbalanced cannot have a fixed threshold, and Section 10 is why. An imbalance ratio of 1.97 is exactly correct at 64 flows on 16 queues and a serious defect at 64 000. The MAC does not know the flow count, so the threshold belongs to software — and a monitor that hard-codes one alarms on correct behaviour on a lightly loaded port, which is the fastest way to have every alarm ignored.

And it teaches that upstream_corruption is the only verdict in Module 18 that is about somebody else's equipment. Every other finding in these seven chapters is about the MAC, its driver, its interconnect or its memory system. This one says a device several hops away is corrupting frames and regenerating a valid FCS over the corruption — which is a fault in a switch, reported by an end station, because the end station is the only place both checks exist.

Deliberately simplified: the starvation scan is combinational over sixteen queues. p_steered_sums_to_total checks only queue zero, because a sixteen-way sum in a property is expensive; a real assertion sums them. And the thresholds are literals throughout where several of them — the imbalance ratio above all — must be registers.

Production implication: none_of_the_above for the sixth and last time in Module 18. Six monitors now cover the MAC's interfaces, its ring protocol, both DMA directions, its bus shaping, its interrupt policy and its offload. A port on which all six assert has no known fault in any contract this module defines — which is roughly thirty hypotheses removed, and is the evidence that finally sends an investigation to the application, the network, or the offered load.

15. Flow Steering, and Why It Is Not RSS

RSS spreads traffic. Flow steering places it, and the two are frequently conflated because they both end with a queue number.

Steering exists because Section 10's distribution is a statistical result and some flows need a guarantee. A latency-sensitive connection pinned to a core with nothing else on it gets that core's cache and its interrupt budget; the same connection hashed across sixteen queues gets whichever it landed on, alongside whatever else landed there.

And the mechanism is a small exact-match table checked before the hash:

Which introduces a failure mode RSS does not have: a table that runs out.

An RSS table's size does not depend on the flow count — 128 entries serve any number of flows, because the hash folds them. A steering table holds one entry per steered flow, so it is a finite resource that software allocates, and software that steers more flows than the table holds gets silent fallbacks to RSS.

The silence is the problem, and it is worth a counter: a flow that software believes is pinned and that is actually being hashed behaves correctly and slowly, which is the hardest kind of performance bug to attribute.

And there is a second failure that is more interesting because it is a correctness issue rather than a performance one.

Steering rules and RSS can disagree about a connection's two directions. A rule pinning (A:1234 → B:80) to queue 3 says nothing about (B:80 → A:1234) — so the reverse direction is hashed, and lands wherever Section 8's symmetric hash puts it, which is a different queue unless somebody installed the mirror rule.

Which is exactly the split-state problem Section 8's symmetry was introduced to prevent, reintroduced by a mechanism that overrides it. A steering rule must be installed as a pair or the symmetry is gone, and nothing in the hardware enforces that.

The general shape is worth naming because it recurs whenever an override sits in front of a general policy: the override inherits none of the policy's invariants. Section 8's hash is symmetric by construction. A steering rule is symmetric only if somebody wrote both halves, and the hardware cannot tell a deliberate asymmetry from a missing rule.

16. What Offload Assumes About the Traffic

Chapter 18.1 §17 listed what the MAC assumes about its system. Offload adds a list of a different kind: assumptions about what is inside the frames.

Rows six and seven are the undetected pair and they are the dangerous ones.

Row six: Chapter 13.2's tag adds four octets before the EtherType, and a double tag adds eight. A parser expecting one tag on a double-tagged frame reads the inner tag's TPID as an EtherType, concludes the frame is not IP, and declines — which is safe. A parser expecting none on a single-tagged frame reads four octets into the tag as an EtherType, may match IPv4 by coincidence, and then computes a checksum over the wrong span. The decline path is not taken because the parse appeared to succeed.

Row seven is the partial-checksum contract's exposure. Section 4's inserter patches sixteen bits at an offset software supplied in the descriptor. A wrong offset patches sixteen bits of payload, and the frame goes out with corrupted data and a checksum computed over it — so the far end accepts it. This is the transmit-side equivalent of Chapter 18.4 §4's stale buffer pointer: a plausible value, acted on irreversibly, with no way to validate it.

The mitigation for row seven is one comparison and it is frequently absent:

The checksum offset must lie within the frame, and within the first N octets where a legitimate L4 header can be. A value beyond the frame is a refusal; a value beyond, say, 128 octets is at least suspicious.

And the general observation is the one that distinguishes offload from everything else in Module 18. Every earlier chapter's assumptions were about the system — memory latency, bus ordering, driver behaviour. These are about the traffic, which no party controls: the frames arrive from the network, built by devices with different stacks, different tunnelling and different tagging.

Which is why every offload engine must have a decline path and software must have a fallback, and why offload_coverage_pct from Section 13 is a deployment-dependent number rather than a design one.

17. Restoring the End-to-End Check

Section 5 identified an unprotected path and deferred its mitigation. This section builds it, because it is cheap and almost nobody does it.

The gap: with offload, no check covers the data between the MAC's checksum engine and the host buffer. Four structures sit in that gap — an asynchronous FIFO, a scatter-gather write engine, a release barrier and a reorder buffer — and each can rearrange or drop octets.

The mitigation is one more instance of Section 3's engine, at the far end of the internal path.

And the comparison's result is a bit in the descriptor: the checksum was the same at both ends of the internal path.

What it does and does not cover is worth being exact about:

Rows two to four are where Module 18's complexity lives, and they are covered. Rows six and seven need ECC and a software check respectively, and are outside what a MAC can do.

The cost is genuinely small. Section 2's engine is a ones-complement adder tree — eight 64-bit adders and a fold on a 512-bit datapath — and the second instance runs on data the DMA is already streaming. Against Chapter 18.5 §18's 5 500 flops for burst shaping, or Chapter 18.2's 2 900 for an in-flight table, a few hundred gates is noise.

So why is it rare?

Row one is circular and is the strongest reason it stays rare. A corruption in the DMA path produces a frame that passes every check software has — Section 5 — so it is recorded as an application bug, a protocol anomaly, or nothing at all. The absence of reports is not evidence of absence; it is evidence that the only instrument that would report it has not been built.

And there is a weaker version that costs nothing at all and is worth mentioning: have software verify the checksum on a sampled fraction of frames. One frame in a thousand, at 0.1% of Section 2's CPU cost — 0.103% of a core at 100 Gb/s — gives a statistical check across the whole path including DRAM. It will not catch a rare corruption quickly; it will catch a systematic one immediately, and a systematic corruption is what an RTL bug produces.

18. The Cost, Accounted

Eight blocks, and this chapter's cost is dominated by one structure that is not in the list: the per-queue replication.

About 5 650 flops — the largest of Module 18's seven chapters alongside Chapter 18.5's 5 500 — and queue_set is nearly half of it purely because sixteen queues need sixteen of everything.

And that is the chapter's real cost, which the table understates because the replicated structures belong to earlier chapters:

Seven extra queues is about 16 800 flops of replication, against Module 18's single-queue total of roughly 25 700 — so an eight-queue port is about 1.65× the logic of a single-queue one, which is much better than 8× and much worse than free.

What is not replicated is the interesting half:

The receive FIFO being shared is what keeps the memory bill flat, and it is why Module 18's SRAM total does not change with the queue count:

Module 18 complete: about 49 KiB of SRAM and 25 700 flops for a 100 Gb/s port's entire single-queue system interface — 42 500 at eight queues, plus 512 KiB of DRAM rings and 8 MiB of receive buffers that belong to the driver — which remain, as Chapter 18.3 §18 observed, the largest number in the module and the one nobody counts.

19. Properties Worth Asserting, and One Worth Refusing

Offload's properties are about a parse; steering's are about a partition. The rejected one is about the partition, and it is the last in Module 18.

Checksum offload.

// A verdict is only offered when a checksum was actually computed.
p_verdict_requires_computation:
  assert property (@(posedge clk) disable iff (!rst_n)
    result.l4_checksum_ok |-> result.checksum_computed)
  else $error("a checksum verdict was reported without a computation");

// Every decline condition suppresses the computation.
p_decline_suppresses:
  assert property (@(posedge clk) disable iff (!rst_n)
    (parse.is_fragment || !parse.parse_ok) |-> !result.checksum_computed)
  else $error("a checksum was computed on a frame that should have declined");

// A fragment is always declined -- there is no L4 header to check.
p_fragment_declined:
  assert property (@(posedge clk) disable iff (!rst_n)
    (beat_eof && parse.is_fragment) |=>
      (c_decline_fragment > $past(c_decline_fragment)))
  else $error("an IP fragment was not declined");

// IPv4 options move the L4 offset, so they must decline.
p_options_declined:
  assert property (@(posedge clk) disable iff (!rst_n)
    (beat_eof && parse.has_options) |-> !result.checksum_computed)
  else $error("a frame with IPv4 options was offloaded");

// Every frame produces exactly one outcome.
p_one_outcome_per_frame:
  assert property (@(posedge clk) disable iff (!rst_n)
    beat_eof |=> ((c_computed + c_declined) ==
                  $past(c_computed + c_declined) + 1))
  else $error("a frame produced zero or two checksum outcomes");

// The accumulator is cleared at the start of a frame.
p_accumulator_reset:
  assert property (@(posedge clk) disable iff (!rst_n)
    (beat_valid && beat_sof) |=> (acc == $past(beat_sum)))
  else $error("the checksum accumulator carried across a frame boundary");

Transmit insertion.

// A patch is only emitted at the end of a frame.
p_patch_at_eof:
  assert property (@(posedge clk) disable iff (!rst_n)
    out_patch_valid |-> $past(beat_eof))
  else $error("a checksum patch was emitted mid-frame");

// The patch offset lies inside the frame.
p_patch_within_frame:
  assert property (@(posedge clk) disable iff (!rst_n)
    out_patch_valid |-> (out_patch_offset < frame_bytes))
  else $error("a checksum patch targeted an offset past the frame");

// Insertion is refused when the parse failed and partial mode is off.
p_insertion_refused:
  assert property (@(posedge clk) disable iff (!rst_n)
    (csum_enable && !cfg_partial_mode && !parse.parse_ok) |->
      insertion_impossible)
  else $error("insertion proceeded on an unparseable frame");

// Partial mode never depends on the parse.
p_partial_ignores_parse:
  assert property (@(posedge clk) disable iff (!rst_n)
    cfg_partial_mode |-> !insertion_impossible)
  else $error("partial mode refused a frame on parse grounds");

Segmentation.

// The sequence number advances by exactly the payload.
p_seq_advances_by_payload:
  assert property (@(posedge clk) disable iff (!rst_n)
    (seg_valid && seg_ready) |=> (seq == $past(seq) + $past(this_payload)))
  else $error("the TCP sequence number did not advance by the payload");

// No segment exceeds the MSS.
p_segment_within_mss:
  assert property (@(posedge clk) disable iff (!rst_n)
    seg_valid |-> (seg_payload_bytes <= req_mss))
  else $error("a segment exceeded the MSS");

// PSH and FIN appear only on the last segment.
p_flags_on_last_only:
  assert property (@(posedge clk) disable iff (!rst_n)
    (seg_set_fin || seg_set_psh) |-> seg_is_last)
  else $error("a terminating flag appeared on a non-final segment");

// The segments' payloads sum to the request.
p_segments_sum_to_request:
  assert property (@(posedge clk) disable iff (!rst_n)
    (seg_valid && seg_ready && seg_is_last) |-> (remaining == this_payload))
  else $error("the final segment did not consume the remainder");

// An invalid MSS is refused rather than clamped.
p_bad_mss_refused:
  assert property (@(posedge clk) disable iff (!rst_n)
    mss_invalid |-> !req_ready)
  else $error("an invalid MSS was accepted");

// Exactly one first segment per request.
p_one_first_segment:
  assert property (@(posedge clk) disable iff (!rst_n)
    (seg_valid && seg_ready && seg_is_first) |=>
      (!seg_is_first throughout seg_is_last[->1]))
  else $error("two segments were marked first");