Hash-Based Distribution and Frame Ordering

Chapter 15.1 treated the distributor as a black box with exactly one property: the same flow always takes the same member. This chapter opens it, and the box turns out to contain three independent decisions, each of which can quietly ruin the aggregate.

Which fields define a flow. Get this wrong and a router behind the aggregate is one conversation, or a single TCP connection is a thousand.

Which function maps them to a number. Get this wrong and the hash correlates with a field the traffic does not vary.

And how that number becomes a member index. Get this wrong — and almost every design does — and losing one member of four moves 74.9% of the flows when the theoretical minimum is 25%.

That third number is the chapter's centre. A modulo is the obvious reduction, it is what Chapter 15.1 §5 wrote, and it moves three times as much traffic at a failover as it has to. The fix costs 96 octets and is not in most designs.

1. Scope — What This Chapter Owns

This chapter owns the distribution function: field selection, the hash, the reduction to a member index, the rebalance behaviour when the member count changes, the fairness that results, and the flow that defeats all of it.

It does not own the aggregate. Chapter 15.1 built the member table, the compaction, link-down detection and the failover sequence. This chapter supplies the one function that chapter treated as opaque.

It does not own the ordering constraint — Chapter 15.1 §2 derived it: every frame of one conversation must take one member, because Ethernet has no reorder window and two members' queues can differ by 524 µs at 1 Gb/s. This chapter takes that as given and works entirely inside it.

It does not own negotiation. Whether the far end agrees about the membership at all is Chapter 15.3. The distribution function is deliberately not negotiated — 802.1AX requires only that each end be consistent with itself — and Section 17 explains why that is the right choice and what it costs.

And it does not own hashing in general. Chapter 12.5 §4 built an address hash for a set-associative table and §9 derived what an uneven distribution costs it. Section 8 recomputes that arithmetic for four members instead of 2048 sets, and the answer is much worse for a reason that is structural rather than accidental.

2. Why Per-Frame Distribution Is Not Available

Chapter 15.1 §2 established the constraint. This section states its cost precisely, because the cost is large and a design that does not know it will keep looking for a way round.

Per-frame round-robin distribution would be perfect.

Every row above the last favours per-frame distribution and the last one settles it.

The reordering is not marginal. Two members whose egress queues differ by 512 of Chapter 14.1's 128-octet cells differ by 65 536 octets of backlog — 524 µs at 1 Gb/s, 52.4 µs at 10. A round-robin distributor placing consecutive frames alternately on those two members delivers frame n+1 half a millisecond before frame n, continuously, on a link with no errors and no loss.

And there is no repair available at any layer.

Ethernet cannot fix it — Chapter 5.1's frame has no sequence number and no receiver has a reorder buffer.

The transport reads it as loss. Three duplicate acknowledgements is TCP's fast-retransmit trigger; a reordering path generates them continuously, so the sender halves its window over and over on a path that is dropping nothing.

And a storage protocol needs a reassembly buffer sized by the maximum reordering depth, which is the maximum queue-depth difference across members — a quantity no protocol designer was ever given.

So the distributor's freedom is exactly one bit wide: it may choose a member for a flow, once. Everything in this chapter is about making that single choice well, and Section 14 is about the case where no choice is good enough.

3. RTL 1 — Field Selection

The first stage extracts a key. It is the stage that decides what a conversation is, and it is the one most often left at its reset value.

// -----------------------------------------------------------------------
// laghash_pkg -- shared types for the LAG distribution function.
// -----------------------------------------------------------------------
package laghash_pkg;

  localparam int MAX_MEMBERS = 8;
  localparam int MEMBER_W    = $clog2(MAX_MEMBERS);   // 3
  localparam int KEY_W       = 208;                   // full 5-tuple + MACs
  localparam int HASH_W      = 16;

  // Which fields participate. Every bit is a policy decision with a
  // measurable consequence -- section 4's table.
  typedef struct packed {
    logic use_da;      // destination MAC   -- 48
    logic use_sa;      // source MAC        -- 48
    logic use_ethtype; //                   -- 16
    logic use_vid;     // 13.2's VID        -- 12
    logic use_sip;     // IPv4 source       -- 32
    logic use_dip;     // IPv4 destination  -- 32
    logic use_l4sp;    // TCP/UDP source    -- 16
    logic use_l4dp;    // TCP/UDP dest      -- 16
  } field_sel_t;

  // Parsed header fields, produced by 19.2's frame parser and 13.2's
  // tag decoder. This module consumes them; it does not parse.
  typedef struct packed {
    logic [47:0] da;
    logic [47:0] sa;
    logic [15:0] ethtype;
    logic [11:0] vid;
    logic        l3_valid;   // IPv4 present and parseable
    logic [31:0] sip;
    logic [31:0] dip;
    logic        l4_valid;   // TCP or UDP, not a fragment
    logic [15:0] l4sp;
    logic [15:0] l4dp;
  } hdr_t;

  typedef logic [KEY_W-1:0]  key_t;
  typedef logic [HASH_W-1:0] hash_t;

endpackage

// -----------------------------------------------------------------------
// flow_field_selector -- builds the hash key from the selected fields.
//
// Two rules that are not obvious and are both load-bearing:
//   1. A field that is not selected contributes ZERO, not its value
//      masked -- otherwise a deselected field still shifts the others.
//   2. A field that is selected but NOT PRESENT (no IP header, a
//      fragment) must be handled explicitly, because falling back
//      silently collapses every such frame onto one member.
// -----------------------------------------------------------------------
module flow_field_selector
  import laghash_pkg::*;
(
  input  logic       clk,
  input  logic       rst_n,

  input  logic       in_valid,
  input  hdr_t       in_hdr,
  input  field_sel_t cfg_sel,

  output logic       out_valid,
  output key_t       out_key,
  output logic       out_degraded,    // a selected field was absent
  output logic [31:0] c_no_l3,
  output logic [31:0] c_no_l4,
  output logic [31:0] c_degraded
);

  key_t k;
  logic deg, miss_l3, miss_l4;

  always_comb begin
    k       = '0;
    miss_l3 = cfg_sel.use_sip | cfg_sel.use_dip;
    miss_l4 = cfg_sel.use_l4sp | cfg_sel.use_l4dp;
    miss_l3 = miss_l3 & ~in_hdr.l3_valid;
    miss_l4 = miss_l4 & ~in_hdr.l4_valid;
    deg     = miss_l3 | miss_l4;

    // Each field occupies a FIXED slice. A deselected field leaves its
    // slice zero rather than shifting its neighbours, so changing the
    // selection changes which bits vary and never which bits mean what.
    if (cfg_sel.use_da)      k[47:0]      = in_hdr.da;
    if (cfg_sel.use_sa)      k[95:48]     = in_hdr.sa;
    if (cfg_sel.use_ethtype) k[111:96]    = in_hdr.ethtype;
    if (cfg_sel.use_vid)     k[123:112]   = in_hdr.vid;

    if (cfg_sel.use_sip  && in_hdr.l3_valid) k[155:124] = in_hdr.sip;
    if (cfg_sel.use_dip  && in_hdr.l3_valid) k[187:156] = in_hdr.dip;
    if (cfg_sel.use_l4sp && in_hdr.l4_valid) k[203:188] = in_hdr.l4sp;
    if (cfg_sel.use_l4dp && in_hdr.l4_valid) k[207:204] = in_hdr.l4dp[3:0];
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      out_valid <= 1'b0; out_key <= '0; out_degraded <= 1'b0;
      c_no_l3 <= '0; c_no_l4 <= '0; c_degraded <= '0;
    end else begin
      out_valid    <= in_valid;
      out_key      <= k;
      out_degraded <= deg;
      if (in_valid && miss_l3) c_no_l3    <= c_no_l3 + 1;
      if (in_valid && miss_l4) c_no_l4    <= c_no_l4 + 1;
      if (in_valid && deg)     c_degraded <= c_degraded + 1;
    end
  end

endmodule

Classification: a fixed-slice key assembler with a degradation flag. Combinational, one register stage.

What it teaches: that a deselected field must contribute zero rather than being packed out, and the reason is stability of meaning. A design that concatenates only the selected fields changes every bit position when the selection changes, so a management change to the hash policy relocates every flow — which is Section 12's disruption triggered by a configuration edit rather than by a failure. A fixed-slice key relocates flows only because the selected values changed, which is the intent.

And it teaches that out_degraded is the counter nobody has and everybody needs. A design configured to hash on the 5-tuple, carrying traffic that is not IP — ARP, LLDP, Chapter 15.3's LACPDUs, PPPoE, MPLS, IPv6 on a parser that only handles v4 — produces an all-zero L3/L4 key for every one of those frames. They all hash identically, and they all land on the same member.

Deliberately simplified: IPv4 only, and use_l4dp truncates to four bits to fit the 208-bit key, which is a real bug shape rather than an elegance. A production key is wider or the fields are folded, and folding is where an implementation quietly loses entropy — Section 5's callout follows what that does.

Production implication: the reset value of field_sel_t decides the behaviour of every aggregate that nobody configured, and a reset value of MAC-only is the common choice and the worst one behind a router. Section 4's second row: every frame arriving from beyond a router carries the router's source MAC and the local gateway's destination MAC, so the entire internet is one flow and a 4 × 10 aggregate carries all of it on one member.

4. What Fields Define a Flow

The selection decides what the distributor considers "one conversation", and the four common choices behave completely differently depending on what is behind the aggregate.

Column four is the one that decides deployments. A MAC-based hash behind a router sees exactly one source MAC and one destination MAC for all traffic crossing that router, so every conversation on the far side of it is one flow — and the aggregate carries all of it on one member, for ever, with the other three idle.

And the last column is the same problem from the other end. No selection makes one TCP connection into several flows, because there is nothing in its frames that varies. The 5-tuple is constant for the life of the connection by definition. Section 14 is that fact.

==

Which gives the selection rule, and it has two halves that pull in opposite directions:

Select the widest key whose fields actually vary in this deployment. Behind a router that means L3 and L4; behind a switch, MAC addresses are already sufficient and cheaper.

And never select a field the parser cannot reliably produce. A 5-tuple hash on a link carrying 30% non-IP traffic sends all of that 30% to one member — Section 3's c_degraded — which is worse than the MAC hash it replaced.

The two together explain why the useful default is "5-tuple when available, MAC pair otherwise", and why that policy needs the degradation counter to be safe. Without it, the fallback is invisible, and the fallback is where all the imbalance is.

5. RTL 2 — The Hash

A 208-bit key must become a small number, and the function has to be uniform over keys that are anything but.

// -----------------------------------------------------------------------
// flow_hash_crc -- reduces the flow key to 16 bits with a CRC.
//
// CRC is the right family here for the same reason it was right in
// chapter 6: it is linear over GF(2), so its output bits are each a
// parity of a fixed subset of input bits, and a single input bit flip
// changes the output in a way determined only by the polynomial.
//
// The key property this needs and 6.2's CRC-32 also has: every output
// bit depends on MANY input bits. A hash whose low bits depend only on
// the key's low bits inherits the key's structure exactly.
// -----------------------------------------------------------------------
module flow_hash_crc
  import laghash_pkg::*;
#(
  // CRC-16-CCITT: x^16 + x^12 + x^5 + 1. Chosen because its taps are
  // spread, so every output bit mixes widely separated key bits.
  parameter logic [15:0] POLY = 16'h1021,
  parameter logic [15:0] INIT = 16'hFFFF,
  // A per-device seed. Two switches in a path must not hash identically
  // -- section 7's callout explains what happens when they do.
  parameter logic [15:0] SEED = 16'h0000
)(
  input  logic  clk,
  input  logic  rst_n,
  input  logic  in_valid,
  input  key_t  in_key,
  output logic  out_valid,
  output hash_t out_hash
);

  // Fully unrolled: 208 bits of key folded in one cycle. The logic is a
  // 208 x 16 GF(2) matrix multiply -- a tree of XORs, depth ceil(log2)
  // of the worst column's population, which for CRC-16 is about 8.
  function automatic hash_t crc16_all(input key_t d);
    hash_t c;
    int i;
    begin
      c = INIT ^ SEED;
      for (i = KEY_W-1; i >= 0; i--) begin
        if (c[15] ^ d[i]) c = {c[14:0], 1'b0} ^ POLY;
        else              c = {c[14:0], 1'b0};
      end
      crc16_all = c;
    end
  endfunction

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      out_valid <= 1'b0;
      out_hash  <= '0;
    end else begin
      out_valid <= in_valid;
      out_hash  <= crc16_all(in_key);
    end
  end

endmodule

Classification: a combinational GF(2) matrix multiply, registered. One cycle, no state, no back-pressure.

What it teaches: that linearity is the property being bought, not scrambling. A CRC's output bit j is the parity of a fixed subset of input bits, determined entirely by the polynomial. This means the distribution's uniformity is analysable — a key field that varies over a range maps to output bits whose dependence on that field can be written down — where an ad-hoc mixing function's behaviour on structured input can only be measured.

And it teaches why SEED is a parameter rather than a constant, which is the single most consequential line in the module. Two switches in series that hash identically produce correlated distributions: a flow that hashed to member 2 at the first switch hashes to member 2 at the second, so the second aggregate's imbalance is not independent of the first's — it is a copy of it. A per-device seed decorrelates them at zero cost.

Deliberately simplified: the loop is written bit-serially for readability and is intended to be unrolled entirely by synthesis. A real implementation writes the 208 × 16 XOR matrix directly — Chapter 6.4 built exactly this transformation for CRC-32 over a 64-bit datapath — because the unrolled form's logic depth is what determines whether the hash fits in the frame budget.

Production implication: CRC-16 over a 208-bit key is a 16 × 208 matrix and the deepest output bit is an XOR of roughly half the key bits — about 104 terms, a tree of depth 7. At Chapter 12.1 §12's 28 ns per frame that is comfortable; at 100 Gb/s with a 3.1 ns budget it is not, and the hash is then pipelined into two stages or the key is narrowed. Narrowing the key to save depth is the tempting move and it silently reverses Section 4's selection decision.

6. RTL 3 — Reducing the Hash to a Member Index

Sixteen bits must become a member number between zero and N−1. This is the stage where the chapter's central number comes from.

// -----------------------------------------------------------------------
// hash_to_member -- three reductions, side by side, so the difference
// is measurable rather than argued about.
//
//   MODE_MOD    : idx = hash % N          -- the obvious one
//   MODE_MASK   : idx = hash[k-1:0]       -- power-of-two N only
//   MODE_TABLE  : idx = table[hash[B-1:0]] -- an indirection, section 13
//
// Sections 7 and 12 show that the three are indistinguishable while the
// membership is stable and differ by a factor of three when it changes.
// -----------------------------------------------------------------------
module hash_to_member
  import laghash_pkg::*;
#(
  parameter int BUCKET_BITS = 8            // 256 buckets for MODE_TABLE
)(
  input  logic                  clk,
  input  logic                  rst_n,

  input  logic                  in_valid,
  input  hash_t                 in_hash,
  input  logic [MEMBER_W:0]     active_count,
  input  logic [1:0]            mode,       // 0 mod, 1 mask, 2 table

  // MODE_TABLE's bucket map, written by section 13's engine.
  input  logic [MEMBER_W-1:0]   bucket_map [1<<BUCKET_BITS],

  output logic                  out_valid,
  output logic [MEMBER_W-1:0]   out_member,
  output logic                  out_pow2,    // the count is a power of two
  output logic [31:0]           c_by_mode [3]
);

  logic [MEMBER_W-1:0] m_mod, m_mask, m_tab;
  logic                pow2;

  // A count is a power of two when exactly one bit is set.
  assign pow2 = (active_count != 0) &&
                ((active_count & (active_count - 1)) == 0);

  always_comb begin
    m_mod  = '0;
    m_mask = '0;
    m_tab  = bucket_map[in_hash[BUCKET_BITS-1:0]];

    if (active_count != 0)
      m_mod = MEMBER_W'(in_hash % {13'd0, active_count});

    // Masking is only correct for a power-of-two count. For any other
    // count it is not merely uneven -- it names members that do not
    // exist, so the design MUST fall back.
    if (pow2)
      m_mask = MEMBER_W'(in_hash & (active_count - 1));
    else
      m_mask = m_mod;
  end

  always_ff @(posedge clk or negedge rst_n) begin
    int i;
    if (!rst_n) begin
      out_valid <= 1'b0; out_member <= '0; out_pow2 <= 1'b0;
      for (i = 0; i < 3; i++) c_by_mode[i] <= '0;
    end else begin
      out_valid <= in_valid;
      out_pow2  <= pow2;
      unique case (mode)
        2'd0: out_member <= m_mod;
        2'd1: out_member <= m_mask;
        default: out_member <= m_tab;
      endcase
      if (in_valid) c_by_mode[mode[1:0] > 2 ? 2 : mode] <=
                    c_by_mode[mode[1:0] > 2 ? 2 : mode] + 1;
    end
  end

endmodule

Classification: three parallel reductions with a select. The modulo is the expensive one and the table is the correct one.

What it teaches: that masking is not an optimisation of the modulo — it is a different function that happens to agree when N is a power of two. For N = 3 the mask produces indices 0 through 7 and five of them name nothing, so a design that masks must detect the non-power-of-two case and fall back, which means it contains the divider anyway. The mask saves logic only on aggregates whose member count never leaves a power of two, and a member failure is precisely what takes it away from one.

And it teaches the shape of the fix without yet giving the numbers. MODE_TABLE puts a fixed number of buckets — 256 here — between the hash and the members. The hash's divisor never changes, so a flow's bucket is permanent; only the bucket-to-member assignment moves, and Section 12 shows that this is the entire difference between 74.9% and 25%.

Deliberately simplified: the modulo is a % operator, which synthesises into a restoring divider with latency proportional to the operand width. Production designs use a reciprocal multiply — precompute 2^k / N when the membership changes, then multiply and shift — which turns a multi-cycle divide into one multiply, at the cost of a small table indexed by N.

Production implication: out_pow2 exists to make a silent mode change visible. A four-member aggregate masks; the same aggregate with one member down does not, so the reduction function changes at exactly the moment the traffic is redistributing. A design that reports its member count and not its reduction mode gives an operator no way to know that the arithmetic changed under them, and Section 12's disruption is attributed to the failover rather than to the mode switch that came with it.

7. Modulo-N Against Power-of-Two

While the membership is stable the two reductions are indistinguishable. The difference appears entirely at a failover, and it is threefold.

Which means the mask is available exactly when the member count is 1, 2, 4 or 8 — and a member failure moves it off every one of those. A four-member aggregate losing one member becomes a three-member aggregate, so the design that masked in normal operation divides during the failure, and it must contain the divider to be correct.

The saving is therefore in area during the common case and nothing else, and the cost is a reduction function that changes behaviour at the worst moment.

And there is a second, larger consequence which is the whole reason this section exists. Both reductions take the hash modulo something that changes — whether by division or by masking — so both move flows that had no reason to move.

8. Chapter 12.5's Arithmetic, Recomputed for a Four-Member LAG

Chapter 12.5 §9 asked what fraction of a hash table's nominal capacity is usable when the hash spreads unevenly. The same question for an aggregate has a much worse answer, and the reason is structural.

First, the two problems side by side.

Row four is the entire difference. A set-associative table absorbs a collision: two addresses hashing to the same set both fit, because the set holds four. An aggregate has no associativity available at all — Chapter 15.1 §2 forbids putting one flow on two members — so every collision is felt.

And row two matters almost as much. Concentration in a balls-in-bins process improves as the bin count rises at fixed load, and 2048 bins is three orders of magnitude more than 4.

Recomputing exactly, for F flows into 4 members with a uniform hash:

==

And the second finding, which reverses an instinct: more members is worse at a fixed flow count.

Sixteen members carrying 64 flows uses half of itself. Adding members raises nominal capacity and lowers the flows-per-member ratio, and the ratio is what sets the concentration — so the eighth and sixteenth members deliver progressively less of what they promise.

Which gives the module's real sizing rule and it is not the obvious one: an aggregate's member count should be chosen against the flow count it will carry, not against the bandwidth it should nominally provide. Four members and 64 flows is 79.1%; sixteen members and the same 64 flows is 50.8%. In member-rates of usable capacity that is 4 × 0.791 = 3.16 against 16 × 0.508 = 8.13 — twelve extra links bought 4× the nominal capacity and 2.57× the usable.

9. RTL 4 — Measuring the Distribution

Section 8's table is what a perfect hash does. A real one does worse and the difference has to be measured, because it is the only way to tell a bad hash from a small flow count.

// -----------------------------------------------------------------------
// distribution_monitor -- per-member octet and flow counts over a
// window, the observed imbalance, and the EXPECTED imbalance for the
// flow count actually present.
//
// The observed figure alone is uninterpretable: 15.1 section 13 showed
// a ratio of 2.13 is correct at four flows and a defect at four
// thousand. The comparison is the measurement.
// -----------------------------------------------------------------------
module distribution_monitor
  import laghash_pkg::*;
#(
  parameter int WINDOW_CYCLES = 500000,   // 1 ms at 500 MHz
  parameter int SKETCH_BITS   = 12        // 4096-bucket flow presence
)(
  input  logic                clk,
  input  logic                rst_n,

  input  logic                tx_valid,
  input  logic [MEMBER_W-1:0] tx_member,
  input  logic [13:0]         tx_octets,
  input  hash_t               tx_hash,
  input  logic                tx_degraded,   // section 3's flag

  input  logic [MEMBER_W:0]   active_count,

  output logic [31:0]         win_octets [MAX_MEMBERS],
  output logic [15:0]         observed_x100,
  output logic [15:0]         expected_x100,
  output logic                distribution_suspect,
  output logic [15:0]         flows_est,
  output logic [15:0]         degraded_pct_x100,
  output logic [31:0]         c_windows
);

  logic [31:0] acc  [MAX_MEMBERS];
  logic [31:0] cyc;
  logic [(1<<SKETCH_BITS)-1:0] seen;
  logic [31:0] deg_cnt, all_cnt;

  logic [15:0] popc;
  always_comb begin
    int i;
    popc = '0;
    for (i = 0; i < (1<<SKETCH_BITS); i++) popc = popc + 16'(seen[i]);
  end

  // Section 8's table, as a step function of flows per member. The
  // hardware carries the shape; the exact values live in the chapter.
  function automatic logic [15:0] expected_ratio_x100(
      input logic [15:0] f, input logic [MEMBER_W:0] n);
    logic [15:0] fpm;
    begin
      fpm = (n == 0) ? 16'd0 : (f / 16'(n));
      if      (fpm <  16'd1)   expected_ratio_x100 = 16'd250;
      else if (fpm <  16'd2)   expected_ratio_x100 = 16'd213;
      else if (fpm <  16'd4)   expected_ratio_x100 = 16'd176;
      else if (fpm <  16'd8)   expected_ratio_x100 = 16'd153;
      else if (fpm <  16'd16)  expected_ratio_x100 = 16'd137;
      else if (fpm <  16'd32)  expected_ratio_x100 = 16'd126;
      else if (fpm <  16'd64)  expected_ratio_x100 = 16'd119;
      else if (fpm <  16'd128) expected_ratio_x100 = 16'd113;
      else if (fpm <  16'd256) expected_ratio_x100 = 16'd109;
      else                     expected_ratio_x100 = 16'd104;
    end
  endfunction

  always_ff @(posedge clk or negedge rst_n) begin
    int m;
    if (!rst_n) begin
      for (m = 0; m < MAX_MEMBERS; m++) begin
        acc[m] <= '0; win_octets[m] <= '0;
      end
      cyc <= '0; seen <= '0; deg_cnt <= '0; all_cnt <= '0;
      observed_x100 <= 16'd100; expected_x100 <= 16'd100;
      distribution_suspect <= 1'b0; flows_est <= '0;
      degraded_pct_x100 <= '0; c_windows <= '0;
    end else begin
      if (tx_valid) begin
        acc[tx_member] <= acc[tx_member] + tx_octets;
        seen[tx_hash[SKETCH_BITS-1:0]] <= 1'b1;
        all_cnt <= all_cnt + 1;
        if (tx_degraded) deg_cnt <= deg_cnt + 1;
      end

      if (cyc == WINDOW_CYCLES-1) begin
        automatic logic [31:0] total, mx;
        automatic logic [15:0] exp_v;
        total = '0; mx = '0;
        for (m = 0; m < MAX_MEMBERS; m++) begin
          win_octets[m] <= acc[m];
          total = total + acc[m];
          if ((m < int'(active_count)) && (acc[m] > mx)) mx = acc[m];
          acc[m] <= '0;
        end

        exp_v = expected_ratio_x100(popc, active_count);
        observed_x100 <= (total == 0) ? 16'd100
                       : 16'((mx * active_count * 100) / total);
        expected_x100 <= exp_v;
        flows_est     <= popc;

        // Suspect only when the observation exceeds the expectation by
        // more than a fifth. Inside that band the flow count explains it.
        distribution_suspect <=
          (total != 0) &&
          (((mx * active_count * 100) / total) > ((exp_v * 12) / 10));

        degraded_pct_x100 <= (all_cnt == 0) ? 16'd0
                           : 16'((deg_cnt * 10000) / all_cnt);

        seen <= '0; deg_cnt <= '0; all_cnt <= '0;
        cyc  <= '0;
        c_windows <= c_windows + 1;
      end else begin
        cyc <= cyc + 1'b1;
      end
    end
  end

endmodule

Classification: a windowed accumulator with an embedded expectation model. It produces a judgement, not a measurement.

What it teaches: that a fairness measurement needs a null hypothesis in hardware. Every other counter in this track reports what happened; this one reports what happened against what should have happened, and without the second half the first is unusable. A ratio of 2.13 is simultaneously the expected value at four flows and a serious defect at four thousand, and no threshold on the ratio alone can separate them.

And it teaches why degraded_pct_x100 sits in the same module. Section 3's degraded frames all carry an identical L3/L4 key and therefore an identical hash, so they land on one member and inflate the observed imbalance without the flow count noticing — the sketch sees one bucket, so flows_est is not misled, but the octets are all on one member. A LAG showing distribution_suspect with degraded_pct_x100 at 3000 has a parser problem, not a hash problem.

Deliberately simplified: the imbalance uses maximum against mean, which is one number where three are informative. Production designs also carry the minimum and the variance: a maximum 30% above the mean with every other member equal is a single heavy flow — Section 14 — while the same maximum with a spread underneath it is a poor hash. The two have different remedies and the same headline figure.

Production implication: the 4096-bit sketch is 512 octets and it is the cheapest useful thing in the module. Without a flow estimate an operator has no way to interpret any imbalance figure at all, which is why the common outcome of an imbalance investigation is a hash-policy change that does nothing — Section 22's first misconception. 512 octets converts the question from a debate into a comparison.

10. What Balls-in-Bins Says About Fairness at Realistic Flow Counts

Section 8 gave the arithmetic. This is what it means for the three deployments an aggregate is actually used in.

Rows three to five are the same failure at different degrees, and only the last one is fixable.

The router row is a configuration error — the 5-tuple is available and the fields vary; the hash was simply told to ignore them. Section 3's field_sel_t reset value is the culprit and one write fixes it.

The storage row is a workload the mechanism does not fit. Four flows into four members is 47.0% with a perfect hash and no configuration changes that.

And the single-transfer row is Section 14, which no distributor can address.

Which produces the honest guidance a datasheet cannot give: aggregation's capacity benefit is a function of flow count, and the flow count is a property of the traffic rather than of the network. A four-member aggregate is 96.9% efficient in a core and 25% efficient to a single server, and the hardware is identical.

And it explains the pattern in real deployments that looks like inconsistency and is not: aggregates in a fabric core are sized for capacity and aggregates to an endpoint are sized for availability — Chapter 15.1 §4's two-member configuration carrying well under one member's load, which is not wasteful because capacity was never what it was buying.

11. RTL 5 — The Rebalance, Measured

Section 12's number needs an instrument that runs in silicon, because the disruption happens during a failover and a simulation of the hash is not evidence about the deployed one.

// -----------------------------------------------------------------------
// rebalance_model -- when the member count changes, measures what
// fraction of flows actually moved.
//
// It keeps the OLD reduction alongside the new one for a window and
// compares. This is 2 x 3 bits per tracked bucket -- 192 octets for
// 256 buckets -- and it converts section 12's theory into a
// per-deployment measurement.
// -----------------------------------------------------------------------
module rebalance_model
  import laghash_pkg::*;
#(
  parameter int BUCKET_BITS   = 8,
  parameter int OBSERVE_CYCLES = 5000000   // 10 ms at 500 MHz
)(
  input  logic                clk,
  input  logic                rst_n,

  input  logic                count_changed,
  input  logic [MEMBER_W:0]   old_count,
  input  logic [MEMBER_W:0]   new_count,

  input  logic                tx_valid,
  input  hash_t               tx_hash,
  input  logic [MEMBER_W-1:0] tx_member,     // the NEW assignment

  output logic [15:0]         moved_pct_x100,
  output logic [15:0]         ideal_pct_x100,
  output logic                observing,
  output logic [31:0]         c_rebalances
);

  logic [31:0] obs_cnt;
  logic [31:0] n_seen, n_moved;
  logic [MEMBER_W:0] old_n;

  // What the OLD membership would have chosen for this hash.
  logic [MEMBER_W-1:0] would_have;
  always_comb begin
    would_have = '0;
    if (old_n != 0) would_have = MEMBER_W'(tx_hash % {13'd0, old_n});
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      observing <= 1'b0; obs_cnt <= '0; n_seen <= '0; n_moved <= '0;
      moved_pct_x100 <= '0; ideal_pct_x100 <= '0;
      old_n <= '0; c_rebalances <= '0;
    end else begin
      if (count_changed) begin
        observing    <= 1'b1;
        obs_cnt      <= '0;
        n_seen       <= '0;
        n_moved      <= '0;
        old_n        <= old_count;
        c_rebalances <= c_rebalances + 1;
        // The theoretical minimum: only the flows that were ON the
        // departed member have to move. 1/old_count of them.
        ideal_pct_x100 <= (old_count == 0) ? 16'd0
                        : 16'(10000 / 16'(old_count));
      end else if (observing) begin
        obs_cnt <= obs_cnt + 1;

        if (tx_valid) begin
          n_seen <= n_seen + 1;
          if (tx_member != would_have) n_moved <= n_moved + 1;
        end

        if (obs_cnt == OBSERVE_CYCLES-1) begin
          observing      <= 1'b0;
          moved_pct_x100 <= (n_seen == 0) ? 16'd0
                          : 16'((n_moved * 10000) / n_seen);
        end
      end
    end
  end

endmodule

Classification: a differential observer. It computes a counterfactual and compares it against reality.

What it teaches: that the disruption a failover causes is measurable at run time and almost never measured. The old divisor is one register; the counterfactual is one modulo; the comparison is one inequality. For 192 octets of state a design can report that its last failover moved 74.9% of flows against an ideal of 25%, which is the difference between a known cost and an invisible one.

And it teaches what "ideal" means here precisely. The minimum possible disruption is exactly the flows that were on the member that left — 1/old_count of them, which for four members is 25%. Every flow beyond that moved for arithmetic reasons rather than for physical ones, and each one costs Chapter 15.1 §8's drain and an ordering risk.

Deliberately simplified: frames are sampled as they pass rather than enumerated, so moved_pct_x100 is weighted by traffic rather than by flow count. That is arguably the better measurement — a moved flow carrying nothing costs nothing — but it is not what Section 12's table reports, and comparing the two directly without knowing that is comparing a flow fraction against an octet fraction.

Production implication: moved_pct_x100 against ideal_pct_x100 is the one number that justifies Section 13's table. A design showing 7490 against 2500 has a three-fold excess to remove for 96 octets; one already showing 2500 has nothing to gain. Without the measurement the argument for the indirection table is theoretical, and theoretical arguments lose to "it works today".

12. When N Changes, Three Quarters of the Flows Move

This is the chapter's central number and it is much larger than the intuition it replaces.

When a member fails, the flows that have to move are the ones that were on it. For a four-member aggregate that is one quarter.

A modulo moves three quarters.

The excess is N − 1, and it grows with the aggregate. A sixteen-member aggregate losing one member relocates 93.7% of its flows when 6.2% needed to move — fifteen times more disruption than the failure required.

==

The reason is arithmetic rather than implementation. h % 4 and h % 3 are unrelated functions. A flow whose hash is 17 goes to member 1 under four members and member 2 under three, and nothing about member 1 having survived enters the calculation. The modulo does not know which member left; it only knows there is one fewer.

And every one of those unnecessary moves costs the same as a necessary one:

Chapter 15.1 §8's drain, held until the remaining queues are shallow enough that overtaking is impossible. An ordering risk if the drain times out on a congested aggregate. And a burst of traffic arriving at a member whose queue was sized for its previous share.

So a four-member aggregate losing one member does three times the drain work, takes three times the reordering exposure, and delivers three times the transient load imbalance that the physical event demanded.

The fix is Section 13 and it costs 96 octets.

13. RTL 6 — The Redirection Table

One level of indirection between the hash and the members removes the entire excess, and the reason it works is that it gives the reduction a divisor that never changes.

// -----------------------------------------------------------------------
// consistent_ring_map -- a bucket table between the hash and the
// members.
//
// The hash is reduced modulo a CONSTANT bucket count, so a flow's
// bucket never moves. Only the bucket-to-member assignment changes,
// and on a member failure only that member's buckets are reassigned.
// Section 12's excess disappears: exactly 1/N of the flows move.
// -----------------------------------------------------------------------
module consistent_ring_map
  import laghash_pkg::*;
#(
  parameter int BUCKET_BITS = 8            // 256 buckets, 96 octets
)(
  input  logic                clk,
  input  logic                rst_n,

  input  logic                member_left,
  input  logic [MEMBER_W-1:0] departed,
  input  logic                member_joined,
  input  logic [MEMBER_W-1:0] joined,
  input  logic [MEMBER_W:0]   active_count,
  input  logic [MAX_MEMBERS-1:0] active_mask,

  output logic [MEMBER_W-1:0] bucket_map [1<<BUCKET_BITS],
  output logic                remapping,
  output logic [15:0]         buckets_moved,
  output logic [31:0]         c_remaps
);

  localparam int NB = 1 << BUCKET_BITS;

  logic [MEMBER_W-1:0] map_q [NB];
  logic [BUCKET_BITS:0] idx;
  logic [MEMBER_W-1:0]  rr;          // round-robin over survivors
  logic [15:0]          moved;

  typedef enum logic [1:0] { R_IDLE, R_INIT, R_HEAL, R_JOIN } rstate_e;
  rstate_e st;

  // The next active member after rr, cyclically. Spreading a departed
  // member's buckets round-robin over the survivors keeps them even;
  // giving them all to one survivor would balance the FLOWS and
  // unbalance the LOAD.
  function automatic logic [MEMBER_W-1:0] next_active(
      input logic [MEMBER_W-1:0] from);
    int i;
    logic [MEMBER_W-1:0] c;
    begin
      next_active = from;
      for (i = 1; i <= MAX_MEMBERS; i++) begin
        c = MEMBER_W'((int'(from) + i) % MAX_MEMBERS);
        if (active_mask[c]) begin
          next_active = c;
          break;
        end
      end
    end
  endfunction

  always_ff @(posedge clk or negedge rst_n) begin
    int b;
    if (!rst_n) begin
      for (b = 0; b < NB; b++) map_q[b] <= '0;
      st <= R_INIT; idx <= '0; rr <= '0; moved <= '0;
      buckets_moved <= '0; c_remaps <= '0;
    end else begin
      unique case (st)
        R_INIT: begin
          // Initial fill: buckets round-robin over the active members.
          map_q[idx[BUCKET_BITS-1:0]] <= rr;
          rr  <= next_active(rr);
          if (idx == NB-1) begin
            st  <= R_IDLE;
            idx <= '0;
          end else idx <= idx + 1'b1;
        end

        R_IDLE: begin
          moved <= '0;
          if (member_left) begin
            st <= R_HEAL; idx <= '0; c_remaps <= c_remaps + 1;
          end else if (member_joined) begin
            st <= R_JOIN; idx <= '0; c_remaps <= c_remaps + 1;
          end
        end

        R_HEAL: begin
          // ONLY the departed member's buckets are touched. Every other
          // bucket keeps its member, so every flow not on the departed
          // member stays exactly where it was.
          if (map_q[idx[BUCKET_BITS-1:0]] == departed) begin
            map_q[idx[BUCKET_BITS-1:0]] <= next_active(rr);
            rr    <= next_active(rr);
            moved <= moved + 1'b1;
          end
          if (idx == NB-1) begin
            st            <= R_IDLE;
            idx           <= '0;
            buckets_moved <= moved;
          end else idx <= idx + 1'b1;
        end

        R_JOIN: begin
          // A joining member takes an even share -- every Nth bucket.
          // This DOES move flows that did not have to move; a join is
          // a deliberate act and can be scheduled, unlike a failure.
          if (idx[BUCKET_BITS-1:0] % BUCKET_BITS'(active_count) == '0) begin
            map_q[idx[BUCKET_BITS-1:0]] <= joined;
            moved <= moved + 1'b1;
          end
          if (idx == NB-1) begin
            st            <= R_IDLE;
            idx           <= '0;
            buckets_moved <= moved;
          end else idx <= idx + 1'b1;
        end
      endcase
    end
  end

  always_comb begin
    int b;
    for (b = 0; b < NB; b++) bucket_map[b] = map_q[b];
  end

  assign remapping = (st != R_IDLE);

endmodule

Classification: a small rewritable map with a healing sequencer. 256 entries, one rewrite per member event.

What it teaches: that the fix is to move the changing divisor out of the flow's path. A flow's bucket is hash % 256 and 256 never changes, so the flow's bucket is permanent for the life of the connection. What changes is one column of a table. A member's departure touches only the buckets that named it — one quarter of them for a four-member aggregate — and every other flow is untouched by construction rather than by luck.

And it teaches why the departed member's buckets are spread round-robin rather than handed to one survivor. Handing them to one survivor moves the same number of flows and unbalances the result: three members with a quarter each and one with a half. Spreading them keeps the post-failure distribution as even as the pre-failure one was, which matters because the aggregate now has less capacity and needs all of it.

Deliberately simplified: the heal sequencer walks all 256 buckets at one per cycle — 512 ns at 500 MHz — and the map is read combinationally throughout, so a lookup during R_HEAL may see a partially updated table. That is safe for correctness (every entry always names an active member, because a bucket is rewritten before the departed member is removed from active_mask) and it does mean a flow can move twice if it is looked up mid-sweep. Production designs double-buffer the map — 192 octets instead of 96 — and swap atomically.

Production implication: the bucket count sets the granularity of the balance. 256 buckets over four members is 64 buckets each, so a member's share is quantised to 1/256 of the traffic — finer than any imbalance worth acting on. Over sixteen members it is sixteen buckets each, and the quantisation starts to matter: one bucket is 6.25% of a member's share. The rule is at least 16 buckets per member, and 64 is comfortable — which for an eight-member aggregate means 512 buckets and 192 octets.

14. The Elephant Flow

Every technique in this chapter distributes flows. None of them can distribute a flow, and that is not a limitation of the technique.

Chapter 15.1 §2's constraint is absolute: all frames of one conversation take one member. So a conversation offering more than one member's rate is throttled to one member's rate, on an aggregate with idle capacity, permanently.

And the failure is completely silent. The aggregate is up. Every member is healthy. No counter increments, no frame is dropped that would not have been dropped on a single link, and Chapter 15.1 §15's conformant is high. The only symptom is a transfer running at a quarter of the rate the aggregate's name implies.

==

Three responses exist and it is worth being clear that only the third is a solution.

One — split the conversation at a higher layer. Multiple TCP connections between the same pair of hosts have different L4 ports, so the 5-tuple varies and the hash spreads them. This is what a well-written storage client does and it is why parallel-stream file transfer tools exist. It is not something the network can do.

Two — detect the elephant and pin it deliberately. Section 15's detector finds the flow; a design can then place it on the least-loaded member rather than the hashed one, which does not raise its ceiling but stops it colliding with other large flows. This is worth doing and is frequently mistaken for a fix.

Three — do not use an aggregate. A single 100 Gb/s link carries a 40 Gb/s conversation and a 4 × 25 aggregate does not. When the workload is a small number of large flows, one fast link beats several slow ones, and the aggregate's availability advantage has to be bought some other way.

The reason this deserves a section rather than a footnote is that the first response is invisible to the network team and the third contradicts the procurement decision. So the usual outcome is the second, applied repeatedly, on a problem it cannot solve.

15. RTL 7 — Detecting an Elephant

Finding a heavy flow with no per-flow state is the interesting constraint, and a count-min sketch does it in 1.5 KiB.

// -----------------------------------------------------------------------
// elephant_detector -- identifies flows whose octet rate exceeds a
// fraction of a member's capacity, without a per-flow table.
//
// Four independent hash slices index four counter banks; a flow's
// estimate is the MINIMUM across the four, which bounds the
// over-estimate caused by collisions and never under-estimates.
// -----------------------------------------------------------------------
module elephant_detector
  import laghash_pkg::*;
#(
  parameter int ROWS      = 4,
  parameter int ROW_BITS  = 8,           // 256 counters per row
  parameter int CNT_W     = 24,          // octets per window
  parameter int WINDOW_CYCLES = 500000,  // 1 ms
  // A member at 10 Gb/s carries 1.25 MB in 1 ms. A flow taking a
  // quarter of that is an elephant worth naming.
  parameter int THRESHOLD = 312500
)(
  input  logic       clk,
  input  logic       rst_n,

  input  logic       tx_valid,
  input  hash_t      tx_hash,
  input  logic [13:0] tx_octets,

  output logic       elephant_seen,
  output hash_t      elephant_hash,
  output logic [CNT_W-1:0] elephant_octets,
  output logic [15:0] c_elephants,
  output logic [CNT_W-1:0] heaviest_octets
);

  logic [CNT_W-1:0] cms [ROWS][1<<ROW_BITS];
  logic [31:0]      cyc;

  // Four independent row indices from one 16-bit hash. Slicing a CRC
  // gives near-independent indices because each output bit is a parity
  // of a different subset of key bits -- section 5's linearity, used
  // for a second purpose.
  function automatic logic [ROW_BITS-1:0] row_idx(
      input hash_t h, input int r);
    case (r)
      0: row_idx = h[7:0];
      1: row_idx = h[15:8];
      2: row_idx = h[7:0]  ^ h[15:8];
      default: row_idx = {h[3:0], h[11:8]} ^ h[15:8];
    endcase
  endfunction

  logic [CNT_W-1:0] est;
  always_comb begin
    int r;
    est = '1;
    for (r = 0; r < ROWS; r++)
      if (cms[r][row_idx(tx_hash, r)] < est) est = cms[r][row_idx(tx_hash, r)];
  end

  always_ff @(posedge clk or negedge rst_n) begin
    int r, i;
    if (!rst_n) begin
      for (r = 0; r < ROWS; r++)
        for (i = 0; i < (1<<ROW_BITS); i++) cms[r][i] <= '0;
      cyc <= '0; elephant_seen <= 1'b0; elephant_hash <= '0;
      elephant_octets <= '0; c_elephants <= '0; heaviest_octets <= '0;
    end else begin
      if (tx_valid) begin
        for (r = 0; r < ROWS; r++)
          cms[r][row_idx(tx_hash, r)] <=
            cms[r][row_idx(tx_hash, r)] + CNT_W'(tx_octets);

        // The estimate AFTER this frame, compared against the threshold.
        if ((est + CNT_W'(tx_octets)) >= CNT_W'(THRESHOLD)) begin
          if (!elephant_seen || ((est + CNT_W'(tx_octets)) > elephant_octets)) begin
            elephant_hash   <= tx_hash;
            elephant_octets <= est + CNT_W'(tx_octets);
          end
          if (!elephant_seen) c_elephants <= c_elephants + 1;
          elephant_seen <= 1'b1;
        end

        if ((est + CNT_W'(tx_octets)) > heaviest_octets)
          heaviest_octets <= est + CNT_W'(tx_octets);
      end

      if (cyc == WINDOW_CYCLES-1) begin
        for (r = 0; r < ROWS; r++)
          for (i = 0; i < (1<<ROW_BITS); i++) cms[r][i] <= '0;
        cyc <= '0;
        elephant_seen   <= 1'b0;
        heaviest_octets <= '0;
      end else cyc <= cyc + 1'b1;
    end
  end

endmodule

Classification: a count-min sketch — a probabilistic heavy-hitter detector with a one-sided error.

What it teaches: that the minimum across rows is what makes the structure usable, and the error is one-sided in the safe direction. Every row's counter for a flow includes that flow's octets plus any colliding flows', so every row over-estimates. The minimum over four independent rows is therefore still an over-estimate and never an under-estimate — so a flow the sketch calls small definitely is small, and a flow it calls large might be several medium ones. For this purpose that is exactly the right asymmetry: a false elephant costs an unnecessary look; a missed elephant costs the investigation this module exists to prevent.

And it teaches a second use for Section 5's linearity. The four row indices are slices and XORs of one 16-bit CRC, and they are near-independent because each CRC output bit is a parity of a different subset of key bits. A non-linear hash gives no such guarantee, so building four independent indices from one output would require four hashes.

Deliberately simplified: elephant_hash records the hash, not the flow, so an operator is given a 16-bit number and no way to turn it back into a conversation. Production designs latch the key — Section 3's 208 bits — for the top few offenders, at 26 octets each. A detector that names a flow an operator cannot identify has done half a job, and that half is the cheap half.

Production implication: 4 × 256 × 24 bits is 3 KiB, and reporting heaviest_octets every window costs nothing more. The value of the second output is that it works when there is no elephant: a heaviest_octets at 5% of a member's capacity says the imbalance in Section 9 is not a heavy flow, which eliminates the most expensive hypothesis first — and Section 21's debugging order depends on being able to do that.

16. RTL 8 — Conformance for a Distributor

The distributor's promise is one sentence and the monitor checks that sentence and no more.

// -----------------------------------------------------------------------
// distributor_conformance_monitor -- one bit.
//
// It asserts the ordering obligation and the structural invariants.
// It does NOT assert fairness: section 19's rejected property is
// exactly that assertion, and section 14 shows why it cannot hold.
// -----------------------------------------------------------------------
module distributor_conformance_monitor
  import laghash_pkg::*;
(
  input  logic       clk,
  input  logic       rst_n,

  input  logic       flow_moved_while_stable,  // the ordering violation
  input  logic       member_out_of_range,
  input  logic       masked_with_non_pow2,
  input  logic       cfg_all_fields_off,
  input  logic       cfg_seed_is_default,      // section 7's correlation
  input  logic [15:0] degraded_pct_x100,

  output logic       conformant,
  output logic [7:0] fault_vector,
  output logic [31:0] c_order_violations
);

  logic v_move, v_range, v_mask, v_nofield, v_seed, v_degraded;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      v_move <= 1'b0; v_range <= 1'b0; v_mask <= 1'b0;
      v_nofield <= 1'b0; v_seed <= 1'b0; v_degraded <= 1'b0;
      c_order_violations <= '0;
    end else begin
      // The one promise. A flow that changes member while the
      // membership is unchanged has reordered, and nothing recovers it.
      if (flow_moved_while_stable) begin
        v_move             <= 1'b1;
        c_order_violations <= c_order_violations + 1;
      end

      if (member_out_of_range)  v_range <= 1'b1;
      if (masked_with_non_pow2) v_mask  <= 1'b1;

      // Standing configuration properties -- wrong from power-on.
      v_nofield <= cfg_all_fields_off;
      v_seed    <= cfg_seed_is_default;

      // More than a fifth of frames hashing on an absent field means
      // the selection does not match the traffic -- section 4.
      if (degraded_pct_x100 > 16'd2000) v_degraded <= 1'b1;
    end
  end

  assign conformant   = !(v_move || v_range || v_mask ||
                          v_nofield || v_seed || v_degraded);
  assign fault_vector = {2'b00, v_degraded, v_seed, v_nofield,
                         v_mask, v_range, v_move};

endmodule

Classification: a fault aggregator with two standing configuration terms, one true safety violation and one statistical threshold.

What it teaches: that v_move is the only row that is a genuine correctness failure and the others are all configuration or quality. A flow that changes member while the membership is stable has reordered traffic, and Chapter 15.1 §2 established that nothing above recovers from that. Every other fault bit describes a distributor that is working and working badly.

And it teaches why cfg_seed_is_default is a fault at all. A default seed is not wrong on one device; it is wrong on a fabric, because Section 7's callout showed identical hashes preserving an imbalance across every hop. A conformance bit that fires on it is a device reporting a property of the network it is about to be part of, which is unusual and defensible: the condition is locally checkable and its consequence is not.

Deliberately simplified: flow_moved_while_stable is an input, and producing it needs either per-flow state or Section 11's counterfactual. The practical implementation reuses rebalance_model's comparison while observing is low — if the membership has not changed and the recomputed assignment differs from the actual one, something moved a flow that should not have.

Production implication: conformant here says the distributor kept its ordering promise and is configured sanely. It says nothing about balance, and it is high on the Section 14 aggregate delivering 25% of its nominal rate to one large transfer. That is correct and it is the thing operators most want it to mean. The balance question is answered by distribution_suspect against expected_x100, which is a judgement rather than a conformance claim — and the two are deliberately separate outputs because one of them can be true while the other is false in both directions.

17. What the Distributor Can and Cannot Promise

Put the guarantees and the non-guarantees side by side, because the boundary between them is where every argument about LAG performance actually sits.

Rows four and five are the pair worth separating carefully, because they are routinely conflated and only the first has any mathematics behind it.

Section 8's arithmetic distributes flows. A uniform hash gives each flow an equal chance of each member, and the balls-in-bins table follows.

Nothing distributes octets. Flow sizes in real traffic are heavy-tailed — a small number of conversations carry most of the bytes — so even a perfectly uniform flow distribution produces an octet distribution as uneven as the flow-size distribution is. Section 8's 96.9% at 4096 flows assumes those flows are equal; at realistic size distributions the same 4096 flows do considerably worse, and Section 15's detector exists because the largest of them dominates.

Which is the honest framing of the whole chapter: the distributor's job is to be uniform over flows, and uniformity over flows is not what anybody wanted. What was wanted is uniformity over bytes, and the mechanism has no access to a frame's flow's future size — it must choose a member on the first frame, before anything is known.

And that is the setup for Section 19's rejected property, which is the assertion that the member loads are balanced — a claim about octets, made by a mechanism that can only act on flows, on a workload where one flow can exceed a member.

18. The Cost of Distribution, Accounted

Three stages and six instruments, and the whole thing is smaller than a single frame buffer.

The fourth row is the chapter's best return and the ninth puts it in proportion. 96 octets takes a four-member aggregate's failover disruption from 74.9% of flows to 25%, and from 93.7% to 6.2% on a sixteen-member one. Nothing else in Module 15 has that ratio.

And the instruments cost more than the mechanism, which is the right way round. The redirection table is 96 octets; the four things that measure whether the distributor is working are 3.7 KiB — and without them Section 21's investigation has no order to proceed in.

Compared against the batch's other mechanisms:

Three of the four are kibibytes and one is a mebibyte, and the pattern from Chapter 15.1 §17 holds: these store pointers and indices into things that already exist; PFC reserves buffer against a future.

19. Properties Worth Asserting, and One Worth Refusing

The properties divide by stage: the key, the hash, the reduction, the ordering obligation, the redirection table, and the instruments.

Group 1 — the key.

// P1. A deselected field contributes zero. Changing the selection must
// not relocate flows whose selected fields did not change.
property p_deselected_is_zero;
  @(posedge clk) disable iff (!rst_n)
  (in_valid && !cfg_sel.use_sip) |=> (out_key[155:124] == 32'd0);
endproperty

// P2. Each field occupies a fixed slice regardless of what else is
// selected -- the slices never shift.
property p_slices_are_fixed;
  @(posedge clk) disable iff (!rst_n)
  (in_valid && cfg_sel.use_da) |=> (out_key[47:0] == $past(in_hdr.da));
endproperty

// P3. A selected field that is absent raises degraded. Falling back
// silently collapses every such frame onto one member.
property p_absent_field_flags;
  @(posedge clk) disable iff (!rst_n)
  (in_valid && cfg_sel.use_sip && !in_hdr.l3_valid) |=> out_degraded;
endproperty

// P4. Degraded frames still produce a key -- the mechanism degrades,
// it does not stall.
property p_degraded_still_emits;
  @(posedge clk) disable iff (!rst_n)
  in_valid |=> out_valid;
endproperty

// P5. The key is a pure function of the header and the selection.
property p_key_is_a_function;
  @(posedge clk) disable iff (!rst_n)
  (in_valid && (in_hdr == h) && (cfg_sel == s)) |=> (out_key == keyof(h, s));
endproperty

Group 2 — the hash.

// P6. The hash is deterministic: one key, one value, always.
property p_hash_is_deterministic;
  @(posedge clk) disable iff (!rst_n)
  (in_valid && (in_key == k)) |=> (out_hash == crc16_all(k));
endproperty

// P7. One cycle of latency, no back-pressure, no stall.
property p_hash_latency;
  @(posedge clk) disable iff (!rst_n)
  in_valid |=> out_valid;
endproperty

// P8. Avalanche: flipping any single key bit changes at least one
// output bit. A CRC guarantees this for every bit the polynomial
// reaches, which for a spread polynomial is all of them.
property p_single_bit_avalanche;
  @(posedge clk) disable iff (!rst_n)
  (crc16_all(k) != crc16_all(k ^ (1 << b)));
endproperty

// P9. The seed participates: two devices with different seeds produce
// different hashes for the same key.
property p_seed_decorrelates;
  @(posedge clk) disable iff (!rst_n)
  (SEED != OTHER_SEED) |-> (crc16_all(k) != crc16_other(k));
endproperty

Group 3 — the reduction.

// P10. The member index is always in range for the active count.
property p_member_in_range;
  @(posedge clk) disable iff (!rst_n)
  out_valid |-> (out_member < active_count);
endproperty

// P11. Masking is used only when the count is a power of two.
property p_mask_only_pow2;
  @(posedge clk) disable iff (!rst_n)
  ((mode == 2'd1) && out_valid && !out_pow2) |-> (out_member == m_mod);
endproperty

// P12. pow2 detection is exact.
property p_pow2_detection;
  @(posedge clk) disable iff (!rst_n)
  out_pow2 <-> ((active_count != 0) &&
                ((active_count & (active_count - 1)) == 0));
endproperty

// P13. Mask and modulo agree whenever the count is a power of two.
property p_mask_agrees_when_pow2;
  @(posedge clk) disable iff (!rst_n)
  out_pow2 |-> (m_mask == m_mod);
endproperty

// P14. The table mode never depends on the active count at all --
// that is the entire point of section 13.
property p_table_ignores_count;
  @(posedge clk) disable iff (!rst_n)
  ((mode == 2'd2) && $stable(bucket_map)) |-> $stable(m_tab);
endproperty

Group 4 — the ordering obligation.

// P15. THE promise. While the membership is stable, one flow takes one
// member. Everything else in Module 15 rests on this.
property p_flow_is_pinned;
  @(posedge clk) disable iff (!rst_n)
  ($stable(active_count) && $stable(bucket_map) && (tx_hash == h))
    |-> (tx_member == $past(member_for(h)));
endproperty

// P16. A flow moves only while the map or the membership is changing.
property p_moves_only_during_change;
  @(posedge clk) disable iff (!rst_n)
  flow_moved |-> (remapping || $changed(active_count));
endproperty

// P17. An ordering violation is always counted, never silent.
property p_violation_is_counted;
  @(posedge clk) disable iff (!rst_n)
  flow_moved_while_stable |=> $changed(c_order_violations);
endproperty

Group 5 — the redirection table.

// P18. Every bucket always names an active member. There is no window
// in which a lookup can return a dead port.
property p_bucket_names_active;
  @(posedge clk) disable iff (!rst_n)
  active_mask[bucket_map[b]];
endproperty

// P19. A heal touches ONLY the departed member's buckets.
property p_heal_touches_only_departed;
  @(posedge clk) disable iff (!rst_n)
  ((st == R_HEAL) && (map_q[idx] != departed)) |=> $stable(map_q[idx]);
endproperty

// P20. The number of buckets moved by a heal equals the number the
// departed member held -- no more.
property p_heal_moves_exactly_its_share;
  @(posedge clk) disable iff (!rst_n)
  $fell(remapping) |-> (buckets_moved == $past(count_of(departed)));
endproperty

// P21. A heal terminates in exactly NB cycles.
property p_heal_terminates;
  @(posedge clk) disable iff (!rst_n)
  $rose(remapping) |-> ##[1:(1<<BUCKET_BITS)+1] !remapping;
endproperty

// P22. The departed member's buckets are spread, not handed to one
// survivor: no survivor gains more than ceil(share/(N-1)) buckets.
property p_heal_spreads;
  @(posedge clk) disable iff (!rst_n)
  $fell(remapping) |-> (max_gain <= ((share + active_count - 2) / (active_count - 1)));
endproperty

Group 6 — the instruments.

// P23. The observed imbalance is at least 100: a maximum cannot be
// below a mean.
property p_observed_floor;
  @(posedge clk) disable iff (!rst_n)
  window_tick |=> (observed_x100 >= 16'd100);
endproperty

// P24. suspect is raised only against the expectation, never against a
// fixed threshold.
property p_suspect_uses_expectation;
  @(posedge clk) disable iff (!rst_n)
  distribution_suspect |-> (observed_x100 > ((expected_x100 * 12) / 10));
endproperty

// P25. The expectation is computed from the flow count measured in the
// same window.
property p_expectation_is_current;
  @(posedge clk) disable iff (!rst_n)
  window_tick |=> (expected_x100 == expected_ratio_x100(flows_est, active_count));
endproperty

// P26. The count-min estimate never under-estimates. The error is
// one-sided, which is what makes the sketch usable.
property p_cms_never_under;
  @(posedge clk) disable iff (!rst_n)
  est >= true_octets_for(tx_hash);
endproperty

// P27. An elephant report implies the threshold was crossed.
property p_elephant_implies_threshold;
  @(posedge clk) disable iff (!rst_n)
  $rose(elephant_seen) |-> (elephant_octets >= CNT_W'(THRESHOLD));
endproperty

// P28. The sketch clears every window, so an elephant is a rate rather
// than an accumulation.
property p_sketch_clears;
  @(posedge clk) disable iff (!rst_n)
  (cyc == WINDOW_CYCLES-1) |=> (cms[0][0] == '0);
endproperty

// P29. The rebalance model's ideal is 1/old_count, which is the number
// of flows that were physically on the departed member.
property p_ideal_is_one_over_n;
  @(posedge clk) disable iff (!rst_n)
  count_changed |=> (ideal_pct_x100 == 16'(10000 / $past(old_count)));
endproperty

// P30. Standing property: at least one field is selected. A key of all
// zeros sends every frame to member 0.
property p_some_field_selected;
  @(posedge clk) disable iff (!rst_n)
  !cfg_all_fields_off;
endproperty

// P31. conformant is exactly the conjunction it claims.
property p_conformant_definition;
  @(posedge clk) disable iff (!rst_n)
  conformant <-> (fault_vector == 8'h00);
endproperty

Every property above is about the distributor's behaviour — determinism, range, pinning, termination, and the honesty of the instruments. None of them is about how much traffic each member ends up carrying, and the reason is the property this chapter refuses.

20. Verification Scenarios

Seventy-one scenarios. The ones that matter most have expected outcomes in which the distributor is perfect and the aggregate delivers a quarter of its rate.

Field selection

The hash

Reduction

Distribution, at a perfect hash

Rebalance

The elephant

Ordering

The directed test random stimulus will not produce

Random traffic cannot produce the modulo-rebalance measurement, because it requires a counterfactual: the fraction of flows whose member differs from what the previous membership would have chosen. A random run observes only what happened. Producing the comparison requires driving the same flow set through the same distributor under two different member counts and correlating per flow, which is a directed construction rather than a stimulus distribution — and no coverage metric asks for it.

Setup: a four-member aggregate, MODE_MOD, CRC-16 with a non-default seed, 5-tuple selection. Ten thousand distinct flows, each identified by a unique L4 source port, each offering one frame per 10 µs so that traffic weight and flow weight coincide and the two measurements can be compared directly.

Stimulus, four phases. Phase 1: run all 10 000 flows for 10 ms with four members; record each flow's member. Phase 2: fail member 1. Phase 3: run the identical 10 000 flows for 10 ms; record each flow's member. Phase 4: repeat phases 1 to 3 with MODE_TABLE and 256 buckets.

Oracle:

Rows 2 and 11 are why this is not found by accident: the distributor looks identical before the failure and identical after it. The entire difference is in row 4, which exists only during the transition and only as a comparison against a counterfactual nobody computes.

And row 8 is the sentence to take away: 4990 flows — half the traffic on the aggregate — were reordered, drained and relocated because the divisor changed, on members that never failed.

21. Debugging an Uneven LAG

Five questions, and the order matters because four of the five hypotheses produce the same symptom.

Step 1 — eliminate the arithmetic. observed_x100 against expected_x100 and flows_est. Chapter 15.1 §13 and Section 8: a ratio of 2.13 on four members with four flows is the expected value. If distribution_suspect is low, the aggregate is as balanced as it can be and the remaining conversation is about the workload, not the hash.

Step 2 — eliminate the elephant. heaviest_octets against a member's capacity. A single flow above one member's share makes balance impossible — Section 19's rejected property — and no configuration change helps. This is checked second because it is the most expensive hypothesis to investigate and the cheapest to eliminate.

Step 3 — check the degradation. degraded_pct_x100. A 5-tuple hash on traffic that is 30% non-IP puts all of that 30% on one member, and the symptom is an imbalance with a healthy flow count. The remedy is a selection change, not a hash change.

Step 4 — check the selection. field_sel_t against what is behind the aggregate. MAC-only behind a router is one flow — Section 4's second row — and it is the single most common real cause. flows_est reading 1 or 2 on a busy aggregate is this, and it is diagnostic on its own.

Step 5 — check the seed. cfg_seed_is_default on every switch in the path. Identical hashes preserve an imbalance across hops, so an aggregate three hops in can be unbalanced by a decision made at the first, and its own configuration is blameless.

And the separate question, asked only during a failover: moved_pct_x100 against ideal_pct_x100. A three-fold excess is a reduction-mode problem and costs 96 octets to fix — and it is invisible at every other moment, which is why it has to be asked deliberately rather than discovered.

22. Common Misconceptions

1 — "The LAG is unbalanced, so the hash is bad."

The wrong model: an uneven distribution implies a defective hash function.

What it costs: hash-policy changes that do nothing, repeatedly. Section 8: four flows into four members has an expected maximum of 2.13 with a perfect hash — 47.0% usable — and at sixteen flows the expected ratio is still 1.53.

The corrected model: compare the observed imbalance against the expected imbalance for the measured flow count. flows_est costs 512 octets and converts the question from a debate into a comparison. A ratio of 2.13 at four flows is arithmetic; the same ratio at four thousand is a defect.

2 — "A modulo is the natural way to pick a member."

The wrong model: hash % N is obvious, correct and free.

What it costs: 74.9% of flows relocated when a four-member aggregate loses one member, against a physical requirement of 25% — and 93.7% against 6.2% on a sixteen-member one. Every unnecessary move pays Chapter 15.1 §8's drain and an ordering risk.

The corrected model: put a fixed bucket count between the hash and the members. The flow's bucket never changes because 256 never changes, and a member's departure rewrites only the buckets that named it. 96 octets, and the excess disappears entirely.