Bonding Links and Failover

Chapter 14.4 subdivided one link into eight independently stoppable classes. This module does the opposite: it presents several physical links as one logical link, and the two problems turn out to be the same problem read in different directions.

The obvious reason to bond links is capacity. Four 10 Gb/s links instead of one, without buying 40 Gb/s optics or rewiring anything.

The obvious reason is not the real one. Capacity from aggregation is conditional and imperfect — Section 13 shows a four-member LAG delivering 47.2% of its nominal capacity with sixteen conversations on it and nothing wrong. The reason aggregation is deployed almost everywhere is the other one: a link can fail and the logical link survives.

And the whole module is shaped by a single constraint that no standard states as a requirement, because it was never a choice.

Frames of one conversation must arrive in the order they were sent. Ethernet does not reorder. Chapter 12.3's forwarding decision picks one port for one frame; Chapter 12.6's two disciplines both preserve order; every layer above assumes it. The instant a design distributes frames across four links with different queue depths, it has built a reordering machine — and everything Module 15 does is a response to that.

1. Scope — What This Chapter Owns

This chapter owns the aggregate as an object: what bonding gives and does not give, the ordering constraint, member selection at the coarsest level, link-down detection and its true cost, redistribution, and what a failover does to Chapter 12.5's forwarding table.

It does not own the distribution function. How a frame chooses a member — field selection, the hash, modulo against masking, and the rebalance when the member count changes — is Chapter 15.2. This chapter treats the distributor as a black box with one property: it is deterministic per flow. Section 2 explains why that property, and not any other, is the one that matters.

It does not own negotiation. Which links are members, whether both ends agree, and how a mis-cable is caught is Chapter 15.3. This chapter assumes a statically configured aggregate — which is a real deployment, and Section 16 shows what it cannot detect.

It does not own the forwarding table. Chapter 12.5 built it: 8192 entries, 2048 sets, four ways, a 16.4 µs full sweep. Section 11 asks what happens to those entries when a member disappears and finds that the answer depends on a field-width decision made long before aggregation was configured.

And it does not own flow control. Chapter 14.2 and Chapter 14.4 both operate on a link, and an aggregate is not one. Section 16's callout follows what that does to a PAUSE frame arriving on one member of four.

2. The Constraint That Shapes Everything

Put this in front of everything else, because every subsequent design decision is a consequence of it and none of them makes sense without it.

Frames belonging to one conversation must be delivered in the order they were transmitted.

Ethernet has never had a sequence number, a reassembly buffer or a reorder window. Chapter 5.1's frame has no field that would support one. The ordering guarantee is structural: one destination has one entry in Chapter 12.5's table, that entry names one port, and one port is a FIFO. Order is preserved because there was never an opportunity to break it.

Aggregation creates the opportunity.

And the amount of reordering available is not small. Two members whose egress queues differ by 512 cells of Chapter 14.1's 128-octet buffer differ by 65 536 octets of backlog — 524 µs at 1 Gb/s. A frame sent second on the empty member arrives half a millisecond before a frame sent first on the full one.

So the distributor cannot be free to place frames anywhere. It has exactly one obligation, and it is not fairness:

Every frame of one conversation must go to the same member.

That single rule is the reason the module looks the way it does. It is why distribution is per flow rather than per frame — Chapter 15.2 §2. It is why a member's failure moves flows rather than rebalancing traffic. It is why one conversation can never exceed one member's rate, which is Section 13's finding and the elephant flow of Chapter 15.2 §14. And it is why Section 9's failover cost has a term that has nothing to do with link speed.

3. RTL 1 — The Aggregator's Member Table

An aggregate is a small table and a mask. The interesting decisions are which fields it holds and which of them the forwarding path is allowed to see.

// -----------------------------------------------------------------------
// lag_pkg -- shared types for link aggregation.
// -----------------------------------------------------------------------
package lag_pkg;

  localparam int NUM_PORTS   = 24;
  localparam int NUM_LAGS    = 8;
  localparam int MAX_MEMBERS = 8;

  localparam int PORT_W   = $clog2(NUM_PORTS);   // 5
  localparam int LAG_W    = $clog2(NUM_LAGS);    // 3
  localparam int MEMBER_W = $clog2(MAX_MEMBERS); // 3

  // A forwarding-table entry names either a physical port or an
  // aggregate. Section 11 shows this one bit deciding the whole cost
  // of a failover.
  typedef struct packed {
    logic                is_lag;
    logic [PORT_W-1:0]   port;   // valid when is_lag == 0
    logic [LAG_W-1:0]    lag;    // valid when is_lag == 1
  } egress_id_t;

  typedef logic [MAX_MEMBERS-1:0] member_vec_t;

  // A member is configured, and separately it is usable. The two are
  // different and conflating them is section 21's first misconception.
  typedef struct packed {
    member_vec_t configured;   // an operator put it in the aggregate
    member_vec_t collecting;   // we accept frames from it
    member_vec_t distributing; // we may send frames on it
  } lag_state_t;

endpackage

// -----------------------------------------------------------------------
// lag_member_table -- per aggregate: which ports belong, which are
// usable, and the compacted index the distributor works against.
//
// The distributor must see a DENSE index space. A four-member LAG whose
// second member is down must present three members numbered 0,1,2 --
// not a sparse mask -- or 15.2's modulo lands on a dead port.
// -----------------------------------------------------------------------
module lag_member_table
  import lag_pkg::*;
(
  input  logic                    clk,
  input  logic                    rst_n,

  // Configuration, written by management.
  input  logic                    cfg_we,
  input  logic [LAG_W-1:0]        cfg_lag,
  input  member_vec_t             cfg_configured,
  input  logic [PORT_W-1:0]       cfg_port [MAX_MEMBERS],

  // Live per-member usability, from section 6's monitor.
  input  member_vec_t             link_up   [NUM_LAGS],

  // Lookup, one aggregate per cycle.
  input  logic [LAG_W-1:0]        q_lag,
  output logic [MEMBER_W:0]       q_active_count,
  output logic [PORT_W-1:0]       q_port_of [MAX_MEMBERS],
  output member_vec_t             q_active,
  output logic                    q_lag_down       // no usable member
);

  member_vec_t       configured_q [NUM_LAGS];
  logic [PORT_W-1:0] port_q       [NUM_LAGS][MAX_MEMBERS];

  always_ff @(posedge clk or negedge rst_n) begin
    int l, m;
    if (!rst_n) begin
      for (l = 0; l < NUM_LAGS; l++) begin
        configured_q[l] <= '0;
        for (m = 0; m < MAX_MEMBERS; m++) port_q[l][m] <= '0;
      end
    end else if (cfg_we) begin
      configured_q[cfg_lag] <= cfg_configured;
      for (m = 0; m < MAX_MEMBERS; m++) port_q[cfg_lag][m] <= cfg_port[m];
    end
  end

  // Active = configured AND up. The compaction below turns the sparse
  // mask into a dense list the distributor can index with a modulo.
  member_vec_t active;
  assign active = configured_q[q_lag] & link_up[q_lag];

  always_comb begin
    int m;
    logic [MEMBER_W:0] n;
    n = '0;
    for (m = 0; m < MAX_MEMBERS; m++) q_port_of[m] = '0;

    for (m = 0; m < MAX_MEMBERS; m++) begin
      if (active[m]) begin
        q_port_of[n] = port_q[q_lag][m];
        n = n + 1'b1;
      end
    end
    q_active_count = n;
  end

  assign q_active     = active;
  assign q_lag_down   = (active == '0);

endmodule

Classification: a small configuration store with a combinational compaction. No datapath, one cycle, read every frame.

What it teaches: that the distributor needs a dense index and the configuration is sparse, and the compaction between them is where a failover physically happens. A four-member LAG whose member 1 fails becomes a three-member LAG with members numbered 0, 1, 2 — and member 2 of the new numbering is member 3 of the old. Every flow whose hash landed on index 2 has just moved to a different physical port, which Section 9 prices and Chapter 15.2 §12 examines properly.

And it teaches the difference between collecting and distributing, which is the single most useful distinction in the whole module. A member may be accepted from and not sent on, or the reverse, and the two must be settable independently because a member being added should start collecting before it starts distributing. If it distributes first, frames arrive at a neighbour that is not yet accepting them and are discarded — a link that comes up and loses traffic for the duration of the disagreement.

Deliberately simplified: link_up arrives as a wire with no timing model, so this module sees a member fail instantaneously. Section 7 shows the real detection taking anywhere from 10 µs to 90 seconds depending entirely on the mechanism, and Section 9's cost is dominated by that interval rather than by anything in this module.

Production implication: q_lag_down is the aggregate having no usable member, and it must be distinct from the aggregate not existing. A forwarding entry pointing at a down aggregate is not stale — the aggregate is real and configured — and frames for it should be discarded and counted rather than flooded, because flooding an entire aggregate's traffic across every other port is Chapter 12.4's 23× amplification triggered by a cable somebody unplugged.

4. What Bonding Actually Gives You

Four claims are made for aggregation. Two are true, one is conditional and one is false, and the false one is the one on the datasheet.

The fourth row follows directly from Section 2. All frames of one conversation take one member, so one conversation's ceiling is one member's rate, whatever N is. A 4 × 10 Gb/s aggregate is not a 40 Gb/s link for anybody; it is four 10 Gb/s links with a shared name.

And the difference matters commercially, because the two are sold as the same thing. A storage array attached over a 4 × 10 aggregate, running one large transfer, moves it at 10 Gb/s — and the switch's aggregate counter reads 25% utilised with three members nearly idle. Nothing is broken and nothing can be tuned.

The availability claim, by contrast, is unconditional and large:

Which is why aggregates are usually sized for redundancy rather than for throughput, and why the common configuration is two members carrying well under 50% of one member's capacity. The second link is not there to add bandwidth. It is there so the first one can fail.

5. RTL 2 — Member Selection

The hash itself belongs to the next chapter. What belongs here is the wrapper: turning an egress decision that names an aggregate into one that names a port, and the two failure cases that wrapper must handle.

// -----------------------------------------------------------------------
// lag_member_selector -- resolves an egress_id_t to a physical port.
//
// The hash itself is opaque here and is built in 15.2. The contract it
// must satisfy is exactly one property: the same flow_key always yields
// the same value for a given member count. Everything else -- field
// choice, polynomial, uniformity -- is 15.2's problem.
// -----------------------------------------------------------------------
module lag_member_selector
  import lag_pkg::*;
(
  input  logic                clk,
  input  logic                rst_n,

  input  logic                req_valid,
  input  egress_id_t          req_egress,
  input  logic [15:0]         req_flow_hash,   // opaque, from 15.2

  input  logic [MEMBER_W:0]   lag_active_count,
  input  logic [PORT_W-1:0]   lag_port_of [MAX_MEMBERS],
  input  logic                lag_down,

  output logic                out_valid,
  output logic [PORT_W-1:0]   out_port,
  output logic                out_discard,     // aggregate has no member
  output logic [31:0]         c_via_lag,
  output logic [31:0]         c_lag_down_drop,
  output logic [31:0]         c_per_member [MAX_MEMBERS]
);

  logic [MEMBER_W-1:0] sel;

  // Modulo by the ACTIVE count, not by the configured count. 15.2
  // section 7 shows why this one choice decides how much traffic moves
  // when a member fails.
  always_comb begin
    sel = '0;
    if (lag_active_count != 0)
      sel = MEMBER_W'(req_flow_hash % {9'd0, lag_active_count});
  end

  always_ff @(posedge clk or negedge rst_n) begin
    int m;
    if (!rst_n) begin
      out_valid <= 1'b0; out_port <= '0; out_discard <= 1'b0;
      c_via_lag <= '0; c_lag_down_drop <= '0;
      for (m = 0; m < MAX_MEMBERS; m++) c_per_member[m] <= '0;
    end else begin
      out_valid   <= 1'b0;
      out_discard <= 1'b0;

      if (req_valid) begin
        if (!req_egress.is_lag) begin
          // An ordinary port. Nothing to resolve.
          out_valid <= 1'b1;
          out_port  <= req_egress.port;
        end else if (lag_down) begin
          // The aggregate exists and has no usable member. Discard and
          // count -- do NOT fall back to flooding, which would turn one
          // unplugged cable into 12.4's 23x amplification.
          out_discard     <= 1'b1;
          c_lag_down_drop <= c_lag_down_drop + 1;
        end else begin
          out_valid          <= 1'b1;
          out_port           <= lag_port_of[sel];
          c_via_lag          <= c_via_lag + 1;
          c_per_member[sel]  <= c_per_member[sel] + 1;
        end
      end
    end
  end

endmodule

Classification: a one-cycle resolver sitting between the forwarding decision and the egress queue. Combinational selection, registered output.

What it teaches: that the modulo is taken against the active count and this is a decision rather than an obvious step. Taking it against the configured count keeps a flow's index stable when a member fails — but that index may name a dead port, so the design needs a second mapping from index to live member, and every flow that lands on the dead index moves. Taking it against the active count means the divisor changes, so nearly every flow moves. Neither is free; Chapter 15.2 §12 quantifies both and Section 9 uses the answer.

And it teaches that out_discard on a down aggregate is a deliberate refusal to do the natural thing. The forwarding table said this destination is reachable via LAG 3, LAG 3 has no members, and the tempting fallback is to treat the destination as unknown and flood. That converts one failed aggregate into Chapter 12.4's amplification across every other port — a 24-port switch replicating 23 times, for every frame, for as long as the aggregate is down.

Deliberately simplified: the modulo is written as %, which synthesises into a divider unless the count is a power of two. Production designs either restrict aggregates to power-of-two member counts — losing three-member and five-member aggregates entirely — or use a reciprocal multiply, and Chapter 15.2 §7 shows why that restriction has a second, larger consequence for how much traffic moves at a failover.

Production implication: c_per_member[m] is a frame counter and the thing an operator needs is an octet counter. A member carrying many small frames and one carrying few large ones look balanced by frame count and are not, and Chapter 13.2 §4's callout made exactly this point about the PCP histogram. Section 13's imbalance argument is entirely about bandwidth, and a frame counter cannot see it.

6. RTL 3 — Link-Down Detection

This module is short and it is the one that dominates Section 9's cost. Everything expensive about a failover happens before this module produces its output.

// -----------------------------------------------------------------------
// lag_link_monitor -- decides whether a member is usable.
//
// Three sources, three latencies spanning four orders of magnitude.
// Section 7's table is this module's specification.
// -----------------------------------------------------------------------
module lag_link_monitor
  import lag_pkg::*;
#(
  parameter int DEBOUNCE_CYCLES = 5000   // 10 us at 500 MHz
)(
  input  logic       clk,
  input  logic       rst_n,

  // Source 1 -- PHY, asynchronous, fastest. 3.8's link status.
  input  logic       phy_link_up,
  // Source 2 -- PCS sync, 3.5's 64B/66B block lock.
  input  logic       pcs_block_lock,
  // Source 3 -- protocol liveness, 15.3's LACP timeout. Slowest by
  // four orders of magnitude and the only one that catches a
  // one-way failure.
  input  logic       proto_alive,
  input  logic       proto_enabled,

  output logic       member_up,
  output logic [1:0] down_reason,      // 0 none, 1 phy, 2 pcs, 3 proto
  output logic [31:0] c_transitions,
  output logic [31:0] c_flaps          // up->down->up inside the window
);

  logic [15:0] debounce;
  logic        raw_up, up_q;

  assign raw_up = phy_link_up && pcs_block_lock &&
                  (!proto_enabled || proto_alive);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      debounce <= '0; up_q <= 1'b0;
      c_transitions <= '0; c_flaps <= '0;
      down_reason <= 2'd0;
    end else begin
      // A DOWN transition is taken IMMEDIATELY. A member that has
      // failed must stop being chosen before anything is debounced,
      // because every frame sent at it in the meantime is lost.
      if (up_q && !raw_up) begin
        up_q          <= 1'b0;
        debounce      <= '0;
        c_transitions <= c_transitions + 1;
        if      (!phy_link_up)    down_reason <= 2'd1;
        else if (!pcs_block_lock) down_reason <= 2'd2;
        else                      down_reason <= 2'd3;
      end
      // An UP transition is debounced. A flapping member that is
      // repeatedly readmitted moves flows on every transition, and
      // section 9's cost is paid each time.
      else if (!up_q && raw_up) begin
        if (debounce == DEBOUNCE_CYCLES-1) begin
          up_q          <= 1'b1;
          debounce      <= '0;
          down_reason   <= 2'd0;
          c_transitions <= c_transitions + 1;
        end else begin
          debounce <= debounce + 1'b1;
          if (debounce == 16'd0) c_flaps <= c_flaps + 1;
        end
      end else begin
        debounce <= '0;
      end
    end
  end

  assign member_up = up_q;

endmodule

Classification: an asymmetric debouncer with a three-way reason code.

What it teaches: that down is immediate and up is debounced, and the asymmetry is not a preference. Every frame sent at a failed member between the failure and its detection is lost, so detection must be as fast as the evidence allows. A member returning is a different situation entirely — nothing is lost by waiting, and readmitting a flapping member costs Section 9's redistribution every time it flaps. A symmetric debouncer is wrong in one direction whichever value it takes.

And it teaches why proto_alive exists alongside two physical sources. A PHY reports the local receiver's state. A member whose transmit direction has failed and whose receive direction has not is up by both physical measures and carries nothing — the classic one-way fibre failure, a broken TX strand or a dirty connector on one side. Only a protocol that expects to hear from the far end detects it, which is Chapter 15.3's LACP, and it is the strongest argument for running it.

Deliberately simplified: DEBOUNCE_CYCLES is a constant, so a member that flaps at a period just longer than the debounce is readmitted every time. Production designs use an exponential hold-down — each successive flap increases the wait — which is the standard response and costs one counter and a shift.

Production implication: down_reason decides who is called. Reason 1 is a cable or an optic. Reason 2 is a signal-integrity problem — Chapter 3.7's FEC is failing to correct, which is a physical-layer investigation. Reason 3 is a far-end configuration or a one-way failure, which is a different team entirely. One "member down" counter sends all three to the same place, and two of the three will be wrong.

7. Detection Time Is Not a Property of the Line Rate

The question "how fast does a LAG fail over" has an answer that surprises people, and the surprise is that the link's speed does not appear in it.

Four orders of magnitude, and the line rate appears in exactly one row — the PCS one, which is the fastest and least used.

The reason polling dominates in practice is architectural rather than technical. Chapter 4.5's MDIO is a management bus, not a datapath signal, and a switch that learns about link state by reading a PHY register learns about it at the polling interval regardless of how quickly the PHY knew. The information was available in 10 µs and is consumed 100 ms later.

And what does scale with line rate is the damage:

In maximum frames, the top-left and bottom-right of that table are 1 frame and 610 million.

So the engineering question is never "how fast is the link" — it is "which of the six mechanisms is actually in the path", and the answer is usually the slowest one that is enabled. A design with PHY interrupts wired, PCS lock monitored and LACP running detects in 10 µs and the LACP timeout never fires. A design that polls MDIO at one second detects in one second, and the 100 Gb/s member it is monitoring has discarded 12.5 GB.

8. RTL 4 — Redistributing a Failed Member

Detection produces a mask change. This module is what happens next, and its correctness condition is about frames that are already in flight.

// -----------------------------------------------------------------------
// lag_redistributor -- sequences a member's removal so that no frame of
// a moved flow can overtake an earlier frame of the same flow.
//
// The naive implementation -- change the mask and carry on -- reorders.
// A flow moving from a member with 400 cells queued to one with 10
// delivers its next frame 15.6 us ahead of its previous one at 1 Gb/s.
// -----------------------------------------------------------------------
module lag_redistributor
  import lag_pkg::*;
#(
  parameter int DRAIN_LIMIT = 500000    // 1 ms at 500 MHz
)(
  input  logic              clk,
  input  logic              rst_n,

  input  logic              member_failed,
  input  logic [MEMBER_W-1:0] failed_member,

  // Per-member egress occupancy, from 14.1's queue model.
  input  logic [15:0]       member_occ [MAX_MEMBERS],
  input  member_vec_t       member_active,

  output member_vec_t       distributing,   // what the selector may use
  output logic              quiescing,      // a move is in progress
  output logic [31:0]       c_moves,
  output logic [31:0]       c_drain_timeout,
  output logic [31:0]       last_drain_cycles
);

  typedef enum logic [1:0] { S_RUN, S_STOP, S_DRAIN, S_DONE } state_e;
  state_e      state;
  logic [31:0] drain_cnt;
  member_vec_t dist_q;

  // The maximum occupancy across the REMAINING members. A moved flow
  // must not be placed on a member until the queue it left has drained
  // past the point where overtaking is possible.
  logic [15:0] max_remaining_occ;
  always_comb begin
    int m;
    max_remaining_occ = '0;
    for (m = 0; m < MAX_MEMBERS; m++)
      if (dist_q[m] && (member_occ[m] > max_remaining_occ))
        max_remaining_occ = member_occ[m];
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      state <= S_RUN; dist_q <= '1; drain_cnt <= '0;
      c_moves <= '0; c_drain_timeout <= '0; last_drain_cycles <= '0;
    end else begin
      unique case (state)
        S_RUN: begin
          dist_q <= member_active;
          if (member_failed) begin
            // Step 1 -- stop distributing to the failed member
            // immediately. Frames already queued on it are lost; there
            // is nothing to be done about them and section 9 counts them.
            dist_q[failed_member] <= 1'b0;
            drain_cnt <= '0;
            state     <= S_STOP;
            c_moves   <= c_moves + 1;
          end
        end

        S_STOP: begin
          // Step 2 -- hold the moved flows until the remaining members'
          // queues are shallow enough that a newly placed frame cannot
          // overtake one placed before the move.
          state <= S_DRAIN;
        end

        S_DRAIN: begin
          drain_cnt <= drain_cnt + 1;
          if (max_remaining_occ == 16'd0) begin
            last_drain_cycles <= drain_cnt;
            state <= S_DONE;
          end else if (drain_cnt == DRAIN_LIMIT) begin
            // A congested aggregate may never fully drain. Proceed and
            // record it: a bounded reordering risk is preferable to an
            // unbounded outage, and the counter says which was chosen.
            c_drain_timeout   <= c_drain_timeout + 1;
            last_drain_cycles <= drain_cnt;
            state             <= S_DONE;
          end
        end

        S_DONE: begin
          dist_q <= member_active;
          state  <= S_RUN;
        end
      endcase
    end
  end

  assign distributing = dist_q;
  assign quiescing    = (state != S_RUN);

endmodule

Classification: a four-state sequencer that trades a bounded outage against a reordering risk, and reports which it took.

What it teaches: that a failover reorders unless something prevents it, and preventing it costs time. A flow moving from a member with 400 queued cells to one with 10 has its next frame delivered 15.6 µs ahead of its previous one at 1 Gb/s, which is exactly the duplicate-acknowledgement generator Section 2's callout described. The drain state exists solely to close that window.

And it teaches that c_drain_timeout is a design admitting which of two bad things it chose. A congested aggregate may never reach zero occupancy, so waiting for a clean drain is an unbounded outage. The module waits 1 ms, then proceeds and says so — and an operator seeing c_drain_timeout rising knows that the reordering risk was taken and can correlate it with the transport's retransmission counters.

Deliberately simplified: the drain condition is max_remaining_occ == 0, which is stricter than necessary. The real condition is that the queue a flow left has drained, not that every remaining queue is empty, so a per-flow-aware design releases much sooner. Tracking that requires knowing which flows moved, which requires per-flow state, which is precisely what a hash-based distributor exists to avoid — the trade is stated here and taken in Chapter 15.2 §13.

Production implication: last_drain_cycles converted to time is the number an operator needs and never has. A failover that takes 1 ms of drain on top of 1 s of detection is a 1.001 s event dominated entirely by detection; one that takes 1 ms on top of a 10 µs interrupt is a 1.01 ms event dominated entirely by drain. The two have completely different remedies and the same symptom, which is a brief loss of traffic.

9. What a Failover Costs, Derived

Four terms, and they differ by five orders of magnitude, so the sum is always the largest one.

Term 2 is a single clock and is never the answer. Term 3 is bounded by a parameter. Term 1 spans four orders of magnitude and is chosen by an architectural decision made elsewhere. And term 4 is either zero or very large, decided by one bit in the forwarding table's entry format.

In octets lost, at the two ends of the range:

And the drain term is worth isolating, because it is the only one this chapter's RTL controls. At 100 Gb/s, 1 ms of drain is 12.5 MB — which is larger than a PHY-interrupt detection's entire loss. On a fast-detecting design the ordering hold is the dominant cost of the failover, and on a polled design it is a rounding error.

Which reverses the intuition about where to optimise. A design polling at one second should wire the PHY interrupt and not think about the drain. A design already detecting in 10 µs should shorten the drain — by tracking which flows moved rather than waiting for every queue to empty, at the cost of per-flow state.

10. RTL 5 — The MAC Table After a Failover

This module exists only in designs that made the wrong choice in Section 11, and it is presented because those designs are common.

// -----------------------------------------------------------------------
// lag_mac_table_purge -- sweeps 12.5's forwarding table removing every
// entry that points at a failed member.
//
// A table storing egress_id_t with is_lag = 1 needs none of this. A
// table storing a physical port number needs all of it, and the sweep
// is the cheap part.
// -----------------------------------------------------------------------
module lag_mac_table_purge
  import lag_pkg::*;
#(
  parameter int NUM_SETS = 2048,
  parameter int WAYS     = 4
)(
  input  logic       clk,
  input  logic       rst_n,

  input  logic       start,
  input  logic [PORT_W-1:0] dead_port,

  // Table access, arbitrated against the lookup path.
  output logic       sweep_req,
  output logic [$clog2(2048)-1:0] sweep_set,
  input  logic       sweep_gnt,
  input  logic       entry_valid   [WAYS],
  input  logic [PORT_W-1:0] entry_port [WAYS],
  output logic [WAYS-1:0]   entry_clear,

  output logic       busy,
  output logic [31:0] c_purged,
  output logic [31:0] c_sweeps,
  output logic [31:0] last_sweep_cycles
);

  logic [$clog2(2048)-1:0] set_idx;
  logic [31:0]             cycles;

  always_ff @(posedge clk or negedge rst_n) begin
    int w;
    if (!rst_n) begin
      busy <= 1'b0; set_idx <= '0; cycles <= '0;
      c_purged <= '0; c_sweeps <= '0; last_sweep_cycles <= '0;
      entry_clear <= '0;
    end else begin
      entry_clear <= '0;

      if (!busy) begin
        if (start) begin
          busy     <= 1'b1;
          set_idx  <= '0;
          cycles   <= '0;
          c_sweeps <= c_sweeps + 1;
        end
      end else begin
        cycles <= cycles + 1;

        // The sweep YIELDS to the lookup path. A forwarding table that
        // stalls for a sweep stalls the switch, and 12.5 section 16's
        // budget has no room for it.
        if (sweep_gnt) begin
          for (w = 0; w < WAYS; w++) begin
            if (entry_valid[w] && (entry_port[w] == dead_port)) begin
              entry_clear[w] <= 1'b1;
              c_purged       <= c_purged + 1;
            end
          end

          if (set_idx == NUM_SETS-1) begin
            busy              <= 1'b0;
            last_sweep_cycles <= cycles;
          end else begin
            set_idx <= set_idx + 1'b1;
          end
        end
      end
    end
  end

  assign sweep_req = busy;
  assign sweep_set = set_idx;

endmodule

Classification: a background table sweep that yields to the lookup path. It is a maintenance engine, not a datapath.

What it teaches: that the sweep is cheap and its consequence is not. 2048 sets at one set per granted cycle is 2048 cycles — 4.1 µs at 500 MHz, and even yielding to the lookup path on most cycles it completes in well under 100 µs. Then every purged address is an unknown unicast and Chapter 12.4 takes over: 4096 purged addresses on a 24-port switch is 143 MB of flooded traffic, replicated 23 times, until each address is relearned by Chapter 12.2's source-address learning on a return frame.

And it teaches that yielding is mandatory rather than polite. Chapter 12.5 §16 established the lookup budget at four cycles inside Chapter 12.1 §12's 28 ns per frame; a sweep that took the table for 2048 consecutive cycles would stall forwarding for 4.1 µs, which at 100 Gb/s is 51 kB of arriving frames with nowhere to go.

Deliberately simplified: the sweep compares entry_port against one dead port. A failure that takes down several members at once — a line card, a shared optic module — needs a mask comparison, and running the sweep once per dead port multiplies both the sweep and the flood.

Production implication: c_purged is the number of addresses that will be flooded, which makes it the best available predictor of the failover's real impact. An operator watching c_purged at 4096 knows to expect a flood; watching it at 12 knows the failover will be invisible. The counter costs nothing and turns Section 9's fourth term from a category into a quantity.

11. Chapter 12.5's Table Now Points at a Member That Is Gone

One bit in the forwarding-table entry decides whether Section 10's module is needed at all, and the bit is usually chosen before anybody has thought about aggregation.

Rows four to six are the entire argument and they are decided by row one: one extra bit per entry.

Chapter 13.4 §8 recomputed Chapter 12.5's table at a wider key and found the general rule — widening what you store is priced in area and nothing else. This is that rule again with a much better return: 8192 entries × 1 bit is 1 KiB, against a 80 KiB table, and it removes a 143 MB flood.

And the learning side is where designs get it wrong, because it is not automatic. Chapter 12.2 learns the ingress port, which for a frame arriving on member 2 of LAG 3 is port 7. A design that stores port 7 has just installed exactly the entry the aggregate-id scheme was meant to avoid, and it will do so on every learned address regardless of how the entry format was designed.

So the aggregate-id design has two parts and both are required:

One — the entry stores egress_id_t with is_lag set. Two — learning maps an ingress port to its aggregate before writing the entry, so a frame on port 7 installs "LAG 3" and not "port 7".

Miss the second and the first is decoration. The table looks like an aggregate-aware table, every entry contains a physical port, and a member failure purges and floods exactly as if the feature had never been implemented. Section 19's scenario 44 is that configuration.

12. RTL 6 — Aggregator Bandwidth Accounting

An aggregate's counters are the sum of its members' counters, and the sum is wrong in a specific and instructive way.

// -----------------------------------------------------------------------
// lag_bandwidth_accountant -- per-aggregate octet and frame totals,
// per-member breakdown, and the imbalance figure section 13 needs.
//
// The membership can change while this is accumulating, and section 18's
// rejected property is the assumption that it cannot.
// -----------------------------------------------------------------------
module lag_bandwidth_accountant
  import lag_pkg::*;
#(
  parameter int WINDOW_CYCLES = 500000   // 1 ms at 500 MHz
)(
  input  logic              clk,
  input  logic              rst_n,

  input  logic              tx_valid,
  input  logic [MEMBER_W-1:0] tx_member,
  input  logic [13:0]       tx_octets,

  input  member_vec_t       distributing,
  input  logic              membership_changed,

  output logic [47:0]       agg_octets,
  output logic [47:0]       member_octets [MAX_MEMBERS],

  // Windowed view, from which the imbalance is computed.
  output logic [31:0]       win_member_octets [MAX_MEMBERS],
  output logic [15:0]       imbalance_x100,      // max/mean, x100
  output logic              window_spans_change, // the window is not comparable
  output logic [31:0]       c_windows,
  output logic [31:0]       c_dirty_windows
);

  logic [31:0] win_cnt;
  logic [31:0] acc [MAX_MEMBERS];
  logic        dirty;

  always_ff @(posedge clk or negedge rst_n) begin
    int m;
    if (!rst_n) begin
      agg_octets <= '0; win_cnt <= '0; dirty <= 1'b0;
      c_windows <= '0; c_dirty_windows <= '0;
      imbalance_x100 <= '0; window_spans_change <= 1'b0;
      for (m = 0; m < MAX_MEMBERS; m++) begin
        member_octets[m] <= '0; acc[m] <= '0; win_member_octets[m] <= '0;
      end
    end else begin
      if (tx_valid) begin
        agg_octets              <= agg_octets + tx_octets;
        member_octets[tx_member]<= member_octets[tx_member] + tx_octets;
        acc[tx_member]          <= acc[tx_member] + tx_octets;
      end

      // A membership change poisons the window it lands in. The totals
      // are still correct; the COMPARISON between members is not,
      // because they were not all present for the same interval.
      if (membership_changed) dirty <= 1'b1;

      if (win_cnt == WINDOW_CYCLES-1) begin
        automatic logic [31:0] total, mx;
        automatic int n;
        total = '0; mx = '0; n = 0;
        for (m = 0; m < MAX_MEMBERS; m++) begin
          win_member_octets[m] <= acc[m];
          total = total + acc[m];
          if (distributing[m]) begin
            n = n + 1;
            if (acc[m] > mx) mx = acc[m];
          end
          acc[m] <= '0;
        end

        // imbalance = max / mean = max * n / total, scaled by 100.
        imbalance_x100 <= (total == 0) ? 16'd100
                        : 16'((mx * n * 100) / total);

        window_spans_change <= dirty;
        c_windows           <= c_windows + 1;
        if (dirty) c_dirty_windows <= c_dirty_windows + 1;
        dirty   <= 1'b0;
        win_cnt <= '0;
      end else begin
        win_cnt <= win_cnt + 1'b1;
      end
    end
  end

endmodule

Classification: a dual-horizon accumulator — lifetime totals and a windowed view — with a validity flag on the window.

What it teaches: that imbalance_x100 is a ratio of a maximum to a mean and both have to come from the same population. Dividing by n, the count of currently distributing members, is correct only if that count held for the whole window. A member removed halfway through contributed half a window's octets and is not counted in n, so mx × n / total overstates the imbalance by roughly the fraction of the window it was present for.

And it teaches why window_spans_change is an output rather than a suppression. The tempting design discards a poisoned window. This one publishes it and marks it, because a failover is exactly when an operator is looking, and a graph with a hole where the interesting event was is worse than one with a flagged point.

Deliberately simplified: imbalance_x100 uses a max against a mean, which is the right shape and a coarse instrument. Production designs also report the minimum — a max/mean of 1.3 with every other member equal is a different situation from one member idle — and the two together cost one more comparator.

Production implication: member_octets are lifetime totals and they do not decrease when a member is removed, which is correct and confuses everybody. A member that failed an hour ago still shows its accumulated octets, so a naive dashboard computing "share of aggregate" from lifetime counters attributes traffic to a link that has been dark for an hour. The windowed view exists for exactly this reason, and Section 18's rejected property is the assertion that the lifetime totals still add up across a membership change.

13. Why N Links Is Not N× Throughput

Section 4 called the capacity claim conditional. This is the condition, and it is about the number of conversations rather than about anything in the hardware.

Section 2's rule places all frames of one flow on one member. So F flows land on N members by whatever the distributor decides, and even a perfect distributor is throwing F balls into N bins. The aggregate's usable capacity is set by the fullest bin, because that member saturates first.

For a four-member LAG, with a perfectly uniform hash:

Four flows on a four-member LAG use 47.2% of it, and the hash is perfect. Not misconfigured, not unlucky, not biased — the arithmetic of throwing four balls into four bins gives an expected maximum of 2.12.

And this is the same arithmetic as Chapter 12.5 §9's hash table, where 4096 addresses into 2048 four-way sets produced a 1.86% overflow probability and 84.6% usable capacity. Same distribution, different consequence: there a full set evicted an entry, here a full member throttles a flow. Chapter 15.2 §8 does the recomputation properly and derives the variance rather than quoting the mean.

The practical reading has three parts.

A LAG carrying a few large flows is a bad LAG and no amount of hash tuning fixes it, because the distribution is the problem and the distribution is forced by Section 2.

A LAG carrying thousands of small flows is nearly ideal — 96.9% at 4096 flows — which is why aggregates in a data-centre core work well and aggregates to a single storage array do not.

And two members are never enough to hide a hot flow. A two-member aggregate carrying one dominant conversation puts all of it on one member, so the aggregate reads 50% utilised with one link saturated — the worst-looking case of Section 4's fourth row, and the most common aggregate there is.

And the arithmetic is the same in both directions of the aggregate, independently. Chapter 15.2 §7's callout shows why: each end hashes on its own, so a four-flow conversation set can be 47.2% efficient outbound and differently unlucky inbound, and neither figure predicts the other.

And the flow count is the diagnostic, not the imbalance. An operator seeing imbalance_x100 at 212 on a four-member LAG should count the flows before touching anything: at four flows that is the expected value.

14. RTL 7 — LAG Telemetry

Six numbers, and the useful ones are the two that describe the aggregate's inputs rather than its outputs.

// -----------------------------------------------------------------------
// lag_telemetry -- what an operator needs to decide whether an
// aggregate is behaving, and to tell an unbalanced LAG from a LAG
// carrying few flows.
// -----------------------------------------------------------------------
module lag_telemetry
  import lag_pkg::*;
#(
  parameter int FLOW_TRACK_BITS = 12    // 4096-bucket flow-presence sketch
)(
  input  logic              clk,
  input  logic              rst_n,

  input  logic              tx_valid,
  input  logic [15:0]       tx_flow_hash,
  input  logic [MEMBER_W-1:0] tx_member,

  input  logic [15:0]       imbalance_x100,
  input  logic [MEMBER_W:0] active_count,
  input  logic [MEMBER_W:0] configured_count,
  input  logic [31:0]       detect_cycles,     // section 6 to section 8
  input  logic              window_tick,

  output logic [15:0]       distinct_flows_est,
  output logic [15:0]       expected_imbalance_x100,
  output logic              imbalance_is_expected,
  output logic [15:0]       redundancy_pct,
  output logic [31:0]       detect_us,
  output logic [31:0]       c_failovers
);

  // A presence sketch, not a flow table: one bit per hash bucket,
  // cleared each window. Popcount estimates the distinct flow count
  // well enough to interpret the imbalance, at 512 octets of state.
  logic [(1<<FLOW_TRACK_BITS)-1:0] seen;
  logic [15:0] popc;

  always_comb begin
    int i;
    popc = '0;
    for (i = 0; i < (1<<FLOW_TRACK_BITS); i++) popc = popc + 16'(seen[i]);
  end

  // E[max]/mean for F balls in N bins, from a small lookup indexed by
  // flows-per-member. Section 13's table, in hardware.
  function automatic logic [15:0] expected_ratio(input logic [15:0] f,
                                                 input logic [MEMBER_W:0] n);
    logic [15:0] fpm;
    fpm = (n == 0) ? 16'd0 : (f / 16'(n));
    if      (fpm <  16'd2)   return 16'd212;
    else if (fpm <  16'd4)   return 16'd177;
    else if (fpm <  16'd16)  return 16'd153;
    else if (fpm <  16'd64)  return 16'd126;
    else if (fpm <  16'd256) return 16'd113;
    else                     return 16'd106;
  endfunction

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      seen <= '0; distinct_flows_est <= '0;
      expected_imbalance_x100 <= 16'd100;
      imbalance_is_expected <= 1'b1;
      redundancy_pct <= '0; detect_us <= '0; c_failovers <= '0;
    end else begin
      if (tx_valid) seen[tx_flow_hash[FLOW_TRACK_BITS-1:0]] <= 1'b1;

      if (window_tick) begin
        distinct_flows_est      <= popc;
        expected_imbalance_x100 <= expected_ratio(popc, active_count);
        // A 20% tolerance around the expectation. Outside it, the
        // distribution is the problem; inside it, the flow count is.
        imbalance_is_expected   <=
          (imbalance_x100 <= ((expected_ratio(popc, active_count) * 12) / 10));
        seen <= '0;
      end

      redundancy_pct <= (configured_count == 0) ? 16'd0
                      : 16'((100 * active_count) / configured_count);
      detect_us      <= detect_cycles / 500;   // 500 MHz
      if (detect_cycles != 0 && $past(detect_cycles) == 0)
        c_failovers <= c_failovers + 1;
    end
  end

endmodule

Classification: an estimator, not a counter. It produces interpretations rather than raw events.

What it teaches: that an imbalance figure is uninterpretable without a flow count, and the flow count is cheap. Section 13's table gives the expected max-to-mean ratio as a function of flows per member; a 4096-bit presence sketch — 512 octets — estimates the flow count well enough to use it. imbalance_is_expected then answers the operator's real question, which is not "is this LAG balanced" but "is this LAG as balanced as it could possibly be".

And it teaches that detect_us exists because Section 7's four orders of magnitude are otherwise invisible. A design that fails over in 10 µs and one that takes 1 s look identical afterwards: the aggregate is up, traffic is flowing, and the only difference is 12.5 GB that nobody counted. Measuring the interval turns an architectural assumption into a number.

Deliberately simplified: the presence sketch never ages within a window and uses one bit per bucket, so it saturates and under-counts once the flow count approaches the bucket count. At 4096 buckets it is accurate to a few hundred flows and optimistic beyond that — which is acceptable because Section 13's curve is flat there: anything above 1024 flows per four members is above 93.9%.

Production implication: redundancy_pct is the number an availability requirement is actually written against, and it is not the same as "the aggregate is up". A four-member LAG running on one member is up, carrying traffic, and one failure from an outage — redundancy_pct reads 25% and q_lag_down is low. An alarm on aggregate-down fires after the last member fails, which is exactly one failure too late.

15. RTL 8 — Conformance for an Aggregate

The aggregate's promise is narrower than it appears, and the monitor's job is to check the narrow one.

// -----------------------------------------------------------------------
// lag_conformance_monitor -- one bit per aggregate.
//
// It asserts that this device distributed correctly and preserved
// ordering. It does NOT assert that the far end agrees about the
// membership -- that is 15.3's problem and section 16's failure.
// -----------------------------------------------------------------------
module lag_conformance_monitor
  import lag_pkg::*;
(
  input  logic       clk,
  input  logic       rst_n,

  input  logic       cfg_no_members,        // configured empty
  input  logic       cfg_mixed_rate,        // members at different rates
  input  logic       flow_moved_while_busy, // moved with queue non-empty
  input  logic       sent_on_down_member,
  input  logic       learned_physical_port, // section 11's second half
  input  logic       drain_timed_out,

  output logic       conformant,
  output logic [7:0] fault_vector,
  output logic [31:0] c_order_risk,
  output logic [31:0] c_sent_on_down
);

  logic v_empty, v_mixed, v_order, v_down, v_learn, v_drain;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      v_empty <= 1'b0; v_mixed <= 1'b0; v_order <= 1'b0;
      v_down  <= 1'b0; v_learn <= 1'b0; v_drain <= 1'b0;
      c_order_risk <= '0; c_sent_on_down <= '0;
    end else begin
      // Standing properties: wrong from configuration time, not from
      // the first frame. 14.1 section 17 and 13.4 section 14.
      v_empty <= cfg_no_members;
      v_mixed <= cfg_mixed_rate;

      if (flow_moved_while_busy) begin
        v_order      <= 1'b1;
        c_order_risk <= c_order_risk + 1;
      end
      if (sent_on_down_member) begin
        v_down         <= 1'b1;
        c_sent_on_down <= c_sent_on_down + 1;
      end
      if (learned_physical_port) v_learn <= 1'b1;
      if (drain_timed_out)       v_drain <= 1'b1;
    end
  end

  assign conformant   = !(v_empty || v_mixed || v_order ||
                          v_down || v_learn || v_drain);
  assign fault_vector = {2'b00, v_drain, v_learn, v_down,
                         v_order, v_mixed, v_empty};

endmodule

Classification: a fault aggregator with two standing configuration terms and four runtime ones.

What it teaches: that cfg_mixed_rate is a configuration fault and not a capability. An aggregate whose members run at different rates has a distributor that cannot be fair by construction — the hash gives each member an equal share of flows and one member has a tenth of the capacity — so a 1 Gb/s member in a 4 × 10 aggregate saturates at 10% of the load the others carry. 802.1AX requires members to be the same speed and duplex, and the requirement exists for this reason rather than for any electrical one.

And it teaches that learned_physical_port is the check that catches Section 11's half-implemented design. A table that stores egress_id_t and a learning path that writes a physical port produces a switch which looks aggregate-aware and floods at every failover. The bit fires the first time learning writes an entry whose port is a LAG member with is_lag clear, which is a one-comparison check on a path that already has the aggregate lookup available.

Deliberately simplified: flow_moved_while_busy is an input, which assumes something upstream can tell that a flow moved rather than that the mask changed. Detecting the first requires per-flow state, so a real implementation approximates it as "the mask changed while any member's queue was non-empty" — conservative, and it will flag moves that were harmless.

Production implication: conformant here does not mean frames are in order end to end. It means this device did not knowingly create a reordering opportunity. A frame can still be reordered by the far end's distributor, by a mis-cable this device cannot see (Section 16), or by Chapter 12.6's different disciplines on different members — and an aggregate whose conformance bit is high can still deliver out of order for reasons that live at the other end of the cable.

16. Asymmetric Aggregation and the Two-Ended Problem

An aggregate has two ends and each one distributes independently. Nothing in a statically configured LAG makes them agree, and three of the four disagreements are silent.

The fourth row is the one that produces a spanning-tree event rather than a performance problem. A carries frames for one destination across four ports; B, treating them as four separate ports, sees the same source address arriving on four different ports — Chapter 12.2 §9's flapping entry, at the rate the distributor moves flows, which for a busy aggregate is thousands of times a second.

And the third row is legal and asymmetric on purpose. 802.1AX does not require the two ends to use the same distribution function, because the distribution function only has to be consistent with itself. A → B traffic balances one way and B → A the other, so a conversation can use member 1 outbound and member 3 inbound, and both ends are correct.

Which has a consequence for every measurement in this chapter: member_octets on A and member_octets on B describe different distributions of the same conversations, and comparing them is comparing two independent hashes. An operator finding member 2 hot on A and member 4 hot on B has found nothing at all.

The first two rows are what the next chapter but one exists to fix, and the fix is not subtle: exchange the membership, refuse to aggregate links whose partners disagree, and detect the mis-cable directly.

And there is a fourth disagreement, which is deliberate rather than accidental: the two ends need not be the same number of devices.

An aggregate whose members terminate on two different switches — sold as MLAG, vPC or a multi-chassis LAG — is the configuration that makes the availability argument complete. A four-member aggregate on one switch survives three member failures and not one switch failure; split across two chassis it survives either.

It also breaks two things this chapter assumed, and both are worth naming because they are why the feature is proprietary rather than standard.

The first row is handled by synchronising the tables between the chassis — an inter-switch link carrying learned addresses, which is exactly Chapter 12.2's learning promoted to a protocol. The second is handled by making both chassis behave as one for the far end, which requires them to agree on an identity the far end sees.

And the fourth row is simply not handled. A flow moving from a member on switch A to a member on switch B cannot be drained, because A cannot see B's queue depth, so a multi-chassis failover reorders and every implementation accepts that it does. The window is the queue-depth difference across two devices, which is larger than the 524 µs Section 2 computed for one.

Which is the honest summary of MLAG: it converts a member failure and a chassis failure into the same event, at the cost of an ordering guarantee that a single-chassis aggregate could keep. 802.1AX does not standardise it, and the reason is that the inter-chassis state synchronisation has no interoperable definition — every vendor's is different, and two vendors' switches cannot form one.

17. The Cost of Aggregation, Accounted

The state is trivial and one line of it is worth 139 000 times its size.

The third row is the whole table's point. One bit per forwarding entry — 1 KiB — removes Section 10's sweep and the 143 MB flood that follows it, and it is the cheapest structural decision in Module 15.

And it is regularly not taken, for a reason worth naming: the entry format is decided when the forwarding table is designed, which is Chapter 12.5's subject, and aggregation is configured years later by somebody who cannot change it. By the time the cost is visible, the width is fixed in silicon.

Compare it against the two mechanisms this batch has already priced:

Two of the three cost about two kibibytes and one costs six hundred times more, and the difference is Section 5 of the previous chapter: VOQs and aggregation store pointers to things that already exist, and PFC reserves space against a future. Storage is sized by the number of cases; a reservation is sized by the worst case times the number of claimants.

18. Properties Worth Asserting, and One Worth Refusing

The properties divide by what they protect: the membership, the selection, the ordering, the failover sequence, the accounting and the configuration.

Group 1 — membership is well formed.

// P1. Active membership is a subset of configured membership. A member
// cannot be usable without being configured.
property p_active_subset_configured;
  @(posedge clk) disable iff (!rst_n)
  (q_active & ~configured_q[q_lag]) == '0;
endproperty

// P2. The compaction is dense: the first q_active_count entries of
// q_port_of are valid and the rest are not consulted.
property p_compaction_is_dense;
  @(posedge clk) disable iff (!rst_n)
  (q_active_count == n) |-> (q_port_of[n-1] inside {port_q[q_lag]});
endproperty

// P3. The count matches the mask's population.
property p_count_matches_mask;
  @(posedge clk) disable iff (!rst_n)
  q_active_count == $countones(q_active);
endproperty

// P4. An aggregate with no active member reports down, and reporting
// down is not the same as not existing.
property p_lag_down_definition;
  @(posedge clk) disable iff (!rst_n)
  q_lag_down <-> (q_active == '0);
endproperty

// P5. distributing is a subset of collecting. A member we will not
// accept frames from must not be sent frames.
property p_distribute_implies_collect;
  @(posedge clk) disable iff (!rst_n)
  (lag_state.distributing & ~lag_state.collecting) == '0;
endproperty

Group 2 — selection is correct and safe.

// P6. A resolved port is always a member of the aggregate that was
// named, and always an active one.
property p_port_is_an_active_member;
  @(posedge clk) disable iff (!rst_n)
  (out_valid && $past(req_egress.is_lag)) |->
    (out_port inside {q_port_of[0:q_active_count-1]});
endproperty

// P7. A frame is never sent on a member that is down. This is the
// property a failover exists to preserve.
property p_never_send_on_down;
  @(posedge clk) disable iff (!rst_n)
  out_valid |-> link_up[member_of(out_port)];
endproperty

// P8. A down aggregate discards rather than flooding. Falling back to
// flood turns one cable into 12.4's 23x amplification.
property p_down_lag_discards;
  @(posedge clk) disable iff (!rst_n)
  (req_valid && req_egress.is_lag && lag_down) |=> (out_discard && !out_valid);
endproperty

// P9. A non-aggregate egress passes through untouched.
property p_non_lag_passthrough;
  @(posedge clk) disable iff (!rst_n)
  (req_valid && !req_egress.is_lag) |=> (out_valid && (out_port == $past(req_egress.port)));
endproperty

// P10. The selector is a function: the same hash and the same active
// count always give the same member.
property p_selection_is_deterministic;
  @(posedge clk) disable iff (!rst_n)
  (req_valid && (req_flow_hash == h) && (lag_active_count == n))
    |=> (out_port == expected_port(h, n));
endproperty

Group 3 — ordering, the constraint from Section 2.

// P11. While the membership is stable, a flow never changes member.
// This is the distributor's single obligation.
property p_flow_pinned_while_stable;
  @(posedge clk) disable iff (!rst_n)
  ($stable(lag_active_count) && (req_flow_hash == h))
    |=> (out_port == $past(port_for(h)));
endproperty

// P12. A flow is only moved while the redistributor is quiescing.
property p_moves_only_while_quiescing;
  @(posedge clk) disable iff (!rst_n)
  flow_moved |-> quiescing;
endproperty

// P13. The failed member stops being distributed to in the same cycle
// the failure is observed. Frames sent at it after that are lost.
property p_failed_member_dropped_immediately;
  @(posedge clk) disable iff (!rst_n)
  member_failed |=> !distributing[$past(failed_member)];
endproperty

// P14. Down is immediate and up is debounced -- the asymmetry is a
// specification, not a preference.
property p_down_is_immediate;
  @(posedge clk) disable iff (!rst_n)
  $fell(raw_up) |=> !member_up;
endproperty

// P15. An up transition waits the full debounce.
property p_up_is_debounced;
  @(posedge clk) disable iff (!rst_n)
  $rose(member_up) |-> ($past(debounce) == DEBOUNCE_CYCLES-1);
endproperty

Group 4 — the failover sequence.

// P16. The redistributor visits its states in order and never skips
// the drain.
property p_sequencer_order;
  @(posedge clk) disable iff (!rst_n)
  (state == S_STOP) |=> (state == S_DRAIN);
endproperty

// P17. The drain either completes or times out. It never waits forever.
property p_drain_terminates;
  @(posedge clk) disable iff (!rst_n)
  (state == S_DRAIN) |-> ##[1:DRAIN_LIMIT+1] (state == S_DONE);
endproperty

// P18. A timed-out drain is always recorded. A design that took the
// reordering risk must say so.
property p_timeout_is_recorded;
  @(posedge clk) disable iff (!rst_n)
  ((state == S_DRAIN) && (drain_cnt == DRAIN_LIMIT)) |=> $changed(c_drain_timeout);
endproperty

// P19. The table sweep yields: it never holds the table on a cycle the
// lookup path wanted it. 12.5 section 16's four-cycle budget.
property p_sweep_yields;
  @(posedge clk) disable iff (!rst_n)
  (sweep_req && lookup_req) |-> !sweep_gnt;
endproperty

// P20. The sweep clears exactly the entries naming the dead port.
property p_sweep_is_exact;
  @(posedge clk) disable iff (!rst_n)
  entry_clear[w] |-> (entry_valid[w] && (entry_port[w] == dead_port));
endproperty

// P21. A sweep terminates: 2048 granted cycles, no more.
property p_sweep_terminates;
  @(posedge clk) disable iff (!rst_n)
  $rose(busy) |-> ##[1:$] $fell(busy);
endproperty

Group 5 — accounting, and what a window can claim.

// P22. Instantaneously, the aggregate total is the sum of the members.
// This holds every cycle and says nothing about any interval.
property p_instantaneous_sum;
  @(posedge clk) disable iff (!rst_n)
  agg_octets == member_octets.sum();
endproperty

// P23. A window containing a membership change is marked. The totals
// are still right; the comparison between members is not.
property p_dirty_window_marked;
  @(posedge clk) disable iff (!rst_n)
  (window_tick && $past(membership_changed)) |-> window_spans_change;
endproperty

// P24. A marked window is still published. Suppressing it removes the
// data from exactly the moment an operator is looking.
property p_dirty_window_published;
  @(posedge clk) disable iff (!rst_n)
  window_spans_change |-> $changed(c_windows);
endproperty

// P25. Lifetime counters are monotonic. A removed member keeps its
// history and does not zero.
property p_lifetime_monotonic;
  @(posedge clk) disable iff (!rst_n)
  member_octets[m] >= $past(member_octets[m]);
endproperty

// P26. imbalance_x100 is at least 100 -- a maximum cannot be below a
// mean -- and equals 100 only when every active member is equal.
property p_imbalance_floor;
  @(posedge clk) disable iff (!rst_n)
  window_tick |=> (imbalance_x100 >= 16'd100);
endproperty

// P27. The expectation is evaluated against the same flow count the
// sketch produced, so the comparison is self-consistent.
property p_expectation_uses_current_estimate;
  @(posedge clk) disable iff (!rst_n)
  window_tick |=> (expected_imbalance_x100 ==
                   expected_ratio(distinct_flows_est, active_count));
endproperty

Group 6 — the configuration.

// P28. A standing property: members of an aggregate run at the same
// rate. The distributor cannot be fair otherwise.
property p_members_same_rate;
  @(posedge clk) disable iff (!rst_n)
  !cfg_mixed_rate;
endproperty

// P29. Learning writes an aggregate, never a member port. Section 11's
// second half, and the check that catches the half-implemented design.
property p_learning_writes_aggregate;
  @(posedge clk) disable iff (!rst_n)
  (learn_we && port_is_lag_member(learn_port)) |-> learn_entry.is_lag;
endproperty

// P30. Redundancy is reported before it is exhausted, not after.
property p_redundancy_precedes_down;
  @(posedge clk) disable iff (!rst_n)
  (redundancy_pct <= 16'd25) |-> !q_lag_down;
endproperty

// P31. conformant is exactly the conjunction it claims.
property p_conformant_definition;
  @(posedge clk) disable iff (!rst_n)
  conformant <-> (fault_vector == 8'h00);
endproperty

Every property above is evaluated at an instant or over a bounded window whose membership did not change. P22 is deliberately instantaneous, and the property this chapter refuses is what happens when somebody integrates it.

19. Verification Scenarios

Seventy scenarios. The most important ones have expected outcomes in which nothing is wrong and the aggregate delivers under half its nominal capacity.

Membership and compaction

Selection

Detection

Failover and ordering

The forwarding table

Distribution and capacity

Two ends

The directed test random stimulus will not produce

Random traffic will not produce the half-implemented aggregate-aware table, because it is not a traffic condition at all — it is a configuration in which two mechanisms disagree, and both are individually correct. A random test either configures the table format or it does not; nothing in a stimulus generator varies the learning path's entry format independently of the table's. And the failure is invisible until a member fails, which random stimulus does not do at a useful rate either.

Setup: a 24-port switch, LAG 3 with four members on ports 6, 7, 8, 9. The forwarding table stores egress_id_t and is_lag is honoured on lookup. The learning path writes {is_lag: 0, port: ingress_port} — the natural implementation of Chapter 12.2, unchanged from before aggregation existed.

Stimulus, in three phases. Phase 1: 4096 distinct source addresses send one frame each from behind the aggregate, distributed across all four members by the far end's hash. Phase 2: quiesce for 1 ms. Phase 3: fail member 2 — port 8.

Oracle:

Rows 1 to 6 are the finding: a switch that is entirely correct, forwards every frame, meets its latency budget, and has silently defeated the feature it appears to implement. Row 3 is the only signal available before phase 3, and row 15 is the same test with one lookup added to the learning path.

20. Debugging a LAG

Five questions in order. Three of them are answered before looking at any traffic.

Step 1 — is the aggregate whole? redundancy_pct and q_active_count. A four-member LAG at 75% has already had a failure nobody was told about, and this is the single most common finding: aggregates run degraded for weeks because "the LAG is up" is what gets alarmed.

Step 2 — is the configuration legal? cfg_mixed_rate and cfg_no_members. Both are standing properties and neither depends on traffic, so both are answerable from a configuration dump. A mixed-rate aggregate cannot be balanced by any distributor.

Step 3 — how long does a failover actually take? detect_us after a deliberate member removal. Section 7's four orders of magnitude are invisible otherwise, and the remedy differs completely between the ends of the range: a one-second detection is fixed with a pin, a ten-microsecond one is fixed by shortening the drain.

Step 4 — is the imbalance real? imbalance_x100 against expected_imbalance_x100. A ratio of 2.12 on a four-member LAG with four flows is the arithmetic, not a fault. Only when imbalance_is_expected is low is the distribution worth investigating — and then it is Chapter 15.2's subject.

Step 5 — does a failover flood? c_purged and learned_physical_port. Non-zero c_purged means the table stores physical ports, whatever the entry format claims, and the flood that follows is predictable from the count: 35 kB per purged address on a 24-port switch.

And the finding that ends an investigation: redundancy_pct at 100, conformant high, imbalance_is_expected high, c_purged zero, detect_us in the tens. That is an aggregate doing everything available to it, and a conversation that is slow on it is slow because one conversation gets one member — Section 13's ceiling, which no configuration changes.

21. Common Misconceptions

1 — "A 4 × 10 Gb/s aggregate is a 40 Gb/s link."

The wrong model: bonding adds the rates.

What it costs: a storage array attached over an aggregate that moves one large transfer at 10 Gb/s, with the switch reporting 25% utilisation and three members idle. Nothing is broken and nothing can be tuned.

The corrected model: Section 2's constraint puts all frames of one conversation on one member, so one conversation's ceiling is one member's rate whatever N is. An aggregate adds capacity across conversations and never within one. The availability claim is the unconditional one: four members survive three failures.

2 — "The LAG is up, so we are fine."

The wrong model: aggregate state is a boolean.

What it costs: aggregates running degraded for weeks. A four-member LAG on one member is up, carrying traffic, and one failure from an outage — and an alarm on aggregate-down fires after the last member fails, which is exactly one failure too late.

The corrected model: alarm on redundancy_pct, not on q_lag_down. The number that matters is how many failures remain survivable, and it falls from 100% to 25% through three events that a boolean cannot report.

3 — "The hash is unfair — member 2 is carrying twice the traffic."

The wrong model: an imbalance means the distributor is biased.

What it costs: hash tuning that cannot help. Section 13: four flows into four bins gives an expected maximum of 2.12 with a perfect hash. At sixteen flows the expected ratio is still 1.53.

The corrected model: compare the observed imbalance against the expected imbalance for the current flow count. imbalance_is_expected is the signal; a ratio of 2.12 at four flows is the arithmetic and a ratio of 2.12 at 4096 flows is a real defect. Count the flows before touching the hash.

4 — "Failover time is a property of the switch."

The wrong model: a faster switch fails over faster.

What it costs: buying the wrong thing. Section 7's four orders of magnitude come from which mechanism is in the path — a PHY interrupt, an MDIO poll, or an LACP timeout — and the line rate appears in only one row of that table. The mask update is 2 ns on every switch ever built.

The corrected model: measure detect_us. A one-second failover is a polling interval and is fixed with an interrupt pin; a ten-microsecond one is already at the mechanism's floor and its remaining cost is the ordering drain.

5 — "Both ends of an aggregate use the same distribution."

The wrong model: the two ends are symmetric.

What it costs: hours comparing per-member counters that were never comparable. 802.1AX does not require the two ends to hash the same way, because a distribution function only has to be consistent with itself. A conversation can legitimately use member 1 outbound and member 3 inbound.

The corrected model: each end's distribution is its own. member_octets on A and on B describe two independent hashes of the same conversations, and an operator finding member 2 hot on one end and member 4 hot on the other has found nothing.

6 — "A PAUSE frame on a member pauses the aggregate."

The wrong model: the aggregate is the link.

What it costs: either a fourfold multiplication of Chapter 14.2 §15's 88% collateral, or — worse — a design that removes the paused member from distribution and reorders every flow on it for a condition that clears in microseconds.

The corrected model: a PAUSE is consumed by the port that receives it, so member 2 stops and the aggregate runs at 75%. And the measurement consequence is worth knowing: a paused member looks under-utilised, so imbalance measured during a pause episode is misleading and window_spans_change does not fire, because the membership did not change.

22. Interview Reasoning

Q1 — Why is distribution per flow rather than per frame?

Because Ethernet has no reordering mechanism and every layer above assumes there is nothing to reorder. Per-frame distribution would give perfect balance — the elephant flow and Section 13's 47.2% both disappear — and it would deliver frames out of order whenever two members' queues differ, which at 512 cells of difference is 524 µs at 1 Gb/s. TCP reads that as loss and halves its window; a storage protocol needs a reassembly buffer nobody sized. The constraint is inherited rather than chosen, and every technique in Module 15 works inside it.

Q2 — How long does a LAG take to fail over?

Between 10 µs and 90 seconds, and the line rate is not in the answer. It is set by which detection mechanism is in the path: a PHY interrupt is ~10 µs, MDIO polling is the polling interval, an LACP slow timeout is 90 s. The mask update is 2 ns and the ordering drain is bounded by a parameter. What does scale with rate is the damage: 1 s of detection on a 100 Gb/s member is 12.5 GB.

Q3 — A member fails. What happens to the forwarding table?

Either nothing or a sweep and a flood, decided by one bit in the entry format. An entry storing an aggregate id is still correct — the distributor picks a different member. An entry storing a physical port is now wrong, so 2048 sets are swept, every matching entry purged, and each purged address becomes an unknown unicast: 4096 of them on a 24-port switch is 143 MB of flooded traffic. The extra bit costs 1 KiB across 8192 entries.

Q4 — And what is the trap in that design?

Learning. Chapter 12.2 records the ingress port, which for a frame on member 2 of LAG 3 is port 7. A table whose entry format supports aggregates and whose learning path writes physical ports produces exactly the entries the format was meant to avoid, on every learned address, and floods at every failover as though the feature were absent. The fix is one 24-entry lookup on the learning path; the check is an assertion that a learned entry naming a LAG member has is_lag set.

Q5 — Four members and four conversations. What fraction of the aggregate is usable?

47.2%, with a perfect hash. Four balls into four bins has an expected maximum of 2.12 against a mean of 1, and the fullest member saturates first. Sixteen flows give 65.2%, 256 give 88.5%, 4096 give 96.9%. It is the same balls-in-bins arithmetic as Chapter 12.5 §9's hash table, with a different consequence: there a full set evicted an entry, here a full member throttles a flow. The diagnostic for an unbalanced aggregate is the flow count, not the imbalance.

Q6 — Why can't you assert that the aggregate's octet total equals the sum of its members' over a measurement window?

Because the set being summed can change inside the window. The identity holds at every instant and fails over any interval containing a membership change: a member removed at 40% contributed 40% of a window and is no longer in the set; one added at 70% contributed 30% and was absent for the rest. All the totals are correct and the identity between them is not — integration and set-membership do not commute. The fix is a guard: assert the windowed form only when window_spans_change is low, and publish the flag so a discontinuity in a graph has an explanation next to it.

23. Understanding Check

24. What's Next

This chapter treated the distributor as a black box with one property — deterministic per flow — and everything above rests on that one guarantee. The next chapter opens the box.

Three questions are left unanswered here and each one turns out to matter more than it looks.

What defines a flow? Section 2 required all frames of one conversation to take one member and never said what a conversation is. The answer is a field selection, and it decides whether a router behind the aggregate looks like one flow or a million.

What makes the hash uniform? Section 13's arithmetic assumed a perfect hash and derived 47.2% at four flows. A real polynomial over real traffic does worse, and the amount worse is computable.

And what happens when N changes? Section 5 chose a modulo against the active count and Section 9 priced the consequence at "nearly every flow moves". Chapter 15.2 quantifies both choices and shows the technique that moves only the flows that have to.

Chapter 15.2 — Hash-Based Distribution and Frame Ordering builds the field selector, the polynomial, the reduction to a member index, and the elephant flow that no hash can split.

Then Chapter 15.3 — LACP replaces Section 16's four silent disagreements with a negotiation, and shows the mis-cable that static configuration cannot see.

One thread runs from this chapter into both. Section 11's finding was that a mechanism designed before aggregation existed — Chapter 12.2's learning — records a fact that is no longer the fact anybody wants. Chapter 15.3 finds the same shape at the other end of the cable: a statically configured aggregate records what an operator intended, and the thing worth knowing is what is actually plugged in.