Skip to content
VLSI Mentor

Ethernet · Module 15

LACP

Negotiating aggregation instead of configuring it: the 128-octet LACPDU, the state machine, and the failure where every link is up and a quarter of the traffic goes to the wrong device.

Chapter 15.1 §16 listed four ways the two ends of an aggregate can disagree and found that a statically configured aggregate detects none of them. Chapter 15.2 added a fifth that is deliberate and correct. This chapter is about the other four.

Static configuration records what an operator intended. The useful fact is what is actually plugged in, and no local measurement produces it.

Chapter 15.1 §6 built three detection sources — PHY loss of signal, PCS block lock, and a protocol liveness input that was left as a stub. All three of the first two answer the same question: is this end receiving a signal. None of them answers the question aggregation actually needs, which is whether the thing at the other end of this cable is the same thing as at the other end of the other three.

The failure that makes the point is a mis-cabled member. Four cables from one switch. Three reach the intended neighbour and one reaches a different device entirely. Every link is up, every PHY is healthy, every PCS is locked, Chapter 15.1 §15's conformant is high — and one quarter of the aggregate's traffic is being delivered to a device it was never addressed to.

LACP costs 1184 bit/s and finds it in three seconds.

1. Scope — What This Chapter Owns

This chapter owns the negotiation: the LACPDU and its fields, the actor/partner exchange, the three states a member can be in, the selection logic, the periodic timer and its timeout, churn, and the mis-cable that motivates all of it.

It does not own the aggregate. Chapter 15.1 built the member table, the compaction, the failover sequence and the forwarding-table consequence. LACP decides which ports go in that table and nothing about what happens afterwards.

It does not own distribution. Chapter 15.2 built the field selector, the hash and the reduction. 802.1AX does not negotiate any of itChapter 15.2 §7's callout gave the reason — so two ends running LACP still hash independently and their per-member counters remain incomparable.

It does not own link detection. Chapter 15.1 §6's PHY and PCS sources are unchanged and remain the fast path: 10 µs and 106 ns respectively. LACP is the slow path — 3 seconds at best — and Section 13 shows that it is nonetheless the only source that catches an entire class of failure.

And it does not own the physical layer. Chapter 11.2's auto-negotiation settles speed and duplex on one link; LACP settles membership across several. The two are frequently confused because both are described as negotiation, and Section 2's table separates them.

2. What Static Configuration Cannot Detect

Four failures, and what makes them worth a protocol is that the local evidence is identical in every case to the evidence of a healthy link.

FailureWhat every local source reportsWhat actually happened
member count mismatchfour links upthe far end put three in its aggregate
port set mismatchfour links upthe far end's fourth is a different port
mis-cabled memberfour links upone cable reaches a different device
far end not aggregatingfour links upit treats them as four independent ports
one-way failurelink up — the local receiver is fineour transmit is dead

The middle column is the same in all five rows. Chapter 15.1 §6's phy_link_up and pcs_block_lock both describe this end's receiver, and this end's receiver is perfectly happy in every case.

And the consequences are not small.

The mis-cable delivers 25% of the aggregate's traffic to an unrelated device, which then floods it — Chapter 12.4's amplification, applied to traffic that had a perfectly good destination.

The far end not aggregating produces Chapter 12.2 §9's flapping entry at the rate the distributor moves flows, which on a busy aggregate is thousands of times a second, on every address behind us.

The one-way failure is a member that is chosen by Chapter 15.2's distributor for its full share of flows and carries none of them.

==

Five distinct aggregation failures produce identical local evidence. A member count mismatch, a port set mismatch, a mis-cabled member reaching a different device, a far end that is not aggregating at all, and a one-way transmit failure all leave this end's PHY reporting link up and its PCS reporting block lock, because both describe the local receiver rather than the far end's identity or intent. Auto-negotiation is equally satisfied: on a mis-cabled member there is a real device that negotiates ten gigabit full duplex correctly. The only source that separates the five is a statement from the far end about what it believes itself to be, which is exactly the content of an LACPDU.Member countdiffersthey aggregate 3Mis-cabled memberreaches switch CFar end notaggregatingfour separate portsphy_link_upup in all fivepcs_block_locklocked in all fiveAuto-negotiation10G full, correctIdentical localviewnothing separates themThe far end'sstatementactor system id12
Figure 1 — five failures, one local view: every detection source a switch owns reports the same thing in all of them.

What all four need is the same thing: a statement from the far end about what it believes. That is the entire content of the protocol — there is no negotiation in the sense of proposing and counter-proposing. Each end periodically says what it is and what it thinks its partner is, and aggregation happens only where the two statements are consistent.

3. RTL 1 — Building an LACPDU

The frame is 128 octets and every field in it is a claim about identity. Nothing in it is a request.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// lacp_pkg -- shared types for 802.1AX Link Aggregation Control.
// -----------------------------------------------------------------------
package lacp_pkg;

  localparam int MAX_MEMBERS = 8;
  localparam int MEMBER_W    = $clog2(MAX_MEMBERS);

  // Slow-protocol destination, distinct from 14.2's control DA.
  localparam logic [47:0] SLOW_DA   = 48'h01_80_C2_00_00_02;
  localparam logic [15:0] SLOW_ET   = 16'h8809;
  localparam logic [7:0]  LACP_SUB  = 8'h01;
  localparam logic [7:0]  LACP_VER  = 8'h01;

  // The eight state bits, carried once for the actor and once for the
  // partner. Bits 2 and 3 are the two roles 15.1 section 3 kept apart.
  typedef struct packed {
    logic expired;         // bit 7
    logic defaulted;       // bit 6 -- using defaults, not heard from
    logic distributing;    // bit 5 -- 15.1's distributing
    logic collecting;      // bit 4 -- 15.1's collecting
    logic synchronised;    // bit 3 -- IN SYNC with the partner
    logic aggregation;     // bit 2 -- willing to aggregate at all
    logic timeout_short;   // bit 1 -- 1 = fast (1 s), 0 = slow (30 s)
    logic activity;        // bit 0 -- 1 = active, will send unprompted
  } lacp_state_t;

  // What identifies an aggregatable entity. Two ports may join the
  // same aggregate only if all four of these match -- section 8.
  typedef struct packed {
    logic [15:0] sys_pri;
    logic [47:0] sys_id;     // the device's own MAC
    logic [15:0] key;        // the aggregation key
    logic [15:0] port_pri;
    logic [15:0] port_id;
  } lacp_id_t;

  typedef enum logic [1:0] {
    MUX_DETACHED, MUX_WAITING, MUX_ATTACHED, MUX_COLLECT_DIST
  } mux_state_e;

endpackage
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// lacpdu_builder -- emits one 128-octet LACPDU.
//
// The frame states what WE are (actor) and what we BELIEVE the partner
// to be (partner). The second half is a mirror, and section 19's
// rejected property is about how stale that mirror is.
// -----------------------------------------------------------------------
module lacpdu_builder
  import lacp_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        req_valid,
  input  lacp_id_t    actor,
  input  lacp_state_t actor_state,
  input  lacp_id_t    partner,        // what we last heard, or defaults
  input  lacp_state_t partner_state,
  input  logic [15:0] collector_max_delay,
  output logic        req_ready,

  output logic        tx_valid,
  output logic [7:0]  tx_octet,
  output logic        tx_sop,
  output logic        tx_eop,
  output logic [31:0] c_sent
);

  // 6 DA + 6 SA + 2 ET + 110 payload = 124; FCS appended downstream
  // brings the frame to 128. No padding: the payload is well above
  // 5.6's 46-octet minimum.
  localparam int BODY = 124;

  logic [7:0] b [0:BODY-1];
  logic [6:0] idx;
  logic       busy;

  always_comb begin
    int p;
    for (p = 0; p < BODY; p++) b[p] = 8'h00;

    for (p = 0; p < 6; p++) b[p]     = SLOW_DA[47 - 8*p -: 8];
    for (p = 0; p < 6; p++) b[6 + p] = actor.sys_id[47 - 8*p -: 8];
    b[12] = SLOW_ET[15:8];  b[13] = SLOW_ET[7:0];
    b[14] = LACP_SUB;       b[15] = LACP_VER;

    // Actor TLV: type 0x01, length 20.
    b[16] = 8'h01; b[17] = 8'd20;
    b[18] = actor.sys_pri[15:8]; b[19] = actor.sys_pri[7:0];
    for (p = 0; p < 6; p++) b[20 + p] = actor.sys_id[47 - 8*p -: 8];
    b[26] = actor.key[15:8];      b[27] = actor.key[7:0];
    b[28] = actor.port_pri[15:8]; b[29] = actor.port_pri[7:0];
    b[30] = actor.port_id[15:8];  b[31] = actor.port_id[7:0];
    b[32] = actor_state;
    // b[33..35] reserved

    // Partner TLV: type 0x02, length 20. This is our BELIEF about them.
    b[36] = 8'h02; b[37] = 8'd20;
    b[38] = partner.sys_pri[15:8]; b[39] = partner.sys_pri[7:0];
    for (p = 0; p < 6; p++) b[40 + p] = partner.sys_id[47 - 8*p -: 8];
    b[46] = partner.key[15:8];      b[47] = partner.key[7:0];
    b[48] = partner.port_pri[15:8]; b[49] = partner.port_pri[7:0];
    b[50] = partner.port_id[15:8];  b[51] = partner.port_id[7:0];
    b[52] = partner_state;
    // b[53..55] reserved

    // Collector TLV: type 0x03, length 16.
    b[56] = 8'h03; b[57] = 8'd16;
    b[58] = collector_max_delay[15:8]; b[59] = collector_max_delay[7:0];
    // b[60..71] reserved

    // Terminator TLV: type 0x00, length 0, then 50 reserved octets.
    b[72] = 8'h00; b[73] = 8'd0;
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      idx <= '0; busy <= 1'b0; c_sent <= '0;
    end else if (!busy) begin
      if (req_valid) begin
        busy   <= 1'b1;
        idx    <= '0;
        c_sent <= c_sent + 1;
      end
    end else begin
      if (idx == BODY-1) begin busy <= 1'b0; idx <= '0; end
      else                     idx  <= idx + 1'b1;
    end
  end

  assign req_ready = !busy;
  assign tx_valid  = busy;
  assign tx_octet  = b[idx];
  assign tx_sop    = busy && (idx == 7'd0);
  assign tx_eop    = busy && (idx == BODY-1);

endmodule

Classification: a protocol encoder with no state of its own. One frame per request, fixed length, no padding.

What it teaches: that the frame carries two identities and one of them is hearsay. The actor TLV is a fact about this device; the partner TLV is what this device last heard, which the far end will read as our report of its state. So the partner TLV is a mirror — and Section 18's rejected property is about how much a mirror can be trusted when the round trip is one second.

And it teaches why the source address is actor.sys_id rather than a port address. Chapter 14.4 §3's pfc_frame_builder used the port's address for exactly the opposite reason. Here the frame is a claim about the device, and all of a device's ports must claim the same identity — otherwise a four-member aggregate looks like four one-member aggregates to the far end, which is Section 9's selection logic failing on a design that got this wrong.

Deliberately simplified: the reserved octets are transmitted as zero and the terminator TLV is emitted with no provision for optional TLVs between it and the collector. Production designs must skip unknown TLVs rather than assume the layout, because 802.1AX has added TLVs since — and a parser with hard-coded offsets past the collector fails on a conformant neighbour.

Production implication: c_sent is the counter that separates "we are not sending" from "they are not hearing". Paired with Section 5's c_received, it is the whole of the first debugging step: c_sent rising and c_received flat is a one-way path, which is precisely the failure Chapter 15.1 §6's physical sources cannot see and the reason proto_alive was left as an input there.

4. The LACPDU Format, Field by Field

Every field is a claim, and it is worth knowing which claims matter.

OffsetFieldOctetsWhat it decides
0DA 01:80:C2:00:00:026Slow Protocols — never forwarded by a bridge
6SA — the device's MAC6all ports claim one identity
12EtherType 0x88092Slow Protocols
14subtype 0x011LACP, against 0x02 Marker
15version1
16actor TLV type 0x01, length 202
18actor system priority2which end decides — Section 9
20actor system id6the device
26actor key2which ports may aggregate together
28actor port priority2which ports are chosen if not all fit
30actor port id2
32actor state1the eight bits
36partner TLV — type 0x02, length 2020our belief about them
56collector TLV, max delay16how long we may buffer during a move
72terminator, then 50 reserved52
payload total110frame 128, wire 148

And the eight state bits, because they carry the whole of the negotiation:

BitNameMeaning
0activityactive — we send unprompted; passive — only in reply
1timeout1 = fast (1 s / 3 s), 0 = slow (30 s / 90 s)
2aggregationwilling to aggregate; clear means individual
3synchronisedour view of the partner matches what they sent
4collectingChapter 15.1 §3's receive role
5distributingChapter 15.1 §3's transmit role
6defaultedwe have heard nothing and are using defaults
7expiredthe receive timer fired

Bit 3 is the negotiation's result and bits 4 and 5 are its consequence. A member becomes usable only when both ends report synchronised, and the two roles are then enabled in Chapter 15.1 §3's order — collecting first, distributing second.

Bit 6 is the one worth dwelling on. Defaulted means this end is transmitting a partner TLV it made up. The far end therefore knows that our mirror of it is fiction, which is a remarkable thing for a protocol to say about its own message and is exactly why the mirror is usable at all — Section 18's rejected property is what happens when a design forgets to read this bit.

And the frame's cost is negligible in both modes:

ModePeriodFrames/sBit/sOn 1 Gb/sOn 100 Gb/s
fast1 s111841.18 × 10⁻⁴%1.18 × 10⁻⁶%
slow30 s0.03339.53.95 × 10⁻⁶%3.95 × 10⁻⁸%

Fast mode costs one part in a million of a gigabit link and detects a failure 30× sooner. There is no bandwidth argument for slow mode, and Section 13 shows what the argument actually is.

5. RTL 2 — Parsing an LACPDU

The receive side has one job the builder does not: deciding whether what arrived is about us.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// lacpdu_parser -- validates and extracts an LACPDU.
//
// The partner TLV in a RECEIVED frame is the far end's belief about
// US. Comparing it against what we actually are is how section 7's
// synchronised bit gets set, and it is the only place in the protocol
// where a device learns whether it has been heard correctly.
// -----------------------------------------------------------------------
module lacpdu_parser
  import lacp_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        rx_valid,
  input  logic [7:0]  rx_octet,
  input  logic        rx_sop,
  input  logic        rx_eop,
  input  logic        rx_fcs_ok,

  input  lacp_id_t    our_actor,       // what we believe we are

  output logic        pdu_valid,
  output lacp_id_t    rx_actor,        // what THEY say they are
  output lacp_state_t rx_actor_state,
  output lacp_id_t    rx_partner,      // what they think WE are
  output lacp_state_t rx_partner_state,
  output logic        they_see_us,     // their partner TLV matches us

  output logic [31:0] c_received,
  output logic [31:0] c_bad_fcs,
  output logic [31:0] c_wrong_subtype,
  output logic [31:0] c_mismatched_mirror
);

  logic [6:0]  idx;
  logic        in_frame, da_ok, et_ok, sub_ok;
  logic [15:0] et_q;
  lacp_id_t    a_q, p_q;
  lacp_state_t as_q, ps_q;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      idx <= '0; in_frame <= 1'b0; da_ok <= 1'b0;
      et_ok <= 1'b0; sub_ok <= 1'b0; et_q <= '0;
      a_q <= '0; p_q <= '0; as_q <= '0; ps_q <= '0;
      pdu_valid <= 1'b0; they_see_us <= 1'b0;
      c_received <= '0; c_bad_fcs <= '0;
      c_wrong_subtype <= '0; c_mismatched_mirror <= '0;
    end else begin
      pdu_valid <= 1'b0;

      if (rx_valid && rx_sop) begin
        idx <= 7'd1; in_frame <= 1'b1; sub_ok <= 1'b0;
        da_ok <= (rx_octet == SLOW_DA[47:40]);
      end else if (rx_valid && in_frame) begin
        idx <= idx + 1'b1;

        if (idx <= 7'd5)
          if (rx_octet != SLOW_DA[47 - 8*idx -: 8]) da_ok <= 1'b0;

        if (idx == 7'd12) et_q[15:8] <= rx_octet;
        if (idx == 7'd13) et_ok <= ({et_q[15:8], rx_octet} == SLOW_ET);
        if (idx == 7'd14) sub_ok <= (rx_octet == LACP_SUB);

        // Actor TLV.
        if (idx == 7'd18) a_q.sys_pri[15:8] <= rx_octet;
        if (idx == 7'd19) a_q.sys_pri[7:0]  <= rx_octet;
        if (idx >= 7'd20 && idx <= 7'd25)
          a_q.sys_id[47 - 8*(idx-20) -: 8] <= rx_octet;
        if (idx == 7'd26) a_q.key[15:8]      <= rx_octet;
        if (idx == 7'd27) a_q.key[7:0]       <= rx_octet;
        if (idx == 7'd28) a_q.port_pri[15:8] <= rx_octet;
        if (idx == 7'd29) a_q.port_pri[7:0]  <= rx_octet;
        if (idx == 7'd30) a_q.port_id[15:8]  <= rx_octet;
        if (idx == 7'd31) a_q.port_id[7:0]   <= rx_octet;
        if (idx == 7'd32) as_q               <= rx_octet;

        // Partner TLV -- their mirror of us.
        if (idx == 7'd38) p_q.sys_pri[15:8] <= rx_octet;
        if (idx == 7'd39) p_q.sys_pri[7:0]  <= rx_octet;
        if (idx >= 7'd40 && idx <= 7'd45)
          p_q.sys_id[47 - 8*(idx-40) -: 8] <= rx_octet;
        if (idx == 7'd46) p_q.key[15:8]      <= rx_octet;
        if (idx == 7'd47) p_q.key[7:0]       <= rx_octet;
        if (idx == 7'd48) p_q.port_pri[15:8] <= rx_octet;
        if (idx == 7'd49) p_q.port_pri[7:0]  <= rx_octet;
        if (idx == 7'd50) p_q.port_id[15:8]  <= rx_octet;
        if (idx == 7'd51) p_q.port_id[7:0]   <= rx_octet;
        if (idx == 7'd52) ps_q               <= rx_octet;
      end

      if (rx_valid && rx_eop) begin
        in_frame <= 1'b0;
        if (da_ok && et_ok) begin
          if      (!sub_ok)    c_wrong_subtype <= c_wrong_subtype + 1;
          else if (!rx_fcs_ok) c_bad_fcs       <= c_bad_fcs + 1;
          else begin
            c_received       <= c_received + 1;
            pdu_valid        <= 1'b1;
            rx_actor         <= a_q;
            rx_actor_state   <= as_q;
            rx_partner       <= p_q;
            rx_partner_state <= ps_q;

            // Do they have us right? If their mirror does not match
            // what we are, they are talking about somebody else -- or
            // they have not heard our latest state yet.
            they_see_us <= (p_q.sys_id  == our_actor.sys_id) &&
                           (p_q.key     == our_actor.key)    &&
                           (p_q.port_id == our_actor.port_id);
            if (!((p_q.sys_id  == our_actor.sys_id) &&
                  (p_q.key     == our_actor.key)    &&
                  (p_q.port_id == our_actor.port_id)))
              c_mismatched_mirror <= c_mismatched_mirror + 1;
          end
        end
      end
    end
  end

endmodule

Classification: a protocol decoder with a validity gate and one comparison that is the whole protocol's hinge.

What it teaches: that they_see_us is where a device discovers whether it has been heard. Everything else in an LACPDU is the far end describing itself; this one field is the far end describing us, and comparing it against what we actually are closes the loop. Without it there is no way to distinguish a partner that is talking to us from one that is talking past us — which is Section 11's mis-cable exactly.

And it teaches that c_mismatched_mirror is transiently normal and persistently fatal. After any state change, the far end's mirror is one round trip out of date, so the counter increments for up to a second in fast mode. It rising continuously means the far end's partner TLV never matches us, which means it is mirroring somebody else — the mis-cable, in one counter.

Deliberately simplified: the parser assumes the TLVs are at their standard offsets and stops at the collector. 802.1AX permits additional TLVs and a conformant parser walks the type/length chain, skipping unknown types until the terminator. A hard-coded parser rejects nothing and mis-parses everything on a neighbour that adds one, which is the same failure shape as Chapter 13.2 §7's fixed-offset parse.

Production implication: c_wrong_subtype catches Marker protocol frames — subtype 0x02, Section 16 — arriving at a device that does not implement them. That is not an error and must not be counted as one, so a production design separates "a slow protocol we do not implement" from "a malformed LACPDU". Conflating them produces an error counter that rises during perfectly normal operation, which is the fastest way to make an error counter ignored.

6. RTL 3 — The Actor/Partner State Machine

The mux machine decides when a member is usable, and its four states are the protocol's actual content.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// lacp_state_machine -- the per-port mux machine.
//
// DETACHED -> WAITING -> ATTACHED -> COLLECTING_DISTRIBUTING
//
// The WAITING state exists so that several ports selecting the same
// aggregate at the same time settle together rather than one at a
// time; without it, a four-member aggregate comes up as a
// one-member aggregate, then two, then three -- and 15.2 section 12's
// rebalance runs at every step.
// -----------------------------------------------------------------------
module lacp_state_machine
  import lacp_pkg::*;
#(
  parameter int AGGREGATION_WAIT = 1000000   // 2 s at 500 MHz
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        pdu_valid,
  input  lacp_id_t    rx_actor,
  input  lacp_state_t rx_actor_state,
  input  logic        they_see_us,
  input  logic        rx_timeout,       // section 12's expiry
  input  logic        port_enabled,     // 15.1 section 6's physical up
  input  logic        selected,         // section 8 chose this port

  output mux_state_e  mux,
  output logic        collecting,
  output logic        distributing,
  output lacp_state_t our_state,
  output logic [31:0] c_attach,
  output logic [31:0] c_detach
);

  logic [31:0] wait_cnt;
  logic        partner_sync;

  // We are synchronised when they have us right AND they say they are
  // synchronised too. Both halves are required: their sync bit alone
  // could be about a different neighbour.
  assign partner_sync = they_see_us && rx_actor_state.synchronised;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      mux <= MUX_DETACHED; wait_cnt <= '0;
      collecting <= 1'b0; distributing <= 1'b0;
      c_attach <= '0; c_detach <= '0;
      our_state <= '0;
    end else begin
      // Any loss of the physical port or of the partner drops us all
      // the way out. There is no partial state.
      if (!port_enabled || rx_timeout || !selected) begin
        if (mux != MUX_DETACHED) c_detach <= c_detach + 1;
        mux          <= MUX_DETACHED;
        collecting   <= 1'b0;
        distributing <= 1'b0;
        wait_cnt     <= '0;
      end else begin
        unique case (mux)
          MUX_DETACHED: begin
            collecting <= 1'b0; distributing <= 1'b0;
            if (selected) begin
              mux      <= MUX_WAITING;
              wait_cnt <= '0;
            end
          end

          MUX_WAITING: begin
            wait_cnt <= wait_cnt + 1;
            if (wait_cnt == AGGREGATION_WAIT-1) mux <= MUX_ATTACHED;
          end

          MUX_ATTACHED: begin
            // 15.1 section 3's order: collect BEFORE distributing, so
            // the far end is never sent frames on a member we are not
            // yet accepting from.
            collecting <= 1'b1;
            if (partner_sync && rx_actor_state.collecting) begin
              mux      <= MUX_COLLECT_DIST;
              c_attach <= c_attach + 1;
            end
          end

          MUX_COLLECT_DIST: begin
            collecting   <= 1'b1;
            distributing <= 1'b1;
          end
        endcase
      end

      our_state.synchronised <= (mux == MUX_ATTACHED) ||
                                (mux == MUX_COLLECT_DIST);
      our_state.collecting   <= collecting;
      our_state.distributing <= distributing;
      our_state.aggregation  <= 1'b1;
      our_state.defaulted    <= rx_timeout;
      our_state.expired      <= rx_timeout;
    end
  end

endmodule

Classification: a four-state mux machine with one timed state and an unconditional drop-out.

What it teaches: that MUX_WAITING is a settling delay and not a safety margin, and removing it produces a specific and expensive symptom. Four ports of an aggregate come up within milliseconds of each other; without the wait, port 1 attaches alone, then port 2 joins, then port 3, then port 4. Each join is a membership change, so Chapter 15.2 §12's rebalance runs four times — and with a modulo reduction that relocates most of the flows on each occasion, for a link that is simply coming up.

And it teaches that partner_sync requires two things and a design that checks one is broken in a way that only appears on a mis-cable. The far end saying I am synchronised is not enough: it may be synchronised with somebody else. The conjunction with they_see_us is what makes the claim about this pairing, and Section 11 is the failure it prevents.

Deliberately simplified: the machine drops all the way to MUX_DETACHED on any loss, including a single missed timeout. Production designs pass through an EXPIRED state first, in which the port keeps collecting while transmitting at the fast rate to re-establish quickly — so a brief loss costs a fast-mode round trip rather than Chapter 15.2 §12's full rebalance and a re-attach.

Production implication: c_detach is the churn counter's raw input and the number an operator actually needs. A member that attaches and detaches repeatedly costs a rebalance each wayChapter 15.1 §6's flapping arithmetic, now driven by the protocol rather than by the cable — and Section 14's churn detector exists because the symptom of protocol churn is identical to the symptom of a flapping link.

==

A port progresses through four mux states. Detached means neither role is active. Selected ports enter waiting, a timed state whose purpose is to let several ports selecting the same aggregate settle together, so a four-port bring-up causes one membership change rather than four. Attached enables collecting only: the port accepts frames from the far end and sends none. Only when the partner reports that it is both synchronised and collecting does the port enter collecting-and-distributing and become a full member. The handshake is therefore: I am collecting, I see you are collecting, I am distributing — two round trips and zero frames lost, which static configuration cannot achieve because it has no way to know when the far end is ready. Any loss of the physical port, the partner or the selection returns the port to detached immediately, with no partial state.DETACHEDneither roleWAITINGports settle togetherATTACHEDcollecting onlyCOLLECT + DISTa full memberI am collectingbit 4 setI see youcollectingtheir bit 4I am distributingbit 5 set12
Figure 2 — the mux machine's four states, and the handshake that loses no frames in either direction.

7. Three States a Member Can Be In

A statically configured member is in or out. An LACP member has three states and the middle one is where every interesting failure sits.

StatecollectingdistributingWhat it means
detached00not in the aggregate — physical down, or unselected, or the partner disagrees
attached10we accept from it and do not send on it
collecting/distributing11a full member

The middle row is the state that makes the protocol safe, and it is exactly Chapter 15.1 §3's collecting/distributing split arriving with a mechanism behind it.

A port that comes up starts collecting first. Frames the far end sends on it are accepted; nothing is sent on it until the far end says it is collecting too. So there is no window in which either end sends into a port the other is not accepting from — which a statically configured aggregate cannot avoid, because it has no way to know when the far end is ready.

And the ordering is the reason the two bits are carried in the LACPDU at all. Each end reads the other's collecting bit and only then sets its own distributing. The handshake is: I am collecting → I see you are collecting → I am distributing. Two round trips, two seconds in fast mode, and no frame is lost in either direction.

Which prices the whole protocol against static configuration in one row:

staticLACP
time to bring a member into serviceimmediate~2 sAGGREGATION_WAIT plus a round trip
frames lost bringing it upup to a round trip's worthzero
frames lost if the far end is not readyall of them, until it iszero

A design optimising the first row is optimising the wrong one, because bringing a member up is a scheduled event and losing traffic on it is not.

8. RTL 4 — Aggregation Key Matching

Which ports may join one aggregate is decided by a comparison of four fields, and the comparison is done on both ends independently.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// lacp_key_matcher -- decides whether a port may join an aggregate.
//
// Two ports aggregate together only if BOTH ends agree they should.
// Locally that means the same actor key. Remotely it means the same
// partner system id AND the same partner key. Section 11's mis-cable
// fails the remote half while passing the local one.
// -----------------------------------------------------------------------
module lacp_key_matcher
  import lacp_pkg::*;
(
  input  logic     clk,
  input  logic     rst_n,

  input  logic     pdu_valid [MAX_MEMBERS],
  input  lacp_id_t our_actor  [MAX_MEMBERS],
  input  lacp_id_t rx_actor   [MAX_MEMBERS],   // the partner, per port
  input  logic     partner_valid [MAX_MEMBERS],

  output logic [MAX_MEMBERS-1:0] selected,
  output logic [MAX_MEMBERS-1:0] individual,   // cannot aggregate
  output logic [47:0]            chosen_partner_sys,
  output logic [15:0]            chosen_partner_key,
  output logic [31:0]            c_wrong_partner,
  output logic [31:0]            c_selections
);

  // The reference pairing is the LOWEST-numbered port that has a
  // partner. Every other port must match it or be excluded.
  logic        have_ref;
  logic [47:0] ref_sys;
  logic [15:0] ref_key, ref_akey;

  always_comb begin
    int m;
    have_ref = 1'b0;
    ref_sys  = '0; ref_key = '0; ref_akey = '0;
    for (m = 0; m < MAX_MEMBERS; m++) begin
      if (!have_ref && partner_valid[m]) begin
        have_ref = 1'b1;
        ref_sys  = rx_actor[m].sys_id;
        ref_key  = rx_actor[m].key;
        ref_akey = our_actor[m].key;
      end
    end
  end

  always_ff @(posedge clk or negedge rst_n) begin
    int m;
    if (!rst_n) begin
      selected <= '0; individual <= '0;
      chosen_partner_sys <= '0; chosen_partner_key <= '0;
      c_wrong_partner <= '0; c_selections <= '0;
    end else begin
      chosen_partner_sys <= ref_sys;
      chosen_partner_key <= ref_key;

      for (m = 0; m < MAX_MEMBERS; m++) begin
        if (!partner_valid[m]) begin
          // Heard nothing. Not selected, and not individual either --
          // we simply do not know yet.
          selected[m]   <= 1'b0;
          individual[m] <= 1'b0;
        end else begin
          // Four conditions, all required.
          automatic logic ok;
          ok = have_ref &&
               (our_actor[m].key    == ref_akey) &&   // local intent
               (rx_actor[m].sys_id  == ref_sys)  &&   // same device
               (rx_actor[m].key     == ref_key);      // same far aggregate

          if (!ok && (rx_actor[m].sys_id != ref_sys)) begin
            // A DIFFERENT DEVICE at the other end of this cable.
            // Section 11: every physical source says this member is
            // healthy, and it must not be aggregated.
            c_wrong_partner <= c_wrong_partner + 1;
            individual[m]   <= 1'b1;
          end else begin
            individual[m] <= 1'b0;
          end

          if (ok && !selected[m]) c_selections <= c_selections + 1;
          selected[m] <= ok;
        end
      end
    end
  end

endmodule

Classification: an all-pairs consistency check reduced to a comparison against a reference. One cycle, combinational reference selection.

What it teaches: that aggregation requires agreement on four fields and only one of them is local. Our own key expresses our intent to aggregate these ports. The partner's system id and key express the far end's, and a design that checks only the local key aggregates any four cables an operator configured regardless of where they go — which is static configuration with extra frames.

And it teaches that individual is a distinct outcome from "not selected", which a two-state design collapses. Not selected means we do not know yet — no LACPDU has arrived. individual means we know, and the answer is no: there is a device at the other end and it is the wrong one. The first is a state to wait in; the second is a fault to report.

Deliberately simplified: the reference is the lowest-numbered port with a partner, which means a single mis-cabled cable on the lowest port makes the other three look wrong. Production designs take the majority pairing as the reference, so one anomalous cable is excluded rather than three. The comparison is the same; the choice of reference decides which side of it the fault lands on, and getting it backwards turns a one-member fault into a three-member one.

Production implication: chosen_partner_sys is the single most useful line of an aggregate's status output and is frequently absent. It answers "what is actually at the other end of this bundle" with a MAC address, and comparing it against the neighbour an operator believes is there resolves Section 11 in one read — without it, the mis-cable is diagnosed by walking to the rack.

9. The Selection Logic, and Why Both Ends Must Agree

Two devices independently choosing which of their ports to aggregate must reach the same answer, and there is no negotiation step in which they compare answers.

The mechanism is a tie-break rather than a conversation: one end is designated the decider, and both ends compute the decision that end would make.

StepWhat decides
1the lower system priority wins; ties broken by the lower system id
2the winner's port priorities order the candidate ports
3ports beyond the aggregate's capacity become standby
4both ends apply the winner's ordering, not their own

Step 4 is the whole point and it is why the system priority exists at all. If each end ordered ports by its own priorities, a six-port bundle limited to four members would produce different four-port subsets at the two ends — and the two subsets' intersection would be the actual aggregate, with the rest carrying traffic in one direction only.

So the protocol does not agree on a set; it agrees on who decides, and derives the set from that. Which is a much smaller thing to agree on, and it needs no round trip: both ends already have both system ids, from the actor TLVs they have exchanged.

And the standby state matters more than it looks. A six-port bundle with a four-member limit has two ports fully negotiated, synchronised, and not distributingheld in reserve. When a member fails, a standby port is promoted, so the aggregate returns to four members without Chapter 15.1 §7's detection cost being followed by a capacity loss. Static configuration has no equivalent: a sixth cable is either a member or it is not connected.

10. RTL 5 — Detecting a Mis-Cable

Section 8's key matcher produces the evidence. This module turns it into a report an operator can act on.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// miscable_detector -- names the specific failure static configuration
// cannot see: a member whose far end is a different device.
//
// Every physical source in 15.1 section 6 reports this member healthy.
// The ONLY evidence is the actor system id in its LACPDUs, which no
// local measurement produces.
// -----------------------------------------------------------------------
module miscable_detector
  import lacp_pkg::*;
#(
  parameter int CONFIRM_PDUS = 3      // three consecutive, not one
)(
  input  logic     clk,
  input  logic     rst_n,

  input  logic     pdu_valid  [MAX_MEMBERS],
  input  lacp_id_t rx_actor   [MAX_MEMBERS],
  input  logic [47:0] expected_partner_sys,   // the majority pairing
  input  logic     expected_valid,

  input  logic [MAX_MEMBERS-1:0] port_enabled,   // 15.1's physical up

  output logic [MAX_MEMBERS-1:0] miscabled,
  output logic [47:0]            found_sys [MAX_MEMBERS],
  output logic [31:0]            c_miscables,
  output logic [15:0]            traffic_at_risk_pct
);

  logic [3:0] streak [MAX_MEMBERS];

  always_ff @(posedge clk or negedge rst_n) begin
    int m, n_bad, n_up;
    if (!rst_n) begin
      for (m = 0; m < MAX_MEMBERS; m++) begin
        streak[m] <= '0; found_sys[m] <= '0;
      end
      miscabled <= '0; c_miscables <= '0; traffic_at_risk_pct <= '0;
    end else begin
      for (m = 0; m < MAX_MEMBERS; m++) begin
        if (pdu_valid[m] && expected_valid) begin
          if (rx_actor[m].sys_id != expected_partner_sys) begin
            found_sys[m] <= rx_actor[m].sys_id;
            if (streak[m] != 4'hF) streak[m] <= streak[m] + 1'b1;
            if (streak[m] == CONFIRM_PDUS-1) begin
              miscabled[m] <= 1'b1;
              c_miscables  <= c_miscables + 1;
            end
          end else begin
            streak[m]    <= '0;
            miscabled[m] <= 1'b0;
          end
        end
        // Losing the port clears the finding: an absent cable is a
        // different fault with a different remedy.
        if (!port_enabled[m]) begin
          streak[m]    <= '0;
          miscabled[m] <= 1'b0;
        end
      end

      // What fraction of the aggregate's traffic a STATIC config would
      // be sending to the wrong device. This is the number that makes
      // the finding urgent rather than interesting.
      n_bad = 0; n_up = 0;
      for (m = 0; m < MAX_MEMBERS; m++) begin
        if (port_enabled[m]) n_up = n_up + 1;
        if (miscabled[m])    n_bad = n_bad + 1;
      end
      traffic_at_risk_pct <= (n_up == 0) ? 16'd0
                           : 16'((n_bad * 100) / n_up);
    end
  end

endmodule

Classification: a confirmation counter over a comparison. It adds no capability to Section 8 and adds the reporting that makes it usable.

What it teaches: that the confirmation streak exists because a single mismatched LACPDU is normal. During bring-up, before a partner is established, and immediately after a topology change, a device can legitimately receive one LACPDU whose actor differs from the expectation — the expectation was itself derived from partial information. Three consecutive is 3 seconds in fast mode and is unambiguous.

And it teaches that traffic_at_risk_pct is the output that changes the conversation. miscabled[2] is a finding. "25% of this aggregate's traffic would be going to the wrong device" is an incident, and it is the same information with the arithmetic done. A design that reports the bit and not the percentage has left the most important step to a human under time pressure.

Deliberately simplified: expected_partner_sys is an input, and deriving it well is the subtle part — the majority pairing across the bundle, not the first port's, for the reason Section 8's simplification described. On a two-member aggregate with one mis-cable there is no majority, and the correct behaviour is to report both as unresolved rather than to pick one.

Production implication: found_sys[m] is the MAC address of the device actually at the other end, and Chapter 5.3's OUI in its top 24 bits usually names the vendor. An operator given "port 8 reaches 00:1B:21:… and ports 5–7 reach 00:25:90:…" has both the fault and a strong hint about which rack to walk to — and given only "member 8 excluded" has neither.

11. The Mis-Cable Static Configuration Cannot See

This is the failure the whole protocol justifies itself with, and it is worth walking through slowly because every local indicator is green.

Four cables leave switch A intended for switch B. Three reach B. The fourth reaches switch C — a different device, in a different rack, plugged into the wrong port during a maintenance window.

What each detection source reports:

SourceReportsCorrect?
Chapter 15.1 §6 phy_link_upupyes — there is a device there
Chapter 15.1 §6 pcs_block_locklockedyes — the PHY is receiving valid blocks
Chapter 11.2's auto-negotiation10 Gb/s full duplexyes — it negotiated correctly
Chapter 15.1 §15 conformanthighyes — nothing this device did is wrong
the operator's configurationfour membersit records the intention
c_miscables1the only source that knows

And under static configuration, the consequences:

25% of the aggregate's traffic is delivered to switch C, which has no entry for those destinations and floods them — Chapter 12.4's 23× amplification on a 24-port switch, applied to traffic that had a perfectly good destination.

Switch B sees the aggregate's source addresses on three ports rather than four, which is harmless.

Switch C sees them on one port and learns themChapter 12.2so C's forwarding table now points at A for addresses that live behind B, and traffic C receives for those destinations goes to A and is forwarded again. A forwarding loop is not created, because the frames do reach their destination eventually; what is created is a path that carries a quarter of the traffic through a device nobody intended.

And return traffic is worse. Switch C, distributing across its own aggregate to A, has no reason to use the mis-cabled port at all — so the failure is asymmetric: A sends into C and C does not send back, which makes the octet counters on the two ends of that cable wildly different and is the one local signal that could have hinted at it.

==

Switch A has a four-port aggregate. Three cables reach switch B as intended and one reaches switch C, an unrelated device. Under static configuration all four are members, so a quarter of the aggregate's traffic is delivered to C, which has no forwarding entry for those destinations and floods them at Chapter 12.4's twenty-three-fold amplification, while also learning A's source addresses on that port so its own table now points back at A. Return traffic never uses that cable, because C's distributor has no reason to select it, which leaves the two ends' octet counters on that single cable wildly asymmetric — the one local hint available, and one nobody graphs. Under LACP, three LACPDUs establish that the actor system id on that port differs from the other three, the member is marked individual and excluded within three seconds, and the aggregate runs as a healthy three-member LAG at seventy-five percent capacity with an alarm naming the port and the MAC address it actually reaches.Switch A4-port aggregatePort 8 reaches Cevery indicator greenStatic: 4 members25% to switch CC floods it23x, and learns usLACP: 3 PDUs3 s in fast modeMember excludedindividual, not selectedfound_sys names Cthe OUI names the vendor12
Figure 3 — the mis-cable, and the two paths its traffic takes under static configuration and under LACP.

LACP refuses the member. Three LACPDUs — 3 seconds in fast mode — establish that the actor system id on that port is not the one on the other three, individual is set, selected is clear, and the aggregate runs as a healthy three-member LAG at 75% capacity while an alarm names the port and the MAC address it actually reaches.

Which is the whole trade, in one line: 1184 bit/s buys the difference between a three-member aggregate with an alarm and a four-member aggregate quietly misdelivering a quarter of its traffic.

12. RTL 6 — The Periodic Timer and Timeout

Two timers per port, in a three-to-one relationship that is the protocol's only tolerance for loss.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// lacp_periodic_timer -- transmit cadence and receive expiry.
//
// The transmit period is chosen by the PARTNER's timeout bit, not by
// ours: a partner that says it wants fast timeouts is telling us to
// transmit every second. This inversion is the most commonly
// implemented backwards thing in the protocol.
// -----------------------------------------------------------------------
module lacp_periodic_timer
  import lacp_pkg::*;
#(
  parameter int CLK_HZ = 500000000
)(
  input  logic       clk,
  input  logic       rst_n,

  input  logic       port_enabled,
  input  logic       our_activity,        // active or passive
  input  logic       partner_activity,
  input  logic       partner_wants_fast,  // partner state bit 1
  input  logic       we_want_fast,        // our own preference, sent out

  input  logic       pdu_received,

  output logic       tx_now,
  output logic       rx_timeout,
  output logic       rx_expired_once,
  output logic [31:0] c_timeouts,
  output logic [31:0] c_tx_periods,
  output logic [31:0] since_last_rx_ms
);

  localparam int FAST_PERIOD = CLK_HZ;          // 1 s
  localparam int SLOW_PERIOD = CLK_HZ * 30;     // 30 s
  localparam int MS_DIV      = CLK_HZ / 1000;

  logic [35:0] tx_cnt, rx_cnt, ms_cnt;
  logic [35:0] tx_period, rx_limit;

  // TRANSMIT at the rate the PARTNER asked for.
  assign tx_period = partner_wants_fast ? 36'(FAST_PERIOD)
                                        : 36'(SLOW_PERIOD);
  // EXPIRE after three of OUR chosen periods -- we asked for this rate,
  // so we are entitled to expect it.
  assign rx_limit  = we_want_fast ? 36'(3 * FAST_PERIOD)
                                  : 36'(3 * SLOW_PERIOD);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      tx_cnt <= '0; rx_cnt <= '0; ms_cnt <= '0;
      tx_now <= 1'b0; rx_timeout <= 1'b0; rx_expired_once <= 1'b0;
      c_timeouts <= '0; c_tx_periods <= '0; since_last_rx_ms <= '0;
    end else begin
      tx_now <= 1'b0;

      if (!port_enabled) begin
        tx_cnt <= '0; rx_cnt <= '0; rx_timeout <= 1'b0;
      end else begin
        // A passive port transmits only if its partner is active.
        // Two passive ports never aggregate -- section 13's table.
        if (our_activity || partner_activity) begin
          if (tx_cnt >= tx_period - 1) begin
            tx_cnt       <= '0;
            tx_now       <= 1'b1;
            c_tx_periods <= c_tx_periods + 1;
          end else tx_cnt <= tx_cnt + 1'b1;
        end

        if (pdu_received) begin
          rx_cnt          <= '0;
          rx_timeout      <= 1'b0;
          ms_cnt          <= '0;
          since_last_rx_ms <= '0;
        end else begin
          if (rx_cnt < rx_limit) begin
            rx_cnt <= rx_cnt + 1'b1;
            if (rx_cnt == rx_limit - 1) begin
              rx_timeout      <= 1'b1;
              rx_expired_once <= 1'b1;
              c_timeouts      <= c_timeouts + 1;
            end
          end
          if (ms_cnt == MS_DIV-1) begin
            ms_cnt <= '0;
            since_last_rx_ms <= since_last_rx_ms + 1;
          end else ms_cnt <= ms_cnt + 1'b1;
        end
      end
    end
  end

endmodule

Classification: two independent counters with an asymmetric parameterisation. No datapath, one bit of output that matters.

What it teaches: that the transmit rate is chosen by the partner and the expiry by us, and getting the two the same way round breaks the protocol quietly. A device that transmits at its own preferred rate while its partner expects the other either floods a slow partner with 30× the frames it asked for, or starves a fast partner and is timed out every three seconds. The second is worse and it is the one that occurs: a fast-mode partner paired with a device transmitting slowly sees a timeout every 3 seconds, detaches, re-attaches, and produces Section 14's churn indefinitely.

And it teaches that since_last_rx_ms is worth its 32 bits. Is the partner talking to us is answered by a counter; "how long ago" is answered only by a timestamp, and it is what separates a partner that stopped a moment ago from one that has never spoken. A freshly configured port and a failed neighbour both read c_received = 0.

Deliberately simplified: the 36-bit counters at 500 MHz cover 30 seconds with room, and a production design would use a prescaled millisecond tick rather than counting core clocks — 15 billion cycles for a slow timeout is a lot of toggling for a timer whose resolution requirement is seconds.

Production implication: c_timeouts rising on a link whose physical layer is clean is the one-way failure Chapter 15.1 §6's proto_alive input was reserved for. Our transmit direction is dead: the far end hears nothing, stops replying, our receive timer expires. Both PHYs report up, both PCSs are locked, and the only evidence in the entire system is this counter.

13. Timeout Arithmetic — Fast and Slow

The choice between the two modes looks like a bandwidth trade and is not.

fastslowRatio
periodic transmit1 s30 s30×
timeout3 s90 s30×
frames per second10.033
bandwidth1184 bit/s39.5 bit/s30×
of a 1 Gb/s link1.18 × 10⁻⁴%3.95 × 10⁻⁶%
of a 100 Gb/s link1.18 × 10⁻⁶%3.95 × 10⁻⁸%

Fast mode costs one part in a million of a gigabit link. There is no bandwidth case for slow mode at any line rate this track has covered.

What the choice actually costs is measured in octets lost during a failure the physical layer cannot see:

Detection1 Gb/s10 Gb/s100 Gb/s
fast — 3 s0.38 GB3.75 GB37.5 GB
slow — 90 s11.25 GB112.5 GB1125 GB

And both numbers are enormous, which is the point that reframes the whole comparison. LACP is not the failure detector for an ordinary link failure — Chapter 15.1 §7 established that a PHY interrupt does that in 10 µs, six orders of magnitude faster. LACP's timeout is the detector of last resort for the failures nothing else sees, and for those the choice is between losing 37.5 GB and losing 1125 GB.

So the real argument for slow mode is not bandwidth — it is CPU. On a switch whose LACP runs in software, a 48-port chassis in fast mode processes 48 LACPDUs per second and must transmit 48, with jitter tolerances measured in seconds. That is trivial today and was not always, and slow mode is the default in many implementations for a reason that expired around the time the hardware stopped being the constraint.

The recommendation that falls out is unambiguous: fast mode, on every aggregate, unless something specific forbids it — and the number that justifies it is the 1125 GB, not the 39.5 bit/s.

One more row completes the picture, because the activity bit interacts with all of this:

Our activityPartner activityResult
activeactiveboth transmit; normal
activepassivewe transmit, they reply — works
passiveactivethey transmit, we reply — works
passivepassiveneither ever transmits — no aggregate, ever

The last row is a configuration that produces a bundle of links that never aggregate and never report an error, because nothing has gone wrong: both ends are waiting to be spoken to. At least one end must be active, and a design whose default is passive has shipped an aggregate that only works if the neighbour was configured differently.

14. RTL 7 — Churn Detection

A member that repeatedly attaches and detaches is worse than one that stays down, and the protocol has a name for the condition.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// lacp_churn_detector -- reports a port that fails to reach a stable
// synchronised state within a bounded time.
//
// Churn is the protocol-level version of 15.1 section 6's flapping
// member, and it has an additional cause that a flapping cable does
// not: two ends that disagree in a way that oscillates.
// -----------------------------------------------------------------------
module lacp_churn_detector
  import lacp_pkg::*;
#(
  parameter int CHURN_LIMIT   = 2000000000/4,  // ~1 s at 500 MHz
  parameter int WINDOW_CYCLES = 5000000000/10  // ~1 s window
)(
  input  logic       clk,
  input  logic       rst_n,

  input  logic       port_enabled,
  input  logic       actor_sync,
  input  logic       partner_sync,
  input  logic       attach_pulse,
  input  logic       detach_pulse,

  output logic       actor_churn,
  output logic       partner_churn,
  output logic [15:0] attaches_per_window,
  output logic [31:0] c_churn_events,
  output logic [31:0] longest_unsync_ms
);

  logic [31:0] a_unsync, p_unsync, win, ms_div;
  logic [15:0] att_cnt;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      a_unsync <= '0; p_unsync <= '0; win <= '0; ms_div <= '0;
      att_cnt <= '0; attaches_per_window <= '0;
      actor_churn <= 1'b0; partner_churn <= 1'b0;
      c_churn_events <= '0; longest_unsync_ms <= '0;
    end else if (!port_enabled) begin
      a_unsync <= '0; p_unsync <= '0;
      actor_churn <= 1'b0; partner_churn <= 1'b0;
    end else begin
      // Actor churn: WE cannot reach synchronised. Usually our own
      // selection logic oscillating -- section 8's reference moving.
      if (actor_sync) begin
        a_unsync    <= '0;
        actor_churn <= 1'b0;
      end else begin
        a_unsync <= a_unsync + 1;
        if (a_unsync == CHURN_LIMIT-1) begin
          actor_churn    <= 1'b1;
          c_churn_events <= c_churn_events + 1;
        end
        if (a_unsync > longest_unsync_ms * 500000)
          longest_unsync_ms <= a_unsync / 500000;
      end

      // Partner churn: THEY cannot reach synchronised. A different
      // fault entirely and usually theirs, which is why it is a
      // separate bit rather than a shared "not synchronised".
      if (partner_sync) begin
        p_unsync      <= '0;
        partner_churn <= 1'b0;
      end else begin
        p_unsync <= p_unsync + 1;
        if (p_unsync == CHURN_LIMIT-1) begin
          partner_churn  <= 1'b1;
          c_churn_events <= c_churn_events + 1;
        end
      end

      if (attach_pulse) att_cnt <= att_cnt + 1'b1;

      if (win == WINDOW_CYCLES-1) begin
        attaches_per_window <= att_cnt;
        att_cnt <= '0;
        win     <= '0;
      end else win <= win + 1'b1;
    end
  end

endmodule

Classification: two independent stall watchdogs plus a rate counter. It reports and changes nothing.

What it teaches: that actor churn and partner churn are separate bits because they name different parties, and the whole of Module 14's counter discipline says why. We cannot synchronise is a local fault — Section 8's reference oscillating, a selection function disagreeing with itself. "They cannot synchronise" is theirs. One combined not synchronised bit sends every investigation to the same place, and half of them are at the wrong end of the cable.

And it teaches that attaches_per_window is the number that distinguishes churn from a slow start. A member that takes ten seconds to attach once is a bring-up. A member attaching four times a second is costing Chapter 15.2 §12's rebalance eight times a second — 74.9% of flows relocated on each, with a modulo reduction — and the aggregate spends more time redistributing than forwarding.

Deliberately simplified: the millisecond conversion divides by a hard-coded 500 000, which ties the module to a 500 MHz clock. A production design takes a millisecond tick as an input, because a switch's LACP logic frequently runs on a slower management clock than its datapath.

Production implication: longest_unsync_ms on a healthy network is the empirical basis for CHURN_LIMIT, exactly as Chapter 14.4 §13's longest_stall was for STALL_LIMIT. A threshold chosen from first principles is a guess; one set several times above the observed maximum will not cry wolf — and a churn detector that fires spuriously is disabled, taking the real detections with it.

15. RTL 8 — Conformance for a Negotiated Aggregate

The monitor's job is to separate what this device did wrong from what the negotiation found wrong at the other end.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// lacp_conformance_monitor -- one bit per aggregate.
//
// It distinguishes OUR faults from FINDINGS about the far end. A
// mis-cable is not a defect in this device and must not clear its
// conformance bit -- it is the protocol working.
// -----------------------------------------------------------------------
module lacp_conformance_monitor
  import lacp_pkg::*;
(
  input  logic       clk,
  input  logic       rst_n,

  // Our faults.
  input  logic       tx_period_wrong,     // ignored the partner's rate
  input  logic       sync_without_mirror, // set sync without they_see_us
  input  logic       distributed_early,   // distributing before partner collecting
  input  logic       both_passive,        // configuration: never aggregates
  input  logic       no_active_end,

  // Findings about the far end -- NOT our faults.
  input  logic [MAX_MEMBERS-1:0] miscabled,
  input  logic       partner_churn,
  input  logic       partner_timeout,

  output logic       conformant,
  output logic [7:0] fault_vector,
  output logic [7:0] finding_vector,
  output logic       aggregate_degraded
);

  logic v_period, v_mirror, v_early, v_passive;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      v_period <= 1'b0; v_mirror <= 1'b0;
      v_early <= 1'b0; v_passive <= 1'b0;
    end else begin
      if (tx_period_wrong)     v_period <= 1'b1;
      if (sync_without_mirror) v_mirror <= 1'b1;
      if (distributed_early)   v_early  <= 1'b1;
      // Standing configuration property: at least one end active.
      v_passive <= both_passive || no_active_end;
    end
  end

  assign conformant = !(v_period || v_mirror || v_early || v_passive);

  assign fault_vector   = {4'b0000, v_passive, v_early, v_mirror, v_period};
  assign finding_vector = {5'b00000, partner_timeout, partner_churn,
                           |miscabled};

  // A separate output: the aggregate is smaller than configured for a
  // reason the protocol UNDERSTOOD. This is the alarm an operator
  // wants, and it is not a conformance failure.
  assign aggregate_degraded = (|miscabled) || partner_churn || partner_timeout;

endmodule

Classification: a two-vector fault aggregator. One vector is about us and one is about the world.

What it teaches: that a mis-cable must not clear this device's conformance bit, and the temptation to make it do so is strong. A quarter of the aggregate is excluded, an alarm is firing, and the natural instinct is for conformant to go low. But nothing this device did is wrong — it detected a fault correctly and responded correctly, and a conformance bit that goes low on a correct detection cannot be used to answer "is my LACP implementation working".

And it teaches that aggregate_degraded is what operations actually alarms on, which is neither vector alone. It fires on findings and not on faults, so an operator gets paged for the cable and an engineer gets the fault vector for the implementation. Merging them produces one alarm that means two unrelated things.

Deliberately simplified: sync_without_mirror is presented as an input, and producing it means checking that our_state.synchronised is never set while they_see_us is low. That check is one comparison and it catches the single most damaging implementation bug in the protocol — a device that synchronises on the partner's sync bit alone, which aggregates a mis-cabled member enthusiastically.

Production implication: the two vectors want different retention. fault_vector should be sticky — an implementation bug that occurred once is worth knowing about for ever. finding_vector should be live, because a mis-cable that has been fixed is no longer a finding and an operator needs the alarm to clear. A design that makes both sticky produces an aggregate that reports a mis-cable months after the cable was moved.

16. The Marker Protocol and Ordering During a Move

Chapter 15.1 §8 solved the reordering problem at a failover by waiting for queues to drain. 802.1AX defines a mechanism that does it exactly instead of conservatively, and almost nobody implements it.

The problem: moving a flow from member 2 to member 3 reorders it if member 2 still holds frames of that flow. Chapter 15.1 §8's answer was to wait until every remaining queue is empty — conservative, bounded by a parameter, and it times out on a congested aggregate.

The Marker protocol's answer is to ask.

StepWhat happens
1stop sending the flow on member 2
2send a Marker PDU on member 2 — subtype 0x02, same 128-octet frame shape
3the far end replies Marker Response on the same member
4the response proves every earlier frame on member 2 has been received
5start sending the flow on member 3

Step 4 is the exact condition Chapter 15.1 §8 approximated. The Marker is an ordinary frame in the same queue as the data, so a response to it means everything queued ahead of it has left — which is precisely, and only, what the ordering constraint requires.

Chapter 15.1 §8's drainMarker
conditionevery remaining queue emptythis member's earlier frames delivered
precisionconservativeexact
duration on an idle aggregateimmediateone round trip — ~1 µs on a short link
duration on a congested aggregatetimes out at 1 msas long as the queue takes
reordering riskon timeoutnone
requires the far endnoyes — it must implement Marker Response

The last row is why it is rare. Marker is optional; a device must respond to a Marker if it receives one, but the standard does not require anybody to send them — so an implementation that relies on Marker must fall back when the far end does not respond, which means implementing the drain anyway.

And that is the honest reason the drain is what ships: it works unilaterally. Chapter 15.1 §8's c_drain_timeout is the price of not needing the neighbour's cooperation, and the Marker protocol is the mechanism that would remove it if both ends could be assumed.

17. What LACP Costs Against Static Configuration

Put the whole comparison in one place, because the protocol is frequently disabled on the grounds that static is simpler and the grounds are true.

staticLACP
configuration efforttwo ends, independentlytwo ends, independently
bandwidthzero1184 bit/s fast, 39.5 slow
state per porta membership bit~40 octets — two lacp_id_t, two state bytes, two timers
logicnonea parser, a builder, a mux machine, a matcher
time to bring a member upimmediate~2 s
frames lost bringing a member upup to a round tripzero
member count mismatchundetectedrefused
mis-cabled memberundetected — 25% misdeliveredrefused in 3 s, MAC reported
far end not aggregatingundetected — table flapsrefused
one-way failureundetecteddetected at timeout
standby membersnot possiblenegotiated, promoted on failure
ordering during a movedrain onlydrain, or Marker if both ends have it
fails whenthe cabling is wrongthe far end does not run it

The last row is the only real argument against it and it is a shrinking one. A device that does not run LACP will not aggregate with one that requires it, so an aggregate to a host with a simple NIC bond, or to an appliance with a fixed configuration, may have to be static.

And the state cost is worth putting in proportion against the batch:

MechanismStateWhat it buys
LACP — this chapter~320 octets, 8 portsthe mis-cable, the one-way failure, the mismatch
aggregation — Chapter 15.1 §172.19 KiBsurvives a member failure
distribution — Chapter 15.2 §183.8 KiBthe failover moves 25% instead of 74.9%
PFC — Chapter 14.4 §181.45 MiBcollateral 88% → 11%

LACP is the smallest mechanism in Module 15 and it is the only one that can see outside the device. Everything else in the module reasons about state this switch holds; LACP's entire value is that it carries a statement from somebody else, and 320 octets is what holding that statement costs.

Which is the module's closing observation and it generalises past Ethernet: the cheapest information a distributed system can acquire is usually the information it cannot derive locally at any price.

18. Properties Worth Asserting, and One Worth Refusing

The properties divide by what they protect: the frame, the parse, the mux machine, the selection, the timers, and the configuration.

Group 1 — the frame.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P1. The LACPDU is addressed to the Slow Protocols multicast and
// carries the Slow Protocols EtherType and the LACP subtype.
property p_frame_identity;
  @(posedge clk) disable iff (!rst_n)
  tx_sop |-> ##12 (tx_octet == SLOW_ET[15:8]) ##1 (tx_octet == SLOW_ET[7:0])
                  ##1 (tx_octet == LACP_SUB);
endproperty

// P2. The frame is exactly 124 body octets -- 128 with the FCS. No
// padding: the payload is well above 5.6's 46-octet minimum.
property p_frame_length;
  @(posedge clk) disable iff (!rst_n)
  tx_sop |-> ##123 tx_eop;
endproperty

// P3. The source address is the DEVICE's identity, not the port's.
// All of a device's ports must claim one identity or the far end
// sees several one-member aggregates.
property p_sa_is_device_id;
  @(posedge clk) disable iff (!rst_n)
  tx_sop |-> ##6 (tx_octet == actor.sys_id[47:40]);
endproperty

// P4. The actor TLV always carries our current state, not a snapshot
// taken when the request was queued.
property p_actor_state_is_current;
  @(posedge clk) disable iff (!rst_n)
  (tx_valid && (idx == 7'd32)) |-> (tx_octet == actor_state);
endproperty

// P5. The builder never overlaps frames.
property p_no_overlap;
  @(posedge clk) disable iff (!rst_n)
  tx_sop |-> !tx_sop throughout (tx_eop [->1]);
endproperty

Group 2 — the parse.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P6. Nothing is committed on a bad FCS.
property p_no_commit_bad_fcs;
  @(posedge clk) disable iff (!rst_n)
  (rx_eop && !rx_fcs_ok) |-> !pdu_valid;
endproperty

// P7. A non-LACP slow protocol is not an error. Marker frames arrive
// legitimately at a device that does not implement them.
property p_marker_is_not_an_error;
  @(posedge clk) disable iff (!rst_n)
  (rx_eop && da_ok && et_ok && !sub_ok) |-> !$changed(c_bad_fcs);
endproperty

// P8. pdu_valid is one cycle wide -- the mux machine reacts to a pulse.
property p_commit_is_a_pulse;
  @(posedge clk) disable iff (!rst_n)
  pdu_valid |=> !pdu_valid;
endproperty

// P9. they_see_us is exactly the three-field comparison it claims.
property p_mirror_comparison;
  @(posedge clk) disable iff (!rst_n)
  pdu_valid |-> (they_see_us == ((rx_partner.sys_id  == our_actor.sys_id) &&
                                 (rx_partner.key     == our_actor.key)    &&
                                 (rx_partner.port_id == our_actor.port_id)));
endproperty

Group 3 — the mux machine.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P10. Collecting is never enabled after distributing. 15.1 section 3's
// ordering, and the reason no frame is lost bringing a member up.
property p_collect_before_distribute;
  @(posedge clk) disable iff (!rst_n)
  $rose(distributing) |-> collecting;
endproperty

// P11. We never distribute unless the partner says it is collecting.
property p_distribute_needs_partner_collecting;
  @(posedge clk) disable iff (!rst_n)
  distributing |-> rx_actor_state.collecting;
endproperty

// P12. Losing the port, the partner or the selection detaches
// immediately. There is no partial state to linger in.
property p_loss_detaches;
  @(posedge clk) disable iff (!rst_n)
  (!port_enabled || rx_timeout || !selected) |=> (mux == MUX_DETACHED);
endproperty

// P13. Detached means neither role is active.
property p_detached_is_inert;
  @(posedge clk) disable iff (!rst_n)
  (mux == MUX_DETACHED) |-> (!collecting && !distributing);
endproperty

// P14. WAITING lasts exactly AGGREGATION_WAIT, so several ports
// selecting together settle together.
property p_waiting_is_timed;
  @(posedge clk) disable iff (!rst_n)
  $rose(mux == MUX_WAITING) |-> ##AGGREGATION_WAIT (mux == MUX_ATTACHED);
endproperty

// P15. Synchronised requires BOTH they_see_us and their sync bit.
// Checking only the second aggregates a mis-cabled member.
property p_sync_needs_mirror;
  @(posedge clk) disable iff (!rst_n)
  our_state.synchronised |-> they_see_us;
endproperty

Group 4 — the selection.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P16. A selected port's partner is the same device as the reference.
property p_selected_same_partner;
  @(posedge clk) disable iff (!rst_n)
  selected[m] |-> (rx_actor[m].sys_id == chosen_partner_sys);
endproperty

// P17. And the same far-end aggregation key.
property p_selected_same_key;
  @(posedge clk) disable iff (!rst_n)
  selected[m] |-> (rx_actor[m].key == chosen_partner_key);
endproperty

// P18. A port whose partner is a DIFFERENT device is individual, never
// selected. The mis-cable, as a property.
property p_wrong_partner_is_individual;
  @(posedge clk) disable iff (!rst_n)
  (partner_valid[m] && (rx_actor[m].sys_id != chosen_partner_sys))
    |=> (individual[m] && !selected[m]);
endproperty

// P19. Not having heard is distinct from having heard the wrong thing.
property p_silence_is_not_individual;
  @(posedge clk) disable iff (!rst_n)
  !partner_valid[m] |-> !individual[m];
endproperty

// P20. A mis-cable finding needs CONFIRM_PDUS consecutive
// disagreements -- one is normal during bring-up.
property p_miscable_needs_confirmation;
  @(posedge clk) disable iff (!rst_n)
  $rose(miscabled[m]) |-> ($past(streak[m]) == CONFIRM_PDUS-1);
endproperty

// P21. Losing the port clears the finding: an absent cable is a
// different fault with a different remedy.
property p_port_down_clears_miscable;
  @(posedge clk) disable iff (!rst_n)
  !port_enabled[m] |=> !miscabled[m];
endproperty

Group 5 — the timers.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P22. We transmit at the rate the PARTNER asked for.
property p_tx_rate_follows_partner;
  @(posedge clk) disable iff (!rst_n)
  partner_wants_fast |-> (tx_period == FAST_PERIOD);
endproperty

// P23. We expire after three of OUR chosen periods.
property p_expiry_follows_us;
  @(posedge clk) disable iff (!rst_n)
  we_want_fast |-> (rx_limit == 3*FAST_PERIOD);
endproperty

// P24. Any received PDU resets the receive timer.
property p_rx_resets_timer;
  @(posedge clk) disable iff (!rst_n)
  pdu_received |=> (rx_cnt == '0);
endproperty

// P25. A passive port with a passive partner never transmits. This is
// a legal configuration that never aggregates, and the property makes
// it visible rather than mysterious.
property p_passive_pair_is_silent;
  @(posedge clk) disable iff (!rst_n)
  (!our_activity && !partner_activity) |-> !tx_now;
endproperty

// P26. Churn is raised only after CHURN_LIMIT of continuous unsync.
property p_churn_needs_persistence;
  @(posedge clk) disable iff (!rst_n)
  $rose(actor_churn) |-> ($past(a_unsync) == CHURN_LIMIT-1);
endproperty

// P27. Synchronising clears the churn evidence.
property p_sync_clears_churn;
  @(posedge clk) disable iff (!rst_n)
  actor_sync |=> (a_unsync == '0);
endproperty

Group 6 — the configuration and the monitor.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P28. Standing property: at least one end is active.
property p_an_active_end_exists;
  @(posedge clk) disable iff (!rst_n)
  !no_active_end;
endproperty

// P29. A mis-cable is a FINDING, not a fault. It must not clear this
// device's conformance bit -- the protocol worked.
property p_finding_is_not_a_fault;
  @(posedge clk) disable iff (!rst_n)
  (|miscabled && (fault_vector == 8'h00)) |-> conformant;
endproperty

// P30. But it must raise the degraded alarm.
property p_finding_degrades;
  @(posedge clk) disable iff (!rst_n)
  |miscabled |-> aggregate_degraded;
endproperty

// P31. conformant is exactly the conjunction of OUR faults.
property p_conformant_definition;
  @(posedge clk) disable iff (!rst_n)
  conformant <-> (fault_vector == 8'h00);
endproperty

Every property above is about state this device holds or messages it has actually received. P15 is the one that prevents Section 11's mis-cable from being aggregated, and it is deliberately phrased as a check on our own synchronised bit rather than as a claim about the partner — which is the subject of the property this chapter refuses.

19. Verification Scenarios

Sixty-nine scenarios. The ones that matter show a link that every physical source calls healthy and the protocol refuses.

The frame

#ScenarioExpected
1Build an LACPDU124 body octets, 128 with FCS, 148 on the wire
2SameDA 01:80:C2:00:00:02, EtherType 0x8809
3Samesubtype 0x01, version 0x01
4Sameno padding — the payload is 110 octets
5SAthe device's MAC, not the port's
6Four ports of one deviceall four claim the same system id
7A design using per-port SAsfar end sees four one-member aggregates
8Actor TLV at offset 16type 0x01, length 20
9Partner TLV at offset 36type 0x02, length 20
10Fast mode1184 bit/s — 1.18 × 10⁻⁴% of a 1 Gb/s link
11Slow mode39.5 bit/s

The parse

#ScenarioExpected
12Well-formed LACPDUpdu_valid, one cycle
13Bad FCSno commit, c_bad_fcs + 1
14Subtype 0x02 — Markerc_wrong_subtype, not an error
15Same, counted as bad FCSan error counter that rises in normal operation
16Partner TLV matches usthey_see_us
17Partner TLV names a different portc_mismatched_mirror + 1
18Same, once, after a state changenormal — the mirror is one round trip stale
19Same, continuouslythe mis-cable
20An unknown TLV before the terminatorhard-coded parser mis-parses; a chain-walker skips it

The mux machine

#ScenarioExpected
21Port comes up, selectedDETACHED → WAITING
22After AGGREGATION_WAIT→ ATTACHED, collecting = 1, distributing = 0
23Partner reports collecting and synchronised→ COLLECT_DIST
24Four ports up within 1 ms, with WAITINGone membership change
25Same, without WAITINGfour changes — Chapter 15.2 §12 runs four times
26Same, modulo reduction74.9% of flows relocated, four times
27distributing set before collectingP10 fails
28Partner not yet collectingwe do not distribute
29Physical port drops→ DETACHED immediately
30Receive timeout→ DETACHED
31Same, with an EXPIRED statekeeps collecting, transmits fast, recovers in one round trip

Selection and the mis-cable

#ScenarioExpected
32Four cables, all to switch Bfour selected
33Three to B, one to CC's port individual, three selected
34Same, phy_link_up on the fourthup
35Same, pcs_block_locklocked
36Same, auto-negotiation10 Gb/s full duplex, correct
37Same, Chapter 15.1 §15 conformanthigh
38Same, under static configuration25% of traffic to switch C
39SameC floods it — 23× amplification
40SameC learns our addresses on that port
41Samereturn traffic never uses that cable — asymmetric octet counters
42Same, under LACPrefused in 3 s, traffic_at_risk_pct = 25
43found_sys reportedC's MAC — the OUI names the vendor
44Mis-cable on the lowest port, first-port referencethree ports wrongly excluded
45Same, majority referenceone port excluded
46Two-member LAG, one mis-cabledno majority — both unresolved
47One mismatched PDU during bring-upnot a finding — needs 3
48Port goes down while mis-cabledfinding cleared — a different fault
49Local key checked, partner notaggregates any four cables — static with extra frames
50Six ports, four-member limittwo standby, fully negotiated, not distributing
51A member fails, standby presentpromoted — no capacity loss

Timers and churn

#ScenarioExpected
52Fast modeperiodic 1 s, timeout 3 s
53Slow mode30 s / 90 s
54Partner wants fast, we transmit slowpartner times out every 3 s
55Samedetach, re-attach, churn indefinitely
56100 Gb/s member, fast timeout37.5 GB lost
57100 Gb/s member, slow timeout1125 GB lost
581 Gb/s member, slow timeout11.25 GB
59Ordinary link failure, PHY interrupt wired10 µs — LACP's timer never fires
60One-way failure, our TX deadboth PHYs up, both PCSs locked
61Samec_sent rising, c_received flat
62Samec_timeoutsthe only evidence in the system
63Both ends passiveneither transmits — no aggregate, no error
64Actor cannot synchronise for 1 sactor_churn
65Partner cannot synchronisepartner_churn — a separate bit
66Both merged into one bithalf the investigations go to the wrong end
67Member attaching four times a secondrebalance eight times a second
68Mis-cable present, conformanthigh — the protocol worked
69Same, aggregate_degradedhigh — the alarm operations wants

The directed test random stimulus will not produce

A mis-cable is not a traffic condition and not a fault injection — it is a topology in which one link's far end is a different, entirely healthy device. Random stimulus varies frames; it does not reconnect cables. And a fault-injection framework that models link failures models them as loss of signal, which is the one thing this failure does not produce.

Setup: three switches. A has a four-port aggregate, key 100, ports 5–8. B has a matching four-port aggregate, key 100. C is an unrelated switch with an ordinary access port, no aggregate. A's ports 5, 6 and 7 are cabled to B. A's port 8 is cabled to C. All four links negotiate 10 Gb/s full duplex and come up. A's forwarding table has 4096 addresses reachable via the aggregate.

Stimulus, two runs. Run 1: static configuration on both ends, all four ports members. Offer 4096 flows to the aggregate for 100 ms. Run 2: identical cabling, LACP fast mode on A and B, C not running LACP.

Oracle:

#ObservableRun 1 — staticRun 2 — LACPWhy it matters
1phy_link_up, port 8upupthere is a device there
2pcs_block_lock, port 8lockedlockedvalid blocks arriving
3auto-negotiation, port 810G full10G fullit negotiated correctly
4Chapter 15.1 §15 conformanthighhighnothing A did is wrong
5members in distributing43the finding
6traffic on port 8≈25%0
7frames delivered to C≈25% of the aggregate0the damage
8C's flood amplification23×Chapter 12.4
9C's table learns A's addressesyesnoa path nobody intended
10return traffic on port 8≈00asymmetric counters — the one local hint
11c_mismatched_mirror, port 8n/arising continuously
12miscabled[3]n/aset after 3 PDUs — 3 s
13found_sys[3]n/aC's MACthe OUI names the vendor
14traffic_at_risk_pctn/a25the incident, not the finding
15aggregate_degradedlowhighwhat operations is paged on
16aggregate capacity4 members, 25% misdelivered3 members, 75%, correctthe trade
17LACPDUs from Cn/anone — C does not run itport 8 never gets a partner
18individual[3] against selected[3]n/aindividual only after a partner is heardsilence is not a finding

Rows 1 to 4 are identical in both runs and all four are green. That is the whole argument for the protocol: every indicator a switch can produce from its own hardware says this member is healthy, and one of them is delivering a quarter of the aggregate's traffic to the wrong building.

And row 17 is the honest caveat. C does not run LACP, so port 8 never acquires a partner at all — it is excluded for silence rather than for disagreement. Row 18 records that difference, because a mis-cable to a device that does run LACP is caught by the system-id comparison and one to a device that does not is caught by the timeout. Both work; they are different mechanisms and the counters should not conflate them.

20. Debugging LACP

Five questions, in order. Three of them are answered before any traffic is examined.

Step 1 — is anything being sent, and is anything arriving? c_sent against c_received. c_sent rising with c_received flat is a one-way path — our transmit is dead or the far end is not running LACP, and Section 19's row 17 separates those. Both flat is a passive/passive pair, which is a configuration that never aggregates and never errors.

Step 2 — what is actually at the other end? chosen_partner_sys and found_sys[m]. This is the read that resolves a mis-cable in seconds and the one most status outputs omit. Chapter 5.3's OUI in the top 24 bits usually names the vendor, which is frequently enough to identify the rack.

Step 3 — is a member excluded, and for which of two reasons? individual[m] against selected[m]. individual means we heard a partner and it was the wrong one — a mis-cable to an LACP-speaking device. Neither set means we have heard nothing at all, which is silence: the far end is not running LACP, or our transmit is dead, or the cable reaches something that does not speak it.

Step 4 — is the timing agreed? The partner's timeout bit against our transmit period. A device transmitting slowly to a partner that asked for fast is timed out every 3 seconds, and the symptom — a member that attaches and detaches endlessly — looks exactly like a flapping cable. c_timeouts rising while phy_link_up never falls separates them.

Step 5 — is it churning, and whose churn? actor_churn against partner_churn, and attaches_per_window. Actor churn is ours — usually Section 8's reference oscillating. Partner churn is theirs. And attaches_per_window above one or two is costing Chapter 15.2 §12's rebalance on every transition, which on a modulo reduction is 74.9% of flows each time.

And the state that ends an investigation: c_sent and c_received both rising, chosen_partner_sys matching the expected neighbour, all configured members selected, c_timeouts zero, attaches_per_window zero. That is a negotiated aggregate with nothing to find, and any remaining complaint about it belongs to Chapter 15.2's distribution or to Chapter 15.1 §13's flow count.

21. Common Misconceptions

1 — "LACP is how a LAG fails over."

The wrong model: the protocol detects member failures.

What it costs: a failover that takes 3 seconds instead of 10 microseconds — 37.5 GB on a 100 Gb/s member. Chapter 15.1 §7: a PHY interrupt detects loss of signal in ~10 µs and PCS block-lock loss is 106 ns at 10 Gb/s.

The corrected model: LACP is the detector of last resort, for the failures the physical layer cannot see — a one-way transmit failure, a mis-cable, a far end that stopped aggregating. Its timeout should almost never fire on a healthy network, and a design relying on it for ordinary failures has left the fast detectors unwired.

2 — "Slow mode saves bandwidth."

The wrong model: 30-second periods are the conservative choice.

What it costs: 1125 GB on a 100 Gb/s member instead of 37.5 GB, to save 1144 bit/s — which is 1.1 × 10⁻⁶% of the link.

The corrected model: the real historical argument was CPU, not bandwidth: a 48-port chassis running LACP in software processing 48 PDUs per second was once a load. It is not now. Fast mode on every aggregate unless something specific forbids it.

3 — "The far end says it is synchronised, so we are aggregated."

The wrong model: the partner's sync bit is sufficient.

What it costs: a mis-cabled member aggregated enthusiastically. The far end may be perfectly synchronised with somebody else — switch C in Section 11 is synchronised with whatever it is actually aggregating.

The corrected model: synchronisation requires two things — their sync bit and they_see_us, the comparison of their partner TLV against our own identity. p_sync_needs_mirror is the property, and a design checking only the first bit is the single most damaging implementation bug in the protocol.

4 — "A mis-cabled member means the switch is broken."

The wrong model: an alarm on an aggregate implies a device fault.

What it costs: an escalation to the wrong team, and a conformance bit that cannot answer "is my LACP implementation working". Nothing the switch did is wrong: it detected a cabling fault correctly and excluded the member correctly.

The corrected model: faults and findings are separate vectors. conformant describes this device; aggregate_degraded describes the world. Operations alarms on the second and engineering reads the first, and merging them produces one alarm meaning two unrelated things.

5 — "Both ends will agree because they run the same standard."

The wrong model: conformance implies identical behaviour.

What it costs: hours on an aggregate where four of eight members carry traffic one way only. Two switches with different maximum member counts — eight and four — both negotiate all eight successfully, then one selects eight and the other selects four with four standby.

The corrected model: the selection is computed independently at both ends and never exchanged, so the two must agree about the function and not merely about the inputs. The protocol converges anyway — because collecting and distributing report state rather than intention, so each end reacts to what the other is observed to be doing.

6 — "LACP will fix the imbalance."

The wrong model: negotiation covers everything about the aggregate.

What it costs: LACP enabled on an aggregate whose real problem is Chapter 15.2 §10's flow count, and no change. 802.1AX deliberately does not negotiate the distribution function — each end need only be consistent with itself.

The corrected model: LACP settles membership. Chapter 15.2 settles distribution, unilaterally and per direction. A four-flow aggregate is 47.0% efficient with LACP and without it, and the two mechanisms answer completely different questions.

22. Interview Reasoning

Q1 — What does LACP detect that a PHY cannot?

Anything about the identity of the device at the other end. A PHY reports that its own receiver has signal; PCS reports block lock; auto-negotiation settles speed and duplex. All three are perfectly happy on a mis-cabled member — there is a real device there, it negotiated correctly, the link is up. The only evidence that it is the wrong device is the actor system id in its LACPDUs, which no local measurement produces. Same for a one-way transmit failure: both ends' receivers are fine and only a protocol that expects to hear back notices.

Q2 — Walk through a mis-cable under static configuration.

Four cables, three to B, one to C. All four PHYs up, all four in the aggregate. 25% of the traffic goes to C, which has no entry for those destinations and floods them at Chapter 12.4's 23× on a 24-port switch. C learns our addresses on that port, so C now believes those stations are reachable via us. Return traffic never uses the cable — C's own distributor has no reason to — so the octet counters on the two ends of that one cable are wildly asymmetric, which is the single local hint available and nobody graphs it. LACP refuses the member in 3 seconds and reports C's MAC.

Q3 — Why does the mux machine have a WAITING state?

So that several ports selecting the same aggregate settle together. Four ports come up within milliseconds; without the wait, port 1 attaches alone, then 2, then 3, then 4 — four membership changes, so Chapter 15.2 §12's rebalance runs four times, and with a modulo reduction that relocates 74.9% of the flows on each occasion. For a link that is simply coming up.

Q4 — Which end chooses the transmit period, and which the timeout?

The partner chooses our transmit period and we choose our own timeout. A partner advertising the fast timeout bit is telling us to transmit every second; we expire after three of the periods we asked for, because we asked for them. Getting this backwards produces a device that transmits slowly to a partner expecting fast, which times out every 3 seconds, detaches, re-attaches, and churns for ever — with a symptom identical to a flapping cable.

Q5 — Two devices must select the same subset of ports. How, with no message that carries the answer?

They agree on who decides rather than on the set. The lower system priority wins, ties broken by the lower system id; both ends then apply the winner's port priorities, not their own. Both already have both system ids from the actor TLVs. If each ordered by its own priorities, a six-port bundle limited to four would produce different subsets and the intersection would carry traffic one way only. And when the two disagree about the selection function — different member limits, say — the collecting and distributing bits make it converge on the intersection anyway, because they report state rather than intention.

Q6 — Why can't you assert that the two ends agree?

Because the partner TLV describes us as we were one round trip ago — up to a full periodic interval, 1 second in fast mode. Any local state change makes the mirror wrong for exactly that interval, by design, so an equality assertion fires on every normal transition and ends up disabled into uselessness. The assertable form is convergence within a bound: after a state change, they_see_us becomes true within twice the partner's period. And a partner TLV with the defaulted bit set must not be compared at all — the far end is volunteering that the value is invented.

23. Understanding Check

==

Module 15's three chapters form one argument about names. Chapter 15.1 introduces the aggregate, a name that outlives the member it replaces, so a forwarding entry stays valid when a member fails; it costs about 2.19 kibibytes and its hard part is finding every mechanism that already held the old name, which turned out to be Chapter 12.2's learning. Chapter 15.2 builds the function that maps a conversation to a member, costing about 3.8 kibibytes, and finds that the obvious modulo reduction relocates 74.9 percent of flows when 25 percent had to, and that no function can split a conversation larger than a member. This chapter replaces the operator's intention with the far end's statement for 1184 bits per second and about 320 octets, and is the only mechanism in the module whose value comes from information the device cannot derive locally at any price.Module 15three names15.1 — theaggregatea name outliving a member15.2 — the flowa name for a conversation15.3 — the partnera name only they can give2.19 KiBsurvives a failure3.8 KiB25% instead of 74.9%320 octetssees outside the device12
Figure 4 — Module 15 in one line each: three mechanisms, three kinds of name, and only one that can see outside the device.

24. What's Next

Module 15 is complete, and its three chapters turned out to be one argument about names.

Chapter 15.1 introduced a name that outlives the thing it names — an aggregate, so a forwarding entry stays valid when a member fails — and found that the work is never the name but finding every mechanism that already holds the old one. Chapter 12.2's learning was the one it had to find.

Chapter 15.2 built the function that maps a conversation to a member, and found that the obvious reduction moves three quarters of the flows when a quarter had to, and that no function at all can split a conversation larger than a member.

And this chapter replaced the operator's intention with the far end's statement, for 1184 bit/s — and found four failures that every local indicator calls healthy.

Module 16 — Precision Time Protocol — changes the question entirely. Instead of asking where a frame should go, it asks when something happened.

Chapter 16.1 — Why Sub-Microsecond Sync Is a Hardware Problem establishes what needs synchronised clocks at all, derives the error budget of a software timestamp from interrupt latency, scheduler jitter and stack traversal, and shows where in the datapath the timestamp actually has to be taken.

Then the module builds the message exchange, the timestamp point at the MAC/PHY boundary, the servo that closes the loop, and the asymmetries that limit what any of it can achieve.

One thread carries directly across, and it is this chapter's. LACP's whole value was that it carried a statement from somebody else about something no local measurement could produce. Module 16 is that problem in its purest form: a clock cannot measure its own error — it can only measure its difference from another clock, and that difference is contaminated by everything the measurement crossed to arrive.

Continue learning

Standards & specifications

Governing standard
IEEE Std 802.3 (Ethernet)(opens IEEE in a new tab)

Defines the Ethernet MAC, the media-independent interfaces and the physical-layer sublayers, including framing, access control, auto-negotiation and per-rate PHY specifications. VLAN tagging, priority and time-sensitive shaping are defined by IEEE 802.1, not by 802.3.

This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.

Where this fits

Part of the Ethernet curriculum.