Ethernet · Module 15
LACP
Negotiating aggregation instead of configuring it: the 128-octet LACPDU, the state machine, and the failure where every link is up and a quarter of the traffic goes to the wrong device.
Chapter 15.1 §16 listed four ways the two ends of an aggregate can disagree and found that a statically configured aggregate detects none of them. Chapter 15.2 added a fifth that is deliberate and correct. This chapter is about the other four.
Static configuration records what an operator intended. The useful fact is what is actually plugged in, and no local measurement produces it.
Chapter 15.1 §6 built three detection sources — PHY loss of signal, PCS block lock, and a protocol liveness input that was left as a stub. All three of the first two answer the same question: is this end receiving a signal. None of them answers the question aggregation actually needs, which is whether the thing at the other end of this cable is the same thing as at the other end of the other three.
The failure that makes the point is a mis-cabled member. Four cables from one switch. Three reach the intended neighbour and one reaches a different device entirely. Every link is up, every PHY is healthy, every PCS is locked, Chapter 15.1 §15's conformant is high — and one quarter of the aggregate's traffic is being delivered to a device it was never addressed to.
LACP costs 1184 bit/s and finds it in three seconds.
1. Scope — What This Chapter Owns
This chapter owns the negotiation: the LACPDU and its fields, the actor/partner exchange, the three states a member can be in, the selection logic, the periodic timer and its timeout, churn, and the mis-cable that motivates all of it.
It does not own the aggregate. Chapter 15.1 built the member table, the compaction, the failover sequence and the forwarding-table consequence. LACP decides which ports go in that table and nothing about what happens afterwards.
It does not own distribution. Chapter 15.2 built the field selector, the hash and the reduction. 802.1AX does not negotiate any of it — Chapter 15.2 §7's callout gave the reason — so two ends running LACP still hash independently and their per-member counters remain incomparable.
It does not own link detection. Chapter 15.1 §6's PHY and PCS sources are unchanged and remain the fast path: 10 µs and 106 ns respectively. LACP is the slow path — 3 seconds at best — and Section 13 shows that it is nonetheless the only source that catches an entire class of failure.
And it does not own the physical layer. Chapter 11.2's auto-negotiation settles speed and duplex on one link; LACP settles membership across several. The two are frequently confused because both are described as negotiation, and Section 2's table separates them.
2. What Static Configuration Cannot Detect
Four failures, and what makes them worth a protocol is that the local evidence is identical in every case to the evidence of a healthy link.
| Failure | What every local source reports | What actually happened |
|---|---|---|
| member count mismatch | four links up | the far end put three in its aggregate |
| port set mismatch | four links up | the far end's fourth is a different port |
| mis-cabled member | four links up | one cable reaches a different device |
| far end not aggregating | four links up | it treats them as four independent ports |
| one-way failure | link up — the local receiver is fine | our transmit is dead |
The middle column is the same in all five rows. Chapter 15.1 §6's phy_link_up and pcs_block_lock both describe this end's receiver, and this end's receiver is perfectly happy in every case.
And the consequences are not small.
The mis-cable delivers 25% of the aggregate's traffic to an unrelated device, which then floods it — Chapter 12.4's amplification, applied to traffic that had a perfectly good destination.
The far end not aggregating produces Chapter 12.2 §9's flapping entry at the rate the distributor moves flows, which on a busy aggregate is thousands of times a second, on every address behind us.
The one-way failure is a member that is chosen by Chapter 15.2's distributor for its full share of flows and carries none of them.
==
What all four need is the same thing: a statement from the far end about what it believes. That is the entire content of the protocol — there is no negotiation in the sense of proposing and counter-proposing. Each end periodically says what it is and what it thinks its partner is, and aggregation happens only where the two statements are consistent.
3. RTL 1 — Building an LACPDU
The frame is 128 octets and every field in it is a claim about identity. Nothing in it is a request.
// -----------------------------------------------------------------------
// lacp_pkg -- shared types for 802.1AX Link Aggregation Control.
// -----------------------------------------------------------------------
package lacp_pkg;
localparam int MAX_MEMBERS = 8;
localparam int MEMBER_W = $clog2(MAX_MEMBERS);
// Slow-protocol destination, distinct from 14.2's control DA.
localparam logic [47:0] SLOW_DA = 48'h01_80_C2_00_00_02;
localparam logic [15:0] SLOW_ET = 16'h8809;
localparam logic [7:0] LACP_SUB = 8'h01;
localparam logic [7:0] LACP_VER = 8'h01;
// The eight state bits, carried once for the actor and once for the
// partner. Bits 2 and 3 are the two roles 15.1 section 3 kept apart.
typedef struct packed {
logic expired; // bit 7
logic defaulted; // bit 6 -- using defaults, not heard from
logic distributing; // bit 5 -- 15.1's distributing
logic collecting; // bit 4 -- 15.1's collecting
logic synchronised; // bit 3 -- IN SYNC with the partner
logic aggregation; // bit 2 -- willing to aggregate at all
logic timeout_short; // bit 1 -- 1 = fast (1 s), 0 = slow (30 s)
logic activity; // bit 0 -- 1 = active, will send unprompted
} lacp_state_t;
// What identifies an aggregatable entity. Two ports may join the
// same aggregate only if all four of these match -- section 8.
typedef struct packed {
logic [15:0] sys_pri;
logic [47:0] sys_id; // the device's own MAC
logic [15:0] key; // the aggregation key
logic [15:0] port_pri;
logic [15:0] port_id;
} lacp_id_t;
typedef enum logic [1:0] {
MUX_DETACHED, MUX_WAITING, MUX_ATTACHED, MUX_COLLECT_DIST
} mux_state_e;
endpackage// -----------------------------------------------------------------------
// lacpdu_builder -- emits one 128-octet LACPDU.
//
// The frame states what WE are (actor) and what we BELIEVE the partner
// to be (partner). The second half is a mirror, and section 19's
// rejected property is about how stale that mirror is.
// -----------------------------------------------------------------------
module lacpdu_builder
import lacp_pkg::*;
(
input logic clk,
input logic rst_n,
input logic req_valid,
input lacp_id_t actor,
input lacp_state_t actor_state,
input lacp_id_t partner, // what we last heard, or defaults
input lacp_state_t partner_state,
input logic [15:0] collector_max_delay,
output logic req_ready,
output logic tx_valid,
output logic [7:0] tx_octet,
output logic tx_sop,
output logic tx_eop,
output logic [31:0] c_sent
);
// 6 DA + 6 SA + 2 ET + 110 payload = 124; FCS appended downstream
// brings the frame to 128. No padding: the payload is well above
// 5.6's 46-octet minimum.
localparam int BODY = 124;
logic [7:0] b [0:BODY-1];
logic [6:0] idx;
logic busy;
always_comb begin
int p;
for (p = 0; p < BODY; p++) b[p] = 8'h00;
for (p = 0; p < 6; p++) b[p] = SLOW_DA[47 - 8*p -: 8];
for (p = 0; p < 6; p++) b[6 + p] = actor.sys_id[47 - 8*p -: 8];
b[12] = SLOW_ET[15:8]; b[13] = SLOW_ET[7:0];
b[14] = LACP_SUB; b[15] = LACP_VER;
// Actor TLV: type 0x01, length 20.
b[16] = 8'h01; b[17] = 8'd20;
b[18] = actor.sys_pri[15:8]; b[19] = actor.sys_pri[7:0];
for (p = 0; p < 6; p++) b[20 + p] = actor.sys_id[47 - 8*p -: 8];
b[26] = actor.key[15:8]; b[27] = actor.key[7:0];
b[28] = actor.port_pri[15:8]; b[29] = actor.port_pri[7:0];
b[30] = actor.port_id[15:8]; b[31] = actor.port_id[7:0];
b[32] = actor_state;
// b[33..35] reserved
// Partner TLV: type 0x02, length 20. This is our BELIEF about them.
b[36] = 8'h02; b[37] = 8'd20;
b[38] = partner.sys_pri[15:8]; b[39] = partner.sys_pri[7:0];
for (p = 0; p < 6; p++) b[40 + p] = partner.sys_id[47 - 8*p -: 8];
b[46] = partner.key[15:8]; b[47] = partner.key[7:0];
b[48] = partner.port_pri[15:8]; b[49] = partner.port_pri[7:0];
b[50] = partner.port_id[15:8]; b[51] = partner.port_id[7:0];
b[52] = partner_state;
// b[53..55] reserved
// Collector TLV: type 0x03, length 16.
b[56] = 8'h03; b[57] = 8'd16;
b[58] = collector_max_delay[15:8]; b[59] = collector_max_delay[7:0];
// b[60..71] reserved
// Terminator TLV: type 0x00, length 0, then 50 reserved octets.
b[72] = 8'h00; b[73] = 8'd0;
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
idx <= '0; busy <= 1'b0; c_sent <= '0;
end else if (!busy) begin
if (req_valid) begin
busy <= 1'b1;
idx <= '0;
c_sent <= c_sent + 1;
end
end else begin
if (idx == BODY-1) begin busy <= 1'b0; idx <= '0; end
else idx <= idx + 1'b1;
end
end
assign req_ready = !busy;
assign tx_valid = busy;
assign tx_octet = b[idx];
assign tx_sop = busy && (idx == 7'd0);
assign tx_eop = busy && (idx == BODY-1);
endmoduleClassification: a protocol encoder with no state of its own. One frame per request, fixed length, no padding.
What it teaches: that the frame carries two identities and one of them is hearsay. The actor TLV is a fact about this device; the partner TLV is what this device last heard, which the far end will read as our report of its state. So the partner TLV is a mirror — and Section 18's rejected property is about how much a mirror can be trusted when the round trip is one second.
And it teaches why the source address is actor.sys_id rather than a port address. Chapter 14.4 §3's pfc_frame_builder used the port's address for exactly the opposite reason. Here the frame is a claim about the device, and all of a device's ports must claim the same identity — otherwise a four-member aggregate looks like four one-member aggregates to the far end, which is Section 9's selection logic failing on a design that got this wrong.
Deliberately simplified: the reserved octets are transmitted as zero and the terminator TLV is emitted with no provision for optional TLVs between it and the collector. Production designs must skip unknown TLVs rather than assume the layout, because 802.1AX has added TLVs since — and a parser with hard-coded offsets past the collector fails on a conformant neighbour.
Production implication: c_sent is the counter that separates "we are not sending" from "they are not hearing". Paired with Section 5's c_received, it is the whole of the first debugging step: c_sent rising and c_received flat is a one-way path, which is precisely the failure Chapter 15.1 §6's physical sources cannot see and the reason proto_alive was left as an input there.
4. The LACPDU Format, Field by Field
Every field is a claim, and it is worth knowing which claims matter.
| Offset | Field | Octets | What it decides |
|---|---|---|---|
| 0 | DA 01:80:C2:00:00:02 | 6 | Slow Protocols — never forwarded by a bridge |
| 6 | SA — the device's MAC | 6 | all ports claim one identity |
| 12 | EtherType 0x8809 | 2 | Slow Protocols |
| 14 | subtype 0x01 | 1 | LACP, against 0x02 Marker |
| 15 | version | 1 | — |
| 16 | actor TLV type 0x01, length 20 | 2 | — |
| 18 | actor system priority | 2 | which end decides — Section 9 |
| 20 | actor system id | 6 | the device |
| 26 | actor key | 2 | which ports may aggregate together |
| 28 | actor port priority | 2 | which ports are chosen if not all fit |
| 30 | actor port id | 2 | — |
| 32 | actor state | 1 | the eight bits |
| 36 | partner TLV — type 0x02, length 20 | 20 | our belief about them |
| 56 | collector TLV, max delay | 16 | how long we may buffer during a move |
| 72 | terminator, then 50 reserved | 52 | — |
| — | payload total | 110 | frame 128, wire 148 |
And the eight state bits, because they carry the whole of the negotiation:
| Bit | Name | Meaning |
|---|---|---|
| 0 | activity | active — we send unprompted; passive — only in reply |
| 1 | timeout | 1 = fast (1 s / 3 s), 0 = slow (30 s / 90 s) |
| 2 | aggregation | willing to aggregate; clear means individual |
| 3 | synchronised | our view of the partner matches what they sent |
| 4 | collecting | Chapter 15.1 §3's receive role |
| 5 | distributing | Chapter 15.1 §3's transmit role |
| 6 | defaulted | we have heard nothing and are using defaults |
| 7 | expired | the receive timer fired |
Bit 3 is the negotiation's result and bits 4 and 5 are its consequence. A member becomes usable only when both ends report synchronised, and the two roles are then enabled in Chapter 15.1 §3's order — collecting first, distributing second.
Bit 6 is the one worth dwelling on. Defaulted means this end is transmitting a partner TLV it made up. The far end therefore knows that our mirror of it is fiction, which is a remarkable thing for a protocol to say about its own message and is exactly why the mirror is usable at all — Section 18's rejected property is what happens when a design forgets to read this bit.
And the frame's cost is negligible in both modes:
| Mode | Period | Frames/s | Bit/s | On 1 Gb/s | On 100 Gb/s |
|---|---|---|---|---|---|
| fast | 1 s | 1 | 1184 | 1.18 × 10⁻⁴% | 1.18 × 10⁻⁶% |
| slow | 30 s | 0.033 | 39.5 | 3.95 × 10⁻⁶% | 3.95 × 10⁻⁸% |
Fast mode costs one part in a million of a gigabit link and detects a failure 30× sooner. There is no bandwidth argument for slow mode, and Section 13 shows what the argument actually is.
5. RTL 2 — Parsing an LACPDU
The receive side has one job the builder does not: deciding whether what arrived is about us.
// -----------------------------------------------------------------------
// lacpdu_parser -- validates and extracts an LACPDU.
//
// The partner TLV in a RECEIVED frame is the far end's belief about
// US. Comparing it against what we actually are is how section 7's
// synchronised bit gets set, and it is the only place in the protocol
// where a device learns whether it has been heard correctly.
// -----------------------------------------------------------------------
module lacpdu_parser
import lacp_pkg::*;
(
input logic clk,
input logic rst_n,
input logic rx_valid,
input logic [7:0] rx_octet,
input logic rx_sop,
input logic rx_eop,
input logic rx_fcs_ok,
input lacp_id_t our_actor, // what we believe we are
output logic pdu_valid,
output lacp_id_t rx_actor, // what THEY say they are
output lacp_state_t rx_actor_state,
output lacp_id_t rx_partner, // what they think WE are
output lacp_state_t rx_partner_state,
output logic they_see_us, // their partner TLV matches us
output logic [31:0] c_received,
output logic [31:0] c_bad_fcs,
output logic [31:0] c_wrong_subtype,
output logic [31:0] c_mismatched_mirror
);
logic [6:0] idx;
logic in_frame, da_ok, et_ok, sub_ok;
logic [15:0] et_q;
lacp_id_t a_q, p_q;
lacp_state_t as_q, ps_q;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
idx <= '0; in_frame <= 1'b0; da_ok <= 1'b0;
et_ok <= 1'b0; sub_ok <= 1'b0; et_q <= '0;
a_q <= '0; p_q <= '0; as_q <= '0; ps_q <= '0;
pdu_valid <= 1'b0; they_see_us <= 1'b0;
c_received <= '0; c_bad_fcs <= '0;
c_wrong_subtype <= '0; c_mismatched_mirror <= '0;
end else begin
pdu_valid <= 1'b0;
if (rx_valid && rx_sop) begin
idx <= 7'd1; in_frame <= 1'b1; sub_ok <= 1'b0;
da_ok <= (rx_octet == SLOW_DA[47:40]);
end else if (rx_valid && in_frame) begin
idx <= idx + 1'b1;
if (idx <= 7'd5)
if (rx_octet != SLOW_DA[47 - 8*idx -: 8]) da_ok <= 1'b0;
if (idx == 7'd12) et_q[15:8] <= rx_octet;
if (idx == 7'd13) et_ok <= ({et_q[15:8], rx_octet} == SLOW_ET);
if (idx == 7'd14) sub_ok <= (rx_octet == LACP_SUB);
// Actor TLV.
if (idx == 7'd18) a_q.sys_pri[15:8] <= rx_octet;
if (idx == 7'd19) a_q.sys_pri[7:0] <= rx_octet;
if (idx >= 7'd20 && idx <= 7'd25)
a_q.sys_id[47 - 8*(idx-20) -: 8] <= rx_octet;
if (idx == 7'd26) a_q.key[15:8] <= rx_octet;
if (idx == 7'd27) a_q.key[7:0] <= rx_octet;
if (idx == 7'd28) a_q.port_pri[15:8] <= rx_octet;
if (idx == 7'd29) a_q.port_pri[7:0] <= rx_octet;
if (idx == 7'd30) a_q.port_id[15:8] <= rx_octet;
if (idx == 7'd31) a_q.port_id[7:0] <= rx_octet;
if (idx == 7'd32) as_q <= rx_octet;
// Partner TLV -- their mirror of us.
if (idx == 7'd38) p_q.sys_pri[15:8] <= rx_octet;
if (idx == 7'd39) p_q.sys_pri[7:0] <= rx_octet;
if (idx >= 7'd40 && idx <= 7'd45)
p_q.sys_id[47 - 8*(idx-40) -: 8] <= rx_octet;
if (idx == 7'd46) p_q.key[15:8] <= rx_octet;
if (idx == 7'd47) p_q.key[7:0] <= rx_octet;
if (idx == 7'd48) p_q.port_pri[15:8] <= rx_octet;
if (idx == 7'd49) p_q.port_pri[7:0] <= rx_octet;
if (idx == 7'd50) p_q.port_id[15:8] <= rx_octet;
if (idx == 7'd51) p_q.port_id[7:0] <= rx_octet;
if (idx == 7'd52) ps_q <= rx_octet;
end
if (rx_valid && rx_eop) begin
in_frame <= 1'b0;
if (da_ok && et_ok) begin
if (!sub_ok) c_wrong_subtype <= c_wrong_subtype + 1;
else if (!rx_fcs_ok) c_bad_fcs <= c_bad_fcs + 1;
else begin
c_received <= c_received + 1;
pdu_valid <= 1'b1;
rx_actor <= a_q;
rx_actor_state <= as_q;
rx_partner <= p_q;
rx_partner_state <= ps_q;
// Do they have us right? If their mirror does not match
// what we are, they are talking about somebody else -- or
// they have not heard our latest state yet.
they_see_us <= (p_q.sys_id == our_actor.sys_id) &&
(p_q.key == our_actor.key) &&
(p_q.port_id == our_actor.port_id);
if (!((p_q.sys_id == our_actor.sys_id) &&
(p_q.key == our_actor.key) &&
(p_q.port_id == our_actor.port_id)))
c_mismatched_mirror <= c_mismatched_mirror + 1;
end
end
end
end
end
endmoduleClassification: a protocol decoder with a validity gate and one comparison that is the whole protocol's hinge.
What it teaches: that they_see_us is where a device discovers whether it has been heard. Everything else in an LACPDU is the far end describing itself; this one field is the far end describing us, and comparing it against what we actually are closes the loop. Without it there is no way to distinguish a partner that is talking to us from one that is talking past us — which is Section 11's mis-cable exactly.
And it teaches that c_mismatched_mirror is transiently normal and persistently fatal. After any state change, the far end's mirror is one round trip out of date, so the counter increments for up to a second in fast mode. It rising continuously means the far end's partner TLV never matches us, which means it is mirroring somebody else — the mis-cable, in one counter.
Deliberately simplified: the parser assumes the TLVs are at their standard offsets and stops at the collector. 802.1AX permits additional TLVs and a conformant parser walks the type/length chain, skipping unknown types until the terminator. A hard-coded parser rejects nothing and mis-parses everything on a neighbour that adds one, which is the same failure shape as Chapter 13.2 §7's fixed-offset parse.
Production implication: c_wrong_subtype catches Marker protocol frames — subtype 0x02, Section 16 — arriving at a device that does not implement them. That is not an error and must not be counted as one, so a production design separates "a slow protocol we do not implement" from "a malformed LACPDU". Conflating them produces an error counter that rises during perfectly normal operation, which is the fastest way to make an error counter ignored.
6. RTL 3 — The Actor/Partner State Machine
The mux machine decides when a member is usable, and its four states are the protocol's actual content.
// -----------------------------------------------------------------------
// lacp_state_machine -- the per-port mux machine.
//
// DETACHED -> WAITING -> ATTACHED -> COLLECTING_DISTRIBUTING
//
// The WAITING state exists so that several ports selecting the same
// aggregate at the same time settle together rather than one at a
// time; without it, a four-member aggregate comes up as a
// one-member aggregate, then two, then three -- and 15.2 section 12's
// rebalance runs at every step.
// -----------------------------------------------------------------------
module lacp_state_machine
import lacp_pkg::*;
#(
parameter int AGGREGATION_WAIT = 1000000 // 2 s at 500 MHz
)(
input logic clk,
input logic rst_n,
input logic pdu_valid,
input lacp_id_t rx_actor,
input lacp_state_t rx_actor_state,
input logic they_see_us,
input logic rx_timeout, // section 12's expiry
input logic port_enabled, // 15.1 section 6's physical up
input logic selected, // section 8 chose this port
output mux_state_e mux,
output logic collecting,
output logic distributing,
output lacp_state_t our_state,
output logic [31:0] c_attach,
output logic [31:0] c_detach
);
logic [31:0] wait_cnt;
logic partner_sync;
// We are synchronised when they have us right AND they say they are
// synchronised too. Both halves are required: their sync bit alone
// could be about a different neighbour.
assign partner_sync = they_see_us && rx_actor_state.synchronised;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
mux <= MUX_DETACHED; wait_cnt <= '0;
collecting <= 1'b0; distributing <= 1'b0;
c_attach <= '0; c_detach <= '0;
our_state <= '0;
end else begin
// Any loss of the physical port or of the partner drops us all
// the way out. There is no partial state.
if (!port_enabled || rx_timeout || !selected) begin
if (mux != MUX_DETACHED) c_detach <= c_detach + 1;
mux <= MUX_DETACHED;
collecting <= 1'b0;
distributing <= 1'b0;
wait_cnt <= '0;
end else begin
unique case (mux)
MUX_DETACHED: begin
collecting <= 1'b0; distributing <= 1'b0;
if (selected) begin
mux <= MUX_WAITING;
wait_cnt <= '0;
end
end
MUX_WAITING: begin
wait_cnt <= wait_cnt + 1;
if (wait_cnt == AGGREGATION_WAIT-1) mux <= MUX_ATTACHED;
end
MUX_ATTACHED: begin
// 15.1 section 3's order: collect BEFORE distributing, so
// the far end is never sent frames on a member we are not
// yet accepting from.
collecting <= 1'b1;
if (partner_sync && rx_actor_state.collecting) begin
mux <= MUX_COLLECT_DIST;
c_attach <= c_attach + 1;
end
end
MUX_COLLECT_DIST: begin
collecting <= 1'b1;
distributing <= 1'b1;
end
endcase
end
our_state.synchronised <= (mux == MUX_ATTACHED) ||
(mux == MUX_COLLECT_DIST);
our_state.collecting <= collecting;
our_state.distributing <= distributing;
our_state.aggregation <= 1'b1;
our_state.defaulted <= rx_timeout;
our_state.expired <= rx_timeout;
end
end
endmoduleClassification: a four-state mux machine with one timed state and an unconditional drop-out.
What it teaches: that MUX_WAITING is a settling delay and not a safety margin, and removing it produces a specific and expensive symptom. Four ports of an aggregate come up within milliseconds of each other; without the wait, port 1 attaches alone, then port 2 joins, then port 3, then port 4. Each join is a membership change, so Chapter 15.2 §12's rebalance runs four times — and with a modulo reduction that relocates most of the flows on each occasion, for a link that is simply coming up.
And it teaches that partner_sync requires two things and a design that checks one is broken in a way that only appears on a mis-cable. The far end saying I am synchronised is not enough: it may be synchronised with somebody else. The conjunction with they_see_us is what makes the claim about this pairing, and Section 11 is the failure it prevents.
Deliberately simplified: the machine drops all the way to MUX_DETACHED on any loss, including a single missed timeout. Production designs pass through an EXPIRED state first, in which the port keeps collecting while transmitting at the fast rate to re-establish quickly — so a brief loss costs a fast-mode round trip rather than Chapter 15.2 §12's full rebalance and a re-attach.
Production implication: c_detach is the churn counter's raw input and the number an operator actually needs. A member that attaches and detaches repeatedly costs a rebalance each way — Chapter 15.1 §6's flapping arithmetic, now driven by the protocol rather than by the cable — and Section 14's churn detector exists because the symptom of protocol churn is identical to the symptom of a flapping link.
==
7. Three States a Member Can Be In
A statically configured member is in or out. An LACP member has three states and the middle one is where every interesting failure sits.
| State | collecting | distributing | What it means |
|---|---|---|---|
| detached | 0 | 0 | not in the aggregate — physical down, or unselected, or the partner disagrees |
| attached | 1 | 0 | we accept from it and do not send on it |
| collecting/distributing | 1 | 1 | a full member |
The middle row is the state that makes the protocol safe, and it is exactly Chapter 15.1 §3's collecting/distributing split arriving with a mechanism behind it.
A port that comes up starts collecting first. Frames the far end sends on it are accepted; nothing is sent on it until the far end says it is collecting too. So there is no window in which either end sends into a port the other is not accepting from — which a statically configured aggregate cannot avoid, because it has no way to know when the far end is ready.
And the ordering is the reason the two bits are carried in the LACPDU at all. Each end reads the other's collecting bit and only then sets its own distributing. The handshake is: I am collecting → I see you are collecting → I am distributing. Two round trips, two seconds in fast mode, and no frame is lost in either direction.
Which prices the whole protocol against static configuration in one row:
| static | LACP | |
|---|---|---|
| time to bring a member into service | immediate | ~2 s — AGGREGATION_WAIT plus a round trip |
| frames lost bringing it up | up to a round trip's worth | zero |
| frames lost if the far end is not ready | all of them, until it is | zero |
A design optimising the first row is optimising the wrong one, because bringing a member up is a scheduled event and losing traffic on it is not.
8. RTL 4 — Aggregation Key Matching
Which ports may join one aggregate is decided by a comparison of four fields, and the comparison is done on both ends independently.
// -----------------------------------------------------------------------
// lacp_key_matcher -- decides whether a port may join an aggregate.
//
// Two ports aggregate together only if BOTH ends agree they should.
// Locally that means the same actor key. Remotely it means the same
// partner system id AND the same partner key. Section 11's mis-cable
// fails the remote half while passing the local one.
// -----------------------------------------------------------------------
module lacp_key_matcher
import lacp_pkg::*;
(
input logic clk,
input logic rst_n,
input logic pdu_valid [MAX_MEMBERS],
input lacp_id_t our_actor [MAX_MEMBERS],
input lacp_id_t rx_actor [MAX_MEMBERS], // the partner, per port
input logic partner_valid [MAX_MEMBERS],
output logic [MAX_MEMBERS-1:0] selected,
output logic [MAX_MEMBERS-1:0] individual, // cannot aggregate
output logic [47:0] chosen_partner_sys,
output logic [15:0] chosen_partner_key,
output logic [31:0] c_wrong_partner,
output logic [31:0] c_selections
);
// The reference pairing is the LOWEST-numbered port that has a
// partner. Every other port must match it or be excluded.
logic have_ref;
logic [47:0] ref_sys;
logic [15:0] ref_key, ref_akey;
always_comb begin
int m;
have_ref = 1'b0;
ref_sys = '0; ref_key = '0; ref_akey = '0;
for (m = 0; m < MAX_MEMBERS; m++) begin
if (!have_ref && partner_valid[m]) begin
have_ref = 1'b1;
ref_sys = rx_actor[m].sys_id;
ref_key = rx_actor[m].key;
ref_akey = our_actor[m].key;
end
end
end
always_ff @(posedge clk or negedge rst_n) begin
int m;
if (!rst_n) begin
selected <= '0; individual <= '0;
chosen_partner_sys <= '0; chosen_partner_key <= '0;
c_wrong_partner <= '0; c_selections <= '0;
end else begin
chosen_partner_sys <= ref_sys;
chosen_partner_key <= ref_key;
for (m = 0; m < MAX_MEMBERS; m++) begin
if (!partner_valid[m]) begin
// Heard nothing. Not selected, and not individual either --
// we simply do not know yet.
selected[m] <= 1'b0;
individual[m] <= 1'b0;
end else begin
// Four conditions, all required.
automatic logic ok;
ok = have_ref &&
(our_actor[m].key == ref_akey) && // local intent
(rx_actor[m].sys_id == ref_sys) && // same device
(rx_actor[m].key == ref_key); // same far aggregate
if (!ok && (rx_actor[m].sys_id != ref_sys)) begin
// A DIFFERENT DEVICE at the other end of this cable.
// Section 11: every physical source says this member is
// healthy, and it must not be aggregated.
c_wrong_partner <= c_wrong_partner + 1;
individual[m] <= 1'b1;
end else begin
individual[m] <= 1'b0;
end
if (ok && !selected[m]) c_selections <= c_selections + 1;
selected[m] <= ok;
end
end
end
end
endmoduleClassification: an all-pairs consistency check reduced to a comparison against a reference. One cycle, combinational reference selection.
What it teaches: that aggregation requires agreement on four fields and only one of them is local. Our own key expresses our intent to aggregate these ports. The partner's system id and key express the far end's, and a design that checks only the local key aggregates any four cables an operator configured regardless of where they go — which is static configuration with extra frames.
And it teaches that individual is a distinct outcome from "not selected", which a two-state design collapses. Not selected means we do not know yet — no LACPDU has arrived. individual means we know, and the answer is no: there is a device at the other end and it is the wrong one. The first is a state to wait in; the second is a fault to report.
Deliberately simplified: the reference is the lowest-numbered port with a partner, which means a single mis-cabled cable on the lowest port makes the other three look wrong. Production designs take the majority pairing as the reference, so one anomalous cable is excluded rather than three. The comparison is the same; the choice of reference decides which side of it the fault lands on, and getting it backwards turns a one-member fault into a three-member one.
Production implication: chosen_partner_sys is the single most useful line of an aggregate's status output and is frequently absent. It answers "what is actually at the other end of this bundle" with a MAC address, and comparing it against the neighbour an operator believes is there resolves Section 11 in one read — without it, the mis-cable is diagnosed by walking to the rack.
9. The Selection Logic, and Why Both Ends Must Agree
Two devices independently choosing which of their ports to aggregate must reach the same answer, and there is no negotiation step in which they compare answers.
The mechanism is a tie-break rather than a conversation: one end is designated the decider, and both ends compute the decision that end would make.
| Step | What decides |
|---|---|
| 1 | the lower system priority wins; ties broken by the lower system id |
| 2 | the winner's port priorities order the candidate ports |
| 3 | ports beyond the aggregate's capacity become standby |
| 4 | both ends apply the winner's ordering, not their own |
Step 4 is the whole point and it is why the system priority exists at all. If each end ordered ports by its own priorities, a six-port bundle limited to four members would produce different four-port subsets at the two ends — and the two subsets' intersection would be the actual aggregate, with the rest carrying traffic in one direction only.
So the protocol does not agree on a set; it agrees on who decides, and derives the set from that. Which is a much smaller thing to agree on, and it needs no round trip: both ends already have both system ids, from the actor TLVs they have exchanged.
And the standby state matters more than it looks. A six-port bundle with a four-member limit has two ports fully negotiated, synchronised, and not distributing — held in reserve. When a member fails, a standby port is promoted, so the aggregate returns to four members without Chapter 15.1 §7's detection cost being followed by a capacity loss. Static configuration has no equivalent: a sixth cable is either a member or it is not connected.
10. RTL 5 — Detecting a Mis-Cable
Section 8's key matcher produces the evidence. This module turns it into a report an operator can act on.
// -----------------------------------------------------------------------
// miscable_detector -- names the specific failure static configuration
// cannot see: a member whose far end is a different device.
//
// Every physical source in 15.1 section 6 reports this member healthy.
// The ONLY evidence is the actor system id in its LACPDUs, which no
// local measurement produces.
// -----------------------------------------------------------------------
module miscable_detector
import lacp_pkg::*;
#(
parameter int CONFIRM_PDUS = 3 // three consecutive, not one
)(
input logic clk,
input logic rst_n,
input logic pdu_valid [MAX_MEMBERS],
input lacp_id_t rx_actor [MAX_MEMBERS],
input logic [47:0] expected_partner_sys, // the majority pairing
input logic expected_valid,
input logic [MAX_MEMBERS-1:0] port_enabled, // 15.1's physical up
output logic [MAX_MEMBERS-1:0] miscabled,
output logic [47:0] found_sys [MAX_MEMBERS],
output logic [31:0] c_miscables,
output logic [15:0] traffic_at_risk_pct
);
logic [3:0] streak [MAX_MEMBERS];
always_ff @(posedge clk or negedge rst_n) begin
int m, n_bad, n_up;
if (!rst_n) begin
for (m = 0; m < MAX_MEMBERS; m++) begin
streak[m] <= '0; found_sys[m] <= '0;
end
miscabled <= '0; c_miscables <= '0; traffic_at_risk_pct <= '0;
end else begin
for (m = 0; m < MAX_MEMBERS; m++) begin
if (pdu_valid[m] && expected_valid) begin
if (rx_actor[m].sys_id != expected_partner_sys) begin
found_sys[m] <= rx_actor[m].sys_id;
if (streak[m] != 4'hF) streak[m] <= streak[m] + 1'b1;
if (streak[m] == CONFIRM_PDUS-1) begin
miscabled[m] <= 1'b1;
c_miscables <= c_miscables + 1;
end
end else begin
streak[m] <= '0;
miscabled[m] <= 1'b0;
end
end
// Losing the port clears the finding: an absent cable is a
// different fault with a different remedy.
if (!port_enabled[m]) begin
streak[m] <= '0;
miscabled[m] <= 1'b0;
end
end
// What fraction of the aggregate's traffic a STATIC config would
// be sending to the wrong device. This is the number that makes
// the finding urgent rather than interesting.
n_bad = 0; n_up = 0;
for (m = 0; m < MAX_MEMBERS; m++) begin
if (port_enabled[m]) n_up = n_up + 1;
if (miscabled[m]) n_bad = n_bad + 1;
end
traffic_at_risk_pct <= (n_up == 0) ? 16'd0
: 16'((n_bad * 100) / n_up);
end
end
endmoduleClassification: a confirmation counter over a comparison. It adds no capability to Section 8 and adds the reporting that makes it usable.
What it teaches: that the confirmation streak exists because a single mismatched LACPDU is normal. During bring-up, before a partner is established, and immediately after a topology change, a device can legitimately receive one LACPDU whose actor differs from the expectation — the expectation was itself derived from partial information. Three consecutive is 3 seconds in fast mode and is unambiguous.
And it teaches that traffic_at_risk_pct is the output that changes the conversation. miscabled[2] is a finding. "25% of this aggregate's traffic would be going to the wrong device" is an incident, and it is the same information with the arithmetic done. A design that reports the bit and not the percentage has left the most important step to a human under time pressure.
Deliberately simplified: expected_partner_sys is an input, and deriving it well is the subtle part — the majority pairing across the bundle, not the first port's, for the reason Section 8's simplification described. On a two-member aggregate with one mis-cable there is no majority, and the correct behaviour is to report both as unresolved rather than to pick one.
Production implication: found_sys[m] is the MAC address of the device actually at the other end, and Chapter 5.3's OUI in its top 24 bits usually names the vendor. An operator given "port 8 reaches 00:1B:21:… and ports 5–7 reach 00:25:90:…" has both the fault and a strong hint about which rack to walk to — and given only "member 8 excluded" has neither.
11. The Mis-Cable Static Configuration Cannot See
This is the failure the whole protocol justifies itself with, and it is worth walking through slowly because every local indicator is green.
Four cables leave switch A intended for switch B. Three reach B. The fourth reaches switch C — a different device, in a different rack, plugged into the wrong port during a maintenance window.
What each detection source reports:
| Source | Reports | Correct? |
|---|---|---|
Chapter 15.1 §6 phy_link_up | up | yes — there is a device there |
Chapter 15.1 §6 pcs_block_lock | locked | yes — the PHY is receiving valid blocks |
| Chapter 11.2's auto-negotiation | 10 Gb/s full duplex | yes — it negotiated correctly |
Chapter 15.1 §15 conformant | high | yes — nothing this device did is wrong |
| the operator's configuration | four members | it records the intention |
c_miscables | 1 | the only source that knows |
And under static configuration, the consequences:
25% of the aggregate's traffic is delivered to switch C, which has no entry for those destinations and floods them — Chapter 12.4's 23× amplification on a 24-port switch, applied to traffic that had a perfectly good destination.
Switch B sees the aggregate's source addresses on three ports rather than four, which is harmless.
Switch C sees them on one port and learns them — Chapter 12.2 — so C's forwarding table now points at A for addresses that live behind B, and traffic C receives for those destinations goes to A and is forwarded again. A forwarding loop is not created, because the frames do reach their destination eventually; what is created is a path that carries a quarter of the traffic through a device nobody intended.
And return traffic is worse. Switch C, distributing across its own aggregate to A, has no reason to use the mis-cabled port at all — so the failure is asymmetric: A sends into C and C does not send back, which makes the octet counters on the two ends of that cable wildly different and is the one local signal that could have hinted at it.
==
LACP refuses the member. Three LACPDUs — 3 seconds in fast mode — establish that the actor system id on that port is not the one on the other three, individual is set, selected is clear, and the aggregate runs as a healthy three-member LAG at 75% capacity while an alarm names the port and the MAC address it actually reaches.
Which is the whole trade, in one line: 1184 bit/s buys the difference between a three-member aggregate with an alarm and a four-member aggregate quietly misdelivering a quarter of its traffic.
12. RTL 6 — The Periodic Timer and Timeout
Two timers per port, in a three-to-one relationship that is the protocol's only tolerance for loss.
// -----------------------------------------------------------------------
// lacp_periodic_timer -- transmit cadence and receive expiry.
//
// The transmit period is chosen by the PARTNER's timeout bit, not by
// ours: a partner that says it wants fast timeouts is telling us to
// transmit every second. This inversion is the most commonly
// implemented backwards thing in the protocol.
// -----------------------------------------------------------------------
module lacp_periodic_timer
import lacp_pkg::*;
#(
parameter int CLK_HZ = 500000000
)(
input logic clk,
input logic rst_n,
input logic port_enabled,
input logic our_activity, // active or passive
input logic partner_activity,
input logic partner_wants_fast, // partner state bit 1
input logic we_want_fast, // our own preference, sent out
input logic pdu_received,
output logic tx_now,
output logic rx_timeout,
output logic rx_expired_once,
output logic [31:0] c_timeouts,
output logic [31:0] c_tx_periods,
output logic [31:0] since_last_rx_ms
);
localparam int FAST_PERIOD = CLK_HZ; // 1 s
localparam int SLOW_PERIOD = CLK_HZ * 30; // 30 s
localparam int MS_DIV = CLK_HZ / 1000;
logic [35:0] tx_cnt, rx_cnt, ms_cnt;
logic [35:0] tx_period, rx_limit;
// TRANSMIT at the rate the PARTNER asked for.
assign tx_period = partner_wants_fast ? 36'(FAST_PERIOD)
: 36'(SLOW_PERIOD);
// EXPIRE after three of OUR chosen periods -- we asked for this rate,
// so we are entitled to expect it.
assign rx_limit = we_want_fast ? 36'(3 * FAST_PERIOD)
: 36'(3 * SLOW_PERIOD);
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
tx_cnt <= '0; rx_cnt <= '0; ms_cnt <= '0;
tx_now <= 1'b0; rx_timeout <= 1'b0; rx_expired_once <= 1'b0;
c_timeouts <= '0; c_tx_periods <= '0; since_last_rx_ms <= '0;
end else begin
tx_now <= 1'b0;
if (!port_enabled) begin
tx_cnt <= '0; rx_cnt <= '0; rx_timeout <= 1'b0;
end else begin
// A passive port transmits only if its partner is active.
// Two passive ports never aggregate -- section 13's table.
if (our_activity || partner_activity) begin
if (tx_cnt >= tx_period - 1) begin
tx_cnt <= '0;
tx_now <= 1'b1;
c_tx_periods <= c_tx_periods + 1;
end else tx_cnt <= tx_cnt + 1'b1;
end
if (pdu_received) begin
rx_cnt <= '0;
rx_timeout <= 1'b0;
ms_cnt <= '0;
since_last_rx_ms <= '0;
end else begin
if (rx_cnt < rx_limit) begin
rx_cnt <= rx_cnt + 1'b1;
if (rx_cnt == rx_limit - 1) begin
rx_timeout <= 1'b1;
rx_expired_once <= 1'b1;
c_timeouts <= c_timeouts + 1;
end
end
if (ms_cnt == MS_DIV-1) begin
ms_cnt <= '0;
since_last_rx_ms <= since_last_rx_ms + 1;
end else ms_cnt <= ms_cnt + 1'b1;
end
end
end
end
endmoduleClassification: two independent counters with an asymmetric parameterisation. No datapath, one bit of output that matters.
What it teaches: that the transmit rate is chosen by the partner and the expiry by us, and getting the two the same way round breaks the protocol quietly. A device that transmits at its own preferred rate while its partner expects the other either floods a slow partner with 30× the frames it asked for, or starves a fast partner and is timed out every three seconds. The second is worse and it is the one that occurs: a fast-mode partner paired with a device transmitting slowly sees a timeout every 3 seconds, detaches, re-attaches, and produces Section 14's churn indefinitely.
And it teaches that since_last_rx_ms is worth its 32 bits. Is the partner talking to us is answered by a counter; "how long ago" is answered only by a timestamp, and it is what separates a partner that stopped a moment ago from one that has never spoken. A freshly configured port and a failed neighbour both read c_received = 0.
Deliberately simplified: the 36-bit counters at 500 MHz cover 30 seconds with room, and a production design would use a prescaled millisecond tick rather than counting core clocks — 15 billion cycles for a slow timeout is a lot of toggling for a timer whose resolution requirement is seconds.
Production implication: c_timeouts rising on a link whose physical layer is clean is the one-way failure Chapter 15.1 §6's proto_alive input was reserved for. Our transmit direction is dead: the far end hears nothing, stops replying, our receive timer expires. Both PHYs report up, both PCSs are locked, and the only evidence in the entire system is this counter.
13. Timeout Arithmetic — Fast and Slow
The choice between the two modes looks like a bandwidth trade and is not.
| fast | slow | Ratio | |
|---|---|---|---|
| periodic transmit | 1 s | 30 s | 30× |
| timeout | 3 s | 90 s | 30× |
| frames per second | 1 | 0.033 | — |
| bandwidth | 1184 bit/s | 39.5 bit/s | 30× |
| of a 1 Gb/s link | 1.18 × 10⁻⁴% | 3.95 × 10⁻⁶% | — |
| of a 100 Gb/s link | 1.18 × 10⁻⁶% | 3.95 × 10⁻⁸% | — |
Fast mode costs one part in a million of a gigabit link. There is no bandwidth case for slow mode at any line rate this track has covered.
What the choice actually costs is measured in octets lost during a failure the physical layer cannot see:
| Detection | 1 Gb/s | 10 Gb/s | 100 Gb/s |
|---|---|---|---|
| fast — 3 s | 0.38 GB | 3.75 GB | 37.5 GB |
| slow — 90 s | 11.25 GB | 112.5 GB | 1125 GB |
And both numbers are enormous, which is the point that reframes the whole comparison. LACP is not the failure detector for an ordinary link failure — Chapter 15.1 §7 established that a PHY interrupt does that in 10 µs, six orders of magnitude faster. LACP's timeout is the detector of last resort for the failures nothing else sees, and for those the choice is between losing 37.5 GB and losing 1125 GB.
So the real argument for slow mode is not bandwidth — it is CPU. On a switch whose LACP runs in software, a 48-port chassis in fast mode processes 48 LACPDUs per second and must transmit 48, with jitter tolerances measured in seconds. That is trivial today and was not always, and slow mode is the default in many implementations for a reason that expired around the time the hardware stopped being the constraint.
The recommendation that falls out is unambiguous: fast mode, on every aggregate, unless something specific forbids it — and the number that justifies it is the 1125 GB, not the 39.5 bit/s.
One more row completes the picture, because the activity bit interacts with all of this:
| Our activity | Partner activity | Result |
|---|---|---|
| active | active | both transmit; normal |
| active | passive | we transmit, they reply — works |
| passive | active | they transmit, we reply — works |
| passive | passive | neither ever transmits — no aggregate, ever |
The last row is a configuration that produces a bundle of links that never aggregate and never report an error, because nothing has gone wrong: both ends are waiting to be spoken to. At least one end must be active, and a design whose default is passive has shipped an aggregate that only works if the neighbour was configured differently.
14. RTL 7 — Churn Detection
A member that repeatedly attaches and detaches is worse than one that stays down, and the protocol has a name for the condition.
// -----------------------------------------------------------------------
// lacp_churn_detector -- reports a port that fails to reach a stable
// synchronised state within a bounded time.
//
// Churn is the protocol-level version of 15.1 section 6's flapping
// member, and it has an additional cause that a flapping cable does
// not: two ends that disagree in a way that oscillates.
// -----------------------------------------------------------------------
module lacp_churn_detector
import lacp_pkg::*;
#(
parameter int CHURN_LIMIT = 2000000000/4, // ~1 s at 500 MHz
parameter int WINDOW_CYCLES = 5000000000/10 // ~1 s window
)(
input logic clk,
input logic rst_n,
input logic port_enabled,
input logic actor_sync,
input logic partner_sync,
input logic attach_pulse,
input logic detach_pulse,
output logic actor_churn,
output logic partner_churn,
output logic [15:0] attaches_per_window,
output logic [31:0] c_churn_events,
output logic [31:0] longest_unsync_ms
);
logic [31:0] a_unsync, p_unsync, win, ms_div;
logic [15:0] att_cnt;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
a_unsync <= '0; p_unsync <= '0; win <= '0; ms_div <= '0;
att_cnt <= '0; attaches_per_window <= '0;
actor_churn <= 1'b0; partner_churn <= 1'b0;
c_churn_events <= '0; longest_unsync_ms <= '0;
end else if (!port_enabled) begin
a_unsync <= '0; p_unsync <= '0;
actor_churn <= 1'b0; partner_churn <= 1'b0;
end else begin
// Actor churn: WE cannot reach synchronised. Usually our own
// selection logic oscillating -- section 8's reference moving.
if (actor_sync) begin
a_unsync <= '0;
actor_churn <= 1'b0;
end else begin
a_unsync <= a_unsync + 1;
if (a_unsync == CHURN_LIMIT-1) begin
actor_churn <= 1'b1;
c_churn_events <= c_churn_events + 1;
end
if (a_unsync > longest_unsync_ms * 500000)
longest_unsync_ms <= a_unsync / 500000;
end
// Partner churn: THEY cannot reach synchronised. A different
// fault entirely and usually theirs, which is why it is a
// separate bit rather than a shared "not synchronised".
if (partner_sync) begin
p_unsync <= '0;
partner_churn <= 1'b0;
end else begin
p_unsync <= p_unsync + 1;
if (p_unsync == CHURN_LIMIT-1) begin
partner_churn <= 1'b1;
c_churn_events <= c_churn_events + 1;
end
end
if (attach_pulse) att_cnt <= att_cnt + 1'b1;
if (win == WINDOW_CYCLES-1) begin
attaches_per_window <= att_cnt;
att_cnt <= '0;
win <= '0;
end else win <= win + 1'b1;
end
end
endmoduleClassification: two independent stall watchdogs plus a rate counter. It reports and changes nothing.
What it teaches: that actor churn and partner churn are separate bits because they name different parties, and the whole of Module 14's counter discipline says why. We cannot synchronise is a local fault — Section 8's reference oscillating, a selection function disagreeing with itself. "They cannot synchronise" is theirs. One combined not synchronised bit sends every investigation to the same place, and half of them are at the wrong end of the cable.
And it teaches that attaches_per_window is the number that distinguishes churn from a slow start. A member that takes ten seconds to attach once is a bring-up. A member attaching four times a second is costing Chapter 15.2 §12's rebalance eight times a second — 74.9% of flows relocated on each, with a modulo reduction — and the aggregate spends more time redistributing than forwarding.
Deliberately simplified: the millisecond conversion divides by a hard-coded 500 000, which ties the module to a 500 MHz clock. A production design takes a millisecond tick as an input, because a switch's LACP logic frequently runs on a slower management clock than its datapath.
Production implication: longest_unsync_ms on a healthy network is the empirical basis for CHURN_LIMIT, exactly as Chapter 14.4 §13's longest_stall was for STALL_LIMIT. A threshold chosen from first principles is a guess; one set several times above the observed maximum will not cry wolf — and a churn detector that fires spuriously is disabled, taking the real detections with it.
15. RTL 8 — Conformance for a Negotiated Aggregate
The monitor's job is to separate what this device did wrong from what the negotiation found wrong at the other end.
// -----------------------------------------------------------------------
// lacp_conformance_monitor -- one bit per aggregate.
//
// It distinguishes OUR faults from FINDINGS about the far end. A
// mis-cable is not a defect in this device and must not clear its
// conformance bit -- it is the protocol working.
// -----------------------------------------------------------------------
module lacp_conformance_monitor
import lacp_pkg::*;
(
input logic clk,
input logic rst_n,
// Our faults.
input logic tx_period_wrong, // ignored the partner's rate
input logic sync_without_mirror, // set sync without they_see_us
input logic distributed_early, // distributing before partner collecting
input logic both_passive, // configuration: never aggregates
input logic no_active_end,
// Findings about the far end -- NOT our faults.
input logic [MAX_MEMBERS-1:0] miscabled,
input logic partner_churn,
input logic partner_timeout,
output logic conformant,
output logic [7:0] fault_vector,
output logic [7:0] finding_vector,
output logic aggregate_degraded
);
logic v_period, v_mirror, v_early, v_passive;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
v_period <= 1'b0; v_mirror <= 1'b0;
v_early <= 1'b0; v_passive <= 1'b0;
end else begin
if (tx_period_wrong) v_period <= 1'b1;
if (sync_without_mirror) v_mirror <= 1'b1;
if (distributed_early) v_early <= 1'b1;
// Standing configuration property: at least one end active.
v_passive <= both_passive || no_active_end;
end
end
assign conformant = !(v_period || v_mirror || v_early || v_passive);
assign fault_vector = {4'b0000, v_passive, v_early, v_mirror, v_period};
assign finding_vector = {5'b00000, partner_timeout, partner_churn,
|miscabled};
// A separate output: the aggregate is smaller than configured for a
// reason the protocol UNDERSTOOD. This is the alarm an operator
// wants, and it is not a conformance failure.
assign aggregate_degraded = (|miscabled) || partner_churn || partner_timeout;
endmoduleClassification: a two-vector fault aggregator. One vector is about us and one is about the world.
What it teaches: that a mis-cable must not clear this device's conformance bit, and the temptation to make it do so is strong. A quarter of the aggregate is excluded, an alarm is firing, and the natural instinct is for conformant to go low. But nothing this device did is wrong — it detected a fault correctly and responded correctly, and a conformance bit that goes low on a correct detection cannot be used to answer "is my LACP implementation working".
And it teaches that aggregate_degraded is what operations actually alarms on, which is neither vector alone. It fires on findings and not on faults, so an operator gets paged for the cable and an engineer gets the fault vector for the implementation. Merging them produces one alarm that means two unrelated things.
Deliberately simplified: sync_without_mirror is presented as an input, and producing it means checking that our_state.synchronised is never set while they_see_us is low. That check is one comparison and it catches the single most damaging implementation bug in the protocol — a device that synchronises on the partner's sync bit alone, which aggregates a mis-cabled member enthusiastically.
Production implication: the two vectors want different retention. fault_vector should be sticky — an implementation bug that occurred once is worth knowing about for ever. finding_vector should be live, because a mis-cable that has been fixed is no longer a finding and an operator needs the alarm to clear. A design that makes both sticky produces an aggregate that reports a mis-cable months after the cable was moved.
16. The Marker Protocol and Ordering During a Move
Chapter 15.1 §8 solved the reordering problem at a failover by waiting for queues to drain. 802.1AX defines a mechanism that does it exactly instead of conservatively, and almost nobody implements it.
The problem: moving a flow from member 2 to member 3 reorders it if member 2 still holds frames of that flow. Chapter 15.1 §8's answer was to wait until every remaining queue is empty — conservative, bounded by a parameter, and it times out on a congested aggregate.
The Marker protocol's answer is to ask.
| Step | What happens |
|---|---|
| 1 | stop sending the flow on member 2 |
| 2 | send a Marker PDU on member 2 — subtype 0x02, same 128-octet frame shape |
| 3 | the far end replies Marker Response on the same member |
| 4 | the response proves every earlier frame on member 2 has been received |
| 5 | start sending the flow on member 3 |
Step 4 is the exact condition Chapter 15.1 §8 approximated. The Marker is an ordinary frame in the same queue as the data, so a response to it means everything queued ahead of it has left — which is precisely, and only, what the ordering constraint requires.
| Chapter 15.1 §8's drain | Marker | |
|---|---|---|
| condition | every remaining queue empty | this member's earlier frames delivered |
| precision | conservative | exact |
| duration on an idle aggregate | immediate | one round trip — ~1 µs on a short link |
| duration on a congested aggregate | times out at 1 ms | as long as the queue takes |
| reordering risk | on timeout | none |
| requires the far end | no | yes — it must implement Marker Response |
The last row is why it is rare. Marker is optional; a device must respond to a Marker if it receives one, but the standard does not require anybody to send them — so an implementation that relies on Marker must fall back when the far end does not respond, which means implementing the drain anyway.
And that is the honest reason the drain is what ships: it works unilaterally. Chapter 15.1 §8's c_drain_timeout is the price of not needing the neighbour's cooperation, and the Marker protocol is the mechanism that would remove it if both ends could be assumed.
17. What LACP Costs Against Static Configuration
Put the whole comparison in one place, because the protocol is frequently disabled on the grounds that static is simpler and the grounds are true.
| static | LACP | |
|---|---|---|
| configuration effort | two ends, independently | two ends, independently |
| bandwidth | zero | 1184 bit/s fast, 39.5 slow |
| state per port | a membership bit | ~40 octets — two lacp_id_t, two state bytes, two timers |
| logic | none | a parser, a builder, a mux machine, a matcher |
| time to bring a member up | immediate | ~2 s |
| frames lost bringing a member up | up to a round trip | zero |
| member count mismatch | undetected | refused |
| mis-cabled member | undetected — 25% misdelivered | refused in 3 s, MAC reported |
| far end not aggregating | undetected — table flaps | refused |
| one-way failure | undetected | detected at timeout |
| standby members | not possible | negotiated, promoted on failure |
| ordering during a move | drain only | drain, or Marker if both ends have it |
| fails when | the cabling is wrong | the far end does not run it |
The last row is the only real argument against it and it is a shrinking one. A device that does not run LACP will not aggregate with one that requires it, so an aggregate to a host with a simple NIC bond, or to an appliance with a fixed configuration, may have to be static.
And the state cost is worth putting in proportion against the batch:
| Mechanism | State | What it buys |
|---|---|---|
| LACP — this chapter | ~320 octets, 8 ports | the mis-cable, the one-way failure, the mismatch |
| aggregation — Chapter 15.1 §17 | 2.19 KiB | survives a member failure |
| distribution — Chapter 15.2 §18 | 3.8 KiB | the failover moves 25% instead of 74.9% |
| PFC — Chapter 14.4 §18 | 1.45 MiB | collateral 88% → 11% |
LACP is the smallest mechanism in Module 15 and it is the only one that can see outside the device. Everything else in the module reasons about state this switch holds; LACP's entire value is that it carries a statement from somebody else, and 320 octets is what holding that statement costs.
Which is the module's closing observation and it generalises past Ethernet: the cheapest information a distributed system can acquire is usually the information it cannot derive locally at any price.
18. Properties Worth Asserting, and One Worth Refusing
The properties divide by what they protect: the frame, the parse, the mux machine, the selection, the timers, and the configuration.
Group 1 — the frame.
// P1. The LACPDU is addressed to the Slow Protocols multicast and
// carries the Slow Protocols EtherType and the LACP subtype.
property p_frame_identity;
@(posedge clk) disable iff (!rst_n)
tx_sop |-> ##12 (tx_octet == SLOW_ET[15:8]) ##1 (tx_octet == SLOW_ET[7:0])
##1 (tx_octet == LACP_SUB);
endproperty
// P2. The frame is exactly 124 body octets -- 128 with the FCS. No
// padding: the payload is well above 5.6's 46-octet minimum.
property p_frame_length;
@(posedge clk) disable iff (!rst_n)
tx_sop |-> ##123 tx_eop;
endproperty
// P3. The source address is the DEVICE's identity, not the port's.
// All of a device's ports must claim one identity or the far end
// sees several one-member aggregates.
property p_sa_is_device_id;
@(posedge clk) disable iff (!rst_n)
tx_sop |-> ##6 (tx_octet == actor.sys_id[47:40]);
endproperty
// P4. The actor TLV always carries our current state, not a snapshot
// taken when the request was queued.
property p_actor_state_is_current;
@(posedge clk) disable iff (!rst_n)
(tx_valid && (idx == 7'd32)) |-> (tx_octet == actor_state);
endproperty
// P5. The builder never overlaps frames.
property p_no_overlap;
@(posedge clk) disable iff (!rst_n)
tx_sop |-> !tx_sop throughout (tx_eop [->1]);
endpropertyGroup 2 — the parse.
// P6. Nothing is committed on a bad FCS.
property p_no_commit_bad_fcs;
@(posedge clk) disable iff (!rst_n)
(rx_eop && !rx_fcs_ok) |-> !pdu_valid;
endproperty
// P7. A non-LACP slow protocol is not an error. Marker frames arrive
// legitimately at a device that does not implement them.
property p_marker_is_not_an_error;
@(posedge clk) disable iff (!rst_n)
(rx_eop && da_ok && et_ok && !sub_ok) |-> !$changed(c_bad_fcs);
endproperty
// P8. pdu_valid is one cycle wide -- the mux machine reacts to a pulse.
property p_commit_is_a_pulse;
@(posedge clk) disable iff (!rst_n)
pdu_valid |=> !pdu_valid;
endproperty
// P9. they_see_us is exactly the three-field comparison it claims.
property p_mirror_comparison;
@(posedge clk) disable iff (!rst_n)
pdu_valid |-> (they_see_us == ((rx_partner.sys_id == our_actor.sys_id) &&
(rx_partner.key == our_actor.key) &&
(rx_partner.port_id == our_actor.port_id)));
endpropertyGroup 3 — the mux machine.
// P10. Collecting is never enabled after distributing. 15.1 section 3's
// ordering, and the reason no frame is lost bringing a member up.
property p_collect_before_distribute;
@(posedge clk) disable iff (!rst_n)
$rose(distributing) |-> collecting;
endproperty
// P11. We never distribute unless the partner says it is collecting.
property p_distribute_needs_partner_collecting;
@(posedge clk) disable iff (!rst_n)
distributing |-> rx_actor_state.collecting;
endproperty
// P12. Losing the port, the partner or the selection detaches
// immediately. There is no partial state to linger in.
property p_loss_detaches;
@(posedge clk) disable iff (!rst_n)
(!port_enabled || rx_timeout || !selected) |=> (mux == MUX_DETACHED);
endproperty
// P13. Detached means neither role is active.
property p_detached_is_inert;
@(posedge clk) disable iff (!rst_n)
(mux == MUX_DETACHED) |-> (!collecting && !distributing);
endproperty
// P14. WAITING lasts exactly AGGREGATION_WAIT, so several ports
// selecting together settle together.
property p_waiting_is_timed;
@(posedge clk) disable iff (!rst_n)
$rose(mux == MUX_WAITING) |-> ##AGGREGATION_WAIT (mux == MUX_ATTACHED);
endproperty
// P15. Synchronised requires BOTH they_see_us and their sync bit.
// Checking only the second aggregates a mis-cabled member.
property p_sync_needs_mirror;
@(posedge clk) disable iff (!rst_n)
our_state.synchronised |-> they_see_us;
endpropertyGroup 4 — the selection.
// P16. A selected port's partner is the same device as the reference.
property p_selected_same_partner;
@(posedge clk) disable iff (!rst_n)
selected[m] |-> (rx_actor[m].sys_id == chosen_partner_sys);
endproperty
// P17. And the same far-end aggregation key.
property p_selected_same_key;
@(posedge clk) disable iff (!rst_n)
selected[m] |-> (rx_actor[m].key == chosen_partner_key);
endproperty
// P18. A port whose partner is a DIFFERENT device is individual, never
// selected. The mis-cable, as a property.
property p_wrong_partner_is_individual;
@(posedge clk) disable iff (!rst_n)
(partner_valid[m] && (rx_actor[m].sys_id != chosen_partner_sys))
|=> (individual[m] && !selected[m]);
endproperty
// P19. Not having heard is distinct from having heard the wrong thing.
property p_silence_is_not_individual;
@(posedge clk) disable iff (!rst_n)
!partner_valid[m] |-> !individual[m];
endproperty
// P20. A mis-cable finding needs CONFIRM_PDUS consecutive
// disagreements -- one is normal during bring-up.
property p_miscable_needs_confirmation;
@(posedge clk) disable iff (!rst_n)
$rose(miscabled[m]) |-> ($past(streak[m]) == CONFIRM_PDUS-1);
endproperty
// P21. Losing the port clears the finding: an absent cable is a
// different fault with a different remedy.
property p_port_down_clears_miscable;
@(posedge clk) disable iff (!rst_n)
!port_enabled[m] |=> !miscabled[m];
endpropertyGroup 5 — the timers.
// P22. We transmit at the rate the PARTNER asked for.
property p_tx_rate_follows_partner;
@(posedge clk) disable iff (!rst_n)
partner_wants_fast |-> (tx_period == FAST_PERIOD);
endproperty
// P23. We expire after three of OUR chosen periods.
property p_expiry_follows_us;
@(posedge clk) disable iff (!rst_n)
we_want_fast |-> (rx_limit == 3*FAST_PERIOD);
endproperty
// P24. Any received PDU resets the receive timer.
property p_rx_resets_timer;
@(posedge clk) disable iff (!rst_n)
pdu_received |=> (rx_cnt == '0);
endproperty
// P25. A passive port with a passive partner never transmits. This is
// a legal configuration that never aggregates, and the property makes
// it visible rather than mysterious.
property p_passive_pair_is_silent;
@(posedge clk) disable iff (!rst_n)
(!our_activity && !partner_activity) |-> !tx_now;
endproperty
// P26. Churn is raised only after CHURN_LIMIT of continuous unsync.
property p_churn_needs_persistence;
@(posedge clk) disable iff (!rst_n)
$rose(actor_churn) |-> ($past(a_unsync) == CHURN_LIMIT-1);
endproperty
// P27. Synchronising clears the churn evidence.
property p_sync_clears_churn;
@(posedge clk) disable iff (!rst_n)
actor_sync |=> (a_unsync == '0);
endpropertyGroup 6 — the configuration and the monitor.
// P28. Standing property: at least one end is active.
property p_an_active_end_exists;
@(posedge clk) disable iff (!rst_n)
!no_active_end;
endproperty
// P29. A mis-cable is a FINDING, not a fault. It must not clear this
// device's conformance bit -- the protocol worked.
property p_finding_is_not_a_fault;
@(posedge clk) disable iff (!rst_n)
(|miscabled && (fault_vector == 8'h00)) |-> conformant;
endproperty
// P30. But it must raise the degraded alarm.
property p_finding_degrades;
@(posedge clk) disable iff (!rst_n)
|miscabled |-> aggregate_degraded;
endproperty
// P31. conformant is exactly the conjunction of OUR faults.
property p_conformant_definition;
@(posedge clk) disable iff (!rst_n)
conformant <-> (fault_vector == 8'h00);
endpropertyEvery property above is about state this device holds or messages it has actually received. P15 is the one that prevents Section 11's mis-cable from being aggregated, and it is deliberately phrased as a check on our own synchronised bit rather than as a claim about the partner — which is the subject of the property this chapter refuses.
19. Verification Scenarios
Sixty-nine scenarios. The ones that matter show a link that every physical source calls healthy and the protocol refuses.
The frame
| # | Scenario | Expected |
|---|---|---|
| 1 | Build an LACPDU | 124 body octets, 128 with FCS, 148 on the wire |
| 2 | Same | DA 01:80:C2:00:00:02, EtherType 0x8809 |
| 3 | Same | subtype 0x01, version 0x01 |
| 4 | Same | no padding — the payload is 110 octets |
| 5 | SA | the device's MAC, not the port's |
| 6 | Four ports of one device | all four claim the same system id |
| 7 | A design using per-port SAs | far end sees four one-member aggregates |
| 8 | Actor TLV at offset 16 | type 0x01, length 20 |
| 9 | Partner TLV at offset 36 | type 0x02, length 20 |
| 10 | Fast mode | 1184 bit/s — 1.18 × 10⁻⁴% of a 1 Gb/s link |
| 11 | Slow mode | 39.5 bit/s |
The parse
| # | Scenario | Expected |
|---|---|---|
| 12 | Well-formed LACPDU | pdu_valid, one cycle |
| 13 | Bad FCS | no commit, c_bad_fcs + 1 |
| 14 | Subtype 0x02 — Marker | c_wrong_subtype, not an error |
| 15 | Same, counted as bad FCS | an error counter that rises in normal operation |
| 16 | Partner TLV matches us | they_see_us |
| 17 | Partner TLV names a different port | c_mismatched_mirror + 1 |
| 18 | Same, once, after a state change | normal — the mirror is one round trip stale |
| 19 | Same, continuously | the mis-cable |
| 20 | An unknown TLV before the terminator | hard-coded parser mis-parses; a chain-walker skips it |
The mux machine
| # | Scenario | Expected |
|---|---|---|
| 21 | Port comes up, selected | DETACHED → WAITING |
| 22 | After AGGREGATION_WAIT | → ATTACHED, collecting = 1, distributing = 0 |
| 23 | Partner reports collecting and synchronised | → COLLECT_DIST |
| 24 | Four ports up within 1 ms, with WAITING | one membership change |
| 25 | Same, without WAITING | four changes — Chapter 15.2 §12 runs four times |
| 26 | Same, modulo reduction | 74.9% of flows relocated, four times |
| 27 | distributing set before collecting | P10 fails |
| 28 | Partner not yet collecting | we do not distribute |
| 29 | Physical port drops | → DETACHED immediately |
| 30 | Receive timeout | → DETACHED |
| 31 | Same, with an EXPIRED state | keeps collecting, transmits fast, recovers in one round trip |
Selection and the mis-cable
| # | Scenario | Expected |
|---|---|---|
| 32 | Four cables, all to switch B | four selected |
| 33 | Three to B, one to C | C's port individual, three selected |
| 34 | Same, phy_link_up on the fourth | up |
| 35 | Same, pcs_block_lock | locked |
| 36 | Same, auto-negotiation | 10 Gb/s full duplex, correct |
| 37 | Same, Chapter 15.1 §15 conformant | high |
| 38 | Same, under static configuration | 25% of traffic to switch C |
| 39 | Same | C floods it — 23× amplification |
| 40 | Same | C learns our addresses on that port |
| 41 | Same | return traffic never uses that cable — asymmetric octet counters |
| 42 | Same, under LACP | refused in 3 s, traffic_at_risk_pct = 25 |
| 43 | found_sys reported | C's MAC — the OUI names the vendor |
| 44 | Mis-cable on the lowest port, first-port reference | three ports wrongly excluded |
| 45 | Same, majority reference | one port excluded |
| 46 | Two-member LAG, one mis-cabled | no majority — both unresolved |
| 47 | One mismatched PDU during bring-up | not a finding — needs 3 |
| 48 | Port goes down while mis-cabled | finding cleared — a different fault |
| 49 | Local key checked, partner not | aggregates any four cables — static with extra frames |
| 50 | Six ports, four-member limit | two standby, fully negotiated, not distributing |
| 51 | A member fails, standby present | promoted — no capacity loss |
Timers and churn
| # | Scenario | Expected |
|---|---|---|
| 52 | Fast mode | periodic 1 s, timeout 3 s |
| 53 | Slow mode | 30 s / 90 s |
| 54 | Partner wants fast, we transmit slow | partner times out every 3 s |
| 55 | Same | detach, re-attach, churn indefinitely |
| 56 | 100 Gb/s member, fast timeout | 37.5 GB lost |
| 57 | 100 Gb/s member, slow timeout | 1125 GB lost |
| 58 | 1 Gb/s member, slow timeout | 11.25 GB |
| 59 | Ordinary link failure, PHY interrupt wired | 10 µs — LACP's timer never fires |
| 60 | One-way failure, our TX dead | both PHYs up, both PCSs locked |
| 61 | Same | c_sent rising, c_received flat |
| 62 | Same | c_timeouts — the only evidence in the system |
| 63 | Both ends passive | neither transmits — no aggregate, no error |
| 64 | Actor cannot synchronise for 1 s | actor_churn |
| 65 | Partner cannot synchronise | partner_churn — a separate bit |
| 66 | Both merged into one bit | half the investigations go to the wrong end |
| 67 | Member attaching four times a second | rebalance eight times a second |
| 68 | Mis-cable present, conformant | high — the protocol worked |
| 69 | Same, aggregate_degraded | high — the alarm operations wants |
The directed test random stimulus will not produce
A mis-cable is not a traffic condition and not a fault injection — it is a topology in which one link's far end is a different, entirely healthy device. Random stimulus varies frames; it does not reconnect cables. And a fault-injection framework that models link failures models them as loss of signal, which is the one thing this failure does not produce.
Setup: three switches. A has a four-port aggregate, key 100, ports 5–8. B has a matching four-port aggregate, key 100. C is an unrelated switch with an ordinary access port, no aggregate. A's ports 5, 6 and 7 are cabled to B. A's port 8 is cabled to C. All four links negotiate 10 Gb/s full duplex and come up. A's forwarding table has 4096 addresses reachable via the aggregate.
Stimulus, two runs. Run 1: static configuration on both ends, all four ports members. Offer 4096 flows to the aggregate for 100 ms. Run 2: identical cabling, LACP fast mode on A and B, C not running LACP.
Oracle:
| # | Observable | Run 1 — static | Run 2 — LACP | Why it matters |
|---|---|---|---|---|
| 1 | phy_link_up, port 8 | up | up | there is a device there |
| 2 | pcs_block_lock, port 8 | locked | locked | valid blocks arriving |
| 3 | auto-negotiation, port 8 | 10G full | 10G full | it negotiated correctly |
| 4 | Chapter 15.1 §15 conformant | high | high | nothing A did is wrong |
| 5 | members in distributing | 4 | 3 | the finding |
| 6 | traffic on port 8 | ≈25% | 0 | — |
| 7 | frames delivered to C | ≈25% of the aggregate | 0 | the damage |
| 8 | C's flood amplification | 23× | — | Chapter 12.4 |
| 9 | C's table learns A's addresses | yes | no | a path nobody intended |
| 10 | return traffic on port 8 | ≈0 | 0 | asymmetric counters — the one local hint |
| 11 | c_mismatched_mirror, port 8 | n/a | rising continuously | — |
| 12 | miscabled[3] | n/a | set after 3 PDUs — 3 s | — |
| 13 | found_sys[3] | n/a | C's MAC | the OUI names the vendor |
| 14 | traffic_at_risk_pct | n/a | 25 | the incident, not the finding |
| 15 | aggregate_degraded | low | high | what operations is paged on |
| 16 | aggregate capacity | 4 members, 25% misdelivered | 3 members, 75%, correct | the trade |
| 17 | LACPDUs from C | n/a | none — C does not run it | port 8 never gets a partner |
| 18 | individual[3] against selected[3] | n/a | individual only after a partner is heard | silence is not a finding |
Rows 1 to 4 are identical in both runs and all four are green. That is the whole argument for the protocol: every indicator a switch can produce from its own hardware says this member is healthy, and one of them is delivering a quarter of the aggregate's traffic to the wrong building.
And row 17 is the honest caveat. C does not run LACP, so port 8 never acquires a partner at all — it is excluded for silence rather than for disagreement. Row 18 records that difference, because a mis-cable to a device that does run LACP is caught by the system-id comparison and one to a device that does not is caught by the timeout. Both work; they are different mechanisms and the counters should not conflate them.
20. Debugging LACP
Five questions, in order. Three of them are answered before any traffic is examined.
Step 1 — is anything being sent, and is anything arriving? c_sent against c_received. c_sent rising with c_received flat is a one-way path — our transmit is dead or the far end is not running LACP, and Section 19's row 17 separates those. Both flat is a passive/passive pair, which is a configuration that never aggregates and never errors.
Step 2 — what is actually at the other end? chosen_partner_sys and found_sys[m]. This is the read that resolves a mis-cable in seconds and the one most status outputs omit. Chapter 5.3's OUI in the top 24 bits usually names the vendor, which is frequently enough to identify the rack.
Step 3 — is a member excluded, and for which of two reasons? individual[m] against selected[m]. individual means we heard a partner and it was the wrong one — a mis-cable to an LACP-speaking device. Neither set means we have heard nothing at all, which is silence: the far end is not running LACP, or our transmit is dead, or the cable reaches something that does not speak it.
Step 4 — is the timing agreed? The partner's timeout bit against our transmit period. A device transmitting slowly to a partner that asked for fast is timed out every 3 seconds, and the symptom — a member that attaches and detaches endlessly — looks exactly like a flapping cable. c_timeouts rising while phy_link_up never falls separates them.
Step 5 — is it churning, and whose churn? actor_churn against partner_churn, and attaches_per_window. Actor churn is ours — usually Section 8's reference oscillating. Partner churn is theirs. And attaches_per_window above one or two is costing Chapter 15.2 §12's rebalance on every transition, which on a modulo reduction is 74.9% of flows each time.
And the state that ends an investigation: c_sent and c_received both rising, chosen_partner_sys matching the expected neighbour, all configured members selected, c_timeouts zero, attaches_per_window zero. That is a negotiated aggregate with nothing to find, and any remaining complaint about it belongs to Chapter 15.2's distribution or to Chapter 15.1 §13's flow count.
21. Common Misconceptions
1 — "LACP is how a LAG fails over."
The wrong model: the protocol detects member failures.
What it costs: a failover that takes 3 seconds instead of 10 microseconds — 37.5 GB on a 100 Gb/s member. Chapter 15.1 §7: a PHY interrupt detects loss of signal in ~10 µs and PCS block-lock loss is 106 ns at 10 Gb/s.
The corrected model: LACP is the detector of last resort, for the failures the physical layer cannot see — a one-way transmit failure, a mis-cable, a far end that stopped aggregating. Its timeout should almost never fire on a healthy network, and a design relying on it for ordinary failures has left the fast detectors unwired.
2 — "Slow mode saves bandwidth."
The wrong model: 30-second periods are the conservative choice.
What it costs: 1125 GB on a 100 Gb/s member instead of 37.5 GB, to save 1144 bit/s — which is 1.1 × 10⁻⁶% of the link.
The corrected model: the real historical argument was CPU, not bandwidth: a 48-port chassis running LACP in software processing 48 PDUs per second was once a load. It is not now. Fast mode on every aggregate unless something specific forbids it.
3 — "The far end says it is synchronised, so we are aggregated."
The wrong model: the partner's sync bit is sufficient.
What it costs: a mis-cabled member aggregated enthusiastically. The far end may be perfectly synchronised with somebody else — switch C in Section 11 is synchronised with whatever it is actually aggregating.
The corrected model: synchronisation requires two things — their sync bit and they_see_us, the comparison of their partner TLV against our own identity. p_sync_needs_mirror is the property, and a design checking only the first bit is the single most damaging implementation bug in the protocol.
4 — "A mis-cabled member means the switch is broken."
The wrong model: an alarm on an aggregate implies a device fault.
What it costs: an escalation to the wrong team, and a conformance bit that cannot answer "is my LACP implementation working". Nothing the switch did is wrong: it detected a cabling fault correctly and excluded the member correctly.
The corrected model: faults and findings are separate vectors. conformant describes this device; aggregate_degraded describes the world. Operations alarms on the second and engineering reads the first, and merging them produces one alarm meaning two unrelated things.
5 — "Both ends will agree because they run the same standard."
The wrong model: conformance implies identical behaviour.
What it costs: hours on an aggregate where four of eight members carry traffic one way only. Two switches with different maximum member counts — eight and four — both negotiate all eight successfully, then one selects eight and the other selects four with four standby.
The corrected model: the selection is computed independently at both ends and never exchanged, so the two must agree about the function and not merely about the inputs. The protocol converges anyway — because collecting and distributing report state rather than intention, so each end reacts to what the other is observed to be doing.
6 — "LACP will fix the imbalance."
The wrong model: negotiation covers everything about the aggregate.
What it costs: LACP enabled on an aggregate whose real problem is Chapter 15.2 §10's flow count, and no change. 802.1AX deliberately does not negotiate the distribution function — each end need only be consistent with itself.
The corrected model: LACP settles membership. Chapter 15.2 settles distribution, unilaterally and per direction. A four-flow aggregate is 47.0% efficient with LACP and without it, and the two mechanisms answer completely different questions.
22. Interview Reasoning
Q1 — What does LACP detect that a PHY cannot?
Anything about the identity of the device at the other end. A PHY reports that its own receiver has signal; PCS reports block lock; auto-negotiation settles speed and duplex. All three are perfectly happy on a mis-cabled member — there is a real device there, it negotiated correctly, the link is up. The only evidence that it is the wrong device is the actor system id in its LACPDUs, which no local measurement produces. Same for a one-way transmit failure: both ends' receivers are fine and only a protocol that expects to hear back notices.
Q2 — Walk through a mis-cable under static configuration.
Four cables, three to B, one to C. All four PHYs up, all four in the aggregate. 25% of the traffic goes to C, which has no entry for those destinations and floods them at Chapter 12.4's 23× on a 24-port switch. C learns our addresses on that port, so C now believes those stations are reachable via us. Return traffic never uses the cable — C's own distributor has no reason to — so the octet counters on the two ends of that one cable are wildly asymmetric, which is the single local hint available and nobody graphs it. LACP refuses the member in 3 seconds and reports C's MAC.
Q3 — Why does the mux machine have a WAITING state?
So that several ports selecting the same aggregate settle together. Four ports come up within milliseconds; without the wait, port 1 attaches alone, then 2, then 3, then 4 — four membership changes, so Chapter 15.2 §12's rebalance runs four times, and with a modulo reduction that relocates 74.9% of the flows on each occasion. For a link that is simply coming up.
Q4 — Which end chooses the transmit period, and which the timeout?
The partner chooses our transmit period and we choose our own timeout. A partner advertising the fast timeout bit is telling us to transmit every second; we expire after three of the periods we asked for, because we asked for them. Getting this backwards produces a device that transmits slowly to a partner expecting fast, which times out every 3 seconds, detaches, re-attaches, and churns for ever — with a symptom identical to a flapping cable.
Q5 — Two devices must select the same subset of ports. How, with no message that carries the answer?
They agree on who decides rather than on the set. The lower system priority wins, ties broken by the lower system id; both ends then apply the winner's port priorities, not their own. Both already have both system ids from the actor TLVs. If each ordered by its own priorities, a six-port bundle limited to four would produce different subsets and the intersection would carry traffic one way only. And when the two disagree about the selection function — different member limits, say — the collecting and distributing bits make it converge on the intersection anyway, because they report state rather than intention.
Q6 — Why can't you assert that the two ends agree?
Because the partner TLV describes us as we were one round trip ago — up to a full periodic interval, 1 second in fast mode. Any local state change makes the mirror wrong for exactly that interval, by design, so an equality assertion fires on every normal transition and ends up disabled into uselessness. The assertable form is convergence within a bound: after a state change, they_see_us becomes true within twice the partner's period. And a partner TLV with the defaulted bit set must not be compared at all — the far end is volunteering that the value is invented.
23. Understanding Check
==
24. What's Next
Module 15 is complete, and its three chapters turned out to be one argument about names.
Chapter 15.1 introduced a name that outlives the thing it names — an aggregate, so a forwarding entry stays valid when a member fails — and found that the work is never the name but finding every mechanism that already holds the old one. Chapter 12.2's learning was the one it had to find.
Chapter 15.2 built the function that maps a conversation to a member, and found that the obvious reduction moves three quarters of the flows when a quarter had to, and that no function at all can split a conversation larger than a member.
And this chapter replaced the operator's intention with the far end's statement, for 1184 bit/s — and found four failures that every local indicator calls healthy.
Module 16 — Precision Time Protocol — changes the question entirely. Instead of asking where a frame should go, it asks when something happened.
Chapter 16.1 — Why Sub-Microsecond Sync Is a Hardware Problem establishes what needs synchronised clocks at all, derives the error budget of a software timestamp from interrupt latency, scheduler jitter and stack traversal, and shows where in the datapath the timestamp actually has to be taken.
Then the module builds the message exchange, the timestamp point at the MAC/PHY boundary, the servo that closes the loop, and the asymmetries that limit what any of it can achieve.
One thread carries directly across, and it is this chapter's. LACP's whole value was that it carried a statement from somebody else about something no local measurement could produce. Module 16 is that problem in its purest form: a clock cannot measure its own error — it can only measure its difference from another clock, and that difference is contaminated by everything the measurement crossed to arrive.
Continue learning
Related tutorials
- Related topic
Bonding Links and Failover
Four links presented as one, constrained by an ordering rule nobody wrote down, with a failover whose cost is detection time times line rate — 12.5 GB at 100 Gb/s.
- Related topic
Hash-Based Distribution and Frame Ordering
The distributor's field selection, polynomial and reduction, why a modulo moves three quarters of the flows when one member fails, and the conversation that pins itself to one link.
- Related topic
The Shared-Medium Problem
Why several independent transmitters on one medium is a distributed timing problem, not a formatting problem. Propagation delay makes every station's view of the medium stale, so two locally correct decisions can still collide — and that is the constraint the Ethernet MAC was built around.
- Related topic
CSMA/CD, Collision Domains and Slot Time
Slot time is the parameter the whole half-duplex MAC hangs on: it bounds medium acquisition, bounds a collision fragment, and is the retransmission quantum. Deriving it from round-trip propagation plus jam is what fixes Ethernet's minimum frame size — a timing constant wearing a frame-format costume.
Standards & specifications
- Governing standard
- IEEE Std 802.3 (Ethernet)(opens IEEE in a new tab)
Defines the Ethernet MAC, the media-independent interfaces and the physical-layer sublayers, including framing, access control, auto-negotiation and per-rate PHY specifications. VLAN tagging, priority and time-sensitive shaping are defined by IEEE 802.1, not by 802.3.
This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.
Where this fits
Part of the Ethernet curriculum.
