25G to 100G — Per-Lane Rate and Lane Aggregation

Chapter 9.4 put four byte lanes on XGMII, and they arrive together because they are four sets of traces on one board, laid out to match.

Above 10 Gigabit that stops being true, and the change is not one of degree.

A 100 Gigabit link is built from multiple physical lanes — four fibres, four wavelengths, four differential pairs across a backplane — and they traverse different physical paths. Different lengths, different connectors, different retimers. They arrive at genuinely different times, by hundreds of nanoseconds, and nothing about the transmission makes them line up.

So the data is striped across them, each lane is given a name it repeats, and the receiver puts them back in order.

1. Scope — What This Chapter Owns

This chapter owns the mechanisms that turn several serial links into one Ethernet link.

It covers what sets the per-lane rate, block striping, the PCS lane count and why it is twenty, alignment markers and what they encode, the deskew buffer and the depth the standard implies, lane-mapping validation, and per-lane health.

It does not re-derive what other chapters own. Chapter 3.5 owns block coding; Chapter 9.4 owns 64B/66B and XGMII's byte lanes, whose arithmetic this chapter extends rather than repeats; Chapter 4.4 owns clock compensation, which is a different buffer solving a different problem. Chapter 9.6 owns PAM4 and the FEC that becomes mandatory with it.

The claim this chapter defends: lane aggregation replaces a physical guarantee — wires that arrive together — with a protocol that re-establishes order from markers the data carries; and the design's obligation is not that the lanes arrive close enough, but that it can say, exactly, when they did not.

2. Two Different Things Called a Lane

The word does two jobs in Ethernet and confusing them makes the rest of the chapter incoherent.

An XGMII byte lane is a position within one parallel transfer. Four lanes carry four bytes of the same transfer, on the same clock, across a length-matched board. Chapter 9.4 §9 showed what goes wrong there — a stuck lane, a swap — and every one of those faults is a board fault.

A PCS lane is a stream of 66-bit blocks with its own identity. Blocks are dealt to PCS lanes round-robin, each PCS lane periodically inserts a marker naming itself, and the lanes are then muxed onto whatever physical lanes the medium provides.

And that indirection is the point of the design.

Twenty PCS lanes can be carried on one physical lane, or two, four, five, ten or twenty — because 20 divides by all of them. The PCS does not know or care, and a link built from four 25G lanes and one built from ten 10G lanes present the identical stream to the MAC.

3. What Sets the Per-Lane Rate

Two numbers decide it, and the second is Chapter 9.4's line code.

Step 1 — divide the aggregate rate by the number of physical lanes.

Step 2 — multiply by 66/64, because every 64 payload bits go on the wire as 66.

Read the first three rows together, because they are the industry's actual strategy. 40GBASE-R4 and 100GBASE-R10 run at exactly 10GBASE-R's per-lane rate. The serial technology was not advanced at all — the link was made wider. Four lanes of a proven 10.3125 GBd serialiser is 40 Gigabit; ten of them is 100.

And 100GBASE-R4 is the later generation, in which the per-lane rate did move — to 25.78125 GBd — so 100 Gigabit needed four lanes instead of ten, with the connector, fibre and power savings that follows.

The PCS lane rate closes the arithmetic.

100 Gb/s ÷ 20 PCS lanes = 5 Gb/s of payload per PCS lane 5 × 66/64 = 5.15625 Gb/s of line rate per PCS lane

And muxing five PCS lanes onto each of four physical lanes:

5 × 5.15625 = 25.78125 Gb/s — exactly the physical lane rate above.

4. RTL 1 — Dealing Blocks Onto Lanes

// SYNTHESIZABLE.
//
// Stripes 66-bit blocks across PCS lanes, round-robin.
//
// WHY ROUND-ROBIN AND NOTHING SMARTER:
//   The receiver must reconstruct the original order from lanes that
//   arrive out of order, out of time, and possibly on different fibres.
//   The ONLY thing it has is each lane's alignment marker, which names
//   the lane. So the mapping from lane number to block position has to
//   be derivable from the lane number alone.
//
//   Round-robin gives exactly that: once the receiver knows which lane
//   is lane 7, it knows every block on it was block 7, 27, 47, ...
//
//   A load-balancing or gap-aware distribution would be measurably
//   better at nothing and would require an extra channel to describe
//   itself.
//
// THE NUMBERS:
//   100 Gb/s / 20 PCS lanes            =  5.00000 Gb/s payload per lane
//   5 x 66/64                          =  5.15625 Gb/s of line per lane
//   5 PCS lanes muxed per physical lane = 25.78125 Gb/s  (100GBASE-R4)
package lane_pkg;
 
  localparam int unsigned PCS_LANES  = 20;
  localparam int unsigned BLOCK_BITS = 66;
 
  // Alignment markers repeat every 16384 blocks on each PCS lane
  // (IEEE 802.3 clause 82). One marker in 16384 blocks is an overhead
  // of 1/16384 = 0.0061%, which is why the period is large: the marker
  // must be frequent enough to re-lock quickly and rare enough to cost
  // nothing.
  localparam int unsigned AM_PERIOD  = 16384;
 
  // Receive PCS skew tolerance: 180 ns of Skew, 4 ns of Skew Variation.
  localparam int unsigned MAX_SKEW_NS      = 180;
  localparam int unsigned MAX_SKEW_VAR_NS  = 4;
 
  typedef enum logic [1:0] {
    BLK_DATA,
    BLK_CONTROL,
    BLK_MARKER,
    BLK_INVALID
  } blk_kind_e;
 
endpackage
 
module lane_striper
  import lane_pkg::*;
#(
  parameter int unsigned CNT_W = 24
) (
  input  logic clk,
  input  logic rst_n,
 
  input  logic [BLOCK_BITS-1:0] blk_in,
  input  logic                  blk_valid,
 
  // One block, on one lane, per accepted input.
  output logic [BLOCK_BITS-1:0] lane_blk,
  output logic [4:0]            lane_sel,
  output logic                  lane_valid,
 
  // The deal position, exported so the marker inserter in Section 6 can
  // align its period to lane boundaries rather than to a free-running
  // counter of its own.
  output logic [4:0]            deal_index,
 
  output logic [CNT_W-1:0]      c_blocks
);
 
  logic [4:0] idx_q;
 
  assign deal_index = idx_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      idx_q <= 5'd0; lane_valid <= 1'b0; lane_sel <= 5'd0;
      lane_blk <= '0; c_blocks <= '0;
    end else begin
      lane_valid <= 1'b0;
 
      if (blk_valid) begin
        lane_blk   <= blk_in;
        lane_sel   <= idx_q;
        lane_valid <= 1'b1;
 
        // Wrap at PCS_LANES, not at 32. A five-bit counter free-running
        // to 31 would deal 32 lanes onto 20 and corrupt the mapping in
        // a way that still looks like round-robin.
        idx_q <= (idx_q == 5'(PCS_LANES - 1)) ? 5'd0 : idx_q + 5'd1;
 
        if (!(&c_blocks)) c_blocks <= c_blocks + 1'b1;
      end
    end
  end
 
endmodule

Classification: synthesizable.

What it teaches: that the striping rule is constrained by what the receiver can invert, not by what the transmitter can compute. The receiver's only input is each lane's marker; from that it must recover every block's original position. Round-robin makes the mapping a function of the lane number alone, which is the only kind of mapping a marker can describe.

Deliberately simplified: one block per cycle. A real 100G PCS stripes several blocks per clock, because a single-block-per-cycle design at 100 Gb/s would need a clock nobody can close.

Production implication: the wrap is at PCS_LANES, and the comment says why. A five-bit index that free-runs to 31 deals blocks onto lanes 20 through 31 that do not exist, and depending on the mux, those blocks land back on real lanes in a pattern that is still periodic — so the stream still looks striped, still locks, and delivers blocks in the wrong order. The failure is a reordering, not a corruption, which means the CRC catches it and nothing says why.

5. Why Twenty

The number is not a capacity decision. It is a divisibility decision.

20 divides by 1, 2, 4, 5, 10 and 20 — which is exactly the set of physical lane counts a 100 Gigabit medium might offer.

Every row is a whole number of PCS lanes per physical lane, which is the property that makes the mux trivial and the PCS reusable. A PCS lane count of 16 would have given 1, 2, 4, 8 and 16 and excluded the ten-lane optics that shipped first; a count of 24 would have excluded five.

And 40 Gigabit uses 4 PCS lanes by the same argument at a smaller scale: 40 ÷ 4 = 10 Gb/s payload per PCS lane, × 66/64 = 10.3125 Gb/s, carried on 1, 2 or 4 physical lanes.

6. RTL 2 — The Marker That Names a Lane

// SYNTHESIZABLE.
//
// Inserts alignment markers, which are the only thing in the stream
// that says which lane a lane is.
//
// WHAT A MARKER DOES, and it is three jobs:
//   1. IDENTITY  -- a per-lane unique pattern. Without it, twenty lanes
//        arriving in an unknown order are twenty anonymous streams and
//        there is no way to reconstruct the deal.
//   2. ALIGNMENT -- the markers of all twenty lanes were inserted at
//        the SAME deal position, so at the receiver their arrival times
//        measure the skew directly. The marker is the reference edge.
//   3. INTEGRITY -- a BIP field, computed over the blocks since the
//        last marker, gives a PER-LANE error measure that no
//        frame-level counter can provide.
//
// PERIOD: one marker every 16384 blocks per lane (clause 82).
//   overhead = 1/16384 = 0.0061%
//   The period is a compromise: frequent enough that a receiver relocks
//   quickly and that BIP windows are short, rare enough to cost nothing.
module alignment_marker_inserter
  import lane_pkg::*;
#(
  parameter int unsigned CNT_W = 24
) (
  input  logic clk,
  input  logic rst_n,
 
  input  logic [BLOCK_BITS-1:0] blk_in,
  input  logic [4:0]            lane_in,
  input  logic                  blk_valid,
 
  output logic [BLOCK_BITS-1:0] blk_out,
  output logic [4:0]            lane_out,
  output logic                  blk_out_valid,
  output logic                  marker_inserted,
 
  // Per-lane block counters, so the period is enforced per lane rather
  // than globally -- twenty lanes each need their own phase.
  output logic [CNT_W-1:0]      c_markers,
  output logic [4:0]            last_marker_lane
);
 
  logic [13:0] count_q [PCS_LANES];
  logic [7:0]  bip_q   [PCS_LANES];
 
  integer i;
 
  // A lane's marker carries its number. Real markers are 66-bit
  // patterns with a defined structure and a BIP field; the identity is
  // what matters here.
  function automatic logic [BLOCK_BITS-1:0] make_marker
    (input logic [4:0] lane, input logic [7:0] bip);
    make_marker = {2'b10, 8'h4B, 43'd0, bip, lane};
  endfunction
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      blk_out <= '0; lane_out <= 5'd0; blk_out_valid <= 1'b0;
      marker_inserted <= 1'b0; c_markers <= '0; last_marker_lane <= 5'd0;
      for (i = 0; i < PCS_LANES; i = i + 1) begin
        count_q[i] <= 14'd0;
        bip_q[i]   <= 8'd0;
      end
    end else begin
      blk_out_valid   <= 1'b0;
      marker_inserted <= 1'b0;
 
      if (blk_valid) begin
        lane_out      <= lane_in;
        blk_out_valid <= 1'b1;
 
        if (count_q[lane_in] == 14'(AM_PERIOD - 1)) begin
          // MARKER. It REPLACES nothing -- it is inserted, so the
          // stream gains a block. Which is why the line rate carries a
          // 1/16384 overhead the payload rate does not.
          blk_out          <= make_marker(lane_in, bip_q[lane_in]);
          marker_inserted  <= 1'b1;
          last_marker_lane <= lane_in;
          count_q[lane_in] <= 14'd0;
          // The BIP window closes with the marker that reports it.
          bip_q[lane_in]   <= 8'd0;
          if (!(&c_markers)) c_markers <= c_markers + 1'b1;
        end else begin
          blk_out          <= blk_in;
          count_q[lane_in] <= count_q[lane_in] + 14'd1;
          // Interleaved parity across the blocks since the last marker.
          bip_q[lane_in]   <= bip_q[lane_in] ^ blk_in[7:0];
        end
      end
    end
  end
 
endmodule

Classification: synthesizable.

What it teaches: that the marker carries identity, timing and integrity in one object, and each of the three is unavailable any other way. Identity, because twenty lanes arriving in an unknown order are otherwise anonymous. Timing, because all twenty markers were inserted at the same deal position, so their arrival times at the receiver are a direct measurement of skew. Integrity, because the BIP gives a per-lane error rate, and Chapter 9.4 §10 established that a frame-level counter can never attribute an error to a lane.

Deliberately simplified: the marker's payload is a placeholder. The real encoding is a defined 66-bit pattern per lane with a specified BIP-3/BIP-7 structure; the identity field is the part this module is about.

Production implication: the period is enforced per lane with twenty counters, not once globally. A single global counter would put every lane's marker at a different deal position, destroying the property that makes deskew possible — the markers would no longer be a common reference edge, and their arrival times would measure the counter's phase rather than the link's skew.

7. The Marker Is the Only Name a Lane Has

Three facts follow from the markers being inserted at the same deal position, and they are the whole receive design.

First, a lane's identity is in the stream, not in the wire. A receiver does not know which fibre carries lane 7; it discovers it by reading a marker. Which means lanes may be reordered arbitrarily between the ends — swapped fibres, a differently wired connector, a mux that permutes — and the link still works, because the mapping is recovered rather than assumed.

Second, the markers are a common reference edge. They left together. So the difference in their arrival times is the skew, exactly — not a proxy for it, not an estimate. A receiver that finds all twenty markers has measured the link's skew as a by-product of locking to it.

Third, alignment must precede everything. Blocks cannot be re-interleaved into the original order until every lane's identity and offset are known, so the receive PCS has a hard ordering: find markers → identify lanes → measure offsets → buffer → re-interleave → descramble → decode. A design that starts decoding before alignment completes decodes blocks in the wrong order, which the CRC catches and nothing explains.

8. RTL 3 — Buffering Each Lane Until the Slowest Catches Up

// SYNTHESIZABLE.
//
// Deskew: hold each lane until every lane's marker has arrived, then
// release all twenty in step.
//
// The FIFO depth is not a guess. It is computed from the standard:
//
//   receive PCS Skew tolerance     : 180 ns
//   per-PCS-lane line rate         : 5.15625 Gb/s
//   bits in 180 ns                 : 180e-9 x 5.15625e9 = 928.125
//   blocks of 66 bits              : 928.125 / 66 = 14.06
//   so a lane can be 15 BLOCKS behind the earliest and still be legal
//
//   Skew Variation tolerance       : 4 ns
//   bits in 4 ns                   : 20.625  -> 0.3125 blocks
//   which is why the alignment, once found, does not need to be
//   continuously re-measured: it moves by less than a third of a block.
//
// The depth is therefore ~16 blocks per lane, and a design that guesses
// 8 works on a bench and fails on a long fibre plant.
module lane_deskew_fifo
  import lane_pkg::*;
#(
  parameter int unsigned DEPTH = 16,
  parameter int unsigned DW    = 5,        // $clog2(DEPTH)+1
  parameter int unsigned CNT_W = 16
) (
  input  logic clk,
  input  logic rst_n,
  input  logic clear,
 
  input  logic [PCS_LANES-1:0]                 lane_valid,
  input  logic [PCS_LANES-1:0][BLOCK_BITS-1:0] lane_blk,
  input  logic [PCS_LANES-1:0]                 lane_marker,
 
  output logic                                 align_status,
  output logic [PCS_LANES-1:0][BLOCK_BITS-1:0] aligned_blk,
  output logic                                 aligned_valid,
 
  // Per-lane occupancy at the moment alignment completed. This is the
  // measured skew, in blocks, and it is the number a field engineer
  // needs and nothing else in the design produces.
  output logic [PCS_LANES-1:0][DW-1:0] lane_depth_at_align,
  output logic [DW-1:0]                worst_depth,
 
  // A lane arrived further behind than the buffer can hold. REPORTED,
  // and alignment is refused -- because aligning nineteen lanes and
  // guessing the twentieth produces a stream that decodes.
  output logic                         skew_exceeded,
  output logic [4:0]                   skew_exceeded_lane,
 
  output logic [CNT_W-1:0]             c_alignments,
  output logic                         ever_skew_exceeded
);
 
  logic [DW-1:0]           depth_q [PCS_LANES];
  logic [PCS_LANES-1:0]    seen_q;
  logic                    hunting_q;
 
  integer i;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      align_status <= 1'b0; aligned_valid <= 1'b0; worst_depth <= '0;
      skew_exceeded <= 1'b0; skew_exceeded_lane <= 5'd0;
      c_alignments <= '0; ever_skew_exceeded <= 1'b0;
      seen_q <= '0; hunting_q <= 1'b1;
      for (i = 0; i < PCS_LANES; i = i + 1) begin
        depth_q[i]             <= '0;
        lane_depth_at_align[i] <= '0;
      end
    end else if (clear) begin
      align_status <= 1'b0; seen_q <= '0; hunting_q <= 1'b1;
      skew_exceeded <= 1'b0;
      for (i = 0; i < PCS_LANES; i = i + 1) depth_q[i] <= '0;
      // ever_skew_exceeded is NOT cleared. A plant that has once
      // exceeded the budget is a plant, and clearing a flag does not
      // shorten a fibre.
    end else begin
      aligned_valid <= 1'b0;
      skew_exceeded <= 1'b0;
 
      if (hunting_q) begin
        for (i = 0; i < PCS_LANES; i = i + 1) begin
          if (lane_valid[i]) begin
            if (lane_marker[i] && !seen_q[i]) begin
              // This lane's reference edge. Its current depth IS how
              // far behind the earliest lane it is.
              seen_q[i]              <= 1'b1;
              lane_depth_at_align[i] <= depth_q[i];
              if (depth_q[i] > worst_depth) worst_depth <= depth_q[i];
            end else if (|seen_q) begin
              // At least one lane has shown its marker, so this lane is
              // late and its arrivals are accumulating.
              if (depth_q[i] == DW'(DEPTH - 1)) begin
                // OUT OF RANGE. Refuse alignment; do not wrap the FIFO
                // and align on stale data, which produces a stream that
                // decodes into the wrong order.
                skew_exceeded      <= 1'b1;
                skew_exceeded_lane <= 5'(i);
                ever_skew_exceeded <= 1'b1;
                hunting_q          <= 1'b1;
                seen_q             <= '0;
              end else begin
                depth_q[i] <= depth_q[i] + 1'b1;
              end
            end
          end
        end
 
        if (&seen_q) begin
          // ALL TWENTY. Nineteen is not alignment.
          hunting_q    <= 1'b0;
          align_status <= 1'b1;
          if (!(&c_alignments)) c_alignments <= c_alignments + 1'b1;
        end
      end else begin
        aligned_blk   <= lane_blk;
        aligned_valid <= &lane_valid;
      end
    end
  end
 
endmodule

Classification: synthesizable.

What it teaches: that lane_depth_at_align is the measured skew and is the module's most valuable output. Every other signal says whether the link works. This one says by how much it is working — a plant with a worst depth of 3 blocks has enormous margin, and one at 14 is a single connector change from failing, and both report align_status high.

Deliberately simplified: one block per lane per cycle, and the marker hunt restarts from scratch on any excess. Real implementations hunt continuously and re-verify with a second marker 16384 blocks later, which is what the standard's alignment-marker lock process requires.

Production implication: skew_exceeded refuses alignment rather than wrapping the FIFO. A buffer that wraps aligns nineteen lanes correctly and one on stale data — and the result is a perfectly locked link delivering blocks in the wrong order, which fails CRC on every frame and reports align_status high. Refusing is legible; wrapping is a silent reordering.

9. The Skew Budget, Computed

The receive PCS must tolerate up to 180 ns of skew between lanes and up to 4 ns of skew variation. Turn both into design quantities.

And 180 ns is a large amount of physical difference. At a fibre velocity factor of about 0.681,

180 × 10⁻⁹ × 0.681 × 3 × 10⁸ ≈ 36.8 m

— nearly 37 metres of differential path length between the lanes of one link. Which is not a pathological case: it is a ribbon whose fibres were cut on different reels, a patch panel, and two lanes that took different routes through a building.

The variation number is the one that decides the architecture. Total skew is 14 blocks and must be buffered; skew variation is 0.3125 blocks — less than a third of one block. So the alignment, once established, essentially does not move, and the receiver can find it once and hold it rather than tracking it continuously.

10. RTL 4 — Twenty Lanes, Twenty Distinct Names

// SYNTHESIZABLE.
//
// Validates the lane mapping the markers describe, before the deskew
// buffer is allowed to trust it.
//
// The mapping is recovered from data, so it can be WRONG in ways that
// a physically wired mapping cannot:
//
//   DUPLICATE  -- two lanes claiming the same number. A mux fault, a
//        looped fibre, or a marker corrupted into another lane's value.
//        The reassembly would take two copies of one position and none
//        of another.
//   MISSING    -- a number nobody claims. Nineteen lanes present is not
//        95% of a link: the deal has a hole in it.
//   OUT OF RANGE -- a lane number of 20 or more, from a corrupted
//        marker.
//
// Note what these have in common with Chapter 9.4's lane faults: the
// stream stays STRUCTURALLY legal. Every block parses. Only the
// reassembly is wrong.
module lane_mapping_validator
  import lane_pkg::*;
#(
  parameter int unsigned CNT_W = 16
) (
  input  logic clk,
  input  logic rst_n,
  input  logic clear,
 
  // One entry per physical position; the lane number its marker claims.
  input  logic [PCS_LANES-1:0]       claim_valid,
  input  logic [PCS_LANES-1:0][4:0]  claimed_lane,
  input  logic                       evaluate,
 
  output logic                       mapping_ok,
  output logic                       duplicate_seen,
  output logic                       missing_seen,
  output logic                       out_of_range_seen,
  output logic [PCS_LANES-1:0]       claimed_mask,
  output logic [4:0]                 first_duplicate_lane,
 
  output logic [CNT_W-1:0]           c_evaluations,
  output logic [CNT_W-1:0]           c_rejections,
  output logic                       ever_rejected
);
 
  logic [PCS_LANES-1:0] mask_c;
  logic dup_c, oor_c;
  logic [4:0] dup_lane_c;
 
  always_comb begin
    mask_c     = '0;
    dup_c      = 1'b0;
    oor_c      = 1'b0;
    dup_lane_c = 5'd0;
 
    for (int p = 0; p < PCS_LANES; p = p + 1) begin
      if (claim_valid[p]) begin
        if (claimed_lane[p] >= 5'(PCS_LANES)) begin
          oor_c = 1'b1;
        end else if (mask_c[claimed_lane[p]]) begin
          // Two positions claiming the same lane number. Record the
          // FIRST, because a corrupted mux produces many and only the
          // first names the collision that started it.
          if (!dup_c) begin
            dup_c      = 1'b1;
            dup_lane_c = claimed_lane[p];
          end
        end else begin
          mask_c[claimed_lane[p]] = 1'b1;
        end
      end
    end
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      mapping_ok <= 1'b0; duplicate_seen <= 1'b0; missing_seen <= 1'b0;
      out_of_range_seen <= 1'b0; claimed_mask <= '0;
      first_duplicate_lane <= 5'd0; c_evaluations <= '0;
      c_rejections <= '0; ever_rejected <= 1'b0;
    end else if (clear) begin
      mapping_ok <= 1'b0; duplicate_seen <= 1'b0; missing_seen <= 1'b0;
      out_of_range_seen <= 1'b0; c_evaluations <= '0; c_rejections <= '0;
      // ever_rejected survives: a link that has ever presented a broken
      // mapping is a link with a mux or a fibre plant worth suspecting.
    end else if (evaluate) begin
      claimed_mask         <= mask_c;
      duplicate_seen       <= dup_c;
      out_of_range_seen    <= oor_c;
      missing_seen         <= !(&mask_c);
      first_duplicate_lane <= dup_lane_c;
 
      // ALL THREE must be clean. A mapping with a hole is not a partial
      // mapping, it is a wrong one -- the missing lane's blocks would
      // be filled from whatever the reassembly finds there.
      mapping_ok <= (&mask_c) && !dup_c && !oor_c;
 
      if (!(&c_evaluations)) c_evaluations <= c_evaluations + 1'b1;
      if (!((&mask_c) && !dup_c && !oor_c)) begin
        if (!(&c_rejections)) c_rejections <= c_rejections + 1'b1;
        ever_rejected <= 1'b1;
      end
    end
  end
 
endmodule

Classification: synthesizable.

What it teaches: that &mask_c — every one of the twenty numbers claimed exactly once — is the only acceptable mapping, and the three failure modes it rejects are distinguishable and worth distinguishing. A duplicate says a mux or a fibre is looped; a missing lane says one is dark; out of range says a marker was corrupted. Three different work orders behind one mapping_ok low.

Deliberately simplified: claims are evaluated as a single snapshot. A real receiver accumulates claims as markers are found over many blocks, and the evaluation happens when the last lane reports.

Production implication: the validator sits before the deskew buffer is allowed to act, because deskew on a bad mapping is worse than no deskew. A buffer that aligns to a mapping with a duplicate will take two copies of one block position and none of another, producing a stream that is perfectly aligned, perfectly framed, and wrong — the same failure shape as Chapter 9.4's lane swap, arrived at by a completely different route.

11. RTL 5 — Per-Lane Health, Because the Frame Layer Cannot Attribute

// SYNTHESIZABLE.
//
// Per-lane health from the BIP carried in each lane's marker.
//
// WHY IT MUST BE PER LANE: once the twenty lanes are re-interleaved
// into one stream, error attribution is gone. An FCS failure names the
// frame, not the lane. So ONE degrading lane and TWENTY mildly degraded
// lanes produce the same symptom above the re-interleaver, and the
// remedies are completely different -- replace a fibre, or replace the
// whole plant.
//
// This is Chapter 9.3's 8B1Q4 argument and Chapter 9.4's XGMII argument
// arriving a third time, for a third reason: attribution exists below a
// combining step and nowhere above it.
module multilane_health_monitor
  import lane_pkg::*;
#(
  parameter int unsigned CNT_W = 16,
  // A lane whose BIP error count in one window exceeds this is
  // degrading rather than occasionally unlucky.
  parameter int unsigned DEGRADE_THRESHOLD = 16'd8
) (
  input  logic clk,
  input  logic rst_n,
  input  logic clear,
 
  input  logic [PCS_LANES-1:0] bip_check_valid,
  input  logic [PCS_LANES-1:0] bip_mismatch,
  input  logic                 window_close,
 
  output logic [PCS_LANES-1:0][CNT_W-1:0] c_lane_bip_errors,
  output logic [PCS_LANES-1:0]            lane_degraded,
  output logic [4:0]                      worst_lane,
  // One lane far worse than the rest is a fibre or a connector; twenty
  // equally poor lanes is the plant, the reach, or the transceivers.
  output logic                            single_lane_outlier,
  output logic                            window_valid,
  output logic [PCS_LANES-1:0]            ever_degraded
);
 
  logic [CNT_W-1:0] win_q [PCS_LANES];
  integer i;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      window_valid <= 1'b0; lane_degraded <= '0; ever_degraded <= '0;
      for (i = 0; i < PCS_LANES; i = i + 1) begin
        win_q[i]             <= '0;
        c_lane_bip_errors[i] <= '0;
      end
    end else if (clear) begin
      window_valid <= 1'b0; lane_degraded <= '0;
      for (i = 0; i < PCS_LANES; i = i + 1) begin
        win_q[i]             <= '0;
        c_lane_bip_errors[i] <= '0;
      end
      // ever_degraded survives clear.
    end else begin
      for (i = 0; i < PCS_LANES; i = i + 1) begin
        if (bip_check_valid[i] && bip_mismatch[i]) begin
          if (!(&win_q[i]))             win_q[i]             <= win_q[i] + 1'b1;
          if (!(&c_lane_bip_errors[i])) c_lane_bip_errors[i] <= c_lane_bip_errors[i] + 1'b1;
        end
      end
 
      if (window_close) begin
        window_valid <= 1'b1;
        for (i = 0; i < PCS_LANES; i = i + 1) begin
          if (win_q[i] > CNT_W'(DEGRADE_THRESHOLD)) begin
            lane_degraded[i] <= 1'b1;
            ever_degraded[i] <= 1'b1;
          end else begin
            lane_degraded[i] <= 1'b0;
          end
          win_q[i] <= '0;
        end
      end
    end
  end
 
  // Worst lane, and whether it stands out from the rest.
  always_comb begin
    logic [CNT_W-1:0] hi, lo;
    logic [4:0] hi_idx;
    hi = '0; lo = '1; hi_idx = 5'd0;
    for (int p = 0; p < PCS_LANES; p = p + 1) begin
      if (c_lane_bip_errors[p] > hi) begin
        hi = c_lane_bip_errors[p]; hi_idx = 5'(p);
      end
      if (c_lane_bip_errors[p] < lo) lo = c_lane_bip_errors[p];
    end
    worst_lane = hi_idx;
    // A factor of four between best and worst is a difference twenty
    // nominally identical lanes do not produce by chance.
    single_lane_outlier = window_valid && (hi > (lo << 2));
  end
 
endmodule

Classification: synthesizable.

What it teaches: that the BIP is not redundant with the CRC — it is the only measurement on the link with a lane in it. Chapter 6.3's CRC detects errors in a frame; the BIP detects them in a lane. Once the re-interleaver has run, the lane is gone from the data forever, so the measurement has to exist below it or not at all.

Deliberately simplified: one BIP result per lane per check. The real structure uses BIP-3 and BIP-7 fields with a defined relationship to the marker, and the check is against a locally recomputed value.

Production implication: single_lane_outlier is what turns one alarm into two work orders. Twenty lanes with a similar error rate is a reach, a plant or a transceiver problem — a link-level fact. One lane four times worse than the rest is a fibre or a connector — a physical fact about one path. Both raise exactly the same FCS error rate at the frame layer, and the ratio between the best and worst lane is the only thing that separates them.

12. Per-Lane FEC and End-to-End FEC

Once a link is several lanes, error correction has a placement question that a single-lane link does not.

Per-lane FEC encodes each lane independently. Each lane carries its own parity, corrects its own errors, and a lane's failure is contained within it.

End-to-end FEC encodes the aggregate stream, after striping — so a codeword's symbols are spread across every lane.

And the second row is the argument that decided it.

Striping is interleaving. Blocks are dealt round-robin across twenty lanes, so a burst of errors confined to one lane arrives at the FEC decoder spread out — one symbol here, then nineteen symbols' worth of clean data, then the next. A burst in time becomes scattered single-symbol errors in the codeword, which is precisely the error pattern a Reed–Solomon code is strongest against.

So end-to-end FEC gets burst protection for free from a mechanism that exists for a completely different reason. The striping was designed to make lanes reconstructible; it happens to be an interleaver, and the FEC placement decision follows from that accident.

The cost is the third row. A lane that fails completely removes its share of every codeword rather than one codeword entirely — and whether that is survivable depends on how much of a codeword one lane carries.

13. Properties Worth Asserting, and One Worth Refusing

The organising split here is the same one Chapter 9.4 drew and applied to a new axis. Striping and marker insertion are things this design produces, and their rules are assertable unconditionally. Skew, lane order and lane health are things the medium delivers, and the only assertable properties about them are properties of the response.

Striping and markers — what this design produces

// P1. The deal index never leaves the lane range. A five-bit counter
// free-running to 31 stripes onto lanes that do not exist and still
// looks periodic.
property p_deal_index_in_range;
  @(posedge clk) disable iff (!rst_n)
  deal_index < 5'(PCS_LANES);
endproperty
a_deal_index_in_range: assert property (p_deal_index_in_range);
 
// P2. Round-robin: consecutive blocks go to consecutive lanes, modulo
// the lane count. The whole mapping, in one line.
property p_round_robin;
  @(posedge clk) disable iff (!rst_n)
  lane_valid |=> (deal_index == (($past(deal_index) == 5'(PCS_LANES-1))
                                   ? 5'd0 : $past(deal_index) + 5'd1));
endproperty
a_round_robin: assert property (p_round_robin);
 
// P3. Every accepted input block produces exactly one output block on
// exactly one lane. Striping neither drops nor duplicates.
property p_one_block_one_lane;
  @(posedge clk) disable iff (!rst_n)
  blk_valid |=> lane_valid;
endproperty
a_one_block_one_lane: assert property (p_one_block_one_lane);
 
// P4. Markers are inserted at exactly the period, per lane. A global
// counter would put each lane's marker at a different deal position and
// destroy the common reference edge deskew depends on.
property p_marker_period;
  @(posedge clk) disable iff (!rst_n)
  marker_inserted |-> ($past(count_q[lane_in]) == 14'(AM_PERIOD - 1));
endproperty
a_marker_period: assert property (p_marker_period);
 
// P5. A marker's identity field is the lane it was inserted on. The
// property that makes the receiver's reconstruction possible.
property p_marker_names_its_lane;
  @(posedge clk) disable iff (!rst_n)
  marker_inserted |-> (blk_out[4:0] == lane_out);
endproperty
a_marker_names_its_lane: assert property (p_marker_names_its_lane);

Mapping — every lane claimed exactly once

// P6. A mapping is accepted only when all twenty numbers are claimed.
// Nineteen lanes is not 95% of a link.
property p_mapping_needs_all_lanes;
  @(posedge clk) disable iff (!rst_n)
  mapping_ok |-> (&claimed_mask);
endproperty
a_mapping_needs_all: assert property (p_mapping_needs_all_lanes);
 
// P7. And only when no number is claimed twice.
property p_mapping_no_duplicates;
  @(posedge clk) disable iff (!rst_n)
  mapping_ok |-> !duplicate_seen;
endproperty
a_mapping_no_dup: assert property (p_mapping_no_duplicates);
 
// P8. Each rejection reason is reported. mapping_ok low with no reason
// is a design that refused and cannot say why.
property p_rejection_has_reason;
  @(posedge clk) disable iff (!rst_n)
  ($past(evaluate) && !mapping_ok)
    |-> (duplicate_seen || missing_seen || out_of_range_seen);
endproperty
a_rejection_has_reason: assert property (p_rejection_has_reason);
 
// P9. Rejection history survives clear -- a plant that has presented a
// broken mapping stays suspect.
property p_rejection_sticky;
  @(posedge clk) disable iff (!rst_n)
  ever_rejected |=> ever_rejected;
endproperty
a_rejection_sticky: assert property (p_rejection_sticky);

Deskew — properties of the RESPONSE, not of the medium

// P10. THE ORDERING PROPERTY. Alignment requires every lane's marker.
// Nineteen found and one guessed produces a link that decodes into the
// wrong order and reports itself healthy.
property p_align_needs_all_markers;
  @(posedge clk) disable iff (!rst_n)
  $rose(align_status) |-> $past(&seen_q);
endproperty
a_align_needs_all_markers: assert property (p_align_needs_all_markers);
 
// P11. THE BOUNDARY PROPERTY, and the one the rejected property below
// displaces. If any lane exceeds the buffer, alignment must NOT be
// declared. This is the assertion about behaviour at the edge of the
// operating envelope.
property p_excess_refuses_alignment;
  @(posedge clk) disable iff (!rst_n)
  skew_exceeded |=> !align_status;
endproperty
a_excess_refuses_alignment: assert property (p_excess_refuses_alignment);
 
// P12. And the excess names the lane, so the report is actionable.
property p_excess_names_lane;
  @(posedge clk) disable iff (!rst_n)
  skew_exceeded |-> (skew_exceeded_lane < 5'(PCS_LANES));
endproperty
a_excess_names_lane: assert property (p_excess_names_lane);
 
// P13. No lane's recorded depth ever exceeds the buffer. A depth beyond
// DEPTH means the FIFO wrapped and aligned on stale data.
property p_depth_within_buffer;
  @(posedge clk) disable iff (!rst_n)
  align_status |-> (worst_depth < DW'(DEPTH));
endproperty
a_depth_within_buffer: assert property (p_depth_within_buffer);
 
// P14. Once aligned, the mapping is stable. Skew Variation is 0.3125
// blocks, so an alignment that moves is a bug, not a channel.
property p_alignment_stable;
  @(posedge clk) disable iff (!rst_n)
  (align_status && !clear) |=> $stable(lane_depth_at_align);
endproperty
a_alignment_stable: assert property (p_alignment_stable);
 
// P15. Deskew never runs on a rejected mapping. Aligning to a mapping
// with a duplicate produces a perfectly framed, perfectly wrong stream.
property p_no_deskew_without_mapping;
  @(posedge clk) disable iff (!rst_n)
  align_status |-> mapping_ok;
endproperty
a_no_deskew_without_mapping: assert property (p_no_deskew_without_mapping);

Per-lane health

// P16. A degraded lane is always named -- the flag and the identity
// arrive together, because the flag alone is an FCS counter with extra
// steps.
property p_degrade_names_lane;
  @(posedge clk) disable iff (!rst_n)
  (|lane_degraded) |-> (worst_lane < 5'(PCS_LANES));
endproperty
a_degrade_names_lane: assert property (p_degrade_names_lane);
 
// P17. Degradation history survives clear, because the fibre did not
// change when somebody read a counter.
property p_degrade_history_sticky;
  @(posedge clk) disable iff (!rst_n)
  (|ever_degraded) |=> (|ever_degraded);
endproperty
a_degrade_history_sticky: assert property (p_degrade_history_sticky);

14. Verification Scenarios

Group by which of the three mechanisms is being broken, because striping, mapping and deskew fail in completely different ways and only one of them is visible above the re-interleaver.

Striping and marker insertion

1. A block on every cycle for a full lane cycle — lanes 0 through 19 in order, then back to 0.
2. Gapped input — the deal index holds across the gap; a bubble does not advance the deal.
3. Exactly AM_PERIOD blocks on one lane — one marker, on that lane, at that count and no other.
4. All twenty lanes reaching their period — twenty markers, one per lane, each carrying its own number.
5. Marker identity check across all twenty — every lane's marker names that lane. P5 across its whole range.
6. The BIP window — parity accumulates between markers and resets with the marker that reports it.
7. Deal index at the wrap boundary — 19 → 0, never 19 → 20. The five-bit free-run bug, tested directly.

Mapping validation

8. All twenty lanes claiming distinct numbers — mapping_ok, no flags.
9. Two positions claiming lane 7 — duplicate_seen, first_duplicate_lane = 7, mapping_ok low.
10. Nineteen lanes claiming, one silent — missing_seen, mapping_ok low. Nineteen is not 95% of a link.
11. A marker claiming lane 24 — out_of_range_seen.
12. Duplicate and missing together — both flags; one broken lane produces both, and reporting only one sends the wrong work order.
13. A fully permuted mapping — lanes arriving in reverse order, all twenty distinct: mapping_ok high. A swap is supported, not a fault.
14. clear after a rejection — counters clear, ever_rejected survives.

Deskew

15. Zero skew — every lane's depth 0, worst_depth 0, immediate alignment.
16. One lane 14 blocks late — the standard's worst conforming case: aligns, worst_depth = 14, skew_exceeded low.
17. One lane DEPTH − 1 blocks late — the boundary that must still align.
18. One lane DEPTH blocks late — skew_exceeded high, skew_exceeded_lane naming it, align_status low. The boundary that must fail, and must not wrap.
19. Nineteen markers found, the twentieth never arriving — alignment never declared; the hunt does not complete on a majority.
20. Markers arriving in a different order on a re-lock — alignment is reached regardless; nothing assumes lane 0 arrives first.
21. lane_depth_at_align after alignment — never changes while aligned. P14, which the 0.3125-block skew variation is what licenses.
22. clear mid-hunt — the hunt restarts cleanly, ever_skew_exceeded survives.
23. Alignment attempted with mapping_ok low — refused. P15.

Per-lane health

24. BIP mismatches on lane 11 only — lane_degraded[11], worst_lane = 11, single_lane_outlier high.
25. BIP mismatches spread evenly over all twenty — single_lane_outlier low, with the same aggregate error rate as scenario 24. Identical frame-layer symptom, opposite diagnosis.
26. Errors just under the degrade threshold — no flag; a lane that is occasionally unlucky is not a degrading lane.
27. clear after a degradation — window counters clear, ever_degraded survives.

15. Debugging: Which Layer Owns the Symptom

A multi-lane link has three places a fault can hide, and only one of them is visible from the frame layer.

Three habits.

First, read worst_depth on a healthy link, and write it down. It is the link's margin, and it is the only number that distinguishes a plant with 3 blocks of skew from one with 14 — both of which report align_status high and identical throughput. A link at 14 fails on the next patch-panel change; a link at 3 does not.

Second, treat a small measured skew with a high error rate as an aliasing signature, not as a contradiction. A wrapped buffer reports a plausible number, because DEPTH + 2 blocks of delay looks like 2. The most dangerous wrong answer this subsystem produces is a reassuring one.

Third, compare the best and worst lane BIP before touching any fibre. A ratio near 1 means the whole plant; a ratio of four or more means one path. The aggregate error rate is identical in both cases, so the ratio is the entire diagnosis.

16. Common Misconceptions

"A multi-lane link is a wide bus."

The wrong model: twenty lanes are twenty wires of one parallel interface.

What it costs: you cannot explain why alignment markers exist, why lanes can be swapped freely, or why the receive PCS has a hard ordering before it can decode anything.

The corrected model: they are twenty independent serial links that a protocol re-assembles. Nothing makes them arrive together — the standard tolerates 180 ns between them, which at a fibre velocity factor of 0.681 is nearly 37 metres of differential path. A bus's lanes are aligned by construction; these are aligned by mechanism.

"Alignment markers are framing overhead."

The wrong model: a periodic sync pattern, like a preamble.

What it costs: you miss that they carry identity and integrity as well as timing, and you cannot explain why swapped fibres are supported.

The corrected model: a marker does three jobs. Identity — a lane's number is in the marker, not in the wire, which is why a receiver can reorder arbitrarily permuted lanes. Timing — all twenty were inserted at the same deal position, so their arrival times are the skew, measured rather than estimated. Integrity — the BIP field gives a per-lane error rate that nothing above the re-interleaver can provide. And it costs 1/16384 = 0.0061%.

"Twenty PCS lanes is a capacity choice."

The wrong model: twenty was sized to carry 100 Gigabit.

What it costs: you cannot explain why 40 Gigabit uses four, or why 100GBASE-R4 and 100GBASE-R10 share a PCS.

The corrected model: it is a divisibility choice. 20 divides by 1, 2, 4, 5, 10 and 20 — the plausible physical lane counts — so every one of them gives a whole number of PCS lanes per physical lane. The count was fixed before four-lane 100G optics existed, and needed no change when they arrived, because 4 divides 20.

"Deskew is a control loop that tracks the alignment."

The wrong model: skew drifts, so the aligner follows it.

What it costs: a loop chasing a quantity that moves by a third of a block, which will eventually respond to noise and misalign a working link.

The corrected model: the two numbers in the standard say the opposite things. Skew is 180 ns — 14 blocks — large and static, so it sizes a buffer. Skew variation is 4 ns — 0.3125 blocks — tiny and dynamic, less than a third of one block, so the alignment once found does not move. Find it once, latch it, and re-hunt only on an explicit loss.

"A deskew buffer that overflows will obviously report a large skew."

The wrong model: a failure reports a big number.

What it costs: you ship a design whose worst bug produces a reassuring measurement.

The corrected model: a wrapped FIFO aliases. A lane delayed by DEPTH + 2 blocks presents as a lane delayed by 2, so the link aligns, reports a small and entirely plausible skew, and delivers every block in the wrong order. The symptom is a healthy-looking link with a 100% frame error rate — which is why Section 8 refuses alignment rather than wrapping, and why P13 asserts that no recorded depth ever exceeds the buffer.

17. Interview Reasoning

"What sets the per-lane rate on a 100 Gigabit link?"

Two things, and the strong answer separates them. The lane count — chosen from what a manufacturable serialiser and channel can carry, not from anything about the aggregate — and the line code, which multiplies by 66/64. So 100 ÷ 4 = 25, × 66/64 = 25.78125 GBd for 100GBASE-R4, and 100 ÷ 10 = 10, × 66/64 = 10.3125 GBd for 100GBASE-R10. And the second of those is exactly 10GBASE-R's per-lane rate — the finishing observation, because it means 40 and 100 Gigabit shipped first on a serialiser that already existed. The link was made wider, not faster; the per-lane rate moved a generation later.

"Why do alignment markers exist, and what would break without them?"

Because a lane has no identity in the wire — twenty streams arriving in an unknown order are anonymous, and the round-robin deal cannot be inverted without knowing which lane is which. The strong answer gives all three jobs: identity, which also makes swapped fibres a supported condition rather than a fault; timing, because all twenty markers were inserted at the same deal position so their arrival times measure the skew directly; and integrity, because the BIP is the only per-lane error measure on the link. The finishing point: they cost 1/16384, or 0.0061%, and the period is a compromise between re-lock speed and overhead.

"How deep does a 100G deskew buffer need to be, and how do you know?"

Compute it. The receive PCS must tolerate 180 ns of skew. Each PCS lane runs at 100 ÷ 20 × 66/64 = 5.15625 Gb/s, so 180 ns is 180 × 10⁻⁹ × 5.15625 × 10⁹ = 928.125 bits, which is 928.125 ÷ 66 = 14.06 blocks — so about 16 blocks per lane with margin. The strong answer adds the second number: skew variation is 4 ns, or 20.625 bits, 0.3125 of a block — which is why the alignment is latched rather than tracked. The large static number sizes a buffer; the tiny dynamic one says no control loop is needed.

"Would you assert that the lanes arrive within the deskew window?"

No, and the reason is that it asserts away the only interesting half of the input space. measured_skew is a property of the fibre plant, and this design is the thing built to absorb it — so the property says the input to a tolerance mechanism is within tolerance. The design has a defined behaviour outside that bound: refuse alignment and name the lane. Asserting the bound makes the environment that reaches that behaviour unreachable, so the boundary handling is never exercised, and a 37-metre path mismatch is one patch panel away. It also passes forever, because its greenness is a restatement of the FIFO's depth parameter. Assert the response instead: an excess refuses alignment, names the lane, and no recorded depth ever exceeds the buffer — the last of which is what catches a FIFO that wrapped and aliased a large skew into a small one.

18. Understanding Check

19. What's Next

The claim this chapter defended: lane aggregation replaces a physical guarantee with a protocol.

XGMII's byte lanes arrive together because they are traces on one board. A 100 Gigabit link's lanes traverse different fibres of different lengths and arrive up to 180 ns apart — nearly 37 metres of differential path — so order is re-established from information the data carries rather than from the wiring. Blocks are dealt round-robin, because that is the only mapping a receiver can invert from a lane's own name. Each lane inserts an alignment marker every 16384 blocks, at a cost of 0.0061%, and that one object carries identity, timing and integrity at once.

The two skew numbers then decide the architecture: 180 ns is 14 blocks and sizes a buffer; 4 ns of variation is 0.3125 blocks and says the alignment is latched, not tracked. And the PCS lane count is 20 because it divides by every physical lane count a medium might offer — a constant chosen to fit media nobody had yet proposed, and vindicated when four-lane 100 Gigabit arrived and needed no PCS change at all.

And the design's obligation is not that the lanes arrive close enough. It is to say, exactly, when they did not — which is why the rejected property here asserts the operating envelope, and the replacement asserts the behaviour at its boundary.

Chapter 9.6 — 200G to 800G: PAM4, FEC and the Modern Data-Centre Link takes the generation in which the error correction stops being an option.

PAM4 carries two bits per symbol, so it halves the baud rate for a given bit rate — and it costs about 9.5 dB of signal-to-noise ratio, because three eyes share the amplitude one eye used to have. At that penalty the raw link no longer meets Ethernet's error-rate expectation on its own, so FEC becomes mandatory rather than optional, and the encoding, the lane rate and the latency budget are all decided together. Chapter 9.6 computes the SNR penalty, the codeword arithmetic, and exactly what the FEC adds to Chapter 8.4's decomposition — plus why asserting a post-FEC error rate is the wrong property to write.

The full path is on the Ethernet curriculum index.