Ethernet · Module 3
Link Establishment
A link climbs a ladder of six gates, each attemptable only once the one below it has succeeded. Every gate is a search whose duration depends on the evidence available to it, which is why bring-up time is a distribution and why the only useful question is which gate it stalled at.
Every chapter in Module 3 built a piece of what happens when a link comes up, and each one stopped at its own boundary.
Chapter 3.2 gave signal detect. Chapter 2.6 gave clock recovery and block synchronisation. Chapter 3.4 built the aligner that searches for a block boundary, and deferred lane deskew. Chapter 3.5 explained where the aligner's evidence comes from and — this is the part that matters here — quantified how strong that evidence is. Chapter 3.7 added a codeword structure that must itself be located.
Assembling them is not merely a matter of listing them in order. The interesting structure is why the order is forced, and why the duration of each stage is so different.
The order is forced because each stage can only be attempted once the one below it has succeeded. You cannot recover a clock from a signal that is not there. You cannot find a block boundary in a bit stream whose bit boundaries are unknown. You cannot deskew lanes that have not individually locked. Each stage's precondition is the previous stage's result, and the ladder has no shortcuts.
And every stage is a discovery. The transmitter knows where everything is; the receiver must find out. Chapter 2.2 named that asymmetry for frames and Chapter 3.4 for blocks — here it applies to the entire link, and it has a consequence nobody expects: bring-up time is a sum of search times, and each search takes as long as its evidence is weak.
Which gate does a stalled link stop at, why does that gate take the time it does, and what does the answer tell you to do next?
1. Scope — What This Chapter Owns
This chapter owns: the ladder of assumptions and why it cannot be reordered; alignment markers and what they are for; lane deskew including how skew is measured rather than merely absorbed; auto-negotiation at the depth an RTL engineer needs; the bring-up sequencer as a state machine with real fallback paths; and the timeline instrumentation that makes "which gate, for how long" answerable.
This chapter does not own: the mechanisms it sequences. Signal detect is Chapter 3.2's, clock recovery and elastic buffering are Chapter 2.6's, the block aligner is Chapter 3.4's, the invalid-pattern evidence is Chapter 3.5's, and FEC is Chapter 3.7's. This chapter cites them rather than restating them — its subject is the sequence, the dependencies, and the diagnosis.
It also deliberately does not encode any auto-negotiation page format. The link codeword structure is clause-specific and reproducing a partial version would look complete while being wrong. Section 8 models the negotiation state machine — propose, detect agreement, commit — which is what an integrator designs against and where the bugs live.
The debts it repays: Chapter 3.1 §7 deferred auto-negotiation. Chapter 3.1 §6, Chapter 3.2 §8 and Chapter 3.4 §10 all deferred lane deskew. Chapter 3.4 §13 built a status vector and left the sequence it summarises to here.
2. The Ladder, and Why It Cannot Be Reordered
Take each dependency in turn, because the reason each is forced is different.
Clock lock needs signal. Chapter 2.6 established that no clock is transmitted — timing is extracted from transitions in the data. No signal, no transitions, nothing to lock to. This one is physical.
Block lock needs clock. Chapter 3.4's aligner slices a bit stream at candidate offsets, which presupposes that bit boundaries are known. Without recovered timing there are no bits, only a waveform. This one is definitional.
Marker lock needs block lock. An alignment marker is a block with a recognisable content, so recognising one requires already being able to parse blocks. This one is structural.
Deskew needs marker lock on every lane. Skew is measured by comparing when the same marker arrives on different lanes, so each lane must first be able to recognise markers. This one is a measurement dependency, and Section 7 makes it concrete.
Codeword lock needs deskew. A FEC codeword spans lanes, so its symbols are only in the right order once the lanes are aligned. This one follows from Chapter 3.7's codeword being a structure over an ordered symbol sequence.
Six gates, five dependencies, five different kinds of reason — physical, definitional, structural, measurement, and structural again. That variety is why the ladder is worth understanding rather than memorising: each dependency fails differently, and Section 15's diagnosis uses that.
3. Alignment Markers — What They Actually Solve
A multi-lane PCS distributes blocks across lanes in a fixed rotation. The receiver must undo that, and to undo it, it must know which physical lane carries which position in the rotation.
That is not knowable from the data. Every lane carries valid, legal, well-formed blocks. Nothing in a block says which lane it came from, and the lanes may arrive in any physical order — a cable with its fibres in a different sequence, a connector wired differently, a board routed for convenience.
So the transmitter periodically inserts a marker into each lane, and the marker identifies the lane. The receiver finds markers, reads the identity, and now knows the mapping from physical lane to logical position.
The markers solve a second problem at the same time, and it is the one Section 7 depends on: because the same marker is inserted into every lane at the same moment, the difference in arrival times of that marker across lanes is the skew. The marker is simultaneously an identity label and a timing reference.
On the marker's contents, this chapter is deliberately unspecific. The exact structure is clause-specific, and a partial reproduction would look authoritative while being wrong. What is common across families and what matters architecturally is that a marker is recognisable, lane-identifying, and periodic — and Section 6's RTL models exactly those three properties.
4. Auto-Negotiation — Agreeing What to Try
Everything above assumes the two ends are attempting the same thing. Auto-negotiation is how they come to agree, and it happens before the ladder rather than as part of it.
The problem it solves: a port may support several speeds, several duplex modes, and several optional features. The far end supports its own set. Trying the wrong combination fails at some gate on the ladder — often at clock lock, with a perfectly healthy channel — and the failure is indistinguishable from a hardware fault.
The shape of the solution is the same in every family that has it, even though the encodings differ completely:
- Advertise. Each end transmits what it can do, in a form the other can receive without having agreed on anything yet. This is the hard part: the advertisement must be readable at a common denominator both ends support unconditionally.
- Receive and acknowledge. Each end reads the other's advertisement and signals that it did.
- Resolve. Both ends independently apply the same priority rule to the intersection of the two capability sets. Because the rule is deterministic and both see the same intersection, both reach the same answer without further exchange.
- Commit. Both reconfigure and begin the ladder.
Step 3 is the elegant part and the one worth understanding. There is no negotiation in the bargaining sense — no proposal and counter-proposal. Both ends compute the same function of the same inputs and therefore agree by construction. That is what makes the protocol robust: there is no state in which the two ends have agreed on different answers, because agreement is not communicated, it is derived.
5. RTL 1 — The Bring-Up Sequencer
Two features of this topology are load-bearing and neither is arbitrary.
Losing block lock falls back to CLOCKED, not to DOWN. Timing is still recovered — nothing about the signal changed — so restarting from the bottom would throw away work that is still valid and turn a brief search into a full outage. Each loss falls back exactly as far as its own precondition and no further, which is the same warm-restart argument Chapter 3.1 §7 made for pair training.
Losing marker lock falls back to BLOCKED, not to CLOCKED. Blocks still parse; only the lane identity is in doubt. And this is why Section 3's markers are periodic — the recovery is bounded by one marker interval rather than requiring a new block-lock search.
// SYNTHESIZABLE. The bring-up sequencer.
//
// It sequences; it does not implement. Every input here is produced by a
// block an earlier chapter owns:
//
// signal_ok <- PMD (Chapter 3.2)
// clock_locked <- PMA (Chapter 2.6)
// block_locked <- PCS aligner (Chapters 3.4, 3.5)
// markers_locked <- Section 6
// deskew_ok <- Section 7
//
// The design rule this module exists to enforce: EACH LOSS FALLS BACK
// EXACTLY AS FAR AS ITS OWN PRECONDITION. A sequencer that returns to DOWN
// on any loss converts every brief search into a full outage, and upper
// layers see a link flap where the PHY saw a two-microsecond re-lock.
package bringup_pkg;
typedef enum logic [2:0] {
LS_DOWN = 3'd0, // nothing arriving
LS_SIGNAL = 3'd1, // energy present, no timing
LS_CLOCKED = 3'd2, // timing recovered, no block boundaries
LS_BLOCKED = 3'd3, // blocks parse, lane identity unknown
LS_MARKED = 3'd4, // lanes identified, skew not yet removed
LS_UP = 3'd5
} link_state_e;
endpackage
module bringup_sequencer
import bringup_pkg::*;
#(
// Per-stage watchdogs. A stage that has not completed within its budget
// is STALLED, which is a different report from "still progressing" and
// must be distinguishable -- Section 11 depends on it.
parameter int unsigned T_SIGNAL = 1_000_000,
parameter int unsigned T_CLOCK = 2_000_000,
parameter int unsigned T_BLOCK = 8_000_000,
parameter int unsigned T_MARKER = 4_000_000,
parameter int unsigned T_DESKEW = 1_000_000,
parameter int unsigned TMR_W = 24
) (
input logic clk,
input logic rst_n,
input logic negotiated, // Section 8: both ends agreed what to try
input logic signal_ok,
input logic clock_locked,
input logic block_locked,
input logic markers_locked,
input logic deskew_ok,
output link_state_e state,
output logic link_up,
// How long the current stage has been running. Exposed continuously, not
// only on failure -- a stage that usually takes 2 ms and today takes 40
// is a finding even though it eventually succeeded.
output logic [TMR_W-1:0] stage_elapsed,
output logic stage_stalled,
// Pulses, for the timeline recorder of Section 9.
output logic advanced,
output logic regressed
);
link_state_e state_q, state_d;
logic [TMR_W-1:0] timer_q;
logic [TMR_W-1:0] budget_c;
always_comb begin
unique case (state_q)
LS_DOWN: budget_c = TMR_W'(T_SIGNAL);
LS_SIGNAL: budget_c = TMR_W'(T_CLOCK);
LS_CLOCKED: budget_c = TMR_W'(T_BLOCK);
LS_BLOCKED: budget_c = TMR_W'(T_MARKER);
LS_MARKED: budget_c = TMR_W'(T_DESKEW);
default: budget_c = '1; // LS_UP has no budget
endcase
end
always_comb begin
state_d = state_q;
// Signal loss beats everything. Evaluated FIRST so no later branch can
// advance a stage on stale evidence after the signal has gone -- the
// same priority rule Chapter 3.1 §7 asserted for pair training.
if (!signal_ok && (state_q != LS_DOWN)) begin
state_d = LS_DOWN;
end else begin
unique case (state_q)
LS_DOWN:
// Negotiation must complete before the ladder starts. Without
// this term the ladder can begin against a mode the far end is
// not using, which fails at clock lock with a healthy channel.
if (signal_ok && negotiated) state_d = LS_SIGNAL;
LS_SIGNAL:
if (clock_locked) state_d = LS_CLOCKED;
LS_CLOCKED:
if (block_locked) state_d = LS_BLOCKED;
LS_BLOCKED: begin
if (!block_locked) state_d = LS_CLOCKED; // fall back one rung
else if (markers_locked) state_d = LS_MARKED;
end
LS_MARKED: begin
if (!block_locked) state_d = LS_CLOCKED;
else if (!markers_locked) state_d = LS_BLOCKED; // one rung, not two
else if (deskew_ok) state_d = LS_UP;
end
LS_UP: begin
if (!block_locked) state_d = LS_CLOCKED;
else if (!markers_locked) state_d = LS_BLOCKED;
else if (!deskew_ok) state_d = LS_MARKED;
end
default: state_d = LS_DOWN;
endcase
end
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
state_q <= LS_DOWN;
timer_q <= '0;
advanced <= 1'b0;
regressed <= 1'b0;
end else begin
state_q <= state_d;
// The timer measures time IN A STAGE, so it resets on any change.
if (state_d != state_q) timer_q <= '0;
else if (timer_q != '1) timer_q <= timer_q + 1'b1;
advanced <= (state_d > state_q);
regressed <= (state_d < state_q);
end
end
assign state = state_q;
assign link_up = (state_q == LS_UP);
assign stage_elapsed = timer_q;
assign stage_stalled = (state_q != LS_UP) && (timer_q >= budget_c);
endmoduleClassification: synthesizable.
What it teaches: that fallback depth is a design decision with a large operational consequence. A sequencer returning to LS_DOWN on any loss converts a two-microsecond block re-lock into a full outage that every layer above must absorb. Falling back exactly one rung — to the state whose precondition still holds — keeps a brief search brief.
Deliberately simplified: no per-lane state. The lock inputs here are already-combined across lanes, and Section 7 shows what that combining costs in visibility. A production sequencer usually keeps per-lane block and marker lock and combines them explicitly.
Production implication: stage_stalled and stage_elapsed are separate outputs on purpose. A stage that eventually succeeded after forty times its usual duration is a finding — a marginal channel, a slow far end, a search running with less evidence than expected — and it is completely invisible to any check that only looks at final state. Expose the elapsed time continuously, not only when a watchdog fires.
And negotiated gating the first transition matters. Without it the ladder can begin against a mode the far end is not using, which fails at clock lock with a perfectly healthy channel — Section 4's case two, and one of the hardest bring-up failures to diagnose without this term making the dependency explicit.
6. RTL 2 — Alignment Marker Insertion and Detection
// SYNTHESIZABLE. Alignment marker insertion and detection.
//
// NO CLAUSE'S MARKER FORMAT IS ENCODED HERE. The three properties that
// matter architecturally are:
//
// RECOGNISABLE -- distinguishable from data, so a receiver can find one
// LANE-IDENTIFYING-- carries which logical lane this physical lane is
// PERIODIC -- inserted on a fixed interval, forever
//
// The third is what bounds marker-lock time and what makes Section 7's
// skew measurement possible: the SAME marker goes into every lane at the
// SAME moment, so arrival-time differences ARE the skew.
package marker_pkg;
parameter int unsigned LANES = 4;
parameter int unsigned LANE_W = $clog2(LANES);
endpackage
module marker_inserter
import marker_pkg::*;
#(
// Blocks between markers. The worst-case marker-lock time is exactly this
// many block times, which is a DESIGNED bound rather than a search that
// depends on data content.
parameter int unsigned MARKER_PERIOD = 16384,
parameter int unsigned PER_W = $clog2(MARKER_PERIOD)
) (
input logic clk,
input logic rst_n,
input logic [LANE_W-1:0] lane_id,
input logic blk_valid,
input logic [65:0] blk_in,
output logic blk_ready,
output logic out_valid,
output logic [65:0] out_blk,
output logic out_is_marker,
// Pulses when a marker is inserted. All lanes must pulse on the SAME
// cycle -- Section 7's measurement is meaningless otherwise, and
// Section 13 asserts it.
output logic marker_inserted
);
logic [PER_W-1:0] count_q;
logic insert_c;
assign insert_c = (count_q == PER_W'(MARKER_PERIOD - 1));
// A marker displaces a data block rather than being squeezed between
// them, so the upstream must be held for exactly one cycle.
assign blk_ready = !insert_c;
// The marker carries the lane identity and a recognisable body. The body
// pattern here is illustrative; what matters is that it is distinguishable
// from data and that the identity field exists.
function automatic logic [65:0] make_marker(input logic [LANE_W-1:0] id);
make_marker = {2'b10, 8'hA5, 46'h2A2A_2A2A_2A2A, {(10-LANE_W){1'b0}}, id};
endfunction
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
count_q <= '0;
out_valid <= 1'b0;
out_blk <= '0;
out_is_marker <= 1'b0;
marker_inserted <= 1'b0;
end else begin
out_valid <= 1'b1; // the PCS never goes silent
marker_inserted <= insert_c;
if (insert_c) begin
out_blk <= make_marker(lane_id);
out_is_marker <= 1'b1;
count_q <= '0;
end else begin
out_blk <= blk_valid ? blk_in : 66'0;
out_is_marker <= 1'b0;
count_q <= count_q + 1'b1;
end
end
end
endmodule
// SYNTHESIZABLE. Detection, on the receive side of one lane.
//
// Marker lock is UNLIKE every other gate on the ladder: its duration is
// bounded by when the transmitter next sends a marker, not by how quickly
// evidence accumulates. So a marker lock taking much longer than one
// marker period means markers are NOT ARRIVING -- a different fault from a
// slow search, and one this module makes distinguishable.
module marker_detector
import marker_pkg::*;
#(
parameter int unsigned MARKER_PERIOD = 16384,
// Consecutive markers at the expected spacing before lock is declared.
// More than one, because a data block can coincidentally resemble a
// marker; the SPACING is the strong evidence, not the pattern alone.
parameter int unsigned LOCK_COUNT = 4,
parameter int unsigned CNT_W = 24
) (
input logic clk,
input logic rst_n,
input logic blk_valid,
input logic [65:0] blk_in,
output logic marker_seen,
output logic [LANE_W-1:0] detected_lane_id,
output logic marker_locked,
// Cycles since the last marker. If this exceeds the period by a margin,
// markers have stopped -- which is a transmitter or upstream fault, not
// a receive search problem.
output logic [CNT_W-1:0] since_last_marker,
output logic markers_missing,
output logic [CNT_W-1:0] c_markers_seen,
output logic [CNT_W-1:0] c_markers_missed
);
logic [CNT_W-1:0] gap_q;
logic [$clog2(LOCK_COUNT+1)-1:0] good_q;
logic is_marker_c;
logic spacing_ok_c;
// Recognition: the control sync header plus the marker body pattern.
assign is_marker_c = blk_valid
&& (blk_in[65:64] == 2'b10)
&& (blk_in[63:56] == 8'hA5);
// The spacing check is the real evidence. A lone matching block proves
// little; one arriving exactly a period after the last is convincing.
assign spacing_ok_c = (gap_q >= CNT_W'(MARKER_PERIOD - 2))
&& (gap_q <= CNT_W'(MARKER_PERIOD + 2));
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
gap_q <= '0;
good_q <= '0;
marker_seen <= 1'b0;
marker_locked <= 1'b0;
detected_lane_id <= '0;
c_markers_seen <= '0;
c_markers_missed <= '0;
end else begin
marker_seen <= is_marker_c;
if (is_marker_c) begin
detected_lane_id <= blk_in[LANE_W-1:0];
gap_q <= '0;
if (!(&c_markers_seen)) c_markers_seen <= c_markers_seen + 1'b1;
if (spacing_ok_c) begin
if (good_q < ($clog2(LOCK_COUNT+1))'(LOCK_COUNT)) good_q <= good_q + 1'b1;
if (good_q + 1'b1 >= ($clog2(LOCK_COUNT+1))'(LOCK_COUNT)) marker_locked <= 1'b1;
end else begin
// Right pattern, wrong spacing: a coincidental match. Restart the
// evidence rather than counting it.
good_q <= '0;
end
end else begin
if (blk_valid && (gap_q != '1)) gap_q <= gap_q + 1'b1;
// A marker period passed with no marker. Lock is lost, and the
// cause is upstream rather than here.
if (gap_q > CNT_W'(MARKER_PERIOD + (MARKER_PERIOD >> 2))) begin
marker_locked <= 1'b0;
good_q <= '0;
if (!(&c_markers_missed)) c_markers_missed <= c_markers_missed + 1'b1;
end
end
end
end
assign since_last_marker = gap_q;
assign markers_missing = (gap_q > CNT_W'(MARKER_PERIOD + (MARKER_PERIOD >> 2)));
endmoduleClassification: synthesizable, with the marker body illustrative and the structure exact.
What it teaches: that the spacing is the evidence, not the pattern. A data block can coincidentally match a marker's body — the body is only so many bits — but a block matching and arriving exactly one period after the last one is convincing. That is the same evidence-accumulation reasoning Chapter 3.4 §11 used for block lock, applied to a different quantity, and it is why LOCK_COUNT is greater than one.
Deliberately simplified: the marker body. No clause's format is reproduced, because a partial reproduction would be worse than an obvious placeholder.
Production implication: markers_missing distinguishes two faults that look identical from the outside. Markers arriving but not locking means a receive-side search problem — spacing tolerance, or a lane identity field being misread. Markers not arriving at all means the far end is not inserting them, or block lock on this lane is failing so they cannot be recognised. Same symptom, opposite ends of the link, and only this output separates them.
Later ownership: the marker's normative structure belongs to the clause.
7. RTL 3 — Lane Deskew, and Measuring the Skew
Three chapters deferred this. Here it is, and the important part is that the buffer measures skew rather than merely absorbing it.
// SYNTHESIZABLE. Lane deskew with measurement.
//
// HOW SKEW IS MEASURED: the transmitter inserts the SAME marker into every
// lane at the SAME moment. Any difference in arrival time at the receiver
// is skew accumulated along the physical path -- different fibre lengths,
// different trace lengths, different retimer delays.
//
// So the measurement is: note when each lane's marker arrives, take the
// earliest as the reference, and each lane's lag is its delay.
//
// WHY MEASURING MATTERS MORE THAN ABSORBING: skew drifts with temperature.
// A design that absorbs skew silently works until the day the drift exceeds
// the buffer, and then fails with no warning. A design that REPORTS the
// margin gives months of it.
module lane_deskew
import marker_pkg::*;
#(
parameter int unsigned DEPTH = 64, // blocks of skew absorbable
parameter int unsigned PTR_W = $clog2(DEPTH),
parameter int unsigned BLK_W = 66
) (
input logic clk,
input logic rst_n,
input logic [LANES-1:0] lane_valid,
input logic [BLK_W-1:0] lane_blk [LANES],
input logic [LANES-1:0] lane_marker, // this lane's marker arrived now
input logic [LANES-1:0] lane_locked, // this lane has marker lock
output logic aligned_valid,
output logic [BLK_W-1:0] aligned_blk [LANES],
output logic deskew_ok,
// Measured skew per lane, in blocks, relative to the earliest lane.
output logic [PTR_W:0] lane_skew [LANES],
output logic [PTR_W:0] worst_skew,
output logic [LANE_W-1:0] worst_lane,
// How much of the buffer is unused at the current worst skew. This is the
// number to trend: it shrinks as the system warms, and it reaches zero
// before anything fails.
output logic [PTR_W:0] skew_margin,
// Skew exceeded what the buffer can absorb. Distinct from "not yet
// aligned" -- this one cannot be fixed by waiting.
output logic skew_overflow,
output logic [15:0] c_skew_overflow
);
// Per-lane elastic storage. Each lane writes on arrival and all lanes are
// read together once every lane's marker has been seen.
logic [BLK_W-1:0] fifo [LANES][DEPTH];
logic [PTR_W-1:0] wptr [LANES];
logic [PTR_W-1:0] rptr;
// Marker arrival capture. The first lane to present its marker sets the
// reference; every other lane's lag is counted from there.
logic [LANES-1:0] marker_seen_q;
logic [PTR_W:0] marker_at [LANES];
logic [PTR_W:0] ref_time_q;
logic measuring_q;
logic [PTR_W:0] tick_q;
logic all_markers_c;
assign all_markers_c = &marker_seen_q;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int unsigned l = 0; l < LANES; l++) begin
wptr[l] <= '0;
marker_at[l] <= '0;
lane_skew[l] <= '0;
end
rptr <= '0;
marker_seen_q <= '0;
measuring_q <= 1'b0;
tick_q <= '0;
ref_time_q <= '0;
deskew_ok <= 1'b0;
skew_overflow <= 1'b0;
c_skew_overflow <= '0;
aligned_valid <= 1'b0;
end else begin
// Write side: each lane stores independently, as it arrives.
for (int unsigned l = 0; l < LANES; l++) begin
if (lane_valid[l]) begin
fifo[l][wptr[l]] <= lane_blk[l];
wptr[l] <= wptr[l] + 1'b1;
end
end
// Measurement window: opens on the first marker of a round.
if (|lane_marker && !measuring_q) begin
measuring_q <= 1'b1;
tick_q <= '0;
ref_time_q <= '0;
marker_seen_q <= lane_marker;
for (int unsigned l = 0; l < LANES; l++)
if (lane_marker[l]) marker_at[l] <= '0;
end else if (measuring_q) begin
tick_q <= tick_q + 1'b1;
for (int unsigned l = 0; l < LANES; l++) begin
if (lane_marker[l] && !marker_seen_q[l]) begin
marker_seen_q[l] <= 1'b1;
marker_at[l] <= tick_q;
lane_skew[l] <= tick_q; // lag behind the earliest lane
end
end
// Every lane reported: the round is complete.
if (all_markers_c) begin
measuring_q <= 1'b0;
marker_seen_q <= '0;
deskew_ok <= (&lane_locked) && (tick_q < (PTR_W+1)'(DEPTH));
if (tick_q >= (PTR_W+1)'(DEPTH)) begin
skew_overflow <= 1'b1;
if (!(&c_skew_overflow)) c_skew_overflow <= c_skew_overflow + 1'b1;
end else begin
skew_overflow <= 1'b0;
end
end else if (tick_q >= (PTR_W+1)'(DEPTH)) begin
// A lane's marker has not arrived within the buffer's reach. It
// cannot be aligned no matter how long we wait.
measuring_q <= 1'b0;
marker_seen_q <= '0;
deskew_ok <= 1'b0;
skew_overflow <= 1'b1;
if (!(&c_skew_overflow)) c_skew_overflow <= c_skew_overflow + 1'b1;
end
end
// Read side: all lanes advance together once aligned.
aligned_valid <= deskew_ok;
if (deskew_ok) rptr <= rptr + 1'b1;
end
end
always_comb begin
for (int unsigned l = 0; l < LANES; l++)
aligned_blk[l] = fifo[l][(rptr - PTR_W'(lane_skew[l]))];
end
// Worst lane and the remaining headroom.
logic [PTR_W:0] max_c;
logic [LANE_W-1:0] maxl_c;
always_comb begin
max_c = lane_skew[0];
maxl_c = '0;
for (int unsigned l = 1; l < LANES; l++) begin
if (lane_skew[l] > max_c) begin
max_c = lane_skew[l];
maxl_c = LANE_W'(l);
end
end
worst_skew = max_c;
worst_lane = maxl_c;
skew_margin = (max_c < (PTR_W+1)'(DEPTH)) ? ((PTR_W+1)'(DEPTH) - max_c)
: '0;
end
endmoduleClassification: synthesizable.
What it teaches: that skew is measurable, and the measurement is free because the markers of Section 6 are already being transmitted for a different reason. One mechanism, two purposes — lane identity and a timing reference — which is why periodic markers earn their bandwidth.
Deliberately simplified: a simple indexed buffer rather than a per-lane elastic structure with independent read and write clocks. Production designs need the clock-domain handling of Chapter 2.6's elastic buffer here too.
Production implication: skew_margin is the number to trend, and it is the module's reason for existing. Skew drifts with temperature — different fibre lengths and different trace lengths expand differently, and retimer delays vary. A design that absorbs skew silently works perfectly until the drift exceeds DEPTH, then fails abruptly with no warning. A design that reports the margin shows it shrinking across a warm-up cycle and across seasons, giving months of notice. This is Chapter 3.3's slope-against-cliff argument at a third layer.
And skew_overflow is distinct from "not yet aligned" for a reason. Not-yet-aligned resolves by waiting. Overflow does not — the skew exceeds what the hardware can absorb, and waiting forever will not fix it. Merging them produces a link that appears to be still trying when it has actually failed.
Later ownership: the normative maximum skew a clause permits is clause-specific, and DEPTH must be chosen against it.
8. RTL 4 — Auto-Negotiation State
// SYNTHESIZABLE. Auto-negotiation as a state machine, without a page format.
//
// NO CLAUSE'S LINK CODEWORD IS ENCODED HERE. What is modelled is the shape
// common to every family that negotiates:
//
// ADVERTISE -> ACKNOWLEDGE -> RESOLVE -> COMMIT
//
// The elegant part is RESOLVE. There is no bargaining -- both ends apply the
// SAME deterministic priority rule to the SAME capability intersection, so
// they agree by construction rather than by exchange. There is no state in
// which the two ends have committed to different answers.
module autoneg
#(
parameter int unsigned CAP_W = 8, // one bit per capability
parameter int unsigned T_ADVERTISE = 500_000,
parameter int unsigned T_ACK = 500_000,
parameter int unsigned TMR_W = 24
) (
input logic clk,
input logic rst_n,
input logic an_enable, // some ports are forced, not negotiated
input logic [CAP_W-1:0] local_caps,
// From the far end. Not modelled: how these are carried on the wire.
input logic remote_valid,
input logic [CAP_W-1:0] remote_caps,
input logic remote_ack,
output logic advertising,
output logic local_ack,
output logic negotiated,
output logic [CAP_W-1:0] selected_cap, // one-hot: the agreed mode
// Why negotiation did not complete. Three causes, three owners -- and the
// second is the one that produces the hardest field failures.
output logic no_remote, // far end never advertised
output logic no_common_cap, // both advertised, nothing shared
output logic an_timeout,
// Sticky: negotiation completed without the far end participating, so a
// default was assumed. Section 4's case one, made visible -- this is the
// duplex-mismatch signature.
output logic defaulted
);
typedef enum logic [2:0] {
AN_IDLE, AN_ADVERTISE, AN_ACK, AN_RESOLVE, AN_DONE, AN_FAIL
} an_state_e;
an_state_e state_q, state_d;
logic [TMR_W-1:0] timer_q;
logic [CAP_W-1:0] common_c;
assign common_c = local_caps & remote_caps;
// The deterministic priority rule. Highest set bit wins, and BOTH ENDS
// COMPUTE THE SAME FUNCTION -- which is what makes agreement automatic.
// The rule itself must be identical at both ends; a design whose priority
// differs from its peer's produces two ends committed to different modes,
// each believing negotiation succeeded.
function automatic logic [CAP_W-1:0] resolve(input logic [CAP_W-1:0] c);
resolve = '0;
for (int unsigned i = 0; i < CAP_W; i++)
if (c[i]) resolve = (CAP_W)'(1) << i; // highest set bit survives
endfunction
always_comb begin
state_d = state_q;
unique case (state_q)
AN_IDLE: if (an_enable) state_d = AN_ADVERTISE;
AN_ADVERTISE: begin
if (remote_valid) state_d = AN_ACK;
else if (timer_q >= TMR_W'(T_ADVERTISE)) state_d = AN_FAIL;
end
AN_ACK: begin
if (remote_ack) state_d = AN_RESOLVE;
else if (timer_q >= TMR_W'(T_ACK)) state_d = AN_FAIL;
end
AN_RESOLVE: state_d = (common_c != '0) ? AN_DONE : AN_FAIL;
AN_DONE: if (!an_enable || !remote_valid) state_d = AN_IDLE;
AN_FAIL: if (!an_enable) state_d = AN_IDLE;
default: state_d = AN_IDLE;
endcase
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
state_q <= AN_IDLE;
timer_q <= '0;
selected_cap <= '0;
defaulted <= 1'b0;
end else begin
state_q <= state_d;
if (state_d != state_q) timer_q <= '0;
else if (timer_q != '1) timer_q <= timer_q + 1'b1;
if (state_d == AN_DONE) selected_cap <= resolve(common_c);
// Negotiation was enabled, the far end never spoke, and we proceeded
// on a default. This is the duplex-mismatch signature, and it is
// sticky because by the time anyone looks the link may be "up".
if ((state_q == AN_ADVERTISE) && (state_d == AN_FAIL) && !remote_valid)
defaulted <= 1'b1;
end
end
assign advertising = (state_q == AN_ADVERTISE);
assign local_ack = (state_q == AN_ACK) || (state_q == AN_RESOLVE);
assign negotiated = (state_q == AN_DONE) || !an_enable;
assign no_remote = (state_q == AN_FAIL) && !remote_valid;
assign no_common_cap = (state_q == AN_FAIL) && remote_valid && (common_c == '0);
assign an_timeout = (state_q == AN_FAIL) && (timer_q >= TMR_W'(T_ACK));
endmoduleClassification: synthesizable, with the wire encoding deliberately absent.
What it teaches: that resolution is computed, not exchanged. Both ends apply the same deterministic priority rule to the same capability intersection, so they agree by construction — and there is no state in which they have committed to different answers. That is what makes the protocol robust, and it is why the priority rule must be identical at both ends rather than merely compatible.
Deliberately simplified: everything on the wire. How capabilities are carried, how many pages there are, and how extensions are signalled are all clause-specific.
Production implication: defaulted is sticky and it is the single most valuable output in the module. Section 4's case one — one end fixed, one end negotiating — produces a link that comes up, works at low traffic, and collapses under load. By the time anyone investigates, the link reports up, and the only surviving evidence that a default was assumed rather than negotiated is this bit. Without it the diagnosis requires access to both ends' configuration, which is often not available.
And negotiated is asserted when an_enable is low, so a deliberately forced port does not stall Section 5's sequencer forever. That is a small line with a large consequence — a forced port that never asserts negotiated never leaves LS_DOWN, and the symptom is a link that will not come up with no fault reported anywhere.
9. RTL 5 — The Bring-Up Timeline
This is the module that makes the chapter's central diagnostic question answerable.
// SYNTHESIZABLE INSTRUMENTATION.
//
// Current state answers "is the link up". It cannot answer:
//
// - which gate did it stall at?
// - how long did each gate take?
// - did it get further last time than this time?
// - is bring-up getting slower over weeks?
//
// Those are the questions with owners, and all four need a TIMELINE.
module bringup_timeline
import bringup_pkg::*;
#(
parameter int unsigned TS_W = 32,
parameter int unsigned NSTAGE = 6
) (
input logic clk,
input logic rst_n,
input logic clear,
input logic attempt_start, // a bring-up attempt begins
input link_state_e state,
input logic advanced,
input logic regressed,
input logic stage_stalled,
// Time spent in each stage during the attempt that just finished.
output logic [TS_W-1:0] stage_duration [NSTAGE],
output logic timeline_valid,
// The furthest state reached this attempt, and ever. "Ever" matters: a
// link that once reached UP and now stalls at BLOCKED is a regression,
// and one that has NEVER reached UP was never configured correctly --
// completely different investigations.
output link_state_e furthest_this_attempt,
output link_state_e furthest_ever,
// Where it stalled, if it did.
output link_state_e stall_state,
output logic stall_valid,
output logic [15:0] c_attempts,
output logic [15:0] c_successes,
output logic [15:0] c_regressions,
// Total time of the last successful bring-up. Trend it: a bring-up that
// took 3 ms in January and 300 ms in July is a marginal link, even though
// both succeeded and both report UP.
output logic [TS_W-1:0] last_bringup_time
);
logic [TS_W-1:0] stage_timer_q;
logic [TS_W-1:0] total_timer_q;
link_state_e prev_state_q;
logic in_attempt_q;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int unsigned i = 0; i < NSTAGE; i++) stage_duration[i] <= '0;
stage_timer_q <= '0;
total_timer_q <= '0;
prev_state_q <= LS_DOWN;
in_attempt_q <= 1'b0;
timeline_valid <= 1'b0;
furthest_this_attempt <= LS_DOWN;
furthest_ever <= LS_DOWN;
stall_state <= LS_DOWN;
stall_valid <= 1'b0;
c_attempts <= '0;
c_successes <= '0;
c_regressions <= '0;
last_bringup_time <= '0;
end else begin
if (clear) begin
c_attempts <= '0;
c_successes <= '0;
c_regressions <= '0;
// furthest_ever deliberately NOT cleared: it distinguishes "never
// worked" from "worked and regressed", which no counter can.
end
if (attempt_start) begin
for (int unsigned i = 0; i < NSTAGE; i++) stage_duration[i] <= '0;
stage_timer_q <= '0;
total_timer_q <= '0;
in_attempt_q <= 1'b1;
timeline_valid <= 1'b0;
furthest_this_attempt <= LS_DOWN;
stall_valid <= 1'b0;
if (!(&c_attempts)) c_attempts <= c_attempts + 1'b1;
end else if (in_attempt_q) begin
if (stage_timer_q != '1) stage_timer_q <= stage_timer_q + 1'b1;
if (total_timer_q != '1) total_timer_q <= total_timer_q + 1'b1;
// A state change closes the stage that just ended.
if (state != prev_state_q) begin
stage_duration[prev_state_q] <= stage_timer_q;
stage_timer_q <= '0;
end
if (state > furthest_this_attempt) furthest_this_attempt <= state;
if (state > furthest_ever) furthest_ever <= state;
if (regressed && !(&c_regressions)) c_regressions <= c_regressions + 1'b1;
// Stalled: record WHERE, once, and hold it.
if (stage_stalled && !stall_valid) begin
stall_state <= state;
stall_valid <= 1'b1;
end
if (state == LS_UP) begin
in_attempt_q <= 1'b0;
timeline_valid <= 1'b1;
last_bringup_time <= total_timer_q;
if (!(&c_successes)) c_successes <= c_successes + 1'b1;
end
end
prev_state_q <= state;
end
end
endmoduleClassification: synthesizable instrumentation.
What it teaches: that furthest_ever and furthest_this_attempt answer different questions, and both are needed. A link stalling at LS_BLOCKED that has previously reached LS_UP is a regression — something changed, and the configuration was once right. One that has never reached LS_UP was never configured correctly, and the investigation starts from the far end's settings rather than from the channel. Same current state, opposite starting points.
Deliberately simplified: durations in raw clock cycles rather than a real-time base, and no per-lane breakdown of the stages that are per-lane.
Production implication: last_bringup_time, trended, catches a class of fault nothing else sees. A bring-up that took 3 ms in January and 300 ms in July is a marginal link — the searches are taking longer because the evidence is degrading — and both attempts succeeded and both report LS_UP. This is Chapter 3.1 §15's retrain-rate argument and Chapter 3.3's margin-trend argument arriving at a third quantity: the duration of a successful operation is itself a health metric.
And furthest_ever deliberately survives clear, for the same reason Chapter 3.1's ever_ready does: an operator clearing counters for a clean measurement window would otherwise destroy the one fact that separates "never worked" from "worked and degraded".
10. RTL 6 — Per-Lane Lock, and What Combining It Early Costs
Section 5's sequencer takes block_locked and markers_locked as single bits. On a multi-lane link those bits are already an AND across lanes, and that reduction happened somewhere. Where it happens decides whether the link is diagnosable.
// SYNTHESIZABLE. Per-lane lock, combined late and preserved.
//
// The sequencer needs one bit. An engineer needs LANES bits. Both are cheap;
// only the second is ever omitted, and it is the one that answers the
// question every multi-lane investigation starts with: WHICH LANE?
//
// The asymmetry that makes this worth a module: a link is up only when
// EVERY lane is locked, so the combined bit is dominated by the WORST lane.
// One lane cycling in and out of lock makes the whole link flap, and from
// the combined bit alone that is indistinguishable from all lanes being
// marginal -- a completely different fault with a completely different fix.
module lane_lock_aggregator
import marker_pkg::*;
#(
parameter int unsigned CNT_W = 20
) (
input logic clk,
input logic rst_n,
input logic clear,
input logic [LANES-1:0] lane_block_locked,
input logic [LANES-1:0] lane_marker_locked,
input logic [LANES-1:0] lane_markers_missing,
// What the sequencer of Section 5 consumes.
output logic block_locked,
output logic markers_locked,
// What an engineer consumes. Never reduced.
output logic [LANES-1:0] block_mask,
output logic [LANES-1:0] marker_mask,
// Lock losses per lane. One lane dominating this is a physical claim
// about one fibre or one trace; a uniform spread points at something
// shared -- the reference clock, the supply, the far end.
output logic [CNT_W-1:0] c_block_loss [LANES],
output logic [CNT_W-1:0] c_marker_loss [LANES],
// The lane that has lost lock most often. The single most useful output
// in the module, and the one an AND gate destroys.
output logic [LANE_W-1:0] worst_lane,
output logic worst_lane_valid,
// Which lanes were unlocked at the moment the combined bit last fell.
// By the time anyone reads it the lane has usually recovered, so live
// state says nothing and this is the only surviving evidence.
output logic [LANES-1:0] down_cause_mask,
output logic down_cause_valid,
// Sticky per lane: has this lane EVER locked? Distinguishes a lane that
// has never worked -- miswired, wrong fibre, dead transceiver -- from one
// that works and drops. Different faults, identical combined bit.
output logic [LANES-1:0] ever_locked
);
logic [LANES-1:0] blk_q, mrk_q;
logic combined_q;
assign block_locked = &lane_block_locked;
assign markers_locked = &lane_marker_locked;
assign block_mask = lane_block_locked;
assign marker_mask = lane_marker_locked;
wire combined_c = (&lane_block_locked) && (&lane_marker_locked);
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int unsigned l = 0; l < LANES; l++) begin
c_block_loss[l] <= '0;
c_marker_loss[l] <= '0;
end
blk_q <= '0;
mrk_q <= '0;
combined_q <= 1'b0;
down_cause_mask <= '0;
down_cause_valid <= 1'b0;
ever_locked <= '0;
end else begin
if (clear) begin
for (int unsigned l = 0; l < LANES; l++) begin
c_block_loss[l] <= '0;
c_marker_loss[l] <= '0;
end
// ever_locked deliberately NOT cleared -- it is a property of the
// installation, not of a window an operator happened to choose.
end else begin
for (int unsigned l = 0; l < LANES; l++) begin
if (blk_q[l] && !lane_block_locked[l] && !(&c_block_loss[l]))
c_block_loss[l] <= c_block_loss[l] + 1'b1;
if (mrk_q[l] && !lane_marker_locked[l] && !(&c_marker_loss[l]))
c_marker_loss[l] <= c_marker_loss[l] + 1'b1;
end
end
blk_q <= lane_block_locked;
mrk_q <= lane_marker_locked;
combined_q <= combined_c;
for (int unsigned l = 0; l < LANES; l++)
if (lane_block_locked[l] && lane_marker_locked[l]) ever_locked[l] <= 1'b1;
// Capture on the DOWN edge only. Sampling throughout the outage would
// record recovery rather than failure.
if (combined_q && !combined_c) begin
down_cause_mask <= ~(lane_block_locked & lane_marker_locked);
down_cause_valid <= 1'b1;
end
end
end
// Worst lane by combined loss count.
logic [CNT_W-1:0] max_c;
logic [LANE_W-1:0] maxl_c;
logic any_c;
always_comb begin
max_c = c_block_loss[0] + c_marker_loss[0];
maxl_c = '0;
any_c = (max_c != '0);
for (int unsigned l = 1; l < LANES; l++) begin
automatic logic [CNT_W-1:0] tot = c_block_loss[l] + c_marker_loss[l];
if (tot != '0) any_c = 1'b1;
if (tot > max_c) begin
max_c = tot;
maxl_c = LANE_W'(l);
end
end
worst_lane = maxl_c;
worst_lane_valid = any_c;
end
endmoduleClassification: synthesizable.
What it teaches: that combining is not the problem — combining early is. The sequencer genuinely needs one bit, and producing it costs an AND gate. What costs nothing extra and is nearly always omitted is keeping the inputs. Once reduced, the question every multi-lane investigation opens with — which lane? — has no answer anywhere in the design.
Deliberately simplified: no per-lane timeline. A production design often keeps Section 9's stage durations per lane too, which answers "which lane was slow" as well as "which lane failed".
Production implication: the two masks separate two faults that produce an identical combined bit. One lane dominating c_block_loss is a physical claim about one fibre or one trace — check that path. A uniform spread across lanes points at something shared: the reference clock, the supply, the environment, or the far end. Same symptom, opposite investigations, and only per-lane counts choose between them.
And ever_locked carries the same distinction Chapter 3.1 §9 drew for pairs. A lane that has never locked is miswired, connected to the wrong fibre, or facing a dead transceiver. A lane that locks and drops is marginal. The combined bit reads identically for both, and only this mask separates them — which is why it survives clear.
11. Why Bring-Up Time Is a Sum of Search Times
The ladder's stages take wildly different times, and until Chapter 3.5 it was not possible to say why. Now it is.
| Gate | What bounds its duration | Typical character |
|---|---|---|
| signal detect | the detector's own response time | fast, and fixed |
| clock lock | the recovery loop's settling time | fast, and fixed |
| block lock | evidence quality — how fast a wrong offset is rejected | variable, and code-dependent |
| marker lock | the marker interval — when one is next sent | bounded, and by design |
| deskew | one marker round across all lanes | bounded, and by design |
| codeword lock | codeword structure and its own evidence | variable |
Row three is the one that surprises people, and Chapter 3.5 §11 quantified it. A 4B/5B stream has seven illegal patterns in thirty-two, so a wrong offset is rejected quickly and the search converges fast. A 64b/66b stream has two illegal headers in four, so a wrong offset survives roughly one block in two — the search needs far more blocks to reach the same confidence, and block lock on a 64b/66b link is genuinely, structurally slower.
That is not a defect. It is the price Chapter 3.5 §13 identified for the overhead falling from 25% to 3.125% — the alignment evidence budget shrank along with the overhead, and slower lock is what that costs.
Rows four and five are different in kind. They do not depend on evidence at all; they depend on when the transmitter next acts. Marker lock cannot happen faster than a marker arrives, so the interval is the bound — which makes them the only stages whose duration is predictable from configuration alone.
12. Assertions
Every property below is a property of these teaching models. IEEE 802.3 specifies auto-negotiation behaviour, marker structure and maximum skew per clause; how a particular design sequences the gates, chooses fallback depth and instruments the timeline is an implementation choice.
// ─── Ordering: the ladder cannot be skipped ────────────────────────────────
// The chapter's central claim, asserted. Catches: a sequencer that advances
// on a stale lock indication, declaring a link up on preconditions that no
// longer hold.
property p_ladder_is_monotone_upward;
@(posedge clk) disable iff (!rst_n)
advanced |-> (state == link_state_e'($past(state) + 1));
endproperty
// ─── Safety: signal loss dominates every other transition ──────────────────
// Catches a priority inversion where a stale block_locked lets the machine
// advance after the cable was unplugged. Same rule Chapter 3.1 §13 asserted
// for pair training, at link scope.
property p_signal_loss_wins;
@(posedge clk) disable iff (!rst_n)
(!signal_ok && (state != LS_DOWN)) |=> (state == LS_DOWN);
endproperty
// ─── Ordering: fallback goes exactly one rung ──────────────────────────────
// Catches a sequencer returning to LS_DOWN on any loss, which turns a
// microsecond block re-lock into a full outage every layer above absorbs.
property p_block_loss_falls_back_one;
@(posedge clk) disable iff (!rst_n)
((state == LS_BLOCKED) && signal_ok && !block_locked) |=> (state == LS_CLOCKED);
endproperty
property p_marker_loss_falls_back_one;
@(posedge clk) disable iff (!rst_n)
((state == LS_MARKED) && signal_ok && block_locked && !markers_locked)
|=> (state == LS_BLOCKED);
endproperty
// ─── Causation: negotiation gates the ladder ───────────────────────────────
// Catches a ladder starting before both ends agree what to attempt, which
// fails at clock lock on a healthy channel -- Section 4's hardest case.
property p_ladder_needs_negotiation;
@(posedge clk) disable iff (!rst_n)
((state == LS_DOWN) && !negotiated) |=> (state == LS_DOWN);
endproperty
// ─── Safety: link_up requires every gate ───────────────────────────────────
// Catches a design declaring the link usable on a subset of preconditions.
property p_up_requires_all_gates;
@(posedge clk) disable iff (!rst_n)
link_up |-> (signal_ok && clock_locked && block_locked
&& markers_locked && deskew_ok);
endproperty
// ─── MARKER PROPERTY: all lanes insert simultaneously ──────────────────────
// Section 7's skew measurement is meaningless unless this holds. Catches a
// per-lane counter that can drift, after which measured "skew" is actually
// insertion phase error and the deskew buffer absorbs a fiction.
property p_markers_inserted_together;
@(posedge clk) disable iff (!rst_n)
|marker_inserted_vec |-> &marker_inserted_vec;
endproperty
// ─── Causation: marker lock needs spacing, not just a pattern ──────────────
// Catches lock declared on a single coincidental pattern match, which locks
// onto a data block that happened to resemble a marker.
property p_marker_lock_needs_spacing;
@(posedge clk) disable iff (!rst_n)
$rose(marker_locked) |-> $past(spacing_ok_c);
endproperty
// ─── Safety: deskew requires every lane locked ─────────────────────────────
// Catches alignment declared while a lane is still searching, which reads
// that lane's buffer at an arbitrary offset and corrupts every word.
property p_deskew_needs_all_lanes;
@(posedge clk) disable iff (!rst_n)
deskew_ok |-> (&lane_locked);
endproperty
// ─── Safety: measured skew never exceeds the buffer ────────────────────────
// Catches a read pointer computed from a skew larger than DEPTH, which
// wraps and silently reads the wrong blocks -- data corruption on a link
// reporting alignment.
property p_skew_within_depth;
@(posedge clk) disable iff (!rst_n)
deskew_ok |-> (worst_skew < DEPTH);
endproperty
// ─── Mutual exclusion: overflow is not "still trying" ──────────────────────
// Catches merging the two, producing a link that appears to be progressing
// when it has actually failed and cannot recover by waiting.
property p_overflow_excludes_ok;
@(posedge clk) disable iff (!rst_n)
skew_overflow |-> !deskew_ok;
endproperty
// ─── Conservation: the timeline accounts for the whole attempt ─────────────
// Catches stage durations that drop time, making the timeline unusable for
// exactly the diagnosis it exists for.
property p_stage_durations_sum;
@(posedge clk) disable iff (!rst_n)
timeline_valid |-> (last_bringup_time == stage_duration_sum);
endproperty
// ─── Stability: furthest_ever never decreases ──────────────────────────────
// Catches furthest_ever folded into the clear branch, destroying the
// distinction between "never worked" and "worked and regressed".
property p_furthest_ever_monotone;
@(posedge clk) disable iff (!rst_n)
1'b1 |=> (furthest_ever >= $past(furthest_ever));
endproperty
// ─── Bounded response: a stalled stage is reported ─────────────────────────
// Catches a watchdog that never fires, leaving a link stuck forever with
// nothing recording where.
property p_stall_is_reported;
@(posedge clk) disable iff (!rst_n)
(stage_stalled && (state != LS_UP)) |-> ##[0:2] stall_valid;
endproperty
// ─── Conservation: the combined bit is exactly the AND of the lanes ────────
// Catches a combined signal that drifts from its inputs -- typically a
// registered copy that misses a one-cycle drop, after which the sequencer
// and the per-lane masks disagree about whether the link is locked.
property p_combined_matches_lanes;
@(posedge clk) disable iff (!rst_n)
(block_locked == (&lane_block_locked))
&& (markers_locked == (&lane_marker_locked));
endproperty
// ─── Stability: ever_locked never clears ───────────────────────────────────
// Catches ever_locked folded into the clear branch, which destroys the
// distinction between a lane that has never worked and one that drops.
property p_ever_locked_monotone;
@(posedge clk) disable iff (!rst_n)
1'b1 |=> (ever_locked == ($past(ever_locked) | ever_locked));
endproperty13. Verification
Scenarios
- A clean bring-up, all gates in order. Verify the sequencer passes
LS_DOWNthroughLS_UPwith no regression, thatadvancedpulses five times, and that the timeline records five non-zero stage durations summing tolast_bringup_time. - Signal never arrives. Verify the machine stays in
LS_DOWN,stage_stalledasserts afterT_SIGNAL, andstall_staterecordsLS_DOWNrather than being left invalid. - Signal lost at each of the five higher states. Five runs. Verify every one returns to
LS_DOWNimmediately, including fromLS_UP, and that no intermediate state is passed through. - Block lock lost while
LS_MARKED. Verify fallback toLS_CLOCKED— two rungs, because marker lock's precondition is also gone — and that this is distinguishable in the timeline from a single-rung fallback. - Marker lock lost while
LS_UP. Verify fallback to exactlyLS_BLOCKED, not toLS_CLOCKEDand not toLS_DOWN. - Negotiation never completes. Verify the ladder does not start,
no_remoteasserts, anddefaultedsticks. This is Section 4's case one. - Negotiation with no common capability. Verify
no_common_caprather thanno_remote, since they have different owners. - A forced port with
an_enablelow. Verifynegotiatedis asserted anyway so the ladder proceeds. A forced port that stalls inLS_DOWNforever is a real and easily-introduced bug. - Markers on all lanes, zero skew. Verify
deskew_ok,worst_skewof zero, andskew_marginequal to the fullDEPTH. - Markers with skew just under
DEPTH. Verify alignment succeeds,worst_lanenames the right lane, andskew_marginreads one. - Markers with skew exactly at
DEPTH. The boundary. Verifyskew_overflowrather than a silent wrap of the read pointer. - One lane's marker never arrives. Verify
markers_missingon that lane,deskew_oklow, and that the failure is attributed to the lane rather than reported as a generic deskew failure. - A data block coincidentally matching a marker pattern. Verify marker lock is not declared, because the spacing check fails. Then send genuine markers and verify lock is achieved.
- Markers stop after lock. Verify
marker_lockeddeasserts after the timeout,c_markers_missedadvances, and the sequencer falls back one rung. - Repeated bring-up attempts. Verify
c_attemptsandc_successesdiverge correctly and thatfurthest_this_attemptresets each time whilefurthest_everdoes not. clearduring operation. Verify counters zero andfurthest_everdoes not — the distinction between "never worked" and "worked and regressed" must survive.- Reset at each state. Verify the machine returns to
LS_DOWN, no partial timeline is reported valid, and no stale stall state persists. - One lane losing lock repeatedly, three healthy. Verify
worst_lanenames it,down_cause_maskcaptures it on the down edge, and the combinedblock_lockedbit alone gives no indication which lane it was. - A lane that never locks. Verify its
ever_lockedbit stays low while the others set theirs, and that it survives aclear. This is the miswired-lane case, and it is indistinguishable from a marginal lane on any combined signal.
What the checker must own
- A sequencer reference model implementing the eight transitions independently, including the fallback depths. Fallback depth is the property most likely to be broken by a refactor and the least likely to be noticed.
- A skew injector with independent per-lane delay, configurable up to and beyond
DEPTH. Without it scenarios 10 through 12 cannot be constructed at all. - A structured-payload generator, not merely a random one. Scenario 18 needs it, and so does any honest bring-up-time characterisation.
- Timeline closure checking: stage durations must sum to the total at every completed attempt, or the timeline cannot be used for the diagnosis it exists for.
- Coverage crosses of link state against stall against regression. The bin
(LS_CLOCKED, regressed, eventually LS_UP)must be well populated — that is a false lock recovered from, the normal healthy behaviour Section 12's rejected property would forbid.
14. Debugging — Which Gate, and For How Long
The symptom: a link that will not come up, or one that takes far longer than expected.
Step 1 — read stall_state, or the current state if it is still progressing. One register, and it names the gate. Every gate names a different owner:
| Stalled at | Meaning | Where to go |
|---|---|---|
LS_DOWN | nothing arriving, or negotiation incomplete | check no_remote first, then Chapter 3.2 or Chapter 3.1 |
LS_SIGNAL | signal present, timing not recovered | signal quality, or a rate mismatch between the ends |
LS_CLOCKED | timing recovered, blocks do not parse | a coding mismatch, or a slow search — check the duration |
LS_BLOCKED | blocks parse, markers not found | check markers_missing — arriving-but-not-locking is a different fault from not-arriving |
LS_MARKED | markers found, deskew failing | check skew_overflow — physical path lengths, not electronics |
Step 2 — read stage_elapsed alongside it, because the gate alone is not enough. A stage running for twice its usual duration is searching; one running for a thousand times is stuck. Same state, different problems, and stage_stalled is the design's own opinion about which.
Step 3 — read furthest_ever against furthest_this_attempt. This is the question everyone forgets to ask and it changes where the investigation starts:
furthest_everisLS_UP— this link has worked. Something changed: a cable moved, a module swapped, the temperature rose, the far end reconfigured. Look for the change.furthest_everhas never reachedLS_UP— this link has never worked. It was never configured correctly, and no amount of channel investigation will help. Start at the far end's settings.
Step 4 — if it stalls at LS_DOWN, separate the two causes before anything else. no_remote means the far end never advertised, and defaulted sticking means we proceeded on an assumption. Section 4's case one produces a link that comes up and then misbehaves under load, so defaulted should be checked even on links that appear to be working.
Step 5 — if it stalls at LS_MARKED, read worst_lane and skew_margin. Skew is a claim about physical path lengths, so worst_lane names a specific fibre or a specific trace. And skew_margin shrinking across a warm-up cycle is the early warning: a link that aligns when cold and fails when warm is a skew-margin problem, and the margin was visible for months before it failed.
Step 6 — the link comes up but takes far longer than it used to. Trend last_bringup_time. There is no fault to find at any single instant, and the finding exists only in the sequence. Then read stage_duration to see which gate got slower — a block-lock stage that has doubled means the code's evidence is being degraded by errors, which sends you to Chapter 3.7's pre-correction rate and then to Chapter 3.3's margin.
The method stated once: the gate names the owner, the elapsed time separates searching from stuck, furthest_ever decides whether to look for a change or for a misconfiguration, and the stage durations trended over weeks catch the degradation that every instantaneous reading reports as healthy.
15. Common Misconceptions
"A link either comes up or it does not."
The wrong model: bring-up is binary, so "link down" is a complete description of the problem.
What it costs: every investigation starts from scratch with no owner. You cannot tell a channel problem from a configuration problem from a physical path-length problem, and you check them in an arbitrary order.
The corrected model: bring-up is a ladder of six gates, each attemptable only once the one below it has succeeded. The stalled gate names the owner — LS_SIGNAL means a rate mismatch or signal quality, LS_CLOCKED means a coding mismatch, LS_MARKED means physical path lengths. One register read replaces a search.
"Bring-up time is a fixed property of the PHY."
The wrong model: a link takes as long as its datasheet says.
What it costs: you treat a slower bring-up as a defect and chase a fault that is not there. Or worse, you treat a bring-up that has gradually slowed from 3 ms to 300 ms as normal variation, and miss a genuinely degrading link.
The corrected model: bring-up time is a sum of search times, and searches depend on evidence quality and on the data being carried. Block lock on a 64b/66b link is structurally slower than on 4B/5B, because two illegal headers in four is far weaker evidence than seven illegal groups in thirty-two. And structured payload data makes wrong offsets survive longer than random data does.
"Regression during bring-up means something is broken."
The wrong model: a link that falls back a stage is failing.
What it costs: this is Section 12's rejected property. You assert against regression, it fires on working hardware, and either you distrust the assertion suite or you "fix" the design by preventing lock release — producing a link that false-locks permanently and delivers corrupted data with every gate reporting passed.
The corrected model: every gate is a search, and searches can commit wrongly and must recover. Regression is the recovery mechanism working. The finding is a rate — many regressions without reaching LS_UP, or regressions after the link has been up — never a single occurrence.
"Alignment markers are just a start-up preamble."
The wrong model: markers identify lanes once and are then overhead.
What it costs: you remove them after lock as a bandwidth optimisation, and lose continuous skew verification and bounded re-acquisition. A lane that loses block lock then forces the whole link back down the ladder instead of rejoining at the next marker.
The corrected model: markers are periodic and permanent, and they do three jobs — lane identity, a continuously re-checked timing reference for skew, and a bounded re-acquisition path. Skew drifts with temperature, so a one-time measurement made on cold equipment is worthless by afternoon.
"Link up means link ready."
The wrong model: once link_up asserts, the PHY's job is done and its state is uninteresting.
What it costs: you miss every gradual failure in this module. A link whose bring-up time has increased tenfold, or whose skew margin has halved, or which regresses forty times before succeeding, reports link_up at every instant anyone checks — and is weeks from failing.
The corrected model: link_up is instantaneous state. Health is in the durations and the counts — last_bringup_time trended, skew_margin trended, c_regressions per attempt. This is Chapter 3.3's slope-against-cliff argument at a third layer, and it applies to every gate on the ladder.
16. Interview Reasoning
"A link will not come up. Walk me through your diagnosis."
The weak answer lists layers. The answer that ends the topic names a first register read — the stalled gate — and then says what each possible answer would mean, because each names a different owner. The follow-up that separates real experience is furthest_ever: has this link ever worked? A link that once did has had something change; one that never has was never configured correctly, and those are opposite investigations from the same current state.
"Why would a link come up in 2 ms on the bench and 40 ms in production?"
Because block lock is a search, and search duration depends on how fast a wrong offset is rejected. Random bench data makes wrong offsets fail almost immediately. Production traffic is structured — repeating patterns, idles, compressed payloads — and structured data lets a wrong offset survive longer, sometimes long enough to false-lock and require recovery. No fault exists. Naming this unprompted signals someone who has brought up real links.
"What are alignment markers actually for?"
Three things, and naming only the first is the common incomplete answer: lane identity, so the receiver knows which physical lane carries which logical position; a timing reference, because the same marker enters every lane simultaneously so arrival differences are the skew; and bounded re-acquisition, because a lane that loses lock rejoins at the next marker instead of restarting the link. The strong follow-up is why they are periodic rather than a preamble — skew drifts with temperature, so the measurement must be continuous.
17. Understanding Check
Because each gate's precondition is the previous gate's result, and the reasons differ by rung:
| Gate | Assumes | Why the dependency is forced |
|---|---|---|
| clock lock | a signal exists | physical — timing is extracted from transitions, and no signal means no transitions |
| block lock | timing is known | definitional — slicing at candidate offsets presupposes bit boundaries |
| marker lock | blocks parse | structural — a marker is a block, so recognising one requires parsing blocks |
| deskew | every lane has marker lock | measurement — skew is the arrival-time difference of the same marker |
| codeword lock | lanes are aligned | structural — a codeword spans lanes and needs them in order |
Five dependencies, five different kinds of reason. That variety is why the ladder is worth understanding rather than memorising — each fails differently, and the diagnosis in Section 14 uses that.
The follow-up to be ready for: does every link have all six? No. Marker lock and deskew exist only because there are multiple lanes; a single-lane 10GBASE-R link has three gates. That is a real source of confusion — an engineer whose experience is single-lane expects the link up at block lock, and on a multi-lane link there are two more searches, the second of which cannot begin until every lane has finished the first.
18. What's Next
The claim this chapter defended: a link does not come up; it climbs a ladder of assumptions, and every rung is a search whose duration depends on the evidence available to it.
The ladder cannot be reordered, because each gate's precondition is the previous gate's result — and the five dependencies are forced for five different reasons: physical, definitional, structural, measurement, structural. Each loss falls back exactly one rung, to the state whose precondition still holds, because a sequencer returning to the bottom turns a microsecond re-lock into an outage.
And because every rung is a search, bring-up time is a sum of search times rather than a constant. Chapter 3.5 quantified why: block lock on a code with two illegal headers in four is structurally slower than on one with seven illegal patterns in thirty-two, and structured payload data makes every such search longer than random data does. That is the whole explanation for a link that comes up in 2 ms on the bench and 40 ms in production, and no fault is involved.
Module 3 is complete. Copper and fibre as channels, the analog channel and its two-axis margin, the sublayer split, the codes, the modulation, the correction, and now the sequence that brings them all up. One argument recurred at every layer and is worth carrying out of the module: measure what degrades continuously — margin, pre-correction error rate, skew margin, bring-up duration — and treat what degrades discontinuously as an alarm of last resort.
Chapter 4.1 — Where the MAC Ends and the PHY Begins opens Module 4 by returning to a boundary this module kept crossing. Chapter 3.4 approached it from below, as the top of the PHY, and gave the reconciliation sublayer and the media-independent interface. Module 4 approaches it from above, as the contract a MAC is designed against — and the question it has to answer is the one this module has been implicitly relying on throughout: given any responsibility, which side owns it, and what decides?
The full path is on the Ethernet curriculum index.
Continue learning
Related tutorials
- Related topic
The PHY Layer
Coding, serialisation, clock recovery and line drive all exist because a real channel attenuates, disperses and carries no clock. The PCS, PMA and PMD split follows the same logic — each owns one consequence of physics, and each changes on its own schedule.
- Related topic
Copper Ethernet
From 1000BASE-T onward every twisted pair carries both directions at once, so each receiver hears its own transmitter louder than the far end. Cancelling a known local signal is why BASE-T PHYs are adaptive signal-processing engines, and why pair count and signalling changed at every generation.
- Related topic
Fibre Ethernet
Separate strands per direction delete the echo problem that shapes copper PHYs, and introduce two others: a conversion boundary inside a pluggable module you do not own, and a link that can break in one direction while the far end still reports perfect health.
- Related topic
Differential Signalling and the Analog Channel
A PHY does not read bits off a wire — it infers symbols from a waveform the channel has attenuated, reflected and smeared into its neighbours. Differential signalling, impedance, jitter and the eye are one subject: what margin is left after the channel takes its share.
Standards & specifications
- Governing standard
- IEEE Std 802.3 (Ethernet)(opens IEEE in a new tab)
Defines the Ethernet MAC, the media-independent interfaces and the physical-layer sublayers, including framing, access control, auto-negotiation and per-rate PHY specifications. VLAN tagging, priority and time-sensitive shaping are defined by IEEE 802.1, not by 802.3.
This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.
Where this fits
Part of the Ethernet curriculum.
