PCIe · Module 18
Configuration — Lanes Become a Link
Trained lanes are not yet a Link. Configuration decides which of them belong together, what their logical identities are, and how wide the result is — and nothing above may see any of it until it is committed atomically.
Chapter 18.2 established that somebody is there. Chapter 18.3 established that the two ports can read each other.
Neither established a Link.
What exists at this point is a set of lanes that individually work. A Link is more than that: it is an agreement about which lanes belong together, what each one is called, in what order they carry data, and how wide the result is — agreed by both ends, because a receiver that recombines lanes in a different order than the transmitter split them produces data that was never sent (Chapter 17.2 §7).
How do two adjacent ports turn trained physical lanes into one coherent logical Link, and when does a discovery become a commitment?
1. The Verified Substates
2. Configuration Is Not Configuration Space
3. Link Number Is Not Bus Number
A second collision, and the same shape.
The Link Number is negotiated during Configuration between two adjacent ports (§1), lives in a 9-bit field, and identifies the Link's configuration context at the Physical Layer.
A Bus Number is assigned by software during enumeration (Chapter 7.4), identifies a bus in the hierarchy, and is written into Configuration Space registers.
Different layers, different agents, different lifetimes, and no relationship. A Link Number is settled by hardware before any software runs; a Bus Number is assigned by software long afterwards, over the Link that the Link Number helped establish.
The practical consequence: a training-time Link Number mismatch is a physical-layer problem between two ports. Reassigning bus numbers cannot affect it, and looking at the enumeration code is looking at software that has not started.
4. The Two Ends Are Not Symmetric
§1's substate table shows separate Upstream and Downstream paths through the same canonical substates — and that asymmetry is real.
Downstream Port (toward the Endpoint) Upstream Port (toward the Root)
CONFIGURATION_DS_LW_START CONFIGURATION_US_LW_START
CONFIGURATION_DS_LW_ACCEPT CONFIGURATION_US_LW_ACCEPT
CONFIGURATION_DS_LN_WAIT CONFIGURATION_US_LN_WAIT
CONFIGURATION_DS_LN_ACCEPT CONFIGURATION_US_LN_ACCEPT
both converge at
CONFIGURATION_COMPLETE → CONFIGURATION_IDLE5. Lane Number Is Not Pad Index
A physical lane is a pair of pins. A logical lane is a position in the Link's ordering. They are not the same, and assuming they are produces a design that works on some boards.
physical lane (pins, routing)
→ training
→ logical lane number (position within this Link)Why they can differ. Board layout does not always permit routing physical lane 0 to the position that becomes logical lane 0. Crossing traces to make the numbering match costs layers, area and signal integrity — so the protocol allows the mapping to be established during training instead.
Lane reversal is the case where the ordering is inverted end to end — physical lane 7 becoming logical lane 0, and so on. §1's source lists Polarity Inversion among its features and tracks lane numbering separately, and this chapter teaches the concept while declining the detection algorithm (§1).
6. Three Different Widths
The word "width" means three different things at three stages, and conflating them is where most Configuration confusion comes from.
| Width | Meaning | Established by |
|---|---|---|
| Connector / routed | how many lanes physically exist | the board |
| Candidate | how many have a receiver, and train | 18.2, 18.3 |
| Configured | how many form the operating Link | this chapter |
Each is bounded by the one above and can be smaller. An x16 slot with an x8 card routed to eight lanes has a routed width of 8. If two of those lanes fail to train, the candidate set is 6. And 6 is not a Link width — §7.
A worked example, computed (§18):
routed x8 lanes 0-7 present on the board
candidate 0x3F lanes 0-5 detected and trained (6 lanes)
configured x4 lanes 0-3Two lanes that work perfectly are not used. Not because anything is wrong with them, but because x6 is not a Link width, and the largest legal width the usable set supports is x4.
7. Width Is Not Popcount
One scoping note. The contiguous-from-zero policy above is this chapter's illustrative normalization, consistent with Chapter 17.3 §11. The real grouping rules are specification detail (§1), and §14's classifier therefore takes an already-normalized logical lane mask as input — its job is to recognise legal width classes, not to invent the normalization.
8. Provisional Is Not Committed
The chapter's flagship idea, and the one with the highest cost of failure.
During Configuration, everything discovered is provisional. Width can shrink as lanes are rejected. A lane map under construction is incomplete. A Link Number is proposed before it is agreed.
§1's source pins the boundary exactly: the Link Number "changes to the negotiated Link Number… when the LTSSM changes from CFG_COMPLETE to CFG_IDLE." The result becomes real at a specific transition, not gradually as it is discovered.
9. The Configuration Subtree
Three things to read out of the figure.
The order is width, then lane numbers. The canonical substate names say so — Linkwidth before Lanenum — and it makes sense: how many lanes participate is a smaller question than which one is which, and the second is only meaningful once the first is settled.
Complete and Idle are two states, not one. §1's source shows the negotiated Link Number taking its final value at the transition between them — that edge is the commit (§8), and the diagram labels it as such.
And failure returns toward Detect. Same principle as Chapters 18.2 §6 and 18.3 §8: a state that cannot establish what it needs goes back rather than proceeding on an unfounded configuration.
What is not drawn: the Upstream/Downstream split (§4), every acceptance criterion, and all timeouts. This is the Configuration subtree's phase topology — Chapter 18.1 §6 owns the full map.
10. A Trace
Internal teaching signals, not PCIe pins. Four lanes, one duplicate arriving late.
step 1 2 3 4 5 6 7 8 9
obs_valid 0 1 1 1 1 1 0 0 0
obs_lane - 0 1 2 1 3 - - -
obs_link - 0A 0A 0A 0A 0A - - -
cand_mask 00 01 03 07 07 0F 0F 0F 0F
dup_error 0 0 0 0 1 1 1 1 1
cand_consistent 1 1 1 1 0 0 0 0 0
commit_req 0 0 0 0 0 0 0 1 0
config_valid 0 0 0 0 0 0 0 0 0
active_width - - - - - - - - -
l0_handoff 0 0 0 0 0 0 0 0 0Read steps 2–4. Three lane observations, all distinct, all with the same Link number. cand_mask accumulates one bit at a time and cand_consistent stays high.
Read step 5 — the duplicate. Lane 1 is claimed a second time. dup_error sets and cand_consistent goes low, and it stays low: consistency is not a per-observation verdict but a property of the whole set (§13).
Read step 6. Lane 3 arrives and the mask reaches 0x0F — which would be a perfectly good x4 if the set were consistent. It is not.
Read step 8 — the commit that does not happen. commit_req asserts and config_valid stays low. An inconsistent candidate cannot be committed (§15, P7), so nothing is exposed upward and l0_handoff never rises.
And note active_width across the whole trace. It never takes a value. A candidate is not a configuration (§8), and a design that had exposed the provisional width would have shown x1, then x2, then x4 as observations arrived — three configurations that were never true.
11. RTL — Normalized Configuration Types
// SYNTHESIZABLE. Normalized Configuration types.
// LEGAL LINK WIDTHS are canonical: x1, x2, x4, x8, x16. Everything else
// here -- field widths, the contiguity policy, the encoding -- is this
// chapter's internal normalization (section 1).
package config_pkg;
parameter int MAX_LANES = 16;
// GUARDED. Never zero-width, even at MAX_LANES == 1.
parameter int LANE_W = (MAX_LANES <= 1) ? 1 : $clog2(MAX_LANES);
parameter int LINK_W = 9; // section 1: the vendor link_num field is 9 bits
parameter int WIDTH_W = 5; // holds 0..16
typedef struct packed {
logic valid;
logic [LANE_W-1:0] lane_num; // LOGICAL lane number, not a pad index
logic [LINK_W-1:0] link_num; // NOT a bus number (section 3)
} cfg_obs_t;
// THE COMMITTED CONFIGURATION. One struct, so section 15 can assign it
// in a single nonblocking statement and partial commit is unconstructable.
typedef struct packed {
logic valid;
logic [WIDTH_W-1:0] width; // 0, 1, 2, 4, 8 or 16
logic [MAX_LANES-1:0] lane_mask; // exactly `width` bits set
logic [LINK_W-1:0] link_num;
} link_config_t;
function automatic bit is_legal_width(input logic [WIDTH_W-1:0] w);
return (w inside {5'd1, 5'd2, 5'd4, 5'd8, 5'd16});
endfunction
endpackage12. RTL — Candidate Lane Collector
import config_pkg::*;
// SYNTHESIZABLE. Accumulate the set of logical lanes claimed during this
// Configuration attempt.
// THIS BUILDS A CANDIDATE, NOT A CONFIGURATION (section 8). Nothing here
// is visible above the Physical Layer.
module cfg_candidate_collect (
input logic clk,
input logic rst_n,
input logic state_entry, // entering Configuration: start over
input cfg_obs_t obs,
output logic [MAX_LANES-1:0] cand_mask,
output logic [LINK_W-1:0] cand_link,
output logic cand_link_seen
);
logic [MAX_LANES-1:0] mask_q;
logic [LINK_W-1:0] link_q;
logic seen_q;
assign cand_mask = mask_q;
assign cand_link = link_q;
assign cand_link_seen = seen_q;
wire in_range = (MAX_LANES == 1) ? (obs.lane_num == '0)
: (obs.lane_num < LANE_W'(MAX_LANES));
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
mask_q <= '0; link_q <= '0; seen_q <= 1'b0;
end else if (state_entry) begin
// ==============================================================
// CHECKED FIRST. A candidate set inherited from a previous attempt
// would let lanes that trained under different conditions join this
// Link -- Chapter 18.3 section 12's rule, one layer up.
// ==============================================================
mask_q <= '0; link_q <= '0; seen_q <= 1'b0;
end else if (obs.valid && in_range) begin
// ONE BIT PER OBSERVATION. Setting a range of bits from a width
// would assume contiguity that has not been established yet --
// that is section 14's job, on the finished set.
mask_q[obs.lane_num] <= 1'b1;
if (!seen_q) begin
link_q <= obs.link_num;
seen_q <= 1'b1;
end
end
end
endmoduleClassification: synthesizable.
Architecture. A set accumulator with entry clearing checked first and one bit written per observation.
The first Link number is latched, not overwritten. Later observations are compared against it by §13 rather than replacing it — a design that overwrote would silently adopt whichever partner spoke last and never notice a disagreement.
Failure — four. No entry clear carries a previous attempt's lanes into this one. Setting a range of bits assumes contiguity before it is checked. Overwriting the Link number hides a mismatch. And omitting the range check writes outside the mask.
13. RTL — Duplicate and Range Checker
import config_pkg::*;
// SYNTHESIZABLE. Detect a candidate set that cannot describe a Link.
// TWO LANES CLAIMING THE SAME LOGICAL NUMBER is not a width problem and
// not recoverable by choosing a smaller width -- the set is incoherent,
// and section 15 refuses to commit it at all.
module cfg_consistency (
input logic clk,
input logic rst_n,
input logic state_entry,
input cfg_obs_t obs,
input logic [MAX_LANES-1:0] cand_mask, // set BEFORE this observation
input logic [LINK_W-1:0] cand_link,
input logic cand_link_seen,
output logic dup_error,
output logic range_error,
output logic link_mismatch,
output logic cand_consistent
);
logic dup_q, rng_q, lnk_q;
assign dup_error = dup_q;
assign range_error = rng_q;
assign link_mismatch = lnk_q;
// CONSISTENCY IS A PROPERTY OF THE WHOLE SET, not a verdict on the
// latest observation -- so these are STICKY until the next attempt.
// A design that recomputed it per observation would report the set
// consistent again as soon as a clean observation arrived after a bad
// one (section 18, mutation 5).
assign cand_consistent = !dup_q && !rng_q && !lnk_q;
wire in_range = (MAX_LANES == 1) ? (obs.lane_num == '0)
: (obs.lane_num < LANE_W'(MAX_LANES));
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
dup_q <= 1'b0; rng_q <= 1'b0; lnk_q <= 1'b0;
end else if (state_entry) begin
// Cleared only on a NEW ATTEMPT. Checked first.
dup_q <= 1'b0; rng_q <= 1'b0; lnk_q <= 1'b0;
end else if (obs.valid) begin
if (!in_range) begin
rng_q <= 1'b1;
end else begin
// THE DUPLICATE TEST. Note it reads cand_mask BEFORE this
// observation is applied -- comparing against the post-update
// mask would find every observation to be a duplicate of itself.
if (cand_mask[obs.lane_num]) dup_q <= 1'b1;
end
// A SECOND LINK NUMBER means two different negotiations are being
// observed on one candidate set. Not a width problem either.
if (cand_link_seen && (obs.link_num != cand_link)) lnk_q <= 1'b1;
end
end
endmoduleClassification: synthesizable.
Architecture. Three sticky error flags, cleared only on a new attempt.
Three distinct errors kept separate, because they mean different things: a duplicate means two lanes claim one position, a range error means a malformed identity, and a Link-number mismatch means two negotiations are being conflated. Merged into one "bad" bit, §19 loses three diagnoses.
Why sticky. Consistency describes the set, so a clean observation after a bad one does not repair it — a recomputing implementation would declare the set consistent again and commit an incoherent lane map.
And the duplicate test reads the pre-update mask. Against the post-update mask every observation is its own duplicate, and no configuration ever commits.
Executed (§18): the collector and checker were run over lane sequences — [0,1,2,3] gives mask 0x0F with no error; [0,1,1,3] sets dup_error with mask 0x0B; [0,1,2,9] sets range_error; and [3,2,1,0] — arriving in reverse order — is clean, mask 0x0F. That last case is the point of §5: reversed order is not an error, it is a lane map.
14. RTL — Legal Width Classifier
import config_pkg::*;
// SYNTHESIZABLE. Reduce a candidate lane set to a LEGAL Link width.
// WIDTH IS NOT POPCOUNT (section 7). This is combinational and pure --
// it holds no state, so it can be reasoned about and exhaustively checked
// as a function, which section 18 did over all 65,536 masks.
module cfg_width_class (
input logic [MAX_LANES-1:0] cand_mask,
output logic [WIDTH_W-1:0] width,
output logic [MAX_LANES-1:0] width_mask
);
// Step 1: the CONTIGUOUS RUN FROM LANE 0.
// A gap terminates the run. Lanes beyond a gap are good lanes that this
// Link cannot use (section 7) -- 0x000D has three good lanes and a run
// of one.
logic [MAX_LANES:0] run_ok;
logic [WIDTH_W-1:0] run_len;
always_comb begin
run_ok[0] = 1'b1;
for (int i = 0; i < MAX_LANES; i++)
run_ok[i+1] = run_ok[i] && cand_mask[i];
run_len = '0;
for (int i = 0; i < MAX_LANES; i++)
if (run_ok[i+1]) run_len = WIDTH_W'(i + 1);
end
// Step 2: REDUCE THE RUN TO A LEGAL WIDTH, largest first.
// A run of 6 is not x6. It is x4, and two working lanes go unused.
always_comb begin
if (run_len >= 5'd16) width = 5'd16;
else if (run_len >= 5'd8) width = 5'd8;
else if (run_len >= 5'd4) width = 5'd4;
else if (run_len >= 5'd2) width = 5'd2;
else if (run_len >= 5'd1) width = 5'd1;
else width = 5'd0; // no usable width
end
// Step 3: the mask of exactly the lanes the Link will use.
// BUILT FROM THE WIDTH, never passed through from cand_mask -- so the
// two can never disagree, and Chapter 17.1's distributor cannot be
// handed a lane outside the configured width.
always_comb begin
width_mask = '0;
for (int i = 0; i < MAX_LANES; i++)
if (WIDTH_W'(i) < width) width_mask[i] = 1'b1;
end
endmoduleClassification: synthesizable, purely combinational.
Architecture. Run, then reduce, then rebuild the mask from the width.
Step 3 is the one that prevents a class of bug outright. Passing cand_mask through would let the mask contain lanes outside the configured width — exactly the input Chapter 17.1 §10's distributor must never receive. Rebuilding from the width makes width and mask agree by construction, and §18 verified it: across all 65,536 masks, every width_mask is a subset of cand_mask with a popcount equal to width.
Failure — four. $countones is wrong for 65,530 of 65,536 masks (§7). Not requiring lane 0 accepts 0xFF00 as x8 when the run is empty. Passing the candidate mask through decouples mask from width. And reducing smallest-first returns x1 for a perfectly good x16.
15. RTL — Provisional Register and Atomic Commit
import config_pkg::*;
// SYNTHESIZABLE. The commit boundary (section 8).
// EVERYTHING ABOVE THE PHYSICAL LAYER READS ONLY `cfg` -- the provisional
// values are not brought out. Chapter 17.1's lane distributor and Chapter
// 17.2's recombiner consume this struct, and a half-updated version makes
// them disagree about the shape of the Link.
module cfg_commit (
input logic clk,
input logic rst_n,
input logic state_entry,
// ---- Provisional, from sections 12-14 ---------------------------------
input logic [WIDTH_W-1:0] prov_width,
input logic [MAX_LANES-1:0] prov_mask,
input logic [LINK_W-1:0] prov_link,
input logic prov_link_seen,
input logic cand_consistent,
input logic commit_req, // the Complete -> Idle edge (section 1)
// ---- Committed. The ONLY outputs anything above may use. --------------
output link_config_t cfg,
output logic commit_rejected
);
link_config_t cfg_q;
logic rej_q;
assign cfg = cfg_q;
assign commit_rejected = rej_q;
// ==================================================================
// THE COMMIT GUARD. Every condition must hold; there is no arm that
// commits a partial or incoherent result.
//
// consistent -- no duplicate, no range error, no link mismatch (13)
// legal width -- x0 is not a Link (section 14 returns 0 for an
// unusable set, and that must be refused, not
// committed as a zero-wide Link)
// link seen -- a Link number was actually observed
// ==================================================================
wire commit_ok = cand_consistent
&& is_legal_width(prov_width)
&& prov_link_seen;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
cfg_q <= '0; rej_q <= 1'b0;
end else if (state_entry) begin
// ENTERING CONFIGURATION INVALIDATES THE COMMITTED CONFIGURATION.
// Checked first. Configuration is reached from Polling, so there is
// no operational Link to preserve -- there is nothing above that may
// legally act on a configuration right now, and a stale width left
// exposed could only mislead.
//
// RECOVERY IS THE OPPOSITE CASE and must not copy this arm. A Link
// entering Recovery already HAS a coherent operational configuration
// that stays active until a new one commits (Chapter 18.5). Clearing
// on entry there would tear down a working configuration to
// renegotiate one that may end up identical.
cfg_q <= '0; rej_q <= 1'b0;
end else if (commit_req) begin
if (commit_ok) begin
// ============================================================
// ONE NONBLOCKING ASSIGNMENT TO ONE PACKED STRUCT.
//
// width, lane_mask, link_num and valid become visible in the SAME
// CYCLE. There is no ordering between them because there is no
// sequence -- partial commit is not a bug to be avoided here, it
// is UNCONSTRUCTABLE (section 8).
// ============================================================
cfg_q <= '{ valid : 1'b1,
width : prov_width,
lane_mask : prov_mask,
link_num : prov_link };
end else begin
// A REFUSED COMMIT LEAVES NOTHING EXPOSED and says so. Committing
// a "best effort" configuration would hand the distributor a lane
// map that the receiver is not using.
cfg_q <= '0;
rej_q <= 1'b1;
end
end
end
endmoduleClassification: synthesizable.
Architecture. One packed struct, one assignment, guarded.
The atomicity is structural, not procedural. Nothing sequences the fields because they are one object — so no future edit can introduce an ordering, and no review has to check for one.
And an unusable set is refused, not committed as x0. cfg_width_class returns width 0 when there is no run from lane 0; is_legal_width rejects it. A zero-wide committed Link would satisfy "valid" and carry nothing — the worst possible outcome, because everything above would proceed.
Failure — five. Separate registers for width and mask permit the §18 counterexample. Committing on commit_req unconditionally exposes an incoherent map. Accepting width 0 produces a valid zero-wide Link. Exposing the provisional values lets consumers act on a configuration under construction. And no entry invalidation presents a stale configuration during renegotiation.
16. RTL — Configuration Progress Controller
import config_pkg::*;
// SYNTHESIZABLE. Progress within Configuration only.
// THIS IS NOT THE LTSSM (Chapter 18.1 section 9), and it is not Chapter
// 18.5's Recovery -- speed changes, Up/Down-configure and autonomous width
// changes are NOT modelled here (section 1). It sequences one thing:
// collect, check, classify, commit, hand off.
module cfg_progress #(
parameter int CNT_W = 24
) (
input logic clk,
input logic rst_n,
input logic state_entry,
input logic cand_consistent,
input logic prov_link_seen,
input logic [WIDTH_W-1:0] prov_width,
input logic settle_done, // enough observation to stop collecting
input logic timeout_elapsed,
input link_config_t cfg,
output logic commit_req,
output logic config_done,
output logic config_fail,
output logic l0_handoff
);
typedef enum logic [2:0] {
S_WIDTH, S_LANENUM, S_COMPLETE, S_IDLE, S_FAIL
} st_e;
st_e st_q;
// COMMIT IS AN EDGE, not a level -- section 1 places it at the
// Complete -> Idle transition. A level would re-commit every cycle.
assign commit_req = (st_q == S_COMPLETE) && settle_done;
assign config_done = (st_q == S_IDLE);
assign config_fail = (st_q == S_FAIL);
// ==================================================================
// THE HANDOFF IS GATED ON THE COMMITTED STRUCT, not on reaching Idle.
//
// Reading `cfg.valid` rather than the state means the Link cannot be
// handed to normal operation unless a configuration was actually
// committed -- so a refused commit (section 15) cannot be followed by
// traffic (section 18, mutation 12).
// ==================================================================
assign l0_handoff = (st_q == S_IDLE) && cfg.valid;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
st_q <= S_WIDTH;
end else if (state_entry) begin
st_q <= S_WIDTH;
end else begin
unique case (st_q)
S_WIDTH :
// A CANDIDATE THAT IS ALREADY INCOHERENT FAILS IMMEDIATELY.
// Continuing to collect lanes onto a set with a duplicate wastes
// the rest of the timeout for a result that cannot commit.
if (!cand_consistent) st_q <= S_FAIL;
else if (timeout_elapsed) st_q <= S_FAIL;
else if (settle_done && is_legal_width(prov_width))
st_q <= S_LANENUM;
S_LANENUM :
if (!cand_consistent) st_q <= S_FAIL;
else if (timeout_elapsed) st_q <= S_FAIL;
else if (prov_link_seen) st_q <= S_COMPLETE;
S_COMPLETE :
// The commit fires on the way out, and the result is checked on
// arrival: if it was refused, Idle exposes nothing and the
// handoff does not happen.
if (!cand_consistent) st_q <= S_FAIL;
else if (settle_done) st_q <= S_IDLE;
else if (timeout_elapsed) st_q <= S_FAIL;
S_IDLE, S_FAIL : ;
default : st_q <= S_WIDTH;
endcase
end
end
endmoduleClassification: synthesizable (Configuration only — not the LTSSM, and not Recovery).
Architecture. Width, then lane numbers, then complete, then idle — §9's phase order, with a fail path.
Two decisions worth naming. commit_req is an edge at the Complete → Idle boundary, matching §1's evidence; a level would re-commit every cycle and defeat the point of a commit. And l0_handoff reads cfg.valid, not the state — so a refused commit cannot be followed by traffic, which a state-only gate would permit.
Failure — four. Committing on a level. Gating the handoff on S_IDLE alone. Continuing to collect onto an already-incoherent set. And treating a width the classifier rejected as progress.
17. Assertions
// SVA over the Configuration blocks. These assert the LOCAL contract:
// what may be committed, and what the committed result must satisfy.
// They assert NOTHING about the partner agreeing, about any particular
// width being achievable, or about Configuration succeeding.
// ---- ENVIRONMENT ------------------------------------------------------
assume property (@(posedge clk) disable iff (!rst_n)
obs.valid |-> !$isunknown({obs.lane_num, obs.link_num}));
assume property (@(posedge clk) disable iff (!rst_n)
state_entry |=> !state_entry);
// ---- THE CANDIDATE ----------------------------------------------------
// P1: entering Configuration starts from an empty candidate -- lanes from
// a previous attempt must not join this Link.
property p_entry_clears;
@(posedge clk) disable iff (!rst_n)
state_entry |=> ((cand_mask == '0) && cand_consistent && !cfg.valid);
endproperty
a_entry : assert property (p_entry_clears);
// P2: a lane bit is set ONLY by an in-range observation of that lane.
property p_mask_growth;
@(posedge clk) disable iff (!rst_n)
((cand_mask & ~$past(cand_mask)) != '0)
|-> ($past(obs.valid) && (cand_mask & ~$past(cand_mask))
== (MAX_LANES'(1) << $past(obs.lane_num)));
endproperty
a_growth : assert property (p_mask_growth);
// P3: A DUPLICATE LOGICAL LANE IS DETECTED. Two lanes cannot occupy one
// position (section 13).
property p_dup_detected;
@(posedge clk) disable iff (!rst_n)
(obs.valid && cand_mask[obs.lane_num]) |=> dup_error;
endproperty
a_dup : assert property (p_dup_detected);
// P4: inconsistency is STICKY until the next attempt -- a clean
// observation after a bad one does not repair the set.
property p_sticky;
@(posedge clk) disable iff (!rst_n)
(!cand_consistent && !state_entry) |=> !cand_consistent;
endproperty
a_sticky : assert property (p_sticky);
// ---- THE WIDTH --------------------------------------------------------
// P5: THE COMMITTED WIDTH IS ALWAYS LEGAL. x6 is not a Link.
property p_legal_width;
@(posedge clk) disable iff (!rst_n)
cfg.valid |-> is_legal_width(cfg.width);
endproperty
a_legal : assert property (p_legal_width);
// P6: WIDTH AND MASK AGREE, and the mask uses only candidate lanes.
// The property that keeps Chapter 17.1's distributor safe.
property p_mask_matches_width;
@(posedge clk) disable iff (!rst_n)
cfg.valid |-> (($countones(cfg.lane_mask) == cfg.width)
&& ((cfg.lane_mask & ~cand_mask) == '0));
endproperty
a_match : assert property (p_mask_matches_width);
// ---- THE COMMIT -------------------------------------------------------
// P7: AN INCONSISTENT CANDIDATE IS NEVER COMMITTED.
property p_no_bad_commit;
@(posedge clk) disable iff (!rst_n)
$rose(cfg.valid) |-> $past(cand_consistent);
endproperty
a_guard : assert property (p_no_bad_commit);
// P8: ATOMICITY. Every field of the committed configuration changes in the
// SAME cycle -- no field may move while the others are stable.
// The counterexample in section 18 is exactly the violation of this.
property p_atomic;
@(posedge clk) disable iff (!rst_n)
($changed(cfg.width) || $changed(cfg.lane_mask) || $changed(cfg.link_num))
|-> $changed(cfg.valid) || $rose(state_entry) || $rose(commit_req);
endproperty
a_atomic : assert property (p_atomic);
// P9: the committed configuration is STABLE between commits -- consumers
// may not see it drift.
property p_stable;
@(posedge clk) disable iff (!rst_n)
(cfg.valid && !commit_req && !state_entry) |=> $stable(cfg);
endproperty
a_stable : assert property (p_stable);
// P10: A REFUSED COMMIT EXPOSES NOTHING and reports itself.
property p_reject;
@(posedge clk) disable iff (!rst_n)
(commit_req && !cand_consistent) |=> (!cfg.valid && commit_rejected);
endproperty
a_reject : assert property (p_reject);
// P11: a zero width is never committed as a valid Link.
property p_no_zero_link;
@(posedge clk) disable iff (!rst_n)
cfg.valid |-> (cfg.width != '0);
endproperty
a_nonzero : assert property (p_no_zero_link);
// ---- THE HANDOFF ------------------------------------------------------
// P12: NO HANDOFF TO NORMAL OPERATION WITHOUT A COMMITTED CONFIGURATION.
// The property that connects this chapter to Chapter 18.6.
property p_handoff_needs_commit;
@(posedge clk) disable iff (!rst_n)
l0_handoff |-> cfg.valid;
endproperty
a_handoff : assert property (p_handoff_needs_commit);
// P13: NO NORMAL TRAFFIC DURING CONFIGURATION. Bound against Chapter
// 18.1's gate -- the Link is not a Link yet.
property p_no_traffic;
@(posedge clk) disable iff (!rst_n)
(dut_ltssm.cur_state == ST_CONFIGURATION) |-> !dut_gate.traffic_enabled;
endproperty
a_no_traffic : assert property (p_no_traffic);
// P14: the provisional values never reach a consumer -- structurally, the
// only configuration port is `cfg`.
property p_consumer_uses_committed;
@(posedge clk) disable iff (!rst_n)
dut_distributor.active_mask == (cfg.valid ? cfg.lane_mask : '0);
endproperty
a_consumer : assert property (p_consumer_uses_committed);
// P15: reset.
property p_reset;
@(posedge clk)
!rst_n |=> (!cfg.valid && (cand_mask == '0) && !l0_handoff);
endproperty
a_reset : assert property (p_reset);P6 and P8 are the two that matter most. P6 is what makes Chapter 17.1's distributor safe — width and mask cannot disagree, and the mask cannot contain a lane that was never a candidate. P8 forbids any field moving alone, which is the partial-commit failure stated as a property.
P11 exists because "valid and zero-wide" is the worst outcome. Everything above proceeds on a Link that carries nothing, and no error is reported anywhere.
No liveness. "Configuration eventually completes" depends on a partner, and asserting it would assume away the failure this state exists to detect.
18. Verification, Fault Injection, and Model Verification
Every number in this chapter was executed before publication.
The width classifier — exhaustive
All 65,536 lane masks at MAX_LANES = 16, checked against three independent invariants:
- every width is in 16
- every
width_maskis a subset ofcand_mask - every
width_maskhas a popcount equal towidth
Zero violations.
And the distribution, which is worth seeing:
| Result | Masks | Of 65,536 |
|---|---|---|
| x16 | 1 | 0.002% |
| x8 | 255 | 0.4% |
| x4 | 3,840 | 5.9% |
| x2 | 12,288 | 18.8% |
| x1 | 16,384 | 25.0% |
| no usable width | 32,768 | 50.0% |
Exactly half of all masks yield nothing, because exactly half have lane 0 absent — which is the contiguity policy made visible.
The popcount comparison (§7): popcount equals the legal width for 6 masks out of 65,536 — 0x0000, 0x0001, 0x0003, 0x000F, 0x00FF, 0xFFFF, and nothing else.
The quoted cases, computed:
cand_mask | run | width | width_mask | |
|---|---|---|---|---|
0x00FF | 8 | x8 | 0x00FF | contiguous |
0x001F | 5 | x4 | 0x000F | 5 good lanes, one unused |
0x000D | 1 | x1 | 0x0001 | gap at lane 1 |
0xFF00 | 0 | none | 0x0000 | 8 good lanes, no usable Link |
0xFFFF | 16 | x16 | 0xFFFF |
The collector and checker — executed
| Observation order | Result |
|---|---|
[0,1,2,3] | mask 0x0F, no error — clean x4 |
[0,1,1,3] | dup_error, mask 0x0B |
[0,1,2,9] | range_error |
[3,2,1,0] | mask 0x0F, no error — reversed order is a lane map, not a fault |
Directed tests
- Every legal width — x1, x2, x4, x8, x16 — committed and read back (P5, P6).
MAX_LANES= 1, 2, 4, 8, 16. TheLANE_Wguard is exercised at 1.- Candidate with a gap (
0x000D) — verify x1, not x3. - Candidate without lane 0 (
0xFF00) — verify no commit (P11). - Duplicate at the last observation — verify the whole set is refused (P4, P7).
- Two different Link numbers — verify
link_mismatchand refusal. commit_reqwith an inconsistent candidate — verify nothing is exposed (P10). Required.- Re-entry mid-negotiation — verify the committed configuration is invalidated (P1).
- Reset during Complete (P15).
Mutations
| # | Mutation | Caught by | Lab symptom |
|---|---|---|---|
| 1 | width = $countones(cand_mask) | P5 | x6 committed; distributor drives six lanes the receiver is not recombining |
| 2 | width_mask passed through from cand_mask | P6 | mask and width disagree; lanes outside the width driven |
| 3 | run not anchored at lane 0 | P6 | 0xFF00 committed as x8; no data ever arrives |
| 4 | duplicate test against the post-update mask | P3 | every observation is a duplicate; nothing ever commits |
| 5 | consistency recomputed per observation | P4 | a clean observation after a duplicate commits an incoherent map |
| 6 | no entry clear on the candidate | P1 | lanes from a previous attempt join this Link |
| 7 | width and mask in separate registers | P8 | the counterexample below |
| 8 | commit on a level rather than the edge | P9 | configuration re-commits every cycle; consumers see churn |
| 9 | width 0 committed as valid | P11 | a valid zero-wide Link; everything above proceeds and nothing moves |
| 10 | Link number overwritten by each observation | mismatch check | two negotiations conflated silently |
| 11 | provisional width exposed to the distributor | P14 | width changes as observations arrive |
| 12 | handoff gated on S_IDLE alone | P12 | traffic after a refused commit |
| 13 | traffic enabled during Configuration | P13 | packets into a Link that does not exist |
| 14 | reset leaves the committed configuration | P15 | a stale width survives reset |
| 15 | lane reversal treated as a duplicate | collector test | valid reversed maps rejected; works on one board, fails on another |
19. Debugging
Symptom → which quantity is wrong → signal → distinguishing experiment.
The Link trains but comes up narrower than expected
The most common Configuration observation, and it is usually not a fault.
Read the three widths in order (§6):
- Routed width — the board. An x16 slot does not mean sixteen routed lanes.
- Candidate mask — which lanes detected and trained (18.2, 18.3).
- Configured width — what §14 made of the candidate set.
The single most informative reading is cand_mask against cfg.width. A candidate of 0x001F — five good lanes — configuring as x4 is correct behaviour: x5 is not a Link width, and one working lane goes unused (§7).
But a candidate of 0x000D configuring as x1 points at lane 1, which trained on neither end. One lane's problem, costing three lanes of bandwidth — because a gap truncates the run.
And a candidate of 0xFF00 configuring as nothing is the striking one: eight good lanes and no Link at all, because lane 0 is absent.
It reaches Configuration and returns to Detect
Read the three consistency flags separately (§13) — this is why they are not merged:
| Flag | Meaning | Where to look |
|---|---|---|
dup_error | two lanes claim one logical number | lane mapping, board routing, reversal handling |
range_error | a malformed lane identity | the parser, or the partner |
link_mismatch | two Link numbers on one candidate set | two negotiations conflated — check topology |
One bit would have said "Configuration failed" and left all three open.
Data corruption immediately after the Link comes up
Suspect the commit, not the channel (§18's counterexample).
The signature is specific: LCRC errors (Chapter 14.3) with no PHY errors and no bit errors — a channel that is measurably clean and a Data Link Layer that keeps rejecting packets. Replay usually succeeds, which is what makes it look transient.
The distinguishing experiment: capture cfg.width and cfg.lane_mask in the same cycle and check $countones(lane_mask) == width continuously, not once. A single cycle of disagreement is the whole bug — and P6 with P8 is that check made permanent.
Works on one board, fails on another, same design
Suspect lane mapping (§5).
The distinguishing experiment: transmit a distinct pattern per lane and observe which lane receives which.
- All lanes correct but in reversed order → lane reversal, and the mapping logic is the target.
- One lane's data inverted, order correct → polarity inversion on that lane, a PHY matter (§5).
- Correct on some lanes, absent on others → a training or routing problem, not a mapping one.
Three different investigations, separated in one run — and a merged "data is wrong" observation separates none of them.
It is stuck and someone is reading enumeration code
Stop (§2).
If the LTSSM has not reached normal operation, no software explanation is possible, because no TLP can have been issued. Configuration Space is unreachable by definition, and time spent on the enumeration path is time not spent on the Link.
20. Common Misconceptions
- "Configuration means Configuration Space." The most expensive collision in PCIe (§2).
- "Configuration is done by software." Hardware, two adjacent ports, before software runs.
- "Link Number is Bus Number." Different layers, agents and lifetimes (§3).
- "Lane number equals the physical pad index." Logical position, established during training (§5).
- "Lane reversal and polarity inversion are the same." Across lanes versus within a pair (§5).
- "Width equals the number of working lanes." Correct for 6 masks out of 65,536 (§7).
- "Six good lanes give x6." x6 is not a Link width; x4, with two unused (§6).
- "Eight good lanes always give x8." Not if lane 0 is absent —
0xFF00gives nothing (§18). - "The width is known as soon as lanes train." Provisional until committed (§8).
- "Committing the width and the map one after the other is fine." §18's counterexample.
- "An x16 slot means an x16 Link." Routed, candidate and configured are three different widths (§6).
- "Both ends play the same part." The Upstream/Downstream split is structural (§4).
- "Configuration negotiates the link speed." Speed changes belong to Chapter 18.5 — Recovery.
- "A narrower-than-expected Link is a bug." Usually the correct reduction of an imperfect candidate set (§19).
- "Reaching Configuration.Idle means traffic can flow." Only if a configuration was actually committed (P12).
21. Understanding Check
22. What's Next
Configuration is where lanes become a Link.
Three quantities are established — which lanes belong, what their logical identities are, and how wide the result is — plus a Link identity that both ends agree on, and §1's source pins where that becomes final: the Configuration.Complete → Configuration.Idle transition.
The arithmetic is not what it looks like. Width is not popcount — that equivalence holds for 6 of 65,536 lane masks. A gap truncates the run; a missing lane 0 makes eight good lanes worth nothing; five usable lanes give x4 and one lane goes unused.
And the commit boundary is the idea to carry. Everything discovered is provisional; the committed configuration changes all at once or not at all, because the two blocks that consume it are the two halves of a Link, and one cycle of disagreement between them produces corruption that surfaces as a CRC error on a channel with no errors in it.
Chapter 18.5 — Recovery takes the question this chapter deliberately left alone. A configured Link does not stay configured forever. Speed changes, equalization at a new rate, width changes, and re-synchronization after a disturbance all pass through Recovery — and it is the state a working Link spends its life re-entering, which is a very different problem from establishing one.
The idea to carry forward: a discovery is not a commitment, and the difference has to be a boundary rather than a habit.