x4 Links — Where Multi-Lane Becomes Practical

Chapter 6.2 introduced the coordination problem: two lanes carrying one logical stream must be distributed, reconstructed, and reconciled. That chapter deliberately used the smallest case that has the problem at all.

x4 is where the problem stops being a teaching example.

Why does x4 become the first width where PCIe feels like a practical high-throughput peripheral connection rather than a multi-lane demonstration?

1. Why x4 Earns Its Own Chapter

Two reasons, and only the second is really about engineering.

It is a widely used width. x4 appears commonly in NVMe storage devices, FPGA endpoints, network interface controllers, accelerators, and embedded add-in devices. It is often the width chosen when a function needs meaningful throughput without committing the resources a wider Link demands.

Stated carefully: these are representative uses, not rules. Not all storage devices are x4, x4 is not reserved for storage, and the width a given product uses is a design decision made against that product's requirements. A chapter that said "x4 is the SSD width" would be memorable and wrong.

It is where observability stops being optional. This is the engineering reason.

At x1, "is the Link working" and "is the lane working" are the same question. At x2, they diverge but there are only two lanes, so a mask of two bits is barely a mask. At x4, a design that reports one aggregate link_error bit has genuinely thrown away the information an engineer needs — because the first question on a four-lane Link is almost always which lane, and a single bit cannot answer it.

2. The x4 Link

Four physical lane units. One logical Link.

3. What Scales From x2

The honest answer is: not the category, the count.

x2 established distribution, reconstruction, relative arrival, and status reduction. x4 has exactly those four problems and no fifth one. What changes is how many instances of each exist:

4. Resource Scaling

Four lanes require four lanes' worth of physical and logical resources: transmit and receive PHY instances, package pins, board routes and their length-matching constraints, status bookkeeping, and per-lane monitoring.

That much is arithmetic. What it costs, how it trades against connecting more devices at narrower widths, and how a system provisions a finite lane budget across everything that needs one is a genuine design analysis — and Chapter 6.8 owns it. Chapter 6.4 takes the first step by treating lanes as an allocatable resource; this chapter needs only that width is not free.

5. Microarchitecture — The Lane-Health Plane

Alongside the datapath established in Chapter 6.2, a multi-lane Link carries a second, much smaller structure that x1 did not need: a per-lane status plane.

Its job is not to move data. Its job is to answer three questions that are genuinely different from one another:

• Is each lane individually usable? — a vector, one bit per lane.
• Is the Link as a whole usable? — a scalar, derived from the vector by a reduction.
• If not, which lane went first? — a captured index, which neither of the above can supply.

The third question is the one designs most often fail to answer, because answering it requires capturing something at the moment it happens. By the time an engineer looks, several lanes may be reporting problems, and the order in which they started is frequently the whole diagnosis.

6. RTL — Per-Lane Health and Its Reduction

// SYNTHESIZABLE. Per-lane health vector and its reduction to Link health.
// The event inputs are ABSTRACTIONS supplied by surrounding logic — not PCIe
// signals and carrying no protocol semantics.
module lane_health_vector #(
  parameter int LANES = 4
) (
  input  logic             clk,
  input  logic             rst_n,
 
  input  logic [LANES-1:0] lane_up,         // per-lane availability, live
  input  logic [LANES-1:0] lane_err_event,  // per-lane error pulse
  input  logic             clear_sticky,    // explicit software-style clear
 
  output logic [LANES-1:0] err_sticky,      // per-lane, latched
  output logic             all_lanes_up,
  output logic             any_lane_bad,
  output logic             link_healthy
);
 
  initial begin
    if (LANES < 1) $fatal(1, "LANES must be at least 1");
  end
 
  // Sticky per lane. An error that lasted one cycle must still be visible
  // when someone looks, which may be millions of cycles later. Only an
  // explicit clear or reset removes it.
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n)            err_sticky <= '0;
    else if (clear_sticky) err_sticky <= '0;
    else                   err_sticky <= err_sticky | lane_err_event;
  end
 
  // The reduction, written explicitly rather than folded into one expression,
  // because these are three separate questions with three separate answers.
  assign all_lanes_up = &lane_up;      // every lane individually available
  assign any_lane_bad = |err_sticky;   // any lane has ever reported an error
  assign link_healthy = all_lanes_up && !any_lane_bad;
 
endmodule

Classification: synthesizable.

What it models: the per-lane status plane of a multi-lane Link and the reduction that turns it into a Link-level conclusion.

What it teaches: that link_healthy is a derived value and the vector is the primary one. A design that stores only the scalar has destroyed the information before anyone could read it — and it cannot be reconstructed afterwards. The vector is cheap; the scalar is free to compute from it; the reverse is impossible.

Deliberately simplified: availability is a live input rather than latched state; errors are undifferentiated pulses with no classification or severity; the reduction requires all lanes, with no representation of operating at a reduced width; and there are no counts.

Production implication: a real implementation would classify errors by type and severity, count them per lane against a shared denominator (the discipline from Chapter 5.5), timestamp them so bursts are distinguishable from steady rates, and define what a lane becoming unavailable means for traffic already in flight.

7. RTL — First-Fail Capture

The reduction above answers whether. It cannot answer which came first, and on a four-lane Link that is usually the more valuable question.

// SYNTHESIZABLE. Captures the FIRST lane to report an error and holds it.
// Lowest-index-first priority among events presented in the same cycle.
module first_fail_capture #(
  parameter int LANES = 4
) (
  input  logic                     clk,
  input  logic                     rst_n,
  input  logic [LANES-1:0]         lane_err_event,
  input  logic                     clear,
 
  output logic                     first_valid,
  output logic [$clog2(LANES)-1:0] first_lane
);
 
  // $clog2(1) is 0, which would make first_lane a zero-width port. A
  // single-lane Link has no "which lane" question, so require at least two.
  initial begin
    if (LANES < 2) $fatal(1, "first_fail_capture requires LANES >= 2");
  end
 
  localparam int IDX_W = $clog2(LANES);
 
  // Priority encode. The loop descends so that the LOWEST set index is the
  // last assignment and therefore wins — a deterministic tie-break matters,
  // because simultaneous events on several lanes are a real scenario and an
  // arbitrary winner makes the captured value untrustworthy.
  logic             hit;
  logic [IDX_W-1:0] idx;
  always_comb begin
    hit = 1'b0;
    idx = '0;                       // both assigned unconditionally: no latch
    for (int i = LANES - 1; i >= 0; i--)
      if (lane_err_event[i]) begin
        hit = 1'b1;
        idx = IDX_W'(i);
      end
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      first_valid <= 1'b0;
      first_lane  <= '0;
    end else if (clear) begin
      first_valid <= 1'b0;
      first_lane  <= '0;
    end else if (!first_valid && hit) begin
      // Capture ONCE. Later events on other lanes must not overwrite this —
      // overwriting turns "which lane failed first" into "which lane failed
      // most recently", which is a different and much less useful fact.
      first_valid <= 1'b1;
      first_lane  <= idx;
    end
  end
 
endmodule

Classification: synthesizable.

What it models: first-event capture across a lane vector — a small, generic, and widely reusable observability structure.

What it teaches: two things. First, that capture-once is the whole point: a register that keeps updating records the last event, and by the time anyone reads it the initiating lane is long overwritten. Second, that a deterministic tie-break is required, because simultaneous events on multiple lanes genuinely happen and a design that picks arbitrarily produces a value nobody can reason about.

Deliberately simplified: one capture slot, so a second independent failure episode after a clear is not distinguished from the first; no timestamp; no error classification; and lowest-index priority chosen for determinism rather than because low-numbered lanes matter more.

Production implication: a real implementation would timestamp the capture, record enough context to correlate it with traffic and configuration state, and generally provide more than one slot so an initiating event and a subsequent escalation are both visible.

8. Assertions

// SVA over the modules above. Implementation invariants for THESE designs —
// not PCIe protocol requirements.
 
// CONSISTENCY — P1: the Link-level conclusion matches the vector it was
// derived from. Catches a reduction that drifts from its inputs, e.g. an
// optimised expression that stops tracking a lane after an edit.
property p_reduction_consistent;
  @(posedge clk) disable iff (!rst_n)
  link_healthy == ((&lane_up) && !(|err_sticky));
endproperty
a_reduction_consistent : assert property (p_reduction_consistent);
 
// SAFETY — P2: a healthy Link cannot be claimed while any lane has a sticky
// error. This is the property that would fire if someone "fixed" a noisy
// status by masking lanes out of the reduction.
property p_healthy_excludes_bad_lane;
  @(posedge clk) disable iff (!rst_n)
  link_healthy |-> (err_sticky == '0);
endproperty
a_healthy_excludes_bad : assert property (p_healthy_excludes_bad_lane);
 
// SAFETY — P3: sticky bits only clear on an explicit clear. Catches a lane
// error being erased by an unrelated state change, which loses the evidence
// silently and leaves a Link that "recovered" without anyone knowing why.
property p_sticky_only_clears_explicitly;
  @(posedge clk) disable iff (!rst_n)
  (err_sticky != '0 && !clear_sticky) |=> (err_sticky != '0);
endproperty
a_sticky_persists : assert property (p_sticky_only_clears_explicitly);
 
// MONOTONICITY — P4: within a period without a clear, sticky bits only set.
// Stronger than P3: catches ONE lane's bit clearing while others remain.
property p_sticky_monotonic;
  @(posedge clk) disable iff (!rst_n)
  !clear_sticky |=> ((err_sticky & $past(err_sticky)) == $past(err_sticky));
endproperty
a_sticky_monotonic : assert property (p_sticky_monotonic);
 
// SAFETY — P5: the captured first-fail index never changes once captured.
// This is the entire value of the structure; without it the register records
// the most recent failure and the name of the signal is a lie.
property p_first_lane_stable;
  @(posedge clk) disable iff (!rst_n)
  (first_valid && !clear) |=> (first_valid && $stable(first_lane));
endproperty
a_first_lane_stable : assert property (p_first_lane_stable);
 
// CORRECTNESS — P6: the lane that was captured actually had an event in the
// cycle of capture. Catches an off-by-one in the priority encoder, which
// otherwise produces a plausible index pointing at an innocent lane.
property p_captured_lane_had_event;
  @(posedge clk) disable iff (!rst_n)
  (!first_valid ##1 first_valid) |-> $past(lane_err_event[first_lane]);
endproperty
a_captured_lane_real : assert property (p_captured_lane_had_event);
 
// SAFETY — P7: capture requires an event. No fabricated first-fail.
property p_no_capture_without_event;
  @(posedge clk) disable iff (!rst_n)
  (!first_valid ##1 first_valid) |-> $past(|lane_err_event);
endproperty
a_no_phantom_capture : assert property (p_no_capture_without_event);

P6 is the one ordinary testing misses. A priority encoder with an index error still sets first_valid at the right time, still produces a value in range, and still looks entirely reasonable in a waveform. Nothing about the observable behaviour is wrong except which lane it names — and a test that injects a fault on lane 2 and reads back "lane 1" will often be read as a testbench problem before it is read as a design problem.

P4 is stronger than it looks. P3 permits a state in which one lane's sticky bit clears while another remains set, because err_sticky != '0 still holds. P4 catches it. That failure mode is realistic: per-lane clear logic that clears the wrong index.

9. Verification

Monitors observe: per-lane availability, per-lane error events, the sticky vector, the reduced Link health, the captured first-fail index and its valid bit, and clear operations.

The scoreboard independently predicts: the expected sticky vector as the running OR of injected events since the last clear; the expected Link-health scalar from its own reduction of availability and stickiness; and the expected first-fail index from the first injected event, with the documented lowest-index tie-break applied. It must compute these from the injection log, not by reading the design's own vector.

Scenarios:

• All four healthy. Baseline. No sticky bits, link_healthy asserted, first_valid low.
• Each lane fails independently — all four cases. For each lane i: inject on i alone, verify err_sticky[i] sets, no other bit sets, link_healthy deasserts, and first_lane == i. This is the test that catches indexing errors, and it must cover every index rather than a representative one.
• Two simultaneous lane failures. Inject on two lanes in the same cycle. Verify both sticky bits set and the documented tie-break selects the lower index deterministically — run it for several lane pairs.
• Sequential failures. Inject on lane 3, then later on lane 1. Verify first_lane remains 3. This is P5's scenario made concrete and is the single most valuable test of the capture structure.
• Failure after sustained traffic. Long clean run, then a fault. Verify capture still works and no state has drifted — catches structures that only work near reset.
• Clear and re-arm. Clear, verify the vector and capture reset, then inject again on a different lane and verify the new capture.
• Reset with state set. Reset while sticky bits and a capture are held. Verify a clean restart with no stale index presented.
• One-lane degradation with three healthy. The realistic field scenario: three lanes carrying traffic normally while one reports errors. Verify the Link-level conclusion and the vector disagree in the correct direction — the Link is not healthy, but three lanes still are, and both facts are readable.

Coverage should include: each lane index as the sole failing lane; each lane index as the captured first-fail; simultaneous events across several lane pairs; clear at each vector state; reset at each vector state; and the cross of availability against stickiness.

10. Debugging

The technique that makes multi-lane debugging tractable is width comparison, and x4 is where it first pays real dividends.

Symptom: x1 and x2 work, x4 fails.

Same devices, same generation, same workload, narrower width known good. What that comparison implicates:

• The additional lane paths — lanes 2 and 3 specifically, including their datapaths, PHY instances, and physical routes.
• Aggregate coordination at four — distribution and reconstruction across four fragments rather than two, and any logic whose correctness happens to depend on the count.
• x4-specific configuration — anything that differs between the two widths in setup.
• Board and package routing — the additional routes and their length matching.

What it argues against: transaction semantics. Identical requests were correct at the narrower width, produced by identical logic.

Symptom: one lane consistently reports errors, the others are clean.

The asymmetry is the information. Everything shared — the transaction layer, the aggregate datapath, the Link's configuration, the clock — would be expected to affect all four lanes. One lane failing while three do not points at what is specific to that lane: its physical path, its transmitter and receiver instances, its package and board route, and its lane-local logic.

This is the scenario first_lane and err_sticky were built for, and it is why §7 exists.

Symptom: the failing lane index rotates between runs.

Different diagnosis entirely, and a genuinely useful one. If the fault followed a fixed physical path it would generally stay on the same lane index. A fault that moves is more consistent with something shared: common logic, a shared resource, an environmental factor affecting all lanes, or a marginal condition that happens to bite whichever lane is closest to its limit at the time.

Practical distinction: a fixed index across many runs is evidence for a lane-local physical cause. A rotating index is evidence against it and for a shared cause. Neither is proof — a marginal lane can appear intermittent, and shared logic can have a lane-indexed bug that looks fixed.

11. Common Misconceptions

• "x4 means four PCIe Links." It is one Link four lanes wide — one connection between one pair of components. Four Links would be independently established, configured, and able to fail separately.
• "x4 means four transactions in flight at all times." Width is a property of the transport. How many operations may be outstanding is a transaction-layer mechanism and unrelated to lane count; Modules 10 onward own it.
• "x4 always gives exactly four times x1 application throughput." It gives four times the physical lane count. Delivered throughput approaches a corresponding increase only when the Link was the limiting resource and the rest of the path can sustain the demand — the reasoning from Chapter 5.1. Chapter 6.7 owns the actual arithmetic.
• "All four lanes are interchangeable in every respect." They are equivalent in role — each is one unit of width. They are not identical in implementation: routes differ in length and environment, and PHY instances differ within manufacturing variation. That is precisely why one lane can fail while three do not.
• "If one lane misbehaves, the transaction logic must be wrong." A single-lane symptom argues the opposite. The transaction layer is shared across all four lanes and would generally affect all of them.
• "x4 is the SSD width." x4 is common in storage and also in network controllers, FPGAs, and accelerators; storage devices also appear at other widths. Width is a per-design decision, not a device-class property.
• "x4 adds no verification burden beyond x2." Single-lane failure cases double, multi-lane combinations grow from one to eleven, and per-lane indexing becomes a real defect class. The mechanisms are the same; the state space is not.
• "A wider Link means a newer generation." Width and generation are independent dimensions (Chapter 6.1). Either can change while the other is fixed.

12. Understanding Check

13. What's Next

x4 established what scales with width — instances, state, and debug surface — and the observability structures that make a multi-lane Link diagnosable at all.

It did not ask where the lanes came from. Four lanes were simply present.

Chapter 6.4 — x8 Links asks that question. Lanes are a finite system resource, and committing eight of them to one connection is a choice made against the alternative of connecting more devices at narrower widths. That makes x8 the width where allocation becomes an architectural concern rather than a given — and it introduces a class of hardware invariant this chapter had no need for: every lane belongs to exactly one Link, and no lane belongs to two.