x8 Links — When Lanes Become a Resource to Allocate

Chapter 6.3 treated four lanes as given. They were present, they belonged to the Link, and the engineering was about coordinating and observing them.

That framing quietly assumed something. Lanes do not simply exist for a Link — they are drawn from a finite pool, and every lane given to one connection is a lane not available to another.

What changes when a design commits eight lanes to one device instead of distributing those lanes across multiple narrower Links?

1. The Question x8 Asks

At x1 and x2 the width was small enough that the choice barely registered. At x4 it was a reasonable commitment. At x8, a designer with a limited lane budget is visibly spending a large fraction of it on one connection.

So the architectural question arrives:

Given a pool of eight lane resources, when does dedicating them to one wide path serve the system better than using them to connect several narrower ones?

Both arrangements use exactly eight lanes. They produce very different systems.

One x8 connection concentrates transport capability on a single device. If that device can actually generate and absorb traffic at a high rate, and if the rest of the path can sustain it, the concentration pays.

Two x4 connections provide transport to two devices. Neither gets the peak capability of the x8 arrangement, but two functions are connected instead of one — and if neither could have saturated eight lanes anyway, nothing was lost.

Neither is better. They solve different problems, and which problem a system has is a property of that system.

2. Two Allocations of One Pool

The figure is the chapter's central idea in one picture: the pool is fixed; the connectivity is a decision.

Note what is not shown. There is no mechanism between the pool and the allocations — no negotiation, no configuration protocol, no firmware. That is deliberate. The figure shows the consequence of an allocation, which is what this chapter reasons about, not how one comes to exist.

3. Local Capability Versus System Capacity

This distinction is the reason "wider is better" fails as a design rule, and x8 is where it starts to bite in practice.

An x8 Endpoint has a wide local Link. Whether the system can use it depends on things the Link does not control:

Upstream topology may constrain it. From Chapter 4.4: traffic from an Endpoint traverses a path, and any segment of that path narrower or slower than this Link caps what actually flows. A wide Link behind a constrained upstream segment cannot exceed the constraint.

Root Complex resources may be shared. Multiple paths converging on shared resources contend for them. A Link that could sustain a high rate in isolation may not get the opportunity.

Memory may be the bottleneck. Data has to come from and go somewhere. If the memory subsystem cannot supply or absorb at rate, transport width is not the limit.

The device may not generate enough demand. A function that cannot produce traffic fast enough leaves the Link idle regardless of its width — the utilisation reasoning from Chapter 5.1.

Software may not sustain traffic. Queue depths, completion handling, and access patterns all determine whether demand is continuous or bursty.

Lane allocation creates potential capacity. It does not create useful work.

This is why the "one x8 versus two x4" question has no width-only answer. If the single device could saturate eight lanes and the path can carry it, concentration wins. If it could not, the second arrangement connects an additional function at no throughput cost — and the first arrangement has spent four lanes' worth of pins, PHY, routing, and power on capability nothing will use.

4. Representative Uses

x8 appears commonly in high-performance network interface controllers, accelerators, FPGA cards, storage controllers, and compute devices — broadly, functions with enough sustained demand to make the wider commitment worthwhile, in systems with lanes to spare for it.

Stated carefully: these are representative categories, not rules, and no device class universally uses x8. The same kind of function appears at x4 and at x16 in different products, because the width is chosen against that product's requirements and its platform's lane budget. Product-specific claims are outside this chapter's scope and would date badly.

5. Microarchitecture — The Allocation Plane

Chapter 6.3 introduced the per-lane status plane. x8 adds a second small structure above it: an allocation plane that records which lanes belong to which Link.

Its responsibilities are narrow and its invariants are strict:

• Every lane in the pool is owned by at most one configured group.
• No lane is owned by two groups — ever, including transiently during a change.
• A group is declared ready only if every lane it owns is available.
• The allocation does not change while traffic depends on it.

The last one is the interesting one. An allocation is not merely configuration data; it is configuration data that other logic is actively relying on. Changing it underneath running traffic reassigns lanes that packets are mid-flight on, which is a class of bug with no good failure mode.

6. RTL — Lane Ownership

// SYNTHESIZABLE. Lane-resource ownership over a fixed pool.
// NOT PCIe bifurcation, NOT link training, NOT a platform configuration model.
package lane_alloc_pkg;
  typedef enum logic {
    ALLOC_WIDE  = 1'b0,   // one group owning the whole pool
    ALLOC_SPLIT = 1'b1    // two disjoint groups of half the pool each
  } lane_alloc_e;
endpackage
 
module lane_allocation
  import lane_alloc_pkg::*;
#(
  parameter int LANES = 8
) (
  input  logic             clk,
  input  logic             rst_n,
 
  input  lane_alloc_e      alloc_req,
  input  logic             alloc_commit,   // apply alloc_req now
  input  logic             epoch_active,   // traffic currently depends on the
                                           // allocation; abstract input
  input  logic [LANES-1:0] lane_up,        // per-lane availability
 
  output lane_alloc_e      alloc_q,
  output logic [LANES-1:0] group_a_mask,
  output logic [LANES-1:0] group_b_mask,
  output logic             group_a_ready,
  output logic             group_b_ready,
  output logic             alloc_fault     // sticky: illegal commit attempted
);
 
  localparam int HALF = LANES / 2;
 
  initial begin
    if (LANES < 2)        $fatal(1, "LANES must be at least 2");
    if (LANES % 2 != 0)   $fatal(1, "this model splits the pool in half");
  end
 
  // Ownership masks are DERIVED from the committed allocation, never stored
  // independently. Two stored masks could drift out of agreement with each
  // other; one stored mode with derived masks cannot.
  always_comb begin
    case (alloc_q)
      ALLOC_WIDE: begin
        group_a_mask = {LANES{1'b1}};                      // lanes 0 .. LANES-1
        group_b_mask = '0;                                 // group B unused
      end
      ALLOC_SPLIT: begin
        group_a_mask = {{HALF{1'b0}}, {HALF{1'b1}}};       // lanes 0 .. HALF-1
        group_b_mask = {{HALF{1'b1}}, {HALF{1'b0}}};       // lanes HALF .. end
      end
      default: begin                                       // unreachable; safe
        group_a_mask = '0;
        group_b_mask = '0;
      end
    endcase
  end
 
  // A group is ready only if it owns lanes AND every lane it owns is up.
  // The non-zero test matters: an empty mask trivially satisfies the subset
  // condition, so without it an unconfigured group would report ready.
  assign group_a_ready = (group_a_mask != '0)
                      && ((lane_up & group_a_mask) == group_a_mask);
  assign group_b_ready = (group_b_mask != '0)
                      && ((lane_up & group_b_mask) == group_b_mask);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      alloc_q     <= ALLOC_WIDE;
      alloc_fault <= 1'b0;
    end else if (alloc_commit) begin
      if (epoch_active) begin
        // REFUSE and RECORD. Reassigning lanes while traffic depends on them
        // would move a lane between groups underneath in-flight data. The
        // fault bit is sticky so the illegal attempt is visible afterwards —
        // silently ignoring it would hide a real software or sequencing bug.
        alloc_fault <= 1'b1;
      end else begin
        alloc_q <= alloc_req;
      end
    end
  end
 
endmodule

Classification: synthesizable, with a compile-time typedef in the accompanying package.

What it models: ownership of a finite lane pool by configured groups, and the rule that ownership is exclusive and stable while depended upon.

What it teaches — three things:

1. Derive masks from one stored mode rather than storing masks. Two independently stored masks can drift into an inconsistent state where a lane appears in both or neither. Deriving them from a single committed mode makes that state unrepresentable, which is stronger than checking for it.
2. Readiness must test both ownership and availability. The != '0 guard is not defensive noise — without it an empty mask satisfies (lane_up & mask) == mask trivially, and an unconfigured group reports ready. That is a real bug with a subtle cause.
3. Refuse and record, rather than refuse silently. An attempt to reconfigure during active traffic is a sequencing error somewhere. Ignoring it produces a system that mysteriously did not take a configuration; recording it produces one that says so.

Deliberately simplified: two allocations only; a fixed half-and-half split; groups are ready or not with no reduced-width behaviour; epoch_active is an abstract input rather than derived from real traffic state; and there is no mechanism by which an allocation comes to be requested.

Production implication: a real system establishes width through negotiation with the far component (Module 17.3), supports more allocation shapes than two, must define what happens to in-flight traffic when a Link goes down, coordinates allocation with the physical lane numbering and ordering the platform actually uses, and reports configuration state through defined interfaces rather than ad-hoc masks.

7. Assertions

The property set here is unlike x4's, because the invariants are about ownership legality rather than health reduction.

// SVA over lane_allocation. Implementation invariants for THIS model —
// not PCIe protocol requirements and not a model of platform configuration.
 
// EXCLUSIVITY — P1: no lane is owned by two groups. This is the invariant the
// whole structure exists to guarantee. A lane in two masks would be driven by
// two Links, which has no correct behaviour.
property p_ownership_disjoint;
  @(posedge clk) disable iff (!rst_n)
  (group_a_mask & group_b_mask) == '0;
endproperty
a_disjoint : assert property (p_ownership_disjoint);
 
// LEGALITY — P2: the wide allocation owns the entire pool and leaves no
// second group. Catches a mask constructed with the wrong width or shift.
property p_wide_owns_all;
  @(posedge clk) disable iff (!rst_n)
  (alloc_q == ALLOC_WIDE) |-> (group_a_mask == {LANES{1'b1}} && group_b_mask == '0);
endproperty
a_wide_owns_all : assert property (p_wide_owns_all);
 
// LEGALITY — P3: the split allocation gives each group exactly half the pool
// and, together with P1, partitions it. $countones catches an off-by-one in
// the mask construction that P1 alone would not.
property p_split_halves;
  @(posedge clk) disable iff (!rst_n)
  (alloc_q == ALLOC_SPLIT) |-> ($countones(group_a_mask) == HALF
                             && $countones(group_b_mask) == HALF);
endproperty
a_split_halves : assert property (p_split_halves);
 
// COVERAGE — P4: together the groups never claim more than the pool. With P1
// and P3 this makes the split a true partition rather than a subset.
property p_within_pool;
  @(posedge clk) disable iff (!rst_n)
  $countones(group_a_mask | group_b_mask) <= LANES;
endproperty
a_within_pool : assert property (p_within_pool);
 
// SAFETY — P5: a group is never ready unless every lane it owns is up.
// Catches readiness computed from the pool rather than from the owned subset,
// which would declare a group ready on the strength of another group's lanes.
property p_ready_implies_lanes_up;
  @(posedge clk) disable iff (!rst_n)
  group_a_ready |-> ((lane_up & group_a_mask) == group_a_mask);
endproperty
a_ready_needs_lanes : assert property (p_ready_implies_lanes_up);
 
// SAFETY — P6: an unconfigured group is never ready. This is the `!= '0`
// guard as an assertion; without the guard the vacuous subset test passes.
property p_empty_group_not_ready;
  @(posedge clk) disable iff (!rst_n)
  (group_b_mask == '0) |-> !group_b_ready;
endproperty
a_empty_not_ready : assert property (p_empty_group_not_ready);
 
// STABILITY — P7: the allocation does not change while traffic depends on it.
// The property that protects in-flight data from having its lanes reassigned.
property p_alloc_stable_during_traffic;
  @(posedge clk) disable iff (!rst_n)
  epoch_active |=> $stable(alloc_q);
endproperty
a_alloc_stable : assert property (p_alloc_stable_during_traffic);
 
// SAFETY — P8: an illegal commit attempt is recorded and stays recorded.
property p_fault_sticky;
  @(posedge clk) disable iff (!rst_n)
  alloc_fault |=> alloc_fault;
endproperty
a_fault_sticky : assert property (p_fault_sticky);

P1 is the property that justifies the module. A lane owned by two groups is not a degraded configuration; it is an incoherent one. Two Links would each believe they may drive it. There is no correct behaviour to fall back on, and no error the hardware could report that would be more useful than preventing the state.

P3 catches what P1 cannot. Disjointness alone is satisfied by masks that are both too small — a shift error producing three lanes and four lanes is disjoint, within the pool, and wrong. Counting the bits is what closes that gap, and the failure it catches produces a system that works while quietly stranding a lane.

P6 encodes a bug that reviews miss. (lane_up & '0) == '0 is true. Without the non-zero guard, an unconfigured group reports ready whenever it is asked — and in the wide allocation, group B is always unconfigured. A reader scanning the readiness expression sees a correct-looking subset test.

P7 is the one ordinary testing rarely reaches, because it requires deliberately attempting something forbidden while traffic is running. A directed test has to be written for it; random stimulus will usually not produce a commit precisely during an active epoch.

8. Verification

Monitors observe: the committed allocation, both ownership masks, both readiness outputs, per-lane availability, commit attempts with their request value, the active-traffic indication, and the fault bit.

Ownership is the kind of property that is easy to check incorrectly, so the environment should carry its own model of what a legal allocation looks like:

// VERIFICATION-ONLY. NOT synthesizable, NOT PCIe protocol state.
// An independent legality model for lane ownership. Deliberately written to
// reason about the masks directly rather than to mirror the design's
// construction of them — a checker that rebuilds the masks the same way the
// design does agrees with the design about a mask bug.
function automatic bit ownership_is_legal(
  input int unsigned lanes,
  input longint unsigned mask_a,
  input longint unsigned mask_b
);
  longint unsigned pool = (lanes >= 64) ? '1 : ((1 << lanes) - 1);
 
  // 1. Exclusive: no lane appears in both groups.
  if ((mask_a & mask_b) != 0)              return 1'b0;
  // 2. Bounded: neither group claims a lane outside the pool.
  if (((mask_a | mask_b) & ~pool) != 0)    return 1'b0;
  // 3. Non-trivial: at least one group owns something.
  if ((mask_a | mask_b) == 0)              return 1'b0;
  return 1'b1;
endfunction
 
// A partition is stronger than mere legality: every lane in the pool is owned.
// The split allocation must satisfy this; the wide allocation satisfies it
// with group B empty. A mask-construction error that strands a lane is legal
// by the function above and NOT a partition — which is exactly the bug that
// otherwise ships, because a stranded lane produces no error of any kind.
function automatic bit ownership_is_partition(
  input int unsigned lanes,
  input longint unsigned mask_a,
  input longint unsigned mask_b
);
  longint unsigned pool = (lanes >= 64) ? '1 : ((1 << lanes) - 1);
  return ownership_is_legal(lanes, mask_a, mask_b)
      && ((mask_a | mask_b) == pool);
endfunction

Classification: verification-only.

What it models: the legality and completeness of a lane-ownership assignment, independently of how the design constructs it.

What it teaches: that legal and complete are different checks, and only the second catches a stranded lane. A shift error yielding three lanes and four lanes is exclusive, bounded, and non-trivial — it passes every legality test and silently wastes a lane's worth of resources. ownership_is_partition is what fails on it.

Deliberately simplified: pools up to 64 lanes, two groups, and no notion of a lane being deliberately unassigned.

Production implication: a real environment would model more than two groups, handle intentionally unassigned lanes as a legal state distinct from an accidental gap, and check ownership against the platform's actual lane numbering rather than a contiguous pool.

The scoreboard independently predicts: the expected masks for each allocation, computed from its own model of the split rather than by reading the design's masks; expected readiness from availability intersected with its own expected masks; and whether each commit attempt should have been accepted or refused. The independence matters especially here — a scoreboard that derives expected masks from group_a_mask verifies nothing about mask construction, which is exactly where P2 and P3 expect defects.

Scenarios:

• Wide allocation, all lanes up. Group A owns the pool and is ready; group B is empty and not ready. This is the case where P6's bug hides.
• Split allocation, all lanes up. Both groups own half and both are ready. Verify disjointness and the bit counts.
• Illegal overlap injected. Force a mask construction that overlaps — by binding to an intentionally broken variant or by driving the masks directly in a bind-based check. Verify P1 fires. A property that has never been observed to fail has not been shown to work, and ownership disjointness is easy to write in a way that is vacuously true.
• Configuration change while idle. Commit a different allocation with epoch_active low. Verify it takes effect on the next cycle and the masks change coherently — both at once, never one before the other.
• Configuration change while active — the prohibited case. Commit with epoch_active high. Verify the allocation does not change (P7) and alloc_fault sets (P8). Then verify a subsequent legal commit while idle still works, so refusing did not wedge the module.
• Individual lane failures in each allocation. For each lane index and each allocation, drop lane_up[i] and verify exactly the group owning that lane loses readiness while the other is unaffected. In the split allocation this is the test that proves the groups are genuinely independent; in the wide allocation it proves any lane loss affects the single group.
• Group independence in the split model. Drive traffic against group A while group B is unavailable. Verify group A's readiness and behaviour are unaffected. Under this teaching model the groups share no lanes, so a dependency would indicate a leak between them.
• Shared resource above both groups. Saturate a resource common to both groups and observe that neither group's readiness changes even though throughput does. This separates "the allocation is fine" from "the system is delivering," which is the chapter's core distinction made checkable.

Coverage should include: each allocation value; each lane index as the sole unavailable lane, crossed with each allocation; commit attempts in both epoch_active states; commit of the same value as currently held; and reset with the fault bit set.

9. Debugging

Reference scenario: an x8-capable device underperforms.

This is the characteristic x8 investigation, and its structure is deliberately different from x4's. There the question was which lane. Here the first several questions are not about lanes at all.

1. Is it actually operating at the intended width? Verify, do not assume. A device capable of x8 that is operating narrower explains a shortfall completely, and every subsequent measurement is misleading until this is settled.

2. Is the local Link busy or idle? The utilisation question from Chapter 5.1. Substantial idle time means the Link was never the limit, and no amount of lane or width investigation will help.

3. Is the upstream topology limiting? From Chapter 4.4: a narrower or slower segment anywhere on the path caps everything behind it. A wide Link behind a constrained segment delivers the segment's capability, not its own.

4. Does the workload generate enough demand? A device that cannot produce or consume traffic at rate leaves capability unused. This is a property of the function and its software, not of the Link.

5. Are lane errors concentrated? Now the lane question — read the per-lane vector from Chapter 6.3. Errors on one index point at that lane's path; errors across all lanes point at something shared; no errors at all removes lane health from the hypothesis set entirely.

6. Is memory or another system resource limiting? Data has to originate and terminate somewhere. A memory subsystem that cannot sustain the rate caps throughput regardless of transport width.

7. Would these lane resources provide more system value elsewhere? Strictly this is architecture rather than debugging — but it is the question the previous six answers actually inform. If the device cannot use eight lanes, the finding is not a defect. It is that the allocation does not match the workload, and four of those lanes could connect something else.

10. Common Misconceptions

• "x8 always beats two x4 Links." It concentrates capability on one device. If that device cannot use it, or the path cannot sustain it, the concentration buys nothing while the alternative connects a second function.
• "Two x4 Links always beat one x8." Equally wrong in the other direction. A device that can genuinely sustain traffic across eight lanes is served worse by four, and splitting to connect something with no demand wastes the split.
• "Lane allocation is a software-only concern." Software may request a configuration, but the hardware must enforce exclusive ownership, derive readiness from owned lanes, and refuse changes while traffic depends on them. Those are hardware invariants, and P1 and P7 exist because they can be violated in RTL.
• "x8 guarantees twice x4 application throughput." It has twice the physical lane count. Delivered throughput approaches a corresponding increase only when the Link was the limiting resource and the rest of the path can sustain the demand. Chapter 6.7 owns the arithmetic.
• "All eight lanes carry unrelated work independently." In one x8 Link, all eight carry the traffic of one connection. Independent groups exist only when the pool is allocated into separate Links — which is a different configuration, not a property of eight lanes.
• "Lane allocation is the same thing as PCIe routing." Allocation decides which lanes form which Link. Routing decides where a transaction goes once it is on a Link. Different layers, different mechanisms; Module 7 onward covers routing.
• "An idle x8 Link is wasting bandwidth." In a system with lanes to spare and nothing else needing them, an idle wide Link costs nothing that another arrangement would have recovered. It is a waste only when those lanes had an alternative use — which is precisely the allocation question.
• "Wider Links remove upstream bottlenecks." Width is local to one Link. A constrained segment elsewhere on the path is unaffected by how wide this Link is, and widening past the constraint changes nothing downstream of it.

11. Understanding Check

12. What's Next

x8 established that lanes are an allocatable resource, that ownership carries hardware invariants, and that local width is potential capacity rather than delivered work.

Chapter 6.5 — x16 Links takes width to the point where the commitment is substantial in every dimension — PHY channels, package and board resources, power, and above all observability. At sixteen lanes a single failure index is no longer enough information: what matters is the distribution of errors across lanes, because four errors on one lane and one error on each of four lanes are diagnostically different situations that any aggregate count reports identically.