Real System Trade-offs — Deciding Where Width Is Worth Spending

Chapter 6.7 can tell you exactly what a Gen5 x16 Link could theoretically carry. It cannot tell you whether that Link should exist.

That is the last question in Module 6, and it is the one an architect actually has to answer.

Given a finite lane budget, how should an architect decide where width is worth spending?

1. What Width Costs

Six dimensions, all real, none quantified here — because the numbers are process-, package-, and platform-specific, and inventing plausible ones would be worse than naming the categories precisely.

Package and connectivity. Each lane requires physical connectivity at both ends. Pins are scarce and contended by everything else the component must connect to.

PHY resources. Each lane needs its own transmit and receive circuitry, occupying silicon area in both components.

Board routing. Each lane is two differential pairs (Chapter 6.1), so N lanes are 2N pairs between the two components — each subject to length matching, impedance control, and layer budget. At the generations where the channel is a first-class constraint (Chapter 5.5), this is a significant share of the design effort.

Power. More high-speed circuitry operating consumes more power, and the thermal consequences propagate into packaging and cooling.

Silicon area. Beyond the PHY: per-lane datapath, buffering, and the observability structures Module 6 has been arguing for since x4.

Opportunity cost. The one that is easiest to forget and often the largest. Lanes assigned here cannot serve another Link. In a platform with a fixed budget, every allocation is simultaneously a decision not to connect something else.

2. What Width Buys — Conditionally

Width raises layer 5 of Chapter 6.7's stack and nothing else. It becomes useful throughput only when every one of the following holds:

• The device can generate sustained demand at the higher rate.
• The destination can absorb it.
• The upstream path can sustain it end to end.
• The workload is bandwidth-sensitive rather than latency- or capacity-sensitive.
• Software keeps the pipeline supplied rather than issuing in bursts with gaps.

If any of those fails, the additional lanes raise a ceiling that nothing reaches, and the cost in §1 is paid in full.

Width creates potential capacity. The workload and the path decide whether it becomes work.

3. The Lane Budget Problem

Make it concrete with a platform that has sixteen lane resources to allocate. All three arrangements below consume exactly sixteen.

The three arrangements are indistinguishable by lane count and by any width-only metric. They differ in what the system can do, and that depends on the workload — which is why the rest of this chapter reasons from workloads rather than from widths.

4. Case Study 1 — A Bandwidth-Heavy Accelerator

An accelerator currently on an x8 Link demonstrably saturates it: high active time, low idle, low stall, and a payload fraction that is already good.

Does x16 help?

Only if all of the following hold. The accelerator can issue more traffic than x8 carries — saturating x8 proves it wanted at least that much, not that it wants more. The host path and memory can sustain the higher rate. And the workload's completion time is actually bounded by transport rather than by computation.

How to find out before committing eight more lanes. Run the same workload at a narrower width and measure the shape of the response. If throughput at x4 is close to half of x8, the workload is scaling with width and probably has more demand than x8 satisfies. If x4 and x8 deliver nearly the same throughput, the workload was never Link-bound and x16 will change nothing.

That measurement costs one experiment and settles an allocation worth eight lanes. It is available before any hardware is designed, using Chapter 6.7's instrumentation on an existing platform.

5. Case Study 2 — Multiple Storage Devices

Four storage devices, each of which saturates an x4 Link under its own workload, against one device on a wider Link.

When the parallel arrangement wins. The workload is genuinely parallel across devices, each device saturates its x4, and the upstream path can carry the aggregate. Four devices doing useful work beats one device with capability it cannot use.

When it does not. If the upstream path — a switch link, a root port, the memory subsystem — cannot carry the sum, adding devices does not add throughput. It divides the same upstream capacity into more streams, and each device gets less.

6. Case Study 3 — A Mixed Workload

A NIC, an accelerator, and a storage device sharing one platform's lane budget and one upstream path. This is the realistic case, and it cannot be answered by comparing peak numbers.

The questions that actually decide it:

Is the function latency-sensitive or bandwidth-sensitive? A latency-sensitive function may gain little from width — its transactions are small and infrequent, so from Chapter 6.7 §6 it is dominated by per-transaction overhead rather than by transport capacity. Lanes spent there buy less than lanes spent on a bandwidth-sensitive function.

Is the demand sustained or bursty? Sustained demand justifies provisioning to it. Bursty demand may be served adequately by a narrower Link with buffering, since the average is what the path must carry.

Do the functions peak simultaneously? Three functions that each need high bandwidth at different times can share an upstream path comfortably. Three that peak together contend, and provisioning each to its individual peak over-provisions the platform for a case that never occurs — or under-provisions the shared segment for the case that does.

What is shared behind them? If all three converge on one upstream segment, its capacity is the real budget and the local widths are a distribution of it.

The method: architect from the workload. Start from what each function needs and when, work out what the shared path must carry, then decide widths — rather than starting from the widest configuration available and working down.

7. RTL — Lane Budget Accounting

// SYNTHESIZABLE. Lane-budget accounting over a fixed pool.
// NOT PCIe bifurcation, NOT enumeration, NOT platform firmware.
module lane_budget_checker #(
  parameter int CLIENTS    = 4,
  parameter int POOL_LANES = 16,
  parameter int W_W        = 5    // width field bits: 0..31 requested lanes
) (
  input  logic             clk,
  input  logic             rst_n,
 
  input  logic [W_W-1:0]   req_width [CLIENTS],
  input  logic [CLIENTS-1:0] req_active,
  input  logic             commit,
  input  logic             epoch_active,   // traffic depends on the allocation
 
  output logic [SUM_W-1:0] assigned_total,
  output logic [SUM_W-1:0] headroom,
  output logic             legal,
  output logic             overcommit,
  output logic             zero_width_active,
  output logic             illegal_width,
  output logic             commit_fault    // sticky
);
 
  // Wide enough for every client requesting the maximum representable width.
  localparam int SUM_W = $clog2(CLIENTS * (1 << W_W)) + 1;
 
  initial begin
    if (CLIENTS    < 1) $fatal(1, "CLIENTS must be at least 1");
    if (POOL_LANES < 1) $fatal(1, "POOL_LANES must be at least 1");
    if ((1 << W_W) <= POOL_LANES)
      $fatal(1, "W_W too narrow to express a full-pool request");
  end
 
  // A width is legal if it is a power of two and fits the pool. The widths
  // Module 6 covers are powers of two; a platform supporting others must
  // relax this deliberately rather than by accident.
  function automatic bit width_ok(input logic [W_W-1:0] w);
    return (w != 0) && ((w & (w - 1)) == 0) && (w <= W_W'(POOL_LANES));
  endfunction
 
  logic [SUM_W-1:0] total_c;
  logic             zero_c, illegal_c;
 
  always_comb begin
    total_c   = '0;                 // all three assigned unconditionally:
    zero_c    = 1'b0;               // no latch inference from this block
    illegal_c = 1'b0;
    for (int i = 0; i < CLIENTS; i++)
      if (req_active[i]) begin
        total_c += SUM_W'(req_width[i]);
        if (req_width[i] == '0)     zero_c    = 1'b1;
        else if (!width_ok(req_width[i])) illegal_c = 1'b1;
      end
  end
 
  assign assigned_total    = total_c;
  assign overcommit        = (total_c > SUM_W'(POOL_LANES));
  assign zero_width_active = zero_c;
  assign illegal_width     = illegal_c;
 
  // An active client with zero width is a request to connect nothing, which
  // is almost always a configuration bug rather than an intent.
  assign legal = !overcommit && !zero_c && !illegal_c && (total_c != 0);
 
  // Headroom is only meaningful for a legal allocation. Reporting a value for
  // an illegal one invites it being used.
  assign headroom = legal ? (SUM_W'(POOL_LANES) - total_c) : '0;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      commit_fault <= 1'b0;
    end else if (commit && (epoch_active || !legal)) begin
      // REFUSE AND RECORD. Committing an illegal allocation, or any
      // allocation while traffic depends on the current one, is a sequencing
      // or configuration error upstream. Ignoring it silently produces a
      // system that mysteriously did not take a configuration.
      commit_fault <= 1'b1;
    end
  end
 
endmodule

Classification: synthesizable.

What it models: the accounting a platform must do before an allocation can be applied — does the request set fit, is every requested width representable, and is it safe to apply now.

What it teaches — three things:

1. Overcommit is a summation problem, not a per-client one. Every individual request can be legal while the set is not. The check has to be over the total.
2. "Legal" has several independent failure modes. Overcommitted, zero-width-but-active, and non-power-of-two are different bugs with different causes, and collapsing them into one legal bit destroys the diagnosis — the same argument Chapter 6.3 made about aggregate error bits.
3. Headroom is reported only when it means something. Computing POOL_LANES - total_c for an overcommitted set produces a wrapped value that looks like plenty of spare capacity.

Deliberately simplified: widths are requested rather than negotiated; there is no mapping from a width to specific physical lanes, so this checks the budget rather than the placement; and the pool is contiguous and uniform.

Production implication: a real platform must map allocations to specific physical lanes with the ordering constraints its packaging imposes, negotiate the resulting width with each far component (Module 17.3), define behaviour when a negotiation yields less than requested, and coordinate all of it with discovery and configuration (Module 7).

8. Assertions

// SVA over lane_budget_checker. Implementation invariants for THIS model —
// not PCIe protocol requirements and not a model of platform configuration.
 
// BUDGET — P1: a legal allocation never assigns more lanes than the pool has.
// The invariant the module exists to guarantee. Assigning more lanes than
// exist is not a degraded configuration; it is an impossible one.
property p_legal_fits_pool;
  @(posedge clk) disable iff (!rst_n)
  legal |-> (assigned_total <= SUM_W'(POOL_LANES));
endproperty
a_legal_fits : assert property (p_legal_fits_pool);
 
// CONSISTENCY — P2: overcommit and legal are never both asserted. Catches a
// legality expression that stops tracking one of its inputs after an edit.
property p_overcommit_excludes_legal;
  @(posedge clk) disable iff (!rst_n)
  overcommit |-> !legal;
endproperty
a_overcommit_not_legal : assert property (p_overcommit_excludes_legal);
 
// SAFETY — P3: no active client may request zero width. An active client
// asking to connect nothing is a configuration bug, and treating it as a
// harmless no-op hides it.
property p_no_zero_width_active;
  @(posedge clk) disable iff (!rst_n)
  legal |-> !zero_width_active;
endproperty
a_no_zero_active : assert property (p_no_zero_width_active);
 
// SAFETY — P4: every active client's width is one this model accepts.
// Distinguishing illegal_width from overcommit matters: they are different
// bugs with different causes and different fixes.
property p_widths_representable;
  @(posedge clk) disable iff (!rst_n)
  legal |-> !illegal_width;
endproperty
a_widths_legal : assert property (p_widths_representable);
 
// CORRECTNESS — P5: the total is the sum of the active requests. Catches a
// loop bound that misses the last client — which under-counts, so the
// allocation passes the budget check and overcommits the pool in reality.
property p_total_is_sum;
  @(posedge clk) disable iff (!rst_n)
  assigned_total == total_c;
endproperty
a_total_correct : assert property (p_total_is_sum);
 
// SAFETY — P6: headroom is only reported for a legal allocation, and never
// wraps. POOL_LANES - total_c on an overcommitted set produces a large value
// that reads as abundant spare capacity.
property p_headroom_sane;
  @(posedge clk) disable iff (!rst_n)
  (headroom != '0) |-> (legal && (headroom <= SUM_W'(POOL_LANES)));
endproperty
a_headroom_sane : assert property (p_headroom_sane);
 
// SAFETY — P7: a refused commit is recorded and stays recorded. Silently
// ignoring an illegal or ill-timed commit produces a system that mysteriously
// did not take a configuration, with the symptom far from the cause.
property p_commit_fault_sticky;
  @(posedge clk) disable iff (!rst_n)
  commit_fault |=> commit_fault;
endproperty
a_fault_sticky : assert property (p_commit_fault_sticky);
 
// SAFETY — P8: any commit attempted while traffic depends on the allocation,
// or with an illegal request set, is recorded. This is the property ordinary
// stimulus rarely reaches: it requires deliberately attempting the forbidden.
property p_bad_commit_recorded;
  @(posedge clk) disable iff (!rst_n)
  (commit && (epoch_active || !legal)) |=> commit_fault;
endproperty
a_bad_commit_recorded : assert property (p_bad_commit_recorded);

P5 catches the failure that defeats the whole module. A summation loop that misses a client under-counts the total. The allocation then passes the budget check, legal asserts, headroom reports spare capacity that does not exist — and the platform is overcommitted with every indicator saying it is fine. Nothing else in the design detects this, because every downstream check trusts the total.

P6 is subtle and cheap. POOL_LANES - total_c on an overcommitted set wraps to a large positive number in unsigned arithmetic. A design that computes headroom unconditionally reports abundant spare capacity precisely when there is none — the most dangerous possible time for that number to be wrong.

P8 is the one that needs a directed test. Random stimulus will rarely assert commit in exactly the cycles where epoch_active is high or the request set is illegal. Without a deliberate scenario the property is never exercised, and a property that has never been observed to fire has not been shown to work.

9. Verification-Only — A System Throughput Model

Budget legality says an allocation fits. It says nothing about whether it is a good one. That question needs a model of what the system would actually deliver.

// VERIFICATION-ONLY. NOT synthesizable, NOT PCIe protocol behaviour.
// A system-level teaching model for reasoning about allocations. It predicts
// nothing about a real platform; it makes the STRUCTURE of the argument
// explicit and checkable.
class system_throughput_model;
 
  typedef struct {
    string           name;
    longint unsigned demand_kBps;   // what the device would issue if unlimited
    longint unsigned link_kBps;     // local Link capacity — from Chapter 6.7
    longint unsigned path_kBps;     // dedicated path capacity behind the Link
  } device_t;
 
  // Per device, ignoring contention: the narrowest of the three.
  // This is the "narrowest effective resource" idea as arithmetic.
  function automatic longint unsigned device_ceiling(input device_t d);
    longint unsigned m = d.demand_kBps;
    if (d.link_kBps < m) m = d.link_kBps;
    if (d.path_kBps < m) m = d.path_kBps;
    return m;
  endfunction
 
  // Which resource bound it — the output that actually informs a decision.
  // A number alone says how much; this says what to change to get more.
  function automatic string limiting_resource(input device_t d);
    longint unsigned m = device_ceiling(d);
    if (m == d.demand_kBps) return "demand";   // widening the Link changes nothing
    if (m == d.link_kBps)   return "link";     // width is the constraint
    return "path";                             // topology is the constraint
  endfunction
 
  // A SHARED upstream segment is not another min(): devices contend for it.
  // Proportional scaling is ONE arbitration policy, chosen because it is
  // simple and neutral. Real arbitration may be weighted, priority-based, or
  // strongly non-linear, so treat the shape of the result as the lesson and
  // the exact split as an assumption of this model.
  function automatic void apply_shared_limit(ref longint unsigned est [],
                                             input longint unsigned shared_kBps);
    longint unsigned total = 0;
    foreach (est[i]) total += est[i];
    if (total == 0 || total <= shared_kBps) return;   // no contention
    foreach (est[i]) est[i] = (est[i] * shared_kBps) / total;
  endfunction
 
endclass

Classification: verification-only.

What it models: the ceiling each device faces and which resource imposes it, plus the effect of a shared upstream segment.

What it teaches — and limiting_resource is the part worth keeping: a throughput estimate answers how much; the limiting resource answers what to change. An architecture argument that produces only the first is not actionable, because it does not say whether the fix is more lanes, a different topology, or a workload change.

Deliberately simplified — and these limits matter:

• A single shared segment. Real paths have several, each shared by a different subset of devices.
• Proportional arbitration. Chosen for neutrality. Real arbitration is frequently weighted or priority-based, and can produce sharply different splits from the same inputs.
• Steady state only. No bursts, no queueing, no time-varying demand — so it cannot show that two devices peaking at different times coexist comfortably while two peaking together do not, which §6 identified as a decisive question.
• Bandwidth only. No latency, which is often what actually matters.

Production implication: real capacity planning uses measured demand traces rather than a single rate per device, models queueing and burstiness, accounts for the actual arbitration policy, and treats latency as a first-class objective. This model is for reasoning about the structure of an allocation decision, not for predicting a platform's performance.

10. Unused Width Is Not Automatically Waste

Chapter 6.4 raised this; it deserves resolving before Module 6 closes, because both extreme positions are wrong.

The naive position: an idle wide Link is wasted resources.

The correction: unused capacity is only waste if another allocation would have produced more system value under the design's goals. A platform with lanes to spare and nothing else needing them loses nothing by provisioning generously.

When over-provisioning is genuinely rational:

• Headroom for workloads not yet known, in a platform expected to host varied devices over its life.
• Peak versus average. A workload with a modest average and a high peak may need the width only occasionally, and needing it occasionally is still needing it.
• Uniformity. A platform that provisions every slot identically may be cheaper to build, validate, and support than one tuned per slot — and validation cost is real cost.
• Avoiding a redesign. Width that is difficult to add later can be worth provisioning now, because board and package changes are expensive.

When it is genuinely waste: when those lanes had an alternative use that would have delivered value, and the width was chosen because it was available rather than because it was needed. That is the actual failure mode — provisioning by default rather than by argument.

11. Debugging Versus Architecture

Module 6's most useful classification, and the one that ends most unproductive investigations.

Why this classification is worth more than any single technique. Only the first row is a bug. The other four are cases where the hardware is behaving correctly and an investigation looking for a defect will not find one — and will consume days establishing that.

The measurement that separates them, and it is cheap. Read utilisation from Chapter 6.7's monitor and lane health from Chapter 6.5's telemetry:

• Lane errors present → defect. Go to lane-local diagnosis.
• Clean lanes, high idle → software limitation or provisioning mismatch. The Link was never asked for the traffic.
• Clean lanes, high stall → bottleneck or topology problem. The Link offered and something refused.
• Clean lanes, high active, low payload fraction → transaction size, from Chapter 6.7 §6. Not a width problem at all.
• Clean lanes, high active, good payload fraction, throughput at capacity → nothing is wrong. The Link is doing everything it can, and more width is a genuine option rather than a guess.

Two measurements produce the classification. Without them, every one of the five looks identical from the outside: "it's slower than we expected."

12. Verification

Architectural verification is not RTL verification. The subject is whether an allocation behaves as intended under representative load, and the environment is the model in §9 plus the budget checker in §7.

Scenarios for the budget checker:

• Legal allocation. Widths summing to at most the pool, all powers of two, all active clients non-zero. Verify legal, correct assigned_total, correct headroom.
• Exact fit. Widths summing to exactly the pool. Verify headroom == 0 and legal — the boundary where an off-by-one in the comparison shows.
• Overcommit by one lane. Verify overcommit, !legal, and headroom == 0 rather than a wrapped value.
• Zero-width active client. Verify zero_width_active and !legal.
• Non-power-of-two width. Request x3 or x6. Verify illegal_width fires and is distinguishable from overcommit.
• Commit while traffic depends on the allocation. Verify commit_fault sets and stays set.
• Commit of an illegal allocation while idle. Verify commit_fault also sets — legality and timing are independent reasons to refuse.
• All clients inactive. Verify !legal, since an allocation connecting nothing is a configuration error rather than a valid empty state.

Scenarios for the system model:

• One high-demand device. Verify the ceiling and that limiting_resource names the expected resource as each of demand, link, and path is made the narrowest in turn.
• Several moderate-demand devices, no contention. Sum below the shared limit. Verify no scaling is applied.
• Simultaneous bursts. Sum above the shared limit. Verify proportional scaling and that no device exceeds its own ceiling afterwards.
• One shared upstream bottleneck. A shared limit well below the sum of local capacities. Verify local width increases produce no change in delivered throughput — the scenario that demonstrates the chapter's central point.
• An underutilised wide Link. Demand well below link capacity. Verify limiting_resource returns "demand", so the model reports that widening changes nothing.
• Narrow Links with parallel workloads. Several devices each saturating a modest Link, aggregate below the shared limit. Verify total delivered exceeds the single-wide-Link arrangement — §5's comparison made checkable.
• One device inactive. Verify the remaining devices absorb the freed shared capacity under the model's policy.
• Dynamic workload mix. Sweep the demand vector across several profiles and verify the limiting resource changes as expected — which is the point of computing it rather than only the number.

13. Common Misconceptions

• "The widest Link is always best." Width raises only layer 5 of Chapter 6.7's stack. If the constraint is demand, topology, memory, transaction size, or software, additional lanes raise a ceiling nothing reaches while the costs in §1 are paid in full.
• "Unused lanes are always wasted." Unused capacity is waste only if another allocation would have produced more value under the design's goals. Headroom for future workloads, for peaks, or for uniformity across a platform can all be rational.
• "An x16-capable device deserves x16." What a device supports is not what a system should give it. The decision is about the workload and the budget, and a device that cannot sustain demand at x16 is served identically by less.
• "Lane count alone determines system performance." It determines one segment's potential capacity. Delivered performance depends on the whole path, the workload, and the software — which is why §11's classification has five rows and only one of them is about the Link.
• "Widening the local Link fixes an upstream bottleneck." A constrained segment elsewhere is unaffected by this Link's width. Widening past a constraint changes nothing downstream of it.
• "More devices always need more lanes." More devices need more lanes only if they generate demand that the shared path can carry. Adding devices behind a saturated upstream segment divides the same capacity into more streams.
• "Power is the only cost, and only when traffic is active." The design-time costs — pins, PHY area, board routes and layers, layout effort, validation — are paid whether the width is used or not, and they frequently dominate the decision.
• "Width planning is a software problem." Software may request a configuration, but width determines package connectivity, PHY instances, board routing, and power at design time. Those decisions are made long before any software runs.
• "Every allocation should maximise peak throughput." Peak throughput is one objective among several. Latency, the number of functions connected, cost, power, thermal budget, and platform uniformity are all legitimate and sometimes decisive.
• "The fastest topology is the best system." Best is defined by the design's goals. A system that connects the functions it needs, within its power and cost budget, with adequate performance, is better than a faster one that does not fit.

14. Understanding Check

15. Module 6 Complete

Eight chapters, one argument.

6.1 — x1. A lane is two differential pairs, one per direction. A lane is not a Link. An x1 Link is one lane wide, and width and generation are independent dimensions.

6.2 — x2. The second lane introduces a change in kind: one logical stream must be distributed across two physical paths, reconstructed, and reconciled — problems that do not exist at x1 and do not fundamentally change beyond x2.

6.3 — x4. Multi-lane becomes practical, and observability stops being optional. Failure combinations grow faster than lane count, so the decisive question shifts from how it works to how you will know where it broke.

6.4 — x8. Lanes are a finite resource drawn from a pool. Committing eight to one connection is a choice against connecting more devices, and ownership becomes a hardware invariant.

6.5 — x16. At scale, aggregate error counts stop carrying usable information and telemetry must become distributional. And the wider the Link, the less likely it is the constraint.

6.6 — Aggregation. How lanes actually act as one Link: distribute, transport, absorb arrival differences, reassemble. Lane-local health proves lane transport and never proves aggregate correctness.

6.7 — Throughput. Seven layers from signalling rate to delivered bytes. Every figure belongs to exactly one layer, and a number without its basis is not an answer.

6.8 — Trade-offs. Width is a resource-allocation decision made against a workload, a topology, and a budget — not a number to maximise.

The through-line: a Link's width is a statement about resources, not about performance. Module 6 began by defining the resource precisely and ends by deciding how much of it to spend. Everything in between — coordination, observability, allocation, aggregation, arithmetic — exists to make that final decision an argument rather than a guess.

16. What's Next

Module 6 answered how many lanes a Link has and how they behave. It never asked how the system finds the devices at the other end of those Links, learns what they are, or gives them the addresses software uses to reach them.

Module 7 — Enumeration and Configuration takes that up: how software discovers what is present, how buses and devices are numbered, how configuration space is reached, and how memory and I/O resources are assigned. It is the point at which PCIe stops being a transport and becomes a system that software can use.