PCIe Gen2 — Doubling the Rate, and When That Doesn't Help

Chapter 5.1 established the derivation: signalling rate, encoding efficiency, byte rate — and the discipline of keeping units visible so the steps cannot be skipped.

Gen2 provides an unusually clean opportunity to use it, because it changes exactly one variable:

What changed in Gen2, what deliberately stayed the same, and why does doubling the signalling rate not automatically mean every workload doubles throughput?

The first two questions take a section. The third takes most of the chapter, and it is the part that matters in practice.

1. One Variable Changed

Gen2's headline figure is 5 GT/s per lane, per direction — twice Gen1's 2.5 GT/s.

The encoding is unchanged: still 8b/10b, still 80% efficient.

That combination is what makes the comparison clean. In a controlled experiment you change one variable and hold the rest fixed; Gen2 relative to Gen1 is close to exactly that. The signalling rate doubled, the encoding efficiency did not move, and therefore the encoding-adjusted capacity doubles — with no interaction between the two effects to disentangle.

It is worth appreciating that this is unusual. Later generations change more than the rate, including the encoding scheme itself, and their comparisons are correspondingly less clean. Chapter 5.3 takes up the generation where that happens.

2. The Derivation, Repeated Deliberately

Run the same four steps from Chapter 5.1, with the one changed input.

Written out for Gen2:

Step 1 — signalling rate. 5.0 × 10⁹ transfers/second, per lane, per direction.

Step 2 — apply encoding. 5.0 GT/s × (8 data bits / 10 transmitted bits) = 4.0 Gb/s

Step 3 — convert to bytes. 4.0 Gb/s ÷ 8 bits/byte = 0.5 GB/s

Step 4 — express conventionally. 0.5 GB/s = 500 MB/s (decimal units)

All figures per lane, per direction, theoretical, before higher-layer overhead.

The ratio column is the useful part: because efficiency is constant, the doubling propagates cleanly and the entire improvement is attributable to the signalling rate. That is a rare and clean attribution, and it is exactly what makes Gen2 a good teaching case.

3. Why "Same Encoding" Is the Interesting Part

It would have been possible to improve capacity by making the encoding more efficient instead of — or as well as — raising the rate. Gen2 did not; it kept 8b/10b and pushed the signalling rate.

Two things follow.

The improvement came from the physical layer. Doubling the signalling rate is a signal-integrity problem: shorter unit intervals, tighter timing, more demanding channel behaviour — the concerns Chapter 3.3 described. The protocol above was largely unaffected, which is part of why the transition was manageable.

The 20% encoding cost persisted. At Gen2 rates that overhead is larger in absolute terms than at Gen1: one fifth of 5 GT/s rather than one fifth of 2.5 GT/s. As signalling rates climb, a fixed-percentage encoding cost becomes an increasingly expensive thing to keep paying — which is the pressure that eventually motivated a different approach.

That is the forward bridge, and this chapter goes no further with it: later generations change more than the rate, including the encoding scheme and its efficiency. The specifics belong to Chapter 5.3.

4. When Throughput Does Not Double

Here is the chapter's substance. Doubling a Link's capability doubles observed throughput only if that Link was the constraint. Six common cases where it was not — and how to tell.

Device-limited. The device's own logic cannot produce or consume data fast enough. Its engines, internal buffering, or processing set the rate, and the Link is idle much of the time. Signature: high idle in the utilisation counters from Chapter 5.1 — the source is simply not presenting data.

Memory-limited. The host memory subsystem is the constraint. All device traffic targeting host memory converges there, shared with the processor and other devices, as Chapter 4.1 established. Signature: backpressure from the host side, and throughput that varies with unrelated system memory activity.

Shared-upstream-limited. The device's own Link got faster; a shared segment further along the path did not. A device behind a Switch whose upstream segment is saturated gains nothing from a faster local Link — Chapter 4.2's central point. Signature: other devices behind the same Switch also constrained, and the limit tracks aggregate rather than individual demand.

Overhead-dominated. Small transfers spend a large proportion of Link time on per-packet overhead rather than payload. Doubling the rate moves the overhead and payload through twice as fast, so this case does scale — but the efficiency stays poor, so if the expectation was "Gen2 fixes my small-transfer efficiency," it does not. Signature: measured payload throughput far below capacity while the Link is well utilised, with small beat sizes.

Software-limited. The submission and completion pipeline cannot sustain the rate — driver overhead, interrupt handling, or the rate at which work is queued. Signature: high idle again, but correlated with CPU utilisation rather than device state.

Not actually operating at Gen2. The Link's operational capability must match the assumption. A Link that is not running at the expected rate will not deliver the expected capacity, and this is worth eliminating before any deeper analysis. How the operational rate is established is a later topic — Module 18 — and the point here is only to verify it rather than assume it.

5. Extending the Rate Model

The Chapter 5.1 package generalises to Gen2 with no change to its logic — only new inputs, which is the code-level reflection of §1's single-variable change.

// Compile-time only (elaboration constants and pure functions).
// Extends the Gen1 package: same functions, different inputs. That the logic
// is unchanged is the point — only the signalling rate differs between the
// two generations.
package pcie_gen2_rate_pkg;
  import pcie_rate_pkg::*;
 
  // Transfer rates in MEGA-transfers/s so both are exact integers.
  localparam longint unsigned GEN1_MTPS = 2500;   // 2.5 GT/s
  localparam longint unsigned GEN2_MTPS = 5000;   // 5.0 GT/s
 
  // 8b/10b — identical for both generations.
  localparam int unsigned ENC_NUM = 8;
  localparam int unsigned ENC_DEN = 10;
 
  // Per lane, per direction, theoretical, encoding-adjusted.
  localparam longint unsigned GEN1_MBPS  = encoded_mbits_per_s(GEN1_MTPS, ENC_NUM, ENC_DEN); // 2000
  localparam longint unsigned GEN2_MBPS  = encoded_mbits_per_s(GEN2_MTPS, ENC_NUM, ENC_DEN); // 4000
  localparam longint unsigned GEN1_MBYTES = mbytes_per_s(GEN1_MTPS, ENC_NUM, ENC_DEN);       // 250
  localparam longint unsigned GEN2_MBYTES = mbytes_per_s(GEN2_MTPS, ENC_NUM, ENC_DEN);       // 500
 
  // Ratio expressed in hundredths to stay in integer arithmetic — 200 = 2.00x.
  localparam longint unsigned GEN2_OVER_GEN1_X100 =
      (GEN2_MBYTES * 100) / GEN1_MBYTES;   // 200
 
endpackage

Classification: compile-time only.

What it teaches: that the generational change is a parameter change, not a model change — the same derivation applies, which is precisely the claim §1 makes in prose.

Deliberately simplified: no lane count, since Link width is Module 6's subject; no protocol overhead, which is workload-dependent rather than constant; and a ratio in hundredths to avoid floating point.

What to notice: GEN2_OVER_GEN1_X100 evaluates to exactly 200, confirming the clean 2× at the encoding-adjusted level. That the arithmetic is exact is a consequence of efficiency being identical — a comparison across generations with different encodings would not divide so neatly, which is itself informative.

6. Measuring Where the Time Actually Goes

To answer "why didn't it double," you need to know what the Link was doing. Chapter 5.1's counters classify cycles; this extends them with what a comparative investigation needs.

// Illustrative synthesizable RTL — comparative throughput instrumentation.
// NOT a PCIe controller. Adds to simple cycle classification the two things a
// generation comparison needs: payload-per-beat distribution and a snapshot
// mechanism so windows can be compared like for like.
module throughput_window #(
  parameter int unsigned CNT_W        = 48,
  parameter int unsigned SMALL_BEAT_B = 64      // "small" payload threshold, bytes
) (
  input  logic              clk,
  input  logic              rst_n,
 
  // Observed interface — watched only, never driven.
  input  logic              obs_valid,
  input  logic              obs_ready,
  input  logic [15:0]       obs_bytes,
 
  // Occupancy of the source's outgoing queue, for starvation diagnosis
  input  logic [15:0]       src_occupancy,
 
  input  logic              win_start,
  input  logic              win_stop,
 
  // Snapshot — frozen at win_stop so software reads a coherent set
  output logic [CNT_W-1:0]  snap_cycles,
  output logic [CNT_W-1:0]  snap_transfer,
  output logic [CNT_W-1:0]  snap_stalled,
  output logic [CNT_W-1:0]  snap_idle,
  output logic [CNT_W-1:0]  snap_bytes,
  output logic [CNT_W-1:0]  snap_small_beats,   // beats below the threshold
  output logic [15:0]       snap_occ_watermark,
  output logic              snap_valid
);
 
  logic [CNT_W-1:0] c_cycles, c_transfer, c_stalled, c_idle, c_bytes, c_small;
  logic [15:0]      occ_hi;
  logic             running_q;
 
  wire transfer = obs_valid &&  obs_ready;
  wire stalled  = obs_valid && !obs_ready;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      running_q  <= 1'b0;
      snap_valid <= 1'b0;
      c_cycles   <= '0; c_transfer <= '0; c_stalled <= '0;
      c_idle     <= '0; c_bytes    <= '0; c_small   <= '0;
      occ_hi     <= '0;
    end else if (win_start) begin
      running_q  <= 1'b1;
      snap_valid <= 1'b0;
      c_cycles   <= '0; c_transfer <= '0; c_stalled <= '0;
      c_idle     <= '0; c_bytes    <= '0; c_small   <= '0;
      occ_hi     <= '0;
    end else if (running_q && win_stop) begin
      // Freeze the whole set in one cycle so the snapshot is self-consistent.
      running_q          <= 1'b0;
      snap_cycles        <= c_cycles;
      snap_transfer      <= c_transfer;
      snap_stalled       <= c_stalled;
      snap_idle          <= c_idle;
      snap_bytes         <= c_bytes;
      snap_small_beats   <= c_small;
      snap_occ_watermark <= occ_hi;
      snap_valid         <= 1'b1;
    end else if (running_q) begin
      c_cycles <= c_cycles + 1'b1;
 
      if (transfer)     c_transfer <= c_transfer + 1'b1;
      else if (stalled) c_stalled  <= c_stalled  + 1'b1;
      else              c_idle     <= c_idle     + 1'b1;
 
      if (transfer) begin
        c_bytes <= c_bytes + obs_bytes;
        // Small beats are counted separately: a window dominated by them
        // explains poor efficiency without any Link fault.
        if (obs_bytes < SMALL_BEAT_B[15:0]) c_small <= c_small + 1'b1;
      end
 
      if (src_occupancy > occ_hi) occ_hi <= src_occupancy;
    end
  end
endmodule

Classification: synthesizable.

What it teaches: that a generation comparison needs more than a byte count. Small-beat counting distinguishes an overhead-dominated workload from a genuinely constrained Link, and source occupancy watermark distinguishes a starved source from a blocked one — the two most common explanations for a disappointing upgrade.

Deliberately simplified: one direction observed; a single snapshot rather than rolling history; a fixed small-beat threshold rather than a full distribution; and counters that could saturate on an extremely long window.

What to notice:

• The snapshot freezes atomically at win_stop, so software reads a mutually consistent set. Reading live counters produces values from different instants, and derived percentages then do not sum correctly — a real and confusing measurement bug.
• snap_valid marks the snapshot as populated, so software cannot mistake a cleared window for a measured one.
• Small beats and occupancy are diagnostic, not performance metrics. They exist to explain why the primary numbers look the way they do.

Production RTL would additionally require: both directions instrumented, a proper histogram rather than a single threshold, rolling windows for continuous monitoring, and a defined software-visible access mechanism.

7. Assertions

// SVA over the illustrative instrumentation. Implementation invariants — not
// PCIe protocol requirements.
 
// P1 — CLASSIFICATION EXHAUSTIVE while running. Every counted cycle lands in
// exactly one class. A violation silently corrupts every derived percentage,
// and produces plausible-looking wrong answers rather than obvious ones.
property p_class_sums;
  @(posedge clk) disable iff (!rst_n)
  running_q |-> (c_cycles == c_transfer + c_stalled + c_idle);
endproperty
a_class_sums : assert property (p_class_sums);
 
// P2 — BYTES ONLY ON ACCEPTED BEATS. Counting on valid alone inflates the
// figure by every stalled cycle and can report impossible throughput.
property p_bytes_on_transfer_only;
  @(posedge clk) disable iff (!rst_n)
  (c_bytes != $past(c_bytes)) |-> $past(obs_valid && obs_ready);
endproperty
a_bytes_on_transfer : assert property (p_bytes_on_transfer_only);
 
// P3 — SMALL BEATS ARE A SUBSET OF TRANSFERS. Cannot exceed the number of
// accepted beats; a violation means the classification is double-counting.
property p_small_subset;
  @(posedge clk) disable iff (!rst_n)
  c_small <= c_transfer;
endproperty
a_small_subset : assert property (p_small_subset);
 
// P4 — SNAPSHOT IS ATOMIC AND STABLE. Once taken, the set does not change
// until a new window starts, so software reads a coherent measurement.
property p_snapshot_stable;
  @(posedge clk) disable iff (!rst_n)
  (snap_valid && !win_start) |=> ($stable(snap_cycles) && $stable(snap_bytes)
                                  && $stable(snap_transfer));
endproperty
a_snapshot_stable : assert property (p_snapshot_stable);
 
// P5 — WATERMARK IS MONOTONIC within a window. It records a maximum, so it
// must never decrease while running.
property p_watermark_monotonic;
  @(posedge clk) disable iff (!rst_n)
  (running_q && !win_start) |=> (occ_hi >= $past(occ_hi));
endproperty
a_watermark_monotonic : assert property (p_watermark_monotonic);

P1 is again the critical one, for the same reason as in Gen1: derived percentages are only as trustworthy as the classification, and a classification bug produces believable wrong numbers rather than obvious ones.

P4 catches the atomicity failure. Without it, software can read snap_cycles from one window and snap_bytes from the next, computing a throughput that never occurred. This is genuinely hard to spot in results because the number is plausible — which is precisely why an assertion is worth more than inspection.

P5 catches a watermark that resets or decreases mid-window, which would understate peak pressure and hide the very starvation the counter exists to reveal.

8. Verification: Comparing Generations Honestly

The distinctive verification task here is controlled comparison, and it is easy to do badly.

Same workload, both rate models. Run an identical workload against a Gen1-rate and a Gen2-rate model and compare. The expected outcome depends entirely on whether the workload saturated Gen1 — so measure Gen1 utilisation first, and treat that measurement as a precondition for interpreting the comparison at all.

Saturating workload. Source always has data, sink always accepts, large beats. Should approach the ceiling in both cases and show close to 2× improvement. If it does not, either the model is wrong or something else is limiting — and the counters say which.

Low-demand workload. Source presents data rarely. Both configurations should show similar application throughput, and both should show high idle. This is the case that most often gets misreported as "Gen2 didn't work," and having it as a deliberate test makes the expected outcome explicit.

Upstream-bottleneck scenario. Model a constrained shared segment beyond the Link under test. Doubling the local Link should produce little or no end-to-end improvement — the Module 4 point, made measurable.

Small transfers. Beats well below the threshold. Both configurations show poor efficiency; Gen2 moves the same inefficient traffic faster, so absolute throughput improves while efficiency does not. Distinguishing those two outcomes is the test's purpose.

Sustained large transfers. Closest to theoretical, and still below the encoding-adjusted ceiling because packet and link-level overhead remain. Do not encode a specific expected percentage — it depends on transaction size and implementation, and a hard-coded figure will produce false failures.

Coverage worth defining: each limiting case (device, memory, upstream, overhead, software) exercised at both rates; utilisation dominated by each cycle class; small-beat-dominated and large-beat-dominated windows; snapshot taken during each cycle class.

9. Debugging: "Expected 2×, Observed 1.2×"

This is the report you will actually receive. It is not evidence of a fault, and treating it as one wastes time. A checklist, ordered by likelihood.

1. Was Gen1 actually saturated? The governing question from §4. If the Gen1 measurement showed substantial idle or stall, the Link was not the constraint and 1.2× may be the correct result. Check this before anything else — it resolves the majority of such reports.

2. Is the Link actually operating at the expected rate? Verify rather than assume. A Link not running at Gen2 rates cannot deliver Gen2 capacity, and this is cheap to eliminate.

3. Is the source generating enough demand? High idle means the Link is waiting for data. Doubling the Link's rate does not make a device produce data faster.

4. Is the sink applying backpressure? High stall means something downstream is the constraint — possibly several hops away, per Chapter 4.2.

5. Is a shared upstream segment the limit? If the device sits behind a Switch, its own Link is only the first hop. Check whether other devices on the same shared segment are also constrained — a strong indicator.

6. Is host memory the limit? If the Link is well utilised and the path is clear, the convergence at the memory subsystem may be binding. Correlate with unrelated system memory activity.

7. Are transfers too small? Check the small-beat count. An overhead-dominated workload improves in absolute terms while staying inefficient — which looks like a partial improvement and is arithmetically expected.

8. Is software the limit? Submission and completion processing can cap throughput regardless of Link capability. Correlate idle time with host CPU activity.

9. Was the comparison controlled? Confirm nothing else changed between the two measurements — same workload, same transfer sizes, same system state, same measurement window length.

1.2× is a result, not a failure. It tells you the Link contributed something and that another resource is now dominant. The productive next step is identifying which one, not re-examining the Link.

10. Common Misconceptions

11. Understanding Check

12. Summary

Gen2 changed one variable: the per-lane signalling rate doubled to 5.0 GT/s, while 8b/10b encoding stayed unchanged at 80% efficiency.

The derivation therefore runs identically to Gen1 with one different input:

5.0 GT/s × 8/10 = 4.0 Gb/s → 4.0 Gb/s ÷ 8 = 0.5 GB/s → 500 MB/s

theoretical, encoding-adjusted, per lane, per direction — exactly 2.00× Gen1's 250 MB/s, with the entire improvement attributable to signalling rate.

The important lesson is the one the clean arithmetic does not deliver: useful workload throughput only doubles when the Link was the constraint. Six common alternatives — device-limited, memory-limited, shared-upstream-limited, overhead-dominated, software-limited, or not actually operating at the expected rate — each produce a smaller improvement without anything being broken. The governing question before any investigation is "was Gen1 actually saturated?", and utilisation instrumentation answers it directly.

For measurement, a generation comparison needs more than a byte count: cycle classification to distinguish idle from stalled from transferring, small-beat counting to identify overhead-dominated workloads, and an atomic snapshot so derived figures are self-consistent. And the comparison itself must be controlled — exactly one variable different, verified rather than assumed.

Hold the model: the capacity doubled cleanly; whether your workload doubles depends on what was limiting it.

13. What Comes Next

Gen1 and Gen2 share an encoding, which made their comparison unusually clean. Chapter 5.3 takes up the generation where that stops being true — where more than the signalling rate changes, including the encoding scheme and its efficiency, so the derivation itself must be revisited rather than merely re-parameterised.

Module 6 takes up Link width, which both of these chapters deliberately held at one lane. Its Chapter 6.7 owns throughput calculation combining rate, width, and encoding — the point at which these per-lane figures become system-level numbers.

Revisit PCIe Gen1 for the derivation this chapter re-parameterises, or Endpoints in Topology for why a faster Link may not be the constraint. Browse the full path on the PCIe tutorials index.