Per-Lane / Per-Module Bandwidth

Module 14 asked whether the link can be trusted. This module asks what it delivers when it can.

1. The One-Sentence Model

Raw bandwidth counts transferred bits. Useful bandwidth counts meaningful payload bits. They are never equal, and the ratio between them is a design output rather than a constant.

Every bandwidth mistake in this chapter comes from one of two places: a unit that was not tracked, or a factor between raw and useful that was not accounted for. Sections 4 to 9 handle the first. Sections 18 to 24 handle the second.

2. What This Chapter Owns

Chapter 13.5 already built the throughput argument — initiation interval, the concurrency conjunction, stall attribution. This chapter is not that argument again.

Specifically new here: units, treated as a first-class correctness concern; GT/s against Gb/s and the one place official material lets us verify the conversion; the verified lane / cluster / link terminology; the raw capacity formula and its overflow hazard in RTL; the four-factor efficiency chain — framing, occupancy, utilisation, replay — each defined so they multiply without double-counting; event-qualified counters and the valid-without-ready bug that reports impossible throughput; atomic snapshots; counter rollover; and the roofline argument that the link is frequently not the limit.

3. Sourcing

4. Units, Before Anything Else

This section exists because units cause more bandwidth errors than every other cause combined.

Four rules that eliminate most errors:

Always write the unit in the identifier (§17). rate_gtps, raw_gbps, payload_bytes — never rate or bw.

Always state the direction. A bandwidth figure without a direction is ambiguous by a factor of two.

Always state whether it is raw or useful. These differ by the whole of §18–§24.

And check that dimensions cancel. Every formula in this chapter is written so the units visibly reduce to the claimed result. A formula whose units do not cancel is wrong regardless of how plausible the number looks.

5. From Transfers to Bytes

transfers/s per lane  ──× lanes──▶  transfers/s total
                      ──× bits per transfer──▶  bits/s
                      ──÷ 8──▶  bytes/s
                      ──÷ 10⁹──▶  GB/s     (decimal)
                      ──÷ 2³⁰──▶  GiB/s    (binary)

The two conversions that are not free:

Transfers to bits needs bits_per_transfer, which is a property of the signalling scheme and must not be assumed to be 1 for every mode and revision (§6).

Bytes to GB/s or GiB/s is a choice, and the two differ by about 7.4%:

1 GB/s  = 1,000,000,000 bytes/s
1 GiB/s = 1,073,741,824 bytes/s
ratio   = 1.073741824        →  a 7.4% discrepancy from labelling alone

7.4% is larger than most of the efficiency factors this chapter spends sections on. A design that optimises payload occupancy by 3% and then reports GiB/s as GB/s has lost more than it gained, in the report rather than in the silicon.

6. GT/s Is Not Gb/s — and What Official Material Lets Us Check

GT/s counts transfers. Gb/s counts bits. They coincide only when one transfer carries exactly one bit per lane, and that is a property of the signalling scheme rather than a definition.

Official material lets us verify it for one specific case, which is worth doing rather than assuming. UCIe 2.0 states a dedicated management port can be "a half-width (x8), or higher, UCIe-S delivering 256 Gb/s/direction for each x8 at 32 GT/s" (§3).

Check the arithmetic:

Given (official):   width         = 8 lanes  (x8)
                    transfer rate = 32 GT/s per lane
                    result        = 256 Gb/s per direction
 
Test the 1-bit-per-transfer-per-lane hypothesis:
 
  bits/s = 8 lanes × 32 × 10⁹ transfers/s/lane × 1 bit/transfer
         = 256 × 10⁹ bit/s
         = 256 Gb/s per direction                    ✓ matches
 
Units:  lanes × (transfers / s / lane) × (bits / transfer) = bits / s   ✓ cancel

The hypothesis reproduces the official figure exactly, so for UCIe-S at 32 GT/s the conversion is one bit per transfer per lane. That is a verified data point, not an assumption.

A second verified data point, at a very different scale, from the same sentence: the sideband is "800 Mb/s/direction with 4 bumps", and §3's cluster description gives the sideband as "one 800 MHz clock and one data" per direction. One data lane at 800 MHz giving 800 Mb/s is again one bit per transfer — consistent, and a useful sanity check that the two statements agree.

7. Lane, Cluster, Link — Verified Terminology

These three words are not interchangeable, and using them loosely is how a bandwidth calculation ends up off by the cluster count.

Two consequences that change the arithmetic.

Only the data lanes carry payload. A cluster contains a Valid lane, a tracking lane, a forwarded clock and sideband lanes — and none of them carries transaction payload. They are essential and they are overhead in the physical sense: a standard-package cluster has 16 data lanes but more than 16 signals.

"Per link" needs a cluster count. A figure quoted "per x16" is per cluster on a standard package; a link may aggregate several. Asking "what is the bandwidth?" without stating lanes and clusters is an under-specified question, and §12's worked example states both.

8. Raw One-Direction Capacity

raw_bit_rate_per_direction
    = transfer_rate_per_lane        [transfers / s / lane]
    × active_data_lanes             [lanes]
    × bits_per_transfer             [bits / transfer]
 
Units:  (transfers / s / lane) × lanes × (bits / transfer)  =  bits / s   ✓

Every symbol defined:

Three modelling notes.

This is one direction. The lanes are described as "unidirectional, full-duplex" (§3), so TX and RX are separate resources. §10 develops what that does and does not permit.

active_data_lanes is runtime state, not a constant. A masked lane on a standard package reduces it (14.4 §4); on an advanced package a spare may keep it constant. A capacity model that hard-codes the configured width reports the wrong number after any repair — which is 14.5 §24's stale-status bug arriving in the performance model.

And this is a ceiling. Nothing in §18 onward can raise it; everything can lower the useful figure below it.

9. Bytes, and Which Kind

raw_bytes_per_second = raw_bit_rate / 8
 
raw_GB_per_second    = raw_bytes_per_second / 1e9      [decimal]
raw_GiB_per_second   = raw_bytes_per_second / 2**30    [binary]

Pick one convention, state it, and use it everywhere. Mixing them within a document produces figures that disagree by 7.4% for no physical reason, and reconciling that later costs more than getting it right costs now.

This chapter uses decimal GB/s throughout, because the rate inputs are decimal (32 GT/s means 32 × 10⁹) and mixing a decimal rate with a binary byte conversion is the specific error that produces an unexplainable 7.4%.

10. Direction

The lanes are unidirectional and full-duplex (§3), so a link has a TX capacity and an RX capacity, and they are separate physical resources.

The trap, stated plainly: a link with 256 Gb/s per direction is sometimes quoted as "512 Gb/s aggregate". That number is real and it is not available to a one-directional workload. A bulk read stream uses one direction heavily and the other only for responses and acknowledgements, so quoting the aggregate overstates what that workload can achieve by a factor of two.

And there is a real dependency in the other direction. 13.1 §11 and 14.3 §26 both established that credit returns and acknowledgements ride in headers travelling the opposite way. So a workload that uses one direction almost exclusively still needs the reverse direction to carry control, and if it does not, the forward direction stalls. The directions are separate capacities that are not independent.

11. Where Bandwidth Is Lost

Read the figure as a sequence of shrinking quantities. Semantic payload is what was asked for; transported bytes are more than that because of framing; lane traffic is more again because of replay; and the lanes are idle some of the time. Each arrow is a place bandwidth is lost, and each counter is what makes the loss attributable rather than inferred.

12. Worked Raw Capacity

Using only verified inputs, for a standard-package cluster.

INPUTS (all verified, Section 3)
  package               = standard (UCIe-S)
  data lanes per cluster= 16                        [UCIe 1.0: N = 16 for standard package]
  transfer rate         = 32 GT/s per lane          [UCIe 1.0 / 2.0 rate]
  bits per transfer     = 1                         [verified for UCIe-S at 32 GT/s, Section 6]
  clusters              = 1                         [stated explicitly]
 
RAW, ONE DIRECTION
  raw_bit_rate = 32e9 transfers/s/lane × 16 lanes × 1 bit/transfer
               = 512e9 bit/s
               = 512 Gb/s per direction
 
  raw_bytes    = 512e9 / 8 = 64e9 byte/s
               = 64 GB/s per direction (decimal)
 
CROSS-CHECK against the official x8 figure
  half the lanes should give half the bandwidth:
  8 lanes × 32e9 × 1 = 256e9 bit/s = 256 Gb/s per direction
  → matches the official "256 Gb/s/direction for each x8 at 32 GT/s"   ✓

The cross-check is the point of the example. The x16 figure is derived; the x8 figure is quoted. Deriving a number and then reproducing an independently stated one from the same model is how you establish the model is right — and it is what separates a capacity calculation from a guess.

Two things this figure is not.

It is not the link's bandwidth unless the link is one cluster. Multiple clusters aggregate (§7, §14).

And it is not deliverable. It is the ceiling before any of §18–§24 applies.

13. Lane-Count Scaling

Raw capacity is exactly linear in active data lanes — the lanes are independent signals and each carries its own transfers:

raw_bit_rate ∝ active_data_lanes

Useful throughput is not linear in lanes, and the reasons are the whole of the rest of the chapter: framing overhead is a fixed cost per transport unit rather than per lane; payload occupancy depends on the traffic; cycle utilisation depends on whether the upper layers can keep the link fed; and the replay fraction depends on the error rate. Halving the lanes halves the ceiling and does something else entirely to the delivered figure (§35).

14. Cluster Aggregation

Verified: "Multiple clusters can be aggregated to deliver more performance per Link" (§3).

raw_link_bit_rate = raw_cluster_bit_rate × cluster_count

Two modelling cautions.

Aggregation multiplies the ceiling, not necessarily the delivered throughput. If the binding constraint is upstream — the producer, the outstanding-transaction pool, the credit round trip (13.5 §17) — then adding clusters raises a ceiling that was not the limit. This is §39's roofline argument in its most common form.

And each cluster carries its own non-payload signals. Each brings a Valid lane, a tracking lane, a forwarded clock and sideband (§7). The signal count grows faster than the data-lane count, which is a floorplan and package-escape cost that a pure bandwidth figure does not show.

15. Capacity Arithmetic in RTL

// ILLUSTRATIVE capacity parameters. Note the TYPES — this is where the bug in
// Section 16 lives, and it is an arithmetic bug rather than a logic bug.
//
// Values are illustrative except where marked; the rate is a verified UCIe
// rate and the lane count a verified cluster size (Section 3).
localparam longint RATE_TPS_PER_LANE = 64'd32_000_000_000;  // 32 GT/s, verified rate
localparam int     ACTIVE_DATA_LANES = 16;                  // verified cluster size
localparam int     BITS_PER_TRANSFER = 1;                   // verified for this case (Sec 6)
 
// longint is REQUIRED here. See Section 16.
localparam longint RAW_BITS_PER_SEC =
    RATE_TPS_PER_LANE * longint'(ACTIVE_DATA_LANES) * longint'(BITS_PER_TRANSFER);
 
localparam longint RAW_BYTES_PER_SEC = RAW_BITS_PER_SEC / 64'd8;
 
// Elaboration-time sanity checks. These are free and they catch the whole
// class of bug in Section 16.
initial begin
  assert (RAW_BITS_PER_SEC == 64'd512_000_000_000)
    else $fatal(1, "raw capacity arithmetic wrong: got %0d", RAW_BITS_PER_SEC);
  assert (RAW_BITS_PER_SEC > 0)
    else $fatal(1, "raw capacity overflowed to a non-positive value");
end

Architecture. Compile-time capacity constants, used for sizing decisions and as a reference for the counters of §25 — not as a synthesisable datapath value. Nothing in the datapath needs to know 512 billion of anything.

State. None — these are elaboration-time.

Cycle behaviour. None. That is worth stating explicitly, because the most common misuse of this arithmetic is to compute a rate in hardware at runtime, which requires a divider and buys nothing (§30's rule: hardware counts, software divides).

Contract. ACTIVE_DATA_LANES here is a configured value. The runtime active count comes from the lane mask (14.4 §17) and can be lower. A model that conflates the two reports pre-degradation capacity forever.

Failure. §16.

DV. The initial assertions are the test, and they are elaboration-time so they run in every configuration the design supports. Cheap, and they catch a reparameterisation that silently overflows.

16. Wrong Arithmetic — 32-Bit Overflow

// WRONG — the multiplication is performed in 32-bit integer arithmetic.
localparam int RATE_TPS = 32_000_000_000;      // ← already overflowed
localparam int RAW_BPS  = RATE_TPS * 16;       // ← nonsense

Two separate failures in two lines.

Line 1 overflows before any multiplication. A SystemVerilog int is 32-bit signed, with a maximum of 2,147,483,647. 32 × 10⁹ does not fit. The declaration alone is already wrong.

Line 2 compounds it. Even with a rate that fitted, rate × 16 at 32 bits overflows for any rate above about 134 MT/s.

What makes this a genuinely dangerous bug rather than an obvious one:

The result is not a crash — it is a quietly wrong design decision, which is the worst kind. A buffer sized from an overflowed bandwidth figure is under-provisioned by whatever factor the wrap introduced, and the sizing rationale in the design document is arithmetically indefensible but reads as authoritative.

Three defences, in order of value:

Use longint for anything counting bits, bytes or hertz. 64-bit signed reaches 9.2 × 10¹⁸, which is comfortable for any link bandwidth.

Assert the expected value at elaboration (§15). A single assert against a hand-computed constant catches every variant of this.

And prefer scaled units where the numbers permit. Working in Gb/s rather than bit/s keeps values in the hundreds, which fits an int with enormous margin — at the cost of remembering the scale, which is what §17 is for.

17. Units in Identifiers

// GOOD — the unit is in the name, so a mismatched assignment is visible.
localparam longint rate_gtps_x1000   = 64'd32_000;    // 32.000 GT/s in milli-GT/s
localparam longint raw_gbps          = 64'd512;       // Gb/s
localparam longint raw_gbytes_ps     = 64'd64;        // GB/s
logic [63:0]       transmitted_bytes_q;
logic [63:0]       unique_payload_bytes_q;
logic [31:0]       observation_cycles_q;
 
// BAD — every one of these needs a comment to be safe, and comments drift.
// localparam int rate;
// localparam int bw;
// logic [63:0] count;

Why this is a correctness measure and not a style preference. A bandwidth model has half a dozen quantities that are all "a number of things per second", and the compiler cannot distinguish them. Putting the unit in the identifier moves a class of error from runtime and undetectable to visible at the point of assignment.

The specific errors it prevents: assigning Gb/s to a GB/s variable (factor 8); assigning GiB/s to a GB/s variable (factor 1.074); assigning a per-lane rate to a whole-link variable (factor lanes); and assigning an aggregate figure to a per-direction variable (factor 2). Four common errors, four different factors, all invisible without the naming discipline.

And the same applies to counters. transmitted_bytes_q and unique_payload_bytes_q differ by exactly the replay tax (§23) — naming them bytes_q and bytes2_q guarantees someone eventually divides by the wrong one.

18. The Efficiency Chain

Four factors sit between raw capacity and delivered payload, and they are defined here so they multiply cleanly rather than double-counting.

The definitions are deliberately disjoint, and that is what makes the product valid:

• framing efficiency is about the container's shape and is a constant for a given format;
• occupancy is about how full the containers were;
• utilisation is about whether containers were sent at all in a given cycle;
• the replay fraction is about whether a container carried anything new.

Get the definitions overlapping and the product is wrong. The classic overlap is counting a partially filled unit as both an occupancy loss and a framing loss — it is an occupancy loss only, because the header and CRC cost is the same whether the container is full or not.

19. Framing Efficiency

framing_efficiency = payload_bytes_per_unit / transported_bytes_per_unit

Verified inputs (§3): the Adapter "adds a 2-byte Flit Header and 2-byte CRC", and the Consortium refers to flits of "68B or 256B".

For the 68-byte case, those numbers determine the split arithmetically:

transported bytes per unit = 68 B                      [verified flit size]
overhead                   =  2 B header + 2 B CRC
                           =  4 B                      [verified additions]
payload bytes per unit     = 68 − 4 = 64 B
 
framing_efficiency = 64 / 68 = 0.9412                  → 94.12%
overhead fraction  =  4 / 68 = 0.0588                  →  5.88%

20. Unit Size and Overhead, Symbolically

With a fixed overhead V bytes and a unit of U bytes:

framing_efficiency(U) = (U − V) / U = 1 − V/U

The overhead fraction falls as 1/U, which is the general argument for larger transport units — and the reason 13.5 §18's batching discussion exists.

Read the shape rather than the last row. Efficiency rises steeply and then flattens: going 16 → 68 buys 19 points; going 68 → 256 buys about 4 more. Beyond a certain unit size the framing argument is nearly exhausted, and the remaining losses are occupancy, utilisation and replay — which is why those three get their own sections and framing gets one.

And the cost of a larger unit is latency, which 13.5 §18 developed and 15.2 §22 quantifies: a bigger container takes longer to fill at low load and longer to serialise at any load.

21. Payload Occupancy

Framing efficiency assumes the container is full. Occupancy measures whether it was.

payload_occupancy = payload_bytes_actually_carried / payload_bytes_available_in_units_sent

Worked, illustrative. A stream of 16-byte semantic objects placed one per 64-byte payload area:

payload available per unit = 64 B
payload carried per unit   = 16 B
payload_occupancy          = 16 / 64 = 0.25            → 25%
 
combined with framing (Section 19):
  effective payload fraction = 0.9412 × 0.25 = 0.2353  → 23.5%
 
Meaning: 68 bytes are transported to deliver 16 useful bytes.
Check:    16 / 68 = 0.2353                             ✓ the product is consistent

The check at the end matters. Multiplying the two factors gives the same answer as computing the ratio directly — which is the evidence that the definitions of §18 do not overlap. If they did, the product would disagree with the direct ratio, and that disagreement is the fastest way to find a double-counted factor.

Two design levers, and both belong to earlier chapters. Packing multiple semantic objects into one unit raises occupancy (13.5 §18's batching), at a latency cost. Choosing a unit size matched to the traffic raises it structurally. Neither is a bandwidth mechanism — they are both scheduling decisions whose effect shows up here.

22. Cycle Utilisation

cycle_utilisation = transferring_cycles / observation_cycles

This factor is where every stall in Modules 12–14 lands. A cycle in which nothing is transferred is a cycle of raw capacity permanently lost — capacity is a rate, not a store.

And the first row is the one that must be separated. 13.5 §34 made this point for stall attribution and it applies identically here: an idle cycle because nothing was offered is not a link inefficiency. A utilisation figure that counts it as one makes a lightly loaded link look broken, and hides real stalls inside a large idle count.

So the useful form separates them:

cycle_utilisation_when_offered = transferring_cycles / (observation_cycles − idle_cycles)

Report both. The first says what fraction of the link's capacity the workload used; the second says what fraction of the opportunities the link converted. They answer different questions and a design should not have to choose.

23. The Replay Tax

non_replay_fraction = unique_payload_bytes / total_payload_bytes_transported

A retransmitted object consumes raw capacity and delivers no new payload. 13.5 §29 and 13.4 §25 developed the mechanism; here it is a factor in the model.

What the verified BER figures imply, computed rather than asserted. UCIe 3.0 targets 10⁻¹² at 64 GT/s (§3):

errored bits per lane per second = 64e9 bit/s × 1e-12 = 0.064 /s
                                 → one bit error per lane every ~15.6 s
 
for a 16-lane cluster:
  errors per second = 16 × 0.064 = 1.024 /s
 
if one bit error costs one retransmitted 68B unit:
  retransmitted bytes/s ≈ 1.024 × 68 = 69.6 B/s
 
as a fraction of a 64 GB/s raw stream:
  69.6 / 64e9 ≈ 1.1e-9                              → about 0.0000001%

At the specified BER, the replay tax is numerically negligible for bandwidth. That is the entire point of specifying a BER.

Three qualifications that keep this honest.

The calculation assumes one error costs one unit. A burst can corrupt several consecutive units (14.1 §37), and a retransmission that itself fails costs more.

It says nothing about latency. A rare event that costs a full retry round trip barely moves the mean and dominates the tail — 15.2 §30 is that argument.

And it collapses if the BER is not met. A marginal channel running orders of magnitude worse enters 13.4 §26's amplification chain, and the replay fraction becomes the dominant term. So the counter exists not because the expected value is large, but because the observed value is the fastest way to prove the channel is meeting specification.

24. The Effective-Bandwidth Model

effective_payload_bw
    = raw_lane_bw                  [bit/s]   — Section 8
    × active_lane_fraction         [—]       — degradation, Section 35
    × framing_efficiency           [—]       — Section 19
    × payload_occupancy            [—]       — Section 21
    × cycle_utilisation            [—]       — Section 22
    × non_replay_fraction          [—]       — Section 23
 
Units: bit/s × (dimensionless)⁵ = bit/s                      ✓

Two properties worth stating.

Every factor is at most 1, so the result is at most the raw rate. That is a sanity check with teeth: a measured throughput above the raw rate means a counter or a unit is wrong (§27), not that the link exceeded physics.

And the factors are multiplicative, so they compound quickly. Five factors at 90% each give 59%, not 90%. A design where every factor is "pretty good" can deliver a little over half its ceiling, which is the arithmetic reason performance work feels unrewarding until the dominant factor is found — and §34 is how to find it.

25. Performance Counters

// ILLUSTRATIVE performance counters. No UCIe telemetry register is claimed
// (Section 3). Counters only — hardware counts, software divides (Section 30).
//
// Widths: 64-bit for byte counts because Section 32's rollover analysis says
// 32 bits is not enough at these rates.
logic [63:0] observation_cycles_q;     // cycles in the window
logic [63:0] transfer_cycles_q;        // cycles that actually transferred
logic [63:0] idle_cycles_q;            // nothing offered — NOT a stall (Sec 22)
logic [63:0] transmitted_bytes_q;      // ALL bytes on the lanes, incl. replay + framing
logic [63:0] unique_payload_bytes_q;   // NEW semantic payload only
logic [63:0] replay_bytes_q;           // bytes re-sent
logic [63:0] framing_bytes_q;          // header + CRC bytes
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n || counter_clear) begin
    observation_cycles_q  <= '0; transfer_cycles_q     <= '0;
    idle_cycles_q         <= '0; transmitted_bytes_q   <= '0;
    unique_payload_bytes_q<= '0; replay_bytes_q        <= '0;
    framing_bytes_q       <= '0;
  end else begin
    observation_cycles_q <= observation_cycles_q + 64'd1;
 
    // THE QUALIFICATION: an actual transfer, not merely an offer (Section 27).
    if (beat_transferred) begin
      transfer_cycles_q   <= transfer_cycles_q   + 64'd1;
      transmitted_bytes_q <= transmitted_bytes_q + 64'(beat_bytes);
      framing_bytes_q     <= framing_bytes_q     + 64'(beat_framing_bytes);
 
      if (beat_is_replay) replay_bytes_q         <= replay_bytes_q + 64'(beat_payload_bytes);
      else                unique_payload_bytes_q <= unique_payload_bytes_q + 64'(beat_payload_bytes);
    end else if (!work_offered) begin
      idle_cycles_q <= idle_cycles_q + 64'd1;
    end
  end
end

Architecture. Seven counters, each corresponding to exactly one term of §24's model. That correspondence is the design — a counter that does not map to a model term cannot be interpreted, and a model term with no counter cannot be measured.

State. All observation-window lifetime, cleared together. Clearing them individually breaks every ratio, because the ratios assume a common window.

Cycle behaviour. Everything is qualified by beat_transferred (§27). Note the else if (!work_offered) — idle is counted only when nothing was offered, so a stalled cycle with work pending is neither a transfer nor idle, and falls into the difference that §22's second form measures.

Contract, and it is checkable:

observation_cycles = transfer_cycles + idle_cycles + stalled_cycles
transmitted_bytes  = unique_payload_bytes + replay_bytes + framing_bytes

Both identities must hold at every window boundary. A violation means a counter is double-counting or missing an event, and that is an instrumentation bug rather than a performance finding — the same discipline 13.5 §34 applied to stall attribution.

Failure. §27, and separately: incrementing transmitted_bytes_q by the beat width rather than the valid bytes of a partial final beat, which overstates throughput by the padding.

DV. Assert both identities at every window boundary. Then drive a known workload and confirm each counter independently against a hand-computed expectation.

26. Counter Event Definitions

The counters are only as good as the events, and three of the seven have a subtlety.

The second row is the one that makes or breaks the model. If replayed bytes are counted as unique payload, then non_replay_fraction is 1 by construction and the performance model reports a healthy link during a retry storm — the exact condition 13.5 §30 said useful / transmitted exists to detect.

27. Wrong Counter — Counting valid Instead of valid && ready

// WRONG — counts an OFFER, not a TRANSFER.
if (tx_valid) begin
  transfer_cycles_q   <= transfer_cycles_q   + 64'd1;
  transmitted_bytes_q <= transmitted_bytes_q + 64'(beat_bytes);
end

The failure, with a three-cycle stall on one beat:

The measurement reports exactly twice the bytes that moved.

Four properties make this the best bug in the chapter.

It reports throughput above the physical maximum. With enough backpressure the figure exceeds the raw lane capacity — and that is the diagnostic signature: a measured throughput greater than the ceiling is always an instrumentation bug, never a discovery (§24).

Its magnitude depends on backpressure, so it varies with load. At full rate with no stalls, the buggy and correct counters agree exactly. The error appears only under congestion, which is when the measurement is most needed.

It makes every derived figure wrong in a correlated way. Utilisation, efficiency and the replay fraction all use these counters, so they stay mutually consistent while all being wrong — and mutual consistency is exactly what an engineer uses to gain confidence in a measurement.

And it is the same root error as 14.1 §12's CRC bug, in a different structure: a state machine advancing on valid rather than on valid && ready. Any block that consumes a stream must be qualified by the handshake, and instrumentation is not an exception.

28. SVA — Counters Increment Only on Transfer

// MANDATORY. The property that catches Section 27.
property p_transfer_cycles_only_on_transfer;
  @(posedge clk) disable iff (!rst_n || counter_clear)
    (!beat_transferred) |=> $stable(transfer_cycles_q);
endproperty
a_transfer_cycles_only_on_transfer: assert property (p_transfer_cycles_only_on_transfer);
 
property p_bytes_only_on_transfer;
  @(posedge clk) disable iff (!rst_n || counter_clear)
    (!beat_transferred) |=> $stable(transmitted_bytes_q);
endproperty
a_bytes_only_on_transfer: assert property (p_bytes_only_on_transfer);
 
// Replay bytes never counted as unique payload — Section 26's second row.
property p_replay_not_counted_as_unique;
  @(posedge clk) disable iff (!rst_n || counter_clear)
    (beat_transferred && beat_is_replay) |=> $stable(unique_payload_bytes_q);
endproperty
a_replay_not_counted_as_unique: assert property (p_replay_not_counted_as_unique);
 
// The two partition identities of Section 25, checked rather than assumed.
property p_cycle_partition;
  @(posedge clk) disable iff (!rst_n)
    window_end |-> (observation_cycles_q ==
                    transfer_cycles_q + idle_cycles_q + stalled_cycles_q);
endproperty
a_cycle_partition: assert property (p_cycle_partition);
 
property p_byte_partition;
  @(posedge clk) disable iff (!rst_n)
    window_end |-> (transmitted_bytes_q ==
                    unique_payload_bytes_q + replay_bytes_q + framing_bytes_q);
endproperty
a_byte_partition: assert property (p_byte_partition);
 
// The physical ceiling — a sanity property with real teeth (Section 24).
property p_never_exceeds_raw_capacity;
  @(posedge clk) disable iff (!rst_n)
    window_end |-> (transmitted_bytes_q <=
                    (observation_cycles_q * BYTES_PER_CYCLE_MAX));
endproperty
a_never_exceeds_raw_capacity: assert property (p_never_exceeds_raw_capacity);

Architecture. Six properties in three groups: event qualification (the first three), partition integrity (the next two), and a physical sanity bound (the last).

Why the last one earns its place. It is the cheapest possible guard against an entire class of instrumentation bug, and it needs no reference model — it simply asserts that the design did not report moving more bytes than its lanes can carry. §27's bug fails it under sustained backpressure, immediately.

Contract. All six are over the instrumentation's own signals. They should be permanently enabled, because instrumentation bugs are silent by nature and only surface as inexplicable numbers much later.

DV. The first three need backpressure in the stimulus; the last needs sustained load. A regression that runs the link at partial load with ready tied high satisfies all six vacuously.

29. The Snapshot Problem

Software reading a multi-word counter while it is incrementing can read a torn value.

Concretely, with a 64-bit counter read as two 32-bit halves:

The read is off by nearly 4 billion, and nothing anywhere reports an error. The two halves were individually correct and belonged to different instants.

Three remedies, and the first is the standard one:

The third is worth naming as a trap. Freezing the counters to read them means the observation window has a hole in it, and the hole is exactly when software was looking — so a system that polls frequently measures a link it has been repeatedly blinding.

30. Snapshot RTL

// ILLUSTRATIVE atomic snapshot. One request freezes a coherent set; software
// reads the shadow copy while the live counters keep counting.
logic [63:0] snap_cycles_q, snap_transfer_q, snap_tx_bytes_q;
logic [63:0] snap_unique_q, snap_replay_q,  snap_framing_q;
logic        snap_valid_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    snap_valid_q <= 1'b0;
  end else if (snapshot_req) begin
    // ALL captured on the SAME edge — that is the atomicity.
    snap_cycles_q   <= observation_cycles_q;
    snap_transfer_q <= transfer_cycles_q;
    snap_tx_bytes_q <= transmitted_bytes_q;
    snap_unique_q   <= unique_payload_bytes_q;
    snap_replay_q   <= replay_bytes_q;
    snap_framing_q  <= framing_bytes_q;
    snap_valid_q    <= 1'b1;
  end
end

Architecture. A shadow copy of the whole counter set, captured on one clock edge. The atomicity is structural — one edge, one always_ff, all six — and not procedural.

State. Per snapshot. The live counters are untouched and keep counting, which is what distinguishes this from the freeze approach (§29).

Cycle behaviour. One edge. The moment one counter is captured in a different cycle from another, the set is no longer coherent and cross-counter ratios — which is what every figure in this chapter is — become meaningless.

Contract, and it is the section's point. Software must read only the shadow. Reading a mix of live and snapshot values reintroduces the tearing the snapshot exists to prevent, in a subtler form: the ratios would be computed from two different instants.

And the ratios are why atomicity matters more here than for a single counter. unique_payload_bytes / observation_cycles taken from two different instants is not merely imprecise — under a changing load it can be outside the physically possible range, which then looks like §27's bug and sends the investigation to the wrong place.

Failure. Capturing counters in a for loop across several cycles, or snapshotting some counters and letting software read others live.

DV. Snapshot under a rapidly changing load and check the shadow set satisfies §28's partition identities. A torn snapshot fails them, which is a neat reuse of a property written for a different purpose.

31. Rollover, and Why Deltas Are the Answer

How long until a counter wraps, at the raw capacity of §12?

raw byte rate = 64e9 byte/s
 
32-bit byte counter:  2**32  = 4.295e9 bytes
  → 4.295e9 / 64e9   = 0.067 s          ← wraps in 67 MILLISECONDS
 
64-bit byte counter:  2**64  = 1.845e19 bytes
  → 1.845e19 / 64e9  = 2.88e8 s
                     ≈ 9.1 years

A 32-bit byte counter is useless at these rates — it wraps faster than most software polls. A 64-bit counter is comfortable for the life of the part. That is the arithmetic behind §25's widths, and it is worth doing rather than picking 64 by habit.

Three practices follow.

Prefer deltas between snapshots to absolute values. (snap_now − snap_prev) is correct across at most one wrap under modular arithmetic, and it is what a monitoring agent actually wants. It also makes the measurement window explicit rather than "since some reset nobody remembers".

Do not saturate these. They are accounting, not diagnostics — 14.5 §11's rule cuts the other way here. A saturated byte counter silently under-reports every subsequent delta, whereas a wrapped one is exactly recoverable by modular subtraction. Saturate diagnostics; let accounting wrap and subtract modularly.

And ensure the poll interval is shorter than the wrap interval, or a delta is ambiguous by an unknown number of wraps. With 64-bit counters this is trivially satisfied; with 32-bit ones it is a real constraint that must be documented.

32. Worked End-to-End Example

One fully specified case. Verified inputs are marked; every other input is illustrative and marked.

INPUTS
  transfer rate            = 32 GT/s per lane        [VERIFIED rate]
  data lanes (cluster)     = 16                      [VERIFIED, standard package]
  clusters                 = 1                       [stated]
  bits per transfer        = 1                       [VERIFIED for UCIe-S @ 32 GT/s]
  flit size                = 68 B                    [VERIFIED size]
  framing overhead         = 4 B (2 hdr + 2 CRC)     [VERIFIED additions]
  payload occupancy        = 0.85                    [ILLUSTRATIVE]
  cycle utilisation        = 0.90                    [ILLUSTRATIVE]
  replay fraction          = 0.02                    [ILLUSTRATIVE]
  active lane fraction     = 1.00                    [no degradation]
 
STEP 1 — raw, one direction
  raw_bit_rate = 32e9 × 16 × 1 = 512e9 bit/s = 512 Gb/s
  raw_byte_rate = 512e9 / 8    = 64e9 byte/s = 64 GB/s
 
STEP 2 — framing efficiency
  framing_efficiency = (68 − 4) / 68 = 64/68 = 0.9412
 
STEP 3 — apply the chain (Section 24)
  effective = 64 GB/s
            × 1.00      (active lane fraction)
            × 0.9412    (framing)
            × 0.85      (occupancy)
            × 0.90      (utilisation)
            × 0.98      (non-replay = 1 − 0.02)
 
  step by step:
    64      × 1.00   = 64.00 GB/s
    64.00   × 0.9412 = 60.24 GB/s
    60.24   × 0.85   = 51.20 GB/s
    51.20   × 0.90   = 46.08 GB/s
    46.08   × 0.98   = 45.16 GB/s
 
RESULT
  effective payload bandwidth ≈ 45.2 GB/s per direction
  overall efficiency          = 45.16 / 64 = 0.7056   → 70.6%
 
INTERPRETATION
  About 29% of the raw capacity is consumed by framing, partial containers,
  idle cycles and retransmission. None of those is a defect; all four are
  design and workload properties, and three of the four are measurable with
  the counters of Section 25.

Note what the arithmetic shows about where the loss is. Framing costs 3.76 GB/s, occupancy costs 9.04, utilisation costs 5.12, replay costs 0.92. Occupancy dominates, and it is a scheduling decision rather than a link property — which §33 develops.

33. Sensitivity — Which Factor Dominates

Change one factor at a time from §32's baseline of 45.16 GB/s:

Four readings.

Width and rate are the only factors that can double anything. Everything else is a percentage improvement on what remains. That is the structural argument for the width-and-rate trade (§36) and the reason UCIe's own evolution is expressed in those terms.

Occupancy is the largest soft factor here, and it is free in silicon — it is a packing decision. A performance effort that starts with framing rather than occupancy is optimising the smaller term, and §32's arithmetic says so explicitly.

Framing is nearly exhausted at 68 B. Moving to a much larger unit buys about 4.5% and costs latency (§20). Worth knowing before proposing it as a throughput fix.

And eliminating replay entirely buys 2%, which at the specified BER is already the achieved value — §23 showed the real figure is nine orders of magnitude smaller. The 0.02 in the baseline is an illustrative stand-in for a channel not meeting specification, and if replay is genuinely 2% the correct action is to fix the channel, not to optimise around it.

34. Width Degradation, Quantified

Verified: "the standard package supports width degradation to handle failures" (§3). 14.4 built the mechanism; here is its bandwidth consequence.

x16 → x8 at the same rate:
  raw:  512 Gb/s → 256 Gb/s        exactly half — lanes are independent

But delivered throughput does not necessarily halve, and both directions of error are common:

The third row is the counter-intuitive one and it is real. A workload with one outstanding transaction and a 50-cycle response latency uses 2% of the link (13.5 §13). Halving the width changes almost nothing, because the link was idle 98% of the time either way.

"Width halved, so throughput halved" is a hypothesis, not a conclusion. The measurement that settles it is cycle utilisation before and after — if it was well below 1 before, the link was not the constraint.

35. Rate Scaling

Verified rates (§3): UCIe 3.0 raises the maximum "from 32 GT/s to 64 GT/s (as well as adding support for 48GT/s)", and "mandates the existing quarter-rate clocking for 48/64 GT/s."

raw, one direction, x16, 1 bit/transfer:
  32 GT/s → 32e9 × 16 = 512 Gb/s = 64 GB/s
  48 GT/s → 48e9 × 16 = 768 Gb/s = 96 GB/s
  64 GT/s → 64e9 × 16 = 1024 Gb/s = 128 GB/s

And the rate has a reliability cost that belongs in the model. The verified BER targets are 10⁻¹⁵ at 48 GT/s and 10⁻¹² at 64 GT/s — three orders of magnitude worse for a 33% rate gain. §23 showed the bandwidth impact is still negligible at specification; the impact that matters is on tail latency (15.2 §30) and on the sizing of the retry path (14.3 §27).

36. Width Against Rate

The same raw bandwidth can be reached by different combinations, and they are not equivalent:

raw = rate × lanes × bits_per_transfer
 
  16 lanes @ 64 GT/s = 1024 Gb/s
  32 lanes @ 32 GT/s = 1024 Gb/s      same raw figure, different everything else

UCIe's own history demonstrates both directions, and 13.5 §27 develops it: UCIe-3D goes wide and slow — "even at 4 GT/s frequency, there are orders of magnitude improvement in bandwidth density", with "the need for (de)serialization, CRC, replay, etc." eliminated by choosing an appropriate BER — while UCIe 3.0 goes faster and pays for it with a BER three orders of magnitude worse and the retry path that absorbs it.

Bandwidth alone cannot choose between them. The choice is made on package escape, power, timing and whether the error rate lets you delete the reliability machinery.

37. Bandwidth Density

Official material reports bandwidth density, and it is a different metric from bandwidth.

What density is for, and what it is not.

It answers a floorplan question: given this much die edge, how much bandwidth can I get? That is the question a chiplet architect actually has, because shoreline is the scarce resource.

It is not a throughput figure and must never be quoted as one. "1300+" is not GB/s — it is a density with its own units, at a stated bump pitch, for a stated packaging option. Quoting it without the pitch, the units and the revision is exactly the marketing usage §2 of this chapter's brief rules out.

And the ratios are comparisons against a stated baseline. "20X the most efficient PCIe SERDES" is a comparison whose value depends entirely on the baseline. It is a real engineering argument for on-package interconnect over long-reach SERDES, and it says nothing about what any particular link delivers.

38. The Roofline — Is UCIe Even the Bottleneck?

The link's bandwidth is a ceiling. The workload's throughput is bounded by the minimum of several ceilings, and the link is frequently not the lowest.

achieved_throughput ≤ min(
    link_effective_bandwidth,     // this chapter
    producer_rate,                // can the source generate this fast?
    consumer_rate,                // can the sink absorb it?
    memory_or_media_bandwidth,    // the far side's real limit
    concurrency_limit             // 13.5 Section 17's conjunction
)

Two symptoms that look like a link problem and are not.

High raw capacity, low application throughput, low cycle utilisation. The link is idle most cycles. The constraint is upstream or downstream, and adding lanes or raising the rate changes nothing — 13.5 §15's single-outstanding-transaction case is the extreme version, using 2% of the link with no defect anywhere.

High cycle utilisation, low useful throughput. The link is busy and delivering little, which points at occupancy or replay — and unique_payload_bytes / transmitted_bytes distinguishes them immediately.

Before optimising a link, measure its cycle utilisation. If it is well below 1, the link is not the bottleneck and no amount of bandwidth will help.

39. The Measurement Scoreboard

// Verification-only reference model. Not synthesisable.
//
// Its job is NOT to predict throughput — it is to prove the INSTRUMENTATION
// is trustworthy, so the numbers can be believed.
class bandwidth_scoreboard;
 
  longint exp_transfer_cycles, exp_transmitted_bytes;
  longint exp_unique_bytes,    exp_replay_bytes, exp_framing_bytes;
  longint exp_idle_cycles,     exp_observation_cycles;
 
  // ---- Independently counted from the monitor, not from the DUT counters.
  function void on_beat(bit transferred, bit is_replay,
                        int payload_bytes, int framing_bytes, bit work_offered);
    exp_observation_cycles++;
    if (transferred) begin
      exp_transfer_cycles++;
      exp_transmitted_bytes += payload_bytes + framing_bytes;
      exp_framing_bytes     += framing_bytes;
      if (is_replay) exp_replay_bytes += payload_bytes;
      else           exp_unique_bytes  += payload_bytes;
    end else if (!work_offered) begin
      exp_idle_cycles++;
    end
  endfunction
 
  // ---- Check 1: every DUT counter matches the independent count.
  function void check_counters(longint dut_transfer, longint dut_tx_bytes,
                               longint dut_unique,   longint dut_replay);
    if (dut_transfer  != exp_transfer_cycles)
      $error("transfer cycles: dut %0d, expected %0d", dut_transfer, exp_transfer_cycles);
    if (dut_tx_bytes  != exp_transmitted_bytes)
      $error("transmitted bytes: dut %0d, expected %0d", dut_tx_bytes, exp_transmitted_bytes);
    if (dut_unique    != exp_unique_bytes)
      $error("unique payload bytes: dut %0d, expected %0d — replay counted as unique?",
             dut_unique, exp_unique_bytes);
    if (dut_replay    != exp_replay_bytes)
      $error("replay bytes: dut %0d, expected %0d", dut_replay, exp_replay_bytes);
  endfunction
 
  // ---- Check 2: the physical ceiling was never exceeded (Section 24).
  function void check_ceiling(longint bytes, longint cycles, int bytes_per_cycle_max);
    if (bytes > cycles * bytes_per_cycle_max)
      $error("IMPOSSIBLE: %0d bytes in %0d cycles exceeds %0d B/cycle — counter or unit bug",
             bytes, cycles, bytes_per_cycle_max);
  endfunction
 
  // ---- Check 3: the decomposed model reproduces the measured throughput.
  //      This is what validates the MODEL rather than the counters.
  function void check_model(real measured_gbps, real raw_gbps, real framing_eff,
                            real occupancy, real utilisation, real non_replay);
    real modelled = raw_gbps * framing_eff * occupancy * utilisation * non_replay;
    if ($abs(modelled - measured_gbps) > 0.01 * measured_gbps)
      $error("MODEL MISMATCH: modelled %0.3f, measured %0.3f — a factor is missing or double-counted",
             modelled, measured_gbps);
  endfunction
 
endclass

Architecture. Three checks with genuinely different purposes: counter correctness, a physical sanity bound, and model validity.

Check 1 is why the scoreboard exists. The counters are the basis of every performance conclusion, so an independently derived count is the only thing that establishes they can be trusted. Its most valuable single line is the unique comparison, whose error message names the likely cause — §26's replay-counted-as-unique.

Check 2 restates §28's ceiling property at the scoreboard level, and catches unit errors that a per-cycle assertion cannot see because they only manifest in the accumulated total.

Check 3 validates the model, not the hardware, and it is the check most performance environments lack. If the five factors multiplied do not reproduce the measured throughput within tolerance, either a factor is missing from the decomposition or two of them overlap (§18) — and that is a modelling bug that would otherwise be discovered by an architect months later trying to reconcile the numbers.

40. Coverage

covergroup cg_bandwidth @(posedge clk);
  option.per_instance = 1;
 
  // --- Configuration space. Width and rate must both vary.
  cp_active_lanes : coverpoint active_data_lanes {
    bins x4 = {4}; bins x8 = {8}; bins x16 = {16}; bins x32 = {32};
    bins degraded = {[1:3],[5:7],[9:15]};        // post-degradation odd widths
  }
  cp_rate : coverpoint rate_sel { bins r32 = {R_32}; bins r48 = {R_48}; bins r64 = {R_64}; }
 
  // --- Utilisation. Full rate and idle both matter.
  cp_utilisation : coverpoint utilisation_bucket {
    bins idle    = {0};
    bins low     = {[1:3]};
    bins high    = {[4:8]};
    bins full    = {9};                          // proves the ceiling is reachable
  }
 
  // --- Payload occupancy, including the pathological case.
  cp_occupancy : coverpoint occupancy_bucket {
    bins tiny   = {0};                           // near-empty containers
    bins low    = {[1:4]};
    bins high   = {[5:8]};
    bins full   = {9};                           // fully packed units
  }
 
  // --- Partial final beats — Section 26's byte-counting subtlety.
  cp_partial_beat : coverpoint final_beat_valid_bytes {
    bins each[] = {[1:BEAT_BYTES]};
  }
 
  // --- Replay present, so the non-replay factor is exercised.
  cp_replay : coverpoint replay_active;
 
  // --- Counter mechanics.
  cp_snapshot_under_load : coverpoint snapshot_taken_while_counting;
  cp_counter_high        : coverpoint counter_near_rollover;   // Section 31
 
  // --- Crosses that carry the information.
  x_lanes_rate     : cross cp_active_lanes, cp_rate;
  x_util_occupancy : cross cp_utilisation, cp_occupancy;   // both low = worst case
  x_replay_util    : cross cp_replay, cp_utilisation;      // busy but not useful
  x_partial_replay : cross cp_partial_beat, cp_replay;
endcovergroup

Five bins whose value is being non-zero:

cp_utilisation.full. Without it the ceiling has never been approached, so §28's p_never_exceeds_raw_capacity has never been near its bound and the raw-capacity model is unvalidated against reality.

cp_occupancy.tiny crossed with high utilisation. The busy-but-unproductive case — a link at full cycle utilisation delivering very little useful payload, which is the condition §38's useful / transmitted ratio exists to expose.

cp_partial_beat — every bin. §26's partial-beat byte counting fails on specific valid-byte counts, and the sweep is cheap.

cp_snapshot_under_load. §30's atomicity is only tested when a snapshot is taken while the counters are moving quickly.

cp_active_lanes.degraded. Post-degradation widths are not powers of two — a masked lane gives 15, not 8 — and a model that assumes a power-of-two width computes the wrong capacity after every repair.

41. Debug Taxonomy

42. Debug Checklist

1. What exact transfer rate, and which revision? 48 and 64 GT/s are UCIe 3.0; mixing revisions produces a part nobody built (§35).
2. How many data lanes are active right now? Not configured, not total signals (§7, §8).
3. How many clusters? "Per link" needs a cluster count (§14).
4. How many bits per transfer? Verified as 1 for UCIe-S at 32 GT/s; do not assume elsewhere (§6).
5. Per direction or aggregate? A factor of two hides here (§10).
6. Decimal GB/s or binary GiB/s? A factor of 1.074 hides here (§9).
7. What framing format, and what is the exact overhead? 2 B header + 2 B CRC for the stated format; do not extrapolate (§19).
8. What is the payload occupancy? Usually the largest soft factor (§33).
9. What is the cycle utilisation — and is idle separated from stalled? (§22).
10. What is the replay fraction? At specification it should be negligible; if it is not, the channel is the problem (§23).
11. Are the counters event-qualified? valid && ready, always (§27).
12. Do the two partition identities hold? If not, stop — the numbers cannot be trusted (§25).
13. Did width or rate degrade? And was utilisation already at 1 before it did (§34)?
14. Is the snapshot atomic, and is the poll interval shorter than the wrap? (§30, §31).
15. Is the link actually the bottleneck? Measure utilisation before optimising anything (§38).
16. Does the decomposed model reproduce the measurement? If not, a factor is missing or double-counted (§39).

43. Common Misconceptions

"GT/s equals Gb/s." GT/s counts transfers; Gb/s counts bits. They coincide only when one transfer carries one bit per lane, which is a property of the signalling scheme. Official material verifies it for UCIe-S at 32 GT/s — 8 lanes × 32 GT/s = 256 Gb/s — and does not license generalising it (§6).

"Bidirectional bandwidth is available to one direction." TX and RX are separate physical resources. A one-directional workload gets the per-direction figure, and quoting the aggregate overstates it by two — while still needing the reverse direction for credits and acknowledgements (§10).

"Lane count alone determines useful throughput." It determines the raw ceiling exactly. Useful throughput is that ceiling multiplied by framing efficiency, payload occupancy, cycle utilisation and the non-replay fraction — five factors at 90% each give 59% (§24).

"Flit overhead is the only efficiency loss." At 68 B it is 5.9%, and in the worked example it is the smallest of the three soft losses. Occupancy cost more than twice as much (§32).

"A busy link is a productive link." A link retransmitting, or shipping quarter-full containers, is fully utilised and delivering little. unique_payload_bytes / transmitted_bytes is the ratio that separates them (§38).

"Replay traffic counts as useful bandwidth." It consumes capacity and delivers no new payload. Counting it as unique payload makes the non-replay factor 1 by construction and hides a retry storm completely (§26).

"Counting valid cycles measures transfers." It measures offers. Under backpressure it reports more bytes than physically moved — sometimes more than the link can carry, which is always an instrumentation bug and never a discovery (§27).

"Counters cannot overflow." A 32-bit byte counter wraps in 67 milliseconds at 64 GB/s. Use 64 bits, take deltas between snapshots, and let accounting counters wrap and subtract modularly rather than saturating (§31).

"A x16-to-x8 degradation always halves application performance." It exactly halves the raw ceiling. What happens to delivered throughput depends on whether the link was the binding constraint — and if utilisation was well below 1, the answer can be almost nothing (§34).

"Theoretical bandwidth equals workload throughput." Throughput is bounded by the minimum of the link, the producer, the consumer, the far-side media and the concurrency pools. Measure cycle utilisation before optimising the link (§38).

"Bandwidth density is a throughput number." It is bandwidth per unit of shoreline or area, at a stated bump pitch and packaging option. It answers a floorplan question and must never be quoted as GB/s (§37).

44. Understanding Check

45. Summary and What Comes Next

Raw bandwidth counts transferred bits; useful bandwidth counts meaningful payload bits. The gap is a decomposition of measurable factors, not a constant.

Units are a correctness concern. GT/s against Gb/s (a factor of bits_per_transfer), bits against bytes (8), GB/s against GiB/s (1.074), per-direction against aggregate (2), per-lane against per-link (lanes), per-cluster against per-link (clusters). Six factors, all invisible without discipline — so put the unit in the identifier and check that dimensions cancel.

GT/s equals Gb/s per lane only when one transfer carries one bit. Official material verifies that for UCIe-S at 32 GT/s — 8 lanes at 32 GT/s giving 256 Gb/s per direction — and does not license generalising it.

Lane, cluster and link are three different things. A cluster is "the unit of construction" with 16 or 64 data lanes plus Valid, tracking, a forwarded clock and sideband; only the data lanes carry payload, and multiple clusters aggregate per link.

Raw capacity is linear in active data lanes and needs 64-bit arithmetic. A 32-bit int cannot hold 32 × 10⁹, and the overflow compiles cleanly, produces a plausible number, and silently mis-sizes whatever was computed from it.

Four disjoint factors separate raw from useful — framing, occupancy, utilisation, non-replay — and their product must reproduce the direct ratio, which is how you prove they do not overlap. In the worked case, occupancy dominated and framing was the smallest of the three soft terms.

Counters must be event-qualified. valid && ready, never valid — the latter reports throughput above the physical ceiling under backpressure, and every derived figure stays mutually consistent while all of them are wrong. Both partition identities must be checked at every window boundary, snapshots must be atomic, and byte counters need 64 bits because 32 bits wrap in 67 milliseconds at these rates.

And measure cycle utilisation before optimising anything. Width degradation halves the ceiling and may barely change delivered throughput; a busy link may be delivering almost nothing; and the producer, the consumer, the far-side media or a concurrency pool is frequently the real limit. The link's bandwidth is a ceiling, and the workload meets whichever ceiling is lowest.

Bandwidth answers how much work can pass once the pipeline is full. Latency asks a different question: how long does one transaction spend waiting, transforming, crossing, executing and returning?

• 15.2 — Latency Anatomy — hop-by-hop decomposition, timestamping across clock domains, queue against service time, and why the tail matters more than the mean.

Browse the full path on the UCIe tutorials index.