Ethernet · Module 24
Ethernet against InfiniBand
Credit-based flow control removes the drop and reserves a round trip of buffer for it — 2.32% of a merchant switch at 3 m and 91.47% at 2 km, per link, per lane, forever.
Chapter 21.6 is an entire chapter about a frame that is gone with no reporter. InfiniBand does not have that chapter, because its links do not drop — and this chapter derives what that costs and what it puts back.
The mechanism is one sentence. A sender transmits only into buffer space the receiver has explicitly guaranteed, counted in credits, and the receiver returns credits as it drains. A frame is never discarded for want of space, because a frame is never sent into space that does not exist.
And the first-order consequence is that a whole class of Ethernet engineering disappears.
| Ethernet | InfiniBand | |
|---|---|---|
| congestion produces | a drop, with no reporter | backpressure, immediately |
| the transport must | detect loss and retransmit | not |
| Chapter 21.6's apparatus | necessary | has nothing to diagnose |
| the receiver's buffer | sized for the traffic you expect | sized for a round trip, mandatory |
Row four is where the bill arrives and Sections 2 to 5 are about it.
1. Scope — A Drop Removed and a Reservation Created
This chapter owns four derivations, and every one of them is an arithmetic consequence of the credit loop.
| What is derived | |
|---|---|
| Sections 2 to 3 | the reserved buffer per link per hop at 400 Gb/s — 25.5 kB at 3 m, 980.5 KiB at 2 km |
| Sections 4 to 5 | the same, priced in BCE against Chapter 23.3's 5.62 × 10⁸ switch, at one and at eight virtual lanes |
| Sections 6 to 9 | what the credits put back: deterministic head-of-line blocking and a congestion tree that grows at one hop per credit loop |
| Sections 10 to 13 | where RoCE sits, what it inherits from each parent, and the go-back-N amplification it pays for a drop |
What this chapter does not own. It is not an InfiniBand tutorial: the verbs API, the subnet manager's routing algorithm, the queue-pair state machine and the management datagram format are all outside it. It uses exactly as much InfiniBand as the comparison needs, and derives every quantity rather than quoting it.
It also does not claim that lossless is wrong. It claims that lossless is paid for in a currency that does not appear on the datasheet, and Section 19 totals the bill.
2. The Credit Loop, and the Buffer It Reserves
A lossless link's reserved buffer is the bandwidth-delay product of its credit return loop. Derive the loop first.
Three things happen between a receiver freeing space and the sender being able to use it.
| Stage | What it is | Controlled by |
|---|---|---|
| the credit travels to the sender | one-way propagation | physics |
| the sender's next packet travels back | one-way propagation again | physics |
| credit generation and processing | detect, build the message, transmit, consume | both implementations |
| one maximum frame already in flight | the sender committed before the credit arrived | the sender |
credit loop = 2 x propagation + credit turnaround + one maximum frameAt 400 Gb/s, a 300 ns turnaround and a 9 000-octet maximum frame — using Chapter 8.2 §3's velocity figures of 5.05 ns/m for copper and 4.90 ns/m for fibre.
| Link | 2 × propagation | One frame | Loop | Reserved buffer |
|---|---|---|---|---|
| 3 m DAC copper | 30.3 ns | 180 ns | 510.3 ns | 25 515 B = 24.9 KiB |
| 30 m AOC | 294 ns | 180 ns | 774 ns | 38 700 B = 37.8 KiB |
| 100 m fibre | 980 ns | 180 ns | 1 460 ns | 73 000 B = 71.3 KiB |
| 2 km fibre | 19 600 ns | 180 ns | 20 080 ns | 1 004 000 B = 980.5 KiB |
A lossless 400 Gb/s link over two kilometres must reserve just under a megabyte of receive buffer, per direction, before it carries a single useful bit.
Three properties of that table decide everything downstream.
One — the reservation is not a peak requirement, it is a floor. A buffer smaller than the loop does not merely risk overflow; it guarantees the sender stalls, because credits cannot return fast enough to keep it fed. Chapter 21.8 §5's ceiling applies exactly: the achievable rate is the credit count divided by the loop time, and the loop time is mostly propagation.
Two — it scales with the rate and the rate has moved by 400×. Propagation has not moved at all since Chapter 1.6's 10 Mb/s. So the reservation at 400 Gb/s is forty thousand times what it was, for exactly the same cable.
Three — it is per direction, per link, per virtual lane. Section 4 multiplies by all three.
3. RTL 1 — The Lossless Package and the Credit-Loop Model
// ---------------------------------------------------------------------
// ibcmp_pkg -- the constants for a lossless-link comparison, with every
// derived figure computed rather than written as a literal.
//
// Unit: Chapter 23.3 Section 2's bitcell equivalent.
// 1 BCE = one bit of usable on-die SRAM = 0.35 GE
// 1 flip-flop = 20 BCE
// Chapter 19.7 Section 19's MAC receive datapath = 283 320 BCE
// Chapter 23.3's 64-port 6.4 Tb/s switch = 5.62e8 BCE
// ---------------------------------------------------------------------
package ibcmp_pkg;
localparam int unsigned DATAPATH_BCE = 283_320;
localparam int unsigned BCE_PER_FLOP = 20;
localparam int unsigned SWITCH_BCE_E5 = 5_620; // 5.62e8 / 1e5
localparam int unsigned SWITCH_PORTS = 64;
localparam int unsigned SWITCH_BUF_MB = 64;
// Chapter 8.2 Section 3's velocity figures, in hundredths of a
// nanosecond per metre, so integer arithmetic keeps the precision.
localparam int unsigned NS_PER_M_CU_X100 = 505; // 5.05 ns/m
localparam int unsigned NS_PER_M_FI_X100 = 490; // 4.90 ns/m
// The credit turnaround: detect, build, transmit, consume. Both
// implementations contribute and neither publishes it.
localparam int unsigned CREDIT_TURNAROUND_NS = 300;
// The frame already committed when the credit arrived.
localparam int unsigned MAX_FRAME_OCTETS = 9_000;
typedef enum logic [1:0] {
MEDIA_COPPER = 2'd0,
MEDIA_FIBRE = 2'd1
} media_e;
// ---- the credit loop -------------------------------------------------
function automatic int unsigned prop_ns_x100(int unsigned metres,
media_e m);
return metres * ((m == MEDIA_COPPER) ? NS_PER_M_CU_X100
: NS_PER_M_FI_X100);
endfunction
function automatic int unsigned frame_ns(int unsigned gbps);
// octets x 8 bits / Gb/s gives nanoseconds directly.
return (MAX_FRAME_OCTETS * 8) / gbps;
endfunction
function automatic int unsigned credit_loop_ns(int unsigned metres,
media_e m,
int unsigned gbps);
return ((2 * prop_ns_x100(metres, m)) / 100)
+ CREDIT_TURNAROUND_NS
+ frame_ns(gbps);
endfunction
// ---- the reservation -------------------------------------------------
// rate x time: Gb/s x ns / 8 gives bytes.
function automatic int unsigned reserved_bytes(int unsigned metres,
media_e m,
int unsigned gbps);
return (gbps * credit_loop_ns(metres, m, gbps)) / 8;
endfunction
function automatic int unsigned reserved_bce(int unsigned metres,
media_e m,
int unsigned gbps,
int unsigned lanes);
return reserved_bytes(metres, m, gbps) * 8 * lanes;
endfunction
// ---- reporting -------------------------------------------------------
function automatic int unsigned datapaths_milli(int unsigned bce);
return (bce / DATAPATH_BCE) * 1000
+ (((bce % DATAPATH_BCE) * 1000) / DATAPATH_BCE);
endfunction
// Share of Chapter 23.3's switch, in hundredths of a per cent, so a
// fraction below one per cent is still a nonzero integer.
function automatic int unsigned switch_share_x100(int unsigned bce_e2);
// bce_e2 is the figure divided by 100, to keep the product in range.
return (bce_e2 * 10_000) / (SWITCH_BCE_E5 * 1_000);
endfunction
endpackage// ---------------------------------------------------------------------
// credit_loop_model -- what a lossless link must reserve, derived from
// the medium rather than configured.
//
// The output that matters is reservation_is_a_floor, which is always
// high: a buffer below the loop does not risk overflow, it guarantees
// a stall, and the two failures look nothing alike.
// ---------------------------------------------------------------------
module credit_loop_model
import ibcmp_pkg::*;
(
input logic clk,
input logic rst_n,
input logic [15:0] metres,
input logic [1:0] media, // media_e
input logic [15:0] rate_gbps,
input logic [31:0] buffer_bytes, // what the design implements
output logic [31:0] loop_ns,
output logic [31:0] prop_share_pct,
output logic [31:0] reserved_bytes_o,
output logic [31:0] reserved_bce_o,
output logic reservation_is_a_floor,
output logic buffer_insufficient,
output logic [15:0] achievable_gbps,
output logic [31:0] ethernet_reservation_bce,
output logic [31:0] c_stall_events
);
always_comb begin
loop_ns = 32'(credit_loop_ns(int'(metres), media_e'(media),
int'(rate_gbps)));
reserved_bytes_o = 32'(reserved_bytes(int'(metres), media_e'(media),
int'(rate_gbps)));
reserved_bce_o = reserved_bytes_o * 32'd8;
// How much of the loop is physics. Above 2 km it is essentially all
// of it, which is why no implementation choice helps.
prop_share_pct = (loop_ns == 0) ? 32'd0
: ((2 * 32'(prop_ns_x100(int'(metres), media_e'(media))))
/ 100) * 32'd100 / loop_ns;
// THE structural point of the chapter.
reservation_is_a_floor = 1'b1;
buffer_insufficient = (buffer_bytes < reserved_bytes_o);
// Chapter 21.8 Section 5's ceiling: if the buffer is short, the
// rate is the buffer divided by the loop.
achievable_gbps = (loop_ns == 0) ? rate_gbps
: (buffer_insufficient
? 16'((buffer_bytes * 32'd8) / loop_ns)
: rate_gbps);
// An Ethernet link reserves nothing. It drops instead.
ethernet_reservation_bce = 32'd0;
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) c_stall_events <= '0;
else if (buffer_insufficient) c_stall_events <= c_stall_events + 32'd1;
end
endmoduleClassification: a bandwidth-delay product with a medium's velocity factor in it, producing a floor rather than a target.
What it teaches: that reservation_is_a_floor is hard-wired high, and that a buffer below the floor fails differently from one that is merely small. A short Ethernet buffer drops under burst, which is intermittent and load-dependent. A short credit buffer caps the link's rate permanently at buffer ÷ loop, which is Chapter 21.8 §5's ceiling with a new denominator. At 100 m and 400 Gb/s a 32 kB buffer delivers 32 768 × 8 ÷ 1 460 = 179 Gb/s — 44.9% of the link, steadily, on an idle fabric.
And it teaches that prop_share_pct is the number that decides whether engineering can help. At 3 m it is 5.9% — the turnaround and the frame dominate, and a faster credit path is worth building. At 2 km it is 97.6%, and nothing an implementation does moves the reservation at all.
Deliberately simplified: CREDIT_TURNAROUND_NS is a single constant where the real figure is the sum of two implementations' internal delays, is not published by either, and varies with how busy the return direction is — a credit message queued behind data is Chapter 14.2 §11's problem in a new protocol. The model assumes one maximum frame in flight where a sender with several small frames committed has less. And achievable_gbps assumes the buffer is the only limit, ignoring that a short buffer also increases the credit message rate, which consumes return bandwidth.
Production implication: the reservation is set at design time from an assumed cable length, and the cable is chosen by whoever installs the rack. A switch designed for 3 m direct-attach copper and deployed with 100 m optics needs 2.9 times the buffer — 73 000 bytes against 25 515 — and the symptom is a link that negotiates, passes every test, and runs at 34.9% of its rate with no errors anywhere. buffer_insufficient is one comparison evaluated at configuration time, and it converts that into a message at bring-up. A design that does not compute it has made the cable length an undocumented requirement.
4. The Reservation at Fabric Scale, in BCE
One link's reservation is a number on a datasheet. Sixty-four of them is a die area, and this section computes it against a switch that exists.
Chapter 23.3 §2's 64-port 100 Gb/s merchant ASIC is 5.62 × 10⁸ BCE, of which 91.0% is a 64 MB packet buffer. Take the same port count at 400 Gb/s and ask what a lossless version would have to reserve.
| Link, 64 ports at 400 Gb/s | Per link | × 64, BCE | Datapaths | Share of the switch |
|---|---|---|---|---|
| 3 m DAC copper | 204 120 BCE | 1.306 × 10⁷ | 46.1 | 2.32% |
| 30 m AOC | 309 600 BCE | 1.981 × 10⁷ | 69.9 | 3.53% |
| 100 m fibre | 584 000 BCE | 3.738 × 10⁷ | 131.9 | 6.65% |
| 2 km fibre | 8 032 000 BCE | 5.140 × 10⁸ | 1 814.4 | 91.47% |
At two kilometres, the reservation alone is 91.47% of an entire merchant switch ASIC — and the switch still needs a packet buffer, a forwarding table and a pipeline.
Row four is the chapter's headline and it deserves to be stated as an impossibility rather than as a cost. A 64-port 400 Gb/s lossless switch over 2 km links cannot be built as a single die at Chapter 23.3's area, because the reservation alone consumes what that die spends on everything.
Rows one to three are the ones that get built, and the reason lossless fabrics look the way they do is visible in them.
| Why the deployment looks like this | |
|---|---|
| short links | row one is 2.32% and row four is 91.47% — a factor of 39.4 |
| direct-attach copper inside a rack | 3 m is the cheapest row and it is not close |
| a separate long-reach tier that is not lossless | because row four is unaffordable |
And the same table read the other way gives the capacity question.
Chapter 23.3's 64 MB buffer, spent entirely on reservations.
| Link | Reservation per port | Ports a 64 MB buffer can serve |
|---|---|---|
| 3 m | 25 515 B | 2 630 |
| 100 m | 73 000 B | 919 |
| 2 km | 1 004 000 B | 66 |
Sixty-six ports at 2 km, from a buffer sized for a 64-port switch — so the buffer is fully consumed by reservation and there is nothing left to buffer congestion with, which is the point of having a buffer.
5. RTL 2 — The Reservation Model
// ---------------------------------------------------------------------
// reservation_model -- the fabric-scale figure, and the comparison
// against a switch that exists.
//
// The output that matters is over_commitment_allowed, which is 0.
// Every other buffer in this track is affordable because it is shared
// against a statistical argument; this one is not.
// ---------------------------------------------------------------------
module reservation_model
import ibcmp_pkg::*;
(
input logic clk,
input logic rst_n,
input logic [15:0] ports,
input logic [31:0] per_link_bce,
input logic [31:0] switch_buffer_mb,
output logic [31:0] fabric_reservation_e2, // BCE / 100
output logic [31:0] reservation_dp_milli,
output logic [15:0] share_of_switch_x100,
output logic exceeds_switch,
output logic [31:0] ports_a_buffer_serves,
output logic over_commitment_allowed,
output logic [31:0] lossy_equivalent_bce,
output logic [31:0] c_infeasible_configs
);
always_comb begin
fabric_reservation_e2 = (per_link_bce / 32'd100) * 32'(ports);
reservation_dp_milli = 32'(datapaths_milli(
(fabric_reservation_e2 * 32'd100)));
share_of_switch_x100 = 16'(switch_share_x100(fabric_reservation_e2));
exceeds_switch = (share_of_switch_x100 >= 16'd10_000);
// How many ports the switch's existing buffer could reserve for.
ports_a_buffer_serves = (per_link_bce == 0) ? 32'd0
: ((switch_buffer_mb * 1024 * 1024 * 8)
/ per_link_bce);
// THE structural difference from every other buffer in this track.
over_commitment_allowed = 1'b0;
// A lossy fabric reserves nothing and drops instead.
lossy_equivalent_bce = 32'd0;
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) c_infeasible_configs <= '0;
else if (exceeds_switch) c_infeasible_configs <= c_infeasible_configs + 32'd1;
end
endmoduleClassification: a multiplication by the port count, and a constant zero that is the reason the multiplication cannot be undone.
What it teaches: that over_commitment_allowed is zero and that this is where the arithmetic becomes unaffordable rather than merely large. Chapter 23.3 §9's shared buffer works because 63 of 64 ports are usually not congested, and the dynamic threshold hands the idle capacity to whoever needs it. A reservation must hold at the instant every port is busy, so the statistical argument is unavailable and the figure is the sum rather than the maximum.
And it teaches that ports_a_buffer_serves answers the design question directly. Chapter 23.3's 64 MB buffer reserves for 2 630 ports at 3 m, 919 at 100 m, and 66 at 2 km — so the same silicon is comfortable, adequate and exactly at the edge, and the parameter that moved is the cable.
Deliberately simplified: per_link_bce is supplied rather than computed, so the module trusts credit_loop_model's output — a real deployment binds them. The port count is uniform, where a switch with a mixture of 3 m and 2 km links has a reservation that is the sum of the mixture rather than a multiple of anything. And share_of_switch_x100 compares against a 100 Gb/s switch's total while the reservations are computed at 400, which is deliberate: the comparison asks what the reservation would consume of a die that exists, not of a hypothetical one.
Production implication: the mixture case is the one that produces a surprising bill. A leaf switch with 48 server-facing 3 m copper ports and 16 spine-facing 2 km optical ports reserves 48 × 204 120 + 16 × 8 032 000 = 1.383 × 10⁸ BCE — 24.6% of Chapter 23.3's entire switch, of which the 16 long links are 92.9%. A budget computed from the average link length would report a quarter of that, and the design would come up unable to reach line rate on precisely the uplinks Chapter 23.1 depends on. The reservation is a sum over links and never a product with a mean.
6. Virtual Lanes, and Why the Reservation Multiplies
Section 4's figures are for one credit pool. A fabric that needs to avoid head-of-line blocking between traffic classes needs one pool per class, and the reservation multiplies by the class count.
Why a second pool is necessary at all is Chapter 14.3's argument, reached from the other direction. A single credit pool means one blocked destination stops every packet on the link, because the packet at the head cannot move and nothing behind it can pass. That is head-of-line blocking made mandatory rather than probable.
| Chapter 14.3 §4's input-queued switch | A single-pool lossless link | |
|---|---|---|
| the cause | a FIFO can offer only its head | the same |
| under uniform traffic | 58.6% throughput — 2 − √2 | the same bound applies |
| the remedy | virtual output queues | virtual lanes |
| what the remedy costs | N queues and a matching scheduler | N reservations |
Row four is where the two diverge. Chapter 14.3 §11's virtual output queues cost queue state and a scheduler, which that chapter §14 priced. Virtual lanes cost a full reservation each, because each lane's credits are independent and a lane may not borrow another's promised space.
So multiply Section 4's table by the lane count.
| Link, 64 ports at 400 Gb/s | 1 lane | 8 lanes, BCE | Datapaths | Share of the switch |
|---|---|---|---|---|
| 3 m DAC copper | 2.32% | 1.045 × 10⁸ | 368.9 | 18.60% |
| 30 m AOC | 3.53% | 1.585 × 10⁸ | 559.5 | 28.21% |
| 100 m fibre | 6.65% | 2.990 × 10⁸ | 1 055.4 | 53.20% |
| 2 km fibre | 91.47% | 4.112 × 10⁹ | 14 515.0 | 731.74% |
Eight virtual lanes over 100 m consume 53.2% of a merchant switch in reservation alone, and over 2 km they consume seven of them.
Three consequences follow and each is visible in a real fabric.
One — lane counts are small in practice. InfiniBand's specification allows fifteen data lanes plus a management lane; deployments use two to four, and Section 4's table says why: the reservation is linear in the lane count and the die is not.
Two — the lanes are not equal. A design that gives one lane the full reservation and the others a fraction has a lossless lane and several nearly-lossless ones, which is a legitimate engineering answer and is exactly what Chapter 14.4's per-priority headroom does on Ethernet.
Three — the share is computed against a lossy switch's die and the lossless switch still needs everything else. The reservation is in addition to the packet buffer, the forwarding tables and the pipeline, not instead of them.
7. RTL 3 — The Virtual-Lane Partition
// ---------------------------------------------------------------------
// vl_partition -- independent credit pools, and the reservation each
// one obliges.
//
// The output that matters is borrow_permitted, which is 0. A lane with
// spare promised space may not lend it to a starving neighbour, and
// that single fact is why the reservation is a sum rather than a max.
// ---------------------------------------------------------------------
module vl_partition
import ibcmp_pkg::*;
#(
parameter int unsigned LANES = 8
)(
input logic clk,
input logic rst_n,
input logic [31:0] per_lane_reserved_bytes,
input logic [31:0] total_buffer_bytes,
input logic [3:0] lane_sel,
input logic packet_arrives,
input logic packet_drains,
input logic [15:0] packet_bytes,
output logic [31:0] reservation_total_bytes,
output logic [31:0] reservation_total_bce,
output logic fits_in_buffer,
output logic [15:0] buffer_used_pct,
output logic borrow_permitted,
output logic lane_blocked,
output logic head_of_line_across_lanes,
output logic [31:0] occupancy [LANES],
output logic [31:0] c_lane_blocks,
output logic [31:0] c_wasted_reserved_bytes
);
always_comb begin
reservation_total_bytes = per_lane_reserved_bytes * 32'(LANES);
reservation_total_bce = reservation_total_bytes * 32'd8;
fits_in_buffer = (reservation_total_bytes <= total_buffer_bytes);
buffer_used_pct = (total_buffer_bytes == 0) ? 16'd0
: 16'((reservation_total_bytes * 32'd100)
/ total_buffer_bytes);
// THE reason the reservation is a sum. Chapter 23.3 Section 9's
// dynamic threshold lends idle space between ports; a credit
// promise cannot be lent, because it was already promised.
borrow_permitted = 1'b0;
lane_blocked = (occupancy[lane_sel] + 32'(packet_bytes))
> per_lane_reserved_bytes;
// And the reason lanes exist: a block on one lane does not stop
// the others, which a single pool could not offer.
head_of_line_across_lanes = 1'b0;
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int i = 0; i < LANES; i++) occupancy[i] <= '0;
c_lane_blocks <= '0; c_wasted_reserved_bytes <= '0;
end else begin
if (packet_arrives && !lane_blocked)
occupancy[lane_sel] <= occupancy[lane_sel] + 32'(packet_bytes);
if (packet_drains && (occupancy[lane_sel] >= 32'(packet_bytes)))
occupancy[lane_sel] <= occupancy[lane_sel] - 32'(packet_bytes);
if (packet_arrives && lane_blocked)
c_lane_blocks <= c_lane_blocks + 32'd1;
// The quantity nobody reports: space that was promised, is empty,
// and cannot be used by the lane that needs it.
c_wasted_reserved_bytes <= '0;
for (int i = 0; i < LANES; i++)
c_wasted_reserved_bytes <= c_wasted_reserved_bytes
+ (per_lane_reserved_bytes - occupancy[i]);
end
end
endmoduleClassification: N independent pools with an explicit prohibition on sharing, and a counter for the space the prohibition wastes.
What it teaches: that borrow_permitted is zero and that this single bit separates a reservation from every other buffer in this track. Chapter 23.3 §9's dynamic threshold is exactly borrowing — one port holds 27.8 MB because 63 others are idle — and it is what makes a 64 MB buffer serve a 6.4 Tb/s switch. Remove borrowing and the same guarantee costs the sum instead of the maximum, which at eight lanes is a factor of eight.
And it teaches that c_wasted_reserved_bytes is a quantity no fabric reports. On a link where one lane carries everything and seven are idle, seven eighths of the reservation is empty, promised, and unusable — and there is no counter in any deployed lossless fabric that says so, because from each lane's own point of view nothing is wrong.
Deliberately simplified: the occupancy accumulator is written as a loop for clarity and a synthesisable form maintains a running total incrementally. head_of_line_across_lanes is hard-wired zero, which is the guarantee lanes are supposed to provide; a real design must prove it, because a shared arbiter, a shared output queue or a shared scheduler can reintroduce the coupling the lanes were bought to remove. And the per-lane reservation is uniform, where a real design gives different lanes different guarantees.
Production implication: the coupling the simplification hides is the bug this structure actually has. Eight virtual lanes that share one egress scheduler are eight queues and one head, and Chapter 14.3 §16's warning applies exactly: virtual output queues do not fix blocking that has moved into the scheduler. The symptom is a fabric with lanes configured, reservations paid for, and head-of-line blocking that behaves as though there were one lane — and the only way to see it is to measure per-lane throughput under a load that congests exactly one of them. A design that reports c_lane_blocks per lane makes that a five-minute test; one that reports an aggregate makes it invisible.
8. What the Credits Put Back — Blocking That Is Required Rather Than Likely
A lossless fabric does not remove congestion. It converts loss into stalling, and the stalling propagates upstream at a rate this section derives.
Start with what happens at the congested point. An egress port is oversubscribed. On Ethernet, Chapter 14.1 §4: the buffer fills and frames are discarded, and the congestion is confined to that port. On a credit link, the buffer fills and credits stop returning, so the upstream sender stops — and the upstream sender's own buffer now fills.
The propagation rate is one hop per credit loop.
| Link | One credit loop | A 5-stage fabric stalls end to end in |
|---|---|---|
| 3 m DAC copper | 0.510 µs | 2.55 µs |
| 30 m AOC | 0.774 µs | 3.87 µs |
| 100 m fibre | 1.460 µs | 7.30 µs |
| 2 km fibre | 20.080 µs | 100.40 µs |
On a short-reach lossless fabric, a single congested egress port stalls the entire five-stage topology in under three microseconds, deterministically.
And the reach of the damage is Chapter 14.3 §8's table, with the probabilities removed.
| Frames dropped | Pairs affected | On a credit fabric | |
|---|---|---|---|
| congested egress, no backpressure | many | 23 | does not occur |
| one hop of backpressure | few | up to 529 | certain rather than possible |
| two hops | very few | up to 12 167 | certain rather than possible |
The word that changed in column four is the whole difference. Chapter 14.3's "up to" is a bound on a probabilistic spread — PAUSE is asserted when a watermark is crossed, and a watermark is crossed sometimes. A credit sender stops the instant its credits run out, which is not a threshold and not a policy. So the congestion tree on a lossless fabric is not a risk; it is the mechanism working.
9. RTL 4 — The Congestion Tree
// ---------------------------------------------------------------------
// congestion_tree -- how far a stall reaches and how long it takes to
// get there and back.
//
// Chapter 14.3 Section 9 measured a blast radius as a bitmap of
// affected pairs. This does the same for a fabric where the spread is
// certain rather than probable, and adds the two things that changes:
// a growth RATE, and a drain time.
// ---------------------------------------------------------------------
module congestion_tree
import ibcmp_pkg::*;
(
input logic clk,
input logic rst_n,
input logic tick_100ns,
input logic [15:0] radix, // ports per switch
input logic [15:0] stages,
input logic [31:0] credit_loop_ns_i,
input logic congestion_start,
input logic congestion_clear,
output logic [15:0] hops_reached,
output logic [31:0] pairs_affected,
output logic [31:0] full_stall_ns,
output logic [31:0] drain_ns,
output logic tree_is_deterministic,
output logic fully_stalled,
output logic [31:0] c_tree_events,
output logic [31:0] c_stall_ns_total
);
logic [31:0] elapsed_ns;
logic active;
always_comb begin
// Chapter 14.3 Section 8's counts: one switch's ingresses, then
// their upstreams, then theirs.
pairs_affected = (hops_reached == 16'd0) ? 32'(radix) - 32'd1
: ((hops_reached == 16'd1)
? (32'(radix) - 32'd1) * (32'(radix) - 32'd1)
: (32'(radix) - 32'd1) * (32'(radix) - 32'd1)
* (32'(radix) - 32'd1));
full_stall_ns = credit_loop_ns_i * 32'(stages);
// The tree unwinds the same way it grew.
drain_ns = full_stall_ns;
// THE difference from Chapter 14.3. There, backpressure MIGHT
// spread. Here the upstream is REQUIRED to stop.
tree_is_deterministic = 1'b1;
fully_stalled = active && (elapsed_ns >= full_stall_ns);
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
elapsed_ns <= '0; hops_reached <= '0; active <= 1'b0;
c_tree_events <= '0; c_stall_ns_total <= '0;
end else begin
if (congestion_start && !active) begin
active <= 1'b1; elapsed_ns <= '0; hops_reached <= '0;
c_tree_events <= c_tree_events + 32'd1;
end
if (congestion_clear) begin
active <= 1'b0;
end
if (active && tick_100ns) begin
elapsed_ns <= elapsed_ns + 32'd100;
c_stall_ns_total <= c_stall_ns_total + 32'd100;
if ((elapsed_ns + 32'd100) >= (credit_loop_ns_i
* (32'(hops_reached) + 32'd1))
&& (hops_reached < stages))
hops_reached <= hops_reached + 16'd1;
end
end
end
endmoduleClassification: a spreading model whose distinguishing output is a hard-wired 1 where Chapter 14.3's equivalent would carry a probability.
What it teaches: that full_stall_ns is a small number on short links and that this is the uncomfortable result. At 3 m and five stages it is 2.55 µs — so a lossless fabric reaches total stall faster than a lossy one reaches its first drop, because a drop requires a buffer to fill and a stall requires only a credit to be withheld.
And it teaches that drain_ns equals full_stall_ns, so the event's total duration is twice the fabric's depth in credit loops. At 2 km that is 200.8 µs of a fabric doing nothing, triggered by one oversubscribed egress port, with no frame lost and no counter anywhere reporting an error.
Deliberately simplified: pairs_affected uses Chapter 14.3 §8's uniform-radix expansion, which assumes every upstream port feeds the congested path and no port appears twice — a real topology's tree is smaller and its shape depends on the routing. The growth is modelled as one hop per full credit loop, where the true rate is one hop per credit exhaustion, which happens sooner on a busy link and later on an idle one. And congestion_clear stops the model immediately, where a real tree keeps growing for a loop after the cause is gone, because the stalls in flight have not yet been told.
Production implication: the counter worth having is c_stall_ns_total rather than c_tree_events, and the reason is that the two answer different questions. An event count says how often congestion occurred; a stalled-nanosecond total says what fraction of the fabric's time was spent not moving — and on a lossless fabric that is the only loss-equivalent quantity there is. A fabric reporting zero drops and 8% stalled time has lost 8% of its capacity, and an operator comparing its drop counter against an Ethernet fabric's will conclude it is performing better. The comparison is only meaningful if the stalled time is put beside the drop count, which requires the lossless fabric to keep a counter its whole design philosophy says it does not need.
10. Where RoCE Sits, and What It Inherits from Each Parent
RDMA over Converged Ethernet is not a compromise between the two fabrics. It is InfiniBand's transport placed on Ethernet's link layer, and it inherits a specific list from each.
State the layering precisely, because the loose version causes the misconfiguration this section ends with.
| Layer | InfiniBand | RoCEv1 | RoCEv2 |
|---|---|---|---|
| transport | IB — queue pairs, verbs | IB, unchanged | IB, unchanged |
| network | IB — LIDs, subnet manager | none — EtherType 0x8915 | IP |
| link | IB — credits | Ethernet | Ethernet |
| routable | within a subnet | no — one L2 domain | yes |
| lossless from | credits, by construction | PFC, if configured | PFC, if configured |
Row five is the one that matters and it is the source of every RoCE deployment problem.
What RoCE inherits from InfiniBand: a transport that was designed on the assumption that the link does not drop. The queue-pair state machine, the reliable-connected service and the completion semantics all come from a world where a packet in flight arrives.
What RoCE inherits from Ethernet: a link that does drop, unless somebody configured priority flow control on every switch on the path, on the right priority, with the right headroom — which is Chapter 14.4's mechanism and Chapter 14.2 §9's arithmetic, applied by hand across a fabric.
RoCE is a lossless transport on a link that is lossless only if somebody configured it to be, and the transport has no way to find out whether they did.
Price the mismatch. The InfiniBand transport's reliable-connected service recovers a loss by go-back-N: on detecting a gap, the receiver discards everything after it and the sender retransmits from the lost packet onward.
| At 400 Gb/s, 1 µs round trip | MTU 1 500 | MTU 9 000 |
|---|---|---|
| window in flight | 50 000 B | 50 000 B |
| packets in the window | 33.3 | 5.6 |
| retransmitted for one loss | 50 000 B | 50 000 B |
| amplification | 33.3× | 5.6× |
And the throughput consequence.
| Drop rate | Retransmission overhead, MTU 1 500 | MTU 9 000 |
|---|---|---|
| 10⁻⁴ | 0.33% | 0.06% |
| 10⁻³ | 3.33% | 0.56% |
| 10⁻² | 33.3% | 5.56% |
A drop rate of one in a thousand — which is unremarkable on a congested Ethernet fabric and which Chapter 21.6 would call mild — costs 3.33% of a 400 Gb/s link under go-back-N. At one in a hundred it costs a third of the link, and the fabric reports a 1% drop rate, which sounds fine.
11. RTL 5 — The RoCE Window Model
// ---------------------------------------------------------------------
// roce_window -- what a drop costs a transport that assumed it would
// not happen, and what the repair costs.
//
// The output that matters is link_losslessness_is_configured, which is
// an INPUT masquerading as a property: the transport cannot determine
// it and behaves as though it were true.
// ---------------------------------------------------------------------
module roce_window
import ibcmp_pkg::*;
(
input logic clk,
input logic rst_n,
input logic [15:0] rate_gbps,
input logic [15:0] rtt_ns,
input logic [15:0] mtu_octets,
input logic selective_repeat,
input logic [15:0] queue_pairs,
input logic pfc_configured_everywhere, // nobody can verify this
input logic packet_lost,
output logic [31:0] window_bytes,
output logic [15:0] window_packets,
output logic [31:0] retransmit_bytes,
output logic [15:0] amplification_x10,
output logic [31:0] bitmap_bits,
output logic [31:0] bitmap_bce,
output logic link_losslessness_is_configured,
output logic transport_assumes_lossless,
output logic [31:0] c_retransmit_bytes,
output logic [31:0] c_losses
);
always_comb begin
window_bytes = (32'(rate_gbps) * 32'(rtt_ns)) / 32'd8;
window_packets = (mtu_octets == 0) ? 16'd0
: 16'(window_bytes / 32'(mtu_octets));
// Go-back-N discards the whole window; selective repeat does not.
retransmit_bytes = selective_repeat ? 32'(mtu_octets) : window_bytes;
amplification_x10 = (mtu_octets == 0) ? 16'd0
: 16'((retransmit_bytes * 32'd10) / 32'(mtu_octets));
// One bit per packet in the window, per queue pair.
bitmap_bits = selective_repeat
? ((32'(window_packets) + 32'd1) * 32'(queue_pairs))
: 32'd0;
bitmap_bce = bitmap_bits; // an array: 1 BCE per bit
// THE honest signal. It is wired from configuration because no
// measurement in the transport can establish it.
link_losslessness_is_configured = pfc_configured_everywhere;
// And this is always true, whatever the link actually does.
transport_assumes_lossless = 1'b1;
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
c_retransmit_bytes <= '0; c_losses <= '0;
end else if (packet_lost) begin
c_losses <= c_losses + 32'd1;
c_retransmit_bytes <= c_retransmit_bytes + retransmit_bytes;
end
end
endmoduleClassification: a window model whose most important signal is an input the design cannot verify.
What it teaches: that transport_assumes_lossless is hard-wired high while link_losslessness_is_configured comes from a register somebody wrote, and that the gap between those two is where every RoCE deployment problem lives. The transport behaves identically whether PFC is configured or not; the only difference is the drop rate, and the only symptom is throughput.
And it teaches that amplification_x10 reports 333 at a 1 500-octet MTU and 56 at 9 000 — a six-fold change from one configuration field, which is a far larger lever than the framing efficiency that MTU discussions usually turn on.
Deliberately simplified: the window is the full bandwidth-delay product, where a real queue pair's window is also bounded by the receiver's advertised credit and by the number of outstanding work requests. retransmit_bytes charges a full window for every loss, where a burst of consecutive losses inside one window costs one retransmission rather than several. And bitmap_bce prices the bitmap alone, ignoring the out-of-order reassembly state that selective repeat also requires — which is the larger cost and is the same reorder buffer Chapter 24.1 §4 priced at 400 000 BCE.
Production implication: pfc_configured_everywhere is a lie in almost every deployment and it is a lie in a specific way. PFC is configured on the switches somebody remembered, and one switch on one path without it converts that path from lossless to lossy while every other path stays lossless — so the fabric's drop rate is zero on most flows and non-trivial on a few. The symptom is a small number of queue pairs running at a fraction of the rest, with no errors, no drops visible at the endpoints, and a fabric-wide drop rate that rounds to zero. The diagnostic is per-queue-pair retransmission bytes, which c_retransmit_bytes provides and which no fabric-level counter can substitute for, because the fabric's average is exactly the statistic that hides it.
12. Where a Lossless Fabric Actually Fails
Sections 2 to 11 priced the mechanism. This section names the failure it has that a lossy fabric structurally cannot, and the answer is deadlock.
The mechanism in four sentences. A packet holds buffer space at switch A and needs space at switch B. B's space is held by a packet that needs space at C. C's is held by a packet that needs space at A. Nothing can move, no credit can be returned, and no timeout in the protocol resolves it — because every participant is behaving exactly as specified.
| A lossy fabric | A credit fabric | |
|---|---|---|
| a full buffer | drops, and the cycle breaks | withholds credit, and the cycle holds |
| who resolves it | the buffer, automatically | nobody |
| the fabric's state | recovers | permanent |
Row one is the whole reason a drop-based fabric cannot deadlock. A drop is an escape: the moment any participant discards a packet, the dependency it represented is gone. Chapter 21.6's silent, unreported, undiagnosable drop is also, structurally, the thing that keeps an Ethernet fabric live.
The drop that a whole chapter of this track was written to diagnose is the same drop that makes deadlock impossible. Removing it removes both.
The standard defence is a routing restriction rather than a hardware mechanism, and it is worth naming because it constrains the topology rather than the silicon.
| Defence | What it does | What it costs |
|---|---|---|
| up-down routing | forbids the turns that close a cycle | path diversity — some shortest paths become illegal |
| dimension-ordered routing | orders the dimensions a packet may turn in | the same, on a mesh or torus |
| escape virtual lanes | one lane is deadlock-free by construction | a full reservation for a lane carrying almost nothing |
| a timeout and discard | breaks the cycle by dropping | the fabric is no longer lossless |
Row four is the honest one and it is what real fabrics do. A credit-based fabric that has been stalled for longer than any legitimate loop discards, and every deployed lossless fabric has such a mechanism — so "lossless" means "does not drop for congestion", not "does not drop."
The timeout must exceed the credit loop by a comfortable factor, and the loop depends on the cable.
| Link | Credit loop | 10× the loop | 100× |
|---|---|---|---|
| 3 m copper | 0.510 µs | 5.10 µs | 51.0 µs |
| 100 m fibre | 1.460 µs | 14.60 µs | 146.0 µs |
| 2 km fibre | 20.080 µs | 200.8 µs | 2 008 µs |
A single timeout value across a fabric with mixed link lengths is either too short for the long links or 39× too long for the short ones, and the second is the worse error: a 2 ms stall on a 3 m link is four thousand credit loops of a fabric doing nothing before anything is discarded.
13. RTL 6 — The Deadlock Detector
// ---------------------------------------------------------------------
// deadlock_detector -- the mechanism that makes a lossless fabric
// survivable, and the admission that it is not lossless.
//
// The output that matters is is_still_lossless, which goes LOW the
// first time the detector fires. A fabric that has never fired it is
// lossless; one that has is a fabric with a long timeout.
// ---------------------------------------------------------------------
module deadlock_detector
import ibcmp_pkg::*;
#(
parameter int unsigned LANES = 8
)(
input logic clk,
input logic rst_n,
input logic tick_us,
input logic [31:0] credit_loop_ns_i,
input logic [15:0] timeout_multiplier, // x the credit loop
input logic [7:0] lane_has_work, // one bit per lane
input logic [7:0] lane_has_credit,
output logic [31:0] timeout_us,
output logic [15:0] loops_before_discard,
output logic [7:0] lane_starved,
output logic discard_required,
output logic [2:0] discard_lane,
output logic is_still_lossless,
output logic [31:0] c_discards,
output logic [31:0] c_starved_us
);
logic [31:0] starved_us [LANES];
logic fired;
always_comb begin
timeout_us = ((credit_loop_ns_i * 32'(timeout_multiplier)) + 32'd999)
/ 32'd1000;
loops_before_discard = timeout_multiplier;
// A lane with work and no credit is starved. It may be a normal
// congestion stall or the head of a cycle, and nothing here can
// tell the difference -- which is why the remedy is a timeout.
lane_starved = lane_has_work & ~lane_has_credit;
// THE admission.
is_still_lossless = !fired;
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int i = 0; i < LANES; i++) starved_us[i] <= '0;
discard_required <= 1'b0; discard_lane <= '0;
fired <= 1'b0; c_discards <= '0; c_starved_us <= '0;
end else begin
discard_required <= 1'b0;
if (tick_us) begin
for (int i = 0; i < LANES; i++) begin
if (lane_starved[i]) begin
starved_us[i] <= starved_us[i] + 32'd1;
c_starved_us <= c_starved_us + 32'd1;
if ((starved_us[i] + 32'd1) >= timeout_us) begin
discard_required <= 1'b1;
discard_lane <= 3'(i);
starved_us[i] <= '0;
fired <= 1'b1;
c_discards <= c_discards + 32'd1;
end
end else begin
starved_us[i] <= '0;
end
end
end
end
end
endmoduleClassification: a starvation timer per lane, whose firing is the protocol admitting it drops.
What it teaches: that is_still_lossless is a latched bit and that every deployed lossless fabric has one. The guarantee is not no packet is ever discarded; it is no packet is discarded for want of space, which is a narrower claim and a useful one. The fabric that has fired this detector has dropped a packet, and the transport above it must handle that — which is why RoCE's go-back-N exists and why Section 10's amplification arithmetic matters even on a fabric that claims not to drop.
And it teaches that lane_starved cannot distinguish a deadlock from ordinary congestion. A lane with work and no credit is exactly what Section 8's congestion tree produces, and exactly what a routing cycle produces, and the detector has no way to tell them apart. So the timeout must be long enough that no legitimate congestion stall reaches it — which is the 39.4× problem from the previous section, arriving as a parameter that cannot be right for every link at once.
Deliberately simplified: the detector is per lane at one port where a real deadlock is a property of a cycle spanning several switches, and no single port can see it. A fabric-wide detector needs either a central collector or a distributed probe, and both are Chapter 22.2 §20's class 104 territory: the guarantee is a relation between nodes and every local observation is consistent with both hypotheses. timeout_multiplier is a single value where a design with mixed link lengths needs a per-port one. And the loop over LANES is written for clarity rather than for synthesis.
Production implication: c_starved_us is the counter to keep and c_discards is the one that gets kept. A fabric reporting zero discards has told you nothing — it may be healthy, or it may be stalling for 90% of every second and never quite reaching the timeout. Starved microseconds is the lossless fabric's equivalent of a drop counter, and it is the only quantity that can be compared against an Ethernet fabric's loss rate on equal terms. A comparison of drop counters between a lossy and a lossless fabric is not a comparison at all, because one of them converted its losses into a quantity the other does not measure, and Section 20's rejected property is precisely that.
14. What a Lossless Fabric Must Never Do
Five prohibitions, each of which this chapter's structures make possible.
| # | Never | Because |
|---|---|---|
| 1 | advertise more credits than the buffer behind them | the promise is the mechanism; a broken promise is a lost packet on a fabric whose transport does not expect one |
| 2 | let one virtual lane draw on another's reservation | Section 7: the space was already promised, and lending it converts a guarantee into a probability |
| 3 | report zero drops as evidence of health | Section 13: the loss became stalled time, and the drop counter cannot see it |
| 4 | use one deadlock timeout across links of different lengths | Section 12: the credit loop varies 39.4× and the timeout must exceed it on every link |
| 5 | assume RoCE's link is lossless because the transport requires it | Section 11: pfc_configured_everywhere is a register somebody wrote, not a measurement |
Row three is the one that produces a wrong decision rather than a bug, and it is worth stating at length.
An operator comparing a lossy and a lossless fabric reads both drop counters. The Ethernet fabric reports a drop rate; the lossless fabric reports zero. The comparison concludes the second is better and it has not measured the quantity the second one spends.
| Lossy fabric | Lossless fabric | |
|---|---|---|
| frames lost | 0.1% | 0 |
| time stalled | not measured, and small | not measured, and possibly 8% |
| capacity delivered | 99.9% minus the retransmissions | 92% |
| what the counters say | worse | better |
The correct comparison is delivered capacity, and it requires the lossless fabric to keep c_starved_us — a counter its design philosophy says it does not need, because nothing is being lost.
15. RTL 7 — Lossless Telemetry
// ---------------------------------------------------------------------
// ib_cmp_telemetry -- the counters a lossless fabric needs and does
// not usually have.
//
// The design rule: on a fabric that does not drop, the loss-equivalent
// quantity is TIME, so every counter here is a duration or a rate
// rather than an event count.
// ---------------------------------------------------------------------
module ib_cmp_telemetry
import ibcmp_pkg::*;
#(
parameter int unsigned LANES = 8
)(
input logic clk,
input logic rst_n,
input logic clear,
input logic tick_us,
input logic [7:0] lane_starved_i,
input logic credit_returned,
input logic packet_sent,
input logic discard_fired,
input logic tree_active,
input logic [31:0] reserved_bytes_i,
input logic [31:0] occupied_bytes_i,
output logic [47:0] c_starved_us_total,
output logic [47:0] c_lane_starved_us [LANES],
output logic [47:0] c_credits_returned,
output logic [47:0] c_packets_sent,
output logic [47:0] c_tree_us,
output logic [47:0] c_discards_o,
output logic [31:0] elapsed_us,
output logic [15:0] stalled_pct,
output logic [15:0] reservation_idle_pct,
output logic drop_counter_is_meaningful
);
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n || clear) begin
c_starved_us_total <= '0; c_credits_returned <= '0;
c_packets_sent <= '0; c_tree_us <= '0; c_discards_o <= '0;
elapsed_us <= '0;
for (int i = 0; i < LANES; i++) c_lane_starved_us[i] <= '0;
end else begin
if (credit_returned) c_credits_returned <= c_credits_returned + 48'd1;
if (packet_sent) c_packets_sent <= c_packets_sent + 48'd1;
if (discard_fired) c_discards_o <= c_discards_o + 48'd1;
if (tick_us) begin
elapsed_us <= elapsed_us + 32'd1;
if (tree_active) c_tree_us <= c_tree_us + 48'd1;
for (int i = 0; i < LANES; i++) begin
if (lane_starved_i[i]) begin
c_lane_starved_us[i] <= c_lane_starved_us[i] + 48'd1;
c_starved_us_total <= c_starved_us_total + 48'd1;
end
end
end
end
end
always_comb begin
// THE number. On a lossless fabric this is the loss rate.
stalled_pct = (elapsed_us == 0) ? 16'd0
: 16'((c_tree_us * 48'd100) / 48'(elapsed_us));
// Section 7's wasted promise, as a percentage.
reservation_idle_pct = (reserved_bytes_i == 0) ? 16'd0
: 16'(((reserved_bytes_i - occupied_bytes_i)
* 32'd100) / reserved_bytes_i);
// And the honest label on the counter everybody reads.
drop_counter_is_meaningful = 1'b0;
end
endmoduleClassification: a counter bank in which the primary quantity is a duration, because the fabric has converted its losses into one.
What it teaches: that drop_counter_is_meaningful is hard-wired zero and that this is the module's whole argument. A lossless fabric's drop counter reports zero by construction; it is a statement about the design rather than about the deployment, and reading it as health is Chapter 20.4 §20's class 91 in a new place — a metric whose value the mechanism was built to determine.
And it teaches that stalled_pct is the comparable quantity. A fabric at 8% stalled has delivered 92% of its capacity, which is the same outcome as a lossy fabric with an 8% effective loss rate — and only one of the two reports a number an operator would react to.
Deliberately simplified: reservation_idle_pct uses instantaneous occupancy where the useful statistic is a time average, and a high-percentile figure would be better still, because the reservation exists for the worst instant rather than the mean. The per-lane starvation counters do not distinguish congestion from a routing cycle, which Section 13 established no local observation can. And all counters are in one clock domain, where a real fabric's per-port counters are in the port's recovered clock — Chapter 18.1 §19's class 74 again.
Production implication: the counter that will be missing is c_lane_starved_us per lane, and its absence hides the failure Section 7's production note described. A fabric reporting an aggregate stalled time of 3% may have one lane stalled 24% of the time and seven idle, which is what a shared egress scheduler behind independent lanes produces — and it is indistinguishable from mild uniform congestion in the aggregate. Per-lane durations cost eight 48-bit counters, 384 flops, 7 680 BCE, 0.027 datapaths, and they are the difference between knowing that the lanes were bought for nothing and not.
16. RTL 8 — The Lossless Conformance Monitor
// ---------------------------------------------------------------------
// ib_cmp_conformance -- the checks that hold a lossless claim to what
// it actually promises.
//
// Every check here exists because "lossless" is a narrower claim than
// the word suggests, and the narrowing is what the monitor enforces.
// ---------------------------------------------------------------------
module ib_cmp_conformance
import ibcmp_pkg::*;
(
input logic clk,
input logic rst_n,
input logic [31:0] advertised_credit_bytes,
input logic [31:0] backing_buffer_bytes,
input logic [31:0] reserved_bytes_i,
input logic [31:0] required_bytes_i,
input logic lane_borrowed,
input logic discard_fired,
input logic claims_lossless,
input logic [31:0] timeout_us_i,
input logic [31:0] credit_loop_ns_i,
input logic packet_sent,
input logic credits_available_i,
output logic v_overadvertised,
output logic v_under_reserved,
output logic v_lane_borrow,
output logic v_lossless_claim_stale,
output logic v_timeout_too_short,
output logic v_sent_without_credit,
output logic [5:0] violations,
output logic conformant
);
always_comb begin
// 1. Prohibition 1 -- the promise must be backed.
v_overadvertised = (advertised_credit_bytes > backing_buffer_bytes);
// 2. Section 2 -- the reservation is a floor, not a target.
v_under_reserved = (reserved_bytes_i < required_bytes_i);
// 3. Prohibition 2 -- a lane may not draw on a neighbour's promise.
v_lane_borrow = lane_borrowed;
// 4. Section 13 -- a fabric that has discarded is not lossless,
// and must stop saying it is.
v_lossless_claim_stale = claims_lossless && discard_fired;
// 5. Prohibition 4 -- the timeout must exceed the loop it is
// protecting, with margin.
v_timeout_too_short = ((timeout_us_i * 32'd1000)
< (credit_loop_ns_i * 32'd10));
// 6. The gate itself.
v_sent_without_credit = packet_sent && !credits_available_i;
violations = { v_sent_without_credit, v_timeout_too_short,
v_lossless_claim_stale, v_lane_borrow,
v_under_reserved, v_overadvertised };
conformant = (violations == 6'b000000);
end
endmoduleClassification: six checks, of which one audits a claim rather than a behaviour.
What it teaches: that v_lossless_claim_stale is a check on a marketing statement and belongs in silicon anyway. A fabric that has fired its deadlock discard has dropped a packet; if its status register still reports lossless operation, every consumer of that register is being misled, including the transport above it, which is making retransmission decisions on the assumption. One sticky bit fixes it.
And it teaches that v_under_reserved is the check that would have caught Section 12's cabling failure. A fabric provisioned for 30 m and cabled at 2 km has reserved_bytes_i at 38 700 and required_bytes_i at 1 004 000, a shortfall of 96.1% — and the resulting link runs at 3.9% of its rate with no other symptom. The check is one comparison at configuration time.
Deliberately simplified: required_bytes_i must be supplied, which means somebody must know the cable length — and on an optical link the design genuinely cannot measure it without a round-trip ranging mechanism, which Chapter 16.2 provides on Ethernet and which a fabric could borrow. v_lane_borrow is an input, so the monitor trusts the partition to report its own violation. And there is no check that the fabric's routing is deadlock-free, because that is a property of a topology and a routing function, not of a port — no monitor at a port can write it.
Production implication: the missing topology check is the important one and it is worth saying where it does belong. Deadlock freedom is established by the routing algorithm, at the subnet manager, before any traffic flows — up-down or dimension-ordered routing is a proof obligation discharged by construction, not a runtime check. So a fabric whose routing has been hand-modified for a link failure has lost the proof and nothing in the hardware knows. The one observable consequence is c_discards beginning to count, which is why a lossless fabric's discard counter going from zero to non-zero is a topology event rather than a traffic event, and the correct response is to re-examine the routing rather than the load.
17. The Two Fabrics, Priced Side by Side
Everything this chapter derived, at 64 ports and 400 Gb/s, in one unit.
| Mechanism | Ethernet, BCE | Lossless, BCE | Note |
|---|---|---|---|
| reserved buffer, 100 m, 1 lane | 0 | 3.738 × 10⁷ | 131.9 datapaths, 6.65% of a merchant switch |
| the same, 8 lanes | 0 | 2.990 × 10⁸ | 53.2% of the switch |
| the same, 2 km, 1 lane | 0 | 5.140 × 10⁸ | 91.47% — the switch is consumed |
| flow-control counters, 8 lanes, 64 ports | ≈ 4 × 10⁴ — PAUSE timers | 655 360 | 2.31 datapaths |
| PFC headroom, 8 priorities, 100 m | 1.376 × 10⁸ | — | Ethernet's own version of the same cost |
| deadlock machinery | 0 — a drop breaks every cycle | a timer per lane, plus a routing restriction | the restriction costs path diversity |
| congestion blast radius, 2 hops | up to 12 167 pairs | 12 167 pairs, certainly | the same number, a different modality |
Row five is the honest row and it is the one an Ethernet advocate should be made to read. 8 × 33 584 bytes across 64 ports is 1.376 × 10⁸ BCE — 24.5% of Chapter 23.3's switch — so an Ethernet fabric that actually enables PFC on eight priorities has paid 46.0% of what the credit fabric pays, for a weaker guarantee.
Lossless is not a property a fabric has or lacks. It is a bandwidth-delay product somebody bought, and both fabrics offer it at a price that scales with the cable.
And the four things each fabric gets in return.
| Ethernet gets | A credit fabric gets | |
|---|---|---|
| 1 | no reservation — the buffer is shared and over-committed 2 793% at idle | no Chapter 21.6 — every congestion event is visible as a stall |
| 2 | no deadlock, structurally | no retransmission in the common case |
| 3 | arbitrary reach | a simpler transport |
| 4 | a drop with no reporter | a stall with no counter |
Rows four are the symmetric admission and Section 14's callout is about them.
18. What the Comparison Assumes
Eight assumptions, each with the direction the chapter moves if it fails.
| # | Assumption | If it is false |
|---|---|---|
| 1 | the credit turnaround is 300 ns | every reservation moves by 400 Gb/s × Δ ÷ 8 bytes — 50 bytes per nanosecond |
| 2 | a maximum frame is 9 000 octets | a 1 500-octet fabric saves 150 ns of loop — 7 500 bytes at 3 m, 29% of that row |
| 3 | propagation is 4.90 ns/m in fibre | Chapter 8.2 §3's figure; every medium is within 5% and the 2 km row is unaffected in substance |
| 4 | the switch is 64 ports | the fabric-scale figures scale linearly; the per-link ones do not move |
| 5 | Chapter 23.3's switch is 5.62 × 10⁸ BCE | the share percentages scale inversely; the absolute reservations do not move |
| 6 | eight virtual lanes | linear — two lanes is a quarter of Section 6's figures |
| 7 | a 1 µs round trip for RoCE's window | the amplification is unchanged; the window and the bitmap scale together |
| 8 | BCE applies | Section 19 examines it and it holds, with one caveat that is worth the space |
Assumption 2 deserves a sentence because it is a lever a design actually has. A fabric that forbids jumbo frames shortens its credit loop by 150 ns at 400 Gb/s, which at 3 m is 29% of the whole loop and at 2 km is 0.7%. So the MTU is a meaningful buffer lever on short links and irrelevant on long ones — the opposite of its effect on Chapter 8.3's efficiency curve, where it matters everywhere equally.
19. The Cost, Accounted — in BCE
This chapter's blocks.
| Block | Flops | BCE | × the datapath |
|---|---|---|---|
credit_loop_model | 32 | 640 | 0.002 |
reservation_model | 32 | 640 | 0.002 |
vl_partition, 8 lanes | 8 × 32 + 64 | 6 400 | 0.023 |
congestion_tree | 112 | 2 240 | 0.008 |
roce_window | 64 | 1 280 | 0.005 |
deadlock_detector, 8 lanes | 8 × 32 + 73 | 6 580 | 0.023 |
ib_cmp_telemetry, 8 lanes | 8 × 48 + 256 | 12 800 | 0.045 |
ib_cmp_conformance | 0 — combinational | 0 | 0 |
| this chapter's additions | 1 513 | 30 580 | 0.108 |
The whole instrumentation apparatus is 0.108 of a MAC datapath, against reservations measured in hundreds of datapaths — a ratio of about ten thousand to one. That is the argument for keeping every counter in this chapter: they are free, and without them the expensive part is unobservable.
And the designs the blocks describe.
| BCE | × the datapath | % of Chapter 23.3's switch | |
|---|---|---|---|
| reservation, 64 ports, 3 m, 1 lane | 1.306 × 10⁷ | 46.1 | 2.32% |
| reservation, 64 ports, 100 m, 8 lanes | 2.990 × 10⁸ | 1 055.4 | 53.20% |
| reservation, 64 ports, 2 km, 8 lanes | 4.112 × 10⁹ | 14 515.0 | 731.74% |
| Chapter 23.3's whole Ethernet switch | 5.62 × 10⁸ | 1 985 | 100% |
| Chapter 24.1 §4's PCIe return path, one link | 425 088 | 1.500 | 0.08% |
Row three is a fabric that cannot exist on one die and row five is the same mechanism at 1/1322nd the scale, which is the honest way to see what this chapter is about: PCIe and InfiniBand both pay a bandwidth-delay product, and the difference is that PCIe's latency is a microsecond of silicon and InfiniBand's is a kilometre of glass.
20. Properties Worth Asserting, and One Worth Refusing
Fifty-one properties in six groups, and the last group exists only because one word in the fabric's datasheet means less than it appears to.
Group A — the credit loop (9).
// A1. The loop is propagation, turnaround and one frame, and nothing else.
p_cl_composition: assert property (@(posedge clk) disable iff (!rst_n)
(metres != 0) |-> (loop_ns ==
(((2 * 32'(prop_ns_x100(int'(metres), media_e'(media)))) / 100)
+ 32'(CREDIT_TURNAROUND_NS) + 32'(frame_ns(int'(rate_gbps))))));
// A2. The reservation is the loop times the rate.
p_cl_reservation: assert property (@(posedge clk) disable iff (!rst_n)
(reserved_bytes_o == (32'(rate_gbps) * loop_ns) / 32'd8));
// A3. It is a floor, always, and never a target.
p_cl_is_floor: assert property (@(posedge clk) disable iff (!rst_n)
(reservation_is_a_floor == 1'b1));
// A4. A short buffer caps the rate rather than risking overflow.
p_cl_short_caps: assert property (@(posedge clk) disable iff (!rst_n)
buffer_insufficient |-> (achievable_gbps < rate_gbps));
// A5. And an adequate buffer does not.
p_cl_adequate: assert property (@(posedge clk) disable iff (!rst_n)
!buffer_insufficient |-> (achievable_gbps == rate_gbps));
// A6. Propagation dominates the loop on long links.
p_cl_prop_dominates: assert property (@(posedge clk) disable iff (!rst_n)
(metres >= 16'd1000) |-> (prop_share_pct >= 32'd90));
// A7. And does not on short ones.
p_cl_prop_minor: assert property (@(posedge clk) disable iff (!rst_n)
(metres <= 16'd10) |-> (prop_share_pct <= 32'd20));
// A8. An Ethernet link reserves nothing.
p_cl_ethernet_zero: assert property (@(posedge clk) disable iff (!rst_n)
(ethernet_reservation_bce == 32'd0));
// A9. A shortfall is counted every cycle it persists.
p_cl_stall_counted: assert property (@(posedge clk) disable iff (!rst_n)
buffer_insufficient |=> (c_stall_events == $past(c_stall_events) + 32'd1));Group B — the fabric-scale reservation (8).
// B1. The fabric figure is the per-link figure times the ports.
p_rs_scales: assert property (@(posedge clk) disable iff (!rst_n)
(fabric_reservation_e2 == (per_link_bce / 32'd100) * 32'(ports)));
// B2. Over-commitment is never permitted.
p_rs_no_borrow: assert property (@(posedge clk) disable iff (!rst_n)
(over_commitment_allowed == 1'b0));
// B3. A lossy fabric reserves nothing.
p_rs_lossy_zero: assert property (@(posedge clk) disable iff (!rst_n)
(lossy_equivalent_bce == 32'd0));
// B4. Exceeding the switch is flagged rather than silently accepted.
p_rs_exceeds_flagged: assert property (@(posedge clk) disable iff (!rst_n)
(share_of_switch_x100 >= 16'd10_000) |-> exceeds_switch);
// B5. And counted.
p_rs_infeasible_counted: assert property (@(posedge clk) disable iff (!rst_n)
exceeds_switch |=> (c_infeasible_configs ==
$past(c_infeasible_configs) + 32'd1));
// B6. The ports a buffer serves falls as the reservation rises.
p_rs_monotone: assert property (@(posedge clk) disable iff (!rst_n)
(per_link_bce > $past(per_link_bce)) |->
(ports_a_buffer_serves <= $past(ports_a_buffer_serves)));
// B7. Zero ports is never reported for a nonzero buffer.
p_rs_nonzero: assert property (@(posedge clk) disable iff (!rst_n)
((switch_buffer_mb != 0) && (per_link_bce <= 32'd1_000_000)) |->
(ports_a_buffer_serves != 32'd0));
// B8. The datapath figure follows the BCE figure.
p_rs_dp_follows: assert property (@(posedge clk) disable iff (!rst_n)
(fabric_reservation_e2 == 32'd0) |-> (reservation_dp_milli == 32'd0));Group C — virtual lanes (8).
// C1. The total is the sum over lanes, never the maximum.
p_vl_sum_not_max: assert property (@(posedge clk) disable iff (!rst_n)
(reservation_total_bytes == per_lane_reserved_bytes * 32'(LANES)));
// C2. Borrowing is forbidden, which is why C1 is a sum.
p_vl_no_borrow: assert property (@(posedge clk) disable iff (!rst_n)
(borrow_permitted == 1'b0));
// C3. A lane never exceeds its own reservation.
p_vl_within_lane: assert property (@(posedge clk) disable iff (!rst_n)
(occupancy[lane_sel] <= per_lane_reserved_bytes));
// C4. An arrival that would exceed it is blocked rather than accepted.
p_vl_block: assert property (@(posedge clk) disable iff (!rst_n)
(packet_arrives && ((occupancy[lane_sel] + 32'(packet_bytes))
> per_lane_reserved_bytes)) |-> lane_blocked);
// C5. And is counted.
p_vl_block_counted: assert property (@(posedge clk) disable iff (!rst_n)
(packet_arrives && lane_blocked) |=>
(c_lane_blocks == $past(c_lane_blocks) + 32'd1));
// C6. Lanes do not head-of-line block one another -- the guarantee
// they were bought for, stated so a scheduler cannot quietly break it.
p_vl_independent: assert property (@(posedge clk) disable iff (!rst_n)
(head_of_line_across_lanes == 1'b0));
// C7. The reservation fits or the configuration is rejected.
p_vl_fits: assert property (@(posedge clk) disable iff (!rst_n)
fits_in_buffer |-> (reservation_total_bytes <= total_buffer_bytes));
// C8. Occupancy never goes negative on a drain.
p_vl_no_underflow: assert property (@(posedge clk) disable iff (!rst_n)
(packet_drains && (occupancy[lane_sel] < 32'(packet_bytes))) |=>
(occupancy[lane_sel] == $past(occupancy[lane_sel])));Group D — the congestion tree (8).
// D1. The spread is certain, not probable.
p_ct_deterministic: assert property (@(posedge clk) disable iff (!rst_n)
(tree_is_deterministic == 1'b1));
// D2. Full stall is the loop times the depth.
p_ct_stall_time: assert property (@(posedge clk) disable iff (!rst_n)
(full_stall_ns == credit_loop_ns_i * 32'(stages)));
// D3. The drain takes as long as the growth.
p_ct_drain_symmetry: assert property (@(posedge clk) disable iff (!rst_n)
(drain_ns == full_stall_ns));
// D4. Hops never exceed the fabric's depth.
p_ct_hops_bounded: assert property (@(posedge clk) disable iff (!rst_n)
(hops_reached <= stages));
// D5. Hops never go backwards while the tree is active.
p_ct_monotone: assert property (@(posedge clk) disable iff (!rst_n)
(tree_is_deterministic && !congestion_clear) |=>
(hops_reached >= $past(hops_reached)));
// D6. A new event is counted once, not per cycle.
p_ct_event_once: assert property (@(posedge clk) disable iff (!rst_n)
(congestion_start && $past(congestion_start)) |=>
(c_tree_events == $past(c_tree_events)));
// D7. Stalled time accrues only while active.
p_ct_time_accrues: assert property (@(posedge clk) disable iff (!rst_n)
(!tree_is_deterministic || congestion_clear) |=>
(c_stall_ns_total == $past(c_stall_ns_total)));
// D8. Full stall is reached only after the whole depth's worth of loops.
p_ct_full_after: assert property (@(posedge clk) disable iff (!rst_n)
fully_stalled |-> (hops_reached == stages));Group E — RoCE and the transport (9).
// E1. The window is the bandwidth-delay product.
p_rc_window: assert property (@(posedge clk) disable iff (!rst_n)
(window_bytes == (32'(rate_gbps) * 32'(rtt_ns)) / 32'd8));
// E2. Go-back-N retransmits the window.
p_rc_gbn: assert property (@(posedge clk) disable iff (!rst_n)
!selective_repeat |-> (retransmit_bytes == window_bytes));
// E3. Selective repeat retransmits one packet.
p_rc_sr: assert property (@(posedge clk) disable iff (!rst_n)
selective_repeat |-> (retransmit_bytes == 32'(mtu_octets)));
// E4. And costs a bit per packet in the window, per queue pair.
p_rc_bitmap: assert property (@(posedge clk) disable iff (!rst_n)
selective_repeat |-> (bitmap_bits ==
((32'(window_packets) + 32'd1) * 32'(queue_pairs))));
// E5. Go-back-N costs no bitmap.
p_rc_no_bitmap: assert property (@(posedge clk) disable iff (!rst_n)
!selective_repeat |-> (bitmap_bits == 32'd0));
// E6. A larger MTU reduces the amplification.
p_rc_mtu_helps: assert property (@(posedge clk) disable iff (!rst_n)
(!selective_repeat && (mtu_octets > $past(mtu_octets))) |->
(amplification_x10 <= $past(amplification_x10)));
// E7. The transport assumes lossless regardless of the link.
p_rc_assumes: assert property (@(posedge clk) disable iff (!rst_n)
(transport_assumes_lossless == 1'b1));
// E8. And the link's actual state is a configuration input.
p_rc_config_input: assert property (@(posedge clk) disable iff (!rst_n)
(link_losslessness_is_configured == pfc_configured_everywhere));
// E9. Retransmitted bytes accumulate at the amplified rate.
p_rc_accumulate: assert property (@(posedge clk) disable iff (!rst_n)
packet_lost |=> (c_retransmit_bytes ==
$past(c_retransmit_bytes) + $past(retransmit_bytes)));Group F — deadlock, telemetry and conformance (9).
// F1. The timeout exceeds the credit loop with margin.
p_dl_timeout_margin: assert property (@(posedge clk) disable iff (!rst_n)
((timeout_us * 32'd1000) >= (credit_loop_ns_i * 32'd10)));
// F2. A starved lane is one with work and no credit.
p_dl_starved_def: assert property (@(posedge clk) disable iff (!rst_n)
(lane_starved == (lane_has_work & ~lane_has_credit)));
// F3. A discard clears the lane's starvation timer.
p_dl_discard_clears: assert property (@(posedge clk) disable iff (!rst_n)
discard_required |=> (starved_us[$past(discard_lane)] == 32'd0));
// F4. And permanently retires the lossless claim.
p_dl_claim_retired: assert property (@(posedge clk) disable iff (!rst_n)
discard_required |=> !is_still_lossless);
// F5. The claim, once retired, never returns.
p_dl_claim_sticky: assert property (@(posedge clk) disable iff (!rst_n)
!is_still_lossless |=> !is_still_lossless);
// F6. A fabric claiming lossless after a discard is a violation.
p_cf_stale_claim: assert property (@(posedge clk) disable iff (!rst_n)
(claims_lossless && discard_fired) |-> v_lossless_claim_stale);
// F7. The drop counter is labelled as uninformative.
p_tl_drop_label: assert property (@(posedge clk) disable iff (!rst_n)
(drop_counter_is_meaningful == 1'b0));
// F8. Stalled percentage is bounded by 100.
p_tl_pct_bounded: assert property (@(posedge clk) disable iff (!rst_n)
(stalled_pct <= 16'd100));
// F9. Conformance is the disjunction of its six checks.
p_cf_vector: assert property (@(posedge clk) disable iff (!rst_n)
conformant |-> (violations == 6'b000000));Coverage — states a lossless fabric only reaches under a specific load.
c_cl_short_buffer: cover property (@(posedge clk) buffer_insufficient);
c_cl_long_link: cover property (@(posedge clk) metres >= 16'd1000);
c_rs_exceeds: cover property (@(posedge clk) exceeds_switch);
c_vl_blocked: cover property (@(posedge clk) lane_blocked);
c_vl_one_lane_hot: cover property (@(posedge clk)
(occupancy[0] > (per_lane_reserved_bytes / 32'd2)) &&
(occupancy[1] == 32'd0));
c_ct_full_stall: cover property (@(posedge clk) fully_stalled);
c_ct_two_hops: cover property (@(posedge clk) hops_reached >= 16'd2);
c_rc_gbn_loss: cover property (@(posedge clk) packet_lost && !selective_repeat);
c_rc_sr_loss: cover property (@(posedge clk) packet_lost && selective_repeat);
c_dl_starved: cover property (@(posedge clk) lane_starved != 8'd0);
c_dl_discard: cover property (@(posedge clk) discard_required);
c_dl_not_lossless: cover property (@(posedge clk) !is_still_lossless);
c_tl_stalled_high: cover property (@(posedge clk) stalled_pct >= 16'd5);
c_cf_stale: cover property (@(posedge clk) v_lossless_claim_stale);21. Verification Scenarios
Fifty-eight scenarios in six groups, plus one directed test random stimulus will not produce.
Group 1 — the credit loop (10).
| # | Scenario | Expect |
|---|---|---|
| 1 | 3 m copper, 400 Gb/s | loop 510.3 ns; 25 515 B reserved |
| 2 | 30 m fibre, 400 Gb/s | loop 774 ns; 38 700 B |
| 3 | 100 m fibre, 400 Gb/s | loop 1 460 ns; 73 000 B |
| 4 | 2 km fibre, 400 Gb/s | loop 20 080 ns; 1 004 000 B |
| 5 | prop_share_pct at 3 m | 5.9% — implementation can help |
| 6 | prop_share_pct at 2 km | 97.6% — nothing can |
| 7 | a 32 kB buffer at 100 m | buffer_insufficient; 179 Gb/s — 44.9% |
| 8 | a 73 kB buffer at 100 m | adequate; full rate |
| 9 | provisioned for 30 m, cabled at 2 km | 38 700 B against 1 004 000 needed; 15.4 Gb/s — 3.9% |
| 10 | ethernet_reservation_bce at any setting | 0 |
Group 2 — the fabric-scale reservation (10).
| # | Scenario | Expect |
|---|---|---|
| 11 | 64 ports, 3 m, 1 lane | 1.306 × 10⁷ BCE; 46.1 datapaths; 2.32% |
| 12 | 64 ports, 100 m, 1 lane | 3.738 × 10⁷; 131.9 datapaths; 6.65% |
| 13 | 64 ports, 2 km, 1 lane | 5.140 × 10⁸; 1 814.4 datapaths; 91.47% |
| 14 | 64 ports, 2 km, 8 lanes | exceeds_switch; 731.74% |
| 15 | a 64 MB buffer at 3 m | 2 630 ports served |
| 16 | the same at 100 m | 919 ports |
| 17 | the same at 2 km | 66 ports — the buffer is the reservation |
| 18 | 48 × 3 m plus 16 × 2 km | 1.383 × 10⁸ BCE; 24.6%; the 16 long links are 92.9% |
| 19 | the same, budgeted from the mean length | a quarter of the requirement; the uplinks miss line rate |
| 20 | over_commitment_allowed at any setting | 0 |
Group 3 — virtual lanes (9).
| # | Scenario | Expect |
|---|---|---|
| 21 | 8 lanes, 100 m | 584 000 B per port; 2.990 × 10⁸ BCE for 64; 53.20% |
| 22 | 2 lanes, 100 m | 146 000 B per port; 13.30% |
| 23 | one lane at capacity, seven empty | 7/8 of the reservation idle and unusable |
| 24 | an arrival that would exceed a lane's share | lane_blocked; c_lane_blocks increments |
| 25 | the same lane drains, then the arrival retries | accepted |
| 26 | a drain larger than the occupancy | no underflow — p_vl_no_underflow |
| 27 | borrow_permitted at any setting | 0 |
| 28 | the reservation exceeding the buffer | fits_in_buffer low |
| 29 | PFC, 8 priorities, 100 m, 64 ports | 1.376 × 10⁸ BCE — 46.0% of the credit scheme's |
Group 4 — the congestion tree (10).
| # | Scenario | Expect |
|---|---|---|
| 30 | congestion at 3 m, 5 stages | full stall in 2.55 µs |
| 31 | the same at 100 m | 7.30 µs |
| 32 | the same at 2 km | 100.40 µs |
| 33 | the drain after the cause clears | the same duration again |
| 34 | radix 24, 0 hops | 23 pairs — Chapter 14.3 §8's figure |
| 35 | radix 24, 1 hop | 529 pairs |
| 36 | radix 24, 2 hops | 12 167 pairs |
| 37 | tree_is_deterministic at any setting | 1 — the distinction from Chapter 14.3 |
| 38 | congestion asserted for two consecutive cycles | one event counted, not two |
| 39 | 8% of a second stalled | stalled_pct reports 8; c_drops reports 0 |
Group 5 — RoCE (9).
| # | Scenario | Expect |
|---|---|---|
| 40 | 400 Gb/s, 1 µs, MTU 1 500, go-back-N | window 50 000 B; 33.3 packets; 33.3× amplification |
| 41 | the same at MTU 9 000 | 5.6 packets; 5.6× amplification |
| 42 | selective repeat, MTU 1 500 | 1.0×; a 34-bit bitmap per queue pair |
| 43 | 16 384 queue pairs with that bitmap | 557 056 BCE; 1.97 datapaths; 1.1% of a NIC |
| 44 | 10⁻³ drop rate, MTU 1 500, go-back-N | 3.33% of the link retransmitted |
| 45 | 10⁻² drop rate, MTU 1 500, go-back-N | 33.3% of the link |
| 46 | 10⁻³ drop rate, MTU 9 000, go-back-N | 0.56% |
| 47 | pfc_configured_everywhere low | the transport behaves identically; only throughput moves |
| 48 | transport_assumes_lossless at any setting | 1 |
Group 6 — deadlock, telemetry, conformance (10).
| # | Scenario | Expect |
|---|---|---|
| 49 | timeout at 10× the loop, 3 m | 5.10 µs |
| 50 | timeout at 100× the loop, 2 km | 2 008 µs |
| 51 | one timeout value across mixed lengths | too short for 2 km or 39.4× too long for 3 m |
| 52 | a lane starved past the timeout | discard_required; is_still_lossless goes low |
| 53 | the claim after a discard | stays low — p_dl_claim_sticky |
| 54 | claims_lossless high with discard_fired | v_lossless_claim_stale |
| 55 | advertised credit above the backing buffer | v_overadvertised |
| 56 | a timeout below 10× the loop | v_timeout_too_short |
| 57 | drop_counter_is_meaningful at any setting | 0 |
| 58 | all six checks clear | conformant high |
22. Debugging a Lossless Fabric
Six symptoms, and five of them present as a number that is fine.
| Symptom | First question | Where to look |
|---|---|---|
| a link at a fixed fraction of its rate, no errors | is buffer_insufficient set? | Section 3 — the cable is longer than the buffer was provisioned for |
| zero drops and an unhappy application | what is stalled_pct? | Section 15 — the loss became time and the drop counter cannot see it |
| a fabric-wide slowdown from one hot port | how many hops has hops_reached reached? | Section 9 — the tree is the mechanism working |
| lanes configured and behaving like one lane | per-lane throughput under a single-lane congestion | Section 21's directed test — the scheduler is shared |
| a discard counter that has just left zero | has the routing changed? | Section 16 — a discard is a topology event, not a traffic one |
| a few RoCE queue pairs far slower than the rest | per-queue-pair c_retransmit_bytes | Section 11 — one switch on one path without PFC |
Row two is the one that is hardest to convince anybody of, because the fabric's own telemetry is reporting a perfect score.
23. Misconceptions
Misconception 1 — "lossless means no packet is ever dropped."
The wrong model: a credit-based fabric never discards, so a transport on it never needs to retransmit.
What it costs: a transport with no retransmission path, deployed on a fabric that discards on deadlock timeout. Section 13: every deployed lossless fabric has a starvation timer that discards, because without it a routing cycle is permanent. The guarantee is no drop for want of buffer, not no drop.
The corrected model: is_still_lossless is a latched bit and it can go low. A fabric that has fired its discard has dropped a packet and must stop claiming otherwise, which is v_lossless_claim_stale.
Misconception 2 — "the credit counters are what flow control costs."
The wrong model: credits are a handful of registers per port.
What it costs: a buffer budget wrong by two orders of magnitude. Section 4: the counters for 8 lanes across 64 ports are 655 360 BCE — 2.31 datapaths — and the reservation they account for at 100 m is 3.738 × 10⁷, a factor of 57.
The corrected model: the credit is a promise of buffer and the promise is the cost. Chapter 24.1 §7 made the same point about PCIe with a factor of 6.9; the factor here is larger because the latency is a cable rather than a chip.
Misconception 3 — "PFC makes Ethernet lossless, so the comparison is moot."
The wrong model: enable priority flow control and Ethernet becomes InfiniBand.
What it costs: a fabric that is lossless on the priority somebody configured, on the switches somebody remembered, and lossy everywhere else. Section 6: PFC's headroom is sized to a one-way dead time and a credit reservation to a round trip, so the credit scheme is about 2× on every link — and Section 11: pfc_configured_everywhere is a register somebody wrote, and the transport cannot verify it.
The corrected model: PFC gives Ethernet an optional, per-priority, emergency version of what credits do continuously and mandatorily. The guarantee is weaker and the cost is real: 8 priorities at 100 m is 1.376 × 10⁸ BCE, 24.5% of a merchant switch.
Misconception 4 — "a lossless fabric performs better because its drop counter is zero."
The wrong model: compare the two fabrics' loss rates and pick the lower one.
What it costs: choosing a fabric that delivers less capacity. Section 14: a fabric with a zero drop counter and 8% stalled time has delivered 92%, which is the same outcome as an 8% loss rate, and only one of the two reports a number an operator reacts to.
The corrected model: the comparable quantity is delivered capacity, and on a lossless fabric the loss-equivalent counter is c_starved_us. A comparison of drop counters is not a comparison, and Section 20's class 113 is the assertion form of the same mistake.
Misconception 5 — "long links are just more expensive."
The wrong model: the reservation scales with distance, so a 2 km fabric costs somewhat more than a 3 m one.
What it costs: a design that budgets linearly and discovers a cliff. Section 12: the ratio is 39.4× and it appears identically in four different quantities — buffer, share of the switch, stall time and timeout — and at 2 km with 8 lanes the reservation is 490 MB, which is not on-die memory at all.
The corrected model: past a few hundred metres the mechanism leaves the die and the unit changes from BCE to bytes and a bandwidth, which is Section 19's caveat and the reason every deployed lossless fabric is a rack, a row or a chassis.
Misconception 6 — "RoCE is InfiniBand on Ethernet, so it inherits InfiniBand's reliability."
The wrong model: the transport is the same, therefore the behaviour is.
What it costs: a 33.3× retransmission amplification on a link that drops. Section 10: RoCE inherits the IB transport, which was designed for a link that does not drop, and Ethernet's link, which does unless configured otherwise. At a 10⁻³ drop rate and a 1 500-octet MTU that is 3.33% of a 400 Gb/s link.
The corrected model: a protocol's efficiency decisions are made against an assumed error rate, and moving it to a different link changes the rate without changing the decisions. The repair is selective repeat — 34 bits per queue pair, 1.1% of a NIC — or a 9 000-octet MTU, which reduces the amplification six-fold for the cost of one configuration field.
24. Interview Questions
Six, with what a strong answer contains.
1. Why must a lossless link reserve buffer, and how much?
Because a credit is a promise that space exists, and the sender must be able to keep transmitting while credits are in flight. The reservation is the bandwidth-delay product of the credit return loop: two propagations, a turnaround, and one maximum frame. A strong answer puts numbers on it — 25 515 B at 3 m and 400 Gb/s, 1 004 000 B at 2 km — and observes that the ratio, 39.4×, is the chapter's only real parameter and it is chosen by whoever ran the cable.
2. What does credit-based flow control put back in exchange for the drop it removes?
Head-of-line blocking that is required rather than probable, and a congestion tree that grows one hop per credit loop. A strong answer cites Chapter 14.3 §8's blast-radius numbers — 23 pairs, 529 at one hop, 12 167 at two — and makes the sharp point: on a credit fabric those are certainties rather than bounds, because the upstream is not merely likely to stall, it is required to. At 3 m and five stages the whole fabric stalls in 2.55 µs.
3. Why can an Ethernet fabric not deadlock?
Because a drop breaks every cycle. A full buffer on Ethernet discards, and the moment any participant discards, the dependency it represented is gone. A credit fabric's full buffer withholds credit instead, so a cyclic dependency holds forever. The strong answer draws the uncomfortable conclusion: Chapter 21.6's silent, unreported, undiagnosable drop is the same drop that keeps the fabric live, and removing it removes both.
4. A 400 Gb/s lossless link runs at 3.9% of its rate with no errors. What do you check?
The cable length against the provisioned buffer. 3.9% × 400 Gb/s is 15.4 Gb/s, and 38 700 bytes ÷ 20.08 µs is exactly that — a link provisioned for 30 m and cabled at 2 km. A strong answer names the structural point: a short credit buffer does not risk overflow, it caps the rate permanently, which is Chapter 21.8 §5's ceiling with the credit loop as the denominator.
5. Where does RoCE sit, and what does it inherit from each parent?
InfiniBand's transport on Ethernet's link. From InfiniBand: queue pairs, verbs, and a reliable-connected service that assumes the link does not drop. From Ethernet: a link that does, unless PFC is configured on every switch on the path. The strong answer prices the mismatch — go-back-N amplifies one loss by 33.3× at a 1 500-octet MTU and 5.6× at 9 000 — and names the repair as 34 bits per queue pair, 1.1% of a NIC.
6. A lossless fabric reports zero drops. What have you learned?
That it is a lossless fabric. The counter is a statement about the design rather than about the deployment. The quantity that moved is stalled time, which most implementations do not count at all: a fabric at 8% stalled has delivered 92% of its capacity and reported a perfect score. A strong answer names Section 20's class 113 — a property whose outcome space is a partition the mechanism invalidated — and gives the repair: assert a residence-time bound with three branches, and a stall budget that can fail.
25. Questions and Answers
26. What's Next
Chapter 24.1 set Ethernet against a fabric that answers a different question, and this chapter against one that answers the same question differently. Chapter 24.3 asks what a design gets by answering it outside a standard altogether.
The question is one every chapter of Module 23 answered implicitly by staying inside one. A proprietary fabric is free of Ethernet's preamble, its interframe gap, its frame check sequence and its forty-eight-bit addresses — and Chapter 24.3 begins by deriving what those actually cost as a fraction of the wire, at both ends of Chapter 8.3 §2's curve.
The answer at the small end is 45.24% and at the large end 0.42%, and the chapter's first punch is that the fabrics which actually leave the standard are the ones carrying large transfers. So they leave for four tenths of a per cent — which means the win, if there is one, is somewhere else entirely.
It is, and the chapter finds it in the lookup rather than the framing: Chapter 23.3 §2's 36-stage pipeline is 30 ns per hop, and a source-routed fabric that carries its route in the header decodes in three stages. Then it prices what leaving the standard obliges a vendor to rebuild — a management protocol, a time protocol, a congestion protocol, test equipment, a second source and an entire verification apparatus with no reference implementation to check against — and closes with Chapter 23.6 §19's argument: the win is real, and it is not in the unit people quote it in.
Continue learning
Related tutorials
- Related topic
Backpressure and Head-of-Line Blocking
Every queue is work-conserving, every link runs at line rate, every switch is non-blocking — and the composition delivers 58.6% of capacity. The bound has been known since 1987.
- Related topic
Priority Flow Control and Lossless Ethernet
PFC subdivides a link into eight independently stoppable classes, costs eight headrooms instead of one, and admits a failure Ethernet has never had: a cycle of classes each waiting for the next.
- Related topic
Ethernet against PCIe
An address that cannot miss, a read that comes back, and the 425 088 bits of state a 400 Gb/s return path must hold — against an Ethernet transmitter that holds nothing.
- Related topic
PCIe vs Ethernet — Where the Cost of Overload Lands
The same overload into two fabrics: one stalled the sender 59,405 times and lost nothing, the other discarded 59,405 frames. That single choice explains why one needs TCP and the other does not.
Standards & specifications
- Governing standard
- IEEE Std 802.3 (Ethernet)(opens IEEE in a new tab)
Defines the Ethernet MAC, the media-independent interfaces and the physical-layer sublayers, including framing, access control, auto-negotiation and per-rate PHY specifications. VLAN tagging, priority and time-sensitive shaping are defined by IEEE 802.1, not by 802.3.
This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.
Where this fits
Part of the Ethernet curriculum.
