Skip to content
VLSI Mentor

Ethernet · Module 24

Ethernet against InfiniBand

Credit-based flow control removes the drop and reserves a round trip of buffer for it — 2.32% of a merchant switch at 3 m and 91.47% at 2 km, per link, per lane, forever.

Chapter 21.6 is an entire chapter about a frame that is gone with no reporter. InfiniBand does not have that chapter, because its links do not drop — and this chapter derives what that costs and what it puts back.

The mechanism is one sentence. A sender transmits only into buffer space the receiver has explicitly guaranteed, counted in credits, and the receiver returns credits as it drains. A frame is never discarded for want of space, because a frame is never sent into space that does not exist.

And the first-order consequence is that a whole class of Ethernet engineering disappears.

EthernetInfiniBand
congestion producesa drop, with no reporterbackpressure, immediately
the transport mustdetect loss and retransmitnot
Chapter 21.6's apparatusnecessaryhas nothing to diagnose
the receiver's buffersized for the traffic you expectsized for a round trip, mandatory

Row four is where the bill arrives and Sections 2 to 5 are about it.


1. Scope — A Drop Removed and a Reservation Created

Congestion at an egress port, handled two ways. On the Ethernet path the buffer fills and frames are discarded, the congestion stays confined to that port, and the transport above must detect the loss and retransmit. Chapter twenty one point six's entire diagnostic apparatus exists because the discard has no reporter. On the credit path the buffer fills and credits stop returning, so the upstream sender stops, its own buffer fills, and the stall propagates one hop per credit loop until the whole fabric is stopped. Nothing is lost, the drop counter reads zero, and the cost has become stalled time, which most implementations do not count. The credit path also requires a permanent reserved buffer of one round trip per link per virtual lane, and a routing restriction to prevent deadlock, neither of which the Ethernet path needs.Egress congestsbuffer fillsDiscardEthernetConfined23 pairsNo reporterChapter 21.6 existsWithhold creditlosslessStall spreads1 hop per loopNo counterc_drops reads 0Reservation1 round trip, alwaysRoutingrestrictionor it deadlocks12
Figure 1 — the drop is removed at one end of the fabric and reappears at the other as a reservation, a stall and a routing restriction.

This chapter owns four derivations, and every one of them is an arithmetic consequence of the credit loop.

What is derived
Sections 2 to 3the reserved buffer per link per hop at 400 Gb/s — 25.5 kB at 3 m, 980.5 KiB at 2 km
Sections 4 to 5the same, priced in BCE against Chapter 23.3's 5.62 × 10⁸ switch, at one and at eight virtual lanes
Sections 6 to 9what the credits put back: deterministic head-of-line blocking and a congestion tree that grows at one hop per credit loop
Sections 10 to 13where RoCE sits, what it inherits from each parent, and the go-back-N amplification it pays for a drop

What this chapter does not own. It is not an InfiniBand tutorial: the verbs API, the subnet manager's routing algorithm, the queue-pair state machine and the management datagram format are all outside it. It uses exactly as much InfiniBand as the comparison needs, and derives every quantity rather than quoting it.

It also does not claim that lossless is wrong. It claims that lossless is paid for in a currency that does not appear on the datasheet, and Section 19 totals the bill.


2. The Credit Loop, and the Buffer It Reserves

The credit loop at four hundred gigabits per second. A receiver frees space and emits a credit. The credit propagates to the sender, taking five point zero five nanoseconds per metre on copper or four point nine zero on fibre. The sender's credit processing and the receiver's credit generation together take about three hundred nanoseconds. The sender's next packet propagates back, taking the same one way time again. And one maximum frame of nine thousand octets, which the sender had already committed, occupies one hundred and eighty nanoseconds. The sum is the loop, and the reservation is the loop times the rate. At three metres of copper the loop is five hundred and ten point three nanoseconds and the reservation is twenty five thousand five hundred and fifteen bytes. At two kilometres of fibre the loop is twenty thousand and eighty nanoseconds and the reservation is one million four thousand bytes, a ratio of thirty nine point three times.Receiver freesemits a creditPropagation out5.05 or 4.90 ns/mTurnaround≈ 300 ns, both endsPropagation backthe same againOne max frame180 ns at 400 Gb/sThe loop510 ns to 20 080 ns3 m: 25 515 B2.32% of a switch2 km: 1 004 000 B91.47% of a switch12
Figure 2 — four terms make the loop and one of them is a cable, which is why the reservation varies 39.4× across deployments of the same silicon.

A lossless link's reserved buffer is the bandwidth-delay product of its credit return loop. Derive the loop first.

Three things happen between a receiver freeing space and the sender being able to use it.

StageWhat it isControlled by
the credit travels to the senderone-way propagationphysics
the sender's next packet travels backone-way propagation againphysics
credit generation and processingdetect, build the message, transmit, consumeboth implementations
one maximum frame already in flightthe sender committed before the credit arrivedthe sender
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
credit loop = 2 x propagation + credit turnaround + one maximum frame

At 400 Gb/s, a 300 ns turnaround and a 9 000-octet maximum frame — using Chapter 8.2 §3's velocity figures of 5.05 ns/m for copper and 4.90 ns/m for fibre.

Link2 × propagationOne frameLoopReserved buffer
3 m DAC copper30.3 ns180 ns510.3 ns25 515 B = 24.9 KiB
30 m AOC294 ns180 ns774 ns38 700 B = 37.8 KiB
100 m fibre980 ns180 ns1 460 ns73 000 B = 71.3 KiB
2 km fibre19 600 ns180 ns20 080 ns1 004 000 B = 980.5 KiB

A lossless 400 Gb/s link over two kilometres must reserve just under a megabyte of receive buffer, per direction, before it carries a single useful bit.

Three properties of that table decide everything downstream.

One — the reservation is not a peak requirement, it is a floor. A buffer smaller than the loop does not merely risk overflow; it guarantees the sender stalls, because credits cannot return fast enough to keep it fed. Chapter 21.8 §5's ceiling applies exactly: the achievable rate is the credit count divided by the loop time, and the loop time is mostly propagation.

Two — it scales with the rate and the rate has moved by 400×. Propagation has not moved at all since Chapter 1.6's 10 Mb/s. So the reservation at 400 Gb/s is forty thousand times what it was, for exactly the same cable.

Three — it is per direction, per link, per virtual lane. Section 4 multiplies by all three.


3. RTL 1 — The Lossless Package and the Credit-Loop Model

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// ibcmp_pkg -- the constants for a lossless-link comparison, with every
// derived figure computed rather than written as a literal.
//
// Unit: Chapter 23.3 Section 2's bitcell equivalent.
//   1 BCE = one bit of usable on-die SRAM = 0.35 GE
//   1 flip-flop = 20 BCE
//   Chapter 19.7 Section 19's MAC receive datapath = 283 320 BCE
//   Chapter 23.3's 64-port 6.4 Tb/s switch = 5.62e8 BCE
// ---------------------------------------------------------------------
package ibcmp_pkg;

  localparam int unsigned DATAPATH_BCE   = 283_320;
  localparam int unsigned BCE_PER_FLOP   = 20;
  localparam int unsigned SWITCH_BCE_E5  = 5_620;   // 5.62e8 / 1e5
  localparam int unsigned SWITCH_PORTS   = 64;
  localparam int unsigned SWITCH_BUF_MB  = 64;

  // Chapter 8.2 Section 3's velocity figures, in hundredths of a
  // nanosecond per metre, so integer arithmetic keeps the precision.
  localparam int unsigned NS_PER_M_CU_X100 = 505;   // 5.05 ns/m
  localparam int unsigned NS_PER_M_FI_X100 = 490;   // 4.90 ns/m

  // The credit turnaround: detect, build, transmit, consume. Both
  // implementations contribute and neither publishes it.
  localparam int unsigned CREDIT_TURNAROUND_NS = 300;

  // The frame already committed when the credit arrived.
  localparam int unsigned MAX_FRAME_OCTETS = 9_000;

  typedef enum logic [1:0] {
    MEDIA_COPPER = 2'd0,
    MEDIA_FIBRE  = 2'd1
  } media_e;

  // ---- the credit loop -------------------------------------------------
  function automatic int unsigned prop_ns_x100(int unsigned metres,
                                               media_e m);
    return metres * ((m == MEDIA_COPPER) ? NS_PER_M_CU_X100
                                         : NS_PER_M_FI_X100);
  endfunction

  function automatic int unsigned frame_ns(int unsigned gbps);
    // octets x 8 bits / Gb/s gives nanoseconds directly.
    return (MAX_FRAME_OCTETS * 8) / gbps;
  endfunction

  function automatic int unsigned credit_loop_ns(int unsigned metres,
                                                 media_e m,
                                                 int unsigned gbps);
    return ((2 * prop_ns_x100(metres, m)) / 100)
         + CREDIT_TURNAROUND_NS
         + frame_ns(gbps);
  endfunction

  // ---- the reservation -------------------------------------------------
  // rate x time: Gb/s x ns / 8 gives bytes.
  function automatic int unsigned reserved_bytes(int unsigned metres,
                                                 media_e m,
                                                 int unsigned gbps);
    return (gbps * credit_loop_ns(metres, m, gbps)) / 8;
  endfunction

  function automatic int unsigned reserved_bce(int unsigned metres,
                                               media_e m,
                                               int unsigned gbps,
                                               int unsigned lanes);
    return reserved_bytes(metres, m, gbps) * 8 * lanes;
  endfunction

  // ---- reporting -------------------------------------------------------
  function automatic int unsigned datapaths_milli(int unsigned bce);
    return (bce / DATAPATH_BCE) * 1000
         + (((bce % DATAPATH_BCE) * 1000) / DATAPATH_BCE);
  endfunction

  // Share of Chapter 23.3's switch, in hundredths of a per cent, so a
  // fraction below one per cent is still a nonzero integer.
  function automatic int unsigned switch_share_x100(int unsigned bce_e2);
    // bce_e2 is the figure divided by 100, to keep the product in range.
    return (bce_e2 * 10_000) / (SWITCH_BCE_E5 * 1_000);
  endfunction

endpackage
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// credit_loop_model -- what a lossless link must reserve, derived from
// the medium rather than configured.
//
// The output that matters is reservation_is_a_floor, which is always
// high: a buffer below the loop does not risk overflow, it guarantees
// a stall, and the two failures look nothing alike.
// ---------------------------------------------------------------------
module credit_loop_model
  import ibcmp_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [15:0] metres,
  input  logic  [1:0] media,            // media_e
  input  logic [15:0] rate_gbps,
  input  logic [31:0] buffer_bytes,     // what the design implements

  output logic [31:0] loop_ns,
  output logic [31:0] prop_share_pct,
  output logic [31:0] reserved_bytes_o,
  output logic [31:0] reserved_bce_o,
  output logic        reservation_is_a_floor,
  output logic        buffer_insufficient,
  output logic [15:0] achievable_gbps,
  output logic [31:0] ethernet_reservation_bce,
  output logic [31:0] c_stall_events
);
  always_comb begin
    loop_ns          = 32'(credit_loop_ns(int'(metres), media_e'(media),
                                          int'(rate_gbps)));
    reserved_bytes_o = 32'(reserved_bytes(int'(metres), media_e'(media),
                                          int'(rate_gbps)));
    reserved_bce_o   = reserved_bytes_o * 32'd8;

    // How much of the loop is physics. Above 2 km it is essentially all
    // of it, which is why no implementation choice helps.
    prop_share_pct = (loop_ns == 0) ? 32'd0
                   : ((2 * 32'(prop_ns_x100(int'(metres), media_e'(media))))
                      / 100) * 32'd100 / loop_ns;

    // THE structural point of the chapter.
    reservation_is_a_floor = 1'b1;
    buffer_insufficient    = (buffer_bytes < reserved_bytes_o);

    // Chapter 21.8 Section 5's ceiling: if the buffer is short, the
    // rate is the buffer divided by the loop.
    achievable_gbps = (loop_ns == 0) ? rate_gbps
                    : (buffer_insufficient
                       ? 16'((buffer_bytes * 32'd8) / loop_ns)
                       : rate_gbps);

    // An Ethernet link reserves nothing. It drops instead.
    ethernet_reservation_bce = 32'd0;
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n)                    c_stall_events <= '0;
    else if (buffer_insufficient)  c_stall_events <= c_stall_events + 32'd1;
  end
endmodule

Classification: a bandwidth-delay product with a medium's velocity factor in it, producing a floor rather than a target.

What it teaches: that reservation_is_a_floor is hard-wired high, and that a buffer below the floor fails differently from one that is merely small. A short Ethernet buffer drops under burst, which is intermittent and load-dependent. A short credit buffer caps the link's rate permanently at buffer ÷ loop, which is Chapter 21.8 §5's ceiling with a new denominator. At 100 m and 400 Gb/s a 32 kB buffer delivers 32 768 × 8 ÷ 1 460 = 179 Gb/s — 44.9% of the link, steadily, on an idle fabric.

And it teaches that prop_share_pct is the number that decides whether engineering can help. At 3 m it is 5.9% — the turnaround and the frame dominate, and a faster credit path is worth building. At 2 km it is 97.6%, and nothing an implementation does moves the reservation at all.

Deliberately simplified: CREDIT_TURNAROUND_NS is a single constant where the real figure is the sum of two implementations' internal delays, is not published by either, and varies with how busy the return direction is — a credit message queued behind data is Chapter 14.2 §11's problem in a new protocol. The model assumes one maximum frame in flight where a sender with several small frames committed has less. And achievable_gbps assumes the buffer is the only limit, ignoring that a short buffer also increases the credit message rate, which consumes return bandwidth.

Production implication: the reservation is set at design time from an assumed cable length, and the cable is chosen by whoever installs the rack. A switch designed for 3 m direct-attach copper and deployed with 100 m optics needs 2.9 times the buffer — 73 000 bytes against 25 515 — and the symptom is a link that negotiates, passes every test, and runs at 34.9% of its rate with no errors anywhere. buffer_insufficient is one comparison evaluated at configuration time, and it converts that into a message at bring-up. A design that does not compute it has made the cable length an undocumented requirement.


4. The Reservation at Fabric Scale, in BCE

One link's reservation is a number on a datasheet. Sixty-four of them is a die area, and this section computes it against a switch that exists.

Chapter 23.3 §2's 64-port 100 Gb/s merchant ASIC is 5.62 × 10⁸ BCE, of which 91.0% is a 64 MB packet buffer. Take the same port count at 400 Gb/s and ask what a lossless version would have to reserve.

Link, 64 ports at 400 Gb/sPer link× 64, BCEDatapathsShare of the switch
3 m DAC copper204 120 BCE1.306 × 10⁷46.12.32%
30 m AOC309 600 BCE1.981 × 10⁷69.93.53%
100 m fibre584 000 BCE3.738 × 10⁷131.96.65%
2 km fibre8 032 000 BCE5.140 × 10⁸1 814.491.47%

At two kilometres, the reservation alone is 91.47% of an entire merchant switch ASIC — and the switch still needs a packet buffer, a forwarding table and a pipeline.

Row four is the chapter's headline and it deserves to be stated as an impossibility rather than as a cost. A 64-port 400 Gb/s lossless switch over 2 km links cannot be built as a single die at Chapter 23.3's area, because the reservation alone consumes what that die spends on everything.

Rows one to three are the ones that get built, and the reason lossless fabrics look the way they do is visible in them.

Why the deployment looks like this
short linksrow one is 2.32% and row four is 91.47% — a factor of 39.4
direct-attach copper inside a rack3 m is the cheapest row and it is not close
a separate long-reach tier that is not losslessbecause row four is unaffordable

And the same table read the other way gives the capacity question.

Chapter 23.3's 64 MB buffer, spent entirely on reservations.

LinkReservation per portPorts a 64 MB buffer can serve
3 m25 515 B2 630
100 m73 000 B919
2 km1 004 000 B66

Sixty-six ports at 2 km, from a buffer sized for a 64-port switch — so the buffer is fully consumed by reservation and there is nothing left to buffer congestion with, which is the point of having a buffer.


5. RTL 2 — The Reservation Model

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// reservation_model -- the fabric-scale figure, and the comparison
// against a switch that exists.
//
// The output that matters is over_commitment_allowed, which is 0.
// Every other buffer in this track is affordable because it is shared
// against a statistical argument; this one is not.
// ---------------------------------------------------------------------
module reservation_model
  import ibcmp_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [15:0] ports,
  input  logic [31:0] per_link_bce,
  input  logic [31:0] switch_buffer_mb,

  output logic [31:0] fabric_reservation_e2,   // BCE / 100
  output logic [31:0] reservation_dp_milli,
  output logic [15:0] share_of_switch_x100,
  output logic        exceeds_switch,
  output logic [31:0] ports_a_buffer_serves,
  output logic        over_commitment_allowed,
  output logic [31:0] lossy_equivalent_bce,
  output logic [31:0] c_infeasible_configs
);
  always_comb begin
    fabric_reservation_e2 = (per_link_bce / 32'd100) * 32'(ports);

    reservation_dp_milli  = 32'(datapaths_milli(
                                 (fabric_reservation_e2 * 32'd100)));

    share_of_switch_x100  = 16'(switch_share_x100(fabric_reservation_e2));
    exceeds_switch        = (share_of_switch_x100 >= 16'd10_000);

    // How many ports the switch's existing buffer could reserve for.
    ports_a_buffer_serves = (per_link_bce == 0) ? 32'd0
                          : ((switch_buffer_mb * 1024 * 1024 * 8)
                             / per_link_bce);

    // THE structural difference from every other buffer in this track.
    over_commitment_allowed = 1'b0;

    // A lossy fabric reserves nothing and drops instead.
    lossy_equivalent_bce = 32'd0;
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n)              c_infeasible_configs <= '0;
    else if (exceeds_switch) c_infeasible_configs <= c_infeasible_configs + 32'd1;
  end
endmodule

Classification: a multiplication by the port count, and a constant zero that is the reason the multiplication cannot be undone.

What it teaches: that over_commitment_allowed is zero and that this is where the arithmetic becomes unaffordable rather than merely large. Chapter 23.3 §9's shared buffer works because 63 of 64 ports are usually not congested, and the dynamic threshold hands the idle capacity to whoever needs it. A reservation must hold at the instant every port is busy, so the statistical argument is unavailable and the figure is the sum rather than the maximum.

And it teaches that ports_a_buffer_serves answers the design question directly. Chapter 23.3's 64 MB buffer reserves for 2 630 ports at 3 m, 919 at 100 m, and 66 at 2 km — so the same silicon is comfortable, adequate and exactly at the edge, and the parameter that moved is the cable.

Deliberately simplified: per_link_bce is supplied rather than computed, so the module trusts credit_loop_model's output — a real deployment binds them. The port count is uniform, where a switch with a mixture of 3 m and 2 km links has a reservation that is the sum of the mixture rather than a multiple of anything. And share_of_switch_x100 compares against a 100 Gb/s switch's total while the reservations are computed at 400, which is deliberate: the comparison asks what the reservation would consume of a die that exists, not of a hypothetical one.

Production implication: the mixture case is the one that produces a surprising bill. A leaf switch with 48 server-facing 3 m copper ports and 16 spine-facing 2 km optical ports reserves 48 × 204 120 + 16 × 8 032 000 = 1.383 × 10⁸ BCE24.6% of Chapter 23.3's entire switch, of which the 16 long links are 92.9%. A budget computed from the average link length would report a quarter of that, and the design would come up unable to reach line rate on precisely the uplinks Chapter 23.1 depends on. The reservation is a sum over links and never a product with a mean.


6. Virtual Lanes, and Why the Reservation Multiplies

Section 4's figures are for one credit pool. A fabric that needs to avoid head-of-line blocking between traffic classes needs one pool per class, and the reservation multiplies by the class count.

Why a second pool is necessary at all is Chapter 14.3's argument, reached from the other direction. A single credit pool means one blocked destination stops every packet on the link, because the packet at the head cannot move and nothing behind it can pass. That is head-of-line blocking made mandatory rather than probable.

Chapter 14.3 §4's input-queued switchA single-pool lossless link
the causea FIFO can offer only its headthe same
under uniform traffic58.6% throughput — 2 − √2the same bound applies
the remedyvirtual output queuesvirtual lanes
what the remedy costsN queues and a matching schedulerN reservations

Row four is where the two diverge. Chapter 14.3 §11's virtual output queues cost queue state and a scheduler, which that chapter §14 priced. Virtual lanes cost a full reservation each, because each lane's credits are independent and a lane may not borrow another's promised space.

So multiply Section 4's table by the lane count.

Link, 64 ports at 400 Gb/s1 lane8 lanes, BCEDatapathsShare of the switch
3 m DAC copper2.32%1.045 × 10⁸368.918.60%
30 m AOC3.53%1.585 × 10⁸559.528.21%
100 m fibre6.65%2.990 × 10⁸1 055.453.20%
2 km fibre91.47%4.112 × 10⁹14 515.0731.74%

Eight virtual lanes over 100 m consume 53.2% of a merchant switch in reservation alone, and over 2 km they consume seven of them.

Three consequences follow and each is visible in a real fabric.

One — lane counts are small in practice. InfiniBand's specification allows fifteen data lanes plus a management lane; deployments use two to four, and Section 4's table says why: the reservation is linear in the lane count and the die is not.

Two — the lanes are not equal. A design that gives one lane the full reservation and the others a fraction has a lossless lane and several nearly-lossless ones, which is a legitimate engineering answer and is exactly what Chapter 14.4's per-priority headroom does on Ethernet.

Three — the share is computed against a lossy switch's die and the lossless switch still needs everything else. The reservation is in addition to the packet buffer, the forwarding tables and the pipeline, not instead of them.


7. RTL 3 — The Virtual-Lane Partition

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// vl_partition -- independent credit pools, and the reservation each
// one obliges.
//
// The output that matters is borrow_permitted, which is 0. A lane with
// spare promised space may not lend it to a starving neighbour, and
// that single fact is why the reservation is a sum rather than a max.
// ---------------------------------------------------------------------
module vl_partition
  import ibcmp_pkg::*;
#(
  parameter int unsigned LANES = 8
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [31:0] per_lane_reserved_bytes,
  input  logic [31:0] total_buffer_bytes,
  input  logic  [3:0] lane_sel,
  input  logic        packet_arrives,
  input  logic        packet_drains,
  input  logic [15:0] packet_bytes,

  output logic [31:0] reservation_total_bytes,
  output logic [31:0] reservation_total_bce,
  output logic        fits_in_buffer,
  output logic [15:0] buffer_used_pct,
  output logic        borrow_permitted,
  output logic        lane_blocked,
  output logic        head_of_line_across_lanes,
  output logic [31:0] occupancy [LANES],
  output logic [31:0] c_lane_blocks,
  output logic [31:0] c_wasted_reserved_bytes
);
  always_comb begin
    reservation_total_bytes = per_lane_reserved_bytes * 32'(LANES);
    reservation_total_bce   = reservation_total_bytes * 32'd8;

    fits_in_buffer  = (reservation_total_bytes <= total_buffer_bytes);
    buffer_used_pct = (total_buffer_bytes == 0) ? 16'd0
                    : 16'((reservation_total_bytes * 32'd100)
                          / total_buffer_bytes);

    // THE reason the reservation is a sum. Chapter 23.3 Section 9's
    // dynamic threshold lends idle space between ports; a credit
    // promise cannot be lent, because it was already promised.
    borrow_permitted = 1'b0;

    lane_blocked = (occupancy[lane_sel] + 32'(packet_bytes))
                 > per_lane_reserved_bytes;

    // And the reason lanes exist: a block on one lane does not stop
    // the others, which a single pool could not offer.
    head_of_line_across_lanes = 1'b0;
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int i = 0; i < LANES; i++) occupancy[i] <= '0;
      c_lane_blocks <= '0; c_wasted_reserved_bytes <= '0;
    end else begin
      if (packet_arrives && !lane_blocked)
        occupancy[lane_sel] <= occupancy[lane_sel] + 32'(packet_bytes);
      if (packet_drains && (occupancy[lane_sel] >= 32'(packet_bytes)))
        occupancy[lane_sel] <= occupancy[lane_sel] - 32'(packet_bytes);
      if (packet_arrives && lane_blocked)
        c_lane_blocks <= c_lane_blocks + 32'd1;

      // The quantity nobody reports: space that was promised, is empty,
      // and cannot be used by the lane that needs it.
      c_wasted_reserved_bytes <= '0;
      for (int i = 0; i < LANES; i++)
        c_wasted_reserved_bytes <= c_wasted_reserved_bytes
                                 + (per_lane_reserved_bytes - occupancy[i]);
    end
  end
endmodule

Classification: N independent pools with an explicit prohibition on sharing, and a counter for the space the prohibition wastes.

What it teaches: that borrow_permitted is zero and that this single bit separates a reservation from every other buffer in this track. Chapter 23.3 §9's dynamic threshold is exactly borrowing — one port holds 27.8 MB because 63 others are idle — and it is what makes a 64 MB buffer serve a 6.4 Tb/s switch. Remove borrowing and the same guarantee costs the sum instead of the maximum, which at eight lanes is a factor of eight.

And it teaches that c_wasted_reserved_bytes is a quantity no fabric reports. On a link where one lane carries everything and seven are idle, seven eighths of the reservation is empty, promised, and unusable — and there is no counter in any deployed lossless fabric that says so, because from each lane's own point of view nothing is wrong.

Deliberately simplified: the occupancy accumulator is written as a loop for clarity and a synthesisable form maintains a running total incrementally. head_of_line_across_lanes is hard-wired zero, which is the guarantee lanes are supposed to provide; a real design must prove it, because a shared arbiter, a shared output queue or a shared scheduler can reintroduce the coupling the lanes were bought to remove. And the per-lane reservation is uniform, where a real design gives different lanes different guarantees.

Production implication: the coupling the simplification hides is the bug this structure actually has. Eight virtual lanes that share one egress scheduler are eight queues and one head, and Chapter 14.3 §16's warning applies exactly: virtual output queues do not fix blocking that has moved into the scheduler. The symptom is a fabric with lanes configured, reservations paid for, and head-of-line blocking that behaves as though there were one lane — and the only way to see it is to measure per-lane throughput under a load that congests exactly one of them. A design that reports c_lane_blocks per lane makes that a five-minute test; one that reports an aggregate makes it invisible.


8. What the Credits Put Back — Blocking That Is Required Rather Than Likely

A congestion tree on a credit based fabric. One egress port congests at stage zero and twenty three ingress egress pairs on that switch are affected. One credit loop later the upstream switches have exhausted their credits and up to five hundred and twenty nine pairs are affected. A second loop later, up to twelve thousand one hundred and sixty seven. On a five stage fabric with three metre links the whole topology is stalled in two point five five microseconds; with one hundred metre links, seven point three microseconds; with two kilometre links, one hundred point four microseconds. And the drain after the congestion clears takes exactly the same time again, because it is the same process running backwards. The distinction from Ethernet backpressure is that the upstream is required to stop rather than likely to, so these counts are certainties rather than bounds.Stage 023 pairs+1 loop529 pairs+2 loops12 167 pairsFully stalled5 stages3 m: 2.55 µsto full stall2 km: 100.4 µsto full stallDrainthe same time againCertain, notlikelythe upstream must stop12
Figure 3 — the congestion tree, growing one hop per credit loop, and the same table Chapter 14.3 derived with the word 'possibly' removed.

A lossless fabric does not remove congestion. It converts loss into stalling, and the stalling propagates upstream at a rate this section derives.

Start with what happens at the congested point. An egress port is oversubscribed. On Ethernet, Chapter 14.1 §4: the buffer fills and frames are discarded, and the congestion is confined to that port. On a credit link, the buffer fills and credits stop returning, so the upstream sender stops — and the upstream sender's own buffer now fills.

The propagation rate is one hop per credit loop.

LinkOne credit loopA 5-stage fabric stalls end to end in
3 m DAC copper0.510 µs2.55 µs
30 m AOC0.774 µs3.87 µs
100 m fibre1.460 µs7.30 µs
2 km fibre20.080 µs100.40 µs

On a short-reach lossless fabric, a single congested egress port stalls the entire five-stage topology in under three microseconds, deterministically.

And the reach of the damage is Chapter 14.3 §8's table, with the probabilities removed.

Frames droppedPairs affectedOn a credit fabric
congested egress, no backpressuremany23does not occur
one hop of backpressurefewup to 529certain rather than possible
two hopsvery fewup to 12 167certain rather than possible

The word that changed in column four is the whole difference. Chapter 14.3's "up to" is a bound on a probabilistic spread — PAUSE is asserted when a watermark is crossed, and a watermark is crossed sometimes. A credit sender stops the instant its credits run out, which is not a threshold and not a policy. So the congestion tree on a lossless fabric is not a risk; it is the mechanism working.


9. RTL 4 — The Congestion Tree

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// congestion_tree -- how far a stall reaches and how long it takes to
// get there and back.
//
// Chapter 14.3 Section 9 measured a blast radius as a bitmap of
// affected pairs. This does the same for a fabric where the spread is
// certain rather than probable, and adds the two things that changes:
// a growth RATE, and a drain time.
// ---------------------------------------------------------------------
module congestion_tree
  import ibcmp_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,
  input  logic        tick_100ns,

  input  logic [15:0] radix,             // ports per switch
  input  logic [15:0] stages,
  input  logic [31:0] credit_loop_ns_i,
  input  logic        congestion_start,
  input  logic        congestion_clear,

  output logic [15:0] hops_reached,
  output logic [31:0] pairs_affected,
  output logic [31:0] full_stall_ns,
  output logic [31:0] drain_ns,
  output logic        tree_is_deterministic,
  output logic        fully_stalled,
  output logic [31:0] c_tree_events,
  output logic [31:0] c_stall_ns_total
);
  logic [31:0] elapsed_ns;
  logic        active;

  always_comb begin
    // Chapter 14.3 Section 8's counts: one switch's ingresses, then
    // their upstreams, then theirs.
    pairs_affected = (hops_reached == 16'd0) ? 32'(radix) - 32'd1
                   : ((hops_reached == 16'd1)
                      ? (32'(radix) - 32'd1) * (32'(radix) - 32'd1)
                      : (32'(radix) - 32'd1) * (32'(radix) - 32'd1)
                        * (32'(radix) - 32'd1));

    full_stall_ns = credit_loop_ns_i * 32'(stages);

    // The tree unwinds the same way it grew.
    drain_ns = full_stall_ns;

    // THE difference from Chapter 14.3. There, backpressure MIGHT
    // spread. Here the upstream is REQUIRED to stop.
    tree_is_deterministic = 1'b1;

    fully_stalled = active && (elapsed_ns >= full_stall_ns);
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      elapsed_ns <= '0; hops_reached <= '0; active <= 1'b0;
      c_tree_events <= '0; c_stall_ns_total <= '0;
    end else begin
      if (congestion_start && !active) begin
        active <= 1'b1; elapsed_ns <= '0; hops_reached <= '0;
        c_tree_events <= c_tree_events + 32'd1;
      end
      if (congestion_clear) begin
        active <= 1'b0;
      end
      if (active && tick_100ns) begin
        elapsed_ns       <= elapsed_ns + 32'd100;
        c_stall_ns_total <= c_stall_ns_total + 32'd100;
        if ((elapsed_ns + 32'd100) >= (credit_loop_ns_i
                                       * (32'(hops_reached) + 32'd1))
            && (hops_reached < stages))
          hops_reached <= hops_reached + 16'd1;
      end
    end
  end
endmodule

Classification: a spreading model whose distinguishing output is a hard-wired 1 where Chapter 14.3's equivalent would carry a probability.

What it teaches: that full_stall_ns is a small number on short links and that this is the uncomfortable result. At 3 m and five stages it is 2.55 µsso a lossless fabric reaches total stall faster than a lossy one reaches its first drop, because a drop requires a buffer to fill and a stall requires only a credit to be withheld.

And it teaches that drain_ns equals full_stall_ns, so the event's total duration is twice the fabric's depth in credit loops. At 2 km that is 200.8 µs of a fabric doing nothing, triggered by one oversubscribed egress port, with no frame lost and no counter anywhere reporting an error.

Deliberately simplified: pairs_affected uses Chapter 14.3 §8's uniform-radix expansion, which assumes every upstream port feeds the congested path and no port appears twice — a real topology's tree is smaller and its shape depends on the routing. The growth is modelled as one hop per full credit loop, where the true rate is one hop per credit exhaustion, which happens sooner on a busy link and later on an idle one. And congestion_clear stops the model immediately, where a real tree keeps growing for a loop after the cause is gone, because the stalls in flight have not yet been told.

Production implication: the counter worth having is c_stall_ns_total rather than c_tree_events, and the reason is that the two answer different questions. An event count says how often congestion occurred; a stalled-nanosecond total says what fraction of the fabric's time was spent not moving — and on a lossless fabric that is the only loss-equivalent quantity there is. A fabric reporting zero drops and 8% stalled time has lost 8% of its capacity, and an operator comparing its drop counter against an Ethernet fabric's will conclude it is performing better. The comparison is only meaningful if the stalled time is put beside the drop count, which requires the lossless fabric to keep a counter its whole design philosophy says it does not need.


10. Where RoCE Sits, and What It Inherits from Each Parent

RDMA over Converged Ethernet is not a compromise between the two fabrics. It is InfiniBand's transport placed on Ethernet's link layer, and it inherits a specific list from each.

State the layering precisely, because the loose version causes the misconfiguration this section ends with.

LayerInfiniBandRoCEv1RoCEv2
transportIB — queue pairs, verbsIB, unchangedIB, unchanged
networkIB — LIDs, subnet managernone — EtherType 0x8915IP
linkIB — creditsEthernetEthernet
routablewithin a subnetno — one L2 domainyes
lossless fromcredits, by constructionPFC, if configuredPFC, if configured

Row five is the one that matters and it is the source of every RoCE deployment problem.

What RoCE inherits from InfiniBand: a transport that was designed on the assumption that the link does not drop. The queue-pair state machine, the reliable-connected service and the completion semantics all come from a world where a packet in flight arrives.

What RoCE inherits from Ethernet: a link that does drop, unless somebody configured priority flow control on every switch on the path, on the right priority, with the right headroom — which is Chapter 14.4's mechanism and Chapter 14.2 §9's arithmetic, applied by hand across a fabric.

RoCE is a lossless transport on a link that is lossless only if somebody configured it to be, and the transport has no way to find out whether they did.

Price the mismatch. The InfiniBand transport's reliable-connected service recovers a loss by go-back-N: on detecting a gap, the receiver discards everything after it and the sender retransmits from the lost packet onward.

At 400 Gb/s, 1 µs round tripMTU 1 500MTU 9 000
window in flight50 000 B50 000 B
packets in the window33.35.6
retransmitted for one loss50 000 B50 000 B
amplification33.3×5.6×

And the throughput consequence.

Drop rateRetransmission overhead, MTU 1 500MTU 9 000
10⁻⁴0.33%0.06%
10⁻³3.33%0.56%
10⁻²33.3%5.56%

A drop rate of one in a thousand — which is unremarkable on a congested Ethernet fabric and which Chapter 21.6 would call mild — costs 3.33% of a 400 Gb/s link under go-back-N. At one in a hundred it costs a third of the link, and the fabric reports a 1% drop rate, which sounds fine.


11. RTL 5 — The RoCE Window Model

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// roce_window -- what a drop costs a transport that assumed it would
// not happen, and what the repair costs.
//
// The output that matters is link_losslessness_is_configured, which is
// an INPUT masquerading as a property: the transport cannot determine
// it and behaves as though it were true.
// ---------------------------------------------------------------------
module roce_window
  import ibcmp_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [15:0] rate_gbps,
  input  logic [15:0] rtt_ns,
  input  logic [15:0] mtu_octets,
  input  logic        selective_repeat,
  input  logic [15:0] queue_pairs,
  input  logic        pfc_configured_everywhere,   // nobody can verify this

  input  logic        packet_lost,

  output logic [31:0] window_bytes,
  output logic [15:0] window_packets,
  output logic [31:0] retransmit_bytes,
  output logic [15:0] amplification_x10,
  output logic [31:0] bitmap_bits,
  output logic [31:0] bitmap_bce,
  output logic        link_losslessness_is_configured,
  output logic        transport_assumes_lossless,
  output logic [31:0] c_retransmit_bytes,
  output logic [31:0] c_losses
);
  always_comb begin
    window_bytes   = (32'(rate_gbps) * 32'(rtt_ns)) / 32'd8;
    window_packets = (mtu_octets == 0) ? 16'd0
                   : 16'(window_bytes / 32'(mtu_octets));

    // Go-back-N discards the whole window; selective repeat does not.
    retransmit_bytes = selective_repeat ? 32'(mtu_octets) : window_bytes;

    amplification_x10 = (mtu_octets == 0) ? 16'd0
                      : 16'((retransmit_bytes * 32'd10) / 32'(mtu_octets));

    // One bit per packet in the window, per queue pair.
    bitmap_bits = selective_repeat
                ? ((32'(window_packets) + 32'd1) * 32'(queue_pairs))
                : 32'd0;
    bitmap_bce  = bitmap_bits;     // an array: 1 BCE per bit

    // THE honest signal. It is wired from configuration because no
    // measurement in the transport can establish it.
    link_losslessness_is_configured = pfc_configured_everywhere;

    // And this is always true, whatever the link actually does.
    transport_assumes_lossless = 1'b1;
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      c_retransmit_bytes <= '0; c_losses <= '0;
    end else if (packet_lost) begin
      c_losses           <= c_losses + 32'd1;
      c_retransmit_bytes <= c_retransmit_bytes + retransmit_bytes;
    end
  end
endmodule

Classification: a window model whose most important signal is an input the design cannot verify.

What it teaches: that transport_assumes_lossless is hard-wired high while link_losslessness_is_configured comes from a register somebody wrote, and that the gap between those two is where every RoCE deployment problem lives. The transport behaves identically whether PFC is configured or not; the only difference is the drop rate, and the only symptom is throughput.

And it teaches that amplification_x10 reports 333 at a 1 500-octet MTU and 56 at 9 000a six-fold change from one configuration field, which is a far larger lever than the framing efficiency that MTU discussions usually turn on.

Deliberately simplified: the window is the full bandwidth-delay product, where a real queue pair's window is also bounded by the receiver's advertised credit and by the number of outstanding work requests. retransmit_bytes charges a full window for every loss, where a burst of consecutive losses inside one window costs one retransmission rather than several. And bitmap_bce prices the bitmap alone, ignoring the out-of-order reassembly state that selective repeat also requires — which is the larger cost and is the same reorder buffer Chapter 24.1 §4 priced at 400 000 BCE.

Production implication: pfc_configured_everywhere is a lie in almost every deployment and it is a lie in a specific way. PFC is configured on the switches somebody remembered, and one switch on one path without it converts that path from lossless to lossy while every other path stays lossless — so the fabric's drop rate is zero on most flows and non-trivial on a few. The symptom is a small number of queue pairs running at a fraction of the rest, with no errors, no drops visible at the endpoints, and a fabric-wide drop rate that rounds to zero. The diagnostic is per-queue-pair retransmission bytes, which c_retransmit_bytes provides and which no fabric-level counter can substitute for, because the fabric's average is exactly the statistic that hides it.


12. Where a Lossless Fabric Actually Fails

Sections 2 to 11 priced the mechanism. This section names the failure it has that a lossy fabric structurally cannot, and the answer is deadlock.

The mechanism in four sentences. A packet holds buffer space at switch A and needs space at switch B. B's space is held by a packet that needs space at C. C's is held by a packet that needs space at A. Nothing can move, no credit can be returned, and no timeout in the protocol resolves it — because every participant is behaving exactly as specified.

A lossy fabricA credit fabric
a full bufferdrops, and the cycle breakswithholds credit, and the cycle holds
who resolves itthe buffer, automaticallynobody
the fabric's staterecoverspermanent

Row one is the whole reason a drop-based fabric cannot deadlock. A drop is an escape: the moment any participant discards a packet, the dependency it represented is gone. Chapter 21.6's silent, unreported, undiagnosable drop is also, structurally, the thing that keeps an Ethernet fabric live.

The drop that a whole chapter of this track was written to diagnose is the same drop that makes deadlock impossible. Removing it removes both.

The standard defence is a routing restriction rather than a hardware mechanism, and it is worth naming because it constrains the topology rather than the silicon.

DefenceWhat it doesWhat it costs
up-down routingforbids the turns that close a cyclepath diversity — some shortest paths become illegal
dimension-ordered routingorders the dimensions a packet may turn inthe same, on a mesh or torus
escape virtual lanesone lane is deadlock-free by constructiona full reservation for a lane carrying almost nothing
a timeout and discardbreaks the cycle by droppingthe fabric is no longer lossless

Row four is the honest one and it is what real fabrics do. A credit-based fabric that has been stalled for longer than any legitimate loop discards, and every deployed lossless fabric has such a mechanism — so "lossless" means "does not drop for congestion", not "does not drop."

The timeout must exceed the credit loop by a comfortable factor, and the loop depends on the cable.

LinkCredit loop10× the loop100×
3 m copper0.510 µs5.10 µs51.0 µs
100 m fibre1.460 µs14.60 µs146.0 µs
2 km fibre20.080 µs200.8 µs2 008 µs

A single timeout value across a fabric with mixed link lengths is either too short for the long links or 39× too long for the short ones, and the second is the worse error: a 2 ms stall on a 3 m link is four thousand credit loops of a fabric doing nothing before anything is discarded.


13. RTL 6 — The Deadlock Detector

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// deadlock_detector -- the mechanism that makes a lossless fabric
// survivable, and the admission that it is not lossless.
//
// The output that matters is is_still_lossless, which goes LOW the
// first time the detector fires. A fabric that has never fired it is
// lossless; one that has is a fabric with a long timeout.
// ---------------------------------------------------------------------
module deadlock_detector
  import ibcmp_pkg::*;
#(
  parameter int unsigned LANES = 8
)(
  input  logic        clk,
  input  logic        rst_n,
  input  logic        tick_us,

  input  logic [31:0] credit_loop_ns_i,
  input  logic [15:0] timeout_multiplier,   // x the credit loop
  input  logic  [7:0] lane_has_work,        // one bit per lane
  input  logic  [7:0] lane_has_credit,

  output logic [31:0] timeout_us,
  output logic [15:0] loops_before_discard,
  output logic  [7:0] lane_starved,
  output logic        discard_required,
  output logic  [2:0] discard_lane,
  output logic        is_still_lossless,
  output logic [31:0] c_discards,
  output logic [31:0] c_starved_us
);
  logic [31:0] starved_us [LANES];
  logic        fired;

  always_comb begin
    timeout_us = ((credit_loop_ns_i * 32'(timeout_multiplier)) + 32'd999)
               / 32'd1000;

    loops_before_discard = timeout_multiplier;

    // A lane with work and no credit is starved. It may be a normal
    // congestion stall or the head of a cycle, and nothing here can
    // tell the difference -- which is why the remedy is a timeout.
    lane_starved = lane_has_work & ~lane_has_credit;

    // THE admission.
    is_still_lossless = !fired;
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int i = 0; i < LANES; i++) starved_us[i] <= '0;
      discard_required <= 1'b0; discard_lane <= '0;
      fired <= 1'b0; c_discards <= '0; c_starved_us <= '0;
    end else begin
      discard_required <= 1'b0;
      if (tick_us) begin
        for (int i = 0; i < LANES; i++) begin
          if (lane_starved[i]) begin
            starved_us[i]  <= starved_us[i] + 32'd1;
            c_starved_us   <= c_starved_us + 32'd1;
            if ((starved_us[i] + 32'd1) >= timeout_us) begin
              discard_required <= 1'b1;
              discard_lane     <= 3'(i);
              starved_us[i]    <= '0;
              fired            <= 1'b1;
              c_discards       <= c_discards + 32'd1;
            end
          end else begin
            starved_us[i] <= '0;
          end
        end
      end
    end
  end
endmodule

Classification: a starvation timer per lane, whose firing is the protocol admitting it drops.

What it teaches: that is_still_lossless is a latched bit and that every deployed lossless fabric has one. The guarantee is not no packet is ever discarded; it is no packet is discarded for want of space, which is a narrower claim and a useful one. The fabric that has fired this detector has dropped a packet, and the transport above it must handle that — which is why RoCE's go-back-N exists and why Section 10's amplification arithmetic matters even on a fabric that claims not to drop.

And it teaches that lane_starved cannot distinguish a deadlock from ordinary congestion. A lane with work and no credit is exactly what Section 8's congestion tree produces, and exactly what a routing cycle produces, and the detector has no way to tell them apart. So the timeout must be long enough that no legitimate congestion stall reaches it — which is the 39.4× problem from the previous section, arriving as a parameter that cannot be right for every link at once.

Deliberately simplified: the detector is per lane at one port where a real deadlock is a property of a cycle spanning several switches, and no single port can see it. A fabric-wide detector needs either a central collector or a distributed probe, and both are Chapter 22.2 §20's class 104 territory: the guarantee is a relation between nodes and every local observation is consistent with both hypotheses. timeout_multiplier is a single value where a design with mixed link lengths needs a per-port one. And the loop over LANES is written for clarity rather than for synthesis.

Production implication: c_starved_us is the counter to keep and c_discards is the one that gets kept. A fabric reporting zero discards has told you nothing — it may be healthy, or it may be stalling for 90% of every second and never quite reaching the timeout. Starved microseconds is the lossless fabric's equivalent of a drop counter, and it is the only quantity that can be compared against an Ethernet fabric's loss rate on equal terms. A comparison of drop counters between a lossy and a lossless fabric is not a comparison at all, because one of them converted its losses into a quantity the other does not measure, and Section 20's rejected property is precisely that.


14. What a Lossless Fabric Must Never Do

Five prohibitions, each of which this chapter's structures make possible.

#NeverBecause
1advertise more credits than the buffer behind themthe promise is the mechanism; a broken promise is a lost packet on a fabric whose transport does not expect one
2let one virtual lane draw on another's reservationSection 7: the space was already promised, and lending it converts a guarantee into a probability
3report zero drops as evidence of healthSection 13: the loss became stalled time, and the drop counter cannot see it
4use one deadlock timeout across links of different lengthsSection 12: the credit loop varies 39.4× and the timeout must exceed it on every link
5assume RoCE's link is lossless because the transport requires itSection 11: pfc_configured_everywhere is a register somebody wrote, not a measurement

Row three is the one that produces a wrong decision rather than a bug, and it is worth stating at length.

An operator comparing a lossy and a lossless fabric reads both drop counters. The Ethernet fabric reports a drop rate; the lossless fabric reports zero. The comparison concludes the second is better and it has not measured the quantity the second one spends.

Lossy fabricLossless fabric
frames lost0.1%0
time stallednot measured, and smallnot measured, and possibly 8%
capacity delivered99.9% minus the retransmissions92%
what the counters sayworsebetter

The correct comparison is delivered capacity, and it requires the lossless fabric to keep c_starved_usa counter its design philosophy says it does not need, because nothing is being lost.


15. RTL 7 — Lossless Telemetry

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// ib_cmp_telemetry -- the counters a lossless fabric needs and does
// not usually have.
//
// The design rule: on a fabric that does not drop, the loss-equivalent
// quantity is TIME, so every counter here is a duration or a rate
// rather than an event count.
// ---------------------------------------------------------------------
module ib_cmp_telemetry
  import ibcmp_pkg::*;
#(
  parameter int unsigned LANES = 8
)(
  input  logic        clk,
  input  logic        rst_n,
  input  logic        clear,
  input  logic        tick_us,

  input  logic  [7:0] lane_starved_i,
  input  logic        credit_returned,
  input  logic        packet_sent,
  input  logic        discard_fired,
  input  logic        tree_active,
  input  logic [31:0] reserved_bytes_i,
  input  logic [31:0] occupied_bytes_i,

  output logic [47:0] c_starved_us_total,
  output logic [47:0] c_lane_starved_us [LANES],
  output logic [47:0] c_credits_returned,
  output logic [47:0] c_packets_sent,
  output logic [47:0] c_tree_us,
  output logic [47:0] c_discards_o,
  output logic [31:0] elapsed_us,
  output logic [15:0] stalled_pct,
  output logic [15:0] reservation_idle_pct,
  output logic        drop_counter_is_meaningful
);
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin
      c_starved_us_total <= '0; c_credits_returned <= '0;
      c_packets_sent <= '0; c_tree_us <= '0; c_discards_o <= '0;
      elapsed_us <= '0;
      for (int i = 0; i < LANES; i++) c_lane_starved_us[i] <= '0;
    end else begin
      if (credit_returned) c_credits_returned <= c_credits_returned + 48'd1;
      if (packet_sent)     c_packets_sent     <= c_packets_sent + 48'd1;
      if (discard_fired)   c_discards_o       <= c_discards_o + 48'd1;
      if (tick_us) begin
        elapsed_us <= elapsed_us + 32'd1;
        if (tree_active) c_tree_us <= c_tree_us + 48'd1;
        for (int i = 0; i < LANES; i++) begin
          if (lane_starved_i[i]) begin
            c_lane_starved_us[i] <= c_lane_starved_us[i] + 48'd1;
            c_starved_us_total   <= c_starved_us_total + 48'd1;
          end
        end
      end
    end
  end

  always_comb begin
    // THE number. On a lossless fabric this is the loss rate.
    stalled_pct = (elapsed_us == 0) ? 16'd0
                : 16'((c_tree_us * 48'd100) / 48'(elapsed_us));

    // Section 7's wasted promise, as a percentage.
    reservation_idle_pct = (reserved_bytes_i == 0) ? 16'd0
                         : 16'(((reserved_bytes_i - occupied_bytes_i)
                                * 32'd100) / reserved_bytes_i);

    // And the honest label on the counter everybody reads.
    drop_counter_is_meaningful = 1'b0;
  end
endmodule

Classification: a counter bank in which the primary quantity is a duration, because the fabric has converted its losses into one.

What it teaches: that drop_counter_is_meaningful is hard-wired zero and that this is the module's whole argument. A lossless fabric's drop counter reports zero by construction; it is a statement about the design rather than about the deployment, and reading it as health is Chapter 20.4 §20's class 91 in a new place — a metric whose value the mechanism was built to determine.

And it teaches that stalled_pct is the comparable quantity. A fabric at 8% stalled has delivered 92% of its capacity, which is the same outcome as a lossy fabric with an 8% effective loss rate — and only one of the two reports a number an operator would react to.

Deliberately simplified: reservation_idle_pct uses instantaneous occupancy where the useful statistic is a time average, and a high-percentile figure would be better still, because the reservation exists for the worst instant rather than the mean. The per-lane starvation counters do not distinguish congestion from a routing cycle, which Section 13 established no local observation can. And all counters are in one clock domain, where a real fabric's per-port counters are in the port's recovered clock — Chapter 18.1 §19's class 74 again.

Production implication: the counter that will be missing is c_lane_starved_us per lane, and its absence hides the failure Section 7's production note described. A fabric reporting an aggregate stalled time of 3% may have one lane stalled 24% of the time and seven idle, which is what a shared egress scheduler behind independent lanes produces — and it is indistinguishable from mild uniform congestion in the aggregate. Per-lane durations cost eight 48-bit counters, 384 flops, 7 680 BCE, 0.027 datapaths, and they are the difference between knowing that the lanes were bought for nothing and not.


16. RTL 8 — The Lossless Conformance Monitor

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// ib_cmp_conformance -- the checks that hold a lossless claim to what
// it actually promises.
//
// Every check here exists because "lossless" is a narrower claim than
// the word suggests, and the narrowing is what the monitor enforces.
// ---------------------------------------------------------------------
module ib_cmp_conformance
  import ibcmp_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [31:0] advertised_credit_bytes,
  input  logic [31:0] backing_buffer_bytes,
  input  logic [31:0] reserved_bytes_i,
  input  logic [31:0] required_bytes_i,
  input  logic        lane_borrowed,
  input  logic        discard_fired,
  input  logic        claims_lossless,
  input  logic [31:0] timeout_us_i,
  input  logic [31:0] credit_loop_ns_i,
  input  logic        packet_sent,
  input  logic        credits_available_i,

  output logic        v_overadvertised,
  output logic        v_under_reserved,
  output logic        v_lane_borrow,
  output logic        v_lossless_claim_stale,
  output logic        v_timeout_too_short,
  output logic        v_sent_without_credit,
  output logic  [5:0] violations,
  output logic        conformant
);
  always_comb begin
    // 1. Prohibition 1 -- the promise must be backed.
    v_overadvertised = (advertised_credit_bytes > backing_buffer_bytes);

    // 2. Section 2 -- the reservation is a floor, not a target.
    v_under_reserved = (reserved_bytes_i < required_bytes_i);

    // 3. Prohibition 2 -- a lane may not draw on a neighbour's promise.
    v_lane_borrow = lane_borrowed;

    // 4. Section 13 -- a fabric that has discarded is not lossless,
    //    and must stop saying it is.
    v_lossless_claim_stale = claims_lossless && discard_fired;

    // 5. Prohibition 4 -- the timeout must exceed the loop it is
    //    protecting, with margin.
    v_timeout_too_short = ((timeout_us_i * 32'd1000)
                           < (credit_loop_ns_i * 32'd10));

    // 6. The gate itself.
    v_sent_without_credit = packet_sent && !credits_available_i;

    violations = { v_sent_without_credit, v_timeout_too_short,
                   v_lossless_claim_stale, v_lane_borrow,
                   v_under_reserved, v_overadvertised };
    conformant = (violations == 6'b000000);
  end
endmodule

Classification: six checks, of which one audits a claim rather than a behaviour.

What it teaches: that v_lossless_claim_stale is a check on a marketing statement and belongs in silicon anyway. A fabric that has fired its deadlock discard has dropped a packet; if its status register still reports lossless operation, every consumer of that register is being misled, including the transport above it, which is making retransmission decisions on the assumption. One sticky bit fixes it.

And it teaches that v_under_reserved is the check that would have caught Section 12's cabling failure. A fabric provisioned for 30 m and cabled at 2 km has reserved_bytes_i at 38 700 and required_bytes_i at 1 004 000, a shortfall of 96.1% — and the resulting link runs at 3.9% of its rate with no other symptom. The check is one comparison at configuration time.

Deliberately simplified: required_bytes_i must be supplied, which means somebody must know the cable length — and on an optical link the design genuinely cannot measure it without a round-trip ranging mechanism, which Chapter 16.2 provides on Ethernet and which a fabric could borrow. v_lane_borrow is an input, so the monitor trusts the partition to report its own violation. And there is no check that the fabric's routing is deadlock-free, because that is a property of a topology and a routing function, not of a port — no monitor at a port can write it.

Production implication: the missing topology check is the important one and it is worth saying where it does belong. Deadlock freedom is established by the routing algorithm, at the subnet manager, before any traffic flows — up-down or dimension-ordered routing is a proof obligation discharged by construction, not a runtime check. So a fabric whose routing has been hand-modified for a link failure has lost the proof and nothing in the hardware knows. The one observable consequence is c_discards beginning to count, which is why a lossless fabric's discard counter going from zero to non-zero is a topology event rather than a traffic event, and the correct response is to re-examine the routing rather than the load.


17. The Two Fabrics, Priced Side by Side

Three questions that decide between a drop based and a credit based fabric. First, how long are the links: the reservation is two point three two per cent of a merchant switch at three metres and ninety one point four seven per cent at two kilometres, so below about thirty metres a credit fabric is affordable and above a few hundred it is not. Second, can the transport tolerate loss cheaply: go back N amplifies one loss by thirty three point three times at a fifteen hundred octet maximum transmission unit, while selective repeat costs thirty four bits per queue pair, which is one point one per cent of a network interface card. Third, is a stall cheaper than a drop for this workload: a stall reaches twelve thousand one hundred and sixty seven conversations in two hops and a drop reaches one, so a synchronised collective prefers drops and block storage does not mind stalls.Which fabric?three questionsLink length2.32% at 3 m, 91.47% at2 kmLoss tolerance33.3× vs 34 bits/QPStall vs drop12 167 vs 1Short reachcredits affordableSelective repeatremoves the reason tobuyA collectiveprefers dropsRoCEthe industry's bet12
Figure 4 — three questions decide the choice, and each has a number in this chapter attached to it.

Everything this chapter derived, at 64 ports and 400 Gb/s, in one unit.

MechanismEthernet, BCELossless, BCENote
reserved buffer, 100 m, 1 lane03.738 × 10⁷131.9 datapaths, 6.65% of a merchant switch
the same, 8 lanes02.990 × 10⁸53.2% of the switch
the same, 2 km, 1 lane05.140 × 10⁸91.47% — the switch is consumed
flow-control counters, 8 lanes, 64 ports≈ 4 × 10⁴ — PAUSE timers655 3602.31 datapaths
PFC headroom, 8 priorities, 100 m1.376 × 10⁸Ethernet's own version of the same cost
deadlock machinery0 — a drop breaks every cyclea timer per lane, plus a routing restrictionthe restriction costs path diversity
congestion blast radius, 2 hopsup to 12 167 pairs12 167 pairs, certainlythe same number, a different modality

Row five is the honest row and it is the one an Ethernet advocate should be made to read. 8 × 33 584 bytes across 64 ports is 1.376 × 10⁸ BCE — 24.5% of Chapter 23.3's switch — so an Ethernet fabric that actually enables PFC on eight priorities has paid 46.0% of what the credit fabric pays, for a weaker guarantee.

Lossless is not a property a fabric has or lacks. It is a bandwidth-delay product somebody bought, and both fabrics offer it at a price that scales with the cable.

And the four things each fabric gets in return.

Ethernet getsA credit fabric gets
1no reservation — the buffer is shared and over-committed 2 793% at idleno Chapter 21.6 — every congestion event is visible as a stall
2no deadlock, structurallyno retransmission in the common case
3arbitrary reacha simpler transport
4a drop with no reportera stall with no counter

Rows four are the symmetric admission and Section 14's callout is about them.


18. What the Comparison Assumes

Eight assumptions, each with the direction the chapter moves if it fails.

#AssumptionIf it is false
1the credit turnaround is 300 nsevery reservation moves by 400 Gb/s × Δ ÷ 8 bytes — 50 bytes per nanosecond
2a maximum frame is 9 000 octetsa 1 500-octet fabric saves 150 ns of loop — 7 500 bytes at 3 m, 29% of that row
3propagation is 4.90 ns/m in fibreChapter 8.2 §3's figure; every medium is within 5% and the 2 km row is unaffected in substance
4the switch is 64 portsthe fabric-scale figures scale linearly; the per-link ones do not move
5Chapter 23.3's switch is 5.62 × 10⁸ BCEthe share percentages scale inversely; the absolute reservations do not move
6eight virtual laneslinear — two lanes is a quarter of Section 6's figures
7a 1 µs round trip for RoCE's windowthe amplification is unchanged; the window and the bitmap scale together
8BCE appliesSection 19 examines it and it holds, with one caveat that is worth the space

Assumption 2 deserves a sentence because it is a lever a design actually has. A fabric that forbids jumbo frames shortens its credit loop by 150 ns at 400 Gb/s, which at 3 m is 29% of the whole loop and at 2 km is 0.7%. So the MTU is a meaningful buffer lever on short links and irrelevant on long ones — the opposite of its effect on Chapter 8.3's efficiency curve, where it matters everywhere equally.


19. The Cost, Accounted — in BCE

This chapter's blocks.

BlockFlopsBCE× the datapath
credit_loop_model326400.002
reservation_model326400.002
vl_partition, 8 lanes8 × 32 + 646 4000.023
congestion_tree1122 2400.008
roce_window641 2800.005
deadlock_detector, 8 lanes8 × 32 + 736 5800.023
ib_cmp_telemetry, 8 lanes8 × 48 + 25612 8000.045
ib_cmp_conformance0 — combinational00
this chapter's additions1 51330 5800.108

The whole instrumentation apparatus is 0.108 of a MAC datapath, against reservations measured in hundreds of datapaths — a ratio of about ten thousand to one. That is the argument for keeping every counter in this chapter: they are free, and without them the expensive part is unobservable.

And the designs the blocks describe.

BCE× the datapath% of Chapter 23.3's switch
reservation, 64 ports, 3 m, 1 lane1.306 × 10⁷46.12.32%
reservation, 64 ports, 100 m, 8 lanes2.990 × 10⁸1 055.453.20%
reservation, 64 ports, 2 km, 8 lanes4.112 × 10⁹14 515.0731.74%
Chapter 23.3's whole Ethernet switch5.62 × 10⁸1 985100%
Chapter 24.1 §4's PCIe return path, one link425 0881.5000.08%

Row three is a fabric that cannot exist on one die and row five is the same mechanism at 1/1322nd the scale, which is the honest way to see what this chapter is about: PCIe and InfiniBand both pay a bandwidth-delay product, and the difference is that PCIe's latency is a microsecond of silicon and InfiniBand's is a kilometre of glass.


20. Properties Worth Asserting, and One Worth Refusing

Fifty-one properties in six groups, and the last group exists only because one word in the fabric's datasheet means less than it appears to.

Group A — the credit loop (9).

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// A1. The loop is propagation, turnaround and one frame, and nothing else.
p_cl_composition: assert property (@(posedge clk) disable iff (!rst_n)
  (metres != 0) |-> (loop_ns ==
    (((2 * 32'(prop_ns_x100(int'(metres), media_e'(media)))) / 100)
     + 32'(CREDIT_TURNAROUND_NS) + 32'(frame_ns(int'(rate_gbps))))));

// A2. The reservation is the loop times the rate.
p_cl_reservation: assert property (@(posedge clk) disable iff (!rst_n)
  (reserved_bytes_o == (32'(rate_gbps) * loop_ns) / 32'd8));

// A3. It is a floor, always, and never a target.
p_cl_is_floor: assert property (@(posedge clk) disable iff (!rst_n)
  (reservation_is_a_floor == 1'b1));

// A4. A short buffer caps the rate rather than risking overflow.
p_cl_short_caps: assert property (@(posedge clk) disable iff (!rst_n)
  buffer_insufficient |-> (achievable_gbps < rate_gbps));

// A5. And an adequate buffer does not.
p_cl_adequate: assert property (@(posedge clk) disable iff (!rst_n)
  !buffer_insufficient |-> (achievable_gbps == rate_gbps));

// A6. Propagation dominates the loop on long links.
p_cl_prop_dominates: assert property (@(posedge clk) disable iff (!rst_n)
  (metres >= 16'd1000) |-> (prop_share_pct >= 32'd90));

// A7. And does not on short ones.
p_cl_prop_minor: assert property (@(posedge clk) disable iff (!rst_n)
  (metres <= 16'd10) |-> (prop_share_pct <= 32'd20));

// A8. An Ethernet link reserves nothing.
p_cl_ethernet_zero: assert property (@(posedge clk) disable iff (!rst_n)
  (ethernet_reservation_bce == 32'd0));

// A9. A shortfall is counted every cycle it persists.
p_cl_stall_counted: assert property (@(posedge clk) disable iff (!rst_n)
  buffer_insufficient |=> (c_stall_events == $past(c_stall_events) + 32'd1));

Group B — the fabric-scale reservation (8).

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// B1. The fabric figure is the per-link figure times the ports.
p_rs_scales: assert property (@(posedge clk) disable iff (!rst_n)
  (fabric_reservation_e2 == (per_link_bce / 32'd100) * 32'(ports)));

// B2. Over-commitment is never permitted.
p_rs_no_borrow: assert property (@(posedge clk) disable iff (!rst_n)
  (over_commitment_allowed == 1'b0));

// B3. A lossy fabric reserves nothing.
p_rs_lossy_zero: assert property (@(posedge clk) disable iff (!rst_n)
  (lossy_equivalent_bce == 32'd0));

// B4. Exceeding the switch is flagged rather than silently accepted.
p_rs_exceeds_flagged: assert property (@(posedge clk) disable iff (!rst_n)
  (share_of_switch_x100 >= 16'd10_000) |-> exceeds_switch);

// B5. And counted.
p_rs_infeasible_counted: assert property (@(posedge clk) disable iff (!rst_n)
  exceeds_switch |=> (c_infeasible_configs ==
                      $past(c_infeasible_configs) + 32'd1));

// B6. The ports a buffer serves falls as the reservation rises.
p_rs_monotone: assert property (@(posedge clk) disable iff (!rst_n)
  (per_link_bce > $past(per_link_bce)) |->
    (ports_a_buffer_serves <= $past(ports_a_buffer_serves)));

// B7. Zero ports is never reported for a nonzero buffer.
p_rs_nonzero: assert property (@(posedge clk) disable iff (!rst_n)
  ((switch_buffer_mb != 0) && (per_link_bce <= 32'd1_000_000)) |->
    (ports_a_buffer_serves != 32'd0));

// B8. The datapath figure follows the BCE figure.
p_rs_dp_follows: assert property (@(posedge clk) disable iff (!rst_n)
  (fabric_reservation_e2 == 32'd0) |-> (reservation_dp_milli == 32'd0));

Group C — virtual lanes (8).

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// C1. The total is the sum over lanes, never the maximum.
p_vl_sum_not_max: assert property (@(posedge clk) disable iff (!rst_n)
  (reservation_total_bytes == per_lane_reserved_bytes * 32'(LANES)));

// C2. Borrowing is forbidden, which is why C1 is a sum.
p_vl_no_borrow: assert property (@(posedge clk) disable iff (!rst_n)
  (borrow_permitted == 1'b0));

// C3. A lane never exceeds its own reservation.
p_vl_within_lane: assert property (@(posedge clk) disable iff (!rst_n)
  (occupancy[lane_sel] <= per_lane_reserved_bytes));

// C4. An arrival that would exceed it is blocked rather than accepted.
p_vl_block: assert property (@(posedge clk) disable iff (!rst_n)
  (packet_arrives && ((occupancy[lane_sel] + 32'(packet_bytes))
                      > per_lane_reserved_bytes)) |-> lane_blocked);

// C5. And is counted.
p_vl_block_counted: assert property (@(posedge clk) disable iff (!rst_n)
  (packet_arrives && lane_blocked) |=>
    (c_lane_blocks == $past(c_lane_blocks) + 32'd1));

// C6. Lanes do not head-of-line block one another -- the guarantee
//     they were bought for, stated so a scheduler cannot quietly break it.
p_vl_independent: assert property (@(posedge clk) disable iff (!rst_n)
  (head_of_line_across_lanes == 1'b0));

// C7. The reservation fits or the configuration is rejected.
p_vl_fits: assert property (@(posedge clk) disable iff (!rst_n)
  fits_in_buffer |-> (reservation_total_bytes <= total_buffer_bytes));

// C8. Occupancy never goes negative on a drain.
p_vl_no_underflow: assert property (@(posedge clk) disable iff (!rst_n)
  (packet_drains && (occupancy[lane_sel] < 32'(packet_bytes))) |=>
    (occupancy[lane_sel] == $past(occupancy[lane_sel])));

Group D — the congestion tree (8).

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// D1. The spread is certain, not probable.
p_ct_deterministic: assert property (@(posedge clk) disable iff (!rst_n)
  (tree_is_deterministic == 1'b1));

// D2. Full stall is the loop times the depth.
p_ct_stall_time: assert property (@(posedge clk) disable iff (!rst_n)
  (full_stall_ns == credit_loop_ns_i * 32'(stages)));

// D3. The drain takes as long as the growth.
p_ct_drain_symmetry: assert property (@(posedge clk) disable iff (!rst_n)
  (drain_ns == full_stall_ns));

// D4. Hops never exceed the fabric's depth.
p_ct_hops_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  (hops_reached <= stages));

// D5. Hops never go backwards while the tree is active.
p_ct_monotone: assert property (@(posedge clk) disable iff (!rst_n)
  (tree_is_deterministic && !congestion_clear) |=>
    (hops_reached >= $past(hops_reached)));

// D6. A new event is counted once, not per cycle.
p_ct_event_once: assert property (@(posedge clk) disable iff (!rst_n)
  (congestion_start && $past(congestion_start)) |=>
    (c_tree_events == $past(c_tree_events)));

// D7. Stalled time accrues only while active.
p_ct_time_accrues: assert property (@(posedge clk) disable iff (!rst_n)
  (!tree_is_deterministic || congestion_clear) |=>
    (c_stall_ns_total == $past(c_stall_ns_total)));

// D8. Full stall is reached only after the whole depth's worth of loops.
p_ct_full_after: assert property (@(posedge clk) disable iff (!rst_n)
  fully_stalled |-> (hops_reached == stages));

Group E — RoCE and the transport (9).

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// E1. The window is the bandwidth-delay product.
p_rc_window: assert property (@(posedge clk) disable iff (!rst_n)
  (window_bytes == (32'(rate_gbps) * 32'(rtt_ns)) / 32'd8));

// E2. Go-back-N retransmits the window.
p_rc_gbn: assert property (@(posedge clk) disable iff (!rst_n)
  !selective_repeat |-> (retransmit_bytes == window_bytes));

// E3. Selective repeat retransmits one packet.
p_rc_sr: assert property (@(posedge clk) disable iff (!rst_n)
  selective_repeat |-> (retransmit_bytes == 32'(mtu_octets)));

// E4. And costs a bit per packet in the window, per queue pair.
p_rc_bitmap: assert property (@(posedge clk) disable iff (!rst_n)
  selective_repeat |-> (bitmap_bits ==
    ((32'(window_packets) + 32'd1) * 32'(queue_pairs))));

// E5. Go-back-N costs no bitmap.
p_rc_no_bitmap: assert property (@(posedge clk) disable iff (!rst_n)
  !selective_repeat |-> (bitmap_bits == 32'd0));

// E6. A larger MTU reduces the amplification.
p_rc_mtu_helps: assert property (@(posedge clk) disable iff (!rst_n)
  (!selective_repeat && (mtu_octets > $past(mtu_octets))) |->
    (amplification_x10 <= $past(amplification_x10)));

// E7. The transport assumes lossless regardless of the link.
p_rc_assumes: assert property (@(posedge clk) disable iff (!rst_n)
  (transport_assumes_lossless == 1'b1));

// E8. And the link's actual state is a configuration input.
p_rc_config_input: assert property (@(posedge clk) disable iff (!rst_n)
  (link_losslessness_is_configured == pfc_configured_everywhere));

// E9. Retransmitted bytes accumulate at the amplified rate.
p_rc_accumulate: assert property (@(posedge clk) disable iff (!rst_n)
  packet_lost |=> (c_retransmit_bytes ==
                   $past(c_retransmit_bytes) + $past(retransmit_bytes)));

Group F — deadlock, telemetry and conformance (9).

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// F1. The timeout exceeds the credit loop with margin.
p_dl_timeout_margin: assert property (@(posedge clk) disable iff (!rst_n)
  ((timeout_us * 32'd1000) >= (credit_loop_ns_i * 32'd10)));

// F2. A starved lane is one with work and no credit.
p_dl_starved_def: assert property (@(posedge clk) disable iff (!rst_n)
  (lane_starved == (lane_has_work & ~lane_has_credit)));

// F3. A discard clears the lane's starvation timer.
p_dl_discard_clears: assert property (@(posedge clk) disable iff (!rst_n)
  discard_required |=> (starved_us[$past(discard_lane)] == 32'd0));

// F4. And permanently retires the lossless claim.
p_dl_claim_retired: assert property (@(posedge clk) disable iff (!rst_n)
  discard_required |=> !is_still_lossless);

// F5. The claim, once retired, never returns.
p_dl_claim_sticky: assert property (@(posedge clk) disable iff (!rst_n)
  !is_still_lossless |=> !is_still_lossless);

// F6. A fabric claiming lossless after a discard is a violation.
p_cf_stale_claim: assert property (@(posedge clk) disable iff (!rst_n)
  (claims_lossless && discard_fired) |-> v_lossless_claim_stale);

// F7. The drop counter is labelled as uninformative.
p_tl_drop_label: assert property (@(posedge clk) disable iff (!rst_n)
  (drop_counter_is_meaningful == 1'b0));

// F8. Stalled percentage is bounded by 100.
p_tl_pct_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  (stalled_pct <= 16'd100));

// F9. Conformance is the disjunction of its six checks.
p_cf_vector: assert property (@(posedge clk) disable iff (!rst_n)
  conformant |-> (violations == 6'b000000));

Coverage — states a lossless fabric only reaches under a specific load.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
c_cl_short_buffer:  cover property (@(posedge clk) buffer_insufficient);
c_cl_long_link:     cover property (@(posedge clk) metres >= 16'd1000);
c_rs_exceeds:       cover property (@(posedge clk) exceeds_switch);
c_vl_blocked:       cover property (@(posedge clk) lane_blocked);
c_vl_one_lane_hot:  cover property (@(posedge clk)
                      (occupancy[0] > (per_lane_reserved_bytes / 32'd2)) &&
                      (occupancy[1] == 32'd0));
c_ct_full_stall:    cover property (@(posedge clk) fully_stalled);
c_ct_two_hops:      cover property (@(posedge clk) hops_reached >= 16'd2);
c_rc_gbn_loss:      cover property (@(posedge clk) packet_lost && !selective_repeat);
c_rc_sr_loss:       cover property (@(posedge clk) packet_lost && selective_repeat);
c_dl_starved:       cover property (@(posedge clk) lane_starved != 8'd0);
c_dl_discard:       cover property (@(posedge clk) discard_required);
c_dl_not_lossless:  cover property (@(posedge clk) !is_still_lossless);
c_tl_stalled_high:  cover property (@(posedge clk) stalled_pct >= 16'd5);
c_cf_stale:         cover property (@(posedge clk) v_lossless_claim_stale);

21. Verification Scenarios

Fifty-eight scenarios in six groups, plus one directed test random stimulus will not produce.

Group 1 — the credit loop (10).

#ScenarioExpect
13 m copper, 400 Gb/sloop 510.3 ns; 25 515 B reserved
230 m fibre, 400 Gb/sloop 774 ns; 38 700 B
3100 m fibre, 400 Gb/sloop 1 460 ns; 73 000 B
42 km fibre, 400 Gb/sloop 20 080 ns; 1 004 000 B
5prop_share_pct at 3 m5.9% — implementation can help
6prop_share_pct at 2 km97.6% — nothing can
7a 32 kB buffer at 100 mbuffer_insufficient; 179 Gb/s — 44.9%
8a 73 kB buffer at 100 madequate; full rate
9provisioned for 30 m, cabled at 2 km38 700 B against 1 004 000 needed; 15.4 Gb/s — 3.9%
10ethernet_reservation_bce at any setting0

Group 2 — the fabric-scale reservation (10).

#ScenarioExpect
1164 ports, 3 m, 1 lane1.306 × 10⁷ BCE; 46.1 datapaths; 2.32%
1264 ports, 100 m, 1 lane3.738 × 10⁷; 131.9 datapaths; 6.65%
1364 ports, 2 km, 1 lane5.140 × 10⁸; 1 814.4 datapaths; 91.47%
1464 ports, 2 km, 8 lanesexceeds_switch; 731.74%
15a 64 MB buffer at 3 m2 630 ports served
16the same at 100 m919 ports
17the same at 2 km66 ports — the buffer is the reservation
1848 × 3 m plus 16 × 2 km1.383 × 10⁸ BCE; 24.6%; the 16 long links are 92.9%
19the same, budgeted from the mean lengtha quarter of the requirement; the uplinks miss line rate
20over_commitment_allowed at any setting0

Group 3 — virtual lanes (9).

#ScenarioExpect
218 lanes, 100 m584 000 B per port; 2.990 × 10⁸ BCE for 64; 53.20%
222 lanes, 100 m146 000 B per port; 13.30%
23one lane at capacity, seven empty7/8 of the reservation idle and unusable
24an arrival that would exceed a lane's sharelane_blocked; c_lane_blocks increments
25the same lane drains, then the arrival retriesaccepted
26a drain larger than the occupancyno underflow — p_vl_no_underflow
27borrow_permitted at any setting0
28the reservation exceeding the bufferfits_in_buffer low
29PFC, 8 priorities, 100 m, 64 ports1.376 × 10⁸ BCE — 46.0% of the credit scheme's

Group 4 — the congestion tree (10).

#ScenarioExpect
30congestion at 3 m, 5 stagesfull stall in 2.55 µs
31the same at 100 m7.30 µs
32the same at 2 km100.40 µs
33the drain after the cause clearsthe same duration again
34radix 24, 0 hops23 pairs — Chapter 14.3 §8's figure
35radix 24, 1 hop529 pairs
36radix 24, 2 hops12 167 pairs
37tree_is_deterministic at any setting1 — the distinction from Chapter 14.3
38congestion asserted for two consecutive cyclesone event counted, not two
398% of a second stalledstalled_pct reports 8; c_drops reports 0

Group 5 — RoCE (9).

#ScenarioExpect
40400 Gb/s, 1 µs, MTU 1 500, go-back-Nwindow 50 000 B; 33.3 packets; 33.3× amplification
41the same at MTU 9 0005.6 packets; 5.6× amplification
42selective repeat, MTU 1 5001.0×; a 34-bit bitmap per queue pair
4316 384 queue pairs with that bitmap557 056 BCE; 1.97 datapaths; 1.1% of a NIC
4410⁻³ drop rate, MTU 1 500, go-back-N3.33% of the link retransmitted
4510⁻² drop rate, MTU 1 500, go-back-N33.3% of the link
4610⁻³ drop rate, MTU 9 000, go-back-N0.56%
47pfc_configured_everywhere lowthe transport behaves identically; only throughput moves
48transport_assumes_lossless at any setting1

Group 6 — deadlock, telemetry, conformance (10).

#ScenarioExpect
49timeout at 10× the loop, 3 m5.10 µs
50timeout at 100× the loop, 2 km2 008 µs
51one timeout value across mixed lengthstoo short for 2 km or 39.4× too long for 3 m
52a lane starved past the timeoutdiscard_required; is_still_lossless goes low
53the claim after a discardstays low — p_dl_claim_sticky
54claims_lossless high with discard_firedv_lossless_claim_stale
55advertised credit above the backing bufferv_overadvertised
56a timeout below 10× the loopv_timeout_too_short
57drop_counter_is_meaningful at any setting0
58all six checks clearconformant high

22. Debugging a Lossless Fabric

Six symptoms, and five of them present as a number that is fine.

SymptomFirst questionWhere to look
a link at a fixed fraction of its rate, no errorsis buffer_insufficient set?Section 3 — the cable is longer than the buffer was provisioned for
zero drops and an unhappy applicationwhat is stalled_pct?Section 15 — the loss became time and the drop counter cannot see it
a fabric-wide slowdown from one hot porthow many hops has hops_reached reached?Section 9 — the tree is the mechanism working
lanes configured and behaving like one laneper-lane throughput under a single-lane congestionSection 21's directed test — the scheduler is shared
a discard counter that has just left zerohas the routing changed?Section 16 — a discard is a topology event, not a traffic one
a few RoCE queue pairs far slower than the restper-queue-pair c_retransmit_bytesSection 11 — one switch on one path without PFC

Row two is the one that is hardest to convince anybody of, because the fabric's own telemetry is reporting a perfect score.


23. Misconceptions

Misconception 1 — "lossless means no packet is ever dropped."

The wrong model: a credit-based fabric never discards, so a transport on it never needs to retransmit.

What it costs: a transport with no retransmission path, deployed on a fabric that discards on deadlock timeout. Section 13: every deployed lossless fabric has a starvation timer that discards, because without it a routing cycle is permanent. The guarantee is no drop for want of buffer, not no drop.

The corrected model: is_still_lossless is a latched bit and it can go low. A fabric that has fired its discard has dropped a packet and must stop claiming otherwise, which is v_lossless_claim_stale.

Misconception 2 — "the credit counters are what flow control costs."

The wrong model: credits are a handful of registers per port.

What it costs: a buffer budget wrong by two orders of magnitude. Section 4: the counters for 8 lanes across 64 ports are 655 360 BCE — 2.31 datapaths — and the reservation they account for at 100 m is 3.738 × 10⁷, a factor of 57.

The corrected model: the credit is a promise of buffer and the promise is the cost. Chapter 24.1 §7 made the same point about PCIe with a factor of 6.9; the factor here is larger because the latency is a cable rather than a chip.

Misconception 3 — "PFC makes Ethernet lossless, so the comparison is moot."

The wrong model: enable priority flow control and Ethernet becomes InfiniBand.

What it costs: a fabric that is lossless on the priority somebody configured, on the switches somebody remembered, and lossy everywhere else. Section 6: PFC's headroom is sized to a one-way dead time and a credit reservation to a round trip, so the credit scheme is about 2× on every link — and Section 11: pfc_configured_everywhere is a register somebody wrote, and the transport cannot verify it.

The corrected model: PFC gives Ethernet an optional, per-priority, emergency version of what credits do continuously and mandatorily. The guarantee is weaker and the cost is real: 8 priorities at 100 m is 1.376 × 10⁸ BCE, 24.5% of a merchant switch.

Misconception 4 — "a lossless fabric performs better because its drop counter is zero."

The wrong model: compare the two fabrics' loss rates and pick the lower one.

What it costs: choosing a fabric that delivers less capacity. Section 14: a fabric with a zero drop counter and 8% stalled time has delivered 92%, which is the same outcome as an 8% loss rate, and only one of the two reports a number an operator reacts to.

The corrected model: the comparable quantity is delivered capacity, and on a lossless fabric the loss-equivalent counter is c_starved_us. A comparison of drop counters is not a comparison, and Section 20's class 113 is the assertion form of the same mistake.

Misconception 5 — "long links are just more expensive."

The wrong model: the reservation scales with distance, so a 2 km fabric costs somewhat more than a 3 m one.

What it costs: a design that budgets linearly and discovers a cliff. Section 12: the ratio is 39.4× and it appears identically in four different quantities — buffer, share of the switch, stall time and timeout — and at 2 km with 8 lanes the reservation is 490 MB, which is not on-die memory at all.

The corrected model: past a few hundred metres the mechanism leaves the die and the unit changes from BCE to bytes and a bandwidth, which is Section 19's caveat and the reason every deployed lossless fabric is a rack, a row or a chassis.

Misconception 6 — "RoCE is InfiniBand on Ethernet, so it inherits InfiniBand's reliability."

The wrong model: the transport is the same, therefore the behaviour is.

What it costs: a 33.3× retransmission amplification on a link that drops. Section 10: RoCE inherits the IB transport, which was designed for a link that does not drop, and Ethernet's link, which does unless configured otherwise. At a 10⁻³ drop rate and a 1 500-octet MTU that is 3.33% of a 400 Gb/s link.

The corrected model: a protocol's efficiency decisions are made against an assumed error rate, and moving it to a different link changes the rate without changing the decisions. The repair is selective repeat — 34 bits per queue pair, 1.1% of a NIC — or a 9 000-octet MTU, which reduces the amplification six-fold for the cost of one configuration field.


24. Interview Questions

Six, with what a strong answer contains.

1. Why must a lossless link reserve buffer, and how much?

Because a credit is a promise that space exists, and the sender must be able to keep transmitting while credits are in flight. The reservation is the bandwidth-delay product of the credit return loop: two propagations, a turnaround, and one maximum frame. A strong answer puts numbers on it — 25 515 B at 3 m and 400 Gb/s, 1 004 000 B at 2 km — and observes that the ratio, 39.4×, is the chapter's only real parameter and it is chosen by whoever ran the cable.

2. What does credit-based flow control put back in exchange for the drop it removes?

Head-of-line blocking that is required rather than probable, and a congestion tree that grows one hop per credit loop. A strong answer cites Chapter 14.3 §8's blast-radius numbers — 23 pairs, 529 at one hop, 12 167 at two — and makes the sharp point: on a credit fabric those are certainties rather than bounds, because the upstream is not merely likely to stall, it is required to. At 3 m and five stages the whole fabric stalls in 2.55 µs.

3. Why can an Ethernet fabric not deadlock?

Because a drop breaks every cycle. A full buffer on Ethernet discards, and the moment any participant discards, the dependency it represented is gone. A credit fabric's full buffer withholds credit instead, so a cyclic dependency holds forever. The strong answer draws the uncomfortable conclusion: Chapter 21.6's silent, unreported, undiagnosable drop is the same drop that keeps the fabric live, and removing it removes both.

4. A 400 Gb/s lossless link runs at 3.9% of its rate with no errors. What do you check?

The cable length against the provisioned buffer. 3.9% × 400 Gb/s is 15.4 Gb/s, and 38 700 bytes ÷ 20.08 µs is exactly that — a link provisioned for 30 m and cabled at 2 km. A strong answer names the structural point: a short credit buffer does not risk overflow, it caps the rate permanently, which is Chapter 21.8 §5's ceiling with the credit loop as the denominator.

5. Where does RoCE sit, and what does it inherit from each parent?

InfiniBand's transport on Ethernet's link. From InfiniBand: queue pairs, verbs, and a reliable-connected service that assumes the link does not drop. From Ethernet: a link that does, unless PFC is configured on every switch on the path. The strong answer prices the mismatch — go-back-N amplifies one loss by 33.3× at a 1 500-octet MTU and 5.6× at 9 000 — and names the repair as 34 bits per queue pair, 1.1% of a NIC.

6. A lossless fabric reports zero drops. What have you learned?

That it is a lossless fabric. The counter is a statement about the design rather than about the deployment. The quantity that moved is stalled time, which most implementations do not count at all: a fabric at 8% stalled has delivered 92% of its capacity and reported a perfect score. A strong answer names Section 20's class 113 — a property whose outcome space is a partition the mechanism invalidated — and gives the repair: assert a residence-time bound with three branches, and a stall budget that can fail.


25. Questions and Answers


26. What's Next

Chapter 24.1 set Ethernet against a fabric that answers a different question, and this chapter against one that answers the same question differently. Chapter 24.3 asks what a design gets by answering it outside a standard altogether.

The question is one every chapter of Module 23 answered implicitly by staying inside one. A proprietary fabric is free of Ethernet's preamble, its interframe gap, its frame check sequence and its forty-eight-bit addresses — and Chapter 24.3 begins by deriving what those actually cost as a fraction of the wire, at both ends of Chapter 8.3 §2's curve.

The answer at the small end is 45.24% and at the large end 0.42%, and the chapter's first punch is that the fabrics which actually leave the standard are the ones carrying large transfers. So they leave for four tenths of a per cent — which means the win, if there is one, is somewhere else entirely.

It is, and the chapter finds it in the lookup rather than the framing: Chapter 23.3 §2's 36-stage pipeline is 30 ns per hop, and a source-routed fabric that carries its route in the header decodes in three stages. Then it prices what leaving the standard obliges a vendor to rebuild — a management protocol, a time protocol, a congestion protocol, test equipment, a second source and an entire verification apparatus with no reference implementation to check against — and closes with Chapter 23.6 §19's argument: the win is real, and it is not in the unit people quote it in.

Continue learning

Standards & specifications

Governing standard
IEEE Std 802.3 (Ethernet)(opens IEEE in a new tab)

Defines the Ethernet MAC, the media-independent interfaces and the physical-layer sublayers, including framing, access control, auto-negotiation and per-rate PHY specifications. VLAN tagging, priority and time-sensitive shaping are defined by IEEE 802.1, not by 802.3.

This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.

Where this fits

Part of the Ethernet curriculum.