Priority Flow Control and Lossless Ethernet

Chapter 14.2 §6 established that PAUSE is all or nothing: it stops a link, and a link carries every conversation crossing it. §15 measured the collateral at 88%. Chapter 14.3 followed that collateral across a topology and found the blocking spreading to conversations that never touched a congested port.

Both chapters ended in the same place, and it was a counting problem. A link is one resource, so stopping it is one decision, and one decision cannot distinguish traffic that deserves stopping from traffic that does not.

Chapter 13.2 §3 put three bits in the tag. Eight values. This chapter spends them.

Priority Flow Control — 802.1Qbb — replaces the single PAUSE with a vector of eight, one per priority. A link stops being one stoppable resource and becomes eight. And that is the only structural move available, because the frame already carries the field, the switch already maps it to a queue, and nothing else in the frame distinguishes one conversation's urgency from another's.

The move works. It also costs eight headrooms rather than one — at 100 Gb/s that is 11.98% of Chapter 14.1's entire 12 MiB buffer pool consumed before a single frame is queued — and it admits a failure mode this track has never met.

Ethernet has never deadlocked. It cannot: a switch that runs out of buffer discards, the backlog moves, and the system makes progress by losing something. A lossless class removes that. And a mechanism that never discards, arranged in a cycle, is a mechanism that can stop permanently with no fault, no error and no counter moving.

1. Scope — What This Chapter Owns

This chapter owns the subdivision: the PFC frame and its class-enable vector, the per-class watermarks and timers, the per-class headroom arithmetic and why the total is not what it first looks like, the per-class transmit gate, the lossless class, and the deadlock that losslessness admits.

It does not own the tag. Chapter 13.2 §3 decoded PCP and handed it on; this chapter is the consumer that field was waiting for.

It does not own the queues. Chapter 13.4 §9 mapped PCP into egress queues and §11 scheduled among them. PFC needs those queues to exist and does not build them.

It does not own the buffer pool. Chapter 14.1 §5 built the shared pool with its reserve, shared and limit regions. PFC's headroom is a new claim on that pool and this chapter prices it.

It does not own the dead time. Chapter 14.2 §5 derived it — 13.32 µs at 1 Gb/s, dominated by a neighbour finishing a frame it had already started. Every number in Section 8 rests on that derivation and none of it changes.

And it does not own head-of-line blocking. Chapter 14.3 §4 derived the 2 − √2 bound and §11 removed it with virtual output queues. Section 11 here shows that PFC moves that blocking from the port to the class and does not remove any of it — which is the single most over-claimed thing about the mechanism.

2. What the Three Bits Buy

The whole of PFC is one substitution. Read the two frames side by side and everything else in the chapter is a consequence.

The last row is the one to notice. The PFC frame carries five times the information and costs exactly the same on the wire, because Chapter 5.6's 46-octet minimum payload was already padding 42 octets of nothing in the PAUSE frame. PFC spends 16 of those padding octets on eight timers and gets them free.

Which is worth stating plainly because it removes an argument that never applies. There is no bandwidth case against PFC. The control frame is the same size, arrives at the same rate, and consumes the same 672 ns at 1 Gb/s. Everything PFC costs, it costs in buffer and in reasoning — never in link capacity.

And the eight bits are not eight independent PAUSEs. They are one frame that carries eight decisions, which means:

Every PFC frame restates all eight classes. A device pausing class 3 and leaving class 5 running does not send "pause 3" — it sends a vector with bit 3 set, bit 5 clear, and a timer for every enabled bit. The frame is a complete statement of the sender's current wishes, not a delta.

So a lost PFC frame loses eight decisions at once, and the next frame restores all eight. This is better than a delta protocol and worth understanding as a deliberate choice — Chapter 14.2 §19's callout observed that Ethernet has no return path and no acknowledgement, so a mechanism whose messages are idempotent complete states recovers from a loss on its next transmission, while one whose messages are increments never recovers at all.

3. RTL 1 — Building a PFC Frame

The builder is Chapter 14.2 §3's with one field replaced and one array added. It is presented in full because the differences are where the mechanism lives.

// -----------------------------------------------------------------------
// pfc_pkg -- shared types for 802.1Qbb priority flow control.
// -----------------------------------------------------------------------
package pfc_pkg;

  localparam int NUM_CLASSES = 8;
  localparam int CELL_OCTETS = 128;   // 14.1's buffer cell
  localparam int Q_DEPTH     = 4096;  // cells per port, 14.1 section 5
  localparam int OCC_W       = 13;    // ceil(log2(4096)) + 1

  // The 802.3 MAC control opcodes this track has met.
  localparam logic [15:0] OPCODE_PAUSE = 16'h0001;   // 14.2
  localparam logic [15:0] OPCODE_PFC   = 16'h0101;   // this chapter

  // Reserved multicast destination, identical for both -- 14.2 section 2.
  localparam logic [47:0] CTRL_DA = 48'h01_80_C2_00_00_01;
  localparam logic [15:0] CTRL_ET = 16'h8808;        // MAC Control

  // One quantum is 512 bit times at the port's rate -- 14.2 section 2.
  localparam int QUANTUM_BITS = 512;

  typedef logic [NUM_CLASSES-1:0]  class_vec_t;
  typedef logic [15:0]             quanta_t;
  typedef quanta_t [NUM_CLASSES-1:0] quanta_vec_t;
  typedef logic [OCC_W-1:0]        occ_t;

  // A class is either lossy (may discard, 14.1's behaviour) or lossless
  // (must not discard, and therefore must be paused before it overflows).
  typedef enum logic [0:0] { CLASS_LOSSY = 1'b0, CLASS_LOSSLESS = 1'b1 } class_kind_e;

endpackage

// -----------------------------------------------------------------------
// pfc_frame_builder -- emits one 802.1Qbb PFC frame from a request.
//
// The frame is a complete statement of the sender's wishes for all eight
// classes. A class whose enable bit is clear is being RELEASED, not
// left alone -- see section 2.
// -----------------------------------------------------------------------
module pfc_frame_builder
  import pfc_pkg::*;
(
  input  logic              clk,
  input  logic              rst_n,

  // Request: the full desired state of all eight classes.
  input  logic              req_valid,
  input  class_vec_t        req_enable,     // 1 = pause this class
  input  quanta_vec_t       req_quanta,     // per class, meaningful if enabled
  input  logic [47:0]       local_sa,
  output logic              req_ready,

  // Octet-serial frame output to the transmit path.
  output logic              tx_valid,
  output logic [7:0]        tx_octet,
  output logic              tx_sop,
  output logic              tx_eop
);

  // 6 DA + 6 SA + 2 ET + 2 opcode + 2 vector + 16 timers + 26 pad = 60
  // octets; the FCS appended downstream brings the frame to 64.
  localparam int BODY_OCTETS = 60;
  localparam int PAD_OCTETS  = 26;

  logic [7:0] body [0:BODY_OCTETS-1];
  logic [5:0] idx;
  logic       busy;

  // ---------------------------------------------------------------------
  // Assemble the body combinationally from the request. The vector is
  // padded to 16 bits with the high half reserved and transmitted zero.
  // ---------------------------------------------------------------------
  always_comb begin
    int p;
    for (p = 0; p < BODY_OCTETS; p++) body[p] = 8'h00;

    for (p = 0; p < 6; p++) body[p]     = CTRL_DA[47 - 8*p -: 8];
    for (p = 0; p < 6; p++) body[6 + p] = local_sa[47 - 8*p -: 8];

    body[12] = CTRL_ET[15:8];
    body[13] = CTRL_ET[7:0];
    body[14] = OPCODE_PFC[15:8];
    body[15] = OPCODE_PFC[7:0];

    // Class-enable vector: 8 reserved bits then the 8 class bits.
    body[16] = 8'h00;
    body[17] = req_enable;

    // Eight 16-bit timers, class 0 first. A disabled class transmits
    // zero quanta -- a receiver that honours it releases immediately,
    // which is the same outcome as the enable bit being clear.
    for (p = 0; p < NUM_CLASSES; p++) begin
      body[18 + 2*p]     = req_enable[p] ? req_quanta[p][15:8] : 8'h00;
      body[18 + 2*p + 1] = req_enable[p] ? req_quanta[p][7:0]  : 8'h00;
    end
    // body[34 .. 59] remain zero: the 26 pad octets.
  end

  // ---------------------------------------------------------------------
  // Serialise.
  // ---------------------------------------------------------------------
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      idx  <= '0;
      busy <= 1'b0;
    end else if (!busy) begin
      if (req_valid) begin
        busy <= 1'b1;
        idx  <= '0;
      end
    end else begin
      if (idx == BODY_OCTETS-1) begin
        busy <= 1'b0;
        idx  <= '0;
      end else begin
        idx <= idx + 1'b1;
      end
    end
  end

  assign req_ready = !busy;
  assign tx_valid  = busy;
  assign tx_octet  = body[idx];
  assign tx_sop    = busy && (idx == 6'd0);
  assign tx_eop    = busy && (idx == BODY_OCTETS-1);

endmodule

Classification: a transmit-side protocol encoder. Deterministic, no back-pressure of its own, one frame per request.

What it teaches: that the enable vector and the timer array carry redundant information, and the redundancy is deliberate. A class can be released either by clearing its enable bit or by sending it zero quanta, and a conformant receiver does the same thing in both cases. Two encodings of one intent look like a specification defect and are not — the vector lets a receiver decide which timers to even look at, which matters because Section 8's timer bank is eight counters and reloading all eight on every frame is eight writes where the vector allows one.

And it teaches the padding point from Section 2 concretely. The body is 60 octets of which 26 are pad. Chapter 14.2's PAUSE body was 60 octets of which 42 were pad. The mechanism grew by 16 octets and the frame did not grow at all.

Deliberately simplified: the request is a single-cycle handshake with no queue behind it, and a new request arriving while busy is high is simply not accepted. Production designs need at least one pending slot, because Section 6's watermark logic can cross two thresholds in consecutive cycles and the second decision must not be lost — a dropped release is Chapter 14.2 §19's stranded link, now on one class instead of the whole port.

Production implication: local_sa is an input rather than a parameter because the source address of a control frame is the port's own address, and on a switch every port has a different one. A design that ties this to a chip-level constant emits eight ports' worth of control frames all claiming the same source, which Chapter 12.2's learning on the neighbour will faithfully record — installing one MAC address against eight different ports in turn, and producing exactly Chapter 12.2 §9's flapping entry from a mechanism that has nothing to do with the data path.

4. RTL 2 — Parsing the Vector

The receive side has one job the PAUSE parser did not: deciding which of eight timers a frame touches.

// -----------------------------------------------------------------------
// pfc_vector_parser -- recognises a PFC frame and extracts the eight
// (enable, quanta) pairs.
//
// Gate order matters and is the same as 14.2 section 4: cheapest and most
// selective checks first, so a data frame is rejected in the first six
// octets rather than after sixty.
// -----------------------------------------------------------------------
module pfc_vector_parser
  import pfc_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        rx_valid,
  input  logic [7:0]  rx_octet,
  input  logic        rx_sop,
  input  logic        rx_eop,
  input  logic        rx_fcs_ok,

  output logic        pfc_valid,      // one pulse, after FCS
  output class_vec_t  pfc_enable,
  output quanta_vec_t pfc_quanta,

  output logic [31:0] c_seen,         // opcode matched and FCS good
  output logic [31:0] c_bad_fcs,      // opcode matched and FCS bad
  output logic [31:0] c_wrong_da      // opcode matched, DA did not
);

  logic [5:0]   idx;
  logic         da_ok, op_ok, in_frame;
  logic [15:0]  et, opc;
  class_vec_t   enable_q;
  quanta_vec_t  quanta_q;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      idx <= '0; da_ok <= 1'b0; op_ok <= 1'b0; in_frame <= 1'b0;
      et <= '0; opc <= '0; enable_q <= '0; quanta_q <= '0;
      c_seen <= '0; c_bad_fcs <= '0; c_wrong_da <= '0;
      pfc_valid <= 1'b0; pfc_enable <= '0; pfc_quanta <= '0;
    end else begin
      pfc_valid <= 1'b0;

      if (rx_valid && rx_sop) begin
        idx <= 6'd1; in_frame <= 1'b1; da_ok <= 1'b1; op_ok <= 1'b0;
        if (rx_octet != CTRL_DA[47:40]) da_ok <= 1'b0;
      end else if (rx_valid && in_frame) begin
        idx <= idx + 1'b1;

        // Gate 1 -- destination address, octets 1..5.
        if (idx <= 6'd5) begin
          if (rx_octet != CTRL_DA[47 - 8*idx -: 8]) da_ok <= 1'b0;
        end

        // Gate 2 -- EtherType, octets 12..13.
        if (idx == 6'd12) et[15:8] <= rx_octet;
        if (idx == 6'd13) et[7:0]  <= rx_octet;

        // Gate 3 -- opcode, octets 14..15.
        if (idx == 6'd14) opc[15:8] <= rx_octet;
        if (idx == 6'd15) begin
          opc[7:0] <= rx_octet;
          op_ok    <= (et == CTRL_ET) &&
                      ({opc[15:8], rx_octet} == OPCODE_PFC);
        end

        // Octet 16 is reserved. Octet 17 is the class-enable vector.
        if (idx == 6'd17) enable_q <= rx_octet;

        // Octets 18..33 are the eight timers, class 0 first.
        if (idx >= 6'd18 && idx <= 6'd33) begin
          automatic int cls = (idx - 18) >> 1;
          if (idx[0] == 1'b0) quanta_q[cls][15:8] <= rx_octet;
          else                quanta_q[cls][7:0]  <= rx_octet;
        end
      end

      if (rx_valid && rx_eop) begin
        in_frame <= 1'b0;
        if (op_ok) begin
          if      (!da_ok)     c_wrong_da <= c_wrong_da + 1;
          else if (!rx_fcs_ok) c_bad_fcs  <= c_bad_fcs  + 1;
          else begin
            c_seen     <= c_seen + 1;
            pfc_valid  <= 1'b1;
            pfc_enable <= enable_q;
            pfc_quanta <= quanta_q;
          end
        end
      end
    end
  end

endmodule

Classification: a receive-side protocol decoder with a validity gate. It is a filter first and a parser second.

What it teaches: that the frame must be complete and correct before any of its eight decisions is acted on. The parser holds enable_q and quanta_q in shadow registers and commits them in one pulse at end-of-frame, after rx_fcs_ok. A design that applied each timer as it arrived would act on a corrupted vector, and the corruption's effect is not symmetric — a flipped bit that sets an enable stops a class that should run, while a flipped bit that clears one releases a lossless class whose buffer is about to overflow. The second is a frame loss on a class the operator was told could not lose frames.

And it teaches why c_wrong_da is separated from c_bad_fcs. Both are frames that matched the opcode and were not honoured, and they mean completely different things: a bad FCS is a physical-layer problem — Chapter 6.3's residue check, a cable or an SFP — while a wrong destination address is a device generating malformed control frames, which is a firmware problem on the neighbour. One counter conflating them sends an engineer to the wrong layer.

Deliberately simplified: the parser assumes the PFC frame is untagged. A PFC frame may carry an 802.1Q tag, and Chapter 13.2 §7 established what that does to every offset after octet 12 — the opcode moves from 14 to 18, the vector from 17 to 21, and the timers from 18 to 22. A parser with hard-coded offsets silently fails to recognise a tagged PFC frame, and the symptom is a neighbour that never seems to honour pause, which is Section 22's second misconception exactly.

Production implication: c_seen is the counter that answers the first question in any PFC investigation — is the neighbour sending PFC at all — and it is the one most often missing. Without it, a link that drops frames on a lossless class has two indistinguishable explanations: the neighbour never asserted pause, or it asserted pause and we ignored it. Those are faults on opposite ends of the cable, and one counter separates them.

5. Eight Reserves, One Wire

The headroom question has an answer that is wrong in an interesting way, and getting it right is the difference between a mechanism that works and a buffer budget that does not fit.

Chapter 14.2 §5 derived the requirement. Between deciding to pause and traffic stopping, the neighbour keeps sending, and the queue must hold everything that arrives in that window. At 1 Gb/s over 100 m the window is 13.32 µs and the reserve is 1.63 KiB.

Now there are eight classes. Two answers present themselves and both are wrong.

The first wrong answer: each class needs one eighth. The reasoning is that the wire carries 1.63 KiB during the dead time no matter what, so dividing it eight ways covers the total. It does not, because nothing forces the traffic to spread across classes. A dead window in which every arriving octet belongs to class 3 is entirely ordinary — one storage array sending at line rate is exactly that — and class 3's 209-octet share overflows on the second frame.

So each class must reserve the full window: 1.63 KiB, all eight of them.

The second wrong answer: the total is therefore eight times 1.63 KiB, which the wire could never deliver, so seven eighths of it is waste that a cleverer design would reclaim. The premise is right and the conclusion is wrong.

The reserve is 8× and the occupancy is never more than 1×, and both statements are correct at the same time, because the eight classes are not paused at the same moment. Class 3 crosses its watermark now and enters its dead window; class 5 crosses its watermark 400 µs later and enters a different one. The octets that fill class 3's reserve and the octets that fill class 5's reserve are different octets arriving at different times. Nothing about the wire's serial nature prevents each class, in its own window, from receiving a full window's worth.

Which means the reserve cannot be shared and cannot be reclaimed. Chapter 14.1 §5's shared pool works precisely because a port that is not congested is not holding memory; a PFC headroom is held by a class that is behaving perfectly, against a window that has not started yet, and releasing it is releasing the guarantee.

87.5% of a PFC deployment's headroom is, at every instant, memory reserved against a future that will not arrive at this instant. That is not waste. It is what a guarantee costs when eight of them are independent.

6. RTL 3 — Per-Class Watermarks and Headroom

The watermark logic is Chapter 14.2 §7's, instantiated eight times, with the headroom computed per class at elaboration and one new output the single-class version had no reason to produce.

// -----------------------------------------------------------------------
// pfc_class_watermarks -- per-class occupancy tracking, watermark
// crossing and the assert/release decision for all eight classes.
//
// Headroom is derived from the link's dead time, not chosen as a fraction
// of the queue -- 14.2 section 9's argument, now eight times over.
// -----------------------------------------------------------------------
module pfc_class_watermarks
  import pfc_pkg::*;
#(
  parameter int DEAD_NS       = 13316,  // 14.2 section 5, 1 Gb/s over 100 m
  parameter int RATE_MBPS     = 1000,
  parameter int CLASS_CELLS   = 512,    // cells budgeted to each class
  parameter int HYSTERESIS    = 64      // cells, section 7's duration
)(
  input  logic       clk,
  input  logic       rst_n,

  input  logic       enq_valid,
  input  logic [2:0] enq_class,
  input  logic       deq_valid,
  input  logic [2:0] deq_class,

  input  class_vec_t lossless_mask,     // which classes must not discard

  output class_vec_t assert_pause,      // level: this class wants stopping
  output logic       change,            // pulse when assert_pause changes
  output occ_t       occ [NUM_CLASSES],

  output logic       cfg_infeasible,    // headroom does not fit the budget
  output logic [15:0] headroom_cells
);

  // Headroom in octets = dead time x rate / 8, then in 14.1 cells.
  // 13316 ns x 1000 Mb/s / 8 = 1664.5 octets -> 13 cells at 128 octets.
  localparam int HEADROOM_OCTETS = (DEAD_NS * RATE_MBPS) / 8000;
  localparam int HEADROOM_CELLS  = (HEADROOM_OCTETS + CELL_OCTETS - 1)
                                   / CELL_OCTETS;

  localparam int HIGH_WM = CLASS_CELLS - HEADROOM_CELLS;
  localparam int LOW_WM  = HIGH_WM - HYSTERESIS;

  // A configuration in which the headroom does not fit inside the class's
  // budget cannot implement the mechanism at all -- 14.2 section 9's
  // watermark_infeasible, now per class and eight times more likely.
  localparam bit INFEASIBLE = (HIGH_WM <= 0) || (LOW_WM <= 0);

  assign cfg_infeasible = INFEASIBLE;
  assign headroom_cells = HEADROOM_CELLS[15:0];

  occ_t       occ_q [NUM_CLASSES];
  class_vec_t asrt_q, asrt_d;

  always_comb begin
    int c;
    asrt_d = asrt_q;
    for (c = 0; c < NUM_CLASSES; c++) begin
      // A lossy class never asserts: 14.1's discard is its back pressure.
      if (!lossless_mask[c]) begin
        asrt_d[c] = 1'b0;
      end else if (occ_q[c] >= occ_t'(HIGH_WM)) begin
        asrt_d[c] = 1'b1;
      end else if (occ_q[c] <= occ_t'(LOW_WM)) begin
        asrt_d[c] = 1'b0;
      end
    end
  end

  always_ff @(posedge clk or negedge rst_n) begin
    int c;
    if (!rst_n) begin
      for (c = 0; c < NUM_CLASSES; c++) occ_q[c] <= '0;
      asrt_q <= '0;
      change <= 1'b0;
    end else begin
      if (enq_valid) occ_q[enq_class] <= occ_q[enq_class] + 1'b1;
      if (deq_valid) occ_q[deq_class] <= occ_q[deq_class] - 1'b1;

      change <= (asrt_d != asrt_q);
      asrt_q <= asrt_d;
    end
  end

  assign assert_pause = asrt_q;
  always_comb begin
    int c;
    for (c = 0; c < NUM_CLASSES; c++) occ[c] = occ_q[c];
  end

endmodule

Classification: a threshold detector with hysteresis, replicated eight times, plus an elaboration-time feasibility check.

What it teaches: that lossless_mask gates the assertion, and a lossy class asserting pause is the most common PFC misconfiguration there is. A lossy class already has a back-pressure mechanism — Chapter 14.1's discard — and adding PFC on top does not make it more reliable. It makes it slower and gives it a deadlock risk it did not have. The mask exists so that enabling PFC on a port does not enable it on all eight classes, which is what a design without the mask forces.

And it teaches that cfg_infeasible is a property of the configuration, evaluated at elaboration, sitting alongside runtime state. Chapter 14.1 §17's callout made this argument for reserves_fit and Chapter 13.4 §14 for injective: a headroom that does not fit is wrong from power-on, not from the first drop, and a conformance bit that only reports observed violations says nothing has gone wrong yet when the honest statement is that this configuration cannot be correct.

Deliberately simplified: CLASS_CELLS is a fixed per-class budget, so eight classes on a 4096-cell queue take 512 each whether they need it or not. Production designs allocate dynamically from Chapter 14.1 §5's shared pool — but the headroom portion must remain static and per class, because Section 5 established that a headroom held against a window that has not started cannot be lent out. So a production design has a dynamic body and a static reserve on the same queue, which is a harder invariant than either alone.

Production implication: HEADROOM_CELLS is computed from DEAD_NS and RATE_MBPS rather than configured, and both are per port on a switch with mixed link rates. A 24-port switch with sixteen 1 Gb/s and eight 25 Gb/s ports needs two different headrooms, and Section 7's table shows they differ by a factor of 1.9. A design that parameterises the module once and instantiates it 24 times has given the fast ports the slow ports' headroom — which is 1664 octets against a requirement of 3165, and every pause on those eight ports is issued too late.

7. The Headroom Arithmetic, at Four Line Rates

Section 5 established that each class reserves a full dead window. Chapter 14.2 §5 derived the window. Multiplying gives the number a PFC deployment actually has to fit, and one row of it does not fit at all.

The last row is not a tight configuration. It is an impossible one. Eight lossless classes on a 100 Gb/s link across 2 km require nearly twice the entire port's buffer for headroom alone, before a single frame is queued. cfg_infeasible fires and the mechanism cannot be deployed — not tuned, not compromised, not deployed.

And the same arithmetic against Chapter 14.1 §5's whole 12 MiB shared pool, for all 24 ports:

Read the fourth row against Chapter 14.2's single-class equivalent, which is one eighth of it: 0.18 MiB, 1.51%. PFC at 100 Gb/s takes 12.11% of the switch's entire buffer and PAUSE takes 1.51%, for the same eight-class link, and the difference is entirely the independence of the eight windows.

Which is the chapter's central cost sentence: the three bits are free in the frame and cost an eighth of the buffer.

8. RTL 4 — The Eight Timers

The receiving side holds eight independent countdowns. The interesting part is not the counting; it is what happens when one expires.

// -----------------------------------------------------------------------
// pfc_timer_bank -- eight independent pause countdowns, one per class.
//
// A quantum is 512 bit times at the port's rate. The prescaler converts
// the port clock into quantum ticks so the timers are rate-independent.
// -----------------------------------------------------------------------
module pfc_timer_bank
  import pfc_pkg::*;
#(
  parameter int CLK_MHZ   = 156,   // port clock
  parameter int RATE_MBPS = 10000  // 512 bits at 10 Gb/s = 51.2 ns
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        pfc_valid,     // one pulse from the parser
  input  class_vec_t  pfc_enable,
  input  quanta_vec_t pfc_quanta,

  output class_vec_t  paused,        // level: this class is stopped
  output logic [31:0] c_expiry [NUM_CLASSES],   // timer ran to zero
  output logic [31:0] c_release [NUM_CLASSES]   // explicitly released
);

  // Quantum period in port-clock cycles. 512 bits at RATE_MBPS is
  // 512000 / RATE_MBPS nanoseconds; times CLK_MHZ / 1000 gives cycles.
  localparam int QUANTUM_CYCLES = (QUANTUM_BITS * CLK_MHZ) / RATE_MBPS;

  logic [15:0] tick_div;
  logic        tick;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      tick_div <= '0;
      tick     <= 1'b0;
    end else begin
      if (tick_div == QUANTUM_CYCLES-1) begin
        tick_div <= '0;
        tick     <= 1'b1;
      end else begin
        tick_div <= tick_div + 1'b1;
        tick     <= 1'b0;
      end
    end
  end

  quanta_t timer [NUM_CLASSES];

  always_ff @(posedge clk or negedge rst_n) begin
    int c;
    if (!rst_n) begin
      for (c = 0; c < NUM_CLASSES; c++) begin
        timer[c]     <= '0;
        c_expiry[c]  <= '0;
        c_release[c] <= '0;
      end
    end else begin
      // A PFC frame is a COMPLETE statement -- every class is written,
      // including the ones being released. Section 2.
      if (pfc_valid) begin
        for (c = 0; c < NUM_CLASSES; c++) begin
          if (pfc_enable[c] && (pfc_quanta[c] != 16'd0)) begin
            timer[c] <= pfc_quanta[c];
          end else begin
            if (timer[c] != 16'd0) c_release[c] <= c_release[c] + 1;
            timer[c] <= 16'd0;
          end
        end
      end else if (tick) begin
        for (c = 0; c < NUM_CLASSES; c++) begin
          if (timer[c] != 16'd0) begin
            timer[c] <= timer[c] - 1'b1;
            if (timer[c] == 16'd1) c_expiry[c] <= c_expiry[c] + 1;
          end
        end
      end
    end
  end

  always_comb begin
    int c;
    for (c = 0; c < NUM_CLASSES; c++) paused[c] = (timer[c] != 16'd0);
  end

endmodule

Classification: eight parallel down-counters sharing one prescaler, with a reload port that always writes all eight.

What it teaches: that the timer is a dead-man's switch and expiry is the safe state. A class is paused only while its timer is non-zero, so losing every subsequent PFC frame releases the class after at most its remaining quanta — a link that goes quiet resumes rather than stalling. This is the opposite polarity from a design that pauses on a signal and resumes on another, where a lost resume strands the class for ever, and Chapter 14.2 §19 established that Ethernet acknowledges nothing.

And it teaches why c_expiry and c_release are separate counters, because the two mean different things about the neighbour. A release is the neighbour saying it is done. An expiry is the neighbour saying nothing and the timer running out. A healthy PFC link is dominated by releases; one dominated by expiries has a neighbour whose refresh interval is longer than the quanta it grants, which works but oscillates — the class resumes, floods, and is paused again, at exactly Chapter 14.2 §9's hysteresis-too-small frequency.

Deliberately simplified: QUANTUM_CYCLES is an integer division and truncates. At 156 MHz and 10 Gb/s it is 512 × 156 / 10000 = 7 cycles where the exact value is 7.987, so every quantum is 12.4% short and every pause ends early. Production designs carry a fractional accumulator. The error is systematically in the unsafe direction — a short pause releases the neighbour before the queue has drained — which is why the truncation matters more than its size.

Production implication: the prescaler is per port because the quantum is defined in bit times, and a 24-port switch with mixed rates has ports whose quantum differs by 100×. A 512-bit quantum is 512 ns at 1 Gb/s and 5.12 ns at 100 Gb/s — so the same quanta value in the same field means 33.55 ms on one port and 335.5 µs on another. A management interface that reports pause duration in quanta without naming the port's rate is reporting an uninterpretable number, and Section 22's third misconception is engineers comparing them across ports.

9. RTL 5 — Per-Class Transmit Gating

Chapter 14.2 §11 gated a whole transmitter. This gates a scheduler's eligibility set, which is a smaller change than it sounds and a larger one than it looks.

// -----------------------------------------------------------------------
// pfc_tx_class_gate -- masks paused classes out of the egress
// scheduler's candidate set.
//
// This sits BETWEEN 13.4 section 11's scheduler and the queues. The
// scheduler is unchanged; what changes is which queues it is allowed
// to see.
// -----------------------------------------------------------------------
module pfc_tx_class_gate
  import pfc_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  class_vec_t  paused,          // from the timer bank
  input  class_vec_t  q_nonempty,      // from the eight egress queues
  input  logic        tx_in_frame,     // a frame is mid-transmission

  input  logic        ctrl_pending,    // a PFC frame of our own to send

  output class_vec_t  eligible,        // what the scheduler may choose
  output logic        ctrl_grant,      // control frame bypasses everything
  output logic        all_blocked,     // work waiting, nothing eligible

  output logic [31:0] c_blocked_cycles,
  output logic [31:0] c_gated [NUM_CLASSES]
);

  // Rule 1 -- a frame already started is finished. Ethernet cannot abort
  // a frame in progress; 12.6 section 8 established that a truncated
  // frame is discarded by every receiver. This is the dead time's
  // dominant term, seen from the sending side.
  logic mid_frame_hold;
  assign mid_frame_hold = tx_in_frame;

  // Rule 2 -- control frames bypass the gate entirely. A device that is
  // pausing its neighbour must be able to send the PAUSE even while its
  // own classes are stopped. 14.2 section 11's rule, and the reason a
  // general control bypass is dangerous.
  assign ctrl_grant = ctrl_pending && !mid_frame_hold;

  always_comb begin
    int c;
    for (c = 0; c < NUM_CLASSES; c++) begin
      eligible[c] = q_nonempty[c] && !paused[c];
    end
  end

  assign all_blocked = (q_nonempty != '0) && (eligible == '0) && !ctrl_grant;

  always_ff @(posedge clk or negedge rst_n) begin
    int c;
    if (!rst_n) begin
      c_blocked_cycles <= '0;
      for (c = 0; c < NUM_CLASSES; c++) c_gated[c] <= '0;
    end else begin
      if (all_blocked) c_blocked_cycles <= c_blocked_cycles + 1;
      for (c = 0; c < NUM_CLASSES; c++) begin
        if (q_nonempty[c] && paused[c]) c_gated[c] <= c_gated[c] + 1;
      end
    end
  end

endmodule

Classification: a combinational mask on a scheduler's request vector, plus a bypass and two instrumentation counters.

What it teaches: that PFC is implemented as an eligibility mask and not as eight gates. Chapter 13.4 §11's scheduler already chooses among non-empty queues; PFC narrows the set it chooses from and changes nothing else. Which is why PFC is cheap in logic and why it inherits every property of the scheduler underneath it — including strict priority's starvation, which Chapter 13.4 §11's callout named as the default behaviour of the default discipline.

And it teaches that all_blocked is the signal the whole chapter turns on. Work is waiting, no class may send, and the port is idle. On a lossy link this state resolves itself — the queues fill, Chapter 14.1's allocator refuses, frames are discarded, and the backlog moves. On a lossless class it does not resolve, and Section 13 is about detecting the case where it never will.

Deliberately simplified: ctrl_grant is a single bypass with no queue behind it, so two control frames wanting to leave in the same window lose one. Production designs give control frames their own small queue at strict top priority — Chapter 14.2 §11's callout derived the requirement from the other direction: a PAUSE queued behind a full transmit queue arrives 4.19 ms late, by which point the congestion it reported has cleared or overflowed.

Production implication: c_gated[c] counts cycles in which class c had work and was not allowed to send it, and it is the per-class version of Chapter 14.2 §12's effective_rate_pct. Its use is to answer the operator's actual question, which is never "is PFC working" but "which class is paying for it". A link with c_gated[3] at 40% and every other class near zero has one class being throttled and seven unaffected — which is precisely the outcome PFC was deployed to produce, and without the per-class counter it is indistinguishable from a link that is simply idle.

10. PFC Moves the Blocking, It Does Not Remove It

This is the most over-claimed property of the mechanism and the arithmetic is unambiguous.

Chapter 14.3 §4 derived the 2 − √2 bound: a switch whose ingress ports hold frames in FIFO queues delivers 58.6% of capacity under uniform random traffic, because a head frame that cannot be forwarded blocks everything behind it including frames for idle destinations.

PFC changes the head. It does not change that there is one.

The fourth row is the finding. Eight FIFO queues are eight FIFO queues, and each one has a head, and each head blocks its own queue for exactly the reason Chapter 14.3 §2 laid out: the frame behind it is bound for an idle port and cannot reach it.

So a switch with PFC and without VOQs is a 58.6% switch with eight-way class isolation. The isolation is real and valuable — a paused storage class no longer stops a voice class — and the capacity loss is untouched.

And the two mechanisms are orthogonal in the strict sense, which is why a real lossless fabric needs both:

VOQs index the ingress queue by destination and remove head-of-line blocking within one switch. PFC indexes the pause by class and removes collateral between switches. Neither substitutes for the other, and a design that deploys PFC expecting VOQs' benefit has bought class isolation and been told it bought throughput.

11. RTL 6 — Per-Class Head-of-Line Detection

Section 10's claim needs an instrument. This is Chapter 14.3 §5's detector, re-indexed by class.

// -----------------------------------------------------------------------
// pfc_class_hol_detector -- per class, distinguishes a head frame that
// is blocked because its destination is congested from one blocked
// because its class is paused.
//
// The two look identical at the queue and mean opposite things: the
// first says this switch is a cause, the second says it is a conduit.
// -----------------------------------------------------------------------
module pfc_class_hol_detector
  import pfc_pkg::*;
#(
  parameter int NUM_PORTS = 24
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic                    head_valid  [NUM_CLASSES],
  input  logic [$clog2(24)-1:0]   head_dest   [NUM_CLASSES],
  input  logic                    head_granted[NUM_CLASSES],

  input  class_vec_t              paused,       // our own tx gate
  input  logic [NUM_PORTS-1:0]    dest_busy,    // egress genuinely full
  input  logic [NUM_PORTS-1:0]    dest_xoff,    // egress paused by neighbour

  output logic [31:0] c_hol_cycles   [NUM_CLASSES],
  output logic [31:0] c_first_order  [NUM_CLASSES],  // our congestion
  output logic [31:0] c_second_order [NUM_CLASSES],  // somebody else's
  output logic [31:0] c_self_paused  [NUM_CLASSES],  // our class is stopped
  output logic [31:0] c_none         [NUM_CLASSES]   // blocked, no reason
);

  always_ff @(posedge clk or negedge rst_n) begin
    int c;
    if (!rst_n) begin
      for (c = 0; c < NUM_CLASSES; c++) begin
        c_hol_cycles[c]   <= '0;
        c_first_order[c]  <= '0;
        c_second_order[c] <= '0;
        c_self_paused[c]  <= '0;
        c_none[c]         <= '0;
      end
    end else begin
      for (c = 0; c < NUM_CLASSES; c++) begin
        if (head_valid[c] && !head_granted[c]) begin
          c_hol_cycles[c] <= c_hol_cycles[c] + 1;

          // Attribution, in priority order. A class that is itself
          // paused explains the block regardless of the destination.
          if (paused[c])                    c_self_paused[c]  <= c_self_paused[c]  + 1;
          else if (dest_xoff[head_dest[c]]) c_second_order[c] <= c_second_order[c] + 1;
          else if (dest_busy[head_dest[c]]) c_first_order[c]  <= c_first_order[c]  + 1;
          else                              c_none[c]         <= c_none[c]         + 1;
        end
      end
    end
  end

endmodule

Classification: a four-way attribution counter bank, replicated per class. It changes no behaviour and decides every investigation.

What it teaches: that PFC adds a third reason a head can be blocked, and it is the one that is new. Chapter 14.3 §15 had two — the destination is congested, or the destination is paused by its own neighbour. PFC adds: this class is itself stopped, which is not a property of the destination at all. A design that keeps Chapter 14.3's two-way attribution under PFC will charge every self-paused cycle to second_order, and conclude that an upstream switch is causing blocking that this switch's own neighbour requested.

And it teaches that the ordering of the if chain is a specification, not an optimisation. A cycle in which the class is paused and the destination is busy is charged to self_paused, because that is the condition the operator can act on — the pause is a decision some device made, and the destination's state is irrelevant while the gate is shut.

Deliberately simplified: dest_xoff is presented as a wire from the egress side, which assumes the egress and ingress of one switch share a clock domain and a view. On a chassis switch they are different line cards and the state arrives over an internal bus with latency; a stale dest_xoff mis-attributes a first-order block as second-order. Production designs timestamp the state or accept a bounded attribution error and say so.

Production implication: c_none[c] is the per-class version of Chapter 14.3 §15's none_of_the_above, and under PFC it acquires a second meaning worth knowing. A head blocked with no destination congestion, no upstream pause and no self-pause is a scheduler that is not granting — Chapter 13.4 §11's strict priority starving a low class while a high one is continuously busy. PFC did not cause it and PFC makes it visible, because the per-class counter separates a class that is stopped from a class that is merely never chosen.

12. The Lossless Class

Every chapter of Module 14 so far has treated dropping as the thing a switch is forced into. It is time to say the other half, because a lossless class is a decision to give it up.

Chapter 12.1 §6 derived the arithmetic: 23 ports offering line rate to one must discard 95.7% of it, and discarding is the switch's specified response. Chapter 14.1 spent twenty sections on where that discard physically happens. Neither chapter said what the discard is for.

It is for progress. A switch that discards always has somewhere to put the next frame, because it can always make room. A switch that has promised never to discard cannot make room, and when it has none it stops.

And the case for promising anyway is real and specific.

The fifth row is the entire commercial case for PFC. A storage or RDMA transport that recovers by timeout turns a single dropped frame into a 192× latency spike, and it does so on a workload where the ninety-ninth percentile is the number being sold. Mean throughput barely moves — at a frame loss rate of one in a hundred thousand, 0.04% of transfers retry — and the tail moves by more than two orders of magnitude.

So losslessness is not bought for bandwidth. It is bought for the tail, and the tail is what a storage array is judged on.

The last row is what it costs, and Sections 13 to 15 are that row.

13. RTL 7 — The Deadlock Detector

A deadlock cannot be detected from inside a switch. What can be detected is the local signature, and the distinction is the reason Section 19's rejected property is what it is.

// -----------------------------------------------------------------------
// pfc_deadlock_detector -- watches for the LOCAL SIGNATURE of a
// cyclic buffer dependency on a lossless class.
//
// It cannot see a cycle. A cycle is a property of a graph spanning
// several switches, and this module can see one node's edges. What it
// can see is: this class has been paused continuously for longer than
// any legitimate pause, its queue is non-empty, and nothing has left.
// -----------------------------------------------------------------------
module pfc_deadlock_detector
  import pfc_pkg::*;
#(
  // Four times the longest single grant the standard allows, in port
  // clocks. At 10 Gb/s, 65535 quanta is 3.355 ms; 4x is 13.4 ms.
  parameter int STALL_LIMIT = 6710784
)(
  input  logic        clk,
  input  logic        rst_n,

  input  class_vec_t  lossless_mask,
  input  class_vec_t  paused,          // our tx gate, from the timer bank
  input  class_vec_t  q_nonempty,
  input  class_vec_t  deq_fire,        // a frame actually left this class

  output class_vec_t  suspect,         // local signature present
  output logic [31:0] c_suspect [NUM_CLASSES],
  output logic [31:0] longest_stall [NUM_CLASSES],

  // The only honest escape: after the signature persists, discard from
  // the class. This BREAKS the lossless promise deliberately and says
  // so -- section 15's condition 3.
  output class_vec_t  emergency_drop
);

  logic [31:0] stall [NUM_CLASSES];

  always_ff @(posedge clk or negedge rst_n) begin
    int c;
    if (!rst_n) begin
      for (c = 0; c < NUM_CLASSES; c++) begin
        stall[c]         <= '0;
        c_suspect[c]     <= '0;
        longest_stall[c] <= '0;
      end
      suspect        <= '0;
      emergency_drop <= '0;
    end else begin
      for (c = 0; c < NUM_CLASSES; c++) begin
        // Any forward progress resets the evidence.
        if (deq_fire[c] || !q_nonempty[c] || !lossless_mask[c]) begin
          stall[c]   <= '0;
          suspect[c] <= 1'b0;
          emergency_drop[c] <= 1'b0;
        end else if (paused[c]) begin
          stall[c] <= stall[c] + 1;

          if (stall[c] > longest_stall[c]) longest_stall[c] <= stall[c];

          if (stall[c] == STALL_LIMIT) begin
            suspect[c]   <= 1'b1;
            c_suspect[c] <= c_suspect[c] + 1;
          end
          // A second full interval with no progress: stop believing
          // the neighbour and make room.
          if (stall[c] == (2*STALL_LIMIT)) emergency_drop[c] <= 1'b1;
        end
      end
    end
  end

endmodule

Classification: a per-class stall watchdog with two thresholds — a warning and a policy override.

What it teaches: that the detector detects a symptom and not the condition, and the module's comment says so because a design that forgets it will trust the signal. The local signature has two causes: a genuine cycle, and a neighbour that is legitimately congested for a very long time. Nothing observable at this switch separates them, because the difference lies in whether the wait-for graph closes three hops away.

And it teaches what emergency_drop really is. It is the switch withdrawing the lossless guarantee unilaterally, and it is the only escape a single device has. Section 15's derivation makes this precise: dropping breaks Coffman's third condition, and the third condition is the only one a lone switch can reach. A design without an escape is a design that stops for ever; a design with one has a guarantee with an asterisk. Both are defensible and only one of them is honest about which it is.

Deliberately simplified: STALL_LIMIT is a parameter and the correct value depends on the port's rate, because the longest legitimate pause is 65 535 quanta and a quantum is 512 bit times. At 1 Gb/s that is 33.55 ms and at 100 Gb/s it is 335.5 µs — a factor of 100. A single limit across a mixed-rate switch is either 100× too slow on the fast ports or fires spuriously on the slow ones.

Production implication: longest_stall[c] is a high-water mark and is the counter that makes the threshold defensible. An operator cannot choose STALL_LIMIT from first principles — it depends on how long real congestion lasts on this network, which nobody knows in advance. The high-water mark on a healthy network is the empirical answer, and a threshold set at several times the observed maximum is a threshold that will not cry wolf. Without it, the parameter is a guess, and Section 22's sixth misconception is a network where the detector was disabled because it kept firing.

14. A Cycle of Classes: the Smallest Case

The failure needs a concrete instance before it needs a theory, and the smallest instance fits on one line.

Three switches. One lossless class. Traffic that happens to go round.

Nothing in that sequence is a fault. Every switch obeyed the standard, every pause was correctly issued against a correctly computed watermark, every timer is being refreshed on schedule, and every frame is intact.

And the system will not move again. The three buffers hold 1536 cells — 192 KiB of frames that are each one hop from their destination, and each one is waiting for a hop that is waiting for it.

Two switches are enough if the traffic turns around. A sends class-3 frames to B, B sends class-3 frames to A, and each switch's ingress buffer from the other drains toward the other. This requires traffic that goes A → B → A, which normal forwarding does not produce — but a link failure and a reroute produce it for the milliseconds before the tables converge, and a millisecond is 6 710 784 clocks.

Three switches is the smallest cycle that needs no forwarding loop at all. A → B → C → A is three ordinary paths, each perfectly sensible on its own, and the cycle exists in the buffer dependency graph rather than in the forwarding tables. No device has a loop. No spanning tree is violated. There is no configuration a network operator could inspect that would show it.

15. The Deadlock Condition, Derived

Section 14 is an instance. The condition behind it was stated in 1971 and has four parts, and the useful thing about it is that a design can inspect which parts it has removed.

All four must hold simultaneously for a deadlock. Removing any one makes it impossible.

And the third row is the whole story of this chapter.

Lossy Ethernet does not have condition 3. A switch under pressure preempts — it takes a frame that is holding a cell and throws it away, and the cell is free. That is not a failure mode; it is the preemption that makes the other three conditions harmless. Chapter 12.1 §6 called discarding the switch's specified response and was more right than it needed to be: it is also the reason a switched Ethernet has never required a deadlock detector.

PFC's entire purpose is to remove condition 3, because "never discard" is exactly "no preemption". The mechanism does not accidentally admit deadlock. It admits deadlock by construction, and the admission is the feature being purchased.

Which leaves condition 4 as the only remaining defence, and it is a graph property.

The first three all require something outside the switch to be true, and the switch cannot check any of them. The fourth is the only one a device can do by itself, and it does it by giving back exactly what was bought.

That is the honest summary of lossless Ethernet: it converts a local, measurable, attributable frame loss into a global, unobservable, topological correctness requirement. Whether that is a good trade depends entirely on whether the topology is under one organisation's control — which in a data-centre fabric it usually is, and in a campus network it never is.

16. RTL 8 — Conformance for a Lossless Class

A conformance bit for PFC has to say something narrower than the operator wants to hear, and the narrowness is the point.

// -----------------------------------------------------------------------
// pfc_conformance_monitor -- one bit per port summarising whether the
// PFC implementation is behaving to specification.
//
// It does NOT say the network is deadlock-free. Section 15 established
// that no device can say that. It says this device did what it promised.
// -----------------------------------------------------------------------
module pfc_conformance_monitor
  import pfc_pkg::*;
(
  input  logic       clk,
  input  logic       rst_n,

  input  logic       cfg_infeasible,      // elaboration-time, section 6
  input  class_vec_t lossless_mask,

  input  class_vec_t overflow_fire,       // a lossless class overflowed
  input  class_vec_t late_arrival,        // frames after the dead time
  input  class_vec_t lossy_asserted,      // a lossy class asserted pause
  input  class_vec_t escape_taken,        // emergency_drop fired
  input  logic       tagged_pfc_seen,     // a tagged PFC frame we ignored

  output logic       conformant,
  output logic [7:0] fault_vector,
  output logic [31:0] c_overflow,
  output logic [31:0] c_late,
  output logic [31:0] c_escape
);

  logic v_overflow, v_late, v_lossy_pause, v_escape, v_tagged;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      v_overflow <= 1'b0; v_late <= 1'b0; v_lossy_pause <= 1'b0;
      v_escape   <= 1'b0; v_tagged <= 1'b0;
      c_overflow <= '0;   c_late  <= '0;  c_escape <= '0;
    end else begin
      // A lossless class that overflowed is the mechanism failing at
      // its single promise. Sticky: one occurrence is the finding.
      if ((overflow_fire & lossless_mask) != '0) begin
        v_overflow <= 1'b1;
        c_overflow <= c_overflow + 1;
      end

      // Frames arriving after the dead time expired means the neighbour
      // is not honouring our pause -- 14.2 section 15's ambiguity, now
      // per class.
      if (late_arrival != '0) begin
        v_late <= 1'b1;
        c_late <= c_late + 1;
      end

      // A lossy class asserting pause is a configuration error: it has
      // 14.1's discard already and gains only a deadlock risk.
      if ((lossy_asserted & ~lossless_mask) != '0) v_lossy_pause <= 1'b1;

      // The escape is not a fault in the mechanism -- it is the
      // mechanism being overridden, and it must be visible.
      if (escape_taken != '0) begin
        v_escape <= 1'b1;
        c_escape <= c_escape + 1;
      end

      if (tagged_pfc_seen) v_tagged <= 1'b1;
    end
  end

  // The standing property sits inside the conjunction: a headroom that
  // does not fit is wrong from power-on, not from the first drop.
  // 14.1 section 17 and 13.4 section 14 made the same argument.
  assign conformant = !cfg_infeasible &&
                      !v_overflow && !v_late &&
                      !v_lossy_pause && !v_escape && !v_tagged;

  assign fault_vector = {3'b000, v_tagged, v_escape,
                         v_lossy_pause, v_late, v_overflow};

endmodule

Classification: a sticky fault aggregator with one standing configuration term and a five-bit decomposition.

What it teaches: that conformant here means this device kept its promise, and deliberately does not mean the class is lossless. A switch in Section 14's cycle is fully conformant — it never overflowed, never dropped, never paused a lossy class, and its headroom fits. It is also permanently stopped. A design whose conformance bit could go low in that situation would be claiming to detect the undetectable.

And it teaches why v_escape is a fault rather than a normal event. emergency_drop firing means the switch discarded a frame from a class documented as lossless — the promise was broken, by us, deliberately, and correctly. It is still a broken promise, and hiding it inside a policy would leave an operator investigating a storage timeout with no record that the fabric had chosen to drop.

Deliberately simplified: every fault bit is sticky and never clears, so a single event at power-on holds the port non-conformant for ever. Production designs need a write-one-to-clear path and, more importantly, a timestamp on first occurrence — Chapter 14.1 §17's monitor had the same shape and the same gap. Without a timestamp, conformant = 0 says something went wrong and gives an operator no way to correlate it with anything.

Production implication: v_tagged catches the failure Section 4's parser was simplified into. A tagged PFC frame is legal and this parser ignores it, so the bit exists to make the ignoring visible rather than silent. This is the pattern worth generalising from the module: where a design knowingly does not implement part of a specification, the conformance bit should report encountering the unimplemented case, not merely decline to handle it. A parser that silently ignores what it cannot parse is indistinguishable from a neighbour that is not sending.

17. What Dropping Bought That Losslessness Gives Up

Four chapters of Module 14 can now be put in one table, and the shape that emerges is not the one the module started with.

Read the last two rows down the table, because they move in the same direction and it is the opposite of the direction the collateral column moves.

Every step from left to right reduces the collateral and weakens the guarantee. Discarding hurts one frame and always works. Stopping a link hurts every conversation on it and always works. Stopping a class hurts only that class and works only if the fabric has no cycle in that class's dependency graph.

And every step makes the failure harder to see. A drop is a counter. A slow link is a rate. A deadlock is silence — no counter moves, no error is logged, no assertion fires, and the only symptom is that something stopped.

Which is what dropping bought. It bought a failure that announces itself, is attributable to a port, is bounded in effect, and cannot compose into a lock-up. Chapter 14.1 §9 spent an RTL module attributing drops precisely because they can be attributed. Nothing in this chapter can be attributed at all, because the thing to attribute is a property of a graph.

And it is still, frequently, the right trade. Section 12's 192× tail is real, and a storage fabric under one organisation's control, with a designed topology and a routing protocol that forbids cycle-closing turns, has removed condition 4 by construction and can take the guarantee. The trade is bad exactly where the topology is not controlled — which is why PFC is standard in data-centre fabrics and almost unheard of in campus networks, and why that distribution is not an accident of vendor support.

18. The Cost of PFC, Accounted

Put the whole mechanism's price in one place, because two of the four rows are much smaller than expected and one is much larger.

The control state is 1.31 KiB and the headroom is 1.45 MiB. The mechanism's cost is entirely the reserve and not at all the logic, which is the reverse of Chapter 14.3 §14's virtual output queues — those cost 2.3 KiB of pointers and a 24 × 24 arbiter, negligible memory and the switch's tightest timing path.

Two mechanisms, opposite cost profiles, and both are needed:

A design that can afford only one should know which problem it has, and the diagnostic is Chapter 14.3 §5's throughput_pct against this chapter's c_gated. A switch at 58.6% under uniform load needs VOQs. A switch whose voice class stalls whenever storage bursts needs PFC. They are different complaints with different fixes and one shared symptom, which is that something is slow.

19. Properties Worth Asserting, and One Worth Refusing

The properties divide by what they are about: the frame, the timers, the headroom, the gate, the detector, and the configuration. Every one of them is local and decidable, which Section 15 established is not a small requirement here.

Group 1 — the frame is well formed.

// P1. A PFC frame carries the PFC opcode and the control DA. Nothing
// else may be emitted by this builder.
property p_frame_identity;
  @(posedge clk) disable iff (!rst_n)
  tx_sop |-> ##14 (tx_octet == OPCODE_PFC[15:8]) ##1 (tx_octet == OPCODE_PFC[7:0]);
endproperty

// P2. The frame is exactly 60 body octets. 14.2's PAUSE was the same
// length; the extra 16 octets came out of the pad.
property p_frame_length;
  @(posedge clk) disable iff (!rst_n)
  tx_sop |-> ##59 tx_eop;
endproperty

// P3. A class whose enable bit is clear transmits zero quanta. The two
// encodings of "released" must agree -- section 3.
property p_release_encoding_agrees;
  @(posedge clk) disable iff (!rst_n)
  (req_valid && !req_enable[c]) |-> ##(18 + 2*c) (tx_octet == 8'h00)
                                    ##1 (tx_octet == 8'h00);
endproperty

// P4. The builder never starts a frame while one is in progress.
property p_builder_no_overlap;
  @(posedge clk) disable iff (!rst_n)
  tx_sop |-> !tx_sop throughout (tx_eop [->1]);
endproperty

Group 2 — the parser commits only complete, correct frames.

// P5. No decision is published unless the FCS was good.
property p_no_commit_on_bad_fcs;
  @(posedge clk) disable iff (!rst_n)
  (rx_eop && !rx_fcs_ok) |-> !pfc_valid;
endproperty

// P6. A frame whose DA is not the control multicast is never honoured,
// however well formed the rest of it is.
property p_wrong_da_never_honoured;
  @(posedge clk) disable iff (!rst_n)
  (rx_eop && !da_ok) |-> !pfc_valid;
endproperty

// P7. pfc_valid is exactly one cycle wide -- the timer bank reloads on
// a pulse and a level would reload every cycle.
property p_commit_is_a_pulse;
  @(posedge clk) disable iff (!rst_n)
  pfc_valid |=> !pfc_valid;
endproperty

// P8. Every honoured frame increments exactly one of the three
// counters. The decomposition is a partition, not an overlap.
property p_counter_partition;
  @(posedge clk) disable iff (!rst_n)
  (rx_eop && op_ok) |=> ($changed(c_seen) ^ $changed(c_bad_fcs)
                                          ^ $changed(c_wrong_da));
endproperty

Group 3 — the timers behave as a dead-man's switch.

// P9. Every PFC frame writes all eight timers. A complete-state
// message leaves no class untouched -- section 2.
property p_all_eight_written;
  @(posedge clk) disable iff (!rst_n)
  pfc_valid |=> (timer[c] == (pfc_enable[c] ? pfc_quanta[c] : 16'd0));
endproperty

// P10. A timer only ever decreases by one, on a quantum tick.
property p_timer_decrements_on_tick;
  @(posedge clk) disable iff (!rst_n)
  (!pfc_valid && tick && (timer[c] != 0)) |=> (timer[c] == $past(timer[c]) - 1);
endproperty

// P11. A timer never changes on a non-tick cycle without a frame.
property p_timer_stable_between_ticks;
  @(posedge clk) disable iff (!rst_n)
  (!pfc_valid && !tick) |=> $stable(timer[c]);
endproperty

// P12. Expiry is the safe state: a timer reaching zero always releases.
property p_expiry_releases;
  @(posedge clk) disable iff (!rst_n)
  (timer[c] == 16'd0) |-> !paused[c];
endproperty

// P13. A class cannot be paused for longer than the quanta it was
// granted, absent a refresh. The bound is 65535 ticks.
property p_pause_is_bounded;
  @(posedge clk) disable iff (!rst_n)
  (paused[c] && !pfc_valid) |-> ##[1:65535*QUANTUM_CYCLES] !paused[c];
endproperty

Group 4 — the headroom is respected.

// P14. The central safety property of the whole mechanism: while a
// lossless class is not asserting pause, its occupancy is below the
// high watermark, so a full dead window still fits.
property p_headroom_respected;
  @(posedge clk) disable iff (!rst_n)
  (lossless_mask[c] && !assert_pause[c]) |-> (occ[c] < occ_t'(HIGH_WM));
endproperty

// P15. Crossing the high watermark asserts within one cycle.
property p_high_wm_asserts;
  @(posedge clk) disable iff (!rst_n)
  (lossless_mask[c] && (occ[c] >= occ_t'(HIGH_WM))) |=> assert_pause[c];
endproperty

// P16. Hysteresis: release requires falling to LOW, not merely below
// HIGH. Without this the mechanism oscillates -- 14.2 section 9.
property p_release_needs_low;
  @(posedge clk) disable iff (!rst_n)
  ($fell(assert_pause[c])) |-> ($past(occ[c]) <= occ_t'(LOW_WM));
endproperty

// P17. A lossy class never asserts pause. It has 14.1's discard.
property p_lossy_never_pauses;
  @(posedge clk) disable iff (!rst_n)
  !lossless_mask[c] |-> !assert_pause[c];
endproperty

// P18. A lossless class never overflows while the configuration is
// feasible and the neighbour is honouring pause.
property p_no_lossless_overflow;
  @(posedge clk) disable iff (!rst_n)
  (lossless_mask[c] && !cfg_infeasible && !late_arrival[c])
    |-> !overflow_fire[c];
endproperty

Group 5 — the transmit gate.

// P19. A paused class is never chosen by the scheduler.
property p_paused_never_eligible;
  @(posedge clk) disable iff (!rst_n)
  paused[c] |-> !eligible[c];
endproperty

// P20. An unpaused, non-empty class is always eligible. The gate
// subtracts and never adds.
property p_gate_only_subtracts;
  @(posedge clk) disable iff (!rst_n)
  (q_nonempty[c] && !paused[c]) |-> eligible[c];
endproperty

// P21. A frame in progress is never truncated by a pause arriving
// mid-frame -- 12.6 section 8, and the dead time's dominant term.
property p_no_mid_frame_abort;
  @(posedge clk) disable iff (!rst_n)
  (tx_in_frame && $rose(paused[tx_class])) |-> tx_in_frame until_with tx_eop;
endproperty

// P22. Control frames leave even when every class is paused --
// 14.2 section 11's rule 2.
property p_control_bypasses;
  @(posedge clk) disable iff (!rst_n)
  (ctrl_pending && !tx_in_frame) |-> ctrl_grant;
endproperty

// P23. all_blocked means exactly what it says: work waiting, nothing
// permitted, no control frame pending.
property p_all_blocked_definition;
  @(posedge clk) disable iff (!rst_n)
  all_blocked <-> ((q_nonempty != '0) && (eligible == '0) && !ctrl_grant);
endproperty

Group 6 — the attribution and the detector.

// P24. Every blocked head-of-line cycle is charged to exactly one
// cause. Four counters, one increment.
property p_attribution_is_a_partition;
  @(posedge clk) disable iff (!rst_n)
  (head_valid[c] && !head_granted[c]) |=>
    ($changed(c_self_paused[c]) + $changed(c_second_order[c])
     + $changed(c_first_order[c]) + $changed(c_none[c]) == 1);
endproperty

// P25. A self-paused class is charged to self_paused whatever the
// destination is doing. The ordering is a specification -- section 11.
property p_self_pause_wins;
  @(posedge clk) disable iff (!rst_n)
  (head_valid[c] && !head_granted[c] && paused[c])
    |=> $changed(c_self_paused[c]);
endproperty

// P26. The local signature is sound: it is raised whenever its
// definition holds, and claims nothing beyond its definition.
property p_suspect_is_sound;
  @(posedge clk) disable iff (!rst_n)
  (stall[c] >= STALL_LIMIT) |-> suspect[c];
endproperty

// P27. Any forward progress clears the evidence, so the signature
// cannot accumulate across unrelated congestion episodes.
property p_progress_clears;
  @(posedge clk) disable iff (!rst_n)
  deq_fire[c] |=> (stall[c] == 32'd0);
endproperty

// P28. The escape is never taken silently: a discard from a lossless
// class implies emergency_drop was asserted for that class.
property p_escape_is_explicit;
  @(posedge clk) disable iff (!rst_n)
  (lossless_mask[c] && drop_fire[c]) |-> emergency_drop[c];
endproperty

Group 7 — the configuration.

// P29. A standing property: the headroom fits inside the class budget.
// This is wrong from power-on, not from the first drop -- 14.1
// section 17 and 13.4 section 14 made the same argument.
property p_headroom_fits;
  @(posedge clk) disable iff (!rst_n)
  (LOW_WM > 0) && (HIGH_WM > 0);
endproperty

// P30. cfg_infeasible and conformant cannot both be true. The
// configuration term sits inside the conjunction.
property p_infeasible_is_nonconformant;
  @(posedge clk) disable iff (!rst_n)
  cfg_infeasible |-> !conformant;
endproperty

// P31. Every fault bit is sticky: once raised it stays raised until
// reset. A conformance bit that could clear itself would hide the
// event that mattered.
property p_faults_are_sticky;
  @(posedge clk) disable iff (!rst_n)
  $rose(fault_vector[b]) |=> always fault_vector[b];
endproperty

// P32. conformant is exactly the conjunction it claims to be.
property p_conformant_definition;
  @(posedge clk) disable iff (!rst_n)
  conformant <-> (!cfg_infeasible && (fault_vector == 8'h00));
endproperty

Every property above is about this device. P14 is the mechanism's single safety promise, P13 bounds the pause, P24 keeps the attribution honest and P29 catches the configuration that cannot work. None of them says the class is lossless end to end, and none of them says the network makes progress — which is the subject of the one property this chapter refuses.

20. Verification Scenarios

Seventy-three scenarios. Several have expected outcomes in which the switch is fully conformant and permanently stopped.

The frame

The parser

The timers

Headroom and watermarks

The gate and the blocking

Attribution

Losslessness and the cycle

The directed test random stimulus will not produce

A three-switch buffer-dependency cycle cannot be reached by random traffic in any practical simulation, and the reason is worth stating: it requires three independent congestion events to overlap in time, on the same class, with destinations arranged so that each switch's drain path is the next switch in the cycle. Random destination selection produces that arrangement with probability that falls as the cube of the per-switch congestion probability, and the window in which all three must coexist is the dead time. A constrained-random run long enough to hit it is long enough to be impractical.

And the test matters more than any other in the chapter, because it is the only one that exercises the state in which every check passes and the system is dead.

Setup: three switches A, B, C. One lossless class, class 3, lossless_mask = 8'h08. Class budget 512 cells, 10 Gb/s ports, STALL_LIMIT = 6 710 784. Forwarding tables arranged so A's class-3 traffic egresses toward B, B's toward C, C's toward A — three ordinary paths, no forwarding loop, spanning tree intact.

Stimulus: offer class-3 traffic at 1.2× line rate simultaneously at A, B and C, sourced from ports outside the cycle, for a duration exceeding the time to fill 512 cells at each hop — 52.4 µs at 10 Gb/s. Then stop offering.

Oracle — the point of the test is that most of these are pass results:

Rows 1 to 15 are the finding. Fifteen observables, every one of them the value a healthy switch reports, and the fabric has stopped. Row 21 is the control: the same stimulus on a lossy class produces drops, counters, attribution and continuous forward progress — Chapter 14.1's entire apparatus, working.

Rows 16 to 20 are the only things any device can contribute, and all five are properties of the instrument rather than of the network.

21. Debugging PFC

The mechanism has four distinct failure modes and they present through three counters. Take them in this order.

Step 1 — is PFC arriving at all? c_seen on the receiving port. Zero means the neighbour is not sending, and the two causes are that PFC is not enabled there, or that it is sending tagged frames this parser ignores — v_tagged separates them. Non-zero means the mechanism is live and the investigation moves on.

Step 2 — is the configuration possible? cfg_infeasible and headroom_cells. This is a static check and it is the one most often skipped, because it does not depend on traffic and therefore does not appear in a traffic-shaped investigation. Section 7's last row is a real deployment and its symptom — every class permanently paused, queue empty — looks nothing like a buffer problem.

Step 3 — which class is paying? c_gated[c] across all eight. One class high and seven near zero is the mechanism working. All eight high is either a genuinely saturated link or Step 2's negative watermark. All eight near zero with drops occurring means the watermarks are too high — the queue overflows before the pause is issued.

Step 4 — is the neighbour honouring it? v_late and c_late. Frames arriving after the dead time expired means the pause was ignored, and Chapter 14.2 §15's ambiguity applies unchanged: a neighbour that should honour it and does not, against a neighbour that never agreed to. The negotiated capability at link-up is the discriminator and it is available from Chapter 11.3's bring-up rather than from a counter three weeks later.

Step 5 — is anything stalled? suspect[c] and longest_stall[c]. suspect high is not a deadlock diagnosis — Section 15 established that no device can make one. It is an instruction to look at the topology, because the local evidence is exhausted. longest_stall on a healthy network is the empirical basis for STALL_LIMIT, and a network where the parameter was guessed is a network where the detector was eventually disabled.

And the case that ends the investigation without a fault: c_seen non-zero, cfg_infeasible low, one class gated, v_late low, suspect low. That is a link doing exactly what PFC asks of it, and the slowness is a capacity question. Chapter 14.3 §15's argument for none_of_the_above applies here too — an explicit "all five checks pass" is the only state that ends an investigation rather than deferring it.

22. Common Misconceptions

1 — "PFC removes head-of-line blocking."

The wrong model: eight queues instead of one, so the head no longer blocks.

What it costs: a deployment bought to fix throughput that does not fix throughput. Section 10's table: eight FIFO queues have eight heads, and each head blocks its own queue for exactly Chapter 14.3 §2's reason — the frame behind it is bound for an idle port and cannot reach it. The 2 − √2 bound holds inside each of the eight.

The corrected model: PFC indexes the pause by class; virtual output queues index the ingress queue by destination. They are orthogonal, they fix different problems, and a lossless fabric needs both. PFC turns 88% collateral into ~11%. It leaves 58.6% at 58.6%.

2 — "The neighbour is ignoring our PFC frames."

The wrong model: frames keep arriving after a pause, so the far end is non-conformant.

What it costs: an escalation to the wrong vendor. Three causes produce identical evidence: the neighbour genuinely ignores PFC; the neighbour never negotiated the capability; or our own PFC frames are tagged and its parser has hard-coded offsets — Chapter 13.2 §7's point, that a tag moves every offset after octet 12, so the opcode sits at 18 and not 14.

The corrected model: check c_seen on the neighbour, not c_asserted on ours. A device that never counted the frame never ignored it, and v_tagged distinguishes a parser that saw something it could not handle from one that saw nothing.

3 — "A pause of 1000 quanta is a pause of 1000 quanta."

The wrong model: the quanta field is a duration.

What it costs: management output that cannot be compared between ports. A quantum is 512 bit times, so it is 512 ns at 1 Gb/s and 5.12 ns at 100 Gb/s — the same field value means 33.55 ms on one port of a switch and 335.5 µs on another, a factor of 100.

The corrected model: quanta are a rate-relative unit and every report must carry the port's rate. A dashboard that graphs pause duration in quanta across a mixed-rate switch is graphing three different quantities on one axis, and the slow ports will always look worse.

4 — "The switch supports PFC on all eight priorities."

The wrong model: the standard defines eight, so eight are available.

What it costs: a lossless deployment that overflows. Section 7's last row: eight classes at 100 Gb/s over 2 km need 7920 cells of headroom on a 4096-cell queue — 193.4% of it. The eight are available only if the buffer arithmetic permits, and at high rate over distance it does not.

The corrected model: the number of lossless classes a port can support is queue_cells / headroom_cells, computed per port from the link's rate and length. At 100 Gb/s over 2 km that number is four, and a design promising eight has promised what cfg_infeasible will refuse.

5 — "Losslessness means no frames are lost."

The wrong model: the class is configured lossless, so the guarantee is unconditional.

What it costs: the guarantee is conditional on something nobody checked. Section 15's four conditions: PFC removes preemption, which leaves circular wait as the only remaining defence, and circular wait is a property of the topology. A lossless class in a fabric with a cycle in its buffer dependency graph does not lose frames — it stops, permanently, and then a switch discards anyway via emergency_drop.

The corrected model: losslessness is a promise conditional on an acyclic dependency graph, which no device can verify. It is a deployment requirement wearing a configuration bit's clothing, and the honest form of the claim names the condition.

6 — "The deadlock detector kept firing, so we turned it off."

The wrong model: a detector that fires without a deadlock is broken.

What it costs: the only instrument the design has. suspect is sound and not complete: it reports the local signature, which a genuine cycle and a very long legitimate congestion both produce — Section 13's module comment says so and Section 15 explains why nothing better is possible.

The corrected model: the detector's threshold is empirical, not principled. longest_stall on a healthy network is the measurement STALL_LIMIT should be several times larger than, and a threshold derived that way stops crying wolf. A detector that fires often has a threshold below the network's normal congestion duration — which is itself a finding, and disabling it discards both the alarm and the measurement.

23. Interview Reasoning

Q1 — Eight classes, one link. How much headroom does the port need, and why is the obvious answer wrong twice?

Each class needs a full dead window — 1664 octets at 1 Gb/s — so the port needs eight of them, 13 316 octets. The first wrong answer divides one window eight ways, which fails because nothing forces traffic to spread across classes: a dead window in which every octet is class 3 is ordinary. The second wrong answer says eight windows exceed what the wire can deliver and must therefore be reclaimable — true about any single window and false across time, because the eight classes enter their windows at different moments. The reserve is 8× and the occupancy is never more than 1×, and both are correct.

Q2 — Why does PFC cost more buffer at 100 Gb/s than at 1 Gb/s?

Because the dead time's floor is propagation, which does not scale. At 1 Gb/s, 500 ns of propagation is 3.8% of a 13.32 µs window dominated by a neighbour finishing a 1518-octet frame. At 100 Gb/s that frame takes 121 ns and the same 500 ns is 79.6% of the window. The window falls 21× and the rate rises 100×, so the headroom rises 4.7×. The general shape is rate × fixed_delay: faster hardware makes the requirement worse.

Q3 — A switch has PFC enabled and delivers 58.6% under uniform load. Is it broken?

No, and the number is Chapter 14.3's bound rather than anything to do with PFC. PFC gives eight FIFO queues where there was one; each has a head and each head blocks its own queue. The fix for 58.6% is virtual output queues — index the ingress queue by destination, not by class — which costs 2.3 KiB of pointers and 10 ns of scheduler time. PFC and VOQs solve different problems and a fabric needs both.

Q4 — Ethernet has never deadlocked. Why can a lossless class?

Because dropping is a preemption, and preemption is one of the four conditions a deadlock needs. A lossy switch under pressure always has an action available — discard, free a cell, make progress — so no arrangement of neighbours can lock it. PFC's whole purpose is to promise never to discard, which is precisely "no preemption", and that promise removes the only condition a single device could break. What remains is circular wait, which is a property of the buffer dependency graph and therefore of the topology.

Q5 — Can a switch detect that it is deadlocked?

No. It can detect a local signature and nothing more. A cycle is a property of a graph spanning several devices; a switch observes its own edges. A switch in a genuine cycle and a switch one hop from a very congested storage array see identical local state: a paused egress, a full buffer, no dequeue, a timer being refreshed. The states are locally indistinguishable and globally opposite. The only action a lone device can take is emergency_drop — withdrawing the guarantee — and that must be visible, because it means a class documented as lossless has lost a frame.

Q6 — When is PFC the right answer, and when is it not?

It is right where the tail matters and the topology is controlled. A storage or RDMA transport recovering by timeout turns one dropped frame into a 192× latency spike on a 64 KiB transfer; mean throughput barely moves and the ninety-ninth percentile moves by two orders of magnitude. In a data-centre fabric with a designed topology and turn-restricted routing, circular wait is removed by construction and the guarantee holds. It is wrong where the topology is not under one organisation's control, because the guarantee is then conditional on something nobody can check — which is why PFC is standard in fabrics and almost unheard of in campus networks.

24. Understanding Check

25. What's Next

Module 14 is complete, and it turned out not to be about congestion.

Chapter 14.1 found where frames are lost — at the egress, from a shared pool, charged to whichever port asked next. Chapter 14.2 tried to prevent the loss by stopping the sender, at a dead time of 13.32 µs and 88% collateral. Chapter 14.3 followed the stopping across a topology and found 41.4% of a switch's capacity unreachable with nothing broken. And this chapter subdivided the link eight ways, paid an eighth of the buffer for it, and admitted a failure Ethernet had never had.

The module's real subject is that every mechanism it built moves a problem rather than removing one, and that the counters which matter are, without exception, the ones naming a party other than the one reporting. drops_by_ingress names a conversation. pool_drops_caused names a port. second_order_pct names a switch. And this chapter's suspect names nothing at all — because the thing to name is a cycle, and no device can see one.

Module 15 — Link Aggregation — starts from the opposite direction. Instead of subdividing one link, it presents several as one.

Chapter 15.1 — Bonding Links and Failover establishes the constraint that shapes the whole module, which is frame ordering, and derives what a failover actually costs: detection, convergence, and Chapter 12.5's table now pointing at a member that is gone.

Then Chapter 15.2 — Hash-Based Distribution and Frame Ordering builds the flow hash properly and confronts the conversation a per-flow hash cannot split.

And Chapter 15.3 — LACP replaces static configuration with negotiation, and shows the specific failure static configuration cannot detect.

One connection is worth carrying forward. This chapter's guarantee was conditional on a topology property no device can observe. Module 15's aggregation makes a guarantee too — that frames of one conversation arrive in order — and it will turn out to rest on a property of the distributor that a device very much can observe, and usually does not.