The MAC's FIFOs

A MAC has three buffers that are all called FIFOs and are sized by three unrelated arguments. Confusing any two of them produces a design that is wrong by two orders of magnitude in one direction or the other.

Three arguments, and the ratio between the smallest and the largest is 32 768. None of the three is a latency in the sense a system architect means; one is a rate difference, one is a synchroniser's depth, and one is the time a cable takes to carry a decision to the other end and back.

Chapter 18.1 §8 sized the receive FIFO from the system's side — how long the memory subsystem might refuse data. This chapter sizes the same buffer from the MAC's side, and the MAC's side asks a question the system's side never does: what happens when nothing is stalled and nothing is congested and the two clocks are simply not the same frequency.

1. Scope, and Three Buffers That Are Not the Same Buffer

The confusion this chapter exists to prevent has a specific shape. A designer reads Chapter 4.4, learns that two Ethernet clocks differ by up to 200 ppm and that a running total with a non-zero mean is unbounded, and concludes that the datapath FIFO must be sized against that unbounded accumulation. It must not; the elastic buffer already absorbed it, and it did so with one octet.

The opposite error is more common and more expensive. A designer sizes the receive FIFO from the memory stall — Chapter 18.1 §8's 24.41 KiB at 100 Gb/s for a 2 µs stall — and ships a port that cannot do lossless flow control, because the 7.67 KiB the PAUSE dead time needs was never reserved.

Row three is the one worth naming explicitly. The elastic buffer sits between the recovered receive clock and the local clock and its job is to let the PHY insert and delete idle. The datapath FIFO sits between the MAC's clock and the system's and its job is to absorb a consumer that stops. They are on opposite sides of the MAC and neither can do the other's work.

What this chapter owns: the elastic buffer's depth derived from the clock tolerance rather than quoted; the crossing FIFO's synchroniser reserve; the almost-full threshold derived from Chapter 14.2 §5's dead-time components rather than chosen; the crossover between the stall term and the headroom term and the closed form for it; and the transmit FIFO's threshold against Chapter 19.3's commit point.

What it does not own: the PAUSE mechanism itself — Chapter 14.2 — the memory system's stall distribution — Chapter 18.5 — and the elastic buffer's control loop, which Chapter 4.4 built and this chapter only sizes.

What it does not build: the memory interface is Chapter 19.6 and the statistics counters are Chapter 19.7.

2. The Elastic Buffer Is One Octet

Chapter 4.4 §2 established the bound and this section re-derives it at a 512-bit datapath, because the answer is small enough that a reader who has just read that chapter's "unbounded accumulation" will not believe it.

IEEE 802.3 specifies ±100 ppm at each end, so two ends differ by at most 200 ppm. Over a frame of L octets the accumulated difference is L × 2 × 10⁻⁴ octets.

Row two is Chapter 4.4 §2's published 0.3036 and it reproduces exactly, which is the check that this chapter's arithmetic and that one's agree.

And rows one to three say the elastic buffer is one or two octets, which at a 512-bit datapath is a fraction of a beat. The buffer is not beat-granular and cannot be: it inserts and deletes single octets in the interframe gap, which is Chapter 4.4 §3's discharge and is the reason the accumulation is bounded at all.

Row four is the boundary the standard's frame size is protecting. A protocol with unbounded frames would need an unbounded elastic buffer, and Chapter 5.1's maximum frame size is therefore doing work nobody credits it for — it bounds the PHY's buffer as well as the MAC's.

Now the thing a reader of Chapter 4.4 is entitled to object to. That chapter showed that no fixed depth suffices if the accumulation is never discharged: at 200 ppm and a gigabit link the running total grows without limit. Both statements are true and they are about different intervals.

Row four is the whole of the disagreement. Chapter 4.4's unbounded case is a MAC that closes the interframe gap, and Chapter 19.3 §6 is why this MAC does not: the gap is 9 to 12 octets held to a mean of 12, never zero. The bound and the unboundedness are the same statement seen from either side of that guarantee.

So the elastic buffer's depth is one octet, and its correctness depends on a property of the transmitter at the far end. That is an unusual shape and it is worth carrying: a buffer sized by a number this small is only safe because a different device is obeying a rule.

3. RTL 1 — The Elastic Buffer, Sized

// ---------------------------------------------------------------------
// fifod_pkg -- the MAC's three buffers and the parameters that size
// them. Sections 1 through 12.
//
// The point of collecting them in one package is that they look alike
// and are sized by unrelated arguments: a clock tolerance, a
// synchroniser's round trip, and a cable's length. A design that shares
// a depth parameter between them has asserted that they are the same
// problem.
// ---------------------------------------------------------------------
package fifod_pkg;

  localparam int DATA_B      = 64;                // octets per beat
  localparam int DATA_W      = DATA_B * 8;        // 512

  // Chapter 4.4: IEEE 802.3 specifies +/-100 ppm at each end.
  localparam int PPM_EACH    = 100;
  localparam int PPM_TOTAL   = 2 * PPM_EACH;      // 200

  // Chapter 5.1's maximum basic frame and Chapter 5.7's jumbo.
  localparam int MAX_FRAME   = 1518;
  localparam int JUMBO_FRAME = 9000;

  // Chapter 14.2 Section 5's dead-time components, in octets and
  // nanoseconds. The PAUSE frame is 84 octets on the wire including the
  // preamble and the gap; the sender may be one bit into a maximum
  // frame; the cable is what it is.
  localparam int PAUSE_OCTETS = 84;
  localparam int FINISH_OCTETS = MAX_FRAME;
  localparam int RATE_INDEP_OCTETS = PAUSE_OCTETS + FINISH_OCTETS;  // 1602
  localparam int CABLE_M      = 100;
  localparam int NS_PER_M     = 5;
  localparam int PROP_NS      = CABLE_M * NS_PER_M;                 // 500

  // Chapter 18.1 Section 8's worst-case memory stall. NOT a MAC property.
  localparam int STALL_NS     = 2000;

  typedef struct packed {
    logic [15:0] depth_beats;
    logic [15:0] almost_full_beats;
    logic [15:0] headroom_beats;
    logic [15:0] cdc_reserve_beats;
  } fifo_sizing_t;

endpackage

// ---------------------------------------------------------------------
// fifod_elastic_buffer -- one or two octets, and the centring is the
// design. Sections 2 and 3.
//
// This is Chapter 4.4's elastic FIFO with its depth DERIVED rather than
// parameterised, and with the derivation exposed so a reviewer can see
// what frame size it assumed. An elastic buffer sized for a 1518-octet
// maximum and deployed on a jumbo link is one octet short, and one octet
// short means an overrun per frame.
// ---------------------------------------------------------------------
module fifod_elastic_buffer
  import fifod_pkg::*;
#(
  parameter int FRAME_OCTETS = JUMBO_FRAME    // size for the LARGEST
) (
  input  logic              rx_clk,           // recovered from the wire
  input  logic              tx_clk,           // local
  input  logic              rst_n,

  input  logic              wr_en,
  input  logic [7:0]        wr_octet,
  input  logic              in_gap,           // Chapter 5.9's gap: discharge here

  output logic              rd_valid,
  output logic [7:0]        rd_octet,

  // Observability. Sections 15 and 16.
  output logic signed [7:0] offset_from_centre,
  output logic [31:0]       c_inserted,
  output logic [31:0]       c_deleted,
  output logic              gap_too_short,
  output logic              overrun
);

  // The derivation, at elaboration. 200 ppm over FRAME_OCTETS, rounded
  // up, plus one for the centring. For 1518 octets this is 0.3036 -> 1;
  // for 9000 it is 1.8 -> 2. Section 2's table.
  localparam int DRIFT_NUM   = FRAME_OCTETS * PPM_TOTAL;
  localparam int DRIFT_OCTETS = (DRIFT_NUM + 999_999) / 1_000_000;
  localparam int DEPTH       = 2 * (DRIFT_OCTETS + 1);   // both directions
  localparam int CENTRE      = DEPTH / 2;

  logic [7:0] mem [DEPTH];
  int unsigned wr_ptr, rd_ptr, occupancy;

  // AN ELASTIC BUFFER STARTS HALF FULL. Chapter 4.4 Section 6: it must
  // absorb drift in BOTH directions, so its correct idle state is the
  // centre and every decision is distance from centre, not fullness.
  // A design that starts it empty underruns on the first slow frame.
  always_ff @(posedge tx_clk or negedge rst_n) begin
    if (!rst_n) begin
      wr_ptr <= CENTRE; rd_ptr <= 0; occupancy <= CENTRE;
      c_inserted <= '0; c_deleted <= '0;
      gap_too_short <= 1'b0; overrun <= 1'b0;
    end else begin
      if (wr_en) begin
        if (occupancy == DEPTH) overrun <= 1'b1;   // data lost, not idle
        else begin
          mem[wr_ptr % DEPTH] <= wr_octet;
          wr_ptr <= wr_ptr + 1;
          occupancy <= occupancy + 1;
        end
      end

      // The discharge. Chapter 4.4 Section 3: idle is inserted or deleted
      // in the interframe gap and nowhere else, because inside a frame
      // there is nothing that may be added or removed.
      if (in_gap) begin
        if (occupancy > CENTRE) begin
          occupancy  <= occupancy - 1;             // delete an idle octet
          rd_ptr     <= rd_ptr + 1;
          c_deleted  <= c_deleted + 1;
        end else if (occupancy < CENTRE) begin
          occupancy  <= occupancy + 1;             // insert an idle octet
          c_inserted <= c_inserted + 1;
        end
      end

      // Chapter 5.9's floor is 9 octets and the drift per maximum frame
      // is under 2, so one gap is always enough. A gap shorter than the
      // accumulated drift means the far end is non-conformant and the
      // bound in Section 2 no longer holds.
      if (in_gap && (occupancy > CENTRE + DRIFT_OCTETS))
        gap_too_short <= 1'b1;
    end
  end

  assign offset_from_centre = 8'(signed'(int'(occupancy) - CENTRE));
  assign rd_octet  = mem[rd_ptr % DEPTH];
  assign rd_valid  = (occupancy != 0);

endmodule

Classification: a datapath buffer whose depth is four octets and whose correctness rests on the far end's behaviour.

What it teaches: that FRAME_OCTETS is the parameter that matters and it is the one most likely to be left at its default. A buffer sized for Chapter 5.1's 1 518 octets needs one octet of drift; the same port carrying Chapter 5.7's 9 000-octet jumbo frames needs two. One octet short is an overrun on every frame, and the symptom — a steady error rate that appears only when jumbo frames are enabled — reads as an MTU negotiation problem rather than a buffer depth.

And it teaches that the counters are the diagnosis and the offset is the health. c_inserted and c_deleted should be within a factor of a few of each other over a long run if the two clocks are close, and strongly one-sided if they are not — the ratio reads back the sign of the frequency error. A buffer with c_deleted at zero and c_inserted climbing has a local clock that is fast, which is a board problem and not a design one.

Deliberately simplified: the depth calculation assumes the drift rounds up cleanly and adds one for centring, which is correct and slightly generous; a production version accounts for the read and write pointers' own granularity. overrun is sticky and there is no recovery path, because an elastic buffer that has overrun has lost frame data — Chapter 4.4 §14's asymmetry, where underrun emits idle and is harmless and overrun destroys. And in_gap is taken as an input rather than derived, so the block trusts the framing to tell it when discharge is legal.

Production implication: gap_too_short is the one output that reports the far end rather than this design. Chapter 5.9's floor is 9 octets and Section 2's worst drift over a jumbo frame is 1.8, so a conformant partner leaves five times the room needed. A port asserting gap_too_short has a partner that is closing the gap — which is exactly Chapter 4.4 §2's unbounded case arriving as a field failure, and the correct response is to report the partner rather than to deepen the buffer.

4. The Rate Mismatch That Never Averages Out

The elastic buffer handled the wire's clock. The datapath FIFO faces a different mismatch and it is the one that catches people out, because the two sides are both nominally fast enough.

The MAC's receive side runs at 195.3125 MHz × 512 bits — exactly 100 Gb/s — and the system side runs at 250 MHz × 512 bits, which is 128 Gb/s. The system is 28% faster. It is still possible for the FIFO to fill and stay full.

So the tolerance does not matter here, and saying so precisely is the point: at a 28% margin, 200 ppm is noise. The FIFO does not fill because the consumer is slow on average; it fills because the consumer stops.

That is the difference between this buffer and the elastic one in one line.

Row one is why the two arguments cannot be merged. A permanent 200 ppm difference needs a discharge mechanism and a tiny buffer; an intermittent total stop needs no discharge mechanism and a large buffer. A designer who has internalised the first and meets the second reaches for a rate argument and gets a depth of one octet, which is Chapter 18.1 §8's point arriving from the other direction.

And there is a case where the rate argument does bind, which is worth isolating because it is the one this chapter's rejected property is about.

Suppose the system side were nominally equal to the MAC side — both 512 bits at 195.3125 MHz, as they would be if one PLL fed both. Then the tolerance is the whole margin, and a FIFO 512 beats deep drains or fills at 200 ppm:

Row four is the chapter's verification trap and Section 20's rejected property. The failure is real, it is deterministic, and no simulation of ordinary length reaches it.

5. RTL 2 — The Rate Mismatch Tracker

// ---------------------------------------------------------------------
// rate_mismatch_tracker -- measure the two sides' actual rates and
// report the drift, in a form that reaches a threshold in bounded time.
// Sections 4, 5 and 20.
//
// The block exists because the failure it detects takes 13.107 ms to
// appear at 200 ppm across a 512-beat FIFO, and a simulation does not
// run that long. Measuring the RATE reaches a verdict in microseconds;
// waiting for the OCCUPANCY to reach a bound does not. Section 20's
// rejected property is the difference.
// ---------------------------------------------------------------------
module rate_mismatch_tracker
  import fifod_pkg::*;
#(
  parameter int WINDOW_BEATS = 1 << 16,          // 65 536 beats
  parameter int PPM_BUDGET   = PPM_TOTAL         // 200
) (
  input  logic              wr_clk,
  input  logic              rd_clk,
  input  logic              rst_n,

  input  logic              wr_en,
  input  logic              rd_en,

  output logic [31:0]       c_writes,
  output logic [31:0]       c_reads,
  output logic signed [31:0] drift_ppm,
  output logic              drift_exceeds_budget,
  output logic              window_complete,
  output logic [31:0]       c_windows
);

  logic [31:0] w_cnt, r_cnt_sync1, r_cnt_sync2, r_cnt;
  logic [31:0] w_snap, r_snap;

  // The read counter crosses to the write domain. It is a COUNTER, not a
  // pointer, and it is sampled only at a window boundary when both sides
  // are quiescent enough that a skew of a few counts is below the
  // measurement's resolution -- 65 536 beats at 200 ppm is 13.1 counts
  // of drift, so a two-count sampling uncertainty is 15% of the signal
  // and the window has to be long for that reason rather than for any
  // reason about the clocks.
  always_ff @(posedge rd_clk or negedge rst_n) begin
    if (!rst_n) r_cnt <= '0;
    else if (rd_en) r_cnt <= r_cnt + 1;
  end

  always_ff @(posedge wr_clk or negedge rst_n) begin
    if (!rst_n) begin
      w_cnt <= '0; r_cnt_sync1 <= '0; r_cnt_sync2 <= '0;
      w_snap <= '0; r_snap <= '0;
      drift_ppm <= '0; drift_exceeds_budget <= 1'b0;
      window_complete <= 1'b0; c_windows <= '0;
    end else begin
      if (wr_en) w_cnt <= w_cnt + 1;
      r_cnt_sync1 <= r_cnt;
      r_cnt_sync2 <= r_cnt_sync1;

      window_complete <= 1'b0;

      if (w_cnt - w_snap >= WINDOW_BEATS) begin
        // Drift in parts per million over the window.
        drift_ppm <= 32'signed'(((32'signed'(w_cnt - w_snap) -
                                  32'signed'(r_cnt_sync2 - r_snap)) * 1_000_000) /
                                 32'signed'(WINDOW_BEATS));
        w_snap <= w_cnt;
        r_snap <= r_cnt_sync2;
        window_complete <= 1'b1;
        c_windows <= c_windows + 1;
      end

      // A one-sided drift beyond the budget means the FIFO will fill or
      // empty, and WHEN it does is a division rather than an experiment.
      if (window_complete &&
          ((drift_ppm > 32'signed'(PPM_BUDGET)) ||
           (drift_ppm < -32'signed'(PPM_BUDGET))))
        drift_exceeds_budget <= 1'b1;
    end
  end

  assign c_writes = w_cnt;
  assign c_reads  = r_cnt_sync2;

endmodule

Classification: a measurement block that exists to convert an unreachable property into a reachable one.

What it teaches: that a rate is observable in a window and an overflow is not. The FIFO overflows after 2 560 000 beats at 200 ppm; the rate that causes it is measurable in 65 536. That is a factor of 39 in simulation time and it is the whole justification for the block: the same fact, observed through a derivative rather than an integral.

And it teaches why the window has to be long. At 200 ppm, 65 536 beats accumulate 13.1 beats of difference, and the two-flop synchroniser on r_cnt introduces up to two counts of sampling uncertainty — 15% of the signal. A 4 096-beat window accumulates 0.8 beats and the uncertainty exceeds the measurement. The window length is set by the measurement's resolution, not by the clocks, which is an unusual reason for a window length and is worth recognising when choosing one.

Deliberately simplified: r_cnt crosses as a plain binary counter through two flops, which is a multi-bit crossing and is not safe in general — a production version uses a gray code or a handshake, and this one is defensible only because the value is sampled at a window boundary and a few counts of error are within the stated resolution. drift_ppm uses a signed divide by a power of two, which synthesises to a shift and is why WINDOW_BEATS is written as 1 << 16. And drift_exceeds_budget is sticky with no clear.

Production implication: the sign of drift_ppm is the diagnosis and its magnitude is the deadline. Positive means writes outrun reads and the FIFO will overflow; negative means it will underflow — and dividing the remaining depth by the drift gives the time until it happens, which is a number an integrator can act on. A port reporting +200 ppm with 389 beats of margin has 1 945 000 beats, or 9.96 ms, before the first drop. That is a maintenance window rather than an incident, and it is only visible because the rate was measured instead of the overflow being waited for.

6. The Crossing, and What a Synchroniser Costs in Depth

Chapter 18.1 §9 built the asynchronous FIFO with gray-coded pointers and two-flop synchronisers and called its full and empty flags "deliberately pessimistic." This section puts a number on the pessimism.

A synchronised pointer is old. The write side's view of the read pointer has passed through two flops in the write domain; the read side's view of the write pointer has passed through two in the read domain. Neither side ever sees the other's current value.

Chapter 18.1 §8 quoted 24 ns for a 125 MHz to 250 MHz pair and this is the same calculation at this chapter's frequencies.

Eighteen nanoseconds at 195.3125 MHz is 3.56 beats, so four. The write side must stop four beats before the FIFO is actually full, because by the time it sees a full flag the read side has moved on and by the time it stops it has written more.

Row one against row two is the section's result and it inverts where attention usually goes. The clock-domain crossing — the part of the design that gets the careful review, the gray codes, the synchroniser depth arguments — costs 3.25% of what the flow-control headroom costs. The CDC is hard to get right and cheap to pay for; the headroom is easy to compute and expensive.

And the CDC reserve does not scale with the line rate. It is a fixed number of clocks, so at 10 Gb/s with the same clock frequencies it is the same four beats — against a FIFO that is a seventh the size, where it is 7.8% rather than 0.78%. The thing that seems negligible at 100 Gb/s is ten times more significant at 10, which is the opposite of how every other term in this chapter behaves.

Row one is a real design problem. At 1 Gb/s a 1.9 KiB FIFO is 30 beats and four of them are synchroniser reserve; a design that also wants 123 octets of headroom and a 2 µs stall buffer is allocating a third of a small FIFO to overheads. The usual answer at low rates is a narrower datapath — 64 bits rather than 512 — which makes the beats smaller and the reserve proportionally cheaper, and is one of the reasons a 1 Gb/s MAC is not a 100 Gb/s MAC with a slower clock.

7. RTL 3 — The Crossing FIFO

// ---------------------------------------------------------------------
// async_fifo_core -- Chapter 18.1 Section 9's asynchronous FIFO, sized from
// this chapter's arguments and with the reserve made explicit.
// Sections 6 and 7.
//
// Gray-coded pointers and two-flop synchronisers, unchanged. What is
// added is CDC_RESERVE: the write side stops four beats before the FIFO
// is full, because a synchronised pointer is 18.24 ns old and 18.24 ns
// is 3.56 beats at 195.3125 MHz. Section 6.
//
// The reserve is 0.78% of a 100 Gb/s FIFO and 13.2% of a 1 Gb/s one,
// which is the only term in this chapter that gets RELATIVELY more
// expensive as the line rate falls.
// ---------------------------------------------------------------------
module async_fifo_core
  import fifod_pkg::*;
#(
  parameter int AW           = 9,             // 512 beats
  parameter int CDC_RESERVE  = 4
) (
  input  logic                 wr_clk,
  input  logic                 wr_rst_n,
  input  logic                 wr_en,
  input  logic [DATA_W-1:0]    wr_data,
  output logic                 wr_full,
  output logic                 wr_almost_full,
  output logic [AW:0]          wr_occupancy,

  input  logic                 rd_clk,
  input  logic                 rd_rst_n,
  input  logic                 rd_en,
  output logic [DATA_W-1:0]    rd_data,
  output logic                 rd_empty,

  input  logic [AW:0]          almost_full_level,   // Section 9 computes it

  // Observability. Sections 15 and 16.
  output logic [31:0]          c_overflow,
  output logic [31:0]          c_underflow,
  output logic [AW:0]          high_water_mark
);

  localparam int DEPTH = 1 << AW;

  logic [DATA_W-1:0] mem [DEPTH];
  logic [AW:0] w_bin, w_gray, w_gray_r1, w_gray_r2;
  logic [AW:0] r_bin, r_gray, r_gray_w1, r_gray_w2;
  logic [AW:0] r_bin_in_w;

  function automatic logic [AW:0] to_gray(input logic [AW:0] b);
    to_gray = b ^ (b >> 1);
  endfunction

  function automatic logic [AW:0] from_gray(input logic [AW:0] g);
    logic [AW:0] b;
    begin
      b = g;
      for (int i = AW; i > 0; i--) b[i-1] = b[i] ^ g[i-1];
      from_gray = b;
    end
  endfunction

  // ---- write domain -------------------------------------------------
  always_ff @(posedge wr_clk or negedge wr_rst_n) begin
    if (!wr_rst_n) begin
      w_bin <= '0; w_gray <= '0;
      r_gray_w1 <= '0; r_gray_w2 <= '0;
      c_overflow <= '0; high_water_mark <= '0;
    end else begin
      r_gray_w1 <= r_gray;
      r_gray_w2 <= r_gray_w1;

      if (wr_en && !wr_full) begin
        mem[w_bin[AW-1:0]] <= wr_data;
        w_bin  <= w_bin + 1;
        w_gray <= to_gray(w_bin + 1);
      end else if (wr_en && wr_full) begin
        // Chapter 19.1 Section 11: the receive path has no backpressure, so
        // this is a DROP and not a stall. Counting it is the only thing
        // this block can do about it.
        c_overflow <= c_overflow + 1;
      end

      if (wr_occupancy > high_water_mark) high_water_mark <= wr_occupancy;
    end
  end

  assign r_bin_in_w   = from_gray(r_gray_w2);
  assign wr_occupancy = w_bin - r_bin_in_w;

  // FULL is pessimistic by construction: the read pointer we compare
  // against is 18.24 ns old, so the true occupancy is at most this and
  // possibly less. Pessimism in this direction is safe.
  assign wr_full = (wr_occupancy >= (AW+1)'(DEPTH - CDC_RESERVE));

  // ALMOST FULL is where flow control is asserted. Its level comes from
  // Section 10 and is a property of the CABLE, not of this block.
  assign wr_almost_full = (wr_occupancy >= almost_full_level);

  // ---- read domain --------------------------------------------------
  always_ff @(posedge rd_clk or negedge rd_rst_n) begin
    if (!rd_rst_n) begin
      r_bin <= '0; r_gray <= '0;
      w_gray_r1 <= '0; w_gray_r2 <= '0;
      c_underflow <= '0;
    end else begin
      w_gray_r1 <= w_gray;
      w_gray_r2 <= w_gray_r1;

      if (rd_en && !rd_empty) begin
        r_bin  <= r_bin + 1;
        r_gray <= to_gray(r_bin + 1);
      end else if (rd_en && rd_empty) begin
        c_underflow <= c_underflow + 1;
      end
    end
  end

  assign rd_empty = (r_gray == w_gray_r2);
  assign rd_data  = mem[r_bin[AW-1:0]];

endmodule

Classification: the chapter's structural block, and the one with the least new content in it.

What it teaches: that CDC_RESERVE is the only line in this module that this chapter adds to Chapter 18.1 §9's version, and that it is the line a review will skip. The gray codes get the attention; the four beats of reserve are what stop an overflow that the gray codes cannot prevent.

And it teaches that wr_full and wr_almost_full come from unrelated arguments and must not share a parameter. wr_full is a synchroniser property — 18.24 ns, four beats, fixed. wr_almost_full is a cable property — 500 ns of propagation, 123 beats, and it changes if the link is re-terminated. A design with one threshold has conflated a flop count with a cable length.

Deliberately simplified: from_gray is a combinational loop over AW+1 bits in the write domain's timing path, which at 195.3125 MHz and 10 bits is fine and at a wider pointer is not; production designs keep a binary shadow of the synchronised pointer instead. high_water_mark has no clear, which is deliberate — Chapter 18.1 §15's argument that a counter clearing on read destroys the evidence a later reader needs. And c_overflow counts beats rather than frames, so a long frame lost to an overflow shows as many counts.

Production implication: high_water_mark against almost_full_level is the number that says whether flow control is working. A port whose high-water mark never reaches the almost-full level has never needed to pause, and its 123 beats of headroom are pure cost. A port whose high-water mark sits at DEPTH − CDC_RESERVE is dropping, and c_overflow says how much. The interesting case is the one in between — a high-water mark above the almost-full level and below full — which means PAUSE was issued and worked, and is the only direct evidence a design ever gets that Chapter 14.2's loop closed in time.

8. Where the Almost-Full Threshold Comes From

Not from a fraction of the depth. From the length of the cable.

Chapter 14.2 §8 said this as a design rule — "the watermark's position is not a tuning parameter" — and this section derives the number at 100 Gb/s and shows what it is made of.

Between deciding to pause and traffic stopping, four things happen and the sender transmits through all of them. Chapter 14.2 §5's decomposition:

And the headroom is the dead time times the line rate:

headroom_octets = dead_time × R
                = (1602 octets / R + t_prop) × R
                = 1602 + R × t_prop / 8

That closed form is the section's result and the track has not written it down before. The headroom has a rate-independent term of 1 602 octets — the partner's maximum frame plus the PAUSE frame's own 84 — and a rate-proportional term that is the cable.

The closed form reproduces all four published values, which is the check that the decomposition is the same one Chapter 14.2 used rather than a plausible-looking substitute.

Now read the two columns against each other, because they cross.

At 1 Gb/s the headroom is almost entirely the partner's frame. At 100 Gb/s it is almost entirely the cable. The crossover is at about 25 Gb/s, and it matters because the two terms respond to different actions. Reducing the partner's MTU helps at 1 Gb/s and does nothing at 100; shortening the cable helps at 100 and does nothing at 1.

And at 100 Gb/s the headroom is 7 852 octets, which is 122.7 beats — so 123.

9. RTL 4 — The Almost-Full Threshold

// ---------------------------------------------------------------------
// almost_full_threshold -- compute the watermark from the link, not from
// a fraction of the depth. Sections 8, 9 and 11.
//
// Chapter 14.2 Section 8 made this argument with cells and a 1 Gb/s default.
// This is the same computation at beat granularity with the closed form
// of Section 8 made explicit:
//
//   headroom_octets = RATE_INDEP_OCTETS + R * t_prop / 8
//
// where RATE_INDEP_OCTETS is 1 602 -- the partner's maximum frame plus
// the PAUSE frame's own 84 octets on the wire -- and the second term is
// the cable. At 100 Gb/s over 100 m that is 7 852 octets, 122.7 beats,
// so 123.
// ---------------------------------------------------------------------
module almost_full_threshold
  import fifod_pkg::*;
#(
  parameter int LINK_MBPS  = 100_000,
  parameter int CABLE_M    = 100,
  parameter int PARTNER_MTU = MAX_FRAME,
  parameter int DEPTH_BEATS = 512,
  parameter int CDC_RESERVE = 4
) (
  input  logic              clk,
  input  logic              rst_n,

  input  logic [15:0]       tx_backlog_octets,   // ahead of our PAUSE

  output logic [19:0]       dead_time_ns,
  output logic [19:0]       headroom_octets,
  output logic [15:0]       headroom_beats,
  output logic [15:0]       almost_full_level,
  output logic [15:0]       cable_share_pct,
  output logic              threshold_infeasible,
  output logic              threshold_is_trivial
);

  localparam int PROP_NS_LOCAL = CABLE_M * NS_PER_M;
  localparam int FIXED_OCTETS  = PAUSE_OCTETS + PARTNER_MTU;

  // The cable term, in octets: t_prop in ns times the rate in Mb/s,
  // divided by 8000 to reach octets.
  localparam int CABLE_OCTETS  = (PROP_NS_LOCAL * LINK_MBPS) / 8000;

  logic [19:0] backlog_octets;
  assign backlog_octets = 20'(tx_backlog_octets);

  assign headroom_octets = 20'(FIXED_OCTETS) + 20'(CABLE_OCTETS) + backlog_octets;
  assign dead_time_ns    = (headroom_octets * 20'd8000) / 20'(LINK_MBPS);

  assign headroom_beats  = 16'((headroom_octets + 20'(DATA_B) - 20'd1) /
                                20'(DATA_B));

  // The watermark is the depth MINUS the headroom MINUS the synchroniser
  // reserve. A design that sets it at 75% of depth has assumed a cable
  // length and a partner MTU without writing either down. At the default
  // parameters this lands at 512 - 123 - 4 = 385.
  assign almost_full_level = (16'(DEPTH_BEATS) > headroom_beats + 16'(CDC_RESERVE))
                           ? (16'(DEPTH_BEATS) - headroom_beats - 16'(CDC_RESERVE))
                           : 16'd0;

  assign cable_share_pct = 16'((20'(CABLE_OCTETS) * 20'd100) / headroom_octets);

  // A FIFO too small to pause from. Chapter 14.2 Section 8's
  // watermark_infeasible, at beat granularity: if the headroom exceeds
  // the depth there is no level at which a PAUSE arrives in time, and
  // lossless operation is impossible on this link with this FIFO.
  assign threshold_infeasible = (headroom_beats + 16'(CDC_RESERVE)) >=
                                16'(DEPTH_BEATS);

  // And the opposite failure, which is quieter: a watermark so high that
  // the FIFO is effectively never paused from, because the depth dwarfs
  // the headroom and the consumer stalls long before the watermark is
  // reached. Not an error, but it means the headroom is unearned area.
  assign threshold_is_trivial = (headroom_beats * 16'd10) <
                                16'(DEPTH_BEATS);

endmodule

Classification: a parameter calculator, and the only block in this chapter whose output is a design review rather than a signal.

What it teaches: that CABLE_M and PARTNER_MTU are the two parameters that set the watermark, and neither is a property of this chip. A MAC shipped with CABLE_M at 100 and deployed on a 2 km link has a watermark 16 times too high, and the failure is a drop rate under congestion on long links only. Chapter 14.2 §8 named the same hazard in cells; this version names it in beats and adds the synchroniser reserve.

And it teaches that threshold_is_trivial is the check nobody writes. threshold_infeasible is obvious — the headroom does not fit — and gets caught at elaboration. The opposite case is that the headroom is 2% of the depth, which is not an error and means the design reserved area for a mechanism that the depth already covered. At 100 Gb/s the headroom is 24% of a 32 KiB FIFO and is earning its area; at 400 Gb/s against the same 2 µs stall it would be 6%, and a reviewer should be told.

Deliberately simplified: tx_backlog_octets is added as a runtime input and then never sampled or held, so the outputs move when the backlog does — a production design latches the worst case, because Chapter 14.2 §11's point is that a PAUSE queued behind traffic arrives late by exactly the amount that matters. The cable term uses 5 ns/m for both copper and fibre, which is Chapter 14.2's own approximation. And PARTNER_MTU defaults to 1 518 rather than to the jumbo case, which is the more dangerous default of the two.

Production implication: cable_share_pct is the number that tells an integrator which lever to pull. A port reporting 79.6% is cable-dominated — shortening the link or accepting a lossy link are the options, and reducing the partner's MTU does almost nothing. A port reporting 3.8% is frame-dominated, at 1 Gb/s, where the partner's MTU is 96.2% of the headroom and reducing it is the whole of the available saving. The same mechanism, and opposite advice at the two ends of the rate range.

10. The Crossover at 8.54 Gb/s

Chapter 18.1 §18 published a table in which the flow-control headroom is 87% of the receive FIFO at 1 Gb/s and 23.9% at 100, and observed that the shares cross. This section finds where, in closed form.

The FIFO has two terms.

stall_octets    = R × t_stall / 8
headroom_octets = F + R × t_prop / 8        where F = 1 602 octets

Set them equal.

R × t_stall / 8 = F + R × t_prop / 8
R × (t_stall − t_prop) / 8 = F
R* = 8F / (t_stall − t_prop)

With Chapter 18.1 §8's 2 µs stall and 100 m of cable:

R* = 8 × 1 602 / (2 000 ns − 500 ns) = 12 816 / 1 500 ns = 8.544 Gb/s

Below 8.544 Gb/s the flow-control headroom dominates the FIFO; above it, the memory stall does.

Rows one and five are Chapter 18.1 §18's published 87.0% and 23.9% and they reproduce, which is the check.

Three things follow from the closed form and none of them is visible in the table.

First, R* does not depend on the datapath width, the clock, or anything else in the MAC. It is 1 602 octets divided by the difference between two times that belong to other people — the memory system's stall and the cable's propagation. A MAC designer cannot move the crossover.

Second, the crossover moves with the stall, and the stall is the term a system architect thinks is theirs to choose.

Row one is not a rounding artefact. If the memory system's worst stall is shorter than the cable's propagation delay, the headroom term exceeds the stall term at every line rate — there is no crossover — and the FIFO is a flow-control buffer that happens also to cover the memory system. A tighter memory subsystem does not make the FIFO smaller; past a point it stops being the thing the FIFO is for.

And the fixed term is the partner's maximum frame, so the crossover moves with the partner's MTU as well.

Row three changes where a 25 Gb/s port sits. With a 1 518-octet partner, 25 Gb/s is above the crossover and the FIFO is a stall buffer — 6.10 KiB against 3.09 of headroom. With a jumbo partner it is below, and the headroom becomes 10.40 KiB against the same 6.10 — a 79.5% larger FIFO from a decision made entirely at the other end of the cable.

Which is the third of the three external owners arriving. The stall is the memory team's, the cable is the facilities team's, and the partner's MTU belongs to whoever configures the switch this port is plugged into — and of the three, it is the one most likely to change after the chip ships.

Third, and this is the one worth carrying: on a long link there is no crossover at any rate.

On a 2 km link the propagation alone is 10 µs against a 2 µs stall, so the headroom dominates at 1 Gb/s and at 100 Gb/s and everywhere between. At 100 Gb/s over 2 km the headroom is 123.63 KiB against a 24.41 KiB stall buffer — a 148 KiB receive FIFO, of which 82% is the cable alone.

Which is Chapter 18.1 §18's uncomfortable conclusion sharpened: the receive FIFO's size is decided by the network's physical layout, and the decision is usually made by someone who is not in the conversation.

11. RTL 5 — The Depth Calculator

// ---------------------------------------------------------------------
// fifo_depth_calc -- the whole receive FIFO, from three arguments that
// do not talk to each other. Sections 6, 8, 10 and 11.
//
// depth = stall term + headroom term + synchroniser reserve
//
//   stall term  = R * t_stall / 8          -- the memory system's
//   headroom    = 1602 + R * t_prop / 8    -- the cable's and the partner's
//   reserve     = ceil(2/f_wr + 2/f_rd)    -- the synchroniser's
//
// At 100 Gb/s, 2 us and 100 m: 391 + 123 + 4 = 518 -> 512 beats
// after rounding the depth to a power of two and accepting a slightly
// tighter watermark, which Section 11 says is the right trade.
// ---------------------------------------------------------------------
module fifo_depth_calc
  import fifod_pkg::*;
#(
  parameter int LINK_MBPS   = 100_000,
  parameter int STALL_NS_P  = STALL_NS,
  parameter int CABLE_M     = 100,
  parameter int PARTNER_MTU = MAX_FRAME,
  parameter int WR_MHZ      = 195,
  parameter int RD_MHZ      = 250
) (
  output fifo_sizing_t sizing,
  output logic [31:0]  depth_octets,
  output logic [31:0]  crossover_mbps,
  output logic         headroom_dominates,
  output logic         no_crossover
);

  localparam int STALL_OCTETS = (STALL_NS_P * LINK_MBPS) / 8000;
  localparam int PROP_NS_L    = CABLE_M * NS_PER_M;
  localparam int CABLE_OCTETS = (PROP_NS_L * LINK_MBPS) / 8000;
  localparam int FIXED_OCTETS = PAUSE_OCTETS + PARTNER_MTU;
  localparam int HEAD_OCTETS  = FIXED_OCTETS + CABLE_OCTETS;

  // The synchroniser round trip, in write-domain beats. Two flops each
  // way -- Section 6 and Chapter 18.1 Section 8's 24 ns at 125/250 MHz.
  localparam int RT_PS      = (2 * 1_000_000) / WR_MHZ + (2 * 1_000_000) / RD_MHZ;
  localparam int RESERVE    = ((RT_PS * WR_MHZ) + 999_999) / 1_000_000;

  localparam int STALL_BEATS = (STALL_OCTETS + DATA_B - 1) / DATA_B;
  localparam int HEAD_BEATS  = (HEAD_OCTETS  + DATA_B - 1) / DATA_B;
  localparam int RAW_BEATS   = STALL_BEATS + HEAD_BEATS + RESERVE;

  // Round DOWN to a power of two rather than up. Section 11: rounding up
  // doubles the SRAM to buy 1.6% more margin, and the 1.6% comes out of
  // the stall term, which is the term with the least confidence in it.
  function automatic int pow2_floor(input int v);
    int p;
    begin
      p = 1;
      while ((p << 1) <= v) p = p << 1;
      pow2_floor = p;
    end
  endfunction

  localparam int DEPTH_BEATS = pow2_floor(RAW_BEATS);

  // Section 10's closed form: R* = 8F / (t_stall - t_prop). If the
  // propagation exceeds the stall there is no crossover at any rate and
  // the headroom dominates everywhere.
  assign no_crossover     = (STALL_NS_P <= PROP_NS_L);
  assign crossover_mbps   = no_crossover ? 32'd0
                          : 32'((8 * FIXED_OCTETS * 1000) /
                                (STALL_NS_P - PROP_NS_L));
  assign headroom_dominates = no_crossover ||
                              (32'(LINK_MBPS) < crossover_mbps);

  assign sizing.depth_beats       = 16'(DEPTH_BEATS);
  assign sizing.headroom_beats    = 16'(HEAD_BEATS);
  assign sizing.cdc_reserve_beats = 16'(RESERVE);
  assign sizing.almost_full_beats = 16'(DEPTH_BEATS - HEAD_BEATS - RESERVE);
  assign depth_octets             = 32'(DEPTH_BEATS * DATA_B);

endmodule

Classification: an elaboration-time calculator whose value is that its three inputs come from three different people.

What it teaches: that the depth is a sum of three terms owned by three parties and the MAC designer owns none of them. STALL_NS_P belongs to the memory subsystem, CABLE_M and PARTNER_MTU to the network, and WR_MHZ and RD_MHZ to the clocking. A block that takes a DEPTH parameter instead of these five has hidden every one of those conversations, which is Chapter 18.1 §8's "a system property nobody writes down" made structural.

And it teaches that rounding down is the right choice, which is not the obvious one. The raw figure at 100 Gb/s is 518 beats; rounding up gives 1 024 and doubles the SRAM to buy 98% more margin, and rounding down gives 512 and loses 6 beats — 1.16%. The 1.16% comes out of the stall term, which is the least certain of the three: the memory system's worst stall is an estimate, and an estimate accurate to 1% does not exist. Spending 16 KiB of SRAM to protect a 1% error in a number that is itself a guess is the trade a power-of-two reflex makes silently.

Deliberately simplified: pow2_floor forces a power-of-two depth because the pointers are binary and gray-coded, which a memory compiler does not actually require — a 520-beat SRAM is buildable and the pointer logic becomes non-trivial. RT_PS is computed in integer picoseconds from megahertz and is off by rounding at odd frequencies. And the block computes crossover_mbps for a reviewer's benefit and no logic consumes it, which is deliberate: it is a number a design review should see and a design should not depend on.

Production implication: headroom_dominates and no_crossover are the two bits that tell an integrator what kind of FIFO this is. A 1 Gb/s port with headroom_dominates set has a FIFO that exists for flow control and incidentally survives memory stalls; a 100 Gb/s port over 100 m has the reverse. And a port with no_crossover set — any link past about a kilometre — has a FIFO whose size is a property of the fibre plant, which is a conversation with a facilities team rather than an architecture one.

12. Sizing the Transmit FIFO Against 19.3's Commit Point

The receive FIFO is sized against a consumer that stops. The transmit FIFO is sized against a producer that stops, and Chapter 19.3 §11 already fixed the answer without naming a depth.

Chapter 19.3 §5 established the rule: a transmit stage may stall before commit and not after. Chapter 18.4 §9 then derived where commit must sit at 100 Gb/s: cut-through transmit is feasible only to 40.48 Gb/s on a maximum-size frame and 1.707 Gb/s on a minimum-size one against a 300 ns memory stall, so at 100 Gb/s the commit point is at the end of the frame and the assembler is store-and-forward.

Which fixes the transmit FIFO's threshold exactly.

And the depth follows from the threshold rather than the other way round. A FIFO whose depth equals its commit threshold can hold exactly one frame and cannot begin receiving the next while transmitting this one, which costs a full frame time of pipeline every frame. Two frames' worth is the minimum that keeps the wire busy:

But Chapter 18.1 §18 published 9 KiB for the transmit FIFO, not 16. The difference is the fourth row's assumption, and resolving it is worth doing carefully rather than assuming one figure is wrong.

Chapter 18.1's 9 KiB is one jumbo frame plus margin — 144 beats — and it is correct for a design that does not overlap frames in the FIFO. The overlap is not free elsewhere either: a second frame in the transmit FIFO is a second frame whose data has been fetched from host memory and is now committed to being sent, and Chapter 18.4 §7's ring walker has to have fetched it.

Row one is idle only if the fetch does not keep up, and Chapter 18.4 §7's fill threshold exists precisely to make sure it does. So 9 KiB is right when the DMA is keeping ahead and 16 KiB is insurance against the case where it is not — and the honest statement is that this is a choice, not a derivation, which is the opposite of the receive side.

That asymmetry is the section's point. The receive FIFO's depth is forced by three external numbers; the transmit FIFO's is chosen, because the producer is inside the chip and can be made to keep up. Receive has no backpressure and must absorb; transmit has backpressure all the way to the descriptor ring and can push back. Chapter 19.1 §11's rule, arriving as a difference in how two buffers are sized.

13. What the FIFOs Must Never Do

Six prohibitions across three buffers, and two of them are about a buffer doing another buffer's job.

Row two is the one that costs the most engineering time in the field, and the mechanism is worth stating. A receive FIFO that overflows drops beats out of the middle of a frame. The frame that reaches the CRC engine is structurally intact and arithmetically wrong, so Chapter 19.4 §10 classifies it RES_BAD and the port's CRC error counter increments. An operator reading that counter concludes the cable is bad. The only thing that separates the two stories is c_overflow, which is why Section 15 puts it in the structural group and why a design that omits it has made a channel problem indistinguishable from a buffer problem.

Row four is the chapter's most specific prohibition and it is easy to violate accidentally. A parameterised FIFO with a single ALMOST_FULL generic invites a designer to set it from the same reasoning as FULL — a synchroniser argument, four beats. Four beats of headroom at 100 Gb/s is 256 octets against the 7 852 the cable needs, so the PAUSE arrives 7 596 octets late and the FIFO overflows every time flow control is exercised. The failure appears only under congestion, which is the condition least represented in a regression.

Row five is the verification one and it is unusual in being about the testbench. The property "occupancy never exceeds the depth" is true in every simulation of ordinary length whether the design is right or wrong, because at 200 ppm the accumulator needs 2 560 000 beats to violate it. Section 20 refuses it; Section 5's tracker is the constructive replacement.

Row six closes the loop on Section 1. The two errors are opposite and both are reached by the same reasoning — "the clocks differ, therefore size the buffer against the difference" — and the reason it fails is that the datapath FIFO's mismatch is not a rate difference at all. It is a consumer that stops.

14. RTL 6 — The Overflow Guard

// ---------------------------------------------------------------------
// overflow_guard -- make a drop visible, attributable and bounded.
// Sections 13 and 14.
//
// Chapter 19.1 Section 11: the receive path has no backpressure, so an
// overflow is a DROP. This block cannot prevent it. What it can do is
// make sure the drop is counted, attributed to a frame rather than to a
// beat, and distinguishable from Chapter 19.4's CRC errors -- which is
// the whole difference between a two-hour diagnosis and a two-week one.
// ---------------------------------------------------------------------
module overflow_guard
  import fifod_pkg::*;
(
  input  logic              clk,
  input  logic              rst_n,

  input  logic              beat_valid,
  input  logic              beat_sof,
  input  logic              beat_eof,
  input  logic              fifo_full,
  input  logic              almost_full,

  output logic              drop_beat,
  output logic              frame_truncated,

  // Observability. Sections 15 and 16.
  output logic [31:0]       c_beats_dropped,
  output logic [31:0]       c_frames_truncated,
  output logic [31:0]       c_frames_dropped_whole,
  output logic [31:0]       c_almost_full_events,
  output logic [31:0]       c_almost_full_beats,
  output logic              mid_frame_drop
);

  logic dropping_q, af_q;

  assign drop_beat = beat_valid && fifo_full;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      dropping_q <= 1'b0; af_q <= 1'b0;
      frame_truncated <= 1'b0; mid_frame_drop <= 1'b0;
      c_beats_dropped <= '0; c_frames_truncated <= '0;
      c_frames_dropped_whole <= '0;
      c_almost_full_events <= '0; c_almost_full_beats <= '0;
    end else begin
      frame_truncated <= 1'b0;

      if (beat_valid && fifo_full) begin
        c_beats_dropped <= c_beats_dropped + 1;

        // A drop at a frame's FIRST beat loses the whole frame and
        // leaves nothing downstream. A drop in the MIDDLE leaves a
        // partial frame that downstream will complete with a wrong
        // check value -- Chapter 19.4 Section 10 reports RES_BAD and
        // the operator reads it as a cable fault. The two cases must be
        // counted separately or they cannot be told apart at all.
        if (beat_sof) c_frames_dropped_whole <= c_frames_dropped_whole + 1;
        else begin
          dropping_q     <= 1'b1;
          mid_frame_drop <= 1'b1;
        end
      end

      if (beat_valid && beat_eof && dropping_q) begin
        frame_truncated    <= 1'b1;
        c_frames_truncated <= c_frames_truncated + 1;
        dropping_q         <= 1'b0;
      end

      // Almost-full is not an error and its statistics are the evidence
      // that flow control is doing anything. Section 7: a port whose
      // high-water mark never reaches this level has 123 beats of
      // reserved SRAM that has never been used.
      af_q <= almost_full;
      if (almost_full) c_almost_full_beats <= c_almost_full_beats + 1;
      if (almost_full && !af_q) c_almost_full_events <= c_almost_full_events + 1;
    end
  end

endmodule

Classification: an attribution block. It prevents nothing and it decides how long a field failure takes to diagnose.

What it teaches: that c_frames_dropped_whole and c_frames_truncated are different failures and the second is much worse. A frame dropped at its first beat never existed as far as everything downstream is concerned; the statistics are short by one frame and nothing else happens. A frame truncated mid-flight reaches Chapter 19.4's engine as a structurally valid frame with wrong data, and is reported as a CRC error. One of these is a capacity problem stated plainly and the other is a capacity problem disguised as a physical-layer problem.

And it teaches that c_almost_full_events against c_almost_full_beats reads back the shape of the congestion. A thousand events of one beat each is bursty traffic that the FIFO is absorbing; one event of a thousand beats is a consumer that stopped. Chapter 18.1 §8's memory-stall argument predicts the second shape, and a port showing the first is congested for a reason the stall table does not describe.

Deliberately simplified: mid_frame_drop is sticky with no clear and is the signal a bring-up engineer should be told to look at first. The block cannot see whether a frame was dropped entirely between sof and eof when the sof itself was dropped, so c_frames_truncated undercounts in the worst congestion. And almost_full is consumed here only for statistics — the PAUSE it triggers is Chapter 14.2's and lives outside this chapter.

Production implication: the counter a network management system should see is c_frames_dropped_whole plus c_frames_truncated, reported as discards, and the counter it should not see is Chapter 19.4's c_bad alone. A port reporting 400 CRC errors and 380 truncations has a buffer problem; a port reporting 400 CRC errors and no truncations has a cable problem. That subtraction is the entire diagnostic value of this block, and it is available only if both counters exist and are read together — which is Chapter 18.7 §13's three-group telemetry argument arriving with a concrete pair.

15. RTL 7 — FIFO Telemetry

// ---------------------------------------------------------------------
// fifod_telemetry -- three buffers, three groups, and one derived number
// that answers the question a field engineer actually has.
// Section 15.
//
// Operator view: discards, and whether flow control engaged.
// Structural view: occupancy, drift, and which buffer is the bottleneck.
// Derived: the time until the FIFO fills at the measured drift, which
// turns Section 4's 13.107 ms from a trap into a maintenance window.
// ---------------------------------------------------------------------
module fifod_telemetry
  import fifod_pkg::*;
#(
  parameter int DEPTH_BEATS = 512,
  parameter int WR_MHZ      = 195
) (
  input  logic              clk,
  input  logic              rst_n,

  input  logic [15:0]       occupancy,
  input  logic [15:0]       almost_full_level,
  input  logic [15:0]       high_water_mark,
  input  logic signed [31:0] drift_ppm,
  input  logic              drift_valid,
  input  logic [31:0]       c_beats_dropped,
  input  logic [31:0]       c_frames_truncated,
  input  logic [31:0]       c_almost_full_events,
  input  logic [31:0]       elastic_inserted,
  input  logic [31:0]       elastic_deleted,

  // Operator view.
  output logic [31:0]       c_discards,
  output logic [31:0]       c_pause_worthy,

  // Structural view.
  output logic [15:0]       occupancy_pct,
  output logic [15:0]       headroom_used_pct,
  output logic signed [31:0] elastic_bias,

  // Derived. Section 5.
  output logic [31:0]       ms_until_full,
  output logic              drift_is_one_sided,
  output logic              headroom_never_used
);

  assign c_discards     = c_frames_truncated + c_beats_dropped;
  assign c_pause_worthy = c_almost_full_events;

  assign occupancy_pct = 16'((32'(occupancy) * 32'd100) / 32'(DEPTH_BEATS));

  // How far into the reserved headroom the port has ever gone. Zero
  // means the 123 beats have never been touched; 100 means they were
  // exhausted and Section 14 counted drops.
  assign headroom_used_pct =
    (high_water_mark <= almost_full_level) ? 16'd0
    : 16'((32'(high_water_mark - almost_full_level) * 32'd100) /
          (32'(DEPTH_BEATS) - 32'(almost_full_level)));

  assign headroom_never_used = (high_water_mark <= almost_full_level);

  // The elastic buffer's bias reads back the SIGN of the frequency
  // error between this port and its partner. Section 3: a one-sided
  // ratio is a board problem, not a design one.
  assign elastic_bias = 32'signed'(elastic_inserted) -
                        32'signed'(elastic_deleted);

  // Time until the FIFO fills at the measured drift. Section 5: the
  // remaining margin divided by the rate, which is a division rather
  // than an experiment. At 200 ppm with 389 beats of margin this is
  // 9.96 ms, and the point of reporting it is that nobody would ever
  // wait for it in simulation.
  always_comb begin
    if (!drift_valid || drift_ppm <= 0) begin
      ms_until_full      = 32'hFFFF_FFFF;
      drift_is_one_sided = 1'b0;
    end else begin
      ms_until_full = 32'((32'(DEPTH_BEATS - occupancy) * 32'd1_000_000) /
                          (32'(drift_ppm) * 32'(WR_MHZ)));
      drift_is_one_sided = (drift_ppm > 32'signed'(PPM_TOTAL / 4));
    end
  end

endmodule

Classification: an observability block whose two most useful outputs are a subtraction and a division.

What it teaches: that headroom_never_used is a budget finding rather than a fault. A port that has never gone past its almost-full level has 123 beats — 7.67 KiB — of SRAM that has done nothing since tape-out. That is not wrong: the headroom is insurance, and unused insurance is the normal outcome. But it is 24% of the FIFO, and an architect deciding whether the next part supports lossless flow control should be told how often the current one needed it.

And it teaches that ms_until_full converts an unobservable into a schedule. Section 4's failure takes 13.107 ms of continuous one-sided drift to appear and no simulation reaches it; in silicon the drift is measurable in 65 536 beats and the division gives a deadline. A port reporting 9.96 ms is not broken and will be, which is a category of telemetry that hardware rarely provides and software takes for granted.

Deliberately simplified: c_discards adds a beat count to a frame count, which is dimensionally wrong and is defensible only as a "something was lost" alarm — a production register set exposes both separately, and Section 14's argument about attribution is the reason. ms_until_full divides every cycle and must be computed on read. And drift_is_one_sided triggers at a quarter of the 200 ppm budget, which is a threshold chosen rather than derived and should be a register.

Production implication: the operator group here is deliberately two counters wide, and that is the negotiation this block loses every time. A network management system wants discards and pause events; it does not want occupancy percentages or drift in parts per million, and a design that exposes all of them through one register page gets its structural counters escalated as faults — Chapter 18.7 §13's failure, now with a specific example: elastic_bias is a number that looks alarming, is entirely normal, and belongs on a bring-up dashboard.

16. RTL 8 — The FIFO Conformance Monitor

// ---------------------------------------------------------------------
// fifod_conformance_monitor -- verdicts for three buffers. Section 16.
//
// Six verdicts. Two correctness, two configuration, one coverage, and
// one that is a standards claim about the PARTNER rather than about
// this design -- which is the second time in Module 19 that a monitor
// has had to report somebody else's conformance.
// ---------------------------------------------------------------------
module fifod_conformance_monitor
  import fifod_pkg::*;
#(
  parameter int MIN_AF_EVENTS   = 100,
  parameter int MIN_DRIFT_WINDOWS = 16
) (
  input  logic              clk,
  input  logic              rst_n,

  input  logic [31:0]       c_beats_dropped,
  input  logic [31:0]       c_frames_truncated,
  input  logic              elastic_overrun,
  input  logic              gap_too_short,
  input  logic              threshold_infeasible,
  input  logic              threshold_is_trivial,
  input  logic              drift_exceeds_budget,
  input  logic [31:0]       c_almost_full_events,
  input  logic [31:0]       c_windows,
  input  logic              headroom_never_used,

  output logic              data_lost,
  output logic              elastic_undersized,
  output logic              partner_nonconformant,
  output logic              threshold_misconfigured,
  output logic              flow_control_untested,
  output logic              drift_untested,
  output logic              none_of_the_above
);

  always_comb begin
    // Correctness. Either makes every other verdict moot.
    data_lost          = (c_beats_dropped != '0) || (c_frames_truncated != '0);
    elastic_undersized = elastic_overrun;

    // A standards claim about the FAR END. Chapter 5.9's floor is 9
    // octets and Section 2's worst drift over a jumbo frame is 1.8, so a
    // conformant partner leaves five times the room needed. Asserting
    // this is asserting that the partner obeys the gap, which is exactly
    // what Chapter 19.3's class 83 warned about asserting the wrong way
    // -- here it is a REPORT, not a property, which is the difference.
    partner_nonconformant = gap_too_short;

    // Configuration. Both directions are wrong and only one is an error.
    threshold_misconfigured = threshold_infeasible;

    // Coverage. A run that never filled the FIFO has not tested the
    // thing the headroom is for. Section 8: this is the condition least
    // represented in a regression.
    flow_control_untested = (c_almost_full_events < 32'(MIN_AF_EVENTS));

    // And a run that never measured the drift has not tested Section 5,
    // which is the only thing in this chapter that can reach Section 4's
    // failure in bounded time.
    drift_untested = (c_windows < 32'(MIN_DRIFT_WINDOWS));

    none_of_the_above = !data_lost && !elastic_undersized &&
                        !partner_nonconformant && !threshold_misconfigured &&
                        !flow_control_untested && !drift_untested;
  end

endmodule

Classification: a verdict generator, the eleventh in Modules 18 and 19, and the first whose inputs come from three physically separate buffers.

What it teaches: that threshold_is_trivial is an input and does not appear in any verdict. Section 9 computes it and this monitor deliberately ignores it, because a headroom that is 2% of the depth is not a conformance failure — it is an area finding, and putting it in none_of_the_above would fail a perfectly correct design. Deciding which computed facts are verdicts and which are merely reported is the design of a monitor, and this is the clearest instance in the track of a fact being demoted on purpose.

And it teaches that partner_nonconformant is a report about somebody else. Chapter 19.4 §16's fragment_kind_confused was about this design's own framing; this one says the device at the other end of the cable is closing the interframe gap. The correct response is to escalate to the partner's owner, and a design that responds by deepening its elastic buffer has accepted a non-conformant link and hidden it — which Chapter 4.4 §2 already showed does not work, because the accumulation is unbounded.

Deliberately simplified: MIN_AF_EVENTS defaults to 100, which a regression without a congestion scenario will never reach — deliberately, because Section 8's whole argument is that the almost-full path is the least-tested one. data_lost merges two failures Section 14 went to trouble to separate, which is right for a verdict and wrong for a counter. And there is no verdict for the transmit FIFO at all, because Section 12 established that its depth is a choice rather than a derivation and there is nothing to conform to.

Production implication: none_of_the_above for the eleventh time, and this is the first instance where a clean verdict depends on a scenario the regression has to be written to produce. A port asserting it has lost no data, an adequately sized elastic buffer, a conformant partner, a feasible watermark, at least a hundred flow-control events and sixteen drift measurements. The fifth clause is the one that fails, and it fails on designs that are entirely correct — which is the intended behaviour and is worth telling a verification lead before they file it as a bug.

17. What the FIFOs Assume

Nine assumptions across three buffers, and five of them belong to somebody outside this chip.

Rows one to four are the chapter's distinguishing feature and it is worth stating plainly: more of this block's sizing depends on other people than any other block in Modules 18 and 19. Chapter 19.4's engine is correct for any conformant beat sequence; Chapter 19.2's parser is correct for any offset. This FIFO is correct only for a particular cable, a particular partner, and a particular memory subsystem, and none of the three is written on the chip.

Row nine is the quietest failure in the chapter. The almost-full level is computed, the SRAM is allocated, the threshold asserts — and if nothing downstream turns it into a PAUSE frame, the reserve is dead area and the port is lossy. There is no error, no counter, no verdict; Section 15's headroom_never_used is the only symptom and it is indistinguishable from an uncongested link. The distinguishing evidence is c_almost_full_events being non-zero while the partner's transmit rate never drops, which needs the partner's counters to detect.

And two things this chapter does not assume, which is unusual after rows one to nine:

Row two is worth isolating because it is a common design anxiety and it is misplaced. A FIFO sized for a 2 µs stop survives any burst pattern whatsoever, because the arrival rate is bounded by the line rate and the line rate is what the stall term was computed from. Burstiness is a problem for a switch's shared buffer — Chapter 14.1 — and not for a port's private one.

18. Reset, and the Three Domains

Three buffers spanning four clock domains, and the reset sequence is where a correct design most often becomes an incorrect one.

Chapter 19.1 §13's sequencer releases each domain on its own clock. This section is what each buffer must be in when its release happens.

Row one is Chapter 4.4 §6's and it is the single most-repeated bug in elastic buffer design. An ordinary FIFO starts empty and is correct; an elastic FIFO must absorb drift in both directions, so its idle state is the centre. A buffer released empty underruns on the first frame whose recovered clock is slower than the local one, which is half of all links.

Row two is the asynchronous-reset hazard and it is specific. The two pointer domains are released by different reset signals on different clocks, so one side can be running while the other is still held — and the running side's synchronised view of the held side's pointer is zero, which is a legal value. A write side released early sees r_gray at zero, computes an occupancy from its own advancing pointer, and fills the FIFO before the read side exists.

The asymmetry in row three is the reason the release order is specified rather than left to the sequencer. The read side must be released first, or equivalently the write side must be held until the read side's reset has propagated — and "equivalently" is doing work there, because the two are the same statement only if somebody counts the synchroniser's two clocks.

And the third hazard has nothing to do with pointers.

Section 9's almost_full_level is computed from parameters and arrives as an input. A design that registers it has a cycle after release where it is zero, and an almost-full level of zero means almost-full asserts immediately — so the port issues a PAUSE on the first beat after every reset. The symptom is a link that appears to work and runs at a fraction of line rate after any link flap, which is Chapter 19.3 §21's complaint 4 in a different block: state that survives or fails to survive a reset, producing a small persistent anomaly nobody escalates.

Three buffers, three reset requirements, and only one of them is "start empty."

19. The Cost, Accounted

Eight blocks, and for the first time in Module 19 the cost is measured in kilobytes rather than gates.

The flop count is the smallest of Module 19's four datapath chapters and the memory is all of it. Chapter 19.2 was 4 800 flops and no memory; Chapter 19.4 was 2 164 flops and 39 445 XOR terms and no memory. This chapter is 1 100 flops and 41 KiB, and the 41 KiB is about 330 000 bit cells — an order more storage than every flop in Module 19 combined.

Which is why the FIFOs are the item that gets negotiated. Chapter 18.1 §18 put a 100 Gb/s port's FIFOs at 9.1% of a chip's SRAM in its own example, and this chapter's decomposition says where that goes:

Row four is a negative number and it is the only line in the table this chapter controls. Everything else is somebody else's parameter, and the one decision available — round up to 1 024 or down to 512 — is worth 16 KiB against a 1.16% margin in the least certain term.

Module 19's running total, with four chapters built:

Four chapters, 78.8% of the flop estimate and all of the memory — with the memory interface and the statistics counters still to come. The 41 KiB lands exactly on §18's figure, which it should, because §18 took it from Chapter 18.1 §18 and this chapter has now derived rather than quoted it: 24.41 plus 7.67 plus a rounding, and 9 for transmit.

And the derivation moved one number. Chapter 18.1 §18's long-cable figure was 146 KiB lossless to 2 km; Section 10's closed form gives a headroom of 123.63 KiB against a 24.41 KiB stall buffer, so the correct figure is 148 KiB. A 1.4% correction to a number whose point was its size, and it is worth making because the argument it supports is about whether 2 km links are affordable at all.

20. Properties Worth Asserting, and One Worth Refusing

Thirty-one properties in five groups, and the refused one is the most natural assertion anybody writes about a FIFO.

Group 1 — the elastic buffer. Everything is distance from centre.

// After reset the buffer is at its centre, not empty. Chapter 4.4's
// most-repeated bug, asserted.
a_starts_centred: assert property (@(posedge tx_clk)
  $rose(rst_n) |-> (occupancy == CENTRE));

// Distance from centre never exceeds the drift the depth was sized for.
a_within_drift: assert property (@(posedge tx_clk) disable iff (!rst_n)
  (offset_from_centre <= DRIFT_OCTETS) &&
  (offset_from_centre >= -DRIFT_OCTETS));

// Insertion and deletion happen ONLY in the gap. Chapter 4.4 Section 3:
// inside a frame there is nothing that may be added or removed.
a_discharge_in_gap: assert property (@(posedge tx_clk) disable iff (!rst_n)
  ($changed(c_inserted) || $changed(c_deleted)) |-> $past(in_gap));

// Overrun loses data and underrun does not, so overrun is sticky.
a_overrun_sticky: assert property (@(posedge tx_clk) disable iff (!rst_n)
  overrun |=> overrun);

// One discharge per gap is enough, because Chapter 5.9's floor is 9 and
// the worst drift over a jumbo frame is 1.8 octets.
a_one_discharge_suffices: assert property (@(posedge tx_clk) disable iff (!rst_n)
  (in_gap && !gap_too_short) |-> ##1 (offset_from_centre == 0 ||
                                      offset_from_centre == $past(offset_from_centre)));

Group 2 — the crossing FIFO's pointers.

// Gray codes differ by exactly one bit between consecutive values.
a_gray_single_bit: assert property (@(posedge wr_clk) disable iff (!wr_rst_n)
  $changed(w_gray) |-> ($countones(w_gray ^ $past(w_gray)) == 1));

// The synchronised pointer is always a value the source actually held.
a_sync_is_real: assert property (@(posedge wr_clk) disable iff (!wr_rst_n)
  1'b1 |-> !$isunknown(r_gray_w2));

// Occupancy never exceeds the depth minus the reserve, which is what
// wr_full enforces. This is NOT the refused property -- it is about the
// FLAG, and it is checkable in a cycle.
a_full_stops_writes: assert property (@(posedge wr_clk) disable iff (!wr_rst_n)
  wr_full |-> !$rose(w_bin));

// Empty and full are never both true.
a_not_both: assert property (@(posedge wr_clk) disable iff (!wr_rst_n)
  !(wr_full && rd_empty));

// The reserve is at least the synchroniser round trip, at elaboration.
a_reserve_adequate: assert property (@(posedge wr_clk)
  CDC_RESERVE >= (((2*1000000)/WR_MHZ + (2*1000000)/RD_MHZ) * WR_MHZ + 999999) / 1000000);

// A read never returns data that was never written.
a_no_phantom_reads: assert property (@(posedge rd_clk) disable iff (!rd_rst_n)
  (rd_en && !rd_empty) |-> (r_bin != w_gray_r2));

// High-water mark is monotonic. Chapter 18.1 Section 15: a counter that
// clears on read destroys the evidence.
a_hwm_monotonic: assert property (@(posedge wr_clk) disable iff (!wr_rst_n)
  high_water_mark >= $past(high_water_mark));

Group 3 — the thresholds, where the properties are about parameters rather than behaviour.

// The almost-full level leaves room for the headroom AND the reserve.
a_af_leaves_headroom: assert property (@(posedge clk)
  almost_full_level + headroom_beats + CDC_RESERVE <= DEPTH_BEATS);

// The headroom matches Section 8's closed form for the configured link.
a_headroom_closed_form: assert property (@(posedge clk)
  headroom_octets == (PAUSE_OCTETS + PARTNER_MTU +
                      (CABLE_M * NS_PER_M * LINK_MBPS) / 8000) + tx_backlog_octets);

// Almost-full asserts strictly before full, always.
a_af_before_full: assert property (@(posedge wr_clk) disable iff (!wr_rst_n)
  wr_full |-> wr_almost_full);

// And they are not the same threshold. Section 13 row 4.
a_af_is_not_full: assert property (@(posedge clk)
  almost_full_level < (DEPTH_BEATS - CDC_RESERVE));

// Infeasible and trivial are mutually exclusive.
a_threshold_exclusive: assert property (@(posedge clk)
  !(threshold_infeasible && threshold_is_trivial));

// The crossover, when it exists, is where the two terms are equal.
a_crossover_correct: assert property (@(posedge clk)
  !no_crossover |-> ((crossover_mbps * (STALL_NS_P - PROP_NS_L)) ==
                     (8 * (PAUSE_OCTETS + PARTNER_MTU) * 1000)));

Group 4 — drops, and their attribution.

// A drop happens only when the FIFO is full. It is never discretionary.
a_drop_only_when_full: assert property (@(posedge clk) disable iff (!rst_n)
  drop_beat |-> fifo_full);

// A drop at SOF is a whole frame; a drop elsewhere is a truncation.
a_drop_attributed: assert property (@(posedge clk) disable iff (!rst_n)
  (drop_beat && beat_sof) |-> ##1 $changed(c_frames_dropped_whole));

a_truncation_attributed: assert property (@(posedge clk) disable iff (!rst_n)
  (drop_beat && !beat_sof) |-> ##1 mid_frame_drop);

// The receive path never stalls. Chapter 19.1 Section 11.
a_never_backpressures: assert property (@(posedge clk) disable iff (!rst_n)
  beat_valid |-> ##1 ($changed(c_beats_dropped) || $changed(occupancy) ||
                      $past(occupancy) == occupancy));

// Truncation implies a mid-frame drop happened.
a_truncation_needs_drop: assert property (@(posedge clk) disable iff (!rst_n)
  frame_truncated |-> mid_frame_drop);

// Almost-full events count rising edges, not levels.
a_af_edge_counted: assert property (@(posedge clk) disable iff (!rst_n)
  $changed(c_almost_full_events) |-> ($rose(almost_full)));

Group 5 — the drift, where every property is about a rate and none is about an occupancy.

// A window completes exactly every WINDOW_BEATS writes.
a_window_periodic: assert property (@(posedge wr_clk) disable iff (!rst_n)
  window_complete |-> ((w_cnt - $past(w_snap)) >= WINDOW_BEATS));

// The measured drift is bounded by what 200 ppm can produce plus the
// synchroniser's two counts of uncertainty.
a_drift_bounded: assert property (@(posedge wr_clk) disable iff (!rst_n)
  window_complete |-> (drift_ppm <= (PPM_TOTAL + (2 * 1000000) / WINDOW_BEATS)));

// Budget exceeded is sticky.
a_budget_sticky: assert property (@(posedge wr_clk) disable iff (!rst_n)
  drift_exceeds_budget |=> drift_exceeds_budget);

// A positive drift with a bounded margin gives a bounded time to full.
a_time_to_full_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  (drift_valid && drift_ppm > 0) |-> (ms_until_full != 32'hFFFF_FFFF));

// And the reverse: no drift means no deadline.
a_no_drift_no_deadline: assert property (@(posedge clk) disable iff (!rst_n)
  (drift_valid && drift_ppm <= 0) |-> (ms_until_full == 32'hFFFF_FFFF));

Group 6 — coverage.

c_elastic_both_ways:  cover property (@(posedge tx_clk)
  (c_inserted > 0) && (c_deleted > 0));
c_fifo_full:          cover property (@(posedge wr_clk) wr_full);
c_almost_full:        cover property (@(posedge wr_clk) wr_almost_full && !wr_full);
c_drop_at_sof:        cover property (@(posedge clk) drop_beat && beat_sof);
c_truncation:         cover property (@(posedge clk) frame_truncated);
c_drift_measured:     cover property (@(posedge wr_clk) c_windows > 16);

21. Verification Scenarios

Fifty-eight scenarios, plus a five-run directed test whose whole content is a clock frequency and a cable length.

The elastic buffer — 11 scenarios.

Row seven is the parameter bug and row eleven is the reset bug, and both pass every functional test at a nominal clock. Row seven needs jumbo traffic and row eleven needs a slow partner, neither of which a default testbench provides.

The crossing FIFO — 13 scenarios.

Row fourteen against row fifteen is the sizing check and it is the one scenario that directly tests Chapter 18.1 §8's number. The FIFO is sized for exactly 2 µs and a 2.1 µs stall must drop, because a design that survives 2.1 µs has more depth than the calculation asked for and somebody should know why.

The thresholds — 12 scenarios.

Row thirty-two is the scenario that validates the chapter rather than the design. Section 8's closed form is a rederivation of somebody else's table, and a rederivation that does not reproduce the original is a new result rather than a simplification — which is worth checking before building anything on it.

Drops and attribution — 10 scenarios.

Rows forty and forty-one are the pair that justifies Section 14's existence, and the difference between them is one counter. Row forty-six is the failure with no symptom at all.

The drift — 12 scenarios.

Rows forty-eight and forty-nine are the chapter's central verification fact in two lines, and row fifty is the fix. Row forty-seven is the fact that makes the fix necessary rather than merely convenient.

The directed test — five runs random stimulus will not produce.

Three of this chapter's failures need a condition a testbench has to be built to create, and in each case the default testbench creates the opposite.

Row one's "default" is the important entry. Generating two clocks with a 200 ppm offset requires a non-integer clock period, which most testbench clock generators do not naturally produce and which some simulators round away. A testbench with #2.5601ns and #2.56ns is 39 ppm apart, not 200, and a verification engineer who has not checked will report the drift scenario as run.

Construct it. Five runs.

Run B is the long one and it is the only run in this chapter that has to be. At 195.3125 MHz, 2 560 000 beats is 13.107 ms of simulated time, which at a typical RTL simulation rate is minutes to hours. It is worth running once per tape-out and never in a nightly, and Section 5's tracker is what stands in for it the rest of the time.

Run E is eleven cycles long and finds a bug that survives every other run. Release the write domain first, let it see r_gray at zero, and watch the occupancy climb to 508 before the read side exists. The whole scenario is a reset-order change.

The oracle, in four parts:

Row one is the check that the runs are what they claim to be, and it is the row most likely to fail on a testbench whose clock generator rounded the period. A run B reporting a drift of 39 ppm is not run B, and every conclusion drawn from it is about a different design point.