The MAC's Memory Interface

Chapter 18.5 shaped this MAC's bus traffic — 32-beat bursts, 8 outstanding transactions, 16 AXI IDs and a reorder buffer. Chapter 19.1 §11 established that the receive path may never stall. This chapter is what happens where those two facts meet, and they are not compatible as stated.

The collision is precise. Chapter 18.5 §16 sized a reorder buffer at (O − 1) bursts and said the overflow behaviour must be safe. The safe overflow behaviour on an AXI read data channel is to deassert RREADY — to stop accepting responses until the buffer drains. That is legal, it is what the protocol is for, and on this MAC it drops received frames.

Row one is why the stall looks safe and row four is what it costs. The two paths share a read data channel, and a backpressure that is legal for one of them is fatal for the other. A block can be correct in isolation and wrong inside its own loop, which is this chapter's rejected property and is the shape of the whole chapter.

1. Scope, and a Shared Channel

Chapter 18.5 owns the bus protocol: burst shaping, outstanding limits, ID allocation, the ordering contract. Chapter 18.4 owns the descriptor ring and the gather engine. Chapter 18.2 owns the ring's structure. None of the three asked what the MAC's own datapath does to the bus, because at that point the datapath had not been built.

Now it has, and it changes two things.

Row two is the one that makes this chapter necessary. Every mechanism Chapter 18.5 built has a backpressure path in it — that is what AXI's ready signals are — and one of the four things using this interface cannot use any of them.

The four traffic classes, and only one of them may not wait:

Row four is the surprise and it is the chapter's subject. A receive descriptor fetch is a read, so it shares a channel with transmit's frame data — and it is on the receive path's critical loop, because the MAC cannot write a received frame to memory until it knows where the buffer is. The one class that cannot wait is a read, and reads are where all the backpressure lives.

What this chapter owns: the request shaper at 512-bit beats; the outstanding-transaction accounting against Little's law at this datapath; the ID classifier that keeps the four classes separable; what happens when the reorder buffer fills and why only one of the three available answers is safe; the write path's commit tracking; and the correction to Chapter 18.5 §18's memory bill that falls out of it.

What it does not own: the AXI protocol — Chapter 18.5 — the descriptor ring's layout — Chapter 18.2 — and the FIFOs, which are Chapter 19.5's and appear here only as the thing that fills.

What it does not build: the statistics counters are Chapter 19.7, which closes the module.

2. A Full Reorder Buffer, and Three Ways to Handle It

Chapter 18.5 §16 established the rule and left the consequence open.

The rule: with O transactions outstanding in one ordering domain, at most O − 1 responses can have arrived out of order while the design waits for the one it needs. So the reorder buffer holds (O − 1) bursts, and Chapter 18.5 §16 added that the overflow behaviour must be safe.

At this chapter's configuration that is 14 KiB.

Now suppose it fills anyway. There are exactly three things an AXI master can do with a read response it has no room for.

Option 2 is out immediately and it is worth saying why rather than assuming it. AXI has no read retry: a response accepted is a response consumed, and there is no transaction-layer mechanism to ask for it again. Discarding read data means the frame is assembled from a hole, and Chapter 19.4's engine computes a check value over it — so the frame goes out on the wire, well formed, with wrong contents and a correct FCS. That is the worst failure mode in Module 19.

Option 1 is the one every design reaches for, and it is where the chapter's argument is.

Deasserting RREADY is correct AXI and it is correct for the transmit path. Chapter 19.3 §5: a transmit stage may stall before commit. The frame is delayed, the wire idles, throughput falls, and nothing is lost.

But RREADY is a property of the channel, not of a transaction. Deasserting it stops every read response, including the receive path's descriptor fetches — and Chapter 19.1 §11 says the receive path has nowhere to push back to.

Row five is the cost of a legal stall, and the stall was on the transmit path. Nothing about the transmit path is wrong; nothing about the AXI usage is wrong; and frames are lost on the other direction entirely.

Which leaves option 3, and option 3 is not a mechanism. It is a sizing requirement.

Size the buffer at (O − 1) bursts and the full condition is unreachable, because the rule says at most O − 1 responses can be waiting. Chapter 18.5 §16 derived the bound as a sizing guide; this chapter says it is a correctness requirement, and the difference matters because a design that budgets less has not saved memory — it has converted a transmit-path throughput loss into a receive-path frame loss.

And Chapter 18.5 §18 budgeted less. Section 19 is that correction.

3. RTL 1 — The Request Shaper

// ---------------------------------------------------------------------
// memif_pkg -- the MAC datapath's side of Chapter 18.5's bus. Sections 3
// through 13.
//
// Four traffic classes on one interface, and exactly one of them may not
// wait. Everything in this package exists to keep that one separable
// from the other three.
// ---------------------------------------------------------------------
package memif_pkg;

  localparam int DATA_B     = 64;                  // octets per beat
  localparam int DATA_W     = DATA_B * 8;          // 512

  // Chapter 18.5 Section 4's shaping, at this datapath's beat.
  localparam int BURST_BEATS  = 32;
  localparam int BURST_OCTETS = BURST_BEATS * DATA_B;   // 2048 -- 2 KiB
  localparam int MAX_OUTSTANDING = 8;
  localparam int NUM_IDS         = 16;

  // Chapter 18.5 Section 16's bound, as a REQUIREMENT rather than a guide.
  // Section 2: at (O-1) bursts the full state is unreachable; below it,
  // a transmit stall becomes a receive drop.
  localparam int REORDER_BURSTS  = MAX_OUTSTANDING - 1;          // 7
  localparam int REORDER_OCTETS  = REORDER_BURSTS * BURST_OCTETS; // 14 KiB

  // The four classes. Only RX_DESC and RX_DATA sit on the receive path's
  // loop, and only RX_DESC is a READ -- which is why it shares a channel
  // with the class most likely to stall it.
  typedef enum logic [1:0] {
    CLASS_TX_DATA = 2'd0,   // read  -- may stall before commit
    CLASS_TX_DESC = 2'd1,   // read  -- may stall
    CLASS_RX_DATA = 2'd2,   // write -- may not
    CLASS_RX_DESC = 2'd3    // read  -- MAY NOT
  } traffic_class_e;

  typedef struct packed {
    logic [47:0]        addr;
    logic [7:0]         len_beats;
    traffic_class_e     tclass;
    logic [3:0]         id;
    logic               last_of_frame;
  } mem_req_t;

endpackage

// ---------------------------------------------------------------------
// request_shaper -- turn a stream of 512-bit beats into Chapter 18.5's
// 32-beat bursts, and never issue a request the reorder buffer cannot
// absorb the response to. Sections 3 and 4.
//
// The shaper is where Chapter 18.5 Section 4's burst-length decision meets
// Chapter 19.1 Section 3's beat. At 64 octets per beat a 32-beat burst is
// 2 KiB, so a maximum frame is 0.74 bursts and a jumbo frame is 4.39 --
// which means most frames do not fill a burst and the shaper's real job
// is deciding what to do about the remainder.
// ---------------------------------------------------------------------
module request_shaper
  import memif_pkg::*;
(
  input  logic              clk,
  input  logic              rst_n,

  input  logic              req_valid,
  input  logic [47:0]       req_addr,
  input  logic [15:0]       req_octets,
  input  traffic_class_e    req_class,
  output logic              req_ready,

  input  logic              credit_available,   // Section 5's tracker
  output logic              issue,
  output mem_req_t          issued,

  // Observability. Sections 15 and 16.
  output logic [31:0]       c_bursts,
  output logic [31:0]       c_full_bursts,
  output logic [31:0]       c_short_bursts,
  output logic [31:0]       c_beats_requested,
  output logic [15:0]       mean_burst_x100,
  output logic              unaligned_split
);

  logic [15:0] remaining;
  logic [47:0] cursor;
  logic        active;
  logic [7:0]  this_len;

  // A burst may not cross a 4 KiB boundary -- Chapter 18.5 Section 5's
  // alignment splitter. At 2 KiB per burst that means an unaligned
  // start splits one burst into two, which is the 37.04% of transactions
  // that chapter measured and is why the shaper reports it.
  logic [11:0] offset_in_4k;
  logic [7:0]  beats_to_4k;

  assign offset_in_4k = cursor[11:0];
  assign beats_to_4k  = 8'((12'd4096 - offset_in_4k) / 12'(DATA_B));

  always_comb begin
    this_len = 8'(BURST_BEATS);
    if (remaining < 16'(BURST_OCTETS))
      this_len = 8'((remaining + 16'(DATA_B) - 16'd1) / 16'(DATA_B));
    if (beats_to_4k < this_len) this_len = beats_to_4k;
  end

  assign req_ready = !active;
  assign issue     = active && credit_available;

  assign issued.addr          = cursor;
  assign issued.len_beats     = this_len;
  assign issued.tclass        = req_class;
  assign issued.id            = '0;                 // Section 7 assigns it
  assign issued.last_of_frame = (remaining <= 16'(this_len) * 16'(DATA_B));

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      remaining <= '0; cursor <= '0; active <= 1'b0;
      c_bursts <= '0; c_full_bursts <= '0; c_short_bursts <= '0;
      c_beats_requested <= '0; unaligned_split <= 1'b0;
    end else begin
      if (req_valid && req_ready) begin
        remaining <= req_octets;
        cursor    <= req_addr;
        active    <= 1'b1;
        if (req_addr[11:0] != '0) unaligned_split <= 1'b1;
      end else if (issue) begin
        c_bursts          <= c_bursts + 1;
        c_beats_requested <= c_beats_requested + 32'(this_len);
        if (this_len == 8'(BURST_BEATS)) c_full_bursts  <= c_full_bursts + 1;
        else                             c_short_bursts <= c_short_bursts + 1;

        cursor    <= cursor + 48'(this_len) * 48'(DATA_B);
        if (remaining <= 16'(this_len) * 16'(DATA_B)) begin
          remaining <= '0;
          active    <= 1'b0;
        end else begin
          remaining <= remaining - 16'(this_len) * 16'(DATA_B);
        end
      end
    end
  end

  assign mean_burst_x100 = (c_bursts == 0) ? 16'd0
                         : 16'((c_beats_requested * 32'd100) / c_bursts);

endmodule

Classification: a datapath transform, and the block where Chapter 18.5's bus decisions meet Chapter 19.1's beat.

What it teaches: that a 32-beat burst at this datapath is 2 KiB, and most Ethernet frames are smaller than that. Chapter 5.1's maximum basic frame is 1 518 octets — 0.74 of a burst — so a transmit path fetching whole frames issues short bursts almost always, and mean_burst_x100 on a port carrying ordinary traffic reads far below 3 200. Chapter 18.5 §4's efficiency argument was about the bursts a design issues, and this block is where it discovers how many of them it can actually fill.

And it teaches that credit_available gates issuance rather than the response path. The shaper never issues a request whose response it cannot absorb — Section 2's option 3 arriving as one AND gate — which is why there is no overflow handling anywhere in this module. The safety is at the request end, where a stall costs nothing, rather than at the response end, where it costs frames.

Deliberately simplified: beats_to_4k divides by a constant and assumes DATA_B divides 4 096, which it does at 64 octets and would not at 48. The shaper handles one request at a time — req_ready is simply !active — so a real design pipelines the next frame's address computation under the current frame's bursts. And unaligned_split is sticky rather than counted, which loses the rate Chapter 18.5 §5 measured at 37.04%.

Production implication: c_short_bursts against c_full_bursts is the number that says whether Chapter 18.5 §4's burst-length decision is being realised. A port on ordinary traffic reports mostly short bursts and that is correct — the frames are smaller than the burst. A port on jumbo traffic reporting mostly short bursts has an alignment problem, because a 9 000-octet frame is 4.39 bursts and should produce four full ones. The ratio separates "the frames are small" from "the addresses are bad", which are otherwise the same symptom: transactions above the model.

4. Thirty-Two Beats, Eight Outstanding, and Little's Law

Chapter 18.5 §16 concluded that burst length and reorder buffer are the same decision. That is true when the outstanding limit is fixed and false when it is chosen, and the distinction is worth 10 KiB.

Little's law at this interface: to keep 100 Gb/s flowing across a memory latency L, the design must have R × L / burst bursts in flight.

Doubling the burst halves the outstanding count, which is the obvious half. The non-obvious half is what it does to the reorder buffer.

They are the same. (O − 1) × burst where O = ceil(R × L / burst) is R × L − burst, near enough — so the reorder buffer is a property of the bandwidth-latency product and not of the burst length at all.

Which means Chapter 18.5 §16's coupling is real only because O is what the interconnect offers rather than what the design needs.

Row three is the section's result. A design that could ask for the outstanding count it wants pays for the bandwidth-latency product once, whatever burst it chooses. A design given eight outstanding transactions and choosing 32-beat bursts pays 14 KiB for a buffer that Little's law would have sized at 12 — and pays it because the interconnect's limit, not the design's arithmetic, is binding.

And the practical reading is uncomfortable. With O = 8 and 2 KiB bursts, the design sustains 16 KiB in flight, which at 100 Gb/s covers a latency of

16 384 octets × 8 / 100 Gb/s = 1.31 µs

So this configuration is throughput-limited above 1.31 µs of memory latency, which is inside the range Chapter 18.1 §8's stall table treats as ordinary. The outstanding limit, not the burst length and not the reorder buffer, is what sets the ceiling — and it is the one parameter Chapter 18.5 §18 listed as "wanted 5 to 67, typically offered 8 or 16."

5. RTL 2 — The Outstanding Tracker

// ---------------------------------------------------------------------
// memif_outstanding_tracker -- issue a request only if the response has
// somewhere to go. Sections 2, 4 and 5.
//
// This is where Section 2's option 3 lives. The reorder buffer is sized
// at (O-1) bursts and this block guarantees that no more than O
// transactions are outstanding in one ordering domain, so the buffer's
// full state is UNREACHABLE rather than handled.
//
// The credit is per ORDERING DOMAIN, not per interface. Chapter 18.5
// Section 12: an AXI ID is an ordering contract, so transactions with
// different IDs may complete in any order and transactions with the same
// ID may not. The domain is the unit the (O-1) bound applies to.
// ---------------------------------------------------------------------
module memif_outstanding_tracker
  import memif_pkg::*;
#(
  parameter int DOMAINS = 4                    // one per traffic class
) (
  input  logic              clk,
  input  logic              rst_n,

  input  logic              issue,
  input  traffic_class_e    issue_class,
  input  logic              response,
  input  traffic_class_e    response_class,

  output logic              credit_available,
  output logic [3:0]        outstanding [DOMAINS],

  // Observability. Sections 15 and 16.
  output logic [31:0]       c_issued,
  output logic [31:0]       c_completed,
  output logic [3:0]        peak_outstanding,
  output logic [31:0]       c_credit_stalls,
  output logic              bound_violated
);

  // The per-class limit. Chapter 18.5 Section 6's throughput ceiling is a
  // property of the TOTAL, but the reorder buffer's bound is per domain,
  // and a design that tracks only the total can put all 8 in one domain
  // and overflow a buffer sized for 7 bursts of one class.
  localparam int PER_DOMAIN_MAX = MAX_OUTSTANDING;

  logic [3:0] cnt [DOMAINS];
  int unsigned total;

  always_comb begin
    total = 0;
    for (int d = 0; d < DOMAINS; d++) total += cnt[d];
  end

  // A credit exists if BOTH limits hold. The receive descriptor class
  // gets a reserved credit -- Section 10 -- so that a transmit-side
  // saturation cannot starve the one class that may not wait.
  assign credit_available =
      (total < unsigned'(MAX_OUTSTANDING)) ||
      ((issue_class == CLASS_RX_DESC) && (cnt[CLASS_RX_DESC] == '0));

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int d = 0; d < DOMAINS; d++) cnt[d] <= '0;
      c_issued <= '0; c_completed <= '0;
      peak_outstanding <= '0; c_credit_stalls <= '0;
      bound_violated <= 1'b0;
    end else begin
      if (issue) begin
        cnt[issue_class] <= cnt[issue_class] + 1;
        c_issued <= c_issued + 1;
      end
      if (response) begin
        cnt[response_class] <= cnt[response_class] - 1;
        c_completed <= c_completed + 1;
      end

      if (4'(total) > peak_outstanding) peak_outstanding <= 4'(total);
      if (!credit_available) c_credit_stalls <= c_credit_stalls + 1;

      // If this ever fires, Section 2's option 3 has failed and the
      // reorder buffer CAN fill -- at which point the design is back to
      // choosing between option 1 and option 2, and both lose frames.
      if (4'(total) > 4'(PER_DOMAIN_MAX)) bound_violated <= 1'b1;
    end
  end

  assign outstanding = cnt;

endmodule

Classification: the block that makes the rest of the chapter's safety argument true, and it is a counter and a comparison.

What it teaches: that the reserved credit for CLASS_RX_DESC is not fairness, it is correctness. Without it, eight outstanding transmit reads leave no credit for a receive descriptor fetch — and the receive path then waits for a transmit transaction to complete. Chapter 19.1 §11 says it cannot wait, so the reservation is what keeps a saturated transmit path from stalling a receive path that has no stall to give. One credit out of eight, and it costs 12.5% of the outstanding budget.

And it teaches that per-domain and total limits are different checks. The interconnect's limit is a total; Chapter 18.5 §16's (O − 1) bound is per ordering domain. A design that tracks only the total can place all eight in one domain, which is legal against the interconnect and overflows a buffer sized for seven bursts of one class. bound_violated is the assertion that the two agree.

Deliberately simplified: total is a combinational sum over four 4-bit counters in the issue path, which at 195.3125 MHz and four terms is fine and does not generalise to sixteen domains. The reserved credit is one deep, so two receive descriptor fetches in flight still need an ordinary credit. And bound_violated has no recovery, because there is none: if the bound is exceeded the reorder buffer's sizing argument has failed and nothing downstream is trustworthy.

Production implication: peak_outstanding against MAX_OUTSTANDING is the number that says whether the interconnect's limit is binding. A port whose peak never reaches 8 is limited by something else — the shaper's one-request-at-a-time, or the frame rate — and raising the interconnect's limit would buy nothing. A port pinned at 8 with c_credit_stalls climbing is limited by exactly the parameter Chapter 18.5 §18 listed as "wanted 5 to 67, typically offered 8", and Section 4's arithmetic says the ceiling is 1.31 µs of latency.

6. What an AXI ID Means Here

Chapter 18.5 §12 established that an AXI ID is an ordering contract rather than a tag, and priced same-ID operation at 36% of line rate. This section is what the four traffic classes do to that contract.

Two transactions with the same ID must complete in order. Two with different IDs may complete in any order. So an ID is a promise the master makes to itself, and the question this chapter adds is: which of the four classes need to be ordered against which?

Every off-diagonal entry is "no", which is the section's first result and it is not obvious. A transmit frame's data has no ordering relationship with a receive descriptor fetch; they are different rings, different directions and different memory regions. So the four classes are four independent ordering domains and must not share an ID.

The diagonal is where the ordering lives and each entry has a different reason.

Rows one and three look identical and they are not. Transmit data is a read, so out-of-order completion is visible to this design and the reorder buffer fixes it. Receive data is a write, and Chapter 18.5 §12's point is that writes with the same ID are ordered at the slave — so the ordering is free and there is no buffer. The same requirement, and one of them costs 14 KiB.

Now the arithmetic that makes the ID allocation tight. Chapter 18.5 §18 records 16 IDs offered against 50 wanted.

Thirteen IDs for transmit data, and one each for the other three. That is the allocation this chapter uses, and the reason the other three get one each is that none of them needs parallelism: a descriptor fetch is one burst, and receive data's ordering is free at the slave. Spending IDs on them would take them from the only class that can use them.

7. RTL 3 — The ID Classifier

// ---------------------------------------------------------------------
// id_classifier -- map a request to an AXI ID that encodes its ordering
// domain. Sections 6 and 7.
//
// Sixteen IDs, four classes, and the allocation is not equal: transmit
// frame data gets thirteen because it is the only class whose ordering
// requirement permits parallelism within itself. The other three get one
// each and need no more -- a descriptor fetch is one burst, and receive
// writes are ordered at the slave for free.
//
// The classifier is also where the RESERVED id for receive descriptors
// lives. Section 10: the one class that may not wait must never be
// unable to issue because another class has taken every ID.
// ---------------------------------------------------------------------
module id_classifier
  import memif_pkg::*;
(
  input  logic              clk,
  input  logic              rst_n,

  input  logic              req_valid,
  input  traffic_class_e    req_class,
  input  logic [15:0]       req_seq,           // position within a frame

  output logic              id_valid,
  output logic [3:0]        id_out,

  // Observability. Sections 15 and 16.
  output logic [31:0]       c_assigned [4],
  output logic [31:0]       c_id_exhausted,
  output logic [3:0]        tx_ids_in_use,
  output logic              reserved_id_taken
);

  // The allocation. Section 6's table.
  localparam logic [3:0] ID_TX_DESC = 4'd13;
  localparam logic [3:0] ID_RX_DATA = 4'd14;
  localparam logic [3:0] ID_RX_DESC = 4'd15;   // RESERVED -- Section 10
  localparam int         TX_DATA_IDS = 13;     // 0 .. 12

  logic [TX_DATA_IDS-1:0] tx_busy;
  logic [3:0]             next_free;
  logic                   have_free;

  always_comb begin
    have_free = 1'b0;
    next_free = '0;
    for (int i = TX_DATA_IDS-1; i >= 0; i--)
      if (!tx_busy[i]) begin have_free = 1'b1; next_free = 4'(i); end
  end

  always_comb begin
    id_valid = req_valid;
    unique case (req_class)
      CLASS_TX_DESC: id_out = ID_TX_DESC;
      CLASS_RX_DATA: id_out = ID_RX_DATA;
      CLASS_RX_DESC: id_out = ID_RX_DESC;
      default: begin
        // Transmit data. Octets within one frame must be ordered, so a
        // frame's bursts share an ID; different frames may use different
        // IDs and complete in any order, which is where the parallelism
        // Chapter 18.5 Section 12 paid 44 IDs for actually lives.
        id_out   = next_free;
        id_valid = req_valid && have_free;
      end
    endcase
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      tx_busy <= '0;
      for (int c = 0; c < 4; c++) c_assigned[c] <= '0;
      c_id_exhausted <= '0; reserved_id_taken <= 1'b0;
    end else begin
      if (req_valid && id_valid) begin
        c_assigned[req_class] <= c_assigned[req_class] + 1;
        if (req_class == CLASS_TX_DATA && req_seq == '0)
          tx_busy[next_free] <= 1'b1;
      end

      if (req_valid && !id_valid) c_id_exhausted <= c_id_exhausted + 1;

      // The reserved ID must never be handed to anything else. If it is,
      // Section 10's guarantee is gone and a saturated transmit path can
      // stall the receive descriptor fetch.
      if (req_valid && id_valid && id_out == ID_RX_DESC &&
          req_class != CLASS_RX_DESC)
        reserved_id_taken <= 1'b1;
    end
  end

  assign tx_ids_in_use = 4'($countones(tx_busy));

endmodule

Classification: an allocator whose interesting property is that three of its four outputs are constants.

What it teaches: that an unequal allocation is the right one when the ordering requirements are unequal. Thirteen IDs for one class and one each for three others looks arbitrary; it follows directly from Section 6's table, where only transmit data has a requirement that permits parallelism within itself. An allocator that divides sixteen IDs by four classes gives four each, and eleven of them are never used while transmit data runs out.

And it teaches that reserved_id_taken is the check that Section 10's argument still holds in silicon. The reservation is not enforced by the protocol, the interconnect or the arbiter — it is enforced by this case statement — and a later change that adds a fifth traffic class will reach for the nearest free ID and find 15. The counter exists because the failure is a code change rather than a runtime event.

Deliberately simplified: next_free is a priority encoder written as a descending loop and has no release path in this listing — a real design clears tx_busy[i] when the last response for that ID arrives, which needs Section 9's sink to report it. req_seq is used only to detect the first burst of a frame. And c_assigned is an unpacked array of counters, which is fine in simulation and needs flattening for a register interface.

Production implication: c_id_exhausted is the counter that converts Chapter 18.5 §12's "36% of line rate" from a warning into a measurement. A port reporting it climbing is serialising transmit frames that could have overlapped, and the fix is more IDs from the interconnect — which is a negotiation, not a design change. A port reporting zero has thirteen IDs it is not using, and some of them could be given to a second receive queue. The number is small, it is cheap, and it is the only direct evidence a design gets about an interconnect parameter it does not control.

8. When the Reorder Buffer Fills, in Time

Section 2 gave the three options and chose the third. This section is what the first option actually does, cycle by cycle, because the argument for the third depends on it.

Start at the moment the buffer fills and follow both paths.

The two rows diverge at the point where each buffer runs out, and the two buffers run out at different times.

So the receive path has 2.00 µs and the transmit path has 0.74 µs, which is the wrong way round: the path with more time is the one that cannot use it. A transmit stall is self-limiting — the wire idles, the FIFO stops draining, and everything resumes when the buffer clears. A receive stall is not, because the arrival rate does not depend on anything this design does.

Now the probability, because option 1's defenders say the buffer never fills.

They are nearly right and the "nearly" is the problem. Chapter 18.5 §16 listed five ordinary conditions under which responses reorder — two DRAM banks, a refresh, a competing master, an interleaving switch, a retry — and every one of them is intermittent.

Row three is the correction Section 19 makes. Chapter 18.5 §18 budgeted 7 KiB, which is (O − 1) bursts of 16 beats — and that chapter chose 32-beat bursts. With 2 KiB bursts, 7 KiB holds three and a half of them, so the fourth out-of-order response fills it and Section 2's option 3 has quietly become option 1.

And the failure is invisible until it is not. A design with a 7 KiB buffer and 32-beat bursts passes every test on a quiet interconnect, because four simultaneous out-of-order responses do not happen there. It drops frames during DRAM refreshes, which are periodic, brief, and entirely ordinary — and the drops are reported as CRC errors by Chapter 19.5 §14's mechanism.

9. RTL 4 — The Read Reorder Sink

// ---------------------------------------------------------------------
// read_reorder_sink -- accept every read response, always, and reassemble
// them in order. Sections 2, 8 and 9.
//
// RREADY IS TIED HIGH. That is the whole design decision and everything
// else in this module follows from it. Section 2: deasserting RREADY
// stops the shared read data channel, which stalls the receive
// descriptor fetch, which fills Chapter 19.5's FIFO, which drops frames.
// The buffer is sized at (O-1) bursts so that the condition that would
// make a design want to deassert it cannot arise.
// ---------------------------------------------------------------------
module read_reorder_sink
  import memif_pkg::*;
#(
  parameter int SLOTS = REORDER_BURSTS          // 7
) (
  input  logic              clk,
  input  logic              rst_n,

  input  logic              r_valid,
  input  logic [DATA_W-1:0] r_data,
  input  logic [3:0]        r_id,
  input  logic              r_last,
  output logic              r_ready,            // tied high -- see below

  input  logic [3:0]        expected_id,        // the sequence we need
  output logic              out_valid,
  output logic [DATA_W-1:0] out_data,
  output logic              out_last,

  // Observability. Sections 15 and 16.
  output logic [31:0]       c_in_order,
  output logic [31:0]       c_buffered,
  output logic [3:0]        slots_used,
  output logic [3:0]        peak_slots,
  output logic              would_have_stalled,
  output logic              overflow
);

  // THE decision. A design that makes this conditional has moved a
  // transmit-path throughput problem onto the receive path, where the
  // currency is frames rather than nanoseconds. Section 2.
  assign r_ready = 1'b1;

  logic [DATA_W-1:0] slot_data [SLOTS][BURST_BEATS];
  logic [3:0]        slot_id   [SLOTS];
  logic [7:0]        slot_fill [SLOTS];
  logic              slot_busy [SLOTS];

  int unsigned free_slot;
  logic        have_slot;
  int unsigned match_slot;
  logic        have_match;

  always_comb begin
    have_slot = 1'b0; free_slot = 0;
    for (int s = SLOTS-1; s >= 0; s--)
      if (!slot_busy[s]) begin have_slot = 1'b1; free_slot = unsigned'(s); end

    have_match = 1'b0; match_slot = 0;
    for (int s = 0; s < SLOTS; s++)
      if (slot_busy[s] && slot_id[s] == expected_id) begin
        have_match = 1'b1; match_slot = unsigned'(s);
      end
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int s = 0; s < SLOTS; s++) begin
        slot_busy[s] <= 1'b0; slot_fill[s] <= '0; slot_id[s] <= '0;
      end
      c_in_order <= '0; c_buffered <= '0;
      peak_slots <= '0; would_have_stalled <= 1'b0; overflow <= 1'b0;
    end else begin
      if (r_valid) begin
        if (r_id == expected_id) begin
          // Straight through. On a quiet interconnect this is almost
          // every response and the buffer is never touched.
          c_in_order <= c_in_order + 1;
        end else if (have_slot) begin
          slot_busy[free_slot] <= 1'b1;
          slot_id[free_slot]   <= r_id;
          slot_data[free_slot][slot_fill[free_slot]] <= r_data;
          slot_fill[free_slot] <= slot_fill[free_slot] + 1;
          c_buffered <= c_buffered + 1;
        end else begin
          // Section 2's unreachable state. If Section 5's tracker is
          // doing its job this never happens; if the buffer was sized
          // from the wrong burst length -- Section 8's 7 KiB against
          // 2 KiB bursts -- it happens on the fourth out-of-order
          // response and there is nothing correct left to do.
          overflow <= 1'b1;
          would_have_stalled <= 1'b1;
        end
      end

      if (4'(slots_used) > peak_slots) peak_slots <= 4'(slots_used);
    end
  end

  always_comb begin
    slots_used = '0;
    for (int s = 0; s < SLOTS; s++) if (slot_busy[s]) slots_used = slots_used + 1;
  end

  assign out_valid = r_valid && (r_id == expected_id);
  assign out_data  = r_data;
  assign out_last  = r_last;

endmodule

Classification: a buffer whose most important line is a tie-off.

What it teaches: that assign r_ready = 1'b1; is a design decision with a frame-loss argument behind it, and it will be read as laziness by anybody who has not followed Section 2. The comment above it is load-bearing. A reviewer's instinct on a buffer with a full condition is to add flow control; adding it here is correct AXI, correct for the transmit path, and drops received frames.

And it teaches that would_have_stalled is the diagnostic that names the sizing error. It fires only when the buffer was too small — the condition Section 5's tracker is supposed to make impossible — so a non-zero value says the reorder buffer's sizing does not match the outstanding limit and the burst length. That is a parameter mismatch, and it is Chapter 18.5 §18's 7 KiB against 32-beat bursts exactly.

Deliberately simplified: the slots are a two-dimensional register array of 7 × 32 × 512 bits — 114 688 flops if synthesised as written — where a real design uses a single SRAM with a slot-indexed address. The listing is shaped for clarity and would never be built. out_valid bypasses the buffer entirely for in-order responses and does not drain the slots, which a complete design must do when expected_id advances. And slots_used is a combinational population count in the same always block that writes it, which works only because it is computed from registered state.

Production implication: c_in_order against c_buffered is the interconnect's reordering rate and it is the number to capture before trusting a small buffer. A port reporting c_buffered at zero over a week has an interconnect that never reorders — and a design sized from that observation overflows the first time a competing master is enabled, which is Chapter 18.2 §19's class-75 environment argument arriving in a buffer. peak_slots is the honest number: it says how close the worst observed case came to the bound, and on a correctly sized buffer it is allowed to reach SLOTS without anything being wrong.

10. Resolving the Contradiction

Three options, one survivor, and the survivor is not a mechanism.

Section 2 stated the options and Section 8 priced option 1. This section is the resolution written down, because it is the chapter's reason for existing and it is easy to state badly.

The contradiction, precisely:

All five are true and together they are a contradiction, so one of them must be made vacuous. The candidates are premise one and premise four.

Making premise four false is the tempting engineering answer and it does not work. Give the receive descriptors their own AXI port, or their own channel, or a separate interconnect path — and the sharing is gone.

Row two is the one that catches people. Chapter 18.5 §12 established that an ID is an ordering contract, and a natural extension is to assume it is also a flow-control domain. It is not: RREADY gates the channel, and a master that deasserts it stops every ID at once. The reserved ID in Section 7 keeps the receive path able to issue; it does nothing about its ability to receive.

So premise one is the one to make vacuous, and that is a sizing requirement rather than a mechanism.

reorder_buffer ≥ (O − 1) × burst_octets

With the tracker of Section 5 enforcing O and the shaper of Section 3 refusing to issue without a credit, the buffer cannot fill. The full state becomes dead logic and RREADY ties high.

Row five is what the argument comes down to and it is why this chapter takes a position rather than presenting a trade-off. Seven kilobytes buys the removal of a failure mode whose symptom is intermittent frame loss reported as CRC errors. There is no operating point at which the other side of that trade is the right answer.

And the general habit is worth naming, because it recurs. When a block has a failure state that cannot be handled safely, the design effort belongs at the input that creates the state, not at the output that has to survive it. Section 3's shaper refuses to issue; Section 5's tracker counts; Section 9 accepts everything unconditionally and has no error path at all — which is what a correctly bounded design looks like from the inside.

11. RTL 5 — The Backpressure Arbiter

// ---------------------------------------------------------------------
// backpressure_arbiter -- decide which class gets the next request slot,
// under the constraint that one of them may not be made to wait.
// Sections 10 and 11.
//
// This is NOT a fairness arbiter. Three of the four classes may be
// starved indefinitely without losing anything; the fourth loses frames
// after 391 beats. So the policy is strict priority with a reservation,
// and the interesting part is proving the low-priority classes still
// make progress -- which Section 20's rejected property is about.
// ---------------------------------------------------------------------
module backpressure_arbiter
  import memif_pkg::*;
(
  input  logic              clk,
  input  logic              rst_n,

  input  logic [3:0]        req_pending,       // one bit per class
  input  logic              credit_available,
  input  logic [15:0]       rx_fifo_occupancy,
  input  logic [15:0]       rx_almost_full,

  output logic              grant_valid,
  output traffic_class_e    grant_class,

  // Observability. Sections 15 and 16.
  output logic [31:0]       c_grants [4],
  output logic [31:0]       c_starved_cycles [4],
  output logic [31:0]       longest_wait [4],
  output logic              urgent_mode,
  output logic              starvation_suspected
);

  logic [15:0] wait_cnt [4];

  // URGENT MODE. When Chapter 19.5's receive FIFO passes its almost-full
  // level the receive classes take absolute priority, because past that
  // point every cycle of delay is 64 octets closer to a drop. Below it
  // the ordinary priority applies and transmit gets a share.
  assign urgent_mode = (rx_fifo_occupancy >= rx_almost_full);

  always_comb begin
    grant_valid = 1'b0;
    grant_class = CLASS_TX_DATA;

    if (credit_available) begin
      if (urgent_mode) begin
        if      (req_pending[CLASS_RX_DESC]) begin grant_class = CLASS_RX_DESC; grant_valid = 1'b1; end
        else if (req_pending[CLASS_RX_DATA]) begin grant_class = CLASS_RX_DATA; grant_valid = 1'b1; end
        else if (req_pending[CLASS_TX_DATA]) begin grant_class = CLASS_TX_DATA; grant_valid = 1'b1; end
        else if (req_pending[CLASS_TX_DESC]) begin grant_class = CLASS_TX_DESC; grant_valid = 1'b1; end
      end else begin
        // Ordinary priority. Receive descriptors still lead, because a
        // descriptor fetch issued early is a drop avoided later, and it
        // is one burst.
        if      (req_pending[CLASS_RX_DESC]) begin grant_class = CLASS_RX_DESC; grant_valid = 1'b1; end
        else if (req_pending[CLASS_TX_DATA]) begin grant_class = CLASS_TX_DATA; grant_valid = 1'b1; end
        else if (req_pending[CLASS_RX_DATA]) begin grant_class = CLASS_RX_DATA; grant_valid = 1'b1; end
        else if (req_pending[CLASS_TX_DESC]) begin grant_class = CLASS_TX_DESC; grant_valid = 1'b1; end
      end
    end
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int c = 0; c < 4; c++) begin
        c_grants[c] <= '0; c_starved_cycles[c] <= '0;
        longest_wait[c] <= '0; wait_cnt[c] <= '0;
      end
      starvation_suspected <= 1'b0;
    end else begin
      for (int c = 0; c < 4; c++) begin
        if (grant_valid && grant_class == traffic_class_e'(c)) begin
          c_grants[c] <= c_grants[c] + 1;
          if (wait_cnt[c] > longest_wait[c]) longest_wait[c] <= wait_cnt[c];
          wait_cnt[c] <= '0;
        end else if (req_pending[c]) begin
          wait_cnt[c]        <= wait_cnt[c] + 1;
          c_starved_cycles[c] <= c_starved_cycles[c] + 1;
        end
      end

      // Transmit descriptors are the lowest priority in both modes, so
      // they are the class that would starve if anything did. This is a
      // REPORT and not a guarantee -- Section 20's rejected property is
      // about why a guarantee cannot be written as a local assertion.
      if (wait_cnt[CLASS_TX_DESC] > 16'd10000) starvation_suspected <= 1'b1;
    end
  end

endmodule

Classification: a priority arbiter, and the block whose correctness cannot be established by looking at it.

What it teaches: that strict priority is the right policy here and that fairness would be the wrong one. Three of the four classes lose time when starved; one loses frames. A round-robin arbiter treats them equally and gives the receive descriptor class a quarter of the slots, which is three times more than it needs and one delay too many when the FIFO is at 391 beats. The asymmetry in what starvation costs is what makes an unfair arbiter correct.

And it teaches that urgent_mode is a second policy rather than a threshold. Below the almost-full level the arbiter balances; above it the receive classes take everything, because Chapter 19.5 §9's level is exactly the point past which delay converts into loss. The two policies share an arbiter and differ in one row of their priority order — receive data moving above transmit data — which is a small change with a specific trigger and is the shape most real arbiters have.

Deliberately simplified: the priority is a fixed if/else chain with no anti-starvation escape at all, so transmit descriptors can in principle wait forever; starvation_suspected reports it rather than preventing it, and a production design adds an aging counter that promotes a starved class after a bounded wait. longest_wait saturates silently at 16 bits. And urgent_mode is combinational from an occupancy that crosses from another clock domain, which needs Chapter 19.5 §7's synchroniser and is elided here.

Production implication: longest_wait[CLASS_TX_DESC] is the number that says whether the strict priority is costing anything. A port reporting a few hundred cycles is fine — a transmit descriptor fetched 500 cycles late delays a frame by 2.6 µs and nothing notices. A port reporting tens of thousands has a receive path saturated enough to starve transmit entirely, which is a real operating point on an asymmetric link and is the case where the aging counter this listing omits earns its flops. The counter distinguishes "the policy is working" from "the policy has taken over."

12. The Write Path, and Why It Is Easier

Everything so far has been about reads. The receive path's frame data is a write, and writes are simpler for three reasons that are worth separating because only one of them is about AXI.

Row four is the whole difference and rows one to three are why. A read response that arrives early must be kept; a write response that arrives early is a bit. So the reorder problem that costs 14 KiB on the read path costs nothing on the write path, and Chapter 18.5 §12's ordering contract is satisfied by the slave rather than by this design.

But the write path has a problem the read path does not, and it is Chapter 18.4 §3's in a new place.

A write that has been issued is not a write that has been seen. Chapter 18.3 §19's class 76 — asserting an effect because its cause was issued — and on the receive path it decides when a descriptor may be marked done.

Row four is the one a design gets wrong by one. A frame is several bursts; the descriptor may be marked done only when the last of them has responded, and a design that marks on the last issued burst tells the driver a frame is ready while part of it is still in the interconnect. Chapter 18.2 §19's class 75 is the driver reading it out of order; this is the driver reading it too early, which is a different failure with the same symptom: a frame in host memory that is partly stale.

And the ordering between the two writes is the part AXI does not help with.

These must be seen by the host in that order and they have different addresses, so Chapter 18.5 §12's same-ID ordering is the only mechanism that guarantees it — and Section 6 gave receive data and receive descriptors different IDs. So the ordering must come from sequencing rather than from the bus: wait for BVALID on the data, then issue the descriptor write. That is a round trip of memory latency per frame, and at 148.8 M frames per second it is not affordable per frame — which is why Chapter 18.6's coalescing batches descriptors, and why the batching is a correctness mechanism as much as an efficiency one.

13. RTL 6 — The Write Commit Tracker

// ---------------------------------------------------------------------
// write_commit_tracker -- know when a frame is actually in memory, which
// is later than when it was written. Sections 12 and 13.
//
// Chapter 18.3 Section 19's class 76: custody is not visibility. A frame is
// several bursts and the descriptor may be marked done only when the
// LAST of them has returned BVALID -- not when the last was issued, and
// not when the first responded.
//
// The block also enforces the ordering AXI will not: frame data before
// descriptor, across two different IDs, by sequencing rather than by the
// bus. Section 12.
// ---------------------------------------------------------------------
module write_commit_tracker
  import memif_pkg::*;
#(
  parameter int FRAMES_IN_FLIGHT = 8
) (
  input  logic              clk,
  input  logic              rst_n,

  input  logic              frame_start,
  input  logic [3:0]        frame_tag,
  input  logic              burst_issued,
  input  logic [3:0]        burst_tag,
  input  logic              frame_last_burst,

  input  logic              b_valid,
  input  logic [3:0]        b_tag,
  input  logic [1:0]        b_resp,

  output logic              frame_committed,
  output logic [3:0]        committed_tag,
  output logic              may_write_descriptor,

  // Observability. Sections 15 and 16.
  output logic [31:0]       c_frames_committed,
  output logic [31:0]       c_write_errors,
  output logic [15:0]       mean_commit_ns_x10,
  output logic              descriptor_before_data,
  output logic              commit_overflow
);

  logic [7:0]  issued   [FRAMES_IN_FLIGHT];
  logic [7:0]  responded[FRAMES_IN_FLIGHT];
  logic        closed   [FRAMES_IN_FLIGHT];   // last burst has been issued
  logic        busy     [FRAMES_IN_FLIGHT];
  logic [31:0] started  [FRAMES_IN_FLIGHT];
  logic [31:0] now;
  logic [31:0] commit_ns_sum;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int f = 0; f < FRAMES_IN_FLIGHT; f++) begin
        issued[f] <= '0; responded[f] <= '0;
        closed[f] <= 1'b0; busy[f] <= 1'b0; started[f] <= '0;
      end
      now <= '0; commit_ns_sum <= '0;
      frame_committed <= 1'b0; committed_tag <= '0;
      c_frames_committed <= '0; c_write_errors <= '0;
      descriptor_before_data <= 1'b0; commit_overflow <= 1'b0;
    end else begin
      now <= now + 1;
      frame_committed <= 1'b0;

      if (frame_start) begin
        if (busy[frame_tag]) commit_overflow <= 1'b1;
        busy[frame_tag]      <= 1'b1;
        issued[frame_tag]    <= '0;
        responded[frame_tag] <= '0;
        closed[frame_tag]    <= 1'b0;
        started[frame_tag]   <= now;
      end

      if (burst_issued) begin
        issued[burst_tag] <= issued[burst_tag] + 1;
        if (frame_last_burst) closed[burst_tag] <= 1'b1;
      end

      if (b_valid) begin
        responded[b_tag] <= responded[b_tag] + 1;
        if (b_resp != 2'b00) c_write_errors <= c_write_errors + 1;

        // COMMIT. Every burst issued, the last one known to be the last,
        // and every one responded. A design that drops the `closed` term
        // commits a frame whose remaining bursts have not been issued
        // yet -- which happens whenever the interconnect is fast and the
        // shaper is slow, and is therefore a rate-dependent bug.
        if (closed[b_tag] && (responded[b_tag] + 8'd1 == issued[b_tag])) begin
          frame_committed    <= 1'b1;
          committed_tag      <= b_tag;
          busy[b_tag]        <= 1'b0;
          c_frames_committed <= c_frames_committed + 1;
          commit_ns_sum      <= commit_ns_sum + (now - started[b_tag]);
        end
      end
    end
  end

  // The descriptor write is gated on the data's commit. Section 12: the
  // two have different IDs, so the bus guarantees nothing and the
  // ordering has to be a sequence.
  assign may_write_descriptor = frame_committed;

  assign mean_commit_ns_x10 = (c_frames_committed == 0) ? 16'd0
    : 16'((commit_ns_sum * 32'd51) / c_frames_committed);   // 5.12 ns/cycle

endmodule

Classification: an ordering enforcer built out of counters, and the only block in this chapter that is about the host's view rather than the bus's.

What it teaches: that closed is the term a design forgets. Counting responses against issues is the obvious commit test and it is wrong: early in a frame, responded + 1 == issued is true after the first burst responds, so a frame commits after one burst. The closed flag says "no more bursts are coming", and without it the bug appears only when the interconnect responds faster than the shaper issues — which is a rate-dependent bug and therefore absent from the simulations that matter.

And it teaches that may_write_descriptor is a sequencing signal doing a job AXI cannot. Chapter 18.5 §12's same-ID ordering is the bus's only ordering guarantee, and Section 6 deliberately gave frame data and descriptors different IDs because they have no ordering requirement between frames. Within a frame they do, and the only mechanism left is to wait. A design that shares an ID instead gets the ordering free and loses the parallelism that the thirteen transmit IDs were for.

Deliberately simplified: now is a free-running cycle counter that wraps at 2³², so a commit measured across a wrap is nonsense — a production version uses a difference computed at capture. mean_commit_ns_x10 multiplies by 51 for 5.12 ns per cycle, which is a fixed-point approximation good to 0.2%. The tracker holds eight frames and commit_overflow is the only response to a ninth, which is adequate because Chapter 18.6's coalescing bounds the outstanding frame count anyway.

Production implication: mean_commit_ns_x10 is the memory system's write latency measured through the MAC, and it is a number the memory team does not otherwise have. Chapter 18.1 §8's stall table is a read property; this is the write side, and it is what decides how early a receive descriptor can be marked done — which feeds directly into Chapter 18.6's coalescing window. A port reporting a mean commit of 800 ns cannot mark descriptors done at the frame rate, and the batching that follows is a consequence of this counter rather than a policy choice.

14. What the Interface Must Never Do

Six prohibitions, and the first is the chapter.

Row two deserves the emphasis it gets, because it is the one failure in Module 19 with no downstream detection at all. A frame assembled from a discarded read response is structurally valid; Chapter 19.3 pads it, Chapter 19.4 computes a correct check value over it, and it leaves the port with a valid FCS and wrong data. Every check in the MAC passes. The receiver's checks pass. The error surfaces at the application layer, if anywhere, and there is no counter anywhere in this design that would have incremented.

Row three is how rows one and two become reachable. Neither is a decision a designer makes; both are what a design is forced into when the reorder buffer fills, and the buffer fills only if something issued a request it had no room for. The prohibition that matters is the one at the input.

Row five is a code-review prohibition rather than a runtime one. Nothing enforces the reservation except Section 7's case statement, and the natural way to add a fifth traffic class is to take the next free ID — which is 15. reserved_id_taken exists because the failure is a future change rather than a current bug.

And the two symptoms this chapter is blamed for and should not be:

15. RTL 7 — Interface Telemetry

// ---------------------------------------------------------------------
// memif_telemetry -- four traffic classes, and the useful numbers are
// the ones that compare them. Section 15.
//
// Operator view: bus utilisation and errors.
// Structural view: per-class grants, waits and reorder depth.
// Derived: the outstanding ceiling in nanoseconds, which is the number
// that says whether the interconnect's limit is what is binding.
// ---------------------------------------------------------------------
module memif_telemetry
  import memif_pkg::*;
(
  input  logic              clk,
  input  logic              rst_n,

  input  logic [31:0]       c_bursts,
  input  logic [31:0]       c_full_bursts,
  input  logic [31:0]       c_beats_requested,
  input  logic [31:0]       c_in_order,
  input  logic [31:0]       c_buffered,
  input  logic [3:0]        peak_slots,
  input  logic [3:0]        peak_outstanding,
  input  logic [31:0]       c_credit_stalls,
  input  logic [31:0]       c_id_exhausted,
  input  logic [31:0]       c_write_errors,
  input  logic [31:0]       longest_wait_tx_desc,
  input  logic              urgent_mode,

  // Operator view.
  output logic [31:0]       c_bus_errors,
  output logic [15:0]       bus_efficiency_pct,

  // Structural view.
  output logic [15:0]       reorder_rate_x10000,
  output logic [15:0]       mean_burst_beats_x10,
  output logic [31:0]       c_urgent_cycles,

  // Derived. Section 4.
  output logic [15:0]       outstanding_ceiling_ns,
  output logic              ceiling_is_binding,
  output logic              reorder_never_seen
);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) c_urgent_cycles <= '0;
    else if (urgent_mode) c_urgent_cycles <= c_urgent_cycles + 1;
  end

  assign c_bus_errors = c_write_errors;

  // Chapter 18.5 Section 4's efficiency: data beats over total beats,
  // where a transaction costs a fixed overhead. Here it is reported as
  // the share of bursts that were FULL, which is the shaper's own
  // contribution and is what Section 3 can influence.
  assign bus_efficiency_pct = (c_bursts == 0) ? 16'd0
    : 16'((c_full_bursts * 32'd100) / c_bursts);

  assign mean_burst_beats_x10 = (c_bursts == 0) ? 16'd0
    : 16'((c_beats_requested * 32'd10) / c_bursts);

  // How often the interconnect reorders. Section 9: a design sized from
  // an observed rate of zero overflows the first time a competing master
  // is enabled, so this number is evidence and never a licence.
  assign reorder_rate_x10000 = ((c_in_order + c_buffered) == 0) ? 16'd0
    : 16'((c_buffered * 32'd10000) / (c_in_order + c_buffered));

  assign reorder_never_seen = (c_buffered == '0) &&
                              (c_in_order > 32'd1_000_000);

  // Section 4: O bursts of BURST_OCTETS in flight covers R x L octets of
  // latency. At 8 x 2 KiB and 100 Gb/s that is 1.31 us -- and a memory
  // system slower than this cannot be kept fed whatever else is done.
  assign outstanding_ceiling_ns =
    16'((32'(MAX_OUTSTANDING) * 32'(BURST_OCTETS) * 32'd8) / 32'd100);

  assign ceiling_is_binding = (peak_outstanding >= 4'(MAX_OUTSTANDING)) &&
                              (c_credit_stalls > 32'd1000);

endmodule

Classification: an observability block whose most valuable output is a constant.

What it teaches: that outstanding_ceiling_ns never changes and belongs in telemetry anyway. It is O × burst × 8 / R — 1 310 ns at this configuration — and it is computed from parameters, not measured. Putting it in a readable register means a system integrator investigating "why is this port at 80% of line rate" can read the answer instead of deriving it, and the answer is usually that the memory latency exceeds it. A constant is telemetry when the person reading it does not have the parameters.

And it teaches that reorder_never_seen is a warning about the testbench, not the design. A port that has processed a million in-order responses and buffered none has an interconnect that has never reordered — and Section 9's argument is that this is exactly the observation that leads to a 7 KiB buffer. The flag exists so the observation is recorded as a fact about the environment rather than absorbed as a fact about the design.

Deliberately simplified: bus_efficiency_pct reports the full-burst share rather than Chapter 18.5 §4's 24 / (24 + bursts) efficiency model, which is a different quantity and is the one this block can influence. A complete design reports both and a reviewer needs to know which is which. outstanding_ceiling_ns hard-codes 100 Gb/s in a divide by 100. And c_urgent_cycles counts without recording how long each episode was, losing the shape that Chapter 19.5 §14's event-versus-beat pair preserves.

Production implication: the pair to read together is ceiling_is_binding and c_credit_stalls. Binding with stalls means the interconnect's outstanding limit is the constraint and the fix is a negotiation — Chapter 18.5 §18's "wanted 5 to 67, typically offered 8." Not binding with stalls means something else is limiting issuance, usually Section 3's one-request-at-a-time shaper, which is a design change this team can make. The two readings send the investigation to different people, which is the whole reason the derived flag exists rather than just the counter.

16. RTL 8 — The Interface Conformance Monitor

// ---------------------------------------------------------------------
// memif_conformance_monitor -- verdicts. Section 16.
//
// Six verdicts, and one of them is about a SIZING parameter rather than
// about behaviour, because this chapter's central failure is a parameter
// mismatch that only appears when an interconnect reorders.
// ---------------------------------------------------------------------
module memif_conformance_monitor
  import memif_pkg::*;
#(
  parameter int REORDER_OCTETS_CFG = REORDER_OCTETS,
  parameter int MIN_REORDER_EVENTS = 100
) (
  input  logic              clk,
  input  logic              rst_n,

  input  logic              overflow,
  input  logic              would_have_stalled,
  input  logic              bound_violated,
  input  logic              reserved_id_taken,
  input  logic              descriptor_before_data,
  input  logic              commit_overflow,
  input  logic [31:0]       c_buffered,
  input  logic [31:0]       c_frames_committed,
  input  logic [3:0]        peak_slots,

  output logic              data_may_be_corrupt,
  output logic              buffer_undersized,
  output logic              ordering_violated,
  output logic              isolation_broken,
  output logic              reorder_untested,
  output logic              none_of_the_above
);

  // The sizing check, at elaboration. Section 10: the buffer must hold
  // (O-1) bursts of the CONFIGURED burst length. Chapter 18.5 Section 18
  // budgeted 7 KiB against 32-beat bursts, which holds 3.5 of them.
  localparam int REQUIRED_OCTETS = (MAX_OUTSTANDING - 1) * BURST_OCTETS;

  always_comb begin
    // Correctness. Either of these means frames are wrong, not late.
    data_may_be_corrupt = overflow || descriptor_before_data;

    // The sizing. This is a STATIC verdict and it is the one that
    // matters, because the dynamic symptom appears only during a DRAM
    // refresh on a busy interconnect.
    buffer_undersized = (REORDER_OCTETS_CFG < REQUIRED_OCTETS) ||
                        would_have_stalled;

    ordering_violated = bound_violated || commit_overflow;
    isolation_broken  = reserved_id_taken;

    // Coverage. Section 9: an interconnect that never reorders has not
    // tested the sink, and a design sized from that run is sized from
    // nothing.
    reorder_untested = (c_frames_committed > 32'd10000) &&
                       (c_buffered < 32'(MIN_REORDER_EVENTS));

    none_of_the_above = !data_may_be_corrupt && !buffer_undersized &&
                        !ordering_violated && !isolation_broken &&
                        !reorder_untested;
  end

endmodule

Classification: a verdict generator, the twelfth in Modules 18 and 19, and the first whose most important verdict is decided at elaboration.

What it teaches: that buffer_undersized fires on a design that has never misbehaved. Its first term is a comparison of two parameters and needs no traffic at all; a design with 7 KiB and 32-beat bursts asserts it on the first cycle after reset, before any response has arrived. That is the right time to find this bug, because the alternative is finding it when a DRAM refresh coincides with a busy interconnect on a customer's board.

And it teaches that reorder_untested is the verdict that will be argued about. A regression on a well-behaved bus functional model produces zero out-of-order responses, so the verdict fires on a perfectly correct design and a verification lead will want it removed. It should not be: Section 9's whole argument is that the sizing decision gets made from the observed reordering rate, and a rate of zero observed over ten thousand frames is the input that produces a 7 KiB buffer. The verdict is the place that observation is recorded.

Deliberately simplified: REORDER_OCTETS_CFG defaults to the package's correct value, so the verdict only fires if an integrator overrides it — which is exactly what happened between Chapter 18.5 §16 and §18, and the default being right makes the check weaker than it should be. data_may_be_corrupt merges two unrelated failures because both mean the same thing to a consumer. And there is no verdict for the arbiter at all, because Section 11's starvation is a report rather than a conformance property — Section 20's rejected class is why.

Production implication: none_of_the_above for the twelfth time, and this instance is the cheapest of them to satisfy and the easiest to satisfy falsely. A port asserting it has a correctly sized reorder buffer, an enforced outstanding bound, an intact ID reservation, ordered commits, and at least a hundred observed reorderings. The last clause is the one that makes the first mean anything: without it, "the buffer never overflowed" is a statement about the interconnect rather than about the buffer — which is Chapter 18.2 §19's class 75 in its natural habitat.

17. The Same Interface at Other Line Rates

Every number in this chapter so far is a 100 Gb/s number. The structure holds at other rates and two of the terms move in opposite directions, which changes which failure a design has to worry about.

Columns three and four cross between 25 and 100 Gb/s, and the crossing is the section's point.

Column three is how long the receive path can survive a stalled read channel. It falls slowly — 15.56 µs to 2.25 µs across a factor of 400 in line rate — because Chapter 19.5 §10's FIFO grows almost as fast as the rate does.

Column four is how much memory latency the interface can hide. It falls in proportion to the rate, because the outstanding limit and the burst length are both fixed by the interconnect. 131 µs at 1 Gb/s and 0.33 µs at 400.

Row four is a different design problem at each end. A 1 Gb/s port with eight outstanding 2 KiB bursts can hide more latency than any memory system will ever exhibit, so its risk is entirely that something stalls the channel for 15 µs. A 400 Gb/s port cannot hide 330 ns, which is less than an ordinary DRAM access — so it is throughput-limited before anything stalls at all, and the reorder buffer's contents are the least of its problems.

And column five says the memory this chapter spends becomes negligible as rates rise. The reorder buffer is 14 KiB at every rate, because (O − 1) × burst has no rate in it. At 1 Gb/s that is 56.2% of the port's on-chip memory and at 400 Gb/s it is 10.5% — so the 7 KiB this chapter argues for is a serious fraction of a slow port and a rounding error on a fast one.

Which inverts where the argument is hardest to win. The failure the 14 KiB prevents is worst at 100 Gb/s and above, where frames are lost fastest; the cost is worst at 1 Gb/s, where the buffer is more than half the port's SRAM. A slow port is where the trade-off is genuinely uncomfortable, and the honest answer there is to reduce the outstanding limit rather than the buffer: O = 4 gives a 6 KiB buffer and still hides 65.5 µs of latency at 1 Gb/s, which is ample.

18. What the Interface Assumes

Ten assumptions, and the first two are the ones that make every other section's argument valid.

Row two is unusual and worth dwelling on: it is an assumption whose falsity would be good news. An interconnect that gives the MAC two genuinely independent read paths — separate ports that are not merged downstream — removes the contradiction entirely, and this chapter's 7 KiB could be saved. No mainstream SoC interconnect offers that, and the ones that appear to usually merge at the memory controller, so the assumption holds by default rather than by design.

Row ten is the one that reads oddly in a list of assumptions and it is real. Section 16's reorder_untested cannot be cleared on a system whose interconnect never reorders, and such systems exist — a single-master point-to-point link to a single memory bank. On those the verdict is permanently asserted and the design is permanently correct, which is a monitor giving a false alarm for a true reason. The right response is to disable the verdict with a documented justification rather than to lower its threshold, because lowering it makes the check meaningless everywhere else.

And one thing deliberately not assumed:

That is the difference between this chapter and Chapter 18.5. That chapter sized against latency and had to argue about distributions. This one sizes against a count the design itself enforces, so the sizing is exact and the latency only affects throughput. Sizing against something you control rather than something you observe is what makes a bound provable.

19. The Cost, Accounted — and a 7 KiB Correction

Eight blocks, and the chapter's memory is the item that changes a published total.

The reorder buffer is the block and everything else is control. The sink's 14 KiB is an SRAM with a slot-indexed address, not the register array Section 9's listing shows — that listing would be 114 688 flops and nobody builds it.

Now the correction. Chapter 18.5 §18 listed both sizes and used the smaller in its total:

Row four is the error and it is a pairing error rather than an arithmetic one. Both figures are correct; the wrong one was paired with the chosen burst length, which is the kind of mistake a table of alternatives invites. Chapter 18.5 §16 even names the coupling — "burst length and reorder buffer are the same decision" — and §18 then decoupled them by two lines of a table.

And Section 8 says what the error costs rather than what it measures. A 7 KiB buffer holds 3.5 of the 2 KiB bursts, so the fourth out-of-order response overflows it — and Section 2's options are then options 1 and 2, both of which lose received frames. The correction is not a budget tidy-up; it is the difference between a design whose failure state is unreachable and one whose failure state is reached during DRAM refreshes.

Module 19's running total, with five chapters built:

Five chapters, 92.0% of the flop estimate, and 34% more memory than it allowed — because §18 counted the FIFOs and not the reorder buffer, which it had inherited from Chapter 18.5's side of the boundary. The reorder buffer belongs to whichever module owns the bus interface, and it was counted in neither.

Chapter 19.7's twenty counters are what is left, and at roughly 1 000 flops the module will land within a few per cent of Chapter 19.1 §18's figure — which is a better outcome than the CRC estimate managed and is worth noting as the exception rather than the rule.

20. Properties Worth Asserting, and One Worth Refusing

Thirty-two properties in five groups, and the refused one is a progress property that is true of a block and false of a system.

Group 1 — the shaper.

// No burst exceeds Chapter 18.5's chosen length.
a_burst_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  issue |-> (issued.len_beats <= 8'(BURST_BEATS)));

// No burst crosses a 4 KiB boundary.
a_no_4k_cross: assert property (@(posedge clk) disable iff (!rst_n)
  issue |-> ((issued.addr[11:0] + 12'(issued.len_beats) * 12'(DATA_B)) <= 12'd4096));

// Never issue without a credit. Section 10's whole safety argument.
a_credit_required: assert property (@(posedge clk) disable iff (!rst_n)
  issue |-> credit_available);

// The cursor advances by exactly what was issued.
a_cursor_advances: assert property (@(posedge clk) disable iff (!rst_n)
  issue |=> (cursor == $past(cursor) + $past(this_len) * DATA_B));

// A frame's last burst is marked and only the last one is.
a_last_marked: assert property (@(posedge clk) disable iff (!rst_n)
  (issue && issued.last_of_frame) |=> !active);

Group 2 — the outstanding bound, which is what makes the buffer's full state unreachable.

// The bound Section 5 exists to enforce.
a_bound_holds: assert property (@(posedge clk) disable iff (!rst_n)
  (outstanding[0] + outstanding[1] + outstanding[2] + outstanding[3])
    <= MAX_OUTSTANDING);

// Issues and completions balance.
a_balance: assert property (@(posedge clk) disable iff (!rst_n)
  c_issued >= c_completed);

// A completion never arrives for a class with nothing outstanding.
a_no_orphan_response: assert property (@(posedge clk) disable iff (!rst_n)
  response |-> (outstanding[response_class] != '0));

// The receive-descriptor reservation. Section 7.
a_rx_desc_can_issue: assert property (@(posedge clk) disable iff (!rst_n)
  (req_pending[CLASS_RX_DESC] && outstanding[CLASS_RX_DESC] == '0)
    |-> credit_available);

// And the bound is never violated, which would make Section 10 false.
a_never_violated: assert property (@(posedge clk) disable iff (!rst_n)
  !bound_violated);

Group 3 — the sink, where the most important property is about a constant.

// RREADY IS TIED HIGH. Section 9: this is the chapter's design decision
// and it is assertable.
a_rready_high: assert property (@(posedge clk) r_ready == 1'b1);

// Slots never exceed the bound the buffer was sized for.
a_slots_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  slots_used <= 4'(SLOTS));

// Overflow never happens. Not because it is handled -- because Group 2
// makes it unreachable.
a_no_overflow: assert property (@(posedge clk) disable iff (!rst_n)
  !overflow);

// The sizing, at elaboration. Section 19's correction, as a check.
a_sized_correctly: assert property (@(posedge clk)
  REORDER_OCTETS >= (MAX_OUTSTANDING - 1) * BURST_OCTETS);

// An in-order response bypasses the buffer entirely.
a_in_order_bypasses: assert property (@(posedge clk) disable iff (!rst_n)
  (r_valid && r_id == expected_id) |-> (out_valid && !$changed(slots_used)));

// Buffered responses come out in sequence order, never in arrival order.
a_emitted_in_order: assert property (@(posedge clk) disable iff (!rst_n)
  out_valid |-> ($past(expected_id) != expected_id || out_last));

Group 4 — ordering and commitment.

// A descriptor is written only after its frame's data has committed.
a_data_before_descriptor: assert property (@(posedge clk) disable iff (!rst_n)
  may_write_descriptor |-> frame_committed);

// A frame commits only when every issued burst has responded AND the
// last burst is known to be the last. The `closed` term is the one a
// design forgets -- Section 13.
a_commit_complete: assert property (@(posedge clk) disable iff (!rst_n)
  frame_committed |-> (closed[committed_tag] &&
                       (responded[committed_tag] == issued[committed_tag])));

// Custody is not visibility. Chapter 18.3's class 76, asserted.
a_not_on_issue: assert property (@(posedge clk) disable iff (!rst_n)
  burst_issued |-> !frame_committed);

// A write error is counted and never silently absorbed.
a_write_error_counted: assert property (@(posedge clk) disable iff (!rst_n)
  (b_valid && b_resp != 2'b00) |=> $changed(c_write_errors));

// Two frames never share a tag while both are in flight.
a_tag_unique: assert property (@(posedge clk) disable iff (!rst_n)
  (frame_start && busy[frame_tag]) |-> commit_overflow);

// The reserved ID goes only to receive descriptors.
a_reserved_id: assert property (@(posedge clk) disable iff (!rst_n)
  (id_valid && id_out == 4'd15) |-> (req_class == CLASS_RX_DESC));

// Ordering domains never share an ID.
a_domains_distinct: assert property (@(posedge clk) disable iff (!rst_n)
  (id_valid && req_class != CLASS_TX_DATA) |-> (id_out >= 4'd13));

Group 5 — the arbiter, where every property is about a single grant.

// A grant requires a pending request and a credit.
a_grant_needs_request: assert property (@(posedge clk) disable iff (!rst_n)
  grant_valid |-> (req_pending[grant_class] && credit_available));

// One grant at a time.
a_single_grant: assert property (@(posedge clk) disable iff (!rst_n)
  grant_valid |-> $onehot(4'b0001 << grant_class));

// In urgent mode a pending receive request always wins.
a_urgent_priority: assert property (@(posedge clk) disable iff (!rst_n)
  (urgent_mode && grant_valid && req_pending[CLASS_RX_DESC])
    |-> (grant_class == CLASS_RX_DESC));

// Urgent mode tracks the FIFO's level and nothing else.
a_urgent_from_fifo: assert property (@(posedge clk) disable iff (!rst_n)
  urgent_mode == (rx_fifo_occupancy >= rx_almost_full));

// Receive descriptors lead in both modes.
a_rx_desc_leads: assert property (@(posedge clk) disable iff (!rst_n)
  (grant_valid && req_pending[CLASS_RX_DESC]) |-> (grant_class == CLASS_RX_DESC));

Group 6 — coverage.

c_reorder_seen:     cover property (@(posedge clk) c_buffered > 0);
c_slots_full:       cover property (@(posedge clk) slots_used == 4'(SLOTS));
c_urgent:           cover property (@(posedge clk) urgent_mode);
c_id_exhausted:     cover property (@(posedge clk) c_id_exhausted > 0);
c_multi_burst_frame: cover property (@(posedge clk)
                      frame_committed && issued[committed_tag] > 8'd1);
c_credit_stall:     cover property (@(posedge clk) !credit_available);

21. Verification Scenarios

Fifty-seven scenarios, plus a five-run directed test whose content is an interconnect's reordering behaviour.

The shaper — 10 scenarios.