Throughput Problems

Every chapter in Module 21 so far has had an event to count. This one has none: the link is correct, nothing is dropped, nothing is damaged, and the number is smaller than the rate on the label.

Six for six, and the throughput is 65 Gb/s on a 100 Gb/s port.

Because four ceilings sit below the line rate and only one of them is ever quoted.

The fourth is the one nobody computes and it is frequently the binding one.

achievable rate = outstanding × burst / memory latency

Row two is an ordinary interconnect — Chapter 18.5 §16's eight outstanding, a kilobyte burst, a microsecond of memory latency — and it caps a 100 Gb/s port at 65.5% with nothing dropped, nothing damaged, and every counter in Module 21 reading zero.

And one existing register separates that from a framing limit.

Chapter 21.7 §5's outstanding count, pinned at its maximum while the throughput is below the line rate, is a memory-side ceiling. Normal, and it is not.

1. Scope, and Every Counter Reading Zero

Scope: a link that is correct and slower than its rate implies. The four ceilings, which one is binding, and the single reading that separates a memory-side limit from a protocol-side one.

Not in scope: a link that is losing something. Chapter 21.6 owns silent drops and Chapter 21.5 owns a duplex mismatch's loss — and the boundary is arithmetic: if frames_in exceeds frames_delivered, this is not the chapter. Nor a link that is down: Chapter 21.4 owns that, and its boundary is link_up.

Row four is a diagnosis by elimination and that is unusual enough to state. Every other chapter in Module 21 begins from an event — a dropped frame, a failing check, a collision, a stall. This one begins from an absence of events and a comparison between two numbers, one of which is on a label.

And the comparison is harder than it looks, because "the rate on the label" is four different numbers.

A complaint of "we are only getting 60 Gb/s on a 100 Gb/s link" does not say which of the four is meant, and Chapter 8.3 §4's throughput-against-goodput distinction is the first half of sorting it out. The second half is this chapter's: which ceiling is actually binding.

Three things this chapter is and is not.

Row three is the most important and the least satisfying. A link carrying 30 Gb/s because nothing is sending more than 30 Gb/s is not slow; it is idle, and distinguishing that from a ceiling is Section 13's whole subject — because it needs evidence about what was offered, which the receiver does not have.

2. Four Ceilings, and Which One Binds

Every ceiling below the line rate is arithmetic over parameters, so all four are computable before any measurement.

Ceiling 1 — the line rate. 100 Gb/s of octets on the wire, including preamble, framing and the interframe gap. Nothing in the design changes it.

Ceiling 2 — framing efficiency. Chapter 8.3 §2: a frame carries p payload octets in p + 38 on the wire.

A link carrying minimum-size frames delivers 54.8 Gb/s of payload at full line rate, and that is not a fault — it is Chapter 8.3 §2's curve, and a complaint of "only 55%" on minimum-size traffic has already been answered.

Ceiling 3 — the frame-rate ceiling. Chapter 8.3 §6: a design that can begin one frame every k cycles cannot exceed f/k frames per second, whatever the line rate. Chapter 19.1 §4 gives the budget: 1.312 cycles per frame at 100 Gb/s and minimum size, so a design needing two cycles per frame is capped at 65.6% on that traffic and is unconstrained on maximum-size traffic, where the budget is 24.03 cycles.

Ceiling 4 — the outstanding limit. Chapter 19.6 §4's Little's law, rearranged:

achievable rate = outstanding × burst / memory latency

And ceilings 2, 3 and 4 depend on completely different things, which is what makes them separable.

Row three is the discriminator and it is the chapter's method in one line: the memory-side ceiling is a rate in octets per second and does not move when the frame size does. The two protocol-side ceilings both move with size — one steeply below 256 octets, the other linearly all the way up — so sweeping the frame size separates them in one experiment.

3. RTL 1 — The Throughput Package and the Ceiling Calculator

// ---------------------------------------------------------------------
// thrudiag_pkg -- four ceilings, all of them arithmetic over parameters,
// and the one axis that separates two of them from the other two.
//
// The package's claim: every ceiling in this chapter is computable
// BEFORE any traffic, and the diagnosis is which of the four is lowest
// rather than what the link measured.
// ---------------------------------------------------------------------
package thrudiag_pkg;

  typedef enum logic [2:0] {
    C_LINE      = 3'd0,   // the label
    C_FRAMING   = 3'd1,   // Chapter 8.3 Section 2
    C_FRAMERATE = 3'd2,   // Chapter 8.3 Section 6
    C_OUTSTAND  = 3'd3,   // Chapter 19.6 Section 4's Little's law
    C_NOT_OFFERED = 3'd4, // Section 13 -- the commonest of all
    C_NONE      = 3'd7
  } ceiling_e;

  // Chapter 8.3 Section 2's fixed overhead: 14 header, 4 check value,
  // 8 preamble and SFD, 12 interframe gap.
  localparam int WIRE_OVERHEAD = 38;

  // Which ceilings move when the frame size does. This is the whole
  // discriminator -- Section 8.
  localparam logic [4:0] SIZE_DEPENDENT   = 5'b0_0110;  // framing, framerate
  localparam logic [4:0] SIZE_INDEPENDENT = 5'b1_1001;  // line, outstanding, offered

  function automatic int framing_eff_ppt(int payload);
    return (payload * 1000) / (payload + WIRE_OVERHEAD);
  endfunction

  // Chapter 19.6 Section 4, rearranged: the rate a bounded number of
  // outstanding requests can sustain across a given latency.
  // Returned in megabits per second to stay in 32 bits.
  function automatic int little_rate_mbps(int outstanding, int burst_octets,
                                          int latency_ns);
    if (latency_ns == 0) return 1000000;
    return (outstanding * burst_octets * 8 * 1000) / latency_ns;
  endfunction

  // Chapter 19.1 Section 4's per-frame cycle budget at 100 Gb/s.
  // A design needing more cycles than this cannot reach line rate.
  function automatic int cycles_available_x1000(int wire_len_octets);
    // (L + 20) * 8 * f / R, with f = 195.3125 MHz and R = 100 Gb/s.
    return ((wire_len_octets + 20) * 8 * 1953) / 1000;
  endfunction

endpackage

Classification: a package of four closed-form ceilings and one five-bit mask.

What it teaches: that SIZE_DEPENDENT against SIZE_INDEPENDENT is the chapter's entire diagnostic apparatus. Framing efficiency and the frame-rate ceiling are both functions of the frame's length; the outstanding limit is a rate in octets per second and does not know what a frame is. So a frame-size sweep separates the two groups without measuring anything about the design at all.

And it teaches that C_NOT_OFFERED is in the enum for a reason. The commonest cause of a link running below its rate is that nothing is sending faster — and it is not a ceiling in the design, it is a property of the workload. Leaving it out of the enum means it is diagnosed as one of the other four, which is Section 13's subject and Section 14's first prohibition.

Deliberately simplified: framing_eff_ppt uses a fixed 38-octet overhead, where Chapter 8.3 §10 showed it is not always 38 — VLAN tags, preemption fragments and jumbo configurations all move it. cycles_available_x1000 hardcodes 195.3125 MHz and 100 Gb/s. little_rate_mbps saturates at 1 Tb/s to avoid a divide by zero. And the latency is a parameter the design does not own, which is Section 20's rejected class in one line.

Production implication: the last simplification is the chapter's structural point and it belongs in a datasheet rather than in a comment. little_rate_mbps(8, 1024, 1000) is 65.5 Gb/s — so a design with eight outstanding requests and kilobyte bursts cannot reach 100 Gb/s across a one-microsecond interconnect, and no amount of RTL changes that. The parameter that has to move is outstanding or burst, and both are negotiated with the integration — Chapter 18.5 §16's observation that the interconnect offers eight regardless of what the design asks for is exactly why this ceiling is so often the binding one.

// ---------------------------------------------------------------------
// ceiling_calculator -- compute all four and report which is lowest.
//
// Every input is a parameter or a measured latency. No traffic is
// required to run this, which is the point: the binding ceiling is
// knowable before anybody complains.
// ---------------------------------------------------------------------
module ceiling_calculator
  import thrudiag_pkg::*;
#(
  parameter int LINE_MBPS      = 100000,
  parameter int OUTSTANDING    = 8,
  parameter int BURST_OCTETS   = 1024,
  parameter int CYCLES_PER_FRAME_X1000 = 2000    // the design's budget
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [15:0] mean_payload,
  input  logic [15:0] mean_wire_len,
  input  logic [31:0] memory_latency_ns,

  output logic [31:0] ceil_line_mbps,
  output logic [31:0] ceil_framing_mbps,
  output logic [31:0] ceil_framerate_mbps,
  output logic [31:0] ceil_outstanding_mbps,
  output ceiling_e    binding,
  output logic [31:0] binding_mbps,
  output logic [15:0] binding_share_ppt
);
  logic [31:0] avail_x1000;

  always_comb begin
    ceil_line_mbps = 32'(LINE_MBPS);

    ceil_framing_mbps =
      (32'(LINE_MBPS) * 32'(framing_eff_ppt(int'(mean_payload)))) / 32'd1000;

    // Chapter 8.3 Section 6: if the design needs more cycles per frame
    // than the wire allows, the shortfall is the ratio.
    avail_x1000 = 32'(cycles_available_x1000(int'(mean_wire_len)));
    ceil_framerate_mbps = (avail_x1000 >= 32'(CYCLES_PER_FRAME_X1000))
      ? 32'(LINE_MBPS)
      : (32'(LINE_MBPS) * avail_x1000) / 32'(CYCLES_PER_FRAME_X1000);

    ceil_outstanding_mbps =
      32'(little_rate_mbps(OUTSTANDING, BURST_OCTETS, int'(memory_latency_ns)));

    binding_mbps = ceil_line_mbps;
    binding      = C_LINE;
    if (ceil_framing_mbps     < binding_mbps) begin
      binding_mbps = ceil_framing_mbps;     binding = C_FRAMING;   end
    if (ceil_framerate_mbps   < binding_mbps) begin
      binding_mbps = ceil_framerate_mbps;   binding = C_FRAMERATE; end
    if (ceil_outstanding_mbps < binding_mbps) begin
      binding_mbps = ceil_outstanding_mbps; binding = C_OUTSTAND;  end

    binding_share_ppt = 16'((binding_mbps * 32'd1000) / 32'(LINE_MBPS));
  end
endmodule

Classification: a four-way minimum, and every input is a parameter except one.

What it teaches: that memory_latency_ns is the only input that is not a design parameter, and it is the one that decides the answer. Change it from 500 to 2 000 and the outstanding ceiling moves from 131.1 Gb/s to 32.8 — with the design unchanged. Chapter 18.5 §16 made the same observation about the reorder buffer; here it is the throughput itself.

And it teaches that binding_share_ppt is the number a report should lead with. "This port's ceiling is 65.5% of its line rate, and the binding constraint is the outstanding limit" is a complete answer to "why is it slow" — computed from four parameters and one measured latency, before anybody runs a benchmark.

Deliberately simplified: the minimum is a sequence of if statements that later ones can override, so ties go to the last checked — the same style defect Chapter 21.2 §7 made deliberately. CYCLES_PER_FRAME_X1000 is a parameter the design must supply and rarely documents. And C_NOT_OFFERED is never selected — the block computes ceilings and has no idea what was offered, which is Section 13's job.

Production implication: the missing C_NOT_OFFERED is why this block alone produces confident wrong answers. A port carrying 30 Gb/s with a 65.5 Gb/s outstanding ceiling will be reported as outstanding-limited, and it may simply be that nothing is sending 65 Gb/s. The ceiling is real and it is not the explanation; the two are distinguished by whether the outstanding count is pinned, which is Section 8's single measurement and costs one register read.

4. The Framing Ceiling Is a Function of Size Alone

Chapter 8.3 §2's curve is the first ceiling and the one most often mistaken for a fault.

A link at full line rate carrying minimum-size frames delivers 54.8 Gb/s of payload, and a monitoring system that reports payload throughput against a 100 Gb/s label shows a 45% shortfall on a perfectly healthy port.

Chapter 8.3 §4's distinction is the vocabulary for it.

And the two differ by 45.2% on minimum-size traffic and 2.5% on maximum-size traffic, so a report that does not say which it is measuring is ambiguous by up to a factor of two.

The curve's shape is what makes it recognisable in a sweep.

Steep below 256 and essentially flat above 1 000 — Chapter 8.3 §2 said exactly that — and a measured throughput that follows this curve as the frame size varies is framing-limited and nothing else.

A throughput that rises steeply below 256 octets and flattens above 1 000 is the framing ceiling. A throughput that does not move with frame size at all is the outstanding ceiling. Those two shapes are the diagnosis.

And one caveat that Chapter 8.3 §10 already made.

The overhead is not always 38, so a curve fitted with 38 will sit slightly above a tagged link's measurements — by a quarter of a point at maximum size and by nearly five at minimum size, which is enough to matter when the question is whether a shortfall is explained.

5. RTL 2 — The Goodput Separator

// ---------------------------------------------------------------------
// goodput_separator -- Chapter 8.3 Section 5's separate accumulator,
// used to say which of the four ceilings a measurement should be
// compared against.
//
// A number is only slow relative to a ceiling, and the ceiling depends
// on what the number counts. Most throughput complaints are a mismatch
// between the two.
// ---------------------------------------------------------------------
module goodput_separator
  import thrudiag_pkg::*;
#(
  parameter int LINE_MBPS = 100000
)(
  input  logic        clk,
  input  logic        rst_n,
  input  logic        tick_us,

  input  logic        frame_done,
  input  logic [15:0] wire_len,
  input  logic [15:0] payload_len,

  output logic [47:0] wire_octets,
  output logic [47:0] payload_octets,
  output logic [31:0] throughput_mbps,
  output logic [31:0] goodput_mbps,
  output logic [15:0] measured_eff_ppt,
  output logic [15:0] expected_eff_ppt,
  output logic        efficiency_explained,
  output logic [31:0] elapsed_us
);
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      wire_octets <= '0; payload_octets <= '0; elapsed_us <= '0;
    end else begin
      if (frame_done) begin
        wire_octets    <= wire_octets + 48'(wire_len) + 48'd20;
        payload_octets <= payload_octets + 48'(payload_len);
      end
      if (tick_us) elapsed_us <= elapsed_us + 32'd1;
    end
  end

  always_comb begin
    throughput_mbps = (elapsed_us == 32'd0) ? 32'd0
                    : 32'((wire_octets * 48'd8) / 48'(elapsed_us));
    goodput_mbps    = (elapsed_us == 32'd0) ? 32'd0
                    : 32'((payload_octets * 48'd8) / 48'(elapsed_us));

    measured_eff_ppt = (wire_octets == 48'd0) ? 16'd0
                     : 16'((payload_octets * 48'd1000) / wire_octets);

    // Chapter 8.3 Section 2's curve, for this link's mean payload.
    expected_eff_ppt = (payload_octets == 48'd0) ? 16'd0
      : 16'(framing_eff_ppt(int'((payload_octets * 48'd1) /
                                 (wire_octets / 48'd84 + 48'd1))));

    // If the measured efficiency matches the curve, the shortfall
    // between goodput and the line rate is entirely framing and
    // nothing is wrong.
    efficiency_explained = (measured_eff_ppt + 16'd20 >= expected_eff_ppt) &&
                           (measured_eff_ppt <= expected_eff_ppt + 16'd20);
  end
endmodule

Classification: two accumulators and a comparison against a closed-form curve.

What it teaches: that efficiency_explained answers the commonest throughput complaint before any other work is done. A link delivering 54.8 Gb/s of payload at 100 Gb/s of wire has a measured efficiency of 54.76% and an expected one of 54.76% — the shortfall is completely explained and the link is at line rate. Chapter 8.3 §5 built the separate accumulator for exactly this and almost nothing deployed has one.

And it teaches that wire_octets must include the twenty octets no frame contains. Chapter 8.3 §3's block counts what the wire carried: the frame, plus eight octets of preamble and start delimiter, plus twelve of interframe gap. A counter that omits them understates the wire's occupancy by 24% on minimum-size traffic and makes a saturated link look idle.

Deliberately simplified: expected_eff_ppt estimates the mean payload from the wire octets by dividing by 84, which is correct only for minimum-size traffic — a real implementation tracks the mean payload directly. The tolerance of 20 parts per thousand is a judgement. elapsed_us counts since reset with no windowing. And 48-bit divisions appear four times combinationally.

Production implication: the twenty-octet term is the one that gets omitted and it changes the conclusion. A counter reporting only frame octets on a saturated minimum-size link reads 76.2 Gb/s of 100 — a 24% apparent shortfall on a link with no spare capacity at all — and the diagnosis that follows is a search for a ceiling that does not exist. Chapter 8.3 §13 called this "a link that is full and looks idle"; it is the first thing to rule out and it is one addition per frame.

6. The Outstanding Ceiling Is 65.5 Gb/s and Nothing Says So

Chapter 19.6 §4 derived the outstanding count a design needs; rearranged, it is the rate a bounded count can sustain.

achievable rate = outstanding × burst / memory latency

With Chapter 18.5 §16's eight outstanding, a kilobyte burst and a microsecond of memory latency:

8 × 1024 octets × 8 bits / 1 µs = 65.5 Gb/s

The 100 Gb/s row is nowhere in that table's first column at eight outstanding, and Chapter 19.6 §4 already made the point that matters: "the interconnect offers 8 outstanding regardless of what the design asks for" — so the count is frequently not the design's choice at all.

Three properties of this ceiling make it the hardest of the four to see.

First, it produces no errors and no drops. A design that cannot issue another request simply waits, and the wire idles. Chapter 19.6 §8's asymmetry applies: on the transmit path the wire idles and nothing is lost. Every counter in Module 21 reads zero.

Second, it is independent of the frame size. The limit is octets per second across the memory interface; a 64-octet frame and a 9 000-octet one consume the same bandwidth per octet. So sweeping the frame size does not move it at all, which is Section 8's discriminator.

Third, it depends on a number the design does not own. Memory latency is the interconnect's, the memory controller's and the system's load — not the MAC's — and a design integrated into a slower system has a lower ceiling with no RTL change.

Which makes it the one ceiling that cannot be fixed by the team that is blamed for it, and the one that a datasheet's throughput figure silently assumes away: a MAC advertised at 100 Gb/s is advertised with an assumed memory latency, and the assumption is almost never printed.

A 100 Gb/s MAC with eight outstanding requests and kilobyte bursts is a 65.5 Gb/s MAC across a one-microsecond interconnect, and no counter in the design will ever say so.

7. RTL 3 — The Little's-Law Bound

// ---------------------------------------------------------------------
// littles_law_bound -- compute the outstanding ceiling from a MEASURED
// latency rather than an assumed one.
//
// The design knows its outstanding limit and its burst size. What it
// does not know is the latency, and that is the term that decides the
// answer -- so measuring it is the block's whole purpose.
// ---------------------------------------------------------------------
module littles_law_bound
  import thrudiag_pkg::*;
#(
  parameter int OUTSTANDING  = 8,
  parameter int BURST_OCTETS = 1024,
  parameter int LINE_MBPS    = 100000
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        req_issued,
  input  logic        resp_returned,
  input  logic [31:0] cycle_count,

  output logic [31:0] latency_ns_mean,
  output logic [31:0] latency_ns_max,
  output logic [31:0] ceiling_mbps,
  output logic [15:0] ceiling_share_ppt,
  output logic        below_line_rate,
  output logic [31:0] c_samples
);
  logic [31:0] outstanding_now, total_latency, issue_stamp;

  // Little's law the other way: measure the latency by measuring the
  // occupancy and the rate, which needs no per-request tagging.
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      outstanding_now <= '0; total_latency <= '0;
      c_samples <= '0; latency_ns_max <= '0; issue_stamp <= '0;
    end else begin
      if (req_issued && !resp_returned)      outstanding_now <= outstanding_now + 32'd1;
      else if (resp_returned && !req_issued) outstanding_now <= outstanding_now - 32'd1;

      if (req_issued)    issue_stamp <= cycle_count;
      if (resp_returned) begin
        total_latency <= total_latency + (cycle_count - issue_stamp);
        c_samples     <= c_samples + 32'd1;
        if ((cycle_count - issue_stamp) > latency_ns_max)
          latency_ns_max <= (cycle_count - issue_stamp);
      end
    end
  end

  always_comb begin
    latency_ns_mean = (c_samples == 32'd0) ? 32'd0
                    : (total_latency / c_samples);

    ceiling_mbps = 32'(little_rate_mbps(OUTSTANDING, BURST_OCTETS,
                                        int'(latency_ns_mean)));

    ceiling_share_ppt = 16'((ceiling_mbps * 32'd1000) / 32'(LINE_MBPS));
    below_line_rate   = (ceiling_mbps < 32'(LINE_MBPS));
  end
endmodule

Classification: a latency measurement whose only purpose is to be divided into two parameters.

What it teaches: that the latency is measurable and almost nobody measures it. The design issues requests and receives responses; a single timestamp and a subtraction gives the round trip, and dividing it into outstanding × burst gives a ceiling that can be compared against the line rate before any throughput test is run. The number that decides whether a 100 Gb/s port can reach 100 Gb/s is available for the cost of one register and a subtraction.

And it teaches that latency_ns_max matters as much as the mean. Little's law uses the mean and the mean gives the sustained ceiling; the maximum is what Chapter 21.7 §11's stall budget cares about, because a single long response is a stall. One measurement, two consumers, two different statistics — and a block reporting only the mean is useless to the other chapter.

Deliberately simplified: issue_stamp holds one timestamp, so with several requests in flight it measures the most recent issue against every response — a real implementation tags per ID, which Chapter 19.6 §7's classifier already does. cycle_count is assumed to be nanoseconds. The occupancy counter mishandles a simultaneous issue and return. And ceiling_mbps uses the mean latency where the achievable rate under a varying latency is bounded by something closer to the mean of the reciprocal.

Production implication: the per-ID tagging is not an extra cost because Chapter 19.6 §7 built the classifier already. An interface that distinguishes read IDs for reordering can timestamp per ID for free — one register per outstanding slot, eight of them — and the resulting latency distribution is the single most useful number a memory interface can report. It bounds the throughput by Little's law, it bounds the stall by Chapter 21.7 §10, and it sizes Chapter 19.6 §19's reorder buffer, which is three chapters' worth of arithmetic from one measurement.

8. One Measurement Separates Memory From Protocol

The four ceilings divide into two groups by what they depend on, and one existing register tells them apart.

A frame-size sweep separates rows two and three from row four, and it is a real experiment: four runs at 64, 256, 1 024 and 1 518 octets.

But a sweep needs a traffic generator and a maintenance window, and Chapter 21.7 §5 already built the instrument that answers it from one register.

If the outstanding count is pinned at its maximum while the throughput is below the line rate, the ceiling is the memory path. If it is not pinned, the ceiling is not the memory path.

The reasoning is Little's law read backwards. A design limited by its outstanding count has every slot occupied at all times — that is what the limit binding means. A design limited by framing or by its frame rate has spare slots, because the memory path is keeping up with a wire that is delivering less than it could.

Four rows, three distinct readings of one register, and the fourth — an outstanding count that is persistently low — is Section 13's not-offered case.

One counter, built for a different purpose in Chapter 19.6 §5 and read as three regimes in Chapter 21.7 §5, answers the question this chapter exists for and costs a register read.

And the frame-size sweep remains worth running for one thing the register cannot do.

Row two is the sweep's remaining value and it is small, because framing and the frame-rate ceiling have the same owner and often the same fix — larger frames. The expensive distinction is memory against protocol, and that one is free.

And there is a second free reading that narrows the protocol side, which the register alone cannot do.

The wire's occupancy separates framing from the frame-rate ceiling without a sweep, because the two differ in whether the wire has gaps.

Row one is the distinction and it is exact. A framing-limited link is saturated: every octet of the 45.2% overhead is a real octet on the wire, so the wire rate is 100% of line and only the payload rate is short. A frame-rate-limited or outstanding-limited link has idle wire, because the frames that would have filled it were never sent.

Two free readings — the wire's octet rate and the outstanding count's regime — separate all four ceilings, and neither of them needs a traffic generator, a maintenance window or a frame-size sweep. That is the chapter's method, and the sweep of Section 8's first table is a confirmation rather than a diagnosis.

9. RTL 4 — The Binding-Ceiling Selector

// ---------------------------------------------------------------------
// binding_ceiling_selector -- combine the computed ceilings with the
// one measured regime, and name the constraint.
//
// The ceilings say what is POSSIBLE. The regime says which possibility
// is actually binding right now, which is the difference between a
// datasheet and a diagnosis.
// ---------------------------------------------------------------------
module binding_ceiling_selector
  import thrudiag_pkg::*;
#(
  parameter int LINE_MBPS = 100000
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [31:0] ceil_framing_mbps,
  input  logic [31:0] ceil_framerate_mbps,
  input  logic [31:0] ceil_outstanding_mbps,
  input  logic [31:0] measured_mbps,

  // Chapter 21.7 Section 5's regime, reused unchanged.
  input  logic        outstanding_pinned,
  input  logic        outstanding_low,
  input  logic        regime_valid,

  output ceiling_e    binding,
  output logic [31:0] headroom_mbps,
  output logic        measurement_explained,
  output logic        unexplained_shortfall,
  output logic [15:0] explained_ppt
);
  logic [31:0] lowest;

  always_comb begin
    lowest = ceil_framing_mbps;
    if (ceil_framerate_mbps   < lowest) lowest = ceil_framerate_mbps;
    if (ceil_outstanding_mbps < lowest) lowest = ceil_outstanding_mbps;

    // The regime decides between the computed possibilities.
    if      (!regime_valid)        binding = C_NONE;
    else if (outstanding_low)      binding = C_NOT_OFFERED;
    else if (outstanding_pinned)   binding = C_OUTSTAND;
    else if (ceil_framing_mbps <= ceil_framerate_mbps) binding = C_FRAMING;
    else                                                binding = C_FRAMERATE;

    headroom_mbps = (lowest > measured_mbps) ? (lowest - measured_mbps) : 32'd0;

    explained_ppt = (lowest == 32'd0) ? 16'd0
                  : 16'((measured_mbps * 32'd1000) / lowest);

    // Within 5% of the binding ceiling is as good as the design gets.
    measurement_explained = (explained_ppt >= 16'd950);

    // Below every ceiling by a margin nothing accounts for. This is
    // the interesting case and the chapter's honest residue.
    unexplained_shortfall = regime_valid && !measurement_explained &&
                            !outstanding_low;
  end
endmodule

Classification: a selector that uses computed ceilings for the candidates and a measured regime for the choice.

What it teaches: that unexplained_shortfall is the output worth building the block for. A link measuring well below every computed ceiling, with the outstanding count neither pinned nor low, is not explained by any of the four — and that is a finding rather than a failure of the method. It means a fifth ceiling exists: a queue discipline, a shaper, a policer, or a bottleneck outside this design entirely.

And it teaches that the regime outranks the arithmetic. The lowest computed ceiling is a candidate; which one is binding is decided by one register. A design whose framing ceiling is 54.8 Gb/s and whose outstanding ceiling is 65.5 is framing-limited at minimum size and outstanding-limited at maximum size — the same design, the same parameters, and only the regime says which regime it is in today.

Deliberately simplified: measurement_explained uses a 5% band with no account of measurement noise or window length. headroom_mbps is computed against the lowest ceiling rather than against the binding one, which differ when the regime contradicts the arithmetic. And C_NOT_OFFERED is selected from a low outstanding count alone, where a low count is also what an idle design looks like.

Production implication: the ambiguity in C_NOT_OFFERED is real and it is resolved one level up. An outstanding count that is persistently low means the memory path is not busy, which is consistent with "nothing is being offered" and with "something upstream of the memory path is the bottleneck" — a full receive FIFO with a low outstanding count would be the second. Chapter 19.5 §15's occupancy telemetry is the tie-breaker: a low outstanding count with an empty FIFO is not offered; with a full FIFO it is a bottleneck between them.

10. The Same Ceiling Is a Different Fraction at Every Rate

Chapter 20.6 §8 showed that a coverage model's size is quadratic in the beat width, so the same model means different things at different rates. The same argument applies here and the consequence is sharper, because the ceiling is an absolute number.

The outstanding ceiling — eight requests, a kilobyte burst, a microsecond of latency — is 65.5 Gb/s, full stop. It does not scale with the line rate; the line rate scales past it.

The same design, the same interconnect, the same memory system — and the constraint goes from entirely invisible to the dominant term between 25 and 100 Gb/s.

An interconnect that was never the bottleneck at 10 Gb/s becomes the bottleneck at 100 without anything about it changing. The rate moved past it.

Which explains a pattern that otherwise looks like a regression.

Row six is the finding. The integration that was adequate at 10 Gb/s is the same integration, and the only thing that moved is the number on the label. Chapter 18.5 §16's observation — the interconnect offers eight outstanding regardless of what the design asks for — is what makes the ceiling a constant across a decade of products.

And the other three ceilings scale differently, which is worth tabulating because the mixture is what makes the diagnosis rate-dependent.

Row two is Chapter 19.1 §4's table: the cycles available per frame fall from 84 at 1 Gb/s to 1.312 at 100 — a factor of 64 — so a per-frame design that was comfortable at 1 Gb/s is impossible at 100. That is a design constraint that tightens with the rate.

Row three is a constraint that does not tighten and does not loosen; the rate simply overtakes it.

And that difference decides who is asked to fix it. A frame-rate ceiling is the MAC team's; an outstanding ceiling is the integration's, and the MAC team is the one that gets the ticket — because the symptom is a MAC that does not reach its advertised rate.

11. RTL 5 — The Rate-Normalised Reporter

// ---------------------------------------------------------------------
// rate_normalised_reporter -- report every ceiling twice: as an
// absolute rate and as a share of THIS link's line rate.
//
// Section 10: the outstanding ceiling is an absolute number and the
// line rate is not. A report that gives only the share is useless
// across products; one that gives only the absolute is useless within
// one.
// ---------------------------------------------------------------------
module rate_normalised_reporter
  import thrudiag_pkg::*;
#(
  parameter int LINE_MBPS = 100000
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [31:0] ceil_framing_mbps,
  input  logic [31:0] ceil_framerate_mbps,
  input  logic [31:0] ceil_outstanding_mbps,
  input  logic [31:0] measured_mbps,

  output logic [15:0] framing_share_ppt,
  output logic [15:0] framerate_share_ppt,
  output logic [15:0] outstanding_share_ppt,
  output logic [15:0] measured_share_ppt,
  output logic        outstanding_would_bind_at_next_rate,
  output logic [31:0] rate_at_which_it_binds_mbps,
  output logic        absolute_and_share_disagree
);
  always_comb begin
    framing_share_ppt     = 16'((ceil_framing_mbps     * 32'd1000) / 32'(LINE_MBPS));
    framerate_share_ppt   = 16'((ceil_framerate_mbps   * 32'd1000) / 32'(LINE_MBPS));
    outstanding_share_ppt = 16'((ceil_outstanding_mbps * 32'd1000) / 32'(LINE_MBPS));
    measured_share_ppt    = 16'((measured_mbps         * 32'd1000) / 32'(LINE_MBPS));

    // Section 10's forward look: the outstanding ceiling is an
    // absolute number, so it binds at every line rate above it.
    rate_at_which_it_binds_mbps = ceil_outstanding_mbps;
    outstanding_would_bind_at_next_rate =
      (ceil_outstanding_mbps < (32'(LINE_MBPS) * 32'd4));

    // A ceiling that is 100% of this link's rate and an absolute
    // number below the next product's is the case Section 10 is about.
    absolute_and_share_disagree =
      (outstanding_share_ppt >= 16'd1000) && outstanding_would_bind_at_next_rate;
  end
endmodule

Classification: a reporter whose only content is that one number has two meanings.

What it teaches: that absolute_and_share_disagree is a forward-looking verdict and the only one in Module 21. A ceiling that is 100% of this link's rate — invisible, non-binding, entirely healthy — and an absolute number below four times it will bind on the next product in the family. The information exists today, on a link where nothing is wrong, and it is one comparison.

And it teaches why both forms must be reported. A share is comparable within a product and meaningless across a family; an absolute rate is comparable across a family and hard to interpret within one. Chapter 20.6 §15 made the same argument about coverage figures at different beat widths: report the parameter beside the result so two products cannot be compared by accident.

Deliberately simplified: the "next rate" is hardcoded as four times the current one, which matches the 10 → 25 → 100 → 400 progression roughly and not exactly. rate_at_which_it_binds_mbps is the ceiling itself, which is correct and reads oddly. And the block assumes the ceiling stays constant across products, which is true of the outstanding limit and false of the memory latency.

Production implication: the assumption that the latency stays constant is the optimistic one and the pessimistic reality makes the warning stronger. Memory latency at a higher line rate is usually worse, not the same — more agents, more contention, a deeper interconnect — so the ceiling that is 65.5 Gb/s today may be 40 on the next platform. A forward-looking report that assumes a constant latency understates the problem, which is the right direction for a warning to be wrong in.

12. Where the Shortfall Actually Goes

A link measured at 65 Gb/s on a 100 Gb/s port has 35 Gb/s that did not happen. This section accounts for it.

Case one — framing, on minimum-size traffic at full line rate.

Nothing is lost and nothing is slow. Chapter 8.3 §2's overhead is 38 octets of every 84 — 45.2% by construction — and the only fault here is comparing payload against a wire-rate label.

Case two — the outstanding limit, on any traffic.

The wire has gaps and nothing was dropped. Chapter 19.6 §8's asymmetry is why: on the transmit path a stalled read channel drains the FIFO and the wire idles — recoverable, nothing lost. The shortfall is unsent frames rather than lost ones, and no counter in a MAC counts the frames it did not send.

Case three — the frame-rate ceiling, on minimum-size traffic.

Cases two and three produce the same symptom — an idle wire and a shortfall — with completely different owners, and Section 8's register read separates them: the outstanding count is pinned in case two and normal in case three.

Case four — nothing was offered.

And this is the commonest case of all, which is why Section 13 exists: a receiver cannot distinguish "I could not keep up" from "nobody asked me to" without evidence about the offered load, and the offered load is upstream.

The four cases, side by side:

Four cases, no losses, four different answers, and the whole of Module 21's counting apparatus reads zero in every one of them. The distinguishing evidence is two numbers that are not counters: the wire's occupancy and the outstanding count's regime.

13. RTL 6 — The Offered-Load Discriminator

// ---------------------------------------------------------------------
// offered_load_discriminator -- the hardest question in the chapter:
// was the link slow, or was nothing asked of it?
//
// A receiver has no direct evidence about what was offered. What it
// has is the interframe gap distribution: a saturated wire has
// minimum gaps and an under-offered one does not.
// ---------------------------------------------------------------------
module offered_load_discriminator
  import thrudiag_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        frame_start,
  input  logic [15:0] gap_octets,        // measured since the last frame
  input  logic        fifo_nonempty,
  input  logic        outstanding_low,

  output logic [31:0] c_frames,
  output logic [31:0] c_min_gap_frames,
  output logic [15:0] saturation_ppt,
  output logic        wire_saturated,
  output logic        not_offered,
  output logic        bottleneck_upstream_of_memory
);
  // Chapter 5.9's floor is 12 octets nominal, 9 to 12 under
  // Chapter 19.3 Section 6's deficit. A frame arriving at the floor
  // means the sender had another one ready.
  localparam int GAP_FLOOR = 12;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      c_frames <= '0; c_min_gap_frames <= '0;
    end else if (frame_start) begin
      c_frames <= c_frames + 32'd1;
      if (gap_octets <= 16'(GAP_FLOOR)) c_min_gap_frames <= c_min_gap_frames + 32'd1;
    end
  end

  always_comb begin
    saturation_ppt = (c_frames == 32'd0) ? 16'd0
                   : 16'((c_min_gap_frames * 32'd1000) / c_frames);

    // Most frames arriving at the minimum gap means the sender was
    // never idle -- the wire is offered more than it is carrying.
    wire_saturated = (c_frames > 32'd10000) && (saturation_ppt > 16'd900);

    // A low outstanding count and an unsaturated wire is the simplest
    // explanation there is: nothing was asked for.
    not_offered = outstanding_low && !wire_saturated;

    // A low outstanding count with a NON-EMPTY receive FIFO is the
    // interesting one: something between the wire and memory is the
    // bottleneck, and it is neither of them.
    bottleneck_upstream_of_memory = outstanding_low && fifo_nonempty;
  end
endmodule

Classification: a gap-distribution test, and the only instrument in the chapter that observes the wire rather than a register.

What it teaches: that the interframe gap distribution is the receiver's only evidence about the offered load. Chapter 5.9's floor is twelve octets; a frame arriving at the floor means the sender had another one ready and sent it as soon as it legally could. Ninety per cent of frames at the floor is a saturated offered load, and the shortfall is ours. A broad gap distribution means the sender was idle and the shortfall is theirs.

And it teaches that bottleneck_upstream_of_memory is the case the other instruments cannot reach. A low outstanding count with a non-empty receive FIFO means the memory path is idle while frames are waiting — so the bottleneck is between the two: Chapter 19.6 §11's arbiter, a descriptor fetch that has not returned, or a request shaper that is not shaping. Neither the wire nor the memory is the constraint and both look healthy.

Deliberately simplified: gap_octets must be measured by something upstream, and a MAC that does not expose it makes this whole block unavailable — the gap is discarded at the parser in most designs. The 90% threshold is a judgement. GAP_FLOOR uses twelve where Chapter 19.3 §6's deficit produces nine to twelve, so a lane-aligning transmitter's frames at a nine-octet gap are not counted as saturating.

Production implication: the gap measurement is the missing instrument and it is nearly free. A counter of frames arriving at or below a thirteen-octet gap — one comparator and one counter, about 99 flops — turns "was the link slow or was nothing offered" from an unanswerable question into a ratio. It is the only evidence a receiver can have about a sender's intent, and Chapter 5.9's floor is what makes it meaningful: a gap at the floor is a sender that could not legally have sent sooner.

14. What a Throughput Diagnosis Must Never Do

Six prohibitions. Two are about which number is being compared, two about which ceiling, and two about what an absence means.

Row two is the prohibition that inverts the diagnosis. Chapter 8.3 §3's counter records what the wire carried: the frame plus eight octets of preamble and start delimiter plus twelve of interframe gap. A counter that records only frame octets reads 76.2 Gb/s on a saturated 100 Gb/s minimum-size link — a 24% apparent shortfall with no spare capacity at all — and the investigation that follows searches for a ceiling that does not exist. Chapter 8.3 §13 named it: a link that is full and looks idle.

Row four is Section 10's argument as a reporting rule. A ceiling quoted as "100% of line rate" is invisible on the product it was measured on and may be the binding constraint on the next one — because the outstanding limit is an absolute 65.5 Gb/s and the line rate is what moves. Reporting both forms is one extra division.

And the two that look like caution and are not:

Both search for a fault in a system that may be behaving perfectly, which is what all six share — and in this chapter, more than in any other in Module 21, the most likely answer is that nothing is wrong at all.

15. RTL 7 — Throughput Telemetry

// ---------------------------------------------------------------------
// thrudiag_telemetry -- three groups: what was measured, which ceiling
// binds, and what would change it.
//
// The third group is unique to this chapter. Every other diagnosis in
// Module 21 reports a fault; this one reports a LIMIT, and a limit has
// a lever attached to it.
// ---------------------------------------------------------------------
module thrudiag_telemetry
  import thrudiag_pkg::*;
#(
  parameter int LINE_MBPS    = 100000,
  parameter int OUTSTANDING  = 8,
  parameter int BURST_OCTETS = 1024
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [31:0] throughput_mbps,
  input  logic [31:0] goodput_mbps,
  input  ceiling_e    binding,
  input  logic [31:0] binding_mbps,
  input  logic [31:0] memory_latency_ns,
  input  logic        wire_saturated,
  input  logic        unexplained_shortfall,

  // Measured.
  output logic [31:0] wire_rate_mbps,
  output logic [31:0] payload_rate_mbps,
  output logic [15:0] wire_share_ppt,

  // Binding.
  output ceiling_e    binding_out,
  output logic [15:0] binding_share_ppt,
  output logic        at_the_ceiling,

  // Levers.
  output logic [31:0] gain_from_double_outstanding_mbps,
  output logic [31:0] gain_from_double_burst_mbps,
  output logic [31:0] gain_from_half_latency_mbps,
  output logic        no_lever_in_this_design
);
  assign wire_rate_mbps    = throughput_mbps;
  assign payload_rate_mbps = goodput_mbps;
  assign binding_out       = binding;

  always_comb begin
    wire_share_ppt    = 16'((throughput_mbps * 32'd1000) / 32'(LINE_MBPS));
    binding_share_ppt = 16'((binding_mbps    * 32'd1000) / 32'(LINE_MBPS));
    at_the_ceiling    = (throughput_mbps * 32'd20) >= (binding_mbps * 32'd19);

    // Little's law is a product of three terms, so each lever is a
    // factor of two -- capped at the line rate, which the others are
    // not.
    gain_from_double_outstanding_mbps = (binding == C_OUTSTAND)
      ? ((binding_mbps * 32'd2 > 32'(LINE_MBPS)) ?
         (32'(LINE_MBPS) - binding_mbps) : binding_mbps) : 32'd0;
    gain_from_double_burst_mbps = gain_from_double_outstanding_mbps;
    gain_from_half_latency_mbps = gain_from_double_outstanding_mbps;

    // Framing and the frame-rate ceiling have no lever the MAC owns:
    // one is the standard's overhead and one is the design's pipeline.
    no_lever_in_this_design = (binding == C_FRAMING) ||
                              (binding == C_NOT_OFFERED);
  end
endmodule

Classification: a reporter whose third group turns a limit into a decision.

What it teaches: that the three levers on the outstanding ceiling are worth exactly the same amount and cost completely different things. Little's law is a product: doubling the outstanding count, doubling the burst size and halving the latency all double the ceiling. But the outstanding count is usually the interconnect's — Chapter 18.5 §16 — the burst size is the design's within the interconnect's rules, and the latency is the system's. Same gain, three different negotiations.

And it teaches that no_lever_in_this_design is an honest output rather than a defeat. A framing-limited link has no lever the MAC owns: the overhead is 38 octets by the standard and the only change is larger frames, which is the application's decision. A not-offered link has no lever at all. Reporting that is more useful than reporting a ceiling somebody will try to raise.

Deliberately simplified: the three lever gains are computed identically and the block does not say which is cheapest, which is the question an integrator actually asks. at_the_ceiling uses a 5% band as integer arithmetic. And gain_from_double_burst_mbps ignores that Chapter 19.6 §4 showed doubling the burst halves the required outstanding count — so the two levers interact rather than compose.

Production implication: that interaction is the useful correction and Chapter 19.6 §4 already derived it. Doubling the burst halves the outstanding count needed and leaves the reorder buffer unchanged — (O − 1) × burst is R × L − burst, near enough — so doubling the burst is the lever that costs nothing in buffering. Doubling the outstanding count doubles the reorder buffer, which is Chapter 19.6 §19's 12 KiB becoming 24. Same throughput gain, and one of the two is free.

16. RTL 8 — The Throughput Conformance Monitor

// ---------------------------------------------------------------------
// thrudiag_conformance_monitor -- six verdicts, and the first two are
// about whether the right numbers are being compared at all.
// ---------------------------------------------------------------------
module thrudiag_conformance_monitor
  import thrudiag_pkg::*;
#(
  parameter int LINE_MBPS = 100000
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        counts_include_gap,
  input  logic        comparing_goodput_to_line,
  input  logic [31:0] measured_mbps,
  input  logic [31:0] binding_mbps,
  input  ceiling_e    binding,
  input  logic        regime_valid,
  input  logic        wire_saturated,
  input  logic        outstanding_low,
  input  logic        gap_measurable,

  output logic        gap_not_counted,
  output logic        wrong_ceiling_compared,
  output logic        offered_load_unknown,
  output logic        regime_unavailable,
  output logic        shortfall_unexplained,
  output logic        ceiling_is_absolute,
  output logic        diagnosis_sound
);
  // Section 14's row two: a full link reads 76.2% without the twenty
  // octets of preamble and gap.
  assign gap_not_counted = !counts_include_gap;

  // Section 14's row one.
  assign wrong_ceiling_compared = comparing_goodput_to_line;

  // Section 13: without the gap distribution a receiver cannot tell a
  // slow link from an unoffered one.
  assign offered_load_unknown = !gap_measurable && outstanding_low;

  assign regime_unavailable = !regime_valid;

  assign shortfall_unexplained = regime_valid &&
                                 ((measured_mbps * 32'd20) < (binding_mbps * 32'd19)) &&
                                 !outstanding_low;

  // Section 10: the outstanding ceiling does not scale with the line
  // rate, so a share is not portable across products.
  assign ceiling_is_absolute = (binding == C_OUTSTAND);

  assign diagnosis_sound = !gap_not_counted && !wrong_ceiling_compared &&
                           !regime_unavailable;

  always_ff @(posedge clk) begin
    if (rst_n && gap_not_counted)
      $display("[thrudiag] octet counts exclude the 20-octet preamble and gap: a saturated minimum-size link will read 76.2%%");
  end
endmodule

Classification: an auditor whose first two verdicts are about the measurement rather than the link.

What it teaches: that gap_not_counted and wrong_ceiling_compared are both soundness terms and both are mistakes in the observer. Neither is a property of the link, neither is a property of the traffic, and both produce a shortfall that does not exist — 24% for the first and up to 45% for the second. They are the only two failures in Module 21 that are entirely the measurement's.

And it teaches that offered_load_unknown is excluded from soundness deliberately. A platform that does not expose the interframe gap cannot answer Section 13's question — and that is a property of the instrumentation, not of the investigation. The same split the previous four chapters made for their own blind spots.

Deliberately simplified: counts_include_gap and comparing_goodput_to_line are inputs describing how somebody else's monitoring system is configured, which no hardware block can know. shortfall_unexplained uses the same 5% band as the telemetry. And ceiling_is_absolute fires on every outstanding-limited link, which is a report rather than a verdict.

Production implication: the two measurement verdicts belong in the tooling rather than in the RTL, and that is where they will actually prevent the mistake. A monitoring system that knows whether its octet counters include the interframe gap can label its own graphs, and one that knows whether it is charting throughput or goodput can pick the right ceiling to draw beside it. Both are one field in a schema, and together they account for the majority of throughput tickets that turn out to be nothing at all.

17. The Budget, Closed

Four ceilings, one measured regime, and a worked example that accounts for every gigabit.

Take a 100 Gb/s port measuring 63 Gb/s of wire throughput on 1 518-octet frames, with no errors and nothing dropped.

The link is at 96.2% of what it can do and 63% of what its label says, and both numbers are correct. The binding ceiling is 65.5 Gb/s and the design is essentially at it.

The accounting, gigabit by gigabit:

Row two is 55% of the total shortfall and it is nobody's bug. Chapter 18.5 §16's interconnect offers eight outstanding regardless of what the design asks for; Chapter 19.6 §4's Little's law turns that into 65.5 Gb/s at a microsecond of latency, and no RTL change moves it.

And the three levers, priced against Chapter 19.6 §4's reorder-buffer arithmetic:

Row one is free and row two costs 12 KiB, which is Chapter 19.6 §4's central finding restated: the reorder buffer is R × L − burst, so it depends on the bandwidth-latency product and not on the burst length. Doubling the burst halves the outstanding count needed and leaves the buffer the same size — the only one of the three levers with no cost at all.

Which makes the complete diagnosis three lines.

The binding ceiling is the outstanding limit at 65.5 Gb/s. The design is at 96.2% of it. Doubling the burst size to 2 KiB raises the ceiling to 131.1 Gb/s and leaves the reorder buffer unchanged.

And the same arithmetic on the same design at three other line rates:

The binding constraint changes twice across the family — from the line rate, to framing, to the memory path — and the design is identical in all four rows. Chapter 20.6 §8's lesson exactly: the same absolute number is a different fraction at each rate, and which constraint dominates is a property of the rate rather than of the design.

18. What the Diagnosis Assumes

Nine assumptions. Three are about the arithmetic, three about the measurement and three about the system — and the one most often false is the fifth.

Row three is the assumption that makes the ceiling self-referential and it is worth the paragraph. Memory latency is load-dependent, and the load includes this design's own traffic. So the ceiling computed at low load is optimistic: pushing towards it raises the latency, which lowers the ceiling.

The true ceiling is a fixed point — the rate at which the latency it induces produces exactly that rate — and measuring the latency at idle overstates it, sometimes by a factor of two. The practical correction is to measure the latency while pushing, which is what a throughput test does anyway.

Row nine is the assumption that halves the work and occasionally doubles it. The transmit path reads from memory and the receive path writes to it; they have different latencies, different outstanding limits and different burst sizes — and Chapter 19.6 §12 noted the write path is the easier of the two. A single ceiling figure for a full-duplex port is two figures averaged, and the binding one may be only one direction's.

And three things deliberately not assumed:

Row one is the chapter's whole stance and it is unusual for Module 21. Every other chapter begins from something being wrong; this one begins from a number and has to establish whether anything is wrong at all — and in three of Section 12's four cases the answer is no.

And a tenth assumption, which is about the window rather than the link and which every chapter in this module has now had to make.

Row four is why the windowing matters more here than anywhere else in Module 21. Every other chapter's instruments are counters of rare events, where a long window is an advantage; this chapter's are occupancies and rates, where a long window is an average over states the link is no longer in. A ceiling computed from a lifetime-average latency describes no moment that ever happened.

Row three is the one that is already right, because Chapter 21.7 §5's regime test is an occupancy over a window by construction — and it is the measurement this chapter leans on hardest.

19. The Cost, Accounted

A throughput diagnosis is arithmetic over parameters plus two measurements, so almost none of it is state.

Six of the nine blocks are pure combinational logic, which is what happens when the diagnosis is four closed-form ceilings and a minimum. The 352 flops are entirely the three things that must be measured: the octets, the latency and the gap.

And the additions to a design are three, all small.

Row three is free and it is the one most often missing. Chapter 8.3 §3's counter records what the wire carried, preamble and gap included, and a counter that omits them makes a saturated minimum-size link read 76.2%. One addend per frame.

Row one is the highest-value measurement in Module 21 and Chapter 21.7 §7 argued the same thing from the other side. A latency distribution per outstanding ID bounds the throughput by Little's law, bounds the stall by Chapter 21.7 §10, and sizes Chapter 19.6 §19's reorder buffer — three chapters' worth of arithmetic from one instrument, and Chapter 19.6 §7's ID classifier already provides the tagging.

Module 21's complete instrumentation bill, all eight chapters:

Less Chapter 21.6 §17's redundant pair, about 2 204 flops — 15.6% of Chapter 19.7 §19's 14 166-flop datapath — for every diagnosis eight chapters have built. Against Chapter 19.4's 5 397 correction-barrel XOR terms, which were built without argument because they are a correctness requirement, and which is the comparison Module 21 has now made eight times.

20. Properties Worth Asserting, and One Worth Refusing

Thirty-three properties and eight covers, in four groups: the ceilings, the measurement, the regime, and the levers.

Group one — the four ceilings, which are all closed forms.

// Every ceiling is arithmetic over parameters, so every property here
// is checkable with no traffic at all -- which is the opposite of
// Chapter 21.4's class 97 and is deliberate.
p_framing_monotone: assert property (@(posedge clk) disable iff (!rst_n)
                      (mean_payload > 16'd0) |->
                        (framing_eff_ppt(int'(mean_payload)) <= 1000));

p_framing_rises:    assert property (@(posedge clk) disable iff (!rst_n)
                      (mean_payload > 16'd46) |->
                        (framing_eff_ppt(int'(mean_payload)) >
                         framing_eff_ppt(46)));

p_line_is_highest:  assert property (@(posedge clk) disable iff (!rst_n)
                      (ceil_framing_mbps <= ceil_line_mbps) &&
                      (ceil_framerate_mbps <= ceil_line_mbps));

p_binding_is_min:   assert property (@(posedge clk) disable iff (!rst_n)
                      (binding_mbps <= ceil_framing_mbps) &&
                      (binding_mbps <= ceil_framerate_mbps) &&
                      (binding_mbps <= ceil_outstanding_mbps));

p_little_product:   assert property (@(posedge clk) disable iff (!rst_n)
                      (memory_latency_ns > 32'd0) |->
                        (ceil_outstanding_mbps ==
                         32'(little_rate_mbps(OUTSTANDING, BURST_OCTETS,
                                              int'(memory_latency_ns)))));

p_latency_inverse:  assert property (@(posedge clk) disable iff (!rst_n)
                      ($past(memory_latency_ns) < memory_latency_ns) |->
                        (ceil_outstanding_mbps <= $past(ceil_outstanding_mbps)));

p_share_bounded:    assert property (@(posedge clk) disable iff (!rst_n)
                      binding_share_ppt <= 16'd1000);

p_framerate_cycles: assert property (@(posedge clk) disable iff (!rst_n)
                      (cycles_available_x1000(int'(mean_wire_len)) >=
                       CYCLES_PER_FRAME_X1000) |->
                        (ceil_framerate_mbps == 32'(LINE_MBPS)));

Group two — the measurement, and the two ways it goes wrong.

p_wire_includes_gap:assert property (@(posedge clk) disable iff (!rst_n)
                      frame_done |=> (wire_octets ==
                        $past(wire_octets) + 48'($past(wire_len)) + 48'd20));

p_goodput_le_thru:  assert property (@(posedge clk) disable iff (!rst_n)
                      goodput_mbps <= throughput_mbps);

p_eff_bounded:      assert property (@(posedge clk) disable iff (!rst_n)
                      measured_eff_ppt <= 16'd1000);

p_eff_matches_curve:assert property (@(posedge clk) disable iff (!rst_n)
                      efficiency_explained |->
                        (measured_eff_ppt + 16'd20 >= expected_eff_ppt));

p_thru_le_line:     assert property (@(posedge clk) disable iff (!rst_n)
                      throughput_mbps <= 32'(LINE_MBPS));

p_gap_not_counted:  assert property (@(posedge clk) disable iff (!rst_n)
                      !counts_include_gap |-> gap_not_counted);

p_wrong_ceiling:    assert property (@(posedge clk) disable iff (!rst_n)
                      comparing_goodput_to_line |-> wrong_ceiling_compared);

p_sound_needs_both: assert property (@(posedge clk) disable iff (!rst_n)
                      diagnosis_sound |->
                        (!gap_not_counted && !wrong_ceiling_compared));

Group three — the regime, reused from Chapter 21.7 §5.

p_pinned_is_memory: assert property (@(posedge clk) disable iff (!rst_n)
                      (outstanding_pinned && regime_valid &&
                       (measured_mbps < 32'(LINE_MBPS))) |->
                        (binding == C_OUTSTAND));

p_low_is_offered:   assert property (@(posedge clk) disable iff (!rst_n)
                      (outstanding_low && regime_valid) |->
                        (binding == C_NOT_OFFERED));

p_normal_is_protocol:assert property (@(posedge clk) disable iff (!rst_n)
                      (regime_valid && !outstanding_pinned && !outstanding_low)
                        |-> ((binding == C_FRAMING) || (binding == C_FRAMERATE)));

p_regime_needed:    assert property (@(posedge clk) disable iff (!rst_n)
                      (binding != C_NONE) |-> regime_valid);

p_saturated_gap:    assert property (@(posedge clk) disable iff (!rst_n)
                      wire_saturated |-> (saturation_ppt > 16'd900));

p_not_offered_pair: assert property (@(posedge clk) disable iff (!rst_n)
                      not_offered |-> (outstanding_low && !wire_saturated));

p_upstream_case:    assert property (@(posedge clk) disable iff (!rst_n)
                      bottleneck_upstream_of_memory |->
                        (outstanding_low && fifo_nonempty));

p_unexplained:      assert property (@(posedge clk) disable iff (!rst_n)
                      unexplained_shortfall |-> (!measurement_explained &&
                                                 !outstanding_low));

Group four — the levers and the rate-normalisation.

p_levers_equal:     assert property (@(posedge clk) disable iff (!rst_n)
                      (gain_from_double_outstanding_mbps ==
                       gain_from_double_burst_mbps) &&
                      (gain_from_double_burst_mbps ==
                       gain_from_half_latency_mbps));

p_lever_only_if_out:assert property (@(posedge clk) disable iff (!rst_n)
                      (gain_from_double_burst_mbps > 32'd0) |->
                        (binding == C_OUTSTAND));

p_no_lever_framing: assert property (@(posedge clk) disable iff (!rst_n)
                      (binding == C_FRAMING) |-> no_lever_in_this_design);

p_absolute_flag:    assert property (@(posedge clk) disable iff (!rst_n)
                      (binding == C_OUTSTAND) |-> ceiling_is_absolute);

p_shares_computed:  assert property (@(posedge clk) disable iff (!rst_n)
                      (outstanding_share_ppt ==
                       16'((ceil_outstanding_mbps * 32'd1000) / 32'(LINE_MBPS))));

p_forward_warning:  assert property (@(posedge clk) disable iff (!rst_n)
                      absolute_and_share_disagree |->
                        (outstanding_share_ppt >= 16'd1000));

p_at_ceiling_band:  assert property (@(posedge clk) disable iff (!rst_n)
                      at_the_ceiling |->
                        ((throughput_mbps * 32'd20) >= (binding_mbps * 32'd19)));

p_headroom_zero:    assert property (@(posedge clk) disable iff (!rst_n)
                      (measured_mbps >= binding_mbps) |-> (headroom_mbps == 32'd0));

And eight covers, because six of these states need a saturated offered load a default regression does not generate.

c_framing_binds:   cover property (@(posedge clk) binding == C_FRAMING);
c_framerate_binds: cover property (@(posedge clk) binding == C_FRAMERATE);
c_outstand_binds:  cover property (@(posedge clk) binding == C_OUTSTAND);
c_not_offered:     cover property (@(posedge clk) binding == C_NOT_OFFERED);
c_unexplained:     cover property (@(posedge clk) unexplained_shortfall);
c_saturated:       cover property (@(posedge clk) wire_saturated);
c_upstream:        cover property (@(posedge clk) bottleneck_upstream_of_memory);
c_forward_warn:    cover property (@(posedge clk) absolute_and_share_disagree);

21. Verification Scenarios

Fifty-eight scenarios, plus a five-run directed test whose variable is the memory latency — a number the design does not own and no frame can change.

The ceilings — 12 scenarios.

The measurement — 10 scenarios.

The regime, reused from Chapter 21.7 — 9 scenarios.

The offered load — 9 scenarios.

Rate normalisation — 9 scenarios.

The levers and the residue — 9 scenarios.

And the directed test, because random stimulus will not produce it.

The case: one design, one traffic pattern, five memory latencies — and the throughput changes by a factor of eleven with nothing in the RTL touched.

No frame-level generator reaches this. The variable is the interconnect's response time, which is not a field of any frame and not a parameter of the design. Chapter 20.5 §10's two categories — wire-producible and design-internal — do not contain it, and Chapter 21.7 §21 needed the same third category for the same reason.

The oracle is four-part and rows two and four are the controls.

Row two is the control that makes the test conclusive. Runs A to D are bit-for-bit the same design and the throughput spans 97.5 down to 11.9 Gb/s — a factor of 8.2 — with nothing about the MAC changed and no counter moving anywhere.

Row three is the single measurement the chapter is built on. The regime is normal in run A, where the framing ceiling binds at 97.5, and pinned in runs B, C and D, where the memory path does. One register read separates the two groups and it costs nothing.

Run E is the fix, and it is one parameter. Doubling the burst from 1 KiB to 2 KiB doubles the ceiling from 65.5 to 131.1 and leaves Chapter 19.6 §4's reorder buffer at 12 KiB — because (O − 1) × burst is R × L − burst, near enough, so it depends on the bandwidth-latency product rather than on the burst length. The same gain from doubling the outstanding count would cost 12 KiB more buffer.

And run D is the one worth keeping in a regression. At 5.5 µs of memory latency a 100 Gb/s MAC achieves 11.9 Gb/s — 11.9% of its label, with a perfect link, no drops and every counter clean. A design verified only at run A's latency has never seen its own binding constraint.

22. Debugging a Slow, Correct Link

Six complaints.

Complaint 1 — "the link is only doing 76% and nothing is wrong."

A saturated minimum-size link with frame-only octet counters reads 76.19%, and the number is exact rather than approximate: 64 octets counted of 84 on the wire. Chapter 8.3 §13 called it a link that is full and looks idle; the fix is one addend per frame.

Complaint 2 — "we only get 55% of the line rate."

Fifty-four point eight per cent is Chapter 8.3 §2's curve at 46 octets of payload, and the link is at line rate. The complaint is a comparison of goodput against a wire-rate label — Section 14's first prohibition — and the right ceiling is 54.8, which the link is meeting.

Complaint 3 — "we get 65 Gb/s whatever we do."

A throughput that is identical at 64 and 9 000 octets is the signature, because framing and the frame-rate ceiling both move with size and the outstanding limit does not. Doubling the burst to 2 KiB is the free lever — Chapter 19.6 §4 — and it doubles the ceiling with no change to the reorder buffer.

Complaint 4 — "it was fine at 10 Gb/s and it is slow at 100."

The ceiling is an absolute rate and the line rate is what moved past it. Chapter 18.5 §16's interconnect offers eight outstanding regardless of what the design asks for, so the same integration that was invisible at 10 Gb/s is the dominant term at 100 — and the MAC team gets the ticket because the symptom is a MAC below its rate.

Complaint 5 — "the throughput is below every ceiling we computed."

Rows one and two together are the commonest answer in the chapter and it is not a fault. A low outstanding count with an empty receive FIFO means the memory path was never asked for anything, and the shortfall is upstream. A low count with a full FIFO is the interesting case: Chapter 19.6 §11's arbiter or a descriptor fetch, and neither the wire nor the memory is the constraint.

Complaint 6 — "we doubled the outstanding count and ran out of buffer."

Row four is the lever that should have been pulled first. Doubling the burst and doubling the outstanding count give exactly the same throughput gain, and one leaves the reorder buffer at 12 KiB while the other takes it to 24 — because the buffer is R × L − burst and depends on the bandwidth-latency product rather than on the burst length.

Complaint 7 — "the benchmark says 97 Gb/s and production says 40."