Skip to content
VLSI Mentor

Ethernet · Module 26

Performance and Debug Review

Seven thresholds the design cannot set, a five-hop latency that is 4.564 µs or 41.941 µs depending on the percentile, and a complete observability apparatus for 0.0820% of the die.

Chapter 26.1 §12 ended with a list of seven thresholds nobody in the room could set, and called that list the review's most valuable output. This chapter is what happens to the list.

Because a deferred threshold does not stay deferred. It becomes a default, and this track has derived four defaults and found every one of them wrong by a large factor in a real deployment.

The defaultWrong byWhereDerived in
300 s MAC ageing8× too long1 000 container teardowns an hourChapter 25.2 §8
75% pause watermark79× too conservative at 1 Gb/s over 100 m; 34 cells SHORT at 100 Gb/s over 2 kmboth ends of one product's rangeChapter 14.2 §9
90% occupancy alarmunreachable — the ceiling is 80.5%any four-way tableChapter 25.5 §8
1% storm controlcorrect on broadcast; discards user data on unknown unicastany 64-port switchChapter 25.5 §12

Row two is the one to sit with. The same 75% is safe by a factor of seventy-nine on one link in a product's supported range and fails by 34 cells on another — and the failure mode is that every pause is issued too late, the queue overflows during the dead time, and the drops look exactly like the drops PAUSE was deployed to prevent.

A threshold the design cannot derive is not a gap in the specification. It is a parameter, and the review's product is the list of them with an owner against each.

And there is a second, quieter deferral in Chapter 26.1, which is the whole of Sections 6 to 9 here. That chapter's latency reconciliation produced 912.88 ns for one hop and 4.564 µs for five — and immediately flagged that every term in it was uncongested. It could not say what the latency is under load, because that is a distribution and an architecture review has no traffic.

This chapter has traffic, so it can put the number back.

Five hops, 100 Gb/s, offered load 0.95LatencyAgainst the structural floor
the structural floor — Chapter 26.1 §84.564 µs1.00×
median7.184 µs1.57×
99th percentile23.148 µs5.07×
99.99th percentile41.941 µs9.19×

Four numbers where a review usually has one, and the difference between the first and the last is the difference between a design that meets a 5 µs budget and one that misses it by eight times.

What this chapter does not own. Chapter 26.1 covered architecture and integration and Chapter 26.2 covered RTL and verification closure; Chapter 26.4 covers how all of it is probed in an interview. This is the pass held when the design performs and somebody has to say what "performs" means, and then build the instrument that will say whether it still does in the field.


1. Scope — A Number, a Percentile and an Instrument

Three rules, and the third is the one this chapter adds to the series.

Rule one, from Chapter 26.1 §1: every item is a number.

Rule two, from Chapter 26.2 §1: every number carries its denominator.

Rule three, and it is specific to performance: every latency figure carries a percentile and the window it was measured over.

The quoted figureWhat it assertsComplete?
"latency is 912.88 ns"the structural floor, uncongestedno — it is the condition the design is rarely in
"latency is 912.88 ns at p50"a median, over an unstated windowno — a median over 100 samples is not a median
"4.564 µs floor; 23.148 µs at p99 over 10⁵ samples at ρ = 0.95"a floor, a percentile, a window and a loadyes

Row three has four numbers and every one of them can be wrong in a stateable way, which is the test this series has applied since Chapter 26.1.

And row three is why the percentile is not a unit. A metre is a unit; a percentile is an index into a distribution, and an index is meaningless without the population it indexes. So the honest unit for a latency claim in this chapter is a triple — value, percentile, window length — and Section 12 shows what happens when the third element is left out, which is that the second one becomes a fiction.

The chapter's four results.

ResultValueSection
the seven deferred thresholds, with owners and measurements7, of which 1 needs silicon4
five-hop latency at p99.99, ρ = 0.9541.941 µs — 9.19× the floor6
why Chapter 26.1 §8's 487× is not a percentileP(W > 2 224 µs) at ρ = 0.99 is 2.04 × 10⁻²³⁷8
the observability bill1 443 840 BCE — 5.10 datapaths, 0.2569% of the switch10

The last row is the one that settles most arguments before they start. Everything this chapter asks a design to measure — timestamps, histograms, watermarks, per-set refusal counters, the counter block, the timers — costs a quarter of one per cent of Chapter 23.3's 5.62 × 10⁸ BCE switch. The objection to observability has never been area; it is that nobody derived the number, so the discussion happened in adjectives.


2. The Twenty Items

Every row is a number, the chapter that derives it, and — for the seven that this chapter exists to close — who owns the threshold.

#The itemThis design's numberDerived inOwner
1structural latency, one hop912.88 nsChapter 26.1 §8the design
2structural latency, five hops4.564 µsChapter 26.1 §8the design
3design-owned share of a hop8.76% — 80 ns of 912.88Chapter 26.1 §8the design
4the offered load the budget is stated atρ = 0.95Section 6the deployment
5five-hop latency at p9923.148 µs — 5.07× the floorSection 6derived from 4
6five-hop latency at p99.9941.941 µs — 9.19×Section 6derived from 4
7samples the p99.99 claim needs100 001 — 4.081 ms of trafficSection 12structural
8on-die latency window depth1 024 — resolving p99.902, not p99.99Section 12the design
9maximum queueing one port can hold2 224 µs — 27.8 MB, 108 616 cellsChapter 23.3 §9the design
10the load at which that becomes reachableρ ≥ 1 — an incast, not a tailSection 8the deployment
11time for a 64-way incast to fill it35.302 µsSection 8derived
12buffer size64 MBChapter 23.3 §9THE DEPLOYMENT
13MAC table size128k, storing 105 465 of 131 072 offeredChapter 25.5 §6THE DEPLOYMENT
14ageing interval300 s default; 833 stale entries at 10 000 teardowns/hChapter 25.2 §8THE ORCHESTRATOR
15replay windowthe fabric's worst-case path skewChapter 25.3 §15THE TOPOLOGY
16storm-control setting1% caps one port at 0.98% of 9.524 GppsChapter 25.5 §12THE DEPLOYMENT
17flap thresholdthe legitimate move rateChapter 25.2 §16THE SITE
18pause watermark99.7% to 74.2% across the product's rangeChapter 14.2 §9THE CABLE PLANT
19occupancy alarmmust be ≤ 80% — the ceiling is 80.5%Chapter 25.5 §8the design — structural
20observability area1 443 840 BCE — 5.10 datapaths, 0.2569%Section 10the design

Rows 12 to 18 are the seven Chapter 26.1 §12 deferred, and their owner column is in capitals because that is the review's finding: not one of them belongs to the design team, and all seven have a default in the RTL.

Row 19 is the counter-example that makes the point. The occupancy alarm looks environmental and is not — it is structural, because the ceiling is a property of the table's associativity — so it is the one threshold in this group the design can set correctly with no outside input, and the industry default of 90% is wrong anyway.

Row 8 against row 7 is the chapter's second finding and the subject of Section 12. A design that carries a 1 024-sample latency window and reports a p99.99 is reporting the maximum of 1 024 samples, whose expected percentile is 99.902 and whose value here is 5.615 µs against a true p99.99 of 7.475 µs — understated by 24.88%. The number is not noisy; it is biased, low, by construction, and it gets better-looking as the window gets smaller.


3. RTL 1 — The Performance Package and the Threshold Model

The package holds the seven thresholds, the three kinds of owner and the constants the rest of the chapter computes with. The module decides what state one threshold is in.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// perfreview_pkg -- the seven deferred thresholds of Chapter 26.1
// Section 12, the four outcomes a threshold can be in, and the latency
// constants Sections 6 to 12 derive.
//
// Times are in nanoseconds unless a name says otherwise. Areas are in
// BCE, one bit of usable on-die SRAM, 0.35 gate-equivalents, twenty to
// a flip-flop -- the unit Chapter 19.7 Section 19 established.
// ---------------------------------------------------------------------
package perfreview_pkg;

  // The seven. Each one has a default in the RTL and an owner who is
  // not on the design team.
  typedef enum logic [2:0] {
    TH_BUFFER,      // the burst distribution      -- the deployment
    TH_TABLE,       // the address population      -- the deployment
    TH_AGEING,      // the address recycle rate    -- the orchestrator
    TH_REPLAY,      // the fabric's path skew      -- the topology
    TH_STORM,       // the legitimate flood rate   -- the deployment
    TH_FLAP,        // the legitimate move rate    -- the site
    TH_WATERMARK    // the link's dead time        -- the cable plant
  } threshold_e;

  localparam int N_THRESH = 7;

  typedef enum logic [1:0] {
    OWN_DESIGN,       // derivable here
    OWN_STRUCTURAL,   // follows from the topology of the design itself
    OWN_EXTERNAL,     // somebody outside the room
    OWN_SILICON       // nobody, until first bring-up -- Section 17
  } owner_e;

  typedef enum logic [1:0] {
    TH_UNSET,       // no value at all
    TH_DEFAULT,     // a value, chosen by an implementer, not exposed
    TH_DEFERRED,    // owner named, parameter exposed
    TH_BOUND        // owner answered, value recorded
  } th_state_e;

  // Latency constants, from Chapter 26.1 Section 8's reconciliation.
  localparam int HOP_FLOOR_NS      = 913;      // 912.88, rounded
  localparam int HOPS              = 5;
  localparam int FIVE_HOP_FLOOR_NS = 4564;     // 4.564 us
  localparam int DESIGN_OWNED_NS   = 80;       // lookup 30 + fabric 50
  localparam int DESIGN_OWNED_PPM  = 87623;    // 8.7623 per cent

  // The buffer, from Chapter 23.3 Section 9. One port claiming its
  // dynamic-threshold share is 108 616 cells of 256 octets.
  localparam int  CELL_OCTETS        = 256;
  localparam int  MAX_CELLS_ONE_PORT = 108616;
  localparam int  MAX_QUEUE_US       = 2224;

  // Percentiles are an index into a distribution, not a unit, so this
  // package carries the window alongside every one of them. Section 12.
  typedef struct packed {
    logic [31:0] value_ns;
    logic [19:0] percentile_ppm;   // 999900 is p99.99
    logic [31:0] window_samples;
  } latency_claim_t;

  // The rule Section 12 derives: a window of W samples can support a
  // percentile no finer than 1 - 1/W. Expressed as the minimum window
  // for ten expected exceedances, which is the usable form.
  function automatic int unsigned min_window(int unsigned pct_ppm);
    if (pct_ppm >= 1000000) return 32'hFFFF_FFFF;
    return (10 * 1000000) / (1000000 - pct_ppm);
  endfunction

  // Observability, Section 10, in BCE at 64 ports.
  localparam int PORTS             = 64;
  localparam int DATAPATH_BCE      = 283320;
  localparam int SWITCH_BCE_DIV1K  = 562000;   // 5.62e8 / 1000

endpackage

Classification, Model or Production? A CLASSIFICATION MODEL. It decides which of four states one threshold is in, and — critically — it distinguishes a value that was chosen from a value that was deferred and then defaulted, which is the distinction Chapter 26.1 §12's scenarios 47 and 48 turned on.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// threshold_model -- one threshold in, its state and its owner out.
//
// The module's whole value is the DEFAULT state. A threshold with a
// value looks identical to one with a derivation in every report this
// track has examined, and the difference is whether anybody could
// change it in the field.
// ---------------------------------------------------------------------
module threshold_model
  import perfreview_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        th_valid,
  input  threshold_e  th_id,
  input  logic        has_value,        // the RTL has a number
  input  logic        is_exposed,       // the number is a register
  input  logic        owner_named,      // somebody is accountable
  input  logic        owner_answered,   // and they supplied a value
  input  logic        needs_measurement,// only silicon can supply it

  output logic        out_valid,
  output th_state_e   state,
  output owner_e      owner,
  output logic        became_a_default, // the failure Section 4 names
  output logic        deferred_correctly
);

  // Which owner each threshold has. Six external, one structural, and
  // Section 17's one that no conversation can close.
  function automatic owner_e owner_of(threshold_e t, logic meas);
    if (meas) return OWN_SILICON;
    case (t)
      TH_BUFFER, TH_TABLE, TH_STORM : return OWN_EXTERNAL;
      TH_AGEING, TH_FLAP            : return OWN_EXTERNAL;
      TH_REPLAY                     : return OWN_STRUCTURAL;
      TH_WATERMARK                  : return OWN_EXTERNAL;
      default                       : return OWN_DESIGN;
    endcase
  endfunction

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      out_valid          <= 1'b0;
      state              <= TH_UNSET;
      owner              <= OWN_DESIGN;
      became_a_default   <= 1'b0;
      deferred_correctly <= 1'b0;
    end else begin
      out_valid <= th_valid;
      if (th_valid) begin
        owner <= owner_of(th_id, needs_measurement);

        if (owner_answered && has_value) begin
          state              <= TH_BOUND;
          became_a_default   <= 1'b0;
          deferred_correctly <= 1'b0;
        end else if (owner_named && is_exposed) begin
          // The correct handling: named, exposed, awaiting an answer.
          state              <= TH_DEFERRED;
          became_a_default   <= 1'b0;
          deferred_correctly <= 1'b1;
        end else if (has_value) begin
          // A number, no owner, not exposed. This is the state every
          // one of the opening table's four defaults was in.
          state              <= TH_DEFAULT;
          became_a_default   <= 1'b1;
          deferred_correctly <= 1'b0;
        end else begin
          state              <= TH_UNSET;
          became_a_default   <= 1'b0;
          deferred_correctly <= 1'b0;
        end
      end
    end
  end

endmodule

What it teaches: that TH_DEFAULT and TH_BOUND look identical from outside the design and are opposites. Both have a value; both produce a working product; only one of them can be corrected when the deployment turns out to differ from the implementer's assumption. The module also teaches that is_exposed is doing the real work — a threshold with a named owner and no register is still a default, because naming an owner who cannot change anything is a bookkeeping entry rather than a deferral.

Deliberately simplified: owner_of is a case statement where a real design reads the owner from a specification, so the mapping is hard-coded in exactly the way this chapter argues against — the honest form makes the owner a field in the same register that exposes the parameter, so the two travel together. needs_measurement is also an input rather than a derivation, where Section 17's argument is that exactly one threshold has this property and the reason is structural.

Production implication: the output worth wiring to a build-time check is became_a_default, and it should fail the build. A parameter with a value, no owner and no register is a deployment decision made at RTL time, and the four rows of this chapter's opening table are the price of making it. The cost of the check is one assertion per parameter in a parameter package; the cost of skipping it is a product whose pause watermark is 34 cells short on the longest link it is sold for, discovered by a customer.


4. The Seven Environmental Thresholds, Closed

Seven thresholds that a design cannot derive for itself, each with the party who owns it. Buffer size, table size and storm control belong to the deployment and are closed by a measurement from a production site. The ageing interval belongs to the orchestrator, which already emits the teardown rate. The replay window belongs to the topology and is closed by a design document. The flap threshold belongs to the site. And the pause watermark belongs to the cable plant, but its dead time includes the response latency of a partner device that somebody else made, so no single owner can supply it and it stays open until first bring up measures it. Six of the seven close with a conversation or a query. Each one has a default in the register transfer level design today, and this track has shown four of those defaults to be wrong by a large factor in a real deployment: three hundred second ageing is eight times too long, a seventy five per cent pause watermark is thirty four cells short at a hundred gigabits over two kilometres, a ninety per cent occupancy alarm is unreachable against a ceiling of eighty point five, and one per cent storm control on unknown unicast discards legitimate user data.7 thresholdsChapter 26.1 §12The deploymentbuffer, table, stormThe orchestratorageing — it emits therateTopology andsitereplay window, flapThe partnerdevicewatermark — nobodyowns itClosed by aconversation6 of 7Closed atbring-up1 of 7 — a procedure,not a valueLeft as defaults4 proven wrong inthis track12
Figure 1 — seven thresholds, seven owners, and the one that no conversation can close.

Chapter 26.1 §12 named them. This section says, for each one, what measurement sets it, who can make that measurement, and what the design must expose so the answer can be used.

ThresholdThe measurement that sets itWho can make itWhat the design must expose
buffer sizethe burst-length distribution at the deployment's offered loadthe deployment, from a week of telemetryper-queue maximum watermark, and the cell occupancy
MAC table sizethe address population, counted, plus the recycle ratethe deploymentChapter 25.5 §16's effective-capacity register, and per-set refusals
ageing intervalthe teardown rate — 10 000 an hour gives 833 stale entriesthe orchestrator, which already emits itthe interval as a register, and a stale-entry count
replay windowthe fabric's worst-case path skewthe topology's designer, from a documentthe window as a register, and a reorder-depth histogram
storm controlthe legitimate flood rate, separately for broadcast and unknown unicastthe deploymentTWO settings, not one — Chapter 25.5 §12
flap thresholdthe legitimate move ratethe sitethe threshold, and a per-address move counter
pause watermarkthe dead time — Section 17NOBODY, until first bring-upthe watermark as a register, and the measured dead time

Three of the seven need more than a register, and those three are where the observability bill of Section 10 comes from.

One — the buffer threshold needs a watermark, not an occupancy. Chapter 23.3 §9's buffer holds 2 224 µs at one port and a sampled occupancy will almost never catch the peak, because the peak is 35.302 µs wide (Section 8) and a management poll is seconds apart. The instrument is a per-queue maximum-since-last-read register — 512 queues at 32 bits, 16 384 BCE, 0.06 of a datapath — and without it the burst distribution cannot be measured at all.

Two — the storm-control threshold is two thresholds. Chapter 25.5 §12's finding is that a limiter on broadcast is correct and a limiter on unknown unicast discards legitimate user data to work around a table refusing inserts. A design exposing one setting invites one setting, so the exposure is itself the design decision: two registers, and the default on the unknown-unicast one is off.

Three — the MAC table threshold needs the refusals, not the occupancy. Chapter 25.5 §6's arithmetic: a 128k four-way table offered 131 072 addresses stores 105 465 and refuses 25 607 — 19.537% — while the occupancy counter reads 80.5%. So the occupancy metric cannot detect the condition; the per-set refusal counter can, at 32 768 sets by 4 bits — 131 072 BCE, 0.46 datapaths, 0.023% of the switch — and it turns a theorem into a histogram.

For four of the seven the design already has the instrument and hides it. For three of them the instrument does not exist and costs less than half a MAC datapath each.


5. RTL 2 — The Deferral Register

Seven thresholds, seven owners, seven registers. This module is the register file and the audit of it.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// deferral_register -- the seven thresholds as live, writable state,
// each with its owner, its default, whether it has been answered, and
// whether the design exposes it.
//
// The module is the artefact Chapter 26.1 Section 12 said an
// architecture review should produce. Here it is, as registers.
// ---------------------------------------------------------------------
module deferral_register
  import perfreview_pkg::*;
(
  input  logic            clk,
  input  logic            rst_n,

  // The management write port -- this is the exposure.
  input  logic            wr_en,
  input  threshold_e      wr_id,
  input  logic [31:0]     wr_value,

  // The review's annotations, loaded at configuration time.
  input  logic            cfg_en,
  input  threshold_e      cfg_id,
  input  logic            cfg_owner_named,
  input  logic            cfg_exposed,
  input  logic            cfg_needs_silicon,

  input  logic [31:0]     defaults [N_THRESH],

  output logic [31:0]     value      [N_THRESH],
  output logic            answered   [N_THRESH],
  output logic            exposed    [N_THRESH],
  output logic            owner_named[N_THRESH],
  output logic            silicon_gated [N_THRESH],

  output logic [3:0]      n_bound,
  output logic [3:0]      n_deferred,
  output logic [3:0]      n_defaulted,
  output logic [3:0]      n_silicon_gated,
  output logic            all_thresholds_disposed
);

  integer i;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (i = 0; i < N_THRESH; i = i + 1) begin
        value[i]         <= defaults[i];
        answered[i]      <= 1'b0;
        exposed[i]       <= 1'b0;
        owner_named[i]   <= 1'b0;
        silicon_gated[i] <= 1'b0;
      end
    end else begin
      if (cfg_en) begin
        owner_named[cfg_id]   <= cfg_owner_named;
        exposed[cfg_id]       <= cfg_exposed;
        silicon_gated[cfg_id] <= cfg_needs_silicon;
      end
      // A write is the owner answering. It is the ONLY way answered
      // becomes high, which is why exposure and accountability are the
      // same mechanism rather than two.
      if (wr_en && exposed[wr_id]) begin
        value[wr_id]    <= wr_value;
        answered[wr_id] <= 1'b1;
      end
    end
  end

  // The four counts of Section 4, recomputed combinationally so a
  // management read always sees the current disposition.
  always_comb begin
    n_bound         = '0;
    n_deferred      = '0;
    n_defaulted     = '0;
    n_silicon_gated = '0;
    for (int j = 0; j < N_THRESH; j = j + 1) begin
      if (silicon_gated[j] && !answered[j])
        n_silicon_gated = n_silicon_gated + 4'd1;
      else if (answered[j])
        n_bound = n_bound + 4'd1;
      else if (owner_named[j] && exposed[j])
        n_deferred = n_deferred + 4'd1;
      else
        n_defaulted = n_defaulted + 4'd1;
    end
    // Disposed means every threshold is in a state somebody chose.
    // A defaulted threshold is the one state nobody chose.
    all_thresholds_disposed = (n_defaulted == 4'd0);
  end

endmodule

Classification, Model or Production? PRODUCTION-INTENT. Every design that ships with configurable thresholds has most of this; what it does not have is the owner_named and exposed annotations beside the values, which is the part that makes the register file a review artefact rather than a configuration block.

What it teaches: that a write only lands if the parameter is exposed, and that this is not a safety feature — it is the definition. A threshold that can be changed only by rebuilding is not exposed, whatever the documentation says, and the module makes the two statements the same statement. It also teaches why n_silicon_gated is counted before n_bound: a threshold awaiting a measurement that does not exist yet is in a different state from one awaiting an answer from a person, and Section 17 is about the single member of that category.

Deliberately simplified: the annotations are loaded through a configuration port where in practice they are compile-time constants in a parameter package, so a real implementation makes them localparam and this module's cfg_en path becomes a generate-time elaboration. The register file also has no access control — any writer is treated as the owner — where a real design partitions the address space so that the orchestrator's thresholds and the cable plant's thresholds are not writable by the same agent. And defaults arrives as an unpacked array input, which is a testbench convenience rather than a synthesisable form.

Production implication: all_thresholds_disposed is the bit to read at the end of bring-up and it should be high. It is low exactly when some parameter has a value nobody chose, which is Chapter 26.1 §12's failure rendered as one bit of status. The operational value is that it is readable from the field: a support engineer looking at a misbehaving deployment can ask the part whether any of its seven thresholds is a default, and get an answer in one register read instead of a three-week correspondence with the vendor.


6. The Percentile the Floor Was Missing

A five hop latency budget at one hundred gigabits per second, evaluated four ways. The structural floor from the architecture review is four point five six four microseconds and it describes an idle network. At ninety five per cent offered load the median is seven point one eight four microseconds, which is one point five seven times the floor. The ninety ninth percentile at the same load is twenty three point one four eight microseconds, which is five point zero seven times. The ninety nine point nine ninth percentile is forty one point nine four one microseconds, which is nine point one nine times. Against a five microsecond specification the structural floor passes with eight point seven per cent margin and the deep tail fails by eight point three nine times, and both readings are defensible because the specification never said which one it meant. Uncongested, the design owns only eight point seven six per cent of a hop because propagation is fifty three point seven per cent. Congested, eighty nine point one per cent of the deep tail is queueing, and queueing is the buffer's size and the scheduler's policy, which are the design's.4.564 µsstructural floor —idle7.184 µsp50 at ρ = 0.95 —1.57×23.148 µsp99 — 5.07×41.941 µsp99.99 — 9.19×A 5 µs budgetone sentence, fourverdictsPASSES8.7% margin, on thefloorFAILS 8.39×on the deep tailDesign owns8.76%idle — 89.1% whencongested12
Figure 2 — one five-hop budget, four defensible answers, and a factor of nine between the first and the last.

Chapter 26.1 §8 produced seven latency terms and one total, flagged that every term was uncongested, and said the review's obligation was to name a percentile. This section names it.

The queueing model, stated so it can be argued with. One hop is a single server with Poisson arrivals and the frame-length distribution of Chapter 26.2 §8 — mean 510.09 wire octets, 40.81 ns, so the server rate is 2.4505 × 10⁷ frames per second. Waiting time in queue then has the closed form P(W > t) = ρ · e^(−(1−ρ)μt), and a percentile inverts it.

Five hops, and every cell is the floor plus five independent queues.

Offered load ρp50p90p99p99.9p99.99
0.504.564 µs5.2216.1617.1008.040
0.805.0446.6869.03511.38413.733
0.905.7649.04813.74618.44423.142
0.957.18413.75123.14832.54441.941
0.9918.50251.34198.323145.305192.287

The top-left cell is Chapter 26.1 §8's 4.564 µs, and it is the only cell in the table an architecture review can produce. Everything else needs an offered load.

Three readings of the table, and the third is the one that changes a design.

One — the floor is the answer only at the top-left. At ρ = 0.5 and p50 the queueing contribution is 0.000 µs and the structural figure is exact. One row down and four columns right, at ρ = 0.8 and p99.99, it is 13.733 µs — three times the floor — and neither figure is wrong.

Two — the spread within one row is larger than the spread between rows near the middle. At ρ = 0.95 the latency runs 7.184 to 41.941 µs across the percentile axis, a factor of 5.84; from ρ = 0.5 to ρ = 0.95 at p50 it runs 4.564 to 7.184, a factor of 1.57. So a specification that names a load and not a percentile has pinned down the less important of the two axes.

Three — the design owns 8.76% of the floor and essentially all of the queueing. Chapter 26.1 §8's finding was that propagation is 53.7% of a hop and the design owns 80 ns of 912.88. Under load that inverts: at ρ = 0.95 and p99.99, 37.377 of 41.941 µs is queueing — 89.1% — and queueing is the buffer's size and the scheduler's policy, which are the design's.

Uncongested, a latency argument is about physics and the design is a bystander. Congested, it is entirely about the design. The percentile is what decides which conversation the review is having.

And the practical form of the review item, which is four numbers rather than one.

ComponentThis design
the structural floor4.564 µs, five hops
the load the specification is written atρ = 0.95
the percentilep99.99
the figure41.941 µs

A review that records only the first has recorded the one number that is never the answer, which is the sentence Chapter 26.1 §8 ended on, now with the other three supplied.


7. RTL 3 — The Latency Percentile Estimator

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// latency_percentile -- estimate a percentile from a stream of measured
// latencies, and REFUSE to report one the window cannot support.
//
// The refusal is the module's reason for existing. Section 12 shows a
// 1024-sample window reporting a p99.99 that is really a p99.902,
// biased low by 24.88 per cent, with no indication that anything is
// wrong with the number.
// ---------------------------------------------------------------------
module latency_percentile
  import perfreview_pkg::*;
#(
  parameter int unsigned WINDOW    = 1024,
  // Log-spaced buckets, so the tail is resolved without storing samples.
  parameter int unsigned N_BUCKETS = 32,
  parameter int unsigned BUCKET_SHIFT = 6    // 64 ns per bucket at the base
)(
  input  logic          clk,
  input  logic          rst_n,
  input  logic          clear,

  input  logic          sample_valid,
  input  logic [31:0]   sample_ns,

  // What the caller wants, in parts per million: 999900 is p99.99.
  input  logic [19:0]   want_ppm,

  output logic [31:0]   estimate_ns,
  output logic [31:0]   samples_seen,
  output logic [31:0]   window_required,
  output logic          estimate_is_supported,   // the refusal
  output logic [19:0]   best_supported_ppm,
  output logic [31:0]   observed_max_ns,
  output latency_claim_t claim                   // value, percentile, window
);

  logic [31:0] bucket [N_BUCKETS];
  logic [31:0] total;

  // Log-spaced bucket index: the leading one of the sample, floored.
  function automatic int unsigned bucket_of(logic [31:0] ns);
    int unsigned b;
    b = 0;
    for (int k = 31; k >= 0; k = k - 1)
      if (ns[k]) begin b = k; break; end
    if (b < BUCKET_SHIFT) return 0;
    if ((b - BUCKET_SHIFT) >= N_BUCKETS) return N_BUCKETS - 1;
    return b - BUCKET_SHIFT;
  endfunction

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin
      for (int i = 0; i < N_BUCKETS; i = i + 1) bucket[i] <= '0;
      total           <= '0;
      observed_max_ns <= '0;
    end else if (sample_valid) begin
      bucket[bucket_of(sample_ns)] <= bucket[bucket_of(sample_ns)] + 32'd1;
      total <= total + 32'd1;
      if (sample_ns > observed_max_ns) observed_max_ns <= sample_ns;
    end
  end

  // Walk the buckets from the top until the requested tail mass is
  // accounted for. The answer is a bucket boundary, which is why the
  // buckets are log-spaced: a linear scale would resolve the median
  // beautifully and the tail not at all.
  logic [31:0] target_tail;
  logic [31:0] running;
  logic [31:0] est;

  always_comb begin
    samples_seen    = total;
    window_required = min_window(want_ppm);
    // The refusal: ten expected exceedances, or the number is a maximum
    // wearing a percentile's name.
    estimate_is_supported = (total >= window_required) && (total != 0);

    // The finest percentile this window can actually support.
    best_supported_ppm = (total == 0) ? 20'd0
                       : 20'(1000000 - (10 * 1000000) / int'(total));

    target_tail = 32'((longint'(total) *
                       longint'(1000000 - int'(want_ppm))) / 1000000);
    running = '0;
    est     = observed_max_ns;
    for (int i = N_BUCKETS - 1; i >= 0; i = i - 1) begin
      running = running + bucket[i];
      if (running >= target_tail) begin
        est = 32'(32'd1 << (i + BUCKET_SHIFT));
        break;
      end
    end
    estimate_ns = est;

    claim.value_ns       = est;
    claim.percentile_ppm = want_ppm;
    claim.window_samples = total;
  end

endmodule

Classification, Model or Production? PRODUCTION-INTENT. A 32-bucket log-spaced histogram per port is the structure real switches carry, and estimate_is_supported is the one output they do not.

What it teaches: that a percentile and its window travel together or the percentile is fiction. claim is a three-field struct for exactly this reason — value, percentile, window — and a management interface that returns only the first field has returned a number that cannot be checked. The module also teaches why the buckets are log-spaced: a linear histogram with 32 buckets over the p50-to-p99.99 range at ρ = 0.95, which is 1.437 µs to 8.388 µs per hop, puts twenty-nine buckets below the median and three in the tail. Log spacing puts about five in each decade, which is where the resolution is needed.

Deliberately simplified: the estimate is a bucket boundary, so its resolution is one octave — at the tail that is a factor of two, which is coarse. A real implementation interpolates within the bucket using its count, which costs a multiply and a divide and recovers most of the resolution. The bucket_of leading-one search is written as a loop with a break for readability; in silicon it is a priority encoder. And min_window uses ten expected exceedances, which is a convention rather than a theorem — the right number depends on how much variance in the estimate is acceptable, and ten gives roughly ±30%.

Production implication: wire estimate_is_supported to the management read and return an error rather than a number when it is low. The alternative — returning the best available estimate with a note — is worse than it sounds, because the note is lost the first time somebody copies the value into a spreadsheet. A register that refuses to answer a question it cannot answer is the only reliable way to stop a 1 024-sample maximum from being quoted as a p99.99 for the product's life, which Section 12 shows is a 24.88% understatement that looks entirely plausible.


8. Why 487× Is Not a Percentile

Queueing on a switch has two regimes and they need different kinds of number. Below an offered load of one there is a steady state, the waiting time has a distribution, and a percentile is the right question: at ninety five per cent load the ninety nine point nine ninth percentile of five hop latency is forty one point nine four one microseconds. In that regime the buffer's maximum of two thousand two hundred and twenty four microseconds is unreachable: at ninety nine per cent load the probability of a wait that long is two times ten to the minus two hundred and thirty seventh. At or above an offered load of one there is no steady state and no percentile. An incast of N senders into one port fills the queue at N minus one line rates and drains at one, so the buffer's maximum share of twenty seven point eight megabytes fills in two thousand two hundred and twenty four microseconds with two senders and in thirty five point three zero two microseconds with sixty four, while the drain takes two thousand two hundred and twenty four microseconds in every case. The asymmetry is exactly N minus one, and an all reduce in an artificial intelligence training cluster is an N way incast by construction.Queueingtwo regimesρ < 1a steady state existsρ ≥ 1no steady stateA percentile41.941 µs at p99.992 224 µs?P = 2e-237 at ρ =0.99A fill time35.302 µs at 64sendersA drain time2 224 µs, whatever Nis63 : 1the asymmetry is N −112
Figure 3 — two regimes, and the buffer's maximum belongs to the one with no steady state.

Chapter 26.1 §8 put the buffer's maximum queueing delay — 2 224 µs — against the five-hop floor and got a ratio of 487. That number is correct and it is not a tail of Section 6's distribution. It belongs to a different regime, and confusing the two is the most common error in a performance review.

Ask Section 6's model what it thinks of 2 224 µs.

Offered loadP(W > 2 224 µs)
ρ = 0.950 to the precision of any arithmetic
ρ = 0.992.04 × 10⁻²³⁷

At ρ = 0.99 the model says a frame waits 2 224 µs about once in every 5 × 10²³⁶ frames, which at 24.5 million frames a second is a wait longer than the age of the universe by two hundred and twenty orders of magnitude. The buffer's maximum is not a percentile of the steady-state queue. It is unreachable in steady state at any load below one.

And yet the buffer fills, routinely, in real fabrics. The resolution is that ρ < 1 is the wrong model for the event that fills it.

An incast is ρ ≥ 1. N senders transmit at line rate into one egress port; the port drains at line rate; the queue grows at (N − 1) × line rate until the senders stop. There is no steady state and no percentile — there is a fill time.

SendersFill rateTime to reach 27.8 MBDrain time
212.5 GB/s2 224 µs2 224 µs
437.5 GB/s741.3 µs2 224 µs
887.5 GB/s317.7 µs2 224 µs
16187.5 GB/s148.3 µs2 224 µs
32387.5 GB/s71.7 µs2 224 µs
64787.5 GB/s35.302 µs2 224 µs

The drain column does not move, because drain is always one line rate. So the asymmetry is exactly N − 1a 64-way incast fills one port's maximum share in 35.302 µs and takes 2 224 µs to recover, a ratio of 63:1 — and during that recovery every frame behind it carries the full 2 224 µs.

Steady-state queueing is a distribution and needs a percentile. Incast is an event and needs a fill time and a recovery time. A review that has one and not the other has described half its design's latency behaviour.

And Chapter 23.2 is the deployment where the second half dominates. An all-reduce is an N-way incast by construction — every participant sends to the same reducer at the same instant — so the regime this section describes is not an edge case in that market, it is the workload.

Which gives the review three latency numbers instead of one, and they answer three different questions.

NumberRegimeQuestion it answers
4.564 µsidlewhat does the structure cost?
41.941 µsρ = 0.95, p99.99what does a loaded but stable fabric cost?
2 224 µs after 35.302 µsincast, ρ ≥ 1what does a collective cost?

Three numbers, three regimes, and Chapter 26.1 §8 could produce only the first, because the other two need an offered load and a traffic pattern that an architecture review does not have. This is what it means for a threshold to be environmental.


9. RTL 4 — The Incast Model

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// incast_model -- the regime Section 8 separates from Section 6. No
// percentiles here, because there is no steady state: an incast has a
// fill time, a peak and a recovery time.
//
// The module's output that matters is regime_is_overload, because every
// queueing statistic computed while it is high is meaningless.
// ---------------------------------------------------------------------
module incast_model
  import perfreview_pkg::*;
#(
  // Octets per second per sender at 100 Gb/s, in units of 1e6 oct/s.
  parameter int unsigned LINE_MOCT_S = 12500,
  parameter int unsigned CAP_OCTETS  = 27800000    // 108 616 cells
)(
  input  logic          clk,
  input  logic          rst_n,

  input  logic          eval,
  input  logic [7:0]    n_senders,
  input  logic [31:0]   burst_octets_per_sender,

  output logic          regime_is_overload,
  output logic [31:0]   fill_rate_moct_s,
  output logic [31:0]   fill_time_us,
  output logic [31:0]   drain_time_us,
  output logic [31:0]   asymmetry,        // drain / fill, which is N-1
  output logic          buffer_overflows,
  output logic [31:0]   octets_dropped,
  output logic [31:0]   peak_queue_octets
);

  logic [63:0] offered;
  logic [63:0] peak;

  always_comb begin
    // Overload is a comparison of rates, not a probability. One sender
    // into one port is rho = 1 exactly and never builds a queue; two
    // is the first overload case.
    regime_is_overload = (n_senders > 8'd1);

    fill_rate_moct_s = regime_is_overload
                     ? 32'((int'(n_senders) - 1) * int'(LINE_MOCT_S))
                     : 32'd0;

    // Microseconds to reach the cap, at the fill rate. Units: octets
    // divided by 1e6 octets per second gives microseconds directly.
    fill_time_us = (fill_rate_moct_s == 0) ? 32'hFFFF_FFFF
                 : 32'(CAP_OCTETS / fill_rate_moct_s);

    // Drain is always one line rate, whatever N is. This is the row
    // of Section 8's table that does not move.
    drain_time_us = 32'(CAP_OCTETS / LINE_MOCT_S);

    asymmetry = (fill_time_us == 0) ? 32'd0
              : 32'(drain_time_us / fill_time_us);

    // How much the senders actually offer, against what fits.
    offered = 64'(n_senders) * 64'(burst_octets_per_sender);
    peak    = (offered > 64'(CAP_OCTETS)) ? 64'(CAP_OCTETS) : offered;

    peak_queue_octets = 32'(peak);
    buffer_overflows  = (offered > 64'(CAP_OCTETS));
    octets_dropped    = buffer_overflows
                      ? 32'(offered - 64'(CAP_OCTETS)) : 32'd0;
  end

endmodule

Classification, Model or Production? A BEHAVIOURAL MODEL — this is NOT how a buffer works. A real shared buffer allocates in cells with a dynamic threshold, the senders do not start simultaneously, and congestion control reacts within a round trip. The model computes the bound, not the trajectory.

What it teaches: that the drain time does not depend on N and the fill time does, which is the whole shape of an incast and the reason it is not describable as a percentile. Sixty-four senders fill the buffer 63 times faster than two do and recover at exactly the same speed — so as a fabric gets wider, the event gets sharper and the recovery does not, and the ratio of the two is the number a scheduler has to survive.

Deliberately simplified: the senders are synchronous and unthrottled, which is the worst case and not the common one; a real all-reduce has skew between participants and a congestion-control loop that reacts in about one round trip. The cap is the single-port maximum from Chapter 23.3 §9 rather than a dynamic threshold that shrinks as other ports queue — so CAP_OCTETS is itself a function of the rest of the switch's state, and treating it as a constant overstates one port's share whenever anything else is busy. And octets_dropped assumes drops happen at the tail, where a real design may use a drop policy that discards from elsewhere.

Production implication: regime_is_overload is the signal to gate every queueing statistic on. A p99 latency computed over an interval that contained an incast is not a p99 of anything — it is a mixture of two distributions with no common parameter, and the mixture weight is the fraction of the interval spent in overload. The cheapest correct behaviour is to keep two histograms, one for each regime, switched by this bit, which doubles Section 10's histogram cost from 0.231 datapaths to 0.463 — 0.0233% of the switch — and makes both numbers mean something.


10. What the Measurement Costs in Silicon

Sections 4, 6 and 8 asked a design to measure seven thresholds, a latency distribution and an overload regime. This section prices the instruments, in BCE, against Chapter 23.3's 64-port switch.

Seven structures, and every row names the section that demanded it.

InstrumentGeometryBitsBCE× a datapath% of the switch
ingress timestamp through the pipeline64 ports × 48 b × 36 stages110 592110 5920.390.0197%
latency histogram64 × 32 buckets × 32 b65 53665 5360.230.0117%
latency window FIFO64 × 1 024 × 16 b1 048 5761 048 5763.700.1866%
per-queue maximum watermark512 queues × 32 b16 38416 3840.060.0029%
per-set refusal counter32 768 sets × 4 b131 072131 0720.460.0233%
the counter block at Chapter 19.7's widths64 × 992 b63 48863 4880.220.0113%
dead-time and flap timers64 × 4 × 32 b8 1928 1920.030.0015%
TOTAL1 443 8401 443 8405.100.2569%

Everything this chapter asks a 6.4 Tb/s switch to measure costs five MAC receive datapaths — a quarter of one per cent of its on-die state.

Four observations, and the last is the one worth taking to a product meeting.

One — the window FIFO is 72.6% of the bill on its own, and Section 12 shows it is also the item that cannot do the job it was added for. The histogram at 0.23 datapaths resolves the same tail sixteen times cheaper, which makes the largest line in this table also the most questionable one.

Two — the counter block reconciles with Chapter 23.3's own bill, and slightly in the safe direction. That chapter budgeted RMON at 64 × 27 × 64 b = 110 592 BCE, 0.39 of a datapath. Chapter 19.7 §2's actual design is four counters at 64 bits and twenty-three at 32 — 992 bits per port, 63 488 BCE — so the switch chapter over-budgeted by a factor of 1.742 and the reconciliation lands 0.17 of a datapath better than planned. Both figures are rounding errors at this scale and that is the point: no observability argument has ever been won or lost on area.

Three — the per-set refusal counter is the highest-value row per BCE. Chapter 25.5 §6's table refuses 19.537% of offered addresses while the occupancy counter reads 80.5%; the refusal counter is the only instrument that can see it, and it costs 0.46 of a datapath and 0.0233% of the die.

Four — the whole bill is smaller than one rounding decision on the packet buffer. Chapter 23.3 §6: the buffer is 5.12 × 10⁸ BCE, 91.0% of the switch, and a 10% trim of it saves 181 datapaths. This chapter's complete observability bill is 5.10. Trimming the buffer by 0.282% pays for all of it.


11. RTL 5 — The Latency Histogram and Watermark Bank

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// obs_bank -- the instruments Section 10 prices, as one block: a
// log-spaced latency histogram, a per-queue maximum watermark, and the
// regime bit that says which histogram a sample belongs in.
//
// The watermark is the structure Section 4 argues for over a sampled
// occupancy: a 35-microsecond peak is invisible to a poll seconds apart.
// ---------------------------------------------------------------------
module obs_bank
  import perfreview_pkg::*;
#(
  parameter int unsigned N_QUEUES  = 512,
  parameter int unsigned N_BUCKETS = 32,
  parameter int unsigned BUCKET_SHIFT = 6
)(
  input  logic          clk,
  input  logic          rst_n,

  // Latency samples, tagged with the regime they were taken in.
  input  logic          lat_valid,
  input  logic [31:0]   lat_ns,
  input  logic          in_overload,       // from Section 9

  // Queue occupancy, sampled every cycle rather than polled.
  input  logic          occ_valid,
  input  logic [8:0]    occ_queue,
  input  logic [31:0]   occ_cells,

  // Management read, which clears the watermark it returns.
  input  logic          rd_en,
  input  logic [8:0]    rd_queue,

  output logic [31:0]   rd_watermark,
  output logic [31:0]   hist_steady  [N_BUCKETS],
  output logic [31:0]   hist_overload[N_BUCKETS],
  output logic [31:0]   n_steady,
  output logic [31:0]   n_overload,
  output logic [19:0]   overload_share_ppm
);

  logic [31:0] watermark [N_QUEUES];

  function automatic int unsigned bucket_of(logic [31:0] ns);
    int unsigned b;
    b = 0;
    for (int k = 31; k >= 0; k = k - 1)
      if (ns[k]) begin b = k; break; end
    if (b < BUCKET_SHIFT) return 0;
    if ((b - BUCKET_SHIFT) >= N_BUCKETS) return N_BUCKETS - 1;
    return b - BUCKET_SHIFT;
  endfunction

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int i = 0; i < N_BUCKETS; i = i + 1) begin
        hist_steady[i]   <= '0;
        hist_overload[i] <= '0;
      end
      for (int q = 0; q < N_QUEUES; q = q + 1) watermark[q] <= '0;
      n_steady     <= '0;
      n_overload   <= '0;
      rd_watermark <= '0;
    end else begin
      // Two histograms, switched by the regime bit. Section 9's
      // production note: a percentile over a mixture is a percentile
      // of nothing.
      if (lat_valid) begin
        if (in_overload) begin
          hist_overload[bucket_of(lat_ns)] <=
            hist_overload[bucket_of(lat_ns)] + 32'd1;
          n_overload <= n_overload + 32'd1;
        end else begin
          hist_steady[bucket_of(lat_ns)] <=
            hist_steady[bucket_of(lat_ns)] + 32'd1;
          n_steady <= n_steady + 32'd1;
        end
      end

      // The watermark: maximum since last read, updated every cycle.
      if (occ_valid && (occ_cells > watermark[occ_queue]))
        watermark[occ_queue] <= occ_cells;

      // Read-and-clear, so consecutive reads are disjoint intervals
      // rather than a running maximum that saturates at the first peak.
      if (rd_en) begin
        rd_watermark        <= watermark[rd_queue];
        watermark[rd_queue] <= '0;
      end
    end
  end

  always_comb begin
    overload_share_ppm = ((n_steady + n_overload) == 0) ? 20'd0
                       : 20'((longint'(n_overload) * 1000000)
                             / longint'(n_steady + n_overload));
  end

endmodule

Classification, Model or Production? PRODUCTION-INTENT, and the read-and-clear watermark is the part most worth copying verbatim.

What it teaches: that read-and-clear turns a maximum into a time series. A running maximum saturates at the first peak and reports the same value forever; clearing on read makes each read the maximum over the interval since the previous one, so a management system polling every second gets a per-second peak — which is the burst distribution Section 4 needs to set the buffer threshold. It also teaches that the regime bit costs one flip-flop and doubles the histogram, which is the correct trade because the alternative is one histogram whose percentiles are meaningless.

Deliberately simplified: the watermark array is 512 deep with a single write port and a read-modify-write on management access, which races with an occupancy update to the same queue in the same cycle — a real implementation arbitrates, and the loss is at most one sample. The histograms have no overflow handling: at 24.5 M frames per second a 32-bit bucket wraps in 175 seconds if every frame lands in it, which is Chapter 19.7 §14's wrapping problem arriving in a new structure, and the honest fix is the same one — document them as differencing counters and read them faster than they wrap.

Production implication: overload_share_ppm is the number that makes every other latency figure interpretable, and it is one divide. A p99 quoted alongside "overload share 0 ppm" is a steady-state p99 and Section 6's table applies; a p99 quoted alongside "overload share 40 000 ppm" is a mixture and the right response is to read the two histograms separately. Without the share, a latency regression between two software releases is uninterpretable, because the traffic may simply have become burstier — and that is the most common false alarm in switch performance work.


12. The Window Is the Resolution

A percentile is an order statistic and needs members above it. In a window of W samples the highest percentile with even one expected member above it is one minus one over W. A window of one thousand and twenty four therefore tops out at ninety nine point nine zero two per cent, so a wire labelled ninety nine point nine nine actually carries the maximum of that window, which at ninety five per cent offered load is five point six one five microseconds against a true value of seven point four seven five, understated by twenty four point eight eight per cent. A window of two hundred and fifty six is worse and cheaper, returning four point four eight four microseconds, so the reported number improves as the instrument shrinks. Storing enough samples to support a genuine ninety nine point nine ninth percentile needs a hundred thousand and one of them, which across sixty four ports at sixteen bits is one hundred and two million bits, three hundred and sixty one point four media access control datapaths, eighteen point two two one per cent of the switch. A thirty two bucket logarithmically spaced histogram supports the same percentile for sixty five thousand five hundred and thirty six bits, which is nought point two three of a datapath, because it stores counts rather than samples and counts have no depth limit.An orderstatisticneeds members aboveitWindow cap = 1 −1/W1 024 → p99.902W = 2564.484 µs — 40.02% lowW = 1 0245.615 µs — 24.88% lowTrue p99.997.475 µsA 100 001-deepFIFO361.4 dp — 18.221%A 32-buckethistogram0.23 dp — 0.0117%Cheaper readsbetterthe incentive runsbackwards12
Figure 4 — a sample store's percentile is capped by its depth; a histogram's is capped by how long you wait.

Section 7's module refuses to report a percentile its window cannot support. This section is the arithmetic behind the refusal, and it is the sharpest result in the chapter.

A percentile is an order statistic, and an order statistic needs members above it. In a window of W samples, the highest percentile with even one expected member above it is 1 − 1/W. Ask for anything finer and the estimator has nothing to return but the window's maximum.

WindowHighest percentile with one expected sampleWhat it returns at ρ = 0.95Against a true p99.99 of 7.475 µs
25699.609%4.484 µs40.02% low
1 02499.902%5.615 µs24.88% low
4 09699.976%6.747 µs9.75% low
16 38499.994%7.878 µs5.39% HIGH
65 53699.998%9.010 µs20.53% high

Rows one to three are the failure and rows four and five are the other failure, and they are not symmetric.

Below the crossover, the estimator returns the window maximum and calls it whatever percentile was asked for. The number is biased low, it is plausible, and it gets better-looking as the window gets smaller — which is exactly backwards, because a smaller window is a cheaper design. A 256-sample window reports 4.484 µs for a p99.99 that is really 7.475, and 4.484 µs is a very satisfactory number to put in a datasheet.

Above the crossover the estimator is returning a real order statistic of a longer window, so it is high relative to p99.99 simply because it is a different, finer percentile — which is a labelling error rather than a measurement error, and it is caught by reading the window length.

Below the crossover the number is wrong and looks right. Above it the number is right and is labelled wrong. Only the window length distinguishes the two, and it is the field that never travels.

The other direction: what window does a percentile need? The convention this chapter uses is ten expected exceedances, which gives an estimate accurate to roughly ±30%.

PercentileMinimum windowTraffic at 24.505 Mfps
p991 00040.8 µs
p99.910 000408 µs
p99.99100 0014.081 ms
p99.9991 000 00140.808 ms

Four milliseconds of traffic for a p99.99 — which is nothing. So the constraint is not the traffic; it is the storage. A 100 001-sample window on 64 ports at 16 bits is 1.024 × 10⁸ BCE — 361.4 datapaths, 18.221% of the switch — which is why no design carries one and why the window is 1 024.

And that is the resolution of the whole section: the FIFO is the wrong instrument.

InstrumentBCEDatapathsCan it report a p99.99?
1 024-sample FIFO1 048 5763.70NO — it returns a p99.902, 24.88% low
100 001-sample FIFO1.024 × 10⁸361.4yes, at 18.221% of the die
32-bucket log histogram65 5360.23yes, to one octave, at 0.0117%

A histogram does not store samples; it stores counts, and a count has no window limit. Sixteen times cheaper than the FIFO that cannot answer the question, fifteen hundred times cheaper than the FIFO that can — and its only cost is resolution within a bucket, which interpolation recovers most of.

So Section 10's largest line item is also its most questionable, and this chapter's recommendation is to delete it: keep the histogram, keep the watermark, drop the window FIFO, and the observability bill falls from 5.10 datapaths to 1.40 — 0.0703% of the switch.


13. RTL 6 — The Window Auditor

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// window_auditor -- given a latency claim, decide whether the window
// it was measured over can support the percentile it names, and say
// which percentile it can support instead.
//
// This is the module that would have stopped a 1024-sample maximum from
// being published as a p99.99 for the life of a product.
// ---------------------------------------------------------------------
module window_auditor
  import perfreview_pkg::*;
#(
  // Expected exceedances required for an estimate to be supported.
  parameter int unsigned EXCEEDANCES = 10
)(
  input  logic             clk,
  input  logic             rst_n,

  input  logic             claim_valid,
  input  latency_claim_t   claim,
  input  logic             from_histogram,   // counts, not samples

  output logic             out_valid,
  output logic             supported,
  output logic [19:0]      supportable_ppm,
  output logic [31:0]      window_required,
  output logic [31:0]      window_shortfall,
  output logic [15:0]      n_claims,
  output logic [15:0]      n_unsupported,
  output logic [15:0]      n_histogram_claims,
  output logic             any_unsupported_claim
);

  logic [31:0] need;
  logic [19:0] best;

  always_comb begin
    // The window a claimed percentile needs.
    need = (claim.percentile_ppm >= 20'd1000000) ? 32'hFFFF_FFFF
         : 32'((EXCEEDANCES * 1000000) /
               (1000000 - int'(claim.percentile_ppm)));

    // The finest percentile this window supports. A histogram is not
    // window-limited in the same way: it accumulates counts, so its
    // limit is the total sample count rather than a stored depth.
    best = (claim.window_samples == 0) ? 20'd0
         : 20'(1000000 - (EXCEEDANCES * 1000000)
                         / int'(claim.window_samples));
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      out_valid          <= 1'b0;
      supported          <= 1'b0;
      supportable_ppm    <= '0;
      window_required    <= '0;
      window_shortfall   <= '0;
      n_claims           <= '0;
      n_unsupported      <= '0;
      n_histogram_claims <= '0;
    end else begin
      out_valid <= claim_valid;
      if (claim_valid) begin
        n_claims        <= n_claims + 16'd1;
        window_required <= need;
        supportable_ppm <= best;

        if (from_histogram)
          n_histogram_claims <= n_histogram_claims + 16'd1;

        if (claim.window_samples >= need) begin
          supported        <= 1'b1;
          window_shortfall <= '0;
        end else begin
          supported        <= 1'b0;
          window_shortfall <= need - claim.window_samples;
          n_unsupported    <= n_unsupported + 16'd1;
        end
      end
    end
  end

  always_comb begin
    any_unsupported_claim = (n_unsupported != 16'd0);
  end

endmodule

Classification, Model or Production? A REVIEW INSTRUMENT, and one that belongs in the management interface rather than in a spreadsheet.

What it teaches: that supportable_ppm is more useful than supported. A boolean says the claim is bad; the supportable percentile says what the claim should have said — a 1 024-sample window supports p99.902, so the honest label for its number is p99.902 and the number itself is fine. The module turns a wrong measurement into a correct one by changing its name, which is the cheapest repair available anywhere in this chapter.

Deliberately simplified: from_histogram is a flag where the real distinction is structural — a histogram's limit is its total count and a FIFO's is its depth, so the two need different best formulas and this module uses one. EXCEEDANCES = 10 is also a convention rather than a derivation: the right value depends on the acceptable variance in the estimate, and a design specifying its latency at p99.99 with a ±10% tolerance needs closer to a hundred. And the integer divides truncate, which makes supportable_ppm pessimistic by up to one part per million — the safe direction.

Production implication: put this check in the datasheet generation flow, not only in the management interface. Every latency figure that reaches a customer should have passed through a comparison between its percentile and its window, and the comparison is two divides. The failure it prevents is the most durable kind in this chapter: a number that is wrong by 24.88%, plausible, published, quoted back by customers, and defended internally for years — because nothing in the flow ever recorded the window it came from.


14. What a Performance Review Must Never Do

Six prohibitions, and each one has a named example earlier in this track.

#NeverBecause
1quote a latency without a percentile and a loadSection 6 — one budget, four answers, spanning 9.19×
2quote a percentile without its windowSection 12 — a 1 024-sample p99.99 is 24.88% low and plausible
3treat a bound as a tailSection 8 — P(W > 2 224 µs) at ρ = 0.99 is 2 × 10⁻²³⁷
4compute a percentile across an incastSection 9 — a mixture of two distributions is a percentile of neither
5leave an environmental threshold at a defaultSection 4 — four defaults, four deployments, four failures
6set an alarm without checking the metric can reach itChapter 25.5 §8 — 90% on a ceiling of 80.5%

Prohibition 6 is the one that needs no measurement at all, which makes it the cheapest item in Section 2 and the one most often wrong. The check is one comparison: the alarm's threshold against the metric's maximum achievable value, and Chapter 25.5 §8's four-way table shows the two can differ by nearly ten points with nothing in the flow noticing.

Prohibition 3 deserves its own sentence because it is subtle and respectable. 2 224 µs is a real number, derived correctly in Chapter 23.3 §9, and it is the right answer to "how bad can it get". It is the wrong answer to "what is the p99.99", and the two questions sound the same in a meeting. The distinguishing test is whether the regime has a steady state: if the offered load is below one, the bound is unreachable; if it is at or above one, there is no percentile to ask for.


15. RTL 7 — Performance Telemetry

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// perf_telemetry -- the numbers a performance review reads at any
// moment, and six flags that each name a sentence somebody must write.
// ---------------------------------------------------------------------
module perf_telemetry
  import perfreview_pkg::*;
(
  input  logic          clk,
  input  logic          rst_n,
  input  logic          clear,

  // From the deferral register, Section 5.
  input  logic [3:0]    n_bound,
  input  logic [3:0]    n_deferred,
  input  logic [3:0]    n_defaulted,
  input  logic [3:0]    n_silicon_gated,

  // From the percentile estimator and the auditor, Sections 7 and 13.
  input  logic          claim_valid,
  input  logic          claim_supported,
  input  latency_claim_t claim,

  // From the observability bank, Section 11.
  input  logic [31:0]   n_steady,
  input  logic [31:0]   n_overload,
  input  logic [31:0]   watermark_peak_cells,

  // Alarm configuration, for prohibition 6.
  input  logic [19:0]   alarm_threshold_ppm,
  input  logic [19:0]   metric_ceiling_ppm,

  output logic [19:0]   thresholds_disposed_ppm,
  output logic [19:0]   overload_share_ppm,
  output logic [19:0]   buffer_peak_share_ppm,
  output logic [15:0]   c_unsupported_claims,
  output logic [15:0]   c_claims,
  output logic          v_threshold_defaulted,
  output logic          v_claim_without_window,
  output logic          v_claim_unsupported,
  output logic          v_percentile_over_mixture,
  output logic          v_alarm_unreachable,
  output logic          v_awaiting_silicon
);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin
      c_unsupported_claims <= '0;
      c_claims             <= '0;
    end else if (claim_valid) begin
      c_claims <= c_claims + 16'd1;
      if (!claim_supported)
        c_unsupported_claims <= c_unsupported_claims + 16'd1;
    end
  end

  always_comb begin
    thresholds_disposed_ppm =
      20'((int'(N_THRESH - n_defaulted) * 1000000) / N_THRESH);

    overload_share_ppm = ((n_steady + n_overload) == 0) ? 20'd0
                       : 20'((longint'(n_overload) * 1000000)
                             / longint'(n_steady + n_overload));

    buffer_peak_share_ppm =
      20'((longint'(watermark_peak_cells) * 1000000)
          / longint'(MAX_CELLS_ONE_PORT));

    // Six flags, each naming a specific prohibition of Section 14.
    v_threshold_defaulted     = (n_defaulted != 4'd0);
    v_claim_without_window    = claim_valid && (claim.window_samples == 0);
    v_claim_unsupported       = (c_unsupported_claims != 16'd0);
    // A percentile is being quoted over an interval that contained an
    // overload event. Prohibition 4.
    v_percentile_over_mixture = claim_valid && (n_overload != 32'd0)
                                            && (n_steady   != 32'd0);
    // Prohibition 6, and it needs no measurement -- just a comparison.
    v_alarm_unreachable       = (alarm_threshold_ppm > metric_ceiling_ppm);
    v_awaiting_silicon        = (n_silicon_gated != 4'd0);
  end

endmodule

Classification, Model or Production? PRODUCTION-INTENT. Every input here exists in a modern switch; the outputs are the comparisons nobody wires up.

What it teaches: that v_alarm_unreachable costs one comparator and catches a whole class of failure. Chapter 25.5 §8's occupancy alarm at 90% on a metric that tops out at 80.5% is a check that can never fire, and the design already knows both numbers — the alarm is a register and the ceiling follows from the table's associativity. One comparison at configuration time, and the failure becomes a configuration error rather than a silence lasting the product's life.

Deliberately simplified: metric_ceiling_ppm is an input where a real design derives it from the table's geometry and its load factor, so a change to the associativity silently invalidates the ceiling unless the derivation is in the same place. v_percentile_over_mixture also fires whenever both histograms are non-empty, which is almost always on a busy fabric — the useful form thresholds the overload share, and the threshold is itself a judgement. And buffer_peak_share_ppm uses the single-port maximum as its denominator, which Section 9 already noted is not a constant.

Production implication: the flag to act on first is v_threshold_defaulted, because it is the only one that describes a decision rather than a measurement. It is high exactly when some parameter of Section 4's seven has a value nobody chose, and it is readable from the field — which means a support engineer can distinguish "this deployment is configured wrongly" from "this deployment is configured by nobody" in one register read, and those two conditions have completely different remedies.


16. RTL 8 — The Performance Conformance Monitor

Seven checks. A design that passes all seven has performance figures somebody can act on; one that fails any of them has figures that will be misquoted.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ---------------------------------------------------------------------
// perf_conformance -- the sign-off gate for Section 2's twenty items.
//
// Same design rule as Chapter 26.2 Section 16: every check must be able
// to fail on a design somebody would really submit, and the gate says
// so about itself.
// ---------------------------------------------------------------------
module perf_conformance
  import perfreview_pkg::*;
(
  input  logic          clk,
  input  logic          rst_n,
  input  logic          evaluate,

  // Section 5.
  input  logic [3:0]    n_defaulted,
  input  logic [3:0]    n_silicon_gated,
  input  logic          silicon_items_documented,

  // Sections 6, 7 and 13.
  input  logic          latency_has_percentile,
  input  logic          latency_has_load,
  input  logic          latency_has_window,
  input  logic [15:0]   n_unsupported_claims,

  // Sections 8 and 9.
  input  logic          overload_regime_reported,
  input  logic [19:0]   overload_share_ppm,

  // Section 14, prohibition 6.
  input  logic [19:0]   alarm_threshold_ppm,
  input  logic [19:0]   metric_ceiling_ppm,

  // Section 10.
  input  logic [31:0]   observability_bce,
  input  logic          observability_budgeted,

  output logic          c1_thresholds_disposed,
  output logic          c2_latency_complete,
  output logic          c3_claims_supported,
  output logic          c4_regime_separated,
  output logic          c5_alarm_reachable,
  output logic          c6_observability_budgeted,
  output logic          c7_silicon_items_named,
  output logic [2:0]    n_failures,
  output logic          conformant,
  output logic          gate_can_fail
);

  logic [2:0] fails;

  always_comb begin
    // 1. No threshold sits at a value nobody chose.
    c1_thresholds_disposed = (n_defaulted == 4'd0);

    // 2. Every latency figure carries a percentile, a load and a
    //    window. Section 1's rule three, as a conjunction.
    c2_latency_complete    = latency_has_percentile &&
                             latency_has_load &&
                             latency_has_window;

    // 3. No published claim exceeds what its window supports.
    c3_claims_supported    = (n_unsupported_claims == 16'd0);

    // 4. The two regimes are reported separately, or the share is zero.
    c4_regime_separated    = overload_regime_reported ||
                             (overload_share_ppm == 20'd0);

    // 5. Every alarm threshold is inside its metric's range.
    c5_alarm_reachable     = (alarm_threshold_ppm <= metric_ceiling_ppm);

    // 6. The instruments are in the area budget rather than added
    //    during implementation. Section 10 is 0.2569 per cent.
    c6_observability_budgeted = observability_budgeted &&
                                (observability_bce != 32'd0);

    // 7. Items that cannot be closed before silicon are NAMED rather
    //    than guessed. Section 17.
    c7_silicon_items_named = (n_silicon_gated == 4'd0) ||
                             silicon_items_documented;

    fails = 3'(!c1_thresholds_disposed)    + 3'(!c2_latency_complete)
          + 3'(!c3_claims_supported)       + 3'(!c4_regime_separated)
          + 3'(!c5_alarm_reachable)        + 3'(!c6_observability_budgeted)
          + 3'(!c7_silicon_items_named);

    // A design that supplies no latency claims and no thresholds would
    // pass every check above. It is not conformant; it is empty.
    gate_can_fail = latency_has_percentile || (n_defaulted != 4'd0) ||
                    (observability_bce != 32'd0);
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_failures <= '0;
      conformant <= 1'b0;
    end else if (evaluate) begin
      n_failures <= fails;
      conformant <= (fails == 3'd0) && gate_can_fail;
    end
  end

endmodule

Classification, Model or Production? A SIGN-OFF GATE.

What it teaches: that check 4's or is the right shape and an and would be wrong. A fabric that never enters overload has nothing to separate, so requiring two histograms on a design that legitimately has one distribution would push teams to fabricate the second. The check asks for separation only when the mixture exists, which is the general form of a gate that does not punish a correct design for being simple. And gate_can_fail repeats Chapter 26.2 §16's defence for the same reason: every check here is a conjunction over supplied quantities, and a design supplying none of them passes all seven.

Deliberately simplified: latency_has_percentile, has_load and has_window are three bits where a real gate parses the performance report and confirms each figure carries all three — which is a document check rather than a hardware one, and the honest implementation runs it in the datasheet flow. metric_ceiling_ppm is a single value for a single alarm where a real design has a dozen alarms with a dozen ceilings, so the check generalises to a loop. And observability_budgeted is a bit that somebody sets, which is exactly the kind of input Chapter 26.1 §10 calls self-witnessed.

Production implication: run checks 1, 5 and 7 at configuration time, not at sign-off. All three are comparisons over constants the design already holds: a threshold's state, an alarm against a ceiling, and whether the silicon-gated items have documentation. They cost three comparators and they convert three of this chapter's failure modes from field escapes into build errors. Checks 2, 3 and 4 need a report, and check 6 needs a floorplan — so the gate naturally splits into a build-time half and a review-time half, and the build-time half is the one that catches the failures with the longest tail.


17. The One Item Only Silicon Can Fill

Six of Section 4's seven thresholds close with a conversation, a query or a document. One does not, and it is worth being precise about why rather than treating it as a harder version of the same problem.

The pause watermark's threshold is the link's dead time, and Chapter 14.2 §9 decomposes it into terms that come from three different places.

TermWhere the number comes fromKnown before silicon?
the local PHY's transmit latencythe PHY's datasheetas a specification, with a range
propagation, both directionsthe cable plant's lengthas a range the product supports
the partner's receive-to-pause-response latencya device somebody else madeNO
the local receive path to the pause generatorthis design's RTLyes

Row three is the reason this threshold is different. It is not a number nobody has measured; it is a number that belongs to a device that is not part of this product, and it varies across every partner the product will ever be connected to. No conversation closes it, because there is no single owner to have the conversation with.

And the consequence, from Chapter 14.2 §9's table, is large.

LinkDead timeHeadroomWatermark on a 4 096-cell queue
1 Gb/s, 100 m13.32 µs13 cells4 083 — 99.7%
10 Gb/s, 100 m1.78 µs18 cells4 078 — 99.6%
100 Gb/s, 100 m0.63 µs62 cells4 034 — 98.5%
10 Gb/s, 2 km fibre11.2 µs110 cells3 986 — 97.3%
100 Gb/s, 2 km fibre10.6 µs1 058 cells3 038 — 74.2%

The watermark spans 99.7% to 74.2% across one product's supported range, and the intuitive default of 75% is safe by a factor of seventy-nine on row one and 34 cells short on row five. On row five the mechanism fails: every pause is issued too late, the queue overflows during the dead time, and the drops look exactly like the drops PAUSE was deployed to prevent.

The honest review entry for this item is not a value. It is: "the watermark is computed from a measured dead time; the register is at address X; the measurement procedure is Y; the default is Z and is correct only for links shorter than L."

Which is a different kind of review output and worth naming as such: an item closed by a PROCEDURE rather than by a number. The review cannot supply the value and can supply everything else — the register, the formula, the measurement method, and the boundary of the default's validity — and a design that ships those four things has deferred correctly in the strongest available sense.


18. What the Review Assumes

Six assumptions, stated because a reader on a different fabric needs to know which numbers survive the journey.

#AssumptionIf it is false
1each hop is M/M/1 with the mean frame of Chapter 26.2 §8Section 6's whole table moves; the shape of the argument does not
2the five hops are independentcorrelated queues make the five-hop tail heavier than the sum
3arrivals are Poissonreal traffic is burstier, so every percentile here is optimistic
4the single-port buffer share is a constant 27.8 MBa dynamic threshold shrinks it when other ports queue
5ten expected exceedances is enough for a percentilea tighter tolerance needs a larger window and Section 12's costs rise
6Chapter 14.2 §9's dead-time decomposition is completea fourth term would move Section 17's watermark further

Assumptions 2 and 3 both push the same way and it is worth saying so plainly: this chapter's percentiles are optimistic.

Assumption 2 first. Five hops treated as independent queues means the five-hop tail is the sum of five independent tails. In a real fabric the hops share traffic — a burst that queues at hop one arrives at hop two as a burst — so the correlation is positive and the true tail is heavier than Section 6's table. How much heavier depends on the topology and the hashing, which is Chapter 23.1's subject rather than this one's.

Assumption 3 is larger. Poisson arrivals are the most benign arrival process with a given mean; real data-centre traffic is bursty at every timescale, and the standard correction is a factor of several on the tail. So 41.941 µs at ρ = 0.95 and p99.99 is a floor on the tail rather than an estimate of it — which means the direction of every conclusion in Sections 6 and 14 is safe and the magnitudes are understated.

Assumption 4 matters for Section 8 rather than Section 6. Chapter 23.3 §9's 108 616 cells is what one port can claim when the rest of the switch is idle. During an incast the other ports are usually not idle, so the real cap is lower and the fill time is shorter than Section 8's table — the asymmetry of 63:1 holds, and both of its terms shrink together.

Assumption 1 is the one a reader is most likely to want to substitute, and the substitution is cheap: the model enters only through μ = 2.4505 × 10⁷ frames per second, so a different frame-size distribution rescales every queueing term by the ratio of the means and leaves the structural floor alone.

Assumption 6 cannot be discharged from inside the chapter, exactly as Chapter 26.2 §18's assumption 6 could not. A fourth dead-time term would make Section 17's 1 058-cell headroom larger and the 75% default worse, so the direction is known and the magnitude is not — and the review's honest form is to name the three terms it has and say the decomposition is the one Chapter 14.2 §9 derived.


19. The Cost, Accounted — in BCE and Percentiles

This chapter prices two different kinds of thing and they do not share a unit. The instruments are silicon and go in BCE. The latency figures are not costs at all — they are measurements, and each one carries a percentile, a load and a window, which is the triple Section 1 established.

The silicon, at 64 ports, against Chapter 23.3's 5.62 × 10⁸ BCE.

StructureBCE× a MAC receive datapath% of the switch
ingress timestamp, 48 b × 36 stages110 5920.390.0197%
latency histogram, steady65 5360.230.0117%
latency histogram, overload65 5360.230.0117%
per-queue maximum watermark16 3840.060.0029%
per-set refusal counter131 0720.460.0233%
Chapter 19.7's counters at true widths63 4880.220.0113%
dead-time and flap timers8 1920.030.0015%
the seven threshold registers2240.0010.00004%
RECOMMENDED TOTAL461 0241.630.0820%
the 1 024-sample window FIFO, NOT recommended1 048 5763.700.1866%
total if the FIFO is kept1 509 6005.330.2686%

The complete measurement apparatus for a 6.4 Tb/s switch — every instrument this chapter argues for — is 1.63 MAC receive datapaths and 0.0820% of the die.

And the reason the recommended total is smaller than Section 10's is Section 12's finding. The window FIFO is 72.6% of Section 10's bill and it is the only instrument in the list that cannot answer the question it was added for — a 1 024-sample window supports p99.902, not p99.99, and reports 5.615 µs against a true 7.475, understated by 24.88%. The histogram answers the same question for 1/16 of the area, and the second histogram for the overload regime costs less than the FIFO's rounding error.

The measurements, which are not costs and are reported as triples.

FigureValuePercentileWindowLoad
five-hop structural floor4.564 µsidle
five-hop median7.184 µsp50≥ 20ρ = 0.95
five-hop tail23.148 µsp99≥ 1 000ρ = 0.95
five-hop deep tail41.941 µsp99.99≥ 100 001ρ = 0.95
incast peak, one port2 224 µsρ ≥ 1, 64 senders
incast fill time35.302 µsρ ≥ 1, 64 senders

Rows one, five and six have no percentile and that is correct, not an omission: a structural floor is a deterministic sum and an incast is an event. Only the middle three are order statistics, and each one names the window it needs.


20. Properties Worth Asserting, and One Worth Refusing

Fifty-two properties in six groups, over the thresholds, the percentiles, the regimes and the instruments.

Group A — the threshold model, Section 3.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// 1. A threshold with an answer is BOUND, whatever else is true.
p_answered_is_bound: assert property (@(posedge clk) disable iff (!rst_n)
  (out_valid && $past(owner_answered) && $past(has_value))
    |-> (state == TH_BOUND));

// 2. Naming an owner without exposing the parameter does not defer it.
p_named_not_exposed: assert property (@(posedge clk) disable iff (!rst_n)
  (out_valid && $past(owner_named) && !$past(is_exposed) &&
   $past(has_value) && !$past(owner_answered))
    |-> (state == TH_DEFAULT && became_a_default));

// 3. Correct deferral needs both, and is exactly both.
p_deferred_needs_both: assert property (@(posedge clk) disable iff (!rst_n)
  (out_valid && deferred_correctly) |-> (state == TH_DEFERRED));

// 4. No value at all is UNSET, which is honest and rare.
p_no_value_unset: assert property (@(posedge clk) disable iff (!rst_n)
  (out_valid && !$past(has_value) && !$past(owner_named))
    |-> (state == TH_UNSET));

// 5. became_a_default and deferred_correctly are mutually exclusive.
p_default_xor_deferred: assert property (@(posedge clk) disable iff (!rst_n)
  out_valid |-> !(became_a_default && deferred_correctly));

// 6. A threshold needing a measurement is owned by silicon, whatever
//    its identity. Section 17.
p_measurement_is_silicon: assert property (@(posedge clk) disable iff (!rst_n)
  (out_valid && $past(needs_measurement)) |-> (owner == OWN_SILICON));

// 7. The replay window is the one structural member of the seven.
p_replay_structural: assert property (@(posedge clk) disable iff (!rst_n)
  (out_valid && $past(th_id) == TH_REPLAY && !$past(needs_measurement))
    |-> (owner == OWN_STRUCTURAL));

c_default_seen: cover property (@(posedge clk) out_valid && became_a_default);
c_silicon_seen: cover property (@(posedge clk) out_valid && owner == OWN_SILICON);

Group B — the deferral register, Section 5.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// 8. A write only lands if the parameter is exposed. Exposure and
//    accountability are one mechanism, not two.
p_write_needs_exposure: assert property (@(posedge clk) disable iff (!rst_n)
  (wr_en && !exposed[wr_id]) |-> ##1 (value[$past(wr_id)] ==
                                      $past(value[wr_id])));

// 9. And a landed write marks the threshold answered.
p_write_answers: assert property (@(posedge clk) disable iff (!rst_n)
  (wr_en && exposed[wr_id]) |-> ##1 answered[$past(wr_id)]);

// 10. The four counts partition the seven exactly.
p_threshold_partition: assert property (@(posedge clk) disable iff (!rst_n)
  (n_bound + n_deferred + n_defaulted + n_silicon_gated == 4'(N_THRESH)));

// 11. Disposed means no defaults, and nothing else.
p_disposed_definition: assert property (@(posedge clk) disable iff (!rst_n)
  all_thresholds_disposed == (n_defaulted == 4'd0));

// 12. A silicon-gated threshold is never counted bound until answered.
p_silicon_not_bound: assert property (@(posedge clk) disable iff (!rst_n)
  (silicon_gated[0] && !answered[0]) |-> (n_silicon_gated != 4'd0));

// 13. Reset leaves every threshold at its default, which is the state
//     the chapter exists to eliminate -- so it must be visible.
p_reset_is_default: assert property (@(posedge clk)
  $rose(rst_n) |-> ##1 (n_defaulted != 4'd0 || n_silicon_gated != 4'd0));

c_all_disposed: cover property (@(posedge clk) all_thresholds_disposed);

Group C — the percentile estimator, Section 7.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// 14. A claim carries all three fields or it is not a claim.
p_claim_complete: assert property (@(posedge clk) disable iff (!rst_n)
  (claim.value_ns != 32'd0) |-> (claim.window_samples != 32'd0));

// 15. supported is exactly the window comparison.
p_supported_definition: assert property (@(posedge clk) disable iff (!rst_n)
  (samples_seen != 0) |->
    (estimate_is_supported == (samples_seen >= window_required)));

// 16. The finest supportable percentile rises with the window and
//     never exceeds the requested one when unsupported.
p_best_below_want: assert property (@(posedge clk) disable iff (!rst_n)
  (!estimate_is_supported && samples_seen != 0)
    |-> (best_supported_ppm < want_ppm));

// 17. The estimate never exceeds the observed maximum.
p_estimate_le_max: assert property (@(posedge clk) disable iff (!rst_n)
  (samples_seen != 0) |-> (estimate_ns <= observed_max_ns ||
                           estimate_ns == 32'(32'd1 << BUCKET_SHIFT)));

// 18. An empty window supports nothing.
p_empty_unsupported: assert property (@(posedge clk) disable iff (!rst_n)
  (samples_seen == 32'd0) |-> !estimate_is_supported);

// 19. A finer percentile never needs a smaller window.
p_window_monotone: assert property (@(posedge clk) disable iff (!rst_n)
  (want_ppm > $past(want_ppm)) |-> (window_required >= $past(window_required)));

// 20. The claim's percentile is what was asked for, never what was
//     achievable -- the module reports the gap rather than hiding it.
p_claim_reports_request: assert property (@(posedge clk) disable iff (!rst_n)
  claim.percentile_ppm == want_ppm);

c_unsupported_claim: cover property (@(posedge clk)
  samples_seen != 0 && !estimate_is_supported);
c_supported_claim: cover property (@(posedge clk) estimate_is_supported);

Group D — the incast model, Section 9.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// 21. One sender is never an overload. rho = 1 exactly builds no queue.
p_one_sender_no_overload: assert property (@(posedge clk) disable iff (!rst_n)
  (n_senders == 8'd1) |-> !regime_is_overload);

// 22. Fill rate is (N-1) line rates, exactly.
p_fill_rate: assert property (@(posedge clk) disable iff (!rst_n)
  regime_is_overload |-> (fill_rate_moct_s ==
                          32'((int'(n_senders) - 1) * int'(LINE_MOCT_S))));

// 23. Drain time does not depend on N. This is Section 8's fixed row.
p_drain_invariant: assert property (@(posedge clk) disable iff (!rst_n)
  (drain_time_us == 32'(CAP_OCTETS / LINE_MOCT_S)));

// 24. So the asymmetry is N-1.
p_asymmetry_is_n_minus_1: assert property (@(posedge clk) disable iff (!rst_n)
  (regime_is_overload && fill_time_us != 32'd0)
    |-> (asymmetry <= 32'(int'(n_senders))));

// 25. Fill time falls as senders rise.
p_fill_monotone: assert property (@(posedge clk) disable iff (!rst_n)
  (n_senders > $past(n_senders) && $past(n_senders) > 8'd1)
    |-> (fill_time_us <= $past(fill_time_us)));

// 26. Peak queue never exceeds the cap.
p_peak_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  (peak_queue_octets <= 32'(CAP_OCTETS)));

// 27. Drops occur exactly when the offer exceeds the cap.
p_drop_definition: assert property (@(posedge clk) disable iff (!rst_n)
  buffer_overflows == ((64'(n_senders) * 64'(burst_octets_per_sender))
                       > 64'(CAP_OCTETS)));

c_incast_64: cover property (@(posedge clk) n_senders == 8'd64);
c_overflow: cover property (@(posedge clk) buffer_overflows);

Group E — the observability bank and the window auditor, Sections 11 and 13.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// 28. A sample lands in exactly one histogram.
p_one_histogram: assert property (@(posedge clk) disable iff (!rst_n)
  (lat_valid && in_overload) |-> ##1 (n_overload == $past(n_overload) + 32'd1)
                                  && (n_steady   == $past(n_steady)));

// 29. And the other way.
p_steady_histogram: assert property (@(posedge clk) disable iff (!rst_n)
  (lat_valid && !in_overload) |-> ##1 (n_steady == $past(n_steady) + 32'd1));

// 30. The watermark never falls except on a read.
p_watermark_monotone: assert property (@(posedge clk) disable iff (!rst_n)
  (!rd_en) |-> ##1 (rd_watermark == $past(rd_watermark)));

// 31. A read clears, so consecutive reads are disjoint intervals.
p_read_clears: assert property (@(posedge clk) disable iff (!rst_n)
  rd_en |-> ##2 (rd_en && rd_queue == $past(rd_queue, 2))
                 |-> (rd_watermark <= $past(rd_watermark, 2)));

// 32. Overload share is bounded.
p_share_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  (overload_share_ppm <= 20'd1000000));

// 33. The auditor's shortfall is zero exactly when supported.
p_shortfall_zero: assert property (@(posedge clk) disable iff (!rst_n)
  (out_valid) |-> (supported == (window_shortfall == 32'd0)));

// 34. And an unsupported claim increments the counter, every time.
p_unsupported_counted: assert property (@(posedge clk) disable iff (!rst_n)
  (claim_valid && claim.window_samples < need)
    |-> ##1 (n_unsupported == $past(n_unsupported) + 16'd1));

// 35. supportable_ppm is below a million, always -- no window supports
//     the hundredth percentile, which is the maximum and not a
//     percentile at all.
p_supportable_below_max: assert property (@(posedge clk) disable iff (!rst_n)
  (out_valid && claim.window_samples != 32'd0)
    |-> (supportable_ppm < 20'd1000000));

c_histogram_claim: cover property (@(posedge clk) claim_valid && from_histogram);
c_both_regimes: cover property (@(posedge clk)
  n_steady != 32'd0 && n_overload != 32'd0);

Group F — telemetry and the gate, Sections 15 and 16.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// 36. The defaulted flag is exactly the count being non-zero.
p_defaulted_flag: assert property (@(posedge clk) disable iff (!rst_n)
  v_threshold_defaulted == (n_defaulted != 4'd0));

// 37. An alarm above its metric's ceiling is flagged. One comparator,
//     and it needs no measurement at all.
p_alarm_flag: assert property (@(posedge clk) disable iff (!rst_n)
  v_alarm_unreachable == (alarm_threshold_ppm > metric_ceiling_ppm));

// 38. Chapter 25.5 Section 8's case, concretely: 90 per cent on a
//     ceiling of 80.5 fires the flag.
p_alarm_805: assert property (@(posedge clk) disable iff (!rst_n)
  (alarm_threshold_ppm == 20'd900000 && metric_ceiling_ppm == 20'd805000)
    |-> v_alarm_unreachable);

// 39. A claim with no window is flagged before it is audited.
p_no_window_flag: assert property (@(posedge clk) disable iff (!rst_n)
  (claim_valid && claim.window_samples == 32'd0) |-> v_claim_without_window);

// 40. The mixture flag needs both regimes present.
p_mixture_needs_both: assert property (@(posedge clk) disable iff (!rst_n)
  v_percentile_over_mixture |-> (n_steady != 32'd0 && n_overload != 32'd0));

// 41. Disposed share is a proper fraction of seven.
p_disposed_share: assert property (@(posedge clk) disable iff (!rst_n)
  (thresholds_disposed_ppm <= 20'd1000000));

// 42. Gate check 1 fails on any default.
p_c1_fails_on_default: assert property (@(posedge clk) disable iff (!rst_n)
  (evaluate && n_defaulted != 4'd0) |-> ##1 !c1_thresholds_disposed);

// 43. Check 2 is a conjunction of all three fields.
p_c2_conjunction: assert property (@(posedge clk) disable iff (!rst_n)
  (evaluate && !latency_has_window) |-> ##1 !c2_latency_complete);

// 44. Check 4 passes when there is no mixture to separate.
p_c4_no_mixture: assert property (@(posedge clk) disable iff (!rst_n)
  (evaluate && overload_share_ppm == 20'd0) |-> ##1 c4_regime_separated);

// 45. Check 5 is the same comparator as the telemetry flag, inverted.
p_c5_matches_flag: assert property (@(posedge clk) disable iff (!rst_n)
  (evaluate) |-> ##1 (c5_alarm_reachable != $past(v_alarm_unreachable)));

// 46. Check 7 passes when the silicon-gated items are documented, and
//     when there are none.
p_c7_silicon: assert property (@(posedge clk) disable iff (!rst_n)
  (evaluate && n_silicon_gated == 4'd0) |-> ##1 c7_silicon_items_named);

// 47. An empty submission is never conformant.
p_empty_not_conformant: assert property (@(posedge clk) disable iff (!rst_n)
  (evaluate && !latency_has_percentile && n_defaulted == 4'd0 &&
   observability_bce == 32'd0) |-> ##1 !conformant);

// 48. Conformance requires the gate to be non-trivial.
p_conformant_needs_gate: assert property (@(posedge clk) disable iff (!rst_n)
  conformant |-> $past(gate_can_fail));

// 49. And the verdict holds between evaluations.
p_verdict_stable: assert property (@(posedge clk) disable iff (!rst_n)
  !evaluate |-> ##1 (conformant == $past(conformant)));

c_gate_conformant: cover property (@(posedge clk) evaluate && conformant);
c_gate_empty: cover property (@(posedge clk)
  evaluate && observability_bce == 32'd0);
c_alarm_unreachable: cover property (@(posedge clk) v_alarm_unreachable);
c_awaiting_silicon: cover property (@(posedge clk) v_awaiting_silicon);

21. Verification Scenarios

Fifty-eight scenarios in six groups, plus one directed test random stimulus will not produce.

Group 1 — thresholds and their states (10).

#ScenarioExpect
1value, owner named, exposed, answeredTH_BOUND
2value, owner named, exposed, unansweredTH_DEFERRED; deferred_correctly
3value, owner named, NOT exposedTH_DEFAULT; became_a_default — naming without exposing
4value, no owner, not exposedTH_DEFAULT
5no value, no ownerTH_UNSET — honest, and rare
6the watermark with needs_measurementOWN_SILICON
7the replay windowOWN_STRUCTURAL
8the buffer, table, storm, ageing, flap thresholdsOWN_EXTERNAL
9a write to an unexposed thresholdthe value does not change
10a write to an exposed onevalue updates; answered high

Group 2 — the deferral register (9).

#ScenarioExpect
11all seven at defaultsn_defaulted = 7; all_thresholds_disposed low
12all seven answeredn_bound = 7; disposed high
13six deferred, one silicon-gatedn_deferred = 6; n_silicon_gated = 1; disposed high
14the same with the watermark defaultedn_defaulted = 1; disposed LOW
15after resetevery threshold at its default; disposed low
16the four countsalways sum to 7
17the watermark answeredmoves from n_silicon_gated to n_bound
18thresholds_disposed_ppm with 2 defaults714 285 ppm
19a support engineer's single register readdistinguishes "misconfigured" from "unconfigured"

Group 3 — percentiles and windows (10).

#ScenarioExpect
20p99 requested, 1 024 samplessupported — 1 000 required
21p99.99 requested, 1 024 samplesNOT supported; shortfall 98 977
22the same, reported anyway5.615 µs — a p99.902 wearing p99.99's name
23the true p99.99 at ρ = 0.957.475 µs — the claim is 24.88% low
24a 256-sample window4.484 µs — 40.02% low, and cheaper
25a 16 384-sample window7.878 µs — 5.39% HIGH, and correctly labelled p99.994
26p99.99 from a 32-bucket histogram at 10⁵ countssupported — counts, not depth
27the same histogram at 10⁴ countsnot supported; hist_p9999_valid low
28best_supported_ppm at 1 024 samples990 235 ppm — p99.0235, at ten exceedances
29an empty windowunsupported; no claim published

Group 4 — the two regimes (10).

#ScenarioExpect
30ρ = 0.5, five hops, p504.564 µs — the structural floor exactly
31ρ = 0.95, p507.184 µs — 1.57× the floor
32ρ = 0.95, p9923.148 µs — 5.07×
33ρ = 0.95, p99.9941.941 µs — 9.19×
34ρ = 0.99, p99.99192.287 µs — 42.1×
35P(W > 2 224 µs) at ρ = 0.992.04 × 10⁻²³⁷ — not a tail
362 senders into one portfill 2 224 µs; drain 2 224 µs; 1:1
3764 sendersfill 35.302 µs; drain 2 224 µs; 63:1
381 senderregime_is_overload low — no queue builds
39a p99 computed across an incastv_percentile_over_mixture

Group 5 — the instruments (10).

#ScenarioExpect
40a latency sample in overloadlands in hist_overload only
41a sample in steady statehist_steady only
42occupancy 1 000 then 400 cellswatermark holds 1 000
43a management readreturns 1 000, then clears to 0
44the next interval peaking at 400returns 400 — a per-interval peak
45a running maximum insteadreturns 1 000 forever — the failure read-and-clear fixes
46a 35.302 µs peak against a 1 s pollinvisible to a sampled occupancy; caught by the watermark
47the full bill at 64 ports1 443 840 BCE; 5.10 datapaths; 0.2569%
48the bill without the window FIFO461 024 BCE; 1.63 datapaths; 0.0820%
49a 100 001-sample FIFO at 64 ports1.024 × 10⁸ BCE — 18.221% of the switch

Group 6 — telemetry and the gate (9).

#ScenarioExpect
50alarm 90%, ceiling 80.5%v_alarm_unreachable; check 5 fails
51alarm 70%, ceiling 80.5%clear
52a latency figure with no percentilecheck 2 fails
53a figure with a percentile and no windowcheck 2 fails; v_claim_without_window
54overload share 0, one histogramcheck 4 PASSES — nothing to separate
55overload share 41 000 ppm, one histogramcheck 4 fails
56one threshold defaultedcheck 1 fails
57a submission with no claims and no thresholdsgate_can_fail low; conformant LOW
58all seven checks clear on a real designconformant high

22. Debugging a Performance Number That Met Its Target

Six symptoms, and all six begin with a figure that passed.

SymptomFirst questionWhere to look
an application missing deadlines on a fabric inside its latency budgetwhich percentile was the budget?Section 6 — the floor and the p99.99 differ by 9.19×
a tail latency that never varies between releaseshow deep is the window?Section 12 — a 1 024-sample window reports its own maximum
a p99 that tripled with no design changewhat was the overload share?Section 21's directed test — 41 000 ppm
drops on the longest links onlyis the pause watermark a default?Section 17 — 75% is 34 cells short at 100 Gb/s over 2 km
an occupancy alarm that has never firedcan the metric reach it?Chapter 25.5 §8 — 90% on a ceiling of 80.5%
a parameter the field cannot changeis it exposed or documented?Section 4 — a PDF is not a register

Row two has a signature worth memorising. A tail-latency figure that is suspiciously stable across releases, loads and traffic profiles is usually not a stable design — it is an estimator pinned at its window's ceiling. The test is one register read: compare the reported percentile against the window length, and if the window cannot support the percentile, the number has been the same since the histogram was wired up.

Row four has a procedure and it takes ten minutes. Read the watermark register; if it is the reset default, read the link's length and rate and compute Chapter 14.2 §9's headroom. On a 100 Gb/s 2 km link the requirement is 1 058 cells of a 4 096-cell queue and the 75% default reserves 1 024 — and the 34-cell shortfall is the entire fault.


23. Misconceptions

Six, in the wrong-model / what-it-costs / corrected-model form this track has used since Chapter 1.1.

Misconception 1 — "The latency is 4.564 microseconds."

Wrong model. A design has a latency, it was computed in the architecture review, and it is in the datasheet.

What it costs. Section 6's table: the same five hops are 4.564 µs, 7.184, 23.148 or 41.941 depending on two numbers nobody supplied — the offered load and the percentile. Against a 5 µs budget the design passes with 8.7% margin or fails by 8.39×, and both readings are defensible. The cost is that the first one is computed in month two and propagates everywhere, and the fourth is computed by a customer.

Corrected model. A latency figure is a triple: value, percentile, window — and a fourth field, the offered load, for anything above the floor. Chapter 26.1 §8 can produce the floor and nothing else, which is why it flagged the gap rather than filling it, and this chapter fills it with three numbers rather than one.

Misconception 2 — "The buffer holds 2 224 microseconds, so that is the worst-case latency."

Wrong model. The deepest queue times the drain rate is the tail.

What it costs. Section 8: at ρ = 0.99 the probability of a 2 224 µs wait in steady state is 2.04 × 10⁻²³⁷. Treating the bound as a tail makes every percentile look absurd and the usual response is to stop computing percentiles at all — which is how a design ends up with one latency number again. The bound is correct and it belongs to a regime with no steady state.

Corrected model. Two regimes, two kinds of number. Steady state, ρ < 1: a distribution, so a percentile. Incast, ρ ≥ 1: an event, so a fill time and a recovery time — 35.302 µs and 2 224 µs at 64 senders, an asymmetry of 63:1. A review needs both and they are not comparable.

Misconception 3 — "Our p99.99 is 5.6 microseconds."

Wrong model. The histogram reports a p99.99 and the register says 5 615 ns.

What it costs. The window is 1 024 samples and the highest percentile it can support is 99.902%, so the number is the window's maximum wearing a finer statistic's name — understated by 24.88% against a true 7.475 µs. And the incentive gradient runs the wrong way: a 256-sample window reports 4.484 µs, so every pressure toward a cheaper instrument improves the reported figure.

Corrected model. A percentile's name is a claim about the sample it came from, and the claim is checkable in one divide. Either report the percentile the window supports — p99.902, the same number honestly labelled — or use a structure whose limit is its accumulated count rather than its depth. A 32-bucket log histogram supports any percentile at 0.23 of a datapath, which is sixteen times cheaper than the FIFO that cannot.

Misconception 4 — "The p99 regressed, so something in the design got slower."

Wrong model. Latency is a property of the switch, so a change in latency is a change in the switch.

What it costs. Section 21's directed test: two traffic profiles at identical mean load, identical frame sizes and identical sample counts produce 23.1 µs and 61.4 µs — a factor of 2.66 with the design untouched. The cost is the investigation, which goes to the scheduler and the buffer allocation and finds nothing, because 4.1% of the interval was a different regime.

Corrected model. A percentile over a mixture is a percentile of neither population. Keep two histograms, switched by the overload bit, and report the share alongside every figure. The steady-state p99 in both profiles is 23.1 µs, and the second histogram that shows it costs 0.0117% of the die.

Misconception 5 — "That threshold is documented, so the deployment can set it."

Wrong model. The default is in the datasheet with an explanation, which is what documenting a parameter means.

What it costs. Chapter 25.2 §8's 300-second ageing default is 8× too long at 1 000 container teardowns an hour. Documented, correcting it needs silicon. Exposed, it is one register write. Four of this chapter's opening defaults are in this state, and each one has a deployment where it fails.

Corrected model. Exposure is a register address, and the review's item is that address. Chapter 26.1 §12's DEFERRED outcome requires two things and only one of them is a sentence: name the owner, and give them somewhere to write the answer. A parameter with an owner and no register is still a default.

Misconception 6 — "We cannot afford to instrument all of that."

Wrong model. Counters, histograms and timestamps are a real area cost that has to be traded against function.

What it costs. Nothing, and that is the finding. Section 10's complete apparatus is 1 443 840 BCE — 5.10 MAC receive datapaths, 0.2569% of Chapter 23.3's switch, and the recommended subset is 1.63 datapaths and 0.0820%. A proposal to double the MAC table costs 44 datapaths and is routinely approved. Trimming the packet buffer by 0.282% pays for the whole bill.

Corrected model. Price observability against the buffer, not against the datapath. The habit of comparing every structure to a MAC receive datapath is right at MAC scale and wrong at switch scale by three orders of magnitude, and it is why the observability conversation is always conducted in adjectives — nobody had derived the number, so there was nothing to say.


24. Interview Questions

Six, with what a strong answer contains.

1. What is the latency of your switch?

A strong answer asks which regime and then gives three numbers. 4.564 µs structural over five hops; 41.941 µs at ρ = 0.95, p99.99; and 2 224 µs after a 35.302 µs fill under a 64-way incast. The strongest part is naming the third, because an all-reduce is an N-way incast by construction and in an AI fabric that regime is the workload rather than an edge case.

2. Your tail latency has not moved in six months. Good or bad?

Suspicious. A tail figure that is stable across releases, loads and traffic profiles is usually an estimator pinned at its window's ceiling. The check is one comparison: the reported percentile against the window length. A 1 024-sample window supports p99.902 and reports 5.615 µs whatever the design does, so the number was fixed the day the histogram was wired up.

3. The buffer holds 2 224 microseconds. What percentile is that?

None — and the answer is the point. At ρ = 0.99 the steady-state probability of that wait is 2.04 × 10⁻²³⁷. It is a bound reachable only at ρ ≥ 1, which is an incast and has no steady state, so it has a fill time and a recovery time instead of a percentile. A candidate who names the regime rather than a number has understood the distinction the whole of Section 8 is about.

4. Which of your thresholds can your design not set?

Seven, and a strong answer names the owners. Buffer size, table size and storm control belong to the deployment; ageing to the orchestrator; the replay window to the topology; the flap threshold to the site; and the pause watermark to nobody, because its dead time includes the partner's response latency. The strongest half is the last one: six close with a conversation and one closes with a measurement at bring-up, and the correct deliverable for that one is a procedure and a validity boundary rather than a value.

5. Why is a 90% occupancy alarm on a four-way table wrong?

Because the table's achievable occupancy is 80.5%. Chapter 25.5 §6: offered 131 072 addresses, a 128k four-way table stores 105 465 and refuses 25 607 — 19.537% — while the occupancy reads 80.5%. The alarm cannot fire and its silence reads as health. The repair costs one comparator at configuration time and needs no measurement at all.

6. How much die would you spend on performance instrumentation?

The arithmetic, not an opinion. The complete bill at 64 ports is 1 443 840 BCE, 5.10 MAC datapaths, 0.2569% of a 5.62 × 10⁸ BCE switch; the recommended subset, after deleting the window FIFO that cannot report the percentile it was added for, is 1.63 datapaths and 0.0820%. Against a packet buffer that is 91.0% of the die, a 0.282% trim pays for all of it — and the right comparison is against the buffer, because comparing against a MAC datapath is a habit from a different scale.


25. Questions and Answers


26. What's Next

This chapter closed the deferrals Chapter 26.1 §12 opened and put the percentile back into a latency budget that had only a floor.

Four results worth carrying forward.

ResultValueWhy it travels
the percentile spread at one load5.84× from p50 to p99.99 at ρ = 0.95it is larger than the spread across loads, so the percentile is the axis to specify
a bound is not a tailP(W > 2 224 µs) at ρ = 0.99 is 2.04 × 10⁻²³⁷the test is whether the regime has a steady state
a window's ceiling1 − 1/W, and 1 024 gives p99.902it is arithmetic, not a property of any design
the observability bill1.63 datapaths, 0.0820% of a 6.4 Tb/s switchit ends the adjective conversation

And one deliverable that is not a number. Section 17's watermark item is closed by a register, a formula, a measurement procedure and a validity boundary — an item that only silicon can fill, handled by publishing everything except the value.

What remains of Module 26, and of the track.

NextWhat it adds
Chapter 26.4 — How Ethernet Is Actually Probedthe whole of it as an interview subject: what is being tested behind each question, what the track's 124 rejected-property classes have in common, and which single question separates a candidate who has the model from one who has the vocabulary

The sentence this chapter ends on is the track's, with a third edge added. Chapter 25.6 §20 and Chapter 26.1 §21 reached it and Chapter 26.2 §20 sharpened it: a check is worth exactly the set of observations that would have refuted it. Section 20 adds that the set can be non-empty, correctly sized and drawn from a sample that does not contain the quantity being checked — so a refutation set has to be a set of observations that could occur, and a p99.99 in a 1 024-sample window is a demand for an observation the instrument cannot make.


Continue learning

Standards & specifications

Governing standard
IEEE Std 802.3 (Ethernet)(opens IEEE in a new tab)

Defines the Ethernet MAC, the media-independent interfaces and the physical-layer sublayers, including framing, access control, auto-negotiation and per-rate PHY specifications. VLAN tagging, priority and time-sensitive shaping are defined by IEEE 802.1, not by 802.3.

This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.

Where this fits

Part of the Ethernet curriculum.