Ethernet · Module 26
Performance and Debug Review
Seven thresholds the design cannot set, a five-hop latency that is 4.564 µs or 41.941 µs depending on the percentile, and a complete observability apparatus for 0.0820% of the die.
Chapter 26.1 §12 ended with a list of seven thresholds nobody in the room could set, and called that list the review's most valuable output. This chapter is what happens to the list.
Because a deferred threshold does not stay deferred. It becomes a default, and this track has derived four defaults and found every one of them wrong by a large factor in a real deployment.
| The default | Wrong by | Where | Derived in |
|---|---|---|---|
| 300 s MAC ageing | 8× too long | 1 000 container teardowns an hour | Chapter 25.2 §8 |
| 75% pause watermark | 79× too conservative at 1 Gb/s over 100 m; 34 cells SHORT at 100 Gb/s over 2 km | both ends of one product's range | Chapter 14.2 §9 |
| 90% occupancy alarm | unreachable — the ceiling is 80.5% | any four-way table | Chapter 25.5 §8 |
| 1% storm control | correct on broadcast; discards user data on unknown unicast | any 64-port switch | Chapter 25.5 §12 |
Row two is the one to sit with. The same 75% is safe by a factor of seventy-nine on one link in a product's supported range and fails by 34 cells on another — and the failure mode is that every pause is issued too late, the queue overflows during the dead time, and the drops look exactly like the drops PAUSE was deployed to prevent.
A threshold the design cannot derive is not a gap in the specification. It is a parameter, and the review's product is the list of them with an owner against each.
And there is a second, quieter deferral in Chapter 26.1, which is the whole of Sections 6 to 9 here. That chapter's latency reconciliation produced 912.88 ns for one hop and 4.564 µs for five — and immediately flagged that every term in it was uncongested. It could not say what the latency is under load, because that is a distribution and an architecture review has no traffic.
This chapter has traffic, so it can put the number back.
| Five hops, 100 Gb/s, offered load 0.95 | Latency | Against the structural floor |
|---|---|---|
| the structural floor — Chapter 26.1 §8 | 4.564 µs | 1.00× |
| median | 7.184 µs | 1.57× |
| 99th percentile | 23.148 µs | 5.07× |
| 99.99th percentile | 41.941 µs | 9.19× |
Four numbers where a review usually has one, and the difference between the first and the last is the difference between a design that meets a 5 µs budget and one that misses it by eight times.
What this chapter does not own. Chapter 26.1 covered architecture and integration and Chapter 26.2 covered RTL and verification closure; Chapter 26.4 covers how all of it is probed in an interview. This is the pass held when the design performs and somebody has to say what "performs" means, and then build the instrument that will say whether it still does in the field.
1. Scope — A Number, a Percentile and an Instrument
Three rules, and the third is the one this chapter adds to the series.
Rule one, from Chapter 26.1 §1: every item is a number.
Rule two, from Chapter 26.2 §1: every number carries its denominator.
Rule three, and it is specific to performance: every latency figure carries a percentile and the window it was measured over.
| The quoted figure | What it asserts | Complete? |
|---|---|---|
| "latency is 912.88 ns" | the structural floor, uncongested | no — it is the condition the design is rarely in |
| "latency is 912.88 ns at p50" | a median, over an unstated window | no — a median over 100 samples is not a median |
| "4.564 µs floor; 23.148 µs at p99 over 10⁵ samples at ρ = 0.95" | a floor, a percentile, a window and a load | yes |
Row three has four numbers and every one of them can be wrong in a stateable way, which is the test this series has applied since Chapter 26.1.
And row three is why the percentile is not a unit. A metre is a unit; a percentile is an index into a distribution, and an index is meaningless without the population it indexes. So the honest unit for a latency claim in this chapter is a triple — value, percentile, window length — and Section 12 shows what happens when the third element is left out, which is that the second one becomes a fiction.
The chapter's four results.
| Result | Value | Section |
|---|---|---|
| the seven deferred thresholds, with owners and measurements | 7, of which 1 needs silicon | 4 |
| five-hop latency at p99.99, ρ = 0.95 | 41.941 µs — 9.19× the floor | 6 |
| why Chapter 26.1 §8's 487× is not a percentile | P(W > 2 224 µs) at ρ = 0.99 is 2.04 × 10⁻²³⁷ | 8 |
| the observability bill | 1 443 840 BCE — 5.10 datapaths, 0.2569% of the switch | 10 |
The last row is the one that settles most arguments before they start. Everything this chapter asks a design to measure — timestamps, histograms, watermarks, per-set refusal counters, the counter block, the timers — costs a quarter of one per cent of Chapter 23.3's 5.62 × 10⁸ BCE switch. The objection to observability has never been area; it is that nobody derived the number, so the discussion happened in adjectives.
2. The Twenty Items
Every row is a number, the chapter that derives it, and — for the seven that this chapter exists to close — who owns the threshold.
| # | The item | This design's number | Derived in | Owner |
|---|---|---|---|---|
| 1 | structural latency, one hop | 912.88 ns | Chapter 26.1 §8 | the design |
| 2 | structural latency, five hops | 4.564 µs | Chapter 26.1 §8 | the design |
| 3 | design-owned share of a hop | 8.76% — 80 ns of 912.88 | Chapter 26.1 §8 | the design |
| 4 | the offered load the budget is stated at | ρ = 0.95 | Section 6 | the deployment |
| 5 | five-hop latency at p99 | 23.148 µs — 5.07× the floor | Section 6 | derived from 4 |
| 6 | five-hop latency at p99.99 | 41.941 µs — 9.19× | Section 6 | derived from 4 |
| 7 | samples the p99.99 claim needs | 100 001 — 4.081 ms of traffic | Section 12 | structural |
| 8 | on-die latency window depth | 1 024 — resolving p99.902, not p99.99 | Section 12 | the design |
| 9 | maximum queueing one port can hold | 2 224 µs — 27.8 MB, 108 616 cells | Chapter 23.3 §9 | the design |
| 10 | the load at which that becomes reachable | ρ ≥ 1 — an incast, not a tail | Section 8 | the deployment |
| 11 | time for a 64-way incast to fill it | 35.302 µs | Section 8 | derived |
| 12 | buffer size | 64 MB | Chapter 23.3 §9 | THE DEPLOYMENT |
| 13 | MAC table size | 128k, storing 105 465 of 131 072 offered | Chapter 25.5 §6 | THE DEPLOYMENT |
| 14 | ageing interval | 300 s default; 833 stale entries at 10 000 teardowns/h | Chapter 25.2 §8 | THE ORCHESTRATOR |
| 15 | replay window | the fabric's worst-case path skew | Chapter 25.3 §15 | THE TOPOLOGY |
| 16 | storm-control setting | 1% caps one port at 0.98% of 9.524 Gpps | Chapter 25.5 §12 | THE DEPLOYMENT |
| 17 | flap threshold | the legitimate move rate | Chapter 25.2 §16 | THE SITE |
| 18 | pause watermark | 99.7% to 74.2% across the product's range | Chapter 14.2 §9 | THE CABLE PLANT |
| 19 | occupancy alarm | must be ≤ 80% — the ceiling is 80.5% | Chapter 25.5 §8 | the design — structural |
| 20 | observability area | 1 443 840 BCE — 5.10 datapaths, 0.2569% | Section 10 | the design |
Rows 12 to 18 are the seven Chapter 26.1 §12 deferred, and their owner column is in capitals because that is the review's finding: not one of them belongs to the design team, and all seven have a default in the RTL.
Row 19 is the counter-example that makes the point. The occupancy alarm looks environmental and is not — it is structural, because the ceiling is a property of the table's associativity — so it is the one threshold in this group the design can set correctly with no outside input, and the industry default of 90% is wrong anyway.
Row 8 against row 7 is the chapter's second finding and the subject of Section 12. A design that carries a 1 024-sample latency window and reports a p99.99 is reporting the maximum of 1 024 samples, whose expected percentile is 99.902 and whose value here is 5.615 µs against a true p99.99 of 7.475 µs — understated by 24.88%. The number is not noisy; it is biased, low, by construction, and it gets better-looking as the window gets smaller.
3. RTL 1 — The Performance Package and the Threshold Model
The package holds the seven thresholds, the three kinds of owner and the constants the rest of the chapter computes with. The module decides what state one threshold is in.
// ---------------------------------------------------------------------
// perfreview_pkg -- the seven deferred thresholds of Chapter 26.1
// Section 12, the four outcomes a threshold can be in, and the latency
// constants Sections 6 to 12 derive.
//
// Times are in nanoseconds unless a name says otherwise. Areas are in
// BCE, one bit of usable on-die SRAM, 0.35 gate-equivalents, twenty to
// a flip-flop -- the unit Chapter 19.7 Section 19 established.
// ---------------------------------------------------------------------
package perfreview_pkg;
// The seven. Each one has a default in the RTL and an owner who is
// not on the design team.
typedef enum logic [2:0] {
TH_BUFFER, // the burst distribution -- the deployment
TH_TABLE, // the address population -- the deployment
TH_AGEING, // the address recycle rate -- the orchestrator
TH_REPLAY, // the fabric's path skew -- the topology
TH_STORM, // the legitimate flood rate -- the deployment
TH_FLAP, // the legitimate move rate -- the site
TH_WATERMARK // the link's dead time -- the cable plant
} threshold_e;
localparam int N_THRESH = 7;
typedef enum logic [1:0] {
OWN_DESIGN, // derivable here
OWN_STRUCTURAL, // follows from the topology of the design itself
OWN_EXTERNAL, // somebody outside the room
OWN_SILICON // nobody, until first bring-up -- Section 17
} owner_e;
typedef enum logic [1:0] {
TH_UNSET, // no value at all
TH_DEFAULT, // a value, chosen by an implementer, not exposed
TH_DEFERRED, // owner named, parameter exposed
TH_BOUND // owner answered, value recorded
} th_state_e;
// Latency constants, from Chapter 26.1 Section 8's reconciliation.
localparam int HOP_FLOOR_NS = 913; // 912.88, rounded
localparam int HOPS = 5;
localparam int FIVE_HOP_FLOOR_NS = 4564; // 4.564 us
localparam int DESIGN_OWNED_NS = 80; // lookup 30 + fabric 50
localparam int DESIGN_OWNED_PPM = 87623; // 8.7623 per cent
// The buffer, from Chapter 23.3 Section 9. One port claiming its
// dynamic-threshold share is 108 616 cells of 256 octets.
localparam int CELL_OCTETS = 256;
localparam int MAX_CELLS_ONE_PORT = 108616;
localparam int MAX_QUEUE_US = 2224;
// Percentiles are an index into a distribution, not a unit, so this
// package carries the window alongside every one of them. Section 12.
typedef struct packed {
logic [31:0] value_ns;
logic [19:0] percentile_ppm; // 999900 is p99.99
logic [31:0] window_samples;
} latency_claim_t;
// The rule Section 12 derives: a window of W samples can support a
// percentile no finer than 1 - 1/W. Expressed as the minimum window
// for ten expected exceedances, which is the usable form.
function automatic int unsigned min_window(int unsigned pct_ppm);
if (pct_ppm >= 1000000) return 32'hFFFF_FFFF;
return (10 * 1000000) / (1000000 - pct_ppm);
endfunction
// Observability, Section 10, in BCE at 64 ports.
localparam int PORTS = 64;
localparam int DATAPATH_BCE = 283320;
localparam int SWITCH_BCE_DIV1K = 562000; // 5.62e8 / 1000
endpackageClassification, Model or Production? A CLASSIFICATION MODEL. It decides which of four states one threshold is in, and — critically — it distinguishes a value that was chosen from a value that was deferred and then defaulted, which is the distinction Chapter 26.1 §12's scenarios 47 and 48 turned on.
// ---------------------------------------------------------------------
// threshold_model -- one threshold in, its state and its owner out.
//
// The module's whole value is the DEFAULT state. A threshold with a
// value looks identical to one with a derivation in every report this
// track has examined, and the difference is whether anybody could
// change it in the field.
// ---------------------------------------------------------------------
module threshold_model
import perfreview_pkg::*;
(
input logic clk,
input logic rst_n,
input logic th_valid,
input threshold_e th_id,
input logic has_value, // the RTL has a number
input logic is_exposed, // the number is a register
input logic owner_named, // somebody is accountable
input logic owner_answered, // and they supplied a value
input logic needs_measurement,// only silicon can supply it
output logic out_valid,
output th_state_e state,
output owner_e owner,
output logic became_a_default, // the failure Section 4 names
output logic deferred_correctly
);
// Which owner each threshold has. Six external, one structural, and
// Section 17's one that no conversation can close.
function automatic owner_e owner_of(threshold_e t, logic meas);
if (meas) return OWN_SILICON;
case (t)
TH_BUFFER, TH_TABLE, TH_STORM : return OWN_EXTERNAL;
TH_AGEING, TH_FLAP : return OWN_EXTERNAL;
TH_REPLAY : return OWN_STRUCTURAL;
TH_WATERMARK : return OWN_EXTERNAL;
default : return OWN_DESIGN;
endcase
endfunction
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
out_valid <= 1'b0;
state <= TH_UNSET;
owner <= OWN_DESIGN;
became_a_default <= 1'b0;
deferred_correctly <= 1'b0;
end else begin
out_valid <= th_valid;
if (th_valid) begin
owner <= owner_of(th_id, needs_measurement);
if (owner_answered && has_value) begin
state <= TH_BOUND;
became_a_default <= 1'b0;
deferred_correctly <= 1'b0;
end else if (owner_named && is_exposed) begin
// The correct handling: named, exposed, awaiting an answer.
state <= TH_DEFERRED;
became_a_default <= 1'b0;
deferred_correctly <= 1'b1;
end else if (has_value) begin
// A number, no owner, not exposed. This is the state every
// one of the opening table's four defaults was in.
state <= TH_DEFAULT;
became_a_default <= 1'b1;
deferred_correctly <= 1'b0;
end else begin
state <= TH_UNSET;
became_a_default <= 1'b0;
deferred_correctly <= 1'b0;
end
end
end
end
endmoduleWhat it teaches: that TH_DEFAULT and TH_BOUND look identical from outside the design and are opposites. Both have a value; both produce a working product; only one of them can be corrected when the deployment turns out to differ from the implementer's assumption. The module also teaches that is_exposed is doing the real work — a threshold with a named owner and no register is still a default, because naming an owner who cannot change anything is a bookkeeping entry rather than a deferral.
Deliberately simplified: owner_of is a case statement where a real design reads the owner from a specification, so the mapping is hard-coded in exactly the way this chapter argues against — the honest form makes the owner a field in the same register that exposes the parameter, so the two travel together. needs_measurement is also an input rather than a derivation, where Section 17's argument is that exactly one threshold has this property and the reason is structural.
Production implication: the output worth wiring to a build-time check is became_a_default, and it should fail the build. A parameter with a value, no owner and no register is a deployment decision made at RTL time, and the four rows of this chapter's opening table are the price of making it. The cost of the check is one assertion per parameter in a parameter package; the cost of skipping it is a product whose pause watermark is 34 cells short on the longest link it is sold for, discovered by a customer.
4. The Seven Environmental Thresholds, Closed
Chapter 26.1 §12 named them. This section says, for each one, what measurement sets it, who can make that measurement, and what the design must expose so the answer can be used.
| Threshold | The measurement that sets it | Who can make it | What the design must expose |
|---|---|---|---|
| buffer size | the burst-length distribution at the deployment's offered load | the deployment, from a week of telemetry | per-queue maximum watermark, and the cell occupancy |
| MAC table size | the address population, counted, plus the recycle rate | the deployment | Chapter 25.5 §16's effective-capacity register, and per-set refusals |
| ageing interval | the teardown rate — 10 000 an hour gives 833 stale entries | the orchestrator, which already emits it | the interval as a register, and a stale-entry count |
| replay window | the fabric's worst-case path skew | the topology's designer, from a document | the window as a register, and a reorder-depth histogram |
| storm control | the legitimate flood rate, separately for broadcast and unknown unicast | the deployment | TWO settings, not one — Chapter 25.5 §12 |
| flap threshold | the legitimate move rate | the site | the threshold, and a per-address move counter |
| pause watermark | the dead time — Section 17 | NOBODY, until first bring-up | the watermark as a register, and the measured dead time |
Three of the seven need more than a register, and those three are where the observability bill of Section 10 comes from.
One — the buffer threshold needs a watermark, not an occupancy. Chapter 23.3 §9's buffer holds 2 224 µs at one port and a sampled occupancy will almost never catch the peak, because the peak is 35.302 µs wide (Section 8) and a management poll is seconds apart. The instrument is a per-queue maximum-since-last-read register — 512 queues at 32 bits, 16 384 BCE, 0.06 of a datapath — and without it the burst distribution cannot be measured at all.
Two — the storm-control threshold is two thresholds. Chapter 25.5 §12's finding is that a limiter on broadcast is correct and a limiter on unknown unicast discards legitimate user data to work around a table refusing inserts. A design exposing one setting invites one setting, so the exposure is itself the design decision: two registers, and the default on the unknown-unicast one is off.
Three — the MAC table threshold needs the refusals, not the occupancy. Chapter 25.5 §6's arithmetic: a 128k four-way table offered 131 072 addresses stores 105 465 and refuses 25 607 — 19.537% — while the occupancy counter reads 80.5%. So the occupancy metric cannot detect the condition; the per-set refusal counter can, at 32 768 sets by 4 bits — 131 072 BCE, 0.46 datapaths, 0.023% of the switch — and it turns a theorem into a histogram.
For four of the seven the design already has the instrument and hides it. For three of them the instrument does not exist and costs less than half a MAC datapath each.
5. RTL 2 — The Deferral Register
Seven thresholds, seven owners, seven registers. This module is the register file and the audit of it.
// ---------------------------------------------------------------------
// deferral_register -- the seven thresholds as live, writable state,
// each with its owner, its default, whether it has been answered, and
// whether the design exposes it.
//
// The module is the artefact Chapter 26.1 Section 12 said an
// architecture review should produce. Here it is, as registers.
// ---------------------------------------------------------------------
module deferral_register
import perfreview_pkg::*;
(
input logic clk,
input logic rst_n,
// The management write port -- this is the exposure.
input logic wr_en,
input threshold_e wr_id,
input logic [31:0] wr_value,
// The review's annotations, loaded at configuration time.
input logic cfg_en,
input threshold_e cfg_id,
input logic cfg_owner_named,
input logic cfg_exposed,
input logic cfg_needs_silicon,
input logic [31:0] defaults [N_THRESH],
output logic [31:0] value [N_THRESH],
output logic answered [N_THRESH],
output logic exposed [N_THRESH],
output logic owner_named[N_THRESH],
output logic silicon_gated [N_THRESH],
output logic [3:0] n_bound,
output logic [3:0] n_deferred,
output logic [3:0] n_defaulted,
output logic [3:0] n_silicon_gated,
output logic all_thresholds_disposed
);
integer i;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (i = 0; i < N_THRESH; i = i + 1) begin
value[i] <= defaults[i];
answered[i] <= 1'b0;
exposed[i] <= 1'b0;
owner_named[i] <= 1'b0;
silicon_gated[i] <= 1'b0;
end
end else begin
if (cfg_en) begin
owner_named[cfg_id] <= cfg_owner_named;
exposed[cfg_id] <= cfg_exposed;
silicon_gated[cfg_id] <= cfg_needs_silicon;
end
// A write is the owner answering. It is the ONLY way answered
// becomes high, which is why exposure and accountability are the
// same mechanism rather than two.
if (wr_en && exposed[wr_id]) begin
value[wr_id] <= wr_value;
answered[wr_id] <= 1'b1;
end
end
end
// The four counts of Section 4, recomputed combinationally so a
// management read always sees the current disposition.
always_comb begin
n_bound = '0;
n_deferred = '0;
n_defaulted = '0;
n_silicon_gated = '0;
for (int j = 0; j < N_THRESH; j = j + 1) begin
if (silicon_gated[j] && !answered[j])
n_silicon_gated = n_silicon_gated + 4'd1;
else if (answered[j])
n_bound = n_bound + 4'd1;
else if (owner_named[j] && exposed[j])
n_deferred = n_deferred + 4'd1;
else
n_defaulted = n_defaulted + 4'd1;
end
// Disposed means every threshold is in a state somebody chose.
// A defaulted threshold is the one state nobody chose.
all_thresholds_disposed = (n_defaulted == 4'd0);
end
endmoduleClassification, Model or Production? PRODUCTION-INTENT. Every design that ships with configurable thresholds has most of this; what it does not have is the owner_named and exposed annotations beside the values, which is the part that makes the register file a review artefact rather than a configuration block.
What it teaches: that a write only lands if the parameter is exposed, and that this is not a safety feature — it is the definition. A threshold that can be changed only by rebuilding is not exposed, whatever the documentation says, and the module makes the two statements the same statement. It also teaches why n_silicon_gated is counted before n_bound: a threshold awaiting a measurement that does not exist yet is in a different state from one awaiting an answer from a person, and Section 17 is about the single member of that category.
Deliberately simplified: the annotations are loaded through a configuration port where in practice they are compile-time constants in a parameter package, so a real implementation makes them localparam and this module's cfg_en path becomes a generate-time elaboration. The register file also has no access control — any writer is treated as the owner — where a real design partitions the address space so that the orchestrator's thresholds and the cable plant's thresholds are not writable by the same agent. And defaults arrives as an unpacked array input, which is a testbench convenience rather than a synthesisable form.
Production implication: all_thresholds_disposed is the bit to read at the end of bring-up and it should be high. It is low exactly when some parameter has a value nobody chose, which is Chapter 26.1 §12's failure rendered as one bit of status. The operational value is that it is readable from the field: a support engineer looking at a misbehaving deployment can ask the part whether any of its seven thresholds is a default, and get an answer in one register read instead of a three-week correspondence with the vendor.
6. The Percentile the Floor Was Missing
Chapter 26.1 §8 produced seven latency terms and one total, flagged that every term was uncongested, and said the review's obligation was to name a percentile. This section names it.
The queueing model, stated so it can be argued with. One hop is a single server with Poisson arrivals and the frame-length distribution of Chapter 26.2 §8 — mean 510.09 wire octets, 40.81 ns, so the server rate is 2.4505 × 10⁷ frames per second. Waiting time in queue then has the closed form P(W > t) = ρ · e^(−(1−ρ)μt), and a percentile inverts it.
Five hops, and every cell is the floor plus five independent queues.
| Offered load ρ | p50 | p90 | p99 | p99.9 | p99.99 |
|---|---|---|---|---|---|
| 0.50 | 4.564 µs | 5.221 | 6.161 | 7.100 | 8.040 |
| 0.80 | 5.044 | 6.686 | 9.035 | 11.384 | 13.733 |
| 0.90 | 5.764 | 9.048 | 13.746 | 18.444 | 23.142 |
| 0.95 | 7.184 | 13.751 | 23.148 | 32.544 | 41.941 |
| 0.99 | 18.502 | 51.341 | 98.323 | 145.305 | 192.287 |
The top-left cell is Chapter 26.1 §8's 4.564 µs, and it is the only cell in the table an architecture review can produce. Everything else needs an offered load.
Three readings of the table, and the third is the one that changes a design.
One — the floor is the answer only at the top-left. At ρ = 0.5 and p50 the queueing contribution is 0.000 µs and the structural figure is exact. One row down and four columns right, at ρ = 0.8 and p99.99, it is 13.733 µs — three times the floor — and neither figure is wrong.
Two — the spread within one row is larger than the spread between rows near the middle. At ρ = 0.95 the latency runs 7.184 to 41.941 µs across the percentile axis, a factor of 5.84; from ρ = 0.5 to ρ = 0.95 at p50 it runs 4.564 to 7.184, a factor of 1.57. So a specification that names a load and not a percentile has pinned down the less important of the two axes.
Three — the design owns 8.76% of the floor and essentially all of the queueing. Chapter 26.1 §8's finding was that propagation is 53.7% of a hop and the design owns 80 ns of 912.88. Under load that inverts: at ρ = 0.95 and p99.99, 37.377 of 41.941 µs is queueing — 89.1% — and queueing is the buffer's size and the scheduler's policy, which are the design's.
Uncongested, a latency argument is about physics and the design is a bystander. Congested, it is entirely about the design. The percentile is what decides which conversation the review is having.
And the practical form of the review item, which is four numbers rather than one.
| Component | This design |
|---|---|
| the structural floor | 4.564 µs, five hops |
| the load the specification is written at | ρ = 0.95 |
| the percentile | p99.99 |
| the figure | 41.941 µs |
A review that records only the first has recorded the one number that is never the answer, which is the sentence Chapter 26.1 §8 ended on, now with the other three supplied.
7. RTL 3 — The Latency Percentile Estimator
// ---------------------------------------------------------------------
// latency_percentile -- estimate a percentile from a stream of measured
// latencies, and REFUSE to report one the window cannot support.
//
// The refusal is the module's reason for existing. Section 12 shows a
// 1024-sample window reporting a p99.99 that is really a p99.902,
// biased low by 24.88 per cent, with no indication that anything is
// wrong with the number.
// ---------------------------------------------------------------------
module latency_percentile
import perfreview_pkg::*;
#(
parameter int unsigned WINDOW = 1024,
// Log-spaced buckets, so the tail is resolved without storing samples.
parameter int unsigned N_BUCKETS = 32,
parameter int unsigned BUCKET_SHIFT = 6 // 64 ns per bucket at the base
)(
input logic clk,
input logic rst_n,
input logic clear,
input logic sample_valid,
input logic [31:0] sample_ns,
// What the caller wants, in parts per million: 999900 is p99.99.
input logic [19:0] want_ppm,
output logic [31:0] estimate_ns,
output logic [31:0] samples_seen,
output logic [31:0] window_required,
output logic estimate_is_supported, // the refusal
output logic [19:0] best_supported_ppm,
output logic [31:0] observed_max_ns,
output latency_claim_t claim // value, percentile, window
);
logic [31:0] bucket [N_BUCKETS];
logic [31:0] total;
// Log-spaced bucket index: the leading one of the sample, floored.
function automatic int unsigned bucket_of(logic [31:0] ns);
int unsigned b;
b = 0;
for (int k = 31; k >= 0; k = k - 1)
if (ns[k]) begin b = k; break; end
if (b < BUCKET_SHIFT) return 0;
if ((b - BUCKET_SHIFT) >= N_BUCKETS) return N_BUCKETS - 1;
return b - BUCKET_SHIFT;
endfunction
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n || clear) begin
for (int i = 0; i < N_BUCKETS; i = i + 1) bucket[i] <= '0;
total <= '0;
observed_max_ns <= '0;
end else if (sample_valid) begin
bucket[bucket_of(sample_ns)] <= bucket[bucket_of(sample_ns)] + 32'd1;
total <= total + 32'd1;
if (sample_ns > observed_max_ns) observed_max_ns <= sample_ns;
end
end
// Walk the buckets from the top until the requested tail mass is
// accounted for. The answer is a bucket boundary, which is why the
// buckets are log-spaced: a linear scale would resolve the median
// beautifully and the tail not at all.
logic [31:0] target_tail;
logic [31:0] running;
logic [31:0] est;
always_comb begin
samples_seen = total;
window_required = min_window(want_ppm);
// The refusal: ten expected exceedances, or the number is a maximum
// wearing a percentile's name.
estimate_is_supported = (total >= window_required) && (total != 0);
// The finest percentile this window can actually support.
best_supported_ppm = (total == 0) ? 20'd0
: 20'(1000000 - (10 * 1000000) / int'(total));
target_tail = 32'((longint'(total) *
longint'(1000000 - int'(want_ppm))) / 1000000);
running = '0;
est = observed_max_ns;
for (int i = N_BUCKETS - 1; i >= 0; i = i - 1) begin
running = running + bucket[i];
if (running >= target_tail) begin
est = 32'(32'd1 << (i + BUCKET_SHIFT));
break;
end
end
estimate_ns = est;
claim.value_ns = est;
claim.percentile_ppm = want_ppm;
claim.window_samples = total;
end
endmoduleClassification, Model or Production? PRODUCTION-INTENT. A 32-bucket log-spaced histogram per port is the structure real switches carry, and estimate_is_supported is the one output they do not.
What it teaches: that a percentile and its window travel together or the percentile is fiction. claim is a three-field struct for exactly this reason — value, percentile, window — and a management interface that returns only the first field has returned a number that cannot be checked. The module also teaches why the buckets are log-spaced: a linear histogram with 32 buckets over the p50-to-p99.99 range at ρ = 0.95, which is 1.437 µs to 8.388 µs per hop, puts twenty-nine buckets below the median and three in the tail. Log spacing puts about five in each decade, which is where the resolution is needed.
Deliberately simplified: the estimate is a bucket boundary, so its resolution is one octave — at the tail that is a factor of two, which is coarse. A real implementation interpolates within the bucket using its count, which costs a multiply and a divide and recovers most of the resolution. The bucket_of leading-one search is written as a loop with a break for readability; in silicon it is a priority encoder. And min_window uses ten expected exceedances, which is a convention rather than a theorem — the right number depends on how much variance in the estimate is acceptable, and ten gives roughly ±30%.
Production implication: wire estimate_is_supported to the management read and return an error rather than a number when it is low. The alternative — returning the best available estimate with a note — is worse than it sounds, because the note is lost the first time somebody copies the value into a spreadsheet. A register that refuses to answer a question it cannot answer is the only reliable way to stop a 1 024-sample maximum from being quoted as a p99.99 for the product's life, which Section 12 shows is a 24.88% understatement that looks entirely plausible.
8. Why 487× Is Not a Percentile
Chapter 26.1 §8 put the buffer's maximum queueing delay — 2 224 µs — against the five-hop floor and got a ratio of 487. That number is correct and it is not a tail of Section 6's distribution. It belongs to a different regime, and confusing the two is the most common error in a performance review.
Ask Section 6's model what it thinks of 2 224 µs.
| Offered load | P(W > 2 224 µs) |
|---|---|
| ρ = 0.95 | 0 to the precision of any arithmetic |
| ρ = 0.99 | 2.04 × 10⁻²³⁷ |
At ρ = 0.99 the model says a frame waits 2 224 µs about once in every 5 × 10²³⁶ frames, which at 24.5 million frames a second is a wait longer than the age of the universe by two hundred and twenty orders of magnitude. The buffer's maximum is not a percentile of the steady-state queue. It is unreachable in steady state at any load below one.
And yet the buffer fills, routinely, in real fabrics. The resolution is that ρ < 1 is the wrong model for the event that fills it.
An incast is ρ ≥ 1. N senders transmit at line rate into one egress port; the port drains at line rate; the queue grows at (N − 1) × line rate until the senders stop. There is no steady state and no percentile — there is a fill time.
| Senders | Fill rate | Time to reach 27.8 MB | Drain time |
|---|---|---|---|
| 2 | 12.5 GB/s | 2 224 µs | 2 224 µs |
| 4 | 37.5 GB/s | 741.3 µs | 2 224 µs |
| 8 | 87.5 GB/s | 317.7 µs | 2 224 µs |
| 16 | 187.5 GB/s | 148.3 µs | 2 224 µs |
| 32 | 387.5 GB/s | 71.7 µs | 2 224 µs |
| 64 | 787.5 GB/s | 35.302 µs | 2 224 µs |
The drain column does not move, because drain is always one line rate. So the asymmetry is exactly N − 1 — a 64-way incast fills one port's maximum share in 35.302 µs and takes 2 224 µs to recover, a ratio of 63:1 — and during that recovery every frame behind it carries the full 2 224 µs.
Steady-state queueing is a distribution and needs a percentile. Incast is an event and needs a fill time and a recovery time. A review that has one and not the other has described half its design's latency behaviour.
And Chapter 23.2 is the deployment where the second half dominates. An all-reduce is an N-way incast by construction — every participant sends to the same reducer at the same instant — so the regime this section describes is not an edge case in that market, it is the workload.
Which gives the review three latency numbers instead of one, and they answer three different questions.
| Number | Regime | Question it answers |
|---|---|---|
| 4.564 µs | idle | what does the structure cost? |
| 41.941 µs | ρ = 0.95, p99.99 | what does a loaded but stable fabric cost? |
| 2 224 µs after 35.302 µs | incast, ρ ≥ 1 | what does a collective cost? |
Three numbers, three regimes, and Chapter 26.1 §8 could produce only the first, because the other two need an offered load and a traffic pattern that an architecture review does not have. This is what it means for a threshold to be environmental.
9. RTL 4 — The Incast Model
// ---------------------------------------------------------------------
// incast_model -- the regime Section 8 separates from Section 6. No
// percentiles here, because there is no steady state: an incast has a
// fill time, a peak and a recovery time.
//
// The module's output that matters is regime_is_overload, because every
// queueing statistic computed while it is high is meaningless.
// ---------------------------------------------------------------------
module incast_model
import perfreview_pkg::*;
#(
// Octets per second per sender at 100 Gb/s, in units of 1e6 oct/s.
parameter int unsigned LINE_MOCT_S = 12500,
parameter int unsigned CAP_OCTETS = 27800000 // 108 616 cells
)(
input logic clk,
input logic rst_n,
input logic eval,
input logic [7:0] n_senders,
input logic [31:0] burst_octets_per_sender,
output logic regime_is_overload,
output logic [31:0] fill_rate_moct_s,
output logic [31:0] fill_time_us,
output logic [31:0] drain_time_us,
output logic [31:0] asymmetry, // drain / fill, which is N-1
output logic buffer_overflows,
output logic [31:0] octets_dropped,
output logic [31:0] peak_queue_octets
);
logic [63:0] offered;
logic [63:0] peak;
always_comb begin
// Overload is a comparison of rates, not a probability. One sender
// into one port is rho = 1 exactly and never builds a queue; two
// is the first overload case.
regime_is_overload = (n_senders > 8'd1);
fill_rate_moct_s = regime_is_overload
? 32'((int'(n_senders) - 1) * int'(LINE_MOCT_S))
: 32'd0;
// Microseconds to reach the cap, at the fill rate. Units: octets
// divided by 1e6 octets per second gives microseconds directly.
fill_time_us = (fill_rate_moct_s == 0) ? 32'hFFFF_FFFF
: 32'(CAP_OCTETS / fill_rate_moct_s);
// Drain is always one line rate, whatever N is. This is the row
// of Section 8's table that does not move.
drain_time_us = 32'(CAP_OCTETS / LINE_MOCT_S);
asymmetry = (fill_time_us == 0) ? 32'd0
: 32'(drain_time_us / fill_time_us);
// How much the senders actually offer, against what fits.
offered = 64'(n_senders) * 64'(burst_octets_per_sender);
peak = (offered > 64'(CAP_OCTETS)) ? 64'(CAP_OCTETS) : offered;
peak_queue_octets = 32'(peak);
buffer_overflows = (offered > 64'(CAP_OCTETS));
octets_dropped = buffer_overflows
? 32'(offered - 64'(CAP_OCTETS)) : 32'd0;
end
endmoduleClassification, Model or Production? A BEHAVIOURAL MODEL — this is NOT how a buffer works. A real shared buffer allocates in cells with a dynamic threshold, the senders do not start simultaneously, and congestion control reacts within a round trip. The model computes the bound, not the trajectory.
What it teaches: that the drain time does not depend on N and the fill time does, which is the whole shape of an incast and the reason it is not describable as a percentile. Sixty-four senders fill the buffer 63 times faster than two do and recover at exactly the same speed — so as a fabric gets wider, the event gets sharper and the recovery does not, and the ratio of the two is the number a scheduler has to survive.
Deliberately simplified: the senders are synchronous and unthrottled, which is the worst case and not the common one; a real all-reduce has skew between participants and a congestion-control loop that reacts in about one round trip. The cap is the single-port maximum from Chapter 23.3 §9 rather than a dynamic threshold that shrinks as other ports queue — so CAP_OCTETS is itself a function of the rest of the switch's state, and treating it as a constant overstates one port's share whenever anything else is busy. And octets_dropped assumes drops happen at the tail, where a real design may use a drop policy that discards from elsewhere.
Production implication: regime_is_overload is the signal to gate every queueing statistic on. A p99 latency computed over an interval that contained an incast is not a p99 of anything — it is a mixture of two distributions with no common parameter, and the mixture weight is the fraction of the interval spent in overload. The cheapest correct behaviour is to keep two histograms, one for each regime, switched by this bit, which doubles Section 10's histogram cost from 0.231 datapaths to 0.463 — 0.0233% of the switch — and makes both numbers mean something.
10. What the Measurement Costs in Silicon
Sections 4, 6 and 8 asked a design to measure seven thresholds, a latency distribution and an overload regime. This section prices the instruments, in BCE, against Chapter 23.3's 64-port switch.
Seven structures, and every row names the section that demanded it.
| Instrument | Geometry | Bits | BCE | × a datapath | % of the switch |
|---|---|---|---|---|---|
| ingress timestamp through the pipeline | 64 ports × 48 b × 36 stages | 110 592 | 110 592 | 0.39 | 0.0197% |
| latency histogram | 64 × 32 buckets × 32 b | 65 536 | 65 536 | 0.23 | 0.0117% |
| latency window FIFO | 64 × 1 024 × 16 b | 1 048 576 | 1 048 576 | 3.70 | 0.1866% |
| per-queue maximum watermark | 512 queues × 32 b | 16 384 | 16 384 | 0.06 | 0.0029% |
| per-set refusal counter | 32 768 sets × 4 b | 131 072 | 131 072 | 0.46 | 0.0233% |
| the counter block at Chapter 19.7's widths | 64 × 992 b | 63 488 | 63 488 | 0.22 | 0.0113% |
| dead-time and flap timers | 64 × 4 × 32 b | 8 192 | 8 192 | 0.03 | 0.0015% |
| TOTAL | — | 1 443 840 | 1 443 840 | 5.10 | 0.2569% |
Everything this chapter asks a 6.4 Tb/s switch to measure costs five MAC receive datapaths — a quarter of one per cent of its on-die state.
Four observations, and the last is the one worth taking to a product meeting.
One — the window FIFO is 72.6% of the bill on its own, and Section 12 shows it is also the item that cannot do the job it was added for. The histogram at 0.23 datapaths resolves the same tail sixteen times cheaper, which makes the largest line in this table also the most questionable one.
Two — the counter block reconciles with Chapter 23.3's own bill, and slightly in the safe direction. That chapter budgeted RMON at 64 × 27 × 64 b = 110 592 BCE, 0.39 of a datapath. Chapter 19.7 §2's actual design is four counters at 64 bits and twenty-three at 32 — 992 bits per port, 63 488 BCE — so the switch chapter over-budgeted by a factor of 1.742 and the reconciliation lands 0.17 of a datapath better than planned. Both figures are rounding errors at this scale and that is the point: no observability argument has ever been won or lost on area.
Three — the per-set refusal counter is the highest-value row per BCE. Chapter 25.5 §6's table refuses 19.537% of offered addresses while the occupancy counter reads 80.5%; the refusal counter is the only instrument that can see it, and it costs 0.46 of a datapath and 0.0233% of the die.
Four — the whole bill is smaller than one rounding decision on the packet buffer. Chapter 23.3 §6: the buffer is 5.12 × 10⁸ BCE, 91.0% of the switch, and a 10% trim of it saves 181 datapaths. This chapter's complete observability bill is 5.10. Trimming the buffer by 0.282% pays for all of it.
11. RTL 5 — The Latency Histogram and Watermark Bank
// ---------------------------------------------------------------------
// obs_bank -- the instruments Section 10 prices, as one block: a
// log-spaced latency histogram, a per-queue maximum watermark, and the
// regime bit that says which histogram a sample belongs in.
//
// The watermark is the structure Section 4 argues for over a sampled
// occupancy: a 35-microsecond peak is invisible to a poll seconds apart.
// ---------------------------------------------------------------------
module obs_bank
import perfreview_pkg::*;
#(
parameter int unsigned N_QUEUES = 512,
parameter int unsigned N_BUCKETS = 32,
parameter int unsigned BUCKET_SHIFT = 6
)(
input logic clk,
input logic rst_n,
// Latency samples, tagged with the regime they were taken in.
input logic lat_valid,
input logic [31:0] lat_ns,
input logic in_overload, // from Section 9
// Queue occupancy, sampled every cycle rather than polled.
input logic occ_valid,
input logic [8:0] occ_queue,
input logic [31:0] occ_cells,
// Management read, which clears the watermark it returns.
input logic rd_en,
input logic [8:0] rd_queue,
output logic [31:0] rd_watermark,
output logic [31:0] hist_steady [N_BUCKETS],
output logic [31:0] hist_overload[N_BUCKETS],
output logic [31:0] n_steady,
output logic [31:0] n_overload,
output logic [19:0] overload_share_ppm
);
logic [31:0] watermark [N_QUEUES];
function automatic int unsigned bucket_of(logic [31:0] ns);
int unsigned b;
b = 0;
for (int k = 31; k >= 0; k = k - 1)
if (ns[k]) begin b = k; break; end
if (b < BUCKET_SHIFT) return 0;
if ((b - BUCKET_SHIFT) >= N_BUCKETS) return N_BUCKETS - 1;
return b - BUCKET_SHIFT;
endfunction
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int i = 0; i < N_BUCKETS; i = i + 1) begin
hist_steady[i] <= '0;
hist_overload[i] <= '0;
end
for (int q = 0; q < N_QUEUES; q = q + 1) watermark[q] <= '0;
n_steady <= '0;
n_overload <= '0;
rd_watermark <= '0;
end else begin
// Two histograms, switched by the regime bit. Section 9's
// production note: a percentile over a mixture is a percentile
// of nothing.
if (lat_valid) begin
if (in_overload) begin
hist_overload[bucket_of(lat_ns)] <=
hist_overload[bucket_of(lat_ns)] + 32'd1;
n_overload <= n_overload + 32'd1;
end else begin
hist_steady[bucket_of(lat_ns)] <=
hist_steady[bucket_of(lat_ns)] + 32'd1;
n_steady <= n_steady + 32'd1;
end
end
// The watermark: maximum since last read, updated every cycle.
if (occ_valid && (occ_cells > watermark[occ_queue]))
watermark[occ_queue] <= occ_cells;
// Read-and-clear, so consecutive reads are disjoint intervals
// rather than a running maximum that saturates at the first peak.
if (rd_en) begin
rd_watermark <= watermark[rd_queue];
watermark[rd_queue] <= '0;
end
end
end
always_comb begin
overload_share_ppm = ((n_steady + n_overload) == 0) ? 20'd0
: 20'((longint'(n_overload) * 1000000)
/ longint'(n_steady + n_overload));
end
endmoduleClassification, Model or Production? PRODUCTION-INTENT, and the read-and-clear watermark is the part most worth copying verbatim.
What it teaches: that read-and-clear turns a maximum into a time series. A running maximum saturates at the first peak and reports the same value forever; clearing on read makes each read the maximum over the interval since the previous one, so a management system polling every second gets a per-second peak — which is the burst distribution Section 4 needs to set the buffer threshold. It also teaches that the regime bit costs one flip-flop and doubles the histogram, which is the correct trade because the alternative is one histogram whose percentiles are meaningless.
Deliberately simplified: the watermark array is 512 deep with a single write port and a read-modify-write on management access, which races with an occupancy update to the same queue in the same cycle — a real implementation arbitrates, and the loss is at most one sample. The histograms have no overflow handling: at 24.5 M frames per second a 32-bit bucket wraps in 175 seconds if every frame lands in it, which is Chapter 19.7 §14's wrapping problem arriving in a new structure, and the honest fix is the same one — document them as differencing counters and read them faster than they wrap.
Production implication: overload_share_ppm is the number that makes every other latency figure interpretable, and it is one divide. A p99 quoted alongside "overload share 0 ppm" is a steady-state p99 and Section 6's table applies; a p99 quoted alongside "overload share 40 000 ppm" is a mixture and the right response is to read the two histograms separately. Without the share, a latency regression between two software releases is uninterpretable, because the traffic may simply have become burstier — and that is the most common false alarm in switch performance work.
12. The Window Is the Resolution
Section 7's module refuses to report a percentile its window cannot support. This section is the arithmetic behind the refusal, and it is the sharpest result in the chapter.
A percentile is an order statistic, and an order statistic needs members above it. In a window of W samples, the highest percentile with even one expected member above it is 1 − 1/W. Ask for anything finer and the estimator has nothing to return but the window's maximum.
| Window | Highest percentile with one expected sample | What it returns at ρ = 0.95 | Against a true p99.99 of 7.475 µs |
|---|---|---|---|
| 256 | 99.609% | 4.484 µs | 40.02% low |
| 1 024 | 99.902% | 5.615 µs | 24.88% low |
| 4 096 | 99.976% | 6.747 µs | 9.75% low |
| 16 384 | 99.994% | 7.878 µs | 5.39% HIGH |
| 65 536 | 99.998% | 9.010 µs | 20.53% high |
Rows one to three are the failure and rows four and five are the other failure, and they are not symmetric.
Below the crossover, the estimator returns the window maximum and calls it whatever percentile was asked for. The number is biased low, it is plausible, and it gets better-looking as the window gets smaller — which is exactly backwards, because a smaller window is a cheaper design. A 256-sample window reports 4.484 µs for a p99.99 that is really 7.475, and 4.484 µs is a very satisfactory number to put in a datasheet.
Above the crossover the estimator is returning a real order statistic of a longer window, so it is high relative to p99.99 simply because it is a different, finer percentile — which is a labelling error rather than a measurement error, and it is caught by reading the window length.
Below the crossover the number is wrong and looks right. Above it the number is right and is labelled wrong. Only the window length distinguishes the two, and it is the field that never travels.
The other direction: what window does a percentile need? The convention this chapter uses is ten expected exceedances, which gives an estimate accurate to roughly ±30%.
| Percentile | Minimum window | Traffic at 24.505 Mfps |
|---|---|---|
| p99 | 1 000 | 40.8 µs |
| p99.9 | 10 000 | 408 µs |
| p99.99 | 100 001 | 4.081 ms |
| p99.999 | 1 000 001 | 40.808 ms |
Four milliseconds of traffic for a p99.99 — which is nothing. So the constraint is not the traffic; it is the storage. A 100 001-sample window on 64 ports at 16 bits is 1.024 × 10⁸ BCE — 361.4 datapaths, 18.221% of the switch — which is why no design carries one and why the window is 1 024.
And that is the resolution of the whole section: the FIFO is the wrong instrument.
| Instrument | BCE | Datapaths | Can it report a p99.99? |
|---|---|---|---|
| 1 024-sample FIFO | 1 048 576 | 3.70 | NO — it returns a p99.902, 24.88% low |
| 100 001-sample FIFO | 1.024 × 10⁸ | 361.4 | yes, at 18.221% of the die |
| 32-bucket log histogram | 65 536 | 0.23 | yes, to one octave, at 0.0117% |
A histogram does not store samples; it stores counts, and a count has no window limit. Sixteen times cheaper than the FIFO that cannot answer the question, fifteen hundred times cheaper than the FIFO that can — and its only cost is resolution within a bucket, which interpolation recovers most of.
So Section 10's largest line item is also its most questionable, and this chapter's recommendation is to delete it: keep the histogram, keep the watermark, drop the window FIFO, and the observability bill falls from 5.10 datapaths to 1.40 — 0.0703% of the switch.
13. RTL 6 — The Window Auditor
// ---------------------------------------------------------------------
// window_auditor -- given a latency claim, decide whether the window
// it was measured over can support the percentile it names, and say
// which percentile it can support instead.
//
// This is the module that would have stopped a 1024-sample maximum from
// being published as a p99.99 for the life of a product.
// ---------------------------------------------------------------------
module window_auditor
import perfreview_pkg::*;
#(
// Expected exceedances required for an estimate to be supported.
parameter int unsigned EXCEEDANCES = 10
)(
input logic clk,
input logic rst_n,
input logic claim_valid,
input latency_claim_t claim,
input logic from_histogram, // counts, not samples
output logic out_valid,
output logic supported,
output logic [19:0] supportable_ppm,
output logic [31:0] window_required,
output logic [31:0] window_shortfall,
output logic [15:0] n_claims,
output logic [15:0] n_unsupported,
output logic [15:0] n_histogram_claims,
output logic any_unsupported_claim
);
logic [31:0] need;
logic [19:0] best;
always_comb begin
// The window a claimed percentile needs.
need = (claim.percentile_ppm >= 20'd1000000) ? 32'hFFFF_FFFF
: 32'((EXCEEDANCES * 1000000) /
(1000000 - int'(claim.percentile_ppm)));
// The finest percentile this window supports. A histogram is not
// window-limited in the same way: it accumulates counts, so its
// limit is the total sample count rather than a stored depth.
best = (claim.window_samples == 0) ? 20'd0
: 20'(1000000 - (EXCEEDANCES * 1000000)
/ int'(claim.window_samples));
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
out_valid <= 1'b0;
supported <= 1'b0;
supportable_ppm <= '0;
window_required <= '0;
window_shortfall <= '0;
n_claims <= '0;
n_unsupported <= '0;
n_histogram_claims <= '0;
end else begin
out_valid <= claim_valid;
if (claim_valid) begin
n_claims <= n_claims + 16'd1;
window_required <= need;
supportable_ppm <= best;
if (from_histogram)
n_histogram_claims <= n_histogram_claims + 16'd1;
if (claim.window_samples >= need) begin
supported <= 1'b1;
window_shortfall <= '0;
end else begin
supported <= 1'b0;
window_shortfall <= need - claim.window_samples;
n_unsupported <= n_unsupported + 16'd1;
end
end
end
end
always_comb begin
any_unsupported_claim = (n_unsupported != 16'd0);
end
endmoduleClassification, Model or Production? A REVIEW INSTRUMENT, and one that belongs in the management interface rather than in a spreadsheet.
What it teaches: that supportable_ppm is more useful than supported. A boolean says the claim is bad; the supportable percentile says what the claim should have said — a 1 024-sample window supports p99.902, so the honest label for its number is p99.902 and the number itself is fine. The module turns a wrong measurement into a correct one by changing its name, which is the cheapest repair available anywhere in this chapter.
Deliberately simplified: from_histogram is a flag where the real distinction is structural — a histogram's limit is its total count and a FIFO's is its depth, so the two need different best formulas and this module uses one. EXCEEDANCES = 10 is also a convention rather than a derivation: the right value depends on the acceptable variance in the estimate, and a design specifying its latency at p99.99 with a ±10% tolerance needs closer to a hundred. And the integer divides truncate, which makes supportable_ppm pessimistic by up to one part per million — the safe direction.
Production implication: put this check in the datasheet generation flow, not only in the management interface. Every latency figure that reaches a customer should have passed through a comparison between its percentile and its window, and the comparison is two divides. The failure it prevents is the most durable kind in this chapter: a number that is wrong by 24.88%, plausible, published, quoted back by customers, and defended internally for years — because nothing in the flow ever recorded the window it came from.
14. What a Performance Review Must Never Do
Six prohibitions, and each one has a named example earlier in this track.
| # | Never | Because |
|---|---|---|
| 1 | quote a latency without a percentile and a load | Section 6 — one budget, four answers, spanning 9.19× |
| 2 | quote a percentile without its window | Section 12 — a 1 024-sample p99.99 is 24.88% low and plausible |
| 3 | treat a bound as a tail | Section 8 — P(W > 2 224 µs) at ρ = 0.99 is 2 × 10⁻²³⁷ |
| 4 | compute a percentile across an incast | Section 9 — a mixture of two distributions is a percentile of neither |
| 5 | leave an environmental threshold at a default | Section 4 — four defaults, four deployments, four failures |
| 6 | set an alarm without checking the metric can reach it | Chapter 25.5 §8 — 90% on a ceiling of 80.5% |
Prohibition 6 is the one that needs no measurement at all, which makes it the cheapest item in Section 2 and the one most often wrong. The check is one comparison: the alarm's threshold against the metric's maximum achievable value, and Chapter 25.5 §8's four-way table shows the two can differ by nearly ten points with nothing in the flow noticing.
Prohibition 3 deserves its own sentence because it is subtle and respectable. 2 224 µs is a real number, derived correctly in Chapter 23.3 §9, and it is the right answer to "how bad can it get". It is the wrong answer to "what is the p99.99", and the two questions sound the same in a meeting. The distinguishing test is whether the regime has a steady state: if the offered load is below one, the bound is unreachable; if it is at or above one, there is no percentile to ask for.
15. RTL 7 — Performance Telemetry
// ---------------------------------------------------------------------
// perf_telemetry -- the numbers a performance review reads at any
// moment, and six flags that each name a sentence somebody must write.
// ---------------------------------------------------------------------
module perf_telemetry
import perfreview_pkg::*;
(
input logic clk,
input logic rst_n,
input logic clear,
// From the deferral register, Section 5.
input logic [3:0] n_bound,
input logic [3:0] n_deferred,
input logic [3:0] n_defaulted,
input logic [3:0] n_silicon_gated,
// From the percentile estimator and the auditor, Sections 7 and 13.
input logic claim_valid,
input logic claim_supported,
input latency_claim_t claim,
// From the observability bank, Section 11.
input logic [31:0] n_steady,
input logic [31:0] n_overload,
input logic [31:0] watermark_peak_cells,
// Alarm configuration, for prohibition 6.
input logic [19:0] alarm_threshold_ppm,
input logic [19:0] metric_ceiling_ppm,
output logic [19:0] thresholds_disposed_ppm,
output logic [19:0] overload_share_ppm,
output logic [19:0] buffer_peak_share_ppm,
output logic [15:0] c_unsupported_claims,
output logic [15:0] c_claims,
output logic v_threshold_defaulted,
output logic v_claim_without_window,
output logic v_claim_unsupported,
output logic v_percentile_over_mixture,
output logic v_alarm_unreachable,
output logic v_awaiting_silicon
);
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n || clear) begin
c_unsupported_claims <= '0;
c_claims <= '0;
end else if (claim_valid) begin
c_claims <= c_claims + 16'd1;
if (!claim_supported)
c_unsupported_claims <= c_unsupported_claims + 16'd1;
end
end
always_comb begin
thresholds_disposed_ppm =
20'((int'(N_THRESH - n_defaulted) * 1000000) / N_THRESH);
overload_share_ppm = ((n_steady + n_overload) == 0) ? 20'd0
: 20'((longint'(n_overload) * 1000000)
/ longint'(n_steady + n_overload));
buffer_peak_share_ppm =
20'((longint'(watermark_peak_cells) * 1000000)
/ longint'(MAX_CELLS_ONE_PORT));
// Six flags, each naming a specific prohibition of Section 14.
v_threshold_defaulted = (n_defaulted != 4'd0);
v_claim_without_window = claim_valid && (claim.window_samples == 0);
v_claim_unsupported = (c_unsupported_claims != 16'd0);
// A percentile is being quoted over an interval that contained an
// overload event. Prohibition 4.
v_percentile_over_mixture = claim_valid && (n_overload != 32'd0)
&& (n_steady != 32'd0);
// Prohibition 6, and it needs no measurement -- just a comparison.
v_alarm_unreachable = (alarm_threshold_ppm > metric_ceiling_ppm);
v_awaiting_silicon = (n_silicon_gated != 4'd0);
end
endmoduleClassification, Model or Production? PRODUCTION-INTENT. Every input here exists in a modern switch; the outputs are the comparisons nobody wires up.
What it teaches: that v_alarm_unreachable costs one comparator and catches a whole class of failure. Chapter 25.5 §8's occupancy alarm at 90% on a metric that tops out at 80.5% is a check that can never fire, and the design already knows both numbers — the alarm is a register and the ceiling follows from the table's associativity. One comparison at configuration time, and the failure becomes a configuration error rather than a silence lasting the product's life.
Deliberately simplified: metric_ceiling_ppm is an input where a real design derives it from the table's geometry and its load factor, so a change to the associativity silently invalidates the ceiling unless the derivation is in the same place. v_percentile_over_mixture also fires whenever both histograms are non-empty, which is almost always on a busy fabric — the useful form thresholds the overload share, and the threshold is itself a judgement. And buffer_peak_share_ppm uses the single-port maximum as its denominator, which Section 9 already noted is not a constant.
Production implication: the flag to act on first is v_threshold_defaulted, because it is the only one that describes a decision rather than a measurement. It is high exactly when some parameter of Section 4's seven has a value nobody chose, and it is readable from the field — which means a support engineer can distinguish "this deployment is configured wrongly" from "this deployment is configured by nobody" in one register read, and those two conditions have completely different remedies.
16. RTL 8 — The Performance Conformance Monitor
Seven checks. A design that passes all seven has performance figures somebody can act on; one that fails any of them has figures that will be misquoted.
// ---------------------------------------------------------------------
// perf_conformance -- the sign-off gate for Section 2's twenty items.
//
// Same design rule as Chapter 26.2 Section 16: every check must be able
// to fail on a design somebody would really submit, and the gate says
// so about itself.
// ---------------------------------------------------------------------
module perf_conformance
import perfreview_pkg::*;
(
input logic clk,
input logic rst_n,
input logic evaluate,
// Section 5.
input logic [3:0] n_defaulted,
input logic [3:0] n_silicon_gated,
input logic silicon_items_documented,
// Sections 6, 7 and 13.
input logic latency_has_percentile,
input logic latency_has_load,
input logic latency_has_window,
input logic [15:0] n_unsupported_claims,
// Sections 8 and 9.
input logic overload_regime_reported,
input logic [19:0] overload_share_ppm,
// Section 14, prohibition 6.
input logic [19:0] alarm_threshold_ppm,
input logic [19:0] metric_ceiling_ppm,
// Section 10.
input logic [31:0] observability_bce,
input logic observability_budgeted,
output logic c1_thresholds_disposed,
output logic c2_latency_complete,
output logic c3_claims_supported,
output logic c4_regime_separated,
output logic c5_alarm_reachable,
output logic c6_observability_budgeted,
output logic c7_silicon_items_named,
output logic [2:0] n_failures,
output logic conformant,
output logic gate_can_fail
);
logic [2:0] fails;
always_comb begin
// 1. No threshold sits at a value nobody chose.
c1_thresholds_disposed = (n_defaulted == 4'd0);
// 2. Every latency figure carries a percentile, a load and a
// window. Section 1's rule three, as a conjunction.
c2_latency_complete = latency_has_percentile &&
latency_has_load &&
latency_has_window;
// 3. No published claim exceeds what its window supports.
c3_claims_supported = (n_unsupported_claims == 16'd0);
// 4. The two regimes are reported separately, or the share is zero.
c4_regime_separated = overload_regime_reported ||
(overload_share_ppm == 20'd0);
// 5. Every alarm threshold is inside its metric's range.
c5_alarm_reachable = (alarm_threshold_ppm <= metric_ceiling_ppm);
// 6. The instruments are in the area budget rather than added
// during implementation. Section 10 is 0.2569 per cent.
c6_observability_budgeted = observability_budgeted &&
(observability_bce != 32'd0);
// 7. Items that cannot be closed before silicon are NAMED rather
// than guessed. Section 17.
c7_silicon_items_named = (n_silicon_gated == 4'd0) ||
silicon_items_documented;
fails = 3'(!c1_thresholds_disposed) + 3'(!c2_latency_complete)
+ 3'(!c3_claims_supported) + 3'(!c4_regime_separated)
+ 3'(!c5_alarm_reachable) + 3'(!c6_observability_budgeted)
+ 3'(!c7_silicon_items_named);
// A design that supplies no latency claims and no thresholds would
// pass every check above. It is not conformant; it is empty.
gate_can_fail = latency_has_percentile || (n_defaulted != 4'd0) ||
(observability_bce != 32'd0);
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
n_failures <= '0;
conformant <= 1'b0;
end else if (evaluate) begin
n_failures <= fails;
conformant <= (fails == 3'd0) && gate_can_fail;
end
end
endmoduleClassification, Model or Production? A SIGN-OFF GATE.
What it teaches: that check 4's or is the right shape and an and would be wrong. A fabric that never enters overload has nothing to separate, so requiring two histograms on a design that legitimately has one distribution would push teams to fabricate the second. The check asks for separation only when the mixture exists, which is the general form of a gate that does not punish a correct design for being simple. And gate_can_fail repeats Chapter 26.2 §16's defence for the same reason: every check here is a conjunction over supplied quantities, and a design supplying none of them passes all seven.
Deliberately simplified: latency_has_percentile, has_load and has_window are three bits where a real gate parses the performance report and confirms each figure carries all three — which is a document check rather than a hardware one, and the honest implementation runs it in the datasheet flow. metric_ceiling_ppm is a single value for a single alarm where a real design has a dozen alarms with a dozen ceilings, so the check generalises to a loop. And observability_budgeted is a bit that somebody sets, which is exactly the kind of input Chapter 26.1 §10 calls self-witnessed.
Production implication: run checks 1, 5 and 7 at configuration time, not at sign-off. All three are comparisons over constants the design already holds: a threshold's state, an alarm against a ceiling, and whether the silicon-gated items have documentation. They cost three comparators and they convert three of this chapter's failure modes from field escapes into build errors. Checks 2, 3 and 4 need a report, and check 6 needs a floorplan — so the gate naturally splits into a build-time half and a review-time half, and the build-time half is the one that catches the failures with the longest tail.
17. The One Item Only Silicon Can Fill
Six of Section 4's seven thresholds close with a conversation, a query or a document. One does not, and it is worth being precise about why rather than treating it as a harder version of the same problem.
The pause watermark's threshold is the link's dead time, and Chapter 14.2 §9 decomposes it into terms that come from three different places.
| Term | Where the number comes from | Known before silicon? |
|---|---|---|
| the local PHY's transmit latency | the PHY's datasheet | as a specification, with a range |
| propagation, both directions | the cable plant's length | as a range the product supports |
| the partner's receive-to-pause-response latency | a device somebody else made | NO |
| the local receive path to the pause generator | this design's RTL | yes |
Row three is the reason this threshold is different. It is not a number nobody has measured; it is a number that belongs to a device that is not part of this product, and it varies across every partner the product will ever be connected to. No conversation closes it, because there is no single owner to have the conversation with.
And the consequence, from Chapter 14.2 §9's table, is large.
| Link | Dead time | Headroom | Watermark on a 4 096-cell queue |
|---|---|---|---|
| 1 Gb/s, 100 m | 13.32 µs | 13 cells | 4 083 — 99.7% |
| 10 Gb/s, 100 m | 1.78 µs | 18 cells | 4 078 — 99.6% |
| 100 Gb/s, 100 m | 0.63 µs | 62 cells | 4 034 — 98.5% |
| 10 Gb/s, 2 km fibre | 11.2 µs | 110 cells | 3 986 — 97.3% |
| 100 Gb/s, 2 km fibre | 10.6 µs | 1 058 cells | 3 038 — 74.2% |
The watermark spans 99.7% to 74.2% across one product's supported range, and the intuitive default of 75% is safe by a factor of seventy-nine on row one and 34 cells short on row five. On row five the mechanism fails: every pause is issued too late, the queue overflows during the dead time, and the drops look exactly like the drops PAUSE was deployed to prevent.
The honest review entry for this item is not a value. It is: "the watermark is computed from a measured dead time; the register is at address X; the measurement procedure is Y; the default is Z and is correct only for links shorter than L."
Which is a different kind of review output and worth naming as such: an item closed by a PROCEDURE rather than by a number. The review cannot supply the value and can supply everything else — the register, the formula, the measurement method, and the boundary of the default's validity — and a design that ships those four things has deferred correctly in the strongest available sense.
18. What the Review Assumes
Six assumptions, stated because a reader on a different fabric needs to know which numbers survive the journey.
| # | Assumption | If it is false |
|---|---|---|
| 1 | each hop is M/M/1 with the mean frame of Chapter 26.2 §8 | Section 6's whole table moves; the shape of the argument does not |
| 2 | the five hops are independent | correlated queues make the five-hop tail heavier than the sum |
| 3 | arrivals are Poisson | real traffic is burstier, so every percentile here is optimistic |
| 4 | the single-port buffer share is a constant 27.8 MB | a dynamic threshold shrinks it when other ports queue |
| 5 | ten expected exceedances is enough for a percentile | a tighter tolerance needs a larger window and Section 12's costs rise |
| 6 | Chapter 14.2 §9's dead-time decomposition is complete | a fourth term would move Section 17's watermark further |
Assumptions 2 and 3 both push the same way and it is worth saying so plainly: this chapter's percentiles are optimistic.
Assumption 2 first. Five hops treated as independent queues means the five-hop tail is the sum of five independent tails. In a real fabric the hops share traffic — a burst that queues at hop one arrives at hop two as a burst — so the correlation is positive and the true tail is heavier than Section 6's table. How much heavier depends on the topology and the hashing, which is Chapter 23.1's subject rather than this one's.
Assumption 3 is larger. Poisson arrivals are the most benign arrival process with a given mean; real data-centre traffic is bursty at every timescale, and the standard correction is a factor of several on the tail. So 41.941 µs at ρ = 0.95 and p99.99 is a floor on the tail rather than an estimate of it — which means the direction of every conclusion in Sections 6 and 14 is safe and the magnitudes are understated.
Assumption 4 matters for Section 8 rather than Section 6. Chapter 23.3 §9's 108 616 cells is what one port can claim when the rest of the switch is idle. During an incast the other ports are usually not idle, so the real cap is lower and the fill time is shorter than Section 8's table — the asymmetry of 63:1 holds, and both of its terms shrink together.
Assumption 1 is the one a reader is most likely to want to substitute, and the substitution is cheap: the model enters only through μ = 2.4505 × 10⁷ frames per second, so a different frame-size distribution rescales every queueing term by the ratio of the means and leaves the structural floor alone.
Assumption 6 cannot be discharged from inside the chapter, exactly as Chapter 26.2 §18's assumption 6 could not. A fourth dead-time term would make Section 17's 1 058-cell headroom larger and the 75% default worse, so the direction is known and the magnitude is not — and the review's honest form is to name the three terms it has and say the decomposition is the one Chapter 14.2 §9 derived.
19. The Cost, Accounted — in BCE and Percentiles
This chapter prices two different kinds of thing and they do not share a unit. The instruments are silicon and go in BCE. The latency figures are not costs at all — they are measurements, and each one carries a percentile, a load and a window, which is the triple Section 1 established.
The silicon, at 64 ports, against Chapter 23.3's 5.62 × 10⁸ BCE.
| Structure | BCE | × a MAC receive datapath | % of the switch |
|---|---|---|---|
| ingress timestamp, 48 b × 36 stages | 110 592 | 0.39 | 0.0197% |
| latency histogram, steady | 65 536 | 0.23 | 0.0117% |
| latency histogram, overload | 65 536 | 0.23 | 0.0117% |
| per-queue maximum watermark | 16 384 | 0.06 | 0.0029% |
| per-set refusal counter | 131 072 | 0.46 | 0.0233% |
| Chapter 19.7's counters at true widths | 63 488 | 0.22 | 0.0113% |
| dead-time and flap timers | 8 192 | 0.03 | 0.0015% |
| the seven threshold registers | 224 | 0.001 | 0.00004% |
| RECOMMENDED TOTAL | 461 024 | 1.63 | 0.0820% |
| the 1 024-sample window FIFO, NOT recommended | 1 048 576 | 3.70 | 0.1866% |
| total if the FIFO is kept | 1 509 600 | 5.33 | 0.2686% |
The complete measurement apparatus for a 6.4 Tb/s switch — every instrument this chapter argues for — is 1.63 MAC receive datapaths and 0.0820% of the die.
And the reason the recommended total is smaller than Section 10's is Section 12's finding. The window FIFO is 72.6% of Section 10's bill and it is the only instrument in the list that cannot answer the question it was added for — a 1 024-sample window supports p99.902, not p99.99, and reports 5.615 µs against a true 7.475, understated by 24.88%. The histogram answers the same question for 1/16 of the area, and the second histogram for the overload regime costs less than the FIFO's rounding error.
The measurements, which are not costs and are reported as triples.
| Figure | Value | Percentile | Window | Load |
|---|---|---|---|---|
| five-hop structural floor | 4.564 µs | — | — | idle |
| five-hop median | 7.184 µs | p50 | ≥ 20 | ρ = 0.95 |
| five-hop tail | 23.148 µs | p99 | ≥ 1 000 | ρ = 0.95 |
| five-hop deep tail | 41.941 µs | p99.99 | ≥ 100 001 | ρ = 0.95 |
| incast peak, one port | 2 224 µs | — | — | ρ ≥ 1, 64 senders |
| incast fill time | 35.302 µs | — | — | ρ ≥ 1, 64 senders |
Rows one, five and six have no percentile and that is correct, not an omission: a structural floor is a deterministic sum and an incast is an event. Only the middle three are order statistics, and each one names the window it needs.
20. Properties Worth Asserting, and One Worth Refusing
Fifty-two properties in six groups, over the thresholds, the percentiles, the regimes and the instruments.
Group A — the threshold model, Section 3.
// 1. A threshold with an answer is BOUND, whatever else is true.
p_answered_is_bound: assert property (@(posedge clk) disable iff (!rst_n)
(out_valid && $past(owner_answered) && $past(has_value))
|-> (state == TH_BOUND));
// 2. Naming an owner without exposing the parameter does not defer it.
p_named_not_exposed: assert property (@(posedge clk) disable iff (!rst_n)
(out_valid && $past(owner_named) && !$past(is_exposed) &&
$past(has_value) && !$past(owner_answered))
|-> (state == TH_DEFAULT && became_a_default));
// 3. Correct deferral needs both, and is exactly both.
p_deferred_needs_both: assert property (@(posedge clk) disable iff (!rst_n)
(out_valid && deferred_correctly) |-> (state == TH_DEFERRED));
// 4. No value at all is UNSET, which is honest and rare.
p_no_value_unset: assert property (@(posedge clk) disable iff (!rst_n)
(out_valid && !$past(has_value) && !$past(owner_named))
|-> (state == TH_UNSET));
// 5. became_a_default and deferred_correctly are mutually exclusive.
p_default_xor_deferred: assert property (@(posedge clk) disable iff (!rst_n)
out_valid |-> !(became_a_default && deferred_correctly));
// 6. A threshold needing a measurement is owned by silicon, whatever
// its identity. Section 17.
p_measurement_is_silicon: assert property (@(posedge clk) disable iff (!rst_n)
(out_valid && $past(needs_measurement)) |-> (owner == OWN_SILICON));
// 7. The replay window is the one structural member of the seven.
p_replay_structural: assert property (@(posedge clk) disable iff (!rst_n)
(out_valid && $past(th_id) == TH_REPLAY && !$past(needs_measurement))
|-> (owner == OWN_STRUCTURAL));
c_default_seen: cover property (@(posedge clk) out_valid && became_a_default);
c_silicon_seen: cover property (@(posedge clk) out_valid && owner == OWN_SILICON);Group B — the deferral register, Section 5.
// 8. A write only lands if the parameter is exposed. Exposure and
// accountability are one mechanism, not two.
p_write_needs_exposure: assert property (@(posedge clk) disable iff (!rst_n)
(wr_en && !exposed[wr_id]) |-> ##1 (value[$past(wr_id)] ==
$past(value[wr_id])));
// 9. And a landed write marks the threshold answered.
p_write_answers: assert property (@(posedge clk) disable iff (!rst_n)
(wr_en && exposed[wr_id]) |-> ##1 answered[$past(wr_id)]);
// 10. The four counts partition the seven exactly.
p_threshold_partition: assert property (@(posedge clk) disable iff (!rst_n)
(n_bound + n_deferred + n_defaulted + n_silicon_gated == 4'(N_THRESH)));
// 11. Disposed means no defaults, and nothing else.
p_disposed_definition: assert property (@(posedge clk) disable iff (!rst_n)
all_thresholds_disposed == (n_defaulted == 4'd0));
// 12. A silicon-gated threshold is never counted bound until answered.
p_silicon_not_bound: assert property (@(posedge clk) disable iff (!rst_n)
(silicon_gated[0] && !answered[0]) |-> (n_silicon_gated != 4'd0));
// 13. Reset leaves every threshold at its default, which is the state
// the chapter exists to eliminate -- so it must be visible.
p_reset_is_default: assert property (@(posedge clk)
$rose(rst_n) |-> ##1 (n_defaulted != 4'd0 || n_silicon_gated != 4'd0));
c_all_disposed: cover property (@(posedge clk) all_thresholds_disposed);Group C — the percentile estimator, Section 7.
// 14. A claim carries all three fields or it is not a claim.
p_claim_complete: assert property (@(posedge clk) disable iff (!rst_n)
(claim.value_ns != 32'd0) |-> (claim.window_samples != 32'd0));
// 15. supported is exactly the window comparison.
p_supported_definition: assert property (@(posedge clk) disable iff (!rst_n)
(samples_seen != 0) |->
(estimate_is_supported == (samples_seen >= window_required)));
// 16. The finest supportable percentile rises with the window and
// never exceeds the requested one when unsupported.
p_best_below_want: assert property (@(posedge clk) disable iff (!rst_n)
(!estimate_is_supported && samples_seen != 0)
|-> (best_supported_ppm < want_ppm));
// 17. The estimate never exceeds the observed maximum.
p_estimate_le_max: assert property (@(posedge clk) disable iff (!rst_n)
(samples_seen != 0) |-> (estimate_ns <= observed_max_ns ||
estimate_ns == 32'(32'd1 << BUCKET_SHIFT)));
// 18. An empty window supports nothing.
p_empty_unsupported: assert property (@(posedge clk) disable iff (!rst_n)
(samples_seen == 32'd0) |-> !estimate_is_supported);
// 19. A finer percentile never needs a smaller window.
p_window_monotone: assert property (@(posedge clk) disable iff (!rst_n)
(want_ppm > $past(want_ppm)) |-> (window_required >= $past(window_required)));
// 20. The claim's percentile is what was asked for, never what was
// achievable -- the module reports the gap rather than hiding it.
p_claim_reports_request: assert property (@(posedge clk) disable iff (!rst_n)
claim.percentile_ppm == want_ppm);
c_unsupported_claim: cover property (@(posedge clk)
samples_seen != 0 && !estimate_is_supported);
c_supported_claim: cover property (@(posedge clk) estimate_is_supported);Group D — the incast model, Section 9.
// 21. One sender is never an overload. rho = 1 exactly builds no queue.
p_one_sender_no_overload: assert property (@(posedge clk) disable iff (!rst_n)
(n_senders == 8'd1) |-> !regime_is_overload);
// 22. Fill rate is (N-1) line rates, exactly.
p_fill_rate: assert property (@(posedge clk) disable iff (!rst_n)
regime_is_overload |-> (fill_rate_moct_s ==
32'((int'(n_senders) - 1) * int'(LINE_MOCT_S))));
// 23. Drain time does not depend on N. This is Section 8's fixed row.
p_drain_invariant: assert property (@(posedge clk) disable iff (!rst_n)
(drain_time_us == 32'(CAP_OCTETS / LINE_MOCT_S)));
// 24. So the asymmetry is N-1.
p_asymmetry_is_n_minus_1: assert property (@(posedge clk) disable iff (!rst_n)
(regime_is_overload && fill_time_us != 32'd0)
|-> (asymmetry <= 32'(int'(n_senders))));
// 25. Fill time falls as senders rise.
p_fill_monotone: assert property (@(posedge clk) disable iff (!rst_n)
(n_senders > $past(n_senders) && $past(n_senders) > 8'd1)
|-> (fill_time_us <= $past(fill_time_us)));
// 26. Peak queue never exceeds the cap.
p_peak_bounded: assert property (@(posedge clk) disable iff (!rst_n)
(peak_queue_octets <= 32'(CAP_OCTETS)));
// 27. Drops occur exactly when the offer exceeds the cap.
p_drop_definition: assert property (@(posedge clk) disable iff (!rst_n)
buffer_overflows == ((64'(n_senders) * 64'(burst_octets_per_sender))
> 64'(CAP_OCTETS)));
c_incast_64: cover property (@(posedge clk) n_senders == 8'd64);
c_overflow: cover property (@(posedge clk) buffer_overflows);Group E — the observability bank and the window auditor, Sections 11 and 13.
// 28. A sample lands in exactly one histogram.
p_one_histogram: assert property (@(posedge clk) disable iff (!rst_n)
(lat_valid && in_overload) |-> ##1 (n_overload == $past(n_overload) + 32'd1)
&& (n_steady == $past(n_steady)));
// 29. And the other way.
p_steady_histogram: assert property (@(posedge clk) disable iff (!rst_n)
(lat_valid && !in_overload) |-> ##1 (n_steady == $past(n_steady) + 32'd1));
// 30. The watermark never falls except on a read.
p_watermark_monotone: assert property (@(posedge clk) disable iff (!rst_n)
(!rd_en) |-> ##1 (rd_watermark == $past(rd_watermark)));
// 31. A read clears, so consecutive reads are disjoint intervals.
p_read_clears: assert property (@(posedge clk) disable iff (!rst_n)
rd_en |-> ##2 (rd_en && rd_queue == $past(rd_queue, 2))
|-> (rd_watermark <= $past(rd_watermark, 2)));
// 32. Overload share is bounded.
p_share_bounded: assert property (@(posedge clk) disable iff (!rst_n)
(overload_share_ppm <= 20'd1000000));
// 33. The auditor's shortfall is zero exactly when supported.
p_shortfall_zero: assert property (@(posedge clk) disable iff (!rst_n)
(out_valid) |-> (supported == (window_shortfall == 32'd0)));
// 34. And an unsupported claim increments the counter, every time.
p_unsupported_counted: assert property (@(posedge clk) disable iff (!rst_n)
(claim_valid && claim.window_samples < need)
|-> ##1 (n_unsupported == $past(n_unsupported) + 16'd1));
// 35. supportable_ppm is below a million, always -- no window supports
// the hundredth percentile, which is the maximum and not a
// percentile at all.
p_supportable_below_max: assert property (@(posedge clk) disable iff (!rst_n)
(out_valid && claim.window_samples != 32'd0)
|-> (supportable_ppm < 20'd1000000));
c_histogram_claim: cover property (@(posedge clk) claim_valid && from_histogram);
c_both_regimes: cover property (@(posedge clk)
n_steady != 32'd0 && n_overload != 32'd0);Group F — telemetry and the gate, Sections 15 and 16.
// 36. The defaulted flag is exactly the count being non-zero.
p_defaulted_flag: assert property (@(posedge clk) disable iff (!rst_n)
v_threshold_defaulted == (n_defaulted != 4'd0));
// 37. An alarm above its metric's ceiling is flagged. One comparator,
// and it needs no measurement at all.
p_alarm_flag: assert property (@(posedge clk) disable iff (!rst_n)
v_alarm_unreachable == (alarm_threshold_ppm > metric_ceiling_ppm));
// 38. Chapter 25.5 Section 8's case, concretely: 90 per cent on a
// ceiling of 80.5 fires the flag.
p_alarm_805: assert property (@(posedge clk) disable iff (!rst_n)
(alarm_threshold_ppm == 20'd900000 && metric_ceiling_ppm == 20'd805000)
|-> v_alarm_unreachable);
// 39. A claim with no window is flagged before it is audited.
p_no_window_flag: assert property (@(posedge clk) disable iff (!rst_n)
(claim_valid && claim.window_samples == 32'd0) |-> v_claim_without_window);
// 40. The mixture flag needs both regimes present.
p_mixture_needs_both: assert property (@(posedge clk) disable iff (!rst_n)
v_percentile_over_mixture |-> (n_steady != 32'd0 && n_overload != 32'd0));
// 41. Disposed share is a proper fraction of seven.
p_disposed_share: assert property (@(posedge clk) disable iff (!rst_n)
(thresholds_disposed_ppm <= 20'd1000000));
// 42. Gate check 1 fails on any default.
p_c1_fails_on_default: assert property (@(posedge clk) disable iff (!rst_n)
(evaluate && n_defaulted != 4'd0) |-> ##1 !c1_thresholds_disposed);
// 43. Check 2 is a conjunction of all three fields.
p_c2_conjunction: assert property (@(posedge clk) disable iff (!rst_n)
(evaluate && !latency_has_window) |-> ##1 !c2_latency_complete);
// 44. Check 4 passes when there is no mixture to separate.
p_c4_no_mixture: assert property (@(posedge clk) disable iff (!rst_n)
(evaluate && overload_share_ppm == 20'd0) |-> ##1 c4_regime_separated);
// 45. Check 5 is the same comparator as the telemetry flag, inverted.
p_c5_matches_flag: assert property (@(posedge clk) disable iff (!rst_n)
(evaluate) |-> ##1 (c5_alarm_reachable != $past(v_alarm_unreachable)));
// 46. Check 7 passes when the silicon-gated items are documented, and
// when there are none.
p_c7_silicon: assert property (@(posedge clk) disable iff (!rst_n)
(evaluate && n_silicon_gated == 4'd0) |-> ##1 c7_silicon_items_named);
// 47. An empty submission is never conformant.
p_empty_not_conformant: assert property (@(posedge clk) disable iff (!rst_n)
(evaluate && !latency_has_percentile && n_defaulted == 4'd0 &&
observability_bce == 32'd0) |-> ##1 !conformant);
// 48. Conformance requires the gate to be non-trivial.
p_conformant_needs_gate: assert property (@(posedge clk) disable iff (!rst_n)
conformant |-> $past(gate_can_fail));
// 49. And the verdict holds between evaluations.
p_verdict_stable: assert property (@(posedge clk) disable iff (!rst_n)
!evaluate |-> ##1 (conformant == $past(conformant)));
c_gate_conformant: cover property (@(posedge clk) evaluate && conformant);
c_gate_empty: cover property (@(posedge clk)
evaluate && observability_bce == 32'd0);
c_alarm_unreachable: cover property (@(posedge clk) v_alarm_unreachable);
c_awaiting_silicon: cover property (@(posedge clk) v_awaiting_silicon);21. Verification Scenarios
Fifty-eight scenarios in six groups, plus one directed test random stimulus will not produce.
Group 1 — thresholds and their states (10).
| # | Scenario | Expect |
|---|---|---|
| 1 | value, owner named, exposed, answered | TH_BOUND |
| 2 | value, owner named, exposed, unanswered | TH_DEFERRED; deferred_correctly |
| 3 | value, owner named, NOT exposed | TH_DEFAULT; became_a_default — naming without exposing |
| 4 | value, no owner, not exposed | TH_DEFAULT |
| 5 | no value, no owner | TH_UNSET — honest, and rare |
| 6 | the watermark with needs_measurement | OWN_SILICON |
| 7 | the replay window | OWN_STRUCTURAL |
| 8 | the buffer, table, storm, ageing, flap thresholds | OWN_EXTERNAL |
| 9 | a write to an unexposed threshold | the value does not change |
| 10 | a write to an exposed one | value updates; answered high |
Group 2 — the deferral register (9).
| # | Scenario | Expect |
|---|---|---|
| 11 | all seven at defaults | n_defaulted = 7; all_thresholds_disposed low |
| 12 | all seven answered | n_bound = 7; disposed high |
| 13 | six deferred, one silicon-gated | n_deferred = 6; n_silicon_gated = 1; disposed high |
| 14 | the same with the watermark defaulted | n_defaulted = 1; disposed LOW |
| 15 | after reset | every threshold at its default; disposed low |
| 16 | the four counts | always sum to 7 |
| 17 | the watermark answered | moves from n_silicon_gated to n_bound |
| 18 | thresholds_disposed_ppm with 2 defaults | 714 285 ppm |
| 19 | a support engineer's single register read | distinguishes "misconfigured" from "unconfigured" |
Group 3 — percentiles and windows (10).
| # | Scenario | Expect |
|---|---|---|
| 20 | p99 requested, 1 024 samples | supported — 1 000 required |
| 21 | p99.99 requested, 1 024 samples | NOT supported; shortfall 98 977 |
| 22 | the same, reported anyway | 5.615 µs — a p99.902 wearing p99.99's name |
| 23 | the true p99.99 at ρ = 0.95 | 7.475 µs — the claim is 24.88% low |
| 24 | a 256-sample window | 4.484 µs — 40.02% low, and cheaper |
| 25 | a 16 384-sample window | 7.878 µs — 5.39% HIGH, and correctly labelled p99.994 |
| 26 | p99.99 from a 32-bucket histogram at 10⁵ counts | supported — counts, not depth |
| 27 | the same histogram at 10⁴ counts | not supported; hist_p9999_valid low |
| 28 | best_supported_ppm at 1 024 samples | 990 235 ppm — p99.0235, at ten exceedances |
| 29 | an empty window | unsupported; no claim published |
Group 4 — the two regimes (10).
| # | Scenario | Expect |
|---|---|---|
| 30 | ρ = 0.5, five hops, p50 | 4.564 µs — the structural floor exactly |
| 31 | ρ = 0.95, p50 | 7.184 µs — 1.57× the floor |
| 32 | ρ = 0.95, p99 | 23.148 µs — 5.07× |
| 33 | ρ = 0.95, p99.99 | 41.941 µs — 9.19× |
| 34 | ρ = 0.99, p99.99 | 192.287 µs — 42.1× |
| 35 | P(W > 2 224 µs) at ρ = 0.99 | 2.04 × 10⁻²³⁷ — not a tail |
| 36 | 2 senders into one port | fill 2 224 µs; drain 2 224 µs; 1:1 |
| 37 | 64 senders | fill 35.302 µs; drain 2 224 µs; 63:1 |
| 38 | 1 sender | regime_is_overload low — no queue builds |
| 39 | a p99 computed across an incast | v_percentile_over_mixture |
Group 5 — the instruments (10).
| # | Scenario | Expect |
|---|---|---|
| 40 | a latency sample in overload | lands in hist_overload only |
| 41 | a sample in steady state | hist_steady only |
| 42 | occupancy 1 000 then 400 cells | watermark holds 1 000 |
| 43 | a management read | returns 1 000, then clears to 0 |
| 44 | the next interval peaking at 400 | returns 400 — a per-interval peak |
| 45 | a running maximum instead | returns 1 000 forever — the failure read-and-clear fixes |
| 46 | a 35.302 µs peak against a 1 s poll | invisible to a sampled occupancy; caught by the watermark |
| 47 | the full bill at 64 ports | 1 443 840 BCE; 5.10 datapaths; 0.2569% |
| 48 | the bill without the window FIFO | 461 024 BCE; 1.63 datapaths; 0.0820% |
| 49 | a 100 001-sample FIFO at 64 ports | 1.024 × 10⁸ BCE — 18.221% of the switch |
Group 6 — telemetry and the gate (9).
| # | Scenario | Expect |
|---|---|---|
| 50 | alarm 90%, ceiling 80.5% | v_alarm_unreachable; check 5 fails |
| 51 | alarm 70%, ceiling 80.5% | clear |
| 52 | a latency figure with no percentile | check 2 fails |
| 53 | a figure with a percentile and no window | check 2 fails; v_claim_without_window |
| 54 | overload share 0, one histogram | check 4 PASSES — nothing to separate |
| 55 | overload share 41 000 ppm, one histogram | check 4 fails |
| 56 | one threshold defaulted | check 1 fails |
| 57 | a submission with no claims and no thresholds | gate_can_fail low; conformant LOW |
| 58 | all seven checks clear on a real design | conformant high |
22. Debugging a Performance Number That Met Its Target
Six symptoms, and all six begin with a figure that passed.
| Symptom | First question | Where to look |
|---|---|---|
| an application missing deadlines on a fabric inside its latency budget | which percentile was the budget? | Section 6 — the floor and the p99.99 differ by 9.19× |
| a tail latency that never varies between releases | how deep is the window? | Section 12 — a 1 024-sample window reports its own maximum |
| a p99 that tripled with no design change | what was the overload share? | Section 21's directed test — 41 000 ppm |
| drops on the longest links only | is the pause watermark a default? | Section 17 — 75% is 34 cells short at 100 Gb/s over 2 km |
| an occupancy alarm that has never fired | can the metric reach it? | Chapter 25.5 §8 — 90% on a ceiling of 80.5% |
| a parameter the field cannot change | is it exposed or documented? | Section 4 — a PDF is not a register |
Row two has a signature worth memorising. A tail-latency figure that is suspiciously stable across releases, loads and traffic profiles is usually not a stable design — it is an estimator pinned at its window's ceiling. The test is one register read: compare the reported percentile against the window length, and if the window cannot support the percentile, the number has been the same since the histogram was wired up.
Row four has a procedure and it takes ten minutes. Read the watermark register; if it is the reset default, read the link's length and rate and compute Chapter 14.2 §9's headroom. On a 100 Gb/s 2 km link the requirement is 1 058 cells of a 4 096-cell queue and the 75% default reserves 1 024 — and the 34-cell shortfall is the entire fault.
23. Misconceptions
Six, in the wrong-model / what-it-costs / corrected-model form this track has used since Chapter 1.1.
Misconception 1 — "The latency is 4.564 microseconds."
Wrong model. A design has a latency, it was computed in the architecture review, and it is in the datasheet.
What it costs. Section 6's table: the same five hops are 4.564 µs, 7.184, 23.148 or 41.941 depending on two numbers nobody supplied — the offered load and the percentile. Against a 5 µs budget the design passes with 8.7% margin or fails by 8.39×, and both readings are defensible. The cost is that the first one is computed in month two and propagates everywhere, and the fourth is computed by a customer.
Corrected model. A latency figure is a triple: value, percentile, window — and a fourth field, the offered load, for anything above the floor. Chapter 26.1 §8 can produce the floor and nothing else, which is why it flagged the gap rather than filling it, and this chapter fills it with three numbers rather than one.
Misconception 2 — "The buffer holds 2 224 microseconds, so that is the worst-case latency."
Wrong model. The deepest queue times the drain rate is the tail.
What it costs. Section 8: at ρ = 0.99 the probability of a 2 224 µs wait in steady state is 2.04 × 10⁻²³⁷. Treating the bound as a tail makes every percentile look absurd and the usual response is to stop computing percentiles at all — which is how a design ends up with one latency number again. The bound is correct and it belongs to a regime with no steady state.
Corrected model. Two regimes, two kinds of number. Steady state, ρ < 1: a distribution, so a percentile. Incast, ρ ≥ 1: an event, so a fill time and a recovery time — 35.302 µs and 2 224 µs at 64 senders, an asymmetry of 63:1. A review needs both and they are not comparable.
Misconception 3 — "Our p99.99 is 5.6 microseconds."
Wrong model. The histogram reports a p99.99 and the register says 5 615 ns.
What it costs. The window is 1 024 samples and the highest percentile it can support is 99.902%, so the number is the window's maximum wearing a finer statistic's name — understated by 24.88% against a true 7.475 µs. And the incentive gradient runs the wrong way: a 256-sample window reports 4.484 µs, so every pressure toward a cheaper instrument improves the reported figure.
Corrected model. A percentile's name is a claim about the sample it came from, and the claim is checkable in one divide. Either report the percentile the window supports — p99.902, the same number honestly labelled — or use a structure whose limit is its accumulated count rather than its depth. A 32-bucket log histogram supports any percentile at 0.23 of a datapath, which is sixteen times cheaper than the FIFO that cannot.
Misconception 4 — "The p99 regressed, so something in the design got slower."
Wrong model. Latency is a property of the switch, so a change in latency is a change in the switch.
What it costs. Section 21's directed test: two traffic profiles at identical mean load, identical frame sizes and identical sample counts produce 23.1 µs and 61.4 µs — a factor of 2.66 with the design untouched. The cost is the investigation, which goes to the scheduler and the buffer allocation and finds nothing, because 4.1% of the interval was a different regime.
Corrected model. A percentile over a mixture is a percentile of neither population. Keep two histograms, switched by the overload bit, and report the share alongside every figure. The steady-state p99 in both profiles is 23.1 µs, and the second histogram that shows it costs 0.0117% of the die.
Misconception 5 — "That threshold is documented, so the deployment can set it."
Wrong model. The default is in the datasheet with an explanation, which is what documenting a parameter means.
What it costs. Chapter 25.2 §8's 300-second ageing default is 8× too long at 1 000 container teardowns an hour. Documented, correcting it needs silicon. Exposed, it is one register write. Four of this chapter's opening defaults are in this state, and each one has a deployment where it fails.
Corrected model. Exposure is a register address, and the review's item is that address. Chapter 26.1 §12's DEFERRED outcome requires two things and only one of them is a sentence: name the owner, and give them somewhere to write the answer. A parameter with an owner and no register is still a default.
Misconception 6 — "We cannot afford to instrument all of that."
Wrong model. Counters, histograms and timestamps are a real area cost that has to be traded against function.
What it costs. Nothing, and that is the finding. Section 10's complete apparatus is 1 443 840 BCE — 5.10 MAC receive datapaths, 0.2569% of Chapter 23.3's switch, and the recommended subset is 1.63 datapaths and 0.0820%. A proposal to double the MAC table costs 44 datapaths and is routinely approved. Trimming the packet buffer by 0.282% pays for the whole bill.
Corrected model. Price observability against the buffer, not against the datapath. The habit of comparing every structure to a MAC receive datapath is right at MAC scale and wrong at switch scale by three orders of magnitude, and it is why the observability conversation is always conducted in adjectives — nobody had derived the number, so there was nothing to say.
24. Interview Questions
Six, with what a strong answer contains.
1. What is the latency of your switch?
A strong answer asks which regime and then gives three numbers. 4.564 µs structural over five hops; 41.941 µs at ρ = 0.95, p99.99; and 2 224 µs after a 35.302 µs fill under a 64-way incast. The strongest part is naming the third, because an all-reduce is an N-way incast by construction and in an AI fabric that regime is the workload rather than an edge case.
2. Your tail latency has not moved in six months. Good or bad?
Suspicious. A tail figure that is stable across releases, loads and traffic profiles is usually an estimator pinned at its window's ceiling. The check is one comparison: the reported percentile against the window length. A 1 024-sample window supports p99.902 and reports 5.615 µs whatever the design does, so the number was fixed the day the histogram was wired up.
3. The buffer holds 2 224 microseconds. What percentile is that?
None — and the answer is the point. At ρ = 0.99 the steady-state probability of that wait is 2.04 × 10⁻²³⁷. It is a bound reachable only at ρ ≥ 1, which is an incast and has no steady state, so it has a fill time and a recovery time instead of a percentile. A candidate who names the regime rather than a number has understood the distinction the whole of Section 8 is about.
4. Which of your thresholds can your design not set?
Seven, and a strong answer names the owners. Buffer size, table size and storm control belong to the deployment; ageing to the orchestrator; the replay window to the topology; the flap threshold to the site; and the pause watermark to nobody, because its dead time includes the partner's response latency. The strongest half is the last one: six close with a conversation and one closes with a measurement at bring-up, and the correct deliverable for that one is a procedure and a validity boundary rather than a value.
5. Why is a 90% occupancy alarm on a four-way table wrong?
Because the table's achievable occupancy is 80.5%. Chapter 25.5 §6: offered 131 072 addresses, a 128k four-way table stores 105 465 and refuses 25 607 — 19.537% — while the occupancy reads 80.5%. The alarm cannot fire and its silence reads as health. The repair costs one comparator at configuration time and needs no measurement at all.
6. How much die would you spend on performance instrumentation?
The arithmetic, not an opinion. The complete bill at 64 ports is 1 443 840 BCE, 5.10 MAC datapaths, 0.2569% of a 5.62 × 10⁸ BCE switch; the recommended subset, after deleting the window FIFO that cannot report the percentile it was added for, is 1.63 datapaths and 0.0820%. Against a packet buffer that is 91.0% of the die, a 0.282% trim pays for all of it — and the right comparison is against the buffer, because comparing against a MAC datapath is a habit from a different scale.
25. Questions and Answers
26. What's Next
This chapter closed the deferrals Chapter 26.1 §12 opened and put the percentile back into a latency budget that had only a floor.
Four results worth carrying forward.
| Result | Value | Why it travels |
|---|---|---|
| the percentile spread at one load | 5.84× from p50 to p99.99 at ρ = 0.95 | it is larger than the spread across loads, so the percentile is the axis to specify |
| a bound is not a tail | P(W > 2 224 µs) at ρ = 0.99 is 2.04 × 10⁻²³⁷ | the test is whether the regime has a steady state |
| a window's ceiling | 1 − 1/W, and 1 024 gives p99.902 | it is arithmetic, not a property of any design |
| the observability bill | 1.63 datapaths, 0.0820% of a 6.4 Tb/s switch | it ends the adjective conversation |
And one deliverable that is not a number. Section 17's watermark item is closed by a register, a formula, a measurement procedure and a validity boundary — an item that only silicon can fill, handled by publishing everything except the value.
What remains of Module 26, and of the track.
| Next | What it adds |
|---|---|
| Chapter 26.4 — How Ethernet Is Actually Probed | the whole of it as an interview subject: what is being tested behind each question, what the track's 124 rejected-property classes have in common, and which single question separates a candidate who has the model from one who has the vocabulary |
The sentence this chapter ends on is the track's, with a third edge added. Chapter 25.6 §20 and Chapter 26.1 §21 reached it and Chapter 26.2 §20 sharpened it: a check is worth exactly the set of observations that would have refuted it. Section 20 adds that the set can be non-empty, correctly sized and drawn from a sample that does not contain the quantity being checked — so a refutation set has to be a set of observations that could occur, and a p99.99 in a 1 024-sample window is a demand for an observation the instrument cannot make.
Continue learning
Related tutorials
- Related topic
Full Link Bring-Up Sequence
Eight stages from power-on to a legally transmitting MAC, and the ordering is a data dependency. Each stage measures what the next assumes, and a stage run early does not fail — it succeeds against garbage.
- Related topic
Packet Capture Analysis
A capture reaches 14 of Module 21's 57 faults, holds 84 milliseconds per gigabyte, and drops 12% of them itself — and it is still the only instrument that shows relationships between frames.
- Related topic
"Switches Eliminate Broadcasts"
One flooding port takes 98.4% of a 64-port switch's frame budget, a 128k table refuses 25 607 of 131 072 addresses, and the occupancy counter reads 80.5% throughout.
- Related topic
Architecture and Integration Review
Twenty review items, each a number somebody has to produce and a chapter that derives it — with a readiness model, a budget reconciler and a claims auditor that make the review itself able to fail.
Standards & specifications
- Governing standard
- IEEE Std 802.3 (Ethernet)(opens IEEE in a new tab)
Defines the Ethernet MAC, the media-independent interfaces and the physical-layer sublayers, including framing, access control, auto-negotiation and per-rate PHY specifications. VLAN tagging, priority and time-sensitive shaping are defined by IEEE 802.1, not by 802.3.
This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.
Where this fits
Part of the Ethernet curriculum.
