Ethernet · Module 24
Ethernet against Proprietary Fabrics
A framing tax worth 0.111% on the traffic these fabrics carry, a lookup worth 137.5 ns across five hops, 8.9% of a die — and twenty-nine chapters of ecosystem to rebuild.
Every chapter of Module 23 answered this chapter's question implicitly, by staying inside a standard. This chapter asks what a design gets by leaving one, and the honest answer is smaller and stranger than the marketing.
The framing tax is the argument everyone reaches for first, so derive it first and get it out of the way.
| At a 64-octet frame — 84 octets on the wire | Octets | Share of the wire |
|---|---|---|
| preamble and start delimiter — Chapter 5.2 | 8 | 9.5238% |
| interframe gap — Chapter 5.9 | 12 | 14.2857% |
| destination and source addresses — Chapter 5.3 | 12 | 14.2857% |
| frame check sequence — Chapter 5.8 | 4 | 4.7619% |
| EtherType — Chapter 5.5 | 2 | 2.3810% |
| total | 38 | 45.2381% |
And at the other end of Chapter 8.3 §2's curve.
| At a 9 000-octet payload — 9 038 octets on the wire | Octets | Share of the wire |
|---|---|---|
| preamble and start delimiter | 8 | 0.0885% |
| interframe gap | 12 | 0.1328% |
| addresses | 12 | 0.1328% |
| frame check sequence | 4 | 0.0443% |
| EtherType | 2 | 0.0221% |
| total | 38 | 0.4204% |
A fabric that deleted every octet of Ethernet's framing would gain 45.24% at minimum frame size and 0.4204% at jumbo — a ratio of 107.6 — and the fabrics that actually leave the standard are the ones carrying large transfers.
So they leave for four tenths of a per cent. Which means the win, if there is one, is somewhere else — and the rest of this chapter goes to find it.
1. Scope — What Leaving Actually Buys and What It Obliges
This chapter owns four derivations and one conclusion.
| What is derived | |
|---|---|
| Sections 2 to 3 | the framing tax, at both ends of the curve, and what deleting it is worth |
| Sections 4 to 7 | the lookup, which is where the real win is: 30 ns per hop against 2.5 |
| Sections 8 to 11 | what a proprietary fabric must BUILD to replace the ecosystem it left |
| Sections 12 to 13 | what it cannot delete — the buffer and the SerDes, which are 91% of the die |
What this chapter does not own. It names no vendor and models no specific fabric. Every proprietary interconnect is different and all of them make the same four decisions, which is what makes the comparison possible at all: source routing instead of a lookup, a shorter header, a credit or token link layer, and a controlled endpoint population.
It also does not argue that leaving is wrong. Several of the most successful interconnects in the industry left, deliberately, with clear eyes. The argument is that the win is real and it is not in the unit people quote it in — which is Chapter 23.6 §19's closing thesis arriving in a commercial setting rather than a physical one.
2. The Framing Tax, and What Deleting It Is Worth
Take the table apart field by field and ask what a fabric that deleted each one would actually save, because three of the five cannot be deleted at all.
| Field | Octets | Can a proprietary fabric delete it? |
|---|---|---|
| preamble and SFD | 8 | NO — a receiver's clock recovery needs a known pattern |
| interframe gap | 12 | partly — it can be shortened, not removed |
| addresses | 12 | YES — replaced by a route or a short endpoint id |
| frame check sequence | 4 | NO — a corrupted frame must be detectable |
| EtherType | 2 | YES — if the fabric carries one payload type |
Rows one and four are the ones people assume are free and are not.
The preamble is not decoration. Chapter 5.2 §3 established what it is for: a receiver arriving at an idle line must lock its clock recovery and find the frame's first bit, and both need a known pattern. A fabric that deletes it must either run a continuously-scrambled line with block synchronisation — which is what Chapter 3.5's 64B/66B already does above 10 Gb/s — or reintroduce a preamble under another name.
And in fact modern Ethernet already did this. Above 10 Gb/s the eight preamble octets are a legacy the PCS no longer needs for synchronisation; they persist because the MAC interface was defined around them. So a proprietary fabric's preamble saving is real, small, and the same saving 802.3 could have taken and did not, for compatibility.
The frame check sequence cannot go. Chapter 5.8 §2: a 32-bit CRC detects all burst errors up to 32 bits and all single, double and odd-weight errors. A fabric without one delivers corrupted data silently, and every fabric anybody trusts has an equivalent — PCIe has an LCRC, InfiniBand has both a variant and an invariant CRC. The field does not disappear; it is renamed.
So the deletable tax is smaller than the total.
| Total tax | Deletable | Actually saved | |
|---|---|---|---|
| 64-octet frame | 45.24% | addresses 12 + EtherType 2 = 14 octets | 16.67% |
| 1 518-octet frame | 2.47% | the same 14 octets | 0.91% |
| 9 018-octet frame | 0.42% | the same 14 octets | 0.155% |
The honest framing saving on jumbo traffic is 0.155% of the wire, and that is the number a proprietary fabric's framing argument is actually worth.
One qualification in the fabric's favour, and it is worth stating because it is the strongest version of the argument. A fabric that knows its endpoint population can use a short route field instead of twelve address octets — Section 4 derives 30 bits for five hops of 64-port switches — so the replacement is smaller rather than absent. Four octets against twelve is an eight-octet saving, and at 64 octets that is 9.52% of the wire, which is the largest honest framing number in this chapter.
3. RTL 1 — The Fabric Package and the Framing-Tax Model
// ---------------------------------------------------------------------
// propfab_pkg -- constants for comparing a standard frame format
// against a fabric that has left the standard.
//
// Unit: Chapter 23.3 Section 2's bitcell equivalent.
// 1 BCE = one bit of usable on-die SRAM = 0.35 GE
// 1 flip-flop = 20 BCE
// Chapter 19.7 Section 19's MAC receive datapath = 283 320 BCE
// Chapter 23.3's 64-port 6.4 Tb/s switch = 5.62e8 BCE
// ---------------------------------------------------------------------
package propfab_pkg;
localparam int unsigned DATAPATH_BCE = 283_320;
localparam int unsigned BCE_PER_FLOP = 20;
localparam int unsigned SWITCH_PORTS = 64;
// ---- the Ethernet frame's fixed octets -------------------------------
localparam int unsigned OCT_PREAMBLE = 8; // Chapter 5.2
localparam int unsigned OCT_IFG = 12; // Chapter 5.9
localparam int unsigned OCT_ADDRESSES = 12; // Chapter 5.3
localparam int unsigned OCT_FCS = 4; // Chapter 5.8
localparam int unsigned OCT_ETHERTYPE = 2; // Chapter 5.5
localparam int unsigned OCT_FIXED = OCT_PREAMBLE + OCT_IFG
+ OCT_ADDRESSES + OCT_FCS
+ OCT_ETHERTYPE; // 38
// Which of them a proprietary fabric can actually delete. The
// preamble and the FCS cannot go; the gap can only be shortened.
localparam int unsigned OCT_DELETABLE = OCT_ADDRESSES + OCT_ETHERTYPE;
// ---- Chapter 23.3's forwarding state, in BCE -------------------------
// The three arrays a source-routed fabric does not need.
localparam int unsigned ACL_TCAM_BCE_E4 = 2_100; // 2.10e7 / 1e4
localparam int unsigned L3_LPM_BCE_E4 = 1_680; // 1.68e7
localparam int unsigned MAC_TBL_BCE_E4 = 1_260; // 1.26e7
localparam int unsigned SWITCH_BCE_E4 = 56_200; // 5.62e8
// ---- the pipeline ----------------------------------------------------
localparam int unsigned LOOKUP_STAGES = 36; // Chapter 23.3 Sec 2
localparam int unsigned SR_STAGES = 3; // shift and demux
localparam int unsigned CORE_MHZ = 1_200;
typedef enum logic [1:0] {
ADDRESSING_NAME = 2'd0, // a 48-bit name, looked up
ADDRESSING_ROUTE = 2'd1, // a port index per hop
ADDRESSING_ID = 2'd2 // a short endpoint identifier
} addressing_e;
// ---- derived: framing ------------------------------------------------
function automatic int unsigned wire_octets(int unsigned payload);
// payload + 14 header + 4 FCS + 8 preamble + 12 gap
return payload + OCT_FIXED;
endfunction
function automatic int unsigned tax_ppm(int unsigned payload);
return (OCT_FIXED * 1_000_000) / wire_octets(payload);
endfunction
function automatic int unsigned deletable_ppm(int unsigned payload);
return (OCT_DELETABLE * 1_000_000) / wire_octets(payload);
endfunction
// ---- derived: a source route ------------------------------------------
function automatic int unsigned bits_per_hop(int unsigned radix);
int unsigned b; int unsigned n;
b = 0; n = radix - 1;
while (n > 0) begin b = b + 1; n = n >> 1; end
return b;
endfunction
function automatic int unsigned route_bits(int unsigned radix,
int unsigned hops);
return bits_per_hop(radix) * hops;
endfunction
// ---- derived: latency -------------------------------------------------
function automatic int unsigned stage_ns_x100(int unsigned stages);
return (stages * 100_000) / CORE_MHZ;
endfunction
function automatic int unsigned datapaths_milli(int unsigned bce);
return (bce / DATAPATH_BCE) * 1000
+ (((bce % DATAPATH_BCE) * 1000) / DATAPATH_BCE);
endfunction
endpackage// ---------------------------------------------------------------------
// framing_tax_model -- what a frame format costs at a given payload,
// and how much of that a fabric can actually escape.
//
// The output that matters is undeletable_ppm: the preamble and the
// check sequence survive any protocol change, so the saving is always
// smaller than the tax and usually by a factor of nearly three.
// ---------------------------------------------------------------------
module framing_tax_model
import propfab_pkg::*;
(
input logic clk,
input logic rst_n,
input logic [15:0] payload_octets,
input logic [15:0] rate_gbps,
input logic [1:0] addressing, // addressing_e
input logic [15:0] route_octets, // if a route replaces addresses
output logic [31:0] wire_octets_o,
output logic [31:0] tax_ppm_o,
output logic [31:0] deletable_ppm_o,
output logic [31:0] undeletable_ppm,
output logic [31:0] actual_saving_ppm,
output logic [31:0] packet_ns_x100,
output logic [31:0] packets_per_s_k,
output logic preamble_is_deletable,
output logic fcs_is_deletable,
output logic [31:0] c_frames_modelled
);
always_comb begin
wire_octets_o = 32'(wire_octets(int'(payload_octets)));
tax_ppm_o = 32'(tax_ppm(int'(payload_octets)));
deletable_ppm_o = 32'(deletable_ppm(int'(payload_octets)));
undeletable_ppm = tax_ppm_o - deletable_ppm_o;
// A route REPLACES the addresses rather than removing them, so the
// saving is the difference, not the whole field.
actual_saving_ppm =
(addressing == 2'(ADDRESSING_NAME)) ? 32'd0
: ((32'(OCT_DELETABLE) > 32'(route_octets))
? (((32'(OCT_DELETABLE) - 32'(route_octets)) * 1_000_000)
/ wire_octets_o)
: 32'd0);
// Chapter 8.3's other quantity: the frame RATE, which is what
// decides the pipeline width rather than the efficiency.
packet_ns_x100 = (rate_gbps == 0) ? 32'd0
: (wire_octets_o * 8 * 100) / 32'(rate_gbps);
packets_per_s_k = (packet_ns_x100 == 0) ? 32'd0
: (100_000_000 / packet_ns_x100);
// The two that survive any protocol change.
preamble_is_deletable = 1'b0;
fcs_is_deletable = 1'b0;
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) c_frames_modelled <= '0;
else c_frames_modelled <= c_frames_modelled + 32'd1;
end
endmoduleClassification: a division, reported in parts per million so the jumbo case does not round to zero.
What it teaches: that actual_saving_ppm is the number the framing argument is worth and it is not tax_ppm_o. At a 64-octet frame with a four-octet route replacing fourteen octets of address and type, the module reports 119 048 ppm — 11.90%. At 9 018 octets it reports 1 106 ppm — 0.111%. The tax is 45.24% and 0.42% respectively, so the saving is 26.3% of the tax at both ends, because the deletable fraction is a constant share of a constant.
And it teaches that packets_per_s_k is the quantity a fabric designer should be arguing about. At 400 Gb/s the module reports 595 238 thousand packets per second at 64 octets and 5 532 at 9 018 — a factor of 107.6 — and it is the packet rate that decided Chapter 23.3 §2's eight-wide pipeline, which is real silicon. The efficiency is a percentage; the packet rate is a pipeline.
Deliberately simplified: the model treats the interframe gap as fully present, where a fabric with tighter clock tolerance can shorten it — Chapter 4.4 §4's deletion rate is what bounds how far. route_octets is a constant where a real source route's length varies with the path's hop count, so a fabric with mixed path lengths has a variable-length header, which is a parser cost the model does not charge. And the preamble is charged at eight octets even above 10 Gb/s, where the PCS no longer needs it for synchronisation and it survives as a MAC-interface convention.
Production implication: a variable-length route header is the detail that turns a clean idea into a parser problem. Chapter 19.2 §6 established that a parser's cost is set by how far into the frame a field's position depends on the frame's own content, and a route whose length is the hop count is exactly that: the payload's offset is not knowable until the route has been read. The usual fix is a fixed-width route field sized for the worst path — 30 bits for five hops of 64-port switches, rounded to four octets — which wastes bits on short paths and is what nearly every such fabric does, for the same reason Ethernet's addresses are a fixed six octets.
4. The Lookup, Which Is Where the Win Actually Is
Section 2 found the framing saving to be a tenth of a per cent on the traffic that matters. The lookup is a different order of magnitude, and it is the thing worth leaving a standard for.
Chapter 23.3 §2 derived the forwarding pipeline's depth from the packet rate: 36 stages at 1.2 GHz, which is 30 ns and is exactly Chapter 23.1 §4's lookup term. Those thirty-six stages do a specific job.
| Stage group | What it does | Why it is deep |
|---|---|---|
| parse | locate the fields, which Chapter 13.2's tag can move | the offsets depend on the content |
| MAC lookup | hash, probe, resolve collisions — Chapter 12.5 | a 128k-entry table is an SRAM access plus comparison |
| L3 lookup | longest-prefix match | several dependent memory accesses |
| ACL | a TCAM match against 16k rules | wide, and the result must be resolved by priority |
| resolve and queue | combine the results, pick an egress queue | the outputs disagree and a policy orders them |
Now the source-routed alternative. A fabric whose endpoints are enumerated by one authority can put the path in the header: one port index per hop.
a 64-port switch needs 6 bits to name an egress port
5 hops x 6 bits = 30 bits of routeAnd the switch's work becomes a shift and a demultiplex.
| Named forwarding | Source routed | |
|---|---|---|
| what the switch does | parse, hash, probe, match, resolve | take the low 6 bits, shift the field right |
| pipeline stages | 36 | 3 |
| at 1.2 GHz | 30.00 ns | 2.50 ns |
| over 5 hops | 150.0 ns | 12.5 ns |
| saving over 5 hops | — | 137.5 ns |
Twelve times the per-hop latency, and 137.5 nanoseconds saved across a five-stage fabric — which is a real number on a real workload, unlike the framing.
Put that beside the traffic the fabric carries and it becomes a decision rather than a fact.
| Workload | Transfer | Fabric latency at 5 hops | Does 137.5 ns matter? |
|---|---|---|---|
| Chapter 23.2's all-reduce | 15.984 GB per participant | 150 ns of 1.279 s ideal | no — 0.00001% |
| a 4 kB storage read | 4 096 octets | 150 ns of ~100 µs | no — 0.15% |
| a synchronisation primitive | 8 octets, and everybody waits | 150 ns of ~300 ns | YES — nearly half |
| a remote atomic | 8 octets, round trip | 300 ns of ~500 ns | YES |
Rows three and four are the whole case for a proprietary fabric and they are a small, specific, real case. A fabric that exists to make short synchronising operations fast has a genuine 12× lever that Ethernet cannot offer, because Ethernet's addressing model requires a lookup by construction — Chapter 24.1 §2 established that a name must be resolved and a position need not.
5. RTL 2 — The Source-Route Decoder
// ---------------------------------------------------------------------
// source_route_decoder -- the whole forwarding decision of a
// proprietary fabric, in three pipeline stages.
//
// Set this against Chapter 23.3's thirty-six. The module is short
// because that IS the comparison: the decision is a shift.
// ---------------------------------------------------------------------
module source_route_decoder
import propfab_pkg::*;
#(
parameter int unsigned RADIX = 64,
parameter int unsigned MAX_HOPS = 5,
parameter int unsigned ROUTE_BITS = 30
)(
input logic clk,
input logic rst_n,
input logic hdr_valid,
input logic [ROUTE_BITS-1:0] route_in,
input logic [3:0] hops_remaining_in,
output logic egress_valid,
output logic [5:0] egress_port,
output logic [ROUTE_BITS-1:0] route_out,
output logic [3:0] hops_remaining_out,
output logic route_exhausted,
output logic lookup_needed,
output logic [31:0] stages_used,
output logic [31:0] latency_ns_x100,
output logic [31:0] named_latency_ns_x100,
output logic [31:0] latency_ratio_x10,
output logic [31:0] c_hops_decoded,
output logic [31:0] c_exhausted
);
localparam int unsigned HOP_BITS = 6; // bits_per_hop(64)
// Stage 1: register the header. Stage 2: select. Stage 3: shift.
logic [ROUTE_BITS-1:0] route_s1, route_s2;
logic [3:0] hops_s1, hops_s2;
logic valid_s1, valid_s2;
always_comb begin
stages_used = 32'(SR_STAGES);
latency_ns_x100 = 32'(stage_ns_x100(int'(SR_STAGES)));
named_latency_ns_x100 = 32'(stage_ns_x100(int'(LOOKUP_STAGES)));
latency_ratio_x10 = (latency_ns_x100 == 0) ? 32'd0
: (named_latency_ns_x100 * 32'd10)
/ latency_ns_x100;
// THE comparison, as a constant. A source-routed fabric performs
// no lookup at all, so no table can miss and none needs to exist.
lookup_needed = 1'b0;
route_exhausted = valid_s2 && (hops_s2 == 4'd0);
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
route_s1 <= '0; route_s2 <= '0; hops_s1 <= '0; hops_s2 <= '0;
valid_s1 <= 1'b0; valid_s2 <= 1'b0;
egress_valid <= 1'b0; egress_port <= '0; route_out <= '0;
hops_remaining_out <= '0;
c_hops_decoded <= '0; c_exhausted <= '0;
end else begin
// Stage 1 -- capture.
valid_s1 <= hdr_valid;
route_s1 <= route_in;
hops_s1 <= hops_remaining_in;
// Stage 2 -- select this hop's port from the low bits.
valid_s2 <= valid_s1;
route_s2 <= route_s1;
hops_s2 <= hops_s1;
// Stage 3 -- emit and shift the route down by one hop.
egress_valid <= valid_s2 && (hops_s2 != 4'd0);
egress_port <= route_s2[HOP_BITS-1:0];
route_out <= route_s2 >> HOP_BITS;
hops_remaining_out <= (hops_s2 == 4'd0) ? 4'd0 : (hops_s2 - 4'd1);
if (valid_s2 && (hops_s2 != 4'd0))
c_hops_decoded <= c_hops_decoded + 32'd1;
if (valid_s2 && (hops_s2 == 4'd0))
c_exhausted <= c_exhausted + 32'd1;
end
end
endmoduleClassification: three registers and a shift, standing in for thirty-six stages of lookup.
What it teaches: that latency_ratio_x10 reports 120 — twelve times — and that the whole difference is the absence of a table. The module contains no memory, no hash, no comparison against stored state and no miss path, because a route is not resolved against anything. Chapter 24.1 §2 made the same observation about PCIe's decode and this is its extreme form: a source route does not even decode, it indexes.
And it teaches that lookup_needed being zero removes an entire failure class rather than a structure. No table means no miss, no flood, no aging, no capacity limit and no thrash — Chapter 12.5 §7's entire chapter has no subject here. What replaces it is a route computed by somebody else, before the packet existed, and that somebody is the fabric manager.
Deliberately simplified: the route field is fixed at 30 bits for five hops of 64-port switches, so a fabric with deeper paths or wider switches needs a different constant — and a variable-length route is the parser problem Section 3's production note described. There is no error path: a route naming a port that is down, or a route whose hop count disagrees with the actual topology, produces a packet sent somewhere wrong and the module cannot tell. And the three stages assume the route field is already aligned at a fixed offset, which requires the fabric to have no variable-length headers at all.
Production implication: the missing error path is where a source-routed fabric's worst failure lives and it is the exact inverse of Ethernet's. On Ethernet a stale forwarding entry sends a frame to the wrong port and the destination does not answer, so the entry ages out and the system heals — Chapter 12.5 §9's aging is a self-repair mechanism. On a source-routed fabric a stale route is in the header of every packet the sender emits, and it keeps being wrong until the fabric manager recomputes and redistributes. The blast radius is every packet from that sender to that destination, for the whole reconvergence interval, and the only defence is a fast manager — which is a piece of software, on a control plane, that Section 8 prices.
6. What the Lookup's Deletion Is Worth, in BCE
Twelve times the latency is the visible win. The area saving is the invisible one, and it is smaller than anybody expects.
Chapter 23.3 §2's table, with the three arrays a source-routed fabric does not need marked.
| A 64-port 100 Gb/s merchant ASIC | BCE | × the datapath | Deleted by source routing? |
|---|---|---|---|
| packet buffer, 64 MB | 5.12 × 10⁸ | 1 807 | NO |
| ACL TCAM, 16k × 256 b | 2.10 × 10⁷ | 74.0 | yes |
| L3 host and LPM, 128k × 128 b | 1.68 × 10⁷ | 59.2 | yes |
| MAC table, 128k × 96 b | 1.26 × 10⁷ | 44.4 | yes |
| RMON counters | 1.11 × 10⁵ | 0.4 | NO |
| queue state, 512 queues | 3.3 × 10⁴ | 0.1 | NO |
| total | 5.62 × 10⁸ | 1 985 | — |
The three deletable rows sum to 5.04 × 10⁷ BCE — 178 datapaths — which is 9.0% of the switch.
Leaving the standard removes 9.0% of the silicon and 100% of the ecosystem, and Section 8 prices the second half.
Three observations, and the third is the one that decides the argument.
One — the buffer is untouched and it is 91.1%. Chapter 23.3 §2 established that the packet buffer dominates a switch by three orders of magnitude over its control logic, and the buffer's size is set by the bandwidth-delay product of the congestion it must absorb, which is a property of the traffic and the topology rather than of the protocol. A proprietary fabric's buffer is the same size.
Two — the SerDes is untouched too, and it is not in the table at all. Chapter 23.3's table counts on-die state; the physical layer is analogue area, and at 64 ports of 100 Gb/s it is a large fraction of the die that BCE does not price. A proprietary fabric at the same rate uses the same SerDes, because the SerDes is where the standards bodies' work is genuinely irreplaceable and because nobody wants to redo it.
Three — the 9.0% is an upper bound and the real figure is smaller. A proprietary fabric still needs some classification: a way to pick a queue, a way to identify control traffic, a way to apply a policy. It deletes the general mechanisms and reintroduces specific ones, and a fabric with eight traffic classes and a small rule set still carries a classifier.
| Ethernet's general mechanism | A fabric's specific replacement | Saving | |
|---|---|---|---|
| destination | 128k-entry MAC table | a route field | 1.26 × 10⁷ BCE |
| routing | 128k-entry LPM | nothing — the route is complete | 1.68 × 10⁷ |
| policy | 16k × 256 b TCAM | perhaps 256 rules | 2.10 × 10⁷ less ~3 × 10⁵ |
| queueing | 512 queues of state | the same | 0 |
So the achievable saving is about 5.01 × 10⁷ BCE — 8.9% of the switch — and the fabric has given up every mechanism in Modules 12 and 13 to get it.
7. RTL 3 — The Fabric Area Model
// ---------------------------------------------------------------------
// fabric_area_model -- what a switch costs with and without the
// general forwarding mechanisms, in BCE.
//
// The output that matters is undeletable_share_pct, which stays above
// ninety whatever the protocol, because the packet buffer is sized by
// the traffic rather than by the frame format.
// ---------------------------------------------------------------------
module fabric_area_model
import propfab_pkg::*;
(
input logic clk,
input logic rst_n,
input logic [1:0] addressing, // addressing_e
input logic [15:0] replacement_rules, // a fabric still classifies
input logic [15:0] rule_width_bits,
output logic [31:0] deleted_bce_e4,
output logic [31:0] reintroduced_bce_e4,
output logic [31:0] net_saving_bce_e4,
output logic [15:0] net_saving_pct_x10,
output logic [31:0] undeletable_bce_e4,
output logic [15:0] undeletable_share_pct,
output logic [31:0] saving_datapaths_milli,
output logic buffer_is_protocol_independent,
output logic serdes_is_shared,
output logic [31:0] c_configs_modelled
);
always_comb begin
// The three arrays a route makes unnecessary.
deleted_bce_e4 = (addressing == 2'(ADDRESSING_NAME))
? 32'd0
: (32'(ACL_TCAM_BCE_E4) + 32'(L3_LPM_BCE_E4)
+ 32'(MAC_TBL_BCE_E4));
// And what a fabric still needs: a small classifier.
reintroduced_bce_e4 = (32'(replacement_rules)
* 32'(rule_width_bits)) / 32'd10_000;
net_saving_bce_e4 = (deleted_bce_e4 > reintroduced_bce_e4)
? (deleted_bce_e4 - reintroduced_bce_e4) : 32'd0;
net_saving_pct_x10 = 16'((net_saving_bce_e4 * 32'd1000)
/ 32'(SWITCH_BCE_E4));
undeletable_bce_e4 = 32'(SWITCH_BCE_E4) - net_saving_bce_e4;
undeletable_share_pct = 16'((undeletable_bce_e4 * 32'd100)
/ 32'(SWITCH_BCE_E4));
saving_datapaths_milli = 32'(datapaths_milli(net_saving_bce_e4
* 32'd10_000));
// The two facts that cap the saving, stated as constants because
// they are true of every protocol rather than of this one.
buffer_is_protocol_independent = 1'b1;
serdes_is_shared = 1'b1;
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) c_configs_modelled <= '0;
else c_configs_modelled <= c_configs_modelled + 32'd1;
end
endmoduleClassification: a subtraction from Chapter 23.3's table, with the reintroduced cost charged back.
What it teaches: that undeletable_share_pct reports 91 or higher in every configuration, and that the two constants explaining it — buffer_is_protocol_independent and serdes_is_shared — are properties of physics and economics rather than of either protocol. The buffer's size is a bandwidth-delay product of the congestion, which does not care what the header looks like. The SerDes is where an industry's cumulative analogue effort lives and no fabric vendor redoes it, which is why a proprietary fabric's physical layer is usually an Ethernet or a PCIe one.
And it teaches that reintroduced_bce_e4 is the term that gets left out. A fabric with 256 rules of 256 bits reintroduces 65 536 bits — 6.55 × 10⁴ BCE, which is small — but a fabric that needs a thousand rules and a stateful classifier has given most of the saving back. The model charges it so the comparison is honest.
Deliberately simplified: the reintroduced classifier is priced as an SRAM array at 1 BCE per bit, where a TCAM-based one is five times that — Chapter 23.3 §2's factor of 2.5 for the cell and 2 for storing value and mask. The model does not price the fabric manager, which is a control-plane cost rather than a die cost and which Section 9 handles separately. And it takes Chapter 23.3's specific table sizes as the baseline, so a part with a smaller table has a proportionally smaller saving.
Production implication: the table sizes are the lever nobody pulls, and it is worth being explicit because it changes the whole argument. A 128k-entry MAC table is sized for a switch that might be deployed anywhere. A switch deployed as a leaf in Chapter 23.1's fabric sees a few thousand addresses, and a 32k-entry table would be ample — saving 9.45 × 10⁶ BCE, 33.4 datapaths, 1.7% of the die, while remaining fully standard. So a large part of the proprietary fabric's area advantage is available to an Ethernet part that knows its deployment, and the reason merchant silicon does not take it is that a merchant part does not know its deployment. The comparison is therefore not standard against proprietary; it is general-purpose against special-purpose, and that is a much older and better-understood trade.
8. What a Proprietary Fabric Must Build
Sections 2 to 7 found the technical savings: 0.155% of the wire and 8.9% of the die. This section is the bill on the other side, and it is not measured in bitcell equivalents.
A standard is not a frame format. It is a frame format plus everything that exists because the frame format is public, and a fabric that leaves must supply all of it.
| What Ethernet supplies | Where this track covers it | What a proprietary fabric must build |
|---|---|---|
| link bring-up and capability exchange | Chapter 11.1 to Chapter 11.4 — four chapters | its own negotiation, with its own failure modes |
| neighbour discovery and topology | Chapter 15.3 | a discovery protocol and a manager |
| segmentation and isolation | Chapter 13.1 to Chapter 13.4 — four chapters | a tenancy model, if it needs one |
| congestion control | Chapter 14.1 to Chapter 14.4 — four chapters | its own, and Chapter 24.2 says what that costs |
| time synchronisation | Chapter 16.1 to Chapter 16.5 — five chapters | a time protocol, or none |
| statistics and management | Chapter 19.7 | its own counter definitions and its own tooling |
| capture and analysis | Chapter 21.9 | a tap, a capture device, a decoder — all of them |
| verification and conformance | Chapter 20.1 to Chapter 20.6 — six chapters | a reference model with one author |
Twenty-nine chapters of this track are about standardised behaviour that has nothing to do with framing or forwarding — 22.5% of the Ethernet track as it stands — and a proprietary fabric must supply an equivalent for every row or decide to go without.
Leaving the standard deletes 8.9% of the die and obliges the vendor to rebuild twenty-nine chapters' worth of ecosystem, most of which is software and none of which is a differentiator.
Two rows deserve more than a table entry.
Capture and analysis is the one that hurts during bring-up. Chapter 21.9's entire apparatus presumes a tap, a capture device that can keep up, a decoder that knows the format, and a corpus of known-good traffic to compare against. All four are commercially available for Ethernet at every rate and none exists for a new fabric on day one. So the fabric's first silicon is debugged with the fabric's own counters, which is Chapter 21.9 §20's class 102 territory: the observation apparatus is inside the system it observes.
Verification is the one that hurts permanently, and Section 20's rejected property is about it. Chapter 20.3's scoreboard compares a design against a reference model. For Ethernet the reference is a public specification with a thousand independent implementations to check against. For a proprietary fabric the reference is a document the same organisation wrote, and when the design disagrees with it, amending the document is cheaper than fixing the design.
9. RTL 4 — The Ecosystem Cost Model
// ---------------------------------------------------------------------
// ecosystem_cost_model -- what a fabric must supply for itself, counted
// in mechanisms rather than in area, because none of it is on the die.
//
// The output that matters is independent_implementations, which is
// one, and which is the input to every other conclusion in Section 8.
// ---------------------------------------------------------------------
module ecosystem_cost_model
import propfab_pkg::*;
(
input logic clk,
input logic rst_n,
input logic is_standard,
input logic [15:0] mechanisms_needed, // discovery, time, etc.
input logic [15:0] mechanisms_supplied, // by the standard
input logic [15:0] vendors,
output logic [15:0] mechanisms_to_build,
output logic [15:0] independent_implementations,
output logic spec_ambiguity_is_discoverable,
output logic conformance_has_an_oracle,
output logic capture_tooling_exists,
output logic [15:0] second_source_count,
output logic single_vendor_risk,
output logic [31:0] c_mechanisms_unsupplied
);
always_comb begin
mechanisms_to_build = is_standard
? ((mechanisms_needed > mechanisms_supplied)
? (mechanisms_needed - mechanisms_supplied)
: 16'd0)
: mechanisms_needed;
independent_implementations = is_standard ? vendors : 16'd1;
// THE consequence. A specification needs two independent readings
// before anybody can discover that it has two readings.
spec_ambiguity_is_discoverable = (independent_implementations >= 16'd2);
// And a conformance check needs an oracle its own author did not
// write. Section 20's rejected property is exactly this.
conformance_has_an_oracle = (independent_implementations >= 16'd2);
// Chapter 21.9's apparatus presumes a decoder somebody sells.
capture_tooling_exists = is_standard;
second_source_count = is_standard ? vendors : 16'd1;
single_vendor_risk = (second_source_count < 16'd2);
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) c_mechanisms_unsupplied <= '0;
else c_mechanisms_unsupplied <= 32'(mechanisms_to_build);
end
endmoduleClassification: a model with no datapath, whose outputs are all consequences of one integer being 1.
What it teaches: that independent_implementations is the input everything else depends on, and it is 1 for a proprietary fabric by definition. From that one value follow spec_ambiguity_is_discoverable false, conformance_has_an_oracle false, capture_tooling_exists false and single_vendor_risk true — four consequences from one fact, none of which is about the frame format.
And it teaches why mechanisms_to_build is 29 rather than 3. A team leaving a standard budgets for the frame format, the forwarding and the link layer. The twenty-six other rows — discovery, time, congestion, segmentation, statistics, capture, conformance — are not in the plan and are not optional, and each is a small project that the standard supplied for free.
Deliberately simplified: the model counts mechanisms rather than effort, where the rows differ by orders of magnitude — a discovery protocol is weeks and a time protocol accurate to Chapter 16.5's limits is years. vendors is a static input where a standard's implementation count grows over decades and a proprietary fabric's stays at one by policy. And the model treats "supplied by the standard" as binary, where several rows are supplied partially: Ethernet supplies PTP, and a deployment still has to configure it.
Production implication: capture_tooling_exists is the row that determines a programme's schedule and it is almost never in the plan. A new fabric's first silicon arrives with no external way to see what is on the wire, so every bring-up bug is diagnosed from the design's own counters — which are exactly the structures most likely to be wrong in first silicon. The teams that get this right build a capture and replay path into the first chip — a mirror port, a timestamped ring buffer, a way to inject a known frame — and treat it as a tape-out requirement rather than a debug convenience. Chapter 21.9 §2 established that an apparatus whose capacity is below the thing it observes records its own drops as the subject's; an apparatus that is the subject has the same problem and no external witness at all.
10. What It Cannot Delete — the Buffer and the SerDes
Two structures survive every protocol change, they are most of the die and most of the schedule, and a fabric comparison that omits them is comparing the small halves.
The buffer first. Chapter 23.3 §2's 64 MB packet buffer is 5.12 × 10⁸ BCE — 91.1% of the switch — and its size is set by a chain of reasoning in which the frame format appears nowhere.
| Step | What decides it | Chapter |
|---|---|---|
| congestion exists | several ingress ports may target one egress | Chapter 14.1 §4 |
| the overlap must go somewhere | or it is discarded | Chapter 14.1 |
| how much is a bandwidth-delay product | the burst's duration times the excess rate | Chapter 23.3 §9 |
| and it is shared, over-committed 2 793% at idle | because ports are rarely all congested at once | Chapter 23.3 §9 |
None of those four steps mentions a header. A proprietary fabric carrying the same traffic across the same topology at the same rate needs the same buffer, and Chapter 24.2 established that a fabric which tries to avoid it by going lossless pays more rather than less.
Then the SerDes, which is not in Chapter 23.3's table at all because that table counts on-die state.
| What it is | Who supplies it | |
|---|---|---|
| the serialiser and clock recovery | analogue, at 100 Gb/s per lane | an IP vendor, or the same one for both |
| the equalisation | Chapter 22.1's echo and feed-forward machinery | the same |
| the line code and scrambler | Chapter 3.5's 64B/66B | standardised |
| the forward error correction | Chapter 9.6's RS-FEC | standardised |
A proprietary fabric at 400 Gb/s uses an Ethernet or a PCIe SerDes, because that is where an industry's accumulated analogue effort lives and nobody redoes it. So the physical layer — the hardest, longest-lead, most schedule-critical part of any high-rate interconnect — is shared.
The two most expensive things in a switch are protocol-independent. A proprietary fabric differentiates in 8.9% of the die and the header, and shares everything else with the standard it left.
11. RTL 5 — The Undeletable Core
// ---------------------------------------------------------------------
// undeletable_core -- what both fabrics must contain, priced so the
// differentiated fraction can be stated honestly.
//
// The outputs that matter are shared_share_pct and the two booleans
// that explain it: the buffer is sized by traffic and the SerDes by
// physics, and neither cares about the header.
// ---------------------------------------------------------------------
module undeletable_core
import propfab_pkg::*;
(
input logic clk,
input logic rst_n,
input logic [31:0] buffer_bce_e4,
input logic [31:0] forwarding_bce_e4,
input logic [31:0] other_bce_e4,
input logic [15:0] serdes_lanes,
input logic fabric_is_proprietary,
output logic [31:0] total_bce_e4,
output logic [31:0] shared_bce_e4,
output logic [31:0] differentiated_bce_e4,
output logic [15:0] shared_share_pct,
output logic [15:0] differentiated_share_pct_x10,
output logic buffer_sized_by_traffic,
output logic serdes_sized_by_physics,
output logic serdes_is_standard_ip,
output logic [15:0] shared_serdes_lanes,
output logic [31:0] c_evaluations
);
always_comb begin
total_bce_e4 = buffer_bce_e4 + forwarding_bce_e4 + other_bce_e4;
// A proprietary fabric deletes the forwarding state and keeps the
// rest. A standard one keeps everything.
shared_bce_e4 = fabric_is_proprietary
? (buffer_bce_e4 + other_bce_e4)
: total_bce_e4;
differentiated_bce_e4 = total_bce_e4 - shared_bce_e4;
shared_share_pct = (total_bce_e4 == 0) ? 16'd100
: 16'((shared_bce_e4 * 32'd100) / total_bce_e4);
differentiated_share_pct_x10 =
(total_bce_e4 == 0) ? 16'd0
: 16'((differentiated_bce_e4 * 32'd1000)
/ total_bce_e4);
// The three facts that cap every proprietary fabric's advantage.
buffer_sized_by_traffic = 1'b1;
serdes_sized_by_physics = 1'b1;
serdes_is_standard_ip = 1'b1;
// And the lane count is the same on both sides at the same rate.
shared_serdes_lanes = serdes_lanes;
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) c_evaluations <= '0;
else c_evaluations <= c_evaluations + 32'd1;
end
endmoduleClassification: a partition of a die into the part a protocol decides and the part it does not.
What it teaches: that shared_share_pct reports 91 for a proprietary fabric built on Chapter 23.3's numbers, and that the figure is a floor rather than a coincidence: the buffer is 91.1% before anything else is counted. A fabric arguing for a transformative silicon advantage is arguing about 8.9%, and that number is the module's entire output.
And it teaches that the SerDes is absent from the arithmetic and dominant in the schedule. shared_serdes_lanes is reported so a comparison cannot quietly omit it: 64 ports of 400 Gb/s is 256 lanes of 100 Gb/s, each with Chapter 22.1's equalisation machinery, and both fabrics buy the same IP from the same vendors.
Deliberately simplified: the model prices only on-die state and the SerDes is counted in lanes rather than in area, because analogue area does not convert to BCE — it is transistors laid out for matching and noise rather than for density, and Chapter 23.3 §2's gate-equivalent derivation does not apply to it. other_bce_e4 lumps counters and queue state together, which at 0.03% of the die is defensible. And the model assumes the proprietary fabric deletes the whole forwarding block, where Section 7 established it reintroduces a classifier.
Production implication: the SerDes being shared is the fact that most changes a programme's risk profile, and it cuts in the proprietary fabric's favour. A new fabric that reuses a proven 100 Gb/s Ethernet SerDes has eliminated the single largest schedule risk in a high-rate interconnect, because the analogue block is the one that comes back from the fab wrong and takes a respin to fix. A fabric that decides to build its own physical layer as well has taken on a two-year programme to save nothing, since the line code and FEC are public and the equalisation is the same problem. The right shape for a proprietary fabric is therefore a standard physical layer with a proprietary layer above it — which is what Chapter 24.2's RoCE does from the other direction, and it is why the two designs converge in the middle.
12. Where a Proprietary Fabric Actually Fails
Not on bandwidth, not on latency, and not on silicon. It fails on the two things Section 9's model derived from independent_implementations being one.
Failure one — the specification has never been read by anyone who might disagree with it.
A specification's ambiguities are found by two teams implementing it independently and producing incompatible parts. That is what an interoperability event is for, and it is why Chapter 20.6's reusable agent is checkable at all: there is a population of implementations to check against.
| A standard | A proprietary fabric | |
|---|---|---|
| readings of the specification | many, independent | one, by the authors |
| how ambiguity surfaces | two parts fail to interoperate | it does not |
| what happens when design and spec disagree | one of them is wrong and the market says which | the spec is amended |
| the conformance oracle | other implementations | the same document |
Row three is the mechanism and it is not dishonesty. When first silicon disagrees with the specification, amending the specification is a one-line change and fixing the design is a respin. The amendment is made by engineers acting reasonably, and the effect is that the specification converges on describing whatever the design does — at which point conformance to it establishes nothing.
Failure two — there is no second source, and the consequence is not a price.
| What a second source provides | |
|---|---|
| price | a negotiation |
| supply | a fallback when a fab allocation is short |
| roadmap | an alternative if the vendor's priorities change |
| longevity | a part in ten years, when the original vendor has moved on |
The last row is the one that decides infrastructure purchases and it is not about cost at all. A data centre built on a fabric with one supplier has coupled its refresh cycle to that supplier's product plan, and there is no engineering mitigation.
13. RTL 6 — The Second-Source Model
// ---------------------------------------------------------------------
// second_source_model -- the consequences of independent_implementations
// being one, made into signals.
//
// There is no datapath here and that is deliberate: every risk this
// module reports is real, none of it is on the die, and a design
// review that only looks at silicon sees none of it.
// ---------------------------------------------------------------------
module second_source_model
import propfab_pkg::*;
(
input logic clk,
input logic rst_n,
input logic [15:0] implementations,
input logic [15:0] spec_amendments_since_freeze,
input logic [15:0] amendment_budget,
input logic independent_model_exists,
input logic model_authors_saw_design,
output logic oracle_is_independent,
output logic conformance_is_meaningful,
output logic ambiguity_discoverable,
output logic spec_churn_exceeded,
output logic price_negotiable,
output logic supply_has_fallback,
output logic roadmap_is_coupled,
output logic [3:0] risks,
output logic [31:0] c_amendments
);
always_comb begin
// An oracle is independent only if somebody wrote it from the
// document WITHOUT seeing the design. Seeing the design converts
// the model into a second description of the same artefact.
oracle_is_independent = independent_model_exists
&& !model_authors_saw_design;
conformance_is_meaningful = oracle_is_independent
|| (implementations >= 16'd2);
ambiguity_discoverable = (implementations >= 16'd2);
spec_churn_exceeded = (spec_amendments_since_freeze > amendment_budget);
price_negotiable = (implementations >= 16'd2);
supply_has_fallback = (implementations >= 16'd2);
roadmap_is_coupled = (implementations < 16'd2);
risks = { roadmap_is_coupled, spec_churn_exceeded,
!conformance_is_meaningful, !ambiguity_discoverable };
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) c_amendments <= '0;
else c_amendments <= 32'(spec_amendments_since_freeze);
end
endmoduleClassification: a risk register expressed as combinational logic, so that a fact usually argued about is evaluated instead.
What it teaches: that oracle_is_independent requires two conditions and teams routinely satisfy only the first. A separate reference model exists in most programmes; a reference model whose authors never saw the design is rare, because the fastest way to write one is to read the RTL. model_authors_saw_design being true collapses the oracle into a second description of the same artefact, which is Section 12's class 114 with extra steps.
And it teaches that spec_churn_exceeded is a measurable, ignored signal. A specification amended repeatedly after a verification freeze is evidence about the verification rather than about the specification, and counting the amendments costs nothing. A programme that freezes a document and then amends it eleven times has, in effect, verified eleven different designs.
Deliberately simplified: implementations is an integer where the useful quantity is independent readings — two vendors who licensed the same reference design are one reading, not two, and this is why some multi-vendor standards behave like single-vendor ones. The risk vector treats all four risks as equal, where roadmap coupling is a decade-scale commercial exposure and spec churn is a quarter-scale engineering one. And there is no term for partial independence, which is what a standard with one dominant implementation actually has.
Production implication: the fix for oracle_is_independent is organisational and it is cheap relative to what it buys. A second team, reading the specification alone, writing a behavioural model before seeing any RTL — and the productive output is not the model but the list of questions they had to ask, because every question is an ambiguity a standard would have found at a plugfest. A programme that runs this exercise and records the questions has reconstructed the one thing leaving a standard gave up, at a cost of a few engineer-months rather than a few engineer-decades. A programme that skips it ships a fabric whose specification says whatever the design does.
14. What a Proprietary Fabric Must Never Do
Five prohibitions, and three of them are about how the fabric describes itself rather than about how it behaves.
| # | Never | Because |
|---|---|---|
| 1 | quote a framing advantage at minimum frame size | Section 2: it is 45.24% there and 0.155% on the traffic the fabric carries |
| 2 | amend the specification to match the design after freeze | Section 20: the oracle and the design share a custodian and the amendment is always cheaper |
| 3 | claim a silicon advantage without naming the fraction | Section 6: it is 8.9% of the die; the buffer and the SerDes are shared |
| 4 | ship first silicon with no capture path | Section 9: no external decoder exists, so the design's own counters are the only witness |
| 5 | use a variable-length route field without a parser argument | Section 5: the payload's offset then depends on the frame's own content — Chapter 19.2 §6 |
Row four is the one that costs schedule and it is the cheapest to fix.
A fabric's first silicon arrives into a world with no tap, no capture device and no decoder, so every bring-up question — did the packet leave? did it arrive? what did it look like? — is answered by the chip's own counters. Chapter 21.9 §20's class 102 is precisely this hazard: an observation apparatus whose own failures are recorded as the subject's. Here the apparatus is the subject, so there is not even an independent instrument to be wrong.
The fix is three structures and they are small.
| Structure | What it gives | Cost |
|---|---|---|
| a mirror port | traffic on a wire an external instrument can see | one egress port's worth of replication |
| a timestamped ring buffer | the last N frames, after a trigger | Chapter 21.9's capture sink — a few thousand BCE per entry |
| a frame injector | a known frame at a known instant | a register bank and a mux |
All three are tape-out requirements rather than debug conveniences, because a respin to add them costs more than the whole programme saves by leaving the standard.
15. RTL 7 — Fabric Telemetry
// ---------------------------------------------------------------------
// propfab_telemetry -- what a fabric with no external instrumentation
// must report about itself, and the one field that says whether the
// report can be trusted.
//
// Design rule: on a fabric with no second implementation, every
// counter is also a claim, so each one carries its provenance.
// ---------------------------------------------------------------------
module propfab_telemetry
import propfab_pkg::*;
(
input logic clk,
input logic rst_n,
input logic clear,
input logic hop_decoded,
input logic route_exhausted_i,
input logic frame_rx,
input logic [15:0] frame_octets,
input logic frame_dropped,
input logic mirror_active,
input logic capture_armed,
input logic injector_used,
input logic external_decoder_present,
output logic [47:0] c_hops,
output logic [47:0] c_route_errors,
output logic [47:0] c_frames,
output logic [47:0] c_octets,
output logic [47:0] c_drops,
output logic [47:0] c_mirror_frames,
output logic [31:0] mean_frame_octets,
output logic [31:0] framing_tax_ppm_measured,
output logic self_witnessed_only,
output logic capture_path_present
);
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n || clear) begin
c_hops <= '0; c_route_errors <= '0; c_frames <= '0;
c_octets <= '0; c_drops <= '0; c_mirror_frames <= '0;
end else begin
if (hop_decoded) c_hops <= c_hops + 48'd1;
if (route_exhausted_i) c_route_errors <= c_route_errors + 48'd1;
if (frame_rx) begin
c_frames <= c_frames + 48'd1;
c_octets <= c_octets + 48'(frame_octets);
end
if (frame_dropped) c_drops <= c_drops + 48'd1;
if (mirror_active && frame_rx)
c_mirror_frames <= c_mirror_frames + 48'd1;
end
end
always_comb begin
mean_frame_octets = (c_frames == 0) ? 32'd0
: 32'(c_octets / c_frames);
// The tax, measured on the traffic that actually ran, rather than
// quoted at minimum frame size.
framing_tax_ppm_measured =
(mean_frame_octets == 0) ? 32'd0
: 32'(tax_ppm(int'(mean_frame_octets)));
// THE field. Without an external decoder, every number above is a
// claim the design makes about itself.
self_witnessed_only = !external_decoder_present;
capture_path_present = mirror_active || capture_armed || injector_used;
end
endmoduleClassification: a counter bank whose distinguishing output is a statement about the counters' own trustworthiness.
What it teaches: that framing_tax_ppm_measured settles Section 2's argument empirically rather than rhetorically. A fabric reporting a mean frame of 1 500 octets measures a tax of 24 707 ppm — 2.47%; one reporting 9 000 measures 4 204 ppm — 0.42%. The quoted 45.24% corresponds to a mean frame of 46 octets, which the counter will never report on the workloads these fabrics exist for.
And it teaches that self_witnessed_only is the honest label on everything else in the module. Chapter 21.9 §20's class 102 established that an instrument inside the system it measures records its own failures as the subject's; a fabric with no external decoder has only such instruments, and a design that says so in a status bit is doing more than most.
Deliberately simplified: mean_frame_octets is a running mean where the distribution matters more — a bimodal workload of 64-octet control and 9 000-octet bulk has a mean that describes neither. c_mirror_frames counts only while mirroring is active, so it cannot say what fraction of traffic was ever visible externally. And the counters share a clock domain with the datapath, which for a multi-port fabric means the per-port counters must be synchronised before comparison — Chapter 18.1 §19's class 74 once more.
Production implication: the bimodal case is the one that produces a wrong decision, and it is common. A fabric whose frames are 90% 64-octet synchronisation messages and 10% 9 000-octet bulk — which is 94% of its octets — has a mean frame of 958 octets, corresponding to a tax of 3.8% against the true figures of 45.24% and 0.42% for its two populations. The mean describes neither. The useful report is a histogram, which Chapter 21.6 §4 argued for in a different context and for the same reason: a mean over a bimodal distribution is a number that no member of the distribution is near. Sixteen bins of a 48-bit counter is 768 bits, 15 360 BCE, 0.054 datapaths, and it turns a misleading mean into a description.
16. RTL 8 — The Fabric Conformance Monitor
// ---------------------------------------------------------------------
// propfab_conformance -- the checks that survive Section 20's problem,
// because they follow from arithmetic rather than from a document.
//
// Every check here is chosen for one property: the specification
// cannot be amended to make it pass.
// ---------------------------------------------------------------------
module propfab_conformance
import propfab_pkg::*;
(
input logic clk,
input logic rst_n,
input logic [47:0] octets_in,
input logic [47:0] octets_out,
input logic [47:0] octets_dropped,
input logic [47:0] octets_resident,
input logic window_end,
input logic [15:0] route_bits_used,
input logic [15:0] radix,
input logic [15:0] max_hops,
input logic hop_decoded_i,
input logic [3:0] hops_remaining_i,
input logic egress_valid_i,
input logic claims_conformance,
input logic oracle_independent_i,
output logic v_conservation,
output logic v_route_too_short,
output logic v_hop_after_exhaustion,
output logic v_conformance_unfounded,
output logic [3:0] violations,
output logic conformant
);
always_comb begin
// 1. Nothing is created or destroyed. Arithmetic, not a document.
v_conservation = window_end
&& (octets_in != (octets_out + octets_dropped
+ octets_resident));
// 2. The route field must be able to name every hop. Arithmetic
// again: ceil(log2(radix)) x max_hops.
v_route_too_short = (32'(route_bits_used)
< 32'(route_bits(int'(radix), int'(max_hops))));
// 3. A packet whose route is exhausted must not be forwarded.
v_hop_after_exhaustion = egress_valid_i && (hops_remaining_i == 4'd0);
// 4. Section 20's class 114, as a check: a conformance claim needs
// an oracle its own authors did not write.
v_conformance_unfounded = claims_conformance && !oracle_independent_i;
violations = { v_conformance_unfounded, v_hop_after_exhaustion,
v_route_too_short, v_conservation };
conformant = (violations == 4'b0000);
end
endmoduleClassification: four checks selected by a single criterion — that no amendment to a document can make them pass.
What it teaches: that v_conservation is the model check for a fabric whose specification is mutable, because octets in equalling octets out plus dropped plus resident is arithmetic. Nobody can revise it. Chapter 19.7 §19's class 87 is the warning that comes with it — a conservation law over counters committed at different pipeline depths is false for a reason that has nothing to do with the design — so the four terms must be sampled consistently or the check is worse than useless.
And it teaches that v_conformance_unfounded puts Section 20's argument into silicon. A design asserting conformance against a self-authored oracle is making a claim its verification cannot support, and one bit records the fact. It is the only check in this chapter whose subject is a process rather than a signal, and it is included deliberately, because on a fabric with one implementation the process is the risk.
Deliberately simplified: octets_resident must be supplied from the buffer's occupancy and is the term most likely to be sampled at the wrong instant. v_route_too_short is an elaboration check — the route width is fixed at synthesis and the radix at deployment, so it should be a compile-time assertion rather than a runtime one. And oracle_independent_i is an input, so the monitor trusts an organisational fact it cannot verify, which is exactly the recursion class 114 describes and which no amount of RTL escapes.
Production implication: the conservation check is the one to implement and it is routinely implemented wrongly. The four counters must be sampled in the same cycle from the same clock domain, which on a 64-port fabric means a snapshot mechanism rather than four independent reads — a freeze signal, a shadow register per counter, and a single read of the shadows. Without it the check fails intermittently on a correct design, the team disables it, and the one property in the suite that a specification amendment could not have falsified is the one that gets turned off.
17. The Two Fabrics, Priced Side by Side
Everything this chapter derived, in one table.
| Quantity | Ethernet | Proprietary | The gap |
|---|---|---|---|
| framing tax at 64 octets | 45.24% | 37.84% | 7.40 points — 11.90% less wire time |
| framing tax at 9 018 octets | 0.4204% | 0.3101% | 0.110 points — 0.111% less wire time |
| per-hop forwarding latency | 30.00 ns — 36 stages | 2.50 ns — 3 stages | 12× |
| across 5 hops | 150.0 ns | 12.5 ns | 137.5 ns |
| forwarding state | 5.04 × 10⁷ BCE | ≈ 3 × 10⁵ | 8.9% of the die |
| packet buffer | 5.12 × 10⁸ BCE | the same | none |
| SerDes, 64 × 400 Gb/s | 256 lanes of standard IP | the same | none |
| ecosystem mechanisms to build | 0 | 29 chapters' worth | all of it |
| independent implementations | many | 1 | the conformance oracle |
Rows six and seven are blank in the gap column and they are 91% of the die.
A proprietary fabric buys 137.5 nanoseconds and 8.9% of a switch, and pays for it with the conformance oracle, the capture tooling, the second source and twenty-nine chapters of ecosystem.
And the decision is a workload question rather than an engineering one.
| Workload | What 137.5 ns is worth | Verdict |
|---|---|---|
| Chapter 23.2's all-reduce, 15.984 GB/participant | 0.00001% of a 1.279 s job | not worth leaving for |
| a 4 kB storage read, ~100 µs | 0.15% | not worth leaving for |
| an 8-octet remote atomic, ~500 ns round trip | 55% | worth leaving for |
| a barrier across 1 024 participants | the slowest path sets the pace — Chapter 23.2 §4 | worth leaving for |
Rows three and four are real, specific and narrow, and they are the honest case for every proprietary interconnect that has succeeded.
18. What the Comparison Assumes
Eight assumptions, each with the direction the chapter moves if it fails.
| # | Assumption | If it is false |
|---|---|---|
| 1 | the fabric carries large transfers | the framing argument improves; at 64 octets the honest saving is 11.90 points rather than 0.111 |
| 2 | a route is four octets | a shorter route improves the saving proportionally; a variable-length one costs a parser |
| 3 | Chapter 23.3's pipeline is 36 stages | that chapter derived it from the packet rate; a shallower one narrows the 12× |
| 4 | source-route decode is 3 stages | a design that also classifies needs more, and the 12× falls toward 6× |
| 5 | the table sizes are 128k MAC, 128k LPM, 16k ACL | a part sized for its deployment already has most of the 8.9% |
| 6 | the SerDes is shared | a fabric building its own takes a two-year programme to save nothing |
| 7 | the packet buffer is protocol-independent | true by Section 10's four-step derivation; a lossless fabric pays more — Chapter 24.2 §4 |
| 8 | BCE applies | Section 19 examines it and it holds, with the SerDes as an explicit exclusion |
Assumption 5 is the one that reframes the argument and it is worth a sentence. A 32k-entry MAC table instead of 128k saves 9.44 × 10⁶ BCE — 33.3 datapaths, 1.7% of the die — while remaining fully standard. So a large part of the proprietary fabric's area advantage is available to any Ethernet part that knows where it will be deployed, and the comparison is really general-purpose against special-purpose rather than standard against proprietary.
19. The Cost, Accounted — in BCE
This chapter's blocks.
| Block | Flops | BCE | × the datapath |
|---|---|---|---|
framing_tax_model | 32 | 640 | 0.002 |
source_route_decoder, 3 stages | 3 × 35 + 64 | 3 380 | 0.012 |
fabric_area_model | 32 | 640 | 0.002 |
ecosystem_cost_model | 32 | 640 | 0.002 |
undeletable_core | 32 | 640 | 0.002 |
second_source_model | 32 | 640 | 0.002 |
propfab_telemetry | 336 | 6 720 | 0.024 |
propfab_conformance | 0 — combinational | 0 | 0 |
| this chapter's additions | 665 | 13 300 | 0.047 |
Six of the eight blocks are models rather than datapaths, which is characteristic of a comparison chapter: most of it is arithmetic about designs rather than logic inside one. The two that carry real state are the decoder and the telemetry, and together they are 10 100 BCE — 0.036 datapaths.
And the designs the blocks describe.
| BCE | × the datapath | % of Chapter 23.3's switch | |
|---|---|---|---|
| forwarding state a source route deletes | 5.04 × 10⁷ | 178 | 9.0% |
| net of the classifier it reintroduces | 5.01 × 10⁷ | 177 | 8.9% |
| the packet buffer, unchanged | 5.12 × 10⁸ | 1 807 | 91.1% |
| a 32k MAC table's saving, inside the standard | 9.44 × 10⁶ | 33.3 | 1.7% |
| Chapter 23.3's whole switch | 5.62 × 10⁸ | 1 985 | 100% |
Row four against row two is the chapter's quietest finding: 18.8% of the proprietary fabric's entire area advantage is available by choosing a smaller table and staying inside the standard.
20. Properties Worth Asserting, and One Worth Refusing
Fifty-one properties in six groups, and the refused one is the top-level property of every verification plan in this chapter's subject.
Group A — the framing tax (9).
// A1. The wire slot is the payload plus thirty-eight fixed octets.
p_ft_wire_slot: assert property (@(posedge clk) disable iff (!rst_n)
(wire_octets_o == 32'(payload_octets) + 32'(OCT_FIXED)));
// A2. The tax falls as the payload grows -- Chapter 8.3's hyperbola.
p_ft_monotone: assert property (@(posedge clk) disable iff (!rst_n)
(payload_octets > $past(payload_octets)) |->
(tax_ppm_o <= $past(tax_ppm_o)));
// A3. Only the addresses and the type are deletable.
p_ft_deletable_set: assert property (@(posedge clk) disable iff (!rst_n)
(deletable_ppm_o == ((32'(OCT_DELETABLE) * 1_000_000) / wire_octets_o)));
// A4. The preamble survives any protocol change.
p_ft_preamble_stays: assert property (@(posedge clk) disable iff (!rst_n)
(preamble_is_deletable == 1'b0));
// A5. And so does the check sequence.
p_ft_fcs_stays: assert property (@(posedge clk) disable iff (!rst_n)
(fcs_is_deletable == 1'b0));
// A6. So the undeletable part is always the larger share.
p_ft_undeletable_larger: assert property (@(posedge clk) disable iff (!rst_n)
(undeletable_ppm > deletable_ppm_o));
// A7. A route REPLACES rather than removes, so the saving is a
// difference and never the whole field.
p_ft_route_replaces: assert property (@(posedge clk) disable iff (!rst_n)
((addressing != 2'(ADDRESSING_NAME)) && (route_octets >= 16'(OCT_DELETABLE)))
|-> (actual_saving_ppm == 32'd0));
// A8. A named-addressing fabric saves nothing by definition.
p_ft_named_saves_nothing: assert property (@(posedge clk) disable iff (!rst_n)
(addressing == 2'(ADDRESSING_NAME)) |-> (actual_saving_ppm == 32'd0));
// A9. The packet rate is the wire slot divided into the line rate.
p_ft_packet_rate: assert property (@(posedge clk) disable iff (!rst_n)
(rate_gbps != 0) |->
(packet_ns_x100 == (wire_octets_o * 8 * 100) / 32'(rate_gbps)));Group B — the source route (9).
// B1. No lookup is performed, ever.
p_sr_no_lookup: assert property (@(posedge clk) disable iff (!rst_n)
(lookup_needed == 1'b0));
// B2. The decode is three stages.
p_sr_three_stages: assert property (@(posedge clk) disable iff (!rst_n)
(stages_used == 32'(SR_STAGES)));
// B3. And twelve times faster than a lookup pipeline.
p_sr_ratio: assert property (@(posedge clk) disable iff (!rst_n)
(latency_ratio_x10 == 32'd120));
// B4. The egress port is the route's low bits.
p_sr_port_is_low_bits: assert property (@(posedge clk) disable iff (!rst_n)
egress_valid |-> (egress_port == $past(route_s2[5:0])));
// B5. The route shifts down by exactly one hop.
p_sr_shift: assert property (@(posedge clk) disable iff (!rst_n)
egress_valid |-> (route_out == ($past(route_s2) >> 6)));
// B6. Hops remaining decrements by one and never wraps.
p_sr_hops_decrement: assert property (@(posedge clk) disable iff (!rst_n)
(egress_valid && ($past(hops_s2) != 4'd0)) |->
(hops_remaining_out == $past(hops_s2) - 4'd1));
// B7. An exhausted route is not forwarded.
p_sr_no_forward_exhausted: assert property (@(posedge clk) disable iff (!rst_n)
route_exhausted |-> !egress_valid);
// B8. And it is counted.
p_sr_exhausted_counted: assert property (@(posedge clk) disable iff (!rst_n)
route_exhausted |=> (c_exhausted == $past(c_exhausted) + 32'd1));
// B9. The route field is wide enough for the topology. An elaboration
// check, written as a property so it appears in the report.
p_sr_route_wide_enough: assert property (@(posedge clk) disable iff (!rst_n)
(ROUTE_BITS >= route_bits(RADIX, MAX_HOPS)));Group C — area (8).
// C1. A named-addressing fabric deletes nothing.
p_ar_named_deletes_nothing: assert property (@(posedge clk) disable iff (!rst_n)
(addressing == 2'(ADDRESSING_NAME)) |-> (deleted_bce_e4 == 32'd0));
// C2. A routed one deletes exactly the three arrays.
p_ar_routed_deletes_three: assert property (@(posedge clk) disable iff (!rst_n)
(addressing != 2'(ADDRESSING_NAME)) |->
(deleted_bce_e4 == (32'(ACL_TCAM_BCE_E4) + 32'(L3_LPM_BCE_E4)
+ 32'(MAC_TBL_BCE_E4))));
// C3. The reintroduced classifier is charged back.
p_ar_reintroduction_charged: assert property (@(posedge clk) disable iff (!rst_n)
(replacement_rules != 16'd0) |-> (reintroduced_bce_e4 != 32'd0));
// C4. The net saving never exceeds the gross.
p_ar_net_bounded: assert property (@(posedge clk) disable iff (!rst_n)
(net_saving_bce_e4 <= deleted_bce_e4));
// C5. And it is a single-digit percentage of the switch.
p_ar_single_digit: assert property (@(posedge clk) disable iff (!rst_n)
(net_saving_pct_x10 <= 16'd100));
// C6. The undeletable share is above ninety, always.
p_ar_undeletable_dominates: assert property (@(posedge clk) disable iff (!rst_n)
(undeletable_share_pct >= 16'd90));
// C7. The buffer is sized by traffic rather than by the header.
p_ar_buffer_independent: assert property (@(posedge clk) disable iff (!rst_n)
(buffer_sized_by_traffic == 1'b1));
// C8. And the SerDes is the same on both sides at the same rate.
p_ar_serdes_shared: assert property (@(posedge clk) disable iff (!rst_n)
(serdes_is_shared && serdes_is_standard_ip &&
(shared_serdes_lanes == serdes_lanes)));Group D — the ecosystem (8).
// D1. A proprietary fabric has one implementation by definition.
p_ec_one_impl: assert property (@(posedge clk) disable iff (!rst_n)
!is_standard |-> (independent_implementations == 16'd1));
// D2. Ambiguity needs two readings to be discoverable.
p_ec_ambiguity: assert property (@(posedge clk) disable iff (!rst_n)
(independent_implementations < 16'd2) |-> !ambiguity_discoverable);
// D3. And so does a conformance oracle.
p_ec_oracle: assert property (@(posedge clk) disable iff (!rst_n)
(independent_implementations < 16'd2) |-> !conformance_has_an_oracle);
// D4. Capture tooling exists only for a standard.
p_ec_capture: assert property (@(posedge clk) disable iff (!rst_n)
(capture_tooling_exists == is_standard));
// D5. A single implementation couples the roadmap.
p_ec_roadmap: assert property (@(posedge clk) disable iff (!rst_n)
(second_source_count < 16'd2) |-> single_vendor_risk);
// D6. Every mechanism the standard does not supply must be built.
p_ec_build_the_rest: assert property (@(posedge clk) disable iff (!rst_n)
is_standard |-> (mechanisms_to_build ==
((mechanisms_needed > mechanisms_supplied)
? (mechanisms_needed - mechanisms_supplied) : 16'd0)));
// D7. A model whose authors saw the design is not an oracle.
p_ec_model_blind: assert property (@(posedge clk) disable iff (!rst_n)
model_authors_saw_design |-> !oracle_is_independent);
// D8. Spec churn past its budget is a reportable condition.
p_ec_churn: assert property (@(posedge clk) disable iff (!rst_n)
(spec_amendments_since_freeze > amendment_budget) |-> spec_churn_exceeded);Group E — telemetry (8).
// E1. The measured tax follows the measured mean frame.
p_tl_measured_tax: assert property (@(posedge clk) disable iff (!rst_n)
(mean_frame_octets != 32'd0) |->
(framing_tax_ppm_measured == 32'(tax_ppm(int'(mean_frame_octets)))));
// E2. Without an external decoder, the fabric witnesses only itself.
p_tl_self_witness: assert property (@(posedge clk) disable iff (!rst_n)
(self_witnessed_only == !external_decoder_present));
// E3. A capture path is present if any of the three mechanisms is.
p_tl_capture_path: assert property (@(posedge clk) disable iff (!rst_n)
(capture_path_present == (mirror_active || capture_armed || injector_used)));
// E4. Octets accumulate with frames.
p_tl_octets_track: assert property (@(posedge clk) disable iff (!rst_n)
frame_rx |=> (c_octets == $past(c_octets) + 48'($past(frame_octets))));
// E5. Mirrored frames never exceed received frames.
p_tl_mirror_bounded: assert property (@(posedge clk) disable iff (!rst_n)
(c_mirror_frames <= c_frames));
// E6. A route error is counted exactly once.
p_tl_route_error_once: assert property (@(posedge clk) disable iff (!rst_n)
route_exhausted_i |=> (c_route_errors == $past(c_route_errors) + 48'd1));
// E7. Clearing zeroes every counter.
p_tl_clear: assert property (@(posedge clk) disable iff (!rst_n)
clear |=> ((c_frames == 48'd0) && (c_octets == 48'd0) &&
(c_drops == 48'd0)));
// E8. The mean is bounded by the maximum frame.
p_tl_mean_bounded: assert property (@(posedge clk) disable iff (!rst_n)
(c_frames != 0) |-> (mean_frame_octets <= 32'd9018));Group F — conformance, chosen so no amendment can make them pass (9).
// F1. Conservation. Arithmetic, not a document.
p_cf_conservation: assert property (@(posedge clk) disable iff (!rst_n)
window_end |-> (octets_in == octets_out + octets_dropped
+ octets_resident));
// F2. The route field must name every hop.
p_cf_route_width: assert property (@(posedge clk) disable iff (!rst_n)
!v_route_too_short);
// F3. An exhausted route must not be forwarded.
p_cf_no_forward: assert property (@(posedge clk) disable iff (!rst_n)
!v_hop_after_exhaustion);
// F4. A conformance claim needs an oracle its authors did not write.
p_cf_oracle_required: assert property (@(posedge clk) disable iff (!rst_n)
claims_conformance |-> oracle_independent_i);
// F5. The violation vector is the monitor's own disjunction.
p_cf_vector: assert property (@(posedge clk) disable iff (!rst_n)
conformant |-> (violations == 4'b0000));
// F6. Conservation is checked only at a consistent sample point --
// Chapter 19.7 Section 19's class 87 as a guard rather than a trap.
p_cf_sample_consistent: assert property (@(posedge clk) disable iff (!rst_n)
window_end |-> $stable(snapshot_taken));
// F7. The reference model's version is pinned at sign-off.
p_cf_version_pinned: assert property (@(posedge clk) disable iff (!rst_n)
signoff_requested |-> (reference_version == SIGNOFF_SPEC_VERSION));
// F8. And an independently written model agrees.
p_cf_independent_agrees: assert property (@(posedge clk) disable iff (!rst_n)
transaction_observed |-> (observed == independent_model_predicts));
// F9. Amendments after freeze are budgeted.
p_cf_amendment_budget: assert property (@(posedge clk) disable iff (!rst_n)
signoff_requested |-> (spec_amendments_since_freeze <= AMENDMENT_BUDGET));Coverage — the states a fabric comparison only reaches with a real workload.
c_ft_min_frame: cover property (@(posedge clk) payload_octets <= 16'd46);
c_ft_jumbo: cover property (@(posedge clk) payload_octets >= 16'd9000);
c_ft_route_bigger: cover property (@(posedge clk) route_octets >= 16'd14);
c_sr_exhausted: cover property (@(posedge clk) route_exhausted);
c_sr_max_hops: cover property (@(posedge clk) hops_remaining_in == 4'd5);
c_ar_reintroduced: cover property (@(posedge clk) reintroduced_bce_e4 != 32'd0);
c_ar_small_table: cover property (@(posedge clk) net_saving_pct_x10 <= 16'd20);
c_ec_no_oracle: cover property (@(posedge clk) !oracle_is_independent);
c_ec_churn: cover property (@(posedge clk) spec_churn_exceeded);
c_ec_blind_model: cover property (@(posedge clk) !model_authors_saw_design);
c_tl_self_only: cover property (@(posedge clk) self_witnessed_only);
c_tl_no_capture: cover property (@(posedge clk) !capture_path_present);
c_cf_conservation: cover property (@(posedge clk) v_conservation);
c_cf_unfounded: cover property (@(posedge clk) v_conformance_unfounded);21. Verification Scenarios
Fifty-eight scenarios in six groups, plus one directed test random stimulus will not produce.
Group 1 — the framing tax (10).
| # | Scenario | Expect |
|---|---|---|
| 1 | 46-octet payload | 84-octet slot; 452 381 ppm — 45.24% |
| 2 | 256-octet payload | 294-octet slot; 129 252 ppm — 12.93% |
| 3 | 1 500-octet payload | 1 538-octet slot; 24 707 ppm — 2.47% |
| 4 | 9 000-octet payload | 9 038-octet slot; 4 204 ppm — 0.42% |
| 5 | a 4-octet route replacing 14 | 119 048 ppm saved at 46; 1 106 at 9 000 |
| 6 | a 14-octet route | saving is zero — p_ft_route_replaces |
| 7 | ADDRESSING_NAME | saving is zero by definition |
| 8 | preamble_is_deletable and fcs_is_deletable | both 0, at every setting |
| 9 | 400 Gb/s at 46 octets | 595 238 k packets/s; 1.68 ns each |
| 10 | 400 Gb/s at 9 000 octets | 5 532 k packets/s; 180.76 ns each — a factor of 107.6 |
Group 2 — the source route (10).
| # | Scenario | Expect |
|---|---|---|
| 11 | radix 64, 5 hops | 6 bits per hop; 30 bits of route |
| 12 | radix 32, 5 hops | 5 bits per hop; 25 bits |
| 13 | radix 128, 7 hops | 7 bits per hop; 49 bits |
| 14 | a 30-bit field for radix 64, 5 hops | p_sr_route_wide_enough holds |
| 15 | a 24-bit field for the same | v_route_too_short |
| 16 | one hop decoded | egress is the low 6 bits; route shifts by 6 |
| 17 | five hops in sequence | five decodes, hops reaching zero |
| 18 | a sixth hop after exhaustion | not forwarded; c_exhausted increments |
| 19 | lookup_needed at any setting | 0 |
| 20 | latency, 3 stages against 36 at 1.2 GHz | 2.50 ns against 30.00; ratio 120 |
Group 3 — area (9).
| # | Scenario | Expect |
|---|---|---|
| 21 | ADDRESSING_ROUTE, no classifier | 5.04 × 10⁷ BCE deleted; 178 datapaths; 9.0% |
| 22 | the same with 256 rules of 256 bits | 6.55 × 10⁴ reintroduced; net 8.9% |
| 23 | 1 024 rules of 512 bits | 5.24 × 10⁵ reintroduced; net still 8.9% |
| 24 | a TCAM-based classifier, 256 × 256 b at ×5 | 3.28 × 10⁵ reintroduced; net still 8.9% |
| 25 | ADDRESSING_NAME | nothing deleted; undeletable_share_pct = 100 |
| 26 | undeletable_share_pct at any setting | ≥ 90 — p_ar_undeletable_dominates |
| 27 | 64 ports at 400 Gb/s | 256 SerDes lanes, both fabrics |
| 28 | a 32k MAC table inside the standard | 9.44 × 10⁶ BCE saved; 33.3 datapaths; 1.7% |
| 29 | row 28 against row 22 | 18.8% of the advantage, without leaving |
Group 4 — the ecosystem (10).
| # | Scenario | Expect |
|---|---|---|
| 30 | is_standard high, 12 vendors | ambiguity_discoverable; conformance_has_an_oracle |
| 31 | is_standard low | independent_implementations = 1; all four risks |
| 32 | 29 mechanisms needed, 29 supplied | 0 to build |
| 33 | 29 needed, 0 supplied | 29 to build |
| 34 | an independent model whose authors saw the RTL | oracle_is_independent low |
| 35 | an independent model written from the document alone | oracle_is_independent high; conformance is meaningful |
| 36 | 11 amendments against a budget of 2 | spec_churn_exceeded |
| 37 | capture_tooling_exists on a proprietary fabric | low |
| 38 | one vendor | no price negotiation, no supply fallback, roadmap coupled |
| 39 | two vendors who licensed one reference design | the model reports 2; the readings are 1 — see Section 13 |
Group 5 — telemetry (9).
| # | Scenario | Expect |
|---|---|---|
| 40 | all frames at 1 500 octets | mean 1 500; measured tax 24 707 ppm |
| 41 | all frames at 9 000 | mean 9 000; 4 204 ppm |
| 42 | 90% at 64, 10% at 9 000 | mean 958; 38 153 ppm — 3.82%, describing neither |
| 43 | no external decoder | self_witnessed_only high |
| 44 | a mirror port active | capture_path_present; mirrored ≤ received |
| 45 | a route exhaustion | c_route_errors increments once |
| 46 | clear asserted | every counter zero next cycle |
| 47 | zero frames | mean is 0 and the tax is 0, not a division by zero |
| 48 | 16-bin histogram instead of a mean | 768 bits; 15 360 BCE; 0.054 datapaths |
Group 6 — conformance (10).
| # | Scenario | Expect |
|---|---|---|
| 49 | octets balance at a window end | no v_conservation |
| 50 | octets short by one frame | v_conservation |
| 51 | the four terms sampled in different cycles | v_conservation fires on a correct design — class 87 |
| 52 | a snapshot mechanism added | it stops firing |
| 53 | a 24-bit route for radix 64, 5 hops | v_route_too_short |
| 54 | forwarding after exhaustion | v_hop_after_exhaustion |
| 55 | claims_conformance with no independent oracle | v_conformance_unfounded |
| 56 | the same with one | clean |
| 57 | spec amended after the version pin | p_cf_version_pinned fires |
| 58 | all four checks clear | conformant high |
22. Debugging a Fabric With No Second Implementation
Six symptoms, and the first question is different from every other debugging section in this track, because there is nothing to compare against.
| Symptom | First question | Where to look |
|---|---|---|
| packets arriving at the wrong endpoint | is the route stale, or is the decode wrong? | Section 5 — a stale route repeats on every packet until the manager recomputes |
| a bring-up bug with no visible traffic | is there a mirror port or a capture ring? | Section 14 — if not, the design's own counters are the only witness |
| counters that do not balance | were the four terms sampled in one cycle? | Section 16 — Chapter 19.7 §19's class 87 |
| a regression that improved without an RTL change | what is spec_amendments_since_freeze? | Section 21's directed test |
| two endpoints that disagree about a field | which reading of the specification is each using? | Section 12 — there is no third party to settle it |
| latency worse than the 12× predicted | is a classifier in the path? | Section 7 — a reintroduced classifier adds stages back |
Row five is the one with no procedure, and saying so is more useful than pretending otherwise.
On a standard, two implementations disagreeing about a field is settled by a third: a plugfest, a published test suite, a reference implementation, or simply the weight of a thousand parts that read it the other way. On a proprietary fabric there is no third, so the disagreement is settled by whoever wrote the document, which means it is settled by seniority rather than by evidence.
23. Misconceptions
Misconception 1 — "Ethernet's framing overhead is 45%."
The wrong model: the tax is a property of the frame format, so it is the same everywhere.
What it costs: a fabric built to recover an overhead it does not have. Section 2: 45.24% is the tax at a 46-octet payload and 0.4204% at 9 000, and the fabrics that leave the standard carry large transfers. The honest deletable saving on jumbo traffic is 0.155% of the wire.
The corrected model: the tax is 38 ÷ (payload + 38), a hyperbola whose interesting region is entirely at the small end — Chapter 8.3 §2 — and quoting the small end for traffic at the large end is choosing the wrong point on a curve everybody knows.
Misconception 2 — "a proprietary fabric can delete the preamble and the check sequence."
The wrong model: those are Ethernet conventions and a new protocol need not have them.
What it costs: a receiver that cannot find the first bit, or a fabric that delivers corrupted data silently. Section 2: the preamble exists for clock recovery and frame delimitation — Chapter 5.2 §3 — and the FCS because wires corrupt. PCIe has an LCRC and InfiniBand has two CRCs; the field is renamed, not removed.
The corrected model: only the addresses and the EtherType are deletable, and a route replaces rather than removes them. Four octets against fourteen is the real saving.
Misconception 3 — "leaving the standard saves most of the switch."
The wrong model: forwarding tables are what a switch is, so deleting them is transformative.
What it costs: a programme justified by an area saving of 8.9%. Section 6: the three deletable arrays are 5.04 × 10⁷ BCE against a 5.62 × 10⁸ BCE switch, and the packet buffer alone is 91.1%. The SerDes is not in the table and is shared.
The corrected model: the two most expensive things in a switch are protocol-independent — the buffer is sized by the traffic and the SerDes by physics — so a proprietary fabric differentiates in 8.9% of the die and the header.
Misconception 4 — "the win is bandwidth."
The wrong model: a purpose-built fabric moves more data than a general-purpose one.
What it costs: competing where the gap is tenths of a per cent. Both fabrics use the same SerDes at the same rate, and Chapter 9.6 took the standard to 800 Gb/s. Section 4: the real win is the lookup — 30 ns against 2.5, twelve times, 137.5 ns across five hops — and it is a latency, quoted as a bandwidth.
The corrected model: name the workload. For Chapter 23.2's all-reduce, 137.5 ns is 0.00001% of the job. For an 8-octet remote atomic on a 500 ns round trip it is 55%. The fabric is worth building for the second.
Misconception 5 — "the ecosystem is a few tools we will write later."
The wrong model: a standard supplies a frame format; everything else is optional tooling.
What it costs: a first silicon with no external decoder, debugged by its own counters. Section 8: twenty-nine chapters of this track — 22.5% of it — are standardised behaviour with nothing to do with framing or forwarding, and Chapter 21.9's entire apparatus presumes a tap, a capture device and a decoder that nobody sells for a new fabric.
The corrected model: the ecosystem is a network effect, so the vendor pays the full cost of every tool rather than a share of it. Build the capture path into first silicon or accept that bring-up has no external witness.
Misconception 6 — "our conformance suite passes, so the design conforms."
The wrong model: a green regression against the reference model means the design matches the specification.
What it costs: sign-off on self-consistency. Section 20's class 114: the design and the oracle share a custodian, so the cheapest repair for a failure is to amend the oracle, and Section 21's directed test shows a property going from 412 failures to zero with unchanged RTL.
The corrected model: an oracle is independent only if its authors never saw the design. The repair is a second team reading the document alone — a few engineer-months, and it reconstructs the one thing leaving the standard gave up. The properties that survive are the ones derived from arithmetic rather than from a document.
24. Interview Questions
Six, with what a strong answer contains.
1. What does Ethernet's framing actually cost?
Thirty-eight octets, and the percentage depends entirely on the payload. 38 ÷ (payload + 38): 45.24% at a 46-octet payload, 2.47% at 1 500, 0.4204% at 9 000. A strong answer goes further: only fourteen of the thirty-eight are deletable — the addresses and the type — because the preamble is needed for clock recovery and the FCS because wires corrupt. So the honest saving on jumbo traffic is 0.155%, and a route field replacing the addresses makes it 0.111%.
2. Where is a proprietary fabric's real advantage?
The lookup. Chapter 23.3 §2's forwarding pipeline is 36 stages at 1.2 GHz — 30 ns; a source-routed decode is a shift and a demultiplex, three stages, 2.50 ns. That is 12× per hop and 137.5 ns across five hops. A strong answer names the structural reason: a name must be resolved against a table and a route is an index, so the saving is not an optimisation, it is the absence of a mechanism.
3. And then what does 137.5 nanoseconds buy?
It depends entirely on the operation. For Chapter 23.2's all-reduce moving 15.984 GB per participant over 1.279 s, it is 0.00001%. For an 8-octet remote atomic with a 500 ns round trip it is 55%. The strong answer says that the fabrics which succeed are the ones built for the second case, and that the comparison cannot be made without naming the workload and the percentile.
4. How much of a switch does leaving the standard actually save?
8.9%. The three deletable arrays are ACL TCAM 2.10 × 10⁷, L3/LPM 1.68 × 10⁷, MAC table 1.26 × 10⁷ — 5.04 × 10⁷ BCE against a 5.62 × 10⁸ BCE switch — less a small reintroduced classifier. The packet buffer is 91.1% and is protocol-independent, and the SerDes is not in the table and is the same IP on both sides. A strong answer adds the quiet finding: a 32k-entry MAC table instead of 128k saves 1.7% while remaining fully standard, which is 18.8% of the whole advantage.
5. What does a fabric with one implementation give up in verification?
The oracle. A conformance property compares a design against a reference model built from a specification; on a standard that specification has a thousand independent readings and cannot be revised to accommodate one part. On a proprietary fabric the design and the document share a custodian, so the cheapest repair for a failure is an amendment, and nothing in the environment distinguishes an amendment from a fix. A strong answer names the repair — a second team writing a model from the document alone, never having seen the RTL — and the cheap partial repair: properties derived from arithmetic, like conservation, which no amendment can falsify.
6. Somebody shows you a fabric comparison. What do you ask?
Four questions. What payload size does the framing argument assume? Is the latency win in the lookup or the framing? What fraction of the die is actually different? How many independent implementations exist? A strong answer notes that the last one determines four separate consequences — ambiguity discovery, the conformance oracle, capture tooling and the second source — and that none of them appears in a technical comparison. The best answers close with the general point: a market's units are a model of what can be compared, not of what is scarce, and the binding quantity here — tail latency on small operations under load — cannot be compared across vendors and so is never published.
25. Questions and Answers
26. What's Next
Module 24 is complete. Three chapters set Ethernet against a load/store fabric, a lossless HPC fabric and a fabric that left the standard, and each derived a number rather than ticking a box.
| Chapter | The number |
|---|---|
| Chapter 24.1 | 425 088 BCE of return-path state at 400 Gb/s, against an Ethernet transmitter's zero |
| Chapter 24.2 | 91.47% of a merchant switch reserved, at 2 km, per lane |
| this chapter | 137.5 ns and 8.9% of a die, against twenty-nine chapters of ecosystem |
Module 25 turns from what Ethernet is not to what engineers believe it is. Six myths, each of which causes real bugs and bad interviews, and each corrected by deriving the number that kills it rather than by asserting the correction.
Chapter 25.1 takes the largest of them — that Ethernet and TCP/IP are the same thing. It builds the layer boundary in RTL and derives what a MAC can and cannot see: a receive MAC parses eighteen octets of a 1 518-octet frame — 1.19% — and reads exactly zero octets of the IP and TCP headers. Then it prices what believing the myth costs in silicon, because Chapter 23.4's connection context cache is 66.2% of that NIC and every bit of it is above the Ethernet boundary.
Chapter 25.2 takes the second — that MAC addresses are globally unique forever — and derives the collision arithmetic at data-centre scale, which is not the arithmetic people quote.
Continue learning
Related tutorials
- Related topic
PCIe vs Ethernet — Where the Cost of Overload Lands
The same overload into two fabrics: one stalled the sender 59,405 times and lost nothing, the other discarded 59,405 frames. That single choice explains why one needs TCP and the other does not.
- Related topic
The Shared-Medium Problem
Why several independent transmitters on one medium is a distributed timing problem, not a formatting problem. Propagation delay makes every station's view of the medium stale, so two locally correct decisions can still collide — and that is the constraint the Ethernet MAC was built around.
- Related topic
CSMA/CD, Collision Domains and Slot Time
Slot time is the parameter the whole half-duplex MAC hangs on: it bounds medium acquisition, bounds a collision fragment, and is the retransmission quantum. Deriving it from round-trip propagation plus jam is what fixes Ethernet's minimum frame size — a timing constant wearing a frame-format costume.
- Related topic
Packet Switching
A circuit allocates capacity in advance and guarantees it; a packet network allocates on demand and guarantees nothing. The exchange is measurable in RTL — idle reserved slots against buffered, delayed and occasionally dropped packets — and it is why a packet must describe its own extent and destination.
Standards & specifications
- Governing standard
- IEEE Std 802.3 (Ethernet)(opens IEEE in a new tab)
Defines the Ethernet MAC, the media-independent interfaces and the physical-layer sublayers, including framing, access control, auto-negotiation and per-rate PHY specifications. VLAN tagging, priority and time-sensitive shaping are defined by IEEE 802.1, not by 802.3.
This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.
Where this fits
Part of the Ethernet curriculum.
