Inside a Merchant Switch ASIC

Five chapters have priced their mechanisms in flip-flops and five have hit the same wall: a block whose size is set by the data rather than by the design. This chapter meets it at full scale, so it starts by fixing the unit.

The replacement is derived in Section 2 and used everywhere after it.

1 BCE = the area of one bit of on-die SRAM, including array overhead

1 flip-flop ≈ 7 gate equivalents; 1 SRAM bit ≈ 0.35; so 1 flop = 20 BCE

Chapter 19.7 §19's 14 166-flop MAC receive datapath is 283 320 BCE, and every figure in Modules 19 to 23 can be restated in it without changing what it means.

Then the switch.

A merchant switch ASIC's on-die state is nearly two thousand times a MAC receive datapath, and 91% of it is one array: the packet buffer.

And the pipeline that moves data through it is not the expensive part at all.

Row five closes a loop. Chapter 23.1 §4 assumed 0.030 µs for lookup and fabric without deriving it; a 36-stage pipeline at 1.2 GHz is exactly that, which is the depth a real forwarding pipeline has.

1. Scope — A Pipeline, and a Unit That Can Price It

Scope: the forwarding pipeline of a merchant switch ASIC, from ingress port to egress queue, and an area unit in which its parts can be compared.

Not in scope: the fabric it sits in. Chapter 23.1 treated a switch as a port count and a forwarding function; this chapter opens the function. What the switch is wired to is that chapter's, and what a collective does to it is Chapter 23.2's.

Two jobs, and the first has to come first.

Job 1 is not bookkeeping. Five chapters have reported a mechanism's cost and then said the number was meaningless, which is an admission that the track has been measuring with the wrong instrument since Module 19. The flop was the right unit for a MAC datapath because a MAC datapath is almost all flops. It is the wrong unit the moment a design contains a memory, and a switch ASIC is 91% memory.

Three things the chapter establishes about the pipeline itself.

Row two is the chapter's structural finding and Section 20's refused property. A packet entering a switch stops being one object: its bytes go to a memory and its metadata goes down a 36-stage pipeline, and they are rejoined at the egress queue. Every per-stage property in the pipeline is a property of the descriptor, and none of them says anything about the bytes — which is why a pipeline can be proved correct stage by stage and still corrupt packets.

2. The Unit: Bitcell Equivalents, and Why the Flop Failed

Derive a unit in which a register and a memory can be compared, using quantities that are properties of a process rather than of a design.

Start with gate equivalents, which are process-portable by construction.

1 GE = the area of one 2-input NAND at the node in question

Row four is the one to use, because a memory bit that cannot be read is not a memory bit: the overhead is part of the cost. A macro's array efficiency — bitcell area divided by total macro area — is typically 60 to 70%, which turns 0.2 GE into about 0.35.

Which gives the unit.

1 BCE (bitcell equivalent) = 0.35 GE = one bit of usable on-die SRAM

1 flip-flop = 7 GE ÷ 0.35 GE = 20 BCE

A flip-flop costs about twenty times what a bit of SRAM costs. That single ratio is what the flop unit has been hiding for five chapters.

Restating the track's anchor figure.

And the five escapes, now priced honestly.

Row one is the clearest demonstration that the old unit was wrong. Chapter 22.1 §19 reported a 512-entry, 2-bit delay line as 1 024 flops — 7.2% of a MAC datapath — and noted it would be a RAM in silicon. In BCE it is 1 024 bits: 0.4% of a datapath, twenty times smaller than the flop count said.

Row four is the unit's own boundary and it is worth naming. An 8 GB capture buffer is DRAM on a host, not on-die SRAM; BCE does not apply to it and neither does any on-die unit. The rule the chapter adopts is:

Row three's 2.5 needs a word. A TCAM cell stores a value and a mask and performs a parallel compare, so it is roughly 16 transistors against SRAM's 6 plus match-line logic — about 2.5× the area per stored bit, and it is stored twice (value and mask). Section 8 uses both factors.

3. RTL 1 — The Switch Package and the Area Model

// ---------------------------------------------------------------------
// swasic_pkg -- a 64-port 100 Gb/s merchant switch, and the unit in
// which its parts can be compared.
//
// Every earlier chapter's package held rates and sizes. This one holds
// an AREA MODEL as well, because the chapter's subject is a design
// whose cost is 91% memory and the track's usual unit cannot say so.
// ---------------------------------------------------------------------
package swasic_pkg;

  // ---- the switch -------------------------------------------------------
  localparam int PORTS        = 64;
  localparam int RATE_GBPS    = 100;
  localparam int CAPACITY_TBPS= (PORTS * RATE_GBPS) / 1000;   // 6.4
  localparam int CORE_MHZ     = 1200;

  // ---- the area unit ----------------------------------------------------
  // 1 BCE = one bit of usable on-die SRAM, including array overhead.
  // A flip-flop is 7 gate equivalents; an SRAM bit with overhead is
  // 0.35; so a flop is 20 BCE. A TCAM bit is about 2.5, and it is
  // stored twice -- value and mask.
  localparam int BCE_PER_FLOP = 20;
  localparam int BCE_PER_SRAM = 1;
  localparam int BCE_PER_TCAM = 5;    // 2.5 per bit x 2 bits stored

  // Chapter 19.7 Section 19's anchor, restated.
  localparam int DATAPATH_FLOPS = 14166;
  localparam int DATAPATH_BCE   = DATAPATH_FLOPS * BCE_PER_FLOP;  // 283320

  function automatic int bce_of_flops(int n);   return n * BCE_PER_FLOP; endfunction
  function automatic int bce_of_sram (int bits); return bits * BCE_PER_SRAM; endfunction
  function automatic int bce_of_tcam (int entries, int width);
    return entries * width * BCE_PER_TCAM;
  endfunction

  // ---- the state, sized ---------------------------------------------------
  localparam int BUFFER_MB      = 64;
  localparam int BUFFER_BITS    = BUFFER_MB * 1000 * 1000 * 8;
  localparam int CELL_BYTES     = 256;          // buffer allocation unit
  localparam int CELLS_TOTAL    = (BUFFER_MB * 1000 * 1000) / CELL_BYTES;

  localparam int MAC_ENTRIES    = 128 * 1024;
  localparam int MAC_WIDTH      = 96;
  localparam int L3_ENTRIES     = 128 * 1024;
  localparam int L3_WIDTH       = 128;
  localparam int ACL_ENTRIES    = 16 * 1024;
  localparam int ACL_WIDTH      = 256;

  localparam int QUEUES_PER_PORT= 8;
  localparam int QUEUES_TOTAL   = PORTS * QUEUES_PER_PORT;

  // ---- the pipeline -------------------------------------------------------
  // 9.524 Gpps aggregate at minimum size against a 1.2 GHz clock is
  // 7.94 packets per cycle, so the pipeline is 8 wide.
  localparam int PIPE_WIDTH     = 8;
  // 36 stages at 1.2 GHz is 30 ns, which is exactly Chapter 23.1
  // Section 4's lookup-and-fabric term.
  localparam int PIPE_STAGES    = 36;
  localparam int DESC_BITS      = 256;

  // ---- the two halves of a packet -----------------------------------------
  // Section 11: a packet entering the switch stops being one object.
  typedef enum logic {
    HALF_PAYLOAD = 1'b0,   // to the buffer
    HALF_DESC    = 1'b1    // down the pipeline
  } packet_half_e;

  typedef enum logic [2:0] {
    ST_PARSE   = 3'd0,
    ST_LOOKUP  = 3'd1,
    ST_ACL     = 3'd2,
    ST_ADMIT   = 3'd3,     // the shared buffer's threshold decision
    ST_ENQUEUE = 3'd4,
    ST_SCHED   = 3'd5,
    ST_REJOIN  = 3'd6,     // descriptor meets payload again
    ST_EGRESS  = 3'd7
  } pipe_stage_e;

endpackage

Classification: a parameter package carrying an area model, which no earlier package in the track needed.

What it teaches: that BCE_PER_FLOP = 20 is the only constant here that is a property of a process rather than of a design. Every other number — port count, buffer size, table depth — is a choice somebody made. The ratio between a flip-flop and an SRAM bit is a fact about silicon, and it is what makes the rest of the package comparable at all.

And it teaches that packet_half_e exists. Nothing in Modules 19 to 22 needed it: a MAC processes a frame as one object. A switch splits it — payload to a memory, descriptor to a pipeline — and Section 11 shows that every per-stage property in pipe_stage_e is a property of one half.

Deliberately simplified: BCE_PER_TCAM = 5 folds the 2.5× cell factor and the ×2 for value-and-mask into one integer, which is convenient and hides the two effects. CELL_BYTES = 256 is a buffer allocation unit and a real design's choice trades internal fragmentation against pointer overhead — a 64-octet frame in a 256-byte cell wastes 75%. PIPE_STAGES = 36 is derived backwards from a latency target rather than forwards from the work. And there is no notion of a second pipeline for the egress direction, which most merchant parts have.

Production implication: the cell-size fragmentation hidden in CELL_BYTES is a real capacity loss and it is workload-dependent in the worst way. A 64-octet frame occupying a 256-byte cell uses 25% of it, so a buffer of 64 MB holds 250 000 cells and only 16 MB of minimum-size frames — a quarter of its nameplate. A collective's 1 500-octet frames occupy 6 cells and waste 1.2%. So the same buffer absorbs 80 µs of maximum-size traffic and 20 µs of minimum-size, and a design validated on large frames discovers the difference during an incast of small ones. Publish the cell occupancy, not just the cell count.

// ---------------------------------------------------------------------
// bce_area_model -- add up a design's state in one unit and report what
// fraction each part is.
//
// The block exists so that "the buffer dominates" is a measurement
// rather than an assertion, and so that a proposal to double a table
// can be compared against a proposal to halve the buffer.
// ---------------------------------------------------------------------
module bce_area_model
  import swasic_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [31:0] buffer_bits,
  input  logic [31:0] mac_bits,
  input  logic [31:0] l3_bits,
  input  logic [31:0] acl_entries,
  input  logic [31:0] acl_width,
  input  logic [31:0] control_flops,

  output logic [63:0] buffer_bce,
  output logic [63:0] table_bce,
  output logic [63:0] tcam_bce,
  output logic [63:0] logic_bce,
  output logic [63:0] total_bce,
  output logic [15:0] buffer_share_pct,
  output logic [31:0] datapaths_equivalent,
  output logic        memory_dominates
);
  always_comb begin
    buffer_bce = 64'(buffer_bits) * 64'(BCE_PER_SRAM);
    table_bce  = (64'(mac_bits) + 64'(l3_bits)) * 64'(BCE_PER_SRAM);
    tcam_bce   = 64'(acl_entries) * 64'(acl_width) * 64'(BCE_PER_TCAM);
    logic_bce  = 64'(control_flops) * 64'(BCE_PER_FLOP);

    total_bce = buffer_bce + table_bce + tcam_bce + logic_bce;

    buffer_share_pct = (total_bce == 0) ? 16'd0
                     : 16'((buffer_bce * 100) / total_bce);

    // The comparison the whole track has been making, now in a unit
    // that survives a memory.
    datapaths_equivalent = 32'(total_bce / 64'(DATAPATH_BCE));

    memory_dominates = ((buffer_bce + table_bce + tcam_bce) >
                        (logic_bce * 64'd10));
  end
endmodule

Classification: an adder with a denominator, and the denominator is the track's anchor figure.

What it teaches: that memory_dominates is true by three orders of magnitude, not marginally. A 6.4 Tb/s switch's memory is 5.62 × 10⁸ BCE and its control logic — even generously at 100 000 flops — is 2 × 10⁶. The memory is 281 times the logic, which is why a switch ASIC's floorplan is a picture of memories with logic in the gaps, and why a chapter that prices it in flip-flops says nothing.

And it teaches that datapaths_equivalent is the right way to carry Modules 19 to 22's habit forward. The comparison against Chapter 19.7 §19's datapath has been the track's yardstick for five modules; in BCE it still works, and it gives 1 985 rather than an error.

Deliberately simplified: control_flops is one input covering the parser, the scheduler, the queue state and everything else — a real model separates them. All arithmetic is in 64-bit integers because 5.62 × 10⁸ overflows 32 bits at the multiply. memory_dominates uses a factor of ten, which for this design is absurdly conservative. And there is no notion of the analog area at all — 64 SerDes lanes at 100 Gb/s are a large fraction of a real die and are not state.

Production implication: the missing SerDes area is the omission that matters for a floorplan and it changes the conclusion's shape rather than its direction. A 6.4 Tb/s switch's 64 high-rate SerDes occupy a substantial share of the die and scale with the port count rather than with the buffer, so a decision to double the buffer and a decision to double the port count consume different resources and cannot be traded against each other. BCE prices the state correctly and says nothing about the I/O; a complete area argument needs both, and the two halves belong to different engineering groups.

4. A 6.4 Tb/s Switch, Accounted in BCE

Every piece of on-die state in a 64-port 100 Gb/s merchant switch, in one table and one unit.

Four observations, and the last is the useful one.

One — the buffer is 91.0% of the state. Everything else in the table together is 9%, and the three tables that people argue about in product meetings — MAC, L3 and ACL — are 8.8% between them. A proposal to double the MAC table costs 44 datapaths; a proposal to shave 10% off the buffer saves 181.

Two — the TCAM is the most expensive bit in the design. Sixteen thousand ACL entries at 256 bits is 4.19 Mbit of information, which as SRAM would be 4.19 × 10⁶ BCE. As TCAM it is 2.10 × 10⁷ — five times more — because each bit is stored twice and each stored bit costs about 2.5 SRAM bits. That factor of five is why ACL depth is rationed and MAC depth is not.

Three — the counters are nothing. Chapter 19.7's 27 RMON counters, replicated across 64 ports at 64 bits, are 1.11 × 10⁵ BCE — 0.4 of a datapath, 0.02% of the switch. The track has spent five modules pricing instrumentation against a MAC datapath; at switch scale the entire statistics subsystem is a rounding error, and the argument for adding a counter is no longer an area argument at all.

Four — and the ratio between the two halves of the design is 281.

So a switch ASIC is a memory system with a forwarding pipeline attached, and every subsequent section of this chapter is about one or the other of those two things. Sections 5 to 8 and 13 are the pipeline; Sections 9 to 12 are the memory; Section 17 is why they have different timing character.

5. RTL 2 — The Ingress Parser

// ---------------------------------------------------------------------
// ingress_parser -- extract the fields the pipeline needs, at 8 packets
// per cycle.
//
// The parser is where a packet becomes a DESCRIPTOR. Everything
// downstream of this block operates on 256 bits of metadata; the
// packet's bytes have gone somewhere else, and Section 11 is about
// what that costs.
// ---------------------------------------------------------------------
module ingress_parser
  import swasic_pkg::*;
#(
  parameter int WIDTH = PIPE_WIDTH        // 8 packets per cycle
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [WIDTH-1:0]        pkt_valid,
  input  logic [WIDTH-1:0][511:0] pkt_head,      // first 64 octets
  input  logic [WIDTH-1:0][5:0]   pkt_port,

  output logic [WIDTH-1:0][DESC_BITS-1:0] desc,
  output logic [WIDTH-1:0]        desc_valid,
  output logic [WIDTH-1:0]        parse_deep,    // needed more than 64 octets
  output logic [WIDTH-1:0]        parse_failed,
  output logic [31:0]             c_parsed,
  output logic [31:0]             c_deep,
  output logic [31:0]             c_failed,
  output logic [15:0]             headroom_pct
);
  // Chapter 5.1's frame, Chapter 13.2's tag, Chapter 5.5's EtherType
  // resolution -- all of it has to happen in a FIXED number of stages,
  // because the pipeline cannot stall.
  localparam int PARSE_STAGES = 6;

  function automatic logic [DESC_BITS-1:0] build_desc(
      logic [511:0] head, logic [5:0] port);
    automatic logic [DESC_BITS-1:0] d = '0;
    d[47:0]    = head[511:464];     // destination address
    d[95:48]   = head[463:416];     // source address
    d[111:96]  = head[415:400];     // EtherType or length
    d[123:112] = head[396:385];     // VLAN ID, if tagged
    d[126:124] = head[399:397];     // priority
    d[132:127] = port;
    d[255:133] = '0;                // lookup results fill the rest
    return d;
  endfunction

  always_comb begin
    for (int i = 0; i < WIDTH; i++) begin
      desc[i]       = build_desc(pkt_head[i], pkt_port[i]);
      desc_valid[i] = pkt_valid[i];
      // A header chain longer than the 64 octets the parser can see in
      // its fixed stage budget. The pipeline CANNOT stall, so a deep
      // parse is a punt, not a retry.
      parse_deep[i]   = pkt_valid[i] && (pkt_head[i][415:400] == 16'h8100)
                                     && (pkt_head[i][383:368] == 16'h8100);
      parse_failed[i] = pkt_valid[i] && (pkt_head[i][415:400] < 16'd46);
    end
    // The pipeline's real budget: 7.94 packets per cycle needed against
    // WIDTH available.
    headroom_pct = 16'(((WIDTH * 100) - 794) / 8);
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      c_parsed <= '0; c_deep <= '0; c_failed <= '0;
    end else begin
      for (int i = 0; i < WIDTH; i++) begin
        if (desc_valid[i])   c_parsed <= c_parsed + 32'd1;
        if (parse_deep[i])   c_deep   <= c_deep   + 32'd1;
        if (parse_failed[i]) c_failed <= c_failed + 32'd1;
      end
    end
  end
endmodule

Classification: a fixed-stage field extractor that cannot stall and therefore cannot retry.

What it teaches: that parse_deep is a punt rather than a retry, and that is forced by the pipeline's structure. Chapter 19.2 §19's MAC parser could stall its datapath; a switch's parser cannot, because 63 other ports are feeding the same pipeline and a stall propagates to all of them. So a header chain the parser cannot resolve in its fixed budget is sent to a slower path or dropped, and the choice is made at design time.

And it teaches that the descriptor is 256 bits while the packet may be 12 144. Everything after this block operates on 2.1% of the packet; the rest has gone to the buffer. Every property written about the pipeline from here on is a property of those 256 bits, which is Section 20's subject.

Deliberately simplified: build_desc reads fixed bit positions, so it assumes an untagged Ethernet II frame and gets a tagged one wrong — a real parser is a state machine over a header chain. parse_deep detects exactly one case, a double VLAN tag. parse_failed tests the length field against 46 without Chapter 5.5's length-or-type disambiguation. And headroom_pct bakes the 7.94 packets-per-cycle figure as the literal 794, which is correct for 64 ports of 100 Gb/s and wrong for anything else.

Production implication: the fixed parse budget is what decides which protocols a switch supports, and it is a silicon decision that outlives the protocols. A parser sized for 64 octets of header handles Ethernet, one VLAN tag, IPv4 and TCP; the same parser meets a packet with two VLAN tags, an MPLS stack and IPv6 extension headers and cannot reach the fields it needs. The punt path is typically orders of magnitude slower — a CPU, or a recirculation that costs a second pass through the pipeline — so a small fraction of deep-parse traffic costs far more than its share suggests. Count c_deep and alarm on it: a rising deep-parse rate is a protocol the switch was not designed for, arriving in production.

6. The Parser's Budget Is a Cycle Count, Not a Gate Count

The pipeline's shape is decided by one division, and it is not an area calculation.

Start with the packet rate.

minimum frame on the wire = 84 octets = 672 bits

per port at 100 Gb/s = 100 × 10⁹ ÷ 672 = 148.81 Mpps

64 ports = 9.524 Gpps

Then the clock.

A switch ASIC's pipeline is wide because the packet rate exceeds the clock, and by a factor that is not marginal: at any achievable core frequency the pipeline must process several packets every cycle. That is the single structural fact that separates a switch's forwarding path from Chapter 19.2's MAC parser, which handles one frame at a time.

And the consequence is that everything downstream is replicated.

Row four is a number worth sitting with. The metadata moving through the pipeline is 2.44 Tb/s, against the packet data's 6.4 Tb/s in each direction. The descriptors are 38% of the payload bandwidth, which is why the pipeline's wiring is a floorplan problem and not an afterthought.

Row two is the one that decides the table architecture. Eight lookups per cycle into a 128k-entry table cannot be served by a single-ported SRAM, so the table is either banked, replicated, or pipelined at a higher internal rate. Section 8 prices the three.

And the depth is set by latency, not by work.

The budget is met exactly, which is not a coincidence — it is how the number was chosen. Chapter 23.1 §4 assumed 30 ns for lookup and fabric and did not derive it; 36 stages at 1.2 GHz is where that assumption comes from, and a design that wants a deeper ACL must either raise the clock or accept a longer hop.

A switch pipeline's width is set by the packet rate and its depth by a latency target. Neither is set by how much logic the work requires, which is why Section 19's area table is almost entirely memory.

7. RTL 3 — The Lookup Stage

// ---------------------------------------------------------------------
// lookup_stage -- 8 lookups per cycle into a 128k-entry table, and a
// control plane writing to the same table at the same time.
//
// The read-during-update problem is this block's real subject, and
// Section 20's refused property is a cousin of it: the pipeline reads a
// table that a second agent is mutating, and "the table's contents"
// is not a well-defined referent at the moment of a lookup.
// ---------------------------------------------------------------------
module lookup_stage
  import swasic_pkg::*;
#(
  parameter int WIDTH   = PIPE_WIDTH,
  parameter int ENTRIES = MAC_ENTRIES,
  parameter int BANKS   = 16
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [WIDTH-1:0]                   q_valid,
  input  logic [WIDTH-1:0][47:0]             q_addr,
  input  logic [WIDTH-1:0][11:0]             q_vlan,

  // the control plane, writing while the data plane reads
  input  logic                               cp_write,
  input  logic [16:0]                        cp_index,
  input  logic [MAC_WIDTH-1:0]               cp_entry,
  input  logic                               cp_epoch,       // toggles per batch

  output logic [WIDTH-1:0]                   hit,
  output logic [WIDTH-1:0][5:0]              egress_port,
  output logic [WIDTH-1:0]                   read_during_write,
  output logic [WIDTH-1:0]                   epoch_at_lookup,
  output logic [15:0]                        bank_conflicts,
  output logic [31:0]                        c_lookups,
  output logic [31:0]                        c_rdw,
  output logic [31:0]                        c_bank_conflict
);
  // Eight lookups per cycle cannot come from a single-ported SRAM, so
  // the table is banked and the hash decides the bank. Two queries
  // hashing to the same bank in the same cycle CONFLICT, and the
  // pipeline cannot stall -- Section 6.
  function automatic logic [3:0] bank_of(logic [47:0] a, logic [11:0] v);
    return 4'(a[3:0] ^ a[11:8] ^ v[3:0]);
  endfunction

  logic [BANKS-1:0][3:0] bank_users;

  always_comb begin
    bank_conflicts = 16'd0;
    for (int b = 0; b < BANKS; b++) bank_users[b] = 4'd0;
    for (int i = 0; i < WIDTH; i++)
      if (q_valid[i])
        bank_users[bank_of(q_addr[i], q_vlan[i])] =
          bank_users[bank_of(q_addr[i], q_vlan[i])] + 4'd1;
    for (int b = 0; b < BANKS; b++)
      if (bank_users[b] > 4'd1)
        bank_conflicts = bank_conflicts + 16'(bank_users[b] - 4'd1);

    for (int i = 0; i < WIDTH; i++) begin
      hit[i]         = q_valid[i];
      egress_port[i] = 6'(q_addr[i][5:0]);
      // The referent problem, made a signal: this lookup landed in the
      // same cycle as a control-plane write. It is not an ERROR -- the
      // design is allowed to return either version -- but the property
      // that says "the result matches the table" has no table to mean.
      read_during_write[i] = q_valid[i] && cp_write;
      epoch_at_lookup[i]   = cp_epoch;
    end
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      c_lookups <= '0; c_rdw <= '0; c_bank_conflict <= '0;
    end else begin
      for (int i = 0; i < WIDTH; i++) begin
        if (q_valid[i])           c_lookups <= c_lookups + 32'd1;
        if (read_during_write[i]) c_rdw     <= c_rdw + 32'd1;
      end
      c_bank_conflict <= c_bank_conflict + 32'(bank_conflicts);
    end
  end
endmodule

Classification: a banked table with two hazards, one spatial and one temporal.

What it teaches: that bank_conflicts is the price of not being able to stall. Eight queries per cycle into 16 banks is balls-in-bins again — Chapter 23.2 §10's arithmetic, at a different scale — and the expected number of banks receiving two or more queries is not zero. The pipeline cannot stall, so a conflict must be resolved by replication, by a higher internal clock, or by deferring one query to the next cycle, and the third costs a stage of latency that the fixed-depth budget has to contain.

And it teaches that epoch_at_lookup exists because read_during_write is not an error. A control plane writing a forwarding entry while the data plane reads it is normal and constant — a switch learns addresses, ages them and installs routes continuously. The design's requirement is not that a lookup match "the table"; it is that each lookup return some self-consistent entry. The epoch bit is what makes that requirement checkable, and Section 20 is the property that gets written instead.

Deliberately simplified: bank_of is an XOR fold where a real design uses a proper hash and often several. hit is asserted unconditionally, so there is no miss path, no default route and no flood behaviour. egress_port is taken from the address's low bits rather than from a table read — the block models the access pattern, not the lookup. And the control plane writes one entry at a time, where a real installation is a multi-word update whose intermediate states must never be readable.

Production implication: the multi-word update is where read-during-write becomes a correctness problem rather than a definitional one. A forwarding entry wider than the memory's write width is installed in two or more cycles, and a lookup landing between them reads half the old entry and half the new — an address that points at no port, or at the wrong one. The remedies are a valid bit written last, a shadow bank with an atomic pointer flip, or a read-modify-write lock, and the first is the cheapest: write the entry with its valid bit clear, then set the bit in a single word. Chapter 17.2 §13's atomic installation is the same idea for a schedule; here it is one bit rather than a whole shadow table.

8. Three Table Technologies, and What Each Costs Per Entry

A switch needs three kinds of lookup and they are built from three different memories. The BCE unit makes them comparable for the first time.

Priced per entry.

A TCAM entry costs 13.3 times a MAC entry, and the factor decomposes cleanly:

That is the whole reason ACL tables are small and MAC tables are large, and it is an area fact rather than an architectural preference. A switch with 128k MAC entries and 16k ACL entries spends 1.26 × 10⁷ BCE on the first and 2.10 × 10⁷ on the second — the eight-times-smaller table costs more.

And there is a third cost TCAM has and SRAM does not.

Row one is why TCAM depth is limited by power as often as by area. A TCAM compares the key against every entry simultaneously; that is what makes it a one-cycle longest-prefix engine and what makes its energy per lookup proportional to its depth. A 16k-entry TCAM searched 8 times per cycle at 1.2 GHz performs 1.57 × 10¹⁴ entry-comparisons per second — 4.03 × 10¹⁶ bit-comparisons, and the power that implies is a real constraint on the die.

Which gives the design rule the section exists for.

Put a lookup in SRAM if its question can be asked as an exact match, and in TCAM only if it cannot. Every ternary bit costs five SRAM bits and its entry is compared a hundred and fifty-seven trillion times a second.

The algorithmic LPM in row three of the first table is the middle path and it is worth naming. A longest-prefix match can be done in SRAM with a multi-level trie or a compressed structure, at 128 BCE per entry instead of 1 280 — a factor of ten — at the cost of several dependent memory accesses instead of one parallel compare. In a 36-stage pipeline there is room for several dependent accesses, which is exactly why modern merchant parts implement LPM algorithmically and reserve TCAM for the ACL, where the key is genuinely ternary and no exact-match reformulation exists.

9. RTL 4 — The Shared Buffer and Its Dynamic Threshold

// ---------------------------------------------------------------------
// shared_buffer_threshold -- Chapter 14.1 Section 6's alpha-times-free
// rule, at 64 ports and 64 MB instead of 24 ports and 12 MiB.
//
// The buffer is 91% of the switch's state, so its admission rule is
// the single most consequential piece of control logic on the die.
// ---------------------------------------------------------------------
module shared_buffer_threshold
  import swasic_pkg::*;
#(
  parameter int RESERVE_CELLS = 512,      // per port, guaranteed
  parameter int ALPHA_SHIFT   = 1         // alpha = 1/2
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        req_valid,
  input  logic [5:0]  req_port,
  input  logic [15:0] req_cells,
  input  logic        release_valid,
  input  logic [5:0]  release_port,
  input  logic [15:0] release_cells,

  output logic        admit,
  output logic [31:0] free_cells,
  output logic [31:0] shared_free,
  output logic [31:0] port_limit,
  output logic [31:0] port_occupancy [PORTS],
  output logic [31:0] c_admits,
  output logic [31:0] c_refusals,
  output logic [15:0] overcommit_pct,
  output logic        limit_is_relative
);
  localparam int CELLS      = CELLS_TOTAL;                 // 250 000
  localparam int RESERVED   = PORTS * RESERVE_CELLS;       // 32 768
  localparam int SHARED     = CELLS - RESERVED;            // 217 232

  logic [31:0] used_cells;

  always_comb begin
    free_cells  = 32'(CELLS) - used_cells;
    shared_free = (free_cells > 32'(RESERVED))
                ? (free_cells - 32'(RESERVED)) : 32'd0;

    // Chapter 14.1 Section 6's rule: a port's allowance is a FRACTION of
    // what is currently free, not a fixed quantity. That is what makes
    // Section 20's refused property refuse-able.
    port_limit = shared_free >> ALPHA_SHIFT;

    admit = req_valid &&
            ((port_occupancy[req_port] < 32'(RESERVE_CELLS)) ||
             ((port_occupancy[req_port] - 32'(RESERVE_CELLS) + 32'(req_cells))
                <= port_limit));

    // Section 10: the sum of what every port MAY hold, against what
    // exists. Chapter 14.1's 612% at 24 ports, recomputed at 64.
    overcommit_pct = 16'(((32'(PORTS) * (32'(RESERVE_CELLS) + port_limit))
                          * 100) / 32'(CELLS));

    // Always true, and it is the point: this is a RELATIVE mechanism
    // and any absolute guarantee read from it is an inference the
    // mechanism does not support.
    limit_is_relative = 1'b1;
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      used_cells <= '0; c_admits <= '0; c_refusals <= '0;
      for (int p = 0; p < PORTS; p++) port_occupancy[p] <= '0;
    end else begin
      if (req_valid && admit) begin
        port_occupancy[req_port] <= port_occupancy[req_port] + 32'(req_cells);
        used_cells <= used_cells + 32'(req_cells);
        c_admits   <= c_admits + 32'd1;
      end else if (req_valid) begin
        c_refusals <= c_refusals + 32'd1;
      end
      if (release_valid) begin
        port_occupancy[release_port] <= (port_occupancy[release_port]
                                         > 32'(release_cells))
                                      ? port_occupancy[release_port]
                                        - 32'(release_cells) : 32'd0;
        used_cells <= (used_cells > 32'(release_cells))
                    ? used_cells - 32'(release_cells) : 32'd0;
      end
    end
  end
endmodule

Classification: an admission rule whose threshold is a function of every other port's behaviour.

What it teaches: that port_limit is recomputed every cycle from shared_free, which 63 other ports are changing. A port's allowance is not a number the port owns — it is α times whatever is left, so a port that was admitted at 100 000 cells a moment ago may be refused at 20 000 now, having done nothing. Chapter 14.1 §6 established the rule and its motivation; this block is it at 64 ports, where the number of parties changing the threshold is nearly three times larger.

And it teaches that limit_is_relative is asserted unconditionally as a warning. The mechanism provides a fraction of what is free. An absolute claim — this port can always buffer X — is not something the mechanism delivers, and Section 20's refused property is exactly that inference.

Deliberately simplified: port_limit is one number for all ports, so a design with per-class or per-queue thresholds is inexpressible — real switches have eight queues per port, each with its own limit. admit reads port_occupancy combinationally and the same array is written sequentially, which is a model. overcommit_pct uses the instantaneous limit, so it moves every cycle. And there is no hysteresis, so a port hovering at its limit oscillates between admit and refuse on consecutive cells.

Production implication: the missing hysteresis is what produces a pathology worth naming, because its symptom is a fairness complaint rather than a loss one. A port at its limit admits a cell, which reduces shared_free, which lowers the limit, which refuses the next cell, which is released, which raises the limit — and the port ends up admitted roughly half the time in a pattern set by the interaction rather than by the traffic. Two ports doing this can lock into an anti-phase in which one consistently wins, and the loser's drops look like a per-port defect. Add hysteresis — a refusal holds for a few cells — and the oscillation stops; without it the switch is fair on average and unfair on any timescale a flow can feel.

10. Over-commitment, Scaled From 12 MiB to 64 MB

Chapter 14.1 §6 computed a 612% over-commitment on a 24-port switch with a 12 MiB pool. This section runs the same arithmetic on the part this chapter is about.

The pool, in cells.

Then the over-commitment, at three pool states.

Read the table downward and the mechanism's whole design becomes visible. The over-commitment is enormous when the pool is empty and vanishes as it fills, because the threshold is a fraction of what is free rather than a fixed quantity. That is the property a fixed limit cannot have, and Chapter 14.1 §6's argument — a fixed limit is too generous when the pool is empty and too mean when it is idle — is what the table shows numerically.

And the scaling against Chapter 14.1's 24-port case is worth stating.

Row three's two numbers are not comparable and that is the point. Chapter 14.1's 612% is a fixed over-commitment, computed once from constants; this chapter's 2 793% is an instantaneous one that falls to 13.1% as the pool fills. A design that quotes a single over-commitment figure for a dynamic-threshold buffer has quoted one point on a curve, and usually the most alarming one.

The buffer's depth, in the unit that matters operationally.

And the cell-size correction that Section 3 flagged. Those figures assume the buffer is filled efficiently. A 64-octet frame in a 256-octet cell uses 25% of it, so a buffer absorbing minimum-size traffic holds a quarter of its nameplate: 64 MB becomes 16 MB, and 80 µs becomes 20.

A 64 MB buffer is 80 microseconds of one port's line rate for large frames and 20 for small ones, and the difference is a cell size nobody outside the design team knows.

Which connects directly to Chapter 23.2 §14. That chapter showed per-packet spraying needs a reassembly buffer equal to the fabric's worst-case path skew times the line rate, and the skew is set by the deepest buffer a packet can queue behind. With 64 MB shared across 64 ports and a dynamic threshold, a single congested port can hold up to α × free of the shared region — 108 616 cells, 27.8 MB, 2 224 µs at 100 Gb/s — so the skew a sprayer must absorb is not the per-port average but the worst case a single port can reach.

11. RTL 5 — The Ingress-to-Egress Crossing

// ---------------------------------------------------------------------
// descriptor_payload_split -- the moment a packet stops being one
// object.
//
// The payload goes to the buffer; the descriptor goes down 36 stages of
// pipeline. They are rejoined at the egress queue. Every per-stage
// property between here and there is a property of ONE HALF, and
// Section 20's refused class is composing them into a claim about the
// packet.
// ---------------------------------------------------------------------
module descriptor_payload_split
  import swasic_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        pkt_valid,
  input  logic [15:0] pkt_octets,
  input  logic [DESC_BITS-1:0] desc_in,

  // the payload half
  output logic        buf_write,
  output logic [15:0] buf_cells,
  output logic [31:0] buf_ptr,
  // the descriptor half
  output logic        desc_out_valid,
  output logic [DESC_BITS-1:0] desc_out,
  output logic [31:0] desc_ptr,

  // the rejoin
  input  logic        egress_ready,
  input  logic [31:0] egress_desc_ptr,
  output logic        rejoin_ok,
  output logic        rejoin_mismatch,

  output logic [31:0] c_split,
  output logic [31:0] c_rejoined,
  output logic [31:0] c_orphan_payload,
  output logic [31:0] c_orphan_desc,
  output logic        halves_travel_separately
);
  logic [31:0] next_ptr;

  always_comb begin
    buf_cells = 16'((pkt_octets + 16'(CELL_BYTES) - 16'd1)
                    / 16'(CELL_BYTES));
    buf_write = pkt_valid;
    buf_ptr   = next_ptr;

    desc_out_valid = pkt_valid;
    // The pointer is the ONLY thing connecting the two halves. It is
    // 32 bits of the 256-bit descriptor, and if it is wrong the
    // descriptor is perfectly well-formed and points at somebody
    // else's bytes.
    desc_out = {desc_in[DESC_BITS-1:32], next_ptr};
    desc_ptr = next_ptr;

    rejoin_ok       = egress_ready && (egress_desc_ptr == next_ptr);
    rejoin_mismatch = egress_ready && (egress_desc_ptr != next_ptr);

    // Always true. Stated as a signal because every property
    // downstream needs to know it.
    halves_travel_separately = 1'b1;
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      next_ptr <= '0; c_split <= '0; c_rejoined <= '0;
      c_orphan_payload <= '0; c_orphan_desc <= '0;
    end else begin
      if (pkt_valid) begin
        next_ptr <= next_ptr + 32'(buf_cells);
        c_split  <= c_split + 32'd1;
      end
      if (rejoin_ok)       c_rejoined       <= c_rejoined + 32'd1;
      // A payload whose descriptor was dropped mid-pipeline leaks a
      // buffer cell. A descriptor whose payload was never written
      // emits garbage. Neither is visible to any per-stage property.
      if (buf_write && !desc_out_valid) c_orphan_payload <= c_orphan_payload + 32'd1;
      if (desc_out_valid && !buf_write) c_orphan_desc    <= c_orphan_desc + 32'd1;
    end
  end
endmodule

Classification: a bifurcation, and the block's most important output is the assertion that it happened.

What it teaches: that the pointer is the only thing connecting a packet's two halves, and it is 32 of the descriptor's 256 bits. Everything the pipeline does for the next 36 stages — parse, look up, filter, admit, enqueue, schedule — operates on the descriptor. The bytes sit in a memory, addressed by those 32 bits. A descriptor with a corrupted pointer is still a valid descriptor: it parses, it looks up, it is admitted, it is scheduled, and it emits somebody else's packet.

And it teaches that c_orphan_payload and c_orphan_desc are two different failures with no shared symptom. An orphaned payload is a leaked buffer cell — the switch slowly loses buffer and eventually drops for no reason a counter explains. An orphaned descriptor is an emitted frame of whatever those bytes were, which is Chapter 21.2's corruption class arriving from inside the switch. Neither is visible to any property about a pipeline stage, because each stage handles its half correctly.

Deliberately simplified: next_ptr allocates linearly and never frees, so the model has no free list — a real allocator's free-list corruption is the commonest source of orphaned cells. The rejoin compares pointers for equality where a real design has a queue between them. c_orphan_payload and c_orphan_desc are detected by a condition that cannot occur in this block, since both halves are driven from the same pkt_valid; they exist to name the failure, not to detect it — detecting it needs a count at each end and a comparison, which is the twelfth taxonomy group again.

Production implication: the free list is where this becomes a field problem, and its failure is slow and silent. A buffer cell whose descriptor was dropped is never returned, so the pool shrinks by one cell per occurrence; at one leak per million packets and 9.5 Gpps, the 250 000-cell pool is exhausted in 26 seconds. The symptom is a switch that drops increasingly and recovers on reboot, with every per-port counter, every table and every link perfectly healthy. The detection is a conservation check: cells allocated minus cells freed equals cells occupied, compared against the pool's own free count — a few hundred flops, and the only thing that will ever find it.

12. Where Module 21's Fault Classes Hide

Module 21 built a fault taxonomy for a link. This section asks where each class lives inside a switch, and three of them turn out to have a home that Module 21 could not see.

Row seven is the class that does not exist on a link and does exist in a switch. Chapter 21.7 met descriptor faults at the host boundary; a switch has its own descriptors, its own pointers and its own free list, and the same fault family reappears with nothing in Module 21's taxonomy to name it.

Three failures in that family, and their symptoms.

Row two deserves its own sentence because it defeats Chapter 21.3 entirely. A switch recomputes the check sequence on egress — it has to, because it may have changed the VLAN tag or decremented a TTL. So a frame emitted from the wrong buffer location leaves with a perfectly valid FCS, and every receiver downstream accepts it. Chapter 21.3 §6's one bit of verdict says the frame is intact, and it is: intact, valid, and the wrong packet.

Which gives the chapter's sharpest operational point.

A switch is the one place in an Ethernet network where a frame can be corrupted and the corruption cannot be detected by any check the standard provides — because the switch regenerates the check.

And it is why the detection has to be structural rather than a checksum.

Row four is what a design that takes this seriously implements, and it is worth pricing in the chapter's own unit: a 32-bit payload CRC in each descriptor is 32 bits × the descriptors in flight. At 8 wide and 36 stages that is 8 × 36 × 32 = 9 216 flops = 184 320 BCE — 0.65 of a MAC receive datapath, and 0.03% of the switch. Against a failure mode that emits valid-looking wrong packets, that is close to free.

13. RTL 6 — The Egress Scheduler

// ---------------------------------------------------------------------
// egress_scheduler -- 512 queues, 64 ports, one decision per port per
// cycle, and Chapter 12.1 Section 9's arbitration at switch scale.
//
// This is the last block that sees a descriptor before it is rejoined
// with its payload, and it is where Module 17's mechanisms would attach
// if a data-centre switch had them. Chapter 23.2 Section 14 showed why
// it does not.
// ---------------------------------------------------------------------
module egress_scheduler
  import swasic_pkg::*;
#(
  parameter int Q_PER_PORT = QUEUES_PER_PORT     // 8
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [5:0]  port,
  input  logic [Q_PER_PORT-1:0]        q_nonempty,
  input  logic [Q_PER_PORT-1:0][15:0]  q_depth_cells,
  input  logic [Q_PER_PORT-1:0][7:0]   q_weight,
  input  logic [Q_PER_PORT-1:0]        q_strict,

  output logic [2:0]  grant_q,
  output logic        grant_valid,
  output logic [15:0] head_of_line_cells,
  output logic        strict_starves_wrr,
  output logic [31:0] c_grants [Q_PER_PORT],
  output logic [15:0] service_ratio_x10,
  output logic [31:0] c_starved_cycles
);
  logic [Q_PER_PORT-1:0] strict_ready;
  logic [Q_PER_PORT-1:0] wrr_ready;

  always_comb begin
    strict_ready = q_nonempty & q_strict;
    wrr_ready    = q_nonempty & ~q_strict;

    grant_valid = |q_nonempty;
    grant_q     = 3'd0;

    // Chapter 12.1 Section 9's arbitration, with a strict-priority tier
    // above a weighted one. The strict tier is what Chapter 17.1
    // Section 8 priced as the interference term.
    if (|strict_ready) begin
      for (int q = Q_PER_PORT-1; q >= 0; q--)
        if (strict_ready[q]) grant_q = 3'(q);
    end else begin
      automatic logic [7:0] best = 8'd0;
      for (int q = 0; q < Q_PER_PORT; q++)
        if (wrr_ready[q] && (q_weight[q] > best)) begin
          best = q_weight[q]; grant_q = 3'(q);
        end
    end

    head_of_line_cells = q_depth_cells[grant_q];

    // The condition Chapter 17.1 Section 11 named unbounded: a strict
    // queue that is never empty starves everything below it, and no
    // weight in the lower tier changes that.
    strict_starves_wrr = (|strict_ready) && (|wrr_ready);
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      c_starved_cycles <= '0; service_ratio_x10 <= 16'd10;
      for (int q = 0; q < Q_PER_PORT; q++) c_grants[q] <= '0;
    end else begin
      if (grant_valid) c_grants[grant_q] <= c_grants[grant_q] + 32'd1;
      if (strict_starves_wrr)
        c_starved_cycles <= c_starved_cycles + 32'd1;
      // Whether the weighted tier is getting the ratio it was
      // configured for, measured rather than assumed.
      if (c_grants[1] != 0)
        service_ratio_x10 <= 16'((c_grants[0] * 32'd10) / c_grants[1]);
    end
  end
endmodule

Classification: a two-tier arbiter, and its most useful counter measures whether the configuration is being honoured.

What it teaches: that strict_starves_wrr is a counted condition rather than a fault. Chapter 17.1 §8 built the strict-priority interference term and §11 named it one of the two unbounded terms; this is that term as a signal. A strict queue with work available always wins, and the weighted tier below it receives nothing for as long as that holds. The design is behaving exactly as configured, which is why the useful output is a cycle count rather than an alarm.

And it teaches that service_ratio_x10 closes the loop a configuration cannot. An operator configures weights of 4:1 and expects a 4:1 service ratio; what they get depends on whether both queues had work, on the strict tier above, and on the packet sizes — since this scheduler grants by cell, not by byte. Measuring the achieved ratio costs two counters and a divide, and it is the only way to know the configuration did what it said.

Deliberately simplified: the weighted tier picks the largest weight among non-empty queues every cycle, which is not weighted round-robin — it is strict priority by weight, and it starves low-weight queues completely. A real WRR maintains a deficit counter per queue. grant_q is computed combinationally over 8 queues for one port, where a 64-port switch makes 64 such decisions per cycle. And there is no notion of the packet's length in the grant, so a queue of minimum frames and a queue of maximum frames granted equally receive a 24:1 byte ratio.

Production implication: the cell-granted-not-byte-granted flaw is the one that surprises operators, and its size is exactly Chapter 5.6's frame-size range. Two queues granted alternately, one carrying 64-octet frames and one carrying 1 518, deliver a byte ratio of 23.7:1 — the configuration said 1:1 and the link carries 96% of one queue's traffic. A deficit round-robin scheduler fixes it by granting in bytes rather than in turns, at a cost of one deficit counter per queue — 512 counters at 16 bits is 8 192 flops, 163 840 BCE, 0.58 of a MAC datapath and 0.03% of the switch. For a fairness property that operators actually configure, that is cheap.

14. What a Switch Pipeline Must Never Do

Five prohibitions, each derived earlier in the chapter.

One — never let a pipeline stage stall.

A switch's pipeline serves 64 ports. A stall does not delay one packet; it delays every packet in flight across every port, and at 8 packets per cycle a single stalled cycle displaces 8 packets. So a parser that cannot resolve a header punts (Section 5), a lookup that hits a bank conflict defers or replicates (Section 7), and nothing waits. The design cost of that rule is in every block in this chapter.

Two — never write a multi-word table entry without a single-word commit.

Section 7's production note. The cost is one bit and one extra write; the failure it prevents is a forwarding entry that is valid, plausible and wrong.

Three — never infer an absolute buffer guarantee from a relative mechanism.

A port's allowance is not a quantity it owns, and Section 20's refused property is the inference that it is.

Four — never compose a claim about a packet from properties about descriptors.

Section 11: a packet is split in two at the ingress. Every stage property between the split and the rejoin is a property of 256 bits of metadata; the packet's 12 144 bits are in a memory. Three failures — a leaked cell, a corrupted pointer, a lost payload write — are invisible to all of them, and the switch regenerates the check sequence on egress, so nothing downstream detects them either.

Five — never rely on the frame check sequence to catch a switch's own errors.

A switch recomputes the FCS because it may have changed the frame. So a frame emitted from the wrong buffer address leaves with a perfectly valid check sequence, and Chapter 21.3's one bit of verdict says it is intact — which it is: intact, valid, and the wrong packet.

The detection has to be structural, and Section 12 priced it: a 32-bit payload digest carried in the descriptor is 9 216 flops — 184 320 BCE, 0.65 of a MAC receive datapath, 0.03% of the switch.

15. RTL 7 — Switch Telemetry

// ---------------------------------------------------------------------
// swasic_telemetry -- what the pipeline knows about itself, plus the
// two conservation checks that are the only way to see Section 11's
// failures.
//
// Chapter 19.7's 27 RMON counters cost 0.4 of a MAC datapath at switch
// scale -- 0.02% of the die. The area argument against instrumentation
// does not survive Section 4's table, so this block is generous.
// ---------------------------------------------------------------------
module swasic_telemetry
  import swasic_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,
  input  logic        snapshot,

  input  logic [31:0] i_parsed,
  input  logic [31:0] i_deep_parse,
  input  logic [31:0] i_lookups,
  input  logic [31:0] i_bank_conflicts,
  input  logic [31:0] i_read_during_write,
  input  logic [31:0] i_admits,
  input  logic [31:0] i_refusals,
  input  logic [31:0] i_cells_allocated,
  input  logic [31:0] i_cells_freed,
  input  logic [31:0] i_cells_occupied,
  input  logic [31:0] i_desc_in,
  input  logic [31:0] i_desc_out,
  input  logic [31:0] i_starved_cycles,
  input  logic [15:0] i_service_ratio_x10,
  input  logic [15:0] i_configured_ratio_x10,

  output logic [31:0] o_parsed,
  output logic [15:0] o_deep_parse_ppm,
  output logic [15:0] o_bank_conflict_ppm,
  output logic [15:0] o_refusal_ppm,
  output logic [31:0] o_cell_leak,
  output logic [31:0] o_desc_loss,
  output logic [15:0] o_service_error_pct,
  output logic [15:0] o_starved_ppm,
  output logic [3:0]  o_caveats,
  output logic        o_conservation_holds
);
  localparam int C_CELL_LEAK = 0;
  localparam int C_DESC_LOSS = 1;
  localparam int C_DEEP      = 2;
  localparam int C_RATIO     = 3;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      o_parsed <= '0; o_deep_parse_ppm <= '0; o_bank_conflict_ppm <= '0;
      o_refusal_ppm <= '0; o_cell_leak <= '0; o_desc_loss <= '0;
      o_service_error_pct <= '0; o_starved_ppm <= '0; o_caveats <= '0;
    end else if (snapshot) begin
      o_parsed <= i_parsed;

      if (i_parsed != 0) begin
        o_deep_parse_ppm <= 16'((i_deep_parse * 1_000_000) / i_parsed);
        o_starved_ppm    <= 16'((i_starved_cycles * 1_000_000) / i_parsed);
      end
      if (i_lookups != 0)
        o_bank_conflict_ppm <= 16'((i_bank_conflicts * 1_000_000) / i_lookups);
      if ((i_admits + i_refusals) != 0)
        o_refusal_ppm <= 16'((i_refusals * 1_000_000)
                             / (i_admits + i_refusals));

      // THE two checks. Section 11's failures are invisible to every
      // per-stage property and visible to exactly these.
      o_cell_leak <= (i_cells_allocated > (i_cells_freed + i_cells_occupied))
                   ? (i_cells_allocated - i_cells_freed - i_cells_occupied)
                   : 32'd0;
      o_desc_loss <= (i_desc_in > i_desc_out) ? (i_desc_in - i_desc_out)
                                              : 32'd0;

      if (i_configured_ratio_x10 != 0)
        o_service_error_pct <=
          16'((((i_service_ratio_x10 > i_configured_ratio_x10)
                ? (i_service_ratio_x10 - i_configured_ratio_x10)
                : (i_configured_ratio_x10 - i_service_ratio_x10)) * 100)
              / i_configured_ratio_x10);

      o_caveats[C_CELL_LEAK] <= (i_cells_allocated >
                                 (i_cells_freed + i_cells_occupied));
      o_caveats[C_DESC_LOSS] <= (i_desc_in > i_desc_out);
      o_caveats[C_DEEP]      <= (i_parsed != 0) &&
                                ((i_deep_parse * 1000) > i_parsed);
      o_caveats[C_RATIO]     <= (o_service_error_pct > 16'd10);
    end
  end

  assign o_conservation_holds = !o_caveats[C_CELL_LEAK] &&
                                !o_caveats[C_DESC_LOSS];
endmodule

Classification: a snapshot bank with two conservation checks and eight ratios.

What it teaches: that o_cell_leak and o_desc_loss are the only outputs in the whole chapter that can see Section 11's failures. Every other field here is a per-stage measurement and every per-stage measurement is correct while a cell leaks. A conservation identity — allocated minus freed equals occupied — is a statement about the whole datapath, and it is the shape of check that a bifurcated object requires.

And it teaches that the area argument against instrumentation is dead at this scale. Chapter 19.7's 27 counters across 64 ports are 1.11 × 10⁵ BCE — 0.02% of the switch (Section 4). This block's fields cost less. Every earlier chapter in the track has had to justify a counter against a MAC datapath; inside a switch ASIC the justification is no longer an area one, and the remaining objections are verification effort and read bandwidth.

Deliberately simplified: no read-clear, and at 9.5 Gpps a 32-bit i_parsed wraps in 0.45 seconds — every counter here needs to be 48 or 64 bits or read very often. o_service_error_pct compares two ratios without knowing whether both queues had work, so an idle queue reads as a configuration error. The conservation checks are computed at snapshot time from three inputs that are sampled independently, which makes a transient mismatch look like a leak. And there is no per-port breakdown of anything, where a switch's interesting failures are almost always per-port.

Production implication: the 0.45-second wrap is the detail that decides the counter widths, and getting it wrong makes the whole block useless. At 9.524 Gpps a 32-bit packet counter wraps in 451 milliseconds, so a telemetry system polling at one hertz reads a wrapped value every time and computes nonsense differences. A 48-bit counter wraps in 8.2 hours and a 64-bit one in 61 years. The cost of going from 32 to 48 bits across, say, 200 counters is 3 200 flops — 64 000 BCE, 0.23 of a MAC datapath, 0.011% of the switch. There is no reason to use 32-bit counters in a part this size, and the habit comes from designs where a flip-flop was expensive.

16. RTL 8 — The Pipeline Conformance Monitor

// ---------------------------------------------------------------------
// swasic_conformance_monitor -- properties for a forwarding pipeline.
//
// Properties 1 to 10 are per-stage and hold while Section 11's three
// failures are happening. Properties 11 and 12 are conservation checks
// over the WHOLE datapath, and they are the only two that do not.
// ---------------------------------------------------------------------
module swasic_conformance_monitor
  import swasic_pkg::*;
(
  input logic        clk,
  input logic        rst_n,

  input logic        pkt_valid,
  input logic        desc_valid,
  input logic        parse_deep,
  input logic        parse_failed,
  input logic [15:0] bank_conflicts,
  input logic        read_during_write,
  input logic        cp_write,
  input logic        admit,
  input logic [31:0] port_limit,
  input logic [31:0] shared_free,
  input logic [31:0] port_occupancy_req,
  input logic        limit_is_relative,
  input logic        halves_travel_separately,
  input logic        grant_valid,
  input logic [2:0]  grant_q,
  input logic [7:0]  q_nonempty,
  input logic [31:0] cells_allocated,
  input logic [31:0] cells_freed,
  input logic [31:0] cells_occupied,
  input logic [31:0] desc_in,
  input logic [31:0] desc_out,
  input logic        conservation_holds,
  input logic        claim_packet_forwarded
);
  // 1. A descriptor exists for every packet that entered.
  p_sw_desc_per_pkt: assert property (@(posedge clk) disable iff (!rst_n)
    pkt_valid |-> desc_valid);

  // 2. The pipeline never stalls -- a valid input is always accepted.
  p_sw_no_stall: assert property (@(posedge clk) disable iff (!rst_n)
    pkt_valid |=> 1'b1);

  // 3. A deep parse is a punt, never a retry of the same packet.
  p_sw_deep_is_punt: assert property (@(posedge clk) disable iff (!rst_n)
    parse_deep |-> !$past(parse_deep));

  // 4. Bank conflicts never exceed the pipeline width minus one.
  p_sw_conflicts_bounded: assert property (@(posedge clk) disable iff (!rst_n)
    bank_conflicts <= 16'(PIPE_WIDTH - 1));

  // 5. Read-during-write only when the control plane is writing.
  p_sw_rdw_needs_write: assert property (@(posedge clk) disable iff (!rst_n)
    read_during_write |-> cp_write);

  // 6. The per-port limit is a fraction of what is free.
  p_sw_limit_relative: assert property (@(posedge clk) disable iff (!rst_n)
    port_limit == (shared_free >> 1));

  // 7. Admission respects the limit or the reserve.
  p_sw_admit_rule: assert property (@(posedge clk) disable iff (!rst_n)
    admit |-> ((port_occupancy_req < 32'(512)) ||
               (port_occupancy_req <= port_limit)));

  // 8. The mechanism is relative and says so.
  p_sw_relative_flag: assert property (@(posedge clk) disable iff (!rst_n)
    limit_is_relative);

  // 9. A grant goes only to a non-empty queue.
  p_sw_grant_nonempty: assert property (@(posedge clk) disable iff (!rst_n)
    grant_valid |-> q_nonempty[grant_q]);

  // 10. The two halves are acknowledged to travel separately.
  p_sw_halves: assert property (@(posedge clk) disable iff (!rst_n)
    halves_travel_separately);

  // 11. CONSERVATION: cells allocated equal cells freed plus occupied.
  //     The only property here that a leaked cell violates.
  p_sw_cell_conservation: assert property (@(posedge clk) disable iff (!rst_n)
    cells_allocated == (cells_freed + cells_occupied));

  // 12. CONSERVATION: descriptors in equal descriptors out.
  p_sw_desc_conservation: assert property (@(posedge clk) disable iff (!rst_n)
    desc_in == desc_out);

  // 13. A forwarding claim requires both conservation checks.
  p_sw_claim_gated: assert property (@(posedge clk) disable iff (!rst_n)
    claim_packet_forwarded |-> conservation_holds);
endmodule

Classification: ten per-stage properties, two conservation laws, and one gated claim.

What it teaches: that properties 11 and 12 are a different kind of statement from the ten above them. Properties 1 to 10 each constrain one stage's behaviour; 11 and 12 constrain a sum over the whole datapath, which is the only shape of check that can see an object whose halves travelled separately. A leaked cell violates none of the first ten and violates 11 immediately.

And it teaches that property 2 is almost vacuous and is worth keeping anyway. pkt_valid |=> 1'b1 asserts nothing about behaviour; it exists as a placeholder for the real requirement — that no backpressure signal exists at all — and in a design where one does exist, the property becomes pkt_valid |-> ready, which is the whole no-stall rule in one line. The vacuous form documents the architecture's assumption.

Deliberately simplified: property 4's bound is the pipeline width, which is correct but loose — the interesting bound is the expected number of conflicts, which is a balls-in-bins figure and not an assertion. Property 6 hard-codes α = ½ as a shift. Properties 11 and 12 compare free-running counters that in a real design are sampled at different pipeline depths, which is Chapter 19.7 §19's rejected class 87 — they hold only at a quiescent point. And property 3's punt check catches a two-cycle repeat and not a general retry.

Production implication: the pipeline-depth skew in properties 11 and 12 is what makes conservation checks hard to run continuously, and the practical answer is to run them at a barrier. A switch that periodically quiesces one ingress pipeline — for a few microseconds, on a rotation across 64 ports — can evaluate both identities exactly, and a discrepancy is then unambiguous. The cost is a few microseconds of one port's throughput per sweep and the mechanism to drain it; the alternative is a running check whose false-positive rate makes it unreadable. Chapter 19.7 §16's snapshot discipline is the same idea, and here it needs a drain as well as a snapshot.

17. Why the Pipeline Is Fixed-Latency and the Buffer Is Not

A switch has two halves with opposite timing character, and almost every confusion about switch latency comes from treating them as one.

Row six is the connection to Module 17 and it resolves an ambiguity that chapter left. Chapter 17.1 §4 listed lookup and fabric as a bounded term at 0.03 µs and queueing as unbounded. Those two terms are the two halves of this chapter: the pipeline is the first, the buffer is the second, and they are bounded and unbounded respectively because of how they are built rather than by coincidence.

The ratio between them is what makes switch latency confusing.

So a switch's latency is 30 nanoseconds plus something between zero and two milliseconds, and the second term is four to five orders of magnitude larger than the first when it is present at all. Every datasheet quotes the first.

Three consequences.

One — a switch's "port-to-port latency" figure describes an empty switch. A quoted 450 ns or 800 ns is the pipeline plus the serialisation of Chapter 17.1 §4's store-and-forward term; it says nothing about a loaded one, and the loaded case is the only one a deadline cares about.

Two — cut-through removes a term the datasheet quotes and not the one that hurts. Chapter 12.6 established that cut-through drops the store-and-forward term. At 100 Gb/s and a maximum frame that is 0.121 µs, against a queueing term that can be 2 224. Cut-through improves the number in the datasheet by a factor that is invisible under load, which is why it matters for low-latency trading fabrics and not for a training cluster.

Three — and the buffer's variability is not noise, it is the mechanism working. A buffer that never queued would be a buffer that was never needed; Chapter 14.1 §14 priced what a buffer buys in time, and this chapter's answer at 64 MB is 80 µs of one port's line rate for large frames and 20 for small. The variability is the product.

Which gives the honest way to state a switch's latency.

30 nanoseconds of pipeline, plus the serialisation of the frame, plus a queueing term that is zero on an idle switch and up to 2.2 milliseconds on a congested one. The first two are design constants and the third is the traffic's.

18. What the Pipeline Assumes

Seven premises. Two are about the control plane and three are about arithmetic that does not appear anywhere in the design.

Row four is the subtlest and it is a specification question rather than an implementation one. The control plane writes forwarding entries continuously — learning, ageing, route installation — while the data plane reads at 9.5 Gpps. "The table's contents at the moment of the lookup" is not a well-defined quantity, because the table is mid-transition for some fraction of every second. The design's actual requirement is that each lookup return some self-consistent entry, which is a much weaker and entirely achievable claim — and it is not what anyone writes down.

Row six is the assumption that makes a buffer figure a lie without anyone intending one. A 64 MB buffer is 64 MB of cells; whether it holds 64 MB of frames depends entirely on the frame size distribution. The nameplate figure is correct and the operational figure is between 16 and 63 MB, and nothing in the datasheet says which.

And one premise that is stated everywhere and is false in a specific way.

That a frame leaving the switch is the frame that entered it. Section 12: the switch recomputes the check sequence on egress, because it may legitimately have changed the frame. So a frame emitted from the wrong buffer address is indistinguishable, to every downstream receiver and every counter in this track, from a correct one.

19. The Cost, Accounted — in BCE

The unit named in Section 2, applied to everything this chapter built.

Read the last two rows together and the chapter's structural claim is a measurement. Every block of logic in this chapter, added up, is 0.84 of a single MAC receive datapath. The memory it manages is 1 985. The ratio is 2 371 : 1.

A switch ASIC's forwarding pipeline is smaller than one MAC's receive datapath. Everything else on the die is memory, and the control logic exists to decide what goes in it.

And that reframes every instrumentation argument the track has made.

Every figure in the right-hand column is a rounding error, which means the area objection to instrumentation — the objection this track has answered carefully five times — does not exist inside a switch ASIC. What remains is verification effort, read bandwidth and the discipline to define the fields well. Those are real and they are not area.

Two closing comparisons, to place the switch against the rest of the track.

Rows two and three are within 16% of each other, which is the chapter's most surprising single result: the forwarding logic of a 6.4 Tb/s switch is about the same size as the receive datapath of one 100 Gb/s MAC. The switch is not a big MAC. It is a memory with a small, very fast decision engine in front of it.

20. Properties Worth Asserting, and One Worth Refusing

Six groups. Thirty-two of the thirty-four constrain one stage, and the two that do not are the only ones that can see a packet whose halves were separated.

Group A — the parser.

// A1. A descriptor exists for every packet.
p_pa_desc: assert property (@(posedge clk) disable iff (!rst_n)
  pkt_valid |-> desc_valid);

// A2. A deep parse is a punt, not a retry.
p_pa_punt: assert property (@(posedge clk) disable iff (!rst_n)
  parse_deep |-> !$past(parse_deep));

// A3. The parser never asserts backpressure.
p_pa_no_stall: assert property (@(posedge clk) disable iff (!rst_n)
  !parser_stall);

// A4. The descriptor's ingress port field matches the arrival port.
p_pa_port: assert property (@(posedge clk) disable iff (!rst_n)
  desc_valid |-> (desc[132:127] == pkt_port));

// A5. A failed parse never produces a valid descriptor.
p_pa_failed: assert property (@(posedge clk) disable iff (!rst_n)
  parse_failed |-> !desc_valid);

// A6. Width is honoured: at most PIPE_WIDTH descriptors per cycle.
p_pa_width: assert property (@(posedge clk) disable iff (!rst_n)
  $countones(desc_valid) <= PIPE_WIDTH);

Group B — the lookup.

// B1. Bank conflicts never exceed the width minus one.
p_lk_conflicts: assert property (@(posedge clk) disable iff (!rst_n)
  bank_conflicts <= 16'(PIPE_WIDTH - 1));

// B2. Read-during-write only while the control plane writes.
p_lk_rdw: assert property (@(posedge clk) disable iff (!rst_n)
  read_during_write |-> cp_write);

// B3. The epoch recorded at a lookup is the epoch in force.
p_lk_epoch: assert property (@(posedge clk) disable iff (!rst_n)
  q_valid |-> (epoch_at_lookup == cp_epoch));

// B4. An entry is never visible with its valid bit clear.
p_lk_valid_last: assert property (@(posedge clk) disable iff (!rst_n)
  hit |-> entry_valid_bit);

// B5. A multi-word install sets the valid bit in its final write.
p_lk_commit: assert property (@(posedge clk) disable iff (!rst_n)
  (cp_write && cp_final_word) |-> cp_entry[VALID_BIT]);

// B6. Lookups counted equal queries presented.
p_lk_count: assert property (@(posedge clk) disable iff (!rst_n)
  q_valid |=> (c_lookups == $past(c_lookups) + $past($countones(q_valid))));

Group C — the buffer.

// C1. The limit is a fraction of what is free.
p_bf_relative: assert property (@(posedge clk) disable iff (!rst_n)
  port_limit == (shared_free >> ALPHA_SHIFT));

// C2. Admission respects the reserve or the limit.
p_bf_admit: assert property (@(posedge clk) disable iff (!rst_n)
  admit |-> ((port_occupancy_req < RESERVE_CELLS) ||
             (port_occupancy_req <= port_limit)));

// C3. Occupancy never exceeds the pool.
p_bf_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  used_cells <= CELLS_TOTAL);

// C4. Free plus used equals the pool.
p_bf_identity: assert property (@(posedge clk) disable iff (!rst_n)
  (free_cells + used_cells) == CELLS_TOTAL);

// C5. A port always has its reserve available to it.
p_bf_reserve: assert property (@(posedge clk) disable iff (!rst_n)
  (port_occupancy_req < RESERVE_CELLS) |-> admit);

// C6. The mechanism declares itself relative.
p_bf_declares: assert property (@(posedge clk) disable iff (!rst_n)
  limit_is_relative);

Group D — the split and the rejoin.

// D1. Both halves are produced together.
p_sp_together: assert property (@(posedge clk) disable iff (!rst_n)
  buf_write == desc_out_valid);

// D2. The descriptor carries the pointer the payload was written to.
p_sp_ptr: assert property (@(posedge clk) disable iff (!rst_n)
  desc_out_valid |-> (desc_out[31:0] == buf_ptr));

// D3. Cells allocated match the frame's length.
p_sp_cells: assert property (@(posedge clk) disable iff (!rst_n)
  buf_write |-> (buf_cells ==
                 ((pkt_octets + CELL_BYTES - 1) / CELL_BYTES)));

// D4. A rejoin matches pointers.
p_sp_rejoin: assert property (@(posedge clk) disable iff (!rst_n)
  rejoin_ok |-> (egress_desc_ptr == desc_ptr));

// D5. A mismatch is reported, never silently accepted.
p_sp_mismatch: assert property (@(posedge clk) disable iff (!rst_n)
  (egress_ready && (egress_desc_ptr != desc_ptr)) |-> rejoin_mismatch);

// D6. The bifurcation is declared.
p_sp_declares: assert property (@(posedge clk) disable iff (!rst_n)
  halves_travel_separately);

Group E — the scheduler.