MAC Block Architecture

Module 18 treated the MAC's internals as a boundary. Module 19 opens it, and this chapter lays out the blocks and the clock domains each sits in — in the way Chapter 18.1 laid out the interfaces, by making the numbers do the arguing.

Here is the number the whole chapter turns on.

One and a third clock cycles per frame.

Which settles an architectural question before any block is drawn: every per-frame operation in the MAC — address filtering, length checking, statistics, the DMA request — has 0.312 cycles of slack after the data beat. So no per-frame block can be a state machine that runs once per frame; all of them must be pipelined across frames, and several frames are in flight at once.

And there is a second consequence that follows from the same arithmetic and is easier to miss.

The gap between frames is 20 octets — Chapter 5.2's 8 octets of preamble and SFD plus Chapter 8.3's 12 octets of interframe gap. A 512-bit beat is 64 octets.

Since 20 is less than 64, a single beat can contain the end of one frame and the beginning of the next.

Above 128 bits, every per-beat block must be able to hold two frames' state simultaneously — a CRC accumulator finishing one frame and starting another in the same cycle, a parser closing one header and opening another. That is not an optimisation; it is a structural requirement of the interface width.

This chapter establishes both of those and then lays out the blocks against them.

1. Scope, and What Module 19 Inherits

Modules 1 to 17 described what a MAC does. Module 18 described how it attaches to a system. This module builds it, and this chapter is the floorplan.

What this chapter does not build: the parser is Chapter 19.2, the assembler Chapter 19.3, the CRC integration Chapter 19.4, the FIFOs Chapter 19.5, the memory interface Chapter 19.6 and the statistics counters Chapter 19.7. Those forward references are bold and unlinked because those chapters are not yet published.

And it inherits four domains from Chapter 18.1 §6, unchanged:

That chapter established the crossings and their cost. This one establishes which block sits where, which is a different question and is the one that decides the design's shape.

2. The Blocks, and Which Domain Each Sits In

A MAC is about a dozen blocks. Listing them with their domain settles most of the architecture, because the domain decides what a block may talk to directly.

Receive path, in the recovered clock:

The crossing:

Transmit path, in the transmit clock:

Host domain:

Two placements in those tables are worth arguing about, because both are frequently got wrong.

The address filter is in the receive domain and not the host domain. Chapter 7.4's filter decides whether a frame is kept, and a frame that is not kept should not consume Chapter 18.1 §9's FIFO or any of Module 18's bandwidth. Filtering after the crossing works and wastes the whole receive path on frames destined elsewhere — which on a promiscuous-adjacent link is most of them.

And the statistics counters are in the datapath domains, not the host domain, for the same reason Chapter 18.1 §14's telemetry had to be: the events they count happen there, and a counter in the host domain would need every event crossed individually. Their readback is the crossing, and that is one handshake for the whole set.

3. RTL 1 — A Receive Pipeline Stage

The shape every block in the receive path takes, and the two fields that make it work at 100 Gb/s.

// -----------------------------------------------------------------------
// macarch_pkg -- the MAC's internal datapath and domain map.
// -----------------------------------------------------------------------
package macarch_pkg;

  localparam int DP_BYTES = 64;          // 512-bit at 100 Gb/s
  localparam int IFG_B    = 12;          // 8.3
  localparam int PRE_B    = 8;           // 5.2
  localparam int GAP_B    = IFG_B + PRE_B;

  // Which clock domain a block lives in. 18.1 section 6 established
  // the four; this is where each BLOCK is placed.
  typedef enum logic [1:0] {
    DOM_RX   = 2'd0,
    DOM_TX   = 2'd1,
    DOM_HOST = 2'd2,
    DOM_PTP  = 2'd3
  } domain_e;

  // The internal beat. Two fields make 100 Gb/s possible and are
  // absent from a naive design: `frame_id`, because several frames
  // are in the pipeline at once, and `second_frame_offset`, because
  // above a 128-bit datapath one beat can carry the end of one frame
  // and the start of the next -- GAP_B is 20 and a beat is 64.
  typedef struct packed {
    logic [DP_BYTES*8-1:0]            data;
    logic [$clog2(DP_BYTES+1)-1:0]    bytes;
    logic                             sof;
    logic                             eof;
    logic                             err;
    logic [7:0]                       frame_id;      // pipeline tag
    logic                             has_second;    // two frames here
    logic [$clog2(DP_BYTES)-1:0]      second_offset; // where the next begins
    logic [7:0]                       second_id;
  } beat_t;

  // A block's per-frame result, carried alongside rather than
  // computed in the frame's own cycles -- section 6.
  typedef struct packed {
    logic [7:0]  frame_id;
    logic        valid;
    logic [15:0] length;
    logic        addr_match;
    logic        crc_ok;
    logic        runt;
    logic        giant;
  } frame_result_t;

endpackage

// -----------------------------------------------------------------------
// rx_pipeline_stage -- the template every receive block follows.
//
// One beat in, one beat out, one cycle of latency, no stalling. The
// per-frame work happens on a SIDE channel tagged by frame_id,
// because section 4's budget is 1.31 cycles per frame and a
// per-frame computation cannot fit inside one frame's beats.
// -----------------------------------------------------------------------
module rx_pipeline_stage
  import macarch_pkg::*;
#(
  parameter int STAGE_ID = 0
)(
  input  logic          clk_rx,
  input  logic          rst_rx_n,

  input  beat_t         in_beat,
  input  logic          in_valid,

  output beat_t         out_beat,
  output logic          out_valid,

  // The side channel: a result for a frame that may have left this
  // stage several cycles ago.
  output frame_result_t result,
  output logic          result_valid,

  output logic [31:0]   c_beats,
  output logic [31:0]   c_frames,
  output logic [31:0]   c_dual_frame_beats,
  output logic          stage_stalled          // must never assert
);

  // A receive stage may NOT stall. There is no backpressure toward
  // the wire -- 7.2 -- so a stage that cannot accept a beat drops
  // data. Every receive stage is therefore fixed-latency and
  // unconditional, and the signal exists to prove it.
  assign stage_stalled = 1'b0;

  always_ff @(posedge clk_rx or negedge rst_rx_n) begin
    if (!rst_rx_n) begin
      out_beat  <= '0;
      out_valid <= 1'b0;
      c_beats <= '0; c_frames <= '0; c_dual_frame_beats <= '0;
    end else begin
      out_beat  <= in_beat;
      out_valid <= in_valid;

      if (in_valid) begin
        c_beats <= c_beats + 1;
        if (in_beat.eof) c_frames <= c_frames + 1;
        // A beat carrying two frames is the 100 Gb/s case and it is
        // worth counting, because a design that has never seen one
        // has not been tested at rate.
        if (in_beat.has_second)
          c_dual_frame_beats <= c_dual_frame_beats + 1;
      end
    end
  end

endmodule

Classification: a fixed-latency, non-stalling pipeline register with a tagged side channel.

What it teaches: that frame_id is not optional above about 10 Gb/s, and its absence is what makes a naive design fail at rate. Section 4's budget gives 1.31 cycles per frame at 100 Gb/s, and a pipeline of any depth therefore holds several frames simultaneously — so a per-frame result computed in stage 5 belongs to a frame that left stage 2 four cycles ago. Without a tag there is no way to say which frame a result is about, and the design works only if exactly one frame is in flight, which it cannot be.

And it teaches that has_second is a structural requirement rather than an optimisation. The gap between frames is 20 octets; a 512-bit beat is 64. So a beat straddles a frame boundary whenever the previous frame's last octets and the next frame's first octets fall in the same 64-octet window — which at minimum frame size happens constantly. A block that assumes one frame per beat loses the leading octets of every frame that shares a beat.

Deliberately simplified: the stage does nothing except register — it is a template, and the real stages put combinational logic between input and output. stage_stalled is tied to zero rather than derived, which is the listing asserting the discipline rather than checking it. And the beat carries only one second_offset, so a beat containing parts of three frames — possible at a 1024-bit datapath, where a beat is 128 octets and two gaps fit — is not representable.

Production implication: c_dual_frame_beats should be non-zero on any port that has run at rate with small frames, and a value of zero is evidence the design has not been tested there. At 100 Gb/s with 64-octet frames essentially every beat carries two frames — the frame slot is 1.31 beats — so a regression reporting zero has been run with large frames or below line rate, and the whole dual-frame path is unexercised.

4. The Cycle Budget, Derived

Section 3 asserted 1.31 cycles per frame. This section derives it and the rest of the table, because the derivation is what makes the consequences unavoidable.

A frame occupies (F + 20) octets of wire time — its own octets plus Chapter 5.2's preamble and SFD and Chapter 8.3's interframe gap. At a line rate r that is (F + 20) × 8 / r seconds, and at a datapath clock f that is (F + 20) × 8 × f / r cycles.

And the data beats a frame needs, ceil(F × 8 / width):

Subtract for the slack:

Three things in that table decide the architecture.

First, the GMII row is 20 cycles at every frame size, which is the gap in octets at one octet per cycle. A 1 Gb/s MAC has twenty spare cycles between frames and can afford a per-frame state machine that takes fifteen of them.

Second, the 100 Gb/s row has essentially none. 0.312 cycles at minimum frame size and 0.031 at maximum — so a per-frame block has no cycles of its own at all, and Section 6 is about what to do instead.

Third, the bottom-right cell is negative, which looks like an error and is the dual-frame case appearing in the arithmetic. A 9000-octet frame needs 141 whole beats and its wire slot is 140.94 beat-times — the discrepancy is the 20-octet gap being smaller than a beat, so consecutive jumbo frames share a beat and the deficit is made up from the neighbour's slot. The arithmetic is not wrong; the assumption that a frame occupies whole beats is.

And the slack's behaviour across the table is worth stating as a rule, because it is the opposite of the instinct:

The faster the interface, the fewer cycles per frame — and the slack falls faster than the beats do, because the gap is a fixed number of octets and a beat is a growing one.

Twenty octets is twenty cycles on an 8-bit datapath and less than a third of one on a 512-bit datapath — and the whole of a MAC's per-frame housekeeping used to happen in that gap.

5. RTL 2 — A Transmit Pipeline Stage

The mirror, and it differs in one respect that follows directly from Section 11: a transmit stage may stall.

// -----------------------------------------------------------------------
// tx_pipeline_stage -- the transmit template.
//
// The asymmetry with section 3: a transmit stage may apply
// backpressure, because the design controls when frames go out. A
// receive stage may not, because the wire does not stop. That single
// difference changes the handshake and everything built on it.
// -----------------------------------------------------------------------
module tx_pipeline_stage
  import macarch_pkg::*;
#(
  parameter int STAGE_ID = 0
)(
  input  logic          clk_tx,
  input  logic          rst_tx_n,

  input  beat_t         in_beat,
  input  logic          in_valid,
  output logic          in_ready,        // the receive template has none

  output beat_t         out_beat,
  output logic          out_valid,
  input  logic          out_ready,

  // Once transmission has started, this stage MUST deliver every
  // remaining beat of the frame. 18.4 section 6: a gap mid-frame is
  // an underrun and the frame is already on the wire.
  input  logic          frame_committed,

  output logic [31:0]   c_beats,
  output logic [31:0]   c_stall_cycles,
  output logic          stalled_after_commit    // must never assert
);

  logic occupied;

  assign in_ready  = !occupied || out_ready;
  assign out_valid = occupied;

  // The invariant that makes the transmit path safe. Stalling before
  // a frame starts costs latency; stalling after it has started
  // costs the frame. 18.4's underrun guard is downstream of this and
  // catches the consequence; this catches the cause.
  assign stalled_after_commit = frame_committed && occupied && !out_ready;

  always_ff @(posedge clk_tx or negedge rst_tx_n) begin
    if (!rst_tx_n) begin
      occupied <= 1'b0; out_beat <= '0;
      c_beats <= '0; c_stall_cycles <= '0;
    end else begin
      if (in_valid && in_ready) begin
        out_beat <= in_beat;
        occupied <= 1'b1;
        c_beats  <= c_beats + 1;
      end else if (out_ready) begin
        occupied <= 1'b0;
      end

      if (occupied && !out_ready) c_stall_cycles <= c_stall_cycles + 1;
    end
  end

endmodule

Classification: a skid-buffer-free single-register stage with a commit-time stall invariant.

What it teaches: that in_ready exists on the transmit template and not on the receive one, and that one difference propagates through the whole design. A transmit stage may stall because the design decides when frames go out — Chapter 18.4 §16's conclusion that falling behind costs throughput and never data. A receive stage may not, because Chapter 7.2's path has no backpressure toward the partner: a stalled receive stage is a dropped frame.

And it teaches that stalled_after_commit catches the cause where Chapter 18.4 §8's guard catches the consequence. Once the first octet is on the wire there is no pause — so a transmit stage that stalls after commit produces an underrun, and the guard's job is then to corrupt the FCS so the far end discards the frame. The guard is a mitigation. This signal says the mitigation was needed, and which stage needed it.

Deliberately simplified: in_ready combinationally depends on out_ready, so backpressure ripples through the whole pipeline in one cycle — a real design inserts skid buffers and breaks the path every few stages. There is no skid buffer here at all, so the stage drops a beat if in_valid and !in_ready coincide, which the handshake is supposed to prevent and which nothing here checks. And frame_committed is an input from elsewhere, where a real stage derives it from having passed the first beat onward.

Production implication: c_stall_cycles per stage is how an integrator finds which stage is the transmit path's limit, and the answer is usually not the one people expect. The CRC generator and the assembler get the attention; the stall is almost always at the FIFO boundary — Chapter 18.4 §7's fill policy — because that is where the memory system's behaviour enters the datapath. Per-stage counters cost a few flops each and turn a pipeline into something that can be profiled.

6. Per-Beat Work and Per-Frame Work

Section 4 established that a per-frame block has 0.312 cycles at 100 Gb/s. This section is what a design does about it, and the answer is a structural change rather than a faster implementation.

Sort the MAC's blocks by billing unit:

The left column scales naturally: more octets per cycle means a wider datapath and the same number of cycles. A 512-bit CRC engine is sixty-four times the logic of an 8-bit one and runs at the same rate, which is Chapter 6.4's trade and is affordable.

The right column does not scale, and at 100 Gb/s it has 0.312 cycles.

Three responses exist and only one of them works.

Row three's mechanism is Section 3's frame_id. The per-frame block is given a frame's worth of metadata and a tag, takes as many cycles as it needs, and emits a result tagged with the same identifier. Something downstream matches the result to the frame, which has long since passed.

And the depth this buys is worth computing, because it is generous.

Twelve frames in flight at 100 Gb/s, which gives a per-frame block sixteen cycles to do work it had 0.312 cycles for — a factor of fifty-one, bought with a tag and a matching structure.

And the 1 Gb/s row shows why this is a high-rate technique. At 84 cycles per frame a 16-stage pipeline holds 0.19 frames — less than one — so the naive structure works, the tag is unnecessary, and a design written for 1 Gb/s and scaled will be missing it.

Which is the section's practical warning. The per-frame blocks in a 1 Gb/s MAC are state machines that run once per frame in the gap, and they are correct. The same blocks at 100 Gb/s are wrong in a way that is not a timing failure — they produce results for the wrong frames, which is a functional bug that synthesis will not flag and that appears only when frames are close together.

7. RTL 3 — The Domain Boundary Registry

A design artefact rather than a datapath block: it records which signal crosses which boundary, so that every crossing has an owner.

// -----------------------------------------------------------------------
// domain_boundary_registry -- an enumerated list of every crossing,
// and the mechanism each uses.
//
// 18.1 section 6 established four domains and their costs. The
// failure this block exists to prevent is not a wrong mechanism --
// it is a crossing nobody noticed, which is why the structure is a
// registry rather than a synchroniser.
// -----------------------------------------------------------------------
module domain_boundary_registry
  import macarch_pkg::*;
#(
  parameter int NUM_CROSSING = 12
)(
  input  logic       clk_host,
  input  logic       rst_host_n,

  // One entry per crossing, declared at elaboration.
  input  domain_e    from_domain [NUM_CROSSING],
  input  domain_e    to_domain   [NUM_CROSSING],
  input  logic [1:0] mechanism   [NUM_CROSSING],   // 0 sync, 1 fifo, 2 handshake
  input  logic [7:0] width       [NUM_CROSSING],

  // Observed activity, so an unused crossing is visible.
  input  logic [NUM_CROSSING-1:0] crossing_active,

  output logic [31:0] c_activity [NUM_CROSSING],
  output logic        same_domain_declared,     // a crossing that is not one
  output logic        multibit_on_synchroniser, // the classic bug
  output logic [15:0] crossings_never_active
);

  // A "crossing" whose two domains are the same is either a mistake
  // in the registry or a signal that used to cross and no longer
  // does. Both are worth knowing; neither is detectable from the RTL.
  logic same_dom;
  always_comb begin
    int i;
    same_dom = 1'b0;
    for (i = 0; i < NUM_CROSSING; i++)
      if (from_domain[i] == to_domain[i]) same_dom = 1'b1;
  end
  assign same_domain_declared = same_dom;

  // The single most common CDC bug: a multi-bit value on a two-flop
  // synchroniser. 18.1 section 14 established why -- the bits arrive
  // on different cycles and the value observed was never held.
  logic multibit;
  always_comb begin
    int i;
    multibit = 1'b0;
    for (i = 0; i < NUM_CROSSING; i++)
      if ((mechanism[i] == 2'd0) && (width[i] > 8'd1)) multibit = 1'b1;
  end
  assign multibit_on_synchroniser = multibit;

  logic [15:0] never;
  always_comb begin
    int i;
    never = '0;
    for (i = 0; i < NUM_CROSSING; i++)
      if (c_activity[i] == 32'd0) never = never + 16'd1;
  end
  assign crossings_never_active = never;

  always_ff @(posedge clk_host or negedge rst_host_n) begin
    int i;
    if (!rst_host_n)
      for (i = 0; i < NUM_CROSSING; i++) c_activity[i] <= '0;
    else
      for (i = 0; i < NUM_CROSSING; i++)
        if (crossing_active[i]) c_activity[i] <= c_activity[i] + 1;
  end

endmodule

Classification: a declarative registry with two static checks and one activity measurement.

What it teaches: that multibit_on_synchroniser is a check a registry can make and RTL inspection usually does not. Chapter 18.1 §14 established the rule — a multi-bit value on a two-flop synchroniser is read as a mixture of old and new bits — and the bug is easy to introduce because the synchroniser is the right mechanism for the single-bit case and looks right for the multi-bit one. A registry that records each crossing's width catches it at elaboration, before simulation.

And it teaches that crossings_never_active is worth measuring. A crossing that never fires in a full regression has not been verified, and clock-domain crossings are the one class of logic where "it was never exercised" is not a coverage gap but a reason the design may be broken and appear fine. A crossing exercised once per reset has been tested at exactly one phase relationship.

Deliberately simplified: everything is an elaboration-time constant fed as an input, which real designs express as a package of parameters or generate from a spreadsheet — and several tools generate the CDC report from the netlist rather than from a declaration. The registry cannot detect a crossing nobody declared, which is the failure it most wants to catch, so it is a discipline supported by a check rather than a check that stands alone.

Production implication: the registry's real value is as a document that is compiled. A CDC list in a specification drifts from the RTL within one revision; a list that is elaborated, checked and counted cannot drift without breaking a check. Chapter 18.2 §12's argument about reference drivers applies unchanged: a requirement embodied in code that is exercised is a requirement that survives, and one written in prose is one the second engineer does not read.

8. RTL 4 — The Cycle Budget Checker

Section 4's arithmetic, in hardware, so that a design running below its intended rate says so.

// -----------------------------------------------------------------------
// cycle_budget_checker -- measures the actual cycles per frame and
// compares against what the configured line rate implies.
//
// Section 4's table is a design-time calculation. This is the
// run-time measurement, and the gap between them is how a design
// discovers it is not running at the rate it believes.
// -----------------------------------------------------------------------
module cycle_budget_checker
  import macarch_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        beat_valid,
  input  logic        frame_eof,
  input  logic [15:0] frame_bytes,

  input  logic [15:0] cfg_line_rate_mbps,
  input  logic [31:0] cfg_clk_hz_div_1k,

  output logic [15:0] measured_cycles_x10,     // per frame, x10
  output logic [15:0] expected_cycles_x10,
  output logic [31:0] c_frames,
  output logic [31:0] c_cycles,
  output logic [15:0] worst_beats_per_frame,
  output logic        running_below_rate,
  output logic        budget_exceeded
);

  logic [31:0] cycles_this_frame;
  logic [15:0] beats_this_frame;

  // Expected cycles per frame = (F + GAP) * 8 * f / r.
  //   With f in kHz and r in Mbps: (F+20) * 8 * f_kHz / (r_Mbps*1000)
  wire [47:0] num = 48'(frame_bytes + 16'(GAP_B)) * 48'd8 *
                    48'(cfg_clk_hz_div_1k);
  wire [47:0] den = 48'(cfg_line_rate_mbps) * 48'd1000;
  wire [47:0] exp10 = (den == '0) ? 48'd0 : ((num * 48'd10) / den);

  assign expected_cycles_x10 = exp10[15:0];

  always_comb begin
    measured_cycles_x10 = (c_frames == '0) ? 16'd0
                          : 16'((c_cycles * 32'd10) / c_frames);
  end

  // Running below rate: the measured cycles per frame exceed the
  // expected by more than 10%. The MAC is idling between frames,
  // which means something upstream is not delivering -- the PHY, the
  // partner, or a link that negotiated a lower speed than configured.
  assign running_below_rate = (c_frames > 32'd1000) &&
                              (measured_cycles_x10 >
                               (expected_cycles_x10 + (expected_cycles_x10 / 16'd10)));

  // The budget is exceeded if a frame took FEWER cycles than its
  // beats require, which is impossible and means the beat accounting
  // is wrong -- usually the dual-frame case being miscounted.
  assign budget_exceeded = (worst_beats_per_frame != 16'd0) &&
                           (measured_cycles_x10 <
                            (worst_beats_per_frame * 16'd10));

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      cycles_this_frame <= '0; beats_this_frame <= '0;
      c_frames <= '0; c_cycles <= '0; worst_beats_per_frame <= '0;
    end else begin
      cycles_this_frame <= cycles_this_frame + 1;
      if (beat_valid) beats_this_frame <= beats_this_frame + 16'd1;

      if (frame_eof) begin
        c_frames <= c_frames + 1;
        c_cycles <= c_cycles + cycles_this_frame;
        if (beats_this_frame > worst_beats_per_frame)
          worst_beats_per_frame <= beats_this_frame;
        cycles_this_frame <= '0;
        beats_this_frame  <= '0;
      end
    end
  end

endmodule

Classification: a run-time comparison of measured cycles per frame against a configuration-derived expectation.

What it teaches: that the gap between a design-time budget and a measured one is a diagnosis nobody has. Section 4's table says a 100 Gb/s port has 1.31 cycles per frame at minimum size. A port measuring 4.2 is not running at 100 Gb/s — the link negotiated slower, the partner is sending larger frames, or the traffic is not saturating — and every one of those is worth knowing and none produces an error.

And it teaches that budget_exceeded catches a specific accounting bug. A frame cannot take fewer cycles than its beats require. If the measurement says it did, the beat count is wrong — and the usual cause is Section 3's dual-frame case: a beat carrying two frames counted once, so the second frame appears to have arrived in zero cycles. The check is a comparison and it catches a class of bug that otherwise shows up as statistics that do not add up.

Deliberately simplified: two 48-bit divides evaluated combinationally, which a real design computes once when the line rate is configured — there are only a handful of rates. cycles_this_frame counts from the previous eof and therefore includes the gap, which is what Section 4's budget means and is worth stating because the alternative convention differs by 20 octets. And the frame's own length is used for the expectation, so a port with mixed frame sizes gets an expectation that varies per frame and a measurement that is an average.

Production implication: running_below_rate is the first counter in this track that detects the link not being what the configuration says, from inside the MAC. Chapter 11.4's speed mismatch, a partner that is not saturating, an application that is not offering load — all three produce more cycles per frame than the budget implies, and an integrator chasing a throughput problem can rule out the MAC in one register read. It does not distinguish the three causes; it distinguishes them all from "the MAC is slow", which is where such investigations start.

9. Which Blocks Are Timing-Critical

Section 6 sorted blocks by billing unit. This section sorts them by how hard they are to close timing on, and the two orderings are almost unrelated.

A block's timing difficulty is set by the logic depth between registers, not by how often it runs.

Row one is the MAC's critical path at every width above about 64 bits, and Chapter 6.4 explains why.

Column two is the matrix's size and column three is the network's. A cell is a term only if it is a one, and over GF(2) the generated matrix is about half ones — 8 512 terms at 512 bits, 51.95% of the dense bound — which Chapter 19.4 §2 generates and counts.

The depth grows logarithmically and the area grows linearly — so doubling the datapath doubles the CRC's gates and adds one level of XOR, which is the trade that makes wide datapaths work at all. At 195.3125 MHz a nine-level XOR tree has 5.12 ns — 569 ps per level — which is comfortable; at 390.625 MHz an eight-level tree has 2.56 ns and 320 ps per level, which is not, and is why 100 Gb/s uses 512 bits at 195 MHz rather than 256 at 390.

Row five deserves the note it was promised, because the statistics block is easy per instance and hard in aggregate.

An RMON counter set is about twenty counters, and every frame updates several of them. At 100 Gb/s:

Fifteen counter updates per cycle is fine — they are different counters and update in parallel. What is not fine is two frames updating the same counter in the same cycle, which Section 3's dual-frame beat makes routine: a beat containing the end of one frame and the start of another can complete two frames' worth of statistics at once.

So every counter needs an adder, not an increment — and the adder's second operand is "how many frames completed this cycle", which is zero, one or two.

That is a small change and it is invisible until a design is run at rate with small frames, which is Section 3's c_dual_frame_beats argument again.

And the section's general conclusion is worth stating because it inverts the attention a design usually gets:

The parser is wide and looks frightening and is a mux. The statistics block is twenty counters and looks trivial and needs an adder per counter and a dual-frame case. The CRC engine is the deepest path in the MAC and is the one block whose structure Chapter 6.4 already settled, which is why it rarely causes trouble: it is hard and it is solved.

10. RTL 5 — The Backpressure Chain

The block that expresses Section 11's asymmetry structurally, so that a receive stage cannot accidentally acquire a stall.

// -----------------------------------------------------------------------
// block_backpressure_chain -- propagates readiness along the
// transmit path and REFUSES to along the receive path.
//
// The refusal is the point. A receive stage with a ready signal is a
// receive stage that can stall, and a stalled receive stage drops
// frames -- 7.2. The chain makes the direction explicit so a review
// can see it.
// -----------------------------------------------------------------------
module block_backpressure_chain
  import macarch_pkg::*;
#(
  parameter int STAGES    = 8,
  parameter bit IS_RECEIVE = 1'b1
)(
  input  logic              clk,
  input  logic              rst_n,

  input  logic [STAGES-1:0] stage_can_accept,
  input  logic              sink_ready,

  output logic [STAGES-1:0] stage_ready,
  output logic              source_ready,

  output logic [31:0]       c_backpressure_cycles [STAGES],
  output logic [31:0]       c_source_stalls,
  output logic              receive_path_stalled     // must never assert
);

  // On the RECEIVE path, every stage is unconditionally ready. The
  // generate is the enforcement: there is no expression by which a
  // receive stage's ready could become conditional.
  generate
    if (IS_RECEIVE) begin : g_rx
      assign stage_ready  = {STAGES{1'b1}};
      assign source_ready = 1'b1;
    end else begin : g_tx
      // On the TRANSMIT path, readiness propagates backwards from
      // the sink. Stage i is ready if stage i+1 is ready or stage i
      // is empty.
      assign stage_ready[STAGES-1] = sink_ready;
      for (genvar i = STAGES-2; i >= 0; i--)
        assign stage_ready[i] = stage_ready[i+1] || stage_can_accept[i+1];
      assign source_ready = stage_ready[0];
    end
  endgenerate

  // The invariant. If a receive path ever reports a stage unable to
  // accept, the design has acquired backpressure it cannot have --
  // and the frames it would have dropped are already gone.
  assign receive_path_stalled = IS_RECEIVE && !(&stage_can_accept);

  always_ff @(posedge clk or negedge rst_n) begin
    int i;
    if (!rst_n) begin
      for (i = 0; i < STAGES; i++) c_backpressure_cycles[i] <= '0;
      c_source_stalls <= '0;
    end else begin
      for (i = 0; i < STAGES; i++)
        if (!stage_ready[i]) c_backpressure_cycles[i] <=
                             c_backpressure_cycles[i] + 1;
      if (!source_ready) c_source_stalls <= c_source_stalls + 1;
    end
  end

endmodule

Classification: a generate-guarded readiness chain that makes one direction structurally incapable of stalling.

What it teaches: that the receive path's lack of backpressure is best enforced by construction rather than by review. A receive stage with a ready output is a stage somebody will eventually make conditional — the change looks safe, it passes every test where the memory system is fast, and it drops frames when the memory system is not. The generate makes the conditional form unwritable on the receive side.

And it teaches that c_backpressure_cycles per stage is the transmit path's profile. Chapter 18.4 §16 established the transmit path as a pipeline whose depth is set by memory latency; this counter says which stage the depth is insufficient at, which is almost always the FIFO boundary and is occasionally not.

Deliberately simplified: the transmit chain's readiness is combinational across all eight stages, so backpressure ripples end to end in one cycle — at 195 MHz across eight stages that is a real path and a production design breaks it with skid buffers. receive_path_stalled uses a reduction AND over the whole vector, which is correct and would be pipelined. And stage_can_accept is an input on the receive side that the generate ignores, which is deliberate: the signal exists so the invariant can be checked.

Production implication: c_source_stalls on the transmit chain, divided by the total cycles, is the fraction of time the transmit path could not accept a frame — and it is the number to compare against Chapter 18.4 §14's c_starve_cycles. The two measure the same congestion from opposite ends: starvation means the gather engine had nothing; source stalls mean it had something and the pipeline would not take it. A port showing both is a port whose problem moves, and a port showing neither while running below rate has a problem in neither place.

11. Backpressure, and the One Direction That Has None

Section 10 enforced an asymmetry. This section is where it comes from and what it costs, because it is the single structural fact that makes the two halves of a MAC different designs.

The transmit path controls its own rate. A frame goes out when the MAC decides; if the memory system is slow, the wire idles and throughput falls — Chapter 18.4 §16's conclusion that falling behind costs throughput and never data.

The receive path does not. A frame arrives when the partner sends it. Chapter 7.2 established there is no way to say "wait" — Chapter 14.2's PAUSE operates between frames and takes 13.32 µs to take effect at 1 Gb/s, which is nearly twenty minimum-size frame times.

Row four is Chapter 18.4 §18's finding and it follows directly from row one. A transmit FIFO holds one maximum frame — 9 KiB. A receive FIFO holds whatever arrives during the worst memory stall — 24.41 KiB at 100 Gb/s for 2 µs — because there is nothing else to do with those octets.

And the asymmetry propagates into every design decision in the MAC.

Row five is the one a verification plan gets wrong. A transmit path is verified by measuring what it achieves; a receive path is verified by proving something never happens — and the second needs an assertion on every stage rather than a performance measurement at the end.

There is one partial exception worth naming, because it is the only lever the receive path has.

Chapter 14.2's flow control is backpressure toward the partner, not toward the wire — it stops the next frame, not this one. So it converts a receive-path overflow into a throughput loss, which is exactly the transmit path's trade, at the cost of Chapter 18.1 §18's headroom and Chapter 14.2's 88% collateral.

Which is the honest summary of what flow control buys: it gives the receive path the transmit path's property, at a price, and with a dead time long enough that the FIFO must cover it anyway.

12. RTL 6 — The Reset Sequencer

Four domains, four resets, and an order that matters — Chapter 18.1 §16 established the rule and this is the mechanism.

// -----------------------------------------------------------------------
// macarch_reset_sequencer -- asynchronous assertion, synchronous
// release, in a defined order.
//
// 18.1 section 16: assert asynchronously because the receive clock
// may not be running; release synchronously or the domain leaves
// reset in a mixture. The ORDER is this block's addition: host,
// then transmit, then receive, with enable held clear throughout.
// -----------------------------------------------------------------------
module macarch_reset_sequencer
  import macarch_pkg::*;
#(
  parameter int HOLD = 8
)(
  input  logic  rst_async_n,          // the SoC's reset, asynchronous

  input  logic  clk_host,
  input  logic  clk_tx,
  input  logic  clk_rx,               // may be ABSENT
  input  logic  clk_ptp,

  input  logic  rx_clk_present,       // 18.1 section 7
  input  logic  clk_lost_event,       // the cable was pulled

  output logic  rst_host_n,
  output logic  rst_tx_n,
  output logic  rst_rx_n,
  output logic  rst_ptp_n,

  output logic  mac_enable_permitted,
  output logic [31:0] c_rx_resets,
  output logic  released_out_of_order    // must never assert
);

  // Each domain: a two-flop synchroniser on the release. Assertion
  // is asynchronous through the flops' own reset, so a domain whose
  // clock is stopped still enters reset.
  logic [1:0] host_sync, tx_sync, ptp_sync;
  logic [1:0] rx_sync;

  always_ff @(posedge clk_host or negedge rst_async_n)
    if (!rst_async_n) host_sync <= 2'b00;
    else              host_sync <= {host_sync[0], 1'b1};
  assign rst_host_n = host_sync[1];

  // Transmit is released only after host. The dependency is explicit
  // rather than implied by timing.
  always_ff @(posedge clk_tx or negedge rst_async_n)
    if (!rst_async_n) tx_sync <= 2'b00;
    else              tx_sync <= {tx_sync[0], rst_host_n};
  assign rst_tx_n = tx_sync[1];

  // Receive is released after transmit AND only while its clock is
  // present. A domain whose clock has stopped is held in reset, so
  // that when the clock returns the domain starts clean -- 18.1
  // section 16's partial-frame case.
  always_ff @(posedge clk_rx or negedge rst_async_n)
    if (!rst_async_n) rx_sync <= 2'b00;
    else              rx_sync <= {rx_sync[0], (rst_tx_n && rx_clk_present)};
  assign rst_rx_n = rx_sync[1];

  always_ff @(posedge clk_ptp or negedge rst_async_n)
    if (!rst_async_n) ptp_sync <= 2'b00;
    else              ptp_sync <= {ptp_sync[0], rst_host_n};
  assign rst_ptp_n = ptp_sync[1];

  // The real release: the MAC does nothing until every domain is out.
  assign mac_enable_permitted = rst_host_n && rst_tx_n &&
                                rst_rx_n && rst_ptp_n;

  // Out of order: a downstream domain out of reset while an upstream
  // one is not.
  assign released_out_of_order = (rst_rx_n && !rst_tx_n) ||
                                 (rst_tx_n && !rst_host_n);

  always_ff @(posedge clk_host or negedge rst_async_n)
    if (!rst_async_n) c_rx_resets <= '0;
    else if (clk_lost_event) c_rx_resets <= c_rx_resets + 1;

endmodule

Classification: per-domain release synchronisers chained into an explicit order, with a clock-presence gate.

What it teaches: that the receive domain's reset is gated on its clock being present, and that this is not defensive but necessary. Chapter 18.1 §16 established the case: the receive clock stops when the cable is pulled, every flop in the domain freezes at whatever value it held — including a half-received frame's state — and when the clock returns the domain resumes from there. The partial frame is then reassembled onto the front of the next one. Holding the domain in reset while the clock is absent makes the return clean.

And it teaches that mac_enable_permitted is the real release and the resets are only its preconditions. Chapter 18.1 §3's register file resets ctrl_q to zero and nothing in the MAC does anything until software writes the enable — so the reset order matters for the domains' internal consistency, and the MAC's externally visible behaviour is gated by one bit that software controls.

Deliberately simplified: the clk_rx always-block is clocked by a signal that may not be toggling, which is exactly the point and will make a linter complain. rx_clk_present is assumed to be stable and glitch-free, which it is not — it comes from a detector in another domain and needs its own synchroniser. And released_out_of_order is combinational across three asynchronous domains, which is Section 19's rejected class in miniature and is why it is a debug signal rather than an assertion's operand.

Production implication: c_rx_resets counts cable events, and it is the counter an operator reads when a port is intermittently unavailable. A value that climbs steadily is a marginal connector or a partner that is resetting; a value that jumps in bursts is somebody re-patching. Neither is a MAC fault, and both are invisible from the link-status register alone, which reports the current state and not how often it has changed.

13. Bringing Four Domains Up in Order

Section 12 built the sequencer. This section is the argument for the order it enforces, because three of the four dependencies are not obvious.

The order is host, then transmit, then receive, with PTP alongside host.

Row four is the binding constraint and it is a fact about the world rather than a design choice. The receive clock is recovered from the incoming signal — Chapter 18.1 §6 — so at power-on, before the partner is transmitting, it does not exist. A sequencer that waits for the receive domain to come out of reset waits for a partner that may never arrive.

Which forces the enable bit to be the real release, and it is worth stating why the alternative fails.

Row three is the only one that lets a MAC transmit before it has ever received, which is necessary: Chapter 11.1's negotiation requires transmitting to a partner that is not yet transmitting back.

And the release order has a second consequence in the other direction, which Section 12's clock gate handles and which is worth seeing explicitly.

Consider the cable being pulled mid-frame.

Row six is the failure and its symptom is a single corrupted frame after every cable event — which is one CRC error per unplug, attributed to the connector, and is actually the MAC.

One more ordering question that Module 18 raised and this chapter can now answer.

Chapter 18.1 §16 noted that releasing the receive domain before the host domain fills Chapter 18.1 §9's FIFO against a read pointer that cannot move, losing frames. With Section 12's order that cannot happen: the receive domain is released last and only after transmit, which is after host. So the FIFO's reader is always alive before its writer, and the failure Module 18 identified is removed by construction rather than by timing.

14. RTL 7 — Architecture Telemetry

The counters that make Sections 4, 6 and 9's arithmetic checkable in silicon.

// -----------------------------------------------------------------------
// macarch_telemetry -- the MAC's internal observability.
// -----------------------------------------------------------------------
module macarch_telemetry
  import macarch_pkg::*;
#(
  parameter int STAGES = 8
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic        beat_valid,
  input  logic        frame_sof,
  input  logic        frame_eof,
  input  logic        dual_frame_beat,
  input  logic [7:0]  frames_in_pipeline,
  input  logic [7:0]  result_frame_id,
  input  logic        result_valid,
  input  logic        result_unmatched,
  input  logic [STAGES-1:0] stage_stalled,

  output logic [31:0] c_beats,
  output logic [31:0] c_frames,
  output logic [31:0] c_dual_frame_beats,
  output logic [31:0] c_results,
  output logic [31:0] c_results_unmatched,
  output logic [7:0]  peak_frames_in_pipeline,
  output logic [31:0] c_stage_stall [STAGES],
  output logic [15:0] beats_per_frame_x10,
  output logic [15:0] dual_frame_pct
);

  always_comb begin
    beats_per_frame_x10 = (c_frames == '0) ? 16'd0
                          : 16'((c_beats * 32'd10) / c_frames);
    dual_frame_pct = (c_beats == '0) ? 16'd0
                     : 16'((c_dual_frame_beats * 32'd100) / c_beats);
  end

  always_ff @(posedge clk or negedge rst_n) begin
    int i;
    if (!rst_n) begin
      c_beats <= '0; c_frames <= '0; c_dual_frame_beats <= '0;
      c_results <= '0; c_results_unmatched <= '0;
      peak_frames_in_pipeline <= '0;
      for (i = 0; i < STAGES; i++) c_stage_stall[i] <= '0;
    end else begin
      if (beat_valid) begin
        c_beats <= c_beats + 1;
        if (dual_frame_beat) c_dual_frame_beats <= c_dual_frame_beats + 1;
      end
      if (frame_eof) c_frames <= c_frames + 1;

      if (result_valid) begin
        c_results <= c_results + 1;
        // A result whose frame_id matches nothing in flight is the
        // section 6 failure: the tag was reused, or the pipeline is
        // deeper than the tag space.
        if (result_unmatched)
          c_results_unmatched <= c_results_unmatched + 1;
      end

      if (frames_in_pipeline > peak_frames_in_pipeline)
        peak_frames_in_pipeline <= frames_in_pipeline;

      for (i = 0; i < STAGES; i++)
        if (stage_stalled[i]) c_stage_stall[i] <= c_stage_stall[i] + 1;
    end
  end

endmodule

Classification: a pipeline-occupancy and tag-matching accountant.

What it teaches: that c_results_unmatched is the counter that catches Section 6's structure failing, and nothing else will. A per-frame result tagged with a frame_id that matches no frame in flight means the tag space is smaller than the pipeline's occupancy — so tags wrapped and a result was attributed to the wrong frame. The symptom without this counter is statistics that are slightly wrong and address-filter decisions applied to the wrong frame, both of which are functional bugs that look like noise.

And it teaches that peak_frames_in_pipeline is the number that sizes the tag. Section 6's arithmetic says twelve frames at 100 Gb/s with a 16-stage pipeline; an 8-bit tag is ample and a 3-bit one is not, and the measurement is what turns that from a calculation into a fact. A design whose peak reaches the tag space's size has already produced unmatched results.

Deliberately simplified: two combinational divides; result_unmatched arrives as an input from a matching structure this block does not contain; and the stall counters are per stage where a real design may aggregate them. frames_in_pipeline is also an input rather than derived, which hides the structure that tracks it — an occupancy counter incremented on sof and decremented on the result.

Production implication: dual_frame_pct is the measurement that says whether a design has been exercised at rate, and it is a one-line addition to a regression report. At 100 Gb/s with minimum-size frames it should be near 100% — the frame slot is 1.31 beats, so essentially every beat straddles a boundary. A regression reporting 0% has run with large frames or below line rate, and the entire dual-frame path — every block's two-frame case, Section 9's statistics adders, the parser's overlapping headers — is unverified.

15. RTL 8 — The Architecture Conformance Monitor

The last block of the chapter, and its verdicts are about the architecture rather than about the traffic.

// -----------------------------------------------------------------------
// macarch_conformance_monitor -- is the architecture's arithmetic
// holding in silicon?
// -----------------------------------------------------------------------
module macarch_conformance_monitor
  import macarch_pkg::*;
#(
  parameter int STAGES  = 8,
  parameter int TAG_W   = 8
)(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [31:0] c_beats,
  input  logic [31:0] c_frames,
  input  logic [31:0] c_dual_frame_beats,
  input  logic [31:0] c_results_unmatched,
  input  logic [7:0]  peak_frames_in_pipeline,
  input  logic [15:0] beats_per_frame_x10,
  input  logic [15:0] dual_frame_pct,
  input  logic [15:0] measured_cycles_x10,
  input  logic [15:0] expected_cycles_x10,

  input  logic        receive_path_stalled,
  input  logic        released_out_of_order,
  input  logic        multibit_on_synchroniser,
  input  logic        budget_exceeded,
  input  logic        running_below_rate,
  input  logic        stalled_after_commit,

  output logic        architecture_ok,
  output logic        tag_space_marginal,
  output logic        dual_frame_untested,
  output logic        cdc_fault,
  output logic        reset_order_fault,
  output logic        rate_mismatch,
  output logic        none_of_the_above
);

  // Fatal: the receive path acquired backpressure, or a frame was
  // stalled after it started transmitting.
  wire datapath_fault = receive_path_stalled | stalled_after_commit |
                        (c_results_unmatched != 32'd0);

  assign cdc_fault        = multibit_on_synchroniser;
  assign reset_order_fault = released_out_of_order;

  // The tag space must exceed the pipeline's occupancy with margin.
  // At half, the design is one traffic pattern from wrapping.
  assign tag_space_marginal = (peak_frames_in_pipeline >
                               (8'((1 << TAG_W)) >> 1));

  // A design that has never seen a dual-frame beat has not been run
  // at rate with small frames -- section 3.
  assign dual_frame_untested = (c_beats > 32'd1_000_000) &&
                               (c_dual_frame_beats == 32'd0);

  assign rate_mismatch = running_below_rate | budget_exceeded;

  assign architecture_ok = !datapath_fault && !cdc_fault &&
                           !reset_order_fault;

  assign none_of_the_above = architecture_ok && !tag_space_marginal &&
                             !dual_frame_untested && !rate_mismatch;

  // ---- properties -------------------------------------------------

  p_receive_never_stalls:
    assert property (@(posedge clk) disable iff (!rst_n)
      !receive_path_stalled)
    else $error("the receive path acquired backpressure it cannot have");

  p_never_stall_after_commit:
    assert property (@(posedge clk) disable iff (!rst_n)
      !stalled_after_commit)
    else $error("a transmit stage stalled after the frame was committed");

  p_no_unmatched_results:
    assert property (@(posedge clk) disable iff (!rst_n)
      c_results_unmatched == 32'd0)
    else $error("a per-frame result matched no frame in flight");

  p_tag_space_sufficient:
    assert property (@(posedge clk) disable iff (!rst_n)
      peak_frames_in_pipeline < (1 << TAG_W))
    else $error("pipeline occupancy reached the tag space");

  p_reset_order:
    assert property (@(posedge clk) disable iff (!rst_n)
      !released_out_of_order)
    else $error("a domain was released before its predecessor");

endmodule

Classification: an architecture-level verdict generator whose findings are about structure rather than traffic.

What it teaches: that dual_frame_untested is a coverage verdict expressed as a run-time signal, which is unusual and is worth the bit. Every other monitor in this track reports what went wrong. This one reports that something has never happened — and for the dual-frame path, never happening is the problem, because the path exists and is unexercised. A million beats with no dual-frame beat is evidence about the testbench, reported by the design.

And it teaches that tag_space_marginal fires well before p_tag_space_sufficient does. The property catches the occupancy reaching the tag space — by which point results have already been misattributed. The verdict fires at half, which is a design margin rather than a failure, and is the only warning available before the failure is silent and functional.

Deliberately simplified: the thresholds are literals — half the tag space, a million beats — where production takes them from registers. datapath_fault combines three signals from three different clock domains in one expression, which is Section 19's rejected class territory and is why these are debug signals rather than a synthesised alarm. And architecture_ok says nothing about the blocks the chapter did not build, which is most of the MAC.

Production implication: none_of_the_above for the seventh time across Modules 18 and 19, and this one covers a different kind of thing from the six before it. Module 18's six monitors ruled out faults in the MAC's contracts — with the system, the driver, the bus, the network. This one rules out faults in the MAC's own structure: the pipeline's tagging, the domain crossings, the reset order, the backpressure discipline. A port on which all seven assert has no known fault in any contract or any structure this track has defined.

16. Datapath Width Against Clock Frequency

Section 9 established the CRC engine as the critical path. This section is the trade that follows, because width and frequency are interchangeable and the exchange rate is not linear.

To carry r bits per second a datapath needs width × clock ≥ r. At 100 Gb/s:

And the CRC engine's timing at each:

Which settles the choice and explains why 100 Gb/s MACs are 512 bits wide. The tree's depth grows logarithmically with the width and the cycle time grows linearly — so going wider buys time faster than it costs depth, and the two curves cross decisively.

So why stop at 512? Three reasons and only the third is fundamental.

Row three is the real limit and it is the chapter's arithmetic closing on itself. At 1024 bits a 64-octet frame's whole wire slot is 0.656 cycles — less than one — so two frames complete in most cycles and every per-frame structure must handle two completions per cycle rather than at most two per beat. The dual-frame case becomes a triple-frame case, and Section 3's beat structure with one second_offset is insufficient.

Which gives a clean rule for choosing a datapath width:

Choose the narrowest width whose CRC tree closes timing, subject to the frame slot remaining above one cycle.

And the last column is the chapter's opening table, arrived at from the other direction — the widths are not chosen to give a convenient cycle budget; the cycle budget is what the width and the CRC's timing leave behind.

17. What Module 19's Remaining Chapters Inherit

This chapter's two numbers constrain every block the rest of Module 19 builds. Stating what each inherits is the floorplan's last job.

Two rows deserve expansion because their constraint is not obvious from the chapter title.

The CRC engine's dual-frame requirement is the sharpest. At 512 bits, a beat can contain the last octets of frame n and the first octets of frame n+1. So the engine must finalise one accumulator and initialise another in the same cycle — which means two accumulators, a mux, and a boundary offset rather than one accumulator and a reset. Chapter 6.4 built the matrix; this is the packaging around it, and it roughly doubles the engine's register count.

And the counters' requirement is the one most likely to be missed. Section 9: 15.24 counter updates per cycle at 100 Gb/s, which is fine because they are different counters — except when two frames complete in one cycle and both update the same counter. A counter with an increment advances by one and loses a frame; a counter with an adder advances by two. The difference is one adder per counter and it is invisible below about 25 Gb/s.

And there is one thing the remaining chapters do not inherit, which is worth stating because it is a common confusion.

Module 18's structures are not in the MAC. Chapter 18.2's rings, Chapter 18.3's DMA, Chapter 18.6's coalescer live in the host domain and attach to the MAC through Chapter 19.6's memory interface. The MAC proper — the blocks in Section 2's first three tables — ends at the FIFO.

The FIFO is the seam, and it is the only block that belongs to both — which is why Chapter 18.1 §9 built one and Chapter 19.5 will build it again, from the other side.

18. The Cost, Accounted

Eight blocks, and this chapter's honest accounting is that most of them are not blocks.

The templates dominate and they are the datapath. Eight receive stages at 600 flops is 4 800 flops, and 600 is almost entirely the 512-bit beat register: 512 bits of data plus the byte count, the flags and Section 3's two tags.

So a pipeline stage is 545 flops of beat and a handful of control — and the dual-frame fields are 23 of them, 4.2%, which is the cost of Section 3's structural requirement and is trivial.

And the whole receive datapath, eight stages deep:

Against Module 18's 25 700 for the system interface, which is worth noting: the MAC's own datapath is less than half the logic of attaching it to a system — and that ratio is the whole argument for Module 18 having been seven chapters.

And the XOR terms are the real area. This table first carried 32 768 — two dense 32 × 512 matrices — and Chapter 19.4 §19 showed that figure was wrong three times. Each matrix is 8 512 terms rather than 16 384; each direction needs two of them, because Section 3's beat can carry two frames; and the correction networks a partial final word needs were not counted at all. The first two errors nearly cancel — 34 048 against 32 768 — and the third is the whole gap: 39 445 terms, 20.4% above the original estimate. Chapter 6.4 established the structure; this is the first time its area has been placed alongside the rest of the MAC, and Chapter 19.4 is where the placing was checked.

Memory: none. The FIFOs are Chapter 19.5's and were counted in Chapter 18.1 §18 — 32 KiB receive, 9 KiB transmit at 100 Gb/s — and this chapter's blocks add zero bytes.

Which gives the MAC's complete bill, for the first time in this track:

Thirty-eight thousand flops and fifty-five kilobytes for a 100 Gb/s Ethernet port, plus the DRAM that belongs to the driver — which is a small block by any modern measure and took two modules and fourteen chapters to specify.

19. Properties Worth Asserting, and One Worth Refusing

A pipeline's properties are about beats, tags and boundaries. The rejected one is about a frame, and that is the problem.

Pipeline stages.

// A receive stage never stalls -- there is no backpressure to the wire.
p_rx_never_stalls:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    !stage_stalled)
  else $error("a receive stage stalled");

// A receive stage has exactly one cycle of latency, always.
p_rx_fixed_latency:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    in_valid |=> out_valid)
  else $error("a receive stage swallowed a beat");

// The beat passes through unaltered by the template.
p_beat_preserved:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    in_valid |=> (out_beat.data == $past(in_beat.data)))
  else $error("a pipeline stage altered the beat");

// A transmit stage may stall, but never after commit.
p_tx_no_stall_after_commit:
  assert property (@(posedge clk_tx) disable iff (!rst_tx_n)
    !stalled_after_commit)
  else $error("a transmit stage stalled after the frame was committed");

// Transmit readiness propagates: a ready sink makes the source ready.
p_tx_ready_propagates:
  assert property (@(posedge clk_tx) disable iff (!rst_tx_n)
    sink_ready |-> source_ready)
  else $error("a ready sink did not make the source ready");

Beats and frame boundaries.

// A beat's byte count never exceeds the datapath.
p_bytes_within_datapath:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    in_valid |-> (in_beat.bytes <= DP_BYTES))
  else $error("a beat claimed more bytes than the datapath carries");

// A dual-frame beat's second offset lies inside the beat.
p_second_offset_valid:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    (in_valid && in_beat.has_second) |->
      (in_beat.second_offset < DP_BYTES))
  else $error("a second-frame offset fell outside the beat");

// A dual-frame beat carries both frames' tags and they differ.
p_second_id_differs:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    (in_valid && in_beat.has_second) |->
      (in_beat.second_id != in_beat.frame_id))
  else $error("a dual-frame beat carried the same tag twice");

// A dual-frame beat is also an end of frame.
p_second_implies_eof:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    (in_valid && in_beat.has_second) |-> in_beat.eof)
  else $error("a beat carried a second frame without ending the first");

// Below a 160-bit datapath, dual-frame beats are impossible.
p_narrow_has_no_dual:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    (DP_BYTES <= GAP_B) |-> !in_beat.has_second)
  else $error("a dual-frame beat on a datapath narrower than the gap");

// Start and end alternate on a given tag.
p_sof_eof_alternate_per_tag:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    (in_valid && in_beat.sof) |=>
      (!(in_valid && in_beat.sof && (in_beat.frame_id == $past(in_beat.frame_id)))
       throughout (in_valid && in_beat.eof)[->1]))
  else $error("two starts on one tag without an intervening end");

Tags and per-frame results.

// Every result carries a tag that is currently in flight.
p_result_tag_in_flight:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    result_valid |-> tag_in_flight[result.frame_id])
  else $error("a result named a frame that is not in the pipeline");

// The pipeline never holds more frames than the tag space.
p_occupancy_within_tags:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    frames_in_pipeline < (1 << 8))
  else $error("pipeline occupancy reached the tag space");

// A tag is not reused while its frame is in flight.
p_no_tag_reuse:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    (in_valid && in_beat.sof) |-> !tag_in_flight[in_beat.frame_id])
  else $error("a tag was reused while its frame was still in the pipeline");

// Every frame that enters produces exactly one result.
p_one_result_per_frame:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    (in_valid && in_beat.sof) |-> ##[1:PIPE_MAX]
      (result_valid && (result.frame_id == $past(in_beat.frame_id))))
  else $error("a frame produced zero or several results");

// Unmatched results never occur.
p_no_unmatched:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    c_results_unmatched == 32'd0)
  else $error("a result matched no frame");

Domains and reset.

// No multi-bit value on a two-flop synchroniser.
p_no_multibit_sync:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    !multibit_on_synchroniser)
  else $error("a multi-bit value crosses on a synchroniser");

// No declared crossing has the same domain on both sides.
p_crossings_are_crossings:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    !same_domain_declared)
  else $error("a declared crossing has one domain on both sides");

// Domains are released in order.
p_reset_order_held:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    !released_out_of_order)
  else $error("a domain was released before its predecessor");

// The receive domain is held in reset while its clock is absent.
p_rx_held_without_clock:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    !rx_clk_present |-> ##[0:4] !rst_rx_n)
  else $error("the receive domain ran without its clock present");

// The enable is only permitted once every domain is out of reset.
p_enable_requires_all:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    mac_enable_permitted |-> (rst_host_n && rst_tx_n && rst_rx_n && rst_ptp_n))
  else $error("the MAC was enabled with a domain still in reset");

Budget and telemetry.

// A frame never takes fewer cycles than its beats require.
p_cycles_at_least_beats:
  assert property (@(posedge clk) disable iff (!rst_n)
    !budget_exceeded)
  else $error("a frame completed in fewer cycles than its beats");

// Counters are monotonic.
p_counters_monotonic:
  assert property (@(posedge clk) disable iff (!rst_n)
    ##1 ((c_beats >= $past(c_beats)) && (c_frames >= $past(c_frames))))
  else $error("an architecture counter decreased");

// Dual-frame beats never exceed total beats.
p_dual_le_total:
  assert property (@(posedge clk) disable iff (!rst_n)
    c_dual_frame_beats <= c_beats)
  else $error("more dual-frame beats than beats");

// The peak occupancy only rises.
p_peak_monotonic:
  assert property (@(posedge clk) disable iff (!rst_n)
    ##1 (peak_frames_in_pipeline >= $past(peak_frames_in_pipeline)))
  else $error("the peak pipeline occupancy decreased");

20. Verification Scenarios

Fifty-eight scenarios, plus a six-run directed test that requires frames spaced at the minimum interframe gap and held there.

Pipeline stages — 10 scenarios.

Frame boundaries and dual-frame beats — 11 scenarios.

Tags and results — 10 scenarios.

Domains and reset — 9 scenarios.

Cycle budget — 9 scenarios.

Verdicts and counters — 9 scenarios.

The directed test — six runs random stimulus will not produce.

Every failure this chapter is about requires frames to be close together, and constrained-random frame generation does not naturally produce that. A generator that randomises the interframe gap produces minimum-gap frames occasionally; the dual-frame path, the tag pressure and the twelve-frames-in-flight case all require them sustained for thousands of frames.

And the tag-wrap failure needs more than that: the pipeline must be full when the wrap occurs, which is a specific state reached only after sustained back-to-back traffic.

Construct it. Six runs, one variable: the spacing and the tag width.

Run A is the default regression and it finds nothing. Random gaps put the pipeline at low occupancy most of the time; twelve frames in flight requires minimum gaps held, and a random generator produces a sustained minimum-gap burst with vanishing probability.

Run D is the failure and it needs all three conditions at once: minimum-size frames, minimum gaps, and a tag narrower than the occupancy. Remove any one and it passes — which runs E and F demonstrate.

Run E is the frame-size control. 1518-octet frames give 24.03 cycles each, so a 16-stage pipeline holds 0.66 frames and a 3-bit tag is ample. The same design, the same tag, and no failure.

Run F is the datapath-width control. At 128 bits the beat is 16 octets, below the 20-octet gap, so dual-frame beats cannot occur — and the frame slot is 5.25 cycles, so a 16-stage pipeline holds 3.05 frames and eight tags suffice. The same tag width, and no failure.

The oracle, in four parts:

Row four is the one that shows the consequence rather than the mechanism, and it is the check most likely to be omitted. A misattributed result does not produce an error — it produces a frame counted in the wrong bucket, an address-filter decision applied to the wrong frame, and a length recorded against a neighbour. Only a scoreboard comparing per-frame results against expectations finds it, and c_results_unmatched is the cheap proxy that points at the cause.

21. Debugging a MAC's Internals

Three complaints, and all three produce symptoms that look like something else entirely.

Complaint 1 — "statistics do not match a packet capture."

Row three is the tell and it is why this bug survives bring-up. At 1 Gb/s the pipeline holds 0.19 frames — tags cannot wrap. The same design at 100 Gb/s holds 12.19, and a tag narrower than that misattributes results. A design validated at the lower rate and shipped at the higher one has this, and its symptom is statistics that are slightly wrong.

Complaint 2 — "one CRC error after every cable unplug."

Row three is the signature and row four is the conclusion that is hard to reach. A single CRC error per cable event is the receive domain resuming mid-frame — Chapter 18.1 §16's frozen flops — and it is attributed to the connector by everyone who has ever debugged a cable.

Complaint 3 — "the port runs below rate and every block reports healthy."

Row four is the useful outcome and it is what Section 15's none_of_the_above delivers: a port whose architecture monitor is clean and whose cycle budget says the frames are arriving slowly has a link or a load problem, not a MAC problem — which is where a throughput investigation should have started and rarely does.

And the two symptoms this chapter is systematically blamed for:

22. Misconceptions

Misconception 1 — "a faster interface just means a wider datapath."

The wrong model: scale the width with the rate; everything else is unchanged.

What it costs: a design whose per-frame blocks worked at 1 Gb/s and produce results for the wrong frames at 100. The width scaled; the cycles per frame did not — they fell from 84.000 to 1.312.

The corrected model: the per-beat work scales with width and the per-frame work does not. At 0.312 cycles of slack, every per-frame block must be pipelined across frames with a tag, and a design carried forward from a lower rate is missing the tag entirely. Sections 4, 6.

Misconception 2 — "one beat, one frame."

The wrong model: a beat belongs to a frame; frame boundaries fall on beat boundaries.

What it costs: the leading octets of every frame that shares a beat with its predecessor — which at 100 Gb/s with minimum-size frames is essentially every frame.

The corrected model: the gap between frames is 20 octets and a 512-bit beat is 64 — so above a 128-bit datapath a beat routinely carries the end of one frame and the start of the next. Every per-beat block needs two frames' state, and the beat structure needs a second offset and a second tag. Section 3.

Misconception 3 — "the receive path can push back if it has to."

The wrong model: add a ready signal to the receive stages so they can stall when the FIFO is full.

What it costs: dropped frames, because a stalled receive stage has nowhere to put the octets still arriving. And the change looks safe: it passes every test where the memory system is fast.

The corrected model: Chapter 7.2's path has no backpressure toward the wire, and Chapter 14.2's PAUSE takes 13.32 µs to take effect — nearly twenty minimum-size frame times. Every receive stage is fixed-latency and unconditional, enforced by construction rather than by review. Sections 10, 11.

Misconception 4 — "the CRC engine is the hard block, so it gets the attention."

The wrong model: the CRC is the deepest path, so that is where the design effort goes.

What it costs: attention spent on a block Chapter 6.4 already solved, while the statistics counters — twenty counters that look trivial — need an adder each because two frames can complete in one cycle.

The corrected model: the CRC engine is the critical path and it is also solved: a 512-bit matrix is ten XOR levels in a 5.12 ns cycle, which is comfortable. The blocks that cause trouble are the ones that look easy and acquire a dual-frame case, which is the statistics block and the parser. Section 9.

Misconception 5 — "wider is always better."

The wrong model: the CRC tree's depth grows logarithmically and the cycle time linearly, so go as wide as possible.

What it costs: a 1024-bit datapath at 100 Gb/s, where a 64-octet frame's whole wire slot is 0.656 cycles — below one — so two frames complete in most cycles and Section 3's single-second_offset beat structure is insufficient.

The corrected model: choose the narrowest width whose CRC tree closes timing, subject to the frame slot staying above one cycle. At 100 Gb/s that is 512 bits at 195.3125 MHz — 1.312 cycles per frame, which is the tightest budget that still has a frame per cycle. Section 16.

Misconception 6 — "the property says the current frame's CRC is checked before forwarding."

The wrong model: assert that a forwarded frame has a valid CRC result; it passes; the requirement is covered.

What it costs: a property that passes on a design forwarding frame 7 on frame 4's CRC, because at steady state both the antecedent and the consequent are true every cycle and they are about different frames.

The corrected model: at 1.31 cycles per frame a 16-stage pipeline holds twelve frames, so "the current frame" names twelve things. Every reference must become a tag — in the design and in the property — and the count is the line rate divided by the per-frame cycle budget, not a design choice. Section 19.

23. Interview Questions

Q1 — "How many clock cycles does a 100 Gb/s MAC have per minimum-size frame?"

1.312, and the derivation matters more than the number. A 64-octet frame occupies 84 octets of wire time including Chapter 5.2's preamble and Chapter 8.3's gap; at 100 Gb/s that is 6.72 ns; at CGMII's 195.3125 MHz that is 1.312 cycles, of which one is the data beat. So a per-frame block has 0.312 cycles — which is not a budget any logic meets, and the answer is to pipeline across frames with a tag rather than to make the block faster.

Q2 — "Can a single beat contain two frames?"

Above a 128-bit datapath, yes, and routinely. The gap between frames is 20 octets; a 512-bit beat is 64. So whenever a frame ends more than 20 octets before a beat boundary, the next frame's first octets are in the same beat. At 128 bits a beat is 16 octets and cannot span a gap; at 256 bits it is 32 and can. Every per-beat block above that width needs two frames' state — a CRC engine finalising one accumulator and initialising another in the same cycle, which roughly doubles its register count.

Q3 — "Why can a transmit stage stall and a receive stage not?"

Because the design controls when frames go out and does not control when they arrive. A transmit stall costs throughput — Chapter 18.4 §16 — a receive stall costs frames, because Chapter 7.2's path has no backpressure toward the wire and Chapter 14.2's PAUSE takes 13.32 µs. The discipline should be enforced structurally: a receive stage with no ready output is a stage nobody can later make conditional.

Q4 — "Which block is the MAC's critical path, and why is it not usually the problem?"

The CRC engine, and it is not the problem because Chapter 6.4 solved it. A 512-bit matrix is 32 × 512 GF(2) terms, about ten XOR levels, in a 5.12 ns cycle — 0.51 ns per level, which is comfortable. The blocks that cause trouble are the ones that look trivial: twenty RMON counters at 148.81 Mfps is 2 976 million updates per second, 15.24 per cycle, and two frames completing in one cycle means each counter needs an adder rather than an increment.

Q5 — "Why is a 100 Gb/s MAC 512 bits wide rather than 256 or 1024?"

256 bits needs 390.625 MHz, giving a nine-level CRC tree 2.56 ns — 0.28 ns per level, which is very hard. 1024 bits gives an eleven-level tree 10.24 ns, which is easy — and puts the frame slot at 0.656 cycles, below one, so two frames complete in most cycles and the beat structure needs three frames' worth of offsets. 512 bits at 195.3125 MHz is the narrowest width whose CRC closes comfortably while the frame slot stays above one cycle.

Q6 — "A property says a frame is not forwarded until its CRC is checked. It passes. Are you satisfied?"

Not at 100 Gb/s. At 1.31 cycles per frame a 16-stage pipeline holds twelve frames, so "the current frame" names twelve things — and the property relates the frame being forwarded to whatever CRC result is valid this cycle, which is a different frame. At steady state both sides are true every cycle, so it passes while the design forwards frame 7 on frame 4's result. The fix is the same one the datapath needed: a tag, and a result array indexed by it.

24. Understanding Check

25. What's Next