Skip to content
VLSI Mentor

Ethernet · Module 18

The Ethernet MAC as an SoC IP Block

A MAC integrated into an SoC demands 1.143 times its line rate in memory bandwidth, crosses four clock domains, and at 100 Gb/s asks for 1.79 bus transactions per cycle.

Modules 1 to 17 treated the MAC as a thing with a wire on one side and frames on the other. Module 18 is about the other side.

A MAC in a switch is a port. A MAC in an SoC is a bus master that happens to have a cable attached — and almost every hard problem in the next seven chapters comes from that sentence rather than from Ethernet.

Here is the number the whole module turns on. A 1 Gb/s MAC receiving 64-octet frames does not ask the system for 1 Gb/s of memory bandwidth. It asks for 1.143 Gb/s, because every frame drags a descriptor fetch and a status writeback along with it, and those are 32 octets against a 64-octet frame.

Line rateFrame dataDescriptor trafficTotal, one directionDescriptor share
1 Gb/s0.762 Gb/s0.381 Gb/s1.143 Gb/s33.3%
10 Gb/s7.619 Gb/s3.810 Gb/s11.429 Gb/s33.3%
25 Gb/s19.048 Gb/s9.524 Gb/s28.571 Gb/s33.3%
100 Gb/s76.190 Gb/s38.095 Gb/s114.286 Gb/s33.3%

A third of the memory traffic is bookkeeping, and it is a third at every line rate because the ratio is set by the frame size rather than by the rate. At 1518-octet frames the same column reads 2.1%.

This chapter frames the problem and makes the numbers do the arguing, in the same way Chapter 16.1 did for Module 16. It builds the MAC's four interfaces, establishes the clock domains and what crosses them, and derives the memory bandwidth and transaction rate a system must supply. The mechanisms that manage that traffic — rings, both DMA directions, AXI shaping, interrupts, offload — are Chapter 18.2 onward.

What it does not do is design the MAC's internals. The parser, the assembler, the CRC integration and the FIFOs are Module 19, and this chapter deliberately stops at their boundaries.


1. Scope, and the Sentence the Module Turns On

A MAC integrated into an SoC is a peripheral with four interfaces, and only one of them carries Ethernet.

InterfaceDirectionCarriesBuilt in
CPU registersslaveconfiguration, status, ring base addressesSection 3
Memorymasterframe data, descriptors, statusSection 5
PHY / xMIIbidirectionalthe frames themselvesSection 7
Interruptoutputcompletion signallingSection 11

Row two is the one that makes the MAC different from almost every other peripheral on the chip. A UART, an SPI controller, a GPIO block and a timer are all slaves: the CPU reads and writes them and they never initiate a transaction. A MAC initiates, because a frame arrives without being asked for and must be in memory before software knows it exists.

And a master's requirements are different in kind. A slave needs an address decode and a handful of registers. A master needs a bandwidth allocation, a transaction budget, an ordering model, a coherency story, and a place in the system's arbitration — and every one of those is a system-integration negotiation rather than a MAC design decision.

What this chapter establishes:

SectionEstablishes
2the four interfaces and what each one costs
4the memory bandwidth, derived, at four line rates
6the clock domains and every crossing
10why one interrupt per frame is arithmetically impossible
12the transaction rate, which is a harder wall than the bandwidth
16reset, and why four domains do not share one
17what the MAC assumes about the system it lands in

What it does not establish, and where it goes instead:

QuestionChapter
how the descriptors are organisedChapter 18.2
how a received frame reaches memoryChapter 18.3
how a transmitted frame leaves itChapter 18.4
how the bursts are shaped for AXIChapter 18.5
how the interrupts are coalescedChapter 18.6
what the MAC's internals look likeModule 19

The forward references above that are bold rather than linked are chapters that are not yet published, which is this track's convention.

One more piece of framing, because it decides how the rest of the module reads. Modules 12 to 17 were about what Ethernet does on the wire. This module is about what Ethernet does to the chip it is on — and the two have almost nothing in common except the frames passing between them.


An Ethernet MAC integrated into a system-on-chip presents four interfaces. The CPU register interface is a slave carrying configuration and status over a few dozen registers, and its bandwidth cost is negligible. The memory interface is a bus master carrying frame data, descriptors and status, and it demands 1.143 times the line rate in each direction because every frame drags a 16-octet descriptor fetch and a 16-octet status writeback with it. The PHY or xMII interface carries the frames themselves and its width and clock are fixed by the standard, ranging from four bits at 25 megahertz for MII to 512 bits at 195.3125 megahertz for 100 gigabit CGMII. The interrupt interface is one output wire costing a single flop, and at 1 gigabit per second with minimum-size frames it asks for 1.488 million interrupts per second, which at a 2 microsecond service routine is 297.6 percent of one CPU. The asymmetry is the chapter's opening argument: the interface with the smallest gate cost has by far the largest system cost, and the two interfaces an integration review spends its time on, registers and PHY, are the two that never cause a problem.The MACan SoC IP blockCPU registersslave — a few dozenMemoryMASTER — 1.143x line ratePHY / xMIIfixed by the standardInterruptone flopCost: nonenever breaksCost: 28.6 GB/sat 100 Gb/s, both waysCost: 297.6% of a CPUat 1 Gb/s, 64-octet framesThe asymmetryleast gates, most system12
Figure 1 — four interfaces, and the one with the smallest hardware cost has the largest system cost.

2. The Four Interfaces, and What Each One Costs

Take them one at a time, because the cost profile of each is different and integration mistakes come from assuming they are alike.

The CPU register interface is the cheapest and it is the one that gets the most attention. A few dozen 32-bit registers on APB or AXI-Lite: base addresses, enables, status, counters. Bandwidth: negligible — a few accesses per frame at most, and on a well-designed MAC, none at all in the steady state.

The memory interface is the expensive one. Section 4's table: 1.143 times the line rate in each direction, which at 100 Gb/s is 228.6 Gb/s — 28.6 GB/s — of a system's memory bandwidth, for one port.

The PHY interface is fixed by the standard and costs pins.

InterfaceWidthClockRate
MII4 bits25 MHz0.100 Gb/s
GMII8 bits125 MHz1.000 Gb/s
XGMII64 bits156.25 MHz10.000 Gb/s
25G XXVGMII64 bits390.625 MHz25.000 Gb/s
100G CGMII512 bits195.3125 MHz100.000 Gb/s

Read the last two rows against each other. 25 Gb/s and 100 Gb/s are reached by completely different means — one raises the clock to 390 MHz on a narrow bus, the other widens the bus to 512 bits and keeps the clock under 200 MHz — and which choice a design faces is decided by the PHY, not the MAC.

The interrupt interface is one wire and it is the one that breaks first. Section 10 derives it: one interrupt per frame at 1 Gb/s with 64-octet frames is 1.488 million interrupts per second, which at a 2 µs interrupt service routine is 297.6% of one CPU. The wire is free and what it asks of software is not.

Put the four costs side by side:

InterfaceGate costSystem costWhere it breaks
CPU registerssmallnegligiblenever
memorymoderate1.143× line ratebandwidth, then transactions
PHYpinsfixed by the standardpin count at 100G
interruptone flopCPU timeimmediately

The asymmetry is the point. The interface with the smallest hardware cost has the largest system cost, and the two interfaces an integration review spends its time on — registers and PHY — are the two that never cause a problem.


3. RTL 1 — The Register File, and the Domain It Is Not In

The register file looks like the easiest block in the MAC and it contains the module's first hard problem: almost nothing it reports lives in its own clock domain.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// macsoc_pkg -- shared types for an Ethernet MAC as an SoC IP block.
// -----------------------------------------------------------------------
package macsoc_pkg;

  localparam int ADDR_W   = 64;          // host physical address
  localparam int REG_AW   = 12;          // 4 KiB register window
  localparam int DATA_W   = 32;          // register data width
  localparam int NUM_CNT  = 16;          // RMON-style counters -- 19.7

  // The register map, kept deliberately small. Everything an operator
  // needs in the steady state is a counter; everything else is written
  // once at bring-up.
  typedef enum logic [REG_AW-1:0] {
    REG_CTRL        = 12'h000,
    REG_STATUS      = 12'h004,
    REG_RX_RING_LO  = 12'h010,
    REG_RX_RING_HI  = 12'h014,
    REG_RX_RING_LEN = 12'h018,
    REG_TX_RING_LO  = 12'h020,
    REG_TX_RING_HI  = 12'h024,
    REG_TX_RING_LEN = 12'h028,
    REG_IRQ_EN      = 12'h030,
    REG_IRQ_STATUS  = 12'h034,
    REG_COALESCE    = 12'h038,
    REG_CNT_BASE    = 12'h100
  } reg_addr_e;

  // Which domain a register's CONTENT lives in. Section 6's subject,
  // and the reason this file is not a simple array of flops.
  typedef enum logic [1:0] {
    DOM_HOST = 2'd0,                     // the bus clock -- no crossing
    DOM_RX   = 2'd1,                     // the receive recovered clock
    DOM_TX   = 2'd2,                     // the transmit clock
    DOM_PTP  = 2'd3                      // 16.3's timestamp clock
  } domain_e;

  typedef struct packed {
    logic        enable;
    logic        promiscuous;
    logic        loopback;
    logic [15:0] max_frame;              // 5.4's MTU
  } ctrl_t;

endpackage
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// mac_register_file -- the CPU-facing slave.
//
// Configuration flows OUT of the host domain and is synchronised into
// the domains that use it. Counters flow IN from other domains and are
// synchronised on the way back. Neither direction is a simple flop,
// and section 19's rejected property is about what happens when a
// verification engineer forgets that.
// -----------------------------------------------------------------------
module mac_register_file
  import macsoc_pkg::*;
(
  input  logic                clk_host,
  input  logic                rst_host_n,

  // APB-style slave -- deliberately the simplest possible.
  input  logic                psel,
  input  logic                penable,
  input  logic                pwrite,
  input  logic [REG_AW-1:0]   paddr,
  input  logic [DATA_W-1:0]   pwdata,
  output logic [DATA_W-1:0]   prdata,
  output logic                pready,
  output logic                pslverr,

  // Configuration out, in the HOST domain. Every consumer synchronises.
  output ctrl_t               cfg_ctrl,
  output logic [ADDR_W-1:0]   cfg_rx_ring_base,
  output logic [ADDR_W-1:0]   cfg_tx_ring_base,
  output logic [15:0]         cfg_rx_ring_len,
  output logic [15:0]         cfg_tx_ring_len,
  output logic [31:0]         cfg_coalesce,

  // Counters in, ALREADY synchronised by their owners. This module
  // does not cross domains itself; it refuses to hold a value that
  // has not been handed to it in its own clock.
  input  logic [31:0]         cnt_value  [NUM_CNT],
  input  logic                cnt_valid  [NUM_CNT],

  input  logic [31:0]         irq_status_sync,
  output logic [31:0]         irq_enable,
  output logic                irq_ack,

  output logic                cfg_write_while_enabled
);

  ctrl_t             ctrl_q;
  logic [ADDR_W-1:0] rx_base_q, tx_base_q;
  logic [15:0]       rx_len_q,  tx_len_q;
  logic [31:0]       coalesce_q, irq_en_q;
  logic [31:0]       cnt_shadow [NUM_CNT];

  wire access = psel && penable;
  wire wr     = access && pwrite;

  // A ring base written while the MAC is enabled is the classic
  // integration bug: the DMA is mid-descriptor against the old base
  // and the next fetch uses the new one. The register file cannot
  // prevent it -- it can only refuse it and say so.
  assign cfg_write_while_enabled =
    wr && ctrl_q.enable &&
    (paddr == REG_RX_RING_LO || paddr == REG_RX_RING_HI ||
     paddr == REG_TX_RING_LO || paddr == REG_TX_RING_HI ||
     paddr == REG_RX_RING_LEN || paddr == REG_TX_RING_LEN);

  always_ff @(posedge clk_host or negedge rst_host_n) begin
    int i;
    if (!rst_host_n) begin
      ctrl_q     <= '0;
      rx_base_q  <= '0; tx_base_q <= '0;
      rx_len_q   <= '0; tx_len_q  <= '0;
      coalesce_q <= '0; irq_en_q  <= '0;
      irq_ack    <= 1'b0;
      for (i = 0; i < NUM_CNT; i++) cnt_shadow[i] <= '0;
    end else begin
      irq_ack <= 1'b0;

      // Counters land here only when their owner asserts valid, which
      // that owner does from ITS synchroniser's output. The shadow is
      // therefore always a value that was coherent at some instant --
      // just not necessarily this one.
      for (i = 0; i < NUM_CNT; i++)
        if (cnt_valid[i]) cnt_shadow[i] <= cnt_value[i];

      if (wr && !cfg_write_while_enabled) begin
        case (paddr)
          REG_CTRL:        {ctrl_q.max_frame, ctrl_q.loopback,
                            ctrl_q.promiscuous, ctrl_q.enable}
                             <= pwdata[18:0];
          REG_RX_RING_LO:  rx_base_q[31:0]  <= pwdata;
          REG_RX_RING_HI:  rx_base_q[63:32] <= pwdata;
          REG_RX_RING_LEN: rx_len_q         <= pwdata[15:0];
          REG_TX_RING_LO:  tx_base_q[31:0]  <= pwdata;
          REG_TX_RING_HI:  tx_base_q[63:32] <= pwdata;
          REG_TX_RING_LEN: tx_len_q         <= pwdata[15:0];
          REG_IRQ_EN:      irq_en_q         <= pwdata;
          REG_COALESCE:    coalesce_q       <= pwdata;
          REG_IRQ_STATUS:  irq_ack          <= 1'b1;   // write-1-to-clear
          default: ;
        endcase
      end
    end
  end

  always_comb begin
    prdata  = '0;
    pslverr = 1'b0;
    if (paddr >= REG_CNT_BASE && paddr < REG_CNT_BASE + (NUM_CNT * 4))
      prdata = cnt_shadow[(paddr - REG_CNT_BASE) >> 2];
    else begin
      case (paddr)
        REG_CTRL:        prdata = {13'b0, ctrl_q.max_frame, ctrl_q.loopback,
                                   ctrl_q.promiscuous, ctrl_q.enable};
        REG_STATUS:      prdata = {31'b0, ctrl_q.enable};
        REG_RX_RING_LO:  prdata = rx_base_q[31:0];
        REG_RX_RING_HI:  prdata = rx_base_q[63:32];
        REG_RX_RING_LEN: prdata = {16'b0, rx_len_q};
        REG_TX_RING_LO:  prdata = tx_base_q[31:0];
        REG_TX_RING_HI:  prdata = tx_base_q[63:32];
        REG_TX_RING_LEN: prdata = {16'b0, tx_len_q};
        REG_IRQ_EN:      prdata = irq_en_q;
        REG_IRQ_STATUS:  prdata = irq_status_sync;
        REG_COALESCE:    prdata = coalesce_q;
        default:         pslverr = access && !pwrite;
      endcase
    end
  end

  assign pready           = 1'b1;
  assign cfg_ctrl         = ctrl_q;
  assign cfg_rx_ring_base = rx_base_q;
  assign cfg_tx_ring_base = tx_base_q;
  assign cfg_rx_ring_len  = rx_len_q;
  assign cfg_tx_ring_len  = tx_len_q;
  assign cfg_coalesce     = coalesce_q;
  assign irq_enable       = irq_en_q;

endmodule

Classification: a single-domain register slave with an explicit refusal to cross domains itself.

What it teaches: that a register file's hard problem is not the decode, it is the provenance of every value it holds. cnt_shadow is a host-domain flop; the counter it shadows increments in the receive recovered clock, which is a different domain at a different frequency with no fixed phase relationship. The module deliberately does not synchronise it — it accepts an already-synchronised value with a valid, so that exactly one block owns each crossing and a reviewer can find every crossing by looking for the synchronisers rather than by reading every register.

And it teaches that cfg_write_while_enabled is the integration bug that survives every unit test. A driver that reconfigures a ring base without disabling the MAC passes in simulation, because the testbench's DMA is idle when the write lands. On real traffic the DMA is mid-descriptor against the old base and the next fetch uses the new one — so it writes a frame into whatever the old base's neighbour happens to be, which is memory belonging to something else.

Deliberately simplified: the counter shadow is 32 bits and a real RMON counter set is a mixture of 32-bit and 64-bit values, which reintroduces a problem this module dodges: a 64-bit counter cannot be read atomically over a 32-bit bus. Chapter 19.7's subject is the latch-on-read-low convention that fixes it, and this file's flat 32-bit array assumes it away. The register map is also flat rather than banked per queue, which Chapter 18.7's multi-queue makes untenable.

Production implication: cfg_write_while_enabled should be both a refusal and an interrupt. Refusing silently means a driver bug becomes a mysterious absence of traffic; refusing loudly means the driver author sees it on the first run. This is the same argument Chapter 14.1 §17 made about configuration errors generally — a value that is wrong from the moment it is written should be rejected at the moment it is written, not diagnosed from its consequences three layers away.


4. The Memory Bandwidth, Derived

Every frame that arrives costs the memory system more than the frame. Section 1's table asserted the number; this section derives it, because the derivation is where the design levers are.

Start with the frame rate. A frame on the wire occupies its own octets plus 8 octets of preamble and SFDChapter 4.2and 12 octets of interframe gap. So a 64-octet frame occupies 84 octets of wire time.

Line rate64-octet frames1518-octet frames
1 Gb/s1.4881 Mfps0.0813 Mfps
10 Gb/s14.8810 Mfps0.8127 Mfps
25 Gb/s37.2024 Mfps2.0319 Mfps
100 Gb/s148.8095 Mfps8.1274 Mfps

Now the per-frame memory cost on receive. Three accesses:

AccessSizeWhy
descriptor fetch16 octetswhere does this frame go
frame data writethe framethe frame itself
status writeback16 octetshow long was it, was it good

So the ratio is (F + 32) / F, and the whole table follows from it.

Frame sizeAmplificationAt 1 Gb/sAt 100 Gb/s
641.500×1.143 Gb/s114.286 Gb/s
1281.250×1.081 Gb/s108.1 Gb/s
2561.125×1.043 Gb/s104.3 Gb/s
5121.063×1.023 Gb/s102.3 Gb/s
15181.021×1.008 Gb/s100.780 Gb/s

Two things about that table are worth stating carefully, because both are easy to get backwards.

The amplification factor is 1.5× at 64 octets and the bandwidth figure is 1.143×. Those are not inconsistent: the amplification is against the frame's own octets, and the bandwidth is against the line rate — and a line carrying 64-octet frames is only delivering 76.19% of its rate as frame data in the first place, because 20 of every 84 octets are preamble and gap. 0.7619 × 1.5 = 1.143.

And the descriptor share is 33.3% at every line rate. It is a function of frame size alone, so a design cannot outrun it by being faster — which is the opposite of most bandwidth problems, where the overhead is a fixed cost that amortises as the rate rises.

Both directions at once, worst case:

Line rateRX + TX, 64-octet framesIn bytes
1 Gb/s2.286 Gb/s0.286 GB/s
10 Gb/s22.857 Gb/s2.857 GB/s
25 Gb/s57.143 Gb/s7.143 GB/s
100 Gb/s228.571 Gb/s28.571 GB/s

The last row is the one to take to a system architect. A single 100 Gb/s Ethernet port at minimum frame size demands 28.6 GB/s of memory bandwidth — which is a substantial fraction of a DDR4 channel's theoretical peak and more than its achievable one, for one peripheral, before the CPU has touched the data.

And software touches it afterwards, which doubles the figure again if the stack copies. This is why Chapter 18.3's scatter-gather and Chapter 18.7's offload exist — not as optimisations but as the difference between a port that runs at line rate and one that does not.

The design levers the derivation exposes, in order of size:

LeverEffect on the 33.3%Whose chapter
larger frames33.3% to 2.1%the network's, not the MAC's
descriptor prefetch in batchesamortises the fetchChapter 18.2
writeback batchingamortises the statusChapter 18.3
smaller descriptorslinear in the sizeChapter 18.2
a faster busnone — the ratio is unchangednobody's

Row five is the one people reach for and it does not help. Doubling the bus width halves the time and leaves the ratio at 33.3%; the descriptor traffic scales with the frame traffic exactly, which is what makes it a structural cost rather than a bottleneck.


A single received frame costs the memory system three bus transactions rather than one. First a 16-octet descriptor fetch asks where the frame should go. Then the frame's own octets are written. Then a 16-octet status writeback records how long it was and whether it passed the frame check sequence. For a 64-octet frame that is 96 octets of memory traffic for 64 octets of frame, an amplification of 1.5 times on the frame's own octets, which against the line rate is 1.143 times because only 76.19 percent of the wire carries frame data at minimum size. The descriptor traffic is therefore 33.3 percent of the total at every line rate, from 0.381 gigabits per second at 1 gigabit to 38.095 gigabits per second at 100 gigabit, because the ratio depends on the frame size and not on the rate. At 1518-octet frames the same share is 2.1 percent. The consequence is that the overhead cannot be outrun by a faster link, which is the opposite of most fixed costs, and the only levers are larger frames, smaller descriptors, or batching several descriptors into one transaction.One frame arrives64 octetsDescriptor fetch16 octets — readFrame data write64 octetsStatus writeback16 octets — write96 octets moved1.5x the framevs the line rate0.7619 x 1.5 = 1.143Descriptors: 33.3%at EVERY line rateAt 1518 octets2.1%The only leversbigger frames, orbatching12
Figure 2 — one frame, three transactions, and a third of the memory traffic that never amortises.

5. RTL 2 — The Memory Master

The MAC's memory interface is the block that turns a frame into bus transactions, and its whole job is to keep Section 12's transaction rate down without violating anything.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// mac_memory_master -- the MAC's bus-mastering front end.
//
// It accepts logical requests (fetch this descriptor, write this frame
// data, post this status) and issues bus transactions. It enforces the
// two rules the bus imposes and the MAC does not naturally respect:
// a burst may not cross a 4 KiB boundary, and the number of
// outstanding transactions is bounded.
// -----------------------------------------------------------------------
module mac_memory_master
  import macsoc_pkg::*;
#(
  parameter int BUS_BYTES    = 32,       // 256-bit data bus
  parameter int MAX_OUTSTAND = 16,
  parameter int MAX_BEATS    = 16
)(
  input  logic               clk,
  input  logic               rst_n,

  // Logical request in.
  input  logic               req_valid,
  output logic               req_ready,
  input  logic               req_write,
  input  logic [ADDR_W-1:0]  req_addr,
  input  logic [15:0]        req_bytes,
  input  logic [1:0]         req_kind,   // 0 desc, 1 data, 2 status

  // Bus address channel out.
  output logic               ax_valid,
  input  logic               ax_ready,
  output logic               ax_write,
  output logic [ADDR_W-1:0]  ax_addr,
  output logic [7:0]         ax_len,     // beats minus one
  output logic [2:0]         ax_size,

  // Completion in -- one per issued transaction.
  input  logic               done_valid,
  input  logic               done_error,

  output logic [31:0]        c_txn_desc,
  output logic [31:0]        c_txn_data,
  output logic [31:0]        c_txn_status,
  output logic [31:0]        c_split_4k,
  output logic [31:0]        c_stall_outstanding,
  output logic [31:0]        c_bus_error,
  output logic               outstanding_full
);

  localparam int SIZE_ENC = $clog2(BUS_BYTES);

  logic [ADDR_W-1:0] cur_addr;
  logic [15:0]       cur_bytes;
  logic [1:0]        cur_kind;
  logic              busy;

  logic [$clog2(MAX_OUTSTAND+1)-1:0] outstanding;

  // How many bytes may this burst cover before the 4 KiB boundary?
  // A burst that crosses one is illegal on AXI and the MAC has no
  // reason to respect a boundary it cannot see -- a frame lands
  // wherever the descriptor's buffer pointer says.
  logic [12:0] to_boundary;
  assign to_boundary = 13'h1000 - {1'b0, cur_addr[11:0]};

  logic [15:0] max_burst_bytes;
  assign max_burst_bytes = MAX_BEATS * BUS_BYTES;

  logic [15:0] this_burst;
  always_comb begin
    this_burst = cur_bytes;
    if (this_burst > max_burst_bytes) this_burst = max_burst_bytes;
    if (this_burst > {3'b0, to_boundary}) this_burst = {3'b0, to_boundary};
  end

  assign outstanding_full = (outstanding == MAX_OUTSTAND[$bits(outstanding)-1:0]);
  assign req_ready        = !busy;

  assign ax_valid = busy && !outstanding_full;
  assign ax_write = (cur_kind != 2'd0);          // only a fetch reads
  assign ax_addr  = cur_addr;
  assign ax_size  = SIZE_ENC[2:0];
  assign ax_len   = (( this_burst + BUS_BYTES - 1) / BUS_BYTES) - 1;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      busy <= 1'b0; cur_addr <= '0; cur_bytes <= '0; cur_kind <= '0;
      outstanding <= '0;
      c_txn_desc <= '0; c_txn_data <= '0; c_txn_status <= '0;
      c_split_4k <= '0; c_stall_outstanding <= '0; c_bus_error <= '0;
    end else begin
      if (!busy && req_valid) begin
        busy      <= 1'b1;
        cur_addr  <= req_addr;
        cur_bytes <= req_bytes;
        cur_kind  <= req_kind;
      end

      if (busy && outstanding_full)
        c_stall_outstanding <= c_stall_outstanding + 1;

      if (ax_valid && ax_ready) begin
        outstanding <= outstanding + 1;

        case (cur_kind)
          2'd0: c_txn_desc   <= c_txn_desc   + 1;
          2'd1: c_txn_data   <= c_txn_data   + 1;
          default: c_txn_status <= c_txn_status + 1;
        endcase

        // A split happened if the boundary, not the beat limit, was
        // what shortened this burst. Counting it separately is what
        // makes a misaligned buffer pool visible.
        if ({3'b0, to_boundary} < cur_bytes &&
            {3'b0, to_boundary} < max_burst_bytes)
          c_split_4k <= c_split_4k + 1;

        if (this_burst >= cur_bytes) begin
          busy      <= 1'b0;
          cur_bytes <= '0;
        end else begin
          cur_addr  <= cur_addr  + {{(ADDR_W-16){1'b0}}, this_burst};
          cur_bytes <= cur_bytes - this_burst;
        end
      end

      if (done_valid) begin
        outstanding <= outstanding - 1;
        if (done_error) c_bus_error <= c_bus_error + 1;
      end

      // Simultaneous issue and completion cancel.
      if (ax_valid && ax_ready && done_valid)
        outstanding <= outstanding;
    end
  end

endmodule

Classification: a request splitter with an outstanding-transaction governor and per-kind accounting.

What it teaches: that the MAC's memory interface exists to reconcile two things that have no reason to agree — a frame, whose length is decided by whoever sent it, and a bus burst, whose legality is decided by a 4 KiB boundary and a beat limit the frame has never heard of. A 1518-octet frame landing 3 KiB into a page becomes two bursts, not one, and no amount of care in the MAC changes that; only the buffer pool's alignment does.

And it teaches that c_txn_desc, c_txn_data and c_txn_status must be separate counters. One combined transaction count cannot distinguish a port moving large frames efficiently from a port moving small frames and drowning in bookkeeping — and Section 4's whole argument is that those two look identical in bandwidth and completely different in transaction rate. Three counters make the ratio readable; one makes it invisible.

Deliberately simplified: a real master pipelines requests rather than handling one at a time. This module goes busy on a request and stays busy until its last burst issues, so a descriptor fetch cannot overlap a data write — which at Section 12's transaction rates is exactly the serialisation a 100 Gb/s port cannot afford. Chapter 18.5's subject is the request pipeline that removes it. The write data channel is also absent entirely, because this chapter is about the interface's shape rather than its datapath, and done_valid collapses AXI's separate read and write response channels into one.

Production implication: c_stall_outstanding is the counter that tells an integrator whether the MAC or the system is the limit. Rising with traffic means the memory system is not completing fast enough and the MAC is idling on its own outstanding limit — the fix is a larger limit, or a memory system that responds sooner. Flat at zero under load means the MAC is issuing at the rate it wants and the limit is elsewhere. Without it, a port that runs at 60% of line rate produces an argument between two teams and no evidence.


6. The Clock Domains, and What Crosses Them

A MAC in an SoC has four clock domains and no two of them are related. This section names them, names every crossing, and derives what a crossing costs.

DomainSourceTypical frequencyWhat lives in it
hostthe SoC's bus clock250–400 MHzregisters, the memory master, descriptors
receiverecovered from the wire125 MHz to 390.625 MHzthe parser, receive counters
transmitthe local reference125 MHz to 390.625 MHzthe assembler, transmit counters
PTPthe synchronised clockoften a separate referenceChapter 16.3's capture

Row two is the one that surprises people coming from other peripherals. The receive clock is recovered from the incoming signal — it is the far end's transmit clock, arriving over the cable. So it is not a clock the SoC generates, not a clock it can stop, and not a clock that is synchronous with anything else on the chip. It exists because a partner is transmitting, and it stops when the partner stops.

Which makes a crossing out of nearly everything interesting.

CrossingDirectionWhat crossesMechanism
host → rxoutconfiguration: enable, MTU, promiscuoustwo-flop synchroniser
host → txoutconfiguration, the transmit requesttwo-flop synchroniser
rx → hostinframe dataasynchronous FIFO — Section 9
rx → hostincountershandshake, Section 14
tx → hostincompletion, countershandshake
ptp → hostintimestampsChapter 16.3 §3's capture register

And here is what a crossing costs, in the units that matter.

DomainTwo flopsThree flops
GMII 125 MHz16.000 ns24.000 ns
XGMII 156.25 MHz12.800 ns19.200 ns
CGMII 195.3125 MHz10.240 ns15.360 ns
XXVGMII 390.625 MHz5.120 ns7.680 ns
host 250 MHz8.000 ns12.000 ns
host 400 MHz5.000 ns7.500 ns

A full handshake is a round trip and therefore roughly the sum of two of those — about 24 ns from a GMII receive domain to a 250 MHz host, or 20.8 ns from XGMII.

Now put that latency against the frame rate, because the comparison is the module's second big number.

Line rate64-octet frames per µsFrames arriving in one 30 ns handshake
1 Gb/s1.490.045
10 Gb/s14.880.446
25 Gb/s37.201.116
100 Gb/s148.814.464

Read the last row. At 100 Gb/s with minimum frames, four and a half frames arrive in the time it takes one value to cross from the receive domain to the host domain. A design that synchronises per frame does not work; a design that asserts a property relating a receive-domain signal to a host-domain one is asserting something about two instants that are four frames apart, which is Section 19's rejected class.

The engineering consequence is a rule and it is worth stating flatly: cross data in a FIFO and cross status in a handshake, and never cross either one signal at a time. Section 9's asynchronous FIFO crosses a whole frame's worth of octets with one pointer synchronisation; Section 14's telemetry crosses a whole counter set with one handshake. The cost is amortised over the payload, and it has to be.


7. RTL 3 — The PHY Boundary

The xMII side is where the MAC stops being a bus master and starts being an Ethernet device, and the adapter's job is to make five different physical interfaces look like one.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// phy_boundary_adapter -- normalises the xMII family to one internal
// interface.
//
// MII, GMII, XGMII, XXVGMII and CGMII differ in width and clock and
// agree on almost nothing else. Everything above this module sees a
// width-W stream with a start, an end and a byte-enable on the last
// beat. Everything below sees the standard's signalling.
// -----------------------------------------------------------------------
module phy_boundary_adapter
  import macsoc_pkg::*;
#(
  parameter int XMII_BYTES = 8,          // 1 = MII/GMII, 8 = XGMII, 64 = CGMII
  parameter int INT_BYTES  = 8
)(
  input  logic                      clk_rx,     // RECOVERED from the wire
  input  logic                      rst_rx_n,

  // xMII receive side.
  input  logic [XMII_BYTES*8-1:0]   xgmii_rxd,
  input  logic [XMII_BYTES-1:0]     xgmii_rxc,  // control, not data
  input  logic                      rx_clk_present,

  // Normalised internal stream out, still in clk_rx.
  output logic                      s_valid,
  output logic [INT_BYTES*8-1:0]    s_data,
  output logic [$clog2(INT_BYTES+1)-1:0] s_bytes,
  output logic                      s_sof,
  output logic                      s_eof,
  output logic                      s_error,

  output logic [31:0]               c_sof,
  output logic [31:0]               c_eof,
  output logic [31:0]               c_local_fault,
  output logic [31:0]               c_clock_lost,
  output logic                      rx_clock_stopped
);

  // The control characters that matter. 4.2's SFD does not appear on
  // XGMII -- the start control character replaces it, which is one of
  // the family's least obvious incompatibilities.
  localparam logic [7:0] XG_START = 8'hFB;
  localparam logic [7:0] XG_TERM  = 8'hFD;
  localparam logic [7:0] XG_ERROR = 8'hFE;
  localparam logic [7:0] XG_IDLE  = 8'h07;

  logic in_frame;

  // A recovered clock that stops is not an error condition on the
  // wire -- it is the partner having stopped transmitting, or the
  // cable having been removed. It is, however, fatal to anything
  // clocked by it, which is why the detector lives in the HOST
  // domain in a real design and is modelled here as an input.
  assign rx_clock_stopped = !rx_clk_present;

  logic [INT_BYTES-1:0] is_start, is_term, is_error;
  always_comb begin
    int i;
    for (i = 0; i < INT_BYTES; i++) begin
      is_start[i] = xgmii_rxc[i] && (xgmii_rxd[i*8 +: 8] == XG_START);
      is_term[i]  = xgmii_rxc[i] && (xgmii_rxd[i*8 +: 8] == XG_TERM);
      is_error[i] = xgmii_rxc[i] && (xgmii_rxd[i*8 +: 8] == XG_ERROR);
    end
  end

  // How many bytes of this beat belong to the frame? Everything up to
  // the terminate character.
  logic [$clog2(INT_BYTES+1)-1:0] bytes_this_beat;
  always_comb begin
    int i;
    bytes_this_beat = INT_BYTES[$bits(bytes_this_beat)-1:0];
    for (i = INT_BYTES-1; i >= 0; i--)
      if (is_term[i]) bytes_this_beat = i[$bits(bytes_this_beat)-1:0];
  end

  always_ff @(posedge clk_rx or negedge rst_rx_n) begin
    if (!rst_rx_n) begin
      in_frame <= 1'b0;
      s_valid <= 1'b0; s_sof <= 1'b0; s_eof <= 1'b0; s_error <= 1'b0;
      s_data <= '0; s_bytes <= '0;
      c_sof <= '0; c_eof <= '0; c_local_fault <= '0; c_clock_lost <= '0;
    end else begin
      s_valid <= 1'b0; s_sof <= 1'b0; s_eof <= 1'b0; s_error <= 1'b0;

      if (rx_clock_stopped) begin
        c_clock_lost <= c_clock_lost + 1;
        in_frame     <= 1'b0;
      end else if (|is_start && !in_frame) begin
        in_frame <= 1'b1;
        s_valid  <= 1'b1;
        s_sof    <= 1'b1;
        s_data   <= xgmii_rxd;
        s_bytes  <= INT_BYTES[$bits(s_bytes)-1:0];
        c_sof    <= c_sof + 1;
      end else if (in_frame) begin
        s_valid <= 1'b1;
        s_data  <= xgmii_rxd;
        if (|is_term) begin
          in_frame <= 1'b0;
          s_eof    <= 1'b1;
          s_bytes  <= bytes_this_beat;
          c_eof    <= c_eof + 1;
        end else begin
          s_bytes <= INT_BYTES[$bits(s_bytes)-1:0];
        end
        if (|is_error) begin
          s_error       <= 1'b1;
          c_local_fault <= c_local_fault + 1;
        end
      end
    end
  end

endmodule

Classification: a width- and encoding-normaliser at an asynchronous boundary, with an explicit clock-presence input.

What it teaches: that the xMII family's differences are not width alone. A GMII receive path has Chapter 4.2's preamble and SFD as data; an XGMII path has a start control character instead, and the preamble that follows it is padding whose only job is to keep the octet count right. A parser written against one and ported to the other looks for an SFD that is not there — which is a bring-up failure that presents as "the link is up and no frames arrive."

And it teaches that a recovered clock stopping is a first-class event, not an error. The receive clock exists because a partner is transmitting. When the cable is unplugged it simply stops — and every flop in the receive domain stops with it, including any counter that was going to report the problem. The detector therefore cannot live in the domain it is detecting, which is why rx_clk_present is an input from elsewhere rather than something this module derives.

Deliberately simplified: XMII_BYTES and INT_BYTES are separate parameters and the module assumes they are equal — the width conversion that makes a 512-bit CGMII stream into a 256-bit internal one is absent, and it is not trivial, because a start character may land in either half. Alignment is also assumed: a real XGMII start character may appear in lane 0 or lane 4, and the realignment that follows is Chapter 19.2's problem. Deficit-idle-count and the interframe-gap accounting that goes with it are absent entirely.

Production implication: c_sof and c_eof should be compared, and the comparison is one of the most useful two-counter checks in a MAC. They are equal on a healthy link. c_sof exceeding c_eof by a growing margin means frames are starting and not terminating — a truncation at the PHY, or a start character being seen mid-frame, both of which point at the serdes rather than at the MAC. The two counters cost almost nothing and separate a PHY fault from a MAC fault before a waveform is opened, which is Chapter 21.1's whole method in one comparison.


8. Why the Crossing FIFO Is Not Optional, and How Deep

A synchroniser crosses a signal. A frame is not a signal, and the block that crosses it has to be sized against something the MAC does not control: how long the memory system might make it wait.

The receive path has no backpressure toward the wire. Chapter 7.2 established it and it is worth restating in this chapter's terms: when a frame is arriving, it arrives. The MAC cannot ask the partner to pause mid-frame — Chapter 14.2's PAUSE operates between frames and takes 13.3 µs to take effect — so every octet that arrives must go somewhere immediately.

Which makes the receive FIFO's depth a direct function of the worst-case time before the memory system accepts data.

Line rate0.5 µs stall1 µs2 µs5 µs
1 Gb/s0.06 KiB0.12 KiB0.24 KiB0.61 KiB
10 Gb/s0.61 KiB1.22 KiB2.44 KiB6.10 KiB
25 Gb/s1.53 KiB3.05 KiB6.10 KiB15.26 KiB
100 Gb/s6.10 KiB12.21 KiB24.41 KiB61.04 KiB

The bottom-right cell is the one to look at. A 100 Gb/s MAC that must survive a 5 µs memory stall needs 61 KiB of on-chip FIFO, which is a substantial SRAM for one peripheral — and 5 µs is not a pathological stall on a system with a DRAM refresh, a competing master and an arbiter that is not aware Ethernet has a deadline.

And the column headings are not a design choice — they are a system property nobody writes down. The MAC's integrator knows the FIFO depth; the memory subsystem's owner knows the worst-case latency; and in most projects neither number is stated in the other team's terms until a port starts dropping frames under load.

This is the shape of every integration failure in this module, so state it once:

The MAC knowsThe system knowsNobody owns
FIFO depthworst-case memory latencywhether one covers the other
transaction rate offeredtransaction rate acceptedthe gap
interrupts generatedinterrupt service timeCPU load
burst length requestedburst length honouredefficiency

Every row is a pair of numbers held by different people, and every row's failure looks like "the Ethernet is slow".

One more thing the depth table does not show, and it is the reason the FIFO is asynchronous rather than merely elastic. The two sides run on unrelated clocks: a recovered receive clock at 156.25 MHz nominal and a host clock at 250 MHz, with no common reference and a frequency ratio that is not rational. A synchronous FIFO with a rate-matching read is not sufficient; the pointers themselves must cross.


The MAC spans four unrelated clock domains. The host domain runs at 250 to 400 megahertz from the system-on-chip's bus clock and holds the registers, the memory master and the descriptors. The receive domain runs at 125 to 390.625 megahertz on a clock recovered from the incoming signal, which means it is the far end's transmit clock arriving over the cable, and it stops entirely when the partner stops transmitting. The transmit domain runs from the local reference. The PTP domain runs from the synchronised clock and holds the timestamp capture. Configuration crosses outward from host to receive and transmit through two-flop synchronisers. Frame data crosses inward from receive to host through an asynchronous FIFO with Gray-coded pointers. Counters cross inward through a handshake that snapshots the whole set in one source-clock cycle. A full handshake round trip is about 24 nanoseconds from a 125 megahertz receive domain to a 250 megahertz host, and at 100 gigabits per second with minimum-size frames 4.46 frames arrive during that time, which is why per-frame synchronisation is impossible and why a property relating a host-domain signal to a receive-domain one refers to two instants four frames apart.Host domain250-400 MHz — the busReceive domainRECOVERED from the wireTransmit domainlocal referencePTP domain16.3's captureTwo-flop syncconfig outwardAsync FIFOframe data inwardHandshake snapshotcounters inward24 ns round trip125 MHz to 250 MHz4.46 frames arrivein that time, at 100 Gb/s12
Figure 3 — four domains, six crossings, and a receive clock that arrives over the cable.

9. RTL 4 — The Asynchronous FIFO

One block, two clocks, and the whole of its correctness in the way the pointers are encoded.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// async_fifo_cdc -- the receive-to-host frame crossing.
//
// Gray-coded pointers, two-flop synchronisers, and a deliberately
// pessimistic full/empty. The pessimism is the point: a synchronised
// pointer is always STALE, so the reader sees fewer entries than
// exist and the writer sees less space than exists. Both errors are
// safe; the opposite of either is not.
// -----------------------------------------------------------------------
module async_fifo_cdc
  import macsoc_pkg::*;
#(
  parameter int WIDTH = 72,              // 64 data + 8 control
  parameter int DEPTH = 512              // must be a power of two
)(
  // Write side -- the recovered receive clock.
  input  logic               clk_w,
  input  logic               rst_w_n,
  input  logic               w_en,
  input  logic [WIDTH-1:0]   w_data,
  output logic               w_full,
  output logic [31:0]        c_overflow,

  // Read side -- the host clock.
  input  logic               clk_r,
  input  logic               rst_r_n,
  input  logic               r_en,
  output logic [WIDTH-1:0]   r_data,
  output logic               r_empty,
  output logic [31:0]        c_underflow
);

  localparam int AW = $clog2(DEPTH);

  logic [WIDTH-1:0] mem [DEPTH];

  logic [AW:0] w_bin, w_gray, w_gray_r1, w_gray_r2;
  logic [AW:0] r_bin, r_gray, r_gray_w1, r_gray_w2;

  function automatic logic [AW:0] to_gray(input logic [AW:0] b);
    return b ^ (b >> 1);
  endfunction

  // ---- write side -------------------------------------------------
  wire [AW:0] w_bin_next = w_bin + {{AW{1'b0}}, (w_en && !w_full)};

  always_ff @(posedge clk_w or negedge rst_w_n) begin
    if (!rst_w_n) begin
      w_bin <= '0; w_gray <= '0; c_overflow <= '0;
      r_gray_w1 <= '0; r_gray_w2 <= '0;
    end else begin
      if (w_en && !w_full) mem[w_bin[AW-1:0]] <= w_data;
      if (w_en &&  w_full) c_overflow <= c_overflow + 1;
      w_bin  <= w_bin_next;
      w_gray <= to_gray(w_bin_next);

      // The read pointer arrives here two flops late. Everything the
      // write side believes about the FIFO's occupancy is therefore
      // at least two read-clock periods out of date -- and that is
      // exactly why it is safe.
      r_gray_w1 <= r_gray;
      r_gray_w2 <= r_gray_w1;
    end
  end

  // Full: the write pointer has wrapped onto the read pointer. In
  // Gray code that is the top two bits inverted and the rest equal.
  assign w_full = (w_gray == {~r_gray_w2[AW:AW-1], r_gray_w2[AW-2:0]});

  // ---- read side --------------------------------------------------
  wire [AW:0] r_bin_next = r_bin + {{AW{1'b0}}, (r_en && !r_empty)};

  always_ff @(posedge clk_r or negedge rst_r_n) begin
    if (!rst_r_n) begin
      r_bin <= '0; r_gray <= '0; c_underflow <= '0;
      w_gray_r1 <= '0; w_gray_r2 <= '0;
    end else begin
      if (r_en && r_empty) c_underflow <= c_underflow + 1;
      r_bin  <= r_bin_next;
      r_gray <= to_gray(r_bin_next);
      w_gray_r1 <= w_gray;
      w_gray_r2 <= w_gray_r1;
    end
  end

  assign r_empty = (r_gray == w_gray_r2);
  assign r_data  = mem[r_bin[AW-1:0]];

endmodule

Classification: a dual-clock FIFO with Gray-coded pointer crossing and deliberately conservative flags.

What it teaches: that Gray coding is not an encoding convenience, it is the only reason the crossing works at all. A binary pointer going from 0111 to 1000 changes four bits; sampled by an unrelated clock mid-transition, any of sixteen values may be captured, and twelve of them are pointers the FIFO never held. A Gray-coded pointer changes exactly one bit per increment, so a mid-transition sample yields either the old value or the new one — both of which are real pointers the FIFO actually had.

And it teaches that both flags are wrong in the safe direction and must be. w_full compares against a read pointer that is two read-clock periods stale, so the write side believes the FIFO is fuller than it is and refuses writes it could have accepted. r_empty is stale the other way and the reader believes it is emptier. Both errors cost throughput; the opposite of either loses data, and there is no encoding that makes the flags exact, because exactness would require a common clock.

Deliberately simplified: the depth is a power of two, which the Gray comparison requires and a real design may not want. Almost-full and almost-empty thresholds are absent, and a production receive FIFO needs almost-full to drive Chapter 14.2's flow control with enough headroom for that chapter's 13.3 µs dead time — which is 1.62 KiB at 1 Gb/s and 162.4 KiB at 100. The memory is a flat array rather than an inferred dual-port SRAM, and there is no reset synchronisation between the two domains, which Section 16 is about.

Production implication: c_overflow is the single most important counter in a MAC's receive path and it is the one most often missing. An overflow means the frame currently arriving lost octets, and the frame will fail Chapter 6.3's FCS check as a result — so the symptom is a CRC error and the cause is a memory stall. An integrator seeing CRC errors and no overflow counter replaces the cable; one seeing CRC errors rising in step with c_overflow looks at the arbiter, which is where the problem is.


10. The Interrupt Interface, and Why One Per Frame Is Impossible

The interrupt is one wire and one flop. It is also the interface that fails first, and it fails by arithmetic rather than by any design error.

Take the naive design: assert an interrupt when a frame has been placed in memory.

Line rateFrame sizeInterrupts per secondAt a 2 µs ISR
1 Gb/s641.488 M297.6% of one CPU
1 Gb/s15180.081 M16.3%
10 Gb/s6414.881 M2976.2%
10 Gb/s15180.813 M162.5%
25 Gb/s6437.202 M7440.5%
100 Gb/s64148.810 M29761.9%
100 Gb/s15188.127 M1625.5%

Row one settles it. A 1 Gb/s port — the slowest rate anybody still builds — at minimum frame size asks for three CPUs to do nothing but service its interrupts. This is not a 100 Gb/s problem that arrives later; it is a problem at the entry-level rate.

And row two is why the problem hides. At 1518-octet frames the same port asks for 16.3% of one CPU, which is high but survivable — so a design tested with large frames passes and the same design meets a burst of minimum-size frames and locks the system up. The frame-size distribution decides whether the design works, and it is chosen by whoever is sending.

Three mechanisms exist and they are not alternatives — a working design uses all three.

MechanismWhat it doesCostChapter
coalescing by countone interrupt per N frameslatency up to N frame timesChapter 18.6
coalescing by timeone interrupt per Tlatency up to TChapter 18.6
polling under loadno interrupts at all while busyCPU spinthe driver's

Row three is worth naming even though it is software, because it changes what the hardware must provide. A driver that disables the interrupt and polls the ring while traffic is heavy needs the ring's state to be readable without an interrupt — which is Chapter 18.2's ownership bit, and it is one of that mechanism's less obvious justifications.

And the combination has a property worth stating: coalescing by count alone can stall indefinitely. A threshold of 32 frames with 31 frames received and no thirty-second holds those 31 frames until one more arrives, which on a quiet link may be seconds. The timer is what bounds it, so the two mechanisms are not alternatives — the count bounds the CPU cost and the timer bounds the latency, and a design with only one of them fails in whichever direction it omitted.


11. RTL 5 — The Interrupt Interface

One output, and everything interesting is in when it is allowed to go high.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// mac_interrupt_iface -- event aggregation, masking and coalescing.
//
// The coalescing here is deliberately minimal -- a count threshold
// and a timer, which is the smallest thing that bounds both the CPU
// cost and the latency. Chapter 18.6's is adaptive.
// -----------------------------------------------------------------------
module mac_interrupt_iface
  import macsoc_pkg::*;
#(
  parameter int NUM_EVENTS = 8
)(
  input  logic                   clk_host,
  input  logic                   rst_host_n,

  // Events, ALREADY in the host domain. Anything arriving from the
  // receive or transmit domain has been through a handshake first.
  input  logic [NUM_EVENTS-1:0]  evt,

  // Which event indices are "a frame completed" and therefore
  // subject to coalescing. Errors are never coalesced.
  input  logic [NUM_EVENTS-1:0]  evt_coalescable,

  input  logic [31:0]            cfg_coalesce,   // {timer_us, count}
  input  logic [31:0]            irq_enable,
  input  logic                   irq_ack,
  input  logic                   tick_1us,

  output logic                   irq,
  output logic [31:0]            irq_status,

  output logic [31:0]            c_irq_raised,
  output logic [31:0]            c_irq_by_count,
  output logic [31:0]            c_irq_by_timer,
  output logic [31:0]            c_events_total,
  output logic                   coalesce_misconfigured
);

  wire [15:0] cfg_count    = cfg_coalesce[15:0];
  wire [15:0] cfg_timer_us = cfg_coalesce[31:16];

  // A count threshold with no timer stalls whenever the traffic stops
  // one frame short. This is the single most common coalescing
  // misconfiguration and the design can see it at configuration time.
  assign coalesce_misconfigured = (cfg_count > 16'd1) && (cfg_timer_us == '0);

  logic [15:0] pending_count;
  logic [15:0] timer_us;
  logic [31:0] status_q;

  wire [NUM_EVENTS-1:0] evt_masked  = evt & irq_enable[NUM_EVENTS-1:0];
  wire [NUM_EVENTS-1:0] evt_coal    = evt_masked &  evt_coalescable;
  wire [NUM_EVENTS-1:0] evt_urgent  = evt_masked & ~evt_coalescable;

  wire count_reached = (pending_count >= cfg_count) && (cfg_count != '0);
  wire timer_expired = (timer_us    >= cfg_timer_us) && (cfg_timer_us != '0)
                       && (pending_count != '0);

  always_ff @(posedge clk_host or negedge rst_host_n) begin
    if (!rst_host_n) begin
      pending_count <= '0; timer_us <= '0; status_q <= '0; irq <= 1'b0;
      c_irq_raised <= '0; c_irq_by_count <= '0; c_irq_by_timer <= '0;
      c_events_total <= '0;
    end else begin
      c_events_total <= c_events_total + $countones(evt);

      // Urgent events -- errors, overflow, a bus fault -- go straight
      // through. Coalescing an error is how a design turns a
      // diagnosable fault into a mysterious one.
      if (|evt_urgent) begin
        status_q <= status_q | {{(32-NUM_EVENTS){1'b0}}, evt_urgent};
        irq      <= 1'b1;
        if (!irq) c_irq_raised <= c_irq_raised + 1;
      end

      if (|evt_coal) begin
        status_q      <= status_q | {{(32-NUM_EVENTS){1'b0}}, evt_coal};
        pending_count <= pending_count + $countones(evt_coal);
      end

      if (tick_1us && pending_count != '0)
        timer_us <= timer_us + 1;

      if (count_reached || timer_expired) begin
        irq           <= 1'b1;
        pending_count <= '0;
        timer_us      <= '0;
        if (!irq) c_irq_raised <= c_irq_raised + 1;
        if (count_reached) c_irq_by_count <= c_irq_by_count + 1;
        else               c_irq_by_timer <= c_irq_by_timer + 1;
      end

      if (irq_ack) begin
        irq      <= 1'b0;
        status_q <= '0;
      end
    end
  end

  assign irq_status = status_q;

endmodule

Classification: an event aggregator with a two-term coalescing policy and an explicit non-coalescable class.

What it teaches: that coalescing must have an exemption and errors are it. A design that coalesces uniformly delays an overflow report by up to the timer's value — and the overflow's cause, a memory stall, is exactly the condition that makes the timer expire slowly. So the report of a problem is delayed by the problem. evt_coalescable is one mask bit per event and it removes the whole class of failure.

And it teaches why c_irq_by_count and c_irq_by_timer must be separate. They say which term is binding. Almost all by-count means the traffic is heavy and the coalescing is doing its job; almost all by-timer means the threshold is never reached and the interrupts are latency-bound rather than rate-bound — so the count is set too high for this traffic and every frame is waiting the full timer. One combined counter cannot distinguish a well-tuned port from a badly tuned one.

Deliberately simplified: irq is level-sensitive with a single acknowledge, and the acknowledge clears the whole status word — which races with an event arriving in the same cycle and loses it. A production design uses write-1-to-clear per bit with a set-priority-over-clear arrangement, and Chapter 18.6's adaptive coalescing replaces the fixed thresholds with ones that track the arrival rate. There is also one interrupt for the whole MAC rather than one per queue, which Chapter 18.7's multi-queue requires.

Production implication: coalesce_misconfigured deserves to be a loud flag rather than a quiet one, because the misconfiguration it catches produces a symptom nobody attributes to coalescing. A count of 32 with no timer on a link carrying 31 frames and then going quiet holds those 31 frames indefinitely — and the report is "some packets arrive seconds late, and only sometimes." The configuration is checkable in one comparison at write time, and diagnosing it from the symptom takes days.


12. The Transaction Rate, Which Is the Harder Wall

Section 4's bandwidth is solvable by widening the bus. Section 12's transaction rate is not, and it is the number that decides whether a design works.

A received frame costs three bus transactions in the simplest design: a descriptor fetch, a data write, a status writeback. Multiply by the frame rate.

Line rate64-octet frame rateTransactions/sAt 250 MHzAt 400 MHz
1 Gb/s1.49 Mfps4.46 M0.018/cycle0.011/cycle
10 Gb/s14.88 Mfps44.64 M0.179/cycle0.112/cycle
25 Gb/s37.20 Mfps111.61 M0.446/cycle0.279/cycle
100 Gb/s148.81 Mfps446.43 M1.786/cycle1.116/cycle

The bottom two cells are both greater than one, and a single AXI address channel issues at most one transaction per cycle. So a 100 Gb/s MAC receiving minimum-size frames cannot present its traffic through one address channel at any clock frequency a modern SoC runs at — 446 million transactions per second would need a 446 MHz bus doing nothing else, and the write channel needs its own.

Contrast that with the bandwidth, which is entirely solvable:

Line rateBandwidth neededWidth at 250 MHzWidth at 400 MHz
1 Gb/s1.14 Gb/s8 bits8 bits
10 Gb/s11.43 Gb/s64 bits32 bits
25 Gb/s28.57 Gb/s128 bits128 bits
100 Gb/s114.30 Gb/s512 bits512 bits

512 bits at 250 MHz is a wide bus and it is a bus that exists. The transaction rate has no such answer, because widening the bus does not reduce the number of transactions — a descriptor fetch is 16 octets whether the bus is 64 bits or 512, and it is still one transaction.

Which is why Module 18's remaining chapters are, almost without exception, about transaction count rather than bandwidth:

MechanismTransactions removedChapter
fetching 8 descriptors in one burst7 of every 8 fetchesChapter 18.2
batching status writebacksmost of the writebacksChapter 18.3
larger buffers, fewer scatter fragmentsthe per-fragment transactionsChapter 18.3
coalescing interruptsthe interrupt, not a bus transactionChapter 18.6
multi-queue with several mastersnone — it parallelises themChapter 18.7

Row one alone takes the 100 Gb/s figure from 446 M to 316 M transactions per second148.81 × (1/8 + 1 + 1)which is 1.26 per cycle at 250 MHz and still above one. Row two, batching the writebacks eight at a time as well, takes it to 186 M — 0.74 per cycle, which fits.

So the honest summary of Module 18 is this: the mechanisms exist because three transactions per frame does not fit, and they get it under one.


13. RTL 6 — The Datapath FIFOs and Their Watermarks

Section 9 built one crossing. A MAC has several FIFOs and the interesting part is the watermarks, because they are what turn a depth into a flow-control decision.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// mac_datapath_fifos -- occupancy accounting and watermarks for the
// receive and transmit paths.
//
// The depths are elsewhere (section 9). What lives here is the part
// integration gets wrong: WHEN to tell the far end to stop, and how
// much must remain free after telling it.
// -----------------------------------------------------------------------
module mac_datapath_fifos
  import macsoc_pkg::*;
#(
  parameter int RX_DEPTH_B  = 16384,     // octets
  parameter int TX_DEPTH_B  = 16384
)(
  input  logic               clk_host,
  input  logic               rst_host_n,

  input  logic [15:0]        rx_occupancy_b,   // synchronised in
  input  logic [15:0]        tx_occupancy_b,

  // 14.2's headroom, in octets, for this line rate. The integrator
  // supplies it because it depends on the CABLE, not on the MAC.
  input  logic [15:0]        cfg_headroom_b,

  input  logic               cfg_flow_control_en,

  output logic               xoff_request,
  output logic               rx_nearly_full,
  output logic               tx_starving,

  output logic [31:0]        c_xoff_asserted,
  output logic [31:0]        c_rx_high_water,
  output logic [31:0]        c_tx_underrun_risk,
  output logic               headroom_impossible
);

  // The watermark is the depth minus the headroom. If the headroom
  // exceeds the depth there is no watermark that works: by the time
  // the far end stops, the FIFO has already overflowed. This is a
  // CONFIGURATION fault and it is silent without this flag.
  wire [16:0] watermark = {1'b0, RX_DEPTH_B[15:0]} - {1'b0, cfg_headroom_b};
  assign headroom_impossible = (cfg_headroom_b >= RX_DEPTH_B[15:0]);

  assign rx_nearly_full = !headroom_impossible &&
                          ({1'b0, rx_occupancy_b} >= watermark);

  assign xoff_request   = cfg_flow_control_en && rx_nearly_full;

  // A transmit FIFO that empties mid-frame causes an underrun, which
  // corrupts a frame already on the wire -- 7.3's subject. The risk
  // threshold is one maximum frame's worth still unsent.
  assign tx_starving = (tx_occupancy_b < 16'd1518) && (tx_occupancy_b != '0);

  always_ff @(posedge clk_host or negedge rst_host_n) begin
    if (!rst_host_n) begin
      c_xoff_asserted <= '0; c_rx_high_water <= '0; c_tx_underrun_risk <= '0;
    end else begin
      if (xoff_request) c_xoff_asserted <= c_xoff_asserted + 1;
      if ({16'b0, rx_occupancy_b} > c_rx_high_water)
        c_rx_high_water <= {16'b0, rx_occupancy_b};
      if (tx_starving) c_tx_underrun_risk <= c_tx_underrun_risk + 1;
    end
  end

endmodule

Classification: watermark computation with an explicit impossibility check and a high-water recorder.

What it teaches: that the receive watermark is not a property of the FIFO, it is a property of the cable. Chapter 14.2 §8 derived the headroom: the octets that will still arrive after the decision to stop the partner is made, which is propagation delay plus the partner's dead time plus the pause frame's own transmission. A 2 km 1 Gb/s link needs about 1.62 KiB of headroom; a 2 m link needs almost none — so the same MAC in two deployments has two correct watermarks, and neither is discoverable from the MAC's own parameters.

And it teaches that headroom_impossible is a fault a design can detect and usually does not. A 100 Gb/s MAC needs 162.4 KiB of headroom over a long link. A 16 KiB receive FIFO cannot provide it at any watermark — the correct conclusion is that this MAC cannot do lossless flow control on this link, and the honest thing is to say so at configuration time rather than to assert XOFF at a watermark that cannot work and drop frames anyway.

Deliberately simplified: occupancy arrives already synchronised and as a flat octet count, when a real design tracks it in FIFO words and must convert. There is one watermark rather than the hysteresis pair a real design needs — a single threshold oscillates, asserting and deasserting XOFF on adjacent octets, and Chapter 14.2 §12's XON threshold is the missing half. Per-class occupancy is absent entirely, which Chapter 14.4 requires and which multiplies the whole structure by eight.

Production implication: c_rx_high_water is the counter that makes the FIFO's depth an evidence-based decision instead of a guess. It records the deepest the FIFO ever got, so a deployed system reports whether 16 KiB was generous or marginal — and a port whose high-water mark sits at 95% of depth is one memory-system hiccup from dropping frames, with no error counter to say so. The depth chosen at design time is a prediction; this counter is the measurement, and it is the only way the next revision's depth is better than this one's.


14. RTL 7 — Telemetry Across Four Domains

Every counter in this chapter lives in the domain that increments it, and the host has to read them all. This block is how they get there, and it is the answer to Section 6's arithmetic.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// soc_integration_telemetry -- one handshake, a whole counter set.
//
// Section 6: at 100 Gb/s, 4.46 frames arrive during a single
// synchroniser round trip. Crossing per counter per update is not
// possible; crossing the SET on request is.
// -----------------------------------------------------------------------
module soc_integration_telemetry
  import macsoc_pkg::*;
#(
  parameter int NUM_CNT_LOCAL = 8
)(
  // Source domain -- one instance per domain, parameterised.
  input  logic                clk_src,
  input  logic                rst_src_n,
  input  logic [31:0]         cnt_src [NUM_CNT_LOCAL],

  // Host domain.
  input  logic                clk_host,
  input  logic                rst_host_n,
  input  logic                req,                 // host asks for a snapshot
  output logic [31:0]         cnt_host [NUM_CNT_LOCAL],
  output logic                cnt_host_valid,

  output logic [31:0]         c_snapshots,
  output logic [31:0]         c_req_while_busy
);

  // ---- request crossing: host -> source ---------------------------
  logic req_tgl_host, req_tgl_s1, req_tgl_s2, req_tgl_s3;
  logic busy_host;

  always_ff @(posedge clk_host or negedge rst_host_n) begin
    if (!rst_host_n) begin
      req_tgl_host <= 1'b0; busy_host <= 1'b0; c_req_while_busy <= '0;
    end else begin
      if (req && !busy_host) begin
        req_tgl_host <= ~req_tgl_host;
        busy_host    <= 1'b1;
      end else if (req && busy_host) begin
        c_req_while_busy <= c_req_while_busy + 1;
      end
    end
  end

  always_ff @(posedge clk_src or negedge rst_src_n) begin
    if (!rst_src_n) begin
      req_tgl_s1 <= 1'b0; req_tgl_s2 <= 1'b0; req_tgl_s3 <= 1'b0;
    end else begin
      req_tgl_s1 <= req_tgl_host;
      req_tgl_s2 <= req_tgl_s1;
      req_tgl_s3 <= req_tgl_s2;
    end
  end

  wire req_pulse_src = req_tgl_s2 ^ req_tgl_s3;

  // ---- the snapshot ----------------------------------------------
  // Taken in ONE source-clock cycle. Every counter in the set is
  // therefore consistent with every other, which is the entire
  // reason this block exists rather than a synchroniser per counter.
  logic [31:0] snap [NUM_CNT_LOCAL];
  logic        ack_tgl_src;

  always_ff @(posedge clk_src or negedge rst_src_n) begin
    int i;
    if (!rst_src_n) begin
      for (i = 0; i < NUM_CNT_LOCAL; i++) snap[i] <= '0;
      ack_tgl_src <= 1'b0;
    end else if (req_pulse_src) begin
      for (i = 0; i < NUM_CNT_LOCAL; i++) snap[i] <= cnt_src[i];
      ack_tgl_src <= ~ack_tgl_src;
    end
  end

  // ---- acknowledge crossing: source -> host -----------------------
  logic ack_tgl_h1, ack_tgl_h2, ack_tgl_h3;

  always_ff @(posedge clk_host or negedge rst_host_n) begin
    int i;
    if (!rst_host_n) begin
      ack_tgl_h1 <= 1'b0; ack_tgl_h2 <= 1'b0; ack_tgl_h3 <= 1'b0;
      cnt_host_valid <= 1'b0; c_snapshots <= '0;
      for (i = 0; i < NUM_CNT_LOCAL; i++) cnt_host[i] <= '0;
    end else begin
      ack_tgl_h1 <= ack_tgl_src;
      ack_tgl_h2 <= ack_tgl_h1;
      ack_tgl_h3 <= ack_tgl_h2;
      cnt_host_valid <= 1'b0;

      // The snapshot registers are stable by now -- they were written
      // before ack_tgl_src toggled, and that toggle took three host
      // clocks to arrive. No synchroniser on the DATA is needed or
      // wanted; the handshake is what makes it safe.
      if (ack_tgl_h2 ^ ack_tgl_h3) begin
        for (i = 0; i < NUM_CNT_LOCAL; i++) cnt_host[i] <= snap[i];
        cnt_host_valid <= 1'b1;
        busy_host      <= 1'b0;
        c_snapshots    <= c_snapshots + 1;
      end
    end
  end

endmodule

Classification: a four-phase toggle handshake carrying a consistent multi-word snapshot, with no synchroniser on the data path.

What it teaches: that the data does not get synchronised and must not be. A synchroniser on each of eight 32-bit counters is 256 flops and it is still wrong, because each counter would cross independently and the host would see eight values captured at eight different instants. The handshake makes one instant: the snapshot is taken in a single source-clock cycle, and the acknowledge's three-flop journey guarantees the snapshot registers are stable long before the host reads them. The data path is combinational and safe because the control path is slow.

And it teaches why a toggle rather than a pulse crosses the request. A single-cycle pulse in a 250 MHz host domain is invisible to a 125 MHz receive domain — the sampling clock may simply miss it. A toggle is a level change and cannot be missed, and the edge detector on the far side reconstructs the pulse. This is the standard construction and it is standard because the obvious alternative silently loses requests, at a rate that depends on the frequency ratio and therefore varies between deployments.

Deliberately simplified: one instance per domain means the register file sees several snapshot interfaces and must sequence them, and the sequencing is absent. The counters are 32 bits and wrap, with no sticky overflow flag — so a host that reads infrequently cannot tell one wrap from two. And busy_host is assigned in two always blocks in this listing, which is a deliberate simplification for readability and would not elaborate cleanly; a real design tracks it in one.

Production implication: c_req_while_busy is the counter that catches a monitoring agent polling faster than the handshake completes. A snapshot from a 125 MHz domain takes roughly 24 ns of crossing plus the request's own journey, so a host polling every 20 ns gets fewer snapshots than it asks for — and without this counter it silently reads stale values and computes rates that are wrong by whatever fraction it missed. The failure is a plausible number rather than an error, which is the hardest kind to notice.


15. RTL 8 — The Integration Conformance Monitor

The last block asserts the properties that are checkable on one side of the boundary, and is explicit about which ones are not.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// macsoc_conformance_monitor -- what an integrator can check from the
// host side alone.
//
// Every property here has operands sampled in ONE clock domain. The
// ones that would relate a host-domain signal to a receive-domain
// one are absent by design, and section 19 explains why they are not
// merely hard but unevaluable.
// -----------------------------------------------------------------------
module macsoc_conformance_monitor
  import macsoc_pkg::*;
(
  input  logic        clk_host,
  input  logic        rst_host_n,

  input  ctrl_t       cfg_ctrl,
  input  logic        cfg_write_while_enabled,
  input  logic        headroom_impossible,
  input  logic        coalesce_misconfigured,

  input  logic [31:0] c_overflow,
  input  logic [31:0] c_stall_outstanding,
  input  logic [31:0] c_bus_error,
  input  logic [31:0] c_split_4k,
  input  logic [31:0] c_txn_desc,
  input  logic [31:0] c_txn_data,
  input  logic [31:0] c_irq_by_timer,
  input  logic [31:0] c_irq_by_count,
  input  logic [31:0] c_req_while_busy,
  input  logic [31:0] c_sof_host,        // snapshotted, section 14
  input  logic [31:0] c_eof_host,        // snapshotted in the SAME snapshot

  output logic        integration_ok,
  output logic        cfg_fault,
  output logic        memory_system_marginal,
  output logic        interrupt_mistuned,
  output logic        phy_truncating,
  output logic        alignment_poor,
  output logic        none_of_the_above
);

  // Configuration faults: checkable immediately, always fatal.
  assign cfg_fault = cfg_write_while_enabled |
                     headroom_impossible     |
                     coalesce_misconfigured;

  // The memory system is marginal if the MAC is stalling on its own
  // outstanding limit or, worse, has overflowed. Overflow is the
  // symptom that presents as a CRC error -- section 9.
  assign memory_system_marginal = (c_overflow != '0) ||
                                  (c_stall_outstanding > 32'd1000) ||
                                  (c_bus_error != '0);

  // Almost all interrupts by timer means the count threshold is never
  // reached: every frame is waiting the full timer for no reason.
  assign interrupt_mistuned = (c_irq_by_timer > (c_irq_by_count << 3)) &&
                              (c_irq_by_timer > 32'd100);

  // c_sof and c_eof come from the SAME snapshot, so this comparison
  // is legitimate. Comparing them across two snapshots would not be.
  assign phy_truncating = (c_sof_host > c_eof_host + 32'd1);

  // A 4 KiB split on a meaningful fraction of data transactions means
  // the buffer pool is not page-aligned -- 18.3's subject.
  assign alignment_poor = (c_txn_data != '0) &&
                          (c_split_4k > (c_txn_data >> 4));

  assign integration_ok = !cfg_fault && !memory_system_marginal &&
                          !phy_truncating;

  // 14.3's habit: an explicit "nothing here is wrong" is what ends an
  // investigation rather than deferring it.
  assign none_of_the_above = integration_ok && !interrupt_mistuned &&
                             !alignment_poor && (c_req_while_busy == '0);

  // ---- properties -------------------------------------------------
  // Every one of these samples only host-domain signals.

  p_cfg_fault_is_sticky_until_fixed:
    assert property (@(posedge clk_host) disable iff (!rst_host_n)
      headroom_impossible |-> ##1 (headroom_impossible ||
                                   $changed(cfg_ctrl)))
    else $error("headroom fault cleared without a configuration change");

  p_overflow_implies_not_ok:
    assert property (@(posedge clk_host) disable iff (!rst_host_n)
      (c_overflow != '0) |-> !integration_ok)
    else $error("receive overflow reported while integration_ok is high");

  p_exactly_one_verdict_or_none:
    assert property (@(posedge clk_host) disable iff (!rst_host_n)
      none_of_the_above |-> (!cfg_fault && !memory_system_marginal &&
                             !interrupt_mistuned && !phy_truncating &&
                             !alignment_poor))
    else $error("none_of_the_above asserted alongside a finding");

  p_desc_txn_never_exceeds_data_txn_by_much:
    assert property (@(posedge clk_host) disable iff (!rst_host_n)
      (c_txn_data > 32'd1000) |-> (c_txn_desc <= (c_txn_data << 1)))
    else $error("descriptor fetches far exceed data writes -- ring starvation");

endmodule

Classification: a host-domain verdict generator whose entire design constraint is that it may only look at host-domain signals.

What it teaches: that the monitor's most important property is which signals it refuses to take. There is no input from the receive clock domain in the port list. c_sof_host and c_eof_host are snapshotted values from Section 14's handshake, explicitly noted as coming from the same snapshot — because comparing two counters captured at two different instants is comparing two different moments in the traffic, and at 100 Gb/s two snapshots 30 ns apart are four and a half frames apart.

And it teaches that the verdicts partition into two kinds with completely different urgency. cfg_fault is a configuration error that was wrong the moment it was written and is fixable in a register write. memory_system_marginal is a system property that no register write fixes — it needs a different arbiter, a deeper FIFO, or less competing traffic. A monitor that reports them as one bit sends an integrator to the wrong team.

Deliberately simplified: the thresholds are literals — 1000 stalls, a factor of 8 on the interrupt ratio, one sixteenth on the split ratio — where a production monitor takes them from registers, because what counts as marginal depends on the deployment. The counters are compared as absolute values rather than as rates, so a long-running system eventually trips every threshold; a real monitor works on deltas between snapshots. p_cfg_fault_is_sticky_until_fixed also assumes cfg_ctrl changing implies the headroom was reconfigured, which is looser than it should be.

Production implication: none_of_the_above is the signal that ends an investigation. Chapter 14.3 §15 made the argument in a different context and it applies unchanged here: on a device where several distinct conditions produce the symptom "the Ethernet is slow", an explicit statement that none of them is present is as informative as any of them — and it is the only one that lets an integrator stop looking at the MAC and start looking at the application.


16. Reset, and Why Four Domains Do Not Share One

Reset is the part of integration that gets designed last and debugged first, and a four-domain block has four of them whether the design admits it or not.

The rule is short and the reason is not obvious: a reset may be asserted asynchronously and must be released synchronously to the domain it resets.

Asserting asynchronously is necessary because the clock may not be running — and in this MAC's receive domain the clock is recovered from the wire, so at power-on there is no clock at all. A reset that needs a clock edge to take effect never takes effect.

Releasing synchronously is necessary because a reset released asynchronously may release close to a clock edge, and different flops in the domain will then see different reset states on the same edge. The domain comes out of reset in a mixture, which for a state machine means an illegal state and for a FIFO pointer means a pointer that is not zero.

DomainClock at power-onReset sourceRelease
hostrunningthe SoC's reset controllersynchronised to host clock
receiveabsent until the partner transmitshost reset, plus clock-losssynchronised to the recovered clock
transmitrunning from the local referencehost resetsynchronised to transmit clock
PTPrunninghost resetsynchronised to PTP clock

And the crossings make the order matter, which is the part that is genuinely subtle.

Consider Section 9's asynchronous FIFO with the two sides reset independently. If the write side is released first, it writes into a FIFO whose read pointer is still held at zero — which is harmless. If the read side is released first, it reads from a FIFO whose write pointer is held at zero and whose r_empty is therefore asserted — also harmless. Neither order breaks it, because both flags are conservative.

Now consider the receive domain's reset being released while the host domain is still held. Frames arrive, the FIFO fills, and w_full asserts against a read pointer that will never move until the host comes out of reset. Nothing is corrupted and every frame after the FIFO fills is lost — which is the correct behaviour and looks exactly like a bug.

The rule that falls out: release the domains in the order host, transmit, receive, and hold the MAC's enable bit clear until all four are out. cfg_ctrl.enable is the real release and the resets are only the preconditions for it — which is why Section 3's register file resets ctrl_q to zero and nothing in the MAC does anything until software writes it.

One more case, and it is specific to this block. The receive clock stops when the cable is removed. Every flop in the receive domain freezes at whatever value it held, including the FIFO's write pointer and Gray code. When the cable is replugged, the clock restarts and those flops resume from a state that was valid — so the FIFO is not corrupted, but it may hold a partial frame from before the unplug, which will be reassembled onto the front of the next one. The receive domain must therefore be reset on clock return, not merely allowed to continue — and c_clock_lost from Section 7 is what triggers it.


A 100 gigabit MAC receiving minimum-size frames can be judged by bandwidth or by transaction rate, and the two measurements disagree. By bandwidth it needs 114.3 gigabits per second, which a 512-bit bus at 250 megahertz supplies at 89 percent utilisation — high but ordinary, and solvable by widening the bus. By transaction rate it offers three transactions per frame at 148.81 million frames per second, which is 446 million transactions per second, or 1.79 per cycle at 250 megahertz, and a single AXI address channel issues at most one per cycle. Widening the bus does not help because a 16-octet descriptor fetch is one transaction at any width. The fix is fewer transactions per frame: fetching eight descriptors in one burst takes the figure to 316 million transactions per second or 1.26 per cycle, and batching the status writebacks eight at a time takes it to 186 million or 0.74 per cycle, which fits. This is why every remaining chapter of Module 18 is about transaction count rather than bandwidth.100 Gb/s, 64-octetframes148.81 MfpsBy bandwidth114.3 Gb/s needed512 bits at 250 MHz89% — comfortableBy transaction rate3 per frame446 M txn/s1.79 per cycle —impossibleA wider buschanges nothing hereBatch 8 descriptors316 M — 1.26/cycleBatch the writebacks186 M — 0.74/cycle, fits12
Figure 4 — the same workload measured two ways, giving opposite verdicts about whether 100 Gb/s is feasible.

17. What the MAC Assumes About the System It Lands In

A MAC's datasheet specifies what the MAC does. Integration failures come from what it assumes, and assumptions are not usually written down. Here are this chapter's, stated as a checklist.

#The assumptionIf it is falseEstablished in
1the memory system supplies 1.143× the line rate, both directionsframes drop under loadSection 4
2the bus accepts the transaction rate, not just the bandwidththe port runs below line rateSection 12
3worst-case memory latency fits in the receive FIFOoverflow, reported as CRC errorsSections 8, 9
4descriptor buffers are page-alignedevery burst splits at 4 KiBSection 5
5the CPU can service the interrupt ratethe system livelocksSection 10
6software never rewrites a ring base while enableda frame lands in unrelated memorySection 3
7the four resets are released in orderan illegal state at bring-upSection 16
8the flow-control headroom fits in the FIFOlossless is not achievableSection 13

Rows 1, 2, 3 and 5 are the ones that cause the failures nobody can localise, because all four present as the same symptom — "the Ethernet is slow" or "we see CRC errors" — and none of them is an Ethernet problem.

And here is the asymmetry that makes them hard: the MAC can detect every one of them and cannot fix any of them.

AssumptionDetectable byFixable by
bandwidthc_stall_outstandingthe system's arbiter
transaction ratec_txn_* ratiosChapter 18.2's batching
latency vs FIFOc_overflow, c_rx_high_watera deeper FIFO — a silicon change
alignmentc_split_4kthe driver's allocator
interrupt ratec_irq_raisedChapter 18.6's coalescing

The right column names four different owners and none of them is the MAC team, which is the single most useful thing this chapter has to say about integration. The MAC's job is to make each assumption's violation visible and attributable; everything after that belongs to somebody else, and the counters are what get it to them.

The checklist has a practical form worth stating: before integrating a MAC, write down the eight numbers on the right-hand side of the system's answer to the eight assumptions. Most projects can answer three of them. The other five are discovered during bring-up, at a cost measured in weeks.


18. The Integration Cost, Accounted

Put the eight blocks' costs side by side, because the distribution is not where the effort goes.

BlockApproximate costWhere it lands
mac_register_file~1200 flopslogic
mac_memory_master~800 flops + address arithmeticlogic
phy_boundary_adapter~400 flopslogic
async_fifo_cdc~120 flops + the SRAMmemory
mac_interrupt_iface~200 flopslogic
mac_datapath_fifos~200 flopslogic
soc_integration_telemetry~300 flops per domain instancelogic
macsoc_conformance_monitor~100 flopslogic, and usually not synthesised

The logic totals roughly 3300 flops, which on a modern SoC is nothing. The memory does not:

Line rateReceive FIFO for a 2 µs stallPlus 14.2's headroom, 2 kmTotal SRAM
1 Gb/s0.24 KiB1.62 KiB~2 KiB
10 Gb/s2.44 KiB16.2 KiB~19 KiB
25 Gb/s6.10 KiB40.6 KiB~47 KiB
100 Gb/s24.41 KiB162.4 KiB~187 KiB

The bottom row is the chapter's last surprise. A 100 Gb/s MAC that wants lossless operation over a 2 km link needs about 187 KiB of on-chip SRAM for its receive FIFO aloneand the flow-control headroom is 87% of it. The stall buffer, which is what everybody sizes, is the small part.

And a transmit FIFO roughly doubles it, though with a different sizing argument: the transmit side must hold enough to survive a memory stall without underrunning mid-frame, which Chapter 7.3 established corrupts a frame already on the wire.

Against the system:

SizeOf a 4 MiB on-chip SRAM budget
1 Gb/s MAC, both directions~4 KiB0.10%
10 Gb/s~38 KiB0.93%
100 Gb/s~374 KiB9.1%

9.1% of a chip's SRAM for one port's FIFOs is a real number and it is the one that gets negotiated, usually by reducing the headroom and accepting that the link is not lossless — which is a network-architecture decision being made by an SRAM budget, and it is made that way more often than anybody admits.


19. Properties Worth Asserting, and One Worth Refusing

Every property below samples its operands in a single clock domain. That constraint is not a stylistic preference and this section ends by explaining why.

Register-file properties — host domain.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// A configuration write to a ring base while the MAC is enabled must
// be refused, not applied. Applying it points the DMA at a new base
// mid-descriptor.
p_ring_base_write_refused_when_enabled:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    (psel && penable && pwrite && cfg_ctrl.enable &&
     (paddr == REG_RX_RING_LO)) |=> $stable(cfg_rx_ring_base))
  else $error("ring base changed while the MAC was enabled");

// A refused write must be reported, or a driver bug becomes an
// unexplained absence of traffic.
p_refused_write_flags:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    (psel && penable && pwrite && cfg_ctrl.enable &&
     (paddr inside {REG_RX_RING_LO, REG_RX_RING_HI,
                    REG_TX_RING_LO, REG_TX_RING_HI}))
      |-> cfg_write_while_enabled)
  else $error("a ring-base write was refused silently");

// Reads never have side effects. A counter that clears on read cannot
// be sampled by two agents.
p_read_is_side_effect_free:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    (psel && penable && !pwrite) |=> $stable(cfg_ctrl))
  else $error("a register read changed configuration state");

// An unmapped read errors rather than returning zero, because zero is
// a plausible value for every register in the map.
p_unmapped_read_errors:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    (psel && penable && !pwrite && (paddr == 12'hFFC)) |-> pslverr)
  else $error("an unmapped read returned data instead of an error");

// The counter shadow only updates when its owner says the value is
// coherent.
p_counter_shadow_needs_valid:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    !cnt_valid[0] |=> $stable(cnt_shadow[0]))
  else $error("a counter shadow updated without its valid");

Memory-master properties — host domain.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// A burst may not cross a 4 KiB boundary. This is the bus's rule and
// the MAC has no natural reason to respect it.
p_burst_within_4k:
  assert property (@(posedge clk) disable iff (!rst_n)
    (ax_valid && ax_ready) |->
      ((ax_addr[11:0] + ((ax_len + 1) << ax_size)) <= 13'h1000))
  else $error("a burst crossed a 4 KiB boundary");

// Outstanding transactions never exceed the limit the system agreed.
p_outstanding_bounded:
  assert property (@(posedge clk) disable iff (!rst_n)
    outstanding <= MAX_OUTSTAND)
  else $error("outstanding transactions exceeded the agreed limit");

// A completion with no transaction outstanding is a bus-level fault
// or a tracking bug, and both are fatal.
p_no_orphan_completion:
  assert property (@(posedge clk) disable iff (!rst_n)
    done_valid |-> (outstanding != '0))
  else $error("a completion arrived with nothing outstanding");

// The burst length never exceeds the parameterised maximum.
p_len_bounded:
  assert property (@(posedge clk) disable iff (!rst_n)
    (ax_valid && ax_ready) |-> (ax_len < MAX_BEATS))
  else $error("burst length exceeded MAX_BEATS");

// A request is not accepted while one is in progress.
p_no_overlapping_requests:
  assert property (@(posedge clk) disable iff (!rst_n)
    busy |-> !req_ready)
  else $error("a request was accepted while another was in progress");

// A descriptor fetch is a read; data and status are writes.
p_kind_implies_direction:
  assert property (@(posedge clk) disable iff (!rst_n)
    (ax_valid && (cur_kind == 2'd0)) |-> !ax_write)
  else $error("a descriptor fetch was issued as a write");

// Every issued transaction eventually completes. Without a bound the
// MAC's outstanding counter leaks and the port stops.
p_transaction_completes:
  assert property (@(posedge clk) disable iff (!rst_n)
    (ax_valid && ax_ready) |-> ##[1:$] done_valid)
  else $error("an issued transaction never completed");

// A remaining byte count only decreases.
p_bytes_monotonic:
  assert property (@(posedge clk) disable iff (!rst_n)
    busy |=> (cur_bytes <= $past(cur_bytes)))
  else $error("the remaining byte count increased");

PHY-boundary properties — receive domain, all operands local.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// A start character inside a frame is a framing error, not a new frame.
p_no_nested_start:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    (in_frame && |is_start) |-> s_error)
  else $error("a start character mid-frame was treated as a new frame");

// Every frame that starts eventually terminates or errors.
p_frame_terminates:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    s_sof |-> ##[1:$] (s_eof || s_error))
  else $error("a frame started and never terminated");

// The last beat's byte count is within the bus width.
p_eof_bytes_legal:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    s_eof |-> (s_bytes <= INT_BYTES))
  else $error("the final beat claimed more bytes than the bus carries");

// A start of frame is never asserted twice without an end between.
p_sof_eof_alternate:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    s_sof |=> (!s_sof throughout (s_eof || s_error)[->1]))
  else $error("two starts without an intervening end");

// Data is only valid inside a frame.
p_valid_only_in_frame:
  assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
    s_valid |-> (in_frame || s_sof))
  else $error("valid data outside a frame");

Asynchronous-FIFO properties — each in its own domain, and note that there is no property relating the two.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Write domain: no write when full.
p_no_write_when_full:
  assert property (@(posedge clk_w) disable iff (!rst_w_n)
    (w_en && w_full) |-> ##1 (c_overflow > $past(c_overflow)))
  else $error("a write into a full FIFO was not counted as an overflow");

// Write domain: the Gray pointer changes by at most one bit.
p_write_gray_single_bit:
  assert property (@(posedge clk_w) disable iff (!rst_w_n)
    ##1 ($countones(w_gray ^ $past(w_gray)) <= 1))
  else $error("the write Gray pointer changed more than one bit");

// Read domain: the same, on the read side.
p_read_gray_single_bit:
  assert property (@(posedge clk_r) disable iff (!rst_r_n)
    ##1 ($countones(r_gray ^ $past(r_gray)) <= 1))
  else $error("the read Gray pointer changed more than one bit");

// Read domain: no read when empty.
p_no_read_when_empty:
  assert property (@(posedge clk_r) disable iff (!rst_r_n)
    (r_en && r_empty) |-> ##1 (c_underflow > $past(c_underflow)))
  else $error("a read from an empty FIFO was not counted");

// Write domain: the synchronised read pointer only ever advances.
p_sync_read_ptr_monotonic:
  assert property (@(posedge clk_w) disable iff (!rst_w_n)
    ##1 (r_gray_w2 == $past(r_gray_w2) ||
         $countones(r_gray_w2 ^ $past(r_gray_w2)) == 1))
  else $error("the synchronised read pointer jumped");

Interrupt properties — host domain.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// An urgent event is never coalesced.
p_urgent_not_coalesced:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    |evt_urgent |=> irq)
  else $error("an error event was delayed by coalescing");

// The interrupt is never asserted with an empty status word.
p_irq_implies_status:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    irq |-> (irq_status != '0))
  else $error("the interrupt was asserted with no status bit set");

// An acknowledge clears the interrupt.
p_ack_clears:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    irq_ack |=> !irq)
  else $error("the interrupt survived its acknowledge");

// A masked event never raises the interrupt.
p_mask_is_respected:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    ((evt & ~irq_enable[NUM_EVENTS-1:0]) != '0 &&
     (evt & irq_enable[NUM_EVENTS-1:0]) == '0 && !irq) |=> !irq)
  else $error("a masked event raised the interrupt");

// Pending events with a configured timer are bounded in time.
p_pending_bounded_by_timer:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    ((pending_count != '0) && (cfg_timer_us != '0))
      |-> ##[1:$] (irq || (pending_count == '0)))
  else $error("pending frames were held with a timer configured");

// A count threshold with no timer is flagged.
p_no_timer_is_flagged:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    ((cfg_count > 16'd1) && (cfg_timer_us == '0))
      |-> coalesce_misconfigured)
  else $error("a stalling coalescing configuration was not flagged");

Watermark properties — host domain.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// XOFF is only requested when flow control is enabled.
p_xoff_requires_enable:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    xoff_request |-> cfg_flow_control_en)
  else $error("XOFF requested with flow control disabled");

// An impossible headroom suppresses the watermark rather than
// producing one that cannot work.
p_impossible_headroom_suppresses:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    headroom_impossible |-> !rx_nearly_full)
  else $error("a watermark was computed from an impossible headroom");

// The high-water mark never decreases.
p_high_water_monotonic:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    ##1 (c_rx_high_water >= $past(c_rx_high_water)))
  else $error("the high-water mark decreased");

Telemetry properties — each side in its own domain.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// Host domain: a snapshot is only requested when not busy.
p_no_request_while_busy:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    (req && busy_host) |=> (c_req_while_busy > $past(c_req_while_busy)))
  else $error("a request while busy was not counted");

// Host domain: valid is a single cycle.
p_snapshot_valid_pulses:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    cnt_host_valid |=> !cnt_host_valid)
  else $error("the snapshot valid was held for more than a cycle");

// Source domain: the snapshot is written in one cycle, so every
// element changes together or none does.
p_snapshot_is_atomic:
  assert property (@(posedge clk_src) disable iff (!rst_src_n)
    (!req_pulse_src) |=> $stable(snap[0]) && $stable(snap[1]))
  else $error("snapshot elements changed outside a request");

// Host domain: a request is eventually answered.
p_request_completes:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    (req && !busy_host) |-> ##[1:$] cnt_host_valid)
  else $error("a snapshot request was never answered");

Monitor properties — host domain.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// A configuration fault and an all-clear verdict cannot coexist.
p_fault_excludes_ok:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    cfg_fault |-> !integration_ok)
  else $error("integration_ok asserted alongside a configuration fault");

// The truncation verdict uses two counters from ONE snapshot.
p_truncation_uses_one_snapshot:
  assert property (@(posedge clk_host) disable iff (!rst_host_n)
    phy_truncating |-> $past(cnt_host_valid, 1))
  else $error("a truncation verdict was computed from unsynchronised counts");

20. Verification Scenarios

Fifty-eight scenarios, grouped by which of Section 17's assumptions each one tests. The last group is the directed test, and it is here because random stimulus will not produce it.

Register interface — 8 scenarios.

#ScenarioExpected
1write every register, read backvalues match
2read an unmapped addresspslverr, no data
3write a ring base with enable clearaccepted
4write a ring base with enable setrefused, cfg_write_while_enabled
5write REG_IRQ_STATUSirq_ack pulses, status clears
6read a counter shadow with no cnt_validprevious value, unchanged
7back-to-back writes to adjacent registersboth land
8write REG_CTRL with max_frame above 9000accepted; the MTU is the integrator's

Memory master — 10 scenarios.

#ScenarioExpected
964-octet write, alignedone burst, 2 beats at 256 bits
101518-octet write starting 3 KiB into a pagetwo bursts, c_split_4k increments
111518-octet write, page-alignedone or more bursts, c_split_4k flat
1216-octet descriptor fetchone beat, read direction
13a request arriving while busyreq_ready low, request held
14MAX_OUTSTAND reachedax_valid low, c_stall_outstanding rises
15completion with done_errorc_bus_error increments
16a write ending exactly on a 4 KiB boundaryone burst, no split
17a write of zero bytesno transaction issued
18issue and completion in the same cycleoutstanding unchanged

PHY boundary — 9 scenarios.

#ScenarioExpected
19a minimum frame, start to terminates_sof, beats, s_eof, c_sof == c_eof
20a start character while in_frames_error, not a new frame
21a terminate character in lane 0s_bytes is zero for that beat
22a terminate in the last lanes_bytes is the full width
23an error character mid-frames_error, c_local_fault rises
24rx_clk_present drops mid-framein_frame clears, c_clock_lost rises
25rx_clk_present returnsthe next start is clean
26idle characters onlyno s_valid
27back-to-back frames at minimum IFGboth delivered, counts equal

Clock crossing — 9 scenarios.

#ScenarioExpected
28write clock 10× the read clockw_full asserts, no data lost
29read clock 10× the write clockr_empty asserts often, no underflow
30write into a full FIFOc_overflow rises, write discarded
31read from an empty FIFOc_underflow rises
32Gray pointer across a wrapone bit changes at DEPTH-1 to 0
33both resets released simultaneouslyboth pointers zero, r_empty high
34write reset released firstwrites accepted, reads blocked
35read reset released firstr_empty high, no spurious read
36the write clock stops mid-framepointers freeze, no corruption

Interrupts — 8 scenarios.

#ScenarioExpected
37count threshold 1, one frameimmediate interrupt
38count 32, 31 frames, timer 100 µsinterrupt at 100 µs, c_irq_by_timer
39count 32, 32 framesimmediate, c_irq_by_count
40count 32, timer 0, 31 framesno interrupt; coalesce_misconfigured set
41an error event during coalescingimmediate interrupt, not delayed
42acknowledge with an event in the same cyclethe event is not lost
43all events maskedno interrupt
44timer expires with zero pendingno interrupt

Watermarks and telemetry — 8 scenarios.

#ScenarioExpected
45occupancy below watermarkxoff_request low
46occupancy crosses the watermarkxoff_request high, counter rises
47headroom exceeds depthheadroom_impossible, no watermark
48flow control disabled at the watermarkrx_nearly_full high, xoff_request low
49transmit FIFO below one maximum frametx_starving
50a snapshot request at idlecnt_host_valid after the round trip
51a second request before the first completesc_req_while_busy rises, first completes
52source counters change during the crossingthe host sees the snapshot's values

The directed test — 6 runs random stimulus will not produce.

Random stimulus explores frame sizes, arrival times and bus delays independently. What it does not produce is the correlation that makes an integration failure: a memory stall of a specific duration arriving during a burst of minimum-size frames, repeated often enough to matter. The reason is combinatorial — the stall must be long enough to matter and short enough not to be an obvious fault, and it must land during a minimum-frame burst rather than between bursts — and the probability of a random generator producing that conjunction is small enough that a long regression will not see it.

Construct it directly. Five runs, one variable.

RunMemory stallTrafficReceive FIFOExpected outcome
A0.5 µs64-octet burst at line rate16 KiB at 10 Gb/soccupancy peaks at 0.61 KiB — fine
B2 µssamesame2.44 KiB — fine, c_rx_high_water rises
C5 µssamesame6.10 KiB — fine, high-water at 38%
D13 µssamesame15.9 KiB — at the edge, no overflow
E14 µssamesameoverflow; CRC errors downstream
F14 µs1518-octet framessameoverflow at the same instant

Run F is the one that makes the test worth constructing, and its result is the opposite of the intuition. The frame size does not change when the overflow happens — the FIFO fills at the line rate regardless of how that rate is divided into frames — so a design tested only with large frames and no stall passes, and the same design tested with large frames and a 14 µs stall fails identically to the small-frame case.

Which makes the stall, not the frame size, the variable that matters for the FIFO — and the frame size the variable that matters for Sections 10 and 12. The two failure modes have disjoint causes and a regression that varies only one of them finds only one.

The oracle for the six runs, checked in four parts:

CheckRun A–DRun E–F
c_overflowzeronon-zero
c_rx_high_waterbelow depthequals depth
downstream FCS errorszeroone per overflowed frame
memory_system_marginallowhigh

Row three is the part an integration test usually gets wrong. The overflow's symptom is a CRC error at the next stage — so a testbench checking only for CRC errors sees runs E and F fail and concludes the receive path corrupts frames. Checking c_overflow alongside is what turns "the MAC is broken" into "the memory system stalled for 14 µs", which is Section 9's production implication demonstrated rather than asserted.

And row four closes the loop to Section 15. The monitor's verdict must agree with the counters; a run where c_overflow is non-zero and memory_system_marginal is low is a monitor bug, not a MAC bug, and the directed test is the only place that pairing is exercised deliberately.


21. Debugging an Integration, in the Order the Evidence Arrives

Every failure in this chapter presents as one of three complaints. Start from the complaint and the counters narrow it in about ten minutes.

Complaint 1 — "we see CRC errors."

CheckIf yesIf no
c_overflow non-zero?a memory stall — go to complaint 2continue
c_sof exceeds c_eof?the PHY is truncating — serdes or cablecontinue
c_local_fault rising?the PHY is reporting errors — layer 1continue
errors on both directions?the cableone direction — that transceiver

Row one is the finding this chapter exists to make possible. Without c_overflow, a CRC error rate caused by a memory stall is indistinguishable from one caused by a marginal optic — and the diagnostic instinct built up over thirty years says optic. The cable gets replaced, the errors persist, and the investigation restarts.

Complaint 2 — "the port does not reach line rate."

CheckIf yesMeaning
c_stall_outstanding rising with load?the memory system is the limitraise MAX_OUTSTAND, or fix the arbiter
c_split_4k above a sixteenth of c_txn_data?the buffer pool is not page-alignedthe driver's allocator
c_txn_desc comparable to c_txn_data?no descriptor batchingChapter 18.2
transaction rate near one per cycle?Section 12's wallbatching, then multi-queue
all four flat, rate still low?the offered load is what it isnone_of_the_above

Complaint 3 — "the system is unresponsive under network load."

CheckIf yesMeaning
c_irq_raised near the frame rate?no coalescing in effectSection 10's arithmetic
c_irq_by_timer far above c_irq_by_count?the count threshold is never reachedmistuned, not absent
coalesce_misconfigured set?a count with no timerframes held indefinitely
both low and the system still stalls?not the MAClook at the driver's receive loop

And the two symptoms most likely to be misattributed, stated plainly because both send an investigation to the wrong team:

SymptomInstinctThis chapter's cause
CRC errorsthe cablea memory stall overflowing the receive FIFO
packets arriving seconds late, sometimesthe networka coalescing count with no timer

Both are diagnosed by one register read and both take days without it, which is the argument for every counter in this chapter in one table.


22. Misconceptions

Misconception 1 — "a MAC needs memory bandwidth equal to its line rate."

The wrong model: a 1 Gb/s port moves 1 Gb/s of data, so it needs 1 Gb/s of memory bandwidth.

What it costs: a memory allocation that is 14.3% short at minimum frame size, which shows up as a port that reaches line rate with large frames and falls short with small ones — and the shortfall is then attributed to the traffic rather than to the allocation.

The corrected model: the MAC moves the frame plus a descriptor fetch plus a status writeback, 32 octets of bookkeeping per frame. At 64-octet frames the memory requirement is 1.143× the line rate; at 1518-octet frames it is 1.008×. Both directions at once doubles it. Section 4.

Misconception 2 — "a wider bus solves it."

The wrong model: the port is limited by bandwidth, so doubling the bus width doubles the headroom.

What it costs: a design iteration that widens the bus from 256 to 512 bits, doubles the area of every datapath register, and produces no measurable improvement — because the limit was transactions, not bytes.

The corrected model: a descriptor fetch is one transaction whether the bus is 64 bits or 512. At 100 Gb/s with minimum frames the MAC offers 446 M transactions per second — 1.79 per cycle at 250 MHz — and one address channel issues one. Widening the bus changes the bandwidth column and leaves the transaction column untouched. Section 12.

Misconception 3 — "the receive FIFO needs to hold a frame."

The wrong model: the FIFO exists to buffer a frame while it is being written to memory, so one maximum frame is enough.

What it costs: a 2 KiB FIFO on a 10 Gb/s port, which overflows on any memory stall longer than 1.6 µs — and the overflow presents as a CRC error, so the FIFO is not suspected.

The corrected model: the FIFO's depth is set by the worst-case time before the memory system accepts data, multiplied by the line rate. At 10 Gb/s a 5 µs stall needs 6.10 KiB, and a lossless configuration needs Chapter 14.2's headroom on top — 16.2 KiB more at 10 Gb/s over 2 km. Sections 8 and 18.

Misconception 4 — "the interrupt is a wire, so it is free."

The wrong model: an interrupt output costs one flop and no bandwidth, so it is the cheapest interface on the block.

What it costs: a system that livelocks the first time it meets a burst of minimum-size frames. At 1 Gb/s that is 1.488 M interrupts per second and 297.6% of one CPU — on the slowest rate anybody still builds.

The corrected model: an interrupt's cost is its rate multiplied by the software cost of servicing one. The hardware cost and the system cost are unrelated, and the interface with the smallest hardware cost has the largest system cost. Coalescing by count bounds the CPU load and coalescing by time bounds the latency; a design needs both. Section 10.

Misconception 5 — "a synchroniser on each signal makes the crossing safe."

The wrong model: clock-domain crossing is solved by two flops, so putting two flops on every signal that crosses makes the design correct.

What it costs: a host that reads eight counters and gets eight values captured at eight different instants — which for a rate computation is wrong by however much the traffic changed in between, and at 100 Gb/s that is four and a half frames per crossing. The numbers are plausible and none of them is right.

The corrected model: a synchroniser is correct for one bit whose value is independently meaningful. A multi-bit value needs Gray coding or a handshake; a set of values that must be mutually consistent needs a snapshot taken in one source-clock cycle and crossed as a unit. Sections 6 and 14.

Misconception 6 — "the assertion failed, so the design is wrong."

The wrong model: an SVA property that fires has found a bug; the design must be fixed until it stops firing.

What it costs: days spent chasing a discrepancy between two counters that were never comparable, and — worse — a "fix" that adds a synchroniser to make the assertion pass, which changes nothing about the design's correctness and hides the next real failure.

The corrected model: a property whose operands are sampled in different clock domains is not false, it is unevaluable — there is no value of a receive-domain counter at a host clock edge. The verdict is an artefact of the simulator's event ordering. Check each domain separately and check the crossing's handshake; the conjunction is the claim you wanted. Section 19.


23. Interview Questions

Q1 — "A 10 Gb/s Ethernet MAC is being integrated. How much memory bandwidth do you ask the system architect for, and what do you say when they ask why it is more than 10 Gb/s?"

Ask for 22.9 Gb/s — 2.86 GB/s — and say it is two directions at 1.143× each. The 1.143 is a descriptor fetch and a status writeback, 32 octets per frame against a 64-octet minimum frame, which is a third of the memory traffic at minimum size and 2.1% at maximum. Then say the number depends on the frame-size distribution and that you have quoted the worst case, because a system architect who discovers that later will not believe the next number you give them.

Q2 — "Why is the receive clock different from every other clock on the chip?"

Because it is recovered from the incoming signal — it is the far end's transmit clock, arriving over the cable. So it is not generated by the SoC, not stoppable by the SoC, not synchronous with anything, and it stops when the partner stops transmitting or the cable is removed. Every flop in that domain freezes with it, including any counter that would report the problem — which is why clock-loss detection cannot live in the domain it detects.

Q3 — "Your 100 Gb/s MAC has a 512-bit AXI interface at 250 MHz. Is that enough?"

For bandwidth, yes — 128 Gb/s against 114.3 needed, 89% utilisation. For transactions, no. Minimum-size frames at 100 Gb/s are 148.8 Mfps, three transactions each, 446 M transactions per second — 1.79 per cycle, and one address channel issues one. The fix is fewer transactions per frame, not a wider bus: batched descriptor fetches take it to 1.26 per cycle and batched writebacks to 0.74, which fits.

Q4 — "How deep is the receive FIFO?"

It depends on a number the MAC does not own: the worst-case latency before the memory system accepts data. Depth is that latency times the line rate. At 10 Gb/s, 5 µs is 6.10 KiB. If the link must be lossless, add Chapter 14.2's headroom — 16.2 KiB at 10 Gb/s over 2 km — and note that the headroom is usually the larger term. Then say which number you were given and by whom, because the depth is a prediction and c_rx_high_water is how the next revision does better.

Q5 — "An integration testbench asserts that the host's frame count equals the MAC's. It fires constantly. What is wrong?"

The property, not the design. The MAC's counter increments on the recovered receive clock and the property samples on the host clock; the two have no phase relationship, so the sample lands inside the counter's transition. A 32-bit counter crossing 0x0000FFFF changes seventeen bits. The quantity the property refers to does not exist in the host's timeframe, so the property is unevaluable rather than false. Check the counter's monotonicity in its own domain, the snapshot's capture in the source domain, and the host's read of the snapshot in the host domain — the three compose into the claim.

Q6 — "You are told the Ethernet is slow. What do you read first?"

Four counters, in this order: c_overflow, c_stall_outstanding, c_split_4k and the ratio of c_txn_desc to c_txn_data. Overflow means a memory stall and CRC errors downstream. Stalls mean the memory system is the limit. Splits mean the buffer pool is not page-aligned. A descriptor count comparable to the data count means no batching. If all four are clean, say so explicitly — an explicit "none of these" is what ends the investigation, and it is the finding that sends it to the application rather than back around the same three hypotheses.


24. Understanding Check


25. What's Next

This chapter framed the problem and produced four numbers. Every remaining chapter in Module 18 is a response to one of them.

The numberWhat it forcesChapter
1.143× the line rate in memory bandwidtha system allocation, not a MAC changethe architect's
33.3% of it is descriptorsbatched fetch and batched writebackChapter 18.2, Chapter 18.3
1.79 transactions per cycle at 100 Gb/sfewer transactions per frameChapter 18.2 onward
297.6% of a CPU at 1 Gb/scoalescing, with both termsChapter 18.6

Chapter 18.2 — Descriptor Rings and the Ownership Model takes the first two. A ring is how the MAC and the driver hand buffers back and forth without a lock, and the handoff is one bit in a descriptor.

And that bit turns out to be the module's hardest problem, for a reason this chapter has set up. The ownership bit is a memory location written by two agents in different clock domains and different coherency domains — the MAC through Section 5's bus master, the driver through the CPU's cache. So the question is not what the bit means; it is whether the other fields of the descriptor are visible when the bit is.

Section 19's class 74 is the verification half of that question and Chapter 18.2 is the design half. A property relating two domains is unevaluable; a memory system that reorders two writes makes a descriptor whose ownership bit is set and whose length field is stale — and the MAC did nothing wrong in either case. The order of the writes against the ownership write is the whole correctness argument, and the barrier that enforces it is the chapter's subject.

Chapter 18.3 — The Receive DMA Path then traces a frame from Section 9's FIFO into host buffers: buffer exhaustion, scatter-gather across pages, and the write-completion ordering that decides when the ownership handoff is allowed to happen. It also prices the two jitter sources Chapter 16.1 §8 listed and left unpriced — DMA arbitration and interrupt coalescing — which this chapter's Sections 10 and 12 have finally made computable.

Continue learning

Standards & specifications

Governing standard
IEEE Std 802.3 (Ethernet)(opens IEEE in a new tab)

Defines the Ethernet MAC, the media-independent interfaces and the physical-layer sublayers, including framing, access control, auto-negotiation and per-rate PHY specifications. VLAN tagging, priority and time-sensitive shaping are defined by IEEE 802.1, not by 802.3.

This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.

Where this fits

Part of the Ethernet curriculum.