Ethernet · Module 18
The Ethernet MAC as an SoC IP Block
A MAC integrated into an SoC demands 1.143 times its line rate in memory bandwidth, crosses four clock domains, and at 100 Gb/s asks for 1.79 bus transactions per cycle.
Modules 1 to 17 treated the MAC as a thing with a wire on one side and frames on the other. Module 18 is about the other side.
A MAC in a switch is a port. A MAC in an SoC is a bus master that happens to have a cable attached — and almost every hard problem in the next seven chapters comes from that sentence rather than from Ethernet.
Here is the number the whole module turns on. A 1 Gb/s MAC receiving 64-octet frames does not ask the system for 1 Gb/s of memory bandwidth. It asks for 1.143 Gb/s, because every frame drags a descriptor fetch and a status writeback along with it, and those are 32 octets against a 64-octet frame.
| Line rate | Frame data | Descriptor traffic | Total, one direction | Descriptor share |
|---|---|---|---|---|
| 1 Gb/s | 0.762 Gb/s | 0.381 Gb/s | 1.143 Gb/s | 33.3% |
| 10 Gb/s | 7.619 Gb/s | 3.810 Gb/s | 11.429 Gb/s | 33.3% |
| 25 Gb/s | 19.048 Gb/s | 9.524 Gb/s | 28.571 Gb/s | 33.3% |
| 100 Gb/s | 76.190 Gb/s | 38.095 Gb/s | 114.286 Gb/s | 33.3% |
A third of the memory traffic is bookkeeping, and it is a third at every line rate because the ratio is set by the frame size rather than by the rate. At 1518-octet frames the same column reads 2.1%.
This chapter frames the problem and makes the numbers do the arguing, in the same way Chapter 16.1 did for Module 16. It builds the MAC's four interfaces, establishes the clock domains and what crosses them, and derives the memory bandwidth and transaction rate a system must supply. The mechanisms that manage that traffic — rings, both DMA directions, AXI shaping, interrupts, offload — are Chapter 18.2 onward.
What it does not do is design the MAC's internals. The parser, the assembler, the CRC integration and the FIFOs are Module 19, and this chapter deliberately stops at their boundaries.
1. Scope, and the Sentence the Module Turns On
A MAC integrated into an SoC is a peripheral with four interfaces, and only one of them carries Ethernet.
| Interface | Direction | Carries | Built in |
|---|---|---|---|
| CPU registers | slave | configuration, status, ring base addresses | Section 3 |
| Memory | master | frame data, descriptors, status | Section 5 |
| PHY / xMII | bidirectional | the frames themselves | Section 7 |
| Interrupt | output | completion signalling | Section 11 |
Row two is the one that makes the MAC different from almost every other peripheral on the chip. A UART, an SPI controller, a GPIO block and a timer are all slaves: the CPU reads and writes them and they never initiate a transaction. A MAC initiates, because a frame arrives without being asked for and must be in memory before software knows it exists.
And a master's requirements are different in kind. A slave needs an address decode and a handful of registers. A master needs a bandwidth allocation, a transaction budget, an ordering model, a coherency story, and a place in the system's arbitration — and every one of those is a system-integration negotiation rather than a MAC design decision.
What this chapter establishes:
| Section | Establishes |
|---|---|
| 2 | the four interfaces and what each one costs |
| 4 | the memory bandwidth, derived, at four line rates |
| 6 | the clock domains and every crossing |
| 10 | why one interrupt per frame is arithmetically impossible |
| 12 | the transaction rate, which is a harder wall than the bandwidth |
| 16 | reset, and why four domains do not share one |
| 17 | what the MAC assumes about the system it lands in |
What it does not establish, and where it goes instead:
| Question | Chapter |
|---|---|
| how the descriptors are organised | Chapter 18.2 |
| how a received frame reaches memory | Chapter 18.3 |
| how a transmitted frame leaves it | Chapter 18.4 |
| how the bursts are shaped for AXI | Chapter 18.5 |
| how the interrupts are coalesced | Chapter 18.6 |
| what the MAC's internals look like | Module 19 |
The forward references above that are bold rather than linked are chapters that are not yet published, which is this track's convention.
One more piece of framing, because it decides how the rest of the module reads. Modules 12 to 17 were about what Ethernet does on the wire. This module is about what Ethernet does to the chip it is on — and the two have almost nothing in common except the frames passing between them.
2. The Four Interfaces, and What Each One Costs
Take them one at a time, because the cost profile of each is different and integration mistakes come from assuming they are alike.
The CPU register interface is the cheapest and it is the one that gets the most attention. A few dozen 32-bit registers on APB or AXI-Lite: base addresses, enables, status, counters. Bandwidth: negligible — a few accesses per frame at most, and on a well-designed MAC, none at all in the steady state.
The memory interface is the expensive one. Section 4's table: 1.143 times the line rate in each direction, which at 100 Gb/s is 228.6 Gb/s — 28.6 GB/s — of a system's memory bandwidth, for one port.
The PHY interface is fixed by the standard and costs pins.
| Interface | Width | Clock | Rate |
|---|---|---|---|
| MII | 4 bits | 25 MHz | 0.100 Gb/s |
| GMII | 8 bits | 125 MHz | 1.000 Gb/s |
| XGMII | 64 bits | 156.25 MHz | 10.000 Gb/s |
| 25G XXVGMII | 64 bits | 390.625 MHz | 25.000 Gb/s |
| 100G CGMII | 512 bits | 195.3125 MHz | 100.000 Gb/s |
Read the last two rows against each other. 25 Gb/s and 100 Gb/s are reached by completely different means — one raises the clock to 390 MHz on a narrow bus, the other widens the bus to 512 bits and keeps the clock under 200 MHz — and which choice a design faces is decided by the PHY, not the MAC.
The interrupt interface is one wire and it is the one that breaks first. Section 10 derives it: one interrupt per frame at 1 Gb/s with 64-octet frames is 1.488 million interrupts per second, which at a 2 µs interrupt service routine is 297.6% of one CPU. The wire is free and what it asks of software is not.
Put the four costs side by side:
| Interface | Gate cost | System cost | Where it breaks |
|---|---|---|---|
| CPU registers | small | negligible | never |
| memory | moderate | 1.143× line rate | bandwidth, then transactions |
| PHY | pins | fixed by the standard | pin count at 100G |
| interrupt | one flop | CPU time | immediately |
The asymmetry is the point. The interface with the smallest hardware cost has the largest system cost, and the two interfaces an integration review spends its time on — registers and PHY — are the two that never cause a problem.
3. RTL 1 — The Register File, and the Domain It Is Not In
The register file looks like the easiest block in the MAC and it contains the module's first hard problem: almost nothing it reports lives in its own clock domain.
// -----------------------------------------------------------------------
// macsoc_pkg -- shared types for an Ethernet MAC as an SoC IP block.
// -----------------------------------------------------------------------
package macsoc_pkg;
localparam int ADDR_W = 64; // host physical address
localparam int REG_AW = 12; // 4 KiB register window
localparam int DATA_W = 32; // register data width
localparam int NUM_CNT = 16; // RMON-style counters -- 19.7
// The register map, kept deliberately small. Everything an operator
// needs in the steady state is a counter; everything else is written
// once at bring-up.
typedef enum logic [REG_AW-1:0] {
REG_CTRL = 12'h000,
REG_STATUS = 12'h004,
REG_RX_RING_LO = 12'h010,
REG_RX_RING_HI = 12'h014,
REG_RX_RING_LEN = 12'h018,
REG_TX_RING_LO = 12'h020,
REG_TX_RING_HI = 12'h024,
REG_TX_RING_LEN = 12'h028,
REG_IRQ_EN = 12'h030,
REG_IRQ_STATUS = 12'h034,
REG_COALESCE = 12'h038,
REG_CNT_BASE = 12'h100
} reg_addr_e;
// Which domain a register's CONTENT lives in. Section 6's subject,
// and the reason this file is not a simple array of flops.
typedef enum logic [1:0] {
DOM_HOST = 2'd0, // the bus clock -- no crossing
DOM_RX = 2'd1, // the receive recovered clock
DOM_TX = 2'd2, // the transmit clock
DOM_PTP = 2'd3 // 16.3's timestamp clock
} domain_e;
typedef struct packed {
logic enable;
logic promiscuous;
logic loopback;
logic [15:0] max_frame; // 5.4's MTU
} ctrl_t;
endpackage// -----------------------------------------------------------------------
// mac_register_file -- the CPU-facing slave.
//
// Configuration flows OUT of the host domain and is synchronised into
// the domains that use it. Counters flow IN from other domains and are
// synchronised on the way back. Neither direction is a simple flop,
// and section 19's rejected property is about what happens when a
// verification engineer forgets that.
// -----------------------------------------------------------------------
module mac_register_file
import macsoc_pkg::*;
(
input logic clk_host,
input logic rst_host_n,
// APB-style slave -- deliberately the simplest possible.
input logic psel,
input logic penable,
input logic pwrite,
input logic [REG_AW-1:0] paddr,
input logic [DATA_W-1:0] pwdata,
output logic [DATA_W-1:0] prdata,
output logic pready,
output logic pslverr,
// Configuration out, in the HOST domain. Every consumer synchronises.
output ctrl_t cfg_ctrl,
output logic [ADDR_W-1:0] cfg_rx_ring_base,
output logic [ADDR_W-1:0] cfg_tx_ring_base,
output logic [15:0] cfg_rx_ring_len,
output logic [15:0] cfg_tx_ring_len,
output logic [31:0] cfg_coalesce,
// Counters in, ALREADY synchronised by their owners. This module
// does not cross domains itself; it refuses to hold a value that
// has not been handed to it in its own clock.
input logic [31:0] cnt_value [NUM_CNT],
input logic cnt_valid [NUM_CNT],
input logic [31:0] irq_status_sync,
output logic [31:0] irq_enable,
output logic irq_ack,
output logic cfg_write_while_enabled
);
ctrl_t ctrl_q;
logic [ADDR_W-1:0] rx_base_q, tx_base_q;
logic [15:0] rx_len_q, tx_len_q;
logic [31:0] coalesce_q, irq_en_q;
logic [31:0] cnt_shadow [NUM_CNT];
wire access = psel && penable;
wire wr = access && pwrite;
// A ring base written while the MAC is enabled is the classic
// integration bug: the DMA is mid-descriptor against the old base
// and the next fetch uses the new one. The register file cannot
// prevent it -- it can only refuse it and say so.
assign cfg_write_while_enabled =
wr && ctrl_q.enable &&
(paddr == REG_RX_RING_LO || paddr == REG_RX_RING_HI ||
paddr == REG_TX_RING_LO || paddr == REG_TX_RING_HI ||
paddr == REG_RX_RING_LEN || paddr == REG_TX_RING_LEN);
always_ff @(posedge clk_host or negedge rst_host_n) begin
int i;
if (!rst_host_n) begin
ctrl_q <= '0;
rx_base_q <= '0; tx_base_q <= '0;
rx_len_q <= '0; tx_len_q <= '0;
coalesce_q <= '0; irq_en_q <= '0;
irq_ack <= 1'b0;
for (i = 0; i < NUM_CNT; i++) cnt_shadow[i] <= '0;
end else begin
irq_ack <= 1'b0;
// Counters land here only when their owner asserts valid, which
// that owner does from ITS synchroniser's output. The shadow is
// therefore always a value that was coherent at some instant --
// just not necessarily this one.
for (i = 0; i < NUM_CNT; i++)
if (cnt_valid[i]) cnt_shadow[i] <= cnt_value[i];
if (wr && !cfg_write_while_enabled) begin
case (paddr)
REG_CTRL: {ctrl_q.max_frame, ctrl_q.loopback,
ctrl_q.promiscuous, ctrl_q.enable}
<= pwdata[18:0];
REG_RX_RING_LO: rx_base_q[31:0] <= pwdata;
REG_RX_RING_HI: rx_base_q[63:32] <= pwdata;
REG_RX_RING_LEN: rx_len_q <= pwdata[15:0];
REG_TX_RING_LO: tx_base_q[31:0] <= pwdata;
REG_TX_RING_HI: tx_base_q[63:32] <= pwdata;
REG_TX_RING_LEN: tx_len_q <= pwdata[15:0];
REG_IRQ_EN: irq_en_q <= pwdata;
REG_COALESCE: coalesce_q <= pwdata;
REG_IRQ_STATUS: irq_ack <= 1'b1; // write-1-to-clear
default: ;
endcase
end
end
end
always_comb begin
prdata = '0;
pslverr = 1'b0;
if (paddr >= REG_CNT_BASE && paddr < REG_CNT_BASE + (NUM_CNT * 4))
prdata = cnt_shadow[(paddr - REG_CNT_BASE) >> 2];
else begin
case (paddr)
REG_CTRL: prdata = {13'b0, ctrl_q.max_frame, ctrl_q.loopback,
ctrl_q.promiscuous, ctrl_q.enable};
REG_STATUS: prdata = {31'b0, ctrl_q.enable};
REG_RX_RING_LO: prdata = rx_base_q[31:0];
REG_RX_RING_HI: prdata = rx_base_q[63:32];
REG_RX_RING_LEN: prdata = {16'b0, rx_len_q};
REG_TX_RING_LO: prdata = tx_base_q[31:0];
REG_TX_RING_HI: prdata = tx_base_q[63:32];
REG_TX_RING_LEN: prdata = {16'b0, tx_len_q};
REG_IRQ_EN: prdata = irq_en_q;
REG_IRQ_STATUS: prdata = irq_status_sync;
REG_COALESCE: prdata = coalesce_q;
default: pslverr = access && !pwrite;
endcase
end
end
assign pready = 1'b1;
assign cfg_ctrl = ctrl_q;
assign cfg_rx_ring_base = rx_base_q;
assign cfg_tx_ring_base = tx_base_q;
assign cfg_rx_ring_len = rx_len_q;
assign cfg_tx_ring_len = tx_len_q;
assign cfg_coalesce = coalesce_q;
assign irq_enable = irq_en_q;
endmoduleClassification: a single-domain register slave with an explicit refusal to cross domains itself.
What it teaches: that a register file's hard problem is not the decode, it is the provenance of every value it holds. cnt_shadow is a host-domain flop; the counter it shadows increments in the receive recovered clock, which is a different domain at a different frequency with no fixed phase relationship. The module deliberately does not synchronise it — it accepts an already-synchronised value with a valid, so that exactly one block owns each crossing and a reviewer can find every crossing by looking for the synchronisers rather than by reading every register.
And it teaches that cfg_write_while_enabled is the integration bug that survives every unit test. A driver that reconfigures a ring base without disabling the MAC passes in simulation, because the testbench's DMA is idle when the write lands. On real traffic the DMA is mid-descriptor against the old base and the next fetch uses the new one — so it writes a frame into whatever the old base's neighbour happens to be, which is memory belonging to something else.
Deliberately simplified: the counter shadow is 32 bits and a real RMON counter set is a mixture of 32-bit and 64-bit values, which reintroduces a problem this module dodges: a 64-bit counter cannot be read atomically over a 32-bit bus. Chapter 19.7's subject is the latch-on-read-low convention that fixes it, and this file's flat 32-bit array assumes it away. The register map is also flat rather than banked per queue, which Chapter 18.7's multi-queue makes untenable.
Production implication: cfg_write_while_enabled should be both a refusal and an interrupt. Refusing silently means a driver bug becomes a mysterious absence of traffic; refusing loudly means the driver author sees it on the first run. This is the same argument Chapter 14.1 §17 made about configuration errors generally — a value that is wrong from the moment it is written should be rejected at the moment it is written, not diagnosed from its consequences three layers away.
4. The Memory Bandwidth, Derived
Every frame that arrives costs the memory system more than the frame. Section 1's table asserted the number; this section derives it, because the derivation is where the design levers are.
Start with the frame rate. A frame on the wire occupies its own octets plus 8 octets of preamble and SFD — Chapter 4.2 — and 12 octets of interframe gap. So a 64-octet frame occupies 84 octets of wire time.
| Line rate | 64-octet frames | 1518-octet frames |
|---|---|---|
| 1 Gb/s | 1.4881 Mfps | 0.0813 Mfps |
| 10 Gb/s | 14.8810 Mfps | 0.8127 Mfps |
| 25 Gb/s | 37.2024 Mfps | 2.0319 Mfps |
| 100 Gb/s | 148.8095 Mfps | 8.1274 Mfps |
Now the per-frame memory cost on receive. Three accesses:
| Access | Size | Why |
|---|---|---|
| descriptor fetch | 16 octets | where does this frame go |
| frame data write | the frame | the frame itself |
| status writeback | 16 octets | how long was it, was it good |
So the ratio is (F + 32) / F, and the whole table follows from it.
| Frame size | Amplification | At 1 Gb/s | At 100 Gb/s |
|---|---|---|---|
| 64 | 1.500× | 1.143 Gb/s | 114.286 Gb/s |
| 128 | 1.250× | 1.081 Gb/s | 108.1 Gb/s |
| 256 | 1.125× | 1.043 Gb/s | 104.3 Gb/s |
| 512 | 1.063× | 1.023 Gb/s | 102.3 Gb/s |
| 1518 | 1.021× | 1.008 Gb/s | 100.780 Gb/s |
Two things about that table are worth stating carefully, because both are easy to get backwards.
The amplification factor is 1.5× at 64 octets and the bandwidth figure is 1.143×. Those are not inconsistent: the amplification is against the frame's own octets, and the bandwidth is against the line rate — and a line carrying 64-octet frames is only delivering 76.19% of its rate as frame data in the first place, because 20 of every 84 octets are preamble and gap. 0.7619 × 1.5 = 1.143.
And the descriptor share is 33.3% at every line rate. It is a function of frame size alone, so a design cannot outrun it by being faster — which is the opposite of most bandwidth problems, where the overhead is a fixed cost that amortises as the rate rises.
Both directions at once, worst case:
| Line rate | RX + TX, 64-octet frames | In bytes |
|---|---|---|
| 1 Gb/s | 2.286 Gb/s | 0.286 GB/s |
| 10 Gb/s | 22.857 Gb/s | 2.857 GB/s |
| 25 Gb/s | 57.143 Gb/s | 7.143 GB/s |
| 100 Gb/s | 228.571 Gb/s | 28.571 GB/s |
The last row is the one to take to a system architect. A single 100 Gb/s Ethernet port at minimum frame size demands 28.6 GB/s of memory bandwidth — which is a substantial fraction of a DDR4 channel's theoretical peak and more than its achievable one, for one peripheral, before the CPU has touched the data.
And software touches it afterwards, which doubles the figure again if the stack copies. This is why Chapter 18.3's scatter-gather and Chapter 18.7's offload exist — not as optimisations but as the difference between a port that runs at line rate and one that does not.
The design levers the derivation exposes, in order of size:
| Lever | Effect on the 33.3% | Whose chapter |
|---|---|---|
| larger frames | 33.3% to 2.1% | the network's, not the MAC's |
| descriptor prefetch in batches | amortises the fetch | Chapter 18.2 |
| writeback batching | amortises the status | Chapter 18.3 |
| smaller descriptors | linear in the size | Chapter 18.2 |
| a faster bus | none — the ratio is unchanged | nobody's |
Row five is the one people reach for and it does not help. Doubling the bus width halves the time and leaves the ratio at 33.3%; the descriptor traffic scales with the frame traffic exactly, which is what makes it a structural cost rather than a bottleneck.
5. RTL 2 — The Memory Master
The MAC's memory interface is the block that turns a frame into bus transactions, and its whole job is to keep Section 12's transaction rate down without violating anything.
// -----------------------------------------------------------------------
// mac_memory_master -- the MAC's bus-mastering front end.
//
// It accepts logical requests (fetch this descriptor, write this frame
// data, post this status) and issues bus transactions. It enforces the
// two rules the bus imposes and the MAC does not naturally respect:
// a burst may not cross a 4 KiB boundary, and the number of
// outstanding transactions is bounded.
// -----------------------------------------------------------------------
module mac_memory_master
import macsoc_pkg::*;
#(
parameter int BUS_BYTES = 32, // 256-bit data bus
parameter int MAX_OUTSTAND = 16,
parameter int MAX_BEATS = 16
)(
input logic clk,
input logic rst_n,
// Logical request in.
input logic req_valid,
output logic req_ready,
input logic req_write,
input logic [ADDR_W-1:0] req_addr,
input logic [15:0] req_bytes,
input logic [1:0] req_kind, // 0 desc, 1 data, 2 status
// Bus address channel out.
output logic ax_valid,
input logic ax_ready,
output logic ax_write,
output logic [ADDR_W-1:0] ax_addr,
output logic [7:0] ax_len, // beats minus one
output logic [2:0] ax_size,
// Completion in -- one per issued transaction.
input logic done_valid,
input logic done_error,
output logic [31:0] c_txn_desc,
output logic [31:0] c_txn_data,
output logic [31:0] c_txn_status,
output logic [31:0] c_split_4k,
output logic [31:0] c_stall_outstanding,
output logic [31:0] c_bus_error,
output logic outstanding_full
);
localparam int SIZE_ENC = $clog2(BUS_BYTES);
logic [ADDR_W-1:0] cur_addr;
logic [15:0] cur_bytes;
logic [1:0] cur_kind;
logic busy;
logic [$clog2(MAX_OUTSTAND+1)-1:0] outstanding;
// How many bytes may this burst cover before the 4 KiB boundary?
// A burst that crosses one is illegal on AXI and the MAC has no
// reason to respect a boundary it cannot see -- a frame lands
// wherever the descriptor's buffer pointer says.
logic [12:0] to_boundary;
assign to_boundary = 13'h1000 - {1'b0, cur_addr[11:0]};
logic [15:0] max_burst_bytes;
assign max_burst_bytes = MAX_BEATS * BUS_BYTES;
logic [15:0] this_burst;
always_comb begin
this_burst = cur_bytes;
if (this_burst > max_burst_bytes) this_burst = max_burst_bytes;
if (this_burst > {3'b0, to_boundary}) this_burst = {3'b0, to_boundary};
end
assign outstanding_full = (outstanding == MAX_OUTSTAND[$bits(outstanding)-1:0]);
assign req_ready = !busy;
assign ax_valid = busy && !outstanding_full;
assign ax_write = (cur_kind != 2'd0); // only a fetch reads
assign ax_addr = cur_addr;
assign ax_size = SIZE_ENC[2:0];
assign ax_len = (( this_burst + BUS_BYTES - 1) / BUS_BYTES) - 1;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
busy <= 1'b0; cur_addr <= '0; cur_bytes <= '0; cur_kind <= '0;
outstanding <= '0;
c_txn_desc <= '0; c_txn_data <= '0; c_txn_status <= '0;
c_split_4k <= '0; c_stall_outstanding <= '0; c_bus_error <= '0;
end else begin
if (!busy && req_valid) begin
busy <= 1'b1;
cur_addr <= req_addr;
cur_bytes <= req_bytes;
cur_kind <= req_kind;
end
if (busy && outstanding_full)
c_stall_outstanding <= c_stall_outstanding + 1;
if (ax_valid && ax_ready) begin
outstanding <= outstanding + 1;
case (cur_kind)
2'd0: c_txn_desc <= c_txn_desc + 1;
2'd1: c_txn_data <= c_txn_data + 1;
default: c_txn_status <= c_txn_status + 1;
endcase
// A split happened if the boundary, not the beat limit, was
// what shortened this burst. Counting it separately is what
// makes a misaligned buffer pool visible.
if ({3'b0, to_boundary} < cur_bytes &&
{3'b0, to_boundary} < max_burst_bytes)
c_split_4k <= c_split_4k + 1;
if (this_burst >= cur_bytes) begin
busy <= 1'b0;
cur_bytes <= '0;
end else begin
cur_addr <= cur_addr + {{(ADDR_W-16){1'b0}}, this_burst};
cur_bytes <= cur_bytes - this_burst;
end
end
if (done_valid) begin
outstanding <= outstanding - 1;
if (done_error) c_bus_error <= c_bus_error + 1;
end
// Simultaneous issue and completion cancel.
if (ax_valid && ax_ready && done_valid)
outstanding <= outstanding;
end
end
endmoduleClassification: a request splitter with an outstanding-transaction governor and per-kind accounting.
What it teaches: that the MAC's memory interface exists to reconcile two things that have no reason to agree — a frame, whose length is decided by whoever sent it, and a bus burst, whose legality is decided by a 4 KiB boundary and a beat limit the frame has never heard of. A 1518-octet frame landing 3 KiB into a page becomes two bursts, not one, and no amount of care in the MAC changes that; only the buffer pool's alignment does.
And it teaches that c_txn_desc, c_txn_data and c_txn_status must be separate counters. One combined transaction count cannot distinguish a port moving large frames efficiently from a port moving small frames and drowning in bookkeeping — and Section 4's whole argument is that those two look identical in bandwidth and completely different in transaction rate. Three counters make the ratio readable; one makes it invisible.
Deliberately simplified: a real master pipelines requests rather than handling one at a time. This module goes busy on a request and stays busy until its last burst issues, so a descriptor fetch cannot overlap a data write — which at Section 12's transaction rates is exactly the serialisation a 100 Gb/s port cannot afford. Chapter 18.5's subject is the request pipeline that removes it. The write data channel is also absent entirely, because this chapter is about the interface's shape rather than its datapath, and done_valid collapses AXI's separate read and write response channels into one.
Production implication: c_stall_outstanding is the counter that tells an integrator whether the MAC or the system is the limit. Rising with traffic means the memory system is not completing fast enough and the MAC is idling on its own outstanding limit — the fix is a larger limit, or a memory system that responds sooner. Flat at zero under load means the MAC is issuing at the rate it wants and the limit is elsewhere. Without it, a port that runs at 60% of line rate produces an argument between two teams and no evidence.
6. The Clock Domains, and What Crosses Them
A MAC in an SoC has four clock domains and no two of them are related. This section names them, names every crossing, and derives what a crossing costs.
| Domain | Source | Typical frequency | What lives in it |
|---|---|---|---|
| host | the SoC's bus clock | 250–400 MHz | registers, the memory master, descriptors |
| receive | recovered from the wire | 125 MHz to 390.625 MHz | the parser, receive counters |
| transmit | the local reference | 125 MHz to 390.625 MHz | the assembler, transmit counters |
| PTP | the synchronised clock | often a separate reference | Chapter 16.3's capture |
Row two is the one that surprises people coming from other peripherals. The receive clock is recovered from the incoming signal — it is the far end's transmit clock, arriving over the cable. So it is not a clock the SoC generates, not a clock it can stop, and not a clock that is synchronous with anything else on the chip. It exists because a partner is transmitting, and it stops when the partner stops.
Which makes a crossing out of nearly everything interesting.
| Crossing | Direction | What crosses | Mechanism |
|---|---|---|---|
| host → rx | out | configuration: enable, MTU, promiscuous | two-flop synchroniser |
| host → tx | out | configuration, the transmit request | two-flop synchroniser |
| rx → host | in | frame data | asynchronous FIFO — Section 9 |
| rx → host | in | counters | handshake, Section 14 |
| tx → host | in | completion, counters | handshake |
| ptp → host | in | timestamps | Chapter 16.3 §3's capture register |
And here is what a crossing costs, in the units that matter.
| Domain | Two flops | Three flops |
|---|---|---|
| GMII 125 MHz | 16.000 ns | 24.000 ns |
| XGMII 156.25 MHz | 12.800 ns | 19.200 ns |
| CGMII 195.3125 MHz | 10.240 ns | 15.360 ns |
| XXVGMII 390.625 MHz | 5.120 ns | 7.680 ns |
| host 250 MHz | 8.000 ns | 12.000 ns |
| host 400 MHz | 5.000 ns | 7.500 ns |
A full handshake is a round trip and therefore roughly the sum of two of those — about 24 ns from a GMII receive domain to a 250 MHz host, or 20.8 ns from XGMII.
Now put that latency against the frame rate, because the comparison is the module's second big number.
| Line rate | 64-octet frames per µs | Frames arriving in one 30 ns handshake |
|---|---|---|
| 1 Gb/s | 1.49 | 0.045 |
| 10 Gb/s | 14.88 | 0.446 |
| 25 Gb/s | 37.20 | 1.116 |
| 100 Gb/s | 148.81 | 4.464 |
Read the last row. At 100 Gb/s with minimum frames, four and a half frames arrive in the time it takes one value to cross from the receive domain to the host domain. A design that synchronises per frame does not work; a design that asserts a property relating a receive-domain signal to a host-domain one is asserting something about two instants that are four frames apart, which is Section 19's rejected class.
The engineering consequence is a rule and it is worth stating flatly: cross data in a FIFO and cross status in a handshake, and never cross either one signal at a time. Section 9's asynchronous FIFO crosses a whole frame's worth of octets with one pointer synchronisation; Section 14's telemetry crosses a whole counter set with one handshake. The cost is amortised over the payload, and it has to be.
7. RTL 3 — The PHY Boundary
The xMII side is where the MAC stops being a bus master and starts being an Ethernet device, and the adapter's job is to make five different physical interfaces look like one.
// -----------------------------------------------------------------------
// phy_boundary_adapter -- normalises the xMII family to one internal
// interface.
//
// MII, GMII, XGMII, XXVGMII and CGMII differ in width and clock and
// agree on almost nothing else. Everything above this module sees a
// width-W stream with a start, an end and a byte-enable on the last
// beat. Everything below sees the standard's signalling.
// -----------------------------------------------------------------------
module phy_boundary_adapter
import macsoc_pkg::*;
#(
parameter int XMII_BYTES = 8, // 1 = MII/GMII, 8 = XGMII, 64 = CGMII
parameter int INT_BYTES = 8
)(
input logic clk_rx, // RECOVERED from the wire
input logic rst_rx_n,
// xMII receive side.
input logic [XMII_BYTES*8-1:0] xgmii_rxd,
input logic [XMII_BYTES-1:0] xgmii_rxc, // control, not data
input logic rx_clk_present,
// Normalised internal stream out, still in clk_rx.
output logic s_valid,
output logic [INT_BYTES*8-1:0] s_data,
output logic [$clog2(INT_BYTES+1)-1:0] s_bytes,
output logic s_sof,
output logic s_eof,
output logic s_error,
output logic [31:0] c_sof,
output logic [31:0] c_eof,
output logic [31:0] c_local_fault,
output logic [31:0] c_clock_lost,
output logic rx_clock_stopped
);
// The control characters that matter. 4.2's SFD does not appear on
// XGMII -- the start control character replaces it, which is one of
// the family's least obvious incompatibilities.
localparam logic [7:0] XG_START = 8'hFB;
localparam logic [7:0] XG_TERM = 8'hFD;
localparam logic [7:0] XG_ERROR = 8'hFE;
localparam logic [7:0] XG_IDLE = 8'h07;
logic in_frame;
// A recovered clock that stops is not an error condition on the
// wire -- it is the partner having stopped transmitting, or the
// cable having been removed. It is, however, fatal to anything
// clocked by it, which is why the detector lives in the HOST
// domain in a real design and is modelled here as an input.
assign rx_clock_stopped = !rx_clk_present;
logic [INT_BYTES-1:0] is_start, is_term, is_error;
always_comb begin
int i;
for (i = 0; i < INT_BYTES; i++) begin
is_start[i] = xgmii_rxc[i] && (xgmii_rxd[i*8 +: 8] == XG_START);
is_term[i] = xgmii_rxc[i] && (xgmii_rxd[i*8 +: 8] == XG_TERM);
is_error[i] = xgmii_rxc[i] && (xgmii_rxd[i*8 +: 8] == XG_ERROR);
end
end
// How many bytes of this beat belong to the frame? Everything up to
// the terminate character.
logic [$clog2(INT_BYTES+1)-1:0] bytes_this_beat;
always_comb begin
int i;
bytes_this_beat = INT_BYTES[$bits(bytes_this_beat)-1:0];
for (i = INT_BYTES-1; i >= 0; i--)
if (is_term[i]) bytes_this_beat = i[$bits(bytes_this_beat)-1:0];
end
always_ff @(posedge clk_rx or negedge rst_rx_n) begin
if (!rst_rx_n) begin
in_frame <= 1'b0;
s_valid <= 1'b0; s_sof <= 1'b0; s_eof <= 1'b0; s_error <= 1'b0;
s_data <= '0; s_bytes <= '0;
c_sof <= '0; c_eof <= '0; c_local_fault <= '0; c_clock_lost <= '0;
end else begin
s_valid <= 1'b0; s_sof <= 1'b0; s_eof <= 1'b0; s_error <= 1'b0;
if (rx_clock_stopped) begin
c_clock_lost <= c_clock_lost + 1;
in_frame <= 1'b0;
end else if (|is_start && !in_frame) begin
in_frame <= 1'b1;
s_valid <= 1'b1;
s_sof <= 1'b1;
s_data <= xgmii_rxd;
s_bytes <= INT_BYTES[$bits(s_bytes)-1:0];
c_sof <= c_sof + 1;
end else if (in_frame) begin
s_valid <= 1'b1;
s_data <= xgmii_rxd;
if (|is_term) begin
in_frame <= 1'b0;
s_eof <= 1'b1;
s_bytes <= bytes_this_beat;
c_eof <= c_eof + 1;
end else begin
s_bytes <= INT_BYTES[$bits(s_bytes)-1:0];
end
if (|is_error) begin
s_error <= 1'b1;
c_local_fault <= c_local_fault + 1;
end
end
end
end
endmoduleClassification: a width- and encoding-normaliser at an asynchronous boundary, with an explicit clock-presence input.
What it teaches: that the xMII family's differences are not width alone. A GMII receive path has Chapter 4.2's preamble and SFD as data; an XGMII path has a start control character instead, and the preamble that follows it is padding whose only job is to keep the octet count right. A parser written against one and ported to the other looks for an SFD that is not there — which is a bring-up failure that presents as "the link is up and no frames arrive."
And it teaches that a recovered clock stopping is a first-class event, not an error. The receive clock exists because a partner is transmitting. When the cable is unplugged it simply stops — and every flop in the receive domain stops with it, including any counter that was going to report the problem. The detector therefore cannot live in the domain it is detecting, which is why rx_clk_present is an input from elsewhere rather than something this module derives.
Deliberately simplified: XMII_BYTES and INT_BYTES are separate parameters and the module assumes they are equal — the width conversion that makes a 512-bit CGMII stream into a 256-bit internal one is absent, and it is not trivial, because a start character may land in either half. Alignment is also assumed: a real XGMII start character may appear in lane 0 or lane 4, and the realignment that follows is Chapter 19.2's problem. Deficit-idle-count and the interframe-gap accounting that goes with it are absent entirely.
Production implication: c_sof and c_eof should be compared, and the comparison is one of the most useful two-counter checks in a MAC. They are equal on a healthy link. c_sof exceeding c_eof by a growing margin means frames are starting and not terminating — a truncation at the PHY, or a start character being seen mid-frame, both of which point at the serdes rather than at the MAC. The two counters cost almost nothing and separate a PHY fault from a MAC fault before a waveform is opened, which is Chapter 21.1's whole method in one comparison.
8. Why the Crossing FIFO Is Not Optional, and How Deep
A synchroniser crosses a signal. A frame is not a signal, and the block that crosses it has to be sized against something the MAC does not control: how long the memory system might make it wait.
The receive path has no backpressure toward the wire. Chapter 7.2 established it and it is worth restating in this chapter's terms: when a frame is arriving, it arrives. The MAC cannot ask the partner to pause mid-frame — Chapter 14.2's PAUSE operates between frames and takes 13.3 µs to take effect — so every octet that arrives must go somewhere immediately.
Which makes the receive FIFO's depth a direct function of the worst-case time before the memory system accepts data.
| Line rate | 0.5 µs stall | 1 µs | 2 µs | 5 µs |
|---|---|---|---|---|
| 1 Gb/s | 0.06 KiB | 0.12 KiB | 0.24 KiB | 0.61 KiB |
| 10 Gb/s | 0.61 KiB | 1.22 KiB | 2.44 KiB | 6.10 KiB |
| 25 Gb/s | 1.53 KiB | 3.05 KiB | 6.10 KiB | 15.26 KiB |
| 100 Gb/s | 6.10 KiB | 12.21 KiB | 24.41 KiB | 61.04 KiB |
The bottom-right cell is the one to look at. A 100 Gb/s MAC that must survive a 5 µs memory stall needs 61 KiB of on-chip FIFO, which is a substantial SRAM for one peripheral — and 5 µs is not a pathological stall on a system with a DRAM refresh, a competing master and an arbiter that is not aware Ethernet has a deadline.
And the column headings are not a design choice — they are a system property nobody writes down. The MAC's integrator knows the FIFO depth; the memory subsystem's owner knows the worst-case latency; and in most projects neither number is stated in the other team's terms until a port starts dropping frames under load.
This is the shape of every integration failure in this module, so state it once:
| The MAC knows | The system knows | Nobody owns |
|---|---|---|
| FIFO depth | worst-case memory latency | whether one covers the other |
| transaction rate offered | transaction rate accepted | the gap |
| interrupts generated | interrupt service time | CPU load |
| burst length requested | burst length honoured | efficiency |
Every row is a pair of numbers held by different people, and every row's failure looks like "the Ethernet is slow".
One more thing the depth table does not show, and it is the reason the FIFO is asynchronous rather than merely elastic. The two sides run on unrelated clocks: a recovered receive clock at 156.25 MHz nominal and a host clock at 250 MHz, with no common reference and a frequency ratio that is not rational. A synchronous FIFO with a rate-matching read is not sufficient; the pointers themselves must cross.
9. RTL 4 — The Asynchronous FIFO
One block, two clocks, and the whole of its correctness in the way the pointers are encoded.
// -----------------------------------------------------------------------
// async_fifo_cdc -- the receive-to-host frame crossing.
//
// Gray-coded pointers, two-flop synchronisers, and a deliberately
// pessimistic full/empty. The pessimism is the point: a synchronised
// pointer is always STALE, so the reader sees fewer entries than
// exist and the writer sees less space than exists. Both errors are
// safe; the opposite of either is not.
// -----------------------------------------------------------------------
module async_fifo_cdc
import macsoc_pkg::*;
#(
parameter int WIDTH = 72, // 64 data + 8 control
parameter int DEPTH = 512 // must be a power of two
)(
// Write side -- the recovered receive clock.
input logic clk_w,
input logic rst_w_n,
input logic w_en,
input logic [WIDTH-1:0] w_data,
output logic w_full,
output logic [31:0] c_overflow,
// Read side -- the host clock.
input logic clk_r,
input logic rst_r_n,
input logic r_en,
output logic [WIDTH-1:0] r_data,
output logic r_empty,
output logic [31:0] c_underflow
);
localparam int AW = $clog2(DEPTH);
logic [WIDTH-1:0] mem [DEPTH];
logic [AW:0] w_bin, w_gray, w_gray_r1, w_gray_r2;
logic [AW:0] r_bin, r_gray, r_gray_w1, r_gray_w2;
function automatic logic [AW:0] to_gray(input logic [AW:0] b);
return b ^ (b >> 1);
endfunction
// ---- write side -------------------------------------------------
wire [AW:0] w_bin_next = w_bin + {{AW{1'b0}}, (w_en && !w_full)};
always_ff @(posedge clk_w or negedge rst_w_n) begin
if (!rst_w_n) begin
w_bin <= '0; w_gray <= '0; c_overflow <= '0;
r_gray_w1 <= '0; r_gray_w2 <= '0;
end else begin
if (w_en && !w_full) mem[w_bin[AW-1:0]] <= w_data;
if (w_en && w_full) c_overflow <= c_overflow + 1;
w_bin <= w_bin_next;
w_gray <= to_gray(w_bin_next);
// The read pointer arrives here two flops late. Everything the
// write side believes about the FIFO's occupancy is therefore
// at least two read-clock periods out of date -- and that is
// exactly why it is safe.
r_gray_w1 <= r_gray;
r_gray_w2 <= r_gray_w1;
end
end
// Full: the write pointer has wrapped onto the read pointer. In
// Gray code that is the top two bits inverted and the rest equal.
assign w_full = (w_gray == {~r_gray_w2[AW:AW-1], r_gray_w2[AW-2:0]});
// ---- read side --------------------------------------------------
wire [AW:0] r_bin_next = r_bin + {{AW{1'b0}}, (r_en && !r_empty)};
always_ff @(posedge clk_r or negedge rst_r_n) begin
if (!rst_r_n) begin
r_bin <= '0; r_gray <= '0; c_underflow <= '0;
w_gray_r1 <= '0; w_gray_r2 <= '0;
end else begin
if (r_en && r_empty) c_underflow <= c_underflow + 1;
r_bin <= r_bin_next;
r_gray <= to_gray(r_bin_next);
w_gray_r1 <= w_gray;
w_gray_r2 <= w_gray_r1;
end
end
assign r_empty = (r_gray == w_gray_r2);
assign r_data = mem[r_bin[AW-1:0]];
endmoduleClassification: a dual-clock FIFO with Gray-coded pointer crossing and deliberately conservative flags.
What it teaches: that Gray coding is not an encoding convenience, it is the only reason the crossing works at all. A binary pointer going from 0111 to 1000 changes four bits; sampled by an unrelated clock mid-transition, any of sixteen values may be captured, and twelve of them are pointers the FIFO never held. A Gray-coded pointer changes exactly one bit per increment, so a mid-transition sample yields either the old value or the new one — both of which are real pointers the FIFO actually had.
And it teaches that both flags are wrong in the safe direction and must be. w_full compares against a read pointer that is two read-clock periods stale, so the write side believes the FIFO is fuller than it is and refuses writes it could have accepted. r_empty is stale the other way and the reader believes it is emptier. Both errors cost throughput; the opposite of either loses data, and there is no encoding that makes the flags exact, because exactness would require a common clock.
Deliberately simplified: the depth is a power of two, which the Gray comparison requires and a real design may not want. Almost-full and almost-empty thresholds are absent, and a production receive FIFO needs almost-full to drive Chapter 14.2's flow control with enough headroom for that chapter's 13.3 µs dead time — which is 1.62 KiB at 1 Gb/s and 162.4 KiB at 100. The memory is a flat array rather than an inferred dual-port SRAM, and there is no reset synchronisation between the two domains, which Section 16 is about.
Production implication: c_overflow is the single most important counter in a MAC's receive path and it is the one most often missing. An overflow means the frame currently arriving lost octets, and the frame will fail Chapter 6.3's FCS check as a result — so the symptom is a CRC error and the cause is a memory stall. An integrator seeing CRC errors and no overflow counter replaces the cable; one seeing CRC errors rising in step with c_overflow looks at the arbiter, which is where the problem is.
10. The Interrupt Interface, and Why One Per Frame Is Impossible
The interrupt is one wire and one flop. It is also the interface that fails first, and it fails by arithmetic rather than by any design error.
Take the naive design: assert an interrupt when a frame has been placed in memory.
| Line rate | Frame size | Interrupts per second | At a 2 µs ISR |
|---|---|---|---|
| 1 Gb/s | 64 | 1.488 M | 297.6% of one CPU |
| 1 Gb/s | 1518 | 0.081 M | 16.3% |
| 10 Gb/s | 64 | 14.881 M | 2976.2% |
| 10 Gb/s | 1518 | 0.813 M | 162.5% |
| 25 Gb/s | 64 | 37.202 M | 7440.5% |
| 100 Gb/s | 64 | 148.810 M | 29761.9% |
| 100 Gb/s | 1518 | 8.127 M | 1625.5% |
Row one settles it. A 1 Gb/s port — the slowest rate anybody still builds — at minimum frame size asks for three CPUs to do nothing but service its interrupts. This is not a 100 Gb/s problem that arrives later; it is a problem at the entry-level rate.
And row two is why the problem hides. At 1518-octet frames the same port asks for 16.3% of one CPU, which is high but survivable — so a design tested with large frames passes and the same design meets a burst of minimum-size frames and locks the system up. The frame-size distribution decides whether the design works, and it is chosen by whoever is sending.
Three mechanisms exist and they are not alternatives — a working design uses all three.
| Mechanism | What it does | Cost | Chapter |
|---|---|---|---|
| coalescing by count | one interrupt per N frames | latency up to N frame times | Chapter 18.6 |
| coalescing by time | one interrupt per T | latency up to T | Chapter 18.6 |
| polling under load | no interrupts at all while busy | CPU spin | the driver's |
Row three is worth naming even though it is software, because it changes what the hardware must provide. A driver that disables the interrupt and polls the ring while traffic is heavy needs the ring's state to be readable without an interrupt — which is Chapter 18.2's ownership bit, and it is one of that mechanism's less obvious justifications.
And the combination has a property worth stating: coalescing by count alone can stall indefinitely. A threshold of 32 frames with 31 frames received and no thirty-second holds those 31 frames until one more arrives, which on a quiet link may be seconds. The timer is what bounds it, so the two mechanisms are not alternatives — the count bounds the CPU cost and the timer bounds the latency, and a design with only one of them fails in whichever direction it omitted.
11. RTL 5 — The Interrupt Interface
One output, and everything interesting is in when it is allowed to go high.
// -----------------------------------------------------------------------
// mac_interrupt_iface -- event aggregation, masking and coalescing.
//
// The coalescing here is deliberately minimal -- a count threshold
// and a timer, which is the smallest thing that bounds both the CPU
// cost and the latency. Chapter 18.6's is adaptive.
// -----------------------------------------------------------------------
module mac_interrupt_iface
import macsoc_pkg::*;
#(
parameter int NUM_EVENTS = 8
)(
input logic clk_host,
input logic rst_host_n,
// Events, ALREADY in the host domain. Anything arriving from the
// receive or transmit domain has been through a handshake first.
input logic [NUM_EVENTS-1:0] evt,
// Which event indices are "a frame completed" and therefore
// subject to coalescing. Errors are never coalesced.
input logic [NUM_EVENTS-1:0] evt_coalescable,
input logic [31:0] cfg_coalesce, // {timer_us, count}
input logic [31:0] irq_enable,
input logic irq_ack,
input logic tick_1us,
output logic irq,
output logic [31:0] irq_status,
output logic [31:0] c_irq_raised,
output logic [31:0] c_irq_by_count,
output logic [31:0] c_irq_by_timer,
output logic [31:0] c_events_total,
output logic coalesce_misconfigured
);
wire [15:0] cfg_count = cfg_coalesce[15:0];
wire [15:0] cfg_timer_us = cfg_coalesce[31:16];
// A count threshold with no timer stalls whenever the traffic stops
// one frame short. This is the single most common coalescing
// misconfiguration and the design can see it at configuration time.
assign coalesce_misconfigured = (cfg_count > 16'd1) && (cfg_timer_us == '0);
logic [15:0] pending_count;
logic [15:0] timer_us;
logic [31:0] status_q;
wire [NUM_EVENTS-1:0] evt_masked = evt & irq_enable[NUM_EVENTS-1:0];
wire [NUM_EVENTS-1:0] evt_coal = evt_masked & evt_coalescable;
wire [NUM_EVENTS-1:0] evt_urgent = evt_masked & ~evt_coalescable;
wire count_reached = (pending_count >= cfg_count) && (cfg_count != '0);
wire timer_expired = (timer_us >= cfg_timer_us) && (cfg_timer_us != '0)
&& (pending_count != '0);
always_ff @(posedge clk_host or negedge rst_host_n) begin
if (!rst_host_n) begin
pending_count <= '0; timer_us <= '0; status_q <= '0; irq <= 1'b0;
c_irq_raised <= '0; c_irq_by_count <= '0; c_irq_by_timer <= '0;
c_events_total <= '0;
end else begin
c_events_total <= c_events_total + $countones(evt);
// Urgent events -- errors, overflow, a bus fault -- go straight
// through. Coalescing an error is how a design turns a
// diagnosable fault into a mysterious one.
if (|evt_urgent) begin
status_q <= status_q | {{(32-NUM_EVENTS){1'b0}}, evt_urgent};
irq <= 1'b1;
if (!irq) c_irq_raised <= c_irq_raised + 1;
end
if (|evt_coal) begin
status_q <= status_q | {{(32-NUM_EVENTS){1'b0}}, evt_coal};
pending_count <= pending_count + $countones(evt_coal);
end
if (tick_1us && pending_count != '0)
timer_us <= timer_us + 1;
if (count_reached || timer_expired) begin
irq <= 1'b1;
pending_count <= '0;
timer_us <= '0;
if (!irq) c_irq_raised <= c_irq_raised + 1;
if (count_reached) c_irq_by_count <= c_irq_by_count + 1;
else c_irq_by_timer <= c_irq_by_timer + 1;
end
if (irq_ack) begin
irq <= 1'b0;
status_q <= '0;
end
end
end
assign irq_status = status_q;
endmoduleClassification: an event aggregator with a two-term coalescing policy and an explicit non-coalescable class.
What it teaches: that coalescing must have an exemption and errors are it. A design that coalesces uniformly delays an overflow report by up to the timer's value — and the overflow's cause, a memory stall, is exactly the condition that makes the timer expire slowly. So the report of a problem is delayed by the problem. evt_coalescable is one mask bit per event and it removes the whole class of failure.
And it teaches why c_irq_by_count and c_irq_by_timer must be separate. They say which term is binding. Almost all by-count means the traffic is heavy and the coalescing is doing its job; almost all by-timer means the threshold is never reached and the interrupts are latency-bound rather than rate-bound — so the count is set too high for this traffic and every frame is waiting the full timer. One combined counter cannot distinguish a well-tuned port from a badly tuned one.
Deliberately simplified: irq is level-sensitive with a single acknowledge, and the acknowledge clears the whole status word — which races with an event arriving in the same cycle and loses it. A production design uses write-1-to-clear per bit with a set-priority-over-clear arrangement, and Chapter 18.6's adaptive coalescing replaces the fixed thresholds with ones that track the arrival rate. There is also one interrupt for the whole MAC rather than one per queue, which Chapter 18.7's multi-queue requires.
Production implication: coalesce_misconfigured deserves to be a loud flag rather than a quiet one, because the misconfiguration it catches produces a symptom nobody attributes to coalescing. A count of 32 with no timer on a link carrying 31 frames and then going quiet holds those 31 frames indefinitely — and the report is "some packets arrive seconds late, and only sometimes." The configuration is checkable in one comparison at write time, and diagnosing it from the symptom takes days.
12. The Transaction Rate, Which Is the Harder Wall
Section 4's bandwidth is solvable by widening the bus. Section 12's transaction rate is not, and it is the number that decides whether a design works.
A received frame costs three bus transactions in the simplest design: a descriptor fetch, a data write, a status writeback. Multiply by the frame rate.
| Line rate | 64-octet frame rate | Transactions/s | At 250 MHz | At 400 MHz |
|---|---|---|---|---|
| 1 Gb/s | 1.49 Mfps | 4.46 M | 0.018/cycle | 0.011/cycle |
| 10 Gb/s | 14.88 Mfps | 44.64 M | 0.179/cycle | 0.112/cycle |
| 25 Gb/s | 37.20 Mfps | 111.61 M | 0.446/cycle | 0.279/cycle |
| 100 Gb/s | 148.81 Mfps | 446.43 M | 1.786/cycle | 1.116/cycle |
The bottom two cells are both greater than one, and a single AXI address channel issues at most one transaction per cycle. So a 100 Gb/s MAC receiving minimum-size frames cannot present its traffic through one address channel at any clock frequency a modern SoC runs at — 446 million transactions per second would need a 446 MHz bus doing nothing else, and the write channel needs its own.
Contrast that with the bandwidth, which is entirely solvable:
| Line rate | Bandwidth needed | Width at 250 MHz | Width at 400 MHz |
|---|---|---|---|
| 1 Gb/s | 1.14 Gb/s | 8 bits | 8 bits |
| 10 Gb/s | 11.43 Gb/s | 64 bits | 32 bits |
| 25 Gb/s | 28.57 Gb/s | 128 bits | 128 bits |
| 100 Gb/s | 114.30 Gb/s | 512 bits | 512 bits |
512 bits at 250 MHz is a wide bus and it is a bus that exists. The transaction rate has no such answer, because widening the bus does not reduce the number of transactions — a descriptor fetch is 16 octets whether the bus is 64 bits or 512, and it is still one transaction.
Which is why Module 18's remaining chapters are, almost without exception, about transaction count rather than bandwidth:
| Mechanism | Transactions removed | Chapter |
|---|---|---|
| fetching 8 descriptors in one burst | 7 of every 8 fetches | Chapter 18.2 |
| batching status writebacks | most of the writebacks | Chapter 18.3 |
| larger buffers, fewer scatter fragments | the per-fragment transactions | Chapter 18.3 |
| coalescing interrupts | the interrupt, not a bus transaction | Chapter 18.6 |
| multi-queue with several masters | none — it parallelises them | Chapter 18.7 |
Row one alone takes the 100 Gb/s figure from 446 M to 316 M transactions per second — 148.81 × (1/8 + 1 + 1) — which is 1.26 per cycle at 250 MHz and still above one. Row two, batching the writebacks eight at a time as well, takes it to 186 M — 0.74 per cycle, which fits.
So the honest summary of Module 18 is this: the mechanisms exist because three transactions per frame does not fit, and they get it under one.
13. RTL 6 — The Datapath FIFOs and Their Watermarks
Section 9 built one crossing. A MAC has several FIFOs and the interesting part is the watermarks, because they are what turn a depth into a flow-control decision.
// -----------------------------------------------------------------------
// mac_datapath_fifos -- occupancy accounting and watermarks for the
// receive and transmit paths.
//
// The depths are elsewhere (section 9). What lives here is the part
// integration gets wrong: WHEN to tell the far end to stop, and how
// much must remain free after telling it.
// -----------------------------------------------------------------------
module mac_datapath_fifos
import macsoc_pkg::*;
#(
parameter int RX_DEPTH_B = 16384, // octets
parameter int TX_DEPTH_B = 16384
)(
input logic clk_host,
input logic rst_host_n,
input logic [15:0] rx_occupancy_b, // synchronised in
input logic [15:0] tx_occupancy_b,
// 14.2's headroom, in octets, for this line rate. The integrator
// supplies it because it depends on the CABLE, not on the MAC.
input logic [15:0] cfg_headroom_b,
input logic cfg_flow_control_en,
output logic xoff_request,
output logic rx_nearly_full,
output logic tx_starving,
output logic [31:0] c_xoff_asserted,
output logic [31:0] c_rx_high_water,
output logic [31:0] c_tx_underrun_risk,
output logic headroom_impossible
);
// The watermark is the depth minus the headroom. If the headroom
// exceeds the depth there is no watermark that works: by the time
// the far end stops, the FIFO has already overflowed. This is a
// CONFIGURATION fault and it is silent without this flag.
wire [16:0] watermark = {1'b0, RX_DEPTH_B[15:0]} - {1'b0, cfg_headroom_b};
assign headroom_impossible = (cfg_headroom_b >= RX_DEPTH_B[15:0]);
assign rx_nearly_full = !headroom_impossible &&
({1'b0, rx_occupancy_b} >= watermark);
assign xoff_request = cfg_flow_control_en && rx_nearly_full;
// A transmit FIFO that empties mid-frame causes an underrun, which
// corrupts a frame already on the wire -- 7.3's subject. The risk
// threshold is one maximum frame's worth still unsent.
assign tx_starving = (tx_occupancy_b < 16'd1518) && (tx_occupancy_b != '0);
always_ff @(posedge clk_host or negedge rst_host_n) begin
if (!rst_host_n) begin
c_xoff_asserted <= '0; c_rx_high_water <= '0; c_tx_underrun_risk <= '0;
end else begin
if (xoff_request) c_xoff_asserted <= c_xoff_asserted + 1;
if ({16'b0, rx_occupancy_b} > c_rx_high_water)
c_rx_high_water <= {16'b0, rx_occupancy_b};
if (tx_starving) c_tx_underrun_risk <= c_tx_underrun_risk + 1;
end
end
endmoduleClassification: watermark computation with an explicit impossibility check and a high-water recorder.
What it teaches: that the receive watermark is not a property of the FIFO, it is a property of the cable. Chapter 14.2 §8 derived the headroom: the octets that will still arrive after the decision to stop the partner is made, which is propagation delay plus the partner's dead time plus the pause frame's own transmission. A 2 km 1 Gb/s link needs about 1.62 KiB of headroom; a 2 m link needs almost none — so the same MAC in two deployments has two correct watermarks, and neither is discoverable from the MAC's own parameters.
And it teaches that headroom_impossible is a fault a design can detect and usually does not. A 100 Gb/s MAC needs 162.4 KiB of headroom over a long link. A 16 KiB receive FIFO cannot provide it at any watermark — the correct conclusion is that this MAC cannot do lossless flow control on this link, and the honest thing is to say so at configuration time rather than to assert XOFF at a watermark that cannot work and drop frames anyway.
Deliberately simplified: occupancy arrives already synchronised and as a flat octet count, when a real design tracks it in FIFO words and must convert. There is one watermark rather than the hysteresis pair a real design needs — a single threshold oscillates, asserting and deasserting XOFF on adjacent octets, and Chapter 14.2 §12's XON threshold is the missing half. Per-class occupancy is absent entirely, which Chapter 14.4 requires and which multiplies the whole structure by eight.
Production implication: c_rx_high_water is the counter that makes the FIFO's depth an evidence-based decision instead of a guess. It records the deepest the FIFO ever got, so a deployed system reports whether 16 KiB was generous or marginal — and a port whose high-water mark sits at 95% of depth is one memory-system hiccup from dropping frames, with no error counter to say so. The depth chosen at design time is a prediction; this counter is the measurement, and it is the only way the next revision's depth is better than this one's.
14. RTL 7 — Telemetry Across Four Domains
Every counter in this chapter lives in the domain that increments it, and the host has to read them all. This block is how they get there, and it is the answer to Section 6's arithmetic.
// -----------------------------------------------------------------------
// soc_integration_telemetry -- one handshake, a whole counter set.
//
// Section 6: at 100 Gb/s, 4.46 frames arrive during a single
// synchroniser round trip. Crossing per counter per update is not
// possible; crossing the SET on request is.
// -----------------------------------------------------------------------
module soc_integration_telemetry
import macsoc_pkg::*;
#(
parameter int NUM_CNT_LOCAL = 8
)(
// Source domain -- one instance per domain, parameterised.
input logic clk_src,
input logic rst_src_n,
input logic [31:0] cnt_src [NUM_CNT_LOCAL],
// Host domain.
input logic clk_host,
input logic rst_host_n,
input logic req, // host asks for a snapshot
output logic [31:0] cnt_host [NUM_CNT_LOCAL],
output logic cnt_host_valid,
output logic [31:0] c_snapshots,
output logic [31:0] c_req_while_busy
);
// ---- request crossing: host -> source ---------------------------
logic req_tgl_host, req_tgl_s1, req_tgl_s2, req_tgl_s3;
logic busy_host;
always_ff @(posedge clk_host or negedge rst_host_n) begin
if (!rst_host_n) begin
req_tgl_host <= 1'b0; busy_host <= 1'b0; c_req_while_busy <= '0;
end else begin
if (req && !busy_host) begin
req_tgl_host <= ~req_tgl_host;
busy_host <= 1'b1;
end else if (req && busy_host) begin
c_req_while_busy <= c_req_while_busy + 1;
end
end
end
always_ff @(posedge clk_src or negedge rst_src_n) begin
if (!rst_src_n) begin
req_tgl_s1 <= 1'b0; req_tgl_s2 <= 1'b0; req_tgl_s3 <= 1'b0;
end else begin
req_tgl_s1 <= req_tgl_host;
req_tgl_s2 <= req_tgl_s1;
req_tgl_s3 <= req_tgl_s2;
end
end
wire req_pulse_src = req_tgl_s2 ^ req_tgl_s3;
// ---- the snapshot ----------------------------------------------
// Taken in ONE source-clock cycle. Every counter in the set is
// therefore consistent with every other, which is the entire
// reason this block exists rather than a synchroniser per counter.
logic [31:0] snap [NUM_CNT_LOCAL];
logic ack_tgl_src;
always_ff @(posedge clk_src or negedge rst_src_n) begin
int i;
if (!rst_src_n) begin
for (i = 0; i < NUM_CNT_LOCAL; i++) snap[i] <= '0;
ack_tgl_src <= 1'b0;
end else if (req_pulse_src) begin
for (i = 0; i < NUM_CNT_LOCAL; i++) snap[i] <= cnt_src[i];
ack_tgl_src <= ~ack_tgl_src;
end
end
// ---- acknowledge crossing: source -> host -----------------------
logic ack_tgl_h1, ack_tgl_h2, ack_tgl_h3;
always_ff @(posedge clk_host or negedge rst_host_n) begin
int i;
if (!rst_host_n) begin
ack_tgl_h1 <= 1'b0; ack_tgl_h2 <= 1'b0; ack_tgl_h3 <= 1'b0;
cnt_host_valid <= 1'b0; c_snapshots <= '0;
for (i = 0; i < NUM_CNT_LOCAL; i++) cnt_host[i] <= '0;
end else begin
ack_tgl_h1 <= ack_tgl_src;
ack_tgl_h2 <= ack_tgl_h1;
ack_tgl_h3 <= ack_tgl_h2;
cnt_host_valid <= 1'b0;
// The snapshot registers are stable by now -- they were written
// before ack_tgl_src toggled, and that toggle took three host
// clocks to arrive. No synchroniser on the DATA is needed or
// wanted; the handshake is what makes it safe.
if (ack_tgl_h2 ^ ack_tgl_h3) begin
for (i = 0; i < NUM_CNT_LOCAL; i++) cnt_host[i] <= snap[i];
cnt_host_valid <= 1'b1;
busy_host <= 1'b0;
c_snapshots <= c_snapshots + 1;
end
end
end
endmoduleClassification: a four-phase toggle handshake carrying a consistent multi-word snapshot, with no synchroniser on the data path.
What it teaches: that the data does not get synchronised and must not be. A synchroniser on each of eight 32-bit counters is 256 flops and it is still wrong, because each counter would cross independently and the host would see eight values captured at eight different instants. The handshake makes one instant: the snapshot is taken in a single source-clock cycle, and the acknowledge's three-flop journey guarantees the snapshot registers are stable long before the host reads them. The data path is combinational and safe because the control path is slow.
And it teaches why a toggle rather than a pulse crosses the request. A single-cycle pulse in a 250 MHz host domain is invisible to a 125 MHz receive domain — the sampling clock may simply miss it. A toggle is a level change and cannot be missed, and the edge detector on the far side reconstructs the pulse. This is the standard construction and it is standard because the obvious alternative silently loses requests, at a rate that depends on the frequency ratio and therefore varies between deployments.
Deliberately simplified: one instance per domain means the register file sees several snapshot interfaces and must sequence them, and the sequencing is absent. The counters are 32 bits and wrap, with no sticky overflow flag — so a host that reads infrequently cannot tell one wrap from two. And busy_host is assigned in two always blocks in this listing, which is a deliberate simplification for readability and would not elaborate cleanly; a real design tracks it in one.
Production implication: c_req_while_busy is the counter that catches a monitoring agent polling faster than the handshake completes. A snapshot from a 125 MHz domain takes roughly 24 ns of crossing plus the request's own journey, so a host polling every 20 ns gets fewer snapshots than it asks for — and without this counter it silently reads stale values and computes rates that are wrong by whatever fraction it missed. The failure is a plausible number rather than an error, which is the hardest kind to notice.
15. RTL 8 — The Integration Conformance Monitor
The last block asserts the properties that are checkable on one side of the boundary, and is explicit about which ones are not.
// -----------------------------------------------------------------------
// macsoc_conformance_monitor -- what an integrator can check from the
// host side alone.
//
// Every property here has operands sampled in ONE clock domain. The
// ones that would relate a host-domain signal to a receive-domain
// one are absent by design, and section 19 explains why they are not
// merely hard but unevaluable.
// -----------------------------------------------------------------------
module macsoc_conformance_monitor
import macsoc_pkg::*;
(
input logic clk_host,
input logic rst_host_n,
input ctrl_t cfg_ctrl,
input logic cfg_write_while_enabled,
input logic headroom_impossible,
input logic coalesce_misconfigured,
input logic [31:0] c_overflow,
input logic [31:0] c_stall_outstanding,
input logic [31:0] c_bus_error,
input logic [31:0] c_split_4k,
input logic [31:0] c_txn_desc,
input logic [31:0] c_txn_data,
input logic [31:0] c_irq_by_timer,
input logic [31:0] c_irq_by_count,
input logic [31:0] c_req_while_busy,
input logic [31:0] c_sof_host, // snapshotted, section 14
input logic [31:0] c_eof_host, // snapshotted in the SAME snapshot
output logic integration_ok,
output logic cfg_fault,
output logic memory_system_marginal,
output logic interrupt_mistuned,
output logic phy_truncating,
output logic alignment_poor,
output logic none_of_the_above
);
// Configuration faults: checkable immediately, always fatal.
assign cfg_fault = cfg_write_while_enabled |
headroom_impossible |
coalesce_misconfigured;
// The memory system is marginal if the MAC is stalling on its own
// outstanding limit or, worse, has overflowed. Overflow is the
// symptom that presents as a CRC error -- section 9.
assign memory_system_marginal = (c_overflow != '0) ||
(c_stall_outstanding > 32'd1000) ||
(c_bus_error != '0);
// Almost all interrupts by timer means the count threshold is never
// reached: every frame is waiting the full timer for no reason.
assign interrupt_mistuned = (c_irq_by_timer > (c_irq_by_count << 3)) &&
(c_irq_by_timer > 32'd100);
// c_sof and c_eof come from the SAME snapshot, so this comparison
// is legitimate. Comparing them across two snapshots would not be.
assign phy_truncating = (c_sof_host > c_eof_host + 32'd1);
// A 4 KiB split on a meaningful fraction of data transactions means
// the buffer pool is not page-aligned -- 18.3's subject.
assign alignment_poor = (c_txn_data != '0) &&
(c_split_4k > (c_txn_data >> 4));
assign integration_ok = !cfg_fault && !memory_system_marginal &&
!phy_truncating;
// 14.3's habit: an explicit "nothing here is wrong" is what ends an
// investigation rather than deferring it.
assign none_of_the_above = integration_ok && !interrupt_mistuned &&
!alignment_poor && (c_req_while_busy == '0);
// ---- properties -------------------------------------------------
// Every one of these samples only host-domain signals.
p_cfg_fault_is_sticky_until_fixed:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
headroom_impossible |-> ##1 (headroom_impossible ||
$changed(cfg_ctrl)))
else $error("headroom fault cleared without a configuration change");
p_overflow_implies_not_ok:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
(c_overflow != '0) |-> !integration_ok)
else $error("receive overflow reported while integration_ok is high");
p_exactly_one_verdict_or_none:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
none_of_the_above |-> (!cfg_fault && !memory_system_marginal &&
!interrupt_mistuned && !phy_truncating &&
!alignment_poor))
else $error("none_of_the_above asserted alongside a finding");
p_desc_txn_never_exceeds_data_txn_by_much:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
(c_txn_data > 32'd1000) |-> (c_txn_desc <= (c_txn_data << 1)))
else $error("descriptor fetches far exceed data writes -- ring starvation");
endmoduleClassification: a host-domain verdict generator whose entire design constraint is that it may only look at host-domain signals.
What it teaches: that the monitor's most important property is which signals it refuses to take. There is no input from the receive clock domain in the port list. c_sof_host and c_eof_host are snapshotted values from Section 14's handshake, explicitly noted as coming from the same snapshot — because comparing two counters captured at two different instants is comparing two different moments in the traffic, and at 100 Gb/s two snapshots 30 ns apart are four and a half frames apart.
And it teaches that the verdicts partition into two kinds with completely different urgency. cfg_fault is a configuration error that was wrong the moment it was written and is fixable in a register write. memory_system_marginal is a system property that no register write fixes — it needs a different arbiter, a deeper FIFO, or less competing traffic. A monitor that reports them as one bit sends an integrator to the wrong team.
Deliberately simplified: the thresholds are literals — 1000 stalls, a factor of 8 on the interrupt ratio, one sixteenth on the split ratio — where a production monitor takes them from registers, because what counts as marginal depends on the deployment. The counters are compared as absolute values rather than as rates, so a long-running system eventually trips every threshold; a real monitor works on deltas between snapshots. p_cfg_fault_is_sticky_until_fixed also assumes cfg_ctrl changing implies the headroom was reconfigured, which is looser than it should be.
Production implication: none_of_the_above is the signal that ends an investigation. Chapter 14.3 §15 made the argument in a different context and it applies unchanged here: on a device where several distinct conditions produce the symptom "the Ethernet is slow", an explicit statement that none of them is present is as informative as any of them — and it is the only one that lets an integrator stop looking at the MAC and start looking at the application.
16. Reset, and Why Four Domains Do Not Share One
Reset is the part of integration that gets designed last and debugged first, and a four-domain block has four of them whether the design admits it or not.
The rule is short and the reason is not obvious: a reset may be asserted asynchronously and must be released synchronously to the domain it resets.
Asserting asynchronously is necessary because the clock may not be running — and in this MAC's receive domain the clock is recovered from the wire, so at power-on there is no clock at all. A reset that needs a clock edge to take effect never takes effect.
Releasing synchronously is necessary because a reset released asynchronously may release close to a clock edge, and different flops in the domain will then see different reset states on the same edge. The domain comes out of reset in a mixture, which for a state machine means an illegal state and for a FIFO pointer means a pointer that is not zero.
| Domain | Clock at power-on | Reset source | Release |
|---|---|---|---|
| host | running | the SoC's reset controller | synchronised to host clock |
| receive | absent until the partner transmits | host reset, plus clock-loss | synchronised to the recovered clock |
| transmit | running from the local reference | host reset | synchronised to transmit clock |
| PTP | running | host reset | synchronised to PTP clock |
And the crossings make the order matter, which is the part that is genuinely subtle.
Consider Section 9's asynchronous FIFO with the two sides reset independently. If the write side is released first, it writes into a FIFO whose read pointer is still held at zero — which is harmless. If the read side is released first, it reads from a FIFO whose write pointer is held at zero and whose r_empty is therefore asserted — also harmless. Neither order breaks it, because both flags are conservative.
Now consider the receive domain's reset being released while the host domain is still held. Frames arrive, the FIFO fills, and w_full asserts against a read pointer that will never move until the host comes out of reset. Nothing is corrupted and every frame after the FIFO fills is lost — which is the correct behaviour and looks exactly like a bug.
The rule that falls out: release the domains in the order host, transmit, receive, and hold the MAC's enable bit clear until all four are out. cfg_ctrl.enable is the real release and the resets are only the preconditions for it — which is why Section 3's register file resets ctrl_q to zero and nothing in the MAC does anything until software writes it.
One more case, and it is specific to this block. The receive clock stops when the cable is removed. Every flop in the receive domain freezes at whatever value it held, including the FIFO's write pointer and Gray code. When the cable is replugged, the clock restarts and those flops resume from a state that was valid — so the FIFO is not corrupted, but it may hold a partial frame from before the unplug, which will be reassembled onto the front of the next one. The receive domain must therefore be reset on clock return, not merely allowed to continue — and c_clock_lost from Section 7 is what triggers it.
17. What the MAC Assumes About the System It Lands In
A MAC's datasheet specifies what the MAC does. Integration failures come from what it assumes, and assumptions are not usually written down. Here are this chapter's, stated as a checklist.
| # | The assumption | If it is false | Established in |
|---|---|---|---|
| 1 | the memory system supplies 1.143× the line rate, both directions | frames drop under load | Section 4 |
| 2 | the bus accepts the transaction rate, not just the bandwidth | the port runs below line rate | Section 12 |
| 3 | worst-case memory latency fits in the receive FIFO | overflow, reported as CRC errors | Sections 8, 9 |
| 4 | descriptor buffers are page-aligned | every burst splits at 4 KiB | Section 5 |
| 5 | the CPU can service the interrupt rate | the system livelocks | Section 10 |
| 6 | software never rewrites a ring base while enabled | a frame lands in unrelated memory | Section 3 |
| 7 | the four resets are released in order | an illegal state at bring-up | Section 16 |
| 8 | the flow-control headroom fits in the FIFO | lossless is not achievable | Section 13 |
Rows 1, 2, 3 and 5 are the ones that cause the failures nobody can localise, because all four present as the same symptom — "the Ethernet is slow" or "we see CRC errors" — and none of them is an Ethernet problem.
And here is the asymmetry that makes them hard: the MAC can detect every one of them and cannot fix any of them.
| Assumption | Detectable by | Fixable by |
|---|---|---|
| bandwidth | c_stall_outstanding | the system's arbiter |
| transaction rate | c_txn_* ratios | Chapter 18.2's batching |
| latency vs FIFO | c_overflow, c_rx_high_water | a deeper FIFO — a silicon change |
| alignment | c_split_4k | the driver's allocator |
| interrupt rate | c_irq_raised | Chapter 18.6's coalescing |
The right column names four different owners and none of them is the MAC team, which is the single most useful thing this chapter has to say about integration. The MAC's job is to make each assumption's violation visible and attributable; everything after that belongs to somebody else, and the counters are what get it to them.
The checklist has a practical form worth stating: before integrating a MAC, write down the eight numbers on the right-hand side of the system's answer to the eight assumptions. Most projects can answer three of them. The other five are discovered during bring-up, at a cost measured in weeks.
18. The Integration Cost, Accounted
Put the eight blocks' costs side by side, because the distribution is not where the effort goes.
| Block | Approximate cost | Where it lands |
|---|---|---|
mac_register_file | ~1200 flops | logic |
mac_memory_master | ~800 flops + address arithmetic | logic |
phy_boundary_adapter | ~400 flops | logic |
async_fifo_cdc | ~120 flops + the SRAM | memory |
mac_interrupt_iface | ~200 flops | logic |
mac_datapath_fifos | ~200 flops | logic |
soc_integration_telemetry | ~300 flops per domain instance | logic |
macsoc_conformance_monitor | ~100 flops | logic, and usually not synthesised |
The logic totals roughly 3300 flops, which on a modern SoC is nothing. The memory does not:
| Line rate | Receive FIFO for a 2 µs stall | Plus 14.2's headroom, 2 km | Total SRAM |
|---|---|---|---|
| 1 Gb/s | 0.24 KiB | 1.62 KiB | ~2 KiB |
| 10 Gb/s | 2.44 KiB | 16.2 KiB | ~19 KiB |
| 25 Gb/s | 6.10 KiB | 40.6 KiB | ~47 KiB |
| 100 Gb/s | 24.41 KiB | 162.4 KiB | ~187 KiB |
The bottom row is the chapter's last surprise. A 100 Gb/s MAC that wants lossless operation over a 2 km link needs about 187 KiB of on-chip SRAM for its receive FIFO alone — and the flow-control headroom is 87% of it. The stall buffer, which is what everybody sizes, is the small part.
And a transmit FIFO roughly doubles it, though with a different sizing argument: the transmit side must hold enough to survive a memory stall without underrunning mid-frame, which Chapter 7.3 established corrupts a frame already on the wire.
Against the system:
| Size | Of a 4 MiB on-chip SRAM budget | |
|---|---|---|
| 1 Gb/s MAC, both directions | ~4 KiB | 0.10% |
| 10 Gb/s | ~38 KiB | 0.93% |
| 100 Gb/s | ~374 KiB | 9.1% |
9.1% of a chip's SRAM for one port's FIFOs is a real number and it is the one that gets negotiated, usually by reducing the headroom and accepting that the link is not lossless — which is a network-architecture decision being made by an SRAM budget, and it is made that way more often than anybody admits.
19. Properties Worth Asserting, and One Worth Refusing
Every property below samples its operands in a single clock domain. That constraint is not a stylistic preference and this section ends by explaining why.
Register-file properties — host domain.
// A configuration write to a ring base while the MAC is enabled must
// be refused, not applied. Applying it points the DMA at a new base
// mid-descriptor.
p_ring_base_write_refused_when_enabled:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
(psel && penable && pwrite && cfg_ctrl.enable &&
(paddr == REG_RX_RING_LO)) |=> $stable(cfg_rx_ring_base))
else $error("ring base changed while the MAC was enabled");
// A refused write must be reported, or a driver bug becomes an
// unexplained absence of traffic.
p_refused_write_flags:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
(psel && penable && pwrite && cfg_ctrl.enable &&
(paddr inside {REG_RX_RING_LO, REG_RX_RING_HI,
REG_TX_RING_LO, REG_TX_RING_HI}))
|-> cfg_write_while_enabled)
else $error("a ring-base write was refused silently");
// Reads never have side effects. A counter that clears on read cannot
// be sampled by two agents.
p_read_is_side_effect_free:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
(psel && penable && !pwrite) |=> $stable(cfg_ctrl))
else $error("a register read changed configuration state");
// An unmapped read errors rather than returning zero, because zero is
// a plausible value for every register in the map.
p_unmapped_read_errors:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
(psel && penable && !pwrite && (paddr == 12'hFFC)) |-> pslverr)
else $error("an unmapped read returned data instead of an error");
// The counter shadow only updates when its owner says the value is
// coherent.
p_counter_shadow_needs_valid:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
!cnt_valid[0] |=> $stable(cnt_shadow[0]))
else $error("a counter shadow updated without its valid");Memory-master properties — host domain.
// A burst may not cross a 4 KiB boundary. This is the bus's rule and
// the MAC has no natural reason to respect it.
p_burst_within_4k:
assert property (@(posedge clk) disable iff (!rst_n)
(ax_valid && ax_ready) |->
((ax_addr[11:0] + ((ax_len + 1) << ax_size)) <= 13'h1000))
else $error("a burst crossed a 4 KiB boundary");
// Outstanding transactions never exceed the limit the system agreed.
p_outstanding_bounded:
assert property (@(posedge clk) disable iff (!rst_n)
outstanding <= MAX_OUTSTAND)
else $error("outstanding transactions exceeded the agreed limit");
// A completion with no transaction outstanding is a bus-level fault
// or a tracking bug, and both are fatal.
p_no_orphan_completion:
assert property (@(posedge clk) disable iff (!rst_n)
done_valid |-> (outstanding != '0))
else $error("a completion arrived with nothing outstanding");
// The burst length never exceeds the parameterised maximum.
p_len_bounded:
assert property (@(posedge clk) disable iff (!rst_n)
(ax_valid && ax_ready) |-> (ax_len < MAX_BEATS))
else $error("burst length exceeded MAX_BEATS");
// A request is not accepted while one is in progress.
p_no_overlapping_requests:
assert property (@(posedge clk) disable iff (!rst_n)
busy |-> !req_ready)
else $error("a request was accepted while another was in progress");
// A descriptor fetch is a read; data and status are writes.
p_kind_implies_direction:
assert property (@(posedge clk) disable iff (!rst_n)
(ax_valid && (cur_kind == 2'd0)) |-> !ax_write)
else $error("a descriptor fetch was issued as a write");
// Every issued transaction eventually completes. Without a bound the
// MAC's outstanding counter leaks and the port stops.
p_transaction_completes:
assert property (@(posedge clk) disable iff (!rst_n)
(ax_valid && ax_ready) |-> ##[1:$] done_valid)
else $error("an issued transaction never completed");
// A remaining byte count only decreases.
p_bytes_monotonic:
assert property (@(posedge clk) disable iff (!rst_n)
busy |=> (cur_bytes <= $past(cur_bytes)))
else $error("the remaining byte count increased");PHY-boundary properties — receive domain, all operands local.
// A start character inside a frame is a framing error, not a new frame.
p_no_nested_start:
assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
(in_frame && |is_start) |-> s_error)
else $error("a start character mid-frame was treated as a new frame");
// Every frame that starts eventually terminates or errors.
p_frame_terminates:
assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
s_sof |-> ##[1:$] (s_eof || s_error))
else $error("a frame started and never terminated");
// The last beat's byte count is within the bus width.
p_eof_bytes_legal:
assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
s_eof |-> (s_bytes <= INT_BYTES))
else $error("the final beat claimed more bytes than the bus carries");
// A start of frame is never asserted twice without an end between.
p_sof_eof_alternate:
assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
s_sof |=> (!s_sof throughout (s_eof || s_error)[->1]))
else $error("two starts without an intervening end");
// Data is only valid inside a frame.
p_valid_only_in_frame:
assert property (@(posedge clk_rx) disable iff (!rst_rx_n)
s_valid |-> (in_frame || s_sof))
else $error("valid data outside a frame");Asynchronous-FIFO properties — each in its own domain, and note that there is no property relating the two.
// Write domain: no write when full.
p_no_write_when_full:
assert property (@(posedge clk_w) disable iff (!rst_w_n)
(w_en && w_full) |-> ##1 (c_overflow > $past(c_overflow)))
else $error("a write into a full FIFO was not counted as an overflow");
// Write domain: the Gray pointer changes by at most one bit.
p_write_gray_single_bit:
assert property (@(posedge clk_w) disable iff (!rst_w_n)
##1 ($countones(w_gray ^ $past(w_gray)) <= 1))
else $error("the write Gray pointer changed more than one bit");
// Read domain: the same, on the read side.
p_read_gray_single_bit:
assert property (@(posedge clk_r) disable iff (!rst_r_n)
##1 ($countones(r_gray ^ $past(r_gray)) <= 1))
else $error("the read Gray pointer changed more than one bit");
// Read domain: no read when empty.
p_no_read_when_empty:
assert property (@(posedge clk_r) disable iff (!rst_r_n)
(r_en && r_empty) |-> ##1 (c_underflow > $past(c_underflow)))
else $error("a read from an empty FIFO was not counted");
// Write domain: the synchronised read pointer only ever advances.
p_sync_read_ptr_monotonic:
assert property (@(posedge clk_w) disable iff (!rst_w_n)
##1 (r_gray_w2 == $past(r_gray_w2) ||
$countones(r_gray_w2 ^ $past(r_gray_w2)) == 1))
else $error("the synchronised read pointer jumped");Interrupt properties — host domain.
// An urgent event is never coalesced.
p_urgent_not_coalesced:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
|evt_urgent |=> irq)
else $error("an error event was delayed by coalescing");
// The interrupt is never asserted with an empty status word.
p_irq_implies_status:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
irq |-> (irq_status != '0))
else $error("the interrupt was asserted with no status bit set");
// An acknowledge clears the interrupt.
p_ack_clears:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
irq_ack |=> !irq)
else $error("the interrupt survived its acknowledge");
// A masked event never raises the interrupt.
p_mask_is_respected:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
((evt & ~irq_enable[NUM_EVENTS-1:0]) != '0 &&
(evt & irq_enable[NUM_EVENTS-1:0]) == '0 && !irq) |=> !irq)
else $error("a masked event raised the interrupt");
// Pending events with a configured timer are bounded in time.
p_pending_bounded_by_timer:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
((pending_count != '0) && (cfg_timer_us != '0))
|-> ##[1:$] (irq || (pending_count == '0)))
else $error("pending frames were held with a timer configured");
// A count threshold with no timer is flagged.
p_no_timer_is_flagged:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
((cfg_count > 16'd1) && (cfg_timer_us == '0))
|-> coalesce_misconfigured)
else $error("a stalling coalescing configuration was not flagged");Watermark properties — host domain.
// XOFF is only requested when flow control is enabled.
p_xoff_requires_enable:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
xoff_request |-> cfg_flow_control_en)
else $error("XOFF requested with flow control disabled");
// An impossible headroom suppresses the watermark rather than
// producing one that cannot work.
p_impossible_headroom_suppresses:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
headroom_impossible |-> !rx_nearly_full)
else $error("a watermark was computed from an impossible headroom");
// The high-water mark never decreases.
p_high_water_monotonic:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
##1 (c_rx_high_water >= $past(c_rx_high_water)))
else $error("the high-water mark decreased");Telemetry properties — each side in its own domain.
// Host domain: a snapshot is only requested when not busy.
p_no_request_while_busy:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
(req && busy_host) |=> (c_req_while_busy > $past(c_req_while_busy)))
else $error("a request while busy was not counted");
// Host domain: valid is a single cycle.
p_snapshot_valid_pulses:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
cnt_host_valid |=> !cnt_host_valid)
else $error("the snapshot valid was held for more than a cycle");
// Source domain: the snapshot is written in one cycle, so every
// element changes together or none does.
p_snapshot_is_atomic:
assert property (@(posedge clk_src) disable iff (!rst_src_n)
(!req_pulse_src) |=> $stable(snap[0]) && $stable(snap[1]))
else $error("snapshot elements changed outside a request");
// Host domain: a request is eventually answered.
p_request_completes:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
(req && !busy_host) |-> ##[1:$] cnt_host_valid)
else $error("a snapshot request was never answered");Monitor properties — host domain.
// A configuration fault and an all-clear verdict cannot coexist.
p_fault_excludes_ok:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
cfg_fault |-> !integration_ok)
else $error("integration_ok asserted alongside a configuration fault");
// The truncation verdict uses two counters from ONE snapshot.
p_truncation_uses_one_snapshot:
assert property (@(posedge clk_host) disable iff (!rst_host_n)
phy_truncating |-> $past(cnt_host_valid, 1))
else $error("a truncation verdict was computed from unsynchronised counts");20. Verification Scenarios
Fifty-eight scenarios, grouped by which of Section 17's assumptions each one tests. The last group is the directed test, and it is here because random stimulus will not produce it.
Register interface — 8 scenarios.
| # | Scenario | Expected |
|---|---|---|
| 1 | write every register, read back | values match |
| 2 | read an unmapped address | pslverr, no data |
| 3 | write a ring base with enable clear | accepted |
| 4 | write a ring base with enable set | refused, cfg_write_while_enabled |
| 5 | write REG_IRQ_STATUS | irq_ack pulses, status clears |
| 6 | read a counter shadow with no cnt_valid | previous value, unchanged |
| 7 | back-to-back writes to adjacent registers | both land |
| 8 | write REG_CTRL with max_frame above 9000 | accepted; the MTU is the integrator's |
Memory master — 10 scenarios.
| # | Scenario | Expected |
|---|---|---|
| 9 | 64-octet write, aligned | one burst, 2 beats at 256 bits |
| 10 | 1518-octet write starting 3 KiB into a page | two bursts, c_split_4k increments |
| 11 | 1518-octet write, page-aligned | one or more bursts, c_split_4k flat |
| 12 | 16-octet descriptor fetch | one beat, read direction |
| 13 | a request arriving while busy | req_ready low, request held |
| 14 | MAX_OUTSTAND reached | ax_valid low, c_stall_outstanding rises |
| 15 | completion with done_error | c_bus_error increments |
| 16 | a write ending exactly on a 4 KiB boundary | one burst, no split |
| 17 | a write of zero bytes | no transaction issued |
| 18 | issue and completion in the same cycle | outstanding unchanged |
PHY boundary — 9 scenarios.
| # | Scenario | Expected |
|---|---|---|
| 19 | a minimum frame, start to terminate | s_sof, beats, s_eof, c_sof == c_eof |
| 20 | a start character while in_frame | s_error, not a new frame |
| 21 | a terminate character in lane 0 | s_bytes is zero for that beat |
| 22 | a terminate in the last lane | s_bytes is the full width |
| 23 | an error character mid-frame | s_error, c_local_fault rises |
| 24 | rx_clk_present drops mid-frame | in_frame clears, c_clock_lost rises |
| 25 | rx_clk_present returns | the next start is clean |
| 26 | idle characters only | no s_valid |
| 27 | back-to-back frames at minimum IFG | both delivered, counts equal |
Clock crossing — 9 scenarios.
| # | Scenario | Expected |
|---|---|---|
| 28 | write clock 10× the read clock | w_full asserts, no data lost |
| 29 | read clock 10× the write clock | r_empty asserts often, no underflow |
| 30 | write into a full FIFO | c_overflow rises, write discarded |
| 31 | read from an empty FIFO | c_underflow rises |
| 32 | Gray pointer across a wrap | one bit changes at DEPTH-1 to 0 |
| 33 | both resets released simultaneously | both pointers zero, r_empty high |
| 34 | write reset released first | writes accepted, reads blocked |
| 35 | read reset released first | r_empty high, no spurious read |
| 36 | the write clock stops mid-frame | pointers freeze, no corruption |
Interrupts — 8 scenarios.
| # | Scenario | Expected |
|---|---|---|
| 37 | count threshold 1, one frame | immediate interrupt |
| 38 | count 32, 31 frames, timer 100 µs | interrupt at 100 µs, c_irq_by_timer |
| 39 | count 32, 32 frames | immediate, c_irq_by_count |
| 40 | count 32, timer 0, 31 frames | no interrupt; coalesce_misconfigured set |
| 41 | an error event during coalescing | immediate interrupt, not delayed |
| 42 | acknowledge with an event in the same cycle | the event is not lost |
| 43 | all events masked | no interrupt |
| 44 | timer expires with zero pending | no interrupt |
Watermarks and telemetry — 8 scenarios.
| # | Scenario | Expected |
|---|---|---|
| 45 | occupancy below watermark | xoff_request low |
| 46 | occupancy crosses the watermark | xoff_request high, counter rises |
| 47 | headroom exceeds depth | headroom_impossible, no watermark |
| 48 | flow control disabled at the watermark | rx_nearly_full high, xoff_request low |
| 49 | transmit FIFO below one maximum frame | tx_starving |
| 50 | a snapshot request at idle | cnt_host_valid after the round trip |
| 51 | a second request before the first completes | c_req_while_busy rises, first completes |
| 52 | source counters change during the crossing | the host sees the snapshot's values |
The directed test — 6 runs random stimulus will not produce.
Random stimulus explores frame sizes, arrival times and bus delays independently. What it does not produce is the correlation that makes an integration failure: a memory stall of a specific duration arriving during a burst of minimum-size frames, repeated often enough to matter. The reason is combinatorial — the stall must be long enough to matter and short enough not to be an obvious fault, and it must land during a minimum-frame burst rather than between bursts — and the probability of a random generator producing that conjunction is small enough that a long regression will not see it.
Construct it directly. Five runs, one variable.
| Run | Memory stall | Traffic | Receive FIFO | Expected outcome |
|---|---|---|---|---|
| A | 0.5 µs | 64-octet burst at line rate | 16 KiB at 10 Gb/s | occupancy peaks at 0.61 KiB — fine |
| B | 2 µs | same | same | 2.44 KiB — fine, c_rx_high_water rises |
| C | 5 µs | same | same | 6.10 KiB — fine, high-water at 38% |
| D | 13 µs | same | same | 15.9 KiB — at the edge, no overflow |
| E | 14 µs | same | same | overflow; CRC errors downstream |
| F | 14 µs | 1518-octet frames | same | overflow at the same instant |
Run F is the one that makes the test worth constructing, and its result is the opposite of the intuition. The frame size does not change when the overflow happens — the FIFO fills at the line rate regardless of how that rate is divided into frames — so a design tested only with large frames and no stall passes, and the same design tested with large frames and a 14 µs stall fails identically to the small-frame case.
Which makes the stall, not the frame size, the variable that matters for the FIFO — and the frame size the variable that matters for Sections 10 and 12. The two failure modes have disjoint causes and a regression that varies only one of them finds only one.
The oracle for the six runs, checked in four parts:
| Check | Run A–D | Run E–F |
|---|---|---|
c_overflow | zero | non-zero |
c_rx_high_water | below depth | equals depth |
| downstream FCS errors | zero | one per overflowed frame |
memory_system_marginal | low | high |
Row three is the part an integration test usually gets wrong. The overflow's symptom is a CRC error at the next stage — so a testbench checking only for CRC errors sees runs E and F fail and concludes the receive path corrupts frames. Checking c_overflow alongside is what turns "the MAC is broken" into "the memory system stalled for 14 µs", which is Section 9's production implication demonstrated rather than asserted.
And row four closes the loop to Section 15. The monitor's verdict must agree with the counters; a run where c_overflow is non-zero and memory_system_marginal is low is a monitor bug, not a MAC bug, and the directed test is the only place that pairing is exercised deliberately.
21. Debugging an Integration, in the Order the Evidence Arrives
Every failure in this chapter presents as one of three complaints. Start from the complaint and the counters narrow it in about ten minutes.
Complaint 1 — "we see CRC errors."
| Check | If yes | If no |
|---|---|---|
c_overflow non-zero? | a memory stall — go to complaint 2 | continue |
c_sof exceeds c_eof? | the PHY is truncating — serdes or cable | continue |
c_local_fault rising? | the PHY is reporting errors — layer 1 | continue |
| errors on both directions? | the cable | one direction — that transceiver |
Row one is the finding this chapter exists to make possible. Without c_overflow, a CRC error rate caused by a memory stall is indistinguishable from one caused by a marginal optic — and the diagnostic instinct built up over thirty years says optic. The cable gets replaced, the errors persist, and the investigation restarts.
Complaint 2 — "the port does not reach line rate."
| Check | If yes | Meaning |
|---|---|---|
c_stall_outstanding rising with load? | the memory system is the limit | raise MAX_OUTSTAND, or fix the arbiter |
c_split_4k above a sixteenth of c_txn_data? | the buffer pool is not page-aligned | the driver's allocator |
c_txn_desc comparable to c_txn_data? | no descriptor batching | Chapter 18.2 |
| transaction rate near one per cycle? | Section 12's wall | batching, then multi-queue |
| all four flat, rate still low? | the offered load is what it is | none_of_the_above |
Complaint 3 — "the system is unresponsive under network load."
| Check | If yes | Meaning |
|---|---|---|
c_irq_raised near the frame rate? | no coalescing in effect | Section 10's arithmetic |
c_irq_by_timer far above c_irq_by_count? | the count threshold is never reached | mistuned, not absent |
coalesce_misconfigured set? | a count with no timer | frames held indefinitely |
| both low and the system still stalls? | not the MAC | look at the driver's receive loop |
And the two symptoms most likely to be misattributed, stated plainly because both send an investigation to the wrong team:
| Symptom | Instinct | This chapter's cause |
|---|---|---|
| CRC errors | the cable | a memory stall overflowing the receive FIFO |
| packets arriving seconds late, sometimes | the network | a coalescing count with no timer |
Both are diagnosed by one register read and both take days without it, which is the argument for every counter in this chapter in one table.
22. Misconceptions
Misconception 1 — "a MAC needs memory bandwidth equal to its line rate."
The wrong model: a 1 Gb/s port moves 1 Gb/s of data, so it needs 1 Gb/s of memory bandwidth.
What it costs: a memory allocation that is 14.3% short at minimum frame size, which shows up as a port that reaches line rate with large frames and falls short with small ones — and the shortfall is then attributed to the traffic rather than to the allocation.
The corrected model: the MAC moves the frame plus a descriptor fetch plus a status writeback, 32 octets of bookkeeping per frame. At 64-octet frames the memory requirement is 1.143× the line rate; at 1518-octet frames it is 1.008×. Both directions at once doubles it. Section 4.
Misconception 2 — "a wider bus solves it."
The wrong model: the port is limited by bandwidth, so doubling the bus width doubles the headroom.
What it costs: a design iteration that widens the bus from 256 to 512 bits, doubles the area of every datapath register, and produces no measurable improvement — because the limit was transactions, not bytes.
The corrected model: a descriptor fetch is one transaction whether the bus is 64 bits or 512. At 100 Gb/s with minimum frames the MAC offers 446 M transactions per second — 1.79 per cycle at 250 MHz — and one address channel issues one. Widening the bus changes the bandwidth column and leaves the transaction column untouched. Section 12.
Misconception 3 — "the receive FIFO needs to hold a frame."
The wrong model: the FIFO exists to buffer a frame while it is being written to memory, so one maximum frame is enough.
What it costs: a 2 KiB FIFO on a 10 Gb/s port, which overflows on any memory stall longer than 1.6 µs — and the overflow presents as a CRC error, so the FIFO is not suspected.
The corrected model: the FIFO's depth is set by the worst-case time before the memory system accepts data, multiplied by the line rate. At 10 Gb/s a 5 µs stall needs 6.10 KiB, and a lossless configuration needs Chapter 14.2's headroom on top — 16.2 KiB more at 10 Gb/s over 2 km. Sections 8 and 18.
Misconception 4 — "the interrupt is a wire, so it is free."
The wrong model: an interrupt output costs one flop and no bandwidth, so it is the cheapest interface on the block.
What it costs: a system that livelocks the first time it meets a burst of minimum-size frames. At 1 Gb/s that is 1.488 M interrupts per second and 297.6% of one CPU — on the slowest rate anybody still builds.
The corrected model: an interrupt's cost is its rate multiplied by the software cost of servicing one. The hardware cost and the system cost are unrelated, and the interface with the smallest hardware cost has the largest system cost. Coalescing by count bounds the CPU load and coalescing by time bounds the latency; a design needs both. Section 10.
Misconception 5 — "a synchroniser on each signal makes the crossing safe."
The wrong model: clock-domain crossing is solved by two flops, so putting two flops on every signal that crosses makes the design correct.
What it costs: a host that reads eight counters and gets eight values captured at eight different instants — which for a rate computation is wrong by however much the traffic changed in between, and at 100 Gb/s that is four and a half frames per crossing. The numbers are plausible and none of them is right.
The corrected model: a synchroniser is correct for one bit whose value is independently meaningful. A multi-bit value needs Gray coding or a handshake; a set of values that must be mutually consistent needs a snapshot taken in one source-clock cycle and crossed as a unit. Sections 6 and 14.
Misconception 6 — "the assertion failed, so the design is wrong."
The wrong model: an SVA property that fires has found a bug; the design must be fixed until it stops firing.
What it costs: days spent chasing a discrepancy between two counters that were never comparable, and — worse — a "fix" that adds a synchroniser to make the assertion pass, which changes nothing about the design's correctness and hides the next real failure.
The corrected model: a property whose operands are sampled in different clock domains is not false, it is unevaluable — there is no value of a receive-domain counter at a host clock edge. The verdict is an artefact of the simulator's event ordering. Check each domain separately and check the crossing's handshake; the conjunction is the claim you wanted. Section 19.
23. Interview Questions
Q1 — "A 10 Gb/s Ethernet MAC is being integrated. How much memory bandwidth do you ask the system architect for, and what do you say when they ask why it is more than 10 Gb/s?"
Ask for 22.9 Gb/s — 2.86 GB/s — and say it is two directions at 1.143× each. The 1.143 is a descriptor fetch and a status writeback, 32 octets per frame against a 64-octet minimum frame, which is a third of the memory traffic at minimum size and 2.1% at maximum. Then say the number depends on the frame-size distribution and that you have quoted the worst case, because a system architect who discovers that later will not believe the next number you give them.
Q2 — "Why is the receive clock different from every other clock on the chip?"
Because it is recovered from the incoming signal — it is the far end's transmit clock, arriving over the cable. So it is not generated by the SoC, not stoppable by the SoC, not synchronous with anything, and it stops when the partner stops transmitting or the cable is removed. Every flop in that domain freezes with it, including any counter that would report the problem — which is why clock-loss detection cannot live in the domain it detects.
Q3 — "Your 100 Gb/s MAC has a 512-bit AXI interface at 250 MHz. Is that enough?"
For bandwidth, yes — 128 Gb/s against 114.3 needed, 89% utilisation. For transactions, no. Minimum-size frames at 100 Gb/s are 148.8 Mfps, three transactions each, 446 M transactions per second — 1.79 per cycle, and one address channel issues one. The fix is fewer transactions per frame, not a wider bus: batched descriptor fetches take it to 1.26 per cycle and batched writebacks to 0.74, which fits.
Q4 — "How deep is the receive FIFO?"
It depends on a number the MAC does not own: the worst-case latency before the memory system accepts data. Depth is that latency times the line rate. At 10 Gb/s, 5 µs is 6.10 KiB. If the link must be lossless, add Chapter 14.2's headroom — 16.2 KiB at 10 Gb/s over 2 km — and note that the headroom is usually the larger term. Then say which number you were given and by whom, because the depth is a prediction and c_rx_high_water is how the next revision does better.
Q5 — "An integration testbench asserts that the host's frame count equals the MAC's. It fires constantly. What is wrong?"
The property, not the design. The MAC's counter increments on the recovered receive clock and the property samples on the host clock; the two have no phase relationship, so the sample lands inside the counter's transition. A 32-bit counter crossing 0x0000FFFF changes seventeen bits. The quantity the property refers to does not exist in the host's timeframe, so the property is unevaluable rather than false. Check the counter's monotonicity in its own domain, the snapshot's capture in the source domain, and the host's read of the snapshot in the host domain — the three compose into the claim.
Q6 — "You are told the Ethernet is slow. What do you read first?"
Four counters, in this order: c_overflow, c_stall_outstanding, c_split_4k and the ratio of c_txn_desc to c_txn_data. Overflow means a memory stall and CRC errors downstream. Stalls mean the memory system is the limit. Splits mean the buffer pool is not page-aligned. A descriptor count comparable to the data count means no batching. If all four are clean, say so explicitly — an explicit "none of these" is what ends the investigation, and it is the finding that sends it to the application rather than back around the same three hypotheses.
24. Understanding Check
25. What's Next
This chapter framed the problem and produced four numbers. Every remaining chapter in Module 18 is a response to one of them.
| The number | What it forces | Chapter |
|---|---|---|
| 1.143× the line rate in memory bandwidth | a system allocation, not a MAC change | the architect's |
| 33.3% of it is descriptors | batched fetch and batched writeback | Chapter 18.2, Chapter 18.3 |
| 1.79 transactions per cycle at 100 Gb/s | fewer transactions per frame | Chapter 18.2 onward |
| 297.6% of a CPU at 1 Gb/s | coalescing, with both terms | Chapter 18.6 |
Chapter 18.2 — Descriptor Rings and the Ownership Model takes the first two. A ring is how the MAC and the driver hand buffers back and forth without a lock, and the handoff is one bit in a descriptor.
And that bit turns out to be the module's hardest problem, for a reason this chapter has set up. The ownership bit is a memory location written by two agents in different clock domains and different coherency domains — the MAC through Section 5's bus master, the driver through the CPU's cache. So the question is not what the bit means; it is whether the other fields of the descriptor are visible when the bit is.
Section 19's class 74 is the verification half of that question and Chapter 18.2 is the design half. A property relating two domains is unevaluable; a memory system that reorders two writes makes a descriptor whose ownership bit is set and whose length field is stale — and the MAC did nothing wrong in either case. The order of the writes against the ownership write is the whole correctness argument, and the barrier that enforces it is the chapter's subject.
Chapter 18.3 — The Receive DMA Path then traces a frame from Section 9's FIFO into host buffers: buffer exhaustion, scatter-gather across pages, and the write-completion ordering that decides when the ownership handoff is allowed to happen. It also prices the two jitter sources Chapter 16.1 §8 listed and left unpriced — DMA arbitration and interrupt coalescing — which this chapter's Sections 10 and 12 have finally made computable.
Continue learning
Related tutorials
- Related topic
The MAC/PHY Boundary in RTL
What five chapters described as one boundary is three in silicon: a data boundary at the port list, a clock boundary inside the elastic buffer, and a reset boundary that is an order rather than a place. Confusing any two produces a specific, recognisable integration failure.
- Related topic
MII — The Original Media Independent Interface
Sixteen signals, four bits each way, and both clocks sourced by the PHY. Its defining failure is a transmit clock that stops — and every assertion sampled by that clock is green while it does.
- Related topic
Descriptor Rings and the Ownership Model
A ring hands buffers between a DMA engine and a driver with one bit and no lock — provided the descriptor's other fields are visible before that bit is, which no memory system promises.
- Related topic
The Receive DMA Path
A received frame crosses seven boundaries between the MAC's FIFO and a host buffer, and the last two contribute 700 ns and 100 microseconds of jitter to a 24.2 ns clock.
Standards & specifications
- Governing standard
- IEEE Std 802.3 (Ethernet)(opens IEEE in a new tab)
Defines the Ethernet MAC, the media-independent interfaces and the physical-layer sublayers, including framing, access control, auto-negotiation and per-rate PHY specifications. VLAN tagging, priority and time-sensitive shaping are defined by IEEE 802.1, not by 802.3.
This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.
Where this fits
Part of the Ethernet curriculum.
