Ethernet · Module 14
Where Frames Are Lost
One congested port consumes a 12 MiB shared pool in 4.4 ms, and then every other port's next frame fails to allocate. The drop lands where the allocation failed, not where the congestion is.
Chapter 12.1 §6 established the arithmetic and then set it aside. Twenty-three ports offering line rate to one port offer 23 Gb/s to a 1 Gb/s wire; 95.7% must be discarded, and that is not a defect but the switch's specified response.
What that chapter did not ask is where the discard happens — and the answer is not the port everybody names.
On a modern switch the egress queues are not separate memories. They are lists of pointers into one shared pool, and a port's "buffer" is an accounting entry rather than a partition. So a congested port does not fill its own buffer. It consumes the pool — and at 23:1 oversubscription a 12 MiB pool is gone in 4.4 milliseconds.
After which every other port's next frame fails to allocate.
The drop is recorded on whichever port was trying to enqueue when the pool ran out, which is any port at all. The congestion is on the port holding the pool. Those are different ports, and nothing in a drop counter connects them.
1. Scope — What This Chapter Owns
This chapter owns where frames are actually lost: the four candidate places, why the egress queue is where the backlog builds, how a shared pool changes the answer, what queue delay actually is, how long a buffer buys, and why a drop counter does not localise congestion on its own.
It does not own the decision to discard — Chapter 12.1 §6 and §9 established that discarding is the specified response and built the tail-drop policy. This chapter asks where the discard physically happens.
It does not own the mechanism that avoids it — Chapter 14.2 owns the PAUSE frame, Chapter 14.3 owns what backpressure does when it propagates, and Chapter 14.4 owns per-priority flow control.
And it does not own scheduling. Chapter 13.4 §11's class scheduler decides which queue transmits; this chapter asks what happens when none of them can, because there is nowhere to put the next arrival.
2. Four Places a Frame Can Be Lost
Trace a frame from one station to another and count the buffers it must fit into. There are four, and only one of them is the one people mean by "the switch dropped it".
| # | Buffer | Sized in | Overflows when | Reported by |
|---|---|---|---|---|
| 1 | ingress FIFO | a few frames | the fabric cannot keep up | the switch, rarely |
| 2 | the shared pool | megabytes | any port is congested for long enough | the switch — on the wrong port |
| 3 | egress queue | pointers into the pool | its share is exhausted | the switch |
| 4 | the receiving host's ring | hundreds of descriptors | the host cannot keep up | the host, if anybody looks |
Buffer 1 is almost never the answer, and the reason is Chapter 12.5 §14's arithmetic: the forwarding path is sized to sustain line rate on every port simultaneously — 35.71 Mpps on a 24-port gigabit switch — so an ingress FIFO that overflows means the lookup is missing its deadline, which is Chapter 12.3 §11's problem rather than a congestion one.
Buffers 2 and 3 are the same memory seen two ways, and separating them is most of this chapter.
And buffer 4 is the one nobody counts. A host whose interrupt handler cannot keep up drops frames in its descriptor ring — and Chapter 12.4 §8 showed a broadcast storm puts every host in exactly that state at 7.4 cores of load. The network is not dropping those frames; the host is, and the switch's counters are clean throughout.
3. RTL 1 — The Queue That Builds
One egress port, one queue, and the arithmetic that decides how fast it fills.
// -----------------------------------------------------------------------
// congest_pkg -- shared types for congestion and flow control.
// -----------------------------------------------------------------------
package congest_pkg;
// Why an enqueue failed. The distinction between the first two is this
// chapter's subject: one is this port's problem and one is somebody
// else's, and a single "drop" counter merges them.
typedef enum logic [2:0] {
EF_OK = 3'd0,
EF_QUEUE_LIMIT = 3'd1, // this port has reached its own limit
EF_POOL_EMPTY = 3'd2, // the SHARED pool is exhausted -- Section 8
EF_NO_RESERVE = 3'd3, // below reserve and the reserve is spent
EF_PORT_DOWN = 3'd4
} enq_fail_e;
// Where a buffer's occupancy sits relative to its thresholds. Chapter
// 14.2's PAUSE and Chapter 14.4's per-priority pause both trigger on
// these rather than on "full".
typedef enum logic [1:0] {
WM_EMPTY = 2'd0,
WM_LOW = 2'd1, // below the resume threshold
WM_HIGH = 2'd2, // above the assert threshold
WM_FULL = 2'd3
} watermark_e;
localparam int CELL_OCTETS = 128; // allocation granularity
endpackage// -----------------------------------------------------------------------
// egress_queue_model -- one port's queue, its watermarks, and the rate at
// which it fills.
//
// Chapter 12.1 Section 9 built the tail-drop policy on this queue. What
// that chapter treated as a fixed depth is, on a shared-buffer switch, a
// claim against a pool -- which is Section 5's subject and the reason
// this module reports its occupancy in CELLS rather than octets.
// -----------------------------------------------------------------------
module egress_queue_model
import congest_pkg::*;
#(
parameter int MAX_CELLS = 4096, // this port's limit, not its memory
parameter int HIGH_CELLS = 3072,
parameter int LOW_CELLS = 1024,
parameter int CNT_W = 32
)(
input logic clk,
input logic rst_n,
input logic enq_valid,
input logic [7:0] enq_cells, // frame size in cells
input logic pool_granted, // Section 5 said yes
input logic deq_valid,
input logic [7:0] deq_cells,
output logic enq_accept,
output enq_fail_e enq_fail,
output logic [15:0] occupancy_cells,
output logic [15:0] peak_cells,
output watermark_e watermark,
output logic above_high,
output logic below_low,
output logic [CNT_W-1:0] c_enq,
output logic [CNT_W-1:0] c_drop_limit,
output logic [CNT_W-1:0] c_drop_pool,
output logic [CNT_W-1:0] c_cells_dropped
);
always_comb begin
enq_accept = 1'b0;
enq_fail = EF_OK;
if (enq_valid) begin
// THE TWO REFUSALS, KEPT APART. Reaching this port's own limit is
// this port's congestion. Failing to get a cell from the pool is
// somebody else's congestion arriving here -- Section 10.
if ((occupancy_cells + 16'(enq_cells)) > 16'(MAX_CELLS))
enq_fail = EF_QUEUE_LIMIT;
else if (!pool_granted)
enq_fail = EF_POOL_EMPTY;
else
enq_accept = 1'b1;
end
end
always_comb begin
if (occupancy_cells == 16'd0) watermark = WM_EMPTY;
else if (occupancy_cells >= 16'(MAX_CELLS)) watermark = WM_FULL;
else if (occupancy_cells >= 16'(HIGH_CELLS)) watermark = WM_HIGH;
else if (occupancy_cells <= 16'(LOW_CELLS)) watermark = WM_LOW;
else watermark = WM_LOW;
end
// Chapter 14.2's PAUSE and Chapter 14.4's per-priority pause both
// trigger on the HIGH watermark and release on the LOW one, never on
// "full" -- by the time a queue is full the frames that would have
// been paused have already arrived.
assign above_high = (occupancy_cells >= 16'(HIGH_CELLS));
assign below_low = (occupancy_cells <= 16'(LOW_CELLS));
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
occupancy_cells <= '0;
peak_cells <= '0;
c_enq <= '0;
c_drop_limit <= '0;
c_drop_pool <= '0;
c_cells_dropped <= '0;
end else begin
if (enq_valid && enq_accept) begin
occupancy_cells <= occupancy_cells + 16'(enq_cells);
c_enq <= c_enq + 1'b1;
if ((occupancy_cells + 16'(enq_cells)) > peak_cells)
peak_cells <= occupancy_cells + 16'(enq_cells);
end else if (enq_valid) begin
c_cells_dropped <= c_cells_dropped + CNT_W'(enq_cells);
if (enq_fail == EF_QUEUE_LIMIT) begin
if (!(&c_drop_limit)) c_drop_limit <= c_drop_limit + 1'b1;
end else if (enq_fail == EF_POOL_EMPTY) begin
if (!(&c_drop_pool)) c_drop_pool <= c_drop_pool + 1'b1;
end
end
if (deq_valid) occupancy_cells <= occupancy_cells - 16'(deq_cells);
end
end
endmoduleClassification: synthesizable.
What it teaches: that c_drop_limit and c_drop_pool must be separate counters, and that merging them into a single "drops" figure destroys the only information that says whose congestion this is. Reaching this port's own limit means this port is oversubscribed — Chapter 12.1 §6's arithmetic, and the remedy is capacity here. Failing to obtain a cell from the pool means some other port is oversubscribed and has consumed the shared memory, and the remedy is somewhere else entirely.
And it teaches why the watermarks are at 75% and 25% rather than at "full". Chapter 14.2's PAUSE and Chapter 14.4's per-priority pause both trigger on the high watermark, because a mechanism that waits for full has already lost the frames it was meant to protect — the ones in flight during the signal's propagation, which Chapter 14.2 quantifies.
Deliberately simplified: occupancy in fixed 128-octet cells rather than octets. Cell allocation is what real switches do — a 64-octet frame and a 128-octet frame both consume one cell — and it means a queue's occupancy in cells and its occupancy in useful octets differ by up to 2× on small frames, which is a real and frequently forgotten inefficiency.
Production implication: peak_cells against MAX_CELLS is the number that distinguishes a buffer that is adequately sized from one that has never been tested. A port whose peak occupancy has never exceeded 10% of its limit has a buffer that has never been exercised — which is not evidence it is big enough, only that nothing has yet asked. A port whose peak equals its limit has been dropping, and c_drop_limit says how often.
4. Why the Backlog Builds at the Egress
Congestion in a switched network has exactly one shape, and naming it precisely rules out most of the places people look for it.
A queue builds where the arrival rate exceeds the service rate, for as long as it does. In a switch there is only one place that can happen: the egress port, because that is the only point where several sources converge on one server.
| Point | Arrivals | Service | Can a backlog build |
|---|---|---|---|
| ingress port | one link | the fabric, sized for line rate | no — Chapter 12.5 §14 |
| the lookup | one per frame | 28 ns budget | no — a miss floods rather than queues |
| the fabric | N ports | N ports | no, if it is non-blocking |
| the egress port | up to N−1 sources | one wire | yes — and this is the only one |
Everything above the egress row is sized to keep up by construction. Chapter 12.1 §12 established the frame-rate budget and Chapter 12.5 §14 showed the table meets it with margin. A backlog at any of them is a design failing to meet its own specification, not congestion.
And the egress row cannot be sized to keep up, because the arrival rate is not bounded by anything the switch controls. Chapter 12.1 §6's 23:1 is the limit case, and no fabric, buffer or scheduler changes the arithmetic.
Which gives the chapter's first firm conclusion: congestion is an egress phenomenon, and every other buffer in Section 2's list either does not fill or fills for a different reason.
5. RTL 2 — The Shared Pool
And here is where the simple picture breaks. A port's queue is a list of pointers, and the memory those pointers reference belongs to everybody.
// -----------------------------------------------------------------------
// buffer_pool_manager -- one memory, many ports, three regions.
//
// The three regions are the whole design. A per-port RESERVE that only
// that port may use; a SHARED region any port may draw from; and a
// per-port LIMIT on how much of the shared region one port may hold. The
// sum of the reserves plus the shared region is the pool; the sum of the
// LIMITS deliberately exceeds it -- Section 18's rejected property.
// -----------------------------------------------------------------------
module buffer_pool_manager
import congest_pkg::*;
#(
parameter int N_PORTS = 24,
parameter int POOL_CELLS = 98304, // 12 MiB at 128 octets per cell
parameter int RESERVE_CELLS = 512, // per port, guaranteed
parameter int LIMIT_CELLS = 24576, // per port, of the SHARED region
parameter int CNT_W = 32
)(
input logic clk,
input logic rst_n,
input logic req_valid,
input logic [4:0] req_port,
input logic [7:0] req_cells,
input logic rel_valid,
input logic [4:0] rel_port,
input logic [7:0] rel_cells,
output logic grant,
output enq_fail_e fail,
output logic [17:0] pool_free,
output logic [17:0] shared_free,
output logic [17:0] held [N_PORTS],
output logic [4:0] largest_holder,
output logic [15:0] largest_share_pct,
output logic pool_exhausted,
output logic [CNT_W-1:0] c_grant,
output logic [CNT_W-1:0] c_refuse_limit,
output logic [CNT_W-1:0] c_refuse_pool,
output logic [CNT_W-1:0] overcommit_ratio_x100
);
localparam int RESERVED_TOTAL = N_PORTS * RESERVE_CELLS;
localparam int SHARED_TOTAL = POOL_CELLS - RESERVED_TOTAL;
logic [17:0] shared_used;
// A port's reserve is spent first and belongs to nobody else. Only what
// it holds ABOVE the reserve comes from the shared region.
function automatic logic [17:0] shared_held(input logic [17:0] h);
shared_held = (h > 18'(RESERVE_CELLS)) ? (h - 18'(RESERVE_CELLS)) : 18'd0;
endfunction
always_comb begin
automatic logic [17:0] h = held[req_port];
automatic logic [17:0] sh = shared_held(h);
grant = 1'b0;
fail = EF_OK;
if (req_valid) begin
if ((h + 18'(req_cells)) <= 18'(RESERVE_CELLS)) begin
// WITHIN THE RESERVE. Always granted -- this is the guarantee
// that makes the shared region safe to over-commit.
grant = 1'b1;
end else if ((sh + 18'(req_cells)) > 18'(LIMIT_CELLS)) begin
// This port has taken as much of the shared region as it is
// allowed. Its congestion is now its own problem, which is the
// entire purpose of the limit.
fail = EF_QUEUE_LIMIT;
end else if ((shared_used + 18'(req_cells)) > 18'(SHARED_TOTAL)) begin
// THE SHARED REGION IS GONE. Somebody else took it -- Section 8.
fail = EF_POOL_EMPTY;
end else begin
grant = 1'b1;
end
end
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int p = 0; p < N_PORTS; p++) held[p] <= '0;
shared_used <= '0;
c_grant <= '0;
c_refuse_limit <= '0;
c_refuse_pool <= '0;
largest_holder <= '0;
largest_share_pct <= '0;
pool_exhausted <= 1'b0;
end else begin
if (req_valid && grant) begin
held[req_port] <= held[req_port] + 18'(req_cells);
if (held[req_port] >= 18'(RESERVE_CELLS))
shared_used <= shared_used + 18'(req_cells);
c_grant <= c_grant + 1'b1;
end else if (req_valid) begin
if (fail == EF_QUEUE_LIMIT) begin
if (!(&c_refuse_limit)) c_refuse_limit <= c_refuse_limit + 1'b1;
end else begin
if (!(&c_refuse_pool)) c_refuse_pool <= c_refuse_pool + 1'b1;
end
end
if (rel_valid) begin
held[rel_port] <= held[rel_port] - 18'(rel_cells);
if (held[rel_port] > 18'(RESERVE_CELLS))
shared_used <= shared_used - 18'(rel_cells);
end
// WHO IS HOLDING THE POOL. This is the number that turns a drop on
// port 3 into a finding about port 7 -- Section 10.
begin
automatic logic [17:0] hi = '0;
automatic logic [4:0] hp = '0;
for (int p = 0; p < N_PORTS; p++)
if (held[p] > hi) begin hi = held[p]; hp = 5'(p); end
largest_holder <= hp;
largest_share_pct <= 16'((hi * 18'd100) / 18'(POOL_CELLS));
end
pool_exhausted <= (shared_used >= 18'(SHARED_TOTAL));
end
end
assign shared_free = 18'(SHARED_TOTAL) - shared_used;
assign pool_free = shared_free;
// THE OVER-COMMITMENT, COMPUTED. The sum of what every port is allowed
// to take, against what exists. A value above 100 is the design working
// as intended -- Section 18.
assign overcommit_ratio_x100 =
CNT_W'((((N_PORTS * (RESERVE_CELLS + LIMIT_CELLS)) * 100) / POOL_CELLS));
endmoduleClassification: synthesizable.
What it teaches: that the three regions are the whole design and each does a different job. The reserve is what makes the shared region safe to over-commit — every port can always enqueue something, so an exhausted pool degrades throughput rather than disconnecting a port. The shared region is where the value is: most ports are idle most of the time, and sharing their unused memory with whichever port is busy is the reason a pool exists at all. And the limit is what stops one port taking everything.
And overcommit_ratio_x100 above 100 is the design working. With 24 ports, a 512-cell reserve and a 24 576-cell limit each, the sum of allowances is 24 × 25 088 = 602 112 cells against a 98 304-cell pool — a ratio of 612%. That is not a bug and it is not sloppy accounting. It is the entire economic argument for a shared buffer: the sum of what everybody may take exceeds what exists, because they will not all take it at once.
Deliberately simplified: a single shared region with one limit per port. Production designs make the limit dynamic — a port's allowance is a fraction of the currently free pool rather than a constant — so a port congested while the pool is empty gets far less than one congested while it is idle. That is strictly better and it makes the accounting considerably harder to reason about.
Production implication: largest_holder and largest_share_pct are the pair that make Section 10 possible. A drop on port 3 with largest_holder = 7 at 85% is a complete diagnosis — port 3 could not allocate because port 7 is holding the pool — and neither half of it is available from a per-port drop counter.
6. The Shared Pool, and Who Owns It
Put numbers on a 24-port switch with a 12 MiB pool and the over-commitment becomes concrete.
| Quantity | Cells | Octets |
|---|---|---|
| the pool | 98 304 | 12 MiB |
| per-port reserve, 24 ports | 24 × 512 = 12 288 | 1.5 MiB |
| the shared region | 86 016 | 10.5 MiB |
| per-port limit on the shared region | 24 576 | 3 MiB |
Σ (reserve + limit) over 24 ports | 602 112 | 73.5 MiB |
| over-commitment | — | 612% |
Six times more memory is promised than exists, and the design depends on it. A pool sized so that every port could simultaneously take its full allowance would need 73.5 MiB — six times the silicon, to serve a case that occurs when every port on the switch is congested at once, at which point the switch has a capacity problem no buffer solves.
And the limit is doing real work. Without it, one port could take the entire 86 016-cell shared region:
| Per-port limit | One port's maximum hold | What the other 23 keep |
|---|---|---|
| none | 86 016 cells — the whole shared region | their reserves only: 512 cells each |
| 24 576 | 24 576 — 28.6% of the shared region | 61 440 cells to share |
| 8 192 | 8 192 — 9.5% | 77 824 cells |
With no limit, a single congested port reduces every other port to its 512-cell reserve — 64 KiB, which at 2:1 oversubscription absorbs 512 × 128 × 8 ÷ 1e9 = 0.52 ms and then drops.
7. RTL 3 — Who Is Holding the Pool
A drop says a frame could not be enqueued. It does not say why the memory was gone, and the answer is a different port.
// -----------------------------------------------------------------------
// shared_pool_accountant -- attributes pool exhaustion to the port that
// caused it, which is not the port that suffered it.
//
// Chapter 12.1 Section 11 made the same argument one level down: c_drops
// says an egress port is oversubscribed, and drops_by_ingress says which
// conversation is doing it. This is that argument applied to the pool.
// -----------------------------------------------------------------------
module shared_pool_accountant
import congest_pkg::*;
#(
parameter int N_PORTS = 24,
parameter int CNT_W = 32,
parameter int WIN = 1_000_000
)(
input logic clk,
input logic rst_n,
input logic [17:0] held [N_PORTS],
input logic [17:0] shared_total,
input logic pool_exhausted,
input logic drop_valid,
input logic [4:0] drop_port,
input enq_fail_e drop_reason,
output logic [CNT_W-1:0] drops_suffered [N_PORTS],
output logic [CNT_W-1:0] pool_drops_caused [N_PORTS],
output logic [4:0] victim_port,
output logic [4:0] culprit_port,
output logic victim_is_not_culprit,
output logic [15:0] culprit_share_pct,
output logic window_valid
);
logic [CNT_W-1:0] win;
// WHO HELD THE POOL AT THE MOMENT OF THE DROP. Sampling this at the
// drop rather than periodically is what makes the attribution sound --
// a periodic sample tells you who is usually busy, not who was holding
// the memory the instant a frame was refused.
logic [4:0] hi_port;
logic [17:0] hi_held;
always_comb begin
hi_port = 5'd0;
hi_held = 18'd0;
for (int p = 0; p < N_PORTS; p++)
if (held[p] > hi_held) begin hi_held = held[p]; hi_port = 5'(p); end
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int p = 0; p < N_PORTS; p++) begin
drops_suffered[p] <= '0;
pool_drops_caused[p] <= '0;
end
win <= '0;
victim_port <= '0;
culprit_port <= '0;
victim_is_not_culprit <= 1'b0;
culprit_share_pct <= '0;
window_valid <= 1'b0;
end else begin
window_valid <= 1'b0;
if (drop_valid) begin
drops_suffered[drop_port] <= drops_suffered[drop_port] + 1'b1;
win <= win + 1'b1;
// THE ATTRIBUTION. A drop caused by an exhausted POOL is charged
// to whoever is holding it, not to the port that could not
// allocate. A drop caused by this port's own LIMIT is charged
// here, because that is this port's congestion.
if (drop_reason == EF_POOL_EMPTY) begin
pool_drops_caused[hi_port] <= pool_drops_caused[hi_port] + 1'b1;
victim_port <= drop_port;
culprit_port <= hi_port;
victim_is_not_culprit <= (drop_port != hi_port);
culprit_share_pct <= 16'((hi_held * 18'd100) / shared_total);
end
end
if (win >= CNT_W'(WIN)) begin
win <= '0;
window_valid <= 1'b1;
end
end
end
endmoduleClassification: synthesizable.
What it teaches: that the attribution must be sampled at the moment of the drop, and a periodic sample is a different and much weaker measurement. A periodic sample says who is usually busy. Sampling held[] at the instant a frame is refused says who was holding the memory that frame needed — and on a switch where several ports take turns being congested, those give different answers.
And victim_is_not_culprit is the bit worth exposing on its own. It is high whenever a port dropped because of somebody else, which is the condition that makes a per-port drop counter misleading rather than merely incomplete. With it low, a drop on port 3 means port 3 is oversubscribed. With it high, port 3 is a bystander.
Deliberately simplified: a combinational maximum over 24 holdings, evaluated every cycle. A production accountant maintains the maximum incrementally, because a 24-way comparison tree on the allocation path is not free at the rates Chapter 12.1 §12 established.
Production implication: pool_drops_caused is the counter that changes what an operator does, and it inverts the usual reading. A port with a large drops_suffered and a small pool_drops_caused is a victim — investigating its traffic finds nothing wrong. A port with a small drops_suffered and a large pool_drops_caused is the problem, and it may be showing no drops of its own at all, because it is winning every allocation.
8. One Congested Port Can Starve Twenty-Three
Do the arithmetic on how fast a single oversubscribed port consumes the shared region, because the answer is measured in milliseconds.
A port receiving 23:1 oversubscription — Chapter 12.1 §6's limit case — is offered 23 Gb/s and drains 1 Gb/s, so it accumulates at 22 Gb/s.
| Shared region | Time for one port to consume it at 22 Gb/s excess |
|---|---|
| 10.5 MiB — no per-port limit | 4.0 ms |
| 3 MiB — with the 24 576-cell limit | 1.1 ms |
And what the other 23 ports have left, in each case:
| other ports' available memory | absorbs 2:1 oversubscription for | |
|---|---|---|
| no limit | 512 cells — 64 KiB each | 0.52 ms |
| with the limit | 61 440 ÷ 23 = 2671 cells — 334 KiB each | 2.7 ms |
A factor of five in burst tolerance, from one comparison in the allocation path.
And the timescales are what make this hard to catch. 4 milliseconds is far below the sampling interval of any management system — a pool that is exhausted, drops frames on every port, and recovers, all between two SNMP polls — while the counters it left behind say only that several ports dropped frames at some point in the last five minutes.
9. RTL 4 — Attributing a Drop
Five distinct reasons a frame is discarded, and a single counter for all of them is the commonest instrumentation failure in a switch.
// -----------------------------------------------------------------------
// drop_attribution -- every discard, classified and attributed.
//
// Chapter 12.3 Section 10's deciding_gate did this for the forwarding
// decision. This does it for the buffer, and the reasons are unrelated
// to each other in exactly the same way.
// -----------------------------------------------------------------------
module drop_attribution
import congest_pkg::*;
#(
parameter int N_PORTS = 24,
parameter int CNT_W = 32,
parameter int WIN = 1_000_000
)(
input logic clk,
input logic rst_n,
input logic drop_valid,
input enq_fail_e reason,
input logic [4:0] egress_port,
input logic [4:0] ingress_port,
input logic [4:0] pool_holder,
output logic [CNT_W-1:0] c_by_reason [5],
output logic [CNT_W-1:0] c_by_egress [N_PORTS],
output logic [CNT_W-1:0] c_by_ingress [N_PORTS],
output logic window_valid,
output logic [15:0] pool_drop_share_pct,
output logic mostly_somebody_else,
output logic [4:0] dominant_ingress,
output logic [4:0] dominant_holder
);
logic [CNT_W-1:0] win, win_pool;
logic [CNT_W-1:0] holder_hits [N_PORTS];
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int i = 0; i < 5; i++) c_by_reason[i] <= '0;
for (int p = 0; p < N_PORTS; p++) begin
c_by_egress[p] <= '0; c_by_ingress[p] <= '0; holder_hits[p] <= '0;
end
win <= '0; win_pool <= '0;
window_valid <= 1'b0;
pool_drop_share_pct <= '0;
mostly_somebody_else<= 1'b0;
dominant_ingress <= '0;
dominant_holder <= '0;
end else begin
window_valid <= 1'b0;
if (drop_valid) begin
c_by_reason[reason[2:0]] <= c_by_reason[reason[2:0]] + 1'b1;
// THREE ATTRIBUTIONS PER DROP, and they answer three questions.
// Egress: where did it fail. Ingress: whose traffic was it.
// Holder: whose memory was it waiting for.
c_by_egress[egress_port] <= c_by_egress[egress_port] + 1'b1;
c_by_ingress[ingress_port] <= c_by_ingress[ingress_port] + 1'b1;
win <= win + 1'b1;
if (reason == EF_POOL_EMPTY) begin
win_pool <= win_pool + 1'b1;
holder_hits[pool_holder] <= holder_hits[pool_holder] + 1'b1;
end
end
if (win >= CNT_W'(WIN)) begin
automatic logic [CNT_W-1:0] hi = '0;
automatic logic [4:0] hp = '0;
automatic logic [CNT_W-1:0] hi2 = '0;
automatic logic [4:0] hp2 = '0;
for (int p = 0; p < N_PORTS; p++) begin
if (holder_hits[p] > hi) begin hi = holder_hits[p]; hp = 5'(p); end
if (c_by_ingress[p] > hi2) begin hi2 = c_by_ingress[p]; hp2 = 5'(p); end
end
pool_drop_share_pct <= 16'((win_pool * CNT_W'(100)) / win);
// MORE THAN HALF OF THIS PORT'S DROPS WERE SOMEBODY ELSE'S FAULT.
// The single most useful bit in the module.
mostly_somebody_else <= (win_pool > (win >> 1));
dominant_holder <= hp;
dominant_ingress <= hp2;
win <= '0; win_pool <= '0;
for (int p = 0; p < N_PORTS; p++) holder_hits[p] <= '0;
window_valid <= 1'b1;
end
end
end
endmoduleClassification: synthesizable.
What it teaches: that one drop deserves three attributions, and they answer three unrelated questions. Where did the enqueue fail is the egress port. Whose traffic was refused is the ingress port — Chapter 12.1 §11's drops_by_ingress. And whose memory was it waiting for is the pool holder, which exists only on a shared-buffer switch and has no analogue in that chapter.
And mostly_somebody_else is the derived bit that changes an investigation's direction. Above half, this port's drop counter is not about this port — it is a symptom of Section 8's pool exhaustion, and every minute spent examining this port's traffic is wasted.
Deliberately simplified: three flat counter arrays. Production instrumentation keeps a small two-dimensional histogram of (egress, holder) pairs, because the useful finding is often port 3's drops are 90% attributable to port 7 rather than either port's total.
Production implication: c_by_reason has five entries and a switch that exposes one "discards" figure has summed them. EF_QUEUE_LIMIT and EF_POOL_EMPTY are the two that matter and they point in opposite directions — the first says this port is oversubscribed and needs capacity, the second says this port is fine and somebody else is holding the memory. A summed counter cannot distinguish add a faster uplink here from find the port that is hoarding, and those are the two most common remedies.
10. Where a Frame Is Lost Is Not Where the Congestion Is
Put Sections 7 and 8 together and state the conclusion plainly, because it inverts the reading of the most-watched counter on a switch.
Three questions, three answers, and on a shared-buffer switch they are frequently three different ports:
| Question | Answer | Available from |
|---|---|---|
| where did the enqueue fail | the egress port that was refused | a per-port drop counter |
| whose traffic was refused | the ingress port the frame came from | Chapter 12.1 §11's drops_by_ingress |
| whose memory was it waiting for | the port holding the pool | largest_holder — Section 5 |
A per-port drop counter answers only the first, and the first is the least useful of the three.
The pattern this produces is worth recognising by shape:
| Observation | What it usually means |
|---|---|
drops on one port, EF_QUEUE_LIMIT dominant | that port is oversubscribed — capacity, here |
drops on many ports, EF_POOL_EMPTY dominant | one port is holding the pool — find it |
| drops on many ports, no single holder | genuine switch-wide overload |
| no drops on the busiest port, drops elsewhere | the busiest port is the culprit |
The last row is the one that reads as impossible and is the normal case. A port holding the pool is winning its allocations — it already has the memory, so its enqueues succeed until its own limit stops them, and those refusals are counted as EF_QUEUE_LIMIT rather than as pool exhaustion. Meanwhile every other port's EF_POOL_EMPTY drops accumulate.
So the port causing the problem can genuinely show fewer drops than its victims, and an investigation ranked by drop count starts at the wrong end.
11. RTL 5 — Estimating Queue Delay
Occupancy is a number of cells. What anybody actually wants to know is how long a frame will wait, and Little's law converts one into the other.
// -----------------------------------------------------------------------
// queue_delay_estimator -- occupancy to latency, in hardware.
//
// L = lambda * W, so W = L / lambda: the wait a frame joining the queue
// now will experience is the current occupancy divided by the drain rate.
// It is exact for a work-conserving server and it is the number an
// application cares about, which occupancy is not.
// -----------------------------------------------------------------------
module queue_delay_estimator
import congest_pkg::*;
#(
parameter int LINK_MBPS = 1000,
parameter int CNT_W = 32
)(
input logic clk,
input logic rst_n,
input logic [15:0] occupancy_cells,
input logic sample,
output logic [19:0] delay_ns,
output logic [19:0] peak_delay_ns,
output logic [19:0] p99_delay_ns, // coarse, from a histogram
output logic delay_over_1ms,
output logic [CNT_W-1:0] hist [8] // log2 buckets of delay
);
// Cells to bits, bits to nanoseconds. At 128 octets per cell and
// 1000 Mb/s: one cell is 1024 bits and drains in 1024 ns.
localparam int NS_PER_CELL = (CELL_OCTETS * 8 * 1000) / LINK_MBPS;
assign delay_ns = 20'(occupancy_cells) * 20'(NS_PER_CELL);
// A log2 histogram rather than a mean. Chapter 12.6 Section 6
// established that jitter is what real-time traffic is engineered
// against, and a mean delay hides the tail entirely.
function automatic logic [2:0] bucket(input logic [19:0] d);
if (d < 20'd1000) bucket = 3'd0; // < 1 us
else if (d < 20'd4000) bucket = 3'd1;
else if (d < 20'd16000) bucket = 3'd2;
else if (d < 20'd64000) bucket = 3'd3;
else if (d < 20'd256000) bucket = 3'd4; // < 256 us
else if (d < 20'd512000) bucket = 3'd5;
else if (d < 20'd1000000) bucket = 3'd6; // < 1 ms
else bucket = 3'd7; // >= 1 ms
endfunction
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int i = 0; i < 8; i++) hist[i] <= '0;
peak_delay_ns <= '0;
p99_delay_ns <= '0;
delay_over_1ms <= 1'b0;
end else if (sample) begin
hist[bucket(delay_ns)] <= hist[bucket(delay_ns)] + 1'b1;
if (delay_ns > peak_delay_ns) peak_delay_ns <= delay_ns;
delay_over_1ms <= (delay_ns >= 20'd1000000);
// A coarse 99th percentile: the lowest bucket boundary above which
// fewer than one percent of samples fall. Computed by walking the
// histogram from the top, which is a handful of comparisons.
begin
automatic logic [CNT_W-1:0] tot = '0;
automatic logic [CNT_W-1:0] acc = '0;
for (int i = 0; i < 8; i++) tot = tot + hist[i];
for (int i = 7; i >= 0; i--) begin
acc = acc + hist[i];
if (acc > (tot / 100)) begin
p99_delay_ns <= 20'd1000 << (3 * i);
break;
end
end
end
end
end
endmoduleClassification: synthesizable; the percentile walk is a small sequential loop in a real design rather than the combinational one shown.
What it teaches: that occupancy and delay are the same measurement in different units, and only one of them is actionable. L ÷ λ converts cells into nanoseconds exactly — at 128 octets per cell and 1 Gb/s, one cell is 1024 ns — and "this queue holds 3000 cells" means nothing to an application while "a frame joining now waits 3.07 ms" is a service-level statement.
And it teaches why the histogram is log₂ rather than a mean. Chapter 12.6 §6 established that jitter, not mean latency, is what real-time traffic is engineered against — a receive buffer is sized against the worst case minus the best. A mean delay of 40 µs with a 3 ms tail and a flat 40 µs are the same number and completely different networks, and only the histogram separates them.
Deliberately simplified: a fixed drain rate. A real port's drain rate varies with frame size — Chapter 12.1 §12's interframe gap and preamble overhead means a queue of minimum-length frames drains slower in bits than one of maximum-length frames — so the estimate is optimistic by up to 31% on small frames.
Production implication: delay_over_1ms is a threshold worth a standing alert rather than a graph. A millisecond of queueing delay is longer than most datacentre round trips and longer than the timeout of several storage protocols — and Section 12's table shows it arrives at about 99% utilisation, which a five-minute average will report as a comfortable 70%.
12. Queue Delay, Measured
Queueing delay is not linear in utilisation. It is hyperbolic, and the whole of it happens in the last few percent.
For 1518-octet frames at 1 Gb/s, one frame's service time is 12.144 µs. Mean wait against utilisation:
| Utilisation | Mean wait | In frames |
|---|---|---|
| 50% | 12.1 µs | 1.0 |
| 70% | 28.3 µs | 2.3 |
| 90% | 109.3 µs | 9.0 |
| 95% | 230.7 µs | 19.0 |
| 99% | 1202.3 µs | 99.0 |
From 50% to 90% the delay grows 9×. From 90% to 99% it grows another 11× — and 99% utilisation is a link an operator would describe as nearly full rather than broken.
Which is the shape Chapter 12.6 §10 relied on and did not derive. That chapter argued cut-through's benefit vanishes under load because queueing dominates; here is the arithmetic: at 90% utilisation the queueing term is 109 µs against a 12.144 µs serialisation term — nine times larger — and cut-through attacks only the smaller one.
And it explains why averaging destroys the measurement. A link at 99% for six seconds and idle for the rest of a five-minute interval averages 2% utilisation and had a millisecond of queueing delay throughout those six seconds. Every frame crossing in that window was late; the graph shows an idle link.
13. RTL 6 — How Long a Buffer Buys
A buffer's size is usually quoted in bytes. What it is actually worth is a duration, and the conversion depends on the oversubscription ratio it is absorbing.
// -----------------------------------------------------------------------
// burst_absorption_model -- how long this buffer holds out.
//
// A buffer absorbs a RATE MISMATCH for a DURATION. Quoting its size in
// bytes hides both variables; quoting the duration it buys at a stated
// oversubscription ratio states them.
// -----------------------------------------------------------------------
module burst_absorption_model
import congest_pkg::*;
#(
parameter int LINK_MBPS = 1000,
parameter int CNT_W = 32
)(
input logic clk,
input logic rst_n,
input logic [15:0] free_cells,
input logic [5:0] oversub_ratio, // N:1, so excess = (N-1) x link
input logic sample,
output logic [19:0] absorb_us,
output logic [19:0] worst_absorb_us,
output logic under_100us,
output logic under_10us,
input logic fill_started,
input logic fill_ended,
output logic [19:0] last_burst_us,
output logic [19:0] longest_burst_us,
output logic [CNT_W-1:0] c_bursts,
output logic buffer_was_adequate
);
// Excess rate = (ratio - 1) x link. Time = bits / excess.
// At 128 octets per cell and 1000 Mb/s, one cell is 1024 bits and
// drains in 1.024 us at the link rate.
localparam int NS_PER_CELL = (CELL_OCTETS * 8 * 1000) / LINK_MBPS;
always_comb begin
if (oversub_ratio <= 6'd1) absorb_us = 20'hFFFFF; // no excess
else absorb_us = (20'(free_cells) * 20'(NS_PER_CELL)) /
(20'(oversub_ratio) - 20'd1) / 20'd1000;
end
assign under_100us = (absorb_us < 20'd100);
assign under_10us = (absorb_us < 20'd10);
logic [19:0] burst_ns;
logic in_burst;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
worst_absorb_us <= 20'hFFFFF;
burst_ns <= '0;
in_burst <= 1'b0;
last_burst_us <= '0;
longest_burst_us <= '0;
c_bursts <= '0;
buffer_was_adequate <= 1'b1;
end else begin
if (sample && (absorb_us < worst_absorb_us))
worst_absorb_us <= absorb_us;
// MEASURE THE BURSTS THAT ACTUALLY ARRIVE, not the ones the buffer
// was sized for. A buffer sized for 1 ms on a network whose bursts
// are 20 us is oversized by 50x, and nothing else says so.
if (fill_started && !in_burst) begin
in_burst <= 1'b1;
burst_ns <= '0;
end else if (in_burst) begin
burst_ns <= burst_ns + 20'd8; // 8 ns per cycle at 125 MHz
if (fill_ended) begin
in_burst <= 1'b0;
last_burst_us <= burst_ns / 20'd1000;
if ((burst_ns / 20'd1000) > longest_burst_us)
longest_burst_us <= burst_ns / 20'd1000;
if (!(&c_bursts)) c_bursts <= c_bursts + 1'b1;
end
end
// The buffer was adequate if it absorbed every burst that arrived.
buffer_was_adequate <= (longest_burst_us < worst_absorb_us);
end
end
endmoduleClassification: synthesizable.
What it teaches: that a buffer's size is two variables collapsed into one number, and separating them is the whole design conversation. A buffer absorbs a rate mismatch for a duration, and size = excess_rate × duration. Quoting 512 KiB says neither; quoting 4.2 ms at 2:1 or 182 µs at 24:1 says both — and those are the same 512 KiB.
And longest_burst_us against worst_absorb_us is the only honest sizing evidence. A buffer sized for a millisecond on a network whose real bursts are 20 µs is oversized by 50×, paying Chapter 12.6 §13's reservation cost and Section 12's delay for headroom nothing uses. Measuring the bursts that actually arrive is the only way to know, and no datasheet figure substitutes.
Deliberately simplified: a fixed cycle time in the burst timer and integer division for the absorption estimate. A production model computes the estimate once per window rather than combinationally, since a 20-bit divide is not free.
Production implication: under_10us deserves an alert and under_100us deserves a graph. A buffer that will absorb less than 10 µs of the current oversubscription is one burst away from dropping, and 10 µs is roughly one maximum-length frame at 1 Gb/s — so the buffer has effectively no headroom left. The signal fires while the port is still passing traffic cleanly, which is the only useful time for it to fire.
14. What a Buffer Actually Buys, in Time
Convert every plausible buffer size into the duration it absorbs at three oversubscription ratios, because the durations are shorter than the sizes suggest.
| Buffer | 2:1 | 4:1 | 24:1 |
|---|---|---|---|
| 32 KiB | 262 µs | 87 µs | 11 µs |
| 122 KiB — Chapter 12.1 §6 | 999 µs | 333 µs | 44 µs |
| 512 KiB | 4.19 ms | 1.40 ms | 182 µs |
| 1 MiB | 8.39 ms | 2.80 ms | 365 µs |
Read the right-hand column. At Chapter 12.1 §6's limit case of 23 other ports offering line rate to one, a megabyte of buffer absorbs 365 microseconds — and then drops at 95.7%, exactly as that chapter's arithmetic said it would.
Which sets the honest expectation for what buffering achieves. It does not fix oversubscription; Chapter 12.1 §6 established that nothing does. What it does is absorb transients — a simultaneous burst from several sources that ends before the buffer fills — and the whole design question is whether real bursts are shorter than the buffer's duration.
And the left-hand column is the one worth reading against Section 12's delay table. A 1 MiB buffer at 2:1 absorbs 8.39 ms, and a frame arriving when it is full waits 8.39 ms to be transmitted. Those are the same number — Little's law — so a buffer sized to absorb an 8 ms burst has committed to an 8 ms tail latency for anything that arrives during one.
15. RTL 7 — Congestion Telemetry
Six mechanisms, and one place to read whether this switch is congested, who is causing it, and how close the buffer is to its limit.
// -----------------------------------------------------------------------
// congestion_telemetry -- the per-window state of the buffer subsystem.
//
// Every quantity is derived from signals the datapath already has. The
// value is that a single "discards" figure conflates six unrelated
// conditions, and each of the six has a different remedy.
// -----------------------------------------------------------------------
module congestion_telemetry
import congest_pkg::*;
#(
parameter int N_PORTS = 24,
parameter int CNT_W = 32,
parameter int WIN = 1_000_000
)(
input logic clk,
input logic rst_n,
input logic frame_valid,
input logic drop_valid,
input enq_fail_e drop_reason,
input logic [17:0] shared_free,
input logic [17:0] shared_total,
input logic [15:0] largest_share_pct,
input logic [19:0] p99_delay_ns,
input logic [19:0] worst_absorb_us,
input logic pool_exhausted,
output logic window_valid,
output logic [15:0] drop_rate_ppm,
output logic [15:0] pool_free_pct,
output logic [15:0] pool_exhausted_ppm,
output logic congested,
output logic one_port_dominant,
output logic delay_bound_exceeded,
output logic headroom_critical,
output logic [CNT_W-1:0] c_frames,
output logic [CNT_W-1:0] c_drops
);
logic [CNT_W-1:0] win, win_drop, win_exh;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
win <= '0; win_drop <= '0; win_exh <= '0;
c_frames <= '0; c_drops <= '0;
window_valid <= 1'b0;
drop_rate_ppm <= '0;
pool_free_pct <= 16'd100;
pool_exhausted_ppm <= '0;
congested <= 1'b0;
one_port_dominant <= 1'b0;
delay_bound_exceeded <= 1'b0;
headroom_critical <= 1'b0;
end else begin
window_valid <= 1'b0;
if (frame_valid) begin win <= win + 1'b1; c_frames <= c_frames + 1'b1; end
if (drop_valid) begin win_drop <= win_drop + 1'b1; c_drops <= c_drops + 1'b1; end
if (pool_exhausted) win_exh <= win_exh + 1'b1;
if (win >= CNT_W'(WIN)) begin
// PARTS PER MILLION, not percent. A drop rate of 0.05% is 500 ppm
// and matters; as an integer percentage it is zero.
drop_rate_ppm <= 16'((win_drop * CNT_W'(1_000_000)) / win);
pool_exhausted_ppm <= 16'((win_exh * CNT_W'(1_000_000)) / win);
pool_free_pct <= 16'((18'(shared_free) * 18'd100) / shared_total);
congested <= (win_drop != '0) || (win_exh != '0);
// ONE PORT HOLDING MORE THAN HALF THE POOL. Section 8's condition,
// as a bit.
one_port_dominant <= (largest_share_pct > 16'd50);
// A millisecond of queueing -- Section 12 puts this at about 99%
// utilisation, which a five-minute average reports as 70%.
delay_bound_exceeded <= (p99_delay_ns >= 20'd1000000);
// Less than ten microseconds of absorption left: roughly one
// maximum-length frame at 1 Gb/s.
headroom_critical <= (worst_absorb_us < 20'd10);
win <= '0; win_drop <= '0; win_exh <= '0;
window_valid <= 1'b1;
end
end
end
endmoduleClassification: synthesizable.
What it teaches: that the drop rate belongs in parts per million rather than percent, and the reason is the same one Chapter 13.2 §11 gave for scaling tag overhead by 10 000. A drop rate of 0.05% is 500 ppm and is a serious problem for a storage protocol; as an integer percentage it is zero — and a metric that reads zero in the range where it matters is a metric nobody reads.
And the four derived bits are four different conditions with four different remedies. congested says frames are being lost. one_port_dominant says one port is holding the pool — Section 8. delay_bound_exceeded says the queueing delay has passed a millisecond, which is a latency problem even if nothing is dropping. And headroom_critical says the next burst will drop, which fires while the port is still clean.
Deliberately simplified: one global window. Production telemetry keeps the window per port and the pool statistics globally, because the pool is one resource and the queues are twenty-four.
Production implication: delay_bound_exceeded catches the failure that has no drops at all. Section 12's table shows a millisecond of queueing arrives at about 99% utilisation — a link that is passing every frame, dropping nothing, and adding a millisecond to every round trip. No drop counter fires, no error is logged, and every latency-sensitive application on that path is failing its budget. It is the one condition in this chapter that a loss-oriented investigation never finds.
16. The Host Is a Buffer Too
Section 2 counted four buffers and this chapter has spent five sections on two of them. The fourth is the one that drops most frames on a healthy network, and no switch counter sees it.
A receiving station holds arriving frames in a descriptor ring — a few hundred entries, refilled by the driver as it consumes them. When the ring is full the network card discards, and the discard is counted in a driver statistic.
| switch buffer | host descriptor ring | |
|---|---|---|
| size | megabytes | hundreds of frames |
| at 1 Gb/s minimum frames | 12 MiB ≈ 98 000 frames | 512 frames — 344 µs |
| overflows because | offered load exceeds an egress rate | the CPU did not run |
| duration of the cause | milliseconds to indefinitely | microseconds |
| loss pattern | a predictable fraction, sustained | bursty and opportunistic |
| reported where | a management interface | a driver counter nobody polls |
The middle rows are the important ones. A switch drops because of an arithmetic condition that persists — Chapter 12.1 §6's rate mismatch. A host drops because of a scheduling condition that lasts microseconds — an interrupt coalescing timer, a scheduler delay, a page fault in a driver — and then clears.
And Chapter 12.4 §8 established the one case where the two meet. A broadcast storm puts every host at 7.4 cores of interrupt load, so every host's ring overflows continuously — while the switch, forwarding correctly at 6.7% link utilisation, reports nothing at all.
Which is why an investigation should establish which buffer before examining any of them, and the establishing question is cheap: does the receiver's driver report ring-full discards? If yes, the network is delivering more than the host can take and no switch change helps. If no, and frames are still missing, the loss is upstream — and Sections 9 and 10 say how to find which port caused it.
17. RTL 8 — Conformance for a Shared Resource
The monitor's difficulty is that the property it most needs to check — that the pool is not over-committed — is one the design deliberately violates.
// -----------------------------------------------------------------------
// congestion_conformance_monitor -- checks a buffer subsystem whose
// design premise is that it promises more than it has.
//
// The invariant is NOT that allocations sum to the pool. It is that the
// RESERVES do, that no port exceeds its limit, that accounting balances,
// and that every drop is attributed. Section 18 is about the difference.
// -----------------------------------------------------------------------
module congestion_conformance_monitor
import congest_pkg::*;
#(
parameter int N_PORTS = 24,
parameter int POOL_CELLS = 98304,
parameter int RESERVE_CELLS = 512,
parameter int LIMIT_CELLS = 24576,
parameter int CNT_W = 32
)(
input logic clk,
input logic rst_n,
input logic [17:0] held [N_PORTS],
input logic [17:0] shared_used,
input logic grant,
input logic [4:0] req_port,
input logic drop_valid,
input enq_fail_e drop_reason,
input logic drop_attributed,
output logic [CNT_W-1:0] v_over_limit, // a port exceeded its limit
output logic [CNT_W-1:0] v_over_pool, // held exceeds what exists
output logic [CNT_W-1:0] v_reserve_denied, // refused within the reserve
output logic [CNT_W-1:0] v_accounting_drift,
output logic [CNT_W-1:0] v_drop_unattributed,
output logic reserves_fit, // a STANDING property
output logic conformant
);
logic [17:0] total_held;
always_comb begin
total_held = 18'd0;
for (int p = 0; p < N_PORTS; p++) total_held = total_held + held[p];
end
// THE STANDING PROPERTY THAT IS TRUE. The reserves must fit; the
// LIMITS deliberately do not -- Section 18.
assign reserves_fit = ((N_PORTS * RESERVE_CELLS) <= POOL_CELLS);
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
v_over_limit <= '0;
v_over_pool <= '0;
v_reserve_denied <= '0;
v_accounting_drift <= '0;
v_drop_unattributed <= '0;
end else begin
// No port holds more than its reserve plus its limit.
for (int p = 0; p < N_PORTS; p++)
if (held[p] > 18'(RESERVE_CELLS + LIMIT_CELLS))
if (!(&v_over_limit)) v_over_limit <= v_over_limit + 1'b1;
// THE REAL CONSERVATION LAW. What is held cannot exceed what
// exists -- unlike the sum of ALLOWANCES, which is 612% of it.
if (total_held > 18'(POOL_CELLS))
if (!(&v_over_pool)) v_over_pool <= v_over_pool + 1'b1;
// THE GUARANTEE. A request within a port's reserve is never
// refused, whatever the pool's state. This is what makes the
// over-commitment safe.
if (drop_valid && (held[req_port] < 18'(RESERVE_CELLS)))
if (!(&v_reserve_denied)) v_reserve_denied <= v_reserve_denied + 1'b1;
// Shared accounting must match the sum of holdings above reserve.
begin
automatic logic [17:0] sh = 18'd0;
for (int p = 0; p < N_PORTS; p++)
if (held[p] > 18'(RESERVE_CELLS))
sh = sh + (held[p] - 18'(RESERVE_CELLS));
if (sh != shared_used)
if (!(&v_accounting_drift))
v_accounting_drift <= v_accounting_drift + 1'b1;
end
// Every drop is attributed -- Section 9. An unattributed drop is a
// number with no finding attached to it.
if (drop_valid && !drop_attributed)
if (!(&v_drop_unattributed))
v_drop_unattributed <= v_drop_unattributed + 1'b1;
end
end
assign conformant = (v_over_limit == '0) && (v_over_pool == '0) &&
(v_reserve_denied == '0) && (v_accounting_drift == '0) &&
(v_drop_unattributed == '0) && reserves_fit;
endmoduleClassification: synthesizable, and intended to stay in silicon.
What it teaches: the distinction between the conservation law that is true and the budget property that is not. total_held ≤ POOL_CELLS is a real invariant — the switch cannot hold more memory than exists, and a violation is a bug in the accounting. Σ (reserve + limit) ≤ POOL_CELLS is false by 612% and is the design working. They look like the same kind of statement and only one of them is checkable.
And v_reserve_denied is the guarantee that makes the over-commitment defensible. A port below its reserve must never be refused, whatever the pool's state — so an exhausted pool degrades every port's throughput and disconnects none of them. Without that guarantee, over-committing would mean a port could be unable to enqueue anything at all, which is a different and much worse failure than a reduced share.
Deliberately simplified: a full 24-way sum every cycle for the accounting check. A production monitor maintains a running total incrementally and reconciles it against a full sweep periodically, which is Chapter 12.5 §17's v_occupancy_drift argument applied to cells instead of entries.
Production implication: reserves_fit is a compile-time property exposed as a signal, and it is the one number a buffer configuration can get catastrophically wrong. A design whose reserves alone exceed the pool cannot honour its own guarantee — some port will be refused within its reserve, at which point the over-commitment is no longer safe and the whole argument for sharing collapses. It costs a comparison of two constants and it is checkable before the first frame.
18. Properties Worth Asserting, and One Worth Refusing
The conservation law holds; the budget property does not. Every property here is on the right side of that line, and the rejected one is on the wrong side.
The queue
// P1. A frame is enqueued only when both the port's limit and the pool
// permit it. Two independent conditions, two distinct refusals.
property p_enqueue_needs_both;
@(posedge clk) disable iff (!rst_n)
(enq_valid && enq_accept) |-> (pool_granted &&
((occupancy_cells + 16'(enq_cells)) <= 16'(MAX_CELLS)));
endproperty
a_enqueue_conditions: assert property (p_enqueue_needs_both);
// P2. THE TWO REFUSALS ARE KEPT APART. This port's own limit is this
// port's congestion; an exhausted pool is somebody else's.
property p_refusals_distinguished;
@(posedge clk) disable iff (!rst_n)
(enq_valid && !enq_accept)
|-> (enq_fail inside {EF_QUEUE_LIMIT, EF_POOL_EMPTY, EF_NO_RESERVE});
endproperty
a_refusal_classified: assert property (p_refusals_distinguished);
// P3. Occupancy never exceeds the port's limit.
property p_occupancy_bounded;
@(posedge clk) disable iff (!rst_n)
(occupancy_cells <= 16'(MAX_CELLS));
endproperty
a_occupancy_bounded: assert property (p_occupancy_bounded);
// P4. Watermarks trigger at the HIGH threshold, never at full. A
// mechanism that waits for full has already lost the frames in flight
// during its signal's propagation -- Chapter 14.2 quantifies them.
property p_high_watermark_before_full;
@(posedge clk) disable iff (!rst_n)
above_high |-> (occupancy_cells < 16'(MAX_CELLS)) or (watermark == WM_FULL);
endproperty
a_watermark_early: assert property (p_high_watermark_before_full);
// P5. Peak occupancy is retained, not averaged away.
property p_peak_monotonic;
@(posedge clk) disable iff (!rst_n)
$stable(rst_n) |-> (peak_cells >= $past(peak_cells));
endproperty
a_peak_retained: assert property (p_peak_monotonic);The pool
// P6. THE CONSERVATION LAW. What is held cannot exceed what exists.
// This is the invariant that IS true -- contrast the budget property in
// the rejected callout.
property p_held_within_pool;
@(posedge clk) disable iff (!rst_n)
(total_held <= 18'(POOL_CELLS));
endproperty
a_conservation: assert property (p_held_within_pool);
// P7. THE GUARANTEE. A request within a port's own reserve is NEVER
// refused, whatever the pool's state. This is what makes the
// over-commitment safe.
property p_reserve_always_granted;
@(posedge clk) disable iff (!rst_n)
(req_valid && ((held[req_port] + 18'(req_cells)) <= 18'(RESERVE_CELLS)))
|-> grant;
endproperty
a_reserve_honoured: assert property (p_reserve_always_granted);
// P8. No port exceeds its reserve plus its share limit.
property p_limit_enforced;
@(posedge clk) disable iff (!rst_n)
(held[req_port] <= 18'(RESERVE_CELLS + LIMIT_CELLS));
endproperty
a_limit_enforced: assert property (p_limit_enforced);
// P9. The reserves fit the pool. THIS budget property is true and must
// be -- a design whose reserves alone exceed the pool cannot honour P7.
property p_reserves_fit;
@(posedge clk) disable iff (!rst_n)
reserves_fit;
endproperty
a_reserves_fit: assert property (p_reserves_fit);
// P10. Shared accounting matches the sum of holdings above reserve.
property p_shared_accounting_balances;
@(posedge clk) disable iff (!rst_n)
(v_accounting_drift == $past(v_accounting_drift));
endproperty
a_accounting_balances: assert property (p_shared_accounting_balances);
// P11. A release returns exactly what was taken.
property p_release_matches_grant;
@(posedge clk) disable iff (!rst_n)
(rel_valid && (rel_cells != 8'd0)) |-> (held[rel_port] >= 18'(rel_cells));
endproperty
a_release_sane: assert property (p_release_matches_grant);Attribution
// P12. Every drop is attributed to an egress port, an ingress port and,
// when the pool was the cause, a holder.
property p_every_drop_attributed;
@(posedge clk) disable iff (!rst_n)
drop_valid |-> drop_attributed;
endproperty
a_drop_attributed: assert property (p_every_drop_attributed);
// P13. A pool-exhaustion drop is charged to the HOLDER, not to the port
// that could not allocate -- Section 10's whole point.
property p_pool_drop_charged_to_holder;
@(posedge clk) disable iff (!rst_n)
(drop_valid && (drop_reason == EF_POOL_EMPTY))
|=> (pool_drops_caused[$past(pool_holder)] >
$past(pool_drops_caused[$past(pool_holder)]));
endproperty
a_charged_to_holder: assert property (p_pool_drop_charged_to_holder);
// P14. The holder is sampled AT THE DROP. A periodic sample says who is
// usually busy, not who held the memory this frame needed.
property p_holder_sampled_at_drop;
@(posedge clk) disable iff (!rst_n)
(drop_valid && (drop_reason == EF_POOL_EMPTY))
|-> (pool_holder == largest_holder);
endproperty
a_sampled_at_drop: assert property (p_holder_sampled_at_drop);
// P15. victim_is_not_culprit is high exactly when the port that dropped
// is not the port holding the pool.
property p_victim_culprit_distinguished;
@(posedge clk) disable iff (!rst_n)
(drop_valid && (drop_reason == EF_POOL_EMPTY))
|=> (victim_is_not_culprit == ($past(drop_port) != $past(pool_holder)));
endproperty
a_victim_culprit: assert property (p_victim_culprit_distinguished);
// P16. Exactly one reason counter moves per drop.
property p_one_reason_per_drop;
@(posedge clk) disable iff (!rst_n)
drop_valid |=> ($countones({$changed(c_by_reason[1]), $changed(c_by_reason[2]),
$changed(c_by_reason[3]), $changed(c_by_reason[4])}) == 1);
endproperty
a_one_reason: assert property (p_one_reason_per_drop);Delay and absorption
// P17. LITTLE'S LAW, as an identity. Delay is occupancy divided by the
// drain rate, exactly, for a work-conserving server.
property p_delay_is_occupancy_over_rate;
@(posedge clk) disable iff (!rst_n)
sample |-> (delay_ns == (20'(occupancy_cells) * 20'(NS_PER_CELL)));
endproperty
a_littles_law: assert property (p_delay_is_occupancy_over_rate);
// P18. Delay is reported as a HISTOGRAM, not a mean. Chapter 12.6
// Section 6: jitter is what real-time traffic is engineered against, and
// a mean hides the tail.
property p_histogram_maintained;
@(posedge clk) disable iff (!rst_n)
sample |=> ($changed(hist[0]) || $changed(hist[1]) || $changed(hist[2]) ||
$changed(hist[3]) || $changed(hist[4]) || $changed(hist[5]) ||
$changed(hist[6]) || $changed(hist[7]));
endproperty
a_histogram: assert property (p_histogram_maintained);
// P19. A deeper buffer means a longer wait at full occupancy. The two
// are the same number, which is why "add buffer" is a latency decision.
property p_deeper_buffer_is_longer_wait;
@(posedge clk) disable iff (!rst_n)
(sample && (occupancy_cells > $past(occupancy_cells)))
|-> (delay_ns > $past(delay_ns));
endproperty
a_buffer_is_delay: assert property (p_deeper_buffer_is_longer_wait);
// P20. Absorption time is free cells divided by the excess rate -- a
// buffer's worth is a DURATION, not a size.
property p_absorption_is_a_duration;
@(posedge clk) disable iff (!rst_n)
(sample && (oversub_ratio > 6'd1))
|-> (absorb_us == ((20'(free_cells) * 20'(NS_PER_CELL)) /
(20'(oversub_ratio) - 20'd1) / 20'd1000));
endproperty
a_absorption: assert property (p_absorption_is_a_duration);
// P21. The bursts that ACTUALLY arrive are measured, not the ones the
// buffer was sized for.
property p_real_bursts_measured;
@(posedge clk) disable iff (!rst_n)
fill_ended |=> (c_bursts > $past(c_bursts));
endproperty
a_bursts_measured: assert property (p_real_bursts_measured);Telemetry and conformance
// P22. The drop rate is in parts per million. 0.05% is 500 ppm and
// matters; as an integer percentage it is zero.
property p_drop_rate_has_resolution;
@(posedge clk) disable iff (!rst_n)
(window_valid && (c_drops != '0)) |-> (drop_rate_ppm != 16'd0);
endproperty
a_ppm_resolution: assert property (p_drop_rate_has_resolution);
// P23. A latency violation is reported even when nothing is dropping --
// the one condition a loss-oriented investigation never finds.
property p_delay_reported_without_loss;
@(posedge clk) disable iff (!rst_n)
(window_valid && (p99_delay_ns >= 20'd1000000)) |-> delay_bound_exceeded;
endproperty
a_latency_visible: assert property (p_delay_reported_without_loss);
// P24. headroom_critical fires while the port is still clean, which is
// the only useful time.
property p_headroom_warns_early;
@(posedge clk) disable iff (!rst_n)
(window_valid && (worst_absorb_us < 20'd10)) |-> headroom_critical;
endproperty
a_headroom_early: assert property (p_headroom_warns_early);
// P25. One port holding more than half the pool is reported as such.
property p_dominance_reported;
@(posedge clk) disable iff (!rst_n)
(window_valid && (largest_share_pct > 16'd50)) |-> one_port_dominant;
endproperty
a_dominance: assert property (p_dominance_reported);
// P26. Conformance INCLUDES reserves_fit -- a standing property of the
// configuration, not a history of events.
property p_conformant_includes_reserves;
@(posedge clk) disable iff (!rst_n)
conformant |-> reserves_fit;
endproperty
a_conformant_reserves: assert property (p_conformant_includes_reserves);
// P27. Conformance means the accounting is sound and every drop is
// attributed -- never that no frame was lost.
property p_conformant_definition;
@(posedge clk) disable iff (!rst_n)
conformant |-> ((v_over_limit == '0) && (v_over_pool == '0) &&
(v_reserve_denied == '0) && (v_accounting_drift == '0) &&
(v_drop_unattributed == '0));
endproperty
a_conformant_def: assert property (p_conformant_definition);
// P28. Chapter 12.1's discard policy survives: a drop happens only when
// there is genuinely nowhere to put the frame.
property p_drop_only_when_no_room;
@(posedge clk) disable iff (!rst_n)
drop_valid |-> ((occupancy_cells + 16'(enq_cells) > 16'(MAX_CELLS)) ||
(shared_free < 18'(enq_cells)));
endproperty
a_drop_justified: assert property (p_drop_only_when_no_room);
// P29. An exhausted pool degrades every port and disconnects none --
// P7's guarantee, stated as an outcome.
property p_exhaustion_degrades_not_disconnects;
@(posedge clk) disable iff (!rst_n)
pool_exhausted |-> (held[req_port] >= 18'd1 || grant);
endproperty
a_degrade_not_cut: assert property (p_exhaustion_degrades_not_disconnects);
// P30. A port's own limit refusal is counted separately from a pool
// refusal, because the remedies are in different places.
property p_limit_and_pool_counted_apart;
@(posedge clk) disable iff (!rst_n)
(drop_valid && (drop_reason == EF_QUEUE_LIMIT)) |-> $stable(c_refuse_pool);
endproperty
a_counted_apart: assert property (p_limit_and_pool_counted_apart);19. Verification Scenarios
Seventy-one scenarios. Several have expected outcomes in which a correct switch drops frames, and one has an expected outcome in which the port causing the problem shows no drops at all.
The queue
| # | Scenario | Expected |
|---|---|---|
| 1 | Enqueue with room and a pool grant | accepted |
| 2 | Enqueue at the port's limit | EF_QUEUE_LIMIT |
| 3 | Enqueue with room, pool exhausted | EF_POOL_EMPTY |
| 4 | The two refusals over a run | counted separately |
| 5 | Occupancy at HIGH_CELLS | above_high, WM_HIGH |
| 6 | Occupancy at LOW_CELLS | below_low |
| 7 | Occupancy at MAX_CELLS | WM_FULL — and the pause should already have fired |
| 8 | Peak occupancy over a run | retained, never averaged |
| 9 | A port whose peak never exceeds 10% | its buffer has never been exercised |
| 10 | 64-octet frame, 128-octet cells | consumes one cell — 2× internal waste |
The pool
| # | Scenario | Expected |
|---|---|---|
| 11 | Request within the reserve, pool exhausted | granted — the guarantee |
| 12 | Request above the reserve, shared region free | granted |
| 13 | Request above the reserve, at the port's limit | EF_QUEUE_LIMIT |
| 14 | Request above the reserve, shared region empty | EF_POOL_EMPTY |
| 15 | Σ (reserve + limit) against the pool | 612% — and correct |
| 16 | Σ reserves against the pool | 12.5% — must fit, and does |
| 17 | Total held at any instant | ≤ the pool, always |
| 18 | One port at 23:1, no per-port limit | consumes 10.5 MiB in 4.0 ms |
| 19 | Same, with the 24 576-cell limit | consumes 3 MiB in 1.1 ms |
| 20 | Other ports' absorption, no limit | 0.52 ms each |
| 21 | Other ports' absorption, with the limit | 2.7 ms each — 5× better |
| 22 | Release | returns exactly what was taken |
| 23 | Shared accounting against the sum above reserve | balances |
Attribution
| # | Scenario | Expected |
|---|---|---|
| 24 | Any drop | attributed to egress, ingress and holder |
| 25 | Pool-exhaustion drop | charged to the holder, not the victim |
| 26 | Holder sampled | at the drop, not periodically |
| 27 | Drop on port 3 while port 7 holds 85% | victim_port = 3, culprit_port = 7 |
| 28 | Same | victim_is_not_culprit high |
| 29 | Port 7 holding the pool | may show zero drops of its own |
| 30 | Ports 1–6 and 8–24 | scattered EF_POOL_EMPTY drops |
| 31 | An investigation ranked by drop count | starts at the wrong end |
| 32 | mostly_somebody_else above half | this port's counter is not about this port |
| 33 | A summed "discards" figure | cannot distinguish the two remedies |
Delay
| # | Scenario | Expected |
|---|---|---|
| 34 | 3000 cells queued at 1 Gb/s | 3.07 ms — Little's law |
| 35 | 50% utilisation, 1518-octet frames | 12.1 µs mean wait |
| 36 | 90% utilisation | 109.3 µs — 9 frames |
| 37 | 99% utilisation | 1202 µs — 99 frames |
| 38 | 50% → 90% | 9×; 90% → 99% another 11× |
| 39 | Cut-through at 90% utilisation | attacks the 12.144 µs term against 109 µs |
| 40 | 99% for 6 s in a 5-minute window | averages 2% — the graph shows an idle link |
| 41 | Delay reported as a mean | hides the tail entirely |
| 42 | Delay reported as a log₂ histogram | the tail is visible |
| 43 | A millisecond of queueing, nothing dropping | delay_bound_exceeded — no drop counter fires |
Absorption
| # | Scenario | Expected |
|---|---|---|
| 44 | 122 KiB at 2:1 | 999 µs |
| 45 | 122 KiB at 24:1 | 44 µs |
| 46 | 1 MiB at 24:1 | 365 µs, then 95.7% discard |
| 47 | 512 KiB at 4:1 | 1.40 ms |
| 48 | A buffer sized for 1 ms, real bursts of 20 µs | oversized 50× |
| 49 | longest_burst_us against worst_absorb_us | the only honest sizing evidence |
| 50 | Absorption under 10 µs | headroom_critical — one frame of headroom |
| 51 | Buffer size quoted in bytes | hides both variables — rate and duration |
| 52 | 1 MiB at 2:1 absorbing 8.39 ms | a frame arriving when full waits 8.39 ms |
The host, telemetry and conformance
| # | Scenario | Expected |
|---|---|---|
| 53 | Host ring, 512 descriptors at 1 Gb/s | 344 µs of minimum frames |
| 54 | Host loss under a broadcast storm | continuous, while the switch reports nothing |
| 55 | Switch link at 6.7%, hosts at 7.4 cores | Chapter 12.4 §8 — the switch is fine |
| 56 | An unexplained-loss investigation | check the receiver's ring first |
| 57 | Drop rate of 0.05% | 500 ppm — zero as an integer percentage |
| 58 | One port above 50% of the pool | one_port_dominant |
| 59 | Reserves exceeding the pool | reserves_fit low — the guarantee cannot be honoured |
| 60 | Held exceeding the pool | v_over_pool — an accounting bug |
| 61 | A refusal within a port's reserve | v_reserve_denied — the guarantee broken |
| 62 | Healthy run, one million frames | conformant high throughout |
| 63 | Non-blocking fabric, 23:1 offered to one port | still 95.7% discarded — the fabric is not the constraint |
| 64 | Dynamic limit α × free, lone congested port | takes most of the pool; contention tightens it automatically |
| 65 | Same, 22 ports congested | every port's allowance shrinks without a global computation |
| 66 | Drop rate reported as an integer percentage | 0 across the entire diagnostic range |
| 67 | reserves_fit low at power-on | the guarantee cannot be honoured — before any frame |
| 68 | 512-descriptor host ring at 10 Gb/s | 34 µs — ten times less headroom than at 1 Gb/s |
| 69 | Switch loss over a long window | a steady fraction — systematic |
| 70 | Host loss over the same window | bursty — opportunistic, and the distinguishing property |
| 71 | Hit rate reported in whole percent | too coarse — 99% and 99.9% are different networks |
What this chapter does not fix
Every number here describes loss and delay. None of it prevents either, and the boundary is worth stating before Chapter 14.2 attempts the prevention.
| Problem | Does buffering help? | Why |
|---|---|---|
| a transient burst shorter than the absorption time | yes — completely | that is what a buffer is for |
| sustained oversubscription | no | Chapter 12.1 §6 — arithmetic, not capacity |
| queueing delay | no — it causes it | Little's law: W = L ÷ λ |
| the host's ring overflowing | no | a different buffer, for a different reason |
| a drop counter pointing at the wrong port | no — attribution does | Sections 9 and 10 |
The first row is the whole value and the second is the whole limit. A buffer converts a burst into a delay and a sustained excess into a delayed discard — and Section 14's table shows that at 24:1 even a megabyte buys 365 µs before the arithmetic reasserts itself.
Which leaves exactly one avenue unexplored: tell the sender to stop. That is Chapter 14.2's subject, and Chapter 12.1 §6 already predicted how it goes.
20. Debugging Congestion
Every row produces a switch forwarding correctly. Several produce a drop counter that points at the wrong port, and one produces no drops at all.
| Symptom | Likely cause | The observable that decides it |
|---|---|---|
| Loss scattered across many ports | one port holding the pool | largest_holder, pool_drops_caused |
| Loss on one port only | that port is oversubscribed | EF_QUEUE_LIMIT dominant — capacity, here |
| The busiest port shows no drops | it is the culprit and is winning allocations | pool_drops_caused inverts the ranking |
| Latency spikes, zero drops | queueing at ~99% utilisation | delay_bound_exceeded, p99_delay_ns |
| Utilisation graph shows 2%, users report loss | a 99% burst averaged away | the delay histogram, not the mean |
| Loss appears and clears between polls | a 4 ms pool exhaustion | pool_exhausted_ppm — the event was real |
| Adding buffer fixed the loss and broke latency | Little's law — they are the same number | delay_ns at the new occupancy |
| One port's absorption dropped without a config change | another port raised its holding | shared_free, largest_share_pct |
| Loss under a broadcast storm, switch clean | the hosts' rings, not the network | Chapter 12.4 §8 — 7.4 cores per host |
| Loss with no switch counter moving | the receiver's descriptor ring | ask the driver, not the switch |
| A port refused within its reserve | the guarantee is broken | v_reserve_denied — over-commitment is now unsafe |
| Every port at its 512-cell reserve | one port took the whole shared region | no per-port limit configured |
| Buffer refused while the pool sits mostly idle | a fixed per-port limit, too mean when uncontended | a dynamic limit α × free scales with what is free |
| Every port's allowance shrank without a config change | a dynamic limit, working as intended | shared_free — the mechanism is self-limiting |
| A metric that has read zero since it was added | the ratio's scale is wrong for its range | ppm for drop rates, hundredths of a percent for overhead |
| Loss began after a switch was replaced with a faster one | a non-blocking fabric delivers congestion faster | Section 4's callout — the buffer now fills at the full excess rate |
| A steady loss percentage across many hours | systematic — the network | switch loss is a predictable fraction; host loss is bursty |
| Buffer adequate on the bench, dropping in production | real bursts exceed worst_absorb_us | longest_burst_us — measure the bursts that arrive |
21. Common Misconceptions
1 — "Each port has its own buffer."
The wrong model: a port's buffer is a partition of memory belonging to it.
What it costs: every conclusion drawn from a drop counter. On a shared-buffer switch a port's "buffer" is an accounting entry against a pool, so a congested port does not fill its own memory — it consumes everybody's, and at 23:1 a 12 MiB pool is gone in 4.4 ms.
The corrected model: three regions — a per-port reserve that is guaranteed, a shared region any port may draw from, and a per-port limit on how much of the shared region one port may hold. The reserve is what makes an exhausted pool degrade throughput rather than disconnect a port, and the limit is what stops one port taking everything.
2 — "The sum of the port buffers should equal the total buffer."
The wrong model: the accounting must balance.
What it costs: either six times the memory or the loss of sharing. Σ (reserve + limit) on a 24-port switch is 602 112 cells against a 98 304-cell pool — 612% — and that over-commitment is the design. Making it balance means either sizing the pool at 73.5 MiB, or cutting every port to 4096 cells and abandoning the ability to borrow idle ports' memory.
The corrected model: the invariant that holds is total_held ≤ POOL_CELLS — what is actually held can never exceed what exists. The sum of the allowances is a feature, not a bug, and Section 18's rejected property confuses the two.
3 — "A drop counter tells you which port is congested."
The wrong model: the port reporting drops is the port with the problem.
What it costs: an investigation that examines twenty-three innocent ports. The drop lands on whichever port asked for a cell when the pool ran out — chosen by arrival order — while the congested port is winning its allocations and may show no drops at all.
The corrected model: a drop needs three attributions: where the enqueue failed, whose traffic it was, and whose memory it was waiting for. pool_drops_caused charges pool exhaustion to the holder, which inverts the ranking back to the useful order.
4 — "More buffer means less loss."
The wrong model: buffering is free, so deeper is better.
What it costs: latency, at a fixed exchange rate. Little's law is an identity here: W = L ÷ λ. A buffer that absorbs an 8.39 ms burst makes a frame arriving when it is full wait 8.39 ms — the same number. Chapter 12.1 §6 said a frame delivered after tens of milliseconds is often less useful than one discarded promptly, and this is the arithmetic.
The corrected model: a buffer converts loss into delay, and which is preferable depends entirely on the traffic. Bulk transfer wants depth; real-time traffic wants shallowness, because a frame past its deadline must be discarded anyway having consumed the bandwidth twice. Chapter 14.4's per-class treatment exists because one depth cannot serve both.
5 — "A quiet drop counter means no congestion."
The wrong model: if nothing is dropping, the link is healthy.
What it costs: the failure that has no drops at all. Section 12's table puts a millisecond of queueing delay at about 99% utilisation — a link passing every frame, dropping nothing, and adding a millisecond to every round trip. No counter fires, no error is logged, and every latency-sensitive application on that path is missing its budget.
The corrected model: congestion has two symptoms and loss is the second one. Delay arrives first, grows hyperbolically, and is invisible to a loss-oriented investigation. delay_bound_exceeded and a p99 histogram are what see it.
6 — "The network is where packets are lost."
The wrong model: unexplained loss means a network problem.
What it costs: the wrong investigation. A switch discards only when offered load exceeds an egress rate for longer than the buffer absorbs — 999 µs at 2:1 with a 122 KiB buffer — which real traffic rarely sustains outside a genuine capacity problem. A host discards whenever its 512-descriptor ring is full, which happens on a microsecond scale for scheduling reasons.
The corrected model: the default hypothesis should be the receiver, and the switch should be ruled out. The distinguishing property is that switch loss is systematic — Chapter 12.1 §6's predictable fraction, sustained — while host loss is opportunistic. Ask the driver for its ring-full count before opening a switch's management interface.
22. Interview Reasoning
Q1 — "Where does a switch actually drop a frame?"
Reason through it. At the egress, because that is the only point where several sources converge on one server — every other stage is sized to keep up by construction, and a backlog at the ingress FIFO or the lookup means a design failing its own specification rather than congestion. The strong answer then complicates it: on a shared-buffer switch the egress queue is a list of pointers into one pool, so the memory that runs out belongs to everybody. A congested port consumes the pool — at 23:1 a 12 MiB pool in 4.4 ms — and the drop is recorded on whichever port was enqueueing when it ran out, which is chosen by arrival order and is usually not the congested one.
Q2 — "The sum of your per-port buffer allowances is 612% of the pool. Is that a bug?"
Reason through it. No — it is the reason a shared buffer exists. A pool sized so every port could hold its full allowance simultaneously would need 73.5 MiB instead of 12 — six times the memory — to serve a case in which every port is congested at once, at which point Chapter 12.1 §6 established that no buffer helps. The strong answer names the invariant that is true: total_held ≤ POOL_CELLS, because the switch cannot hold more memory than exists. And it names what makes the over-commitment safe: a per-port reserve that is never refused, so an exhausted pool degrades every port's share and disconnects none — and a per-port limit, without which one congested port reduces the other 23 from 2.7 ms of absorption to 0.52.
Q3 — "Your busiest port shows fewer drops than several idle ones. Explain."
Reason through it. The busiest port is the culprit and it is winning its allocations. It already holds the memory; its enqueues succeed until its own limit stops them, and those refusals are counted as EF_QUEUE_LIMIT rather than pool exhaustion. Every other port's occasional request finds the shared region empty and is counted as EF_POOL_EMPTY. The strong answer names the repair: attribute each pool-exhaustion drop to whoever was holding the pool at that instant, not to the port refused — pool_drops_caused — which reverses the ranking back to the useful order. And it notes why the holder must be sampled at the drop: a periodic sample says who is usually busy, not who held the memory this frame needed.
Q4 — "Latency is up, nothing is dropping, and utilisation reads 70%. What is happening?"
Reason through it. Queueing delay at a utilisation the average is hiding. Delay is hyperbolic: 12.1 µs at 50%, 109 µs at 90%, 1202 µs at 99% for maximum-length frames at 1 Gb/s. A link at 99% for six seconds in a five-minute window averages 2% — and every frame in those six seconds was more than a millisecond late. The strong answer names the measurement that sees it: a log₂ histogram of occupancy ÷ drain rate, not a mean — Chapter 12.6 §6 established that jitter, not mean latency, is what real-time traffic is engineered against, and a mean of 40 µs with a 3 ms tail and a flat 40 µs are the same number and completely different networks.
Q5 — "A colleague proposes doubling the buffer to stop the loss. What do you say?"
Reason through it. That it will work, and state what it costs. Little's law is an identity for a work-conserving server: W = L ÷ λ — so a buffer that absorbs an 8.39 ms burst makes a frame arriving when it is full wait 8.39 ms. Absorbing more loss means introducing exactly that much more delay, at a fixed exchange rate. The strong answer makes it a question about the traffic: bulk transfer wants depth, because a retransmission costs a round trip and a millisecond costs a millisecond. Real-time traffic wants shallowness, because a frame past its deadline must be discarded anyway having consumed the bandwidth twice. And it names the real evidence: measure longest_burst_us against worst_absorb_us — a buffer sized for a millisecond on a network whose bursts are 20 µs is oversized 50×.
Q6 — "Users report packet loss. Where do you look first?"
Reason through it. The receiver's descriptor ring, not the switch. A switch discards only when offered load exceeds an egress rate for longer than the buffer absorbs — 999 µs at 2:1 with a 122 KiB buffer — which real traffic rarely sustains outside a genuine capacity problem. A host's ring holds 512 frames, 344 µs at 1 Gb/s, and overflows for scheduling reasons lasting microseconds. The strong answer gives the order: is it in the network at all (a driver statistic, seconds to check); if so, which buffer (EF_QUEUE_LIMIT against EF_POOL_EMPTY, two remedies in different places); and only then which port — after Section 10's attribution, because ranking by drops examines the victims first. And it names the exception: switch loss is systematic, a predictable fraction sustained indefinitely, while host loss is opportunistic — so a steady percentage points at the network and a bursty one points at the host.
23. Understanding Check
24. What's Next
This chapter found where frames are lost and priced what a buffer buys. It did not ask whether the loss can be prevented.
Chapter 12.1 §6 rejected the obvious answer. Applying backpressure to an ingress port converts a local congestion into a global one: the port feeding a congested egress is throttled, and so is every other conversation entering through that same port, including ones bound for completely idle egress ports.
Chapter 14.2 — PAUSE Frames builds the mechanism 802.3x specified anyway. It is a MAC control frame carrying a duration in quanta of 512 bit times, and its semantics are all-or-nothing: a PAUSE stops everything the receiving port would send, for as long as it says. Section 3's high watermark is what triggers it, and the reason the trigger is at 75% rather than at full is the frames already in flight when the signal leaves.
Then Chapter 14.3 — Backpressure and Head-of-Line Blocking shows that the pathology Chapter 12.1 predicted is exactly what happens — with a throughput bound that has been known since 1987 and is worse than most people guess.
Continue learning
Related tutorials
- Related topic
The Shared-Medium Problem
Why several independent transmitters on one medium is a distributed timing problem, not a formatting problem. Propagation delay makes every station's view of the medium stale, so two locally correct decisions can still collide — and that is the constraint the Ethernet MAC was built around.
- Related topic
CSMA/CD, Collision Domains and Slot Time
Slot time is the parameter the whole half-duplex MAC hangs on: it bounds medium acquisition, bounds a collision fragment, and is the retransmission quantum. Deriving it from round-trip propagation plus jam is what fixes Ethernet's minimum frame size — a timing constant wearing a frame-format costume.
- Related topic
Packet Switching
A circuit allocates capacity in advance and guarantees it; a packet network allocates on demand and guarantees nothing. The exchange is measurable in RTL — idle reserved slots against buffered, delayed and occasionally dropped packets — and it is why a packet must describe its own extent and destination.
- Related topic
From Coax to Twisted Pair to Switched Links
Coax, repeater, hub, bridge, switch — four steps, and only the last touched contention. A repeater reproduces a signal and cannot buffer, so it spends collision-domain budget and partitions nothing; a bridge holds the whole frame, and that buffer is what makes every other capability possible.
Standards & specifications
- Governing standard
- IEEE Std 802.3 (Ethernet)(opens IEEE in a new tab)
Defines the Ethernet MAC, the media-independent interfaces and the physical-layer sublayers, including framing, access control, auto-negotiation and per-rate PHY specifications. VLAN tagging, priority and time-sensitive shaping are defined by IEEE 802.1, not by 802.3.
This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.
Where this fits
Part of the Ethernet curriculum.
