Skip to content
VLSI Mentor

Ethernet · Module 14

Where Frames Are Lost

One congested port consumes a 12 MiB shared pool in 4.4 ms, and then every other port's next frame fails to allocate. The drop lands where the allocation failed, not where the congestion is.

Chapter 12.1 §6 established the arithmetic and then set it aside. Twenty-three ports offering line rate to one port offer 23 Gb/s to a 1 Gb/s wire; 95.7% must be discarded, and that is not a defect but the switch's specified response.

What that chapter did not ask is where the discard happens — and the answer is not the port everybody names.

On a modern switch the egress queues are not separate memories. They are lists of pointers into one shared pool, and a port's "buffer" is an accounting entry rather than a partition. So a congested port does not fill its own buffer. It consumes the pool — and at 23:1 oversubscription a 12 MiB pool is gone in 4.4 milliseconds.

After which every other port's next frame fails to allocate.

The drop is recorded on whichever port was trying to enqueue when the pool ran out, which is any port at all. The congestion is on the port holding the pool. Those are different ports, and nothing in a drop counter connects them.

1. Scope — What This Chapter Owns

This chapter owns where frames are actually lost: the four candidate places, why the egress queue is where the backlog builds, how a shared pool changes the answer, what queue delay actually is, how long a buffer buys, and why a drop counter does not localise congestion on its own.

It does not own the decision to discardChapter 12.1 §6 and §9 established that discarding is the specified response and built the tail-drop policy. This chapter asks where the discard physically happens.

It does not own the mechanism that avoids itChapter 14.2 owns the PAUSE frame, Chapter 14.3 owns what backpressure does when it propagates, and Chapter 14.4 owns per-priority flow control.

And it does not own scheduling. Chapter 13.4 §11's class scheduler decides which queue transmits; this chapter asks what happens when none of them can, because there is nowhere to put the next arrival.

2. Four Places a Frame Can Be Lost

Trace a frame from one station to another and count the buffers it must fit into. There are four, and only one of them is the one people mean by "the switch dropped it".

#BufferSized inOverflows whenReported by
1ingress FIFOa few framesthe fabric cannot keep upthe switch, rarely
2the shared poolmegabytesany port is congested for long enoughthe switch — on the wrong port
3egress queuepointers into the poolits share is exhaustedthe switch
4the receiving host's ringhundreds of descriptorsthe host cannot keep upthe host, if anybody looks

Buffer 1 is almost never the answer, and the reason is Chapter 12.5 §14's arithmetic: the forwarding path is sized to sustain line rate on every port simultaneously — 35.71 Mpps on a 24-port gigabit switch — so an ingress FIFO that overflows means the lookup is missing its deadline, which is Chapter 12.3 §11's problem rather than a congestion one.

Buffers 2 and 3 are the same memory seen two ways, and separating them is most of this chapter.

And buffer 4 is the one nobody counts. A host whose interrupt handler cannot keep up drops frames in its descriptor ring — and Chapter 12.4 §8 showed a broadcast storm puts every host in exactly that state at 7.4 cores of load. The network is not dropping those frames; the host is, and the switch's counters are clean throughout.

A frame crossing a switched network must fit into four buffers in sequence. The ingress FIFO holds a few frames and overflows only if the forwarding path is missing its deadline, which is a lookup problem rather than a congestion one. The shared buffer pool holds megabytes and is consumed by whichever port is congested, so it overflows when any single port has been oversubscribed for long enough. The egress queue is not a separate memory at all but a list of pointers into that shared pool, so its exhaustion and the pool's are the same event seen two ways. And the receiving host's descriptor ring holds a few hundred entries and overflows whenever the host cannot keep up, for reasons that have nothing to do with the network, reported only in a driver statistic somebody must know to ask for. The switch's loss is systematic and predictable while the host's is opportunistic, which is why the switch is worth studying even though the host is where loss more often happens.Sending stationoffers frames1 — Ingress FIFOa few frames; rarely theanswer2 — Shared poolmegabytes, consumed byone port3 — Egress queuepointers into the pool4 — Host ringhundreds of descriptorsReported in a driverstatif anybody asksThe switch's loss issystematica predictable fraction,indefinitely12
Figure 1 — four buffers between two stations, and the one that is easiest to blame is not the one that usually overflows.

3. RTL 1 — The Queue That Builds

One egress port, one queue, and the arithmetic that decides how fast it fills.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// congest_pkg -- shared types for congestion and flow control.
// -----------------------------------------------------------------------
package congest_pkg;

  // Why an enqueue failed. The distinction between the first two is this
  // chapter's subject: one is this port's problem and one is somebody
  // else's, and a single "drop" counter merges them.
  typedef enum logic [2:0] {
    EF_OK          = 3'd0,
    EF_QUEUE_LIMIT = 3'd1,  // this port has reached its own limit
    EF_POOL_EMPTY  = 3'd2,  // the SHARED pool is exhausted -- Section 8
    EF_NO_RESERVE  = 3'd3,  // below reserve and the reserve is spent
    EF_PORT_DOWN   = 3'd4
  } enq_fail_e;

  // Where a buffer's occupancy sits relative to its thresholds. Chapter
  // 14.2's PAUSE and Chapter 14.4's per-priority pause both trigger on
  // these rather than on "full".
  typedef enum logic [1:0] {
    WM_EMPTY = 2'd0,
    WM_LOW   = 2'd1,   // below the resume threshold
    WM_HIGH  = 2'd2,   // above the assert threshold
    WM_FULL  = 2'd3
  } watermark_e;

  localparam int CELL_OCTETS = 128;   // allocation granularity

endpackage
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// egress_queue_model -- one port's queue, its watermarks, and the rate at
// which it fills.
//
// Chapter 12.1 Section 9 built the tail-drop policy on this queue. What
// that chapter treated as a fixed depth is, on a shared-buffer switch, a
// claim against a pool -- which is Section 5's subject and the reason
// this module reports its occupancy in CELLS rather than octets.
// -----------------------------------------------------------------------
module egress_queue_model
  import congest_pkg::*;
#(
  parameter int MAX_CELLS  = 4096,      // this port's limit, not its memory
  parameter int HIGH_CELLS = 3072,
  parameter int LOW_CELLS  = 1024,
  parameter int CNT_W      = 32
)(
  input  logic                 clk,
  input  logic                 rst_n,

  input  logic                 enq_valid,
  input  logic [7:0]           enq_cells,       // frame size in cells
  input  logic                 pool_granted,    // Section 5 said yes
  input  logic                 deq_valid,
  input  logic [7:0]           deq_cells,

  output logic                 enq_accept,
  output enq_fail_e            enq_fail,

  output logic [15:0]          occupancy_cells,
  output logic [15:0]          peak_cells,
  output watermark_e           watermark,
  output logic                 above_high,
  output logic                 below_low,

  output logic [CNT_W-1:0]     c_enq,
  output logic [CNT_W-1:0]     c_drop_limit,
  output logic [CNT_W-1:0]     c_drop_pool,
  output logic [CNT_W-1:0]     c_cells_dropped
);

  always_comb begin
    enq_accept = 1'b0;
    enq_fail   = EF_OK;

    if (enq_valid) begin
      // THE TWO REFUSALS, KEPT APART. Reaching this port's own limit is
      // this port's congestion. Failing to get a cell from the pool is
      // somebody else's congestion arriving here -- Section 10.
      if ((occupancy_cells + 16'(enq_cells)) > 16'(MAX_CELLS))
        enq_fail = EF_QUEUE_LIMIT;
      else if (!pool_granted)
        enq_fail = EF_POOL_EMPTY;
      else
        enq_accept = 1'b1;
    end
  end

  always_comb begin
    if      (occupancy_cells == 16'd0)             watermark = WM_EMPTY;
    else if (occupancy_cells >= 16'(MAX_CELLS))    watermark = WM_FULL;
    else if (occupancy_cells >= 16'(HIGH_CELLS))   watermark = WM_HIGH;
    else if (occupancy_cells <= 16'(LOW_CELLS))    watermark = WM_LOW;
    else                                            watermark = WM_LOW;
  end

  // Chapter 14.2's PAUSE and Chapter 14.4's per-priority pause both
  // trigger on the HIGH watermark and release on the LOW one, never on
  // "full" -- by the time a queue is full the frames that would have
  // been paused have already arrived.
  assign above_high = (occupancy_cells >= 16'(HIGH_CELLS));
  assign below_low  = (occupancy_cells <= 16'(LOW_CELLS));

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      occupancy_cells <= '0;
      peak_cells      <= '0;
      c_enq           <= '0;
      c_drop_limit    <= '0;
      c_drop_pool     <= '0;
      c_cells_dropped <= '0;
    end else begin
      if (enq_valid && enq_accept) begin
        occupancy_cells <= occupancy_cells + 16'(enq_cells);
        c_enq           <= c_enq + 1'b1;
        if ((occupancy_cells + 16'(enq_cells)) > peak_cells)
          peak_cells <= occupancy_cells + 16'(enq_cells);
      end else if (enq_valid) begin
        c_cells_dropped <= c_cells_dropped + CNT_W'(enq_cells);
        if (enq_fail == EF_QUEUE_LIMIT) begin
          if (!(&c_drop_limit)) c_drop_limit <= c_drop_limit + 1'b1;
        end else if (enq_fail == EF_POOL_EMPTY) begin
          if (!(&c_drop_pool)) c_drop_pool <= c_drop_pool + 1'b1;
        end
      end

      if (deq_valid) occupancy_cells <= occupancy_cells - 16'(deq_cells);
    end
  end

endmodule

Classification: synthesizable.

What it teaches: that c_drop_limit and c_drop_pool must be separate counters, and that merging them into a single "drops" figure destroys the only information that says whose congestion this is. Reaching this port's own limit means this port is oversubscribedChapter 12.1 §6's arithmetic, and the remedy is capacity here. Failing to obtain a cell from the pool means some other port is oversubscribed and has consumed the shared memory, and the remedy is somewhere else entirely.

And it teaches why the watermarks are at 75% and 25% rather than at "full". Chapter 14.2's PAUSE and Chapter 14.4's per-priority pause both trigger on the high watermark, because a mechanism that waits for full has already lost the frames it was meant to protect — the ones in flight during the signal's propagation, which Chapter 14.2 quantifies.

Deliberately simplified: occupancy in fixed 128-octet cells rather than octets. Cell allocation is what real switches do — a 64-octet frame and a 128-octet frame both consume one cell — and it means a queue's occupancy in cells and its occupancy in useful octets differ by up to 2× on small frames, which is a real and frequently forgotten inefficiency.

Production implication: peak_cells against MAX_CELLS is the number that distinguishes a buffer that is adequately sized from one that has never been tested. A port whose peak occupancy has never exceeded 10% of its limit has a buffer that has never been exercised — which is not evidence it is big enough, only that nothing has yet asked. A port whose peak equals its limit has been dropping, and c_drop_limit says how often.

4. Why the Backlog Builds at the Egress

Congestion in a switched network has exactly one shape, and naming it precisely rules out most of the places people look for it.

A queue builds where the arrival rate exceeds the service rate, for as long as it does. In a switch there is only one place that can happen: the egress port, because that is the only point where several sources converge on one server.

PointArrivalsServiceCan a backlog build
ingress portone linkthe fabric, sized for line ratenoChapter 12.5 §14
the lookupone per frame28 ns budgetno — a miss floods rather than queues
the fabricN portsN portsno, if it is non-blocking
the egress portup to N−1 sourcesone wireyes — and this is the only one

Everything above the egress row is sized to keep up by construction. Chapter 12.1 §12 established the frame-rate budget and Chapter 12.5 §14 showed the table meets it with margin. A backlog at any of them is a design failing to meet its own specification, not congestion.

And the egress row cannot be sized to keep up, because the arrival rate is not bounded by anything the switch controls. Chapter 12.1 §6's 23:1 is the limit case, and no fabric, buffer or scheduler changes the arithmetic.

Which gives the chapter's first firm conclusion: congestion is an egress phenomenon, and every other buffer in Section 2's list either does not fill or fills for a different reason.

5. RTL 2 — The Shared Pool

And here is where the simple picture breaks. A port's queue is a list of pointers, and the memory those pointers reference belongs to everybody.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// buffer_pool_manager -- one memory, many ports, three regions.
//
// The three regions are the whole design. A per-port RESERVE that only
// that port may use; a SHARED region any port may draw from; and a
// per-port LIMIT on how much of the shared region one port may hold. The
// sum of the reserves plus the shared region is the pool; the sum of the
// LIMITS deliberately exceeds it -- Section 18's rejected property.
// -----------------------------------------------------------------------
module buffer_pool_manager
  import congest_pkg::*;
#(
  parameter int N_PORTS      = 24,
  parameter int POOL_CELLS   = 98304,   // 12 MiB at 128 octets per cell
  parameter int RESERVE_CELLS = 512,    // per port, guaranteed
  parameter int LIMIT_CELLS  = 24576,   // per port, of the SHARED region
  parameter int CNT_W        = 32
)(
  input  logic                 clk,
  input  logic                 rst_n,

  input  logic                 req_valid,
  input  logic [4:0]           req_port,
  input  logic [7:0]           req_cells,

  input  logic                 rel_valid,
  input  logic [4:0]           rel_port,
  input  logic [7:0]           rel_cells,

  output logic                 grant,
  output enq_fail_e            fail,

  output logic [17:0]          pool_free,
  output logic [17:0]          shared_free,
  output logic [17:0]          held [N_PORTS],
  output logic [4:0]           largest_holder,
  output logic [15:0]          largest_share_pct,
  output logic                 pool_exhausted,

  output logic [CNT_W-1:0]     c_grant,
  output logic [CNT_W-1:0]     c_refuse_limit,
  output logic [CNT_W-1:0]     c_refuse_pool,
  output logic [CNT_W-1:0]     overcommit_ratio_x100
);

  localparam int RESERVED_TOTAL = N_PORTS * RESERVE_CELLS;
  localparam int SHARED_TOTAL   = POOL_CELLS - RESERVED_TOTAL;

  logic [17:0] shared_used;

  // A port's reserve is spent first and belongs to nobody else. Only what
  // it holds ABOVE the reserve comes from the shared region.
  function automatic logic [17:0] shared_held(input logic [17:0] h);
    shared_held = (h > 18'(RESERVE_CELLS)) ? (h - 18'(RESERVE_CELLS)) : 18'd0;
  endfunction

  always_comb begin
    automatic logic [17:0] h  = held[req_port];
    automatic logic [17:0] sh = shared_held(h);
    grant = 1'b0;
    fail  = EF_OK;

    if (req_valid) begin
      if ((h + 18'(req_cells)) <= 18'(RESERVE_CELLS)) begin
        // WITHIN THE RESERVE. Always granted -- this is the guarantee
        // that makes the shared region safe to over-commit.
        grant = 1'b1;
      end else if ((sh + 18'(req_cells)) > 18'(LIMIT_CELLS)) begin
        // This port has taken as much of the shared region as it is
        // allowed. Its congestion is now its own problem, which is the
        // entire purpose of the limit.
        fail = EF_QUEUE_LIMIT;
      end else if ((shared_used + 18'(req_cells)) > 18'(SHARED_TOTAL)) begin
        // THE SHARED REGION IS GONE. Somebody else took it -- Section 8.
        fail = EF_POOL_EMPTY;
      end else begin
        grant = 1'b1;
      end
    end
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int p = 0; p < N_PORTS; p++) held[p] <= '0;
      shared_used           <= '0;
      c_grant               <= '0;
      c_refuse_limit        <= '0;
      c_refuse_pool         <= '0;
      largest_holder        <= '0;
      largest_share_pct     <= '0;
      pool_exhausted        <= 1'b0;
    end else begin
      if (req_valid && grant) begin
        held[req_port] <= held[req_port] + 18'(req_cells);
        if (held[req_port] >= 18'(RESERVE_CELLS))
          shared_used <= shared_used + 18'(req_cells);
        c_grant <= c_grant + 1'b1;
      end else if (req_valid) begin
        if (fail == EF_QUEUE_LIMIT) begin
          if (!(&c_refuse_limit)) c_refuse_limit <= c_refuse_limit + 1'b1;
        end else begin
          if (!(&c_refuse_pool)) c_refuse_pool <= c_refuse_pool + 1'b1;
        end
      end

      if (rel_valid) begin
        held[rel_port] <= held[rel_port] - 18'(rel_cells);
        if (held[rel_port] > 18'(RESERVE_CELLS))
          shared_used <= shared_used - 18'(rel_cells);
      end

      // WHO IS HOLDING THE POOL. This is the number that turns a drop on
      // port 3 into a finding about port 7 -- Section 10.
      begin
        automatic logic [17:0] hi = '0;
        automatic logic [4:0]  hp = '0;
        for (int p = 0; p < N_PORTS; p++)
          if (held[p] > hi) begin hi = held[p]; hp = 5'(p); end
        largest_holder    <= hp;
        largest_share_pct <= 16'((hi * 18'd100) / 18'(POOL_CELLS));
      end

      pool_exhausted <= (shared_used >= 18'(SHARED_TOTAL));
    end
  end

  assign shared_free = 18'(SHARED_TOTAL) - shared_used;
  assign pool_free   = shared_free;

  // THE OVER-COMMITMENT, COMPUTED. The sum of what every port is allowed
  // to take, against what exists. A value above 100 is the design working
  // as intended -- Section 18.
  assign overcommit_ratio_x100 =
    CNT_W'((((N_PORTS * (RESERVE_CELLS + LIMIT_CELLS)) * 100) / POOL_CELLS));

endmodule

Classification: synthesizable.

What it teaches: that the three regions are the whole design and each does a different job. The reserve is what makes the shared region safe to over-commit — every port can always enqueue something, so an exhausted pool degrades throughput rather than disconnecting a port. The shared region is where the value is: most ports are idle most of the time, and sharing their unused memory with whichever port is busy is the reason a pool exists at all. And the limit is what stops one port taking everything.

And overcommit_ratio_x100 above 100 is the design working. With 24 ports, a 512-cell reserve and a 24 576-cell limit each, the sum of allowances is 24 × 25 088 = 602 112 cells against a 98 304-cell pool — a ratio of 612%. That is not a bug and it is not sloppy accounting. It is the entire economic argument for a shared buffer: the sum of what everybody may take exceeds what exists, because they will not all take it at once.

Deliberately simplified: a single shared region with one limit per port. Production designs make the limit dynamic — a port's allowance is a fraction of the currently free pool rather than a constant — so a port congested while the pool is empty gets far less than one congested while it is idle. That is strictly better and it makes the accounting considerably harder to reason about.

Production implication: largest_holder and largest_share_pct are the pair that make Section 10 possible. A drop on port 3 with largest_holder = 7 at 85% is a complete diagnosis — port 3 could not allocate because port 7 is holding the pool — and neither half of it is available from a per-port drop counter.

6. The Shared Pool, and Who Owns It

Put numbers on a 24-port switch with a 12 MiB pool and the over-commitment becomes concrete.

QuantityCellsOctets
the pool98 30412 MiB
per-port reserve, 24 ports24 × 512 = 12 2881.5 MiB
the shared region86 01610.5 MiB
per-port limit on the shared region24 5763 MiB
Σ (reserve + limit) over 24 ports602 11273.5 MiB
over-commitment612%

Six times more memory is promised than exists, and the design depends on it. A pool sized so that every port could simultaneously take its full allowance would need 73.5 MiB — six times the silicon, to serve a case that occurs when every port on the switch is congested at once, at which point the switch has a capacity problem no buffer solves.

And the limit is doing real work. Without it, one port could take the entire 86 016-cell shared region:

Per-port limitOne port's maximum holdWhat the other 23 keep
none86 016 cells — the whole shared regiontheir reserves only: 512 cells each
24 57624 576 — 28.6% of the shared region61 440 cells to share
8 1928 192 — 9.5%77 824 cells

With no limit, a single congested port reduces every other port to its 512-cell reserve64 KiB, which at 2:1 oversubscription absorbs 512 × 128 × 8 ÷ 1e9 = 0.52 ms and then drops.

A shared buffer pool of twelve mebibytes is divided into three logical regions on a twenty-four port switch. Each port has a guaranteed reserve of five hundred and twelve cells, sixty-four kibibytes, which no other port may take, and the twenty-four reserves together account for one and a half mebibytes. The remaining ten and a half mebibytes form a shared region that any port may draw from. Each port is additionally limited to twenty-four thousand five hundred and seventy-six cells of that shared region, which is three mebibytes or twenty-eight point six percent of it. Adding every port's reserve to its limit gives seventy-three and a half mebibytes of promised memory against twelve mebibytes that exist, an over-commitment of six hundred and twelve percent. That over-commitment is deliberate and is the entire economic argument for sharing, because most ports are idle most of the time and a pool sized for every port to take its full allowance simultaneously would need six times the silicon to serve a case in which the switch already has a capacity problem no buffer solves.Pool — 12 MiB98 304 cellsReserves — 1.5 MiB512 cells per port,guaranteedShared — 10.5 MiBany port may drawLimit — 3 MiB perport28.6% of the sharedregionΣ allowances = 73.5MiB612% of the poolDeliberatenot every port is busy atonceWithout the limitone port takes all 10.5MiB12
Figure 2 — reserve, shared region and limit: the sum of allowances is 612% of the pool, and that over-commitment is the reason a shared buffer is worth building.

7. RTL 3 — Who Is Holding the Pool

A drop says a frame could not be enqueued. It does not say why the memory was gone, and the answer is a different port.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// shared_pool_accountant -- attributes pool exhaustion to the port that
// caused it, which is not the port that suffered it.
//
// Chapter 12.1 Section 11 made the same argument one level down: c_drops
// says an egress port is oversubscribed, and drops_by_ingress says which
// conversation is doing it. This is that argument applied to the pool.
// -----------------------------------------------------------------------
module shared_pool_accountant
  import congest_pkg::*;
#(
  parameter int N_PORTS = 24,
  parameter int CNT_W   = 32,
  parameter int WIN     = 1_000_000
)(
  input  logic             clk,
  input  logic             rst_n,

  input  logic [17:0]      held [N_PORTS],
  input  logic [17:0]      shared_total,
  input  logic             pool_exhausted,

  input  logic             drop_valid,
  input  logic [4:0]       drop_port,
  input  enq_fail_e        drop_reason,

  output logic [CNT_W-1:0] drops_suffered [N_PORTS],
  output logic [CNT_W-1:0] pool_drops_caused [N_PORTS],
  output logic [4:0]       victim_port,
  output logic [4:0]       culprit_port,
  output logic             victim_is_not_culprit,
  output logic [15:0]      culprit_share_pct,
  output logic             window_valid
);

  logic [CNT_W-1:0] win;

  // WHO HELD THE POOL AT THE MOMENT OF THE DROP. Sampling this at the
  // drop rather than periodically is what makes the attribution sound --
  // a periodic sample tells you who is usually busy, not who was holding
  // the memory the instant a frame was refused.
  logic [4:0]  hi_port;
  logic [17:0] hi_held;

  always_comb begin
    hi_port = 5'd0;
    hi_held = 18'd0;
    for (int p = 0; p < N_PORTS; p++)
      if (held[p] > hi_held) begin hi_held = held[p]; hi_port = 5'(p); end
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int p = 0; p < N_PORTS; p++) begin
        drops_suffered[p]    <= '0;
        pool_drops_caused[p] <= '0;
      end
      win                   <= '0;
      victim_port           <= '0;
      culprit_port          <= '0;
      victim_is_not_culprit <= 1'b0;
      culprit_share_pct     <= '0;
      window_valid          <= 1'b0;
    end else begin
      window_valid <= 1'b0;

      if (drop_valid) begin
        drops_suffered[drop_port] <= drops_suffered[drop_port] + 1'b1;
        win <= win + 1'b1;

        // THE ATTRIBUTION. A drop caused by an exhausted POOL is charged
        // to whoever is holding it, not to the port that could not
        // allocate. A drop caused by this port's own LIMIT is charged
        // here, because that is this port's congestion.
        if (drop_reason == EF_POOL_EMPTY) begin
          pool_drops_caused[hi_port] <= pool_drops_caused[hi_port] + 1'b1;
          victim_port                <= drop_port;
          culprit_port               <= hi_port;
          victim_is_not_culprit      <= (drop_port != hi_port);
          culprit_share_pct          <= 16'((hi_held * 18'd100) / shared_total);
        end
      end

      if (win >= CNT_W'(WIN)) begin
        win          <= '0;
        window_valid <= 1'b1;
      end
    end
  end

endmodule

Classification: synthesizable.

What it teaches: that the attribution must be sampled at the moment of the drop, and a periodic sample is a different and much weaker measurement. A periodic sample says who is usually busy. Sampling held[] at the instant a frame is refused says who was holding the memory that frame needed — and on a switch where several ports take turns being congested, those give different answers.

And victim_is_not_culprit is the bit worth exposing on its own. It is high whenever a port dropped because of somebody else, which is the condition that makes a per-port drop counter misleading rather than merely incomplete. With it low, a drop on port 3 means port 3 is oversubscribed. With it high, port 3 is a bystander.

Deliberately simplified: a combinational maximum over 24 holdings, evaluated every cycle. A production accountant maintains the maximum incrementally, because a 24-way comparison tree on the allocation path is not free at the rates Chapter 12.1 §12 established.

Production implication: pool_drops_caused is the counter that changes what an operator does, and it inverts the usual reading. A port with a large drops_suffered and a small pool_drops_caused is a victim — investigating its traffic finds nothing wrong. A port with a small drops_suffered and a large pool_drops_caused is the problem, and it may be showing no drops of its own at all, because it is winning every allocation.

8. One Congested Port Can Starve Twenty-Three

Do the arithmetic on how fast a single oversubscribed port consumes the shared region, because the answer is measured in milliseconds.

A port receiving 23:1 oversubscription — Chapter 12.1 §6's limit case — is offered 23 Gb/s and drains 1 Gb/s, so it accumulates at 22 Gb/s.

Shared regionTime for one port to consume it at 22 Gb/s excess
10.5 MiB — no per-port limit4.0 ms
3 MiB — with the 24 576-cell limit1.1 ms

And what the other 23 ports have left, in each case:

other ports' available memoryabsorbs 2:1 oversubscription for
no limit512 cells — 64 KiB each0.52 ms
with the limit61 440 ÷ 23 = 2671 cells — 334 KiB each2.7 ms

A factor of five in burst tolerance, from one comparison in the allocation path.

And the timescales are what make this hard to catch. 4 milliseconds is far below the sampling interval of any management system — a pool that is exhausted, drops frames on every port, and recovers, all between two SNMP polls — while the counters it left behind say only that several ports dropped frames at some point in the last five minutes.

9. RTL 4 — Attributing a Drop

Five distinct reasons a frame is discarded, and a single counter for all of them is the commonest instrumentation failure in a switch.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// drop_attribution -- every discard, classified and attributed.
//
// Chapter 12.3 Section 10's deciding_gate did this for the forwarding
// decision. This does it for the buffer, and the reasons are unrelated
// to each other in exactly the same way.
// -----------------------------------------------------------------------
module drop_attribution
  import congest_pkg::*;
#(
  parameter int N_PORTS = 24,
  parameter int CNT_W   = 32,
  parameter int WIN     = 1_000_000
)(
  input  logic             clk,
  input  logic             rst_n,

  input  logic             drop_valid,
  input  enq_fail_e        reason,
  input  logic [4:0]       egress_port,
  input  logic [4:0]       ingress_port,
  input  logic [4:0]       pool_holder,

  output logic [CNT_W-1:0] c_by_reason [5],
  output logic [CNT_W-1:0] c_by_egress [N_PORTS],
  output logic [CNT_W-1:0] c_by_ingress [N_PORTS],

  output logic             window_valid,
  output logic [15:0]      pool_drop_share_pct,
  output logic             mostly_somebody_else,
  output logic [4:0]       dominant_ingress,
  output logic [4:0]       dominant_holder
);

  logic [CNT_W-1:0] win, win_pool;
  logic [CNT_W-1:0] holder_hits [N_PORTS];

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int i = 0; i < 5; i++) c_by_reason[i] <= '0;
      for (int p = 0; p < N_PORTS; p++) begin
        c_by_egress[p] <= '0; c_by_ingress[p] <= '0; holder_hits[p] <= '0;
      end
      win <= '0; win_pool <= '0;
      window_valid        <= 1'b0;
      pool_drop_share_pct <= '0;
      mostly_somebody_else<= 1'b0;
      dominant_ingress    <= '0;
      dominant_holder     <= '0;
    end else begin
      window_valid <= 1'b0;

      if (drop_valid) begin
        c_by_reason[reason[2:0]] <= c_by_reason[reason[2:0]] + 1'b1;
        // THREE ATTRIBUTIONS PER DROP, and they answer three questions.
        // Egress: where did it fail. Ingress: whose traffic was it.
        // Holder: whose memory was it waiting for.
        c_by_egress[egress_port]   <= c_by_egress[egress_port] + 1'b1;
        c_by_ingress[ingress_port] <= c_by_ingress[ingress_port] + 1'b1;
        win <= win + 1'b1;

        if (reason == EF_POOL_EMPTY) begin
          win_pool               <= win_pool + 1'b1;
          holder_hits[pool_holder] <= holder_hits[pool_holder] + 1'b1;
        end
      end

      if (win >= CNT_W'(WIN)) begin
        automatic logic [CNT_W-1:0] hi = '0;
        automatic logic [4:0] hp = '0;
        automatic logic [CNT_W-1:0] hi2 = '0;
        automatic logic [4:0] hp2 = '0;
        for (int p = 0; p < N_PORTS; p++) begin
          if (holder_hits[p] > hi)      begin hi = holder_hits[p];  hp = 5'(p); end
          if (c_by_ingress[p] > hi2)    begin hi2 = c_by_ingress[p]; hp2 = 5'(p); end
        end
        pool_drop_share_pct <= 16'((win_pool * CNT_W'(100)) / win);
        // MORE THAN HALF OF THIS PORT'S DROPS WERE SOMEBODY ELSE'S FAULT.
        // The single most useful bit in the module.
        mostly_somebody_else <= (win_pool > (win >> 1));
        dominant_holder      <= hp;
        dominant_ingress     <= hp2;
        win <= '0; win_pool <= '0;
        for (int p = 0; p < N_PORTS; p++) holder_hits[p] <= '0;
        window_valid <= 1'b1;
      end
    end
  end

endmodule

Classification: synthesizable.

What it teaches: that one drop deserves three attributions, and they answer three unrelated questions. Where did the enqueue fail is the egress port. Whose traffic was refused is the ingress port — Chapter 12.1 §11's drops_by_ingress. And whose memory was it waiting for is the pool holder, which exists only on a shared-buffer switch and has no analogue in that chapter.

And mostly_somebody_else is the derived bit that changes an investigation's direction. Above half, this port's drop counter is not about this port — it is a symptom of Section 8's pool exhaustion, and every minute spent examining this port's traffic is wasted.

Deliberately simplified: three flat counter arrays. Production instrumentation keeps a small two-dimensional histogram of (egress, holder) pairs, because the useful finding is often port 3's drops are 90% attributable to port 7 rather than either port's total.

Production implication: c_by_reason has five entries and a switch that exposes one "discards" figure has summed them. EF_QUEUE_LIMIT and EF_POOL_EMPTY are the two that matter and they point in opposite directions — the first says this port is oversubscribed and needs capacity, the second says this port is fine and somebody else is holding the memory. A summed counter cannot distinguish add a faster uplink here from find the port that is hoarding, and those are the two most common remedies.

10. Where a Frame Is Lost Is Not Where the Congestion Is

Put Sections 7 and 8 together and state the conclusion plainly, because it inverts the reading of the most-watched counter on a switch.

Three questions, three answers, and on a shared-buffer switch they are frequently three different ports:

QuestionAnswerAvailable from
where did the enqueue failthe egress port that was refuseda per-port drop counter
whose traffic was refusedthe ingress port the frame came fromChapter 12.1 §11's drops_by_ingress
whose memory was it waiting forthe port holding the poollargest_holder — Section 5

A per-port drop counter answers only the first, and the first is the least useful of the three.

The pattern this produces is worth recognising by shape:

ObservationWhat it usually means
drops on one port, EF_QUEUE_LIMIT dominantthat port is oversubscribed — capacity, here
drops on many ports, EF_POOL_EMPTY dominantone port is holding the pool — find it
drops on many ports, no single holdergenuine switch-wide overload
no drops on the busiest port, drops elsewherethe busiest port is the culprit

The last row is the one that reads as impossible and is the normal case. A port holding the pool is winning its allocations — it already has the memory, so its enqueues succeed until its own limit stops them, and those refusals are counted as EF_QUEUE_LIMIT rather than as pool exhaustion. Meanwhile every other port's EF_POOL_EMPTY drops accumulate.

So the port causing the problem can genuinely show fewer drops than its victims, and an investigation ranked by drop count starts at the wrong end.

11. RTL 5 — Estimating Queue Delay

Occupancy is a number of cells. What anybody actually wants to know is how long a frame will wait, and Little's law converts one into the other.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// queue_delay_estimator -- occupancy to latency, in hardware.
//
// L = lambda * W, so W = L / lambda: the wait a frame joining the queue
// now will experience is the current occupancy divided by the drain rate.
// It is exact for a work-conserving server and it is the number an
// application cares about, which occupancy is not.
// -----------------------------------------------------------------------
module queue_delay_estimator
  import congest_pkg::*;
#(
  parameter int LINK_MBPS = 1000,
  parameter int CNT_W     = 32
)(
  input  logic             clk,
  input  logic             rst_n,

  input  logic [15:0]      occupancy_cells,
  input  logic             sample,

  output logic [19:0]      delay_ns,
  output logic [19:0]      peak_delay_ns,
  output logic [19:0]      p99_delay_ns,       // coarse, from a histogram
  output logic             delay_over_1ms,
  output logic [CNT_W-1:0] hist [8]            // log2 buckets of delay
);

  // Cells to bits, bits to nanoseconds. At 128 octets per cell and
  // 1000 Mb/s: one cell is 1024 bits and drains in 1024 ns.
  localparam int NS_PER_CELL = (CELL_OCTETS * 8 * 1000) / LINK_MBPS;

  assign delay_ns = 20'(occupancy_cells) * 20'(NS_PER_CELL);

  // A log2 histogram rather than a mean. Chapter 12.6 Section 6
  // established that jitter is what real-time traffic is engineered
  // against, and a mean delay hides the tail entirely.
  function automatic logic [2:0] bucket(input logic [19:0] d);
    if      (d < 20'd1000)    bucket = 3'd0;   // < 1 us
    else if (d < 20'd4000)    bucket = 3'd1;
    else if (d < 20'd16000)   bucket = 3'd2;
    else if (d < 20'd64000)   bucket = 3'd3;
    else if (d < 20'd256000)  bucket = 3'd4;   // < 256 us
    else if (d < 20'd512000)  bucket = 3'd5;
    else if (d < 20'd1000000) bucket = 3'd6;   // < 1 ms
    else                      bucket = 3'd7;   // >= 1 ms
  endfunction

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int i = 0; i < 8; i++) hist[i] <= '0;
      peak_delay_ns  <= '0;
      p99_delay_ns   <= '0;
      delay_over_1ms <= 1'b0;
    end else if (sample) begin
      hist[bucket(delay_ns)] <= hist[bucket(delay_ns)] + 1'b1;
      if (delay_ns > peak_delay_ns) peak_delay_ns <= delay_ns;
      delay_over_1ms <= (delay_ns >= 20'd1000000);

      // A coarse 99th percentile: the lowest bucket boundary above which
      // fewer than one percent of samples fall. Computed by walking the
      // histogram from the top, which is a handful of comparisons.
      begin
        automatic logic [CNT_W-1:0] tot = '0;
        automatic logic [CNT_W-1:0] acc = '0;
        for (int i = 0; i < 8; i++) tot = tot + hist[i];
        for (int i = 7; i >= 0; i--) begin
          acc = acc + hist[i];
          if (acc > (tot / 100)) begin
            p99_delay_ns <= 20'd1000 << (3 * i);
            break;
          end
        end
      end
    end
  end

endmodule

Classification: synthesizable; the percentile walk is a small sequential loop in a real design rather than the combinational one shown.

What it teaches: that occupancy and delay are the same measurement in different units, and only one of them is actionable. L ÷ λ converts cells into nanoseconds exactly — at 128 octets per cell and 1 Gb/s, one cell is 1024 ns — and "this queue holds 3000 cells" means nothing to an application while "a frame joining now waits 3.07 ms" is a service-level statement.

And it teaches why the histogram is log₂ rather than a mean. Chapter 12.6 §6 established that jitter, not mean latency, is what real-time traffic is engineered against — a receive buffer is sized against the worst case minus the best. A mean delay of 40 µs with a 3 ms tail and a flat 40 µs are the same number and completely different networks, and only the histogram separates them.

Deliberately simplified: a fixed drain rate. A real port's drain rate varies with frame size — Chapter 12.1 §12's interframe gap and preamble overhead means a queue of minimum-length frames drains slower in bits than one of maximum-length frames — so the estimate is optimistic by up to 31% on small frames.

Production implication: delay_over_1ms is a threshold worth a standing alert rather than a graph. A millisecond of queueing delay is longer than most datacentre round trips and longer than the timeout of several storage protocols — and Section 12's table shows it arrives at about 99% utilisation, which a five-minute average will report as a comfortable 70%.

12. Queue Delay, Measured

Queueing delay is not linear in utilisation. It is hyperbolic, and the whole of it happens in the last few percent.

For 1518-octet frames at 1 Gb/s, one frame's service time is 12.144 µs. Mean wait against utilisation:

UtilisationMean waitIn frames
50%12.1 µs1.0
70%28.3 µs2.3
90%109.3 µs9.0
95%230.7 µs19.0
99%1202.3 µs99.0

From 50% to 90% the delay grows 9×. From 90% to 99% it grows another 11× — and 99% utilisation is a link an operator would describe as nearly full rather than broken.

Which is the shape Chapter 12.6 §10 relied on and did not derive. That chapter argued cut-through's benefit vanishes under load because queueing dominates; here is the arithmetic: at 90% utilisation the queueing term is 109 µs against a 12.144 µs serialisation term — nine times larger — and cut-through attacks only the smaller one.

And it explains why averaging destroys the measurement. A link at 99% for six seconds and idle for the rest of a five-minute interval averages 2% utilisation and had a millisecond of queueing delay throughout those six seconds. Every frame crossing in that window was late; the graph shows an idle link.

13. RTL 6 — How Long a Buffer Buys

A buffer's size is usually quoted in bytes. What it is actually worth is a duration, and the conversion depends on the oversubscription ratio it is absorbing.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// burst_absorption_model -- how long this buffer holds out.
//
// A buffer absorbs a RATE MISMATCH for a DURATION. Quoting its size in
// bytes hides both variables; quoting the duration it buys at a stated
// oversubscription ratio states them.
// -----------------------------------------------------------------------
module burst_absorption_model
  import congest_pkg::*;
#(
  parameter int LINK_MBPS = 1000,
  parameter int CNT_W     = 32
)(
  input  logic             clk,
  input  logic             rst_n,

  input  logic [15:0]      free_cells,
  input  logic [5:0]       oversub_ratio,   // N:1, so excess = (N-1) x link
  input  logic             sample,

  output logic [19:0]      absorb_us,
  output logic [19:0]      worst_absorb_us,
  output logic             under_100us,
  output logic             under_10us,

  input  logic             fill_started,
  input  logic             fill_ended,
  output logic [19:0]      last_burst_us,
  output logic [19:0]      longest_burst_us,
  output logic [CNT_W-1:0] c_bursts,
  output logic             buffer_was_adequate
);

  // Excess rate = (ratio - 1) x link. Time = bits / excess.
  // At 128 octets per cell and 1000 Mb/s, one cell is 1024 bits and
  // drains in 1.024 us at the link rate.
  localparam int NS_PER_CELL = (CELL_OCTETS * 8 * 1000) / LINK_MBPS;

  always_comb begin
    if (oversub_ratio <= 6'd1) absorb_us = 20'hFFFFF;   // no excess
    else absorb_us = (20'(free_cells) * 20'(NS_PER_CELL)) /
                     (20'(oversub_ratio) - 20'd1) / 20'd1000;
  end

  assign under_100us = (absorb_us < 20'd100);
  assign under_10us  = (absorb_us < 20'd10);

  logic [19:0] burst_ns;
  logic        in_burst;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      worst_absorb_us     <= 20'hFFFFF;
      burst_ns            <= '0;
      in_burst            <= 1'b0;
      last_burst_us       <= '0;
      longest_burst_us    <= '0;
      c_bursts            <= '0;
      buffer_was_adequate <= 1'b1;
    end else begin
      if (sample && (absorb_us < worst_absorb_us))
        worst_absorb_us <= absorb_us;

      // MEASURE THE BURSTS THAT ACTUALLY ARRIVE, not the ones the buffer
      // was sized for. A buffer sized for 1 ms on a network whose bursts
      // are 20 us is oversized by 50x, and nothing else says so.
      if (fill_started && !in_burst) begin
        in_burst <= 1'b1;
        burst_ns <= '0;
      end else if (in_burst) begin
        burst_ns <= burst_ns + 20'd8;      // 8 ns per cycle at 125 MHz
        if (fill_ended) begin
          in_burst      <= 1'b0;
          last_burst_us <= burst_ns / 20'd1000;
          if ((burst_ns / 20'd1000) > longest_burst_us)
            longest_burst_us <= burst_ns / 20'd1000;
          if (!(&c_bursts)) c_bursts <= c_bursts + 1'b1;
        end
      end

      // The buffer was adequate if it absorbed every burst that arrived.
      buffer_was_adequate <= (longest_burst_us < worst_absorb_us);
    end
  end

endmodule

Classification: synthesizable.

What it teaches: that a buffer's size is two variables collapsed into one number, and separating them is the whole design conversation. A buffer absorbs a rate mismatch for a duration, and size = excess_rate × duration. Quoting 512 KiB says neither; quoting 4.2 ms at 2:1 or 182 µs at 24:1 says both — and those are the same 512 KiB.

And longest_burst_us against worst_absorb_us is the only honest sizing evidence. A buffer sized for a millisecond on a network whose real bursts are 20 µs is oversized by 50×, paying Chapter 12.6 §13's reservation cost and Section 12's delay for headroom nothing uses. Measuring the bursts that actually arrive is the only way to know, and no datasheet figure substitutes.

Deliberately simplified: a fixed cycle time in the burst timer and integer division for the absorption estimate. A production model computes the estimate once per window rather than combinationally, since a 20-bit divide is not free.

Production implication: under_10us deserves an alert and under_100us deserves a graph. A buffer that will absorb less than 10 µs of the current oversubscription is one burst away from dropping, and 10 µs is roughly one maximum-length frame at 1 Gb/s — so the buffer has effectively no headroom left. The signal fires while the port is still passing traffic cleanly, which is the only useful time for it to fire.

14. What a Buffer Actually Buys, in Time

Convert every plausible buffer size into the duration it absorbs at three oversubscription ratios, because the durations are shorter than the sizes suggest.

Buffer2:14:124:1
32 KiB262 µs87 µs11 µs
122 KiBChapter 12.1 §6999 µs333 µs44 µs
512 KiB4.19 ms1.40 ms182 µs
1 MiB8.39 ms2.80 ms365 µs

Read the right-hand column. At Chapter 12.1 §6's limit case of 23 other ports offering line rate to one, a megabyte of buffer absorbs 365 microseconds — and then drops at 95.7%, exactly as that chapter's arithmetic said it would.

Which sets the honest expectation for what buffering achieves. It does not fix oversubscription; Chapter 12.1 §6 established that nothing does. What it does is absorb transients — a simultaneous burst from several sources that ends before the buffer fills — and the whole design question is whether real bursts are shorter than the buffer's duration.

And the left-hand column is the one worth reading against Section 12's delay table. A 1 MiB buffer at 2:1 absorbs 8.39 ms, and a frame arriving when it is full waits 8.39 ms to be transmitted. Those are the same number — Little's law — so a buffer sized to absorb an 8 ms burst has committed to an 8 ms tail latency for anything that arrives during one.

15. RTL 7 — Congestion Telemetry

Six mechanisms, and one place to read whether this switch is congested, who is causing it, and how close the buffer is to its limit.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// congestion_telemetry -- the per-window state of the buffer subsystem.
//
// Every quantity is derived from signals the datapath already has. The
// value is that a single "discards" figure conflates six unrelated
// conditions, and each of the six has a different remedy.
// -----------------------------------------------------------------------
module congestion_telemetry
  import congest_pkg::*;
#(
  parameter int N_PORTS = 24,
  parameter int CNT_W   = 32,
  parameter int WIN     = 1_000_000
)(
  input  logic             clk,
  input  logic             rst_n,

  input  logic             frame_valid,
  input  logic             drop_valid,
  input  enq_fail_e        drop_reason,
  input  logic [17:0]      shared_free,
  input  logic [17:0]      shared_total,
  input  logic [15:0]      largest_share_pct,
  input  logic [19:0]      p99_delay_ns,
  input  logic [19:0]      worst_absorb_us,
  input  logic             pool_exhausted,

  output logic             window_valid,
  output logic [15:0]      drop_rate_ppm,
  output logic [15:0]      pool_free_pct,
  output logic [15:0]      pool_exhausted_ppm,
  output logic             congested,
  output logic             one_port_dominant,
  output logic             delay_bound_exceeded,
  output logic             headroom_critical,
  output logic [CNT_W-1:0] c_frames,
  output logic [CNT_W-1:0] c_drops
);

  logic [CNT_W-1:0] win, win_drop, win_exh;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      win <= '0; win_drop <= '0; win_exh <= '0;
      c_frames <= '0; c_drops <= '0;
      window_valid         <= 1'b0;
      drop_rate_ppm        <= '0;
      pool_free_pct        <= 16'd100;
      pool_exhausted_ppm   <= '0;
      congested            <= 1'b0;
      one_port_dominant    <= 1'b0;
      delay_bound_exceeded <= 1'b0;
      headroom_critical    <= 1'b0;
    end else begin
      window_valid <= 1'b0;

      if (frame_valid) begin win <= win + 1'b1; c_frames <= c_frames + 1'b1; end
      if (drop_valid)  begin win_drop <= win_drop + 1'b1; c_drops <= c_drops + 1'b1; end
      if (pool_exhausted) win_exh <= win_exh + 1'b1;

      if (win >= CNT_W'(WIN)) begin
        // PARTS PER MILLION, not percent. A drop rate of 0.05% is 500 ppm
        // and matters; as an integer percentage it is zero.
        drop_rate_ppm      <= 16'((win_drop * CNT_W'(1_000_000)) / win);
        pool_exhausted_ppm <= 16'((win_exh  * CNT_W'(1_000_000)) / win);
        pool_free_pct      <= 16'((18'(shared_free) * 18'd100) / shared_total);

        congested            <= (win_drop != '0) || (win_exh != '0);
        // ONE PORT HOLDING MORE THAN HALF THE POOL. Section 8's condition,
        // as a bit.
        one_port_dominant    <= (largest_share_pct > 16'd50);
        // A millisecond of queueing -- Section 12 puts this at about 99%
        // utilisation, which a five-minute average reports as 70%.
        delay_bound_exceeded <= (p99_delay_ns >= 20'd1000000);
        // Less than ten microseconds of absorption left: roughly one
        // maximum-length frame at 1 Gb/s.
        headroom_critical    <= (worst_absorb_us < 20'd10);

        win <= '0; win_drop <= '0; win_exh <= '0;
        window_valid <= 1'b1;
      end
    end
  end

endmodule

Classification: synthesizable.

What it teaches: that the drop rate belongs in parts per million rather than percent, and the reason is the same one Chapter 13.2 §11 gave for scaling tag overhead by 10 000. A drop rate of 0.05% is 500 ppm and is a serious problem for a storage protocol; as an integer percentage it is zero — and a metric that reads zero in the range where it matters is a metric nobody reads.

And the four derived bits are four different conditions with four different remedies. congested says frames are being lost. one_port_dominant says one port is holding the pool — Section 8. delay_bound_exceeded says the queueing delay has passed a millisecond, which is a latency problem even if nothing is dropping. And headroom_critical says the next burst will drop, which fires while the port is still clean.

Deliberately simplified: one global window. Production telemetry keeps the window per port and the pool statistics globally, because the pool is one resource and the queues are twenty-four.

Production implication: delay_bound_exceeded catches the failure that has no drops at all. Section 12's table shows a millisecond of queueing arrives at about 99% utilisation — a link that is passing every frame, dropping nothing, and adding a millisecond to every round trip. No drop counter fires, no error is logged, and every latency-sensitive application on that path is failing its budget. It is the one condition in this chapter that a loss-oriented investigation never finds.

16. The Host Is a Buffer Too

Section 2 counted four buffers and this chapter has spent five sections on two of them. The fourth is the one that drops most frames on a healthy network, and no switch counter sees it.

A receiving station holds arriving frames in a descriptor ring — a few hundred entries, refilled by the driver as it consumes them. When the ring is full the network card discards, and the discard is counted in a driver statistic.

switch bufferhost descriptor ring
sizemegabyteshundreds of frames
at 1 Gb/s minimum frames12 MiB ≈ 98 000 frames512 frames — 344 µs
overflows becauseoffered load exceeds an egress ratethe CPU did not run
duration of the causemilliseconds to indefinitelymicroseconds
loss patterna predictable fraction, sustainedbursty and opportunistic
reported wherea management interfacea driver counter nobody polls

The middle rows are the important ones. A switch drops because of an arithmetic condition that persists — Chapter 12.1 §6's rate mismatch. A host drops because of a scheduling condition that lasts microseconds — an interrupt coalescing timer, a scheduler delay, a page fault in a driver — and then clears.

And Chapter 12.4 §8 established the one case where the two meet. A broadcast storm puts every host at 7.4 cores of interrupt load, so every host's ring overflows continuously — while the switch, forwarding correctly at 6.7% link utilisation, reports nothing at all.

Which is why an investigation should establish which buffer before examining any of them, and the establishing question is cheap: does the receiver's driver report ring-full discards? If yes, the network is delivering more than the host can take and no switch change helps. If no, and frames are still missing, the loss is upstream — and Sections 9 and 10 say how to find which port caused it.

17. RTL 8 — Conformance for a Shared Resource

The monitor's difficulty is that the property it most needs to check — that the pool is not over-committed — is one the design deliberately violates.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// -----------------------------------------------------------------------
// congestion_conformance_monitor -- checks a buffer subsystem whose
// design premise is that it promises more than it has.
//
// The invariant is NOT that allocations sum to the pool. It is that the
// RESERVES do, that no port exceeds its limit, that accounting balances,
// and that every drop is attributed. Section 18 is about the difference.
// -----------------------------------------------------------------------
module congestion_conformance_monitor
  import congest_pkg::*;
#(
  parameter int N_PORTS       = 24,
  parameter int POOL_CELLS    = 98304,
  parameter int RESERVE_CELLS = 512,
  parameter int LIMIT_CELLS   = 24576,
  parameter int CNT_W         = 32
)(
  input  logic             clk,
  input  logic             rst_n,

  input  logic [17:0]      held [N_PORTS],
  input  logic [17:0]      shared_used,
  input  logic             grant,
  input  logic [4:0]       req_port,
  input  logic             drop_valid,
  input  enq_fail_e        drop_reason,
  input  logic             drop_attributed,

  output logic [CNT_W-1:0] v_over_limit,        // a port exceeded its limit
  output logic [CNT_W-1:0] v_over_pool,         // held exceeds what exists
  output logic [CNT_W-1:0] v_reserve_denied,    // refused within the reserve
  output logic [CNT_W-1:0] v_accounting_drift,
  output logic [CNT_W-1:0] v_drop_unattributed,
  output logic             reserves_fit,        // a STANDING property
  output logic             conformant
);

  logic [17:0] total_held;

  always_comb begin
    total_held = 18'd0;
    for (int p = 0; p < N_PORTS; p++) total_held = total_held + held[p];
  end

  // THE STANDING PROPERTY THAT IS TRUE. The reserves must fit; the
  // LIMITS deliberately do not -- Section 18.
  assign reserves_fit = ((N_PORTS * RESERVE_CELLS) <= POOL_CELLS);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      v_over_limit        <= '0;
      v_over_pool         <= '0;
      v_reserve_denied    <= '0;
      v_accounting_drift  <= '0;
      v_drop_unattributed <= '0;
    end else begin
      // No port holds more than its reserve plus its limit.
      for (int p = 0; p < N_PORTS; p++)
        if (held[p] > 18'(RESERVE_CELLS + LIMIT_CELLS))
          if (!(&v_over_limit)) v_over_limit <= v_over_limit + 1'b1;

      // THE REAL CONSERVATION LAW. What is held cannot exceed what
      // exists -- unlike the sum of ALLOWANCES, which is 612% of it.
      if (total_held > 18'(POOL_CELLS))
        if (!(&v_over_pool)) v_over_pool <= v_over_pool + 1'b1;

      // THE GUARANTEE. A request within a port's reserve is never
      // refused, whatever the pool's state. This is what makes the
      // over-commitment safe.
      if (drop_valid && (held[req_port] < 18'(RESERVE_CELLS)))
        if (!(&v_reserve_denied)) v_reserve_denied <= v_reserve_denied + 1'b1;

      // Shared accounting must match the sum of holdings above reserve.
      begin
        automatic logic [17:0] sh = 18'd0;
        for (int p = 0; p < N_PORTS; p++)
          if (held[p] > 18'(RESERVE_CELLS))
            sh = sh + (held[p] - 18'(RESERVE_CELLS));
        if (sh != shared_used)
          if (!(&v_accounting_drift))
            v_accounting_drift <= v_accounting_drift + 1'b1;
      end

      // Every drop is attributed -- Section 9. An unattributed drop is a
      // number with no finding attached to it.
      if (drop_valid && !drop_attributed)
        if (!(&v_drop_unattributed))
          v_drop_unattributed <= v_drop_unattributed + 1'b1;
    end
  end

  assign conformant = (v_over_limit       == '0) && (v_over_pool        == '0) &&
                      (v_reserve_denied   == '0) && (v_accounting_drift == '0) &&
                      (v_drop_unattributed == '0) && reserves_fit;

endmodule

Classification: synthesizable, and intended to stay in silicon.

What it teaches: the distinction between the conservation law that is true and the budget property that is not. total_held ≤ POOL_CELLS is a real invariant — the switch cannot hold more memory than exists, and a violation is a bug in the accounting. Σ (reserve + limit) ≤ POOL_CELLS is false by 612% and is the design working. They look like the same kind of statement and only one of them is checkable.

And v_reserve_denied is the guarantee that makes the over-commitment defensible. A port below its reserve must never be refused, whatever the pool's state — so an exhausted pool degrades every port's throughput and disconnects none of them. Without that guarantee, over-committing would mean a port could be unable to enqueue anything at all, which is a different and much worse failure than a reduced share.

Deliberately simplified: a full 24-way sum every cycle for the accounting check. A production monitor maintains a running total incrementally and reconciles it against a full sweep periodically, which is Chapter 12.5 §17's v_occupancy_drift argument applied to cells instead of entries.

Production implication: reserves_fit is a compile-time property exposed as a signal, and it is the one number a buffer configuration can get catastrophically wrong. A design whose reserves alone exceed the pool cannot honour its own guarantee — some port will be refused within its reserve, at which point the over-commitment is no longer safe and the whole argument for sharing collapses. It costs a comparison of two constants and it is checkable before the first frame.

Mean queueing delay for maximum length frames on a one gigabit link grows hyperbolically with utilisation. At fifty percent utilisation a frame waits about twelve microseconds, which is one frame's service time. At ninety percent it waits one hundred and nine microseconds, which is nine frames. At ninety-nine percent it waits one thousand two hundred microseconds, which is ninety-nine frames and more than a millisecond. From fifty to ninety percent the delay grows nine times, and from ninety to ninety-nine percent it grows another eleven times, so nearly all of the delay occurs in the last few percent of utilisation. This is why cut-through switching's benefit vanishes under load, since at ninety percent utilisation the queueing term is nine times the serialisation term that cut-through attacks. It is also why averaging destroys the measurement: a link at ninety-nine percent for six seconds and idle for the rest of a five minute interval averages two percent utilisation while every frame crossing during those six seconds was more than a millisecond late.50% — 12.1 µsone frame70% — 28.3 µs2.3 frames90% — 109.3 µs9 frames99% — 1202 µs99 framesCut-through attacks12.144 µsthe smaller term at 90%Five-minute averagereports 2% for a 99%burstA p99 histogram seesita mean does not12
Figure 3 — queueing delay is hyperbolic in utilisation, and averaging over any interval longer than the burst hides all of it.
A shared buffer subsystem has three accounting statements that look alike and are not. The first is a real conservation law: the total memory held across all ports can never exceed the pool that exists, and a violation is a bug in the accounting. The second is a budget property that is deliberately false: the sum of every port's reserve plus its limit is six hundred and twelve percent of the pool, because the design's entire value comes from promising more than it has on the expectation that not every port is congested at once. The third is the guarantee that makes the over-commitment safe: a request within a port's own reserve is never refused, whatever the pool's state, so an exhausted pool degrades every port's throughput rather than disconnecting any port. Asserting the second statement as though it were the first produces a design that either wastes six times the memory or fails an assertion continuously while working correctly.Held ≤ poola real conservation lawCheckable, and itholdsa violation is anaccounting bugΣ allowances ≤ poolfalse by 612% —deliberatelyThe over-commitmentis the valuenot every port is busy atonceWithin the reserve:never refusedthe guaranteeExhaustion degrades,never disconnectswhat makes it defensibleAssert the budgetinstead6× the memory, or afailing assertion12
Figure 4 — the conservation law that holds, the budget property that does not, and the guarantee that makes the difference safe.

18. Properties Worth Asserting, and One Worth Refusing

The conservation law holds; the budget property does not. Every property here is on the right side of that line, and the rejected one is on the wrong side.

The queue

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P1. A frame is enqueued only when both the port's limit and the pool
// permit it. Two independent conditions, two distinct refusals.
property p_enqueue_needs_both;
  @(posedge clk) disable iff (!rst_n)
  (enq_valid && enq_accept) |-> (pool_granted &&
    ((occupancy_cells + 16'(enq_cells)) <= 16'(MAX_CELLS)));
endproperty
a_enqueue_conditions: assert property (p_enqueue_needs_both);

// P2. THE TWO REFUSALS ARE KEPT APART. This port's own limit is this
// port's congestion; an exhausted pool is somebody else's.
property p_refusals_distinguished;
  @(posedge clk) disable iff (!rst_n)
  (enq_valid && !enq_accept)
    |-> (enq_fail inside {EF_QUEUE_LIMIT, EF_POOL_EMPTY, EF_NO_RESERVE});
endproperty
a_refusal_classified: assert property (p_refusals_distinguished);

// P3. Occupancy never exceeds the port's limit.
property p_occupancy_bounded;
  @(posedge clk) disable iff (!rst_n)
  (occupancy_cells <= 16'(MAX_CELLS));
endproperty
a_occupancy_bounded: assert property (p_occupancy_bounded);

// P4. Watermarks trigger at the HIGH threshold, never at full. A
// mechanism that waits for full has already lost the frames in flight
// during its signal's propagation -- Chapter 14.2 quantifies them.
property p_high_watermark_before_full;
  @(posedge clk) disable iff (!rst_n)
  above_high |-> (occupancy_cells < 16'(MAX_CELLS)) or (watermark == WM_FULL);
endproperty
a_watermark_early: assert property (p_high_watermark_before_full);

// P5. Peak occupancy is retained, not averaged away.
property p_peak_monotonic;
  @(posedge clk) disable iff (!rst_n)
  $stable(rst_n) |-> (peak_cells >= $past(peak_cells));
endproperty
a_peak_retained: assert property (p_peak_monotonic);

The pool

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P6. THE CONSERVATION LAW. What is held cannot exceed what exists.
// This is the invariant that IS true -- contrast the budget property in
// the rejected callout.
property p_held_within_pool;
  @(posedge clk) disable iff (!rst_n)
  (total_held <= 18'(POOL_CELLS));
endproperty
a_conservation: assert property (p_held_within_pool);

// P7. THE GUARANTEE. A request within a port's own reserve is NEVER
// refused, whatever the pool's state. This is what makes the
// over-commitment safe.
property p_reserve_always_granted;
  @(posedge clk) disable iff (!rst_n)
  (req_valid && ((held[req_port] + 18'(req_cells)) <= 18'(RESERVE_CELLS)))
    |-> grant;
endproperty
a_reserve_honoured: assert property (p_reserve_always_granted);

// P8. No port exceeds its reserve plus its share limit.
property p_limit_enforced;
  @(posedge clk) disable iff (!rst_n)
  (held[req_port] <= 18'(RESERVE_CELLS + LIMIT_CELLS));
endproperty
a_limit_enforced: assert property (p_limit_enforced);

// P9. The reserves fit the pool. THIS budget property is true and must
// be -- a design whose reserves alone exceed the pool cannot honour P7.
property p_reserves_fit;
  @(posedge clk) disable iff (!rst_n)
  reserves_fit;
endproperty
a_reserves_fit: assert property (p_reserves_fit);

// P10. Shared accounting matches the sum of holdings above reserve.
property p_shared_accounting_balances;
  @(posedge clk) disable iff (!rst_n)
  (v_accounting_drift == $past(v_accounting_drift));
endproperty
a_accounting_balances: assert property (p_shared_accounting_balances);

// P11. A release returns exactly what was taken.
property p_release_matches_grant;
  @(posedge clk) disable iff (!rst_n)
  (rel_valid && (rel_cells != 8'd0)) |-> (held[rel_port] >= 18'(rel_cells));
endproperty
a_release_sane: assert property (p_release_matches_grant);

Attribution

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P12. Every drop is attributed to an egress port, an ingress port and,
// when the pool was the cause, a holder.
property p_every_drop_attributed;
  @(posedge clk) disable iff (!rst_n)
  drop_valid |-> drop_attributed;
endproperty
a_drop_attributed: assert property (p_every_drop_attributed);

// P13. A pool-exhaustion drop is charged to the HOLDER, not to the port
// that could not allocate -- Section 10's whole point.
property p_pool_drop_charged_to_holder;
  @(posedge clk) disable iff (!rst_n)
  (drop_valid && (drop_reason == EF_POOL_EMPTY))
    |=> (pool_drops_caused[$past(pool_holder)] >
         $past(pool_drops_caused[$past(pool_holder)]));
endproperty
a_charged_to_holder: assert property (p_pool_drop_charged_to_holder);

// P14. The holder is sampled AT THE DROP. A periodic sample says who is
// usually busy, not who held the memory this frame needed.
property p_holder_sampled_at_drop;
  @(posedge clk) disable iff (!rst_n)
  (drop_valid && (drop_reason == EF_POOL_EMPTY))
    |-> (pool_holder == largest_holder);
endproperty
a_sampled_at_drop: assert property (p_holder_sampled_at_drop);

// P15. victim_is_not_culprit is high exactly when the port that dropped
// is not the port holding the pool.
property p_victim_culprit_distinguished;
  @(posedge clk) disable iff (!rst_n)
  (drop_valid && (drop_reason == EF_POOL_EMPTY))
    |=> (victim_is_not_culprit == ($past(drop_port) != $past(pool_holder)));
endproperty
a_victim_culprit: assert property (p_victim_culprit_distinguished);

// P16. Exactly one reason counter moves per drop.
property p_one_reason_per_drop;
  @(posedge clk) disable iff (!rst_n)
  drop_valid |=> ($countones({$changed(c_by_reason[1]), $changed(c_by_reason[2]),
                              $changed(c_by_reason[3]), $changed(c_by_reason[4])}) == 1);
endproperty
a_one_reason: assert property (p_one_reason_per_drop);

Delay and absorption

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P17. LITTLE'S LAW, as an identity. Delay is occupancy divided by the
// drain rate, exactly, for a work-conserving server.
property p_delay_is_occupancy_over_rate;
  @(posedge clk) disable iff (!rst_n)
  sample |-> (delay_ns == (20'(occupancy_cells) * 20'(NS_PER_CELL)));
endproperty
a_littles_law: assert property (p_delay_is_occupancy_over_rate);

// P18. Delay is reported as a HISTOGRAM, not a mean. Chapter 12.6
// Section 6: jitter is what real-time traffic is engineered against, and
// a mean hides the tail.
property p_histogram_maintained;
  @(posedge clk) disable iff (!rst_n)
  sample |=> ($changed(hist[0]) || $changed(hist[1]) || $changed(hist[2]) ||
              $changed(hist[3]) || $changed(hist[4]) || $changed(hist[5]) ||
              $changed(hist[6]) || $changed(hist[7]));
endproperty
a_histogram: assert property (p_histogram_maintained);

// P19. A deeper buffer means a longer wait at full occupancy. The two
// are the same number, which is why "add buffer" is a latency decision.
property p_deeper_buffer_is_longer_wait;
  @(posedge clk) disable iff (!rst_n)
  (sample && (occupancy_cells > $past(occupancy_cells)))
    |-> (delay_ns > $past(delay_ns));
endproperty
a_buffer_is_delay: assert property (p_deeper_buffer_is_longer_wait);

// P20. Absorption time is free cells divided by the excess rate -- a
// buffer's worth is a DURATION, not a size.
property p_absorption_is_a_duration;
  @(posedge clk) disable iff (!rst_n)
  (sample && (oversub_ratio > 6'd1))
    |-> (absorb_us == ((20'(free_cells) * 20'(NS_PER_CELL)) /
                       (20'(oversub_ratio) - 20'd1) / 20'd1000));
endproperty
a_absorption: assert property (p_absorption_is_a_duration);

// P21. The bursts that ACTUALLY arrive are measured, not the ones the
// buffer was sized for.
property p_real_bursts_measured;
  @(posedge clk) disable iff (!rst_n)
  fill_ended |=> (c_bursts > $past(c_bursts));
endproperty
a_bursts_measured: assert property (p_real_bursts_measured);

Telemetry and conformance

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// P22. The drop rate is in parts per million. 0.05% is 500 ppm and
// matters; as an integer percentage it is zero.
property p_drop_rate_has_resolution;
  @(posedge clk) disable iff (!rst_n)
  (window_valid && (c_drops != '0)) |-> (drop_rate_ppm != 16'd0);
endproperty
a_ppm_resolution: assert property (p_drop_rate_has_resolution);

// P23. A latency violation is reported even when nothing is dropping --
// the one condition a loss-oriented investigation never finds.
property p_delay_reported_without_loss;
  @(posedge clk) disable iff (!rst_n)
  (window_valid && (p99_delay_ns >= 20'd1000000)) |-> delay_bound_exceeded;
endproperty
a_latency_visible: assert property (p_delay_reported_without_loss);

// P24. headroom_critical fires while the port is still clean, which is
// the only useful time.
property p_headroom_warns_early;
  @(posedge clk) disable iff (!rst_n)
  (window_valid && (worst_absorb_us < 20'd10)) |-> headroom_critical;
endproperty
a_headroom_early: assert property (p_headroom_warns_early);

// P25. One port holding more than half the pool is reported as such.
property p_dominance_reported;
  @(posedge clk) disable iff (!rst_n)
  (window_valid && (largest_share_pct > 16'd50)) |-> one_port_dominant;
endproperty
a_dominance: assert property (p_dominance_reported);

// P26. Conformance INCLUDES reserves_fit -- a standing property of the
// configuration, not a history of events.
property p_conformant_includes_reserves;
  @(posedge clk) disable iff (!rst_n)
  conformant |-> reserves_fit;
endproperty
a_conformant_reserves: assert property (p_conformant_includes_reserves);

// P27. Conformance means the accounting is sound and every drop is
// attributed -- never that no frame was lost.
property p_conformant_definition;
  @(posedge clk) disable iff (!rst_n)
  conformant |-> ((v_over_limit == '0) && (v_over_pool == '0) &&
                  (v_reserve_denied == '0) && (v_accounting_drift == '0) &&
                  (v_drop_unattributed == '0));
endproperty
a_conformant_def: assert property (p_conformant_definition);

// P28. Chapter 12.1's discard policy survives: a drop happens only when
// there is genuinely nowhere to put the frame.
property p_drop_only_when_no_room;
  @(posedge clk) disable iff (!rst_n)
  drop_valid |-> ((occupancy_cells + 16'(enq_cells) > 16'(MAX_CELLS)) ||
                  (shared_free < 18'(enq_cells)));
endproperty
a_drop_justified: assert property (p_drop_only_when_no_room);

// P29. An exhausted pool degrades every port and disconnects none --
// P7's guarantee, stated as an outcome.
property p_exhaustion_degrades_not_disconnects;
  @(posedge clk) disable iff (!rst_n)
  pool_exhausted |-> (held[req_port] >= 18'd1 || grant);
endproperty
a_degrade_not_cut: assert property (p_exhaustion_degrades_not_disconnects);

// P30. A port's own limit refusal is counted separately from a pool
// refusal, because the remedies are in different places.
property p_limit_and_pool_counted_apart;
  @(posedge clk) disable iff (!rst_n)
  (drop_valid && (drop_reason == EF_QUEUE_LIMIT)) |-> $stable(c_refuse_pool);
endproperty
a_counted_apart: assert property (p_limit_and_pool_counted_apart);

19. Verification Scenarios

Seventy-one scenarios. Several have expected outcomes in which a correct switch drops frames, and one has an expected outcome in which the port causing the problem shows no drops at all.

The queue

#ScenarioExpected
1Enqueue with room and a pool grantaccepted
2Enqueue at the port's limitEF_QUEUE_LIMIT
3Enqueue with room, pool exhaustedEF_POOL_EMPTY
4The two refusals over a runcounted separately
5Occupancy at HIGH_CELLSabove_high, WM_HIGH
6Occupancy at LOW_CELLSbelow_low
7Occupancy at MAX_CELLSWM_FULL — and the pause should already have fired
8Peak occupancy over a runretained, never averaged
9A port whose peak never exceeds 10%its buffer has never been exercised
1064-octet frame, 128-octet cellsconsumes one cell — 2× internal waste

The pool

#ScenarioExpected
11Request within the reserve, pool exhaustedgranted — the guarantee
12Request above the reserve, shared region freegranted
13Request above the reserve, at the port's limitEF_QUEUE_LIMIT
14Request above the reserve, shared region emptyEF_POOL_EMPTY
15Σ (reserve + limit) against the pool612% — and correct
16Σ reserves against the pool12.5% — must fit, and does
17Total held at any instant≤ the pool, always
18One port at 23:1, no per-port limitconsumes 10.5 MiB in 4.0 ms
19Same, with the 24 576-cell limitconsumes 3 MiB in 1.1 ms
20Other ports' absorption, no limit0.52 ms each
21Other ports' absorption, with the limit2.7 ms each — 5× better
22Releasereturns exactly what was taken
23Shared accounting against the sum above reservebalances

Attribution

#ScenarioExpected
24Any dropattributed to egress, ingress and holder
25Pool-exhaustion dropcharged to the holder, not the victim
26Holder sampledat the drop, not periodically
27Drop on port 3 while port 7 holds 85%victim_port = 3, culprit_port = 7
28Samevictim_is_not_culprit high
29Port 7 holding the poolmay show zero drops of its own
30Ports 1–6 and 8–24scattered EF_POOL_EMPTY drops
31An investigation ranked by drop countstarts at the wrong end
32mostly_somebody_else above halfthis port's counter is not about this port
33A summed "discards" figurecannot distinguish the two remedies

Delay

#ScenarioExpected
343000 cells queued at 1 Gb/s3.07 ms — Little's law
3550% utilisation, 1518-octet frames12.1 µs mean wait
3690% utilisation109.3 µs — 9 frames
3799% utilisation1202 µs — 99 frames
3850% → 90%; 90% → 99% another 11×
39Cut-through at 90% utilisationattacks the 12.144 µs term against 109 µs
4099% for 6 s in a 5-minute windowaverages 2% — the graph shows an idle link
41Delay reported as a meanhides the tail entirely
42Delay reported as a log₂ histogramthe tail is visible
43A millisecond of queueing, nothing droppingdelay_bound_exceeded — no drop counter fires

Absorption

#ScenarioExpected
44122 KiB at 2:1999 µs
45122 KiB at 24:144 µs
461 MiB at 24:1365 µs, then 95.7% discard
47512 KiB at 4:11.40 ms
48A buffer sized for 1 ms, real bursts of 20 µsoversized 50×
49longest_burst_us against worst_absorb_usthe only honest sizing evidence
50Absorption under 10 µsheadroom_critical — one frame of headroom
51Buffer size quoted in byteshides both variables — rate and duration
521 MiB at 2:1 absorbing 8.39 msa frame arriving when full waits 8.39 ms

The host, telemetry and conformance

#ScenarioExpected
53Host ring, 512 descriptors at 1 Gb/s344 µs of minimum frames
54Host loss under a broadcast stormcontinuous, while the switch reports nothing
55Switch link at 6.7%, hosts at 7.4 coresChapter 12.4 §8 — the switch is fine
56An unexplained-loss investigationcheck the receiver's ring first
57Drop rate of 0.05%500 ppm — zero as an integer percentage
58One port above 50% of the poolone_port_dominant
59Reserves exceeding the poolreserves_fit low — the guarantee cannot be honoured
60Held exceeding the poolv_over_pool — an accounting bug
61A refusal within a port's reservev_reserve_denied — the guarantee broken
62Healthy run, one million framesconformant high throughout
63Non-blocking fabric, 23:1 offered to one portstill 95.7% discarded — the fabric is not the constraint
64Dynamic limit α × free, lone congested porttakes most of the pool; contention tightens it automatically
65Same, 22 ports congestedevery port's allowance shrinks without a global computation
66Drop rate reported as an integer percentage0 across the entire diagnostic range
67reserves_fit low at power-onthe guarantee cannot be honoured — before any frame
68512-descriptor host ring at 10 Gb/s34 µs — ten times less headroom than at 1 Gb/s
69Switch loss over a long windowa steady fraction — systematic
70Host loss over the same windowbursty — opportunistic, and the distinguishing property
71Hit rate reported in whole percenttoo coarse — 99% and 99.9% are different networks

What this chapter does not fix

Every number here describes loss and delay. None of it prevents either, and the boundary is worth stating before Chapter 14.2 attempts the prevention.

ProblemDoes buffering help?Why
a transient burst shorter than the absorption timeyes — completelythat is what a buffer is for
sustained oversubscriptionnoChapter 12.1 §6 — arithmetic, not capacity
queueing delayno — it causes itLittle's law: W = L ÷ λ
the host's ring overflowingnoa different buffer, for a different reason
a drop counter pointing at the wrong portno — attribution doesSections 9 and 10

The first row is the whole value and the second is the whole limit. A buffer converts a burst into a delay and a sustained excess into a delayed discard — and Section 14's table shows that at 24:1 even a megabyte buys 365 µs before the arithmetic reasserts itself.

Which leaves exactly one avenue unexplored: tell the sender to stop. That is Chapter 14.2's subject, and Chapter 12.1 §6 already predicted how it goes.

20. Debugging Congestion

Every row produces a switch forwarding correctly. Several produce a drop counter that points at the wrong port, and one produces no drops at all.

SymptomLikely causeThe observable that decides it
Loss scattered across many portsone port holding the poollargest_holder, pool_drops_caused
Loss on one port onlythat port is oversubscribedEF_QUEUE_LIMIT dominant — capacity, here
The busiest port shows no dropsit is the culprit and is winning allocationspool_drops_caused inverts the ranking
Latency spikes, zero dropsqueueing at ~99% utilisationdelay_bound_exceeded, p99_delay_ns
Utilisation graph shows 2%, users report lossa 99% burst averaged awaythe delay histogram, not the mean
Loss appears and clears between pollsa 4 ms pool exhaustionpool_exhausted_ppm — the event was real
Adding buffer fixed the loss and broke latencyLittle's law — they are the same numberdelay_ns at the new occupancy
One port's absorption dropped without a config changeanother port raised its holdingshared_free, largest_share_pct
Loss under a broadcast storm, switch cleanthe hosts' rings, not the networkChapter 12.4 §8 — 7.4 cores per host
Loss with no switch counter movingthe receiver's descriptor ringask the driver, not the switch
A port refused within its reservethe guarantee is brokenv_reserve_denied — over-commitment is now unsafe
Every port at its 512-cell reserveone port took the whole shared regionno per-port limit configured
Buffer refused while the pool sits mostly idlea fixed per-port limit, too mean when uncontendeda dynamic limit α × free scales with what is free
Every port's allowance shrank without a config changea dynamic limit, working as intendedshared_free — the mechanism is self-limiting
A metric that has read zero since it was addedthe ratio's scale is wrong for its rangeppm for drop rates, hundredths of a percent for overhead
Loss began after a switch was replaced with a faster onea non-blocking fabric delivers congestion fasterSection 4's callout — the buffer now fills at the full excess rate
A steady loss percentage across many hourssystematic — the networkswitch loss is a predictable fraction; host loss is bursty
Buffer adequate on the bench, dropping in productionreal bursts exceed worst_absorb_uslongest_burst_us — measure the bursts that arrive

21. Common Misconceptions

1 — "Each port has its own buffer."

The wrong model: a port's buffer is a partition of memory belonging to it.

What it costs: every conclusion drawn from a drop counter. On a shared-buffer switch a port's "buffer" is an accounting entry against a pool, so a congested port does not fill its own memory — it consumes everybody's, and at 23:1 a 12 MiB pool is gone in 4.4 ms.

The corrected model: three regions — a per-port reserve that is guaranteed, a shared region any port may draw from, and a per-port limit on how much of the shared region one port may hold. The reserve is what makes an exhausted pool degrade throughput rather than disconnect a port, and the limit is what stops one port taking everything.

2 — "The sum of the port buffers should equal the total buffer."

The wrong model: the accounting must balance.

What it costs: either six times the memory or the loss of sharing. Σ (reserve + limit) on a 24-port switch is 602 112 cells against a 98 304-cell pool — 612% — and that over-commitment is the design. Making it balance means either sizing the pool at 73.5 MiB, or cutting every port to 4096 cells and abandoning the ability to borrow idle ports' memory.

The corrected model: the invariant that holds is total_held ≤ POOL_CELLS — what is actually held can never exceed what exists. The sum of the allowances is a feature, not a bug, and Section 18's rejected property confuses the two.

3 — "A drop counter tells you which port is congested."

The wrong model: the port reporting drops is the port with the problem.

What it costs: an investigation that examines twenty-three innocent ports. The drop lands on whichever port asked for a cell when the pool ran out — chosen by arrival order — while the congested port is winning its allocations and may show no drops at all.

The corrected model: a drop needs three attributions: where the enqueue failed, whose traffic it was, and whose memory it was waiting for. pool_drops_caused charges pool exhaustion to the holder, which inverts the ranking back to the useful order.

4 — "More buffer means less loss."

The wrong model: buffering is free, so deeper is better.

What it costs: latency, at a fixed exchange rate. Little's law is an identity here: W = L ÷ λ. A buffer that absorbs an 8.39 ms burst makes a frame arriving when it is full wait 8.39 ms — the same number. Chapter 12.1 §6 said a frame delivered after tens of milliseconds is often less useful than one discarded promptly, and this is the arithmetic.

The corrected model: a buffer converts loss into delay, and which is preferable depends entirely on the traffic. Bulk transfer wants depth; real-time traffic wants shallowness, because a frame past its deadline must be discarded anyway having consumed the bandwidth twice. Chapter 14.4's per-class treatment exists because one depth cannot serve both.

5 — "A quiet drop counter means no congestion."

The wrong model: if nothing is dropping, the link is healthy.

What it costs: the failure that has no drops at all. Section 12's table puts a millisecond of queueing delay at about 99% utilisation — a link passing every frame, dropping nothing, and adding a millisecond to every round trip. No counter fires, no error is logged, and every latency-sensitive application on that path is missing its budget.

The corrected model: congestion has two symptoms and loss is the second one. Delay arrives first, grows hyperbolically, and is invisible to a loss-oriented investigation. delay_bound_exceeded and a p99 histogram are what see it.

6 — "The network is where packets are lost."

The wrong model: unexplained loss means a network problem.

What it costs: the wrong investigation. A switch discards only when offered load exceeds an egress rate for longer than the buffer absorbs — 999 µs at 2:1 with a 122 KiB buffer — which real traffic rarely sustains outside a genuine capacity problem. A host discards whenever its 512-descriptor ring is full, which happens on a microsecond scale for scheduling reasons.

The corrected model: the default hypothesis should be the receiver, and the switch should be ruled out. The distinguishing property is that switch loss is systematicChapter 12.1 §6's predictable fraction, sustained — while host loss is opportunistic. Ask the driver for its ring-full count before opening a switch's management interface.

22. Interview Reasoning

Q1 — "Where does a switch actually drop a frame?"

Reason through it. At the egress, because that is the only point where several sources converge on one server — every other stage is sized to keep up by construction, and a backlog at the ingress FIFO or the lookup means a design failing its own specification rather than congestion. The strong answer then complicates it: on a shared-buffer switch the egress queue is a list of pointers into one pool, so the memory that runs out belongs to everybody. A congested port consumes the pool — at 23:1 a 12 MiB pool in 4.4 ms — and the drop is recorded on whichever port was enqueueing when it ran out, which is chosen by arrival order and is usually not the congested one.

Q2 — "The sum of your per-port buffer allowances is 612% of the pool. Is that a bug?"

Reason through it. No — it is the reason a shared buffer exists. A pool sized so every port could hold its full allowance simultaneously would need 73.5 MiB instead of 12 — six times the memory — to serve a case in which every port is congested at once, at which point Chapter 12.1 §6 established that no buffer helps. The strong answer names the invariant that is true: total_held ≤ POOL_CELLS, because the switch cannot hold more memory than exists. And it names what makes the over-commitment safe: a per-port reserve that is never refused, so an exhausted pool degrades every port's share and disconnects none — and a per-port limit, without which one congested port reduces the other 23 from 2.7 ms of absorption to 0.52.

Q3 — "Your busiest port shows fewer drops than several idle ones. Explain."

Reason through it. The busiest port is the culprit and it is winning its allocations. It already holds the memory; its enqueues succeed until its own limit stops them, and those refusals are counted as EF_QUEUE_LIMIT rather than pool exhaustion. Every other port's occasional request finds the shared region empty and is counted as EF_POOL_EMPTY. The strong answer names the repair: attribute each pool-exhaustion drop to whoever was holding the pool at that instant, not to the port refused — pool_drops_causedwhich reverses the ranking back to the useful order. And it notes why the holder must be sampled at the drop: a periodic sample says who is usually busy, not who held the memory this frame needed.

Q4 — "Latency is up, nothing is dropping, and utilisation reads 70%. What is happening?"

Reason through it. Queueing delay at a utilisation the average is hiding. Delay is hyperbolic: 12.1 µs at 50%, 109 µs at 90%, 1202 µs at 99% for maximum-length frames at 1 Gb/s. A link at 99% for six seconds in a five-minute window averages 2% — and every frame in those six seconds was more than a millisecond late. The strong answer names the measurement that sees it: a log₂ histogram of occupancy ÷ drain rate, not a mean — Chapter 12.6 §6 established that jitter, not mean latency, is what real-time traffic is engineered against, and a mean of 40 µs with a 3 ms tail and a flat 40 µs are the same number and completely different networks.

Q5 — "A colleague proposes doubling the buffer to stop the loss. What do you say?"

Reason through it. That it will work, and state what it costs. Little's law is an identity for a work-conserving server: W = L ÷ λ — so a buffer that absorbs an 8.39 ms burst makes a frame arriving when it is full wait 8.39 ms. Absorbing more loss means introducing exactly that much more delay, at a fixed exchange rate. The strong answer makes it a question about the traffic: bulk transfer wants depth, because a retransmission costs a round trip and a millisecond costs a millisecond. Real-time traffic wants shallowness, because a frame past its deadline must be discarded anyway having consumed the bandwidth twice. And it names the real evidence: measure longest_burst_us against worst_absorb_usa buffer sized for a millisecond on a network whose bursts are 20 µs is oversized 50×.

Q6 — "Users report packet loss. Where do you look first?"

Reason through it. The receiver's descriptor ring, not the switch. A switch discards only when offered load exceeds an egress rate for longer than the buffer absorbs — 999 µs at 2:1 with a 122 KiB buffer — which real traffic rarely sustains outside a genuine capacity problem. A host's ring holds 512 frames, 344 µs at 1 Gb/s, and overflows for scheduling reasons lasting microseconds. The strong answer gives the order: is it in the network at all (a driver statistic, seconds to check); if so, which buffer (EF_QUEUE_LIMIT against EF_POOL_EMPTY, two remedies in different places); and only then which port — after Section 10's attribution, because ranking by drops examines the victims first. And it names the exception: switch loss is systematic, a predictable fraction sustained indefinitely, while host loss is opportunistic — so a steady percentage points at the network and a bursty one points at the host.

23. Understanding Check

24. What's Next

This chapter found where frames are lost and priced what a buffer buys. It did not ask whether the loss can be prevented.

Chapter 12.1 §6 rejected the obvious answer. Applying backpressure to an ingress port converts a local congestion into a global one: the port feeding a congested egress is throttled, and so is every other conversation entering through that same port, including ones bound for completely idle egress ports.

Chapter 14.2 — PAUSE Frames builds the mechanism 802.3x specified anyway. It is a MAC control frame carrying a duration in quanta of 512 bit times, and its semantics are all-or-nothing: a PAUSE stops everything the receiving port would send, for as long as it says. Section 3's high watermark is what triggers it, and the reason the trigger is at 75% rather than at full is the frames already in flight when the signal leaves.

Then Chapter 14.3 — Backpressure and Head-of-Line Blocking shows that the pathology Chapter 12.1 predicted is exactly what happens — with a throughput bound that has been known since 1987 and is worse than most people guess.

Continue learning

Standards & specifications

Governing standard
IEEE Std 802.3 (Ethernet)(opens IEEE in a new tab)

Defines the Ethernet MAC, the media-independent interfaces and the physical-layer sublayers, including framing, access control, auto-negotiation and per-rate PHY specifications. VLAN tagging, priority and time-sensitive shaping are defined by IEEE 802.1, not by 802.3.

This page also covers RTL structure, verification approach and debugging technique. Those are engineering practice built on the standard, not requirements the standard itself imposes.

Where this fits

Part of the Ethernet curriculum.