Interrupts, Coalescing and Completion

Chapter 18.1 §10 showed that one interrupt per frame is arithmetically impossible — 297.6% of a CPU at 1 Gb/s with minimum-size frames. Chapter 18.3 §14 built a fixed count-and-timer coalescer that fixes it. This chapter shows why a fixed policy is wrong at both ends and what replaces it.

A fixed threshold has to be chosen for one arrival rate, and the rate varies by five decades.

Read the two right-hand columns down. The fixed policy adds 50 µs of latency to an idle link where there is nothing to amortise, and at 100 Gb/s its count binds so hard that it delivers 4.65 million interrupts per second anyway. The count-of-one policy is correct at the top of the table and unusable at the bottom.

The adaptive answer is one line: make the threshold proportional to the measured arrival rate.

The interrupt rate is 50 000 per second in four consecutive rows spanning a factor of 18 300 in arrival rate. That is the policy's whole point: it is self-normalising, and the CPU cost is held at 10% of a core until the clamp binds.

And it has a cost this chapter's second half is about. Chapter 18.3 §15 priced fixed coalescing against Chapter 16.5's 24.2 ns clock at 4 132×. The adaptive policy's delay is five times smaller — and varies with traffic the measured flow does not control, which for a measurement is a different and worse kind of error.

1. Scope, and the Mechanism That Must Delay

An interrupt coalescer is the only block in Module 18 whose purpose is to make something later.

Every other mechanism in these six chapters exists to move data sooner or more cheaply. This one deliberately withholds a notification so that several can be delivered together, and its correctness argument is therefore upside down: the question is not whether it delivers, but whether it delivers soon enough, and "eventually" is not an answer.

What this chapter does not cover: the interrupt's delivery — MSI, a wire, an interrupt controller's own latency — is the SoC's, not the MAC's, and Chapter 18.1 §2 already established that its hardware cost is one flop. And multi-queue interrupt affinity is Chapter 18.7, which is the last chapter of the module.

One framing note that decides the whole chapter. A coalescer is a control loop: it measures an input rate, computes a threshold, and that threshold gates the output whose timing feeds back into the measurement. Loops of that shape can lock up, and Section 6 exists because "it probably will not" is not a design.

2. What a Fixed Policy Gets Wrong, at Both Ends

Chapter 18.1 §11 and Chapter 18.3 §14 both built a count-and-timer coalescer and both noted that a count with no timer stalls. This section is the fuller failure.

A fixed policy has two parameters and each is wrong in one direction.

And the two failures do not occur at the same time, which is what makes a single setting impossible: the count is too small at high rates and too large at low ones, so any fixed value is wrong at one end of the range.

The idle case is the one designs get wrong most often, and it is worth being precise about. A link receiving ten frames a second with a count of 256 and a timer of 100 µs delivers every frame 50 µs late on average — for no benefit whatsoever, because ten interrupts per second costs 0.002% of a CPU and there is nothing to amortise.

The threshold is somewhere around the third row, and a fixed policy cannot know which row it is in.

The high-rate case is the one that is usually measured and it has a subtler failure. At 100 Gb/s with minimum-size frames, a count of 256 is reached in 1.7 µs — so the count binds constantly, the timer never fires, and the port delivers 581 000 interrupts per second, which is 116% of a core. The policy is working exactly as configured and is still not enough.

Which gives the requirement in one sentence: the threshold must be large when the rate is high and small when it is low, and nothing about a constant satisfies that.

And there is a third failure worth naming because it is not about either parameter. A fixed count measured in frames treats a 64-octet frame and a 9000-octet frame as equal work for the CPU, and they are not — the per-frame software cost dominates, which is why they are roughly equal, but the per-octet cost is not zero. Section 4's policy inherits this and Section 17 says when it matters.

3. RTL 1 — The Arrival-Rate Estimator

The policy's input, and the block where a bad choice of filter produces a loop that oscillates.

// -----------------------------------------------------------------------
// coalesce_pkg -- adaptive interrupt coalescing.
// -----------------------------------------------------------------------
package coalesce_pkg;

  localparam int RATE_W   = 24;          // frames per second, up to 16.7 M
  localparam int N_MAX    = 256;
  localparam int NUM_EVT  = 8;

  // The policy's one tunable: the latency the design is willing to
  // spend filling a batch. Everything else is derived from it and
  // from the measured rate.
  //   N = clamp(rate * target_latency, 1, N_MAX)
  typedef struct packed {
    logic [15:0] target_us;              // section 4's L
    logic [15:0] n_max;
    logic [15:0] floor_timer_us;         // section 6's stall proof
    logic        ptp_bypass;             // 18.3 section 14
    logic        adaptive_enable;
  } coalesce_cfg_t;

  // Events that must never be coalesced -- 18.3 section 14's mask,
  // restated because this chapter changes everything around it and
  // not this.
  localparam int EVT_RX_FRAME   = 0;
  localparam int EVT_TX_DONE    = 1;
  localparam int EVT_RX_ERROR   = 2;     // never coalesced
  localparam int EVT_TX_UNDERRUN= 3;     // never coalesced -- 18.4
  localparam int EVT_OVERFLOW   = 4;     // never coalesced -- 18.1
  localparam int EVT_PTP        = 5;     // bypassed if configured

endpackage

// -----------------------------------------------------------------------
// arrival_rate_estimator -- an exponential moving average of the
// frame arrival rate, in frames per second.
//
// The filter's time constant is the design's only real freedom here
// and it has to sit between two bounds: slow enough that the loop
// does not chase noise, fast enough that a burst is tracked before
// it ends. Section 6's stall proof depends on neither.
// -----------------------------------------------------------------------
module arrival_rate_estimator
  import coalesce_pkg::*;
#(
  parameter int WIN_US    = 128,         // measurement window
  parameter int ALPHA_LOG = 3            // EMA weight = 1/8
)(
  input  logic                clk,
  input  logic                rst_n,

  input  logic                frame_arrived,
  input  logic                tick_1us,

  output logic [RATE_W-1:0]   rate_fps,          // frames per second
  output logic [15:0]         window_count,
  output logic                rate_valid,

  output logic [31:0]         c_windows,
  output logic [RATE_W-1:0]   peak_rate,
  output logic                rate_saturated
);

  logic [15:0]       win_us;
  logic [15:0]       win_frames;
  logic [RATE_W-1:0] ema;
  logic              have_one;

  // Frames in the window, scaled to frames per second. With
  // WIN_US = 128, one frame in the window is 7812 frames/s, which
  // sets the estimator's resolution -- and is why the clamp to 1 in
  // section 5 matters: below 7812 frames/s the estimate is zero.
  wire [RATE_W-1:0] win_rate =
    RATE_W'((32'(win_frames) * 32'd1_000_000) / 32'(WIN_US));

  assign rate_saturated = (win_frames == 16'hFFFF);
  assign rate_fps       = ema;
  assign window_count   = win_frames;
  assign rate_valid     = have_one;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      win_us <= '0; win_frames <= '0; ema <= '0; have_one <= 1'b0;
      c_windows <= '0; peak_rate <= '0;
    end else begin
      if (frame_arrived && !rate_saturated)
        win_frames <= win_frames + 16'd1;

      if (tick_1us) begin
        if (win_us >= 16'(WIN_US) - 16'd1) begin
          // Close the window: fold this window's rate into the EMA.
          //   ema <- ema + (win_rate - ema) / 2^ALPHA_LOG
          ema <= have_one
                   ? (ema + ((win_rate - ema) >>> ALPHA_LOG))
                   : win_rate;
          have_one   <= 1'b1;
          win_us     <= '0;
          win_frames <= '0;
          c_windows  <= c_windows + 1;
          if (win_rate > peak_rate) peak_rate <= win_rate;
        end else begin
          win_us <= win_us + 16'd1;
        end
      end
    end
  end

endmodule

Classification: a windowed counter feeding an exponential moving average, with an explicit resolution floor.

What it teaches: that the estimator's resolution sets the policy's low-rate behaviour and must be reasoned about rather than discovered. With a 128 µs window, one frame is 7 812 frames per second — so any rate below that measures as zero, and the threshold computed from it would be zero too. Section 5's clamp to 1 is not defensive coding; it is the only thing standing between a zero rate and a threshold that never fires.

And it teaches why the filter is an EMA rather than an instantaneous window. A raw per-window rate on bursty traffic swings between zero and the peak every window, and a threshold computed from it swings with it — so a burst that ends just after the threshold rises leaves frames waiting behind a threshold sized for traffic that has stopped. The EMA's time constant, 2^ALPHA_LOG windows, is what stops the loop chasing its own input — and at ALPHA_LOG = 3 and a 128 µs window that is about 1 ms, which is slow enough to be stable and fast enough to track a burst.

Deliberately simplified: the win_rate computation is a real divide by a parameter, which a production design replaces with a shift when WIN_US is a power of two — 128 is, and the listing divides anyway for clarity. The EMA's subtraction can underflow when the rate falls, because win_rate - ema is computed in an unsigned width; a real design keeps it signed. And the window is closed by a 1 µs tick, which assumes such a tick exists — it does, because Chapter 18.3 §14's coalescer needed one.

Production implication: peak_rate against rate_fps is how an integrator discovers that the traffic is burstier than the average suggests. A port whose EMA reads 50 000 frames per second and whose peak window read 1 500 000 is a port whose adaptive threshold is chronically too small during bursts — so the interrupt rate spikes exactly when the CPU is busiest. The fix is a faster filter, and the counter is the only evidence that one is needed.

4. The Policy: a Threshold Proportional to Rate

One equation, and everything in this chapter follows from it.

N = clamp(rate × L, 1, N_MAX), where L is the latency the design is willing to spend filling a batch.

Why this form and not another. The interrupt rate a policy produces is rate / N. Substituting gives rate / (rate × L) = 1 / L — a constant. The policy holds the interrupt rate at 1/L regardless of the arrival rate, which is exactly the self-normalising property Section 2 asked for.

At L = 20 µs, 1/L is 50 000 interrupts per second.

Rows four to six are the policy working: the arrival rate spans a factor of 18.3 and the interrupt rate does not move at all.

Rows one to three are the clamp at the bottom doing its job. Below 50 000 frames per second the computed N is less than one, so the clamp pins it at one and every frame interrupts immediately — which is correct, because at that rate there is nothing to amortise and the CPU cost is 10% or less.

Rows seven to nine are the clamp at the top, and it is where the policy stops working. At 148.8 M frames per second, rate × L is 2 976, clamped to 256 — so the interrupt rate rises to 581 000 per second and the CPU cost to 116%. The policy has run out of threshold.

Raising N_MAX fixes the arithmetic and costs something real, which is the trade Section 9 prices:

Row three is the setting that restores the invariant and it costs 10 µs of mean latency — which is L/2, and is the same 10 µs every other row in the first table pays. The policy is consistent; only the clamp was breaking it.

And N_MAX is not free, because a batch of 2 976 frames is 2 976 descriptors the driver must process in one interrupt service routine — and Section 17 is about what that requires of software.

5. RTL 2 — The Adaptive Threshold

Section 4's equation, with the two clamps that make it a policy rather than a formula.

// -----------------------------------------------------------------------
// adaptive_threshold -- N = clamp(rate * L, 1, N_MAX).
//
// The multiply is by a microsecond figure against a per-second rate,
// so the scaling is a division by 1e6. Both clamps are load-bearing:
// the lower one is what section 6's stall proof rests on, and the
// upper one is what bounds the driver's per-interrupt work.
// -----------------------------------------------------------------------
module adaptive_threshold
  import coalesce_pkg::*;
(
  input  logic                clk,
  input  logic                rst_n,

  input  coalesce_cfg_t       cfg,
  input  logic [RATE_W-1:0]   rate_fps,
  input  logic                rate_valid,

  output logic [15:0]         threshold_n,
  output logic                clamped_low,
  output logic                clamped_high,

  output logic [31:0]         c_clamp_low_cycles,
  output logic [31:0]         c_clamp_high_cycles,
  output logic [15:0]         n_min_seen,
  output logic [15:0]         n_max_seen,
  output logic                cfg_target_zero        // a configuration fault
);

  // rate [frames/s] x target [us] / 1e6 = frames per target window.
  // At 1.488 M frames/s and 20 us that is 29.76 -> 29.
  wire [47:0] product = 48'(rate_fps) * 48'(cfg.target_us);
  wire [47:0] scaled  = product / 48'd1_000_000;

  wire [15:0] raw = (scaled > 48'hFFFF) ? 16'hFFFF : scaled[15:0];

  // A target of zero would make N zero at every rate, and a zero
  // threshold never fires. It is a configuration fault, not a
  // "disable adaptation" encoding -- that is cfg.adaptive_enable.
  assign cfg_target_zero = cfg.adaptive_enable && (cfg.target_us == 16'd0);

  wire [15:0] n_hi = (cfg.n_max == 16'd0) ? 16'(N_MAX) : cfg.n_max;

  always_comb begin
    if (!cfg.adaptive_enable || !rate_valid || cfg_target_zero) begin
      threshold_n  = 16'd1;
      clamped_low  = 1'b1;
      clamped_high = 1'b0;
    end else if (raw < 16'd1) begin
      threshold_n  = 16'd1;
      clamped_low  = 1'b1;
      clamped_high = 1'b0;
    end else if (raw > n_hi) begin
      threshold_n  = n_hi;
      clamped_low  = 1'b0;
      clamped_high = 1'b1;
    end else begin
      threshold_n  = raw;
      clamped_low  = 1'b0;
      clamped_high = 1'b0;
    end
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      c_clamp_low_cycles <= '0; c_clamp_high_cycles <= '0;
      n_min_seen <= 16'hFFFF; n_max_seen <= '0;
    end else begin
      if (clamped_low)  c_clamp_low_cycles  <= c_clamp_low_cycles + 1;
      if (clamped_high) c_clamp_high_cycles <= c_clamp_high_cycles + 1;
      if (threshold_n < n_min_seen) n_min_seen <= threshold_n;
      if (threshold_n > n_max_seen) n_max_seen <= threshold_n;
    end
  end

endmodule

Classification: a scaled multiply with two clamps, each of which is a policy decision rather than an overflow guard.

What it teaches: that the lower clamp is the stall proof's foundation and must be unconditional. N is never zero, never — not when the rate is zero, not when the estimator has no valid window, not when adaptation is disabled, not when the configuration is nonsense. Every path in the always_comb produces at least one, because Section 6's argument begins "the threshold is at least one" and a single path that produces zero destroys it.

And it teaches that the upper clamp is a software parameter wearing a hardware costume. N_MAX bounds the number of frames a driver must process in one interrupt service routine. Raising it from 256 to 2 976 restores Section 4's invariant at 100 Gb/s — and asks the driver to walk 2 976 descriptors before returning, which is a design decision in a completely different codebase. Section 17 is about what that requires.

Deliberately simplified: the division by a million is a real divide of a 48-bit product, evaluated combinationally every cycle, which no design does — a production block computes N once per estimator window, since the rate only changes then. raw < 1 is written as a comparison against a 16-bit constant on a value that has already been truncated, so a rate whose product exceeds 65 535 before scaling is mishandled. And there is no hysteresis on the clamps, so a rate hovering at 50 000 frames per second flips clamped_low every window.

Production implication: c_clamp_low_cycles and c_clamp_high_cycles together say whether the policy is operating in its designed region at all. Mostly low-clamped means the link is quiet and coalescing is doing nothing — which is correct and worth knowing, because it means the latency the design is worrying about is not being paid. Mostly high-clamped means N_MAX is the binding constraint and Section 4's last table applies. A port that is never in the middle is a port whose L is set for the wrong traffic.

6. Proving It Cannot Stall

The user of a coalescer wants one guarantee: every frame is signalled within a bounded time of arriving. This section proves it, and the proof is short enough to be worth doing properly because the loop's structure is the kind that can lock up.

The hazard, stated first. The threshold is computed from a rate measured from arrivals. If arrivals stop, the rate decays — but the decay takes an estimator time constant, about 1 ms at Section 3's settings, and frames already pending were accumulated under a threshold sized for traffic that has ended. A design relying on the threshold falling to rescue them has an unbounded wait whenever the estimator is slow.

So the proof cannot rest on the adaptation. It rests on the floor timer.

The mechanism has three parts:

The proof, in four steps.

Step 1 — if N = 1, the interrupt fires on the arriving frame. The pending count reaches 1, which is N, and the wait is zero.

Step 2 — if N > 1, a frame's arrival either completes a batch or does not. If it completes one, the wait is zero for that frame. If not, the pending count is at least 1 and non-zero.

Step 3 — a non-zero pending count implies the floor timer is armed, by the second rule, and it was armed at the instant the count first became non-zero — which is at or before this frame's arrival.

Step 4 — the floor timer is unconditional and expires after T_floor. So the interrupt fires no later than T_floor after the first pending frame arrived, which is no later than T_floor after any frame in the batch arrived.

Every frame is signalled within T_floor of its arrival, for every threshold, every rate and every configuration.

The third rule is where designs go wrong and it deserves emphasis. A timer restarted on each arrival never expires while traffic continues — and under heavy traffic that is harmless, because the count fires first. Under a trickle just below the count threshold it is fatal: frames arrive often enough to keep restarting the timer and rarely enough never to fill the batch, and the batch is held indefinitely.

And the second row's failure has the worst possible signature: it requires a specific arrival rate to appear, so it is invisible in a regression that tests idle and saturated and nothing between.

One more case to close the proof: what if the rate estimate is wrong? A rate estimate that is too high gives a threshold that is too large, and the floor timer bounds it. A rate estimate too low gives a threshold that is too small, which costs interrupts and no latency. Neither direction breaks the bound, which is the property that makes the estimator's accuracy a performance question rather than a correctness one — and is why Section 3's filter can be tuned freely.

7. RTL 3 — The Stall-Proof Timer

Section 6's proof in RTL, and the whole block is three rules.

// -----------------------------------------------------------------------
// stall_proof_timer -- the bound that does not depend on the policy.
//
// Three rules, and the third is the one designs get wrong:
//   1. armed when pending goes 0 -> 1
//   2. never restarted by a later arrival
//   3. unconditional -- no rate, no threshold, no configuration can
//      prevent it expiring
// -----------------------------------------------------------------------
module stall_proof_timer
  import coalesce_pkg::*;
(
  input  logic                clk,
  input  logic                rst_n,

  input  logic [15:0]         cfg_floor_us,
  input  logic                tick_1us,

  input  logic                pending_nonzero,
  input  logic                batch_flushed,

  output logic                floor_expired,
  output logic [15:0]         age_us,
  output logic                armed,

  output logic [31:0]         c_floor_flushes,
  output logic [15:0]         worst_age_us,
  output logic                floor_disabled_unsafely
);

  // A floor of zero removes the bound entirely and makes section 6's
  // proof false. There is no legitimate configuration in which the
  // floor should be absent, so it is flagged rather than honoured as
  // a disable.
  assign floor_disabled_unsafely = (cfg_floor_us == 16'd0);

  wire [15:0] floor = floor_disabled_unsafely ? 16'd1000 : cfg_floor_us;

  // Rule 3: unconditional. Nothing in this expression refers to the
  // threshold, the rate, or any enable.
  assign floor_expired = armed && (age_us >= floor);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      armed <= 1'b0; age_us <= '0;
      c_floor_flushes <= '0; worst_age_us <= '0;
    end else begin
      // Rule 1: arm on the transition into non-empty.
      if (pending_nonzero && !armed) begin
        armed  <= 1'b1;
        age_us <= '0;
      end

      // Rule 2: while armed, the age advances and NOTHING resets it
      // except a flush. An arriving frame does not touch it.
      if (armed && tick_1us) begin
        age_us <= age_us + 16'd1;
        if ((age_us + 16'd1) > worst_age_us)
          worst_age_us <= age_us + 16'd1;
      end

      if (batch_flushed) begin
        armed  <= 1'b0;
        age_us <= '0;
        if (floor_expired) c_floor_flushes <= c_floor_flushes + 1;
      end
    end
  end

endmodule

Classification: a one-shot age counter with an arm-on-first-pending rule and no restart path.

What it teaches: that the absence of a signal is the block's most important feature. frame_arrived is not in the port list. The timer cannot be restarted by an arrival because it cannot see one — which is Section 6's rule 3 enforced structurally rather than by discipline, and it is worth building that way precisely because the restarting version looks so reasonable.

And it teaches that floor_disabled_unsafely substitutes a default rather than honouring a zero. A floor of zero makes Section 6's proof false — there is then no bound at all, at any rate, in any configuration. The block falls back to 1 000 µs and says so, which is Chapter 18.4 §7's fail-closed argument applied to a timer: when a parameter's absence removes a guarantee, substitute a conservative value and flag it rather than obeying.

Deliberately simplified: worst_age_us is updated on every tick while armed, which is one comparator running continuously where a real design samples it at the flush. The age is 16 bits of microseconds — 65.5 ms — which is ample and unchecked for overflow. And floor_expired is combinational from age_us, so it asserts in the same cycle the comparison becomes true and stays asserted until the flush, which the aggregator must treat as a level rather than a pulse.

Production implication: worst_age_us is the measured version of Section 6's bound, and it should never exceed cfg_floor_us by more than a tick or two. A port whose worst age is 4 ms with a 1 ms floor has a flush path that is not responding to the timer — which is a bug in the aggregator rather than in the timer, and it is exactly the class of failure a bound nobody measures hides for ever. The counter costs sixteen flops and is the only evidence the proof holds in silicon.

8. RTL 4 — The Completion Aggregator

Where the threshold, the timer and the never-coalesced events meet.

// -----------------------------------------------------------------------
// completion_aggregator -- decides when the interrupt fires.
//
// Four ways to fire, in priority order: an urgent event, the PTP
// bypass, the adaptive threshold, the floor timer. The first two are
// immediate; the last two are the coalescing policy.
// -----------------------------------------------------------------------
module completion_aggregator
  import coalesce_pkg::*;
(
  input  logic                clk,
  input  logic                rst_n,

  input  coalesce_cfg_t       cfg,
  input  logic [NUM_EVT-1:0]  evt,
  input  logic [NUM_EVT-1:0]  evt_urgent_mask,
  input  logic [15:0]         threshold_n,
  input  logic                floor_expired,
  input  logic                irq_ack,

  output logic                irq,
  output logic [NUM_EVT-1:0]  irq_status,
  output logic                pending_nonzero,
  output logic                batch_flushed,
  output logic [15:0]         pending,

  output logic [31:0]         c_irq_total,
  output logic [31:0]         c_by_urgent,
  output logic [31:0]         c_by_bypass,
  output logic [31:0]         c_by_threshold,
  output logic [31:0]         c_by_floor,
  output logic [31:0]         c_frames_signalled,
  output logic [15:0]         worst_batch
);

  logic [NUM_EVT-1:0] status_q;

  wire [NUM_EVT-1:0] urgent = evt & evt_urgent_mask;
  wire               is_ptp = evt[EVT_PTP] && cfg.ptp_bypass;
  wire [NUM_EVT-1:0] coal   = evt & ~evt_urgent_mask &
                              ~(cfg.ptp_bypass ? (NUM_EVT'(1) << EVT_PTP)
                                               : NUM_EVT'(0));

  wire fire_urgent    = |urgent;
  wire fire_bypass    = is_ptp;
  wire fire_threshold = (pending >= threshold_n) && (pending != 16'd0);
  wire fire_floor     = floor_expired && (pending != 16'd0);

  wire fire = fire_urgent | fire_bypass | fire_threshold | fire_floor;

  assign pending_nonzero = (pending != 16'd0);
  assign batch_flushed   = fire;
  assign irq_status      = status_q;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      pending <= '0; status_q <= '0; irq <= 1'b0;
      c_irq_total <= '0; c_by_urgent <= '0; c_by_bypass <= '0;
      c_by_threshold <= '0; c_by_floor <= '0;
      c_frames_signalled <= '0; worst_batch <= '0;
    end else begin
      if (|coal) begin
        pending  <= pending + 16'($countones(coal));
        status_q <= status_q | coal;
      end
      if (|urgent) status_q <= status_q | urgent;
      if (is_ptp)  status_q <= status_q | (NUM_EVT'(1) << EVT_PTP);

      if (fire) begin
        irq <= 1'b1;
        if (!irq) c_irq_total <= c_irq_total + 1;

        // Attribution, in priority order. Exactly one is credited,
        // which is what makes the four counters a partition rather
        // than four overlapping tallies.
        if      (fire_urgent)    c_by_urgent    <= c_by_urgent + 1;
        else if (fire_bypass)    c_by_bypass    <= c_by_bypass + 1;
        else if (fire_threshold) c_by_threshold <= c_by_threshold + 1;
        else                     c_by_floor     <= c_by_floor + 1;

        c_frames_signalled <= c_frames_signalled + {16'b0, pending};
        if (pending > worst_batch) worst_batch <= pending;
        pending <= '0;
      end

      if (irq_ack) begin
        irq      <= 1'b0;
        status_q <= '0;
      end
    end
  end

endmodule

Classification: a four-source firing decision with priority attribution.

What it teaches: that the four counters must partition rather than overlap, and the else if chain is what makes them do it. A design counting each firing condition independently credits a single interrupt to two or three causes whenever conditions coincide — which they do constantly, because the floor timer and the threshold both become true on a busy link. A partition lets an operator read the four counters as fractions that sum to one; overlapping tallies cannot be interpreted at all.

And it teaches that the urgent and bypass paths do not touch pending. An error event fires the interrupt and leaves the coalesced batch's count intact, so the batch is not credited as flushed by an event that had nothing to do with it. The fire signal does flush it — which is a deliberate and slightly generous choice: an urgent interrupt carries the pending frames along with it, which is free and reduces the next batch's wait.

Deliberately simplified: $countones on the event vector every cycle; status_q is assigned in three separate branches in one always_ff, which is legal and will make a linter unhappy. The acknowledge clears the whole status word, which races with an event in the same cycle and loses it — Chapter 18.1 §11 flagged the same simplification, and the fix is write-1-to-clear per bit with set priority over clear.

Production implication: the ratio c_by_floor / c_irq_total is the single number that says whether the adaptive policy is doing anything. Near zero means the threshold is firing — the policy is in its designed region. Near one means every interrupt is the floor timer — the threshold is never reached, so the port is latency-bound rather than rate-bound and the adaptive machinery is inert. That is a correct outcome on a quiet link and a misconfiguration on a busy one, and c_by_threshold against the arrival rate distinguishes them.

9. What the Adaptive Policy Costs in Latency

Section 4 gave the interrupt rate. This section gives the other side of the trade, and the result is that the cost is constant where the policy is working.

The mean wait of a frame in a batch of N arriving at rate r is (N − 1) / 2r. Substituting N = r × L:

mean wait = (rL − 1) / 2r ≈ L / 2 whenever rL ≫ 1.

So the policy trades a constant L/2 of latency for a constant 1/L of interrupt rate, and neither depends on the traffic.

Rows four and five are the policy at L/2 = 10 µs, as predicted. Row three is below it because N is only 1.6 and the approximation does not hold at small N. Rows six and seven are below it because the clamp has taken over.

Compare against the two fixed policies, on the same traffic:

The fixed policy's latency is worse than the adaptive one everywhere the link is not saturated, which is most of the time — 50 µs against 0 to 9.66 µs. And at the bottom two rows it is better, because its count binds harder and delivers 4.65 million interrupts per second to achieve it.

So on latency and on CPU the adaptive policy wins across the range, and the honest summary of the trade is:

Row five is the actual argument. A fixed policy tuned for one operating point is competitive at that point; the adaptive policy is competitive at all of them with one setting, which is what a general-purpose NIC needs because it does not know what traffic it will carry.

And L is now the only parameter, which is the second half of the win. Chapter 18.1 §11's fixed coalescer had two — a count and a timer — whose correct values depend on each other and on the traffic. This one has L, whose meaning is stated directly: "I am willing to spend L/2 of mean latency, and in exchange I will take 1/L interrupts per second." Both halves of that sentence are what the operator actually cares about.

10. RTL 5 — The Latency Accountant

The block that measures what Section 9 derived, because a derived latency and a delivered one are different things.

// -----------------------------------------------------------------------
// coalesce_latency_accountant -- per-frame delay, measured.
//
// Section 9 predicts (N-1)/2r. This measures it. The two differ
// whenever the arrival process is not smooth, which is always, and
// the difference is what section 11's jitter argument is about.
// -----------------------------------------------------------------------
module coalesce_latency_accountant
  import coalesce_pkg::*;
#(
  parameter int MAX_TRACK = 16           // frames whose age is tracked
)(
  input  logic                clk,
  input  logic                rst_n,

  input  logic                frame_arrived,
  input  logic                batch_flushed,
  input  logic                tick_1us,

  input  logic                frame_is_ptp,

  output logic [31:0]         c_delay_sum_us,
  output logic [31:0]         c_delay_n,
  output logic [15:0]         worst_delay_us,
  output logic [15:0]         best_delay_us,
  output logic [31:0]         c_ptp_delay_sum_us,
  output logic [31:0]         c_ptp_delay_n,
  output logic [15:0]         ptp_worst_us,
  output logic [15:0]         ptp_best_us,
  output logic [15:0]         mean_delay_us,
  output logic [15:0]         ptp_spread_us          // section 11
);

  // Only the OLDEST frame's age is tracked exactly; the rest are
  // approximated from it. Tracking every frame's arrival instant
  // would be MAX_TRACK timestamps, which at N up to 2976 is not
  // affordable -- section 13's note on what this costs.
  logic [15:0] oldest_age;
  logic        have_oldest;
  logic [15:0] ptp_age;
  logic        have_ptp;

  always_comb begin
    mean_delay_us = (c_delay_n == '0) ? 16'd0
                                      : 16'(c_delay_sum_us / c_delay_n);
    ptp_spread_us = (ptp_worst_us > ptp_best_us)
                      ? (ptp_worst_us - ptp_best_us) : 16'd0;
  end

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      oldest_age <= '0; have_oldest <= 1'b0;
      ptp_age <= '0; have_ptp <= 1'b0;
      c_delay_sum_us <= '0; c_delay_n <= '0;
      worst_delay_us <= '0; best_delay_us <= 16'hFFFF;
      c_ptp_delay_sum_us <= '0; c_ptp_delay_n <= '0;
      ptp_worst_us <= '0; ptp_best_us <= 16'hFFFF;
    end else begin
      if (frame_arrived && !have_oldest) begin
        have_oldest <= 1'b1;
        oldest_age  <= '0;
      end
      if (frame_arrived && frame_is_ptp && !have_ptp) begin
        have_ptp <= 1'b1;
        ptp_age  <= '0;
      end

      if (tick_1us) begin
        if (have_oldest) oldest_age <= oldest_age + 16'd1;
        if (have_ptp)    ptp_age    <= ptp_age + 16'd1;
      end

      if (batch_flushed) begin
        if (have_oldest) begin
          c_delay_sum_us <= c_delay_sum_us + {16'b0, oldest_age};
          c_delay_n      <= c_delay_n + 1;
          if (oldest_age > worst_delay_us) worst_delay_us <= oldest_age;
          if (oldest_age < best_delay_us)  best_delay_us  <= oldest_age;
        end
        if (have_ptp) begin
          c_ptp_delay_sum_us <= c_ptp_delay_sum_us + {16'b0, ptp_age};
          c_ptp_delay_n      <= c_ptp_delay_n + 1;
          if (ptp_age > ptp_worst_us) ptp_worst_us <= ptp_age;
          if (ptp_age < ptp_best_us)  ptp_best_us  <= ptp_age;
        end
        have_oldest <= 1'b0;
        have_ptp    <= 1'b0;
      end
    end
  end

endmodule

Classification: an oldest-frame age tracker with a separate channel for timestamped traffic.

What it teaches: that the PTP delay must be measured separately from the aggregate, because the aggregate hides it. A batch of 256 frames containing one PTP message contributes one sample to the mean — and the mean is dominated by 255 frames whose latency nobody cares about. Chapter 16.1's whole argument is about the one frame, so it needs its own accumulator, its own worst case, and its own best case.

And it teaches that ptp_spread_us — worst minus best — is the number Section 11 is about, not the mean. A delay that is always 50 µs is a bias; a delay that is between 0 and 9.66 µs is an uncertainty. The mean is similar in both cases and the spread is not, and for a synchronisation measurement the spread is what survives every correction.

Deliberately simplified: only the oldest frame's age is tracked, so the per-frame mean is approximated by the batch's oldest — which over-estimates the mean by roughly a factor of two, and is stated so the number can be interpreted. Tracking every frame would need N timestamps and N reaches 2 976. mean_delay_us is a combinational divide. And best_delay_us initialised to 0xFFFF will read as 65 535 on a port that has never flushed, which software must special-case.

Production implication: ptp_spread_us is the number to put in a synchronisation report, and it is the one Module 16 has been unable to obtain. Chapter 16.5 derived an error budget from the clock, the servo and the path; this measures the term the host adds, per port, in the field. A port reporting a 24.2 ns clock and a 9 660 ns PTP spread has a synchronisation problem that is entirely in its own device driver's interrupt configuration — and nothing in Module 16 could have told it so.

11. A Varying Delay Against a Constant One

This section is the chapter's central claim and it runs against the obvious reading of Section 9's tables.

By every throughput and latency measure the adaptive policy is better. For a measurement it may be worse, and the reason is not magnitude.

Consider a PTP event message — Chapter 16.2's Sync or Delay_Req. It is small, isolated, and arrives tens of times per second. Its delay through the coalescer depends on what other traffic is doing.

Read the fixed column down the first six rows: it is 50.0 µs, six times, across a factor of 5 000 in background rate. It is a constant, and a constant is a bias.

Read the adaptive column: 0, 0, 0, 3.85, 9.66, 9.39, 8.57, 0.86. It is a variable, and it varies with traffic the PTP flow does not control and cannot observe.

Why the distinction matters more than the magnitude.

So the comparison that matters is not 50 µs against 9.66 µs. It is:

Row one is the theoretical best and row two is what actually happens, because nobody calibrates out a NIC driver's coalescing timer — the value is not exported, it changes when the driver is updated, and the person tuning the clock is not the person tuning the NIC.

Which makes the honest conclusion a split verdict, and it is worth stating plainly:

The adaptive policy is better for throughput, better for latency, better for CPU, and worse for anything that needs a stable delay. It reduces the error by 5× and makes the remainder uncorrectable.

And the resolution is the one Chapter 18.3 §14 already built: the PTP bypass. A frame identified as PTP skips the coalescer entirely, so its delay is zero under either policy — and the argument above becomes irrelevant, which is the best outcome available.

The bypass costs tens of interrupts per second — Chapter 16.2's message rate — and it is the only configuration in which a 24.2 ns clock is delivered as a 24.2 ns clock. Section 13 prices what happens without it.

12. RTL 6 — The Jitter Reporter

A histogram, because a mean and a worst case cannot distinguish a bias from a spread and Section 11's whole argument is that distinction.

// -----------------------------------------------------------------------
// jitter_reporter -- a coarse delay histogram for timestamped frames.
//
// Section 11: the mean is similar for a bias and for an uncertainty;
// the SHAPE is not. A histogram is the smallest structure that shows
// the difference, and eight buckets is enough to see it.
// -----------------------------------------------------------------------
module jitter_reporter
  import coalesce_pkg::*;
#(
  parameter int NUM_BUCKET = 8
)(
  input  logic                clk,
  input  logic                rst_n,

  input  logic                sample_valid,
  input  logic [15:0]         sample_us,
  input  logic                sample_is_ptp,

  input  logic [15:0]         cfg_bucket_us,        // width of bucket 0

  output logic [31:0]         hist     [NUM_BUCKET],
  output logic [31:0]         hist_ptp [NUM_BUCKET],
  output logic [31:0]         c_samples,
  output logic [31:0]         c_samples_ptp,
  output logic [2:0]          modal_bucket,
  output logic                distribution_is_bimodal
);

  // Logarithmic buckets: 0, 1x, 2x, 4x, 8x, 16x, 32x, 64x the unit.
  // Logarithmic because the interesting range spans 0 to 100 us and
  // the resolution that matters is near the bottom.
  logic [2:0] bucket;
  always_comb begin
    logic [15:0] u;
    u = (cfg_bucket_us == 16'd0) ? 16'd1 : cfg_bucket_us;
    if      (sample_us <  u)            bucket = 3'd0;
    else if (sample_us < (u << 1))      bucket = 3'd1;
    else if (sample_us < (u << 2))      bucket = 3'd2;
    else if (sample_us < (u << 3))      bucket = 3'd3;
    else if (sample_us < (u << 4))      bucket = 3'd4;
    else if (sample_us < (u << 5))      bucket = 3'd5;
    else if (sample_us < (u << 6))      bucket = 3'd6;
    else                                bucket = 3'd7;
  end

  // The modal bucket, and whether a second mode holds a comparable
  // share. A bias lives in one bucket; 16.1's coalescing structure --
  // "usually T, occasionally 0" -- lives in two.
  logic [31:0] max1, max2;
  always_comb begin
    int i;
    max1 = '0; max2 = '0; modal_bucket = 3'd0;
    for (i = 0; i < NUM_BUCKET; i++) begin
      if (hist_ptp[i] > max1) begin
        max2 = max1;
        max1 = hist_ptp[i];
        modal_bucket = i[2:0];
      end else if (hist_ptp[i] > max2) begin
        max2 = hist_ptp[i];
      end
    end
  end

  // Two modes, each at least a quarter of the samples: the shape
  // 16.1 section 8's callout described and could not measure.
  assign distribution_is_bimodal =
    (c_samples_ptp > 32'd100) && (max2 > (c_samples_ptp >> 2));

  always_ff @(posedge clk or negedge rst_n) begin
    int i;
    if (!rst_n) begin
      for (i = 0; i < NUM_BUCKET; i++) begin
        hist[i] <= '0; hist_ptp[i] <= '0;
      end
      c_samples <= '0; c_samples_ptp <= '0;
    end else if (sample_valid) begin
      hist[bucket] <= hist[bucket] + 1;
      c_samples    <= c_samples + 1;
      if (sample_is_ptp) begin
        hist_ptp[bucket] <= hist_ptp[bucket] + 1;
        c_samples_ptp    <= c_samples_ptp + 1;
      end
    end
  end

endmodule

Classification: a logarithmic delay histogram with a bimodality test.

What it teaches: that Chapter 16.1 §8's callout described a distribution shape and had no way to observe it. That callout's table said a PTP message waits the full timer when isolated and nothing when it lands in a burst — "a delay that is usually one value and occasionally another" — and identified it as the worst possible structure for a measurement. distribution_is_bimodal is that observation made into a signal, and it costs eight counters and a two-maximum scan.

And it teaches why the buckets are logarithmic. The interesting range is 0 to 100 µs and the resolution that matters is at the bottom: the difference between 0 and 1 µs decides whether the bypass is working, while the difference between 50 and 60 µs decides nothing. Linear buckets across 100 µs would put every bypassed sample in bucket zero and every coalesced one in bucket four or five, which is exactly two bits of information.

Deliberately simplified: the two-maximum scan runs combinationally over eight entries every cycle. hist_ptp and hist are both 32-bit and never reset, so a long-running port's histogram describes its whole lifetime rather than its current state — a real design double-buffers and resets on read. And the bimodality test is crude: two modes each above a quarter catches the shape described and would miss a three-moded distribution entirely.

Production implication: the pair modal_bucket and distribution_is_bimodal is what an operator should read before blaming the network for poor synchronisation. A single mode at bucket zero is the bypass working. A single mode at bucket five is a constant delay — a bias, correctable in principle. Two modes is Section 11's uncertainty, and it means the PTP traffic is sharing a coalescer with bulk traffic whose rate varies — which is a configuration problem in the host, not a problem with the clock, the servo, the cable or the switch.

13. Pricing Adaptivity Against Chapter 16.1's Timestamp Path

Chapter 18.3 §15 settled Chapter 16.1 §8's second unpriced term at 100 µs and 4 132×. This section revises it for an adaptive policy, and the revision is not simply smaller.

The full table, with Chapter 18.3 §13's first term alongside:

Row three's "in principle" is doing a lot of work and it is worth being blunt about why it fails in practice.

To subtract a known bias, three things must be true:

Row three is the one that fails. The person tuning the PTP servo is not the person tuning the NIC driver's coalescing; the coalescing value changes when the driver is updated; and no PTP implementation this track is aware of reads it. So the 2 066× is what a deployed system actually carries.

And the adaptive policy's 399× is what a deployed system carries too, because there is nothing to read — the delay is a function of the neighbours' traffic, moment to moment.

So adaptivity improves the practical figure by 5.2× and does not solve anything, which is the section's conclusion and the reason the bypass is not optional.

One more effect the adaptive policy introduces and the fixed one does not, and it is genuinely new.

The adaptive delay depends on the background rate. So a PTP message's delay is correlated with how busy the link is — and a link's business is correlated with time of day, with the application's load, and with whatever else the machine is doing.

A slowly varying, load-correlated error is the hardest kind for Chapter 16.4's servo, because it looks exactly like a real frequency error: the offset trends in one direction for hours and then trends back. A servo with an integral term will chase it, and pull the clock away from the truth by whatever fraction of the drift its loop bandwidth admits.

Which is a genuine argument for the fixed policy in one narrow case: a device whose only job is time synchronisation and which carries no bulk traffic. There the fixed policy's constant is honest and the adaptive policy's variability is pure harm. For everything else the adaptive policy plus the bypass wins on every axis, and the bypass is what makes the comparison moot.

14. RTL 7 — Coalescing Telemetry

The counters that let an operator tell any of Section 13's cases apart.

// -----------------------------------------------------------------------
// coalesce_telemetry -- the policy's observable state.
// -----------------------------------------------------------------------
module coalesce_telemetry
  import coalesce_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  logic [RATE_W-1:0] rate_fps,
  input  logic [15:0] threshold_n,
  input  logic        clamped_low,
  input  logic        clamped_high,
  input  logic        irq_fired,
  input  logic        by_threshold,
  input  logic        by_floor,
  input  logic        by_urgent,
  input  logic        by_bypass,
  input  logic [15:0] batch_size,
  input  logic [15:0] delay_us,
  input  logic        delay_is_ptp,

  output logic [31:0] c_irq,
  output logic [31:0] c_frames,
  output logic [31:0] c_by [4],
  output logic [31:0] c_low_windows,
  output logic [31:0] c_high_windows,
  output logic [15:0] mean_batch_x10,
  output logic [15:0] mean_n_x10,
  output logic [31:0] c_n_sum,
  output logic [31:0] c_n_samples,
  output logic [RATE_W-1:0] rate_now,
  output logic [15:0] amortisation_x10        // frames per interrupt
);

  always_comb begin
    mean_batch_x10   = (c_irq == '0) ? 16'd0
                                     : 16'((c_frames * 32'd10) / c_irq);
    mean_n_x10       = (c_n_samples == '0) ? 16'd0
                                           : 16'((c_n_sum * 32'd10) / c_n_samples);
    amortisation_x10 = mean_batch_x10;
  end

  assign rate_now = rate_fps;

  always_ff @(posedge clk or negedge rst_n) begin
    int i;
    if (!rst_n) begin
      c_irq <= '0; c_frames <= '0;
      for (i = 0; i < 4; i++) c_by[i] <= '0;
      c_low_windows <= '0; c_high_windows <= '0;
      c_n_sum <= '0; c_n_samples <= '0;
    end else begin
      if (irq_fired) begin
        c_irq    <= c_irq + 1;
        c_frames <= c_frames + {16'b0, batch_size};
        if      (by_urgent)    c_by[0] <= c_by[0] + 1;
        else if (by_bypass)    c_by[1] <= c_by[1] + 1;
        else if (by_threshold) c_by[2] <= c_by[2] + 1;
        else if (by_floor)     c_by[3] <= c_by[3] + 1;

        // The THRESHOLD in force at each firing, averaged. This is
        // not the same as the mean batch size: a batch flushed by
        // the floor timer is smaller than the threshold that was
        // set, and the gap between the two is how much the policy
        // is asking for versus getting.
        c_n_sum     <= c_n_sum + {16'b0, threshold_n};
        c_n_samples <= c_n_samples + 1;
      end

      if (clamped_low)  c_low_windows  <= c_low_windows + 1;
      if (clamped_high) c_high_windows <= c_high_windows + 1;
    end
  end

endmodule

Classification: a policy-state accumulator separating the threshold requested from the batch achieved.

What it teaches: that mean_n_x10 and mean_batch_x10 are different numbers and the gap between them is the policy's efficiency. The threshold is what the policy asked for; the batch size is what it got. A port with a mean threshold of 30 and a mean batch of 4 is flushing on the floor timer three times in four — the traffic is burstier than the EMA believes, and the amortisation the policy was designed for is not happening.

And it teaches that amortisation_x10 — frames per interrupt — is the number that should appear in a performance report, because it is directly comparable across policies. Chapter 18.3 §12's mean_batch_x100 made the same argument for descriptor writebacks; this is the interrupt-path version, and both exist because a configured maximum describes an intention and this describes an outcome.

Deliberately simplified: three combinational divides. c_by is a 4-entry array indexed by a chain of else if in the sequential block, which is correct and would be a case statement in production. The counters never reset, so a port's numbers describe its uptime rather than its current behaviour — the same limitation every telemetry block in Module 18 has and a real design fixes with windowed counters.

Production implication: c_by[3] / c_irq — the floor-timer fraction — read together with rate_now is the diagnostic that separates three situations that look identical. A high floor fraction at a low rate is correct: the link is quiet, N is clamped at 1, and the timer is irrelevant. A high floor fraction at a high rate means the threshold is never reached, so the EMA is over-estimating and the policy is delivering latency without amortisation. A low floor fraction at any rate means the threshold is doing the work, which is the designed behaviour.

15. RTL 8 — The Coalescing Conformance Monitor

The verdicts, and one of them is a proof obligation rather than a measurement.

// -----------------------------------------------------------------------
// coalesce_conformance_monitor -- is the policy behaving as derived?
// -----------------------------------------------------------------------
module coalesce_conformance_monitor
  import coalesce_pkg::*;
(
  input  logic        clk,
  input  logic        rst_n,

  input  coalesce_cfg_t cfg,
  input  logic [31:0] c_irq,
  input  logic [31:0] c_frames,
  input  logic [31:0] c_by [4],
  input  logic [31:0] c_low_windows,
  input  logic [31:0] c_high_windows,
  input  logic [15:0] mean_batch_x10,
  input  logic [15:0] mean_n_x10,
  input  logic [15:0] worst_age_us,
  input  logic [15:0] ptp_spread_us,
  input  logic [31:0] c_ptp_delay_n,
  input  logic        distribution_is_bimodal,
  input  logic [RATE_W-1:0] rate_now,

  input  logic        floor_disabled_unsafely,
  input  logic        cfg_target_zero,

  output logic        coalescing_ok,
  output logic        bound_violated,          // the proof obligation
  output logic        cfg_fault,
  output logic        clamp_high_binding,
  output logic        threshold_never_reached,
  output logic        ptp_not_bypassed,
  output logic        ptp_delay_unstable,
  output logic        none_of_the_above
);

  assign cfg_fault = floor_disabled_unsafely | cfg_target_zero;

  // Section 6 proves every frame is signalled within the floor. This
  // checks it in silicon. A violation means the flush path is not
  // responding to the timer -- the proof is about the policy and
  // this is about the implementation.
  assign bound_violated = (worst_age_us > (cfg.floor_timer_us + 16'd2));

  // N_MAX is binding: section 4's last table applies.
  assign clamp_high_binding = (c_high_windows > (c_low_windows + 32'd1000));

  // Almost every interrupt is the floor timer while the link is busy:
  // the threshold is set for traffic that is not arriving.
  assign threshold_never_reached =
    (c_irq > 32'd1000) && (c_by[3] > ((c_irq >> 1) + (c_irq >> 2))) &&
    (rate_now > RATE_W'(100_000));

  // PTP frames are being coalesced: section 13's 399x or 2066x.
  assign ptp_not_bypassed = (c_ptp_delay_n > 32'd100) &&
                            (c_by[1] == 32'd0);

  // And the shape 16.1 section 8's callout described.
  assign ptp_delay_unstable = distribution_is_bimodal ||
                              (ptp_spread_us > 16'd1000);

  assign coalescing_ok = !bound_violated && !cfg_fault;

  assign none_of_the_above = coalescing_ok && !clamp_high_binding &&
                             !threshold_never_reached && !ptp_not_bypassed &&
                             !ptp_delay_unstable;

  // ---- properties -------------------------------------------------

  p_bound_holds:
    assert property (@(posedge clk) disable iff (!rst_n)
      !bound_violated)
    else $error("a frame waited longer than the floor timer -- section 6's bound");

  p_floor_never_disabled:
    assert property (@(posedge clk) disable iff (!rst_n)
      !floor_disabled_unsafely)
    else $error("the stall-proof floor was configured to zero");

  p_frames_ge_irq:
    assert property (@(posedge clk) disable iff (!rst_n)
      c_frames >= c_irq)
    else $error("more interrupts than frames signalled");

  p_attribution_partitions:
    assert property (@(posedge clk) disable iff (!rst_n)
      (c_by[0] + c_by[1] + c_by[2] + c_by[3]) == c_irq)
    else $error("the firing attribution does not partition the interrupts");

  p_fault_excludes_ok:
    assert property (@(posedge clk) disable iff (!rst_n)
      cfg_fault |-> !coalescing_ok)
    else $error("coalescing_ok asserted with a configuration fault");

endmodule

Classification: a verdict generator containing one runtime check of a design-time proof.

What it teaches: that bound_violated is a different kind of signal from every other verdict in Module 18. The others report a condition the design did not prevent. This one reports that a proof is false in silicon — Section 6 argues that no frame waits longer than the floor, and this checks the argument rather than the behaviour. A violation is not a tuning problem or a system problem; it means the flush path is not responding to the timer, which is an RTL bug in the aggregator.

And it teaches that p_attribution_partitions is worth asserting even though it looks like bookkeeping. Section 8's four counters are only interpretable as fractions if they sum to the total — and the else if chain that makes them do so is exactly the kind of structure a later optimisation flattens into parallel ifs. The property costs an adder and preserves the counters' meaning against a change nobody would flag as risky.

Deliberately simplified: bound_violated allows two microseconds of slack for the flush path, which is a magic number rather than a derived one. The thresholds elsewhere are literals. ptp_not_bypassed fires whenever the bypass counter is zero and PTP frames have been seen, which is correct and will also fire on a port that carries PTP and deliberately does not bypass it — a configuration that Section 13's last paragraph argues is sometimes right.

Production implication: ptp_delay_unstable is the verdict that closes Module 16's loop, and it is the first signal in this track that reports a synchronisation problem from inside the host. Chapter 16.5 could measure the clock's error; it could not see the transport. This says the PTP frames' delay through the interrupt path is bimodal or spread over more than a microsecond — against a 24.2 ns clock — and names the cause as the host rather than the network.

16. The Transmit Side, Which Is Not Symmetric

Everything to this point has been about receive completions. The transmit side needs the same mechanism and a different setting, and the reason is Chapter 18.4's asymmetry.

A receive completion tells software a frame has arrived and must be processed. A transmit completion tells software a buffer may be freed — Chapter 18.4 §13 — and nothing else.

Row four is the shape: transmit completions are not urgent until suddenly they are. A driver with 4 096 transmit descriptors and a pool of buffers does not care whether a completion arrives in 10 µs or 10 ms — until the pool is empty, at which point the next transmit blocks and the completion's latency becomes the application's.

Which argues for a much larger L on the transmit side, and a different floor.

And it argues for something the receive side does not need: a pool-pressure trigger. A transmit completion should fire immediately when the free-buffer count falls below a watermark, regardless of count or timer — because at that moment the completion's latency is on the application's critical path.

The last row is the transmit side's equivalent of the PTP bypass: a condition under which the coalescing policy must be suspended because the thing it is trading against has changed. And like the bypass it is cheap — a comparison against a watermark the driver writes — and like the bypass it is frequently absent, which produces a port whose transmit throughput collapses whenever the buffer pool is tight and recovers when it is not.

Which is worth naming as a symptom, because it is diagnosed as a memory problem. A transmit path that runs at full rate with a large pool and at a fraction of it with a small one looks like a memory-bandwidth limitation and is a coalescing configuration. c_by[3] on the transmit coalescer — the floor fraction — distinguishes them: near one means completions are timer-bound and the pool is waiting on a timer that does not know it matters.

17. What Software Must Do for Any of This to Work

The hardware delivers one interrupt for N frames. Everything after that is the driver's, and three of its obligations are not obvious.

Obligation 1 — process the whole batch in one service routine. A driver that handles one frame per interrupt has undone the entire mechanism: the hardware coalesced 256 frames into one interrupt and the software takes 256 interrupts' worth of time to handle it, or worse, leaves 255 frames unprocessed until the next one.

The loop must run until the ring is empty, not until one frame is found — and that is a real structural requirement, because N can be 2 976 at Section 4's raised clamp.

Obligation 2 — bound the loop. A loop that runs until the ring is empty never terminates on a saturated link, because frames arrive as fast as it consumes them. So it must have a budget — a frame count or a time — and return, leaving the interrupt re-armed if work remains.

Row three is the standard structure and it is what every modern driver does, under one name or another. Its budget interacts with Section 4's N_MAX: a budget of 64 frames against a hardware threshold of 256 means four interrupts per hardware batch, which quietly restores a quarter of the interrupt rate the policy removed.

The software budget and the hardware threshold must be chosen together, and a budget below N_MAX puts the interrupt rate back up by their ratio.

Obligation 3 — do not re-enable the interrupt until the ring is drained. A driver that acknowledges the interrupt at the start of its loop takes a second interrupt for frames that arrive while it is still running the first — which on a busy link is an interrupt storm layered on top of a working coalescer.

And there is a fourth obligation that belongs to the system rather than to the driver, and it is the one that makes the arithmetic real.

Section 4's policy holds the interrupt rate at 50 000 per second. At a 2 µs service routine that is 10% of a core — and 2 µs is an assumption. A service routine that takes 20 µs because it touches uncached descriptor memory — Chapter 18.2 §14's non-cacheable ring — costs 100% of a core at the same interrupt rate.

So L is chosen against an ISR cost the MAC does not know, which is Chapter 18.1 §17's assumption table gaining one more row — and the only honest way to set it is to measure the ISR and divide.

One consequence worth stating, because it inverts a common instinct. The right value of L is larger on a slow CPU and smaller on a fast one. A design tuned on a development board with a fast core and tested on a deployed system with a slower one will be under-coalesced by exactly the ratio of their ISR costs — and the symptom is CPU saturation at a traffic level the lab never reproduced.

18. The Cost, Accounted

Eight blocks, and this chapter is the cheapest in Module 18 by a wide margin.

About 1 670 flops — against Chapter 18.5's 5 500 and Chapter 18.2's 4 800 — and a third of it is the histogram, which is pure instrumentation.

The arithmetic is where the real cost is:

Every one of them runs once per 128 µs, which at 250 MHz is once per 32 000 cycles — so all three can be a single shared sequential unit taking dozens of cycles, and the combinational versions in Sections 3 and 5 are written for readability rather than for synthesis.

And there is no memory at all. This chapter adds zero bytes to Module 18's SRAM bill, which now stands complete for a 100 Gb/s port:

Which makes this the module's best value by a distance. Sixteen hundred flops and no memory, to take a 1 Gb/s port from 297.6% of a CPU to 10% — and to hold it at 10% across a factor of 18 300 in arrival rate.

The comparison worth drawing is against Chapter 18.2's in-flight table: 2 900 flops to make a correctness requirement affordable. This chapter's 1 670 flops make a performance requirement satisfiable, and performance work is usually the expensive kind. The inversion is because the mechanism is a control loop rather than a datapath — it computes a number once per 128 µs and gates something already present.

19. Properties Worth Asserting, and One Worth Refusing

A coalescer's properties are mostly about bounds, which makes the rejected one particularly close to the right answer: it is the same property with the bound removed.

The rate estimator.

// The estimate never exceeds what the window could have counted.
p_rate_bounded_by_window:
  assert property (@(posedge clk) disable iff (!rst_n)
    window_count <= 16'hFFFF)
  else $error("the window counter saturated without being flagged");

// The window closes every WIN_US microseconds, without exception.
p_window_closes_on_time:
  assert property (@(posedge clk) disable iff (!rst_n)
    (tick_1us && (win_us == WIN_US - 1)) |=> (win_us == '0))
  else $error("a measurement window failed to close");

// Saturation is reported rather than silently clipping the rate.
p_saturation_flagged:
  assert property (@(posedge clk) disable iff (!rst_n)
    (win_frames == 16'hFFFF) |-> rate_saturated)
  else $error("the window counter saturated without flagging");

// The peak only rises.
p_peak_monotonic:
  assert property (@(posedge clk) disable iff (!rst_n)
    ##1 (peak_rate >= $past(peak_rate)))
  else $error("the peak rate decreased");

The threshold.

// THE property this chapter rests on: N is never zero, on any path.
p_threshold_at_least_one:
  assert property (@(posedge clk) disable iff (!rst_n)
    threshold_n >= 16'd1)
  else $error("the coalescing threshold was zero -- section 6's proof is void");

// N never exceeds the configured maximum.
p_threshold_within_max:
  assert property (@(posedge clk) disable iff (!rst_n)
    threshold_n <= ((cfg.n_max == 16'd0) ? 16'(N_MAX) : cfg.n_max))
  else $error("the threshold exceeded N_MAX");

// Adaptation disabled means a threshold of one, not an arbitrary value.
p_disabled_means_one:
  assert property (@(posedge clk) disable iff (!rst_n)
    !cfg.adaptive_enable |-> (threshold_n == 16'd1))
  else $error("adaptation was disabled and the threshold was not one");

// Exactly one clamp flag at a time.
p_clamps_exclusive:
  assert property (@(posedge clk) disable iff (!rst_n)
    !(clamped_low && clamped_high))
  else $error("both clamp flags asserted");

// A zero target is a fault, not a disable encoding.
p_zero_target_is_a_fault:
  assert property (@(posedge clk) disable iff (!rst_n)
    (cfg.adaptive_enable && (cfg.target_us == 16'd0)) |-> cfg_target_zero)
  else $error("a zero coalescing target was accepted");

The stall-proof timer.

// Armed exactly when there is something pending.
p_armed_iff_pending:
  assert property (@(posedge clk) disable iff (!rst_n)
    armed |-> pending_nonzero)
  else $error("the floor timer was armed with nothing pending");

// The age is NEVER reset except by a flush -- section 6's rule 2.
p_age_only_reset_by_flush:
  assert property (@(posedge clk) disable iff (!rst_n)
    (armed && !batch_flushed) |=> (age_us >= $past(age_us)))
  else $error("the floor timer was restarted -- the stall bug");

// The floor is never configured away.
p_floor_never_zero:
  assert property (@(posedge clk) disable iff (!rst_n)
    !floor_disabled_unsafely)
  else $error("the stall-proof floor was set to zero");

// Once expired, it stays expired until the flush.
p_expiry_is_sticky:
  assert property (@(posedge clk) disable iff (!rst_n)
    (floor_expired && !batch_flushed) |=> floor_expired)
  else $error("the floor expiry deasserted without a flush");

The aggregator.

// Urgent events are never delayed.
p_urgent_immediate:
  assert property (@(posedge clk) disable iff (!rst_n)
    |(evt & evt_urgent_mask) |=> irq)
  else $error("an urgent event was coalesced");

// A bypassed PTP frame is never delayed.
p_ptp_bypass_immediate:
  assert property (@(posedge clk) disable iff (!rst_n)
    (evt[EVT_PTP] && cfg.ptp_bypass) |=> irq)
  else $error("a PTP frame was coalesced despite the bypass");

// The interrupt is never raised with nothing to report.
p_irq_implies_status:
  assert property (@(posedge clk) disable iff (!rst_n)
    irq |-> (irq_status != '0))
  else $error("the interrupt was raised with an empty status word");

// The pending count clears on every firing.
p_pending_clears:
  assert property (@(posedge clk) disable iff (!rst_n)
    batch_flushed |=> (pending == 16'd0))
  else $error("the pending count survived a flush");

// The pending count never exceeds the threshold by more than one
// frame's worth of arrivals.
p_pending_bounded:
  assert property (@(posedge clk) disable iff (!rst_n)
    pending <= (threshold_n + 16'(NUM_EVT)))
  else $error("the pending count ran past the threshold");

// Exactly one cause is credited per firing.
p_one_cause_per_firing:
  assert property (@(posedge clk) disable iff (!rst_n)
    batch_flushed |=>
      ((c_by_urgent + c_by_bypass + c_by_threshold + c_by_floor) ==
       $past(c_by_urgent + c_by_bypass + c_by_threshold + c_by_floor) + 1))
  else $error("a firing was credited to zero or several causes");

Latency and jitter.

// A measured delay never exceeds the floor, by construction.
p_measured_delay_within_floor:
  assert property (@(posedge clk) disable iff (!rst_n)
    worst_delay_us <= (cfg.floor_timer_us + 16'd2))
  else $error("a measured delay exceeded the floor timer");

// The PTP accumulator only advances on PTP samples.
p_ptp_channel_is_ptp_only:
  assert property (@(posedge clk) disable iff (!rst_n)
    (batch_flushed && !have_ptp) |=> $stable(c_ptp_delay_n))
  else $error("a non-PTP sample entered the PTP delay channel");

// The histogram's total equals the sample count.
p_histogram_totals:
  assert property (@(posedge clk) disable iff (!rst_n)
    (hist[0] + hist[1] + hist[2] + hist[3] +
     hist[4] + hist[5] + hist[6] + hist[7]) == c_samples)
  else $error("the delay histogram lost a sample");

// A bypassed PTP frame lands in bucket zero.
p_bypassed_is_bucket_zero:
  assert property (@(posedge clk) disable iff (!rst_n)
    (sample_valid && sample_is_ptp && (sample_us == 16'd0)) |=>
      (hist_ptp[0] == $past(hist_ptp[0]) + 1))
  else $error("a zero-delay sample did not land in bucket zero");

Verdicts.

// The bound holds.
p_bound_never_violated:
  assert property (@(posedge clk) disable iff (!rst_n)
    !bound_violated)
  else $error("section 6's bound was violated in silicon");

// A configuration fault excludes an OK verdict.
p_cfg_fault_excludes_ok:
  assert property (@(posedge clk) disable iff (!rst_n)
    cfg_fault |-> !coalescing_ok)
  else $error("coalescing_ok asserted with a configuration fault");

// Frames signalled never exceed frames arrived.
p_signalled_le_arrived:
  assert property (@(posedge clk) disable iff (!rst_n)
    c_frames_signalled <= c_frames_arrived)
  else $error("more frames were signalled than arrived");

// An all-clear excludes every finding.
p_clear_excludes_findings:
  assert property (@(posedge clk) disable iff (!rst_n)
    none_of_the_above |-> (!bound_violated && !cfg_fault &&
                           !ptp_not_bypassed && !ptp_delay_unstable))
  else $error("none_of_the_above asserted alongside a finding");

20. Verification Scenarios

Fifty-seven scenarios, plus a six-run directed test that requires an arrival process random stimulus will not produce on its own.

Rate estimation — 9 scenarios.

The threshold — 9 scenarios.

The stall proof — 11 scenarios.

Aggregation and bypass — 10 scenarios.

Latency and jitter — 9 scenarios.