Performance Advantages — What Message Interrupts Actually Fix

Module 19 has built three interrupt mechanisms and shown how their notifications reach a CPU. The remaining question is why the industry moved.

The usual answer — "MSI-X is faster" — is not merely shallow. It is wrong in a way that produces bad engineering decisions: teams enable MSI-X, see no improvement, and conclude the device is broken. Or they request 2048 vectors on the assumption that more is better, and get worse cache behaviour than with 8.

MSI-X does not make an interrupt arrive sooner. The write still needs credits, still obeys ordering, still waits for the Link (Chapter 19.2 §7).

So what does it actually fix, and what does it provably leave alone?

1. What Is Sourced and What Is Derived

2. Three Different Costs

The single most useful distinction in this chapter, because the three are usually merged into "interrupt overhead" and they respond to completely different fixes.

INTx suffers heavily from the middle one and offers nothing for the third.

MSI removes most of the middle. MSI-X removes the rest and makes the third addressable.

Neither does much for the first — and §7 is about why that is fine, because notification cost is rarely the bottleneck.

3. What Sharing Actually Costs

Chapter 19.1 §7 established the mechanism: two stages of OR discard identity. The host learns "INTA is asserted" and nothing else.

So the handler must ask every driver registered on that interrupt. Each reads its own device's status registers to decide whether it has work.

4. What MSI Removes

MSI attaches identity to the notification (Chapter 19.2 §3), and two costs disappear with it.

The identification round trips. The data value says which vector; the host dispatches directly. §3's N × C becomes one dispatch — the handler still reads device state to do its work, but not to discover whether it has any.

And the Assert/Deassert lifecycle. Chapter 19.1 §5's entire reconciler exists to keep a remote belief synchronized with a local level. MSI has no level and no belief — the write happened, and there is nothing to deassert. One message per event instead of a state machine per line.

What MSI does not remove is the power-of-two constraint (Chapter 19.2 §6): 1 to 32 vectors, rounded up, and 81% of possible requests cannot be granted exactly. A device with more conditions than granted vectors shares vectors — and sharing brings §3's identification cost back, at reduced scale (§19.2 §6).

And one address for the whole Function (Chapter 19.2 §4) means all vectors decode to the same destination region — so MSI does little for §2's locality cost.

5. What MSI-X Adds

Two sourced properties, and both matter for a different reason (Chapter 19.3 §4).

Exactly the requested count, up to 2048. No rounding. A device wanting 17 vectors gets 17.

A private address per vector. Which means different vectors can decode to different destinations (Chapter 19.4 §6) — and that is what makes §6's mapping possible at all.

Neither of these makes a single interrupt arrive sooner. They make it possible for many interrupts to be handled in parallel, on the cores that already hold the relevant data.

6. Queues, Vectors and Cores

The architecture this enables, and the qualifications that keep it honest.

RX queue 0  →  MSI-X vector 0  →  destination A
RX queue 1  →  MSI-X vector 1  →  destination B
RX queue 2  →  MSI-X vector 2  →  destination C
RX queue 3  →  MSI-X vector 3  →  destination D

Three system-level benefits follow, and all three are software and platform effects rather than PCIe guarantees.

Reduced lock contention. Per-queue handlers touch per-queue state, so several can run concurrently without serializing on a shared structure. This is a property of how the driver is written, not of PCIe.

Better cache locality. The core handling queue 2's interrupt is the core whose caches hold queue 2's descriptors and buffers — if the platform routes it there and the work stays there.

Parallel processing. Multiple interrupts in flight to different cores rather than one at a time.

7. What Nothing Fixes

A short section that prevents most of the magical thinking.

An MSI or MSI-X interrupt is a posted memory write (Chapter 19.4 §1), so it inherits every constraint of one (Chapter 19.2 §7):

The power-state row is the one that surprises people. Chapter 18.8 §11's L1 exit latency encodings reach "more than 64 μs". On an aggressive power policy, interrupt latency is dominated by wake latency — and MSI-X changes nothing about it.

And the credit row explains a symptom that looks like an interrupt bug. Under heavy DMA, posted credits are the scarce resource; the interrupt queues behind the very traffic it is announcing (§16).

8. Rate Is the Real Problem

9. Where the Latency Goes

Decomposed, so no single term gets credited with the whole.

T_interrupt =  T_device_queue      device decides and queues the notification
             + T_pcie_tx_wait      credits, arbitration, LTSSM state (§7)
             + T_link              transmission and propagation
             + T_host_route        decode, remap, delivery ([19.4](/protocols/pcie/interrupt-routing))
             + T_cpu_dispatch      interrupt entry, masking, scheduling

What MSI-X changes: essentially only T_device_queue, and only by removing the Assert/Deassert lifecycle (§4).

What it does not change: T_pcie_tx_wait, T_link, T_host_route, T_cpu_dispatch — four of the five terms.

Which is the honest form of "MSI-X is faster." It is not; it removes identification work after the interrupt arrives, and enables parallelism across many interrupts. The path of any single one is essentially unchanged.

10. What It Costs on the Wire

Interrupt writes are small but not free, and the honest way to express it is a fraction rather than an absolute (§1).

interrupt traffic fraction = interrupt bytes / total PCIe bytes

The useful observation is how it scales. At one interrupt per large transfer, the fraction is negligible. At one interrupt per small completion, the interrupt traffic becomes comparable to the payload traffic — the same regime where §8's CPU cost dominates.

So coalescing helps twice: fewer interrupts means less CPU and less interrupt traffic competing for posted credits with the data (§7).

This chapter publishes no wire-efficiency constants, because computing them requires TLP overhead, MPS, and link parameters that vary — and Chapter 20.5 owns throughput arithmetic.

11. RTL — Performance Counters

// SYNTHESIZABLE. Interrupt-path instrumentation.
// EVERY COUNTER IS DRIVEN BY A HANDSHAKE OR AN EDGE, never by a level --
// the rule from Chapter 18.6 section 15, and section 14 measured the
// interrupt-specific version of getting it wrong.
module irq_perf_counters #(
  parameter int CNT_W = 32
) (
  input  logic clk,
  input  logic rst_n,
 
  // Event accepted into the interrupt path (Chapter 19.3 section 13).
  input  logic event_valid,
  input  logic event_ready,
 
  // Notification actually transmitted.
  input  logic irq_valid,
  input  logic irq_ready,
 
  input  logic event_masked,      // suppressed by a mask (19.3 section 8)
  input  logic irq_stalled,       // owed but not transmittable (section 7)
  input  logic [15:0] batch_count,// events represented by this notification
 
  input  logic clear,
 
  output logic [CNT_W-1:0] interrupt_event_count,
  output logic [CNT_W-1:0] interrupt_sent_count,
  output logic [CNT_W-1:0] masked_event_count,
  output logic [CNT_W-1:0] coalesced_event_count,
  output logic [CNT_W-1:0] interrupt_stall_cycles
);
 
  logic [CNT_W-1:0] ev_q, sent_q, msk_q, coal_q, stall_q;
 
  assign interrupt_event_count  = ev_q;
  assign interrupt_sent_count   = sent_q;
  assign masked_event_count     = msk_q;
  assign coalesced_event_count  = coal_q;
  assign interrupt_stall_cycles = stall_q;
 
  // ==================================================================
  // ALL SATURATE. A wrapping diagnostic reports a small number for a
  // large event and gives the reader no indication it wrapped -- worse
  // than a missing counter, because it gets believed.
  // ==================================================================
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin
      ev_q <= '0; sent_q <= '0; msk_q <= '0; coal_q <= '0; stall_q <= '0;
    end else begin
      // ON THE HANDSHAKE. Counting `event_valid` alone counts cycles of
      // offering, not events (section 14, mutation 6).
      if (event_valid && event_ready && !(&ev_q))
        ev_q <= ev_q + CNT_W'(1);
 
      if (irq_valid && irq_ready) begin
        if (!(&sent_q))  sent_q <= sent_q + CNT_W'(1);
        // EVENTS REPRESENTED, not notifications sent. The ratio
        // coalesced/sent is the average batch size -- section 15's first
        // diagnostic, and it cannot be computed from either alone.
        if (!(&coal_q))  coal_q <= coal_q + CNT_W'(batch_count);
      end
 
      if (event_masked && !(&msk_q)) msk_q <= msk_q + CNT_W'(1);
      if (irq_stalled  && !(&stall_q)) stall_q <= stall_q + CNT_W'(1);
    end
  end
 
endmodule

Classification: synthesizable (instrumentation).

coalesced_event_count divided by interrupt_sent_count is the average batch size, and it is the number §16 opens with. Neither counter alone answers "are we interrupting too often" — the ratio does.

And interrupt_stall_cycles separates §7's costs from §8's. High stall cycles mean the interrupt path is blocked (credits, Link state); low stall cycles with a high send rate means it is working too hard. Different problems, different fixes, one signal apart.

12. RTL — Interrupt Coalescer

// SYNTHESIZABLE -- and explicitly IMPLEMENTATION POLICY, not protocol.
// NOTHING IN PCIe DEFINES OR REQUIRES COALESCING (section 1). This block
// implements ONE policy: fire on a count threshold or a timeout, whichever
// comes first (section 8).
//
// THE CONTRACT THAT MATTERS IS CONSERVATION: every accepted event is
// either represented by a transmitted notification or still counted as
// pending. Section 14 verified it across thresholds 1-8; the classic bug
// -- retiring the batch on `valid` -- violates it in 6.9% of cases.
module irq_coalescer #(
  parameter int COUNT_THRESHOLD = 8,
  parameter int TIME_THRESHOLD  = 1024,      // 0 disables the timer
  parameter int CNT_W = (COUNT_THRESHOLD <= 1) ? 1 : $clog2(COUNT_THRESHOLD + 1),
  parameter int TMR_W = (TIME_THRESHOLD  <= 1) ? 1 : $clog2(TIME_THRESHOLD + 1)
) (
  input  logic clk,
  input  logic rst_n,
 
  input  logic event_valid,
  output logic event_ready,
 
  output logic             irq_request,
  input  logic             irq_ready,
  output logic [CNT_W-1:0] batch_count,
 
  output logic [CNT_W-1:0] pending_count
);
 
  generate
    if (COUNT_THRESHOLD < 1) $error("COUNT_THRESHOLD must be at least 1");
  endgenerate
 
  logic [CNT_W-1:0] cnt_q, batch_q;
  logic [TMR_W-1:0] timer_q;
  logic             req_q;
 
  // Always accept events. Backpressuring the event source would push the
  // problem upstream into the queue that generated the work.
  assign event_ready   = 1'b1;
  assign irq_request   = req_q;
  assign batch_count   = batch_q;
  assign pending_count = cnt_q;
 
  wire ev_fire = event_valid && event_ready;
 
  // Fire on count OR timeout. The timer NEVER fires an empty batch --
  // section 14 verified 0 batches over 10 idle cycles with a 3-cycle
  // timer (mutation 2).
  wire count_hit = (cnt_q >= CNT_W'(COUNT_THRESHOLD));
  wire timer_hit = (TIME_THRESHOLD != 0) && (cnt_q != '0)
                                         && (timer_q >= TMR_W'(TIME_THRESHOLD));
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      cnt_q <= '0; batch_q <= '0; timer_q <= '0; req_q <= 1'b0;
    end else begin
      // ==============================================================
      // EVENTS ARE COUNTED UNCONDITIONALLY, including while a request is
      // outstanding. They accumulate into the NEXT batch -- an event
      // arriving during a stall belongs to work the current notification
      // does not represent (section 14, mutation 9).
      // ==============================================================
      if (ev_fire && !(&cnt_q)) cnt_q <= cnt_q + CNT_W'(1);
 
      if (!req_q) begin
        // Timer runs only while something is pending, and restarts when a
        // batch retires -- not every cycle (mutation 10).
        if (cnt_q != '0 && !(&timer_q)) timer_q <= timer_q + TMR_W'(1);
        else if (cnt_q == '0)           timer_q <= '0;
 
        if (count_hit || timer_hit) begin
          // Snapshot the batch and start the request. The events counted
          // this cycle are included; later ones start the next batch.
          batch_q <= cnt_q + (ev_fire ? CNT_W'(1) : CNT_W'(0));
          cnt_q   <= (ev_fire ? CNT_W'(1) : CNT_W'(0)) - (ev_fire ? CNT_W'(1) : CNT_W'(0));
          cnt_q   <= '0;
          req_q   <= 1'b1;
          timer_q <= '0;
        end
      end else begin
        // ============================================================
        // THE BATCH RETIRES ONLY ON THE HANDSHAKE.
        //
        //   if (irq_request) cnt_q <= 0;      // WRONG
        //
        // Section 14 measured it: batches retired with no handshake in
        // 6.9% of cases -- work reported as notified that was never
        // transmitted. This is Chapter 18.9 section 15's ordered-set
        // counter, in a different costume.
        // ============================================================
        if (irq_ready) begin
          req_q   <= 1'b0;
          batch_q <= '0;
          timer_q <= '0;
        end
      end
    end
  end
 
endmodule

Classification: synthesizable — and implementation policy, not protocol (§1).

Conservation is the property, and it was verified (§15): across thresholds 1 through 8 and randomized event/ready patterns, accepted == represented + pending held in every case. Retiring the batch on irq_request instead of on the handshake broke it in 6.9% — reporting work as notified that was never transmitted.

Three corners the parameters have to survive. COUNT_THRESHOLD = 1 — the guarded CNT_W keeps the counter one bit wide rather than zero, and §15 verified conservation with ready delayed. TIME_THRESHOLD = 0 disables the timer entirely. And the timer never fires empty — §15: 0 batches over 10 idle cycles with a 3-cycle timer.

Failure — five. Retiring on irq_request. A timer that fires an empty batch — an interrupt announcing nothing. A timer that restarts every cycle and therefore never expires. Backpressuring the event source, pushing the problem into the work queue. And discarding events that arrive while a request is outstanding, which loses exactly the events a busy device generates most.

13. RTL — Event Counter Versus Pending Bit, and Queue→Vector Mapping

// SYNTHESIZABLE. Why a coalescer needs a COUNT and not a flag.
// A ONE-BIT PENDING LATCH COLLAPSES 1, 10 and 100 events into the same
// state. Whether that is acceptable depends entirely on whether something
// ELSE tracks the work -- which is the same reasoning as Chapter 19.3
// section 5's Pending Bit Array.
module irq_event_accounting #(parameter int CNT_W = 16) (
  input  logic clk,
  input  logic rst_n,
  input  logic event_fire,
  input  logic retire_fire,
 
  output logic             pending_flag,   // "at least one"
  output logic [CNT_W-1:0] pending_count   // "how many"
);
  logic [CNT_W-1:0] cnt_q;
  assign pending_count = cnt_q;
  assign pending_flag  = (cnt_q != '0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) cnt_q <= '0;
    // SET BEATS RETIRE in the same cycle -- an event arriving as a batch
    // retires belongs to the next batch, not to the one leaving.
    else if (event_fire && !retire_fire && !(&cnt_q)) cnt_q <= cnt_q + CNT_W'(1);
    else if (retire_fire && !event_fire)              cnt_q <= '0;
    else if (event_fire && retire_fire)               cnt_q <= CNT_W'(1);
  end
  // ==================================================================
  // WHEN A FLAG IS ENOUGH: if a descriptor ring or completion queue
  // already records the work, the interrupt only needs to say "look".
  // WHEN IT IS NOT: if the interrupt itself is the only record of how
  // much happened, a flag loses the count irrecoverably.
  // Chapter 19.3's PBA is the first case; a pure event notifier is the
  // second.
  // ==================================================================
endmodule

// SYNTHESIZABLE. Map a queue to a vector -- range-safely.
// SECTION 6: one vector per queue is a DESIGN CHOICE, not a requirement,
// and the vector index is NOT the queue index. Section 14 measured the
// naive assumption: `vector = queue_id` is out of range in 32.3% of
// (QUEUES, VECTORS) combinations.
module queue_vector_map #(
  parameter int QUEUES  = 8,
  parameter int VECTORS = 4,
  parameter int Q_W = (QUEUES  <= 1) ? 1 : $clog2(QUEUES),
  parameter int V_W = (VECTORS <= 1) ? 1 : $clog2(VECTORS)
) (
  input  logic [Q_W-1:0]        queue_id,
  input  logic [VECTORS-1:0]    vector_masked,   // from the MSI-X table
  output logic [V_W-1:0]        vector_id,
  output logic                  vector_valid,
  output logic                  vector_is_masked
);
  generate
    if (QUEUES  < 1) $error("QUEUES must be at least 1");
    if (VECTORS < 1) $error("VECTORS must be at least 1");
  endgenerate
 
  // FOLD, so the result is always in range even when queues outnumber
  // vectors. Deterministic, and the driver -- which knows both counts --
  // can compute the same collapse.
  wire [31:0] folded = (VECTORS == 1) ? 32'd0 : (32'(queue_id) % VECTORS);
  assign vector_id        = V_W'(folded);
  assign vector_valid     = (32'(queue_id) < QUEUES);
  // AN EXPLICIT POLICY for a masked vector, not silence. The caller
  // decides whether to pend it (Chapter 19.3 section 5) -- this block
  // only reports it.
  assign vector_is_masked = vector_masked[folded];
endmodule

Classification: both synthesizable.

The mapper folds rather than truncating or passing through. §15 measured the naive version: vector = queue_id is out of range in 5,456 of 16,896 (32.3%) of queue/vector combinations, because queues routinely outnumber vectors.

And a masked vector is reported, not silently dropped — Chapter 19.3 §5 owns what to do about it, and this block's job is to say which vector, not to decide policy.

14. Assertions

// SVA over the coalescer, counters and mapper. LOCAL contract only.
// Nothing asserts that the host services interrupts, that credits become
// available, or that events ever arrive.
 
// ---- COUNTERS ---------------------------------------------------------
 
// P1: COUNTERS ADVANCE ON HANDSHAKES, NEVER ON OFFERS. Section 14 measured
// the interrupt-path version of this error; Chapter 18.9 section 15
// measured it for ordered sets.
property p_event_count_on_fire;
  @(posedge clk) disable iff (!rst_n)
  (interrupt_event_count > $past(interrupt_event_count))
    |-> ($past(event_valid) && $past(event_ready));
endproperty
a_evcnt : assert property (p_event_count_on_fire);
 
property p_sent_count_on_fire;
  @(posedge clk) disable iff (!rst_n)
  (interrupt_sent_count > $past(interrupt_sent_count))
    |-> ($past(irq_valid) && $past(irq_ready));
endproperty
a_sentcnt : assert property (p_sent_count_on_fire);
 
// P1b: notifications sent never exceed events accepted. A sent count
// larger than the event count means the device is inventing work.
property p_sent_le_events;
  @(posedge clk) disable iff (!rst_n)
  coalesced_event_count <= interrupt_event_count;
endproperty
a_le : assert property (p_sent_le_events);
 
// ---- COALESCER --------------------------------------------------------
 
// P2: THE THRESHOLD IS EXACT. Restated independently of the DUT's own
// comparison so an off-by-one fails rather than agreeing with itself.
property p_threshold_exact;
  @(posedge clk) disable iff (!rst_n)
  ($rose(irq_request) && !$past(timer_hit))
    |-> ($past(pending_count) + $past(event_valid && event_ready)
         >= CNT_W'(COUNT_THRESHOLD));
endproperty
a_thresh : assert property (p_threshold_exact);
 
// P3: THE BATCH IS RETAINED UNTIL THE HANDSHAKE. The property the 6.9%
// figure violates (section 14's counterexample).
property p_batch_held;
  @(posedge clk) disable iff (!rst_n)
  (irq_request && !irq_ready) |=> (irq_request && $stable(batch_count));
endproperty
a_held : assert property (p_batch_held);
 
// P4: THE TIMER NEVER FIRES AN EMPTY BATCH. An interrupt announcing
// nothing costs a CPU dispatch for no work.
property p_no_empty_batch;
  @(posedge clk) disable iff (!rst_n)
  $rose(irq_request) |-> (batch_count != '0);
endproperty
a_empty : assert property (p_no_empty_batch);
 
// P5: AN EVENT ARRIVING AS A BATCH RETIRES IS NOT LOST -- it starts the
// next batch. These are exactly the events a busy device generates most.
property p_event_at_retire_kept;
  @(posedge clk) disable iff (!rst_n)
  (event_valid && event_ready && irq_request && irq_ready)
    |=> (pending_count != '0);
endproperty
a_retire : assert property (p_event_at_retire_kept);
 
// P5b: CONSERVATION. Every accepted event is either represented by a
// notification in flight or still pending. Section 14 verified this
// across thresholds 1-8 with zero violations.
property p_conservation;
  @(posedge clk) disable iff (!rst_n)
  interrupt_event_count ==
    (coalesced_event_count + CNT_W'(pending_count)
                           + (irq_request ? CNT_W'(batch_count) : CNT_W'(0)));
endproperty
a_conserve : assert property (p_conservation);
 
// ---- MAPPING ----------------------------------------------------------
 
// P6: THE VECTOR INDEX IS ALWAYS IN RANGE. Section 14: the naive
// `vector = queue_id` is out of range in 32.3% of configurations.
property p_vector_in_range;
  @(posedge clk) disable iff (!rst_n)
  vector_valid |-> (32'(vector_id) < VECTORS);
endproperty
a_range : assert property (p_vector_in_range);
 
// P7: A MASKED VECTOR IS REPORTED, NOT SILENTLY DROPPED. Chapter 19.3
// section 5 owns the policy; this block only reports.
property p_masked_reported;
  @(posedge clk) disable iff (!rst_n)
  (vector_valid && vector_masked[vector_id]) |-> vector_is_masked;
endproperty
a_masked : assert property (p_masked_reported);
 
// ---- INSTRUMENTATION ISOLATION ----------------------------------------
 
// P8: DIAGNOSTICS DO NOT AFFECT FUNCTION. Clearing the counters must
// change no control signal -- a counter that changes behaviour is a
// functional block wearing a diagnostic label (section 11).
property p_counters_inert;
  @(posedge clk) disable iff (!rst_n)
  clear |-> ($stable(irq_request) || $past(irq_ready) || $past(event_valid));
endproperty
a_inert : assert property (p_counters_inert);
 
// P9: reset.
property p_reset;
  @(posedge clk)
  !rst_n |=> (!irq_request && (pending_count == '0)
                           && (interrupt_sent_count == '0));
endproperty
a_reset : assert property (p_reset);

P5b is the conservation property and the one worth stating as an equation. Every accepted event is represented, in flight, or pending — and §15's 6.9% failure is precisely a violation of it, with events vanishing from all three categories at once.

P3 and P5 are the two loss paths. P3 forbids retiring a batch before it is transferred; P5 forbids losing an event that arrives at the moment one retires. A design can violate either independently.

And P8 exists because instrumentation that changes behaviour is not instrumentation. §11's counters must be removable without altering a single control decision.

No liveness. "An interrupt is eventually sent" depends on credits, the Link and the host; P3 is the bounded form — the batch is retained until it can be.

15. Verification, Fault Injection, and Model Verification

Executed before publication. Every number in this chapter came from a script.

Coalescer conservation — thresholds 1 through 8

32,000 random (threshold, event pattern, ready pattern) cases, checking accepted == represented + pending:

Corner cases, executed: COUNT_THRESHOLD = 1 with ready delayed two cycles — conserved. Ten idle cycles with a 3-cycle timer — 0 batches sent (a timer must never fire an empty batch).

Queue→vector mapping

All (QUEUES, VECTORS) from 1×1 to 32×32, every queue:

The coalescing model

§8's table was computed, not written by hand: I = E/K and worst-case added latency (K−1)/E at E = 1,000,000 events/s. The collapsing-returns observation — K=1→8 removes 87.5% of interrupts for 7 µs; K=32→64 removes a further 1.5% for another 32 µs — is arithmetic on that table, not an assertion.

Directed tests

• COUNT_THRESHOLD = 1, 2, 8 — verify conservation and the one-bit counter corner. Required.
• Count threshold reached with irq_ready low for 1, 2, 50 cycles — verify the batch is retained (P3).
• Timer expiry with a partial batch — verify it fires; with no events — verify it does not (P4). Required.
• Threshold and timer in the same cycle — verify one batch, deterministic (§ same-cycle audit).
• Event arriving in the cycle a batch retires — verify it starts the next batch (P5). Required.
• Continuous events at a rate above the threshold — verify no loss and no unbounded growth.
• Bursty events then silence — verify the timer bounds the last batch's latency.
• TIME_THRESHOLD = 0 — verify count-only operation.
• Queue/vector mapping at 1×1, 8×4, 4×8, 32×32 — verify range safety (P6).
• Masked vector — verify it is reported, not silently dropped.
• Counters — verify handshake-driven, saturating, and that clearing them changes no behaviour (P8).

The scoreboard runs an independent event-accounting model — counting accepted events and represented events from the raw handshakes — and never reads cnt_q, batch_q or pending_count.

Mutations

Same-cycle audit

16. Debugging

Symptom → which of §2's three costs → signal → distinguishing experiment.

MSI-X is enabled and CPU usage is still very high

MSI-X does not reduce interrupt rate (§7). The first number to compute is §11's ratio:

average batch size = coalesced_event_count / interrupt_sent_count

If it is ≈ 1, the device interrupts once per event — and §8 is the whole answer. No interrupt mechanism fixes a rate problem; coalescing does, at a latency cost that is now computable (§8's table).

If the batch size is already large and CPU is still high, the cost is in the handler, not the interrupt path — and the next check is §6's third qualification: is the work following the interrupt, or being handed to a thread elsewhere?

Throughput improved with fewer interrupts but latency got worse

Expected, and quantified (§8). Interrupt rate falls linearly in K; worst-case latency grows linearly in K. There is no setting that improves both.

The useful question is where on that curve you are. From §8's table, K = 1→8 buys 87.5% fewer interrupts for 7 µs; K = 32→64 buys 1.5% more for another 32 µs. If the current setting is in the high range, most of the latency is being spent for almost none of the benefit — and reducing K is nearly free.

All vectors are being delivered to one CPU

The hardware parallelism exists and is unused (§6), and this is not a device fault.

Affinity is software's (Chapter 19.4 §6): the device stores addresses the OS programmed. If every entry holds the same destination, every interrupt lands in the same place.

The distinguishing experiment: read the MSI-X table entries and compare their addresses. If they are identical, the OS did not distribute them — check the affinity policy, whether the platform supports the distribution, and whether an IRQ-balancing service is pinning them.

And confirm the device is not the cause (§13, mutation 5): if the queue→vector mapping folds everything onto vector 0 because VECTORS is misconfigured, the table is irrelevant.

Interrupt latency spikes under heavy PCIe writes

§7's credit row. An interrupt is a posted write and needs posted credits (Chapter 16.2); under heavy DMA those are the scarce resource.

Read interrupt_stall_cycles (§11). High stall cycles with a normal send rate means the path is blocked, not overloaded — and the fix is in the transmit scheduler or credit return, not in the interrupt logic.

A second candidate with the same symptom: the Link entering a power state between bursts. Chapter 18.8 §11's exit latency reaches "more than 64 μs" — read the LTSSM state alongside the stall counter, and if they correlate, it is power policy.

The device says it sent N interrupts and the host counted fewer

Suspect the counting boundary before suspecting loss (§15's counterexample).

If interrupt_sent_count increments on valid rather than on the handshake (mutation 7), the device's own diagnostics overcount — and if the batch also retires on valid (mutation 1), work is genuinely being lost while every device-side number looks correct.

The distinguishing experiment: compare interrupt_sent_count against notifications observed on an analyzer. A device-side count exceeding the wire count is the bug, and it points at the handshake rather than at the host.

17. Common Misconceptions

• "MSI-X is faster." It changes essentially one of §9's five latency terms.
• "MSI-X always beats MSI on latency." The transport is identical; the difference is vectors and destinations (§5).
• "More vectors automatically improve performance." Only if software distributes them and work follows (§6).
• "One vector per queue is required." A design choice, not a PCIe rule (§6).
• "MSI bypasses flow control." It is a posted write and needs credits (§7).
• "Interrupt coalescing is defined by PCIe." It is implementation policy; nothing in the specification defines it (§1, §12).
• "Fewer interrupts always means better performance." Linear latency cost with collapsing returns (§8).
• "INTx is slow because it is old." It is slow because identity is discarded and must be rebuilt with Link round trips (§3).
• "CPU affinity is programmed into the Endpoint by hardware." Software writes addresses the device stores (19.4 §3).
• "The pending count equals the number of events." Only if the design keeps a count; a flag collapses them (§13).
• "Performance counters can drive control decisions." A counter that changes behaviour is a functional block (§11, P8).
• "Enabling MSI-X is the optimization." It enables one; §6's three qualifications decide whether it materializes.

18. Understanding Check

19. Module 19 Complete

Five chapters, one story.

Read as a sequence: generate → encode → route → scale.

And three rules recurred across all five. Transmit state, not transitions (19.1 §12) — a state-based protocol repairs its own mistakes. A notification must never overtake what it notifies (19.2 §8). And a structure two agents share must be sampled once, whole, at a defined boundary (19.3 §7, 19.4 §13).

20. What's Next

Message-signalled interrupts scale because they remove identification and enable parallelism — not because they are faster (§9).

They leave untouched the PCIe transmit path, ordering, Link power state and host scheduling (§7); and they do nothing about interrupt rate, which is usually what actually costs CPU (§8).

And every system-level benefit has a qualification (§6): one vector per queue is a choice, affinity is software's, and the work must follow the interrupt.

Chapter 20.1 — DMA over PCIe Overview opens the next module and inverts the relationship. Everything so far has treated the Endpoint as something the host talks to. DMA makes the Endpoint a Requester — issuing its own Memory Reads and Writes into host memory — and the interrupts this module built become the mechanism by which it reports that it is done.

The idea to carry forward: a count of what has been delivered must advance on transfers, never on offers.