MSI-X — Moving the Interrupt Configuration Into Memory

Chapter 19.2 ended on three limits, and all three have the same cause.

MSI stores its configuration inside a Configuration Space capability — a small, fixed structure. There is room for one address and one 16-bit data value (§19.2 §4), so vectors have to be encoded in the low bits of that one value, counts must be powers of two, and the ceiling is 32.

MSI-X's answer is to stop storing the configuration in Configuration Space.

The table lives in Memory Space, behind a BAR — ordinary device memory the host maps and writes. A structure with no fixed size limit can hold a full entry per vector, and everything MSI could not do follows.

What does moving the table into memory buy, and what does it cost?

1. The Verified Sources

2. The Table Is Device Memory

The structural change in one picture.

MSI                                    MSI-X
───                                    ─────
Configuration Space capability         Configuration Space capability
  MSI Enable                             MSIX Enable, Function Mask
  ONE address                            Table BIR + offset  ─┐
  ONE 16-bit data                        PBA   BIR + offset  ─┤
  32 mask bits                                                │
  32 pending bits                                             │
                                                              ▼
                                       Memory Space, behind a BAR:
                                         entry 0:  address, data, control
                                         entry 1:  address, data, control
                                         ...
                                         entry N:  address, data, control
                                         + Pending Bit Array

The capability no longer holds the configuration. It holds a pointer to it — a BAR indicator and an offset (§1).

3. An Entry Per Vector

Each table entry holds what MSI kept one copy of, plus its own control.

And separately, the Pending Bit Array — one bit per vector, in its own memory region (§5).

Compare with Chapter 19.2 §4's MSI: one address, one 16-bit data, for the entire Function. MSI-X gives every vector its own everything.

The immediate consequence is that vectors no longer share an address. In MSI, all 32 vectors write to the same place and are distinguished by data bits (Chapter 19.2 §5). In MSI-X, vector 5 and vector 6 may target completely different addresses — which is what makes per-vector CPU affinity possible: different vectors delivered to different interrupt controllers, and therefore different cores.

And the device does not interpret any of it (§6). It reads the entry and transmits the write. The address is opaque, the data is opaque, and there is nothing to compose.

4. What the Table Buys

The two rows that change system design most are granularity and address.

Granularity (§16): a device wanting 17 vectors gets exactly 17, not 32 with 15 wasted. Across requests 1–32, MSI cannot grant 81% exactly (Chapter 19.2 §15); MSI-X grants every one of them.

Per-vector addresses let affinity be expressed in the table itself rather than by remapping every interrupt at the host. The addresses are still software-programmed and platform-decoded (Chapter 19.4) — the device stores a destination it does not choose and cannot interpret. Queue 3's completions can be delivered to the core that owns queue 3 — which is the mechanism behind every multi-queue NIC and NVMe device.

And 2048 is not an arbitrary ceiling — it is 64× MSI's 32, which is what makes per-queue interrupts practical at all.

5. Two Structures, Possibly Two BARs

§1 gives the table and the PBA independent BAR indicators and offsets. They may live in the same BAR at different offsets, or in different BARs entirely.

Why separate them? They have different access patterns. The table is written by software and read by hardware. The PBA is written by hardware and read by software. Different directions, different frequencies — and a device may want them in memory regions with different properties.

The PBA's role is the same as MSI's pending bits (Chapter 19.2 §9): a masked vector's interrupt sets its pending bit rather than being discarded, and unmasking delivers it. Verified across every mask and pending combination (§16): zero interrupts lost.

The design consequence (§12): the PBA is hardware-owned state that software reads. A device that let software writes into the PBA region clear bits arbitrarily would be letting software discard interrupts it had not serviced — so §12 treats the PBA as read-only to the host and owned by the interrupt logic.

6. The Device Does Not Interpret the Entry

A short section, because the rule is short and the failures from breaking it are not.

Software writes an address and a data value into a table entry. The device reads them and transmits a memory write. That is all.

The device must not: compute or adjust the address; compose the data from anything (there is no vector encoding — §3); assume any relationship between entries; or validate that the address is "reasonable."

Because the device has no idea what the values mean. They are host-side interrupt-controller addresses whose layout is a platform property. An "obviously wrong" address may be exactly right on a system the device has never seen.

And the failure from getting this wrong is severe. A device that modified an address would be issuing a memory write to a location software did not choose — which is not a mangled interrupt, it is a stray DMA write into host memory. §15's mutation 2, and the reason §13's RTL passes both fields through untouched.

7. Reading a Table Software Is Writing

8. Two Masks

§1 gives two independent controls:

Per-vector Mask — in each table entry's Vector Control field. Masks one vector.

Function Mask — Bit 30 of the capability. Masks every vector of the Function at once.

They serve different purposes. The per-vector mask is the fine-grained control software uses around reprogramming (§7) and re-entrancy. The Function Mask is a single-write global off switch — useful when software must quiesce a device's interrupts entirely without walking a 2048-entry table.

The combining rule is an OR: a vector is masked if either its own mask bit or the Function Mask is set. Verified (§16): across every combination of per-vector mask, Function Mask and pending state, zero interrupts lost — a masked vector always sets its pending bit.

The RTL consequence (§12): both must be checked, and a design that checked only the per-vector bit would ignore a global quiesce that software believed had taken effect.

9. Everything MSI Inherited, MSI-X Inherits Too

MSI-X interrupts are transmitted on the same path as data (§1), so Chapter 19.2 §§7–8 apply unchanged:

• Posted credits — no credits, no interrupt (Chapter 16.2)
• Ordering — the interrupt must not overtake the data it announces (Chapter 13.4)
• LTSSM state — transmittable only in L0 (Chapter 18.6 §7)
• Power states — a pending interrupt wakes the Link and then waits (Chapter 18.8)

The ordering requirement is worth restating because MSI-X makes it easier to get wrong. With 2048 vectors and per-queue affinity, a natural design gives each queue its own interrupt path — and separate paths have no ordering relationship (Chapter 19.2 §15).

A per-queue interrupt must be ordered behind that queue's data, which is the guarantee that matters. It does not need ordering against another queue's data, and demanding that would serialize independent queues for nothing. §13's interlock is therefore per-vector, not global.

10. The Table

Three things to read out of the figure.

The capability holds no addresses or data — only enable/mask controls and two pointers. That is §2's whole change.

The table and PBA are located independently (§5), and may be in different BARs.

And the entry is the unit. Address, data and mask travel together for one vector — which is why §7's partial-write hazard is about entries, not about the table as a whole.

11. A Trace

Internal teaching signals. Table size 8, and vector 3 is being reprogrammed.

cycle              1     2     3     4     5     6     7     8     9    10
event_vec3         0     1     0     0     0     0     0     1     0     0
msix_enable        1     1     1     1     1     1     1     1     1     1
func_mask          0     0     0     0     0     0     0     0     0     0
entry3_mask        0     0     1     1     1     0     0     0     0     0
 
sw_write_entry3    0     0     0     1     1     0     0     0     0     0
data_pending       0     1     1     0     0     0     0     0     0     0
 
msix_valid         0     0     0     0     0     0     0     0     1     1
msix_ready         0     0     0     0     0     0     0     0     0     1
msix_addr         --    --    --    --    --    --    --    --  NEW   NEW
pba_bit3           0     1     1     1     1     1     1     1     1     0

Read cycle 2. An event on vector 3 — and nothing is transmitted. data_pending is high (§9), so the interrupt cannot overtake its data. pba_bit3 sets, recording the obligation.

Read cycle 3. Software masks entry 3 before modifying it (§7).

Read cycles 4–5. The entry is rewritten. The device does not read it — the mask forbids it — so no half-written entry can be used. This is the mask acting as mutual exclusion, not as "don't interrupt me."

Read cycle 6. The mask clears. The entry is now coherent.

Read cycles 9–10. The pending interrupt is transmitted — to NEW, the reprogrammed address. The event happened before the reprogramming; the delivery happens after; and it goes to the new destination, which is correct: software moved the vector and expects it there.

And read pba_bit3 at cycle 10. It clears on the accepted transfer, not when the write was offered. §12's rule, and mutation 8's failure.

12. RTL — Types, Table Access, and the PBA

// SYNTHESIZABLE. Normalized MSI-X types.
// 1..2048 VECTORS is canonical (section 1), encoded as size-1 in the
// capability. The entry's FIELD CONTENTS are named as the sources name
// them; no byte layout is published in this chapter.
package msix_pkg;
 
  parameter int MSIX_MAX_VEC = 2048;
  parameter int VEC_W        = 11;    // $clog2(2048)
 
  typedef struct packed {
    logic [63:0] address;    // OPAQUE -- transmitted verbatim (section 6)
    logic [31:0] data;       // OPAQUE
    logic        mask;       // Vector Control mask bit
  } msix_entry_t;
 
  typedef struct packed {
    logic            enable;        // MSIX Enable      (section 1, bit 31)
    logic            function_mask; // MSIX Function Mask (bit 30)
    logic [VEC_W:0]  table_size;    // 1..2048, from capability + 1
  } msix_cfg_t;
 
  // ==================================================================
  // A VECTOR IS MASKED IF EITHER MASK IS SET (section 8).
  //
  // Checking only the per-vector bit ignores a global quiesce software
  // believes has taken effect (section 15, mutation 4).
  // ==================================================================
  function automatic bit vector_masked(input msix_cfg_t c, input msix_entry_t e);
    return c.function_mask || e.mask;
  endfunction
 
  // Range check. Section 16: over table sizes 1..64, 23.9% of index
  // values representable in the index WIDTH are OUT OF RANGE. The width
  // does not bound the index; only the size does.
  function automatic bit vector_in_range(input msix_cfg_t c,
                                         input logic [VEC_W-1:0] v);
    return ({1'b0, v} < c.table_size);
  endfunction
 
endpackage

import msix_pkg::*;
 
// SYNTHESIZABLE. Read one table entry -- safely.
// THE TABLE IS DEVICE MEMORY SOFTWARE WRITES CONCURRENTLY (section 7).
// The protocol's mutual-exclusion primitive is the MASK: software masks a
// vector before modifying its entry. The device's half of that contract is
// to NEVER READ A MASKED ENTRY FOR TRANSMISSION.
module msix_entry_fetch (
  input  logic clk,
  input  logic rst_n,
 
  input  msix_cfg_t        cfg,
  input  logic             req,
  input  logic [VEC_W-1:0] vec,
 
  // Table storage. Software writes this through the BAR; those writes are
  // NOT visible to this port as a separate interface -- they simply change
  // what is stored, whenever they arrive.
  input  msix_entry_t      tbl_rdata,
  output logic [VEC_W-1:0] tbl_raddr,
 
  output msix_entry_t      entry,
  output logic             entry_valid,
  output logic             refused_masked,
  output logic             refused_range
);
 
  msix_entry_t ent_q;
  logic        val_q, rm_q, rr_q;
 
  assign tbl_raddr      = vec;
  assign entry          = ent_q;
  assign entry_valid    = val_q;
  assign refused_masked = rm_q;
  assign refused_range  = rr_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      ent_q <= '0; val_q <= 1'b0; rm_q <= 1'b0; rr_q <= 1'b0;
    end else begin
      val_q <= 1'b0; rm_q <= 1'b0; rr_q <= 1'b0;
      if (req) begin
        // ==========================================================
        // RANGE FIRST. An out-of-range index would read outside the
        // table -- whatever memory happens to follow it -- and transmit
        // a write to an address that is not an interrupt destination at
        // all (section 15, mutation 3).
        // ==========================================================
        if (!vector_in_range(cfg, vec)) begin
          rr_q <= 1'b1;
        end
        // ==========================================================
        // THEN THE MASK, AND ONLY THEN IS THE ENTRY USED.
        //
        // Checking the mask AFTER accepting the entry has already used a
        // possibly half-written entry (section 7). The mask is the
        // mutual-exclusion primitive; consulting it late defeats it.
        // ==========================================================
        else if (vector_masked(cfg, '{address:'0, data:'0,
                                      mask: tbl_rdata.mask})) begin
          rm_q <= 1'b1;
        end else begin
          // CAPTURED WHOLE, in one assignment. Nothing downstream
          // re-reads the table -- an entry's fields belong together, and
          // software may rewrite them a cycle later (Chapter 11.6 §7's
          // rule, applied to a structure software owns).
          ent_q <= tbl_rdata;
          val_q <= 1'b1;
        end
      end
    end
  end
 
endmodule

Classification: both synthesizable.

Range before mask, and mask before use. An out-of-range index reads outside the table; a masked entry may be mid-rewrite (§7). Both refusals are reported rather than silently swallowed, because both indicate a real problem — one in the device's own vector mapping, one in software's discipline.

And the entry is captured whole. Nothing downstream re-reads the table, because software may rewrite it a cycle later.

Failure — four. Mask checked after the entry is accepted — the §7 hazard. No range check, relying on index width (§16: 23.9% of representable values are out of range). Function Mask ignored (§8). And re-reading the table at transmit time.

import msix_pkg::*;
 
// SYNTHESIZABLE. The Pending Bit Array.
// HARDWARE-OWNED, SOFTWARE-READABLE (section 5). A masked vector's
// interrupt sets its bit rather than being discarded; unmasking delivers
// it. Section 16 verified zero losses across every mask/pending combination.
module msix_pba #(parameter int N = 64) (
  input  logic clk,
  input  logic rst_n,
 
  input  logic             set_req,     // interrupt owed but cannot be sent
  input  logic [VEC_W-1:0] set_vec,
 
  input  logic             clr_req,     // ...and it was ACTUALLY transmitted
  input  logic [VEC_W-1:0] clr_vec,
 
  output logic [N-1:0]     pba
);
 
  logic [N-1:0] pba_q;
  assign pba = pba_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) pba_q <= '0;
    else begin
      // ==============================================================
      // SET BEATS CLEAR (Chapter 19.1 section 11's rule).
      //
      // An event arriving in the cycle a transmission completes must not
      // be lost -- the transmitted interrupt announced the earlier state,
      // not this new event.
      //
      // NOTE what is absent: any software write path. The PBA is
      // hardware-owned. Letting software clear bits would let it discard
      // interrupts it had not serviced (section 5).
      // ==============================================================
      if (set_req && (set_vec < VEC_W'(N))) pba_q[set_vec] <= 1'b1;
      else if (clr_req && (clr_vec < VEC_W'(N))) pba_q[clr_vec] <= 1'b0;
    end
  end
 
endmodule

Classification: synthesizable.

Set beats clear, for the same reason as Chapter 19.1 §11: an event arriving as a transmission completes describes something the transmitted interrupt did not announce.

And there is no software write path. The PBA is hardware's record of what it owes; software reads it to see what is pending, and clears the condition by unmasking so the interrupt is delivered — not by erasing the record.

13. RTL — MSI-X Request Owner

import msix_pkg::*;
 
// SYNTHESIZABLE. Issue one MSI-X interrupt.
// SAME OWNERSHIP PATTERN as Chapter 19.2's MSI request owner, with two
// differences: the address and data come from a TABLE ENTRY rather than
// being composed (section 6), and the ordering interlock is PER-VECTOR
// (section 9) so independent queues are not serialized against each other.
module msix_request #(parameter int N = 64) (
  input  logic clk,
  input  logic rst_n,
 
  input  msix_cfg_t        cfg,
  input  logic             event_pulse,
  input  logic [VEC_W-1:0] event_vec,
 
  // ==============================================================
  // PER-VECTOR ORDERING INTERLOCK (section 9).
  //
  // High while posted writes THIS vector announces are still outstanding.
  // A GLOBAL interlock would be correct but would serialize independent
  // queues against each other for no benefit -- and with 2048 vectors and
  // per-queue affinity, that is the whole point of MSI-X.
  // ==============================================================
  input  logic [N-1:0]     data_pending_per_vec,
 
  // Table read (section 12)
  output logic             fetch_req,
  output logic [VEC_W-1:0] fetch_vec,
  input  msix_entry_t      fetch_entry,
  input  logic             fetch_valid,
  input  logic             fetch_masked,
 
  // Transmit path -- the SAME path as data (section 9)
  output logic             tx_valid,
  output logic [63:0]      tx_addr,
  output logic [31:0]      tx_data,
  input  logic             tx_ready,
  input  logic             link_ok,
 
  output logic             pba_set,
  output logic [VEC_W-1:0] pba_set_vec,
  output logic             pba_clr,
  output logic [VEC_W-1:0] pba_clr_vec
);
 
  typedef enum logic [1:0] { S_IDLE, S_FETCH, S_SEND } st_e;
  st_e st_q;
 
  msix_entry_t     held_q;
  logic [VEC_W-1:0] vec_q;
 
  assign fetch_req = (st_q == S_FETCH);
  assign fetch_vec = vec_q;
 
  // ADDRESS AND DATA STRAIGHT FROM THE ENTRY. Nothing composed, nothing
  // adjusted (section 6). A device that modified the address would be
  // issuing a stray DMA write, not an interrupt.
  assign tx_addr = held_q.address;
  assign tx_data = held_q.data;
 
  wire vec_data_pending = data_pending_per_vec[vec_q];
 
  assign tx_valid = (st_q == S_SEND) && cfg.enable
                 && !vec_data_pending && link_ok;
 
  assign pba_clr     = tx_valid && tx_ready;   // ON THE HANDSHAKE only
  assign pba_clr_vec = vec_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      st_q <= S_IDLE; held_q <= '0; vec_q <= '0;
      pba_set <= 1'b0; pba_set_vec <= '0;
    end else begin
      pba_set <= 1'b0;
 
      if (event_pulse && cfg.enable) begin
        // EVERY event records itself in the PBA first. If it can be sent
        // immediately the bit clears on the handshake; if not, the record
        // is already there. Nothing depends on the event recurring.
        pba_set     <= 1'b1;
        pba_set_vec <= event_vec;
      end
 
      unique case (st_q)
        S_IDLE :
          if (event_pulse && cfg.enable) begin
            vec_q <= event_vec;
            st_q  <= S_FETCH;
          end
 
        S_FETCH :
          // A masked or out-of-range vector does NOT proceed. The PBA bit
          // is already set, so unmasking will deliver it later (section 5).
          if (fetch_masked)     st_q <= S_IDLE;
          else if (fetch_valid) begin
            held_q <= fetch_entry;   // captured whole
            st_q   <= S_SEND;
          end else                   st_q <= S_IDLE;   // range refusal
 
        S_SEND :
          if (tx_valid && tx_ready) st_q <= S_IDLE;
 
        default : st_q <= S_IDLE;
      endcase
    end
  end
 
endmodule

Classification: synthesizable.

The per-vector interlock is the MSI-X-specific decision. Chapter 19.2 §13's MSI had one interlock because it effectively had one interrupt stream. With per-queue vectors, a global interlock would make queue 7's interrupt wait for queue 2's data — correct, but pointlessly serializing the independence MSI-X exists to provide (§9).

And every event sets its PBA bit first. Whether it transmits immediately or not, the obligation is recorded before anything can go wrong — so a stall, a mask, or a link outage cannot lose it.

Failure — five. A global interlock serializes independent queues. No interlock reintroduces Chapter 19.2 §8's corruption. Composing or adjusting the address (§6). Clearing the PBA on tx_valid rather than the handshake. And re-reading fetch_entry at transmit time, after software may have rewritten it.

14. Assertions

// SVA over the MSI-X blocks. LOCAL contract only.
 
// ---- TABLE ACCESS -----------------------------------------------------
 
// P1: A MASKED ENTRY IS NEVER USED FOR TRANSMISSION. The device's half of
// section 7's mutual-exclusion contract.
property p_masked_not_fetched;
  @(posedge clk) disable iff (!rst_n)
  entry_valid |-> !vector_masked(cfg, entry);
endproperty
a_mask : assert property (p_masked_not_fetched);
 
// P2: THE VECTOR INDEX IS ALWAYS IN RANGE. Section 16: the index WIDTH
// does not bound it -- 23.9% of representable values exceed the table.
property p_in_range;
  @(posedge clk) disable iff (!rst_n)
  entry_valid |-> vector_in_range(cfg, $past(vec));
endproperty
a_range : assert property (p_in_range);
 
// P3: the entry is captured WHOLE and stable while it is used.
property p_entry_stable;
  @(posedge clk) disable iff (!rst_n)
  (tx_valid && !tx_ready) |=> ($stable(tx_addr) && $stable(tx_data));
endproperty
a_stable : assert property (p_entry_stable);
 
// ---- TRANSMISSION -----------------------------------------------------
 
// P4: THE ADDRESS AND DATA ARE TRANSMITTED VERBATIM (section 6). The
// device does not compute, adjust or compose them.
property p_verbatim;
  @(posedge clk) disable iff (!rst_n)
  tx_valid |-> ((tx_addr == held_q.address) && (tx_data == held_q.data));
endproperty
a_verbatim : assert property (p_verbatim);
 
// P5: NO INTERRUPT BEFORE THE DATA IT ANNOUNCES -- per vector (section 9).
property p_ordering;
  @(posedge clk) disable iff (!rst_n)
  tx_valid |-> !data_pending_per_vec[vec_q];
endproperty
a_order : assert property (p_ordering);
 
// P5b: and it uses the ORDINARY transmit path, so ordering applies at all.
property p_same_path;
  @(posedge clk) disable iff (!rst_n)
  (tx_valid && tx_ready) |-> dut_txpath.accepted_this_cycle;
endproperty
a_path : assert property (p_same_path);
 
// P6: enabled, and the Link permits traffic.
property p_enabled_and_link;
  @(posedge clk) disable iff (!rst_n) tx_valid |-> (cfg.enable && link_ok);
endproperty
a_en : assert property (p_enabled_and_link);
 
// ---- PENDING ----------------------------------------------------------
 
// P7: EVERY EVENT RECORDS ITSELF. Nothing depends on the event recurring.
property p_event_records;
  @(posedge clk) disable iff (!rst_n)
  (event_pulse && cfg.enable) |=> pba[$past(event_vec)];
endproperty
a_rec : assert property (p_event_records);
 
// P8: A MASKED VECTOR'S INTERRUPT IS NOT LOST -- section 16 verified zero
// losses across every mask/pending combination.
property p_masked_pends;
  @(posedge clk) disable iff (!rst_n)
  (fetch_masked && cfg.enable) |-> pba[vec_q];
endproperty
a_pend : assert property (p_masked_pends);
 
// P9: THE FUNCTION MASK MASKS EVERYTHING (section 8).
property p_func_mask;
  @(posedge clk) disable iff (!rst_n) cfg.function_mask |-> !tx_valid;
endproperty
a_fmask : assert property (p_func_mask);
 
// P10: a pending bit clears ONLY on an accepted transfer of that vector.
property p_clear_on_fire;
  @(posedge clk) disable iff (!rst_n)
  ($past(pba[v]) && !pba[v])
    |-> ($past(tx_valid) && $past(tx_ready) && ($past(vec_q) == VEC_W'(v)));
endproperty
a_clr : assert property (p_clear_on_fire);
 
// P11: SET BEATS CLEAR -- an event arriving as a transmission completes
// is not swallowed by it.
property p_set_wins;
  @(posedge clk) disable iff (!rst_n)
  (set_req && clr_req && (set_vec == clr_vec)) |=> pba[set_vec];
endproperty
a_setwins : assert property (p_set_wins);
 
// P12: SOFTWARE CANNOT CLEAR A PENDING BIT. The PBA is hardware-owned
// (section 5); letting software erase it would discard unserviced work.
property p_pba_hw_owned;
  @(posedge clk) disable iff (!rst_n)
  ($past(pba[v]) && !pba[v]) |-> !$past(sw_write_to_pba_region);
endproperty
a_owned : assert property (p_pba_hw_owned);
 
// P13: MSI-X and MSI are mutually exclusive; and neither coexists with
// INTx (Chapter 19.1 P14, Chapter 19.2 P13, from the third side).
property p_one_mechanism;
  @(posedge clk) disable iff (!rst_n)
  cfg.enable |-> (!dut_msi.cfg.enable && !dut_intx.msg.valid);
endproperty
a_excl : assert property (p_one_mechanism);
 
// P14: reset.
property p_reset;
  @(posedge clk) !rst_n |=> (!tx_valid && (pba == '0));
endproperty
a_reset : assert property (p_reset);

P1 and P2 are the table-safety pair, and both guard reads that would otherwise use data the device has no right to. P1 is §7's concurrency contract; P2 is the range check the index width does not provide.

P5 with P5b carry Chapter 19.2 §8's ordering requirement into MSI-X unchanged — and P5 is per-vector, which is the difference that keeps independent queues independent (§9).

And P12 is the property that keeps the PBA meaningful. A record software can erase is not a record of what hardware owes.

15. Verification, Fault Injection, and Model Verification

Executed before publication.

Vector index range — exhaustive

Over every table size 1 through 64, every index value representable in that size's index width:

Worked examples: table size 3 needs a 2-bit index, and index 3 is representable and invalid. Size 33 needs 6 bits, and indices 33–63 are all representable and invalid.

The conclusion the number supports: the index width does not bound the index. Only the runtime table size does, which is why §12 range-checks against cfg.table_size rather than relying on VEC_W.

Masking — exhaustive

Every combination of per-vector mask, Function Mask and pending state: 0 interrupts lost. A masked vector always sets its pending bit.

Capacity, against MSI

Directed tests

• Table sizes 1, 2, 3, 32, 33, 2048 — verify range refusal at size and above (P2). Required, and §15's measured hazard.
• Index equal to table_size — the boundary; verify refusal, not a read.
• Entry masked — verify no fetch for transmission and the PBA bit set (P1, P8). Required.
• Function Mask set with per-vector masks clear — verify nothing transmits (P9). Required.
• Software rewrites an entry while masked — verify no transmission uses a partial entry (§7). Required, and the mutation-1 test.
• Software rewrites an entry while unmasked — verify the device's behaviour is at least safe (captured whole, P3), while noting the convention was violated.
• data_pending_per_vec set for vector A only — verify vector B still transmits (§9's independence). Required.
• Transmit path stalls — verify hold and field stability (P3).
• Event during a transmission of the same vector — verify set beats clear (P11).
• link_ok low — verify no transmission and the PBA bit retained (P6).
• Reprogrammed address delivered after the event — §11's trace, cycles 9–10.
• MSI-X and MSI both enabled — verify exclusion (P13).

The scoreboard maintains its own table model and pending model, reading only (event, table contents, masks, transfers) — and never fetch_entry, pba, or the controller state.

Mutations

Same-cycle audit

16. Debugging

Symptom → table, masking, or ordering? → signal → distinguishing experiment.

Memory corruption that only happens during driver load or affinity changes

Suspect the table-read ordering (§15's counterexample) — this is mutation 1, and its clustering is the clue.

Reprogramming happens at driver load, on CPU-affinity changes, and on queue reconfiguration. Corruption clustered at exactly those moments, and absent during steady-state traffic, points at an entry read while software was rewriting it.

The distinguishing experiment: have the driver mask, rewrite, and unmask an entry repeatedly under load, and watch for writes to addresses that are neither the old nor the new one. A spliced address is unmistakable once you look for it — it matches neither.

And check the device's ordering directly: is the mask consulted before the entry is used (P1), or after?

Interrupts arrive on some vectors and never on others

Check the range first (§15). With a table size that is not a power of two, indices between size and the next power of two are representable and invalid — 23.9% of representable values across sizes 1–64.

The distinguishing experiment: compare the failing vector numbers against table_size. If every failing vector is ≥ table_size, it is a range bug (mutation 3) — the device is computing indices its own table does not have.

A second candidate: a device that assumed a power-of-two table and masked the index rather than range-checking it. That folds high vectors onto low ones, so the symptom is "vector 40's interrupts arrive as vector 8" rather than "vector 40 never arrives."

Software masks a vector and interrupts keep arriving

Check whether the Function Mask is what was set (§8, mutation 4). A device that checks only the per-vector bit ignores a global quiesce.

And check vec_mask handling versus MSI: in MSI-X per-vector masking is architectural, so unlike Chapter 19.2 §9 there is no capability bit to consult — a device that did not implement it is simply wrong.

Interrupts are lost when the driver reprograms or quiesces

Read the PBA (§5). If bits are set for the missing vectors, nothing was lost — they are pending, and unmasking will deliver them.

If the PBA is empty, the events were dropped — mutation 9 (masked and discarded) or mutation 10 (not recorded before a refused fetch). P7 and P8 are those checks made permanent.

And if the PBA clears without a corresponding transmission, it is mutation 8 or 11: cleared on tx_valid rather than the handshake, or cleared by a software write the device should not have honoured (P12).

The handler runs but the buffer is wrong

Identical to Chapter 19.2 §16, and the same three checks: is there an interlock (P5), is it on the same transmit path (P5b), and is Relaxed Ordering set (Chapter 11.6)?

With one MSI-X-specific addition. If the interlock is global rather than per-vector (mutation 5), the symptom is not corruption but latency — every queue's interrupt waiting on every other queue's data. Correct, and it throws away exactly the independence MSI-X was adopted for.

17. Common Misconceptions

• "MSI-X is just MSI with more vectors." The configuration moved into Memory Space, and everything follows from that (§2).
• "The MSI-X table is in Configuration Space." It is behind a BAR, in Memory Space (§1).
• "The table and PBA are in the same place." They have independent BAR indicators and offsets (§5).
• "Vector counts must be powers of two." That is MSI; MSI-X grants any count, 1 to 2048 (§4).
• "All vectors share one address." Each entry has its own address and data (§3).
• "The vector number is encoded in the data." That is MSI's workaround; MSI-X has no such encoding (§3).
• "The device interprets the entry." Address and data are opaque and transmitted verbatim (§6).
• "The index width bounds the vector." 23.9% of representable values are out of range (§15).
• "Masking only means don't interrupt me." It is also the mutual-exclusion primitive for reprogramming (§7).
• "Software can reprogram an entry safely without masking." It cannot; the device may read it mid-write (§7).
• "Per-vector masking is optional." It is optional in MSI; architectural in MSI-X (§4).
• "The Function Mask is redundant." It quiesces without walking a 2048-entry table (§8).
• "Software clears pending bits." The PBA is hardware-owned; software unmasks and the interrupt is delivered (§5).
• "MSI-X interrupts bypass ordering and credits." They are posted writes on the ordinary path (§9).
• "Per-queue vectors need a global ordering interlock." Per-vector, or the queues are serialized (§9).

18. Understanding Check

19. What's Next

MSI-X moves the interrupt configuration out of Configuration Space and into memory, and every difference follows.

What it buys (§4): 2048 vectors instead of 32; exactly the requested count instead of the next power of two; a private address and data per vector, which is what makes per-queue CPU affinity possible; and architectural per-vector masking.

What it costs (§7): the table is ordinary memory that software writes while hardware reads it, and an entry read mid-rewrite produces a write to an address that never existed. The Mask bit is the mutual-exclusion primitive that closes it — which is why masking in MSI-X is a safety mechanism, not only a convenience.

And everything MSI inherited from being a memory write, MSI-X inherits too (§9) — credits, ordering, LTSSM state — with the ordering interlock kept per-vector so independent queues stay independent.

Chapter 19.4 — Interrupt Routing takes the question all three chapters have deferred: how does the write actually reach the host? INTx assertions travel as messages through a hierarchy; MSI and MSI-X are memory writes that must be routed and decoded by something that turns them into interrupts. Chapter 19.5 then closes the module by making the performance argument quantitative.

The idea to carry forward: when two agents share a structure with no lock, one of them must own the primitive that keeps them apart — and the hardware's half is to never use what it was told not to.