"MSI Is Just a Software Interrupt" — It Is a Posted Memory Write

The belief: an interrupt is a signal. The device raises it, the CPU takes it, and the mechanism is a notification channel that sits outside the data path.

MSI is a Memory Write TLP. It travels the same path as the payload, obeys the same ordering rules, and consumes the same posted credit. Once you know that, the data-before-interrupt race stops being a mystery and becomes an ordinary write-ordering question.

1. Why a Competent Engineer Believes It

Four reinforcing reasons, and none of them is careless.

The software experience is a signal. From the driver, an interrupt is a handler that fires. There is no packet visible anywhere in that experience — you register a handler and it gets called.

The name says interrupt. Every operating-systems course teaches interrupts as an asynchronous signal from a device, and for a pin-based interrupt that is exactly what it is. MSI inherits the vocabulary of a mechanism it replaced.

Legacy interrupts genuinely were signals. INTx is a level assertion on a physical conductor (19.1). An engineer who learned interrupts on that model is not remembering wrongly; they are remembering an earlier mechanism.

And the abstraction usually holds. In a system that sequences correctly, an interrupt does arrive after its data, and the myth predicts every observation. It fails only in the race, which is intermittent by construction.

2. The Locally True Kernel

From the host's perspective, MSI is an interrupt: a vector fires and a handler runs. For writing or reasoning about the handler, the signal model is correct.

That covers a great deal: which core takes it, vector allocation, affinity, handler latency, interrupt storms and coalescing policy (26.4 §12). A driver author reasoning about handler structure is right to think of it as an interrupt.

The scope boundary is the device side. The moment you ask when the interrupt arrives relative to anything else the device wrote, the signal model has no answer — because signals and writes are ordered by different things, and an MSI is a write.

3. The Hidden Assumption

The myth assumes the notification travels a channel with its own delivery semantics, independent of the data.

It does not. An MSI is a Memory Write TLP to an address the platform has configured to mean "interrupt" (19.2, 19.3). Consequences:

The last row is the whole chapter. The replacement model:

An MSI is a Posted Memory Write that lands on an address the platform interprets as an interrupt. Its arrival is ordered relative to other writes by exactly the rules that order any two writes, and the device — not the interrupt mechanism — is responsible for issuing it after the data it announces is visible.

"Visible", not "issued". §9 measured the difference at 46.3 percentage points of nothing.

4. The Root-Cause Tree

"Spurious interrupt" is the expensive wrong turn. It sends the investigation to interrupt configuration — vectors, masking, affinity — all of which are correct. The device raised exactly one interrupt for exactly one event; it simply raised it too early.

5. The Minimal Counterexample

Two writes. No concurrency beyond that, no load, no multiple queues.

device wants to report one completed operation:
 
  W1:  write the status record to host memory
  W2:  write the MSI message
 
if W1 and W2 are issued in the same cycle, and they take different
paths or different amounts of time:
 
  host is interrupted
  handler reads the status record
  the record still holds the PREVIOUS operation's result

If MSI were a signal, W2 would not be a write and there would be nothing to reorder. It is, so there is.

The disproof does not even require reordering. It requires only that two independently-issued writes can complete in either order — which is the default unless something makes it otherwise.

6. The Two Writes, in Sequence

Two readings.

Both messages leave the same actor through the same actor. There is no separate interrupt path in the diagram because there is none in the device — the MSI is a write like the one above it.

The only difference between the halves is where the accepted arrow sits. That single dependency is the fix, and §11 is the RTL of it.

7. The RTL the Myth Produces

The wrong RTL. Written by someone who believes the interrupt is a side channel.

// WRONG — the interrupt is raised from the same condition that produces the
// status write, because "the interrupt is a signal and the write is data".
module completion_report_wrong (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        op_complete,          // an operation finished
  input  logic [31:0] op_result,
  // status write port
  output logic        st_valid,
  output logic [31:0] st_data,
  input  logic        st_ready,
  // interrupt request port
  output logic        irq_valid,
  input  logic        irq_ready
);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      st_valid <= 1'b0; irq_valid <= 1'b0; st_data <= '0;
    end else begin
      if (op_complete) begin
        st_valid  <= 1'b1;
        st_data   <= op_result;
        irq_valid <= 1'b1;                  // <-- raised in the SAME cycle
      end
      if (st_valid  && st_ready)  st_valid  <= 1'b0;
      if (irq_valid && irq_ready) irq_valid <= 1'b0;
    end
  end
 
endmodule

Why it passes basic tests. In a testbench where both ports are always ready and the host model reads the status register synchronously, the ordering is never exercised. The handler always finds the right value, because nothing introduced a delay between the two writes.

Three defects.

The two writes are issued concurrently and race (§5). Nothing sequences them.

irq_valid clears on its own ready, independently of the status write. The interrupt's lifetime is unrelated to the record's.

And a second op_complete while st_valid is still high overwrites st_data — the previous result is lost and the interrupt count no longer matches the event count. That is a second, independent bug the same mental model produces, because a signal-shaped interrupt has no notion of a record it must stay consistent with.

8. The Failure Timeline

cycle    0   op_complete for operation A. st_data <= A, st_valid <= 1,
             irq_valid <= 1.  Both writes now offered.
cycle    1   irq_ready is high; the MSI write is accepted and leaves.
             st_ready is LOW — the posted-write path is congested.
cycle   14   the MSI write lands in the host's interrupt controller.
cycle   15   the CPU takes the interrupt and enters the handler.
cycle   16   the handler reads the status record in host memory.
             It contains operation Z — the PREVIOUS report.
             (first divergence observed here; it happened at cycle 1)
cycle   31   the status write is finally accepted and lands.
             Nobody is looking any more.
 
--- and the second, independent failure ---
 
cycle   40   op_complete for operation B while st_valid is still 1
cycle   40   st_data <= B.  Operation A's result is overwritten, unreported.
             irq_valid was already 1, so no second interrupt is raised.
             One interrupt, two events, one lost result.

The first divergence is cycle 1 — the MSI accepted before the status write.

Why the symptom points elsewhere. The handler ran, so the interrupt worked. The status record contained a valid value, so the register file works. The natural conclusion is "spurious interrupt", which sends the investigation to vector configuration and masking, all of which are correct.

And it is load-dependent. At cycle 1 the failure required st_ready to be low — congestion. On an idle bench both writes are accepted immediately and the bug does not exist.

9. Measured — When the Interrupt Is Issued

Three readings.

Row 1 is a coin flip, and that is the honest expectation. Two writes issued together with identical latency distributions arrive in either order with equal probability. There is no bias to exploit and no "usually fine".

Row 2 is the trap. Issuing the interrupt after the status write barely helps — 46.3% versus 48.8% — because issue is not visibility. The status write has left the engine and may still be behind the MSI in a queue, in a different path, or simply slower. A design that "fixed" the race by reordering two always_ff statements has fixed nothing measurable.

Row 3 is zero by construction. Once the interrupt is gated on the status write being accepted, the ordering is a property of the design rather than of the race.

One scope caution. This model deliberately says nothing about what the ordering rules guarantee for two writes on the same path — that is 13.4's subject and it is a real mechanism. What the model measures is the case a device must not depend on: two writes whose relative arrival the device has not constrained.

10. Same-Cycle Audit

11. The Corrected RTL

// The interrupt is a WRITE, and it is issued only after the write it
// announces has been accepted. The state machine is the ordering contract.
module completion_report_correct (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        op_complete,
  input  logic [31:0] op_result,
  output logic        op_ready,             // backpressure to the engine
  // status write port
  output logic        st_valid,
  output logic [31:0] st_data,
  input  logic        st_ready,
  input  logic        st_accepted,          // the write was ACCEPTED, not merely issued
  // interrupt request port
  output logic        irq_valid,
  input  logic        irq_ready,
  // observability
  output logic [31:0] c_events, c_interrupts,
  output logic        accounting_mismatch
);
 
  typedef enum logic [1:0] { S_IDLE, S_STATUS, S_IRQ } st_e;
  st_e state;
  logic [31:0] held_result;
 
  always_comb begin
    // A new event is accepted only when the previous report has finished.
    // The wrong RTL has no such gate, so a second event overwrites the
    // first's result (§8's second failure, §10 audit A).
    op_ready  = (state == S_IDLE);
    st_valid  = (state == S_STATUS);
    st_data   = held_result;
    // The interrupt is a function of STATE, not of the same condition that
    // produced the status write. It cannot be issued until the state machine
    // has observed st_accepted — which is §9 row 3.
    irq_valid = (state == S_IRQ);
    accounting_mismatch = (c_interrupts > c_events);
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      state <= S_IDLE; held_result <= '0;
      c_events <= '0; c_interrupts <= '0;
    end else begin
      unique case (state)
        S_IDLE:
          if (op_complete && op_ready) begin
            held_result <= op_result;       // SNAPSHOT — not re-read later
            state       <= S_STATUS;
            if (c_events != 32'hFFFF_FFFF) c_events <= c_events + 32'd1;
          end
        S_STATUS:
          // ACCEPTED, not merely st_valid && st_ready. Acceptance is the
          // point at which the record is guaranteed visible to a subsequent
          // reader; issue is not (§9 row 2 measured the difference at ~2pp).
          if (st_valid && st_ready && st_accepted) state <= S_IRQ;
        S_IRQ:
          if (irq_valid && irq_ready) begin
            state <= S_IDLE;
            if (c_interrupts != 32'hFFFF_FFFF) c_interrupts <= c_interrupts + 32'd1;
          end
        default: state <= S_IDLE;
      endcase
    end
  end
 
endmodule

The six lenses.

ARCHITECTURE. The module exists because two Posted writes have no ordering unless the device creates it (§3). The state machine is the ordering contract — there is no other mechanism available.

STATE. state (which phase of the report is outstanding), held_result (a snapshot of the event's result), and two diagnostic counters. held_result is owned by the report from S_IDLE exit to S_IDLE re-entry.

EVENT. op_complete && op_ready begins ownership; st_accepted advances to the interrupt phase; irq_ready ends it.

CONTRACT. The host relies on: when the handler runs, the record is visible and describes the event that caused the interrupt. That is exactly what the S_STATUS → S_IRQ transition guarantees and nothing weaker does.

FAILURE. Gating on st_ready alone instead of st_accepted reproduces §9 row 2 — a 46.3% early rate that looks like a fix and is not.

DV / DEBUG. §12's p1_irq_after_status_accepted is the direct check; p5_counts_agree catches the overwrite failure independently.

12. Assertions

// P1 — the interrupt is never issued before the status write was accepted.
// This is the ordering contract in one line. |-> with $past because
// acceptance is observed in a prior cycle and the state machine encodes it.
// Catches: §7's concurrent issue, and §9 rows 1 and 2.
property p1_irq_after_status_accepted;
  @(posedge clk) disable iff (!rst_n)
    irq_valid |-> $past(state == S_STATUS && st_valid && st_ready && st_accepted);
endproperty
a_p1: assert property (p1_irq_after_status_accepted);
 
// P2 — exactly one interrupt per event. The wrong RTL can raise one
// interrupt for two events (§8) and, with a different stall pattern, two
// for one. This property forbids the second; P5 catches the first.
property p2_one_irq_per_report;
  @(posedge clk) disable iff (!rst_n)
    (irq_valid && irq_ready) |=> !irq_valid;
endproperty
a_p2: assert property (p2_one_irq_per_report);
 
// P3 — the reported result is a SNAPSHOT: it does not change while the
// report is outstanding. Catches a design that drives st_data from the
// engine's live result register (25.6 §5's live-descriptor fault, in a
// new place).
property p3_result_stable_while_reporting;
  @(posedge clk) disable iff (!rst_n)
    (state != S_IDLE) |=> ((state == S_IDLE) || $stable(held_result));
endproperty
a_p3: assert property (p3_result_stable_while_reporting);
 
// P4 — no new event is accepted while a report is outstanding. Without it
// held_result is overwritten and an event is lost with no trace (§10 audit A).
property p4_no_event_during_report;
  @(posedge clk) disable iff (!rst_n)
    (state != S_IDLE) |-> !op_ready;
endproperty
a_p4: assert property (p4_no_event_during_report);
 
// P5 — interrupts never exceed events. An INDEPENDENT check on the same
// invariant P2 addresses, derived from counters rather than from the state
// machine — so a state-machine bug cannot satisfy both.
property p5_counts_agree;
  @(posedge clk) disable iff (!rst_n) !accounting_mismatch;
endproperty
a_p5: assert property (p5_counts_agree);
 
// P6 — a pending interrupt does not survive reset. §10 audit C: it would
// announce a record under a configuration that may have been reprogrammed.
property p6_reset_drops_pending_irq;
  @(posedge clk)
    !rst_n |=> !irq_valid;
endproperty
a_p6: assert property (p6_reset_drops_pending_irq);
 
// Vacuity guards. P1 is satisfied trivially by a testbench whose st_accepted
// is tied high in the same cycle as st_ready — which is the natural BFM to
// write if you believe issue equals visibility.
c1_status_stalled: cover property (@(posedge clk) disable iff (!rst_n)
                     st_valid && !st_ready);
c2_accept_lags:    cover property (@(posedge clk) disable iff (!rst_n)
                     st_valid && st_ready && !st_accepted);
c3_event_during:   cover property (@(posedge clk) disable iff (!rst_n)
                     op_complete && (state != S_IDLE));

c2_accept_lags is the cover that separates a real fix from §9 row 2. If acceptance always coincides with st_ready, then gating on acceptance and gating on ready are the same property, and P1 passes for a design that would still race in a real system.

13. If DV Believes the Same Myth

The testbench models an interrupt as an event, not a write — and once it does, the race is unrepresentable.

The interrupt monitor row is the structural failure. A monitor that fires on irq_valid collapses issue and arrival into one moment, which is precisely the collapse the myth makes. The race cannot be observed by an instrument that does not model it as a transaction.

What a correct testbench does differently. The interrupt is modelled as a write transaction with its own latency, delivered into the same host model that receives the status write — so the host observes them in whatever order they actually arrive. Then the scoreboard's check becomes: at the moment the handler runs, does the record describe the event? That question is unaskable under the signal model.

14. Debugging

On instrument choice. An analyzer is genuinely useful here, and it is the one myth in this module where the ordering is directly visible on the wire: the trace shows the MSI write ahead of the status write. That is a link-visible fact (25.9 §6), unlike 31.4's internal stalls. Set the trigger on the MSI address and read backwards.

15. Review and Interview

The review gate this myth corrupts. A verification review that accepts "interrupt generation is covered — we check every vector fires" has verified the wrong property.

The review question: "What event makes the interrupt eligible to be issued, and how do you know the status record is visible at that point?"

Two clauses, and the second is where designs fail. "After the status write" is not an answer until it says after the status write is accepted — and a reviewer should ask what "accepted" means in this design's fabric, because §9 measured issue-based gating buying almost nothing.

The interview exchange.

Weak answer: "MSI replaced the interrupt pin with a message, so the device signals the CPU without a dedicated wire."

Why it sounds plausible: it is historically accurate and describes the motivation correctly.

Interviewer follow-up: "Your handler runs and reads a stale status record. The interrupt fired exactly once for exactly one event. What happened?"

Where the weak model breaks: under the signal model there is nothing to reorder — the interrupt "arrived", so the data must have. The candidate has no mechanism to explain the observation.

Strong answer: "An MSI is a Posted Memory Write to a configured address. It's ordered relative to the status write by the same rules that order any two writes, so if the device issued them without a dependency, they can arrive in either order — and I'd expect roughly a coin flip. The fix is in the device: don't issue the interrupt until the status write has been accepted. Gating on 'issued' isn't enough, because issue isn't visibility."

Senior follow-up: "You gate on acceptance. What is 'accepted' in your fabric, and what would you check on an analyzer to confirm the ordering held?" — the answer names the point at which the write is guaranteed observable by a subsequent reader, and the analyzer check is the relative position of the two writes in the trace.

16. Misconceptions Inside the Misconception

"MSI-X is a different mechanism, so this doesn't apply." Why it sounds plausible: MSI-X has a table, per-vector masking and more vectors — genuinely more capable. What really happens: it is still a Memory Write to a configured address (19.3). More vectors does not change the transaction type, and the ordering question is identical. What it causes: a design that solved the race for MSI and reintroduced it when moving to MSI-X.

"A memory barrier in the driver fixes it." Why it sounds plausible: barriers are the standard tool for ordering, and this is an ordering problem. What really happens: the race is between two device-originated writes. A barrier orders the host's accesses (26.3 §16 reached the same conclusion). What it causes: a driver-side "fix" that changes nothing, followed by the conclusion that the bug is in the platform.

"Interrupt coalescing makes the race less likely." Why it sounds plausible: fewer interrupts means fewer opportunities to race. What really happens: coalescing changes how many interrupts are raised, not their ordering relative to the records they announce (26.4 §12). A coalesced interrupt announcing a batch has the same requirement for every record in the batch. What it causes: a reduced failure rate mistaken for a fix — the reproducible bug becoming an intermittent one.

"The interrupt tells the handler what happened." Why it sounds plausible: the vector distinguishes sources, so it carries some information. What really happens: the vector identifies which source; the record carries the result. The interrupt is a notification that something is readable. What it causes: handlers that act on the vector alone, which works until two events share a vector — and then the count and the work diverge.

17. Understanding Check

Q1. Your handler occasionally reads a stale status record. Event count and interrupt count match exactly. What does that agreement rule out, and what does it point at?

It rules out the counting failures and points at ordering (§14, P5 versus P1). If the counts agreed, the device raised exactly one interrupt per event — so nothing was lost, duplicated, or overwritten, which eliminates §8's second failure and the whole held_result class. What remains is that the interrupt arrived before the record it announces. §9 measured concurrent issue producing that at 48.8%. The confirming evidence is a sequence number in the record: if the handler repeatedly reads the previous event's number, the ordering is the fault, and the fix is in the device's sequencing, not in interrupt configuration.

Q2. A colleague moves the interrupt assignment below the status-write assignment in the same always_ff and reports the race fixed. Evaluate.

Statement order inside an always_ff does not sequence two Posted writes (§7, §9 row 2). Both assignments land on the same clock edge via NBA, so both writes are still issued in the same cycle — nothing changed at all. Even a genuine one-cycle delay would not fix it: §9 measured gating on the status write being issued at 46.3% against 48.8% for concurrent, because issue is not visibility. The dependency has to be on acceptance (P1), and c2_accept_lags is what proves a testbench can tell the two apart.

Q3. Why can an interrupt monitor that fires on irq_valid never observe this bug, no matter how much stimulus you run?

Because it collapses issue and arrival into one moment (§13). The bug is that the interrupt arrives before the status record becomes visible; a monitor sampling irq_valid observes the interrupt at the instant the device offers it, so the two events it would need to compare are not both represented. The instrument has the same blind spot as the mental model — which is the recurring shape across this module. The fix is to model the interrupt as a write transaction with latency, delivered into the same host model that receives the status write, so their arrival order is observable.

Q4. In §11, held_result is a snapshot. What breaks if st_data is driven from the engine's live result register instead?

A second operation completing while the report is outstanding changes the value being written (P3, §10 audit A). The status write is offered across several cycles under backpressure; if st_data follows the engine's live register, the value that eventually lands may describe a different operation from the one that triggered the report. The host then receives one interrupt describing an event whose result was overwritten — and neither the event counter nor the interrupt counter notices, because both incremented once. This is 25.6 §5's live-descriptor fault appearing in the reporting path.

Q5. State the replacement model and use it to explain why a driver-side memory barrier does not help.

"An MSI is a Posted Memory Write that lands on an address the platform interprets as an interrupt; its arrival is ordered relative to other writes by the rules that order any two writes, and the device is responsible for issuing it after the data it announces is visible" (§3). A memory barrier orders the host's accesses relative to each other. Both writes here originate at the device, so there is no host access sequence for a barrier to constrain — the ordering must be created where the writes are issued (§16). This is the same conclusion 26.3 §16 reached about an SSD's completion-queue entry and its interrupt.

18. What Comes Next

The first five chapters corrected beliefs about what things are — a fabric, a transaction type, an address window, a transfer, a notification. Each replacement model was a statement about identity.

31.6 corrects a belief about time. The link trains at boot and then, in the usual mental picture, stops being a thing that changes. It does not stop — and the consequence for every mechanism in this module is that the state they depend on has a lifetime nobody wrote down.