Management Over CXL.io

Chapter 7.2 ended on a sticky bit: hardware sets it, software clears it, and a same-cycle race decides which wins. That bit answers exactly one question — did something happen at least once? — and for a device in a production fleet that answer is almost never enough.

This chapter is about the gap between what hardware knows and what software can find out.

1. The Engineering Problem — The Device Knows, and Cannot Say

A device detects an internal error. It is the only component in the system that has the full picture: which unit, which cycle, what state, what preceded it. Microseconds later that context is gone — the pipeline has advanced, the buffers have turned over, and the only thing left is whatever the device deliberately wrote down.

Everything after that is reconstruction from a record.

So the design question in management is not "does the device detect errors". It is: what does the device write down, how much of it survives to software, and can software tell when something did not survive? Three failure shapes dominate, and they are opposites of each other:

The third is the dangerous one. A device that reports nothing is obviously broken. A device that reports a truncated record with no indication of truncation produces a confident, wrong root-cause analysis — and the fleet ships with the real bug still in it.

2. The One-Sentence Model

Management is evidence handling. Hardware witnesses an event once. Every structure on the reporting path — sticky bit, counter, queue, interrupt — keeps part of that event and discards the rest, and a management bug is nearly always evidence that was discarded by a structure that never said so.

Call it the evidence path. The design question at every stage is not "did this work" but "what did this stage throw away, and is the throwing-away itself visible?"

3. What This Chapter Owns

Deliberately not repeated: the MSI-X table, vector delivery and the PCIe interrupt mechanism itself, which the PCIe track owns. This chapter is about the logic behind the interrupt — what decided to raise it, and what was recorded before it went out.

4. The Evidence Ladder

Four structures, in increasing order of what they preserve and what they cost.

And what each one throws away:

That last row is the one engineers get wrong. An interrupt is not evidence. It is a doorbell. It says "look", and if the record it points at has already overflowed, the doorbell is worse than useless — it produces a handler that runs, finds a truncated log, and returns having learned nothing.

A real device uses all four, and the discipline is knowing which question each one can and cannot answer.

5. Teaching-model boundary

6. RTL 1 — A Queue Records What, Not Merely That

The step up from a sticky bit is a structure that keeps content and order. It also introduces the failure a sticky bit cannot have: it can fill up.

module event_fifo #(
  parameter int unsigned DEPTH = 8,
  parameter bit          SILENT_DROP = 1'b0   // 1 = overflow is not reported
) (
  input  logic        clk, rst_n,
  input  logic        ev_valid,
  input  logic [1:0]  ev_class,      // teaching values, no CXL encoding
  input  logic [2:0]  ev_source,
  input  logic        sw_pop,
  output logic [1:0]  head_class,
  output logic [2:0]  head_source,
  output logic [7:0]  head_seq,
  output logic        empty, full, accepted,
  output logic [4:0]  level_q, peak_q,
  output logic [15:0] n_pushed_q, n_dropped_q,
  output logic        overflow_flag,     // software-visible: evidence was lost
  output logic        overflow_err, underflow_err
);
  assign empty    = (level_q == 5'd0);
  assign full     = (level_q == DEPTH[4:0]);
  assign accepted = ev_valid && !full;
  assign do_push  = accepted;
  assign do_pop   = sw_pop && !empty;

Three decisions in those five lines are worth naming.

accepted is gated by !full. An event arriving at a full queue is not written. That is a loss, and it is correct — the alternative is corrupting eight good records to store a ninth.

overflow_flag is a separate output from overflow_err. The first is software-visible state: it survives, and software can read it. The second is a verification signal. The distinction matters because the whole point is that the loss must reach software, not just the testbench.

peak_q records the high-water mark. Current occupancy tells you nothing after a burst has drained. Peak tells you how close you came, which is the number that decides whether the depth is adequate.

Simulated with twelve events into a depth-eight queue:

=== EXP1: a burst larger than the queue ===
  12 events into depth 8 : level=8 peak=8 pushed=8 dropped=4
  every event was pushed or counted as dropped        : ok
  overflow reported to software: correct=1 | silent-drop variant=0
  the loss of evidence is itself recorded             : ok

Every one of those numbers is asserted, not merely printed: level is checked against 8, pushed against 8, dropped against 4, and the conservation pushed + dropped == 12 is checked separately so that a counter that drifts in both directions cannot hide.

The SILENT_DROP variant is instantiated side by side and differs in exactly one respect: overflow_flag stays at zero. It loses the same four events. The two designs are indistinguishable from the outside except in whether they admit it, and that is the entire lesson of this module.

Occupancy under simultaneous push and pop

A queue that is being filled and drained at once is the normal case, not an edge case, and it is where occupancy counters break:

      case ({do_push, do_pop})
        2'b10: level_q <= level_q + 5'd1;
        2'b01: level_q <= level_q - 5'd1;
        default: ;                          // both or neither: no change
      endcase

The natural-looking alternative is two independent if statements. Written with non-blocking assignments, the second one wins outright, so a cycle with both a push and a pop decrements. Occupancy then drifts downward under exactly the traffic pattern the queue exists to handle, and because the drift only appears under simultaneous access, a testbench that pushes a burst and then drains it will never see it.

Measured on the real RTL:

=== EXP9: a push and a pop in the same cycle ===
  4 cycles of push+pop together : level=3 pushed 11->15 head_seq 8->12
  occupancy held at 3 across simultaneous push and pop : ok
  arrival order preserved through the same-cycle traffic: ok

head_seq advancing by exactly four confirms the second half: the queue did not merely keep the right number of entries, it kept the right entries.

7. The Management Loop

Before the remaining modules, the shape of the whole exchange. Note which arrows carry evidence and which carry only a signal.

Two properties of that picture do the work.

The interrupt and the evidence travel separately. The doorbell is a signal; the record is read over CXL.io afterwards. This is why the interrupt carrying no information is acceptable — and why the record overflowing while the doorbell still rings is the failure that produces a confident wrong answer.

Software's acknowledge and software's clear are different actions. Acknowledging the interrupt says "I am handling it". Clearing the status says "I have consumed this evidence". Collapsing them into one is a common shortcut and it loses every event that arrives between the two.

8. RTL 2 — The First Error Is the One Worth Keeping

When one fault causes a hundred downstream errors, the hundredth is a symptom and the first is the cause. A capture register that keeps the most recent value throws away the only record that identifies the origin.

  // Capture only when nothing is held, unless the design is the broken shape.
  assign take = err_valid && (KEEP_LATEST || !captured_q);
 
  if (take) begin
    if (captured_q) overwritten_err <= 1'b1;     // destroying the first error
    cap_class_q  <= err_class;
    cap_source_q <= err_source;
    captured_q   <= 1'b1;
  end else if (sw_clear && !err_valid) begin
    captured_q <= 1'b0;
  end
 
  if (err_valid) n_err_q <= n_err_q + 8'd1;      // counted whether captured or not

The last line is the design's second half and it is easy to omit. The capture holds one error; the counter records how many arrived. Without the counter, software reading a single captured error cannot distinguish one isolated fault from a storm of three thousand — and those two situations call for opposite responses.

Note also that n_err_q increments on err_valid, not on take. Counting only what was captured would report exactly one error forever, which is the most confidently misleading telemetry a device can produce.

Three errors arriving in sequence, correct design against the KEEP_LATEST variant:

=== EXP3: the FIRST error is the one worth keeping ===
  three errors arrive : first-capture class=2 source=1 count=3
                        keep-latest  class=0 source=7
  the first error survived the two that followed      : ok
  the keep-latest variant holds the LAST error        : ok
  and the count still says three arrived              : ok

Both instances report count=3. They disagree only on which error they held — and the correct one holds class=2 source=1, the fatal error that started it, while the variant holds class=0 source=7, an informational event from an unrelated unit that happened to arrive last. A root-cause analysis built on the second one investigates the wrong component.

The sw_clear && !err_valid guard is the other subtlety. A clear that lands in the same cycle as a new error would discard the new one, so the clear is suppressed and the collision is recorded in cleared_while_pending_err. This is 7.2's hardware-versus-software race in a different costume, and it has the same resolution: hardware's new information outranks software's stale intent.

9. RTL 3 — One Interrupt Per Burst, Not One Per Event

An uncoalesced notifier converts an error storm into an interrupt storm. The host then spends its cycles entering and leaving the handler rather than draining the queue, which makes the storm last longer — a positive feedback loop that turns a recoverable fault into an unresponsive machine.

  // Fire when something is pending, notification is unmasked, and either the
  // hold-off has expired or we are not coalescing at all.
  assign fire = pending_q && !mask &&
                (NO_COALESCE || (timer_q >= HOLDOFF[7:0]));
  assign irq  = fire;
 
  // An event always becomes pending. Masking suppresses the NOTIFICATION,
  // never the record -- otherwise unmasking reveals nothing.
  if (ev_valid) begin
    if (MASK_LOSES && mask) event_lost_to_mask_err <= 1'b1;   // broken shape
    else                    pending_q <= 1'b1;
  end

Twenty events, coalesced against uncoalesced:

=== EXP4: one interrupt per burst, not one per event ===
  20 events : coalesced interrupts=3 | per-event interrupts=21
  coalescing cut 21 interrupts to 3                 : ok

Twenty-one, not twenty. The uncoalesced instance fires once per cycle that anything is pending, which includes the cycle after the last event arrived and before software acknowledged. That off-by-one is not a defect; it is the honest behaviour of a design with no hold-off, and it is worth noticing because it shows that "one interrupt per event" is not even a stable description of what an uncoalesced notifier does.

The comparison is asserted directly — the test requires the coalesced count to be strictly lower, so a hold-off that silently stopped working would fail rather than pass quietly.

Masking suppresses the doorbell, never the record

=== EXP5: masking hides the notification, not the event ===
  event while masked : pending=1 irq=0 | mask-loses pending=0
  the event is pending even though nothing fired      : ok
  after unmasking : irq raised, total=4

This is the invariant most often broken in a hurry. Software masks an interrupt for a legitimate reason — it is already in the handler, or it is reconfiguring. If masking discards the event, then unmasking reveals nothing and the event is gone forever, with no record that it ever existed. Under the correct design the event is recorded while masked and the interrupt fires on unmask, four cycles later in the transcript above.

Read n_events_q against n_irq_q at the right-hand edge: eight events, one interrupt. The ratio is the entire value of the module, and because both counters are exposed, software can compute it at runtime and tune the hold-off against a real workload rather than a guess.

Note that irq is high for exactly one cycle. fire resets timer_q, so the condition that produced the interrupt is destroyed by the interrupt itself. A design that left the timer at or above the threshold would assert irq every cycle until acknowledged, which is the interrupt storm rebuilt out of a different mistake.

10. RTL 4 — Severity Ordering, and What It Costs

A fatal event must not queue behind a hundred informational ones. Strict priority achieves that, and the same mechanism creates starvation.

    if (FLAT_ORDER) begin
      if      (has_info)    pick_info    = 1'b1;    // no severity notion at all
      else if (has_recover) pick_recover = 1'b1;
      else if (has_fatal)   pick_fatal   = 1'b1;
    end else begin
      if      (has_fatal)   pick_fatal   = 1'b1;
      else if (has_recover) pick_recover = 1'b1;
      else if (has_info)    pick_info    = 1'b1;
    end
 
    // A lower severity served while a higher one is waiting is an inversion.
    if (pick_info    && (has_fatal || has_recover)) inversion_err <= 1'b1;
    if (pick_recover && has_fatal)                  inversion_err <= 1'b1;

The inversion checker is written about the outputs, not restated from the priority equation. That distinction matters: a checker derived from the same expression the design uses is a tautology that passes on any implementation, correct or not. This one compares what was served against what was waiting, which is an independent statement of the requirement.

All three classes waiting at once:

=== EXP6: severity ordering, and what it costs ===
  all three waiting : fatal=1 recover=0 info=0 | flat picks info=1
  the fatal event was served first                    : ok
  fatal gone        : fatal=0 recover=1 info=0
  strict order fatal, recoverable, info               : ok
  30 cycles all-waiting : fatal=31 recover=1 info=1
  sustained fatal starved both lower classes          : ok (by design)

The last line is the honest half. Over thirty cycles with all three classes continuously present, the strict-priority arbiter served fatal 31 times, recoverable once and informational once — and the one service each of the lower classes received happened before the fatal source became continuous. This is not a bug. It is the specified behaviour of strict priority, and the test asserts it deliberately so that anyone changing the arbiter to a weighted or aging scheme sees this test fail and has to make the trade-off consciously.

Both properties are true simultaneously: strict priority is correct for severity and strict priority starves. A design review that only names the first half has not finished.

11. RTL 5 — One Source's Clear Must Not Touch Another

The sticky bank generalises 7.2's single W1C bit to one bit per source, which introduces a failure the single bit cannot have.

    for (k = 0; k < 4; k = k + 1) begin
      // Hardware set wins, per source. A clear of one source must not touch
      // another -- that is the CLEAR_ALL_ON_ANY bug.
      if (ev_set[k])                                    next_sticky[k] = 1'b1;
      else if (CLEAR_ALL_ON_ANY ? (|sw_clear) : sw_clear[k]) next_sticky[k] = 1'b0;
      else                                              next_sticky[k] = sticky_q[k];
    end

CLEAR_ALL_ON_ANY uses |sw_clear — any clear bit wipes every source. It is a plausible shortcut, it passes any test that clears one source at a time and re-reads only that source, and it destroys unrelated evidence in the field.

=== EXP7: one source's clear must not touch another ===
  sources 0 and 2 set : sticky=0101
  clear source 0 only : correct sticky=0100 | clear-all sticky=0000
  source 2 survived a clear aimed at source 0         : ok
  set+clear same cycle on source 1 : sticky=0110 events=3 visible=3
  the hardware set won the race                       : ok
  every event that occurred stayed visible            : ok

A counter bug found by this module's own invariant

The events versus visible pair on that last line exists to measure a real loss: an event whose bit was cleared in the same cycle it was set would be counted as occurred but never observable. On the first run it read events=3 visible=2 — a reported loss on a design that was in fact correct.

The fault was in the counter, not the bank:

      // WRONG: a non-blocking increment inside a loop evaluates the same
      // right-hand side on every iteration, so this counts at most one
      // per cycle however many sources fired.
      for (k = 0; k < 4; k = k + 1)
        if (ev_set[k] && next_sticky[k]) n_visible_q <= n_visible_q + 8'd1;

Four iterations, four assignments to the same variable, all computing n_visible_q + 1 from the same pre-edge value — so the last one wins and the counter advances by one no matter how many sources fired. The companion counter n_set_q was already correct because it accumulated into a combinational variable first and added once. The fix makes the two symmetrical:

    vis_this_cycle = 8'd0;
    for (k = 0; k < 4; k = k + 1)
      if (ev_set[k] && next_sticky[k]) vis_this_cycle = vis_this_cycle + 8'd1;
    ...
    n_visible_q <= n_visible_q + vis_this_cycle;      // one non-blocking add

Worth dwelling on, because the symptom pointed at the wrong module. The invariant said "evidence was lost"; the bank was fine and the instrument measuring the bank was broken. Telemetry that under-reports produces false alarms as readily as it hides real ones, and a counter is a piece of logic that needs verifying like any other.

12. RTL 6 — Three Numbers, Not One

Occurred, recorded, serviced. A device that reports one of them cannot tell software whether it is idle, saturated, or silently discarding.

      // Every event that occurred was either recorded or dropped.
      if (n_occurred_q != n_recorded_q + n_dropped_q) accounting_err <= 1'b1;
      // Software cannot consume more than was recorded.
      if (n_serviced_q > n_recorded_q) serviced_more_than_recorded_err <= 1'b1;

Sixty events under a service rate deliberately slower than the arrival rate:

=== EXP8: three numbers, over 60 events ===
  occurred=60 recorded=48 dropped=12 serviced=20 backlog=28
  conservation occurred == recorded + dropped : 60 == 60
  conservation held                                   : ok
  every count matched an independent oracle           : ok
  software serviced 20 of 48 recorded: backlog 28

Every one of those five numbers is checked twice: once against the conservation law, and once against a separate testbench oracle that counts arrivals, acceptances, drops and services in the testbench's own variables without using any of the DUT's logic. That independence is the point — a testbench that recomputes the expected value using the design's own helper proves only that the helper is self-consistent.

Read the numbers as a diagnosis. dropped=12 says the queue was overrun; backlog=28 says software is behind; serviced=20 against occurred=60 says software saw one third of reality. Any one of those numbers alone supports a wrong conclusion — serviced=20 alone looks like a quiet device.

13. Assertions

Icarus Verilog 13.0 is the simulator available in this environment and does not support concurrent SystemVerilog assertions, so every property below is implemented as synthesisable checker logic and verified in simulation. The assert property form is given as the equivalent statement of intent.

Two of these deserve comment.

"Loss is visible" is the property that separates the two queue variants. Both lose events; only one asserts overflow_flag. Without this property the SILENT_DROP design passes every functional test.

"Masking preserves record" is a next-cycle property, not a same-cycle one. Written as a same-cycle check it would sample pending_q before the edge that sets it and fail on the correct design — the state-versus-transition mistake that has produced a false failure in every batch of this track so far.

14. Mutation Testing

Thirteen mutations, each a plausible mistake rather than a random edit. A mutation is killed if the testbench fails; a mutation that survives means the testbench does not actually test that behaviour.

Twelve killed, one proven equivalent. The first run scored ten of thirteen, and the three survivors had three different causes — which is the useful part.

Simultaneous push and pop — missing stimulus. The mutation is a real bug, and the testbench filled the queue and then drained it, never doing both in one cycle. Adding EXP9 killed it. The lesson generalises: a testbench built from phases systematically misses the concurrency between phases.

Service-versus-record check disabled — vacuous checker. Nothing in the testbench ever drove serviced beyond recorded, so the check had never fired and deleting it changed nothing. A checker that has never fired is not verified; it is merely present. The fix needed a dedicated instance for illegal stimulus, because driving the abuse into the main instance would trip that instance's own checker and be scored as a failure:

  // An instance reserved for illegal stimulus. Driving "serviced" beyond
  // "recorded" into u_eg would trip u_eg's own checker and be scored a
  // failure, so the positive test of that checker needs its own DUT.
  event_accounting #(1'b0) u_ex(...);

  abuse instance flagged serviced>recorded (3>1)   : ok

Lost sticky events counted as visible — genuinely equivalent, and left in the report. The mutation changes ev_set[k] && next_sticky[k] to ev_set[k]. In the correct design these cannot differ: ev_set[k] forces next_sticky[k] = 1'b1 in the first branch of the priority chain, so the second term is always true when the first is. No stimulus can distinguish them.

The guard is not useless — it is what makes the counter still correct if the set/clear priority is ever inverted — but under the correct priority it is unreachable. Reporting this as a coverage hole would be dishonest, and deleting the mutation to make the score read thirteen of thirteen would be worse. Twelve of thirteen with a proof is a better result than thirteen of thirteen without one.

15. Verification Plan

The coverage model that matters here is not a line or toggle target. It is a cross of queue state against arrival: empty, partial, full, and one-below-full, each crossed with arrival, service, both and neither. The "both" column is the one that found the push/pop bug, and it is the column a phase-structured testbench does not have.

16. Debug Lab

// Event queue accept path.
assign accepted = ev_valid && !full;
if (ev_valid && full) n_dropped_q <= n_dropped_q + 16'd1;
// overflow_flag is never set

  12 events into depth 8 : pushed=8 dropped=4
  overflow reported to software: correct=1 | silent-drop variant=0

if (ev_valid && full) begin
  n_dropped_q   <= n_dropped_q + 16'd1;
  overflow_flag <= 1'b1;          // software-visible, sticky until cleared
end

if (do_push) level_q <= level_q + 5'd1;
if (do_pop)  level_q <= level_q - 5'd1;

  4 cycles of push+pop together : level=1 expected 3

case ({do_push, do_pop})
  2'b10: level_q <= level_q + 5'd1;
  2'b01: level_q <= level_q - 5'd1;
  default: ;                          // both or neither: no change
endcase

if (err_valid) begin
  cap_class_q  <= err_class;      // overwrite whatever was held
  cap_source_q <= err_source;
  captured_q   <= 1'b1;
end

  three errors arrive : first-capture class=2 source=1
                        keep-latest  class=0 source=7

assign take = err_valid && !captured_q;    // first wins, hold until cleared
if (take) begin ... end
if (err_valid) n_err_q <= n_err_q + 8'd1;  // but count them all

assign irq = pending_q && !mask;     // no hold-off at all

  20 events : coalesced interrupts=3 | per-event interrupts=21

assign fire = pending_q && !mask && (timer_q >= HOLDOFF[7:0]);
if (fire)             timer_q <= 8'd0;      // the interrupt destroys its own cause
else if (pending_q)   timer_q <= timer_q + 8'd1;

if (ev_valid && !mask) pending_q <= 1'b1;   // masked events are discarded

  event while masked : pending=1 irq=0 | mask-loses pending=0

if (ev_valid) pending_q <= 1'b1;                 // record unconditionally
assign fire = pending_q && !mask && (timer_q >= HOLDOFF[7:0]);   // mask gates delivery only

if (|sw_clear) sticky_q <= 4'd0;      // any clear wipes every source

  sources 0 and 2 set : sticky=0101
  clear source 0 only : correct sticky=0100 | clear-all sticky=0000

for (k = 0; k < 4; k = k + 1) begin
  if (ev_set[k])       next_sticky[k] = 1'b1;    // hardware set wins
  else if (sw_clear[k]) next_sticky[k] = 1'b0;   // per-source clear
  else                 next_sticky[k] = sticky_q[k];
end

for (k = 0; k < 4; k = k + 1)
  if (ev_set[k] && next_sticky[k]) n_visible_q <= n_visible_q + 8'd1;

  set+clear same cycle on source 1 : sticky=0110 events=3 visible=2
FAIL 3 events but 2 visible

vis_this_cycle = 8'd0;                                   // combinational accumulate
for (k = 0; k < 4; k = k + 1)
  if (ev_set[k] && next_sticky[k]) vis_this_cycle = vis_this_cycle + 8'd1;
...
n_visible_q <= n_visible_q + vis_this_cycle;             // one non-blocking add

17. Design Review

Questions worth asking of any management block, in the order that finds problems fastest.

1. For every place evidence can be discarded, which software-readable bit records the discard? If any answer is "none", the log can be silently truncated.
2. What does the capture register hold when N errors arrive? If the answer is "the last", root-cause analysis will name the wrong unit.
3. Does the mask gate delivery or recording? Only delivery is ever correct.
4. What is the interrupt count for a thousand-event storm? If it is a thousand, the design has a livelock under fault conditions.
5. Which class starves under sustained load, and is that acceptable? Strict priority always starves something; the question is whether the right thing starves.
6. Can software distinguish idle from saturated? It needs occurred, recorded and serviced — one number cannot answer it.
7. Does clearing one source touch another? Test with multiple bits set, always.
8. Has every checker in the testbench ever fired? A checker with no positive test is decoration.

18. How This Appears in Real Engineering

Bring-up. The first thing anyone wants from a new device is a way to see inside it. A device whose event queue depth is too shallow produces bring-up logs that are truncated exactly when the interesting thing happens, and the team spends weeks chasing a phantom because the record ends before the cause.

Silicon debug. The peak-occupancy counter is the number that decides the next revision's queue depth. Without it the discussion is opinion; with it, it is arithmetic.

Driver work. The masking invariant is where hardware and software teams most often disagree, because each has a defensible reading of "mask". Settling it in the specification — mask gates delivery, never recording — before either side writes code saves a class of bug that is very hard to reproduce.

Fleet operation. The occurred/recorded/serviced triple is what turns a device from a black box into something operable at scale. A fleet-wide dashboard of dropped and peak finds marginal units before they fail.

Post-silicon. First-error capture is what makes a returned unit diagnosable. If it holds the last error, the return is a coin flip.

19. Common Misconceptions

20. Interview Reasoning

21. Exercises

1. Analysis. A device reports occurred=1000, recorded=1000, serviced=40, dropped=0, peak=8, queue depth 8. Something in that set is inconsistent. Identify it and give the two most likely implementation faults.

2. Design. Extend the event queue so that a fatal event can displace an informational one when the queue is full, rather than being dropped. State the new invariant, and name which existing assertion must be weakened and why.

3. RTL task. Implement aging in the severity arbiter: any class waiting longer than AGE_LIMIT is promoted for one service. Give the new bound on lowest-class latency, and write the assertion that catches a promotion that never fires.

4. DV task. Write the coverage cross that would have caught the simultaneous push-and-pop bug on the first run, and explain why a cross of address against read/write — the natural register-test cross — cannot contain it.

5. Debug task. Field units report interrupts at a steady low rate with an empty event queue every time software looks. Give your investigation order, and name the single pair of counters that settles whether the interrupt logic or the queue is at fault.

6. Design review. A colleague proposes removing the drop counter because "overflow_flag already tells software it lost something". Give the strongest version of that argument, then name what it costs and the specific fleet decision it makes impossible.

22. Summary

Management is evidence handling, and every structure on the path discards something.

• The evidence ladder runs sticky bit, counter, queue, interrupt — and an interrupt carries no evidence at all. It is a doorbell pointing at a record.
• Dropping events at a full queue is correct. Dropping silently is the defect, because a truncated record that looks complete ends the investigation at the wrong place.
• Hold the first error, count all of them. The last error in a cascade names a symptom; the first names the cause.
• Masking gates delivery, never recording. The window where a driver masks is exactly the window where events are most likely.
• Coalescing is a correctness property under load, not an optimisation: twenty events produced twenty-one interrupts uncoalesced and three with a hold-off.
• Strict severity priority is correct and it starves — thirty-one fatal services against one each for the lower classes. Both halves are true and a review must name both.
• A per-source clear must not touch other sources, and single-bit register tests cannot see the bug.
• Three numbers, not one: occurred, recorded and serviced, checked against a conservation law and an independent oracle.
• Verification lesson: a counter is logic and can be wrong. The n_visible_q bug was in the instrument, not the thing measured — and a checker that has never fired is not verified, it is decoration.

Chapter 7.4 turns from one device's internal state to the system view: how software walks a hierarchy it has never seen, decides what each thing is, and bounds a traversal that could otherwise never end.