Fault Management

Chapters 14.1 through 14.4 handled the fault: detected it, recovered from it, retried through it, degraded around it. In every one of those chapters the link ends up working again.

And in every one of them, something happened that the platform needs to know about — after the mechanism that handled it has already moved on.

1. The One-Sentence Model

Error handling restores service. Fault management preserves meaning.

A link can recover perfectly and still leave the system unable to answer the questions that matter: what failed, where, when, how many times, what was done about it, is the link still at full capability, and does software need to act?

Those are not questions the recovery controller asks. It is busy recovering, and by the time it succeeds the evidence has usually been overwritten by its own activity. Fault management is the discipline of making the answers survive.

2. What This Chapter Owns

The last row is the point. Fault management is not a fifth handling mechanism. It is the layer that turns four independent mechanisms' local state into one coherent account of what happened, and exposes it to software and to post-silicon engineers who were not present when it happened.

Specifically new here: the event / fault / incident distinction; first cause against latest cause across mechanisms rather than within one; the fault record and one-shot capture; per-lane counters read as a distribution; detection against isolation as two different bits; the escalation ladder with hysteresis; severity as a derived property rather than a detector attribute; host-visible status; why clearing a status register must not repair anything; interrupt coalescing without losing observability; and the incident timeline that correlates without claiming causation.

And it adds a fourth verification category. Modules 12–15 have used safety, liveness and performance. This chapter adds observability: can an engineer prove why the system behaved as it did? A design can be safe, live and fast and still be unmaintainable, because nothing that happened can be reconstructed.

3. Sourcing

4. Event, Fault, Incident

The distinction that structures everything else, and the one most designs never make explicit.

A worked example, and it is the whole section. Lane 2's margin degrades. What the system observes:

Six observations, one fault, one incident. The fault is lane 2's margin. The incident is the whole episode. Everything else is either a symptom or a response.

Three consequences.

A system that reports six faults has reported one fault five times too often, and has made the operator's job harder rather than easier. Counting events is correct and useful; presenting them as independent faults is not.

The mechanisms cannot make this distinction individually, because each sees only its own domain. The retry mechanism knows retries rose; it cannot know a lane was the cause. Only a layer that observes all of them can group them, which is why fault management is a separate layer rather than a feature of each.

And the grouping is a correlation, never a proof. §33 is explicit about this: a fault manager that presents a timeline is helping; one that asserts causation is guessing, and will occasionally be confidently wrong.

5. First Cause, Latest Cause, and What Software Sees

Within a single incident, the mechanisms fire in an order, and that order is the diagnosis.

Both are needed and they answer different questions, which is why §7's record carries both. But if a design can only afford one, it must keep the first, and the reason is asymmetric: the latest cause can usually be re-observed by looking at the system now, whereas the first cause is gone forever the moment it is overwritten.

Preserve the earliest actionable cause, then accumulate later evidence around it.

The failure this prevents, concretely. In §4's example, a design that keeps only the latest cause reports "width reduced" — or worse, "training timeout". Both are true. Neither mentions lane 2. A post-silicon team then investigates the retraining logic, which is working correctly, for as long as it takes someone to guess.

6. The Fault-Management Path

Three structural properties to read off it.

One collector, many detectors. Every detector reports through the same entry point, which is what makes first-cause ordering meaningful — two detectors writing two independent "first" records cannot be ordered against each other, and §33's timeline becomes unbuildable.

Isolation hangs off policy, not off detection. There is deliberately no edge from the detectors to the isolation mask. §15 and §16 are that missing edge, and it is the most consequential structural decision in the chapter.

And the software path is parallel to the policy path, not in series. Software is informed; it does not sit in the loop that decides whether to isolate. That matters for latency — a policy that waited for a software acknowledgement before acting would take milliseconds — and it matters for §26, because it means clearing a software-visible bit is touching a branch that policy does not read.

7. The Fault Record

// ILLUSTRATIVE internal fault record — NOT a UCIe register definition.
// UCIe 1.1 defines a per-Lane error log register with interrupt capability
// (Section 3); its LAYOUT is not published, and none is invented here.
typedef struct packed {
  logic                 valid;        // an incident is open
  logic [FAULT_W-1:0]   first_cause;  // WHAT started it — latched ONCE
  logic [FAULT_W-1:0]   latest_cause; // WHAT is happening now — updated always
  logic [LAYER_W-1:0]   layer;        // WHICH layer owned the first detector
  logic [LANE_W-1:0]    lane;         // WHICH lane, if lane-attributable
  logic [EPOCH_W-1:0]   epoch;        // WHICH link epoch (14.2 Section 23)
  logic [COUNT_W-1:0]   event_count;  // HOW MANY events in this incident
} fault_record_t;
 
fault_record_t fault_q;

Every field, and why it is there rather than derivable:

Architecture. One record per open incident, holding the minimum that makes an incident explicable to someone who was not there.

State. Mixed lifetimes in one structure, deliberately. first_cause, layer, lane and epoch are latch-once per incident. latest_cause and event_count are accumulate-per-incident. valid gates both. Mixing lifetimes in a struct is normally a smell; here it is the design, because they describe one object and must be read together.

Cycle behaviour. §8.

Contract. The policy engine (§19) reads latest_cause and event_count; the software path (§24) reads all of it; the post-silicon reader cares mostly about first_cause, layer, lane and epoch. Three consumers, one structure, and none of them may write it — a single writer is what keeps first-cause semantics intact.

Failure. §9.

DV. Inject three different causes in successive cycles and check first_cause holds the first while latest_cause tracks. Then check event_count reached 3. Then trigger a recovery mid-incident and confirm epoch still describes the configuration at the first event, not the current one.

8. First-Fault Capture

// ILLUSTRATIVE one-shot capture. The GUARD is the mechanism: !fault_q.valid.
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    fault_q <= '0;
  end else begin
    // --- LATCH ONCE: only on the first event of an incident.
    if (!fault_q.valid && fault_event) begin
      fault_q.valid       <= 1'b1;
      fault_q.first_cause <= fault_cause;
      fault_q.layer       <= fault_layer;
      fault_q.lane        <= fault_lane;
      fault_q.epoch       <= link_epoch_q;        // 14.2 Section 23
    end
 
    // --- ACCUMULATE: separately, and on every event including the first.
    if (fault_event) begin
      fault_q.latest_cause <= fault_cause;
      if (!(&fault_q.event_count))
        fault_q.event_count <= fault_q.event_count + COUNT_W'(1);
    end
 
    // --- CLEAR: an explicit, deliberate action only (Section 26).
    if (fault_record_clear) fault_q.valid <= 1'b0;
  end
end

Architecture. Two independent update paths in one block — a guarded latch and an unguarded accumulator — sharing one fault_event trigger. Separating them is the entire mechanism, and merging them is §9.

State. As §7. Note the clear path sets only valid: the other fields are left intact and are overwritten by the next incident's first event. That is deliberate — a cleared-but-not-yet-overwritten record is still readable by a debugger that arrives late, and clearing valid alone is cheaper than zeroing the struct.

Cycle behaviour. On the first event of an incident both blocks fire: the latch captures and the accumulator sets latest_cause to the same value and event_count to 1. That is correct — first and latest are legitimately equal at the start.

Contract. fault_event must be a single-cycle pulse per event. If a detector holds a level rather than pulsing, event_count counts cycles instead of events and every number downstream is wrong by a factor of the assertion length. That is a real integration hazard and worth an assertion at the collector boundary.

Failure. §9, and separately: putting the clear before the latch in priority so that a clear and an event in the same cycle lose the new event.

DV. Drive fault_event for a single cycle and for ten consecutive cycles from a badly behaved detector, and confirm the count differs by exactly what the pulse discipline predicts — which is how you find a level-versus-pulse mismatch before it corrupts a field return.

9. Wrong RTL — Last Error Wins

// WRONG — one cause register, overwritten by every event.
always_ff @(posedge clk)
  if (fault_event) fault_cause_q <= fault_cause;

The sequence, which is §4's example replayed against this register:

Software reads RETRAIN_TIMEOUT. That is the only surviving fact, and it is the least informative of the five.

Three properties make this the most expensive bug in the chapter.

It costs days, not hours, and it costs them to a different team. The post-silicon engineer sees a training timeout and investigates training — the equalisation, the pattern exchange, the deskew, the LTSM. All of it is working. The actual fault was physical, in a lane, four mechanisms and possibly minutes earlier.

It is invisible in simulation. Every functional test passes; the fault was recorded, and something plausible is in the register. No standard regression asserts that the recorded cause is the right one, which is why §12's assertion and §34's scoreboard check exist.

And it actively misdirects rather than merely under-informing. A missing record says "I don't know" and sends the investigation to first principles. A wrong record says "it was training" with full confidence, and confident wrong information is worse than none.

The last error is the least useful error, because it is the furthest downstream of the cause. A record that keeps only one cause must keep the first.

10. SVA — First Cause Is Sticky

// MANDATORY. The property that makes Section 9 impossible.
property p_first_cause_sticky;
  @(posedge clk) disable iff (!rst_n)
    (fault_q.valid && !fault_record_clear) |=> $stable(fault_q.first_cause);
endproperty
a_first_cause_sticky: assert property (p_first_cause_sticky);
 
// The companions: layer, lane and epoch are latched with it and stay with it.
property p_first_context_sticky;
  @(posedge clk) disable iff (!rst_n)
    (fault_q.valid && !fault_record_clear)
      |=> ($stable(fault_q.layer) && $stable(fault_q.lane) && $stable(fault_q.epoch));
endproperty
a_first_context_sticky: assert property (p_first_context_sticky);
 
// And the record is only ever opened by an actual event.
property p_valid_requires_event;
  @(posedge clk) disable iff (!rst_n)
    $rose(fault_q.valid) |-> $past(fault_event);
endproperty
a_valid_requires_event: assert property (p_valid_requires_event);
 
// The scoreboard-level claim: the captured cause is the one that ACTUALLY
// occurred first. Needs a reference, because the design cannot know it alone.
property p_first_cause_is_truly_first;
  @(posedge clk) disable iff (!rst_n)
    fault_q.valid |-> (fault_q.first_cause == tb_first_injected_cause);
endproperty
a_first_cause_is_truly_first: assert property (p_first_cause_is_truly_first);

Architecture. Three cheap local properties plus one reference-model property, and the split matters.

Why the fourth needs a reference. The first three prove the record is stable; they cannot prove it is correct. A design that latches the second event's cause and then holds it forever satisfies all three. Only a testbench that knows what it injected first can catch that, and it is a real bug — it arises when the collector's arbitration between simultaneous detectors is priority-encoded by severity rather than by arrival.

Contract. fault_record_clear must be the only thing that can disturb the record. Every other path — a recovery, a retrain, a local reset of a neighbouring block, a configuration commit — must leave it alone. That is a constraint on the whole design, not on this block.

DV. These need a multi-event incident. A single-event test satisfies stickiness vacuously, because there is never a second write to reject.

11. Saturating Fault Counters

// ILLUSTRATIVE. Diagnostic counters SATURATE. This is the rule Module 13 and
// 14.1 both established, and it matters most here because this is the counter
// software actually reads.
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n)                                  fault_count_q <= '0;
  else if (fault_event && !(&fault_count_q))   fault_count_q <= fault_count_q + 1'b1;
  else if (fault_count_clear)                  fault_count_q <= '0;

Why the &-reduction saturation test rather than a literal comparison. It is width-independent, so reparameterising COUNT_W cannot break it — a comparison against a hard-coded maximum silently stops saturating when the width changes.

The rule, stated once more because this chapter is where it bites hardest:

Saturate diagnostics. Never saturate accounting. A diagnostic clipped at its maximum reads "at least this many", which is fully actionable. A diagnostic that wraps reads as a small number during a storm — the single most misleading value a fault manager can present, because it says "healthy" at the exact moment the part is worst.

Contrast, so the rule is not applied blindly: 13.1's credit counter must never saturate, because saturation there invents permission. The test is whether the value is a report or a permission — a report may be clipped; a permission may not.

Failure. A wrapping counter, and its signature is in §37: an error count that decreases between two software reads with no clear in between.

DV. Drive 2**COUNT_W + 5 events and confirm the count clips. That is a cheap directed test and it is the only way to find this, because a regression short enough to be practical never reaches the maximum by accident.

12. Per-Lane Fault Counters

// ILLUSTRATIVE per-lane counters. UCIe 1.1 defines a per-LANE error log
// (Section 3) — its layout is not published, and this is an internal
// structure rather than a claimed register map.
logic [ERR_W-1:0] lane_error_count_q [NUM_LANES];
logic [LANE_W-1:0] first_failing_lane_q;
logic              first_lane_valid_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    for (int l = 0; l < NUM_LANES; l++) lane_error_count_q[l] <= '0;
    first_lane_valid_q <= 1'b0;
  end else begin
    for (int l = 0; l < NUM_LANES; l++)
      if (lane_error[l] && !(&lane_error_count_q[l]))
        lane_error_count_q[l] <= lane_error_count_q[l] + 1'b1;
 
    // Which lane went first — latch once, never overwrite (Section 5).
    if (!first_lane_valid_q && (lane_error != '0)) begin
      first_lane_valid_q   <= 1'b1;
      first_failing_lane_q <= LANE_W'(lowest_set_index(lane_error));
    end
  end
end

Architecture. A counter per lane plus a latch-once "which lane went first". The array is what makes §13's distribution readable; the latch is what makes a multi-lane failure attributable.

State. Counters are debug-epoch, saturating. first_failing_lane_q is latch-once per debug epoch — the same discipline as the fault record, applied to the physical resource.

Cycle behaviour. All lanes update in parallel; the first-lane latch uses a priority encode of the error vector. Note the tie-breaking case: if two lanes report in the same cycle, the priority encoder picks the lower index. That is arbitrary and it must be documented, because a debugger reading "lane 3 went first" deserves to know it might have been simultaneous with lane 7.

Contract. These feed §13's interpretation and the verified software path that "system software can then use this information to assess if link repair is needed" (§3). They are therefore a hardware-software interface, and their meaning must remain stable to a reader arriving long afterwards.

Failure. Clearing them on a recovery or a repair. A lane repaired three times shows a count of zero if the counters reset with each repair, and the single most valuable signal — this lane keeps failing — is destroyed.

DV. Inject on a known lane and confirm the count lands on that lane and no other. Inject on two lanes in the same cycle and confirm the documented tie-break. Repair a lane and confirm counts survive.

13. Reading the Distribution

Per-lane counters are useful as a shape, not as a maximum.

The right-hand column is the section. Every one of these readings is a hypothesis, and the fault manager's job is to make the hypothesis available, not to assert it.

Two concrete traps.

Acting on the maximum during a common-mode event. A supply droop raises every lane; a controller that masks the worst one loses capacity and does not fix anything (14.4 §25). The distribution, not the maximum, is what distinguishes the two — and only per-lane counters expose a distribution.

And "one lane hot" does not identify which lane to repair. A marginal lane can degrade a neighbour through crosstalk, so the lane reporting errors may be the victim. Repair count per lane over time is what disambiguates (§31): a lane that keeps failing after repair is probably not the fault.

14. Isolation Scope

A fault manager may remove something from service. What "something" is depends on the architecture:

One design rule that holds across all five: isolation must be reversible only by a deliberate act. A resource that removes itself from service and returns on its own has a policy nobody can reason about — and §16 is the version of that which happens by accident.

15. Unhealthy Is Not Isolated

Two bits, two meanings, and conflating them is §16.

Detection produces evidence. Policy produces decisions. A design in which a detector writes the isolation mask has no policy — it has a reflex.

And the asymmetry in the "clears when" row is deliberate. unhealthy should clear when the lane stops reporting errors, because it is an observation and observations change. isolated must not, because the reason a lane was isolated is usually that it intermittently fails — so the absence of errors while it is out of service is exactly what you would expect, and is not evidence it is fixed.

16. Wrong Design — Observation Becomes Isolation

// WRONG — the detector drives the isolation mask directly.
always_ff @(posedge clk)
  for (int l = 0; l < NUM_LANES; l++)
    if (lane_error[l]) lane_isolated_q[l] <= 1'b1;

Three failures, and they compound.

Transient events cost permanent capacity. At the verified UCIe 3.0 target BER of 10⁻¹² at 64 GT/s, a lane sees an error roughly every 15.6 seconds (13.5 §29). These are expected events that CRC and replay exist to absorb. Isolating on each one removes every lane within minutes, and — because isolation does not self-clear (§15) — none of them comes back.

There is no hysteresis and no persistence test, so a burst counts as many independent reasons to isolate (14.4 §8). One physical disturbance can isolate several lanes at once.

And it makes a common-mode event catastrophic. A supply droop raises errors on every lane simultaneously; this design isolates all of them. The link is now down, permanently, because of a transient that affected nothing structurally. A correct design would have observed a common-mode pattern (§13) and reduced rate instead.

The correct shape separates the two:

// Isolation is a POLICY output. It requires magnitude, persistence, and an
// explicit policy decision — never a bare detector event.
assign isolate_request[l] = lane_degraded[l]          // magnitude AND age (14.4 Sec 8)
                         && policy_permits_isolation  // the ladder allows it (Sec 19)
                         && spare_or_width_available; // there is a way to survive it

17. The Isolation Mask

// ILLUSTRATIVE isolation mask. Written ONLY by policy, and cleared ONLY by an
// explicit re-enable — never by a status clear (Section 26).
logic [NUM_LANES-1:0] lane_isolated_q;
logic [RPR_W-1:0]     lane_isolate_count_q [NUM_LANES];   // how many times, ever
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    lane_isolated_q <= '0;
    for (int l = 0; l < NUM_LANES; l++) lane_isolate_count_q[l] <= '0;
  end else begin
    for (int l = 0; l < NUM_LANES; l++) begin
      // SET: policy only.
      if (isolate_commit[l] && !lane_isolated_q[l]) begin
        lane_isolated_q[l] <= 1'b1;
        if (!(&lane_isolate_count_q[l]))
          lane_isolate_count_q[l] <= lane_isolate_count_q[l] + 1'b1;
      end
      // CLEAR: an explicit, deliberate re-enable ONLY. Note this is a
      // DIFFERENT signal from any status or interrupt clear (Section 26).
      else if (lane_reenable_commit[l]) begin
        lane_isolated_q[l] <= 1'b0;
      end
    end
  end
end

Architecture. One bit per lane plus a lifetime count of how often each has been isolated. The count is what makes §13's "one lane, repeatedly" pattern visible across incidents.

State. lane_isolated_q is life-of-service state — it survives recoveries, repairs, retrains and configuration commits, and is cleared only by a deliberate re-enable. lane_isolate_count_q is life of the part.

Cycle behaviour. Set and clear are mutually exclusive by the else if, with set winning. That priority is deliberate: if a re-enable and a fresh isolate request arrive together, the lane should stay out of service. Isolation is the safe direction.

Contract, and it is the whole point of the block. Two distinct commands exist — isolate_commit and lane_reenable_commit — and neither is driven by any status-register write (§26). The link-control path reads lane_isolated_q and excludes those lanes from the requested map (14.4 §11), which means isolation is an input to the map computation and not a substitute for it — the mask still goes through the atomic commit.

Failure. §27 — the clear path being wired to a status write.

DV. Isolate a lane, then perform every non-isolation operation the design supports — status clear, interrupt clear, recovery, retrain, repair, configuration commit, counter clear — and confirm the lane is still isolated after each. That matrix is small, fully directed, and it is what proves the separation is real rather than intended.

18. SVA — Isolation Changes Only on Policy Commands

// MANDATORY. Isolation is inert to everything except its two commands.
property p_isolation_stable_without_command;
  @(posedge clk) disable iff (!rst_n)
    (!isolate_commit && !lane_reenable_commit) |=> $stable(lane_isolated_q);
endproperty
a_isolation_stable_without_command: assert property (p_isolation_stable_without_command);
 
// The specific claim Section 27 violates, named so a failure is self-explaining.
property p_status_clear_does_not_reenable;
  @(posedge clk) disable iff (!rst_n)
    (status_clear || interrupt_clear) |=> $stable(lane_isolated_q);
endproperty
a_status_clear_does_not_reenable: assert property (p_status_clear_does_not_reenable);
 
// Isolation requires a degraded observation AND policy permission — never a
// bare detector event (Section 16).
property p_isolation_requires_policy;
  @(posedge clk) disable iff (!rst_n)
    $rose(lane_isolated_q[LANE_UT])
      |-> ($past(lane_degraded[LANE_UT]) && $past(policy_permits_isolation));
endproperty
a_isolation_requires_policy: assert property (p_isolation_requires_policy);
 
// And a recovery must not quietly bring an isolated lane back (14.2 Section 24).
property p_isolation_survives_recovery;
  @(posedge clk) disable iff (!rst_n)
    cfg_commit |=> (lane_isolated_q == $past(lane_isolated_q));
endproperty
a_isolation_survives_recovery: assert property (p_isolation_survives_recovery);

Architecture. Four properties pinning the mask from four directions: nothing else writes it, status clears specifically do not, setting it requires policy, and reconfiguration does not disturb it.

Why the second exists when the first subsumes it. Naming the specific violation makes a failure self-explaining to whoever triages it. A general stability property that fails says "something wrote the mask"; the named one says "the status-clear path wrote the mask", which is the difference between a bug hunt and a fix.

Why the fourth matters. 14.4 §17's atomic commit writes the active lane map. If isolation were re-derived at each commit from current health rather than held, a lane that has been quiet while isolated would look healthy and be silently re-enabled — which is §15's "clears when" asymmetry violated through a completely different path.

DV. All four need a lane actually isolated. A regression where isolation never occurs satisfies them vacuously.

19. The Escalation Ladder

// ILLUSTRATIVE severity ladder. NOT normative UCIe states — UCIe publishes no
// severity encoding or escalation policy (Section 3).
typedef enum logic [2:0] {
  FM_HEALTHY  = 3'd0,   // no open incident
  FM_WARN     = 3'd1,   // events recorded; service unaffected
  FM_DEGRADED = 3'd2,   // capability reduced; service continues
  FM_ISOLATED = 3'd3,   // a resource removed from service
  FM_FATAL    = 3'd4    // no viable configuration; new traffic blocked
} fault_state_t;
 
fault_state_t fm_q, fm_d;
 
always_comb begin
  fm_d = fm_q;                                   // default: hold
  unique case (fm_q)
    FM_HEALTHY : if (fault_q.valid)              fm_d = FM_WARN;
    FM_WARN    : if (no_viable_config)           fm_d = FM_FATAL;
                 else if (isolation_active)      fm_d = FM_ISOLATED;
                 else if (capability_reduced)    fm_d = FM_DEGRADED;
                 else if (clean_interval_met)    fm_d = FM_HEALTHY;   // Section 20
    FM_DEGRADED: if (no_viable_config)           fm_d = FM_FATAL;
                 else if (isolation_active)      fm_d = FM_ISOLATED;
                 else if (repair_verified && clean_interval_met)
                                                 fm_d = FM_HEALTHY;
    FM_ISOLATED: if (no_viable_config)           fm_d = FM_FATAL;
                 else if (reenabled && repair_verified && clean_interval_met)
                                                 fm_d = FM_DEGRADED;
    FM_FATAL   : fm_d = FM_FATAL;                // terminal until reset/service
    default    : fm_d = FM_FATAL;                // fail loudly, not silently
  endcase
end
 
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n) fm_q <= FM_HEALTHY;
  else        fm_q <= fm_d;

Architecture. A severity state derived from the record, the isolation mask and the link's capability — not from any detector's identity. §22 is why.

State. Incident lifetime, resting at FM_HEALTHY.

Cycle behaviour. unique case with a default to FM_FATAL — the same reasoning 14.2 §9 gave: a fault manager reaching an illegal encoding must not declare the system healthy. Note every escalating check is evaluated before every de-escalating one, so a state cannot improve and worsen in the same cycle with the improvement winning.

Contract. Downgrading requires three conditions together: a clean interval, a verified repair, and — from FM_ISOLATED — an explicit re-enable. Any one alone is insufficient, and §20 is why.

Failure. De-escalating on a clean interval alone, which flaps (§20). Or making FM_FATAL non-terminal, so a link with no viable configuration keeps announcing recovery.

DV. Walk every arc. Cover FM_ISOLATED → FM_DEGRADED, which needs a re-enable and a verified repair and a clean interval — the arc most likely to be wrong and least likely to be hit randomly.

20. Escalation Hysteresis

A severity state that tracks the instantaneous condition will flap, and flapping severity is worse than a stuck severity because it makes the reporting untrustworthy.

// ILLUSTRATIVE clean-interval requirement. Thresholds are design choices;
// UCIe publishes none (Section 3).
localparam int CLEAN_INTERVAL = 100_000;      // ILLUSTRATIVE cycles
 
logic [$clog2(CLEAN_INTERVAL+1)-1:0] clean_timer_q;
 
assign clean_interval_met = (clean_timer_q == CLEAN_INTERVAL);
 
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n)                          clean_timer_q <= '0;
  else if (fault_event)                clean_timer_q <= '0;        // any event restarts it
  else if (!clean_interval_met)        clean_timer_q <= clean_timer_q + 1'b1;

Architecture. One timer that measures time since the last event, reset by any event. De-escalation requires it to be full, so a single event anywhere in the interval defers the downgrade.

State. Incident lifetime.

Cycle behaviour. The reset-on-event arm is the mechanism. It means the interval measures a genuinely quiet period rather than an average.

Contract, and the three conditions are not interchangeable. A clean interval shows the symptom has stopped. A verified repair shows something was actually done — and matters because a lane can be quiet simply because it is isolated (§15). An explicit re-enable is the deliberate act. A design that de-escalates on quietness alone will announce recovery for a resource that is quiet because it is switched off.

Failure. Two, opposite. Too short an interval flaps FM_WARN ↔ FM_HEALTHY, generating an interrupt storm (§29) from a stable underlying condition. Too long leaves the system reporting degraded long after a genuine repair, which erodes trust in the reporting.

DV. Drive an event exactly at CLEAN_INTERVAL - 1 and confirm the timer restarts and no downgrade occurs. Then let it complete and confirm the downgrade requires the other two conditions as well.

21. Sticky Warning and Fatal Are Different States

// ILLUSTRATIVE. Two sticky bits with genuinely different consequences.
logic fault_warning_q;    // service continues; software should look
logic fault_fatal_q;      // new traffic blocked; escalation required
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    fault_warning_q <= 1'b0;
    fault_fatal_q   <= 1'b0;
  end else begin
    if (fm_q inside {FM_WARN, FM_DEGRADED, FM_ISOLATED}) fault_warning_q <= 1'b1;
    if (fm_q == FM_FATAL)                                fault_fatal_q   <= 1'b1;
 
    // Sticky bits are cleared by an explicit software acknowledgement, which
    // is NOT a repair (Section 26) and NOT a re-enable (Section 17).
    if (status_clear && !(fm_q inside {FM_FATAL})) fault_warning_q <= 1'b0;
  end
end
 
// New traffic is gated by the FATAL state, not by the warning.
assign new_traffic_permitted = !fault_fatal_q && link_operational;

Architecture. Two bits because two audiences act differently. Warning means look; fatal means stop. Collapsing them forces every consumer to guess which one a set bit meant.

State. Both sticky until acknowledged. Note the asymmetry in the clear: the warning bit can be acknowledged while the condition persists, but the fatal bit is not cleared by a status write at all — the condition it describes has not gone away, and letting software clear it would let software un-gate traffic on a link that has no viable configuration.

Cycle behaviour. The warning clear is guarded by !(fm_q inside {FM_FATAL}) so that acknowledging a warning during a fatal condition does not partially reset the picture.

Contract. fault_fatal_q gates admission. That is a real datapath consequence, and it means fatal must not be entered casually — which is why §19 reaches it only from no_viable_config rather than from a count threshold.

Failure. One combined fault_q bit. Software sees "a fault" and either stops for a warning that did not need it, or continues through a fatal condition because previous set-bits were benign.

DV. Assert new traffic is blocked in FM_FATAL and permitted in FM_WARN. Then confirm a status clear during fatal leaves both the fatal bit and the traffic gate untouched.

22. Severity Is Derived, Not Declared

A detector's name says what fired. It does not say how serious it is.

Five inputs to severity, and none is the detector's identity:

• recurrence — how often, in what window (14.2 §19's flap logic is this);
• recoverability — did the handling mechanism succeed?
• scope — one lane, all lanes, one protocol, the whole link (§14);
• data-loss risk — was anything delivered wrongly, or only delayed? 14.1 §19's gate means a detected corruption is not a data-loss event;
• configuration impact — did capability change, and in which direction?

A single CRC error on a link designed for a BER of 10⁻¹² is the mechanism working. Ten thousand CRC errors in a second on the same link is a different fault entirely — and the detector's name is identical in both cases.

The design consequence. A severity lookup table keyed by detector identity is wrong by construction. Severity must be computed from the record's event_count, the clean timer, the isolation mask and the capability state — which is exactly the input set §19's FSM reads.

23. Host and Software Visibility

What software needs, and what UCIe verifiably provides:

Three observations.

The verified surface is real and is more than most links have. A per-lane error log with interrupts, standard-format margin reporting, a software-triggered retrain, and a management transport that explicitly carries error reporting and log retrieval. The architecture assumes software participates, which is why §26's clear semantics matter so much.

The bottom three rows are the ones a design must provide itself. Nothing I could reach establishes how active width, error history or isolation state are exposed. They are still necessary — §37's taxonomy has a signature for each of them being absent — so they are an implementation obligation rather than a specification one.

And "manageability is optional" is a design input, not a footnote. A system that assumes the management transport exists will have no reporting path on a part that did not implement it. The fault record and counters must therefore be readable through whatever access the design does have, and that is a decision to make early.

24. Software-Visible Status

// ILLUSTRATIVE software-visible status. NOT a UCIe register map — no layout
// is published for the per-Lane error log or any status register (Section 3).
// This is the internal structure a design would MAP ONTO its revision's
// actual registers.
typedef struct packed {
  logic                  fatal;          // sticky — traffic blocked
  logic                  warning;        // sticky — look at this
  logic                  degraded;       // capability currently reduced
  logic [FAULT_W-1:0]    first_cause;    // from the record (Section 7)
  logic [FAULT_W-1:0]    latest_cause;
  logic [LANE_W-1:0]     first_lane;
  logic [NUM_LANES-1:0]  isolated_mask;  // what is OUT OF SERVICE right now
  logic [WIDTH_W-1:0]    active_width;   // what capability actually is
  logic [RATE_W-1:0]     active_rate;
  logic [COUNT_W-1:0]    event_count;
} fault_status_t;
 
// The status is a VIEW. It is assembled from the authoritative state and is
// never itself the authority — writing it must not change behaviour (Sec 26).
assign fault_status.fatal         = fault_fatal_q;
assign fault_status.warning       = fault_warning_q;
assign fault_status.degraded      = (fm_q == FM_DEGRADED) || (fm_q == FM_ISOLATED);
assign fault_status.first_cause   = fault_q.first_cause;
assign fault_status.latest_cause  = fault_q.latest_cause;
assign fault_status.first_lane    = first_failing_lane_q;
assign fault_status.isolated_mask = lane_isolated_q;
assign fault_status.active_width  = active_w;              // 14.4 Section 17
assign fault_status.active_rate   = active_cfg_q.rate;
assign fault_status.event_count   = fault_q.event_count;

Architecture. A view, assembled combinationally from state that lives elsewhere. The status register is not a copy that could drift; it is a window.

Why that matters more than it sounds. §37 lists a real signature — "link degraded but software reports full capability" — which is exactly what happens when the status is a registered snapshot updated by the fault path rather than a view of the live configuration. A design that latches active_width into the status when a fault occurs will report the width at fault time forever, which is wrong the moment a repair changes it.

Contract. Including active_width, active_rate and isolated_mask is what lets software answer "is this link still delivering what I think it is?" — and 15.1 shows why that question has a direct performance consequence. Reporting a fault without reporting the resulting capability is half a report.

Failure. A snapshot rather than a view (above). Or omitting isolated_mask, so software knows a fault occurred and cannot tell what is still out of service.

DV. Change width, rate and the isolation mask and confirm the status tracks each without a fault event occurring — proving the view is live rather than fault-triggered.

25. Clear Semantics — Acknowledge Is Not Repair

The single most consequential software-hardware contract in the chapter.

Acknowledging a report and repairing a fault are different operations with different consequences, and no design should let one imply the other.

Four things a clear must not do, each with a reason:

It must not repair hardware. The fault is physical or structural; a register write does not change it. A design where clearing appears to fix things has hidden the fault, not resolved it.

It must not re-enable an isolated resource. §27 is this, and it is the worst of the four.

It must not clear unresolved semantic obligations. 14.2 §12's census still applies — transactions in flight are owed responses regardless of what software acknowledges about a fault report.

And it must not erase forensic history before it has been collected. A field failure is often diagnosed from a dump taken after the system has been running degraded for hours. If the first status read cleared the record, the dump contains nothing. This argues for separating "acknowledge" from "clear counters" as distinct operations, so a monitoring agent polling status does not destroy the evidence a later dump would need.

26. Wrong RTL — Write-1-to-Clear Also Re-Enables the Resource

// WRONG — the status clear is wired to the isolation mask, "so the lane can
// be retried". Nobody writes this deliberately; it is written by convenience.
always_ff @(posedge clk)
  if (status_clear) begin
    fault_warning_q <= 1'b0;
    lane_isolated_q <= '0;          // ← the bug
  end

The loop this creates, and it is stable:

1. Lane 2 degrades persistently; policy isolates it. Capacity is reduced but the link is stable.
2. An interrupt fires. Software reads the status and clears it — the ordinary, correct thing for software to do.
3. Lane 2 silently returns to service.
4. Lane 2 is still bad. Errors resume immediately.
5. Retries rise, recovery runs, policy isolates lane 2 again.
6. An interrupt fires. Return to step 2.

Four properties make this an excellent bug.

The software is behaving correctly. Reading and clearing a status register is exactly what a driver is supposed to do. The bug is entirely in the hardware's coupling, and no amount of software review finds it.

The symptom is an interrupt storm with a healthy-looking link. Between iterations the link works. Error counters climb, the severity state flaps, and the underlying condition — one bad lane — never appears as a stable fact anywhere.

It gets worse under good monitoring. A system that polls status frequently re-enables the lane more often. The better the observability tooling, the more damage it does, which is a genuinely perverse failure mode.

And it is invisible if the testbench never clears status while a lane is isolated. That specific ordering — isolate, then clear — is the whole test, and it is §17's DV matrix.

27. Interrupt Coalescing

A one-interrupt-per-event design fails at exactly the moment it matters most. At a high fault rate, software spends all its time in the handler and never gets to the diagnosis.

// ILLUSTRATIVE coalescing. UCIe publishes no coalescing rule (Section 3);
// the window is a design choice.
localparam int IRQ_WINDOW = 10_000;          // ILLUSTRATIVE cycles
 
logic [$clog2(IRQ_WINDOW+1)-1:0] irq_timer_q;
logic                            irq_pending_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    irq_timer_q   <= '0;
    irq_pending_q <= 1'b0;
  end else begin
    // A new event arms an interrupt only if the window has expired.
    if (fault_event && (irq_timer_q == '0)) begin
      irq_pending_q <= 1'b1;
      irq_timer_q   <= IRQ_WINDOW;
    end else if (irq_timer_q != '0) begin
      irq_timer_q <= irq_timer_q - 1'b1;
    end
 
    if (interrupt_clear) irq_pending_q <= 1'b0;
  end
end
 
assign fault_interrupt = irq_pending_q;

Architecture. A rate limiter on the notification, and nothing else.

Contract, and this is §28's rule in the code: the window gates irq_pending_q and nothing else. fault_count_q, the per-lane counters and the fault record are all updated by fault_event directly, with no reference to the window. Suppressing notifications must never suppress observation.

Failure. Putting the window in front of the collector rather than in front of the interrupt — which throttles the counters too, and then the reported event count understates reality by an unknown factor that depends on the storm's shape.

DV. Drive a burst far denser than the window and confirm exactly one interrupt while the event count reflects every event.

28. Storm Suppression Must Not Suppress Observability

// The property that keeps Section 27 honest.
property p_counters_ignore_irq_window;
  @(posedge clk) disable iff (!rst_n)
    (fault_event && !(&fault_count_q)) |=> (fault_count_q == $past(fault_count_q) + 1);
endproperty
a_counters_ignore_irq_window: assert property (p_counters_ignore_irq_window);
 
// And the first-cause record is armed by the first event regardless of whether
// an interrupt was permitted.
property p_record_ignores_irq_window;
  @(posedge clk) disable iff (!rst_n)
    (fault_event && !fault_q.valid) |=> fault_q.valid;
endproperty
a_record_ignores_irq_window: assert property (p_record_ignores_irq_window);

Rate-limit the notification. Never rate-limit the record. An engineer reading a fault dump must be able to reconstruct what happened even though software was told about it only once.

Why this is worth an assertion rather than a comment. The coupling is easy to introduce during a refactor — someone moves the window check "up" one level to simplify the code, and the counters silently start under-reporting. Neither functional test nor code review reliably catches that; the assertion does, immediately.

29. Persistent History

One incident is a data point. The history is the diagnosis.

// ILLUSTRATIVE lifetime history. All saturating; none cleared by a status
// acknowledgement (Section 25).
logic [15:0] recovery_count_q;        // 14.2
logic [15:0] retry_exhaust_count_q;   // 14.3
logic [15:0] lane_repair_count_q;     // 14.4
logic [15:0] degraded_entry_count_q;  // times capability was reduced
logic [15:0] fatal_count_q;           // times FM_FATAL was entered
logic [15:0] incident_count_q;        // distinct incidents (Section 4)

Six counters, and the ratios between them are what a reader interprets:

The last two rows are why incident_count_q is separate from fault_count_q. Ten thousand events in one incident and ten thousand events across ten thousand incidents describe completely different parts, and a design with only an event counter cannot tell them apart.

Contract. None of these is cleared by a status acknowledgement. They are the part's service history, and they are what a returns-analysis engineer reads first.

30. Timestamping

If the platform has a timebase, capture when. If it does not, cycle counts are still useful — but their limits must be understood.

// ILLUSTRATIVE. Prefer a platform timestamp; fall back to a free-running
// cycle counter. Capture FIRST and LAST for each incident.
logic [TIME_W-1:0] incident_first_time_q;
logic [TIME_W-1:0] incident_last_time_q;
 
always_ff @(posedge clk) begin
  if (fault_event) begin
    if (!fault_q.valid) incident_first_time_q <= time_source;   // latch once
    incident_last_time_q <= time_source;                        // always
  end
end

Two cautions that matter more than the code.

A local cycle counter is not comparable across dies. Two chiplets counting their own cycles produce numbers that cannot be subtracted or ordered against each other — the identical problem 15.2 §18 develops for latency measurement, arriving here for fault correlation. An incident spanning two dies needs a common timebase, or the timeline is per-die and must be presented that way.

And a finite timestamp wraps. The duration last - first is correct across a wrap under modular arithmetic and wrong under a magnitude comparison — the same rule as 14.3 §25's sequence numbers and 15.2 §36's timestamps. An incident lasting longer than the wrap interval cannot be measured at all, which sets a floor on TIME_W given how long incidents realistically last.

31. The Incident Timeline

Correlation is the fault manager's most valuable output and its most dangerous temptation.

A typical timeline, assembled from the six counters and the record:

t0   lane 2 per-lane error count begins rising      ← 14.4 detection
t1   CRC error count begins rising                  ← 14.1 detection
t2   retry count rises                              ← 14.3 response
t3   retry exhaustion on one object                 ← 14.3 escalation
t4   recovery entered                               ← 14.2 response
t5   lane 2 masked; width 16 → 15                   ← 14.4 repair
t6   clean interval begins

What the fault manager may legitimately say: these six observations occurred in this order within one incident, and the first was a per-lane error on lane 2.

What it must not say: lane 2 caused the width reduction.

Three reasons the stronger claim is unsafe.

Ordering is not causation. The lane errors and the CRC errors may share a common cause — a supply event affecting both — with neither causing the other.

The attribution may be wrong at the source. 14.4 §25 established that a lane reporting errors can be a crosstalk victim of a marginal neighbour. The timeline faithfully reports what was observed, and what was observed may name the wrong lane.

And presenting a hypothesis as a conclusion stops the investigation. An engineer told "lane 2 caused this" checks lane 2, finds it marginal-but-passing, and is now confused. An engineer told "lane 2 reported first" checks lane 2 and its neighbours, which is the correct next step.

A fault manager's job is to make the timeline reconstructable. Interpretation belongs to the engineer, and the design should present evidence in a form that supports interpretation rather than replacing it.

32. The Fault Scoreboard

// Verification-only reference model. Not synthesisable.
class fault_scoreboard;
 
  typedef struct {
    int  first_cause;          // what the TESTBENCH injected first
    int  latest_cause;
    int  event_count;
    int  affected_lane;
    int  recovery_actions;
    bit  isolation_expected;
    bit  software_notified;
  } incident_t;
 
  incident_t incidents[$];
  int cur = -1;
  int prev_counts[int];        // per-counter monotonicity tracking
 
  // ---- Check 1: first cause is the one actually injected first.
  function void check_first_cause(int dut_first);
    if (cur < 0) return;
    if (dut_first != incidents[cur].first_cause)
      $error("FIRST CAUSE WRONG: dut %0d, actually %0d — record overwritten or mis-arbitrated",
             dut_first, incidents[cur].first_cause);
  endfunction
 
  // ---- Check 2: every counter is monotonic non-decreasing until an explicit
  //      clear. Catches a wrapping diagnostic (Section 11).
  function void check_monotonic(int id, int value, bit cleared);
    if (!cleared && prev_counts.exists(id) && (value < prev_counts[id]))
      $error("COUNTER WENT BACKWARDS: id %0d, %0d -> %0d — wrap or unexpected clear",
             id, prev_counts[id], value);
    prev_counts[id] = value;
  endfunction
 
  // ---- Check 3: escalation is legal — severity never improves without the
  //      three de-escalation conditions (Section 19).
  function void check_escalation(int prev_sev, int new_sev, bit clean, bit repaired, bit reenabled);
    if (new_sev < prev_sev && !(clean && repaired))
      $error("ILLEGAL DE-ESCALATION: %0d -> %0d without clean interval and verified repair",
             prev_sev, new_sev);
  endfunction
 
  // ---- Check 4: a status clear changed ONLY status. THE Section 26 check.
  function void check_clear_semantics(bit [NUM_LANES-1:0] iso_before,
                                      bit [NUM_LANES-1:0] iso_after,
                                      int cause_before, int cause_after);
    if (iso_before != iso_after)
      $error("CLEAR CHANGED ISOLATION: %0h -> %0h — status clear is coupled to policy",
             iso_before, iso_after);
    if (cause_before != cause_after)
      $error("CLEAR ERASED FIRST CAUSE: %0d -> %0d", cause_before, cause_after);
  endfunction
 
  // ---- Check 5: reported capability matches actual capability (Section 24).
  function void check_capability(int reported_width, int actual_width);
    if (reported_width != actual_width)
      $error("STATUS STALE: reports width %0d, actual %0d", reported_width, actual_width);
  endfunction
 
endclass

Architecture. Five checks over three concerns: record correctness (1), counter integrity (2), and contract correctness (3, 4, 5).

Check 1 is the one that requires the testbench to know ground truth, and it is the only way to catch §9. The design cannot self-check which cause was first.

Check 2 catches a wrapping counter with no knowledge of the width — a counter going backwards without a clear is wrong regardless of why, and this is far more robust than asserting a specific saturation value.

Check 4 is §26's test and it is a before-and-after comparison around a specific stimulus. It cannot be expressed as a per-cycle assertion because it is a claim about what a particular operation did not do. That is the general shape of a contract check: apply the operation, and verify the things it must not touch are untouched.

Check 5 catches the stale-status bug, which no assertion inside the fault path catches because the fault path is not involved — the width changed for reasons that had nothing to do with a fault event.

33. Coverage

covergroup cg_fault_mgmt @(posedge clk);
  option.per_instance = 1;
 
  // --- Severity states and ARCS, including the hard de-escalations.
  cp_severity : coverpoint fm_q;
  cp_arc : coverpoint {fm_prev_q, fm_q} {
    bins h_w    = {{FM_HEALTHY,  FM_WARN}};
    bins w_h    = {{FM_WARN,     FM_HEALTHY}};      // needs clean interval
    bins w_d    = {{FM_WARN,     FM_DEGRADED}};
    bins d_i    = {{FM_DEGRADED, FM_ISOLATED}};
    bins i_d    = {{FM_ISOLATED, FM_DEGRADED}};     // re-enable + repair + clean
    bins any_f  = {{FM_WARN, FM_FATAL}, {FM_DEGRADED, FM_FATAL}, {FM_ISOLATED, FM_FATAL}};
  }
 
  // --- Events per incident. The distinction of Section 4.
  cp_events_per_incident : coverpoint events_in_incident {
    bins single = {1};
    bins few    = {[2:9]};
    bins many   = {[10:999]};
    bins storm  = {[1000:$]};        // needs coalescing to be exercised
  }
 
  // --- Which detector was FIRST. All must occur first at least once.
  cp_first_cause : coverpoint fault_q.first_cause {
    bins crc    = {FC_CRC};
    bins parity = {FC_PARITY};
    bins seq    = {FC_SEQUENCE};
    bins lane   = {FC_LANE};
    bins retry  = {FC_RETRY_EXHAUST};
    bins recov  = {FC_RECOVERY};
  }
 
  // --- Lane distribution shape (Section 13).
  cp_lane_shape : coverpoint lane_error_shape {
    bins one_hot     = {SHAPE_ONE_LANE};
    bins all_lanes   = {SHAPE_COMMON_MODE};   // must NOT lead to isolation
    bins grouped     = {SHAPE_GROUPED};
  }
 
  // --- Isolation, and the operations that must NOT disturb it (Section 17).
  cp_isolated_any : coverpoint (lane_isolated_q != '0);
  cp_op_while_isolated : coverpoint op_while_isolated {
    bins status_clear  = {OP_STATUS_CLEAR};    // Section 26's test
    bins irq_clear     = {OP_IRQ_CLEAR};
    bins recovery      = {OP_RECOVERY};
    bins repair        = {OP_REPAIR};
    bins cfg_commit    = {OP_CFG_COMMIT};
    bins counter_clear = {OP_COUNTER_CLEAR};
  }
 
  // --- Clear behaviour.
  cp_clear_while_faulted : coverpoint status_clear_with_fault_still_active;
  cp_repeated_clear      : coverpoint repeated_clear_same_incident;
 
  // --- Counters.
  cp_counter_saturate : coverpoint fault_count_q { bins max = {'1}; }
 
  // --- Crosses that carry the information.
  x_iso_ops     : cross cp_isolated_any, cp_op_while_isolated;   // Section 26
  x_shape_iso   : cross cp_lane_shape, cp_isolated_any;          // common-mode must not isolate
  x_storm_irq   : cross cp_events_per_incident, cp_irq_count;
  x_firstcause_sev : cross cp_first_cause, cp_severity;
endcovergroup

Six bins whose value is being non-zero, each proving a mechanism ran:

x_iso_ops — every operation crossed with an isolated lane. This is §17's DV matrix as coverage, and it is the direct test for §26. A regression with zero hits here has never checked that a status clear leaves isolation alone.

cp_events_per_incident.storm. Without it, §27's coalescing and §28's observability properties have never been exercised under the condition they exist for.

cp_counter_saturate.max. Proves saturation rather than assuming it.

cp_arc.i_d. De-escalation out of isolation, which needs three conditions simultaneously.

cp_first_cause — every bin. Each detector must be first at least once, or the collector's arbitration between detectors is untested in that direction.

And one whose value should be zero: x_shape_iso with all_lanes crossed with isolation active. A common-mode pattern must not produce lane isolation (§13, §16), and a non-zero count there is a policy bug.

34. Flagship Trace — One Incident, Six Symptoms

Illustrative. Lane 2 degrades. CLEAN_INTERVAL and IRQ_WINDOW shortened for the table.

Eight readings, and most of them are about columns that do not change.

Cycle 1: the first event sets first cause, latest cause, count and severity together. First and latest are equal at the start, which is correct.

Cycles 3–6: latest_cause moves through CRC, RETRY, RECOVERY while first_cause stays at LANE. That is the chapter in four rows. §9's bug is the version where the first-cause column reads RECOVERY at cycle 6 — and the lane is never mentioned in the field report.

Cycles 1–8: seven events, one incident. event_count reaches 7 while incident_count stays at 1. Six of those events are symptoms of one fault (§4).

Cycle 9: isolation happens at cycle 9, not cycle 1. Eight cycles of evidence — magnitude and persistence — before a resource is removed. §16's bug is the version where the Isolated column becomes 0x0004 at cycle 1, and where a common-mode event would have isolated everything.

Cycle 10: the status reports width 15 immediately. The view is live (§24). A snapshot-based status would still report 16.

Cycle 13 is the whole of §26. Software clears status. The IRQ column goes to 0. Every other column is unchanged — first cause LANE, count 7, lane count 3, severity ISOLATED, mask 0x0004. §27's bug is the version where the Isolated column becomes 0x0000 here, and the entire sequence restarts at cycle 14.

Cycle 17: the clean interval is met and severity does not de-escalate. Because lane 2 has not been repaired and not been re-enabled — and it is quiet precisely because it is isolated. §20's three-condition rule, doing exactly what it is for.

And the whole incident is reconstructable afterwards from four surviving facts: first cause LANE, first failing lane 2, seven events, one incident, and a lane isolate count of 1. That is what fault management produces — not a repair, not a recovery, but an account.

35. Debug Taxonomy

36. Debug Checklist

1. What was the first cause? Not the latest — and if the answer is unavailable, that is itself the finding (§9).
2. What is the latest cause? It drives the next action but rarely identifies the fault.
3. Which layer owned the first detector? PHY, Adapter, storage or protocol (14.1 §35).
4. Which lane or link? And is the attribution trustworthy, or could it be an aggressor's victim (§13)?
5. How many events, and how many incidents? Different numbers, different meanings (§4).
6. What recovery or repair actions occurred? From the persistent history (§29).
7. Was capability reduced, and by how much? Active width and rate — and does the status agree with reality (§24)?
8. Is anything currently isolated? The mask, not the health bits (§15).
9. Did software clear only status, or did the clear touch policy? §26 — check the isolation mask across the clear.
10. Did the first cause survive? Compare against what the injection or the timeline says happened first.
11. Did the counters saturate or wrap? A count that decreased without a clear is a wrap (§11).
12. Were the symptoms one incident or several? The timeline (§31).
13. Does the reported width and rate match the actual configuration? A stale view is a real and common bug (§24).
14. Was the fault still present after the clear? If yes, the clear was correctly an acknowledgement and not a repair.
15. What should software do next — and does the report contain enough to decide? If not, the observability is incomplete regardless of how correct the handling was.

37. Common Misconceptions

"An error event and a fault are the same thing." An event is one detector firing once; a fault is the persistent condition underneath; an incident is the whole episode. One degrading lane produces thousands of events, several mechanisms' responses, and one fault (§4).

"The last error is the most useful error." It is the furthest downstream of the cause, and it is usually a response by another mechanism rather than a symptom of the fault. Keeping only the last cause is how a lane-quality problem gets reported as a training timeout (§9).

"Clearing a status register repairs the fault." It acknowledges a report. The fault is physical or structural and a register write does not change it — and a design where clearing appears to fix things has hidden the fault (§25).

"Every error should interrupt software." At a high fault rate that saturates software exactly when diagnosis matters most. Rate-limit the notification; never rate-limit the record (§27, §28).

"One CRC error means the lane should be isolated." At the verified target BER a lane sees an error roughly every 15.6 seconds at 64 GT/s, and CRC and replay exist to absorb them. Isolation requires magnitude, persistence and a policy decision (§16).

"A recovered link has no fault history." Recovery restores service; it does not make the fault not have happened. The history — recoveries, repairs, degradations, incidents — is what distinguishes a part that recovered once from a part that recovers constantly (§29).

"Diagnostic counters may wrap." A wrapped count reads as a small number during a storm, which is the most misleading value available. Saturate diagnostics; never saturate accounting (§11).

"Isolation and detection are the same state." Detection is an observation that changes as conditions change; isolation is a deliberate removal from service that must not clear on its own — not least because an isolated resource is quiet precisely because it is isolated (§15).

"Fatal means the physical link is down." Fatal means no viable configuration remains and new traffic must be blocked. The physical layer may be perfectly capable of moving bits (§21).

"Post-silicon debug can reconstruct the first cause from the final status." It cannot, and this is the assumption that costs the most days. The final status describes the last mechanism to react, and every mechanism between it and the fault has overwritten what it saw (§5, §9).

"Correlation in the timeline proves causation." Ordering is not causation, attribution may name a victim rather than an aggressor, and presenting a hypothesis as a conclusion stops the investigation at the wrong place (§31).

38. Understanding Check

39. Summary — and Module 14 Closed

Error handling restores service; fault management preserves meaning. The four preceding chapters each handle a fault and each leave local state behind. This chapter is the layer that turns that state into one explicable incident for people who were not there.

Event, fault, incident are three different things. One degrading lane produces thousands of events, several mechanisms' responses, and one fault. Counting events is useful; presenting them as independent faults is not, and only a layer that sees every mechanism can group them.

Keep the first cause, accumulate the rest around it. The latest cause can be re-observed; the first cannot. A record that keeps only the last reports "training timeout" for a lane-quality fault — and confident wrong information costs more than none.

Detection is an observation; isolation is a decision. They are two bits with different clear semantics: observation clears when the symptom stops, isolation clears only on a deliberate re-enable — not least because an isolated resource is quiet because it is isolated. A detector wired to the isolation mask has no policy, only a reflex, and a common-mode event will take the whole link down.

Severity is derived, never declared. The same detector fires in benign and serious cases; what separates them is recurrence, recoverability, scope, data-loss risk and configuration impact.

Acknowledging a report is not repairing a fault, and a clear must not repair, re-enable, discard obligations, or erase forensics before collection. The coupled version of this creates a stable interrupt storm driven by correct software behaviour — and gets worse the better the monitoring.

Rate-limit the notification; never rate-limit the record. And saturate diagnostics, never accounting — a wrapped error count reads as healthy at the worst possible moment.

And a timeline is evidence, not a verdict. Ordering is not causation, and attribution can name a victim rather than an aggressor. The design's job is to make the episode reconstructable; the interpretation belongs to the engineer.

Module 14 — Reliability and Error Handling, complete

Reliability tells us whether the link is trustworthy. Module 15 asks a different question: when the link is healthy, how much useful work can it actually carry, and where does the time go?

• 15.1 — Per-Lane / Per-Module Bandwidth — bit rate, lane count, and the difference between raw physical bandwidth and useful payload throughput.

Browse the full path on the UCIe tutorials index.