Error Recovery

Chapter 14.1 built the detectors and stopped deliberately at the gate: a corrupted object is detected, not delivered, and recorded. What happens next was named as belonging elsewhere.

This is elsewhere.

1. The One-Sentence Model

Recovery is controlled loss of assumptions. Normal operation runs on a set of assumptions about the link; an error means one or more of them may no longer hold; recovery is the disciplined process of working out which, rebuilding exactly those, and releasing traffic only when they hold again.

The word doing the work is controlled. An uncontrolled response — reset everything, clear every queue, start again — restores the assumptions perfectly and destroys the semantic state that had nothing wrong with it. The engineering is in restoring the minimum and preserving the maximum.

2. What This Chapter Owns

Recovery is touched by several earlier chapters. This one is the controller, and the boundaries are worth stating before the first line of RTL.

The specific new material here: the six assumptions normal operation makes, and which errors invalidate which; recovery against reset as a state-lifetime argument rather than a slogan; quiescing without revoking ownership; capture before retrain, because retraining destroys the evidence; the safe-action matrix for the five states an outstanding transaction can be in; timeout ambiguity, which is why blind reissue is unsafe; the recovery epoch and requested-versus-active configuration with an atomic commit on a live link; the resume gate as a conjunction rather than a PHY-ready bit; escalation and flap detection; and bounded recovery liveness with its assumptions written down.

3. Sourcing

4. Recovery Is Not Reset

The distinction that structures the whole chapter, and it is a state-lifetime argument rather than a matter of degree.

Recovery should not erase more state than the failure requires.

Three consequences, and the third is the one that produces bugs.

Erasing more is not "safer". It feels safer, because a cleared queue cannot contain a corrupted object. But a cleared queue also cannot contain the valid objects that were in it, and those had owners — Chapter 12.4 established that a transaction whose response never arrives is not a tidy outcome but an unbounded wait ending in a timeout at a layer that cannot explain it. Clearing converts a recoverable transport problem into an unrecoverable semantic one.

"Reset it and see" is a debug technique, not a design. In silicon bring-up it is often the right action; encoding it as the recovery policy means the product's response to a single bit error is to lose in-flight work.

And the boundary is not obvious from inside the recovery controller. The controller sees a trigger and a set of resources. Whether a particular register is "transport state that must be rebuilt" or "semantic state that must survive" is a question about its lifetime, and the only reliable way to answer it is to have written the lifetimes down. §6 is that table.

5. Six Assumptions Normal Operation Makes

Normal traffic runs because six things are assumed true. Recovery exists because an error means at least one may not be.

The table is the argument for minimal recovery (§16). A CRC failure invalidates assumption 5 and possibly 1 — it says nothing at all about the lane map or the rate. Rebuilding all six because one is suspect is the reset behaviour of §4 wearing a recovery label, and it costs the full retraining latency for a fault that a much narrower action would have cleared.

And notice what is not in the table. Semantic transaction state is not an assumption about the link; it is state the layers above own. Nothing in the list justifies touching it, which is §11's bug in one sentence.

6. The State-Lifetime Matrix

Read the lifetime column, not the treatment column. The treatment follows mechanically once the lifetime is known: link-epoch state is rebuilt, per-object and per-transaction state survives, diagnostic state survives everything. A design that cannot say which category a register is in cannot make a correct recovery decision about it, and that is the actual root cause of most of this chapter's bugs.

What is verified and what is not. That RECOVERY exists and that lane masking plus retrain is a real repair action is official (§3). Which state UCIe requires to be retained across a recovery is not something I could establish, so the treatment column is architecture — it is what a correctly layered implementation must do given the lifetimes, not a specification obligation.

7. Recovery Entry Causes

Split deliberately, because conflating these is how a chapter starts asserting a normative trigger list it cannot support.

Officially described conditions that lead to repair or retrain:

• A lane assessed as needing repair. UCIe 1.1: per-lane errors are logged and "system software can then use this information to assess if link repair is needed", with repair being "mask faulty lane and retrain".
• A software-triggered retrain. UCIe 1.1: "system software can trigger periodic retrain of the link to get eye margin information" — note this is a deliberate, healthy-link retrain, not an error response, and a controller must handle it without treating it as a fault.

Representative implementation triggers — plausible, common, and not asserted as UCIe requirements:

The right-hand column is the design's most valuable asset, because it is what makes minimal recovery possible: the trigger tells you which assumption is suspect, and therefore which rebuild is sufficient. A controller that maps every trigger onto one maximal response has discarded that information at the first opportunity.

8. The Recovery Controller

Read the figure for three orderings, not for the state names.

CAPTURE precedes RETRAIN. Diagnostics are latched before the action that destroys them. §13 and §14 are that ordering, and reversing it is the single most damaging thing a recovery controller can do to a post-silicon debug session.

VALIDATE precedes RESUME, and the commit happens in RESUME. The new configuration exists as requested through RECONFIG and VALIDATE, and becomes active only at the commit. §24 to §27 are that ordering.

And FAIL is reachable. A controller with no terminal state loops until something outside it intervenes, which in practice means a watchdog with no context. An explicit failure state that reports why is worth more than an implicit hang.

9. The Recovery FSM

// ILLUSTRATIVE recovery controller — NOT normative UCIe state naming.
// UCIe names RECOVERY as a link state (Chapter 8.6); it does not publish
// substates, and these eight are this chapter's decomposition (Section 3).
typedef enum logic [2:0] {
  REC_IDLE     = 3'd0,   // normal operation
  REC_QUIESCE  = 3'd1,   // stop admission, drain in-flight transfers
  REC_CAPTURE  = 3'd2,   // latch diagnostics BEFORE retraining destroys them
  REC_RETRAIN  = 3'd3,   // re-establish timing / framing (Module 8 owns how)
  REC_RECONFIG = 3'd4,   // stage a new configuration as REQUESTED
  REC_VALIDATE = 3'd5,   // prove the staged configuration works
  REC_RESUME   = 3'd6,   // commit atomically, then release traffic
  REC_FAIL     = 3'd7    // terminal: escalate with context intact
} rec_state_t;
 
rec_state_t rec_q, rec_d;
 
always_comb begin
  rec_d = rec_q;                                  // default: hold
  unique case (rec_q)
    REC_IDLE     : if (recovery_trigger)   rec_d = REC_QUIESCE;
    REC_QUIESCE  : if (quiesce_complete)   rec_d = REC_CAPTURE;
    REC_CAPTURE  : if (capture_done)       rec_d = REC_RETRAIN;
    REC_RETRAIN  : if (attempts_exhausted) rec_d = REC_FAIL;
                   else if (retrain_ok)    rec_d = REC_RECONFIG;
    REC_RECONFIG : if (cfg_staged)         rec_d = REC_VALIDATE;
    REC_VALIDATE : if (cfg_validated)      rec_d = REC_RESUME;
                   else if (validate_fail) rec_d = REC_RETRAIN;   // try again
    REC_RESUME   : if (cfg_commit_done && obligations_resolved)
                                           rec_d = REC_IDLE;
    REC_FAIL     : rec_d = REC_FAIL;                              // terminal
    default      : rec_d = REC_FAIL;   // unreachable: fail LOUDLY, not silently
  endcase
end
 
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n) rec_q <= REC_IDLE;
  else        rec_q <= rec_d;

Architecture. A phase sequencer whose ordering encodes three obligations: quiesce before touching anything, capture before retraining, validate before committing. The controller does not itself retrain, reconfigure or validate — it sequences the units that do, which keeps it small enough to reason about and keeps Module 8's machinery where it belongs.

State. One register, recovery-episode lifetime. REC_IDLE is the resting value. Note this is not the link state of 8.6 — it is the controller driving the transitions between them, and conflating the two produces a design where the link state machine and the recovery sequencer each believe they own the transition.

Cycle behaviour. unique case with an explicit default. Note the default goes to REC_FAIL rather than REC_IDLE — the opposite of 13.4 §9's congestion FSM, and deliberately so. A congestion policy reaching an illegal encoding should relax to the least restrictive state; a recovery controller reaching one should not silently declare the link healthy. The safe direction differs because the consequence of being wrong differs.

Contract. Four consumers depend on this state, and they must all read the same copy: the admission gate (§10), the capture logic (§13), the configuration commit (§26), and the resume gate (§28). obligations_resolved in the RESUME arm is the reliability handshake with 14.3 — §29 is what happens without it.

Failure. Three. Omitting REC_CAPTURE and capturing inside REC_RETRAIN loses the evidence (§14). Omitting the attempts_exhausted arm produces an infinite retrain loop with no report (§18). And driving rec_d = REC_IDLE from REC_VALIDATE on success, skipping REC_RESUME, removes the commit point entirely — which is §25.

DV. Cover every arc including REC_VALIDATE → REC_RETRAIN, which testbenches miss because it requires validation to fail after training succeeded. Assert the phase ordering, the commit property (§27), and the resume property (§28).

10. Quiesce — Stop Admission Without Revoking Ownership

The first phase, and the one with the sharpest correctness rule.

Quiescing has four jobs, and only the first is about stopping:

// ILLUSTRATIVE admission gate during recovery. The recovery term is a
// PREVENTION of new ownership transfer — it is NOT a revocation of ownership
// already transferred (Section 11, and Chapter 13.4 Section 23).
assign semantic_ready = normal_operation      // link state permits traffic
                     && !recovery_pending     // a recovery is starting or running
                     && resources_available;  // Ch 13.1 Section 14's conjunction

Architecture. One more term in an acceptance conjunction that already exists. That is the entire mechanism — recovery does not need a new path to stop traffic, only a new reason to refuse it, and reusing the existing gate means the refusal is automatically correct with respect to every other resource condition.

State. None here. recovery_pending is derived from the FSM and asserted from the moment a trigger is observed, before REC_QUIESCE is entered — because a trigger observed this cycle must not be followed by an acceptance next cycle.

Cycle behaviour. Combinational into the accept decision, for exactly 13.4 §23's reason: a registered gate is a gate that accepts one more object after it has decided not to.

Contract. The upstream layer's obligation is to hold what it cannot hand over. The gate's obligation is to be a prevention, never a revocation. After valid && ready, the object is the transport's responsibility and recovery does not change that — the object may be delayed by the whole recovery, but it may not be dropped.

Failure. §11.

DV. Assert that no acceptance occurs while recovery_pending; then, separately and more importantly, assert that the count of accepted-but-unresolved objects does not decrease across a recovery except through legitimate resolution (§12).

11. Wrong RTL — Recovery Clears the Queues

// WRONG — recovery entry clears accepted work. This is the reset behaviour of
// Section 4 wearing a recovery label.
always_ff @(posedge clk) begin
  if (recovery_enter) begin
    request_valid_q  <= 1'b0;     // accepted requests: gone
    outstanding_q    <= '0;       // tracking entries: gone
    replay_valid_q   <= '0;       // replay entries: gone
  end
end

Why each of the three lines is a distinct bug.

request_valid_q <= 1'b0 loses accepted work with no owner. The producer completed a handshake and released its copy. Nobody is retrying it. Nobody is waiting to be told. The transaction that generated it waits forever, and the only thing that eventually notices is 12.4's timeout, reporting at the wrong layer.

outstanding_q <= '0 destroys the ability to match a response that is still coming. The remote side may already have executed the operation. Its response is in flight right now, and it will arrive after recovery completes to a tracking table that has no record of the request. §34's trace is exactly this. The response is then either dropped silently or matched to a reused identity — a stale-response alias, 14.1 §6's last row.

replay_valid_q <= '0 destroys the recovery mechanism itself, which is the most self-defeating of the three. The replay buffer holds the objects whose delivery is unconfirmed — precisely the objects most likely to need re-sending because of the event that triggered recovery. Clearing it guarantees the loss of the work recovery exists to save (14.3 §14).

And the reason this bug is written is worth naming. It is written because a designer asks "what state might be corrupt?" and answers "anything the failure touched", which includes queues carrying objects that crossed the failing link. The answer is right and the action is wrong: the objects may indeed be untrustworthy, and the correct response is to resolve them through the reliability mechanism, not to forget them. Forgetting is not a resolution — it is the absence of one.

Recovery may re-establish what the failure invalidated. It may not discard obligations that predate the failure.

12. SVA — Accepted Work Survives Recovery

// The chapter's central safety claim. It is a SCOREBOARD-LEVEL property: no
// single RTL signal expresses "this object still exists somewhere", so this
// uses a verification-only census.
//
//   tb_live_objects = accepted objects that are neither delivered, nor
//                     architecturally failed with a reported outcome.
 
// Weak, local, always-on form: the tracking population cannot drop at the
// moment of recovery entry.
property p_no_bulk_loss_at_recovery_entry;
  @(posedge clk) disable iff (!rst_n)
    $rose(recovery_enter) |=> (outstanding_q >= $past(outstanding_q));
endproperty
a_no_bulk_loss_at_recovery_entry: assert property (p_no_bulk_loss_at_recovery_entry);
 
// Strong, scoreboard form: every object accepted before recovery is still
// represented by SOME live state until it is legitimately resolved.
property p_accepted_object_survives(int unsigned id);
  @(posedge clk) disable iff (!rst_n)
    (tb_accepted[id] && !tb_resolved[id])
      |-> (tb_live_somewhere[id]);
endproperty
 
generate for (genvar i = 0; i < MAX_TRACKED; i++) begin : g_survive
  a_accepted_object_survives: assert property (p_accepted_object_survives(i));
end endgenerate

Architecture. Two forms, because they fail at different times and both are useful. The weak one is a cheap always-on tripwire that fires on the exact cycle of §11's bug. The strong one is the real claim and needs a census the design cannot provide for itself.

Why the strong form must be verification-only. "Still represented by some live state" spans the request queue, the outstanding table, the replay buffer and the response path — four structures owned by different blocks. No design signal is the disjunction of all four, and building one purely to assert against would be a synthesised copy of the scoreboard, which then shares its bugs.

Contract. tb_resolved must include every legitimate ending: delivered, completed, or architecturally failed with a reported outcome. That last one matters — a recovery that ends in REC_FAIL may legitimately abandon objects, provided the abandonment is reported. Silent loss is the bug; reported failure is a defined outcome.

Failure it catches. All three lines of §11, and it catches them at the cycle of the clear rather than at the timeout hundreds of thousands of cycles later.

DV. The stimulus must have objects in every state of §21's matrix when recovery is triggered — queued, in flight, remotely accepted, response outstanding, complete. Triggering recovery on an idle link satisfies this property vacuously and proves nothing.

13. Capture Before Retrain

// ILLUSTRATIVE recovery-context capture. Latched ONCE, on entry to REC_CAPTURE,
// before retraining begins. Chapter 14.1 Section 34 built the first-error record;
// this captures the LINK CONTEXT that gives that record meaning.
typedef struct packed {
  logic                    valid;
  err_layer_e              first_layer;      // from 14.1
  err_kind_e               first_kind;       // from 14.1
  logic [SEQ_W-1:0]        first_seq;        // which object
  logic [NUM_LANES-1:0]    lane_mask_at_fail;// the map that was active
  logic [RATE_W-1:0]       rate_at_fail;     // the rate that was active
  logic [2:0]              link_state_at_fail;
  logic [TRY_W-1:0]        retry_count_at_fail;  // from 14.3
  logic [31:0]             cycle_at_fail;
} recovery_context_t;
 
recovery_context_t rec_ctx_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    rec_ctx_q.valid <= 1'b0;
  end else if ((rec_q == REC_CAPTURE) && !rec_ctx_q.valid) begin
    rec_ctx_q.valid               <= 1'b1;
    rec_ctx_q.first_layer         <= first_error_layer_q;   // 14.1
    rec_ctx_q.first_kind          <= first_error_cause_q;   // 14.1
    rec_ctx_q.first_seq           <= first_error_seq_q;
    rec_ctx_q.lane_mask_at_fail   <= active_lane_mask_q;    // 14.4
    rec_ctx_q.rate_at_fail        <= active_rate_q;
    rec_ctx_q.link_state_at_fail  <= link_state_q;          // 8.6
    rec_ctx_q.retry_count_at_fail <= retry_count_q;         // 14.3
    rec_ctx_q.cycle_at_fail       <= cycle_counter;
  end else if (recovery_context_clear) begin
    rec_ctx_q.valid <= 1'b0;        // explicit software/host action only
  end
end

Architecture. A snapshot of the link as it was when it failed, taken at a point where that information still exists. 14.1 §34 preserves the first error; this preserves the configuration in which it occurred, which is what turns "a CRC failed" into "a CRC failed on lane mask 0xFFFE at 32 GT/s after 3 retries."

State. Debug-epoch lifetime — cleared only by an explicit host action, never by a recovery, a retrain, or a local reset. The !rec_ctx_q.valid guard makes it latch-once, so a second recovery within the same debug epoch does not overwrite the first.

Cycle behaviour. One write, in REC_CAPTURE, before REC_RETRAIN is entered. The phase exists purely to create that ordering. It costs one state and a few cycles and it is the difference between a diagnosable failure and a mystery.

Contract. Every field's source must still hold valid data at capture time. That constrains the other blocks: active_lane_mask_q must not have been modified by the trigger handling, and retry_count_q must not have been cleared by the retry logic on entering recovery. Capture is a contract on the whole design, not a local feature.

Failure. §14.

DV. Trigger recovery from each error class and check every field. Then trigger a second recovery and confirm the record still describes the first. Then run a retrain and confirm the record survives it.

14. Wrong RTL — Diagnostics Cleared on Recovery Entry

// WRONG — the recovery controller "starts clean".
always_ff @(posedge clk)
  if (recovery_enter) begin
    first_error_valid_q <= 1'b0;      // 14.1's root cause: gone
    crc_error_count_q   <= '0;        // rate information: gone
    retry_count_q       <= '0;        // 14.3's escalation evidence: gone
  end

What the debug session looks like afterwards. A link fails in the field. Software reads the error state and finds: a training timeout. That is all. The report is accurate and useless.

What actually happened: lane 3's error rate rose, CRC failures began, the retry mechanism retried eleven times, retry exhaustion triggered recovery, and retraining then failed because lane 3 is genuinely bad. Six facts, of which the reported one is the last and least informative.

Three properties make this the most expensive bug in the chapter.

It is invisible in simulation. Every functional test passes. The recovery works. The link recovers or fails as designed. The only thing lost is information, and no assertion in a typical regression checks that information survived — which is why §13's DV step of "trigger a second recovery and confirm the record still describes the first" is worth writing explicitly.

Its cost is paid by someone else, much later. The RTL engineer who writes it never experiences the consequence. The post-silicon engineer who does has no way to know the information ever existed.

And it makes recovery look like the fault. Because the only surviving evidence is from the recovery attempt, the fault is attributed to the recovery machinery — the retraining, the controller, the configuration logic. The actual fault was physical, three mechanisms upstream, and it is now unattributable.

A recovery generates secondary symptoms by design. If the primary evidence is not captured before recovery acts, the only evidence remaining is secondary.

15. Re-establishing — What Module 8 Owns

REC_RETRAIN invokes machinery that already has three chapters, and this one deliberately does not repeat them.

What is not claimed. The exact sequence a UCIe recovery performs, and which of these steps it includes under which conditions, is not something I could establish from official material (§3). What is officially described is one concrete repair action — "mask faulty lane and retrain" — and a software-triggered retrain for margin measurement. The ordering in §9's FSM is this chapter's decomposition and is not asserted as UCIe's.

One genuine reuse worth noting. 8.3 §10–§12 already established candidate-versus-active and validation before commit for training. §24 to §27 apply the same discipline in a harder setting: 8.3 commits when nothing is flowing; a recovery commits on a link that has transport state in flight and obligations outstanding. The pattern is the same; the failure modes are not.

16. Minimal Recovery Against Full Recovery

The diagnostic row is underappreciated and is the strongest argument for trying minimal first. If resolving reliability state clears the condition, the fault was in assumption 5 and the lane map is fine. If a full retrain is performed instead, everything is refreshed, the link works, and nothing has been learned — so the next occurrence is equally mysterious.

The escalation ladder resolves the tension (§17): try minimal, and let repeated failure widen the response. That gets the latency and diagnostic benefits in the common case and the confidence benefit when it is actually needed.

Not a UCIe claim. No official source I could reach states that UCIe uses either strategy, or defines a graduated response. This is architecture.

17. The Escalation Ladder

// ILLUSTRATIVE escalation policy. The LADDER is architecture; the THRESHOLDS
// are design choices. UCIe publishes no escalation policy (Section 3).
typedef enum logic [2:0] {
  ESC_RESOLVE   = 3'd0,   // resolve reliability state only — cheapest
  ESC_RESYNC    = 3'd1,   // re-synchronise framing
  ESC_RETRAIN   = 3'd2,   // full retrain at the current configuration
  ESC_RECONFIG  = 3'd3,   // reduced configuration: mask a lane, or lower rate
  ESC_FAIL      = 3'd4    // terminal
} escalation_e;
 
escalation_e esc_level_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    esc_level_q <= ESC_RESOLVE;
  end else if (recovery_success && (rec_q == REC_RESUME)) begin
    esc_level_q <= ESC_RESOLVE;                 // success resets the ladder
  end else if (recovery_attempt_failed) begin
    if (esc_level_q != ESC_FAIL)
      esc_level_q <= escalation_e'(esc_level_q + 3'd1);   // widen the response
  end else if (flap_detected) begin
    // Section 19: repeated SUCCESSFUL recoveries are also a failure mode.
    if (esc_level_q < ESC_RECONFIG) esc_level_q <= ESC_RECONFIG;
  end
end

Architecture. A monotone ladder that widens on failure and resets on success. Each rung rebuilds strictly more than the one below it, so the response is always the narrowest that has not yet been shown insufficient.

State. Recovery-episode lifetime, and the two reset conditions are the design. recovery_success resets it — otherwise a link with occasional transient events ratchets permanently to the most conservative level. flap_detected jumps it — which is §19's point, and the reason a simple failure counter is not enough.

Cycle behaviour. One writer, three arms, priority ordered so that a success on the same cycle as a flap detection resets rather than escalates — a deliberate choice, since a completed recovery is evidence the current level works.

Contract. esc_level_q selects which steps REC_RETRAIN and REC_RECONFIG actually perform. It must be read at the start of an attempt and held for its duration, or a mid-attempt escalation produces a recovery that performed half of one policy and half of another — which is unreproducible and untestable.

Failure. Two, opposite. Never resetting on success ratchets to the most conservative level permanently, so a link that saw three transient events in its first hour runs in a reduced configuration forever. Resetting on every attempt — including failed ones — prevents escalation entirely, so a persistent fault is retried at the cheapest level indefinitely.

DV. Force failures at each rung and confirm single-step escalation; force a success at each rung and confirm the reset; and confirm that reaching ESC_FAIL is terminal rather than wrapping — an escalation_e'(ESC_FAIL + 1) would wrap to ESC_RESOLVE and produce an infinite ladder, which the guard prevents.

18. The Recovery Attempt Counter

// ILLUSTRATIVE saturating attempt counter. Diagnostic AND policy input.
// NO UCIe attempt limit is published, and RECOVERY_MAX_ATTEMPTS is a design
// choice (Section 3).
localparam int RECOVERY_MAX_ATTEMPTS = 4;      // ILLUSTRATIVE
 
logic [TRY_W-1:0] recovery_attempt_q;
logic [15:0]      recovery_total_q;            // lifetime count — sticky
 
assign attempts_exhausted = (recovery_attempt_q >= TRY_W'(RECOVERY_MAX_ATTEMPTS));
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    recovery_attempt_q <= '0;
    recovery_total_q   <= '0;
  end else begin
    // Per-episode: incremented on each attempt, cleared only on SUCCESS.
    if (rec_q == REC_RESUME && cfg_commit_done)      recovery_attempt_q <= '0;
    else if (retrain_start && !(&recovery_attempt_q)) recovery_attempt_q <= recovery_attempt_q + TRY_W'(1);
 
    // Lifetime: saturating, never cleared by a recovery (Section 6).
    if (retrain_start && !(&recovery_total_q)) recovery_total_q <= recovery_total_q + 16'd1;
  end
end
 
// The limit must be reachable, or REC_FAIL is dead code.
initial assert (RECOVERY_MAX_ATTEMPTS < (2**TRY_W))
  else $fatal(1, "RECOVERY_MAX_ATTEMPTS unreachable for TRY_W — REC_FAIL is unreachable");

Architecture. Two counters with different lifetimes for two different questions: "should this episode escalate or give up?" and "how often does this part recover?"

State. recovery_attempt_q is per recovery episode; recovery_total_q is sticky for the life of the part. Both saturate — they are policy and diagnostic counters, so 14.1 §33's rule applies: saturate diagnostics, never saturate accounting. A wrapping attempt counter would clear attempts_exhausted mid-episode and de-escalate a link that is failing, which is the worst available behaviour.

Cycle behaviour. The clear condition is REC_RESUME && cfg_commit_done — success, not merely leaving the retrain state. Clearing on entering REC_RECONFIG would reset the count on every attempt that got as far as training, so a link that trains successfully and fails validation forever would never exhaust its budget.

Contract. attempts_exhausted feeds the FSM's path to REC_FAIL, and the elaboration assertion guarantees that path is reachable. A limit larger than the counter can represent makes the terminal state unreachable, and the design then loops forever with no report — the failure mode the terminal state existed to prevent.

DV. Fail exactly RECOVERY_MAX_ATTEMPTS - 1 times then succeed, and check the per-episode counter clears while the lifetime counter retains the total. Then fail one more time and check REC_FAIL.

19. Error Storms and the Flap Detector

A subtle failure mode: recovery that keeps succeeding.

ACTIVE → RECOVERY → ACTIVE → RECOVERY → ACTIVE → …

Each recovery completes correctly. The attempt counter of §18 clears every time, so it never escalates. The link is nominally healthy and delivers almost nothing, because every recovery costs the full quiesce-retrain-validate-commit latency plus the restart cost of the entire pipeline (13.5 §8).

// ILLUSTRATIVE flap detector — recoveries within a sliding observation window.
// Thresholds are design choices; UCIe publishes none (Section 3).
localparam int FLAP_WINDOW  = 1_000_000;   // ILLUSTRATIVE observation cycles
localparam int FLAP_LIMIT   = 4;           // ILLUSTRATIVE recoveries in window
 
logic [$clog2(FLAP_WINDOW+1)-1:0] flap_timer_q;
logic [$clog2(FLAP_LIMIT+2)-1:0]  flap_count_q;
 
assign flap_detected = (flap_count_q >= FLAP_LIMIT);
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    flap_timer_q <= '0;
    flap_count_q <= '0;
  end else if (flap_timer_q == FLAP_WINDOW) begin
    flap_timer_q <= '0;
    flap_count_q <= '0;                       // window expires: forget
  end else begin
    flap_timer_q <= flap_timer_q + 1'b1;
    if (recovery_enter && !(&flap_count_q)) flap_count_q <= flap_count_q + 1'b1;
  end
end

Architecture. A rate detector rather than a count detector. The quantity that matters is recoveries per unit time, and neither a bare counter nor a bare timer expresses it.

State. Observation-window lifetime, deliberately forgetting on expiry. That forgetting is the feature: four recoveries spread over a day is a healthy link with occasional transients; four in a millisecond is a link that should be running in a more conservative configuration.

Cycle behaviour. A tumbling window, chosen over a sliding one because it costs two counters instead of a timestamp queue. The cost is a boundary effect — three recoveries just before an expiry and three just after do not trigger — and that is an acceptable trade for a policy input, though it should be known rather than discovered.

Contract. flap_detected jumps the escalation ladder to ESC_RECONFIG (§17). This is the only path by which repeated successful recoveries change behaviour, and without it a design has no response at all to the storm above.

Failure. Omitting it entirely, which is common: the design handles recovery failure and has no concept of recovery frequency. The signature is distinctive and is in §35 — healthy link state, near-zero throughput, high recovery count, zero recovery failures.

DV. Drive FLAP_LIMIT recoveries inside the window and check the escalation; drive them spread across two windows and check it does not fire; and confirm the boundary effect is understood rather than surprising.

20. Recovery and Outstanding Transactions

The deepest section in the chapter, and the reason §11's bug is worse than it looks.

A transaction in flight when recovery begins can be in one of five states, and they are genuinely different problems:

The three middle rows are the whole problem. Row 1 is easy — nothing has left, so nothing is at risk. Row 5 is easy — it is finished. Rows 2, 3 and 4 are states in which local knowledge and remote knowledge disagree, and recovery must act without being able to resolve the disagreement locally.

21. The Safe-Action Matrix

Row 3 deserves its own sentence. Possibly accepted is not a diagnosis a recovery controller can refine. Reissuing a semantic operation that may already have executed converts a transport uncertainty into a semantic corruption — a write applied twice, a coherence transition applied twice, a counter incremented twice. 14.3 exists precisely because the safe way through row 3 is to retransmit the transport object under an identity the receiver can recognise as a repeat, never to regenerate the semantic operation.

Recovery may re-drive transport. It may not re-execute semantics. The distinction is 14.3 §4's semantic-object-versus-attempt separation, and it is the reason a recovery controller must never reach into the layer above and ask it to "just send it again".

Not asserted as normative. No official source I could reach defines what UCIe requires of outstanding transactions across a recovery. This matrix is what the ownership rules of Module 12 and the reliability rules of 14.3 jointly require — sound engineering, not a specification clause.

22. Timeout Ambiguity

The formal reason row 3 cannot be resolved locally.

A timeout proves that no response arrived. It does not prove that the request was not executed.

Four outcomes are consistent with the same observation — a request sent, and no response received:

One observation, four causes, and three of the four make reissue unsafe. Chapter 12.2 established that a response is the only evidence of execution; 12.4 established that a timeout is a policy decision about waiting, not a fact about the remote side.

So the recovery-time rule is:

• Transport-level retransmission is safe when the receiver can identify a repeated attempt as the same object — that is 14.3's exactly-once machinery, and it exists to make rows 2 and 3 tractable.
• Semantic-level reissue is unsafe unless the operation is genuinely idempotent and the architecture says so. Idempotency is a property of the operation, not an assumption a recovery controller may make.

And the fourth row is the trap that makes this concrete. The response is in flight at the moment recovery begins. It will arrive after recovery completes. If the tracking entry was cleared, it arrives at a table that has no record of it — §34.

23. The Recovery Epoch

Everything the link was configured with before a recovery belongs to the old epoch. Anything committed during recovery belongs to the new one. The unsafe state is a mixture.

// ILLUSTRATIVE local epoch counter. THIS IS LOCAL CONTROL AND VERIFICATION
// STATE. No on-wire epoch field is claimed — UCIe publishes none (Section 3).
logic [EPOCH_W-1:0] link_epoch_q;
 
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n)             link_epoch_q <= '0;
  else if (cfg_commit)    link_epoch_q <= link_epoch_q + EPOCH_W'(1);

Architecture. One counter, incremented at the commit point and nowhere else. Its value is not used by the datapath — it is used to tag state so that assertions and the scoreboard can detect mixing, and to give a waveform a single signal that says "the configuration changed here."

State. Link-epoch lifetime, by construction. It wraps, and that is fine: it is used for inequality against a captured value, never for ordering across a long distance.

Cycle behaviour. Increments on cfg_commit (§26), so the epoch boundary and the configuration boundary are the same event by construction rather than by convention.

Contract. Any state tagged with an epoch must be checked against the current one before use. The mixing hazard is what this exists to catch: permission established under the old epoch applied under the new one is 13.1 §15's over-advertisement, and a transport object begun under the old lane map and finished under the new one is 14.4 §22's corruption.

Failure. Incrementing it at REC_QUIESCE instead of at the commit. The epoch then advances even when recovery fails and the old configuration is retained, so the tag no longer identifies a configuration and every check built on it is wrong.

DV. Check the epoch advances exactly once per successful commit and not at all on a recovery that ends in REC_FAIL with the previous configuration still active.

24. Requested Configuration Is Not Active Configuration

// ILLUSTRATIVE two-copy configuration. The same discipline Chapter 8.3
// Sections 10 to 12 applies to training, here on a link with live transport
// state — which is the harder case.
typedef struct packed {
  logic [NUM_LANES-1:0] lane_mask;
  logic [RATE_W-1:0]    rate;
  logic [WIDTH_W-1:0]   width;
  logic [CFG_VER_W-1:0] version;
} cfg_t;
 
cfg_t requested_cfg_q;   // staged during REC_RECONFIG; NOT used by the datapath
cfg_t active_cfg_q;      // what every datapath block actually uses
 
// Staging is free — it is a register write with no consumer.
always_ff @(posedge clk)
  if (rec_q == REC_RECONFIG && cfg_stage_en)
    requested_cfg_q <= cfg_proposal;
 
// Committing is NOT free. One event, one cycle, every consumer together.
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n)          active_cfg_q <= CFG_RESET_DEFAULT;
  else if (cfg_commit) active_cfg_q <= requested_cfg_q;

Architecture. Two copies of the same structure with completely different exposure. requested_cfg_q has no datapath consumers at all — it can be written, rewritten, abandoned, and none of that is observable in traffic. active_cfg_q is read by every block that stripes, reconstructs, times or frames, and changing it changes their behaviour on the next cycle.

State. requested_cfg_q has recovery-episode lifetime and may be discarded if validation fails. active_cfg_q has link-epoch lifetime and is the thing §23's epoch counts.

Cycle behaviour. The asymmetry is the design: staging is a plain enable, commit is a single global event. Note active_cfg_q is not reset by a recovery — a failed recovery leaves the previous configuration in place, which is correct, because a configuration that was working is a better fallback than a default.

Contract. No datapath block may read requested_cfg_q. That is the whole contract, it is not enforceable by the type system, and it is exactly what §25 violates. A useful lint is to make the staged copy structurally unreachable — a separate module boundary that exposes only the active copy.

Failure. §25, and its severity is the reason for the two-copy structure rather than a single register written when ready.

DV. Assert that active_cfg_q changes only on cfg_commit (§27). Then, separately, drive requested_cfg_q with a different configuration throughout a live transfer and assert that no transported bytes change — which proves no consumer is reading the staged copy.

25. Wrong RTL — Exposing the Requested Configuration Early

// WRONG — one configuration register, written as soon as a new value is known.
always_ff @(posedge clk)
  if (new_cfg_known) active_cfg_q <= cfg_proposal;   // takes effect immediately

Why this is deterministic corruption rather than a race. The transmitter begins striping across the new lane map, or at the new rate, from the next cycle. The receiver is still using the old one — it has not validated, has not agreed, and in general does not yet know a change was proposed.

Three properties worth naming.

It is not intermittent. Every byte after the change is misassembled, in the same way, every time. Deterministic corruption is easier to debug than a race and easier to ship than one, because it will be caught the first time the path is exercised — provided the path is exercised.

CRC will catch the corruption but will misattribute it. The receiver computes a CRC over misassembled bytes and it fails. The reported error is a CRC failure — a physical-looking symptom for a configuration bug. 14.1 §45's taxonomy names this signature explicitly: errors that begin exactly at a configuration change are a commit problem, not a channel problem.

And the fix is not "tell the receiver first". Ordering the two sides' updates does not close the window — it moves it. The fix is that neither side changes until both have validated and the change is applied at a defined boundary, which is §26.

26. Atomic Commit

// ILLUSTRATIVE atomic configuration commit. ONE event; every datapath consumer
// observes the change on the same clock edge.
//
// The commit is permitted only when everything that must be true, is.
assign cfg_commit = (rec_q == REC_RESUME)
                 && cfg_validated          // Section 24 staged value proved good
                 && peer_agreed            // both sides will switch
                 && no_transfer_in_flight  // no object spans the boundary (14.4 Sec 23)
                 && !cfg_commit_done_q;    // exactly once per episode
 
// Every consumer reads the SAME register. Not a copy, not a re-timed version.
wire [NUM_LANES-1:0] dp_lane_mask = active_cfg_q.lane_mask;
wire [RATE_W-1:0]    dp_rate      = active_cfg_q.rate;
wire [WIDTH_W-1:0]   dp_width     = active_cfg_q.width;

Architecture. A single commit event gated by a conjunction of four independently meaningful conditions, driving a single register that every consumer reads. The atomicity is structural, not procedural — it comes from there being one register, not from a careful sequence of writes.

State. cfg_commit_done_q is per recovery episode and prevents a double commit, which would advance the epoch twice for one configuration change and desynchronise every epoch-tagged check.

Cycle behaviour. One clock edge. Every consumer changes together because they all read active_cfg_q directly. The moment a consumer takes a registered copy "for timing", atomicity is gone — that copy changes a cycle later, and for one cycle two blocks disagree about the lane map, which is §25's corruption in miniature and much harder to find.

Contract, and each term earns its place. cfg_validated — the staged configuration was proven. peer_agreed — a unilateral change is §25 regardless of validation. no_transfer_in_flight — a transport object must not span two configurations, which 14.4 §23 develops into a property. !cfg_commit_done_q — exactly once.

Failure. Dropping any single term. Without peer_agreed it is §25. Without no_transfer_in_flight an object is striped half one way and half the other. Without cfg_validated the link commits to a configuration that may not work, and the fallback path — which needed the old configuration — has already overwritten it.

DV. Force a commit attempt with each term false in turn and confirm no commit occurs. Then confirm all consumers change on the same edge by sampling every derived signal at the commit cycle — a consumer with a registered copy shows up immediately as a one-cycle skew.

27. SVA — Active Configuration Changes Only on Commit

// MANDATORY. The property that makes Section 25 impossible.
property p_active_cfg_stable_without_commit;
  @(posedge clk) disable iff (!rst_n)
    !cfg_commit |=> $stable(active_cfg_q);
endproperty
a_active_cfg_stable_without_commit: assert property (p_active_cfg_stable_without_commit);
 
// And commit is only legal in the resume phase, after validation.
property p_commit_only_after_validate;
  @(posedge clk) disable iff (!rst_n)
    cfg_commit |-> ((rec_q == REC_RESUME) && cfg_validated && peer_agreed);
endproperty
a_commit_only_after_validate: assert property (p_commit_only_after_validate);
 
// Exactly one commit per recovery episode.
property p_single_commit_per_episode;
  @(posedge clk) disable iff (!rst_n)
    cfg_commit |=> (!cfg_commit throughout (rec_q != REC_IDLE)[->1]);
endproperty
a_single_commit_per_episode: assert property (p_single_commit_per_episode);
 
// The epoch advances with the configuration, and only with it.
property p_epoch_tracks_commit;
  @(posedge clk) disable iff (!rst_n)
    (link_epoch_q != $past(link_epoch_q)) |-> $past(cfg_commit);
endproperty
a_epoch_tracks_commit: assert property (p_epoch_tracks_commit);

Architecture. Four properties pinning the commit from four directions: it is the only way the active configuration changes, it is legal only after validation, it happens once, and the epoch is bound to it.

Why the first is the highest-value assertion in the chapter. It is trivially cheap, always enabled, and it makes an entire class of deterministic corruption structurally impossible. Any path that writes active_cfg_q outside the commit — a debug backdoor, a "quick fix" for a training corner, a reset that half-applies — fires it on the cycle it happens.

Contract. These constrain the design in isolation and need no reference model. The fourth is the one that keeps the scoreboard's epoch tagging trustworthy: if the epoch could advance without a commit, every epoch-based check downstream would be checking a fiction.

DV. They need a recovery that actually changes the configuration, not merely one that retrains at the same settings. A regression whose recoveries all recommit identical values satisfies all four while never exercising a change — cover the changed-configuration case explicitly (§32).

28. The Resume Gate

// ILLUSTRATIVE resume gate. FOUR independent conditions. The temptation is to
// reduce this to phy_ready, and that is Section 29.
assign traffic_may_resume =
       (rec_q == REC_IDLE)          // the controller says recovery is over
    && cfg_commit_done_q            // the configuration is committed (Sec 26)
    && phy_operational              // Module 8: the physical link works
    && reliability_resolved         // 14.3: no unresolved replay obligations
    && flow_control_established;    // 13.1 Section 15: permission re-established
 
assign semantic_ready = traffic_may_resume && resources_available;

Architecture. A conjunction of four conditions owned by four different mechanisms, and the chapter's argument is that all four are necessary and none implies another.

State. None here; each term is derived from its owner.

Cycle behaviour. Combinational, feeding the same acceptance gate as §10 — recovery adds and removes a term rather than building a parallel path.

Contract, term by term, and why each cannot be dropped:

• phy_operational — necessary, and the one everyone remembers. Bits can move.
• cfg_commit_done_q — the two sides agree on how bits are interpreted. A PHY can be perfectly operational under two different lane maps (§25).
• reliability_resolved — outstanding replay obligations from before the recovery are settled. §29 is what happens without this.
• flow_control_established — permission for the new epoch exists. Without it the first object is sent against permission that describes a link that no longer exists (13.1 §15).

PHY ready means bits can move. It does not mean they should.

Failure. §29.

DV. Force each term false in turn with the other three true, and confirm no traffic resumes. Four directed tests, and they find the collapsed-gate design immediately.

29. Wrong Resume — Traffic Before Reliability State Is Aligned

// WRONG — one bit stands in for four conditions.
assign traffic_may_resume = phy_operational;

The failure, concretely. Objects A and B were replay-owned when recovery began — sent, delivery unconfirmed. Recovery completes. phy_operational rises. Fresh object C is admitted and transmitted before A and B have been resolved.

Three distinct damages, depending on what the traffic is.

Ordering is violated. If the protocol requires A before C — and 12.2 established which orderings must hold — the receiver has now observed them in the wrong order. A CRC will not catch this: every object is individually intact.

Duplication becomes possible. If A actually did arrive before the recovery and its confirmation was lost, then the replay of A is a duplicate. Duplicate suppression (14.3 §22) handles that — but only if the receiver's history state is coherent, and resuming before reliability state is aligned is precisely the condition in which it may not be.

And the replay window can be exceeded. New allocations consume replay entries that the unresolved objects still occupy. If C, D, E are admitted before A and B retire, the buffer can fill with a mixture the retry machinery was not designed to disentangle (14.3 §28).

Why the collapsed gate gets written. Because phy_operational is the condition that takes longest — retraining dominates the recovery latency — so it is empirically the last to become true, and a designer observing that concludes it is sufficient. It is the last on a healthy recovery and not on a failing one, and the failing case is the one that matters.

30. Recovery Liveness, Bounded

// LIVENESS, with assumptions stated. Unprovable without them, and a property
// with unstated assumptions gets disabled the first time it fires.
//
//   A1: the physical channel is recoverable at SOME configuration on the ladder
//   A2: the peer participates in recovery
//   A3: training converges within TRAIN_BOUND when the channel is usable
//
// Without A1 this property MUST fail — and correctly so: an unrecoverable
// channel cannot be recovered. That is what REC_FAIL is for.
localparam int RECOVERY_BOUND = RECOVERY_MAX_ATTEMPTS * (TRAIN_BOUND + VALIDATE_BOUND);
 
assume property (@(posedge clk) disable iff (!rst_n)
  (rec_q == REC_RETRAIN) |-> ##[1:TRAIN_BOUND] (retrain_ok || attempts_exhausted));
 
assume property (@(posedge clk) disable iff (!rst_n)
  (rec_q == REC_QUIESCE) |-> ##[1:QUIESCE_BOUND] quiesce_complete);
 
// The controller must LEAVE recovery — to success or to explicit failure.
property p_recovery_terminates;
  @(posedge clk) disable iff (!rst_n)
    (rec_q != REC_IDLE) |-> ##[1:RECOVERY_BOUND] (rec_q inside {REC_IDLE, REC_FAIL});
endproperty
a_recovery_terminates: assert property (p_recovery_terminates);
 
// And a stronger safety companion: no phase may be occupied indefinitely.
property p_no_phase_stalls(rec_state_t s, int unsigned bound);
  @(posedge clk) disable iff (!rst_n)
    (rec_q == s) |-> ##[1:bound] (rec_q != s);
endproperty
a_no_quiesce_stall : assert property (p_no_phase_stalls(REC_QUIESCE,  QUIESCE_BOUND));
a_no_capture_stall : assert property (p_no_phase_stalls(REC_CAPTURE,  4));
a_no_validate_stall: assert property (p_no_phase_stalls(REC_VALIDATE, VALIDATE_BOUND));

Architecture. A bounded termination property plus per-phase bounds, with three environment assumptions written down.

Why bounded rather than s_eventually. An unbounded eventuality is checkable in neither simulation nor bounded formal in any practical sense. A bound derived from the attempt budget and the per-phase bounds is checkable in both, and the derivation itself is useful documentation — it says what the design's worst-case recovery latency actually is, which is a number systems engineers ask for and designs frequently cannot answer.

The consequent is REC_IDLE or REC_FAIL, and that is deliberate. Reaching REC_FAIL is a successful outcome of the property: the controller determined the link is not recoverable and said so. The liveness requirement is that recovery terminates, not that it succeeds — and conflating the two produces a property that fails on correct behaviour whenever a link is genuinely broken.

Contract. The assumptions are obligations on the environment: in formal they constrain it, in simulation they are obligations on the testbench, and a testbench that models a peer which never responds will fail these properties correctly. That case needs its own directed test with the assumptions disabled, checking that the design reaches REC_FAIL rather than hanging.

Failure it catches. A phase whose exit condition can never be satisfied — most commonly REC_QUIESCE waiting for a drain that cannot complete because the thing it is draining needs the link that is down. That deadlock is invisible to every safety property, exactly as 13.3 §19 established, and it is the single most common way a recovery controller hangs with a healthy-looking PHY (§35).

31. The Recovery Scoreboard

// Verification-only reference model. Not synthesisable.
class recovery_scoreboard;
 
  typedef struct {
    bit        accepted;         // ownership transferred to transport
    bit        transport_owned;  // in the replay buffer
    bit        maybe_remote;     // possibly executed remotely (Section 20 row 3)
    bit        response_pending;
    bit        resolved;         // delivered, completed, or REPORTED as failed
    int        epoch_at_accept;
  } obj_t;
 
  obj_t          objects[int];
  int            model_epoch;
  cfg_t          model_active_cfg;
  int            recovery_count, commit_count, loss_count, dup_count;
 
  // ---- Check 1: no silent loss across a recovery. THE core property.
  function void check_no_loss();
    foreach (objects[id])
      if (objects[id].accepted && !objects[id].resolved && !dut_has_state(id)) begin
        loss_count++;
        $error("SILENT LOSS: object %0d accepted, unresolved, no DUT state", id);
      end
  endfunction
 
  // ---- Check 2: no duplicate SEMANTIC acceptance. Transport attempts may
  //      repeat; semantic acceptance may not (Section 21).
  function void check_no_duplicate_semantics(int id);
    if (objects[id].resolved) begin
      dup_count++;
      $error("DUPLICATE SEMANTIC ACCEPT: object %0d resolved twice", id);
    end
  endfunction
 
  // ---- Check 3: no configuration mixing. Every transported object must be
  //      framed entirely within one epoch.
  function void check_no_epoch_mixing(int id, int epoch_at_start, int epoch_at_end);
    if (epoch_at_start != epoch_at_end)
      $error("EPOCH MIXING: object %0d spans epochs %0d..%0d",
             id, epoch_at_start, epoch_at_end);
  endfunction
 
  // ---- Check 4: the model's configuration changes only at commits, and the
  //      epoch tracks them one-for-one.
  function void on_commit(cfg_t new_cfg);
    model_active_cfg = new_cfg;
    model_epoch++;
    commit_count++;
    if (model_epoch != dut_epoch())
      $error("EPOCH DIVERGENCE: model %0d, dut %0d", model_epoch, dut_epoch());
  endfunction
 
  // ---- Check 5: diagnostic context survived.
  function void check_context_preserved(int expected_first_kind);
    if (dut_recovery_context_valid() &&
        dut_recovery_context_first_kind() != expected_first_kind)
      $error("CONTEXT OVERWRITTEN: expected first kind %0d, got %0d",
             expected_first_kind, dut_recovery_context_first_kind());
  endfunction
 
endclass

Architecture. Five checks in three categories: loss and duplication (the semantic-safety pair), configuration integrity (the epoch pair), and diagnostic survival.

Check 1 is the one §11 exists to violate, and it is written as a census over the model's own record of what was accepted rather than as a comparison against a DUT counter. Reading a DUT counter would agree with the DUT about a loss, which is the same reasoning 13.4 §30 applied to occupancy.

Check 2 distinguishes the two kinds of repetition, and the distinction is the module's central idea. A transport object may be transmitted many times; a semantic operation may be accepted once. A scoreboard that counts "deliveries" without that distinction will either flag every legitimate replay as a duplicate or miss a genuine double-execution.

Check 3 needs the epoch at the start and end of each object, which requires the monitor to timestamp both — a real constraint on the testbench and the only way to catch an object framed across a configuration change.

Check 5 is the one usually missing, and it is what turns §14 from an untested assumption into a checked property. It requires the testbench to know which error it injected first.

Failure classes caught that assertions do not. Check 1 spans four structures, so no local assertion expresses it. Check 2 requires knowing an object's semantic identity across multiple transport attempts. Check 5 requires knowing ground truth about injection order.

32. Coverage

covergroup cg_recovery @(posedge clk);
  option.per_instance = 1;
 
  // --- Phases and, more importantly, ARCS.
  cp_phase : coverpoint rec_q;
  cp_arc   : coverpoint {rec_prev_q, rec_q} {
    bins idle_qui   = {{REC_IDLE,     REC_QUIESCE}};
    bins qui_cap    = {{REC_QUIESCE,  REC_CAPTURE}};
    bins cap_retr   = {{REC_CAPTURE,  REC_RETRAIN}};
    bins retr_recfg = {{REC_RETRAIN,  REC_RECONFIG}};
    bins recfg_val  = {{REC_RECONFIG, REC_VALIDATE}};
    bins val_res    = {{REC_VALIDATE, REC_RESUME}};
    bins val_retr   = {{REC_VALIDATE, REC_RETRAIN}};   // validation failed
    bins res_idle   = {{REC_RESUME,   REC_IDLE}};
    bins retr_fail  = {{REC_RETRAIN,  REC_FAIL}};      // budget exhausted
  }
 
  // --- WHAT WAS IN FLIGHT when recovery started. Section 20's five states —
  //     recovery on an idle link proves almost nothing.
  cp_inflight : coverpoint inflight_class_at_trigger {
    bins idle_link    = {IF_NONE};
    bins queued       = {IF_QUEUED};
    bins replay_owned = {IF_REPLAY};
    bins maybe_remote = {IF_MAYBE_REMOTE};
    bins resp_pending = {IF_RESP_PENDING};
    bins mixed        = {IF_MIXED};        // the realistic case
  }
 
  // --- Trigger class (Section 7).
  cp_trigger : coverpoint recovery_trigger_cause {
    bins integrity = {TRG_CRC};
    bins framing   = {TRG_FRAMING};
    bins retry_exh = {TRG_RETRY_EXHAUSTED};
    bins lane      = {TRG_LANE_DEGRADE};
    bins rate      = {TRG_RATE_FAIL};
    bins sw_retrain= {TRG_SOFTWARE};       // a HEALTHY-link retrain (Section 7)
  }
 
  // --- Did the configuration actually CHANGE? Section 27's assertions are
  //     nearly vacuous if every commit recommits identical values.
  cp_cfg_change : coverpoint cfg_changed_at_commit {
    bins unchanged   = {0};
    bins lane_mask   = {1};
    bins rate        = {2};
    bins width       = {3};
  }
 
  // --- Escalation and flap.
  cp_esc_level : coverpoint esc_level_q;
  cp_attempts  : coverpoint recovery_attempt_q {
    bins first = {1};
    bins mid   = {[2:RECOVERY_MAX_ATTEMPTS-1]};
    bins last  = {RECOVERY_MAX_ATTEMPTS};
  }
  cp_flap      : coverpoint flap_detected;
 
  // --- Diagnostics survived.
  cp_ctx_valid  : coverpoint rec_ctx_q.valid;
  cp_second_rec : coverpoint second_recovery_in_debug_epoch;
 
  // --- Crosses that carry the information.
  x_trigger_inflight : cross cp_trigger, cp_inflight;   // the real matrix
  x_cfg_inflight     : cross cp_cfg_change, cp_inflight;
  x_esc_trigger      : cross cp_esc_level, cp_trigger;
  x_ctx_second       : cross cp_ctx_valid, cp_second_rec;  // Section 14
endcovergroup

Five bins whose value is being non-zero, each proving a mechanism was genuinely exercised:

cp_inflight — every bin except idle_link. Recovery on an idle link is the easy case and the one random stimulus produces most often. §12's survival property and §21's whole matrix are vacuous unless objects were actually in flight, and mixed is the realistic case that finds interactions between the rows.

cp_cfg_change — the three changed bins. Without them, §27's four assertions have only ever seen commits that changed nothing.

cp_arc.val_retr and cp_arc.retr_fail. Validation failing after successful training, and budget exhaustion. Both need directed stimulus; neither occurs spontaneously in a working design.

x_ctx_second. A second recovery within one debug epoch, with the context still valid — the direct test for §14.

cp_trigger.sw_retrain. A software-initiated retrain on a healthy link (§7). A controller that treats it as a fault escalates on a routine margin measurement, and only this bin finds that.

33. Flagship Trace — Recovery With Work In Flight

Illustrative. Objects A and B are replay-owned; transaction T has a response outstanding; object D is queued locally and not yet accepted by transport. A lane event triggers recovery at cycle 3.

Eight readings, and the valuable ones are the columns that do not change.

Cycle 3: admission stops, and D is not accepted. D was never handed over, so it stays where it is with its owner intact. That is the difference between preventing ownership transfer and revoking it (§10).

Cycles 3–19: the Replay column never loses an entry, and T is never cleared. Sixteen cycles of recovery — including a full retrain with the PHY down — and the transport and semantic obligations are exactly as they were. §11's bug is the version of this table where those two columns go blank at cycle 3.

Cycle 6: capture precedes retrain by one phase. The context records mask 0xFFFF and rate 32 — the configuration at the time of failure, not the one in force afterwards. At cycle 20 the active mask is 0xFFFE, and without the capture there would be no record that the failure occurred under a different map.

Cycles 10–12: requested is 0xFFFE while active remains 0xFFFF. Three cycles in which the design knows what it wants and has not acted on it. §25's bug is the version of this table where the Active cfg column changes at cycle 10 — and every beat after that is misassembled.

Cycle 13 is the single atomic event. Active configuration and epoch change on one edge. Not the lane mask at cycle 13 and the rate at cycle 14; not a registered copy in one consumer arriving at cycle 14.

Cycles 15–18: replay resolves before new admission. A, B and C are re-driven under the new configuration while the admission gate is still closed. §29's bug is the version where admission opens at cycle 14 — and D overtakes A, B and C.

Cycle 19: all four resume conditions hold, and only then does traffic resume. The PHY has been up since cycle 9. Ten cycles separate "the PHY works" from "traffic may resume", and a design gated on phy_operational alone would have resumed in the middle of the reconfiguration.

Cycle 20: T's response arrives and matches. The transaction was accepted before recovery, was outstanding throughout, and completes normally afterwards. From the Protocol Layer's point of view, the recovery was a latency event. That is the goal.

34. Failure Trace — the Cleared Outstanding Table

The same scenario with §11's outstanding_q <= '0.

What happens at cycle 20, and all three outcomes are bad.

Best case — the response is dropped and counted as unexpected. The requester waits for a response that was delivered and discarded, and times out much later. The error report says "transaction timeout" and points at the Protocol Layer. The actual fault was a register clear at cycle 3, seventeen cycles and four mechanisms away.

Worse — the identity has been reused. 12.4 established that identities are recycled after retirement. If T's identity was reallocated to a new transaction during or after recovery, the stale response matches it. Data belonging to T is delivered as the response to a different request — 14.1 §6's stale-response alias, with every integrity check clean, because the response is perfectly well-formed.

Worst — the operation is reissued. If some layer notices T is missing and regenerates it, the remote executes it a second time. A write applied twice. §22 is the reason: the timeout never proved the first execution did not happen.

And the diagnostic picture is actively misleading. Recovery succeeded. Training converged, the configuration committed, the link runs at full rate, error counters are clean. Everything the recovery machinery reports says the recovery worked — and it did, at the transport level. The damage is one layer up, invisible from where the recovery controller sits.

A recovery that restores the link and loses a transaction has not succeeded. It has converted a transport error into a semantic one, and moved the symptom to a layer that cannot explain it.

35. Debug Taxonomy

36. Debug Checklist

1. What triggered the recovery? Trigger class first — it narrows which assumption (§5) is suspect.
2. Was it the first error? If the context record was overwritten, the answer is unavailable and that is itself the finding (§14).
3. Which objects had been accepted at the trigger cycle? Ownership had transferred; they must still exist.
4. Which were transport- or replay-owned? These resolve through 14.3, not through the recovery controller.
5. Which may have been remote-accepted? §20 row 3 — blind reissue is unsafe for these.
6. Which transaction entries were live, and are they still? §34's signature.
7. Did replay state survive the retrain? Clearing it destroys the mechanism recovery depends on.
8. Which configuration became invalid, and which stayed valid? Rebuilding all six assumptions when one failed is expensive and diagnostically blind (§16).
9. When was the new configuration committed? One cycle, one event — check it.
10. Did every datapath consumer switch on that same edge? A registered copy is a one-cycle disagreement (§26).
11. Did both sides validate before the commit? A unilateral change is §25.
12. Did traffic resume before reliability obligations resolved? §29 — compare the admission-reopen cycle against replay occupancy.
13. How many attempts occurred, and did the escalation ladder move? A ladder that never moves is either a healthy link or a broken reset condition (§17).
14. How many recoveries per unit time? Flap, not failure (§19).
15. What is the first divergence from the reference model? §31 — and check the loss counter before anything else.

37. Common Misconceptions

"Recovery is just a reset." Reset is entitled to destroy broad architectural state; recovery is entitled to rebuild only what the failure invalidated. The difference is a state-lifetime argument, and the practical consequence is that a reset-shaped recovery converts a transport error into an unbounded semantic wait (§4, §11).

"Clearing FIFOs makes recovery safer." It removes objects that might be corrupt and also objects that are fine — and those had owners who have already released their copies. Forgetting is not a resolution; the correct action is to resolve through the reliability mechanism (§11).

"Every transaction should be resent after recovery." A timeout does not prove non-execution, and three of the four causes consistent with "no response" make a reissue unsafe. Retransmit the transport object under an identity the receiver recognises; never regenerate the semantic operation (§21, §22).

"A timeout proves the remote did not execute it." It proves no response arrived. The request may have arrived and executed with its response lost, or be executing right now (§22).

"PHY ready means semantic traffic may resume." It means bits can move. It says nothing about whether the two sides agree on the configuration, whether replay obligations are resolved, or whether permission exists for the new epoch (§28, §29).

"The requested configuration can replace the active one as soon as it is known." Then the transmitter uses the new mapping while the receiver still uses the old, and every subsequent beat is misassembled — deterministically, and reported as a CRC failure that looks physical (§25).

"First-error state may be reset on recovery entry." Recovery generates secondary symptoms by design, so clearing the primary evidence leaves only the secondary. The report becomes "training timeout" and the actual fault becomes unattributable (§14).

"One recovery attempt policy fits every failure." The trigger tells you which assumption is suspect and therefore what the narrowest sufficient response is. Mapping every trigger onto one maximal response discards that information and destroys the diagnostic value of a successful recovery (§7, §16).

"Successful retraining proves old transaction state is safe." Retraining proves the physical link works. It says nothing about whether the tracking table, the replay buffer or the permission state survived correctly — and §34's trace is a recovery in which training succeeded and a transaction was lost.

"Recovery correctness is only an FSM property." The FSM sequences the phases; the correctness lives in what is preserved, when the configuration is committed, and what the resume gate requires. A perfectly correct FSM with §11's clear or §29's collapsed gate is a broken recovery.

"Repeated successful recoveries are fine — they succeeded." Each one costs the full quiesce-retrain-validate-commit latency plus the pipeline restart. A link recovering four times a millisecond is nominally healthy and delivers almost nothing, and no failure counter registers it (§19).

38. Understanding Check

39. Summary and What Comes Next

Recovery is controlled loss of assumptions. Normal operation runs on six — timing, lane map, rate, framing, reliability coherence, resource alignment — and an error means at least one may not hold. The trigger identifies which, and that information is the design's most valuable asset, because it makes minimal recovery possible and gives a successful recovery diagnostic meaning.

Recovery is not reset, and the difference is state lifetime. Link-epoch state is rebuilt; per-object and per-transaction state survives; diagnostic state survives everything. Clearing more is not safer — it converts a recoverable transport error into an unbounded semantic wait, and clearing the replay buffer specifically destroys the mechanism recovery depends on.

Quiesce prevents ownership transfer; it never revokes it. The last moment to decline an object is the cycle before its handshake completes, and after that a recovery may delay it arbitrarily but may not drop it.

Capture precedes retrain. A recovery generates secondary symptoms by design, so the primary evidence must be latched before the action that destroys it. Otherwise the field report says "training timeout" and the actual fault is unattributable.

Outstanding transactions occupy five states and three of them are hard, because local and remote knowledge disagree and no local action can reconcile them. A timeout proves no response arrived; it does not prove the request was not executed — and three of the four causes consistent with that observation make a reissue unsafe. So recovery may re-drive transport and may never re-execute semantics.

Requested configuration is not active configuration, and the commit is one atomic event gated on validation, peer agreement, no object in flight, and exactly-once. Exposing the requested value early is deterministic corruption reported as a CRC failure that looks physical — and the highest-value assertion in the chapter is the cheap one saying the active configuration is stable without a commit.

The resume gate is a conjunction of four conditions, and phy_operational is only one. It is the last to become true on a healthy recovery and not on a failing one, which is exactly why it gets mistaken for sufficient.

And escalate on failure, but detect flap on success. A ladder that widens on failure and resets on success handles persistent faults; only a rate measurement handles the link that recovers correctly four times a millisecond and delivers nothing. Liveness must be bounded and must admit an explicit failure state — terminating is the requirement, succeeding is not.

Recovery repairs the link when its operating assumptions become unsafe. Retry solves a narrower but equally important problem: when the link is still usable, how do we resend only the transport work that was not safely accepted — without executing the semantic transaction twice?

• 14.3 — Retry Mechanisms — the Adapter's replay buffer, acknowledgement processing, the retirement point, and exactly-once delivery across a lost acknowledgement.

Browse the full path on the UCIe tutorials index.