Link Failures

26.1 ruled out the physical family whenever the link reached L0. This chapter is that family, and its central problem is that the indicator everybody reads is a single bit that says the link came back.

It did come back. At eight lanes instead of sixteen, one generation down, after twelve milliseconds in recovery, having replayed twelve hundred flits — and the interrupt that fired says exactly one thing, which is that it is up.

1. The Engineering Problem — Coming Back Is Not The Same As Being Back

A retrain loop spends bandwidth on recovery. Twenty retrains at twelve milliseconds in a one-second window is 240 milliseconds down — a quarter of the link gone, with the link reporting up throughout. Section 5.

A corrected error is not a free error. Twelve hundred CRC errors at thirty-two flits of replay each is 38,400 flits of retransmission, and a health check that only counts uncorrectables reports the link clean. Section 6.

A link runs at its worst lane. Fifteen healthy lanes and one at thirty millivolts against a fifty-millivolt requirement averages to ninety — and the link runs at thirty. Section 7.

And it can come back narrower. Eight lanes of sixteen at one generation down is forty percent of the bandwidth, on a recovery that every indicator calls successful. Section 9.

This chapter against 26.1, stated precisely. That one owns a link that never came up. This one owns a link that comes up, goes down, and comes back — which is why every model here measures what the link is rather than whether it is, and why section 14's weak definition is a link-up interrupt.

2. The One-Sentence Model

A link is sound when it returned to L0, nothing needed correcting either, the worst lane has margin rather than the mean, it returned at the planned width and speed, the error rate stays inside the limit once aged and cornered, and every corner was characterised — and "it came back" is one of those six.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model takes one measurement of a link that is working and computes what it is costing. A real investigation has an analyser, lane margining, eye diagrams, error counters and a thermal chamber, and none of that is reproduced. What is reproduced is the arithmetic each measurement implies — bandwidth lost, margin remaining, headroom to the limit — and the shape of the mistake when only the status bit is read.

Three simplifications are worth stating. Section 7 treats lane margin as a single millivolt figure where a real measurement is a two-dimensional eye. Section 10 uses multiplicative ageing and corner factors where the real relationship is not a product. Section 13 treats drops as a flat rate where a real intermittent has structure. In each case the conclusion is the same and the model is abbreviated.

Each model is built twice — a correct build and a broken build selected by a parameter. Every broken build here is the reading the most visible indicator supports: it came back so it is up, nothing uncorrectable so it is clean, the average margin is fine, any speed works, it passed at room temperature, an hour of quiet is enough. Each is a true statement about the thing it measures, and none of them is the question.

Figure 1 — The interrupt is accurate. It reports the one fact it measures, and the four numbers that decide whether the link is usable are in registers nobody read.

5. RTL 1 — A Retrain Loop Spends The Link's Bandwidth

// RTL 1 - the retrain loop. A link that trains, drops and retrains spends its
// bandwidth on recovery, and the loop rate is the number that matters.
module retrain_loop #(parameter int LINK_IS_UP_AGAIN = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] window_ms, retrains, recovery_ms, nominal_gbps,
  output logic [15:0] down_ms, up_ms, effective_gbps, lost_pct,
  output logic        link_usable,
  output logic [7:0]  n_measures, n_degraded,
  output logic        recovery_ignored_err
);
  logic [31:0] d_q, e_q, p_q;
  assign d_q = {16'd0, retrains} * {16'd0, recovery_ms};
  // A link counted as up whenever it is in L0 ignores the time it was not.
  assign down_ms = (LINK_IS_UP_AGAIN != 0) ? 16'd0
                 : ((d_q > {16'd0, window_ms}) ? window_ms : d_q[15:0]);
  assign up_ms = (window_ms > down_ms) ? (window_ms - down_ms) : 16'd0;
  assign e_q = (window_ms == 16'd0) ? 32'd0
             : (({16'd0, up_ms} * {16'd0, nominal_gbps}) / {16'd0, window_ms});
  assign effective_gbps = (e_q > 32'd65535) ? 16'hFFFF : e_q[15:0];
  assign p_q = (nominal_gbps == 16'd0) ? 32'd0
             : ((({16'd0, nominal_gbps} - e_q) * 32'd100) / {16'd0, nominal_gbps});
  assign lost_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  assign link_usable = (lost_pct <= 16'd5);
  // Recovery time a link spent down, reported as none.
  assign recovery_ignored_err = measure && (retrains != 16'd0)
                                && (recovery_ms != 16'd0) && (down_ms == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_degraded <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (!link_usable) n_degraded <= n_degraded + 8'd1;
    end
  end
endmodule

Seven measurements. A one-second window at 64 Gbps unless stated.

Three measurements degraded when the recovery is charged; none when it is not.

A link that retrains twenty times a second is up twenty times a second, and that is how it gets reported. The link status register is correct on every read. What it cannot show is the 240 milliseconds between the reads — a quarter of the window, spent in recovery, while every counter that samples the register sees L0.

Rows three and four are the threshold and they are one retrain apart. Five retrains is six percent lost and four is four — the line between a link somebody investigates and a link nobody does is a single event per second. Row six reaches exactly five percent on a 100 Gbps link, which is the boundary the rule is written on.

Row seven is the measurement that was never taken. Twenty retrains with no recovery time recorded computes to zero down-time and reads as usable — and the model reports that as a missing measurement rather than as a healthy link. The retrain count alone is not enough; the cost per retrain is the other half.

6. RTL 2 — A Corrected Error Costs A Round Trip

// RTL 2 - corrected is not free. A CRC error that the replay mechanism fixes
// costs a round trip, and the count of them is the health of the link.
module replay_cost #(parameter int CORRECTED_IS_CLEAN = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] flits, crc_errors, replay_flits, uncorrectable,
  output logic [15:0] replayed, overhead_pct, error_per_million, wasted_flits,
  output logic        link_healthy,
  output logic [7:0]  n_measures, n_unhealthy,
  output logic        corrected_hidden_err
);
  logic [31:0] w_q, o_q, e_q;
  assign w_q = {16'd0, crc_errors} * {16'd0, replay_flits};
  assign wasted_flits = (w_q > 32'd65535) ? 16'hFFFF : w_q[15:0];
  // A link counted clean whenever nothing was uncorrectable hides the replays.
  assign replayed = (CORRECTED_IS_CLEAN != 0) ? 16'd0 : crc_errors;
  assign o_q = (flits == 16'd0) ? 32'd0
             : (({16'd0, wasted_flits} * 32'd100) / {16'd0, flits});
  assign overhead_pct = (o_q > 32'd100) ? 16'd100 : o_q[15:0];
  assign e_q = (flits == 16'd0) ? 32'd0
             : (({16'd0, crc_errors} * 32'd1000000) / {16'd0, flits});
  assign error_per_million = (e_q > 32'd65535) ? 16'hFFFF : e_q[15:0];
  assign link_healthy = (replayed == 16'd0) && (uncorrectable == 16'd0);
  // Corrected errors on a link reported as clean.
  assign corrected_hidden_err = measure && (crc_errors != 16'd0)
                                && (replayed == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_unhealthy <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (!link_healthy) n_unhealthy <= n_unhealthy + 8'd1;
    end
  end
endmodule

Five measurements. Fifty thousand flits, thirty-two flits replayed per error.

Three measurements unhealthy when corrections count; one when only uncorrectables do.

CXL's replay mechanism makes a CRC error invisible to the transaction layer and expensive to the link. Twelve hundred errors at thirty-two flits of replay each is thirty-eight thousand four hundred flits of retransmission — seventy-six percent of the traffic in this window — and not one of them reaches an uncorrectable-error counter.

Row four is the case the two views agree on, and it is worth contrasting with row one. An uncorrectable error is unhealthy under both readings, because it is the thing the clean view measures. The disagreement is entirely about the twelve hundred that were fixed.

Row three is the detector's resolution. A single corrected error is enough to call the link unhealthy, which is deliberately strict — the useful metric is the rate, and the boolean exists to make a non-zero rate visible at all.

7. RTL 3 — A Link Runs At Its Worst Lane

// RTL 3 - the worst lane. A link is as good as its worst lane, and an average
// across sixteen hides one that is failing.
module lane_margin #(parameter int AVERAGE_THE_LANES = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] lanes, total_margin_mv, worst_lane_mv, margin_needed_mv,
  output logic [15:0] reported_mv, mean_mv, shortfall_mv, marginal_lanes,
  output logic        margin_adequate,
  output logic [7:0]  n_measures, n_short,
  output logic        averaging_hides_err
);
  assign mean_mv = (lanes == 16'd0) ? 16'd0 : (total_margin_mv / lanes);
  // The link runs at the worst lane, whatever the mean says.
  assign reported_mv = (AVERAGE_THE_LANES != 0) ? mean_mv : worst_lane_mv;
  assign shortfall_mv = (margin_needed_mv > worst_lane_mv)
                        ? (margin_needed_mv - worst_lane_mv) : 16'd0;
  assign marginal_lanes = (shortfall_mv != 16'd0) ? 16'd1 : 16'd0;
  assign margin_adequate = (reported_mv >= margin_needed_mv);
  // A lane below the requirement, on a link reported as having margin.
  assign averaging_hides_err = measure && (shortfall_mv != 16'd0) && margin_adequate;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_short <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (!margin_adequate) n_short <= n_short + 8'd1;
    end
  end
endmodule

Five measurements. Sixteen lanes, fifty millivolts needed.

Three measurements short at the worst lane; one on the average.

A serial link is a conjunction, not an average. Sixteen lanes carry one stream, so a single lane below its margin is the link's margin — and fifteen healthy lanes at ninety-four millivolts are irrelevant to whether the sixteenth errors.

Row one is the arithmetic of the wrong summary. A mean of ninety against a requirement of fifty passes comfortably, and the link fails on one lane. The measurement exists — per-lane margining produces sixteen numbers — and the summary that reaches a review is one of them.

Rows three and four are the requirement and they are one millivolt apart. Exactly fifty is exactly adequate; forty-nine is not. The threshold is a comparison rather than a judgement, which is what makes it reviewable at all.

Figure 3 — Both numbers come from the same sixteen measurements. A serial link is a conjunction, so the mean is arithmetic about a quantity the link does not have.

8. RTL 4 — Every Excursion Out Of L0 Costs Cycles

// RTL 4 - what a recovery costs. Every LTSSM excursion out of L0 is cycles the
// link is not carrying traffic, and the replay buffer drains behind it.
module recovery_cost #(parameter int RECOVERY_IS_INSTANT = 0) (
  input  logic clk, rst_n,
  input  logic        measure,
  input  logic [15:0] excursions, ltssm_cycles, drain_cycles, window_cycles,
  output logic [15:0] per_excursion, total_lost, available, lost_pct,
  output logic        cost_acceptable,
  output logic [7:0]  n_measures, n_costly,
  output logic        excursion_free_err
);
  logic [31:0] t_q, p_q;
  assign per_excursion = (RECOVERY_IS_INSTANT != 0) ? 16'd0
                       : (ltssm_cycles + drain_cycles);
  assign t_q = {16'd0, excursions} * {16'd0, per_excursion};
  assign total_lost = (t_q > {16'd0, window_cycles}) ? window_cycles
                    : ((t_q > 32'd65535) ? 16'hFFFF : t_q[15:0]);
  assign available = (window_cycles > total_lost)
                     ? (window_cycles - total_lost) : 16'd0;
  assign p_q = (window_cycles == 16'd0) ? 32'd0
             : (({16'd0, total_lost} * 32'd100) / {16'd0, window_cycles});
  assign lost_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  assign cost_acceptable = (lost_pct <= 16'd2);
  // Excursions out of L0 costed at nothing.
  assign excursion_free_err = measure && (excursions != 16'd0)
                              && (ltssm_cycles != 16'd0) && (per_excursion == 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_costly <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (!cost_acceptable) n_costly <= n_costly + 8'd1;
    end
  end
endmodule

Six measurements. A fifty-thousand-cycle window, 800 LTSSM cycles plus 200 of drain.

Three measurements too costly; none when recovery is instant.

An LTSSM excursion is not instantaneous and the replay buffer drains behind it. Eight hundred cycles in Recovery plus two hundred to refill is a thousand cycles per excursion, and forty of them in a fifty-thousand-cycle window is eighty percent of the link's time.

Rows three and four are the policy line, one excursion apart. One excursion in fifty thousand cycles is exactly two percent and acceptable; two is four and is not. The number is arguable; having one is what makes an excursion count actionable rather than a statistic nobody compares to anything.

Row six is section 5's failure in a different unit. Excursions counted with no cost per excursion measured reads as free — the count and the cost are two measurements, and only one of them is usually taken.

9. RTL 5 — It Can Come Back Narrower

// RTL 5 - it came back narrower. A link that recovers may recover at a lower
// width or speed, and nothing raises an error because the link is up.
module degraded_return #(parameter int UP_IS_UP = 0) (
  input  logic clk, rst_n,
  input  logic        recover_it,
  input  logic [15:0] planned_lanes, actual_lanes, planned_gen, actual_gen,
  output logic [15:0] lane_ratio_pct, gen_ratio_pct, bandwidth_pct, lost_pct,
  output logic        as_planned,
  output logic [7:0]  n_recoveries, n_degraded,
  output logic        degraded_return_err
);
  logic [31:0] l_q, g_q, b_q;
  assign l_q = (planned_lanes == 16'd0) ? 32'd100
             : (({16'd0, actual_lanes} * 32'd100) / {16'd0, planned_lanes});
  assign lane_ratio_pct = (l_q > 32'd100) ? 16'd100 : l_q[15:0];
  assign g_q = (planned_gen == 16'd0) ? 32'd100
             : (({16'd0, actual_gen} * 32'd100) / {16'd0, planned_gen});
  assign gen_ratio_pct = (g_q > 32'd100) ? 16'd100 : g_q[15:0];
  assign b_q = ({16'd0, lane_ratio_pct} * {16'd0, gen_ratio_pct}) / 32'd100;
  assign bandwidth_pct = (b_q > 32'd100) ? 16'd100 : b_q[15:0];
  assign lost_pct = 16'd100 - bandwidth_pct;
  // A link that is up is reported as recovered, at whatever width it found.
  assign as_planned = (UP_IS_UP != 0) ? 1'b1 : (bandwidth_pct >= 16'd100);
  assign degraded_return_err = recover_it && (bandwidth_pct < 16'd100) && as_planned;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_recoveries <= 8'd0; n_degraded <= 8'd0;
    end else if (recover_it) begin
      n_recoveries <= n_recoveries + 8'd1;
      if (!as_planned) n_degraded <= n_degraded + 8'd1;
    end
  end
endmodule

Six recoveries. A sixteen-lane gen-5 link unless stated.

Four recoveries degraded; none when up is up.

A link that loses lanes during recovery re-trains at the width it can manage, and that is the correct behaviour. It is also forty percent of the bandwidth in row one, arrived at silently, with the link reporting up and the device working.

Row five is why the bar is a hundred rather than ninety. Fifteen lanes of sixteen is ninety-three percent and is not as planned — a lane is gone, and the reason it is gone will take the other fifteen with it eventually. A degraded-but-adequate threshold hides exactly the early warning that matters.

Row six is the missing baseline. With no planned width and speed recorded, every recovery reads as full — and the model reports that as an absent plan rather than a successful recovery. The planned values are a device attribute somebody has to write down.

10. RTL 6 — Inside The Limit Is Not The Same As Far From It

// RTL 6 - inside the limit is not the same as far from it. A bit error rate
// one decade below the specification limit has one decade of headroom.
module ber_headroom #(parameter int IN_SPEC_IS_FINE = 0) (
  input  logic clk, rst_n,
  input  logic        judge,
  input  logic [15:0] measured_e12, limit_e12, ageing_factor, corner_factor,
  output logic [15:0] headroom_x, aged_e12, worst_case_e12, margin_pct,
  output logic        headroom_adequate,
  output logic [7:0]  n_judgements, n_marginal,
  output logic        no_headroom_err
);
  logic [31:0] a_q, w_q, h_q, m_q;
  assign h_q = (measured_e12 == 16'd0) ? 32'd65535
             : ({16'd0, limit_e12} / {16'd0, measured_e12});
  assign headroom_x = (h_q > 32'd65535) ? 16'hFFFF : h_q[15:0];
  assign a_q = {16'd0, measured_e12} * {16'd0, ageing_factor};
  assign aged_e12 = (a_q > 32'd65535) ? 16'hFFFF : a_q[15:0];
  assign w_q = {16'd0, aged_e12} * {16'd0, corner_factor};
  assign worst_case_e12 = (w_q > 32'd65535) ? 16'hFFFF : w_q[15:0];
  assign m_q = (limit_e12 == 16'd0) ? 32'd0
             : (({16'd0, worst_case_e12} * 32'd100) / {16'd0, limit_e12});
  assign margin_pct = (m_q > 32'd65535) ? 16'hFFFF : m_q[15:0];
  // A link inside the limit today is passing; whether it stays there is the
  // question the headroom answers.
  assign headroom_adequate = (IN_SPEC_IS_FINE != 0) ? (measured_e12 <= limit_e12)
                           : (worst_case_e12 <= limit_e12);
  assign no_headroom_err = judge && (worst_case_e12 > limit_e12)
                           && headroom_adequate;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_judgements <= 8'd0; n_marginal <= 8'd0;
    end else if (judge) begin
      n_judgements <= n_judgements + 8'd1;
      if (!headroom_adequate) n_marginal <= n_marginal + 8'd1;
    end
  end
endmodule

Five judgements. Ageing 2×, corner 3×, a limit of 100 unless stated.

Two judgements marginal once aged and cornered; none on today's number.

A link inside its limit today and a link that will be inside it in three years at ninety degrees are different claims. Twenty against a limit of a hundred is five times of headroom, and two-times ageing with a three-times corner factor puts it at a hundred and twenty — outside, on hardware that passes every test today.

Rows three and four are the boundary, and they are one unit of measured rate apart. Sixteen ages and corners to exactly the limit; seventeen to a hundred and two. The sensitivity is what makes the margin worth stating — a small change in today's number is a large change in three years' time.

Row five is honest rather than reassuring. A rate too low to measure has unbounded headroom because there is no measurement to age, and the model says so. It is the correct answer and it is not evidence.

11. RTL 7 — A Corner Nobody Characterised Is A Corner The Field Will Find

// RTL 7 - the corner. A link that passes at room temperature and fails hot is
// a link that was never characterised, not a link that broke.
module corner_coverage #(parameter int ROOM_TEMP_IS_ENOUGH = 0) (
  input  logic clk, rst_n,
  input  logic        characterise,
  input  logic [15:0] corners, corners_tested, failures_found, field_units,
  output logic [15:0] tested, untested, coverage_pct, exposed_units,
  output logic        corners_covered,
  output logic [15:0] truly_untested,
  output logic [7:0]  n_runs, n_gapped,
  output logic        corner_gap_err
);
  logic [31:0] p_q, e_q;
  assign tested = (ROOM_TEMP_IS_ENOUGH != 0) ? 16'd1
                : ((corners_tested > corners) ? corners : corners_tested);
  assign truly_untested = (corners > tested) ? (corners - tested) : 16'd0;
  assign untested = truly_untested;
  assign p_q = (corners == 16'd0) ? 32'd100
             : (({16'd0, tested} * 32'd100) / {16'd0, corners});
  assign coverage_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  // Every untested corner is a fraction of the field that may meet it.
  assign e_q = (corners == 16'd0) ? 32'd0
             : (({16'd0, field_units} * {16'd0, untested}) / {16'd0, corners});
  assign exposed_units = (e_q > 32'd65535) ? 16'hFFFF : e_q[15:0];
  // A room-temperature sign-off reports characterisation complete.
  assign corners_covered = (ROOM_TEMP_IS_ENOUGH != 0) ? 1'b1
                         : (truly_untested == 16'd0);
  // Corners never characterised, on a link reported as covered.
  assign corner_gap_err = characterise && (truly_untested != 16'd0)
                          && corners_covered;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_runs <= 8'd0; n_gapped <= 8'd0;
    end else if (characterise) begin
      n_runs <= n_runs + 8'd1;
      if (!corners_covered) n_gapped <= n_gapped + 8'd1;
    end
  end
endmodule

Five characterisations. Twelve corners, ten thousand field units.

Two characterisations with a corner gap; none when room temperature is taken as characterisation.

A link that passes at room temperature and nominal voltage has been characterised at one of twelve corners. The other eleven are not failures — they are unknowns, and the field is a sample of all twelve in proportion.

Row one is the exposure in units rather than in percentages. Eight untested corners of twelve across ten thousand units is six thousand six hundred and sixty-six units that may meet a corner nobody tried. That number is what turns a characterisation gap into a schedule decision.

Row three is the sensitivity. One corner short of complete is eight hundred and thirty-three units — small enough to argue about and large enough to matter, which is precisely the range where the argument happens.

12. RTL 8 — A Speed Backoff Is A Working Link At Less Bandwidth

// RTL 8 - equalisation. A link that will not hold the top speed can run one
// generation down and work perfectly, at half the bandwidth.
module speed_backoff #(parameter int ANY_SPEED_WORKS = 0) (
  input  logic clk, rst_n,
  input  logic        judge,
  input  logic [15:0] target_gen, stable_gen, per_gen_gbps, lanes,
  output logic [15:0] target_gbps, stable_gbps, lost_gbps, achieved_pct,
  output logic        at_target,
  output logic [7:0]  n_judgements, n_backed_off,
  output logic        backoff_hidden_err
);
  logic [31:0] p_q;
  assign target_gbps = target_gen * per_gen_gbps * lanes;
  assign stable_gbps = stable_gen * per_gen_gbps * lanes;
  assign lost_gbps = (target_gbps > stable_gbps)
                     ? (target_gbps - stable_gbps) : 16'd0;
  assign p_q = (target_gbps == 16'd0) ? 32'd100
             : (({16'd0, stable_gbps} * 32'd100) / {16'd0, target_gbps});
  assign achieved_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  // A link at any speed is a working link, so nothing reports the backoff.
  assign at_target = (ANY_SPEED_WORKS != 0) ? 1'b1 : (stable_gen >= target_gen);
  assign backoff_hidden_err = judge && (stable_gen < target_gen) && at_target;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_judgements <= 8'd0; n_backed_off <= 8'd0;
    end else if (judge) begin
      n_judgements <= n_judgements + 8'd1;
      if (!at_target) n_backed_off <= n_backed_off + 8'd1;
    end
  end
endmodule

Five judgements. Sixteen lanes, 4 Gbps a generation.

Two judgements backed off; none when any speed works.

A link that will not hold its top generation drops one and runs perfectly. That is the mechanism working as designed — and it is sixty-four gigabits a second that the system was budgeted to have. Nothing reports it because nothing is wrong.

Row three is the same mechanism twice. Two generations down is sixty percent of the bandwidth, and each step is individually reasonable — which is how a link arrives at gen 3 without any single decision to run it there.

Row five is section 9's missing baseline again. No recorded target means no backoff is detectable, and the two sections share the finding: a degradation is only visible against a written-down plan.

13. RTL 9 — A Link That Drops Twice A Day Needs More Than A Day

// RTL 9 - how long to watch. A link that drops once a day needs more than a
// day of observation before a quiet link means anything.
module observe_time #(parameter int AN_HOUR_IS_ENOUGH = 0) (
  input  logic clk, rst_n,
  input  logic        judge,
  input  logic [15:0] drops_per_day, watched_hours, links_watched,
  output logic [15:0] expected_drops, hours_for_one, hours_for_four, confidence_pct,
  output logic        quiet_means_fixed,
  output logic [7:0]  n_judgements, n_premature,
  output logic        premature_quiet_err
);
  logic [31:0] e_q, h_q, f_q, c_q;
  // Watching several links at once multiplies the observation.
  assign e_q = ({16'd0, watched_hours} * {16'd0, links_watched}
                * {16'd0, drops_per_day}) / 32'd24;
  assign expected_drops = (e_q > 32'd65535) ? 16'hFFFF : e_q[15:0];
  assign h_q = (drops_per_day == 16'd0) ? 32'd0 : (32'd24 / {16'd0, drops_per_day});
  assign hours_for_one = (h_q > 32'd65535) ? 16'hFFFF : h_q[15:0];
  assign f_q = {16'd0, hours_for_one} * 32'd4;
  assign hours_for_four = (f_q > 32'd65535) ? 16'hFFFF : f_q[15:0];
  assign c_q = (expected_drops > 16'd3) ? 32'd95
             : ({16'd0, expected_drops} * 32'd30);
  assign confidence_pct = (c_q > 32'd100) ? 16'd100 : c_q[15:0];
  assign quiet_means_fixed = (AN_HOUR_IS_ENOUGH != 0) ? 1'b1
                           : (expected_drops > 16'd3);
  assign premature_quiet_err = judge && (expected_drops <= 16'd3)
                               && quiet_means_fixed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_judgements <= 8'd0; n_premature <= 8'd0;
    end else if (judge) begin
      n_judgements <= n_judgements + 8'd1;
      if (!quiet_means_fixed) n_premature <= n_premature + 8'd1;
    end
  end
endmodule

Five judgements. A link dropping twice a day.

Three premature judgements; none when an hour is taken as enough.

An intermittent link drop is the hardest thing in this chapter to conclude anything about, and the arithmetic is the whole method. Twice a day is one drop every twelve hours; four expected drops needs forty-eight link-hours, which is twelve hours on four links or two days on one.

Row five is the lever. Watching four links at once turns two days into twelve hours — the observation multiplies with the population, which is the cheapest accelerant available and the one most often not used because the spare boards are in someone else's rack.

Rows three and four are one hour apart and they are the difference between a conclusion and a hope. Without the rate, "we watched it overnight and it was fine" has no content at all.

14. RTL 10 — A Link Diagnosis Assembled

// RTL 10 - a link diagnosis assembled. Everything that must hold before "the
// link came back" is a claim about what it came back as, and for how long.
module link_signoff #(parameter int IT_CAME_BACK = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       came_back,        // the link returned to L0
  input  logic       no_replays,       // no corrected errors either
  input  logic       worst_lane_ok,    // the worst lane has margin, not the mean
  input  logic       full_bandwidth,   // it returned at the planned width and speed
  input  logic       headroom_ok,      // aged and cornered, it stays inside the limit
  input  logic       corners_covered,  // every corner was characterised
  output logic       link_sound,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_sound,
  output logic       false_recovery_err
);
  assign fail_mask[0] = ~came_back;
  assign fail_mask[1] = ~no_replays;
  assign fail_mask[2] = ~worst_lane_ok;
  assign fail_mask[3] = ~full_bandwidth;
  assign fail_mask[4] = ~headroom_ok;
  assign fail_mask[5] = ~corners_covered;
  // The came-back build is what a link-up interrupt reports.
  assign link_sound = (IT_CAME_BACK != 0) ? came_back : (fail_mask == 6'd0);
  assign false_recovery_err = evaluate && link_sound && (fail_mask != 6'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_sound <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (link_sound) n_sound <= n_sound + 8'd1;
    end
  end
endmodule

Seven configurations.

One configuration sound under the full diagnosis; six under the interrupt.

Row four is the one that makes the point without argument. The link came back at forty percent of the bandwidth, so the interrupt fired because the recovery succeeded — it is reporting the successful outcome of the degradation. It is not being fooled.

Rows two, five and six are three different time horizons. Corrected errors are happening now. No headroom is a prediction about three years. An uncharacterised corner is a statement about what has never been tried. All three are invisible to an indicator that samples the present state, and they need three different instruments.

These six are not 26.1's six. That chapter's failures all end with a device that is not there; these all end with a device that is there and is worse than it should be — which is why they survive bring-up and appear as performance complaints months later.

Figure 4 — Corrected errors are asked first because they are happening now and are already counted; the corner question is asked last because it is the only one whose answer is about experiments nobody has run. Every exit above is a link reporting up.

15. Quantitative Reasoning

Retrain loops. Twenty retrains at 12 ms in a one-second window is 240 ms down and 48 Gbps of 64 — a quarter gone, with the status register reading up on every sample.

Replay cost. Twelve hundred CRC errors at 32 flits each is 38,400 flits replayed — 76% of the traffic — and none of it reaches an uncorrectable-error counter.

Worst lane. Fifteen lanes at 94 mV and one at 30 averages to 90 against a requirement of 50 — and the link runs at 30, twenty short.

Recovery cost. Forty excursions at 1,000 cycles each is 40,000 of a 50,000-cycle window — 80%; one excursion is exactly the two percent line.

Degraded return. Eight lanes of sixteen at one generation down is 40% of the bandwidth; fifteen lanes of sixteen is 93% and still not as planned.

Headroom. A measured rate of 20 against a limit of 100 is five times of headroom today and 120% of the limit once aged twice and cornered three times.

Corners. Four of twelve characterised is 33% covered and 6,666 field units exposed; one corner short is 833.

Speed backoff. Gen 5 to gen 4 on sixteen lanes is 64 Gbps lost of 320 — 80% achieved; two generations is 60%.

Observation. A link dropping twice a day needs 48 link-hours for four expected drops — twelve hours on four links, or two days on one.

The assembled diagnosis. Six properties, seven configurations, one sound. The interrupt called six sound.

16. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Three checks ran before the mutation campaign, and the first one caught something a reading had missed.

The mutual-exclusivity check is now a script rather than a reading. 25.7 introduced it and 26.1 passed ten for ten by eye. This chapter shipped a dead error signal past a visual pass — section 11's corner_gap_err required corners > tested and corners_covered, and corners_covered is (corners − tested) == 0. Listing the expressions is not the check; substituting the definitions is, and a fifty-line script does it mechanically.

The scripted output-listing step reported twenty-eight unasserted nets and four were real — an up-time, an available-cycles count and two corner-coverage figures, every one of them the broken build's own claim.

And every published code block is generated from the verified source file rather than pasted, so the two cannot drift. 26.1 §16 found a block that had; this chapter makes that failure impossible rather than detectable.

Alongside those: every inclusive threshold at exactly equal, every clamp driven past its cap, and both builds asserted on every degenerate case.

Retrain loops. Losses are driven at exactly five percent — which needs a 100 Gbps link, because on 64 Gbps the integer arithmetic skips from four to six.

chk(tGl == 16'd5,   "exactly five percent lost");
chk(tGk == 1'b1,    "which is exactly usable");

Replay cost. An uncorrectable error with no corrected errors behind it is driven, where both views agree the link is unhealthy.

Worst lane. The requirement is driven at exactly fifty millivolts and at forty-nine.

Recovery cost. One excursion is driven at exactly two percent and two at four, and an excursion count with no measured LTSSM cost is driven — which killed one survivor.

Degraded return. Fifteen lanes of sixteen is driven at 93 percent, which is what separates a hundred-percent bar from a ninety-percent one.

chk(gGb == 16'd93, "ninety-three percent of the bandwidth");
chk(gGk == 1'b0,   "which is not as planned - the bar is a hundred");

Headroom. A worst case at exactly the limit is constructed from a measured rate of sixteen against a limit of ninety-six.

Corners. More corners tested than exist is driven, exercising the clamp.

Speed backoff. A link running above its target is driven, where the achieved share clamps at a hundred.

Observation. Exactly four expected drops is driven, three is driven, and the confidence percentage is asserted at both — which killed the last survivor.

The assembled diagnosis. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 306 checks across two testbenches, 163 on the front five models and 143 on the back five, all passing on the unmutated sources.

17. Mutation Testing

Seventy-three mutations were injected one at a time. 73 injected, 73 killed, after six survivors — five of them undriven cases and one of them a mutation that did not compile.

Four of the five surviving cases were the same shape: a guard or a threshold whose boundary the testbench approached and never landed on. Exactly five percent, exactly one excursion's guard, ninety-three percent, three expected drops. Each was one stimulus line.

One mutation did not compile, and the harness reported it as COMPILE-ERROR rather than as a kill. The replacement text left the tail of a ternary behind. That is the same class as batch 024's unmatched anchors — a mutation that was never validly injected is not a mutation that was killed, and a harness that counted it would have reported 73/73 while testing 72.

Reaching exactly five percent needed the link's nominal rate changed from 64 to 100 Gbps. On 64, the integer arithmetic steps from four percent to six and five is unreachable — so the boundary the rule is written on could not be driven at all until the stimulus was re-parameterised. A threshold that no input can produce is a threshold that has never been tested, and it is worth checking that a boundary is reachable before assuming a case will reach it.

18. Verification Strategy

What a testbench for a degradation model must cover.

Make the mutual-exclusivity check a script. This chapter is the evidence: 26.1 passed a visual pass ten for ten, and 26.2 shipped a dead error signal past the same pass. Substituting the definitions is mechanical and a reading is not.

Generate published code blocks from the verified source. 26.1 detected a drifted block; building the chapter by concatenating the simulated files makes drift impossible.

Check that a threshold's boundary is reachable before assuming a case reaches it. Five percent was unreachable on a 64 Gbps link — the integer arithmetic steps over it — and the boundary case had to be re-parameterised to exist at all.

Drive the guard whose input is a missing measurement. §5's recovery time and §8's LTSSM cost are both counts without their cost, and in both the model must report a missing measurement rather than a healthy link.

The cases where the indicator is right. A link with no retrains. An uncorrectable error with no corrections. A worst lane above the requirement. A full-width, full-speed recovery. A rate too low to measure. A link running above target. Six exemptions across nine models.

Counters as a second signature. Ten models, ten pairs of totals, differing in eight. Two are deliberately equal — §6's unhealthy count in the uncorrectable case, and §11's gap count once the report is separated from the truth.

What a real investigation needs that these models do not have. A two-dimensional eye for §7, a non-multiplicative ageing model for §10, and a structured intermittent for §13. All three are abbreviations that preserve the conclusion, and section 26 exercises 3, 6 and 9 are where they come back.

19. Synthesis and Implementation Reality

The link status register is a snapshot and the LTSSM is a history. Section 5 exists because the two are read as if they were the same thing — a register sampled once a second in a link retraining twenty times a second is correct on every read and describes almost none of the window.

Correctable error counters exist and are almost never polled. The replay count in section 6 is in the device's own error registers, and a health check that reads only the uncorrectable status is reading the smaller of two numbers.

Lane margining is a standard capability and it produces per-lane results. Section 7's mistake happens in the summary, not the measurement — sixteen numbers arrive and one number is reported, and the one reported is usually a mean.

Negotiated width and speed are in the link status register alongside the up bit. Section 9's degradation is one register read away from being visible, and the reason it is not read is that the up bit is in the same register and answers the question people ask.

Ageing and temperature dependence are characterisation, not debug. Sections 10 and 11 are decisions made months before any failure, and a link with no corner data is not a link that failed — it is a link nobody asked the question about.

Watching several links at once is the cheapest accelerant in section 13, and the constraint is usually organisational rather than technical: the spare boards exist and belong to somebody else.

20. Silicon Observability

"Planned width and speed, recorded as a device attribute" is the cheapest entry and it enables two others. Sections 9 and 12 are both invisible without it — a degradation is only detectable against a written-down baseline — and the baseline is a constant that somebody has to decide to store.

21. Debug Lab

Symptom. A CXL memory expander in a customer's fleet reports occasional link-up interrupts — a few a day, across a few hundred of ten thousand units. No errors are reported, the devices work, and a support engineer closes the tickets because the links recover.

Step 1 — what does the interrupt mean? Section 14. It means the link came back. The first question is what it came back as, and the negotiated width and speed are in the same register as the up bit. They read eight lanes, gen 4 against a planned sixteen and gen 5.

Step 2 — how much is that? Section 9. Fifty percent of the lanes at eighty percent of the speed is forty percent of the bandwidth, on devices reporting healthy. Nobody had complained about performance because nobody had a baseline to complain against — section 20's planned-width attribute did not exist.

Step 3 — is it a loop? Section 5. Accumulated time in Recovery is added and read back over a day: the affected units are in Recovery about four percent of the time, which is a retrain every few seconds rather than the few-a-day the interrupt count suggested. The interrupt was coalescing.

Step 4 — the correctable errors. Section 6. The replay counters on the affected units show a rate roughly two decades above the healthy population, and the per-lane breakdown puts almost all of it on one lane.

Step 5 — the lane. Section 7. Per-lane margining on a returned unit shows fifteen lanes near ninety millivolts and lane 11 at thirty-one, against a fifty-millivolt requirement. The mean is ninety-one and the summary in the manufacturing test report is the mean.

Step 6 — why these units. Section 11. The corner characterisation covered room temperature and one voltage corner of twelve. The affected units are in a part of the fleet with a warmer inlet — a corner nobody had run, and at that corner lane 11's margin is not marginal but insufficient.

The finding. One marginal lane, visible in manufacturing data that was averaged away, producing a retrain loop at a rate the interrupt count understated by two orders of magnitude, costing sixty percent of the bandwidth on the affected units, at a temperature corner that was never characterised.

The fix. For the fleet: screen on per-lane margin rather than the mean, and re-test at the warm corner. For the product: record the planned width and speed as a device attribute so a degraded return is detectable, latch the negotiated values at every recovery, and report accumulated Recovery time rather than interrupt counts — because the interrupt count was the number that made this look like a few units a day.

What made this hard. Every link came back, every device worked, and every number that would have shown the problem existed — in a manufacturing report that reported a mean, in a register nobody read, and in a corner nobody ran.

22. Design Review

1. What are the planned width and speed, and where are they recorded? Sections 9 and 12 are invisible without a baseline. Section 20.

2. What did the link negotiate at the last recovery? Same register as the up bit. Sections 9 and 19.

3. What is the accumulated time in Recovery, not the entry count? A snapshot cannot show a loop. Sections 5 and 19.

4. What is the correctable error rate, and is it one lane? The number a clean-link check does not read. Sections 6 and 20.

5. Is per-lane margin reported, or a mean? Sixteen numbers arrive and one is reported. Sections 7 and 19.

6. What is the error rate once aged and cornered, against the limit? Today's number is not the claim. Section 10.

7. Which corners were characterised, and what is the field exposure of the rest? A list, not a percentage. Section 11.

8. What is the measured drop rate per unit-hour? Without it, no quiet period means anything. Section 13.

9. How many links can be watched at once? The cheapest accelerant, usually blocked organisationally. Sections 13 and 19.

10. What does a link-up interrupt establish? Section 14 exists because the answer is the last property only.

23. How This Appears In Real Engineering

A support engineer sees link-up interrupts and devices that work, and closes the ticket. That is a reasonable reading of the only evidence presented, and section 21 is what it costs.

A hardware engineer owns the margin data and reports a summary because a summary is what a report format wants. Section 7 happens in the reporting, not the measuring.

A firmware engineer owns whether the negotiated width is latched and whether Recovery time is accumulated. Both are small and both are decided before anybody needs them — which is section 20's whole argument.

A reliability engineer owns sections 10 and 11 and is usually the only person who thinks in terms of three-year numbers. The rest of the organisation reads today's measurement, and the two are different claims.

24. Common Misconceptions

"The link came back, so it recovered." At forty percent of the bandwidth (section 9), one generation down, after a thousand cycles per excursion. The interrupt reports the recovery succeeded, which is true and is not the question.

"There are no errors." No uncorrectable errors. Twelve hundred corrected ones cost 38,400 replayed flits (section 6), and they are in a counter nobody polls.

"The margin is fine — the average is ninety." The link runs at thirty (section 7). A serial link is a conjunction; an average describes a quantity it does not have.

"It's inside the specification limit." Today, at this temperature, on new hardware. Aged twice and cornered three times it is at 120% of the limit (section 10), and that is the claim a three-year deployment is making.

"It passed characterisation." At one corner of twelve (section 11) — 6,666 field units of ten thousand may meet a corner nobody ran.

"We watched it overnight and it was quiet." At two drops a day, overnight on one link is two expected drops and 60% confidence (section 13). Four links for twelve hours is the same evidence in a quarter of the time.

25. Interview Reasoning

"A link-up interrupt fires a few times a day. What do you look at?" The negotiated width and speed — in the same register as the up bit — and the accumulated Recovery time rather than the interrupt count. A candidate who asks how often it is dropping has asked the second question; the first is what it is coming back as.

"Correctable errors are zero and the link keeps retraining. What's going on?" Check whether the counter being read is the uncorrectable one. Corrected errors and replays are separate counters (section 6), and a link that retrains has a replay history.

"Lane margin averages ninety millivolts against a fifty-millivolt requirement. Are you happy?" Not without the per-lane numbers. A link runs at its worst lane (section 7), and a mean of ninety is consistent with one lane at thirty.

"The error rate is a decade inside the limit. Is that enough headroom?" For how long, and at what temperature? Two-times ageing and a three-times corner factor eats a decade and a half (section 10). The follow-up worth asking back: what does the limit assume about the deployment life?

"How long do you watch a link before you believe an intermittent is fixed?" A question about rate and population. Four expected drops (section 13) — at two drops a day, forty-eight link-hours, which is twelve hours on four links. A candidate who names a duration has not asked what the rate is.

26. Exercises

1. Separate the retrain count from the retrain cost. Given an interrupt count and an accumulated Recovery time, derive the per-retrain cost and say which of the two a device should report if it can only report one.

2. Price the replay. For a link at 24,000 corrected errors per million flits with 32 flits of replay each, compute the effective bandwidth and compare it against the same link with a 2% retrain loop. Which costs more?

3. Model the eye. §7 uses a millivolt figure. Replace it with a two-dimensional eye — voltage and timing margin — and state what a single scalar loses.

4. Find the threshold. §5's five-percent line and §8's two-percent line are policy. Derive each from a system requirement rather than asserting it, and say what changes if the requirement is latency rather than bandwidth.

5. Write the baseline. Design the device attribute that records planned width and speed, and state what reads it and when. What happens on a device whose planned width legitimately changes?

6. Age the rate properly. §10 multiplies. Build a model where ageing and temperature interact rather than compose, and re-derive the three-year worst case for a measured rate of 20 against a limit of 100.

7. Plan the corner matrix. For temperature, voltage and process, enumerate the corners and rank them by field exposure. Which four would you run if you could only run four?

8. Detect a backoff. Write the check that distinguishes a link that trained at gen 4 because it could not hold gen 5 from one that was configured for gen 4, using only registers.

9. Size an observation campaign. For drop rates of 2, 0.5 and 0.1 per day on populations of 1, 4 and 40 links, tabulate the hours for four expected drops. Where does the population stop helping?

10. Add the seventh property. Propose one none of §14's six implies, name its section, and construct the configuration where the six hold and it fails. A property that cannot fail alone is not a seventh property.

27. Summary

Coming back is not the same as being back. A link-up interrupt reports one fact accurately — the recovery succeeded — and the four numbers that decide whether the link is usable are in registers nobody read.

A retrain loop spends the link's bandwidth on recovery. Twenty retrains at twelve milliseconds is 240 milliseconds of a one-second window, and a status register sampled once per window is correct on every read and describes none of it.

A corrected error costs a round trip. Twelve hundred of them at thirty-two flits each is 38,400 flits replayed — seventy-six percent of the traffic — and a health check that counts only uncorrectables reports the link clean.

A link runs at its worst lane. Fifteen lanes at ninety-four millivolts and one at thirty averages to ninety against a requirement of fifty. The link runs at thirty, and the number that reaches the report is the mean.

Every excursion out of L0 costs a thousand cycles. Forty of them in fifty thousand is eighty percent of the link's time, and one is exactly the two percent line — so the difference between a link somebody investigates and one nobody does is a single event.

And it can come back narrower. Eight lanes of sixteen at one generation down is forty percent of the bandwidth; fifteen lanes of sixteen is ninety-three percent and is still not as planned, because the lane that went will take the others eventually.

Inside the limit today is not inside it in three years. Five times of headroom aged twice and cornered three times is a hundred and twenty percent of the limit, on hardware that passes every test now.

And a corner nobody characterised is a corner the field runs. Four of twelve is 6,666 units of ten thousand exposed to an experiment that was never done.

A speed backoff is a working link at less bandwidth. Gen 5 to gen 4 is sixty-four gigabits a second, arrived at by a mechanism doing exactly what it should, reported by nothing.

And at two drops a day, four expected drops costs forty-eight link-hours — twelve on four links or two days on one. Without the rate, "we watched it overnight" has no content.

The mutual-exclusivity check became a script because a reading missed one. 26.1 passed ten for ten by eye and this chapter shipped a dead error signal past the same pass — corner_gap_err required a gap and a report of no gap, which is a conjunction that cannot be true. Listing expressions is not substituting definitions.

And every code block in this chapter was generated from its simulated source file rather than pasted, which makes 26.1's drifted-block defect impossible rather than merely detectable.

One mutation did not compile and the harness said so. Like batch 024's unmatched anchors, a mutation that was never validly injected is not a mutation that was killed — 73 of 73 is only a score because the harness distinguishes the two.

And exactly five percent was unreachable until the stimulus was re-parameterised. On a 64 Gbps link the integer arithmetic steps from four percent to six, so the boundary the rule is written on could not be produced by any input — which is worth checking before assuming a case will reach it.

"It came back" is one property of six. The interrupt called six of seven configurations sound when one was — and §21 is a fleet of devices whose links all came back, at forty percent of the bandwidth, retraining every few seconds, because of one lane that a manufacturing report averaged away and a temperature corner nobody ran.

26.3 — Coherency Bugs moves above the link. Everything here is about a link that carries fewer bits than it should; the next chapter is about a link that carries the right bits and delivers the wrong data.