Discovery Failures

Module 25 asked, six times, what a green report establishes. Module 26 starts from a failure that is already happening and asks a different question: given what the trace shows, what is still possible?

The first thing every discovery investigation is told is that the link is up. That is a real observation and it is worth exactly one thing: it removes the physical-layer causes. Fourteen of thirty-six, leaving twenty-two — and a week gets spent on the fourteen anyway, because "the link is up" reads as reassurance rather than as evidence.

1. The Engineering Problem — An Observation Is Worth What It Removes

The last stage reached clears everything before it. Eight discovery stages with five known good and six causes behind each is eighteen candidates of forty-eight — sixty-two percent narrowed, from one observation. Section 5.

Trained is not enumerated. A link at L0 rules out fourteen physical causes and none of the twenty-two protocol ones, and the two families do not overlap. Section 6.

All-ones is not a register value. Twenty-four reads of forty returning every bit set is twenty-four addresses nothing responded to, not twenty-four values worth decoding. Section 7.

Without the DVSEC the host enumerates a working PCIe endpoint. Eighteen CXL features lost of thirty, and the enumeration reports success, because it is one. Section 8.

And a device ready at 140 ms against a 100 ms window is never found. Not intermittently — every boot, and the link trains perfectly first. Section 10.

This chapter against Module 25, stated precisely. Those chapters own whether a check could have seen a bug. This one owns what a bug that is already visible narrows to — which is why every model here converts an observation into a candidate count, and why section 14's weak definition is a link status register reading up.

2. The One-Sentence Model

A CXL device is discovered when the link reaches L0, configuration reads find a responder, the DVSEC is present, the alternate protocol was agreed rather than fallen back from, the device answered inside the host's window, and the trained width is the advertised one — and "the link is up" is one of those six.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model takes one observation from a discovery trace and computes what it removes. A real investigation has a protocol analyser, a host log, a device's own status registers and somebody's memory of a similar bug, and none of that is reproduced. What is reproduced is the arithmetic of narrowing — candidates before, candidates after, and what the difference cost to obtain.

Three simplifications are worth stating. Section 5 treats causes as evenly distributed across stages, which they are not. Section 10 models boot-to-boot variation as a single margin figure rather than a distribution. Section 13 uses a flat failure rate where a real intermittent has structure. In each case the conclusion is the same and the model is abbreviated.

Each model is built twice — a correct build and a broken build selected by a parameter. Every broken build here is a reading of the trace that stops too early: the link is up so it works, all-ones is data, PCIe enumeration is enumeration, a fallback is a success, the host will wait, the width is what was advertised. Each is what the most visible indicator says, which is why each survives a status meeting.

Figure 1 — Both observations are true. One of them removes thirty candidates and the other removes none, and the second is the one that gets reported first because it is the one with a register behind it.

5. RTL 1 — The Last Stage Reached Clears Everything Before It

// RTL 1 - where it stopped. Discovery is a sequence of stages, and the last
// stage reached removes every cause that lives after it.
module stage_reached #(parameter int ONLY_LINK_UP = 0) (
  input  logic clk, rst_n,
  input  logic        observe,
  input  logic [15:0] total_stages, last_ok, causes_per_stage,
  output logic [15:0] cleared_stages, suspect_stages, candidates, narrowed_pct,
  output logic        narrowed,
  output logic [7:0]  n_observations, n_wide,
  output logic        no_narrowing_err
);
  logic [31:0] c_q, p_q, all_q;
  // Everything before the last good stage is ruled out.
  assign cleared_stages = (last_ok > total_stages) ? total_stages : last_ok;
  assign suspect_stages = (total_stages > cleared_stages)
                          ? (total_stages - cleared_stages) : 16'd0;
  assign all_q = {16'd0, total_stages} * {16'd0, causes_per_stage};
  // Knowing only that the link is up clears nothing past the first stage.
  assign c_q = (ONLY_LINK_UP != 0) ? all_q
             : ({16'd0, suspect_stages} * {16'd0, causes_per_stage});
  assign candidates = (c_q > 32'd65535) ? 16'hFFFF : c_q[15:0];
  assign p_q = (all_q == 32'd0) ? 32'd0
             : (((all_q - c_q) * 32'd100) / all_q);
  assign narrowed_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  assign narrowed = (candidates < all_q[15:0]);
  // Stages known good that still leave every cause in play.
  assign no_narrowing_err = observe && (cleared_stages != 16'd0) && !narrowed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_observations <= 8'd0; n_wide <= 8'd0;
    end else if (observe) begin
      n_observations <= n_observations + 8'd1;
      if (!narrowed) n_wide <= n_wide + 8'd1;
    end
  end
endmodule

Five observations. Eight stages, six causes behind each.

Two observations that narrow nothing; all five when only the link state is known.

Discovery is ordered, which is the most useful thing about it. Link training, then configuration access, then capability enumeration, then the CXL-specific structures, then the device's own readiness. A trace showing stage five completed removes every cause in stages one to five — not because those stages are simple, but because they visibly worked.

Row one is the arithmetic of a first look at a trace. Thirty candidates gone for the price of reading which transactions completed. That is the highest-value hour in any discovery investigation, and it happens before any equipment is connected.

Row three is the case with no leverage and it is worth naming. Nothing reached at all leaves all forty-eight in play, and the model reports that honestly rather than manufacturing a narrowing. An investigation that starts here needs a different first measurement — section 12's question — rather than a more careful reading of the same trace.

6. RTL 2 — Trained And Enumerated Are Different Families

// RTL 2 - trained is not enumerated. A link that reaches L0 has passed the
// physical stages and none of the protocol ones, and the two failure sets are
// disjoint.
module link_vs_enum #(parameter int LINK_IS_ENOUGH = 0) (
  input  logic clk, rst_n,
  input  logic        diagnose,
  input  logic [15:0] phys_causes, proto_causes, link_at_l0, cfg_readable,
  output logic [15:0] live_causes, ruled_out, remaining_pct, wrong_family,
  output logic        family_known,
  output logic        physical_ruled_out,
  output logic [7:0]  n_diags, n_unknown,
  output logic        wrong_family_err
);
  logic [31:0] p_q, t_q;
  assign t_q = {16'd0, phys_causes} + {16'd0, proto_causes};
  // L0 rules out the physical family entirely.
  assign physical_ruled_out = (link_at_l0 != 16'd0);
  assign ruled_out = (LINK_IS_ENOUGH != 0) ? 16'd0
                   : (physical_ruled_out ? phys_causes : 16'd0);
  assign live_causes = (t_q[15:0] > ruled_out) ? (t_q[15:0] - ruled_out) : 16'd0;
  assign p_q = (t_q == 32'd0) ? 32'd0
             : (({16'd0, live_causes} * 32'd100) / t_q);
  assign remaining_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  // Time spent on the physical family after L0 was observed.
  assign wrong_family = (physical_ruled_out && (ruled_out == 16'd0))
                        ? phys_causes : 16'd0;
  assign family_known = (ruled_out != 16'd0) || !physical_ruled_out;
  assign wrong_family_err = diagnose && (wrong_family != 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_diags <= 8'd0; n_unknown <= 8'd0;
    end else if (diagnose) begin
      n_diags <= n_diags + 8'd1;
      if (!family_known) n_unknown <= n_unknown + 8'd1;
    end
  end
endmodule

Four diagnoses. Fourteen physical causes, twenty-two protocol ones.

The family is unknown twice with the L0 observation used; three times when it is not.

"The link is up" and "the host sees the device" are different claims about different layers, and the failure sets are disjoint. L0 means the electricals, the training sequences and the lane deskew all worked — fourteen causes gone. It says nothing whatever about whether a configuration read will be answered.

Row one is the week that gets spent. The link-is-enough reading leaves all thirty-six live and sends the investigation at the fourteen it has already disproved — signal integrity, retimer configuration, cable seating — because those are the causes with instruments attached to them.

Row three is the subtler failure and it is why family_known exists. L0 observed, but no physical causes were listed to rule out — so the observation cleared an empty set and established nothing. A narrowing against an empty candidate list is not a narrowing, and the model says so rather than reporting a hundred percent of nothing.

7. RTL 3 — All-Ones Is The Absence Of A Responder

// RTL 3 - all-ones is an answer. A configuration read that returns every bit
// set is not a register value, it is the absence of a responder.
module cfg_read #(parameter int ALL_ONES_IS_DATA = 0) (
  input  logic clk, rst_n,
  input  logic        read_it,
  input  logic [15:0] reads, all_ones_reads, real_reads, decoded_as_valid,
  output logic [15:0] responded, unresponded, misread, responded_pct,
  output logic        responder_present,
  output logic [7:0]  n_reads, n_absent,
  output logic        all_ones_misread_err
);
  logic [31:0] p_q;
  // An all-ones read means nothing claimed the address.
  assign unresponded = (ALL_ONES_IS_DATA != 0) ? 16'd0 : all_ones_reads;
  assign responded = (reads > unresponded) ? (reads - unresponded) : 16'd0;
  assign misread = (ALL_ONES_IS_DATA != 0)
                   ? ((decoded_as_valid > all_ones_reads) ? all_ones_reads
                      : decoded_as_valid) : 16'd0;
  assign p_q = (reads == 16'd0) ? 32'd100
             : (({16'd0, responded} * 32'd100) / {16'd0, reads});
  assign responded_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  assign responder_present = (unresponded == 16'd0);
  // All-ones decoded as a register value.
  assign all_ones_misread_err = read_it && (misread != 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_reads <= 8'd0; n_absent <= 8'd0;
    end else if (read_it) begin
      n_reads <= n_reads + 8'd1;
      if (!responder_present) n_absent <= n_absent + 8'd1;
    end
  end
endmodule

Five read sets. Forty reads unless stated.

Three read sets with no responder; none when all-ones is taken as data.

A PCIe configuration read that nothing claims returns all ones by construction — the root complex synthesises it when no completer answers. It is not a value that happens to be 0xFFFFFFFF; it is the wire protocol's way of saying nobody was there, and every field decoded out of it is fabricated.

Row one is the shape of the wasted day. Twenty-four addresses with no responder, eight of them decoded into plausible-looking capability structures — a vendor ID of 0xFFFF, a capability pointer of 0xFF — and an investigation that follows them is chasing bits the host invented.

Row three is the detector's sensitivity. One all-ones read is enough to say the responder is absent at that address, and it is enough for a decoder to produce one wrong structure. The rule is a comparison, not a judgement: any read returning every bit set is an absence, always.

8. RTL 4 — Without The DVSEC The Host Enumerates A PCIe Device

// RTL 4 - the DVSEC. Without the CXL capability structure a host enumerates a
// perfectly working PCIe endpoint and nothing else.
module dvsec_present #(parameter int PCIE_IS_ENOUGH = 0) (
  input  logic clk, rst_n,
  input  logic        enumerate,
  input  logic [15:0] cxl_features, pcie_features, dvsec_found,
  output logic [15:0] usable, lost_features, degraded_pct, reported_ok,
  output logic        cxl_visible,
  output logic [7:0]  n_enums, n_degraded,
  output logic        silent_fallback_err
);
  logic [31:0] p_q;
  assign cxl_visible = (dvsec_found != 16'd0);
  assign usable = cxl_visible ? (cxl_features + pcie_features) : pcie_features;
  assign lost_features = cxl_visible ? 16'd0 : cxl_features;
  assign p_q = ((cxl_features + pcie_features) == 16'd0) ? 32'd100
             : (({16'd0, usable} * 32'd100)
                / ({16'd0, cxl_features} + {16'd0, pcie_features}));
  assign degraded_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  // A PCIe-only enumeration reports success, because it is one.
  assign reported_ok = (PCIE_IS_ENOUGH != 0) ? 16'd1
                     : (cxl_visible ? 16'd1 : 16'd0);
  assign silent_fallback_err = enumerate && (lost_features != 16'd0)
                               && (reported_ok != 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_enums <= 8'd0; n_degraded <= 8'd0;
    end else if (enumerate) begin
      n_enums <= n_enums + 8'd1;
      if (!cxl_visible) n_degraded <= n_degraded + 8'd1;
    end
  end
endmodule

Five enumerations. Twelve PCIe features unless stated.

Four enumerations degraded — the same four either way, because the DVSEC is there or it is not.

This is the failure that looks most like success. The device enumerates. It appears in the host's device list. It has a vendor ID, a class code, BARs that map. Everything a PCIe enumeration is supposed to produce, it produced — and the eighteen CXL features are not reported missing, because nothing asked for them.

Row one is why it reaches a customer. Forty percent of the feature set, and the only symptom is that the device does not appear as a CXL device — which requires somebody to look for a CXL-specific entry rather than for the device. A bring-up checklist that says "device enumerates" ticks.

Row three is the exemption that keeps the check honest. A device with no CXL features to lose is a PCIe device, correctly enumerated, and the model raises nothing. The error is specifically about features that exist and were not reached.

9. RTL 5 — A Failed Negotiation Falls Back Rather Than Failing

// RTL 5 - alternate protocol negotiation. The CXL handshake happens inside
// link training, and a failure there falls back to PCIe rather than failing.
module altproto_nego #(parameter int FALLBACK_IS_FINE = 0) (
  input  logic clk, rst_n,
  input  logic        train_it,
  input  logic [15:0] attempts, cxl_agreed, ts_mismatches, bandwidth_gbps,
  output logic [15:0] as_cxl, as_pcie, agreed_pct, lost_gbps,
  output logic        negotiated,
  output logic [7:0]  n_trainings, n_fallback,
  output logic        silent_downgrade_err
);
  logic [31:0] p_q;
  assign as_cxl = (cxl_agreed > attempts) ? attempts : cxl_agreed;
  assign as_pcie = (attempts > as_cxl) ? (attempts - as_cxl) : 16'd0;
  assign p_q = (attempts == 16'd0) ? 32'd100
             : (({16'd0, as_cxl} * 32'd100) / {16'd0, attempts});
  assign agreed_pct = (p_q > 32'd100) ? 16'd100 : p_q[15:0];
  // A link that fell back still carries traffic, at the PCIe feature set.
  assign lost_gbps = (as_pcie != 16'd0) ? bandwidth_gbps : 16'd0;
  // Falling back is a working link, so nothing reports it as a failure.
  assign negotiated = (FALLBACK_IS_FINE != 0) ? 1'b1 : (as_pcie == 16'd0);
  assign silent_downgrade_err = train_it && (ts_mismatches != 16'd0)
                                && (as_pcie != 16'd0) && negotiated;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_trainings <= 8'd0; n_fallback <= 8'd0;
    end else if (train_it) begin
      n_trainings <= n_trainings + 8'd1;
      if (!negotiated) n_fallback <= n_fallback + 8'd1;
    end
  end
endmodule

Five training sets. Forty attempts unless stated.

Three training sets with a fallback; none when a fallback is accepted.

The CXL handshake lives inside PCIe link training, in the modified TS1 and TS2 ordered sets, and a device and host that fail to agree still train. They train as PCIe. The link comes up, the device works, and every CXL-specific capability is absent — which is section 8's symptom arriving through a different cause.

Row four is the row that makes the investigation hard. Eight links fell back and no TS mismatch was recorded, because the analyser was not capturing during training or the counter does not exist. The fallback is visible; its cause is not, and the model reports the first without inventing the second.

Row one is what the fallback costs. Not a dead link — a working PCIe link, at the PCIe feature set, with sixty-four gigabits a second of CXL-specific capability simply not there. It is the most expensive silent success in the chapter.

10. RTL 6 — A Device Outside The Window Is Never Found

// RTL 6 - the readiness window. A device that is not ready within the host's
// window is a device the host stops looking for.
module ready_window #(parameter int HOST_WAITS_FOREVER = 0) (
  input  logic clk, rst_n,
  input  logic        boot_it,
  input  logic [15:0] window_ms, device_ready_ms, margin_needed, boots,
  output logic [15:0] margin, late_by, fail_rate_pct, missed_boots,
  output logic        ready_in_time,
  output logic [7:0]  n_boots, n_missed,
  output logic        window_ignored_err
);
  logic [31:0] f_q;
  assign margin = (window_ms > device_ready_ms)
                  ? (window_ms - device_ready_ms) : 16'd0;
  assign late_by = (device_ready_ms > window_ms)
                   ? (device_ready_ms - window_ms) : 16'd0;
  // A host that waits forever never misses; a real one gives up.
  assign ready_in_time = (HOST_WAITS_FOREVER != 0) ? 1'b1 : (late_by == 16'd0);
  // A margin below what the variation needs fails some fraction of boots.
  assign fail_rate_pct = ready_in_time
                         ? ((margin >= margin_needed) ? 16'd0
                            : (16'd100 - ((margin * 16'd100) / ((margin_needed == 16'd0)
                               ? 16'd1 : margin_needed))))
                         : 16'd100;
  assign f_q = ({16'd0, boots} * {16'd0, fail_rate_pct}) / 32'd100;
  assign missed_boots = (f_q > 32'd65535) ? 16'hFFFF : f_q[15:0];
  assign window_ignored_err = boot_it && (late_by != 16'd0) && ready_in_time;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_boots <= 8'd0; n_missed <= 8'd0;
    end else if (boot_it) begin
      n_boots <= n_boots + 8'd1;
      if (!ready_in_time) n_missed <= n_missed + 8'd1;
    end
  end
endmodule

Five boot analyses. A 100 ms window, a thousand boots, 20 ms of variation.

One boot analysis not ready in time; none when the host waits forever.

A host gives a device a bounded time to become ready and then stops looking. Forty milliseconds late is not an intermittent failure — it is every boot, deterministically, on a link that trained perfectly and a device that works correctly forty milliseconds later.

Rows three, four and five are the margin, and it is the part that produces intermittency. Ready exactly at the window is not late and is at a hundred percent risk, because any boot-to-boot variation at all pushes it over. Twenty milliseconds of margin against twenty of variation is exactly enough; nineteen is five percent — fifty boots of a thousand, which is the failure that gets called "flaky hardware."

Row one's broken build is worth reading closely. The waits-forever model reports ready in time while its own margin arithmetic still says every boot is at risk — two outputs from the same model disagreeing, which is what a status bit and a measurement look like when only one of them is being read.

11. RTL 7 — The Trained Width Is Not The Advertised One

// RTL 7 - lane configuration. A device advertising a width the slot cannot
// give trains at the width they agree on, or does not train at all.
module lane_config #(parameter int WIDTH_ALWAYS_AGREES = 0) (
  input  logic clk, rst_n,
  input  logic        train_it,
  input  logic [15:0] device_lanes, slot_lanes, bifurcation_lanes, per_lane_gbps,
  output logic [15:0] agreed_lanes, lost_lanes, agreed_gbps, expected_gbps,
  output logic        width_as_planned,
  output logic [7:0]  n_trainings, n_narrow,
  output logic        width_ignored_err
);
  logic [15:0] slot_actual;
  // Bifurcation splits the slot, so the usable width is the smaller again.
  assign slot_actual = (bifurcation_lanes != 16'd0)
                       ? ((bifurcation_lanes > slot_lanes) ? slot_lanes
                          : bifurcation_lanes)
                       : slot_lanes;
  assign agreed_lanes = (WIDTH_ALWAYS_AGREES != 0) ? device_lanes
                      : ((device_lanes > slot_actual) ? slot_actual : device_lanes);
  assign lost_lanes = (device_lanes > agreed_lanes)
                      ? (device_lanes - agreed_lanes) : 16'd0;
  assign agreed_gbps = agreed_lanes * per_lane_gbps;
  assign expected_gbps = device_lanes * per_lane_gbps;
  assign width_as_planned = (lost_lanes == 16'd0);
  // A narrower link than the device advertised, reported at full width.
  assign width_ignored_err = train_it && (device_lanes > slot_actual)
                             && width_as_planned;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_trainings <= 8'd0; n_narrow <= 8'd0;
    end else if (train_it) begin
      n_trainings <= n_trainings + 8'd1;
      if (!width_as_planned) n_narrow <= n_narrow + 8'd1;
    end
  end
endmodule

Five trainings. A sixteen-lane device at 4 Gbps a lane unless stated.

Two trainings narrower than advertised; none when the width is assumed to agree.

A width mismatch is not a discovery failure on its own — it is a performance failure that arrives disguised as one. The device enumerates, works, and delivers half its bandwidth, and the first symptom is a benchmark rather than an absence.

Row three is the one that costs the most time. A sixteen-lane slot bifurcated in platform configuration presents eight lanes, so the physical slot and the trained width disagree for a reason that is in firmware rather than in hardware. Nothing on the board looks wrong, and the device's own registers report the width it negotiated, which is eight.

Row five is the exemption. A device narrower than its slot trains at its own width with nothing lost, and the model reports it as planned — narrow is only a defect relative to what was advertised.

12. RTL 8 — A Measurement Is Worth The Candidates It Removes

// RTL 8 - what an observation is worth. A measurement is worth the candidates
// it removes, and one that removes none costs its time for nothing.
module observation_value #(parameter int MEASURE_EVERYTHING = 0) (
  input  logic clk, rst_n,
  input  logic        plan_it,
  input  logic [15:0] candidates, removed_by_obs, minutes_each, obs_available,
  output logic [15:0] chosen, remaining, cost_min, per_candidate,
  output logic        worth_taking,
  output logic [7:0]  n_plans, n_wasteful,
  output logic        blind_measurement_err
);
  logic [31:0] c_q;
  // Taking every measurement costs every measurement.
  assign chosen = (MEASURE_EVERYTHING != 0) ? obs_available
                : ((removed_by_obs == 16'd0) ? 16'd0 : 16'd1);
  assign remaining = (candidates > removed_by_obs)
                     ? (candidates - removed_by_obs) : 16'd0;
  assign cost_min = chosen * minutes_each;
  // Minutes per candidate removed is what decides which measurement to take.
  assign c_q = (removed_by_obs == 16'd0) ? 32'hFFFF
             : ({16'd0, minutes_each} / {16'd0, removed_by_obs});
  assign per_candidate = (c_q > 32'd65535) ? 16'hFFFF : c_q[15:0];
  assign worth_taking = (removed_by_obs != 16'd0);
  // A measurement taken that removes no candidate.
  assign blind_measurement_err = plan_it && (removed_by_obs == 16'd0)
                                 && (chosen != 16'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_plans <= 8'd0; n_wasteful <= 8'd0;
    end else if (plan_it) begin
      n_plans <= n_plans + 8'd1;
      if (!worth_taking) n_wasteful <= n_wasteful + 8'd1;
    end
  end
endmodule

Five plans. Eighteen candidates, nine measurements available at twenty minutes each.

Two plans where nothing was worth taking — the same two either way, because value is a fact about the observation.

Debug is a search, and the measurement to take next is the one with the best minutes-per-candidate-removed. Twelve candidates for twenty minutes is under two minutes each; one candidate for twenty minutes is twenty. Both are worth taking and they are not equally worth taking, and the ordering is the entire method.

Row two is the measurement that should not be taken. It removes nothing — the trace already tells you what it would show — and the measure-everything approach takes it regardless, at twenty minutes. Across nine available measurements that is a hundred and eighty minutes for a narrowing that could have been done in twenty.

Row five is where the method stops applying. No candidates left means there is nothing to narrow, and no measurement is worth taking — which is either a solved problem or, more often, a candidate list that was never written down. Section 22's first question exists for the second case.

Figure 3 — Both plans reach six candidates. The difference is nine times the wall clock, and it is decided by asking what each measurement removes before taking any of them.

13. RTL 9 — An Intermittent Needs Enough Boots To Mean Anything

// RTL 9 - an intermittent failure. A discovery bug that appears one boot in
// forty needs enough boots before an absence of it means anything.
module repro_rate #(parameter int ONE_CLEAN_RUN_IS_PROOF = 0) (
  input  logic clk, rst_n,
  input  logic        judge,
  input  logic [15:0] fail_per_thousand, boots_done, boots_each_min,
  output logic [15:0] expected_fails, boots_for_one, hours_needed, confidence_pct,
  output logic        absence_means_fixed,
  output logic [7:0]  n_judgements, n_premature,
  output logic        premature_clean_err
);
  logic [31:0] e_q, b_q, h_q, c_q;
  assign e_q = ({16'd0, boots_done} * {16'd0, fail_per_thousand}) / 32'd1000;
  assign expected_fails = (e_q > 32'd65535) ? 16'hFFFF : e_q[15:0];
  assign b_q = (fail_per_thousand == 16'd0) ? 32'd0
             : (32'd1000 / {16'd0, fail_per_thousand});
  assign boots_for_one = (b_q > 32'd65535) ? 16'hFFFF : b_q[15:0];
  assign h_q = ({16'd0, boots_for_one} * {16'd0, boots_each_min}) / 32'd60;
  assign hours_needed = (h_q > 32'd65535) ? 16'hFFFF : h_q[15:0];
  // Confidence that a clean run means anything is the expected-fail count.
  assign c_q = (expected_fails > 16'd3) ? 32'd95
             : ({16'd0, expected_fails} * 32'd30);
  assign confidence_pct = (c_q > 32'd100) ? 16'd100 : c_q[15:0];
  assign absence_means_fixed = (ONE_CLEAN_RUN_IS_PROOF != 0) ? 1'b1
                             : (expected_fails > 16'd3);
  assign premature_clean_err = judge && (expected_fails <= 16'd3)
                               && absence_means_fixed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_judgements <= 8'd0; n_premature <= 8'd0;
    end else if (judge) begin
      n_judgements <= n_judgements + 8'd1;
      if (!absence_means_fixed) n_premature <= n_premature + 8'd1;
    end
  end
endmodule

Five judgements. A failure at 25 per thousand boots, three minutes a boot.

Three premature judgements; none when one clean run is taken as proof.

A discovery bug that appears one boot in forty is the hardest kind, and the trap is the fix that appears to work. Forty boots after a change is one expected failure — so a clean run is roughly as likely as a coin landing heads, and it gets reported as a fix.

Rows three and four are the threshold and it is a real decision. Four expected failures is enough to take an absence seriously; three is not. The number is arguable and having one is not — without it, "we ran it a bunch of times" is a sentence with no content.

Row five is the honest failure. With no measured rate there is no boot count to aim for and nothing can be concluded — and the model reports that rather than defaulting to confidence. Measuring the rate is the first job, and it costs two hours here.

14. RTL 10 — A Discovery Diagnosis Assembled

// RTL 10 - a discovery diagnosis assembled. Everything that must hold before
// "the link is up" is a claim about the stages past the physical layer.
module discovery_signoff #(parameter int LINK_IS_UP = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       link_up,          // the link reached L0
  input  logic       cfg_responds,     // configuration reads are not all-ones
  input  logic       dvsec_present,    // the CXL capability structure is there
  input  logic       cxl_negotiated,   // alternate protocol agreed, not fallen back
  input  logic       ready_in_time,    // the device answered inside the window
  input  logic       width_as_planned, // the trained width is the advertised one
  output logic       discovered,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_discovered,
  output logic       false_discovery_err
);
  assign fail_mask[0] = ~link_up;
  assign fail_mask[1] = ~cfg_responds;
  assign fail_mask[2] = ~dvsec_present;
  assign fail_mask[3] = ~cxl_negotiated;
  assign fail_mask[4] = ~ready_in_time;
  assign fail_mask[5] = ~width_as_planned;
  // The link-is-up build is what a link status register says.
  assign discovered = (LINK_IS_UP != 0) ? link_up : (fail_mask == 6'd0);
  assign false_discovery_err = evaluate && discovered && (fail_mask != 6'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_discovered <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (discovered) n_discovered <= n_discovered + 8'd1;
    end
  end
endmodule

Seven configurations.

One configuration discovered under the full diagnosis; six under the link status register.

Row four is the one that makes the point without argument. The link fell back to PCIe, so the link is up because the fallback worked — the register is reporting the successful outcome of the failure. It is not being fooled; it is measuring the thing it measures.

Rows two, three and four are one symptom with three causes, and the ordering matters. All-ones reads mean nothing responded at all. A missing DVSEC means the device responded and has no CXL capability. A fallback means the device has the capability and the link did not negotiate for it. The same "no CXL device" ends three different investigations.

Row six is the row that is not a discovery failure at all. A narrower width enumerates fine and delivers half the bandwidth — it belongs in this mask because it is found by the same trace, and it is the reason a discovery investigation should read the negotiated width before it closes.

Figure 4 — The order is by how much each answer removes, not by how easy it is to obtain. A config-space read separates "nothing responded" from everything else in one measurement, which is section 12's criterion applied to the flow itself.

15. Quantitative Reasoning

Stage clearing. Eight stages with five good and six causes each is eighteen candidates of forty-eight — 62% narrowed; seven good is six candidates and 87%.

Families. L0 rules out fourteen physical causes of thirty-six, leaving 61% — and the link-is-enough reading sends the work at fourteen already-cleared causes.

All-ones. Twenty-four of forty reads with no responder is 40% responded; eight of them decoded as values is eight fabricated structures.

The DVSEC. Eighteen CXL features lost of thirty is 40% of the feature set, on an enumeration that reports success.

Negotiation. Eight fallbacks of forty attempts is 80% agreed and 64 Gbps of capability absent, on links that are all up.

Readiness. A device ready at 140 ms against a 100 ms window is 40 ms late and misses every boot; at 81 ms with 20 ms of variation it is 5% — fifty boots of a thousand.

Width. A sixteen-lane device in an eight-lane slot is 32 Gbps against 64, and bifurcation produces the same from a sixteen-lane slot.

Observation value. Twelve candidates removed for twenty minutes is under two minutes each; measuring everything is 180 minutes for the same narrowing.

Reproduction. At 25 failures per thousand boots, forty boots is one expected failure and 30% confidence; 160 boots is four and 95%.

The assembled diagnosis. Six properties, seven configurations, one discovered. The link register called six discovered.

16. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Three checks ran before the mutation campaign.

Every error signal was tested for mutual exclusivity with its own guard — the check 25.7 §16 introduced after three of its models were written with dead error signals. All ten models here passed on the first reading, which is the first time that has happened.

The scripted output-listing step reported twenty-three unasserted nets and three were real — a remaining-percentage, a responded count and a responded percentage, all of them the broken build's own claim. The other twenty were parameter-independent nets whose partner was already asserted, which is the triage the script's own output line asks for.

And every published code block was compared byte-for-byte against the verified source. It found one: section 10's always_ff had acquired a redundant branch in the chapter that was not in the simulated model. A code block that is not the code that was tested is a defect regardless of whether it compiles, and the check is four lines of script.

Alongside those: every inclusive threshold at exactly equal, every clamp driven past its cap, every floor past its boundary, and both builds asserted on every degenerate case.

Stage clearing. The cleared count is driven above the stage count, exercising the clamp, and at zero where the model correctly reports no narrowing.

chk(sGc == 16'd8, "the cleared count clamps at the stage count");
chk(sGn2 == 1'b0, "which is no narrowing");

Families. L0 observed with no physical causes listed is driven — the configuration where the observation clears an empty set and establishes nothing.

All-ones. More reads decoded as valid than were all-ones is driven, exercising the clamp.

The DVSEC. A device with no features at all is driven, which is what killed one of the three survivors.

Negotiation. A fallback with no TS mismatch recorded is driven, separating the observation from its cause, and a full agreement is asserted to lose no bandwidth — which killed the second survivor.

chk(pGl == 16'd0,  "with no bandwidth lost");

Readiness. The margin is driven exactly at what the variation needs and one millisecond short — 0% against 5%.

Width. Bifurcation is driven wider than the slot, exercising the clamp.

Observation value. A measurement removing nothing is driven, where the targeted plan declines it and the measure-everything plan takes it.

Reproduction. The confidence threshold is driven at exactly four expected failures and at three.

The assembled diagnosis. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 285 checks across two testbenches, 144 on the front five models and 141 on the back five, all passing on the unmutated sources.

17. Mutation Testing

Seventy-four mutations were injected one at a time. 74 injected, 74 killed, after three survivors.

Two of the three survivors were undriven degenerate inputs — a device with no features listed, and a training set in which nothing fell back. Both were one case each.

The third was an equivalent mutant, and it is the more interesting one. Changing margin >= margin_needed to > looks like a textbook off-by-one, and at margin == margin_needed the else branch computes 100 - (100 × margin / margin) = 0, which is what the then branch returns. The two paths agree exactly at the boundary, so no stimulus can distinguish them. It was replaced with a mutation on an operand — scaling the shortfall by fifty — which §10 row five kills at nineteen milliseconds of margin.

That is batch 024's finding recurring immediately. A replacement mutation needs its own reachability argument; so does an original one. An inclusive-threshold mutation is only valid when the two branches actually disagree at the equality — and where a threshold guards a computation that is continuous through it, they do not.

18. Verification Strategy

What a testbench for a narrowing model must cover.

Check every error signal for mutual exclusivity before writing a testbench. Ten for ten here, which is what applying 25.7's check while drafting looks like.

Compare every published code block against the verified source. One defect here, found in four lines of script. The chapter is the deliverable and the simulated file is the evidence, and nothing else checks that they are the same text.

Before injecting a threshold mutation, check the branches disagree at the equality. §10's >= guards an expression that evaluates to the same value at the boundary. An equivalent mutant costs a full diagnosis cycle and teaches nothing.

Drive the configuration where an observation clears an empty set. §6 row three — L0 observed and no physical causes listed. A narrowing measured against an empty candidate list is not a narrowing, and a model that reports it as one is flattering itself.

Separate the observation from its cause. §9 row four has a fallback with no TS mismatch recorded. The model reports what is visible and does not invent the reason, which is the difference between a trace and an explanation.

The cases where the quick reading is right. A device with no CXL features. A link that never reached L0. A fallback that did not happen. A slot wider than the device. A measurement that narrows. Five exemptions across nine models.

Counters as a second signature. Ten models, ten pairs of totals, differing in seven. Three are deliberately equal — §8's n_degraded, §12's n_wasteful and §13's judgement counts in one build — because the DVSEC's presence and an observation's value are facts about the world rather than about the reading.

What a real investigation needs that these models do not have. A non-uniform cause distribution for §5, a boot-time distribution rather than a margin for §10, and a structured intermittent for §13. All three are abbreviations that preserve the conclusion, and section 26 exercises 1, 6 and 9 are where they come back.

19. Synthesis and Implementation Reality

The stages in section 5 map to things a protocol analyser prints, which is why the first hour is worth thirty candidates. Link training states, the first configuration read, the capability walk, the DVSEC read, the first CXL.mem transaction — and a capture that starts after link-up loses the one cycle section 9 needs.

All-ones is synthesised by the root complex, not by the device, so it is visible from the host and invisible on the wire. The analyser shows an unanswered request; the host shows 0xFFFFFFFF. Reading both is how you tell an absent responder from a device returning that value legitimately.

The DVSEC is a capability structure with a vendor ID and a specific ID, and a device whose capability list is malformed anywhere before it loses everything after it. A capability-list walk that terminates early is section 8's cause with a different fix — and the walk is printable.

The alternate-protocol handshake is in the training sets, which most captures do not include by default. Section 9 is the strongest argument in this chapter for capturing from reset, because the evidence exists for a few microseconds and then never again.

Readiness windows are firmware policy, and they differ between platforms. A device that works on one host and not another, with no other difference, is section 10 before it is anything else — and the window is usually documented.

Bifurcation is set in platform firmware and reported by the slot, not by the device. Section 11's investigation reads the host's slot configuration, which is a different place from everything else in this chapter.

20. Silicon Observability

"Modified TS1/TS2 exchange outcome, latched" is the one that cannot be added later. Everything else on this list can be read after the fact from a device that is sitting there failing. The negotiation happens once, in microseconds, and if nothing latched the result the evidence is gone — which makes it a design decision rather than a debug one.

21. Debug Lab

Symptom. A CXL type-3 memory expander works on the bring-up host. On a customer's platform, the device enumerates as a PCIe endpoint and no CXL memory appears. The link status register reads up, at the expected speed, on all sixteen lanes.

Step 1 — what did the trace reach? Section 5. The host log shows configuration reads succeeding and a capability walk completing. Five of eight stages good — which clears the physical family and the configuration-access family in one reading, leaving eighteen candidates of forty-eight.

Step 2 — is anything responding? Section 7. Configuration reads return real values, not all-ones. The device is there and answering, which removes the absent-responder branch entirely and is the cheapest measurement in the investigation.

Step 3 — the capability walk. Section 8. The walk completes and the CXL DVSEC is not in it. That is the symptom stated precisely: a device that responds, enumerates, and has no CXL capability structure. Three causes remain — the device never emits it, the walk is truncated, or the device is not in CXL mode.

Step 4 — which of the three. The same device on the bring-up host does present the DVSEC. That single comparison removes two of the three: the device can emit it and the walk is fine. It is not in CXL mode on this platform, and the cause is now upstream of enumeration entirely.

Step 5 — the negotiation. Section 9. The capture on the customer platform starts after link-up, so the training sets are not in it. A capture from reset shows modified TS1 sent and the CXL bits absent from the TS2 response — the host did not agree. The link then trained normally as PCIe, which is why every indicator is green.

Step 6 — why the host did not agree. The platform's firmware enables CXL on a subset of slots, and the device is in one of the others. Nothing on the device is wrong, and nothing on the host is broken — it is a configuration the device cannot see and the host does not report.

The finding. One platform configuration setting, reached in five observations of which four were free — a log read, a capability walk, a cross-platform comparison, and a capture from reset. The fifth, the capture from reset, was the only one that needed equipment, and it was the one that produced the answer.

The fix. For this customer, move the device to a CXL-enabled slot. For the product: latch the alternate-protocol negotiation outcome in a device register (section 20), so the next instance of this is a register read rather than a capture, and report the negotiated protocol in the bring-up checklist alongside link status — because "the link is up" was true throughout and was never the question.

What made this hard. Every indicator a bring-up checklist reads was green, and the one cycle that explained the failure was outside the capture window. The device was correct, the host was correct, and the answer was in a firmware setting that neither reports.

22. Design Review

1. What is the candidate list, written down? Section 12 — a narrowing needs something to narrow, and most investigations never write one.

2. What was the last discovery stage completed? The highest-value observation in the chapter, available from a log. Section 5.

3. Does "the link is up" appear anywhere in the reasoning as evidence of enumeration? It rules out a family and nothing else. Section 6.

4. Are any all-ones reads being decoded as values? They are absences, always. Section 7.

5. Does the capability walk complete, and is the DVSEC in it? Printable, and it separates three causes. Sections 8 and 19.

6. Was the capture started from reset? The negotiation evidence lasts microseconds. Sections 9 and 19.

7. What is the platform's readiness window, and the device's ready time distribution? A margin is only visible as a distribution. Sections 10 and 20.

8. What width and speed did the link negotiate, against what was advertised — and what is the slot's bifurcation? Two different places to look. Section 11.

9. For each remaining measurement, how many candidates would it remove? Minutes per candidate is the ordering. Section 12.

10. For an intermittent, what is the measured rate and how many boots does a clean run need? Section 13 — without the rate, no absence means anything.

23. How This Appears In Real Engineering

A bring-up engineer reads the link status register first because it is the first thing that exists. It is a real observation worth fourteen causes, and the mistake is not reading it — it is stopping there.

A platform engineer owns the firmware settings in sections 9, 10 and 11, none of which the device can see or report. Three of this chapter's six failures are configuration on the other side of the connector.

A device architect decides in section 20 whether the negotiation outcome is latched. That decision is made years before the debug that needs it, and it is the difference between a register read and a capture from reset.

A customer support engineer meets all of this at a distance, with no analyser and one platform. Sections 5, 7 and 8 are the three that work over a phone call, and between them they separate most of the cause set.

24. Common Misconceptions

"The link is up, so the hardware is fine." L0 rules out fourteen causes of thirty-six (section 6) and says nothing about configuration, capabilities or negotiation. It is evidence, not reassurance.

"The device enumerated." As what? A device with no DVSEC enumerates perfectly as a PCIe endpoint (section 8), with eighteen CXL features absent and nothing reporting them missing.

"It returned 0xFFFFFFFF, so that register reads as all ones." It returned nothing, and the root complex synthesised that value (section 7). Every field decoded out of it is fabricated.

"The CXL negotiation failed, so the link would have failed." It falls back (section 9). A failed negotiation produces a working PCIe link, which is the most expensive silent success here.

"It's intermittent, so it's marginal signal integrity." A device 19 ms inside a 20 ms margin fails five percent of boots (section 10) with a perfect link every time. Intermittent discovery is a timing-window symptom before it is a signal one.

"We rebooted it twenty times and it was fine." At 25 failures per thousand, twenty boots is half an expected failure (section 13). A clean run there is a coin toss reported as a fix.

25. Interview Reasoning

"A host doesn't see your CXL device. What do you ask for first?" The last discovery stage completed — not the link status. It is available from a log, it costs nothing, and it removes thirty of forty-eight candidates. A candidate who asks for an analyser first has skipped the free measurement.

"The link status register says up. What does that establish?" That the physical family is cleared — fourteen causes of thirty-six — and nothing about enumeration. The follow-up worth asking back: up as what protocol? Section 9's fallback produces a link that is up and is not CXL.

"Configuration reads come back as all ones. What is your next step?" Not to decode them. All-ones is the absence of a responder, so the next step is upstream — is the device driving the bus at all, is it in reset, is the address routed to it. A candidate who reads a vendor ID out of it has lost the thread.

"It works on your board and not the customer's. What is the shape of that?" A platform setting neither side reports. Sections 9, 10 and 11 are all of this shape — negotiation enablement, readiness window, bifurcation — and the cross-platform comparison is the measurement that splits them from device faults in one step.

"How many boots before you believe an intermittent is fixed?" A question about the rate, not a number. At 25 per thousand, 160 boots for four expected failures (section 13); without a measured rate, no answer is defensible. The first job is measuring the rate, and it costs two hours.

26. Exercises

1. Weight the stages. §5 spreads causes evenly. Assign realistic cause counts to eight discovery stages and recompute the narrowing from a trace that reaches stage five. Which stage is worth the most to clear?

2. Build the cause list. For a CXL type-3 device that does not enumerate, write the candidate list — and say for each entry which observation removes it.

3. Separate the three "no DVSEC" causes. Device never emits it, walk truncated, not in CXL mode. Design the minimum set of observations that distinguishes all three, and order them by §12's criterion.

4. Capture from reset. State what a capture must include to answer §9, and what is lost by starting at link-up. How long is the evidence window?

5. Read a capability list. Take a malformed capability chain and determine what is lost after the break — then write the check that would catch it in the device's own tests.

6. Model readiness as a distribution. §10 uses one margin figure. Replace it with a boot-time distribution and compute the failure rate for windows of 80, 100 and 150 ms.

7. Order the measurements. Given eighteen candidates and five available measurements with known removal counts and costs, produce the optimal order and the total time to a single candidate.

8. Find the bifurcation. State where a sixteen-lane slot presenting eight lanes is configured, and what the device can and cannot see of it.

9. Size an intermittent campaign. For rates of 25, 5 and 1 per thousand at three minutes a boot, compute the hours for 95% confidence. At which rate does the campaign stop being affordable, and what do you do instead?

10. Add the seventh property. Propose one none of §14's six implies, name its section, and construct the configuration where the six hold and it fails. A property that cannot fail alone is not a seventh property.

27. Summary

An observation is worth the candidates it removes, and the most-reported one removes the fewest. "The link is up" clears fourteen causes of thirty-six; "five of eight stages completed" clears thirty of forty-eight, and the second is a log line.

Trained and enumerated are disjoint families. L0 means the electricals, the training sequences and the deskew worked — and nothing whatever about whether a configuration read will be answered.

And a narrowing against an empty candidate list is not a narrowing. L0 observed with no physical causes written down establishes nothing, which is why the candidate list is a deliverable rather than a habit.

All-ones is the absence of a responder. Twenty-four of forty reads unanswered, eight of them decoded into capability structures the host invented — and every field read out of them is fabricated.

Without the DVSEC the host enumerates a working PCIe endpoint. Eighteen features lost of thirty, and the enumeration reports success, because it is one. A bring-up checklist that says "device enumerates" ticks.

A failed negotiation falls back rather than failing. Eight of forty links trained as PCIe, sixty-four gigabits a second of capability absent, every indicator green — and the single cycle that explains it lives in the training sets, for microseconds, unless something latches it.

A device outside the readiness window is never found. Forty milliseconds late is every boot, deterministically, on a perfect link — and nineteen milliseconds of margin against twenty of variation is five percent, which gets called flaky hardware.

And the trained width is not the advertised one. Sixteen lanes in an eight-lane slot is half the bandwidth, and a bifurcated sixteen-lane slot produces the same from firmware that the device cannot see.

A measurement that removes nothing costs its time for nothing. Twelve candidates for twenty minutes is under two minutes each; measuring everything is a hundred and eighty minutes for the same answer.

And at 25 failures per thousand, forty boots is one expected failure. A clean run there is a coin toss, and 160 boots is what four expected failures costs — without a measured rate, no absence means anything at all.

Three checks ran before the mutation campaign and all three found something or proved something. Mutual exclusivity passed ten for ten, the first chapter to do so. The scripted output-listing step found three unasserted claims of twenty-three candidate nets. And a byte-for-byte comparison of every published code block against its verified source found one block that had drifted — a defect that compiles, reads correctly, and is not what was tested.

One of three mutation survivors was an equivalent mutant, and it is worth naming: >= changed to > on a threshold whose two branches compute the same value at the equality. Batch 024's rule about replacement mutations applies to original ones too — a threshold mutation is only valid where the branches actually disagree at the boundary.

"The link is up" is one property of six. The register called six of seven configurations discovered when one was — and §21 is a device that enumerates, answers every configuration read, trains at full width and full speed, and is not a CXL device, because a firmware setting on the other side of the connector said so and nothing on either side reports it.

26.2 — Link Failures takes the family this chapter ruled out. Everything here assumed the link either reached L0 or did not; the next chapter is about a link that reaches L0 and then does not stay there.