Senior Silicon Debug

“Walk me through a DDR bring-up debug session.” It is the only question in this module with no right answer at all, and that is deliberate.

The graded output is not a diagnosis. It is an ordered sequence of decisions, each one justified by the evidence available at the moment it was taken — and an interviewer will interrupt to remove an instrument, make the failure intermittent, or forbid a reset, precisely to see whether the sequence was reasoning or a script.

Two laws, and the second is the one that costs people real weeks:

A candidate who names a mechanism before naming a discriminator has guessed. The answer to every stage is an experiment, not a cause.

Capture before you recover. Every recovery action destroys the state that would have explained the failure, and the pressure to recover is highest exactly when the failure is rarest.

1. What Is Actually Being Graded

Five things, and none of them is whether you find the bug.

Row three is the one that separates levels and it is a habit of speech. A candidate who says so it must be signal integrity has asserted; a candidate who says the rate test has exonerated the digital layers and not chosen among the remaining three has reported. The second is correct, and it is also what a colleague can act on.

And row two has a shape worth stating explicitly. Cheapest first is not laziness — it is information per hour. A configuration read-back takes minutes and eliminates an entire class; a board respin takes weeks and eliminates one hypothesis.

2. The Observability Collapse

Everything in Modules 17 to 23 reasons about signals. In a lab, almost none of them exist.

This is the constraint the entire debug method exists to work around, and naming it in the first minute is the strongest opening available.

Three consequences, and each one converts into an interview sentence.

Everything is inference. CURRICULUM-DERIVED from 28.1 §1, whose result generalises past training: an observable is a projection of several independent failure domains onto one bit, and projections lose information. So the method is hypothesis-and-discriminator rather than observation.

Looking costs the evidence. A reset, a reboot, a retrain and a power-cycle each destroy a different superset of the state — 28.7 §10 owns the table. So let me just reset and look is not a neutral action.

And the configuration is evidence, not context. CURRICULUM-DERIVED from 28.7 §5, which owns configuration provenance as a bring-up artefact rather than a debugging afterthought: the configuration as read back is a first-class observable, and it is the cheapest one available.

3. The Method, in One Shape

CURRICULUM-DERIVED from Module 28, whose seven chapters share one spine. The interview value is being able to state it in six words and then apply it under interruption.

   SYMPTOM  ->  EVIDENCE  ->  HYPOTHESES  ->  DISCRIMINATOR
            ->  RESPONSIBLE LAYER  ->  FIX  ->  REGRESSION

   and the three rules that make it work:

   1  A FAILURE SIGNATURE IS EVIDENCE, NOT A VERDICT.
        the signature narrows the hypothesis set; it does not
        select from it. 28.1, 28.5 each catalogue eleven
        signatures FOR THAT REASON.

   2  ONE VARIABLE -- and "one" requires decomposition.
        28.7 §4: raising the frequency changes the period, every
        derived timing count, calibration validity, channel
        behaviour and the access rate. That is one STEP and five
        VARIABLES.

   3  CAPTURE BEFORE RECOVERY.
        28.7 §10, and §10 of this chapter is the hardware that
        was supposed to make it automatic.

Rule two is the one candidates claim and violate in the same answer. CURRICULUM-DERIVED from 28.7 §4: change one thing is a goal that requires decomposition rather than discipline alone, and the technique is to change the compound variable and decompose only if the result is surprising. And its corollary — do not change a variable and a stage at the same time — is what collapses the method when broken, because the suspect list becomes both.

4. The Session Opens

Q1. Bring-up. The system passes training on every lane, boots, runs for about eleven hours under load, then reports a single uncorrectable memory error and halts. What do you do first?

Nothing to the system. The first three actions cost minutes and are all reads.

   1  READ BACK the configuration -- 28.7 §5.
      what the hardware is USING, not what was intended.
      cost: minutes. eliminates: an entire class.

   2  READ the snapshots BEFORE anything is reset -- 28.7 §10.
      28.2 §15's violation snapshot, 28.5 §13's first-error
      capture, 28.1 §14's training ledger. all three are
      designed to survive until an EXPLICIT clear.
      cost: minutes. and it is now or never.

   3  RECORD the rate and the duration -- 28.5 §3.
      one error in eleven hours is a NUMBER, and the arithmetic
      that uses it needs both terms. neither survives a reboot.

And then the sentence that makes it an answer: at this point I know the configuration, whether a snapshot fired, and the rate. I do not know the layer, and I have not changed anything.

Why the rate matters more than it looks. CURRICULUM-DERIVED from 28.5, whose arithmetic requires the rate and the duration: one error in eleven hours is a different problem from one error per minute, and the difference determines whether a reproduction attempt is even affordable. A candidate who does not write the rate down has given up the ability to plan the rest of the investigation.

The trap in the question's wording, and it should be named unprompted. Uncorrectable implies ECC, and CURRICULUM-DERIVED from 28.5's ECC visibility problem: on-die correction means the observed structure is not the fault structure. So the single uncorrectable error may be the visible tail of a population of corrected ones — and the corrected-error counters, if they exist, are the highest-value read on the list. One visible error and a large corrected count is a completely different problem from one visible error and none.

5. The First Real Decision

Q2. The snapshot fired. It shows a read to one bank, one byte lane wrong, and the configuration read-back matches the intent. What now?

Three hypotheses survive, and the useful move is to rank them by the cost of the experiment that would separate them.

The one-lane fact already did work, and saying so is the answer. A fault confined to one byte lane is evidence against every shared-resource cause — the clock relationship, the command path, the scheduler — because those would affect every lane. Chapter 19.1 §8 owns the layer taxonomy this narrows.

What not to do, and it is the most common wrong move: do not retrain. CURRICULUM-DERIVED from 28.7 §10, retraining destroys the committed settings that were in use when it failed — which are the primary evidence for hypothesis A. Retraining is the experiment you run after recording them, and it is then a real experiment: if the failure clears, the settings were the cause.

And the discipline about hypothesis C: CURRICULUM-DERIVED from 28.6, which owns the escalation criterion and names the three experiments most often mistaken for proof — frequency, lane and ODT. A lower frequency working does not prove a channel problem, and 19.1 §8 states the bound precisely: the rate test exonerates the digital layers and does not choose among the remaining three. So the channel hypothesis becomes admissible when the digital ones are eliminated, not when one experiment is suggestive.

6. The Interviewer Removes the Instruments

Q3 (removing an assumption). There is no snapshot register. The part reports a machine check and nothing else. Now what?

The honest first answer is that the investigation just became far more expensive, and saying so is not a complaint — it is the estimate. Then the method adapts, and it adapts in a specific direction: toward experiments that produce their own evidence.

Row two is the technique worth having. If you cannot observe the lane, choose a stimulus that encodes it — a per-lane distinguishable pattern converts a data error into a lane identifier, and it costs one test rather than a silicon change.

And the deliverable this question is really asking for: a list of the debug registers you would add for the next revision, justified by this session. CURRICULUM-DERIVED from the three snapshot instruments Module 28 builds — 28.2 §15's violation snapshot with a rule mask rather than a rule id, 28.5 §13's first-error capture with structure accumulation, and 28.1 §14's ledger — all three designed to survive until an explicit clear. A candidate who answers this question with a specification is doing the senior job.

7. It Is Intermittent

Q4 (removing another assumption). It reproduced once in eleven hours and has not reproduced since. How do you proceed?

You stop trying to reproduce it and start trying to accelerate it, because a one-in-eleven-hours experiment cannot support any investigation with more than about two steps.

Four acceleration axes, and the third is the one that is usually available:

The validity column is the whole answer. An accelerated failure is only useful if it is the same failure, and establishing that requires the signature to match — which is why the original snapshot's contents are the reference and §4's step two is load-bearing hours later.

And the discipline that must accompany acceleration: CURRICULUM-DERIVED from 28.7 §11 and 28.5, both of which own minimal reproducer reduction applied during the session rather than after: reduce while you can still reproduce. A reproducer that takes eleven hours is not a reproducer, and each reduction step must be validated against the signature, because a reduction that changes the signature has changed the problem.

The senior addition nobody offers: decide the budget out loud. At one failure per eleven hours, a five-step investigation is two weeks of wall-clock. Accelerating is not an optimisation — it is the precondition for investigating at all. Stating the arithmetic is what justifies spending the day on acceleration instead of the day on hypotheses.

8. “Nothing Changed”

Q5 (diagnosing). It worked last week. Nothing changed. Now it fails.

Treat the claim as a hypothesis with a cheap test, not as a constraint on the search.

Row five is the one that reframes the question and it is often the truth. A marginal system does not fail deterministically — it fails at a rate, and a rate that was below the observation threshold last week can cross it with no change anywhere. CURRICULUM-DERIVED from 28.1 §5's narrow-window signature: a one-tap window passes training and reports a pass, so a system can ship marginal and fail later with nothing having changed.

So the discriminating question is: what is the rate now, and what was it then? If nobody measured it then, it worked last week means nobody saw it last week — and those are different claims. Saying so, carefully and without blame, is the answer.

9. One Piece of Real Evidence

Below is the whole of what one session had: a captured snapshot, and what happened to it. Every value is ILLUSTRATIVE.

Read the sw_clear lane. It is zero for the entire trace. Nothing the software did caused this, so the snapshot was cleared by something that was never supposed to clear it — and §11 is that something.

And read cycle 5 before cycle 7. The register did the hard part correctly: the second error did not overwrite the first, which is 28.5's first-error discipline implemented properly. The block is right about the subtle requirement and wrong about the obvious one, which is why it passed review.

10. RTL Review — The Evidence Register

The intended contract:

1. On the first error after an explicit clear, capture the snapshot and set captured.
2. Subsequent errors must not overwrite it. Count them separately — CURRICULUM-DERIVED from 28.5's first-error capture.
3. The snapshot and captured are cleared only by an explicit software clear — not by the functional reset, not by a retrain, not by a recovery action. CURRICULUM-DERIVED from 28.7 §10: capture before recovery, which is only possible if the capture survives the recovery.
4. Count total errors, so a rate is computable — 28.5 §3 needs the rate and the duration.
5. Report an overflow flag if the error counter saturates, so a rate is never silently understated.

// ---------------------------------------------------------------------
// first_error_snapshot -- INTENTIONALLY DEFECTIVE, for review (§10).
//
// CLASSIFICATION: synthesisable debug hardware, ILLUSTRATIVE, and
// CONTAINS A BUG.
//
// WHAT IT IS MEANT TO DO: the five-clause contract above -- capture
// the FIRST error's context and hold it against everything except an
// explicit software clear, so that an operator can recover the system
// and still have the evidence afterwards.
//
// WHY IT EXISTS HERE: 28.7 §10 owns "capture before recovery". This
// block is the hardware that was supposed to make that rule
// automatic, and §9's trace is what it actually does.
//
// HOW TO RUN IT: capture an error, then assert the functional reset
// as a recovery action would.
// EXPECTED RESULT under clause 3: `captured` and `snapshot` SURVIVE.
// EXPECTED TRACE: only a write to sw_clear may return captured to 0.
//
// SYNTHESIS: one snapshot register, one sticky bit, two counters.
// LIMITATIONS: captures ONE error's context. Structure accumulation
// across many errors is 28.5's and is a STATED omission, not the bug.
// ---------------------------------------------------------------------
module first_error_snapshot #(
  parameter int SNAP_W = 64,
  // COUNT, not INDEX: the error counter must REPRESENT its maximum,
  // and clause 5 exists because a saturated counter understates a
  // rate silently -- the instrument failing in the direction that
  // makes the problem look smaller.
  parameter int ERR_W  = 16
)(
  input  logic               clk,
  input  logic               rst_n,        // FUNCTIONAL reset

  input  logic               err,
  input  logic [SNAP_W-1:0]  err_context,
  input  logic               sw_clear,     // explicit software clear

  output logic               captured,
  output logic [SNAP_W-1:0]  snapshot,
  output logic [ERR_W-1:0]   err_count,
  output logic               err_overflow
);
  always_ff @(posedge clk) begin
    if (!rst_n) begin                      // <-- THE DEFECT
      captured     <= 1'b0;
      snapshot     <= '0;
      err_count    <= '0;
      err_overflow <= 1'b0;
    end else if (sw_clear) begin
      captured     <= 1'b0;
      snapshot     <= '0;
      err_count    <= '0;
      err_overflow <= 1'b0;
    end else begin
      if (err) begin
        // Clause 1 and 2: capture the FIRST, hold it against later
        // errors. This part is correct and is why the block passes
        // review -- see §9's cycle 5.
        if (!captured) begin
          captured <= 1'b1;
          snapshot <= err_context;
        end
        // Clause 4 and 5: count every error, and say so if saturated.
        if (err_count == {ERR_W{1'b1}}) err_overflow <= 1'b1;
        else                            err_count    <= err_count + 1'b1;
      end
    end
  end
endmodule

Before reading on: which clause, and what makes this defect different in kind from the previous six in this module?

11. The Defect — The Instrument That Recovery Destroys

The violated clause is 3, and the defect is the first branch: if (!rst_n) clears the snapshot.

Clause 3 says the capture is cleared only by sw_clear. The functional reset clears it too — and the functional reset is exactly what an operator asserts in order to recover.

   the sequence, from §9's trace:

     eleven hours of correct operation
     one error            -> captured = 1, snapshot = E1   CORRECT
     a second error       -> snapshot UNCHANGED             CORRECT
     the operator recovers by asserting the warm reset
                          -> captured = 0, snapshot = 0     CLAUSE 3
     sw_clear             -> never asserted anywhere

   so the evidence of an eleven-hour failure is destroyed by the
   first action any operator takes, and the register's own log
   says nobody cleared it.

And the answer to §10's harder question — what makes this defect different in kind.

The previous six defects in this module produced wrong behaviour. This one produces correct behaviour and destroys the evidence that any behaviour occurred. It is not a bug in the design; it is a bug in the ability to find bugs.

Which is why it is the last one. A debug instrument that does not survive the recovery converts a hard problem into an unsolvable one, and the cost is measured in reproduction attempts: at one failure per eleven hours, each lost snapshot costs another eleven hours and there is no guarantee the next one is the same failure — §7's validity problem, arriving as a consequence.

The correction, and both parts are required:

// CORRECTED. The capture must survive the FUNCTIONAL reset, because
// the functional reset is the recovery action (28.7 §10). It may be
// cleared by a POWER-ON reset -- after which there is no system state
// left to explain anyway -- and by an explicit software write.
//
// por_n is a separate input, not a rename: the point is that the two
// resets have different scopes and this register belongs to the wider
// one. If the platform has only one reset, the snapshot belongs in a
// register that reset does not reach, and saying that is the review
// finding rather than a workaround.
  always_ff @(posedge clk) begin
    if (!por_n) begin
      captured     <= 1'b0;
      snapshot     <= '0;
      err_count    <= '0;
      err_overflow <= 1'b0;
    end else if (sw_clear) begin
      // ... unchanged
    end else begin
      // ... unchanged, and NOT gated by rst_n
    end
  end

And the review finding that is larger than the fix: every debug register in the design needs its reset domain stated as part of its specification, not inherited from whatever reset the surrounding module used. CURRICULUM-DERIVED from 28.7 §10's table — reset, reboot, retrain and power-cycle each destroy a different superset — so a debug register's value is exactly the set of recovery actions it survives, and a register that survives none has no value at all. Grepping the debug block for if (!rst_n) is a real review technique with a real yield.

12. SVA Review — The Ninth Variety, Hidden by the Guard

A property offered as proof that the capture is sticky:

  // Offered as "proves the first-error capture is sticky".
  property p_capture_is_sticky;
    @(posedge clk) disable iff (!rst_n)
      (captured && !sw_clear) |=> captured;
  endproperty
  assert property (p_capture_is_sticky)
    else $error("capture was lost without a software clear");

Q. This property is correct, well-guarded, follows every convention this curriculum teaches — and it cannot fail on the defective block. Why?

Because disable iff (!rst_n) is precisely the blind spot. The defect's mechanism is the functional reset, and the guard kills the evaluation attempt across exactly the cycles the defect acts on. When evaluation resumes, captured is already 0, so the antecedent is false and there is nothing to check.

A property guarded against a condition cannot observe a defect whose mechanism is that condition. The reset guard is correct discipline and it is also a hole, and the hole is shaped exactly like reset behaviour.

This is the ninth variety, and it is the one that could not have been found without the previous eight, because it is the discipline that made them all correct turning into the failure. CURRICULUM-DERIVED from 27.2 §7, which owns reset discipline and what its absence costs — and this is the symmetric finding: what its presence costs, in the one case where reset is the subject rather than the context.

What actually covers clause 3:

  // Guarded by the POWER-ON reset, so the functional reset is INSIDE
  // the checked region rather than outside it. This is the whole fix
  // to the property, and it mirrors the fix to the design.
  property p_capture_survives_functional_reset;
    @(posedge clk) disable iff (!por_n)
      (captured && !sw_clear) |=> captured;
  endproperty
  assert property (p_capture_survives_functional_reset)
    else $error("capture lost across a functional reset");

  // Clause 2, which the defective block satisfies -- worth asserting
  // anyway, because the correction to clause 3 must not break it.
  property p_snapshot_not_overwritten;
    @(posedge clk) disable iff (!por_n)
      (captured && !sw_clear) |=> $stable(snapshot);
  endproperty
  assert property (p_snapshot_not_overwritten)
    else $error("a later error overwrote the first-error snapshot");

  // Clause 5 -- the rate must never be silently understated.
  property p_saturation_is_reported;
    @(posedge clk) disable iff (!por_n)
      (err_count == {ERR_W{1'b1}}) |-> err_overflow;
  endproperty
  assert property (p_saturation_is_reported)
    else $error("error counter saturated without reporting overflow");

  // And the cover WITHOUT WHICH the first property proves nothing:
  // the functional reset must actually be asserted while a capture
  // is held. A regression that never resets after an error reaches
  // neither the defect nor the property that catches it.
  cover property (@(posedge clk) disable iff (!por_n)
                  captured && !rst_n);

  // The antecedent, published separately: a zero here means no error
  // was ever captured, which is a different fact from stickiness.
  cover property (@(posedge clk) disable iff (!por_n) $rose(captured));

The first cover is the stimulus requirement and it is the whole finding. A verification environment that asserts reset only at time zero can never reach this, and CURRICULUM-DERIVED from 28.7 §10, the scenario that matters is error, then reset — a sequence no functional test has any reason to run. So the test that would have found this is a debug-feature test, and debug features are the least-tested logic in most designs precisely because they do not affect function.

Follow-up an interviewer should ask: how would you test debug hardware generally? By running the recovery actions. For each debug register, assert each reset, perform a retrain, and check what survived — then publish the surviving set as the register's specification, per §11. That is a short, mechanical test plan, and almost nobody writes it.

13. What Would You Measure?

Q. What would you require of a design so that the next silicon debug is affordable?

Row eight is the one that is cheap and absent. CURRICULUM-DERIVED from 27.7, which owns the cross-domain correlation problem that follows from having no common time base: without a shared timestamp, two snapshots from two domains cannot be ordered, so which happened first becomes unanswerable — and that is the question most debug sessions turn on.

And the framing that makes this list a deliverable rather than a wish. Every row is justified by a specific session, and each has a cost measured in reproduction hours. At one failure per eleven hours, a missing snapshot costs eleven hours per investigation step — which is the argument that gets debug hardware into a design, and it is the argument the verification engineer is uniquely placed to make.

14. Common Wrong Answers

“I'd reset it and see if it comes back.” §2, §4. A reset destroys every debug register, the captured state and the failing configuration. Capture first.

“I'd retrain to see if that fixes it.” §5. Retraining destroys the committed settings in use when it failed — the primary evidence. Record them, then retrain, and it becomes a real experiment.

“It only fails at speed, so it's signal integrity.” §5. The rate test exonerates the digital layers and chooses among none of the remaining three — 19.1 §8. And frequency, lane and ODT are the three experiments most often mistaken for proof — 28.6.

“The signature tells us it's a timing violation.” §3. A failure signature is evidence, not a verdict. It narrows the hypothesis set; it does not select from it.

“I'll change the frequency and the burst length and see.” §3. Two variables, and neither is one — 28.7 §4. The suspect list becomes their union.

“We'll move to stage 8 and raise the rate together to save time.” §3. A variable and a stage at once, and the method collapses for that failure.

“One uncorrectable error, so one fault.” §4. On-die correction means the observed structure is not the fault structure — read the corrected counters.

“The configuration is what we programmed.” §2, §8. It is what reads back, and the difference is the cheapest finding available.

“It worked last week and nothing changed.” §8. A marginal system fails at a rate; a rate below the observation threshold last week can cross it with nothing having changed. If nobody measured it then, the claim is that nobody saw it.

“It reproduced once, so let's investigate.” §7. A one-in-eleven-hours reproducer supports about two steps. Accelerate first, and validate the acceleration against the signature.

“I'll crank the stimulus until it fails often.” §7. That changes what is being tested, and a new failure may not be the old one. The signature is the reference.

“There's no snapshot, so we're stuck.” §6. Encode the missing observable in the stimulus, re-derive the applied counts by hand, and count the rate — then deliver the register specification for the next revision.

“The stickiness assertion passes, so the capture is safe.” §12. disable iff (!rst_n) is shaped exactly like the defect. Guard with the power-on reset instead.

“Debug registers don't need verifying — they don't affect function.” §12. Which is why they are the least-tested logic in the design, and why the error-then-reset sequence that finds this bug is in nobody's functional test.

“I once saw this exact symptom and it was the DQS gate.” A remembered case is a hypothesis, not a diagnosis, and offering it as a conclusion is the module's central error in its most seductive form. Say it as what it is: I have seen this signature come from a gate misalignment; the discriminator would be the gate's presence result against data correctness — 30.7 §6. The memory is useful; the shortcut is not.

15. Self-Check

1. List six things you can observe in simulation and cannot observe in silicon, and for each name the substitute the method uses.

2. Write the method's seven-stage spine and its three rules from memory. Then give the one rule candidates most often claim and violate in the same answer.

3. A single uncorrectable error after eleven hours. Give your first three actions, the cost of each, and the sentence stating what you know afterwards.

4. Explain why retraining is a destructive action and the condition under which it becomes a legitimate experiment.

5. There is no snapshot register. Give the substitute for the failing lane, and say why encoding an observable in the stimulus is cheaper than a silicon change.

6. Give the four acceleration axes and, for each, what it costs in validity. Then state how you would establish that the accelerated failure is the same failure.

7. Find the defect in §10 without reading §11. Then say what makes it different in kind from every other defect in this module.

8. Explain why p_capture_is_sticky cannot fail on the defective block, state the general rule, and give the cover that the fixed property needs in order to mean anything.

9. Write the debug-register specification you would submit for the next revision, with each row justified by a session step and a cost in reproduction hours.

16. Where This Goes — and the Module Closes

A silicon-debug question is a decision-sequence question. Name the observability collapse first, because it is the constraint everything else works around; make every step an experiment with a named discriminator; order by cost, because cheapest first is information per hour; say what you know and what you do not after each step; capture before you recover; and treat acceleration as the precondition for investigating a rare failure rather than as an optimisation.

Three results close the chapter. A debug register's value is exactly the set of recovery actions it survives, and one that survives none has no value at all. A property guarded against a condition cannot observe a defect whose mechanism is that condition — the ninth variety, and the one that only exists because the other eight established the discipline. And a remembered case is a hypothesis; offering it as a diagnosis is the module's central error in its most seductive form.

And the module closes on the law it opened with.

A STRONG DDR INTERVIEW ANSWER IS NOT A FACT RECALLED. IT IS A CORRECT MENTAL MODEL APPLIED TO THE EVIDENCE GIVEN.

Ten chapters have now tested that claim ten different ways, and the same three moves recur in every one.

Name what is missing before answering. 30.1 refused to answer an average-versus-deadline question without the deadline; 30.3 computed a lower bound and said which constraints could only push it later; 30.8 refused too slow until one of five bandwidths was named. The refusal is the first correct step, not a stall.

Say the dependency, both branches and the distinguishing observation. 30.2's three outcomes, 30.4 §3's two parts whose ranking flips with load, 30.5's legality-before-policy, 30.6's eight-field frame applied to a step never memorised, 30.7's placement rule reproducing a twenty-row table. “It depends” without those three parts is not an answer; with them, it is the best one available.

And ask what the evidence actually proves. Nine chapters produced nine distinct reasons a green result proves nothing — too narrow in time, silent on the key signal, inexpressible without a model, sharing the design's wrong constant, an unasserted range, vacuous, not independent, blind to conservatism, and hidden by its own reset guard. Chapter 30.9 §6 holds the list, and its length is the point: it is a procedure, not an insight.

What none of this required was a remembered number. Every parameter value in this module is labelled ILLUSTRATIVE, every verified figure is attributed to the chapter that verified it, and the strongest answer to every request for a specific value was to say where it comes from instead. That is not a gap in preparation. It is the preparation.