Advanced PCIe Interview Questions — Credits, LTSSM, and MSI-X

27.2 asked how PCIe is assembled. This round asks about the three mechanisms that decide whether it keeps working under load — and the answers are graded on whether you can derive a consequence, not on whether you can name a state.

1. Sources, Scope, and What This Chapter Refuses to Do

2. What the Advanced Level Tests

Three readings.

Rows 1 and 3 are the two most reliable discriminators in a PCIe interview. "Credits are buffers" is nearly right and produces a wrong design (§3), and not knowing the conservation invariant means not being able to detect a leak — which is the failure mode credits actually have (§5).

Row 7 is the one that separates candidates who have debugged silicon. A link in a working state proves the link trained, and proves nothing about whether useful work is moving — and §10's trace is that stated as an exercise.

And row 8 is a change in standard rather than a new topic. At this level the interviewer expects you to be unable to recall some values and able to reason without them.

3. "What Does a Credit Represent?"

15 seconds. "Permission to send. A transmitter may only send a TLP of a given class if it holds credit for it, and the receiver returns credit as the corresponding resource frees. It is a promise about capacity, not the capacity itself."

60 seconds. Add the negation, because it is the whole question: "credits are not buffer space — they are permission that refers to buffer space. The buffer is at the receiver; the credit is an accounting token at the transmitter that says the receiver has room. The distinction matters because it tells you who owns what: the receiver owns the resource and the accounting of what it has freed, the transmitter owns the accounting of what it has consumed, and the protocol keeps those two views consistent" (16.1, 16.5, 16.6).

Then the consequence worth volunteering. "And it means flow control cannot fail by overflowing — a transmitter without credit does not send. It fails by stalling, and the interesting failures are all cases where credit is not returned when it should be" (§5).

4. "Why Are There Several Credit Types?"

The answer is head-of-line blocking, and the strong version says so first.

60 seconds. "Because the transaction classes consume different receiver resources and must not block each other. Posted, non-posted and completion traffic are accounted separately — so a receiver that is out of room for posted writes can still accept completions. If there were a single credit pool, a burst of one class would starve the others, and the class most likely to be starved is the one that frees resources" (16.2, 16.3, 16.4).

Then the failure this prevents, which is the follow-up: "and that is a deadlock, not a slowdown. If completions cannot be accepted because posted traffic consumed the shared resource, the completions that would have freed the outstanding requests never arrive — so nothing drains" (25.8 owns the debug procedure, and §12 asks you to construct the scenario).

And a second dimension worth naming, because interviewers ask: "credits are also accounted separately for header and data, because a TLP consumes both a header slot and a payload allowance, and a stream of small TLPs exhausts header credit while a stream of large ones exhausts data credit." Do not quote credit unit sizes (§1).

5. "How Do You Know Your Credit Accounting Is Correct?"

The invariant, and this is the question that most reliably identifies someone who has debugged a link.

The answer. "Conservation. For each credit type, credits consumed plus credits available equals the configured total — always. Every credit the transmitter consumes must eventually be returned by the receiver, and neither side may create or destroy one. If that invariant holds, the link cannot overflow and cannot leak."

Then the failure mode, which is the reason to know the invariant: "a leak is monotonic. If one credit in a million is lost — consumed by the transmitter and never returned by the receiver — the available count only ever drifts downward. A short test never notices; a soak test wedges the link hours in. And the symptom is a stall with no error: the transmitter is waiting for credit that will never come, the receiver has room, and nothing is wrong on the wire" (25.8).

The follow-up: "How would you catch it?" — an assertion on the invariant, not a timeout on the symptom (§11). "A timeout tells you the link stalled hours after the cause. The conservation check fires on the cycle the accounting breaks."

6. Read the Trace — the Link Is Up and Nothing Moves

"The link is up. Why is nothing moving?"

The transmitter holds no credit and no update has returned (§5). cred_avail fell monotonically to zero and cred_upd never asserted — so this is a credit stall, not a link problem, and link_up being high throughout is exactly why the reported symptom will be "the link is fine but throughput is zero."

"Is this a leak or backpressure?" — the trace cannot distinguish them, and saying so is the correct answer. Legitimate backpressure looks identical over twelve cycles: a receiver that is genuinely full also returns no updates. The discriminator is time and the receiver's state — "I'd check whether the receiver has room. If it has room and is not returning credit, it is a leak or a lost update; if it is genuinely full, it is backpressure and the question moves to why it is not draining."

"What would you add to tell them apart?" — the conservation check (§5, §11) plus a no-credit cycle counter and receiver occupancy. If consumed + available ≠ configured total, it is a leak and the arithmetic proves it without waiting for a stall.

7. The LTSSM — Which Transitions Actually Matter

"Which of these matters most in an interview?"

Three groups, and the middle one carries the questions.

The strong framing. "The list is memorisable and the useful knowledge is which state answers which question. If a link never comes up, the state it stopped in tells you which stage of training failed — that is a bring-up question. If a link is up and misbehaving, the state is L0 and it tells you almost nothing, so the question moves elsewhere. And if the link keeps entering Recovery, that is a signal-integrity or margin question rather than a protocol one" (25.4).

Do not invent transition conditions, ordered sets, substates or timers (§1). An interviewer who asks for the exact exit condition from Polling is asking a specification question, and §14 is the answer.

8. "What Survives a Recovery?"

The advanced version of the question, and the answer separates layers.

60 seconds. "Recovery re-establishes the link's operational state — it retrains and returns to L0. It is a physical and link-layer event. What must not be affected is the transaction layer's outstanding work: a non-posted request issued before recovery is still owed a Completion afterwards, and the requester still has its tag allocated" (18.5, 10.4).

Then the distinction that makes this an advanced answer: "the link layer has its own retransmission mechanism with its own state — the replay buffer and sequence numbers — and that state is a different thing from transaction-layer outstanding requests. A recovery may cause a replay at the link layer without any transaction being reissued at the transaction layer. Conflating the two is where designs go wrong" (14.4, 14.2).

And the failure worth volunteering (25.7): "the failure mode is a requester that clears its outstanding table on a link event. The far end still owes a Completion and eventually sends it, and the requester either drops it as unmatched or — worse — matches it to a tag it has since reallocated."

9. "MSI-X — Why Does It Exist?"

The weak answer is "more vectors." The strong answer names three properties and the ordering one is the interesting part.

The 60-second answer. "MSI-X gives a device many independent interrupt vectors, each with its own address and data in a table, and each independently maskable — so a multi-queue device can direct completions for different queues to different CPUs without them interfering. But the property that matters architecturally is the one MSI already had: an MSI or MSI-X interrupt is delivered as a posted memory write. It is not an out-of-band signal. That means PCIe's ordering rules keep it behind the data writes that preceded it, which is what makes producer-consumer work" (26.1 §6).

The follow-up: "So can the interrupt arrive before the data is visible?"

"On the wire, no — and that is not sufficient." "The ordering holds for the stream the device emitted. Preserving it to the point where a core can observe the data is the Root Complex's obligation, and a design that gives the interrupt write a faster path to the interrupt controller than the data takes to memory delivers an interrupt for data that has not landed" (26.1 §7). That answer is a level above what the question asks for, and it is the correct one.

"MSI-X versus legacy INTx?" — "INTx was a level-sensitive out-of-band signal, shared and requiring the driver to determine the source. MSI and MSI-X are in-band memory writes, so they carry their own identification and are ordered with the data" (19.1, 19.5). Do not quote table size limits or vector counts (§1).

10. "The Link Is in L0 and the Device Is Broken. What Does L0 Tell You?"

Almost nothing, and being able to say that crisply is an advanced marker.

The answer. "L0 means the link trained and is operational. It says the physical and link layers are working. It says nothing about whether useful transactions are being made, whether the device's internal logic is functioning, or whether anything is being delivered correctly."

Three cases where L0 is true and the system is broken, each from a different layer:

And the framing that ties it together: "a healthy link is a precondition, not a diagnosis. The debugging question is always which service boundary stopped delivering, and the link state answers that only for the case where the link itself failed" (25.1).

11. Whiteboard RTL — Credit Accounting, Done Correctly

An advanced interview will ask you to write this. It is short and there are two ways to get it wrong.

// ILLUSTRATIVE. Credit accounting for one class. Two things are load-bearing:
// the single next-state expression, and the fact that "available" is DERIVED
// with a clamp rather than being a subtraction someone trusts.
localparam int CRED_TOTAL = 16;
 
logic [7:0] cred_consumed_q;   // consumed and not yet returned
logic [7:0] cred_avail;
logic       cred_violation_q;  // sticky — should be zero forever
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    cred_consumed_q  <= '0;
    cred_violation_q <= 1'b0;
  end else begin
    // ONE signed next-state expression. Separate if-branches would lose an
    // update whenever a send and a return land in the same cycle, and the
    // count would drift monotonically — exactly §5's leak, self-inflicted.
    cred_consumed_q <= cred_consumed_q
                     + 8'(tx_fire)            // a TLP was SENT (accepted)
                     - 8'(cred_return_fire);  // the receiver returned one
 
    // The invariant, checked rather than assumed (§5).
    if (cred_consumed_q > 8'(CRED_TOTAL)) cred_violation_q <= 1'b1;
  end
end
 
// Clamp rather than trust. If consumed ever exceeds total, an unsigned
// subtraction yields a huge value and can_send asserts with no credit —
// turning an accounting error into an overrun.
assign cred_avail = (cred_consumed_q >= 8'(CRED_TOTAL)) ? 8'd0
                                                        : (8'(CRED_TOTAL) - cred_consumed_q);
assign can_send   = (cred_avail != 8'd0);
 
// MANDATORY. English: credits consumed never exceed the configured total. This
// is §5's conservation invariant, and it fires on the cycle the accounting
// breaks rather than at the stall hours later.
a_credit_conservation: assert property (
  @(posedge clk) disable iff (!rst_n)
    cred_consumed_q <= 8'(CRED_TOTAL)
);
 
// MANDATORY. English: a TLP is never sent without an available credit.
// Catches the can_send gate being bypassed, and stays meaningful even if the
// clamp above is removed.
a_no_send_without_credit: assert property (
  @(posedge clk) disable iff (!rst_n)
    tx_fire |-> (cred_avail != 8'd0)
);

Architecture. One counter, a derived availability, a clamp, and a sticky violation bit.

State. cred_consumed_q only. Tracking available as a register instead is the more common shape and is worse — it has two writers and the same drift problem, and the total is no longer explicit.

Event. tx_fire is the accepted send, not an offer (27.2 §10's general rule); cred_return_fire is the receiver's update (16.6).

Contract. CRED_TOTAL must match what was advertised to the peer. A local constant that disagrees with the advertised value is a silent overrun or a permanent under-utilisation, and it is a configuration contract rather than an RTL one.

Failure. The two ways to get this wrong, and an interviewer may ask for both. Separate if branches for send and return lose an update on coincidence, and the count drifts upward until can_send never asserts — a self-inflicted version of §5's leak. And an unclamped CRED_TOTAL - consumed in unsigned arithmetic produces a large value when consumed exceeds total, so can_send asserts with no credit and the receiver is overrun — turning a stall into data loss.

DV/debug. cred_violation_q readable from silicon converts "the link wedged" into "credit accounting broke at least once before it wedged", which is a different and much shorter investigation.

12. Whiteboard — Construct a Credit Deadlock

"Give me a scenario where a credit-based link deadlocks with no error and no lost packet."

A strong answer builds it rather than describing it (25.8 owns the debug procedure).

The construction. "Take a receiver whose completion credit and posted credit come from one shared resource. A requester issues many reads, so many completions are owed. Meanwhile a burst of posted writes arrives and consumes the shared resource. Now the completions cannot be accepted — and the completions are what would free the requester's outstanding entries, which is what would let the posted traffic drain. Circular wait: nothing errors, nothing is lost, and the link stays in L0."

Then the fix, stated as a principle: "the class that frees resources must always be admittable. Separate credit accounting per class is exactly that guarantee (16.2–16.4), and where a shared resource is unavoidable, reserve a portion for the progress-critical class."

And the observation that makes it an advanced answer: "it needs sustained pressure from one class, so a functional test with light traffic never constructs it. The condition is a coverage item, not a directed test someone thinks of."

13. The Traps

And the honest pattern: five of the seven are cases where the intermediate answer is correct as far as it goes. Advanced grading is about the sentence after it.

14. When You Do Not Know — the Advanced Standard

At this level "I would check the specification" is necessary and insufficient. It must be paired with the part you can derive.

Three readings.

The pairing is what is being graded. A candidate who declines a value and then derives the consequence has demonstrated exactly the thing the value would have been a proxy for.

Fabrication costs more at this level, not less. An advanced interviewer is more likely to know the value and more likely to be testing calibration deliberately.

And if corrected, take the correction and use it. "That's useful — so given that, the consequence for the outstanding window is…" is a better outcome than having been right, because it demonstrates you can work with new information under pressure.

15. Readiness Markers

Ready for this level when, without preparation, you can:

• define a credit as permission, and state what it is not (§3)
• explain who owns the resource, who owns the consumption accounting, and how they stay consistent (§3)
• explain why credit types are separate — head-of-line blocking, not priority (§4)
• state the conservation invariant as an equation (§5)
• explain why a credit leak is monotonic and wedges hours later (§5)
• read a trace and distinguish "credit stall" from "link problem" (§6)
• say why the same trace cannot distinguish a leak from backpressure, and what would (§6)
• group the LTSSM states by which debug question each answers (§7)
• explain what survives a Recovery, and separate link-layer replay from transaction reissue (§8)
• name the failure of a requester that clears its outstanding table on a link event (§8)
• give three reasons MSI-X exists, and identify the ordering one as the architectural one (§9)
• explain why on-the-wire ordering is not sufficient for producer-consumer (§9)
• state three cases where L0 is true and the system is broken (§10)
• write credit accounting with one next-state expression and a clamp, and justify both (§11)
• construct a credit deadlock and state the principle that prevents it (§12)
• decline a value and derive the consequence in the same answer (§14)

Sixteen capabilities, and each cites the section that builds it. A pause on any one is a pointer to a chapter, not a reason to reread this page.

16. What Comes Next

This round covered the three mechanisms that decide whether a link keeps working. The last round asks whether you can own a subsystem.

27.4 — Senior Questions (planned) moves to DMA architecture, performance and silicon debug — the questions where an answer is judged on whether you could be handed the block. The shape changes accordingly: fewer questions with single answers, more scenarios where the correct response is to state what evidence you would need, what you could not conclude without it, and which contract you would insist on before writing RTL.