UCIe Flow Control

Chapters 25.3 to 25.5 were about identity and layering. This one is about resource ownership — and it is where junior and senior answers diverge most sharply, because the junior answer is not wrong, it is just not an answer.

1. What They Ask

"Explain UCIe credit-based flow control end-to-end."

Or: "When exactly do you decrement a credit?" · "What if a consume and a return happen in the same cycle?" · "How would you debug a link that's stuck at zero credits?"

The junior answer arrives immediately and is almost universal: "The sender has credits, decrements when it sends, and the receiver returns them."

That describes the mechanism and says nothing about why it is correct.

2. The One-Sentence Model

A credit is permission to consume one unit of someone else's capacity. Correctness is therefore conservation of ownership, not keeping a counter non-negative — a link can have a perfectly non-negative credit count and be badly broken.

Say that, and the rest of the interview is you explaining what "ownership" implies. Every trap in this chapter — the acceptance event (§8), the return point (§13), the replay question (§16), the recovery baseline (§17) — is a question about when ownership actually changes hands.

3. What They Are Really Testing

And the seventh row is the one that most often surprises candidates. "Flow control is correct and the bandwidth is 40% of what we expected" is a real, common situation — and a candidate who reaches for "credit leak" has misdiagnosed it.

4. What You Can Safely Assert

5. The Answer Ladder

15 SECONDS — the definition that is actually a model.
 
  "A credit is permission to use one unit of the receiver's capacity. The
   sender only sends when it holds permission, so the receiver never has
   to drop anything. The tricky part is that it's distributed state — two
   sides have to agree about who owns what."

60 SECONDS — adds the events and the trap.
 
  "Credit-based flow control means the receiver grants the sender
   permission in advance. The sender holds a count of how much remote
   capacity it's allowed to consume; when it consumes some, the count goes
   down, and when the receiver frees the capacity, it returns credit and
   the count goes back up.
 
   Two details matter more than the counter. First, when you decrement —
   it has to be on the event where the resource is actually committed,
   not just when you offered something. Second, when the receiver returns
   — it has to be when the storage is genuinely reusable, not when a
   pointer moved, or you're advertising capacity that isn't there yet.
 
   And the reason to do this rather than a simple ready/valid is that
   credits let you keep work in flight across a link with round-trip
   latency — the sender doesn't have to wait to find out whether there's
   room."
 
  [STOP — "keep work in flight" is the hook, §25.]

Three properties.

The 15-second version ends on "distributed state", which is the whole subject — and it invites the right follow-up rather than a definition check.

The 60-second version gives two events, not one. Most candidates describe the decrement and forget the return has an equally precise condition (§13).

And the last paragraph answers "why not just ready/valid?" before it is asked (§6), which is a standard follow-up.

6. Why Credits Rather Than a Handshake

And the architectural point: across a die-to-die boundary the round trip is not negligible, so a handshake that requires the sender to learn about space before sending caps throughput at one transfer per round trip. Credits decouple the permission from the transfer — which is exactly what §19 quantifies.

7. The Whiteboard Ledger

Draw this, then talk. It is the conservation statement that §2 promises.

CONCEPTUAL ownership ledger, per class/domain. This is a TEACHING model
of conservation — not a normative UCIe equation (§4).
 
  advertised capacity
      =  credits the sender still HOLDS          (free to use)
       +  capacity committed to work IN FLIGHT   (sent, not yet released)
       +  capacity OCCUPIED at the receiver      (arrived, not yet freed)
       +  returns IN FLIGHT back to the sender   (freed, not yet counted)
 
  Every unit of capacity is in exactly ONE of those four places.
 
  A LEAK  = a unit that is in none of them.
  AN INFLATION = a unit counted in two of them.

Four readings, and this is the drawing that wins the question.

Four places, and the boxes are the answer to "where did my credit go?" — §20's debug walk is just reading them in order.

"In flight" appears twice, in opposite directions, and forgetting the second is a classic error: a return that has been generated but not yet applied is still capacity that exists (21.4 §10).

A leak is an absence and an inflation is a duplicate — two different failures with two different signatures (§20, §21).

And this is why "the counter never went negative" is not a correctness argument (§2): a unit can be missing from the ledger entirely while every counter stays in range.

8. The Acceptance Event

Consume when the resource is actually committed — not when you offered, and not merely when bits left the wire.

Two properties.

Row 3 is the one that trips people who have read about retries. If consumption tracked transmission, every retransmission would consume another credit — which is §16's question answered by accident and probably wrongly.

And "usually yes" on row 2 is deliberate. The correct event is the architecture's allocation event; in most designs that coincides with the accepted handshake, and saying "the architecture's actual allocation event" rather than "valid and ready" is the more precise answer.

9. RTL — The Credit Counter, Done Correctly

// ILLUSTRATIVE ONLY. The centrepiece: a credit counter that survives a
// simultaneous consume and return. This is the code an interviewer is hoping
// you will write, and §10 is what most people write instead.
module credit_counter #(
  parameter int CRD_W    = 12,
  parameter int AMT_W    = 4,          // credits per event may exceed 1 (§12)
  parameter int CAPACITY = 64
) (
  input  logic              clk,
  input  logic              rst_n,
  input  logic              advertise_fire,      // (re)baseline — §17
  input  logic [CRD_W-1:0]  advertised_amount,
  input  logic              consume_fire,        // the ALLOCATION event (§8)
  input  logic [AMT_W-1:0]  consume_amount,
  input  logic              return_fire,         // storage is REUSABLE (§13)
  input  logic [AMT_W-1:0]  return_amount,
  output logic [CRD_W-1:0]  credit_q,
  output logic              underflow_err,
  output logic              overflow_err
);
 
  // ONE next-state expression, computed in SIGNED, EXTENDED arithmetic so the
  // error cases are detectable instead of wrapping into plausible values.
  // Width is CRD_W+2: one bit for the sum, one for the sign.
  logic signed [CRD_W+1:0] credit_next;
 
  always_comb begin
    credit_next = $signed({2'b00, credit_q})
                + (return_fire  ? $signed({{(CRD_W+2-AMT_W){1'b0}}, return_amount})  : '0)
                - (consume_fire ? $signed({{(CRD_W+2-AMT_W){1'b0}}, consume_amount}) : '0);
  end
 
  // A consume that would take the count below zero is a BUG, not a wrap.
  assign underflow_err = (credit_next < 0);
  // Credit above the advertised capacity means someone returned twice (§21).
  assign overflow_err  = (credit_next > $signed({2'b00, CRD_W'(CAPACITY)}));
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n)               credit_q <= '0;
    else if (advertise_fire)  credit_q <= advertised_amount;      // rebaseline
    else if (!underflow_err && !overflow_err)
                              credit_q <= credit_next[CRD_W-1:0];
    // else: HOLD, and let the error flags be observed. Silently clamping here
    // hides the bug and is worse than stopping.
  end
 
endmodule

Architecture. One counter, one next-state expression, and two error outputs that make the impossible cases observable rather than absorbed.

State. The credit count. That is deliberately all — the ledger's other three terms (§7) live elsewhere.

Event. consume_fire is the allocation event (§8). return_fire is the reusable-storage event (§13). advertise_fire rebaselines at an agreement boundary (§17).

Contract. The arithmetic is signed and two bits wider than the counter. That is not defensive padding: an unsigned decrement below zero wraps to a very large number, which then satisfies every "is there credit?" check and lets the sender flood the receiver. The width is what turns a silent corruption into a flag.

Failure. §10 is the classic. Also: clamping instead of flagging hides the bug and produces a link that appears healthy while conserving nothing.

DV/debug. underflow_err and overflow_err are the two most valuable silicon registers in a credit block (21.7 §21) — and credit_min_seen alongside them, because the current value tells you nothing about whether starvation ever happened.

10. Wrong RTL — Two Non-Blocking Assignments

// WRONG — the single most common credit bug, and it is invisible in review.
always_ff @(posedge clk) begin
  if (consume_fire) credit_q <= credit_q - 1'b1;
  if (return_fire)  credit_q <= credit_q + 1'b1;
end

What happens when both fire in the same cycle:

Five properties, and this is the answer an interviewer is listening for.

The last assignment wins. Both branches evaluate, both schedule a non-blocking update to the same register, and the second one is what lands — so the consume is silently discarded.

It only manifests when both events coincide, which is rare at low load and common at high load — the sender is consuming steadily while returns stream back. So it passes directed tests and fails in traffic.

The direction of the error is the dangerous one. Credit inflates, so the sender believes it has more permission than it does — and eventually overruns the receiver (21.3 §8). A deflating error would merely stall.

And it may look like an improvement first. More apparent credit means less throttling, so throughput rises before the receiver corrupts — which is 21.4 §25's inflation signature.

The fix is §9's single next-state expression, and the interview-worthy way to say it is: "any state with more than one source of update gets one next-state expression, not several assignments."

11. Widths, Overflow and Underflow

And the first row is the one to say aloud: "I'd compute the next state in signed arithmetic that's wider than the counter, so an underflow is a detectable condition rather than a very large positive number."

12. A Credit Event Is Not Always One Unit

A common simplification, and a real bug when it is wrong.

Worked, illustrative:

And the interview point: "If an object can consume more than one unit, I'd track both the event count and the amount sum — they detect different bugs, and neither one subsumes the other."

13. When May a Credit Be Returned?

When the storage is genuinely reusable — not when a pointer moved, not when a consumer read the data, and not when an occupancy counter decremented.

And the failure this prevents is a receiver advertising capacity it does not yet have — §14.

14. Wrong RTL — Returning Before the Storage Is Free

// WRONG — three consumers of "release", derived independently, one cycle apart.
assign occupancy_dec   = pop_fire;          // at pop
assign freelist_release = pop_fire_d1;      // ONE CYCLE LATER
assign credit_return   = pop_fire;          // at pop

Four properties.

For one cycle, the far end has been told an entry is available while the free-list still owns it.

Under light load nothing happens. The credit takes a round trip to be used, and by then the free-list has caught up. The bug is invisible.

Under heavy load with a short round trip, the replacement object arrives while the entry is still allocated — dropped, blocked, or written over an entry the design believes is in use.

And the fix is one shared expression, which is the interview answer: "define the storage-reusable event once, and derive the occupancy decrement, the free-list release and the credit return from that same signal" (21.4 §38) — because equalising the pipeline works until somebody adds a stage.

15. RTL — Returning Exactly Once

// ILLUSTRATIVE ONLY (§9). One allocation must produce at most one return.
// A per-entry guard is the cheapest way to make a duplicate unrepresentable.
logic entry_returned_q [N_ENTRIES];
 
assign entry_reusable_fire = pop_fire && read_complete[pop_idx];
 
// All three consumers derive from ONE expression (§14).
assign occupancy_dec    = entry_reusable_fire;
assign freelist_release = entry_reusable_fire;
assign credit_return    = entry_reusable_fire && !entry_returned_q[pop_idx];
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n)
    for (int i = 0; i < N_ENTRIES; i++) entry_returned_q[i] <= 1'b0;
  else begin
    if (credit_return)              entry_returned_q[pop_idx]   <= 1'b1;
    if (entry_alloc_fire)           entry_returned_q[alloc_idx] <= 1'b0;  // rearm
  end
end
 
// MANDATORY. English: an entry never returns credit twice for one allocation.
// Fires at the second return — the cycle inflation begins (§21).
a_return_once_per_alloc: assert property (
  @(posedge clk) disable iff (!rst_n)
    (credit_return && (pop_idx == 0)) |-> !entry_returned_q[0]
);

Architecture. A one-bit-per-entry guard that makes a duplicate return structurally impossible rather than merely unlikely.

State. N_ENTRIES bits.

Event. Set on return, cleared on allocation — so the guard is armed for each new use rather than being a one-shot.

Contract. The guard must be cleared at allocation, not at pop. Clearing at pop re-arms it in the same cycle it was set, and the guard does nothing.

Failure. Without it, 21.3 §18's five duplicate-return mechanisms — a level sampled twice, a pulse crossing a clock domain, a retransmitted return message, a pop-plus-flush, two structures sharing an index — each become an inflation.

DV/debug. In silicon, a first-duplicate register (21.4 §27) naming the offending entry converts "one duplicate exists somewhere" into a fix.

16. The Replay Question

// ILLUSTRATIVE ONLY. Classify the event so the DESIGN's behaviour is
// measurable, without asserting which rule is correct.
typedef enum logic [1:0] {
  ALLOC_NEW        = 2'd0,   // first transmission — requires new capacity
  ALLOC_RETRANSMIT = 2'd1    // a repeat of a committed object
} alloc_class_e;
 
logic [63:0] cnt_consume_new_q;
logic [63:0] cnt_consume_retx_q;
logic [63:0] cnt_attempt_retx_q;   // retransmissions ATTEMPTED

The diagnostic is a ratio. cnt_consume_retx / cnt_attempt_retx is 0.0 if retransmissions never consume, 1.0 if they always do, and anything between is an inconsistency in the design itself — which is a finding regardless of which rule is correct (21.4 §35).

17. Recovery and the Baseline

Distributed credit state must rebaseline consistently, and two failures are common.

And the defence is an epoch (21.4 §21):

// ILLUSTRATIVE ONLY. The return carries the epoch it was generated under.
// A stale return is REJECTED and COUNTED — never silently dropped, because
// a counted rejection is evidence and a silent drop looks like a loss.
logic [EPOCH_W-1:0] credit_epoch_q;
logic [31:0]        stale_return_cnt_q;
 
logic return_valid_this_epoch;
assign return_valid_this_epoch = (rx_return_epoch == credit_epoch_q);
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    credit_epoch_q     <= '0;
    stale_return_cnt_q <= '0;
  end else begin
    if (advertise_fire)  credit_epoch_q <= credit_epoch_q + 1'b1;
    if (rx_return_fire && !return_valid_this_epoch)
      stale_return_cnt_q <= stale_return_cnt_q + 32'd1;
  end
end
 
// The return only counts if its epoch matches.
assign return_fire = rx_return_fire && return_valid_this_epoch;

Architecture. An epoch that increments at each advertisement, carried on returns, with stale returns counted rather than dropped.

Event. The epoch advances at advertise_fire — the rebaseline, which is the only moment the agreement changes.

Contract. The counter matters as much as the rejection. Without it, a correctly-rejected stale return is indistinguishable from a lost one (21.4 §44) — and a debugger will report a transport problem that does not exist.

Failure. No epoch means §17's second row: a straggler inflates the new baseline, and the sender over-issues.

DV/debug. The distribution is the diagnosis: a burst of stale rejections immediately after a recovery is expected and drains; a steady stream during normal operation means the epoch is advancing when nothing should advance it.

18. Assertion Inventory

// MANDATORY. Illustrative properties (§4).
 
// (1) NEVER CONSUME WITHOUT CAPACITY — next-state form, because a same-cycle
// return may legitimately cover the consume. The registered form fires on
// correct hardware and gets deleted.
a_never_consume_without_capacity: assert property (
  @(posedge clk) disable iff (!rst_n) (credit_next >= 0)
);
 
// (2) RETURNS NEVER EXCEED CONSUMES. English: cumulative units restored can
// never exceed units charged. Fires in the PROFITABLE phase of an inflation
// (§21), thousands of cycles before the receiver overflows. The single
// highest-value credit property.
a_no_return_without_consume: assert property (
  @(posedge clk) disable iff (!rst_n)
    (amt_return_q <= amt_consume_q)
);
 
// (3) CREDIT NEVER EXCEEDS ADVERTISED CAPACITY.
a_credit_bounded: assert property (
  @(posedge clk) disable iff (!rst_n) (credit_q <= CRD_W'(CAPACITY))
);
 
// (4) CONSUME ONLY ON THE ALLOCATION EVENT (§8).
a_consume_on_alloc_only: assert property (
  @(posedge clk) disable iff (!rst_n) consume_fire |-> alloc_event
);
 
// (5) NO DUPLICATE RETURN PER ALLOCATION.  [§15]
 
// (6) STALE-EPOCH RETURN IS NOT APPLIED (§17).
a_stale_return_not_applied: assert property (
  @(posedge clk) disable iff (!rst_n)
    (rx_return_fire && (rx_return_epoch != credit_epoch_q)) |-> !return_fire
);
 
// (7) SIMULTANEOUS CONSUME AND RETURN NETS CORRECTLY — the §10 check.
a_simultaneous_nets: assert property (
  @(posedge clk) disable iff (!rst_n)
    (consume_fire && return_fire && (consume_amount == return_amount))
      |=> $stable(credit_q)
);

Two things worth saying about them in an interview.

Property (2) is the one to name if asked for a single assertion. It is two registers, needs no latency model, and fires while an inflation is still making throughput look good — which is the only window in which it is cheap to fix.

And property (1) must be the next-state form. Written against the registered value it fires on legal hardware whenever a return covers a consume in the same cycle, so it gets deleted — taking real over-allocation detection with it.

19. A Correct Link at Forty Per Cent

The follow-up that separates senior candidates, because the instinct is to say "leak."

ILLUSTRATIVE. Symbolic units; no UCIe figure is used (§4).
 
  required outstanding  ~=  target rate  x  round-trip reuse latency
 
  Target rate            1 object / cycle
  Credit round trip      20 cycles       (consume -> use -> release ->
                                          return generated -> transported
                                          -> applied)
  Required outstanding   ~20 objects
  Credit window          8
 
  Sustained rate  ~=  8 / 20  =  0.4 objects / cycle      -> 40%
 
  And the SHAPE on a trace:
     cycles 0-7    8 transfers back to back      credit 8 -> 0
     cycles 8-19   12 idle, waiting for returns  credit 0
     cycles 20-27  8 transfers                   credit 8 -> 0
     ...

Five readings.

Every credit is returned correctly and every conservation check closes. There is no leak, no duplicate, no misattribution — the window is simply smaller than the round trip requires.

The discriminating observation is whether credit recovers. A leak's ceiling ratchets downward and never returns to the full window; a window limit returns to exactly the full window every round trip (21.5 §27).

The sawtooth carries both parameters. Burst length is the window (8); period is the round trip (20) — readable straight off a trace.

Two independent fixes exist and they cost different things: enlarge the window (receiver buffering) or shorten the round trip (design effort in the return path).

And the sentence to say: "Before I looked for a leak I'd check whether credit returns to its full value each cycle. If it does, this isn't a leak — it's a bandwidth-delay product problem."

20. Debug — the Leak

Read staged counters in order and find the first boundary that diverges (21.4 §18).

ILLUSTRATIVE reading at quiescence, one class.
 
  consume            1000
  remote_alloc       1000     d=0    OK
  remote_release      850     d=150  <- this is OCCUPANCY, not a leak
  return_generated    850     d=0    OK
  return_transmitted  850     d=0    OK
  return_received     842     d=8    *** FIRST DIVERGENCE
  return_applied      842     d=0    OK
 
  CONCLUSION: 8 returns lost between transmit and receive.
  INVESTIGATE: the return path, its clock-domain crossing, any filter.
  DO NOT INVESTIGATE: the sender's decrement, the release logic, the
  return generator — all four closed above.

Three properties.

The 150 is occupancy, not a loss — and a tool that flags it reports a leak equal to the receiver's fill level (21.4 §9).

Reading in order matters, because a later mismatch is usually a consequence of an earlier one.

And the "do not investigate" list is half the value — four subsystems eliminated by four closed comparisons, each of which is a plausible hypothesis that would otherwise cost a day.

21. Debug — the Inflation

Four readings.

Phase 3 is the best throughput the link ever achieves and the sickest it has ever been — a performance regression run there records an improvement.

The symptom appears at the receiver, thousands of cycles after the cause at the sender's applier — so the investigation starts in the wrong place.

The discriminating observation is available in phase 2 and is arithmetically impossible: returns_applied > consumes means more returns applied than were ever generated. No latency term can make that legitimate.

And that is §18 property (2) — two registers, checkable every cycle, firing while the link still looks good.

22. Fifteen Follow-Ups

Q1 — "Why credits rather than ready/valid?" §6 — to keep work in flight across a round trip. A handshake caps you at one transfer per round trip.

Q2 — "When exactly do you decrement?" On the architecture's allocation event (§8), which usually coincides with an accepted handshake — not on valid, not on transmission.

Q3 — "What if valid stays high for four cycles?" Nothing is consumed. Consuming on valid allocates four times for one object (25.4 §9).

Q4 — "What if consume and return happen together?" §10 — one next-state expression. Two non-blocking assignments silently discard the consume and inflate credit.

Q5 — "How do you detect a leak?" §20 — staged counters read in order, and remember the alloc-minus-release difference is occupancy.

Q6 — "How do you detect inflation?" §18 property (2): returns can never exceed consumes. Fires in the profitable phase.

Q7 — "When do you return a credit?" When the storage is reusable (§13) — not on a pop, not on an occupancy decrement.

Q8 — "Does a retry consume again?" §16 — it depends on whether the far end retained the allocation, and I would confirm the revision. Meanwhile, classify the event and count both.

Q9 — "How does recovery affect credit state?" §17 — rebaseline consistently, use an epoch, and count stale rejections rather than dropping them.

Q10 — "How can correct flow control reduce bandwidth?" §19 — the window is smaller than the round trip requires. Check whether credit returns to its full value before concluding leak.

Q11 — "How many credits are enough?" Roughly target rate × round-trip reuse latency, plus margin (§19). It is a bandwidth-delay product, and the round trip must be measured rather than assumed.

Q12 — "What is the conservation equation?" §7's four places — held, in flight, occupied, returning. A leak is a unit in none; an inflation is a unit in two.

Q13 — "Can a credit event be more than one unit?" §12 — yes, and then event counts and amount sums detect different bugs.

Q14 — "How do you verify credits?" §18's properties, plus an independent model that does not consume the design's own consume signal (21.3 §10) — a shadow fed from the DUT agrees with the bug.

Q15 — "How would you debug a zero-credit hang?" Read the staged counters (§20), check whether credit ever recovers (§19), check credit_min_seen, and check the return age — because a missing event cannot be counted, only aged (21.4 §29).

23. Bad Answers

And the fifth row is the one worth rehearsing, because it is the most seductive: it sounds like a correctness argument and it is a range check.

24. Whiteboard Self-Check

Close the page. Reproduce all six:

If you can do all six you can answer §22's fifteen follow-ups. If you cannot do 1 or 5, those are the two an interviewer most reliably probes.

25. Controlling the Next Question

And the second row is the highest-value hook in Module 25 so far, because the honest answer — "it depends on retention, and I'd confirm the revision" — is simultaneously the accurate one and the one that demonstrates you know where the specification boundary sits (25.2 §12).

26. Understanding Check

27. Summary

Five things.

A credit is permission to consume remote capacity (§2). Correctness is conservation of ownership; the counter is bookkeeping.

Two events, both precise (§8, §13). Consume on the allocation event; return when the storage is reusable — and derive all three consumers of "release" from one expression.

One next-state expression, signed and extended (§9, §10). Two non-blocking assignments discard the consume, inflate credit, and fail only under load.

Do not guess the replay rule (§16). Both answers fail badly and in opposite directions — classify the event and count both.

And correct flow control can still cap bandwidth (§19). Check whether credit recovers before saying "leak"; the sawtooth's burst length is the window and its period is the round trip.