Congestion Handling

Chapter 13.1 built the permission to send. Chapter 13.2 built the storage that permission reserves. Chapter 13.3 built the propagation of the constraint upstream and proved that a fully backed-up pipeline is a legal steady state rather than a fault.

All three answer what is true. None of them answers what to do about it.

1. The One-Sentence Model

Congestion is a resource condition. Congestion handling is a policy decision. The two are separate, and conflating them is the mistake this chapter exists to prevent.

A queue reaching its high watermark is a fact — measurable, unambiguous, and produced by counters that 13.2 already built. What the hardware does next is a choice: stop admitting new work, prioritise draining, throttle one class, reserve capacity for progress-critical traffic, redirect arbitration, escalate a prolonged stall to something that reports itself, or simply wait.

None of those is implied by the fact. Each is a policy with a cost, and a design that has not chosen deliberately has chosen by accident.

2. What This Chapter Owns

13.1, 13.2 and 13.3 are referenced rather than repeated. This chapter owns the response.

Specifically new here: the four conditions that produce the same symptom and need different responses; per-layer congestion and why the response differs by layer; a hysteretic policy FSM with the escalation path; congestion age as a first-class diagnostic; arbitration policy as a congestion response, including the starvation it can create; drain mode; admission throttling and exactly when it is too late to throttle; retry-induced congestion, which is the bridge into Module 14; and congestion collapse and bufferbloat, the two system-level phenomena that make "add more buffering" the wrong instinct.

3. Sourcing

4. Four Conditions That Produce the Same Symptom

Start here, because the rest of the chapter is organised around telling these apart. All four present to an upstream observer as "I cannot send."

Four observations, and they set up the whole chapter.

The first row needs no response at all, and that is the single most common design error in this area. Backpressure asserting is not a problem; it is the evidence that flow control exists. A policy that reacts to every ready deassertion will thrash — §11 shows exactly how.

The distinguishing variable between rows one and two is duration, which is why §13 makes congestion age a register rather than an afterthought. Occupancy tells you the condition; age tells you which condition it is.

The distinguishing variable between rows two and three is whose progress, not how much. Aggregate throughput can look healthy while one class receives nothing. A single global occupancy counter cannot see this — 13.3 §21 established that per-class instrumentation is the only way.

And the distinguishing variable for row four is whether anything moves at all. 13.3 §19 proved the hard part: a deadlocked system generates no transitions, so every safety property holds vacuously and forever. Congestion policy cannot detect deadlock, because policy reads occupancy and occupancy is perfectly legal in a deadlock. That is why §14's escalation path exists — not to fix deadlock, but to make it visible.

A congestion controller that cannot distinguish these four conditions will apply the wrong response to at least three of them.

5. Congestion Is Per-Layer, and the Response Differs by Layer

"The link is congested" is not a diagnosis. Four distinct conditions wear that description, they have different owners, and the useful response differs in each.

Three consequences, and the third is the one engineers get wrong.

The lowest layer has the fewest options. A PHY that cannot accept a beat can only refuse it. Every meaningful congestion response lives above the point where the resource is scarce, because only an upstream agent can choose not to offer work. Congestion is detected where it happens and handled where the decision can be made — and those are different places, which is why the observation state of §6 must travel.

The remote-side row is the diagnostic trap. Local queues full plus credits at zero looks identical to local congestion, and every local remedy — deeper buffers, better arbitration, drain mode — does nothing for it, because the scarce resource is a buffer on the other die. 13.1 §22's per-domain taxonomy is the instrument: credits at zero with remote occupancy low means a return-path problem; credits at zero with remote occupancy high means the far consumer is genuinely slow. Those need opposite responses and the local symptom is the same.

And a response at the wrong layer can amplify. If the Protocol Layer responds to Adapter congestion by retrying its injection attempt more aggressively, it adds arbitration load to a resource that is already the bottleneck. The correct protocol-side response to downstream congestion is almost always to offer less, not to try harder.

6. Congestion Observation State

Before policy, observation. The condition needs four independent facts, and each has a different lifetime.

// ILLUSTRATIVE congestion observation record. Not a UCIe-defined structure.
//
// Four facts, four lifetimes:
//   active         — this cycle's condition          (combinational-ish, 1 cycle)
//   age            — how long the current episode has lasted (per episode)
//   peak_occupancy — the worst point of this episode  (per episode)
//   cause          — WHY it started, captured once    (per episode, first-only)
typedef struct packed {
  logic                active;
  logic [AGE_W-1:0]    age;
  logic [OCC_W-1:0]    peak_occupancy;
  logic [CAUSE_W-1:0]  cause;
} congestion_state_t;
 
congestion_state_t cg_q;

Architecture. A congestion controller cannot make a good decision from occupancy alone, because occupancy is one scalar sampled at one instant. The record above is the minimum that supports a policy decision and a post-silicon diagnosis — the condition, its persistence, its severity, and its origin.

State. Four fields with three distinct lifetimes. active is per-cycle. age and peak_occupancy are per congestion episode — they must be reset when an episode ends and not before. cause is first-only within the episode: overwriting it on each new contributing reason destroys exactly the information a debugger needs (§12's first-cause argument, which 13.3 §12 established for backpressure and applies unchanged here).

Cycle behaviour. age increments while active, saturating (§13). peak_occupancy takes a maximum. cause latches on the transition into the episode. All four are written by one process.

Contract. The policy FSM (§9) consumes active and age. The diagnostic path consumes peak_occupancy and cause. Those are different consumers with different requirements, which is why mixing them into one "congestion status" register loses value at both ends: the policy does not need peak occupancy, and the debugger cannot use a cause that has been overwritten.

Failure. The characteristic bug is resetting age on any dip below the threshold. A workload that oscillates around the high watermark then produces an episode that is really 4,000 cycles long but reports as a hundred episodes of 40 cycles. The escalation of §14 never triggers, and the log says the link is fine.

DV. Drive an occupancy profile that dips one cycle below the entry threshold mid-episode and check that age continues; check peak_occupancy equals the maximum of the driven profile; check cause holds the first contributing reason when three arrive in successive cycles.

7. Four Policy States, Not Two

A binary congested/not-congested flag cannot express the useful responses, because the useful responses are graded.

The important negative is in the last column of the CONGESTED row. Congestion is not an error, and the policy's job is not to make the traffic go away. There is no legitimate congestion response that discards work whose ownership has already transferred — §23 develops why, and it is the same argument 12.4 made about transaction lifetime.

And note that DRAINING is a distinct state rather than a phase of CONGESTED. The reason is entirely about hysteresis: the exit condition from congestion is not the negation of the entry condition, so the falling edge needs somewhere to live. §10 is that argument.

8. The Policy FSM

Read the figure for two structural properties rather than for the state names.

Every rising transition uses a higher threshold than the falling transition that reverses it. hi_warn against lo_warn, hi_cong against lo_cong. That asymmetry is the hysteresis, and §10 derives why omitting it is a real bug rather than a stylistic preference.

The escalation edge is the only one not driven by occupancy. It is driven by age. A mild congestion that never clears is more dangerous than a severe one that clears in twenty cycles, and no occupancy threshold can express that distinction.

9. The FSM RTL

// ILLUSTRATIVE congestion policy FSM — NOT a normative UCIe state machine.
// UCIe defines no congestion state machine (Section 3). Thresholds, the
// escalation limit, and the state encoding are all design choices.
typedef enum logic [2:0] {
  CG_NORMAL     = 3'd0,
  CG_PRESSURED  = 3'd1,
  CG_CONGESTED  = 3'd2,
  CG_DRAINING   = 3'd3,
  CG_ESCALATE   = 3'd4
} congestion_policy_e;
 
congestion_policy_e cg_state_q, cg_state_d;
 
// Occupancy comes from the buffer of Chapter 13.2 Section 5 — including its
// +1 width, because DEPTH itself must be representable.
wire at_hi_warn = (occupancy_q >= OCC_W'(HI_WARN));
wire at_hi_cong = (occupancy_q >= OCC_W'(HI_CONG));
wire lo_warn_ok = (occupancy_q <  OCC_W'(LO_WARN));
wire lo_cong_ok = (occupancy_q <  OCC_W'(LO_CONG));
wire aged_out   = (cg_q.age    >= AGE_W'(ESCALATE_AGE));
 
always_comb begin
  cg_state_d = cg_state_q;                     // default: hold
  unique case (cg_state_q)
    CG_NORMAL    : if (at_hi_warn) cg_state_d = CG_PRESSURED;
    CG_PRESSURED : if (at_hi_cong) cg_state_d = CG_CONGESTED;
                   else if (lo_warn_ok) cg_state_d = CG_NORMAL;
    CG_CONGESTED : if (aged_out)   cg_state_d = CG_ESCALATE;
                   else if (lo_cong_ok) cg_state_d = CG_DRAINING;
    CG_DRAINING  : if (at_hi_cong) cg_state_d = CG_CONGESTED;
                   else if (lo_warn_ok) cg_state_d = CG_NORMAL;
    CG_ESCALATE  : if (cause_cleared) cg_state_d = CG_DRAINING;
    default      : cg_state_d = CG_NORMAL;     // unreachable; recovers anyway
  endcase
end
 
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n) cg_state_q <= CG_NORMAL;
  else        cg_state_q <= cg_state_d;

Architecture. The policy is deliberately a separate state machine from the datapath. It reads occupancy and age; it drives admission and arbitration hints. It never sits in the ready path — 13.3 §5 established why a control decision must not lengthen the combinational path that carries backpressure, and a five-state FSM in that path would do exactly that.

State. One register, five encodings, per congestion episode in the sense that NORMAL is the resting value. Note the default arm returns to CG_NORMAL rather than holding: an FSM that reaches an unreachable encoding through an upset should recover to the least-restrictive legal state, not lock into the most restrictive one.

Cycle behaviour. unique case with a default — the combination is deliberate. unique gets the synthesis and simulation checking; default guarantees the register is assigned on every path, so no latch is inferred and no state is unreachable-and-sticky. The explicit cg_state_d = cg_state_q default means every arm expresses only its departures, which is what makes the FSM readable against Figure 1.

Contract. Two consumers, and they must read the same state. The admission gate (§22) reads it to decide whether to accept new objects; the arbiter (§17) reads it to decide whether to bias toward progress-critical classes. If those two read different copies — because one is registered and the other is not — the design can be throttling admission while still arbitrating for the class that caused the congestion.

Failure. Two, with different signatures. Omit the lo_* distinction and the FSM oscillates (§11). Omit the CG_DRAINING state and there is nowhere for the falling edge to live, so the design either exits congestion the instant occupancy drops one below the entry threshold — reintroducing the oscillation — or never exits at all.

DV. Cover every arc in Figure 1 including drain → cong, which is the arc that testbenches miss because it requires refilling during a drain. Assert the hysteresis property of §12. Check the escalation path with a stuck consumer.

10. Hysteresis, and Why the Thresholds Must Differ

The single most important line of RTL in this chapter is the one that makes the exit threshold lower than the entry threshold.

A policy whose entry and exit conditions are the same predicate has no state. It is a comparator, and it will change its mind as often as the input crosses the threshold.

13.2 §11 established the same principle for buffer watermarks and derived the ordering constraint. What 13.4 adds is the policy consequence, which is worse than the buffer consequence, because a policy decision has downstream effects that take cycles to take hold and cycles to undo.

The width of the gap is a real design parameter, and it trades two costs against each other:

The lower bound on a sensible gap is not arbitrary. It should exceed the amount of occupancy movement one round of the control loop can cause. If throttling admission takes L cycles to reduce arrival rate — and 13.3 §7 showed L counts registered boundaries in the control path — then occupancy can move by roughly rate × L before the policy's own action is visible. A gap narrower than that is a gap the policy cannot land inside, and the FSM will overshoot in both directions regardless of how carefully the two thresholds were chosen.

// ILLUSTRATIVE threshold derivation with the relationships made explicit
// rather than left to a comment. Values are design choices, not UCIe values.
localparam int DEPTH         = 32;
localparam int CTRL_LATENCY  = 3;                  // registered stages in the
                                                   // throttle path (13.3 §7)
localparam int RATE          = 1;                  // objects per cycle
localparam int LOOP_SWING    = RATE * CTRL_LATENCY; // occupancy the loop can
                                                   // move before it lands
localparam int HI_CONG       = DEPTH - LOOP_SWING - 2;   // enter congestion
localparam int LO_CONG       = HI_CONG - LOOP_SWING - 2; // leave congestion
localparam int HI_WARN       = LO_CONG - 2;              // early warning
localparam int LO_WARN       = HI_WARN - LOOP_SWING;     // clear the warning
 
// The ordering is a build-time requirement, not a runtime hope.
initial begin
  assert (LO_WARN >  0)            else $fatal(1, "LO_WARN underflow");
  assert (LO_WARN <  HI_WARN)      else $fatal(1, "warn hysteresis inverted");
  assert (HI_WARN <  LO_CONG)      else $fatal(1, "warn/cong regions overlap");
  assert (LO_CONG <  HI_CONG)      else $fatal(1, "cong hysteresis inverted");
  assert (HI_CONG <= DEPTH)        else $fatal(1, "HI_CONG exceeds DEPTH");
  assert ((HI_CONG - LO_CONG) >= LOOP_SWING)
    else $fatal(1, "hysteresis gap narrower than one control-loop swing");
end

Architecture. The thresholds are derived from the control-path latency, not chosen by taste. That is the same move 13.3 §8 made for the backpressure watermark, and it has the same payoff: when someone later adds a pipeline stage to the throttle path, CTRL_LATENCY changes and every threshold follows.

State. None — these are elaboration-time constants. That is the point.

Cycle behaviour. None at runtime. The initial assertions run once, at elaboration, and a violation is a build failure rather than a simulation failure that might not be exercised.

Contract. Four thresholds with a strict ordering that the FSM of §9 silently assumes. Encoding that assumption as executable assertions is the difference between a design that fails loudly when reparameterised and one that fails subtly.

Failure. Reparameterise DEPTH from 32 to 8 without these assertions and LO_WARN goes negative, wrapping to a huge unsigned value, so lo_warn_ok is true always and the FSM sits in NORMAL through total saturation. A congestion controller that cannot report congestion, with no simulation error.

DV. Run the elaboration checks under every parameter set the design claims to support — this is a compile-time regression, cheap to run and it catches the class of bug that survives functional testing.

11. Wrong RTL — One Threshold

// WRONG — a single threshold. This is a comparator with a state register
// bolted on, and it behaves like a comparator.
always_ff @(posedge clk) begin
  if (occupancy_q >= OCC_W'(THRESHOLD)) congested_q <= 1'b1;
  else                                   congested_q <= 1'b0;
end

With THRESHOLD = 8, a workload arriving at almost exactly the service rate produces this:

Four consequences, and only the last one is obvious.

Arbitration instability. If the congestion state biases arbitration, the bias flips every cycle. A rotating-priority arbiter whose weights change every cycle is not implementing any policy — the grant sequence becomes a function of the oscillation phase rather than of the request pattern. This can starve a class that happens to request on the wrong phase, which is a fairness bug produced entirely by a hysteresis bug.

Repeated throttling with no net effect. Each throttle assertion takes CTRL_LATENCY cycles to reduce the arrival rate, and it is released before it can take effect. The design pays the full control-path cost and gets none of the benefit.

Throughput loss with no correctness symptom. Every assertion passes. Occupancy never exceeds DEPTH. Nothing is dropped, nothing is duplicated, the scoreboard is clean — and useful throughput is measurably below what the same silicon achieves with a two-threshold policy. This is 13.3 §14's category exactly: a performance failure invisible to functional verification.

And the diagnostics become unreadable. The congestion episode counter reports thousands of episodes. age never exceeds one, so escalation never fires. A post-silicon log showing 40,000 congestion episodes of one cycle each is indistinguishable from noise, and the actual condition — sustained pressure for 40,000 cycles — is nowhere in the data.

The oscillation is not the bug's symptom. The oscillation is the bug, and every consequence above is downstream of it.

12. SVA — the Hysteresis Contract

// The policy contract: CONGESTED cannot clear until occupancy has actually
// fallen below the LOWER exit threshold. This is the property that fails on
// Section 11's single-threshold design and passes on Section 9's FSM.
property p_congestion_needs_exit_condition;
  @(posedge clk) disable iff (!rst_n)
    (cg_state_q == CG_CONGESTED) && (occupancy_q >= OCC_W'(LO_CONG))
      |=> (cg_state_q inside {CG_CONGESTED, CG_ESCALATE});
endproperty
a_congestion_needs_exit_condition:
  assert property (p_congestion_needs_exit_condition);
 
// The mirror: PRESSURED cannot clear above its own exit threshold either.
property p_pressure_needs_exit_condition;
  @(posedge clk) disable iff (!rst_n)
    (cg_state_q == CG_PRESSURED) && (occupancy_q >= OCC_W'(LO_WARN))
      |=> (cg_state_q != CG_NORMAL);
endproperty
a_pressure_needs_exit_condition:
  assert property (p_pressure_needs_exit_condition);
 
// And the one that catches a subtler error: entering congestion requires the
// UPPER threshold. A design that enters at LO_CONG has hysteresis in name only.
property p_congestion_entry_needs_upper;
  @(posedge clk) disable iff (!rst_n)
    (cg_state_q != CG_CONGESTED) && (occupancy_q < OCC_W'(HI_CONG))
      |=> (cg_state_q != CG_CONGESTED);
endproperty
a_congestion_entry_needs_upper:
  assert property (p_congestion_entry_needs_upper);

Architecture. Three properties that between them pin both edges of both hysteresis loops. The third is the one most often omitted, and it is the one that catches a design where somebody "simplified" the entry condition to reuse the exit comparator.

Why the first property permits CG_ESCALATE in its consequent. Escalation is a legal departure from CONGESTED that does not require the exit condition — it is driven by age (§9). Writing the consequent as cg_state_q == CG_CONGESTED would make the property fail on correct escalation behaviour, and the temptation would then be to weaken the property rather than fix it. A property that fires on correct behaviour gets disabled, and then it protects nothing.

Contract. These are safety properties: they say the policy never leaves a restrictive state too early. They say nothing about whether it ever leaves — that is liveness, and §20 handles it separately, for the reason 13.3 §19 made unavoidable.

DV. The properties must be exercised by a stimulus that actually parks occupancy between LO_CONG and HI_CONG for many cycles. Random traffic passes through that band without dwelling in it, so these three assertions can hold vacuously through an entire random regression. Directed dwell stimulus is mandatory — cover it explicitly (§31).

13. Congestion Age, Saturating

// ILLUSTRATIVE congestion age. Saturating, because this is a DIAGNOSTIC and
// POLICY counter, not a correctness counter. Contrast Chapter 13.1: a credit
// counter must NEVER saturate, because saturation there invents permission.
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    cg_q.age <= '0;
  end else if (!congestion_active) begin
    cg_q.age <= '0;                       // episode ended: forget it
  end else if (!(&cg_q.age)) begin        // saturate at all-ones
    cg_q.age <= cg_q.age + AGE_W'(1);
  end
end

Architecture. Age converts a condition into a classification. Occupancy at 30 of 32 says the buffer is nearly full; occupancy at 30 of 32 for 50,000 cycles says something is structurally wrong. The second statement requires a register and the first does not.

State. Per congestion episode. The !congestion_active arm is where §6's failure mode lives: if congestion_active is a raw threshold comparison rather than the FSM's notion of an ongoing episode, a one-cycle dip resets the age and the classification is destroyed.

Cycle behaviour. Three arms, one writer, saturation via &cg_q.age. The & reduction is the idiomatic all-ones test and it is width-independent, so reparameterising AGE_W cannot break it — unlike a literal comparison against a hard-coded maximum.

Contract. Two consumers with different needs. The escalation edge in §9 compares age against a limit that must be well below saturation, or escalation becomes a function of the counter width rather than of the design's intent. The diagnostic path reads the maximum age ever reached, which needs its own sticky register because age is cleared at the end of each episode.

Failure — and it is a subtle one. If ESCALATE_AGE is set at or near the saturation value, then once the counter saturates, aged_out is true forever within the episode, which is correct. But if it is set above the saturation value — easily done by reparameterising AGE_W downward — then aged_out can never be true and escalation is dead code. Guard it:

initial assert (ESCALATE_AGE < (2**AGE_W - 1))
  else $fatal(1, "ESCALATE_AGE unreachable for AGE_W — escalation is dead code");

Deliberately not invented. UCIe defines no congestion timeout, and ESCALATE_AGE is not derived from any specification value (§3). What the value should be is a system question — long enough that legitimate sustained load does not escalate, short enough that a genuine hang is reported before software gives up.

DV. Hold congestion for exactly ESCALATE_AGE - 1 and check no escalation; one more cycle and check escalation. Hold for the full saturation range and check the counter stops rather than wrapping — a wrapping age counter would clear aged_out mid-episode and de-escalate a hang, which is the worst possible behaviour.

14. Duration as Diagnosis

Age is not only a policy input. It is the primary diagnostic axis of this chapter, because the same occupancy means different things at different durations.

The bottom row is the one the policy cannot resolve and must not pretend to. A deadlock presents to a congestion controller as maximum occupancy with age increasing forever, and every response the policy has — throttle, prioritise, drain — is useless, because there is nothing to drain. Escalation exists precisely for this case: not to fix the condition, but to hand it to a mechanism that can.

Congestion policy is a rate-matching mechanism. It cannot resolve a structural circular dependency, and a design that tries to will simply throttle to zero and stay there.

The third row is the interesting one for debug, because four different faults land in it and they are cleanly separable by instrumentation the previous three chapters already built:

• slow consumer — remote occupancy high, credits returning slowly but returning;
• credit leak — credits at zero, remote occupancy low (13.1 §22's fingerprint);
• arbitration starvation — aggregate throughput healthy, one class's grant ratio near zero;
• raised error rate — retry counters climbing, and §25 is the mechanism.

15. Traffic Classes and Three Arbitration Policies

Arbitration is a congestion response. When a shared resource is oversubscribed, the choice of whose work proceeds is the most direct policy lever available.

Official material makes the urgency distinction real rather than hypothetical. UCIe 3.0 describes events "such as power down, wake-up, and low-latency telemetry data" requiring "high-priority notification over others, such as debug dump, which can be large bulk transfers" (§3). That is a latency-critical class contending with a bandwidth-heavy class — the exact conditions under which arbitration policy determines system behaviour.

The two failures are not symmetric, and this is the design insight.

Strict priority fails silently and unboundedly. Nothing overflows, no assertion fires, the high-priority class is served beautifully, and a lower class can wait forever. The failure is invisible at every FIFO boundary (§16).

Round robin fails visibly and boundedly. The latency-critical class waits — measurably, and by an amount you can compute from the number of classes and the maximum service quantum. A bounded, measurable latency degradation is a far better failure than an unbounded invisible one, which is why round robin is the correct default and strict priority is the thing you add deliberately, with a reservation or a deficit mechanism to bound its damage.

And UCIe's own preemption mechanism shows the third way. Rather than choosing between "bulk transfers block urgent events" and "urgent events get a dedicated channel", UCIe 3.0 interrupts an in-progress sideband packet at the next aligned 8 UI boundary and bounds the whole hop-to-hop transfer at 48 UI — 8 to reach the boundary, 8 for the clock-low switch indication, 32 for the priority packet — which at 800 MHz is 60 ns. The design pattern to extract is the bound, not the numbers: a preemption mechanism is valuable in proportion to how tightly you can state its worst case, and UCIe states its worst case as an arithmetic identity you can check (48 UI at 800 MHz is indeed 60 ns).

16. Wrong RTL — Strict Priority With No Reservation

// WRONG for a shared resource with more than one class that must progress.
// Correct only where a lower class genuinely may be starved indefinitely
// without breaking a system-level requirement — which is rarer than it looks.
always_comb begin
  grant = '0;
  for (int c = 0; c < NUM_CLASS; c++) begin
    if (request[c]) begin
      grant[c] = 1'b1;
      break;                      // first requester wins, always
    end
  end
end

The failure trace is boring, which is why it survives review. Class 0 requests continuously. Class 1 requests continuously.

Every FIFO remains within capacity. Class 1's queue is full and stays full — full is legal. Class 0's queue drains as fast as it fills. Occupancy assertions pass. The conservation scoreboard of 13.3 §24 balances exactly: nothing is lost, nothing is duplicated. Total throughput may even be at the theoretical maximum.

And the system has failed. If class 1 carries coherence responses, 11.3 §28's forward-progress classification is broken and 11.4 showed where that ends. If class 1 carries the completion for a transaction the far side is waiting on, 12.4's lifetime is unbounded and something times out.

Starvation is a liveness failure, and liveness failures produce no illegal states. This is the same reason 13.3 §19 gave for deadlock, and it is why safety verification alone cannot sign off an arbiter.

The relationship to congestion is direct. Strict priority is attractive precisely under congestion — when the resource is scarce, serving the important class first feels right. Congestion is therefore the condition under which the worst arbitration policy looks best, which is why the choice must be made at design time against the worst case, not tuned by watching a healthy link.

17. A Rotating-Priority Arbiter

// ILLUSTRATIVE rotating-priority (round-robin) arbiter. Grants the requester
// nearest above the last-granted index, wrapping. Bounded waiting: with N
// classes and one grant per cycle, any continuous requester is served within
// N grants.
module rr_arbiter #(parameter int N = 4) (
  input  logic                 clk,
  input  logic                 rst_n,
  input  logic [N-1:0]         request,
  input  logic                 transfer_done,   // the grant was CONSUMED
  output logic [N-1:0]         grant
);
  localparam int PTR_W = (N > 1) ? $clog2(N) : 1;
 
  logic [PTR_W-1:0] ptr_q;          // index that has HIGHEST priority now
  logic [N-1:0]     mask;           // 1 for indices at or above ptr_q
  logic [N-1:0]     masked_req;
  logic [N-1:0]     masked_grant, plain_grant;
 
  // Priority mask: everything from ptr_q upward is eligible in this round.
  always_comb begin
    mask = '0;
    for (int i = 0; i < N; i++)
      if (i >= int'(ptr_q)) mask[i] = 1'b1;
  end
 
  assign masked_req = request & mask;
 
  // Two fixed-priority picks, one masked and one not. The masked pick wins
  // when it has any requester; otherwise the search wraps to index 0.
  always_comb begin
    masked_grant = '0;
    for (int i = 0; i < N; i++)
      if (masked_req[i]) begin masked_grant[i] = 1'b1; break; end
    plain_grant = '0;
    for (int i = 0; i < N; i++)
      if (request[i])    begin plain_grant[i]  = 1'b1; break; end
  end
 
  assign grant = (masked_req != '0) ? masked_grant : plain_grant;
 
  // THE POINTER MOVES ONLY ON AN ACTUAL TRANSFER. This is the line that
  // distinguishes a working arbiter from Section 18's broken one.
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      ptr_q <= '0;
    end else if ((grant != '0) && transfer_done) begin
      // Advance past the granted index, wrapping.
      for (int i = 0; i < N; i++)
        if (grant[i]) ptr_q <= (i == N-1) ? '0 : PTR_W'(i + 1);
    end
  end
endmodule

Architecture. Fairness with one register. The mask-and-two-picks structure is the standard synthesisable rotating-priority form: the masked pick handles the common case, the unmasked pick provides the wrap, and the select between them is one comparison. It is deliberately not a shift-register-of-tokens design, which costs N flops instead of log2(N) and makes the "who is next" question harder to read in a waveform.

State. ptr_q — one pointer, lifetime is the arbitration epoch, meaning it persists across congestion episodes and across individual grants. It is not per-transaction and it is not per-congestion-episode. Resetting it when congestion clears would restart the rotation at index 0 every time, which biases toward low indices in exactly the workload where fairness matters.

Cycle behaviour. grant is combinational from request and ptr_q. ptr_q updates on the clock edge only when a grant was both issued and consumed. Under sustained backpressure, grant can be asserted for many cycles with transfer_done low, and the pointer must not move during those cycles.

Contract. Two obligations in opposite directions. The arbiter promises the requester that a continuous request is granted within N grant opportunities. The datapath promises the arbiter that transfer_done means the granted object actually moved. If transfer_done is wired to something weaker — the grant itself, or a valid without a ready — the fairness guarantee is void, and §18 is that bug.

Failure. Pointer advance without transfer (§18). Also worth naming: a mask computed from a stale pointer if ptr_q is bypassed for timing, which can double-grant an index within one rotation.

DV. Assert grant-implies-request (§19) and bounded fairness (§20). Cover the wrap case, the all-request case, the single-request case, and — the one usually missed — grant asserted for many cycles with transfer_done low, then finally high, checking the pointer moved exactly once.

18. Wrong RTL — Advancing the Pointer Without a Transfer

// WRONG — the pointer advances whenever a grant is asserted, whether or not
// the granted object actually moved.
always_ff @(posedge clk) begin
  if (grant != '0)
    for (int i = 0; i < N; i++)
      if (grant[i]) ptr_q <= (i == N-1) ? '0 : PTR_W'(i + 1);
end

What goes wrong, and it is worse than "slightly unfair". Consider N = 4, all four classes requesting continuously, and a downstream that accepts one object every four cycles.

Class 3 receives every transfer. Classes 0, 1 and 2 receive none. The pointer has become a function of the stall pattern rather than of the service history, and because the stall pattern is periodic, the bias is deterministic and total.

Three properties of this bug make it dangerous.

It reproduces the strict-priority failure from a design that looks fair. Code review sees a rotating pointer and stops looking. The starvation is structurally identical to §16's, but the RTL contains no priority ordering at all.

Its severity depends on the downstream duty cycle, which means it can pass every test where the downstream is fast and fail catastrophically in the one configuration where it is slow. A design validated at full downstream bandwidth will not see it.

And it can freeze rather than merely bias. If the accept pattern's period is a multiple of N, one index wins forever. If it is coprime with N, service rotates but not fairly. The behaviour is arithmetic, not random, so it will not average out over a long run.

Fairness state must be updated by the event it is accounting for. The pointer records "who was last served", so it must move when someone is served — not when someone is selected.

19. SVA — Grant Implies Request

// Mandatory. A grant to a non-requester wastes the slot and, if the datapath
// acts on it, moves an object that does not exist.
property p_grant_implies_request;
  @(posedge clk) disable iff (!rst_n)
    (grant != '0) |-> ((grant & ~request) == '0);
endproperty
a_grant_implies_request: assert property (p_grant_implies_request);
 
// Mandatory companion — at most one grant. A one-hot violation here means two
// classes believe they own the resource this cycle.
property p_grant_onehot0;
  @(posedge clk) disable iff (!rst_n)
    $onehot0(grant);
endproperty
a_grant_onehot0: assert property (p_grant_onehot0);
 
// And the arbiter's own progress obligation: if anyone requests, someone is
// granted. An arbiter that grants nobody while requests are pending is a
// silent throughput hole, not a safety violation.
property p_no_idle_with_requests;
  @(posedge clk) disable iff (!rst_n)
    (request != '0) |-> (grant != '0);
endproperty
a_no_idle_with_requests: assert property (p_no_idle_with_requests);

Architecture. Three properties covering the three ways an arbiter can be wrong in a single cycle: granting the wrong agent, granting two agents, and granting nobody.

Why the third one matters more than it looks. p_no_idle_with_requests is a safety property that catches a performance bug — the mask-and-wrap logic of §17 has exactly one way to fail this, which is if the plain_grant fallback is omitted or mis-selected. Then a request set entirely below ptr_q gets nothing, the arbiter idles, and the only symptom is throughput. It is worth having a cheap safety assertion that catches a performance bug, because the performance bug otherwise requires a benchmark to find.

Contract. These constrain the arbiter in isolation and need no reference model. They are the assertions to write first and to keep enabled in every regression.

Failure they catch. A bypassed or mis-timed mask produces grant for an index that stopped requesting this cycle — a real hazard when request is combinational from a queue's empty signal.

DV. All three should be exercised by random request patterns, which cover them well. The properties in §20 are the ones random stimulus cannot reach.

20. SVA — Bounded Fairness, With Assumptions

// LIVENESS, explicitly. This property is only meaningful under the two
// assumptions below, and it is unprovable without them.
//
//   A1: the class requests continuously (it does not withdraw)
//   A2: the downstream makes progress (transfer_done occurs)
//
// Without A2 this property MUST fail, and correctly so: a resource nobody can
// use cannot be shared fairly. Without A1 the property is meaningless, because
// a class that withdraws its request is not waiting.
localparam int FAIR_BOUND = N * MAX_SERVICE_CYCLES;
 
// A2 as an explicit assumption — in formal it constrains the environment; in
// simulation it is an obligation on the testbench.
assume property (@(posedge clk) disable iff (!rst_n)
  (grant != '0) |-> ##[0:MAX_SERVICE_CYCLES-1] transfer_done);
 
property p_bounded_fairness(int c);
  @(posedge clk) disable iff (!rst_n)
    request[c] throughout (##[1:FAIR_BOUND] 1'b1)
      |-> ##[1:FAIR_BOUND] (grant[c] && transfer_done);
endproperty
 
generate for (genvar c = 0; c < N; c++) begin : g_fair
  a_bounded_fairness: assert property (p_bounded_fairness(c));
end endgenerate

Architecture. One property per class, bounded rather than eventual. A bounded liveness property is checkable in simulation and in bounded formal; an unbounded s_eventually is neither, in practice.

Why the bound is N × MAX_SERVICE_CYCLES and not N. The rotation guarantees service within N grants, not within N cycles. Converting one to the other requires knowing how long a grant can be held, which is a datapath property, not an arbiter property. Writing the bound as N is the most common error in this assertion, and it produces failures on a correct arbiter with a slow downstream — after which the assertion usually gets deleted.

State. None of its own. It reads the arbiter's interface only, which is what makes it portable across arbiter implementations — swap §17's design for a deficit-weighted one and this property still expresses the requirement.

Contract. The assume is doing real work and must be stated, not implied. This is the classification that matters most on a review: p_grant_implies_request is safety and holds unconditionally; p_bounded_fairness is liveness and holds only under an environment assumption. Mixing them in one list, unlabelled, leads to someone "fixing" a fairness failure that is actually a downstream-stall failure.

Failure it catches. §18's pointer bug, immediately and with a clean counterexample — which is the argument for writing this property even though it costs more thought than the safety ones.

DV. Needs directed stimulus: all classes requesting continuously, with a downstream duty cycle swept across values coprime and non-coprime with N. Random traffic will not hold request[c] continuously for FAIR_BOUND cycles, so the antecedent rarely matches and the property passes vacuously. Cover the antecedent, not just the property (§31).

21. Drain Mode

Once congestion is established, the most useful thing a design can do is often not "throttle harder" but "change what it prioritises."

Drain mode is a policy state in which the design temporarily optimises for reducing backlog rather than for accepting new work. Three components, and each is a separate decision:

The second row's cost is the one to internalise. Draining oldest-first is not free: if the oldest entry is blocked on something the drain cannot influence — a credit that has not returned, a response that has not arrived — then insisting on serving it first converts a partial stall into a total one. That is head-of-line blocking chosen deliberately, and 13.2 §15 established that the structural fix is separate queues rather than a cleverer scheduler.

And the first row's cost is the reason §5 exists. Stopping admission does not remove congestion; it relocates it upstream. In a layered stack that is often exactly right — pressure is cheaper to hold in the Protocol Layer's own queues than in the Adapter's, and holding it upstream keeps the scarce resource free for progress-critical traffic. But "the congestion went away" and "the congestion moved to somewhere I am not instrumenting" look identical, which is why per-layer occupancy instrumentation is not optional.

Not a UCIe mechanism. Drain mode as described here is generic queueing architecture. No official source establishes any UCIe drain behaviour (§3).

22. Admission Throttling, and the Gate It Must Not Replace

// ILLUSTRATIVE admission decision. TWO independent conditions, deliberately
// kept separate:
//
//   resource_available — CORRECTNESS. Accepting without this overflows a
//                        buffer or over-commits a credit (13.1 Section 14,
//                        13.2 Section 8). It is never optional.
//
//   policy_allows      — POLICY. Congestion handling. This is advisory: a
//                        design that ignores it is slow, not broken.
//
// Collapsing these into one signal loses the distinction and, worse, invites
// someone to "optimise away" the correctness half.
assign policy_allows = (cg_state_q != CG_CONGESTED) &&
                       (cg_state_q != CG_DRAINING)  &&
                       (cg_state_q != CG_ESCALATE);
 
assign admit_new = resource_available   // correctness gate — mandatory
                && policy_allows;       // congestion policy — advisory

Architecture. Two gates, one AND, and a comment explaining which is which. The separation is the entire content of this section.

State. None here; both inputs are derived. resource_available is the conjunction 13.1 §14 built — credit, queue space, replay space, tracking entry, each named separately. policy_allows is the FSM of §9.

Cycle behaviour. Combinational. Note that admit_new gates the decision to accept, which must be evaluated before the handshake completes (§23).

Contract. The critical asymmetry: a false negative on policy_allows costs throughput; a false negative on resource_available costs nothing; a false positive on resource_available corrupts state. These are not comparable risks, and code that treats them as one "can I accept?" signal has erased the difference. When someone later profiles the design and finds admit_new on the critical path, the safe optimisation is to pipeline policy_allows — never resource_available.

Failure. Two, in opposite directions. Drop resource_available and rely on the congestion policy to keep occupancy safe: the policy has hysteresis and a control-path latency, so it will admit past DEPTH during a burst and overflow. Drop policy_allows and the design is correct but has no congestion response at all — which is the honest baseline, and preferable to the first error.

DV. Assert the correctness gate independently of policy: admit_new |-> resource_available, unconditionally. Then verify the policy gate is ignorable — force policy_allows high throughout a congestion episode and confirm nothing overflows, because that is what proves the two gates are genuinely independent rather than one gate written twice.

// The gate that must hold regardless of any policy decision.
property p_admit_requires_resources;
  @(posedge clk) disable iff (!rst_n)
    admit_new |-> resource_available;
endproperty
a_admit_requires_resources: assert property (p_admit_requires_resources);

23. Wrong Design — Throttling After Ownership Has Transferred

// WRONG — the object was accepted last cycle, and the congestion policy now
// tries to un-accept it.
always_ff @(posedge clk) begin
  if (accepted_q && (cg_state_q == CG_CONGESTED))
    drop_object <= 1'b1;          // ownership already transferred. Too late.
end

Why this is not a policy choice but a correctness violation. 12.4 established that a completed handshake transfers ownership: after valid && ready, the producer has been told the object is the consumer's responsibility and has released whatever state it was holding for it. The producer may have freed its queue entry, released a credit's accounting, or retired the source transaction.

So dropping it later creates a loss with no owner. Nobody is retrying it, because the producer believes it was delivered. Nobody is waiting to be told, because there is no mechanism to un-accept. The transaction that generated it waits for a response that will never come, and 12.4's timeout is the only thing that eventually notices — a timeout that reports the wrong layer, because the loss happened in the Adapter and the symptom appears at the Protocol Layer.

Congestion policy operates on the admission decision, not on accepted work. The last cycle at which a design may decline an object is the cycle before its handshake completes.

Three corollaries worth stating.

This is why admit_new (§22) is combinational into the accept decision rather than registered. A registered policy signal is a policy that is always one cycle stale, and one cycle stale is exactly enough to accept an object the policy would have refused.

A legitimate discard needs a protocol that says so. Where a system genuinely must shed load, that has to be an architected behaviour with a defined notification — a negative acknowledgement, a poison indication, an error response — so the producer learns the object did not survive. Silent discard is never a valid congestion response. Nothing in official UCIe material describes any such shedding mechanism for the flit path, and none is invented here.

And "drop" is not made safe by dropping only low-priority traffic. The priority of an object says how urgently it should be served, not whether anyone is waiting for it. Something is always waiting.

24. Oldest-First, and What It Costs

Under congestion, selecting which backlogged entry to serve is a second-order policy with first-order effects on tail latency.

// ILLUSTRATIVE age tracking for an oldest-first drain policy. A monotonically
// increasing sequence number stamped at enqueue is cheaper and more robust
// than per-entry counters, because nothing has to be incremented every cycle.
logic [STAMP_W-1:0] stamp_ctr_q;                  // free-running enqueue stamp
logic [STAMP_W-1:0] entry_stamp_q [DEPTH];
 
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n)      stamp_ctr_q <= '0;
  else if (push)   stamp_ctr_q <= stamp_ctr_q + STAMP_W'(1);   // wraps: fine
 
always_ff @(posedge clk)
  if (push) entry_stamp_q[push_idx] <= stamp_ctr_q;
 
// Relative age via modular subtraction — correct across a wrap, unlike a
// plain magnitude comparison of two stamps.
function automatic logic [STAMP_W-1:0] age_of(logic [STAMP_W-1:0] s);
  return stamp_ctr_q - s;
endfunction

Architecture. One free-running counter plus one stamp per entry, and the oldest entry is the one with the greatest age_of. The alternative — an age counter per entry, incremented every cycle — costs DEPTH adders and buys nothing.

State. stamp_ctr_q has arbitration-epoch lifetime and is deliberately allowed to wrap. entry_stamp_q has per-entry lifetime, valid only while the entry is occupied.

Cycle behaviour. The stamp counter advances on push only, not every cycle. That means "age" here measures entries enqueued since, not cycles elapsed — a subtly different quantity that is usually the more useful one for fairness, and the wrong one if what you care about is a timeout. Know which you are measuring.

Contract. The comparison must be modular. stamp_ctr_q - s is correct across a wrap for any distance under 2**STAMP_W; comparing two raw stamps with > is not, and produces an inverted ordering on the wrap. That constrains STAMP_W: it must exceed $clog2(DEPTH) with margin, so no two live entries can be more than half the stamp space apart.

Failure. Undersized STAMP_W inverts the age ordering periodically, so the "oldest-first" policy occasionally serves the newest entry. Symptom: tail latency with a periodic spike, and a drain policy that appears to work on average.

The cost of the policy itself, stated plainly. Oldest-first bounds worst-case residency and therefore reduces timeout risk, which is why it is the right drain policy. But it converts the oldest entry into a head-of-line blocker by construction: if that entry cannot proceed, nothing behind it is considered, even when a younger entry could have been served immediately. Throughput drops while the policy waits for the one thing it has committed to serving first.

Which means the choice is explicit. Oldest-first optimises tail latency at the cost of throughput; newest-first or any-ready optimises throughput at the cost of tail latency. Under a timeout regime the first is usually correct; under a bandwidth regime the second is. 13.2 §15's point stands as the escape from the dilemma: separate queues remove the coupling that forces the choice.

25. Congestion From the Reliability Path

Here is the mechanism that makes this chapter's connection to Module 14 more than a cross-reference.

Retransmission consumes the same resources as new work. Where a link-level retry mechanism is present, a replayed object occupies replay-buffer residency, arbitration opportunities, and link bandwidth — while, per 13.1 §12, correctly consuming no additional credit, because it is the same object.

Congestion can therefore originate entirely in the reliability layer, with the incoming semantic load completely unchanged.

That sentence is worth pausing on, because it breaks the mental model most engineers bring to congestion. Congestion is normally understood as demand exceeding capacity: more work arrived. Here, no additional work arrived at all. The offered transaction rate is identical; the error rate rose; and useful throughput fell.

Four resources are affected, and they are not affected equally:

The last row is a diagnostic gift. Because a correct design does not consume extra credit on replay, the signature of retry-induced congestion is distinctive: replay occupancy high, retry counters climbing, credits normal, offered load unchanged. Contrast 13.1 §12's extra-consume bug, whose signature is credits falling with remote occupancy low. Those two are adjacent conditions with opposite credit behaviour, which is exactly what makes the credit counter a useful discriminator here.

And there is a verified reason this matters more at higher rates. The UCIe 3.0 white paper states the target "bit error rate (BER) is 10⁻¹⁵ for 48 GT/s and 10⁻¹² for 64 GT/s", and that this "ensures that data transmission remains reliable using existing CRC and replay mechanisms, even at increased speeds." Those two figures differ by three orders of magnitude across a single generation step. The mechanism is unchanged; the rate at which it is exercised is not. A replay path sized for the 48 GT/s error rate is being asked to absorb roughly a thousand times more events at 64 GT/s — which is a sizing input, and one of the few places in this chapter where a real number is available.

26. Congestion Amplification

The individual effects of §25 compose into a positive feedback loop, and the loop is the reason error-rate problems present as throughput collapses rather than as error counts.

1. Error rate rises — a marginal channel, a thermal excursion, a rate change.
2. Retries increase. Each error costs at least one retransmission.
3. Replay entries stay occupied longer. An entry cannot retire until delivery is confirmed, and confirmation is now delayed by the retransmission itself.
4. Replay space becomes the binding resource. 13.1 §14's acceptance gate is a conjunction, and this term goes false first.
5. Admission stalls — new objects cannot be accepted even though credits are available and queues have space.
6. Upstream queues grow, and 13.3 §10's wave propagates the stall backward.
7. Congestion policy engages, throttling admission further.
8. Useful throughput collapses — while the link is busy, transmitting mostly retransmissions.

Three properties of this chain make it hard to diagnose from the far end.

The symptom is at step 8 and the cause is at step 1, seven layers of mechanism apart, and every intermediate step looks like a legitimate response to its own input. Nothing in the chain is a bug.

The link looks busy, not idle. Utilisation instrumentation shows high occupancy of the physical path, which reads as "we are at capacity" rather than "we are retransmitting." Distinguishing them requires counting useful flits separately from transmitted flits — which is 13.5's useful-throughput counter, and the single most valuable performance counter a link can have.

And the congestion policy makes it worse, correctly. Step 7 is the policy doing its job: pressure exists, so throttle. Throttling reduces new admissions, which does nothing about the retransmissions occupying the resource. A congestion response that cannot distinguish retransmission from new work will throttle the wrong traffic, which is the argument for a per-reason stall attribution (13.3 §11) rather than a single congestion flag.

Read the chain in reverse when debugging: a throughput collapse with a busy link and a clean scoreboard should send you to the retry counters before the queue sizes.

27. Congestion Collapse Versus Stable Saturation

Two conditions with a similar occupancy signature and completely different meanings.

The definition worth memorising:

Stable saturation: useful throughput stops rising with offered load. Congestion collapse: useful throughput falls as offered load rises.

What makes throughput fall rather than plateau? Every mechanism that consumes a shared resource without delivering work:

• retransmission (§25) — bandwidth spent on objects already sent;
• queue churn — objects occupying residency without progressing, so the same capacity serves fewer completions;
• arbitration overhead — more contenders per grant, so a larger fraction of cycles is spent selecting rather than transferring;
• starvation — a class that is never served still occupies its queue, so its share of the buffer is permanently unavailable;
• excessive buffering (§28) — more in-flight work, longer residency, later feedback.

Note that all five are resource consumption without progress. That is the general definition of the amplifier, and it is the thing to look for when a curve turns over.

The engineering framing is deliberately generic. Congestion collapse is a classical result from packet networks, and it is described here as engineering rather than as UCIe behaviour. No official source establishes any UCIe congestion-collapse characteristic, and no numbers are claimed. What transfers is the measurement discipline: plot useful throughput against offered load, and if it turns over, find which of the five is consuming the difference.

28. Bufferbloat — Why More Buffering Is Not the Answer

The instinctive response to congestion is a deeper queue. It works, for one metric, and hurts four others.

The third and fourth rows are the trap. Adding depth reliably improves the metric people measure first — stall cycles — while degrading the metric users actually feel. A design tuned by minimising stall counters will grow its queues until latency is bad and nobody has noticed, because no counter got worse.

The fifth row is the subtle one and it interacts with everything else in this chapter. 13.3 §7 established that backpressure must begin rate × latency before capacity is exhausted. Deeper buffers do not change that arithmetic — but they do mean the upstream sees pressure later, so a deeper queue delays the moment the congestion policy engages, which is precisely the opposite of what §10's control-loop analysis wants. Depth and responsiveness trade directly.

The sixth row is the one that costs debug time in silicon. A consumer that drains at 90% of the required rate will eventually back up regardless of depth. With a shallow queue, that shows up in seconds, near the cause. With a very deep queue, it shows up much later under a different workload, and the deep queue has meanwhile absorbed and hidden the evidence.

Buffering converts a throughput problem into a latency problem. That is sometimes the right trade, and it is never a fix.

The right question is not "how deep?" but "deep enough for what?" 13.2 §13 gives the inputs: the burst length to absorb, and the round-trip latency to cover. Depth beyond the larger of those two buys latency and area, and nothing else.

29. First Cause, Peak, and Duration

The diagnostic state that makes a congestion episode reconstructable after the fact.

// ILLUSTRATIVE congestion diagnostics. Four quantities, four different
// reasons for existing. This is the record a post-silicon debugger needs and
// the one most designs discover they are missing.
typedef enum logic [2:0] {
  CGC_NONE       = 3'd0,
  CGC_LOCAL_FULL = 3'd1,   // our own queue hit the watermark
  CGC_NO_CREDIT  = 3'd2,   // remote storage unavailable (13.1)
  CGC_REPLAY     = 3'd3,   // replay space exhausted (Section 25)
  CGC_ARB_LOSS   = 3'd4,   // lost arbitration repeatedly (Section 17)
  CGC_DOWNSTREAM = 3'd5    // next stage not ready (13.3)
} cg_cause_e;
 
cg_cause_e          first_cause_q;      // FIRST reason, this episode
cg_cause_e          current_cause_q;    // reason right now
logic [OCC_W-1:0]   peak_occ_q;         // worst occupancy, this episode
logic [AGE_W-1:0]   max_age_q;          // longest episode ever seen — STICKY
logic [15:0]        episode_count_q;    // how many episodes — saturating
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    first_cause_q   <= CGC_NONE;
    current_cause_q <= CGC_NONE;
    peak_occ_q      <= '0;
    max_age_q       <= '0;
    episode_count_q <= '0;
  end else begin
    current_cause_q <= cause_now;                       // always current
 
    if (episode_start) begin
      first_cause_q <= cause_now;                       // latch once
      peak_occ_q    <= occupancy_q;
      if (!(&episode_count_q)) episode_count_q <= episode_count_q + 16'd1;
    end else if (cg_q.active) begin
      if (occupancy_q > peak_occ_q) peak_occ_q <= occupancy_q;
    end
 
    // Sticky across episodes — the worst duration the part ever saw.
    if (cg_q.age > max_age_q) max_age_q <= cg_q.age;
  end
end

Architecture. Five registers that between them answer the five questions a congestion bug report needs: what started it, what is sustaining it, how bad it got, how long the worst one lasted, and how often it happens.

State — and the lifetimes are all different, which is the point. current_cause_q is per-cycle. first_cause_q and peak_occ_q are per-episode. max_age_q and episode_count_q are sticky for the life of the part, because they are what you read out after the fact.

Cycle behaviour. first_cause_q is written only on episode_start. That single else if is the whole mechanism, and it is the line that gets refactored away by someone simplifying the block.

Contract. First cause and current cause must both exist, because they answer different questions and they frequently differ. 13.3 §12 made this argument for backpressure reasons; the congestion case is stronger, because a congestion episode is long enough that the current cause has usually changed several times. An episode that started on CGC_NO_CREDIT and is now reporting CGC_LOCAL_FULL will send a debugger to the wrong die.

Failure. Keeping only the current cause. The episode is real, the duration is real, the peak is real, and the reported cause is whatever happened to be true when someone read the register — which, in a long episode, is almost never the cause.

And one thing to note about episode_count_q saturating. It is a diagnostic, so saturation is safe and wrapping is not: a wrapped episode count can read as a small number during a storm. Contrast 13.1's credit counter, where saturation would invent permission and is therefore forbidden. The rule generalises: saturate diagnostics, never saturate accounting.

30. The Congestion Scoreboard

// Verification-only reference model for congestion policy. Not synthesisable.
//
// The scoreboard's job is NOT to re-derive occupancy — 13.2's scoreboard does
// that. It is to check that the POLICY is a legal function of the observed
// resource state, which is a different claim.
class congestion_scoreboard;
 
  // Observed, independently of the DUT's own registers
  int unsigned  ingress_count;
  int unsigned  egress_count;
  int unsigned  model_occupancy;      // ingress - egress, recomputed
  int unsigned  grants_per_class[NUM_CLASS];
  int unsigned  requests_per_class[NUM_CLASS];
  cg_cause_e    observed_first_cause;
  int unsigned  episode_len;
 
  // ---- Check 1: conservation. Nothing congestion policy does may lose work.
  function void check_conservation(int unsigned dut_occupancy);
    model_occupancy = ingress_count - egress_count;
    if (model_occupancy != dut_occupancy)
      $error("occupancy divergence: model %0d, dut %0d — policy lost or invented work",
             model_occupancy, dut_occupancy);
  endfunction
 
  // ---- Check 2: the policy state is legal for the observed occupancy.
  //      This catches an FSM that entered CONGESTED without cause, and an
  //      FSM that stayed there after the exit condition was met.
  function void check_policy_legal(congestion_policy_e s, int unsigned occ,
                                   int unsigned age);
    if ((s == CG_CONGESTED) && (occ < LO_CONG) && (age < ESCALATE_AGE))
      $error("policy in CONGESTED with occ %0d below LO_CONG and no escalation", occ);
    if ((s == CG_NORMAL) && (occ >= HI_WARN))
      $error("policy in NORMAL with occ %0d at or above HI_WARN", occ);
  endfunction
 
  // ---- Check 3: class service under the stated assumptions.
  //      Reported rather than asserted, because the fairness CLAIM is a
  //      bounded-liveness property (Section 20) and belongs in SVA. What the
  //      scoreboard adds is the DISTRIBUTION, which SVA cannot express.
  function void report_fairness();
    foreach (grants_per_class[c]) begin
      if (requests_per_class[c] > 0 && grants_per_class[c] == 0)
        $error("class %0d requested %0d times and was never granted — starvation",
               c, requests_per_class[c]);
      $display("class %0d: requests %0d grants %0d ratio %0.3f", c,
               requests_per_class[c], grants_per_class[c],
               real'(grants_per_class[c]) / real'(requests_per_class[c] + 1));
    end
  endfunction
 
  // ---- Check 4: first cause matches what the testbench actually caused.
  function void check_first_cause(cg_cause_e dut_first);
    if (dut_first != observed_first_cause)
      $error("first cause mismatch: dut %s, actually %s — cause is being overwritten",
             dut_first.name(), observed_first_cause.name());
  endfunction
 
endclass

Architecture. Four checks with four different characters, and separating them is the design of the scoreboard.

Check 1 is safety and absolute. No congestion policy may lose or invent work. This is the check that catches §23's discard: the model's occupancy and the DUT's diverge by exactly the number of dropped objects, at the cycle they were dropped. It is the reason the scoreboard recomputes occupancy from ingress and egress rather than reading the DUT's counter — reading the DUT's counter would agree with the DUT about a loss.

Check 2 is a policy-legality check and it needs the thresholds. Note it is expressed as two forbidden combinations rather than as a full state derivation. Re-deriving the FSM in the scoreboard would mean maintaining a second copy of the policy, and a bug copied into both is invisible. Checking a small number of impossible combinations is more robust than mirroring the design.

Check 3 deliberately does not assert fairness. Fairness is bounded liveness and belongs in §20's SVA. What the scoreboard contributes is the distribution — grant ratios per class — which no assertion can express and which is the actual instrument for finding §18's bias. A design can satisfy p_bounded_fairness with a wide bound and still be badly skewed; the ratio shows it.

Check 4 requires the testbench to know what it caused, which means the stimulus must record its own injected condition. That is a real constraint on the testbench architecture and it is worth accepting, because first-cause bugs (§29) are otherwise found only in silicon.

Failure the scoreboard catches that assertions do not. Assertions are per-cycle and local; §18's arbiter bias is a statistical property visible only in aggregate. A regression can pass every assertion in this chapter and show a 100:0 grant ratio, and only check 3 reports it.

31. Coverage

covergroup cg_congestion @(posedge clk);
  option.per_instance = 1;
 
  // --- Conditions (Section 4) — all four must be reached, and the last two
  //     require directed stimulus.
  cp_condition : coverpoint congestion_condition {
    bins transient  = {COND_TRANSIENT};
    bins sustained  = {COND_SUSTAINED};
    bins starvation = {COND_STARVE};      // needs a starving arbiter or class
    bins deadlock   = {COND_DEADLOCK};    // needs an injected cyclic wait
  }
 
  // --- Policy states and, more importantly, the ARCS (Section 9).
  cp_state : coverpoint cg_state_q;
  cp_arc   : coverpoint {cg_state_prev_q, cg_state_q} {
    bins norm_pres  = {{CG_NORMAL,    CG_PRESSURED}};
    bins pres_cong  = {{CG_PRESSURED, CG_CONGESTED}};
    bins pres_norm  = {{CG_PRESSURED, CG_NORMAL}};
    bins cong_drain = {{CG_CONGESTED, CG_DRAINING}};
    bins drain_norm = {{CG_DRAINING,  CG_NORMAL}};
    bins drain_cong = {{CG_DRAINING,  CG_CONGESTED}};   // refill during drain
    bins cong_esc   = {{CG_CONGESTED, CG_ESCALATE}};
    bins esc_drain  = {{CG_ESCALATE,  CG_DRAINING}};
  }
 
  // --- The hysteresis band. Section 12's assertions pass VACUOUSLY unless
  //     occupancy actually dwells between the exit and entry thresholds.
  cp_dwell : coverpoint occupancy_q {
    bins below_lo   = {[0 : LO_CONG-1]};
    bins in_band    = {[LO_CONG : HI_CONG-1]};   // the band that matters
    bins at_or_above= {[HI_CONG : DEPTH]};
  }
  cp_dwell_long : coverpoint dwell_in_band_cycles {
    bins brief    = {[1:8]};
    bins sustained= {[9:64]};
    bins long     = {[65:$]};                    // proves Section 12 ran
  }
 
  // --- Bottleneck identity (Section 5). Each resource must be the binding
  //     one at least once, or that path is untested.
  cp_bottleneck : coverpoint first_cause_q {
    bins local     = {CGC_LOCAL_FULL};
    bins credit    = {CGC_NO_CREDIT};
    bins replay    = {CGC_REPLAY};
    bins arb       = {CGC_ARB_LOSS};
    bins downstr   = {CGC_DOWNSTREAM};
  }
 
  // --- Classes and fairness (Sections 15 to 20).
  cp_class_congested : coverpoint congested_class_id;
  cp_grant_class     : coverpoint granted_class_id;
  cp_starve_inject   : coverpoint strict_priority_starvation_injected;
 
  // --- Retry-induced congestion (Section 25) — the Module 14 bridge.
  cp_retry_rate : coverpoint retry_rate_bucket {
    bins none = {0};
    bins low  = {[1:3]};
    bins high = {[4:$]};
  }
 
  // --- Drain and escalation.
  cp_drain_mode : coverpoint drain_mode_active;
  cp_escalated  : coverpoint escalation_asserted;
 
  // --- Crosses that carry the real information.
  x_arb_congested : cross cp_grant_class, cp_state;       // fairness under load
  x_retry_cong    : cross cp_retry_rate, cp_state;        // Section 26's chain
  x_cause_state   : cross cp_bottleneck, cp_state;
  x_class_starve  : cross cp_starve_inject, cp_class_congested;
endcovergroup

Four bins whose value is being non-zero rather than zero, because each is proof a mechanism was actually exercised rather than merely present:

cp_dwell_long.long. Without it, §12's three hysteresis assertions have passed vacuously — occupancy crossed the band without dwelling in it, so the antecedents never held. A regression reporting long = 0 has not tested hysteresis, and that is indistinguishable from hysteresis being absent.

cp_arc.drain_cong. The refill-during-drain arc, which requires the workload to burst again mid-drain. It is the arc most likely to be broken and least likely to be hit by random traffic.

cp_condition.starvation and cp_starve_inject. Starvation must be injected; it does not occur spontaneously in a design whose arbiter works. A zero here means §16's and §18's failure modes were never reachable in the regression.

cp_retry_rate.high crossed with a congested state. This is §26's amplification chain. Testing congestion at a zero error rate and errors at low load exercises neither.

And one bin whose value is being zero: cp_condition.deadlock should be zero in a clean regression and non-zero only in a directed deadlock-injection test, which must exist separately (13.3 §17).

32. Flagship Trace — Eighteen Cycles Through the Whole Policy

Illustrative throughout. DEPTH = 16, HI_WARN = 8, LO_WARN = 6, LO_CONG = 10, HI_CONG = 13. Two classes, rotating priority, one grant per cycle. The consumer stalls at cycle 3 and recovers at cycle 11.

Eight things to read out of it, and the interesting ones are the non-events.

Cycles 3–5: PRESSURED does real work without throttling. Occupancy is rising steadily and the policy has engaged, but admission of mandatory traffic continues. This is the state that a two-state policy cannot express, and it is where the cheapest response — suppressing optional work — lives.

Cycle 6: throttling begins at 14 of 16, not at 16. The two-slot margin is LOOP_SWING from §10: the throttle takes three cycles to reduce arrivals, so waiting until 16 would overflow. Note cycle 7's arrival count is 0 — the throttle worked, and it worked because it started early enough.

Cycles 6–10 are entirely legal and entirely stationary. Occupancy pinned at 14, nothing arriving, nothing departing. Every safety assertion in this chapter, in 13.2, and in 13.3 passes. This is 13.3 §19's vacuity in miniature — and the only thing distinguishing it from a deadlock is that age is being watched, which is why §13's counter is not optional.

Cycle 12 is the whole point of the chapter. Occupancy has fallen to 12, which is below the entry threshold of 13. A single-threshold policy (§11) would clear congestion here — and then re-enter it at cycle 13 or shortly after, because arrivals resume the moment admission reopens. The policy stays CONGESTED for three more cycles because the exit threshold is 10, not 13, and those three cycles are what turn one oscillation into one episode.

Cycle 14 resets age, and that is correct here because the episode genuinely ended — the FSM left CONGESTED via its exit condition rather than dipping momentarily. §6's failure mode is resetting age on a dip; resetting on a real state transition is the intended behaviour.

Cycles 14–16 show why DRAINING is a separate state. Admission is still restricted while occupancy falls from 9 to 8. If DRAINING did not exist, cycle 14 would have gone straight to NORMAL, admission would have reopened at occupancy 9, and the arrivals at cycles 15–17 would have pushed straight back through HI_WARN.

Cycle 17 is the exit, at 5, below LO_WARN = 6 — not at 8, where the warning was entered. Both hysteresis loops have now been demonstrated in one trace.

And the grant column never repeats a class twice in a row, including through the entire stalled region 6–10 where no transfer completes. The pointer did not move during those cycles — §17's transfer_done condition — and yet the alternation continues because grant is combinational from request and a stationary pointer. That is the correct behaviour and it is exactly what §18's bug destroys.

33. Debug Taxonomy

34. Debug Checklist

1. Which queue crossed its high watermark first? Congestion is per-layer (§5) and the first crossing names the layer.
2. What was the ingress rate into that queue? Objects per cycle over the episode, not instantaneous.
3. What was the drain rate out of it? The difference between these two is the whole condition.
4. Which traffic class dominated the occupancy? A single global counter cannot answer this; per-class instrumentation must exist.
5. Did retries rise? If yes, §25 — the load may be unchanged and the cause elsewhere entirely.
6. Were credits exhausted, and was remote occupancy high or low? High means a slow far consumer; low means a return-path fault (13.1 §22).
7. Did arbitration starve a class? Grant ratio per class, over the episode, not overall.
8. Did the pointer move only on actual transfers? §18 — correlate pointer changes against transfer_done.
9. Did admission throttle before the handshake completed? §23 — if anything was discarded after acceptance, stop and fix that first.
10. Did congestion clear below the low watermark, or at the high one? §11 — the answer tells you whether hysteresis exists.
11. Is this pressure, sustained congestion, starvation, or deadlock? §4 — and the discriminator is duration plus whether anything is progressing.
12. What was the first congestion cause? Not the current one (§29).
13. What was peak occupancy? Severity, independent of duration.
14. How long did the longest episode last? max_age_q, sticky across episodes.
15. How many episodes were there? Thousands of short ones is a hysteresis bug, not a congestion condition.
16. Did useful throughput fall as offered load rose? §27 — if so, find the resource being consumed without progress.
17. Would a shallower queue have surfaced this sooner? §28 — a useful counterfactual when the fault was found far from its cause.

35. Common Misconceptions

"Congestion is automatically an error." It is a resource condition, and the ordinary case requires no response at all (§4). Treating every high watermark as a fault produces a design that reports constantly and escalates on healthy traffic. Congestion becomes a fault when it is sustained — which is a duration judgement (§14), and needs a register (§13).

"More buffering always fixes congestion." It converts a throughput problem into a latency problem, delays the feedback that would engage the policy, masks a slow consumer, and costs silicon (§28). It genuinely helps burst absorption, and beyond max(burst, rate × RTT) it buys nothing but latency.

"Strict priority is the safest policy." It is the policy most likely to look correct under congestion and starve a class indefinitely with no illegal state anywhere (§16). Safe means bounded, and strict priority without a reservation bounds nothing.

"If no FIFO overflows, congestion handling is correct." Overflow is a safety property. Starvation, congestion collapse, bufferbloat and deadlock are all achievable with every FIFO permanently within capacity (§4, §16, §27).

"Backpressure and congestion policy are the same thing." Backpressure is the mechanism that carries the constraint (13.3). Congestion policy is the decision about what to do given the constraint. One is plumbing; the other is architecture, and a design can have excellent backpressure and no policy at all.

"Replay does not affect congestion." Retransmission consumes replay residency, link bandwidth and arbitration opportunities, so congestion can rise with the offered load completely unchanged (§25) — and amplify from there (§26).

"A throttle can discard already accepted work." Once the handshake completed, ownership transferred and no discard is legal without an architected notification (§23). The last moment to decline is the cycle before acceptance.

"One occupancy threshold is enough." One threshold produces a comparator with no memory, which changes policy every time the input crosses it, destabilises arbitration, and makes the diagnostics unreadable (§11).

"Starvation is just low throughput for that class." Low throughput is bounded and measurable; starvation is unbounded. If the class carries a progress-critical response, unbounded is a system failure rather than a performance one (§16).

"Congestion state needs only a current-cause field." In a long episode the current cause is almost never the original cause, and the original is the one that identifies the fault (§29).

"The congestion policy will keep occupancy safe, so the resource check is redundant." The policy has hysteresis and control-path latency, so it admits during the interval before its own action takes effect. The correctness gate and the policy gate are independent, and only one of them is optional (§22).

"A busy link means we are at capacity." A link retransmitting is busy and delivering nothing new. Transmitted flits and useful flits are different counters, and only the second answers the question (§26).

36. Understanding Check

37. Summary and What Comes Next

Congestion is a resource condition; congestion handling is a policy decision. The condition is measured by counters 13.2 already built. The response is a choice, and a design that has not chosen deliberately has chosen by accident.

Four conditions produce the same upstream symptom and need four different responses: transient backpressure needs none, sustained congestion needs policy, starvation needs fairness, and deadlock needs neither — it needs to be made visible. Duration separates the first two, whose progress separates the third, and whether anything moves at all separates the fourth.

Congestion is per-layer, and the response lives above the scarce resource. A PHY that cannot accept can only refuse; only an upstream agent can choose to offer less. And credits at zero with remote occupancy low is a return-path fault rather than congestion at all — the same local symptom, an entirely different fix.

The policy needs five states and two thresholds per boundary, because the exit condition is not the negation of the entry condition. The gap between them must exceed one control-loop swing, rate × control_path_latency, or the FSM cannot land inside it. Deriving the thresholds from DEPTH and that latency — with elaboration assertions on their ordering — is what makes the design survive reparameterisation.

Age is the classification input, and it is a saturating counter. Occupancy says how bad; age says what kind. ESCALATE_AGE must be reachable within AGE_W or escalation is dead code, and the age must not reset on a momentary dip or a 40,000-cycle episode reports as a hundred short ones. Saturate diagnostics; never saturate accounting.

Arbitration is a congestion response, and congestion is when the worst arbitration policy looks best. Strict priority serves the important class beautifully and starves the others unboundedly, with every FIFO legal and every safety assertion passing. Rotating priority fails visibly and boundedly instead, which is why it is the default. And fairness state must be updated by the event it accounts for — a pointer that advances on grant rather than on transfer follows the stall pattern and can give one class every transfer, deterministically.

Policy operates on the admission decision and never on accepted work. The correctness resource gate and the policy gate are independent, only one is optional, and the last cycle at which an object may be declined is the cycle before its handshake completes.

And congestion can originate in the reliability path with the offered load unchanged. Retransmission consumes replay residency, bandwidth and arbitration opportunities while correctly consuming no extra credit — so its signature is replay high, retries climbing, credits normal. From there it amplifies: more errors, longer replay residency, stalled admission, growing queues, harder throttling, collapsing useful throughput, on a busy link. The two system-level lessons are that throughput falling as load rises means some resource is being consumed without progress, and that more buffering converts a throughput problem into a latency problem while delaying the feedback that would have fixed it.

Congestion handling keeps the system stable under pressure. The next chapter asks the complementary question: how do we structure the datapath so the pressure appears as late and as rarely as possible?

• 13.5 — Throughput Optimisation — pipelining, outstanding requests, batched flits, and what actually determines the useful transaction rate.

Browse the full path on the UCIe tutorials index.