Backpressure

Chapter 13.1 established how much remote storage a producer is permitted to use. Chapter 13.2 established how much local storage actually exists and how much of it may be promised.

Both chapters produced a number at the receiving end of a pipeline. This chapter is about the awkward fact that the producer which must act on that number is several registered stages away — so by the time it learns to stop, it has already launched work that has to go somewhere.

The consequences split cleanly. One is arithmetic: headroom has to cover the propagation latency, and a design that asserts backpressure with one slot left and takes three cycles to stop overflows by two. The other is structural: a pipeline of independently-stalling stages can reach a state in which every stage is waiting for another and nothing is wrong anywhere — a deadlock through which every safety assertion in this module passes indefinitely.

1. The One-Sentence Model

Backpressure is delayed information about future capacity, so it must begin before the capacity is gone.

Every mechanism below is a way of managing that delay, and every failure is a design that treated the information as instantaneous.

2. What This Chapter Owns

Four earlier chapters have touched backpressure, and the boundaries matter because the overlap would otherwise be substantial.

Chapter 5.5 §2–3 established that a combinational ready-chain across layers cannot work, built the two-entry elastic buffer, and explained why two entries rather than one; its §11–14 established deadlock as a cyclic dependency, that safety assertions pass during one, and bounded progress stated honestly. Chapter 12.1 §7–8 rebuilt the ready-chain argument at FDI and the skid buffer there. Chapter 13.1 built credit accounting; Chapter 13.2 built watermarks and hysteresis.

None of those is re-derived. What this chapter owns:

• the four mechanisms with four scopes, and why they are not interchangeable abstractions;
• the quantitative relationship between propagation latency and required headroom — the local analogue of the bandwidth-delay product;
• deriving a watermark from the propagation latency rather than choosing one;
• backpressure-reason and first-stall-cause state, which is what makes a stall attributable;
• occupancy waves — how a stall moves upstream over several cycles and what that looks like;
• throughput collapse with no correctness failure, as a distinct debugging discipline;
• the wait-for graph, and a concrete deadlock in a UCIe-shaped stack;
• per-class liveness, and why starvation, head-of-line blocking, and deadlock are three different faults.

3. Sourcing

4. Four Mechanisms, Four Scopes

The table this chapter is organised around. These are routinely called "backpressure" interchangeably and they are four different things.

Three things this table settles.

A credit is not a delayed ready. Chapter 13.1 §24 made this point; the addition here is the row above it. A watermark is the delayed local signal — it is an early warning about storage you own. A credit is a reservation on storage you do not own. Conflating them produces a design that either treats remote capacity as observable or treats local capacity as needing a round trip.

They compose as a conjunction, and each can be the binding constraint. Chapter 12.1 §10's acceptance gate is exactly this table turned into an expression, which is why it has four independent terms rather than one.

And they fail with different symptoms and different diagnoses. A ready deassertion is a one-cycle event; a watermark assertion is a stage-local condition; credit exhaustion implicates the far die; a link-state change invalidates work in flight. §27's taxonomy is organised by which one fired, because that determines where to look.

5. Why a Global Ready Chain Cannot Exist

Chapter 12.1 §7 and Chapter 5.5 §2 both established that chaining ready through the stack is unimplementable — the timing path spans the stack, it couples independently-replaceable layers, and it makes the pipeline move in lockstep.

One addition, because it is the strongest form of the argument and it is specific to a die-to-die link: the chain cannot even be written.

remote_ready does not exist as a signal. The far side is on another die, across a package channel, quite possibly in a different clock domain. There is no wire. Whatever a designer writes in place of remote_ready is one of:

• a credit count — registered state reflecting a view of the far side at least a round trip old;
• a link-state bit — registered, and coarser still;
• or a synchronised occupancy flag from a clock-crossing structure, which is registered and delayed by the synchroniser.

All three are registers. So the "combinational chain to the far side" is not a bad design that meets timing poorly; it is a design that cannot be expressed, and any attempt to express it silently substitutes stale registered state for the instantaneous signal the code appears to read.

Every backpressure mechanism except a single local ready is registered state. The only question is how stale, and the answer determines how much headroom you need.

6. Registered Ready Means One More Item Is Already Coming

The mechanism that makes headroom necessary, in its smallest form.

Suppose a stage decides at cycle N that it can accept no more, and its ready is registered — as it must be, for the timing reasons of §5. Then:

• at cycle N the stage's ready is still high, because the new value has not clocked out;
• the upstream stage, seeing ready high, launches an item at cycle N;
• at cycle N+1 the upstream stage sees ready low and stops.

One item was launched after the decision to stop. Chapter 5.5 §3's two-entry elastic buffer exists precisely to hold it, and that chapter explains why two entries rather than one.

What this chapter adds is the generalisation. With one registered boundary, one item. With N registered boundaries between the point of decision and the source, the stop information takes N cycles to arrive, and the source can launch an item in each of them.

7. The N-Stage Relationship

items_launched_after_the_decision  =  launch_rate  ×  propagation_latency

and therefore:

required_headroom  ≳  launch_rate  ×  propagation_latency

where propagation_latency is the number of cycles from "a stage decides to stop" to "the source has stopped", counted in registered boundaries, and launch_rate is items per cycle.

This is the local analogue of Chapter 13.1's bandwidth-delay product, and the parallel is exact:

Note the difference in the last row, because it is the important asymmetry. Too few credits costs throughput and nothing else: the producer simply waits. Too little headroom overflows, because the items were already launched and there is nowhere to put them. So the two errors have opposite severities, and the local one is the dangerous direction.

And the propagation latency is not the pipeline depth. It is the number of registered boundaries the backpressure signal traverses, which can differ from the number the data traverses — a design may pipeline data through four stages while the ready signal is combinational across two of them and registered across the other two. Count the ready path, not the datapath.

8. Deriving the Watermark

Chapter 13.2 §11 built the watermark and its hysteresis. What it did not do — because it did not yet have the propagation latency — is say where the threshold should be.

// ILLUSTRATIVE watermark derivation. The propagation latency and the margin are
// system properties; the arithmetic is not. Symbolic values throughout.
localparam int DEPTH        = 8;   // symbolic
localparam int LAUNCH_RATE  = 1;   // items per cycle the source can launch
localparam int BP_LATENCY   = 3;   // registered boundaries in the READY path
localparam int BP_MARGIN    = 1;   // jitter, arbitration, one cycle of slack
 
// Items that can still arrive after the decision to stop.
localparam int IN_FLIGHT_AFTER_STOP = LAUNCH_RATE * BP_LATENCY;
 
// The watermark must leave room for all of them, plus margin.
localparam int HIGH_WM = DEPTH - IN_FLIGHT_AFTER_STOP - BP_MARGIN;   // 8-3-1 = 4
localparam int LOW_WM  = HIGH_WM / 2;                                // 2
 
// And the relationship must be checked, not assumed (Ch 13.2 §12).
initial begin
  assert (HIGH_WM > 0)
    else $fatal(1, "DEPTH too shallow for this backpressure latency: no valid watermark exists");
  assert (LOW_WM < HIGH_WM)
    else $fatal(1, "hysteresis gap collapsed");
end

Architecture. The watermark is derived rather than chosen. Every term is a system property that can be named, and the elaboration assertion turns "this buffer is too shallow for this control path" from a silicon bug into a build failure.

State. None — parameters.

Cycle behaviour. None, but note what the derivation implies: with DEPTH = 8 and three cycles of propagation, backpressure asserts at half depth. That feels early and it is correct. A design asserting at 7 of 8 has one slot for three cycles of in-flight items.

Contract. The upstream stages rely on being stopped in time. The buffer's hard full protection remains independent (Chapter 13.2 §11).

Failure. §9.

DV. Sweep BP_LATENCY in the parameterisation and verify the elaboration assertion fires when the depth becomes insufficient — testing the guard, not just the design. Then, at each valid setting, drive a sustained burst and verify occupancy never reaches DEPTH.

9. Wrong Pairing — a Late Watermark With a Slow Path

// WRONG — the watermark chosen for "as much buffering as possible" without
// reference to how long the stop takes to arrive.
localparam int HIGH_WM = DEPTH - 1;    // 7 of 8: "one slot of margin"

Architecture. It maximises average occupancy, which is the wrong objective. The watermark's job is not to use the buffer fully; it is to stop the source early enough.

Cycle behaviour. Occupancy reaches 7, backpressure asserts, and three cycles later the source stops. In those three cycles, at one item per cycle, three more items arrive at a buffer with one free slot.

Failure. Two items overflow, deterministically, every time the buffer fills. And note what is not wrong: the occupancy counter, the pointers, the hysteresis, the watermark comparison, and the source's ready handling are all correct. The bug is a parameter chosen without reference to another parameter, which is why Chapter 13.2 §14's overflow was "not a FIFO bug" and this one is not either.

Why it survives testing. It needs the buffer to actually fill, which needs sustained traffic at a rate the consumer cannot match. A directed test that sends a few items and drains them never reaches occupancy 7. It is a load-dependent bug in a parameter, which is about the least likely thing anyone inspects.

And the two fixes are not equivalent. Lowering the watermark to 4 costs average occupancy and fixes it now. Deepening the buffer to 10 costs area and preserves throughput. Which is right depends on whether the buffer was sized for burst absorption or for latency hiding — Chapter 13.2 §13's inputs decide it, and the point is that the decision has to be made rather than defaulted.

10. The Backpressure Wave

The figure's shape is the arithmetic of §7. Three hops, three cycles, three items launched after the decision — and each stage's self-directed arrow is the item it must absorb. Remove any one of those absorption points and the item has nowhere to go.

11. Backpressure Reason State

A stall is not a fact; it is a fact with a cause, and the cause is what a debugging engineer needs.

// ILLUSTRATIVE diagnostic state. NOT normative, and not part of any protocol.
// Encoding one enum per stage turns "the pipeline is stalled" into "stage 2 is
// stalled because the replay store is full", which is the whole value.
typedef enum logic [2:0] {
  BP_NONE         = 3'd0,   // not stalled
  BP_LOCAL_FIFO   = 3'd1,   // this stage's own storage is at its watermark
  BP_REPLAY_FULL  = 3'd2,   // no replay entry available (Ch 9.4 §11)
  BP_NO_CREDIT    = 3'd3,   // remote reservation unavailable (Ch 13.1)
  BP_LINK_DOWN    = 3'd4,   // transport unusable
  BP_DOWNSTREAM   = 3'd5,   // the next stage is not ready — propagated, not local
  BP_ARB_LOST     = 3'd6    // lost arbitration to another class (§21)
} bp_reason_t;
 
bp_reason_t bp_reason_q [NUM_STAGES];
 
// Priority matters: report the ROOT reason, not the propagated one. A stage
// that is both at its watermark and seeing a downstream stall should report
// its own condition, because that is the actionable one.
always_comb begin
  for (int s = 0; s < NUM_STAGES; s++) begin
    unique case (1'b1)
      !link_operational          : bp_reason_c[s] = BP_LINK_DOWN;
      stage_at_watermark[s]      : bp_reason_c[s] = BP_LOCAL_FIFO;
      stage_needs_replay[s] && !replay_space
                                 : bp_reason_c[s] = BP_REPLAY_FULL;
      stage_needs_credit[s] && !credit_available
                                 : bp_reason_c[s] = BP_NO_CREDIT;
      stage_lost_arbitration[s]  : bp_reason_c[s] = BP_ARB_LOST;
      !downstream_ready[s]       : bp_reason_c[s] = BP_DOWNSTREAM;
      default                    : bp_reason_c[s] = BP_NONE;
    endcase
  end
end

Architecture. One enum per stage, computed with an explicit priority so that a stage reports the most actionable cause rather than the most proximate one. BP_DOWNSTREAM is deliberately last: a stage stalled only because the next stage is stalled has nothing to report except a pointer, and the interesting stage is further down.

State. One small register per stage. Diagnostic lifetime — and the registered copy exists so the reason is observable in a waveform at the cycle of interest rather than only combinationally.

Cycle behaviour. Recomputed every cycle. Note that the reason can change while the stall persists — which is §12's subject and the reason a second, sticky field is needed.

Contract. Diagnostics only. This must never gate correctness, and it is worth saying because an enum this convenient invites being used as a control input, at which point a debug change becomes a functional change.

Failure. Putting BP_DOWNSTREAM first in the priority makes every stage in a stalled pipeline report "downstream", which is true and useless: the whole pipeline reports the same thing and none of them names the cause.

DV. Reach every reason value at every stage — that is the coverage cross in §26 — and verify that a stage with two simultaneous causes reports the higher-priority one.

12. First Cause, Not Current Cause

A stall's reason changes while it persists, and the last reason is almost never the useful one.

The realistic sequence: the link goes down, so a stage reports BP_LINK_DOWN. Recovery begins and the link comes back, but by now the replay store is full of un-confirmed objects, so the stage reports BP_REPLAY_FULL. Replay drains, and now credits are exhausted because none were returned during the outage, so it reports BP_NO_CREDIT.

Three reasons, one stall, and only the first one explains it. A design that reports the current reason tells an investigator "no credit", which sends them to Chapter 13.1 for a problem that started as a link event.

// ILLUSTRATIVE first-cause capture. The `if (!valid)` guard is the mechanism,
// and the same discipline as Chapter 12.4 §19's first-failure preservation.
bp_reason_t first_bp_reason_q [NUM_STAGES];
logic       first_bp_valid_q  [NUM_STAGES];
logic [31:0] stall_start_cyc_q [NUM_STAGES];
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    for (int s = 0; s < NUM_STAGES; s++) first_bp_valid_q[s] <= 1'b0;
  end else begin
    for (int s = 0; s < NUM_STAGES; s++) begin
      // Capture on the FIRST cycle of a stall, and hold until the stall ends.
      if (!first_bp_valid_q[s] && (bp_reason_c[s] != BP_NONE)) begin
        first_bp_valid_q[s]  <= 1'b1;
        first_bp_reason_q[s] <= bp_reason_c[s];
        stall_start_cyc_q[s] <= cycle_count_q;
      end
      // Released when the stage is genuinely flowing again, so the field
      // describes THIS stall rather than an older one.
      if (bp_reason_c[s] == BP_NONE) first_bp_valid_q[s] <= 1'b0;
    end
  end
end

Architecture. Write-once per stall episode, cleared when the stall ends. The clearing condition is the design decision: clearing on BP_NONE gives per-episode attribution, while never clearing gives the first stall since reset. Per-episode is more useful, and either is better than the current reason.

State. Per stage, diagnostic lifetime within a stall episode.

Failure. Without the guard, the field tracks the current reason and the cascade above destroys the attribution — and it destroys it identically at every stage, so a multi-stage pipeline reports one misleading answer in chorus.

DV. Inject a link-down stall, recover, and verify the first-cause field still reads BP_LINK_DOWN while the current reason has moved on twice.

13. Occupancy Waves — the Flagship Trace

A three-stage pipeline, each stage a buffer of depth 4 with a watermark at 2 and one cycle of registered ready per boundary. Source launches one item per cycle. The sink stalls at cycle 4.

Six things to read off it, and together they are the chapter.

Cycles 4–8: the wave takes four cycles to reach the source — three boundaries plus the stall's own cycle. During those cycles the source launched four more items, and every one of them found a home because each stage had headroom above its watermark. That is §7's arithmetic working.

Each stage's occupancy peaks one cycle later than the stage below it. That staggered rise is the visual signature of a propagating stall, and it is what to look for in a waveform: if all stages fill simultaneously, the backpressure is not propagating — it is combinational, which is §5's problem.

Cycle 8 is the source stopping, four cycles after the sink did. No item was lost, and the reason is that 4 - 2 = 2 slots of headroom existed per stage and only one item per boundary needed absorbing. Halve the headroom and this trace overflows.

Cycles 9–11 are the correct steady state of a stalled pipeline, and it is worth naming because it looks alarming: every buffer full, nothing moving, every count legal. This is not a deadlock — the sink is stalled for an external reason and will resume. §15 is about telling this state apart from one that will not resume.

Cycle 12–14: the restart wave travels in the same direction with the same delay. So there is a four-cycle bubble at the source after the sink resumes, during which the link is idle while the pipeline is full. That is the throughput cost of registered backpressure, and it is why §14 exists.

And the row that never appears: no cycle where any occupancy exceeds 4. The conservation of §25 holds throughout — every item the source launched is resident in exactly one stage or has been delivered.

14. Throughput Collapse Without a Correctness Failure

A discipline rather than a mechanism, and the reason performance debugging belongs in an RTL chapter.

A pipeline can be entirely correct — no loss, no duplication, no reordering, every assertion passing — and run at a small fraction of its achievable bandwidth. §13's trace shows the mechanism in miniature: a four-cycle bubble after every stall. If the sink stalls frequently, the pipeline spends much of its time in that bubble.

Four causes, and they are distinguishable:

Why this needs its own instrumentation. None of the four raises an error. The functional scoreboard passes, the assertions pass, and the only evidence is counters — cycles stalled per reason, occupancy histograms, and bubble lengths. §25 builds them.

A correct pipeline running at 20 % of its rate is an RTL problem with no functional symptom. The instinct to hand it to a performance team is often the instinct to hand it to people without visibility into the ready path.

15. Starvation, Head-of-Line Blocking, and Deadlock

Three faults that all present as "something is not progressing", with three different structures and three different fixes. Getting them confused wastes more time than any other error in this module.

The distinguishing question is the second row. If something is progressing, it is not a deadlock. If one class is stuck while others move and the queues are separate, it is starvation. If one class is stuck while others move and they share a queue, it is head-of-line blocking — and no scheduler change fixes that, because the ordering was decided when the items were enqueued.

Chapter 13.2 §15 covered the head-of-line structure and Chapter 9.5 §15 covered starvation-versus-deadlock at the credit level. What this chapter owns is the wait-for graph that makes a deadlock provable rather than suspected.

16. The Wait-For Graph

A deadlock is a cycle in a directed graph. Build the graph and the deadlock becomes a finding rather than a hypothesis.

Nodes are things that can wait: a queue, a stage, an agent, a resource pool.

Edges are "waits for": draw an edge from A to B when A cannot make progress until B does something.

A cycle in that graph is a deadlock, and the absence of a cycle is a proof of freedom from deadlock under the modelled dependencies — which is the caveat that matters, because an unmodelled dependency is exactly how designs deadlock in silicon after passing analysis.

The four dependency kinds worth drawing, because these are the ones that create cycles in a flow-controlled stack:

The last edge is where real deadlocks come from, and it is the one people omit — because a design's request and response paths are usually drawn as separate diagrams.

17. A Concrete Deadlock

Now an instance, in a stack shaped like the ones this module has been building.

The setup. Two agents, A and B, over one link. Each sends requests and returns responses. Requests and responses share one outbound queue at each end — a shared-queue decision made for area, and locally reasonable.

The state:

1. A's outbound queue is full of requests destined for B.
2. B's outbound queue is full of requests destined for A.
3. A cannot accept B's requests, because processing them requires space in A's outbound queue to put the responses — and it is full.
4. B cannot accept A's requests, for the mirror reason.
5. So neither side drains. Neither side's queue frees. No credits return in either direction.

The wait-for graph:

A_outbound_queue  →  B_accepts_requests      (A's queue drains only if B takes them)
B_accepts_requests →  B_outbound_queue_space  (B must have room for the response)
B_outbound_queue  →  A_accepts_requests      (B's queue drains only if A takes them)
A_accepts_requests →  A_outbound_queue_space  (A must have room for the response)
A_outbound_queue_space → A_outbound_queue     (space appears only when the queue drains)

That closes. Every node waits on the next, and the last waits on the first.

And notice what is not wrong. No queue overflowed. No credit count is negative or above capacity. No item was lost or duplicated. Every occupancy counter equals its valid population. The conservation equations of Chapter 13.1 §7 and Chapter 13.2 §10 both hold exactly, forever.

The structural fix is to break the cycle, and the standard way is to remove the shared resource that creates edge 2 and 4: give responses their own queue and their own credits, so that returning a response never requires resources that requests can exhaust.

Which is precisely what Chapter 11.3 §28 recorded the CXL specification doing — response and data channels pre-allocated and required to make progress, while request channels are credited and may block indefinitely. That classification is not a stylistic choice; it is the removal of this cycle, written into the protocol. A shared-queue implementation puts the cycle back, and Chapter 13.1 §13 made the same argument in the credit plane.

Anywhere a response requires a resource that requests can exhaust, there is a cycle. Finding them is a matter of drawing the graph including the response edges, which is why §16 insists on that edge kind.

18. The Deadlock Trace

The same scenario, cycle by cycle, so the absence of anything anomalous is visible.

Read the last column. It says "no" at every row, including row 10000, and it is correct at every row:

• occupancy is 4 of 4 — legal, and equal to the valid population;
• credits are 0 — legal, and the conservation equation holds: 0 held + 0 in flight + 4 occupied = 4;
• no push occurred while full; no pop occurred while empty;
• no item was lost, duplicated, or reordered;
• the link is operational and the PHY is fine.

Row 3 is where it became permanent and nothing marks it. There is no edge, no transition, no assertion — the system simply entered a state it cannot leave, and every subsequent cycle is identical to the last.

The only observable difference between row 3 and row 10000 is time, which is why the detection mechanism has to be a progress watchdog rather than a property over state (Chapter 5.5 §13).

19. Why Every Safety Assertion Passes

Chapter 5.5 §12 stated this; §18's trace is the proof, and it is worth extracting the general principle because it determines what verification can and cannot do.

A safety property has the form "nothing bad happens". Formally it is a claim about every reachable state and transition: no overflow, no underflow, no loss, no duplicate, no illegal transition, no push when full.

A deadlocked system produces no transitions. So there is nothing for a safety property to evaluate against, and every one of them holds vacuously and permanently. Adding more safety properties cannot help, because the class of property is wrong.

Three consequences for a verification plan.

A regression whose pass criterion is "no assertion fired" reports a deadlock as a pass — until a test timeout eventually kills it, and the timeout points at whatever happened to be waiting rather than at the cycle in §17.

Coverage does not save you either, because a deadlocked run still filled whatever bins it reached before stopping. A coverage report from a deadlocked test looks like a short test.

So the detection must be a liveness check or a watchdog, and both need assumptions. §20.

20. Liveness, With the Assumptions Written Down

// Illustrative liveness. The `disable iff` is not decoration — it is the list
// of assumptions, and a property without it fires during every injection test,
// gets waived, and then protects nothing.
//
// ASSUMPTIONS:
//   - the clock is running and reset is deasserted
//   - the link is operational
//   - the ultimate consumer eventually drains
//   - no error injection is active
//   - arbitration is fair (see §21 for the per-class form)
property p_stage_eventually_forwards;
  @(posedge clk) disable iff (!rst_n || !link_operational
                             || !consumer_draining || error_injection_active)
    (stage_head_valid[s] && !stage_head_forwarded[s])
      |-> ##[1:MAX_STAGE_LATENCY] stage_head_forwarded[s];
endproperty
a_stage_eventually_forwards: assert property (p_stage_eventually_forwards);

On the honesty of MAX_STAGE_LATENCY. This is a bounded progress property, not a true liveness proof. The bound is an engineering choice — chosen large enough not to fire on legitimate congestion and small enough to catch a genuine stop. Say so in a comment, because a reader who mistakes it for a proof will trust it further than it deserves.

And the assumption that does the most work is consumer_draining. Without it the property is false in a system whose consumer has legitimately stopped, which is a normal condition. With it, the property says something narrower and true: given that the far end is consuming, an item at a stage head must move.

The complementary mechanism is a watchdog, which needs no assumptions because it reports rather than asserts:

// Illustrative progress watchdog. Reports; never gates. Diagnostic lifetime.
logic [31:0] no_progress_cyc_q;     // saturating
logic        no_progress_seen_q;    // sticky
 
wire pipeline_should_move = pipeline_has_items && link_operational;
wire pipeline_moved       = |stage_forward_fire;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    no_progress_cyc_q  <= '0;
    no_progress_seen_q <= 1'b0;
  end else if (pipeline_moved) begin
    no_progress_cyc_q <= '0;                       // progress resets the count
  end else if (pipeline_should_move && !(&no_progress_cyc_q)) begin
    no_progress_cyc_q <= no_progress_cyc_q + 32'd1;
    if (no_progress_cyc_q > WATCHDOG_THRESHOLD) no_progress_seen_q <= 1'b1;
  end
end

Why the watchdog is worth having alongside the property. The property fires in simulation and needs its assumptions to be true. The watchdog works in silicon, where there is no assertion engine, and its sticky bit survives to be read by software after the fact — which is the only evidence a post-silicon deadlock leaves.

21. Per-Class Liveness and Arbitration

Starvation is a liveness failure that a global progress check cannot see, because the pipeline is progressing — just not for everyone.

An arbiter that grants a high-priority class whenever it has work will, under sustained high-priority load, never grant the others. The starved class's queue fills, backpressure propagates to its source, and that source stops. Globally the system is live; for that class it is dead.

// Illustrative per-class progress. One property instance per class, because a
// global check passes while one class starves.
//
// ASSUMPTIONS: as §20, plus that the class's own downstream consumer is draining.
property p_class_eventually_served;
  @(posedge clk) disable iff (!rst_n || !link_operational
                             || !class_consumer_draining[c] || error_injection_active)
    (class_has_work[c] && downstream_ready)
      |-> ##[1:MAX_CLASS_WAIT] class_granted[c];
endproperty
a_class_eventually_served: assert property (p_class_eventually_served);

And the diagnostic that finds it without a property:

// Illustrative — per-class starvation high-water mark. Sticky, diagnostic.
logic [15:0] max_wait_cyc_q [NUM_CLASS];    // longest observed wait, per class
logic [31:0] grants_q       [NUM_CLASS];    // grant counts, for the ratio

The grant ratio is the fastest starvation detector there is. If one class holds 98 % of grants under a mixed workload, that is the finding, and it takes one look at two counters rather than a liveness proof.

Why fairness is a correctness concern here and not only a performance one — Chapter 11.4 §22 built this argument and it is worth the cross-reference: a starved coherence response channel becomes a coherence timeout, and a coherence timeout has no safe recovery. In that context an unfair arbiter is a correctness bug with a performance-shaped symptom.

22. Backpressure Across a Clock Domain

Brief, because Chapter 7.5 §9–10 and Chapter 5.3 §7–8 own the crossing itself. What matters here is the backpressure-specific consequence.

A ready signal must not be synchronised as a status bit. Two reasons, and the second is the one people miss.

The latency becomes part of the propagation path. A two-flop synchroniser adds two destination-clock cycles to the stop information, and §7's arithmetic must include them. A design that sizes headroom from the register count and forgets the synchroniser is short by two.

And a multi-bit occupancy value must never be synchronised bit by bit, because the receiving domain can sample a value that never existed — which for an occupancy comparison can produce a full that is false or, worse, a not full that is false. Chapter 7.5 §10 makes this argument in general; the backpressure form is that a fabricated occupancy value drives a fabricated flow-control decision.

The correct structures are the ones those chapters build: an asynchronous FIFO whose full and empty flags are generated by design in the correct domain, or a proper handshake. The flow-control consequence is only that the crossing's latency counts toward the propagation latency, and it must appear in the watermark derivation of §8.

23. Reset and Recovery

Two interactions, and both are cases where a control signal must not be treated as free to change.

On reset release, ready must not simply be asserted. A stage coming out of reset with ready high advertises capacity before its own state is consistent — its pointers may not be initialised, its occupancy may not be zero yet if the reset is not synchronous across the block. The upstream stage will send into it. The safe shape is that ready is deasserted until the stage's own initialisation is complete, which makes it one more term in Chapter 8.5's dependency graph.

And a reset must not desynchronise ownership. Chapter 12.4 §25's three reset scopes apply: if a stage resets while its upstream believes an item was accepted, that item is lost with the upstream having no knowledge. A reset that clears a stage's storage must either be visible to the upstream stage or be scoped to a point where no item is in flight.

On link recovery, backpressure must engage quickly enough. When the link becomes unusable, new work must stop being accepted — and BP_LINK_DOWN from §11 is that signal. But the items already accepted remain owned according to the transaction and replay policies of Module 12: Chapter 12.4 §24's per-phase table decides their fate, not the backpressure logic. Backpressure stops new work; it does not resolve old work, and a design that uses a link-down condition to flush in-flight items has conflated the two.

24. The Pipeline Conservation Scoreboard

JOIN KEY — the verification-only monitor tag, because an item may be
retransmitted and no protocol field says so.
 
PER STAGE s:
  accepted[s]     — items the stage took from upstream
  forwarded[s]    — items it handed downstream
  resident[s]     — accepted minus forwarded, and it must equal its occupancy
  stall_cycles[s] — per reason, from the §11 enum
  peak_occ[s]
 
PIPELINE:
  source_accepted  =  sink_delivered  +  sum(resident[s])  +  architecturally_aborted
 
PER CLASS c:
  grants[c], max_wait[c]

The five checks, and what each catches.

The conservation equation, every cycle. Backpressure must not create loss or duplication, and this is the check that proves it. A dropped item — §9's overflow — makes the left side exceed the right in the cycle it happens.

resident[s] equals stage s's occupancy counter, computed independently from observed accepts and forwards. Catches Chapter 13.2 §7's drift, and catches an item vanishing inside a stage.

No stage's occupancy exceeded its depth at any point. §9's overflow, caught even if the item is not subsequently lost.

The stall reason distribution is non-degenerate. If 100 % of stall cycles report BP_DOWNSTREAM, no stage ever reported a local cause and the priority logic of §11 is inverted.

And per class, the grant ratio is within a stated bound. §21's starvation check, done with counters rather than properties.

On what the model must not do. It must not read the design's occupancy counters to compute resident[s] — that counter is the suspect. Count accepts and forwards at the interfaces.

25. Performance Monitor

// ILLUSTRATIVE performance counters. These are engineering diagnostics, not an
// analytics product — six numbers that between them explain a throughput
// collapse. Diagnostic lifetime: they survive recovery.
logic [31:0] stall_cyc_by_reason_q [NUM_STAGES][7];   // §11's enum
logic [31:0] occ_histogram_q       [NUM_STAGES][5];   // empty/low/mid/high/full
logic [31:0] credit_zero_cyc_q;                       // Ch 13.1
logic [31:0] replay_full_cyc_q;                       // Ch 9.4
logic [31:0] link_down_cyc_q;
logic [15:0] max_restart_bubble_q;                    // §13's cycles 12-14

Architecture. Each counter eliminates one hypothesis from §14's table. max_restart_bubble_q is the one specific to this chapter: it measures the cost of registered backpressure directly, and a bubble longer than the stage count means the propagation path is longer than the designer thinks.

State. Diagnostic lifetime throughout — surviving link recovery, cleared only by a broad deliberate reset. A counter cleared by the event under investigation is worse than no counter.

Contract. Diagnostics only, never gating.

Failure. Saturating without a sticky companion flag makes a saturated counter indistinguishable from a stopped one.

DV. Verify the reason counters sum to the total stalled cycles — a discrepancy means a stall cycle was attributed to no reason, which is a hole in §11's unique case.

26. Coverage

// Illustrative backpressure coverage. Not UCIe-defined. Every bin reaches a
// failure named in this chapter.
covergroup cg_backpressure @(posedge clk iff bp_event);
 
  cp_stage     : coverpoint stalled_stage;
  cp_reason    : coverpoint bp_reason_q[stalled_stage];       // §11 — all seven
  cp_first     : coverpoint first_bp_reason_q[stalled_stage]; // §12
  cp_duration  : coverpoint stall_duration_bucket {
    bins none = {0}; bins one_cyc = {1}; bins short_stall = {[2:8]}; bins long_stall = {[9:$]};
  }
  cp_occ       : coverpoint stage_occ[stalled_stage] {
    bins empty = {0}; bins below_wm = {[1:HIGH_WM-1]};
    bins at_wm = {HIGH_WM}; bins above_wm = {[HIGH_WM+1:DEPTH-1]}; bins full = {DEPTH};
  }
  cp_skid      : coverpoint skid_captured;            // §6 — the late item
  cp_wave      : coverpoint stages_stalled_count {    // §13 — the wave depth
    bins one = {1}; bins two = {2}; bins all = {NUM_STAGES};
  }
  cp_bubble    : coverpoint restart_bubble_cycles;    // §13 cycles 12-14
  cp_class     : coverpoint granted_class;            // §21
  cp_starve    : coverpoint class_wait_bucket;        // §21
  cp_recovery  : coverpoint stall_during_recovery;    // §23
  cp_cdc       : coverpoint stall_crossed_cdc;        // §22
  cp_overflow  : coverpoint stage_overflow_seen;      // MUST remain zero
  cp_nomove    : coverpoint no_progress_seen_q;       // §20 — MUST remain zero
 
  // Every reason at every stage — the cross §11's priority logic needs.
  x_stage_reason  : cross cp_stage, cp_reason;
  // A stall at every occupancy, especially at and above the watermark — §9.
  x_occ_reason    : cross cp_occ, cp_reason;
  // The full wave: every stage stalled simultaneously — §13 cycle 9.
  x_wave_duration : cross cp_wave, cp_duration;
  // A skid capture at every stage — §6's late item must be absorbed everywhere.
  x_skid_stage    : cross cp_skid, cp_stage;
  // Per-class waiting while another class is granted — §21's starvation.
  x_starve_class  : cross cp_starve, cp_class;
  // A stall beginning during a recovery — §23's ordering.
  x_recovery_stage : cross cp_recovery, cp_stage;
 
endgroup

Three notes on the bins.

cp_overflow and cp_nomove must stay at zero. Writing bins whose value is that they never fill converts two assumptions — no overflow, no deadlock — into checked facts.

cp_first versus cp_reason is the pair that validates §12. A run in which they are always equal has never produced a cascading stall, so the first-cause mechanism is untested.

And cp_wave's all bin is §13's cycle 9 — every stage stalled at once. It is the state that looks like a deadlock and is not, and a regression that never reaches it has never exercised the restart wave.

27. Debug Taxonomy

The pair to internalise is rows six and seven. Identical symptoms, opposite fixes: one needs a fairer arbiter, the other needs separate queues and no arbiter change will help. The distinguishing observation is whether the classes share a queue, which is a structural question answerable by reading the design rather than by simulating it.

28. Debug Checklist

1. Which stage first asserted backpressure? Not which stages are asserting now — the first.
2. What was its reason? §11 — and specifically the first reason, not the current one (§12).
3. What was its occupancy at that moment? Against its watermark and its depth.
4. How much headroom remained above the watermark? §7.
5. How many cycles until the source stopped? Measured, not assumed — this is the propagation latency.
6. How many items were launched in those cycles? Rate times the measured latency.
7. Did every one of them find storage? If not, §9 and the arithmetic is the answer.
8. Did any stage's occupancy exceed its depth? §24's check.
9. Do the stages' occupancies rise in a staggered pattern? If simultaneous, the backpressure is combinational (§5).
10. What was the sender's credit count? §4 — a different mechanism with a different scope.
11. Was replay capacity available? Chapter 9.4 §11.
12. Is anything at all progressing? If yes, it is not a deadlock — go to §15's table.
13. If nothing is progressing, draw the wait-for graph including the response edges. §16 — the response edge is the one usually omitted.
14. Are all safety assertions passing? During a deadlock they will be, and that is information rather than reassurance (§19).
15. Did a reset or a link recovery change ready while items were in flight? §23.
16. Does the pipeline conservation equation balance? §24 — accepted = delivered + resident + aborted.

Steps 5 and 6 are the two that most often produce the answer, and both are measurements rather than inspections. The propagation latency is almost always longer than the designer believes, because synchronisers, arbitration cycles, and combinational-versus-registered boundaries are easy to miscount.

29. Common Misconceptions

"Backpressure is one ready signal." Four mechanisms with four scopes and four latencies — a local ready, a watermark, a credit, and a link state. They compose as a conjunction and each can be the binding constraint, so a design with one "cannot accept" signal cannot say which resource is scarce (§4).

"Combinational ready chains are simplest." Across a die-to-die link the chain cannot even be written, because remote_ready does not exist as a signal — whatever stands in for it is registered state that is at least a round trip stale (§5).

"Registering ready never needs extra storage." It always does. The upstream stage launches one more item after the decision to stop, and with N registered boundaries it launches N. That is what headroom is for (§6, §7).

"Almost-full can assert when one slot remains." Only if the propagation latency is zero. With three cycles of propagation and one item per cycle, asserting at DEPTH−1 overflows by two, deterministically, every time the buffer fills (§9).

"If no FIFO overflows, flow control is correct." §17's deadlock has no overflow, no underflow, no loss, no duplication, and every conservation equation holding exactly — forever (§18).

"Deadlock always causes an assertion failure." A deadlocked system produces no transitions, so every safety property holds vacuously. Adding more safety properties cannot help because the class of property is wrong; detection needs a liveness check or a watchdog (§19).

"Starvation and deadlock are the same." During starvation the system is progressing, just not for one class — so a global progress check passes. And starvation with a shared queue is head-of-line blocking instead, which no scheduler change fixes (§15, §21).

"A legal credit count proves the pipeline will progress." Credits are safety only. In §17's deadlock both sides hold exactly zero credits, legitimately, and the conservation equations hold at every cycle (§18).

"ready may be synchronised like a normal status bit." Its latency becomes part of the propagation path and must appear in the watermark derivation; and a bit-wise crossing of a multi-bit occupancy can fabricate a value that never existed, producing a flow-control decision based on nothing (§22).

"Resetting a blocked stage clears the problem safely." If the upstream believes an item was accepted, resetting the stage loses it silently. A reset that clears storage must be visible to the upstream or scoped to a quiescent point — and on reset release, ready must not simply be asserted (§23).

"Performance stalls are not RTL-debug problems." A correct pipeline can run at a fraction of its rate with no functional symptom, and the four causes are all in the ready path and the buffer sizing. Handing it to someone without visibility into those is handing it to the wrong place (§14).

30. Understanding Check

31. Summary and What Comes Next

Backpressure is delayed information about future capacity, so it must begin before the capacity is gone.

The four mechanisms and their scopes: a local ready that acts this cycle, a watermark that warns within a stage or two, a credit that reserves remote storage a round trip away, and a link state that invalidates the path. They are not interchangeable, they compose as a conjunction, and each can be the binding constraint.

The arithmetic: required_headroom ≳ launch_rate × propagation_latency, with the latency counted in registered boundaries in the ready path — including any synchroniser. That makes the watermark a derived value, DEPTH − rate × latency − margin, guarded by an elaboration assertion so a buffer too shallow for its control path fails the build. Choosing DEPTH−1 instead overflows by two, deterministically, whenever the buffer fills.

The structures: a stall propagates as a wave whose staggered occupancy rise is the signature that backpressure is registered rather than combinational; a fully backed-up pipeline is a correct steady state, not a fault; and the restart wave costs a bubble equal to the propagation latency.

The three faults that look alike: starvation with a fairness fix, head-of-line blocking with a structural fix and no scheduler remedy, and deadlock with neither. The distinguishing question is whether anything is progressing, and then whether the classes share a queue.

And the lesson that outranks the rest: a deadlocked system produces no transitions, so every safety property holds vacuously and forever. Occupancies legal, credits legal, conservation exact, nothing lost — and nothing moving. Detection needs bounded-progress liveness with its assumptions written down, and a watchdog that works in silicon.

Flow control is now complete as a set of mechanisms: permission accounting, the storage behind it, and the propagation of constraints through the pipeline. What none of the three chapters has addressed is what a system should do when congestion is sustained rather than transient — whether to throttle, to drop, to reroute, to signal upward, or simply to wait — and those responses differ by layer and trade against each other:

• 13.4 — Congestion Handling — per-layer congestion responses and their tradeoffs.

Browse the full path on the UCIe tutorials index.