FPGA Accelerator Card — Back-Pressure Has Latency

29.1 traced one command; 29.2 accounted for one transfer. This chapter has neither — a streaming card has no per-item descriptor and no transfer boundary, and the interesting state is a control loop that closes across the PCIe link with a round trip inside it.

1. Sources, Scope, and What This Chapter Refuses to Do

2. Streaming Is a Different Control Problem

Three readings.

Row 3 is the whole chapter. In a descriptor model, the transfer stops because it is finished. In a streaming model it stops because someone asked it to — and between the asking and the stopping, the requests already issued continue to return.

Row 5 is where the two models' arithmetic diverges. Descriptor designs derive outstanding depth from rate × latency to reach a throughput. Streaming designs must additionally derive buffer headroom from the same product — to survive stopping.

And "the consumer's appetite" is not a constant. An accelerator's consumption rate depends on what it is processing, so the loop must handle a consumer that slows down without warning — which is the case §6 gets wrong.

3. The Loop

4. The Fact That Makes This Hard

A read request cannot be cancelled. Once issued, its Completion is coming (10.4), and the FIFO must have room for it whether or not the accelerator still wants it.

So the loop has a dead time, and it is not the signal's propagation delay — it is the round trip of the requests already in flight.

Three readings.

"Stop" means "stop issuing", not "stop receiving." Every design error in §6 comes from conflating those two. There is no mechanism at this layer to un-ask for data.

Which makes the required headroom a product, not a margin. It is outstanding_requests × bytes_per_request — and that quantity is the same one 26.2 §5 derives for throughput. The same product appears twice with opposite roles: large enough to fill the pipe, small enough to fit in the buffer.

And that tension is the streaming design's central trade. More outstanding requests raise achievable bandwidth (29.2 factor 4) and raise the buffer the card must carry. They are not independent knobs.

5. Deriving the Headroom

Step 1 — bytes committed at any instant. 64 tags × 512 B = 32 768 bytes are either in flight or about to be.

Step 2 — therefore the minimum headroom. When the threshold asserts, up to 32 KB may still arrive. So:

threshold_bytes ≤ FIFO_depth − outstanding_bytes

Step 3 — what that means for a given FIFO.

Step 4 — or fix the FIFO and solve for tags. With a 32 KB FIFO and a threshold at half depth (16 KB), the safe outstanding budget is 16 KB ÷ 512 B = 32 tags — which by 26.2 §5 supports 32 × 512 B ÷ 1.0 µs = 16.4 GB/s, below the accelerator's 20 GB/s appetite. The buffer, not the link, is now the throughput limit.

Four readings.

Row 1 of step 3 is a design that cannot be fixed by tuning. With 16 KB of FIFO and 32 KB committed, no threshold value avoids overflow — the only fixes are a bigger FIFO or fewer tags. A team searching for the right threshold will not find one, and §6 is what they build while searching.

Step 4 is the trade stated as a number. The FIFO depth sets a ceiling on outstanding, which sets a ceiling on bandwidth. A card specified for 20 GB/s with a 32 KB FIFO is specified inconsistently, and the arithmetic exposes it before RTL.

The two knobs are coupled through the same product (§4). Raising tags to reach bandwidth raises the headroom requirement proportionally — so "add more tags" is never a free change on a streaming path.

And the honest reading of step 3 is that headroom is expensive. 32 KB of on-chip buffer per stream is real block RAM, and a card with several streams multiplies it. That cost belongs in the architecture conversation, not discovered at floorplan.

6. Wrong RTL — Back-Pressure at "Nearly Full"

The design follows from treating back-pressure as an instantaneous signal.

// WRONG. ILLUSTRATIVE. Back-pressure asserted from FIFO occupancy with a
// generous-looking margin. Every line is reasonable in a local, same-cycle
// flow-control design — and this loop closes across a PCIe round trip.
localparam int FIFO_DEPTH_BYTES = 32768;
localparam int ALMOST_FULL      = 28672;          // BUG 1: 87.5 % — "generous"
 
logic [15:0] fifo_bytes_q;
logic        stop_issue;
 
// BUG 1: the threshold is chosen as a fraction of depth, with no reference to
//        how many bytes are already committed (§5). At 64 tags × 512 B that is
//        32 KB in flight against 4 KB of remaining space.
assign stop_issue = (fifo_bytes_q >= ALMOST_FULL[15:0]);
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    fifo_bytes_q <= '0;
  end else begin
    // BUG 2: two separate branches writing fifo_bytes_q. When a completion
    //        lands and the accelerator consumes in the same cycle, one update
    //        is lost and the occupancy estimate drifts.
    if (cpl_fire)      fifo_bytes_q <= fifo_bytes_q + 16'(cpl_bytes);
    else if (cons_fire) fifo_bytes_q <= fifo_bytes_q - 16'(cons_bytes);
 
    // BUG 3: on overflow the write is simply dropped, silently. There is no
    //        error, no counter, and the stream is now corrupt with no evidence.
    if (cpl_fire && (fifo_bytes_q + 16'(cpl_bytes) > FIFO_DEPTH_BYTES[15:0]))
      fifo_wr_en <= 1'b0;
  end
end

Architecture. An occupancy counter, a percentage threshold, and an issue gate.

State. fifo_bytes_q. The missing state is the outstanding byte count — the quantity §5 shows the threshold must be derived from, and which this design never reads.

Event. stop_issue asserts at 87.5 % occupancy. New requests stop; nothing stops the 32 KB already committed.

Contract. The DMA engine's contract is that the FIFO has room for every Completion it will receive. This design's threshold leaves 4 KB of room for 32 KB of committed data.

Failure — the timeline. The accelerator stalls for 3 µs mid-stream.

First divergence: 1.4 µs — the threshold asserted with less room remaining than the committed traffic. Everything after is arithmetic. The visible symptom is wrong results from the accelerator, with no PCIe error, no dropped TLP on the wire, and a link that behaved perfectly: every Completion was delivered correctly and the device discarded it.

Root cause. A percentage threshold on a loop with a round trip inside it. 87.5 % sounds conservative and is meaningless — the correct threshold is not a fraction of depth, it is depth minus committed bytes (§5).

BUG 2 compounds it. Coincident completion and consumption lose an update, so the occupancy estimate drifts low — the threshold asserts later than it should, reducing the already-insufficient headroom further.

And BUG 3 is why it is undebuggable. The overflow drops writes with no counter and no sticky flag. The only evidence is wrong output, arbitrarily far downstream.

DV/debug. The report is "the accelerator produces wrong results when the host is busy", which points at the host or the accelerator. The link is clean, and the discriminator is a drop counter that does not exist.

7. Corrected RTL — a Derived Threshold and a Committed-Bytes Counter

// CORRECT. ILLUSTRATIVE. Three changes:
//   (a) the threshold is DERIVED from committed bytes, not chosen;
//   (b) occupancy uses one signed next-state expression;
//   (c) an overflow is impossible by construction AND counted if it ever
//       happens, so the invariant is falsifiable rather than assumed.
localparam int FIFO_DEPTH_BYTES = 65536;          // sized per §5 step 3
localparam int MAX_TAGS         = 64;
localparam int BYTES_PER_REQ    = 512;
localparam int COMMITTED_MAX    = MAX_TAGS * BYTES_PER_REQ;   // 32768
 
// The elaboration-time check that makes §5's arithmetic a build failure rather
// than a silicon bug. This is the single most valuable line in the file.
initial begin
  if (FIFO_DEPTH_BYTES <= COMMITTED_MAX)
    $error("FIFO_DEPTH_BYTES (%0d) must exceed COMMITTED_MAX (%0d): no safe threshold exists",
           FIFO_DEPTH_BYTES, COMMITTED_MAX);
end
 
logic [16:0] fifo_bytes_q;
logic [16:0] committed_bytes_q;      // bytes requested and not yet returned
logic [31:0] overflow_drops_q;       // must be zero forever
logic        stop_issue;
 
// (a) Stop issuing when the FIFO could not absorb everything still committed
//     PLUS one more request. This is §5's inequality, evaluated live rather
//     than baked into a constant — so it adapts if fewer tags are in use.
assign stop_issue = (fifo_bytes_q + committed_bytes_q + 17'(BYTES_PER_REQ))
                      > 17'(FIFO_DEPTH_BYTES);
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    fifo_bytes_q <= '0; committed_bytes_q <= '0; overflow_drops_q <= '0;
  end else begin
    // (b) ONE signed next-state expression each, so coincident events net
    //     correctly instead of losing an update (BUG 2).
    fifo_bytes_q <= fifo_bytes_q
                  + 17'(cpl_fire  ? cpl_bytes  : '0)
                  - 17'(cons_fire ? cons_bytes : '0);
 
    // Committed rises when a request is ACCEPTED and falls when its data lands.
    committed_bytes_q <= committed_bytes_q
                       + 17'(req_accept_fire ? BYTES_PER_REQ : '0)
                       - 17'(cpl_fire        ? cpl_bytes     : '0);
 
    // (c) Should be unreachable. Counted anyway — an invariant nobody can
    //     observe is an assumption (§9).
    if (cpl_fire && ((fifo_bytes_q + 17'(cpl_bytes)) > 17'(FIFO_DEPTH_BYTES)))
      overflow_drops_q <= overflow_drops_q + 32'd1;
  end
end

Architecture. Two occupancy-class counters — what is in the FIFO and what is on its way — and a threshold that is a live inequality over both.

State. fifo_bytes_q, committed_bytes_q, and a drop counter. committed_bytes_q is the state §6 lacks, and it is what makes the threshold correct rather than hopeful.

Event. committed_bytes_q rises on request acceptance (valid && ready), not on offering, and falls when the data lands. The two events are deliberately different: acceptance commits the bytes, arrival releases the commitment.

Contract. BYTES_PER_REQ must be the maximum a single request can return, not the typical value. If a request can return more than assumed, the inequality under-reserves — and this is the parameter most likely to be set from an average.

Failure. The residual risk is a request that returns in several Completions (13.3): committed_bytes_q must fall by the bytes actually returned, not by BYTES_PER_REQ per Completion, or the commitment is released too fast. The code above decrements by cpl_bytes, which is correct for split delivery.

And the elaboration check is worth more than the RTL. FIFO_DEPTH_BYTES <= COMMITTED_MAX is §5 row 1 — a configuration with no safe threshold, caught at compile time on every parameter set rather than in a lab.

8. Checks and Assertions

// MANDATORY. English: the FIFO never overflows. This is the invariant the whole
// derivation exists to establish, and it is the property that fails immediately
// if the threshold or the committed accounting is wrong.
a_no_fifo_overflow: assert property (
  @(posedge clk) disable iff (!rst_n)
    (fifo_bytes_q <= 17'(FIFO_DEPTH_BYTES))
);
 
// MANDATORY. English: a request is never accepted when the FIFO could not
// absorb everything already committed plus this request. This is §7(a) as a
// property — it catches the issue gate being bypassed, and it fires BEFORE the
// overflow rather than at it.
a_issue_respects_headroom: assert property (
  @(posedge clk) disable iff (!rst_n)
    req_accept_fire |->
      ((fifo_bytes_q + committed_bytes_q + 17'(BYTES_PER_REQ))
         <= 17'(FIFO_DEPTH_BYTES))
);
 
// MANDATORY. English: committed bytes never exceed the maximum the tag count
// permits. Catches a commitment that is not released when data lands — which
// would throttle the stream permanently and present as a bandwidth bug.
a_committed_bounded: assert property (
  @(posedge clk) disable iff (!rst_n)
    committed_bytes_q <= 17'(COMMITTED_MAX)
);

Reading the three.

The second is the useful one and the first is the safety net. a_no_fifo_overflow fires at the failure; a_issue_respects_headroom fires at the decision that causes it, which is one round trip earlier and identifies the responsible logic directly. Both are worth having, and only the second localises.

All three are same-cycle (|->) because each relates a decision to state evaluated in the same cycle. All three are good formal targets — three counters of context, bounded arithmetic, and the properties are the design intent rather than a restatement of the code.

And a_committed_bounded catches the opposite failure from the one this chapter is about. A commitment never released starves the stream: the threshold stays asserted, throughput collapses, and the symptom is a bandwidth shortfall rather than corruption. Same accounting, opposite direction, and 29.2 §12's decomposition would attribute it to factor 4.

9. Measured Behaviour

A 3 µs accelerator stall injected once per millisecond, across four configurations.

Three readings.

Row 3 is the one that matters: the FIFO is large enough and §6 still drops. 64 KB of depth with 32 KB committed has a safe threshold — 32 KB — and the percentage rule picks 56 KB instead. The bug is the rule, not the buffer, and no amount of extra memory fixes a rule that ignores committed bytes.

Row 5 is how this ships. At 128 KB the percentage threshold happens to leave enough room, so the design works and the rule looks correct. A later cost reduction to 64 KB reintroduces the bug, and the change that caused it looks unrelated.

And rows 1–2 show the elaboration check earning its place. Those configurations are unfixable by tuning, and §7 refuses to build them rather than letting someone search for a threshold that does not exist.

10. The Other Direction

Streaming device→host has the same structure with the roles exchanged, and one asymmetry worth naming.

Two readings.

The egress direction cannot overflow the way ingress can, because the device chooses when to emit and credits prevent it from sending into a full receiver (16.5). The failure mode flips from corruption to a stall, which is a considerably better failure.

And the host-side consumption rate becomes the new hidden parameter. If software drains the destination ring more slowly than the device fills it, the device stalls waiting for buffer availability signalled by software — a loop with software latency inside it rather than a PCIe round trip, and typically far longer. The same headroom reasoning applies with a different, larger dead time.

11. Executable Counterexamples

Case 2 is the minimum reproduction and it needs a consumer stall, not a link event. A test that only varies host or link behaviour cannot produce it — which is why a streaming card's verification must model the accelerator as an independently-stalling agent (§12).

12. Verification

Three readings.

The scoreboard must track byte position, not transaction count. A stream has no transaction boundaries software cares about; a dropped 25 KB is a gap in a byte sequence, and only positional accounting detects it.

"Dropped bytes are a scoreboard error" is worth stating because §6's design treats them as a flow-control outcome. A verification environment that models the drop as legitimate back-pressure will pass the broken design — the environment must encode that a drop is never acceptable.

And the stall-duration coverage bin must exceed a round trip, because a stall shorter than the dead time is absorbed by any threshold. The bug only exists for stalls long enough that committed data outlives the gate closing.

13. Debugging

Three readings.

The clean link is the confusing part, and it is the module's recurring signature (29.1 §13, 29.2 §12): every external measurement exonerates the obvious suspects, because the failure is internal to the device and downstream of correct delivery.

A protocol analyser cannot see this at all. It observes Completions arriving and being ACKed at the link layer (14.2); what the device did with them afterwards is invisible (25.9 §3).

And the cheapest instrument is a single sticky bit. "Did the ingress FIFO ever overflow?" costs one flop and converts an open-ended data-corruption investigation into a one-read answer.

14. Misconceptions

"Back-pressure stops the data." §4: it stops issuing. Completions for already-issued requests cannot be recalled, and the FIFO must absorb all of them.

"87.5 % is a conservative threshold." §5, §9 row 3: conservative against what? With 32 KB committed, a 64 KB FIFO's only safe threshold is 32 KB. A percentage is not a reservation.

"A bigger FIFO fixes it." §9 row 3: 64 KB is big enough and the rule still drops 24 KB. Row 5 shows a bigger FIFO masking the bug, which is worse.

"More outstanding requests is a free throughput win." §4, §5 step 4: on a streaming path it raises the headroom requirement proportionally. The two knobs are coupled through one product.

"The link dropped data." §13: every Completion was delivered and ACKed. The device discarded it.

"Streaming is just DMA without descriptors." §2: removing the descriptor removes the transfer boundary, which is what used to bound the data in flight. The bound has to come from somewhere else.

"We can size the FIFO later." §5, §7: FIFO_DEPTH ≤ MAX_TAGS × BYTES_PER_REQ is a configuration with no safe threshold — it is an architecture error, and the elaboration check makes it a build failure.

15. Understanding Check

Q1. The accelerator's FIFO is 64 KB and back-pressure asserts at 87.5 %. The card drops data. Why doesn't a bigger threshold margin fix it?

Because the threshold must be derived from committed bytes, and a percentage is not a reservation (§5). With 64 tags at 512 B, 32 KB is in flight or committed at any instant, so when the gate closes the FIFO must have at least 32 KB free. A threshold at 87.5 % leaves 8 KB. The safe threshold for that configuration is 32 KB — exactly half depth — and §9 row 3 measures ~24 KB dropped per stall with the percentage rule on a FIFO that is otherwise large enough. The rule is the bug, not the buffer, and the inequality is threshold ≤ depth − outstanding_bytes. Two fixes exist: raise the FIFO, or cut tags — and cutting tags to 32 caps sustainable bandwidth at 32 × 512 B ÷ 1 µs = 16.4 GB/s (§5 step 4), which may be below the accelerator's appetite. The knobs are coupled.

Q2. Why can't the device just cancel the outstanding reads?

There is no mechanism at this layer to un-ask for data (§4). A non-posted read has been issued; its Completion is coming (10.4). So "stop" means stop issuing new requests, and the dead time before the FIFO stops growing is the round trip of what is already committed — not the back-pressure signal's propagation delay. That is why the required headroom is a product (outstanding × bytes_per_request) rather than a margin, and why it is the same product 26.2 §5 derives for throughput. The same quantity appears twice with opposite roles: large enough to fill the pipe, small enough to fit the buffer.

Q3. Every Completion was delivered and ACKed, the link counters are clean, and the results are wrong. Where do you look?

Inside the device, downstream of correct delivery (§13). The fabric did its job — a protocol analyser sees Completions arriving and being acknowledged at the link layer (14.2), and what the device did with them afterwards is invisible on the wire (25.9 §3). The first divergence is the cycle the issue gate closed with less headroom than committed bytes, and the minimum discriminating instrument is a sticky overflow bit on the ingress FIFO — one flop. Without it, the fallback is occupancy high-water versus depth: pinned at depth means the FIFO reached capacity, which on a correct design is impossible. The prevention is the elaboration check (§7), because a configuration where FIFO_DEPTH ≤ MAX_TAGS × BYTES_PER_REQ has no safe threshold at all and should not build.

Q4. Design the verification that would have caught this, including why an ordinary DMA testbench would not.

Model the accelerator as an independently-stalling agent (§12). An ordinary DMA testbench varies host and link behaviour; this bug requires a consumer stall lasting longer than a round trip, which no host- or link-side stimulus produces. Coverage must therefore bin stall duration with bins beyond the round trip, because shorter stalls are absorbed by any threshold and cannot expose the defect.

The checker must treat a drop as an error, not as back-pressure. §6's design discards writes as a flow-control outcome; an environment that models that as legitimate will pass it. So the scoreboard tracks byte position in the stream — a dropped 25 KB is a gap in a byte sequence, invisible to transaction counting.

And two negative tests prove the checker is alive: set FIFO depth below committed bytes and confirm the build fails, and inject a stall and confirm the drop counter stays zero. A checker that cannot fail on case 2 is not checking. Add the coincident completion-and-consumption case, which catches the occupancy estimate drifting from separate if branches.

16. What Comes Next

This chapter's loop closed inside one device, across a PCIe round trip. The next one closes across two independent rate domains.

29.4 traces a SmartNIC, where the arrival process is the network and the service process is PCIe — two rates set by different systems, neither of which can be told to slow down. The headroom question returns in a harsher form: there is no back-pressure path to an external network, so the buffer is the only defence and its sizing is a statistical argument rather than a product.