PCIe · Module 16
Credits — Backpressure Across a Wire You Cannot Reach
An on-chip receiver says no with one wire. A receiver on the far side of a Link cannot, so PCIe replaces the ready signal with advertised capacity and local accounting — six pools, two units, and one rule about spending what you have not been given.
Every interface in this curriculum so far has ended in the same question: may I send this?
On-chip, the answer is a wire. ready is low, so you wait.
There is no such wire across a PCIe Link. The receiver is a package, a connector and a length of board away — a round trip of nanoseconds in a protocol that issues a packet every few. By the time a "stop" signal arrived, the packets it was meant to stop would already have been received and dropped.
So how does a PCIe transmitter know it is allowed to send?
1. The Verified Model
2. Why Not Just Run a Ready Signal
Because the answer would arrive too late to be useful, and PCIe's problem is not "when may I send" but "will this land".
Consider what an on-chip ready actually provides. The receiver evaluates its own occupancy in the same cycle the transmitter is deciding, and the transmitter obeys before committing. The information is zero cycles old.
Across a Link, the same signal would be tens of nanoseconds old, and in that window the transmitter can issue many packets. A late "stop" is not conservative — it is wrong, and the packets it failed to stop have nowhere to go. There is no back-off, no retry at this layer, and a dropped TLP is a reliability failure (Chapter 14.1 §2), not a flow-control event.
3. Three Things a Credit Is Not
Each of these is a real design error, and each produces a different symptom.
Not replay-buffer space
Chapter 14.4 built local storage that retains transmitted packets until acknowledgement retires them. That is the transmitter's own memory, on this side of the Link.
A credit is the receiver's memory, on the far side. The two are independent, and either can block a transmission on its own:
| Replay space | Remote credit | Result |
|---|---|---|
| free | available | send |
| free | exhausted | wait — the receiver cannot store it |
| full | available | wait — you could not replay it if it were lost |
| full | exhausted | wait, for two reasons |
They are also freed by different events. Replay space is freed by an ACK (Chapter 14.2); credit is returned by an UpdateFC (Chapter 15.2). A design that treats one as the other stalls permanently the first time they diverge — and §13 gives the debugging signature.
Not Link bandwidth
Credits answer "may the receiver store this?", not "is there a transmit slot this cycle?"
A Link at 5% utilisation can be completely credit-blocked, and a Link with abundant credit can be bandwidth-saturated. The two constraints are unrelated, and a performance investigation that measures only one will find nothing.
Not a completion guarantee
Credit returned means buffer space was freed. It does not mean the operation finished.
A receiver may free a Posted Write's buffer the moment it hands the write to internal logic — long before the write is visible anywhere. Returned credit says "I can take another", not "the last one is done". That distinction is what makes Chapter 10.3's posting model work at all.
4. Six Pools, Two Dimensions
The taxonomy is a cross product, and both axes exist for a reason.
Header Data
Posted PH PD
Non-Posted NPH NPD
Completion CPLH CPLDWhy the storage axis exists is more concrete. A receiver must do two different things with an arriving TLP: process its header — decode, route, look up, enqueue — and store its payload, if it has one. Those consume different resources in different amounts. A thousand header-only packets and one maximum-payload packet stress completely different parts of the same receiver, and one number could not express both.
5. The Two Units Are Not the Same Kind of Thing
And this is where the most common misconception lives.
A data credit is 4 DW — sixteen bytes. It is a quantity of storage, and a packet's data cost is a division:
n = Roundup(Length / FC unit size)
A header credit is one header. The source defines the unit as "the sum of one maximum-size header and TLP Digest" — that is, the receiver reserves enough room for the largest header it might get, plus a Digest, and calls that one credit.
"One credit equals one byte" is wrong by a factor of sixteen, and the error is conservative in one direction and catastrophic in the other. A design that thinks a credit is a byte will refuse to send when it could — merely slow. A design that computes 16 bytes as one credit when it needs two will overflow the receiver — and §12's counterexample is exactly that.
6. Advertise, Spend, Return
The full loop, and each step belongs to a different chapter.
| Step | What happens | Owned by |
|---|---|---|
| Advertise | the receiver states initial capacity per pool with InitFC | §1's table 6; packets in 15.1 |
| Account | the transmitter holds a local counter per pool | this chapter, §8 |
| Gate | a packet is released only if every pool it needs has room | this chapter, §10 |
| Spend | the counter is decremented as the packet is transmitted | §1's "deducts the amount of credits used"; the policy is 16.5 |
| Free | the receiver drains the buffer | implementation-defined |
| Return | UpdateFC carries a cumulative total back | 15.2; protocol in 16.6 |
Note what the transmitter never does: ask. Every arrow into its counters is unsolicited.
7. A Trace
Internal teaching signals, not PCIe wire signals. One pool, capacity 8 credits.
step 1 2 3 4 5 6 7 8
init_valid 1 0 0 0 0 0 0 0
init_capacity 8 - - - - - - -
consume_valid 0 1 1 1 1 1 0 0
consume_cost - 3 3 3 2 2 - -
return_valid 0 0 0 0 1 0 1 0
return_count - - - - 4 - 2 -
available 0 8 5 2 2 4 2 4
can_consume 0 1 1 0 1 1 1 1Read step 2. Capacity 8 is advertised at step 1 and visible at step 2. A cost-3 packet is eligible and sent; available falls to 5.
Read step 4 — the refusal. Available is 2 and the packet costs 3. can_consume is low, the packet is not sent, and nothing is decremented. The packet stays valid and stable; the credit state is untouched. That is the single most important behaviour in this chapter — a refused packet must leave no trace.
Read step 5 — the same-cycle case. A return of 4 arrives while a cost-2 packet is offered. Effective availability is 2 + 4 = 6, the packet is eligible, and the next value is 6 − 2 = 4. Both events applied, in one expression (§8).
Read step 7. A return with no consumption. Available rises.
And note what never happens: available never goes below zero, and never exceeds 8.
8. RTL — Generic Credit Account
// SYNTHESIZABLE. One normalized credit pool: advertised capacity, local
// accounting, eligibility.
// That a transmitter holds a local count, gates on it, and deducts as it
// sends is NORMATIVE (section 1). That zero means INFINITE at
// initialisation is NORMATIVE (section 1, Table 6 note). The error
// outputs, the same-cycle contract and the capacity ceiling are
// ILLUSTRATIVE IMPLEMENTATION POLICY.
module credit_account #(
// Widths are chosen by the instantiating pool. Header pools are small;
// data pools are not (section 1: PD's minimum advertisement is the
// largest supported MPS divided by the FC unit size).
parameter int CRED_W = 12,
parameter int COST_W = 12
) (
input logic clk,
input logic rst_n,
// ---- Initial advertisement, from InitFC (section 1) -------------------
input logic init_valid,
input logic [CRED_W-1:0] init_capacity,
// NORMATIVE: an advertisement of zero means INFINITE, not empty. It must
// be a distinct STATE, or the pool blocks that class forever (section 6).
input logic init_infinite,
// ---- Return, from Chapter 15.2's decoded update events ----------------
input logic return_valid,
input logic [CRED_W-1:0] return_count,
// ---- Consumption, on the caller's irrevocable send event --------------
input logic consume_valid,
input logic [COST_W-1:0] consume_cost,
// ---- Outputs ----------------------------------------------------------
output logic [CRED_W-1:0] available,
output logic can_consume, // is consume_cost affordable?
output logic initialised,
output logic underflow_error, // consumed without capacity
output logic overflow_error // returned beyond capacity
);
// =====================================================================
// THE SAME-CYCLE CONTRACT -- declared, because leaving it implicit is
// how two blocks end up disagreeing about one cycle.
//
// effective = available + returned-this-cycle
// can_consume = effective >= cost
// next = effective - consumed-this-cycle
//
// A return arriving THIS cycle IS usable by a consumption THIS cycle.
// That is a choice, and the alternative -- returns visible only next
// cycle -- is equally defensible and costs one cycle of latency per
// return. What is NOT defensible is deciding eligibility from the old
// value and then applying both, which can consume capacity that was
// never available.
// =====================================================================
// WIDENED ARITHMETIC. available + returned can exceed CRED_W bits, and
// effective - cost must be evaluated where a negative result is
// REPRESENTABLE rather than wrapped to a huge positive one.
localparam int ACC_W = ((CRED_W > COST_W) ? CRED_W : COST_W) + 2;
logic [CRED_W-1:0] avail_q;
logic [CRED_W-1:0] cap_q; // the advertised ceiling, for checking
logic init_q, inf_q;
logic uf_q, of_q;
assign available = avail_q;
assign initialised = init_q;
assign underflow_error = uf_q;
assign overflow_error = of_q;
wire [ACC_W-1:0] avail_x = ACC_W'(avail_q);
wire [ACC_W-1:0] ret_x = (return_valid && init_q && !inf_q)
? ACC_W'(return_count) : '0;
wire [ACC_W-1:0] cost_x = ACC_W'(consume_cost);
wire [ACC_W-1:0] cap_x = ACC_W'(cap_q);
// Effective capacity for THIS cycle's decision.
wire [ACC_W-1:0] effective = avail_x + ret_x;
// ELIGIBILITY. An infinite pool is always eligible -- it is not a large
// number, it is the absence of a limit (section 6).
assign can_consume = init_q && (inf_q || (effective >= cost_x));
wire do_consume = consume_valid && can_consume;
// Computed once, in one widened expression, from the same `effective`
// the eligibility decision used. Two separate expressions could disagree.
wire [ACC_W-1:0] next_x = effective - (do_consume ? cost_x : '0);
// A return that would push a pool past its advertised ceiling is a
// protocol or decode fault, not capacity (Chapter 15.2 section 6's
// plausibility argument, applied to the accumulated value).
wire ret_overflows = return_valid && init_q && !inf_q && (effective > cap_x);
// THE CLAMP APPLIES TO THE NEXT STATE, NOT TO THE RETURN.
// Clamping to cap_q whenever `effective` exceeded the ceiling would
// DISCARD a consumption happening in the same cycle -- the packet leaves,
// its cost is never deducted, and the pool is left at the full ceiling.
// That INVENTS credit (up to one packet's cost) at the exact moment the
// receiver has one more packet in it, which is the direction that
// overflows. Clamp `next_x` instead: it is already the post-consumption
// value, so the deduction survives the clamp.
wire next_exceeds_cap = (next_x > cap_x);
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
avail_q <= '0; cap_q <= '0;
init_q <= 1'b0; inf_q <= 1'b0;
uf_q <= 1'b0; of_q <= 1'b0;
end else begin
if (init_valid && !init_q) begin
init_q <= 1'b1;
inf_q <= init_infinite;
cap_q <= init_infinite ? '0 : init_capacity;
avail_q <= init_infinite ? '0 : init_capacity;
end else if (init_q && !inf_q) begin
// Clamped at the ceiling rather than allowed to run away, and the
// clamp is REPORTED so it can never be mistaken for normal.
avail_q <= next_exceeds_cap ? cap_q : CRED_W'(next_x);
end
if (ret_overflows) of_q <= 1'b1;
// A consumption offered without capacity must be REFUSED by the
// caller, not absorbed here. If it is asserted anyway, that is a
// contract violation and it is reported rather than silently wrapped.
if (consume_valid && !can_consume) uf_q <= 1'b1;
end
end
endmoduleClassification: synthesizable.
Architecture. One counter, one ceiling, two state bits, and one widened expression that both the eligibility decision and the next-state update are derived from.
The widening is the point. ACC_W is two bits wider than the larger of the credit and cost widths, so available + returned cannot wrap and effective - cost is only ever evaluated when it is non-negative. A design that computed avail_q + return_count - consume_cost at CRED_W would wrap through zero and present a huge availability — the classic credit-underflow bug, and it looks like the receiver granted enormous capacity.
Cycle behaviour.
return_valid | consume_valid | Result |
|---|---|---|
| 0 | 0 | idle |
| 1 | 0 | available rises, clamped and reported at the ceiling |
| 0 | 1, affordable | available falls by the cost |
| 0 | 1, not affordable | can_consume low; nothing changes; reported |
| 1 | 1 | effective = available + returned, decision and update from that one value |
| — | — | infinite pool: always eligible, counter never moves |
Contract. consume_valid is asserted only on the caller's irrevocable send event — §10 and Chapter 16.5 discuss where that boundary sits. return_count comes from Chapter 15.2's decoded return events, which are already differences rather than totals.
Failure — seven. Narrow arithmetic wraps below zero and reports vast availability. Deciding eligibility from avail_q and then applying both events consumes capacity that was never available. Treating an infinite advertisement as zero blocks that class permanently. Consuming on consume_valid alone, without can_consume, overflows the receiver. Letting returns accumulate past the ceiling hides a decode fault as capacity. Clamping silently turns a reportable fault into a mystery. And clamping on ret_overflows rather than on next_x discards a same-cycle consumption — the packet leaves, its cost is never deducted, and the pool is left at the full ceiling, inventing credit in the one direction that overflows the receiver.
Deliberately simplified: one return and one consumption per cycle; a single ceiling rather than per-VC capacity; no reservation state (§10); reinitialisation not modelled.
9. RTL — Atomic Header/Data Pair
// SYNTHESIZABLE. One traffic class, two pools, ATOMIC consumption.
// That a TLP consumes header and (when present) data credits together
// follows from section 1's consumption model. The joint-commit structure
// is the design consequence (section 9's callout).
module credit_pair_gate #(
parameter int CRED_W = 12,
parameter int COST_W = 12
) (
input logic clk,
input logic rst_n,
// ---- Packet descriptor, from the cost deriver -------------------------
input logic pkt_valid,
input logic needs_header,
input logic [COST_W-1:0] header_cost,
input logic needs_data,
input logic [COST_W-1:0] data_cost,
// ---- The irrevocable send event ---------------------------------------
// The caller asserts this ONLY when the packet is actually leaving. If
// the transmit path can still stall after this point, the caller has the
// wrong boundary -- see section 10.
input logic send_fire,
// ---- Pool interfaces ---------------------------------------------------
input logic hdr_return_valid,
input logic [CRED_W-1:0] hdr_return_count,
input logic dat_return_valid,
input logic [CRED_W-1:0] dat_return_count,
input logic init_valid,
input logic [CRED_W-1:0] init_hdr_capacity,
input logic init_hdr_infinite,
input logic [CRED_W-1:0] init_dat_capacity,
input logic init_dat_infinite,
// ---- Outputs -----------------------------------------------------------
output logic eligible,
output logic [CRED_W-1:0] hdr_available,
output logic [CRED_W-1:0] dat_available,
output logic pool_error
);
logic hdr_ok_raw, dat_ok_raw;
logic hdr_uf, hdr_of, dat_uf, dat_of;
logic hdr_init, dat_init;
// JOINT ELIGIBILITY, computed BEFORE either consume input is asserted.
// A pool the packet does not need is trivially satisfied -- a header-only
// packet must not be blocked by, or charged to, the data pool.
wire hdr_ok = !needs_header || hdr_ok_raw;
wire dat_ok = !needs_data || dat_ok_raw;
assign eligible = hdr_init && dat_init && hdr_ok && dat_ok;
// ATOMIC COMMIT. Both consume strobes are driven from the SAME condition,
// so there is no ordering between them and no state in which one pool has
// been charged and the other has not.
wire commit = pkt_valid && eligible && send_fire;
credit_account #(.CRED_W(CRED_W), .COST_W(COST_W)) u_hdr (
.clk, .rst_n,
.init_valid, .init_capacity(init_hdr_capacity),
.init_infinite(init_hdr_infinite),
.return_valid(hdr_return_valid), .return_count(hdr_return_count),
.consume_valid(commit && needs_header), .consume_cost(header_cost),
.available(hdr_available), .can_consume(hdr_ok_raw),
.initialised(hdr_init),
.underflow_error(hdr_uf), .overflow_error(hdr_of)
);
credit_account #(.CRED_W(CRED_W), .COST_W(COST_W)) u_dat (
.clk, .rst_n,
.init_valid, .init_capacity(init_dat_capacity),
.init_infinite(init_dat_infinite),
.return_valid(dat_return_valid), .return_count(dat_return_count),
.consume_valid(commit && needs_data), .consume_cost(data_cost),
.available(dat_available), .can_consume(dat_ok_raw),
.initialised(dat_init),
.underflow_error(dat_uf), .overflow_error(dat_of)
);
assign pool_error = hdr_uf | hdr_of | dat_uf | dat_of;
endmoduleClassification: synthesizable.
Architecture. Two credit_account instances and one commit wire driving both. There is deliberately no sequencing, no per-pool handshake and no intermediate state — the leak in the callout requires an ordering, and this structure has none to exploit.
The !needs_* terms matter more than they look. A header-only packet must be neither blocked by nor charged to the data pool. Omitting the term in eligibility stalls Memory Reads behind a full data pool; omitting it at consume charges them for payload they do not have. Both are in §12's mutation table.
Failure — four. Sequential consumption leaks the first pool's credit when the second is short. Deriving commit separately per pool reintroduces the ordering the structure exists to remove. Charging an unused pool drains it against packets that never used it. And gating eligible on send_fire would make eligibility depend on the send event that is supposed to depend on it — a loop.
10. RTL — Resource Eligibility Gate
Credits are one input to a launch decision, not the whole of it — and this block is where that becomes explicit.
// SYNTHESIZABLE. Hold a packet until every resource it needs is available.
// Credit eligibility is section 9's. The OTHER resources are named here so
// the multi-resource nature of a launch is visible in one place; Chapter
// 16.3 section 12 builds the Memory Read case fully.
// The set of resources modelled: ILLUSTRATIVE.
module credit_resource_gate (
input logic clk,
input logic rst_n,
// ---- The waiting packet ------------------------------------------------
input logic pkt_valid,
output logic pkt_ready,
// ---- Resources, each owned by a different chapter -----------------------
input logic credit_eligible, // section 9 -- REMOTE capacity
input logic replay_space, // Chapter 14.4 -- LOCAL storage
input logic tx_path_ready, // the transmit path this cycle
output logic send_fire,
// Which resource is missing. Not decoration: section 13's first
// debugging question is exactly this, and without it the answer needs a
// simulation run.
output logic blocked_on_credit,
output logic blocked_on_replay,
output logic blocked_on_path
);
// ALL OF THEM, ALWAYS. A launch that satisfied two of three would either
// overflow the receiver, or transmit a packet that could never be
// replayed if the Link lost it (Chapter 14.1 section 2).
wire all_ready = credit_eligible && replay_space && tx_path_ready;
assign pkt_ready = all_ready;
assign send_fire = pkt_valid && all_ready;
// Reported only while a packet is actually waiting -- an idle gate is not
// "blocked", and a flag that asserts when nothing wants to send is noise.
assign blocked_on_credit = pkt_valid && !credit_eligible;
assign blocked_on_replay = pkt_valid && !replay_space;
assign blocked_on_path = pkt_valid && !tx_path_ready;
endmoduleClassification: synthesizable (combinational).
Architecture. A conjunction and three reason flags. It holds no state — the packet is held by whatever queue owns it, and this gate only decides.
That is deliberate, and it fixes the consumption boundary. send_fire is asserted only when the packet is genuinely leaving, and §9 consumes on exactly that. A design that consumed when the scheduler selected the packet — before the transmit path confirmed it could take it — would charge credit for a packet that then stalled, and §13's second scenario is that bug seen from the outside.
11. Assertions
// SVA over credit_account, credit_pair_gate and credit_resource_gate.
// These assert the local accounting contract and the declared same-cycle
// semantics. They assert NOTHING about credit-return timing (an
// environment property, not a design one), nothing about the per-class
// consumption table (Chapters 16.2-16.4), nothing about spend policy
// (Chapter 16.5), and nothing about the update protocol (Chapter 16.6).
// ---- ENVIRONMENT ------------------------------------------------------
// A1: returns are decoded DIFFERENCES from Chapter 15.2, not totals.
assume property (@(posedge clk) disable iff (!rst_n)
return_valid |-> (return_count != '0));
// A2: send_fire is IRREVOCABLE -- the packet really leaves (section 10).
assume property (@(posedge clk) disable iff (!rst_n)
send_fire |-> pkt_valid);
// ---- CONSERVATION AND RANGE -------------------------------------------
// P1: available NEVER underflows. The property a narrow-arithmetic design
// fails by wrapping to a huge positive value.
property p_no_underflow;
@(posedge clk) disable iff (!rst_n)
initialised |-> (available <= cap_q);
endproperty
a_range : assert property (p_no_underflow);
// P2: THE CONSERVATION IDENTITY. A verification accounting statement, NOT
// PCIe wire state: everything advertised and returned is either still
// available or was spent. Ghost counters are testbench state.
property p_conservation;
@(posedge clk) disable iff (!rst_n)
(initialised && !inf_q)
|-> ((g_initial + g_returned - g_consumed) == available);
endproperty
a_conserve : assert property (p_conservation);
// P3: AN INFINITE POOL NEVER BLOCKS. Zero at initialisation means
// unlimited, not empty (section 6). A design that missed this stalls that
// class forever, and the symptom looks like a dead receiver.
property p_infinite_never_blocks;
@(posedge clk) disable iff (!rst_n)
(initialised && inf_q) |-> can_consume;
endproperty
a_infinite : assert property (p_infinite_never_blocks);
// P4: nothing is consumable before initialisation. A transmitter with no
// information must assume NO capacity (section 2).
property p_no_credit_before_init;
@(posedge clk) disable iff (!rst_n)
!initialised |-> !can_consume;
endproperty
a_uninit : assert property (p_no_credit_before_init);
// ---- THE SAME-CYCLE CONTRACT ------------------------------------------
// P5: eligibility uses EFFECTIVE capacity -- available plus this cycle's
// return (section 8's declared contract). Restated independently, so a
// design that decided from the stale value fails.
property p_effective_eligibility;
@(posedge clk) disable iff (!rst_n)
(initialised && !inf_q)
|-> (can_consume ==
((available + (return_valid ? return_count : '0)) >= consume_cost));
endproperty
a_effective : assert property (p_effective_eligibility);
// P6: return and consume in the SAME cycle both apply, exactly once each.
property p_simultaneous_applies_both;
@(posedge clk) disable iff (!rst_n)
(return_valid && consume_valid && can_consume && !inf_q)
|=> (available == ($past(available) + $past(return_count)
- $past(consume_cost)));
endproperty
a_simul : assert property (p_simultaneous_applies_both);
// P7: a return is applied EXACTLY ONCE -- not twice, not zero times.
property p_return_applied_once;
@(posedge clk) disable iff (!rst_n)
(return_valid && !consume_valid && !inf_q && !ret_overflows)
|=> (available == ($past(available) + $past(return_count)));
endproperty
a_return_once : assert property (p_return_applied_once);
// ---- OWNERSHIP --------------------------------------------------------
// P8: NO CREDIT IS CONSUMED MERELY BECAUSE A PACKET IS VALID. The single
// most common credit bug (section 13).
property p_no_consume_without_send;
@(posedge clk) disable iff (!rst_n)
(pkt_valid && !send_fire) |=> ($stable(hdr_available) && $stable(dat_available));
endproperty
a_no_phantom : assert property (p_no_consume_without_send);
// P9: A FAILED MULTI-RESOURCE ELIGIBILITY CHANGES NO POOL. The leak in
// section 9's callout, asserted directly.
property p_failed_eligibility_is_free;
@(posedge clk) disable iff (!rst_n)
(pkt_valid && !eligible)
|=> ($stable(hdr_available) && $stable(dat_available));
endproperty
a_atomic : assert property (p_failed_eligibility_is_free);
// P10: ATOMICITY. Either both needed pools moved, or neither did.
property p_pair_atomic;
@(posedge clk) disable iff (!rst_n)
(commit && needs_header && needs_data)
|=> (!$stable(hdr_available) && !$stable(dat_available));
endproperty
a_pair : assert property (p_pair_atomic);
// P11: ISOLATION. A packet that needs no data credit never touches the
// data pool -- it is neither charged nor blocked by it.
property p_header_only_leaves_data_alone;
@(posedge clk) disable iff (!rst_n)
(commit && needs_header && !needs_data) |=> $stable(dat_available);
endproperty
a_isolation : assert property (p_header_only_leaves_data_alone);
// P12: a credit-starved packet is STABLE. Not dropped, not mutated.
property p_starved_packet_stable;
@(posedge clk) disable iff (!rst_n)
(pkt_valid && !pkt_ready)
|=> (pkt_valid && $stable(header_cost) && $stable(data_cost));
endproperty
a_stable : assert property (p_starved_packet_stable);
// ---- CROSS-RESOURCE NON-INTERFERENCE ----------------------------------
// P13: REPLAY STORAGE IS NOT CREDIT (section 3). A full replay buffer
// blocks the launch and must not alter any credit counter.
property p_replay_does_not_touch_credit;
@(posedge clk) disable iff (!rst_n)
!replay_space |=> ($stable(hdr_available) && $stable(dat_available));
endproperty
a_not_replay : assert property (p_replay_does_not_touch_credit);
// P14: an ACK returns replay storage, never credit (Chapter 14.2).
property p_ack_returns_no_credit;
@(posedge clk) disable iff (!rst_n)
(dut_retire_window.retire_valid && !return_valid)
|=> ($stable(hdr_available) && $stable(dat_available));
endproperty
a_ack_not_credit : assert property (p_ack_returns_no_credit);
// P15: reset and initialisation are explicit.
property p_reset_clears;
@(posedge clk)
!rst_n |=> (!initialised && !can_consume && (available == '0));
endproperty
a_reset : assert property (p_reset_clears);P2 is the chapter's strongest property and it is worth naming precisely. It is a verification accounting identity, not a statement about PCIe wire state: the ghost counters exist only in the testbench. What it catches is any credit that appeared or disappeared without a corresponding event — which is every leak, every double-return and every phantom consumption in one property.
P5 is written as an independent restatement of the same-cycle contract, deliberately. A property that compared can_consume against the module's own effective would agree with a design that decided from the stale value. Restating the arithmetic is the only version that catches it.
P8 and P9 are the pair that catches the most common credit bug. P8 forbids consuming on validity; P9 forbids partial consumption on failed eligibility. A design can pass either alone — one that charges only the header pool on a failed pair satisfies P8 and fails P9.
P13 and P14 are non-interference properties across chapter boundaries, in the style Chapter 15.5 §11 established. They cannot be written inside either module — no single block sees both an ACK and a credit counter — and they are the only mechanism that catches the §3 confusions, because each block in isolation is correct.
No liveness is asserted. "Credits eventually return" is a property of the remote receiver, not of this design, and asserting it would import an assumption this chapter cannot justify.
12. Verification and Fault Injection
The scoreboard maintains its own pool arithmetic — initial capacity, cumulative returns, cumulative consumption — and never reads available, can_consume, or the module's cost helper. Using the DUT's eligibility function as the oracle would verify only that the design agrees with itself.
Range and boundaries
- An empty pool — verify
can_consumelow for any non-zero cost, and that offering a packet changes nothing (P8). - A full pool, and a return on top of it — verify the clamp and
overflow_error(P1). - An over-ceiling return arriving in the same cycle as an affordable send. Verify the consumption is still deducted and the pool does not sit at the ceiling — the clamp applies to the post-consumption value, not to the return. Required test: with capacity 8,
available = 7, a return of 3 and a cost of 5, the pool must land on 5, not 8. - Capacity of exactly 1, with cost 1 and cost 2.
- Consume down to exactly zero, then return from zero.
- Cost exactly equal to available — the exact-fit case, eligible.
- Cost one greater than available — refused, nothing changes. The pair of tests that catch an off-by-one in the comparison.
- Cost greater than the entire advertised capacity — permanently ineligible; verify it does not corrupt state while waiting.
The same-cycle contract
- Return only. Available rises by exactly the count (P7).
- Consume only. Falls by exactly the cost.
- Return and consume together, where the consumption is affordable only with the return counted. The declared-contract test (P5, P6).
- Return and consume together, unaffordable even with the return. Verify refusal and that the return still applies.
- Initialisation in the same cycle as a return.
Infinite credits
- Initialise with
init_infinite. Verifycan_consumeis always high, for any cost (P3), and that the counter never moves. - A return arriving on an infinite pool. Verify it is ignored rather than accumulating.
The pair
- Header available, data insufficient. Verify neither pool moves (P9) — the required test, and the one the leak fails.
- Data available, header insufficient. The mirror case.
- Both available. Verify both move, in the same cycle (P10).
- A header-only packet with the data pool at zero. Verify it is eligible and the data pool is untouched (P11).
- A header-only packet with the data pool full. Verify it is not charged.
Multi-resource
- Credit eligible, replay buffer full. Verify no launch and no credit consumed (P13).
- Replay space free, credit exhausted. Verify no launch, and that
blocked_on_creditnames the reason. - An ACK arriving with no UpdateFC. Verify credit counters are unmoved (P14).
- Reset mid-transaction, and reset with a packet waiting (P15).
Mutations
| # | Mutation | Caught by | Silicon symptom |
|---|---|---|---|
| 1 | consume on pkt_valid rather than send_fire | P8 | credits drain while the Link is idle; traffic stops |
| 2 | narrow arithmetic — counter wraps below zero | P1, at the exact-fit test | availability appears enormous; receiver overflows |
| 3 | return accumulates past the ceiling | P1, overflow_error | phantom capacity; overflow later |
| 4 | header pool decremented before data eligibility | P9, header-available/data-short test | slow leak; one class dies after minutes |
| 5 | a return applied twice | P2, P7 | capacity invented; intermittent overflow |
| 6 | the wrong pool updated | P2 per pool | one class starves while another over-sends |
| 7 | replay-buffer fullness treated as credit exhaustion | P13 | Link stalls when both are momentarily short |
| 8 | header-only packet charged to the data pool | P11 | reads drain the write payload pool |
| 9 | eligibility decided from avail_q, both events then applied | P5 | rare over-send exactly when a return coincides |
| 10 | effective - cost computed at CRED_W | P1, P2 | the wrap of mutation 2, one expression later |
| 11 | infinite advertisement stored as capacity zero | P3 | that class never sends; looks like a dead partner |
| 12 | ceiling clamp applied silently | overflow_error never asserts | a decode fault is invisible until it overflows |
| 13 | clamp keyed on ret_overflows instead of next_x | P2 — conservation breaks by exactly the discarded cost | an implausible return coinciding with a send leaves the pool at full ceiling; receiver overflow |
13. Debugging
A packet is queued, the Link is idle, the replay buffer has space — and nothing sends
This is the canonical credit symptom, and §10's reason flags answer it in one glance.
blocked_on_credit? Then the remote receiver has not advertised enough for this packet's class. Readavailablefor the pool the packet needs — not the aggregate.- Is that pool initialised? An uninitialised pool refuses everything (P4), and this is what a failed FC initialisation looks like from the transmitter.
- Was the advertisement infinite? If the design stored zero as capacity rather than as a flag, the pool blocks forever (P3) — and the symptom is indistinguishable from a receiver that never returns credit.
- Is the pool the packet needs the one you are watching? A Memory Read blocked on NPH while you watch PH is §4's isolation working correctly.
And note what "the Link is idle" does not tell you. Idle means no bandwidth problem. It says nothing about receive capacity, which is the whole point of §3.
Credits decrease while the transmitter is stalled
Consumption is tied to the wrong event.
The counter is moving on packet validity or on scheduler selection rather than on the irrevocable send (§10). Check whether the decrement correlates with pkt_valid or with send_fire — one waveform answers it.
The signature over time is a slow, monotonic drain with no packets on the wire, ending in a Link that has spent all its credit on packets that never left.
The credit count jumps after an update
Almost always the update decode, not the account — Chapter 15.2 §13.
A cumulative total treated as a delta injects an enormous return. Check whether overflow_error fired: if it did, the account caught it at the ceiling and the fault is upstream. If it did not and the count still jumped, the ceiling check is missing.
One traffic class starves while others flow
Check whether this is a bug at all.
Pool isolation means exactly this: Posted traffic flowing while Non-Posted starves is the architecture working (§4). The question is not "why is one class blocked" but "why is that pool not being returned", which points at the remote receiver's drain rate, not at the local transmitter.
It becomes a bug only if the starved pool has capacity and the packet still will not go — and then it is mutation 6 or 8, a wrong-pool update or a wrong-pool charge.
14. Common Misconceptions
- "Credits are Link bandwidth." They are receive storage permission. An idle Link can be fully credit-blocked (§3).
- "Credits are replay-buffer entries." Local TX storage versus remote RX capacity — different resources, freed by different packets (§3).
- "PCIe has a per-packet ready signal from the receiver." It cannot; the latency makes it useless. Capacity is advertised in advance (§2).
- "All traffic shares one credit count." Six are tracked, and the isolation is what prevents a class from being starved by another (§4).
- "Header and data credits are interchangeable." Different resources, different units, tracked separately (§4, §5).
- "One credit is one byte." A data credit is 4 DW — 16 bytes. A header credit is one header (§5).
- "A header credit's size depends on whether the header is 3DW or 4DW." The unit is sized for the largest header plus Digest, so every header costs one (§5).
- "Credit should decrement when the packet becomes valid." On the irrevocable send, or the counter drains while nothing is sent (§10, P8).
- "Returned credit means the operation completed." It means buffer space was freed — possibly long before anything is visible (§3).
- "An ACK returns flow-control credit." An ACK retires replay storage (Chapter 14.2); UpdateFC returns credit (P14).
- "An UpdateFC acknowledges TLP reliability." Different mechanism, different layer, different guarantee (Chapter 15.2).
- "Zero credits advertised means the receiver is broken." Zero at initialisation means infinite (§6, §1).
- "Credit starvation means the PHY is failing." It means the far side is not draining as fast as you are sending — a flow-control condition, not an electrical one.
- "Having enough credits guarantees low latency." It guarantees the packet may land. Queueing, scheduling and the return latency all still apply (Chapter 12.5).
15. Understanding Check
16. What's Next
Backpressure across a Link it cannot reach, solved by inverting who speaks: the receiver advertises, the transmitter accounts, and the count is stale only in the safe direction.
Six pools, because three transaction classes with different lifetimes must not compete for one buffer, and headers and payloads are not the same resource. Two units — one header, or 4 DW of data. And one rule that most of the RTL exists to enforce: check every pool, then charge every pool, in a single event.
Chapter 16.2 — Posted Credits makes PH and PD concrete: what a Memory Write actually costs, the ceiling division that turns a payload length into data credits, and why posted traffic — which needs no Completion — is anything but free.
Chapter 16.3 — Non-Posted Credits then takes NPH and NPD, and the resource question that catches experienced engineers: why a requester with free Tags, free replay space and an idle Link still cannot issue a read.
The idea to carry forward: a credit is not permission to succeed — it is permission to arrive.