PCIe · Module 16
Credit Updates — A Total, Delivered Twice, Counted Once
The receiver never sends a token per packet. It sends a running total that wraps, and the transmitter subtracts a baseline to recover what was freed — so a duplicate adds nothing and a lost update costs nothing.
Every chapter in Module 16 has spent credit. None has explained where it comes back from.
The naive answer would be a token: the receiver frees a buffer, sends back "here is one credit", and the transmitter adds one. It would be simple, and it would be fragile — a lost token is capacity gone forever, and a duplicated one is capacity invented.
PCIe does something better and stranger. The receiver reports a running total of everything it has ever freed, in a counter that wraps. The transmitter subtracts what it last saw and adds the difference.
When does receive storage actually become reusable, how is that progress carried, and how does a wrapping total become live credit exactly once — never twice on a duplicate, and never lost when an update goes missing?
1. The Verified Mechanism
2. Three Counters, and They Are Not Interchangeable
This is the chapter's organising distinction, and every bug in §14 is a confusion between two of these.
| Counter | Lives at | Is | Changes when |
|---|---|---|---|
| Cumulative freed total | the receiver | a protocol value, wraps | the receiver frees a buffer |
| Last-accepted baseline | the transmitter | a local copy of the above | an update is accepted |
| Live available credit | the transmitter | local send permission | a packet is sent, or a difference is added |
3. When Is Storage Actually Free?
Not when the packet arrives, and not when the transaction completes.
TLP arrives
→ occupies the receiver's buffer ← credit is "spent"
→ moved into internal processing / storage
→ THE BUFFER IS NOW REUSABLE ← the freeing event
→ cumulative freed total advances
→ an UpdateFC eventually carries the new total
→ the transmitter differences it into live credit4. A Total, Not a Delta
The single most important sentence in the chapter, and the one bug it prevents is fatal.
The value carried is how much has been freed in total, since initialisation, modulo the counter width. It is not "how much was freed since the last update".
transmitter: returned = new_total − last_total (modulo)
available += returned
last_total = new_total5. The Wrap, and What Makes a Difference Plausible
Start with a small counter so the arithmetic is visible.
A 4-bit cumulative counter, modulus 16:
last_total = 14
new_total = 2
modular difference = (2 − 14) mod 16 = 4And that is correct, because the receiver's progress was 14 → 15 → 0 → 1 → 2 — four freeings. An ordinary signed subtraction gives −12, which is meaningless as a credit return and, if added to an unsigned counter, catastrophic.
Fixed-width unsigned subtraction is modular subtraction — the same fact Chapter 15.2 §6 and Chapter 14.5 §6 relied on. Widening the operands before subtracting destroys it, and §7's RTL is written to make that impossible.
6. A Trace
Internal teaching signals, not PCIe wire signals. One pool, capacity 8, a 4-bit cumulative counter for legibility.
step 1 2 3 4 5 6 7 8
rx_free 1 1 0 1 0 0 1 0
rx_free_count 2 3 - 4 - - 2 -
rx_total 14 1 1 5 5 5 7 7
dirty 1 1 1 1 1 0 1 1
update_valid 0 1 1 1 1 1 0 1
update_ready 0 0 0 1 0 1 0 0
update_total - 14 14 14 5 5 - 7
tx_last 0 0 0 0 14 14 14 5
returned_delta - - - 14 - 7 - -
tx_available 8 8 8 8 8 8 8 8Read step 1. The receiver frees 2 units; the cumulative total moves to 14. dirty is set — there is unadvertised progress.
Read step 2 — the wrap. It frees 3 more. 14 + 3 = 17, and modulo 16 that is 1. The total wrapped and nothing special happened: the counter simply advanced.
Read step 4. An update carrying 14 is taken. Note it carries the total as of when the descriptor was offered (step 2), not the current total of 5 — §8's immutability contract.
Read step 5. The transmitter's baseline becomes 14, and the difference against its previous baseline of 0 is 14.
Read step 6 — the second update. It carries 5. The difference is (5 − 14) mod 16 = 7, which is exactly the freeings between the two updates. dirty clears here because the total just sent matches the current total.
Read step 7 — the race that §9 is about. The receiver frees 2 more in the same region where an update just completed. dirty sets again, because there is new unadvertised progress. A design that cleared dirty unconditionally on the handshake would have lost this.
And note tx_available never moves in this trace, because nothing is being sent — the returns are shown arriving and the pool is already at capacity. The trace is about the counters, not the spending.
7. RTL — Receiver Cumulative Return Counter
// SYNTHESIZABLE. Receiver-side cumulative freed-credit accounting.
// That the receiver reports progress as the RECEIVER EMPTIES ITS BUFFERS,
// via UpdateFC, is NORMATIVE (section 1). That the value is a CUMULATIVE
// TOTAL modulo a fixed field width is at the tier section 1 assigns it --
// hence COUNTER_W is a parameter. The dirty-bit scheduling and the error
// reports are ILLUSTRATIVE IMPLEMENTATION POLICY.
module fc_return_counter #(
parameter int POOLS = 6,
// GUARDED INDEX WIDTH. $clog2(1) is zero and a zero-width port cannot
// index anything -- the corner Chapter 15.2 was fixed for.
parameter int POOL_W = (POOLS <= 1) ? 1 : $clog2(POOLS),
// The PROTOCOL counter width. Deliberately not hardcoded (section 1).
parameter int COUNTER_W = 12
) (
input logic clk,
input logic rst_n,
// ---- Initialisation, from InitFC (section 1) ---------------------------
// Normalized: this module does not implement the InitFC1/InitFC2 state
// machine, it takes the resulting starting total.
input logic init_valid,
input logic [POOL_W-1:0] init_pool,
input logic [COUNTER_W-1:0] init_total,
// ---- The FREEING EVENT, from the receiver's internal machinery ---------
// Asserted when buffer capacity becomes REUSABLE -- not when the packet
// arrived, and not when the transaction completed (section 3).
input logic free_valid,
input logic [POOL_W-1:0] free_pool,
input logic [COUNTER_W-1:0] free_count,
// ---- To the update scheduler -------------------------------------------
output logic [COUNTER_W-1:0] total [POOLS],
output logic [POOLS-1:0] dirty,
// Asserted by the scheduler when an update for this pool has been taken.
input logic sent_valid,
input logic [POOL_W-1:0] sent_pool,
input logic [COUNTER_W-1:0] sent_total,
output logic free_pool_error
);
generate
if (POOLS < 1) $error("POOLS must be at least 1");
if (POOL_W < 1) $error("POOL_W must be at least 1");
if ((POOLS > 1) && ((1 << POOL_W) < POOLS))
$error("POOL_W too narrow to index POOLS");
if (COUNTER_W < 2) $error("COUNTER_W must be at least 2");
endgenerate
logic [COUNTER_W-1:0] total_q [POOLS];
logic [POOLS-1:0] dirty_q;
logic err_q;
assign total = total_q;
assign dirty = dirty_q;
assign free_pool_error = err_q;
// RANGE SAFETY. A POOL_W-wide index can express values >= POOLS whenever
// POOLS is not a power of two. Checked one bit wider, BEFORE any access.
localparam int CHK = POOL_W + 1;
wire free_legal = (CHK'(free_pool) < CHK'(POOLS));
wire init_legal = (CHK'(init_pool) < CHK'(POOLS));
wire sent_legal = (CHK'(sent_pool) < CHK'(POOLS));
wire do_free = free_valid && free_legal;
wire do_sent = sent_valid && sent_legal;
// The total AFTER this cycle's free, for the pool being freed. Computed
// once so the dirty logic and the counter update cannot disagree.
wire [COUNTER_W-1:0] free_next =
do_free ? (total_q[free_pool] + free_count) : '0;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int i = 0; i < POOLS; i++) total_q[i] <= '0;
dirty_q <= '0;
err_q <= 1'b0;
end else begin
if (init_valid && init_legal) begin
total_q[init_pool] <= init_total;
dirty_q[init_pool] <= 1'b0;
end
// MODULAR BY CONSTRUCTION. Fixed-width unsigned addition wraps at
// 2**COUNTER_W, which IS the protocol modulus. Never widen this.
if (do_free) begin
total_q[free_pool] <= free_next;
dirty_q[free_pool] <= 1'b1;
end
// ================================================================
// THE DIRTY-CLEAR RACE (section 9).
//
// Clearing dirty[sent_pool] unconditionally on the handshake LOSES
// any freeing that happened in the SAME cycle: the update carried
// the OLD total, the counter has already advanced past it, and the
// pool is now marked clean with unadvertised progress in it.
//
// The condition is not "was this pool sent" but "does the total the
// update carried still match the total we now hold". Written as an
// explicit comparison so the same-cycle case falls out rather than
// needing to be special-cased.
// ================================================================
if (do_sent) begin
automatic logic [COUNTER_W-1:0] now =
(do_free && (free_pool == sent_pool)) ? free_next
: total_q[sent_pool];
if (sent_total == now) dirty_q[sent_pool] <= 1'b0;
// else: the pool STAYS dirty -- there is newer progress to advertise
end
if (free_valid && !free_legal) err_q <= 1'b1;
end
end
endmoduleClassification: synthesizable.
Architecture. One cumulative counter per pool, one dirty bit per pool, and a comparison rather than an unconditional clear.
The modular arithmetic is structural, not computed. total_q[p] + free_count at COUNTER_W bits wraps at 2**COUNTER_W because that is what fixed-width addition does. There is no modulo operator anywhere, and there must not be — a % invites a widened intermediate, and a widened intermediate is no longer modular.
Cycle behaviour.
| Event | Result |
|---|---|
free | total advances modulo, pool marked dirty |
sent, total still matches | dirty cleared |
sent and free on the same pool, same cycle | dirty stays set — §9 |
sent on a pool with newer progress | dirty stays set |
out-of-range free_pool | nothing touched, reported |
init | total set, dirty cleared |
Failure — six. $clog2(POOLS) in a port width fails to elaborate at POOLS = 1. Indexing before the bounds check corrupts a neighbouring pool at any non-power-of-two POOLS. Clearing dirty unconditionally on the handshake loses a same-cycle freeing (§9). A widened intermediate in the addition destroys the wrap. Advancing the wrong pool's total attributes capacity to the wrong class. And treating free_valid as "packet arrived" returns credit for buffers that are still occupied (§3).
Deliberately simplified: one free and one send per cycle; no InitFC state machine; capacity not modelled here — the plausibility bound lives at the transmitter (§10).
8. RTL — Update Scheduler
// SYNTHESIZABLE. Select one dirty pool and offer a normalized FC update
// descriptor.
// That updates are scheduled periodically per non-infinite pool is
// NORMATIVE (section 1: every 32 microseconds, or 120 with Extended
// Synch). The SELECTION POLICY and the immutability contract below are
// ILLUSTRATIVE IMPLEMENTATION POLICY -- section 1's sources state the
// timing, not the arbitration.
module fc_update_scheduler #(
parameter int POOLS = 6,
parameter int POOL_W = (POOLS <= 1) ? 1 : $clog2(POOLS),
parameter int COUNTER_W = 12
) (
input logic clk,
input logic rst_n,
input logic [COUNTER_W-1:0] total [POOLS],
input logic [POOLS-1:0] dirty,
// Pools advertised as infinite are never scheduled (section 1).
input logic [POOLS-1:0] pool_infinite,
// ---- Normalized update descriptor out ----------------------------------
// Chapter 15.1 owns the DLLP encoding; this is the boundary above it.
output logic upd_valid,
input logic upd_ready,
output logic [POOL_W-1:0] upd_pool,
output logic [COUNTER_W-1:0] upd_total,
// ---- Back to the return counter ----------------------------------------
output logic sent_valid,
output logic [POOL_W-1:0] sent_pool,
output logic [COUNTER_W-1:0] sent_total
);
logic v_q;
logic [POOL_W-1:0] pool_q;
logic [COUNTER_W-1:0] total_q;
assign upd_valid = v_q;
assign upd_pool = pool_q;
assign upd_total = total_q;
wire fire = v_q && upd_ready;
// The descriptor is reported as SENT on the handshake, carrying the
// total it actually contained -- which is what lets the return counter
// decide whether the pool is still dirty (section 7).
assign sent_valid = fire;
assign sent_pool = pool_q;
assign sent_total = total_q;
wire [POOLS-1:0] eligible = dirty & ~pool_infinite;
// Lowest-index dirty pool. A FIXED PRIORITY, stated as such: section 1's
// sources give the update TIMING, not the arbitration, so nothing here
// claims round-robin is mandated. A production scheduler would rotate;
// the liveness argument in section 13 assumes fairness explicitly.
logic any_eligible;
logic [POOL_W-1:0] sel;
always_comb begin
any_eligible = 1'b0;
sel = '0;
for (int p = POOLS-1; p >= 0; p--)
if (eligible[p]) begin
any_eligible = 1'b1;
sel = POOL_W'(p);
end
end
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
v_q <= 1'b0; pool_q <= '0; total_q <= '0;
end else begin
// ================================================================
// IMMUTABILITY CONTRACT: once offered, a descriptor does not change
// until it is taken.
//
// The alternative -- refreshing the offered total as new capacity is
// freed, the way Chapter 15.3's ACK coalescer refreshes its frontier
// -- is tempting and is NOT done here. The reason is that the return
// counter's dirty logic is written against "the total this update
// carried", and a descriptor that mutates under it would make that
// comparison meaningless.
//
// Nothing is lost: new frees keep the pool DIRTY, so they are
// advertised by the next update (section 7).
// ================================================================
if (!v_q && any_eligible) begin
v_q <= 1'b1;
pool_q <= sel;
total_q <= total[sel];
end else if (fire) begin
v_q <= 1'b0;
end
end
end
endmoduleClassification: synthesizable.
Architecture. A priority select and a one-entry output register with an immutable descriptor.
The immutability decision is the interesting one, and it is the opposite of Chapter 15.3's. There, an offered ACK frontier is refreshed to newer progress while it waits, because the frontier is cumulative and naming more is strictly better. Here the total is also cumulative and refreshing would also be safe on the wire — but the receiver's dirty logic compares the sent total against the current total, and a descriptor that mutated after being sampled would break that comparison.
So the two blocks make opposite choices for the same underlying reason: each picks the option that keeps its partner's bookkeeping checkable. Nothing is lost either way — new frees keep the pool dirty and ride the next update.
Failure — four. Mutating the offered descriptor breaks the return counter's dirty comparison. Scheduling an infinite pool wastes update bandwidth on a pool that will never be differenced. Clearing the offering before the handshake loses an update. And asserting sent_valid on offer rather than on fire tells the return counter a pool was advertised when it was not.
Deliberately simplified: fixed priority rather than rotation — stated, not hidden; no periodic timer (§1 gives the intervals, and modelling a 32 µs timer adds nothing to the accounting lesson); one update in flight.
9. The Dirty-Clear Race
The subtlest bug in Module 16, and it is worth its own section because the wrong code looks obviously right.
// WRONG. Two independent updates that are correct in isolation.
if (update_fire)
dirty_q[sel] <= 1'b0;
if (free_fire)
total_q[free_pool] <= total_q[free_pool] + free_count;10. RTL — Transmitter Return Integration
// SYNTHESIZABLE. Convert a received cumulative FC total into newly
// returned credit, exactly once.
// That the value is a cumulative total and the difference is modular is
// the mechanism of section 4, at the tier section 1 assigns it. The
// plausibility bounds and the rejection policy are ILLUSTRATIVE LOCAL
// POLICY (section 5).
module fc_return_integrate #(
parameter int POOLS = 6,
parameter int POOL_W = (POOLS <= 1) ? 1 : $clog2(POOLS),
parameter int COUNTER_W = 12,
parameter int CRED_W = 12
) (
input logic clk,
input logic rst_n,
// ---- Initialised baseline and capacity, per pool -----------------------
input logic init_valid,
input logic [POOL_W-1:0] init_pool,
input logic [COUNTER_W-1:0] init_total,
input logic [CRED_W-1:0] init_capacity,
// ---- Decoded update, from Chapter 15.2 ----------------------------------
input logic upd_valid,
output logic upd_ready,
input logic [POOL_W-1:0] upd_pool,
input logic [COUNTER_W-1:0] upd_total,
// ---- How much is currently outstanding, from Chapter 16.5 --------------
// Sent but not yet returned. The TIGHTER plausibility bound (section 5).
input logic [CRED_W-1:0] outstanding [POOLS],
// ---- Normalized return event, to Chapter 16.5's accounts ---------------
output logic ret_valid,
input logic ret_ready,
output logic [POOL_W-1:0] ret_pool,
output logic [CRED_W-1:0] ret_count,
output logic fc_update_error
);
logic [COUNTER_W-1:0] last_q [POOLS];
logic [CRED_W-1:0] cap_q [POOLS];
logic [POOLS-1:0] init_q;
logic err_q;
localparam int CHK = POOL_W + 1;
wire upd_legal = (CHK'(upd_pool) < CHK'(POOLS));
wire init_legal = (CHK'(init_pool) < CHK'(POOLS));
logic [COUNTER_W-1:0] last_sel;
logic [CRED_W-1:0] cap_sel, out_sel;
logic init_sel;
always_comb begin
last_sel = '0; cap_sel = '0; out_sel = '0; init_sel = 1'b0;
if (upd_legal) begin
last_sel = last_q[upd_pool];
cap_sel = cap_q[upd_pool];
out_sel = outstanding[upd_pool];
init_sel = init_q[upd_pool];
end
end
// ================================================================
// THE MODULAR DIFFERENCE.
//
// Fixed-width unsigned subtraction IS modular subtraction. There is no
// modulo operator here and there must not be: a `%` invites a widened
// intermediate, and a widened intermediate is no longer modular
// (section 5, and Chapter 14.5 section 6).
//
// A DUPLICATE update gives upd_total == last_sel, so returned == 0 and
// nothing is added. That is the whole de-duplication mechanism.
// ================================================================
wire [COUNTER_W-1:0] returned = upd_total - last_sel;
// PLAUSIBILITY, two bounds, both local (section 5).
// - a receiver cannot free more than its buffer holds
// - a transmitter cannot be owed more than it has outstanding
// Compared at COUNTER_W so a large modular difference cannot be
// truncated into a small plausible one.
wire [COUNTER_W-1:0] cap_x = COUNTER_W'(cap_sel);
wire [COUNTER_W-1:0] out_x = COUNTER_W'(out_sel);
wire plausible = init_sel && (returned <= cap_x) && (returned <= out_x);
// ---- Output holding stage ----------------------------------------------
logic ret_v_q;
logic [POOL_W-1:0] ret_p_q;
logic [CRED_W-1:0] ret_c_q;
wire ret_fire = ret_v_q && ret_ready;
assign upd_ready = !ret_v_q || ret_ready;
assign ret_valid = ret_v_q;
assign ret_pool = ret_p_q;
assign ret_count = ret_c_q;
assign fc_update_error = err_q;
wire accept = upd_valid && upd_ready && upd_legal && plausible;
// A zero difference is a legitimate no-op -- a duplicate update. It is
// ACCEPTED (the baseline is already correct) and emits NO event.
wire emit = accept && (returned != '0);
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int i = 0; i < POOLS; i++) begin
last_q[i] <= '0; cap_q[i] <= '0;
end
init_q <= '0; ret_v_q <= 1'b0; ret_p_q <= '0; ret_c_q <= '0;
err_q <= 1'b0;
end else begin
if (init_valid && init_legal && !init_q[init_pool]) begin
last_q[init_pool] <= init_total;
cap_q[init_pool] <= init_capacity;
init_q[init_pool] <= 1'b1;
end
// ============================================================
// THE BASELINE ADVANCES ONLY ON AN ACCEPTED UPDATE.
//
// Advancing it on a REJECTED one would measure the next difference
// from a total the sender never sent -- silently losing every
// credit between the rejected value and the next one, permanently.
// ============================================================
if (accept) last_q[upd_pool] <= upd_total;
// Emit has priority over drain, so a same-cycle accept-and-consume
// holds the new event rather than dropping it.
if (emit) begin
ret_v_q <= 1'b1;
ret_p_q <= upd_pool;
ret_c_q <= CRED_W'(returned);
end else if (ret_fire) begin
ret_v_q <= 1'b0;
end
if (upd_valid && !upd_legal) err_q <= 1'b1;
if (upd_valid && upd_legal && !plausible) err_q <= 1'b1;
end
end
endmoduleClassification: synthesizable.
Architecture. A baseline per pool, a modular subtraction, two plausibility bounds, and an output holding stage.
The outstanding bound is new here and it is tighter than the capacity bound Chapter 15.2 used. A transmitter cannot be owed capacity it never spent — so a return larger than what is currently in flight is impossible regardless of how big the pool is. It catches a corrupted update that happens to fall under the capacity ceiling.
Cycle behaviour.
| Situation | Result |
|---|---|
| first valid update | baseline set, difference emitted |
| duplicate — same total | difference 0, accepted, no event |
| later update after a lost one | difference covers both |
| difference across the wrap | correct, by construction |
| implausible difference | refused, baseline unchanged, reported |
| out-of-range pool | nothing touched, reported |
| update before init | refused, reported |
| emit and consume same cycle | emit wins — event held |
Failure — six. Adding upd_total directly treats a total as a delta and injects hundreds of credits (§4). Widening before subtracting destroys the wrap. Advancing the baseline on a rejected update loses every credit between that value and the next, permanently. Dropping a decoded return event under backpressure throws away capacity that was correctly measured — and unlike a lost update, that is not recoverable. Emitting a zero difference as an event feeds phantom returns downstream. And indexing before the bounds check corrupts a neighbouring pool.
Deliberately simplified: one update per cycle; outstanding supplied by Chapter 16.5; no InitFC sequencing.
11. The Whole Loop
Three things to read out of the figure.
Only one arrow crosses the Link carrying a protocol value — the UpdateFC. Everything else is local on one side or the other. The entire mechanism rests on one packet type and one number.
The freeing event is inside the receiver, between the buffer and the consumer, and PCIe does not specify where exactly (§3). The protocol constrains the accounting, not the pipeline.
And the transmitter's last two steps are a subtraction and an addition — the difference against the baseline, then the addition to live credit. Those are three different counters (§2), and collapsing any two is a distinct bug.
12. Assertions
// SVA over fc_return_counter, fc_update_scheduler and
// fc_return_integrate. These assert the LOCAL accounting contract and the
// declared scheduling policy. Liveness appears only with an explicit
// fairness assumption (P17).
// ---- ENVIRONMENT ------------------------------------------------------
// A1: free_valid means buffer capacity became REUSABLE (section 3) --
// not that a packet arrived, and not that a transaction completed.
assume property (@(posedge clk) disable iff (!rst_n)
free_valid |-> (free_count != '0));
// A2: the update descriptor is stable while offered (section 8).
assume property (@(posedge clk) disable iff (!rst_n)
(upd_valid && !upd_ready) |=> ($stable(upd_pool) && $stable(upd_total)));
// ---- RECEIVER SIDE ----------------------------------------------------
// P1: a free advances ONLY the selected pool.
generate for (genvar p = 0; p < POOLS; p++) begin : g_iso
property p_only_selected_pool;
@(posedge clk) disable iff (!rst_n)
(do_free && (free_pool != POOL_W'(p))) |=> $stable(total_q[p]);
endproperty
a_pool_iso : assert property (p_only_selected_pool);
end endgenerate
// P2: RANGE SAFETY -- an out-of-range pool index touches nothing.
property p_no_oob_access;
@(posedge clk) disable iff (!rst_n)
(free_valid && !free_legal) |=> (free_pool_error && $stable(total_q));
endproperty
a_oob : assert property (p_no_oob_access);
// P3: the cumulative total wraps EXACTLY at the declared modulus. An
// independent restatement, so a widened intermediate fails.
property p_wraps_at_modulus;
@(posedge clk) disable iff (!rst_n)
do_free |=> (total_q[$past(free_pool)]
== COUNTER_W'(($past(total_q[$past(free_pool)])
+ $past(free_count))));
endproperty
a_wrap : assert property (p_wraps_at_modulus);
// P4: dirty is SET whenever unadvertised progress exists.
property p_dirty_when_unadvertised;
@(posedge clk) disable iff (!rst_n)
do_free |=> dirty_q[$past(free_pool)];
endproperty
a_dirty_set : assert property (p_dirty_when_unadvertised);
// P5: THE RACE PROPERTY (section 9). An update handshake clears dirty ONLY
// if no newer progress remains. Same-cycle free on the same pool must
// leave the pool dirty.
property p_dirty_survives_same_cycle_free;
@(posedge clk) disable iff (!rst_n)
(do_sent && do_free && (free_pool == sent_pool))
|=> dirty_q[$past(sent_pool)];
endproperty
a_race : assert property (p_dirty_survives_same_cycle_free);
// P5b: and the general form -- dirty clears only on an exact match.
property p_dirty_clears_only_on_match;
@(posedge clk) disable iff (!rst_n)
(do_sent && (sent_total != total_q[sent_pool]))
|=> dirty_q[$past(sent_pool)];
endproperty
a_dirty_match : assert property (p_dirty_clears_only_on_match);
// ---- SCHEDULER --------------------------------------------------------
// P6: an offered descriptor is IMMUTABLE until taken (section 8).
property p_descriptor_stable;
@(posedge clk) disable iff (!rst_n)
(upd_valid && !upd_ready) |=> (upd_valid && $stable(upd_pool)
&& $stable(upd_total));
endproperty
a_stable : assert property (p_descriptor_stable);
// P7: an infinite pool is never scheduled (section 1).
property p_infinite_not_scheduled;
@(posedge clk) disable iff (!rst_n)
upd_valid |-> !pool_infinite[upd_pool];
endproperty
a_infinite : assert property (p_infinite_not_scheduled);
// P8: sent_valid asserts on the HANDSHAKE, not on the offer.
property p_sent_on_fire;
@(posedge clk) disable iff (!rst_n)
sent_valid |-> (upd_valid && upd_ready);
endproperty
a_sent : assert property (p_sent_on_fire);
// ---- TRANSMITTER SIDE -------------------------------------------------
// P9: A DUPLICATE UPDATE DERIVES ZERO NEW CREDIT. The property a design
// that adds the total outright fails on the first repeat.
property p_duplicate_returns_zero;
@(posedge clk) disable iff (!rst_n)
(upd_valid && upd_legal && (upd_total == last_sel)) |-> !emit;
endproperty
a_duplicate : assert property (p_duplicate_returns_zero);
// P10: the returned amount equals an INDEPENDENT modular reference.
// Restated, so a widened or signed implementation fails rather than
// agreeing with itself.
property p_delta_matches_reference;
@(posedge clk) disable iff (!rst_n)
(upd_valid && upd_legal)
|-> (returned == COUNTER_W'(upd_total - last_sel));
endproperty
a_delta : assert property (p_delta_matches_reference);
// P11: AN ACCEPTED UPDATE ADVANCES THE BASELINE EXACTLY ONCE.
property p_baseline_once;
@(posedge clk) disable iff (!rst_n)
accept |=> (last_q[$past(upd_pool)] == $past(upd_total));
endproperty
a_baseline : assert property (p_baseline_once);
// P12: A REJECTED UPDATE CANNOT CORRUPT THE BASELINE. Advancing it on a
// rejected value loses every credit up to the next one, permanently.
property p_rejected_leaves_baseline;
@(posedge clk) disable iff (!rst_n)
(upd_valid && upd_legal && !plausible) |=> $stable(last_q[$past(upd_pool)]);
endproperty
a_reject : assert property (p_rejected_leaves_baseline);
// P13: newly returned credit is added EXACTLY ONCE. Ghost counters are
// testbench state -- a VERIFICATION ACCOUNTING IDENTITY, not wire state.
property p_returned_once;
@(posedge clk) disable iff (!rst_n)
(ret_valid && ret_ready)
|=> (g_credit_added == $past(g_credit_added) + $past(ret_count));
endproperty
a_once : assert property (p_returned_once);
// P14: NO CREDIT APPEARS WITHOUT A RECEIVER FREE OR AN INITIALISATION.
// Bound across the Link in a two-port testbench: the total credit the
// transmitter has been given never exceeds what the receiver has freed.
property p_no_spontaneous_credit;
@(posedge clk) disable iff (!rst_n)
(g_credit_added <= (g_rx_freed + g_initial_advertised));
endproperty
a_no_magic : assert property (p_no_spontaneous_credit);
// P15: a decoded return event is not lost under backpressure.
property p_event_not_lost;
@(posedge clk) disable iff (!rst_n)
(ret_valid && !ret_ready) |=> (ret_valid && $stable(ret_count)
&& $stable(ret_pool));
endproperty
a_no_loss : assert property (p_event_not_lost);
// ---- CROSS-MECHANISM NON-INTERFERENCE ---------------------------------
// P16: AN ACK CANNOT MOVE THE FC BASELINE. Three feedback loops, three
// mechanisms (section 16).
property p_ack_does_not_touch_fc;
@(posedge clk) disable iff (!rst_n)
(dut_retire_window.retire_valid && !upd_valid) |=> $stable(last_q);
endproperty
a_ack : assert property (p_ack_does_not_touch_fc);
// P16b: nor does a Completion arrival return FC credit.
property p_completion_returns_no_credit;
@(posedge clk) disable iff (!rst_n)
(dut_cpl_arrives && !upd_valid) |=> !ret_valid;
endproperty
a_cpl : assert property (p_completion_returns_no_credit);
// P16c: an FC update never alters replay state.
property p_fc_does_not_touch_replay;
@(posedge clk) disable iff (!rst_n)
(upd_valid && upd_ready) |=> $stable(dut_replay_buffer.occupancy);
endproperty
a_replay : assert property (p_fc_does_not_touch_replay);
// ---- LIVENESS, WITH ITS ASSUMPTION STATED -----------------------------
// F1: FAIRNESS ASSUMPTION. The scheduler eventually selects any dirty
// pool, and the transport eventually accepts. This is an ASSUMPTION about
// the environment and the arbiter, not a property of this design.
assume property (@(posedge clk) disable iff (!rst_n)
s_eventually upd_ready);
// P17: given F1, unadvertised progress is eventually advertised. Stated
// only because its assumption is stated.
property p_dirty_eventually_clears;
@(posedge clk) disable iff (!rst_n)
dirty_q[0] |-> s_eventually (!dirty_q[0]);
endproperty
a_liveness : assert property (p_dirty_eventually_clears);P5 and P5b are the race properties, and P5b is the more general statement: dirty clears only on an exact match between the total sent and the total now held. P5 is the same claim narrowed to the same-cycle case, kept separately because it is the one a directed test targets.
P9 and P10 are the pair that kills the delta misreading. P10 says the difference is computed modularly; P9 says a duplicate produces no event at all. A design that added the total outright fails P9 on the first repeated update, before any overflow occurs.
P12 is the quietest and most damaging property in the chapter. A baseline advanced on a rejected update permanently loses every credit between that value and the next accepted one — and nothing downstream ever notices, because the arithmetic stays self-consistent from then on.
P14 is the cross-Link conservation claim and it is the strongest thing here: credit given cannot exceed credit freed, plus what was advertised at init. It catches every invention — duplicate application, delta misreading, phantom emission — in one property, and it can only be written in a two-port testbench.
P17 is the only liveness property in Module 16, and F1 states its assumption explicitly. Without fairness it is false, and asserting it without F1 would be asserting something this design cannot guarantee (Chapter 16.1 §15).
13. Verification and Fault Injection
The scoreboard runs its own receiver counter and its own modular-difference model, and never calls the DUT's subtraction, dirty logic, or pool selector.
// VERIFICATION-ONLY. Independent models for both ends.
// Deliberately different in structure from the RTL: the receiver model
// counts up in a loop and the transmitter model masks explicitly, so a
// shared arithmetic bug cannot hide in both.
class fc_ref;
int unsigned modulus; // 2**COUNTER_W
int unsigned rx_total[6]; // receiver cumulative
int unsigned tx_baseline[6]; // transmitter baseline
int unsigned tx_added[6]; // cumulative credit added
function void rx_free(int pool, int count);
for (int i = 0; i < count; i++) // counted, not added
rx_total[pool] = (rx_total[pool] + 1) % modulus;
endfunction
function int tx_apply(int pool, int unsigned total);
int unsigned delta;
delta = (total + modulus - tx_baseline[pool]) % modulus; // explicit
tx_baseline[pool] = total;
tx_added[pool] += delta;
return delta; // duplicate yields 0
endfunction
endclassReceiver side
- Free one unit. Verify the total advances by one and
dirtysets. - Several frees before any update. Verify one update carrying the accumulated total, not several.
- Frees on several pools. Verify per-pool isolation (P1).
- An update stalled while the same pool is freed. Verify the descriptor does not change (P6) and the pool stays dirty.
- A free on the same pool in the same cycle as the update handshake. Verify
dirtyremains set (P5) — the required test, and the one §9's wrong code fails. - A free on a different pool in the same cycle as a handshake. Verify the sent pool clears and the other sets.
- The counter wrapping — free past
2**COUNTER_W. Verify it wraps rather than saturating (P3). - An out-of-range
free_pool. Verify nothing touched and reported (P2). - An infinite pool with frees. Verify it is never scheduled (P7).
POOLS= 1, 3, 6, 8 — the parameter corners, including the$clog2(1)case.
Transmitter side
- First valid update. Baseline set, difference emitted.
- The same update twice. Verify the second yields zero and no event (P9) — required.
- The same update ten times. Verify availability does not grow.
- A later update after one is dropped. Verify the difference covers both (§4).
- A difference across the wrap — baseline 14, update 2, modulus 16 → 4. Required, and the test a signed implementation fails.
- An implausible update — larger than capacity, and separately larger than
outstanding. Verify refusal, baseline unchanged (P12), and reported. - An update before initialisation. Verify refusal.
ret_readylow across several updates. Verify the held event is stable (P15).- Emit and consume in the same cycle. Verify the new event is held.
- Available already at capacity when a return arrives (Chapter 16.1 §8's clamp).
End to end
- Send, free, update, send again. The full loop of §11, with the scoreboard checking credit added against credit freed (P14).
- An update coinciding with a consumption — Chapter 16.5 §7's same-cycle contract.
- An ACK with no update. Verify the FC baseline does not move (P16).
- Reset mid-loop, with a dirty pool and a pending descriptor.
Mutations
| # | Mutation | Caught by | Silicon symptom |
|---|---|---|---|
| 1 | free increments the wrong pool's total | P1 | one class over-credited, another starves |
| 2 | dirty never set on free | P4 | credits freed but never advertised; Link stalls until the 32 µs timer |
| 3 | dirty cleared despite a same-cycle free | P5 | credits invisible until the next free; idle-Link hang |
| 4 | offered descriptor mutates under stall | P6, A2 | dirty comparison meaningless; progress lost |
| 5 | pool index used before the bounds check | P2 at POOLS = 6 | neighbouring pool's total corrupted |
| 6 | cumulative total added as a delta | P9, P14 | availability explodes; immediate receiver overflow |
| 7 | duplicate update creates credit | P9 | availability grows every 32 µs with no traffic |
| 8 | difference computed in widened non-modular arithmetic | P10 at the wrap | huge or negative return exactly at rollover |
| 9 | baseline advances on a rejected update | P12 | permanent silent loss of a block of credit |
| 10 | returned credit added twice | P13, P14 | capacity invented; intermittent overflow |
| 11 | FC update alters replay state | P16c | replay buffer corrupted by unrelated traffic |
| 12 | ACK alters the FC baseline | P16 | credit tracking drifts with error rate |
| 13 | counter instantiated at the wrong width | P3 | wrap at the wrong point; periodic mis-accounting |
| 14 | a new free lost while an update is pending | P5b | slow, load-dependent credit leak |
| 15 | zero difference emitted as an event | P9 | phantom returns downstream of a duplicate |
14. Debugging
Credits decrease and never return, though the receiver is draining
Walk the loop of §11; each step eliminates the ones before.
- Is
free_validpulsing at the receiver? If not, the internal consumer is not signalling the free — and note this is not the packet arriving (§3). - Is the cumulative total advancing? If frees pulse and the total is flat, the counter is not being written — or the wrong pool is (mutation 1).
- Is
dirtyset? If the total advanced and dirty is clear, that is the §9 race — and it is the one that presents exactly like this. - Is the scheduler offering an update? If dirty is set and nothing is offered, check
pool_infinite— an infinite pool is deliberately never scheduled. - Is the update reaching the transmitter? Chapter 15.2 owns the decode.
- Is
fc_update_errorset? A rejected update means the difference failed a plausibility bound, and the baseline was deliberately not advanced (P12). - Is
ret_validfiring into the credit account? The last hop.
Step 3 is the one this chapter added, and it has a distinctive tell: the receiver's total and the transmitter's baseline differ by a fixed amount that never changes until unrelated traffic sets dirty again.
Credits jump upward massively near the counter wrap
The modular difference is not modular (§5, mutation 8).
Check for a widened intermediate or a signed comparison. At baseline 4094 and update 2 with a 12-bit counter, the correct difference is 4; a widened subtraction gives −4092, and interpreted unsigned that is an enormous return.
The tell is periodicity. It happens once per counter rollover — every 4096 freeings for a data pool — which makes it look time-based rather than arithmetic.
Credit returns work until the update output stalls
The dirty/descriptor ownership race (§8, §9).
If the offered descriptor mutates under stall, the return counter's comparison is against a value that no longer matches what was sent, and dirty clears when it should not.
Check that upd_total is stable while upd_valid && !upd_ready — one waveform.
A duplicate update visible on the analyzer increases availability twice
The transmitter is treating the cumulative total as a delta (§4, mutation 6).
This is diagnostic on its own. Compare the analyzer's two updates: if they carry the same value and availability rose twice, no modular difference is being computed.
And the corollary is the fastest check in the chapter: on a quiet Link with no traffic and nothing being freed, availability must not change. If it climbs every 32 µs, it is this.
Only one credit class stops returning
Pool selection or that pool's dirty state (mutations 1, 5).
Check whether the receiver's total for that pool is advancing. If it is and no update carries it, the scheduler is skipping it — pool_infinite set wrongly, or the priority select never reaching it. If the total is flat, the frees are being attributed to a different pool.
15. Common Misconceptions
- "The receiver returns one token DLLP per consumed TLP." It publishes a cumulative total; the transmitter differences it (§4).
- "The update value should simply be added to available." That is a total, not a delta. Adding it directly injects hundreds of credits (§4, §13).
- "A duplicate FC update adds credit again." The difference is zero (§4, P9).
- "A lost FC update loses that capacity." The next total includes it (§4).
- "An FC update acknowledges TLP reception." Different loop, different guarantee (§16).
- "Credit returns when the transaction completes." It returns when the buffer becomes reusable — for a Memory Read, potentially long before the Completion (§3).
- "A Memory Read's NPH returns only after the CplD arrives." No — the request buffer is free once the request moves into servicing (§3, Chapter 16.3 §5).
- "ACK and FC Update modify the same state." An ACK retires replay storage; an update returns credit (P16, Chapter 16.1 §3).
- "A replay restores credits." It retransmits a packet. Credit is returned only by the receiver freeing a buffer (Chapter 16.5 §9).
- "A counter wrap means something was reset." It means the counter reached its modulus. Nothing is lost — the difference is still correct (§5).
- "A receiver must emit an update immediately for every freed entry." Updates are scheduled periodically per non-infinite pool, and several frees ride one update (§1).
- "The dirty flag may be cleared whenever an update is sent." Only if no newer progress remains — §9, and it is the subtlest bug in Module 16.
- "All six pools share one cumulative counter." One per pool; a shared counter cannot express per-class progress (P1).
- "An infinite pool still needs updates." It is never scheduled, and §1's watchdog is disabled when all three classes advertised infinite.
16. Three Feedback Loops — Module Synthesis
Module 16 ends where Module 14 began, and the three mechanisms are finally separable.
| Reliability | Flow control | Transaction | |
|---|---|---|---|
| Carrier | ACK / NAK DLLP | InitFC / UpdateFC DLLP | Completion TLP |
| Question answered | was this Link transmission accepted correctly? | does the remote receiver have room for another? | did the operation resolve, and how? |
| Scope | one Link | one Link, one direction | end to end |
| Granularity | a sequence number | a cumulative credit total | one request |
| Frees | replay buffer (14.4) | remote receive buffer | requester context (12.1) |
| Failure | replay, then Link retrain | stall — safe, and possibly forever | Completion timeout |
| Cumulative? | yes — ACK is a frontier | yes — a running total | no — one per request |
17. Understanding Check
18. Module 16 Complete
The loop closes. A receiver frees a buffer, advances a cumulative total, and publishes it periodically. A transmitter differences it against a baseline and adds the result to live credit — exactly once, with duplicates contributing zero and losses repaired by the next update.
Three counters, never interchangeable: the receiver's cumulative total, the transmitter's baseline, and the transmitter's live availability. And the two subtlest bugs in the module are both confusions between them — a total added as a delta, and a dirty bit cleared while newer progress was still unadvertised.
Module 16 as a whole built one idea: backpressure that cannot travel as a wire travels as accounting. Six pools, because three transaction classes with different lifetimes must not compete for one buffer (16.1). Costs of 1 header + Roundup(bytes / 16) on the Posted (16.2), Non-Posted (16.3) and Completion (16.4) pools. A spend that happens on one declared irrevocable event, with a third state for anything in flight (16.5). And a return that is a published total rather than a delivered token.
Module 17 leaves the Link layers entirely for the Physical Layer — serialization, link training, equalization, and the electrical reality underneath every packet Modules 10 to 16 have been moving.
The idea to carry forward: when the answer must survive being lost, send a position, not a quantity.