Skip to content

PCIe · Module 16

Credit Updates — A Total, Delivered Twice, Counted Once

The receiver never sends a token per packet. It sends a running total that wraps, and the transmitter subtracts a baseline to recover what was freed — so a duplicate adds nothing and a lost update costs nothing.

Every chapter in Module 16 has spent credit. None has explained where it comes back from.

The naive answer would be a token: the receiver frees a buffer, sends back "here is one credit", and the transmitter adds one. It would be simple, and it would be fragile — a lost token is capacity gone forever, and a duplicated one is capacity invented.

PCIe does something better and stranger. The receiver reports a running total of everything it has ever freed, in a counter that wraps. The transmitter subtracts what it last saw and adds the difference.

When does receive storage actually become reusable, how is that progress carried, and how does a wrapping total become live credit exactly once — never twice on a duplicate, and never lost when an update goes missing?

1. The Verified Mechanism

2. Three Counters, and They Are Not Interchangeable

This is the chapter's organising distinction, and every bug in §14 is a confusion between two of these.

CounterLives atIsChanges when
Cumulative freed totalthe receivera protocol value, wrapsthe receiver frees a buffer
Last-accepted baselinethe transmittera local copy of the abovean update is accepted
Live available creditthe transmitterlocal send permissiona packet is sent, or a difference is added

3. When Is Storage Actually Free?

Not when the packet arrives, and not when the transaction completes.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
TLP arrives
  → occupies the receiver's buffer                  ← credit is "spent"
  → moved into internal processing / storage
  → THE BUFFER IS NOW REUSABLE                      ← the freeing event
  → cumulative freed total advances
  → an UpdateFC eventually carries the new total
  → the transmitter differences it into live credit

4. A Total, Not a Delta

The single most important sentence in the chapter, and the one bug it prevents is fatal.

The value carried is how much has been freed in total, since initialisation, modulo the counter width. It is not "how much was freed since the last update".

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
transmitter:   returned = new_total − last_total   (modulo)
               available += returned
               last_total = new_total

5. The Wrap, and What Makes a Difference Plausible

Start with a small counter so the arithmetic is visible.

A 4-bit cumulative counter, modulus 16:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
last_total = 14
new_total  =  2
 
modular difference = (2 − 14) mod 16 = 4

And that is correct, because the receiver's progress was 14 → 15 → 0 → 1 → 2 — four freeings. An ordinary signed subtraction gives −12, which is meaningless as a credit return and, if added to an unsigned counter, catastrophic.

Fixed-width unsigned subtraction is modular subtraction — the same fact Chapter 15.2 §6 and Chapter 14.5 §6 relied on. Widening the operands before subtracting destroys it, and §7's RTL is written to make that impossible.

6. A Trace

Internal teaching signals, not PCIe wire signals. One pool, capacity 8, a 4-bit cumulative counter for legibility.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
step             1     2     3     4     5     6     7     8
rx_free          1     1     0     1     0     0     1     0
rx_free_count    2     3     -     4     -     -     2     -
rx_total        14     1     1     5     5     5     7     7
dirty            1     1     1     1     1     0     1     1
 
update_valid     0     1     1     1     1     1     0     1
update_ready     0     0     0     1     0     1     0     0
update_total     -    14    14    14     5     5     -     7
 
tx_last          0     0     0     0    14    14    14     5
returned_delta   -     -     -    14     -     7     -     -
tx_available     8     8     8     8     8     8     8     8

Read step 1. The receiver frees 2 units; the cumulative total moves to 14. dirty is set — there is unadvertised progress.

Read step 2 — the wrap. It frees 3 more. 14 + 3 = 17, and modulo 16 that is 1. The total wrapped and nothing special happened: the counter simply advanced.

Read step 4. An update carrying 14 is taken. Note it carries the total as of when the descriptor was offered (step 2), not the current total of 5 — §8's immutability contract.

Read step 5. The transmitter's baseline becomes 14, and the difference against its previous baseline of 0 is 14.

Read step 6 — the second update. It carries 5. The difference is (5 − 14) mod 16 = 7, which is exactly the freeings between the two updates. dirty clears here because the total just sent matches the current total.

Read step 7 — the race that §9 is about. The receiver frees 2 more in the same region where an update just completed. dirty sets again, because there is new unadvertised progress. A design that cleared dirty unconditionally on the handshake would have lost this.

And note tx_available never moves in this trace, because nothing is being sent — the returns are shown arriving and the pool is already at capacity. The trace is about the counters, not the spending.

7. RTL — Receiver Cumulative Return Counter

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// SYNTHESIZABLE. Receiver-side cumulative freed-credit accounting.
// That the receiver reports progress as the RECEIVER EMPTIES ITS BUFFERS,
// via UpdateFC, is NORMATIVE (section 1). That the value is a CUMULATIVE
// TOTAL modulo a fixed field width is at the tier section 1 assigns it --
// hence COUNTER_W is a parameter. The dirty-bit scheduling and the error
// reports are ILLUSTRATIVE IMPLEMENTATION POLICY.
module fc_return_counter #(
  parameter int POOLS     = 6,
  // GUARDED INDEX WIDTH. $clog2(1) is zero and a zero-width port cannot
  // index anything -- the corner Chapter 15.2 was fixed for.
  parameter int POOL_W    = (POOLS <= 1) ? 1 : $clog2(POOLS),
  // The PROTOCOL counter width. Deliberately not hardcoded (section 1).
  parameter int COUNTER_W = 12
) (
  input  logic clk,
  input  logic rst_n,
 
  // ---- Initialisation, from InitFC (section 1) ---------------------------
  // Normalized: this module does not implement the InitFC1/InitFC2 state
  // machine, it takes the resulting starting total.
  input  logic                  init_valid,
  input  logic [POOL_W-1:0]     init_pool,
  input  logic [COUNTER_W-1:0]  init_total,
 
  // ---- The FREEING EVENT, from the receiver's internal machinery ---------
  // Asserted when buffer capacity becomes REUSABLE -- not when the packet
  // arrived, and not when the transaction completed (section 3).
  input  logic                  free_valid,
  input  logic [POOL_W-1:0]     free_pool,
  input  logic [COUNTER_W-1:0]  free_count,
 
  // ---- To the update scheduler -------------------------------------------
  output logic [COUNTER_W-1:0]  total    [POOLS],
  output logic [POOLS-1:0]      dirty,
  // Asserted by the scheduler when an update for this pool has been taken.
  input  logic                  sent_valid,
  input  logic [POOL_W-1:0]     sent_pool,
  input  logic [COUNTER_W-1:0]  sent_total,
 
  output logic                  free_pool_error
);
 
  generate
    if (POOLS < 1)  $error("POOLS must be at least 1");
    if (POOL_W < 1) $error("POOL_W must be at least 1");
    if ((POOLS > 1) && ((1 << POOL_W) < POOLS))
      $error("POOL_W too narrow to index POOLS");
    if (COUNTER_W < 2) $error("COUNTER_W must be at least 2");
  endgenerate
 
  logic [COUNTER_W-1:0] total_q [POOLS];
  logic [POOLS-1:0]     dirty_q;
  logic                 err_q;
 
  assign total           = total_q;
  assign dirty           = dirty_q;
  assign free_pool_error = err_q;
 
  // RANGE SAFETY. A POOL_W-wide index can express values >= POOLS whenever
  // POOLS is not a power of two. Checked one bit wider, BEFORE any access.
  localparam int CHK = POOL_W + 1;
  wire free_legal = (CHK'(free_pool) < CHK'(POOLS));
  wire init_legal = (CHK'(init_pool) < CHK'(POOLS));
  wire sent_legal = (CHK'(sent_pool) < CHK'(POOLS));
 
  wire do_free = free_valid && free_legal;
  wire do_sent = sent_valid && sent_legal;
 
  // The total AFTER this cycle's free, for the pool being freed. Computed
  // once so the dirty logic and the counter update cannot disagree.
  wire [COUNTER_W-1:0] free_next =
        do_free ? (total_q[free_pool] + free_count) : '0;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int i = 0; i < POOLS; i++) total_q[i] <= '0;
      dirty_q <= '0;
      err_q   <= 1'b0;
    end else begin
      if (init_valid && init_legal) begin
        total_q[init_pool] <= init_total;
        dirty_q[init_pool] <= 1'b0;
      end
 
      // MODULAR BY CONSTRUCTION. Fixed-width unsigned addition wraps at
      // 2**COUNTER_W, which IS the protocol modulus. Never widen this.
      if (do_free) begin
        total_q[free_pool] <= free_next;
        dirty_q[free_pool] <= 1'b1;
      end
 
      // ================================================================
      // THE DIRTY-CLEAR RACE (section 9).
      //
      // Clearing dirty[sent_pool] unconditionally on the handshake LOSES
      // any freeing that happened in the SAME cycle: the update carried
      // the OLD total, the counter has already advanced past it, and the
      // pool is now marked clean with unadvertised progress in it.
      //
      // The condition is not "was this pool sent" but "does the total the
      // update carried still match the total we now hold". Written as an
      // explicit comparison so the same-cycle case falls out rather than
      // needing to be special-cased.
      // ================================================================
      if (do_sent) begin
        automatic logic [COUNTER_W-1:0] now =
          (do_free && (free_pool == sent_pool)) ? free_next
                                                : total_q[sent_pool];
        if (sent_total == now) dirty_q[sent_pool] <= 1'b0;
        // else: the pool STAYS dirty -- there is newer progress to advertise
      end
 
      if (free_valid && !free_legal) err_q <= 1'b1;
    end
  end
 
endmodule

Classification: synthesizable.

Architecture. One cumulative counter per pool, one dirty bit per pool, and a comparison rather than an unconditional clear.

The modular arithmetic is structural, not computed. total_q[p] + free_count at COUNTER_W bits wraps at 2**COUNTER_W because that is what fixed-width addition does. There is no modulo operator anywhere, and there must not be — a % invites a widened intermediate, and a widened intermediate is no longer modular.

Cycle behaviour.

EventResult
freetotal advances modulo, pool marked dirty
sent, total still matchesdirty cleared
sent and free on the same pool, same cycledirty stays set — §9
sent on a pool with newer progressdirty stays set
out-of-range free_poolnothing touched, reported
inittotal set, dirty cleared

Failure — six. $clog2(POOLS) in a port width fails to elaborate at POOLS = 1. Indexing before the bounds check corrupts a neighbouring pool at any non-power-of-two POOLS. Clearing dirty unconditionally on the handshake loses a same-cycle freeing (§9). A widened intermediate in the addition destroys the wrap. Advancing the wrong pool's total attributes capacity to the wrong class. And treating free_valid as "packet arrived" returns credit for buffers that are still occupied (§3).

Deliberately simplified: one free and one send per cycle; no InitFC state machine; capacity not modelled here — the plausibility bound lives at the transmitter (§10).

8. RTL — Update Scheduler

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// SYNTHESIZABLE. Select one dirty pool and offer a normalized FC update
// descriptor.
// That updates are scheduled periodically per non-infinite pool is
// NORMATIVE (section 1: every 32 microseconds, or 120 with Extended
// Synch). The SELECTION POLICY and the immutability contract below are
// ILLUSTRATIVE IMPLEMENTATION POLICY -- section 1's sources state the
// timing, not the arbitration.
module fc_update_scheduler #(
  parameter int POOLS     = 6,
  parameter int POOL_W    = (POOLS <= 1) ? 1 : $clog2(POOLS),
  parameter int COUNTER_W = 12
) (
  input  logic clk,
  input  logic rst_n,
 
  input  logic [COUNTER_W-1:0]  total [POOLS],
  input  logic [POOLS-1:0]      dirty,
  // Pools advertised as infinite are never scheduled (section 1).
  input  logic [POOLS-1:0]      pool_infinite,
 
  // ---- Normalized update descriptor out ----------------------------------
  // Chapter 15.1 owns the DLLP encoding; this is the boundary above it.
  output logic                  upd_valid,
  input  logic                  upd_ready,
  output logic [POOL_W-1:0]     upd_pool,
  output logic [COUNTER_W-1:0]  upd_total,
 
  // ---- Back to the return counter ----------------------------------------
  output logic                  sent_valid,
  output logic [POOL_W-1:0]     sent_pool,
  output logic [COUNTER_W-1:0]  sent_total
);
 
  logic                 v_q;
  logic [POOL_W-1:0]    pool_q;
  logic [COUNTER_W-1:0] total_q;
 
  assign upd_valid = v_q;
  assign upd_pool  = pool_q;
  assign upd_total = total_q;
 
  wire fire = v_q && upd_ready;
 
  // The descriptor is reported as SENT on the handshake, carrying the
  // total it actually contained -- which is what lets the return counter
  // decide whether the pool is still dirty (section 7).
  assign sent_valid = fire;
  assign sent_pool  = pool_q;
  assign sent_total = total_q;
 
  wire [POOLS-1:0] eligible = dirty & ~pool_infinite;
 
  // Lowest-index dirty pool. A FIXED PRIORITY, stated as such: section 1's
  // sources give the update TIMING, not the arbitration, so nothing here
  // claims round-robin is mandated. A production scheduler would rotate;
  // the liveness argument in section 13 assumes fairness explicitly.
  logic            any_eligible;
  logic [POOL_W-1:0] sel;
  always_comb begin
    any_eligible = 1'b0;
    sel          = '0;
    for (int p = POOLS-1; p >= 0; p--)
      if (eligible[p]) begin
        any_eligible = 1'b1;
        sel          = POOL_W'(p);
      end
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      v_q <= 1'b0; pool_q <= '0; total_q <= '0;
    end else begin
      // ================================================================
      // IMMUTABILITY CONTRACT: once offered, a descriptor does not change
      // until it is taken.
      //
      // The alternative -- refreshing the offered total as new capacity is
      // freed, the way Chapter 15.3's ACK coalescer refreshes its frontier
      // -- is tempting and is NOT done here. The reason is that the return
      // counter's dirty logic is written against "the total this update
      // carried", and a descriptor that mutates under it would make that
      // comparison meaningless.
      //
      // Nothing is lost: new frees keep the pool DIRTY, so they are
      // advertised by the next update (section 7).
      // ================================================================
      if (!v_q && any_eligible) begin
        v_q     <= 1'b1;
        pool_q  <= sel;
        total_q <= total[sel];
      end else if (fire) begin
        v_q <= 1'b0;
      end
    end
  end
 
endmodule

Classification: synthesizable.

Architecture. A priority select and a one-entry output register with an immutable descriptor.

The immutability decision is the interesting one, and it is the opposite of Chapter 15.3's. There, an offered ACK frontier is refreshed to newer progress while it waits, because the frontier is cumulative and naming more is strictly better. Here the total is also cumulative and refreshing would also be safe on the wire — but the receiver's dirty logic compares the sent total against the current total, and a descriptor that mutated after being sampled would break that comparison.

So the two blocks make opposite choices for the same underlying reason: each picks the option that keeps its partner's bookkeeping checkable. Nothing is lost either way — new frees keep the pool dirty and ride the next update.

Failure — four. Mutating the offered descriptor breaks the return counter's dirty comparison. Scheduling an infinite pool wastes update bandwidth on a pool that will never be differenced. Clearing the offering before the handshake loses an update. And asserting sent_valid on offer rather than on fire tells the return counter a pool was advertised when it was not.

Deliberately simplified: fixed priority rather than rotation — stated, not hidden; no periodic timer (§1 gives the intervals, and modelling a 32 µs timer adds nothing to the accounting lesson); one update in flight.

9. The Dirty-Clear Race

The subtlest bug in Module 16, and it is worth its own section because the wrong code looks obviously right.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// WRONG. Two independent updates that are correct in isolation.
if (update_fire)
  dirty_q[sel] <= 1'b0;
 
if (free_fire)
  total_q[free_pool] <= total_q[free_pool] + free_count;

10. RTL — Transmitter Return Integration

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// SYNTHESIZABLE. Convert a received cumulative FC total into newly
// returned credit, exactly once.
// That the value is a cumulative total and the difference is modular is
// the mechanism of section 4, at the tier section 1 assigns it. The
// plausibility bounds and the rejection policy are ILLUSTRATIVE LOCAL
// POLICY (section 5).
module fc_return_integrate #(
  parameter int POOLS     = 6,
  parameter int POOL_W    = (POOLS <= 1) ? 1 : $clog2(POOLS),
  parameter int COUNTER_W = 12,
  parameter int CRED_W    = 12
) (
  input  logic clk,
  input  logic rst_n,
 
  // ---- Initialised baseline and capacity, per pool -----------------------
  input  logic                  init_valid,
  input  logic [POOL_W-1:0]     init_pool,
  input  logic [COUNTER_W-1:0]  init_total,
  input  logic [CRED_W-1:0]     init_capacity,
 
  // ---- Decoded update, from Chapter 15.2 ----------------------------------
  input  logic                  upd_valid,
  output logic                  upd_ready,
  input  logic [POOL_W-1:0]     upd_pool,
  input  logic [COUNTER_W-1:0]  upd_total,
 
  // ---- How much is currently outstanding, from Chapter 16.5 --------------
  // Sent but not yet returned. The TIGHTER plausibility bound (section 5).
  input  logic [CRED_W-1:0]     outstanding [POOLS],
 
  // ---- Normalized return event, to Chapter 16.5's accounts ---------------
  output logic                  ret_valid,
  input  logic                  ret_ready,
  output logic [POOL_W-1:0]     ret_pool,
  output logic [CRED_W-1:0]     ret_count,
 
  output logic                  fc_update_error
);
 
  logic [COUNTER_W-1:0] last_q  [POOLS];
  logic [CRED_W-1:0]    cap_q   [POOLS];
  logic [POOLS-1:0]     init_q;
  logic                 err_q;
 
  localparam int CHK = POOL_W + 1;
  wire upd_legal  = (CHK'(upd_pool)  < CHK'(POOLS));
  wire init_legal = (CHK'(init_pool) < CHK'(POOLS));
 
  logic [COUNTER_W-1:0] last_sel;
  logic [CRED_W-1:0]    cap_sel, out_sel;
  logic                 init_sel;
  always_comb begin
    last_sel = '0; cap_sel = '0; out_sel = '0; init_sel = 1'b0;
    if (upd_legal) begin
      last_sel = last_q[upd_pool];
      cap_sel  = cap_q[upd_pool];
      out_sel  = outstanding[upd_pool];
      init_sel = init_q[upd_pool];
    end
  end
 
  // ================================================================
  // THE MODULAR DIFFERENCE.
  //
  // Fixed-width unsigned subtraction IS modular subtraction. There is no
  // modulo operator here and there must not be: a `%` invites a widened
  // intermediate, and a widened intermediate is no longer modular
  // (section 5, and Chapter 14.5 section 6).
  //
  // A DUPLICATE update gives upd_total == last_sel, so returned == 0 and
  // nothing is added. That is the whole de-duplication mechanism.
  // ================================================================
  wire [COUNTER_W-1:0] returned = upd_total - last_sel;
 
  // PLAUSIBILITY, two bounds, both local (section 5).
  //   - a receiver cannot free more than its buffer holds
  //   - a transmitter cannot be owed more than it has outstanding
  // Compared at COUNTER_W so a large modular difference cannot be
  // truncated into a small plausible one.
  wire [COUNTER_W-1:0] cap_x = COUNTER_W'(cap_sel);
  wire [COUNTER_W-1:0] out_x = COUNTER_W'(out_sel);
  wire plausible = init_sel && (returned <= cap_x) && (returned <= out_x);
 
  // ---- Output holding stage ----------------------------------------------
  logic              ret_v_q;
  logic [POOL_W-1:0] ret_p_q;
  logic [CRED_W-1:0] ret_c_q;
 
  wire ret_fire = ret_v_q && ret_ready;
  assign upd_ready = !ret_v_q || ret_ready;
  assign ret_valid = ret_v_q;
  assign ret_pool  = ret_p_q;
  assign ret_count = ret_c_q;
  assign fc_update_error = err_q;
 
  wire accept = upd_valid && upd_ready && upd_legal && plausible;
  // A zero difference is a legitimate no-op -- a duplicate update. It is
  // ACCEPTED (the baseline is already correct) and emits NO event.
  wire emit   = accept && (returned != '0);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int i = 0; i < POOLS; i++) begin
        last_q[i] <= '0; cap_q[i] <= '0;
      end
      init_q <= '0; ret_v_q <= 1'b0; ret_p_q <= '0; ret_c_q <= '0;
      err_q  <= 1'b0;
    end else begin
      if (init_valid && init_legal && !init_q[init_pool]) begin
        last_q[init_pool] <= init_total;
        cap_q[init_pool]  <= init_capacity;
        init_q[init_pool] <= 1'b1;
      end
 
      // ============================================================
      // THE BASELINE ADVANCES ONLY ON AN ACCEPTED UPDATE.
      //
      // Advancing it on a REJECTED one would measure the next difference
      // from a total the sender never sent -- silently losing every
      // credit between the rejected value and the next one, permanently.
      // ============================================================
      if (accept) last_q[upd_pool] <= upd_total;
 
      // Emit has priority over drain, so a same-cycle accept-and-consume
      // holds the new event rather than dropping it.
      if (emit) begin
        ret_v_q <= 1'b1;
        ret_p_q <= upd_pool;
        ret_c_q <= CRED_W'(returned);
      end else if (ret_fire) begin
        ret_v_q <= 1'b0;
      end
 
      if (upd_valid && !upd_legal)                        err_q <= 1'b1;
      if (upd_valid &&  upd_legal && !plausible)          err_q <= 1'b1;
    end
  end
 
endmodule

Classification: synthesizable.

Architecture. A baseline per pool, a modular subtraction, two plausibility bounds, and an output holding stage.

The outstanding bound is new here and it is tighter than the capacity bound Chapter 15.2 used. A transmitter cannot be owed capacity it never spent — so a return larger than what is currently in flight is impossible regardless of how big the pool is. It catches a corrupted update that happens to fall under the capacity ceiling.

Cycle behaviour.

SituationResult
first valid updatebaseline set, difference emitted
duplicate — same totaldifference 0, accepted, no event
later update after a lost onedifference covers both
difference across the wrapcorrect, by construction
implausible differencerefused, baseline unchanged, reported
out-of-range poolnothing touched, reported
update before initrefused, reported
emit and consume same cycleemit wins — event held

Failure — six. Adding upd_total directly treats a total as a delta and injects hundreds of credits (§4). Widening before subtracting destroys the wrap. Advancing the baseline on a rejected update loses every credit between that value and the next, permanently. Dropping a decoded return event under backpressure throws away capacity that was correctly measured — and unlike a lost update, that is not recoverable. Emitting a zero difference as an event feeds phantom returns downstream. And indexing before the bounds check corrupts a neighbouring pool.

Deliberately simplified: one update per cycle; outstanding supplied by Chapter 16.5; no InitFC sequencing.

11. The Whole Loop

A transmitter with available credit sends a TLP and decrements its local count. The packet occupies the remote receiver's buffer. The receiver's internal consumer later drains that buffer, which is the freeing event. The receiver's cumulative freed total advances modulo the counter width and the pool is marked dirty. The update scheduler emits a flow control update carrying that cumulative total. The transmitter subtracts its last accepted baseline from the received total to obtain the newly returned amount, advances its baseline, and adds the returned amount to its live available credit.Send, occupy, free, publish, difference, restoreTransmitterRX bufferRX consumerCumulativetotalUpdateschedulercredit availableTLP - availabledecrementsheld until drainedbuffer reusable -the free eventtotal advances,modulopool marked dirtyUpdateFC carries theTOTALreturned = total -baselinebaseline advances,credit restored
Figure 1 — one credit's full round trip. The transmitter sends and decrements; the packet occupies the receiver's buffer; the receiver's own consumer eventually drains it, which is the freeing event; the cumulative total advances and the pool is marked dirty; the scheduler emits an update carrying the total; the transmitter differences it against its baseline and adds the result to live credit. Note that only two of the six arrows carry protocol values — the rest are local state changes on one side or the other.

Three things to read out of the figure.

Only one arrow crosses the Link carrying a protocol value — the UpdateFC. Everything else is local on one side or the other. The entire mechanism rests on one packet type and one number.

The freeing event is inside the receiver, between the buffer and the consumer, and PCIe does not specify where exactly (§3). The protocol constrains the accounting, not the pipeline.

And the transmitter's last two steps are a subtraction and an addition — the difference against the baseline, then the addition to live credit. Those are three different counters (§2), and collapsing any two is a distinct bug.

12. Assertions

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// SVA over fc_return_counter, fc_update_scheduler and
// fc_return_integrate. These assert the LOCAL accounting contract and the
// declared scheduling policy. Liveness appears only with an explicit
// fairness assumption (P17).
 
// ---- ENVIRONMENT ------------------------------------------------------
// A1: free_valid means buffer capacity became REUSABLE (section 3) --
// not that a packet arrived, and not that a transaction completed.
assume property (@(posedge clk) disable iff (!rst_n)
  free_valid |-> (free_count != '0));
// A2: the update descriptor is stable while offered (section 8).
assume property (@(posedge clk) disable iff (!rst_n)
  (upd_valid && !upd_ready) |=> ($stable(upd_pool) && $stable(upd_total)));
 
// ---- RECEIVER SIDE ----------------------------------------------------
 
// P1: a free advances ONLY the selected pool.
generate for (genvar p = 0; p < POOLS; p++) begin : g_iso
  property p_only_selected_pool;
    @(posedge clk) disable iff (!rst_n)
    (do_free && (free_pool != POOL_W'(p))) |=> $stable(total_q[p]);
  endproperty
  a_pool_iso : assert property (p_only_selected_pool);
end endgenerate
 
// P2: RANGE SAFETY -- an out-of-range pool index touches nothing.
property p_no_oob_access;
  @(posedge clk) disable iff (!rst_n)
  (free_valid && !free_legal) |=> (free_pool_error && $stable(total_q));
endproperty
a_oob : assert property (p_no_oob_access);
 
// P3: the cumulative total wraps EXACTLY at the declared modulus. An
// independent restatement, so a widened intermediate fails.
property p_wraps_at_modulus;
  @(posedge clk) disable iff (!rst_n)
  do_free |=> (total_q[$past(free_pool)]
               == COUNTER_W'(($past(total_q[$past(free_pool)])
                            + $past(free_count))));
endproperty
a_wrap : assert property (p_wraps_at_modulus);
 
// P4: dirty is SET whenever unadvertised progress exists.
property p_dirty_when_unadvertised;
  @(posedge clk) disable iff (!rst_n)
  do_free |=> dirty_q[$past(free_pool)];
endproperty
a_dirty_set : assert property (p_dirty_when_unadvertised);
 
// P5: THE RACE PROPERTY (section 9). An update handshake clears dirty ONLY
// if no newer progress remains. Same-cycle free on the same pool must
// leave the pool dirty.
property p_dirty_survives_same_cycle_free;
  @(posedge clk) disable iff (!rst_n)
  (do_sent && do_free && (free_pool == sent_pool))
    |=> dirty_q[$past(sent_pool)];
endproperty
a_race : assert property (p_dirty_survives_same_cycle_free);
 
// P5b: and the general form -- dirty clears only on an exact match.
property p_dirty_clears_only_on_match;
  @(posedge clk) disable iff (!rst_n)
  (do_sent && (sent_total != total_q[sent_pool]))
    |=> dirty_q[$past(sent_pool)];
endproperty
a_dirty_match : assert property (p_dirty_clears_only_on_match);
 
// ---- SCHEDULER --------------------------------------------------------
 
// P6: an offered descriptor is IMMUTABLE until taken (section 8).
property p_descriptor_stable;
  @(posedge clk) disable iff (!rst_n)
  (upd_valid && !upd_ready) |=> (upd_valid && $stable(upd_pool)
                                           && $stable(upd_total));
endproperty
a_stable : assert property (p_descriptor_stable);
 
// P7: an infinite pool is never scheduled (section 1).
property p_infinite_not_scheduled;
  @(posedge clk) disable iff (!rst_n)
  upd_valid |-> !pool_infinite[upd_pool];
endproperty
a_infinite : assert property (p_infinite_not_scheduled);
 
// P8: sent_valid asserts on the HANDSHAKE, not on the offer.
property p_sent_on_fire;
  @(posedge clk) disable iff (!rst_n)
  sent_valid |-> (upd_valid && upd_ready);
endproperty
a_sent : assert property (p_sent_on_fire);
 
// ---- TRANSMITTER SIDE -------------------------------------------------
 
// P9: A DUPLICATE UPDATE DERIVES ZERO NEW CREDIT. The property a design
// that adds the total outright fails on the first repeat.
property p_duplicate_returns_zero;
  @(posedge clk) disable iff (!rst_n)
  (upd_valid && upd_legal && (upd_total == last_sel)) |-> !emit;
endproperty
a_duplicate : assert property (p_duplicate_returns_zero);
 
// P10: the returned amount equals an INDEPENDENT modular reference.
// Restated, so a widened or signed implementation fails rather than
// agreeing with itself.
property p_delta_matches_reference;
  @(posedge clk) disable iff (!rst_n)
  (upd_valid && upd_legal)
    |-> (returned == COUNTER_W'(upd_total - last_sel));
endproperty
a_delta : assert property (p_delta_matches_reference);
 
// P11: AN ACCEPTED UPDATE ADVANCES THE BASELINE EXACTLY ONCE.
property p_baseline_once;
  @(posedge clk) disable iff (!rst_n)
  accept |=> (last_q[$past(upd_pool)] == $past(upd_total));
endproperty
a_baseline : assert property (p_baseline_once);
 
// P12: A REJECTED UPDATE CANNOT CORRUPT THE BASELINE. Advancing it on a
// rejected value loses every credit up to the next one, permanently.
property p_rejected_leaves_baseline;
  @(posedge clk) disable iff (!rst_n)
  (upd_valid && upd_legal && !plausible) |=> $stable(last_q[$past(upd_pool)]);
endproperty
a_reject : assert property (p_rejected_leaves_baseline);
 
// P13: newly returned credit is added EXACTLY ONCE. Ghost counters are
// testbench state -- a VERIFICATION ACCOUNTING IDENTITY, not wire state.
property p_returned_once;
  @(posedge clk) disable iff (!rst_n)
  (ret_valid && ret_ready)
    |=> (g_credit_added == $past(g_credit_added) + $past(ret_count));
endproperty
a_once : assert property (p_returned_once);
 
// P14: NO CREDIT APPEARS WITHOUT A RECEIVER FREE OR AN INITIALISATION.
// Bound across the Link in a two-port testbench: the total credit the
// transmitter has been given never exceeds what the receiver has freed.
property p_no_spontaneous_credit;
  @(posedge clk) disable iff (!rst_n)
  (g_credit_added <= (g_rx_freed + g_initial_advertised));
endproperty
a_no_magic : assert property (p_no_spontaneous_credit);
 
// P15: a decoded return event is not lost under backpressure.
property p_event_not_lost;
  @(posedge clk) disable iff (!rst_n)
  (ret_valid && !ret_ready) |=> (ret_valid && $stable(ret_count)
                                           && $stable(ret_pool));
endproperty
a_no_loss : assert property (p_event_not_lost);
 
// ---- CROSS-MECHANISM NON-INTERFERENCE ---------------------------------
 
// P16: AN ACK CANNOT MOVE THE FC BASELINE. Three feedback loops, three
// mechanisms (section 16).
property p_ack_does_not_touch_fc;
  @(posedge clk) disable iff (!rst_n)
  (dut_retire_window.retire_valid && !upd_valid) |=> $stable(last_q);
endproperty
a_ack : assert property (p_ack_does_not_touch_fc);
 
// P16b: nor does a Completion arrival return FC credit.
property p_completion_returns_no_credit;
  @(posedge clk) disable iff (!rst_n)
  (dut_cpl_arrives && !upd_valid) |=> !ret_valid;
endproperty
a_cpl : assert property (p_completion_returns_no_credit);
 
// P16c: an FC update never alters replay state.
property p_fc_does_not_touch_replay;
  @(posedge clk) disable iff (!rst_n)
  (upd_valid && upd_ready) |=> $stable(dut_replay_buffer.occupancy);
endproperty
a_replay : assert property (p_fc_does_not_touch_replay);
 
// ---- LIVENESS, WITH ITS ASSUMPTION STATED -----------------------------
 
// F1: FAIRNESS ASSUMPTION. The scheduler eventually selects any dirty
// pool, and the transport eventually accepts. This is an ASSUMPTION about
// the environment and the arbiter, not a property of this design.
assume property (@(posedge clk) disable iff (!rst_n)
  s_eventually upd_ready);
 
// P17: given F1, unadvertised progress is eventually advertised. Stated
// only because its assumption is stated.
property p_dirty_eventually_clears;
  @(posedge clk) disable iff (!rst_n)
  dirty_q[0] |-> s_eventually (!dirty_q[0]);
endproperty
a_liveness : assert property (p_dirty_eventually_clears);

P5 and P5b are the race properties, and P5b is the more general statement: dirty clears only on an exact match between the total sent and the total now held. P5 is the same claim narrowed to the same-cycle case, kept separately because it is the one a directed test targets.

P9 and P10 are the pair that kills the delta misreading. P10 says the difference is computed modularly; P9 says a duplicate produces no event at all. A design that added the total outright fails P9 on the first repeated update, before any overflow occurs.

P12 is the quietest and most damaging property in the chapter. A baseline advanced on a rejected update permanently loses every credit between that value and the next accepted one — and nothing downstream ever notices, because the arithmetic stays self-consistent from then on.

P14 is the cross-Link conservation claim and it is the strongest thing here: credit given cannot exceed credit freed, plus what was advertised at init. It catches every invention — duplicate application, delta misreading, phantom emission — in one property, and it can only be written in a two-port testbench.

P17 is the only liveness property in Module 16, and F1 states its assumption explicitly. Without fairness it is false, and asserting it without F1 would be asserting something this design cannot guarantee (Chapter 16.1 §15).

13. Verification and Fault Injection

The scoreboard runs its own receiver counter and its own modular-difference model, and never calls the DUT's subtraction, dirty logic, or pool selector.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// VERIFICATION-ONLY. Independent models for both ends.
// Deliberately different in structure from the RTL: the receiver model
// counts up in a loop and the transmitter model masks explicitly, so a
// shared arithmetic bug cannot hide in both.
class fc_ref;
  int unsigned modulus;                        // 2**COUNTER_W
  int unsigned rx_total[6];                    // receiver cumulative
  int unsigned tx_baseline[6];                 // transmitter baseline
  int unsigned tx_added[6];                    // cumulative credit added
 
  function void rx_free(int pool, int count);
    for (int i = 0; i < count; i++)            // counted, not added
      rx_total[pool] = (rx_total[pool] + 1) % modulus;
  endfunction
 
  function int tx_apply(int pool, int unsigned total);
    int unsigned delta;
    delta = (total + modulus - tx_baseline[pool]) % modulus;  // explicit
    tx_baseline[pool] = total;
    tx_added[pool] += delta;
    return delta;                              // duplicate yields 0
  endfunction
endclass

Receiver side

  • Free one unit. Verify the total advances by one and dirty sets.
  • Several frees before any update. Verify one update carrying the accumulated total, not several.
  • Frees on several pools. Verify per-pool isolation (P1).
  • An update stalled while the same pool is freed. Verify the descriptor does not change (P6) and the pool stays dirty.
  • A free on the same pool in the same cycle as the update handshake. Verify dirty remains set (P5) — the required test, and the one §9's wrong code fails.
  • A free on a different pool in the same cycle as a handshake. Verify the sent pool clears and the other sets.
  • The counter wrapping — free past 2**COUNTER_W. Verify it wraps rather than saturating (P3).
  • An out-of-range free_pool. Verify nothing touched and reported (P2).
  • An infinite pool with frees. Verify it is never scheduled (P7).
  • POOLS = 1, 3, 6, 8 — the parameter corners, including the $clog2(1) case.

Transmitter side

  • First valid update. Baseline set, difference emitted.
  • The same update twice. Verify the second yields zero and no event (P9) — required.
  • The same update ten times. Verify availability does not grow.
  • A later update after one is dropped. Verify the difference covers both (§4).
  • A difference across the wrap — baseline 14, update 2, modulus 16 → 4. Required, and the test a signed implementation fails.
  • An implausible update — larger than capacity, and separately larger than outstanding. Verify refusal, baseline unchanged (P12), and reported.
  • An update before initialisation. Verify refusal.
  • ret_ready low across several updates. Verify the held event is stable (P15).
  • Emit and consume in the same cycle. Verify the new event is held.
  • Available already at capacity when a return arrives (Chapter 16.1 §8's clamp).

End to end

  • Send, free, update, send again. The full loop of §11, with the scoreboard checking credit added against credit freed (P14).
  • An update coinciding with a consumptionChapter 16.5 §7's same-cycle contract.
  • An ACK with no update. Verify the FC baseline does not move (P16).
  • Reset mid-loop, with a dirty pool and a pending descriptor.

Mutations

#MutationCaught bySilicon symptom
1free increments the wrong pool's totalP1one class over-credited, another starves
2dirty never set on freeP4credits freed but never advertised; Link stalls until the 32 µs timer
3dirty cleared despite a same-cycle freeP5credits invisible until the next free; idle-Link hang
4offered descriptor mutates under stallP6, A2dirty comparison meaningless; progress lost
5pool index used before the bounds checkP2 at POOLS = 6neighbouring pool's total corrupted
6cumulative total added as a deltaP9, P14availability explodes; immediate receiver overflow
7duplicate update creates creditP9availability grows every 32 µs with no traffic
8difference computed in widened non-modular arithmeticP10 at the wraphuge or negative return exactly at rollover
9baseline advances on a rejected updateP12permanent silent loss of a block of credit
10returned credit added twiceP13, P14capacity invented; intermittent overflow
11FC update alters replay stateP16creplay buffer corrupted by unrelated traffic
12ACK alters the FC baselineP16credit tracking drifts with error rate
13counter instantiated at the wrong widthP3wrap at the wrong point; periodic mis-accounting
14a new free lost while an update is pendingP5bslow, load-dependent credit leak
15zero difference emitted as an eventP9phantom returns downstream of a duplicate

14. Debugging

Credits decrease and never return, though the receiver is draining

Walk the loop of §11; each step eliminates the ones before.

  1. Is free_valid pulsing at the receiver? If not, the internal consumer is not signalling the free — and note this is not the packet arriving (§3).
  2. Is the cumulative total advancing? If frees pulse and the total is flat, the counter is not being written — or the wrong pool is (mutation 1).
  3. Is dirty set? If the total advanced and dirty is clear, that is the §9 race — and it is the one that presents exactly like this.
  4. Is the scheduler offering an update? If dirty is set and nothing is offered, check pool_infinite — an infinite pool is deliberately never scheduled.
  5. Is the update reaching the transmitter? Chapter 15.2 owns the decode.
  6. Is fc_update_error set? A rejected update means the difference failed a plausibility bound, and the baseline was deliberately not advanced (P12).
  7. Is ret_valid firing into the credit account? The last hop.

Step 3 is the one this chapter added, and it has a distinctive tell: the receiver's total and the transmitter's baseline differ by a fixed amount that never changes until unrelated traffic sets dirty again.

Credits jump upward massively near the counter wrap

The modular difference is not modular (§5, mutation 8).

Check for a widened intermediate or a signed comparison. At baseline 4094 and update 2 with a 12-bit counter, the correct difference is 4; a widened subtraction gives −4092, and interpreted unsigned that is an enormous return.

The tell is periodicity. It happens once per counter rollover — every 4096 freeings for a data pool — which makes it look time-based rather than arithmetic.

Credit returns work until the update output stalls

The dirty/descriptor ownership race (§8, §9).

If the offered descriptor mutates under stall, the return counter's comparison is against a value that no longer matches what was sent, and dirty clears when it should not.

Check that upd_total is stable while upd_valid && !upd_ready — one waveform.

A duplicate update visible on the analyzer increases availability twice

The transmitter is treating the cumulative total as a delta (§4, mutation 6).

This is diagnostic on its own. Compare the analyzer's two updates: if they carry the same value and availability rose twice, no modular difference is being computed.

And the corollary is the fastest check in the chapter: on a quiet Link with no traffic and nothing being freed, availability must not change. If it climbs every 32 µs, it is this.

Only one credit class stops returning

Pool selection or that pool's dirty state (mutations 1, 5).

Check whether the receiver's total for that pool is advancing. If it is and no update carries it, the scheduler is skipping it — pool_infinite set wrongly, or the priority select never reaching it. If the total is flat, the frees are being attributed to a different pool.

15. Common Misconceptions

  • "The receiver returns one token DLLP per consumed TLP." It publishes a cumulative total; the transmitter differences it (§4).
  • "The update value should simply be added to available." That is a total, not a delta. Adding it directly injects hundreds of credits (§4, §13).
  • "A duplicate FC update adds credit again." The difference is zero (§4, P9).
  • "A lost FC update loses that capacity." The next total includes it (§4).
  • "An FC update acknowledges TLP reception." Different loop, different guarantee (§16).
  • "Credit returns when the transaction completes." It returns when the buffer becomes reusable — for a Memory Read, potentially long before the Completion (§3).
  • "A Memory Read's NPH returns only after the CplD arrives." No — the request buffer is free once the request moves into servicing (§3, Chapter 16.3 §5).
  • "ACK and FC Update modify the same state." An ACK retires replay storage; an update returns credit (P16, Chapter 16.1 §3).
  • "A replay restores credits." It retransmits a packet. Credit is returned only by the receiver freeing a buffer (Chapter 16.5 §9).
  • "A counter wrap means something was reset." It means the counter reached its modulus. Nothing is lost — the difference is still correct (§5).
  • "A receiver must emit an update immediately for every freed entry." Updates are scheduled periodically per non-infinite pool, and several frees ride one update (§1).
  • "The dirty flag may be cleared whenever an update is sent." Only if no newer progress remains — §9, and it is the subtlest bug in Module 16.
  • "All six pools share one cumulative counter." One per pool; a shared counter cannot express per-class progress (P1).
  • "An infinite pool still needs updates." It is never scheduled, and §1's watchdog is disabled when all three classes advertised infinite.

16. Three Feedback Loops — Module Synthesis

Module 16 ends where Module 14 began, and the three mechanisms are finally separable.

ReliabilityFlow controlTransaction
CarrierACK / NAK DLLPInitFC / UpdateFC DLLPCompletion TLP
Question answeredwas this Link transmission accepted correctly?does the remote receiver have room for another?did the operation resolve, and how?
Scopeone Linkone Link, one directionend to end
Granularitya sequence numbera cumulative credit totalone request
Freesreplay buffer (14.4)remote receive bufferrequester context (12.1)
Failurereplay, then Link retrainstall — safe, and possibly foreverCompletion timeout
Cumulative?yes — ACK is a frontieryes — a running totalno — one per request

17. Understanding Check

18. Module 16 Complete

The loop closes. A receiver frees a buffer, advances a cumulative total, and publishes it periodically. A transmitter differences it against a baseline and adds the result to live credit — exactly once, with duplicates contributing zero and losses repaired by the next update.

Three counters, never interchangeable: the receiver's cumulative total, the transmitter's baseline, and the transmitter's live availability. And the two subtlest bugs in the module are both confusions between them — a total added as a delta, and a dirty bit cleared while newer progress was still unadvertised.

Module 16 as a whole built one idea: backpressure that cannot travel as a wire travels as accounting. Six pools, because three transaction classes with different lifetimes must not compete for one buffer (16.1). Costs of 1 header + Roundup(bytes / 16) on the Posted (16.2), Non-Posted (16.3) and Completion (16.4) pools. A spend that happens on one declared irrevocable event, with a third state for anything in flight (16.5). And a return that is a published total rather than a delivered token.

Module 17 leaves the Link layers entirely for the Physical Layer — serialization, link training, equalization, and the electrical reality underneath every packet Modules 10 to 16 have been moving.

The idea to carry forward: when the answer must survive being lost, send a position, not a quantity.