Skip to content

PCIe · Module 23

Completion Logic — The Memory of a Transaction You Already Sent

A read leaves and the request is gone. What remains is a context. Matching returned Completions to the right one, accumulating them by byte coverage, and freeing the context exactly once is where requester-side hardware is won or lost.

Chapter 23.4 built the block that sends a Memory Read. The moment that packet is accepted, the request is gone — and the answer will arrive later, possibly in pieces, possibly out of order with respect to other reads, possibly not at all.

What stands between "sent" and "the data is where it belongs" is a context, and this chapter is about owning it correctly.

1. Sources, Scope, and What Is Already Built

2. Sending Is Not Forgetting

When Chapter 23.4's generator accepts the EOP of a Memory Read, the request is on the wire and the requester holds nothing but a promise.

The context is what makes the promise redeemable. It records what was asked for, who asked, and where the answer belongs — and it must survive until the answer is complete, which may be many microseconds and many other transactions later.

Three properties follow directly, and they are the chapter:

The context is created at the request handshake, not at the offer. A request that is offered and never accepted has consumed nothing.

The context is found by what the Completion carries, and nothing else. There is no "current request" — Chapter 13.3 §2's ordering rule explicitly grants no ordering between different transactions, so the next Completion to arrive belongs to whichever transaction the Completer answered first.

And the context is released exactly once, by exactly one of two events: coverage complete, or a terminal error.

3. Identity Is the Pair, and the Matcher Must Never Default

4. Allocation, and the Same-Cycle Free

A context is allocated when a Non-Posted request transfers. Chapter 23.3 §7 established the three-way binding; here the context half is built properly.

Two invariants make everything else provable:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
a live (Requester ID, Tag) is UNIQUE among contexts
FREE + LIVE == CONTEXTS, always

The first is what makes §3's matcher's "two matches" case an error rather than a tie. The second is what makes a leak detectable in one read.

And the same-cycle case is real, exactly as it was for Tags (23.3 §7): the final Completion for context 3 arrives in the same cycle a new request wants a context. One next-state expression applies both, so the freed slot is immediately allocatable and neither the free nor the allocation can be lost to the other.

The dangerous variant here has a twist Tags do not have. Freeing the context also frees its identity — and if a new request is allocated the same identity in the same cycle, a Completion still in flight for the old transaction will match the new context. §8's terminal handling and §7's result ownership together are what bound that window, and P9 asserts that an identity is not reused until its context has been released and its result consumed.

5. Coverage, Not Packet Count

Chapter 13.3 §1 established the model and this chapter enforces it in one engine.

A read is complete when the requested byte extent is covered — not when some number of Completions has arrived, because the Completer chooses the fragmentation and may cut anywhere RCB permits.

§12 Model 9 measured the cost of getting this wrong. 200,000 reads of 128 B to 1 KiB, fragmented at 64 B and 128 B boundaries — 1,041,364 fragments:

Retirement policyConsequence
coverage completecorrect
retire on the first fragment841,364 fragments (80.8%) become unknown-Tag errors

Read that as a cascade, not a single bug. The first fragment frees the context; every subsequent fragment for that transaction now matches nothing, so it is reported as an unknown identity (§3) — and a design that logs those will produce a flood of errors whose root cause is one line of retirement logic. Chapter 21.4 §10 met the identical bug in a fabric context.

The accumulator itself is simple and its guard is not. Bytes accumulate; an accumulation that would exceed the requested extent is an over-return and must be reported rather than absorbed, because absorbing it means writing past the end of the destination buffer.

6. The Context RAM Answers Late

7. Freeing Early Corrupts the Result

8. The Terminal Error Path

A Completion may report an error status (13.2). When it does, three things must happen and no more:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
mark the context terminal        -- no further payload accounting
release the context exactly once -- via the SAME path as success
publish the error to the requester -- as a result, like any other

The second is the one that gets forgotten, and Chapter 23.3 §8 measured what forgetting costs: cleanup written on the success path only exhausted every Tag and wedged the engine after 14 jobs. The release path must not depend on why the transaction ended.

The first matters too. Once a context is terminal, later fragments for that identity must not accumulate bytes — the transaction's outcome is already decided, and adding payload to it would let a failed read report a byte count that suggests success.

What this chapter does not invent. Completion Timeout duration and policy are not specified here — the chapter models a timeout_event input and treats it as one more terminal reason, and Chapter 25.7 owns diagnosing timeouts. Inventing a timeout value would be exactly the kind of remembered constant §1 refuses.

And retry is not modelled. Whether a failed read is reissued is a policy decision belonging to the requester (software or the DMA engine), not to the context table. The context table's job ends at reporting the outcome, which keeps its contract small enough to verify.

9. The Context Lifecycle

A completion context lifecycle state machine. From FREE, an accepted non-posted request moves to ALLOCATED. ALLOCATED moves to WAITING once the request has been sent. A matched completion with payload moves WAITING to ASSEMBLING. ASSEMBLING remains while byte coverage is incomplete. When coverage is complete it moves to REPORT. A completion with an error status, or a timeout event, moves from WAITING or ASSEMBLING to TERMINAL. TERMINAL moves to REPORT. REPORT holds the result record until it is accepted, then moves to RELEASE, which returns the slot to FREE.FREEALLOCATED - identity boundALLOCATED-identit…WAITINGASSEMBLING- bytecoverageTERMINAL - error or timeoutTERMINAL- erroror…REPORT -resultheldRELEASE- slotreturnedrequest acceptedrequest acceptedrequest sentrequest sentfirst matched CplDfirst matched CplDcoverage completecoverage completeerror or timeouterror or timeouterror or timeouterror or timeoutoutcome decidedoutcome decidedresult acceptedresult acceptedslot freeslot free
Figure 1 — one context's lifecycle. Many contexts occupy this machine concurrently and independently; the diagram describes one slot, not a serialized engine. A slot is allocated on the accepted request, accumulates returned bytes, and is released only after its result record has been accepted — by the same path whether the outcome was success or error.

Four things to read out of the figure.

Many slots run this concurrently. The diagram is one context's lifecycle, not the engine's — an engine with 8 contexts has 8 of these advancing independently, which is why there is no single "current request" (§3).

ASSEMBLING has no exit on a packet count. It leaves on coverage, which is §5 — and the absence of a "first fragment" edge is deliberate, because that edge is the 80.8% bug.

Success and error converge on REPORT, and REPORT is the only path to RELEASE. That convergence is §8's single release path, and it is what P18 asserts.

And RELEASE follows an accepted result, not the completion of the transfer (§7). The gap between REPORT and RELEASE is exactly the window §12 measured at 75.9%, and holding the slot across it is what closes it.

10. RTL — The Completion Engine

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// SYNTHESIZABLE. Completion context types.
// The identity is the PAIR (Chapter 21.4 §3). §12 Model 8: matching on
// Tag alone misattributed 39.7% of completions; the pair, 0.
package cplctx_pkg;
 
  parameter int CONTEXTS = 8;
  parameter int CTX_W    = (CONTEXTS <= 1) ? 1 : $clog2(CONTEXTS);
  parameter int RID_W    = 16;
  parameter int TAG_W    = 8;
  parameter int LEN_W    = 13;
  parameter int ADDR_W   = 64;
 
  typedef struct packed {
    logic [RID_W-1:0] requester_id;
    logic [TAG_W-1:0] tag;
  } txn_id_t;
 
  typedef enum logic [2:0] {
    C_FREE=3'd0, C_ALLOC=3'd1, C_WAIT=3'd2,
    C_ASSEMBLE=3'd3, C_TERMINAL=3'd4, C_REPORT=3'd5
  } ctx_state_e;
 
  typedef enum logic [2:0] {
    ST_NONE=3'd0, ST_OK=3'd1, ST_CPL_ERROR=3'd2,
    ST_TIMEOUT=3'd3, ST_OVERRUN=3'd4, ST_RESET=3'd5
  } outcome_e;
 
  typedef struct packed {
    ctx_state_e        st;
    txn_id_t           id;
    logic [ADDR_W-1:0] local_addr;      // where the answer belongs (20.6 §9)
    logic [LEN_W-1:0]  requested;
    logic [LEN_W-1:0]  received;
    outcome_e          outcome;
  } ctx_t;
 
  function automatic bit id_match(input txn_id_t a, input txn_id_t b);
    return (a.requester_id == b.requester_id) && (a.tag == b.tag);
  endfunction
 
  function automatic bit ctx_live(input ctx_state_e s);
    return (s != C_FREE);
  endfunction
 
endpackage
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import cplctx_pkg::*;
 
// SYNTHESIZABLE. Context allocator (§4). A live identity is UNIQUE, and
// FREE + LIVE == CONTEXTS at every cycle. One next-state expression, so a
// same-cycle free and allocate cannot lose either write.
module context_allocator (
  input  logic clk,
  input  logic rst_n,
 
  input  logic       alloc_req,          // a non-posted request TRANSFERRED
  input  txn_id_t    alloc_id,
  input  logic       release_req,
  input  logic [CTX_W-1:0] release_idx,
 
  input  logic [CONTEXTS-1:0] live_map,  // from the table
  input  txn_id_t             live_id [CONTEXTS],
 
  output logic             alloc_grant,
  output logic [CTX_W-1:0] alloc_idx,
  output logic             err_duplicate_identity,   // sticky
  output logic             err_release_of_free       // sticky
);
  logic [CONTEXTS-1:0] free_q;
  logic ed_q, er_q;
  logic [CTX_W-1:0] pick;
  logic found, dup;
 
  assign err_duplicate_identity = ed_q;
  assign err_release_of_free    = er_q;
 
  always_comb begin
    pick = '0; found = 1'b0;
    for (int i = CONTEXTS-1; i >= 0; i--) if (free_q[i]) begin pick = CTX_W'(i); found = 1'b1; end
    // A live identity must be unique -- it is what makes §3's "two matches"
    // an error rather than a tie to break.
    dup = 1'b0;
    for (int i = 0; i < CONTEXTS; i++)
      if (live_map[i] && id_match(live_id[i], alloc_id)) dup = 1'b1;
  end
 
  assign alloc_grant = alloc_req && found && !dup;
  assign alloc_idx   = pick;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin free_q <= {CONTEXTS{1'b1}}; ed_q <= 1'b0; er_q <= 1'b0; end
    else begin
      if (alloc_req && dup)                     ed_q <= 1'b1;
      if (release_req && free_q[release_idx])   er_q <= 1'b1;
      // ONE next-state expression (§4).
      free_q <= (free_q | (release_req ? (CONTEXTS'(1) << release_idx) : '0))
                        & ~(alloc_grant ? (CONTEXTS'(1) << pick) : '0);
    end
  end
endmodule
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import cplctx_pkg::*;
 
// SYNTHESIZABLE. THE MATCHER. Three outcomes, and it NEVER defaults (§3).
// Defaulting to entry 0 is the natural output of a for-loop whose index is
// initialised to zero -- and it routes every orphan into slot 0's owner.
module completion_matcher (
  input  logic       cpl_valid,
  input  txn_id_t    cpl_id,
  input  ctx_t       ctx [CONTEXTS],
 
  output logic [CONTEXTS-1:0] match_vec,
  output logic                hit,
  output logic                unknown_id,
  output logic                ambiguous,
  output logic [CTX_W-1:0]    sel
);
  always_comb
    for (int i = 0; i < CONTEXTS; i++)
      match_vec[i] = ctx_live(ctx[i].st) && id_match(ctx[i].id, cpl_id);
 
  assign hit        = cpl_valid &&  $onehot(match_vec);
  assign unknown_id = cpl_valid &&  (match_vec == '0);
  assign ambiguous  = cpl_valid && !$onehot0(match_vec);
 
  // `sel` is meaningful ONLY when `hit`. Every consumer gates on `hit`, so
  // the zero default can never be acted upon (P5, P6).
  always_comb begin
    sel = '0;
    if (hit) for (int i = 0; i < CONTEXTS; i++) if (match_vec[i]) sel = CTX_W'(i);
  end
endmodule
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import cplctx_pkg::*;
 
// SYNTHESIZABLE. Tag/data alignment through a synchronous lookup (§6).
// §12 Model 10: the RAM's answer belongs to a DIFFERENT tag than the
// completion now presented in 87.5% of cycles. The fix is to send the
// identity and the payload metadata DOWN THE SAME PIPELINE as the read.
module context_lookup_pipe (
  input  logic clk,
  input  logic rst_n,
 
  // stage 0 -- the completion arrives, the RAM read is launched
  input  logic             s0_valid,
  input  txn_id_t          s0_id,
  input  logic [LEN_W-1:0] s0_bytes,
  input  logic             s0_err,
  input  logic [CTX_W-1:0] s0_idx,
 
  // stage 1 -- the RAM answers
  input  ctx_t             ram_dout,
 
  output logic             s1_valid,
  output txn_id_t          s1_id,          // carried, NOT re-read
  output logic [LEN_W-1:0] s1_bytes,
  output logic             s1_err,
  output logic [CTX_W-1:0] s1_idx,
  output ctx_t             s1_ctx,
  output logic             err_skew        // sticky: identity mismatch
);
  logic             v_q, e_q, skew_q;
  txn_id_t          id_q;
  logic [LEN_W-1:0] b_q;
  logic [CTX_W-1:0] i_q;
 
  assign s1_valid = v_q;   assign s1_id    = id_q;
  assign s1_bytes = b_q;   assign s1_err   = e_q;
  assign s1_idx   = i_q;   assign s1_ctx   = ram_dout;
  assign err_skew = skew_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin v_q<=1'b0; id_q<='0; b_q<='0; e_q<=1'b0; i_q<='0; skew_q<=1'b0; end
    else begin
      v_q<=s0_valid; id_q<=s0_id; b_q<=s0_bytes; e_q<=s0_err; i_q<=s0_idx;
      // The context that emerged must be the one fetched for THIS identity.
      if (v_q && ctx_live(ram_dout.st) && !id_match(ram_dout.id, id_q)) skew_q <= 1'b1;
    end
  end
endmodule
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import cplctx_pkg::*;
 
// SYNTHESIZABLE. THE FLAGSHIP BLOCK. Coverage accumulation, terminal
// handling, and result ownership. Three measured bugs are prevented here:
// retire-on-first-fragment (80.8% of fragments orphaned), free-under-a-
// stalled-result (75.9% corrupted), and absorbing an over-return.
module context_table (
  input  logic clk,
  input  logic rst_n,
 
  // allocation
  input  logic             alloc_grant,
  input  logic [CTX_W-1:0] alloc_idx,
  input  txn_id_t          alloc_id,
  input  logic [ADDR_W-1:0] alloc_local_addr,
  input  logic [LEN_W-1:0] alloc_requested,
  input  logic             request_sent,
 
  // matched completion (post-pipeline, §6)
  input  logic             cpl_hit,
  input  logic [CTX_W-1:0] cpl_idx,
  input  logic [LEN_W-1:0] cpl_bytes,
  input  logic             cpl_error,
  input  logic             timeout_event,        // input, never invented (§8)
  input  logic [CTX_W-1:0] timeout_idx,
 
  // result handoff
  input  logic             result_ready,
 
  output ctx_t             ctx [CONTEXTS],
  output logic [CONTEXTS-1:0] live_map,
  output txn_id_t          live_id [CONTEXTS],
  output logic             result_valid,
  output txn_id_t          result_id,
  output outcome_e         result_outcome,
  output logic [LEN_W-1:0] result_bytes,
  output logic [CTX_W-1:0] result_idx,
  output logic             release_req,
  output logic             err_overrun          // sticky
);
  ctx_t c_q [CONTEXTS];
  logic e_q;
  int   rep;
 
  always_comb begin
    for (int i = 0; i < CONTEXTS; i++) begin
      ctx[i]      = c_q[i];
      live_map[i] = ctx_live(c_q[i].st);
      live_id[i]  = c_q[i].id;
    end
    // The FIRST context in REPORT owns the result output.
    rep = -1;
    for (int i = CONTEXTS-1; i >= 0; i--) if (c_q[i].st == C_REPORT) rep = i;
  end
 
  assign result_valid   = (rep >= 0);
  assign result_id      = (rep >= 0) ? c_q[rep].id       : '0;
  assign result_outcome = (rep >= 0) ? c_q[rep].outcome  : ST_NONE;
  assign result_bytes   = (rep >= 0) ? c_q[rep].received : '0;
  assign result_idx     = (rep >= 0) ? CTX_W'(rep)       : '0;
  // RELEASE follows an ACCEPTED result, never the end of the transfer (§7).
  assign release_req    = result_valid && result_ready;
  assign err_overrun    = e_q;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int i = 0; i < CONTEXTS; i++) c_q[i] <= '0;
      e_q <= 1'b0;
    end else begin
      if (alloc_grant)
        c_q[alloc_idx] <= '{st:C_ALLOC, id:alloc_id, local_addr:alloc_local_addr,
                            requested:alloc_requested, received:'0, outcome:ST_NONE};
      if (request_sent)
        for (int i = 0; i < CONTEXTS; i++)
          if (c_q[i].st == C_ALLOC) c_q[i].st <= C_WAIT;
 
      if (cpl_hit) begin
        automatic logic [LEN_W:0] tot = {1'b0, c_q[cpl_idx].received} + {1'b0, cpl_bytes};
        if (cpl_error) begin
          // TERMINAL: no further payload accounting (§8).
          c_q[cpl_idx].st      <= C_TERMINAL;
          c_q[cpl_idx].outcome <= ST_CPL_ERROR;
        end else if (c_q[cpl_idx].st == C_TERMINAL) begin
          // already decided -- ignore payload, do not "un-fail" the read
        end else if (tot > {1'b0, c_q[cpl_idx].requested}) begin
          e_q <= 1'b1;                                   // REPORT the over-return
          c_q[cpl_idx].st      <= C_TERMINAL;
          c_q[cpl_idx].outcome <= ST_OVERRUN;
        end else begin
          c_q[cpl_idx].received <= tot[LEN_W-1:0];
          // COVERAGE, not packet count (§5). Retiring on the first fragment
          // orphaned 80.8% of all fragments in §12 Model 9.
          if (tot >= {1'b0, c_q[cpl_idx].requested}) begin
            c_q[cpl_idx].st      <= C_REPORT;
            c_q[cpl_idx].outcome <= ST_OK;
          end else c_q[cpl_idx].st <= C_ASSEMBLE;
        end
      end
 
      if (timeout_event && ctx_live(c_q[timeout_idx].st)
                        && (c_q[timeout_idx].st != C_REPORT)) begin
        c_q[timeout_idx].st      <= C_TERMINAL;
        c_q[timeout_idx].outcome <= ST_TIMEOUT;
      end
 
      for (int i = 0; i < CONTEXTS; i++)
        if (c_q[i].st == C_TERMINAL) c_q[i].st <= C_REPORT;   // one path (§8)
 
      // The slot is freed ONLY after the result has been accepted (§7).
      if (release_req) c_q[result_idx] <= '0;
    end
  end
endmodule
Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
import cplctx_pkg::*;
 
// VERIFICATION-ONLY. Independent oracle: a dictionary keyed by identity,
// deliberately NOT sharing the DUT's match function, so a shared
// misconception about identity cannot make both agree.
module completion_oracle #(parameter int MAX = 64) (
  input logic clk, rst_n,
  input logic       alloc, txn_id_t alloc_id, input logic [LEN_W-1:0] alloc_bytes,
  input logic       cpl,   txn_id_t cpl_id,   input logic [LEN_W-1:0] cpl_bytes,
  input logic       retire, txn_id_t retire_id, input logic [LEN_W-1:0] retire_bytes,
 
  output logic err_unknown, err_over, err_early, err_leak,
  output int    outstanding
);
  txn_id_t          k [MAX];
  logic [LEN_W-1:0] want [MAX], got [MAX];
  logic             used [MAX];
  int               n;
 
  assign outstanding = n;
 
  function automatic int find(input txn_id_t q);
    for (int i = 0; i < MAX; i++)
      if (used[i] && (k[i].requester_id == q.requester_id) && (k[i].tag == q.tag)) return i;
    return -1;
  endfunction
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      for (int i=0;i<MAX;i++) used[i]<=1'b0;
      n<=0; err_unknown<=1'b0; err_over<=1'b0; err_early<=1'b0; err_leak<=1'b0;
    end else begin
      if (alloc) for (int i=0;i<MAX;i++) if (!used[i]) begin
        used[i]<=1'b1; k[i]<=alloc_id; want[i]<=alloc_bytes; got[i]<='0; n<=n+1; break;
      end
      if (cpl) begin
        automatic int j = find(cpl_id);
        if (j < 0) err_unknown <= 1'b1;
        else begin
          got[j] <= got[j] + cpl_bytes;
          if ((got[j] + cpl_bytes) > want[j]) err_over <= 1'b1;
        end
      end
      if (retire) begin
        automatic int j = find(retire_id);
        if (j < 0) err_unknown <= 1'b1;
        else begin
          if (got[j] < want[j]) err_early <= 1'b1;      // retired before coverage
          used[j] <= 1'b0; n <= n-1;
        end
      end
    end
  end
endmodule

Classification: five synthesizable, one verification-only.

Failure — eight. Matching on Tag alone (39.7%). Defaulting an unmatched Completion to slot 0. Retiring on the first fragment (80.8% of fragments orphaned). A synchronous lookup without a matched pipeline (87.5%). Freeing the context under a stalled result (75.9%). Absorbing an over-return. Accumulating payload into a terminal context. And freeing on the error path by a different route than success.

11. Same-Cycle Audit and Assertions

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// ==================================================================
// ALLOCATION AND IDENTITY (§3, §4).
// ==================================================================
 
// P1: a context is allocated only on a request TRANSFER, never an offer.
property p_alloc_on_transfer;
  @(posedge clk) disable iff (!rst_n)
    alloc_grant |-> alloc_req;
endproperty
 
// P2: a live identity is UNIQUE. This is what makes "two matches" an
// error rather than a tie (§3).
property p_live_identity_unique;
  @(posedge clk) disable iff (!rst_n)
    (live_map[0] && live_map[1]) |-> !id_match(live_id[0], live_id[1]);
endproperty
 
// P3: allocating a duplicate identity is REFUSED and reported.
property p_duplicate_identity_refused;
  @(posedge clk) disable iff (!rst_n)
    (alloc_req && dup) |-> (!alloc_grant ##1 err_duplicate_identity);
endproperty
 
// P4: FREE + LIVE == CONTEXTS, always -- a leak is one read away.
property p_context_conservation;
  @(posedge clk) disable iff (!rst_n)
    ($countones(free_q) + $countones(live_map) == CONTEXTS);
endproperty
 
// ==================================================================
// MATCHING (§3) -- three outcomes, no default.
// ==================================================================
 
// P5: exactly one outcome per arriving completion.
property p_match_outcome_total;
  @(posedge clk) disable iff (!rst_n)
    cpl_valid |-> $onehot({hit, unknown_id, ambiguous});
endproperty
 
// P6: an unmatched completion applies NOTHING -- it does not default to
// slot 0. §12 Model 8: tag-only matching misattributed 39.7%.
property p_unknown_applies_nothing;
  @(posedge clk) disable iff (!rst_n)
    unknown_id |=> $stable(ctx[0].received);
endproperty
 
// P7: a hit names a context whose identity really matches.
property p_hit_identity_matches;
  @(posedge clk) disable iff (!rst_n)
    hit |-> id_match(ctx[sel].id, cpl_id);
endproperty
 
// P8: same-cycle release and allocate -- the freed slot is allocatable.
property p_same_cycle_release_alloc;
  @(posedge clk) disable iff (!rst_n)
    (release_req && alloc_req) |=> ($countones(free_q) + $countones(live_map) == CONTEXTS);
endproperty
 
// P9: an identity is not reused while its context is still live.
property p_no_identity_reuse_while_live;
  @(posedge clk) disable iff (!rst_n)
    (alloc_grant) |-> !live_map[alloc_idx];
endproperty
 
// ==================================================================
// COVERAGE (§5) -- bytes, never packets.
// ==================================================================
 
// P10: received bytes are monotonic.
property p_received_monotonic;
  @(posedge clk) disable iff (!rst_n)
    ctx_live(ctx[0].st) |-> (ctx[0].received >= $past(ctx[0].received));
endproperty
 
// P11: a context reports only when coverage is complete or terminal.
// §12 Model 9: retiring on the first fragment orphaned 80.8% of fragments.
property p_report_requires_coverage_or_terminal;
  @(posedge clk) disable iff (!rst_n)
    ((ctx[0].st == C_REPORT) && (ctx[0].outcome == ST_OK))
      |-> (ctx[0].received >= ctx[0].requested);
endproperty
 
// P12: received never exceeds requested; an over-return is REPORTED.
property p_no_overrun_absorbed;
  @(posedge clk) disable iff (!rst_n)
    ctx_live(ctx[0].st) |-> (ctx[0].received <= ctx[0].requested);
endproperty
 
// P13: a terminal context accumulates no further payload (§8).
property p_terminal_stops_accounting;
  @(posedge clk) disable iff (!rst_n)
    (ctx[0].st == C_TERMINAL) |=> $stable(ctx[0].received);
endproperty
 
// P14: TAG/DATA ALIGNMENT. The context that emerged from the RAM was
// fetched for the identity presented at the same stage. §12 Model 10:
// 87.5% of cycles are misaligned without this pipeline.
property p_lookup_alignment;
  @(posedge clk) disable iff (!rst_n)
    (s1_valid && ctx_live(s1_ctx.st)) |-> id_match(s1_ctx.id, s1_id);
endproperty
 
// P15: skew is reported stickily rather than silently applied.
property p_skew_reported;
  @(posedge clk) disable iff (!rst_n)
    (s1_valid && ctx_live(s1_ctx.st) && !id_match(s1_ctx.id, s1_id)) |=> err_skew;
endproperty
 
// ==================================================================
// RESULT OWNERSHIP (§7).
// ==================================================================
 
// P16: the context is released only after the result is ACCEPTED.
// §12 Model 11: freeing earlier corrupted 75.9% of stalled results.
property p_release_after_result_accepted;
  @(posedge clk) disable iff (!rst_n)
    release_req |-> (result_valid && result_ready);
endproperty
 
// P17: the result record is STABLE while stalled.
property p_result_stable_under_stall;
  @(posedge clk) disable iff (!rst_n)
    (result_valid && !result_ready) |=>
      (result_valid && $stable(result_id) && $stable(result_outcome)
                    && $stable(result_bytes));
endproperty
 
// P18: every outcome reaches REPORT by the same path -- success and error
// share one release route (§8).
property p_terminal_reaches_report;
  @(posedge clk) disable iff (!rst_n)
    (ctx[0].st == C_TERMINAL) |=> (ctx[0].st == C_REPORT);
endproperty
 
// P19: the result identity is the ORIGINAL request's identity.
property p_result_identity_original;
  @(posedge clk) disable iff (!rst_n)
    result_valid |-> id_match(result_id, ctx[result_idx].id);
endproperty
 
// P20: a released slot is not reported again -- exactly one result.
property p_single_result_per_transaction;
  @(posedge clk) disable iff (!rst_n)
    release_req |=> (ctx[$past(result_idx)].st == C_FREE);
endproperty
 
// ==================================================================
// RESET, RANGE AND NON-INTERFERENCE.
// ==================================================================
 
// P21: reset invalidates every context and publishes nothing.
property p_reset_clears_contexts;
  @(posedge clk)
    (!rst_n) |=> (!result_valid && (live_map == '0));
endproperty
 
// P22: a completion after reset finds no context and is reported unknown,
// rather than matching a stale entry.
property p_no_stale_match_after_reset;
  @(posedge clk) disable iff (!rst_n)
    ($past(!rst_n) && cpl_valid) |-> unknown_id;
endproperty
 
// P23: every index is in range -- CONTEXTS = 1 included.
property p_index_in_range;
  @(posedge clk) disable iff (!rst_n)
    (sel < CTX_W'(CONTEXTS)) && (alloc_idx < CTX_W'(CONTEXTS))
      && (result_idx < CTX_W'(CONTEXTS));
endproperty
 
// P24: releasing a free context is reported, never absorbed.
property p_release_of_free_flagged;
  @(posedge clk) disable iff (!rst_n)
    (release_req && free_q[release_idx]) |=> err_release_of_free;
endproperty
 
// P25: outstanding population equals the number of live contexts.
property p_population_consistent;
  @(posedge clk) disable iff (!rst_n)
    ($countones(live_map) == outstanding);
endproperty
 
// P26: a completion with zero payload bytes does not advance coverage.
property p_zero_byte_no_coverage;
  @(posedge clk) disable iff (!rst_n)
    (cpl_hit && (cpl_bytes == '0) && !cpl_error) |=> $stable(ctx[cpl_idx].received);
endproperty
 
// P27: the timeout input is treated as one terminal reason among others --
// no duration is computed here (§8).
property p_timeout_is_terminal_only;
  @(posedge clk) disable iff (!rst_n)
    (timeout_event && ctx_live(ctx[timeout_idx].st)
      && (ctx[timeout_idx].st != C_REPORT)) |=> (ctx[timeout_idx].st == C_TERMINAL);
endproperty
 
// P28: the oracle never drives the engine.
property p_oracle_non_functional;
  @(posedge clk) disable iff (!rst_n)
    $stable({cpl_hit, alloc_grant}) or !$stable(outstanding);
endproperty

Twenty-eight properties. P1–P9 are allocation and identity; P10–P15 are coverage and the pipeline, and P14 is the one that only exists because a real table lives in RAM; P16–P20 are result ownership, which is the race §7 measured at 75.9% and which no functional test without a stalled consumer will ever reach.

12. Measured Behaviour

13. Verification — DV and Mutations

DV, against the independent identity-keyed oracle of §10 — which does not share the DUT's match function: one request, one Completion · one request, many fragments · many Tags completing out of order · the same Tag from two Requester IDs · an unknown identity · a duplicate identity at allocation · a Completion with error status · a Completion with error status after partial data · an over-return · a zero-byte Completion · a stalled result · a stalled result while new Completions arrive · context exhaustion · CONTEXTS = 1 · reset with contexts outstanding · a Completion in the cycle after reset · a timeout event · a timeout on a context already in REPORT.

#MutationSymptomCaught by
1Create the context on req_validcontexts consumed by requests never sentP1
2Allow a duplicate live identitytwo contexts match one CompletionP2, P3
3Match on Tag alone39.7% misattributed; data into the wrong bufferP7
4Default an unmatched Completion to slot 0every orphan corrupts slot 0's transactionP6
5Retire on the first fragment80.8% of fragments become unknown-identity errorsP11
6Count Completions instead of bytesbreaks on the first Completer that splits differentlyP11
7Interpret Byte Count as this-fragment-onlycoverage never completes, or completes earlyP11, P12
8Absorb an over-return silentlywrites past the end of the destination bufferP12
9Treat a Completion error as successa failed read reports completeP13, P18
10Accumulate payload into a terminal contexta failed read reports a plausible byte countP13
11Free the context on the error path by a different routea Tag leak on errors (23.3 §8)P18, P24
12Ignore the context RAM's latency87.5% of lookups paired with the wrong CompletionP14, P15
13Re-read the identity at stage 1 instead of carrying itthe same skew by another routeP14
14Free the context when coverage completes75.9% of stalled results corrupted (§12)P16
15Pulse the result for one cyclea busy consumer loses the transactionP17
16Let the result record change while stalledthe consumer reads a different transactionP17
17Take the result identity from the newest contextresults attributed to the wrong requestP19
18Report the same context twiceduplicate results; software double-freesP20
19Leave contexts valid through reseta stale Completion matches after restartP21, P22
20Assume reordered Completions are illegalcorrect traffic rejected (13.3 §7)design review
21Serialize all Tags to avoid reorderingthroughput collapses to one outstanding readdesign review
22Let a later fragment overwrite the outcomean error is erased by the fragment after itP13
23Advance coverage on a zero-byte Completiona status-only Completion counts as dataP26
24Truncate the Tag widthtwo identities aliasP23
25Let the live count underflowconservation breaks; leaks become invisibleP4, P25
26Release a free contextthe pool exceeds its sizeP24
27Invent a Completion Timeout duration in this blockan unsourced constant in the datapath (§8)P27
28Retry the read from the context tablepolicy in the wrong block (§8)design review
29Use the DUT's match function in the DV oraclea shared misconception makes both agreedesign review
30Let a debug counter gate the free logicthe instrument changes the engineP28

Two counterexamples worth stating explicitly.

Mutation 14 is the one this chapter exists for. Freeing the context when coverage completes is obviously right — the transaction is done, the slot is no longer needed. It is wrong because the result has not been delivered yet, and §12 Model 11 measured the context being reallocated before the consumer read it in 75.9% of stalled cases. The consumer then reads metadata belonging to a different request — a result that is well-formed, plausible and attributed to the wrong transaction. No test without a stalled result consumer can reach this, which is why it is on the DV list twice.

Mutation 12 is invisible in RTL simulation with a register-based table and appears the moment the table becomes a RAM. The design is correct at 8 contexts held in flops; at 256 contexts in block RAM the lookup gains a cycle, the identity is re-read from the interface at stage 1, and 87.5% of lookups pair a context with the wrong Completion (§12 Model 10). The symptom is that the FPGA build fails and the simulation passes — which is §14's last scenario, and P14 is what makes it a simulation failure instead.

14. Debugging

Symptom — a read hangs although the analyzer shows every Completion arriving. The Completions are not matching. Read err_unknown_id: if it is set, the identity the matcher computes differs from the one the Completions carry — most often Tag-only matching (mutation 3) or a truncated Tag width (mutation 24). If it is clear, check coverage: the accumulator may be short by a fragment because Byte Count was interpreted as this-fragment-only (mutation 7).

Symptom — only split reads fail; single-Completion reads are fine. Retirement on the first fragment (§5, mutation 5). A read answered in one Completion is retired correctly by both the right and the wrong logic. The tell is a flood of unknown-identity errors that begins one fragment after each large read — §12 measured 80.8% of fragments orphaned.

Symptom — corruption appears only with more than one outstanding request. Something is using "the current request" instead of the matched context (§3). At one outstanding transaction there is only one, so the wrong design is accidentally right. This is the same signature as Chapter 23.3 §16's QD1-versus-QD8 case, one layer down.

Symptom — one Completion error eventually wedges the engine. The error path releases the context by a different route than success — or not at all (§8, mutation 11). Read the live-context count at rest: a healthy engine returns to zero between transactions. Chapter 23.3 §14 measured the same defect wedging a DMA engine after 14 jobs.

Symptom — the result's metadata belongs to the next request. The context was freed before the result was consumed (§7, mutation 14). Reproduce by stalling the result consumer — without that stall the window does not exist, which is why it survives to the field.

Symptom — unknown-identity errors immediately after reset. Completions for pre-reset transactions are still in flight and the table is empty, so they correctly report unknown. That is the design working; the bug would be a stale context that matches one of them (mutation 19, P22). Distinguish by whether the errors stop once the in-flight Completions drain.

Symptom — the design passes simulation and fails on the FPGA. Suspect the context RAM's latency (§6, mutation 12). A flop-based table has no lookup delay; a block RAM does, and 87.5% of lookups pair the wrong context. err_skew names it directly — and if the design has no such check, the giveaway is that the failure rate is near 100% rather than intermittent.

15. Misconceptions

"Sending the request is the hard part." Sending is the easy part; remembering is the chapter (§2).

"A Tag identifies the transaction." (Requester ID, Tag) does — Tag alone misattributed 39.7% (§3).

"An unmatched Completion can go to entry 0; it is an error anyway." Then the error corrupts entry 0's transaction (§3, P6).

"One Completion means the read is done." 80.8% of fragments become orphans under that rule (§5).

"Byte Count is how much this Completion carries." It is the remaining count including this one (13.3 §2).

"Count the Completions; the Completer will use the largest payload it may." It is not obliged to (13.3 §1).

"Reordered Completions are a protocol violation." No ordering is implied between different transactions (13.3 §2).

"Serialize the Tags and the problem goes away." So does the throughput (20.5 §5).

"An over-return can be clamped." Clamping writes past the buffer's end; report it (§5, P12).

"An error status still tells you how many bytes arrived." Not once the outcome is terminal — accumulating after that lets a failed read look plausible (§8).

"Free the context as soon as the data is complete." 75.9% of stalled results corrupted (§7).

"A one-cycle result pulse is fine." A busy consumer loses the transaction (23.3 §9).

"A register table and a RAM table behave the same." The RAM answers a cycle late, and 87.5% of lookups pair the wrong Completion (§6).

"The context table should retry failed reads." Retry is the requester's policy, not the table's (§8).

16. Understanding Check

Q1. Endpoint Function 0 and Function 1 each have Tag 5 outstanding. A Completion arrives for Tag 5. Which context receives it? The one whose (Requester ID, Tag) matches the Completion's fields (21.4 §3, §3 here). Matching on Tag alone picks whichever entry the loop found first and misattributed 39.7% of Completions in §12 Model 8 — and misattribution means payload written into the other Function's buffer, not a dropped packet.

Q2. A 512-byte read returns as four 128-byte Completions. Your context is freed on the first. What does the system report? Three unknown-identity errors, and a read that completed with a quarter of its data. §12 Model 9 measured 80.8% of all fragments becoming orphans under that rule. The cascade is the diagnostic clue: errors that begin exactly one fragment after each large read point at retirement logic, not at the fabric.

Q3. Your context table moves from flops to block RAM and the design stops working. What changed? The lookup gained a cycle (§6). The context data now emerges while a different Completion is on the interface — 87.5% of cycles in §12 Model 10. The fix is to pipeline the identity, byte count and status alongside the RAM read so they re-emerge together, and P14 asserts the pairing so this fails in simulation rather than on the board.

Q4. Coverage is complete and the result consumer is stalled. May the context be freed? No (§7). §12 Model 11 measured the slot being reallocated before the consumer read the result in 75.9% of stalled cases, after which the metadata handed out belongs to a different transaction. Either hold the context until the result is accepted (P16), or copy the result into an immutable queue first — but not neither.

Q5. A Completion with an error status arrives for a context that already has 256 of 512 bytes. What happens to the 256 bytes and to the context? The context becomes terminal and accumulates nothing further (§8, P13). The 256 bytes are reported as what arrived, with an error outcome — they are not promoted to a success, and no later fragment may un-fail the transaction (mutation 22). The context is released through the same path a successful one uses (P18), because a release path that depends on the reason is how Chapter 23.3 §8's engine wedged after 14 jobs.

Q6. Why does this chapter treat Completion Timeout as an input rather than computing it? Because the duration and policy are not something this chapter can source (§1, §8), and inventing a constant in the datapath is worse than omitting one. Structurally the timeout is just one more terminal reason among error status and reset — it enters C_TERMINAL like any other (P27) — which keeps the context table's contract small enough to verify. Chapter 25.7 owns diagnosing timeouts.

17. What's Next

Module 23's data path is now complete. 23.1 placed the boundaries, 23.2 decoded inbound addresses, 23.3 drove the transfers, 23.4 built the packets, and this chapter remembered them.

Chapter 23.6 Design Patterns closes the module by naming what has now repeated too often to be a coincidence. This chapter alone contributed four instances: the three-outcome decoder (§3, after 21.1 and 23.2), allocate/use/free with exactly one terminal event (§8), hold until handshake (§7), and metadata that must travel with its data through a pipeline (§6, after 23.4 §3).

Then Module 24 stops building and starts proving. Every model in this chapter was an independent oracle keyed by identity — and 24.1 is about why that independence is the whole point, and what it takes to organize it across three protocol layers rather than one block.