PCIe · Module 23
Completion Logic — The Memory of a Transaction You Already Sent
A read leaves and the request is gone. What remains is a context. Matching returned Completions to the right one, accumulating them by byte coverage, and freeing the context exactly once is where requester-side hardware is won or lost.
Chapter 23.4 built the block that sends a Memory Read. The moment that packet is accepted, the request is gone — and the answer will arrive later, possibly in pieces, possibly out of order with respect to other reads, possibly not at all.
What stands between "sent" and "the data is where it belongs" is a context, and this chapter is about owning it correctly.
1. Sources, Scope, and What Is Already Built
2. Sending Is Not Forgetting
When Chapter 23.4's generator accepts the EOP of a Memory Read, the request is on the wire and the requester holds nothing but a promise.
The context is what makes the promise redeemable. It records what was asked for, who asked, and where the answer belongs — and it must survive until the answer is complete, which may be many microseconds and many other transactions later.
Three properties follow directly, and they are the chapter:
The context is created at the request handshake, not at the offer. A request that is offered and never accepted has consumed nothing.
The context is found by what the Completion carries, and nothing else. There is no "current request" — Chapter 13.3 §2's ordering rule explicitly grants no ordering between different transactions, so the next Completion to arrive belongs to whichever transaction the Completer answered first.
And the context is released exactly once, by exactly one of two events: coverage complete, or a terminal error.
3. Identity Is the Pair, and the Matcher Must Never Default
4. Allocation, and the Same-Cycle Free
A context is allocated when a Non-Posted request transfers. Chapter 23.3 §7 established the three-way binding; here the context half is built properly.
Two invariants make everything else provable:
a live (Requester ID, Tag) is UNIQUE among contexts
FREE + LIVE == CONTEXTS, alwaysThe first is what makes §3's matcher's "two matches" case an error rather than a tie. The second is what makes a leak detectable in one read.
And the same-cycle case is real, exactly as it was for Tags (23.3 §7): the final Completion for context 3 arrives in the same cycle a new request wants a context. One next-state expression applies both, so the freed slot is immediately allocatable and neither the free nor the allocation can be lost to the other.
The dangerous variant here has a twist Tags do not have. Freeing the context also frees its identity — and if a new request is allocated the same identity in the same cycle, a Completion still in flight for the old transaction will match the new context. §8's terminal handling and §7's result ownership together are what bound that window, and P9 asserts that an identity is not reused until its context has been released and its result consumed.
5. Coverage, Not Packet Count
Chapter 13.3 §1 established the model and this chapter enforces it in one engine.
A read is complete when the requested byte extent is covered — not when some number of Completions has arrived, because the Completer chooses the fragmentation and may cut anywhere RCB permits.
§12 Model 9 measured the cost of getting this wrong. 200,000 reads of 128 B to 1 KiB, fragmented at 64 B and 128 B boundaries — 1,041,364 fragments:
| Retirement policy | Consequence |
|---|---|
| coverage complete | correct |
| retire on the first fragment | 841,364 fragments (80.8%) become unknown-Tag errors |
Read that as a cascade, not a single bug. The first fragment frees the context; every subsequent fragment for that transaction now matches nothing, so it is reported as an unknown identity (§3) — and a design that logs those will produce a flood of errors whose root cause is one line of retirement logic. Chapter 21.4 §10 met the identical bug in a fabric context.
The accumulator itself is simple and its guard is not. Bytes accumulate; an accumulation that would exceed the requested extent is an over-return and must be reported rather than absorbed, because absorbing it means writing past the end of the destination buffer.
6. The Context RAM Answers Late
7. Freeing Early Corrupts the Result
8. The Terminal Error Path
A Completion may report an error status (13.2). When it does, three things must happen and no more:
mark the context terminal -- no further payload accounting
release the context exactly once -- via the SAME path as success
publish the error to the requester -- as a result, like any otherThe second is the one that gets forgotten, and Chapter 23.3 §8 measured what forgetting costs: cleanup written on the success path only exhausted every Tag and wedged the engine after 14 jobs. The release path must not depend on why the transaction ended.
The first matters too. Once a context is terminal, later fragments for that identity must not accumulate bytes — the transaction's outcome is already decided, and adding payload to it would let a failed read report a byte count that suggests success.
What this chapter does not invent. Completion Timeout duration and policy are not specified here — the chapter models a timeout_event input and treats it as one more terminal reason, and Chapter 25.7 owns diagnosing timeouts. Inventing a timeout value would be exactly the kind of remembered constant §1 refuses.
And retry is not modelled. Whether a failed read is reissued is a policy decision belonging to the requester (software or the DMA engine), not to the context table. The context table's job ends at reporting the outcome, which keeps its contract small enough to verify.
9. The Context Lifecycle
Four things to read out of the figure.
Many slots run this concurrently. The diagram is one context's lifecycle, not the engine's — an engine with 8 contexts has 8 of these advancing independently, which is why there is no single "current request" (§3).
ASSEMBLING has no exit on a packet count. It leaves on coverage, which is §5 — and the absence of a "first fragment" edge is deliberate, because that edge is the 80.8% bug.
Success and error converge on REPORT, and REPORT is the only path to RELEASE. That convergence is §8's single release path, and it is what P18 asserts.
And RELEASE follows an accepted result, not the completion of the transfer (§7). The gap between REPORT and RELEASE is exactly the window §12 measured at 75.9%, and holding the slot across it is what closes it.
10. RTL — The Completion Engine
// SYNTHESIZABLE. Completion context types.
// The identity is the PAIR (Chapter 21.4 §3). §12 Model 8: matching on
// Tag alone misattributed 39.7% of completions; the pair, 0.
package cplctx_pkg;
parameter int CONTEXTS = 8;
parameter int CTX_W = (CONTEXTS <= 1) ? 1 : $clog2(CONTEXTS);
parameter int RID_W = 16;
parameter int TAG_W = 8;
parameter int LEN_W = 13;
parameter int ADDR_W = 64;
typedef struct packed {
logic [RID_W-1:0] requester_id;
logic [TAG_W-1:0] tag;
} txn_id_t;
typedef enum logic [2:0] {
C_FREE=3'd0, C_ALLOC=3'd1, C_WAIT=3'd2,
C_ASSEMBLE=3'd3, C_TERMINAL=3'd4, C_REPORT=3'd5
} ctx_state_e;
typedef enum logic [2:0] {
ST_NONE=3'd0, ST_OK=3'd1, ST_CPL_ERROR=3'd2,
ST_TIMEOUT=3'd3, ST_OVERRUN=3'd4, ST_RESET=3'd5
} outcome_e;
typedef struct packed {
ctx_state_e st;
txn_id_t id;
logic [ADDR_W-1:0] local_addr; // where the answer belongs (20.6 §9)
logic [LEN_W-1:0] requested;
logic [LEN_W-1:0] received;
outcome_e outcome;
} ctx_t;
function automatic bit id_match(input txn_id_t a, input txn_id_t b);
return (a.requester_id == b.requester_id) && (a.tag == b.tag);
endfunction
function automatic bit ctx_live(input ctx_state_e s);
return (s != C_FREE);
endfunction
endpackageimport cplctx_pkg::*;
// SYNTHESIZABLE. Context allocator (§4). A live identity is UNIQUE, and
// FREE + LIVE == CONTEXTS at every cycle. One next-state expression, so a
// same-cycle free and allocate cannot lose either write.
module context_allocator (
input logic clk,
input logic rst_n,
input logic alloc_req, // a non-posted request TRANSFERRED
input txn_id_t alloc_id,
input logic release_req,
input logic [CTX_W-1:0] release_idx,
input logic [CONTEXTS-1:0] live_map, // from the table
input txn_id_t live_id [CONTEXTS],
output logic alloc_grant,
output logic [CTX_W-1:0] alloc_idx,
output logic err_duplicate_identity, // sticky
output logic err_release_of_free // sticky
);
logic [CONTEXTS-1:0] free_q;
logic ed_q, er_q;
logic [CTX_W-1:0] pick;
logic found, dup;
assign err_duplicate_identity = ed_q;
assign err_release_of_free = er_q;
always_comb begin
pick = '0; found = 1'b0;
for (int i = CONTEXTS-1; i >= 0; i--) if (free_q[i]) begin pick = CTX_W'(i); found = 1'b1; end
// A live identity must be unique -- it is what makes §3's "two matches"
// an error rather than a tie to break.
dup = 1'b0;
for (int i = 0; i < CONTEXTS; i++)
if (live_map[i] && id_match(live_id[i], alloc_id)) dup = 1'b1;
end
assign alloc_grant = alloc_req && found && !dup;
assign alloc_idx = pick;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin free_q <= {CONTEXTS{1'b1}}; ed_q <= 1'b0; er_q <= 1'b0; end
else begin
if (alloc_req && dup) ed_q <= 1'b1;
if (release_req && free_q[release_idx]) er_q <= 1'b1;
// ONE next-state expression (§4).
free_q <= (free_q | (release_req ? (CONTEXTS'(1) << release_idx) : '0))
& ~(alloc_grant ? (CONTEXTS'(1) << pick) : '0);
end
end
endmoduleimport cplctx_pkg::*;
// SYNTHESIZABLE. THE MATCHER. Three outcomes, and it NEVER defaults (§3).
// Defaulting to entry 0 is the natural output of a for-loop whose index is
// initialised to zero -- and it routes every orphan into slot 0's owner.
module completion_matcher (
input logic cpl_valid,
input txn_id_t cpl_id,
input ctx_t ctx [CONTEXTS],
output logic [CONTEXTS-1:0] match_vec,
output logic hit,
output logic unknown_id,
output logic ambiguous,
output logic [CTX_W-1:0] sel
);
always_comb
for (int i = 0; i < CONTEXTS; i++)
match_vec[i] = ctx_live(ctx[i].st) && id_match(ctx[i].id, cpl_id);
assign hit = cpl_valid && $onehot(match_vec);
assign unknown_id = cpl_valid && (match_vec == '0);
assign ambiguous = cpl_valid && !$onehot0(match_vec);
// `sel` is meaningful ONLY when `hit`. Every consumer gates on `hit`, so
// the zero default can never be acted upon (P5, P6).
always_comb begin
sel = '0;
if (hit) for (int i = 0; i < CONTEXTS; i++) if (match_vec[i]) sel = CTX_W'(i);
end
endmoduleimport cplctx_pkg::*;
// SYNTHESIZABLE. Tag/data alignment through a synchronous lookup (§6).
// §12 Model 10: the RAM's answer belongs to a DIFFERENT tag than the
// completion now presented in 87.5% of cycles. The fix is to send the
// identity and the payload metadata DOWN THE SAME PIPELINE as the read.
module context_lookup_pipe (
input logic clk,
input logic rst_n,
// stage 0 -- the completion arrives, the RAM read is launched
input logic s0_valid,
input txn_id_t s0_id,
input logic [LEN_W-1:0] s0_bytes,
input logic s0_err,
input logic [CTX_W-1:0] s0_idx,
// stage 1 -- the RAM answers
input ctx_t ram_dout,
output logic s1_valid,
output txn_id_t s1_id, // carried, NOT re-read
output logic [LEN_W-1:0] s1_bytes,
output logic s1_err,
output logic [CTX_W-1:0] s1_idx,
output ctx_t s1_ctx,
output logic err_skew // sticky: identity mismatch
);
logic v_q, e_q, skew_q;
txn_id_t id_q;
logic [LEN_W-1:0] b_q;
logic [CTX_W-1:0] i_q;
assign s1_valid = v_q; assign s1_id = id_q;
assign s1_bytes = b_q; assign s1_err = e_q;
assign s1_idx = i_q; assign s1_ctx = ram_dout;
assign err_skew = skew_q;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin v_q<=1'b0; id_q<='0; b_q<='0; e_q<=1'b0; i_q<='0; skew_q<=1'b0; end
else begin
v_q<=s0_valid; id_q<=s0_id; b_q<=s0_bytes; e_q<=s0_err; i_q<=s0_idx;
// The context that emerged must be the one fetched for THIS identity.
if (v_q && ctx_live(ram_dout.st) && !id_match(ram_dout.id, id_q)) skew_q <= 1'b1;
end
end
endmoduleimport cplctx_pkg::*;
// SYNTHESIZABLE. THE FLAGSHIP BLOCK. Coverage accumulation, terminal
// handling, and result ownership. Three measured bugs are prevented here:
// retire-on-first-fragment (80.8% of fragments orphaned), free-under-a-
// stalled-result (75.9% corrupted), and absorbing an over-return.
module context_table (
input logic clk,
input logic rst_n,
// allocation
input logic alloc_grant,
input logic [CTX_W-1:0] alloc_idx,
input txn_id_t alloc_id,
input logic [ADDR_W-1:0] alloc_local_addr,
input logic [LEN_W-1:0] alloc_requested,
input logic request_sent,
// matched completion (post-pipeline, §6)
input logic cpl_hit,
input logic [CTX_W-1:0] cpl_idx,
input logic [LEN_W-1:0] cpl_bytes,
input logic cpl_error,
input logic timeout_event, // input, never invented (§8)
input logic [CTX_W-1:0] timeout_idx,
// result handoff
input logic result_ready,
output ctx_t ctx [CONTEXTS],
output logic [CONTEXTS-1:0] live_map,
output txn_id_t live_id [CONTEXTS],
output logic result_valid,
output txn_id_t result_id,
output outcome_e result_outcome,
output logic [LEN_W-1:0] result_bytes,
output logic [CTX_W-1:0] result_idx,
output logic release_req,
output logic err_overrun // sticky
);
ctx_t c_q [CONTEXTS];
logic e_q;
int rep;
always_comb begin
for (int i = 0; i < CONTEXTS; i++) begin
ctx[i] = c_q[i];
live_map[i] = ctx_live(c_q[i].st);
live_id[i] = c_q[i].id;
end
// The FIRST context in REPORT owns the result output.
rep = -1;
for (int i = CONTEXTS-1; i >= 0; i--) if (c_q[i].st == C_REPORT) rep = i;
end
assign result_valid = (rep >= 0);
assign result_id = (rep >= 0) ? c_q[rep].id : '0;
assign result_outcome = (rep >= 0) ? c_q[rep].outcome : ST_NONE;
assign result_bytes = (rep >= 0) ? c_q[rep].received : '0;
assign result_idx = (rep >= 0) ? CTX_W'(rep) : '0;
// RELEASE follows an ACCEPTED result, never the end of the transfer (§7).
assign release_req = result_valid && result_ready;
assign err_overrun = e_q;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int i = 0; i < CONTEXTS; i++) c_q[i] <= '0;
e_q <= 1'b0;
end else begin
if (alloc_grant)
c_q[alloc_idx] <= '{st:C_ALLOC, id:alloc_id, local_addr:alloc_local_addr,
requested:alloc_requested, received:'0, outcome:ST_NONE};
if (request_sent)
for (int i = 0; i < CONTEXTS; i++)
if (c_q[i].st == C_ALLOC) c_q[i].st <= C_WAIT;
if (cpl_hit) begin
automatic logic [LEN_W:0] tot = {1'b0, c_q[cpl_idx].received} + {1'b0, cpl_bytes};
if (cpl_error) begin
// TERMINAL: no further payload accounting (§8).
c_q[cpl_idx].st <= C_TERMINAL;
c_q[cpl_idx].outcome <= ST_CPL_ERROR;
end else if (c_q[cpl_idx].st == C_TERMINAL) begin
// already decided -- ignore payload, do not "un-fail" the read
end else if (tot > {1'b0, c_q[cpl_idx].requested}) begin
e_q <= 1'b1; // REPORT the over-return
c_q[cpl_idx].st <= C_TERMINAL;
c_q[cpl_idx].outcome <= ST_OVERRUN;
end else begin
c_q[cpl_idx].received <= tot[LEN_W-1:0];
// COVERAGE, not packet count (§5). Retiring on the first fragment
// orphaned 80.8% of all fragments in §12 Model 9.
if (tot >= {1'b0, c_q[cpl_idx].requested}) begin
c_q[cpl_idx].st <= C_REPORT;
c_q[cpl_idx].outcome <= ST_OK;
end else c_q[cpl_idx].st <= C_ASSEMBLE;
end
end
if (timeout_event && ctx_live(c_q[timeout_idx].st)
&& (c_q[timeout_idx].st != C_REPORT)) begin
c_q[timeout_idx].st <= C_TERMINAL;
c_q[timeout_idx].outcome <= ST_TIMEOUT;
end
for (int i = 0; i < CONTEXTS; i++)
if (c_q[i].st == C_TERMINAL) c_q[i].st <= C_REPORT; // one path (§8)
// The slot is freed ONLY after the result has been accepted (§7).
if (release_req) c_q[result_idx] <= '0;
end
end
endmoduleimport cplctx_pkg::*;
// VERIFICATION-ONLY. Independent oracle: a dictionary keyed by identity,
// deliberately NOT sharing the DUT's match function, so a shared
// misconception about identity cannot make both agree.
module completion_oracle #(parameter int MAX = 64) (
input logic clk, rst_n,
input logic alloc, txn_id_t alloc_id, input logic [LEN_W-1:0] alloc_bytes,
input logic cpl, txn_id_t cpl_id, input logic [LEN_W-1:0] cpl_bytes,
input logic retire, txn_id_t retire_id, input logic [LEN_W-1:0] retire_bytes,
output logic err_unknown, err_over, err_early, err_leak,
output int outstanding
);
txn_id_t k [MAX];
logic [LEN_W-1:0] want [MAX], got [MAX];
logic used [MAX];
int n;
assign outstanding = n;
function automatic int find(input txn_id_t q);
for (int i = 0; i < MAX; i++)
if (used[i] && (k[i].requester_id == q.requester_id) && (k[i].tag == q.tag)) return i;
return -1;
endfunction
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int i=0;i<MAX;i++) used[i]<=1'b0;
n<=0; err_unknown<=1'b0; err_over<=1'b0; err_early<=1'b0; err_leak<=1'b0;
end else begin
if (alloc) for (int i=0;i<MAX;i++) if (!used[i]) begin
used[i]<=1'b1; k[i]<=alloc_id; want[i]<=alloc_bytes; got[i]<='0; n<=n+1; break;
end
if (cpl) begin
automatic int j = find(cpl_id);
if (j < 0) err_unknown <= 1'b1;
else begin
got[j] <= got[j] + cpl_bytes;
if ((got[j] + cpl_bytes) > want[j]) err_over <= 1'b1;
end
end
if (retire) begin
automatic int j = find(retire_id);
if (j < 0) err_unknown <= 1'b1;
else begin
if (got[j] < want[j]) err_early <= 1'b1; // retired before coverage
used[j] <= 1'b0; n <= n-1;
end
end
end
end
endmoduleClassification: five synthesizable, one verification-only.
Failure — eight. Matching on Tag alone (39.7%). Defaulting an unmatched Completion to slot 0. Retiring on the first fragment (80.8% of fragments orphaned). A synchronous lookup without a matched pipeline (87.5%). Freeing the context under a stalled result (75.9%). Absorbing an over-return. Accumulating payload into a terminal context. And freeing on the error path by a different route than success.
11. Same-Cycle Audit and Assertions
// ==================================================================
// ALLOCATION AND IDENTITY (§3, §4).
// ==================================================================
// P1: a context is allocated only on a request TRANSFER, never an offer.
property p_alloc_on_transfer;
@(posedge clk) disable iff (!rst_n)
alloc_grant |-> alloc_req;
endproperty
// P2: a live identity is UNIQUE. This is what makes "two matches" an
// error rather than a tie (§3).
property p_live_identity_unique;
@(posedge clk) disable iff (!rst_n)
(live_map[0] && live_map[1]) |-> !id_match(live_id[0], live_id[1]);
endproperty
// P3: allocating a duplicate identity is REFUSED and reported.
property p_duplicate_identity_refused;
@(posedge clk) disable iff (!rst_n)
(alloc_req && dup) |-> (!alloc_grant ##1 err_duplicate_identity);
endproperty
// P4: FREE + LIVE == CONTEXTS, always -- a leak is one read away.
property p_context_conservation;
@(posedge clk) disable iff (!rst_n)
($countones(free_q) + $countones(live_map) == CONTEXTS);
endproperty
// ==================================================================
// MATCHING (§3) -- three outcomes, no default.
// ==================================================================
// P5: exactly one outcome per arriving completion.
property p_match_outcome_total;
@(posedge clk) disable iff (!rst_n)
cpl_valid |-> $onehot({hit, unknown_id, ambiguous});
endproperty
// P6: an unmatched completion applies NOTHING -- it does not default to
// slot 0. §12 Model 8: tag-only matching misattributed 39.7%.
property p_unknown_applies_nothing;
@(posedge clk) disable iff (!rst_n)
unknown_id |=> $stable(ctx[0].received);
endproperty
// P7: a hit names a context whose identity really matches.
property p_hit_identity_matches;
@(posedge clk) disable iff (!rst_n)
hit |-> id_match(ctx[sel].id, cpl_id);
endproperty
// P8: same-cycle release and allocate -- the freed slot is allocatable.
property p_same_cycle_release_alloc;
@(posedge clk) disable iff (!rst_n)
(release_req && alloc_req) |=> ($countones(free_q) + $countones(live_map) == CONTEXTS);
endproperty
// P9: an identity is not reused while its context is still live.
property p_no_identity_reuse_while_live;
@(posedge clk) disable iff (!rst_n)
(alloc_grant) |-> !live_map[alloc_idx];
endproperty
// ==================================================================
// COVERAGE (§5) -- bytes, never packets.
// ==================================================================
// P10: received bytes are monotonic.
property p_received_monotonic;
@(posedge clk) disable iff (!rst_n)
ctx_live(ctx[0].st) |-> (ctx[0].received >= $past(ctx[0].received));
endproperty
// P11: a context reports only when coverage is complete or terminal.
// §12 Model 9: retiring on the first fragment orphaned 80.8% of fragments.
property p_report_requires_coverage_or_terminal;
@(posedge clk) disable iff (!rst_n)
((ctx[0].st == C_REPORT) && (ctx[0].outcome == ST_OK))
|-> (ctx[0].received >= ctx[0].requested);
endproperty
// P12: received never exceeds requested; an over-return is REPORTED.
property p_no_overrun_absorbed;
@(posedge clk) disable iff (!rst_n)
ctx_live(ctx[0].st) |-> (ctx[0].received <= ctx[0].requested);
endproperty
// P13: a terminal context accumulates no further payload (§8).
property p_terminal_stops_accounting;
@(posedge clk) disable iff (!rst_n)
(ctx[0].st == C_TERMINAL) |=> $stable(ctx[0].received);
endproperty
// P14: TAG/DATA ALIGNMENT. The context that emerged from the RAM was
// fetched for the identity presented at the same stage. §12 Model 10:
// 87.5% of cycles are misaligned without this pipeline.
property p_lookup_alignment;
@(posedge clk) disable iff (!rst_n)
(s1_valid && ctx_live(s1_ctx.st)) |-> id_match(s1_ctx.id, s1_id);
endproperty
// P15: skew is reported stickily rather than silently applied.
property p_skew_reported;
@(posedge clk) disable iff (!rst_n)
(s1_valid && ctx_live(s1_ctx.st) && !id_match(s1_ctx.id, s1_id)) |=> err_skew;
endproperty
// ==================================================================
// RESULT OWNERSHIP (§7).
// ==================================================================
// P16: the context is released only after the result is ACCEPTED.
// §12 Model 11: freeing earlier corrupted 75.9% of stalled results.
property p_release_after_result_accepted;
@(posedge clk) disable iff (!rst_n)
release_req |-> (result_valid && result_ready);
endproperty
// P17: the result record is STABLE while stalled.
property p_result_stable_under_stall;
@(posedge clk) disable iff (!rst_n)
(result_valid && !result_ready) |=>
(result_valid && $stable(result_id) && $stable(result_outcome)
&& $stable(result_bytes));
endproperty
// P18: every outcome reaches REPORT by the same path -- success and error
// share one release route (§8).
property p_terminal_reaches_report;
@(posedge clk) disable iff (!rst_n)
(ctx[0].st == C_TERMINAL) |=> (ctx[0].st == C_REPORT);
endproperty
// P19: the result identity is the ORIGINAL request's identity.
property p_result_identity_original;
@(posedge clk) disable iff (!rst_n)
result_valid |-> id_match(result_id, ctx[result_idx].id);
endproperty
// P20: a released slot is not reported again -- exactly one result.
property p_single_result_per_transaction;
@(posedge clk) disable iff (!rst_n)
release_req |=> (ctx[$past(result_idx)].st == C_FREE);
endproperty
// ==================================================================
// RESET, RANGE AND NON-INTERFERENCE.
// ==================================================================
// P21: reset invalidates every context and publishes nothing.
property p_reset_clears_contexts;
@(posedge clk)
(!rst_n) |=> (!result_valid && (live_map == '0));
endproperty
// P22: a completion after reset finds no context and is reported unknown,
// rather than matching a stale entry.
property p_no_stale_match_after_reset;
@(posedge clk) disable iff (!rst_n)
($past(!rst_n) && cpl_valid) |-> unknown_id;
endproperty
// P23: every index is in range -- CONTEXTS = 1 included.
property p_index_in_range;
@(posedge clk) disable iff (!rst_n)
(sel < CTX_W'(CONTEXTS)) && (alloc_idx < CTX_W'(CONTEXTS))
&& (result_idx < CTX_W'(CONTEXTS));
endproperty
// P24: releasing a free context is reported, never absorbed.
property p_release_of_free_flagged;
@(posedge clk) disable iff (!rst_n)
(release_req && free_q[release_idx]) |=> err_release_of_free;
endproperty
// P25: outstanding population equals the number of live contexts.
property p_population_consistent;
@(posedge clk) disable iff (!rst_n)
($countones(live_map) == outstanding);
endproperty
// P26: a completion with zero payload bytes does not advance coverage.
property p_zero_byte_no_coverage;
@(posedge clk) disable iff (!rst_n)
(cpl_hit && (cpl_bytes == '0) && !cpl_error) |=> $stable(ctx[cpl_idx].received);
endproperty
// P27: the timeout input is treated as one terminal reason among others --
// no duration is computed here (§8).
property p_timeout_is_terminal_only;
@(posedge clk) disable iff (!rst_n)
(timeout_event && ctx_live(ctx[timeout_idx].st)
&& (ctx[timeout_idx].st != C_REPORT)) |=> (ctx[timeout_idx].st == C_TERMINAL);
endproperty
// P28: the oracle never drives the engine.
property p_oracle_non_functional;
@(posedge clk) disable iff (!rst_n)
$stable({cpl_hit, alloc_grant}) or !$stable(outstanding);
endpropertyTwenty-eight properties. P1–P9 are allocation and identity; P10–P15 are coverage and the pipeline, and P14 is the one that only exists because a real table lives in RAM; P16–P20 are result ownership, which is the race §7 measured at 75.9% and which no functional test without a stalled consumer will ever reach.
12. Measured Behaviour
13. Verification — DV and Mutations
DV, against the independent identity-keyed oracle of §10 — which does not share the DUT's match function: one request, one Completion · one request, many fragments · many Tags completing out of order · the same Tag from two Requester IDs · an unknown identity · a duplicate identity at allocation · a Completion with error status · a Completion with error status after partial data · an over-return · a zero-byte Completion · a stalled result · a stalled result while new Completions arrive · context exhaustion · CONTEXTS = 1 · reset with contexts outstanding · a Completion in the cycle after reset · a timeout event · a timeout on a context already in REPORT.
| # | Mutation | Symptom | Caught by |
|---|---|---|---|
| 1 | Create the context on req_valid | contexts consumed by requests never sent | P1 |
| 2 | Allow a duplicate live identity | two contexts match one Completion | P2, P3 |
| 3 | Match on Tag alone | 39.7% misattributed; data into the wrong buffer | P7 |
| 4 | Default an unmatched Completion to slot 0 | every orphan corrupts slot 0's transaction | P6 |
| 5 | Retire on the first fragment | 80.8% of fragments become unknown-identity errors | P11 |
| 6 | Count Completions instead of bytes | breaks on the first Completer that splits differently | P11 |
| 7 | Interpret Byte Count as this-fragment-only | coverage never completes, or completes early | P11, P12 |
| 8 | Absorb an over-return silently | writes past the end of the destination buffer | P12 |
| 9 | Treat a Completion error as success | a failed read reports complete | P13, P18 |
| 10 | Accumulate payload into a terminal context | a failed read reports a plausible byte count | P13 |
| 11 | Free the context on the error path by a different route | a Tag leak on errors (23.3 §8) | P18, P24 |
| 12 | Ignore the context RAM's latency | 87.5% of lookups paired with the wrong Completion | P14, P15 |
| 13 | Re-read the identity at stage 1 instead of carrying it | the same skew by another route | P14 |
| 14 | Free the context when coverage completes | 75.9% of stalled results corrupted (§12) | P16 |
| 15 | Pulse the result for one cycle | a busy consumer loses the transaction | P17 |
| 16 | Let the result record change while stalled | the consumer reads a different transaction | P17 |
| 17 | Take the result identity from the newest context | results attributed to the wrong request | P19 |
| 18 | Report the same context twice | duplicate results; software double-frees | P20 |
| 19 | Leave contexts valid through reset | a stale Completion matches after restart | P21, P22 |
| 20 | Assume reordered Completions are illegal | correct traffic rejected (13.3 §7) | design review |
| 21 | Serialize all Tags to avoid reordering | throughput collapses to one outstanding read | design review |
| 22 | Let a later fragment overwrite the outcome | an error is erased by the fragment after it | P13 |
| 23 | Advance coverage on a zero-byte Completion | a status-only Completion counts as data | P26 |
| 24 | Truncate the Tag width | two identities alias | P23 |
| 25 | Let the live count underflow | conservation breaks; leaks become invisible | P4, P25 |
| 26 | Release a free context | the pool exceeds its size | P24 |
| 27 | Invent a Completion Timeout duration in this block | an unsourced constant in the datapath (§8) | P27 |
| 28 | Retry the read from the context table | policy in the wrong block (§8) | design review |
| 29 | Use the DUT's match function in the DV oracle | a shared misconception makes both agree | design review |
| 30 | Let a debug counter gate the free logic | the instrument changes the engine | P28 |
Two counterexamples worth stating explicitly.
Mutation 14 is the one this chapter exists for. Freeing the context when coverage completes is obviously right — the transaction is done, the slot is no longer needed. It is wrong because the result has not been delivered yet, and §12 Model 11 measured the context being reallocated before the consumer read it in 75.9% of stalled cases. The consumer then reads metadata belonging to a different request — a result that is well-formed, plausible and attributed to the wrong transaction. No test without a stalled result consumer can reach this, which is why it is on the DV list twice.
Mutation 12 is invisible in RTL simulation with a register-based table and appears the moment the table becomes a RAM. The design is correct at 8 contexts held in flops; at 256 contexts in block RAM the lookup gains a cycle, the identity is re-read from the interface at stage 1, and 87.5% of lookups pair a context with the wrong Completion (§12 Model 10). The symptom is that the FPGA build fails and the simulation passes — which is §14's last scenario, and P14 is what makes it a simulation failure instead.
14. Debugging
Symptom — a read hangs although the analyzer shows every Completion arriving.
The Completions are not matching. Read err_unknown_id: if it is set, the identity the matcher computes differs from the one the Completions carry — most often Tag-only matching (mutation 3) or a truncated Tag width (mutation 24). If it is clear, check coverage: the accumulator may be short by a fragment because Byte Count was interpreted as this-fragment-only (mutation 7).
Symptom — only split reads fail; single-Completion reads are fine. Retirement on the first fragment (§5, mutation 5). A read answered in one Completion is retired correctly by both the right and the wrong logic. The tell is a flood of unknown-identity errors that begins one fragment after each large read — §12 measured 80.8% of fragments orphaned.
Symptom — corruption appears only with more than one outstanding request. Something is using "the current request" instead of the matched context (§3). At one outstanding transaction there is only one, so the wrong design is accidentally right. This is the same signature as Chapter 23.3 §16's QD1-versus-QD8 case, one layer down.
Symptom — one Completion error eventually wedges the engine. The error path releases the context by a different route than success — or not at all (§8, mutation 11). Read the live-context count at rest: a healthy engine returns to zero between transactions. Chapter 23.3 §14 measured the same defect wedging a DMA engine after 14 jobs.
Symptom — the result's metadata belongs to the next request. The context was freed before the result was consumed (§7, mutation 14). Reproduce by stalling the result consumer — without that stall the window does not exist, which is why it survives to the field.
Symptom — unknown-identity errors immediately after reset. Completions for pre-reset transactions are still in flight and the table is empty, so they correctly report unknown. That is the design working; the bug would be a stale context that matches one of them (mutation 19, P22). Distinguish by whether the errors stop once the in-flight Completions drain.
Symptom — the design passes simulation and fails on the FPGA.
Suspect the context RAM's latency (§6, mutation 12). A flop-based table has no lookup delay; a block RAM does, and 87.5% of lookups pair the wrong context. err_skew names it directly — and if the design has no such check, the giveaway is that the failure rate is near 100% rather than intermittent.
15. Misconceptions
"Sending the request is the hard part." Sending is the easy part; remembering is the chapter (§2).
"A Tag identifies the transaction." (Requester ID, Tag) does — Tag alone misattributed 39.7% (§3).
"An unmatched Completion can go to entry 0; it is an error anyway." Then the error corrupts entry 0's transaction (§3, P6).
"One Completion means the read is done." 80.8% of fragments become orphans under that rule (§5).
"Byte Count is how much this Completion carries." It is the remaining count including this one (13.3 §2).
"Count the Completions; the Completer will use the largest payload it may." It is not obliged to (13.3 §1).
"Reordered Completions are a protocol violation." No ordering is implied between different transactions (13.3 §2).
"Serialize the Tags and the problem goes away." So does the throughput (20.5 §5).
"An over-return can be clamped." Clamping writes past the buffer's end; report it (§5, P12).
"An error status still tells you how many bytes arrived." Not once the outcome is terminal — accumulating after that lets a failed read look plausible (§8).
"Free the context as soon as the data is complete." 75.9% of stalled results corrupted (§7).
"A one-cycle result pulse is fine." A busy consumer loses the transaction (23.3 §9).
"A register table and a RAM table behave the same." The RAM answers a cycle late, and 87.5% of lookups pair the wrong Completion (§6).
"The context table should retry failed reads." Retry is the requester's policy, not the table's (§8).
16. Understanding Check
Q1. Endpoint Function 0 and Function 1 each have Tag 5 outstanding. A Completion arrives for Tag 5. Which context receives it? The one whose (Requester ID, Tag) matches the Completion's fields (21.4 §3, §3 here). Matching on Tag alone picks whichever entry the loop found first and misattributed 39.7% of Completions in §12 Model 8 — and misattribution means payload written into the other Function's buffer, not a dropped packet.
Q2. A 512-byte read returns as four 128-byte Completions. Your context is freed on the first. What does the system report? Three unknown-identity errors, and a read that completed with a quarter of its data. §12 Model 9 measured 80.8% of all fragments becoming orphans under that rule. The cascade is the diagnostic clue: errors that begin exactly one fragment after each large read point at retirement logic, not at the fabric.
Q3. Your context table moves from flops to block RAM and the design stops working. What changed? The lookup gained a cycle (§6). The context data now emerges while a different Completion is on the interface — 87.5% of cycles in §12 Model 10. The fix is to pipeline the identity, byte count and status alongside the RAM read so they re-emerge together, and P14 asserts the pairing so this fails in simulation rather than on the board.
Q4. Coverage is complete and the result consumer is stalled. May the context be freed? No (§7). §12 Model 11 measured the slot being reallocated before the consumer read the result in 75.9% of stalled cases, after which the metadata handed out belongs to a different transaction. Either hold the context until the result is accepted (P16), or copy the result into an immutable queue first — but not neither.
Q5. A Completion with an error status arrives for a context that already has 256 of 512 bytes. What happens to the 256 bytes and to the context? The context becomes terminal and accumulates nothing further (§8, P13). The 256 bytes are reported as what arrived, with an error outcome — they are not promoted to a success, and no later fragment may un-fail the transaction (mutation 22). The context is released through the same path a successful one uses (P18), because a release path that depends on the reason is how Chapter 23.3 §8's engine wedged after 14 jobs.
Q6. Why does this chapter treat Completion Timeout as an input rather than computing it?
Because the duration and policy are not something this chapter can source (§1, §8), and inventing a constant in the datapath is worse than omitting one. Structurally the timeout is just one more terminal reason among error status and reset — it enters C_TERMINAL like any other (P27) — which keeps the context table's contract small enough to verify. Chapter 25.7 owns diagnosing timeouts.
17. What's Next
Module 23's data path is now complete. 23.1 placed the boundaries, 23.2 decoded inbound addresses, 23.3 drove the transfers, 23.4 built the packets, and this chapter remembered them.
Chapter 23.6 Design Patterns closes the module by naming what has now repeated too often to be a coincidence. This chapter alone contributed four instances: the three-outcome decoder (§3, after 21.1 and 23.2), allocate/use/free with exactly one terminal event (§8), hold until handshake (§7), and metadata that must travel with its data through a pipeline (§6, after 23.4 §3).
Then Module 24 stops building and starts proving. Every model in this chapter was an independent oracle keyed by identity — and 24.1 is about why that independence is the whole point, and what it takes to organize it across three protocol layers rather than one block.