Completions — The Return Half of a Split Transaction

Chapter 10.1 established the top-level taxonomy of the Transaction Layer: a TLP conveys a Request or a Completion. It owned the first half and named the second.

When a PCIe Request requires a response, what is a Completion, who generates it, and how does the system associate that Completion with the correct outstanding Request?

1. Why a Return Half Exists at All

Chapter 9.6 §14 made an inference this chapter can now justify: a working write proves the entire forward path, so a hanging read is a return-path problem.

That inference rests on an asymmetry. A write hands something over and is done. A read asks — and asking is only half of an operation. Something must come back, carrying the answer and enough identity for the asker to know which of its questions was answered.

The asker is not blocked while it waits. That is the part worth pausing on, because the alternative shaped an earlier generation of buses. A protocol that held the fabric from request to response would make a single slow Completer stall everything behind it. PCIe does not do that: the Request goes, the fabric is free, and the answer arrives later as its own transaction.

The cost of not blocking is state. Something has to remember, for every unresolved operation, what it was and where its answer goes.

That state is the subject of this chapter, and §10's RTL is where it becomes real.

2. The Verified Definitions

Read the Completion definition slowly, because four separate facts are packed into one sentence.

"Terminate, or partially terminate." A Completion may finish a Sequence or only advance it. That single clause is why §7 exists and why §10's table cannot free an entry on just any Completion.

"Always corresponds to a preceding Request." There is no such thing as an unsolicited Completion. One arriving that matches nothing is not a new operation — it is an error condition, and §11 gives it an explicit output rather than dropping it.

"In some cases includes data." Whether a Completion carries a payload depends on what was asked, not on the device. §8 develops this conceptually; the taxonomy is Chapter 13.1's.

"A single Request and zero or more Completions." Zero is in that definition for a reason — and it is the whole of Chapter 10.3.

3. Requester and Completer Are Roles

Chapter 10.1 §4 established this and it is worth one paragraph rather than a section, because the reflex it corrects is strong.

The Completer is "the logical device addressed by a Request." Not "the Endpoint." Not "the device that is not the host." Whichever device a Request is addressed to is the Completer for that Sequence, and the same device is a Requester for Sequences it originates.

So a Root Complex generates Completions too — every time an Endpoint reads host memory, the Root Complex is the addressed device and produces the returning transaction. And an Endpoint receives Completions, for exactly the same reason. A design that can only produce Completions, or only consume them, has implemented half a Transaction Layer.

Which is why §10's outstanding table is not "an Endpoint thing" or "a host thing." Any device that issues Requests requiring Completion needs one, sized for how many operations it intends to have outstanding.

4. The Split-Transaction Model

The mechanism, stated as a lifetime rather than as a diagram:

Request constructed and launched
   → Requester retains context for the operation
   → the fabric is free; other Requests may be issued
   → the Completer services the Request
   → a Completion returns, carrying the Transaction ID
   → the Requester matches it against retained context
   → the operation resolves; the context is released

Two properties of that sequence do the work.

The fabric is not held synchronously while a Request waits for its Completion. Subject to available outstanding state, ordering constraints, flow-control resources and local implementation limits, the Requester may have other work in progress while earlier Requests remain unresolved. That is what "split" means — the transaction is split into two independently transported halves, and the interval between them belongs to nobody. What the Requester may actually do in that interval is bounded — Chapter 10.4 §4 and §13 develop the limits.

The Completion carries identity, not position. It is matched by the Transaction ID it carries, not by the order in which it arrived. §5 shows why that distinction is not academic.

5. Two Requests, Two Completions, Returning Late

The figure's argument is the crossing. Request A leaves first and is answered second. Nothing in the Requester's issue order predicts the return order, so nothing in the Requester's implementation may depend on it.

And notice what the Requester is doing between messages 1 and 6. Issuing more work. That is the entire economic justification for the split-transaction model, and it is also the reason the outstanding table has more than one entry.

6. Correlation — What Identifies a Transaction

The specification is precise about this and the precision is useful.

Two components, combined:

• The Requester ID — the Bus/Device/Function triple that uniquely identifies the Requester (Chapter 10.1 §4). It steers the Completion back to the right device.
• The Tag — "a number assigned to a given Non-posted Request to distinguish Completions for that Request from other Requests." It distinguishes which operation within that device.

"Requester ID and Tag combined form a global identifier for each Transaction within a Hierarchy," and that Transaction ID "is included with all Requests and Completions."

7. One Request Does Not Always Mean One Completion

The Completion definition said "terminate, or partially terminate." The Read Request rules make that concrete: "Read Requests may be completed with one, or in some cases, multiple Completions."

The mental model to take from this is narrow and important:

One logical Request does not always imply exactly one Completion packet. The Sequence ends when the Request has been fully completed, which may take more than one returning transaction.

The design consequence is a single line in §10's contract, and it is the one people get wrong: an outstanding entry must be released when the Sequence ends, not when a Completion arrives. A table that frees on every Completion will free early on the first fragment of a multi-Completion response, and then the remaining fragments arrive as orphans while the entry has already been recycled.

What determines whether a Request is split, and how the fragments relate to one another, is Chapter 13.3's. The boundary parameter, the alignment rules, and the guarantees between fragments are all its material. §11's model carries an abstract cpl_last input for exactly this reason: it represents the fact that something determines the final fragment, without claiming to know what.

8. With Data, or Status Only

The normative statement is conditional: Completions include a header "that, for some types of Completions, will be followed by some number of DW of data."

Conceptually, and only conceptually here:

The rule underneath both rows: what a Completion carries follows from what the original Request asked, not from what the Completer felt like sending. A Completer answering a read produces the data; a Completer answering a non-posted write confirms the outcome.

Everything sharper than that is deferred. The names for the two shapes, when each is used, and the encoding that distinguishes them are Chapter 13.1's subject. §11's model carries a has_data flag and does not name the categories, because naming them here would consume that chapter to save a sentence.

The one thing worth stating firmly. A Completion carrying no data is not a degenerate or failed Completion — it is the correct and complete answer to a Request that did not ask for data. Treating "no payload" as an error is a real bug and it is in §16's list.

9. Microarchitecture — Retained Context

The Requester's problem is not the packet. It is the bookkeeping.

What must be retained per unresolved operation, at minimum:

10. RTL — The Outstanding Request Table

// SYNTHESIZABLE. Tracks operations that are outstanding — launched, not yet
// resolved. Small-N teaching structure: an active bitmap plus a context array.
// The requirement (retain context until the Sequence resolves) is NORMATIVE.
// The key width, the allocation policy, and the structure are ILLUSTRATIVE.
module outstanding_table #(
  parameter int ENTRIES = 8,
  parameter int CTX_W   = 16,
  // Derived. Never zero-width, including at ENTRIES == 1.
  parameter int IDX_W   = (ENTRIES <= 1) ? 1 : $clog2(ENTRIES)
) (
  input  logic              clk,
  input  logic              rst_n,
 
  // ---- Allocate: a Request that requires Completion is being launched ----
  input  logic              alloc_req,
  input  logic [CTX_W-1:0]  alloc_ctx,
  output logic              alloc_gnt,
  output logic [IDX_W-1:0]  alloc_id,
  output logic              table_full,
 
  // ---- Resolve: a Completion's correlation key has arrived --------------
  // Combinational lookup. Reading the table does NOT change it.
  input  logic              resolve_valid,
  input  logic [IDX_W-1:0]  resolve_id,
  output logic              resolve_hit,
  output logic [CTX_W-1:0]  resolve_ctx,
 
  // ---- Free: the Sequence has ended (section 7) -------------------------
  // NOT "a Completion arrived" — the LAST one.
  input  logic              free_valid,
  input  logic [IDX_W-1:0]  free_id,
 
  // ---- Explicit error indications, rather than silent misbehaviour ------
  output logic              unknown_completion,  // resolve of an inactive entry
  output logic              free_error,          // free of an inactive entry
  output logic [IDX_W:0]    active_count         // for the conservation check
);
 
  generate
    if (ENTRIES < 1) $error("ENTRIES must be at least 1");
    if (CTX_W  < 1)  $error("CTX_W must be at least 1");
  endgenerate
 
  logic [ENTRIES-1:0] active_q;
  logic [CTX_W-1:0]   ctx_q [ENTRIES];
  logic [IDX_W:0]     count_q;   // 0..ENTRIES, so IDX_W+1 bits
 
  // ---- First-free selection -------------------------------------------
  // Iterating DOWNWARD means index 0's assignment lands last and therefore
  // wins. Written this way deliberately: the naive upward loop makes the
  // HIGHEST free index win, which contradicts prose that says "lowest free"
  // and produces an allocation order nobody intended.
  // The policy itself is implementation-defined; only the code matching the
  // prose is not optional.
  logic             free_found;
  logic [IDX_W-1:0] free_idx;
  always_comb begin
    free_found = 1'b0;
    free_idx   = '0;
    for (int i = ENTRIES-1; i >= 0; i--) begin
      if (!active_q[i]) begin
        free_found = 1'b1;
        free_idx   = IDX_W'(i);
      end
    end
  end
 
  assign table_full = !free_found;
  assign alloc_gnt  = alloc_req && free_found;
  assign alloc_id   = free_idx;
 
  // ---- Lookup ----------------------------------------------------------
  // Against the CURRENT active set. A key that is not active resolves
  // nothing and is reported, because a Completion "always corresponds to a
  // preceding Request" (section 2) — so one that matches nothing is an error
  // condition, not a new operation.
  assign resolve_hit        = resolve_valid &&  active_q[resolve_id];
  assign unknown_completion = resolve_valid && !active_q[resolve_id];
  assign resolve_ctx        = ctx_q[resolve_id];
 
  assign free_error   = free_valid && !active_q[free_id];
  assign active_count = count_q;
 
  wire do_free  = free_valid && active_q[free_id];
  wire do_alloc = alloc_gnt;
 
  // SAME-CYCLE POLICY, and it falls out structurally rather than by rule:
  // free_idx is chosen from entries that are INACTIVE THIS CYCLE, and
  // free_id names an entry that is ACTIVE this cycle. They can never be the
  // same index, so an entry freed in this cycle becomes allocatable no
  // earlier than the next cycle. That is the conservative contract, and it
  // makes a late Completion for the freed key resolve nothing rather than
  // resolving a brand-new operation.
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      active_q <= '0;
      count_q  <= '0;
      for (int i = 0; i < ENTRIES; i++) ctx_q[i] <= '0;
    end else begin
      if (do_free)  active_q[free_id]  <= 1'b0;
      if (do_alloc) begin
        active_q[alloc_id] <= 1'b1;
        ctx_q[alloc_id]    <= alloc_ctx;
      end
 
      case ({do_alloc, do_free})
        2'b10:   count_q <= count_q + 1'b1;
        2'b01:   count_q <= count_q - 1'b1;
        default: count_q <= count_q;          // both or neither: unchanged
      endcase
    end
  end
 
endmodule

Classification: synthesizable.

Semantics — every dimension:

What it teaches — four things:

1. A table, not a queue. Indexed by correlation key, so §4's out-of-order return is a non-event. A FIFO would be smaller and would corrupt on the first reordering.
2. Reading is not resolving. Lookup is combinational and side-effect-free; freeing is a separate, explicit input driven by end-of-Sequence (§7). Fusing them is the bug that breaks multi-Completion responses.
3. Unknown keys get an output. A Completion matching nothing is a real condition with a real cause — a late arrival after a free, a corrupted key, a reused ID. Silently ignoring it makes §15's second scenario undiagnosable.
4. The same-cycle race is answered structurally, not by a rule in the prose. The allocator scans inactive entries and the free targets an active one; they cannot collide. That is stronger than a comment saying they must not.

Deliberately simplified: a bitmap plus a flat array, appropriate for a handful of entries; no per-entry accumulation state for a multi-Completion Sequence beyond the free-on-last contract; no timeout; a single allocate and a single free port per cycle.

Production implication: a design with many outstanding operations uses RAM, a CAM, or a distributed structure rather than a flat array; accumulates partial results for Sequences that span several Completions (Chapter 13.3); implements the Completion Timeout mechanism (Chapter 25.7); and maps its internal index onto the standardised Tag (Module 11).

11. RTL — Routing a Returning Completion

// SYNTHESIZABLE. Matches a decoded Completion against the outstanding table
// and delivers the result to the local consumer the RETAINED CONTEXT names.
// Routing-by-retained-context is the architectural point (section 9); the
// metadata interface and the destination set are illustrative.
module completion_router #(
  parameter int ENTRIES = 8,
  // Derived. Declared here rather than in the body because the port list
  // needs them.
  parameter int IDX_W   = (ENTRIES <= 1) ? 1 : $clog2(ENTRIES),
  parameter int TOK_W   = 14,
  parameter int DEST_W  = 2,
  parameter int CTX_W   = DEST_W + TOK_W
) (
  input  logic              clk,
  input  logic              rst_n,
 
  // ---- Decoded Completion metadata in ----------------------------------
  input  logic              cpl_valid,
  output logic              cpl_ready,
  input  logic [IDX_W-1:0]  cpl_id,        // internal correlation key
  input  logic              cpl_has_data,
  input  logic              cpl_last,      // terminates the Sequence (section 7)
  input  logic              cpl_error,     // ABSTRACT status — Chapter 13.2 owns the real one
  input  logic [31:0]       cpl_data,
 
  // ---- Outstanding table ------------------------------------------------
  output logic              lookup_valid,
  output logic [IDX_W-1:0]  lookup_id,
  input  logic              lookup_hit,
  input  logic [CTX_W-1:0]  lookup_ctx,
  output logic              free_valid,
  output logic [IDX_W-1:0]  free_id,
 
  // ---- Local consumers, selected by the RETRIEVED CONTEXT ---------------
  output logic              ld_valid,
  input  logic              ld_ready,
  output logic [TOK_W-1:0]  ld_token,
  output logic [31:0]       ld_data,
 
  output logic              cfg_valid,
  input  logic              cfg_ready,
  output logic [TOK_W-1:0]  cfg_token,
  output logic [31:0]       cfg_data,
 
  output logic              err_valid,
  input  logic              err_ready,
  output logic [TOK_W-1:0]  err_token,
 
  // A Completion whose key matches no outstanding entry. One-cycle pulse.
  output logic              orphan
);
 
  // The retained context is DEST_W destination bits above TOK_W handle bits.
  localparam logic [DEST_W-1:0] DEST_LOAD = 2'd0;
  localparam logic [DEST_W-1:0] DEST_CFG  = 2'd1;
 
  // ---- One-deep result stage, so a Completion is never dropped because a
  // local consumer happens to be stalled (section 12).
  logic                 res_valid_q;
  logic [DEST_W-1:0]    res_dest_q;
  logic [TOK_W-1:0]     res_token_q;
  logic [31:0]          res_data_q;
  logic                 res_err_q;
 
  // Destination select, from the RETAINED context — never from the Completion.
  wire [DEST_W-1:0] ctx_dest  = lookup_ctx[CTX_W-1 -: DEST_W];
  wire [TOK_W-1:0]  ctx_token = lookup_ctx[TOK_W-1:0];
 
  // Destination decode. DEST_W is wider than the set of destinations this
  // model represents, so a retained code outside that set is possible and
  // must be given a defined home rather than falling through a mux default.
  wire dest_is_load = (res_dest_q == DEST_LOAD);
  wire dest_is_cfg  = (res_dest_q == DEST_CFG);
  wire dest_invalid = !(dest_is_load || dest_is_cfg);
 
  // An invalid retained destination is routed to the ERROR sink. A result
  // with nowhere legitimate to go is a fault, not something to drop, and
  // routing it explicitly is what makes the one-hot contract below true by
  // construction instead of by assumption about the context's contents.
  wire to_err = res_err_q || dest_invalid;
 
  // Which consumer is being offered this result, and is it taking it?
  // NOTE: sel_ready uses the SAME decode as the output valids below, so no
  // consumer's ready can drain a result that consumer was never offered.
  wire sel_ready = to_err       ? err_ready
                 : dest_is_load ? ld_ready
                                : cfg_ready;
  wire res_take  = res_valid_q && sel_ready;
 
  // READY DEPENDS ON STATE AND THE SELECTED CONSUMER ONLY — never on
  // cpl_valid, so there is no combinational path from the Completion's valid
  // to its own ready, and an orphan is consumed rather than deadlocking.
  assign cpl_ready = !res_valid_q || res_take;
 
  wire cpl_xfer = cpl_valid && cpl_ready;
 
  assign lookup_valid = cpl_valid;
  assign lookup_id    = cpl_id;
 
  wire cpl_matched = cpl_xfer &&  lookup_hit;
  assign orphan    = cpl_xfer && !lookup_hit;
 
  // FREE ON END OF SEQUENCE, not on arrival of any Completion (section 7).
  assign free_valid = cpl_matched && cpl_last;
  assign free_id    = cpl_id;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      res_valid_q <= 1'b0;
      res_dest_q  <= DEST_LOAD;
      res_token_q <= '0;
      res_data_q  <= 32'h0;
      res_err_q   <= 1'b0;
    end else begin
      // Drain first, then refill, so a taken result and a newly matched
      // Completion can occupy the same cycle.
      if (res_take) res_valid_q <= 1'b0;
 
      if (cpl_matched) begin
        res_valid_q <= 1'b1;
        res_dest_q  <= ctx_dest;      // from the TABLE
        res_token_q <= ctx_token;     // from the TABLE
        res_err_q   <= cpl_error;
        res_data_q  <= cpl_has_data ? cpl_data : 32'h0;
      end
    end
  end
 
  assign ld_valid  = res_valid_q && !to_err && dest_is_load;
  assign cfg_valid = res_valid_q && !to_err && dest_is_cfg;
  assign err_valid = res_valid_q &&  to_err;
 
  assign ld_token  = res_token_q;
  assign ld_data   = res_data_q;
  assign cfg_token = res_token_q;
  assign cfg_data  = res_data_q;
  assign err_token = res_token_q;
 
endmodule

Classification: synthesizable.

What it teaches — four things:

1. Destination comes from the table. res_dest_q and res_token_q are loaded from lookup_ctx. Nothing carried by the Completion selects a local consumer, which is §9's rule expressed as a data path.
2. Free is gated on cpl_last, not on arrival. A non-final Completion is routed and the entry stays. This is the one line that makes a multi-Completion Sequence survivable, and it costs nothing to get right at design time and a great deal to retrofit.
3. An orphan is consumed and reported. It does not stall the interface and it does not free anything. Both matter: stalling on an orphan turns a recoverable anomaly into a hang, and freeing on one corrupts an unrelated live entry.
4. Ready never depends on cpl_valid. The discipline of Chapter 9.6 §10, applied to the return path, and the reason an orphan cannot deadlock the router.
5. The destination decode is exhaustive. DEST_W can hold codes this model does not define, so a retained value outside the represented set is routed to the error sink rather than falling through a mux default. A sel_ready that decoded the destination differently from the output valids would let one consumer's ready drain a result it was never offered — a silent loss with no indication anywhere, and the reason P10b and P10c exist.

Deliberately simplified: one 32-bit data beat per Completion rather than a stream; three destinations; an abstract single-bit error rather than a status field; a one-deep result stage; and no accumulation across the Completions of one Sequence.

Production implication: a real return path carries a data stream with its own flow control, decodes the actual Completion header (Module 11), interprets the real status (Chapter 13.2), accumulates fragments of a split response (Chapter 13.3), and reports orphaned Completions through the error mechanisms later modules own.

12. Why the Result Stage Exists

The Completion arrives when the Completer and the fabric decide. The local consumer is ready when it decides. Those are unrelated events, and the gap between them is a design decision that must be made explicitly.

The unacceptable answer is to drop the Completion. It carries the only answer that operation will ever get, and it corresponds to a Request the Requester itself issued. Discarding it converts a working transaction into a hang that will eventually surface as a timeout, far from its cause.

So the router holds it. res_valid_q stays asserted, every field held with it stays stable, and cpl_ready deasserts — pushing back on the Completion path rather than losing data.

And back-pressure on the return path is not free. A local consumer that stops draining eventually stalls the router, which stalls the Completion input, which — in a real design — stops Completion credits being returned and ultimately throttles the Completer. Chapter 10.4 §12 develops that chain, because it is the mechanism by which a slow local consumer becomes a system-wide throughput limit. Module 16 owns the credit half of it.

13. Assertions

// SVA over outstanding_table and completion_router. Implementation
// invariants for THESE designs plus the normative requirement they serve —
// not claims about any PCIe field. Every property refers to explicit RTL.
 
// ENVIRONMENT ASSUMPTIONS. The Completion source owes valid-stability, and
// the free port must not be driven for an entry the environment did not
// observe as active; without these no ownership property can hold.
assume property (@(posedge clk) disable iff (!rst_n)
  (cpl_valid && !cpl_ready) |=> cpl_valid);
assume property (@(posedge clk) disable iff (!rst_n)
  (cpl_valid && !cpl_ready) |=> ($stable(cpl_id) && $stable(cpl_data)
                                 && $stable(cpl_last) && $stable(cpl_error)));
 
// OWNERSHIP — P1: allocation never overwrites an active entry. The property
// that makes a retained context trustworthy for its whole lifetime.
property p_alloc_never_overwrites;
  @(posedge clk) disable iff (!rst_n)
  alloc_gnt |-> !active_q[alloc_id];
endproperty
a_alloc_safe : assert property (p_alloc_never_overwrites);
 
// OWNERSHIP — P2: an active entry's context is immutable while it stays
// active. Written per index with a generate loop, because the claim is about
// every entry rather than about whichever one is being allocated.
generate for (genvar e = 0; e < ENTRIES; e++) begin : g_ctx_stable
  property p_ctx_stable_while_active;
    @(posedge clk) disable iff (!rst_n)
    (active_q[e] && !(do_alloc && (alloc_id == e[IDX_W-1:0])))
      |=> $stable(ctx_q[e]);
  endproperty
  a_ctx_stable : assert property (p_ctx_stable_while_active);
end endgenerate
 
// UNIQUENESS — P3: an entry freed this cycle is not allocated this cycle.
// The same-cycle contract of section 10, asserted rather than assumed so an
// edit to the allocation policy cannot quietly break it.
property p_no_same_cycle_reuse;
  @(posedge clk) disable iff (!rst_n)
  (do_alloc && do_free) |-> (alloc_id != free_id);
endproperty
a_no_reuse_same_cycle : assert property (p_no_same_cycle_reuse);
 
// LEGALITY — P4: a Completion whose key is not active resolves nothing and
// is reported. "A Completion always corresponds to a preceding Request" —
// so one that matches nothing must not be treated as an operation.
property p_unknown_resolves_nothing;
  @(posedge clk) disable iff (!rst_n)
  unknown_completion |-> (!resolve_hit && !free_valid);
endproperty
a_unknown_inert : assert property (p_unknown_resolves_nothing);
 
// LEGALITY — P5: freeing an inactive entry has no effect and is reported.
// Double-free protection, made structural.
property p_free_of_inactive_inert;
  @(posedge clk) disable iff (!rst_n)
  free_error |-> !do_free;
endproperty
a_no_double_free : assert property (p_free_of_inactive_inert);
 
// CORRECTNESS — P6: a hit returns the context that was written for that key.
// Catches an indexing error between the write and read ports, which delivers
// correct data to the wrong consumer — section 15's first scenario.
property p_lookup_returns_written_ctx;
  @(posedge clk) disable iff (!rst_n)
  resolve_hit |-> (resolve_ctx == ctx_q[resolve_id]);
endproperty
a_lookup_exact : assert property (p_lookup_returns_written_ctx);
 
// CONSERVATION — P7: active_count equals the number of active entries. The
// invariant "allocated minus freed equals active", checkable in one line.
property p_count_matches_bitmap;
  @(posedge clk) disable iff (!rst_n)
  active_count == $countones(active_q);
endproperty
a_count_exact : assert property (p_count_matches_bitmap);
 
// CONSERVATION — P8: the table reports full exactly when it is full.
property p_full_is_exact;
  @(posedge clk) disable iff (!rst_n)
  table_full == (active_count == ENTRIES[IDX_W:0]);
endproperty
a_full_exact : assert property (p_full_is_exact);
 
// CONSERVATION — P9: a full table blocks allocation. The back-pressure that
// stops a Requester launching an operation it could not track.
property p_full_blocks_alloc;
  @(posedge clk) disable iff (!rst_n)
  table_full |-> !alloc_gnt;
endproperty
a_full_blocks : assert property (p_full_blocks_alloc);
 
// ROUTING — P10: exactly one local consumer is offered a held result. True
// by construction because the destination decode is exhaustive: load, cfg,
// or the error sink, with no fourth outcome.
property p_result_onehot;
  @(posedge clk) disable iff (!rst_n)
  res_valid_q |-> $onehot({ld_valid, cfg_valid, err_valid});
endproperty
a_result_onehot : assert property (p_result_onehot);
 
// ROUTING — P10b: a retained destination code this model does not represent
// goes to the error sink. Without this the code would fall through a mux
// default, and a result could be drained by a consumer's ready while no
// consumer's valid was asserted — silently losing it.
property p_invalid_dest_to_error;
  @(posedge clk) disable iff (!rst_n)
  (res_valid_q && dest_invalid) |-> (err_valid && !ld_valid && !cfg_valid);
endproperty
a_invalid_dest_routed : assert property (p_invalid_dest_to_error);
 
// ROUTING — P10c: no result is taken by a consumer that was not offered it.
// The property that fails if sel_ready and the output valids ever decode the
// destination differently.
property p_take_matches_offer;
  @(posedge clk) disable iff (!rst_n)
  res_take |-> ((ld_valid && ld_ready) || (cfg_valid && cfg_ready)
                || (err_valid && err_ready));
endproperty
a_take_is_offered : assert property (p_take_matches_offer);
 
// ROUTING — P11: the destination came from the table, not from the
// Completion. Asserted against the values captured at match time.
property p_dest_from_context;
  @(posedge clk) disable iff (!rst_n)
  cpl_matched |=> (res_dest_q  == $past(ctx_dest)
                && res_token_q == $past(ctx_token));
endproperty
a_dest_from_table : assert property (p_dest_from_context);
 
// STABILITY — P12: a held result is stable until its consumer takes it. The
// executable form of "a Completion is never dropped because a local consumer
// is stalled".
property p_result_stable_under_stall;
  @(posedge clk) disable iff (!rst_n)
  (res_valid_q && !sel_ready) |=> (res_valid_q && $stable(res_dest_q)
                                   && $stable(res_token_q)
                                   && $stable(res_data_q)
                                   && $stable(res_err_q));
endproperty
a_result_held : assert property (p_result_stable_under_stall);
 
// CONSERVATION — P13: one matched Completion produces one result offer.
property p_match_produces_result;
  @(posedge clk) disable iff (!rst_n)
  cpl_matched |=> res_valid_q;
endproperty
a_match_fills : assert property (p_match_produces_result);
 
// CONSERVATION — P14: the result stage empties on handoff unless refilled,
// so a result is never delivered twice.
property p_result_not_duplicated;
  @(posedge clk) disable iff (!rst_n)
  (res_take && !cpl_matched) |=> !res_valid_q;
endproperty
a_no_dup_result : assert property (p_result_not_duplicated);
 
// SEQUENCE — P15: only a terminating Completion frees. The one-line contract
// that makes a multi-Completion Sequence survivable (section 7).
property p_free_only_on_last;
  @(posedge clk) disable iff (!rst_n)
  free_valid |-> (cpl_last && lookup_hit);
endproperty
a_free_on_last : assert property (p_free_only_on_last);
 
// RESET — P16: reset clears all outstanding ownership in this block.
// LOCAL BLOCK RESET behaviour only — real Function and Link reset semantics
// are owned by later reset and LTSSM chapters.
property p_reset_clears_ownership;
  @(posedge clk) !rst_n |=> (active_count == '0);
endproperty
a_reset_clears : assert property (p_reset_clears_ownership);
 
// SAFETY — P17: no interface output is ever unknown.
property p_outputs_never_unknown;
  @(posedge clk) disable iff (!rst_n)
  !$isunknown({alloc_gnt, table_full, resolve_hit, unknown_completion,
               free_error, cpl_ready, orphan, ld_valid, cfg_valid, err_valid});
endproperty
a_no_x : assert property (p_outputs_never_unknown);

P11 is the chapter's architectural claim made checkable. "The destination comes from the table" is a statement about where a signal originates, and P11 pins it by comparing what was captured against what the table produced at match time. A router that steered on a Completion-carried field would fail it on the first Completion whose field disagreed with the retained context — which is every Completion, in a correct system, because the Completion has no reason to know.

P15 is the property that would be omitted by someone who had not read the Completion definition carefully. Freeing on every Completion satisfies P4, P5, P7 and P9 and still corrupts a multi-Completion Sequence: the first fragment frees the entry, the second arrives as an orphan, and the operation silently loses data. P15 is one line and it is the difference.

P3 asserts a property the structure already guarantees, on purpose. §10's allocator picks from inactive entries and the free targets an active one, so same-cycle reuse is unrepresentable today. The property exists because that guarantee is a consequence of the allocation policy, and an edit that made the allocator smarter — reusing an entry being freed this cycle to avoid a bubble — would break it silently. P3 makes the optimisation fail loudly instead.

P7 and P8 are the conservation pair. P7 says the counter and the bitmap agree; P8 says fullness is exact. Together they make "allocated − freed = active" checkable without a shadow model, and they catch the classic leak where an entry is marked inactive but never counted down, so the table reports space it does not have.

14. Verification

Monitors observe: every allocate with its context and granted key; every lookup with its hit and returned context; every free; both error outputs; the active count; the Completion interface; and all three consumer handshakes.

Correlation basics

• One Request, one Completion. Allocate, launch, resolve, free. Verify the context returned matches the context written and the entry becomes reusable the next cycle.
• Several outstanding operations, Completions in issue order. Verify each resolves its own entry.
• Several outstanding operations, Completions in the reverse order. The test that a queue-based design fails, and the reason §4 exists. Verify correlation is unaffected by arrival order.
• Interleaved allocate and resolve. Allocate while a Completion for a different entry is resolving. Verify both proceed and the count is right.
• Contexts that differ only in destination. Two entries with the same token but different destinations. Verify each result reaches the correct consumer — this catches a router that ignores ctx_dest.

Sequence lifetime

• A Completion with cpl_last low. Verify the result is routed, the entry stays active, and active_count does not change (P15).
• Several non-final Completions followed by a final one. Verify the entry is freed exactly once, on the last.
• A final Completion for an entry that has received no earlier fragments. The single-Completion case, verified as the degenerate case of the same rule.

Negative tests

• Unknown correlation key. Present a Completion whose key is inactive. Verify unknown_completion pulses, orphan pulses, nothing is freed, no consumer is offered anything, and the interface does not stall (P4).
• Duplicate Completion. Resolve and free an entry, then present the same key again. Verify it is treated as an orphan and does not resolve the entry that has since been allocated to a different operation. Run this test both with the key reallocated and with it still free, because only the first version can corrupt anything.
• Double free. Drive free_valid twice for the same key. Verify free_error pulses on the second and the count does not go below the true value (P5).
• Free of a never-allocated key. Verify the same.
• Allocate against a full table. Verify alloc_gnt stays low (P9) and no entry is disturbed.
• Completion arriving while every consumer is stalled. Verify the result is held stable and cpl_ready deasserts rather than the Completion being dropped (P12).
• Reset with entries outstanding. Verify all ownership clears (P16), and that a Completion arriving after reset for a pre-reset key is an orphan rather than resolving a new entry.

Parameter corners

• ENTRIES = 1. Verify IDX_W is 1, no zero-width vector exists, and full/empty behave — the degenerate case where allocation and freeing must interleave perfectly.
• A non-power-of-two ENTRIES, such as 3 or 5. Verify table_full is exact and no index above ENTRIES-1 is ever produced.
• ENTRIES large enough that the count needs its extra bit. Verify active_count reaches ENTRIES without wrapping.

Coverage should include: every entry index allocated at least once; the table at empty, one-below-full, and full; Completions in issue order, reverse order, and interleaved; cpl_last both set and clear on matched Completions; every destination in the context; all three consumers stalled while a result is held; and each error output asserted.

15. Debugging

Symptom: a read returns valid data, but the wrong local consumer receives it

Classify it first, because the instinct is to suspect the data path and the data is fine. The bytes are correct; the destination is wrong. That is retained context, not transport.

The candidates, in order:

1. The lookup returned the wrong entry's context. An indexing error between the write and read ports of the context array delivers entry j's destination for entry i's key. P6 catches it; by hand, compare the context written at allocate against the context returned at resolve for the same key.
2. The correlation key was reused while the first operation was still outstanding. The late Completion for operation 1 resolves the entry now owned by operation 2 and is delivered to operation 2's consumer. Chapter 10.4 §7 owns this hazard; here the signature is that the wrong consumer receives data that belongs to an operation it never issued.
3. The entry was freed early and reallocated. Same outcome as (2) by a different route — a design that frees on any Completion rather than on the last (P15) does exactly this on a multi-Completion Sequence.
4. The router steered on the Completion rather than on the context. P11's failure. The signature is distinctive: the destination correlates with something about the Completion rather than with the operation that was issued.
5. The context was overwritten while active. An allocation that landed on a live entry — P1's failure.

The observation that separates all five: record the context at allocate and compare it against the context used for routing. They must be identical for a given key across its whole lifetime. If they differ, the fault is in the table (1, 3, 5); if they agree and the delivery is still wrong, the fault is in the router (4); and if the key is being used by two operations at once, it is (2).

Symptom: a Completion arrives and the Requester reports an unknown key

unknown_completion is not a mysterious failure — it is a Completion for an operation this Requester no longer believes it has. There are only a few ways to arrive there.

1. The entry was freed too early. Freeing on arrival rather than on end-of-Sequence (§7). The signature is that the first Completion of a multi-Completion response resolves normally and the rest are orphans.
2. The key was recycled and the late Completion missed its window. If the key had already been reallocated, this would have been the previous scenario instead — so an orphan here means the key happened to be free, which is luck rather than safety.
3. The decoded key is wrong. The metadata feeding this module came from a header decode; an error there produces a key that matches nothing. Compare the decoded key against the key the corresponding Request carried.
4. A local reset or flush cleared the table while an operation was in flight. The Completion is entirely legitimate; the Requester forgot. This is why §13's P16 is labelled as local-block behaviour — clearing local state does not un-issue a Request that is already on the fabric, and real reset semantics are owned by later chapters.
5. The Completion genuinely was not for this Requester. Rare, and it points upstream at routing rather than at this block.

What makes the orphan output worth its logic. Without it, all five of these are silence, and silence at the return path surfaces later as a timeout at a completely different level. An explicit pulse converts an invisible anomaly into a timestamped event with a key attached, which is the difference between a two-hour investigation and a two-minute one.

Symptom: the outstanding table fills and traffic stops

The table filling is a symptom of entries not being freed, and freeing depends on Completions returning. So this is a return-path investigation that looks like a request-path stall.

The ladder:

1. Are Completions arriving at all? If not, this is not a table problem — the operations really are unresolved, and the investigation moves to the Completer, the fabric, or the return path. Watch cpl_valid.
2. Are arriving Completions matching? If orphan is pulsing, entries are not being freed because the Completions cannot find them — the previous scenario, with a full table as the downstream consequence.
3. Are matched Completions freeing? If cpl_last never asserts, every Completion routes and nothing frees. A cpl_last stuck low is a leak with a completely uniform signature: the count rises monotonically and never falls.
4. Is the result stage draining? If a local consumer has stopped taking results, res_valid_q stays high, cpl_ready deasserts, Completions stop being accepted, and entries stop being freed. The root cause is a stalled consumer and the visible symptom is a full table three steps away.
5. Is the request rate simply exceeding the resolution rate? A table at capacity with entries being freed steadily is not a bug — it is the design operating at its outstanding limit, and it is a performance observation. Chapter 10.4 §13 develops what that limit costs.

The distinguishing observation: watch active_count over time. Monotonically rising means a leak (steps 2 or 3). Rising to ENTRIES and staying flat with frees still occurring means step 5 — saturation, not corruption. Those two look identical at the request interface and need opposite responses.

16. Common Misconceptions

• "Every Request gets exactly one Completion." A Sequence is "a single Request and zero or more Completions." Posted Requests get none (Chapter 10.3); a Read Request may be answered by more than one (§7).
• "A Completion is the Data Link Layer's acknowledgment." They are different mechanisms at different layers. A Completion is a Transaction Layer transaction that terminates a Sequence; the Data Link Layer has its own Link-local reliability mechanism, owned by Modules 14–15. A packet can be acknowledged Link-locally and still never produce a Completion.
• "Only the Root Complex receives Completions." Any device that issues a Request requiring Completion receives them. An Endpoint reading host memory is the Requester, and the Root Complex is the Completer.
• "An Endpoint cannot be a Requester." An Endpoint "can be the Requester or Completer of a PCI Express transaction" (Chapter 10.1 §4). The roles are per-Sequence.
• "A Completion always carries data." It carries data "in some cases" — determined by what the Request asked for (§8). A data-free Completion answering a non-posted write is complete and correct.
• "Receiving a Completion means software has consumed the result." It means the transaction resolved at the Transaction Layer. The result still has to reach a local consumer, and §12's stage exists precisely because those are separate events.
• "One outstanding Request at a time means PCIe is synchronous." That is an implementation limit, not a protocol property. The protocol is split-transaction; a design that keeps one operation outstanding has chosen to leave throughput on the table (Chapter 10.4 §13).
• "The internal correlation ID is the PCIe Tag." §10's corr_id is a local table index. The Tag is a standardised field whose width, position, and reuse rules are Module 11's. A design maps one onto the other; they are not the same object.
• "A Completion can be routed without retained Request context." It cannot. The Completion identifies which Request; only the Requester knows where that Request's answer belongs (§9).
• "A duplicate Completion is harmless." If the key has been reallocated, it resolves and possibly frees an unrelated live operation — corrupting a transaction that was proceeding correctly. It is one of the most damaging things that can arrive on the return path.

17. Understanding Check

18. What's Next

This chapter took the return half of the taxonomy: what a Completion is, who produces it, why identity rather than arrival order is the correlation mechanism, and what a Requester must retain while an operation is unresolved.

It rested throughout on a division it did not justify: some Requests require a Completion and some do not.

Chapter 10.3 — Posted Transactions takes the "do not" case: which Requests are posted, what the Requester consequently cannot know, why no Completion does not mean no buffering and no flow control, and where forward-path ownership actually ends. Chapter 10.4 — Non-Posted Transactions takes the "do" case and builds the full lifetime controller that §10's table is one piece of — including the correlation-ID allocation that makes reuse safe. Chapter 10.5 then traces one complete exchange across the fabric, with every boundary this module has established in place.

The idea to carry forward: a Completion tells you which Request it answers; only your own retained context can tell you where that answer goes.