PCIe · Module 12
Memory Read — A Split Transaction, End to End
A PCIe memory read is not a bus read that takes a long time. It is a Request packet, retained correlation state, and one or more Completion packets that arrive later and possibly out of order. The full lifecycle, the RTL that owns each stage, and why protocol resolution and local delivery are different events.
Module 11 built the vocabulary. This chapter spends all of it at once.
A local agent wants to read a device register. What actually happens — from the moment that intent exists until the data is in the requester's hands?
The answer is longer than it looks, and almost none of it resembles a bus read.
1. Where the Read Comes From
Chapter 9.6 left off exactly here. Software has a device's BAR base — call it 0x8000_0000 — and it wants the register at offset 0x120. It issues a load to 0x8000_0120.
That load is not a PCIe transaction. It is a CPU or fabric operation against a system address. Something between the core and the link has to notice that this address belongs to PCIe address space and turn the operation into a packet.
software load from 0x8000_0120
→ the Requester recognises a PCIe-space access
→ a Memory Read Request TLP is constructed and launched
→ address routing carries it to the Endpoint
→ the Endpoint's BAR decode matches, producing internal offset 0x120
→ the resource is read
→ a Completion with Data is constructed and routed back
→ the Requester correlates it to the original operation
→ the load's result is deliveredNine steps. The load instruction is step one and the delivery is step nine, and everything between is packets.
2. This Is Not a Bus Read That Takes Longer
The single most important reframing in the chapter, and it is worth being blunt about.
| A shared-bus read | A PCIe Memory Read | |
|---|---|---|
| The interconnect during the wait | held — it is the transaction | free — other traffic proceeds |
| What identifies the response | being the response — nothing else is happening | a correlation field in the packet |
| Number of response packets | one | one or more |
| Response arrival order | trivially the only one | not necessarily request order |
| What the requester must retain | nothing | context, for as long as the read is outstanding |
3. The Verified Semantics
4. What the Request Conveys
Chapter 11.3 owns the header layout, so this is a list of needs, not a field map.
| The Request must convey | Which mechanism carries it |
|---|---|
| that this is a memory read | Fmt and Type together (11.7 §2) |
| where in memory space | the address, in DW2 or DW2–DW3 by header form |
| how much is wanted | the Length field, in DW |
| which bytes of the first and last DW matter | the byte enables (11.3 §7) |
| who is asking | the Requester ID — which is also the return path (11.5 §7) |
| which operation this is | the Tag, correlating the eventual Completion (10.2 §6) |
| under what policy | the Traffic Class and attributes (11.6) |
Note what is absent: any data, and any description of where the answer should be delivered locally. The Request says who asked, not what the asker intends to do with it. The local destination never leaves the Requester — it is held in the context table (§7), keyed by the same identifier that goes into the Tag.
5. Address, Routing, and the BAR
The address in the Request is a system address, and the fabric routes on it.
Take the worked example. Software reads 0x8000_0120.
0x8000_0120 is placed in the Request's address field
→ each Switch compares it against its downstream Base/Limit windows
→ it falls inside the window covering this Endpoint's subtree
→ the Request is forwarded down that port
→ it reaches the Endpoint
→ the Endpoint's BAR decode matches: BAR base 0x8000_0000, so this is mine
→ internal offset = 0x8000_0120 - 0x8000_0000 = 0x120
→ the resource at offset 0x120 is read6. The Lifecycle
Two arrows in that figure are the ones worth staring at.
The Request leaves and nothing is held. Between message three and message eight, the fabric carries whatever else it likes. The only thing connecting the two halves is the correlation state allocated in message two.
The last two messages are separate events. "Correlated" means the protocol side is done with this packet. "Delivered" means the client took the data. They can be many cycles apart, and §14 is why that distinction has to exist in the RTL rather than only in the diagram.
7. The Requester Cannot Forget
A Memory Read that has been transmitted is not finished. It is outstanding, and something has to remember it.
| What must be retained | Why |
|---|---|
| the correlation identifier | it is how the returning Completion is matched (10.2 §6) |
| the local destination context | the client that asked is long gone; the answer needs somewhere to go |
| the expected extent | so the design knows when the read is finished, not merely answered |
| progress so far | because the answer may arrive in pieces (§9) |
| the start address | for diagnostics, and for placing returned chunks correctly |
This is Chapter 10.4's outstanding-transaction structure, instantiated for one specific transaction class. §12's context table is the implementation.
And it bounds concurrency. A design with N context entries can have at most N reads outstanding. When they are all occupied, no new read can launch — which is not an error, it is the design's chosen depth, and §20 explains why that number is a performance parameter rather than an arbitrary one.
8. How Much One Request May Ask For
Max_Read_Request_Size bounds the Request. Max_Payload_Size does not.
This is Chapter 11.4 §6 arriving where it matters most, so state it once more plainly: a Memory Read Request carries no payload, so there is nothing in it for MPS to bound. What MPS bounds is the Completion, because a Completion with Data does carry a payload.
| MRRS | MPS | |
|---|---|---|
| Register field | Device Control bits 14:12 | Device Control bits 7:5 |
| Bounds | how much a read Request asks for | how much data any one TLP carries |
| Applies to a Memory Read Request | yes | no — it has no payload |
| Applies to its Completions | no | yes |
The exact rules governing how a Completer may divide an answer — how many Completions, at what boundaries, in what order — are Module 13's. This chapter needs only that the division happens and that the Requester must accommodate it.
9. The Request Acceptance Contract
Before any RTL, a decision has to be made explicitly, because leaving it implicit is how identifiers leak.
A Memory Read must not be launched unless a context has been allocated for it and the Request can actually be accepted downstream.
The hazard is the gap between those two things. Allocate a context, then discover the downstream Request path is not ready, and the design is holding an allocated identifier for a Request that has not been sent. If it then abandons the attempt without freeing, the identifier is leaked — permanently consumed, reducing the design's concurrency by one, every time it happens.
9a. Three States for a Correlation Index
The pending stage in §10 needs a vocabulary, because "allocated" is now two different things.
| State | Meaning | Entered on | Left on |
|---|---|---|---|
| FREE | available to a new read | reset, or full resolution | local acceptance |
| RESERVED | owned by a Request that has not launched | loc_valid && loc_ready | outbound handshake |
| OUTSTANDING | owned by a Request in flight | outbound handshake | full resolution |
A validity/reserved bitmap is enough — no per-entry state machine is required, and §11's table carries one bit for each condition.
10. RTL — Memory Read Request Engine
// SYNTHESIZABLE. Accept a local read intent, allocate context, and emit a
// normalized Memory Read Request descriptor.
// Non-posted split semantics and MRRS bounding the request extent:
// NORMATIVE (section 3). The descriptor, the interface and the
// correlation-index scheme: ILLUSTRATIVE.
module mem_read_req_engine #(
parameter int ADDR_W = 64,
parameter int CTXS = 8, // outstanding reads supported
parameter int LEN_W = 11, // DW count; 1..1024 needs 11 bits
parameter int DEST_W = 8, // opaque local destination token
// In the PARAMETER LIST, not the body: it appears in the port list, and a
// body localparam cannot be referenced there. Width-safe at CTXS == 1.
parameter int IDX_W = (CTXS <= 1) ? 1 : $clog2(CTXS)
) (
input logic clk,
input logic rst_n,
// ---- Operative limit, from configuration at RUN TIME -----------------
// DECODED byte limit from Device Control bits 14:12 (section 3). An input
// rather than a parameter: software sets it after elaboration.
input logic [12:0] mrrs_bytes,
input logic mrrs_valid,
// ---- Local read intent -----------------------------------------------
input logic loc_valid,
input logic [ADDR_W-1:0] loc_addr,
input logic [LEN_W-1:0] loc_len_dw,
input logic [DEST_W-1:0] loc_dest,
output logic loc_ready,
// ---- Outbound Request descriptor -------------------------------------
output logic req_valid,
output logic [ADDR_W-1:0] req_addr,
output logic [LEN_W-1:0] req_len_dw,
output logic [IDX_W-1:0] req_ctx_id, // INTERNAL index, NOT a Tag
input logic req_ready,
// ---- Context table interface -----------------------------------------
input logic ctx_free_avail,
input logic [IDX_W-1:0] ctx_free_id,
output logic ctx_reserve, // FREE -> RESERVED
output logic [IDX_W-1:0] ctx_reserve_id,
output logic ctx_launch, // RESERVED -> OUTSTANDING
output logic [IDX_W-1:0] ctx_launch_id,
// ---- Reported conditions ---------------------------------------------
output logic oversize_error, // asked beyond MRRS
output logic no_limit_error // MRRS not known
);
generate
if (CTXS < 1) $error("CTXS must be at least 1");
if (LEN_W < 11) $error("LEN_W must hold 1024 DW");
endgenerate
// MRRS in DW. Four bytes per DW.
wire [10:0] mrrs_dw = mrrs_bytes[12:2];
wire limit_ok = mrrs_valid && (mrrs_dw != 11'd0);
wire size_ok = limit_ok && (loc_len_dw != '0)
&& (loc_len_dw <= LEN_W'(mrrs_dw));
// ---- PENDING REQUEST STAGE ------------------------------------------
// The descriptor offered downstream is REGISTERED, never a live view of a
// combinational free-entry scan. Without this stage, req_ctx_id would be
// driven straight from ctx_free_id — and a Completion freeing a
// lower-numbered entry while req_ready is low would change the correlation
// index of an offer already in progress. That is a valid/ready payload
// stability violation, and it corrupts a Request nobody has sent yet.
logic pend_valid_q;
logic [IDX_W-1:0] pend_ctx_q;
logic [ADDR_W-1:0] pend_addr_q;
logic [LEN_W-1:0] pend_len_q;
wire launch = pend_valid_q && req_ready;
// Accept a new local read when the pending stage is empty, or when the
// pending Request is being launched this cycle (same-cycle replacement).
assign loc_ready = (!pend_valid_q || req_ready)
&& ctx_free_avail && size_ok;
wire accept = loc_valid && loc_ready;
// FREE -> RESERVED happens at local acceptance, so the chosen index is
// removed from the free pool before the offer begins.
assign ctx_reserve = accept;
assign ctx_reserve_id = ctx_free_id;
// RESERVED -> OUTSTANDING happens on the downstream handshake.
assign ctx_launch = launch;
assign ctx_launch_id = pend_ctx_q;
// The offer is driven entirely from registers. Nothing here can move
// while req_ready is low.
assign req_valid = pend_valid_q;
assign req_addr = pend_addr_q;
assign req_len_dw = pend_len_q;
assign req_ctx_id = pend_ctx_q;
logic osz_q, nol_q;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
pend_valid_q <= 1'b0; pend_ctx_q <= '0;
pend_addr_q <= '0; pend_len_q <= '0;
osz_q <= 1'b0; nol_q <= 1'b0;
end else begin
// accept has priority over launch, so a same-cycle launch-and-accept
// leaves the stage holding the NEW request rather than going empty.
if (accept) begin
pend_valid_q <= 1'b1;
pend_ctx_q <= ctx_free_id;
pend_addr_q <= loc_addr;
pend_len_q <= loc_len_dw;
end else if (launch) begin
pend_valid_q <= 1'b0;
end
// Reported, not silently clamped. A design that truncated an oversized
// request to MRRS would return less data than the client asked for and
// tell no one (section 9's principle, applied to size).
if (loc_valid && !limit_ok) nol_q <= 1'b1;
if (loc_valid && limit_ok && !size_ok) osz_q <= 1'b1;
end
end
assign oversize_error = osz_q;
assign no_limit_error = nol_q;
endmoduleClassification: synthesizable.
Architecture. An acceptance gate followed by a single-entry pending-request stage. The gate decides whether a read may start; the stage owns the descriptor while the outbound path is busy.
State. The pending descriptor — validity, reserved context index, address, length — plus two sticky error flags.
Cycle behaviour, and the reservation is the point.
| Event | Transition |
|---|---|
loc_valid && loc_ready | context FREE → RESERVED, descriptor captured |
req_valid && req_ready | context RESERVED → OUTSTANDING, stage empties |
| both in the same cycle | previous request launches, new one captured — accept has priority |
Contract. The caller holds the local request stable while loc_valid is asserted. Downstream relies on the whole offered descriptor — including req_ctx_id — being stable for as long as req_ready is low. The context table guarantees ctx_free_id is genuinely free and, critically, stops offering a reserved index the cycle after ctx_reserve.
Failure — and the first is the one this stage exists to prevent. Driving req_ctx_id directly from the live ctx_free_id scan makes the correlation index of an in-progress offer change under backpressure, because a Completion resolving a lower-numbered entry re-points the free-entry search mid-offer. The Request that eventually launches then carries a different index from the one that was reserved — and the context table has state for one index while the packet advertises another. Every Completion for that read arrives as an unknown context. Beyond that: reserving on loc_valid rather than on accept leaks an index every cycle the TX path stalls; giving launch priority over accept loses a request on same-cycle replacement; clamping an oversized request silently returns short data; and computing req_valid without size_ok launches beyond the configured limit.
Deliberately simplified: no splitting of an oversized local request into multiple Requests; no attributes; no byte enables; no address alignment checks (Module 12 continues).
DV. §17's P1–P4.
11. RTL — Read Context Table
// SYNTHESIZABLE. Retain per-read state for the lifetime of an outstanding
// Memory Read, and match returning Completions against it.
// Retaining context across a split transaction: NORMATIVE consequence of
// non-posted semantics. The fields, the index scheme and the progress
// accounting: ILLUSTRATIVE implementation.
module read_ctx_table #(
parameter int ADDR_W = 64,
parameter int CTXS = 8,
parameter int LEN_W = 11,
parameter int DEST_W = 8,
parameter int IDX_W = (CTXS <= 1) ? 1 : $clog2(CTXS)
) (
input logic clk,
input logic rst_n,
// ---- Reservation and launch, from the request engine ------------------
// FREE -> RESERVED. The descriptor is captured here so the entry is
// complete before the Request is ever offered downstream (section 9a).
input logic reserve,
input logic [IDX_W-1:0] reserve_id,
input logic [ADDR_W-1:0] reserve_addr,
input logic [LEN_W-1:0] reserve_len_dw,
input logic [DEST_W-1:0] reserve_dest,
// RESERVED -> OUTSTANDING, on the outbound handshake.
input logic launch,
input logic [IDX_W-1:0] launch_id,
output logic free_avail,
output logic [IDX_W-1:0] free_id,
// Validity bitmap: an entry is OCCUPIED if reserved or outstanding.
// Exposed so a reset assertion can cover EVERY entry (section 14, P13).
output logic [CTXS-1:0] ctx_occupied_mask,
// ---- Returned Completion chunk ---------------------------------------
input logic cpl_valid,
input logic [IDX_W-1:0] cpl_ctx_id,
input logic [LEN_W-1:0] cpl_dw, // DW returned in this chunk
// ---- Per-chunk outputs, for the assembler ----------------------------
output logic chunk_accept,
output logic [DEST_W-1:0] chunk_dest,
output logic [ADDR_W-1:0] chunk_addr, // where this chunk belongs
output logic read_resolved, // ALL expected data has arrived
// ---- Reported conditions ---------------------------------------------
output logic unknown_ctx_error,
output logic overrun_error,
// An identifier outside 0..CTXS-1 was presented. Distinct from
// unknown_ctx_error, which means in-range but not outstanding.
output logic illegal_id_error
);
typedef struct packed {
logic valid; // RESERVED or OUTSTANDING
logic outstanding; // launched; Completions may arrive
logic [DEST_W-1:0] local_dest;
logic [LEN_W-1:0] remaining_dw;
logic [ADDR_W-1:0] start_addr;
logic [LEN_W-1:0] received_dw;
} read_ctx_t;
read_ctx_t ctx_q [CTXS];
always_comb
for (int i = 0; i < CTXS; i++) ctx_occupied_mask[i] = ctx_q[i].valid;
// ---- RANGE SAFETY ----------------------------------------------------
// IDX_W = $clog2(CTXS) can express values >= CTXS whenever CTXS is not a
// power of two: at CTXS = 5, IDX_W is 3 and 5, 6 and 7 are presentable
// but do not exist. Every array access below is guarded. The comparison
// is one bit wider than the identifier so CTXS stays representable when
// it IS a power of two — a same-width compare truncates CTXS to zero at
// CTXS = 8 and rejects every legal index.
localparam int CHK_W = IDX_W + 1;
wire cpl_id_legal = (CHK_W'(cpl_ctx_id) < CHK_W'(CTXS));
wire reserve_id_legal = (CHK_W'(reserve_id) < CHK_W'(CTXS));
wire launch_id_legal = (CHK_W'(launch_id) < CHK_W'(CTXS));
// Guarded reads. Defaults make an illegal index look absent, so it can
// never satisfy ctx_hit and never be credited.
logic cpl_entry_valid, cpl_entry_outstanding;
logic [LEN_W-1:0] cpl_entry_remaining, cpl_entry_received;
logic [DEST_W-1:0] cpl_entry_dest;
logic [ADDR_W-1:0] cpl_entry_start;
always_comb begin
cpl_entry_valid = 1'b0;
cpl_entry_outstanding = 1'b0;
cpl_entry_remaining = '0;
cpl_entry_received = '0;
cpl_entry_dest = '0;
cpl_entry_start = '0;
if (cpl_id_legal) begin
cpl_entry_valid = ctx_q[cpl_ctx_id].valid;
cpl_entry_outstanding = ctx_q[cpl_ctx_id].outstanding;
cpl_entry_remaining = ctx_q[cpl_ctx_id].remaining_dw;
cpl_entry_received = ctx_q[cpl_ctx_id].received_dw;
cpl_entry_dest = ctx_q[cpl_ctx_id].local_dest;
cpl_entry_start = ctx_q[cpl_ctx_id].start_addr;
end
end
// Free-entry search over the occupancy bitmap. A RESERVED entry is already
// valid, so it is never offered again — which is what keeps the offered
// req_ctx_id from being re-pointed mid-offer (section 9a). Linear for
// clarity at teaching scale; a real design uses a free-list.
logic fa;
logic [IDX_W-1:0] fid;
always_comb begin
fa = 1'b0; fid = '0;
for (int i = CTXS - 1; i >= 0; i--)
if (!ctx_q[i].valid) begin fa = 1'b1; fid = IDX_W'(i); end
end
assign free_avail = fa;
assign free_id = fid;
// Completion matching. A chunk is accepted only for an OUTSTANDING context
// — an unknown identifier, or one that is merely RESERVED, must never
// disturb any entry. A Completion for a Request that has not launched is
// by definition not ours.
wire ctx_hit = cpl_valid && cpl_id_legal
&& cpl_entry_valid && cpl_entry_outstanding;
wire [LEN_W-1:0] rem = cpl_entry_remaining;
wire overrun = ctx_hit && (cpl_dw > rem);
wire fits = ctx_hit && !overrun && (cpl_dw != '0);
assign chunk_accept = fits;
assign chunk_dest = cpl_entry_dest;
// Where this chunk belongs within the read: start + what already arrived.
// Derived from RETAINED state, never from the packet's arrival order.
assign chunk_addr = cpl_entry_start
+ ADDR_W'(cpl_entry_received) * ADDR_W'(4);
// RESOLVED means every expected DW has arrived. It does NOT mean the local
// client has taken the data — that is a separate event (section 14).
assign read_resolved = fits && (cpl_dw == rem);
logic unk_q, ovr_q, ill_q;
assign unknown_ctx_error = unk_q;
assign overrun_error = ovr_q;
assign illegal_id_error = ill_q;
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
for (int i = 0; i < CTXS; i++) ctx_q[i] <= '0;
unk_q <= 1'b0; ovr_q <= 1'b0; ill_q <= 1'b0;
end else begin
// FREE -> RESERVED. The entry becomes occupied immediately, so the
// free-entry search stops offering this index from the next cycle.
// Gated on legality: no out-of-range index ever reaches the array.
if (reserve && reserve_id_legal) begin
ctx_q[reserve_id].valid <= 1'b1;
ctx_q[reserve_id].outstanding <= 1'b0;
ctx_q[reserve_id].local_dest <= reserve_dest;
ctx_q[reserve_id].remaining_dw <= reserve_len_dw;
ctx_q[reserve_id].start_addr <= reserve_addr;
ctx_q[reserve_id].received_dw <= '0;
end
// RESERVED -> OUTSTANDING. Only now may Completions match it.
if (launch && launch_id_legal)
ctx_q[launch_id].outstanding <= 1'b1;
// `fits` already requires cpl_id_legal, so this write is guarded.
if (fits) begin
ctx_q[cpl_ctx_id].received_dw <= cpl_entry_received + cpl_dw;
ctx_q[cpl_ctx_id].remaining_dw <= rem - cpl_dw;
// Freed ONLY when the whole extent has arrived. Freeing on the first
// chunk is section 8's catastrophe.
// Freed ONLY when the whole extent has arrived, and both bits clear
// together so the entry cannot linger as outstanding-but-invalid.
if (cpl_dw == rem) begin
ctx_q[cpl_ctx_id].valid <= 1'b0;
ctx_q[cpl_ctx_id].outstanding <= 1'b0;
end
end
// Reported, never acted on. A chunk for an unknown, out-of-range or
// not-yet-launched context is dropped and flagged; it must not write
// any entry. Out-of-range is reported separately because it points at
// an upstream mapping bug rather than at context lifetime.
if (cpl_valid && !cpl_id_legal) ill_q <= 1'b1;
if (reserve && !reserve_id_legal) ill_q <= 1'b1;
if (launch && !launch_id_legal) ill_q <= 1'b1;
if (cpl_valid && cpl_id_legal
&& !(cpl_entry_valid
&& cpl_entry_outstanding)) unk_q <= 1'b1;
if (overrun) ovr_q <= 1'b1;
end
end
endmoduleClassification: synthesizable.
Architecture. An indexed table with per-entry progress accounting and an explicit three-state lifetime (§9a). The entry is the read, and it lives from reservation — before the Request is even offered — until every expected DW has arrived.
State. Per entry: occupancy, an outstanding bit, local destination, remaining and received DW counts, start address. Plus three sticky error flags and an exported occupancy bitmap.
Range safety. IDX_W is $clog2(CTXS), which can express values ≥ CTXS whenever CTXS is not a power of two — at CTXS = 5, IDX_W is 3 and 5, 6 and 7 are presentable but do not exist. Every array access is therefore guarded, with defaults that make an illegal index look absent so it can never satisfy ctx_hit, and every write path is gated on the same legality. The comparison is one bit wider than the identifier, so CTXS stays representable when it is a power of two; a same-width compare truncates CTXS to zero at CTXS = 8 and rejects every legal index. Chapter 12.3 §8a explains why illegal_id_error is reported separately from unknown_ctx_error.
Cycle behaviour. Reservation writes one entry and marks it occupied; launch promotes it to outstanding; an accepted chunk updates it; it frees on the cycle the last chunk is accepted. A freed entry is reusable the following cycle — the free-entry search sees the cleared occupancy on the next evaluation, so there is no same-cycle reuse and no window in which a straggler could match a newly reserved read.
Contract. The request engine relies on free_id being genuinely free and on a reserved index disappearing from the free pool on the next cycle — that is what keeps §10's offered req_ctx_id stable. The assembler relies on chunk_dest and chunk_addr being derived from retained state, never from the arriving packet, and on read_resolved meaning all expected data has arrived and nothing more.
Failure — six, and the first is the one this chapter is built around. Freeing on any accepted chunk rather than on the last frees the identifier while Completions are in flight (§8). Indexing the table before the legality check lets an out-of-range identifier read or write outside it at any non-power-of-two CTXS. Marking an entry occupied only at launch rather than at reservation re-opens §9a's window, and the offered index moves under backpressure. Deriving chunk_addr from arrival order rather than from received_dw misplaces data whenever chunks interleave with another read's. Accepting a chunk for a context that is occupied but not yet outstanding credits progress to a Request that has not been sent. And omitting the overrun check lets remaining_dw underflow, after which the read never resolves.
Deliberately simplified: no Completion Status handling (Module 13); no timeout; no out-of-order chunk placement within a single read — chunks are assumed to arrive in address order for that read, and §17's P9 states the assumption; a linear free-entry search.
Production implication: a real table handles status, timeout, and whatever ordering the Completer's splitting is permitted to produce. What does not change is that the entry outlives every Completion but the last.
12. Protocol Resolution Is Not Local Delivery
Two events, easily conflated, and conflating them is a real bug rather than a naming preference.
| Protocol resolution | Local delivery | |
|---|---|---|
| Means | all expected data for the Request has arrived | the local client has taken the result |
| Decided by | the Completer and the fabric | the local client's readiness |
| Frees | the correlation identifier and context entry | the result buffer |
| Timing | when the last Completion is accepted | possibly much later |
13. RTL — Result Buffer
// SYNTHESIZABLE. Decouple protocol resolution from local delivery.
// The separation of the two events is an ARCHITECTURAL requirement
// (section 12); the queue depth, the status abstraction and the interface
// are ILLUSTRATIVE.
module read_result_buffer #(
parameter int DEPTH = 4,
parameter int DEST_W = 8,
parameter int DATA_W = 128
) (
input logic clk,
input logic rst_n,
// ---- From the completion path — CANNOT be backpressured --------------
input logic res_valid,
input logic [DEST_W-1:0] res_dest,
input logic [DATA_W-1:0] res_data,
input logic [DATA_W/8-1:0] res_byte_valid,
input logic res_error, // abstracted status
// ---- To the local client — MAY backpressure --------------------------
output logic out_valid,
output logic [DEST_W-1:0] out_dest,
output logic [DATA_W-1:0] out_data,
output logic [DATA_W/8-1:0] out_byte_valid,
output logic out_error,
input logic out_ready,
// Reported: a result arrived with no room. NOT silently dropped.
output logic overflow_error
);
localparam int IDX_W = (DEPTH <= 1) ? 1 : $clog2(DEPTH);
localparam int CNT_W = $clog2(DEPTH + 1);
typedef struct packed {
logic [DEST_W-1:0] dest;
logic [DATA_W-1:0] data;
logic [DATA_W/8-1:0] bv;
logic err;
} res_t;
res_t mem_q [DEPTH];
logic [IDX_W-1:0] wr_q, rd_q;
logic [CNT_W-1:0] cnt_q;
logic ovf_q;
wire full = (cnt_q == CNT_W'(DEPTH));
wire pop = out_valid && out_ready;
// EFFECTIVE capacity, not instantaneous capacity. A slot being vacated
// this cycle is available this cycle: full + pop + push replaces one
// element and loses nothing. Gating push on `!full` alone would drop a
// result at exactly the moment the consumer was draining the buffer —
// the busiest moment, and the one a bench is least likely to hit.
wire can_accept = !full || pop;
wire push = res_valid && can_accept;
assign out_valid = (cnt_q != '0);
assign out_dest = mem_q[rd_q].dest;
assign out_data = mem_q[rd_q].data;
assign out_byte_valid = mem_q[rd_q].bv;
assign out_error = mem_q[rd_q].err;
assign overflow_error = ovf_q;
function automatic logic [IDX_W-1:0] next_idx (input logic [IDX_W-1:0] i);
next_idx = (i == IDX_W'(DEPTH - 1)) ? '0 : (i + IDX_W'(1));
endfunction
always_ff @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
wr_q <= '0; rd_q <= '0; cnt_q <= '0; ovf_q <= 1'b0;
end else begin
if (push) begin
mem_q[wr_q].dest <= res_dest;
mem_q[wr_q].data <= res_data;
mem_q[wr_q].bv <= res_byte_valid;
mem_q[wr_q].err <= res_error;
wr_q <= next_idx(wr_q);
end
if (pop) rd_q <= next_idx(rd_q);
case ({push, pop})
2'b10: cnt_q <= cnt_q + CNT_W'(1);
2'b01: cnt_q <= cnt_q - CNT_W'(1);
default: cnt_q <= cnt_q;
endcase
// Overflow is ONLY the genuinely-no-room case: full, nothing leaving,
// and something arriving. A result that cannot be stored is a REPORTED
// loss — dropping it silently would mean a read that resolved at the
// protocol level and returned nothing.
if (res_valid && full && !pop) ovf_q <= 1'b1;
end
end
endmoduleClassification: synthesizable.
Architecture. A decoupling queue between a producer that cannot be told to wait and a consumer that can. res_valid has no res_ready — deliberately, because by the time a result reaches this buffer the data has already arrived from the fabric and there is nowhere upstream to push it back to.
State. The entry array, pointers, occupancy, and a sticky overflow flag.
Cycle behaviour — and the capacity test is the subtle part. Acceptance is gated on effective capacity, !full || pop, not on instantaneous capacity:
| Occupancy | pop | res_valid | Result |
|---|---|---|---|
below DEPTH | 0 | 1 | accepted, occupancy +1 |
at DEPTH | 1 | 1 | accepted — replacement, occupancy unchanged |
at DEPTH | 0 | 1 | overflow — genuinely no room |
| empty | — | 0 | idle; pop cannot occur |
Because entries are registers rather than a RAM macro, the full+pop+push case has no read/write-collision ambiguity to reason about — the element being read this cycle is presented from mem_q[rd_q] and the new element is written to mem_q[wr_q], and at full those indices are equal only if DEPTH is 1, where the write lands after the read has already been sampled by the consumer. A RAM-based implementation would need an explicit read-before-write or output-register policy stated.
Contract. The client relies on the result being stable while it stalls — an ordinary valid/ready guarantee, but load-bearing here because the data has no other home. The system relies on DEPTH being sized so that overflow does not occur in normal operation; overflow_error exists to make a sizing mistake visible rather than to make it survivable.
Failure — three, and the second is easy to write and hard to find. Dropping a result when full without reporting produces a read that resolved at the protocol level and delivered nothing, with no error anywhere. Gating push on !full alone rather than on !full || pop drops a result at exactly the moment the consumer was draining the buffer — the busiest moment, and the one least likely to appear in a directed test, since it needs the buffer at capacity and a simultaneous pop and push. And beyond those: natural pointer rollover breaks at non-power-of-two DEPTH, and $clog2(DEPTH) without the DEPTH <= 1 guard is zero-width at the stated minimum.
Deliberately simplified: whole-result granularity rather than per-beat streaming; abstracted error status; no per-destination queues.
14. Assertions
// SVA over mem_read_req_engine, read_ctx_table and read_result_buffer.
// LOCAL contracts plus the normative non-posted lifecycle of section 3.
// SAFETY properties first; LIVENESS is section 15 and is stated separately
// with its assumptions.
// REQUEST — P1: no Request is ever offered without a context RESERVED for it
// (section 9a). The offer and the reservation cannot diverge.
property p_no_offer_without_reservation;
@(posedge clk) disable iff (!rst_n)
req_valid |-> ctx_occupied_mask[req_ctx_id];
endproperty
a_offer_reserved : assert property (p_no_offer_without_reservation);
// REQUEST — P2: a reservation is made only for a request the stage takes
// ownership of, and it is released only by launch or reset. Together P1, P2
// and P13a make both the identifier leak AND the mid-offer index change
// unrepresentable.
property p_reserve_only_on_accept;
@(posedge clk) disable iff (!rst_n)
ctx_reserve |-> (loc_valid && loc_ready);
endproperty
a_reserve_on_accept : assert property (p_reserve_only_on_accept);
// REQUEST — P2a: launch happens only on the outbound handshake, and promotes
// the index that was actually offered.
property p_launch_on_handshake;
@(posedge clk) disable iff (!rst_n)
ctx_launch |-> (req_valid && req_ready && (ctx_launch_id == req_ctx_id));
endproperty
a_launch_handshake : assert property (p_launch_on_handshake);
// REQUEST — P3: the request extent never exceeds MRRS (section 3, 8).
property p_within_mrrs;
@(posedge clk) disable iff (!rst_n)
(req_valid && mrrs_valid) |-> (req_len_dw <= LEN_W'(mrrs_dw));
endproperty
a_mrrs : assert property (p_within_mrrs);
// REQUEST — P4: descriptor stable while the TX path stalls.
property p_req_stable_under_stall;
@(posedge clk) disable iff (!rst_n)
(req_valid && !req_ready)
|=> (req_valid && $stable({req_addr, req_len_dw, req_ctx_id}));
endproperty
a_req_stable : assert property (p_req_stable_under_stall);
// CONTEXT — P5: UNIQUENESS. A correlation index is never reserved while it is
// already occupied — whether reserved or outstanding. THE property that
// prevents a straggling Completion from resolving against a later read
// (section 8), now covering the reservation window too.
property p_id_not_reused_while_owned;
@(posedge clk) disable iff (!rst_n)
reserve |-> !ctx_occupied_mask[reserve_id];
endproperty
a_unique_id : assert property (p_id_not_reused_while_owned);
// CONTEXT — P6: ISOLATION. A chunk for an unknown context disturbs nothing.
// (ctx_snapshot is a testbench copy of the whole table.)
property p_unknown_ctx_harmless;
@(posedge clk) disable iff (!rst_n)
(cpl_valid && cpl_id_legal && !(cpl_entry_valid && cpl_entry_outstanding))
|=> (ctx_q == $past(ctx_snapshot)) && unknown_ctx_error;
endproperty
a_unknown_isolated : assert property (p_unknown_ctx_harmless);
// RANGE — P6a: an out-of-range identifier changes NOTHING in the table.
// The structural property; it fails for any design that indexes before it
// checks, at any non-power-of-two CTXS.
property p_illegal_id_inert;
@(posedge clk) disable iff (!rst_n)
((cpl_valid && !cpl_id_legal) || (reserve && !reserve_id_legal)
|| (launch && !launch_id_legal))
|=> ((ctx_q == $past(ctx_snapshot)) && illegal_id_error);
endproperty
a_illegal_inert : assert property (p_illegal_id_inert);
// RANGE — P6b: an out-of-range Completion is never accepted as progress.
property p_illegal_chunk_not_accepted;
@(posedge clk) disable iff (!rst_n)
(cpl_valid && !cpl_id_legal) |-> (!chunk_accept && !read_resolved);
endproperty
a_illegal_chunk : assert property (p_illegal_chunk_not_accepted);
// RANGE — P6c: per-entry write provenance. A table entry changes only when
// THAT entry was the legal selected index. An aggregate check would let a
// write to a neighbouring entry hide.
generate for (genvar g = 0; g < CTXS; g++) begin : g_write_prov
a_write_provenance : assert property (@(posedge clk) disable iff (!rst_n)
!$stable(ctx_q[g])
|-> ($past(reserve && reserve_id_legal && (reserve_id == IDX_W'(g)))
|| $past(launch && launch_id_legal && (launch_id == IDX_W'(g)))
|| $past(fits && (cpl_ctx_id == IDX_W'(g)))
|| $past(!rst_n)));
end endgenerate
// CONTEXT — P7: CONSERVATION. Progress never exceeds the requested extent.
property p_progress_bounded;
@(posedge clk) disable iff (!rst_n)
chunk_accept |=> (ctx_q[$past(cpl_ctx_id)].received_dw
<= $past(ctx_q[$past(cpl_ctx_id)].received_dw)
+ $past(ctx_q[$past(cpl_ctx_id)].remaining_dw));
endproperty
a_progress : assert property (p_progress_bounded);
// CONTEXT — P8: THE CENTRAL LIFETIME PROPERTY. A context is freed ONLY when
// the whole extent has arrived. A partial Completion must not free it.
property p_free_only_on_full_extent;
@(posedge clk) disable iff (!rst_n)
(chunk_accept && !read_resolved) |=> ctx_q[$past(cpl_ctx_id)].valid;
endproperty
a_no_early_free : assert property (p_free_only_on_full_extent);
// CONTEXT — P9: chunk placement comes from RETAINED progress, not from
// arrival order. States the in-read ordering assumption explicitly.
property p_chunk_addr_from_progress;
@(posedge clk) disable iff (!rst_n)
chunk_accept |-> (chunk_addr == cpl_entry_start
+ ADDR_W'(cpl_entry_received) * ADDR_W'(4));
endproperty
a_placement : assert property (p_chunk_addr_from_progress);
// CONTEXT — P10: exactly one local completion event per resolved read.
property p_one_resolution_per_read;
@(posedge clk) disable iff (!rst_n)
read_resolved |=> !read_resolved until_with reserve;
endproperty
// DELIVERY — P11: the result is stable while the local client stalls. The
// property that makes section 12's decoupling real.
property p_result_stable_under_client_stall;
@(posedge clk) disable iff (!rst_n)
(out_valid && !out_ready)
|=> (out_valid && $stable({out_dest, out_data, out_byte_valid, out_error}));
endproperty
a_result_stable : assert property (p_result_stable_under_client_stall);
// DELIVERY — P12: a result is never silently lost. Overflow is reported ONLY
// when there was genuinely no room: full, nothing leaving, something
// arriving.
property p_no_silent_result_loss;
@(posedge clk) disable iff (!rst_n)
(res_valid && full && !pop) |=> overflow_error;
endproperty
a_no_loss : assert property (p_no_silent_result_loss);
// DELIVERY — P12a: FULL + POP + PUSH IS NOT OVERFLOW. The slot being vacated
// this cycle is usable this cycle, so the result is accepted and occupancy
// is unchanged. Catches a `push = res_valid && !full` capacity test.
property p_full_pop_push_accepted;
@(posedge clk) disable iff (!rst_n)
(res_valid && full && pop) |-> (push && !$rose(overflow_error));
endproperty
a_full_pop_push : assert property (p_full_pop_push_accepted);
// DELIVERY — P12b: simultaneous push and pop conserves occupancy at every
// level, not only at full.
property p_simul_conserves;
@(posedge clk) disable iff (!rst_n)
(push && pop) |=> (cnt_q == $past(cnt_q));
endproperty
a_simul_res : assert property (p_simul_conserves);
// RESET — P13: reset clears EVERY context, the pending stage, and the result
// buffer. Written over the whole occupancy bitmap rather than over entry
// zero, so a design that clears only the entry it happens to index cannot
// pass. Defined explicitly because "what happens to reads outstanding at
// reset" is otherwise ambiguous.
property p_reset_clears_all;
@(posedge clk)
!rst_n |=> (!out_valid
&& (ctx_occupied_mask == '0)
&& !pend_valid_q
&& free_avail);
endproperty
a_reset : assert property (p_reset_clears_all);
// RESERVATION — P13a: THE STABILITY PROPERTY THIS ARCHITECTURE EXISTS FOR.
// The offered Request descriptor — including the correlation index — does
// not move while the outbound path stalls, no matter what Completions
// resolve underneath it.
property p_offer_stable_under_tx_stall;
@(posedge clk) disable iff (!rst_n)
(req_valid && !req_ready)
|=> (req_valid && $stable({req_ctx_id, req_addr, req_len_dw}));
endproperty
a_offer_stable : assert property (p_offer_stable_under_tx_stall);
// RESERVATION — P13b: a reserved index is never offered as free again.
property p_reserved_not_reoffered;
@(posedge clk) disable iff (!rst_n)
(reserve) |=> (ctx_occupied_mask[$past(reserve_id)]);
endproperty
a_reserved_held : assert property (p_reserved_not_reoffered);
// RESERVATION — P13c: launch promotes exactly the reserved index, and only
// an index that was reserved.
property p_launch_promotes_reserved;
@(posedge clk) disable iff (!rst_n)
launch |-> (ctx_occupied_mask[launch_id] && !ctx_q[launch_id].outstanding);
endproperty
a_launch_promotes : assert property (p_launch_promotes_reserved);
// RESERVATION — P13d: the states are consistent for EVERY entry. An entry can
// never be outstanding without being occupied, which is what makes FREE /
// RESERVED / OUTSTANDING a partition rather than three independent bits.
// Written as a generate loop because the property is per-entry.
generate for (genvar g = 0; g < CTXS; g++) begin : g_state_excl
a_state_partition : assert property (@(posedge clk) disable iff (!rst_n)
ctx_q[g].outstanding |-> ctx_q[g].valid);
end endgenerate
// RESERVATION — P13e: a Completion for a merely-RESERVED context is rejected,
// not accepted as progress. A Request that has not launched cannot have an
// answer in flight.
property p_no_completion_for_unlaunched;
@(posedge clk) disable iff (!rst_n)
(cpl_valid && cpl_id_legal && cpl_entry_valid && !cpl_entry_outstanding)
|-> !chunk_accept;
endproperty
a_unlaunched_rejected : assert property (p_no_completion_for_unlaunched);
// SAFETY — P14: no descriptor or result output is ever unknown.
property p_outputs_never_unknown;
@(posedge clk) disable iff (!rst_n)
(req_valid |-> !$isunknown({req_addr, req_len_dw, req_ctx_id}))
&& (out_valid |-> !$isunknown({out_dest, out_error}));
endproperty
a_no_x : assert property (p_outputs_never_unknown);P1, P2, P2a and P13a are the reservation contract, split four ways because they fail on four different bugs. P1 says an offer always has a reservation behind it. P2 says a reservation is only made for a request the stage takes ownership of. P2a says launch promotes exactly the index that was offered. P13a is the one that motivated the architecture: the offered descriptor, correlation index included, does not move while the outbound path stalls. A design driving req_ctx_id from a live free-entry scan passes P1, P2 and P2a and fails P13a the first time a Completion resolves during a TX stall.
P5 and P8 are the pair that prevents §8's catastrophe. P8 keeps the entry alive until the extent is complete; P5 ensures that an entry which is alive cannot be handed out again — and P5 now covers the reservation window as well, since an entry becomes occupied before the Request is offered. A design that satisfies P5 but not P8 frees early and then reuses legitimately, and the straggler still lands on the wrong read. Both are needed.
P13e closes the other end of the lifetime. A Completion cannot legitimately exist for a Request that has not launched, so a chunk arriving for a merely-RESERVED context is rejected rather than credited. Without the outstanding bit there is no way to state that: a reserved entry and an outstanding entry would look identical.
P6 is stated over a snapshot of the whole table rather than over the addressed entry, because the failure it catches is a write to some entry — possibly not the one indexed — when an unknown identifier arrives. Checking only the addressed entry would miss a decode that indexed elsewhere — and note that the properties themselves read the guarded cpl_entry_* signals rather than the array, for the same reason the RTL does.
P12a is the assertion for a bug that reads as correct. push = res_valid && !full looks like a capacity check and is one — of the wrong quantity. The slot a pop vacates is available in the same cycle, so refusing a push at full-with-pop discards a result the buffer had room for. P12a states the acceptance directly rather than checking the absence of overflow, because a design could avoid reporting overflow while still dropping the data.
P9 makes an assumption visible rather than hiding it. The model assumes chunks for a single read arrive in address order; the property states the placement rule that follows. If a design must tolerate out-of-order chunks within one read, P9 is the property that must change — and having it written down is what makes that a decision rather than an oversight.
15. Liveness, and Its Assumptions
None of §14 says a read ever completes. That is deliberate: it is not true without assumptions, and asserting it unconditionally would fail on any testbench that models a stalled fabric or an unresponsive Completer.
// LIVENESS. Valid ONLY under the environment assumptions below. State them
// as `assume` so the obligation sits on the environment, where it belongs.
// A1: the TX path eventually accepts an offered Request.
assume property (@(posedge clk) disable iff (!rst_n)
req_valid |-> s_eventually req_ready);
// A2: a launched read is eventually answered in full by the remote system.
// This assumes a functioning fabric AND a functioning Completer — neither is
// guaranteed by PCIe, which is why Completion timeout mechanisms exist.
assume property (@(posedge clk) disable iff (!rst_n)
(req_valid && req_ready) |-> s_eventually read_resolved);
// A3: the local client eventually accepts a presented result.
assume property (@(posedge clk) disable iff (!rst_n)
out_valid |-> s_eventually out_ready);
// L1: under A1-A3, an accepted local read eventually delivers a result.
property p_read_eventually_delivers;
@(posedge clk) disable iff (!rst_n)
(loc_valid && loc_ready) |-> s_eventually (out_valid && out_ready);
endproperty
a_liveness : assert property (p_read_eventually_delivers);
// L2: a context entry is eventually released, so the design does not
// permanently lose concurrency.
property p_context_eventually_freed;
@(posedge clk) disable iff (!rst_n)
alloc |-> s_eventually free_avail;
endproperty
a_ctx_liveness : assert property (p_context_eventually_freed);The three assumptions are not decoration — each names a real way a system fails. A2 in particular is the one PCIe explicitly does not guarantee: a Completion may never arrive, which is why Completion timeout mechanisms exist at all. A design that assumed A2 in its RTL rather than in its assertions would have no timeout handling.
Keeping safety and liveness apart is the discipline. The safety properties in §14 hold unconditionally and are what you run in regression. L1 and L2 hold only in a well-behaved environment, and their value is that they prove the design does not add a deadlock — it does not prove the system makes progress.
16. Verification
Monitors observe: the local read interface; the outbound Request descriptor; the returning Completion chunk interface; the context table's allocation and free events; and the result interface.
The scoreboard maintains its own model, independently:
- Its own outstanding map, keyed by correlation index →
{address, length, local destination, DW received}. Never readctx_q. The context table is the thing under test; using it as the oracle verifies that it agrees with itself. - Its own free-index model. Never read
free_availorfree_idas expected values. - Its own memory model for expected read data (below).
The independent memory model
Expected data must not be dut_memory[address]. Use a deterministic function of the address with no correlation to the address's structure:
expected_dw(addr) = a decorrelated deterministic function of addrA well-chosen function makes address corruption observable. If expected data were, say, the address itself, then a single wrong address bit produces expected data that differs from the received data in that same bit — subtle and easy to miss. A decorrelated function turns any address error into completely different data, which a comparison catches immediately and unambiguously.
Directed tests
- A single read, smallest modelled size. The baseline.
- A single read at the maximum modelled size.
- Several reads outstanding simultaneously, up to
CTXS. CTXSreads outstanding, then one more offered. Verifyloc_readygoes low and nothing is consumed.- Completions returning in request order.
- Completions returning in a different permitted order. Verify each read's data reaches its own destination — the direct test that arrival order is not identity.
- A multi-chunk read, if the multi-Completion path is modelled: verify the context survives every chunk but the last (P8) and that chunks are placed by progress, not arrival (P9).
- Interleaved chunks from two different reads. The strongest correlation test available.
Reservation and offer stability
- Offer a read, stall the TX path, and resolve an unrelated read that frees a lower-numbered context. Verify
req_ctx_iddoes not change (P13a). This is the chapter's most important new test — it is the exact stimulus the previous architecture failed, and no amount of ordinary backpressure testing produces it. - Reserve, then check the free-entry search. Verify the reserved index is no longer offered as free from the next cycle (P13b).
- Same-cycle launch and accept. Verify the pending stage ends holding the new request and the old one launched — not empty, and not still holding the old one.
- A Completion arriving for a reserved-but-not-launched context. Verify it is rejected and reported (P13e), and that no progress is credited.
- Reset with a request pending. Verify
pend_valid_qclears and the reservation is released (P13). CTXS = 1. Reserve the only entry, verifyloc_readydrops, and verify the entry is reusable one cycle after resolution.
Range safety — non-power-of-two CTXS
CTXS = 5, then presentcpl_ctx_id= 5, 6 and 7. Verifyillegal_id_error, no chunk accepted, and no entry changed (P6a–P6c). Required stimulus — atCTXS = 4or8these identifiers are unreachable.CTXS = 5withreserve_idandlaunch_idout of range. Verify no allocation and no promotion.CTXS = 8. Verify every identifier 0–7 is legal — the counter-test for a same-width legality compare.CTXS = 1,3,6,7. Sweep the corners.
Backpressure
- The TX path stalled while a read is offered. Verify no additional reservation is made and the descriptor is stable (P2, P13a).
- The local client stalled for a long period while Completions return. Verify results are buffered, stable (P11), and that the context table does not fill — §12's decoupling, tested directly.
- The result buffer driven to full with the consumer stalled, then one more result. Verify
overflow_errorand no silent loss (P12). - The result buffer at full with the consumer accepting in the same cycle a new result arrives. Verify the result is accepted, occupancy is unchanged, and
overflow_errordoes not set (P12a, P12b). The full+pop+push case is a required test, not a stress case — it is the busiest steady state a buffer sees. - A long run at exactly full occupancy with producer and consumer both at full rate. Verify no result is lost and none is duplicated.
Negative
- A Completion chunk with an unknown correlation index. Verify nothing changes and the error is reported (P6).
- A duplicate Completion for an already-resolved read. Verify it is treated as unknown, not as progress.
- A chunk larger than the remaining extent. Verify
overrun_errorand thatremaining_dwdoes not underflow. - A zero-DW chunk. Verify it is not accepted as progress.
- A read request exceeding MRRS. Verify it is refused and reported, not clamped.
- A read offered with
mrrs_validlow. Verify refusal. - Reset with reads outstanding. Verify all contexts clear and no stale result emerges (P13).
CTXS = 1and a non-power-of-twoDEPTHon the result buffer. Parameter corners.
Which test kills which bug
Stating the mapping explicitly is what makes a test plan a verification argument rather than a checklist.
| Injected fault | What catches it |
|---|---|
| a wrong address bit in the Request | the decorrelated memory model — data mismatch, immediately |
| context freed on the first chunk | P8, and the interleaved-chunk test |
| correlation index reused while outstanding | P5, and the multiple-outstanding test |
| chunk placed by arrival order | P9, and the interleaved-chunk test |
| result delivered to the wrong local destination | the scoreboard's independent destination map |
| the final chunk dropped | the read never resolves — L2 under its assumptions, and a scoreboard timeout |
| duplicated returned data | P7 conservation, and the total-DW check |
| resolution signalled early | P8, and a byte-count comparison at delivery |
| context reserved on offer rather than on acceptance | P2, under TX backpressure |
req_ctx_id driven from the live free-entry scan | P13a, with a Completion resolving during a TX stall |
| Completion credited to a reserved-but-unlaunched context | P13e |
| table indexed before the legality check | P6a/P6c, at non-power-of-two CTXS |
| result dropped at full+pop+push | P12a, and the full-rate steady-state run |
| result dropped when the buffer is genuinely full | P12 |
Coverage should include: every legal request size; one to CTXS reads outstanding; every completion-ordering permutation for two reads; single-chunk and multi-chunk resolution; every reported error condition; TX, completion and client backpressure independently and together; and the parameter minima.
17. Debugging Ladder
Work down this list in order. Each step eliminates everything above it, which is what makes it a ladder rather than a list of suspects.
1. The Request never leaves
Nothing reached the fabric, so nothing downstream can be at fault.
Check, in this order: is loc_valid actually asserted? Is ctx_free_avail high — or are all CTXS entries occupied by reads that never resolved, which is a different bug wearing this one's symptoms? Is req_ready high? Is size_ok — is the request within MRRS, and is mrrs_valid even set?
The distinguishing observation: if ctx_free_avail is low, the problem is previous reads, not this one. Dump the context table and see what is still valid and how old it is.
2. The Request leaves but the Endpoint never sees it
Routing. The packet was launched and did not arrive.
Check the address against each Switch port's Base/Limit windows (Chapter 11.5 §4). Remember that an address matching no downstream window is forwarded upstream, not dropped — so the packet went somewhere, and finding where is faster than asking why it vanished.
Also check the address itself. A Request that leaves with a corrupted address routes perfectly to the wrong place.
3. The Endpoint sees the Request but no resource is hit
Routing worked; BAR decode did not (§5).
Check the BAR value against the Request address, check that the BAR is enabled, and check the internal offset extraction. A window/BAR disagreement is a configuration fault (Chapter 9.5): the fabric believed this address belonged here and the Endpoint disagrees, and one of them was configured wrong.
4. The resource produces data but no Completion leaves
The Completer's Completion path. Check Completion generation, the output queue, and arbitration against other outbound traffic.
Also check the Completer's flow-control credits — a Completion that cannot be sent for lack of credit looks identical to one that was never built.
5. A Completion returns but the Requester reports an unknown context
Correlation lifetime, and this is where §8's bug surfaces.
Three candidates, in order of likelihood. The context was freed early — on the first chunk of a multi-Completion answer (P8). The correlation identifier was reused while the read was still outstanding (P5). Or the returning identifier does not match what was sent, which points at the Tag mapping (Chapter 11.3 §6).
The observation that separates them: compare the Request's identifier against the Completion's. If they match and the entry is invalid, it was freed too early. If they differ, the mapping is wrong.
6. The Completion correlates but the wrong local client gets the data
The retained destination context, or the response router.
The correlation succeeded, so the read was found. What failed is the mapping from that entry to a local destination — check local_dest as written at allocation against what the result carried.
A specific and common cause: the destination was taken from the arriving Completion's order rather than from the entry. That works perfectly until two reads are outstanding, which is why §16 makes multiple-outstanding a required test rather than a stress case.
7. Partial data arrives but the read never retires
Progress accounting.
Check received_dw and remaining_dw against what actually arrived. Three shapes: if remaining_dw never reaches zero, chunks are being under-counted or one was dropped. If it underflowed, the overrun check is missing and the comparison wrapped. If it reached zero but the entry did not free, the resolution condition is comparing the wrong things.
And check the last chunk specifically — a final partial chunk whose DW count is smaller than the others is exactly where an off-by-one lives.
18. Performance — Why Outstanding Depth Is the Parameter
Memory read performance depends on far more than link bandwidth, and the reason is entirely a consequence of the split structure.
19. Memory Read Versus Memory Write
Only as contrast — Chapter 12.2 owns the write path in full.
| Memory Read | Memory Write | |
|---|---|---|
| Transaction class | non-posted | posted (canonical) |
| Request payload | none | the data |
| Response | one or more Completions with Data | none normally |
| Requester state after transmit | retained until resolved | released |
| Bounded by | MRRS (the request) and MPS (the completions) | MPS (the payload) |
| Round trips | one | none |
Everything in this chapter's RTL exists because of the second and fourth rows. A write engine needs no context table, no correlation, no result buffer and no completion path — it needs a segmenter (Chapter 11.4 §10) and nothing else from this chapter.
That asymmetry is why "writes work but reads fail" is such a strong diagnostic (Chapter 11.5 §14): a write exercises the outbound address path and nothing else.
20. Common Misconceptions
- "A Memory Read Request carries the requested data." It carries no payload — its Fmt says no data. The data returns in Completions travelling the other way (§3).
- "A memory read behaves like a synchronous bus read." The link is not held. Between Request and Completion, arbitrary other traffic crosses in both directions (§2).
- "The Request can be forgotten after transmit." It is outstanding. Correlation state, destination context and progress must be retained until it resolves (§7).
- "One Memory Read always gets exactly one Completion." MRRS bounds the Request and MPS bounds each Completion. MRRS greater than MPS is normal, and then the answer arrives in pieces (§8).
- "Completion arrival order identifies the Request." Order is not identity. The correlation field is (§2). This is what P5, P8 and P9 exist for.
- "MPS and MRRS are the same control." Different register fields bounding different quantities. MPS does not bound a read request, which has no payload (§8).
- "The BAR value goes into the Requester ID." The Requester ID is a Bus/Device/Function identity and it is the return path (Chapter 11.5 §7). A BAR is an address window and has nothing to do with it.
- "The Endpoint routes the packet using its BAR." The fabric routes it using Switch windows; the Endpoint's BAR decode runs after arrival, to find the internal offset (§5).
- "A Completion must return before any later Request can issue." Multiple reads may be outstanding — that is the entire point of correlation, and
CTXSis how many (§7, §18). - "Protocol resolution and local acceptance are the same event." Resolution is all data arrived; delivery is the client took it. Conflating them either wastes correlation identifiers on a stalled client or discards data (§12).
- "Returned data can be discarded if the local consumer is stalled." A Completion cannot be backpressured, so the data must be buffered. Discarding it produces a read that succeeded at the protocol level and returned nothing (§12).
- "A Completion timeout means the Request should be resent." A timeout is an error condition with specification-defined handling — not an instruction to retry. The mechanisms and required responses belong to later error-handling material.
21. Understanding Check
22. What's Next
This chapter traced one Memory Read from a software load through Request construction, context allocation, address routing, BAR decode, resource access, Completion return, correlation and local delivery — and found that the interesting engineering is almost all in the parts that are not the packet: the state that has to survive the wait, the identifier that must not be reused, the distinction between an answer arriving and a client taking it.
Chapter 12.2 takes the Memory Write path in full: posted semantics, the payload the Request carries, and why almost none of this chapter's machinery is needed for it.
Chapter 12.3 goes deeper into the Completion return pipeline that this chapter treated as a single arrow — return packetisation, the multi-stage path, and the flow-control interaction. Module 13 then owns the Completion packet itself: status, byte count, lower address, and the rules governing how an answer may be split. Chapter 12.5 takes the performance model that §18 only gestured at.
The idea to carry forward: a split transaction is not a slow transaction — it is two independent packet events joined only by state the Requester chose to keep, and every bug in this chapter is a bug in keeping it.