PCIe · Module 12
Examples — Reasoning Through Whole Transactions
Six worked traces that combine routing, packet type, attributes, payload, outstanding state and completion handling — with the trace tables, the scoreboard's view, the bug signatures, and a fault-injection harness that shows which invariant kills which fault.
Everything needed to reason about PCIe memory traffic is now on the table. Chapter 12.1 traced a read, 12.2 traced a write, 12.3 traced the return path.
This chapter stops explaining and starts working.
Can you now reason through complete PCIe memory transactions without being guided field by field?
1. The Shared Setup
All six examples use one configuration, so the traces can be compared directly.
| Item | Value |
|---|---|
| Endpoint BAR base | 0x8000_0000 |
| BAR size | 64 KB, memory space, enabled |
| Register block | offsets 0x100 – 0x1FF |
Outstanding read contexts (CTXS) | 4 |
| Operative MPS | 128 bytes = 32 DW |
| Operative MRRS | 512 bytes = 128 DW |
| Datapath | 4 DW per beat |
2. Example 1 — A Simple Memory Write
Setup. Software writes 0xA5A5_5A5A to 0x8000_0120, all four bytes valid.
Question to answer before reading on: at what event may each buffer free the data it holds?
Trace
| # | Event | Interface | Owner of address + payload |
|---|---|---|---|
| 1 | producer offers the write | wr_valid=1, wr_ready=1 | producer → posted buffer |
| 2 | producer moves on | — | posted buffer alone |
| 3 | buffer offers the descriptor | tx_valid=1, tx_ready=0 | posted buffer (stalled) |
| 4 | TX path accepts | tx_valid=1, tx_ready=1 | outbound TX |
| 5 | packet routed by address | — | fabric |
| 6 | Endpoint receives; BAR hit | — | Endpoint receive |
| 7 | front end offers to resource | res_valid=1, res_ready=0 | front end (stalled) |
| 8 | resource accepts | res_valid=1, res_ready=1 | resource |
| 9 | register at 0x120 = 0xA5A5_5A5A | — | done |
| 10 | nothing returns | — | — |
Answers
The producer may reuse its data bus after step 1 — the handshake discharged its obligation (Chapter 12.2 §16). The buffer may free its entry after step 4, not step 3: an offer is not a transfer. The front end holds until step 8.
And the trap in step 7: the resource must not update while res_ready is low. A design that applies the write on res_valid writes once per stalled cycle (Chapter 12.2 §11).
Monitor vs. scoreboard
| Monitor sees | Scoreboard expects |
|---|---|
one local write accepted, address 0x8000_0120, data 0xA5A5_5A5A | store {addr, data, mask} keyed by shadow ID |
| one outbound Memory Write descriptor | same address, same data |
one resource write, offset 0x120 | offset = 0x8000_0120 − 0x8000_0000 — computed by the scoreboard's own copy of the BAR base |
| zero Completions | zero Completions — asserted positively, not merely unobserved |
Bug signatures
| Symptom | First suspect |
|---|---|
| register never changes | walk §13's ladder from step 1 |
| register changes twice | update on valid not valid && ready (step 8) |
| right address, wrong data | address and payload came apart (Chapter 12.2 §4) |
| a Completion appears | the write was classified non-posted (Chapter 11.7 §7) |
3. Example 2 — A Simple Memory Read
Setup. Software reads 1 DW from 0x8000_0124.
Trace
| # | Event | pending | ctx[0] | Result buffer | Client |
|---|---|---|---|---|---|
| 1 | read offered, accepted | — | — | — | waiting |
| 2 | ctx 0 RESERVED, descriptor captured | valid | RESERVED | — | waiting |
| 3 | TX stalled; offer held | valid | RESERVED | — | waiting |
| 4 | Request launches | empty | OUTSTANDING, remaining 1 | — | waiting |
| 5 | routed by address; BAR hit; offset 0x124 | empty | OUTSTANDING | — | waiting |
| 6 | resource read; response available | empty | OUTSTANDING | — | waiting |
| 7 | Completion descriptor built, queued | empty | OUTSTANDING | — | waiting |
| 8 | Completion transmitted, routed by ID | empty | OUTSTANDING | — | waiting |
| 9 | correlated to ctx 0; 1 DW credited | empty | retires | 1 entry | waiting |
| 10 | client accepts | empty | FREE | empty | done |
What each step tests
Steps 2–4 are the reservation contract (Chapter 12.1 §9a). During step 3 the offered ctx_id must not move, even if another read resolves and frees a lower-numbered context.
Steps 6–8 are Chapter 12.3 §3. Step 6 is data available; step 8 is packet sent. Three boundaries separate them.
Steps 9 and 10 are different events. The context frees at 9; the buffer frees at 10 (Chapter 12.1 §12).
Monitor vs. scoreboard
The scoreboard stores {ctx_id → address, expected_dw, destination} at step 2 and compares at step 10. Expected data comes from its own decorrelated address→data function, never from the DUT's memory model (Chapter 12.1 §16) — so a single wrong address bit produces completely different data rather than data that differs in the same bit.
4. Example 3 — Two Outstanding Reads, Second Returns First
Setup. Read A — 1 DW from 0x8000_0130, ctx 0, destination CLIENT_X. Read B — 1 DW from 0x8000_0140, ctx 1, destination CLIENT_Y. B's Completion returns first.
Questions to answer before reading on: which context gets B's data, why can arrival order not be used, when may ctx 1 be reused, and what happens if A's context was freed early?
Trace
| # | Event | ctx 0 (A) | ctx 1 (B) |
|---|---|---|---|
| 1 | A launches | OUTSTANDING, rem 1 | — |
| 2 | B launches | OUTSTANDING, rem 1 | OUTSTANDING, rem 1 |
| 3 | B's Completion arrives | OUTSTANDING | correlated → retires |
| 4 | B's data → CLIENT_Y | OUTSTANDING | FREE (reusable next cycle) |
| 5 | A's Completion arrives | correlated → retires | FREE |
| 6 | A's data → CLIENT_X | FREE | FREE |
Answers
Which context gets B's data: ctx 1, because B's Completion carries the correlation field copied from B's Request. Nothing about the ordering enters the decision.
Why arrival order cannot be used. The two reads target different addresses that may be served by different resources with different latencies, across paths with different congestion. PCIe promises nothing about the return order of independent transactions — they are independent transactions that happen to share a Requester.
When ctx 1 may be reused: the cycle after step 4. The free-entry search sees the cleared occupancy on its next evaluation, so there is no same-cycle reuse window in which A's straggler could match a newly reserved read.
What breaks if A's context was freed early. Suppose a bug frees ctx 0 at step 3. A new read C reserves ctx 0 at step 4. A's Completion arrives at step 5 and resolves against C. A's data is delivered to C's client; A never completes and eventually times out. Two reads corrupted, no error reported — Chapter 10.4 §5, reached from the return path.
The order-based design, traced
This is the shortcut worth seeing fail explicitly. A design that pops a FIFO of pending reads on each Completion:
| # | Order-based design does | Correct behaviour |
|---|---|---|
| 3 | pops the head — A — and gives it B's data | credits ctx 1 |
| 3 | delivers to CLIENT_X | delivers to CLIENT_Y |
| 5 | pops B and gives it A's data | credits ctx 0 |
| 5 | delivers to CLIENT_Y | delivers to CLIENT_X |
Both clients receive plausible data from the wrong address. Both reads report success. And with one read outstanding, the design is indistinguishable from a correct one — which is why §12's two-outstanding-with-reversed-return test is required rather than optional.
5. Example 4 — A Split Return
Setup. One read of 64 DW from 0x8000_1000, ctx 2. MPS is 32 DW, so one Completion cannot carry the whole answer.
MRRS is 128 DW, so the Request is legal — MRRS bounds what a read may ask for; MPS bounds what a Completion may carry (Chapter 12.1 §8).
Trace
| # | Event | remaining_dw | active | read_retire |
|---|---|---|---|---|
| 1 | read opens, expecting 64 DW | 64 | yes | — |
| 2 | chunk 1 arrives, 32 DW | 32 | yes | no |
| 3 | chunk 1's data buffered for the client | 32 | yes | no |
| 4 | chunk 2 arrives, 32 DW | 0 | no | yes |
| 5 | result delivered, context free | — | — | — |
What this example is for
Step 2 is the entire point. Valid data has arrived and the read is not finished. A design that retires here frees the correlation index while chunk 2 is in flight (Chapter 12.3 §5).
And step 4 is the other half. A design that never retires — because its condition compares the wrong things — holds ctx 2 forever, and read concurrency drops from 4 to 3 permanently. P5 forbids the first; P6 forbids the second.
How the chunks are placed. Chunk 1 covers 0x8000_1000–0x8000_107F; chunk 2 covers 0x8000_1080 onward. The placement comes from the context's received_dw, not from arrival (Chapter 12.1 §11, P9).
What is deliberately not claimed here. How many Completions a Completer will actually produce, at what boundaries it may divide, and whether this particular division is the one it must choose — Module 13 owns those rules. This example shows the Requester-side consequence of division happening, which is the part Chapter 12.3 owns.
6. Example 5 — Routing Failure vs. BAR-Decode Failure
Two failures that produce similar complaints and require completely different investigations.
Version A — the packet never reaches the Endpoint
Software writes to 0x9000_0000. The register never changes. Nothing appears at the Endpoint's receive interface.
Question: is BAR decode relevant?
No — and this is the discipline the whole chapter is built on. The packet never arrived, so nothing at the Endpoint has run. BAR decode cannot be the cause of a packet that was never delivered to the thing that would have decoded it.
| Inspect | Not yet relevant |
|---|---|
| each Switch port's Base/Limit windows | the Endpoint's BAR value |
| whether the address falls in any downstream window | bar_enabled |
| whether the packet went upstream instead (Chapter 11.5 §4) | the internal offset |
The likely cause: 0x9000_0000 is outside every downstream window, so the packet was forwarded upstream — which is the default direction, not an error. The address is wrong, or the windows were assigned wrong (Chapter 9.5).
Version B — the packet reaches the Endpoint but no BAR hit
Software writes to 0x8000_0120. The register never changes. The packet is observed at the Endpoint's receive interface with the correct address, and unclaimed_write asserts.
Now the boundaries flip.
| Proven correct | Inspect |
|---|---|
| the address in the packet | the BAR value the Endpoint actually holds |
| every Switch's routing decision | bar_enabled — is Memory Space enabled? |
| the fabric path | the decode comparison and the offset computation |
The strongest clue is that this state should not be reachable. The fabric forwarded the packet because a window said this address lives here; the Endpoint says it does not. Windows and BARs disagree, which is a configuration fault, not a decode bug.
Or, far more often: the BAR is right and Memory Space is simply not enabled — an ordinary bring-up mistake that produces exactly this signature.
Why the pair matters
The two versions produce the same user-visible complaint — "my write does not work" — and share no investigative steps. One observation separates them: does the packet appear at the Endpoint's receive interface?
7. Example 6 — A Backpressure Metadata Bug
Setup. A posted write is offered with Relaxed Ordering clear. The TX path stalls for 20 steps. During the stall, software writes a control register that changes the design's attribute policy. The packet departs with Relaxed Ordering set.
Trace
| # | Event | Stored attr | Config | Offered attr |
|---|---|---|---|---|
| 1 | write accepted, RO=0 | RO=0 | RO policy off | RO=0 |
| 2–20 | TX stalled | RO=0 | — | RO=0 (required) |
| 12 | software writes config, RO policy on | RO=0 | RO policy on | RO=1 ← bug |
| 21 | TX accepts | — | — | RO=1 departs |
The analysis
Which contract is violated: valid/ready payload stability. An offer's payload must not change while the offer is in progress. Attributes are payload (Chapter 11.6 §7).
Which architectural rule: attributes are owned packet state from acceptance. Configuration governs transactions at acceptance, not continuously. A block that consults live configuration for an in-flight packet has no coherent notion of when its policy was decided.
Which assertions: Chapter 11.6's P8a — everything the stage holds is stable while the consumer stalls — and P8, which anchors the verdict to the acceptance event rather than to the previous cycle. Chapter 12.2's P2 covers the same shape in the posted buffer.
What the scoreboard reports. It stored {addr, data, mask, attr} at step 1, keyed by shadow ID. At step 21 it compares and reports an attribute mismatch on a transaction whose address and data are perfect — which is exactly the right diagnostic, because it points at the metadata path rather than the datapath.
Why the consequence is real and not cosmetic. The Requester issued this write with default ordering because it had an ordering relationship that mattered. The packet that reached the fabric carries permission to relax exactly the constraints that relationship depended on (Chapter 11.6 §4). The resulting corruption is intermittent, load-dependent, and has no protocol error attached to it.
Why it survives testing. With no backpressure, acceptance and departure are close enough that a configuration write almost never lands between them. It needs a stall and a concurrent configuration write — two stimuli a directed test does not naturally combine.
8. Trace Signals
Illustrative internal RTL interfaces from Chapters 12.1–12.3. Not PCIe wire signals.
A Memory Read, one chunk, with a TX stall and a client stall:
step 1 2 3 4 5 6 7 8 9 10 11 12
loc_valid 1 1 0 0 0 0 0 0 0 0 0 0
loc_ready 1 0 0 0 0 0 0 0 0 0 0 0
req_valid 0 1 1 1 0 0 0 0 0 0 0 0
req_ready 0 0 0 1 0 0 0 0 0 0 0 0
ctx_state - R R O O O O O O - - - R=reserved O=outstanding
cpl_valid 0 0 0 0 0 0 0 1 0 0 0 0
cpl_ctx - - - - - - - 0 - - - -
res_valid 0 0 0 0 0 0 0 0 1 1 1 0
res_ready 0 0 0 0 0 0 0 0 0 0 1 0Read steps 2–4 carefully. req_valid is high for three steps and req_ready for one. ctx_state is R throughout, and the offered ctx_id must be identical at every one of them — that is Chapter 12.1's P13a, and the trace is what it looks like when satisfied.
And steps 9–11. res_valid is high for three steps with res_ready high for one. Exactly one delivery occurs, at step 11.
A posted Memory Write with a resource stall:
step 1 2 3 4 5 6 7 8
wr_valid 1 0 0 0 0 0 0 0
wr_ready 1 0 0 0 0 0 0 0
tx_valid 0 1 1 1 0 0 0 0
tx_ready 0 0 0 1 0 0 0 0
res_valid 0 0 0 0 0 1 1 1
res_ready 0 0 0 0 0 0 0 1
tgt_update 0 0 0 0 0 0 0 1 <-- EXACTLY ONE, at step 8tgt_update is the row that matters. res_valid is high for three steps. A design that updated on valid would show three pulses — Chapter 12.2 §11, visible in one trace.
9. Verification Harness
VERIFICATION-ONLY. Not synthesizable, not a production model. It composes the blocks the previous three chapters built so their interaction can be exercised.
// VERIFICATION-ONLY. A minimal composition harness: request generator, a toy
// fabric that can delay and reorder, an endpoint resource model, a completion
// return path, and an independent scoreboard.
// This is NOT a PCIe endpoint and NOT synthesizable. It models only enough
// behaviour to exercise the interactions taught in Chapters 12.1-12.3.
module mem_txn_harness;
localparam int CTXS = 4;
localparam int ADDR_W = 64;
// ---- Independent data model -----------------------------------------
// NOT the DUT's memory. A decorrelated function, so ANY address error
// produces completely different data rather than data that differs in the
// same bit the address did.
function automatic logic [31:0] model_data (input logic [ADDR_W-1:0] a);
logic [31:0] h;
h = a[31:0] ^ 32'h9E37_79B9;
h = (h << 13) ^ (h >> 7) ^ a[63:32];
h = h * 32'h85EB_CA6B;
return h ^ (h >> 16);
endfunction
// ---- Scoreboard state — owned entirely by the testbench --------------
typedef struct {
bit active;
logic [ADDR_W-1:0] addr;
int expected_dw;
int received_dw;
int dest;
} sb_read_t;
sb_read_t sb_read [CTXS]; // NEVER read from the DUT context table
typedef struct {
logic [ADDR_W-1:0] addr;
logic [31:0] data;
logic [3:0] mask;
} sb_write_t;
sb_write_t sb_write [$]; // expected writes, in issue order
int applied_writes;
int observed_completions_for_writes;
// ---- Toy fabric: delay and optional reorder --------------------------
// Illustrative only. Real fabric behaviour is far richer; this exists so
// the harness can produce out-of-order returns deterministically.
typedef struct {
int delay;
int ctx;
int dw;
} fab_item_t;
fab_item_t fabric_q [$];
task automatic fabric_push (input int ctx, input int dw, input int delay);
fabric_q.push_back('{delay: delay, ctx: ctx, dw: dw});
endtask
// Each tick, decrement delays and release anything that has expired.
// Because delays differ per item, items are RELEASED OUT OF ORDER — which
// is the whole reason the harness exists.
task automatic fabric_tick (output bit fire, output int ctx, output int dw);
fire = 0;
foreach (fabric_q[i]) if (fabric_q[i].delay > 0) fabric_q[i].delay--;
foreach (fabric_q[i])
if (fabric_q[i].delay == 0) begin
fire = 1; ctx = fabric_q[i].ctx; dw = fabric_q[i].dw;
fabric_q.delete(i);
break;
end
endtask
// ---- Scoreboard checks ----------------------------------------------
// Called on every credited chunk. Uses ONLY testbench state.
task automatic sb_credit (input int ctx, input int dw, input bit dut_retire);
bit expect_retire;
// Bounds first, for the reason Chapter 12.3 section 8a gives: an
// out-of-range identifier is a different fault from an inactive one,
// and indexing before checking hides it.
if ((ctx < 0) || (ctx >= CTXS)) begin
$error("chunk credited to out-of-range context %0d (CTXS=%0d)",
ctx, CTXS);
return;
end
if (!sb_read[ctx].active) begin
$error("chunk credited to inactive context %0d", ctx);
return;
end
sb_read[ctx].received_dw += dw;
if (sb_read[ctx].received_dw > sb_read[ctx].expected_dw)
$error("ctx %0d overrun: %0d > %0d",
ctx, sb_read[ctx].received_dw, sb_read[ctx].expected_dw);
expect_retire = (sb_read[ctx].received_dw == sb_read[ctx].expected_dw);
// THE integration check: the DUT's retire decision must agree with an
// independently-computed one. A DUT that retires early disagrees here on
// the first partial chunk of a split return (Example 4).
if (dut_retire != expect_retire)
$error("ctx %0d retire mismatch: dut=%0b expected=%0b",
ctx, dut_retire, expect_retire);
if (expect_retire) sb_read[ctx].active = 0;
endtask
// Called on every observed resource write.
task automatic sb_apply_write (input logic [ADDR_W-1:0] a,
input logic [31:0] d,
input logic [3:0] m);
sb_write_t e;
if (sb_write.size() == 0) begin
$error("resource write with no expected write outstanding -- DUPLICATE");
return;
end
e = sb_write.pop_front();
// Compared as a PAIR. Checking address and data separately would pass
// the split-ownership failure of Chapter 12.2 section 4.
if ((e.addr != a) || (e.data != d) || (e.mask != m))
$error("write mismatch: expected %h/%h/%h got %h/%h/%h",
e.addr, e.data, e.mask, a, d, m);
applied_writes++;
endtask
// End-of-test checks, including the NEGATIVE one.
task automatic sb_final;
foreach (sb_read[i])
if (sb_read[i].active)
$error("ctx %0d never completed: %0d of %0d DW",
i, sb_read[i].received_dw, sb_read[i].expected_dw);
if (sb_write.size() != 0)
$error("%0d expected writes never reached the resource",
sb_write.size());
// A posted write producing no Completion is EXPECTED BEHAVIOUR, so it
// needs a positive check. "We did not look" is not the same as
// "correctly absent".
if (observed_completions_for_writes != 0)
$error("%0d Completions observed for posted writes",
observed_completions_for_writes);
endtask
endmoduleClassification: verification-only.
What it models: correlation, progress accounting, out-of-order return, write application, and an independent data model. What it does not model: any wire format, flow control, ordering rules, error handling, or timing.
The three things worth copying from it. model_data is decorrelated, so an address error is loud rather than subtle. sb_apply_write compares the pair, so split-ownership corruption cannot pass. And sb_final asserts the absence of write Completions positively — a check that is easy to omit and impossible to recover afterwards.
10. Fault Injection
// VERIFICATION-ONLY. Deliberate fault switches, for demonstrating which
// invariant catches which fault. NEVER synthesize this; never leave it
// enabled in a regression that is checking for real bugs.
typedef enum {
FAULT_NONE,
FAULT_DROP_REQUEST, // Request offered but never launched
FAULT_CORRUPT_CTX_ID, // correlation index flipped on the return path
FAULT_DUPLICATE_WRITE, // target updated on valid, not valid && ready
FAULT_DROP_COMPLETION, // a Completion chunk silently discarded
FAULT_CORRUPT_ADDRESS, // one address bit flipped in the Request
FAULT_EARLY_RETIRE, // context retired on any credited chunk
FAULT_ORDER_ROUTING // return data routed by arrival order, not by ID
} fault_e;
fault_e injected = FAULT_NONE;
// Applied at the harness boundary, never inside a DUT module.
function automatic logic [ADDR_W-1:0] maybe_corrupt_addr (
input logic [ADDR_W-1:0] a);
return (injected == FAULT_CORRUPT_ADDRESS) ? (a ^ (1 << 6)) : a;
endfunction
function automatic int maybe_corrupt_ctx (input int c);
return (injected == FAULT_CORRUPT_CTX_ID) ? ((c + 1) % CTXS) : c;
endfunctionWhich invariant kills which fault:
| Fault | Caught by | How it presents |
|---|---|---|
FAULT_DROP_REQUEST | sb_final — context never completes | one read hangs; concurrency drops by one permanently |
FAULT_CORRUPT_CTX_ID | sb_credit — retire mismatch, or credit to an inactive context | data reaches the wrong client; with CTXS=1 it is invisible |
FAULT_DUPLICATE_WRITE | sb_apply_write — "write with no expected write outstanding" | target state wrong; only under resource stall |
FAULT_DROP_COMPLETION | sb_final — partial received_dw | read hangs; looks identical to a lost Request until you check whether the Request left |
FAULT_CORRUPT_ADDRESS | data comparison — model_data is decorrelated, so the mismatch is total | wrong data, right transaction structure |
FAULT_EARLY_RETIRE | sb_credit — retire mismatch on the first partial chunk | fires immediately on Example 4's trace |
FAULT_ORDER_ROUTING | destination comparison, only with two reads returning out of order | both clients get plausible wrong data (Example 3) |
11. Integration Assertions
Deliberately not a repeat of Chapters 12.1–12.3's properties. These are the ones that only make sense across block boundaries.
// INTEGRATION SVA. Cross-block properties over the harness. LOCAL contracts
// of the composed teaching model — not claims about PCIe legality.
// I1: CONSERVATION ACROSS THE WRITE PATH. The target is never updated more
// times than writes were accepted. Catches duplication anywhere in the chain.
property p_no_write_amplification;
@(posedge clk) disable iff (!rst_n)
(applied_writes <= accepted_writes);
endproperty
a_no_amplification : assert property (p_no_write_amplification);
// I2: CLASS EXCLUSIVITY. No transaction is both posted and Completion-
// tracked in the teaching model. Only expressible across the two paths.
property p_posted_never_tracked;
@(posedge clk) disable iff (!rst_n)
posted_write_accepted |-> !ctx_reserve;
endproperty
a_class_exclusive : assert property (p_posted_never_tracked);
// I3: a resolved read corresponds to a read that was previously accepted.
// Rules out a retirement conjured from a Completion for a read never issued.
property p_resolution_has_provenance;
@(posedge clk) disable iff (!rst_n)
read_retire |-> sb_read[chunk_id].active;
endproperty
a_provenance : assert property (p_resolution_has_provenance);
// I4: the local context delivered with a result matches the one recorded at
// issue. The end-to-end form of Example 3's lesson.
property p_local_context_round_trip;
@(posedge clk) disable iff (!rst_n)
(out_valid && out_ready) |-> (out_dest == sb_read[out_ctx].dest);
endproperty
a_ctx_round_trip : assert property (p_local_context_round_trip);
// I5: DEBUG BOUNDARY DISTINGUISHABILITY. A packet that reached the Endpoint
// and found no resource is observably DIFFERENT from a packet that never
// reached it. Example 5 depends on this being true of the design, so it is
// worth asserting rather than assuming.
property p_arrival_distinguishable;
@(posedge clk) disable iff (!rst_n)
ep_rx_valid |-> (ep_bar_hit || unclaimed_write);
endproperty
a_distinguishable : assert property (p_arrival_distinguishable);
// I6: NO CROSS-TRANSACTION CONTAMINATION. Data credited to one context never
// emerges against another's destination.
property p_no_contamination;
@(posedge clk) disable iff (!rst_n)
(out_valid && out_ready) |-> (out_data == model_data(sb_read[out_ctx].addr));
endproperty
a_no_contamination : assert property (p_no_contamination);Liveness, with every assumption stated:
// The fabric, the Completer and the local consumer are all environment.
// PCIe guarantees none of these, which is why Completion timeout exists.
assume property (@(posedge clk) disable iff (!rst_n)
req_valid |-> s_eventually req_ready); // TX eventually accepts
assume property (@(posedge clk) disable iff (!rst_n)
fabric_has_item |-> s_eventually fabric_fires); // fabric eventually forwards
assume property (@(posedge clk) disable iff (!rst_n)
res_valid |-> s_eventually res_ready); // target eventually ready
assume property (@(posedge clk) disable iff (!rst_n)
out_valid |-> s_eventually out_ready); // consumer eventually ready
// L1: under those four, an accepted write reaches the target.
property p_write_reaches_target;
@(posedge clk) disable iff (!rst_n)
posted_write_accepted |-> s_eventually (res_valid && res_ready);
endproperty
// L2: under those four, an accepted read delivers a result.
property p_read_delivers;
@(posedge clk) disable iff (!rst_n)
(loc_valid && loc_ready) |-> s_eventually (out_valid && out_ready && out_final);
endpropertyI1 is a counting property because a value check cannot see duplication. Ten identical writes to an idempotent register leave the same final state as one. Only the count distinguishes them, which is why the scoreboard tracks it.
I5 is unusual and worth keeping. It asserts that a debugging distinction is real — that Example 5's two versions are actually separable by observation. A design where an unmatched packet is silently swallowed at the receive boundary would fail it, and the consequence is that §13's method breaks: there would be no observation distinguishing "never arrived" from "arrived and was discarded."
I6 is the end-to-end contamination check, and it works only because model_data is decorrelated: data from a neighbouring address produces a completely different value rather than a nearly-identical one.
12. Verification Exercises
Do these against the modules in Chapters 12.1–12.3. Each has a specific answer.
Task A — reuse an active correlation index
Mutate the context table so the free-entry search can return an index that is still occupied.
Which check fails first, and under what stimulus? — Chapter 12.1's P5 fails immediately on the reservation, before any Completion returns. If P5 is not bound, the first observable failure is a retire mismatch in sb_credit, and only with two reads outstanding — with one, the reuse is harmless because there is nothing to collide with.
Task B — reject full+pop replacement
Change the result buffer's acceptance to push = res_valid && !full.
What symptom appears? — No functional failure at low rates. Under sustained full-rate return traffic, overflow_error sets and results are lost, and the reads they belonged to never complete. The signature is that it only happens when the buffer is full — that is, when the design is busiest. Chapter 12.1's P12a catches it directly.
Task C — update the target on valid
Drive the resource's write side effect from res_valid instead of res_valid && res_ready.
Which assertion catches the duplication? — Chapter 12.2's P8 fires on the first stalled cycle. I1 catches it in aggregate. sb_apply_write reports "write with no expected write outstanding" on the second application. And with a never-stalling resource, none of them fires — the stimulus must include a resource stall.
Task D — swap two Completion context IDs
Apply FAULT_CORRUPT_CTX_ID so chunks for ctx n are credited to ctx n+1.
What does the monitor see versus what does the scoreboard report? — The monitor sees two perfectly well-formed Completions, each with a valid correlation index, each credited to an active context. Nothing at the packet level is wrong. The scoreboard reports a retire mismatch — because its independently-tracked received_dw for each context no longer matches the DUT's retire decision. The gap between those two views is the entire argument for scoreboard independence: a scoreboard mirroring the DUT's accounting would agree with it and report nothing.
Task E — corrupt routing but not BAR decode
Make the fabric deliver a packet to the wrong Endpoint while leaving every BAR correct.
Which debug boundary localizes it? — The receive-interface observation of Example 5. The packet arrives somewhere, so it is not a routing-to-nowhere failure; it arrives at an Endpoint whose BAR does not match, so unclaimed_write asserts at the wrong component. The localizing question is not "did it arrive" but "did it arrive here" — and the answer identifies the fabric immediately, without touching any BAR.
Task F — split ownership of a posted write
Give the posted buffer separate metadata and payload handshakes with no association mechanism.
What must the scoreboard do to catch it? — Compare the {address, payload} pair, keyed by shadow ID. Two independent checks both pass: every address requested was written, and every payload supplied was delivered. Only the pairing is wrong (Chapter 12.2 §16). And the stimulus must make the payload path slower than the metadata path with two writes close enough to overlap.
13. The Debugging Method
One method, used in every example above.
Find the last boundary where identity, address, payload and ownership are all correct. Then inspect the next boundary.
Three corollaries, each of which saves an afternoon:
- Do not debug the physical layer if the packet never left the Requester. If
req_valid && req_readynever happened, nothing below the Transaction Layer has seen this transaction. - Do not debug BAR decode if the Switch sent the packet to the wrong port. BAR decode runs after arrival; a packet that arrived elsewhere never reached it (Example 5).
- Do not debug Completion fields if the target never produced data. Chapter 12.3 §3's boundary 1 precedes everything about the return packet.
The boundary list, in order, for the two transaction classes:
| # | Memory Read | Memory Write |
|---|---|---|
| 1 | local read accepted | local write accepted |
| 2 | context reserved | buffered with payload |
| 3 | Request launched | Request launched |
| 4 | routed to the Endpoint | routed to the Endpoint |
| 5 | BAR hit, offset computed | BAR hit, offset computed |
| 6 | resource read, data available | resource handshake |
| 7 | Completion descriptor built | target updated — once |
| 8 | Completion transmitted | (nothing returns) |
| 9 | correlated at the Requester | — |
| 10 | progress accounted; retire decided | — |
| 11 | result delivered to the client | — |
Walk it in order. The first row that did not happen is where to look, and every row above it is proven correct by the fact that the next one was reached.
14. Common Misconceptions
- "A timing diagram implies fixed PCIe latency." The traces are event sequences, not cycle counts. Fabric, Completer and consumer delay all vary (§1).
- "A second Completion arriving first is a protocol error." Independent transactions may be served by different resources across different paths. Nothing promises return order (Example 3).
- "A posted write has no state." It has forward-path state at every boundary — just no correlation state (Chapter 12.2 §1).
- "A Memory Read Request carries return payload." It carries no payload at all (Chapter 12.1 §3).
- "A read is done when the target's data becomes available." That is boundary 6 of eleven. Five boundaries and a queue separate it from delivery (§13).
- "A write is done when the local producer drops
valid." That means the producer's handshake completed. The target may not have been touched (Example 1). - "A packet reaching the Endpoint means the BAR hit." Arrival and decode are separate, and
unclaimed_writeis the observable difference (Example 5). - "A BAR hit means the resource consumed the request." The front end still has to hand it over, and the resource can stall (Example 1, step 7).
- "A Completion arriving means the local consumer accepted the data." Retirement and delivery are different events (Chapter 12.1 §12).
- "Queue occupancy and cumulative transaction count are the same metric." One is live state and one is a running total (Chapter 10.5). Occupancy is not expected to be monotonic.
- "Correctness can be proven with happy-path traces." Every bug in this chapter needs a stall, a reorder, or a concurrent configuration write to appear.
- "The testbench may use the DUT's context table as the expected model." Then it agrees with the DUT by construction and verifies nothing (Task D).
15. Understanding Check
16. What's Next
Module 12 has covered memory traffic end to end: 12.1's split read, 12.2's posted write, 12.3's return pipeline, and this chapter's integration of all three.
Chapter 12.5 closes the module with the performance implications these chapters kept setting aside: read latency and outstanding depth, posted-write throughput, packet-size overhead, Completion return bandwidth, and how latency and bandwidth interact on a real path.
Module 13 then takes the Completion packet itself — the Cpl and CplD forms, Completion Status, Byte Count and Lower Address, the rules governing how a response may be split, and the ordering matrix that Chapter 11.6's attributes are permitted to relax.
The idea to carry forward: every transaction is a chain of ownership transfers, and every bug in this chapter is one boundary asserting an ownership it did not have.