Skip to content

PCIe · Module 12

Examples — Reasoning Through Whole Transactions

Six worked traces that combine routing, packet type, attributes, payload, outstanding state and completion handling — with the trace tables, the scoreboard's view, the bug signatures, and a fault-injection harness that shows which invariant kills which fault.

Everything needed to reason about PCIe memory traffic is now on the table. Chapter 12.1 traced a read, 12.2 traced a write, 12.3 traced the return path.

This chapter stops explaining and starts working.

Can you now reason through complete PCIe memory transactions without being guided field by field?

1. The Shared Setup

All six examples use one configuration, so the traces can be compared directly.

ItemValue
Endpoint BAR base0x8000_0000
BAR size64 KB, memory space, enabled
Register blockoffsets 0x1000x1FF
Outstanding read contexts (CTXS)4
Operative MPS128 bytes = 32 DW
Operative MRRS512 bytes = 128 DW
Datapath4 DW per beat

2. Example 1 — A Simple Memory Write

Setup. Software writes 0xA5A5_5A5A to 0x8000_0120, all four bytes valid.

Question to answer before reading on: at what event may each buffer free the data it holds?

Trace

#EventInterfaceOwner of address + payload
1producer offers the writewr_valid=1, wr_ready=1producer → posted buffer
2producer moves onposted buffer alone
3buffer offers the descriptortx_valid=1, tx_ready=0posted buffer (stalled)
4TX path acceptstx_valid=1, tx_ready=1outbound TX
5packet routed by addressfabric
6Endpoint receives; BAR hitEndpoint receive
7front end offers to resourceres_valid=1, res_ready=0front end (stalled)
8resource acceptsres_valid=1, res_ready=1resource
9register at 0x120 = 0xA5A5_5A5Adone
10nothing returns

Answers

The producer may reuse its data bus after step 1 — the handshake discharged its obligation (Chapter 12.2 §16). The buffer may free its entry after step 4, not step 3: an offer is not a transfer. The front end holds until step 8.

And the trap in step 7: the resource must not update while res_ready is low. A design that applies the write on res_valid writes once per stalled cycle (Chapter 12.2 §11).

Monitor vs. scoreboard

Monitor seesScoreboard expects
one local write accepted, address 0x8000_0120, data 0xA5A5_5A5Astore {addr, data, mask} keyed by shadow ID
one outbound Memory Write descriptorsame address, same data
one resource write, offset 0x120offset = 0x8000_0120 − 0x8000_0000computed by the scoreboard's own copy of the BAR base
zero Completionszero Completions — asserted positively, not merely unobserved

Bug signatures

SymptomFirst suspect
register never changeswalk §13's ladder from step 1
register changes twiceupdate on valid not valid && ready (step 8)
right address, wrong dataaddress and payload came apart (Chapter 12.2 §4)
a Completion appearsthe write was classified non-posted (Chapter 11.7 §7)

3. Example 2 — A Simple Memory Read

Setup. Software reads 1 DW from 0x8000_0124.

Trace

#Eventpendingctx[0]Result bufferClient
1read offered, acceptedwaiting
2ctx 0 RESERVED, descriptor capturedvalidRESERVEDwaiting
3TX stalled; offer heldvalidRESERVEDwaiting
4Request launchesemptyOUTSTANDING, remaining 1waiting
5routed by address; BAR hit; offset 0x124emptyOUTSTANDINGwaiting
6resource read; response availableemptyOUTSTANDINGwaiting
7Completion descriptor built, queuedemptyOUTSTANDINGwaiting
8Completion transmitted, routed by IDemptyOUTSTANDINGwaiting
9correlated to ctx 0; 1 DW creditedemptyretires1 entrywaiting
10client acceptsemptyFREEemptydone

What each step tests

Steps 2–4 are the reservation contract (Chapter 12.1 §9a). During step 3 the offered ctx_id must not move, even if another read resolves and frees a lower-numbered context.

Steps 6–8 are Chapter 12.3 §3. Step 6 is data available; step 8 is packet sent. Three boundaries separate them.

Steps 9 and 10 are different events. The context frees at 9; the buffer frees at 10 (Chapter 12.1 §12).

Monitor vs. scoreboard

The scoreboard stores {ctx_id → address, expected_dw, destination} at step 2 and compares at step 10. Expected data comes from its own decorrelated address→data function, never from the DUT's memory model (Chapter 12.1 §16) — so a single wrong address bit produces completely different data rather than data that differs in the same bit.

4. Example 3 — Two Outstanding Reads, Second Returns First

Setup. Read A — 1 DW from 0x8000_0130, ctx 0, destination CLIENT_X. Read B — 1 DW from 0x8000_0140, ctx 1, destination CLIENT_Y. B's Completion returns first.

Questions to answer before reading on: which context gets B's data, why can arrival order not be used, when may ctx 1 be reused, and what happens if A's context was freed early?

Trace

#Eventctx 0 (A)ctx 1 (B)
1A launchesOUTSTANDING, rem 1
2B launchesOUTSTANDING, rem 1OUTSTANDING, rem 1
3B's Completion arrivesOUTSTANDINGcorrelated → retires
4B's data → CLIENT_YOUTSTANDINGFREE (reusable next cycle)
5A's Completion arrivescorrelated → retiresFREE
6A's data → CLIENT_XFREEFREE

Answers

Which context gets B's data: ctx 1, because B's Completion carries the correlation field copied from B's Request. Nothing about the ordering enters the decision.

Why arrival order cannot be used. The two reads target different addresses that may be served by different resources with different latencies, across paths with different congestion. PCIe promises nothing about the return order of independent transactions — they are independent transactions that happen to share a Requester.

When ctx 1 may be reused: the cycle after step 4. The free-entry search sees the cleared occupancy on its next evaluation, so there is no same-cycle reuse window in which A's straggler could match a newly reserved read.

What breaks if A's context was freed early. Suppose a bug frees ctx 0 at step 3. A new read C reserves ctx 0 at step 4. A's Completion arrives at step 5 and resolves against C. A's data is delivered to C's client; A never completes and eventually times out. Two reads corrupted, no error reportedChapter 10.4 §5, reached from the return path.

The order-based design, traced

This is the shortcut worth seeing fail explicitly. A design that pops a FIFO of pending reads on each Completion:

#Order-based design doesCorrect behaviour
3pops the head — A — and gives it B's datacredits ctx 1
3delivers to CLIENT_Xdelivers to CLIENT_Y
5pops B and gives it A's datacredits ctx 0
5delivers to CLIENT_Ydelivers to CLIENT_X

Both clients receive plausible data from the wrong address. Both reads report success. And with one read outstanding, the design is indistinguishable from a correct one — which is why §12's two-outstanding-with-reversed-return test is required rather than optional.

5. Example 4 — A Split Return

Setup. One read of 64 DW from 0x8000_1000, ctx 2. MPS is 32 DW, so one Completion cannot carry the whole answer.

MRRS is 128 DW, so the Request is legal — MRRS bounds what a read may ask for; MPS bounds what a Completion may carry (Chapter 12.1 §8).

Trace

#Eventremaining_dwactiveread_retire
1read opens, expecting 64 DW64yes
2chunk 1 arrives, 32 DW32yesno
3chunk 1's data buffered for the client32yesno
4chunk 2 arrives, 32 DW0noyes
5result delivered, context free

What this example is for

Step 2 is the entire point. Valid data has arrived and the read is not finished. A design that retires here frees the correlation index while chunk 2 is in flight (Chapter 12.3 §5).

And step 4 is the other half. A design that never retires — because its condition compares the wrong things — holds ctx 2 forever, and read concurrency drops from 4 to 3 permanently. P5 forbids the first; P6 forbids the second.

How the chunks are placed. Chunk 1 covers 0x8000_10000x8000_107F; chunk 2 covers 0x8000_1080 onward. The placement comes from the context's received_dw, not from arrival (Chapter 12.1 §11, P9).

What is deliberately not claimed here. How many Completions a Completer will actually produce, at what boundaries it may divide, and whether this particular division is the one it must choose — Module 13 owns those rules. This example shows the Requester-side consequence of division happening, which is the part Chapter 12.3 owns.

6. Example 5 — Routing Failure vs. BAR-Decode Failure

Two failures that produce similar complaints and require completely different investigations.

Version A — the packet never reaches the Endpoint

Software writes to 0x9000_0000. The register never changes. Nothing appears at the Endpoint's receive interface.

Question: is BAR decode relevant?

No — and this is the discipline the whole chapter is built on. The packet never arrived, so nothing at the Endpoint has run. BAR decode cannot be the cause of a packet that was never delivered to the thing that would have decoded it.

InspectNot yet relevant
each Switch port's Base/Limit windowsthe Endpoint's BAR value
whether the address falls in any downstream windowbar_enabled
whether the packet went upstream instead (Chapter 11.5 §4)the internal offset

The likely cause: 0x9000_0000 is outside every downstream window, so the packet was forwarded upstream — which is the default direction, not an error. The address is wrong, or the windows were assigned wrong (Chapter 9.5).

Version B — the packet reaches the Endpoint but no BAR hit

Software writes to 0x8000_0120. The register never changes. The packet is observed at the Endpoint's receive interface with the correct address, and unclaimed_write asserts.

Now the boundaries flip.

Proven correctInspect
the address in the packetthe BAR value the Endpoint actually holds
every Switch's routing decisionbar_enabled — is Memory Space enabled?
the fabric paththe decode comparison and the offset computation

The strongest clue is that this state should not be reachable. The fabric forwarded the packet because a window said this address lives here; the Endpoint says it does not. Windows and BARs disagree, which is a configuration fault, not a decode bug.

Or, far more often: the BAR is right and Memory Space is simply not enabled — an ordinary bring-up mistake that produces exactly this signature.

Why the pair matters

The two versions produce the same user-visible complaint — "my write does not work" — and share no investigative steps. One observation separates them: does the packet appear at the Endpoint's receive interface?

7. Example 6 — A Backpressure Metadata Bug

Setup. A posted write is offered with Relaxed Ordering clear. The TX path stalls for 20 steps. During the stall, software writes a control register that changes the design's attribute policy. The packet departs with Relaxed Ordering set.

Trace

#EventStored attrConfigOffered attr
1write accepted, RO=0RO=0RO policy offRO=0
2–20TX stalledRO=0RO=0 (required)
12software writes config, RO policy onRO=0RO policy onRO=1 ← bug
21TX acceptsRO=1 departs

The analysis

Which contract is violated: valid/ready payload stability. An offer's payload must not change while the offer is in progress. Attributes are payload (Chapter 11.6 §7).

Which architectural rule: attributes are owned packet state from acceptance. Configuration governs transactions at acceptance, not continuously. A block that consults live configuration for an in-flight packet has no coherent notion of when its policy was decided.

Which assertions: Chapter 11.6's P8a — everything the stage holds is stable while the consumer stalls — and P8, which anchors the verdict to the acceptance event rather than to the previous cycle. Chapter 12.2's P2 covers the same shape in the posted buffer.

What the scoreboard reports. It stored {addr, data, mask, attr} at step 1, keyed by shadow ID. At step 21 it compares and reports an attribute mismatch on a transaction whose address and data are perfect — which is exactly the right diagnostic, because it points at the metadata path rather than the datapath.

Why the consequence is real and not cosmetic. The Requester issued this write with default ordering because it had an ordering relationship that mattered. The packet that reached the fabric carries permission to relax exactly the constraints that relationship depended on (Chapter 11.6 §4). The resulting corruption is intermittent, load-dependent, and has no protocol error attached to it.

Why it survives testing. With no backpressure, acceptance and departure are close enough that a configuration write almost never lands between them. It needs a stall and a concurrent configuration write — two stimuli a directed test does not naturally combine.

8. Trace Signals

Illustrative internal RTL interfaces from Chapters 12.112.3. Not PCIe wire signals.

A Memory Read, one chunk, with a TX stall and a client stall:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
step        1  2  3  4  5  6  7  8  9 10 11 12
loc_valid   1  1  0  0  0  0  0  0  0  0  0  0
loc_ready   1  0  0  0  0  0  0  0  0  0  0  0
req_valid   0  1  1  1  0  0  0  0  0  0  0  0
req_ready   0  0  0  1  0  0  0  0  0  0  0  0
ctx_state   -  R  R  O  O  O  O  O  O  -  -  -      R=reserved O=outstanding
cpl_valid   0  0  0  0  0  0  0  1  0  0  0  0
cpl_ctx     -  -  -  -  -  -  -  0  -  -  -  -
res_valid   0  0  0  0  0  0  0  0  1  1  1  0
res_ready   0  0  0  0  0  0  0  0  0  0  1  0

Read steps 2–4 carefully. req_valid is high for three steps and req_ready for one. ctx_state is R throughout, and the offered ctx_id must be identical at every one of them — that is Chapter 12.1's P13a, and the trace is what it looks like when satisfied.

And steps 9–11. res_valid is high for three steps with res_ready high for one. Exactly one delivery occurs, at step 11.

A posted Memory Write with a resource stall:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
step        1  2  3  4  5  6  7  8
wr_valid    1  0  0  0  0  0  0  0
wr_ready    1  0  0  0  0  0  0  0
tx_valid    0  1  1  1  0  0  0  0
tx_ready    0  0  0  1  0  0  0  0
res_valid   0  0  0  0  0  1  1  1
res_ready   0  0  0  0  0  0  0  1
tgt_update  0  0  0  0  0  0  0  1     <-- EXACTLY ONE, at step 8

tgt_update is the row that matters. res_valid is high for three steps. A design that updated on valid would show three pulsesChapter 12.2 §11, visible in one trace.

9. Verification Harness

VERIFICATION-ONLY. Not synthesizable, not a production model. It composes the blocks the previous three chapters built so their interaction can be exercised.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// VERIFICATION-ONLY. A minimal composition harness: request generator, a toy
// fabric that can delay and reorder, an endpoint resource model, a completion
// return path, and an independent scoreboard.
// This is NOT a PCIe endpoint and NOT synthesizable. It models only enough
// behaviour to exercise the interactions taught in Chapters 12.1-12.3.
module mem_txn_harness;
 
  localparam int CTXS   = 4;
  localparam int ADDR_W = 64;
 
  // ---- Independent data model -----------------------------------------
  // NOT the DUT's memory. A decorrelated function, so ANY address error
  // produces completely different data rather than data that differs in the
  // same bit the address did.
  function automatic logic [31:0] model_data (input logic [ADDR_W-1:0] a);
    logic [31:0] h;
    h = a[31:0] ^ 32'h9E37_79B9;
    h = (h << 13) ^ (h >> 7) ^ a[63:32];
    h = h * 32'h85EB_CA6B;
    return h ^ (h >> 16);
  endfunction
 
  // ---- Scoreboard state — owned entirely by the testbench --------------
  typedef struct {
    bit                  active;
    logic [ADDR_W-1:0]   addr;
    int                  expected_dw;
    int                  received_dw;
    int                  dest;
  } sb_read_t;
 
  sb_read_t sb_read [CTXS];       // NEVER read from the DUT context table
 
  typedef struct {
    logic [ADDR_W-1:0] addr;
    logic [31:0]       data;
    logic [3:0]        mask;
  } sb_write_t;
 
  sb_write_t sb_write [$];        // expected writes, in issue order
  int        applied_writes;
  int        observed_completions_for_writes;
 
  // ---- Toy fabric: delay and optional reorder --------------------------
  // Illustrative only. Real fabric behaviour is far richer; this exists so
  // the harness can produce out-of-order returns deterministically.
  typedef struct {
    int          delay;
    int          ctx;
    int          dw;
  } fab_item_t;
 
  fab_item_t fabric_q [$];
 
  task automatic fabric_push (input int ctx, input int dw, input int delay);
    fabric_q.push_back('{delay: delay, ctx: ctx, dw: dw});
  endtask
 
  // Each tick, decrement delays and release anything that has expired.
  // Because delays differ per item, items are RELEASED OUT OF ORDER — which
  // is the whole reason the harness exists.
  task automatic fabric_tick (output bit fire, output int ctx, output int dw);
    fire = 0;
    foreach (fabric_q[i]) if (fabric_q[i].delay > 0) fabric_q[i].delay--;
    foreach (fabric_q[i])
      if (fabric_q[i].delay == 0) begin
        fire = 1; ctx = fabric_q[i].ctx; dw = fabric_q[i].dw;
        fabric_q.delete(i);
        break;
      end
  endtask
 
  // ---- Scoreboard checks ----------------------------------------------
  // Called on every credited chunk. Uses ONLY testbench state.
  task automatic sb_credit (input int ctx, input int dw, input bit dut_retire);
    bit expect_retire;
    // Bounds first, for the reason Chapter 12.3 section 8a gives: an
    // out-of-range identifier is a different fault from an inactive one,
    // and indexing before checking hides it.
    if ((ctx < 0) || (ctx >= CTXS)) begin
      $error("chunk credited to out-of-range context %0d (CTXS=%0d)",
             ctx, CTXS);
      return;
    end
    if (!sb_read[ctx].active) begin
      $error("chunk credited to inactive context %0d", ctx);
      return;
    end
    sb_read[ctx].received_dw += dw;
    if (sb_read[ctx].received_dw > sb_read[ctx].expected_dw)
      $error("ctx %0d overrun: %0d > %0d",
             ctx, sb_read[ctx].received_dw, sb_read[ctx].expected_dw);
 
    expect_retire = (sb_read[ctx].received_dw == sb_read[ctx].expected_dw);
 
    // THE integration check: the DUT's retire decision must agree with an
    // independently-computed one. A DUT that retires early disagrees here on
    // the first partial chunk of a split return (Example 4).
    if (dut_retire != expect_retire)
      $error("ctx %0d retire mismatch: dut=%0b expected=%0b",
             ctx, dut_retire, expect_retire);
 
    if (expect_retire) sb_read[ctx].active = 0;
  endtask
 
  // Called on every observed resource write.
  task automatic sb_apply_write (input logic [ADDR_W-1:0] a,
                                 input logic [31:0] d,
                                 input logic [3:0] m);
    sb_write_t e;
    if (sb_write.size() == 0) begin
      $error("resource write with no expected write outstanding -- DUPLICATE");
      return;
    end
    e = sb_write.pop_front();
    // Compared as a PAIR. Checking address and data separately would pass
    // the split-ownership failure of Chapter 12.2 section 4.
    if ((e.addr != a) || (e.data != d) || (e.mask != m))
      $error("write mismatch: expected %h/%h/%h got %h/%h/%h",
             e.addr, e.data, e.mask, a, d, m);
    applied_writes++;
  endtask
 
  // End-of-test checks, including the NEGATIVE one.
  task automatic sb_final;
    foreach (sb_read[i])
      if (sb_read[i].active)
        $error("ctx %0d never completed: %0d of %0d DW",
               i, sb_read[i].received_dw, sb_read[i].expected_dw);
    if (sb_write.size() != 0)
      $error("%0d expected writes never reached the resource",
             sb_write.size());
    // A posted write producing no Completion is EXPECTED BEHAVIOUR, so it
    // needs a positive check. "We did not look" is not the same as
    // "correctly absent".
    if (observed_completions_for_writes != 0)
      $error("%0d Completions observed for posted writes",
             observed_completions_for_writes);
  endtask
 
endmodule

Classification: verification-only.

What it models: correlation, progress accounting, out-of-order return, write application, and an independent data model. What it does not model: any wire format, flow control, ordering rules, error handling, or timing.

The three things worth copying from it. model_data is decorrelated, so an address error is loud rather than subtle. sb_apply_write compares the pair, so split-ownership corruption cannot pass. And sb_final asserts the absence of write Completions positively — a check that is easy to omit and impossible to recover afterwards.

10. Fault Injection

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// VERIFICATION-ONLY. Deliberate fault switches, for demonstrating which
// invariant catches which fault. NEVER synthesize this; never leave it
// enabled in a regression that is checking for real bugs.
typedef enum {
  FAULT_NONE,
  FAULT_DROP_REQUEST,      // Request offered but never launched
  FAULT_CORRUPT_CTX_ID,    // correlation index flipped on the return path
  FAULT_DUPLICATE_WRITE,   // target updated on valid, not valid && ready
  FAULT_DROP_COMPLETION,   // a Completion chunk silently discarded
  FAULT_CORRUPT_ADDRESS,   // one address bit flipped in the Request
  FAULT_EARLY_RETIRE,      // context retired on any credited chunk
  FAULT_ORDER_ROUTING      // return data routed by arrival order, not by ID
} fault_e;
 
fault_e injected = FAULT_NONE;
 
// Applied at the harness boundary, never inside a DUT module.
function automatic logic [ADDR_W-1:0] maybe_corrupt_addr (
    input logic [ADDR_W-1:0] a);
  return (injected == FAULT_CORRUPT_ADDRESS) ? (a ^ (1 << 6)) : a;
endfunction
 
function automatic int maybe_corrupt_ctx (input int c);
  return (injected == FAULT_CORRUPT_CTX_ID) ? ((c + 1) % CTXS) : c;
endfunction

Which invariant kills which fault:

FaultCaught byHow it presents
FAULT_DROP_REQUESTsb_final — context never completesone read hangs; concurrency drops by one permanently
FAULT_CORRUPT_CTX_IDsb_creditretire mismatch, or credit to an inactive contextdata reaches the wrong client; with CTXS=1 it is invisible
FAULT_DUPLICATE_WRITEsb_apply_write — "write with no expected write outstanding"target state wrong; only under resource stall
FAULT_DROP_COMPLETIONsb_final — partial received_dwread hangs; looks identical to a lost Request until you check whether the Request left
FAULT_CORRUPT_ADDRESSdata comparison — model_data is decorrelated, so the mismatch is totalwrong data, right transaction structure
FAULT_EARLY_RETIREsb_credit — retire mismatch on the first partial chunkfires immediately on Example 4's trace
FAULT_ORDER_ROUTINGdestination comparison, only with two reads returning out of orderboth clients get plausible wrong data (Example 3)

11. Integration Assertions

Deliberately not a repeat of Chapters 12.1–12.3's properties. These are the ones that only make sense across block boundaries.

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// INTEGRATION SVA. Cross-block properties over the harness. LOCAL contracts
// of the composed teaching model — not claims about PCIe legality.
 
// I1: CONSERVATION ACROSS THE WRITE PATH. The target is never updated more
// times than writes were accepted. Catches duplication anywhere in the chain.
property p_no_write_amplification;
  @(posedge clk) disable iff (!rst_n)
  (applied_writes <= accepted_writes);
endproperty
a_no_amplification : assert property (p_no_write_amplification);
 
// I2: CLASS EXCLUSIVITY. No transaction is both posted and Completion-
// tracked in the teaching model. Only expressible across the two paths.
property p_posted_never_tracked;
  @(posedge clk) disable iff (!rst_n)
  posted_write_accepted |-> !ctx_reserve;
endproperty
a_class_exclusive : assert property (p_posted_never_tracked);
 
// I3: a resolved read corresponds to a read that was previously accepted.
// Rules out a retirement conjured from a Completion for a read never issued.
property p_resolution_has_provenance;
  @(posedge clk) disable iff (!rst_n)
  read_retire |-> sb_read[chunk_id].active;
endproperty
a_provenance : assert property (p_resolution_has_provenance);
 
// I4: the local context delivered with a result matches the one recorded at
// issue. The end-to-end form of Example 3's lesson.
property p_local_context_round_trip;
  @(posedge clk) disable iff (!rst_n)
  (out_valid && out_ready) |-> (out_dest == sb_read[out_ctx].dest);
endproperty
a_ctx_round_trip : assert property (p_local_context_round_trip);
 
// I5: DEBUG BOUNDARY DISTINGUISHABILITY. A packet that reached the Endpoint
// and found no resource is observably DIFFERENT from a packet that never
// reached it. Example 5 depends on this being true of the design, so it is
// worth asserting rather than assuming.
property p_arrival_distinguishable;
  @(posedge clk) disable iff (!rst_n)
  ep_rx_valid |-> (ep_bar_hit || unclaimed_write);
endproperty
a_distinguishable : assert property (p_arrival_distinguishable);
 
// I6: NO CROSS-TRANSACTION CONTAMINATION. Data credited to one context never
// emerges against another's destination.
property p_no_contamination;
  @(posedge clk) disable iff (!rst_n)
  (out_valid && out_ready) |-> (out_data == model_data(sb_read[out_ctx].addr));
endproperty
a_no_contamination : assert property (p_no_contamination);

Liveness, with every assumption stated:

Azvya Education Pvt. Ltd.VLSI Mentor
Snippet
// The fabric, the Completer and the local consumer are all environment.
// PCIe guarantees none of these, which is why Completion timeout exists.
assume property (@(posedge clk) disable iff (!rst_n)
  req_valid |-> s_eventually req_ready);          // TX eventually accepts
assume property (@(posedge clk) disable iff (!rst_n)
  fabric_has_item |-> s_eventually fabric_fires); // fabric eventually forwards
assume property (@(posedge clk) disable iff (!rst_n)
  res_valid |-> s_eventually res_ready);          // target eventually ready
assume property (@(posedge clk) disable iff (!rst_n)
  out_valid |-> s_eventually out_ready);          // consumer eventually ready
 
// L1: under those four, an accepted write reaches the target.
property p_write_reaches_target;
  @(posedge clk) disable iff (!rst_n)
  posted_write_accepted |-> s_eventually (res_valid && res_ready);
endproperty
 
// L2: under those four, an accepted read delivers a result.
property p_read_delivers;
  @(posedge clk) disable iff (!rst_n)
  (loc_valid && loc_ready) |-> s_eventually (out_valid && out_ready && out_final);
endproperty

I1 is a counting property because a value check cannot see duplication. Ten identical writes to an idempotent register leave the same final state as one. Only the count distinguishes them, which is why the scoreboard tracks it.

I5 is unusual and worth keeping. It asserts that a debugging distinction is real — that Example 5's two versions are actually separable by observation. A design where an unmatched packet is silently swallowed at the receive boundary would fail it, and the consequence is that §13's method breaks: there would be no observation distinguishing "never arrived" from "arrived and was discarded."

I6 is the end-to-end contamination check, and it works only because model_data is decorrelated: data from a neighbouring address produces a completely different value rather than a nearly-identical one.

12. Verification Exercises

Do these against the modules in Chapters 12.112.3. Each has a specific answer.

Task A — reuse an active correlation index

Mutate the context table so the free-entry search can return an index that is still occupied.

Which check fails first, and under what stimulus?Chapter 12.1's P5 fails immediately on the reservation, before any Completion returns. If P5 is not bound, the first observable failure is a retire mismatch in sb_credit, and only with two reads outstanding — with one, the reuse is harmless because there is nothing to collide with.

Task B — reject full+pop replacement

Change the result buffer's acceptance to push = res_valid && !full.

What symptom appears? — No functional failure at low rates. Under sustained full-rate return traffic, overflow_error sets and results are lost, and the reads they belonged to never complete. The signature is that it only happens when the buffer is full — that is, when the design is busiest. Chapter 12.1's P12a catches it directly.

Task C — update the target on valid

Drive the resource's write side effect from res_valid instead of res_valid && res_ready.

Which assertion catches the duplication?Chapter 12.2's P8 fires on the first stalled cycle. I1 catches it in aggregate. sb_apply_write reports "write with no expected write outstanding" on the second application. And with a never-stalling resource, none of them fires — the stimulus must include a resource stall.

Task D — swap two Completion context IDs

Apply FAULT_CORRUPT_CTX_ID so chunks for ctx n are credited to ctx n+1.

What does the monitor see versus what does the scoreboard report?The monitor sees two perfectly well-formed Completions, each with a valid correlation index, each credited to an active context. Nothing at the packet level is wrong. The scoreboard reports a retire mismatch — because its independently-tracked received_dw for each context no longer matches the DUT's retire decision. The gap between those two views is the entire argument for scoreboard independence: a scoreboard mirroring the DUT's accounting would agree with it and report nothing.

Task E — corrupt routing but not BAR decode

Make the fabric deliver a packet to the wrong Endpoint while leaving every BAR correct.

Which debug boundary localizes it?The receive-interface observation of Example 5. The packet arrives somewhere, so it is not a routing-to-nowhere failure; it arrives at an Endpoint whose BAR does not match, so unclaimed_write asserts at the wrong component. The localizing question is not "did it arrive" but "did it arrive here" — and the answer identifies the fabric immediately, without touching any BAR.

Task F — split ownership of a posted write

Give the posted buffer separate metadata and payload handshakes with no association mechanism.

What must the scoreboard do to catch it?Compare the {address, payload} pair, keyed by shadow ID. Two independent checks both pass: every address requested was written, and every payload supplied was delivered. Only the pairing is wrong (Chapter 12.2 §16). And the stimulus must make the payload path slower than the metadata path with two writes close enough to overlap.

13. The Debugging Method

One method, used in every example above.

Find the last boundary where identity, address, payload and ownership are all correct. Then inspect the next boundary.

Three corollaries, each of which saves an afternoon:

  • Do not debug the physical layer if the packet never left the Requester. If req_valid && req_ready never happened, nothing below the Transaction Layer has seen this transaction.
  • Do not debug BAR decode if the Switch sent the packet to the wrong port. BAR decode runs after arrival; a packet that arrived elsewhere never reached it (Example 5).
  • Do not debug Completion fields if the target never produced data. Chapter 12.3 §3's boundary 1 precedes everything about the return packet.

The boundary list, in order, for the two transaction classes:

#Memory ReadMemory Write
1local read acceptedlocal write accepted
2context reservedbuffered with payload
3Request launchedRequest launched
4routed to the Endpointrouted to the Endpoint
5BAR hit, offset computedBAR hit, offset computed
6resource read, data availableresource handshake
7Completion descriptor builttarget updated — once
8Completion transmitted(nothing returns)
9correlated at the Requester
10progress accounted; retire decided
11result delivered to the client

Walk it in order. The first row that did not happen is where to look, and every row above it is proven correct by the fact that the next one was reached.

14. Common Misconceptions

  • "A timing diagram implies fixed PCIe latency." The traces are event sequences, not cycle counts. Fabric, Completer and consumer delay all vary (§1).
  • "A second Completion arriving first is a protocol error." Independent transactions may be served by different resources across different paths. Nothing promises return order (Example 3).
  • "A posted write has no state." It has forward-path state at every boundary — just no correlation state (Chapter 12.2 §1).
  • "A Memory Read Request carries return payload." It carries no payload at all (Chapter 12.1 §3).
  • "A read is done when the target's data becomes available." That is boundary 6 of eleven. Five boundaries and a queue separate it from delivery (§13).
  • "A write is done when the local producer drops valid." That means the producer's handshake completed. The target may not have been touched (Example 1).
  • "A packet reaching the Endpoint means the BAR hit." Arrival and decode are separate, and unclaimed_write is the observable difference (Example 5).
  • "A BAR hit means the resource consumed the request." The front end still has to hand it over, and the resource can stall (Example 1, step 7).
  • "A Completion arriving means the local consumer accepted the data." Retirement and delivery are different events (Chapter 12.1 §12).
  • "Queue occupancy and cumulative transaction count are the same metric." One is live state and one is a running total (Chapter 10.5). Occupancy is not expected to be monotonic.
  • "Correctness can be proven with happy-path traces." Every bug in this chapter needs a stall, a reorder, or a concurrent configuration write to appear.
  • "The testbench may use the DUT's context table as the expected model." Then it agrees with the DUT by construction and verifies nothing (Task D).

15. Understanding Check

16. What's Next

Module 12 has covered memory traffic end to end: 12.1's split read, 12.2's posted write, 12.3's return pipeline, and this chapter's integration of all three.

Chapter 12.5 closes the module with the performance implications these chapters kept setting aside: read latency and outstanding depth, posted-write throughput, packet-size overhead, Completion return bandwidth, and how latency and bandwidth interact on a real path.

Module 13 then takes the Completion packet itself — the Cpl and CplD forms, Completion Status, Byte Count and Lower Address, the rules governing how a response may be split, and the ordering matrix that Chapter 11.6's attributes are permitted to relax.

The idea to carry forward: every transaction is a chain of ownership transfers, and every bug in this chapter is one boundary asserting an ownership it did not have.