PCIe Packet Transport

Chapter 10.1 drew the boundary: PCIe's transaction layer tunnels unchanged, the UCIe Adapter takes over the data-link role, and an accepted object becomes the transport's responsibility. That was ownership.

This chapter is mechanism. A PCIe transaction enters one die as a packet object with meaning; it leaves the other die as the same object with the same meaning; and in between it is buffered, framed, protected, possibly retried, and reconstructed. Every one of those steps has state, and the state has lifetimes that do not coincide.

The single most important thing this chapter teaches is what happens when those lifetimes are confused — because the resulting bug is a PCIe transaction delivered twice, and a duplicated memory write is not a performance problem.

1. The One-Sentence Model

The transport may retry as often as it likes; the PCIe transaction must be delivered exactly once, in an order PCIe permits, with its meaning intact.

Transport retry and semantic delivery are different events at different layers with different lifetimes. Chapter 9.4 built the machinery that makes retry safe for the transport. This chapter is about making sure that machinery stays invisible above the boundary — because the moment a retransmitted flit becomes a second PCIe transaction, the system is corrupt in a way no CRC will ever detect.

2. First, Correct the Premise

3. What the Mapping Problem Actually Is

Given §2, restate the problem correctly.

It is not: "how do I chop a TLP into pieces and reassemble it?"

It is: how does a PCIe packet object move from one protocol engine to another across a transport that

• has its own transfer quantum, aligned with but not identical to the protocol's;
• provides reliability by retrying that quantum, invisibly;
• must preserve exactly-once delivery and PCIe's ordering rules at the semantic layer;
• and must not require the PCIe engines at either end to know any of it happened.

Four requirements, and the tension between the second and third is where this chapter lives.

4. The Object and Its Lifecycle

// Illustrative PCIe-over-UCIe RTL — not PCIe or UCIe normative encoding.
// Symbolic widths throughout: no flit layout is asserted anywhere.
localparam int PCIE_META_W = 128;      // symbolic: whatever the mapping needs
localparam int PCIE_DATA_W = 512;      // symbolic
localparam int PCIE_BYTES  = PCIE_DATA_W / 8;
 
typedef struct packed {
  logic [PCIE_META_W-1:0] meta;        // PCIe-side descriptor, opaque below
  logic [PCIE_DATA_W-1:0] data;
  logic [PCIE_BYTES-1:0]  byte_valid;
  logic [MON_ID_W-1:0]    mon_id;      // VERIFICATION ONLY — see §10
} pcie_obj_t;

Architecture. A PCIe packet is a descriptor plus payload plus extent. Those three are consumed together and must therefore travel together — Chapter 9.2 §12's rule, and §9 shows why the consequence is worse here.

State. None yet; this is the representation.

Contract. Below the boundary, meta is carried, not interpreted (Chapter 10.1 §7).

Note mon_id explicitly. It is a verification-only tag, present in simulation to let a scoreboard follow one object end to end. It is not a PCIe Tag and not a transport sequence number — §10 is entirely about why conflating those is a serious bug.

The lifecycle, with each stage owning distinct state:

Six stages, six lifetimes, and no two of them end at the same moment. The bugs in this chapter are all cases where two stages' state was assumed to share a lifetime.

5. What Happened to the DLLPs

The §2 correction, developed — because a reader arriving from PCIe will look for them.

PCIe's DLLPs carry data-link functions: acknowledgement and negative acknowledgement for replay, link-level flow-control credit updates, and some power-management signalling. Over UCIe, those functions have a different provider:

The DLLP-carried functions are largely replaced, not tunnelled. Asking "which flit does a flow-control DLLP go in?" is asking about a mechanism that has been substituted.

Three consequences worth stating.

There is one reliability mechanism, not two stacked. Chapter 9.4 §19 warned that layering your own retry over the Adapter's produces two mechanisms with different timeouts interacting. Here that warning is architectural rather than advisory.

Credits may still exist at two layers. Chapter 9.5 §14: UCIe link credits protect transport storage, and the carried protocol may retain its own credits protecting its own queues — PCIe's posted-header and posted-data credits being the named example. Link-level flow control being provided by the Adapter does not automatically mean PCIe's transaction-layer credits vanish.

The exact division is a specification question. "Largely" is load-bearing. Which PCIe link-layer behaviours survive, are replaced, or are handled differently is exactly the detail to read from your revision — and it is why this chapter teaches the shape rather than a mapping table.

6. Classification Before Mapping

// Illustrative — the mapping engine's own view of what it is handling.
// NOT a UCIe or PCIe normative encoding.
typedef enum logic [1:0] {
  OBJ_TLP    = 2'd0,   // a transaction-layer object — tunnels
  OBJ_LINK   = 2'd1,   // link-layer management, handled per §5
  OBJ_OTHER  = 2'd2
} obj_class_t;

Architecture. Different classes have different lifetimes and different destinations, so the mapping engine must know which it is handling before deciding anything else.

State. A class field alongside each object — per-object lifetime.

Cycle behaviour. Assigned at acceptance and never changed afterwards.

Contract. Downstream logic keys buffering and delivery on the class.

Failure. Misclassification delivers a link-management object to the PCIe transaction layer or vice versa. The first produces a PCIe engine seeing something meaningless; the second silently drops something the link needed.

Why this is deliberately coarse. The enum has three values because the mapping engine's own decisions need three. Inventing a fine-grained PCIe packet-type encoding here would be exactly the fabrication §2 forbids — the real classification comes from the interface definition of the revision you implement.

7. Metadata and Payload Must Not Be Pipelined Apart

Chapter 9.2 §12 established this rule for streaming. Here the consequence is qualitatively worse, and it deserves its own treatment.

// WRONG — two paths, different depths.
always_ff @(posedge clk) meta_q  <= obj_in.meta;                 // 1 stage
always_ff @(posedge clk) data_q  <= obj_in.data;
always_ff @(posedge clk) data_q2 <= data_q;                      // 2 stages
// Output pairs meta_q with data_q2 — descriptor N with payload N-1.

In streaming mode, a misaligned header produces a corrupt message that the receiving protocol will probably reject. Here, the descriptor carries PCIe transaction meaning — what kind of request this is, and where it is going. Pair descriptor N with payload N−1 and you have constructed a well-formed PCIe transaction carrying the wrong data.

Think about what that means for a memory write: a correctly-formed write, to a legitimate address, with the previous transaction's payload. It will not be rejected. It will be executed. Nothing in the PCIe stack has any way to detect it, because every field is individually valid.

A misaligned descriptor does not corrupt a packet — it manufactures a valid transaction with wrong contents. This is the most severe failure mode in Module 10, and it is a pipeline-depth bug.

// Illustrative — one object, one pipeline, misalignment impossible by construction.
pcie_obj_t obj_q, obj_q2;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    obj_q  <= '0;
    obj_q2 <= '0;
  end else if (pipe_en) begin
    obj_q  <= obj_in;
    obj_q2 <= obj_q;
  end
end

Architecture. Fields consumed together travel together, and one enable means there is no way to advance one and not another.

Cycle behaviour. A stall stalls the whole object.

Failure. The separated version — which, note, survives review because each always_ff is individually correct, and survives simulation whenever consecutive objects have similar descriptors.

// Illustrative — the whole object holds still under backpressure.
property p_object_stable_under_stall;
  @(posedge clk) disable iff (!rst_n)
    (obj_valid && !obj_ready) |=> ($stable(obj_q) && $stable(obj_valid));
endproperty

8. Backpressure Is a Chain, Not a Signal

// WRONG — PCIe acceptance derived from the physical layer.
assign pcie_pkt_ready = ucie_phy_ready;

Between the PCIe engine and the PHY sit the tunnel queue, the mapping engine, the Adapter's replay store, and the credit pool. The PHY being ready says nothing about any of them, and accepting an object with no room in an intermediate stage means it is overwritten or lost.

// Illustrative — every stage that must hold the object gets a veto.
assign pcie_pkt_ready =
    tunnel_ready              // Ch 10.1 — boundary queue has space
 && map_buffer_space          // this chapter — mapping state available
 && replay_space              // Ch 9.4 — retention possible
 && (tx_credit_q != '0);      // Ch 9.5 — receiver has a slot

Architecture. Accepting a PCIe object commits every stage between here and the far side. Each stage independently can be the constraint.

State. None of its own; a conjunction over four registered facts owned by four mechanisms.

Contract. The PCIe engine relies on acceptance meaning safe handling all the way through.

Failure. Each omission fails on a different die with a different symptom, and knowing which is which is most of the debug value: without map_buffer_space the object is overwritten locally; without replay_space it becomes unrecoverable; without credit the remote buffer overflows.

Module 9 and Module 10 have now assembled a four-way conjunction. Each mechanism holds a veto, and none of them is the others' proxy.

9. The Cross-Layer Rule: Retry Is Not a New Transaction

The heart of the chapter.

Chapter 9.4 gave the transport a replay mechanism: a corrupted flit is retransmitted from retained state. That mechanism operates entirely below the semantic boundary — and it must stay there.

// WRONG — semantic delivery driven by transport arrival.
assign pcie_deliver = flit_received && flit_crc_ok;

If a flit is retried because a confirmation was lost — Chapter 9.4 §12's case, where the original arrived intact — then the receiver sees the same transport object twice. With delivery keyed directly on arrival, the remote PCIe engine sees the same TLP twice.

Consider what that means concretely:

• A memory write delivered twice writes twice. If the write is not idempotent — an increment, a FIFO push, a doorbell — the system state is now wrong, permanently, with no error anywhere.
• A read request delivered twice generates two completions for one request tag, and the requester receives a completion it never asked for.
• A completion delivered twice may retire a tag that has already been reused.

None of these produce a CRC error. None produce a PCIe error. The transport did exactly what it was designed to do.

// Illustrative — semantic delivery is gated on transport RESOLUTION, not arrival.
// Requires an identity the receiver can use to recognise a repeat (Ch 9.4 §12).
assign pcie_deliver = obj_reconstructed          // §11 — complete object
                   && !obj_is_duplicate          // this transport object is new
                   && delivery_order_ok;         // §12 — PCIe ordering satisfied

Architecture. The transport's job is to make an object arrive; the mapping layer's job is to make it arrive to the protocol once. Those are different guarantees and need different logic.

State. Duplicate-detection state — Chapter 9.4 §12's last-delivered identity — plus reconstruction state.

Cycle behaviour. Delivery fires at most once per semantic object, regardless of how many times its transport representation crossed the link.

Contract. The PCIe engine may assume every delivery is a distinct transaction. That assumption is the entire reason PCIe semantics survive the crossing.

Failure. As above — and note that the retry itself is correct. The bug is in what was allowed to observe it.

// Illustrative — a replayed transport object delivers at most one PCIe object.
// Uses the verification-only monitor ID (§4).
property p_replay_does_not_duplicate_delivery;
  @(posedge clk) disable iff (!rst_n)
    (pcie_deliver && (delivered_mon_id == prev_delivered_mon_id_q)) |-> 1'b0;
endproperty

Transport retirement and PCIe semantic retirement are different events. The transport retires an object when delivery is confirmed. PCIe retires a transaction according to PCIe's own rules — a completion returning, a tag being freed. Neither is the other's trigger.

10. PCIe Tag Is Not Transport Identity

A conflation worth its own section, because the two look similar and are unrelated.

Why anyone conflates them. Both are small integers identifying an in-flight thing, both wrap, and both have a "still outstanding" window. It is a natural mistake.

What goes wrong. Using the PCIe Tag as the replay identity means the replay window becomes bounded by the PCIe tag space, and — worse — two transactions with the same tag at different times become indistinguishable to the transport, so a replay can resurrect the wrong one. Using the transport identity to match completions is equally broken: it changes with retries, and it means nothing to the far PCIe engine.

The related trap: PCIe 6.0's flit mode introduced a new header format with, among other things, 14-bit tag support. A design that assumed a narrower tag field breaks on a generation change — which is Chapter 10.1 §7's argument for why the Adapter should not be reading PCIe fields at all.

Two identity spaces, two owners, two lifetimes. They may both be present on the same object, and they must never be the same field.

11. Reconstruction Is All-or-Nothing

// Illustrative — a PCIe object is delivered only when it is complete.
logic                      obj_open_q;        // reconstruction in progress
logic [OBJ_LEN_W-1:0]      obj_bytes_q;       // accumulated so far
logic [OBJ_LEN_W-1:0]      obj_expected_q;    // declared extent
logic                      obj_reconstructed;
 
assign obj_reconstructed = obj_open_q && (obj_bytes_q == obj_expected_q);

Architecture. A partial PCIe object is not a smaller PCIe object — it is not a PCIe object at all. Delivering one hands the protocol engine something malformed at best and misinterpretable at worst.

State. An open flag, an accumulator, and the declared extent — per-object lifetime, cleared on completion or on invalidation.

Cycle behaviour. Accumulates as transport units arrive; obj_reconstructed asserts only on exact match.

Contract. The PCIe engine receives whole objects or nothing.

Failure. Two directions, both bad. Delivering early hands over a truncated transaction. Using >= instead of == accepts an over-length object, which means a length mismatch went undetected and the payload boundary is wrong.

// Illustrative — never deliver a partial object.
property p_no_delivery_before_complete;
  @(posedge clk) disable iff (!rst_n)
    pcie_deliver |-> (obj_bytes_q == obj_expected_q);
endproperty
 
// Illustrative — malformed extents are detected, not delivered.
property p_no_overlength_object;
  @(posedge clk) disable iff (!rst_n)
    obj_open_q |-> (obj_bytes_q <= obj_expected_q);
endproperty

On malformed detection generally: a mapping engine can detect impossible states — a continuation with no object open, an extent exceeding the declared length, a class change mid-object. It should report them abstractly and refuse delivery. Do not invent UCIe error codes for these; the reporting mechanism is defined by the interface you implement.

12. Ordering Is PCIe's, Not Streaming's

A warning against reusing Chapter 9.3 mechanically.

Chapter 9.3 taught ordering domains and per-stream FIFOs — for a protocol UCIe does not interpret, where you defined the domain. PCIe's ordering model is not that. It has producer-consumer rules, relationships between posted and non-posted traffic and completions, and relaxed-ordering attributes that permit specific reorderings. It is a real specification with real subtlety, and it is the PCIe track's subject, not this chapter's.

What this chapter must say is narrower and more useful:

The transport must not introduce reorderings the PCIe engine does not expect. Whatever ordering the mapping and transport preserve must be at least as strong as what PCIe requires between any two objects.

Three practical consequences:

A scheduler must not reorder objects whose PCIe relationship forbids it. If the mapping engine has any freedom in what it transmits next, that freedom is bounded by PCIe's rules — which the mapping engine may not itself know, so the safe default is to preserve the order objects were accepted in.

Retry must not reorder — Chapter 9.3 §10 and 9.4 §9. Go-back-N preserves order structurally; selective retry requires reordering support at the receiver.

Verification must model PCIe's ordering, not a per-stream approximation. Chapter 9.3 §13's rule — the number of expectation queues equals the number of ordering domains — applies, but the domains are PCIe's, and getting them wrong produces either false failures or silent escapes.

13. A Worked Trace: Two Objects, One Retry

The flagship example, connecting every mechanism in Modules 9 and 10. Illustrative latency throughout.

Two PCIe objects, A then B, whose relative order PCIe requires to be preserved.

Six things to read off it, and together they are the module in one table.

Cycle 6: B arrived first and was not delivered. Ordering required A first, so B waits. This is head-of-line blocking (Ch 9.3 §7) at the semantic layer.

Cycle 9: replay does not allocate. The replay store still holds exactly A and B; the send pointer rewound (Ch 9.4 §8). Occupancy is unchanged.

Cycle 10: B is re-sent although it was fine. Go-back-N's cost (Ch 9.4 §9), paid to preserve order structurally.

Cycle 13: B arrives a second time and is not delivered again. This is §9's rule doing its work — the transport delivered B twice, the semantic layer delivered it once.

Cycle 14: B is delivered from the state held since cycle 6, not from the cycle-13 copy. Either source works if they are identical; what matters is exactly one delivery.

Cycle 16: retirement is last. A was retained from cycle 3 to cycle 16 — through a corruption, a retry, and its own successful delivery. Retention is bounded by confirmation, never by transmission.

14. State Lifetimes, Side by Side

The table this chapter exists to produce.

Three rows carry most of the weight. Replay entries do not re-allocate on retry — Chapter 9.4 §8's invariant. Credits are not consumed twice for one object unless the contract says the slot was genuinely released and re-taken. And the PCIe transaction's lifetime is governed by PCIe, entirely independently of everything to its left — which is the formal statement of §9.

15. Verification: Two Models, Because Two Contracts

Chapter 10.1 §12 introduced the split; here is what each model actually holds.

The transport model tracks what Chapter 9.4 defined: allocated, transmitted, unconfirmed, retired; replay occupancy; retry counts. Its invariant is that the transport delivered every object at least once and eventually retired it.

The PCIe semantic model tracks what the protocol engines see: every object offered at the source, in order, and every object delivered at the sink. Its invariant is exactly once, in an order PCIe permits, unmodified.

source_objects[]        — every object the PCIe engine offered, in order
transport_model         — allocation / transmission / retry / retirement
delivered_objects[]     — every object the remote PCIe engine received

The check that only this pairing can make: delivered_objects equals source_objects regardless of how many times the transport model recorded a retransmission. A transport-only model cannot see a duplicate delivery. A semantic-only model cannot explain one.

Why both are needed, in one sentence each. The transport model tells you why — the diverging object was in the unconfirmed window when the error was injected. The semantic model tells you that — the sink saw B twice.

Error injection, and what each targets:

The second and third are the ones that matter most and the ones regressions omit. A lost confirmation is the only way to produce a genuine duplicate, and if it is never injected, §9's entire mechanism ships unexercised.

// Illustrative PCIe-transport coverage — not UCIe- or PCIe-defined.
covergroup cg_pcie_transport @(posedge clk iff obj_event);
 
  cp_class     : coverpoint obj_class;
  cp_outstand  : coverpoint objects_outstanding {
    bins one = {1}; bins few = {[2:3]}; bins many = {[4:$]};
  }
  cp_retry     : coverpoint obj_retry_count {
    bins none = {0}; bins one = {1}; bins several = {[2:$]};
  }
  cp_dup       : coverpoint obj_was_duplicate;      // a repeat arrived
  cp_held      : coverpoint obj_held_for_order;     // waited for a predecessor
  cp_recon     : coverpoint recon_partial;          // reconstruction in progress
 
  // Was a duplicate ever received while another object was held for ordering?
  x_dup_by_held    : cross cp_dup, cp_held;
  // Was a retry ever taken with several objects outstanding?
  x_retry_by_out   : cross cp_retry, cp_outstand;
 
endgroup

Why x_dup_by_held is the valuable cross. A duplicate with nothing else in flight exercises duplicate suppression alone. A duplicate arriving while another object is held for ordering exercises suppression and ordering and reconstruction simultaneously — the configuration of §13's trace, and where these three independently-correct mechanisms interact.

16. Failure Taxonomy

Two rows deserve emphasis. "Valid TLP, wrong payload" is the most dangerous entry in the table because every field is individually valid and the transaction executes — no CRC, no PCIe error, no assertion unless you wrote §7's. And "duplicate PCIe transaction" is the one whose blast radius is unbounded, because a repeated non-idempotent write corrupts system state permanently.

17. Debug Checklist

1. Was the PCIe object accepted at the boundary? If not, §8 — which term of the veto chain was low.
2. What class was assigned? §6 — misclassification sends an object to the wrong consumer.
3. Did metadata and payload stay together? §7 — check whether failures correlate with differing consecutive descriptors.
4. Which mapping-buffer entry held it? Follow the monitor ID.
5. Was a replay entry allocated exactly once? Ch 9.4 §8 — occupancy across the retry.
6. Did the transport retry, and how many times?
7. Did the object arrive more than once? If so, was the second arrival suppressed (§9)?
8. Was a confirmation lost? That is the mechanism that generates genuine duplicates.
9. Was reconstruction complete before delivery? §11 — compare accumulated against declared extent.
10. Was the object held for ordering, and released in the right order? §12.
11. Was it delivered exactly once? Compare source and sink counts, per direction.
12. Was it retired only after confirmation? Not after transmission.
13. Did a reset or recovery occur with objects in flight? §14 — check each row's policy was honoured.
14. Which model diverged first — transport or semantic? §15 — this single question routes the whole investigation.

Step 14 is the highest-yield one. If the transport model is clean and the semantic model diverged, the bug is in the mapping or delivery layer. If the transport model diverged, everything above it is a consequence.

18. Common Misconceptions

"TLPs are chopped into flits by a generic packetizer." UCIe maps PCIe natively with flit sizes aligned to PCIe's own flit definitions. Importing Chapter 9.2's shim model teaches a mapping that does not exist (§2).

"DLLPs are tunnelled like TLPs." The functions DLLPs carry — replay coordination, link flow control, link management — are largely provided by the Adapter's own mechanisms instead (§5).

"PCIe Tag and transport identity are the same thing." Different layers, purposes, lifetimes, scopes, and owners. Conflating them bounds the replay window by the tag space and lets a replay resurrect the wrong transaction (§10).

"A UCIe replay should appear as another PCIe TLP." Then a non-idempotent write executes twice, with no error anywhere. Retry must remain invisible above the semantic boundary (§9).

"PCIe input ready can follow PHY ready." Four independent stages must each be able to hold the object, and each failure lands on a different die (§8).

"If CRC passes, the mapping is correct." CRC proves the bytes arrived as sent. Misalignment, misclassification, and premature delivery all pass CRC (§7, §11, §16).

"Metadata and payload may be pipelined independently." Here that produces a well-formed transaction carrying the wrong data — which executes (§7).

"One scoreboard is enough." A transport model cannot see a duplicate delivery; a semantic model cannot explain one (§15).

"Transport retirement and PCIe retirement are the same event." The transport retires on confirmation; PCIe retires by its own rules. Neither triggers the other (§9, §14).

"All PCIe generations map identically." PCIe 6.0's flit mode changed the transfer model and the header format, including a wider tag. Revision nuance is real (§2, §10).

19. Understanding Check

20. Summary and What Comes Next

The transport may retry as often as it likes; the PCIe transaction must be delivered exactly once, in an order PCIe permits, with its meaning intact.

The premise had to be corrected first. UCIe maps PCIe natively, with flit sizes aligned to PCIe's own flit definitions — PCIe 6.0 being flit-based itself — so this is not a generic packet-shredding problem and Chapter 9.2's shim is the wrong model. And the DLLP-carried functions are largely replaced rather than tunnelled, because replay coordination, link flow control, and link management are what the Adapter already provides.

The mechanisms: six lifecycle stages with six lifetimes, none ending together. Metadata and payload in one pipeline, because here misalignment manufactures a well-formed transaction carrying the wrong data — which executes, with nothing in the PCIe stack able to detect it. Four vetoes on acceptance, each owned by a different mechanism and each failing on a different die. Reconstruction all-or-nothing, with an exact extent match rather than a >=.

The cross-layer rule that is the chapter's reason for existing: semantic delivery is gated on resolution, not arrival. A lost confirmation causes the transport to deliver an object twice; the mapping layer must deliver it once. And PCIe Tag is not transport identity — two spaces, two owners, two lifetimes, never the same field.

For verification, two models: the semantic one says that something was duplicated, lost, or reordered; the transport one says why. And the injection that most regressions omit — a lost confirmation on an object that arrived intact — is the only one that produces a genuine duplicate.

The link now carries PCIe transactions correctly. What has not been addressed is the system question above all of this: what it means to bring up an actual PCIe endpoint whose connectivity is provided by a die-to-die link rather than a slot:

• 10.3 — Endpoint Connectivity — bringing up a PCIe endpoint over a UCIe link.

Browse the full path on the UCIe tutorials index.