PCIe Tunneling Over UCIe

Module 9 built a transport for protocols UCIe does not understand. Its defining property was payload opacity: the Adapter protected bytes without interpreting them, which made one mechanism serve every protocol and left framing, ordering semantics, and message identity as the shim's problem.

Module 10 is the other case, and the difference is larger than it first appears. PCIe is not a protocol UCIe carries blindly — UCIe maps PCIe natively, and the Adapter knows what it is carrying. That changes which layer owns what, which errors mean what, and what a verification engineer has to check.

This chapter establishes those boundaries. Chapter 10.2 does the mechanics.

1. The One-Sentence Model

UCIe replaces the transport path beneath PCIe, not PCIe's transaction semantics. A memory read is still a memory read, with the same requester, the same completion obligation, and the same ordering rules — it simply crosses a package instead of a board.

The value of that sentence is what it protects. PCIe's semantics are the thing an enormous software and IP ecosystem already depends on; if crossing a die boundary changed them, none of that ecosystem would transfer, and the entire point of mapping PCIe would be lost.

2. Prerequisites, and What This Chapter Will Not Re-teach

This chapter assumes PCIe fundamentals. It does not redefine TLPs, requesters and completers, or topology — the PCIe track covers those, and duplicating them here would bury the actual subject.

Useful background, if you need it: PCIe Architecture · Root Complex · Endpoint · Switch

The unique topic here is PCIe semantics over UCIe transport — specifically, where the boundary sits and what crosses it unchanged.

3. Tunnelling Is Not Translation

A distinction worth fixing before anything else, because the words are used loosely and the engineering consequences are opposite.

PCIe over UCIe is the left column. The PCIe engine on each die still speaks PCIe; UCIe carries it between them. Nothing is being converted into a different transaction model, which is why an existing PCIe protocol engine and its verification IP remain useful.

Calling this "translation" invites the wrong questions — how are completions converted, how are ordering rules remapped — when the correct answer is that they are not converted at all, and any design that converts them has introduced a semantic gap where none needed to exist.

4. Native Mapping Is Not Opaque Carriage Either

Having separated tunnelling from translation, separate it from Module 9's model too — because "tunnelling" in the streaming sense and what UCIe does with PCIe are genuinely different.

UCIe maps PCIe and CXL natively, precisely because those ecosystems are widely deployed. The UCIe protocol stack is described as supporting PCIe 6.0 (256B flit), CXL 3.0 (256B flit), and CXL 2.0 (68B flit) — the flit sizes align with the protocols' own definitions rather than being a container the protocol is stuffed into.

The row that matters most is the last. Two streaming chiplets interoperate at the transport level and understand each other only if their private framing agrees. Two PCIe-over-UCIe chiplets interoperate because both speak PCIe, which is the entire commercial argument for mapping it natively.

5. Which Layer Does UCIe Actually Replace?

The question that determines everything else, and the one where the registry's shorthand needs care.

A conventional PCIe stack has a transaction layer producing TLPs, a data link layer providing sequence numbers, ACK/NAK, replay and link-level flow control via DLLPs, and a physical layer. Now look at what UCIe's D2D Adapter does: CRC embedding and checking, retry of flits, link state management, parameter negotiation, and flit-based flow control.

Those are data-link-layer responsibilities. Secondary technical material states the relationship directly — the Adapter layer replaces PCIe data-link-layer functions with a flit-based mechanism that works across multiple protocols, in place of protocol-specific mechanisms such as DLLP acknowledgements.

So the shape is:

The transaction layer is what tunnels. The data link layer is largely what gets replaced.

That single sentence resolves most of the confusion in this subject, and it has a direct consequence for Chapter 10.2: the registry summarises it as "How TLPs / DLLPs map to UCIe FLITs", and the DLLP half of that phrase needs qualifying rather than repeating. If UCIe's Adapter is providing the reliability and link-flow-control functions that DLLPs exist to carry, then those DLLPs are not simply tunnelled — their function is served by a different mechanism.

Two honest caveats. First, this is the architectural shape from vendor-technical description, not a clause I have read; the exact division is a specification question for your revision. Second, "largely" is doing real work in that sentence — which PCIe link-layer behaviours survive, are replaced, or are handled differently is exactly the detail to verify before implementing.

6. What Each Side Owns

The two right-hand columns of that table are what Chapter 10.2 implements. The two left-hand columns are what must survive the crossing untouched.

7. The Adapter Must Not Read PCIe Semantics

Native mapping means the Adapter knows it is carrying PCIe. It does not mean the Adapter should make decisions based on PCIe transaction semantics.

// WRONG — transport behaviour keyed on PCIe transaction state.
if (tlp_tag == blocked_tag_q)
  adapter_stall <= 1'b1;                    // the Adapter now owns PCIe semantics

Four things break at once. The Adapter is now protocol-specific, so it cannot serve CXL or streaming without a second implementation. It is coupled to a PCIe revision — tag width changed with PCIe 6.0's flit-mode header, so this logic breaks on a generation change. It has taken a decision that belongs to the transaction layer, which is the only place that knows what a tag means. And it can no longer be verified independently, because its correctness now depends on PCIe semantics rather than on transport behaviour.

// Illustrative — the Adapter sees a transport object plus defined interface
// metadata, and nothing about what the payload means.
typedef struct packed {
  logic [OBJ_LEN_W-1:0]  length;      // transport extent, not a PCIe field
  logic                  start;       // framing, owned by the mapping
  logic                  end_of_obj;
} tunnel_meta_t;
 
typedef struct packed {
  tunnel_meta_t              meta;
  logic [OBJ_DATA_W-1:0]     data;    // opaque to the Adapter's datapath
  logic [OBJ_BYTES-1:0]      byte_valid;
} tunnel_item_t;

Architecture. The Adapter needs enough metadata to frame, protect, and retry an object. It does not need to know that the object is a memory read.

State. None here — this is the boundary representation.

Contract. The PCIe engine supplies objects and their transport metadata; the Adapter returns them intact.

Failure. Semantic leakage downward, which Chapter 5.4 identified as the failure that makes a layer un-reusable and un-independently-verifiable.

Native mapping is about the format being defined, not about the transport making decisions on transaction meaning. Those are different kinds of knowledge, and conflating them is how a protocol-agnostic Adapter becomes a PCIe-only one.

8. PCIe Link State Is Not UCIe Link State

Two state machines, two purposes, and collapsing them is a common integration error.

UCIe link state — Chapter 8.6's machine: RESET through training to ACTIVE, low power, recovery. It describes whether the die-to-die transport is usable.

PCIe link and device state — LTSSM state, configuration space, enumeration, device power state. It describes whether the PCIe function is usable.

They are related but not equal, and both directions of mismatch occur:

• UCIe ACTIVE, PCIe not ready. The transport is up; the PCIe engine has not completed its own initialisation. Traffic must not flow yet.
• PCIe configured, UCIe not operational. The PCIe side believes it has a link; the transport underneath is in recovery. Traffic must not flow yet.

// WRONG — one bit standing in for two independent state machines.
assign pcie_ready = ucie_link_operational;

This is Chapter 8.5's argument in a new place: a single link_up bit destroys the information needed to debug. When traffic does not flow, the first question is which layer is not ready, and one bit cannot answer it.

// Illustrative — a readiness chain, with each link separately observable.
assign transport_ready  = ucie_link_operational && adapter_resources_ready;
assign tunnel_ready     = transport_ready && tunnel_fifo_space;
assign pcie_may_send    = tunnel_ready && pcie_engine_ready;

Architecture. Several independent conditions must hold, each owned by a different block and each capable of failing on its own.

State. None of its own — a chain of conjunctions over registered facts from different layers, several of which cross clock domains and must be synchronised before use (Chapter 7.5 §9).

Cycle behaviour. Combinational; any term dropping withdraws permission in the same cycle it is observed.

Contract. The PCIe engine gates on pcie_may_send; nothing consumes the intermediate terms functionally, but every one of them is readable for debug.

Failure. Collapsing the chain means a stalled link gives one bit of diagnosis where four were available.

9. Boundary Buffering, and Who Owns an Accepted Object

The boundary needs a queue, and the reason is not only performance.

// Illustrative — a small queue at the PCIe/UCIe boundary.
localparam int TUNNEL_DEPTH = 4;
 
tunnel_item_t                          tun_mem [TUNNEL_DEPTH];
logic [$clog2(TUNNEL_DEPTH+1)-1:0]     tun_count_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n)                          tun_count_q <= '0;
  else if (tun_push && !tun_pop)       tun_count_q <= tun_count_q + 1'b1;
  else if (tun_pop  && !tun_push)      tun_count_q <= tun_count_q - 1'b1;
end

Architecture. The PCIe engine and the UCIe transport run at different rates and stall for different reasons. Without a queue, every UCIe hiccup — a retry, a recovery, a credit exhaustion — immediately stalls the PCIe engine, and PCIe engines generally have their own timing expectations about how long they may be held.

State. A few whole objects plus an occupancy count. Per-PCIe-object lifetime: an entry exists from acceptance until the transport has taken responsibility for it.

Cycle behaviour. Simultaneous push and pop leave the count unchanged.

Contract. This is the crucial one. The moment a PCIe object is accepted into this queue, the PCIe engine is entitled to consider it handed off — and the transport side becomes responsible for it.

Failure. See §10.

// WRONG — acceptance conditioned on the physical link alone.
assign pcie_pkt_ready = ucie_link_operational;

Missing: queue space, Adapter resources, and — per Chapter 9.5 §7 — credit and replay availability further down. Accepting an object with nowhere to put it either overwrites a queued entry or advances a pointer past the end of the buffer.

// Illustrative — acceptance requires somewhere to put it and a path onward.
assign pcie_pkt_ready = tunnel_ready && (tun_count_q != TUNNEL_DEPTH);

10. An Accepted Object Cannot Vanish

The invariant that connects this chapter to everything Module 8 and Module 9 established.

Once the handshake completes at the boundary, the PCIe engine has released its copy. If a UCIe recovery, retry, or link event then discards the object silently, a PCIe transaction disappears — and because PCIe's own completion timeouts are the only thing that will eventually notice, the symptom surfaces far away, much later, as an unrelated-looking timeout.

// WRONG — a transport event erases work the PCIe side already handed over.
if (ucie_recovery_entered)
  tun_count_q <= '0;                        // silently discards accepted objects

This is the fourth appearance of the same rule in this curriculum — Chapter 5.5 at the Adapter boundary, 7.1 at the PHY boundary, 7.5 at a clock loss, 8.6 at a state transition — and it does not change here:

A flush is acceptable. An unannounced flush is not. If the architecture permits abandoning accepted objects, the PCIe side must be told, so its own error handling can act.

// Illustrative — accepted objects are retained or their loss is reported.
property p_accepted_object_not_silently_dropped;
  @(posedge clk) disable iff (!rst_n)
    (tun_push && ucie_recovery_entered) |->
      ##[1:$] (object_transported || object_loss_reported);
endproperty

Note this one is written as a liveness property and needs an assumption that recovery resolves; in simulation it should be bounded. The stronger practical check is a scoreboard (§12) counting objects in and objects out.

11. Two Error Domains

Errors must not be laundered across the boundary, and the distinction is a debug lifeline.

The rule: do not manufacture a PCIe error from a transport event, and do not report a PCIe error as a link fault.

A flit CRC failure that was successfully retried is a transport event that should appear in transport telemetry — turning it into a PCIe error would tell software a transaction was malformed when it was not. Conversely, a genuinely malformed TLP is not a link problem, and reporting it as one sends the investigation to the PHY.

Where they legitimately connect is when a transport failure becomes unrecoverable. If the link cannot deliver an object at all, the PCIe side must eventually learn — but as a defined consequence, such as the transaction failing to complete and PCIe's own timeout mechanism acting, rather than as a fabricated PCIe error code invented by the Adapter. Whether UCIe defines a specific mapping from transport failure to PCIe-visible error is a specification question, and inventing one would be exactly the pre-audit mistake this curriculum now avoids.

12. Two Scoreboards, Because There Are Two Contracts

The verification architecture follows directly from §11, and it is the chapter's main DV point.

The transport scoreboard checks that UCIe carried what it was given: objects in equal objects out, intact, with retries invisible above the transport boundary.

The PCIe semantic scoreboard checks that PCIe semantics survived: every TLP appears exactly once, ordering rules are respected, completions match their requests, and tags are preserved.

Neither implies the other, and the two failure cases are instructive:

• Transport correct, semantics wrong. Every object crossed intact, but the mapping reordered two TLPs whose relative order PCIe requires. The transport scoreboard passes; PCIe is broken.
• Semantics correct, transport wrong. The PCIe side sees valid traffic, but the transport is retrying constantly and throughput is a fraction of expectation. The semantic scoreboard passes; the link is sick.

Error injection at this boundary: a UCIe stall while the PCIe engine has traffic; a retry with objects outstanding; a link recovery with the tunnel queue non-empty; a tunnel queue driven full; and PCIe-side backpressure. In every case the check is the same — no accepted object is lost, duplicated, or reordered, and the readiness chain reports which layer is holding.

// Illustrative tunnel-boundary coverage — not UCIe-defined.
covergroup cg_tunnel @(posedge clk);
  cp_occ    : coverpoint tun_count_q {
    bins empty = {0}; bins mid = {[1:TUNNEL_DEPTH-1]}; bins full = {TUNNEL_DEPTH};
  }
  cp_ready  : coverpoint {ucie_link_operational, tunnel_ready, pcie_engine_ready};
  cp_event  : coverpoint transport_event;    // none / retry / recovery
  // Was the queue non-empty when a transport event occurred?
  x_occ_by_event : cross cp_occ, cp_event;
endgroup

Why that cross. A recovery with an empty tunnel queue proves nothing about ownership. A recovery with objects in the queue is exactly §10's case, and it is the one a regression will otherwise never produce.

13. What Tunnelling Costs

Honest accounting, because "PCIe over UCIe" is sometimes described as free.

Latency is added. The path runs PCIe engine → boundary queue → Adapter → PHY → package → remote PHY → remote Adapter → remote boundary → remote PCIe engine. Each stage is real. Latency-sensitive PCIe traffic — completions in particular — feels this.

Retry adds variance. Chapter 9.4's replay means occasional latency spikes at unpredictable moments, which is different in character from a conventional PCIe link's behaviour.

Bandwidth has overheads. Flit headers and CRC consume capacity; the published packing efficiencies are roughly 94 % for the 68B format and 97.7 % for the 256B latency-optimised format, on payload against header and CRC bytes.

What is not paid is semantic cost: the transaction model, the ordering rules, the completion behaviour, and the software view are unchanged. That is the trade — some latency and some bandwidth overhead, in exchange for keeping an entire ecosystem.

14. Debug Checklist

1. Is the UCIe link operational? Chapter 8.5's checklist first — everything below is meaningless otherwise.
2. Which term of the readiness chain is low? Transport, tunnel, or PCIe engine — this is why §8 keeps them separate.
3. Is the tunnel queue full? Then the transport is the constraint, and Chapter 9.5's credit and replay questions apply.
4. Is the tunnel queue empty while PCIe stalls? Then the PCIe engine is not offering, and the problem is above the boundary.
5. Are transport errors being reported as PCIe errors, or vice versa? §11 — a misfiled error sends the investigation to the wrong layer entirely.
6. Did a recovery occur with objects in the queue? §10 — check they were retained or reported.
7. Do object counts balance across the boundary? In versus out, both directions.
8. Is a PCIe completion timing out? Ask whether the request was ever accepted at the boundary — a lost object presents as a completion timeout far from its cause.
9. Is throughput low with no errors? Transport overhead, credit starvation, or queue depth — not a semantic problem.
10. Do both dies agree on which protocol is mapped? Parameter negotiation is the Adapter's, and a mismatch fails in confusing ways.

Step 5 deserves emphasis. A misfiled error is worse than no error, because it actively directs the investigation to the wrong layer — and both directions of misfiling happen.

15. Common Misconceptions

"PCIe over UCIe converts PCIe into another protocol." The PCIe engine at each end still speaks PCIe; nothing is converted. Calling it translation invites questions whose correct answer is "that does not happen" (§3).

"The Adapter should understand PCIe tags and ordering." Native mapping defines the format; it does not license transport decisions based on transaction meaning. Semantic leakage downward makes the Adapter protocol-specific and revision-coupled (§7).

"UCIe ACTIVE means the PCIe endpoint is ready." Two independent state machines, and both directions of mismatch occur (§8).

"PCIe link state and UCIe link state are one FSM." Collapsing them into one bit destroys the information needed to answer "which layer is not ready?" (§8).

"Tunnelling removes PCIe ordering requirements." Semantics are preserved, which means the ordering rules come along and the mapping must respect them (§6, and Chapter 10.2).

"A UCIe transport error is automatically a PCIe error." A retried CRC failure is a transport event. Manufacturing a PCIe error from it tells software a transaction was malformed when it was not (§11).

"If the tunnel accepted a TLP, it may drop it on link loss." Once accepted, its fate must remain knowable. This rule has now appeared at four boundaries (§10).

"PCIe over UCIe is just a longer wire." UCIe substitutes the data-link and physical functions with a different mechanism, adds latency and retry variance, and changes where errors are detected (§5, §13).

16. Understanding Check

17. Summary and What Comes Next

UCIe replaces the transport path beneath PCIe, not PCIe's transaction semantics. A memory read stays a memory read, which is what allows an entire ecosystem of PCIe IP, verification, and software to transfer to a die-to-die link.

Tunnelling is not translation — nothing is converted, and asking how completions are remapped is asking the wrong question. It is also not Module 9's opaque carriage: UCIe maps PCIe and CXL natively, with flit sizes aligned to those protocols' own definitions, which is why two PCIe-over-UCIe chiplets interoperate on the strength of PCIe rather than a private agreement.

The boundary, stated as a sentence: the transaction layer tunnels; the data link layer is largely what gets replaced, since the Adapter provides CRC, retry, link state, and flit-based flow control — the functions DLLPs otherwise carry. That is the architectural shape, with the exact division flagged as a specification question, and it is why Chapter 10.2 must qualify the "TLPs / DLLPs" shorthand rather than repeat it.

Three implementation rules follow. The Adapter must not read PCIe semantics — native mapping defines a format, not a licence to make transport decisions on transaction meaning. PCIe link state and UCIe link state are different machines, so readiness is a chain whose terms stay separately observable. And an accepted object cannot vanish — the fourth appearance of that rule, and here its symptom is a PCIe completion timeout far from its cause.

Finally, two error domains and two scoreboards. A retried CRC failure is a transport event; a malformed TLP is not a link fault; and a misfiled error sends the investigation to the wrong layer.

This chapter established ownership. The mechanics are next — how PCIe objects are actually represented in the transport, how flits carry them, and what happens to a PCIe transaction when the flit carrying it is replayed:

• 10.2 — PCIe Packet Transport — the mapping machinery, its buffers and their lifetimes, and why transport retry must never become a duplicated PCIe transaction.

Browse the full path on the UCIe tutorials index.