Die-to-Die Communication Flow

Everything in Module 4 has been building to this. You have the three layers (4.1), what each owns and where its state lives (4.2), how representation changes as information descends (4.3), and how the physical link is organised (4.4). This chapter puts a single payload into the top of one die's stack and follows it, cycle by cycle, until the other die's Protocol Layer accepts it.

It is the most implementation-grounded chapter of the module because the answer to "how does a payload get from A to B?" is not a diagram — it is a sequence of local handshakes, each with state, each able to stall, each able to fail. Understanding that sequence is what lets you design a layer, verify a boundary, or debug a link that has stopped delivering.

1. Nobody Sends End-to-End

The single most useful correction in this chapter, and it contradicts the natural mental picture.

No layer transmits a payload to the far die. Each layer does exactly four things, over and over:

1. Accept from its upstream neighbour, when it has somewhere to put the payload.
2. Retain it — the payload is now this layer's responsibility.
3. Transform it into its own representation (Chapter 4.3).
4. Transfer ownership downstream when the downstream neighbour is ready.

End-to-end delivery is what emerges when that four-step cycle runs at every boundary in sequence.

End-to-end delivery is a chain of local handshakes. Nothing sends end-to-end; every layer only ever talks to its immediate neighbour.

This is why "the payload was sent" is not a meaningful statement during debug. The meaningful questions are always local: did this specific boundary hand off, and does the payload now exist on the other side of it?

2. Ownership Transfers on Handshake, Not on Valid

Be precise, because this is where beginners lose payloads.

Asserting valid offers a payload. It does not transfer it. The upstream layer still owns it, must still hold it unchanged, and must be prepared to keep offering it indefinitely.

Ownership transfers only on the cycle where the handshake completes — for the illustrative interface used throughout Module 4, the cycle where valid && ready are both high at the clock edge. Before that edge the payload belongs to the sender; after it, to the receiver. There is no in-between state and no partial transfer.

Two rules follow, and every boundary in this chapter obeys them:

• The sender must hold valid and the payload stable until the handshake completes. Deasserting valid because a cycle passed means the offer was withdrawn, not accepted.
• The receiver must not assert ready unless it can keep what it accepts. Readiness is a promise of storage, which is why Chapter 4.2 insisted a boundary needs state behind it.

3. The Boundary Element

Because every boundary behaves identically, it is worth building the element once and reusing it mentally at each one.

Illustrative boundary element — not UCIe normative interface naming. Real FDI and RDI carry considerably more; this isolates ownership transfer and backpressure.

// One-entry boundary stage. Holds at most one payload, accepts when it
// has room, and presents downstream until the payload is taken.
module one_entry_pipe #(
  parameter int unsigned W = 256
) (
  input  logic         clk,
  input  logic         rst_n,
 
  input  logic [W-1:0] up_data,
  input  logic         up_valid,
  output logic         up_ready,
 
  output logic [W-1:0] dn_data,
  output logic         dn_valid,
  input  logic         dn_ready
);
 
  logic [W-1:0] data_q;
  logic         full_q;
 
  // Accept when empty, or when the held payload is leaving this cycle.
  assign up_ready = !full_q || dn_ready;
  assign dn_valid = full_q;
  assign dn_data  = data_q;
 
  always_ff @(posedge clk) begin
    if (!rst_n) begin
      full_q <= 1'b0;
      data_q <= '0;
    end else begin
      // Fill and drain are independent decisions in the same cycle.
      if (up_valid && up_ready) begin
        full_q <= 1'b1;            // accepting: we now own a payload
        data_q <= up_data;
      end else if (full_q && dn_ready) begin
        full_q <= 1'b0;            // draining with nothing arriving
      end
    end
  end
 
endmodule

Architecture. The minimum honest boundary: it accepts only what it can keep, and it keeps what it accepts until the next stage takes it.

State. full_q is ownership — this stage is responsible for a payload right now. data_q is the payload. That is the entire state, and full_q is the single most useful signal to look at when debugging a stalled link.

Cycle behaviour. up_ready is high when empty or when the held payload is being taken this cycle. That second term is what allows simultaneous drain and fill: on a cycle where dn_ready is high and up_valid is high, the stage hands its payload downstream and accepts a new one on the same edge. Without it, the stage would need an empty cycle between payloads and throughput would halve.

Contract. Upstream may rely on: if up_ready was high, the payload is now retained. Downstream may rely on: while dn_valid is high and dn_ready is low, dn_data does not change.

Failure/DV. Three ways to get this wrong, each with its own assertion in §4. Make up_ready unconditionally high and payloads are dropped. Make it !full_q only and you lose half the throughput (a performance bug, not a correctness one). Assert dn_valid without full_q and you present stale data as real.

4. The Three Boundary Invariants

Illustrative boundary properties — conceptual invariants, not UCIe normative requirements.

// 1. STABILITY. An offered payload must not change while it is being
//    refused -- otherwise the receiver may capture a different payload
//    than the one it decided to accept.
property p_stable_while_stalled;
  @(posedge clk) disable iff (!rst_n)
    (dn_valid && !dn_ready) |=> (dn_valid && $stable(dn_data));
endproperty
assert property (p_stable_while_stalled)
  else $error("payload or valid changed while downstream was stalling");
 
// 2. NO OVERWRITE. Never accept when full and not draining -- the new
//    payload would overwrite one that was never handed on.
property p_no_accept_when_full_and_stalled;
  @(posedge clk) disable iff (!rst_n)
    (full_q && !dn_ready) |-> !up_ready;
endproperty
assert property (p_no_accept_when_full_and_stalled)
  else $error("accepted a payload with no room to keep it");
 
// 3. NO UNDERFLOW. Never claim to be offering a payload we do not hold.
property p_no_valid_when_empty;
  @(posedge clk) disable iff (!rst_n)
    dn_valid |-> full_q;
endproperty
assert property (p_no_valid_when_empty)
  else $error("asserted valid downstream with no payload held");

Each catches a distinct failure. Stability catches corruption — the receiver decided to accept payload A and captured payload B. No-overwrite catches loss — a payload is silently destroyed while the counters still look plausible, which is the hardest form to trace. No-underflow catches fabrication — presenting stale or undefined data as a real payload.

Bind these at every boundary and the local behaviour of the chain is covered. §12 explains precisely what they still do not prove.

5. The Transmit Sequence

Now follow one payload. This is an architectural sequence, not an implementation latency claim — a real design may pipeline any of these steps deeper, and two conforming implementations can take different numbers of cycles.

• Cycle 0. The Protocol Layer asserts valid with the payload. The link is still coming up — the Adapter has not reached its operational state, so ready at the Protocol boundary is low. Nothing transfers. The Protocol Layer holds valid and the payload steady. This is rule one of §2 doing its job: the payload is not lost, it is simply not yet accepted.
• Cycle 1. Bring-up completes and the link becomes operational (Chapter 4.2's FSM reaching its ready state, much of that work having happened over the sideband of Chapter 4.4). The Adapter's boundary stage is empty, so ready rises. valid && ready — the handshake completes and ownership transfers to the Adapter.
• Cycle 2. The Adapter holds the payload (full_q set) and transforms it into its transport representation (Chapter 4.3) — attaching transport metadata such as error-detection information in the modes that use it. It offers the transport unit downstream. Suppose the PHY is busy: ready at that boundary is low. The Adapter holds, payload stable.
• Cycle 3. The PHY becomes ready. The handshake completes and ownership transfers to the PHY. The Adapter's stage drains and, if the Protocol Layer is offering another payload this same cycle, fills again on the same edge — the simultaneous drain-and-fill of §3.
• Later. The PHY maps the transport unit across the module's data lanes (Chapter 4.4), drives it with its forwarded clock, and the transfer crosses the package.

Notice that at every step the pattern is identical: offer, hold while refused, transfer on handshake. The layers differ in what they do with the payload while they hold it; they do not differ in how they hand it over.

6. Crossing the Package Is Not a Layer

A short but important clarification.

The package channel is not another protocol layer. There is no state machine in the package, nothing that accepts or refuses, no handshake. It is a physical medium with physical effects — attenuation, crosstalk, delay — and those effects are the PHY's problem on both sides.

So the chain of handshakes has a gap in the middle: five ownership transfers happen between logic blocks, and one step is bits propagating through conductors. That gap is exactly why the receive path needs validation (§8) — it is the only place in the chain where a payload can be altered by something that is not a design decision.

7. The Receive Sequence

Chapter 4.3 established that receive is not transmit reversed. Here is what that means concretely.

• The receiving PHY recovers the transfer from the lanes: sampling against the forwarded clock, undoing the lane mapping of Chapter 4.4, and reassembling the transport unit. It offers the result upward.
• The receiving Adapter accepts it, then validates — in the modes with error detection enabled, checking the integrity information the transmitting Adapter attached. This is the decision point transmit does not have. It then strips its own transport metadata (Chapter 4.3's containment rule) and offers the semantic unit upward.
• The receiving Protocol Layer accepts it and it becomes a transaction again.

Every one of those steps is still a local handshake with the same three invariants. What differs is the validation step in the middle, and what it implies when it fails.

8. The Error Path

One error case, because the invariant it produces is one of the most important in the module.

Suppose the physical transfer is corrupted in the channel. The receiving PHY recovers something — it cannot know the bits are wrong — and hands it upward. The receiving Adapter validates and the check fails.

What must not happen is that the payload continues upward. A corrupted transport unit is not a valid semantic unit, and delivering it would hand the Protocol Layer a transaction that was never sent.

Corrupted transport must never become accepted protocol payload.

What happens instead depends on the configured mode, and that detail belongs to later modules: where link-level retry is enabled the Adapter can request retransmission; in raw mode the Protocol Layer carries the error-handling responsibility. What is architectural here is the stop condition, which is assertable:

// Illustrative property: a transport unit that failed its integrity check
// must not be delivered upward as valid protocol payload.
// Simplified timing -- real designs pipeline detection, so the indication
// and the payload may be separated by known stages.
property p_no_deliver_on_bad_transport;
  @(posedge clk) disable iff (!rst_n)
    transport_bad |-> !proto_rx_valid;
endproperty
assert property (p_no_deliver_on_bad_transport)
  else $error("payload delivered upward after a failed integrity check");

Contract. The Protocol Layer relies on this absolutely. Its entire model of the link is that anything arriving is genuine; if that fails, no amount of protocol-level care recovers the situation, because the Protocol Layer has no way to know.

Failure mode. Violating it is the worst class of bug in the stack — corrupt data accepted as genuine, propagating into whatever the system does with it, with no error reported anywhere.

9. Backpressure Propagates Backwards — Eventually

Now run the chain in reverse. Suppose the receiving Protocol Layer stops accepting.

1. The receiving Adapter's boundary stage cannot drain. Its full_q stays set.
2. It therefore stops asserting ready toward the receiving PHY.
3. The receiving PHY cannot hand upward; its own storage fills.
4. Eventually the receiving side can no longer accept transfers from the link.
5. The transmitting side must then stop sending — and the mechanism by which that is communicated is flow control, whose details belong to later modules.
6. With nowhere to send, the transmitting Adapter's stage stays full, so it stops asserting ready to the transmitting Protocol Layer.
7. The transmitting Protocol Layer stalls.

A stall at the far end eventually becomes a resource constraint at the source.

The word eventually carries the weight, and this is the section's real content. Backpressure does not propagate instantly, because every stage in the chain holds state. Each buffer absorbs one payload before it has to push back. In the transmit chain of §5 there are several such stages, plus whatever depth exists inside each layer.

That has two consequences worth holding together. Bursts are absorbed — a brief downstream stall need not reach the source at all, which is much of why buffering exists. And finite buffers eventually fill — absorption is capacity, not immunity, so a sustained stall always reaches the source in the end. The delay between the two is exactly the window in which a system can misbehave if flow control is wrong, which is why flow control gets its own treatment later.

Occupancy is the state that makes this visible, and it is worth being able to write:

// Illustrative occupancy tracking for a multi-entry boundary stage.
localparam int unsigned DEPTH = 4;
 
logic [$clog2(DEPTH+1)-1:0] occupancy_q;
logic                       do_accept, do_send;
 
assign do_accept = up_valid && up_ready;
assign do_send   = dn_valid && dn_ready;
 
always_ff @(posedge clk) begin
  if (!rst_n) begin
    occupancy_q <= '0;
  end else begin
    unique case ({do_accept, do_send})
      2'b10:   occupancy_q <= occupancy_q + 1'b1;  // in only
      2'b01:   occupancy_q <= occupancy_q - 1'b1;  // out only
      default: occupancy_q <= occupancy_q;         // both or neither
    endcase
  end
end
 
assign up_ready = (occupancy_q < DEPTH[$clog2(DEPTH+1)-1:0]) || do_send;
assign dn_valid = (occupancy_q != '0);

Cycle behaviour. The default arm is the important one: accepting and sending on the same edge leaves occupancy unchanged, which is the multi-entry form of §3's simultaneous drain-and-fill.

Failure/DV. Occupancy is the one piece of state that can be checked for consistency directly, and both directions matter:

// Illustrative occupancy safety properties.
property p_no_overflow;
  @(posedge clk) disable iff (!rst_n) occupancy_q <= DEPTH;
endproperty
assert property (p_no_overflow) else $error("occupancy exceeded depth");
 
property p_no_underflow;
  @(posedge clk) disable iff (!rst_n) do_send |-> (occupancy_q != '0);
endproperty
assert property (p_no_underflow) else $error("sent from an empty stage");

An overflow means a payload was overwritten — silent loss. An underflow means one was fabricated — sending something never accepted. Both are ownership failures: the first drops it, the second invents it.

10. Ordering

If two payloads A and B are accepted at a boundary in that order, and the architecture requires order preservation at that stage, then B must not overtake A anywhere downstream.

Two observations. The single-entry element of §3 preserves order structurally — one payload at a time, so overtaking is impossible. But multi-entry structures and any parallelism can reorder, and multi-module links (Chapter 4.4) introduce exactly the sort of parallelism where care is needed.

The verification consequence is important and easy to miss: the three invariants of §4 do not prove ordering. A design can satisfy stability, no-overwrite, and no-underflow at every boundary while still delivering B before A — for instance by reloading an output stage from the newest input while an older payload waits elsewhere. Proving order requires either an explicit ordering property tying accepted payloads to emitted ones in sequence, or a scoreboard in a dynamic environment.

What ordering the UCIe contract actually requires, and where, is protocol- and configuration-dependent and belongs to later modules. What belongs here is the engineering habit: if a stage may reorder, say so explicitly and check it, because local invariants will not catch it.

11. Two Waveforms

Make the handshake rule concrete, because this is the single most common junior mistake in interface design.

The broken waveform. The transmitting Adapter has a transport unit ready:

• Cycle N. Adapter asserts valid with the transport unit. The PHY is busy: ready low. No handshake.
• Cycle N+1. The Adapter deasserts valid — its designer assumed presenting it for a cycle was enough, or moved on to other work.
• Cycle N+2. The PHY becomes ready and asserts ready. There is nothing being offered.

The payload was never handed over. Whether it is lost depends on whether the Adapter still holds it — and if the Adapter cleared its own state believing the transfer occurred, it is gone. Nothing reports an error; the symptom is a missing completion much later.

The correct waveform.

• Cycle N. Adapter asserts valid; ready low; payload held.
• Cycle N+1. valid still high, payload unchanged; ready still low.
• Cycle N+2. ready rises. valid && ready — handshake completes, ownership transfers, and only now may the Adapter release its copy.

The rule is simply §2 restated: an offer stands until it is accepted, and the sender's obligation ends only at the handshake.

12. Why Layer-Local Correctness Is Not Enough

A design can satisfy every property in this chapter at every boundary and still fail end-to-end. Three reasons, and they are the argument for keeping an end-to-end environment:

• Ordering is not covered by the local invariants (§10).
• Transformation correctness is not either. §4's properties check that a payload is not lost or altered at a boundary; they say nothing about whether the Adapter's transformation was right, whether metadata was correctly attached and stripped (Chapter 4.3), or whether the lane mapping matches at both ends (Chapter 4.4).
• Shared misunderstanding between neighbours. Each layer is verified against a model of its neighbour. If a layer and the model of its neighbour share a wrong assumption, both pass — the same structural gap Chapter 3.2 identified between endpoint and interoperability verification, now appearing between layers inside one endpoint.

Local invariants prove that the chain does not drop or corrupt what it is given. Only end-to-end verification proves that what the transmitting Protocol Layer sent is what the receiving Protocol Layer got.

13. A Debug Method You Can Reuse

When a payload never arrives, do not guess. Walk the chain and find the first step that answers no — that is where the defect is.

1. Did the transmitting Protocol Layer assert valid with the expected payload?
2. Did the transmit boundary handshake — valid && ready on the same edge — and did the Adapter's full_q set?
3. Did the Adapter present a transport unit downstream?
4. Did the transmitting PHY accept it?
5. Was the link operational at that moment, and were the configured lanes healthy (Chapter 4.4)?
6. Did the receiving PHY recover a transfer?
7. Did the receiving Adapter accept it?
8. Did the integrity check pass, in the modes where one applies?
9. Did the receiving Adapter present the semantic unit upward?
10. Did the receiving Protocol Layer accept it?

Two habits make this powerful. Check handshakes, not valids — step 2 is the difference between "we offered it" and "it was taken", and the §11 waveform lives exactly there. And note where ownership stops: the last stage whose occupancy went high is holding the payload, or was the last to hold it, which localises the fault to one layer or one boundary immediately.

14. Common Misconceptions

15. Understanding Check

16. Summary and What Comes Next

End-to-end delivery is a chain of local handshakes. No layer sends to the far die; each accepts from its neighbour, retains, transforms, and hands on. Ownership transfers only when a handshake completes — asserting valid offers a payload, and until valid && ready coincide at an edge the sender still owns it and must hold it unchanged.

Every boundary is the same element: storage plus occupancy, accepting only when it has room, presenting until taken, with up_ready = !full_q || dn_ready enabling simultaneous drain and fill so throughput is not halved. Three local invariants hold it together — stability under backpressure (catches corruption), no acceptance when full and stalled (catches loss), no valid when empty (catches fabrication) — with occupancy bounds for deeper stages.

Transmit is offer-hold-transfer repeated; the package crossing is not a layer but a physical gap with no handshake, which is why receive adds validation. Corrupted transport must never become accepted protocol payload — the flow stops at the check, and what happens next is mode-dependent. Backpressure propagates all the way back but not instantly, because every stage absorbs one payload first: bursts are absorbed, sustained stalls always arrive, and the gap between those facts is what flow control exists to manage.

Finally, the honest limit: local invariants prove the chain does not drop or corrupt what it is given. They do not prove ordering, transformation correctness, or freedom from a misunderstanding shared between a layer and the model of its neighbour — which is why end-to-end verification remains necessary, and why the debug method walks handshakes and ownership rather than guessing.

That completes the architecture overview. Module 5 takes the same three layers and examines each one properly, starting at the top:

• 5.1 — The Protocol Layer — PCIe, CXL, and Streaming on top of UCIe, and what the Protocol Layer actually does with each.

Browse the full path on the UCIe tutorials index.