Layer Interactions — Following One Transaction Through the Stack

Chapter 3.4 treated the layers as separable, which is what makes ownership decidable. That separability is real, and it is also only half the picture — because in a working system the layers are composed, and composition creates behaviour no single layer exhibits.

This chapter follows one operation all the way through:

How does one PCIe transaction move down a component's stack, across a Link, and back up the far-side stack?

1. The Running Example

One operation, used throughout, chosen because it exercises both directions of the stack and both roles:

An Endpoint has assembled data and needs it written into host memory.

The device engine has a buffer and a destination address. Everything that follows is what happens between that intent and the data landing in memory.

2. Down the Stack at the Source

Step 1 — local intent. A data mover inside the Endpoint has a buffer and a target address, expressed in whatever internal form the device's designers chose. This is not yet a transaction; it is a device-internal request.

Step 2 — Transaction Layer. The request is classified: this is a write, directed at a particular address in the system address map, carrying this much data, requiring nothing returned. A transaction is assembled carrying that information plus the payload. The layer's work is now complete — it hands the packet down and, per Chapter 3.1, proceeds as though it arrives.

Step 3 — Data Link Layer. The packet is accepted for delivery across this one Link. It is given an identity within this relationship, retained in retry storage, and marked unresolved. Integrity information is attached so the neighbour can determine whether it arrived intact. Note what did not happen: nothing about the operation was examined. The packet is opaque here.

Step 4 — Physical Layer. The information is framed, conditioned for the channel, distributed across the connection's resources, serialised, and driven. Per Chapter 3.3, this only proceeds if the connection is available; if not, the boundary refuses and the stall propagates upward — §6.

3. Up the Stack at the Destination

Step 5 — far-side Physical Layer. Signalling is captured, timing recovered, deserialised, reassembled, and de-framed back into logical information. It is passed up.

Step 6 — far-side Data Link Layer. Integrity is checked. If the packet arrived intact, it is accepted and passed up, and the local relationship's bookkeeping records that outcome so the neighbour learns of it. If not, it is not passed up, and the transmitter will eventually resend from its retained copy.

Step 7 — far-side Transaction Layer. Now the packet is interpreted. What operation is this? Is it directed here? Is it legal for this component? Which local consumer owns it? For our write, it is a memory write directed at host memory.

Step 8 — local consumer. The operation reaches whatever actually performs it, and memory is updated.

4. The Response Case, Briefly

A read differs in one structurally important way, and it is worth one short section rather than a repeat of the walkthrough.

The requester's Transaction Layer holds outstanding state from step 2 until a response returns. The response is itself a full transaction: it is created by the completer's Transaction Layer, travels down its stack, across the Link, and up the requester's stack, arriving as an ordinary arriving transaction — which the requester's Transaction Layer must then match to the outstanding request that asked for it.

Two consequences follow, both practical:

• The response's journey is independent. It gets its own delivery state on each hop, its own retries if needed. The request's successful delivery says nothing about the response's.
• The requester's resources stay committed for the whole round trip, not just until the request was delivered. That is why outstanding-tracking capacity, not link capability, is often what limits how many reads a device can have in flight.

5. Through a Switch — Where Delivery Scope Terminates

This is the section that makes hop-local reliability unforgettable, because it shows the mechanism rather than asserting it.

Change the topology: the Endpoint reaches the Root Complex through a Switch. Per Chapter 2.8, that is one path made of two Links.

Now trace the same write:

1. The Endpoint's stack does steps 1–4 exactly as before. Its Data Link Layer assigns delivery state for Link 1 and retains the packet.
2. The Switch's Physical Layer receives it; its Data Link Layer checks integrity and accepts it. The Link 1 relationship is now complete — the Switch reports the outcome to the Endpoint, and the Endpoint may release its retained copy.
3. The Switch forwards the transaction toward the Root Complex. Its Data Link Layer for Link 2 assigns new delivery state — a different identity, in a different relationship — and retains the packet in its own retry storage.
4. The Root Complex receives it, checks integrity, accepts, reports the outcome to the Switch, and passes it up to its Transaction Layer.

6. The Backpressure Chain

Now the interaction that has no single-layer explanation, and which produces some of the most confusing symptoms in real systems.

Backpressure propagates upward, one stage at a time, and each stage absorbs it only until its storage fills:

1. The connection becomes unavailable, or the far side stops accepting. The Physical Layer boundary stops accepting from above (Chapter 3.3 — it must refuse rather than accept what it cannot transmit).
2. The Data Link Layer cannot hand packets down. It continues accepting from above while it has retry storage — but nothing is being resolved, so nothing is released.
3. Retry storage fills. Now the Data Link Layer deasserts ready upward (Chapter 3.2).
4. The Transaction Layer cannot hand packets down. Its own queues fill.
5. The Transaction Layer deasserts ready to the local source.
6. The device engine stalls.

Two properties of this chain are worth internalising.

The symptom appears furthest from the cause. The engine stalls, so a device-level observer sees "the device stopped working" — five stages from the actual condition. Any debugging approach that starts where the symptom was reported starts in the wrong place.

Storage depth changes latency, not outcome. Deeper buffers delay each propagation step. They do not prevent it. A system that appears fine under short disturbances and stalls under long ones has not solved backpressure; it has bought time.

7. Stall Versus Deadlock

Both look like "nothing is moving." They are different conditions with different fixes, and distinguishing them is a genuinely useful skill.

Temporary backpressure — a downstream stage is momentarily busy. Resolves on its own. Normal.

Persistent stall — a downstream condition is not clearing: a connection stays unavailable, resolution indications stop arriving. The chain is working correctly; something at the bottom is genuinely stuck. Fix the bottom condition and the chain drains.

Deadlock — no stage can advance because each waits on another, in a cycle. Nothing external is broken. Fixing "the bottom" is impossible because there is no bottom; the dependency is circular.

The diagnostic that separates them:

Follow the ready chain downward from the stalled source. If it terminates at an external condition, it is a stall. If it comes back around to something waiting on the stalled source, it is a deadlock.

A concrete shape worth recognising: a stage that must accept an incoming item before it can release an outgoing one, paired with a peer doing the same in the opposite direction, and shared storage between them. Neither can proceed. Real interconnects avoid this class of problem by architectural means — separating resources so that classes of traffic cannot block each other — and PCIe has its own provisions, developed in Module 16. The reasoning to carry now is structural: a cycle in the wait-for relation is a deadlock, and no amount of buffering removes a cycle.

8. A Pipeline That Shows Composition

The RTL worth writing here is not another single boundary but stages composed, because the interaction properties only appear in composition.

// Illustrative internal representation — NOT a normative PCIe format.
// trace_id is an internal correlation handle for debug/verification only.
typedef struct packed {
  logic [15:0]  trace_id;   // internal handle — NOT carried by PCIe
  logic [127:0] meta;       // opaque per-item metadata
} pipeline_item_t;

// Illustrative synthesizable RTL — three composed stages, NOT PCIe layers.
// Demonstrates: ownership transfer on handshake, upward backpressure
// propagation, and per-stage occupancy observability.
module layer_pipeline (
  input  logic           clk,
  input  logic           rst_n,
 
  // From the local source
  input  logic           src_valid,
  output logic           src_ready,
  input  pipeline_item_t src_item,
 
  // To the connection boundary
  output logic           out_valid,
  input  logic           out_ready,
  output pipeline_item_t out_item,
 
  // Observability — which stages currently hold an item
  output logic [2:0]     stage_occupied
);
 
  // One item per stage. Real designs use deeper storage per stage; a single
  // register makes the propagation behaviour visible without obscuring it.
  pipeline_item_t s0_item, s1_item, s2_item;   // "TL", "DLL", "PHY boundary"
  logic           s0_full, s1_full, s2_full;
 
  // Ready propagates upward: a stage can accept if it is empty, or if its
  // downstream will take what it holds this cycle.
  wire s2_ready = !s2_full || out_ready;
  wire s1_ready = !s1_full || s2_ready;
  wire s0_ready = !s0_full || s1_ready;
 
  assign src_ready      = s0_ready;
  assign out_valid      = s2_full;
  assign out_item       = s2_item;
  assign stage_occupied = {s2_full, s1_full, s0_full};
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      s0_full <= 1'b0;
      s1_full <= 1'b0;
      s2_full <= 1'b0;
      s0_item <= '0;
      s1_item <= '0;
      s2_item <= '0;
    end else begin
      // Stage 2 (nearest the connection)
      if (s2_ready) begin
        s2_full <= s1_full;
        if (s1_full) s2_item <= s1_item;
      end
      // Stage 1
      if (s1_ready) begin
        s1_full <= s0_full;
        if (s0_full) s1_item <= s0_item;
      end
      // Stage 0 (nearest the source)
      if (s0_ready) begin
        s0_full <= src_valid;
        if (src_valid) s0_item <= src_item;
      end
    end
  end
endmodule

What this models: ownership transferring stage to stage on handshake, with ready computed upward so a downstream stall propagates in the same cycle it occurs.

What is deliberately simplified: one item per stage rather than realistic buffering; no per-stage transformation (real layers change the representation); no error paths; no retry storage — this pipeline has no way to resend, which is precisely what the Data Link Layer would add.

What to notice, and it is the whole point:

• ready is combinational upward. s0_ready depends on s2_ready, so out_ready falling stalls the source in the same cycle. That is the backpressure chain of §6 in five lines of RTL.
• valid flows downward, ready flows upward. Data moves one way; permission moves the other. Every interaction property in this chapter follows from that opposition.
• trace_id travels with the item but means nothing to the hardware. It is not identity in any protocol sense. Confusing an internal handle with transaction identity is a real design error — the pipeline must work if it were removed.
• stage_occupied is observability. Without per-stage occupancy visible, diagnosing where a pipeline is stalled means guessing.

What production RTL would additionally require: per-stage transformation, deeper and correctly-sized buffering, error and status propagation, and CDC if stages sit in different clock domains.

9. Cross-Stage Assertions

Composition invariants — properties no single stage can violate alone.

// SVA over the illustrative pipeline. Internal contract, not PCIe requirements.
 
// P1 — CONSERVATION. Items in minus items out equals items held.
logic [31:0] n_in, n_out;
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin n_in <= '0; n_out <= '0; end
  else begin
    if (src_valid && src_ready) n_in  <= n_in  + 1;
    if (out_valid && out_ready) n_out <= n_out + 1;
  end
end
property p_conservation;
  @(posedge clk) disable iff (!rst_n)
  (n_in - n_out) == ($countones(stage_occupied));
endproperty
a_conservation : assert property (p_conservation);
 
// P2 — STABILITY UNDER STALL. A held item does not change while blocked.
property p_out_stable;
  @(posedge clk) disable iff (!rst_n)
  (out_valid && !out_ready) |=> $stable(out_item);
endproperty
a_out_stable : assert property (p_out_stable);
 
// P3 — NO ACCEPT WITHOUT ROOM. The source is only stalled when genuinely
// unable to progress; conversely, accepting implies room existed.
property p_accept_implies_room;
  @(posedge clk) disable iff (!rst_n)
  (src_valid && src_ready) |-> (!s0_full || s1_ready);
endproperty
a_accept_room : assert property (p_accept_implies_room);
 
// P4 — ORDERING. For this in-order pipeline, trace ids leave as they entered.
// Assumption: the source issues strictly increasing trace_id values.
property p_in_order;
  @(posedge clk) disable iff (!rst_n)
  (out_valid && out_ready) |=> (!(out_valid && out_ready) or
                                out_item.trace_id > $past(out_item.trace_id));
endproperty
a_in_order : assert property (p_in_order);
 
// P5 — LIVENESS, with an explicit environment assumption. Without the
// assumption this is unprovable and would fail on ordinary backpressure.
property p_eventually_drains;
  @(posedge clk) disable iff (!rst_n)
  (out_valid && out_ready[->1]) |-> ##[0:$] !out_valid or out_ready;
endproperty
// assume property (@(posedge clk) s_eventually out_ready);

P1 catches silent loss or duplication inside the pipeline. Without it, an item dropped at stage 1 appears as a transaction that never arrives — a symptom observed at the far end of the system, many boundaries from its cause.

P2 catches the corruption class this module has referred to throughout: a held item changing produces a well-formed packet with mixed content, undetectable by anything downstream.

P4 is the one to be careful with. In-order delivery is a property of this illustrative pipeline, not a general PCIe guarantee — PCIe has its own ordering rules with specific requirements and specific exceptions, and those are Module 13. Asserting in-order behaviour on a design that is permitted to reorder would produce false failures, which is exactly the kind of mislabelled assertion that erodes trust in a suite.

P5 is shown deliberately as a caution. Liveness without an environment assumption is unprovable — the property cannot distinguish "permanently stuck" from "legitimately backpressured." The commented assume is what makes it meaningful, and omitting such assumptions is the most common way liveness assertions become noise.

10. End-to-End Observability

Layer-isolated checks prove each stage. What they cannot prove is that the composition works. The technique that closes the gap:

Error injection by layer, and the symptom each should produce:

• Transaction Layer — inject a malformed internal request or illegal metadata. Expect: rejection at classification, or a well-formed packet that arrives and does the wrong thing. Should not produce retries — if it does, something below is inspecting content it should not.
• Data Link Layer — inject a corruption indication, a stale resolution indication, a duplicate resolution. Expect: retries, and for the stale/duplicate cases, no state corruption. Watch for duplicated transactions at the destination, the signature of a guard failing.
• Physical Layer — drop, corrupt, or hold the transport representation; withdraw availability mid-packet. Expect: integrity failures and retries for corruption; backpressure propagating upward for holds; a reported loss rather than silent discard for withdrawal.

The value of injecting per layer is that it validates the symptom mapping — you learn what each fault class actually looks like at the observation points, so that when a real failure appears you can recognise it.

11. A Full Debug Walkthrough

Symptom: an Endpoint write never updates host memory.

Work the observation chain from Chapter 3.4, bisecting rather than walking.

Bisect first — did the far-side Physical Layer receive it?

No, nothing arrived. The problem is on the transmit side or in transport. Continue on the transmit half.

• Was the request accepted from the engine? If not, the Transaction Layer is stalled — follow ready downward (§6) to find which stage is refusing.
• Was a transaction emitted? If not, the fault is inside the Transaction Layer.
• Did the Data Link Layer accept it? If not, check retry storage occupancy — full storage means resolution is not happening, which is a different problem wearing this symptom.
• Did the Physical Layer transmit it? If it refused, check connection availability. If it accepted and nothing emerged, transport lost it.

Yes, it arrived. The transmit side worked. Continue on the receive half.

• Did the far-side Data Link Layer accept it upward? If not, integrity is failing — expect retries, and expect the transmitter to be resending. That points at transport quality on this hop.
• Did the far-side Transaction Layer pass it up? If not, it was rejected as illegal or misclassified.
• Did it reach the memory-side consumer? If not, receive-side classification sent it elsewhere — check whether another consumer saw something unexpected.
• Did memory update? If everything above is clean and memory did not change, the fault is beyond PCIe entirely: the local consumer or the memory path.

And one branch that catches people: if there is a Switch in the path, "the Endpoint's delivery succeeded" only proves the packet reached the Switch. Repeat the bisection for the second hop (§5). An engineer who treats the first hop's success as end-to-end confirmation will conclude the packet was delivered and look for the bug in memory, which is the wrong half of the system.

12. Common Misconceptions

13. Understanding Check

14. Summary — and the Close of Module 3

A transaction changes representation and responsibility as it moves. Meaning is created once at the source's Transaction Layer and interpreted once at the destination's. Between them, hop-local delivery state is attached, used, and discarded on each Link, and physical transport carries the information across each connection.

Across a Switch, the delivery relationship terminates and a new one begins with a different identity space and separate retry storage. Only meaning crosses unchanged — which is why a delivery confirmation is always a statement about one hop.

Backpressure propagates upward through every stage, each absorbing it until its own storage fills. The engine stalls last and reports first, putting the reported symptom furthest from the cause. Buffer depth changes latency, not outcome. A stall terminates at an external condition; a deadlock has a cycle in the wait-for relation and no bottom to fix.

Module 3 as a whole: 3.1 established what operation is being performed; 3.2 how a packet crosses one Link reliably; 3.3 how information crosses the connection; 3.4 which layer owns a behaviour and how to prove it; and this chapter how they compose.

Layer Responsibilities taught ownership. Layer Interactions taught composition. Ownership tells you where to look; composition tells you why the symptom appeared somewhere else.

Hold the model: same logical operation, different responsibility at every stage.

15. What Comes Next

Module 3 kept the topology deliberately simple — mostly two components and a Link, with one Switch introduced where it mattered. Real systems are not that simple.

Module 4 — PCIe Topology returns to structure with the layered model now in hand: how root-complex-side topology is actually organised, how endpoints and switches attach, how multi-level fabrics are built, and what real systems look like. Everything Module 2 established about paths and convergence becomes concrete, and everything this module established about hop-local scope becomes something you can locate on a real diagram.

Revisit Layer Responsibilities for the ownership method this chapter's walkthrough applies. Browse the full path on the PCIe tutorials index.