The Protocol Layer

Module 4 established the stack and followed a payload across it. Module 5 goes inside each layer and asks what actually has to be built. This chapter takes the top layer, and it exists to correct a specific and damaging simplification: that the Protocol Layer is "where PCIe or CXL sits", as if naming the protocol were the same as describing the hardware.

It is not. The Protocol Layer owns state with a longer lifetime than anything below it, and it owns the only correctness property the layers below cannot check. A link can be electrically flawless, pass every integrity check, and deliver every transport unit in order — and the system can still be wrong, because correctness at this layer is about meaning, and meaning is invisible to everything underneath.

1. The Layer That Owns Meaning

The Protocol Layer hosts the upper-level protocol and is responsible for generating and terminating that protocol's packets. Depending on configuration it may implement the corresponding protocol's transaction layer and that protocol's own flow-control scheme.

What it owns:

• Transaction semantics — what a request is, what a response means, which completion belongs to which request.
• Ordering rules the protocol requires, and the state needed to enforce them.
• Protocol-visible state — outstanding transactions, identifiers, expected completions.

What it must deliberately not know: lane health, lane mapping, bump placement, training progress, electrical behaviour, or any raw PHY state. Chapter 4.2 argued this structurally; this chapter shows what it costs when violated.

The Protocol Layer owns meaning, not movement.

2. What Rides Above, and What Changes

UCIe carries established protocols and a streaming option, and the choice changes what this layer does:

• PCIe — transaction-oriented I/O semantics: requests, completions, and the ordering rules that come with them.
• CXL — coherent and memory-oriented semantics where the system uses them, with correspondingly stronger ordering and state obligations.
• Streaming / raw — a protocol-defined payload carried without those upper-protocol interpretations, for traffic that suits neither.

The architecturally significant point is the responsibility shift first met in Chapter 4.2. In the streaming interface with the raw format, the Adapter's CRC and retry mechanisms are bypassed — which means error protection becomes the Protocol Layer's responsibility. So "what does the Protocol Layer do?" genuinely depends on the configured mode:

The layers are fixed. The responsibility boundary between the top two is mode-dependent — and an engineer who memorised one row will be wrong for the other.

3. State That Outlives the Transfer

Here is the property that most distinguishes this layer, and it follows directly from Chapter 4.3's observation about state lifetimes.

When the Protocol Layer hands a request to the Adapter, the transaction is not finished — it has barely started. The request must travel, be processed remotely, and produce a completion that comes back. Until then the Protocol Layer must remember: what was issued, under what identifier, what response is expected, and what ordering constraints still apply.

So this layer holds state that persists long after the payload has physically left the die. Contrast the lifetimes:

• PHY state ends when the bits are gone.
• Adapter transport state ends when delivery is confirmed (or, with retry enabled, when it is safe to discard the replay copy).
• Protocol transaction state persists until the semantic exchange completes — potentially far longer than any of the above.

That difference is exactly why these cannot be merged into one layer, and it is the reason a Protocol Layer needs real storage rather than a pass-through path.

4. Holding a Request Until It Is Accepted

Start with the simplest piece of state, because it makes the ownership rule concrete.

Illustrative architecture RTL — not UCIe normative signal naming.

// Protocol-side request holding register. The semantic request belongs to
// the Protocol Layer until the boundary handshake completes.
logic [REQ_W-1:0] req_q;
logic             req_valid_q;
 
always_ff @(posedge clk) begin
  if (!rst_n) begin
    req_valid_q <= 1'b0;
    req_q       <= '0;
  end else if (req_valid_q && !fdi_ready) begin
    req_valid_q <= 1'b1;      // stalled: hold the request unchanged
    req_q       <= req_q;
  end else if (new_request) begin
    req_valid_q <= 1'b1;
    req_q       <= new_request_data;
  end else begin
    req_valid_q <= 1'b0;
  end
end

Architecture. The Protocol Layer can generate traffic faster than the layer below accepts it. Something must hold a generated request until it is taken.

State. req_valid_q is ownership of an un-handed-down request; req_q is the request itself.

Cycle behaviour. While stalled, both hold. On acceptance, the register either reloads with a new request or clears.

Contract. The Adapter relies on the payload being stable while it stalls; the Protocol Layer relies on the Adapter retaining what it accepts.

Failure/DV. Advance req_q while stalled and the Adapter captures a different request than the one presented — a semantic corruption that the transport layers will faithfully deliver. This is the boundary invariant of Module 4, and it is only the first of the layer's obligations.

5. Tags: Transaction Identity

Now the state that makes this layer distinctive. A protocol that allows multiple outstanding transactions needs to match each completion to its request, which requires an identifier — a tag — and a table recording what is outstanding.

// Conceptual transaction tag allocator and outstanding record.
parameter int unsigned NUM_TAGS = 16;
 
logic [NUM_TAGS-1:0]        tag_in_use_q;
logic [$clog2(NUM_TAGS)-1:0] alloc_tag;
logic                        tag_available;
 
// Lowest free tag. Real designs often use a priority encoder or free-list.
always_comb begin
  tag_available = (tag_in_use_q != '1);
  alloc_tag     = '0;
  for (int i = NUM_TAGS-1; i >= 0; i--) begin
    if (!tag_in_use_q[i]) alloc_tag = i[$clog2(NUM_TAGS)-1:0];
  end
end
 
always_ff @(posedge clk) begin
  if (!rst_n) begin
    tag_in_use_q <= '0;
  end else begin
    // Claim on issue; release only when the completion is matched.
    if (issue_fire)      tag_in_use_q[alloc_tag]  <= 1'b1;
    if (completion_fire) tag_in_use_q[cmpl_tag]   <= 1'b0;
  end
end

Architecture. Tag allocation is transaction identity management. Without it, a returning completion cannot be attributed to the request that caused it.

State. tag_in_use_q is a bit per tag — the compact form of the outstanding-transaction table. A real design also stores per-tag context (address, length, requester), which is more state with the same lifetime.

Cycle behaviour. A tag is claimed on the cycle a request is issued and released on the cycle its completion is matched. Between those two events — potentially hundreds or thousands of cycles, spanning the entire round trip — the tag is unavailable.

Contract. The remote endpoint will echo this tag back in the completion. That is a protocol-level agreement, invisible to the Adapter and PHY, which treat the tag as ordinary payload bits.

Failure/DV. Reuse a tag while its first transaction is still outstanding and the returning completions become ambiguous: completion for transaction A arrives, is matched against the entry now describing transaction B, and B's requester receives A's data. Nothing below this layer can detect it — the transport was perfect. Hence:

// Illustrative safety property: never issue with a tag already outstanding.
property p_no_tag_reuse;
  @(posedge clk) disable iff (!rst_n)
    issue_fire |-> !tag_in_use_q[alloc_tag];
endproperty
assert property (p_no_tag_reuse)
  else $error("issued a transaction using a tag that is still outstanding");
 
// Illustrative safety property: a completion must match a live transaction.
property p_completion_matches_outstanding;
  @(posedge clk) disable iff (!rst_n)
    completion_fire |-> tag_in_use_q[cmpl_tag];
endproperty
assert property (p_completion_matches_outstanding)
  else $error("completion received for a tag that is not outstanding");

The second property is the one people forget, and it catches a different class of bug: a spurious, duplicated, or stale completion arriving for a transaction that already retired.

6. Ordering Is a Protocol Obligation

Take two requests, A then B, where the protocol requires that their effects become visible in that order.

The link may deliver both perfectly. It may even deliver them in order. That is not sufficient, because ordering is a property of when effects become observable, which depends on how the endpoint issues and retires them — and that is protocol logic.

The critical architectural rule: the Adapter must not invent ordering policy. It does not know which transactions are related, which may pass which, or what the protocol promises. If it reordered on its own initiative it could break a guarantee it cannot see; if it enforced ordering the protocol did not require, it would serialise traffic unnecessarily. Ordering is specified by the protocol and enforced by the layer that implements it.

// Conceptual ordering interlock: a strongly-ordered request must not be
// issued while an earlier one it must not pass is still outstanding.
logic ordered_req_pending_q;
 
always_ff @(posedge clk) begin
  if (!rst_n) begin
    ordered_req_pending_q <= 1'b0;
  end else if (issue_fire && issue_is_ordered) begin
    ordered_req_pending_q <= 1'b1;
  end else if (completion_fire && cmpl_is_ordered) begin
    ordered_req_pending_q <= 1'b0;
  end
end
 
// Gate issue of a new ordered request on the previous one completing.
assign may_issue_ordered = !ordered_req_pending_q;

Architecture. A deliberately minimal interlock — real PCIe or CXL ordering is far richer, and belongs to the chapters that teach those mappings. What it captures is the shape: ordering requires state about what is still in flight, and that state gates issue.

Cycle behaviour. ordered_req_pending_q sets when an ordered request is issued and clears when its completion returns. Between those cycles, further ordered issue is blocked.

Failure/DV. Drop the interlock and two ordered requests can be outstanding simultaneously; their completions may return in either order, and the effects become visible out of order. The symptom is intermittent and load-dependent — it appears only when timing allows overlap, which is exactly why it survives casual testing and fails in the field.

7. A Perfect Link Carrying Wrong Traffic

Put the pieces together into the failure this chapter exists to teach.

Consider a system where:

• The PHY has trained; all configured lanes are healthy; no electrical errors.
• The Adapter reports no integrity failures; every transport unit passes its check; no retries occur.
• Every transport unit is delivered exactly once, in order.

And the system produces wrong results, because a tag was reused: a completion carrying data for transaction A was matched to the table entry now describing transaction B, and B's requester consumed A's data as its own.

Every measurement below the Protocol Layer says the link is perfect — and every one of them is correct. The link is perfect. It faithfully transported a logically wrong transaction.

Protocol correctness sits logically above link correctness. A flawless link will deliver a wrong transaction with complete fidelity.

The debugging consequence is worth internalising: when a system produces wrong data but the link reports no errors, the link is not exonerated — it is simply not the layer being accused. Error counters at zero is evidence about transport, not about semantics.

8. What Must Not Leak In

The mirror of the above. Here is the anti-pattern in its Protocol-Layer form:

// BAD: protocol issue decision derived from raw physical lane state.
always_comb begin
  issue_request = have_work && lane_good[3];
end

It couples protocol behaviour to a specific lane on a specific PHY. Change the link width, change module organisation, or enable lane repair that transparently remaps lanes, and protocol logic breaks. It also destroys the ability to verify the Protocol Layer against an abstract model of the layer below (§9), because that model would have to expose lane detail.

The correct dependency is on abstractions the layers below provide:

// Protocol consumes abstracted status and negotiated capability only.
assign issue_request = have_work
                    && link_operational      // abstract: usable or not
                    && may_issue_ordered     // protocol's own interlock
                    && tag_available;        // protocol's own resource

Notice that two of the three gating terms are this layer's own state. That is the signature of a correctly layered Protocol implementation: it reasons about its own resources and consumes exactly one abstracted fact from below.

9. Verifying This Layer

Protocol verification is a scoreboard problem, not a waveform problem, and the distinction matters.

A protocol scoreboard tracks each issued transaction with its tag and context, the completion expected for it, ordering constraints in force, and the retirement of each entry. Its central check is that every completion matches a live outstanding entry and that the data returned is the data that transaction asked for.

Assertions cover the invariants that must hold every cycle: no tag reuse while outstanding, no completion for a non-outstanding tag, no request retired before the boundary handshake, and ordering interlocks respected. These are all safety properties — they say something bad never happens. A useful liveness companion, checked with a bounded window rather than an unbounded one, is that an outstanding transaction eventually retires while the link is operational and progress is being made.

Coverage should reach the states where bugs live: maximum outstanding transactions, backpressure while many are outstanding, completions returning in an order different from issue where the protocol permits it, tag wrap-around, the link going down with transactions outstanding, and recovery afterwards.

The model to carry: the Adapter and PHY environments can prove transport is faithful. Only a protocol scoreboard can prove the traffic was right.

10. Common Misconceptions

11. Understanding Check

12. Summary and What Comes Next

The Protocol Layer hosts the upper protocol and generates and terminates its packets, owning transaction semantics, ordering rules, and protocol-visible state — and deliberately knowing nothing about lanes, training, or signalling. Its defining property is state lifetime: a transaction remains outstanding long after its payload has physically left the die, which is why this layer needs a request queue, a tag allocator, an outstanding table, and ordering interlocks rather than a pass-through path.

Two consequences dominate. Tags are semantic state — identity, not transport — and reusing one while outstanding misattributes completions with every error counter reading zero, which is the sharpest demonstration that protocol correctness sits above link correctness. And ordering is the protocol's obligation, because the Adapter cannot see which transactions are related or what the protocol promises.

The mode nuance matters: with the streaming interface in raw format, the Adapter's CRC and retry are bypassed and error protection moves up to this layer, so its responsibilities are configuration-dependent even though the layer boundaries are not.

The Protocol Layer knows what the traffic means. Something below must now turn that traffic into reliable link transport while hiding every physical detail — and that layer turns out to hold more state than either of its neighbours:

• 5.2 — The Adapter Layer — buffering, flow control as resource accounting, integrity and retry state, link-management and negotiation state machines, and arbitration between protocols sharing a link.

Browse the full path on the UCIe tutorials index.