CXL Layered Architecture

Chapter 3.4 described a coherence conversation and said nothing about how the messages travelled. That omission was the point: coherence semantics and message transport are separable concerns, and keeping them apart is what made both tractable.

This chapter is that separation stated as architecture.

1. The Engineering Problem — Where Does This Bug Live?

A request is issued by a host and never completes.

Without layering, that sentence has no useful next step. The fault could be a coherence conflict, a malformed request, a dropped message, a bad CRC, a marginal lane, a failed equalisation, or a queue that filled and never drained. Seven candidates in four different engineering disciplines, and no principle for ordering them.

With layering, the same sentence becomes a search. Each layer has a contract with its neighbours, so the question "which contract was broken?" partitions the problem — and the partition is what turns an open-ended investigation into a bounded one.

That is the practical value of a layered architecture, and it is a larger value than the usual justification of modularity. Layering is a debugging tool before it is a design tool.

2. The One-Sentence Model

Each layer's job is defined by what it knows: the protocol layer knows what a request means and nothing about how it travels; the link layer knows whether a transfer arrived and nothing about what it meant; the physical layer knows whether the electrical transfer succeeded and nothing above that — and every bug is a responsibility that landed in a layer without the information to discharge it.

3. What This Chapter Owns

Chapter 2.4 established what is shared with PCIe. This chapter is about responsibility, which is a different question with a different answer.

4. The Mental Model — Layers as Information Boundaries

The usual picture is a stack of boxes. A more useful one is a stack of knowledge:

  ┌─────────────────────────────────────────────┐
  │ PROTOCOL / TRANSACTION                      │
  │ knows: what this request means               │
  │ never knows: how many attempts it took       │
  ├─────────────────────────────────────────────┤
  │ LINK                                        │
  │ knows: whether the transfer arrived intact   │
  │ never knows: what the transfer meant         │
  ├─────────────────────────────────────────────┤
  │ PHYSICAL                                    │
  │ knows: whether the bits made it              │
  │ never knows: that transfers exist            │
  └─────────────────────────────────────────────┘

Read the "never knows" lines as design rules. They are what make the layers independent, and each one is violated by a real defect in Section 15.

The rule generalises the placement test from Chapter 3.1: does the agent making this decision have the information the decision requires? Applied vertically instead of horizontally, it produces layering.

5. What Each Layer Owns

Protocol / transaction layer

Owns: what a request means. Which of the three CXL sub-protocols it belongs to; what address it targets; what coherence state is being requested; which transaction it is, so a response can be matched; what ordering the request requires.

Does not own: whether the message arrived, how many attempts it took, or what the signalling looked like.

The CXL Consortium's material describes this layer's defining property: the transaction layer comprises three dynamically multiplexed sub-protocols on a single link. That multiplexing — the decision about which class this traffic is and how the classes share the link — is protocol-layer work, and it is what Chapter 2.4's arbiter was.

Link layer

Owns: reliable delivery of a unit of transfer. Framing, integrity checking, flow control across the link, and retry when a transfer fails.

Does not own: the meaning of what it carried.

Consortium material describes CXL as dynamically multiplexed I/O, cache and memory traffic in flit format carried on the PCIe physical layer, and the flit-based structure with FEC and CRC that arrived with CXL 3.0 tracks the PCIe generation underneath. The link layer is where "did it arrive?" is answered, and answering it requires integrity information the protocol layer does not have and does not want.

Physical layer

Owns: getting bits across a channel. Signalling, lanes, equalisation, training, and the electrical properties of the link.

Does not own: anything about transfers or transactions.

Consortium material shows the CXL stack sitting on the PCIe PHY with a PCIe/CXL logical PHY above it — which is Chapter 2.4's reuse boundary drawn from the layering side. The physical layer is entirely inherited, and that is why a channel problem in a CXL system is a PCIe problem with PCIe answers.

6. The Contracts Between Layers

A layer boundary is not a line on a diagram. It is a set of promises, and naming them is what makes a violation detectable.

Four of those six are about flow control or exactly-once delivery, which is not a coincidence. Those are the two contracts a layered stack most often gets wrong, and Sections 8 and 10 measure both.

The one to underline: "exactly once". The link layer retries, so the same unit may traverse the wire more than once. If that multiplicity reaches the protocol layer, a transaction appears twice — and the protocol layer has no way to tell a duplicate from a second, legitimate request. Section 10 measures it.

7. CXL and PCIe, From the Layering Side

Chapter 2.4 said what is shared. Layering says why the boundary is where it is.

The boundary sits at the highest layer that can be reused without constraining semantics. One layer lower and CXL would have had to build its own framing for no benefit. One layer higher and it would have inherited PCIe's transaction semantics, which is exactly what Chapter 1.6 showed to be insufficient.

Two consequences worth carrying:

A CXL controller contains a full PCIe stack. Not a subset — a working one, because Chapter 2.4's mode FSM comes up as PCIe unconditionally. Layering makes that cheap to say and expensive to build.

Security lands at different layers. Consortium material notes that CXL.io uses PCIe IDE while CXL defines a similar capability for CXL.mem — the inherited layer brings its inherited protection, and the new semantics needed new protection. That is a clean example of the boundary being visible in a feature rather than in a diagram.

8. RTL 1 — The Boundary as a Queue

Purpose

Make the protocol-to-link boundary explicit, and make the backpressure contract the only channel between them.

// The boundary between the transaction layer and the link layer.
//
// A layer boundary is a CONTRACT: what crosses it, in what order, and what
// happens when the far side cannot accept. This FIFO is the contract made
// physical, and its `ready` is the only channel by which link-layer conditions
// reach the transaction engine.
//
// GENERIC teaching model. NOT a CXL flit, link-layer or queue structure.
module layer_fifo #(
  parameter int unsigned DEPTH = 4,
  parameter int unsigned W     = 24
) (
  input  logic         clk,
  input  logic         rst_n,
  input  logic         in_valid,
  input  logic [W-1:0] in_data,
  output logic         in_ready,
  input  logic         out_ready,
  output logic         out_valid,
  output logic [W-1:0] out_data,
  output logic [7:0]   level_q,
  output logic [7:0]   max_level_q,
  output logic [15:0]  bp_cycles_q,
  output logic         overflow_err,
  output logic         underflow_err
);
  assign in_ready  = (level_q < DEPTH[7:0]);
  assign out_valid = (level_q != 0);
  assign out_data  = mem_q[0];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* ... */ end
    else begin
      if (!in_ready) bp_cycles_q <= bp_cycles_q + 16'd1;
      // ... push / pop / simultaneous, tracking max_level_q ...
      if (level_q > DEPTH[7:0])                      overflow_err  <= 1'b1;
      if (out_ready && !out_valid && (level_q != 0))  underflow_err <= 1'b1;
    end
  end
endmodule

in_ready is doing something conceptually larger than it looks. It is the entire vocabulary the link layer has for talking to the protocol layer about its own condition. A congested link, a retrying link, a link whose far side is slow — all of them reach the transaction engine as the same single bit going low. That is deliberate: if the protocol layer could distinguish those cases, it would be making decisions with link-layer information, which is the coupling layering exists to prevent.

The two error outputs check opposite contract violations. overflow_err means the producer ignored in_ready; underflow_err means the consumer took something that was not offered. Both are contract breaks by a neighbour, which is why a layer checks its own boundaries rather than trusting them.

Simulation evidence

Seven pushes into a 4-deep boundary with the link side unable to accept:

=== EXP1: the layer boundary as a contract ===
  push 0: in_ready=1 level=1
  push 1: in_ready=1 level=2
  push 2: in_ready=1 level=3
  push 3: in_ready=0 level=4
  push 4: in_ready=0 level=4
  push 5: in_ready=0 level=4
 
  link side stalled: level=4 max=4 backpressure_cycles=3 in_ready=0
  -> in_ready is the ONLY path from a link condition to the layer above
  link side drains : level=0 in_ready=1

The level pins at 4 and nothing is lost, because the producer honoured in_ready. That is the whole contract working: the transaction layer knows nothing about why the link stopped, only that it must stop too.

bp_cycles_q = 3 is the number worth instrumenting. It converts "the link seems slow" into a measurement, and it belongs at the boundary rather than inside either layer — which is a general point about layered designs: the interesting performance data is at the interfaces, because that is where one layer's problem becomes another layer's.

9. RTL 2 — Metadata Across the Boundary

Purpose

Transaction identity has to survive the trip down and back. If it does not, everything above breaks in a way that looks like a protocol bug.

// Packing transaction metadata for the trip down the stack, and recovering it
// on the way up.
//
// The lesson is not the encoding -- it is that a layer boundary is where
// information is LOST if the packing and unpacking disagree. TRUNCATE=1 drops
// the top bit of the tag, which is invisible for small tags and destroys the
// association for large ones.
//
// GENERIC teaching model. This is NOT a CXL flit layout, header format or
// field encoding.
module meta_pack #(
  parameter bit TRUNCATE = 1'b0
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        in_valid,
  input  logic [4:0]  in_tag,
  input  logic [1:0]  in_class,
  input  logic [3:0]  in_len,
  input  logic [7:0]  in_payload,
  output logic [23:0] packed_out,
  // recovered on the way back up
  input  logic [23:0] packed_in,
  output logic [4:0]  out_tag,
  output logic [1:0]  out_class,
  output logic [3:0]  out_len,
  output logic [7:0]  out_payload,
  output logic        meta_lost_err
);
  logic [4:0] tag_field;
 
  // TRUNCATE=0: the full tag survives. TRUNCATE=1: the top bit is dropped and
  // silently reconstructed as zero.
  assign tag_field  = TRUNCATE ? {1'b0, in_tag[3:0]} : in_tag;
  assign packed_out = {tag_field, in_class, in_len, in_payload, 5'b0};
 
  assign out_tag     = packed_in[23:19];
  assign out_class   = packed_in[18:17];
  assign out_len     = packed_in[16:13];
  assign out_payload = packed_in[12:5];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) meta_lost_err <= 1'b0;
    // What went in is not what came out. The check has to compare, because
    // nothing about the packed word says a field was dropped.
    else if (in_valid && (tag_field != in_tag)) meta_lost_err <= 1'b1;
  end
endmodule

Nothing about a truncated field is detectable from the packed word. The word is well-formed, the correct width, and internally consistent. Only a comparison against what was handed in can see the loss — which is why meta_lost_err compares rather than validates.

Why the tag specifically. Class, length and payload are all content; the tag is identity, and identity is what allows a response to find its request. Corrupting content produces a wrong answer that something downstream may notice. Corrupting identity produces a correct answer delivered to the wrong requester, which is Chapter 3.2's wrong-slot failure with the corruption moved into the transport.

Simulation evidence

Two packers, identical inputs:

=== EXP2: metadata across the boundary ===
  tag 7  : faithful out_tag=7 | truncating out_tag=7
  tag 20 : faithful out_tag=20 | truncating out_tag=4  <-- association destroyed
           class=1 len=8 payload=a5 all survive both
 
  truncating instance: meta_lost_err=1

Tag 7 survives truncation and tag 20 does not, which is the structural problem: the defect is invisible for every tag below 16. A bring-up test with a handful of outstanding transactions will never see it, and the bug appears when concurrency rises enough for tags to exceed the surviving width — under load, in production.

Note that class, len and payload all survive both packers. Everything that looks like data is fine and only the identity is wrong, so the symptom is a response arriving at the wrong transaction rather than a corrupted value. That points debugging at the coherence logic, which is intact.

10. RTL 3 — Retry, and the Duplicate It Creates

Purpose

Retry belongs in the link layer. Its consequence must not escape it.

// A simplified link layer: send, wait for acknowledgement, retry on failure.
//
// The architectural point is RESPONSIBILITY PLACEMENT. Retry belongs here
// because the link layer knows a transfer failed and the transaction layer
// does not. But retry has a consequence the transaction layer must be
// protected from: the same transaction can appear on the wire twice.
// DEDUP=0 lets that duplicate reach the layer above.
//
// GENERIC teaching model. This is NOT the CXL retry algorithm, CRC scheme,
// sequence-number mechanism or any specification-defined link behaviour.
module link_retry #(
  parameter int unsigned MAX_RETRY = 3,
  parameter bit          DEDUP     = 1'b1
) (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       tx_valid,
  input  logic [4:0] tx_seq,
  output logic       tx_ready,
  input  logic       wire_ack,      // far side received it
  input  logic       wire_nak,      // far side rejected it (bad CRC, etc.)
  output logic       on_wire,
  output logic [4:0] on_wire_seq,
  output logic [7:0] retry_count_q,
  output logic       give_up,
  // upward delivery to the transaction layer
  output logic       deliver,
  output logic [4:0] deliver_seq,
  output logic       dup_delivered_err
);
  assign tx_ready = (state_q == IDLE);
  assign on_wire  = (state_q == SENT);
  assign give_up  = (retry_count_q >= MAX_RETRY[7:0]);
 
  // Delivery upward happens on acknowledgement. DEDUP=1 suppresses a repeat
  // of a sequence number already delivered.
  assign deliver = (state_q == SENT) && wire_ack &&
                   !(DEDUP && have_delivered_q && (last_delivered_q == seq_q));
 
  always_ff @(posedge clk or negedge rst_n) begin
    // ... IDLE: latch seq, clear retry, go SENT
    //     SENT: on ack, deliver upward once and return to IDLE;
    //           on nak, count a retry, or DONE if the limit is reached
  end
endmodule

give_up matters as much as the retry. A link layer that retries without bound turns a permanently failed transfer into a permanently stalled transaction, which surfaces as a hang rather than as an error. Escalating after a bounded number of attempts is what lets the protocol layer see a failure it can act on.

The layering argument for retry being here is the information argument again. The link layer knows the transfer failed — it computed the integrity check. The protocol layer knows only that it handed something down and has heard nothing back, which is indistinguishable from congestion. Only one of them can decide to retry.

Simulation evidence

=== EXP3: retry, and what the layer above must not see ===
  sent seq 11: on_wire=1 retry=0
  nak       : retry=1 on_wire=1
  nak       : retry=2
  ack       : deliver=1 seq=11

Two failed attempts, then success, and the transaction layer sees exactly one delivery. It never learns that three transmissions occurred, which is the contract working: attempt count is link-layer information.

Then the same sequence number is transmitted again and acknowledged — the shape produced when a retry happens because an acknowledgement was lost rather than because the transfer was:

  same seq re-acked:
    with dedup    : deliver=0
    without dedup : deliver=1  <-- the layer above sees it twice
 
  total deliveries of 2 transmissions of one sequence: dedup=1 no-dedup=2

One transaction, two deliveries. The protocol layer above the non-deduplicating link now has two apparently independent requests with the same identity, and it has no way to tell which is real — because from its position they are identical. Whatever it does next is wrong: allocating a second tracker leaks an entry, and matching to the existing one may retire a transaction that is still in flight.

This is the "exactly once" contract from Section 6, and it is broken by a mechanism that is otherwise entirely correct.

11. RTL 4 — Which Layer Handles Which Error

Purpose

An error should be handled by the layer that can do something about it, and escalated only when it cannot.

// Which layer should handle which error?
//
// The rule: an error is handled by the layer that can DO something about it,
// and reported upward only when it cannot be corrected. Getting this wrong in
// either direction is a real defect:
//   -- a correctable link error escalated to the transaction layer causes a
//      transaction to be abandoned that would have succeeded on retry;
//   -- an uncorrectable error absorbed at the link layer silently loses data.
//
// GENERIC teaching model; the error taxonomy here is a teaching abstraction.
module layer_error_route #(
  parameter bit ESCALATE_ALL = 1'b0     // 1 = the bug shape
) (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       err_valid,
  input  logic [1:0] err_kind,   // 0=phy transient, 1=link CRC, 2=link fatal, 3=protocol
  output logic       handle_phy,
  output logic       handle_link,
  output logic       escalate_to_txn,
  output logic       misrouted_err
);
  logic correctable_below;
 
  // Kinds 0 and 1 are recoverable below the transaction layer.
  assign correctable_below = (err_kind == 2'd0) || (err_kind == 2'd1);
 
  assign handle_phy      = err_valid && (err_kind == 2'd0) && !ESCALATE_ALL;
  assign handle_link     = err_valid && (err_kind == 2'd1) && !ESCALATE_ALL;
  assign escalate_to_txn = err_valid && (ESCALATE_ALL || !correctable_below);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) misrouted_err <= 1'b0;
    // Escalated something a lower layer could have fixed.
    else if (escalate_to_txn && correctable_below) misrouted_err <= 1'b1;
  end
endmodule

The rule is symmetric and both directions are defects. Escalating too eagerly abandons transactions that would have succeeded; absorbing too eagerly loses data silently. misrouted_err catches only the first direction — the second requires knowing that an error was uncorrectable, which is information the absorbing layer by definition did not have.

That asymmetry in detectability is worth noticing. Over-escalation is visible from below; over-absorption is visible only from above, and often only as missing data. It is a general reason to prefer escalating when uncertain.

Simulation evidence

=== EXP4: which layer handles which error ===
  phy transient       correct: phy=1 link=0 escalate=0 | escalate-all: escalate=1
  link CRC            correct: phy=0 link=1 escalate=0 | escalate-all: escalate=1
  link fatal          correct: phy=0 link=0 escalate=1 | escalate-all: escalate=1
  protocol violation  correct: phy=0 link=0 escalate=1 | escalate-all: escalate=1
 
  escalate-all instance: misrouted_err=1

Rows three and four are identical between the designs, which is the familiar shape: the escalate-all variant is correct for every error that genuinely needs escalating, and wrong for the two that do not. On a clean link, most of what it sees is nothing — so the defect appears as unexplained transaction failures during marginal-channel operation, precisely when the system should be riding through.

12. Assertions

Icarus does not execute concurrent SVA, so these were not run; the table gives the procedural check.

// L1 — the boundary queue is bounded and honours its own ready.
a_boundary_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  (level_q <= DEPTH) && (!in_ready |-> !(in_valid && in_ready)));
 
// L2 — nothing is taken that was not offered.
a_no_underflow: assert property (@(posedge clk) disable iff (!rst_n)
  (out_ready && out_valid) |-> (level_q != 0));
 
// L3 — metadata survives the boundary unchanged. This is a COMPARISON, not a
// validity check: a truncated field produces a perfectly well-formed word.
a_meta_preserved: assert property (@(posedge clk) disable iff (!rst_n)
  in_valid |-> (out_tag == in_tag) && (out_class == in_class));
 
// L4 — EXACTLY ONCE: one transaction is delivered upward at most once,
// however many times the link transmitted it.
a_deliver_once: assert property (@(posedge clk) disable iff (!rst_n)
  (deliver && have_delivered_q) |-> (deliver_seq != last_delivered_q));
 
// L5 — retry is bounded; a permanently failing transfer escalates.
a_retry_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  (retry_count_q >= MAX_RETRY) |-> ##[1:2] (give_up || tx_ready));
 
// L6 — an error is never escalated past a layer that could correct it.
a_no_misroute: assert property (@(posedge clk) disable iff (!rst_n)
  escalate_to_txn |-> !correctable_below);
 
// L7 — an error is always handled somewhere. Absorbing silently is the
// opposite defect and this is what catches it.
a_error_handled: assert property (@(posedge clk) disable iff (!rst_n)
  err_valid |-> (handle_phy || handle_link || escalate_to_txn));
 
// L8 — LIVENESS: a full boundary eventually drains once the consumer is ready.
a_boundary_drains: assert property (@(posedge clk) disable iff (!rst_n)
  (level_q == DEPTH && out_ready) |-> ##[1:DEPTH] (level_q < DEPTH));

L3 is a comparison and not a validity check, and that distinction recurs across this track. A truncated field yields a well-formed word; only checking it against what was handed in reveals the loss. The same shape appears in Chapter 3.1's address-map overlap and Chapter 3.3's permission check: the output was made well-formed by logic that ran after the mistake.

L4 is the layering property. It says something about what the link layer may expose, not about whether the link layer works — and it is the only property in the table that a correct-in-isolation link layer can fail.

13. Quantitative Reasoning — Boundary Depth

Layer boundaries cost storage, and the amount follows from the rate mismatch they absorb.

If the protocol layer produces at R_p units per cycle and the link layer drains at R_l, a boundary of depth D tolerates a burst before backpressure for:

cycles_of_tolerance  =  D / (R_p − R_l)      while R_p > R_l

The measured run pushed one per cycle into a boundary draining zero, so tolerance was 4 / 1 = 4 cycles — and the trace shows in_ready falling on push 3, exactly as predicted.

Retry changes the arithmetic in a way that is easy to miss. A link that retries occupies the wire for the retried unit as well as the original, so its effective drain rate is lower than its nominal one:

R_l_effective  =  R_l / (1 + expected_retries)

At a 10% retry rate the effective drain is about 0.91 of nominal, so a boundary sized against nominal rate will backpressure under a perfectly healthy link that is simply working slightly harder. That is why bp_cycles_q belongs at the boundary: it distinguishes "the protocol layer is producing too fast" from "the link is spending capacity on retries", and those have different fixes.

14. Debugging by Layer

The practical payoff. A symptom, and the layer to interrogate first:

The discipline is to interrogate the boundaries before the blocks. A request that never completes is visible at the boundary as a unit that went down and nothing came back — one measurement that eliminates the entire protocol layer from suspicion, or confirms it, before anyone opens a coherence trace.

15. Debug Lab

// Pack the transaction into the transfer.
assign packed_out = {in_tag[3:0], in_class, in_len, in_payload, 6'b0};

  tag 7  : faithful out_tag=7 | truncating out_tag=7
  tag 20 : faithful out_tag=20 | truncating out_tag=4  <-- association destroyed
           class=1 len=8 payload=a5 all survive both

assign packed_out = {in_tag, in_class, in_len, in_payload, 5'b0};   // full width
a_meta_preserved: assert property (in_valid |-> (out_tag == in_tag));

// A transfer was acknowledged; hand it up.
assign deliver = (state_q == SENT) && wire_ack;

  same seq re-acked:
    with dedup    : deliver=0
    without dedup : deliver=1  <-- the layer above sees it twice
 
  total deliveries of 2 transmissions of one sequence: dedup=1 no-dedup=2

assign deliver = (state_q == SENT) && wire_ack &&
                 !(have_delivered_q && (last_delivered_q == seq_q));

// Report every error to the transaction layer so nothing is hidden.
assign escalate_to_txn = err_valid;

  phy transient       correct: phy=1 link=0 escalate=0 | escalate-all: escalate=1
  link CRC            correct: phy=0 link=1 escalate=0 | escalate-all: escalate=1
  link fatal          correct: phy=0 link=0 escalate=1 | escalate-all: escalate=1
  protocol violation  correct: phy=0 link=0 escalate=1 | escalate-all: escalate=1
 
  escalate-all instance: misrouted_err=1

assign correctable_below = (err_kind == PHY_TRANSIENT) || (err_kind == LINK_CRC);
assign escalate_to_txn   = err_valid && !correctable_below;

// The transaction layer has work; hand it down.
always_ff @(posedge clk) if (txn_has_work) begin
  boundary_in_valid <= 1'b1;         // in_ready never consulted
  boundary_in_data  <= txn_unit;
end

  push 3: in_ready=0 level=4
  push 4: in_ready=0 level=4
  push 5: in_ready=0 level=4
 
  link side stalled: level=4 max=4 backpressure_cycles=3 in_ready=0

assign boundary_in_valid = txn_has_work;
// the unit is only handed over when BOTH sides agree
assign accepted = boundary_in_valid && boundary_in_ready;

// Infer the class at the link layer from the transfer's size.
assign class_field = (len > 8) ? CLASS_MEM : CLASS_IO;

  tag 20 : faithful out_tag=20 | truncating out_tag=4
           class=1 len=8 payload=a5 all survive both

// protocol layer, which knows what the request IS
assign unit_class = coherent_req ? CLASS_CACHE :
                    hostmem_req  ? CLASS_MEM   : CLASS_IO;
// link layer carries it and never interprets it
assign packed_out = {tag, unit_class, len, payload, 5'b0};

16. Design Review — Reading a Layered Stack

On boundaries. Is every boundary an explicit handshake with a bounded queue? Does the producer treat ready as a condition or advice? Does each layer assert on its own interfaces, or trust its neighbours? Are max_level and backpressure_cycles instrumented at the boundary rather than inside the layers?

On metadata. Is every field the same width on both sides? Is there a comparison of what came back against what went down, or only a validity check? Does any layer derive a field it was handed?

On retry. Is retry bounded, and does exceeding the bound escalate rather than spin? Does the link layer deduplicate, or can a retried unit reach the protocol layer twice? Is the regression injecting lost acknowledgements as well as lost transfers?

On errors. Is there an explicit correctable-versus-not classification? Are both failure directions asserted — nothing escalated that could be fixed below, nothing absorbed that could not? Does the classification live in one place, or is it implicit in scattered conditions?

And the layering question. For each decision in the design, which layer makes it, and does that layer have the information? A decision made a layer too low is a guess; a decision made a layer too high is a coupling. Both work until the traffic changes.

17. How This Appears in Real Engineering

Architect

Layering is the interface-definition exercise. The contracts in Section 6 are what the teams building each layer agree to, and the ones about flow control and exactly-once delivery are the two that get written down last and cause the most integration pain.

RTL engineer

Four disciplines, all measured. Make every boundary an explicit bounded handshake. Size metadata fields identically on both sides and compare the round trip. Deduplicate at the layer that creates duplicates. Classify errors by whether the layer below can correct them.

Verification engineer

The stimulus that finds these defects is boundary-focused: a stalled consumer, tags exceeding every field width, a lost acknowledgement rather than a lost transfer, and each error kind driven individually. Note that all four defects are invisible on a healthy, lightly-loaded link — which is the default bench configuration.

Performance engineer

The interesting counters live at the boundaries, because that is where one layer's problem becomes another's. And remember retry lowers the effective drain rate: R_l / (1 + expected_retries), so a healthy link working harder looks like a producer that is too fast unless bp_cycles_q is there to distinguish them.

Silicon debug

Interrogate boundaries before blocks. "A unit went down and nothing came back" is one measurement that partitions the whole stack, and it is cheaper than any trace. Section 14's table is the triage order.

Firmware and system software

Layer attribution in error reporting is what makes a support case tractable. An error reported as "transaction failed" when the cause was a marginal channel sends the investigation to the wrong team; the layer that detected it should be part of the report.

18. Common Misconceptions

19. Interview Reasoning

20. Exercises

1. Size the boundary. The protocol layer produces 1 unit per cycle; the link drains 1 unit every 2 cycles nominally, with a 15% retry rate. Compute the effective drain rate and the depth needed to absorb a 20-unit burst without backpressure. Then decide whether that depth is worth paying for.

2. Find the third contract break. Section 6 lists six promises. Section 15 breaks four of them. Identify the two not broken by any Debug Lab, construct a defect for one, and write the property that catches it.

3. Widen the tag. The truncation defect appeared at tag 16. If the field were 5 bits and the tag 6, at what concurrency does it appear? Generalise, and state what that implies about the minimum outstanding-transaction count a regression must reach.

4. Classify a new error. A retimer reports a recoverable framing error. Which layer handles it, and which of the two failure directions is the risk if you get it wrong? Write both properties.

5. Argue the boundary. Make the case for placing CXL's reuse boundary one layer lower than it is — CXL defining its own link layer on PCIe's PHY. What would it gain, what would it cost, and which of Chapter 2.4's arguments does it violate?

21. Summary

Layering is about information, not modularity. The protocol layer knows what a request means and must not know how many attempts it took; the link layer knows whether a transfer arrived and must not know what it meant; the physical layer knows whether the bits made it and knows nothing above that. Every defect in this chapter is a decision made by a layer without the information to make it.

Boundaries are contracts, and four of the six are about flow control or exactly-once delivery. Measured, a 4-deep boundary with a stalled consumer pinned at 4, reported 3 backpressure cycles, and lost nothing — because the producer honoured ready, which is the entire vocabulary the lower layer has for its own condition.

Metadata must survive, and a truncated field is not malformed. A five-bit tag in a four-bit field delivered tag 20 as tag 4 while class, length and payload all arrived intact — so the symptom is a correct response at the wrong transaction, which sends debugging to coherence logic that is working perfectly. Only a comparison against what went down reveals it, and the defect is invisible below 16 outstanding transactions.

Retry belongs at the link layer, and so does its consequence. One sequence number transmitted twice and acknowledged twice produced two deliveries without deduplication. The protocol layer cannot fix this, because from its position a duplicate and a second legitimate request are identical — the layer that created the multiplicity must absorb it.

Errors go to the layer that can act. A router escalating everything was correct for the two error kinds that genuinely needed escalating and wrong for the two that were correctable — and the result is worse reliability, because the transaction layer's only response to a correctable error is to give up.

The habit worth keeping is the debugging one. Interrogate boundaries before blocks. "A unit went down and nothing came back" is a single measurement that partitions the entire stack, and it is cheaper than any trace — which is the practical reason layering earns its keep long before anyone reuses a layer.

22. What Comes Next

Five chapters have now built the pieces: what the host owns, what the device owns, what the fabric does, how coherence is coordinated, and which layer owns which responsibility.

Chapter 3.6 assembles them. One request, traced from a CPU core through decode, the host bridge, the link, a fabric, into a device and back — with the latency decomposed at every stage, the places it can stall enumerated, and the counters that tell you which one actually happened.

For adjacent material: Relationship to PCIe has the reuse boundary this chapter reads from the layering side, Coherent Communication Model has the semantics the protocol layer carries, and The CXL Fabric has the queueing this chapter's backpressure contract composes with. The path is on the CXL tutorials index.