The CXL System View

Five chapters have built the pieces. This one assembles them, and the assembly is not a summary — it is where a set of individually correct components becomes a system with properties none of them has alone.

The question this chapter answers is the one a real engineer actually asks: a request went out and something is wrong. Where?

1. The Engineering Problem — "CXL Latency" Is Not a Number

Someone asks what the latency of a CXL memory access is.

There is no answer, and the reason is structural rather than evasive. The access traverses a CPU queue, a coherence check, an address decode, a host bridge, a link, possibly a switch, a device's ingress queue, a device's memory controller, its media, and then the entire return path. Each of those has its own latency, its own queue, and its own worst case, and the total is a sum of terms that vary independently.

Worse, the variance matters more than the mean. A path with a 200-cycle mean and a 250-cycle tail is a usable memory tier. A path with a 200-cycle mean and a 4,000-cycle tail is not, and no single number distinguishes them.

So the useful skill is not knowing a number. It is being able to decompose the number, and then to localise which term is responsible when it moves.

2. The One-Sentence Model

A CXL request is a journey through a series of stages, each holding state, each able to stall for its own reason, and each preserving one thing that must survive to the end — the transaction's identity — so end-to-end behaviour is the sum of per-stage terms, and every performance or correctness question reduces to which stage owns the term that changed.

3. What This Chapter Owns

Each earlier chapter is referenced rather than repeated. What is new here is the composition.

4. The Complete Picture

   CPU core
      │  issue, and the core's own queue
      ▼
   Caches
      │  hit? then none of the rest happens
      ▼
   Coherence / home
      │  serialise, snoop if needed, wait   ── Chapter 3.4
      ▼
   Address decode
      │  local DRAM, HDM, or MMIO           ── Chapter 3.1
      ▼
   CXL host bridge
      │  outstanding table, protocol class   ── Chapters 3.1, 3.5
      ▼
   CXL link
      │  framing, integrity, retry           ── Chapter 3.5
      ▼
   Switch / fabric  (optional)
      │  route, arbitrate, queue             ── Chapter 3.3
      ▼
   CXL device
      │  class dispatch, outstanding slot    ── Chapter 3.2
      ▼
   Accelerator / memory
         media latency, and the way back

Two things are worth noticing before any detail.

The path is asymmetric in cost: a cache hit skips everything below it, so the interesting latency distribution is bimodal and an average across all accesses describes neither mode.

The fabric stage is optional, and that is an architectural fact rather than a diagram convenience. A directly attached device has one fewer queueing point, one fewer routing table, and one fewer place to be starved — which is why Chapter 3.3's failures simply do not exist in a point-to-point system.

5. Walking One Host Read of Device-Attached Memory

Conceptually, without inventing any protocol sequence:

1. A core issues a load. It occupies an entry in the core's own load queue; that entry is not freed until the data returns.
2. The caches miss. Everything below now happens.
3. Coherence resolves. For a line in device-attached memory the host is the coherence authority (Chapter 3.4); if any agent holds a modified copy, that is where the value comes from and the device is never asked.
4. Decode selects the target. Chapter 3.1: this address belongs to host-managed device memory, so the request leaves the socket.
5. The host bridge allocates outstanding state. A tag, a record of the requester, the protocol class. If no slot is free, the request stalls here and never reaches the link.
6. The request crosses the layer boundary and the link. Chapter 3.5: framed, integrity-protected, retried if necessary — and the layers above never learn how many attempts it took.
7. The fabric routes it, if there is one. Chapter 3.3: a routing lookup, possibly arbitration against other hosts, possibly a queue.
8. The device accepts it. Chapter 3.2: class dispatch to the .mem engine, an outstanding slot allocated, a decoupling queue entered.
9. The device's memory controller services it. Media latency, which is the one term in this list that has nothing to do with CXL.
10. The response returns along every stage in reverse, and at each one its identity must still be intact.
11. The host matches the response to its outstanding entry and places the data.
12. The core's load queue entry retires.

Twelve steps, and eleven of them can stall. Only step 9 is a fixed physical cost; everything else is a queue, a table, or an agreement.

And the other direction

A device read of host memory reverses the roles but not the structure: the device allocates its own outstanding slot, the request crosses the link, the host's home agent resolves coherence — possibly snooping CPU caches (Chapter 3.4) — and the response returns to the device's slot.

The asymmetry from Chapter 3.4 shows up here as an extra term. Because the host orchestrates coherence, a device request that needs a snoop pays a round trip inside the host before its own round trip completes. That nesting is the structural reason device-side coherent access latency has a worse tail than host-side.

6. Latency Accounting

Make the decomposition explicit. Illustratively:

T_total  ≈  T_core_queue
         +  T_coherence           (0 if uncontended, large if snooping)
         +  T_decode
         +  T_host_outstanding    (0 if a slot is free, unbounded if not)
         +  T_link
         +  T_fabric_queue        (0 if no fabric)
         +  T_device_queue
         +  T_device_resource     (media)
         +  T_return              (≈ the sum of the transport terms again)

These are terms, not numbers. This chapter deliberately assigns no cycle counts to real CXL hardware, because doing so from memory is how inaccurate figures propagate. What the model gives you is the shape.

Three consequences follow, and they are the practical content:

Most terms are zero most of the time. No coherence conflict, a free outstanding slot, an empty fabric queue. That is why the median is much better than the worst case — and why a mean computed across a mixed workload describes neither.

Any term can dominate. A full outstanding table makes T_host_outstanding unbounded; a congested fabric port makes T_fabric_queue grow without limit until backpressure arrives. A single term going bad makes the media latency irrelevant, which is why "the memory is slow" is usually the wrong hypothesis.

The return path roughly doubles the transport terms. Anything that affects transport is paid twice, so a fabric hop is not one hop of cost.

Little's Law closes the loop

The measured run instrumented accumulated occupancy and a sample count, which is exactly what Little's Law needs:

mean in-flight  =  arrival rate × mean latency

Rearranged, mean latency = mean in-flight / throughput — so occupancy and throughput counters give you latency without timestamping a single request. The measured run reported accumulated occupancy 225 over 127 samples, a mean of 1.77 requests in flight, which is the kind of number a design can be held to and a trace cannot cheaply provide.

7. RTL 1 — The End-to-End Pipeline

Purpose

Make the whole path one module, so a stall anywhere is observable everywhere.

// One request, all the way through: host ingress -> route -> link -> device
// service -> response -> host retire.
//
// Every stage is a valid/ready pair with its own capacity, which is the point:
// a request can stall at any of them, and each stall has a different cause and
// a different fix.
//
// GENERIC integrative teaching model. NOT production CXL RTL.
module e2e_pipeline #(
  parameter int unsigned NTAG = 8
) (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       req_valid,
  input  logic [1:0] req_dest,
  output logic       req_ready,
  output logic       cmpl_valid,
  output logic [4:0] cmpl_tag,
  output logic [1:0] cmpl_dest,
  // per-stage readiness, driven externally to model congestion
  input  logic       link_ready,
  input  logic       fabric_ready,
  input  logic       device_ready,
  input  logic       resource_done,
  output logic [4:0] stage_tag  [4:0],
  output logic [4:0] stage_busy,
  output logic       tag_corrupt_err,
  output logic       stage_overrun_err
);
  // Stage indices: 0 host, 1 link, 2 fabric, 3 device, 4 return
 
  // A stage may advance only if the next stage is free AND its gate is open.
  // Written back-to-front so a FULL pipeline still advances in one cycle --
  // computing these forwards would make every stage wait a cycle for the one
  // ahead, which is a classic and invisible throughput bug.
  assign adv4 = busy_q[4];
  assign adv3 = busy_q[3] && resource_done && (!busy_q[4] || adv4);
  assign adv2 = busy_q[2] && device_ready  && (!busy_q[3] || adv3);
  assign adv1 = busy_q[1] && fabric_ready  && (!busy_q[2] || adv2);
  assign adv0 = busy_q[0] && link_ready    && (!busy_q[1] || adv1);
 
  assign req_ready  = !busy_q[0] || adv0;
  assign cmpl_valid = adv4;
  assign cmpl_tag   = tag_q[4];
 
  // ... shift forward, accept at stage 0, assign the next tag ...
endmodule

The backwards-computed advance chain is the load-bearing structure. Each stage's ability to move depends on the one ahead of it also moving this cycle. Computed forwards, a full pipeline would advance one stage per cycle instead of all of them, which halves throughput and looks like a slow link. Note the timing cost honestly: this is a combinational chain across five stages, so it is a real critical path and a deeper pipeline would need to break it — usually with skid buffers, which trade a cycle of latency for a shorter path.

req_ready = !busy_q[0] || adv0 is the backpressure edge that reaches the CPU. Everything downstream ultimately expresses itself here, which is why a core sees "memory is slow" for causes that have nothing to do with memory.

Simulation evidence — the unloaded case

=== EXP1: one request, every stage clear ===
    cyc 0: busy=00001  cmpl=0 tag=0
    cyc 1: busy=00010  cmpl=0 tag=0
    cyc 2: busy=00100  cmpl=0 tag=0
    cyc 3: busy=01000  cmpl=0 tag=0
    cyc 4: busy=10000  cmpl=1 tag=1
    cyc 5: busy=00000  cmpl=0 tag=1
  -> 5 stages, so an unloaded request takes 5 hops

One request walking cleanly through, one stage per cycle, and the tag arriving as 1 — the value assigned at ingress. That identity survival is the correctness property; the five cycles are the latency floor.

8. RTL 2 — The Watchdog That Names a Stage

Purpose

A timeout that says "stuck" is nearly useless. One that says where is a diagnosis.

// How old is each outstanding request, and which stage was it in when it
// exceeded the limit?
//
// A timeout alone says "something is stuck". A timeout that names the STAGE
// says where to look, which is the difference between an investigation and a
// guess. Real systems have richer error mechanisms; this is the abstraction.
//
// GENERIC teaching model.
module e2e_watchdog #(
  parameter int unsigned LIMIT = 20
) (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       start,
  input  logic [4:0] start_tag,
  input  logic       done,
  input  logic [4:0] done_tag,
  input  logic [4:0] stage_busy,
  output logic [7:0] age_q,
  output logic       timeout,
  output logic [2:0] blame_stage,     // 0 host 1 link 2 fabric 3 device 4 return
  output logic       timeout_latched_q
);
  assign timeout = tracking_q && (age_q >= LIMIT[7:0]);
 
  // Blame the LAST stage that still holds anything -- the furthest point the
  // request reached is the most useful pointer.
  always_comb begin
    blame_stage = 3'd0;
    for (i = 0; i < 5; i = i + 1) if (stage_busy[i]) blame_stage = i[2:0];
  end
  // ... age accumulation, cleared when the tracked tag completes ...
endmodule

Simulation evidence

A request that never completes, with the device path held closed:

=== EXP4: a request that never completes ===
  age=24 timeout=1 latched=1 blame=FABRIC
  -> a bare timeout says 'stuck'; naming the stage says where

blame=FABRIC is the actionable half. The request reached the fabric stage and could not be handed to the device, so it is sitting there — and that single word eliminates the host, the link, the coherence logic and the media from suspicion.

Note the subtlety the trace reveals: the stalled gate was device_ready, and the blamed stage is the fabric. That is correct and it is the general rule — a request stalls in the stage that cannot hand it onward, so blame names the stage holding it and the cause is immediately downstream. Reading it as "the fabric is broken" would be wrong.

9. RTL 3 — Per-Stage Counters

Purpose

Localise a bottleneck without a trace.

// Per-stage instrumentation, so a system-level symptom can be localised to a
// stage without a trace.
//
// The counter that matters is STALL CYCLES PER STAGE. Total latency tells you
// the system is slow; per-stage stalls tell you which stage to fix.
//
// GENERIC teaching model.
module stage_counters (
  input  logic        clk,
  input  logic        rst_n,
  input  logic [4:0]  busy,        // stage holds a request
  input  logic [4:0]  advancing,   // stage will hand it on this cycle
  input  logic        accepted,
  input  logic        completed,
  output logic [15:0] n_accept_q,
  output logic [15:0] n_complete_q,
  output logic [15:0] stall_q     [4:0],
  output logic [15:0] max_stall_q [4:0],
  output logic [15:0] occupancy_sum_q,
  output logic [15:0] samples_q
);
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* ... */ end
    else begin
      if (accepted)  n_accept_q   <= n_accept_q   + 16'd1;
      if (completed) n_complete_q <= n_complete_q + 16'd1;
 
      // Little's Law needs BOTH: accumulated occupancy and a sample count.
      occupancy_sum_q <= occupancy_sum_q + {13'b0, occ};
      samples_q       <= samples_q + 16'd1;
 
      for (i = 0; i < 5; i = i + 1) begin
        // A stall is holding a request and NOT handing it on.
        if (busy[i] && !advancing[i]) begin
          stall_q[i] <= stall_q[i] + 16'd1;
          run_q[i]   <= run_q[i] + 16'd1;
          if (run_q[i] + 16'd1 > max_stall_q[i]) max_stall_q[i] <= run_q[i] + 16'd1;
        end else run_q[i] <= '0;
      end
    end
  end
endmodule

n_accept_q counts acceptances, not offers. A counter incremented when a request is offered measures how hard the producer tried, which is a different and much less useful quantity — and it inflates exactly when the system is congested, so the metric looks best when the system is worst.

Both a total and a maximum run per stage. The total says how much time was lost; the maximum says what the worst single wait was. A stage with 39 stall cycles in runs of 2 and a stage with 39 in one run of 39 are different systems, and only the maximum separates them.

Simulation evidence — a device stall

=== EXP2: the same request with the DEVICE resource slow ===
  device stalled 8 cycles: busy=01000 watchdog_age=16 blame=DEVICE
  stall cycles by stage : host=0 link=0 fabric=0 device=5 return=0
  -> the DEVICE stage is the one holding the request, because that is
     where resource_done gates it

One stage accumulating stalls and every other at zero — an unambiguous localisation from four counters, with no trace and no timestamps.

Simulation evidence — sustained congestion, and a lesson

Sixty requests with the fabric accepting one cycle in three:

=== EXP3: sustained traffic with the FABRIC as the bottleneck ===
  accepted=23 completed=23
  stall cycles : host=39 link=39 fabric=0 device=5 return=0
  max stall run: host=2 link=2 fabric=0 device=5 return=0

The fabric stage reads zero, and the fabric was the bottleneck. This is not a measurement error — it is how backpressure works, and it is the most useful thing in the chapter.

A stage stalls when it is holding a request it cannot hand onward. When the fabric refuses, the stage holding a request is the link stage, which is upstream of the refusal. So the stall accumulates upstream of its cause, and it propagates further upstream as the queue behind it fills — which is why host also reads 39.

Read the stall profile as a gradient. High counts upstream, zero at the bottleneck itself, and the bottleneck is immediately below the last stage with a high count. An engineer who reads "fabric=0" as "the fabric is fine" will look everywhere else, and this is exactly the reasoning error the counters exist to prevent.

Note also accepted=23 from 60 offers. The host bridge refused 37 requests, which is the backpressure reaching the core — and it is why the core sees "memory is slow" for a fabric problem two stages away.

10. RTL 4 — Which Stage Actually Failed

Purpose

Under a cascading failure, every stage reports something. Only one of them started it.

// Record where the FIRST error was detected, and refuse to be overwritten.
//
// Under a cascading failure every stage reports something, and the last
// reporter is usually a victim rather than a cause. Latching the first one is
// what keeps the report pointing at the origin.
//
// LAST_WINS=1 is the bug shape.
//
// GENERIC teaching model.
module error_origin #(
  parameter bit LAST_WINS = 1'b0
) (
  input  logic       clk,
  input  logic       rst_n,
  input  logic [4:0] err_pulse,     // one bit per stage
  output logic [2:0] origin_q,
  output logic       origin_valid_q,
  output logic [4:0] seen_mask_q    // every stage that ever reported
);
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin /* ... */ end
    else begin
      seen_mask_q <= seen_mask_q | err_pulse;
      if (any) begin
        // First writer wins, unless the design lets the last one overwrite.
        if (LAST_WINS || !origin_valid_q) begin
          origin_q       <= LAST_WINS ? highest : lowest;
          origin_valid_q <= 1'b1;
        end
      end
    end
  end
endmodule

seen_mask_q is kept alongside the origin, and both are needed. The origin says where to start; the mask says how far the damage spread. A report with only the origin understates the impact, and a report with only the mask does not say where to look.

Simulation evidence

The fabric fails; the device and then the host report as consequences:

=== EXP5: a cascading failure, and who gets blamed ===
  stages that reported: 01101
  first-wins  : origin=FABRIC
  last-wins   : origin=HOST  <-- blames a victim

The last-wins design reports the host, which is the stage furthest from the cause and the one whose "error" was simply a timeout on a request that never came back. An investigation starting at the host finds a healthy host, and the next step is unclear.

The first-wins design reports the fabric, and the mask 01101 says the device and host were affected too. That is a complete report from two registers.

11. Waveform — One Request, With and Without a Stall

Three readings.

The stall is visible as stage_busy not moving. Cycles 5 through 8 all read 01000 — the request is in the device stage and going nowhere. No other signal is needed to localise it, which is the entire argument for exposing a per-stage occupancy vector.

Three stalled cycles cost three cycles of latency, and nothing else was affected because only one request was in flight. Under load the same three cycles would have backed up behind it, which is why Chapter 3.3's queue measurements and this chapter's max_stall_run matter more than the mean.

The tag arrives as 1. Identity survived five stages down and back, which is the correctness property that makes the whole path usable — and Chapter 3.5 showed what a single truncated field does to it.

12. Assertions

Icarus does not execute concurrent SVA, so these were not run; the table gives the procedural check.

// S1 — a stage never holds two requests.
a_stage_single: assert property (@(posedge clk) disable iff (!rst_n)
  busy_q[s] |-> (tag_q[s] != 5'd0));
 
// S2 — identity survives: what completes is what was issued.
a_tag_preserved: assert property (@(posedge clk) disable iff (!rst_n)
  cmpl_valid |-> (cmpl_tag == expected_tag_for(cmpl_dest)));
 
// S3 — a stage advances only if the next is free or advancing too. This is
// what makes a full pipeline move; computed forwards it silently halves rate.
a_advance_legal: assert property (@(posedge clk) disable iff (!rst_n)
  adv[s] |-> (!busy_q[s+1] || adv[s+1]));
 
// S4 — nothing is accepted when the ingress cannot take it.
a_no_accept_when_full: assert property (@(posedge clk) disable iff (!rst_n)
  !req_ready |-> !(req_valid && req_ready));
 
// S5 — every accepted request eventually completes, GIVEN that every stage
// gate eventually opens. The assumption is the point: without it this is
// unprovable, because a permanently closed gate is a permanent stall.
a_eventually_completes: assert property (@(posedge clk) disable iff (!rst_n)
  (req_valid && req_ready) |-> ##[1:MAX_LATENCY] cmpl_valid);
 
// S6 — a timeout always names a stage that is actually holding something.
a_blame_is_real: assert property (@(posedge clk) disable iff (!rst_n)
  timeout |-> stage_busy[blame_stage]);
 
// S7 — the recorded error origin is never overwritten once valid.
a_origin_sticky: assert property (@(posedge clk) disable iff (!rst_n)
  origin_valid_q |=> (origin_q == $past(origin_q)));
 
// S8 — acceptance is counted on acceptance, never on offer.
a_accept_counted_once: assert property (@(posedge clk) disable iff (!rst_n)
  $rose(n_accept_q != $past(n_accept_q)) |-> $past(req_valid && req_ready));

S5's assumption is the honest part. End-to-end completion cannot be proven from the pipeline's own logic — it depends on every downstream gate eventually opening, and those are outside the module. Stated as an assumption it is a contract on the environment; unstated, the property fails on correct hardware attached to a stuck device. This is the same shape as Chapter 3.4's liveness assumption, and it recurs because liveness at a boundary always depends on the far side.

13. Design Review — the Staff-Engineer Pass

The questions worth asking about any end-to-end path, in the order a reviewer asks them.

Where can this request block? Enumerate every stage. There should be a named queue or table at each, and for each one an answer to "what happens when it is full". A stage with no answer is a stage that drops.

Which queue owns each backpressure edge? Every ready should trace to a specific resource. A ready that is a function of several things is a place where the cause of a stall becomes unrecoverable from observation.

Are tags unique end-to-end? Not per stage — end to end, including the return path. And is the width the same at every boundary? Chapter 3.5 measured what one truncated field does.

Can the fabric reorder requests? If yes, who restores order, and does anything depend on order that has not been told? If no, what enforces it, and what does that cost in head-of-line blocking?

Who preserves ordering, and for what? Ordering guarantees are per-class and per-address; a blanket answer means nobody has thought about it.

What is the state at every stage after reset? Specifically: can a request be in flight while a stage resets, and what happens to it? A stage that resets to empty while a completion is inbound produces an unknown completion.

How are errors surfaced, and who is blamed? First-writer-wins with a spread mask, or last-writer-wins? Measured, the second blames a victim.

How do we observe occupancy? Per stage, with both a total and a maximum run. And is acceptance counted on acceptance or on offer?

What is the maximum in-flight count, and where is it enforced? Usually the host's outstanding table. NSLOT / T_service is the throughput ceiling regardless of link bandwidth.

What happens under simultaneous traffic in all three protocol classes? Do they share queues, budgets, or arbitration? Chapter 2.4 measured what sharing costs.

Can one class starve another, and where is fairness enforced? There should be exactly one answer per contention point, and it should be a bounded number rather than "it works out".

How is latency measured? If the answer is "we timestamp requests", ask about the overhead and the sampling rate. Occupancy plus throughput via Little's Law is cheaper and always on.

14. Bottleneck Localisation — the Procedure

Given "the CXL memory is slow", in order:

1. Is it actually CXL? Check the achieved link mode. Chapter 2.4 showed a link can be up, working and silently running as PCIe.
2. Read acceptance versus offer at the host. Measured, 23 accepted from 60 offers means the host is refusing — the problem is downstream and the core is feeling it as memory latency.
3. Read the per-stage stall profile as a gradient. High upstream, zero at the bottleneck. The bottleneck is immediately below the last stage with a high count.
4. Compare totals against maximum runs. Many short stalls are congestion; one long stall is a stuck resource, and they have different fixes.
5. Check the outstanding table's maximum occupancy. Pinned at capacity means the design is issue-limited, and NSLOT / T_service is the ceiling no bandwidth fixes.
6. Check coherence separately. A high max_stall on the coherence stage with normal counts elsewhere is contention on hot lines — a data-layout problem, per Chapter 3.4.
7. Only then look at the media. It is the one term with a fixed physical cost, and it is almost never what changed.

The ordering is the value. Most investigations start at step 7 and work backwards, which is the expensive direction.

15. Debug Lab

// Each stage assigns its own local tag from its own free list.
if (adv1) tag_q[2] <= local_next_tag;      // not tag_q[1]

    cyc 4: busy=10000  cmpl=1 tag=1
  -> 5 stages, so an unloaded request takes 5 hops

if (adv1) tag_q[2] <= tag_q[1];            // the SAME tag moves forward
a_tag_preserved: assert property (cmpl_valid |-> (cmpl_tag == issued_tag));

// The request has been outstanding too long.
if (age_q >= LIMIT) report_error("transaction timeout");

  age=24 timeout=1 latched=1 blame=FABRIC
  -> a bare timeout says 'stuck'; naming the stage says where

always_comb begin
  blame_stage = 3'd0;
  for (i = 0; i < 5; i = i + 1) if (stage_busy[i]) blame_stage = i[2:0];
end
a_blame_is_real: assert property (timeout |-> stage_busy[blame_stage]);

// A stage advances if the next stage is empty.
assign adv0 = busy_q[0] && link_ready   && !busy_q[1];
assign adv1 = busy_q[1] && fabric_ready && !busy_q[2];
assign adv2 = busy_q[2] && device_ready && !busy_q[3];

  accepted=23 completed=23

assign adv3 = busy_q[3] && resource_done && (!busy_q[4] || adv4);
assign adv2 = busy_q[2] && device_ready  && (!busy_q[3] || adv3);
assign adv1 = busy_q[1] && fabric_ready  && (!busy_q[2] || adv2);
assign adv0 = busy_q[0] && link_ready    && (!busy_q[1] || adv1);

// Count requests entering the system.
if (req_valid) n_accept_q <= n_accept_q + 1;      // offer, not acceptance

  accepted=23 completed=23

if (req_valid && req_ready) n_accept_q <= n_accept_q + 1;

// Record the most recent error, so the report is current.
if (|err_pulse) origin_q <= highest_reporting_stage;

  stages that reported: 01101
  first-wins  : origin=FABRIC
  last-wins   : origin=HOST  <-- blames a victim

if (|err_pulse && !origin_valid_q) begin
  origin_q <= first_reporting_stage; origin_valid_q <= 1'b1;
end
seen_mask_q <= seen_mask_q | err_pulse;      // how far it spread

// Find the bottleneck: the stage with the most stall cycles.
bottleneck = argmax(stall_q);

  stall cycles : host=39 link=39 fabric=0 device=5 return=0
  max stall run: host=2 link=2 fabric=0 device=5 return=0

The bottleneck is immediately BELOW the last stage with a high stall count.
  host=39  link=39  fabric=0  ...  ->  the fabric gate is the constraint

16. How This Appears in Real Engineering

CXL / system architect

The decomposition in Section 6 is the design budget. Each term is owned by a team, and the sum is the number the product is held to — so the architecture work is allocating a latency budget across stages and deciding which terms are allowed to be unbounded. The answer should be "none", which means every stage needs a bound and a backpressure path.

SoC architect

The maximum in-flight count is usually set by the host's outstanding table, and NSLOT / T_service is a throughput ceiling that no amount of link bandwidth raises. That single expression decides whether a memory tier is usable, and it is a sizing decision made early and hard to change late.

RTL engineer

Four disciplines from the measured defects: carry one identity end to end, compute the advance chain from the back, count handshakes rather than offers, and latch the first error rather than the last. The third and fourth are both counter-wiring errors that pass code review.

Verification engineer

End-to-end stimulus is different in kind from per-block stimulus. Sustained load against the analytical throughput ceiling, because the forward-chain defect is invisible with one request in flight. Multi-stage failures, because a single-stage failure cannot distinguish first-wins from last-wins reporting. And a stuck gate at every stage in turn, so the blame logic is checked rather than assumed.

Performance engineer

Section 14 is the procedure. The two things to internalise: acceptance versus offer at the ingress tells you immediately whether the problem is downstream, and the stall profile is a gradient whose zero marks the bottleneck. Little's Law from occupancy and throughput gives you latency continuously without timestamping anything.

Firmware and system software

"CXL latency" is not a number your code can rely on, and the distribution is bimodal — a cache hit skips the entire path. Placement decisions (Chapter 2.5) matter more than any single-access latency, because they change how often the path is taken at all.

System integrator

Most of what you will diagnose is configuration, not silicon: a link that came up as PCIe, a routing table with an overlap, a binding that was never applied. Step 1 of Section 14 exists because it is the most common answer and the cheapest to check.

Silicon debug

Per-stage occupancy is the single most valuable observable in the system. A vector showing where every in-flight request sits localises almost any symptom in one read, and it is far cheaper than a trace. If you can only add one debug feature to a path like this, add that.

17. Common Misconceptions

18. Interview Reasoning

19. Exercises

1. Budget the path. Assign plausible relative weights to the nine terms in Section 6 for a directly attached Type 3 device, then again with one switch. Which term changed most, and by how much once you account for the return path?

2. Compute the ceiling. A host has 24 outstanding slots and the round trip is 350 cycles. What is the maximum sustainable request rate? How many slots would double it, and what else would have to be true for that to help?

3. Read a profile. Given stalls of host=120 link=118 fabric=115 device=0 return=0, name the bottleneck and justify it from the gradient rule. Now given host=4 link=2 fabric=0 device=210 return=0, do the same. Which of the two is more likely a stuck resource, and which is congestion?

4. Break S5. Construct a scenario in which "every accepted request eventually completes" is violated while every safety property in Section 12 holds. State the environment assumption your property needs, and say what the hardware must do when it is violated.

5. Design per-entry ageing. Section 8's watchdog tracks one request and can be cleared by a different request reusing its tag. Sketch the per-entry version: what does it cost in area, and what new observable does it give you that the single-slot version cannot?

20. Summary

A CXL request is a journey through stages, each holding state, each able to stall for its own reason, and each obliged to preserve one thing: the transaction's identity. The measured pipeline delivered tag 1 unchanged after five stages down and back, and that survival is what makes the whole path usable.

Latency is a sum of nine terms, not a number. Most are zero most of the time, any one can dominate, and the transport terms are paid twice. Little's Law closes the loop cheaply — accumulated occupancy 225 over 127 samples gave a mean of 1.77 in flight, which is a continuous latency measurement from two counters and no timestamps.

The most useful measured result is counter-intuitive. With the fabric deliberately made the bottleneck, the stall counters read host=39 link=39 fabric=0 device=5. The bottleneck stage read zero, because a stage stalls when it is holding a request it cannot hand onward — so stalls accumulate upstream of their cause and propagate further upstream as queues fill. Read the profile as a gradient; the bottleneck is immediately below the last stage with a high count. An argmax names the host, and the host is fine.

Three integration defects, none of which is a correctness bug. An advance chain computed forwards preserved every tag, completed every request, violated nothing, and halved sustained throughput. A counter incremented on offer rather than acceptance would have read 60 where 23 requests were accepted — a metric that improves as the system degrades. And last-writer-wins error reporting named the host in a failure the fabric started, because in a cascade the reporting order is roughly the reverse of the causal order.

Instrumentation is what makes the system diagnosable. Per-stage occupancy localises almost any symptom in one read. A timeout that samples occupancy named the stage — blame=FABRIC — and eliminated four others in a word, with the correct reading being "got as far as the fabric" rather than "the fabric is broken". An error origin latched first, with a spread mask alongside, is a complete report from two registers.

And the habit to keep: localise before you optimise, and start at the ingress. Acceptance versus offer at the host says immediately whether the problem is downstream. Most investigations start at the media, which is the one term with a fixed physical cost and almost never the thing that changed.

21. Module 3 Complete — What You Should Now Be Able to Answer

This chapter closes Module 3. The module set out to remove one layer of architectural ambiguity per chapter, and the test is whether these are now answerable without hesitation.

What does the host own? Address decode, outstanding-request tracking, and the coherence point — because each needs a system-wide view and only the host has one.

What does the device own? Which protocol engines exist and therefore what device type it is; dispatch, its own outstanding slots, and rate-matching the link to its internal resources — everything local knowledge suffices for.

What does the fabric do? Routing, arbitration and binding — three kinds of software-written state that can be wrong while every link is healthy.

How is coherence coordinated? By a single home per line that serialises requests, invalidates existing copies before granting a writer, waits for every response, and returns the value from wherever it actually is. Asymmetrically, with the host orchestrating.

Which layer owns which responsibility? The protocol layer owns meaning, the link layer owns arrival, the physical layer owns bits — and each must not know what the layer below it is doing.

How does one request travel through the system? Twelve steps, eleven of which can stall.

Where can it stall? Every queue and every table: the core, coherence, the outstanding tables at both ends, the layer boundaries, the fabric's egress, the device's queue.

What state does hardware retain? Directory state per line, outstanding entries at host and device, routing and binding tables in the fabric, and queue contents at every boundary.

How can it fail? Silently, mostly. A stale read that returns a plausible value; a completion delivered to the wrong requester; a class starved to zero; a permission computed and not enforced; a throughput halved with nothing violated.

How would I verify it? With properties that check a decision against its source rather than against itself, with liveness properties whose environment assumptions are written down, and with stimulus that saturates — because nearly every defect in this module is invisible on a lightly loaded, correctly configured, single-device bench.

Module 4 moves from architecture to the protocols themselves, beginning with CXL.io. For adjacent material, every chapter of this module is on the CXL tutorials index: The CXL Host, The CXL Device, The CXL Fabric, Coherent Communication Model and CXL Layered Architecture.