Architectural Goals

A goal that cannot be violated is not a goal, it is a description. "CXL enables memory expansion" is a description. "CXL enables memory expansion without making the average access materially slower" is a goal, because a design can fail it, and failing it is what most naive implementations do.

This chapter states CXL's architectural goals in the second form, and then implements them — because a goal expressed as logic is a goal you can regress against.

1. The One-Sentence Model

CXL's goals are to attach devices coherently at low latency, expand memory capacity without wrecking access latency, pool and compose resources across hosts, and do all of it on existing platforms — and every one of those is a constraint that a working implementation can still violate.

2. What This Chapter Owns

3. Four Goals, Each Restated as an Obligation

The right-hand column is what the rest of the chapter builds. Notice the shape common to all four rows: each is a conjunction, and the second half is the part implementations drop. A device that caches whenever it wants satisfies "coherent attach". A tier that adds capacity and doubles mean latency satisfies "expansion". A pool that lets one host take everything satisfies "pooling". Each is the goal with its constraint removed.

4. Goal 1 — Low-Latency Coherent Attach

The first goal is the one Module 1 spent seven chapters motivating: a device should participate in the coherence domain rather than being handed copies.

What makes it hard is the "low-latency" half. Coherence has a cost — Chapter 1.7 measured an invalidation round trip on a contended write — and the design goal is that participating coherently must be cheaper than the copying it replaces. A coherent attach that costs more than a DMA copy has satisfied the letter of the goal and defeated its purpose.

What makes it dangerous is the safety half. A device holding a cached copy it is not entitled to is a correctness failure, not a performance one, and it is silent. Section 10's gate is that obligation as logic.

5. Goal 2 — Expansion Without Wrecking Latency

Adding memory over a link is easy. Adding it without destroying the average access time is the goal.

The arithmetic is unforgiving and worth doing explicitly. With a near tier at latency L and a far tier at latency F, and a fraction f of accesses landing far:

mean latency  =  (1 - f) · L  +  f · F

With L = 10 and F = 30 — the teaching values used in Section 8 — the mean is 10 + 20f. A 25% far-access rate costs 50% more mean latency. That is the entire difficulty of tiered memory in one line: the far tier's latency matters much less than the fraction of traffic that reaches it, and that fraction is a placement decision, not a hardware property.

Which yields the real obligation: the design must keep hot data near. Not "must have a fast far tier" — must place well. Section 8 measures both policies on identical stimulus.

6. Goal 3 — Pooling and Composability

Pooling means capacity is allocated where it is needed rather than fixed at build time. That addresses Chapter 1.1's stranding problem, and it introduces one of its own.

A shared resource with no floor is a resource one participant can take entirely. This is Chapter 2.4's shared-outstanding-pool defect at rack scale, and the measurement in Section 9 is correspondingly blunt: first-come-first-served gave one host the entire pool and three hosts nothing at all.

So the obligation has two halves: capacity must be shareable, and every participant must have a guarantee it can rely on. A pool that can be monopolised is worse than fixed allocation, because fixed allocation at least fails predictably.

7. Goal 4 — Compatibility as a First-Class Goal

Compatibility is usually described as a constraint CXL works around. It is better understood as a goal, because it changed the architecture.

Chapter 2.4 showed the mode FSM coming up as PCIe unconditionally: the fallback is not a degraded corner case, it is the default, and CXL is what a capable pair negotiates up to. Chapter 2.3 showed the same principle across revisions — every capability additive and gated on the negotiated revision.

The design consequence is that "it works" is never sufficient evidence. A link that is up may not be CXL; a pair at the same revision may share few features. Every goal in this chapter has a corresponding "and it is actually happening" question, which is why Section 11's counters exist.

8. RTL 1 — Placement, and What Bad Placement Costs

Purpose

Goal 2's obligation, as logic: keep hot data near.

// Decide whether a page lives in local DRAM or in CXL-attached memory.
//
// The architectural goal "expand capacity without wrecking latency" only holds
// if HOT pages stay near. This module is that goal expressed as logic.
//
// MODE 0 -- capacity-only: place far whenever local is full. Simple, and it
//           exiles hot pages the moment local fills.
// MODE 1 -- hotness-aware: keep hot pages local by evicting a cold one.
//
// GENERIC teaching model: thresholds and tier latencies are teaching values.
module placement_policy #(
  parameter int unsigned LOCAL_CAP  = 8,
  parameter int unsigned HOT_THRESH = 6
) (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       mode_hot,          // 0 = capacity-only, 1 = hotness-aware
  input  logic       place_req,
  input  logic [7:0] page_heat,
  input  logic       cold_victim_avail, // a cold local page could be evicted
  output logic       place_local,
  output logic       place_far,
  output logic       evict_cold,
  output logic [7:0] local_used_q,
  output logic       hot_exiled,        // a hot page was sent far
  output logic       overcommit_err
);
  logic is_hot, local_full;
 
  assign is_hot     = (page_heat >= HOT_THRESH[7:0]);
  assign local_full = (local_used_q >= LOCAL_CAP[7:0]);
 
  always_comb begin
    place_local = 1'b0; place_far = 1'b0; evict_cold = 1'b0;
    if (place_req) begin
      if (!local_full) begin
        place_local = 1'b1;
      end else if (mode_hot && is_hot && cold_victim_avail) begin
        // Local is full, but this page is hot and something cold can go far.
        place_local = 1'b1;
        evict_cold  = 1'b1;
      end else begin
        place_far = 1'b1;
      end
    end
  end
 
  // A hot page placed far is the GOAL being violated, not an error condition.
  // Counting it is how the goal becomes testable.
  assign hot_exiled = place_far && is_hot;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      local_used_q <= '0; overcommit_err <= 1'b0;
    end else begin
      // Eviction frees a slot in the same cycle the new page takes one.
      if (place_local && !evict_cold) local_used_q <= local_used_q + 8'd1;
      if (local_used_q > LOCAL_CAP[7:0]) overcommit_err <= 1'b1;
    end
  end
endmodule

hot_exiled is the important output and it is not an error flag. The design is functioning correctly when it asserts; what has failed is the architectural goal. Distinguishing "this violated a rule" from "this violated an intent" is what makes a goal regressable, and most designs instrument only the first.

Simulation evidence

Both policies instantiated on identical stimulus — sixteen pages into eight local slots, alternating hot and cold:

=== EXP1: 16 pages into 8 local slots, alternating hot and cold ===
  capacity-only  : local=8  far=8  hot pages exiled=4
  hotness-aware  : local=12 far=4  hot pages exiled=0
 
  mean access latency (weighted, arbitrary units):
    capacity-only  = 320 / 16 = 20
    hotness-aware  = 240 / 16 = 15

Same hardware, same capacity, same workload — 25% lower mean latency from the placement decision alone. The capacity-only policy exiled four hot pages; the hotness-aware policy exiled none, and it did so by moving cold pages far instead.

Two things follow that are worth stating carefully.

The far tier's latency is not what determined the outcome. Both configurations used LAT_FAR = 30. What differed was f, the fraction of accesses landing far — 50% versus 25% — which is exactly what Section 5's arithmetic predicts. A tiered memory design's latency is dominated by its placement policy, not by its far tier's speed.

And the capacity-only policy is not broken. Every page was placed, nothing overflowed, no invariant fired. It satisfies "memory expansion" completely and misses the goal entirely — which is Section 3's pattern in measured form.

9. RTL 2 — Pooling With a Floor

Purpose

Goal 3's obligation: share capacity without letting one participant take it all.

// Allocate a shared memory pool across four hosts.
//
// MODE 0 -- first-come-first-served: whoever asks first takes what is there.
// MODE 1 -- reserved floor: every host is guaranteed FLOOR blocks that no
//           other host can consume; the remainder is contended.
//
// GENERIC teaching model. This is NOT a CXL fabric-manager allocation policy.
module pool_reservation #(
  parameter int unsigned NHOST = 4,
  parameter int unsigned TOTAL = 16,
  parameter int unsigned FLOOR = 2
) (
  input  logic             clk,
  input  logic             rst_n,
  input  logic             mode_floor,
  input  logic [NHOST-1:0] alloc_req,
  input  logic [NHOST-1:0] free_req,
  output logic [NHOST-1:0] alloc_ok,
  output logic [7:0]       free_q,
  output logic             floor_violated,
  output logic             oversubscribe_err
  // ... per-host held counters elided ...
);
  // How much of the free pool is spoken for by hosts still below their floor.
  always_comb begin
    reserved_for_others = 8'd0;
    for (i = 0; i < NHOST; i = i + 1)
      if (held[i] < FLOOR[7:0]) reserved_for_others = reserved_for_others
                                                    + (FLOOR[7:0] - held[i]);
  end
 
  always_comb begin
    for (i = 0; i < NHOST; i = i + 1) begin
      if (!mode_floor) begin
        // FCFS: anything free is fair game.
        eligible[i] = alloc_req[i] && (free_q != 0);
      end else begin
        // Below your floor you are always served. Above it you may only take
        // what is not reserved for someone still below theirs.
        eligible[i] = alloc_req[i] &&
          ((held[i] < FLOOR[7:0]) ? (free_q != 0)
                                  : (free_q > reserved_for_others));
      end
    end
  end
 
  // Only ONE block leaves the pool per cycle, chosen from a rotating start.
  // Without this the per-host counters and the free counter disagree and the
  // pool oversubscribes -- a real defect, found in simulation.
  always_comb begin
    alloc_ok = '0; granted_any = 1'b0;
    for (j = 0; j < NHOST; j = j + 1) begin
      i = (rr_ptr_q + j) % NHOST;
      if (!granted_any && eligible[i]) begin
        alloc_ok[i] = 1'b1; granted_any = 1'b1;
      end
    end
  end
endmodule

Simulation evidence

Host 0 allocates alone for long enough to drain a 16-block pool, then hosts 1 to 3 start asking:

=== EXP2: host 0 grabs the pool, then hosts 1-3 ask ===
  FCFS          : held=16/0/0/0 free=0
  reserved floor: held=10/2/2/2 free=0
  -> FCFS gave three hosts NOTHING
  -> the floor held for every host

Three hosts received zero blocks. Not a small share, not a delayed share — nothing, permanently, with no error anywhere. First-come-first-served is not a fairness policy with poor characteristics; it is the absence of one.

The floor configuration reserved six blocks (three hosts × two) and let host 0 take the remaining ten. Host 0 got less, and that is the point: a guarantee is capacity you deliberately decline to allocate to whoever asks first. There is no fairness mechanism that does not cost the greedy participant something.

Note also that the floor is enforced against a host that has not yet asked. reserved_for_others counts deficits of hosts below their floor regardless of whether they are currently requesting, which is what makes the guarantee meaningful — a floor that only applies to hosts already asking is not a floor, because by the time they ask the pool is gone.

10. RTL 3 — The Coherent-Mode Gate

Purpose

Goal 1's safety obligation: a device caches host memory only when it is entitled to.

// Permit a device to hold a coherent cached copy only when every precondition
// is true. The architectural goal is "coherent attach"; this is the goal's
// safety condition, and it is a conjunction on purpose.
//
// GENERIC teaching model, NOT a CXL specification mechanism.
module coherent_gate (
  input  logic clk,
  input  logic rst_n,
  input  logic dev_can_cache,       // device type supports caching at all
  input  logic region_coherent,     // this address region participates
  input  logic host_granted,        // host has granted the device permission
  input  logic link_is_cxl,         // not running in PCIe fallback
  input  logic req_cache,
  output logic allow_cache,
  output logic refuse_cache,
  output logic [3:0] refuse_reason, // grant / region / device-type / link
  output logic unsafe_cache_err
);
  logic all_ok;
  assign all_ok = dev_can_cache && region_coherent && host_granted && link_is_cxl;
 
  assign allow_cache  = req_cache &&  all_ok;
  assign refuse_cache = req_cache && !all_ok;
 
  // One bit per cause, so a refusal says WHICH precondition failed. Bundling
  // two causes into one bit makes the two indistinguishable in a log.
  assign refuse_reason = { (req_cache && !host_granted),
                           (req_cache && !region_coherent),
                           (req_cache && !dev_can_cache),
                           (req_cache && !link_is_cxl) };
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) unsafe_cache_err <= 1'b0;
    else if (allow_cache && !all_ok) unsafe_cache_err <= 1'b1;
  end
endmodule

link_is_cxl is the term people forget, and it connects directly to Chapter 2.4's fourth Debug Lab. A link that has fallen back to PCIe carries no coherence, so a device that caches based on device type and host grant alone will cache over a link that cannot maintain the copy. The other three preconditions are all true in that scenario.

Simulation evidence

=== EXP3: when a device may hold a coherent copy ===
  everything true            allow=1 refuse=0 reason=0000
  device type cannot cache   allow=0 refuse=1 reason=0010
  region is not coherent     allow=0 refuse=1 reason=0100
  host has not granted       allow=0 refuse=1 reason=1000
  link fell back to PCIe     allow=0 refuse=1 reason=0001
  -> four of five refused; the condition is a conjunction

Every refusal carries a distinct reason code. That is a deliberate design choice with an operational payoff: a device that reports "caching refused" tells you nothing, and one that reports "caching refused: link is not CXL" ends the investigation immediately.

11. RTL 4 — Making the Goal a Number

Purpose

Goal 2 is only meaningful if mean latency is measured.

// Make the architectural goal measurable: count accesses per tier and
// accumulate weighted latency, so "expansion without wrecking latency"
// becomes a number a regression can fail on.
//
// GENERIC teaching model; the tier latencies are teaching values in
// arbitrary units, not measured device numbers.
module tier_counters #(
  parameter int unsigned LAT_LOCAL = 10,
  parameter int unsigned LAT_FAR   = 30
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        acc_local,
  input  logic        acc_far,
  output logic [15:0] n_local_q,
  output logic [15:0] n_far_q,
  output logic [31:0] lat_sum_q,
  output logic [15:0] n_total_q
);
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_local_q <= '0; n_far_q <= '0; lat_sum_q <= '0; n_total_q <= '0;
    end else begin
      if (acc_local) begin
        n_local_q <= n_local_q + 16'd1;
        lat_sum_q <= lat_sum_q + LAT_LOCAL;
      end
      if (acc_far) begin
        n_far_q   <= n_far_q + 16'd1;
        // Both in one cycle: accumulate both, do not overwrite.
        lat_sum_q <= lat_sum_q + (acc_local ? LAT_LOCAL + LAT_FAR : LAT_FAR);
      end
      n_total_q <= n_total_q + {15'b0,acc_local} + {15'b0,acc_far};
    end
  end
endmodule

The acc_local ? LAT_LOCAL + LAT_FAR : LAT_FAR term exists because of a real hazard. Two non-blocking assignments to lat_sum_q in the same cycle mean the last one wins, so a simultaneous local and far access would silently lose the local contribution. Counters that are quietly wrong are worse than absent ones, because they are believed.

Cost. Two 16-bit counters, a 32-bit accumulator and an adder — a few hundred gates, no timing impact, and it turns Goal 2 from an aspiration into a regression check. That ratio is why counters are the cheapest architectural insurance a design can carry.

What the counters produced

These are the numbers behind Section 8's conclusion, taken from the same run:

The capacity-only run placed 8 local and 8 far for a lat_sum of 320 over 16 accesses; the hotness-aware run placed 12 local and 4 far for a lat_sum of 240 over the same 16.

The measured means match 10 + 20f exactly, which is the check worth doing on any counter: a measurement that agrees with a model you derived independently is a measurement you can trust. One that does not means either the model or the counter is wrong, and finding out which is always worth the time.

12. Assertions

Icarus does not execute concurrent SVA, so these were not run; the table gives the procedural check.

// G1 — the near tier is never overcommitted.
a_local_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  local_used_q <= LOCAL_CAP);
 
// G2 — a placement is exactly one of local or far, never both.
a_place_exclusive: assert property (@(posedge clk) disable iff (!rst_n)
  $onehot0({place_local, place_far}));
 
// G3 — GOAL property, not a safety property: a hot page is not exiled while
// the design could have evicted a cold one instead.
a_hot_stays_near: assert property (@(posedge clk) disable iff (!rst_n)
  (mode_hot && cold_victim_avail) |-> !hot_exiled);
 
// G4 — the pool is never oversubscribed.
a_pool_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  (held[0] + held[1] + held[2] + held[3]) <= TOTAL);
 
// G5 — a host below its floor that asks is always served.
a_floor_honoured: assert property (@(posedge clk) disable iff (!rst_n)
  (mode_floor && alloc_req[h] && (held[h] < FLOOR)) |-> eligible[h]);
 
// G6 — caching is never allowed unless every precondition holds.
a_cache_requires_all: assert property (@(posedge clk) disable iff (!rst_n)
  allow_cache |-> (dev_can_cache && region_coherent && host_granted && link_is_cxl));
 
// G7 — a refusal always names a cause.
a_refusal_explained: assert property (@(posedge clk) disable iff (!rst_n)
  refuse_cache |-> (refuse_reason != '0));

G3 is not like the others. G1, G2, G4, G6 and G7 are safety properties — a violation means something illegal happened. G3 says the design did something legal that defeats its purpose, and the capacity-only policy violates it four times while every other property passes.

That distinction is the chapter's central methodological claim: architectural goals need their own properties, and those properties will not fire on any bug. They fire on designs that work.

13. Debug Lab

// Local is full, so the new page goes to the expansion tier.
if (!local_full) place_local = 1'b1;
else             place_far   = 1'b1;

  capacity-only  : local=8  far=8  hot pages exiled=4   mean latency 20
  hotness-aware  : local=12 far=4  hot pages exiled=0   mean latency 15

if (!local_full)                              place_local = 1'b1;
else if (is_hot && cold_victim_avail) begin   place_local = 1'b1;
                                              evict_cold  = 1'b1; end
else                                          place_far   = 1'b1;

// Whatever is free is available to whoever asks.
eligible[i] = alloc_req[i] && (free_q != 0);

  FCFS          : held=16/0/0/0 free=0
  reserved floor: held=10/2/2/2 free=0

// Deficits of ALL hosts below their floor, asking or not.
for (i = 0; i < NHOST; i++)
  if (held[i] < FLOOR) reserved_for_others += (FLOOR - held[i]);
 
eligible[i] = alloc_req[i] &&
  ((held[i] < FLOOR) ? (free_q != 0) : (free_q > reserved_for_others));

// The device type supports caching and the host granted permission.
assign allow_cache = req_cache && dev_can_cache && region_coherent && host_granted;

  everything true            allow=1 refuse=0 reason=0000
  link fell back to PCIe     allow=0 refuse=1 reason=0001

assign all_ok = dev_can_cache && region_coherent && host_granted && link_is_cxl;
assign refuse_reason = { !host_granted, !region_coherent, !dev_can_cache, !link_is_cxl };

if (acc_local) lat_sum_q <= lat_sum_q + LAT_LOCAL;
if (acc_far)   lat_sum_q <= lat_sum_q + LAT_FAR;   // BUG: last write wins

    capacity-only  = 320 / 16 = 20     (model: 10 + 20*0.50 = 20)
    hotness-aware  = 240 / 16 = 15     (model: 10 + 20*0.25 = 15)

if (acc_far)
  lat_sum_q <= lat_sum_q + (acc_local ? LAT_LOCAL + LAT_FAR : LAT_FAR);

14. How This Appears in Real Engineering

Architecture

The four goals are the acceptance criteria, and each needs a number attached before the design starts. Not "expand memory" but "mean access latency stays under X at the target far-access fraction". Not "pool capacity" but "every host is guaranteed N blocks". Goals without numbers are not testable, and untestable goals are met by whatever the implementation happens to do.

RTL engineer

Three of the four models here are ten to forty lines and each turns a goal into logic: gate coherent caching on every precondition including link mode, make placement hotness-aware rather than capacity-only, and give the pool a floor. The fourth — counters — is the cheapest and the one most often deferred.

Verification engineer

The central point: write properties for the goals, not only for the rules. G3 fires on a design in which nothing is illegal. Stimulus matters as much — a drained pool with a late arrival, a full local tier with a hot page arriving, a link in PCIe fallback. Each of these is a state a random test reaches rarely and a directed test reaches immediately.

Performance engineering

Instrument the far-access fraction before anything else. mean = (1-f)·L + f·F means f dominates, and f is a placement outcome. A tiering investigation that starts by measuring the far tier's latency is measuring the term that matters least.

Firmware and system software

Placement policy usually lives here, which makes Goal 2 largely a software responsibility running on hardware that must expose enough information — access counts per tier, and heat — for the policy to be more than a guess. That is why the observability work in CXL 3.2 matters: policy without measurement is policy without feedback.

15. Common Misconceptions

16. Interview Reasoning

17. Summary

CXL's architectural goals are coherent device attach, memory expansion, pooling and composability, and compatibility. Each is a conjunction whose second term is the one implementations drop, and each was implemented and measured here.

Coherent attach requires a complete gate. Device type, region, host grant and achieved link mode — four preconditions, of which the last is forgotten precisely because the other three are true in a PCIe fallback. Four of five simulated cases refused, each with a distinct reason code.

Expansion requires placement, not just capacity. mean = (1-f)·L + f·F makes the far-access fraction dominant, and that fraction is a policy outcome. Measured on identical hardware: capacity-only placement exiled four hot pages for a mean of 20; hotness-aware placement exiled none for a mean of 15. Same capacity, 25% lower mean latency, from the placement decision alone.

Pooling requires a floor. First-come-first-served gave one host all 16 blocks and three hosts zero, silently. The floor configuration capped the first arrival at 10 and guaranteed every host 2 — and it works only because it reserves against hosts that have not yet asked.

And a goal is only a goal if it is a number. The counters that make Goal 2 regressable cost a few hundred gates, and a wrong counter is worse than none: a lost contribution produces a plausible mean rather than an obviously broken one.

The methodological claim underneath all of it: architectural goals need their own properties. Every defect in Debug Labs 1 and 2 satisfies every safety property in Section 12. They are not designs that break — they are designs that work and miss the point, and only a property written against the intent will say so.

18. What Comes Next

Module 2 is complete: what CXL is, why it is a standard, how it evolved, what it shares with PCIe, and what it is for. Module 3 turns to structure, beginning with Chapter 3.1, The CXL Host — the address decoding, request tracking and home-agent responsibilities that make the host the enforcement point for most of what this chapter called an obligation.

For adjacent material: Relationship to PCIe has the fallback mode the coherent gate depends on, Evolution of CXL has the compatibility mechanisms, and Why Coherent Attach Matters has the coherence cost the first goal trades against. The path is on the CXL tutorials index.