Memory Expansion

Chapter 17.1 treated one memory chiplet as a stateful endpoint. Chapter 11.2 taught the CXL memory-expansion mechanism in depth. This chapter asks what happens when expandable capacity becomes a package-scale tier — several targets, one memory map, and a round trip that is longer than the local one.

1. The One-Sentence Model

Memory expansion makes remote physical capacity participate in one system memory map without pretending that remote capacity has local latency, local bandwidth, local failure behaviour, or local ownership. The word remote keeps its meaning even when the device is inside the same package.

2. What This Chapter Owns

What is new here, and none of it exists in 11.2:

Several targets, one map. 11.2 taught one host and one CXL memory device. This chapter's region table names a target and a tier, and the interesting failures are the ones that need two targets to exist (§11, §30).

Four planes with four different lifetimes. Capacity, mapping, transport and service are independently valid or invalid, and §5's lifetime table is what makes a remap safe or unsafe (§14).

Capacity, bandwidth and concurrency are three separate purchases. §21 and §22 quantify that, and §22's outstanding-depth requirement is the number most expansion designs get wrong.

Temporary and permanent loss are different semantics (§29), and silently rerouting an accepted write is a corruption, not a recovery (§30).

And a stall classifier that names the actual constraint (§32). Composing 11.2's mapping, 17.1's memory-side behaviour and 15.5's attribution discipline produces something none of them has alone: an answer to why an expanded access was slow that is never "memory is slow".

3. Sourcing

4. Four Planes of Expansion

The organising structure of the chapter. Each plane answers a different question, and a system needs all four to be true simultaneously before a single byte can be read.

Two consequences.

No plane implies another. Capacity can exist while the mapping is unprogrammed; the mapping can be valid while the link is down; the link can be operational while the target is in a maintenance window (17.1 §11). §18 is the bug of using one plane's signal as a proxy for all four.

And each plane fails differently. A capacity error is a bring-up problem, a mapping error is silent data corruption, a transport error is a delay or a recovery, and a service limit is latency. The stall classifier in §32 exists to keep them distinguishable at runtime.

5. Plane Lifetimes

The table that decides whether a reconfiguration is safe.

Nothing indexed differently can be reset together correctly (11.5 §4). Read the last column: three rows are rebuilt and every other row is untouched, and no single reset signal can express that split. §28 is what happens when one exists anyway.

6. The Expansion Path

Read the two paths out of the decoder. They differ in every dimension that matters — latency, bandwidth, failure behaviour, and what can go wrong silently. A design that treats them as one memory has already lost the ability to explain its own performance.

7. Remote Means Remote, Even Inside the Package

Expanded memory in the same package is closer than expanded memory on a board. It is not local.

Every row is a reason not to describe expanded memory as "almost local". The honest description is that it is a different tier with a different service contract, and a system that reasons about it as a tier makes correct placement decisions (§24) while one that reasons about it as more RAM does not.

8. The Region Table, Across Several Targets

// ILLUSTRATIVE memory-expansion mapping entry. NOT a CXL or UCIe register
// format (Section 3). Symbolic widths throughout.
typedef struct packed {
  logic                 valid;
  logic [ADDR_W-1:0]    base;      // inclusive
  logic [ADDR_W-1:0]    limit;     // INCLUSIVE — one convention, stated once
  logic [TARGET_W-1:0]  target;    // WHICH expansion chiplet
  logic [TIER_W-1:0]    tier;      // service class — Section 24
  logic [EPOCH_W-1:0]   epoch;     // which configuration produced this binding
} expansion_region_t;
 
expansion_region_t active_region_q [NUM_REGIONS];

Architecture. A small associative range table. target and tier are what make this a package-scale structure rather than 11.2's single-window decoder: several expansion chiplets exist, they may have different service characteristics, and the map must say which one owns each range and what kind of memory it is.

State. NUM_REGIONS entries of flops — small, read every cycle, and written only at a commit event (§12).

Cycle behaviour. Compared combinationally against the incoming address. The comparison is the critical path in many designs, which is why the table is small and why a wide NUM_REGIONS is a timing decision as much as a capability one.

Contract. Every consumer downstream relies on the binding being stable for the life of any request that used it. That is not a property of this table; it is a property of the commit discipline in §12 and the guard in §13.

Failure. Mixing limit conventions — an inclusive limit compared with < gives an off-by-one at every boundary, which 11.2 §8 covers in full. One convention, chosen once, stated in the comment, and never mixed.

DV. Cover first and last address of every region, and the address immediately outside each boundary (§39).

9. Range Matching, Width-Safe

// ILLUSTRATIVE. Inclusive base and inclusive limit — matching the struct.
logic [NUM_REGIONS-1:0] region_match;
 
always_comb
  for (int r = 0; r < NUM_REGIONS; r++)
    region_match[r] = active_region_q[r].valid
                   && (req_addr >= active_region_q[r].base)
                   && (req_addr <= active_region_q[r].limit);

Architecture. A one-hot-or-zero match vector. Zero is a legal outcome — an address in no expansion region is either local or unmapped, and 11.2 §11 establishes that unmapped is a third case with its own handling rather than a synonym for local.

State. None — combinational from the table.

Cycle behaviour. Evaluated once, at acceptance. The result is carried with the request and never recomputed (§16), which is the property §15 violates.

Contract. Downstream logic assumes at most one bit is set. That assumption is asserted in §10 rather than trusted, because the cost of it being false is silent misrouting.

Failure. Comparing with < against an inclusive limit; or using a subtraction that can wrap at the address-space boundary. Both produce an incorrect match at exactly the addresses least likely to be tested.

DV. Directed tests at base, base+1, limit-1, limit, limit+1 for every region.

10. SVA — One Target or None

// MANDATORY. At most one region may claim an address.
property p_at_most_one_region;
  @(posedge clk) disable iff (!rst_n)
    $onehot0(region_match);
endproperty
a_at_most_one_region: assert property (p_at_most_one_region);
 
// And an expansion region must not overlap the local range.
property p_local_and_expansion_disjoint;
  @(posedge clk) disable iff (!rst_n)
    mem_req_fire |-> !(local_range_match && (region_match != '0));
endproperty
a_local_and_expansion_disjoint: assert property (p_local_and_expansion_disjoint);
 
// A committed configuration must be internally consistent — checked at commit,
// so a bad configuration is rejected rather than discovered by a wrong access.
property p_commit_produces_disjoint_regions;
  @(posedge clk) disable iff (!rst_n)
    config_commit_fire |=> regions_are_disjoint(active_region_q);
endproperty
a_commit_produces_disjoint_regions:
  assert property (p_commit_produces_disjoint_regions);

Architecture. Three properties: at most one match, no overlap with local, and a commit-time consistency check.

Why zero matches is not a failure. An address outside every expansion region is a normal case with defined handling. $onehot0 and not $onehot — and getting that wrong turns every local access into an assertion failure.

Why the third property matters most. The first two fire when a bad address is issued, which may be long after the bad configuration was written and in a completely different test. The third fires at the commit itself, naming the configuration rather than its first victim.

DV. Inject an overlapping configuration and confirm the commit-time property fires before any access is made.

11. Wrong RTL — Two Expansion Regions Overlap

// WRONG — a priority encoder silently resolves an overlap that should not exist.
always_comb begin
  sel_target = TARGET_NONE;
  for (int r = 0; r < NUM_REGIONS; r++)
    if (region_match[r]) sel_target = active_region_q[r].target;   // last wins
end

Suppose regions 2 and 5 both claim address X, region 2 naming target A and region 5 naming target B.

Four properties that make this a flagship failure.

Both devices are perfectly healthy. A stores what it was given; B stores what it was given. Neither has any way to know the other exists, let alone that they are both claiming an address.

The transport is perfect. Clean CRC, no retries, no recovery. 16.5 §29's point exactly: transport correctness is a strictly weaker claim.

The symptom depends on the resolution order, which is an implementation detail. A priority encoder that takes the first match routes to A and the system appears to work; the same configuration with a last-wins loop routes to B and corrupts. A synthesis or refactor change can flip it.

And the corruption is at two addresses at once. A's copy of X is stale, and B holds data at an offset that belongs to something else. Neither device can detect either condition.

The fix is not a better priority encoder. It is §10's commit-time check, which refuses the configuration. A priority encoder that resolves an overlap has converted a detectable configuration error into an undetectable data error.

12. Requested and Active Configuration

// ILLUSTRATIVE. Two copies, and only one of them is in the address path.
expansion_region_t requested_region_q [NUM_REGIONS];   // software writes here
expansion_region_t active_region_q    [NUM_REGIONS];   // the decoder reads here
 
logic [EPOCH_W-1:0] active_epoch_q;
 
// Program -> validate -> quiesce -> commit. All four, in order.
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n) begin
    active_epoch_q <= '0;
  end else if (config_commit_fire) begin
    active_region_q <= requested_region_q;      // atomic, whole table
    active_epoch_q  <= active_epoch_q + 1'b1;
  end
 
assign config_commit_allowed =
      requested_config_validated              // Section 10's disjointness, checked
   && (affected_outstanding_total == '0);     // Section 13's guard

Architecture. The requested-versus-active split that 14.4 established for lane configuration, applied to the memory map. Software writes freely into requested_region_q; nothing in the address path can see it until a commit.

State. Two copies of the table plus an epoch counter. The duplication is the cost of atomicity, and it is cheap for a table this size.

Cycle behaviour. The whole table transfers in one cycle. Not entry by entry — a partial commit produces exactly §11's overlap transiently, and a request accepted during that window is misrouted with no configuration error ever having existed.

Contract. Every consumer sees a complete, validated, disjoint configuration or the previous one. Never a mixture.

Failure. Committing without the quiesce guard (§13, §15). Or letting software write active_region_q directly for a "quick" reconfiguration, which is the same bug with the mechanism removed.

DV. Assert that active_region_q changes only on config_commit_fire, and that the epoch advances with it (§14).

13. Outstanding Counts as the Commit Guard

// ILLUSTRATIVE. Per-region AND per-target, because they answer different
// questions and a remap may affect either.
logic [CNT_W-1:0] region_outstanding_q [NUM_REGIONS];
logic [CNT_W-1:0] target_outstanding_q [NUM_TARGETS];
 
// Safe simultaneous allocate and retire — one owner, one case statement.
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n) begin
    for (int r = 0; r < NUM_REGIONS; r++) region_outstanding_q[r] <= '0;
  end else begin
    for (int r = 0; r < NUM_REGIONS; r++) begin
      unique case ({alloc_fire && (alloc_region == r[REG_W-1:0]),
                    retire_fire && (retire_region == r[REG_W-1:0])})
        2'b10: region_outstanding_q[r] <= region_outstanding_q[r] + 1'b1;
        2'b01: region_outstanding_q[r] <= region_outstanding_q[r] - 1'b1;
        default: ;                                  // 2'b00 and 2'b11 hold
      endcase
    end
  end

Architecture. Two counter arrays. Per-region answers "may I change this range's binding?"; per-target answers "may I take this device out of service?" — and a reconfiguration may need either, so a design with only one cannot guard both operations.

State. NUM_REGIONS + NUM_TARGETS counters, each wide enough for the maximum outstanding population. Sized from the outstanding-table depth, not guessed — a counter that can saturate silently permits an unsafe commit.

Cycle behaviour. Allocated when a semantic request is accepted, retired at semantic completion — never at a transport event (16.5 §11). The unique case handles the simultaneous allocate-and-retire cycle in one place with one owner, which is 12.2 §20's rule.

Contract. config_commit_allowed depends on these reading zero for the affected entries. The commit logic trusts them completely, so an undercount permits a remap under live traffic (§15) and an overcount deadlocks reconfiguration forever.

Failure. Two independent if statements for increment and decrement — which on a simultaneous cycle produces one net update instead of none, and the counter drifts. The drift is slow, silent, and eventually permits or forbids a commit incorrectly.

DV. Assert the count equals the population of live entries naming that region (§42's conservation property), checked every cycle rather than at quiesce.

14. SVA — the Active Mapping Is Stable While Work Depends On It

// MANDATORY. The active table changes only at a commit.
property p_active_map_changes_only_at_commit;
  @(posedge clk) disable iff (!rst_n)
    !config_commit_fire |=> $stable(active_region_q);
endproperty
a_active_map_changes_only_at_commit:
  assert property (p_active_map_changes_only_at_commit);
 
// A commit requires zero outstanding against every affected region.
property p_commit_requires_quiesce;
  @(posedge clk) disable iff (!rst_n)
    config_commit_fire |-> (affected_outstanding_total == '0);
endproperty
a_commit_requires_quiesce: assert property (p_commit_requires_quiesce);
 
// The epoch advances with the table, so the two can never disagree.
property p_epoch_tracks_commit;
  @(posedge clk) disable iff (!rst_n)
    config_commit_fire |=> (active_epoch_q == $past(active_epoch_q) + 1);
endproperty
a_epoch_tracks_commit: assert property (p_epoch_tracks_commit);
 
// A live transaction's captured target and epoch never change under it.
property p_captured_route_immutable;
  @(posedge clk) disable iff (!rst_n)
    txn_q[IDX].valid |-> ($stable(txn_q[IDX].target) && $stable(txn_q[IDX].map_epoch));
endproperty
a_captured_route_immutable: assert property (p_captured_route_immutable);

Architecture. Four properties: the table is immutable outside a commit, a commit requires quiescence, the epoch is inseparable from the table, and a captured route is immutable.

Why the third is not bookkeeping. The epoch is the only evidence a returning response has about which configuration decided its destination. If the epoch can advance without the table, or the table without the epoch, that evidence is worthless — and §26's check silently passes on stale responses.

Why the fourth is separate from the first. The table being stable does not prove a transaction's copy of the route is stable; a design can correctly freeze the map and still let a downstream stage re-derive the target from a live signal. 16.5 §16 is that failure at the CXL boundary; this is it at the memory map.

DV. All four are always-on and cheap. The second requires a directed test that attempts a commit with outstanding work — which is §37's second trace and does not occur by chance.

15. Wrong RTL — Remap While a Read Is Outstanding

// WRONG — the response path consults the CURRENT map instead of the one the
// request was accepted under.
assign resp_expected_target = lookup_target(resp_addr, active_region_q);
 
always_ff @(posedge clk)
  if (resp_valid && (resp_target != resp_expected_target))
    drop_or_flag_response();          // ← a correct response, thrown away

The sequence:

1. A read of address X is accepted. Epoch 7 binds X to target A.
2. The request reaches A. A begins servicing it.
3. Software commits a new configuration. Epoch 8 binds X to target B.
4. A's response returns, correctly, naming target A.
5. The response path recomputes the expected target from the CURRENT map: B.
6. A != B  ->  the response is dropped or flagged as an error.
7. The requester waits forever for a response that already arrived.

Three properties.

Both the memory and the transport behaved correctly. A served the read it was given; UCIe delivered the response intact. The only incorrect component is a lookup performed at the wrong time.

The symptom is a hang, and the cause is a configuration change — two events that nothing obviously connects. Without a first-cause record (14.5) the investigation starts at the timeout, several thousand cycles downstream.

And it is worse if the response is not dropped but accepted. A design that recomputes the target and uses the new one to route the returning data will hand A's data to whatever B's context expects. Silent corruption instead of a hang.

The correct form compares against the captured context, not the live map:

// ILLUSTRATIVE. The acceptance-time decision is the authority.
assign resp_ok = txn_q[resp_id].valid
              && (txn_q[resp_id].target    == resp_target)
              && (txn_q[resp_id].map_epoch == resp_epoch);

And note that with §14's quiesce guard, step 3 could not have happened — the commit would have been blocked while the read was outstanding. The captured context is the second line of defence, and a design should have both, because the guard protects against a remap and the capture protects against everything else that might disturb a route.

16. The Expansion Transaction

// ILLUSTRATIVE outstanding entry. Extends Module 12's transaction descriptor
// with the two facts an expansion path needs (12.4 Section 6 is the base).
typedef struct packed {
  logic                 valid;
  logic [REQ_ID_W-1:0]  req_id;      // the requester's identity — carried, not reassigned
  logic [ADDR_W-1:0]    address;
  logic                 is_write;
  logic [TARGET_W-1:0]  target;      // captured at acceptance
  logic [EPOCH_W-1:0]   map_epoch;   // and the configuration that chose it
  logic [REG_W-1:0]     region;      // for Section 13's guard
} expansion_txn_t;
 
expansion_txn_t txn_q [MAX_OUTSTANDING];

Architecture. 12.4 §6's descriptor plus three expansion-specific fields: the target, the epoch that chose it, and the region for the commit guard. Nothing else is added, because everything else the transaction needs is already Module 12's.

State. MAX_OUTSTANDING entries. This depth is a performance parameter, not a convenience — §22 derives it.

Cycle behaviour. Allocated at acceptance with the route decision written into it; freed at semantic completion. The three new fields are write-once.

Contract. The response path (§26), the commit guard (§13) and the stall classifier (§32) all read this entry. It is the single record of what was promised and to whom.

Failure. Omitting map_epoch and relying on target alone — which fails when a target is removed and later re-added, or when two epochs bind the same range to the same target for different reasons. The epoch is what makes "the same target" distinguishable across configurations.

DV. §14's fourth property, plus a scoreboard that predicts the target independently from the epoch-7 configuration (§35).

17. Admission Is a Conjunction Across All Four Planes

// ILLUSTRATIVE. One term per plane, plus the local resources that must hold
// the request. 11.2 Section 24 makes the general argument; the expansion-
// specific terms are the last two.
assign expansion_req_ready =
      region_valid                 // MAPPING  — exactly one owner (Section 10)
   && target_present               // CAPACITY — the device is enumerated and up
   && outstanding_space            // local    — an entry can be pre-allocated
   && ucie_transport_ready         // TRANSPORT— the object can be represented
   && target_admit_credit_q != '0; // SERVICE  — the remote side can take it

Architecture. Five terms, four planes, and each term is owned by a different mechanism. The value of naming them separately is diagnostic: a refusal to admit is immediately attributable.

State. target_admit_credit_q is the only stateful term — a per-target credit reflecting how much the remote memory side has advertised it can absorb. The other four are combinational reads of state owned elsewhere.

Cycle behaviour. Evaluated at acceptance. Sampled, and the result bound to the accepted transaction — not re-evaluated underneath work already accepted, which is 11.5 §9's rule.

Contract. Accepting a request commits every stage between the decoder and the remote media. Each stage can independently be the constraint, and none is a proxy for another.

Failure. §18. Also collapsing target_present and target_admit_credit_q into one signal, which merges "this device exists" with "this device can take work right now" — and those become false for entirely different reasons and on entirely different timescales.

DV. Force each term false in isolation and confirm no admission. Five directed tests, and they catch the "counted but not gated" case that no random regression finds.

18. Wrong RTL — Admission Derived From the Link

// WRONG — transport liveness treated as memory readiness.
assign expansion_req_ready = ucie_link_active;

Everything that can be true while ucie_link_active is asserted:

Three consequences, and the third is the worst.

Row 3 loses the write. A request admitted with no outstanding entry has no owner. For a read that is a hang; for a write, if the far side also cannot take it, the data is simply gone — accepted by the system and stored nowhere.

The link signal is the most locally available and the least informative. It answers one of five questions and is routinely mistaken for all five. This is the sixth chapter in this curriculum to make that point, which is a measure of how often the shortcut is written.

And the failure is load-dependent. At low load the other four terms are almost always true, so the shortcut works. It fails exactly when the system is busy — which is when the consequences are largest and the debugging is hardest.

19. Latency, Decomposed

15.2 established that residence is queue wait plus service. For an expanded access there are six services and five queues, and calling the total "remote latency" hides all of them.

Two readings.

Package traversal is one row of ten, and it is usually not the largest. The intuition that remote memory is slow "because of the wire" is almost always wrong. Queueing at the remote controller, and waiting for bank eligibility, typically dominate — which means the fix for a latency problem is usually a scheduling or concurrency fix, not a physical one.

And every added stage is a place a request can queue. Under load, the queueing terms grow and the service terms do not. So the gap between local and expanded latency widens with load, which is why a lightly-loaded benchmark understates the difference and a saturated one overstates it if the concurrency is wrong (§22).

20. A Worked Latency Decomposition

Illustrative numbers throughout — chosen to make the arithmetic legible, not to describe any product. No source is claimed for any figure in this section.

Compare against an illustrative local access of decode 3 + queue 6 + schedule 9 + media 18 + match 4 = 40 cycles.

Three readings, and the arithmetic is the point.

The expanded access is about 2.5× the local one (101 / 40 ≈ 2.53), not 10× — and not 1.1× either. Both exaggerations appear in practice.

Package traversal is 10 of 101 cycles — under 10%. Doubling the link's physical speed removes at most 5 cycles from a 101-cycle path, roughly 5%. The 22 cycles of remote controller queueing are more than four times as large, and they respond to concurrency and scheduling rather than to link improvements.

And the queueing terms are the load-sensitive ones. At higher load the 6 and 22 grow while the 18 and 5 do not. A decomposition measured at low load will mispredict behaviour at high load, which is why 15.5 §17's warmup discipline applies to memory measurement as much as to link measurement.

21. Capacity Is Not Bandwidth

Adding an expansion chiplet adds capacity. Whether it adds usable bandwidth is a separate question with a separate answer.

A system can double its capacity and measure exactly the same throughput. That is not a defect — it is the expected result when the binding constraint was never capacity.

Two consequences.

The bottleneck must be identified before the purchase. 15.5 §33's residual analysis names which resource is binding; if it names the link or the outstanding depth, more memory changes nothing.

And the outstanding table is the constraint most often overlooked. It is a host-side parameter, it is invisible in every product datasheet, and §22 shows that the expanded tier needs more of it than the local tier does.

22. The Concurrency a Longer Round Trip Demands

The formula, from 13.5 and 15.2, applied to a memory tier:

required_outstanding  ≈  target_throughput × round_trip_latency

Units, before substituting anything (15.1 §4's discipline):

target_throughput   [requests / cycle]
round_trip_latency  [cycles]
required_outstanding [requests]     — dimensionless product, correct

Worked, with §20's illustrative latencies.

Suppose the goal is one request per cycle sustained.

LOCAL     : 1 req/cycle × 40 cycles  = 40 outstanding requests
EXPANDED  : 1 req/cycle × 101 cycles = 101 outstanding requests

Three readings.

The expanded tier needs about 2.5× the outstanding depth for the same throughput — the same ratio as the latency, which is exactly what the formula says. A host sized at 40 entries can saturate local memory and will reach roughly 40/101 ≈ 40% of that rate on the expanded tier, with the media, the link and the remote controller all substantially idle.

And the symptom is indistinguishable from a slow device. Throughput is low, latency per request is normal, no queue is full, no error is reported. Only an outstanding-occupancy counter (§34) distinguishes "the memory is slow" from "we never asked it for enough work at once".

The table must therefore be sized for the slowest tier it serves, or partitioned so the expanded tier gets its own depth. Sizing it for local memory and hoping is the most common expansion performance bug there is.

23. Wrong Assumption — Capacity Purchased, Throughput Expected

CLAIM:   "We added an expansion chiplet with X GB/s of media bandwidth,
          so system memory bandwidth increases by X GB/s."
REALITY: usable increase = min( X,
                                link useful bandwidth to that target,
                                remote controller sustainable rate,
                                host outstanding depth ÷ round-trip latency,
                                workload demand for that address range )

The last row is the assumed case and is frequently the rarest. Establishing which row applies is 15.5's method, and §32 is the runtime instrument that answers it.

24. Placement — Two Tiers, One Map

A system with local and expanded memory has a placement question, and the hardware's job is not to answer it but to make it answerable.

Two consequences for the hardware.

The tier field in §8 is what makes per-tier accounting possible. Without it, counters aggregate across tiers and the resulting averages describe no real access.

And dependent-chase workloads are the ones tiering hurts most. Each access depends on the previous one, so concurrency cannot be increased and §22's remedy is unavailable. For those, the round trip is paid in full every time, and the only fix is placement.

25. Responses Return Out of Order

Two independent reasons, and a system must handle both.

The remote memory reorders (17.1 §17) because that is how it exploits bank parallelism.

And several targets complete independently, so responses from A and B interleave in an order that reflects neither issue order nor each device's internal order.

Response order carries no information. Identity does. 12.2 §14 establishes the general rule and 12.2 §16 shows the "use the oldest outstanding request" bug in full. What expansion adds is that the reordering is now across devices as well as within one, so even a design that correctly handles one device's reordering can be broken by two.

26. SVA — a Response Must Match Its Acceptance Context

// MANDATORY. Identity, target and epoch must all agree with what was captured.
property p_response_matches_acceptance_context;
  @(posedge clk) disable iff (!rst_n)
    resp_valid |-> (txn_q[resp_id].valid
                 && (txn_q[resp_id].target    == resp_target)
                 && (txn_q[resp_id].map_epoch == resp_epoch)
                 && (txn_q[resp_id].address   == resp_addr));
endproperty
a_response_matches_acceptance_context:
  assert property (p_response_matches_acceptance_context);
 
// Exactly one semantic completion per accepted request.
property p_one_completion_per_request;
  @(posedge clk) disable iff (!rst_n)
    (resp_valid && resp_fire) |=> !txn_q[$past(resp_id)].valid;
endproperty
a_one_completion_per_request: assert property (p_one_completion_per_request);
 
// Semantic state is not retired on a transport event (16.5 Section 12).
property p_no_retire_on_transport_event;
  @(posedge clk) disable iff (!rst_n)
    (ucie_tx_fire && !semantic_completion[IDX]) |=> $stable(txn_q[IDX].valid);
endproperty
a_no_retire_on_transport_event:
  assert property (p_no_retire_on_transport_event);

Architecture. Three properties: full-context matching, exactly-once completion, and no transport-triggered retirement.

Why the first checks four things. A response can be wrong in four independent ways — dead transaction, wrong device, stale configuration, wrong address — and with several targets and a mutable map, all four are reachable. With one device and a static map, only the first is.

Why the epoch term is the one that catches §15. Target and address can both match while the configuration differs, which is exactly the case where a route was recomputed. Without the epoch, that check silently passes.

DV. Inject each of the four violations separately. The stale-epoch case requires a commit to have occurred between issue and response, which §14's guard normally prevents — so it must be injected with the guard disabled, to prove the second line of defence works.

27. UCIe Recovery With Memory Requests Live

A transport recovery is a link event. (14.2 §4)

Two things the recovery legitimately changes.

The link epoch advances, so a pre-recovery response arriving afterwards can be identified (16.4 §27).

And the link may return degraded (14.4), which changes §20's latencies and therefore §22's required depth. Bounds derived from the previous configuration must be rescaled (16.5 §27) or a healthy expanded access will be declared failed.

28. Wrong Recovery — the Expansion Table Is Flushed

// WRONG — a transport event is treated as a reason to forget memory obligations.
always_ff @(posedge clk)
  if (ucie_recovery_entered)
    for (int i = 0; i < MAX_OUTSTANDING; i++)
      txn_q[i].valid <= 1'b0;        // every live memory transaction, gone

1. A read of X is outstanding against target A.
2. UCIe enters recovery. This code clears the entry.
3. The recovery completes. A had already serviced the read.
4. A's response returns, correctly, with its identity.
5. The response finds no live transaction.
6. -> the response is orphaned. The requester waits forever for data
     that arrived and was discarded at the boundary.

And for a write it is worse. The write may have landed at A, or may not have — and with the record destroyed, the system no longer knows which. There is nothing to retry and nothing to report.

Three properties.

The transport is blameless. Recovery succeeded, replay worked, the response crossed intact.

The symptom is remote from the cause. A timeout thousands of cycles after a recovery that reported success. 11.2 §18 is this bug for a single device; the expansion version is worse because with several targets only some transactions are affected, so the failure is partial and looks intermittent.

And it is the seventh appearance of this lifetime error in the curriculum — 14.3 §15, 15.2 §15, 11.5 §14, 16.3 §13, 16.4 §26, 16.5 §11, and here. Every time, a semantic obligation was released by a transport signal.

29. Temporary Unavailability Is Not Permanent Loss

// ILLUSTRATIVE per-target availability state. Two failure semantics, and
// merging them is Section 30's bug.
typedef enum logic [1:0] {
  TGT_USABLE      = 2'd0,   // enumerated, mapped, accepting
  TGT_UNAVAILABLE = 2'd1,   // temporarily not accepting — WAIT, do not fail
  TGT_LOST        = 2'd2    // permanently unreachable — explicit failure policy
} target_state_e;
 
target_state_e target_state_q [NUM_TARGETS];
logic [AGE_W-1:0] unavailable_age_q [NUM_TARGETS];   // saturating

Architecture. A three-state per-target register plus a saturating age. The age is what promotes UNAVAILABLE to LOST after a bound — the promotion is a policy decision that must be explicit rather than emergent.

State. NUM_TARGETS two-bit registers and NUM_TARGETS counters. Saturating, because an age that wraps reports a freshly-unavailable target at the moment it has been unavailable longest (13.4 §13).

Cycle behaviour. UNAVAILABLE is entered on any of the four causes and left when the target resumes; the age advances only while unavailable and clears on resumption.

Contract. The admission conjunction (§17) reads this; the failure policy reads this; and the outstanding transactions read it to decide whether to keep waiting. All three depend on the two states being distinguishable.

Failure. Collapsing to one bit. A design with target_ok alone must choose one behaviour for both cases, and either it fails transactions during an ordinary maintenance window or it waits forever for a device that has been removed.

DV. Cover both states, cover the promotion at the bound, and confirm outstanding transactions survive UNAVAILABLE and are explicitly failed on LOST.

30. Wrong Policy — Silently Rerouting an Accepted Write

// WRONG — an accepted write is re-routed to a different target because the
// original became unavailable.
always_comb
  tx_target = (target_state_q[txn_q[id].target] != TGT_USABLE)
            ? next_usable_target()          // ← a DIFFERENT device
            : txn_q[id].target;

Address X belongs to target A. The write is sent to target B.

Four consequences.

X now has two values. A holds the old one; B holds the new one at whatever offset X maps to on B. A subsequent read of X routes to A by the map and returns the stale value — the write is lost, silently.

And B has been corrupted. The offset the write landed on belongs to some other address in B's range. A second, unrelated address is now wrong, with no connection to the write that caused it.

The system reports success. The write was accepted, transported and acknowledged. Every layer's statistics are clean.

And it looks like a reasonable availability feature. "The target went away, so we used another one" is a sentence that sounds like resilience. It is only resilience if the architecture defines migration — with a cutover epoch, coordination of outstanding accesses, and a defined treatment of the old copy (§31). Absent that, rerouting is corruption with good intentions.

An accepted write is an obligation against a specific target. If that target cannot take it, the correct outcomes are to wait (TGT_UNAVAILABLE) or to fail it explicitly and report (TGT_LOST). Choosing a different device is not among them.

31. Migration, and Why It Is a Cutover Problem

Moving a range's contents from one target to another is a legitimate architecture and a large one. This chapter does not build it, and §30 is the reason it must not be improvised.

A correct migration must coordinate at least six things:

Two readings.

The commit machinery in §12 and the guard in §13 are the foundation of migration, not a substitute for it. They make the map change atomically and safely; they say nothing about the data having been moved.

And migration is where all four planes interact at once, which is why it is the hardest operation a tiered memory system performs, and why a design should implement it deliberately or not at all.

32. Stall Attribution — the Classifier

The chapter's most useful RTL. Composing 11.2's mapping, 17.1's memory-side behaviour and 15.5 §10's causal-priority discipline.

// ILLUSTRATIVE. ONE reason per cycle, chosen by causal priority. A cycle in
// which an expanded request could not be admitted has exactly one primary cause.
typedef enum logic [3:0] {
  EXP_ADMITTED     = 4'd0,   // not a stall
  EXP_NO_DEMAND    = 4'd1,   // nothing wanted expanded memory this cycle
  EXP_UNMAPPED     = 4'd2,   // no region owns the address
  EXP_TGT_LOST     = 4'd3,   // permanent (Section 29)
  EXP_TGT_UNAVAIL  = 4'd4,   // temporary (Section 29)
  EXP_NO_SLOT      = 4'd5,   // local outstanding table full  <- Section 22 lives here
  EXP_UCIE_BLOCKED = 4'd6,   // transport cannot represent it
  EXP_TGT_NOCREDIT = 4'd7,   // remote memory side cannot absorb it
  EXP_RECOVERING   = 4'd8,   // link in recovery
  EXP_REMAP_QUIESCE= 4'd9    // deliberately held for a commit (Section 14)
} exp_stall_e;
 
exp_stall_e stall_reason_d;
 
always_comb begin
  unique case (1'b1)
    expansion_req_fire                          : stall_reason_d = EXP_ADMITTED;
    !expansion_req_valid                        : stall_reason_d = EXP_NO_DEMAND;
    (region_match == '0)                        : stall_reason_d = EXP_UNMAPPED;
    (target_state_q[sel_target] == TGT_LOST)    : stall_reason_d = EXP_TGT_LOST;
    recovery_active                             : stall_reason_d = EXP_RECOVERING;
    remap_quiesce_active                        : stall_reason_d = EXP_REMAP_QUIESCE;
    (target_state_q[sel_target] == TGT_UNAVAILABLE)
                                                : stall_reason_d = EXP_TGT_UNAVAIL;
    !outstanding_space                          : stall_reason_d = EXP_NO_SLOT;
    !ucie_transport_ready                       : stall_reason_d = EXP_UCIE_BLOCKED;
    (target_admit_credit_q[sel_target] == '0)   : stall_reason_d = EXP_TGT_NOCREDIT;
    default                                     : stall_reason_d = EXP_ADMITTED;
  endcase
end
 
logic [63:0] stall_cycles_q [10];
always_ff @(posedge clk)
  if (rst_n) stall_cycles_q[stall_reason_d] <= stall_cycles_q[stall_reason_d] + 1'b1;

Architecture. A one-hot causal-priority classifier over the admission conjunction, plus one saturating counter per reason. Every cycle lands in exactly one bin, which is what makes the totals meaningful.

State. Ten 64-bit counters. Diagnostic only — nothing in the datapath reads them, which is why they may be wide and why they must survive events that reset everything else (14.5).

Cycle behaviour. unique case (1'b1) evaluates in order, so the priority encodes causality, not severity. Structural impossibilities come first (unmapped, lost), then deliberate holds (recovery, quiesce), then resources. EXP_NO_DEMAND is second so that idle cycles never inflate a resource bin — the single most common way a classifier lies.

Contract. The sum of all ten equals the elapsed cycle count. That conservation is what makes the residual meaningful (15.5 §12), and it is asserted rather than assumed.

Failure. Independent counters that each increment when their condition holds — which double-counts every cycle where two conditions are true, and all the percentages exceed 100% in a way nobody notices until they are added up.

DV. Assert one-hot classification and cycle conservation. Cover every reason as non-zero, because a reason that never fires is either impossible or untested and the difference matters.

33. Wrong Classification — "Memory Is Slow"

// WRONG — one bucket for everything.
always_ff @(posedge clk)
  if (expansion_req_valid && !expansion_req_fire)
    memory_slow_q <= memory_slow_q + 1'b1;

What this counter cannot distinguish, and what each would have demanded:

Three properties.

Two of the six rows are not problems at all. Maintenance and quiesce are correct behaviour. A single bucket reports them as memory slowness and invites an engineer to "fix" them, which in the maintenance case means starving refresh (17.1 §13) and losing data.

Two rows point at completely different hardware. Table depth is a host-side parameter; remote credit is a device-side one. The single bucket cannot tell you which team owns the problem, which is how weeks are spent.

And the counter is not wrong, merely useless. It counts real stall cycles accurately. Precision without attribution is not diagnosis — which is the whole argument for §32.

34. Observability

// Diagnostic only. Every counter's incrementing EVENT is defined, because an
// undefined event is a counter nobody can interpret.
logic [63:0] local_req_q;              // requests routed to the local tier
logic [63:0] expanded_req_q    [NUM_TARGETS];   // per target, not aggregated
logic [63:0] expanded_bytes_q  [NUM_TARGETS];
logic [63:0] outstanding_occ_sum_q;    // sum of occupancy each cycle -> mean
logic [63:0] outstanding_at_max_q;     // cycles the table was FULL   <- Section 22
logic [63:0] expanded_retry_q  [NUM_TARGETS];
logic [63:0] expanded_recovery_q;
logic [63:0] target_unavail_q  [NUM_TARGETS];   // cycles in TGT_UNAVAILABLE
logic [63:0] remap_quiesce_q;          // cycles held for a commit
logic [63:0] resp_age_max_q    [NUM_TARGETS];   // worst observed round trip

Three properties of the list.

Every counter is per-target where the question is per-target. Aggregating across targets produces averages that describe no real access — which is 15.3 §14's per-link argument, one layer up.

outstanding_at_max_q is the highest-value counter here. It converts §22 from an argument into a measurement, and it is one comparator and one counter.

And they must survive recovery and reconfiguration. A counter cleared by the event you are diagnosing has no evidentiary value (14.5).

35. The Expansion Scoreboard

// Verification-only. THREE models, three identities. This differs from 11.2's
// semantic memory scoreboard: that one models ONE device's memory contents;
// this one models a MULTI-TARGET MAP and must catch a request going to the
// wrong device (Section 11) and a route recomputed under a live request (Section 15).
class expansion_scoreboard;
 
  // ---- Layer 1: MAPPING model. Predicted independently, per epoch.
  typedef struct {
    bit [ADDR_W-1:0] base;
    bit [ADDR_W-1:0] limit;
    int              target;
    int              tier;
  } region_model_t;
  region_model_t regions [int][int];     // [epoch][region] — every epoch retained
  int active_epoch;
 
  function int predict_target(bit [ADDR_W-1:0] a, int epoch);
    foreach (regions[epoch][r])
      if (a >= regions[epoch][r].base && a <= regions[epoch][r].limit)
        return regions[epoch][r].target;
    return -1;                            // unmapped is a legal answer
  endfunction
 
  // ---- Layer 2: TRANSACTION model. What was promised, and under what context.
  typedef struct {
    bit              live;
    bit [ADDR_W-1:0] address;
    bit              is_write;
    int              accepted_target;     // what the DESIGN captured
    int              accepted_epoch;
    int              predicted_target;    // what the MODEL says it should be
    int              completions;         // MUST be <= 1
    bit              crossed_recovery;
  } txn_model_t;
  txn_model_t txns [int];                 // keyed by request identity
 
  // ---- Layer 3: MEMORY CONTENT model, PER TARGET.
  bit [DATA_W-1:0] mem [int][bit [ADDR_W-1:0]];   // [target][address]
 
  // ---- The check only this scoreboard can make (Section 11).
  function void check_route(int id);
    if (txns[id].accepted_target != txns[id].predicted_target)
      $error("MISROUTED id %0d addr %0h: design chose target %0d, epoch-%0d map says %0d",
             id, txns[id].address, txns[id].accepted_target,
             txns[id].accepted_epoch, txns[id].predicted_target);
  endfunction
 
  // ---- The check that catches Section 15.
  function void check_response_context(int id, int resp_target, int resp_epoch);
    if (resp_target != txns[id].accepted_target || resp_epoch != txns[id].accepted_epoch)
      $error("CONTEXT DRIFT id %0d: accepted (t%0d,e%0d), response (t%0d,e%0d)",
             id, txns[id].accepted_target, txns[id].accepted_epoch, resp_target, resp_epoch);
  endfunction
 
  // ---- The check that catches Section 28.
  function void check_recovery_survival(int id);
    if (txns[id].crossed_recovery && !txns[id].live && txns[id].completions == 0)
      $error("TRANSACTION %0d lost across recovery without completing", id);
  endfunction
 
endclass

Architecture. Three models keyed by epoch-and-region, by request identity, and by target-and-address.

Why every epoch is retained in regions[epoch][]. A response arriving after a commit must be checked against the map that was in force when its request was accepted, not the current one. Discarding old epochs makes §15 undetectable — the model would make exactly the mistake the design is being tested for.

Why mem is indexed by target as well as address. §11's misrouting writes correct data to the wrong device. A memory model keyed by address alone sees a consistent value and passes; only a per-target model shows that A's copy is stale and B's is polluted.

How this differs from 11.2 §30. That scoreboard models one device's contents and proves memory semantics. This one models the map across several devices and proves routing — and both are needed, because a system can route perfectly to a device that stores incorrectly, or store perfectly at a device that should never have received the request.

36. Flagship Trace 1 — a Remote Read

Illustrative. Address X in region 2, epoch 7, target A. Cycle numbers are illustrative.

Five readings.

Cycle 3 captures the route, and nothing recomputes it. Cycles 66–67 check against that capture, not against the live map. §15's bug is doing the cycle-67 lookup from active_region_q.

Cycle 13 retires the transport object while the transaction lives for another 55 cycles. §28's bug clears the transaction here.

Cycles 22 to 34 are remote controller queueing — the largest single term. Not the link, not the media. §20's decomposition made visible.

The region count is non-zero from cycle 4 to cycle 68, and during that window §14's guard blocks any commit affecting region 2. That window is what §37 is about.

And the response is matched by identity at cycle 67, not by order. Had a second read to target B been issued at cycle 5 and completed at cycle 40, it would have returned first — which carries no information at all (§25).

37. Flagship Trace 2 — a Remap Attempt Under Live Traffic

Correct design first.

Then the wrong design, which commits immediately at cycle 13:

Three readings.

The guard and the capture are two independent defences and a design should have both. The guard prevents the situation; the capture survives it if the guard is bypassed, mis-sized, or defeated by a debug override.

Cycle 51 shows the real cost of reconfiguration: 38 cycles of waiting for three transactions to drain. That cost is measurable (remap_quiesce_q, §34) and is frequently mistaken for memory slowness (§33).

And the wrong design's failure is invisible at cycle 13. Nothing goes wrong for 53 cycles. The commit succeeds, the counters look normal, and the fault surfaces as an unrelated hang.

38. Flagship Trace 3 — Recovery With Requests Live

Five readings, and this is the chapter's strongest section.

Cycle 25: three transactions live, three preserved. The map is untouched, the counts are untouched, the captured contexts are untouched. §28's bug clears all three here.

Cycle 33 is the fact designs forget. The remote device did not stop. It completed the read during the recovery, and its response was waiting. A host that had given up would have discarded a completed access.

Cycle 40: the link returns narrower. Latency and bandwidth change; not one semantic field does (16.5 §26).

Cycle 41 is the omitted step. §20's decomposition just got longer, so §22's required depth just got larger and every derived bound is now wrong. Without rescaling, the very next access is declared failed on a link that recovered successfully.

And cycle 48 carries two epochs — link epoch 5 and map epoch 7 — which are independent and must both be checked. A design that has only one epoch concept cannot express this.

39. Coverage

covergroup cg_memory_expansion @(posedge clk);
  option.per_instance = 1;
 
  // --- Routing (Sections 8-11).
  cp_route : coverpoint req_route {
    bins local = {0}; bins expanded = {1}; bins unmapped = {2};
  }
  cp_target : coverpoint sel_target { bins each[] = {[0:NUM_TARGETS-1]}; }
  cp_boundary : coverpoint addr_boundary_class {
    bins base    = {0};
    bins base_p1 = {1};
    bins mid     = {2};
    bins limit_m1= {3};
    bins limit   = {4};
    bins outside = {5};              // Section 9's directed set
  }
  cp_overlap_injected : coverpoint overlap_configuration_attempted;
 
  // --- Concurrency (Section 22).
  cp_outstanding : coverpoint outstanding_occupancy {
    bins one  = {1};
    bins few  = {[2:MAX_OUTSTANDING/2]};
    bins deep = {[MAX_OUTSTANDING/2+1:MAX_OUTSTANDING-1]};
    bins full = {MAX_OUTSTANDING};   // Section 22 lives here
  }
  cp_reorder : coverpoint response_reorder_distance {
    bins in_order = {0}; bins near = {[1:3]}; bins far = {[4:$]};
  }
  cp_cross_target_reorder : coverpoint responses_interleaved_across_targets;
 
  // --- Reconfiguration (Sections 12-15).
  cp_remap : coverpoint remap_attempt_context {
    bins idle             = {0};
    bins blocked_by_outst = {1};     // Section 37's first trace
    bins forced_with_outst= {2};     // guard disabled — Section 15
  }
  cp_epoch_mismatch : coverpoint response_epoch_mismatch_observed;
 
  // --- Availability (Sections 29, 30).
  cp_target_state : coverpoint target_state_q_ut {
    bins usable = {0}; bins unavailable = {1}; bins lost = {2};
  }
  cp_unavail_promotion : coverpoint unavailable_promoted_to_lost;
 
  // --- Transport composition (Sections 27, 28).
  cp_link_event : coverpoint link_event_during_outstanding {
    bins none = {0}; bins retry = {1}; bins recovery = {2}; bins degraded = {3};
  }
 
  // --- Attribution (Section 32) — every reason must be reachable.
  cp_stall_reason : coverpoint stall_reason_d { bins each[] = {[0:9]}; }
 
  // --- Crosses that carry the information.
  x_remap_outstanding : cross cp_remap, cp_outstanding;
  x_link_target       : cross cp_link_event, cp_target_state;
  x_reorder_targets   : cross cp_reorder, cp_target;
  x_stall_outstanding : cross cp_stall_reason, cp_outstanding;
endcovergroup

Six bins worth calling out:

cp_outstanding.full. §22's constraint, observed. A regression that never fills the table has not tested the most common expansion performance limit.

cp_remap.blocked_by_outst and .forced_with_outst. The first proves the guard works; the second, run with the guard disabled, proves the captured context is a real second defence rather than dead code.

cp_cross_target_reorder. Responses from two targets interleaving — §25's expansion-specific case, and one that a single-target environment cannot produce.

cp_target_state.lost and cp_unavail_promotion. §29's distinction, exercised. Most environments only ever produce usable.

cp_boundary — all six. §9's off-by-one set, which is cheap and catches the errors that survive everything else.

And cp_stall_reason — every one of the ten. A reason that never fires is either unreachable or untested, and the difference is exactly what a coverage report is for.

40. Debug Taxonomy

Row 4 is the one that reaches management. Capacity was added and nothing got faster is a purchasing question answered by a stall histogram, and §32 answers it in one screenshot — which is the practical argument for building the classifier before it is needed.

41. Debug Checklist

1. Which address, and which region claims it? (§9)
2. Does exactly one region claim it? (§10)
3. Which target did the region name, and which tier? (§8)
4. What target and map epoch were captured at acceptance? (§16)
5. What is the current map epoch? (§12)
6. Did a commit occur between acceptance and response? (§37)
7. Was the commit guarded by outstanding counts, or forced? (§13, §14)
8. Which request identity, and is its entry still live? (§26)
9. Is the path local or expanded? (§6)
10. What is the outstanding occupancy, and how often is it at maximum? (§22, §34)
11. What does the stall histogram say — which of the ten reasons dominates? (§32)
12. What is the target's availability state, and for how long? (§29)
13. Is the remote device in a maintenance window? (17.1 §11)
14. Did a UCIe retry or recovery occur while the request was live? (§27)
15. Did the link return degraded, and were bounds rescaled? (§27, §38)
16. Did the response's target, epoch, identity and address all match the capture? (§26)
17. Was any accepted request rerouted to a different target? (§30)
18. When was the transaction retired — at semantic completion or at a transport event? (§26)
19. Which of the three scoreboard layers diverged first? (§35)

42. SVA Inventory

// The conservation property that ties the counters to reality (Section 13).
property p_region_count_matches_population(int r);
  @(posedge clk) disable iff (!rst_n)
    (region_outstanding_q[r] == count_live_txns_for_region(r));
endproperty
a_region_count_matches_population:
  assert property (p_region_count_matches_population(REG_UT));
 
// The classifier is one-hot and conserves cycles (Section 32).
property p_stall_cycles_conserved;
  @(posedge clk) disable iff (!rst_n)
    (sum_of(stall_cycles_q) == elapsed_cycles_q);
endproperty
a_stall_cycles_conserved: assert property (p_stall_cycles_conserved);
 
// An accepted request's target is never changed by anything (Section 30).
property p_no_reroute_of_accepted_request;
  @(posedge clk) disable iff (!rst_n)
    (txn_q[IDX].valid && tx_fire_for(IDX)) |-> (tx_target == txn_q[IDX].target);
endproperty
a_no_reroute_of_accepted_request:
  assert property (p_no_reroute_of_accepted_request);
 
// An unavailable target does not fail live transactions (Section 29).
property p_unavailable_does_not_fail_transactions;
  @(posedge clk) disable iff (!rst_n)
    (target_state_q[TGT_UT] == TGT_UNAVAILABLE) |=> $stable(txn_q[IDX].valid);
endproperty
a_unavailable_does_not_fail_transactions:
  assert property (p_unavailable_does_not_fail_transactions);

The full inventory for this chapter, gathering what the sections above established:

43. Common Misconceptions

"Expansion memory is just more local memory." It is a different tier with a different latency, a different bandwidth ceiling, different failure modes, and the ability to be reconfigured and to become temporarily unreachable. Every one of those changes how a system must be designed around it (§7).

"More capacity automatically adds bandwidth." Usable bandwidth is the minimum of the media, the link, the remote controller, the host's outstanding depth divided by the round trip, and the workload's demand. Capacity is the only one of the five that the purchase guarantees (§21, §23).

"UCIe being active means the memory can accept a request." The link answers one of five admission questions. The region may be unmapped, the target unenumerated, the local table full, or the remote side unable to absorb — and a request admitted with no outstanding entry has no owner at all (§17, §18).

"The destination can be recomputed after a retry or a recovery." It cannot. The route decided at acceptance is the authority, and recomputing it against a map that may have changed either drops a correct response or delivers data to the wrong context (§15, §16).

"Memory responses return in request order." The remote device reorders to exploit bank parallelism, and several targets complete independently. Response order carries no information; identity does (§25).

"Recovery may clear outstanding reads." It may not. The remote device never stopped and may already have completed the access — so clearing the record orphans a response that arrives correctly, and for a write it destroys the only evidence of whether the data landed (§27, §28).

"A remap can safely affect requests already in flight." It cannot, which is why a commit is guarded by outstanding counts and why the transaction carries its own captured target and epoch as a second defence (§13, §14, §37).

"Remote memory latency is mostly wire delay." In the illustrative decomposition, package traversal is under 10% of the round trip and remote controller queueing is more than twice as large. Doubling the link's speed addresses the smallest term (§19, §20).

"A timeout means the remote write never happened." A timeout means no response arrived. The write may have landed. Re-issuing without knowing is how one write becomes two (12.4 §16).

"If a target becomes unavailable, send the request somewhere else." An accepted write is an obligation against a specific target. Rerouting corrupts two addresses at once and reports success. Wait, or fail explicitly — those are the two correct outcomes (§30).

"Clean transport proves correct memory routing." Every object can cross once with clean CRC while the request was routed to the wrong device by an overlapping region. Transport correctness is a strictly weaker claim (§11, §35).

44. Understanding Check

45. Summary and What Comes Next

Memory expansion makes remote capacity part of one memory map without pretending it is local. Remote keeps its meaning inside the package.

Four planes, four lifetimes. Capacity, mapping, transport and service are independently valid, fail differently, and must each have their own term in the admission conjunction — because the link signal answers one of the five questions and is routinely mistaken for all of them.

The map is committed atomically, guarded by outstanding counts, and captured per transaction. Two independent defences, because the guard prevents the situation and the capture survives it being bypassed.

An overlap resolved by a priority encoder is a detectable configuration error converted into an undetectable data error — corrupting two addresses at once across two perfectly healthy devices, with clean CRC throughout.

Capacity, bandwidth and concurrency are three separate purchases. The expanded tier needs outstanding depth proportional to its longer round trip, and a table sized for local memory reaches a fraction of the rate with everything downstream idle.

A recovery rebuilds a link and changes nothing semantic — and the far side never stopped, so it may already have completed the access whose record a flush would destroy.

Temporary and permanent loss are different semantics, and rerouting an accepted write to a different target is corruption with good intentions.

And no stall is ever "memory slow". Ten reasons, one per cycle, conserving the cycle count — two of which are correct behaviour and two of which name different teams.

This chapter moved capacity toward the system. The next inverts the question: rather than moving all the data to the compute die across that longer round trip, move the operation toward the memory — and find out what that does to ownership, fairness, and exactly-once semantics.

• 17.3 — Near-Memory Compute — compute placed on the memory chiplet.

Browse the full path on the UCIe tutorials index.