Shared Memory Across Chiplets

Module 15 quantified the interconnect. This module asks what it is increasingly built to carry: memory that several dies treat as one.

1. The One-Sentence Model

Shared memory means multiple agents agree on one semantic memory state, even though the physical storage, the caches, the requests and the transports are all distributed.

The word carrying the weight is agree. Two chiplets that can both address location X have shared addressability. Shared memory needs four separate agreements, and addressability is only the first.

2. What This Chapter Owns

Module 11 already taught CXL memory expansion in depth, and this chapter is not that chapter again.

11.2's failures are all visible inside one agent. An overlapping window, a wrong interleave, a cleared outstanding table — each is a defect you can find by inspecting one die.

16.1's characteristic failure cannot be found that way at all. Chiplet A maps X to memory die 0; chiplet B maps X to memory die 1. Both maps are internally consistent, both pass every assertion 11.2 would write, and the system is corrupt (§13). That failure only exists when there is more than one configurator, which is the situation Module 16 is about.

Specifically new here: the four agreements; physical location against system address against latest value; the region map that every die must agree on, with the cross-die disagreement failure; requested against active maps committed atomically with N participants; per-region outstanding counts as a commit guard; atomicity requiring one serialisation point across dies and why local atomicity is not atomicity; and coherence against consistency, which are different guarantees and are routinely conflated.

3. Sourcing

4. Four Agreements, Not One

Three observations.

They are independent, and each can hold while another fails. A system can have perfect address agreement, perfect routing, a well-defined consistency model — and still return stale data because agreement 4 is missing. That is precisely §18's scenario, and it is the whole reason 16.2 exists.

Only the first is about addressing, and it is the one people mean when they say "shared memory". The other three are where the engineering is.

And agreement 4 is optional in a way the others are not. A system with no private caches needs agreements 1 to 3 and not 4 (§17). Adding a cache anywhere adds a requirement that did not exist before — which is a genuinely useful thing to know when scoping a chiplet.

5. Three Different Answers to "Where Is X?"

The distinction the rest of Module 16 rests on.

The third row moves at transaction rate and the first two move at configuration rate. That is why they need different machinery: the map is committed state changed rarely and carefully (§14), while the location of the latest value is tracked per line and changes constantly (16.2).

Physical location tells you where the bits are stored. It does not tell you where the newest bits are. Conflating those two is the single most consequential misunderstanding in shared-memory design, and §18 is what it costs.

6. The Semantic Memory Model

Three quantities per address, and they can all differ:

memory_value[X]  = the value physically stored in memory
latest_value[X]  = the architecturally newest value
owner[X]         = the agent responsible for the latest value, if not memory

In a system with no caches, memory_value == latest_value always and owner is meaningless. That is why §18's cacheless case is genuinely simpler rather than merely smaller.

With caches, all three diverge routinely:

initial:    memory_value = 0,  latest_value = 0,  owner = memory
A caches X and writes 1:
            memory_value = 0,  latest_value = 1,  owner = A

Memory now holds a value that no correct read may return. It is not corrupt, it is not stale in the sense of a bug — it is simply not the newest value, and a system that reads it directly is wrong (§19).

This triple is the backbone of the scoreboard in §33 and of the coherence scoreboard in 16.2 §29. A verification environment that models only memory_value cannot detect a coherence failure at all, because from memory's point of view nothing went wrong.

7. The Shared-Memory Structure

Read the figure for the two boxes on the right. The memory chiplet stores the bits. The home knows where the newest bits are — and those are different jobs that a monolithic mental model collapses into one.

And note that every requester crosses a UCIe boundary. In a monolithic SoC the coherent fabric is on-die and its latency is tens of cycles; here every request, every probe and every response is a die crossing. 16.3 §14 is what that does to the timing of ownership changes.

8. The Region Map

// ILLUSTRATIVE shared-memory routing state. NOT a CXL, CHI or UCIe decoder
// format — no such format is published in any source I could reach (Section 3).
typedef struct packed {
  logic                 valid;   // this region is configured
  logic [ADDR_W-1:0]    base;    // first system address in the region
  logic [ADDR_W-1:0]    limit;   // last  system address, INCLUSIVE (Section 9)
  logic [TARGET_W-1:0]  home;    // which agent is the home for this region
} shared_region_t;
 
shared_region_t region_q [NUM_REGIONS];

Architecture. A small table mapping ranges of the system address space onto homes. It is the mechanism by which "we share memory" becomes a routing decision — and it is the artefact every die must agree about.

State. Configuration lifetime — programmed at boot or on hot-plug, changed rarely, and read on every request. That asymmetry is why §14 separates a requested copy from an active one: writes are rare and expensive to get wrong; reads are constant.

Cycle behaviour. Read combinationally on the request path. Written only through the commit of §15, never in place.

Contract, and it is the whole chapter. Every die that can issue a request into this address space must hold a map that produces the same home for the same address. Nothing in this table enforces that — it is an integration property across N independently programmed tables, and §13 is what its violation looks like.

Failure. §11 (ambiguous home within one table), §13 (disagreement across tables), §17 (map changed under a live request).

DV. Assert §10's one-hot property per table. Then, at the system level, compare every die's resolved home for a swept address set — which is the only check that catches §13 and cannot be written inside any single die.

9. Boundary Convention, Stated Once

// Inclusive limit: the region covers [base, limit].
assign region_match[i] = region_q[i].valid
                      && (addr >= region_q[i].base)
                      && (addr <= region_q[i].limit);

The convention must be stated and then held, because the two plausible conventions differ by exactly one address:

11.2 §8 developed the off-by-one this produces inside one decoder. What 16.1 adds is that the convention is now an interface between teams: if one die's map is programmed inclusive and another's exclusive from the same configuration data, the two dies disagree about exactly one address at every region boundary — and that single address is where the system will fail, rarely and reproducibly.

A size-based encoding avoids the ambiguity entirely by making the convention unrepresentable:

// An alternative that cannot be misread: base plus size.
assign region_match[i] = region_q[i].valid
                      && (addr >= region_q[i].base)
                      && ((addr - region_q[i].base) < region_q[i].size);

10. SVA — At Most One Home

// MANDATORY within one die's map. Zero matches is legal (Section 11);
// two matches is configuration corruption.
property p_at_most_one_home;
  @(posedge clk) disable iff (!rst_n)
    $onehot0(region_match);
endproperty
a_at_most_one_home: assert property (p_at_most_one_home);
 
// And a request may only be issued for an address that HAS a home.
property p_issue_requires_home;
  @(posedge clk) disable iff (!rst_n)
    request_issued |-> $onehot(region_match);
endproperty
a_issue_requires_home: assert property (p_issue_requires_home);

Why zero matches is legal and two are not. An unmapped address is a defined condition — it means this die has no route for that address, and the correct behaviour is a defined error response rather than a guess. Two matches means two homes claim the same address, and whichever the priority encoder picks is arbitrary: the same address could route differently after a resynthesis. That is corruption, not a policy choice.

What these properties cannot catch. They constrain one die's table. They say nothing about whether another die resolves the same address the same way — and §13 is exactly a case where every die satisfies both properties and the system is broken.

11. Wrong Integration — Dies That Disagree

The characteristic failure of shared memory across chiplets, and it exists in no single-agent system.

CHIPLET A's map                    CHIPLET B's map
  region 0: [0x0000_0000 ..          region 0: [0x0000_0000 ..
             0x7FFF_FFFF] → mem0                0x7FFF_FFFF] → mem1
                                                              ^^^^
 
Both maps are internally consistent.
Both satisfy Section 10's one-hot property.
Both dies pass every assertion that can be written inside a die.

What happens. A writes X = 1; the write lands in mem0. B reads X; the read is served from mem1, which has never seen the write. B reads a value that is not stale — it is from a different physical location entirely.

Four properties make this the worst failure class in the chapter.

No die is wrong. There is no defective RTL to find. The defect is in the relationship between two configurations, and every conventional verification artefact — per-die assertions, per-die scoreboards, per-die formal — is blind to it by construction.

The symptom is indistinguishable from a coherence failure. B reads an old value. That is exactly what §19's stale-cache scenario looks like, and it is what a debugger will assume. The distinguishing test is to check whether the value B reads was ever written by anybody — a coherence failure returns a genuinely old value, while this returns a value from an address that was never the target.

It can be asymmetric and therefore intermittent-looking. If only one region of one die's map is wrong, most addresses work. A workload touching mostly-correct regions passes; one touching the wrong region fails, which reads as a data-dependent bug.

And it is a build problem, not a runtime one. The maps came from a configuration source — firmware, a device tree, a fusing step. The fix is in the configuration flow, and the hardware's job is to make the disagreement detectable.

Which is the actionable conclusion:

A multi-die shared-memory system needs a system-level check that every die resolves a swept address set to the same home. That check cannot live inside a die, and without it this failure is found by a workload rather than by verification.

12. Requested Against Active Maps

// ILLUSTRATIVE two-copy map. The same discipline Chapter 14.2 Section 24 used
// for link configuration and 14.4 Section 11 for the lane map, applied to
// address routing on a live system.
shared_region_t requested_region_q [NUM_REGIONS];   // staged; NO consumer
shared_region_t active_region_q    [NUM_REGIONS];   // what routing USES
 
logic [CFG_VER_W-1:0] active_map_version_q;
 
// Staging is free — nothing routes from the requested copy.
always_ff @(posedge clk)
  if (map_stage_en) requested_region_q[stage_idx] <= stage_value;
 
// Committing is not. One event, one edge, every consumer together.
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    active_map_version_q <= '0;
  end else if (map_commit) begin
    active_region_q      <= requested_region_q;      // whole table, one edge
    active_map_version_q <= active_map_version_q + CFG_VER_W'(1);
  end
end

Architecture. Two copies with asymmetric exposure — the pattern this curriculum has now used three times, and for the same reason each time. Software may reprogram the requested copy freely; nothing observes it until the commit.

State. requested_region_q has programming-episode lifetime. active_region_q and active_map_version_q have map-epoch lifetime — and the version counter exists so §13's property is expressible.

Cycle behaviour. The whole table commits on one edge. A per-entry commit is a map that is briefly half-old and half-new, which routes different addresses under different maps — the address-space version of 14.4 §16's one-lane-at-a-time bug.

Contract. No routing logic may read requested_region_q. And the version counter is local control and verification state — no on-wire version field is claimed, and none is published (§3).

Failure. §15's mid-flight change; and committing while requests are outstanding against an affected region (§16).

DV. Drive the requested copy with a completely different map throughout live traffic and assert no request's target changes — which proves nothing routes from the staged copy.

13. SVA — a Request Keeps One Map Version

// MANDATORY. Once a request is accepted, its destination is fixed. Software
// reprogramming the map must not redirect work already in flight.
property p_request_keeps_its_map_version;
  @(posedge clk) disable iff (!rst_n)
    request_completes |-> (req_map_version_q[resp_id] == req_map_version_at_accept[resp_id]);
endproperty
a_request_keeps_its_map_version: assert property (p_request_keeps_its_map_version);
 
// The active map is stable except at a commit.
property p_active_map_stable;
  @(posedge clk) disable iff (!rst_n)
    !map_commit |=> $stable(active_region_q);
endproperty
a_active_map_stable: assert property (p_active_map_stable);
 
// And the version advances with the map, one for one.
property p_version_tracks_commit;
  @(posedge clk) disable iff (!rst_n)
    (active_map_version_q != $past(active_map_version_q)) |-> $past(map_commit);
endproperty
a_version_tracks_commit: assert property (p_version_tracks_commit);

Architecture. Three properties: a request's routing is immutable once accepted, the map changes only at a commit, and the version is bound to the commit.

Why the first is the interesting one. It is not a statement about the map — it is a statement about a request's lifetime. A request captures its map version at acceptance, and that captured version must still describe how the request was routed when it completes. 15.2 §14 made the same argument for timestamps: state captured at acceptance must survive to completion or it describes nothing.

Contract. req_map_version_q is per-request instrumentation, allocated with the transaction (12.4) and released with it.

DV. These need a map commit while requests are outstanding — which is the case §16 is designed to prevent, so the test must either force it or verify the guard blocks it.

14. Wrong RTL — a Map Update That Takes Effect Immediately

// WRONG — routing reads the live map, and software writes it in place.
always_ff @(posedge clk)
  if (sw_write_en) region_q[sw_idx] <= sw_value;      // takes effect at once
 
assign target = resolve_home(region_q, addr);          // reads it every cycle

The failure, at transaction granularity. A read request is a multi-phase operation: the request is issued, and later a response returns and is matched. Suppose the map changes between those phases.

Different phases of one transaction consult different maps. The request went to mem0 and the completion logic believes it went to mem1 — so a response from mem0 matches nothing, and a retry (if any) would be sent to a device that never saw the request.

Three properties.

It is a race with a wide window. The window is the whole transaction latency — in 15.2 §32's worked case, around 96 ns. That is enormous by hardware standards, so a map update under load will hit it.

The symptom is a lost or unmatched response, which points at the response-matching logic (12.2) rather than at the map. The response matching is correct; the premise it was given changed.

And it is exactly 14.2 §25's configuration-commit argument in the address domain. The same discipline applies: stage, validate, commit atomically, and do not let a consumer see the staged value.

15. Region Outstanding Counts as a Commit Guard

// ILLUSTRATIVE per-region outstanding count. The guard that makes a map
// commit safe: do not change a region's routing while work depends on it.
logic [COUNT_W-1:0] region_outstanding_q [NUM_REGIONS];
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    for (int r = 0; r < NUM_REGIONS; r++) region_outstanding_q[r] <= '0;
  end else begin
    for (int r = 0; r < NUM_REGIONS; r++) begin
      unique case ({req_accepted_in_region[r], req_retired_in_region[r]})
        2'b10  : region_outstanding_q[r] <= region_outstanding_q[r] + COUNT_W'(1);
        2'b01  : region_outstanding_q[r] <= region_outstanding_q[r] - COUNT_W'(1);
        default: region_outstanding_q[r] <= region_outstanding_q[r];  // incl. 2'b11
      endcase
    end
  end
end
 
// The commit condition: affected regions must be quiescent.
assign map_commit = map_validated
                 && peer_maps_agreed                 // Section 11's system check
                 && affected_regions_quiescent
                 && !map_commit_done_q;
 
assign affected_regions_quiescent =
    &( ~region_affected_by_commit | region_is_empty );   // per-region reduction

Architecture. One counter per region plus a quiescence conjunction. Per region rather than global is the design decision: a commit touching one region need not wait for traffic to unrelated regions to drain, which matters because a global quiesce on a busy system may never complete.

State. Per region, per map epoch. Note the 2'b11 arm falls into default — accept and retire in the same cycle is net-zero, the same discipline as 13.1 §9 and 14.3 §7.

Cycle behaviour. Incremented on semantic acceptance and decremented on semantic retirement — not on transport events. A count decremented when a response arrives rather than when it is consumed would allow a commit while a response is still in a queue, which is 15.2 §15's retirement-point error with a correctness consequence rather than a measurement one.

Contract. peer_maps_agreed is the system-level check of §11, and it is deliberately a term here: a commit that is locally valid and disagrees with a peer is the failure this chapter exists to prevent.

Failure. Omitting the quiescence term, which is §14. Or making it global, which can deadlock a commit on a permanently busy system.

DV. Attempt a commit with a region non-empty and confirm it is blocked; drain and confirm it proceeds. Then confirm an unaffected region's traffic does not block it.

16. SVA — the Map Is Stable While Requests Depend On It

// The property that makes Section 14 impossible.
property p_no_commit_with_outstanding;
  @(posedge clk) disable iff (!rst_n)
    map_commit |-> affected_regions_quiescent;
endproperty
a_no_commit_with_outstanding: assert property (p_no_commit_with_outstanding);
 
// Per region, expressed directly.
generate for (genvar r = 0; r < NUM_REGIONS; r++) begin : g_region_stable
  a_region_stable_while_busy: assert property (@(posedge clk) disable iff (!rst_n)
    (region_outstanding_q[r] != '0) |=> $stable(active_region_q[r]));
end endgenerate

Architecture. A commit-time guard and a per-region stability claim. The generate form is the stronger of the two, because it holds regardless of how affected_regions_quiescent was computed — a bug in that expression cannot hide behind the first property.

DV. Needs a commit attempted while a region is busy. A regression that only reprograms maps on an idle system satisfies both vacuously.

17. Shared Memory Without Caches

Start with the simple case, because it isolates what caching adds.

With no private caches anywhere, memory_value == latest_value always (§6). Correctness then needs only agreements 1 to 3:

What is not required: any notion of ownership, sharers, invalidation, probes or transient state. Memory is the single point of truth, so there is nothing to keep consistent with it.

This is a real and useful design point, not merely a teaching device. An accelerator chiplet that streams through a buffer and never caches it needs none of 16.2's machinery. The cost of coherence is paid only where caching exists, and knowing that is how a chiplet architect scopes the problem.

18. Shared Memory With Private Caches

Now the hard case, in four steps:

1. memory_value[X] = 0,  latest_value[X] = 0,  owner = memory
2. Chiplet A reads X, caches it, writes 1:
     memory_value[X] = 0        ← memory is NOT updated
     latest_value[X] = 1
     owner[X]        = A
3. Chiplet B reads X.
4. If B's read is served from memory: B sees 0.

B has read a value that no correct execution permits. Nothing was corrupted; no packet was lost; every CRC passed; the address map was correct on both dies. The system is simply wrong, because agreement 4 was missing.

Three things this establishes for the rest of the module.

A correct read of X can no longer be answered by memory alone. Something must know that A owns it — which is the home of Figure 1, and 16.2 is what it must track.

The failure is invisible to every mechanism in Modules 12 to 15. Transport delivered correctly; the transaction completed; the throughput was good. 16.2 §31 develops why a packet-level scoreboard passes while the system is corrupt.

And the window is unbounded. A's dirty copy can sit in its cache indefinitely. There is no timeout after which memory becomes correct again — only an explicit protocol action can move the latest value.

19. Atomicity Needs One Serialisation Point

A read-modify-write is atomic only if no other agent can interleave between the read and the write. Across dies that requires a point through which all conflicting accesses pass.

Chiplet A                          Chiplet B
  read X   → 5                       read X   → 5
  compute  5 + 1 = 6                 compute  5 + 1 = 6
  write X  = 6                       write X  = 6
 
Final: X = 6.  Two increments were performed. One was lost.

Both agents behaved correctly in isolation. The lost update is a property of the pair, and no local mechanism can prevent it.

20. Wrong Design — Local Atomicity

// WRONG at system scope. Each die blocks its OWN competing accesses to the
// line, and neither blocks the other die.
always_ff @(posedge clk) begin
  if (atomic_start && !local_lock_q[line]) local_lock_q[line] <= 1'b1;
  if (atomic_done)                          local_lock_q[line] <= 1'b0;
end

Why it looks correct. Within chiplet A, two cores contending for X are correctly serialised. Every test on die A passes. Single-die verification cannot distinguish this from a correct design.

Why it is not. A's lock is invisible to B. Both dies acquire their own lock simultaneously, both read 5, both write 6. The lock did exactly what it was built to do and the system still lost an update.

Three properties.

Its correctness is a function of the number of dies, which is an unusual and dangerous property. A design validated on a single-die configuration is not validated for the two-die one, and the code is identical.

It scales in the wrong direction. More chiplets means more concurrent holders of independent locks, so the probability of a lost update rises with the thing chiplet architecture exists to increase.

And the symptom is a lost update, which is silent. No error, no timeout, no assertion. The only detection is a scoreboard that models the expected value (§33) — and this is precisely why §6's semantic model is the backbone rather than a nicety.

A lock is a serialisation point only for the agents that consult it. Atomicity across dies requires a point every participant reaches — which is the home, and which is why the home exists.

21. Illustrative Serialisation State

// ILLUSTRATIVE line-granular serialisation at a common point. This is a
// TEACHING microarchitecture, NOT a recommended implementation of full
// coherence — Chapter 16.2 builds the real mechanism.
typedef struct packed {
  logic                valid;    // a serialised operation is in progress
  logic [LINE_W-1:0]   line;
  logic [AGENT_W-1:0]  owner;    // which agent holds it
} atomic_lock_t;
 
atomic_lock_t lock_q [NUM_LOCKS];
 
// A conflicting request must wait, not proceed in parallel.
assign line_busy = |( lock_valid_vec & line_match_vec );

Architecture. Line-granular exclusion at the home, so every participant's request passes through it. That single placement is the whole difference from §20.

State. Per in-progress atomic operation, at the serialisation point rather than at the requester.

Contract. Every conflicting access must route through this point. If any die has a path that bypasses it — a private fast path, a cached copy served locally — the guarantee is void, which is the same requirement §20 violated.

Failure — and it is why this is explicitly not a coherence implementation. A global lock per line serialises all access to that line, including concurrent readers who could safely proceed together. That is correct and unusably slow, and 16.2's ownership model exists precisely to allow shared readers while still serialising writers.

DV. Two dies contending for the same line, with the check that exactly one increment per operation is observed by the scoreboard.

22. Outstanding Requests Per Target

// ILLUSTRATIVE per-target outstanding accounting. Shared memory creates
// several independent remote paths, and each has its own capacity.
logic [COUNT_W-1:0] outstanding_q [NUM_TARGETS];
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    for (int t = 0; t < NUM_TARGETS; t++) outstanding_q[t] <= '0;
  end else begin
    for (int t = 0; t < NUM_TARGETS; t++)
      unique case ({sem_accepted_by_target[t], sem_retired_by_target[t]})
        2'b10  : outstanding_q[t] <= outstanding_q[t] + COUNT_W'(1);
        2'b01  : outstanding_q[t] <= outstanding_q[t] - COUNT_W'(1);
        default: outstanding_q[t] <= outstanding_q[t];
      endcase
  end
end
 
assign may_issue_to[t] = (outstanding_q[t] < COUNT_W'(MAX_PER_TARGET));

Architecture. Per target, not global. A single global count cannot express that one memory chiplet is saturated while another is idle, which is 15.3 §18's imbalance question appearing as a correctness limit rather than a performance one.

State. Per target, per transaction set. Both edges are semantic events, per §15's argument.

Contract. MAX_PER_TARGET must not exceed the tracking capacity actually allocated for that target. Exceeding it means a response can arrive with no entry to match it — 14.2 §34's unmatched-response failure, reached by over-issue rather than by a cleared table.

Failure. A global counter that permits MAX requests all to one target.

DV. Saturate one target and confirm issue to it stops while issue to others continues.

23. SVA — Per-Target Bound and Live-Request Matching

// No target receives more obligations than its tracking permits.
generate for (genvar t = 0; t < NUM_TARGETS; t++) begin : g_target_bound
  a_target_bounded: assert property (@(posedge clk) disable iff (!rst_n)
    (outstanding_q[t] <= COUNT_W'(MAX_PER_TARGET)));
end endgenerate
 
// A response must correspond to a live request — Module 12's rule, at
// multi-target scope.
property p_response_has_live_request;
  @(posedge clk) disable iff (!rst_n)
    response_accepted |-> req_table_q[resp_id].valid;
endproperty
a_response_has_live_request: assert property (p_response_has_live_request);
 
// And the response must come from the target the request was sent to.
property p_response_from_expected_target;
  @(posedge clk) disable iff (!rst_n)
    response_accepted |-> (resp_source == req_table_q[resp_id].target);
endproperty
a_response_from_expected_target: assert property (p_response_from_expected_target);

Why the third property is specific to this chapter. With one target it is trivially true. With several, a response carrying a valid identity from the wrong target is possible — through an identity-space collision across targets, or a misrouted response — and it produces data delivered against a request that went somewhere else. 16.2 does not catch this either, because the data is well-formed and the coherence state is untouched. It is a routing failure with a semantic consequence.

DV. Requires concurrent traffic to at least two targets with overlapping identity values, which is the configuration that makes the collision reachable.

24. Response Order Is Not Request Order

With multiple targets this stops being a subtlety and becomes routine.

A issues:  R1 → mem0   (slow: a DRAM access)
           R2 → mem1   (fast: an SRAM-backed device)
 
Responses arrive: R2, then R1.

Nothing is wrong. Different targets have different service times, different queue depths, and traverse different links with independent congestion (15.3 §29).

So response matching must be by identity, never by arrival position — 12.2 established this and it is stated again here because the multi-target case makes reordering the common case rather than the exception. A design that happens to work with one target because responses return in order will fail immediately with two.

And it interacts with §26. Out-of-order responses are not the same as a relaxed memory model — the transaction layer may reorder freely while the architecture still guarantees a particular visibility order. Conflating them leads to designs that either over-constrain the transport or under-constrain the software contract.

25. Coherence Is Not Consistency

The distinction is mandatory, and the two words are routinely swapped.

A worked example of a consistency failure with perfect coherence:

initially X = 0, Y = 0
 
Chiplet A:  write X = 1        Chiplet B:  read Y  → 1
            write Y = 1                    read X  → 0
 
Every read returned a value some agent wrote. Coherence per location
is intact — X's and Y's individual histories are consistent.
But B saw Y's new value and X's old one, which a strong model forbids.

Nothing here is a coherence bug. Each location behaved correctly on its own. The ordering between locations is a different guarantee, and it is not supplied by the coherence protocol.

26. Coherent Does Not Mean Sequentially Consistent

Explicitly rejected, because it is the most common inference from §25.

A system can be fully coherent and implement a relaxed memory model. Coherence guarantees agreement about each location; it says nothing about the order in which operations to different locations become visible.

Three consequences for a chiplet design.

Adding a coherence protocol does not give you sequential consistency, and a team that assumes it will ship software that breaks on reordering. The consistency model is a separate contract that must be stated.

Relaxed models exist because they are cheaper, and in a chiplet system the saving is larger than on a monolithic die: enforcing a strong order across dies means serialising operations that traverse independent links with independent congestion. The cost of strong ordering scales with the number of dies.

And the enforcement points are architectural. Where a relaxed model requires ordering — at a barrier, a fence, a release — something must actually wait, and in a chiplet system that wait spans a die boundary. Which is a latency the software model must be designed against (15.2 §26).

27. Recovery With Shared-Memory Requests Outstanding

14.2 established the general rule; here is what it means for the address plane specifically.

The row that matters is the first. A UCIe recovery re-establishes a link; it does not invalidate the address space. A design that reloads or clears the region map on recovery has coupled two unrelated lifetimes — and the consequence is that requests issued before the recovery and completing after it may resolve against a different map, which is §14's failure triggered by a link event.

And the second-order point: if a recovery is accompanied by a reconfiguration that changes the map (14.4 changes lanes, not addresses), then §15's quiescence guard applies. Recovery is not a licence to skip it.

28. The Shared-Memory Scoreboard

// Verification-only reference model. Not synthesisable.
//
// Note the deliberate scope: this models ADDRESS and VALUE semantics.
// It does NOT model sharers or coherence state — that is 16.2's scoreboard.
class shared_memory_scoreboard;
 
  // Section 6's semantic model.
  bit [DATA_W-1:0] memory_value [bit [ADDR_W-1:0]];
  bit [DATA_W-1:0] latest_value [bit [ADDR_W-1:0]];
  int              owner        [bit [ADDR_W-1:0]];   // -1 = memory
 
  // Per-die resolved routing, for Section 11's cross-die check.
  int              die_home [int][bit [ADDR_W-1:0]];
 
  typedef struct {
    int requester, target, op, map_version;
    bit [ADDR_W-1:0] addr;
    bit outstanding;
  } req_t;
  req_t reqs[int];
 
  // ---- Check 1: EVERY die resolves an address to the SAME home.
  //      The check that cannot live inside a die (Section 11).
  function void check_map_agreement(bit [ADDR_W-1:0] addr);
    int first = -1;
    foreach (die_home[d]) begin
      if (!die_home[d].exists(addr)) continue;
      if (first == -1) first = die_home[d][addr];
      else if (die_home[d][addr] != first)
        $error("MAP DISAGREEMENT at %0h: a die says %0d, another says %0d",
               addr, first, die_home[d][addr]);
    end
  endfunction
 
  // ---- Check 2: a read returns the LATEST value, not the memory value.
  //      This is what detects Section 18's stale read.
  function void check_read(bit [ADDR_W-1:0] addr, bit [DATA_W-1:0] got);
    if (got !== latest_value[addr])
      $error("STALE READ at %0h: got %0h, latest is %0h (memory holds %0h, owner %0d)",
             addr, got, latest_value[addr], memory_value[addr], owner[addr]);
  endfunction
 
  // ---- Check 3: an atomic increment is not lost (Section 20).
  function void check_atomic(bit [ADDR_W-1:0] addr, int expected_ops);
    if (latest_value[addr] !== bit'(expected_ops))
      $error("LOST UPDATE at %0h: %0d operations applied, value is %0h",
             addr, expected_ops, latest_value[addr]);
  endfunction
 
  // ---- Check 4: a request's map version is stable across its lifetime.
  function void check_map_version(int id, int version_now);
    if (reqs[id].map_version != version_now)
      $error("MAP CHANGED UNDER REQUEST %0d: accepted at v%0d, completing at v%0d",
             id, reqs[id].map_version, version_now);
  endfunction
 
  // ---- Check 5: a response came from the target the request went to.
  function void check_response_source(int id, int src);
    if (reqs[id].target != src)
      $error("WRONG TARGET for request %0d: sent to %0d, response from %0d",
             id, reqs[id].target, src);
  endfunction
 
endclass

Architecture. Five checks over two models: a semantic memory model (memory / latest / owner) and a routing model (per-die resolved home).

Check 1 is the one that has no in-die equivalent. It requires the testbench to hold every die's resolved map and compare them, which is the only way to catch §11. A verification plan without it will not find that failure.

Check 2 compares against latest_value, not memory_value — and that single choice is what makes the scoreboard capable of detecting a coherence failure at all. A scoreboard modelling only memory would report §18's stale read as correct, because memory really does hold 0.

Check 3 is the lost-update detector. It requires the testbench to count the operations it issued, which is a modest constraint and the only way to see §20.

Scope note. This model deliberately does not track sharers, permissions or transient state. Those belong to 16.2 §29's scoreboard, and keeping the two separate is what lets a failure be attributed to addressing rather than to coherence.

29. Coverage

covergroup cg_shared_memory @(posedge clk);
  option.per_instance = 1;
 
  // --- Every home must be reached, and unmapped must be exercised.
  cp_home : coverpoint resolved_home {
    bins each_target[] = {[0:NUM_TARGETS-1]};
    bins unmapped      = {HOME_NONE};
  }
 
  // --- Address boundaries — where Section 9's convention bites.
  cp_addr_position : coverpoint addr_position_in_region {
    bins first        = {0};
    bins interior     = {1};
    bins last         = {2};
    bins just_outside = {3};        // base-1 and limit+1
  }
  cp_adjacent_regions : coverpoint addr_crosses_region_boundary;
 
  // --- Multi-die access patterns. The point of the chapter.
  cp_access_pattern : coverpoint access_pattern {
    bins one_die_only      = {0};
    bins two_dies_same_addr= {1};   // the interesting case
    bins two_dies_diff_addr= {2};
    bins three_dies_same   = {3};
  }
 
  // --- Map reconfiguration.
  cp_map_update : coverpoint map_update_condition {
    bins while_idle        = {0};
    bins attempted_busy    = {1};   // must be BLOCKED by Section 15
    bins after_quiesce     = {2};
  }
  cp_map_disagreement_injected : coverpoint injected_map_mismatch;   // Section 11
 
  // --- Atomicity.
  cp_atomic_contention : coverpoint concurrent_atomic_agents {
    bins one = {1}; bins two = {2}; bins many = {[3:$]};
  }
 
  // --- Ordering and recovery.
  cp_response_order : coverpoint response_vs_request_order {
    bins in_order = {0}; bins reordered = {1};
  }
  cp_recovery_with_outstanding : coverpoint recovery_while_requests_live;
 
  // --- Cache presence — the boundary into 16.2.
  cp_caching : coverpoint caching_enabled;
 
  // --- Crosses that carry the information.
  x_dies_addr      : cross cp_access_pattern, cp_addr_position;
  x_caching_pattern: cross cp_caching, cp_access_pattern;   // Section 18's stale read
  x_map_atomic     : cross cp_map_update, cp_atomic_contention;
endcovergroup

Five bins whose value is being non-zero:

cp_access_pattern.two_dies_same_addr. The whole chapter is about several dies sharing an address. A regression where each die uses its own addresses exercises none of it.

cp_map_disagreement_injected. §11's failure must be injected — it never arises spontaneously, and without it the scoreboard's cross-die check is never validated.

cp_map_update.attempted_busy. A commit attempted while a region is busy, which must be blocked by §15's guard — and is the only way §16's properties are exercised.

cp_atomic_contention.two and .many. §20's lost update needs genuine concurrent contention.

x_caching_pattern with caching enabled and two dies on one address. §18's stale read — the case that motivates the next chapter.

And one bin whose value should be zero in a clean run: any scoreboard STALE READ when caching is disabled. With no caches, memory is the latest value by construction, so a stale read in that configuration is an addressing bug rather than a coherence one.

30. Flagship Trace — Two Chiplets, One Address

Illustrative. A and B both access X, which maps to memory chiplet M0. No caches yet — §31 adds them.

Five readings.

Cycle 5 is where correctness is decided. M0 serialises A's write before B's read. Had they been served by two independent paths, B could have read 0 after A's write completed — §19's lost-update structure applied to a write/read pair.

Memory value and latest value are equal throughout, because there are no caches. That is what makes this the easy case, and §31 breaks it.

Cycles 1–10: the outstanding count rises to 2 and returns to 0, incremented at semantic acceptance and decremented at consumption (§15).

Cycle 11–12: the map commit waits for quiescence. A commit at cycle 3 would have been §14's bug — B's request accepted under v1 and its response resolved under v2.

And the Owner column never changes. With no caches, memory owns everything permanently. The moment a cache appears, that column becomes the most important one in the table — which is §31.

31. The Same Trace With a Cache

Illustrative. Identical addressing; A now caches X and writes it locally.

Cycle 2 is the whole of Module 16 in one row. Memory holds 0; the architecturally newest value is 1; and the agent responsible for it is A rather than memory. Three quantities that were identical in §30 have now diverged and will stay diverged until a protocol action moves the value.

Cycle 5 is the failure, and note what did not go wrong. The address map was correct on both dies. The request routed correctly. The transport delivered it. The response matched. Every mechanism this chapter built worked perfectly, and B read a value it must not see.

Shared addressability got the request to the right place. It could not get it to the right value, because the right value was somewhere else entirely.

That is agreement 4, and it needs its own chapter.

32. Debug Taxonomy

The seventh and eighth rows are the boundary of this chapter. A stale read with correct addressing is not an addressing bug — and the discriminating question is whether the returned value was ever written by anyone. A value nobody wrote points at addressing or corruption; a genuinely old value points at coherence.

33. Debug Checklist

1. Which system address? And which region does it fall in on each die?
2. Which home does each die resolve it to? If they differ, stop — that is §11.
3. Which map version was active when the request was accepted? And at completion (§13)?
4. Was a map commit attempted while the region had outstanding work? (§15).
5. Is the request still outstanding, and against which target? (§22).
6. Did the response come from the target the request was sent to? (§23).
7. Did the response match by identity rather than by arrival order? (§24).
8. Is caching enabled anywhere in the path? If not, a stale read is an addressing bug.
9. Was the value read one that any agent ever wrote? If not, it is routing; if yes, coherence.
10. Does memory hold the latest value, or does some cache? (§6).
11. Was an atomic operation involved, and did every participant pass one serialisation point? (§20).
12. Is this a coherence question or a consistency question? One location, or ordering between locations (§25)?
13. Did a link recovery occur, and did the region map and outstanding state survive it? (§27).
14. Does the shared-memory scoreboard agree — and specifically its cross-die map check? (§28).

34. Common Misconceptions

"A shared address space means shared-memory correctness." Addressability is one of four agreements. Routing, a consistency contract and — where caches exist — coherence are all separately required, and a system can satisfy the first and fail on any of the others (§4).

"Physical memory always holds the newest value." With any private cache anywhere, memory routinely holds a value that no correct read may return. Memory stores the bits; it does not track which bits are newest (§5, §6, §31).

"If UCIe delivers the request, the memory model is correct." Transport delivering an object correctly says nothing about whether the value returned is architecturally permitted. Every mechanism in Modules 12 to 15 can work perfectly while the answer is wrong (§31).

"Atomic means locally atomic on each chiplet." A lock serialises only the agents that consult it. Two dies each holding their own lock both read the old value and both write, and one update is silently lost — and the failure probability rises with the die count (§20).

"Coherence and consistency are the same thing." Coherence is agreement about one location; consistency is the ordering of operations across locations. A system can be perfectly coherent and still let a reader observe writes to two locations in an order a strong model forbids (§25).

"A coherent system must be sequentially consistent." It need not be, and relaxed models exist because strong ordering is expensive — more so across dies, where enforcing it means waiting across a die boundary (§26).

"Memory maps may be updated while traffic runs." A transaction spans phases; if the map changes between them, different phases resolve different targets. Stage, validate, commit atomically, and only when the affected regions are quiescent (§14, §15).

"Response order always matches request order." With several targets of different speeds on independent links, reordering is the normal case rather than the exception. Match by identity (§24).

"Link recovery can clear outstanding memory requests." The address space is not transport state. A recovery re-establishes a link and leaves the map, the versions, the outstanding counts and the tracking entries intact (§27).

"Two chiplets using the same address are guaranteed to reach the same physical target." Only if their maps agree — and each map can be internally consistent while the pair is not. That failure exists in no single-agent system and cannot be found by any per-die check (§11).

35. Understanding Check

36. Summary and What Comes Next

Shared memory means multiple agents agree on one semantic memory state, and agreement has four parts: addressing, routing, ordering and — where caches exist — coherence. Only the first is about addresses.

Three different answers to "where is X" — physical location, system address, and the location of the latest value. The first two change at configuration rate and are committed state; the third changes on every write and needs entirely different machinery.

The region map is the artefact every die must agree about, and its characteristic failure has no single-die equivalent: two internally consistent maps that disagree with each other. Every per-die assertion passes and the system is corrupt — so the check must be at system level, comparing every die's resolved home across a swept address set.

Stage, validate, commit atomically, and only when the affected regions are quiescent. A map that changes under a live transaction routes its phases to different targets, and the window is the whole transaction latency.

Atomicity needs one serialisation point that every participant reaches. A per-die lock is correct within a die and worthless across dies, its correctness depends on the die count, and its failure — a lost update — is completely silent.

Coherence and consistency are different guarantees. One location against ordering across locations; and coherent does not imply sequentially consistent, because relaxed models are cheaper and the saving grows with die count.

And with a cache anywhere, memory routinely holds a value no correct read may return. In §31's trace the address map was right on both dies, the request routed correctly, the transport delivered it and the response matched — and the reader still got the wrong value.

Shared addressing tells every chiplet what location X means. Private caches create the harder question: when several dies hold copies of X, which copy is legal to use and which agent owns the newest data?

• 16.2 — Cache Coherency in Chiplets — ownership, sharers, transient state, and the directory that has to know all three.

Browse the full path on the UCIe tutorials index.