AMD EPYC Case Study

Twenty-five modules built a framework for reading die-to-die architecture. This module turns it outward — onto a shipping product whose vendor documents it in detail.

1. The One-Sentence Model

AMD EPYC is a hub-and-spoke chiplet architecture: compute dies arranged around a single central I/O die, each connected to it by its own dedicated link — and that central die, not any link, owns coherence and the path to memory and I/O.

Two consequences make this worth a chapter. The topology means every compute die is one hop from everything it needs, which is a deliberate and unusual choice (25.7 §10). And it means the hub is a shared resource — which is a different set of design problems from a mesh.

2. What This Chapter Owns

22.2 covered AMD's relationship with the standard. This chapter covers the architecture itself — a different subject, and a more useful one:

The topology, what it buys, and what it costs (§5–§8), read from AMD's own documentation rather than characterised, and audited boundary by boundary.

What the hub owns that no link can (§9) — the answer to 23.3's layer question, in a real product.

What the spoke boundary must guarantee (§10–§12), applying the framework from Modules 21–25 to a documented boundary — including why not building a CCD-to-CCD mesh is itself a defensible decision.

The identity mistake a hub design is most likely to ship (§15–§16) — routing a response on the wire it arrived over, which is correct until the physical and logical views separate.

And why a proprietary fabric is the better teacher here (§17) — because it is documented at the architectural level, which a UCIe deployment currently is not.

3. Sourcing

4. Claim-vs-Evidence

Four readings.

Rows 1–6 are unusually strong for a product architecture. This is vendor documentation about the vendor's own product, which is the best available evidence — and it is far better than what 22.1 and 22.2 could establish about UCIe deployment.

Row 4 is the architecturally decisive one. "The IOD maintains cache coherency" answers 23.3's layer question in a real product: coherence is owned by a die, not by a link.

Row 7 is deliberately hedged (§3). I cite the figure with its series and its link type, and I do not repurpose it as a CCD-to-IOD number — which is the sort of quiet substitution 23.1 §13's conditions exist to prevent.

And the last row is the one to keep empty. This is a chiplet product with a named proprietary fabric, and saying so plainly is more useful than implying otherwise.

5. The Topology

Four things to read.

Every CCD edge terminates at the IOD and nowhere else. There is no CCD-to-CCD link in this documented arrangement — so CCD-to-CCD traffic is two hops through the hub, and that is a deliberate choice rather than an omission.

Memory and I/O also terminate at the hub, which is why the IOD can own coherency: it sits on the path to everything that needs to be coherent.

The fan-out is real. Up to 12 CCDs, each with its own GMI connection (§4 row 6) — twelve dedicated links into one die, which is a substantial physical and arbitration problem concentrated in one place.

And the topology is replication, not functional split. Every CCD is the same kind of thing. That is the opposite of 26.2's arrangement, and §18 is that comparison.

6. Why Hub-and-Spoke

Three readings.

Latency uniformity is the property being bought. In a mesh, a core's memory latency depends on where its die sits; in hub-and-spoke, every CCD is one hop from the hub — so software sees a much more uniform machine, which matters enormously for scheduling and NUMA modelling (22.4 §21).

One link per compute die is a real simplification. A CCD needs one boundary contract, not several — and that is 24.3 §4's cost of a die boundary, paid once per die instead of once per neighbour pair.

And row 7 is the scaling argument. Adding compute means adding a spoke, which does not change any existing CCD's interface — so a CCD can be reused across products with different core counts, which is 24.3 §8's reuse dimension realised concretely.

7. The Cost of the Hub

Hub-and-spoke is not free, and naming the costs is what makes this analysis rather than description.

And the last row is the elegant part of the arrangement. The hub is exactly the die that benefits least from a leading process — memory controllers, PCIe, CXL — while the CCDs are exactly the dies that benefit most. So the topology and the process-affinity argument point the same way, which is a genuinely good architectural fit rather than a coincidence.

8. Auditing the Partition

Run 25.7 §6's scorecard on each boundary rather than on the architecture as a whole. A partition is only as good as its worst cut.

Four readings.

Row 1 is the cut that carries all the risk, and it is the one the architecture repeats twelve times. It is justified by process affinity — but it is also the boundary with the strictest contract in the system (§10) and the one paid on every miss. A good partition can still have an expensive cut; what makes it good is that the cut is expensive for a reason.

Rows 2 and 3 are the least controversial cuts in chiplet design. Memory and I/O on a mature node, next to each other, on the die that owns the path to both — this is 25.2 §5's process-affinity argument with nothing working against it, and the same cut appears in 26.2 §7's I/O tile for the same reason.

Row 4 is the decision that is easiest to miss because it is an absence. Not cutting is also a partition choice. Declining to build a CCD-to-CCD mesh means cross-compute traffic pays two hops — and §12 is whether that is affordable.

And the scorecard's verdict is that this partition is internally consistent. Every boundary either sits where process affinity wants it, or does not exist. That consistency, not any single number, is what makes it a good architecture (24.3 §8).

9. What the Hub Owns That No Link Can

AMD's documentation states the IOD maintains cache coherency (§4 row 4). That single fact settles 23.3's layer question in a shipping product.

Three readings, and this is the chapter's strongest teaching point.

The line between rows 4 and 5 is 23.3 §5's boundary, drawn in a real product. Everything above it is a die's responsibility; everything below it is a link's.

Which means the question "could UCIe replace Infinity Fabric?" is malformed (23.3 §8) — and this architecture shows why concretely. A standard link could in principle occupy row 5. It could not occupy rows 1–4, because those belong to a die that sits at the centre of the graph.

And the interesting question is the one 23.3 §8 poses: could the CCD-to-IOD transport be a standard link while the fabric's semantics stay where they are? That is architecturally coherent, it is Level E inference, and no source I could reach says AMD does it (22.2 §4).

10. What the Spoke Boundary Must Guarantee

Apply the framework. A CCD-to-hub boundary carrying coherent traffic has the strictest possible contract (22.4 §9).

Two readings.

Rows 2 and 3 are why this boundary is harder than a bulk-data boundary. A dropped tensor read is a retry; a dropped coherent request is a stalled core, and the symptom is a hang with a perfectly healthy link.

And row 6 is the one that would corrupt rather than hang. A coherent write issued twice because a transport retry allocated a new semantic identity leaves two agents disagreeing about a cacheline (25.5 §10) — the exact failure coherence exists to prevent.

None of these is a claim about how AMD implements it (§3). They are what any transport carrying coherent traffic must provide, and they are what you would ask about at this boundary.

11. The Dependency-Depth Question

25.7 §10 says bandwidth does not decide a partition — dependency depth does. Apply it here.

Three readings, and this is the analysis the framework enables.

The die crossing is paid on every miss that leaves the CCD, twice per round trip (22.4 §7) — so its cost scales with miss rate, not with bandwidth.

Row 3 is why a large per-CCD cache matters so much in this arrangement. A dependent chain of misses cannot hide the crossing at all; the only defence is not to miss. That makes the CCD's own cache hierarchy part of the interconnect argument, not separate from it.

And row 1 is 21.5 §26's bandwidth-delay product: sustaining a memory rate across the boundary requires enough outstanding misses to cover the round trip — which is a design requirement on the CCD, set by the boundary's latency.

12. The Two-Hop Path

§8 row 4 noted that CCD-to-CCD traffic is not cut — it is routed. That is the architecture's most-questioned decision, and it deserves an answer rather than a shrug.

Three readings, and the answer is not "latency does not matter".

Row 4 is the structural argument, and it is the strongest one. The hub already owns coherence (§9) — so a coherent CCD-to-CCD transfer has to be visible to the hub anyway. A direct link that bypassed it would still need the hub's involvement to stay coherent, which means the second hop is largely paid regardless. The mesh saves less than the hop count suggests.

Rows 2, 3 and 5 are the cost the mesh imposes on every CCD. Eleven links and eleven contracts per die, changing with die count — which destroys the property 26.1 §6 identified as the reuse story: one CCD design, unchanged across SKUs. The mesh buys a hop and sells the modularity.

And the honest cost is row 6. All cross-CCD bandwidth converges on the hub (§7). Workloads with heavy cross-socket-style sharing between compute dies are exactly where this topology is weakest — and naming that, rather than defending the architecture, is what reading it critically means.

13. Illustrative — a Hub Spoke Contract

// ILLUSTRATIVE ONLY. What a spoke into a coherence hub carries. Note that the
// SEMANTIC fields belong to the fabric and the TRANSPORT fields belong to the
// link — the separation 23.3 §5 argues for, and 26.1 §8 observes in a product.
typedef struct packed {
  // SEMANTIC — fabric-owned, stable across every transport attempt
  logic [SEM_W-1:0]   sem_id;
  logic [GEN_W-1:0]   generation;      // which use of that id (25.4 §12)
  logic [3:0]         coh_op;          // request / snoop / response / writeback
  logic [ADDR_W-1:0]  addr_token;      // opaque; not an architectural address
  logic [DOM_W-1:0]   order_domain;
  logic               needs_completion;
  // TOPOLOGY — the hub routes; the spoke does not
  logic [SPOKE_W-1:0] src_spoke;       // which CCD it came from
  logic [EPOCH_W-1:0] cfg_epoch;
  // TRANSPORT — link-owned, and NEVER written by fabric logic
  logic [ATT_W-1:0]   attempt;
} spoke_txn_t;

Architecture. One record with three clearly separated field groups, so the ownership boundary of §9 is visible in the type itself.

State. One entry per outstanding transaction in the hub's tracking table (§14).

Event. Constructed at acceptance on the spoke; attempt increments on retransmission. No branch anywhere writes sem_id or generation on a transport event (25.5 §11).

Contract. src_spoke exists because the hub must route the response back, and it must do so on logical identity rather than on which physical link the request arrived over (25.7 §16) — those coincide today and need not tomorrow.

Failure. If order_domain or the routing decision were derived from transport state, a link event would silently change where a coherent message goes (23.3 §13) — delivered perfectly, to the wrong place, with every integrity check passing.

DV/debug. All three groups belong in a trace (21.7 §16). A trace showing attempt = 3, one sem_id, and one completion is the system working — and being able to say that stops an engineer reporting a duplicate.

14. Illustrative — the Hub's Fan-In Problem

// ILLUSTRATIVE ONLY (§13). Twelve spokes converging on one die is a real
// arbitration and fairness problem concentrated in one place (§7). This is the
// per-spoke accounting that makes starvation observable rather than inferred.
localparam int N_SPOKE = 12;
 
logic [15:0] outstanding_q   [N_SPOKE];
logic [15:0] max_outstanding_q[N_SPOKE];
logic [47:0] served_q        [N_SPOKE];
logic [31:0] wait_age_q      [N_SPOKE];   // cycles WAITING while requesting
logic [31:0] wait_age_max_q  [N_SPOKE];
 
always_ff @(posedge clk or negedge rst_n) begin
  for (int s = 0; s < N_SPOKE; s++) begin
    if (!rst_n || diag_clear) begin
      outstanding_q[s]    <= '0;
      max_outstanding_q[s]<= '0;
      served_q[s]         <= '0;
      wait_age_q[s]       <= '0;
      wait_age_max_q[s]   <= '0;
    end else begin
      // Outstanding as ONE signed next-state expression per spoke, so an
      // accept and a completion in the same cycle net correctly (25.6 §10).
      outstanding_q[s] <= outstanding_q[s]
                        + 16'(accept_fire[s]) - 16'(complete_fire[s]);
      if (outstanding_q[s] > max_outstanding_q[s])
        max_outstanding_q[s] <= outstanding_q[s];
 
      if (grant_fire[s] && xfer_fire) served_q[s] <= served_q[s] + 48'd1;
 
      // Age only while REQUESTING and reset on actual SERVICE — not on grant.
      // A spoke granted into a blocked path was not served (21.5 §29).
      if (grant_fire[s] && xfer_fire)      wait_age_q[s] <= '0;
      else if (request[s] && (wait_age_q[s] != '1)) begin
        wait_age_q[s] <= wait_age_q[s] + 32'd1;
        if ((wait_age_q[s] + 32'd1) > wait_age_max_q[s])
          wait_age_max_q[s] <= wait_age_q[s] + 32'd1;
      end
    end
  end
end
 
// MANDATORY. English: no spoke waits longer than the architecture's fairness
// bound while continuously requesting. Illustrative architectural property —
// the bound must come from the design's own fairness contract, not be guessed
// (21.6 §29). Catches starvation of one CCD by eleven others.
a_no_spoke_starvation: assert property (
  @(posedge clk) disable iff (!rst_n)
    request[0] |-> ##[1:MAX_FAIR_WAIT] (grant_fire[0] && xfer_fire)
);

Architecture. Per-spoke outstanding, high-water, service count and waiting age — five numbers per spoke that make hub fairness measurable.

State. 5 × N_SPOKE counters. At twelve spokes that is modest and it localises a fairness problem to one spoke in a single read.

Event. wait_age_q resets on actual service, not on grant — 21.5 §29: a grant into a blocked path is not service, and resetting on grant hides starvation more effectively the busier the hub is.

Contract. MAX_FAIR_WAIT must be derived from the design's fairness contract. A guessed bound gets raised each time the assertion fires until it cannot fail (21.6 §29).

Failure. Ageing on grant rather than service is the classic error. And a shared age across spokes cannot answer which CCD is starving — which is the only question worth asking here.

DV/debug. wait_age_max per spoke is the fairness evidence. One spoke with a much larger maximum than the others is a fairness bug localised to one arbiter input — and in silicon it is a register read rather than an experiment.

15. Wrong RTL — Routing the Response on the Arrival Link

§13's Failure lens named this hazard. It is worth demonstrating, because it is the mistake a hub design is most likely to ship: it is correct today, it passes every test, and it breaks the first time the physical and logical views stop coinciding.

// WRONG. ILLUSTRATIVE (§13). The hub returns each response over the physical
// link the request ARRIVED on. Written by someone who observed — correctly —
// that spoke 3 always means CCD 3, and turned an observation into a mechanism.
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    for (int e = 0; e < N_ENTRY; e++) rt_link_q[e] <= '0;
  end else begin
    // BUG 1: the routing key is the PHYSICAL arrival link, captured at accept.
    if (accept_fire)
      rt_link_q[alloc_idx] <= arrival_link_id;
 
    // BUG 2: nothing re-derives this if the spoke-to-die mapping changes —
    //        there is no epoch, so a remap cannot invalidate a live entry.
    if (resp_fire)
      resp_link_sel <= rt_link_q[resp_idx];
  end
end

Architecture. One stored field per outstanding entry: which wire it came in on.

State. rt_link_q — physical link identity, captured once and never revalidated.

Event. Captured at accept, consumed at response. Nothing in between can correct it.

Contract. It violates the separation §13's type exists to enforce: the response must be routed on the requester's logical identity, not on a physical attribute of the attempt (25.7 §16).

Failure — the timeline. A spoke is retrained and the platform remaps which physical link serves which logical die, while two requests are outstanding.

Root cause. A physical attribute was used as a logical identity. arrival_link_id and "which CCD this is" coincided at design time and were treated as the same fact. They are two of the three identities (25.5 §11) — and the moment a remap separates them, every live entry is silently wrong.

And the reason it survives verification is that the two views only diverge after a remap with entries already outstanding — a state no directed test constructs unless someone thought of it, and one that recovery testing usually reaches with the pipeline drained.

DV/debug. The symptom is maximally misleading: a hang on the correct die, an unexpected response on an unrelated die, and not one integrity error — the failure signature 21.6 §14 calls delivered-perfectly-to-the-wrong-place. The discriminator is the pair of counts in §16: responses sent to a die versus responses matched at that die.

16. Corrected — Routing on Logical Identity, With an Epoch

// CORRECT. ILLUSTRATIVE (§13). Route on the LOGICAL die, and stamp the config
// epoch so an entry allocated under an old mapping is detected rather than
// silently misrouted (25.10 §11's shadow-validate-quiesce-commit, reduced to
// the one guarantee this table needs).
logic [DIE_W-1:0]   rt_die_q   [N_ENTRY];
logic [EPOCH_W-1:0] rt_epoch_q [N_ENTRY];
logic [EPOCH_W-1:0] cfg_epoch_q;
logic [31:0]        misroute_blocked_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    cfg_epoch_q        <= '0;
    misroute_blocked_q <= '0;
  end else begin
    if (accept_fire) begin
      rt_die_q[alloc_idx]   <= src_die_id;    // LOGICAL, not the wire
      rt_epoch_q[alloc_idx] <= cfg_epoch_q;   // the mapping it was allocated under
    end
 
    // A remap is only committed when nothing is outstanding under the old
    // epoch — so a live entry can never be resolved through a new mapping.
    if (remap_commit && (outstanding_total == '0))
      cfg_epoch_q <= cfg_epoch_q + 1'b1;
 
    // Defence in depth: if an entry ever survives into a new epoch, it is
    // BLOCKED and COUNTED rather than delivered somewhere plausible.
    if (resp_fire && (rt_epoch_q[resp_idx] != cfg_epoch_q))
      misroute_blocked_q <= misroute_blocked_q + 32'd1;
  end
end
 
// Routing resolves the LOGICAL die to a physical link at USE time, through the
// live mapping — so a remap moves future responses without touching any entry.
assign resp_link_sel = die_to_link[ rt_die_q[resp_idx] ];
 
// MANDATORY. English: a response is only emitted while its entry's epoch still
// matches the live configuration. Catches §15's silent misroute directly.
a_resp_epoch_current: assert property (
  @(posedge clk) disable iff (!rst_n)
    resp_fire |-> (rt_epoch_q[resp_idx] == cfg_epoch_q)
);
 
// MANDATORY. English: the mapping never changes while any request is
// outstanding. This is the property that makes the assertion above provable
// rather than merely hopeful.
a_remap_quiesced: assert property (
  @(posedge clk) disable iff (!rst_n)
    $changed(cfg_epoch_q) |-> $past(outstanding_total) == '0
);

Architecture. The routing key becomes the logical die, resolved to a wire at use time; an epoch records which mapping an entry was allocated under.

State. Two fields per entry plus one global epoch — and one counter that makes the impossible case visible if it ever happens.

Event. Resolution moves from allocation to use, which is the whole fix: a remap changes where future responses go without needing to touch a single live entry.

Contract. a_remap_quiesced is the load-bearing property. Quiescing before a remap is what makes the epoch check an assertion rather than a recovery mechanism (25.10 §11).

Failure. The residual risk is a remap that cannot quiesce because something is permanently outstanding — which turns a config change into a hang. outstanding_total non-zero with remap_commit pending is the diagnostic, and it names the blocking entry rather than the remap.

DV/debug. misroute_blocked_q should be zero forever. A non-zero value is not a performance signal — it is proof that the quiesce protocol was violated, and it converts §15's four-thousand-cycle two-die mystery into one register read.

17. Why a Proprietary Fabric Is the Better Teacher

This may look like the wrong case study for a UCIe curriculum. It is the right one, for three reasons.

It is documented. AMD publishes the architectural role of Infinity Fabric, the CCD/IOD structure, the GMI link's existence and the IOD's coherency ownership (§4) — all Level C. By contrast, 22.1 and 22.2 could establish no shipping product whose die-to-die links are UCIe. You cannot read an architecture that has not been published.

It demonstrates the layer boundary in a shipping product (§9). The IOD owns coherence; GMI moves bits. That is 23.3's entire argument, confirmed by a vendor about its own design.

And it shows what a standard link would and would not change (23.2 §8). Every CCD-IOD boundary is inside one company, between dies designed together, on one schedule — which is precisely where co-design wins and a standard's decisive advantages are absent. A candidate who can say that has understood the standard-versus-proprietary trade rather than memorised a preference.

18. Two Kinds of Disaggregation

A preview of 26.2, because the contrast is the point of the module.

And the framing worth carrying into 26.2: EPYC disaggregates by replication; Meteor Lake disaggregates by function. Those are different answers to different pressures, and neither is the general case — which is 25.7 §6's scorecard producing different partitions from different inputs.

19. What the Documentation Cannot Tell You

And the point of the table is that the framework tells you what to ask even where the answer is not public. A reader who can name the recovery-retention question about a documented boundary has understood something the documentation does not contain.

20. Common Misconceptions

"EPYC uses UCIe." §3, §4: its die-to-die links are documented as Infinity Fabric / GMI, and no shipping AMD product with UCIe D2D was established (22.2 §4).

"A chiplet product proves UCIe adoption." 22.1 §12: multi-die products predate the standard, and proprietary interfaces have real advantages at internal boundaries.

"Infinity Fabric is the die-to-die link." §9: it is the fabric — data path, control, coherence role. GMI is the CCD-to-IOD link. Conflating them is 23.3's layer error.

"UCIe could replace Infinity Fabric." §9: a link could occupy the transport row. It could not own coherence, routing, or the path to memory — those belong to a die at the centre of the graph.

"Hub-and-spoke is a compromise." §6: it buys latency uniformity and one boundary contract per compute die, and the hub is exactly the die that wants a mature process. It is a fit, not a fallback.

"The 32 Gbps figure is the CCD-to-IOD rate." §3, §4 row 7: it is documented for xGMI / G-links on the 8005 series, and I did not verify what xGMI connects there.

"More CCDs is strictly better." §7: every spoke converges on one hub, which concentrates arbitration, fairness and fault exposure — §14 is what you would measure.

"No CCD-to-CCD link is a gap in the design." §12: the hub already sits on every coherent path, so a direct link would still need it — the mesh saves less than the hop count suggests, and costs eleven contracts per die.

"The spoke number is the die number." §15: they coincide until a remap separates them, after which every outstanding entry routed on the wire is silently wrong — a hang on one die and an orphan on another, with no integrity error.

21. Understanding Check

22. Summary

Six things.

Hub-and-spoke: identical compute dies around one central die, each with its own link (§5). No CCD-to-CCD link — cross-compute traffic goes through the hub.

It buys latency uniformity and one contract per compute die (§6), and it scales by adding spokes without changing any CCD's interface.

The hub owns coherence, routing, memory and I/O (§9) — and the link owns moving bits across one boundary. That is the layer boundary of 23.3, documented in a shipping product.

The spoke's contract is the strictest kind (§10) because it carries coherent traffic: no silent drop, guaranteed completion, obligation survival, identity stability.

And identity stability is not abstract (§15–§16): a hub that routes a response on the wire it arrived over is correct until a remap separates the physical and logical views — then it hangs one die and orphans a response on another, with every integrity check passing. Route on the logical die, resolve at use time, and quiesce before a remap.

And it is not a UCIe product (§3, §20) — which makes it a better teacher, because it is documented at the architectural level and its boundaries sit exactly where co-design wins.