Server-Class Processors

Module 20 · Chapter 20.6 · Real Industry Case Studies

Project thread — 20.5 composed coherent subsystems. 20.6 scales to the server, closing Module 20; Module 21 is the design-review checklist.

1. The Domain

Server-class processors — Neoverse / Graviton-class — are CHI at its largest scale: dozens of cores across many clusters on a huge coherent mesh, with many distributed home nodes, a large system-level cache, and many memory controllers, often spanning multiple die. It is the most demanding CHI configuration, and the one where a new concern takes over: at this scale the mesh already scales throughput, so latency — specifically non-uniform memory access (NUMA) — becomes the dominant performance factor.

2. Why CHI Here

Server processors use CHI because it is the only AMBA coherence protocol built to scale this far. The distributed, interleaved home nodes (Chapter 15.3) let coherence scale to dozens of cores; the credit-based, deadlock-free mesh (Chapters 14.1, 14.3) stays reliable at server size; DVM (Chapter 12) handles system-wide TLB maintenance; and CHI extends across die/chiplet links so coherence spans a package, not just a chip. Add RAS — poison and error containment (Chapter 11-class concerns) — and CHI provides the coherence, scale, and reliability a server needs. This is CHI at its ceiling, doing what it was ultimately designed for.

3. Key Terms

4. Previous Chapter Connection

This is the capstone — every scalability lever, at maximum. The CPU cluster (Chapter 20.3) — DSU clusters, a CMN mesh, distributed homes, an SLC — is here multiplied; composable subsystems (Chapter 20.5) are how it is built; the full performance model of Module 15 (latency anatomy 15.1, distribution 15.3, memory-bandwidth knee 15.4, directory efficiency 15.5, snoop reduction 15.6, hotspots 15.7) all apply at once.

Two things are new: cross-die coherence (CHI over chiplet links) and RAS (poison/containment, Chapter 11-class). But the defining shift is that the dominant concern flips from throughput to latency — because the mesh scales throughput well, what's left is NUMA. This case study is where the large-SoC interview answer (Chapter 19.10) meets its hardest case, and it sets up the design-review checklist (Module 21) that a system this complex demands.

5. The CHI Configuration

The server-class CHI configuration:

• Scale, multiplied. Dozens of cores across many clusters (RN-F) on a huge coherent mesh, with many distributed home nodes (HN-F) — addresses interleaved — a large distributed SLC, and many memory controllers (SN).
• Cross-die coherence. CHI extends over die-to-die / chiplet links, so the coherent domain spans multiple die — a coherent access can cross a package.
• RAS built in. Poison and error containment keep a single fault from corrupting the whole coherent domain — reliability is first-class at server scale.
• NUMA is inherent. With homes and memory spread across a huge mesh (and across die), access latency is non-uniform — near is fast, far is slow.

The synthesis:

A server-class processor is CHI at maximum scale: dozens of cores, a huge interleaved-home mesh, a large SLC, many memory controllers, cross-die coherence, and RAS. The mesh scales throughput, so the dominant concern becomes NUMA / distant-home latency, and the design — and its software — is about locality.

6. The Domain Constraints

What shapes a server-class CHI design — scale, with latency as the enemy.

• Throughput is (mostly) solved. Distributed, interleaved homes (Chapter 15.3) already scale coherence throughput to dozens of cores — so throughput is not the binding constraint. Latency is.
• NUMA dominates. With homes and memory spread across a huge mesh and across die, latency is non-uniform: a near home/controller is fast; a distant one — far mesh corner or another die — is much slower (Chapter 15.1). This variance dominates real performance.
• Cross-die is slower still. A die-to-die hop adds latency beyond any on-die distance — so cross-die coherent traffic must be rare.
• Reliability at scale. More die, more cores, more memory means more faults — so RAS (poison, containment) is required, not optional.

The point to carry:

At server scale the mesh scales throughput, so the enemy becomes latency — and latency is non-uniform (NUMA): near is fast, far (and cross-die) is slow. The entire design, and the software on top of it, is therefore about locality: keep work near its data, size the SLC to avoid distant-memory trips, and keep cross-die traffic rare. Treat the system as flat/uniform and locality is ignored — the mesh scales throughput while latency caps delivered performance.

7. Engineering Diagram — CHI at server scale

Many clusters on a huge mesh (many HN-F + a large SLC), many memory controllers, and a cross-die link to a second die. Throughput scales with the distributed homes; the dominant concern is NUMA — near is fast, distant / cross-die is slow — so locality governs delivered performance.

8. The Distinctive Design Decisions

What a server-class design does — and why.

The rule to carry: at server scale, throughput scales with distributed homes, so the design decision that matters most is locality. The mesh, the homes, and the SLC scale capacity and throughput; cross-die scales beyond a die; RAS keeps it reliable. But because latency is non-uniform, the delivered performance is set by how local the accesses are — which makes NUMA-awareness (in hardware placement and software scheduling) the defining decision, and the one the pitfall punishes.

9. The Trade-offs

The tensions a server-class design balances.

• Scale vs locality. More cores/homes/memory raise aggregate throughput but make the mesh bigger, so the average home/memory is farther (Chapter 15.1) — scale worsens NUMA. Balanced by placement and locality.
• Monolithic mesh vs multi-die. One huge die has uniform on-die latency but poor yield/cost; chiplets (multi-die) give better economics and more total cores but add cross-die latency. The split trades cost against latency.
• SLC size vs area. A larger SLC turns distant-memory trips into local hits — very valuable when memory is far — but costs area (Chapter 15.5). Sized generously at server scale.
• Performance vs reliability. RAS (poison, containment, checking) costs area/latency but is non-negotiable at server scale — more silicon means more faults to contain.

The point to carry:

The server-class trade is scale vs locality: scaling out raises throughput but pushes homes and memory farther, worsening NUMA, so the design fights back with placement, a large SLC, and NUMA-aware software. Chiplets add cores and cut cost but add cross-die latency to manage. And RAS is a required tax. The art is scaling throughput without letting latency win — the central challenge of the largest CHI systems.

10. A Concrete Walkthrough

A memory access under NUMA — the scenario that defines server-scale performance.

1. Local access — fast. A core reads a line whose home and memory controller are near it on the mesh. Few hops; the SLC or a near controller sources it. Low latency.
2. Distant access — slow. Another core reads a line whose home sits at the far corner of the mesh, and on an SLC miss the memory controller is far too. Many hops each way — much higher latency (Chapter 15.1).
3. Cross-die — slower. A third access targets a home on another die. It crosses the die-to-die link — latency beyond any on-die distance.
4. Same coherence, different latency. All three are functionally identical coherent reads — CHI keeps them correct. But their latencies differ greatly: that variance is NUMA.
5. Locality decides. If software (and placement) keep accesses local, most look like step 1; if not, step 2/3 dominate and performance collapses — even though throughput and correctness are fine.

The three accesses were equally correct and wildly different in latency — the essence of NUMA. The mesh scaled throughput; locality decided delivered performance. The pitfall is treating the system as flat and ignoring this (section 14).

11. Representative Configuration

A server-class CHI configuration. Representative.

// Representative server-class (Neoverse/Graviton-class) CHI configuration (educational).
// CHI at maximum scale: throughput scales; NUMA / distant-home latency is the enemy.
localparam int N_CORES           = 64;     // dozens of cores across many clusters (RN-F)
localparam int N_CLUSTERS        = 16;     // clusters of cores (DSU-style, 20.3)
localparam int MESH_ROWS         = 8;      // HUGE mesh -- bigger mesh => farther homes (NUMA)
localparam int MESH_COLS         = 8;
localparam int N_HOME_NODES      = 32;     // MANY distributed HN-F, address-interleaved (15.3)
localparam int SLC_MB            = 64;     // LARGE distributed SLC -- cut distant-memory trips
localparam int N_MEM_CTRL        = 8;      // MANY controllers -- high aggregate BW (15.4)
 
// New at server scale:
localparam int N_DIE             = 2;      // CROSS-DIE / chiplet coherence over die-to-die
localparam bit RAS_POISON        = 1;      // poison + error containment (reliability at scale)
 
// Dominant concern: NUMA. Throughput SCALES with N_HOME_NODES/N_MEM_CTRL;
// delivered performance is set by LOCALITY -- near home/memory fast, distant/cross-die slow.
// Design + software must place work near its data and keep cross-die traffic rare.

The scale knobs (cores, mesh, homes, SLC, controllers) are the CPU-cluster config (Chapter 20.3) multiplied; N_DIE and RAS_POISON are new. The comment states the defining truth: throughput scales, so NUMA/locality governs delivered performance. This closes the case-study arc — the largest system CHI builds.

12. Analysis View — throughput scales, latency governs

The two forces, at server scale.

• Throughput scales — nearly for free. With many interleaved homes and many controllers, aggregate coherence and memory throughput scale with the counts (Chapters 15.3, 15.4). At this scale, throughput is not the limiter.
• Latency governs — and it's non-uniform. The dominant latency term (Chapter 15.1) grows with mesh size and explodes across die. Because it's non-uniform, the distribution of access distances — not the average — sets delivered performance.
• The SLC attacks the worst term. A large SLC converts distant-memory accesses into on-mesh hits — the single most valuable lever when memory is far (Chapter 15.5).
• Locality is the multiplier. Placement + NUMA-aware software keep most accesses near, turning the latency distribution local — the difference between scaling well and stalling.

The point to carry:

The server-class analysis is a role reversal: at small scale you fight throughput; at server scale throughput scales, so you fight latency — and specifically its non-uniformity (NUMA). The winning moves are a large SLC (kill distant-memory trips), balanced placement (Chapter 15.7), rare cross-die traffic, and NUMA-aware software. This is the complete Module 15 toolkit, aimed at the latency distribution rather than the throughput ceiling.

13. Design Considerations

The deeper checklist for a server-class CHI design.

• Scale throughput with distributed, interleaved homes and many controllers — the throughput half is largely solved this way.
• Attack latency with a large SLC — turn distant-memory trips into on-mesh hits.
• Design for NUMA — expose locality to software; place work near its data.
• Keep cross-die traffic rare — die-to-die hops are the costliest.
• Build in RAS — poison and error containment for reliability at scale.
• Balance placement — avoid hotspots and keep the average access near (15.7).

14. The Pitfall

15. Common Mistakes

• Assuming UMA/flat. Assumption: latency is uniform. Bug: distant accesses dominate (the Pitfall). Fix: design for NUMA.
• Chasing throughput only. Assumption: throughput = performance. Bug: latency caps delivered speed. Fix: attack the latency distribution.
• Undersized SLC at scale. Assumption: small SLC suffices. Bug: constant distant-memory trips. Fix: large SLC.
• Frequent cross-die traffic. Assumption: die-to-die is cheap. Bug: worst-case latency everywhere. Fix: keep cross-die rare.
• NUMA-blind software. Assumption: hardware hides locality. Bug: unplaced work stalls. Fix: NUMA-aware placement.
• Skimping on RAS. Assumption: faults are rare. Bug: one fault corrupts the domain. Fix: poison + containment.

16. Engineering Checklist

• Scale throughput with distributed, interleaved homes and many memory controllers.
• Attack latency with a large, distributed SLC — cut distant-memory trips.
• Design for NUMA — expose locality; place work near its data.
• Keep cross-die coherent traffic rare — die-to-die hops are costliest.
• Build in RAS — poison and error containment for reliability at scale.
• Measure the latency distribution, not just aggregate throughput.

17. Key Takeaways

• Server-class processors are CHI at maximum scale — Neoverse / Graviton-class.
• The mesh scales throughput through distributed, interleaved homes; latency becomes the limiter.
• The dominant concern is NUMA — near is fast, distant and cross-die are slow.
• Cross-die coherence and RAS are new at this scale.
• The winning lever is locality — placement, a large SLC, rare cross-die traffic, NUMA-aware software.
• The pitfall is treating a huge system as flat/uniform; the model here is representative.

18. Quick Revision

Server-class processors. CHI at its largest scale — Neoverse / Graviton-class: dozens of cores across many clusters (RN-F) on a huge coherent mesh, with many distributed, interleaved home nodes (HN-F), a large distributed system-level cache, and many memory controllers, often spanning multiple die via cross-die / chiplet coherence, with RAS (poison, error containment) for reliability at scale. Everything from the CPU-cluster case study (Chapter 20.3) is here multiplied, built by composing subsystems (Chapter 20.5). The defining shift is that the dominant concern flips: the interleaved-home mesh already scales throughput (Chapter 15.3), so what caps delivered performance is latency — and specifically its non-uniformity, NUMA. A near home or controller is fast; a distant one (far mesh corner) is many hops slower; a cross-die one is slower still (Chapter 15.1). So the design — and the software on top — is all about locality: place work near its data, size a large SLC to kill distant-memory trips (Chapter 15.5), keep cross-die traffic rare, and balance placement (Chapter 15.7). The pitfall: treating a huge system as flat/uniform — ignoring NUMA — so unplaced work stalls on distant homes and memory, throughput scales while latency quietly caps real performance, and the cost grows with scale. Representative case study; this closes Module 20's real-world tour, and Module 21 turns the whole track into a design-review checklist.

Coming Next

Chapter 21.1 — Architecture Review Checklist. The case studies are done; the next chapter turns the whole CHI track into a pre-RTL review tool. Chapter 21.1 opens Module 21 — CHI Design-Review Checklist with the architecture review — the questions to ask about node types, home distribution, coherence boundaries, ordering, deadlock freedom, and NUMA before any RTL is written, so a coherent system is reviewed while it is still cheap to change.