AMBA CHI · Module 20 · Real Industry Case Studies

Multi-Core Subsystems

Multi-core subsystems build coherence to compose. A coherent subsystem — a cluster of cores with local fabric and homes — is reusable IP with a well-defined coherence boundary. Instead of one flat monolithic fabric, you build coherent blocks and connect them, so coherence becomes hierarchical: the local fabric keeps the cores coherent, a higher level keeps the subsystems coherent, and the boundary bridges local coherence into the system domain. The design question is where to draw the boundary, trading modularity and independent verification against an extra level's latency. The pitfall composition introduces is a coherence-boundary leak — a subsystem assuming it is the whole coherent domain, or letting internal state escape its boundary. Representative case study, not a product specification.

Intermediate14 min readAMBA CHICoherent IPHierarchical CoherenceComposabilitySubsystems

Module 20 · Chapter 20.5 · Real Industry Case Studies

Project thread — 20.4 was one coherent device. 20.5 composes coherent subsystems; 20.6 scales to a server.

1. The Domain

Multi-core subsystems are coherent building blocks — a cluster of cores with its local coherent interconnect and homes — packaged as reusable IP that composes into a larger system. Instead of one flat, monolithic coherent fabric, a large SoC is built from coherent subsystems connected at coherence boundaries. It is how real large chips are actually built (by integration, not from scratch), which makes it one of the most practically important CHI patterns — and the composability and coherence boundaries are what make it work.

2. Why CHI Here

Composable coherent subsystems use CHI because its layered, node-typed structure lets a subsystem be a self-contained coherent unit that still plugs into a larger CHI system. Inside, the subsystem has its own RN-F / HN-F on a local fabric (Chapters 5, 6); at its boundary, it presents a CHI interface that a higher-level fabric connects to. Because CHI's coherence is home-mediated (Chapter 8) and hierarchical by nature (the CPU cluster's two levels, Chapter 20.3), a subsystem's internal coherence can be bridged into a system-level coherence domain. CHI makes coherent IP composable — which is why large SoCs are built this way.

3. Key Terms

4. Previous Chapter Connection

This chapter generalises the two-level idea. The CPU cluster (Chapter 20.3) had intra-cluster (DSU) and inter-cluster (CMN) coherence; multi-core subsystems make that a composition pattern — each subsystem is an intra-coherent unit, and a higher level makes them inter-coherent. The node types (Chapter 5), home-mediated coherence (Chapter 8), and hierarchy all reappear as the mechanics of composition.

Where 20.3 named a specific architecture (DynamIQ + CMN), 20.5 is the general engineering practice: coherent IP you can verify independently and integrate. This is how the large-SoC interview answer (Chapter 19.10) becomes a build method — assemble coherent subsystems rather than design one enormous fabric. And it introduces the risk that only composition has: the coherence boundary can leak.

5. The CHI Configuration

The composable coherent-subsystem configuration:

Each subsystem is internally coherent. A subsystem holds a cluster of cores (RN-F), a local coherent fabric, and its homes (HN-F) — a self-contained coherence unit.
A coherence boundary at the edge. Each subsystem presents a CHI coherence boundary — the interface where its internal coherence connects to the outside.
Hierarchical coherence composes them. A higher-level fabric connects the subsystems' boundaries, keeping the subsystems coherent with each other — inter-subsystem coherence on top of each subsystem's intra-coherence.
The boundary bridges, it doesn't leak. The boundary translates a subsystem's local coherence into the system domain without exposing internal state — a clean interface.

The synthesis:

A multi-core subsystem is a coherent building block — cores + local fabric + homes — with a well-defined coherence boundary. Systems are built by composing subsystems: hierarchical coherence keeps each subsystem coherent internally and all subsystems coherent with each other, the boundary bridging local coherence into the system domain. It is coherent IP that composes.

6. The Domain Constraints

What shapes a composable-subsystem design.

Reuse and independent verification. A subsystem must be verifiable on its own and reusable across products — so it needs a clean, self-contained coherence unit with a defined boundary, not tangled global state.
Hierarchical coherence. Coherence must work inside each subsystem and across subsystems — two levels, composed. The boundary is where the two levels meet.
The boundary must bridge cleanly. The interface must translate local coherence to the system level without leaking internal assumptions — or composition breaks.
Where to draw the boundary. Bigger subsystems mean fewer boundaries (less bridging) but a larger internal coherence problem; smaller ones mean more boundaries. The split is a design choice.

The point to carry:

The composable subsystem trades a flat monolith for modularity — reuse and independent verification — at the cost of an extra hierarchy level (the boundary and its latency). The whole method depends on the boundary being clean: a subsystem must be a self-contained coherence unit that bridges at its edge without leaking. Get the boundary right and you can build big chips by integration; get it wrong and composition breaks coherence.

7. Engineering Diagram — composing coherent subsystems

Figure 1 — composable coherent subsystems on CHI. Each subsystem is internally coherent — a cluster of cores with a local fabric and homes — and presents a coherence boundary at its edge. A higher-level system fabric connects the boundaries, keeping the subsystems coherent with each other: hierarchical coherence, intra-subsystem plus inter-subsystem. The boundary bridges local coherence into the system domain without leaking internal state.

Two internally-coherent subsystems each present a coherence boundary, and a system fabric composes them into one coherence domain. Coherence is hierarchical — intra-subsystem inside each block, inter-subsystem across the fabric. The boundary bridges without leaking.

8. The Distinctive Design Decisions

What composable subsystems do — and why.

Decision	Subsystem choice	Why
Structure	coherent building blocks	reuse + independent verification
Coherence	hierarchical (intra + inter)	compose blocks into one domain
Boundary	first-class CHI interface	bridge local coherence to system
State	contained, not leaked	keep the boundary clean
Boundary placement	a deliberate split	fewer boundaries vs smaller blocks
Build method	integrate, don't rebuild	large SoCs are assembled

The rule to carry: composability makes the coherence boundary a first-class interface, and hierarchical coherence is how blocks compose. A subsystem is a self-contained coherence unit; the boundary bridges it into the system; the higher level keeps subsystems coherent with each other. The boundary must contain internal state — its cleanliness is what makes composition safe, and its leaks are the one risk composition adds.

9. The Trade-offs

The tensions a composable-subsystem design balances.

Modular vs monolithic. Composable subsystems give reuse and independent verification but add an extra hierarchy level (the boundary) with its latency; a flat monolith avoids the level but is not reusable and is harder to verify whole. For large SoCs, modular wins.
Boundary placement. Bigger subsystems ⇒ fewer boundaries (less bridging overhead) but a larger internal coherence problem per block; smaller subsystems ⇒ more boundaries but simpler blocks. Drawn to balance integration vs internal cost.
Clean boundary vs shortcuts. A clean, self-contained boundary is safe to compose but takes discipline; leaking internal state across it is a shortcut that breaks composition (the Pitfall). Always clean.
Independent vs global verification. Verifying subsystems independently scales, but the composition (the boundaries) must also be verified — the seams are where bugs hide.

The point to carry:

The composable-subsystem trade is modularity vs a flat design: reuse and independent verification against an extra hierarchy level. It is strongly favourable for large SoCs — which is why they are built by integration. But the trade only pays off if the coherence boundary is clean; a leak turns a safe composition into a broken one, so boundary discipline is the price of modularity.

10. A Concrete Walkthrough

A line shared across two subsystems — the scenario that exercises the boundary.

Local sharing stays local. Two cores within subsystem A share a line — subsystem A's own fabric and homes handle it. Intra-subsystem coherence; the boundary is not involved.
Cross-subsystem access. A core in subsystem A needs a line that subsystem B holds. A's local homes miss for that (system) address, so the request goes out through A's coherence boundary.
System fabric routes it. The higher-level fabric routes the request to subsystem B, crossing B's boundary into B's domain.
Boundary bridges coherence. B's boundary presents B's internal state to the system coherently — sourcing the line without exposing B's internal coherence details. Inter-subsystem coherence.
Complete. The line returns to A across the boundaries. Both subsystems stayed internally coherent and became mutually coherent — hierarchically.

The access stayed local when it could and crossed boundaries only when it had to, each boundary bridging cleanly. The pitfall is a boundary that leaks — a subsystem assuming it is the whole domain, so the cross-subsystem step gets stale data or fails to bridge (section 14).

11. Representative Configuration

A composable coherent-subsystem CHI configuration. Representative.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

// Representative composable coherent-subsystem configuration (educational, not a spec).
// Coherent IP building blocks with defined coherence BOUNDARIES, composed hierarchically.
localparam int N_SUBSYSTEMS      = 2;      // coherent subsystems composed into one system
localparam int CORES_PER_SUBSYS  = 4;      // cores per subsystem (RN-F), internally coherent
localparam int HOMES_PER_SUBSYS  = 2;      // local homes (HN-F) inside each subsystem
 
// Each subsystem is a SELF-CONTAINED coherence unit behind a COHERENCE BOUNDARY.
localparam bit COHERENCE_BOUNDARY= 1;      // clean CHI boundary: bridge, do NOT leak state
localparam bit STATE_CONTAINED   = 1;      // internal coherence state stays inside (no leak)
 
// HIERARCHICAL coherence: a higher level composes the subsystems.
localparam int SYSTEM_FABRIC     = 1;      // inter-subsystem coherence across boundaries
// intra-subsystem: local fabric + homes (fast, local)
// inter-subsystem: system fabric bridges boundaries (compose blocks into one domain)
// Boundary placement (subsystem size) trades FEWER boundaries vs SMALLER internal problem.

The configuration names the two levels — internally-coherent subsystems and a system fabric that composes them — and the boundary with its state-containment rule. The knobs (subsystem count/size, homes per subsystem) are the boundary-placement trade of section 9. This is the build-by-integration pattern the server case (Chapter 20.6) scales up.

12. Analysis View — the cost and benefit of composition

What composition adds and saves.

Benefit: reuse + independent verification. A verified coherent subsystem is reused across products and verified once on its own — a large engineering saving that a monolith cannot match.
Cost: an extra hierarchy level. A cross-subsystem access crosses boundaries and the system fabric — extra hops/latency (Chapter 15.1) versus a flat design. Local accesses avoid it.
Locality pays. Keeping sharing within a subsystem avoids the boundary entirely — so placement of work to a subsystem matters, echoing hotspot/locality thinking (Chapters 15.1, 15.7).
Seams need verification. The boundaries are where composition can break — they must be verified as composed, not just each subsystem alone.

The point to carry:

Composition's analysis is engineering-scale benefit (reuse, independent verification, build-by-integration) against a per-access cost (an extra hierarchy level for cross-subsystem traffic) and a verification obligation (the boundaries). For large SoCs the benefit dominates — but only with clean boundaries and locality, so cross-subsystem traffic (the costly path) stays the exception, not the rule.

13. Design Considerations

The deeper checklist for a composable-subsystem design.

Build coherent subsystems as self-contained units — cluster + local fabric + homes.
Define a clean coherence boundary per subsystem — a first-class CHI interface.
Compose hierarchically — a system fabric for inter-subsystem coherence.
Contain internal state — never leak coherence assumptions across the boundary.
Place the boundary to balance fewer boundaries against smaller internal blocks.
Verify the seams — the boundaries, composed, not just each subsystem alone.

14. The Pitfall

Pitfall: coherence-boundary leaks

The mistake unique to composition is a coherence-boundary leak — a subsystem that does not respect its own boundary. Two shapes. First, a subsystem that assumes it is the whole coherent domain: designed and verified as if nothing exists outside it, so when it is composed it does not correctly bridge cross-domain requests — a core in another subsystem reads stale data, because this subsystem never expected an outside requester. Second, a subsystem that leaks internal state or assumptions across its boundary: exposing internal coherence details, sharing IDs or ordering assumptions that only hold inside, so the system fabric and the other subsystems see state they cannot interpret consistently — coherence breaks at the seam. Both are invisible when each subsystem is verified alone (in isolation it is correct) and only appear on composition — which is exactly when they are hardest to debug, because the bug is in the interaction, not in any one block. A leak can produce stale data, a non-bridging boundary, or a deadlock where the two coherence levels disagree about who owns a line.

The fix: treat the boundary as a first-class contract. Make each subsystem a self-contained coherence unit that assumes an outside exists, contain internal state, and bridge — never leak — at the edge. Verify the subsystem composed, driving cross-boundary traffic, not only in isolation.

15. Common Mistakes

Subsystem assumes it is everything. Assumption: nothing outside. Bug: no bridging, stale cross-domain data (the Pitfall). Fix: assume an outside exists.
Leaking internal state. Assumption: internal IDs/ordering are global. Bug: seam breaks. Fix: contain state, bridge cleanly.
Verifying only in isolation. Assumption: correct alone ⇒ correct composed. Bug: seam bugs escape. Fix: verify composed.
Flattening for simplicity. Assumption: skip the boundary. Bug: no reuse, unverifiable whole. Fix: keep subsystems modular.
Wrong boundary placement. Assumption: any split works. Bug: too many boundaries or huge blocks. Fix: balance the split.
Ignoring locality. Assumption: cross-subsystem is free. Bug: constant boundary crossings. Fix: keep sharing local.

16. Engineering Checklist

Build each subsystem as a self-contained coherent unit (cores + fabric + homes).
Define a clean coherence boundary — a first-class CHI interface per subsystem.
Compose hierarchically — a system fabric for inter-subsystem coherence.
Contain internal state; never leak coherence assumptions across the boundary.
Place the boundary to balance boundary count against block size.
Verify composed — drive cross-boundary traffic, not just each subsystem alone.

17. Key Takeaways

Multi-core subsystems are coherent building blocks that compose into a larger system.
Coherence is hierarchical — intra-subsystem plus inter-subsystem.
The coherence boundary is a first-class interface that bridges local coherence to the system.
Composition gives reuse and independent verification at the cost of an extra hierarchy level.
Large SoCs are built by integration, not from one flat fabric.
The pitfall is a coherence-boundary leak; the model here is representative.

18. Quick Revision

Multi-core subsystems. Coherent building blocks — a cluster of cores with its local coherent fabric and homes (HN-F) — packaged as reusable IP with a well-defined coherence boundary, so a large SoC is built by composing subsystems rather than designing one flat, monolithic fabric. Coherence is hierarchical: inside a subsystem the local fabric and homes keep the cores coherent (intra-subsystem), and a higher-level system fabric keeps the subsystems coherent with each other (inter-subsystem), with the coherence boundary bridging a subsystem's internal coherence into the system domain. This generalises the two-level CPU-cluster idea (Chapter 20.3) into a reuse and integration pattern: coherent IP you can verify independently and assemble. The design questions are where to draw the boundary (bigger subsystems ⇒ fewer boundaries but a larger internal coherence problem) and how it bridges — trading modularity and independent verification against an extra hierarchy level's latency, a trade that is strongly favourable for large SoCs. The pitfall — unique to composition — is a coherence-boundary leak: a subsystem that assumes it is the whole coherent domain (so it fails to bridge, and cross-domain reads get stale data) or that leaks internal state/assumptions across its boundary (so the seam breaks coherence). Leaks hide because each subsystem is correct in isolation and only fails on composition; the fix is a first-class boundary contract — contained state, clean bridging, verified composed. Representative case study; 20.6 scales this to a server-class processor.

Coming Next

Chapter 20.6 — Server-Class Processors. Multi-core subsystems composed a few coherent blocks; the next case scales composition to the largest CHI systems. Chapter 20.6 covers server-class processors — Neoverse / Graviton-class designs with a huge coherent mesh, many distributed home nodes and a large system-level cache, many memory controllers, cross-die coherence, and RAS — where the dominant concern becomes NUMA locality and distant-home latency at scale.