AMBA CHI · Module 1 · Cache Coherency Foundations

Cache Hierarchy Review

Coherency does not operate in a vacuum — it operates on a cache hierarchy, and you cannot reason about snoops or a Home Node until you can draw that hierarchy cleanly. This chapter reviews L1, L2, and L3, and the three inclusion policies — inclusive, exclusive, and NINE — that decide whether an outer level must hold a copy of everything an inner level holds. It extends the running project by adding a shared L2 beneath the two CPUs' private L1s, and fixes where the coherency boundary sits: private caches are per-core copies; the shared last-level cache is the point of coherence, and in CHI that ordering point becomes the Home Node.

Foundation11 min readAMBA CHICache HierarchyInclusive CacheLLCPoint of Coherence

Module 1 · Chapter 1.4 · Cache Coherency Foundations

Project thread — building on 1.1-1.3: the two CPUs (CPU0, CPU1) still have private L1 caches, but now they share a per-cluster L2, and a system last-level cache (LLC / L3) sits over one memory. That shared level is where coherency will live.

1. Learning Outcomes

By the end of this lesson you will be able to:

Name the cache levels — L1 (private), L2 (private or shared), L3 / LLC (shared) — and say which is private and which is shared.
Define inclusive, exclusive, and NINE, and give the capacity, tracking, and back-invalidation consequence of each.
Explain back-invalidation and why it is a property of the inclusion policy at the shared level, not of L1.
Place the point of coherence at the shared level and say why it is not simply "memory".
Map the shared point of coherence forward to CHI's Home Node and its directory / snoop filter.

2. Why Should I Learn This?

From 1.1-1.3 you know coherency means all copies of one line must agree. But copies do not live in one place — they live across levels, and coherency has to be defined at exactly one of them. The instant you have L1 above L2 above an LLC, "who holds a copy" and "who must be told about a write" become questions about which level is private, which is shared, and where the ordering point sits. That boundary is the whole game, and the inclusion policy at the shared level decides how cheaply you can track the private copies above it.

A quad-core cluster once shrank its inclusive shared L3 to save area and a hot loop that fit entirely in L1 began to thrash. The cause was structural: an inclusive LLC eviction back-invalidates the inner copies, so shrinking the shared level quietly reached up and tore hot lines out of L1 — you cannot reason about a cache's behavior without knowing its level and its inclusion policy.

3. Core Concept — levels, inclusion, and the coherency boundary

Extend the running project. The two CPUs keep their private L1 caches. Beneath them we add a shared, per-cluster L2, and beneath that a system last-level cache (LLC / L3) over one shared memory.

The levels (inner to outer).

L1 — small, fastest, private to one core; where a load hits first.
L2 — larger, slower; in this project shared by the cluster. In other designs L2 is private and L3 is the shared level — the names move, but "some level is private, some level is shared" does not.
L3 / LLC — largest, slowest cache, shared by the whole system, over DRAM. The last stop before memory.

Inclusion policy — the relationship between an outer level and the inner level(s) it covers.

Inclusive. The outer level holds a superset: every inner line is guaranteed also present in the outer. Cheap to answer "who might have this line" (the outer acts as a directory), but wastes capacity (lines are duplicated) and forces back-invalidation on outer eviction.
Exclusive. A line lives in exactly one level at a time — never duplicated. Maximizes total capacity (L1 plus LLC is the sum, not the max), but the outer level cannot answer "is this cached inside?" on its own.
NINE (non-inclusive non-exclusive). No guarantee either way — a line may or may not be duplicated, and an outer eviction does not force an inner invalidation. Pragmatic and common; it needs an explicit snoop filter / directory because the LLC contents alone are not authoritative.

Where the coherency boundary sits. Private caches (L1, and per-core levels) hold copies — they are requesters. The shared level — the point of coherence — is where every request for an address is serialized and made visible. In this project that is the LLC. It is not simply "memory": under write-back caching the newest value often lives in a cache above the LLC, so the point of coherence must order access and track which private caches hold a line, not just forward to DRAM. In CHI this point becomes the Home Node for an address range (Module 4); for now, just place the boundary correctly. This is still per-address agreement — distinct from consistency / ordering across different addresses (Module 12).

4. Engineering Diagram

Figure 1 — the extended project hierarchy and the coherency boundary. CPU0 and CPU1 each keep a PRIVATE L1 (copies). Beneath them a shared per-cluster L2, then a shared last-level cache (LLC / L3) over one memory. Everything above the boundary holds private copies that must be tracked; the LLC is the shared POINT OF COHERENCE where requests for an address serialize and writes become visible. In CHI that point becomes the Home Node. An inclusive LLC eviction back-invalidates the inner L1/L2 copies.

5. Worked Example — a load walks the hierarchy, then an inclusive back-invalidate

Trace a load descending the levels on a miss, filling as it goes — then watch an inclusive LLC eviction reach back up and invalidate an inner copy:

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

# Extended project: private L1 per core, shared L2, inclusive LLC, over memory.
# Line X initially resides only in memory.  "-" = not present (invalid).
 
Part A — L1 miss -> L2 hit path (X already in L2 from an earlier fill)
Step  Action                         L1(CPU0)  L2(shared)  LLC(incl)  Note
----  -----------------------------  --------  ----------  ---------  ---------------------------
  1   CPU0 load X: L1 miss            -         X           X          probe next level down
  2   L2 lookup X: HIT               -         X           X          no need to reach LLC/memory
  3   fill L1 <- L2                   X         X           X          duplicated (inclusive)
  4   CPU0 load X again: L1 HIT       X         X           X          fast path, no descent
 
Part B — inclusive LLC must evict X (needs the way for another line Y)
Step  Action                         L1(CPU0)  L2(shared)  LLC(incl)  Note
----  -----------------------------  --------  ----------  ---------  ---------------------------
  5   LLC picks X as victim for Y     X         X           X          inclusion invariant at risk
  6   LLC BACK-INVALIDATES inner X    -         -           (evict)    forces L1 + L2 to drop X too
  7   LLC install Y                   -         -           Y          X gone from ALL levels
  8   CPU0 load X: L1 miss again      -         -           Y          must refetch X from memory

Part A shows the fast case — the shared L2 already holds X, so an L1 miss is served one level down without touching the LLC or memory, and the line is duplicated up into L1 because the hierarchy is inclusive. Part B is the surprise: in an inclusive hierarchy the LLC cannot silently drop a line inner caches still hold — the inclusion invariant ("every inner line is in the LLC") would break — so it back-invalidates the inner copies first. A line CPU0 was happily hitting in L1 vanishes because a shared level, several steps away, needed the space.

6. RTL Illustration — a two-level lookup with inclusive back-invalidate

A simplified, behavioral demonstrator — not a coherency controller: one private L1 over a shared L2 over memory, tracking a single line X. A load hits L1, else L2, else fills from memory up through both levels; an L2 (LLC) eviction back-invalidates the L1 copy. It is the hierarchy baseline later modules make coherent.

Azvya Education Pvt. Ltd.VLSI Mentor

Snippet

// chi_hierarchy_demo.sv — private L1 over shared L2 over memory (single line X).
// Simplified/behavioral: models descend/fill and inclusive back-invalidation only.
module chi_hierarchy_demo #(parameter int DW = 8) (
  input  logic          clk,
  input  logic          rst_n,
  input  logic          load,      // CPU0 load of line X
  input  logic          l2_evict,  // LLC/L2 evicts X (inclusive back-invalidate)
  output logic [DW-1:0] rdata,     // data returned to CPU0
  output logic          l1_hit,    // load hit in L1
  output logic          l1_valid,  // L1 holds line X
  output logic          l2_valid   // L2 holds line X
);
  logic [DW-1:0] mem_X, l2_X, l1_X;   // one tracked line per level: value + valid
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      mem_X <= 8'd42;                          // X lives in memory at reset
      l2_X  <= '0; l2_valid <= 1'b0;
      l1_X  <= '0; l1_valid <= 1'b0; l1_hit <= 1'b0;
    end else begin
      l1_hit <= 1'b0;
      if (l2_evict) begin                      // inclusive eviction:
        l2_valid <= 1'b0;
        l1_valid <= 1'b0;                      // inner copy torn out to preserve inclusion
      end else if (load) begin
        if (l1_valid)      l1_hit  <= 1'b1;              // fast path: L1 hit
        else if (l2_valid) begin l1_X <= l2_X;  l1_valid <= 1'b1; end   // fill L1 from L2
        else begin l2_X <= mem_X; l2_valid <= 1'b1;      // fill L2 from memory,
                   l1_X <= mem_X; l1_valid <= 1'b1; end  // and L1 (duplicated: inclusive)
      end
    end
  end
  assign rdata = l1_valid ? l1_X : (l2_valid ? l2_X : mem_X);
endmodule

Invariant note: for an inclusive level, any line valid in L1 must also be valid in L2/LLC (l1_valid implies l2_valid), and an outer eviction must clear the inner copy on the next cycle — the same invalidate a write will later reuse to enforce coherency. A quick behavioral sim (cold load, warm L1 hit, then l2_evict, then reload) shows the reload missing in L1 and refilling with rdata=42, confirming back-invalidation.

7. DebugLab

Inclusive LLC back-invalidates a hot L1 line

INCLUSIVE LLC BACK-INVALIDATION -> FIX THE SHARED LEVEL, NOT L1

Symptom

After the shared L3 (LLC) was shrunk to save area, a hot loop whose working set fits in L1 suddenly thrashes. L1 hit rate collapses even though the loop never grew. Profilers show constant refills of lines the core was actively using. Nothing in the private L1/L2 changed.

Root Cause

The LLC is inclusive, so it must hold a copy of every line present in any inner cache — and when the now-smaller LLC evicts a line to make room, inclusion forces it to back-invalidate that line from the L1/L2 that still hold it, even hot lines. Shrinking the shared level dropped its associativity below the summed footprint of the inner caches, so LLC victims increasingly landed on lines the cores were still using. Each eviction drove a back-invalidate up into L1, and the next access missed and refetched — the thrash. The private caches are innocent; the coupling is structural.

Fix

Break the coupling at the shared level: change the inclusion policy or size the LLC to cover the inner caches. Moving the LLC to NINE stops an eviction from back-invalidating inner copies — at the cost of an explicit snoop filter / directory to track who holds a line. Alternatively, keep inclusion but size the LLC so its associativity comfortably exceeds the summed inner footprint. The principle: back-invalidation is a property of the inclusion policy at the shared level, not a bug in the private caches — the same shared level where CHI later places the Home Node's directory (Modules 4 and 11).

8. Common Mistakes

Calling the LLC "memory." Under write-back caching the newest value can live above the LLC. The point of coherence must order and track, not just forward to DRAM.
Assuming inclusion is free. Inclusive caches duplicate lines (wasted capacity) and cause back-invalidation on outer eviction — a real performance cliff.
Confusing exclusive with NINE. Exclusive guarantees a line is in exactly one level; NINE guarantees nothing either way. They need different tracking.
Placing coherency at L1. L1 is private — it holds copies. Coherency is enforced at the shared level, not inside a private cache.
Fixing back-invalidation by tuning L1. The coupling lives at the shared level's inclusion policy; changing L1 replacement does nothing.

9. Interview Questions

10. Engineering Checklist

I can name the levels — L1 (private), L2 (private or shared), LLC / L3 (shared) — and say which is private and which is shared.
I can define inclusive, exclusive, and NINE and give the capacity / tracking / back-invalidation consequence of each.
I can explain back-invalidation and why it is a property of the inclusion policy at the shared level, not of L1.
I can place the point of coherence at the shared level and say why it is not simply "memory."
I can state how the shared point maps forward to CHI's Home Node and directory / snoop filter.

11. Key Takeaways

The hierarchy has levels with owners. L1 (and often L2) is private to a core; the LLC / L3 is shared by the system over memory.
Inclusion policy is the key choice. Inclusive = outer holds a superset (free snoop filter, but duplication and back-invalidation); exclusive = a line lives in exactly one level (max capacity, weaker tracking); NINE = no guarantee (needs an explicit directory).
Back-invalidation is an inclusion property. An inclusive outer eviction forces inner invalidation — fix it at the shared level, never at L1.
Coherency lives at the shared level. The point of coherence is where accesses serialize and writes become visible — not L1, and not plain memory.
This maps straight to CHI. The shared point of coherence becomes the Home Node with its directory / snoop filter tracking the private copies above it.

12. Quick Revision

Cache hierarchy review. Levels: L1 private (per core), L2 (private or per-cluster shared), LLC / L3 shared over memory. Inclusion policy = relationship between outer and inner: inclusive (outer = superset, so a free snoop filter but duplication plus back-invalidation on outer eviction), exclusive (line in exactly one level, so max capacity but weak tracking), NINE (no guarantee, so it needs an explicit directory / snoop filter). Back-invalidation is an inclusion effect, not an L1 bug — fix at the shared level. Point of coherence = the shared level (LLC), where accesses serialize and writes become visible — not memory (newest value may live in a cache under write-back). In CHI that point is the Home Node.

13. Coming Next

Next — 1.5 The Stale Data Problem. You will add:

how a write into one private cache leaves the other cores' copies — spread across L1, L2, and the LLC — holding the old value,
why the shared point of coherence is the only place that can catch and reconcile that divergence,
the first concrete picture of a snoop/invalidate acting across the levels you just drew.

We keep the same project — CPU0, CPU1, private L1s over a shared L2 and LLC — and zoom from "here is the hierarchy and where coherency sits" into "here is exactly how the data goes stale across it."