Why AXI Is Not Enough

Module 1 · Chapter 1.8 · Cache Coherency Foundations

Project thread — the same two CPUs (CPU0, CPU1), each with a private L1 cache, over one shared memory. Here we connect them with the AXI interconnect you already know, and show it has no way to keep the caches coherent.

1. Learning Outcomes

By the end of this lesson you will be able to:

• Explain that AXI is a per-master, point-to-point read/write protocol with no snoop path, and therefore provides no coherency.
• Trace why an AXI write that correctly reaches memory still leaves a peer core's cached copy stale.
• Name the AXI-only workaround — software clean/flush plus invalidate — and explain why it is slow, fragile, and unscalable.
• Distinguish AXI, ACE, ACE-Lite, and CHI precisely, in one sentence each, and explain why ACE's broadcast snoop does not scale.

2. Why Should I Learn This?

You already know AXI (this series has used its stale-read bug since 1.5). You have probably assumed that "the bus" keeps the system consistent. It does not — AXI was never designed to, and understanding that gap precisely is what makes the entire CHI protocol make sense.

• AXI is per-master. Each master sees its own read/write channels to memory. AXI has no channel that carries "some other master has this line cached."
• No snoop path means no coherency. Coherency needs a write on one master to reach into another master's cache and invalidate it. AXI has no such wire, so the interconnect physically cannot do it — software pays the price by hand.
• This is CHI's starting line. ACE added coherency by broadcasting snoops; that collapses at scale. CHI replaces broadcast with distributed Home Nodes and directories. Knowing why AXI is not enough tells you exactly what CHI had to add.

3. Core Concept — AXI's per-master model, the missing snoop, and the AMBA progression

Start from the running project: CPU0 and CPU1, each with a private L1, now connected by an AXI interconnect to one shared memory.

What AXI actually provides. AXI is a point-to-point, per-master read/write protocol. Each master has its own five channels — write address (AW), write data (W), write response (B), read address (AR), read data (R) — routed by the interconnect to a slave (memory). The interconnect's only job is to transport a master's read/write to the right slave and return the response. It has a per-master view: it knows what this master asked for, not what other masters have cached. Crucially, there is no channel from the interconnect back into a master's cache — no AXI signal says "another master wrote line A, invalidate your copy." Coherency needs exactly that action, and AXI has no wire for it.

Why the bug survives. CPU0 writes shared line A. On a write-back cache the new value may sit in CPU0's L1; even when it reaches memory over AXI, the write terminates at the slave. Nothing snoops CPU1. CPU1's L1 still holds the old A, and its next load hits that stale copy — the identical stale read from 1.5, unfixed, because AXI has no coherency mechanism to fix it with.

The software workaround (and why it hurts). To share correctly on AXI, software must do cache maintenance by hand: the writer cleans/flushes its dirty lines to memory; the reader invalidates its stale lines so the next read misses and refetches. Correct, but expensive and fragile — one missed operation is a silent bug.

The AMBA progression — the ONE authoritative statement of the tiers.

• AXI — point-to-point read/write, per-master view, no coherency, no snoop. Great for non-shared or software-managed traffic.
• ACE (AXI Coherency Extensions) — adds snoop channels (address, response, data) and cache-state signalling on top of AXI so fully coherent masters snoop each other. Coherency is kept by broadcasting snoops to all coherent masters — correct, but broadcast traffic grows with master count, so it does not scale.
• ACE-Lite — a lighter profile for IO-coherent masters. An ACE-Lite master can snoop the fully coherent masters (so a DMA or accelerator reads up-to-date data) but holds no coherent cache of its own to be snooped.
• CHI (Coherent Hub Interface) — a packet/message-based protocol with distributed Home Nodes and directory tracking. Instead of broadcasting, a Home Node knows which masters hold a line and snoops only those, so coherency scales to dozens of agents.

One-line summary: AXI has no snoop; ACE adds broadcast snoop but does not scale; ACE-Lite lets IO masters snoop without being snooped; CHI makes snooping targeted and scalable with Home Nodes and directories.

4. Engineering Diagram

The edge from the interconnect back to CPU1's L1 is drawn only to name what is missing: on a real AXI fabric there is no such path. Coherency is precisely the action that arrow would carry — snoop CPU1's cache and invalidate its stale copy on CPU0's write — and AXI has no channel to put it on.

5. Real Silicon Story — the dual-core that shipped with a flush routine

An automotive SoC paired two Cortex-class cores over a standard AXI interconnect to a shared DDR controller. Each core had a private, write-back L1, and the cores exchanged sensor frames through a shared buffer in DDR. In bring-up, the consumer core kept processing old frames — the producer had clearly written new data, but the consumer read stale bytes. The interconnect was doing its job perfectly: when the producer wrote the buffer, the write reached the AXI interconnect and updated DDR (or sat dirty in the producer's write-back cache). Either way, nothing on AXI reached the consumer core's L1 to say the buffer changed, so its next load hit the stale line and the frame was processed again. AXI has no snoop channel, so the interconnect had no mechanism — and no obligation — to invalidate the consumer's line.

The software team fixed it the only way an AXI-only system can: manual cache maintenance. The producer cleans/flushes the buffer lines after writing; the consumer invalidates those lines before reading, forcing a refetch from DDR. It worked, but every shared access now carried explicit flush/invalidate code, a missed invalidate meant a silent stale-data bug, and the maintenance traffic cost real performance. The next SoC in the family moved the CPU cluster onto a coherent interconnect so the hardware did this automatically. The lesson: AXI moves data between a master and memory correctly, but it has no path into a peer's cache — so on an AXI-only cluster coherency becomes a software burden, not a hardware guarantee.

6. Worked Example — the write reaches memory, the peer stays stale

Trace the shared-buffer bug at the value level over an AXI interconnect. No coherency; software has not yet flushed or invalidated:

# Two private caches over an AXI interconnect to one shared memory.
# Initial: A = 5 everywhere. AXI carries reads/writes only — NO snoop path.
 
Cycle  Actor  Action                         L1(CPU0)  L1(CPU1)  Memory  Note
-----  -----  -----------------------------  --------  --------  ------  ------------------------------
  1    CPU0   AR/R read A (miss -> fetch)     A=5       -         A=5     fetched over AXI, cached
  2    CPU1   AR/R read A (miss -> fetch)     A=5       A=5       A=5     CPU1 caches its own copy
  3    CPU0   store A=6 (write-back, local)   A=6       A=5       A=5     dirty in CPU0's L1 only
  4    CPU0   AW/W write-back A=6 to memory   A=6       A=5       A=6     reaches MEMORY over AXI...
  5    (bus)  no snoop of CPU1               A=6       A=5       A=6     AXI has NO channel to CPU1's L1
  6    CPU1   read A (HIT own L1)             A=6       A=5       A=6     reads STALE 5  <-- BUG survives

Frame the same events the way CHI will later frame coherency — a requester, the shared line, and the snoop that is missing. Look closely at cycles 4 and 5. CPU0's write does reach memory over AXI: address on AW, data on W, and a B response after memory accepts it. But there is no cycle where CPU1's cache is touched, because AXI has no snoop channel to touch it with. CPU1 hits its own stale copy at cycle 6. Getting memory right is not enough; coherency requires reaching the peer cache, and AXI cannot.

7. RTL Illustration — a minimal AXI-lite write path with no snoop output

This is a simplified behavioral demonstrator of the AXI structural gap: a minimal AXI-lite-style write path that accepts an address/data handshake and updates a memory location. Read the port list carefully — there is no snoop or invalidate output anywhere. That absence is the point: an AXI write endpoint has no wire with which to tell a peer cache that a shared line changed.

// axi_write_no_snoop.sv — minimal AXI-lite-style single-location write path (simplified).
// Accepts AW/W handshakes and updates memory; returns a B response.
// STRUCTURAL POINT: there is NO snoop/invalidate output port at all,
// so this endpoint can never tell a peer cache that A changed.
module axi_write_no_snoop #(
  parameter int DW = 8
)(
  input  logic          clk,
  input  logic          rst_n,
  input  logic          awvalid,   // write address valid
  output logic          awready,
  input  logic          wvalid,    // write data valid
  input  logic [DW-1:0] wdata,
  output logic          wready,
  output logic          bvalid,    // write response
  input  logic          bready,
  output logic [DW-1:0] mem_A      // the only visible effect of a write
  // NOTE: no snoop_valid, no invalidate_addr, no peer-cache port exists.
);
  logic aw_done, w_done;
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      mem_A <= 8'd5; awready <= 1'b0; wready <= 1'b0;
      bvalid <= 1'b0; aw_done <= 1'b0; w_done <= 1'b0;
    end else begin
      awready <= awvalid & ~aw_done;          // accept address once
      wready  <= wvalid  & ~w_done;           // accept data once
      if (awvalid & awready) aw_done <= 1'b1;
      if (wvalid  & wready ) begin w_done <= 1'b1; mem_A <= wdata; end // write reaches MEMORY only
      if (aw_done & w_done & ~bvalid) bvalid <= 1'b1;                  // raise B when both accepted
      if (bvalid & bready) begin bvalid <= 1'b0; aw_done <= 1'b0; w_done <= 1'b0; end
      // There is deliberately NO code path that notifies any peer cache.
    end
  end
endmodule

Simulation note: driving one write of A=6 leaves mem_A=6 and returns a B response — the transport half of coherency — while no peer cache is ever notified, because the module has no snoop port to notify it with. That missing port is exactly what ACE and CHI add.

8. DebugLab — "the buffer updates in memory but the other core reads old data"

9. Common Mistakes

• "AXI guarantees coherency." No — AXI is a per-master read/write bus with no snoop channel and no cache-state tracking. It transports data to and from memory and makes no coherency promise whatsoever; coherency arrives only with ACE or CHI.
• Thinking a write reaching memory is enough. A peer cache holding the old line is still stale, and under write-back the newest value may not even be in memory yet. Coherency is about reconciling all cached copies, not just memory.
• Believing AXI has snoops. It does not — AXI has only AW/W/B/AR/R. Snoop channels arrive with ACE.
• "CHI is just AXI with more channels." No — CHI is a packet/message-based coherency protocol with distributed Home Nodes and directories. It is a different model built to snoop only the caches that hold a line, not a wider AXI bus.
• Confusing ACE and ACE-Lite. ACE masters are fully coherent (they snoop and are snooped). ACE-Lite masters are IO-coherent — they can snoop others but hold no coherent cache to be snooped.
• Assuming ACE's broadcast scales. Broadcasting a snoop to every coherent master grows with master count and saturates the fabric — the exact wall CHI's directories were built to avoid.

10. Interview Questions

11. Engineering Checklist

• I can state that AXI is per-master, point-to-point read/write with no snoop and therefore provides no coherency.
• I can show why an AXI write reaching memory still leaves a peer cache stale.
• I can name the AXI-only workaround — software clean/flush + invalidate — and why it is costly and fragile.
• I can distinguish AXI vs ACE vs ACE-Lite vs CHI in one sentence each.
• I can explain why ACE's broadcast snoop does not scale and what CHI replaces it with.
• I can map a master to the right tier: non-coherent (AXI), IO-coherent (ACE-Lite / RN-D), fully coherent (ACE / CHI RN-F).

12. Key Takeaways

• AXI is per-master and has no snoop. It transports point-to-point reads and writes between a master and memory and makes no coherency promise.
• The stale read survives AXI. A write reaching memory does not touch a peer cache, so the reader keeps hitting its old copy — the same bug as 1.5, unfixed.
• Software cache maintenance is the AXI workaround. The writer cleans/flushes and the reader invalidates by hand — correct but slow, fragile, and unscalable.
• ACE / ACE-Lite / CHI are the answer tiers. ACE adds broadcast snoop for fully coherent masters (does not scale); ACE-Lite adds IO-coherent masters; CHI uses distributed Home Nodes and directories to snoop only the caches that hold a line.
• This is why CHI exists. AXI cannot snoop, ACE snoops but cannot scale — CHI makes snooping targeted and scalable.

13. Quick Revision

Why AXI is not enough. AXI = per-master, point-to-point read/write, NO coherency, NO snoop. It moves data between a master and memory only — it has no channel into a peer's cache. So a write reaching memory leaves the other core's copy stale; the same stale read as 1.5 survives. AXI-only fix = software cache maintenance (writer clean/flush, reader invalidate) — correct but slow and fragile. The AMBA answer: ACE adds snoop channels plus cache states for fully coherent masters (broadcast snoop — does not scale); ACE-Lite = IO-coherent masters (snoop others, hold no coherent cache); CHI = packet/message-based with distributed Home Nodes plus directory, snooping only the caches that hold a line — scalable coherency. That is why CHI exists.

14. Coming Next

Next — 1.9 Introduction to Coherency. You will add:

• the formal coherency invariants — single-writer/multiple-reader, write propagation, value consistency — that an interconnect must uphold,
• the vocabulary of coherency actions — snoop, invalidate, update — that AXI lacked and ACE and CHI provide,
• the first framing of the coherency point, where the latest value is tracked, which later becomes the CHI Home Node.

We keep the same two-CPU project and turn from "why AXI cannot fix the stale read" into "what a coherency protocol must guarantee to fix it" — the contract every later CHI mechanism implements.