AMBA CHI · Module 2 · Coherency Protocol Foundations
MOESI Protocol
MESI still forces a cache holding dirty data to write it back to memory the moment another core reads it, even though the data is heading straight into another cache. MOESI removes that memory traffic by adding the Owned state: a dirty line that is also shared. The owner keeps the dirty data, supplies it to readers cache-to-cache, and defers the writeback until it gives the line up. This chapter builds the five-state machine, shows why memory is no longer the source of truth once an owner exists, traces the migratory read that MOESI accelerates, and implements it across SystemVerilog, Verilog, and VHDL. MOESI here is a representative model, not the exact CHI state set.
Foundation15 min readAMBA CHIMOESI ProtocolOwned StateCache-to-CacheCache CoherencyState Machine
Module 2 · Chapter 2.3 · Coherency Protocol Foundations
Project thread — MESI (2.2) made private read-modify-write cheap with Exclusive, but a shared read of dirty data still bounced through memory. MOESI adds Owned to forward dirty data directly. MESIF (2.4) will take the opposite tack — a designated forwarder for clean shared data.
1. Learning Outcomes
By the end of this chapter you should be able to:
- Explain the memory writeback MESI pays whenever dirty data is read-shared.
- Distinguish Owned (dirty and shared) from Modified (dirty and exclusive) and from Shared (clean).
- Trace a peer read that moves a line M to O with no writeback, and identify the data source.
- Identify which agent owns the dirty value, supplies readers, and performs the deferred writeback.
- Implement a representative MOESI tracker in SystemVerilog, Verilog-2001, and VHDL.
- Verify that a read-driven downgrade of Modified issues no writeback and that only one owner exists per line.
2. Why Should I Learn This?
Real workloads move dirty data between cores constantly: a producer writes a buffer and a consumer reads it; a lock or work-queue migrates between threads. Under MESI, every such hand-off writes the dirty line back to memory first, then both caches hold clean copies — a memory write on the critical path even though the data went cache-to-cache.
MOESI's Owned state removes that write. Dirty data can be shared directly, with one cache designated as the owner responsible for supplying it and eventually writing it back. It is the change that makes migratory and producer-consumer sharing efficient, and it forces a sharper idea of what "the current value" means.
3. Key Terms
4. Previous Chapter Connection
Chapter 2.2 added Exclusive so a private read-modify-write costs one transaction. But MESI kept MSI's rule that Shared implies clean — so when a Modified line is read by a peer, MESI must write it back to memory (M→S with writeback) to restore that invariant before both caches hold clean Shared copies.
That writeback is the waste MOESI targets. By allowing a dirty shared state, MOESI lets the owner keep the dirty data and hand a copy to the reader without touching memory. The price is giving up "Shared implies clean" — and being precise about who owns the current value.
5. Core Concept — dirty data can be shared
MOESI keeps M, E, S, I and inserts O between "I own dirty data alone" and "we share clean data":
| State | Copies elsewhere? | Clean/Dirty | May read? | May write? | Supplies readers? |
|---|---|---|---|---|---|
| M Modified | none | dirty | Yes | Yes | (becomes owner on read) |
| O Owned | yes (Shared) | dirty | Yes | No (upgrade first) | Yes (cache-to-cache) |
| E Exclusive | none | clean | Yes | Yes → silent M | — |
| S Shared | maybe | clean-looking | Yes | No (upgrade first) | — |
| I Invalid | — | — | No | No | — |
The payoff transition:
M → O on a peer read. The owner supplies the dirty data directly to the reader (who installs Shared) and stays dirty as Owned — no memory writeback. Memory is not touched; the write is deferred until the owner evicts or is invalidated.
Write permission is unchanged: only M is writable. To write an Owned line you must upgrade O→M, invalidating the Shared copies first. Owned is a read-and-supply role, not a write role.
6. Engineering Mental Model — the librarian with the master copy
Extend the token model:
- S — you hold a photocopy. Under MESI it matched the library's shelf copy; under MOESI it may match an owner's edited master instead.
- O — you hold the edited master (dirty) and have handed out photocopies. You are the librarian: readers get their copies from you, the shelf copy (memory) is out of date, and when you finally leave you must re-shelve the master (writeback).
- M — you hold the edited master and have handed out no copies; you may keep editing.
The librarian idea captures the two owner duties Owned adds: supply (answer reads from your master) and eventual re-shelving (the deferred writeback). Exactly one librarian per line — two would each think the other re-shelves.
7. Engineering Diagram — the MOESI state machine
The one edge to study is M→O: identical trigger to MESI's M→S (a peer read), but the result keeps the data dirty in the owner and skips the memory writeback.
8. Worked Example — a migratory read of dirty data
Two cores, line A. CPU0 has written A and holds it Modified (dirty, value new); memory holds the old value. CPU1 now reads A.
| Step | Action | CPU0 | CPU1 | Memory | Writeback? |
|---|---|---|---|---|---|
| 1 | (start) CPU0 wrote A | M (dirty) | I | stale | — |
| 2 | CPU1 reads A — MESI | M → S | I → S | updated | yes (M→S) |
| 2 | CPU1 reads A — MOESI | M → O | I → S | still stale | no (deferred) |
| 3 | CPU1 reads A again | O (supplies) | S (hit) | stale | no |
| 4 | CPU0 evicts A | O → I | S | updated | yes (deferred, once) |
MOESI performs one memory write — at eviction (Step 4) — no matter how many times the dirty line is read-shared in between. MESI would write memory at Step 2 and again if the line went dirty later. For a producer read repeatedly by consumers, that is a large saving.
9. Transaction Walkthrough — the peer read that creates an owner
CPU1's read of a line CPU0 holds Modified, mapped onto the CHI cast from Module 1. Representative behavioral flow, not a byte-level trace.
- CPU1 pipeline → RN1 → Home Node. A load misses (state I). RN1 requests a readable copy of A from the HN that owns A's range.
- HN directory lookup. The directory shows CPU0 holds A Modified (the current, dirty value). The data source is therefore CPU0, not memory — memory is stale.
- HN → RN0 (snoop): read snoop. Purpose: obtain the current data and let CPU1 share it. CPU0 transitions M → O, keeps the dirty data, and supplies it.
- Data forwarded to CPU1 (cache-to-cache). CPU1 installs A as Shared. Memory is not written. The directory now records CPU0 as Owned and CPU1 as Shared.
- Deferred writeback later. When CPU0 evicts A, or when a peer store invalidates it, CPU0 (the owner) performs the single writeback that finally updates memory.
10. RTL / Hardware View — a per-line MOESI tracker
A representative single-line MOESI tracker. It extends the MESI tracker of 2.2 with the O state, a do_supply output (cache-to-cache data forward), and a peer read that moves M→O without a writeback. Behavioral and simplified: one event per cycle, one line, no data path.
// Representative single-line MOESI coherence tracker (educational, not CHI RTL).
// State: I=000, S=001, E=010, O=011 (dirty+shared), M=100 (dirty+exclusive).
module moesi_line_tracker (
input logic clk,
input logic rst_n,
input logic req_load, // local read
input logic req_store, // local write
input logic snoop_read, // a peer wants a shared (read) copy
input logic snoop_inval, // a peer wants to write / read-exclusive
input logic shared_in, // read-miss verdict: 1 = an owner/sharer holds the line
output logic [2:0] state,
output logic do_busread, // fetch a copy (from the owner if one exists, else memory)
output logic do_invalidate,// ask the HN to invalidate peer copies
output logic do_writeback, // deferred: flush dirty data when giving the line up
output logic do_supply, // forward this cache's data to a reader (cache-to-cache)
output logic can_read,
output logic can_write
);
localparam logic [2:0] I = 3'b000, S = 3'b001, E = 3'b010, O = 3'b011, M = 3'b100;
logic [2:0] next;
always_comb begin
next = state;
do_busread = 1'b0;
do_invalidate = 1'b0;
do_writeback = 1'b0;
do_supply = 1'b0;
if (snoop_inval) begin
if (state == M || state == O) do_writeback = 1'b1; // dirty: deferred wb realised here
next = I;
end
else if (snoop_read) begin
if (state == M) begin
do_supply = 1'b1; // forward dirty data cache-to-cache
next = O; // M -> O: stay dirty, NO writeback
end else if (state == O) begin
do_supply = 1'b1; // owner keeps supplying readers
end else if (state == E) begin
next = S; // clean downgrade
end
end
else if (req_store) begin
unique case (state)
M: next = M; // write hit
E: next = M; // silent upgrade
O: begin do_invalidate = 1'b1; next = M; end // owner upgrade: invalidate sharers
S: begin do_invalidate = 1'b1; next = M; end // shared upgrade
default: begin do_busread = 1'b1; do_invalidate = 1'b1; next = M; end // I: write miss
endcase
end
else if (req_load) begin
if (state == I) begin
do_busread = 1'b1;
next = shared_in ? S : E; // owner/sharer present -> Shared; else Exclusive
end
end
end
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) state <= I;
else state <= next;
assign can_read = (state != I);
assign can_write = (state == M);
endmoduleThe same behavior in Verilog-2001:
// Representative single-line MOESI tracker (Verilog-2001).
module moesi_line_tracker (
input clk,
input rst_n,
input req_load,
input req_store,
input snoop_read,
input snoop_inval,
input shared_in,
output reg [2:0] state,
output reg do_busread,
output reg do_invalidate,
output reg do_writeback,
output reg do_supply,
output can_read,
output can_write
);
localparam I = 3'b000, S = 3'b001, E = 3'b010, O = 3'b011, M = 3'b100;
reg [2:0] next;
always @(*) begin
next = state; do_busread = 1'b0; do_invalidate = 1'b0; do_writeback = 1'b0; do_supply = 1'b0;
if (snoop_inval) begin
if (state == M || state == O) do_writeback = 1'b1;
next = I;
end else if (snoop_read) begin
if (state == M) begin do_supply = 1'b1; next = O; end // M->O: no writeback
else if (state == O) do_supply = 1'b1;
else if (state == E) next = S;
end else if (req_store) begin
case (state)
M: next = M;
E: next = M; // silent
O: begin do_invalidate = 1'b1; next = M; end
S: begin do_invalidate = 1'b1; next = M; end
default: begin do_busread = 1'b1; do_invalidate = 1'b1; next = M; end
endcase
end else if (req_load) begin
if (state == I) begin
do_busread = 1'b1;
next = shared_in ? S : E;
end
end
end
always @(posedge clk or negedge rst_n)
if (!rst_n) state <= I; else state <= next;
assign can_read = (state != I);
assign can_write = (state == M);
endmoduleAnd in VHDL:
-- Representative single-line MOESI tracker (VHDL).
library ieee;
use ieee.std_logic_1164.all;
entity moesi_line_tracker is
port (
clk, rst_n : in std_logic;
req_load, req_store : in std_logic;
snoop_read, snoop_inval : in std_logic;
shared_in : in std_logic;
state : out std_logic_vector(2 downto 0);
do_busread : out std_logic;
do_invalidate : out std_logic;
do_writeback : out std_logic;
do_supply : out std_logic;
can_read, can_write : out std_logic
);
end entity;
architecture rtl of moesi_line_tracker is
constant I : std_logic_vector(2 downto 0) := "000";
constant S : std_logic_vector(2 downto 0) := "001";
constant E : std_logic_vector(2 downto 0) := "010";
constant O : std_logic_vector(2 downto 0) := "011";
constant M : std_logic_vector(2 downto 0) := "100";
signal cur, nxt : std_logic_vector(2 downto 0);
begin
comb : process(cur, req_load, req_store, snoop_read, snoop_inval, shared_in)
begin
nxt <= cur; do_busread <= '0'; do_invalidate <= '0'; do_writeback <= '0'; do_supply <= '0';
if snoop_inval = '1' then
if cur = M or cur = O then do_writeback <= '1'; end if;
nxt <= I;
elsif snoop_read = '1' then
if cur = M then do_supply <= '1'; nxt <= O; -- M->O: no writeback
elsif cur = O then do_supply <= '1';
elsif cur = E then nxt <= S; end if;
elsif req_store = '1' then
if cur = M then nxt <= M;
elsif cur = E then nxt <= M; -- silent
elsif cur = O then do_invalidate <= '1'; nxt <= M;
elsif cur = S then do_invalidate <= '1'; nxt <= M;
else do_busread <= '1'; do_invalidate <= '1'; nxt <= M; end if;
elsif req_load = '1' then
if cur = I then
do_busread <= '1';
if shared_in = '1' then nxt <= S; else nxt <= E; end if;
end if;
end if;
end process;
seq : process(clk, rst_n)
begin
if rst_n = '0' then cur <= I;
elsif rising_edge(clk) then cur <= nxt; end if;
end process;
state <= cur;
can_read <= '0' when cur = I else '1';
can_write <= '1' when cur = M else '0';
end architecture;All three model the identical machine: M→O supply with no writeback, owner upgrade by invalidation, and a single deferred writeback when the dirty line is finally surrendered.
11. Timing View — the read that skips memory
CPU1 reads a line CPU0 holds Modified. Watch memory stay stale while the owner supplies the reader. Timing is representative — real CHI latencies are not fixed cycle counts.
Migratory read — dirty data forwarded cache-to-cache, no memory writeback
6 cyclesUnder MESI the mem_wb row would show a write at t2. Owned is exactly the state that lets the dirty data move sideways instead of down to memory.
12. Verification View — no writeback on a read, one owner per line
Three properties pin the Owned behaviour down.
// Bind to moesi_line_tracker. Encodings: O=3'b011, M=3'b100.
// 1. Write permission exists only in Modified.
property p_write_only_in_M;
@(posedge clk) disable iff (!rst_n) can_write |-> (state == 3'b100);
endproperty
assert property (p_write_only_in_M);
// 2. A peer READ of a Modified line supplies data and does NOT write back.
property p_read_no_writeback;
@(posedge clk) disable iff (!rst_n)
(state == 3'b100) && snoop_read |-> (do_supply && !do_writeback);
endproperty
assert property (p_read_no_writeback);
// 3. Losing a DIRTY line (O or M) to a writer flushes it — the deferred writeback.
property p_deferred_writeback;
@(posedge clk) disable iff (!rst_n)
(state == 3'b011 || state == 3'b100) && snoop_inval |-> do_writeback;
endproperty
assert property (p_deferred_writeback);The system invariants live in a scoreboard or directory model:
For each line: at most one owner —
oCount + mCount <= 1— and a read miss is sourced from that owner whenever one exists (never from stale memory).
- What it proves: the owner supplies readers without a memory write, exactly one cache owns the dirty value, and the deferred writeback is not skipped when ownership is lost.
- What it does not prove: that a read miss actually chose the owner as its data source — that decision lives in the Home Node's routing, not in this tracker. It also does not prove cross-address consistency (Module 12).
- Bug signature when it fails:
do_writebackasserted onM && snoop_read(lost MOESI's benefit),oCount + mCount == 2(two owners, ambiguous writeback), or a reader served stale memory while an owner existed (below).
13. Testbench — drive the supply-and-share, then the deferred flush
Deterministic stimulus; actions sampled while inputs are asserted (pre-edge), state checked after the edge — no sampling race.
module tb_moesi_line_tracker;
logic clk = 0, rst_n;
logic req_load, req_store, snoop_read, snoop_inval, shared_in;
logic [2:0] state;
logic do_busread, do_invalidate, do_writeback, do_supply, can_read, can_write;
int errors = 0;
moesi_line_tracker dut (.*);
always #5 clk = ~clk;
// Apply one event; check pre-edge actions, then post-edge state.
task automatic ev(input logic ld, st, sr, si, sh,
input logic [2:0] exp_state,
input logic exp_br, exp_iv, exp_wb, exp_sp,
input string tag);
logic br, iv, wb, sp;
req_load = ld; req_store = st; snoop_read = sr; snoop_inval = si; shared_in = sh;
#1;
br = do_busread; iv = do_invalidate; wb = do_writeback; sp = do_supply;
if (br !== exp_br || iv !== exp_iv || wb !== exp_wb || sp !== exp_sp) begin
errors++;
$display("FAIL [%s] br/iv/wb/sp = %b/%b/%b/%b (exp %b/%b/%b/%b)",
tag, br, iv, wb, sp, exp_br, exp_iv, exp_wb, exp_sp);
end
@(posedge clk); #1;
req_load = 0; req_store = 0; snoop_read = 0; snoop_inval = 0; shared_in = 0;
if (state !== exp_state) begin
errors++;
$display("FAIL [%s] state=%0d exp=%0d", tag, state, exp_state);
end else
$display("PASS [%s] state=%0d br/iv/wb/sp=%b/%b/%b/%b", tag, exp_state, br, iv, wb, sp);
endtask
initial begin
rst_n = 0; ev(0,0,0,0,0, 3'b000, 0,0,0,0, "reset");
rst_n = 1;
// Become the dirty owner: I -> M (write miss) -> stays M
ev(0,1,0,0,0, 3'b100, 1,1,0,0, "store miss: I->M");
// Peer read: M -> O, SUPPLY, no writeback (the MOESI win)
ev(0,0,1,0,0, 3'b011, 0,0,0,1, "peer read: M->O (supply, no wb)");
// Owner keeps supplying further readers, stays O
ev(0,0,1,0,0, 3'b011, 0,0,0,1, "peer read: O->O (supply)");
// Owner upgrades to write: invalidate sharers, O -> M
ev(0,1,0,0,0, 3'b100, 0,1,0,0, "store: O->M (upgrade)");
// Peer store takes it for write: deferred writeback, M -> I
ev(0,0,0,1,0, 3'b000, 0,0,1,0, "peer store: M->I (deferred wb)");
// Sole read -> Exclusive; a peer read then downgrades clean E -> S (no wb, no supply)
ev(1,0,0,0,0, 3'b010, 1,0,0,0, "load sole: I->E");
ev(0,0,1,0,0, 3'b001, 0,0,0,0, "peer read: E->S (clean)");
// Owner path again then lose to inval while Owned: writeback
ev(0,1,0,0,0, 3'b100, 0,1,0,0, "store: S->M (upgrade)");
ev(0,0,1,0,0, 3'b011, 0,0,0,1, "peer read: M->O (supply)");
ev(0,0,0,1,0, 3'b000, 0,0,1,0, "peer store: O->I (deferred wb)");
if (errors == 0) $display("ALL TESTS PASSED");
else $display("%0d FAILURE(S)", errors);
$finish;
end
endmoduleExpected output:
PASS [reset] state=0 br/iv/wb/sp=0/0/0/0
PASS [store miss: I->M] state=4 br/iv/wb/sp=1/1/0/0
PASS [peer read: M->O (supply, no wb)] state=3 br/iv/wb/sp=0/0/0/1
PASS [peer read: O->O (supply)] state=3 br/iv/wb/sp=0/0/0/1
PASS [store: O->M (upgrade)] state=4 br/iv/wb/sp=0/1/0/0
PASS [peer store: M->I (deferred wb)] state=0 br/iv/wb/sp=0/0/1/0
PASS [load sole: I->E] state=2 br/iv/wb/sp=1/0/0/0
PASS [peer read: E->S (clean)] state=1 br/iv/wb/sp=0/0/0/0
PASS [store: S->M (upgrade)] state=4 br/iv/wb/sp=0/1/0/0
PASS [peer read: M->O (supply)] state=3 br/iv/wb/sp=0/0/0/1
PASS [peer store: O->I (deferred wb)] state=0 br/iv/wb/sp=0/0/1/0
ALL TESTS PASSED14. DebugLab — a reader served stale memory while an owner existed
A reader served stale memory while an owner existed
READ SOURCED FROM STALE MEMORY INSTEAD OF THE OWNER -> WRONG DATAA consumer core reads a producer's buffer and gets an old value, intermittently — even though no error is reported and coherency "looks" fine (states are legal). It reproduces only when the producer's line is Owned or Modified at read time.
The read miss and its data source:
cyc core event state data source value
3 CPU1 load miss I -> S MEMORY 5 (old) <-- sourced from stale memory
- CPU0 (holds A Owned, value = 9, dirty)The reference model flags it: A read of A was sourced from memory while oCount == 1 (CPU0 Owned). Memory held 5; the owner held 9.
Cycle 3, the read's data source — memory — is the earliest wrong event. The reader's own I→S transition is legal; the defect is where the data came from, decided upstream at the Home Node, not in the reader's state machine.
The read was routed to memory instead of snooping the owner. The implementation still assumed "memory is current for Shared lines," which held under MESI but is false under MOESI: once an O copy exists, memory is stale and the owner holds the current value. Skipping the owner snoop returned the old value.
Source a read from the owner whenever the directory shows an O or M copy — snoop the owner for a cache-to-cache supply, and only fall back to memory when no owner exists. This is the whole point of Owned: the owner, not memory, answers reads. Do not "fix" it by writing the owner back to memory on every read — that reintroduces exactly the MESI traffic MOESI removes; route the read to the owner instead.
15. Common Mistakes
- Sourcing a read from memory when an owner exists. Assumption: memory is current for Shared lines. Bug: readers get stale data (the DebugLab). Prevention: an O/M owner is the data source; snoop it.
- Writing back on M→O (peer read). Assumption: sharing dirty data requires flushing it first. Bug: the memory traffic MOESI exists to remove comes right back. Prevention: M→O supplies and stays dirty; no writeback on a read.
- Treating Owned as clean. Assumption: only Modified is dirty. Bug: the deferred writeback is skipped on eviction and the newest value is lost. Prevention: O is dirty; surrendering it always writes back.
- Allowing two owners. Assumption: any dirty copy can supply. Bug: ambiguous writeback responsibility, double or missing writeback. Prevention: exactly one owner (O or M) per line.
- Assuming Shared implies memory is current. Assumption: an S copy matches memory. Bug: false whenever an O owner coexists. Prevention: S may be backed by an owner, not memory.
- Treating MOESI as the CHI state set. Assumption: these five states are the specification. Bug: confusion when CHI's Shared-Dirty / Unique states appear. Prevention: MOESI is the concept; the CHI states are later modules.
16. Engineering Checklist
- A peer read of M moves to O, supplies data cache-to-cache, and does not write back.
- A read miss is sourced from the owner (O or M) when one exists, not from memory.
- Surrendering a dirty line (O or M → I) performs the deferred writeback.
- Only M is writable; O upgrades to M by invalidating sharers.
- At most one owner per line;
can_writeonly in M. - Shared copies are treated as possibly owner-backed, never assumed memory-current.
17. Key Takeaways
- MOESI adds Owned — a dirty, shared line whose owner supplies readers and owes the writeback.
- A peer read of Modified becomes Owned: cache-to-cache supply, no memory writeback.
- Memory is stale whenever an O or M copy exists — the owner, not memory, is the source of truth.
- The dirty writeback is deferred to the moment the owner surrenders the line — one write, not one per read.
- Only M is writable; O is a read-and-supply role that upgrades to M by invalidating sharers.
- MOESI here is representative — the AMD-style dirty-sharing refinement; MESIF (2.4) refines clean sharing instead.
18. Quick Revision
MOESI = MESI + Owned. Five states: M (dirty, exclusive, writable), O (dirty, shared, supplies readers, owes writeback), E (clean, exclusive), S (clean-looking, shared), I (none). Peer read of M → O: supply cache-to-cache, no writeback. Read miss sourced from the owner when one exists — memory is stale under O/M. Deferred writeback when the owner surrenders the line (eviction or invalidation). Store: only from M directly; O/S upgrade by invalidating sharers; E silent. One owner per line;
can_writeonly in M. Representative model, not the CHI spec state set.
Coming Next
Chapter 2.4 — MESIF Protocol. MOESI shares dirty data efficiently. MESIF tackles the other overhead — when several caches share a clean line, which one answers a new reader? MESIF adds the Forward state to designate a single forwarder, so a read miss gets one fast cache-to-cache response instead of a memory fetch or a broadcast of duplicate replies.