AMBA CHI · Module 2 · Coherency Protocol Foundations
MESIF Protocol
MESI and MOESI make dirty data efficient, but leave a clean question: when several caches share the same clean line, who answers a new reader? If memory answers, a fast cache-to-cache transfer is wasted; if every sharer answers, the interconnect drowns in duplicate replies. MESIF adds the Forward state to pick exactly one responder. One sharer holds the line in Forward and supplies read misses; the rest stay quietly Shared, and the forwarder migrates to the newest reader for locality. This chapter builds the five-state machine, contrasts Forward with MOESI's Owned, and implements it across SystemVerilog, Verilog, and VHDL. MESIF here is a representative model, not the exact CHI state set.
Intermediate15 min readAMBA CHIMESIF ProtocolForward StateCache-to-CacheCache CoherencyState Machine
Module 2 · Chapter 2.4 · Coherency Protocol Foundations
Project thread — MOESI (2.3) shared dirty data with an owner. MESIF solves the mirror problem for clean data: which of many identical sharers responds to a read. Chapter 2.5 then consolidates what "ownership" means across all four protocols.
1. Learning Outcomes
By the end of this chapter you should be able to:
- Explain the duplicate-response and slow-memory problem MESI leaves for clean shared data.
- Distinguish Forward (clean, shared, designated responder) from MOESI's Owned (dirty, shared) and from plain Shared (clean, silent).
- Trace a read miss served by the Forward holder, and the forwarder migrating to the new reader.
- Identify why Forward needs no writeback and can be silently dropped, and what that costs.
- Implement a representative MESIF tracker in SystemVerilog, Verilog-2001, and VHDL.
- Verify that at most one Forward holder exists per line and that a store from Forward invalidates the sharers.
2. Why Should I Learn This?
On a modern many-core chip, the busiest lines are often clean and widely shared — read-only code, constants, lookup tables. Many caches hold identical copies. When a new core reads one, MESI has no good answer: memory can supply it (slow, and a cache next door already has it), or every sharer can supply it (a storm of duplicate responses on the interconnect).
MESIF's Forward state fixes this by electing one responder. It is the change that makes clean sharing scale on fast point-to-point fabrics, and it sharpens a key idea — that answering a read is a role a single cache holds, not something every copy does.
3. Key Terms
4. Previous Chapter Connection
Chapter 2.3 added Owned so dirty data can be shared cache-to-cache with one owner responsible for the value and its writeback. But Owned solves the dirty case. For clean shared data — where memory is already current — MESI still either fetches from memory or lets every Shared copy respond.
MESIF is the clean-data counterpart to Owned. Where Owned designates who holds the dirty value, Forward designates who answers reads for a clean value. Same instinct — one cache carries a responsibility the others do not — applied to the opposite (clean) case.
5. Core Concept — one responder for clean shared data
MESIF keeps M, E, S, I and inserts F, a specialized Shared:
| State | Copies elsewhere? | Clean/Dirty | May read? | May write? | Answers read misses? |
|---|---|---|---|---|---|
| M Modified | none | dirty | Yes | Yes | (writes back, downgrades) |
| E Exclusive | none | clean | Yes | Yes → silent M | supplies (sole holder) |
| F Forward | yes (Shared) | clean | Yes | No (upgrade first) | Yes (the one responder) |
| S Shared | yes | clean | Yes | No (upgrade first) | No (silent) |
| I Invalid | — | — | No | No | — |
Two rules define F:
One forwarder. Of all the caches sharing a clean line, exactly one holds it in F and answers reads; the rest hold S and stay silent. A read miss gets a single, fast cache-to-cache response.
F migrates to the newest reader. When the F holder supplies a read, it demotes to S, and the requester installs the line as F. The most recently accessed copy is the responder — good locality, and it spreads the forwarding work.
F is clean — memory is current — so leaving F (a peer store, or eviction) needs no writeback. That also means F can be dropped silently on eviction, temporarily leaving no forwarder until a later read re-establishes one.
6. Engineering Mental Model — the on-call librarian
Reuse the librarian, but for clean copies:
- Many readers hold identical photocopies (S) of a shelf document that matches the master (memory is current).
- One of them is on call (F): when a new reader asks, the on-call librarian hands over a copy — faster than walking to the shelf (memory). The others stay quiet so the new reader isn't buried in duplicate handoffs.
- After handing over, the new reader takes the pager (F), and the previous on-call reverts to an ordinary holder (S). The pager follows the freshest reader.
- Nobody edited anything, so no re-shelving (writeback) is ever needed — and if the on-call reader leaves, they can just go; the next request falls back to the shelf until someone new picks up the pager.
7. Engineering Diagram — the MESIF state machine
The edge to study is F→S: a peer read makes the forwarder supply the data and step down, while the reader (transitioning I→F) picks up the forwarder role.
8. Worked Example — clean sharing with and without Forward
Line A is clean; memory is current. CPU0 already holds A; CPU1 now reads it, then CPU2 reads it.
Without Forward (plain MESI).
| Step | Action | Data source | Interconnect cost |
|---|---|---|---|
| 1 | CPU1 reads A | memory (or all sharers) | slow fetch, or duplicate replies |
| 2 | CPU2 reads A | memory (or all sharers) | slow fetch, or duplicate replies |
With Forward (MESIF).
| Step | Action | CPU0 | CPU1 | CPU2 | Data source |
|---|---|---|---|---|---|
| 0 | CPU0 holds A | F | — | — | — |
| 1 | CPU1 reads A | F → S | I → F | — | CPU0 (one response) |
| 2 | CPU2 reads A | S | F → S | I → F | CPU1 (one response) |
Every read gets exactly one fast cache-to-cache response, and the forwarder rides along with the newest reader. No memory fetch, no duplicate replies, and — because everything is clean — no writeback anywhere.
9. Transaction Walkthrough — the read that moves the forwarder
CPU1's read of a clean line CPU0 forwards, mapped onto the CHI cast from Module 1. Representative behavioral flow, not a byte-level trace.
- CPU1 pipeline → RN1 → Home Node. A load misses (state I). RN1 requests a readable copy of A.
- HN directory lookup. The directory shows clean sharers, with CPU0 as the forwarder (F). Memory is current, but a cache response is faster, so the HN targets the forwarder.
- HN → RN0 (snoop): read snoop. CPU0 (F) supplies the data cache-to-cache and demotes F→S — it is no longer the designated responder.
- Data forwarded to CPU1. CPU1 installs A as F — the new forwarder. The directory records CPU1 as F, CPU0 as S. No writeback (all clean).
- If the forwarder had been gone. Had CPU0 silently evicted its F copy, no cache would answer; the HN sources the read from memory (still correct, just slower) and makes CPU1 the new F.
10. RTL / Hardware View — a per-line MESIF tracker
A representative single-line MESIF tracker. It extends the MESI tracker of 2.2 with the F state and a do_supply output. The reader becomes F when clean sharers exist; the F holder supplies and demotes to S on a peer read. Behavioral and simplified: one event per cycle, one line, no data path.
// Representative single-line MESIF coherence tracker (educational, not CHI RTL).
// State: I=000, S=001, E=010, F=011 (clean, designated forwarder), M=100 (dirty).
module mesif_line_tracker (
input logic clk,
input logic rst_n,
input logic req_load, // local read
input logic req_store, // local write
input logic snoop_read, // a peer wants a shared (read) copy
input logic snoop_inval, // a peer wants to write / read-exclusive
input logic shared_in, // read-miss verdict: 1 = clean sharer(s) already hold the line
output logic [2:0] state,
output logic do_busread, // fetch a copy (from the forwarder if present, else memory)
output logic do_invalidate,// ask the HN to invalidate peer copies
output logic do_writeback, // flush dirty data (only from M)
output logic do_supply, // forward this cache's data to a reader (cache-to-cache)
output logic can_read,
output logic can_write
);
localparam logic [2:0] I = 3'b000, S = 3'b001, E = 3'b010, F = 3'b011, M = 3'b100;
logic [2:0] next;
always_comb begin
next = state;
do_busread = 1'b0;
do_invalidate = 1'b0;
do_writeback = 1'b0;
do_supply = 1'b0;
if (snoop_inval) begin
if (state == M) do_writeback = 1'b1; // only M is dirty; E/F/S are clean
next = I;
end
else if (snoop_read) begin
if (state == F) begin
do_supply = 1'b1; // the forwarder answers, then hands off
next = S; // F -> S: reader becomes the new forwarder
end else if (state == E) begin
do_supply = 1'b1; // sole clean holder supplies
next = S;
end else if (state == M) begin
do_supply = 1'b1; // supply the data...
do_writeback = 1'b1; // ...and flush: MESIF shares only clean
next = S;
end
// S: stays S and stays SILENT (does not respond) ; I: nothing
end
else if (req_store) begin
unique case (state)
M: next = M; // write hit
E: next = M; // silent upgrade
F: begin do_invalidate = 1'b1; next = M; end // forwarder upgrade: invalidate sharers
S: begin do_invalidate = 1'b1; next = M; end // shared upgrade
default: begin do_busread = 1'b1; do_invalidate = 1'b1; next = M; end // I: write miss
endcase
end
else if (req_load) begin
if (state == I) begin
do_busread = 1'b1;
next = shared_in ? F : E; // clean sharers exist -> become the forwarder; else Exclusive
end
end
end
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) state <= I;
else state <= next;
assign can_read = (state != I);
assign can_write = (state == M);
endmoduleThe same behavior in Verilog-2001:
// Representative single-line MESIF tracker (Verilog-2001).
module mesif_line_tracker (
input clk,
input rst_n,
input req_load,
input req_store,
input snoop_read,
input snoop_inval,
input shared_in,
output reg [2:0] state,
output reg do_busread,
output reg do_invalidate,
output reg do_writeback,
output reg do_supply,
output can_read,
output can_write
);
localparam I = 3'b000, S = 3'b001, E = 3'b010, F = 3'b011, M = 3'b100;
reg [2:0] next;
always @(*) begin
next = state; do_busread = 1'b0; do_invalidate = 1'b0; do_writeback = 1'b0; do_supply = 1'b0;
if (snoop_inval) begin
if (state == M) do_writeback = 1'b1;
next = I;
end else if (snoop_read) begin
if (state == F) begin do_supply = 1'b1; next = S; end // hand off
else if (state == E) begin do_supply = 1'b1; next = S; end
else if (state == M) begin do_supply = 1'b1; do_writeback = 1'b1; next = S; end
end else if (req_store) begin
case (state)
M: next = M;
E: next = M; // silent
F: begin do_invalidate = 1'b1; next = M; end
S: begin do_invalidate = 1'b1; next = M; end
default: begin do_busread = 1'b1; do_invalidate = 1'b1; next = M; end
endcase
end else if (req_load) begin
if (state == I) begin
do_busread = 1'b1;
next = shared_in ? F : E;
end
end
end
always @(posedge clk or negedge rst_n)
if (!rst_n) state <= I; else state <= next;
assign can_read = (state != I);
assign can_write = (state == M);
endmoduleAnd in VHDL:
-- Representative single-line MESIF tracker (VHDL).
library ieee;
use ieee.std_logic_1164.all;
entity mesif_line_tracker is
port (
clk, rst_n : in std_logic;
req_load, req_store : in std_logic;
snoop_read, snoop_inval : in std_logic;
shared_in : in std_logic;
state : out std_logic_vector(2 downto 0);
do_busread : out std_logic;
do_invalidate : out std_logic;
do_writeback : out std_logic;
do_supply : out std_logic;
can_read, can_write : out std_logic
);
end entity;
architecture rtl of mesif_line_tracker is
constant I : std_logic_vector(2 downto 0) := "000";
constant S : std_logic_vector(2 downto 0) := "001";
constant E : std_logic_vector(2 downto 0) := "010";
constant F : std_logic_vector(2 downto 0) := "011";
constant M : std_logic_vector(2 downto 0) := "100";
signal cur, nxt : std_logic_vector(2 downto 0);
begin
comb : process(cur, req_load, req_store, snoop_read, snoop_inval, shared_in)
begin
nxt <= cur; do_busread <= '0'; do_invalidate <= '0'; do_writeback <= '0'; do_supply <= '0';
if snoop_inval = '1' then
if cur = M then do_writeback <= '1'; end if;
nxt <= I;
elsif snoop_read = '1' then
if cur = F then do_supply <= '1'; nxt <= S; -- hand off forwarder role
elsif cur = E then do_supply <= '1'; nxt <= S;
elsif cur = M then do_supply <= '1'; do_writeback <= '1'; nxt <= S; end if;
elsif req_store = '1' then
if cur = M then nxt <= M;
elsif cur = E then nxt <= M; -- silent
elsif cur = F then do_invalidate <= '1'; nxt <= M;
elsif cur = S then do_invalidate <= '1'; nxt <= M;
else do_busread <= '1'; do_invalidate <= '1'; nxt <= M; end if;
elsif req_load = '1' then
if cur = I then
do_busread <= '1';
if shared_in = '1' then nxt <= F; else nxt <= E; end if;
end if;
end if;
end process;
seq : process(clk, rst_n)
begin
if rst_n = '0' then cur <= I;
elsif rising_edge(clk) then cur <= nxt; end if;
end process;
state <= cur;
can_read <= '0' when cur = I else '1';
can_write <= '1' when cur = M else '0';
end architecture;All three model the identical machine: the reader becomes the forwarder, the F holder supplies and hands off, and only Modified writes back.
11. Timing View — the forwarder migrates
CPU1 reads a clean line CPU0 forwards. Watch the F role move from CPU0 to CPU1 with one response and no writeback. Timing is representative — real interconnect latencies are not fixed cycle counts.
Clean shared read — one forwarder answers, then the role migrates
6 cyclesUnder plain MESI the rd_src at t2 would be memory (or every sharer at once). Forward turns clean sharing into a single, migrating cache response.
12. Verification View — at most one forwarder, and it stays clean
Three properties pin the Forward behaviour down.
// Bind to mesif_line_tracker. Encodings: F=3'b011, M=3'b100.
// 1. Write permission exists only in Modified.
property p_write_only_in_M;
@(posedge clk) disable iff (!rst_n) can_write |-> (state == 3'b100);
endproperty
assert property (p_write_only_in_M);
// 2. Forward is clean — leaving F (peer store or read hand-off) never writes back.
property p_forward_clean;
@(posedge clk) disable iff (!rst_n) (state == 3'b011) |-> !do_writeback;
endproperty
assert property (p_forward_clean);
// 3. On a peer read, the forwarder answers this cycle...
property p_forward_supplies;
@(posedge clk) disable iff (!rst_n) (state == 3'b011) && snoop_read |-> do_supply;
endproperty
assert property (p_forward_supplies);
// ...and hands off: next cycle it is Shared, not Forward (exactly one forwarder).
property p_forward_hands_off;
@(posedge clk) disable iff (!rst_n)
(state == 3'b011) && snoop_read |=> (state == 3'b001);
endproperty
assert property (p_forward_hands_off);The system invariant lives in a scoreboard or directory model:
For each clean-shared line: at most one cache in F —
fCount <= 1— and every other sharer is silent S. A read miss is answered by that one F holder, or by memory if none exists.
- What it proves: exactly one responder for clean shared data, no duplicate replies, and F never triggers a writeback.
- What it does not prove: that the interconnect actually routed the read to the F holder (a Home Node / snoop-filter decision), nor that a silently-dropped F is handled — the "no cache answered, use memory" fallback must be checked where the read is sourced.
- Bug signature when it fails:
fCount == 2(two forwarders → duplicate responses),do_writebackfrom F (treated Forward as dirty), or a read that got no response because the sole F was evicted and the fallback to memory was missing.
13. Testbench — drive the hand-off and the upgrade
Deterministic stimulus; actions sampled while inputs are asserted (pre-edge), state checked after the edge — no sampling race.
module tb_mesif_line_tracker;
logic clk = 0, rst_n;
logic req_load, req_store, snoop_read, snoop_inval, shared_in;
logic [2:0] state;
logic do_busread, do_invalidate, do_writeback, do_supply, can_read, can_write;
int errors = 0;
mesif_line_tracker dut (.*);
always #5 clk = ~clk;
// Apply one event; check pre-edge actions, then post-edge state.
task automatic ev(input logic ld, st, sr, si, sh,
input logic [2:0] exp_state,
input logic exp_br, exp_iv, exp_wb, exp_sp,
input string tag);
logic br, iv, wb, sp;
req_load = ld; req_store = st; snoop_read = sr; snoop_inval = si; shared_in = sh;
#1;
br = do_busread; iv = do_invalidate; wb = do_writeback; sp = do_supply;
if (br !== exp_br || iv !== exp_iv || wb !== exp_wb || sp !== exp_sp) begin
errors++;
$display("FAIL [%s] br/iv/wb/sp = %b/%b/%b/%b (exp %b/%b/%b/%b)",
tag, br, iv, wb, sp, exp_br, exp_iv, exp_wb, exp_sp);
end
@(posedge clk); #1;
req_load = 0; req_store = 0; snoop_read = 0; snoop_inval = 0; shared_in = 0;
if (state !== exp_state) begin
errors++;
$display("FAIL [%s] state=%0d exp=%0d", tag, state, exp_state);
end else
$display("PASS [%s] state=%0d br/iv/wb/sp=%b/%b/%b/%b", tag, exp_state, br, iv, wb, sp);
endtask
initial begin
rst_n = 0; ev(0,0,0,0,0, 3'b000, 0,0,0,0, "reset");
rst_n = 1;
// Become the forwarder: read miss with clean sharers -> F
ev(1,0,0,0,1, 3'b011, 1,0,0,0, "load shared: I->F (become forwarder)");
if (can_write) begin errors++; $display("FAIL: writable in Forward"); end
// Peer read: F supplies and hands off, no writeback -> S
ev(0,0,1,0,0, 3'b001, 0,0,0,1, "peer read: F->S (supply, hand off, no wb)");
// Now Shared and SILENT: a peer read leaves S untouched, no supply
ev(0,0,1,0,0, 3'b001, 0,0,0,0, "peer read: S stays S (silent)");
// Store from Shared upgrades: invalidate sharers -> M
ev(0,1,0,0,0, 3'b100, 0,1,0,0, "store: S->M (upgrade)");
// Peer store takes it: M is dirty -> writeback, I
ev(0,0,0,1,0, 3'b000, 0,0,1,0, "peer store: M->I (writeback)");
// Sole read -> Exclusive; a peer read supplies and demotes clean E->S (no wb)
ev(1,0,0,0,0, 3'b010, 1,0,0,0, "load sole: I->E");
ev(0,0,1,0,0, 3'b001, 0,0,0,1, "peer read: E->S (supply, no wb)");
// Forwarder upgrade path: I->F then store F->M
ev(1,0,0,0,1, 3'b011, 1,0,0,0, "load shared: I->F");
ev(0,1,0,0,0, 3'b100, 0,1,0,0, "store: F->M (upgrade, invalidate)");
if (errors == 0) $display("ALL TESTS PASSED");
else $display("%0d FAILURE(S)", errors);
$finish;
end
endmoduleExpected output:
PASS [reset] state=0 br/iv/wb/sp=0/0/0/0
PASS [load shared: I->F (become forwarder)] state=3 br/iv/wb/sp=1/0/0/0
PASS [peer read: F->S (supply, hand off, no wb)] state=1 br/iv/wb/sp=0/0/0/1
PASS [peer read: S stays S (silent)] state=1 br/iv/wb/sp=0/0/0/0
PASS [store: S->M (upgrade)] state=4 br/iv/wb/sp=0/1/0/0
PASS [peer store: M->I (writeback)] state=0 br/iv/wb/sp=0/0/1/0
PASS [load sole: I->E] state=2 br/iv/wb/sp=1/0/0/0
PASS [peer read: E->S (supply, no wb)] state=1 br/iv/wb/sp=0/0/0/1
PASS [load shared: I->F] state=3 br/iv/wb/sp=1/0/0/0
PASS [store: F->M (upgrade, invalidate)] state=4 br/iv/wb/sp=0/1/0/0
ALL TESTS PASSED14. DebugLab — two forwarders answer the same read
Two forwarders answer the same read
MISSED F->S HAND-OFF -> TWO FORWARDERS -> DUPLICATE RESPONSESOn a fast interconnect, an occasional duplicate data response to a read — two caches reply with the same clean line. Depending on the fabric this shows up as a protocol error, a dropped second packet, or wasted bandwidth. It appears only for widely-shared clean lines.
The forwarding read and the next one:
cyc core event state->next supply
3 CPU0 snoop_read F -> F 1 <-- supplied but stayed F (no hand-off)
3 CPU1 load miss I -> F - <-- new reader also becomes F
7 CPU2 load miss (snoop) both CPU0 and CPU1 answer -> 2 responsesDirectory scoreboard: fCount == 2 after cycle 3.
Cycle 3, CPU0's snoop response: it supplied the data but its state stayed F instead of demoting to S. That is the earliest wrong event — long before the duplicate responses at cycle 7 expose it.
Forwarding was treated as a pure data supply, not a hand-off. The design let the new reader install F without demoting the old forwarder, so two caches held the responder role. F's whole purpose — exactly one responder — was broken, and the next read drew replies from both.
Make supplying a read a hand-off: the F holder that answers demotes to S as the requester installs F. In the tracker of Section 10 this is the snoop_read branch on state F moving next to S with do_supply. Do not suppress the symptom by ignoring one of the duplicate responses at the requester — enforce the single-forwarder invariant at its source.
15. Common Mistakes
- Skipping the F→S hand-off. Assumption: forwarding is just answering a read. Bug: two forwarders and duplicate responses (the DebugLab). Prevention: the supplier demotes to S as the reader becomes F — one forwarder always.
- Treating Forward as dirty. Assumption: the special shared state must need a writeback. Bug: needless writeback on F→S or F→I. Prevention: F is clean and equals memory; only M writes back.
- Ignoring silent F eviction. Assumption: an F always exists to answer. Bug: a read hangs or errors when the sole forwarder was silently dropped. Prevention: tolerate "no forwarder responded" and source from memory, then elect a new F.
- Letting Shared copies respond. Assumption: any sharer can supply. Bug: the duplicate-response storm F was meant to prevent. Prevention: only F answers; S stays silent.
- Treating Forward as writable. Assumption: holding the forwarder role implies write rights. Bug: a store from F without invalidation leaves stale sharers. Prevention: F is read-only; a store upgrades F→M and invalidates the sharers.
- Treating MESIF as the CHI state set. Assumption: these states are the specification. Bug: confusion when CHI's Home Node picks data sources directly. Prevention: MESIF is the concept; in CHI the directory subsumes the forwarder role.
16. Engineering Checklist
- A read miss with clean sharers installs the requester as F; a sole miss installs E.
- The F holder supplies read misses and demotes to S (hand-off) — exactly one forwarder.
- S copies stay silent; they never answer reads.
- E, F, and S are clean — only M writes back.
- A read with no forwarder falls back to memory (F may have been silently dropped).
- A store from F or S upgrades to M and invalidates the sharers;
can_writeonly in M.
17. Key Takeaways
- MESIF adds Forward — a clean, shared line whose single holder answers read misses.
- One forwarder per line: F supplies, S stays silent — one fast response, no duplicate replies.
- The forwarder migrates to the newest reader (F→S on the supplier, I→F on the reader).
- F is clean — no writeback, and it can be silently dropped, so reads must fall back to memory.
- Forward (clean sharing) is the mirror of MOESI's Owned (dirty sharing); some systems use ideas from both.
- MESIF here is representative — in a directory protocol like CHI the Home Node picks the data source directly.
18. Quick Revision
MESIF = MESI + Forward. Five states: M (dirty, exclusive, writable), E (clean, exclusive), F (clean, shared, the one designated responder), S (clean, shared, silent), I (none). Read miss: sole → E, clean sharers → F (reader becomes forwarder). Peer read of F → supply then F→S hand-off; the reader installs F. Only M writes back; E/F/S are clean. F is losable — silent eviction means reads fall back to memory. Store: E silent; F/S upgrade by invalidating sharers;
can_writeonly in M. One forwarder per line. Representative model, not the CHI spec state set.
Coming Next
Chapter 2.5 — The Ownership Concept. Four protocols, one recurring idea: exactly one cache carries a special responsibility — supplying data, and eventually writing it back. The next chapter steps back from the state letters to define ownership itself: who owns a line, what duties ownership imposes, and how MSI, MESI, MOESI, and MESIF each assign it — the mental model CHI's Home Node and snoop machinery are built on.