Real Coherency Examples

Module 2 · Chapter 2.10 · Coherency Protocol Foundations · Module Capstone

Project thread — this chapter closes Module 2 by comparing the protocols on one workload. Module 3 then leaves the academic protocols behind and asks why AMBA needed CHI at all.

1. Learning Outcomes

By the end of this chapter you should be able to:

• Trace one producer-consumer sequence through MSI, MESI, and MOESI.
• Compare the memory writebacks and cache-to-cache transfers each protocol generates.
• Explain why MESI's Exclusive does not help producer-consumer sharing, but MOESI's Owned does.
• Identify migratory sharing and predict which protocol minimizes its traffic.
• Implement a SystemVerilog model that tallies the traffic of a sequence under different protocols.
• Verify the capstone claim that MOESI removes the memory writebacks MSI and MESI incur here.

2. Why Should I Learn This?

Choosing or evaluating a coherence protocol is a real engineering decision, and it turns on which sharing pattern your workload has. The producer-consumer (migratory) pattern — one core writes, another reads, repeatedly — is everywhere: message queues, streaming buffers, lock hand-offs. On it, the protocols differ sharply, and picking by reputation ("MESI is better than MSI") gives the wrong answer.

This chapter is where the four abstract state machines become a concrete comparison you can reason about and measure — the difference between knowing the states and knowing when each one pays off.

3. Key Terms

4. Previous Chapter Connection

Chapters 2.1–2.4 built MSI, MESI, MOESI, and MESIF; 2.5–2.9 built the machinery and the reading skill. Each protocol was justified by some inefficiency it removed — Exclusive for private writes, Owned for dirty sharing, Forward for clean sharing.

This capstone tests those justifications against one workload. Rather than trusting the per-chapter claims, it runs the same producer-consumer sequence through three protocols and counts the traffic, so you can see the Exclusive and Owned states help — or not — in a case that matters.

5. Core Concept — one sequence, three protocols

The scenario: memory holds A. CPU0 produces (writes A), CPU1 consumes (reads A), repeatedly. Four steps:

1. CPU0 writes A (produce).
2. CPU1 reads A (consume).
3. CPU0 writes A (produce again).
4. CPU1 reads A (consume again).

The one step that separates the protocols is the consume (step 2 and 4): CPU0 holds the line dirty (Modified) and CPU1 wants to read it.

• MSI and MESI: a shared read of a Modified line downgrades M→S and writes the dirty data back to memory first — both caches then hold clean Shared copies. Two consumes → two memory writebacks.
• MOESI: the same read moves M→O (Owned): CPU0 supplies the data cache-to-cache and keeps it dirty, no memory write. Two consumes → zero writebacks, two cache-to-cache transfers.

The result in one line:

On producer-consumer sharing, MSI and MESI behave identically — two memory writebacks — and MOESI removes both. Exclusive (MESI) helps a core that writes its own private line; it does nothing when the line is shared. Owned (MOESI) is the state that fits this pattern.

6. Engineering Mental Model — passing a note in class

Two students pass a note back and forth; a teacher (memory) keeps the official record.

• MSI / MESI: every time the note changes hands, they must first copy it to the teacher's record (writeback), then both work from clean copies. The teacher is on the critical path of every hand-off.
• MOESI: they pass the marked-up note directly to each other (cache-to-cache) and only update the teacher's record once, at the end (deferred writeback). The teacher is out of the loop during the exchange.

MESI's trick — being allowed to edit your own private note silently — is useless here, because the note is shared: the moment the other student reads it, MESI still routes through the teacher. Only MOESI's "keep the marked-up original and hand it over" removes that trip.

7. Engineering Diagram — the MOESI consume

There is no arrow to memory. Under MSI or MESI, this same exchange would add a write from CPU0 to memory before the data reached CPU1 — the writeback MOESI removes.

8. Worked Example — the sequence traced

The four steps, per protocol. wb = memory writeback, c2c = cache-to-cache supply.

MSI (and MESI — identical here):

MOESI:

Tally:

MESI matching MSI is the surprising row: adding Exclusive changed nothing, because Exclusive only helps a core writing a line no one else holds. The consumer's read makes the line shared, so MESI takes MSI's writeback path every time.

9. Transaction Walkthrough — the consume, two ways

Focus on step 2 (the first consume), the step that separates the protocols. Representative behavioral flow.

Under MSI / MESI:

1. CPU1's read miss reaches the Home Node; the directory shows CPU0 holds A Modified.
2. The HN snoops CPU0. Because MSI/MESI keep "Shared implies clean," CPU0 must write its dirty data back to memory and downgrade M→S.
3. Memory, now current, (or CPU0) supplies CPU1, which installs Shared. One writeback on the critical path.

Under MOESI:

1. Same read miss, same directory lookup: CPU0 holds A Modified.
2. The HN snoops CPU0, which supplies the data cache-to-cache and moves M→O — keeping the dirty value, writing nothing to memory.
3. CPU1 installs Shared, backed by CPU0's Owned copy. No writeback; the eventual one is deferred until CPU0 finally evicts.

Same request, same directory, one difference: whether the dirty line is allowed to stay dirty in a cache (Owned) or must be reconciled to memory first.

10. Timing View — producer-consumer under MOESI

Line A's state in each cache across produce, consume, produce. Watch CPU0 oscillate M↔O — never writing back — while CPU1 shares. Timing is representative — real latencies are not fixed cycle counts.

CPU0 never leaves the dirty family (M or O) during the exchange — no A@CPU0 value is ever clean Shared, so memory is never written. Under MSI/MESI, A@CPU0 would drop to S at t2 with a writeback.

11. RTL / Hardware View — a traffic-comparison model

This capstone compares protocols rather than building a new block, so the fitting artifact is one SystemVerilog model that runs the sequence and tallies traffic — a verification-style model, not synthesizable RTL. (Tri-HDL would be artificial here; a comparison is naturally expressed once, as executable checking code.) It walks the four steps under MSI/MESI and MOESI and counts writebacks and cache-to-cache supplies.

// Representative coherency-traffic comparison model (educational, verification
// style — ONE SystemVerilog model, not synthesizable RTL). Runs the same
// producer-consumer sequence under MSI/MESI and MOESI and tallies the traffic.
module coherency_traffic_compare;
  typedef enum {I, S, E, O, M} cstate_e;   // superset covering MSI/MESI/MOESI
 
  // Run produce, consume, produce, consume; report writebacks and c2c supplies.
  task automatic run(input bit moesi, output int writebacks, output int c2c);
    cstate_e a0, a1;                        // line A state in CPU0, CPU1
    writebacks = 0; c2c = 0;
    a0 = I; a1 = I;
    // 1. CPU0 produces (write A): -> Modified.
    a0 = M; a1 = I;
    // 2. CPU1 consumes (read A) while CPU0 holds M.
    if (moesi) begin a0 = O; a1 = S; c2c++;        end   // MOESI: supply, no writeback
    else       begin a0 = S; a1 = S; writebacks++; end   // MSI/MESI: writeback to memory
    // 3. CPU0 produces again (write A): upgrade, invalidate CPU1 -> Modified.
    a0 = M; a1 = I;
    // 4. CPU1 consumes again (read A) while CPU0 holds M.
    if (moesi) begin a0 = O; a1 = S; c2c++;        end
    else       begin a0 = S; a1 = S; writebacks++; end
  endtask
 
  int wb_msi, c2c_msi, wb_moesi, c2c_moesi, errors = 0;
  initial begin
    run(.moesi(0), .writebacks(wb_msi),   .c2c(c2c_msi));    // MSI / MESI
    run(.moesi(1), .writebacks(wb_moesi), .c2c(c2c_moesi));  // MOESI
    $display("MSI/MESI : writebacks=%0d cache-to-cache=%0d", wb_msi, c2c_msi);
    $display("MOESI    : writebacks=%0d cache-to-cache=%0d", wb_moesi, c2c_moesi);
    // Capstone claim: MOESI removes the memory writebacks of migratory sharing.
    if (!(wb_moesi < wb_msi)) begin errors++; $display("FAIL: MOESI did not reduce writebacks"); end
    if (wb_msi   !== 2)       begin errors++; $display("FAIL: expected 2 writebacks under MSI/MESI"); end
    if (c2c_moesi !== 2)      begin errors++; $display("FAIL: expected 2 cache-to-cache under MOESI"); end
    if (errors == 0) $display("ALL CHECKS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

MSI/MESI : writebacks=2 cache-to-cache=0
MOESI    : writebacks=0 cache-to-cache=2
ALL CHECKS PASSED

The model makes the comparison executable: wb_msi == 2, wb_moesi == 0, and MOESI trades the two memory writebacks for two cache-to-cache supplies.

12. Verification View — the capstone claim, checked

The model self-checks; here is what it proves and does not.

// The three checks embedded in the model above, stated as the capstone claim.
// 1. MOESI incurs strictly fewer memory writebacks than MSI/MESI for this pattern.
assert (wb_moesi < wb_msi);
// 2. MSI/MESI write back on every consume (two consumes -> two writebacks).
assert (wb_msi == 2);
// 3. MOESI converts both consumes to cache-to-cache supplies.
assert (c2c_moesi == 2);

The system claim, as a reference-model rule:

For a producer-consumer sequence of K consumes, MSI and MESI incur K memory writebacks; MOESI incurs 0 during the exchange (one deferred writeback at final eviction). MESI equals MSI because Exclusive never applies to a shared line.

• What it proves: MOESI reduces memory traffic on migratory sharing, and MESI does not help this pattern.
• What it does not prove: protocol correctness (that is 2.1–2.4), actual latency or bandwidth numbers (workload- and interconnect-dependent), or that MOESI wins on every pattern (a mostly-private workload favors MESI's Exclusive).
• Bug signature when it fails: wb_moesi non-zero (a MOESI model still writing back on a shared read — Owned not implemented), or wb_msi != 2 (the sequence miscounted).

13. DebugLab — the MESI upgrade that saved nothing

MSI  : writebacks=2 cache-to-cache=0
MESI : writebacks=2 cache-to-cache=0     <-- identical: Exclusive never engaged
MOESI: writebacks=0 cache-to-cache=2

14. Common Mistakes

• Expecting MESI to beat MSI on shared data. Assumption: MESI is universally faster. Bug: no improvement on producer-consumer (the DebugLab). Prevention: Exclusive helps private RMW; Owned helps shared-dirty.
• Assuming cache-to-cache is always a win. Assumption: any cache transfer beats memory. Bug: on some fabrics a remote cache is no faster than memory. Prevention: cache-to-cache helps when it is genuinely faster and saves a writeback — usually, but measure.
• Forgetting the deferred writeback. Assumption: MOESI eliminates the writeback. Bug: budgeting zero memory writes; the deferred one still happens at eviction. Prevention: MOESI defers to one, it does not delete.
• Counting transactions, not their cost. Assumption: fewer transactions is always better. Bug: a writeback-plus-read can cost more than one cache-to-cache even at equal counts. Prevention: weight by type and latency, not raw count.
• Generalizing from one pattern. Assumption: the best protocol here is best everywhere. Bug: a private-heavy workload favors MESI, not MOESI. Prevention: profile the sharing pattern before choosing.
• Treating these as the CHI protocol. Assumption: MSI/MESI/MOESI are CHI. Bug: CHI's states are a superset with more nuance. Prevention: this is the reasoning that carries into CHI (Module 3 onward).

15. Engineering Checklist

• Identify the workload's sharing pattern (private, producer-consumer, clean-shared) before choosing a protocol.
• For producer-consumer, prefer a protocol with Owned (MOESI-like dirty sharing).
• Reserve Exclusive benefits for private read-modify-write workloads.
• Count traffic by type (writeback, cache-to-cache, upgrade), not raw transaction count.
• Remember MOESI's writeback is deferred, not removed.
• Validate a protocol change against the actual pattern, with a traffic model or measurement.

16. Key Takeaways

• On producer-consumer sharing, MSI and MESI are identical — two memory writebacks — and MOESI removes them (cache-to-cache, deferred writeback).
• MESI's Exclusive helps private data, not shared data — the consumer's read makes the line shared, defeating it.
• MOESI's Owned fits migratory sharing — the dirty line stays in caches and is handed over directly.
• Compare protocols by traffic type and pattern, not reputation; the right choice depends on the workload.
• A deferred writeback is fewer, not none — the saving grows with the number of consumes.
• These examples are representative — the reasoning that carries into CHI's own states from Module 3.

17. Quick Revision

Real coherency examples. Producer-consumer: CPU0 writes, CPU1 reads, repeatedly. The consume is the differentiator. MSI = MESI: a shared read of Modified writes back to memory (M→S + wb) — K consumes, K writebacks; Exclusive never engages because the line is shared. MOESI: the read is M→O, cache-to-cache, no writeback (deferred to eviction). Result: MSI/MESI = 2 writebacks; MOESI = 0 (+ 2 cache-to-cache). Exclusive helps private RMW; Owned helps shared-dirty. Match the protocol to the pattern; count traffic by type. Representative model, not the complete CHI spec.

Coming Next

Module 3 — Why CHI Exists. Module 2 built the academic protocols and the reasoning behind them. Module 3 leaves them for the real question: why did AMBA need a new protocol at all? Chapter 3.1 traces the AMBA generations — AHB, AXI, ACE — and the coherency and scalability walls each hit, setting up exactly the problems CHI was created to solve.