Ownership in CXL

13.1 established who may read and who may write.

13.2 turned every one of those decisions into a window with a duration.

Neither chapter answered the question those mechanisms exist to protect: when a line is dirty and shared, who is responsible for the data.

1. The Engineering Problem — A Duty, Not A Privilege

The word ownership is misleading. In everyday use it means a right — the owner decides, the owner benefits, the owner may exclude others. In a coherency protocol it means almost the opposite.

An owner is the agent that holds a value memory does not have. That is not a benefit. It is an obligation with three parts:

It must answer reads. Memory's copy is stale. If another agent asks for the line, memory's answer would be wrong, so the owner has to supply it instead. The owner cannot decline, and it cannot delegate without a handover.

It must not disappear silently. A shared, clean copy can be dropped without telling anyone — memory still has the value, and so may other agents. A dirty copy cannot. Dropping it without writing it back destroys the only current version of the data.

It must be exactly one agent. Two owners means two answers to the same read, and nothing in the protocol says which is right. Zero owners for a dirty line means the current value has no source at all.

The reason MESI needs a fifth state is that it has nowhere to record this duty. In MESI, when a modified line is read remotely, the modifying agent has to drop to S — and S carries no obligation. So before it can drop, it must push the data to memory, because otherwise the duty would evaporate along with the state that implied it.

That writeback is pure overhead. The data was not needed in memory; it was needed by the reader, who is about to get it anyway. MESI performs it because it has no way to say "I still owe this."

MOESI adds a state whose entire content is that sentence.

2. The One-Sentence Model

Ownership is a debt, not a right. Exactly one agent holds a value memory does not have; that agent must answer every read of the line and must not drop it without settling; and the debt can be transferred, but never merely abandoned.

Call it one debtor per line. Every defect in this chapter is a line with two debtors, no debtor, or a debtor that walked away.

3. What This Chapter Owns

Deferred:

The distinction against 13.4 is worth stating precisely, because it is easy to blur. This chapter owns the duty: who has it, what it obliges, how it moves, what it costs, and what happens when it is lost. 13.4 owns the state space: the full enumeration, the encoding, the transition table and the machinery that applies it. The Owned state appears here because ownership is what it means, not because this is where the state diagram lives.

4. Teaching-Model Boundary

Every model below is a teaching model. It is written to make one idea measurable, compiled and simulated with Icarus Verilog 13.0, and checked by a testbench whose oracle is structurally different from the design.

What these models are not: they are not a coherency controller. There is no request pipeline, no retry queue, no directory cache, no ECC, no power management, and no CXL opcode encoding anywhere in this chapter. A production owner tracker is a directory cache with eviction policy, back-invalidation, and a state machine per outstanding transaction.

Three conventions carry over from 13.1 and 13.2 and are used without further comment.

Every checker tests cond !== 1'b1, never !cond. On an uninitialised signal !cond is x, which is not true, so a naive check passes vacuously and reports nothing. The stricter form fails on x as it should.

Where a monitor cannot be reached in a correct design, the module carries a FAULT_INJECT parameter and the testbench instantiates a second copy of the same source with the hook enabled. This chapter needs it three times: for the two-responder case, for the directory that names a non-holder, and for the entry that forgets who owes the data. A monitor that has never fired is a monitor nobody has tested.

Where two protocols are compared, they are the same source under a parameter. moesi_line takes HAS_OWNED, and the testbench runs both builds on identical stimulus. The MESI-versus-MOESI numbers in this chapter are a measured difference between two instances, not two separately written models that could differ for uninteresting reasons.

5. RTL 1 — MOESI: The State That Owes

The whole chapter is in five lines of this module. Start with what the states mean rather than what they are called:

  // O is dirty. That is the whole point: the data differs from memory and one
  // agent is carrying that difference on memory's behalf.
  assign dirty     = (st_q == M) || (st_q == O);
  assign is_owner  = (st_q == M) || (st_q == O);
  // Only the owner answers reads. A line with no owner is answered by memory.
  assign must_supply = rem_rd && is_owner;

dirty and is_owner are the same expression, and that is not an accident — it is the invariant. A dirty line always has an owner, and an owner always holds dirty data. If those two ever come apart, either memory's stale value is about to be served as current, or an agent is answering reads it has no authority to answer.

Now the transition that distinguishes the two protocols:

  // In MESI a remote read of a modified line must push the data to memory,
  // because S carries no duty. In MOESI the owner keeps it and supplies it,
  // so the writeback is deferred -- possibly forever, if the line is
  // eventually invalidated by a write that supersedes it.
  assign wb_on_downgrade = rem_rd && (st_q == M) && (HAS_OWNED == 0);
  // Dropping a dirty line always costs a writeback, whether M or O.
  assign wb_on_evict     = evict && ((st_q == M) || (st_q == O));

and in the next-state logic:

    end else if (rem_rd) begin
      if (st_q == E)      st_n = S;
      else if (st_q == M) st_n = (HAS_OWNED != 0) ? O : S;
      // O stays O: it keeps the duty and supplies the data again.
    end

HAS_OWNED is a parameter, so the same source compiles as MESI and as MOESI. The testbench instantiates both and drives them from one set of stimulus:

  moesi_line #(1) moesi(...);
  // The SAME source as MESI, so the cost of not having O is measured, not claimed.
  moesi_line #(0) mesi (...);

Drive a read miss, a local write, and then a remote read. Measured:

  remote read of M : MOESI -> state 4 (O), MESI -> state 1 (S)
  writebacks on that downgrade : MOESI=0  MESI=1
  owner supplied the data      : 1 time(s)
  evicting an owned line       : writebacks=1

One line of stimulus, two different costs. MESI wrote to memory; MOESI did not, and answered the read itself. Note the fourth line carefully: eviction still costs a writeback. MOESI does not eliminate the writeback, it defers it — and if the line is later invalidated by another agent's write, the deferred writeback is never performed at all. That is where the saving actually comes from.

Two further measurements from the same run pin down what O is not:

  downgrades : E->1 (S=1)   read miss with sharers -> 1 (S=1)

A remote read of an E line goes to S, not to O, because E is clean — there is no debt to carry. And a read miss when other agents already hold the line lands in S rather than E, because exclusivity is a claim about the rest of the system, not about the local cache.

6. Waveform — The Downgrade That Costs Nothing

Transcribed from the printed cycle trace of the dual-build model above, which runs a MOESI and a MESI instance on identical stimulus and prints both state columns every cycle.

The is_owner row is the whole idea. It is high from the local write until the eviction, and it does not drop at the downgrade. Write permission is lost at cycle 3; the obligation is not. In the MESI column the obligation is discharged at cycle 2 by a writeback nobody asked for, and the agent is left in S with no responsibility and no useful copy.

The supply row shows the owner answering twice — once at cycle 2 and once at cycle 4 — while wb_mesi fires once and never again. Over the whole trace:

  totals: MOESI downgrade-writebacks=0  MESI=1
          MOESI evict-writebacks=1      MESI=0
          MOESI owner supplies=2        MESI=1

Read those three lines together, because taken alone the first is misleading. The total writeback count is one either way. What differs is when it happens and whether it happens at all: MESI paid at the downgrade, unconditionally; MOESI paid at the eviction, and would have paid nothing had the line been invalidated by another agent's write first. MOESI's saving is a probability, not a certainty — and the last row is the certainty: the owner supplied the data twice, so two reads were answered without touching memory at all.

7. RTL 2 — The Owner Registry

The state machine above tracks one agent's view. The registry tracks the system's: at most one owner per line, and ownership is not the same as holding a copy.

  // A second claim while an owner exists is the ownership form of the
  // two-writer failure: two agents would both answer reads, with two answers.
  assign two_owner_err     = claim && own_v_q && (claim_agent != own_id_q);
  // A release from an agent that does not own it must not clear the duty.
  assign wrong_release_err = release_own && own_v_q && (release_agent != own_id_q);
  // Copies exist and nobody owes the data: the dirty value has no supplier.
  assign orphan_dirty_err  = !own_v_q && (n_holders != 3'd0) && release_own;

Three different failures, three separate signals, and the separation is the point. Merging them into one owner_err tells an engineer that something is wrong with ownership and nothing about whether the fix is in arbitration, in identity checking, or in the eviction path.

The claim path refuses rather than overwrites:

      if (claim && !own_v_q) begin
        own_v_q <= 1'b1; own_id_q <= claim_agent; n_claims <= n_claims + 8'd1;
      end

The guard is !own_v_q, not claim alone. A second claimant does not get the line and does not displace the incumbent — it is refused and reported, and the requesting agent must retry after the current owner releases. Measured:

  registry : two_owner_err=1 wrong_release_err=1 orphan_dirty_err=1

Each of the three abuse cases was driven and each was caught. The wrong_release_err case is the subtle one: agent 3 asked to release a line owned by agent 1. Without the identity check the duty would have been cleared, leaving a dirty line with no owner while agent 1 still believed it was responsible. The check costs a two-bit comparison.

8. RTL 3 — Who Answers A Read

Exactly one responder, and which one depends on whether the line is dirty:

  // The owner supplies whenever there is one; memory supplies only a clean
  // line with no owner. A dirty line with no owner has no correct source.
  assign owner_supplies  = read_req &&  has_owner && (owner_id != requester);
  assign memory_supplies = read_req && !line_dirty
                           && ((FAULT_INJECT != 0) || !has_owner);
  assign no_supplier_err = read_req && !has_owner &&  line_dirty;
  assign two_supplier_err= owner_supplies && memory_supplies;

The owner_id != requester term is easy to omit and easy to justify omitting — surely an agent that owns a line does not ask for it. It does, when the request came from a different core behind the same cache, or when a speculative fetch races an ownership acquisition. Without the term the model reports that the owner supplied data to itself, which inflates the supply counter and hides the fact that the access never went anywhere.

two_supplier_err cannot fire in the correct design: owner_supplies requires has_owner and memory_supplies requires its negation. That makes it an unreachable checker — the exact defect class this chapter's convention exists for. The FAULT_INJECT term drops the !has_owner guard, the testbench instantiates a second copy of the same source with the hook enabled, and the monitor is proven to fire when the condition it watches actually occurs.

Measured across four supply cases:

  supply   : owner=1 memory=1 dirty-with-no-owner=1 self=0 two_responders(faulty)=1

Read the third figure carefully. A dirty line with no owner has no correct source at all — memory's copy is stale and no agent has admitted to holding the current one. That is not a performance problem or a latency problem. It is a data-loss condition that has already happened, detected after the fact. Every other mechanism in this chapter exists to make sure that number stays at zero.

9. RTL 4 — Moving The Duty

Ownership transfer is where the windows from 13.2 become dangerous. The old owner must stop answering; the new owner must start. Do those in the wrong order and there is an interval in which a read of the line has no responder.

The model makes the ordering explicit as a three-phase handover:

  // The old owner keeps answering until the new one has acknowledged.
  // Overlap, not a gap, is what makes a transfer safe.
  assign old_answers = (ph_q == HANDING);
  assign new_answers = (ph_q == SETTLING) || (ph_q == IDLE && responder_valid_q);
  assign new_owner_live  = new_answers;
  assign gap_err         = read_arrives && !responder_valid;

The design deliberately exports new_owner_live rather than keeping new_answers internal. A directory needs to know when it may publish the new owner, and an output that nothing observes is an output no testbench can check — a point section 18 returns to, because the first version of this model had exactly that flaw.

Reads were driven in every phase. Measured:

  handover : gaps=1  transfers=1  covered during send->ack=1  new-owner-live during hand=0

One gap, and it was the deliberate one — a read issued before any agent had ever owned the line. Every read during the handover itself was answered: by the old owner while the data was still in flight, by the new owner once it had arrived. new_owner_live was low throughout the handing phase, which is the check that the new owner does not start answering with data it does not have yet.

The phase machine also refuses to complete on its own:

        SETTLING: if (data_acked) begin
                    ph_q <= IDLE;
                    responder_q <= new_q; responder_valid_q <= 1'b1;
                    n_transfers <= n_transfers + 8'd1;
                  end

Held in SETTLING for two cycles with no acknowledgement, the phase does not advance and n_transfers stays at zero. A handover that completes on a timer rather than on an acknowledgement is a handover that can publish an owner which never received the data.

10. RTL 5 — The Drop That Loses Data

A shared copy may be discarded without telling anyone. An owned copy may not. The model is small enough to read in one pass and the parameter is what makes it useful:

  // A shared copy may always be discarded silently: someone else, or memory,
  // still has the value. ALLOW_SILENT_OWNED extends that permission to the
  // owner -- the same source can therefore be run as a correct protocol and
  // as a broken one, and the difference measured rather than asserted.
  assign silent_ok = drop && ((is_shared && !is_owned) ||
                              (is_owned && ALLOW_SILENT_OWNED != 0));
  assign needs_wb  = drop && is_owned && mem_stale && (ALLOW_SILENT_OWNED == 0);
  assign data_lost = drop && is_owned && mem_stale && !wrote_back;

Both builds were driven with the same drop of a dirty owned line. Measured:

  drops    : shared silent_ok=1 lost=0 | owned silent_ok=0 needs_wb=1 lost=1
  policy   : lost strict=1 permissive=1 | writeback demanded strict=1 permissive=0
  clean owned drop demands a writeback: 0

The second line is the one worth sitting with, because the naive expectation is wrong. Both policies lost the data. The permissive build did not lose it more often — it lost it just as often, and gave the agent no indication that anything was owed. The strict build raised needs_wb; the permissive build reported the drop as legal. The difference between a correct protocol and a broken one here is not the outcome under identical stimulus. It is whether the protocol tells the agent to do something else.

The third line separates the debt from the state. An owned but clean line — one whose value memory already has — owes nothing, and dropping it loses nothing. mem_stale is the term that matters, not is_owned. A model that demanded a writeback for every owned drop would be conservative and wrong, and section 18 injects exactly that mutation.

11. RTL 6 — Stale Memory And Who Owes It

The registry in section 7 tracked ownership as an identity. This model tracks it as an invariant over time: whenever memory is stale, exactly one agent owes the writeback.

  // The checkable invariant, evaluated on the registered state.
  assign unowed_stale_err = stale_q && !held_q;
  assign false_wb_err     = writeback && (!held_q || wb_agent != agent_q);

unowed_stale_err reads the registered state rather than the incoming request, which makes it a statement about what the directory currently believes rather than about what it is being asked to do. That distinction is what makes it usable as a continuous monitor in silicon: it is true or false at every clock edge, with no request in flight required.

This is the third monitor in the chapter that cannot fire in a correct design, so the same convention applies:

      if (write) begin
        stale_q <= 1'b1;
        // FAULT_INJECT drops the duty assignment while still marking memory
        // stale -- the exact shape of a directory that forgets an owner.
        held_q  <= (FAULT_INJECT != 0) ? 1'b0 : 1'b1;

Measured:

  duty     : unowed_stale strict=0 faulty=1  false_wb=1

The faulty build reproduces a real and common directory bug: the entry is marked dirty but the owner field is not written, usually because the two live in different pipeline stages and one of them was flushed. The result is a line that memory will not serve correctly and no agent will serve at all — and it is silent until someone reads it.

false_wb_err catches the mirror-image failure. A writeback arriving from an agent that is not the recorded owner was driven, and the model rejected it: memory stayed stale, the duty stayed with agent 3, and the counter did not advance. Accepting it would have marked memory clean using data from an agent that may have been holding a stale copy of its own.

12. RTL 7 — The Directory Entry

Everything so far has been per-line facts in isolation. A directory holds them together, and the interesting failures are the ones where two fields of the same entry disagree.

  // Both invariants read the registered state, so they are true statements
  // about what the directory currently believes -- not about a request.
  assign owner_not_sharer_err = ov_q && !sh_q[ow_q];
  assign empty_with_owner_err = ov_q && (sh_q == 4'd0);

The first says the named owner must be a member of the sharer set. The second says an entry with no holders must name nobody. Both are unreachable in the correct design, and both are proven by a FAULT_INJECT build that removes the membership guard on set_owner.

The removal path is where the two fields are most likely to drift apart:

      if (remove_) begin
        sh_q[agent] <= 1'b0;
        n_evictions <= n_evictions + 8'd1;
        // Removing the owner from the sharer set must also drop the duty,
        // otherwise the directory names an agent that no longer has the line.
        if (ov_q && agent == ow_q) ov_q <= 1'b0;
      end

Note that the guard is ov_q && agent == ow_q and not ov_q alone. Removing a plain sharer must leave the duty exactly where it was. Both cases were driven. Measured:

  directory : owner_not_sharer=0 empty_with_owner=0 evictions=3
  faulty directory : owner_not_sharer=1 empty_with_owner=1

A separate clear_owner path releases the duty while the copy is retained — agent 0 kept its line and stopped owing the data, which is the state a writeback leaves behind. Ownership and residence are different facts and the entry stores them separately, because collapsing them into one field makes both of the above invariants unstateable.

13. RTL 8 — Two Agents, One Line

Two agents ask to own the same line in the same cycle. Exactly one may win, and the loser must be told to retry rather than silently dropped:

  // With both asking, one preference decides. Fairness alternates it so a
  // steady stream from A cannot starve B.
  assign a_wins  = (req_a && !req_b) ? 1'b1 :
                   (!req_a && req_b) ? 1'b0 : !prefer_b_q;
  assign grant_a = req_a && a_wins;
  assign grant_b = (FAULT_INJECT != 0) ? req_b : (req_b && !a_wins);
  assign retry_a = req_a && !grant_a;
  assign retry_b = req_b && !grant_b;
  assign double_grant_err = grant_a && grant_b;
  assign lost_req_err = (req_a && !grant_a && !retry_a)
                     || (req_b && !grant_b && !retry_b);

The first two terms of a_wins handle the uncontested cases explicitly. Without them a lone request from B would lose to a preference that favours A, and the requester would retry forever against no competitor — a livelock that only appears under light load, which is the worst possible time to discover it.

Four rounds were driven: idle, B alone, A and B contested with fairness off, and contested again with fairness on. Measured:

  arbitration : A won=1 then B won=1 | double_grant correct=0 faulty=1

with n_grants at four and n_retries at two or more. One grant per round, never zero and never two. With fairness off, two consecutive contested rounds returned the same winner — the preference does not move on its own — and enabling it handed the next round to B.

lost_req_err deserves a note, because the faulty build does not trip it. FAULT_INJECT grants both requesters, and both are answered, so no request is lost. The bug is not a dropped request; it is two owners. A monitor that only watched for unanswered requests would report a clean run on a design that had just created two responders for one line.

14. RTL 9 — When The Owner Is Across The Link

Ownership is a latency decision as well as a correctness one. When the owner is on the far side of a CXL link, every read of that line becomes a link round trip rather than a memory access:

        remote_q <= owner_valid && owner_is_remote;
        target_q <= (owner_valid && owner_is_remote) ? LINK_CYCLES[7:0]
                                                     : MEM_CYCLES[7:0];

The owner_valid term is not decoration. A line with no owner is served by memory regardless of where the requester is, so charging it the link latency would attribute a cost to ownership that ownership did not cause. Three reads were driven — local owner, remote owner, no owner. Measured:

  latency : owner local=3  owner across the link=6  no owner=3 (cycles)

with total occupancy at 12 cycles for the three reads and the remote and local counters at one and two respectively.

Cache-to-cache transfer is not automatically a saving. The received wisdom is that supplying from a cache beats going to memory, and on one die it usually does. Across a link the comparison inverts: six cycles to ask a remote owner against three to read local memory. A line whose owner sits on the far side of a link and is read frequently by agents on the near side is a line whose owned state is actively costing latency on every access — and the fix is to force a writeback and let memory answer, which is precisely the operation MOESI exists to avoid. Module 18 quantifies this properly; the model here only establishes that the sign of the effect depends on where the owner is.

15. RTL 10 — What The Owned State Actually Saved

Every mechanism counter in this chapter can be clean while the protocol is doing no good at all. This model measures the outcome:

      if (downgrade &&  had_owned) wb_saved <= wb_saved + 16'd1;
      if (downgrade && !had_owned) wb_done  <= wb_done  + 16'd1;

with the share computed at a width that cannot wrap:

  logic [16:0] total_sup;     // one bit wider than the sum of two 16-bit counts
  logic [31:0] weighted;      // 16x100 needs 23 bits; 16 would silently wrap
  assign total_sup = {1'b0, from_owner} + {1'b0, from_mem};
  assign weighted  = {16'd0, from_owner} * 32'd100;
  assign owner_share_pct = (total_sup == 17'd0) ? 8'd0
                         : (weighted / {15'd0, total_sup});

The width commentary is not pedantry. Verilog sizes an expression from its operands, not from its destination, so from_owner * 100 in 16 bits wraps above 655 supplies — a counter that reports a plausible percentage and is wrong under exactly the sustained load an engineer would use it to investigate. This defect class appeared eight times across Module 12 and is now checked explicitly in every counter this track writes.

Eight downgrades were driven through a build with the Owned state and eight identical ones through a build without it. Measured:

  writebacks over 8 downgrades : with O=8  without O=8
  reads answered by the owner  : 75%

Eight writebacks avoided against eight performed, on identical stimulus. The 75% figure is three owner supplies against one from memory — three reads that never reached memory at all.

The empty-sample guard is checked before any read is issued: with no supplies recorded the share reports zero, not a hundred. A ratio computed from an empty sample is the most common way a performance counter lies, and it lies most convincingly at the start of a run.

16. Quantitative Reasoning

Everything below follows from the measured numbers above.

What one downgrade costs. MESI's writeback on an M-to-S downgrade moves one cache line to memory. At a 64-byte line and a 10-cycle memory write, eight downgrades cost 512 bytes of write bandwidth and 80 cycles of memory occupancy that MOESI does not spend. The measured wb_saved=8 against wb_done=8 is that difference at the protocol level, before any bandwidth model is applied.

When the saving is real and when it is not. MOESI defers a writeback; it does not remove it. The saving is realised only if the line is invalidated before it is evicted — a write from another agent supersedes the value and the deferred writeback is never performed. If the line is instead evicted from the owner's cache, the writeback happens anyway, one measured wb_on_evict per eviction, and MOESI has bought nothing but a delay. The Owned state's benefit is therefore a function of the workload's write-sharing pattern, not a property of the protocol.

Where the owner sits changes the sign. At the measured 6-cycle link round trip against a 3-cycle local memory read, a remotely-owned line costs double on every read. With owner_share_pct at 75%, three of four reads take six cycles instead of three, so mean read latency for that line rises from 3 to 5.25 cycles — a 75% increase caused entirely by the state that was supposed to make things faster.

The directory cost of naming an owner. A four-agent sharer vector needs four bits; naming one of four owners needs two more plus a valid bit. Seven bits per line at a 64-byte line is roughly 1.4% of the tracked capacity in directory storage — before any consideration of how the directory itself is cached, which Modules 15 and 16 own.

Handover exposure. The three-phase transfer holds the old owner responsible until the new one acknowledges. On the measured link that overlap is the full round trip. A design that instead released the old owner at data_sent would open a window equal to the acknowledgement latency, during which gap_err would fire on any read — one measured gap in this chapter, and it was the deliberate pre-ownership one.

17. Assertions

Presented as SystemVerilog and executed as procedural checkers — see section 19.

A dirty line always has exactly one owner.

property p_dirty_implies_owner;
  @(posedge clk) disable iff (!rst_n)  dirty |-> is_owner;
endproperty

An owner always holds read permission.

property p_owner_can_read;
  @(posedge clk) disable iff (!rst_n)  is_owner |-> can_read;
endproperty

A remote read of a modified line writes back only without the Owned state.

property p_downgrade_writeback;
  @(posedge clk) disable iff (!rst_n)
    (rem_rd && state == M) |-> (wb_on_downgrade == (HAS_OWNED == 0));
endproperty

Never two owners.

property p_single_owner;
  @(posedge clk) disable iff (!rst_n)  claim && has_owner |-> two_owner_err;
endproperty

Exactly one responder per read.

property p_one_responder;
  @(posedge clk) disable iff (!rst_n)
    read_req |-> $countones({owner_supplies, memory_supplies, no_supplier_err}) == 1;
endproperty

A handover never leaves a read unanswered.

property p_no_gap;
  @(posedge clk) disable iff (!rst_n)
    (phase != 2'd0 && read_arrives) |-> responder_valid;
endproperty

The new owner does not answer before it has the data.

property p_new_owner_not_early;
  @(posedge clk) disable iff (!rst_n)  (phase == 2'd1) |-> !new_owner_live;
endproperty

Stale memory always has a debtor.

property p_stale_is_owed;
  @(posedge clk) disable iff (!rst_n)  mem_stale |-> duty_held;
endproperty

A writeback comes only from the recorded owner.

property p_writeback_identity;
  @(posedge clk) disable iff (!rst_n)
    (writeback && !false_wb_err) |-> (wb_agent == duty_agent);
endproperty

The directory never names a non-holder.

property p_owner_is_sharer;
  @(posedge clk) disable iff (!rst_n)  owner_valid |-> sharers[owner];
endproperty

Never two grants for one line.

property p_single_grant;
  @(posedge clk) disable iff (!rst_n)  !(grant_a && grant_b);
endproperty

Every request is answered, granted or retried.

property p_no_lost_request;
  @(posedge clk) disable iff (!rst_n)  req_a |-> (grant_a || retry_a);
endproperty

18. Mutation Testing

An assertion that has never failed has never been tested. 98 mutations were injected into the ten models, one at a time, each a single-line change that a competent engineer could plausibly write. Every one must make the testbench print RESULT: FAIL.

Representative mutations, all killed:

Sixteen mutations survived the first run, and none of them was patched away. Each was classified and the testbench was extended:

Five stimulus gaps. The bench never drove a remote read of an E line, never set others_have on a read miss, never populated the top bit of the sharer vector, never had an agent request a line it already owned, and never removed a plain sharer while an owner existed. Five cases added, five mutations killed.

Four unobserved outputs. n_grants, n_lost, owner_share_pct before any sample, and the arbiter's behaviour with no requester at all were computed and never read. A mutation cannot be killed by a value nobody checks.

Three unreachable checkers. two_supplier_err, owner_not_sharer_err and empty_with_owner_err are all unreachable in the correct design — that is what makes them invariants. Each needed a FAULT_INJECT build of the same source, and each was then proven to fire.

Three missing stability checks. The arbiter's preference was never checked for stability across two rounds with fairness off; the handover was never held in SETTLING without an acknowledgement; and a clean owned line was never dropped.

One provably equivalent mutation, replaced rather than recorded. The mutation that made new_answers true during the handing phase changed nothing observable, because responder_valid was already high via old_answers and responder still selected the old owner. It was unkillable not because the bench was weak but because the signal was internal. The correct response was to fix the design, not the mutation: new_owner_live was added as an output, because a directory genuinely needs to know when the new owner may be published. The mutation then became observable and was killed.

A survivor is a finding about the testbench, not a nuisance. Recording one as an acceptable escape converts a verification gap into a documented feature.

19. Verification Strategy

The oracle must not be the design. Each testbench models the same behaviour in a structurally different representation, so a bug in one cannot reproduce itself in the other.

For moesi_line the design holds one five-valued state. The oracle holds four independent booleans — read permission, write permission, dirtiness, and the duty — updated by an event function that knows nothing about state encodings:

  // Oracle: permission, dirtiness and DUTY held as three independent booleans.
  // The design stores one five-valued state; the oracle stores what it means.
  integer o_rd, o_wr, o_dirty, o_duty;

with the transition that matters written as the meaning rather than the state change:

      else if (rr) begin
        if (o_wr) begin o_wr=0; end        // downgrade loses write permission
        // duty and dirtiness SURVIVE the downgrade in MOESI
      end

For sharer_vector the design holds a bit vector; the oracle holds four plain integers and a name, so a vector indexing bug cannot appear identically in both. For owner_transfer the design holds a phase machine; the oracle holds a single "who is on the hook" scalar advanced by events.

Every displayed value is a captured signal. No $display in these benches prints a literal. Where a value is sampled before a later event changes it, it is latched into a named integer first:

    sO=state; sMs=ms_state; sWbD=n_wb_downgrade; sMsWbD=ms_nwbd;

A summary line that prints what the author expected rather than what the design produced is worse than no summary at all, because it survives the bug.

Delta-cycle discipline. A continuous assignment read in the same delta as its driver changes returns the previous value. The waveform trace in section 6 initially printed the supply and writeback columns one cycle late for exactly this reason; a #1 settle before each sample corrected it. The corrected trace is the one shown.

Coverage recorded: 146 assertion sites across the three testbenches, all five MOESI states entered, both downgrade paths from M exercised, all four supply combinations driven, the handover exercised in all three phases with reads in each, both drop policies driven on identical stimulus, and the arbiter driven idle, uncontested, contested, and contested-with-fairness.

20. Synthesis and Implementation Reality

The Owned state costs one encoding, not one bit. Four states fit in two bits; five do not. Every directory entry and every cache tag grows from two state bits to three — a 50% increase in state storage across the whole tracked capacity, for one extra state. That is the real price of MOESI, and it is paid on every line whether or not the line is ever owned.

The owner field is the expensive part. Naming one owner out of N agents costs ceil(log2(N)) bits plus a valid bit per line. At 64 agents that is seven bits per line on top of the sharer vector, and the sharer vector itself is the dominant term — which is why real directories compress it, and why Modules 15 and 16 own that discussion rather than this one.

The invariant monitors are nearly free. dirty && !is_owner is one gate on two signals already present. owner_not_sharer_err is a mux into the sharer vector and an inverter. These are not verification-only constructs to be stripped at synthesis; they are cheap enough to leave in silicon, and section 21 argues they should be.

The arbiter is on the critical path. a_wins feeds grant_a and grant_b combinationally, and in a real design that path continues into the directory write. At scale the arbitration is pipelined, which introduces its own window — a request granted in one cycle and recorded in the next, with the interval available for a second grant if the pipeline is not interlocked. The teaching model is flat on purpose; the pipelined version is a 13.4 and Module 14 concern.

Reset matters more than it looks. own_v_q and held_q must reset to zero. A directory that comes out of reset believing some line is owned by agent zero will refuse the first genuine claim on that line and report two_owner_err against a system that has done nothing wrong.

21. Silicon Observability

The counters in these models are not decoration. Each answers a question that cannot be answered from a waveform after the fact, because the failures in this chapter are silent by construction.

Three of the error signals belong in silicon, not just in simulation. unowed_stale_err, owner_not_sharer_err and no_supplier_err all detect states from which no correct behaviour is possible. Each is a handful of gates. A machine check raised the moment one of them asserts turns a silent data corruption into a diagnosable fault with a line address attached — and the alternative is discovering it as a wrong answer in an application, days later, with no state left to inspect.

n_claims minus n_releases should equal the number of currently owned lines. A drift between them over a long run is the signature of a leaked duty: an agent that took ownership and was reset, or a release message dropped on a link. Neither shows up as an error until something reads the line.

22. Debug Lab

  drops : owned silent_ok=0 needs_wb=1 lost=1

assign needs_wb  = drop && is_owned && mem_stale && (ALLOW_SILENT_OWNED == 0);
assign data_lost = drop && is_owned && mem_stale && !wrote_back;

  arbitration : double_grant correct=0 faulty=1

assign grant_b = req_b && !a_wins;               // never unconditionally on request
if (claim && !own_v_q) begin                     // refuse, do not overwrite
  own_v_q <= 1'b1; own_id_q <= claim_agent;
end

  duty : unowed_stale strict=0 faulty=1  false_wb=1

assign unowed_stale_err = stale_q && !held_q;    // registered state, always evaluable

  latency : owner local=3  owner across the link=6  no owner=3 (cycles)

assign target_q_next = (owner_valid && owner_is_remote) ? LINK_CYCLES : MEM_CYCLES;

  handover : gaps=1  transfers=1  covered during send->ack=1  new-owner-live during hand=0

assign old_answers = (ph_q == HANDING);          // old owner answers until the ack
assign new_owner_live = new_answers;             // and the directory waits for this
SETTLING: if (data_acked) begin ... end          // never on elapsed time

  directory : owner_not_sharer=0 empty_with_owner=0 evictions=3
  faulty directory : owner_not_sharer=1 empty_with_owner=1

if (ov_q && agent == ow_q) ov_q <= 1'b0;         // remove the owner, drop the duty
assign owner_not_sharer_err = ov_q && !sh_q[ow_q];

  arbitration : A won=1 then B won=1 | double_grant correct=0 faulty=1

assign a_wins = (req_a && !req_b) ? 1'b1 :
                (!req_a && req_b) ? 1'b0 : !prefer_b_q;

  writebacks over 8 downgrades : with O=8  without O=8
  reads answered by the owner  : 75%

else if (st_q == M) st_n = (HAS_OWNED != 0) ? O : S;

23. Design Review

What was built. Ten models: a MOESI line whose HAS_OWNED parameter makes it a MESI line, a registry enforcing one owner, a supply path with exactly one responder, a three-phase transfer, a drop detector parameterised by policy, a duty tracker with a fault-injection hook, a directory entry with two structural invariants, an arbiter that refuses rather than overwrites, a latency model for a remote owner, and an aggregate that measures what the Owned state actually saved.

What was measured. MOESI reached O where MESI reached S, with zero writebacks against one on identical stimulus. The owner supplied two reads. Evicting an owned line still cost a writeback. Three registry abuse cases were each caught. Four supply combinations produced exactly one responder each, with the dirty-no-owner case detected. One handover gap, and it was the deliberate one. Both drop policies lost the same data and only one demanded a writeback. Eight writebacks saved against eight performed. A remote owner cost 6 cycles against 3 for local memory. The owner answered 75% of reads.

What would be different in production. The directory would be a cache with its own eviction policy and back-invalidation. Ownership transfer would be pipelined and would carry a transaction identifier, because the flat handover here cannot support more than one migration at a time. The sharer vector would be compressed rather than one bit per agent. The arbiter would be pipelined, with an interlock covering the window the flat version does not have. None of that changes the invariant; all of it changes how many places the invariant can be violated.

The strongest argument against this design. The Owned state costs a third state bit on every line in the system, and it pays off only on lines that are written and then read by another agent and then invalidated before eviction. On a workload without that pattern it is pure cost. That argument is correct, and the measured wb_saved against n_wb_evict comparison is exactly how to settle it for a given workload rather than arguing it in the abstract.

What would be built differently next time. new_owner_live should have been an output from the start. It was added because a mutation proved it unobservable, but the design reason was already there — a directory cannot publish a new owner without it. An internal signal that a real consumer would need is a design smell before it is a verification one.

24. How This Appears In Real Engineering

In an architecture review, the question is not whether to implement MOESI. It is whether the workload has the write-then-share-then-invalidate pattern that makes the fifth state pay for its encoding. The measurement that settles it is wb_saved against n_wb_evict on a trace, and it is cheap to instrument in a model long before RTL exists.

In a coherency controller bring-up, the first three counters to check are no_supplier_err, unowed_stale_err, and n_claims minus n_releases. All three are silent failures — the system keeps running and returns wrong data — and all three are trivially observable if the monitors were put in.

In a performance investigation on a multi-host system, n_remote against n_local for a hot line frequently explains a latency anomaly that looks like cache thrashing. The instinct that a cache hit beats a memory access is correct on one die and wrong across a link.

In a verification plan review, the question to ask about every invariant monitor is how do you know it works. If the answer is that it has never fired, the monitor is untested. This chapter needed three fault-injection builds to answer that question for three monitors, and the same argument applies to every "this cannot happen" check in a coherency controller.

In silicon debug, the distinction between gap_err and new_owner_live is the difference between two bugs that produce identical symptoms. Designing the observability to separate them costs two signals at RTL time and saves a week of bisection after tapeout.

25. Common Misconceptions

"The owner is the agent that has the most rights to the line." The owner has fewer rights than a modifier — it has lost write permission — and more obligations than a sharer. Measured: in state O the model reports can_write=0 and is_owner=1 simultaneously.

"MOESI eliminates the writeback." It defers it. Measured: evicting an owned line still produced wb_on_evict=1. The saving is realised only when the line is invalidated before it is evicted.

"Cache-to-cache transfer is always faster than memory." Measured across a link: 6 cycles from a remote owner against 3 from local memory. The comparison depends entirely on where the owner sits.

"A dirty line with no owner is a recoverable error." It is a data-loss condition that has already occurred. The current value has no source; memory's copy is stale. Detection is after the fact, which is why the drop path must prevent it rather than the supply path detect it.

"Ownership and holding a copy are the same thing." They are separate fields for a reason. An agent can hold a line and owe nothing — that is state S. The directory in section 12 stores both, and the two invariants it checks are precisely statements about their relationship.

"If no error counter fired, the run was clean." Three of the monitors in this chapter cannot fire in a correct design. A zero from a monitor that has never been proven reachable is not evidence of anything. This is the reason for the FAULT_INJECT convention.

"The arbiter only matters under contention." The livelock in section 13 appears under light load, where a lone requester loses to a static preference. Uncontested cases need explicit handling.

"A writeback can come from any agent holding the line." Only from the recorded owner. A writeback from a non-owner may carry a stale value and would mark memory clean using it. Measured: false_wb_err=1 on exactly that stimulus, with memory left stale.

26. Interview Reasoning

27. Exercises

1. Calculation. A 64-agent system tracks 4 MB of lines at 64 bytes each. Compute the directory storage for a full sharer vector plus an owner field and a valid bit, then compute it again for MESI without the owner field, and express the difference as a percentage of tracked capacity.

2. Analysis. A workload reports wb_saved=1200 and n_wb_evict=1150. State what fraction of the deferred writebacks were ultimately performed anyway, what that says about the workload's invalidation pattern, and whether the Owned state is paying for its encoding on this workload.

3. RTL task. Extend owner_transfer to support two concurrent migrations for different lines. State what state that requires per migration, and the failure that becomes possible if the two share a single phase register.

4. Assertion task. Write the property proving that a dirty line always has exactly one owner. Then explain why it passes trivially on a design where dirty is derived from is_owner, and what independent source of dirtiness is required to make it meaningful.

5. Design task. Add a mechanism that lets an owner shed the duty without evicting the line — a writeback that leaves the copy in S. State which of the models in this chapter must change, and what new failure mode you have introduced.

6. Testbench design. Design the stimulus that distinguishes a correct arbiter from one that grants both requesters. Explain why a monitor watching for unanswered requests reports a clean run on the faulty design, and name the counter that separates them.

7. Debug task. A system reports rising n_claims with n_releases flat, and no error counters set. Give your investigation order, name the condition that would eventually be violated, and explain why it is silent until then.

8. Design review. A colleague proposes removing the owner_id != requester term from the supply path, arguing that an agent never requests a line it owns. Give the strongest version of that argument, then name two situations in which it is false and the measurement that would reveal them.

28. Summary

Ownership is a debt, not a right.

• The owner holds a value memory does not have. dirty and is_owner are the same expression in the model, and that identity is the invariant: a dirty line always has an owner, an owner always holds dirty data.
• MESI writes back because it has nowhere to record the debt. Measured on identical stimulus from one parameterised source: MOESI reached O with 0 writebacks, MESI reached S with 1.
• MOESI defers the writeback, it does not remove it. Evicting an owned line still cost 1 writeback. The saving is realised only when the line is invalidated before eviction.
• Exactly one owner, enforced by refusal. A second claimant was rejected rather than allowed to displace the incumbent, and all three registry abuse cases — two_owner_err, wrong_release_err, orphan_dirty_err — were caught at 1 each.
• Exactly one responder per read. Owner, memory, or nobody — and the third case, a dirty line with no owner, is a data-loss condition detected after the fact rather than prevented.
• Transfers overlap, they do not abut. Reads driven in all three phases produced 1 gap across the run, and it was the deliberate pre-ownership one; new_owner_live stayed low while the data was in flight.
• Both drop policies lost the same data. Measured lost=1 each; only the strict build raised needs_wb. The difference between correct and broken here is what the protocol tells the agent, not the outcome under identical stimulus.
• Where the owner sits changes the sign of the optimisation. 6 cycles from a remote owner against 3 from local memory, with the owner answering 75% of reads.
• The aggregate settles the argument. 8 writebacks saved against 8 performed on identical stimulus, with the percentage computed at a width that cannot wrap and an empty sample reporting zero rather than a hundred.
• Verification: 146 assertion sites, 98 of 98 mutations killed, zero surviving. Sixteen first-run escapes were five stimulus gaps, four unobserved outputs, three unreachable checkers needing fault-injection hooks, three missing stability checks, and one provably equivalent mutation that was fixed in the design rather than recorded as an escape.

Next: 13.4 CXL State Management, which takes the states this chapter used informally and builds the full space: the encoding, the transition table, the machinery that applies it, and what has to be true for a transition to be legal at all.