Coherent Write Flows

14.1 built the four-phase shape of a coherent read: classify, gather, accept, complete.

A write has the same shape and one extra phase, and that phase is the whole subject. A read needs data. A write needs data and permission, and permission is not granted on request — it is acquired, by a transaction of its own, with a duration set by agents the writer does not control.

1. The Engineering Problem — Permission Is Acquired, Not Granted

An agent holding a line in E or M writes it in a cycle. Nothing in this chapter applies to that case.

Every other write has to obtain write permission first, and three things make that harder than obtaining data.

Permission requires everyone else to give something up. A fetch asks one supplier for a copy. An upgrade asks N sharers to stop reading, and it is not finished until every one of them has acknowledged. The duration is set by the slowest, so adding sharers moves the worst case, not the mean.

The old value stays readable for the whole acquisition. Between the moment the writer decides to write and the moment the last sharer acknowledges, every sharer is still serving reads of the value the writer is about to replace. 13.2 measured that window as a property of a link; here it is a property of a flow, and section 10 measures how many reads are served inside it.

The agent that already has the data still cannot write. This is the case that produces the most wasted bandwidth in a real system. An agent in S holds the line; a design that treats "I need to write and cannot" as "I need the line" re-fetches data it already has, on every upgrade in the system. Section 7 measures that: the faulty build turned two upgrades into two extra fetches.

The correctness argument from 14.1 still applies. Every step of a write flow must be an edge the graph in 13.4 actually has — and the one that catches designs is S straight to M, which looks obviously right and does not exist.

2. The One-Sentence Model

A write is a read plus an acquisition. The acquisition is a transaction with a fan-out, a completion condition that names every sharer rather than counting them, and a window during which the value being replaced is still the one everybody else can see.

Call it acquire, then write. Every defect in this chapter is a write that happened before the acquisition finished, or an acquisition that finished before it should have.

3. What This Chapter Owns

Deferred:

The arbitration between two competing writers is 13.3's ground and is not rebuilt here. Section 11 uses a fixed priority and spends its attention on what happens to the loser, which is a flow question rather than an arbitration one.

4. Teaching-Model Boundary

Every model below is a teaching model, compiled and simulated with Icarus Verilog 13.0, checked by a testbench whose oracle is structurally different from the design.

What these models are not: a coherency controller. There is no link layer, no credit scheme, no store buffer, no write combining and no memory-consistency model. A production write path has a store queue in front of everything here and a merge policy this chapter does not model.

The conventions from Module 13 and 14.1 carry over. Every checker tests cond !== 1'b1. Unreachable monitors get a FAULT_INJECT build of the same source. Comparisons are one source under a parameter, instantiated twice, driven from one stimulus stream. Every displayed value is a captured signal, and every combinational sample is preceded by a settle.

This chapter uses eight parameterised twin builds — GRANT_ON_REQUEST, COUNT_ONLY, FAULT_INJECT on the classifier, FIRE_AND_FORGET, FAULT_INJECT and DROP_LOSER on the race, SKIP_READ_FOR_PARTIAL, GRANT_IS_ENOUGH and EARLY_M.

5. RTL 1 — Five Phases, And The Extra One

A read has four phases. A write has five, and the extra one is the only phase in which the agent already has the data and still cannot proceed:

  // Writing while an invalidation is still outstanding is the write-side form
  // of the exposure window: a sharer is still serving reads of the old value.
  assign write_before_acks_err = (ph_q == UPG) && !acks_done && ack;
  // GRANT_ON_REQUEST treats a permission request as a permission grant, which
  // is the single most common write-flow bug.
  assign skipped_upgrade_err = (GRANT_ON_REQUEST != 0) && need_upg_q
                               && (ph_q == DONE);

and the phase ordering that matters most:

        DATA: if (data_arrived) ph_q <= need_upg_q ? UPG : DONE;

A fetch is not permission. A write to a line the agent does not hold fetches the data and then still upgrades, because arriving data says nothing about whether anyone else is reading. A design that goes straight from the fetch to done has skipped the acquisition entirely.

Three flows were driven. Measured:

  flow    : flows=3 direct=1 upgrades=1 fetches=1 | write-before-acks=1
            upgrade held without acks, phase stayed 2
  holds   : upgrade phase=2 fetch phase=3 | idle busy=0 done-phase ack flagged=0

Both hold checks matter more than they look. The flow held in the upgrade phase across multiple cycles with no acknowledgements and did not advance; it held in the fetch phase with no data and did not advance. A phase machine that advances on elapsed time passes every directed test in which the event happens to be prompt — and prompt is what a directed test naturally produces.

The GRANT_ON_REQUEST build reached done on the same stimulus that sent the correct build to the upgrade phase. It is the shape of a design that treats asking for permission as having it.

6. RTL 2 — The Upgrade Is A Transaction

Not a permission change. A transaction, with a beginning, a fan-out, and a completion condition:

  // COUNT_ONLY grants when enough acks have arrived without checking WHICH
  // agents sent them -- so a duplicate from one sharer covers for a silent one.
  assign granted = busy_q && ((COUNT_ONLY != 0) ? (n_acks_c >= need_c)
                                                : (got_q == want_q));
  assign dup_ack_err   = ack_valid && busy_q &&  got_q[ack_from];
  assign stray_ack_err = ack_valid && busy_q && !want_q[ack_from];

got_q == want_q, not a count. That is the entire difference between the two builds, and it is worth stating why counting is wrong: a duplicate acknowledgement from one sharer raises the count without any second sharer having given the line up. The count reaches the threshold, the write is granted, and a sharer that never answered is still serving reads.

Three sharers were asked, with a duplicate and a stray injected. Measured:

  upgrade : needed=3 seen=3 max fan-out=4 | dup=1 stray=1

Two error signals for two different failures:

• dup_ack_err — one sharer acknowledged twice. Usually a retransmission, and the second must not count toward the set.
• stray_ack_err — an acknowledgement from an agent that was never asked. That is a directory whose sharer list disagrees with reality, which is 13.5's filter problem surfacing in a flow.

The peak fan-out is latched at 4, from an earlier upgrade that was abandoned. A narrower upgrade afterwards did not reduce it — the same latching discipline as every duration counter in this module, for the same reason: the widest fan-out is the one that explains a latency outlier, and it has long since finished by the time anyone looks.

The oracle for the upgrade is a per-agent boolean set rather than a bit vector, so a vector-indexing bug in the design cannot appear identically in the reference.

7. RTL 3 — Four Writes, And The Expensive Mistake

  // The distinction that matters: an upgrade already HAS the data. Fetching it
  // again is wasted bandwidth on every upgrade in the system.
  assign blocked  = acc && transient;
  assign direct   = acc && !transient && writable;
  assign upgrade  = acc && !transient && readable && !writable;
  assign fetch    = acc && !transient && (!readable
                    || ((FAULT_INJECT != 0) && readable && !writable));

The whole state space was driven. Measured:

  classify: direct=2 upgrade=2 fetch=1 blocked=3 | faulty build fetches=3 multi=1

Two direct writes (E and M), two upgrades (S and O — both hold the data), one fetch (only I holds nothing), and three blocked (the transients).

The faulty build turned both upgrades into fetches: three fetches instead of one. Each of those is a full cache line moved across the interconnect for data the agent already had in its own array. On a workload with any write sharing, that is the difference between an upgrade transaction and an upgrade transaction plus a line transfer, on every single upgrade.

Note where O sits. An owned line holds the data and lacks write permission, so it upgrades exactly like S — the duty it carries from 13.3 does not help it write. A classifier that only tests for S sends every owned write down the fetch path.

multi_class_err is unreachable in the correct design, so the same source carries a FAULT_INJECT build in which a shared line is classified as both an upgrade and a fetch. Measured: correct=0, faulty=1.

No data moves in this diagram at all. The writer already had the line; the entire transaction is about the other two agents agreeing to stop reading it. That is why an upgrade and a fetch are different flows with different costs, and why conflating them is expensive.

8. Waveform — The Window Between Deciding And Writing

Transcribed from the printed cycle trace of the assembled flow in section 15.

The early_er row is high for four consecutive cycles, and during every one of them the line is marked writable while sharers have not given it up. That is not a transient glitch — it is a four-cycle window in which the single-writer invariant is false and any read from a sharer returns a value the writer believes it has replaced.

The line and earlyM rows diverge at cycle 3 and reconverge at cycle 7. Everything outside that window looks identical, which is why the failure survives any test that samples only at the start and the end of the flow.

9. RTL 4 — Revoking Is Not Asking

A read's snoop asks. A write's invalidation revokes. The difference is that a sharer which has not answered is still serving reads:

  // FIRE_AND_FORGET grants as soon as the invalidations are sent, without
  // waiting for any acknowledgement.
  assign granted = (FIRE_AND_FORGET != 0) ? done_q : (done_q && all_revoked);
  // The measurement that matters: a sharer still serving reads AFTER the
  // writer was told it may proceed.
  assign stale_read_err = granted && read_by_sharer && hold_q[reading_sharer];

Three sharers were revoked, with a read served by an unacknowledged sharer in the middle. Measured:

  revoke  : holders 3 -> 0, granted=1 stale=0 | fire-and-forget granted early=1 stale=1
            grant held with one sharer outstanding=0

The correct build served no stale read; the fire-and-forget build served one. And note precisely what stale_read_err tests: not that a sharer served a read during the revocation — that is legal and expected — but that a sharer served a read after the writer was told it may proceed. Those two are cycles apart and only the second is a violation.

The grant held with one sharer outstanding for as long as that sharer took. There is no timeout on the grant, deliberately: a design that grants after a fixed wait has converted a correctness property into a latency assumption.

10. RTL 5 — The Window, Measured As A Flow Property

13.2 measured the exposure window as a property of a link. This measures it as a property of a flow: how long the old value stays readable after the writer decided to replace it, and how many reads are served inside it.

          cnt_q <= cnt_q + 8'd1;
          // Latch the peak: the window that caused the stale read has closed
          // long before anyone investigates.
          if (cnt_q + 8'd1 > max_window) max_window <= cnt_q + 8'd1;

Measured across three windows — one with reads inside it, one shorter, one that never closed:

  window  : open 5 cycles, reads served inside it=5, peak latched=5, timeout=1

Five reads were served from the value the writer had already decided to replace. Every one of them is correct: the sharers still hold valid copies and the write has not happened. The number matters because it is the size of the correctness argument — if the write had been permitted at the start of that window, all five would have been stale.

The shorter window afterwards did not reduce the latched peak, and a window that ran past its bound raised still_exposed_err rather than waiting indefinitely. A flow that waits forever for an acknowledgement that will never arrive is indistinguishable from one that is merely slow, and the timeout is the only thing that separates them.

11. RTL 6 — Two Writers, And What The Loser Has To Do

Two agents write one line. The arbitration is 13.3's ground; what is new here is the loser's situation:

  // The point of the model: B held the data and could have upgraded. After A
  // writes, B's copy is invalid, so B's next attempt is a FETCH -- a different
  // flow with a different cost, not the same request retried.
  assign b_now_needs_fetch = reclass_b && b_had_data;
  assign two_writers_err = grant_a && grant_b;

Measured across three builds:

  race    : reclass=1 B-needs-fetch=1 | two_writers correct=0 faulty=1 lost(drop build)=1

B did not lose a race and get to try again. B lost a race and now has a different problem. Before the race B held the line in S and needed an upgrade — a transaction with no data movement. After A's write, B's copy is invalid and B needs a fetch and an upgrade. The retry is not the same request; it is a more expensive flow, and a design that simply re-issues the original request will re-issue an upgrade for a line it no longer holds.

The three builds again produce three different failures. The correct one reclassifies. The FAULT_INJECT build grants both writers — and its lost_req_err is clean, because every requester was answered. The DROP_LOSER build neither grants nor reclassifies, and is the only build that can reach the lost-request monitor at all.

12. RTL 7 — Writing Part Of A Line

Coherency works on lines; software writes bytes:

  // A full-line write overwrites every byte, so the old contents are
  // irrelevant and no read is needed. A partial write is a read-modify-write.
  assign needs_read  = wr && !full_line && !have_line
                       && (SKIP_READ_FOR_PARTIAL == 0);
  // Writing part of a line the agent does not hold means the untouched bytes
  // were never read. Whatever ends up there is invented.
  assign fabricated_err = wr && !full_line && !have_line && may_write;

Three writes were driven — full-line without the line, partial with it, partial without it. Measured:

  partial : full=1 partial=2 needs_read=1 fabricated correct=0 skip-read=1

A full-line write needs no read at all, and that is a real optimisation rather than a shortcut: every byte is being replaced, so the old contents are genuinely irrelevant. A design that fetches the line first for a full-line write is moving 64 bytes it will immediately overwrite.

A partial write without the line is a read-modify-write, and skipping the read does not produce a slightly-wrong line — it produces a line whose untouched bytes were never read from anywhere. The SKIP_READ_FOR_PARTIAL build wrote anyway and fabricated_err fired. Whatever those bytes contain came from the cache's previous occupant, from reset state, or from nothing.

13. RTL 8 — A Write Needs Permission And An Order

  // GRANT_IS_ENOUGH retires on the permission alone, so the write is performed
  // without ever being placed in the coherence order.
  assign complete = live_q && ((GRANT_IS_ENOUGH != 0) ? g_q : (g_q && o_q));
  assign unordered_write_err = complete && !o_q;

Measured:

  complete: grant-only build retired early=1 unordered=1 tag_mismatch=1

The grant and the ordering response answer different questions. The grant says you may write this — every sharer has given it up. The ordering response says your write has been placed in the coherence order — the serialisation point has decided where this write sits relative to every other access to the line. A write that is permitted but unordered has no defined position, and 13.1's "a read returns the latest write in the coherence order" has no meaning for it.

Both halves are tag-matched, in both directions. A wrong-tagged grant and a wrong-tagged ordering response were both driven and both reported, and neither counted. The ordering half is the one usually left unchecked, and a mutation removing that check survives any test that only mismatches the grant.

14. RTL 9 — What The Write Paths Cost

  logic [31:0] weighted;   // 16 x 100 needs 23 bits; 16 would silently wrap
  assign upgrade_pct  = (total == 17'd0) ? 8'd0 : (weighted / {15'd0, total});
  assign mean_sharers = (n_upgrade == 16'd0) ? 8'd0
                      : ({16'd0, total_sharers} / {16'd0, n_upgrade});

Ten writes: five direct, two upgrades at 30 and 50 cycles with 1 and 3 sharers, three fetches at 80, 60 and 70. Measured:

  cost    : direct=5 upgrade=2 fetch=3 | upgrade share=20% mean upg=40 fetch=70 sharers=2

An upgrade cost 40 cycles against a fetch's 70 — because an upgrade moves no data, only agreement. That is the number that justifies having it as a separate flow: conflating the two costs 30 cycles and a cache line on every upgrade in the system.

mean_sharers is the counter nobody adds and the one that explains an upgrade latency distribution. An upgrade with one sharer and an upgrade with seven are the same transaction with wildly different durations, and averaging their latencies without knowing the fan-out produces a number that describes neither.

15. RTL 10 — The Flow Assembled

Classify, fetch if needed, upgrade, done — with the invariant a read does not have:

  // The invariant: the line may not be M while sharers are still outstanding.
  assign writable_too_soon_err = (line_q == M) && pend_q;

Three writes were driven — direct, upgrade, and fetch-then-upgrade. Measured:

  assembled: writes=3 upgrade stages=2 | writable-too-soon correct=0 early-M=1
  holds   : upgrade stage=3 fetch stage=2 (both wait, neither advances)

Two of the three writes needed an upgrade stage, including the one that started from I — because a fetch delivers data and the sharers still have to be dealt with. The flow enters the write-intent transient, fetches, moves to the upgrade transient, and only then reaches M.

The EARLY_M build marked the line writable on entering the upgrade stage rather than on completing it, and the monitor caught it. That is the failure the waveform in section 8 shows in full: four cycles of a line marked writable while sharers are still reading it.

16. Quantitative Reasoning

An upgrade is cheaper than a fetch and not free. Measured: 40 cycles against 70, with no data movement. At a 64-byte line, conflating them costs 64 bytes of interconnect traffic plus 30 cycles on every upgrade. At the measured 20% upgrade share, that is 20% of all writes paying a full line transfer for data the agent already held.

Fan-out sets the tail, not the mean. An upgrade completes when the slowest sharer answers, so adding sharers moves the worst case. With the measured mean fan-out of 2 and a per-sharer response uniformly distributed, the expected upgrade duration is the maximum of two draws rather than one — and at a fan-out of 7 it is the maximum of seven, which is far closer to the distribution's upper bound than to its mean.

The exposure window is the acquisition latency. Measured at 5 cycles with 5 reads served inside it. Every one was correct; all five would have been stale had the write been permitted at the window's start. Doubling the acquisition latency doubles the number of reads that must be served from the old value — which is a correctness cost only if the design gets the grant condition wrong, and a latency cost regardless.

A full-line write saves a whole line transfer. No read is needed, because every byte is replaced. At a 64-byte line and a 70-cycle fetch, recognising full-line writes saves 64 bytes and 70 cycles each time. A store buffer that merges adjacent partial writes into full-line writes is therefore worth its area precisely to the extent that it turns partials into fulls.

Two-part completion costs two comparators per outstanding write. The grant tag and the ordering tag are matched independently against the transaction identity. At 16 outstanding writes and a 3-bit tag that is 96 bits of comparison — against the alternative, which is a write performed with permission it has but an order it does not.

17. Assertions

Presented as SystemVerilog and executed as procedural checkers — see section 19.

A write never completes while invalidations are outstanding.

property p_no_early_write;
  @(posedge clk) disable iff (!rst_n)
    (phase == UPG && !acks_done) |-> !write_done;
endproperty

An upgrade is granted only when every named sharer has acknowledged.

property p_all_sharers;
  @(posedge clk) disable iff (!rst_n)  granted |-> (acks_seen == acks_needed);
endproperty

A duplicate acknowledgement does not advance the set.

property p_no_dup_credit;
  @(posedge clk) disable iff (!rst_n)
    (ack_valid && already_acked) |-> dup_ack_err;
endproperty

An acknowledgement from an agent never asked is reported.

property p_no_stray;
  @(posedge clk) disable iff (!rst_n)
    (ack_valid && !in_sharer_set) |-> stray_ack_err;
endproperty

Exactly one classification per write.

property p_one_wclass;
  @(posedge clk) disable iff (!rst_n)
    acc |-> $countones({direct, upgrade, fetch, blocked}) == 1;
endproperty

An agent that holds the line never fetches it again.

property p_no_refetch;
  @(posedge clk) disable iff (!rst_n)  (acc && readable) |-> !fetch;
endproperty

No sharer serves a read after the writer is granted.

property p_no_stale_read;
  @(posedge clk) disable iff (!rst_n)
    (granted && read_by_sharer) |-> !still_holding[reading_sharer];
endproperty

Never two writers granted for one line.

property p_single_writer;
  @(posedge clk) disable iff (!rst_n)  !(grant_a && grant_b);
endproperty

A losing writer is reclassified, never dropped.

property p_loser_reclassified;
  @(posedge clk) disable iff (!rst_n)  req_b |-> (grant_b || reclass_b);
endproperty

A partial write never proceeds without the line.

property p_no_fabrication;
  @(posedge clk) disable iff (!rst_n)
    (wr && !full_line && !have_line) |-> !may_write;
endproperty

A write completes only when permitted and ordered.

property p_permitted_and_ordered;
  @(posedge clk) disable iff (!rst_n)  complete |-> (got_grant && got_order);
endproperty

A line is never writable while sharers are outstanding.

property p_not_writable_early;
  @(posedge clk) disable iff (!rst_n)  (line_st == M) |-> !sharers_pending;
endproperty

18. Mutation Testing

90 mutations were injected into the ten models, one at a time, each a single-line change a competent engineer could plausibly write. Every one must make the testbench print RESULT: FAIL.

Representative mutations, all killed:

Ten mutations survived the first run. None was patched away.

Six stimulus gaps. The bench never held the flow in the fetch phase with no data, never held the assembled flow in its fetch stage, never checked busy while idle, never exercised the top bit of the sharer vector, never issued a second upgrade start while one was running, and never issued a second revocation start while one was running. Six cases added, six mutations killed.

Two unobserved outputs. The completion flag sampled while an acknowledgement was actually asserted in the done phase — the check existed but sampled a cycle early, so a mutation that flagged every completion as premature passed. And the latched peak fan-out, which needed a wider upgrade earlier in the run before a narrower one could prove the latch.

One unreachable checker. lost_req_err cannot fire in either the correct build or the both-granted build, because both answer every requester. It needed a third build, DROP_LOSER, exactly as 14.1 needed one for the same monitor in the read race. That the same shape recurred in two consecutive chapters is worth noting: a monitor for "nothing was lost" is almost always unreachable, because a design that loses requests is not a design anyone writes deliberately.

One display defect found by the discipline rather than by a mutation. A summary line printed nc_upgrade after a later stimulus had incremented it, so the printed value did not match the value the assertion checked. The counter was latched into a named integer at the assertion point. No mutation caught this — the rule that every displayed value must be a captured signal did.

19. Verification Strategy

The oracle must not be the design. Each testbench models the same behaviour in a structurally different representation.

For write_classify the design produces four one-hot booleans; the oracle produces a single integer outcome code derived from two facts — do I hold it, may I write it — so a design that asserts two outcomes cannot be reproduced by a reference that structurally holds one.

For upgrade_txn the design holds a bit vector and compares it against the wanted set. The oracle holds four per-agent booleans and a function that asks whether all are clear:

  function integer o_all_acked;
    begin o_all_acked = (o0==0) && (o1==0) && (o2==0) && (o3==0); end
  endfunction

That shape is what makes the count-only bug visible: the oracle has no notion of a count at all, so a build that reaches its threshold with a duplicate cannot agree with it.

For inval_fanout the design holds a holder vector; the oracle holds four plain integers, so a vector-indexing bug cannot appear identically in both.

For write_completion the design has one complete expression; the oracle holds two independent booleans and an explicit AND.

Every displayed value is a captured signal, latched into a named integer before any later event changes it — a rule that caught a real defect in this chapter's own summary output.

Delta-cycle discipline. Every sample of a combinational output is preceded by a settle. Two checks in this chapter additionally needed an extra clock: the EARLY_M build commits its premature M one cycle into the upgrade stage, so sampling at stage entry sees the two builds agreeing.

Coverage recorded: 151 assertion sites across three testbenches; all four write classifications driven across the whole state space; upgrades driven at fan-outs of 4 and 3 with duplicates, strays and a second start; revocations driven with a read served by an unacknowledged sharer and a second start; windows driven long, short and unterminated; the race driven uncontested in both directions and contested across three builds; partial writes driven full, partial-with-line and partial-without; completion driven with each half alone, both, and mismatched tags in both directions; and the assembled flow driven direct, upgrade, and fetch-then-upgrade with both stages held.

20. Synthesis and Implementation Reality

The upgrade tracker is a bit vector per outstanding write. One "asked" bit and one "acknowledged" bit per potential sharer, plus the comparison between them. At 64 agents that is 128 bits per outstanding write, and the equality comparison across it is on the grant path — which is why real designs compress the sharer representation, and why Modules 15 and 16 own that.

The equality test, not a counter, is the expensive-looking part — and it must stay. A population count and a threshold comparison is narrower logic than a 64-bit equality. It is also wrong, for the reason section 6 measures. Where timing forces the count, the duplicate-acknowledgement check becomes mandatory rather than optional, because the count is only safe if duplicates cannot reach it.

The exposure window counter is per outstanding write and the peak is global. The per-write counter controls the timeout; the latched global peak is a diagnostic. Sharing one counter across writes makes the timeout fire on whichever write is unluckiest rather than on the one that is actually stuck.

Byte enables travel with the write from the core. The full-line test is an AND-reduce over them, which is cheap, but it has to happen before the flow classifies — a design that classifies first and discovers the write is full-line afterwards has already issued the fetch.

writable_too_soon_err is two comparisons and belongs in silicon. It compares the line state against the sharer-pending flag, both of which already exist. It is unreachable in a correct design, which is exactly why a zero reading proves nothing without the EARLY_M build behind it in the regression.

Reset must leave no upgrade pending. A write path that comes out of reset with a sharer-pending flag set will never grant, and one that comes out with the flag clear and the line in the upgrade transient will grant immediately for a transaction that never asked anybody.

21. Silicon Observability

Five error signals belong in silicon. write_before_acks_err, stray_ack_err, stale_read_err, unordered_write_err and writable_too_soon_err all detect states from which no correct behaviour is possible, and each is a handful of gates over signals that already exist. All five are unreachable in a correct design, so each needs a fault-injection build behind it before a zero reading means anything.

mean_sharers is the counter that makes an upgrade latency histogram interpretable. Without it, an upgrade at fan-out 1 and one at fan-out 7 are averaged into a number describing neither. With it, a rising mean explains a rising latency without any change in the interconnect.

n_partial against n_full is a software finding. A workload dominated by partial writes is forcing a read-modify-write on every store to a line it does not hold, and the fix is in the store buffer's merge policy or the data layout rather than in the coherency path.

22. Debug Lab

  revoke  : granted=1 stale=0 | fire-and-forget granted early=1 stale=1

assign granted = done_q && all_revoked;    // never on elapsed time

  upgrade : needed=3 seen=3 max fan-out=4 | dup=1 stray=1

assign granted     = busy_q && (got_q == want_q);   // the set, not a count
assign dup_ack_err = ack_valid && busy_q && got_q[ack_from];

  classify: direct=2 upgrade=2 fetch=1 blocked=3 | faulty build fetches=3 multi=1

assign upgrade = acc && !transient && readable && !writable;   // S and O both
assign fetch   = acc && !transient && !readable;

  complete: grant-only build retired early=1 unordered=1 tag_mismatch=1

assign complete = live_q && got_grant && got_order;

  partial : full=1 partial=2 needs_read=1 fabricated correct=0 skip-read=1

assign needs_read = wr && !full_line && !have_line;
assign may_write  = wr && (full_line || have_line);

  race    : reclass=1 B-needs-fetch=1 | two_writers correct=0 faulty=1 lost=1

assign reclass_b         = req_b && req_a;
assign b_now_needs_fetch = reclass_b && b_had_data;

  window  : open 5 cycles, reads served inside it=5, peak latched=5, timeout=1

assign still_exposed_err = exp_q && (cnt_q >= TIMEOUT);
if (cnt_q + 8'd1 > max_window) max_window <= cnt_q + 8'd1;

  assembled: writes=3 upgrade stages=2 | writable-too-soon correct=0 early-M=1

UPG: if (acks_done) begin
       line_q <= M; pend_q <= 1'b0; st_q <= DONE;    // only on completion
     end

23. Design Review

What was built. Ten models: a five-phase flow machine with a grant-on-request twin, an upgrade transaction with a count-only twin, a four-way classifier with a fault-injection build, an invalidation fan-out with a fire-and-forget twin, an exposure-window meter, a race resolver in three builds, a partial-write gate with a skip-read twin, a two-half completion with a grant-only twin, a cost model, and an assembled flow with an early-M twin.

What was measured. Three flows with both hold checks passing and premature completion caught. An upgrade granted only on the full acknowledgement set, with a duplicate and a stray both reported and neither counted, at a latched peak fan-out of 4. Four classifications with the faulty build turning two upgrades into three fetches. A revocation that served no stale read against a fire-and-forget build that served one. A window open 5 cycles with 5 reads served inside it, a shorter one that did not reduce the peak, and one that timed out. A race whose loser needed a different flow, not a retry. A partial write refused without the line while the skip-read build fabricated bytes. A completion needing both halves with mismatched tags caught in both directions. Upgrades at 40 cycles against fetches at 70, at a mean fan-out of 2. An assembled flow where the early-M build was writable four cycles too soon.

What would be different in production. A store queue sits in front of everything here, with a merge policy that turns partial writes into full-line writes and changes the measured partial/full ratio entirely. The sharer vector is compressed. The acquisition is pipelined, which reintroduces a window between the grant decision and the state update. None of that changes the five phases; all of it multiplies where they can be got wrong.

The strongest argument against this design. Comparing the full acknowledgement set is wider logic than counting acknowledgements against a threshold, and it sits on the grant path. That argument is correct about the timing, and the honest response is not to reject it but to state the condition: a count is safe only if duplicates cannot reach it. If timing forces the count, the duplicate check moves from optional to mandatory, and it must be upstream of the counter rather than beside it.

What would be built differently next time. The exposure-window model and the invalidation fan-out both track the same acquisition and were written as separate modules. That was right for teaching — one measures duration, the other measures correctness — but it means the timeout and the grant condition read the same event through two paths that could disagree. In production they are one structure, and the model should have said so rather than leaving a reader to infer it.

24. How This Appears In Real Engineering

In a microarchitecture review, the question that separates a specified write path from an aspirational one is what the grant condition actually is. "When the invalidations have been sent" and "when they have been acknowledged" are one word apart in a specification and several cycles apart in silicon.

In a performance investigation, n_upgrade against n_fetch is the first ratio to look at on a write-heavy workload. A fetch count close to the write count on a workload with sharing means lines are being fetched that the agent already held.

In bring-up, mean_sharers makes an upgrade latency histogram interpretable. Without it, fan-out 1 and fan-out 7 are averaged into a number that describes neither, and a rising latency looks like an interconnect regression when it is a sharing-pattern change.

In a verification plan review, ask whether the upgrade grant has been tested with a duplicate acknowledgement. That single case separates a design that names its sharers from one that counts them, and no ordinary directed test produces it.

In silicon debug, the distinction between "the acquisition is slow" and "the acquisition is stuck" does not exist without a bound. Both present as a line that stops accepting writes, and only a timeout that reports rather than completes tells them apart.

In a design review of somebody else's write path, ask what a losing writer does next. If the answer is "retries", the design is replaying a request that encodes an assumption the winner has already invalidated.

25. Common Misconceptions

"A write is a read with a different opcode." A write needs data and permission, and permission is acquired by a separate transaction with its own fan-out and duration. Measured: two of three writes needed an upgrade stage, including the one that started with no data at all.

"An agent that has the line can write it." Only in E or M. Measured: S and O both hold the data and both need an upgrade, and treating either as a miss re-fetches a line already present.

"A fetch delivers permission along with the data." It does not. Arriving data says nothing about whether anyone else is reading. Measured: the flow from I fetched and then still upgraded.

"Counting acknowledgements is equivalent to naming them." A duplicate from one sharer raises a count without a second sharer having given the line up. Measured: the count-only build reaches its threshold with one sharer silent.

"The invalidation is done when it is sent." Until a sharer acknowledges, it is still serving reads. Measured: the fire-and-forget build granted at issue and a sharer served the old value after the grant.

"A read served during a revocation is a stale read." It is not — the sharer holds a valid copy and the write has not happened. The violation is a read served after the writer was granted, which is cycles later. Conflating them produces an alarm that fires on legal traffic.

"A partial write is just a smaller write." It is a read-modify-write. The bytes it does not touch must come from a line the agent actually holds, and skipping the read invents them.

"A permission grant means the write is done." The grant says every sharer gave the line up; the ordering response says where the write sits in the coherence order. A write with the first and not the second has no defined position.

26. Interview Reasoning

27. Exercises

1. Calculation. A workload issues 1000 writes at a 20% upgrade share, with upgrades at 40 cycles and fetches at 70. Compute the total write latency, then compute it again for a design that re-fetches on every upgrade, and express the difference in both cycles and bytes at a 64-byte line.

2. Analysis. An upgrade's per-sharer response latency is uniformly distributed between 5 and 25 cycles. Explain why the expected upgrade duration at a fan-out of 4 is not 15, state roughly what it is, and name the counter that would let you check your estimate against silicon.

3. RTL task. Extend upgrade_txn to support a sharer that responds with "I no longer hold this line" rather than an acknowledgement. State whether that response should advance the set, and the directory failure it reveals.

4. Assertion task. Write the property proving a line is never writable while sharers are outstanding. Then explain why it passes trivially on a design that derives the sharer-pending flag from the line state, and what independent source of that flag is required to make it meaningful.

5. Design task. Add a speculative write path that performs the store into a shadow buffer before the acquisition completes. State what must be squashed if the acquisition fails, which of this chapter's monitors must change, and what new failure mode you have introduced.

6. Testbench design. Design the stimulus that distinguishes an upgrade which names its sharers from one which counts acknowledgements. Explain why every test with prompt, distinct acknowledgements passes on both, and state the minimum stimulus that separates them.

7. Debug task. A system reports rising n_fetch with a flat n_upgrade on a workload known to have write sharing. Give your investigation order, name the state most likely being misclassified, and explain why the bug is invisible on a workload without that state.

8. Design review. A colleague proposes removing the byte-enable test and always fetching the line before a write, arguing that it removes a special case. Give the strongest version of that argument, then the cost at a 64-byte line and a 70-cycle fetch, and the workload characteristic that decides which is right.

28. Summary

A write is a read plus an acquisition.

• Permission is acquired, not granted. Measured: two of three writes needed an upgrade stage, including the one that started from I — because a fetch delivers data and says nothing about who else is reading.
• The upgrade names its sharers. got_q == want_q, never a count: a duplicate from one sharer would otherwise cover for a silent one. Duplicate and stray acknowledgements were both reported and neither counted, at a latched peak fan-out of 4.
• An upgrade already has the data. S and O both hold the line and need only permission. The build that re-fetched them turned two upgrades into three fetches — a full line transfer per upgrade, forever.
• An invalidation that was sent is not one that happened. The fire-and-forget build granted at issue and a sharer served the old value after the grant; the correct build served none.
• The window is real and its reads are correct. 5 cycles, 5 reads served from the value being replaced. All five would have been stale had the write been permitted at the window's start.
• A losing writer has a different problem, not a delayed one. It held the line and needed an upgrade; after the winner's write it needs a fetch and an upgrade.
• A partial write is a read-modify-write. Refused without the line; the skip-read build performed it and fabricated the bytes it never read. A full-line write correctly needs no read at all.
• A write needs permission and a position. The grant says the sharers gave it up; the ordering response says where the write sits in the coherence order. The grant-only build retired one half short.
• Upgrades cost 40 cycles against fetches at 70, at a mean fan-out of 2 — the number that justifies having the two as separate flows.
• The line becomes writable at the end of the acquisition, not the start. The early-M build was writable for four cycles while sharers still held it, with everything before and after looking identical.
• Verification: 151 assertion sites, 90 of 90 mutations killed, zero surviving. Ten first-run escapes were six stimulus gaps, two unobserved outputs, and one unreachable checker needing a third build — plus one display defect caught by the captured-signal rule rather than by any mutation.

Next: 14.3 Ownership-Transfer Flows, which takes the duty 13.3 defined and moves it between agents as an actual message sequence — including the case where a third agent asks for the line while it is in mid-air.