Coherent Read Flows

Module 13 built the machinery — permissions, windows, ownership, a state space and a domain boundary — and deferred the flows five separate times.

This is where that deferral is paid. A coherent read is not one thing, it is four, and only one of them is not a transaction at all.

1. The Engineering Problem — A Read Is A Transaction

A read that hits is an access: the line is present, permission is sufficient, and the data is returned in a cycle. Nothing in this chapter applies to it.

Every other read is a transaction — a sequence of messages with a beginning, a middle in which the line is in a state no diagram names, and an end that is not the moment the data arrives. Three things make that hard.

The line is in a transient for the whole middle. 13.4 established that a transaction spends most of its life in a state between the old one and the new one. A read flow is what drives that transient, and a flow that abandons it leaves a line nothing owns.

The data does not come from where the request went. A read miss for a line another agent has modified goes to the serialisation point and is answered by a peer. Three parties, two of which never talk to each other directly, and the requester has to know which of them is allowed to answer.

The flow can be interrupted at every phase. A snoop for the same line can arrive mid-flow. The home can tell the requester to try again. A response can arrive for a transaction that has already retired. Each of these is legitimate traffic, and each has a wrong answer that looks like a reasonable optimisation.

The correctness argument for the whole module is one sentence: a flow is a sequence of state transitions, and every one of them must be an edge the graph in 13.4 actually has. Section 6 builds that check as RTL and uses it on a real flow.

2. The One-Sentence Model

A read is a path, not a step. It classifies itself from the line's state, walks a sequence of legal transitions through a transient, accepts data only from the agent the serialisation point named, and is finished only when every outstanding answer has arrived.

Call it classify, gather, accept, complete. Every defect in this chapter is a phase skipped, a phase left early, or a phase that accepted something from the wrong place.

3. What This Chapter Owns

Deferred:

This chapter owns the read and nothing else. Where a write appears it is stimulus, not subject. The state graph is used here and owned by 13.4 — section 6 walks it and does not extend it.

4. Teaching-Model Boundary

Every model below is a teaching model, compiled and simulated with Icarus Verilog 13.0, checked by a testbench whose oracle is structurally different from the design.

What these models are not: a coherency controller. There is no link layer, no credit scheme, no MSHR allocation policy, no address decode and no error correction. A production read path has one state machine per outstanding transaction, a deep response-matching structure, and a retry policy tuned against real contention traces.

The conventions from Module 13 carry over and are used without further comment. Every checker tests cond !== 1'b1, never !cond. Unreachable monitors get a FAULT_INJECT build of the same source. Comparisons between two policies are one source under a parameter, instantiated twice and driven from one stimulus stream. Every displayed value is a captured signal. Every combinational sample is preceded by a settle.

This chapter uses eight parameterised twin builds — SKIP_SNOOP, FAULT_INJECT on the classifier, FIRST_WINS, TRUST_ANY_SOURCE, FAULT_INJECT and DROP_LOSER on the race, NO_CEILING, DATA_IS_ENOUGH and LEAK_TRANSIENT. Every comparison below is a measured difference between two instances of one file.

5. RTL 1 — The Four Phases Of A Read

A read miss has four phases, and the phase is not the line's state. The line is in a transient for most of this; the phase is what the flow is doing about it.

  // Accepting data while a snoop is still outstanding is the single most
  // common read-flow bug: the data may be superseded by what the snoop returns.
  assign data_before_snoop_err = data_arrived && (ph_q == SNOOP) && !snoop_done;
  assign early_complete_err    = ack && (ph_q != DATA) && (ph_q != DONE);

and the phase machine itself:

        ISSUED: begin
                  // SKIP_SNOOP goes straight for the data even when peers hold
                  // the line -- fast, and wrong.
                  if (snoop_needed_q) begin
                    ph_q <= SNOOP; n_snooped <= n_snooped + 8'd1;
                  end else begin
                    ph_q <= DATA;  n_direct  <= n_direct  + 8'd1;
                  end
                end

Three flows were driven: one with no peers, one that must snoop, and one acknowledged before its data arrived. Measured:

  flow    : flows=3 snooped=1 direct=1 | data-before-snoop=1 early-complete=1

Both premature-completion cases were caught. The flow held in the data phase across multiple cycles with no data and did not advance; it held in the snoop phase with no snoop response and did not advance. Those two hold checks matter more than they look: a phase machine that advances on elapsed time rather than on an event will pass every directed test in which the event happens to be prompt.

The SKIP_SNOOP build went straight for the data on the same stimulus that sent the correct build to the snoop phase. It is faster, it is simpler, and the data it accepts may be the value a peer is about to supersede.

Memory has no line in this diagram at all, and that is the point. The data moved between two caches, and the only reason the serialisation point was involved is that somebody has to decide which peer answers and in what order competing requests are served.

6. RTL 2 — Every Flow Is A Path Through The Graph

This is the correctness argument for the whole module, built as RTL.

  // The same eight states and six causes as 13.4. This module does not own the
  // graph -- it walks it, which is exactly the boundary between the modules.

The model exposes one question: is this step an edge the graph has?

  assign illegal_path_err = step_en && !edge_legal;
  assign path_complete    = step_en && edge_legal && (to_st <= O);

The testbench sweeps all 8 source states × 6 causes × 8 destinations = 384 combinations against an independently written edge list — the same list 13.4 used, transcribed separately so a transcription error cannot appear in both. Measured:

  graph   : 28 legal edges of 384; illegal steps taken=1

Twenty-eight edges, matching 13.4 exactly. The sweep is run with the step counter disabled, because 356 illegal combinations pouring into a counter the path walk uses would make that counter meaningless — a small point, and the first version of this bench got it wrong.

Then the flow from section 5 is walked as a path:

  I --(local read)--> IS_D      legal, not yet landed
  IS_D --(fill)-----> E         legal, landed on a stable state

and the flow the SKIP_SNOOP build would take:

  I --(local read)--> E         NO SUCH EDGE

That is the whole argument. A design that fetches a line straight into E without passing through the fill transient is not taking a shortcut through the state machine — it is taking a transition the state machine does not have, and every invariant that depends on the transient holding the line is unenforced for the duration. The check costs a lookup and it is the cheapest correctness argument available in this module.

7. RTL 3 — Which Read Is This?

Four outcomes, exactly one per access, decided from the line's state and where the current data lives:

  // A hit is the only read that is not a flow at all. Everything else is a
  // transaction, and which one depends on where the current data lives.
  assign hit       = acc && (resident || ((FAULT_INJECT != 0) && transient));
  assign blocked   = acc && transient;
  assign miss_fwd  = acc && !resident && !transient &&  peer_has_dirty;
  assign miss_mem  = acc && !resident && !transient && !peer_has_dirty;

The whole state space was driven, for a clean and a dirty peer. Measured:

  classify: hit=4 mem=1 fwd=1 blocked=4 | multi correct=0 faulty=1

Four hits — S, E, M and O all hold a readable copy, which is the 13.1 permission table restated. Four blocked — the three transients plus one more, and every one of those is an access that failed for a reason that has nothing to do with permission. One forward and one memory fetch, distinguished only by whether a peer holds dirty data.

The miss_fwd versus miss_mem split is the one that matters, and it is not a performance distinction. A miss with a dirty peer must be forwarded: memory's copy is stale, so fetching it returns a value that was superseded. The oracle for this is four mutually exclusive integer outcomes rather than four booleans, so a design bug that asserts two of them cannot be reproduced by a reference that structurally has only one.

multi_class_err cannot fire in a correct design — the four conditions are mutually exclusive by construction. That makes it an unreachable checker, so the same source carries a FAULT_INJECT build that drops the transient guard from the hit test and classifies an in-flight line as both a hit and blocked. Measured: correct=0, faulty=1.

8. Waveform — A Read Miss End To End

Transcribed from the printed cycle trace of the assembled flow in section 15.

The line row is in a transient for four of the seven cycles, which is 13.4's measurement restated from the flow's side rather than the line's.

The landing state is the detail worth pausing on. The fill lands in S, not E, because the peer that supplied the data still holds its copy. A design that lands every fill in E has granted exclusivity while another agent is still reading — and the flow that produced it looks identical up to the final cycle.

The trace continues past what the waveform shows. A second flow is started and abandoned mid-gather:

  11      0  0  0    gather  IS_D     0       0
  12      0  0  0    idle    I        0       1

At cycle 12 the correct build has restored the line to I and the LEAK_TRANSIENT build has not — it is idle with the line still in IS_D, a transient with no flow driving it and nothing that will ever complete it. That is the last column, and it is the failure section 15 exists to demonstrate.

9. RTL 4 — Gathering Every Answer

A read miss asks every peer that might hold the line. The flow may not proceed until all of them have answered:

  // FIRST_WINS declares the gather complete as soon as one peer supplies data,
  // which leaves the other peers' answers to arrive after the flow has moved on.
  assign all_answered = busy_q && ((FIRST_WINS != 0) ? data_q : (cnt == 3'd0));
  assign dup_rsp_err  = rsp_valid && busy_q && seen_q[rsp_peer];
  assign two_data_err = rsp_valid && busy_q && rsp_has_data && data_q
                        && !seen_q[rsp_peer];

Three peers were asked; the second to answer supplied the data. Measured:

  gather  : outstanding falls to 0, all_answered=1, data from peer 3
            first-wins build complete with a peer still outstanding=1
            dup_rsp=1 two_data=1 responses=6

The FIRST_WINS build declared the gather complete with a peer still outstanding, and that peer's response then arrives after the flow has moved on — where it is either dropped, or applied to a transaction that no longer exists. Waiting for "enough" answers rather than all of them is the shape of that bug, and it is attractive because the data is already in hand.

The two error signals separate failures that look identical from the outside:

• dup_rsp_err — one peer answered twice. Usually a retransmission, and the second copy must not be counted.
• two_data_err — two different peers both claimed to hold the data. That is a directory that named two suppliers, which is the 13.3 two-owner failure surfacing in a flow.

The && !seen_q[rsp_peer] term is what keeps them apart. Without it, a repeat from the peer that already supplied data is reported as two claimants — an alarm that fires on a retransmission, which is exactly the kind of false positive that gets a monitor disabled. Measured on that stimulus: duplicate reported, two-claimants not.

The oracle for the gather is four plain integers counting who still owes an answer, so a bit-vector indexing bug in the design cannot reproduce itself in the reference.

10. RTL 5 — The Data Comes From Somewhere Else

The requester sent one message, to the serialisation point. The data arrives from a peer it never contacted. It must accept that data from exactly one agent — the one the home named.

  // The requester accepts data only from the agent the home named. Accepting
  // from anyone is how a stale copy from a peer that was just invalidated
  // gets installed as current.
  assign accept = data_valid && armed_q
                  && ((TRUST_ANY_SOURCE != 0) || (data_from == src_q));
  assign wrong_source_err = data_valid && armed_q && (data_from != src_q);
  assign unsolicited_err  = data_valid && !armed_q;

Data was driven from the wrong peer, then the right one, then with nothing outstanding. Measured:

  forward : rejected=1 forwarded=1 wrong_source=1 unsolicited=1

The wrong-source case is not paranoia. A peer that held the line a moment ago, was invalidated, and had a data message already in flight will deliver a stale copy that is perfectly well-formed. The requester has no way to tell it is stale except that it came from an agent the home did not name. The TRUST_ANY_SOURCE build accepted it on identical stimulus.

unsolicited_err catches the other direction — data arriving with nothing outstanding. That is a duplicate, a response to a transaction that already timed out, or a response for a line this agent never requested, and all three are worth separating from a response that is merely late.

The req && home_directs guard is worth its own sentence. A request the home does not redirect leaves the requester expecting nothing from a peer, and the model checks that explicitly: with no forwarder named, expect_from_peer stays low and any peer data is unsolicited.

11. RTL 6 — A Read Racing A Snoop

A read for a line crosses a snoop for the same line. Both are legitimate, and they cannot both be applied to the same starting state:

  // The snoop wins: it carries an ordering decision the serialisation point has
  // already made, while the read can be replayed against whatever it leaves.
  assign serve_snoop = snoop_req;
  assign serve_read  = (FAULT_INJECT != 0) ? read_req : (read_req && !snoop_req);
  assign read_replay = read_req && snoop_req && (FAULT_INJECT == 0)
                       && (DROP_LOSER == 0);
  assign both_served_err = serve_read && serve_snoop;
  assign dropped_err = (read_req && !serve_read && !read_replay)
                    || (snoop_req && !serve_snoop);

The snoop wins, and the priority is not arbitrary. A snoop carries an ordering decision the serialisation point has already committed to; the read has not been ordered yet and can be replayed against whatever the snoop leaves behind. Reversing the priority means a snoop that has already been counted as delivered is applied to a state the read has since changed.

Three builds were driven on identical stimulus. Measured:

  race    : replays=1 both_served correct=0 faulty=1 dropped(drop build)=1

Read those three numbers together, because they are three different bugs:

• The correct build served the snoop, refused the read, and replayed it.
• The FAULT_INJECT build served both, applying two changes to one starting state — and note that its dropped_err is clean, because every request was answered. A monitor watching for unanswered requests reports a healthy run on this bug.
• The DROP_LOSER build neither served the read nor replayed it. It is the only build that can reach the dropped-request monitor at all, which is why it exists.

12. RTL 7 — The Read That Has To Be Retried

A read the home cannot serve yet is nacked and retried. A retry budget with no ceiling is a livelock that never reports itself:

  // NO_CEILING retries forever, so a line that is permanently contended never
  // produces a report -- it just stops making progress, silently.
  assign give_up      = act_q && (NO_CEILING == 0) && (try_q >= MAX_TRIES[3:0]);
  assign livelock_err = act_q && (try_q >= MAX_TRIES[3:0]);

with the peak latched rather than sampled:

          // Latch the peak: the contended line is long gone by the time
          // anyone investigates.
          if ({4'd0, try_q} + 8'd1 > max_tries_seen)
            max_tries_seen <= {4'd0, try_q} + 8'd1;

Measured across a read that succeeded on its third attempt and one that never succeeded:

  retry   : peak attempts=4 give_up=1 | no-ceiling build still active=1

The bounded build gave up and reported a livelock; the unbounded build was still retrying nothing. That is the difference between a fault an engineer can find and a system that has quietly stopped making progress on one address.

Two details in the bench are worth naming because both hid a real mutation. Waiting is not retrying — the attempt counter must only advance on a nack, and a build that advances it every cycle while active passes any test that never idles mid-flow. And a later shorter attempt must not reduce the latched peak — the check needs a short run that does retry, because a run with no nack never touches the latch at all.

13. RTL 8 — When Is A Read Actually Finished

A read completes when the data and the completion response have both arrived, for the same transaction:

  // DATA_IS_ENOUGH retires on the data alone, which loses the response that
  // carries the final permission the requester is allowed to assume.
  assign complete = live_q && ((DATA_IS_ENOUGH != 0) ? d_q : (d_q && c_q));
  assign tag_mismatch_err = (data_en && live_q && (data_tag != tag_q))
                         || (comp_en && live_q && (comp_tag != tag_q));
  assign half_complete_err = complete && !(d_q && c_q);

Measured:

  complete: data-only build declared complete early=1 half_complete=1 tag_mismatch=1

The data is not the answer. The data tells the requester what the value is; the completion response tells it what permission it now holds and that the serialisation point has finished ordering the transaction. Retiring on the data alone means the requester proceeds with an assumption about its permission that nothing confirmed — and if the flow was a forwarded read, that assumption is often E when the correct answer is S.

Both halves are tag-matched, in both directions. A wrong-tagged data beat and a wrong-tagged response were both driven while a transaction was live, and both were reported and neither counted. That is the 13.2 identity-matching result applied to a two-part completion, and the response half is the one usually left unchecked.

The retirement path clears both halves. Without that, a fresh transaction inherits the previous one's state and completes on the arrival of a single beat — measured explicitly by checking that with nothing outstanding, neither half is still showing.

14. RTL 9 — What Each Path Cost

  logic [31:0] weighted;    // 16 x 100 needs 23 bits; 16 would silently wrap
  assign total    = {1'b0, n_hit} + {1'b0, n_mem} + {1'b0, n_fwd};
  assign weighted = {16'd0, n_hit} * 32'd100;
  assign hit_rate_pct = (total == 17'd0) ? 8'd0 : (weighted / {15'd0, total});

Ten reads: seven hits, two memory fetches at 40 and 60 cycles, one forward at 20. Measured:

  cost    : hits=7 fetches=2 forwards=1 | hit rate=70% mean fetch=50 fwd=20

The forward was faster than the fetch — 20 cycles against 50 — which is the single-die case where cache-to-cache transfer is the win it is reputed to be. 13.3 measured the opposite across a CXL link, and the two results are not in conflict: the comparison depends entirely on whether the supplying peer is nearer than memory. Separating the two means the counter can answer the question rather than assuming it.

The width comment and the empty-sample guard are the same defect classes as Modules 12 and 13. The hit rate reports zero before any access, not a hundred, and the product is computed in 32 bits because a 16-bit multiply by 100 wraps above 655 samples.

15. RTL 10 — The Flow Assembled

Classify, gather, fill, done — with one invariant that spans the whole thing:

  // The invariant: when the flow is not running, the line must be stable. A
  // flow that aborts without restoring the line leaks a transient nobody owns.
  assign stuck_transient_err = (st_q == IDLE) && (line_q > 4'd4);

and the landing state that depends on how the data arrived:

        FILL:  if (data_ok) begin
                 line_q <= fwd_q ? S : E;        // forwarded lines are shared
                 st_q   <= DONE;
               end

Four flows were driven — a hit, a forwarded miss, a memory miss, and an abandoned flow. Measured:

  assembled: completed=3 aborted=1 stuck_transient correct=0 leaking=1
  holds   : gather stage=2 fill stage=3 (both wait, neither advances)

A hit reached the done stage without entering a transient at all, which is the section 7 classification showing up in the flow. A forwarded miss landed SHARED and a memory miss landed EXCLUSIVE, from identical flow logic differing only in whether a peer supplied the data.

The abort is the interesting case. The correct build restored the line to I; the LEAK_TRANSIENT build left it in IS_D with the flow idle. That line is now unreachable: every access to it classifies as blocked, and nothing will ever complete the transient because the flow that owned it is gone. It is a permanent, silent loss of one cache line, and the only thing that detects it is a monitor comparing the flow's stage against the line's state.

16. Quantitative Reasoning

How much of a read is the transient. Measured on the assembled flow: the line was in IS_D for four of the seven cycles a forwarded miss took — 57%. That fraction is set by the snoop round trip, not by the workload, so it rises with link latency and is the reason 13.4's transient discussion is not an implementation detail.

What the classification is worth. At the measured 70% hit rate, seven of ten reads are not transactions at all. The remaining three cost 50 cycles (fetch) or 20 (forward). Mean read latency is therefore 0.7 × 1 + 0.2 × 50 + 0.1 × 20 = 12.7 cycles — and moving one fetch to a forward moves it to 9.7, a 24% improvement from a change that touches only where the data lives.

Snoop fan-out sets the floor on a miss. The flow cannot proceed until every asked peer answers, so the gather costs the slowest response, not the average. With four peers and a 6-cycle mean response at a 12-cycle worst case, the expected gather is much closer to 12 than to 6 — and adding peers moves the maximum, not the mean. This is why snoop filters exist, and Modules 15 and 16 own that.

The retry budget is a latency-versus-detection trade. A budget of 4 caps the wasted work on a contended line at four round trips before the flow gives up and reports. An unbounded retry has no cap and no report. At the measured peak of 4 attempts, the difference between the two builds is a bounded 4-round-trip loss against an unbounded silent stall.

Completion needs two tags, not one. Matching both the data and the response against the transaction identity costs one comparator per half, per outstanding transaction. At 16 outstanding transactions and a 3-bit tag that is 96 bits of comparison — trivially cheap against the alternative, which is retiring the wrong transaction the first time two are in flight.

17. Assertions

Presented as SystemVerilog and executed as procedural checkers — see section 19.

Data is never accepted while a snoop is outstanding.

property p_no_early_data;
  @(posedge clk) disable iff (!rst_n)
    (phase == SNOOP && !snoop_done) |-> !data_accepted;
endproperty

A flow does not complete before its data.

property p_no_early_complete;
  @(posedge clk) disable iff (!rst_n)  ack |-> (phase == DATA || phase == DONE);
endproperty

Every step of a flow is an edge the graph has.

property p_legal_path;
  @(posedge clk) disable iff (!rst_n)  step_en |-> edge_legal;
endproperty

Exactly one classification per access.

property p_one_class;
  @(posedge clk) disable iff (!rst_n)
    acc |-> $countones({hit, miss_mem, miss_fwd, blocked}) == 1;
endproperty

A miss with a dirty peer is never served from memory.

property p_no_stale_fetch;
  @(posedge clk) disable iff (!rst_n)  (acc && peer_has_dirty) |-> !miss_mem;
endproperty

The gather completes only when every peer has answered.

property p_full_gather;
  @(posedge clk) disable iff (!rst_n)  all_answered |-> (outstanding == 0);
endproperty

Never two peers claiming the data.

property p_one_supplier;
  @(posedge clk) disable iff (!rst_n)
    (rsp_valid && rsp_has_data && data_seen && !already_seen) |-> two_data_err;
endproperty

Data is accepted only from the named forwarder.

property p_named_source;
  @(posedge clk) disable iff (!rst_n)  accept |-> (data_from == expected_source);
endproperty

A read and a snoop are never both applied to one state.

property p_no_double_apply;
  @(posedge clk) disable iff (!rst_n)  !(serve_read && serve_snoop);
endproperty

Every request is served, replayed, or reported.

property p_nothing_dropped;
  @(posedge clk) disable iff (!rst_n)  read_req |-> (serve_read || read_replay);
endproperty

A read completes only with both halves.

property p_both_halves;
  @(posedge clk) disable iff (!rst_n)  complete |-> (got_data && got_comp);
endproperty

An idle flow leaves no line in a transient.

property p_no_stranded_line;
  @(posedge clk) disable iff (!rst_n)  (stage == IDLE) |-> (line_st <= O);
endproperty

18. Mutation Testing

94 mutations were injected into the ten models, one at a time, each a single-line change a competent engineer could plausibly write. Every one must make the testbench print RESULT: FAIL.

Representative mutations, all killed:

Twenty mutations survived the first run. None was patched away; each was classified and either the testbench or the design was extended.

Thirteen stimulus gaps. The bench never held a flow in the data phase with no data, never held it in the snoop phase with no response, never held the assembled flow in the gather or the fill, never idled while a retry was active, never issued a second gather start while one was running, never issued a request the home declined to redirect, never sent a wrong-tagged completion response (only a wrong-tagged data beat), never started a second transaction while one was live, never repeated a response from the peer that had already supplied data, and never checked the outstanding count while idle. Thirteen cases added, thirteen mutations killed.

Five unobserved outputs. The outstanding count on a closed gather, the response count after a duplicate, the attempt count while merely waiting, the data half after retirement, and the hit rate before any access were all computed and never read.

Two unreachable checkers. multi_class_err and dropped_err are both unreachable in a correct design — that is what makes them invariants. The first needed a FAULT_INJECT build that drops the transient guard; the second needed a third build of read_snoop_race, DROP_LOSER, because neither the correct build nor the both-served build ever drops a request. Adding a parameter specifically to reach a monitor is not a workaround; it is the only honest way to know the monitor works.

Three findings worth stating separately, because each hid a mutation and each is a real lesson:

1. The exhaustive sweep must not feed the path counter. Sweeping 384 combinations with the step enable high poured 356 illegal steps into the counter the path walk uses, making it meaningless. The sweep reads the combinational legality only.
2. Waiting is not retrying. A build that advances the attempt counter every cycle while a flow is active, rather than only on a nack, passes every test in which the flow never idles.
3. A short run that does not retry cannot test the peak latch. The latch lives inside the nack branch, so the "a later shorter window must not reduce the peak" check needs a shorter run that does retry.

A survivor is a finding about the testbench, not a nuisance. Recording one as an acceptable escape converts a verification gap into a documented feature.

19. Verification Strategy

The oracle must not be the design. Each testbench models the same behaviour in a structurally different representation.

For read_classify the design produces four one-hot booleans. The oracle produces a single integer outcome code, so a design bug that asserts two of them cannot be reproduced by a reference that structurally has only one:

  function integer o_class(input integer st, input integer dirty);
    begin
      if      (st > 4)            o_class = 3;
      else if (st != 0)           o_class = 0;
      else if (dirty != 0)        o_class = 2;
      else                        o_class = 1;
    end
  endfunction

For path_check the design is a nested case statement over states and causes. The oracle is an explicit edge list written as integer comparisons, transcribed independently from 13.4's table, and the two are compared across all 384 combinations rather than on a sampled subset. Exhaustion is what finds an edge that exists and should not — a directed test can only find edges that are missing, because nobody writes a test for a transition they did not intend to implement.

For snoop_fanout the design holds a bit vector. The oracle holds four plain integers, so a vector-indexing bug cannot appear identically in both.

For read_completion the design has one complete expression. The oracle holds two independent booleans and an explicit AND, so a design that collapses the two halves cannot be validated by a reference that has already collapsed them.

Every displayed value is a captured signal. No $display in these benches prints a literal, and where a value is sampled before a later event changes it, it is latched into a named integer first.

Delta-cycle discipline. Every sample of a combinational output is preceded by a settle, because a continuous assignment read in the same delta as its driver changes returns the previous value. The waveform harness has a #1 at the top of its per-cycle task for exactly this reason.

Coverage recorded: 161 assertion sites across three testbenches; all four read classifications driven across the whole state space; all 384 graph combinations swept; gathers driven with duplicates, two claimants, and a second start mid-gather; forwarding driven with the right source, the wrong source, and nothing outstanding; the race driven uncontested in both directions and contested across three builds; retries driven to success, to the budget, and while idle; completion driven with each half alone, both, and mismatched tags in both directions; and the assembled flow driven as a hit, a forwarded miss, a memory miss and an abort.

20. Synthesis and Implementation Reality

The phase machine is per outstanding transaction, not per line. The model here has one; a real read path has one per MSHR, and the MSHR count is what bounds how many misses can be in flight. Every structure in this chapter is replicated that many times.

The gather is the expensive structure. It needs one bit per peer for "asked", one per peer for "answered", plus the supplier identity — and it is written on every response and read every cycle to compute the outstanding count. That combinational reduction across the peer vector is on the flow's critical path, and at large peer counts it is pipelined, which introduces a window in which the count is stale.

Two-tag completion matching is a small CAM. Each outstanding transaction's tag is compared against the incoming data tag and the incoming response tag, every cycle, in parallel. It is not large, but it is wide and it is on the response path.

The retry counter must be per transaction and must reset. A shared counter turns one contended line into a budget exhaustion for every flow behind it. The latched peak, by contrast, is deliberately global — it is a diagnostic, not a control.

stuck_transient_err is a handful of gates and belongs in silicon. It compares the flow's stage against the line's state, both of which already exist. A machine check on it converts a permanently unreachable cache line into a fault with an address attached, and the alternative is a line that silently stops serving accesses for the lifetime of the system.

Reset must leave no flow active and no line in a transient. A read path that comes out of reset with a stage register in GATHER will wait forever for responses to a request that was never sent — and its line is in a transient nothing will complete.

21. Silicon Observability

Five error signals belong in silicon, not just in simulation. data_before_snoop_err, two_data_err, wrong_source_err, half_complete_err and stuck_transient_err all detect states from which no correct behaviour is possible, and all five are a handful of gates over signals that already exist. Each is unreachable in a correct design, so a zero reading proves nothing until a fault-injection build has demonstrated it firing.

n_started minus n_completed should equal the transactions currently in flight. A persistent drift means a flow was abandoned — an MSHR freed without its phase machine retiring — and it is silent until the line is touched again, at which point it classifies as blocked forever.

mean_fwd against mean_mem is the counter that decides a placement question. If forwarding is slower than fetching on this system, the peers holding the data are further away than memory, and the fix is where the data lives rather than anything in the read path.

22. Debug Lab

  classify: hit=4 mem=1 fwd=1 blocked=4 | multi correct=0 faulty=1

assign miss_fwd = acc && !resident && !transient &&  peer_has_dirty;
assign miss_mem = acc && !resident && !transient && !peer_has_dirty;

  assembled: completed=3 aborted=1 stuck_transient correct=0 leaking=1

GATHER: if (abort) begin
          line_q <= I;                 // restore before releasing the flow
          st_q   <= IDLE;
        end

  flow    : flows=3 snooped=1 direct=1 | data-before-snoop=1 early-complete=1

SNOOP: if (snoop_done) ph_q <= DATA;      // never on elapsed time

  gather  : first-wins build complete with a peer still outstanding=1

assign all_answered = busy_q && (outstanding_count == 0);

  forward : rejected=1 forwarded=1 wrong_source=1 unsolicited=1

assign accept = data_valid && armed_q && (data_from == expected_source);

  race    : replays=1 both_served correct=0 faulty=1 dropped(drop build)=1

assign serve_snoop = snoop_req;
assign serve_read  = read_req && !snoop_req;
assign read_replay = read_req && snoop_req;

  retry   : peak attempts=4 give_up=1 | no-ceiling build still active=1

assign give_up      = act_q && (try_q >= MAX_TRIES);
assign livelock_err = act_q && (try_q >= MAX_TRIES);

  complete: data-only build declared complete early=1 half_complete=1 tag_mismatch=1

assign complete = live_q && got_data && got_comp;
assign tag_mismatch_err = (data_en && live_q && (data_tag != tag_q))
                       || (comp_en && live_q && (comp_tag != tag_q));

23. Design Review

What was built. Ten models: a four-phase flow machine with a snoop-skipping twin, a path checker that walks 13.4's graph, a four-way classifier with a fault-injection build, a snoop gather with a first-wins twin, a forwarding acceptor with a trust-anyone twin, a race resolver in three builds, a bounded retry with an unbounded twin, a two-half completion with a data-only twin, a cost model, and an assembled flow with a transient-leaking twin.

What was measured. Three flows with both premature-completion cases caught. 28 legal edges out of 384, matching 13.4 exactly, with the skip-snoop flow shown to have no edge. Four classifications across the whole state space with exactly one firing per access. A gather that stayed open with a peer outstanding while the first-wins build had already closed. Data rejected from an unnamed source and accepted from the named one. A race in which the snoop won, the read was replayed, and three different builds produced three different failures. A retry that gave up at a latched peak of 4 while the unbounded build was still going. A completion that needed both halves, with mismatched tags caught in both directions. A 70% hit rate with forwards at 20 cycles against fetches at 50. An abort that restored the line while the leaking build stranded it.

What would be different in production. Every structure here is replicated per MSHR. The gather's outstanding count is pipelined at scale, which reintroduces a stale-count window. The retry policy is tuned against real contention traces rather than a fixed budget. The forwarding path carries an address and a full transaction identity rather than a two-bit peer id. None of that changes the four phases; all of it multiplies the places they can be got wrong.

The strongest argument against this design. Waiting for every snoop response before accepting data costs the slowest peer on every miss, and the data is usually already in hand when the first peer answers. That argument is correct about the latency, and the measured consequence of acting on it is that abandoned peers' responses arrive after the flow has retired — where they are applied to a transaction that no longer exists or dropped, with the peer left holding a copy the requester believes was invalidated. The honest form of the optimisation is to use the data early and retire late, which is a different design from the one the FIRST_WINS build implements.

What would be built differently next time. The exhaustive graph sweep and the path counter should have been separated from the start. Feeding 356 illegal combinations into the counter the real path walk uses made that counter meaningless, and it took a mutation to notice. A sweep is a query, not a sequence of steps, and the RTL should have made that distinction structurally rather than leaving it to the testbench.

24. How This Appears In Real Engineering

In a microarchitecture review, the question that separates a specified read path from an aspirational one is what happens to a snoop that arrives mid-flow. If the answer is "it is handled by the arbiter", the design has a race nobody has enumerated.

In bring-up, n_started minus n_completed and stuck_transient_err are the first two things to look at. Both detect abandoned transactions, both are silent in every other view, and both are trivially cheap if they were designed in.

In a performance investigation, mean_fwd against mean_mem decides whether cache-to-cache transfer is helping. It is a single-die instinct that holds on a die and inverts across a link, and the counter is the only thing that will say which case this system is in.

In a verification plan review, the question to ask about the flow machine is whether it has been held in every phase with the advancing event absent. A phase machine driven by elapsed time passes every test in which the event is prompt, and prompt is what a directed test naturally produces.

In silicon debug, the distinction between a dropped read and a slow read does not exist at the requester — both are a timeout. The counter that separates them lives in the responder and has to be designed in before tapeout.

In a design review of somebody else's read path, ask when the transaction retires. If the answer is "when the data arrives", the design is assuming a permission that nothing confirmed, and on a forwarded read that assumption is usually wrong in the direction that breaks the single-writer invariant.

25. Common Misconceptions

"A read is an access." Only when it hits. Measured: 7 of 10 reads were hits and not transactions at all; the other 3 were multi-message flows spending most of their life in a transient.

"The data arriving means the read is done." The data says what the value is; the completion response says what permission the requester now holds. Measured: the data-only build declared completion one half short, and half_complete_err fired.

"The response comes from wherever the request went." On a forwarded read the requester never sends the supplying peer a message. Measured: data accepted from the named forwarder, rejected from another peer, and reported when it arrived with nothing outstanding.

"Once you have the data you can stop snooping." The other peers' responses carry state changes the flow needed to observe. Measured: the first-wins build closed its gather with a peer still outstanding.

"A read losing a race should be retried by the requester." It should be replayed against the state the winner left. A write that was legal against one state may need a different flow against the next, and simply reissuing the same request re-enters the classifier from the top — which is a property of the surrounding pipeline, not of the race resolver.

"An unbounded retry is safer than giving up." It is the opposite. Measured: the bounded build reported a livelock; the unbounded build made no progress and raised nothing, which presents as a performance problem rather than a fault.

"Every fill lands in Exclusive." Only when nothing else holds the line. Measured: a forwarded fill landed SHARED and a memory fill landed EXCLUSIVE, from identical flow logic differing only in whether a peer supplied the data.

"An aborted flow just frees the transaction." It must also restore the line. Measured: the leaking build left a line in a fill transient with no flow driving it — permanently unreachable, and detected by nothing except a monitor comparing stage against state.

26. Interview Reasoning

27. Exercises

1. Calculation. A system reads at a 70% hit rate, with fetches at 50 cycles and forwards at 20. Compute the mean read latency, then compute it again if half the fetches become forwards, and express the improvement as a percentage.

2. Analysis. A gather asks 8 peers whose response latencies are uniformly distributed between 4 and 16 cycles. Explain why the expected gather time is not 10, state what it is closer to, and name the structural change that reduces it.

3. RTL task. Extend snoop_fanout to support two concurrent gathers for different lines. State what state that requires per gather, and the failure that becomes possible if the two share one supplier register.

4. Assertion task. Write the property proving a read completes only with both halves. Then explain why it passes trivially on a design where the response is generated from the data beat, and what independent source of the completion response is required to make it meaningful.

5. Design task. Add a speculative-use path that forwards the data to the requesting core before the gather completes, without retiring the transaction. State what must be squashed if a snoop response contradicts the speculation, and which of this chapter's monitors must change.

6. Testbench design. Design the stimulus that distinguishes a flow machine driven by events from one driven by elapsed time. Explain why a test that drives the advancing event promptly passes on both, and state the minimum stimulus that separates them.

7. Debug task. A system reports n_started rising with n_completed tracking it exactly, and yet one address is permanently blocked. Give your investigation order, name the monitor that detects it directly, and explain why the transaction counters do not.

8. Design review. A colleague proposes removing the source check on forwarded data, arguing that the home would not name a peer that cannot supply the line. Give the strongest version of that argument, then the situation in which it is false, and the measurement that would reveal it.

28. Summary

A read is a path, not a step.

• Only a hit is not a transaction. Measured across the whole state space: 4 hits, 1 memory fetch, 1 forward, and 4 blocked — the blocked case being more common than either permission problem.
• Every flow is a path through 13.4's graph. 28 legal edges out of 384 combinations, matching that chapter exactly, and the skip-snoop flow was shown to have no edge at all.
• A miss with a dirty peer is never served from memory. Memory's copy is stale by definition; fetching it completes normally and returns a superseded value.
• The gather completes on answers, not on data. The first-wins build closed with one of three peers still outstanding, abandoning responses that arrive after the flow has retired.
• The data comes from an agent the requester never contacted, and it must be accepted only from the one the serialisation point named. Rejected from another peer, accepted from the named one, reported when unsolicited.
• In a race the snoop wins and the read is replayed. Three builds produced three distinct failures, and the both-served build's dropped_err stayed clean — a monitor watching for unanswered requests reports a healthy run on it.
• A retry needs a ceiling and a report. Latched peak of 4 attempts, with the unbounded build still retrying nothing and raising nothing.
• A read finishes on both halves. The data says what the value is; the response says what permission was granted. The data-only build declared completion one half short.
• A forwarded fill lands SHARED, a memory fill lands EXCLUSIVE, from identical logic differing only in whether a peer supplied the data.
• An abandoned flow must restore its line. The leaking build stranded a line in a fill transient — permanently unreachable, detected by nothing but a monitor comparing stage against state.
• Verification: 161 assertion sites, 94 of 94 mutations killed, zero surviving. Twenty first-run escapes were thirteen stimulus gaps, five unobserved outputs, and two unreachable checkers — one of which needed a third build of the same module to be reachable at all.

Next: 14.2 Coherent Write Flows, which takes the same four-phase shape and adds the thing a read never needs: permission the agent does not yet have, acquired as a transaction of its own.