Senior Coherency Question

27.7 designed a fabric. This chapter walks a single line through it — take a coherent transaction from one agent to another, end to end — and it is the question that separates somebody who has read about coherency from somebody who has debugged it.

The snoop invalidates it. True of one case out of several, and it is where most answers stop: one message, one outcome, no states, no bias, no ordering, no completion, and nothing about what happens when a step does not arrive.

1. The Engineering Problem — A Vocabulary Is Not A Walk

The walk is the answer. Eleven steps, four of them named, is thirty-six percent of a transaction — and naming the participants is not sequencing them. Section 5.

A snoop does not always invalidate. The same hundred lines snooped for a read give zero invalidations and thirty downgrades, and thirty lines of data moving that an invalidate-only answer never mentions. Section 6.

Bias decides what an access costs. Four hundred of a thousand device accesses resolved through the host at six hundred nanoseconds each is three hundred and thirty microseconds against a hundred and fifty — a hundred and eighty of pure bias penalty. Section 7.

The host tracks what the device holds. Four thousand lines wanted against a thousand-entry directory is two thousand nine hundred and seventy-six forced back-invalidations, each one a line taken from a device that was using it. Section 8.

The data arriving is not the transaction finishing. Data in hand at two hundred nanoseconds with three acknowledgements owed resolves at three hundred and eighty — a hundred and eighty nanoseconds during which the requester has the data and does not yet own it. Section 12.

This chapter against 27.3, stated precisely. That one owns why a device would want a coherent cache and what it costs. This one owns the transaction itself — the sequence, the states, the ordering and the completion — at the level of detail a senior interview actually asks for. 26.7 owns finding one of these when it has gone wrong in silicon.

2. The One-Sentence Model

A coherent transaction has been walked when the snoop is named, when every step from request to completion is in the sequence, when the line's states are named at each step, when bias is accounted for, when what must be ordered is distinguished from what need not be, and when completion is defined as something other than the data arriving — and "the snoop invalidates it" is one of those six.

3. What This Chapter Owns

Some vocabulary, because the senior question is asked in words that each carry a precise meaning and are often used loosely.

A request node is an agent that can ask for a line. A host core is one; a CXL Type 1 or Type 2 device with a caching agent is another. "Cross-RN" means a transaction whose requester and current owner are different request nodes — the case where the line has to move between agents rather than being fetched from memory.

A home node is the serialisation point for an address. Every request for a given line goes to the same home, and that is what makes the order well-defined: two racing requests are ordered because the home decides which one it processes first. The home holds the directory — section 8's subject — and issues the snoops.

A snoop is a message from the home to an agent that may hold the line, telling it what is about to happen and what it must do about it. Section 6 is entirely about the "what it must do" part, which is where the invalidate-only answer goes wrong.

Bias is which side of a Type 2 device's memory is authoritative right now. In device bias the device reaches its own memory directly; in host bias the same access is resolved through the host's coherency. Section 7.

And completion is the point at which the requester both has the data and owns the state it was granted. Those are two different moments, and section 12 is the distance between them.

4. Teaching-Model Boundary

Every model in this chapter is a teaching model, not a coherency implementation. It computes the one relationship the section is about and nothing else. There is no state machine, no message encoding and no protocol engine anywhere in this file.

Each model is built twice from one source. A parameter selects between the measured build, which counts what the transaction actually does, and the recited build, which reports what the short answer says. Every section's headline number is the gap between them.

5. RTL 1 — The Walk Is The Answer

Start with the shape of the question, because it decides what a good answer even looks like.

The senior coherency question is not a definition question. It does not ask what MESI stands for or what a snoop is. It asks you to take one line from one agent to another and say what happens, in order, with the states named at each step — and the answer is graded on whether the sequence is complete, not on whether the vocabulary is right.

That distinction matters because the two are easy to confuse from the inside. An answer full of correct nouns — requester, home, snoop, directory, writeback — feels like an answer. Naming the participants is not sequencing them, and the model exists to put a number on the difference.

// RTL 1 - a walk is a sequence of steps, and an answer that names four of
// eleven has walked a third of the transaction. The senior question is not
// "do you know what a snoop is", it is "can you get a line from one agent to
// another without leaving a step out".
module walk_completeness #(parameter int NAMING_THE_PARTS_IS_WALKING = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] steps_total, steps_named, parts_named, detail_wanted,
  output logic [15:0] named_ok, steps_missing, walk_pct, detail_given,
  output logic        walked,
  output logic [7:0]  n_evals, n_partial,
  output logic        walk_err
);
  logic [31:0] w_q;
  logic [15:0] true_missing;
  logic        truly_partial;
  // An answer cannot name more steps than the transaction has.
  assign named_ok = (steps_named > steps_total) ? steps_total : steps_named;
  assign true_missing = steps_total - named_ok;
  assign steps_missing = (NAMING_THE_PARTS_IS_WALKING != 0) ? 16'd0 : true_missing;
  // Naming the participants is not the same as sequencing them; the parts
  // view reports its own count of nouns as though it were the walk.
  assign detail_given = (parts_named > detail_wanted) ? detail_wanted : parts_named;
  assign w_q = (steps_total == 16'd0) ? 32'd100
             : (({16'd0, named_ok} * 32'd100) / {16'd0, steps_total});
  assign walk_pct = w_q[15:0];
  assign walked = (steps_missing == 16'd0) && (steps_total != 16'd0);
  // No steps_total guard: true_missing is steps_total less a minimum against
  // steps_total, so a walk with no steps already has nothing missing.
  assign truly_partial = (true_missing != 16'd0);
  assign walk_err = evaluate && truly_partial && walked;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_partial <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_partial) n_partial <= n_partial + 8'd1;
    end
  end
endmodule

Eleven steps with four named is thirty-six percent of a transaction, with the parts all correctly named alongside — and the naming view reports a completed walk.

It is worth writing the eleven steps down, because "the walk" is otherwise an instruction nobody can check themselves against. The requester misses and issues a request to the home for the address. The home serialises it against anything else outstanding for that line. The home consults the directory to find out who holds it and in what state. The home issues snoops to whichever agents the directory names. Each snooped agent acts on its copy — invalidating, downgrading, or doing nothing, which is section 6. An agent holding it modified supplies the data, either to the requester directly or through the home. The home updates the directory to record the new owner and state. The data reaches the requester. The requester acknowledges, and any agent owed an acknowledgement receives it. The home considers the transaction complete and is free to process the next request for that line. And the requester installs the line in the granted state — which is not always the state it asked for.

Figure 1 — the walk with nothing left out. Every message on it is a step an answer has to account for, and the two worth noticing are the ones that are easy to omit. The peer tells the home it downgraded rather than invalidated — which is section 6, and which an invalidate-only answer replaces with silence. And the requester acknowledges at the end, after the data has already arrived, which is section 12: the transaction is not over when the data lands.

The fifth case is the one this section is named for. The parts named and nothing sequenced — every noun correct, zero steps in order — reports eleven missing steps and the parts still correctly named, which is exactly what a confident recital looks like from the outside.

The third case is the boundary worth stating. An answer claiming more steps than the transaction has is clamped to the eleven that exist and reported as a complete walk, not as extra credit. Inventing steps is not coverage, and the clamp says so.

The degenerate case bounds it at the other end: a transaction with no steps is fully walked and is not a walk, which the model reports as a hundred percent and declines to call complete.

Two of the eleven are worth dwelling on because they are the ones that separate a good answer from a complete one, and both are invisible on a diagram.

The home's serialisation is a step, not a property. It is easy to say "the home orders them" as though ordering were something the address space has. It is an action taken by a structure with finite capacity: the home takes this request, marks the line busy, and everything else for that address waits. That is where section 11's retries come from, and an answer that treats serialisation as ambient rather than as a step has no place to put them.

The granted state is not always the requested state. A requester asking for exclusive ownership may be granted shared, because another agent's copy could not be invalidated in time or because the home's policy chose otherwise — and the requester then has to notice and retry its write. An answer that assumes the requester gets what it asked for has removed a whole class of behaviour, and it is the class that produces the hardest livelock bugs.

The clamp on detail is worth a line because it makes a point the section is otherwise silent about. Detail beyond what the question wants is not extra credit — the model clamps it and reports what was asked for. An answer that describes the flit encoding of the snoop message in a question about the transaction sequence has not answered more of the question; it has answered a different one.

6. RTL 2 — A Snoop Does Not Always Invalidate

The weak definition, taken seriously enough to measure.

A snoop is a question with several possible answers, and which one applies depends on two things the short answer never mentions: what the requester intends to do with the line, and what state the snooped agent is holding it in.

A requester that intends to write needs exclusive ownership, so every other copy has to go — that is an invalidate, and it is the case the short answer describes. A requester that intends to read needs only a shared copy, so an agent holding the line modified downgrades to shared and keeps it, and an agent already holding it shared does nothing at all.

And in both of the read cases something happens that the invalidate-only answer has no place for: a modified line's data has to move, because the copy in memory is stale and somebody has to supply the current one.

// RTL 2 - a snoop does not always invalidate. What a snoop does to a line
// depends on what the requester wants and what state the line is in, and an
// answer with one outcome in it has one third of the machine - and never
// mentions the writeback, which is the part that costs time.
module snoop_effect #(parameter int A_SNOOP_INVALIDATES = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] lines_snooped, want_write, held_modified, held_shared,
  output logic [15:0] invalidated, downgraded, data_moved, invalidate_pct,
  output logic        invalidate_only,
  output logic [7:0]  n_evals, n_downgrades,
  output logic        effect_err
);
  logic [31:0] i_q;
  logic [15:0] true_inval, true_down, true_moved, held_ok, shared_ok;
  logic        truly_downgraded;
  // A device cannot hold more lines modified or shared than were snooped.
  assign held_ok = (held_modified > lines_snooped) ? lines_snooped : held_modified;
  assign shared_ok = (held_shared > lines_snooped) ? lines_snooped : held_shared;
  // A write request invalidates whatever is held. A read request downgrades a
  // modified line to shared and leaves a shared line alone.
  assign true_inval = (want_write != 16'd0) ? (held_ok + shared_ok) : 16'd0;
  assign true_down  = (want_write != 16'd0) ? 16'd0 : held_ok;
  // Data moves whenever a modified line is snooped, whatever the requester
  // wanted; a clean line needs no writeback. An invalidate-only answer never
  // mentions it.
  assign true_moved = held_ok;
  assign invalidated = (A_SNOOP_INVALIDATES != 0) ? (held_ok + shared_ok) : true_inval;
  assign downgraded  = (A_SNOOP_INVALIDATES != 0) ? 16'd0 : true_down;
  assign data_moved  = (A_SNOOP_INVALIDATES != 0) ? 16'd0 : true_moved;
  assign i_q = (lines_snooped == 16'd0) ? 32'd0
             : (({16'd0, invalidated} * 32'd100) / {16'd0, lines_snooped});
  assign invalidate_pct = (i_q > 32'd100) ? 16'd100 : i_q[15:0];
  // The claim the answer makes: every snoop ends in an invalidate.
  assign invalidate_only = (downgraded == 16'd0) && (lines_snooped != 16'd0);
  // No lines_snooped guard: held_ok is a minimum against lines_snooped, so a
  // snoop that did not happen already downgrades nothing.
  assign truly_downgraded = (true_down != 16'd0);
  assign effect_err = evaluate && truly_downgraded && invalidate_only;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_downgrades <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_downgraded) n_downgrades <= n_downgrades + 8'd1;
    end
  end
endmodule

A hundred lines snooped for a read, thirty of them held modified and fifty shared, gives zero invalidations, thirty downgrades and thirty lines of data moving — while the invalidate-only view invalidates eighty and moves nothing.

Figure 2 — two answers to the same snoop, and the upper one is not merely imprecise. It throws away eighty lines that should have survived, which on a real device is eighty cache misses that did not have to happen, and it moves no data at all, which on a line held modified means the reader gets a stale copy. Stated as a design rather than as an answer, the recited version is a coherency bug.

The first case is the one the short answer gets right, and it is why the short answer survives. A write snoop does invalidate everything held, both views report eighty, and the only disagreement is the thirty lines of writeback that the invalidate-only view never mentions. Most people's mental image of a snoop is formed on write traffic, where the rule holds.

The fifth case is the quietest and the most instructive. A read snoop of lines held only shared does nothing whatsoever — no invalidation, no downgrade, no data movement — and both views report the same verdict from completely different machines: the measured build says "this is not an invalidate" and the invalidate-only build says "it is, and here are fifty of them". They agree on the headline and disagree on every number under it, which is the shape of an answer that cannot be caught by asking for the conclusion.

The third case is where the recited version becomes arithmetically impossible. A device claiming to hold more lines than were snooped is clamped to the hundred that exist, while the invalidate-only view invalidates two hundred — more lines than the snoop touched — and its percentage saturates at a hundred rather than reporting the two hundred percent that would have given it away.

It is worth naming the states explicitly, because "modified" and "shared" are doing specific work above. A line held modified is dirty and exclusive: this agent has the only copy and it differs from memory, so the data must move on any snoop — that is the writeback, and it is why the third row of the model exists. A line held shared is clean: memory has the same bytes, several agents may hold it, and a read snoop needs to do nothing at all because nothing about the line is about to change. A line held exclusive is clean and sole — the agent may write it without asking anybody, which is the state that makes a subsequent write cheap, and a snoop finds nothing to write back.

That gives the actual rule, which is two questions rather than one. Does the requester's intent require this agent to stop holding the line? Write: yes. Read: no. Does this agent hold data that memory does not have? Modified: yes. Shared or exclusive: no. The first question decides invalidate versus downgrade; the second decides whether data moves. The recited answer collapses both into one, which is why it gets write traffic right — the two questions happen to have the same answer there — and why it gets everything else wrong.

The sixth case is the empty one and it is worth keeping. A write snoop of a device holding nothing is an invalidate over an empty set: correct, free, and a real event, since the home may snoop on the strength of a directory entry that is conservative rather than exact. That is what a snoop filter's false positive looks like from the device's side, and a model that refused to call it an invalidate would be wrong about a case that happens constantly.

7. RTL 3 — Bias Decides What An Access Costs

The third thing a senior answer has to account for, and the one that is specific to CXL rather than to coherency in general.

A Type 2 device has memory that both sides can reach. Bias is which side is authoritative for a given page right now. In device bias the device reads its own memory directly, at local latency, with the host guaranteed not to have a cached copy. In host bias the same access is resolved through the host's coherency — the device asks, the host checks, and the data comes back having travelled the long way.

The two costs are not close to each other, and an answer that treats bias as a configuration detail has left out the largest term in the device's access cost.

// RTL 3 - bias is the part of the answer that decides what an access costs.
// A device reading its own memory in device bias goes straight to it; in host
// bias the same read is resolved through the host, and the difference is the
// whole reason the mechanism exists.
module bias_state #(parameter int BIAS_IS_A_DETAIL = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] dev_accesses, in_host_bias, host_resolve_ns, local_ns,
  output logic [15:0] host_bias_ok, resolved_local, total_us, penalty_us,
  output logic        no_penalty,
  output logic [7:0]  n_evals, n_penalised,
  output logic        bias_err
);
  logic [31:0] raw_host, raw_local, raw_total, raw_flat;
  logic [15:0] true_total, flat_total, true_penalty;
  logic        truly_penalised;
  // A device cannot have more accesses in host bias than it makes.
  assign host_bias_ok = (in_host_bias > dev_accesses) ? dev_accesses : in_host_bias;
  assign resolved_local = dev_accesses - host_bias_ok;
  assign raw_host  = {16'd0, host_bias_ok} * {16'd0, host_resolve_ns};
  assign raw_local = {16'd0, resolved_local} * {16'd0, local_ns};
  assign raw_total = (raw_host + raw_local) / 32'd1000;
  assign true_total = (raw_total > 32'd9999) ? 16'd9999 : raw_total[15:0];
  // The bias-is-a-detail view charges every access the local price.
  assign raw_flat = ({16'd0, dev_accesses} * {16'd0, local_ns}) / 32'd1000;
  assign flat_total = (raw_flat > 32'd9999) ? 16'd9999 : raw_flat[15:0];
  assign true_penalty = (true_total > flat_total) ? (true_total - flat_total) : 16'd0;
  assign total_us   = (BIAS_IS_A_DETAIL != 0) ? flat_total : true_total;
  assign penalty_us = (BIAS_IS_A_DETAIL != 0) ? 16'd0 : true_penalty;
  assign no_penalty = (penalty_us == 16'd0) && (dev_accesses != 16'd0);
  // No dev_accesses guard: every term of the penalty is scaled by the access
  // count, so a device that makes none is already unpenalised.
  assign truly_penalised = (true_penalty != 16'd0);
  assign bias_err = evaluate && truly_penalised && no_penalty;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_penalised <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_penalised) n_penalised <= n_penalised + 8'd1;
    end
  end
endmodule

A thousand device accesses with four hundred in host bias, at six hundred nanoseconds against a hundred and fifty local, is three hundred and thirty microseconds where a fully-local device would take a hundred and fifty — a hundred and eighty of pure bias penalty.

The fifth case draws the boundary of the argument honestly. When host resolution costs the same as a local access, bias is free — the penalty is the difference between the two paths and nothing else, and the model reports zero. That is not a realistic configuration, but stating it is what makes the section a measurement rather than a slogan: bias costs what the two paths differ by, and on a system where they do not differ it costs nothing.

The last case is the one that inverts the intuition. A local access that costs nothing — a device with its memory effectively on-die — makes the entire two hundred and forty microseconds attributable to host bias, and the flat view reports a device doing no work at all. The cheaper the local path, the larger the fraction of the cost that bias accounts for, which is the opposite of the way "it is just a detail" is usually argued.

The third case is worth knowing because it is the failure mode bias exists to prevent. Every access in host bias — a device that has not transitioned any of its pages back — takes six hundred microseconds where a hundred and fifty would do, and it is a completely functional system. Nothing is broken. The device is simply paying the host's coherency cost for memory it owns, which is what a missing or mis-triggered bias transition produces, and it is invisible except as a performance number.

The degenerate case bounds the model: a device that makes no accesses has no bias story, and the model reports nothing rather than inventing a penalty.

What actually moves a page between the two states is worth knowing, because it is the part a design can control. Device bias is entered by the device asking for it and the host agreeing, which requires the host to flush anything it holds cached from that page — so the transition has a cost paid once, and it is not small. Host bias is entered when the host needs coherent access, typically because software on the host is about to read results the device produced.

That gives the shape of a well-behaved workload: transition once at the start of a phase, run the whole phase in the right bias, transition back. And it gives the shape of a badly-behaved one: fine-grained interleaving of host and device access to the same page, which either flips bias repeatedly — paying the transition cost each time — or settles in host bias and pays section 7's penalty on every device access forever. The second is more common, because it is what happens when nobody transitions at all.

The third case in the stimulus is exactly that steady state, and it deserves its own sentence. Every access in host bias is a completely functional system running at a quarter of its intended speed, with no error anywhere, and the only way to see it is to know what the local path would have cost.

8. RTL 4 — The Host Tracks What The Device Holds

The fourth constraint, and the one that makes coherent device attach cost something on the host side.

Every line a device caches occupies an entry in the host's directory. The host has to know who holds what in order to snoop correctly, and the structure that records it is finite — sized when the host silicon was designed, against an expectation of how many agents would attach and how much they would cache.

A directory that cannot hold what the devices want does not fail. It evicts, and evicting a directory entry means back-invalidating the line in the device that was using it, whether or not the device was finished with it.

// RTL 4 - the host tracks what the device holds, in a structure with a size.
// A coherent device attach is not free on the host side: every line the
// device caches occupies a directory entry, and a directory that fills has to
// evict somebody - which means back-invalidating a line the device is using.
module directory_budget #(parameter int TRACKING_IS_FREE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] dir_entries, lines_wanted, devices, entries_per_dev,
  output logic [15:0] lines_held, dir_short, evict_forced, held_pct,
  output logic        fits_directory,
  output logic [7:0]  n_evals, n_evicting,
  output logic        dir_err
);
  logic [31:0] raw_demand, h_q;
  logic [15:0] demand, true_short;
  logic        truly_evicting;
  // Every device wants entries, and the demand is what they want together.
  assign raw_demand = {16'd0, devices} * {16'd0, entries_per_dev};
  assign demand = (raw_demand > 32'd9999) ? 16'd9999
                : ((lines_wanted > raw_demand[15:0]) ? lines_wanted : raw_demand[15:0]);
  assign lines_held = (demand > dir_entries) ? dir_entries : demand;
  assign true_short = (demand > dir_entries) ? (demand - dir_entries) : 16'd0;
  assign dir_short = (TRACKING_IS_FREE != 0) ? 16'd0 : true_short;
  // A directory that cannot hold what is wanted forces a back-invalidate for
  // every line it cannot track.
  assign evict_forced = dir_short;
  assign h_q = (demand == 16'd0) ? 32'd100
             : (({16'd0, lines_held} * 32'd100) / {16'd0, demand});
  assign held_pct = h_q[15:0];
  assign fits_directory = (dir_short == 16'd0) && (demand != 16'd0);
  assign truly_evicting = (true_short != 16'd0) && (dir_entries != 16'd0);
  assign dir_err = evaluate && truly_evicting && fits_directory;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_evicting <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_evicting) n_evicting <= n_evicting + 8'd1;
    end
  end
endmodule

Four thousand lines wanted against a thousand-and-twenty-four-entry directory is two thousand nine hundred and seventy-six forced back-invalidations and a quarter of what the devices want tracked — and the tracking-is-free view reports a directory that fits.

The last case is the one worth checking on a multi-device system. The devices' own aggregate demand exceeding the line count — four devices wanting five hundred entries each while only five hundred lines were nominally requested — means the binding number is the sum across devices, not the figure any one of them quoted. A directory sized from a single device's working set is undersized the moment a second device attaches.

The fifth case is the degenerate one and the model declines to draw a conclusion from it. A host with no directory at all is short of every line, and that is not an undersized directory — it is a host that cannot support coherent device attach, which is a different conversation and a different answer.

The boundary is driven deliberately. A directory sized exactly to the demand fits, because a structure sized precisely to its load has been sized correctly.

What back-invalidation actually does to the device is worth connecting to section 6, because the two interact in a way that surprises people. A back-invalidate is a snoop the device did not cause and cannot predict. It arrives because some other device wanted directory space, it takes a line the device was actively using, and from the device's point of view it is indistinguishable from a peer wanting the line. Directory pressure therefore shows up as an unexplained miss rate on a device whose own working set has not changed — which is one of the harder symptoms in 26.7's territory, because nothing about the affected device is wrong.

The arithmetic has a consequence for system design that the model states plainly and that is easy to miss when devices are added one at a time. Directory demand is the sum across every attached agent, and it is the sum of their working sets rather than of their cache sizes — but a device with a large cache will use it, so the two converge. A host sized for one device is undersized for four, and the failure is graceful in the worst way: no error, no report, just a device whose hit rate has quietly fallen because another device attached.

This is also why a snoop filter is not the same thing as a directory, and the distinction matters when reading a specification. A directory is exact: it names who holds what. A filter is conservative: it may name an agent that does not hold the line, producing a harmless extra snoop, but it may never omit one that does. The filter is smaller, which is the point, and the price is the unnecessary snoops — which land on the device as section 6's sixth case, an invalidate over an empty set.

9. RTL 5 — Not Everything Is Ordered

The fifth thing, and the one where over-answering costs marks.

An answer that orders every step of the transaction is safe-sounding and wrong. What has to be ordered is what the coherence result depends on: the home's serialisation of competing requests for the same line, the directory update against the snoop responses, the acknowledgement against the completion. Everything else is free to proceed concurrently and does — snoops to several agents go out together, data and the home's response travel independently, and different addresses do not order against each other at all.

The difference is not academic. It is the difference between a protocol that scales with the number of agents and one that does not.

// RTL 5 - not everything in a coherent transaction is ordered, and an answer
// that orders all of it has described a slower machine than the one that
// exists. What must be ordered is what the coherence result depends on;
// everything else is free to complete out of order and usually does.
module ordering_rules #(parameter int EVERYTHING_IS_ORDERED = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] ops, must_order, op_ns, lanes,
  output logic [15:0] ordered_ops, parallel_ops, total_ns, serial_pct,
  output logic [15:0] serial_ns, parallel_ns,
  output logic        all_ordered,
  output logic [7:0]  n_evals, n_parallel,
  output logic        order_err
);
  logic [31:0] raw_par, raw_total, s_q, raw_ser;
  logic [15:0] must_ok, true_parallel, lanes_ok, par_ns, ser_ns;
  logic        truly_parallel;
  // An answer cannot require ordering on more operations than there are.
  assign must_ok = (must_order > ops) ? ops : must_order;
  assign true_parallel = ops - must_ok;
  assign ordered_ops  = (EVERYTHING_IS_ORDERED != 0) ? ops : must_ok;
  assign parallel_ops = (EVERYTHING_IS_ORDERED != 0) ? 16'd0 : true_parallel;
  // Ordered operations serialise; the rest share the available lanes.
  assign lanes_ok = (lanes == 16'd0) ? 16'd1 : lanes;
  assign raw_ser = {16'd0, ordered_ops} * {16'd0, op_ns};
  assign ser_ns = (raw_ser > 32'd9999) ? 16'd9999 : raw_ser[15:0];
  assign raw_par = ({16'd0, parallel_ops} * {16'd0, op_ns}) / {16'd0, lanes_ok};
  assign par_ns = (raw_par > 32'd9999) ? 16'd9999 : raw_par[15:0];
  assign serial_ns = ser_ns;
  assign parallel_ns = par_ns;
  assign raw_total = {16'd0, ser_ns} + {16'd0, par_ns};
  assign total_ns = (raw_total > 32'd9999) ? 16'd9999 : raw_total[15:0];
  assign s_q = (ops == 16'd0) ? 32'd0
             : (({16'd0, ordered_ops} * 32'd100) / {16'd0, ops});
  assign serial_pct = s_q[15:0];
  assign all_ordered = (parallel_ops == 16'd0) && (ops != 16'd0);
  // No ops guard: true_parallel is ops less a minimum against ops, so a
  // transaction with no operations already has none in parallel.
  assign truly_parallel = (true_parallel != 16'd0);
  assign order_err = evaluate && truly_parallel && all_ordered;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_parallel <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_parallel) n_parallel <= n_parallel + 8'd1;
    end
  end
endmodule

Twenty operations with six that must be ordered, at fifty nanoseconds each across four lanes, is four hundred and seventy-five nanoseconds — three hundred serial and a hundred and seventy-five in parallel — where ordering all twenty gives a thousand.

The fifth case is the one that makes the distinction operational rather than theoretical. Parallel operations with one lane to run on take exactly as long as ordering them would, and the everything-is-ordered view cannot tell the two apart. They are not the same thing: one is a protocol constraint and the other is an implementation limit, and they have different fixes — the first cannot be removed and the second is a matter of provisioning. An answer that conflates them has described the symptom correctly and the cause not at all.

The last case is the sharpest boundary. One operation free of ordering out of twenty is enough to make "everything is ordered" false, and the model says so at ninety-five percent serialised. The claim is not approximately true at ninety-five percent; it is false, and the one operation is where the concurrency lives.

The sixth case is the other end. Nothing that has to be ordered at all — twenty independent operations across four lanes, two hundred and fifty nanoseconds — is what different addresses look like, and it is the case that makes the protocol scale.

It is worth being specific about what the six ordered operations actually are, because "what the coherence result depends on" is a definition rather than a list. The home's decision about which of two competing requests for the same line goes first — that is the serialisation, and everything else for that address is behind it. The directory update against the snoop responses, because a directory that records the new owner before the old one has confirmed it released the line describes a state that does not exist. And the completion against the acknowledgements, because a transaction declared complete while an agent still believes it holds the line is a coherency violation rather than a performance problem.

Everything else is free. Snoops to several agents go out together and their responses arrive in any order. The data and the home's response travel independently and either may arrive first. And requests for different addresses do not order against each other at all, which is the property that lets the protocol scale with agents — if they did, the home would be a global serialisation point rather than a per-address one, and the whole design would be a bus.

10. RTL 6 — The End-To-End Number Is A Sum

The sixth thing, and the one that decides whether the answer is useful to anybody afterwards.

A walk that finishes with a latency number has answered the question. A walk that finishes with the terms of that number has answered the next one too — which is where the time goes, and therefore what could be done about it.

The terms are not interchangeable. The link crossing is a function of the physical topology, which is 27.7's subject. The agent's own pipeline is a design parameter of the device. The directory lookup is a property of the host. The snoop is a function of how many agents hold the line and how fast the slowest of them responds. Four different owners, four different fixes, and a single total identifies none of them.

// RTL 6 - the end-to-end number is a sum, and the answer has to name the
// terms. A senior walk that ends at "and then the data comes back" has not
// said where the time went, which is the only part a performance engineer
// can act on.
module latency_chain #(parameter int IT_IS_ONE_NUMBER = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] link_ns, agent_ns, dir_ns, snoop_ns,
  output logic [15:0] terms_named, total_ns, largest_ns, largest_pct,
  output logic        one_number,
  output logic [7:0]  n_evals, n_unnamed,
  output logic        chain_err
);
  logic [31:0] raw_total, l_q;
  logic [15:0] true_total, true_largest, ab, cd;
  logic        truly_unnamed;
  assign raw_total = {16'd0, link_ns} + {16'd0, agent_ns}
                   + {16'd0, dir_ns} + {16'd0, snoop_ns};
  assign true_total = (raw_total > 32'd9999) ? 16'd9999 : raw_total[15:0];
  // The largest term is where the time actually is, and it is the term an
  // answer has to be able to point at.
  assign ab = (link_ns > agent_ns) ? link_ns : agent_ns;
  assign cd = (dir_ns > snoop_ns) ? dir_ns : snoop_ns;
  assign true_largest = (ab > cd) ? ab : cd;
  // The one-number view reports the total and nothing about its shape.
  assign terms_named = (IT_IS_ONE_NUMBER != 0) ? 16'd1 : 16'd4;
  assign largest_ns  = (IT_IS_ONE_NUMBER != 0) ? 16'd0 : true_largest;
  assign total_ns = true_total;
  assign l_q = (true_total == 16'd0) ? 32'd0
             : (({16'd0, true_largest} * 32'd100) / {16'd0, true_total});
  assign largest_pct = l_q[15:0];
  // The claim the one-number answer makes: there is no term worth naming.
  assign one_number = (largest_ns == 16'd0) && (true_total != 16'd0);
  // No true_total guard: the largest term is a maximum over the four terms
  // that make the total, so a chain with no time already has no term to name.
  assign truly_unnamed = (true_largest != 16'd0);
  assign chain_err = evaluate && truly_unnamed && one_number;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_unnamed <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_unnamed) n_unnamed <= n_unnamed + 8'd1;
    end
  end
endmodule

A three-hundred-nanosecond chain of eighty, forty, thirty and a hundred and fifty is half the time in the snoop — and the one-number view reports three hundred nanoseconds and nothing to act on.

The sixth case is the one that keeps the technique honest. Four equal terms still produce a largest term — seventy-five nanoseconds, a quarter of the chain — and naming it tells you almost nothing, because there is no bottleneck to name. The measurement is only actionable when the distribution is uneven, and reporting a dominant term that dominates nothing is a way of sounding precise without being useful.

The last case is the opposite extreme and it is the one where the method pays best. The whole chain in one term means naming the term is naming the problem, and the one-number view reports the same three hundred nanoseconds it reports for the evenly-spread case. Two chains with completely different remedies, one number, no way to tell them apart.

The degenerate case bounds it: a chain with no latency in it has no dominant term, and the model reports four terms, no total, and declines to call any of them the answer.

The four terms have four different remedies and that is the practical point of splitting them. The link crossing is topology: fewer hops, a shorter reach, a different placement — 27.7's section 6, and usually the most expensive to change. The agent's own pipeline is the device's design, fixed at silicon and negotiable only by choosing a different part. The directory lookup is the host's, and it is the one term that a snoop filter can improve. The snoop is bounded by the slowest agent that must respond, which means it is a function of how many agents hold the line and of the worst of them — so it gets worse as the system grows, and it is the term most likely to dominate on a large fabric.

That last property is why the snoop is the term worth measuring first on any system with more than two caching agents. It is the only one of the four that grows with the number of participants, and an answer that quotes a two-agent latency for a system that will run eight has quoted the wrong number.

11. RTL 7 — Two Agents Can Want The Same Line

The seventh thing, and the one that turns a walk into a protocol.

A single request walked end to end is a story. Two requests for the same line at the same time is the reason the protocol has a home node at all. The home serialises them: one is processed, the other is either queued behind it or told to retry, and the retry is work that the happy-path walk never counts.

An answer that walks one transaction and stops has described the easy case. The question of what the second requester experiences is where the design is.

// RTL 7 - two agents can want the same line at the same time, and what the
// protocol does about it is the part of the walk that separates an answer
// from a recital. Somebody wins, somebody retries, and the retry is work
// that the happy-path walk never counts.
module race_window #(parameter int ONE_AT_A_TIME = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] requests, contended, retry_ns, base_ns,
  output logic [15:0] winners, retried, retry_cost_ns, retry_pct,
  output logic        uncontended,
  output logic [7:0]  n_evals, n_raced,
  output logic        race_err
);
  logic [31:0] raw_cost, r_q;
  logic [15:0] cont_ok, true_retried, true_cost;
  logic        truly_raced;
  // A request set cannot contain more contended requests than requests.
  assign cont_ok = (contended > requests) ? requests : contended;
  // One of the contenders wins outright; the others are told to retry.
  assign winners = (cont_ok == 16'd0) ? 16'd0 : 16'd1;
  assign true_retried = (cont_ok > 16'd0) ? (cont_ok - 16'd1) : 16'd0;
  assign retried = (ONE_AT_A_TIME != 0) ? 16'd0 : true_retried;
  assign raw_cost = {16'd0, true_retried} * {16'd0, retry_ns};
  assign true_cost = (raw_cost > 32'd9999) ? 16'd9999 : raw_cost[15:0];
  assign retry_cost_ns = (ONE_AT_A_TIME != 0) ? 16'd0 : true_cost;
  assign r_q = (base_ns == 16'd0) ? 32'd0
             : (({16'd0, retry_cost_ns} * 32'd100) / {16'd0, base_ns});
  assign retry_pct = (r_q > 32'd999) ? 16'd999 : r_q[15:0];
  assign uncontended = (retried == 16'd0) && (requests != 16'd0);
  // No requests guard: cont_ok is a minimum against requests, so a
  // transaction with no requests already has nobody retrying.
  assign truly_raced = (true_retried != 16'd0);
  assign race_err = evaluate && truly_raced && uncontended;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_raced <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_raced) n_raced <= n_raced + 8'd1;
    end
  end
endmodule

Four requests contending for one line is one winner and three retries — seven hundred and fifty nanoseconds, nearly twice the base latency of the transaction itself — and the one-at-a-time view reports a clean run.

The fifth case is the boundary and it is worth being precise about. One contender is not contention. A single requester for a line wins uncontested, retries nothing, and costs nothing extra — the race begins at the second requester, and a model that alarmed at the first would be alarming on every transaction in the system.

The last case separates the two things a retry costs. A retry that costs no time is still a retry — the model reports three of them at zero nanoseconds — because the time is only half of it. The other half is that the requester's transaction was re-ordered relative to everything else it was doing, which is an ordering event whether or not it was slow. An answer that prices races purely in nanoseconds has missed the half that causes the hard bugs.

The sixth case is the honest gap. A race with no base latency to measure it against reports the retry cost and declines to express it as a proportion, because a percentage of nothing is not a number.

Contention on a coherent line is also not evenly distributed in practice, which makes the model's uniform treatment the optimistic case. Real contention concentrates: a lock, a queue head, a shared counter, a flag that several agents poll. Those lines see the same few agents racing continuously, and the cost is not a one-off retry but a steady-state penalty on every access to that address — while every other line in the system is uncontended and cheap.

That is why an average latency across all lines hides it completely. A workload where one line in ten thousand is savagely contended and the rest are free shows an unremarkable average and a tail that is entirely made of that one line. Section 10's split says where the time goes within a transaction; this is the version of the same argument across transactions, and both are cases where the mean is the wrong summary.

The third case is the one that shows how bad it can get. More contenders than requests, clamped to sixty-three retries, saturates both the retry cost and its proportion of the base latency — a line that every agent in the system wants at once, which is what a poorly-chosen synchronisation variable looks like.

12. RTL 8 — The Data Arriving Is Not The Transaction Finishing

The eighth thing, and the step most answers are missing.

The requester can have the data and not yet own it. The data arrives when whichever agent had it sends it, which can be early. The state the requester was granted is settled when every acknowledgement the grant depends on has been collected — the snooped agents confirming what they did, the home recording it, the requester confirming receipt.

Between those two moments the requester holds bytes it is not yet permitted to act on in the way it intends. Every protocol handles this, and the handling is the part of the walk that gets left out, because the interesting-looking part — the data movement — has already happened.

// RTL 8 - the data arriving is not the transaction finishing. A requester can
// have the data in hand while the coherence state it was granted is still
// being resolved elsewhere, and an answer that ends at "the data comes back"
// has stopped one step early.
module completion_semantics #(parameter int DATA_IS_DONE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] txns, data_ns, resolve_ns, acks_owed,
  output logic [15:0] data_at_ns, resolved_at_ns, completion_gap_ns, gap_pct,
  output logic        done_on_data,
  output logic [7:0]  n_evals, n_gapped,
  output logic        completion_err
);
  logic [31:0] raw_res, g_q;
  logic [15:0] true_resolved, true_gap;
  logic        truly_gapped;
  assign data_at_ns = data_ns;
  // Resolution waits for every acknowledgement the grant depends on.
  assign raw_res = {16'd0, data_ns} + ({16'd0, acks_owed} * {16'd0, resolve_ns});
  assign true_resolved = (raw_res > 32'd9999) ? 16'd9999 : raw_res[15:0];
  assign true_gap = true_resolved - data_at_ns;
  assign resolved_at_ns   = (DATA_IS_DONE != 0) ? data_at_ns : true_resolved;
  assign completion_gap_ns = (DATA_IS_DONE != 0) ? 16'd0 : true_gap;
  assign g_q = (data_at_ns == 16'd0) ? 32'd0
             : (({16'd0, completion_gap_ns} * 32'd100) / {16'd0, data_at_ns});
  assign gap_pct = (g_q > 32'd999) ? 16'd999 : g_q[15:0];
  assign done_on_data = (completion_gap_ns == 16'd0) && (txns != 16'd0);
  assign truly_gapped = (true_gap != 16'd0) && (txns != 16'd0);
  assign completion_err = evaluate && truly_gapped && done_on_data;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_gapped <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_gapped) n_gapped <= n_gapped + 8'd1;
    end
  end
endmodule

Data in hand at two hundred nanoseconds with three acknowledgements owed at sixty each resolves at three hundred and eighty — a hundred and eighty nanoseconds of gap, ninety percent of the data latency again — and the data-is-done view reports a two-hundred-nanosecond transaction.

The last case is the one that makes the section matter for performance rather than only for correctness. A resolution far longer than the transfer — data at a hundred nanoseconds, resolution at four thousand one hundred — means the data movement was the fast part and everything the requester experiences as latency is the coherence settling. Optimising the data path on that transaction would be optimising four percent of it.

The second and sixth cases are the two the data-is-done view gets right, and they are worth stating so the section is not read as a blanket rule. A transaction owing no acknowledgements is done when the data arrives, and so is one whose acknowledgements resolve instantly. Those are real configurations — an uncontended line that nobody else holds is exactly the first — which is why the shortcut survives: it is correct on the easy case, and the easy case is most of them.

The fifth case is worth sitting with. Acknowledgements owed on a transfer that took no time at all gives a transaction whose entire duration is coherence resolution, and the data-is-done view reports it as instantaneous. That is not a contrived input: a line already in the requester's own cache hierarchy, needing only a state upgrade, transfers no data and still has to be resolved.

The degenerate case bounds the model. A gap on a transaction set with no transactions is a defined quantity nobody experiences, and the model reports the gap while declining to call it a completion fault.

The reason this step exists at all is worth stating, because "wait for acknowledgements" sounds like bookkeeping. It is what makes the coherence result observable in the right order. If the requester could act on the data the instant it arrived, it could publish a result derived from that line before the previous owner had finished relinquishing it — and another agent reading that result could then read the old value of the line. The acknowledgement is what forbids that sequence.

Which is why the completion gap cannot be optimised away, only overlapped. A design can let the requester proceed speculatively, or pipeline the next transaction into the gap, or shorten it by reducing how many agents must acknowledge — but it cannot declare the transaction finished early, because the ordering guarantee is the product being sold. An answer that treats the gap as overhead has misunderstood which part is the feature.

The third case is worth reading for a reason that has nothing to do with the clamp. A resolution driven past the model's ceiling still reports a gap of nine hundred and ninety-nine against a data latency of nine thousand — eleven percent — which is the shape of a system where the data transfer genuinely is the dominant cost. Those exist. The section's argument is not that resolution always dominates; it is that it is never zero and is usually unmeasured.

13. RTL 9 — What Happens When A Step Does Not

The ninth thing, and the one that turns a walk into a design.

Every step in the sequence has a way of not completing. The snoop can go to an agent that does not respond. The data can be lost. The acknowledgement can be dropped. The directory update can race with a competing transaction. For each of them the protocol has an answer, and the answer is always one of three things: a timeout, a retry, or an error escalation.

What it is never is silence. A step with no handling does not fail fast — it waits for whatever timeout sits above it, and if there is none it waits for the message that is never coming, which is a hang rather than an error.

// RTL 9 - a walk is only a design if it says what happens when a step does
// not. Every step in a coherent transaction has a way of not completing, and
// the answer to "what then" is a timeout, a retry, or an error - never
// silence.
module failure_mode #(parameter int IT_ALWAYS_COMPLETES = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] steps, steps_with_handling, timeout_ns, step_ns,
  output logic [15:0] handled_ok, unhandled, hang_ns, handled_pct,
  output logic        fully_handled,
  output logic [7:0]  n_evals, n_unhandled,
  output logic        failure_err
);
  logic [31:0] raw_hang, h_q;
  logic [15:0] true_unhandled, true_hang;
  logic        truly_unhandled;
  // A walk cannot handle more steps than it has.
  assign handled_ok = (steps_with_handling > steps) ? steps : steps_with_handling;
  assign true_unhandled = steps - handled_ok;
  assign unhandled = (IT_ALWAYS_COMPLETES != 0) ? 16'd0 : true_unhandled;
  // An unhandled step does not fail fast; it waits for whatever timeout sits
  // above it, and if there is none it waits for the step that never comes.
  assign raw_hang = (true_unhandled == 16'd0) ? 32'd0
                  : ((timeout_ns == 16'd0) ? 32'd9999 : {16'd0, timeout_ns});
  assign true_hang = (raw_hang > 32'd9999) ? 16'd9999 : raw_hang[15:0];
  assign hang_ns = (IT_ALWAYS_COMPLETES != 0) ? 16'd0 : true_hang;
  assign h_q = (steps == 16'd0) ? 32'd100
             : (({16'd0, handled_ok} * 32'd100) / {16'd0, steps});
  assign handled_pct = h_q[15:0];
  assign fully_handled = (unhandled == 16'd0) && (steps != 16'd0);
  // No steps guard: true_unhandled is steps less a minimum against steps, so
  // a walk with no steps already has none unhandled.
  assign truly_unhandled = (true_unhandled != 16'd0);
  assign failure_err = evaluate && truly_unhandled && fully_handled;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_unhandled <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_unhandled) n_unhandled <= n_unhandled + 8'd1;
    end
  end
endmodule

Eleven steps with seven handled leaves four with no failure path and a five-microsecond wait when one of them does not complete — sixty-three percent of the walk designed, and the always-completes view reports a walk.

The fifth case is the difference between a slow system and a dead one. Unhandled steps with no timeout above them produce an unbounded wait rather than a five-microsecond one, and the distinction is the whole of the design: a timeout turns a hang into an error, which turns an unrecoverable system into a recoverable one. It does not make the underlying failure less likely; it makes it survivable.

The last case is the sharpest. One unhandled step in a walk of two hundred is ninety-nine percent handled and is still a hang waiting to happen, because the failure will find the one step rather than the hundred and ninety-nine. Coverage of failure paths is not a percentage that can be traded off against schedule in the way coverage of functionality sometimes can.

The sixth case is what a happy-path answer looks like measured. Nothing handled at all — every step correct, no step with a failure story — is a walk that describes what happens when everything works, which is a description rather than a design.

14. RTL 10 — A Senior Coherency Answer Assembled

Nine sections of inputs. This one puts them in one place and makes the confident answer visible as what it is: one bit of six.

// RTL 10 - a senior coherency answer assembled. Nine sections of inputs, one
// summary. "The snoop invalidates it" is bit 0: true of one case, and one
// sixth of what makes a walk an answer.
module coherency_signoff #(parameter int INVALIDATION_IS_THE_ANSWER = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic        snoop_named, walk_complete, states_named,
  input  logic        bias_accounted, ordering_stated, completion_defined,
  output logic [5:0]  fail_mask,
  output logic [15:0] conditions_met, sound_pct,
  output logic        sound,
  output logic [7:0]  n_evals, n_sound, n_claimed,
  output logic        signoff_err
);
  logic [31:0] s_q;
  logic        truly_sound, claimed;
  assign fail_mask[0] = ~snoop_named;
  assign fail_mask[1] = ~walk_complete;
  assign fail_mask[2] = ~states_named;
  assign fail_mask[3] = ~bias_accounted;
  assign fail_mask[4] = ~ordering_stated;
  assign fail_mask[5] = ~completion_defined;
  assign conditions_met = {15'd0, snoop_named} + {15'd0, walk_complete}
                        + {15'd0, states_named} + {15'd0, bias_accounted}
                        + {15'd0, ordering_stated} + {15'd0, completion_defined};
  assign s_q = ({16'd0, conditions_met} * 32'd100) / 32'd6;
  // No clamp: conditions_met sums six one-bit values, so the quotient cannot
  // exceed a hundred and a ceiling would be unreachable code.
  assign sound_pct = s_q[15:0];
  assign truly_sound = (fail_mask == 6'd0);
  // The invalidation view reads bit 0 and stops.
  assign claimed = (INVALIDATION_IS_THE_ANSWER != 0) ? snoop_named : truly_sound;
  assign sound = claimed;
  assign signoff_err = evaluate && !truly_sound && claimed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_sound <= 8'd0; n_claimed <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_sound) n_sound <= n_sound + 8'd1;
      if (claimed)     n_claimed <= n_claimed + 8'd1;
    end
  end
endmodule

The stimulus walks all six bits one at a time. When the snoop has been named and any one of the other five fails, the assembled model reports that the answer is not sound and the invalidation view reports an answer.

Across the eight evaluations, the assembled model calls one answer sound and the invalidation view calls six of them an answer.

The bit order is by how far into the answer each condition appears. Bit 0 is the first sentence. Bit 1 is the body. Bit 2 is the detail that makes the body checkable. Bits 3 and 4 are the two things a senior answer adds that an intermediate one does not, and bit 5 is last because completion is the step at the end that almost everybody omits.

"The snoop invalidates it" is bit 0, and it is true of the write case. That is what makes it the right weak definition for this chapter: it is not a wrong statement, it is a correct statement about one of three outcomes, offered as though it were the mechanism. On write traffic it is right, and write traffic is where most people's mental model of coherency was formed — which is exactly why it survives until somebody asks about a read.

The five other bits fail independently, which is what makes the mask a summary rather than a score. A complete walk can have no states named in it. States can be named with bias entirely absent, which is the most common shape of a good non-CXL coherency answer given to a CXL question. Ordering can be stated by somebody who thinks everything is ordered. And completion can be left undefined by an answer that got every other part right — which is the single most common omission, because the data arriving feels like the end.

Figure 4 — the mask ordered by how far into the answer each condition appears, which is also the order in which an interviewer stops hearing them. The first two are audible in the opening minute. The middle two are what a senior answer adds over an intermediate one. The last is at the end of a sequence that has already delivered the data, which is why it is omitted by answers that get everything else right — the interesting-looking part has finished, and the transaction has not.

15. Quantitative Reasoning

Thirty-six percent of a transaction from four of eleven steps, with the parts all correctly named alongside.

Zero invalidations and thirty downgrades on a read snoop of a hundred lines — against eighty invalidations and no data movement from the recited answer.

A hundred and eighty microseconds of bias penalty on a thousand accesses, four hundred of them resolved through the host. Three hundred and thirty against a hundred and fifty.

Two thousand nine hundred and seventy-six forced back-invalidations from four thousand lines against a thousand-entry directory — each one a line taken from a device that was using it.

Four hundred and seventy-five nanoseconds against a thousand, from distinguishing the six operations that must be ordered from the fourteen that need not be.

Half the chain in the snoop — a hundred and fifty nanoseconds of three hundred, in a term with a different owner from the other three.

Three retries at seven hundred and fifty nanoseconds, a hundred and eighty-seven percent of the base latency, from four requests wanting one line.

A hundred and eighty nanoseconds between the data arriving and the transaction completing — ninety percent of the data latency again, on a transaction the short answer calls two hundred nanoseconds.

Four steps of eleven with no failure path, and an unbounded wait rather than a five-microsecond one when no timeout sits above them.

One answer of eight sound; the invalidation view counts six. The assembled model's summary, and the chapter's.

16. Assertions

The testbenches carry 593 checks across ten models.

Every output of every model is asserted as a value, in both builds. The output listing step reported nothing on the first run — the first chapter in six where it found no gap — and one afterwards, when publishing the serial and parallel split added an output that the recited build's assertions had not yet reached.

Both builds are asserted on every degenerate case. A transaction with no steps, a snoop that did not happen, a device that makes no accesses, a directory nobody uses, a host with no directory, a transaction with no operations, a chain with no latency, a request set with no requests, a transaction set with no transactions, a transfer that took no time, and a walk with no steps in it.

Every clamp that an input can reach is driven past its limit exactly once. More steps named than the transaction has, more lines held than were snooped, more accesses in host bias than the device makes, a device demand past the directory model's ceiling, a serial chain and a parallel set that each saturate on their own, a latency chain past its ceiling, more contenders than requests, a resolution past its ceiling, and a timeout too long to report.

Every error output is checked in both directions in every case. Section 5's second and third cases, section 6's first and fifth, section 7's fifth, section 9's second and sixth, section 11's second and fifth, section 12's second and sixth exist to assert the quiet half — a complete walk, an over-claim, a write snoop, a read of shared lines, bias that costs nothing, a genuinely serial transaction, an uncontended run, a single contender, a transaction owing nothing, and acknowledgements that resolve instantly. Each is a case where the recited answer is right, and a model that alarmed on them would be unusable.

17. Mutation Testing

128 mutations, 128 killed. Sixty-four against each testbench. The first run killed a hundred and twenty and left eight, and the eight are the chapter's result.

Seven of the eight survivors were one defect, and it is batch 027's finding in a new form. A guard written as (A != 0) && (S != 0) is dead when A is already zero whenever S is — and it is, whenever A is a difference against S or a minimum taken against it. Seven of the ten models had one: the walk's step count, the snoop's line count, the device's access count, the transaction's operation count, the chain's total, the request count and the step count again.

Last batch the same relationship appeared as a clamp whose numerator was a minimum against its denominator. This is the boolean form of the identical mistake, and it was found the same way — by mutations that survived rather than by reading. domcheck.py now decides both: it computes the set of signals forced to zero by a given signal being zero, closing over minimums, differences, products and the branches of a ternary, and reports any guard whose second conjunct is in that set. Run across every chapter in Modules 25 to 28 it finds two more, in 27.6 and 27.7. Neither was ever the target of a mutation — verified by re-reading both mutation lists — so no published kill count is affected.

The eighth survivor was worth more than its share. A "raise the ceiling" mutation is not a test of a clamp. Two of this chapter's clamp mutations raised 9999 to 99999 on quantities that reached a hundred thousand — so the clamp fired identically and the mutation was equivalent. The mutation that actually tests a clamp is removing it, and letting the value wrap. Four batches of drive every clamp past its limit had never distinguished the two.

The same eighth run also exposed a real assertion gap in two separate models: a percentage whose numerator was swapped from the clamped value to the raw claim survived, because the over-claim case asserted every output except the percentage. Both now assert it.

18. Verification Strategy

Write the steps down before answering. Section 5. Eleven of them, and the list is checkable.

Ask what the requester intends before saying what the snoop does. Section 6. Write invalidates, read downgrades, and a modified line's data moves either way.

Account for bias on any Type 2 device. Section 7. It is the largest term in the device's access cost and it is specific to CXL.

Ask how many entries the host directory has, and how many devices will attach. Section 8. The binding number is the sum across devices.

Separate what must be ordered from what happens to be serial. Section 9. The second is an implementation limit with a different fix.

Name the terms of the latency, not just the total. Section 10. Four terms, four owners.

Say what the second requester experiences. Section 11. One transaction walked is the easy case.

Define completion as something other than the data arriving. Section 12.

Give every step a failure path, and put a timeout above the ones that have none. Section 13. A timeout turns a hang into an error.

19. Synthesis and Implementation Reality

The home node is a real structure with a real cost. Serialisation means a per-address ordering point, which means state per outstanding transaction and a structure that can fill — and a full one back-pressures requests that have nothing to do with each other.

Snoop filters exist because directories are expensive. A filter is a conservative approximation of the directory: it may say an agent holds a line that it does not, producing a harmless unnecessary snoop, but never the reverse. Section 8's pressure is what they are built to relieve.

Bias transitions are not free either. Moving a page from host bias to device bias requires the host to be certain it holds nothing cached from that page, which is a flush, which is a cost paid once per transition. A workload that flips bias repeatedly pays section 7's penalty and the transition cost.

Completion acknowledgements are why a coherent fabric has more message classes than a non-coherent one. They must not be blocked behind the requests that depend on them, which is a deadlock-avoidance requirement and one of the structural reasons the flit layer separates traffic the way it does.

And the timeouts of section 13 are usually much longer than anything else in the system — microseconds against nanoseconds — because their job is to catch a failure, not to bound a latency. A transaction that hits one is not slow; it is broken.

20. Silicon Observability

Free, and on paper. Which protocols an agent implements and what states it supports. Sections 5 and 6's vocabulary.

Cheap. The directory's entry count, from the host's specification. Section 8's supply side.

Moderate. Per-agent hit and miss rates, which show directory pressure as an unexplained miss rate on a device whose working set has not changed. Section 8's demand side.

Moderate. The bias state of a device's pages over time, which is a driver-visible quantity on most implementations. Section 7.

Expensive. The split of an end-to-end latency into its four terms. Section 10 needs either a trace with timestamps at each hop or a set of experiments that move one term at a time.

Expensive, and the reason 26.7 exists. Retry counts and the identity of what lost a race. Section 11's numbers are visible only to an agent that counts them, and not every agent does.

Unobtainable after the fact. Which step of a hung transaction did not complete. Section 13's whole argument is that a timeout is what converts this from unobtainable into a reported error.

21. Debug Lab

A device's coherent traffic is slower than the model predicted and nothing reports an error.

Step 1 — check the bias state of the pages being accessed. Section 7. A device in host bias for memory it owns is the largest single factor and the cheapest to check.

Step 2 — check the device's miss rate against its own working set. Section 8. Misses on lines the device did not evict itself are back-invalidations, and they mean directory pressure from somewhere else.

Step 3 — split the latency. Section 10. Link, agent, directory, snoop — and the answer is usually that one of them is most of it.

Step 4 — check for contention on specific lines. Section 11. A hot line serialises every requester behind one home, and the retry cost does not appear as an error anywhere.

Step 5 — check whether the transactions are completing or merely delivering. Section 12. A requester waiting on acknowledgements looks idle and is not.

Step 6 — if something is hung rather than slow, find the step with no timeout above it. Section 13. That is where it is waiting.

Steps 1 and 2 are reads from existing counters, which makes the first half of this cheap. Steps 3 to 5 need instrumentation that not every system has, which is why steps 1 and 2 are first.

22. Design Review

Walk the transaction. All of it, in order, with the states named.

For each snoop: what does the requester intend, what state is the line held in, and what does the snooped agent do about it?

Which lines have data that has to move, and where does it go?

For a Type 2 device: what fraction of accesses are in host bias, and what does that cost?

How many directory entries does the host have, and what is the aggregate demand across every device that will attach?

Which operations must be ordered, and which are serial only because of how many lanes there are?

What are the four terms of the end-to-end latency, and which one dominates?

What does the second requester for a contended line experience?

When is the transaction complete, and how is that different from when the data arrived?

Which steps have no failure path, and what timeout sits above them?

23. How This Appears In Real Engineering

The answer is confident, correct as far as it goes, and stops at the interesting part.

The most common shape is section 12. The walk is complete up to the data arriving — request, serialisation, directory, snoop, data — and then it ends, because that is where the data movement finishes and the data movement is what the diagram shows. The acknowledgements, the directory update and the completion are a third of the transaction's latency on a contended line and they are not in the answer at all.

The second shape is section 6, and it is a genuine mental model rather than an omission. Coherency was learned on write traffic, where a snoop does invalidate, and the read cases — downgrade, no-op, and the writeback that accompanies both — were never separately learned. The answer is not a simplification of a complete model; it is a complete model of a simpler protocol.

The third is section 7 and it is specific to this domain. A perfectly good coherency answer is given to a CXL question — home nodes, directories, MESI states, all correct — with no mention of bias, because bias is not part of general coherency. It is the part of the question that was actually about CXL, and it is missing.

The fourth is section 13 and it shows up in design documents rather than in interviews. Every step has a description and four of them have no failure path, because the failure paths are written last and the schedule ends first. The system works, ships, and hangs once a month in a way nobody can attribute, which is section 13's unbounded wait with a real customer attached.

The pattern is that the parts of a coherent transaction that are easy to draw are not the parts that cost time or cause bugs, and every failure above is a step that a diagram does not have a shape for.

24. Common Misconceptions

"The snoop invalidates it." On a write. On a read it downgrades, and the data still moves. Section 6.

"I named the requester, the home, the directory and the snoop." Those are the parts. The question asked for the sequence. Section 5.

"Bias is a configuration detail." It is the largest term in a Type 2 device's access cost. Section 7.

"Coherent attach costs the device, not the host." Every cached line is a host directory entry. Section 8.

"The transaction is ordered." Parts of it. Ordering all of it describes a slower protocol. Section 9.

"It takes three hundred nanoseconds." In four terms with four different owners. Section 10.

"Then the data comes back and we are done." The data came back. Section 12.

"The second requester just waits." It retries, and the retry is an ordering event as well as a delay. Section 11.

"Every step completes." Four of eleven have no path for what happens if one does not. Section 13.

25. Interview Reasoning

"Walk me through a coherent read where another device has the line modified." Requester misses and asks the home. The home serialises against anything outstanding for that address and consults the directory, which names the peer as holding it modified. The home snoops the peer. Because the requester wants to read, the peer downgrades to shared rather than invalidating, and supplies the data — to the requester directly, or through the home, depending on the implementation. The peer tells the home what state it kept. The home updates the directory to show both agents sharing. The requester acknowledges, the home completes the transaction and is free to process the next request for that line, and the requester installs the line shared.

"What if the requester wanted to write instead?" Then the peer invalidates rather than downgrading, the directory records a single owner, and the requester installs it modified. The data still moves, because the peer's copy was the current one.

"Where does the time go?" Four terms: the link crossing, the agent's own pipeline, the directory lookup and the snoop. The snoop is usually the largest because it is bounded by the slowest agent that has to respond, and the four have different owners — so a total tells you nothing about what to fix.

"This is a Type 2 device. What have I not asked about yet?" Bias. If the pages are in host bias the device's accesses to its own memory are resolved through the host, which can be four times the local cost, and nothing about the transaction walk reveals it.

"Two agents request the same line at once. What happens?" The home serialises them — that is what a home node is for. One is processed; the other is queued or retried. The retry costs time and re-orders that requester's traffic relative to everything else it was doing, and the second effect causes the harder bugs.

"When is the transaction complete?" Not when the data arrives. When every acknowledgement the granted state depends on has been collected and the home has recorded the new state. On an uncontended line those are nearly the same moment, which is why the shortcut survives.

"What if the snoop response never comes?" A timeout, which turns a hang into an error the system can report and recover from. If no timeout sits above that step, the transaction waits for a message that is not coming, and so does everything ordered behind it.

26. Exercises

1. Write the eleven steps from memory, then check them against section 5. Which did you omit, and is it before or after the data moves?

2. Two hundred lines snooped, 60 held modified and 90 shared. Compute invalidations, downgrades and lines of data moving, for a read and for a write. Which number is the same in both?

3. A device makes 4,000 accesses, 1,500 in host bias. Local is 120 ns, host resolution 700 ns. Compute the total and the bias penalty. What fraction of the penalty disappears if half those pages transition to device bias?

4. A host has 2,048 directory entries. Six devices attach, each caching 800 lines. Compute the shortfall and the forced back-invalidations. How many devices can attach before the directory is the constraint?

5. Thirty operations, nine of which must be ordered, at 40 ns each across three lanes. Compute the serial time, the parallel time and the total. Redo it with one lane and say which number is a protocol constraint.

6. A chain of 120, 60, 200 and 90 ns. Compute the total, the largest term and its share. Which term would you attack, and who owns it?

7. Eight requests contend for one line, retry cost 300 ns, base latency 500 ns. Compute the retries and the total retry cost. What is the cost to the eighth requester specifically?

8. Data at 150 ns, five acknowledgements at 80 ns each. Compute the resolution point and the completion gap. At what acknowledgement count does the gap exceed the data latency?

9. Extend the assembled model with a seventh bit for a condition this chapter does not cover. Justify its position using the rule that the ordering is by how far into the answer each condition appears.

27. Summary

The walk is the answer, and naming the participants is not sequencing them — eleven steps, and four of them is thirty-six percent of a transaction.

A snoop does not always invalidate. Write invalidates, read downgrades, a shared line under a read snoop does nothing, and a modified line's data moves in every case.

Bias decides what a Type 2 device's access costs, and it is the part of a CXL coherency question that a general coherency answer does not contain.

The host tracks what the device holds, in a finite structure, and a full one takes lines back from devices that were using them.

Not everything is ordered, and ordering all of it describes a slower protocol than the one being asked about.

The end-to-end number is a sum of four terms with four different owners, and the total identifies none of them.

Two agents can want the same line, and what the second one experiences is where the protocol is.

The data arriving is not the transaction finishing — a hundred and eighty nanoseconds apart on a contended line, and that step is the one most answers omit.

Every step needs a failure path, and a timeout is what turns a hang into an error.

Six bits, and "the snoop invalidates it" is one of them. One answer of eight is sound; the invalidation view counts six.