CXL vs CHI

28.1 placed CXL against the protocol it is built on. This chapter places it against the protocol it is most often confused with — what is the difference between CXL and CHI? — and it is the harder comparison, because this time both sides really are coherency protocols.

They are both coherent. True of both, which is more than the last chapter's opening could claim, and it is still where most answers stop: no scope, no symmetry, no home placement, no transport and nothing about what happens when half of the domain disappears.

1. The Engineering Problem — Where The Domain Stops

The difference is where the coherency domain stops. Four agents of sixteen across a package boundary is a domain whose worst-case reach is the far side for everybody — one agent across sets it for all of them. Section 5.

CHI is symmetric between its request nodes and CXL is not. Sixty of two hundred operations that have to be asked for is two thousand four hundred units of asking a symmetric fabric would never have spent. Section 6.

CHI distributes the home; CXL's home is the host. Sixty-four addresses on one home is a host round trip where eight homes would cost a fabric hop — and it is the same address space either way. Section 7.

A package boundary is a failure boundary. Four agents across it is a quarter of the domain that can disappear independently, and four recovery paths nobody has enumerated. Section 9.

A bridge between them is a translation. Nine of fourteen message kinds with an equivalent is five approximations, and the approximations are where the correctness argument lives. Section 13.

This chapter against 28.1, stated precisely. That comparison had an easy half — one side is not coherent at all. This one has no easy half, which is why the weak definition here is true of both sides rather than roughly true of one, and why every section is about a property that coherence alone does not fix.

2. The One-Sentence Model

A comparison of CXL against CHI is sound when both are named as coherency protocols, when the domain's scope is stated, when the symmetry between agents is stated, when the home's placement is stated, when the transport is stated, and when the failure boundary is stated — and "they are both coherent" is one of those six.

3. What This Chapter Owns

Some vocabulary, because both protocols use the same words for structures that sit in different places.

CHI is Arm's coherent hub interface — the protocol that carries coherent traffic between the agents on a die or a package, over a network on chip. It is what the cores, the caches, the memory controllers and the on-die accelerators of a server SoC talk over.

A request node is an agent that issues requests. CHI distinguishes fully-coherent request nodes, which hold cached state and answer snoops, from I/O-coherent ones, which do not. The fully-coherent ones are peers of each other, and that symmetry is section 6's subject.

A home node is the serialisation point for an address, exactly as in 27.8. The difference is that CHI has many of them spread across the mesh and CXL has the host's, which is section 7.

A network on chip is the transport — credited, parallel, wide, and with latency measured in a handful of cycles. CXL's transport is a serialised link carrying fixed-size flits, which is section 8.

And the package boundary is the line neither protocol crosses in the same way. CHI stops at it; CXL was built to cross it. Everything in this chapter is a consequence of that single geometric fact.

4. Teaching-Model Boundary

Every model in this chapter is a teaching model, not a protocol implementation. It computes the one relationship the section is about and nothing else. There is no mesh, no link and no coherency engine anywhere in this file.

Each model is built twice from one source. A parameter selects between the measured build, which counts what the two protocols decide differently, and the both-are-coherent build, which counts the shared property as settling the question. Every section's headline number is the gap between them.

5. RTL 1 — Where The Domain Stops

Start with the geometry, because every other difference follows from it.

CHI's coherency domain is a die or a package. Every agent in it is reachable in a handful of cycles over a wide parallel fabric that the same team laid out, powered from the same supply, clocked from the same source, and failing as one piece. That is not a limitation the protocol works around — it is the assumption the protocol is built on.

CXL's coherency domain crosses a package boundary. The agents are on different silicon, in different packages, possibly in different chassis, connected by a serialised link with retimers in it. Coherence still works; everything about what coherence costs changes.

// RTL 1 - the difference is where the coherency domain stops. CHI is a
// coherency fabric on a die or a package; CXL extends coherence across a
// package boundary and a serialised link. Every other difference is a
// consequence of that one line on a floorplan.
module domain_scope #(parameter int COHERENT_IS_COHERENT = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] agents_total, agents_on_die, on_die_ns, off_die_ns,
  output logic [15:0] on_die_ok, agents_off_die, worst_ns, off_die_pct,
  output logic        one_domain,
  output logic [7:0]  n_evals, n_spanning,
  output logic        scope_err
);
  logic [31:0] o_q;
  logic [15:0] true_off, true_worst;
  logic        truly_spanning;
  assign on_die_ok = (agents_on_die > agents_total) ? agents_total : agents_on_die;
  assign true_off = agents_total - on_die_ok;
  assign agents_off_die = (COHERENT_IS_COHERENT != 0) ? 16'd0 : true_off;
  // The domain's worst-case reach is the slowest agent in it, and one agent
  // across the boundary sets it for everybody.
  assign true_worst = (true_off != 16'd0) ? off_die_ns : on_die_ns;
  assign worst_ns = (COHERENT_IS_COHERENT != 0) ? on_die_ns : true_worst;
  assign o_q = (agents_total == 16'd0) ? 32'd0
             : (({16'd0, true_off} * 32'd100) / {16'd0, agents_total});
  assign off_die_pct = o_q[15:0];
  assign one_domain = (agents_off_die == 16'd0) && (agents_total != 16'd0);
  assign truly_spanning = (true_off != 16'd0);
  assign scope_err = evaluate && truly_spanning && one_domain;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_spanning <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_spanning) n_spanning <= n_spanning + 8'd1;
    end
  end
endmodule

Sixteen agents with twelve on the die is four across the boundary, and the domain's worst-case reach becomes the far one for all sixteen — and the coherent-is-coherent view reports the on-die reach for everybody.

Figure 1 — the one fact the rest of the chapter follows from. Both readings describe the same sixteen agents and the same correct coherency. The upper one reports the domain by its typical member; the lower one reports it by its worst, which is the only reading a designer can use — a coherency domain is bounded by its slowest participant, because any line may be held by any of them and a snoop has to reach whoever holds it.

The last case makes that concrete in the sharpest form. One agent across the boundary sets the worst-case reach for the entire domain, at six percent of the agent count. There is no averaging available: a snoop that has to reach the far agent takes the far agent's time, and the fifteen on-die agents wait for it exactly as if they were all off-die.

The sixth case is the honest limit and it is worth including because it isolates the variable. A boundary that costs the same as staying on the die — a hypothetical zero-cost link — leaves the latency identical and the model still declines to call it one domain, because section 9's failure argument is untouched by latency. The boundary is two things, a distance and a failure line, and only the first of them can be engineered away.

The fifth case is the other pole: every agent across the boundary is a purely link-attached domain, which is what a host with only CXL-attached coherent devices looks like, and which is a coherency domain with no mesh in it at all.

The degenerate case bounds the model: a domain with no agents cannot span a boundary, and the model reports the on-die reach by default while declining to call it one domain.

It is worth being concrete about what the boundary costs, because "off die" is a category rather than a number. A die crossing is a few millimetres of wire and a few cycles. A package crossing is a SerDes: serialise, equalise, transmit, receive, deserialise, and do it again coming back — plus the flit framing of section 8, plus any retimer in between. That is where a hundred-nanosecond-class figure comes from, and none of it is protocol overhead. It is physics and framing, and no amount of protocol design removes it.

Which is why the comparison is not a contest. CHI is faster across its domain because its domain is smaller, and CXL reaches further because it accepted the cost of reaching. A design that wants both properties in one protocol is asking for a smaller package.

The third case is the clamp and it makes a small point precisely. Placing agents that do not exist on the die changes nothing, because the domain is bounded by the agents it has — and a floorplan that claims capacity for agents nobody is building is not a coherency argument.

There is a second-order consequence worth carrying forward. The worst-case reach is what the protocol's timeouts have to be set from, not the typical one. A domain with one agent across a boundary has to accommodate that agent's response time in every timeout that could involve it, which stretches the timeouts for the fifteen on-die agents too — 27.8 section 13's timeouts, sized by the slowest possible participant.

6. RTL 2 — CHI Is Symmetric And CXL Is Not

The second thing, and the one that decides what an agent may do on its own.

CHI's fully-coherent request nodes are peers. Any of them may request any line in any state; the home serialises them and none of them is privileged. Two accelerators on the same mesh have exactly the same rights over memory as the cores do.

CXL is asymmetric by construction. The host is the coherency master; the device participates. A device may cache host memory through CXL.cache, and it does so under a protocol where the host's agent is the one that resolves, tracks and decides. A device cannot snoop the host, and the operations that a peer would simply issue are operations a CXL device has to ask for.

// RTL 2 - CHI is symmetric between its fully-coherent request nodes; CXL is
// not. On CXL the host is the coherency master and the device is a
// participant, which decides what a device may do on its own and what it must
// ask for.
module symmetry_model #(parameter int PEERS_ARE_PEERS = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] ops, ops_needing_host, agents, ask_cost,
  output logic [15:0] host_ops, own_ops, ask_total, own_pct,
  output logic        symmetric,
  output logic [7:0]  n_evals, n_asymmetric,
  output logic        symmetry_err
);
  logic [31:0] a_q, o_q;
  logic [15:0] need_ok, true_host, true_own;
  logic        truly_asymmetric;
  assign need_ok = (ops_needing_host > ops) ? ops : ops_needing_host;
  assign true_host = need_ok;
  assign true_own = ops - need_ok;
  assign host_ops = (PEERS_ARE_PEERS != 0) ? 16'd0 : true_host;
  assign own_ops  = (PEERS_ARE_PEERS != 0) ? ops : true_own;
  // Every operation that has to be asked for costs a round trip the symmetric
  // fabric would not have taken.
  assign a_q = {16'd0, true_host} * {16'd0, ask_cost};
  assign ask_total = (a_q > 32'd9999) ? 16'd9999 : a_q[15:0];
  assign o_q = (ops == 16'd0) ? 32'd0
             : (({16'd0, own_ops} * 32'd100) / {16'd0, ops});
  assign own_pct = o_q[15:0];
  assign symmetric = (host_ops == 16'd0) && (ops != 16'd0);
  assign truly_asymmetric = (true_host != 16'd0);
  assign symmetry_err = evaluate && truly_asymmetric && symmetric;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_asymmetric <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_asymmetric) n_asymmetric <= n_asymmetric + 8'd1;
    end
  end
endmodule

Sixty of two hundred operations needing the host is two thousand four hundred units of asking and seventy percent of the traffic self-directed — and the peers-are-peers view reports symmetry.

The fifth case separates the structure from its price, and it is the one that matters for the argument. Asking that costs nothing is still asking: the model reports the sixty operations with a zero cost beside them, because asymmetry is a statement about rights rather than about latency. A device that must request something cannot proceed without a reply, however fast the reply is, and that dependency is what a symmetric fabric does not have.

The last case is the boundary. One operation in two hundred is enough to make "they are peers" false, for the same reason 28.1 section 7 gives: the claim is about capability, and one thing the agent cannot do alone is one thing too many.

The second case is what symmetry actually looks like when it is present, and it is worth stating because the model must be able to report it. An agent that needs nobody issues all two hundred operations itself, at no asking cost, and both views agree. That is CHI between two fully-coherent request nodes, and it is the thing CXL deliberately did not build.

The third case is the far end and it is a real device. Every operation asked for — a participant with no independent rights at all — is what an I/O-coherent agent is, on either protocol, and it is the correct design for a device that reads host memory occasionally and caches nothing.

The reason CXL is asymmetric is worth stating, because the asymmetry reads as a limitation and is a deliberate simplification. A symmetric protocol requires every agent to trust every other agent's coherency implementation. On a die that is reasonable: one team, one verification programme, one tape-out. Across a link it is not — the device is another vendor's silicon, verified by another team, and a coherency bug in it would corrupt the host's memory.

Making the host the master bounds the damage. The device's obligations are narrower, its state space is smaller, the host validates what it is told, and the failure modes a misbehaving device can produce are constrained. That is the same reasoning as 27.6 section 9's argument about Type 2's verification cost, applied at the protocol level rather than at the device's.

It is also why a device cannot snoop the host. Not because the mechanism would be hard, but because doing so would let a device demand that host caches give up lines — which is a capability nobody wants to hand to a card. The asymmetry is a trust boundary drawn on the same line as the package boundary, and section 9 draws the failure boundary on it too.

The third case is the arithmetic at its limit. More asking than there are operations, clamped to the operations that exist, still reports eight thousand units of asking — because a device that asks for everything is doing something real and expensive, not something impossible.

7. RTL 3 — Where The Home Sits

The third thing, and the one that decides how far a request travels before it is ordered.

CHI spreads the home across the mesh. An address's serialisation point sits near the memory controller that owns it, which means a request travels a few hops, is ordered, and is answered — and adding memory adds homes, so the serialisation capacity grows with the system.

CXL's home is the host. Every coherent request from a device is resolved by the host's home agent, wherever the device and the memory happen to be. That is a fixed point, it does not move nearer, and it does not multiply when devices are added.

// RTL 3 - CHI distributes the home across the mesh so that an address's
// serialisation point is near its memory; CXL's home is the host. That
// decides how far a request travels before it is ordered, and it does not
// scale the same way.
module home_placement #(parameter int A_HOME_IS_A_HOME = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] addresses, homes, hop_ns, host_ns,
  output logic [15:0] homes_ok, per_home, reach_ns, spread_pct,
  output logic        home_is_near,
  output logic [7:0]  n_evals, n_far,
  output logic        home_err
);
  logic [31:0] p_q, s_q;
  logic [15:0] true_reach;
  logic        truly_far;
  // More homes means each one serialises fewer addresses and sits closer to
  // the memory it orders.
  assign homes_ok = (homes > addresses) ? addresses : homes;
  assign p_q = (homes_ok == 16'd0) ? 32'd9999
             : ({16'd0, addresses} / {16'd0, homes_ok});
  assign per_home = (p_q > 32'd9999) ? 16'd9999 : p_q[15:0];
  // A single home at the host costs the host round trip; a distributed home
  // costs a hop on the fabric.
  assign true_reach = (homes_ok > 16'd1) ? hop_ns : host_ns;
  assign reach_ns = (A_HOME_IS_A_HOME != 0) ? hop_ns : true_reach;
  assign s_q = (addresses == 16'd0) ? 32'd0
             : (({16'd0, homes_ok} * 32'd100) / {16'd0, addresses});
  assign spread_pct = s_q[15:0];
  assign home_is_near = (reach_ns <= hop_ns) && (addresses != 16'd0);
  assign truly_far = (true_reach > hop_ns);
  assign home_err = evaluate && truly_far && home_is_near;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_far <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_far) n_far <= n_far + 8'd1;
    end
  end
endmodule

Sixty-four addresses on a single home is all sixty-four serialised at one point, at a host round trip rather than a fabric hop — and the a-home-is-a-home view charges a hop for it.

Figure 2 — the same address space, two placements. Nothing about the protocol logic differs between them: the home does the same job, consults the same directory, issues the same snoops. What differs is how far the request travelled to reach it, and on a distributed home that distance is a design parameter while on a centralised one it is the host's location. The second number is not a worse implementation of the first; it is the same mechanism with the geometry of section 5 applied to it.

The sixth case isolates the variable cleanly. A host round trip that costs a hop makes a single home near, and the model says so. Home placement costs exactly what the distance costs — the centralisation is not intrinsically expensive, it is expensive because the host is far, which is section 5 again.

The third case is the other extreme and it is worth knowing as a limit. A home per address is as near as a home can be, and the model clamps at that rather than crediting more homes than addresses. Real meshes are nowhere near it; the point is that the technique has a ceiling and CHI's value is in being partway up the curve rather than at the top of it.

The last case is the shape at real scale. Sixty thousand addresses on one home saturates the per-home figure and rounds the spread to nothing — which is what a memory map looks like when a single serialisation point owns all of it, and is the structural reason a centralised home is a scaling question rather than only a latency one.

The fourth case bounds the model at the degenerate end. Homes declared for a space with no addresses reports no homes and falls back to the host reach, which is an empty map rather than a near home.

The scaling half of this is the part that outlives the latency half, and it is worth separating. A distributed home multiplies the serialisation capacity as well as shortening the distance. Sixteen homes can each be resolving a different address at the same time; one home resolves one address at a time and queues the rest. On a workload with many agents touching many addresses, that queueing is a throughput limit rather than a latency one, and it does not appear in any single access's measurement.

That is the structural reason section 10's provisioning argument exists. A centralised home is a fixed resource shared by every device that attaches, so adding devices makes each one's serialisation worse — which is the opposite of a mesh, where adding agents and adding homes happen together.

The fifth case is the honest degenerate one and it is a real configuration rather than an artefact. Addresses with no home at all is what an address range looks like before anybody has decided who orders it, and the model falls back to the host — which is also what the system does, and is the reason an unconfigured range performs like a remote one.

8. RTL 4 — A Mesh And A Link Are Different Transports

The fourth thing, and the one that is easiest to forget because it is below the protocol.

CHI runs on a network on chip. Wide parallel buses, credited channels, a handful of cycles per hop, and a message that occupies as many bytes as it needs. There is no framing overhead to speak of because there is no frame.

CXL runs on a serialised link in fixed-size flits. Every message is carried in whole flits, and the remainder of the last one crosses the link as padding. That is not waste in any avoidable sense — it is what framing costs — but it is a cost the mesh does not have, and it applies to every small message the protocol sends.

// RTL 4 - the transports are different in kind. CHI runs on a network on
// chip with credited channels and wide parallel buses; CXL runs on a
// serialised link in fixed-size flits. The same message costs a different
// amount and arrives with different jitter.
module transport_model #(parameter int A_MESSAGE_IS_A_MESSAGE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] messages, msg_bytes, flit_bytes, noc_width,
  output logic [15:0] flits_used, noc_beats, pack_waste, efficiency_pct,
  output logic        transport_free,
  output logic [7:0]  n_evals, n_packed,
  output logic        transport_err
);
  logic [31:0] f_q, b_q, e_q;
  logic [15:0] per_msg, true_flits, true_beats, true_waste;
  logic        truly_packed;
  // A message is carried in whole flits, and the remainder of the last one is
  // padding that crosses the link anyway.
  assign per_msg = (flit_bytes == 16'd0) ? 16'd1
                 : (((msg_bytes + flit_bytes - 16'd1) / flit_bytes) == 16'd0
                    ? 16'd1 : ((msg_bytes + flit_bytes - 16'd1) / flit_bytes));
  assign f_q = {16'd0, messages} * {16'd0, per_msg};
  assign true_flits = (f_q > 32'd9999) ? 16'd9999 : f_q[15:0];
  assign flits_used = true_flits;
  assign b_q = (noc_width == 16'd0) ? 32'd9999
             : (({16'd0, messages} * {16'd0, msg_bytes}) / {16'd0, noc_width});
  assign true_beats = (b_q > 32'd9999) ? 16'd9999 : b_q[15:0];
  assign noc_beats = true_beats;
  assign true_waste = (({16'd0, true_flits} * {16'd0, flit_bytes}) > ({16'd0, messages} * {16'd0, msg_bytes}))
                    ? 16'd1 : 16'd0;
  assign pack_waste = (A_MESSAGE_IS_A_MESSAGE != 0) ? 16'd0 : true_waste;
  assign e_q = (true_flits == 16'd0) ? 32'd100
             : (({16'd0, messages} * 32'd100) / {16'd0, true_flits});
  // No clamp: every message costs at least one flit, so true_flits is at
  // least messages and the quotient cannot exceed a hundred.
  assign efficiency_pct = e_q[15:0];
  assign transport_free = (pack_waste == 16'd0) && (messages != 16'd0);
  assign truly_packed = (true_waste != 16'd0);
  assign transport_err = evaluate && truly_packed && transport_free;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_packed <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_packed) n_packed <= n_packed + 8'd1;
    end
  end
endmodule

A hundred twenty-byte messages in sixty-four-byte flits is a hundred flits carrying two thousand bytes of content across six thousand four hundred bytes of link — and the a-message-is-a-message view reports no padding.

The last case inverts an intuition worth correcting. A flit smaller than the message costs more framing, not less — twenty-byte messages in sixteen-byte flits take two flits each, with padding in the second, and the efficiency halves. Flit size is not a knob that makes framing cheaper; it trades the padding on small messages against the flit count on large ones, and a protocol picks one point on that curve for all of its traffic.

The third case is what the mesh side actually looks like. A transport with no flit size at all pads nothing, because there is no frame to pad to — which is the honest model of a wide parallel bus, and is why the comparison is between two kinds of transport rather than two sizes of the same one.

The fifth case keeps the two costs separate deliberately. An undeclared fabric width saturates the beat count and leaves the flit count and the padding unchanged, because the two transports are costed independently. A comparison that folded them into one number would have to choose units, and the units are the difference.

The sixth case is where both ceilings flatten the comparison at once, and the model reports it rather than hiding it: two saturated counts still carry the framing difference, because the padding verdict is structural rather than a magnitude.

The degenerate case bounds it: a comparison with no messages reports full efficiency of nothing, which is an idle transport rather than a free one.

The credit question is worth a paragraph because it is where the two transports differ in behaviour rather than in arithmetic. A network on chip is credited per channel, and an agent that has no credit simply does not send — back-pressure is local, immediate and precise. A serialised link carries credit for several protocols interleaved in the same flit stream, so back-pressure on one class interacts with the others through the arbitration between them.

That coupling is the part with no analogue on the mesh. A CXL.mem stream starved of credit and a CXL.cache stream with credit to spare are sharing a flit stream, and the arbitration between them is a design decision with correctness consequences — 27.8 section 13's deadlock-avoidance requirement is exactly this problem. On a mesh the channels are physically separate and the question does not arise.

The fourth case is the model's own boundary and it is worth reading as a statement about parallel buses. A transport with no flit size is unframed by construction, and the model reports no padding rather than inventing some. That is what a wide on-die bus is, and it is why section 8's whole cost exists on one side of the comparison and not the other.

9. RTL 5 — A Package Boundary Is A Failure Boundary

The fifth thing, and the one with the largest consequence for system design.

Agents on one die fail together and are designed to. A die loses power, a clock stops, a thermal event trips — the whole domain goes at once, and the recovery is a reset of the thing that failed. Nobody writes a recovery path for half a mesh disappearing, because half a mesh cannot.

Agents across a link fail independently. A device can be removed, lose power, be reset by its own firmware, or fail a link and go away while the host keeps running — and the host's coherency state still says that device holds lines. Every one of those is a recovery path somebody has to design, and they have no analogue on a mesh.

// RTL 5 - a package boundary is a failure boundary. Agents on one die fail
// together and are designed to; agents across a link fail independently, and
// a coherency domain that spans the boundary has to have an answer for one
// half disappearing.
module failure_boundary #(parameter int ONE_DOMAIN_ONE_FATE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] agents, agents_across, recovery_paths, handled_paths,
  output logic [15:0] across_ok, blast_agents, unhandled, blast_pct,
  output logic        fate_shared,
  output logic [7:0]  n_evals, n_exposed,
  output logic        boundary_err
);
  logic [31:0] b_q;
  logic [15:0] handled_ok, true_blast, true_unhandled;
  logic        truly_exposed;
  assign across_ok = (agents_across > agents) ? agents : agents_across;
  assign true_blast = across_ok;
  assign blast_agents = (ONE_DOMAIN_ONE_FATE != 0) ? 16'd0 : true_blast;
  assign handled_ok = (handled_paths > recovery_paths) ? recovery_paths : handled_paths;
  assign true_unhandled = recovery_paths - handled_ok;
  assign unhandled = (ONE_DOMAIN_ONE_FATE != 0) ? 16'd0 : true_unhandled;
  assign b_q = (agents == 16'd0) ? 32'd0
             : (({16'd0, true_blast} * 32'd100) / {16'd0, agents});
  assign blast_pct = b_q[15:0];
  assign fate_shared = (blast_agents == 16'd0) && (unhandled == 16'd0)
                     && (agents != 16'd0);
  assign truly_exposed = (true_blast != 16'd0) || (true_unhandled != 16'd0);
  assign boundary_err = evaluate && truly_exposed && fate_shared;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_exposed <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_exposed) n_exposed <= n_exposed + 8'd1;
    end
  end
endmodule

Four agents of sixteen across the boundary with six recovery paths and two handled is a quarter of the domain able to disappear independently, and four paths with no answer — and the one-domain-one-fate view reports a single fate.

The sixth case is the one that stops this from being an argument about link-attached designs only. Nothing across the boundary and no recovery path handled still reports six unhandled paths, because an on-die domain has failure answers of its own — they are simply easier, and easier is not the same as absent.

The fifth case is what an early design document looks like. Agents across a boundary with no recovery path enumerated at all reports the blast radius and zero unhandled paths, which reads as containment and is the absence of the question. The two are distinguishable only by asking how many paths were enumerated, which is why the model publishes both counts.

The third case is the worst geometry and it is the common one at first. The whole domain across the boundary puts a hundred percent of the agents in one blast radius, which is what a host with all of its coherent agents on links actually has — and it is 27.7 section 10's argument arriving one layer down.

The degenerate case bounds it: an uncounted domain is not a contained one, and the model reports the unhandled paths while declining to call the fate shared.

The recovery paths worth enumerating are short enough to list, and listing them is most of the work. A device removed while holding lines — hot-remove, surprise or planned. A device that resets itself and comes back with no memory of what it held. A link that goes down and retrains, during which outstanding transactions are in an unknown state. A device that stops responding without going away, which is the worst case because nothing signals it. And a host that resets while devices hold its lines, which is the mirror and is often forgotten entirely.

Each of those needs a policy, a timeout and a way back to a consistent state, and the policies are not obvious: a host that forcibly drops a departed device's lines is making an assumption about what that device did with them. 27.8 section 13's argument — that a step with no failure path does not fail fast, it hangs — applies to every one.

The second case is the mesh's position and it is worth stating as a positive rather than as a foil. Nothing across the boundary and every path handled is a domain that genuinely does share a fate, and the model reports it. That is not the mesh being simpler by accident; it is the mesh's designers having been allowed to assume something CXL's cannot.

10. RTL 6 — They Scale By Different Mechanisms

The sixth thing, and the one that decides which protocol a system can grow into.

A mesh scales by tiling. Add agents, add mesh nodes, add homes; the fabric grows with the design and the serialisation capacity grows with it. The limit is what the floorplan and the power budget allow, and it is decided when the die is laid out.

A host scales by provisioning. Ports, directory entries, decoders — all fixed at silicon, all shared across every device that attaches, and none of them growing because another device arrived. Two very different limits, and the one that binds first is not always the obvious one.

// RTL 6 - the two fabrics scale by different mechanisms. CHI adds agents to
// a mesh and the home count grows with it; CXL adds devices to a host whose
// directory and port count are fixed at silicon. One scales by tiling and the
// other by provisioning.
module scaling_mechanism #(parameter int BOTH_SCALE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] agents_wanted, mesh_capacity, host_ports, dir_entries_each,
  output logic [15:0] mesh_ok, host_ok, agents_short, host_pct,
  output logic        scales_alike,
  output logic [7:0]  n_evals, n_short,
  output logic        scaling_err
);
  logic [31:0] d_q, h_q;
  logic [15:0] dir_limit, true_short;
  logic        truly_short;
  // A mesh is sized by how many agents it was laid out for.
  assign mesh_ok = (agents_wanted > mesh_capacity) ? mesh_capacity : agents_wanted;
  // A host is sized by its ports and by the directory each attached agent
  // needs, whichever binds first.
  assign d_q = (dir_entries_each == 16'd0) ? 32'd9999
             : (32'd9999 / {16'd0, dir_entries_each});
  // No clamp: the quotient of 9999 by a non-zero divisor cannot exceed 9999,
  // and the zero case already reports exactly that, so a ceiling is dead.
  assign dir_limit = d_q[15:0];
  assign host_ok = (host_ports > dir_limit) ? dir_limit
                 : ((agents_wanted > host_ports) ? host_ports : agents_wanted);
  assign true_short = (agents_wanted > host_ok) ? (agents_wanted - host_ok) : 16'd0;
  assign agents_short = (BOTH_SCALE != 0) ? 16'd0 : true_short;
  assign h_q = (agents_wanted == 16'd0) ? 32'd100
             : (({16'd0, host_ok} * 32'd100) / {16'd0, agents_wanted});
  assign host_pct = h_q[15:0];
  assign scales_alike = (agents_short == 16'd0) && (agents_wanted != 16'd0);
  assign truly_short = (true_short != 16'd0);
  assign scaling_err = evaluate && truly_short && scales_alike;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_short <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_short) n_short <= n_short + 8'd1;
    end
  end
endmodule

Thirty-two agents wanted against a mesh laid out for sixty-four and a host with eight ports is twenty-four agents with nowhere to attach — a quarter of them placed — and the both-scale view reports the same mechanism.

The sixth case is the one worth taking to a review, because it is the limit people do not check. A directory requirement that binds before the ports do — a hundred ports available and four agents attachable, because each one wants two thousand directory entries — is a host that looks generously provisioned and is not. The binding constraint is whichever of the two runs out first, and a specification that quotes port count alone has quoted the one that is easier to make large.

The last case is the boundary. A host one port short places thirty-one of thirty-two and is still short, because the question is whether the system the design wants can be built, and one agent that cannot attach is a design that cannot ship as drawn.

The third case shows both mechanisms running out together and is instructive about the asymmetry. A hundred agents wanted exceeds both the mesh's layout and the host's provisioning, and the two have entirely different remedies: the mesh needs a new floorplan and a new die, and the host needs a bigger part or another host. Neither is a configuration change, which is the honest version of "they both scale".

The degenerate case bounds it: a comparison with no agents wanted places everything and is an unasked question rather than a scaling result.

There is a third scaling axis neither model counts and a review should ask about. Snoop traffic grows with the number of agents that might hold a line, on both protocols, and it grows faster than the agent count if the sharing pattern is wide. A mesh absorbs that because its channels are wide and parallel; a link carries it in the same flit stream as everything else, which is section 8's coupling appearing as a scaling limit.

That is why snoop filters matter more on the CXL side. 27.8 section 8's directory pressure is a host-side resource question; the snoop traffic it generates is a link-side bandwidth question, and the two are the same structure seen from opposite ends.

The second case is the configuration that makes the section fair. A host with a port for every agent does scale to the mesh's count, and the model says so — provisioning is not inherently inferior, it is inherently fixed, and a host provisioned for the design in front of it is adequate for that design and nothing beyond it.

11. RTL 7 — The State Sets Do Not Map One To One

The seventh thing, and the one that turns a bridge from a wire into a design.

Coherency protocols carry state, and the state sets differ. CHI has states for lines that are dirty but shared, for ownership passed without data, for partial holdings — distinctions that exist because an on-die fabric can afford to track them and benefits from doing so. The device side of CXL's protocol does not express all of them.

A state with no equivalent has to fall back to a more conservative one that both sides understand. That is always correct and never free: every line held in the fallback pays the difference, in extra transactions, extra writebacks, or lost sharing.

// RTL 7 - the state sets are not the same and the difference is not cosmetic.
// CHI carries states CXL's device-side protocol does not express, and a
// design that assumes a one-to-one mapping has states with nowhere to go.
module state_mapping #(parameter int STATES_MAP_ACROSS = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] states_total, states_mapped, lines, fallback_cost,
  output logic [15:0] mapped_ok, states_lost, fallback_total, mapped_pct,
  output logic        maps_cleanly,
  output logic [7:0]  n_evals, n_lost,
  output logic        mapping_err
);
  logic [31:0] f_q, m_q;
  logic [15:0] true_lost;
  logic        truly_lost;
  assign mapped_ok = (states_mapped > states_total) ? states_total : states_mapped;
  assign true_lost = states_total - mapped_ok;
  assign states_lost = (STATES_MAP_ACROSS != 0) ? 16'd0 : true_lost;
  // A state with no equivalent has to fall back to a more conservative one,
  // and every line held in it pays the difference.
  assign f_q = (true_lost == 16'd0) ? 32'd0
             : ({16'd0, lines} * {16'd0, fallback_cost});
  assign fallback_total = (f_q > 32'd9999) ? 16'd9999 : f_q[15:0];
  assign m_q = (states_total == 16'd0) ? 32'd100
             : (({16'd0, mapped_ok} * 32'd100) / {16'd0, states_total});
  assign mapped_pct = m_q[15:0];
  assign maps_cleanly = (states_lost == 16'd0) && (states_total != 16'd0);
  assign truly_lost = (true_lost != 16'd0);
  assign mapping_err = evaluate && truly_lost && maps_cleanly;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_lost <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_lost) n_lost <= n_lost + 8'd1;
    end
  end
endmodule

Seven states with four mapped is three with nowhere to go, and six thousand units of conservative fallback across two thousand lines — fifty-seven percent of the state set, and the states-map-across view reports a clean mapping.

The last case separates the protocol question from the traffic question, and it is the one that makes this section a comparison rather than a benchmark. States lost on a cache holding nothing still reports three states unmapped, at zero cost — because the mapping is a property of the two protocols and the cost is a property of the workload. A design that reports only the cost will conclude the mapping is fine on a quiet system and discover otherwise under load.

The fifth case is the worst version. Nothing mapped at all puts every line in the most conservative state both sides understand, which is functionally correct and throws away the whole reason for having a rich state set.

The second case is the target and is achievable for a restricted subset: every state with an equivalent, no fallback, and a mapping that costs nothing. That is what a carefully-chosen intersection looks like, and choosing it deliberately is better than discovering it.

The degenerate case bounds the model: a comparison with no state set named reports a clean mapping of nothing, which is an unwritten mapping rather than a clean one.

What the fallback actually costs is worth making concrete, because "conservative" sounds free. A line that would have been held dirty-shared but must be held modified cannot be shared at all, so a second reader takes a miss that would not have happened. A line that would have been held with ownership but no data has to fetch the data. A partial holding that has to become a whole one drags bytes nobody wanted. Each of those is a transaction that the richer state set would have avoided, repeated for every line in that state for as long as it is held.

Which is why the cost scales with the working set rather than with the state count. Three unmapped states sound minor; three unmapped states across a two-thousand-line working set is six thousand units of extra work, and across fifty thousand lines it saturates the model. The protocol difference is small and the workload multiplies it.

The third case is the clamp and it says the expected thing precisely. More states mapped than exist is bounded by the state set that was named, because a mapping table with entries for states neither protocol has is a table nobody checked.

12. RTL 8 — Each Wins Where Its Boundary Is

The eighth thing, and the direction a one-fabric answer never goes.

An agent that can be on the die belongs on the die. A core, a cache, an on-die accelerator, a memory controller — all of them are in the mesh's latency tier by construction, and putting any of them behind a link costs the tier difference on every access they make.

An agent that cannot be on the die was never a CHI candidate. A device in another package, a memory expander on a card, an accelerator from another vendor — none of them can be in the mesh at all, whatever their coherency needs, and for those the comparison is not a choice.

// RTL 8 - each fabric wins where its boundary is. An agent that belongs on
// the die belongs on CHI, and putting it behind a link costs it the latency
// tier it was designed for; an agent that cannot be on the die was never a
// CHI candidate whatever its coherency needs.
module where_each_wins #(parameter int ONE_FABRIC_FITS_ALL = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] agents, must_be_off_die, on_die_ns, off_die_ns,
  output logic [15:0] off_ok, on_die_agents, misplaced_ns, on_die_pct,
  output logic        one_fabric,
  output logic [7:0]  n_evals, n_misplaced,
  output logic        fit_err
);
  logic [31:0] m_q, p_q;
  logic [15:0] true_on, true_cost, penalty;
  logic        truly_misplaced;
  assign off_ok = (must_be_off_die > agents) ? agents : must_be_off_die;
  assign true_on = agents - off_ok;
  assign on_die_agents = (ONE_FABRIC_FITS_ALL != 0) ? 16'd0 : true_on;
  // Putting an on-die agent across the link costs it the tier difference on
  // every access it makes.
  assign penalty = (off_die_ns > on_die_ns) ? (off_die_ns - on_die_ns) : 16'd0;
  assign m_q = {16'd0, true_on} * {16'd0, penalty};
  assign true_cost = (m_q > 32'd9999) ? 16'd9999 : m_q[15:0];
  assign misplaced_ns = (ONE_FABRIC_FITS_ALL != 0) ? 16'd0 : true_cost;
  assign p_q = (agents == 16'd0) ? 32'd0
             : (({16'd0, true_on} * 32'd100) / {16'd0, agents});
  assign on_die_pct = p_q[15:0];
  assign one_fabric = (misplaced_ns == 16'd0) && (agents != 16'd0);
  assign truly_misplaced = (true_cost != 16'd0);
  assign fit_err = evaluate && truly_misplaced && one_fabric;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_misplaced <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_misplaced) n_misplaced <= n_misplaced + 8'd1;
    end
  end
endmodule

Twelve agents with three that must be off die is nine belonging on it, and two thousand and seventy nanoseconds of misplacement if they are not — and the one-fabric view reports a fit.

The second case is the case CXL exists for and it is worth stating as a positive. Every agent has to be off the die — a host and a population of link-attached devices — and there is no misplacement at all, because nothing could have been on the mesh. CHI is not an option there; it is not a worse option. That distinction matters in a review, because a comparison that ranks the two protocols will reach for the wrong one whenever the geometry has already decided.

The fifth case is the mirror. Every agent belongs on the die and is put behind a link anyway, which is a mesh workload paying a link's latency on every access, and is what building a chiplet-based design with the wrong interconnect looks like.

The sixth case isolates the variable. A boundary that costs nothing removes the misplacement entirely, because the fit question is about the tier rather than about the boundary — which is the same separation section 5 makes between distance and failure, applied to placement.

The degenerate case bounds it: a comparison with no agents misplaces nothing and reports no fit.

The middle of this comparison is where the industry actually is, and it is worth naming because both models treat it as a boundary case. Chiplets are agents that are neither on the die nor across a system boundary — multiple dies in one package, close enough that the latency is far better than a link and worse than a mesh, and sharing a substrate that fails more together than two cards do and less than one die does.

That geometry is why die-to-die interconnects exist as a third category rather than as a choice between these two, and it is 27.6's question asked about packaging instead of about protocols. The models here bound the ends of that spectrum; the interesting designs sit between them, and the honest answer to "CXL or CHI for a chiplet" is usually "neither, and here is why".

The third case is the clamp and it reinforces the section's frame. More off-die agents claimed than exist changes nothing, because the placement question is about the agents a design has.

13. RTL 9 — A Bridge Is A Translation

The ninth thing, and the one that comes up the moment a real system has both.

Systems with both exist and are ordinary. A server SoC with a CHI mesh and CXL ports has a component that translates between them — requests from a CXL device become CHI requests on the mesh, and the mesh's snoops become CXL snoops going out. That component is not a wire.

Every message kind with an equivalent is translated and every one without has to be approximated, and the approximations are where a bridge's correctness argument lives — because an approximation is a place where the bridge chose a behaviour the original protocol did not specify.

// RTL 9 - a bridge between them is a translation, not a wire. Every message
// that has an equivalent is translated and every one that does not has to be
// approximated, and the approximations are where a bridge's correctness
// arguments live.
module bridge_cost #(parameter int A_BRIDGE_IS_A_WIRE = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic [15:0] msg_kinds, kinds_mapped, blocks, block_cost,
  output logic [15:0] mapped_ok, approximated, bridge_total, mapped_pct,
  output logic        bridge_is_cheap,
  output logic [7:0]  n_evals, n_approximated,
  output logic        bridge_err
);
  logic [31:0] b_q, m_q;
  logic [15:0] true_approx, true_cost;
  logic        truly_approximated;
  assign mapped_ok = (kinds_mapped > msg_kinds) ? msg_kinds : kinds_mapped;
  assign true_approx = msg_kinds - mapped_ok;
  assign approximated = (A_BRIDGE_IS_A_WIRE != 0) ? 16'd0 : true_approx;
  assign b_q = {16'd0, blocks} * {16'd0, block_cost};
  assign true_cost = (b_q > 32'd9999) ? 16'd9999 : b_q[15:0];
  assign bridge_total = (A_BRIDGE_IS_A_WIRE != 0) ? 16'd0 : true_cost;
  assign m_q = (msg_kinds == 16'd0) ? 32'd100
             : (({16'd0, mapped_ok} * 32'd100) / {16'd0, msg_kinds});
  assign mapped_pct = m_q[15:0];
  assign bridge_is_cheap = (approximated == 16'd0) && (bridge_total == 16'd0)
                         && (msg_kinds != 16'd0);
  assign truly_approximated = (true_approx != 16'd0) || (true_cost != 16'd0);
  assign bridge_err = evaluate && truly_approximated && bridge_is_cheap;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_approximated <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_approximated) n_approximated <= n_approximated + 8'd1;
    end
  end
endmodule

Fourteen message kinds with nine mapped is five approximations and seven hundred and twenty units of bridge logic — sixty-four percent translated — and the a-bridge-is-a-wire view reports a wire.

The third case is the one that makes the section's claim precise. Every kind translated and blocks still needed is a complete translation that costs seven hundred and twenty units of logic — because even a one-to-one mapping needs buffering, credit conversion, ordering maintenance and the state to track outstanding transactions across the boundary. A perfect mapping is not a wire either.

The last case separates the two costs the model tracks. Approximations with no logic counted still reports five, because the approximations are the correctness argument and the block count is the schedule. A bridge with few blocks and many approximations is cheap to build and hard to be sure of, which is the more dangerous of the two shapes.

The fifth case is the far end. Nothing translated at all is a protocol conversion rather than a bridge, and the distinction is worth keeping: a bridge preserves semantics message by message, and a conversion re-implements them.

The degenerate case bounds it: bridge blocks on a comparison with no message set named is a bridge nobody has specified, and both views decline to call it cheap.

14. RTL 10 — A CXL-Against-CHI Comparison Assembled

Nine sections of inputs. This one puts them in one place and makes the confident answer visible as what it is: one bit of six.

// RTL 10 - a CXL-against-CHI comparison assembled. Nine sections of inputs,
// one summary. "They are both coherent" is bit 0: true of both, and one
// sixth of a comparison.
module comparison_signoff #(parameter int COHERENT_IS_THE_ANSWER = 0) (
  input  logic clk, rst_n,
  input  logic        evaluate,
  input  logic        coherent_named, scope_stated, symmetry_stated,
  input  logic        home_stated, transport_stated, blast_stated,
  output logic [5:0]  fail_mask,
  output logic [15:0] conditions_met, sound_pct,
  output logic        sound,
  output logic [7:0]  n_evals, n_sound, n_claimed,
  output logic        signoff_err
);
  logic [31:0] s_q;
  logic        truly_sound, claimed;
  assign fail_mask[0] = ~coherent_named;
  assign fail_mask[1] = ~scope_stated;
  assign fail_mask[2] = ~symmetry_stated;
  assign fail_mask[3] = ~home_stated;
  assign fail_mask[4] = ~transport_stated;
  assign fail_mask[5] = ~blast_stated;
  assign conditions_met = {15'd0, coherent_named} + {15'd0, scope_stated}
                        + {15'd0, symmetry_stated} + {15'd0, home_stated}
                        + {15'd0, transport_stated} + {15'd0, blast_stated};
  assign s_q = ({16'd0, conditions_met} * 32'd100) / 32'd6;
  // No clamp: conditions_met sums six one-bit values, so the quotient cannot
  // exceed a hundred and a ceiling would be unreachable code.
  assign sound_pct = s_q[15:0];
  assign truly_sound = (fail_mask == 6'd0);
  // The both-are-coherent view reads bit 0 and stops.
  assign claimed = (COHERENT_IS_THE_ANSWER != 0) ? coherent_named : truly_sound;
  assign sound = claimed;
  assign signoff_err = evaluate && !truly_sound && claimed;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_evals <= 8'd0; n_sound <= 8'd0; n_claimed <= 8'd0;
    end else if (evaluate) begin
      n_evals <= n_evals + 8'd1;
      if (truly_sound) n_sound <= n_sound + 8'd1;
      if (claimed)     n_claimed <= n_claimed + 8'd1;
    end
  end
endmodule

The stimulus walks all six bits one at a time. When both have been named as coherency protocols and any one of the other five fails, the assembled model reports that the comparison is not sound and the both-are-coherent view reports a comparison.

Across the eight evaluations, the assembled model calls one comparison sound and the both-are-coherent view calls six of them a comparison.

The bit order is by how much of the difference each condition carries, and the first of them carries most of it: scope decides the latency tier, the home's usefulness, the transport's necessity and the failure boundary's existence all at once. The rest are the consequences, ordered by how far from the geometry they sit.

"They are both coherent" is bit 0, and it is more true than the last chapter's opening. Both sides really are coherency protocols with homes, snoops, states and ordering — which is what makes this the harder comparison and the weaker stopping point. The shared property is genuinely shared and settles nothing, because coherence is a contract about what agents observe and every difference in this chapter is about the cost of honouring it across a given distance.

The five other bits fail independently. Scope can be stated with no symmetry, which is the most common half-answer: somebody who knows CXL crosses a package and has not noticed the host is privileged. Symmetry can be stated with no home placement. The home can be placed by somebody who has not thought about the transport underneath it. And the failure boundary is almost never in the answer at all, which is the same omission 28.1 section 10 found in the other comparison — failure semantics are the last thing a protocol answer reaches and the first thing a system architect needs.

Figure 4 — the mask ordered by how much of the difference each condition carries, and the first decision carries most of it. Scope is upstream of everything else on the diagram: it decides the latency tier, whether a distributed home is even available, whether framing exists, and whether the failure boundary is a question at all. An answer that gets the first box right and stops still has more of the comparison than one that gets the last four right without it — which is unusual among this module's masks and is why this one is not ordered by visibility.

15. Quantitative Reasoning

Four agents of sixteen across a boundary, and the domain's worst-case reach becomes two hundred and fifty nanoseconds for all sixteen — one agent is enough.

Two thousand four hundred units of asking for sixty of two hundred operations, with seventy percent of the traffic self-directed and the claim of symmetry false anyway.

Sixty-four addresses on one home at a host round trip, against eight homes at a fabric hop — a factor of sixteen on the same address space.

A hundred flits carrying two thousand bytes across six thousand four hundred bytes of link, and a smaller flit doubling the count rather than reducing the padding.

A quarter of the domain in one blast radius and four recovery paths of six unanswered, on a failure the mesh side never sees.

Twenty-four agents of thirty-two with nowhere to attach, and four rather than a hundred when the directory binds before the ports.

Three states of seven with nowhere to map and six thousand units of conservative fallback across two thousand lines — at zero cost on an idle cache and unchanged as a protocol fact.

Two thousand and seventy nanoseconds of misplacement for nine agents that belonged on the die, and none at all when every agent had to be off it.

Five approximations of fourteen message kinds, plus seven hundred and twenty units of logic that a perfect mapping would still need.

One comparison of eight sound; the both-are-coherent view counts six. The assembled model's summary, and the chapter's.

16. Assertions

The testbenches carry 544 checks across ten models.

Every output of every model is asserted as a value, in both builds. The output listing step reported nothing on either testbench, the fifth chapter running.

Both builds are asserted on every degenerate case. A domain with no agents, a comparison with no operations, an address space with no addresses, addresses with no home, a transport with no messages, a fabric with no width declared, a fabric with no flit size, an uncounted domain, a comparison with no agents wanted, a state set with nothing in it, and a bridge with no message set named.

Every clamp that an input can reach is driven past its limit exactly once. More on-die agents than exist, more asking than operations, an asking cost that saturates, more homes than addresses, a per-home figure that saturates, flit and beat counts that both saturate, more agents across the boundary than the domain has, more recovery handlers than paths, more agents wanted than the mesh holds, more states mapped than exist, a fallback cost that saturates, more off-die agents than exist, a misplacement cost that saturates, more message kinds translated than exist, and a bridge cost that saturates.

Every percentage whose numerator is clamped is asserted both in the case that over-claims and in the degenerate case — the second half of that sentence is new to this chapter and section 17 explains why.

Every error output is checked in both directions in every case. Section 5's second, third and fourth cases, section 6's second and fourth, section 7's second, third and sixth, section 8's second, third and fourth, section 9's second, section 10's second and fourth, section 11's second, third and fourth, section 12's second, third, fourth and sixth, and section 13's second and fourth exist to assert the quiet half. Each is a case where the both-are-coherent view is right, and a model that alarmed on them would be unusable.

17. Mutation Testing

116 mutations, 116 killed. Fifty-five against the first testbench, sixty-one against the second. The first run killed a hundred and twelve and left four.

Two of the four survivors were one new rule. A percentage's zero-guard — the value it reports when its denominator is zero — is only tested if the degenerate case asserts that percentage, and in two models the degenerate case asserted every other output and not the percentage. That is the mirror of the rule 27.9 extracted for the over-claim case, and together they make one instruction: a percentage needs asserting at both ends — where its numerator is clamped and where its denominator is zero. Applying only the first half left exactly half the guards untested across five chapters.

The other two were dominated clamps that domcheck.py cannot see, and they are worth recording because they show the check's boundary. Its rule is that a percentage's numerator is a minimum against its denominator; these two were bounded by a different relationship — an efficiency whose numerator is at most its denominator because every message costs at least one flit, and a quotient of a constant that cannot exceed the constant. Both are unreachable ceilings and neither matches the pattern, which means the campaign is still doing work the static check cannot, and that the script's value is in the class it covers rather than in completeness.

Both were removed rather than tested, as the batch's standing rule requires, and each carries a comment saying why there is no clamp there.

18. Verification Strategy

Ask where the coherency domain stops. Section 5. Everything else is downstream of the geometry.

Ask whether the agents are peers. Section 6. One operation an agent cannot issue alone makes the answer no.

Ask where the home sits and how many there are. Section 7. The distance is the cost and the count is the scaling.

Ask what the transport is. Section 8. Framing is a cost a mesh does not have and a link cannot avoid.

Ask what happens when one side disappears. Section 9. A mesh has no reason to contain that answer and a link-attached domain must.

Ask which limit binds — ports or directory entries. Section 10. The one that is easy to make large is the one that gets quoted.

Ask which states have no equivalent, and what a line held in the fallback costs. Section 11.

Ask whether the agent could have been on the die at all. Section 12. If it could not, the comparison is not a choice.

And if the system has both, ask how many message kinds are approximated. Section 13. The approximations are the correctness argument.

19. Synthesis and Implementation Reality

A CHI mesh is laid out with the die. Its topology, its home placement and its capacity are floorplan decisions, which is why "add another agent" is a tape-out and not a configuration.

A CXL host port carries a home agent's worth of work for its devices — 27.8 section 8's directory, extended across a link — which is the silicon consequence of section 7's centralised home.

The flit layer is where section 8's framing lives, and it is also where interleaving between CXL.io, CXL.cache and CXL.mem happens, so the padding argument interacts with the arbitration between protocols.

A CHI-to-CXL bridge is a real product, present in every server SoC with CXL ports, and it is where section 13's approximations are made — usually conservatively, which is correct and costs performance.

And chiplets are where the comparison is genuinely live. Multiple dies in one package, coherent between them, is a geometry that is neither clearly on-die nor clearly across a system boundary, and it is the reason die-to-die interconnects exist as a third category rather than as a choice between these two.

20. Silicon Observability

Free, and from the floorplan. Which agents are on the die. Section 5's whole argument, and it is decided before any code runs.

Free, and on paper. The mesh's agent capacity and the host's port count. Section 10's two limits, one of which is usually quoted and one of which is usually not.

Cheap. The host's directory entries and per-agent requirement. Section 10's binding constraint.

Cheap. The flit size and the fabric width. Section 8.

Moderate. The fraction of an agent's operations that require the host. Section 6 needs a trace that distinguishes them, which not every agent's counters do.

Moderate. Measured reach from an agent to its home. Section 7, from the same observation points 27.9 section 11 argues for.

Expensive. The state-mapping table between two protocols. Section 11 is a specification comparison rather than a measurement.

Expensive, and usually undocumented. Which bridge messages are approximated and how. Section 13, and it is the question a bridge's users most need answered.

21. Debug Lab

A coherent accelerator behind a CXL port performs unlike the same design's on-die predecessor.

Step 1 — check where the agent is. Section 5. If it moved across a package boundary the tier changed, and that is the answer most of the time.

Step 2 — check what fraction of its operations need the host. Section 6. An agent written for a symmetric fabric will issue operations that now cost a round trip.

Step 3 — check where the home for its addresses is. Section 7. A distributed home on the predecessor and a centralised one now is a second latency change on the same access.

Step 4 — check the state it is holding lines in. Section 11. A state that fell back to something conservative produces extra transactions with nothing else wrong.

Step 5 — check the bridge. Section 13. An approximated message kind is a place where the behaviour is legitimately different from the mesh's.

Step 6 — only then look at the link. 26.6. Bandwidth and retries are real and they are not usually the difference between these two designs.

Steps 1 to 3 are reads from a floorplan and a specification, which makes the first half of this free.

22. Design Review

Where does the coherency domain stop, and how many agents are across the boundary?

Are the agents peers, and which operations can one not issue alone?

Where are the homes, how many are there, and how far is a request from one?

What is the transport, and what does framing cost the small messages?

What happens when an agent across the boundary disappears, and how many recovery paths are enumerated?

Which binds first — the host's ports or its directory entries?

Which states have no equivalent, and what does a line held in the fallback cost?

Could this agent have been on the die at all?

If the system has both, how many message kinds does the bridge approximate?

23. How This Appears In Real Engineering

The comparison is made by somebody who knows one of the two well, and the missing half is always the same shape.

The most common is an on-die background. Somebody who has built a CHI agent assumes symmetry, because on a mesh it is true, and designs a CXL device that issues operations it is not entitled to issue. The discovery is late, in integration, and the fix is architectural.

The second is a host background. Somebody who has built the host side assumes the home is near, because on their side it is, and sizes a device's outstanding-transaction capacity for a mesh's round trip. The device stalls under load with nothing wrong anywhere.

The third is section 9 and it is the one that reaches production. A link-attached coherent design ships with no answer for a device disappearing, because the predecessor was on-die and the question had never come up. The failure is rare, and when it happens the host's directory holds lines for an agent that is gone.

The fourth is section 11 and it is quiet. A state falls back conservatively at the bridge, every transaction is correct, and the design runs at a fraction of its predecessor's hit rate for a reason that shows up in no error counter.

The pattern is that each protocol's assumptions are invisible from inside it, and the comparison's value is almost entirely in naming the assumption the reader is not carrying.

24. Common Misconceptions

"They are both coherent." True, and one condition of six. Section 14.

"CXL is CHI over a link." The scope, the symmetry, the home placement and the failure model all differ. Section 5.

"The agents are peers." The host is the coherency master. Section 6.

"A home is a home." One is a few hops away and one is at the host. Section 7.

"The transport is a detail." Framing is a cost a mesh does not have. Section 8.

"One coherency domain, one fate." Not across a package boundary. Section 9.

"They both scale." One by tiling and one by provisioning, and neither is a configuration change. Section 10.

"The states map across." Three of seven do not. Section 11.

"CXL replaces CHI." For agents that could never have been on the die, there was nothing to replace. Section 12.

"The bridge is a wire." Five approximations and seven hundred and twenty units of logic. Section 13.

25. Interview Reasoning

"What is the difference between CXL and CHI?" Both are coherency protocols, so the difference is not coherence — it is where the domain stops. CHI is an on-die or on-package fabric; CXL crosses a package boundary over a serialised link. From that follow all the rest: CHI's agents are symmetric peers while CXL's host is the coherency master, CHI distributes the home across the mesh while CXL's home is the host, CHI runs on a wide credited network on chip while CXL runs in fixed-size flits, and — the one people forget — CHI's agents share a fate and CXL's do not, so a link-attached domain needs recovery paths a mesh has no reason to contain.

"Is CXL just CHI over a link?" No, and the symmetry is the clearest reason. On CHI two fully-coherent request nodes have the same rights over memory. On CXL the device participates in a protocol the host masters — it cannot snoop the host, and the operations a peer would issue are operations it has to ask for.

"Why does the home's placement matter?" Because it is where a request is ordered. CHI puts the home near the memory it orders, so a request travels a few hops. CXL's home is the host, so a device's request travels to the host and back regardless of where the memory is — and that number does not improve by adding devices.

"What breaks that would not break on a mesh?" An agent disappearing. On a die, half the domain cannot vanish while the other half runs. Across a link it can, and the host's directory still records the lines it held — so there has to be a timeout, a recovery and a policy, and those have no analogue on the mesh.

"When would you use each?" It is usually not a choice. If the agent can be on the die, it belongs on the mesh and putting it behind a link costs the tier. If it cannot — another package, another vendor, a card — then CHI was never available and CXL is the only coherent option. The interesting middle is chiplets, which is why die-to-die interconnects are a third category.

"Your SoC has both. What sits between them?" A bridge, and it is not a wire. Message kinds with an equivalent are translated; the ones without are approximated conservatively, and those approximations are where a bridge's correctness argument and most of its performance loss both live.

26. Exercises

1. A domain of 24 agents with 19 on the die, on-die reach 18 ns, off-die 300 ns. Compute the off-die fraction and the worst-case reach. What does the worst case become with 23 on the die?

2. 500 operations, 90 needing the host at 55 units each. Compute the asking total and the self-directed fraction. At what count does the claim of symmetry become true?

3. 256 addresses. Compute the addresses per home and the reach at 1, 4 and 16 homes, with a hop of 12 ns and a host round trip of 280 ns. Where does the curve stop paying?

4. 400 messages of 30 bytes. Compute the flits at flit sizes of 16, 32 and 128 bytes, and the efficiency at each. Which flit size would you choose and what does it cost the large messages?

5. A 40-agent domain with 12 across a boundary, 9 recovery paths, 4 handled. Compute the blast radius and the unhandled count. Which of the 9 would you write first?

6. A host with 24 ports where each agent needs 700 directory entries against a budget of 9,999. Compute the attachable count. Which limit binds, and what would you change?

7. 11 states with 6 mapped, 5,000 lines at 4 units of fallback each. Compute the loss and the cost. What is the cost at 50,000 lines, and is it the same protocol problem?

8. 20 agents of which 6 must be off die, on-die 18 ns and off-die 280 ns. Compute the misplacement. What does it become if 14 must be off die?

9. A bridge with 22 message kinds, 15 mapped, 9 blocks at 150 units. Compute the approximations and the cost. Which number would you want documented as a user of the bridge?

10. Extend the assembled model with a seventh bit for a condition this chapter does not cover. Justify its position using the rule that the ordering is by how much of the difference each condition carries.

27. Summary

The difference is where the coherency domain stops, and one agent across a package boundary sets the worst-case reach for all of them.

CHI is symmetric and CXL is not — the host is the coherency master, and one operation an agent cannot issue alone makes the peer claim false.

CHI distributes the home and CXL's home is the host, so a request's distance to its serialisation point is a design parameter on one and a fixed geometry on the other.

A mesh and a link are different transports, and framing is a cost the mesh does not have and the link cannot avoid.

A package boundary is a failure boundary, and a domain that spans one needs recovery paths a mesh has no reason to contain.

They scale by different mechanisms — tiling against provisioning — and neither is a configuration change.

The state sets do not map one to one, and a line held in the fallback pays for the difference on every access.

Each wins where its boundary is, and for an agent that could never be on the die the comparison was never a choice.

A bridge is a translation, and the approximations are where its correctness argument lives.

Six bits, and "they are both coherent" is one of them. One comparison of eight is sound; the both-are-coherent view counts six.