The CXL Host

Module 2 established what CXL is and what it is for. Module 3 turns to structure, and it starts with the host — because almost every obligation Chapter 2.5 named is enforced on the host side.

The reason is not hierarchy. It is information: the host is the only agent that knows the whole system. A device knows its own memory and its own caches. Only the host knows the complete address map, every agent that might hold a copy of a line, and which requests are outstanding across all of them. Responsibilities follow information, and that is what this chapter is about.

1. The One-Sentence Model

The CXL host owns the three things only a system-wide view makes possible — decoding an address to the target that owns it, tracking every request it has issued until the matching completion arrives, and serialising conflicting accesses to a line so that coherence holds — and it is the enforcement point for all of them because no device has the information to do any of them.

2. What This Chapter Owns

Message formats, opcodes and encodings are not this chapter's subject. Structure is.

3. Three Responsibilities, One Reason

The third column is the whole argument. These are not tasks assigned to the host by convention — they are tasks that require information only the host has, and a design that pushes any of them toward the device is pushing a decision to an agent that cannot make it correctly.

The practical corollary is worth stating early: when a CXL system misbehaves, the host is where the evidence is. Section 11's counters exist for that reason.

4. Responsibility 1 — Decoding the Address

A physical address arriving at the host must be resolved to exactly one target: local DRAM, host-managed device memory living on a CXL device, or MMIO.

This is the responsibility that makes CXL memory system memory rather than a device buffer. Chapter 2.1 made the definitional claim; address decode is where it becomes true. A range in the system map that happens to be serviced across a link is memory that software addresses like any other, and the decode is what places it there.

The information argument is clearest here. A device knows the ranges it hosts. It has no idea what else exists, so it cannot answer "does this address belong to me or to something else?" — only "is this in my range?", which is a different and insufficient question when ranges can overlap.

5. Responsibility 2 — Tracking What Is Outstanding

A host that issues a request over a link must, when a completion arrives, know which request it belongs to. That means a tag, a table, and a lifetime rule.

The lifetime rule is the part that goes wrong, and it goes wrong in one specific way: a tag is not free when the request is sent, and it is not free when the completion arrives at the link — it is free when the completion has been consumed. Freeing early makes the tag available for reuse while a completion for it is still in flight, and the host then routes returning data using an entry that belongs to a different request.

Section 9 measures both lifetimes side by side.

6. Responsibility 3 — Being the Coherence Point

Chapter 1.7 established the single-writer invariant: at most one agent may hold a writable copy of a line, and no readable copies may coexist with it. Something has to enforce that, and the enforcing agent is called the home agent.

Its job is serialisation. Two agents asking for the same line at once must be ordered, one of them must wait, and any existing copies must be invalidated before a writer is granted. That waiting is not an implementation weakness — it is the cost of agreement, and Section 10 measures it.

Why it must be the host. Enforcing the invariant requires knowing every agent that might hold a copy. A device knows its own caches; the host knows all of them. There is no device-side implementation of this that is correct, only ones that work while nothing else caches the line.

7. RTL 1 — The Address Decoder

Purpose

Resolve an address to exactly one owner — and detect a map that cannot do so.

// Host address decode: which target owns this physical address?
//
// The host is the only agent that knows the whole map, which is why decode
// lives here and not in the device. GENERIC teaching model -- the ranges are
// teaching values and no CXL address-map mechanism is modelled.
//
// OVERLAP=1 shifts the HDM base so it overlaps local DRAM, to show what a
// programming error in the map does.
module addr_decode #(
  parameter int unsigned OVERLAP = 0
) (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        req_valid,
  input  logic [31:0] addr,
  output logic        to_local,
  output logic        to_hdm,      // host-managed device memory
  output logic        to_mmio,
  output logic        unmapped,
  output logic        overlap_err  // two decoders claimed the same address
);
  localparam logic [31:0] LOCAL_BASE = 32'h0000_0000, LOCAL_TOP = 32'h3FFF_FFFF;
  localparam logic [31:0] HDM_BASE   = (OVERLAP != 0) ? 32'h2000_0000 : 32'h4000_0000;
  localparam logic [31:0] HDM_TOP    = 32'h7FFF_FFFF;
  localparam logic [31:0] MMIO_BASE  = 32'hF000_0000, MMIO_TOP = 32'hFFFF_FFFF;
 
  logic hit_local, hit_hdm, hit_mmio;
 
  assign hit_local = (addr >= LOCAL_BASE) && (addr <= LOCAL_TOP);
  assign hit_hdm   = (addr >= HDM_BASE)   && (addr <= HDM_TOP);
  assign hit_mmio  = (addr >= MMIO_BASE)  && (addr <= MMIO_TOP);
 
  // Priority makes the OUTPUT one-hot even when the ranges overlap. That is
  // the trap: the decode looks correct while the map is wrong.
  assign to_local = req_valid &&  hit_local;
  assign to_hdm   = req_valid && !hit_local && hit_hdm;
  assign to_mmio  = req_valid && !hit_local && !hit_hdm && hit_mmio;
  assign unmapped = req_valid && !(hit_local || hit_hdm || hit_mmio);
 
  // Detect the MAP error, not the output. This is the check people omit.
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) overlap_err <= 1'b0;
    else if (req_valid && (({2'b0,hit_local} + {2'b0,hit_hdm} + {2'b0,hit_mmio}) > 3'd1))
      overlap_err <= 1'b1;
  end
endmodule

unmapped must exist as an explicit output. An address matching nothing is a real condition — a software bug, a partially configured map, a device removed while software still holds a pointer — and a decoder without an unmapped output either drops it silently or lets it fall into whichever range the default branch happens to select. Both are worse than an error.

The distinction between hit_* and to_* carries the whole lesson. The to_* outputs are one-hot by construction because of the priority chain, so an overlapping map produces a perfectly well-formed decode. Only hit_* — the raw range matches — reveals that the map is broken.

Simulation evidence

Two decoders, identical stimulus, one with a 512 MB overlap between local DRAM and HDM:

=== EXP1: host address decode, clean map vs a 512MB overlap ===
  0x00001000   clean: L=1 H=0 M=0 U=0 | overlapping: L=1 H=0 M=0
  0x25000000   clean: L=1 H=0 M=0 U=0 | overlapping: L=1 H=0 M=0
  0x50000000   clean: L=0 H=1 M=0 U=0 | overlapping: L=0 H=1 M=0
  0xF0000100   clean: L=0 H=0 M=1 U=0 | overlapping: L=0 H=0 M=1
  0x80000000   clean: L=0 H=0 M=0 U=1 | overlapping: L=0 H=0 M=0
 
  clean map overlap_err=0   overlapping map overlap_err=1

Every decoded output is identical between the two configurations. The overlapping map produces exactly the same routing on all five addresses — including 0x25000000, which sits inside the overlap. The priority chain resolves it to local, which is a defensible answer and possibly the wrong one.

Only overlap_err distinguishes them, and it exists because the module checks the raw range matches rather than the decoded outputs. This is a general and under-used technique: check the input condition that should be impossible, not the output that priority logic has already made safe. Output-based assertions on a priority decoder are guaranteed to pass and therefore worthless.

Row five is the second point. 0x80000000 is above HDM's top and below MMIO's base — a genuine hole. The clean decoder reports unmapped; the overlapping one reports nothing at all on any output, which in a design without an unmapped signal is a request that vanishes.

8. Why Decode Placement Is an Architectural Choice

Worth pausing on, because it is the chapter's thesis in miniature.

Decode could in principle be distributed: each target could answer "is this mine?" and the fabric could route to whoever says yes. That design works exactly as long as no two targets ever claim the same address — and there is no agent in it that can detect when two do, because each one only sees its own answer.

Centralising decode at the host means one agent holds the whole map, so overlaps are detectable, holes are detectable, and there is a single place to instrument. The cost is that the host must be told the map, which is why enumeration and configuration matter as much as they do, and why Chapter 2.4's inherited enumeration was so valuable: the mechanism for telling the host what exists already existed.

9. RTL 2 — The Outstanding-Transaction Table

Purpose

Match every completion to the request that produced it, and never reuse a live tag.

// Track every request the host has issued and not yet completed.
//
// The host must match an arriving completion to the request that produced it,
// which means a tag, and tags are a finite resource. Freeing a tag before the
// completion is consumed is the classic defect; EARLY_FREE=1 shows it.
//
// GENERIC teaching model, NOT a CXL tag or transaction-ID mechanism.
module outstanding_table #(
  parameter int unsigned NTAG      = 8,
  parameter bit          EARLY_FREE = 1'b0
) (
  input  logic                    clk,
  input  logic                    rst_n,
  input  logic                    issue,
  input  logic [31:0]             issue_addr,
  input  logic                    cmpl_valid,
  input  logic [$clog2(NTAG)-1:0] cmpl_tag,
  output logic                    issue_ok,
  output logic [$clog2(NTAG)-1:0] issue_tag,
  output logic [31:0]             cmpl_addr,
  output logic                    cmpl_matched,
  output logic                    orphan_cmpl_err,  // completion for a free tag
  output logic                    tag_reuse_err,    // allocated a live tag
  output logic [7:0]              inflight_q
);
  logic [NTAG-1:0] busy_q;
  logic [31:0]     addr_q [NTAG-1:0];
  integer          i;
  logic            found;
 
  // Lowest free tag.
  always_comb begin
    issue_tag = '0; found = 1'b0;
    for (i = NTAG-1; i >= 0; i = i - 1)
      if (!busy_q[i]) begin issue_tag = i[$clog2(NTAG)-1:0]; found = 1'b1; end
  end
  // No free tag means NO ISSUE. Backpressure, not reuse.
  assign issue_ok = issue && found;
 
  assign cmpl_addr    = addr_q[cmpl_tag];
  assign cmpl_matched = cmpl_valid && busy_q[cmpl_tag];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      busy_q <= '0; inflight_q <= '0;
      orphan_cmpl_err <= 1'b0; tag_reuse_err <= 1'b0;
      for (i = 0; i < NTAG; i = i + 1) addr_q[i] <= '0;
    end else begin
      if (issue_ok) begin
        if (busy_q[issue_tag]) tag_reuse_err <= 1'b1;
        busy_q[issue_tag] <= 1'b1;
        addr_q[issue_tag] <= issue_addr;
      end
      if (cmpl_valid) begin
        // A completion for a tag nobody owns means the tag was freed too soon
        // -- or a duplicate arrived. Either way the host is about to route
        // data using a stale address.
        if (!busy_q[cmpl_tag]) orphan_cmpl_err <= 1'b1;
        if (!EARLY_FREE) busy_q[cmpl_tag] <= 1'b0;
      end
      // BUG shape: free on ISSUE-side accounting rather than on completion.
      if (EARLY_FREE && issue_ok) busy_q[issue_tag] <= 1'b0;
 
      inflight_q <= inflight_q + {7'b0, issue_ok} - {7'b0, cmpl_matched};
    end
  end
endmodule

Backpressure is the design decision here. issue_ok requires a free tag, so a full table stalls issue rather than reusing. That is the correct trade every time: a stall is a performance cost and a reused tag is a data-corruption cost, and they are not comparable quantities.

Timing. The free-tag search is a priority encoder across NTAG bits, combinational and in the issue path. At larger tag counts this becomes real, and the standard answer is a registered free-list rather than a search — an implementation change with no behavioural difference, which is why it is left as a search here.

Simulation evidence

Both lifetimes instantiated on identical stimulus. Four requests issued, no completions:

=== EXP2: tags -- freeing on completion vs freeing on issue ===
  after 4 issues, no completions:
    free-on-completion : inflight=4 next tag=4
    free-on-issue      : inflight=4 next tag=0  <-- tags handed back

next tag=0 after four outstanding requests is the defect in one number. The early-free table has already returned every tag to the pool while all four requests are still in flight, so the next issue will reuse tag 0 — which currently belongs to a request whose completion has not arrived.

Then a completion arrives for tag 2:

  completion for tag 2 (sampled while the completion is present):
    free-on-completion : matched=1 addr=40000080 orphan=0
    free-on-issue      : matched=0 addr=00000000 orphan=0  <-- unmatched
 
  one cycle later (the error flags are registered):
    free-on-completion : inflight=3 orphan_cmpl_err=0
    free-on-issue      : inflight=4 orphan_cmpl_err=1  <-- cannot place the data

The correct table resolves tag 2 to 0x40000080 — the third request's address, which is exactly right — and retires it. The early-free table has no entry: matched=0, the returned address is 0x00000000, and orphan_cmpl_err asserts a cycle later.

Read that address carefully. 0x00000000 is not a null pointer or an obvious failure; it is a plausible-looking address in local DRAM. A host that trusts an unmatched lookup writes returning device data into low physical memory. The error flag is what converts silent corruption into a reported fault, and it costs one comparison.

Finally, the pool is deliberately exhausted:

=== EXP3: exhausting the tag pool ===
  10 issues into 8 tags: issue_ok=0 inflight=8
  -> the host refuses to issue rather than reusing a live tag

Eight in flight, ten attempted, issue_ok low. The host stalls. That is the whole answer to "what happens when you run out of tags", and any other answer is a bug.

10. RTL 3 — The Home Agent

Purpose

Serialise conflicting access to one line and enforce the single-writer invariant.

// The host's coherence point for one cache line.
//
// The home agent is what makes the host the host: it serialises conflicting
// requests to a line, snoops sharers, and only then grants.
//
//   00 I  no cached copies       01 S  shared, one or more readers
//   10 E  exclusive to one agent 11 T  transient -- snoops outstanding
//
// GENERIC pedagogical coherence model. NOT MESI, NOT MOESI, and NOT the CXL
// coherence protocol; no CXL message, opcode or state encoding is claimed.
module home_agent_line #(
  parameter int unsigned NAGENT = 4
) (
  input  logic              clk,
  input  logic              rst_n,
  input  logic [NAGENT-1:0] req_shared,
  input  logic [NAGENT-1:0] req_excl,
  input  logic [NAGENT-1:0] snoop_ack,
  output logic [1:0]        state_q,
  output logic [NAGENT-1:0] sharers_q,
  output logic [NAGENT-1:0] owner_q,
  output logic [NAGENT-1:0] snoop_out,
  output logic [NAGENT-1:0] grant_shared,
  output logic [NAGENT-1:0] grant_excl,
  output logic              busy,
  output logic              two_writers_err,
  output logic              stale_reader_err
);
  localparam logic [1:0] I = 2'b00, S = 2'b01, E = 2'b10, T = 2'b11;
  logic [NAGENT-1:0] awaiting_q, pend_excl_q;
 
  assign busy = (state_q == T);
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      state_q <= I; sharers_q <= '0; owner_q <= '0; /* ... */
    end else begin
      grant_shared <= '0; grant_excl <= '0; snoop_out <= '0;
      case (state_q)
        I: begin
          if (|req_excl) begin
            // Nobody holds it -- grant immediately, no snoops needed.
            owner_q    <= req_excl & (~req_excl + 1);   // lowest requester
            grant_excl <= req_excl & (~req_excl + 1);
            state_q    <= E;
          end else if (|req_shared) begin
            sharers_q <= req_shared; grant_shared <= req_shared; state_q <= S;
          end
        end
        S: begin
          if (|req_excl) begin
            // Must invalidate every sharer BEFORE granting. This is the
            // serialisation the home agent exists to perform.
            snoop_out   <= sharers_q;
            awaiting_q  <= sharers_q;
            pend_excl_q <= req_excl & (~req_excl + 1);
            state_q     <= T;
          end else if (|req_shared) begin
            sharers_q <= sharers_q | req_shared; grant_shared <= req_shared;
          end
        end
        E: begin
          if (|req_shared || |req_excl) begin
            snoop_out   <= owner_q;
            awaiting_q  <= owner_q;
            pend_excl_q <= (|req_excl) ? (req_excl & (~req_excl + 1)) : '0;
            state_q     <= T;
          end
        end
        T: begin
          // Nothing is granted while snoops are outstanding. That wait IS the
          // cost of coherence, and it is why the transient state exists.
          if ((awaiting_q & ~snoop_ack) == '0) begin
            sharers_q <= '0;
            if (|pend_excl_q) begin
              owner_q <= pend_excl_q; grant_excl <= pend_excl_q; state_q <= E;
            end else begin
              owner_q <= '0; sharers_q <= req_shared;
              grant_shared <= req_shared;
              state_q <= (|req_shared) ? S : I;
            end
            awaiting_q <= '0; pend_excl_q <= '0;
          end else begin
            awaiting_q <= awaiting_q & ~snoop_ack;
          end
        end
      endcase
      // ... invariant checks ...
    end
  end
endmodule

The T state is the entire design. Without it there is no place to be while snoops are outstanding, and a design with no transient state has only two options: grant immediately, which breaks the invariant, or block the whole agent, which serialises every line against every other. The transient state is what makes "wait for exactly this line's snoops" expressible.

awaiting_q & ~snoop_ack and not a counter. A vector says which agents have not acknowledged; a counter says only how many. When an acknowledgement goes missing, the vector names the agent and the counter does not, and that difference is the difference between a five-minute debug and a week.

req_excl & (~req_excl + 1) isolates the lowest set bit — the standard idiom, and here it is what guarantees that two simultaneous writers produce exactly one owner.

Simulation evidence — the invalidation round trip

Three agents share the line, then a fourth asks to write:

=== EXP4: the home agent serialising a conflicting write ===
  cyc A: state=01 sharers=0111 grant_shared=0111
  cyc B: state=11 snoop_out=0111 busy=1  <-- invalidate before granting
  cyc C: state=11 busy=1 grant_excl=0000
  cyc D: state=11 busy=1 grant_excl=0000
  cyc E: state=10 busy=0 grant_excl=1000 owner=1000 sharers=0000
  cyc F: state=10 owner=1000 sharers=0000
  -> home agent held the writer for 3 cycles; max busy run=3

Three cycles between the request and the grant, and nothing illegal happened in any of them. Cycles B through D are the transient state: snoops issued to all three sharers, acknowledgements collected one per cycle, no grant. At cycle E the last acknowledgement lands, sharers clears to 0000 and owner becomes 1000 in the same transition — the invariant never has a window in which a writer and a reader coexist.

This is Chapter 1.7's coherence cost measured at the agent that pays it. Three cycles here is a teaching number, but the shape is not: the wait scales with the number of sharers, so the most-shared lines are the most expensive to write. That is why false sharing is a performance disaster and not merely an inefficiency.

Simulation evidence — two simultaneous writers

=== EXP5: two agents ask to write the same line in the same cycle ===
  both asked: state=11 snoop_out=1000 busy=1 (the current owner is snooped first)
  owner acks: state=10 owner=0010 grant_excl=0010
  settled   : owner=0010 sharers=0000 two_writers_err=0
  -> of two simultaneous writers exactly one owns the line

Two requesters, one owner, and two_writers_err never asserts. The prior owner is snooped, its acknowledgement arrives, and the lowest of the two requesters is granted. The loser is not refused — it is deferred, which is the difference between serialisation and arbitration and the reason the home agent needs pend_excl_q rather than simply dropping what it cannot serve.

11. RTL 4 — Host Counters

Purpose

Make the host's behaviour visible, since it is where the evidence is.

// Host-side visibility: where requests went and how long the coherence point
// held them. GENERIC teaching model.
module host_counters (
  input  logic        clk,
  input  logic        rst_n,
  input  logic        acc_local,
  input  logic        acc_hdm,
  input  logic        ha_busy,
  output logic [15:0] n_local_q,
  output logic [15:0] n_hdm_q,
  output logic [15:0] ha_busy_cycles_q,
  output logic [15:0] ha_max_busy_q      // worst single stall, not the mean
);
  logic [15:0] run_q;
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_local_q <= '0; n_hdm_q <= '0; ha_busy_cycles_q <= '0;
      ha_max_busy_q <= '0; run_q <= '0;
    end else begin
      if (acc_local) n_local_q <= n_local_q + 16'd1;
      if (acc_hdm)   n_hdm_q   <= n_hdm_q   + 16'd1;
      if (ha_busy) begin
        ha_busy_cycles_q <= ha_busy_cycles_q + 16'd1;
        run_q            <= run_q + 16'd1;
        if (run_q + 16'd1 > ha_max_busy_q) ha_max_busy_q <= run_q + 16'd1;
      end else begin
        run_q <= '0;
      end
    end
  end
endmodule

ha_max_busy_q is the counter that earns its area. Total busy cycles gives a utilisation figure; the maximum run gives the worst single stall any requester experienced. A home agent that is busy 5% of the time with a maximum run of 200 cycles is a very different system from one busy 5% of the time with a maximum run of 4, and an average cannot tell them apart. Tail latency is what users experience, and only a maximum captures it.

The measured run reported home-agent busy cycles=4 and max busy run=3, consistent with the single three-cycle invalidation in EXP4 plus one cycle in EXP5's transient — which is the check worth doing on any counter: does the total agree with the events you can count by hand?

12. Assertions

Icarus does not execute concurrent SVA, so these were not run; the table gives the procedural check.

// H1 — decode selects at most one target.
a_decode_onehot: assert property (@(posedge clk) disable iff (!rst_n)
  $onehot0({to_local, to_hdm, to_mmio}));
 
// H2 — every valid request is routed or explicitly reported unmapped.
a_decode_total: assert property (@(posedge clk) disable iff (!rst_n)
  req_valid |-> (to_local || to_hdm || to_mmio || unmapped));
 
// H3 — MAP property, not an output property: no two ranges claim an address.
// H1 passes on a broken map because priority hides the overlap; only this fires.
a_map_disjoint: assert property (@(posedge clk) disable iff (!rst_n)
  req_valid |-> $onehot0({hit_local, hit_hdm, hit_mmio}));
 
// H4 — a live tag is never reallocated.
a_no_tag_reuse: assert property (@(posedge clk) disable iff (!rst_n)
  issue_ok |-> !busy_q[issue_tag]);
 
// H5 — every completion matches a live entry.
a_cmpl_matched: assert property (@(posedge clk) disable iff (!rst_n)
  cmpl_valid |-> busy_q[cmpl_tag]);
 
// H6 — SINGLE-WRITER INVARIANT: at most one owner, ever.
a_single_writer: assert property (@(posedge clk) disable iff (!rst_n)
  $countones(owner_q) <= 1);
 
// H7 — no reader coexists with a writer.
a_no_stale_reader: assert property (@(posedge clk) disable iff (!rst_n)
  (state_q == E) |-> (sharers_q == '0));
 
// H8 — nothing is granted while snoops are outstanding.
a_no_grant_in_transient: assert property (@(posedge clk) disable iff (!rst_n)
  (state_q == T) |-> ((grant_excl == '0) && (grant_shared == '0)));
 
// H9 — LIVENESS: the transient state always terminates.
a_transient_terminates: assert property (@(posedge clk) disable iff (!rst_n)
  (state_q == T) |-> ##[1:SNOOP_TIMEOUT] (state_q != T));

H3 is the property this chapter exists to teach. H1 — the one everybody writes — passed on both maps, because a priority chain makes its output one-hot no matter how broken the ranges are. Only a property written against the raw range matches detects the overlap. Asserting the output of logic designed to guarantee that output is a tautology, and tautologies are the most comfortable kind of test to have.

H9 is the liveness property. A transient state that never exits is a line that is permanently unwritable, presenting as a hang rather than as a coherence error, and no safety property can express it.

13. Debug Lab

// The decode is one-hot, so the map must be fine.
assert property ($onehot0({to_local, to_hdm, to_mmio}));

  0x25000000   clean: L=1 H=0 M=0 U=0 | overlapping: L=1 H=0 M=0
  clean map overlap_err=0   overlapping map overlap_err=1

a_map_disjoint: assert property (req_valid |-> $onehot0({hit_local, hit_hdm, hit_mmio}));

// The request has been sent; the slot is available again.
if (issue_ok) begin
  busy_q[issue_tag] <= 1'b1;
  ...
end
if (issue_ok) busy_q[issue_tag] <= 1'b0;    // freed on issue, not completion

  after 4 issues, no completions:
    free-on-completion : inflight=4 next tag=4
    free-on-issue      : inflight=4 next tag=0  <-- tags handed back
 
  completion for tag 2:
    free-on-completion : matched=1 addr=40000080 orphan=0
    free-on-issue      : matched=0 addr=00000000 orphan=1

if (cmpl_valid) begin
  if (!busy_q[cmpl_tag]) orphan_cmpl_err <= 1'b1;
  busy_q[cmpl_tag] <= 1'b0;
end

// Wrap around when we run out. Stalling would hurt throughput.
assign issue_tag = next_tag_q;
assign issue_ok  = issue;              // always accept

=== EXP3: exhausting the tag pool ===
  10 issues into 8 tags: issue_ok=0 inflight=8
  -> the host refuses to issue rather than reusing a live tag

assign issue_ok = issue && found;      // no free tag, no issue

S: if (|req_excl) begin
     snoop_out  <= sharers_q;              // send the snoops
     owner_q    <= req_excl & (~req_excl + 1);
     grant_excl <= req_excl & (~req_excl + 1);   // and grant immediately
     state_q    <= E;
   end

  cyc B: state=11 snoop_out=0111 busy=1  <-- invalidate before granting
  cyc C: state=11 busy=1 grant_excl=0000
  cyc D: state=11 busy=1 grant_excl=0000
  cyc E: state=10 busy=0 grant_excl=1000 owner=1000 sharers=0000

S: if (|req_excl) begin
     snoop_out <= sharers_q; awaiting_q <= sharers_q;
     pend_excl_q <= req_excl & (~req_excl + 1);
     state_q <= T;                        // grant NOTHING yet
   end
T: if ((awaiting_q & ~snoop_ack) == '0) begin
     sharers_q <= '0; owner_q <= pend_excl_q;
     grant_excl <= pend_excl_q; state_q <= E;
   end

// Count the outstanding snoops; when it hits zero we are done.
if (snoop_ack_any) ack_count_q <= ack_count_q - 1;
if (ack_count_q == 0) state_q <= E;

  cyc B: state=11 snoop_out=0111 busy=1
  cyc C: state=11 busy=1 grant_excl=0000

awaiting_q <= awaiting_q & ~snoop_ack;               // idempotent
if ((awaiting_q & ~snoop_ack) == '0) state_q <= E;   // done when the set empties

14. Design Review — Reading a Host Implementation

The five defects above are what a reviewer is looking for. Reduced to questions worth asking about any host-side implementation:

On decode. Is there an explicit unmapped output, or does an unmatched address fall into a default branch? Is any assertion checking the raw range matches rather than the priority-resolved outputs? Are the range boundaries themselves in the regression, since an overlap is only visible from inside it?

On tracking. When exactly is a tag freed — on issue, on the completion arriving, or on the data being placed? What happens when the pool is empty: stall, or wrap? Is there a check that an arriving completion corresponds to a live entry, and what does the design do with the data if it does not?

On the home agent. Is anything granted while snoops are outstanding? Are sharers cleared in the same transition that installs an owner, or a cycle later? Is the outstanding set a vector or a count? Is there a timeout, and does exceeding it report which agent failed to respond?

On instrumentation. Is the maximum home-agent stall recorded, or only the total? A mean cannot distinguish a system with a 200-cycle worst-case stall from one with a 4-cycle worst case at the same utilisation, and the tail is what users experience.

One structural question above all of them. For each decision the design makes, does the agent making it have the information required? That is the test this whole chapter is built on, and it is the question that predicts which responsibilities can be moved and which cannot.

15. How This Appears in Real Engineering

Architecture

The host is where system-wide information lives, so it is where system-wide decisions belong. The practical form of that principle is a placement test: before assigning a responsibility to a device, ask what it would need to know. Address decode, completion matching and coherence enforcement all fail that test, which is why all three are host-side in every coherent interconnect, not just CXL.

RTL engineer

Four blocks and four disciplines. Decode with an explicit unmapped path. A tag whose lifetime ends at data placement, with issue backpressured when the pool is empty. A transient state that grants nothing and tracks outstanding snoops as a vector. And counters that record maxima, not just totals.

Verification engineer

The two properties that matter most are the two nobody writes. $onehot0(hit_*) catches a broken map that the usual output assertion cannot; a transient-state timeout catches a hang that no safety property can express. Stimulus requirements follow from the measured defects: addresses inside every range boundary, many outstanding requests with out-of-order completions, a deliberately exhausted tag pool, variable per-agent snoop latency, and a dropped acknowledgement.

Firmware and system software

The address map is software-programmed, which makes overlaps a software bug with a hardware symptom — and one that produces no error unless the hardware checks the raw matches. Reporting unmapped and overlap_err to firmware turns a silent data-placement fault into a boot-time diagnostic.

Performance engineering

The home agent is the serialisation point, so it is the first place to look when latency degrades under concurrency rather than under load. The signature is a rising maximum busy run with unchanged utilisation, and its usual cause is contention on a small number of hot lines — which is a data-layout problem, not a hardware one.

16. Common Misconceptions

17. Interview Reasoning

18. Summary

The CXL host owns three responsibilities, and one principle explains all of them: each requires information only a system-wide view provides.

Address decode resolves a physical address to exactly one owner, which is what makes device-attached memory system memory. It is also the only place a broken map is detectable — and measured, an overlapping map produced decoded outputs identical to a clean one on every test address. Only a property written against the raw range matches fired, because a priority chain makes the resolved output one-hot no matter how wrong the ranges are.

The outstanding-transaction table matches completions to requests, and its correctness reduces to a lifetime rule: a tag is free when the data is placed, not when the request is sent. Freeing early reported tag 0 available with four requests in flight, and resolved a completion to 0x00000000 — a plausible address in low DRAM, which is what makes the resulting corruption undebuggable. When the pool is empty the host stalls, because a stall costs cycles and a reused tag costs correctness.

The home agent serialises conflicting access to a line. Measured: three sharers, a fourth agent requesting exclusive access, three cycles in a transient state with snoops out and nothing granted, then the owner installed and the sharer set cleared in one transition. Two simultaneous writers produced exactly one owner and one deferred requester. The wait scales with sharers, which is why hot lines are expensive to write.

Two methodological points carry beyond this chapter. An assertion that the logic under test guarantees structurally is a tautology — the one-hot decode check passed on a map with a 512 MB overlap. And a maximum matters more than a mean for anything a requester waits on, because two systems at identical utilisation with worst-case stalls of 4 and 200 cycles are not the same machine.

19. What Comes Next

Chapter 3.2, The CXL Device, is the other side of the link: what the device owns, why those responsibilities are the ones local knowledge is sufficient for, and how the three device classes differ in what they must implement. Read together, 3.1 and 3.2 answer the placement question this chapter kept returning to — which decisions can move, and which cannot.

For adjacent material: Architectural Goals has the obligations the host enforces, Why Coherent Attach Matters has the single-writer invariant the home agent implements, and Relationship to PCIe has the arbitration the host applies to its side of the link. The path is on the CXL tutorials index.