Transaction Processing

24.1 got a flit to the right engine. This chapter is what the engine does with it, and the distance between those two sentences is most of the silicon in a CXL device.

A request arriving at a .mem engine has to be decoded against several address ranges, given a tag it can be identified by later, checked against everything already in flight to the same address, possibly split, issued, and then retired against a completion that may arrive in any order. Each of those is a place where a pipeline that "works" in a directed test returns the wrong data under load.

And one of them returns the wrong data silently, which is section 12.

1. The Engineering Problem — Six Places A Request Goes Wrong

A pipeline issues one per cycle only while nothing stalls. Eight stalled cycles in sixty-four is 87% throughput, not 100%. Section 5.

Nothing issues without a tag. A 32-tag pool with 32 in flight admits nothing — and a pipeline that ignores the pool issues a transaction it cannot name. Section 6.

Completions return out of order. Tag 5 arriving while the window head is tag 3 must be held, not retired — retiring it hands tag 5's data to tag 3's requester. Section 7.

A device answers to several ranges. Address 2100 is configuration and 4200 is a register; a one-range decoder routes both into memory. Section 8.

And a read must not pass a write to its own address. Section 10 holds it or forwards; an unchecked pipeline returns what was there before. Section 10.

This chapter against 24.1, stated precisely. That one owns getting a flit to an engine. This one owns what happens to a request inside one — which is why sections 6, 7, 10 and 12 have no counterpart there: they are all consequences of a request having a lifetime longer than a cycle.

2. The One-Sentence Model

A transaction pipeline is correct when requests complete, nothing issues without a tag, completions retire against their own tag rather than their arrival order, a read cannot pass a write to its address, suspect data stays marked, and the replay buffer covers a round trip — and every defect below is a pipeline through which traffic flows and answers arrive wrong.

3. What This Chapter Owns

Deferred:

4. Teaching-Model Boundary

Every model is a small synchronous block isolating one decision. A real transaction pipeline is a decoder, a tag allocator, a scoreboard, a hazard CAM, a splitter, a reorder buffer and a replay queue, and none of that is reproduced. What is reproduced is the decision each has to get right, and the shape of the mistake when it does not.

Three simplifications are worth stating. Section 5 takes stall cycles as an input rather than deriving them. Section 7 compares one arriving tag against one window head rather than maintaining a window. Section 11 counts one write in flight rather than a CAM of them. In each case the conclusion is the same and the bookkeeping is abbreviated.

Each model is built twice — a correct build and a broken build selected by a parameter. Every broken build here is a pipeline that is simpler and passes a directed test: no tag bound, no reorder buffer, one address range, no hazard check, no splitter, no poison bit. Each works when one request is in flight at a time, which is exactly how a first bring-up test drives it.

Figure 1 — Three questions before a request may issue and one after it returns. Each of the three has its own model below, and each of the three has a version that answers "yes" without checking.

5. RTL 1 — A Pipeline Issues One Per Cycle Only While Nothing Stalls

// RTL 1 - pipeline throughput. A pipeline issues one transaction per cycle only
// while nothing behind it stalls, and the initiation interval is what it delivers.
module pipeline_rate #(parameter int ASSUME_FULL_RATE = 0) (
  input  logic clk, rst_n,
  input  logic       measure,
  input  logic [7:0] depth_stages, stall_cycles, window_cycles,
  output logic [7:0] issued, initiation_interval, throughput_pct,
  output logic       at_rate,
  output logic [7:0] n_measures, n_degraded,
  output logic       stall_ignored_err
);
  logic [15:0] i_q, t_q, v_q;
  // A stalled cycle is a cycle in which nothing was issued.
  assign i_q = (ASSUME_FULL_RATE != 0) ? {8'd0, window_cycles}
             : ((window_cycles > stall_cycles)
                ? ({8'd0, window_cycles} - {8'd0, stall_cycles}) : 16'd0);
  assign issued = (i_q > 16'd255) ? 8'hFF : i_q[7:0];
  assign v_q = (issued == 8'd0) ? 16'd255
             : ({8'd0, window_cycles} / {8'd0, issued});
  assign initiation_interval = (v_q > 16'd255) ? 8'hFF : v_q[7:0];
  assign t_q = (window_cycles == 8'd0) ? 16'd0
             : (({8'd0, issued} * 16'd100) / {8'd0, window_cycles});
  assign throughput_pct = (t_q > 16'd255) ? 8'hFF : t_q[7:0];
  assign at_rate = (throughput_pct >= 8'd90);
  // A window with stall cycles in it reported as issuing every cycle.
  assign stall_ignored_err = measure && (stall_cycles != 8'd0)
                             && (issued == window_cycles);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_measures <= 8'd0; n_degraded <= 8'd0;
    end else if (measure) begin
      n_measures <= n_measures + 8'd1;
      if (!at_rate) n_degraded <= n_degraded + 8'd1;
    end
  end
endmodule

Seven measurements. A 64-cycle window.

Five degraded when the stalls are counted; one when they are not.

An initiation interval of one is a claim about the pipeline and eighty-seven percent is a claim about the traffic. The two are compatible — nothing about the pipeline changed — and only the second is a throughput number. A datasheet that quotes the first is describing a machine with no contention behind it.

Row three is the interval the datasheet would notice. At half the cycles stalled the initiation interval reads two, which looks like a pipeline change and is not; the pipeline still accepts one per cycle and is offered one every other cycle. Distinguishing the two needs the stall counter, not the rate.

Row seven is what a counter reads after an off-by-one. More stalled cycles than the window contains is impossible in a correct measurement and entirely possible after a double-count; the floor reports zero issued rather than wrapping to a large positive, which is the difference between an obviously broken measurement and a plausible one.

6. RTL 2 — Nothing Issues Without A Tag

// RTL 2 - tag allocation. Every outstanding transaction holds a tag, and a
// pipeline can have no more in flight than it has tags to name them with.
module tag_pool #(parameter int UNLIMITED_TAGS = 0) (
  input  logic clk, rst_n,
  input  logic       allocate,
  input  logic [7:0] tag_count, in_flight,
  output logic [7:0] free_tags, headroom,
  output logic       can_issue, pool_empty,
  output logic [7:0] n_allocations, n_blocked,
  output logic       tag_overrun_err
);
  // Tags in use are transactions in flight; a pipeline that ignores the pool
  // issues a transaction it cannot name.
  assign free_tags = (in_flight < tag_count) ? (tag_count - in_flight) : 8'd0;
  assign pool_empty = (free_tags == 8'd0);
  assign can_issue = (UNLIMITED_TAGS != 0) ? allocate : (allocate && !pool_empty);
  assign headroom = free_tags;
  // A transaction issued with no tag left to identify its completion.
  assign tag_overrun_err = allocate && can_issue && pool_empty;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_allocations <= 8'd0; n_blocked <= 8'd0;
    end else if (allocate) begin
      n_allocations <= n_allocations + 8'd1;
      if (!can_issue) n_blocked <= n_blocked + 8'd1;
    end
  end
endmodule

Five allocation attempts. A 32-tag pool.

Three blocked by the tag pool; none without one.

A tag is not a permission, it is a name. A transaction issued without one has no way for its completion to be matched back to it — so the failure is not that too much is in flight, it is that a returning response cannot be attributed at all. Section 7 is where that becomes visible.

Row four is the configuration a register default produces. A tag count of zero — a field never programmed — makes the pool permanently empty, and a bounded pipeline stalls forever and visibly. An unbounded one issues everything and matches nothing, which is a much harder bring-up failure than a stall.

Row five is what an occupancy counter does after a leak. More in flight than the pool holds is impossible if every tag is returned and entirely possible if one is not; the floor keeps the pool empty rather than wrapping to a large free count, which would let the pipeline issue against tags that are already outstanding.

7. RTL 3 — Completions Retire By Tag, Not By Arrival

// RTL 3 - completion reorder. Responses return out of order, and a pipeline that
// retires them in arrival order returns the wrong data to the wrong requester.
module completion_reorder #(parameter int RETIRE_IN_ARRIVAL_ORDER = 0) (
  input  logic clk, rst_n,
  input  logic       arrive,
  input  logic [7:0] arriving_tag, expected_tag, window_head,
  output logic [7:0] retire_tag, distance,
  output logic       in_order, retire_now,
  output logic [7:0] n_arrivals, n_held,
  output logic       misretire_err
);
  // A reorder buffer retires the head of the window; arrival order retires
  // whatever turned up.
  assign in_order = (arriving_tag == window_head);
  assign distance = (arriving_tag > window_head) ? (arriving_tag - window_head)
                                                 : (window_head - arriving_tag);
  assign retire_tag = (RETIRE_IN_ARRIVAL_ORDER != 0) ? arriving_tag : window_head;
  assign retire_now = arrive && ((RETIRE_IN_ARRIVAL_ORDER != 0) ? 1'b1 : in_order);
  // A completion retired against a tag other than the one the window expects.
  assign misretire_err = arrive && retire_now && (retire_tag != expected_tag);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_arrivals <= 8'd0; n_held <= 8'd0;
    end else if (arrive) begin
      n_arrivals <= n_arrivals + 8'd1;
      if (!retire_now) n_held <= n_held + 8'd1;
    end
  end
endmodule

Seven arrivals.

Four held by the reorder buffer; none retired out of order by the arrival-order design — which is the point.

Retiring by arrival hands one requester another's data. Row one is the whole failure: tag 5's completion arrives, the window is waiting for tag 3, and the arrival-order design retires tag 5's data against tag 3's entry. Nothing errors; the requester gets a plausible value from the wrong address.

Row seven is a different check and it fires in both builds. When the scoreboard expects tag 3 and the hardware's window head is tag 5, the two disagree — and misretire_err reports it regardless of which retirement policy is in use, because the disagreement is upstream of the policy. A check that only ever fires on the broken build is a check that cannot detect a broken scoreboard, and section 17 records that this case was demanded by two surviving mutations.

Rows one and three are the same distance in opposite directions, driven separately because the distance calculation has two branches and reasoning that they are symmetric is not the same as testing both.

Three cycles of wrong data with no error signal anywhere. That is the property that makes this the hardest defect in the chapter to find in simulation: the data is well-formed, the transaction completes, and only a scoreboard that knows which tag asked for what can tell that it is wrong.

8. RTL 4 — A Device Answers To Several Ranges

// RTL 4 - inbound address decode. A device answers to several ranges, and a
// decoder with one range routes configuration and register traffic into memory.
module address_decode #(parameter int ONE_RANGE = 0) (
  input  logic clk, rst_n,
  input  logic        decode,
  input  logic [15:0] addr, hdm_base, hdm_size, cfg_base, cfg_size,
  input  logic [15:0] mmio_base, mmio_size,
  output logic [1:0]  true_target, target,   // 0 none, 1 hdm, 2 cfg, 3 mmio
  output logic        in_hdm, in_cfg, in_mmio, unrouted,
  output logic [7:0]  n_decodes, n_unrouted,
  output logic        misroute_err
);
  // No separate unmapped-range guard is needed: with a size of zero the upper
  // bound equals the base, so `addr >= base && addr < base` is false for every
  // address and the range already excludes itself.
  assign in_hdm  = (addr >= hdm_base)  && (addr < (hdm_base  + hdm_size));
  assign in_cfg  = (addr >= cfg_base)  && (addr < (cfg_base  + cfg_size));
  assign in_mmio = (addr >= mmio_base) && (addr < (mmio_base + mmio_size));
  // The truth is decoded unconditionally, so the check below does not depend on
  // the decoder being tested.
  assign true_target = in_hdm ? 2'd1 : (in_cfg ? 2'd2 : (in_mmio ? 2'd3 : 2'd0));
  // A single-range decoder sends everything to host-managed memory.
  assign target = (ONE_RANGE != 0) ? 2'd1 : true_target;
  assign unrouted = (target == 2'd0);
  // An access delivered to a range that does not contain it.
  assign misroute_err = decode && (target != true_target);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_decodes <= 8'd0; n_unrouted <= 8'd0;
    end else if (decode) begin
      n_decodes <= n_decodes + 8'd1;
      if (unrouted) n_unrouted <= n_unrouted + 8'd1;
    end
  end
endmodule

Eight decodes. Memory at 0 for 1024, configuration at 2048 for 256, registers at 4096 for 512.

Two unrouted by the range decoder; none by the single-range decoder.

A misrouted access does not fail, it succeeds against the wrong thing. Row two writes a configuration value into host memory and returns a completion; nothing in the transaction is malformed and the register it was meant for keeps its old value. That is a class of bug that survives every test which does not read the register back.

Row four is the access that must be refused. An address in no range is not a memory access with an unusual offset — it is an access this device does not answer to, and returning data for it is worse than returning an error. The single-range decoder returns data.

Row seven is the state before enumeration. With no memory range programmed, every access is unrouted and a correct decoder says so; the single-range decoder routes traffic into a window that has no backing at all, which is the bring-up failure that presents as garbage reads rather than as a stall.

9. RTL 5 — Merge Writes To A Line

// RTL 5 - write merging. Several partial writes to one line become one line
// write; unmerged, each partial write is a read-modify-write of its own.
module write_merge #(parameter int NO_MERGE = 0) (
  input  logic clk, rst_n,
  input  logic       write,
  input  logic [7:0] writes, bytes_each, line_bytes,
  output logic [7:0] covered_bytes, merged_accesses, unmerged_accesses, saved,
  output logic       full_line, saves,
  output logic [7:0] n_writes, n_no_saving,
  output logic       merge_ignored_err
);
  logic [15:0] c_q, t_q;
  assign t_q = {8'd0, writes} * {8'd0, bytes_each};
  assign c_q = (t_q > {8'd0, line_bytes}) ? {8'd0, line_bytes} : t_q;
  assign covered_bytes = (c_q > 16'd255) ? 8'hFF : c_q[7:0];
  assign full_line = (line_bytes != 8'd0) && (covered_bytes == line_bytes);
  // Every unmerged partial write is a read and a write of its own.
  assign unmerged_accesses = ((({8'd0, writes} * 16'd2) > 16'd255)
                              ? 8'hFF : (writes * 8'd2));
  assign merged_accesses = (NO_MERGE != 0) ? unmerged_accesses
                         : ((writes == 8'd0) ? 8'd0 : (full_line ? 8'd1 : 8'd2));
  // No floor is needed here: merged_accesses is at most 2 and never exceeds
  // 2 * writes, so the subtraction is non-negative in the merging build, and
  // the two are equal in the unmerged one.
  assign saved = unmerged_accesses - merged_accesses;
  assign saves = (merged_accesses < unmerged_accesses);
  // Several writes to one line issued as separate read-modify-writes.
  assign merge_ignored_err = write && (writes > 8'd1)
                             && (merged_accesses == unmerged_accesses);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_writes <= 8'd0; n_no_saving <= 8'd0;
    end else if (write) begin
      n_writes <= n_writes + 8'd1;
      if (!saves) n_no_saving <= n_no_saving + 8'd1;
    end
  end
endmodule

Six write groups. A 64-byte line.

One with no saving when merging; all six without it.

Eight accesses become one, and the mechanism is a buffer rather than an algorithm. Holding partial writes until the line is covered turns four read-modify-writes into a single full-line write — and the read half of each RMW is the expensive part, because it is a memory round trip that the merged version never makes.

Row two is the honest limit. Two writes covering half a line still need the rest of the line read before it can be written back, so merging saves half the accesses rather than seven-eighths. Merging is worth doing there and it is not the same win.

Row five is what an unconfigured line size does. With no line width the merger cannot know when a line is covered, so it falls back to read-modify-write — which is correct, conservative, and exactly the behaviour that makes a missing register default look like a performance bug rather than a functional one.

Figure 3 — The same four writes, eight times the memory traffic. The dashed path's cost is the four reads, none of which the workload asked for and each of which is a full memory round trip.

10. RTL 6 — A Read Must Not Pass A Write To Its Address

// RTL 6 - the read-after-write hazard. A read to an address with a write still
// in flight must not be allowed to see what was there before it.
module hazard_check #(parameter int IGNORE_HAZARD = 0) (
  input  logic clk, rst_n,
  input  logic        check,
  input  logic        write_in_flight, forwarding_enabled,
  input  logic [15:0] read_addr, write_addr,
  output logic [1:0]  action,             // 0 issue, 1 forward, 2 hold
  output logic        hazard, stale,
  output logic [7:0]  n_checks, n_held,
  output logic        stale_read_err
);
  assign hazard = write_in_flight && (read_addr == write_addr);
  // A pipeline that does not check simply issues the read.
  assign action = (IGNORE_HAZARD != 0) ? 2'd0
                : (hazard ? (forwarding_enabled ? 2'd1 : 2'd2) : 2'd0);
  assign stale = (action == 2'd0) && hazard;
  // A read issued past a write to the same address.
  assign stale_read_err = check && stale;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_checks <= 8'd0; n_held <= 8'd0;
    end else if (check) begin
      n_checks <= n_checks + 8'd1;
      if (action == 2'd2) n_held <= n_held + 8'd1;
    end
  end
endmodule

Five hazard checks.

Two reads held; none by the unchecked pipeline.

Both terms of the hazard are required and rows three and four show why. A write to a different address is not a hazard, and a matching address with nothing in flight is not either — issuing in both cases is correct, and a check that fires on either alone would stall a pipeline for no reason.

Forwarding is the better answer where the design can afford it. Row two returns the in-flight write's data directly rather than waiting for it to land, which costs a datapath from the write buffer into the read return path — and turns a stall into a bypass. Whether it is worth the wires is a per-design question; what is not optional is that one of the two happens.

Row five is address zero, driven deliberately. A hazard model that compares against a "no address" sentinel of zero would treat this as no hazard; there is no such sentinel here, and there must not be one — every address is an address, and the in-flight flag is what says whether a write exists.

11. RTL 7 — Split What Exceeds The Maximum Payload

// RTL 7 - splitting a request. A request larger than the maximum payload becomes
// several, and the last one carries whatever is left over.
module split_transaction #(parameter int ASSUME_ATOMIC = 0) (
  input  logic clk, rst_n,
  input  logic       split,
  input  logic [7:0] request_bytes, max_payload_bytes,
  output logic [7:0] packets, last_packet_bytes, tail_fill_pct,
  output logic       efficient,
  output logic [7:0] n_requests, n_inefficient,
  output logic       oversized_err
);
  logic [15:0] p_q, l_q, f_q;
  assign p_q = (ASSUME_ATOMIC != 0) ? 16'd1
             : ((max_payload_bytes == 8'd0) ? 16'd0
                : (({8'd0, request_bytes} + {8'd0, max_payload_bytes} - 16'd1)
                   / {8'd0, max_payload_bytes}));
  assign packets = (p_q > 16'd255) ? 8'hFF : p_q[7:0];
  assign l_q = (packets == 8'd0) ? 16'd0
             : ({8'd0, request_bytes}
                - (({8'd0, packets} - 16'd1) * {8'd0, max_payload_bytes}));
  assign last_packet_bytes = (l_q > 16'd255) ? 8'hFF : l_q[7:0];
  assign f_q = (max_payload_bytes == 8'd0) ? 16'd0
             : (({8'd0, last_packet_bytes} * 16'd100) / {8'd0, max_payload_bytes});
  assign tail_fill_pct = (f_q > 16'd255) ? 8'hFF : f_q[7:0];
  assign efficient = (tail_fill_pct >= 8'd50);
  // A request beyond the maximum payload issued as one packet.
  assign oversized_err = split && (max_payload_bytes != 8'd0)
                         && (request_bytes > max_payload_bytes) && (packets == 8'd1);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_requests <= 8'd0; n_inefficient <= 8'd0;
    end else if (split) begin
      n_requests <= n_requests + 8'd1;
      if (!efficient) n_inefficient <= n_inefficient + 8'd1;
    end
  end
endmodule

Seven requests. A 64-byte maximum payload.

Four inefficient when split; two when sent atomically.

Row four is the pathological shape and it is one byte away from the best case. Sixty-five bytes becomes two packets, the second carrying a single byte — and the second packet's header costs the same as the first's. A requester that rounds its transfers to the payload size avoids this entirely, and one that does not pays a full packet for a byte.

Row one is the same effect at scale, and it is the more common one: a 200-byte transfer against a 64-byte maximum leaves an 8-byte tail. The tail fill is a property of the requester's size distribution, not of the splitter, which is why section 20 asks for it as a histogram.

Rows two and four together are the ceiling driven both ways — an exact multiple, where the ceiling and a truncating divide agree, and a remainder, where they do not. Section 17 records why both were needed.

12. RTL 8 — Poison Must Survive The Pipeline

// RTL 8 - poison must survive the pipeline. Data that arrived bad has to leave
// bad, and a stage that drops the marker delivers corruption as a good response.
module poison_propagation #(parameter int DROP_POISON = 0) (
  input  logic clk, rst_n,
  input  logic       check,
  input  logic       response_valid, poisoned_in, ecc_uncorrectable,
  output logic       poison_out, delivered_as_good, data_suspect,
  output logic [7:0] n_responses, n_poisoned,
  output logic       silent_corruption_err
);
  assign data_suspect = poisoned_in || ecc_uncorrectable;
  // A stage that does not carry the marker forward clears it.
  assign poison_out = (DROP_POISON != 0) ? 1'b0 : data_suspect;
  assign delivered_as_good = response_valid && data_suspect && !poison_out;
  // Suspect data handed to the requester with no marker on it.
  assign silent_corruption_err = check && delivered_as_good;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_responses <= 8'd0; n_poisoned <= 8'd0;
    end else if (check) begin
      n_responses <= n_responses + 8'd1;
      if (response_valid && poison_out) n_poisoned <= n_poisoned + 8'd1;
    end
  end
endmodule

Five responses examined.

Three marked poisoned on the way out; none by the dropping stage.

This is the only defect in the chapter that produces wrong data with no way to know. Every other failure here is visible in a scoreboard, a latency, or a stall counter. A cleared poison bit turns detected corruption into undetected corruption, and the requester consumes it as a normal value.

Two independent reasons set the marker and both are driven separately. Data can arrive already poisoned from the far side, or fail error correction inside this device — and a stage that carries only one of the two is half a propagation path. Section 17's mutation set injects both omissions.

Row five is the case that keeps the delivery check honest. Suspect data on a cycle with no valid response is not a delivery, so it is not a corruption — the marker is still set and nothing is handed over. A check without the validity term would fire on every idle cycle after an error.

13. RTL 9 — The Replay Buffer Must Cover A Round Trip

// RTL 9 - the replay buffer. Everything sent must be held until it is
// acknowledged, so the buffer has to cover a round trip's worth of traffic.
module replay_buffer #(parameter int NO_REPLAY_LIMIT = 0) (
  input  logic clk, rst_n,
  input  logic       size_it,
  input  logic [7:0] rtt_cycles, flits_per_cycle, buffer_entries,
  output logic [7:0] entries_needed, achievable_pct,
  output logic       sufficient,
  output logic [7:0] n_sizings, n_short,
  output logic       replay_ignored_err
);
  logic [15:0] n_q, a_q;
  assign n_q = {8'd0, rtt_cycles} * {8'd0, flits_per_cycle};
  assign entries_needed = (n_q > 16'd255) ? 8'hFF : n_q[7:0];
  // Without a replay buffer bound the link is assumed to run at full rate.
  assign a_q = (NO_REPLAY_LIMIT != 0) ? 16'd100
             : ((entries_needed == 8'd0) ? 16'd100
                : ((({8'd0, buffer_entries} * 16'd100) / {8'd0, entries_needed})
                   > 16'd100 ? 16'd100
                   : (({8'd0, buffer_entries} * 16'd100)
                      / {8'd0, entries_needed})));
  assign achievable_pct = (a_q > 16'd255) ? 8'hFF : a_q[7:0];
  assign sufficient = (achievable_pct >= 8'd100);
  // A buffer smaller than a round trip, reported as running at full rate.
  assign replay_ignored_err = size_it && (buffer_entries < entries_needed)
                              && (achievable_pct == 8'd100);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_sizings <= 8'd0; n_short <= 8'd0;
    end else if (size_it) begin
      n_sizings <= n_sizings + 8'd1;
      if (!sufficient) n_short <= n_short + 8'd1;
    end
  end
endmodule

Six sizings. A 64-cycle round trip.

Three short when the round trip is counted; none when it is not.

The replay buffer is a bandwidth-delay product and nothing else. Everything transmitted must be retained until it is acknowledged, so the buffer sets the maximum number of unacknowledged flits, which sets the achievable rate directly. A buffer half the size runs the link at half rate with no error, no stall counter and no obvious symptom.

Row five is the sizing trap when the link widens. Doubling the flit rate doubles the requirement, so a buffer sized for one flit per cycle runs a two-flit link at half rate — and the change that caused it was a link upgrade, not a buffer change.

Row three is the clamp. A buffer larger than a round trip does not exceed full rate; it buys margin against jitter rather than throughput, and a model without the clamp would report 200% and invite someone to spend area on it.

14. RTL 10 — A Transaction Pipeline Assembled

// RTL 10 - a CXL transaction pipeline assembled. Everything that must hold
// before requests that complete are requests that completed correctly.
module transaction_model #(parameter int REQUESTS_COMPLETE = 0) (
  input  logic clk, rst_n,
  input  logic       evaluate,
  input  logic       requests_complete,   // traffic flows end to end
  input  logic       tags_bounded,        // nothing issues without a tag
  input  logic       completions_reordered,// responses retire against their own tag
  input  logic       hazards_checked,     // a read cannot pass a write to its address
  input  logic       poison_propagated,   // suspect data stays marked
  input  logic       replay_sized,        // the buffer covers a round trip
  output logic       correct,
  output logic [5:0] fail_mask,
  output logic [7:0] n_eval, n_correct,
  output logic       false_correct_err
);
  assign fail_mask[0] = ~requests_complete;
  assign fail_mask[1] = ~tags_bounded;
  assign fail_mask[2] = ~completions_reordered;
  assign fail_mask[3] = ~hazards_checked;
  assign fail_mask[4] = ~poison_propagated;
  assign fail_mask[5] = ~replay_sized;
  // The traffic-flows build is what a pipeline bring-up reports.
  assign correct = (REQUESTS_COMPLETE != 0) ? requests_complete
                                            : (fail_mask == 6'd0);
  assign false_correct_err = evaluate && correct && (fail_mask != 6'd0);

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      n_eval <= 8'd0; n_correct <= 8'd0;
    end else if (evaluate) begin
      n_eval <= n_eval + 8'd1;
      if (correct) n_correct <= n_correct + 8'd1;
    end
  end
endmodule

One correct configuration of six, and four false claims.

"Traffic flows end to end" is what a pipeline bring-up reports and it is right about one of the six. Row four is the one that will never be found by a functional test: the pipeline is correct in every observable way and hands over corrupted data as good, which needs an injected fault and a scoreboard that knows what was injected.

Figure 4 — Five exits and four of them are the pipeline working. Only the first is a defect in the request; the other three are the pipeline refusing to do something unsafe or breaking work into pieces, which is why "a request is not issuing" is the normal state of a loaded engine rather than a symptom.

15. Quantitative Reasoning

Pipeline rate. Eight stalled cycles of sixty-four: 56 issued, 87% throughput — and an initiation interval that still reads one.

Tags. A 32-tag pool with 32 in flight admits nothing; an unbounded pipeline issues a transaction with no name for its completion.

Reorder. Tag 5 arriving against a window head of 3 is held; retiring it posts tag 5's data against tag 3's entry, for three cycles with no error signal.

Decode. 512 is memory, 2100 is configuration, 4200 is a register and 1024 is nothing — a one-range decoder claims all four for memory.

Write merging. Four sixteen-byte writes into a 64-byte line: one access merged against eight unmerged, and the four saved reads are full memory round trips.

Hazards. A read behind an in-flight write to the same address is held or forwarded; issued, it returns the previous value.

Splitting. 200 bytes against a 64-byte maximum is four packets with an 8-byte tail — 12% fill; 65 bytes is two packets with a one-byte tail.

Poison. Three of five responses carry a marker out; a dropping stage carries none, and the requester consumes corruption as a value.

Replay. A 64-cycle round trip at one flit a cycle needs 64 entries; 32 sustain 50% of the rate, and two flits a cycle need 128.

The assembled model. Six properties, six configurations, one correct. The traffic-flows milestone reported five.

16. Assertions

Every check is an explicit comparison against an exact value. Icarus Verilog 13.0 has no concurrent assertion support, so each is a procedural comparison against 1'b1, and every one is an equality.

Every inclusive threshold is driven at exactly equal, every ceiling both on and off its boundary, and every floor past it — the three rules this batch inherits.

Pipeline rate. Throughput of exactly 90% is constructed from six stalled cycles in sixty-four, and more stalled cycles than the window holds is asserted to floor at zero issued.

chk(rGi == 8'd0, "which floors the issue count at nothing, not a wrap");
chk(rGt == 8'd0, "at no throughput");

Tags. A pool with one tag left is driven against one with none, and an in-flight count above the pool is asserted to floor rather than wrap.

Reorder. Both distance directions are driven, and a scoreboard that disagrees with the window head is asserted to be flagged by both builds.

chk(oGr == 8'd5,  "so the buffer retires tag 5");
chk(oGe == 1'b1,  "and flags that the scoreboard expected tag 3");

Decode. The first and last byte of the memory range are both driven, the first byte of the configuration range is driven, and an unmapped range is asserted to route nothing.

Write merging. Coverage capped at the line is driven, and the unconfigured line size is asserted to fall back to read-modify-write.

Hazards. Both reasons a hazard does not exist are driven separately, and a hazard at address zero is asserted a hazard.

Splitting. An exact multiple and a remainder are both driven, and a tail fill of exactly 50% is constructed from 160 bytes against a 64-byte maximum.

Poison. Both reasons for suspicion are driven separately, and suspect data with no valid response is asserted as not a delivery.

Replay. A buffer exactly covering a round trip is driven, and an oversized buffer is asserted to clamp at full rate.

The assembled model. Every fail mask is asserted as an exact six-bit value, and each of the six bits is driven false alone.

Totals: 291 checks across two testbenches, 151 on the front five models and 140 on the back five, all passing on the unmutated sources.

17. Mutation Testing

Sixty mutations were injected one at a time. 60 injected, 60 killed, after three survivors.

Two survivors were a modelling artifact rather than a stimulus gap, and that is a distinct thing.

Section 7's in_order and misretire_err both survived because every case in the testbench set expected_tag equal to window_head. With the two always identical, comparing against either gives the same answer — so both mutations were unobservable. The gap was not in what was driven; it was in what was allowed to differ.

The fix is a case where the scoreboard and the hardware disagree, which is exactly the inconsistency misretire_err exists to detect. Driving arriving_tag = 5, window_head = 5 and expected_tag = 3 kills both mutations, and it makes the check's purpose clearer: it fires in both builds, because a scoreboard disagreeing with the window head is a defect upstream of whichever retirement policy is in use.

The third survivor was a genuinely dominated guard. Section 8's three size != 0 terms survived removal, and the reason is provable: with a size of zero the upper bound equals the base, so addr >= base && addr < base is false for every address — the range already excludes itself. All three were deleted with a comment naming the domination, and the mutation replaced with one that makes the memory range's lower bound exclusive, which needed the first byte of the range driven.

Section 9's saving floor was deleted before mutating for the same reason. merged_accesses is at most 2 and never exceeds 2 * writes, so the subtraction is non-negative in the merging build and the two are equal in the unmerged one — the floor's false branch is reachable only at equality, where both branches give zero.

The complete set was re-run after every stimulus change, per standing discipline, and all sixty held.

18. Verification Strategy

What a testbench for a transaction pipeline must cover.

Let every pair of inputs that could differ, differ. This chapter's two hardest survivors were not an undriven case — they were two inputs that the testbench always tied together. A model with two nominally independent inputs is only exercised when they disagree, and the disagreement is often the condition the check exists for.

Prove domination before writing a guard, and delete it with a comment when it holds. Two guards here were dominated — a range size against a base-plus-size comparison, and a saving floor against a bounded subtrahend — and both were removed before mutating rather than after.

Drive both directions of every two-branch expression. Section 7's distance and section 12's two reasons for suspicion are each two branches, and each needed both driven; reasoning that they are symmetric is not testing them.

The cases where the simpler pipeline is right. A window with no stalls. A pool with a tag left. A completion that is the head. An address in the one range. A single write of a whole line. A read to a different address. A request within the payload maximum. A clean response. Eight cases across nine models, each exempted explicitly, and each the case a directed bring-up test drives first.

Counters as a second signature. Ten models, ten pairs of totals, differing in all ten — five degraded against one, three blocked against none, four held against none, three poisoned against none.

What a real pipeline needs that these models do not have. State and identity. A reorder buffer holds a window, not a head; a hazard check holds a CAM, not a flag; a tag pool holds which tags, not how many. Every model here abbreviates the bookkeeping and keeps the decision, and section 26 exercises 2, 4 and 6 are where the bookkeeping comes back.

19. Synthesis and Implementation Reality

Section 5's stall counter is the cheapest thing in the chapter and the most often absent. One counter per engine, incremented when valid is high and ready is low, and it is the only way to distinguish a pipeline that cannot go faster from traffic that is not asking it to.

Section 6's tag pool is a free-list, and its depth is a bandwidth-delay product exactly as section 13's replay buffer is — the number of tags bounds the outstanding transactions, which bounds the achievable rate over a round trip. The two are the same calculation applied to different resources, which is why sizing one without the other is a common mistake.

Section 7's reorder buffer is the largest structure here. It holds a window of outstanding transactions with their requester identity, and it is the difference between a device that pipelines and one that completes in order — which is why it is also the first thing a schedule-pressured design proposes to simplify.

Section 8's decoder is a handful of comparators and its difficulty is entirely in the register programming: the ranges come from enumeration, and a decoder that is correct against wrong bases is wrong.

Section 9's merge buffer competes with section 13's replay buffer for the same SRAM, and both are sized from traffic rather than from protocol. They are the two places where a device's area is decided by a workload assumption.

Section 12's poison bit is one wire per data path, and it is dropped when a datapath is retimed and the marker is not carried through the added stage. That is a mechanical error rather than a design one, which is why section 20 asks for a counter rather than a review.

20. Silicon Observability

"Responses with poison set, in and out" is the counter this chapter argues hardest for, because section 12's defect has no other signature. Two counters that must be equal is a much stronger check than one counter that should be low — and if they diverge, the difference is the number of corruptions delivered as good data.

21. Debug Lab

Symptom. A device passes functional bring-up on all three protocols. Under a mixed read/write workload from four hosts: read bandwidth is 60% of target, one host reports occasional wrong data with correct ECC, and a soak test produces a single unexplained application crash after nine hours.

Step 1 — is the pipeline stalling or is it not being asked? Cycles valid with ready low: 41% of cycles. Section 5, and the pipeline is stalling rather than idling — which eliminates "the hosts are not offering enough" and points inside.

Step 2 — what is it stalling on? Cycles with the tag pool empty: 38%. The pool is 32 tags against a round trip that, measured, holds 51 transactions' worth of latency. Section 6, and the tag count was set from an earlier link generation with half the latency.

Step 3 — confirm with the other bound. Replay buffer occupancy, maximum: at its limit, 64 of 64. Section 13, the same bandwidth-delay product bounding a different resource. Both were sized from the same stale round-trip number, which is why they failed together and why fixing one alone would have moved the bottleneck rather than removing it.

Step 4 — the wrong data. Reads held and reads forwarded: the forwarded counter reads zero, always, and forwarding is enabled in the configuration register. The forwarding path exists and never fires — its address comparator was synthesised against a registered copy of the write address that updates a cycle late. Section 10, and the reads that should have forwarded were issued instead.

Step 5 — why only occasionally? The window is one cycle wide: a read must arrive exactly one cycle after the write for the comparison to miss. At 41% stall the pipeline rarely produces that pattern, which is why it took a four-host workload to surface it.

Step 6 — the crash. Responses with poison set, in and out: in reads 3, out reads 2. Section 12. One stage in the read return path was retimed during timing closure and the poison bit was not carried through the added register. One corrupted line in nine hours, delivered as good data.

The finding. Three defects with two causes. A stale round-trip number sized both the tag pool and the replay buffer, and a late-stage timing fix dropped a marker that nothing checked. The wrong-data path was a third, independent, and would have been found on day one by a counter that never existed.

The fix. Re-derive both depths from the measured round trip — 51 transactions means 64 tags and 64 entries, and the two must be re-derived together whenever the link changes. Fix the forwarding comparator's timing. Carry the poison bit through the retimed stage, and add the in-and-out counters so the next retiming cannot do it silently.

What made this hard. Every symptom had a different owner — throughput, data correctness and stability — and two of them shared a root cause that neither team's counters would have shown. The one counter that would have caught the third, poison in against poison out, is a pair rather than a single value, which is why nobody had built it.

22. Design Review

1. Is there a stall counter per engine? Without it you cannot tell a slow pipeline from a quiet one. Section 5.

2. How many tags, and derived from what round trip? It is a bandwidth-delay product and it goes stale when the link changes. Section 6.

3. Is the replay buffer derived from the same round trip, at the same time? Sections 6 and 13 fail together. Section 13.

4. Does the reorder buffer retire by tag or by arrival? Arrival order hands one requester another's data with no error. Section 7.

5. How many address ranges does the decoder know, and what happens to an address in none of them? An unrouted access must be refused, not served. Section 8.

6. Does the merge buffer know the line size, and where does that value come from? An unconfigured line silently becomes read-modify-write. Section 9.

7. Is the hazard comparator against the live write address or a registered copy? A cycle late is a stale read. Sections 10 and 21.

8. What is the packet-count histogram for real traffic? A one-byte tail packet costs a whole header. Section 11.

9. Is there a poison-in and a poison-out counter, and are they compared? It is the only signature the defect has. Sections 12 and 20.

10. Which of the six properties does "traffic flows end to end" imply? Section 14 exists because the answer is the first one only.

23. How This Appears In Real Engineering

A pipeline designer owns sections 5, 6 and 13 together, and they are one calculation: the round trip sets the outstanding count, which sets the tag pool and the replay buffer, which set the achievable rate. The failure mode in section 21 is not that anyone computed it wrongly; it is that the link changed and nobody recomputed it.

A memory-side designer owns sections 9 and 11, and both are workload-shaped rather than protocol-shaped. Merge-buffer depth and payload-splitting efficiency are decided by the requester's size distribution, which means the numbers come from a trace rather than from a specification.

A verification engineer meets sections 7 and 12, which are the two defects that produce correct-looking wrong data. Neither is findable without a scoreboard that knows which tag asked for what and an injected-fault flow that knows what it injected. Everything else in this chapter shows up as a stall, a latency or an error bit; these two do not.

A physical-design or timing engineer owns the way section 12 actually breaks. Nobody removes a poison bit deliberately — it is dropped when a datapath is retimed and the marker is not carried through the new stage, which is a mechanical omission that a functional test cannot see and a counter pair can.

24. Common Misconceptions

"The pipeline has an initiation interval of one." Empty. With eight stalls in sixty-four it delivers 87%. Section 5.

"Tags are just flow control." A tag is a name — without one, a completion cannot be matched to its request. Section 6.

"Completions arrive in order most of the time." Retire by arrival and the one time they do not, a requester gets another's data with no error. Section 7.

"The device is a memory device, so decode to memory." 2100 is configuration and 4200 is a register. Section 8.

"Four small writes are four writes." They are eight memory accesses unmerged, and one merged. Section 9.

"The read and the write are to the same address, so ordering handles it." Only if something checks. Section 10.

"The request is 200 bytes." It is four packets, the last carrying eight. Section 11.

"ECC caught it, so we are safe." Only if the marker survives every stage. Section 12.

"The replay buffer is a link-layer detail." It bounds the achievable rate exactly as the tag pool does. Section 13.

"Traffic flows end to end." One property of six. Section 14.

25. Interview Reasoning

Q. A pipeline with an initiation interval of one is delivering 60% of its target. Where do you look?

At the stall counter first — cycles with valid high and ready low. It separates "the pipeline cannot go faster" from "nothing is asking it to", and those have opposite fixes. If it is stalling, the next two counters are cycles with the tag pool empty and replay buffer occupancy, because both bound the outstanding count.

Q. Why do the tag pool and the replay buffer fail together?

Because they are the same bandwidth-delay product applied to different resources. The round trip sets how many transactions can be outstanding, and both the tag count and the buffer depth have to cover it — so a link change that lengthens the round trip under-sizes both at once, and fixing one just moves the bottleneck.

Q. What is wrong with retiring completions in arrival order?

It hands one requester another's data. If tag 5's completion arrives while the window is waiting for tag 3, retiring it posts tag 5's data against tag 3's entry — the transaction completes, the data is well-formed, and nothing errors. It is findable only with a scoreboard that knows which tag asked for what.

Q. A read returns the value that was there before a write that was already in flight. Explain.

The hazard comparator missed. Either there is no check at all, or — more commonly — the comparator is against a registered copy of the write address that updates a cycle late, so a read arriving exactly one cycle behind the write compares against the previous address and issues. The window is one cycle wide, which is why it needs sustained mixed traffic to surface.

Q. Your device delivers one corrupted line every few hours with ECC reporting clean. What happened?

A stage dropped the poison marker. ECC detected the error and set the bit; a later stage cleared it, so the requester consumed detected corruption as a normal value. It is almost always mechanical — a datapath retimed during timing closure with the marker not carried through the new register — and the only signature is a poison-in count that exceeds the poison-out count.

Q. How would you make section 12 findable?

Two counters that must be equal, rather than one that should be low. Poison set on the way in and poison set on the way out, compared continuously — if they diverge, the difference is exactly the number of corruptions delivered as good data, and it catches the next retiming as well as this one.

26. Exercises

1. Derive section 5's stall cycles from a producer and a consumer rate rather than taking them as an input.

2. Give RTL 3 a real window of outstanding tags and show how deep it must be for a given completion-reordering distribution.

3. Combine RTL 2 and RTL 9: show that the tag count and the buffer depth are the same calculation, and find the round trip at which a given pair under-sizes.

4. Replace RTL 6's single in-flight write with a CAM of them and find the CAM depth at which the hazard rate stops falling.

5. Drive RTL 7 with a realistic request-size distribution and find the maximum payload that minimises total header overhead.

6. Extend RTL 4 to programmable ranges and show which register-programming errors produce silent misroutes rather than unrouted accesses.

7. Model RTL 6's comparator against a registered write address and find the traffic pattern that produces a stale read.

8. Add a poison-in and poison-out counter pair to RTL 8 and show that a dropped marker is detectable in one cycle.

9. Model section 21 end to end: a stale round-trip number, a late comparator and a retimed datapath under four-host traffic.

10. Add a seventh property to RTL 10. If it is implied by one of the six, say which; if not, give the pipeline it catches that the current mask calls correct.

27. Summary

24.1 got a flit to the right engine and found that every resource three protocols share is a defect. This chapter is inside one engine, and the finding rhymes: every question a request must be asked before it issues has a version that answers yes without checking.

A pipeline issues one per cycle only while nothing stalls. Eight stalled cycles in sixty-four is 87% throughput with an initiation interval that still reads one — the pipeline is a claim and the throughput is a measurement.

Nothing issues without a tag, because a tag is a name rather than a permission. A 32-tag pool with 32 in flight admits nothing, and an unbounded pipeline issues a transaction whose completion can never be matched.

Completions retire by tag, not by arrival. Tag 5 arriving against a window head of 3 must be held; retiring it gives one requester another's data for three cycles with no error signal anywhere.

A device answers to several ranges. 512 is memory, 2100 is configuration, 4200 is a register and 1024 is nothing at all — and a one-range decoder serves every one of them from memory, successfully.

Merge writes to a line. Four sixteen-byte writes are one access merged and eight unmerged, and the four saved reads are full memory round trips the workload never asked for.

A read must not pass a write to its own address. Held or forwarded is correct; issued, it returns the previous value — and in section 21 the comparator existed and was a cycle late.

Split what exceeds the maximum payload. 200 bytes is four packets with an 8-byte tail, and 65 bytes is two packets for one byte of payload.

Poison must survive every stage. Three of five responses carry the marker out and a dropping stage carries none — the only defect here that turns detected corruption into undetected corruption.

And the replay buffer is a bandwidth-delay product, the same calculation as the tag pool: 64 entries for a 64-cycle round trip, and 32 entries run the link at half rate with no error and no stall.

Two mutations survived because the testbench tied two independent inputs together — the scoreboard's expected tag and the hardware's window head — which is a modelling artifact rather than an undriven case. A model with two nominally independent inputs is only exercised when they disagree. Two further guards were proved dominated and deleted before mutating.

Traffic flowing end to end is one property of six. The milestone a pipeline bring-up reports called five of six pipelines correct when one was — and section 21 is a device that passed every functional test and delivered one corrupted line every nine hours.

24.3 — CXL Memory Controllers goes one layer further in: what happens to a request after it leaves this pipeline and meets DRAM, where the ordering the pipeline worked so hard to preserve meets a scheduler that reorders on purpose.