Near-Memory Compute

Chapter 17.2 made remote capacity part of one memory map and measured the cost: an illustrative round trip roughly 2.5× the local one. This chapter asks the inverse question — what if the operation travels instead of the data?

1. The One-Sentence Model

Near-memory compute trades data movement for command movement. It does not win because compute is closer; it wins when the bytes avoided across the expensive boundary greatly exceed the command and result bytes added — and it costs a set of new obligations that the host can no longer take back.

2. What This Chapter Owns

What is new here:

A quantitative decision rule. §5's byte-movement equation and §6's suitability table decide whether near-memory compute helps at all for a given operation. Everything after §6 is about paying that benefit's cost correctly.

A command is an obligation the host cannot withdraw. Once accepted, the memory side owns it — through link retries, through recoveries, through the host's own timeouts (§9, §31).

Result capacity must be reserved before execution. §23 is a deadlock that has nothing to do with the protocol and everything to do with the order of two decisions.

Fairness against the host is a correctness-adjacent property. §18 is a design that is functionally perfect and makes the system unusable.

And exactly-once is the hard part. §32's blind re-issue is the flagship failure, and §33's idempotence distinction is what makes recovery tractable.

3. Sourcing

4. Three Places the Work Can Happen

Two distinctions that must not blur.

Near-memory compute is a normal logic engine in a normal process, placed on the memory-side die where it can reach the memory controller without crossing the link. It is ordinary hardware in an unusual place.

In-memory compute changes what the array itself does, and is bounded by what the storage technology can be made to compute. The two have different capability envelopes and different failure modes, and calling both "processing in memory" hides that. This chapter is about the first.

5. The Byte-Movement Equation

The decision rule, and it must be computed before any design work.

HOST METHOD
traffic_host = input_bytes_across_boundary + result_bytes_across_boundary
 
NEAR-MEMORY METHOD
traffic_nmc  = command_bytes + result_bytes + control_and_coherence_bytes
 
BENEFIT
traffic_saved = traffic_host - traffic_nmc

Worked, with illustrative numbers. A reduction over a 4 MiB buffer producing a 64-byte result. Command descriptor: 64 bytes. Control and coherence overhead: illustratively 1 KiB.

traffic_host = 4 MiB + 64 B
             = 4,194,304 + 64
             = 4,194,368 bytes
 
traffic_nmc  = 64 + 64 + 1,024
             = 1,152 bytes
 
traffic_saved = 4,194,368 - 1,152
              = 4,193,216 bytes
 
ratio        = 4,194,368 / 1,152
             ≈ 3,641×

Now a case where it does not help. An element-wise transform over the same 4 MiB buffer, writing 4 MiB of output that the host then needs.

traffic_host = 4 MiB in + 4 MiB out (written back)   = 8,388,608 bytes
traffic_nmc  = 64 (command) + 4 MiB (result to host) + 1,024
                                                      = 4,195,392 bytes
traffic_saved = 8,388,608 - 4,195,392 = 4,193,216 bytes
ratio         = 8,388,608 / 4,195,392 ≈ 2.0×

And a case where it loses. A transform whose output stays in memory and is never read by the host, compared against a host that never needed to read the input either — because the operation could have been expressed as a memory-to-memory copy the memory side already performs.

Three readings.

The reduction wins by three orders of magnitude; the transform wins by 2×. Both are wins, and they justify entirely different amounts of engineering. Computing the ratio first is what keeps the effort proportionate.

The result size is the dominant variable. In the first case the result is 64 bytes; in the second it is 4 MiB. result_bytes appears in both sides of the subtraction and does not cancel — it is subtracted from the saving once, which is why a result-preserving operation is a weak candidate.

And control_and_coherence_bytes is the term most often set to zero in a proposal. It covers ownership handoff, cache maintenance and completion traffic (§27, §30). At 1 KiB it is negligible against 4 MiB and decisive against a 2 KiB operation — which is why fine-grained near-memory operations rarely pay.

6. When It Helps, and When It Does Not

Two consequences.

Row 3 is the one that surprises people. A near-memory engine is generally less capable per byte than the host's compute. If the operation is compute-bound, moving it toward the memory moves it to slower arithmetic — and the boundary was never the constraint.

And row 5 compounds with §5. Every host synchronisation adds control traffic and a full round trip. An operation that synchronises every few kilobytes has converted its data movement into control movement at the same order of magnitude, which is the whole benefit spent on coordination.

7. The Near-Memory Path

Read the arbiter. The engine and the host reach the same controller, so every byte the engine reads locally is a byte of controller bandwidth the host did not get. §15 through §21 are that contention, and it is the difference between a near-memory design that helps and one that makes the system worse.

8. The Command Descriptor

// ILLUSTRATIVE near-memory command. NOT a UCIe, CXL or JEDEC structure, and no
// field corresponds to any specified field of any of them (Section 3).
typedef enum logic [OP_W-1:0] {
  NMC_REDUCE_SUM = 'd0,
  NMC_REDUCE_MAX = 'd1,
  NMC_TRANSFORM  = 'd2,
  NMC_FILL       = 'd3
} nmc_op_e;
 
typedef struct packed {
  logic [CMD_ID_W-1:0]  cmd_id;     // the HOST's identity for this command
  logic [GEN_W-1:0]     generation; // Section 14 — what makes a stale completion detectable
  nmc_op_e              opcode;
  logic [ADDR_W-1:0]    src_base;
  logic [ADDR_W-1:0]    dst_base;
  logic [LEN_W-1:0]     length;     // bytes
  logic [CTX_W-1:0]     context;    // ownership / permission context (Section 30)
  logic [RES_W-1:0]     result_bytes;  // how much the host must be able to receive
} nmc_cmd_t;

Architecture. One descriptor carrying identity, operation, extents, an ownership context, and — critically — how large the result will be. That last field is what makes §22's admission decision possible before execution.

State. One register per pipeline stage on the way in, plus an entry in the command table (§10).

Cycle behaviour. Formed at the host, transmitted once, and held stable while offered (12.1's handshake discipline). Nothing recomputes any field on the memory side.

Contract. The memory side must be able to determine, from the descriptor alone, whether it can accept the obligation. A descriptor that omits result_bytes forces the engine to accept blind and discover the problem after executing — which is §24.

Failure. Omitting generation and relying on cmd_id alone (§14). Or deriving result_bytes from opcode and length on the memory side, which couples the admission logic to the operation set and breaks the moment an operation's result size becomes data-dependent.

DV. Assert the descriptor is stable under stall, and that every field the memory side uses was carried rather than derived.

9. Acceptance Transfers an Obligation

The sentence the rest of the chapter depends on.

When a command handshake completes, the memory side owns an obligation that the host cannot withdraw. Not by timing out, not by resetting its own tracking, and not because the link went away.

Three consequences.

A command is not a packet. 14.3's replay machinery makes a transport object re-sendable precisely because re-sending it has no semantic effect. A command has semantic effect, so the same reasoning does not transfer (§32).

The host's tracking entry must therefore outlive its own uncertainty. §12 is the bug of freeing it early, and §31 is the situation that tempts it.

And the memory side must be able to report completion or failure even when it cannot deliver a result — otherwise a command that executed successfully and could not return its output leaves both sides permanently unsure.

10. The Command Table and Its States

// ILLUSTRATIVE command state on the memory side. Not a protocol state machine.
typedef enum logic [2:0] {
  CMD_FREE          = 3'd0,
  CMD_ACCEPTED      = 3'd1,   // obligation owned; result capacity reserved
  CMD_WAIT_MEMORY   = 3'd2,   // needs controller bandwidth
  CMD_EXECUTING     = 3'd3,   // a chunk is in flight
  CMD_RESULT_READY  = 3'd4,   // output produced, not yet delivered
  CMD_COMPLETE      = 3'd5,   // result consumed by the host
  CMD_FAILED        = 3'd6    // explicit failure, reported
} nmc_state_e;
 
typedef struct packed {
  logic                  valid;
  logic [CMD_ID_W-1:0]   id;
  logic [GEN_W-1:0]      generation;
  nmc_state_e            state;
  nmc_op_e               opcode;
  logic [ADDR_W-1:0]     cursor;          // where the next chunk starts
  logic [LEN_W-1:0]      remaining;       // bytes still to process
  logic [LEN_W-1:0]      bytes_processed; // monotonic — Section 34's property
  logic                  result_reserved; // Section 23
  logic [RES_W-1:0]      result_bytes;
} nmc_cmd_state_t;
 
nmc_cmd_state_t cmd_q [MAX_COMMANDS];

Architecture. One entry per accepted command, holding the obligation, the execution cursor, and the reservation. Everything a chunked, restartable, exactly-once execution needs.

State. MAX_COMMANDS entries. The depth is a concurrency parameter — one command at a time serialises long operations behind short ones and wastes the engine during memory stalls.

Cycle behaviour. state has exactly one next-state owner (§11). cursor and remaining advance only on a completed chunk, never on a chunk that was issued and not retired — which is the 13.4 §18 pointer rule applied to execution progress.

Contract. The host's table and this one are two records of one obligation. They may disagree about state and must never disagree about existence — the host may believe a command is still executing when it is CMD_RESULT_READY, and that is fine; the host believing it is gone when it is CMD_EXECUTING is §12.

Failure. bytes_processed advancing on issue rather than on completion, which makes the progress counter lie in exactly the situation — a stalled chunk — where it is most consulted.

DV. Assert bytes_processed + remaining == length for every live command, and that bytes_processed is monotonic (§34).

11. The Execution FSM

// ILLUSTRATIVE. One next-state owner, explicit priority, unique case.
always_comb begin
  nxt = cmd_q[i].state;
  unique case (cmd_q[i].state)
    CMD_FREE:         if (accept_fire[i])                nxt = CMD_ACCEPTED;
    CMD_ACCEPTED:                                        nxt = CMD_WAIT_MEMORY;
    CMD_WAIT_MEMORY:  if (fatal_error[i])                nxt = CMD_FAILED;
                      else if (chunk_grant[i])           nxt = CMD_EXECUTING;
    CMD_EXECUTING:    if (fatal_error[i])                nxt = CMD_FAILED;
                      else if (chunk_done[i] && (cmd_q[i].remaining == '0))
                                                         nxt = CMD_RESULT_READY;
                      else if (chunk_done[i])            nxt = CMD_WAIT_MEMORY;  // YIELD
    CMD_RESULT_READY: if (result_consumed[i])            nxt = CMD_COMPLETE;
    CMD_COMPLETE:     if (host_ack[i])                   nxt = CMD_FREE;
    CMD_FAILED:       if (host_ack[i])                   nxt = CMD_FREE;
    default:                                             nxt = CMD_FAILED;
  endcase
end
 
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n) cmd_q[i].state <= CMD_FREE;
  else        cmd_q[i].state <= nxt;

Architecture. Seven states, one owner, one unique case. The transition that matters most is CMD_EXECUTING → CMD_WAIT_MEMORY on a completed chunk with work remaining — that is the yield, and it is what §21's design lacks.

State. Three bits per command. CMD_COMPLETE and CMD_FAILED both require a host acknowledgement before returning to CMD_FREE, which is what prevents the identity from being recycled while a completion is still in flight (§13, §14).

Cycle behaviour. fatal_error is checked before progress in both working states, so an error cannot be overtaken by a chunk completing in the same cycle. unique case makes the priority explicit and lets the tool prove the arms are disjoint.

Contract. The arbiter (§17) sees CMD_WAIT_MEMORY as a request; the result path sees CMD_RESULT_READY; the host sees CMD_COMPLETE or CMD_FAILED. Three consumers, three states, no overlap.

Failure. Going straight from CMD_EXECUTING to CMD_FREE on the last chunk, skipping CMD_RESULT_READY and CMD_COMPLETE — which frees the entry while the result is still in the return path, and the result then arrives with no owner.

DV. Cover every state and every legal transition; assert that no illegal transition occurs, which is 12.4 §10's legal-transition function.

12. Wrong RTL — the Host Retires the Command on Transmission

// WRONG — the host frees its command entry when UCIe accepts the descriptor.
always_ff @(posedge clk)
  if (ucie_tx_fire)
    host_cmd_valid_q[id] <= 1'b0;      // ← transport launch, not command completion

Two failures, and the second is worse.

The result arrives with no owner. The engine executes and returns a result naming id; the host's entry is gone; the result is discarded and the work is wasted invisibly.

And then the identity is reused. id is now free, so the host allocates it to a new command. The old command's result returns and is matched to the new command — a stale completion alias, which is 12.2 §25's immediate-reuse bug with a semantic operation attached to it.

Row 55 is a wrong answer delivered confidently. The host acts on A's result as if it were B's. Nothing anywhere reports an error, because at the transport level everything was delivered exactly once to a live identity.

The fix is two-part, and both parts are needed: the entry is freed only at semantic completion (§9), and the identity carries a generation so that even a mistakenly-early free cannot produce an alias (§14).

13. SVA — Identity Is Not Recycled Under a Live Command

// MANDATORY. A command identity is unique among live commands.
property p_cmd_id_unique_while_live;
  @(posedge clk) disable iff (!rst_n)
    host_alloc_fire |-> !id_currently_live(alloc_cmd_id);
endproperty
a_cmd_id_unique_while_live: assert property (p_cmd_id_unique_while_live);
 
// The host entry is not freed by a transport event (Section 12).
property p_host_entry_not_freed_on_tx;
  @(posedge clk) disable iff (!rst_n)
    (ucie_tx_fire && !semantic_completion[ID])
      |=> $stable(host_cmd_valid_q[ID]);
endproperty
a_host_entry_not_freed_on_tx: assert property (p_host_entry_not_freed_on_tx);
 
// A completion must name a live command AND the current generation.
property p_completion_matches_live_generation;
  @(posedge clk) disable iff (!rst_n)
    completion_valid |-> (host_cmd_valid_q[comp_id]
                       && (host_cmd_gen_q[comp_id] == comp_gen));
endproperty
a_completion_matches_live_generation:
  assert property (p_completion_matches_live_generation);

Architecture. Three properties: uniqueness at allocation, no transport-triggered free, and generation-checked completion matching.

Why the third is the safety net. The first two prevent §12 from arising. The third makes §12 detectable if it arises anyway — a stale result carrying generation 3 cannot match an entry now at generation 4, so the alias becomes a reported orphan instead of a wrong answer.

DV. The alias case requires an early free followed by a reallocation followed by a late result. That is a three-event sequence that no random test produces, and it is §43's cp_stale_completion bin.

14. Generation, and Why an Identity Alone Is Not Enough

// ILLUSTRATIVE. Identity plus generation. NOT a UCIe or CXL field (Section 3).
logic [GEN_W-1:0] host_cmd_gen_q [MAX_COMMANDS];
 
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n)
    for (int i = 0; i < MAX_COMMANDS; i++) host_cmd_gen_q[i] <= '0;
  else if (host_alloc_fire)
    host_cmd_gen_q[alloc_cmd_id] <= host_cmd_gen_q[alloc_cmd_id] + 1'b1;

Architecture. A per-identity counter incremented on every allocation. The pair {cmd_id, generation} is unique over a far longer window than cmd_id alone, which is 12.2 §24's generation quarantine applied to a semantic command.

State. One counter per identity, sized so wrap is not reachable within the lifetime of any possible in-flight completion. GEN_W is not a free parameter — it is derived from the maximum time a completion can be delayed, which after a recovery can be very long.

Cycle behaviour. Incremented at allocation, never elsewhere. The generation travels in the descriptor and returns in the completion (§8).

Contract. The completion matcher (§13) compares both fields. A design that carries the generation but does not check it has paid the cost and kept none of the benefit.

Failure. Sizing GEN_W at one bit and calling it a toggle — which distinguishes consecutive uses and fails on the third. Or resetting generations on a link recovery, which destroys exactly the history the check needs at exactly the moment stale completions are most likely.

DV. Cover a completion arriving with a stale generation and confirm it is reported as an orphan rather than matched (§13).

15. The Engine Competes With the Host

The section most near-memory designs underestimate.

The engine's reads and writes go to the same memory controller the host's requests do. Four resources are shared:

Near-memory compute does not add memory bandwidth. It relocates the consumer of existing memory bandwidth from the far side of the link to the near side. The total is unchanged; only who competes for it has changed — and now the competitors are on the same side of the arbiter.

Two consequences.

The host's memory latency is now a function of engine activity. A long engine operation can multiply host access latency without the link showing any load at all — which is §18's diagnostic signature.

And the benefit in §5 is measured in link bytes while the cost here is measured in controller cycles. They are different currencies. A design that reports only the link saving has reported half the transaction.

16. Arbitration Policies

The recommendation, stated as one. For a shared controller, a reserved minimum for the host plus a bandwidth cap on the engine is usually right: it bounds the worst case for interactive traffic and lets the engine use everything else. Weights alone do not bound host latency, because a weight is a ratio and a ratio under saturation still allows a long queue.

17. A Bounded-Fairness Arbiter

// ILLUSTRATIVE controller arbiter. A reserved host share plus an engine cap.
// Fairness state advances ONLY on an actual grant that fired.
logic                    host_req;
logic                    nmc_req;
logic                    grant_host;
logic                    grant_nmc;
 
logic [AGE_W-1:0]        host_wait_q;      // saturating
logic [QUOTA_W-1:0]      nmc_quota_q;      // engine's remaining budget this window
logic [WIN_W-1:0]        window_q;         // rolling arbitration window
 
wire host_starved = (host_wait_q >= HOST_SERVICE_BOUND);
wire nmc_capped   = (nmc_quota_q == '0);
 
always_comb begin
  grant_host = 1'b0;
  grant_nmc  = 1'b0;
  unique case (1'b1)
    host_req && host_starved : grant_host = 1'b1;   // bounded override
    host_req && nmc_capped   : grant_host = 1'b1;
    nmc_req  && !nmc_capped  : grant_nmc  = 1'b1;
    host_req                 : grant_host = 1'b1;
    nmc_req                  : grant_nmc  = 1'b1;   // cap ignored when host idle
    default                  : ;
  endcase
end
 
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n) begin
    host_wait_q <= '0;
    nmc_quota_q <= NMC_QUOTA;
    window_q    <= '0;
  end else begin
    // Window refresh restores the engine's budget.
    if (window_q == ARB_WINDOW - 1) begin
      window_q    <= '0;
      nmc_quota_q <= NMC_QUOTA;
    end else begin
      window_q <= window_q + 1'b1;
      if (grant_nmc && ctrl_fire && (nmc_quota_q != '0))
        nmc_quota_q <= nmc_quota_q - 1'b1;
    end
 
    // Host age advances only when the host wanted service and did not get it.
    if (host_req && !(grant_host && ctrl_fire))
      host_wait_q <= (host_wait_q == AGE_MAX) ? AGE_MAX : host_wait_q + 1'b1;
    else if (grant_host && ctrl_fire)
      host_wait_q <= '0;
  end

Architecture. Two independent mechanisms. The quota caps the engine's share within a window; the age-based override bounds the host's worst case regardless of the quota. Either alone leaves a hole — a quota does not bound host latency within a window, and an override alone lets the engine consume everything up to the bound every time.

State. One saturating age, one quota counter, one window counter. The age saturates because a wrapping age reports a freshly-waiting host at the moment it has waited longest (13.4 §13).

Cycle behaviour. Grants are combinational. Both the quota decrement and the age reset are qualified by ctrl_fire — a grant the controller did not accept has served nobody, and crediting it is the most-repeated arbiter bug in this curriculum (13.4 §18).

Contract. The host's interactive latency requirement and the engine's throughput requirement are both properties of this block, and neither is visible at its interface — which is why §19 asserts both.

Failure. Setting NMC_QUOTA to the full window, which removes the cap while leaving the code that looks like one. Or omitting the fifth arm, which idles the controller when the engine is capped and the host has nothing to do — losing throughput for a fairness guarantee nobody needed at that moment.

DV. Saturate both sources and measure worst-case host wait against HOST_SERVICE_BOUND; then idle the host and confirm the engine reaches full controller utilisation.

18. Wrong Policy — the Engine Monopolises the Controller

// WRONG — a long operation streams the controller until it finishes.
assign grant_nmc  = nmc_req;              // engine always wins
assign grant_host = !nmc_req;

Illustrative arithmetic. A 4 MiB operation at 64 bytes per controller access is 65,536 accesses. If each occupies the controller for an illustrative 4 cycles, the operation holds it for 262,144 cycles.

Four properties, and the fourth is why it survives review.

It is functionally perfect. Every host access eventually completes with correct data. No assertion about correctness fires, and a functional regression passes cleanly.

The system is nonetheless unusable. A host access that takes a quarter of a million cycles is, from software's point of view, a hang.

The diagnostic signature points at the wrong place. The UCIe link is idle — that was the entire goal of the design — so every link-level metric says the interconnect is healthy. The bottleneck is a memory arbiter nobody is looking at, several layers below where the symptom appears.

And the near-memory metrics look excellent. Bytes avoided across the boundary: maximal. Engine utilisation: 100%. Every number the feature was justified by is at its best value while the system is at its worst — which is why §38's counters must include host-starvation cycles, or the design is optimised against a partial view.

19. SVA — Both Parties Make Progress, Under Assumptions

// LIVENESS, bounded, both directions, with assumptions stated (15.2 Section 36).
//
//   A1: the controller eventually accepts a granted request
//   A2: a requester keeps requesting until served
//   A3: maintenance terminates (17.1 Section 14)
assume property (@(posedge clk) disable iff (!rst_n)
  (grant_host || grant_nmc) |-> ##[1:CTRL_ACCEPT_BOUND] ctrl_fire);
assume property (@(posedge clk) disable iff (!rst_n)
  (host_req && !ctrl_fire) |=> host_req);
 
property p_host_served_within_bound;
  @(posedge clk) disable iff (!rst_n)
    host_req |-> ##[1:HOST_SERVICE_BOUND] (grant_host && ctrl_fire);
endproperty
a_host_served_within_bound: assert property (p_host_served_within_bound);
 
// The engine must not starve either — a DIFFERENT claim with a different bound.
property p_nmc_served_within_bound;
  @(posedge clk) disable iff (!rst_n)
    nmc_req |-> ##[1:NMC_SERVICE_BOUND] (grant_nmc && ctrl_fire);
endproperty
a_nmc_served_within_bound: assert property (p_nmc_served_within_bound);
 
// And the cap is real: the engine's share within a window is bounded.
property p_engine_share_bounded;
  @(posedge clk) disable iff (!rst_n)
    (window_q == ARB_WINDOW - 1) |-> (nmc_grants_this_window <= NMC_QUOTA);
endproperty
a_engine_share_bounded: assert property (p_engine_share_bounded);

Architecture. Two liveness bounds in opposite directions, plus a share bound.

Why both directions. §18 starves the host; a design over-corrected to fix it starves the engine, and commands then never finish — which looks like a broken engine and is a broken arbiter. A single fairness property with one generous bound cannot distinguish them.

Why the third is not implied by the first two. Bounded service says eventually within N. The share bound says how much of the window the engine may take, which is what actually determines host latency in the common case rather than the worst one.

DV. Prove all three. Then set NMC_QUOTA to the full window and confirm the first property fails — verifying the cap is load-bearing and not decoration.

20. Chunking

// ILLUSTRATIVE. A long command is executed as a sequence of bounded chunks so
// that arbitration opportunities occur throughout, not only at the end.
logic [LEN_W-1:0] this_chunk;
 
assign this_chunk = (cmd_q[i].remaining > CHUNK_MAX)
                  ? CHUNK_MAX[LEN_W-1:0]
                  : cmd_q[i].remaining;
 
always_ff @(posedge clk)
  if (chunk_done[i]) begin
    cmd_q[i].cursor          <= cmd_q[i].cursor + this_chunk;
    cmd_q[i].remaining       <= cmd_q[i].remaining - this_chunk;
    cmd_q[i].bytes_processed <= cmd_q[i].bytes_processed + this_chunk;
  end

Architecture. Execution in bounded units, with a return to CMD_WAIT_MEMORY between them (§11). The chunk boundary is an arbitration opportunity, and it is the only mechanism that bounds §18 structurally rather than by policy.

State. cursor, remaining and bytes_processed per command. Three fields rather than one because they answer three questions: where to read next, how much is left, and how much has been done — and the last is the one an observer needs (§38).

Cycle behaviour. All three advance only on chunk_done, in one place, together. A design that advances the cursor on chunk issue re-reads or skips data if the chunk is refused.

Contract. bytes_processed + remaining == length is invariant for a live command (§34). The arbiter relies on the yield happening; the host relies on progress being observable.

Failure. Choosing CHUNK_MAX too large re-creates §18 at a smaller scale; too small adds per-chunk overhead and arbitration churn. The size is bounded below by efficiency and above by HOST_SERVICE_BOUND — and stating that relationship is what makes it a design parameter rather than a magic number.

DV. Cover remaining < CHUNK_MAX, exactly CHUNK_MAX, and many multiples; cover a chunk boundary coinciding with a maintenance window and with a link recovery.

21. Wrong Design — One Command Holds the Controller to Completion

// WRONG — no chunking. The engine holds the controller for the whole length.
always_ff @(posedge clk)
  if (state == CMD_EXECUTING)
    issue_memory_access(cursor + progress);   // until progress == length

This is §18 expressed structurally rather than as a policy, and it is worse in one specific way: no arbitration policy can fix it. A perfectly fair arbiter has nothing to arbitrate between, because the engine never yields.

Two readings.

The failure scales with the input, which the hardware does not control. A design tested with 4 KiB commands passes; the same design given a 1 GiB command produces a multi-second stall. Nothing about the hardware changed.

And chunking must be in the execution path, not the submission path. Splitting the work into many small commands at the host achieves the same yielding, at the cost of many descriptors crossing the link — which spends the §5 benefit on exactly the traffic near-memory compute exists to avoid. The engine must chunk internally.

22. Admission Is a Conjunction — Including the Result

// ILLUSTRATIVE. Every resource the command will need, checked BEFORE accepting.
assign cmd_admit =
      cmd_slot_available                 // a table entry (Section 10)
   && (result_slots_q >= cmd_result_need)// RESULT capacity — Section 23
   && context_valid                      // ownership / permission (Section 30)
   && engine_operational                 // the engine is not failed or quiesced
   && !maintenance_lockout;              // 17.1 Section 11, if the design gates on it

Architecture. Five terms. The second is the one designs omit, and §24 is why omitting it is a deadlock rather than a performance issue.

State. result_slots_q is the only stateful term (§25).

Cycle behaviour. Evaluated at acceptance, and the reservation is taken in the same cycle as the acceptance — not later, not optimistically. Reserve-then-accept, never accept-then-hope.

Contract. Accepting a command promises that it can execute and that its output can be held. The host relies on that promise to know its command will make progress.

Failure. §24. Also treating context_valid as a static configuration rather than a per-command check, which allows a command to be admitted against a buffer whose ownership the host has since reclaimed (§30).

DV. Force each term false alone and confirm no admission. Five directed tests.

23. Result Capacity Must Be Reserved Before Execution

The subtlest failure in the chapter, and it is a genuine deadlock.

A command that executes and has nowhere to put its result cannot make progress and cannot be undone. It holds an engine, a table entry, and whatever controller resources it acquired — indefinitely.

The cycle, step by step:

1. Result buffers are a finite resource, sized RESULT_SLOTS.
2. Commands are admitted without checking result capacity.
3. Enough commands execute that all result buffers are occupied by
   results the host has not yet consumed.
4. Another admitted command finishes executing. It has no slot.
5. It cannot complete, so it does not release its table entry, its
   engine resources, or its controller reservations.
6. The host is willing to consume results — but the return path or the
   completion ordering requires the blocked command to make progress first.
7. -> deadlock. Every component is behaving correctly.

Three properties.

No single component is wrong. The buffers are correctly sized for their purpose; the engine correctly executes; the host correctly consumes. The error is the order of two decisions — admit, then discover.

It cannot be recovered by waiting. Unlike a congestion stall, no amount of time frees anything, because the thing that would free a buffer is blocked behind the command that needs one.

And it is load-dependent and late. With few commands in flight the buffers are never exhausted. The deadlock appears when concurrency rises, which is exactly when the feature is being demonstrated.

24. Wrong RTL — Admission Ignores Result Space

// WRONG — result capacity is checked when the result is produced, not when
// the command is accepted.
assign cmd_admit = cmd_slot_available && engine_operational;
 
// ... and much later:
always_ff @(posedge clk)
  if (result_ready[i] && (result_slots_q == '0))
    stall_command(i);                  // ← nothing will ever un-stall it (Section 23)

The comparison that makes the fix obvious:

Reserve before you commit. This is the same principle as 14.3 §29's "credits checked, replay space not" — a resource needed later in an operation must be secured before the operation starts, or the operation becomes unabortable at the point it discovers the shortage.

25. Result-Slot Accounting

// ILLUSTRATIVE. Reserve at admission, release at consumption. Simultaneous
// reserve and release handled in ONE place with ONE owner.
logic [RES_CNT_W-1:0] result_slots_q;
 
always_ff @(posedge clk or negedge rst_n)
  if (!rst_n) begin
    result_slots_q <= RESULT_SLOTS[RES_CNT_W-1:0];
  end else begin
    unique case ({cmd_admit_fire, result_consumed_fire})
      2'b10: result_slots_q <= result_slots_q - cmd_result_need;
      2'b01: result_slots_q <= result_slots_q + freed_result_slots;
      2'b11: result_slots_q <= result_slots_q - cmd_result_need + freed_result_slots;
      default: ;                       // 2'b00 — hold
    endcase
  end

Architecture. One counter, one owner, four explicit cases. The 2'b11 arm is written out rather than left to two independent statements, because that is the cycle where a design with separate ifs loses an update.

State. One counter, initialised to RESULT_SLOTS. Not a free-list unless results are variable-sized and non-contiguous — a counter suffices when a command's need is known at admission and the buffer is managed as a pool.

Cycle behaviour. Decremented by the command's stated need (§8's result_bytes, converted to slots), not by one. A design that reserves one slot per command and then produces a multi-slot result has reserved the wrong quantity and reaches §23 anyway.

Contract. §22's admission reads this. The counter is the only thing standing between the design and §23, so its accuracy is a correctness property rather than a performance one.

Failure. Two independent if statements — which on a simultaneous admit-and-consume cycle applies one update and the counter drifts. The drift is monotonic in one direction over time, so the design eventually either refuses all commands or admits past capacity.

DV. §26's bounds and conservation properties, checked every cycle. Cover the simultaneous case explicitly, because it is where the bug lives and it is rare in a lightly-loaded test.

26. SVA — Result Accounting Is Bounded and Conserved

// MANDATORY. The counter never exceeds its pool and never underflows.
property p_result_slots_bounded;
  @(posedge clk) disable iff (!rst_n)
    (result_slots_q <= RESULT_SLOTS);
endproperty
a_result_slots_bounded: assert property (p_result_slots_bounded);
 
// Conservation: free slots plus reserved slots equals the pool.
property p_result_slots_conserved;
  @(posedge clk) disable iff (!rst_n)
    (result_slots_q + total_reserved_by_live_commands() == RESULT_SLOTS);
endproperty
a_result_slots_conserved: assert property (p_result_slots_conserved);
 
// A command is never admitted without sufficient reservation (Section 22).
property p_admit_implies_reservation;
  @(posedge clk) disable iff (!rst_n)
    cmd_admit_fire |-> (result_slots_q >= cmd_result_need);
endproperty
a_admit_implies_reservation: assert property (p_admit_implies_reservation);
 
// A command that reaches RESULT_READY always has somewhere to put its result.
property p_result_ready_has_space;
  @(posedge clk) disable iff (!rst_n)
    (cmd_q[IDX].state == CMD_RESULT_READY) |-> cmd_q[IDX].result_reserved;
endproperty
a_result_ready_has_space: assert property (p_result_ready_has_space);

Architecture. Bounds, conservation, admission implication, and the end-to-end guarantee.

Why conservation is the valuable one. Bounds catch gross errors. Conservation catches the slow drift from §25's two-if bug, which stays within bounds for a long time and is invisible to every other check.

Why the fourth exists given the third. The third checks the decision; the fourth checks that the reservation survived from admission to result time — which a design that reserves correctly and then releases early would violate.

DV. All four always-on. The conservation property needs a testbench function that sums live reservations, which is the same shape as 17.2 §42's count-matches-population property.

27. Coherence, and Who Holds the Newest Value

A near-memory engine that reads or writes host-visible memory is another agent in the system.

Two viable architectures, and both are defensible:

Coherent participation. The engine is an agent; its reads obtain permission and its writes invalidate cached copies. Correct by construction, and it adds coherence traffic to §5's control_and_coherence_bytes — which for fine-grained operations can erase the benefit entirely.

Explicit ownership handoff. The buffer is transferred to the engine before execution and back afterwards, with software responsible for the transitions (§30). Cheaper per operation, and it moves a correctness obligation into software where a missed handoff produces §28 with no hardware fault at all.

28. Wrong Architecture — a Memory-Local Writer Behind Coherent Caches

// WRONG at system scope — the engine writes memory directly with no coherence
// participation and no ownership protocol, while the host may cache the line.
always_ff @(posedge clk)
  if (chunk_done[i])
    write_memory(cmd_q[i].dst_base + offset, computed_value);

1. The host reads address X. The line is cached, clean, on the host.
2. A near-memory command computes a new value for X and writes it to memory.
3. The host reads X again. Its cache hits. It returns the OLD value.
4. -> the host reads stale data. Transport was perfect. Memory holds the
     correct value. Every component did exactly what it was told.

And the reverse direction is worse.

1. The host writes X. The line is cached DIRTY on the host; memory is stale.
2. A near-memory command reads X from memory. It reads the OLD value.
3. The command computes a result from stale input and returns it,
     confidently and with no error.
4. Later the host evicts its dirty line, overwriting whatever the engine wrote.

Three properties.

Neither direction produces an error anywhere. No CRC failure, no timeout, no assertion. The only detector is a model that knows where the newest value lives (16.2 §31's argument, at a new agent).

The second direction is the dangerous one because the result is wrong, not just a subsequent read. A wrong answer computed from stale input propagates into everything downstream and cannot be traced back to a cache line.

And the fix is architectural, not a patch. Either the engine participates in coherence, or software performs an explicit ownership handoff around every operation (§30). There is no third option in which the engine simply writes and everything is fine.

29. Atomicity, and the Scope of Serialisation

If a near-memory engine performs read-modify-write updates, the serialisation must cover every agent that can touch the location — not only other near-memory commands.

An "atomic" operation that is atomic only with respect to its own class is not atomic. It is a mutual-exclusion mechanism with a scope that does not match the sharing.

Two consequences.

Genuine atomicity requires one serialisation point that every agent goes through (16.1 §19), which in practice means the coherence protocol or an equivalent system-wide mechanism. 11.3 and 16.2 are where those live. This chapter does not define one, and a design that invents a local lock has invented a false guarantee.

And the safest near-memory operations are the ones that do not need atomicity — reductions over a buffer nobody else is writing, transforms into a distinct output range. Restricting the operation set to those is a legitimate and common architectural choice, and it is far cheaper than solving the general case.

30. Explicit Ownership Handoff

The non-coherent alternative, stated at the level this chapter can support.

1. The host ensures its caches hold no stale or dirty copies of the buffer.
2. Ownership of the buffer is handed to the engine — a software-visible
   transition, not a hardware event.
3. The command executes. During this window the host must not access
   the buffer.
4. The engine completes and ownership returns to the host.
5. The host may access the buffer again.

// ILLUSTRATIVE context gate. The engine refuses commands whose buffer context
// is not currently owned by it. This is a HARDWARE CHECK on a SOFTWARE
// protocol — it cannot make software correct, only make violations visible.
logic [NUM_CTX-1:0] ctx_owned_by_engine_q;
 
assign context_valid = ctx_owned_by_engine_q[cmd_in.context];

Architecture. A per-context ownership bit consulted at admission (§22). The hardware cannot enforce the software protocol; it can refuse commands that visibly violate it, which converts a class of silent corruption into a reported error.

State. NUM_CTX bits, written by whatever mechanism performs the handoff.

Contract. Software guarantees step 3. The hardware guarantees only that a command against an unowned context is refused — which catches the common ordering mistake and none of the concurrent-access ones.

Failure. Treating the gate as sufficient. It is a check, not a guarantee, and a design that documents it as enforcement invites exactly the concurrent access it cannot see.

DV. Cover a command against an unowned context and confirm refusal; and cover a host access during step 3 to confirm the environment can produce the violation the hardware cannot catch — which is a scoreboard check, not an assertion.

31. Recovery While a Command Executes

The ambiguity that makes near-memory compute harder than memory access.

1. A command is accepted. The engine begins executing.
2. UCIe enters recovery. The link is unusable.
3. THE ENGINE KEEPS EXECUTING. It is local to the memory; the link is
   irrelevant to its progress.
4. The command completes. Its result sits in a reserved buffer.
5. The completion cannot be delivered until the link returns.
6. The host's timeout expires somewhere between step 2 and step 5.

A timeout tells the host that no completion arrived. It does not tell the host whether the command executed. (12.4 §16) For a memory read that ambiguity is inconvenient. For a command with side effects it is the whole problem.

32. Wrong Recovery — the Host Re-Issues the Command

// WRONG — a timeout is treated as evidence the command did not execute.
always_ff @(posedge clk)
  if (cmd_outstanding[id] && (age_q[id] >= CMD_TIMEOUT))
    reissue_command(id);            // ← it may have already executed. Once.

Illustrative, with a non-idempotent operation. A command that adds a computed delta into a destination range.

Four properties.

Every component behaved correctly. The engine executed the command it was given, twice, because it was given it twice. The link recovered successfully. The timeout fired at a reasonable bound.

The corruption is silent and unbounded. Nothing detects a double application, and a design that re-issues on every timeout can apply an operation many times under a flapping link.

And the host's belief is wrong in a specific, actionable-looking way. It has a completion, a result, and no errors. It will act on V + Δ while memory holds V + 2Δ.

The correct behaviour depends on §33's classification, and for a non-idempotent command it is to query, not re-issue: the host re-establishes the command's state from the memory side using {cmd_id, generation}, and re-issues only if the memory side confirms the command was never accepted.

33. Idempotent and Non-Idempotent Commands

Three consequences.

Idempotence is a property of the operation and its operands together, not of the opcode. A transform reading and writing the same range is not idempotent even though a transform into a distinct range is. A design that classifies by opcode alone will classify some commands wrongly, and the wrong direction is catastrophic.

So the classification belongs in the descriptor or is derived conservatively. When in doubt, treat a command as non-idempotent — the cost is a query path; the cost of the opposite error is silent corruption.

And the query path is what makes exactly-once achievable at all. Without it, the host's only options after a timeout are to re-issue, which may double-apply, or to give up, which may abandon a completed operation and leave the memory in a state nobody has recorded. Neither is acceptable, which is why the memory side must retain command state across a recovery (§31).

34. SVA — At Most One Semantic Execution

// MANDATORY. The property that makes Section 32 detectable. Uses a verification
// reference model, because "this is the same command" is testbench knowledge.
int unsigned tb_semantic_executions [int];   // reference command id -> count
 
always @(posedge clk)
  if (cmd_semantically_executed) tb_semantic_executions[tb_ref_cmd_id]++;
 
property p_at_most_one_execution(int unsigned rid);
  @(posedge clk) disable iff (!rst_n)
    (tb_semantic_executions[rid] <= 1);
endproperty
a_at_most_one_execution: assert property (p_at_most_one_execution(REF_UT));
 
// Progress is monotonic and never exceeds the command length.
property p_progress_bounded_and_monotonic;
  @(posedge clk) disable iff (!rst_n)
    cmd_q[IDX].valid |-> (cmd_q[IDX].bytes_processed <= cmd_q[IDX].length)
                      && (cmd_q[IDX].bytes_processed >= $past(cmd_q[IDX].bytes_processed));
endproperty
a_progress_bounded_and_monotonic:
  assert property (p_progress_bounded_and_monotonic);
 
// The invariant that ties the three progress fields together (Section 20).
property p_progress_conserved;
  @(posedge clk) disable iff (!rst_n)
    cmd_q[IDX].valid |->
      (cmd_q[IDX].bytes_processed + cmd_q[IDX].remaining == cmd_q[IDX].length);
endproperty
a_progress_conserved: assert property (p_progress_conserved);
 
// A command is retired only after its result was consumed (Section 11).
property p_retire_after_result_consumed;
  @(posedge clk) disable iff (!rst_n)
    (cmd_q[IDX].state == CMD_FREE) && $past(cmd_q[IDX].valid)
      |-> $past(result_consumed[IDX] || cmd_failed_reported[IDX]);
endproperty
a_retire_after_result_consumed: assert property (p_retire_after_result_consumed);

Architecture. Four properties: exactly-once semantics, bounded monotonic progress, progress conservation, and retirement discipline.

Why the first must be verification-only. The wire carries {cmd_id, generation}. The knowledge that a re-issued command is semantically the same operation belongs to whatever generated the traffic — synthesising it into the design would build a duplicate-detector with the same blind spots as the thing it checks.

Why conservation is separate from bounds. Bounds catch a runaway; conservation catches the case where bytes_processed advanced and remaining did not, which is §20's separate-update bug and stays within bounds indefinitely.

DV. The first needs a recovery, a timeout, and a re-issue in sequence — §42's third trace, and it never occurs spontaneously.

35. What Actually Limits Near-Memory Throughput

achieved_nmc_throughput  <=  min(
    local memory bandwidth available to the engine,   // Section 15's share
    engine compute throughput,
    command issue rate across UCIe,
    result return rate across UCIe,
    result buffer drain rate                          // Section 23
)

Worked, with illustrative numbers. An engine capable of 100 GB/s of arithmetic; a memory share of 40 GB/s under §17's cap; a command rate of 10⁶ commands/s over 4 MiB each; a result return of 64 B per command.

memory share       = 40 GB/s                                  <- BINDING
engine compute     = 100 GB/s
command issue      = 1e6 cmd/s × 4 MiB = 4.19e12 B/s ≈ 4190 GB/s
result return      = 1e6 cmd/s × 64 B  = 6.4e7  B/s  ≈ 0.064 GB/s of link
                     (negligible — which is Section 5's whole point)
 
achieved <= min(40, 100, 4190, 0.064-as-link-cost) = 40 GB/s

Three readings.

The memory share binds, and it is a design parameter (§17's cap), not a physical limit. Raising the cap raises this bound and lowers host service. The trade is explicit, which is the value of having a cap at all.

Adding engine compute does nothing. 100 GB/s of arithmetic against a 40 GB/s memory share means 60% of the engine is idle by construction. More ALUs is the intuitive fix and the wrong one.

And UCIe is nowhere near binding — 0.064 GB/s of result traffic against a link that carries orders of magnitude more. That is the near-memory benefit working exactly as intended, and it is also why "the UCIe link is the NMC bottleneck" is almost always wrong.

36. Operational Intensity at the Engine

The roofline question, stated at the engine rather than at the system.

operational_intensity  =  operations / bytes read from local memory   [ops/byte]
 
if  operational_intensity × memory_share  <  engine_peak_ops
      -> MEMORY-BOUND. More ALUs do nothing (§35).
 
if  operational_intensity × memory_share  >  engine_peak_ops
      -> COMPUTE-BOUND. The engine is the constraint, and the operation was
         probably a poor near-memory candidate to begin with (§6, row 3).

Two consequences.

Most good near-memory candidates are memory-bound at the engine, because §6 selected for low arithmetic intensity. So the engine should be sized to consume its memory share and no more — and an engine sized far above that is silicon spent on a bound it cannot reach.

And a compute-bound near-memory operation is a warning sign. It means the arithmetic is substantial, which means the host's compute — generally more capable — would have done it faster, and the only remaining justification is the byte saving in §5. That justification can still hold, but it should be checked rather than assumed.

37. Batching

Several small operations can be carried in one descriptor.

Two readings.

Batching helps most where §5's benefit is weakest. For a 4 MiB reduction the 64-byte descriptor is already negligible; for a 2 KiB operation the descriptor is 3% of the traffic and batching matters. So batching is a fine-grained-operation technique.

And it interacts with chunking rather than replacing it. A batch is still one obligation, and an engine that executes a whole batch without yielding has re-created §21 at batch scale. Chunking must operate within the batch.

38. Making "It Helped" Measurable

// Diagnostic only. Every counter's incrementing EVENT is defined.
logic [63:0] cmds_accepted_q;
logic [63:0] cmds_completed_q;
logic [63:0] cmds_failed_q;
logic [63:0] semantic_executions_q;      // must equal cmds_completed + cmds_failed
logic [63:0] local_bytes_read_q;         // bytes the engine read from local memory
logic [63:0] result_bytes_returned_q;    // bytes returned across UCIe
logic [63:0] host_bytes_avoided_q;       // local_bytes_read - result_bytes_returned
logic [63:0] engine_mem_stall_q;         // engine wanted the controller, did not get it
logic [63:0] host_starved_cycles_q;      // host_wait_q at HOST_SERVICE_BOUND  <- Section 18
logic [63:0] result_slot_stall_q;        // admission refused for result capacity
logic [63:0] cmd_recovery_spanned_q;     // commands live across a UCIe recovery

Three properties of this list.

The first two must be reported together, always. host_bytes_avoided_q alone is the number a proposal quotes; host_starved_cycles_q is the number that decides whether the feature shipped or was reverted. §18 is a design where the first is maximal and the second is catastrophic.

semantic_executions_q must equal completions plus failures. Any excess is §32 happening in production. It is one comparison and it is the only field evidence of double-application there is.

And these must survive a recovery (14.5), because the events worth diagnosing are the ones that also reset things.

39. The Command Scoreboard

// Verification-only. THREE models: what was commanded, what memory should now
// contain, and what the host received.
class nmc_scoreboard;
 
  // ---- Layer 1: COMMAND SEMANTIC model.
  typedef struct {
    int              id;
    int              generation;
    int              opcode;
    bit [ADDR_W-1:0] src_base;
    bit [ADDR_W-1:0] dst_base;
    int              length;
    bit              accepted;
    int              semantic_executions;   // MUST be <= 1  (Section 34)
    int              bytes_processed;
    int              result_bytes_expected;
    bit              result_observed;
    bit              retired;
    bit              idempotent;            // Section 33 — from operands, not opcode
    bit              spanned_recovery;
  } cmd_model_t;
  cmd_model_t cmds [int];                   // keyed by {id, generation}
 
  // ---- Layer 2: MEMORY EFFECT model. What SHOULD have changed, and only that.
  bit [DATA_W-1:0] expected_mem [bit [ADDR_W-1:0]];
  bit              expected_written [bit [ADDR_W-1:0]];
 
  // ---- Layer 3: RESULT / COMPLETION model.
  typedef struct {
    bit              delivered;
    bit [DATA_W-1:0] value;
    int              matched_cmd_key;
  } result_model_t;
  result_model_t results [int];
 
  // ---- The check that catches Section 32.
  function void check_exactly_once(int key);
    if (cmds[key].semantic_executions > 1)
      $error("COMMAND %0d gen %0d executed %0d times (must be <= 1) — Section 32",
             cmds[key].id, cmds[key].generation, cmds[key].semantic_executions);
  endfunction
 
  // ---- The check that catches Section 28 — memory changed where it should not.
  function void check_no_collateral_writes(bit [ADDR_W-1:0] a, bit [DATA_W-1:0] got);
    if (!expected_written.exists(a) && got !== expected_mem[a])
      $error("ADDRESS %0h changed but no command wrote it — Section 28/29", a);
  endfunction
 
  // ---- The check that catches Section 12's alias.
  function void check_completion_binding(int comp_key, int expected_key);
    if (comp_key != expected_key)
      $error("COMPLETION bound to key %0d, expected %0d — stale alias (Section 12)",
             comp_key, expected_key);
  endfunction
 
  // ---- The check that catches Section 23 without waiting for the deadlock.
  function void check_reservation(int key);
    if (cmds[key].accepted && (cmds[key].result_bytes_expected > 0)
                           && !reservation_held(key))
      $error("COMMAND %0d accepted without a result reservation — Section 24", key);
  endfunction
 
endclass

Architecture. Three models keyed by command-and-generation, by address, and by completion.

Why Layer 2 tracks expected_written as well as expected_mem. A near-memory engine that writes outside its declared destination range is a class of bug no value comparison catches — the values it wrote may even be plausible. Tracking which addresses should have changed is what makes a stray write detectable (§28, §29).

Why the command key is {id, generation}. §12's alias is precisely a completion binding to the right id and the wrong instance. A model keyed by id alone reproduces the design's bug and passes.

And why reservation is checked at acceptance. §23's deadlock has no detectable moment — the system simply stops. Checking the invariant at admission reports the cause instead of waiting for the symptom that never arrives.

40. Flagship Trace 1 — a Reduction Command

Illustrative. A reduction over a range, executed in chunks, producing a small result. Cycle numbers illustrative.

Six readings.

Cycle 8 reserves and accepts in the same cycle. §24's design accepts here and reserves at cycle 63 — by which time it may be too late (§23).

Cycles 20, 32 and 44 are the yields. The host gets the controller between every chunk. §21's design has no rows between cycle 12 and cycle 63.

bytes_processed + remaining is 4096 at every row — §34's conservation property, visible.

Cycle 71 releases the slot on consumption, not on production at cycle 63. Releasing at 63 would let a new command reserve a buffer that still holds an undelivered result.

Cycle 72 retires the host entry after the result was consumed — not at cycle 3 when UCIe accepted the descriptor, which is §12.

And the whole command occupied the controller for roughly 32 of 72 cycles, leaving the rest to the host. That ratio is NMC_QUOTA doing its job (§17).

41. Flagship Trace 2 — Contention

Correct design, with chunk yielding and a host reserve.

Wrong design (§21 — no chunking, engine strict priority).

Three readings.

The correct design's worst host wait is 4 cycles; the wrong design's is over a quarter of a million. Same operation, same engine, same memory. The only difference is whether the engine yields.

host_wait_q saturates at AGE_MAX in the wrong design, which is why it must saturate rather than wrap — a wrapping counter would report a small number at cycle 100,000 and the starvation would be invisible in exactly the run that demonstrates it.

And the link is idle in both traces. Every link-level metric is identical. The difference is entirely inside the memory-side arbiter, which is why §38's host_starved_cycles_q exists.

42. Flagship Trace 3 — Recovery During Execution

Six readings, and this is the chapter's strongest section.

Cycle 35 is the fact that makes near-memory different from memory access. The engine is local to the memory; the link's state is irrelevant to its progress. A design that assumes a link event pauses the far side is wrong about the one thing that most matters here.

Cycle 52: the command finished during the recovery. All 4096 bytes processed, result produced, slot holding it. The host knows none of this.

Cycle 60: the timeout fires against a command that has already succeeded. The timeout was correct and its conclusion would be wrong.

Cycle 61 is the entire lesson. A re-issue here produces §32's double application. The command is non-idempotent by assumption, so the only safe action is to ask.

Cycles 62–66 are the query path, keyed by {id, generation} — which is why §14's generation must survive the recovery rather than being reset by it.

And cycle 75 releases the slot at consumption, 23 cycles after the result was produced. The reservation held across the entire recovery, which is why it had to be taken at admission rather than at result time.

43. Coverage

covergroup cg_nmc @(posedge clk);
  option.per_instance = 1;
 
  // --- Command shape (Sections 20, 37).
  cp_length : coverpoint cmd_length_class {
    bins tiny        = {0};    // shorter than one chunk
    bins one_chunk   = {1};    // exactly CHUNK_MAX
    bins many_chunks = {2};
    bins huge        = {3};    // Section 21's regime
  }
  cp_chunk_boundary : coverpoint chunk_boundary_event {
    bins none = {0}; bins mid = {1}; bins final = {2};
  }
  cp_batched : coverpoint command_was_batched;
 
  // --- Resources (Sections 22-26).
  cp_cmd_slots : coverpoint cmd_table_occupancy {
    bins empty = {0}; bins mid = {[1:MAX_COMMANDS-1]}; bins full = {MAX_COMMANDS};
  }
  cp_result_slots : coverpoint result_slots_q {
    bins none = {0};                       // admission MUST refuse here
    bins few  = {[1:2]};
    bins many = {[3:$]};
  }
  cp_simultaneous_reserve_release : coverpoint admit_and_consume_same_cycle;
  cp_admit_refused : coverpoint admit_refusal_reason {
    bins slot = {0}; bins result = {1}; bins context = {2}; bins engine = {3};
  }
 
  // --- Contention (Sections 15-19).
  cp_contention : coverpoint contention_class {
    bins host_only   = {0};
    bins engine_only = {1};
    bins both        = {2};                // the only interesting one
  }
  cp_host_wait : coverpoint host_wait_q {
    bins none = {0};
    bins some = {[1:HOST_SERVICE_BOUND-1]};
    bins at_bound = {HOST_SERVICE_BOUND};  // the override firing
  }
  cp_quota_exhausted : coverpoint nmc_quota_exhausted_in_window;
  cp_maint_during_exec : coverpoint maintenance_during_execution;
 
  // --- Semantics (Sections 27-34).
  cp_idempotent : coverpoint command_idempotence {
    bins idempotent = {0}; bins non_idempotent = {1};
  }
  cp_coherence : coverpoint coherence_interaction {
    bins none = {0}; bins host_cached_clean = {1}; bins host_cached_dirty = {2};
  }
  cp_stale_completion : coverpoint stale_generation_completion_arrived;
 
  // --- Transport composition (Sections 31, 32).
  cp_link_event : coverpoint link_event_during_command {
    bins none = {0};
    bins retry = {1};
    bins recovery_before_exec = {2};
    bins recovery_during_exec = {3};       // THE case — Section 42
    bins recovery_after_result= {4};
  }
  cp_timeout_after_exec : coverpoint host_timeout_after_execution_completed;
 
  // --- Crosses that carry the information.
  x_recovery_idem   : cross cp_link_event, cp_idempotent;      // Section 32
  x_contention_len  : cross cp_contention, cp_length;          // Section 41
  x_result_admit    : cross cp_result_slots, cp_admit_refused; // Section 24
  x_coh_write       : cross cp_coherence, cp_idempotent;       // Section 28
endcovergroup

Seven bins worth calling out:

cp_result_slots.none crossed with cp_admit_refused.result. §24's fix, proven active. If the first bin is hit and the second is not, admission is ignoring result capacity and §23 is reachable.

cp_simultaneous_reserve_release. §25's exact cycle, and the one a lightly-loaded test never produces.

cp_link_event.recovery_during_exec crossed with cp_idempotent.non_idempotent. §32's flagship failure. This cross is the single most valuable bin in the chapter, and it requires deliberate injection.

cp_timeout_after_exec. §42's cycle 60 — a timeout firing against a command that already succeeded. Most environments never construct it.

cp_host_wait.at_bound. §17's override firing, proving the bound is real.

cp_coherence.host_cached_dirty. §28's second and worse direction, where the engine computes from stale input.

And cp_contention.both. Host and engine requesting simultaneously — the only class in which the arbiter does anything at all, and a surprisingly common gap.

44. Debug Taxonomy

Row 1 is the one that reaches production. The near-memory feature's own metrics are at their best while the system is at its worst, so a design reviewed on host_bytes_avoided_q alone will ship §18.

45. Debug Checklist

1. Which command — identity and generation? (§14)
2. Was it accepted, and at what cycle? (§9)
3. Was result capacity reserved at acceptance? (§22, §26)
4. What is the command's state, and how long has it been there? (§11)
5. What is bytes_processed, and does it plus remaining equal length? (§34)
6. Is bytes_processed advancing? If not, is it waiting on the controller? (§15)
7. What is the engine's quota, and is it exhausted in this window? (§17)
8. What is the host's worst wait, and did the override fire? (§17, §19)
9. Is CHUNK_MAX bounded above by HOST_SERVICE_BOUND? (§20)
10. How many result slots are free, and how many are held by undelivered results? (§25)
11. Was any command admitted while result slots were zero? (§24)
12. Is the operation idempotent — judged from operands, not opcode? (§33)
13. Did a UCIe retry or recovery occur during execution? (§31)
14. Did the host time out, and did it re-issue or query? (§32, §33)
15. Does semantic_executions_q equal completions plus failures? (§38)
16. Is the buffer cached on the host, clean or dirty? (§28)
17. Was an ownership handoff performed, and is the context owned by the engine? (§30)
18. Did the engine write outside its declared destination range? (§39)
19. When was the host's entry freed — at semantic completion or at a transport event? (§12)
20. Which of the three scoreboard layers diverged first? (§39)

46. Common Misconceptions

"Near-memory compute is automatically faster." It wins only when the bytes avoided across the boundary greatly exceed the command, result and coherence bytes added. A reduction over 4 MiB wins by roughly three orders of magnitude; a transform that returns the same volume it consumed wins by about 2×; and a compute-bound operation may lose outright (§5, §6).

"Closer compute means lower latency for every workload." The engine is generally less capable per byte than the host's compute. For a compute-bound operation, moving it toward memory moves it to slower arithmetic against a boundary that was never the constraint (§6, §36).

"It eliminates memory bandwidth limits." It relocates the consumer of existing memory bandwidth from the far side of the link to the near side. The total is unchanged, and now the engine and the host compete at the same arbiter (§15).

"UCIe bandwidth is the main near-memory bottleneck." In the worked example the result traffic is a fraction of a GB/s against a link carrying far more. The binding constraint is almost always the engine's share of the memory controller — which is a design parameter, not a physical one (§35).

"Commands can be retried like packets." A transport object is re-sendable precisely because re-sending has no semantic effect. A command has semantic effect, so a blind re-issue after a timeout can apply a non-idempotent operation twice, silently and without any error anywhere (§9, §32).

"Result capacity can be allocated after execution." It cannot. A command that executes and has nowhere to put its result holds an engine, a table entry and controller resources indefinitely, and no amount of waiting frees anything. Reserve at admission (§23, §24).

"Memory-local writes are automatically coherent." They are not. A host cache can hold a stale clean copy, in which case the host reads old data; or a dirty copy, in which case the engine reads stale input and computes a confidently wrong answer. Either coherent participation or an explicit ownership handoff is required (§27, §28).

"One long operation should keep the controller until it finishes." An unchunked 1 GiB command can occupy the controller for millions of accesses, during which host memory latency is effectively unbounded. The design is functionally perfect and the system is unusable, and no arbitration policy can fix it because the engine never yields (§18, §21).

"A timeout means the command did not execute." The engine is local to the memory and does not stop when the link does. It may have completed during the recovery with its result waiting — so the correct response for a non-idempotent command is to query, not to re-issue (§31, §32, §42).

"More compute units always improve near-memory throughput." If the operation is memory-bound at the engine — which §6 selected for — then additional ALUs are idle by construction. 100 GB/s of arithmetic against a 40 GB/s memory share leaves 60% of the engine unused (§35, §36).

47. Understanding Check

48. Summary and What Comes Next

Near-memory compute trades data movement for command movement, and it wins only when the bytes avoided greatly exceed the command, result and coherence bytes added. A reduction wins by orders of magnitude; a result-preserving transform wins by about 2×; a compute-bound operation may lose.

A command is an obligation the host cannot withdraw. Not by timing out, not by resetting, not because the link went away — so the host's entry outlives its own uncertainty, and the identity carries a generation so a stale completion becomes a reported orphan rather than a wrong answer.

Reserve before you commit. Result capacity checked at admission is the difference between backpressure and an unrecoverable deadlock holding an engine, a table entry and controller state.

The engine does not add memory bandwidth; it relocates the consumer of it. Host and engine now contend at the same arbiter, which needs a reserved host share, an engine quota, and two liveness bounds in opposite directions.

Chunking is structural, not policy. An engine that never yields gives the arbiter nothing to arbitrate, and the failure scales with an input length the hardware does not control.

A memory-local writer behind coherent caches is wrong in two directions — a stale host read, and a confidently wrong result computed from stale input — and both are silent.

The engine does not stop when the link does. A timeout says no completion arrived; it never says the command did not execute. For a non-idempotent command the only safe response is to query, and idempotence is a property of the operands, not the opcode.

And the metrics must be reported in pairs. Bytes avoided is the number a proposal quotes; host-starvation cycles is the number that decides whether the feature ships.

This chapter moved the operation toward the memory and found that the memory controller became the contended resource. The next chapter confronts a memory subsystem whose entire value is that it has many controllers and many independent paths — and asks what it takes to keep all of them usefully busy from the far side of a chiplet boundary.

• 17.4 — HBM Integration — HBM stacks attached over UCIe.

Browse the full path on the UCIe tutorials index.