AI Accelerator — What the Attach Model Hides

The four chapters before this one measured things: microseconds (29.1), an efficiency product (29.2), a deterministic headroom (29.3), a loss distribution (29.4). This one is about a choice that decides which of those a programmer is allowed to see at all.

1. Sources, Scope, and What This Chapter Refuses to Do

2. The Two Models, Described by What the Programmer Sees

Four readings.

Row 2 is the entire trade. An explicit copy is expensive and legible; a coherent access is cheap to write and illegible. Every other row follows from that one.

Row 3 is where debugging diverges. With a copy there is a moment to assert about — 29.1's retirement, 29.3's completion. A coherent access has no such moment in the source code, so a correctness question about when data became valid has no line to point at.

Row 4 is the one that produces silicon bugs, and §6–§9 are about it. "Exactly one side at a time, by construction" is a very strong property that programmers get for free from an explicit copy and must be engineered deliberately in the other model.

And row 7 is the honest summary. The coherent model does not remove the transfer arithmetic of the previous four chapters — it relocates it below the programmer's visibility. The bytes still cross the link; the headroom still binds; the utilisation still climbs. Nobody can see it in the program text.

3. The Attach Boundary

4. The Break-Even — When Each Model Wins

Explicit copy pays for the whole buffer regardless of f:

T_copy = 168 + f × 65536 × k × 0.001 µs

Coherent attach pays only for what it touches, at the remote price:

T_attach = f × 65536 × k × 0.5 µs

Setting them equal and solving for the product f × k:

168 = f × k × 65536 × (0.5 − 0.001) → f × k ≈ 0.0051

Five readings.

The break-even is astonishingly low: about half a percent of the buffer, touched once. For dense work — anything that reads most of its inputs — explicit copy wins by two to three orders of magnitude. The bulk transfer amortises so well that it is not a close call.

Which reframes the usual argument. Coherent attach is not, on this arithmetic, a performance technique for dense workloads. It wins where access is sparse and unpredictable — pointer-chasing, sparse embeddings, large tables of which a tiny fraction is consulted per step. That is a real and important class, and it is a narrower class than the enthusiasm suggests.

And the third reading is the one worth taking to a design review: reuse strengthens the copy case, not the attach case. Both models pay k times, but copy pays the local price. Every additional access widens copy's lead — so "we reuse the data heavily" is an argument for copying, not against it, which is the opposite of the intuition that reuse justifies a shared view.

The real reason to want a coherent attach is usually not in this table. It is programmability — the model in §2 row 1, where the program simply accesses memory. That is a legitimate and substantial benefit, and it should be argued on its own terms rather than as a performance claim the arithmetic does not support.

And one honest limitation: f is often unknown at design time. A workload whose touched fraction depends on input data has no fixed f, and the model choice is then a bet on a distribution. §12 treats that as a decision requiring evidence rather than a preference.

5. What Each Model Does to the Failure Surface

This is the part the break-even table cannot express.

Three readings.

Row 5 is the most consequential and the least discussed. In the copy model, a correctness bug about timing has a place to be fixed — move the copy, add a wait, reorder. In the coherent model the corresponding bug has no corresponding line, and the fix is a change to synchronisation the programmer may not have known was load-bearing.

Row 4 is why 29.2's instrumentation becomes mandatory rather than nice. Under explicit copy, bytes crossed can be counted in software. Under a coherent attach, if the hardware does not count, nobody counts — and the efficiency cascade becomes unmeasurable from the host.

And row 6 explains the characteristic experience of the coherent model. The profile shows the compute engine stalling and attributes it to compute, because that is where the stall is observed. The cause is data movement that has no representation in the profile at all.

6. The State Machine the Coherent Model Requires

Ownership is the minimum this model needs, and it is a four-state problem — not two.

Three readings.

PENDING is the state designs omit, and §7 is what happens when they do. It exists for the same structural reason as 29.3 §4's dead time: a request crosses a link and is answered later, so there is an interval during which the device has asked and does not yet have. A two-state design has nowhere to put that interval, so it puts it in the wrong state.

And an epoch is required alongside the state, for the reason 29.1 §8 needed a generation: a response can arrive after the request that prompted it has been abandoned — because the line was invalidated, the context switched, or a reset occurred. Without an epoch, a stale grant is indistinguishable from a fresh one.

The transition that matters is PENDING → OWNED, and it must be driven by the answer, never by the request. That single distinction is this chapter's RTL content, and it is the same request-versus-grant confusion that produces §7's timeline.

7. Wrong RTL — the Missing State

// WRONG. ILLUSTRATIVE. A two-state ownership tracker. It reads as obviously
// correct because the ONLY missing thing is an interval, and intervals are
// exactly what does not appear in a state list.
typedef enum logic [1:0] { S_NONE, S_OWNED } own_e;   // BUG 1: no PENDING
 
own_e own_q;
logic [63:0] line_data_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    own_q <= S_NONE;
  end else begin
    case (own_q)
      S_NONE: begin
        // BUG 1: the state advances on the REQUEST being sent, not on a grant.
        //        From here on, the device believes it owns the line while the
        //        request is still in flight.
        if (own_req_fire) own_q <= S_OWNED;
      end
      S_OWNED: begin
        if (invalidate_fire) own_q <= S_NONE;
      end
      default: own_q <= S_NONE;
    endcase
 
    // BUG 2: a grant is applied whenever one arrives, with no check that it
    //        corresponds to the request currently outstanding. A grant for an
    //        abandoned request installs stale data (§6's epoch argument).
    if (grant_fire) line_data_q <= grant_data;
  end
end
 
// BUG 3: writes are permitted from S_OWNED, which BUG 1 can enter without ever
//        having been granted exclusivity. Two writers, silently.
assign write_permitted = (own_q == S_OWNED);

Architecture. Two states, a data register, and a write permission derived from the state.

State. own_q and line_data_q. The missing state is PENDING, and the missing field is an epoch — so the design cannot represent either "I have asked" or "which asking this answers."

Event. own_q advances on own_req_fire — the request leaving. That is the defect in one signal name: the transition is driven by the device's own action rather than by the answer.

Contract. write_permitted is consumed by the datapath as "exclusivity has been established." It actually means "a request was sent at some point and no invalidate has arrived since."

Failure — the timeline. The host and the device both write the same line. The device's request takes 600 ns.

First divergence: 20 ns — a write performed from a state the design entered on a request. Everything after is a consequence.

Root cause. The state machine conflates asking with having, because there is no state for the interval between them. The grant at 600 ns then overwrites the device's local write with the version it fetched, so the device's own store vanishes.

BUG 2 is the second half. Even with a PENDING state, applying any grant that arrives installs data that may answer an abandoned request. The epoch is what makes a grant matchable to its request (29.1 §8 established this pattern for completions; here it protects ownership).

And BUG 3 is why it is silent. The lost write is a data value, not a protocol event. No PCIe error, no timeout, no counter — the fabric delivered everything correctly, and the accelerator computed on a line whose history is unrecoverable.

DV/debug. The report is "results differ between runs" or "results differ from the CPU reference by a tiny amount." A protocol analyser is useless — the transactions were all legal. The discriminator is a state trace showing a write issued while a request was outstanding, which is precisely what a PENDING state would have made a detectable condition.

8. Corrected RTL — Request, Grant, Epoch

// CORRECT. ILLUSTRATIVE. Four states, an epoch, and one rule: writes require a
// state the design can only reach by receiving an answer.
typedef enum logic [1:0] { S_NONE, S_PENDING, S_SHARED, S_OWNED } own_e;
 
own_e        own_q;
logic [3:0]  epoch_q;          // increments whenever a request is abandoned
logic [3:0]  req_epoch_q;      // epoch stamped on the outstanding request
logic [63:0] line_data_q;
logic [31:0] stale_grant_q;    // grants dropped as stale — observability
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    own_q <= S_NONE; epoch_q <= '0; req_epoch_q <= '0; stale_grant_q <= '0;
  end else begin
    case (own_q)
      S_NONE: if (own_req_fire) begin
        own_q       <= S_PENDING;              // the request, not the ownership
        req_epoch_q <= epoch_q;                // stamp it
      end
 
      S_PENDING: begin
        // The ONLY path to OWNED: a grant whose epoch matches this request.
        if (grant_fire && (grant_epoch == req_epoch_q))
          own_q <= grant_is_excl ? S_OWNED : S_SHARED;
        // Abandonment bumps the epoch, so a late grant can never be applied.
        else if (invalidate_fire || req_abort_fire) begin
          own_q   <= S_NONE;
          epoch_q <= epoch_q + 4'd1;
        end
      end
 
      S_SHARED: begin
        if (invalidate_fire)      own_q <= S_NONE;
        else if (upgrade_req_fire) begin
          own_q       <= S_PENDING;
          req_epoch_q <= epoch_q;
        end
      end
 
      S_OWNED: if (invalidate_fire) begin
        own_q   <= S_NONE;
        epoch_q <= epoch_q + 4'd1;             // any view we held is now void
      end
 
      default: own_q <= S_NONE;
    endcase
 
    // Data is installed ONLY from a matching grant. A mismatched grant is
    // counted, never applied — an invariant nobody can observe is an assumption.
    if (grant_fire) begin
      if (grant_epoch == req_epoch_q) line_data_q  <= grant_data;
      else                            stale_grant_q <= stale_grant_q + 32'd1;
    end
  end
end
 
// Writes require OWNED, which is now reachable only via a matched grant.
assign write_permitted = (own_q == S_OWNED);
assign read_permitted  = (own_q == S_OWNED) || (own_q == S_SHARED);

Architecture. Four states, two epoch registers, a stale-grant counter, and two permission signals instead of one.

State. own_q, epoch_q, req_epoch_q. epoch_q increments on abandonment, never on success — so a stale grant is one whose stamp no longer matches, and matching is a single equality.

Event. S_PENDING → S_OWNED fires only on grant_fire && epoch match. Every other route into OWNED has been removed, which is the entire correction.

Contract. grant_epoch must be the epoch the requester stamped, echoed back. If the responder does not carry it, this scheme cannot work — and that is a bus-interface requirement, not something the tracker can enforce alone. It must appear in the interface specification.

Failure. The realistic residual is epoch wraparound: four bits wrap after 16 abandonments, so a grant delayed across a full wrap could match falsely. The width is a derivation, not a guess, and it belongs beside the declaration:

EPOCH_BITS ≥ ceil( log2( max_grant_latency × max_abandon_rate ) ) + 1

Worked, illustratively. If a grant can take up to 2 µs and the tracker can abandon a request at most once every 50 ns — one per invalidate, at the fastest rate invalidates can arrive — then at most 2 000 ÷ 50 = 40 abandonments fit inside one grant's flight time. ceil(log2(40)) + 1 = 7 bits. Four bits is not enough for that configuration, and §13's third negative case is how you demonstrate it rather than argue about it.

The + 1 is the part that gets dropped. Without it the epoch space is exactly the number of abandonments, so the last value wraps onto the outstanding stamp. One spare bit costs one flop and removes the boundary case entirely.

And read_permitted being separate from write_permitted is the SHARED state earning its place. A read-only view is useful and safe; collapsing SHARED into OWNED to save a state means every read requires exclusivity, which serialises readers for no reason.

9. Checks and Assertions

// MANDATORY. English: a write is never permitted unless the state is OWNED and
// OWNED was entered from PENDING via a matched grant. This is §7 BUG 1 and
// BUG 3 as a single property.
a_no_write_without_grant: assert property (
  @(posedge clk) disable iff (!rst_n)
    write_permitted |-> (own_q == S_OWNED)
);
 
// MANDATORY. English: OWNED is only ever entered from PENDING. Catches any new
// transition added later that shortcuts the handshake — the exact regression
// this chapter exists to prevent.
a_owned_only_from_pending: assert property (
  @(posedge clk) disable iff (!rst_n)
    ((own_q == S_OWNED) && (own_q != $past(own_q))) |-> ($past(own_q) == S_PENDING)
);
 
// MANDATORY. English: line data changes only on a grant whose epoch matched.
// Catches §7 BUG 2 — a stale grant installing data.
a_data_only_on_matched_grant: assert property (
  @(posedge clk) disable iff (!rst_n)
    ($changed(line_data_q)) |->
      ($past(grant_fire) && ($past(grant_epoch) == $past(req_epoch_q)))
);
 
// MANDATORY. English: a request is outstanding only in PENDING. Catches a
// request issued from a state that already holds a view, which would produce
// two outstanding requests with one epoch stamp.
a_request_only_when_pending: assert property (
  @(posedge clk) disable iff (!rst_n)
    own_req_fire |=> (own_q == S_PENDING)
);

Reading the four.

The second is the one that pays for itself over the design's lifetime. Today's code has one path into OWNED; the risk is the optimisation someone adds next quarter — a fast path, a prefetch, a "we already know we'll get it" shortcut. a_owned_only_from_pending rejects all of them mechanically, which is worth more than any review comment.

The third uses $changed with $past on both the event and the comparison, because the wrong behaviour is data appearing from an unmatched grant. It is the assertion that would have caught §7's 600 ns line — the moment the grant overwrote the device's own store.

And all four are strong formal targets. Four states, a 4-bit epoch, one data register: the state space is tiny and the properties are the design intent. This is the rare case where formal is the cheaper option, and §14 treats it that way.

10. Measured Behaviour

Four readings.

Row 1 is why this ships. At short request latency the PENDING interval is almost nothing, so the two-state design is almost right — and "almost right" at 10 ns means a bring-up bench with a close-coupled model finds nothing. The bug's incidence is proportional to the interval the missing state represents.

Which makes this a bug that appears when something gets further away. Moving from an on-die model to a real link, adding a switch hop, or raising load all lengthen the request latency and all increase the defect rate — while none of them touch the tracker's code. The change that exposes it looks unrelated to it.

Row 4 is the operating point that matters, and the rate deserves converting into something a team can feel. 9 100 lost writes per 10⁶ is roughly 1 in 110 contended accesses. At an illustrative 2 million contended accesses per second, that is about 18 000 corrupted values per second — and each one is a plausible-looking number rather than an error. High enough to change results, low enough to be blamed on numerical noise, which in a workload that tolerates small variation is the worst combination available.

And the scaling is roughly linear in request latency, which the table shows directly: 600 ns → 2 µs is a 3.3× latency increase and a 3.4× increase in lost writes at fixed contention. So the defect rate is, to first order, the fraction of time the device spends in the state the design does not have.

And the last column is the argument for the stale-grant counter. §8 does not merely avoid the failure; it reports that the condition occurred — 61 stale grants correctly dropped. A silent correct design and a silent broken design look identical, and the counter is what separates them.

11. Choosing an Attach Model — the Evidence Each Answer Needs

Three readings.

Row 3 applies 29.4 §4's statistical lesson to a different quantity. A workload whose touched fraction varies by input has a distribution of f, and choosing by its mean repeats exactly the error of choosing a buffer by average occupancy. The tail is what decides.

Row 4 is the intellectually honest position and it is a good one. Programmability is a real benefit; a model where the program simply accesses memory removes a whole class of orchestration bugs and a great deal of code. It should be chosen for that, with §4's cost stated, rather than defended with a performance claim the arithmetic contradicts.

And row 5 is the tie-breaker this module has earned. After four chapters of instruments that could not see what mattered, the model whose failures have a place to be fixed is the better default. 29.1's missing stage, 29.2's undecomposable ratio, 29.3's silent drop, 29.4's aliased burst — all four were visibility failures. The attach model is a chance to choose visibility up front.

12. Executable Counterexamples

Case 1 is listed first deliberately. It is the test most likely to be run and it cannot find this bug — §10 row 1. A close-coupled bring-up environment is structurally blind here, and the fix is a latency knob in the model, not more tests.

And case 6 is the honest one: it fails §8 too. A 4-bit epoch is not universally sufficient; the width is a derivation (§8), and this case is how you demonstrate the requirement rather than assert it.

13. Verification

Four readings.

Formal is the right primary tool and that is unusual in this module. 29.1–29.4 all needed statistical or long-run stimulus that formal handles badly. This chapter's property is a small-state safety invariant — exactly formal's strength — and §9's four assertions are close to a complete specification of the tracker.

Request latency must be a randomised parameter, not a constant, because §10 shows the defect rate is proportional to it. A suite with one latency value is testing one point on a curve whose interesting end is the far one.

"Stale-grant count non-zero" as a coverage point is the same idea as 29.4 §12's near-full residency: cover the state where the protection engaged. A suite where stale_grant_q never increments has never exercised the epoch, and the epoch is half the fix.

And the reset row is a real question with a real answer. A reset during PENDING must bump the epoch or clear req_epoch_q to a value no grant can match — otherwise an in-flight grant applies after reset. §8 clears both to zero on reset, which means a grant stamped 0 could match, and that is worth either fixing or documenting.

14. Debugging

Four readings.

Row 3 is what makes this the hardest bug in the module. The other four chapters produced symptoms that were wrong — dropped packets, missing microseconds, a shortfall. This one produces a symptom that is indistinguishable from an expected, benign phenomenon. Small non-deterministic numerical differences between an accelerator and a CPU reference are normal, so the bug hides inside a known-acceptable noise band.

The sticky bit is one flop and it is decisive. "Was a write ever permitted while a request was outstanding?" is a yes/no that converts an open-ended numerical investigation into a state-machine bug report. It is the cheapest instrument in this module and it answers the hardest question.

And it is worth noting what all five chapters share. 29.1: a stage the instrument omitted. 29.2: a ratio that could not be decomposed. 29.3: a drop with no counter. 29.4: a burst between two samples. 29.5: a write from a state that should not exist. Every one was legible from inside the device and invisible from outside it, and in every case the fix cost less than the investigation.

The general lesson is a design habit rather than a technique. Instrument the thing that would be impossible to reconstruct afterwards — a peak, a cause, an epoch, a "did this ever happen" bit. None of them cost meaningful area, and each one replaces a week.

15. Misconceptions

"A coherent attach is faster because it avoids the copy." §4: for dense access it is 10²–10³× slower, because a bulk copy amortises and per-line remote access does not. Break-even is f × k ≈ 0.005.

"Heavy reuse justifies a shared view." §4: reuse strengthens the copy case — both models pay k times, and copy pays the local price. Reuse is an argument for copying.

"Coherency means we don't have to think about data movement." §2 row 7, §5 row 4: the bytes still cross the link. The arithmetic moves below the programmer's visibility; it does not disappear.

"The device requested ownership, so it owns the line." §6, §7: a request crosses a link and is answered later. The interval between is a state, and omitting it permits a write into a line the device does not hold.

"If a grant arrives, apply it." §7 BUG 2, §8: a grant may answer an abandoned request. Without an epoch, a stale grant is indistinguishable from a fresh one.

"It passed bring-up, so the tracker is correct." §10 row 1, §12 case 1: at 10 ns request latency the defect rate is ~0. The bug's incidence scales with the latency the bring-up environment minimised.

"There's no error, so the data is fine." §7, §14: the lost write is a value, not an event. Every transaction was legal and the fabric can prove it.

"Choose the attach model on performance." §11 rows 4–5: programmability is a legitimate reason to choose a coherent attach, and it should be argued as such. And when the models are close, the legible failure surface should decide (§5).

16. Understanding Check

Q1. An accelerator has a 4 MiB working set. Should the host copy it or attach coherently?

Compute the break-even; for dense access it is not close (§4). With an illustrative 168 µs bulk copy at 25 GB/s, 0.001 µs local access and 0.5 µs remote access over 65 536 lines, the two models are equal when f × k ≈ 0.0051 — where f is the touched fraction and k the accesses per touched line. Touch the whole buffer once and explicit copy wins by roughly 196×; with 8× reuse, by ~1 400×. Coherent attach only wins below about half a percent of the buffer touched — sparse embeddings, pointer chasing, large tables consulted thinly.

Two consequences engineers get backwards. Reuse favours copying, because both models pay k times and copy pays the local price — so "we reuse heavily" argues for the copy. And if f is input-dependent, the decision needs its distribution, not its mean (§11 row 3), which is 29.4 §4's lesson applied to a different quantity. The defensible reason to attach coherently is usually programmability (§11 row 4) — a real benefit, worth stating on its own terms rather than as a performance claim.

Q2. Why is if (own_req_fire) own_q <= S_OWNED; wrong, and what exactly goes wrong?

It advances on the device's own action instead of on the answer, and there is no state for the interval between them (§6, §7). A request crosses a link and is answered later — structurally the same dead time as 29.3 §4's in-flight completions — and a two-state machine has nowhere to put it, so it puts it in OWNED.

§7's timeline: at 0 ns the request is sent and own_q becomes OWNED; at 20 ns the device writes into a line it does not own; at 100 ns the host writes the legitimate value; at 600 ns the grant arrives with the host's data and overwrites the device's store. The device then reads a value with its own write missing, computes on it, and no error is raised anywhere — the lost write is a value, not an event, and every transaction was legal.

The fix has two halves. A PENDING state whose only exit to OWNED is grant_fire, and an epoch stamped on the request and echoed on the grant — because a grant may answer a request that was abandoned by an invalidate or a reset, and without a stamp a stale grant is indistinguishable from a fresh one.

Q3. This passed bring-up and fails in the lab. Explain why, and how you would have caught it.

Because the defect rate is proportional to request latency, and bring-up minimised it (§10). At 10 ns the PENDING interval is almost nothing and the two-state design is almost right: ~0 failures. At 600 ns under contention it is ~9 100 lost writes per 10⁶ — roughly 1 in 110 contended accesses. So the change that exposes it is anything that makes the peer further away: a real link instead of a model, an extra switch hop, higher load. None of them touch the tracker's code, which is why the exposing change looks unrelated.

The catch is a stimulus change, not more tests. Make request latency a first-class randomised parameter with bins far above the bring-up value, and add an adversarial peer agent contending for the same line (§13). Then prefer formal: four states, a 4-bit epoch and one data register is a tiny state space, and §9's four properties are close to a full specification — this is the rare case where formal is the cheaper tool, and it is the opposite of 29.1–29.4, which all needed long-run statistical stimulus.

Q4. The report is "results differ slightly from the CPU reference and differ between runs." Why is this the hardest failure in the module, and what is the cheapest instrument?

Because the symptom is indistinguishable from a phenomenon that is normal and expected (§14). Small non-deterministic differences between an accelerator and a CPU reference arise legitimately from reduction order and precision, so this bug hides inside a known-acceptable noise band — unlike the other four chapters, whose symptoms were unambiguously wrong. Meanwhile every transaction was legal and a protocol analyser shows nothing (25.9), so the fabric is exonerated and the compute kernel takes the blame.

The cheapest instrument is a sticky bit: "was a write ever permitted while an ownership request was outstanding?" One flop, a yes/no answer, and it converts an open-ended numerical investigation into a state-machine bug report. stale_grant_q corroborates — a non-zero count on a design that claims stale grants cannot occur.

And this is the module's pattern. A missing stage, an undecomposable ratio, an uncounted drop, an aliased burst, a write from an impossible state: every one was legible from inside the device and invisible from outside it. The habit that follows is to instrument whatever could not be reconstructed afterwards — a peak, a cause, an epoch, a did-this-ever-happen bit. None cost meaningful area, and each replaces a week.

17. Module 29 Complete

Where this module sits. Module 26 described what each endpoint class is. Module 29 traced what each one does and, in every case, found that the hard part was not the protocol — it was knowing what actually happened. The next module continues at the system level, and 30.1 Architecture Checklist turns these five investigations into checks that run before silicon rather than after it.