AI Accelerators — When the Link Is Not the Bottleneck

Module 26 has read four endpoint classes whose performance lived on the PCIe link. This one closes the module with an endpoint where it usually does not — and where a chapter about PCIe has to explain why a PCIe number does not predict the system's throughput.

1. Sources, Scope, and What This Chapter Refuses to Do

2. Where PCIe Sits

In most accelerator systems PCIe is the host boundary: it carries commands, working-set transfers in and results out. It is not usually the path the computation runs over.

Three readings.

Rows 4 and 5 are where the execution bandwidth is, and neither crosses PCIe. That is the structural reason a fast link does not make compute faster — it was never on the compute path.

Row 6 matters because it is frequently assumed to be PCIe and frequently is not. Vendors document dedicated scale-up interconnects for multi-accelerator traffic; assuming collectives ride the host link produces a system model that is wrong about the dominant traffic in distributed training.

And rows 2–3 are the ones a system architect actually controls. They are per-job costs, they are on PCIe, and whether they are visible in the wall-clock depends entirely on whether they overlap with compute (§6).

3. Four Bandwidths, Not One

Three readings, and the middle one is the whole point.

The gap between rows 1 and 2 is one to two orders of magnitude, and it is not a defect — it is the architecture working. Device memory feeds compute at every layer; the host link moves a working set once. They are sized for completely different duty cycles.

Which is why the two figures in row 2 must not be compared with each other either. They are different products, different memory generations, different capacities. They appear together to establish a magnitude, and both are peak figures (22.6's discipline applies: a peak bounds a ceiling and predicts no workload).

And the practical consequence is a rule about claims. "We upgraded to a faster PCIe generation" predicts a change in rows 1's contribution to wall-clock — which may be zero if row 1 was already hidden behind compute (§6). The honest statement names the path.

4. The System Boundary

5. Command and Data Are Different Problems

Two readings.

They fail in opposite regimes, which is why one benchmark cannot characterise the interface (§10). Small jobs expose the command path's latency; large jobs expose the data path's bandwidth — and a system tuned on one is untested on the other.

And they are separable resources, which is the enabling fact for §6. Command submission for job N+1 can proceed while job N computes, because they use different mechanisms.

6. The Serialisation Trap

The dominant performance question at this boundary is not how fast the link is. It is whether the link is busy at the same time as the compute.

Per job: input copy 20 ms, compute 50 ms, output copy 10 ms.

Serialised — copy in, wait, compute, wait, copy out, wait, next job:

T_job = 20 + 50 + 10 = 80 ms → 12.5 jobs/s, and the link is idle for 50 of every 80 ms while compute is idle for 30.

Pipelined — three independent resources, steady state:

T_job ≈ max(20, 50, 10) = 50 ms → 20 jobs/s

Four readings, and the last two rows are the chapter in one table.

Halving the input copy time buys 14 % serialised and exactly nothing pipelined. In the pipelined case the input copy was already hidden behind compute, so making it faster changes a term that was not in the maximum. This is why a PCIe upgrade can produce no measurable improvement while being genuinely faster.

The pipelined formula holds only under stated conditions, and they are worth naming: at least three jobs' worth of buffering in device memory, independent copy and compute resources, no dependency forcing job N+1's input to wait on job N's output, and enough queue depth that the host is never the limit. Break any one and the formula degrades toward the serial sum.

And max() means the slowest stage sets the rate, so the useful optimisation target is whichever stage that is. If compute is the maximum, the link is irrelevant to throughput — which is the situation §3's magnitude gap makes common.

7. RTL — a Three-Stage Job Tracker

Overlap requires that jobs in different stages be tracked independently. That is the mechanism.

// ILLUSTRATIVE. A job table that permits one job per stage concurrently. The
// generation field is what makes a completion attributable after a slot is
// reused (§8), and the per-stage occupancy counts are what make §6's pipeline
// observable rather than assumed.
typedef enum logic [2:0] {
  J_FREE, J_COPY_IN, J_READY, J_EXECUTE, J_COPY_OUT, J_COMPLETE
} jstage_e;
 
localparam int N_JOB = 8;
 
jstage_e            job_stage_q [N_JOB];
logic [GEN_W-1:0]   job_gen_q   [N_JOB];
logic [ADDR_W-1:0]  job_host_buf_q [N_JOB];
logic [ADDR_W-1:0]  job_dev_buf_q  [N_JOB];
logic [LEN_W-1:0]   job_bytes_q [N_JOB];
logic [3:0]         job_deps_q  [N_JOB];      // outstanding dependencies
 
// Per-stage occupancy — the pipeline's own instrumentation (§14).
logic [3:0] n_copy_in_q, n_execute_q, n_copy_out_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    for (int j = 0; j < N_JOB; j++) begin
      job_stage_q[j] <= J_FREE;
      job_gen_q[j]   <= '0;
    end
    n_copy_in_q <= '0; n_execute_q <= '0; n_copy_out_q <= '0;
  end else begin
    // Admission on ACCEPTANCE (valid && ready), never on offering — the
    // general rule from 23.3, and counting the offer overstates depth.
    if (submit_valid && submit_ready) begin
      job_stage_q[alloc_j]    <= J_COPY_IN;
      job_gen_q[alloc_j]      <= job_gen_q[alloc_j] + 1'b1;   // NEW generation
      job_host_buf_q[alloc_j] <= submit_host_buf;
      job_dev_buf_q[alloc_j]  <= submit_dev_buf;
      job_bytes_q[alloc_j]    <= submit_bytes;
      job_deps_q[alloc_j]     <= submit_deps;
    end
 
    // Stage advance is EVENT-driven, and each transition has exactly one cause.
    for (int j = 0; j < N_JOB; j++) begin
      unique case (job_stage_q[j])
        J_COPY_IN:  if (copy_in_done[j])  job_stage_q[j] <= J_READY;
        // READY waits on dependencies, so an independent job can overtake a
        // blocked one — which is what keeps the pipeline full (§6).
        J_READY:    if (job_deps_q[j] == '0 && exec_grant[j])
                                          job_stage_q[j] <= J_EXECUTE;
        J_EXECUTE:  if (exec_done[j])     job_stage_q[j] <= J_COPY_OUT;
        J_COPY_OUT: if (copy_out_done[j]) job_stage_q[j] <= J_COMPLETE;
        // Retire only when the host has consumed the completion record —
        // freeing earlier is §8.
        J_COMPLETE: if (cpl_consumed[j])  job_stage_q[j] <= J_FREE;
        default: ;
      endcase
    end
 
    // One signed next-state expression per counter, so an entry and an exit in
    // the same cycle net correctly instead of losing one.
    n_copy_in_q  <= n_copy_in_q  + 4'(enter_copy_in)  - 4'(exit_copy_in);
    n_execute_q  <= n_execute_q  + 4'(enter_execute)  - 4'(exit_execute);
    n_copy_out_q <= n_copy_out_q + 4'(enter_copy_out) - 4'(exit_copy_out);
  end
end

Architecture. A job table where stage is per-job, not global — which is the structural difference between §6's two rows. A global stage register would enforce serialisation by construction.

State. Stage, generation, buffers, byte count, dependency count. job_deps_q is what allows an independent job to overtake a blocked one, and without it a single dependency stalls the whole pipeline.

Event. Admission on valid && ready; each stage advance has exactly one named cause; retirement requires the host to have consumed the completion record.

Contract. The host assumes a job's completion record is readable and correct when it is signalled, and that a slot is not reused until it has consumed it. §8 is what happens when the second half is skipped.

Failure. A global stage register serialises everything (§6's first row). Per-stage occupancy counters near zero on two of three stages is the observable signature, and §14 uses it.

DV/debug. n_copy_in_q, n_execute_q, n_copy_out_q are the pipeline's self-report. All three non-zero simultaneously is the definition of overlap working; one non-zero at a time is §6's serialised case measured rather than assumed.

8. Wrong RTL — Slot Freed on Fetch

// WRONG. ILLUSTRATIVE. The command slot is released as soon as the accelerator
// has fetched the command. The reasoning is sound-sounding: the command's
// CONTENTS have been consumed, so the host may reuse the slot. It confuses
// consuming the descriptor with completing the job.
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    for (int j = 0; j < N_JOB; j++) job_stage_q[j] <= J_FREE;
  end else begin
    if (submit_valid && submit_ready) begin
      job_stage_q[alloc_j] <= J_COPY_IN;
      job_dev_buf_q[alloc_j] <= submit_dev_buf;
    end
    for (int j = 0; j < N_JOB; j++) begin
      // BUG 1: the slot is freed when the command has been FETCHED, while the
      //        job is still executing. The host may now submit into this slot.
      if (job_stage_q[j] == J_COPY_IN && cmd_fetched[j])
        job_stage_q[j] <= J_FREE;
      // BUG 2: no generation, so a completion carries only the slot index and
      //        cannot be attributed to a particular use of that slot.
      if (exec_done[j]) cpl_slot <= j[$clog2(N_JOB)-1:0];
    end
  end
end

Architecture. A slot table freed at fetch, with completions identified by slot index alone.

State. Stage and device buffer. No generation — which is what makes BUG 1 corrupting rather than merely early.

Event. cmd_fetched[j] frees the slot.

Contract. The host's contract is that a free slot may be reused. This design offers a slot back while the job that occupied it is still running.

Failure — the timeline. Job A occupies slot 2; the host reuses slot 2 for job B while A is executing.

First divergence: t1 — the slot was freed on fetch. The visible symptom is at t5, and it is a plausible-looking wrong answer.

Root cause. Consuming a command and completing a job are different events, and the slot's lifetime must be bounded by the second. BUG 2 removes the last line of defence: with a generation, the completion at t3 would carry A's generation and be detectably stale.

And it survives testing because it needs slot pressure. With eight slots and fewer than eight jobs in flight, the host never reuses a slot while its previous occupant is live — so the bug requires a queue depth the test suite may never reach.

9. Corrected — Retire on Consumption, Attribute by Generation

// CORRECT. ILLUSTRATIVE. Two changes: the slot lives until the host has
// consumed the completion record (§7's J_COMPLETE), and every completion
// carries the generation of the job that produced it.
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    for (int j = 0; j < N_JOB; j++) begin
      job_stage_q[j] <= J_FREE; job_gen_q[j] <= '0;
    end
    stale_cpl_q <= '0;
  end else begin
    if (submit_valid && submit_ready) begin
      job_stage_q[alloc_j] <= J_COPY_IN;
      job_gen_q[alloc_j]   <= job_gen_q[alloc_j] + 1'b1;
    end
 
    for (int j = 0; j < N_JOB; j++) begin
      if (exec_done[j] && (job_stage_q[j] == J_EXECUTE)) begin
        cpl_slot <= j[$clog2(N_JOB)-1:0];
        cpl_gen  <= job_gen_q[j];              // attribution travels with it
        job_stage_q[j] <= J_COPY_OUT;
      end
      // The slot returns to FREE only after the host has read the record.
      if ((job_stage_q[j] == J_COMPLETE) && cpl_consumed[j])
        job_stage_q[j] <= J_FREE;
    end
 
    // Defence in depth: a completion whose generation does not match the slot's
    // current generation belongs to a previous use. Count it, never deliver it.
    if (cpl_in_valid && (cpl_in_gen != job_gen_q[cpl_in_slot]))
      stale_cpl_q <= stale_cpl_q + 32'd1;
  end
end

Architecture. Slot lifetime bounded by host consumption, plus a generation that makes any residual mis-attribution detectable.

State. job_gen_q per slot; stale_cpl_q should be zero forever.

Event. Retirement requires cpl_consumed[j] — an event that comes from the host side, not from the accelerator's own progress.

Contract. The host must acknowledge consumption, which is a real obligation and must be in the driver contract. A design that retires on a timer instead reintroduces §8 with a delay, and the delay makes it rarer rather than absent.

Failure. The residual risk is a generation width too narrow: at 4 bits a slot reused sixteen times while one completion is outstanding aliases as current. Unlikely, and worth stating rather than discovering.

10. Checks and Assertions

// MANDATORY. English: a job slot is never returned to FREE while the job
// occupying it is still executing. This is §8 BUG 1 stated directly.
a_no_free_while_executing: assert property (
  @(posedge clk) disable iff (!rst_n)
    ($fell(job_stage_q[chk] != J_FREE))
      |-> ($past(job_stage_q[chk]) == J_COMPLETE)
);
 
// MANDATORY. English: a completion is delivered only when its generation
// matches the slot's current generation. Catches §8's t4 mis-attribution even
// when slot pressure causes reuse.
a_cpl_generation_matches: assert property (
  @(posedge clk) disable iff (!rst_n)
    cpl_deliver_fire |-> (cpl_in_gen == job_gen_q[cpl_in_slot])
);
 
// MANDATORY. English: the three stage-occupancy counters never exceed the job
// table. Catches a stage transition that increments one counter without
// decrementing the other — slow drift that eventually stalls admission.
a_stage_counts_bounded: assert property (
  @(posedge clk) disable iff (!rst_n)
    (n_copy_in_q + n_execute_q + n_copy_out_q) <= 4'(N_JOB)
);

Reading the three.

The first uses $fell on "not free" because the interesting event is release, and it requires the prior stage to have been J_COMPLETE. That is exactly the lifetime rule, and it is a good formal target — small state, one slot at a time.

The second is defence in depth and holds regardless of whether the lifetime rule was implemented correctly, which is why it is the one to keep if only one survives.

And the third protects the instrumentation §14 depends on. A counter that drifts makes the pipeline's self-report untrustworthy — and every conclusion in §14 rests on those three numbers.

11. Measured Behaviour

Setup. 1 000 jobs at §6's profile (20/50/10 ms), three configurations.

Three readings.

Row 4 is the measurement that changes how a team argues about upgrades. Halving the input copy time produced zero throughput change — because compute was the maximum and the copy was already hidden. Link utilisation fell from 40 % to 20 %, which looks like an improvement and moved nothing.

Row 2 shows the pipeline is depth-limited before it is bandwidth-limited. Two slots cannot keep three stages busy; the buffering requirement is a consequence of the stage count, not of the bandwidth.

And the last column is the cheapest diagnostic in this chapter. How many stages are non-zero at once distinguishes rows 1, 2 and 3 immediately, and it is three counters (§7).

12. Small Jobs and Large Jobs Are Different Systems

Two readings.

One benchmark cannot characterise this interface, which is 22.6's argument in its sharpest form. A latency-oriented small-job benchmark and a throughput-oriented batched benchmark measure different bottlenecks in the same hardware, and a claim from one does not transfer.

And the small-job regime is where the command path's design matters (§5). Launch overhead is per-job, so at small job sizes the doorbell round trip and the completion round trip are a meaningful fraction of wall-clock — which makes the batching and coalescing mechanisms from 26.4 §5 relevant here for exactly the same reason.

13. Where CXL Fits — At Architecture Level Only

The chapter's registry purpose mentions CXL, and the honest treatment is short.

Three readings.

The architectural point worth keeping is the second row. Coherence is a protocol between agents with a point of serialisation, not a property a link confers — and a host-attached accelerator obtains it by participating in the host's coherence domain, not by having a faster link.

And the fourth row is deliberately empty. I do not assert that any AI accelerator ships as a CXL device. The vendor documentation carried in §1 describes PCIe host connectivity and proprietary scale-up fabrics; CXL support is a per-product claim requiring per-product evidence.

Which leaves the useful framing: CXL changes what the host boundary can express — coherent access to device-attached memory, rather than explicit copies — and it does not change §3's magnitude gap. Device memory still feeds compute; the host boundary still moves a working set.

14. Debugging — Which Boundary Diverged

The reported symptom is almost always "we are not getting the performance we expected", and the counters that answer it are stage-oriented rather than link-oriented.

Three readings.

The single most useful reading is stages concurrently non-zero (§11 last column). One means serialised, three means overlapped, and it settles the largest available performance question with three registers.

And compute-active cycles is what makes a "PCIe upgrade" argument falsifiable. If compute is active ~100 % of the time, the link is not the bottleneck and no link change will help — which is a claim the team can check before purchasing.

The failure mode to avoid is link-only instrumentation. A dashboard showing PCIe utilisation at 40 % invites the conclusion that the link has headroom, which is true and irrelevant — §11 row 3 has 40 % link utilisation at 100 % compute utilisation, and that is the optimal configuration.

15. Misconceptions

"PCIe bandwidth determines AI accelerator performance." §3, §6: the compute path runs over device-local memory, one to two orders of magnitude wider, and §11 row 4 measures a halved input copy producing zero throughput change.

"A faster PCIe generation makes compute proportionally faster." §6: it changes a term that may not be in the max(). Serialised it bought 14 %; pipelined it bought nothing.

"HBM bandwidth and PCIe bandwidth are comparable numbers." §3: different paths, different duty cycles, both peaks. The gap is the architecture working, not a bottleneck.

"Command completion means the DMA completed." §8: consuming a command and completing a job are different events, and conflating them frees a slot while the job runs.

"Copy and compute must be serial." §6: they are separable resources, and §11 measures +60 % throughput from overlapping them.

"CXL replaces PCIe for accelerators." §13: CXL changes what the host boundary can express and does not change §3's magnitude gap. And no product claim is made here.

"The endpoint link tells us the accelerator's internal topology." §2: internal fabric and scale-up traffic do not cross the host link, so the endpoint reveals nothing about either.

"Link utilisation of 40 % means we have headroom." §14: §11's optimal configuration runs at 40 % link utilisation and 100 % compute utilisation.

16. Understanding Check

Q1. A team upgrades to a faster PCIe generation and measures no throughput improvement. Explain, with arithmetic.

Because the transfer time was already hidden behind compute (§6). At 20 ms in, 50 ms compute, 10 ms out, a pipelined system's steady-state job time is max(20, 50, 10) = 50 ms. Halving the input copy gives max(10, 50, 10) = still 50 ms — zero change, and §11 row 4 measures exactly that while link utilisation falls from 40 % to 20 %. Serialised, the same change buys 14 % (80 → 70 ms), which is why the upgrade may have helped a different system. The falsifiable check before purchasing is compute-active cycles (§14): near 100 % means compute is the maximum and no link change moves throughput. And the pipelined formula's conditions must hold — at least three jobs' buffering, independent copy and compute resources, and no dependency chaining job N+1's input to job N's output.

Q2. Why is comparing a PCIe generation's bandwidth with an accelerator's memory bandwidth a category error?

They are different paths with different duty cycles (§3). Device memory feeds the compute path at every layer, continuously; the host link moves a working set once per job, or amortised across many. The documented magnitudes make the point — 5.3 TB/s and 8 TB/s of device memory in two Class-C vendor sources against a host link derived from generation × width × efficiency (6.7) — and the gap is the architecture working, not a bottleneck to close. Two further disciplines apply: the two memory figures must not be compared with each other either (different products, generations and capacities, both peaks), and any comparison must state generation, width, direction, raw-versus-useful and which path — because a claim that omits the path is unfalsifiable.

Q3. A job slot is freed when the accelerator fetches its command. What breaks, and what exact state prevents it?

Consuming a command is not completing a job (§8). The slot is advertised free while the job executes; the host submits a new job into it, overwriting the device buffer the running job is reading; the original job's completion arrives carrying only a slot index and is attributed to the new job. §8's timeline: job B is reported complete before it started, and its result buffer contains A's output computed from B's input. No error anywhere. Two pieces of state fix it (§9): slot lifetime bounded by host consumption of the completion record, not by the accelerator's own progress; and a generation per slot travelling with every completion, so a residual mis-attribution is detectable and countable rather than delivered. And it survives testing because it needs slot pressure — with eight slots and fewer than eight jobs in flight the host never reuses a live slot.

Q4. An accelerator reports 40 % PCIe link utilisation and throughput below target. Name the counters that identify the bottleneck.

Start with stages concurrently non-zero (§11, §14) — one means serialised, three means overlapped, and it settles the largest question with three registers. Then compute-active cycles: near 100 % means compute is the maximum and 40 % link utilisation is the optimal configuration, not headroom (§11 row 3). If compute is not saturated, jobs offered versus accepted separates a host-side limit from a device-side one; PCIe-stall cycles (no credit, no tag) versus device-memory stall cycles separates which service path is binding; and outstanding DMA count tests the window (26.2 §5). For small jobs the right counter is the completion-latency distribution instead (§12), because launch overhead rather than bandwidth dominates — and that is a different regime measured by a different instrument.

17. Module 26 Complete

Six endpoint classes, six different places where the performance and the correctness actually lived.

Three readings that close the module.

Every one of those failures produced zero PCIe errors. No CRC event, no Unsupported Request, no Completion Timeout in any of the six. The protocol was satisfied in every case, and the defect was in what a component did with a correctly delivered transaction — which is why 25.9's closing argument runs through the whole module.

Four of six were producer-consumer visibility failures — an MSI, a phase bit, a batched status flag, and a job slot. Every device that writes into host memory and then signals is solving the same problem, and each solved it in a different place.

And the module's method was consistent: identify what the endpoint is trying to accomplish, name what PCIe does not own, find the state that crosses the hardware/software boundary, and locate the bottleneck by measurement rather than by the link's headline number.