Accelerator Growth

Chapter 1.3 priced the copy. It assumed the thing at the far end of the copy — the accelerator — without asking why it is there.

This chapter asks. The answer is not "because AI", and it is not a list of product categories. It is an engineering argument about where performance comes from once one general-purpose processor stops being the cheapest way to get it, and about the interconnect and memory problem that argument creates as a side effect.

1. The One-Sentence Model

Specialisation buys throughput per watt and per square millimetre by removing generality the workload does not need. Every engine that buys it becomes an independent initiator with its own memory appetite — so a system that adds engines adds offered load, and offered load converges on shared resources that were not multiplied.

Two halves, and the second is the one this track needs. The first half explains why accelerators exist. The second explains why a system full of them develops a problem no individual accelerator has.

2. What This Chapter Owns

Not here: why accelerator-local memory runs out (Ch 1.5), what conventional attach provides (Ch 1.6), or coherent attach as a mechanism (Ch 1.7). No CXL mechanism appears in this chapter.

3. Why Specialisation Wins

A general-purpose core spends most of its transistors and most of its energy on not knowing what comes next. Branch prediction, out-of-order scheduling, register renaming, speculation, coherent caches, precise exceptions — none of that performs arithmetic. It exists so that arbitrary code runs quickly, and it is superb at that job.

For a workload whose shape is known in advance, that machinery is overhead. If the computation is a long sequence of identical operations over a predictable address pattern, then a design that assumes exactly that can spend its area on arithmetic instead of on prediction.

The argument is worth stating as an inequality rather than a slogan:

A specialised engine wins when the fraction of a general-purpose core's cost that the workload does not need is larger than the flexibility the workload gives up.

That inequality has two important consequences most treatments skip.

It is workload-conditional, not universal. The same specialised engine that is fifty times better on the workload it was built for can be worse than a core on a workload one branch away from it. Specialisation is a bet on the shape of the computation.

It runs out. Once the fixed-function part of the design is nearly all arithmetic, further specialisation has little left to remove. The remaining scaling comes from replication — more lanes, more engines — which is precisely what turns a compute problem into an interconnect and memory problem.

4. Four Ways to Build a Compute Engine

The four families below are design centres of gravity, not categories with hard walls. Modern parts routinely borrow from each other, and any statement that begins "GPUs cannot…" is usually wrong about some current product. What follows is what each family optimises for when it has to choose.

The engineering content is in the third column. Each family bought its advantage by giving something up, and the thing given up is what determines where it fails.

CPU. Few powerful threads, deep speculation, a coherent cache hierarchy, and hardware that keeps a single instruction stream fast when the next branch is unpredictable. It is the only engine in the list you can hand arbitrary code and expect a reasonable result.

GPU. Very many concurrent threads over regular work, with latency hidden by having enough other threads ready rather than by predicting anything. That trade is the origin of the memory behaviour Chapter 1.5 examines: thousands of lanes only stay busy if the memory system keeps them fed.

FPGA. Configurable logic, so the datapath is built for the problem instead of the problem being mapped onto a fixed datapath. That buys deterministic, low-latency streaming — which is why protocol adaptation and line-rate packet work suit it — at a clock rate well below a hardened design.

NPU-class engines. Structures aimed at matrix and tensor operations, dataflow between stages, and reduced numerical precision where the application tolerates it. The reuse such engines depend on is Chapter 1.3 §9 applied in silicon: fetch an operand once, apply it many times.

5. The Arithmetic of Adding Engines

Specialisation raises the operation rate of one engine. Replication raises the operation rate of a system. Both raise the demand on everything the engines share, and the second raises it faster.

Take a deliberately simple model. Write P for the operations per cycle a general-purpose core sustains, and describe an accelerator as L lanes each performing Q operations per cycle:

core        =  P              operations/cycle
accelerator =  L × Q          operations/cycle

With illustrative values P = 8, L = 128, Q = 2:

accelerator  = 128 × 2 = 256 operations/cycle
ratio        = 256 / 8 = 32× the core's rate

Now attach the consequence, using the arithmetic intensity of Chapter 1.1. If the workload performs I operations per byte moved, the byte rate the engine demands is its operation rate divided by I:

demand (bytes/cycle) = operations per cycle / I
 
at I = 0.5 flop/byte:
  core        =   8 / 0.5  =   16 bytes/cycle
  accelerator = 256 / 0.5  =  512 bytes/cycle

The 32× compute ratio became a 32× bandwidth demand ratio, because intensity is a property of the algorithm and replacing the engine did not change it. This is the single most useful sentence in the chapter for sizing arguments: specialisation multiplies memory demand by the same factor it multiplies compute, unless the specialised design also raises reuse.

Which is exactly why serious accelerators invest in reuse structures — local memory, scratchpads, tiling — rather than only in arithmetic. Raising I is the only way to add compute without adding proportional memory demand.

Aggregate load against fixed service

Now put several engines in one machine. Write A_i for the traffic engine i offers and S for what the shared resource can service:

offered   =  Σ A_i          grows with engine count
service   =  S              set when the platform was designed
utilisation of the shared resource = min(1, offered / S)

When offered exceeds S, the excess does not become throughput. It becomes queueing, backpressure and idle engines — Chapter 1.2 §11 in a different costume. The next section measures it in hardware rather than asserting it.

6. Peak Is a Ceiling; Utilisation Is the Number

Every accelerator datasheet quotes a peak. Peak is the rate at which the engine performs arithmetic when nothing is missing — no operand is late, no result is refused, no queue is empty.

achieved throughput  =  peak × utilisation

That equation is trivial and its consequence is not: utilisation is set almost entirely by things outside the engine. An accelerator running at 30% is rarely 30% because its arithmetic is slow. It is at 30% because for 70% of its cycles an operand had not arrived or a result could not be written.

So the useful decomposition is not "how fast is the engine" but "where did the cycles go":

observed cycles = active + starved + stalled
 
  active   the engine had operands and could deliver results
  starved  the engine had no operands            → look upstream
  stalled  the engine had operands, output blocked → look downstream

Section 12 builds that counter in RTL and measures it, because a decomposition you cannot obtain from silicon is a decomposition nobody uses.

7. The Attach Problem

Figure 1 contains the whole difficulty. Each engine was justified on its own: it computes something faster or more efficiently than the alternative. Nothing in that justification mentioned the other engines, and nothing in it mentioned the shared path.

Three consequences follow, and they are what the rest of Module 1 is about.

Every engine is an initiator. A passive peripheral responds. These engines start work, generate long streams of memory traffic, and keep going without being asked again — Chapter 1.2 called them peers, and this is where they came from.

Every engine wants memory near it. Proximity buys bandwidth, so a serious engine acquires local memory — which is bandwidth bought with reachability, and it creates the domains Chapter 1.3 §8 priced and Chapter 1.5 examines in detail.

Every engine attaches through something. In a conventional machine it attaches as a device, on an I/O path, with the ownership model that implies. Whether that attach model is adequate for a compute peer is exactly the question Chapter 1.6 takes up.

8. The Hardware of an Accelerator Front-End

Sections 9 to 12 build the four structures that sit between software and a specialised engine. Together they are a minimal but complete front-end: a queue that accepts work, a tag pool that bounds how much of it can be in flight, a shared path that serialises engines onto one service point, and the counters that say where the cycles went.

9. RTL 1 — The Work Queue

Purpose

Software submits work faster than the engine consumes it, or slower. A queue absorbs the difference and tells software when it cannot.

// Command submission queue for one accelerator front-end. `occupancy_q` is an
// output rather than an internal signal on purpose: queue depth is the single
// most useful performance counter on a submission path, and a queue that
// cannot report its occupancy cannot be tuned after silicon.
module accel_work_queue #(
  parameter int unsigned DEPTH = 4,
  parameter int unsigned CMD_W = 16
) (
  input  logic             clk,
  input  logic             rst_n,
  input  logic             sub_valid,
  output logic             sub_ready,
  input  logic [CMD_W-1:0] sub_cmd,
  output logic             deq_valid,
  input  logic             deq_ready,
  output logic [CMD_W-1:0] deq_cmd,
  output logic [$clog2(DEPTH+1)-1:0] occupancy_q
);
  localparam int unsigned PTR_W = $clog2(DEPTH);
  logic [CMD_W-1:0] mem_q [DEPTH];
  logic [PTR_W-1:0] wr_q, rd_q;
  logic sub_fire, deq_fire;
 
  assign sub_ready = (occupancy_q != DEPTH[$clog2(DEPTH+1)-1:0]);
  assign deq_valid = (occupancy_q != '0);
  assign sub_fire  = sub_valid && sub_ready;
  assign deq_fire  = deq_valid && deq_ready;
  assign deq_cmd   = mem_q[rd_q];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      wr_q <= '0; rd_q <= '0; occupancy_q <= '0;
    end else begin
      if (sub_fire) begin
        mem_q[wr_q] <= sub_cmd;
        wr_q <= (wr_q == PTR_W'(DEPTH-1)) ? '0 : wr_q + 1'b1;
      end
      if (deq_fire) rd_q <= (rd_q == PTR_W'(DEPTH-1)) ? '0 : rd_q + 1'b1;
      // Simultaneous push and pop cancel; writing this as one case is what
      // keeps occupancy exact when both happen in the same cycle.
      case ({sub_fire, deq_fire})
        2'b10: occupancy_q <= occupancy_q + 1'b1;
        2'b01: occupancy_q <= occupancy_q - 1'b1;
        default: occupancy_q <= occupancy_q;
      endcase
    end
  end
endmodule

Architectural role

The boundary between the software submission path of Chapter 1.2 §6 and the engine. Its depth decides how large a burst software can deposit before it must wait.

State

Two pointers, one occupancy counter, and the storage array. The occupancy counter is $clog2(DEPTH+1) bits, not $clog2(DEPTH), because it must represent DEPTH itself.

Cycle behaviour and simulation evidence

Pushing every cycle with no drain, DEPTH = 4 — verbatim from the Icarus run:

=== EXP4: work queue DEPTH=4, no drain ===
  push 0: sub_ready=1 occupancy=1
  push 1: sub_ready=1 occupancy=2
  push 2: sub_ready=1 occupancy=3
  push 3: sub_ready=0 occupancy=4    <-- full: backpressure to software
  push 4: sub_ready=0 occupancy=4
  push 5: sub_ready=0 occupancy=4

Contract

sub_ready low means the command was not accepted; software must retry it, not assume it landed. Occupancy never exceeds DEPTH.

Synthesis

A small RAM or flop array, two pointer counters, one occupancy counter, two comparators. The storage array is deliberately not reset, so it infers RAM rather than flops.

Failure shape

A queue that reports full one entry early wastes a slot silently — no functional symptom, just throughput nobody can explain. A queue that accepts a push when full overwrites an unread command, and the work simply never happens.

10. RTL 2 — The Tag Pool

Purpose

An engine with many operations in flight must know which completion belongs to which request. A credit counter says how many; a tag pool says which.

// Outstanding-operation slots as a TAG POOL. Unlike a plain credit counter, a
// pool must also say WHICH slot was allocated, because a completion returns a
// tag and the engine must find the state that belongs to it.
module tag_pool #(
  parameter int unsigned NTAG = 8
) (
  input  logic                     clk,
  input  logic                     rst_n,
  input  logic                     alloc_req,
  output logic                     alloc_valid,
  output logic [$clog2(NTAG)-1:0]  alloc_tag,
  input  logic                     free_valid,
  input  logic [$clog2(NTAG)-1:0]  free_tag,
  output logic [NTAG-1:0]          busy_q,
  output logic                     free_to_idle_err
);
  localparam int unsigned TW = $clog2(NTAG);
  logic [NTAG-1:0] avail;
  logic [NTAG-1:0] pick;
 
  assign avail = ~busy_q;
  assign pick  = avail & (~avail + 1'b1);      // lowest free slot
  assign alloc_valid = alloc_req && (avail != '0);
 
  always_comb begin
    alloc_tag = '0;
    for (int unsigned i = 0; i < NTAG; i++)
      if (pick[i]) alloc_tag = TW'(i);
  end
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      busy_q <= '0;
      free_to_idle_err <= 1'b0;
    end else begin
      // A completion for a tag that is not outstanding is a protocol error.
      // Latching it makes a silent failure visible — see Debug Lab 2.
      if (free_valid && !busy_q[free_tag]) free_to_idle_err <= 1'b1;
      if (alloc_valid) busy_q[alloc_tag] <= 1'b1;
      if (free_valid)  busy_q[free_tag]  <= 1'b0;
    end
  end
endmodule

Architectural role

The resource that bounds concurrency. Chapter 1.1 derived that sustaining bandwidth over a latency needs bandwidth × latency bytes in flight; the tag count is where that requirement becomes a design-time number, and where it is frozen.

State

One busy bit per tag, plus a sticky error flag. NTAG bits of state buy NTAG operations of concurrency.

Cycle behaviour and simulation evidence

Allocating continuously with NTAG = 8, then freeing tag 3, then illegally freeing it again — verbatim:

=== EXP3: tag pool, NTAG=8 ===
  busy after each allocation: 00000001, 00000011, 00000111, 00001111,
                              00011111, 00111111, 01111111, 11111111
  9th alloc: valid=0  (pool exhausted, requester must wait)
  after freeing tag 3: busy=11110111
  free-to-idle error latch = 1   (illegal free detected and latched)

The exhaustion line is the architectural point. A requester with 8 tags cannot have a ninth operation outstanding no matter how much link bandwidth is available, so tag count is a bandwidth parameter disguised as a bookkeeping one.

Contract

A tag is allocated at most once before it is freed. A free for an idle tag is illegal and is reported rather than silently applied.

Synthesis

NTAG flip-flops, a lowest-set-bit encoder, a small decoder, one error flop. The avail & (~avail + 1) idiom synthesises to a borrow chain; for large NTAG a tree-based leading-one detector replaces it.

Failure shape

Freeing a tag early — on a timeout, or on a partial response — lets it be reallocated while the original completion is still in flight. The late completion then lands on a live entry belonging to a different request, and the corruption appears far from the tag pool.

11. RTL 3 — The Shared Path, Measured

Purpose

This is the module the chapter exists for. It puts N engines onto one service point and counts what each one actually received, so the scaling law is measured rather than asserted.

// N engines converge on ONE service point. This module exists to MEASURE the
// scaling law, not to demonstrate fairness: it counts the grants each engine
// wins so a testbench can show that per-engine share falls as N rises.
module shared_path #(
  parameter int unsigned N          = 4,
  parameter int unsigned CNT_W      = 16,
  // 1 = refill the eligibility mask to all-ones at a round boundary.
  // 0 = refill to ~pick (exclude the just-served requester).
  // The two policies are NOT equivalent under saturated load; the testbench
  // measures the difference.
  parameter bit          REFILL_ALL = 1'b1
) (
  input  logic         clk,
  input  logic         rst_n,
  input  logic [N-1:0] req,
  input  logic         svc_ready,
  output logic [N-1:0] grant,
  output logic         grant_valid,
  output logic [CNT_W-1:0] served_q   [N],
  output logic [CNT_W-1:0] offered_q  [N],
  output logic [CNT_W-1:0] svc_busy_q
);
  logic [N-1:0] mask_q, eligible, pick;
 
  assign eligible    = ((req & mask_q) != '0) ? (req & mask_q) : req;
  assign pick        = eligible & (~eligible + 1'b1);
  assign grant_valid = svc_ready && (req != '0);
  assign grant       = grant_valid ? pick : '0;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      mask_q <= '1; svc_busy_q <= '0;
      for (int unsigned i = 0; i < N; i++) begin
        served_q[i]  <= '0;
        offered_q[i] <= '0;
      end
    end else begin
      if (grant_valid) begin
        mask_q <= ((req & mask_q & ~pick) == '0)
                    ? (REFILL_ALL ? {N{1'b1}} : ~pick)
                    : (mask_q & ~pick);
        svc_busy_q <= svc_busy_q + 1'b1;
      end
      // Offered and served are counted separately. The gap between them IS
      // the contention, and a design that counts only `served` cannot see it.
      for (int unsigned i = 0; i < N; i++) begin
        if (req[i])   offered_q[i] <= offered_q[i] + 1'b1;
        if (grant[i]) served_q[i]  <= served_q[i]  + 1'b1;
      end
    end
  end
endmodule

Simulation evidence — the scaling law

Four engines, all requesting every cycle, service ready every cycle, 40 service cycles. Verbatim:

=== EXP1: N=4 engines all requesting, service ready every cycle ===
  svc_busy=40  served: e0=10 e1=10 e2=10 e3=10
  offered: e0=40 e1=40 e2=40 e3=40
  share: e0=25% e1=25% e2=25% e3=25%

Then the same service point with only two engines asking, over a further 40 service cycles:

=== EXP2: same service, only 2 engines requesting ===
  svc_busy=80  served: e0=30 e1=30

Engine 0 went from 10 grants in the first 40 cycles to 20 more in the next 40 — from a 25% share to a 50% share. The service point did not change. The only thing that changed was how many engines were sharing it.

That is the law, stated so it can be applied to a system nobody has built yet:

per-engine share  =  S / N        (when every engine is saturated)
offered : served  =  N : 1        (each engine offered 40, received 10)

Doubling engine count does not double useful throughput once the shared resource is saturated. It halves each engine's share and quadruples nobody's.

A measured fairness pathology

The REFILL_ALL parameter exists because the other refill policy behaves differently, and the difference is invisible in a short trace. Same stimulus, both policies, side by side in the same simulation:

  [all-ones refill] served: e0=10 e1=10 e2=10 e3=10   <-- even
  [~pick   refill ] served: e0=13 e1=13 e2=7  e3=7    <-- NOT even

The ~pick policy — refilling the eligibility mask to everyone except the requester just served — prevents a back-to-back win at a round boundary, which is a real property. It also shortens rounds, and the shortened rounds systematically drop the highest indices. The grant order under saturated load is 0,1,2,3, 0,1,2, 0,1,3, 0,1,2, …, so engines 0 and 1 receive roughly 1.9× the share of engines 2 and 3.

Two lessons, and the second is the more valuable one.

A fairness scheme is not fair because it was designed to be. The eligibility mask looks symmetric in the source. It is not symmetric in behaviour, and no amount of reading the RTL reveals that — only counting does.

A per-requester safety assertion can hide it. A bounded-wait property written for requester 0 passes comfortably under the ~pick policy, because requester 0 is the favoured one. The assertion is true and the arbiter is unfair simultaneously. Fairness is a property of the distribution, so it needs a measurement over a long run, not a per-cycle check.

12. RTL 4 — Utilisation Attribution

Purpose

To answer "why is the engine idle" with evidence rather than opinion, and to do it in a way that cannot double-count.

// Utilisation attribution for one accelerator. The three counters are mutually
// exclusive BY CONSTRUCTION so that active + starved + stalled == observed,
// which is what makes the ratios interpretable.
module accel_util #(
  parameter int unsigned CNT_W = 32
) (
  input  logic clk,
  input  logic rst_n,
  input  logic clear,
  input  logic observe,        // engine is powered on and has work assigned
  input  logic has_input,      // operands are present
  input  logic out_ready,      // downstream can take the result
  output logic [CNT_W-1:0] observed_q,
  output logic [CNT_W-1:0] active_q,
  output logic [CNT_W-1:0] starved_q,
  output logic [CNT_W-1:0] stalled_q
);
  logic active_c, starved_c, stalled_c;
 
  // Priority is deliberate: no input is starvation even if the output is also
  // blocked, because the upstream fix is the one that matters first.
  assign starved_c = observe && !has_input;
  assign stalled_c = observe &&  has_input && !out_ready;
  assign active_c  = observe &&  has_input &&  out_ready;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n || clear) begin
      observed_q <= '0; active_q <= '0; starved_q <= '0; stalled_q <= '0;
    end else if (observe) begin
      observed_q <= observed_q + 1'b1;
      if (active_c)  active_q  <= active_q  + 1'b1;
      if (starved_c) starved_q <= starved_q + 1'b1;
      if (stalled_c) stalled_q <= stalled_q + 1'b1;
    end
  end
endmodule

Key invariant

observed == active + starved + stalled, exactly, with no residue. The three conditions are constructed as a priority chain rather than as three independent expressions, which is what guarantees it. A set of counters that can double-count produces percentages summing to more than 100 and destroys confidence in the whole measurement.

Simulation evidence

Twenty observed cycles: ten fully fed, six with no operands, four with the output blocked. Verbatim:

=== EXP5: utilisation over 20 observed cycles ===
  observed=20 active=10 starved=6 stalled=4
  utilisation = active/observed = 50%

The partition is exact — 10 + 6 + 4 = 20 — and the diagnosis is immediate: 30% of the engine's life was spent waiting for operands and 20% waiting for somewhere to put results. No amount of arithmetic improvement addresses either number.

Contract, synthesis, failure shape

Counters advance only while observe is asserted, so an idle engine does not dilute the ratios. Four counters with enables and three small comparators. The characteristic bug is counting stalled cycles as active because the enable used has_input alone — which inflates utilisation exactly when the design is in trouble, and is Debug Lab 3.

13. Assertions

Bind-ready properties for the four modules. Icarus does not support concurrent assertions, so these were not executed; the mapping table says which testbench check verified each invariant procedurally.

// Q1 — the queue never exceeds its physical depth.
a_q_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  occupancy_q <= DEPTH);
 
// Q2 — a full queue accepts nothing. Backpressure must be real, not advisory.
a_q_full_blocks: assert property (@(posedge clk) disable iff (!rst_n)
  (occupancy_q == DEPTH) |-> !(sub_valid && sub_ready));
 
// T1 — a tag is never allocated while it is already outstanding. Violating
// this means two live requests share one piece of state.
a_tag_not_double_alloc: assert property (@(posedge clk) disable iff (!rst_n)
  alloc_valid |-> !busy_q[alloc_tag]);
 
// T2 — a completion for an idle tag is a protocol error, and the flag is
// sticky so one occurrence in a long regression is still visible at the end.
a_tag_err_sticky: assert property (@(posedge clk) disable iff (!rst_n)
  free_to_idle_err |=> free_to_idle_err);
 
// A1 — at most one engine is granted per cycle.
a_grant_onehot0: assert property (@(posedge clk) disable iff (!rst_n)
  $onehot0(grant));
 
// A2 — a grant implies a request, and never reaches a busy service.
a_grant_legal: assert property (@(posedge clk) disable iff (!rst_n)
  ((grant & ~req) == '0) && (!svc_ready |-> (grant == '0)));
 
// U1 — the utilisation partition is exact. If this fails the percentages are
// meaningless and every conclusion drawn from them is unsafe.
a_util_partition: assert property (@(posedge clk) disable iff (!rst_n || clear)
  observed_q == active_q + starved_q + stalled_q);

Note what A1 and A2 do not cover. Both are safety properties, and Section 11 measured an arbiter that satisfies both while distributing service 13/13/7/7. Fairness is not expressible as a per-cycle property, which is why the counters exist.

14. Debug Lab

Each failure below was produced by injecting the bug into the real module and re-running the same testbench. The numbers are actual simulator output.

// Round-boundary refill excludes the just-served requester.
mask_q <= ((req & mask_q & ~pick) == '0) ? (~pick) : (mask_q & ~pick);
 
// The fairness check that "proves" the arbiter is fair:
a_bounded_fairness: assert property (@(posedge clk) disable iff (!rst_n)
  wait0_q < N);            // <-- checks requester 0, the favoured one

  [all-ones refill] served: e0=10 e1=10 e2=10 e3=10
  [~pick   refill ] served: e0=13 e1=13 e2=7  e3=7

mask_q <= ((req & mask_q & ~pick) == '0) ? {N{1'b1}} : (mask_q & ~pick);

// Timeout handling frees the tag so the requester is not wedged forever.
if (timeout_fired) busy_q[timeout_tag] <= 1'b0;   // nothing fences the old request

  after freeing tag 3: busy=11110111
  free-to-idle error latch = 1   (illegal free detected and latched)

if (timeout_fired) quarantine_q[timeout_tag] <= 1'b1;   // not reallocatable yet
assign avail = ~busy_q & ~quarantine_q;

// "Active" means the engine had operands to work on.
assign active_c = observe && has_input;      // BUG: ignores out_ready

GOOD: observed=20 active=10 starved=6 stalled=4  util=50%  sum=20
BUG : observed=20 active=14 starved=6 stalled=4  util=70%  sum=24

assign starved_c = observe && !has_input;
assign stalled_c = observe &&  has_input && !out_ready;
assign active_c  = observe &&  has_input &&  out_ready;

15. When Adding Engines Stops Helping

The diagnostic question is always the same: the system has more compute than it had; where did the throughput go? The answers separate cleanly if the right things are counted.

Two habits make this tractable, and both come from earlier chapters.

Compare against a computed bound. With N saturated engines and a service point granting once per cycle, each engine's ceiling is 1/N of the service rate. If measured share matches that, the system is behaving correctly and the fix is more service capacity — not more engines.

Separate the control plane from the data plane. An engine starved while the interconnect is idle is Chapter 1.2's submission bottleneck, and no amount of extra bandwidth addresses it.

16. How This Appears in Real Engineering

SoC architect

Questions about ratios and topology, asked before RTL exists. How many engines share one memory controller, and what is their combined demand at the intensity the target workload actually has? Where does arbitration happen — once, or in a hierarchy? What happens when every engine bursts simultaneously, which is the case datasheet numbers never describe? Does giving each engine local memory strand capacity another engine needs? Is the interconnect deliberately oversubscribed, and on what assumption about concurrent activity?

Accelerator RTL engineer

Section 8's four structures are the daily work: queue depths, tag counts, arbitration policy, and the counters that make all three tunable after silicon. The sizing judgement recurs — tag count comes from a rate multiplied by a round-trip latency, frozen at design time. The habit worth building is instrumenting offered as well as served, because a design that reports only what it achieved cannot explain why.

Verification engineer

These failures are load-dependent and silent. Saturate every requester and hold it. Exhaust the tag pool and try one more. Free a tag that is not outstanding. Fill the queue and keep pushing. Reset with tags outstanding and confirm the pool returns empty rather than to a stale bitmap. And — the finding this chapter demonstrates — run long enough to measure fairness, because the safety properties pass while the distribution is 13/13/7/7.

Interconnect architect

Aggregate offered load against service capacity, per traffic class, at realistic concurrency rather than worst-case-everything. Whether one engine's burst can starve another's latency-sensitive traffic. Where buffering absorbs bursts and where it merely relocates the stall. The uncomfortable question is which engines were sized assuming the others would be idle.

Performance engineer

active / observed is the headline; starved and stalled are the diagnosis, localising the problem upstream or downstream in one measurement. Offered-versus-served per engine separates contention from slowness. Peak numbers are useful only as the denominator of a utilisation ratio.

Firmware and driver engineer

Queue depth and submission batching decide whether the engines are fed — Chapter 1.3 §12 is the arithmetic. Tag and buffer lifetime rules decide whether completions are matched safely. Whether a timeout frees a resource the hardware may still be using is a correctness decision that lives in this role's code, and Debug Lab 2 is what happens when it is made casually.

Datacentre architect

The unit of purchase is a node with a fixed ratio of engines to memory to interconnect, and workloads do not arrive with that ratio. A node whose accelerators saturate while its memory is half free is the stranding argument of Chapter 1.1 §11, seen from procurement.

17. Common Misconceptions

18. Interview Reasoning

19. Summary

Specialisation wins by removing generality a known workload does not need, and the win is real: a design that assumes the shape of its computation spends on arithmetic what a general-purpose core spends on prediction. It is also conditional and finite — conditional because the assumption can be violated, finite because once a design is nearly all arithmetic, further gains come from replication.

Replication is where the trouble starts. Arithmetic intensity belongs to the algorithm, so multiplying an engine's operation rate multiplies its byte demand by the same factor unless the design also raises reuse. And every engine added is another independent initiator offering load to shared resources that were not multiplied with it. The measured law, from four saturated engines on one service point: each received 25% of the service rate; with two engines the same pair received 50%. Offered was 40 per engine, served was 10 — a 4:1 gap that is the contention.

Which makes utilisation, not peak, the number worth carrying. observed = active + starved + stalled turns an engine's life into a diagnosis: starved points upstream, stalled points downstream, and neither is addressed by faster arithmetic. The four structures in Sections 9 to 12 are where those limits live in hardware, and each bounds something different.

Two findings from building them are worth keeping. A fairness scheme is not fair because it was designed to be — a symmetric-looking arbiter measured 13/13/7/7 while every safety assertion passed, because fairness is a distribution and assertions are per-cycle. And an instrumentation partition that does not sum is a wrong answer that looks right — counters reporting 70% against a true 50% summed to 24 over a 20-cycle window, which is arithmetically impossible and therefore trivial to detect if anyone checks.

The durable form: specialisation multiplies compute and memory demand together; replication multiplies offered load without multiplying service. What follows is an attach problem, not a compute problem.

20. What Comes Next

Every engine in Figure 1 wants memory close to it, because proximity is what buys bandwidth. Each therefore acquires local memory, and each becomes an island — fast for its owner, unreachable by anything else, fixed in size when the part was designed.

Chapter 1.5 takes the hardest case, where the working set outgrows the island entirely and the failure is capacity rather than bandwidth. Chapter 1.6 asks whether the way these engines attach to a host is adequate for a compute peer — a question about semantics rather than bandwidth. Chapter 1.7 makes the case for the answer.

For the hardware behind this chapter, the AXI track covers arbitration, outstanding depth tuning and backpressure in production detail; the PCIe track covers accelerator attach and copy-compute overlap and scalability limits. The full path is on the CXL tutorials index.