CPU-Centric Computing Limits

Chapter 1.1 established that memory cannot supply what compute can consume. That argument was about a resource: bytes per second, bytes of capacity, joules per byte moved.

This chapter is about authority — which agent in the system is allowed to own memory, decide what moves, hold coherent state, translate an address, and start work. In a conventional server that agent is overwhelmingly the CPU, and that arrangement is the reason a great many systems work at all. It is also, past a certain amount of heterogeneity, the thing that stops them scaling.

1. The One-Sentence Model

A CPU-centric architecture makes every important resource — memory, coherence, translation, and the right to initiate work — subordinate to the CPU. That is an excellent trade when devices are peripherals and a progressively worse one as devices become compute peers, because coordination, copying and serialisation grow with the number of peers while the central agent does not.

Two things follow that are easy to state and easy to get wrong.

The CPU can become a bottleneck without touching a single byte of the data. Control-plane work — mapping, descriptors, doorbells, completions, synchronisation — is separate from data-plane work, and each can saturate independently.

Centralisation is not a mistake. It buys simple ownership, one place to enforce policy, and a correctness story an engineer can hold in their head. The engineering question is never "central or distributed"; it is which resources must stay central and which have outgrown it. Section 17 is entirely about that question, and the rest of the chapter is the evidence needed to answer it.

2. What This Chapter Owns

Deliberately not here: the energy accounting of moving bytes (Ch 1.3), why accelerators proliferated (Ch 1.4), discrete-GPU memory limits (Ch 1.5), what PCIe does and does not provide (Ch 1.6), and the case for coherent attach as a mechanism (Ch 1.7). This chapter builds the pressure those chapters resolve. No CXL protocol mechanism appears in it.

3. The Architecture That Worked

Before criticising the model, it is worth being precise about why it was a good one — because most of its advantages are still real.

In the classical arrangement the CPU is the only general-purpose initiator. Memory hangs off its controller. Devices sit below it in an I/O hierarchy and exist to move data into and out of CPU-owned memory. Software running on the CPU decides what happens next.

That structure buys five things, and they are not small:

• One owner of memory. Allocation, protection and lifetime are decided in one place, by code that already has to be correct.
• One coherence domain. Caches belong to CPU cores. A single hardware protocol keeps them consistent, and software mostly does not have to think about it.
• One translation authority. Virtual-to-physical mapping lives with the CPU's page tables and its MMU. Isolation between processes is enforced at one boundary.
• One place to set policy. Scheduling, priority, quality of service, and security checks all sit on the path everything already takes.
• A tractable failure model. When there is one initiator, "who did this?" has one answer, which makes debugging and post-mortem analysis enormously easier.

Hold Figure 1 as the reference picture. Everything that follows is a way of asking what happens to it when the boxes on the right stop being passive.

4. Two Different CPUs: the Execution Engine and the Orchestrator

The single most useful distinction in this chapter is that "the CPU" names two roles that saturate independently.

The CPU as execution engine runs the application's arithmetic. It is on the data plane: it loads, computes, stores. When this role is the limit, you see high instructions-per-cycle on useful work, cores pinned at 100%, and the profile dominated by the application's own functions.

The CPU as orchestrator decides what happens and tells other agents to do it. It is on the control plane: allocate, map, build a descriptor, ring a doorbell, take an interrupt, reap a completion, signal a waiter. When this role is the limit, cores are also busy — but the profile is dominated by the driver, the runtime, the interrupt path and synchronisation, and the useful arithmetic per busy cycle is close to zero.

Two more roles matter and are easy to forget because they are usually invisible:

The CPU as coherence authority. The hardware coherence protocol is built around CPU caches. A device that wants a consistent view of the same data either participates in that protocol or is kept outside it and made consistent by software.

The CPU as translation authority. Devices are given access to memory through mappings the CPU's software creates and the platform's translation hardware enforces. A device does not simply "have" an address space; it is granted one.

5. When Devices Stopped Being Peripherals

The classical model assumes devices are passive: they have no significant state of their own, they do not decide anything, and they exist to shuttle bytes into and out of CPU-owned memory. That assumption held for a long time. It no longer describes what is plugged into a modern server.

A contemporary node may contain GPUs, FPGAs, NPUs and other domain-specific accelerators, SmartNICs and DPUs, storage accelerators, compression and encryption engines, and memory expanders. Not all of them behave alike, and it would be wrong to claim every device has every property below. But across that population, the following are now common rather than exotic:

• They execute independently. Once given work, the device runs a program of its own for a long time relative to a memory access.
• They generate enormous memory traffic. A device can be a larger consumer of bandwidth than the cores.
• They hold local memory. Capacity that is fast for the device and not directly usable by anything else.
• They cache shared data. A device that reads the same lines repeatedly wants them close, which means it wants a cache, which means it wants coherence.
• They talk to each other. Device-to-device exchange is a normal traffic pattern, not a special case.
• They need low-latency access to shared state. Queues, flags, counters and control structures that both host and device read and write.

Every one of those properties is in tension with a structure that assumes the CPU is the sole owner and initiator. A device with its own long-running program does not want to ask permission per operation. A device with a cache needs its cached copy to be correct with respect to the CPU's. A device that exchanges data with another device does not benefit from staging it through host memory. The architecture is not wrong — it is being asked a question it was not designed for.

Once devices become compute peers rather than passive peripherals, a structure built around one universal centre starts producing movement, serialisation and ownership boundaries that the workload never asked for.

6. The Control Path, Priced

"CPU orchestration overhead" is a phrase that means nothing until you can name the steps and put a number on them. Here are the steps.

Different device classes and runtimes use different subsets of this list, and a well-engineered stack removes or amortises several of them. Treat it as the vocabulary of the control plane, not as a claim that every device pays every item on every operation.

• Buffer allocation — memory the device may use has to exist and be pinned or otherwise guaranteed resident.
• Mapping — that memory has to be made reachable by the device, which means an entry in a translation structure and a device-visible address handed back.
• Descriptor construction — a record describing the work: source, destination, length, flags, completion instructions.
• Queue management — a submission ring with producer and consumer indices, and the memory ordering discipline that keeps them consistent between two agents.
• Doorbell — an MMIO write telling the device that new work exists. An uncached store to device memory space, which is far more expensive than a normal store.
• Cache maintenance — where the device is not coherent with CPU caches, explicit flushes before the device reads and invalidations after it writes.
• Completion handling — reading a completion record, matching it to the request, and freeing the resources.
• Interrupt processing — the device signals; the CPU takes an interrupt, enters the handler, wakes a waiter, and returns.
• Synchronisation — the fences, locks and atomics that make all of the above safe between the submitting thread, the completion path and the device.
• Ownership transitions — the bookkeeping that says a buffer now belongs to the device, and later that it belongs to the host again.

Worked example A — when the control plane is the limit

The following model is illustrative. Real per-submission costs span orders of magnitude depending on device class, batching, interrupt versus polling, and whether the path is in the kernel or user space. The arithmetic is what matters; substitute your own measured cost.

Assume, illustratively:
 
  host cost per submission   = 2 µs   (descriptor + doorbell + completion + wake)
 
  one core sustains          = 1 / 2e-6
                             = 500,000 submissions/s
 
  4 cores dedicated to I/O   = 2,000,000 submissions/s
 
Each accelerator operation needs 2 submissions (work in, result out):
 
  control-plane capacity     = 2,000,000 / 2
                             = 1,000,000 operations/s
 
If one accelerator can execute 400,000 operations/s:
 
  accelerators supportable   = 1,000,000 / 400,000
                             = 2.5

Two and a half. Four host cores dedicated entirely to feeding devices support fewer than three of them, and the fourth accelerator in the chassis is idle hardware regardless of how much memory bandwidth is free.

Now change one input. Batch 32 operations per doorbell and per completion event:

  per-operation host cost    ≈ 2 µs / 32
                             ≈ 62.5 ns
 
  control-plane capacity     ≈ 4 cores × (1 / 62.5e-9) / 2
                             ≈ 32,000,000 operations/s
 
  accelerators supportable   ≈ 80

That contrast is the honest version of the argument. The control plane is not a fixed tax; it is a design surface, and batching, polling, user-space submission and hardware queue management have all moved it by more than an order of magnitude. What does not change is the structure: per-operation host work exists, it is proportional to the number of operations somebody has to describe, and it competes with the application for the same cores. Amortisation pushes the wall back. It does not remove it, and it costs latency — a batch of 32 does not start until 32 requests exist, or a timeout fires.

7. What DMA Fixes, and What It Leaves

The most common beginner model of this subject is "DMA solves CPU involvement." It is worth being exact about what DMA does, because the half of the problem it does not touch is the half this chapter is about.

What DMA genuinely removes: the CPU executing the transfer. Without it, moving n bytes costs the CPU n bytes of load and store work, and the cores are on the data plane for the whole transfer. With it, a device engine performs the transfer while the CPU does something else. That is a large, real win and it is why every serious device has a DMA engine. The PCIe track covers the mechanism in DMA over PCIe and its failure modes in Host Memory Access.

What DMA leaves behind is everything in Section 6 except the copy loop itself:

Read the last row twice. DMA changes who executes the copy. It does not change where the data has to be or who owns it. A device that must operate out of host memory still needs its input placed there and its output read back from there, and a device with its own local memory still needs the data staged across. That placement question is Section 8, and it is architectural rather than a matter of who runs the loop.

8. Data-Movement Amplification and Memory Islands

Give a device its own memory and it gets bandwidth and proximity that host memory cannot match. That is a real engineering win, and Chapter 1.1 priced why proximity is what buys the bandwidth. The cost is that the system now has two memory domains and neither can address the other directly, so any data both sides need exists twice and has to be moved.

A realistic path for one job looks like this:

storage                        (source of record)
   ↓  DMA into host memory
host DRAM                      (staging copy)
   ↓  transfer across the I/O link
device-local memory            (working copy)
   ↓  device loads
device execution
   ↓  results written back
device-local memory
   ↓  transfer across the I/O link
host DRAM                      (result copy)

Each arrow is a byte read from one memory and written to another. The CPU may execute none of them and the argument is unaffected: the placement forced the movement, not the copy loop.

Worked example B — copy amplification

working set                S = 4 GiB
                             = 4 × 2^30
                             = 4.295e9 bytes
 
link traffic  = host→device + device→host
              = 2 × 4.295e9
              = 8.59e9 bytes
 
at an illustrative sustained 25 GB/s across the link:
 
  transfer time = 8.59e9 / 25e9
                = 0.344 s
 
if the computation itself takes 0.100 s:
 
  copy : compute   = 3.4 : 1
  device utilisation = 0.100 / (0.344 + 0.100)
                     = 22.5%

Overlap the two — double-buffer so a transfer runs while the previous chunk computes — and the time becomes max(0.344, 0.100) = 0.344 s, for a device utilisation of about 29%. Better, and still copy-bound: overlapping hides the smaller phase behind the larger one, it does not remove the larger one.

Now count the DRAM traffic the same job caused, which the link figure hides:

every link byte is also read from one DRAM and written to another:
 
  DRAM traffic ≈ 2 × 8.59e9
               = 17.2e9 bytes
 
useful arithmetic performed on: 4.295e9 bytes

Four times the working set moves through DRAM so that one copy of it can be computed on. That ratio is the amplification, and it is a property of the ownership structure — two domains, neither addressable from the other — not of any particular link or engine.

9. Shared State Without Shared Memory

Copying is the visible cost. The subtler one appears when the host and a device need to work on the same data rather than on their own copies of it.

Consider the smallest possible case. The CPU writes a value; the device must read it, modify it, and the CPU must then read the result.

CPU writes X          → the new value sits in a CPU cache, dirty
device wants X        → the device is outside the CPU's coherence domain
device reads memory   → it sees the OLD value unless the line is written back
device modifies X     → the new value is in device memory or a device cache
CPU reads X           → it hits its own stale cached copy

Without hardware that keeps the two views consistent, correctness has to be manufactured in software: flush the line before the device reads, invalidate before the CPU reads back, place a fence at each ownership transition, and structure the program so that no other thread touches the buffer while the device owns it.

That works. It is what a great deal of production software does. Its costs are:

• It is expensive. Cache maintenance over a large buffer costs real time and pollutes the cache. A fence orders more than the one buffer you cared about.
• It is coarse. Ownership moves in whole buffers because per-line software tracking is impractical, so a device that touches 5% of a buffer still forces maintenance on 100% of it.
• It is a correctness cliff. A missing flush is a data-dependent, timing-dependent, occasionally-reproducible wrong answer. It does not crash and it does not warn.
• It forbids fine-grained sharing. A shared queue, a flag, a work-stealing structure — anything where both sides read and write the same lines frequently — is impractical when every transition costs a flush.

The alternative is to let the device participate in the coherence protocol so the hardware keeps the views consistent. That is what "coherent attach" means as an architectural property, and it is the direction Chapter 1.7 argues for. It also changes the programming model: a device that can read host memory directly does not need the data copied at all.

Worked example C — sharing is not automatically cheaper

The tempting conclusion is that eliminating copies eliminates cost. It does not. Copies move data in bulk, sequentially, at the highest bandwidth the link can sustain. Direct fine-grained access moves less data, but does so as scattered requests at higher latency and lower achieved bandwidth.

Write S for the working-set size, f for the fraction of it the device actually touches, k for how many times it touches each byte without local reuse, B_bulk for the achieved bandwidth of a bulk transfer, and B_fine for the achieved bandwidth of fine-grained remote access.

copy model    : bytes = 2 × S          time = 2S / B_bulk
access model  : bytes = k × f × S      time = kfS / B_fine
 
access is faster when
 
    kfS / B_fine   <   2S / B_bulk
 
    k × f          <   2 × (B_fine / B_bulk)

Substitute the two cases that matter:

B_fine / B_bulk = 0.5   →  access wins while  k × f  <  1.0
B_fine / B_bulk = 0.2   →  access wins while  k × f  <  0.4

Read that as an engineering rule. If fine-grained access achieves half of bulk bandwidth, sharing wins whenever the device effectively touches less than the whole working set. If it achieves a fifth, sharing only wins on genuinely sparse access — touch 40% of the data once, or 20% of it twice, and you are already behind the copy.

The two variables that move B_fine most are locality and concurrency, both of which Chapter 1.1 already priced: a device that caches what it fetches turns k back towards 1, and a device that keeps enough requests in flight to satisfy Little's Law raises B_fine towards its ceiling. This is exactly why device-side caching and deep outstanding-request support are architectural requirements rather than optimisations — Sections 13 and 14 build both in hardware.

10. Address Translation — the Coordination Nobody Sees

There is a third centralised authority beside memory ownership and coherence, and it is the one most likely to be omitted from an interview answer.

A device does not naturally speak the same addresses as the application that owns its data. The application works in virtual addresses; memory is physical; the mapping between them belongs to the CPU's page tables, and it changes underneath a running program. Bridging that gap costs coordination:

• Explicit mappings. The classical approach hands the device a device-visible address for a specific buffer, established for the duration of the operation and torn down afterwards. Set-up and tear-down are per-operation control-plane work — items 1 and 2 of Section 6.
• Platform translation hardware. An IOMMU (or equivalent) translates and range-checks device accesses, which is what makes it safe to give a device access to memory at all. It brings its own translation caches, its own invalidation traffic when a mapping changes, and its own miss cost.
• Pinning. Memory a device may access generally cannot be moved or paged out while the mapping exists, which constrains the memory manager.
• Shared virtual memory. The more capable model lets a device operate on the same virtual addresses as the process, so a pointer means the same thing on both sides. It removes per-buffer mapping work and makes pointer-rich data structures usable by the device — at the cost of the device participating in translation and invalidation, which is more system coordination, not less.

That last line is the point worth carrying: shared virtual memory removes bookkeeping from the software path by adding participation to the hardware path. It is a relocation of coordination, not a deletion of it, and evaluating that trade is exactly the kind of judgement a system architect is paid for.

This is as much translation as this chapter needs. It exists here only to complete the list of things the CPU centrally owns: memory, coherence, translation, and the right to initiate.

11. The Hardware Shape of a Central Resource

Everything so far has been architectural. It has a hardware shape, and the shape is always the same: several requesters converge on one service point.

That structure appears at every level of a system. Cores and devices converge on a memory controller. Device engines converge on a link. Completion sources converge on an interrupt controller. Submission queues converge on the software thread that drains them. In each case the same principle applies, and it is the most important sentence in this section:

Adding requesters increases offered load. It does not increase service capacity.

A shared service point has a service rate. If the aggregate offered load stays below it, the queue in front stays short and latency is roughly the service time. As offered load approaches the service rate, the queue grows, latency grows with it, and — because a queue is finite — the point eventually refuses new work and that refusal travels back to the requesters. Nothing is broken. The structure is behaving exactly as designed; there is simply more demand than the centre can absorb.

The next four sections build Figure 3 in RTL. The reason to write the hardware rather than describe it is that the architectural claims of this chapter — serialisation, fairness, latency hiding, backpressure coupling — are all structural properties of small pieces of logic, and they are far more convincing when you can see the state that produces them.

12. RTL 1 — The Centralized Arbiter

One service point, several requesters, one winner per cycle. This is the smallest complete model of centralisation in hardware.

// A single arbitration point for N requesters. The architectural content is in
// three places: only one grant can be issued per cycle; a grant is issued only
// when the downstream service can accept it; and the eligibility mask is what
// stops a persistently-asserting requester from starving its neighbours.
module central_arbiter #(
  parameter int unsigned N = 4
) (
  input  logic         clk,
  input  logic         rst_n,
  input  logic [N-1:0] req,        // level-held: a requester keeps req high until granted
  input  logic         svc_ready,  // the shared service can accept work this cycle
  output logic [N-1:0] grant,      // one-hot, or all-zero
  output logic         grant_valid
);
 
  // One bit per requester: 1 = has not yet been served in this round. Round
  // robin here means "serve every asserting requester once, then refill" —
  // the simplest fair policy that is easy to reason about and to assert on.
  logic [N-1:0] mask_q;
  logic [N-1:0] eligible;
  logic [N-1:0] pick;
 
  // Requests still allowed to win this round. When every asserted request has
  // already been served, fall back to the full request vector — that fallback
  // is the round boundary.
  assign eligible = ((req & mask_q) != '0) ? (req & mask_q) : req;
 
  // Isolate the lowest set bit: x & (~x + 1) is two's-complement negation
  // ANDed with the original, which leaves exactly one bit set (or zero).
  // Evaluated in an N-bit context, so no width extension surprises.
  assign pick = eligible & (~eligible + 1'b1);
 
  // No grant is ever issued to a service that cannot take it. Without this
  // term the arbiter would spend a turn on work the service drops, and the
  // requester would have to re-arbitrate for a slot it already won.
  assign grant_valid = svc_ready && (req != '0);
  assign grant       = grant_valid ? pick : '0;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      mask_q <= '1;
    end else if (grant_valid) begin
      // Clear the winner. If that empties the eligible set, start the next
      // round with everyone except the just-served requester — which is what
      // prevents the same requester winning twice across a round boundary.
      mask_q <= ((req & mask_q & ~pick) == '0) ? (~pick) : (mask_q & ~pick);
    end
  end
 
endmodule

Architecture. One decision point, one winner, one service. Every requester's throughput is a share of the service's rate, not of its own request rate — which is the hardware statement of "adding requesters does not add capacity."

State. mask_q, N bits. That is the entire fairness mechanism; everything else is combinational.

Cycle behaviour. With all four requests asserted and svc_ready held high, grants come out one per cycle in index order — 0001, 0010, 0100, 1000 — then the mask refills and the pattern repeats. With svc_ready low, grant is all-zero and mask_q holds: the arbiter does not advance while the service is stalled.

Expected simulation output. Driving req = 4'b1111, svc_ready = 1 for six cycles after reset and printing grant each cycle:

t=1 grant=0001
t=2 grant=0010
t=3 grant=0100
t=4 grant=1000
t=5 grant=0001   <-- new round
t=6 grant=0010

A plain rotation, which is what round-robin should look like under saturated symmetric load. A fixed-priority arbiter would print 0001 on every cycle forever, and that difference is the entire content of Debug Lab 3.

The ~pick refill is worth one more trace, because its value is invisible in the run above. Take req = 4'b0011 with requester 1 already served this round, so mask_q = 4'b0001:

eligible = 0011 & 0001 = 0001   → grant = 0001 (requester 0), round ends here
 
refill with ~pick  → mask_q = 1110
  next cycle: eligible = 0011 & 1110 = 0010 → grant = 0010 (requester 1)
 
refill with all-ones → mask_q = 1111
  next cycle: eligible = 0011 & 1111 = 0011 → grant = 0001 (requester 0 again)

The all-ones refill lets the requester that just won take the first slot of the new round as well. Excluding it costs one gate and removes a back-to-back win that a fairness measurement would otherwise have to explain.

Synthesis. A priority encoder over N bits plus an N-bit register. The ~eligible + 1'b1 idiom synthesises to a borrow chain and is the standard low-cost way to isolate the least-significant set bit; for large N a tree-based leading-one detector is the usual replacement. No latches — eligible, pick, grant and grant_valid are all continuous assignments with no conditional path that leaves them undriven.

Verification — what to prove about an arbiter

The properties below are safety invariants of the arbitration contract, not restatements of the implementation. They are what you would bind to any arbiter, including one you did not write.

// At most one requester is granted in any cycle. If this fails, two agents
// believe they own the shared service and the failure appears downstream as
// corrupted or interleaved traffic, far from the arbiter.
a_grant_onehot0: assert property (@(posedge clk) disable iff (!rst_n)
  $onehot0(grant));
 
// A grant is never issued to a requester that is not asking. The symptom of a
// violation is a requester receiving a completion it has no state for.
a_grant_implies_req: assert property (@(posedge clk) disable iff (!rst_n)
  (grant & ~req) == '0);
 
// Work is never handed to a service that cannot accept it.
a_no_grant_when_busy: assert property (@(posedge clk) disable iff (!rst_n)
  !svc_ready |-> (grant == '0));
 
// Progress: whenever a grant is issued, the fairness state must change.
// A round-robin arbiter whose mask never moves is a fixed-priority arbiter
// wearing the wrong name, and this is the cheapest way to catch that.
a_round_advances: assert property (@(posedge clk) disable iff (!rst_n)
  grant_valid |=> (mask_q != $past(mask_q)));

Bounded fairness needs a little state of its own, so the checker keeps a counter rather than trying to express it as one property. This is idiomatic and it reports a cycle count when it fails, which s_eventually does not:

// How long requester 0 has been asking while the service was able to serve.
logic [$clog2(N+1)-1:0] wait0_q;
 
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n)                   wait0_q <= '0;
  else if (grant[0] || !req[0]) wait0_q <= '0;
  else if (svc_ready)           wait0_q <= wait0_q + 1'b1;
end
 
// One round is at most N grants, so a held request must win inside N ready
// cycles. Starvation shows up here as a specific number, not as a hang.
a_bounded_fairness: assert property (@(posedge clk) disable iff (!rst_n)
  wait0_q < N);

The stimulus that actually tests this. All requesters asserting simultaneously and holding; svc_ready toggling irregularly, including long low periods; requesters withdrawing mid-arbitration; and reset asserted while grants are in flight. A regression that raises one request at a time satisfies every property above vacuously and proves nothing about fairness.

13. RTL 2 — The Outstanding-Transaction Tracker

Chapter 1.1 derived that sustaining bandwidth requires bandwidth × latency bytes in flight. This is the hardware that decides how many a requester is allowed.

// A requester may have at most MAX_OUTSTANDING requests awaiting completion.
// This single number decides whether the requester can cover the round-trip
// latency of whatever it is talking to — see the arithmetic below the module.
module outstanding_tracker #(
  parameter int unsigned MAX_OUTSTANDING = 8
) (
  input  logic clk,
  input  logic rst_n,
  input  logic issue_req,    // the requester wants to send a request this cycle
  input  logic cpl_valid,    // a completion returned this cycle
  output logic can_issue,    // a credit is available
  output logic issue_fire    // a request actually left this cycle
);
 
  localparam int unsigned CNT_W = $clog2(MAX_OUTSTANDING + 1);
 
  logic [CNT_W-1:0] outstanding_q;
 
  // The whole flow-control policy is this comparison. Everything upstream of
  // it — queue depth, arbitration, buffering — exists to keep it satisfiable.
  assign can_issue  = (outstanding_q < CNT_W'(MAX_OUTSTANDING));
  assign issue_fire = issue_req && can_issue;
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      outstanding_q <= '0;
    end else begin
      // Issue and completion in the same cycle cancel. Writing this as one
      // case rather than two independent increment/decrement statements is
      // what keeps the count exact when both happen together — the case a
      // directed test usually misses and random stimulus hits constantly.
      unique case ({issue_fire, cpl_valid})
        2'b10:   outstanding_q <= outstanding_q + 1'b1;
        2'b01:   outstanding_q <= outstanding_q - 1'b1;
        default: outstanding_q <= outstanding_q;   // 2'b00 and 2'b11
      endcase
    end
  end
 
endmodule

Architecture. A credit counter, and the sizing question that goes with it. From Little's Law, the concurrency needed to sustain a target rate is the product of that rate and the round-trip latency:

requests in flight = request rate × round-trip latency
 
to sustain one 64-byte request every 2 ns (32 GB/s) at 300 ns round trip:
 
  = (1 / 2e-9) × 300e-9
  = 150 requests in flight
 
with MAX_OUTSTANDING = 8, the achievable rate is instead
 
  = 8 / 300e-9
  = 26.7 million requests/s
  = 26.7e6 × 64 B
  ≈ 1.7 GB/s

A tracker sized at 8 against a 300 ns round trip delivers about 5% of the target rate, and no amount of link bandwidth changes that. This is the most common cause of an engine that "should" be fast and is not, and it is a design-time decision frozen in a parameter.

State. One counter, $clog2(MAX_OUTSTANDING+1) bits wide — 4 bits for a limit of 8, because the count must represent the value 8 itself.

Cycle behaviour. With issue_req held high and no completions, issue_fire pulses for 8 cycles and then stops; can_issue falls in the cycle the count reaches 8. Each returning completion re-opens exactly one slot.

Expected simulation output. Holding issue_req = 1 with completions arriving from cycle 12:

t=1..8   issue_fire=1  outstanding=1..8
t=9      issue_fire=0  outstanding=8   can_issue=0   <-- credit-starved
t=12     cpl_valid=1   outstanding=8   issue_fire=1  <-- one in, one out

Synthesis. One up/down counter and a comparator. The unique case gives the tool a complete, mutually exclusive decode — it does not create priority logic, and it documents the simultaneous case for the next reader. Note that unique also enables a simulation check for an unexpected combination, which is free.

Verification — the four properties that matter

// The count never exceeds the configured limit. If it does, the requester has
// more state outstanding than it has resources to track, and the extra
// completions will land on entries that have been reused.
a_no_overflow: assert property (@(posedge clk) disable iff (!rst_n)
  outstanding_q <= CNT_W'(MAX_OUTSTANDING));
 
// A completion with nothing outstanding is a protocol error. In hardware the
// decrement wraps rather than going negative, so the symptom is a tracker that
// reads FULL forever and a requester that never issues again — a silent hang,
// not a crash. This property turns that into a named failure at the cycle it
// happened.
a_no_underflow: assert property (@(posedge clk) disable iff (!rst_n)
  (cpl_valid && !issue_fire) |-> (outstanding_q != '0));
 
// Flow control is genuinely enforced, not merely advertised.
a_blocked_when_full: assert property (@(posedge clk) disable iff (!rst_n)
  (outstanding_q == CNT_W'(MAX_OUTSTANDING)) |-> !issue_fire);
 
// Simultaneous issue and completion leave the count unchanged.
a_simultaneous_is_neutral: assert property (@(posedge clk) disable iff (!rst_n)
  (issue_fire && cpl_valid) |=> $stable(outstanding_q));

The stimulus that actually tests this. Sustained issue pressure until the tracker is full and stays full; completions returning out of order and in bursts; issue and completion asserted in the same cycle, repeatedly; a completion injected with nothing outstanding, to prove the underflow property has teeth; and reset asserted with the count non-zero, to confirm it returns to zero rather than to a stale value.

14. RTL 3 — Backpressure, and How It Travels Upstream

The arbiter decides who wins. The tracker decides how much can be in flight. This is the block that connects them, and it is where centralisation stops being an abstraction: when the shared service stalls, the queue in front of it fills, and the refusal propagates to every requester at once.

// A depth-DEPTH request queue with a ready/valid interface on both sides.
// The architectural content is one signal: `in_ready`. Deasserting it is the
// only way this block has of telling an upstream requester to stop, and that
// refusal is the mechanism by which congestion at a shared service point
// becomes a stall at an engine that has nothing to do with the congestion.
module req_queue #(
  parameter int unsigned DEPTH = 4,
  parameter int unsigned W     = 32
) (
  input  logic         clk,
  input  logic         rst_n,
  // upstream — the requester side
  input  logic         in_valid,
  output logic         in_ready,
  input  logic [W-1:0] in_data,
  // downstream — the arbiter / shared service side
  output logic         out_valid,
  input  logic         out_ready,
  output logic [W-1:0] out_data
);
 
  localparam int unsigned PTR_W = $clog2(DEPTH);
  localparam int unsigned CNT_W = $clog2(DEPTH + 1);   // must represent DEPTH itself
 
  logic [W-1:0]     mem_q [DEPTH];
  logic [PTR_W-1:0] wr_ptr_q;
  logic [PTR_W-1:0] rd_ptr_q;
  logic [CNT_W-1:0] count_q;
 
  logic in_fire;
  logic out_fire;
 
  // Ready/valid: a transfer occurs only when both sides agree in the same
  // cycle. Neither side may wait for the other before deciding — that rule is
  // what keeps the protocol deadlock-free.
  assign in_ready  = (count_q != CNT_W'(DEPTH));
  assign out_valid = (count_q != '0);
  assign in_fire   = in_valid  && in_ready;
  assign out_fire  = out_valid && out_ready;
  assign out_data  = mem_q[rd_ptr_q];
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      wr_ptr_q <= '0;
      rd_ptr_q <= '0;
      count_q  <= '0;
    end else begin
      // The storage array is deliberately not reset — it infers RAM, and
      // resetting it would infer flops instead. count_q guarantees no entry is
      // read before it is written.
      if (in_fire) begin
        mem_q[wr_ptr_q] <= in_data;
        wr_ptr_q <= (wr_ptr_q == PTR_W'(DEPTH-1)) ? '0 : wr_ptr_q + 1'b1;
      end
      if (out_fire) begin
        rd_ptr_q <= (rd_ptr_q == PTR_W'(DEPTH-1)) ? '0 : rd_ptr_q + 1'b1;
      end
      unique case ({in_fire, out_fire})
        2'b10:   count_q <= count_q + 1'b1;
        2'b01:   count_q <= count_q - 1'b1;
        default: count_q <= count_q;
      endcase
    end
  end
 
endmodule

Architecture. Elasticity between a producer and a shared consumer. A queue absorbs a burst; it does not raise the consumer's service rate. Once the average offered load exceeds that rate, the queue is permanently full and its only remaining function is to convert the excess into a stall.

State. Two pointers, one count, and the storage array. The count is $clog2(DEPTH+1) bits rather than $clog2(DEPTH) because it must represent DEPTH itself — a genuinely common off-by-one that produces a queue reporting empty when it is full.

Cycle behaviour. With out_ready low, count_q climbs by one per accepted push. In the cycle it reaches DEPTH, in_ready falls, and the upstream requester stalls from that cycle onwards.

Expected simulation output. Pushing every cycle from reset with out_ready tied low, DEPTH = 4:

(count is the registered value during that cycle; in_ready is combinational from it)
 
t=0  count=0  in_ready=1  in_fire=1
t=1  count=1  in_ready=1  in_fire=1
t=2  count=2  in_ready=1  in_fire=1
t=3  count=3  in_ready=1  in_fire=1
t=4  count=4  in_ready=0  in_fire=0   <-- full: backpressure and stall, same cycle
t=5  count=4  in_ready=0  in_fire=0

Synthesis. A small dual-port RAM or a flop array depending on DEPTH and W, two pointer counters, one occupancy counter, and two comparators. Nothing here is latch-prone: every output is either a continuous assignment or a fully-reset flop.

What changes, and why. out_ready falls at t1 — the shared service can no longer accept work. count_q continues to rise because the queue still can, and that is exactly its job: for three cycles the requester is unaware anything is wrong. At t4 the count reaches DEPTH; in_ready is combinational from the count, so it and in_fire both fall in that same cycle and the requester is stalled from t4 onwards. When out_ready returns at t6 one entry drains, and in_ready recovers a cycle later once the count has actually dropped.

What a bug would look like. If in_ready were computed as count_q < DEPTH-1 — the off-by-one above — the queue would stall one entry early and never use its last slot, costing throughput with no functional symptom. If it were computed as count_q != DEPTH+1, the queue would accept a push when full and overwrite an entry that had not been read: no stall, no error, and one silently lost request whose completion never arrives. The second failure eventually presents as a hung requester with a non-zero outstanding count, which is the Debug Lab in Section 18.

Verification — proving backpressure is honoured

// Occupancy stays within the physical depth. A violation means an entry was
// overwritten before it was read, and the lost request's completion never
// arrives — a hang whose root cause is several thousand cycles earlier.
a_count_bounded: assert property (@(posedge clk) disable iff (!rst_n)
  count_q <= CNT_W'(DEPTH));
 
// Backpressure is real: a full queue accepts nothing.
a_full_blocks_push: assert property (@(posedge clk) disable iff (!rst_n)
  (count_q == CNT_W'(DEPTH)) |-> !in_fire);
 
// An empty queue offers nothing.
a_empty_offers_nothing: assert property (@(posedge clk) disable iff (!rst_n)
  (count_q == '0) |-> !out_valid);
 
// Ready/valid stability: once a requester asserts valid it must hold both
// valid and data until the transfer is accepted. Without this the queue can
// sample a value the requester has already withdrawn.
a_valid_stable_until_ready: assert property (@(posedge clk) disable iff (!rst_n)
  (in_valid && !in_ready) |=> (in_valid && $stable(in_data)));
 
// Simultaneous push and pop leave the occupancy unchanged — the same
// cancellation case as the tracker in Section 13.
a_simultaneous_is_neutral: assert property (@(posedge clk) disable iff (!rst_n)
  (in_fire && out_fire) |=> $stable(count_q));

The stimulus that actually tests this. Sustained backpressure long enough to hold the queue full for thousands of cycles, not a few; out_ready toggling every cycle, which exercises the simultaneous push-and-pop path continuously; a burst arriving in the exact cycle the queue fills; and reset asserted with the queue partly full. The last one catches designs that reset the pointers but not the count, or the reverse — a queue that comes out of reset believing it holds three entries it does not have.

15. RTL 4 — Completion Aggregation and the Interrupt Path

Sections 12 to 14 modelled the request direction. The return direction has its own convergence point, and in a CPU-centric system it is a particularly sharp one: every device's completions eventually have to become work for host software.

// Completions from NDEV device engines are counted into one pending total.
// An interrupt is raised only once COALESCE completions are outstanding —
// interrupt coalescing, in its smallest honest form. The host acknowledges,
// and the count clears.
//
// The architectural point: the host's notification path is a single resource,
// and coalescing is the standard way of trading latency for host CPU cost.
module cpl_aggregator #(
  parameter int unsigned NDEV     = 4,
  parameter int unsigned COALESCE = 4,
  parameter int unsigned CNT_W    = 16
) (
  input  logic            clk,
  input  logic            rst_n,
  input  logic [NDEV-1:0] cpl_pulse,        // one-cycle pulse per completing engine
  input  logic            host_ack,         // host consumed the pending count
  output logic            irq,
  output logic            overflow_sticky
);
 
  logic [CNT_W-1:0] pending_q;
  logic [CNT_W-1:0] arrivals;
  logic [CNT_W-1:0] next_sum;
 
  // Several engines may complete in the same cycle, so arrivals is a popcount
  // rather than a single bit. This is the hardware statement of "completions
  // are concurrent, host processing of them is not."
  assign arrivals = CNT_W'($countones(cpl_pulse));
  assign next_sum = pending_q + arrivals;
 
  assign irq = (pending_q >= CNT_W'(COALESCE));
 
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      pending_q       <= '0;
      overflow_sticky <= 1'b0;
    end else if (host_ack) begin
      // The host consumed everything it was told about. Completions arriving
      // in this same cycle were NOT included in that total, so they must be
      // carried forward rather than cleared — see the Debug Lab in Section 18.
      pending_q <= arrivals;
    end else begin
      // Unsigned wrap detection: a sum smaller than one of its addends wrapped.
      if (next_sum < pending_q) overflow_sticky <= 1'b1;
      pending_q <= next_sum;
    end
  end
 
endmodule

Architecture. A convergence point on the return path. NDEV engines complete concurrently; one host is told about it. COALESCE is the dial between two failure modes — set it low and the host drowns in interrupts, set it high and completion latency grows and short jobs sit unnoticed.

State. One pending counter and one sticky overflow bit. The sticky bit is diagnostic rather than functional, and it is the cheapest useful instrumentation in the module.

Cycle behaviour. With COALESCE = 4 and single completions arriving on consecutive cycles, irq rises in the cycle after the fourth arrival and stays high until host_ack.

Expected simulation output. Four engines completing over several cycles, COALESCE = 4:

t=1  cpl_pulse=0001  arrivals=1  pending=1  irq=0
t=2  cpl_pulse=0110  arrivals=2  pending=3  irq=0
t=3  cpl_pulse=1000  arrivals=1  pending=4  irq=1   <-- threshold reached
t=4  host_ack=1      arrivals=1  pending=1  irq=0   <-- arrival in the ack cycle survives

Line 4 is the whole point of the module. A naive implementation writes zero on host_ack and loses that completion permanently.

Synthesis. An adder, a comparator, a popcount over NDEV bits, and two registers. $countones is IEEE 1800 and is supported by the mainstream synthesis tools; where a flow rejects it, an explicit adder tree over the pulse bits is the direct replacement and produces the same logic.

Verification — completions must not evaporate

// A completion arriving in the same cycle as the acknowledgement must survive.
// This is the single most valuable property in the module and the one a
// directed test almost never generates.
a_ack_race_preserves_arrivals: assert property (@(posedge clk) disable iff (!rst_n)
  (host_ack && (|cpl_pulse)) |=> (pending_q != '0));
 
// Without an acknowledgement the pending count never decreases. A clear-on-read
// side effect somewhere else in the design shows up here immediately.
a_no_silent_drop: assert property (@(posedge clk) disable iff (!rst_n)
  !host_ack |=> (pending_q >= $past(pending_q)));
 
// The overflow flag is sticky, so a single wrap anywhere in a long regression
// is still visible at the end of it.
a_overflow_is_sticky: assert property (@(posedge clk) disable iff (!rst_n)
  overflow_sticky |=> overflow_sticky);
 
// The counter must be wide enough that overflow is unreachable in a healthy
// design. If this fires, CNT_W or the acknowledgement rate is wrong — the
// property is a sizing check, not a functional one.
a_no_overflow_in_regression: assert property (@(posedge clk) disable iff (!rst_n)
  !overflow_sticky);

The stimulus that actually tests this. All NDEV engines completing in the same cycle, repeatedly; host_ack asserted in the same cycle as arrivals, which is the property above; a long acknowledgement outage to drive the count towards its width limit; and an acknowledgement with nothing pending. Note that the last property is deliberately one a hostile test can break — that is what makes it a sizing check worth running.

16. What Happens When You Add Accelerators

The four modules above each have a fixed capacity. Adding engines does not change any of them. What it does change is the amount of coordination the system has to perform, and that grows faster than the engine count.

Worked example D — coordination does not scale linearly

Count the structures a host must maintain. The per-device figures are illustrative of a common pattern rather than a claim about any product:

per device: 1 submission queue + 1 completion queue per host thread,
            1 interrupt vector, 1 doorbell region, 1 mapping context
 
with 32 host threads:
 
   1 device  →   64 queues,   1 vector
   4 devices →  256 queues,   4 vectors
  16 devices → 1024 queues,  16 vectors

Queue count grows with the product of devices and threads, and every one of those queues is memory that must stay resident, indices two agents must keep consistent, and a structure some code has to poll or be interrupted about.

Now count the data movement when devices need each other's results and every exchange is staged through host memory:

all-to-all exchange among N devices, host-mediated:
 
  transfers = N × (N − 1)
 
   2 devices →    2 transfers
   4 devices →   12 transfers
   8 devices →   56 transfers
  16 devices →  240 transfers

Each of those transfers is two DRAM traversals and two link traversals, and all of them converge on the same host memory. Sixteen devices produce twenty times the host-mediated traffic of four, for four times the compute. The host did not get slower; it was simply never the part being scaled.

The general form is worth stating plainly, because it is what "does not scale" actually means here:

Offered load grows with the number of engines. Coordination grows with the number of relationships between them. Service capacity at the centre grows with neither.

Fixed attachment, and what it strands

There is one more consequence of the CPU owning memory, and it is a fleet-level one. Chapter 1.1 introduced stranding as the utilisation axis of the memory wall; here it is a direct consequence of the ownership structure rather than of any resource shortfall.

Server A          Server B
  CPU:   busy       CPU:   idle
  DRAM:  90% used   DRAM:  70% free

Server A is short of memory. Server B has memory. Nothing can be done about it, because the DRAM in Server B is behind Server B's memory controller, addressable only by Server B's cores, and mapped by Server B's page tables. The same is true one level down: memory behind socket 0 is not equally usable by socket 1, and memory inside a device is not usable by anything except that device.

The capacity is not missing. It is attached to the wrong owner, and attachment was decided when the machine was built. That is a topology property, and no improvement in DRAM technology addresses it.

17. Centralization Is a Design Choice, Not a Mistake

It would be easy to read Sections 11 to 16 as an argument that central resources are bad. They are not, and an engineer who draws that conclusion will design worse systems.

What centralisation genuinely buys:

• One ordering point. An arbiter creates a single, total order over shared access. Enormous amounts of downstream logic get simpler because of it, and a great many correctness arguments become tractable.
• Simple ownership. One agent decides who has what. Lifetime, protection and cleanup have one implementation.
• Policy in one place. Priority, quality of service, rate limiting and security checks sit on a path everything already takes, so they cannot be bypassed.
• A tractable failure model. One place to instrument, one place to look, one answer to "who did this?".
• Cheap correctness. Distributed state — multiple agents caching, deciding and recovering independently — is the single largest source of protocol complexity in any system. Avoiding it when you can is good engineering, not timidity.

What it costs at scale:

• Serialisation. One winner per cycle, however many are asking.
• Concentrated bandwidth. Every path crosses the centre, so the centre needs the sum of everyone's bandwidth.
• A shared performance domain. One congested requester slows unrelated ones through backpressure, as Figure 4 showed.
• Coordination overhead. Every participant must be told about, mapped for, and reported to.
• A shared blast radius. The failure model that was simple is also singular.

The engineering question is per-resource, not per-system. Ordering and policy have excellent reasons to stay central. Bulk movement and read-mostly sharing have almost none — they are central in a conventional system as a side effect of the CPU owning memory, not because anyone decided they should be.

That distinction is the one this chapter exists to establish, and Section 22 is where it points.

18. Debugging: What CPU-Centric Scaling Failure Looks Like

Scaling failures in this class share a frustrating property: nothing reports an error. The system is slower than it should be, every component says it is healthy, and the binding resource is whichever one nobody instrumented. What follows is the reasoning, not a list of vendor commands.

Symptom — accelerator utilisation is low and the link is not busy

Work through the possibilities in order of how cheaply they can be eliminated:

• Submission starvation. The host is not producing work fast enough. Check submission rate against the device's service rate; if the queue in front of the device is usually empty, the problem is upstream and Section 6's arithmetic applies.
• Insufficient concurrency. The device has work but too few requests in flight to cover the round trip. Section 13's arithmetic; the counter to look at is outstanding occupancy, not bandwidth.
• Serialised copy and compute. The device is idle during transfers. Compare total copy time against total compute time — Section 8's example B.
• Synchronisation. The device finishes and waits for the host to notice. Look at the gap between completion posted and completion processed, which is Section 15's coalescing trade.

Symptom — CPU utilisation rises as devices are added, and throughput does not

This is the control-plane signature, and it is diagnostic on its own. Attribute the CPU time before doing anything else: if it is in the driver, the interrupt path, the runtime's synchronisation, or memory management rather than in the application, the bottleneck is orchestration. Section 6's remedies apply — larger batches, fewer notifications, submission paths that do not enter the kernel — and buying faster cores does not.

Symptom — throughput stops scaling past N devices

Something shared saturated. The candidates, and what distinguishes them:

The single most useful habit: compare against a computed bound rather than against the previous configuration. "It got 8% faster" is not a diagnosis. "Aggregate demand is 340 GB/s against a 409 GB/s bound, so we are at 83% of the memory-controller limit" is.

// The completion path decrements unconditionally, on the assumption that a
// completion only ever arrives for a request that was issued.
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n)          outstanding_q <= '0;
  else begin
    if (issue_fire)    outstanding_q <= outstanding_q + 1'b1;
    if (cpl_valid)     outstanding_q <= outstanding_q - 1'b1;
  end
end

unique case ({issue_fire, cpl_valid})
  2'b10:   outstanding_q <= outstanding_q + 1'b1;
  2'b01:   outstanding_q <= outstanding_q - 1'b1;
  default: outstanding_q <= outstanding_q;
endcase
 
a_no_underflow: assert property (@(posedge clk) disable iff (!rst_n)
  (cpl_valid && !issue_fire) |-> (outstanding_q != '0));

// The host has acknowledged, so clear the pending count.
always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n)         pending_q <= '0;
  else if (host_ack)  pending_q <= '0;                    // <-- loses same-cycle arrivals
  else                pending_q <= pending_q + arrivals;
end

else if (host_ack)  pending_q <= arrivals;   // credit this cycle's arrivals
 
a_ack_race_preserves_arrivals: assert property (@(posedge clk) disable iff (!rst_n)
  (host_ack && (|cpl_pulse)) |=> (pending_q != '0));

// Fixed priority: index 0 wins whenever it asks.
always_comb begin
  grant = '0;
  for (int i = 0; i < N; i++)
    if (req[i] && grant == '0) grant[i] = 1'b1;
end

assign eligible = ((req & mask_q) != '0) ? (req & mask_q) : req;
assign pick     = eligible & (~eligible + 1'b1);
 
a_bounded_fairness: assert property (@(posedge clk) disable iff (!rst_n)
  wait0_q < N);

19. How This Appears in Real Engineering

SoC architect

The questions are about authority and convergence, and they are asked before any RTL exists. Where does orchestration live, and how much of it is on a critical path? Which agents may initiate traffic, and which may only respond? Which resources are shared, and what is each one's service rate against the aggregate demand? Where does traffic serialise, and is that serialisation buying an ordering guarantee somebody needs? Who owns coherence, and does any device need to participate in it? Who translates addresses, and what does a translation miss cost? Where must buffering exist to absorb bursts, and how deep? And the question that catches the most architecture: what happens to everyone else when one master stalls?

RTL and microarchitecture engineer

This chapter's four modules are the vocabulary. Request queues and their depths. Arbiters and their fairness policy. Credit schemes and outstanding-transaction tables. Tag allocation and completion matching for out-of-order returns. Reorder buffers where the consumer needs in-order delivery from an out-of-order source. Backpressure paths and the timing they close on. Starvation, which is a liveness property and therefore invisible to the safety checks most blocks ship with. The recurring judgement is sizing: depth, credits and tag count are all instances of the same Little's Law arithmetic, and all three are frozen at design time.

Verification engineer

The failures in this class are load-dependent and silent, so the interesting stimulus is never a correct single transaction. Every requester active simultaneously and holding. Queues held full for long periods rather than momentarily. Completions returning out of order, in bursts, and in the least convenient order. Simultaneous assertion of any two control signals that can occur together — the two Debug Labs above are both exactly this. Resource exhaustion at every limit, and one beyond it. Reset while traffic is in flight, with non-zero counters. Timeouts, and what the design does when one fires. And starvation scenarios run long enough for unfairness to be measurable, since a fairness bug that costs one requester 5% is invisible in a thousand-cycle test.

Performance engineer

The work is attribution, and it needs both a rate and an occupancy. Queue occupancy distinguishes a saturated service from a starved one. Request latency separates queueing delay from service time. Issue rate against service rate says which side of the queue is binding. Outstanding-depth saturation says whether a requester is credit-limited — the single most common cause of an engine that underperforms its link. Stall-cycle attribution says what a stalled engine was waiting for. And control-plane utilisation — CPU time in driver, interrupt and runtime paths, measured separately from application time — is the counter that distinguishes this chapter's bottleneck from Chapter 1.1's, and the one most often missing.

System software and firmware engineer

The control plane is this role's implementation. Descriptor formats and the ordering rules that make a ring safe between two agents. Queue configuration: how many, how deep, per-core or shared. Interrupt configuration, affinity and coalescing thresholds. Mapping and pinning policy, and what it costs the memory manager. Synchronisation between the submitting thread, the completion path and the device. Buffer ownership protocols, and the cache maintenance that goes with them where the device is not coherent. Device initialisation and teardown, including what happens when a device disappears while it owns host memory. Every item in Section 6's list is somebody's code, and its cost is measured in this role's profile.

20. Common Misconceptions

21. Interview Reasoning

CPU-centric scaling is a favoured senior-level topic because it cannot be answered from memory. Every question below has a shallow answer that sounds right, and the follow-up is what separates the two.

22. Summary

A CPU-centric architecture concentrates four kinds of authority in one agent: ownership of memory, ownership of coherence, ownership of address translation, and the right to initiate work. That concentration is why an enormous number of systems are correct, debuggable and simple to reason about, and none of those benefits have gone away.

What has changed is the population of devices. Engines that execute independently, hold local memory, cache shared data and exchange results with each other are peers, and a structure with one universal owner has no authority to give them without changing the structure. The costs show up in four places. Control-plane work — mapping, descriptors, doorbells, completions, synchronisation — is per operation and independent of whether the CPU touches a byte, so a system can be CPU-bound with the data path idle. Copies are forced by memory domains that cannot address each other, and they amplify: roughly four times a working set can traverse DRAM so that one copy of it is computed on. Software-managed consistency for shared data is expensive, coarse and a correctness cliff. And fixed attachment strands capacity behind an owner that does not need it.

In hardware the pattern is always the same shape: requesters converge on a service point, an arbiter serialises them, a queue absorbs the difference, and when the service stalls the refusal propagates to everyone at once. The four modules in Sections 12 to 15 are that shape, and their parameters — grant policy, outstanding depth, queue depth, coalescing threshold — are where the architectural argument becomes a number somebody has to choose. Adding requesters raises offered load; it raises nothing else.

Hold two conclusions together, because holding only one of them produces bad architecture. Centralisation is not the problem — a single ordering point, one owner and unbypassable policy are worth paying for, and distributed state is the most expensive thing in any system. The problem is centralising resources that never needed a central opinion, which in a conventional machine means bulk data movement and read-mostly sharing, both of which are central only as a side effect of the CPU owning memory.

Which produces the question the rest of Module 1 answers: which of these resources has genuinely outgrown a single owner, and what would it take to move only those without giving up coherence, isolation or compatibility?

23. What Comes Next

The direction the industry took follows from separating the two conclusions above.

Ordering, policy, isolation and scheduling stay where they are — the host still runs the operating system, still owns the process address space, still schedules, and still initiates most work. Nothing in this track argues otherwise, and any claim that a link protocol removes the CPU from a system should be treated as a warning about the source.

What changes is narrower and more interesting: the attachment and sharing model, so that certain memory and coherence relationships no longer have to look like traditional peripheral I/O behind CPU-local resources. A device that can address host memory coherently does not need its input staged. Memory that is not behind exactly one socket is not stranded by that socket's utilisation. A device holding cached copies of shared lines does not need software flush and invalidate at every handoff. Each of those removes one specific consequence of this chapter — and none of them removes the CPU.

Module 1 continues by pricing the pieces this chapter has only named. Chapter 1.3 puts numbers on what moving bytes costs in energy and time. Chapter 1.4 explains why accelerators proliferated in the first place and what that did to device attach. Chapter 1.5 examines why discrete accelerator memory cannot grow with working sets. Chapter 1.6 is the honest accounting of what conventional device attach provides and where it stops. Chapter 1.7 then makes the case for coherent attach as a mechanism — which is the first chapter in this track where the answer, rather than the pressure, is the subject.

For the link-side view of several arguments here, the PCIe track covers DMA over PCIe and host memory access in depth, interrupt performance for the completion path, and AI accelerators for copy-compute overlap. The AXI track builds the hardware of this chapter properly: outstanding depth tuning, backpressure and stalls, and arbitration. The full path is on the CXL tutorials index.