Multi-Core Scaling

Module 15 · Chapter 15.4 · CHI Performance

Project thread — 15.3 scaled aggregate throughput by distributing the fabric. 15.4 finds the shared-resource ceiling on per-core; 15.5 turns to the directory.

1. Learning Outcomes

By the end of this chapter you should be able to:

• Distinguish aggregate scaling (15.3) from per-core scaling under contention.
• State that total throughput is min(N × per-core demand, shared bandwidth).
• Locate the knee — where combined demand meets the shared resource — at N* = shared BW ÷ demand.
• Explain that past the knee, per-core throughput falls as ~1/N.
• Diagnose wasted cores added past a saturated shared resource.
• Implement a representative per-core throughput model in SystemVerilog, Verilog-2001, and VHDL.

2. Why Should I Learn This?

Chapter 15.3 showed how to make aggregate throughput grow with node count by distributing the fabric. But a user of a single core does not care about aggregate throughput — they care about their core's performance, and that can fall even as aggregate rises. The two metrics diverge the moment cores start contending for a shared resource that the fabric cannot distribute away: memory bandwidth, a fixed pool sized by the memory controllers, not the mesh.

The behavior has a sharp knee. Below it, the cores' combined demand fits within the shared bandwidth, and each core runs at full speed — total scales linearly. Above it, the shared resource is saturated: total throughput is pinned, and the fixed total is split among more cores, so per-core throughput drops as 1/N. Cores added past the knee are not just wasted — they degrade everyone else through contention. Knowing where the knee is tells you when to stop adding cores or, better, when to scale the shared resource alongside them. This chapter is that ceiling and how to see it coming.

3. Key Terms

4. Previous Chapter Connection

Chapter 15.3 removed the centralized-home bottleneck so aggregate throughput could scale. This chapter is the bottleneck that distribution cannot remove — a genuinely shared resource. The mesh can distribute the home and the directory, but all the traffic that misses the caches still converges on the memory controllers, whose bandwidth is a fixed pool.

It is the same Amdahl idea as 15.3, applied to a resource you cannot distribute. In 15.3 the bottleneck (a single home) was artificial — the fix was to spread it across many homes. Here the bottleneck (memory bandwidth) is fundamental — you cannot hash it away, only add more of it (more channels). So 15.3 and 15.4 are the two halves of scaling: distribute what can be distributed (the fabric), and provision what cannot (the shared memory bandwidth). This chapter is the second half, and it sets up the directory-efficiency levers of 15.5–15.6 that reduce the demand on that shared pool.

5. Core Concept — a shared pool with a knee

Total throughput is min(combined demand, shared bandwidth); below the knee it scales with cores, above it the shared resource saturates and per-core falls as 1/N.

• Total is a min. With N cores each demanding d bandwidth and a shared pool of B, total throughput is min(N × d, B) — the cores get what they ask for or the pool's capacity, whichever is smaller.
• Below the knee: linear. While N × d < B, the pool is not full, so total ≈ N × d and per-core stays at d. Adding cores adds throughput.
• The knee. At N* = B ÷ d, combined demand equals the shared bandwidth. This is the last core count that scales.
• Above the knee: 1/N. For N > N*, the pool is saturated: total is pinned at B, and per-core = B ÷ N — falling as more cores split the fixed total.

The synthesis:

With N cores sharing a fixed bandwidth pool B, each demanding d, total throughput is min(N × d, B). Below the knee N* = B ÷ d the pool has room and throughput scales; above it the pool saturates, total is pinned at B, and per-core throughput falls as B ÷ N (~1/N). Cores added past the knee win no aggregate throughput and degrade per-core through contention.

6. Engineering Mental Model — straws in one drink

Think of a group sharing a single large drink (the memory bandwidth) through straws (cores).

• With a few straws, each person sips as fast as they like — the drink flows freely, and adding a straw lets one more person drink at full speed. Total consumption rises.
• But the drink can only flow out so fast — its neck is a fixed limit (the shared bandwidth). Once enough straws are pulling to hit that limit, the drink is coming out as fast as it can.
• Add more straws past that point and the same flow is now split among more people — each person gets less. Total consumption is unchanged (the neck is maxed); per-straw consumption drops.
• Beyond the limit, every new straw makes everyone else's sip weaker. The straws are not adding capacity; they are dividing a fixed flow.

The neck of the drink is the memory bandwidth; the straws are cores. Up to the neck's limit, more straws help; past it, more straws only share out a fixed flow, and per-straw performance falls as 1/N.

7. Engineering Diagram — cores sharing a fixed pool

Below the knee the pool has room and total tracks N·d; above it the pool is full, total is B, and per-core is B/N. The knee is where the two regimes meet. The DebugLab adds cores on the right side of the knee and expects the left side's linear scaling.

8. Below vs Above the Knee

The two regimes contrasted.

The rule to carry: there is a core count past which the shared resource, not the cores, sets performance — and adding cores there is counterproductive. Below the knee cores are the constraint (compute-bound) and adding them helps; above it the shared bandwidth is the constraint (memory-bound) and adding cores only divides a fixed total. The knee N* = B ÷ d is the design's scaling limit for that workload — and lowering d (Chapters 15.5–15.6 reduce memory traffic) or raising B (more channels) is the only way to move it.

9. Moving the Knee

How to scale past the shared-resource ceiling.

• Raise the shared bandwidth B. Add memory channels / controllers so the pool is larger. The knee N* = B ÷ d moves right — more cores scale before saturation.
• Lower the per-core demand d. Reduce each core's memory traffic — bigger or smarter caches, a system-level cache (Chapter 13.2), snoop/directory efficiency (Chapters 15.5–15.6). Lower d also moves the knee right.
• Balance cores and bandwidth. Provision B and core count together so the knee sits beyond the target core count — cores and bandwidth scale in step.
• Or accept the ceiling. If B cannot grow, stop adding cores at the knee — cores past it are wasted silicon that degrade per-core performance.

The point to carry:

Multi-core scaling exposes a design truth that a single-core mindset misses: adding compute can reduce delivered performance. Below the knee, more cores is strictly good; above it, more cores is strictly bad for the per-core experience, because they contend for a saturated pool and the fixed total is spread thinner. This non-monotonicity means "how many cores" is not "as many as fit" — it is a balance against the shared bandwidth, and the right answer is the knee N* for the target workload. The corollary is that cache and coherence efficiency are throughput features, not just latency features: every miss avoided lowers d, which pushes the knee right and lets more cores scale before the memory wall. This is why Chapters 15.5–15.6 (directory efficiency, snoop reduction) belong in a performance module — they reduce the demand on the shared pool, and reducing d is exactly what buys more scalable cores.

10. Scaling a Chip — before and after the knee

A shared memory pool of B = 64 GB/s; each core demands d = 8 GB/s. Knee at N* = 64 ÷ 8 = 8 cores.

1. 4 cores. Combined demand 32 GB/s < 64. Total = 32 GB/s; per-core = 8 GB/s (full). Below the knee.
2. 8 cores. Combined demand 64 GB/s = B — right at the knee. Total = 64 GB/s; per-core = 8 GB/s. Last count that scales.
3. 16 cores. Combined demand 128 GB/s > 64 — the pool saturates. Total = 64 GB/s (unchanged); per-core = 64 ÷ 16 = 4 GB/s — halved.
4. 32 cores. Total still 64 GB/s; per-core = 64 ÷ 32 = 2 GB/s — a quarter of full. The 24 cores past the knee added zero total throughput and halved per-core twice over.

The system scaled to the knee (8 cores) and then stopped — past it, cores only divided a fixed 64 GB/s. The DebugLab is going from 8 to 16 cores expecting 2× total.

11. RTL / Hardware View — a per-core throughput model

Total throughput is the min of combined demand and the shared pool; per-core is total over N. Representative analysis model.

// Representative per-core throughput model (educational).
// total = min(N * per_core_demand, shared_bw). Below the knee (N*d < B) total scales
// with N and per-core holds; at/above the knee the shared pool saturates, total pins at
// B, and per-core = B / N (~1/N). Adding cores past the knee wins no total throughput.
module chi_percore_throughput #(parameter W = 32) (
  input  logic [W-1:0] num_cores,     // N
  input  logic [W-1:0] per_core_demand, // d
  input  logic [W-1:0] shared_bw,     // B (fixed pool)
  output logic [2*W-1:0] total_tput,  // min(N*d, B)
  output logic [W-1:0]   per_core_tput,// total / N
  output logic           saturated    // N*d >= B
);
  logic [2*W-1:0] demand;
  always_comb begin
    demand        = num_cores * per_core_demand;         // combined demand N*d
    saturated     = (demand >= shared_bw);
    total_tput    = saturated ? shared_bw : demand;      // min(N*d, B)
    per_core_tput = (num_cores != 0) ? (total_tput / num_cores) : '0;
  end
endmodule

The same behavior in Verilog-2001:

// Representative per-core throughput model (Verilog-2001).
module chi_percore_throughput #(parameter W = 32) (
  input  [W-1:0]   num_cores, per_core_demand, shared_bw,
  output [2*W-1:0] total_tput,
  output [W-1:0]   per_core_tput,
  output           saturated
);
  wire [2*W-1:0] demand = num_cores * per_core_demand;
  assign saturated    = (demand >= shared_bw);
  assign total_tput   = saturated ? shared_bw : demand;
  assign per_core_tput = (num_cores != 0) ? (total_tput / num_cores) : {W{1'b0}};
endmodule

And in VHDL:

-- Representative per-core throughput model (VHDL).
library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
 
entity chi_percore_throughput is
  generic ( W : integer := 32 );
  port (
    num_cores, per_core_demand, shared_bw : in unsigned(W-1 downto 0);
    total_tput    : out unsigned(2*W-1 downto 0);
    per_core_tput : out unsigned(W-1 downto 0);
    saturated     : out std_logic
  );
end entity;
 
architecture rtl of chi_percore_throughput is
begin
  process (num_cores, per_core_demand, shared_bw)
    variable demand : unsigned(2*W-1 downto 0);
    variable tot    : unsigned(2*W-1 downto 0);
  begin
    demand := num_cores * per_core_demand;
    if demand >= resize(shared_bw, 2*W) then
      saturated <= '1';
      tot := resize(shared_bw, 2*W);
    else
      saturated <= '0';
      tot := demand;
    end if;
    total_tput <= tot;
    if num_cores /= 0 then
      per_core_tput <= resize(tot / num_cores, W);
    else
      per_core_tput <= (others => '0');
    end if;
  end process;
end architecture;

All three cap total at the shared pool (min(N*d, B)) and divide by N for per-core — so past the knee, total is flat and per-core falls. The DebugLab assumes total keeps rising and per-core stays flat past saturation.

12. Verification View — total is capped, per-core falls past the knee

The properties tie throughput to the shared pool: total never exceeds B, and it is pinned once saturated.

// Bind to chi_percore_throughput.
// 1. Total throughput never exceeds the shared bandwidth pool.
property p_total_capped;
  @(*) total_tput <= shared_bw;
endproperty
 
// 2. Once saturated, total is pinned at the shared bandwidth (adding cores adds nothing).
property p_saturated_pins_total;
  @(*) saturated |-> (total_tput == shared_bw);
endproperty
 
// 3. Below the knee, total scales with demand (per-core holds at d).
property p_unsaturated_scales;
  @(*) (!saturated) |-> (total_tput == num_cores * per_core_demand);
endproperty

The system point, beyond the checks:

The model formalizes the knee as the point where the saturated flag flips, and the two regimes it separates are the crux of every capacity-planning conversation. Before the flip, throughput questions are compute questions — how many cores, how fast. After the flip, they are memory questions — how much bandwidth — and answering a memory-bound problem with more cores is the classic capacity-planning error. The saturated flag is therefore the single most important output: it tells you which resource you are actually buying when you scale. A design that ignores it will keep adding cores into a saturated pool, spending area and power for negative marginal per-core return, and the failure is invisible in aggregate metrics (total throughput looks "fine" — it is at B) while every user's individual experience degrades. This is why per-core throughput, not just aggregate, must be a tracked metric: aggregate hides saturation, per-core reveals it.

• What it proves: total is capped at B and pinned once saturated; below the knee it scales.
• What it does not prove: the real per-core demand d — that requires workload profiling.
• Bug signature: provisioning that adds cores while saturated is already true.

13. Testbench — cores past the knee add no total throughput

Sweeps core count across the knee and checks total pins while per-core falls.

module tb_chi_percore_throughput;
  localparam W = 32;
  logic [W-1:0] num_cores, per_core_demand, shared_bw;
  logic [2*W-1:0] total_tput;
  logic [W-1:0] per_core_tput;
  logic saturated;
  int errors = 0;
 
  chi_percore_throughput #(.W(W)) dut (.*);
 
  task show(input int n);
    begin
      num_cores = n; #1;
      $display("N=%0d total=%0d per_core=%0d %s", n, total_tput, per_core_tput,
               saturated ? "(saturated)" : "");
    end
  endtask
 
  initial begin
    per_core_demand = 8; shared_bw = 64;   // knee at N* = 8
    show(4);   // below knee: total 32, per-core 8
    show(8);   // at knee:    total 64, per-core 8
    if (total_tput !== 64 || per_core_tput !== 8) begin errors++; $display("FAIL at knee"); end
    show(16);  // above:      total 64 (pinned), per-core 4
    if (total_tput !== 64) begin errors++; $display("FAIL total not pinned past knee"); end
    if (per_core_tput !== 4) begin errors++; $display("FAIL per-core not halved"); end
    show(32);  // above:      total 64 (pinned), per-core 2
    if (total_tput !== 64 || per_core_tput !== 2) begin errors++; $display("FAIL 32-core"); end
 
    if (errors == 0) $display("ALL TESTS PASSED");
    else             $display("%0d FAILURE(S)", errors);
    $finish;
  end
endmodule

Expected output:

N=4 total=32 per_core=8
N=8 total=64 per_core=8
N=16 total=64 per_core=4 (saturated)
N=32 total=64 per_core=2 (saturated)
ALL TESTS PASSED

14. DebugLab — adding cores past a saturated shared resource

shared memory BW B = 64 GB/s; per-core demand d = 8 GB/s -> knee N* = 8 cores
at 8 cores: total = 64 GB/s (pool full), per-core = 8 GB/s
"scale up": 8 -> 16 cores, expecting total 128 GB/s
  -> combined demand 128 > 64 -> pool SATURATED
  -> total pinned at 64 GB/s (NOT 128), per-core = 64/16 = 4 GB/s (HALVED)
  -> 8 added cores: 0 extra total throughput, everyone slower via contention
correct: raise B (more memory channels) or lower d (caching) to move the knee

15. Common Mistakes

• Adding cores past the knee. Assumption: linear scaling continues. Bug: zero total gain, per-core falls (the DebugLab). Prevention: compute N*.
• Tracking only aggregate throughput. Assumption: aggregate reflects the user. Bug: saturation hidden, per-core degrades. Prevention: track per-core too.
• Not scaling memory with cores. Assumption: bandwidth is free. Bug: memory-bound plateau. Prevention: scale B with core count.
• Ignoring per-core demand. Assumption: d is negligible. Bug: knee hit early. Prevention: reduce d via caching.
• Assuming the fabric is the limit. Assumption: a better mesh helps. Bug: memory is the wall. Prevention: identify the shared resource.
• Confusing with aggregate scaling. Assumption: 15.3 covers it. Bug: distribution does not add bandwidth. Prevention: 15.3 distributes; 15.4 provisions.

16. Engineering Checklist

• Identify the shared resource (usually memory bandwidth B).
• Estimate the per-core demand d for the workload.
• Compute the knee N* = B ÷ d — the scaling limit.
• Track per-core throughput, not just aggregate.
• To scale past N*, raise B (more channels) or lower d (caching, efficiency).
• If neither is possible, stop adding cores at the knee.

17. Key Takeaways

• Total throughput is min(N × per-core demand, shared bandwidth).
• Below the knee N* = B ÷ d, throughput scales and per-core holds.
• Above the knee the shared pool saturates: total pins at B, per-core falls as 1/N.
• Cores past the knee add no total throughput and degrade per-core via contention.
• Move the knee by raising B (more memory) or lowering d (caching, efficiency).
• Distribution (15.3) cannot fix a shared resource; the model here is representative.

18. Quick Revision

Multi-core scaling. Distributing the fabric (Chapter 15.3) grows aggregate throughput, but per-core throughput is limited by a genuinely shared resource the mesh cannot distribute — usually memory bandwidth, a fixed pool B. With N cores each demanding d, total throughput is min(N × d, B). Below the knee N* = B ÷ d, the pool has room: total scales ≈ N × d and per-core holds at d (compute-bound). At and above the knee the pool saturates: total is pinned at B, and the fixed total splits among more cores, so per-core throughput falls as B ÷ N (~1/N) (memory-bound). Cores added past the knee win zero aggregate throughput and degrade per-core through contention — wasted silicon that slows everyone. The failure to avoid: adding cores expecting the below-knee linear scaling to continue when the shared pool is already saturated. Move the knee right by raising B (more memory channels) or lowering d (caching, a system-level cache, directory/snoop efficiency — Chapters 15.5–15.6), or stop adding cores at the knee. Track per-core throughput, since aggregate hides saturation. Distribution fixes an artificial bottleneck (15.3); a shared resource must be provisioned, not distributed. Representative model; 15.5 turns to directory efficiency, one way to lower d.

Coming Next

Chapter 15.5 — Directory Efficiency. One way to lower per-core memory demand is to make coherence itself efficient, starting with the directory. Chapter 15.5 covers directory efficiency — how the directory's hit and miss rates drive snoop precision, why an undersized directory evicts entries and forces broadcast snoops and back-invalidations, and how sizing the directory to the aggregate cache footprint keeps the hit rate high.