Why Concurrency Appears in Engineering Code

Part I · Module 26 · Chapter 26.1 · Why Concurrency Appears in Engineering Code

1. The Machine This Is Measured On

Every number in this module comes from one machine, and its shape explains several of them:

8
4
4

That is hw.ncpu, hw.perflevel0.physicalcpu and hw.perflevel1.physicalcpu on an Apple M3: eight cores, of which four are performance cores and four are efficiency cores. std::thread::hardware_concurrency() reports 8 and does not mention the asymmetry.

That matters for reading §3's table. A workload split evenly across eight threads puts half its work on cores that are deliberately slower, so the ceiling is not 8× — it is somewhere between 4× and 8× depending on how much the efficiency cores contribute. Nothing in this chapter reaches 4×, and that is a property of the hardware rather than of the code.

The general lesson is worth more than the specific numbers: hardware_concurrency() is a count, not a promise of uniform cores, and on any modern laptop or phone some of those cores are slower on purpose.

2. The Three Reasons That Are Real

A parallel regression driver. You have four hundred simulations, each independent, each taking minutes. Running eight at a time is the clearest legitimate use of concurrency in this field: the work was already independent, the results do not interact, and the only shared state is a results directory. Note that processes are often the better tool here — a crashed simulation takes down one process instead of the driver — which §6 returns to.

A background writer. A model producing diagnostics should not block on the filesystem. Chapter 22.9 §3 measured per-cycle logging at 9 GB over a hundred million cycles, and Chapter 23.1 §7 measured a flush per line costing 17× for identical bytes; moving the writing to a thread lets the computation continue while the I/O drains. The shared state is one queue, which is a problem with a well-known shape.

A long analysis over partitionable data. A sweep over independent inputs, a transformation of a large array, a search that divides. This is §3's first two rows and it is the case where the measurement is encouraging.

What the three have in common is that the independence exists before the threads do. Threading expresses a structure the problem already has.

3. What Threads Actually Buy

Three workloads over the same 64 MB of doubles, differing only in what the threads share. Each figure is the best of three within a run; the table gives the median speedup over five whole-process runs with the range.

Compute-bound and independent — sin(x) * cos(x) per element, each thread on its own slice:

Memory-bound — a plain sum over the same array:

The same loop plus one shared atomic counter — every element does a fetch_add on a single std::atomic<long>:

The third row is the chapter. Adding threads makes it monotonically worse, and at eight threads it is about 7.7× slower than the single-threaded version. Not "fails to scale" — actively slower than not threading at all, with every increment of parallelism making it worse.

The arithmetic in all three loops is comparable. The only difference is that the third one has a single memory location every thread writes to, so the cache line holding it must be transferred between cores on every operation, and the cores spend their time passing it back and forth instead of computing. Chapter 26.4 §5 measures the same effect through a mutex, and the shape is identical.

Two observations about the first two rows, because the honest reading is less flattering than the headline.

Doubling from one to two threads gives almost exactly 2× in both. That is the real win, and it is available on any machine.

Four to eight threads adds very little — 2.76 to 3.73 and 3.05 to 3.90 — which is §1's asymmetry showing up: the second four threads run on efficiency cores. A naive reading of hardware_concurrency() == 8 predicts 8× and gets less than half of it.

4. The Threshold: What a Thread Costs to Start

B. the cost of starting and joining one thread
   std::thread + join (empty body) :    …  ns = … microseconds

Over seven runs the median is 14.4 µs with a range of 11.2–27.3 µs — the range is wide because thread creation goes through the operating system and competes with everything else running.

So a task handed to a thread must be worth tens of microseconds before the thread pays for itself, and if you intend to create a thread per item you need items worth milliseconds. That is the arithmetic behind thread pools: create the threads once, hand them work many times.

For comparison, from Chapter 26.4 §3: a mutex lock and unlock is about 4.6 ns, and an uncontended atomic increment about 1.9 ns. Starting a thread is roughly three thousand times the cost of taking a lock, which is why the decision to create a thread is categorically different from the decision to use a synchronisation primitive.

5. The Reasons That Are Not Good Enough

"It will be faster." Not a hypothesis until there is a number. Chapter 25.8 §3 measured one program reporting three different hot spots under three builds; Chapter 25.10 §3 measured four of five plausible optimisations doing nothing or harm. Threading is a much larger change than any of those, and §3's third row is what it looks like when the guess is wrong.

"The machine has eight cores." Measured here, eight cores gave 3.73× on the most favourable workload, because four of them are efficiency cores. The count is not the speedup.

"It is only a background task." A background thread shares the address space, so it shares every object either thread touches. Chapter 26.3 is what that costs when one of them is a std::vector somebody resizes.

"The work is almost independent." "Almost" is where this module lives. §3's third row is almost independent — one counter — and it is 7.7× slower.

"We can add synchronisation later." Chapter 26.3 §5's measurement is the answer: a data race is undefined behaviour, and the version without synchronisation produced the correct answer in every run because the optimiser had collapsed the loop. A program that looks right without synchronisation has told you nothing.

"Threads make it more responsive." Sometimes true and worth separating from throughput. A UI or a server that must not block benefits from concurrency even with no speedup at all, because the goal is latency rather than work per second. That is a real reason; it is just not the same reason as §3's first two rows, and conflating them leads to threading a batch tool that nobody is waiting on.

6. Processes, and When They Are the Better Answer

For the parallel regression driver in §2, separate processes are usually better than threads, and the reasons are worth naming because the choice is often made by habit.

Isolation. A simulation that segfaults takes down its own process. The same code in a thread takes down the driver and the other seven simulations with it.

No shared mutable state by construction. The entire subject of this module does not arise. There is no race to have, because there is no shared address space.

The work is already a program. A simulator is an executable; running eight of them is a shell loop or a job scheduler, not a C++ design problem.

What processes cost is communication — you exchange data through files, pipes or sockets rather than by sharing memory — and startup, which is milliseconds rather than §4's microseconds. For a task measured in minutes, both costs are irrelevant.

The rule that follows: threads for sharing, processes for isolation. If the parallel units need to exchange a lot of data cheaply, threads. If they are independent jobs, processes — and Chapter 25.7 §7's observation applies, that a leaking process is cleaned up when it exits while a leaking thread accumulates for the life of the driver.

7. What the Rest of This Module Covers

The frozen blurb for Module 26 is deliberate about its scope, and it is worth knowing what you will and will not have at the end:

What is not here: lock-free data structures, memory-order micro-optimisation beyond Chapter 26.8's model, coroutines, std::async and futures beyond a mention, and parallel algorithms. Those are each a subject, and the frozen blurb's judgement is that an engineer who can use a mutex and a condition variable correctly is better served than one who has been shown a lock-free queue.

8. The Table

9. What to Actually Do

• Ask whether the independence already exists. If it does, threading expresses it; if not, threading creates shared mutable state.
• Use the three legitimate reasons — a parallel job driver, a background writer, a partitionable analysis — and be suspicious of any other.
• Measure before threading, exactly as Chapter 25.10 §1 requires for any optimisation, and remember threading is harder to revert than the changes that discipline was written for.
• Expect 2× from two threads and much less than 8× from eight. Measured: 3.73× and 3.90× on an 8-core machine with 4 efficiency cores.
• Treat any shared mutable location as a scaling problem, not just a correctness one. One counter took a 3.9× speedup to 0.13×.
• Do not create a thread per item unless each item is worth far more than ~14 µs. Use a pool.
• Prefer processes for independent jobs, for isolation and because the whole of this module then does not apply.
• Separate responsiveness from throughput. Both are real reasons and they need different designs.
• Do not plan to add synchronisation later. Chapter 26.3 §5 measures why the unsynchronised version looking correct proves nothing.

10. Mistakes Worth Making Once

Threading because the machine has eight cores. 3.73× at best here.

Threading a four-second batch tool. Two seconds saved, a race class acquired.

One shared counter in an otherwise independent loop. 0.13× at eight threads.

A thread per work item. 14 µs of overhead per item.

Assuming hardware_concurrency() cores are equivalent. Four of these are efficiency cores.

Threads where processes would do. One crash takes down the driver.

"We will add the locks later." The unsynchronised version will look correct — Chapter 26.3 §5.

Conflating responsiveness with throughput. Different goals, different designs.

11. Interview Relevance

The frozen curriculum marks this chapter Medium, which is right — it is the framing chapter rather than a technical one — and it still comes up as "when would you use threads?", usually expecting a list.

The answer that lands gives the list and then the discriminator: the three real cases are a parallel job driver, a background writer and a partitionable analysis, and what they share is that the independence exists before the threads do. Threading expresses structure; it does not create it.

The measurement to quote is the contrast, because it is memorable and it makes the point no list can: three workloads with comparable arithmetic, two scaling to 3.73× and 3.90× on eight cores, and the third — the same loop plus one shared atomic counter — getting monotonically worse to 0.13×, about 7.7 times slower than not threading at all.

If the follow-up is about cost, that starting a thread is ~14 µs against ~4.6 ns for a mutex lock explains thread pools in one sentence. And if it is about judgement, the callout's framing is the strongest thing to say: the benefit is throughput, which is measurable and reversible, and the cost is a class of defect that testing cannot find — so the decision has to come from the structure of the problem rather than from a benchmark.

12. What You Should Be Able to Say Now

• That this machine is an Apple M3 with 8 cores, 4 performance and 4 efficiency, and that hardware_concurrency() reports 8 without mentioning it.
• The three legitimate reasons, and that what unites them is pre-existing independence.
• The measured scaling: compute-bound 1.98× / 2.76× / 3.73×, memory-bound 1.97× / 3.05× / 3.90× at 2, 4 and 8 threads.
• That one shared atomic counter gives 0.57× / 0.23× / 0.13× — monotonically worse, about 7.7× slower than single-threaded at 8 threads.
• That 1 → 2 threads is the reliable win and 4 → 8 adds little on this hardware.
• That starting and joining a thread costs a median 14.4 µs [11.2–27.3], against ~4.6 ns for a mutex lock — a factor of about 3,000, which is the argument for a pool.
• Why the decision is not a performance decision: the benefit is measurable and reversible, the cost is a defect class that testing cannot find.
• When processes are better — isolation, already-separate jobs — and the rule threads for sharing, processes for isolation.
• What this module deliberately does not cover.

13. Exercises

Exercise 1 — your machine. Report hardware_concurrency() and, if your platform exposes it, the performance/efficiency core split. Predict your ceiling for an independent workload before measuring it.

Exercise 2 — the three workloads. Build §3's three loops and reproduce the speedup tables at 1, 2, 4 and 8 threads, as medians over five runs with ranges.

Exercise 3 — the one shared word. Take the compute-bound loop and add a single shared fetch_add per element. Confirm the direction reverses. Then give each thread its own counter and combine at the end, and confirm the scaling returns.

Exercise 4 — the threshold. Measure thread creation on your machine, then find the task size at which creating a thread per task breaks even against doing them serially.

Exercise 5 — processes. Run eight instances of a small program with a shell loop and compare the wall-clock time against eight threads doing the same work. Then make one of them crash and observe the difference.

14. Summary

• This machine is an Apple M3: 8 cores, 4 performance and 4 efficiency. hardware_concurrency() reports 8 and says nothing about the asymmetry, which is why nothing here reaches 4×.
• The three legitimate reasons are a parallel job driver, a background writer, and a partitionable analysis — and what unites them is that the independence exists before the threads do.
• Measured on 64 MB of doubles, medians of five runs: compute-bound 1.98× / 2.76× / 3.73× and memory-bound 1.97× / 3.05× / 3.90× at 2, 4 and 8 threads. The reliable win is 1 → 2; 4 → 8 adds little.
• The same loop plus one shared atomic counter: 0.57× / 0.23× / 0.13× — monotonically worse with every thread, about 7.7× slower than single-threaded. One word of shared mutable state reversed the sign of the result.
• Starting and joining a thread costs a median 14.4 µs [11.2–27.3], against ~4.6 ns for a mutex lock — roughly 3,000×, which is why thread pools exist and why a thread per item is wrong.
• The decision is not a performance decision. The benefit is throughput, measurable and reversible; the cost is a class of defect testing cannot find — races that give the right answer, that vanish under optimisation, that nearly disappear when you add a printf. So decide from the structure of the problem.
• Threads for sharing, processes for isolation. For independent jobs, processes remove this entire module from the problem.

Chapter 26.2 starts the mechanics, with the one detail about std::thread that terminates your process if you get it wrong.