Shared State and Data Races

Part I · Module 26 · Chapter 26.3 · Shared State and Data Races

1. The Definition, in Four Parts

A data race occurs when:

1. two or more threads access the same memory location,
2. at least one access is a write,
3. the accesses are not ordered by any synchronisation, and
4. the location is not an atomic object.

All four are necessary. Remove any one and there is no race:

Two clauses are worth expanding because they are where the subtlety lives.

"Memory location" is defined by the standard and is finer-grained than "object". Distinct scalar members of a struct are distinct locations, so two threads writing s.a and s.b is not a race. Adjacent bit-fields in the same sequence are one location, so two threads writing two different bit-fields of the same struct is a race — which is a genuine trap in register-modelling code.

"Not ordered" means there is no happens-before relationship. A join() creates one, which is why Chapter 26.2 §8's "write a result, then join, then read it" is correct. A printf does not, which is why Chapter 26.9's debugging advice is more subtle than it looks.

2. It Is Undefined Behaviour, Not a Wrong Answer

The frozen purpose insists on this and it is the whole chapter. The standard says a program with a data race has undefined behaviour — Chapter 25.1 §1's strongest category, where no requirement is imposed on the program at all.

So the available outcomes are not "the counter is a bit low". They include:

• the counter is low, which is what people expect;
• the counter is exactly right, which §3 measures;
• a value that was never stored by anything, because a read and a write overlapped;
• the compiler having hoisted the read out of a loop, so a flag change is never observed at all;
• a branch eliminated because the compiler proved the condition invariant on the assumption nothing else writes it.

The last two are the ones people do not anticipate, and they follow from exactly the same licence Chapter 25.1 §3 measured: x + 1 < x compiled to mov w0, #0; ret because signed overflow is undefined. A compiler optimising single-threaded code may assume no other thread is writing — that assumption is what makes almost all optimisation legal — and a racy program is one where the assumption is false.

3. The Measurement

Four threads, 200,000 increments each, expecting 800,000. Three versions of one counter:

static long            plain = 0;
static volatile long   vol   = 0;
static std::atomic<long> atom{0};

void bumpPlain(int n) { for (int i=0;i<n;++i) ++plain; }
void bumpVol(int n)   { for (int i=0;i<n;++i) ++vol; }
void bumpAtom(int n)  { for (int i=0;i<n;++i) atom.fetch_add(1, std::memory_order_relaxed); }

A. four threads, 200000 increments each, expecting 800000
   plain long                   got  800000 of  800000  
   volatile long                got  …     of  800000  <-- LOST UPDATES
   atomic<long> relaxed         got  800000 of  800000

The plain long — the one with the data race and no protection whatsoever — produced the correct answer. Every run. The volatile long lost roughly 60% of its updates, with values across eleven runs between 249,186 and 354,556 — a median of 326,849, so between 56% and 69% of the increments vanished depending on nothing but timing.

If you were testing these three implementations to decide which was correct, the measurement would tell you that the unprotected one works and the volatile one is broken. Both have the identical defect, and the atomic one is the only correct program of the three.

4. Why, Exactly

The assembly settles it. Both loops, at -O2:

	ldr	x10, [x9, _plain@PAGEOFF]
	add	x8, x10, x8
	add	x8, x8, #1
	str	x8, [x9, _plain@PAGEOFF]

That is the whole of bumpPlain's work: one load, one add of the entire count, one store. The compiler recognised that for (i < n) ++plain adds n to plain, and collapsed the loop. Each thread therefore performs a single read-modify-write, and four threads colliding on one instruction pair is unlikely enough that it never happened in any run.

	ldr	x9, [x8, _vol@PAGEOFF]
	add	x9, x9, #1
	str	x9, [x8, _vol@PAGEOFF]
	subs	w0, w0, #1
	b.ne	LBB1_2

And that is bumpVol: a load, an add, a store, inside a loop — the subs/b.ne is the loop still being there. volatile requires every access in the abstract machine to be a real access, so the collapse is forbidden and each thread performs 200,000 separate read-modify-write pairs. Four threads doing that on one location collide constantly, and the lost updates are the result.

So volatile did not make the program safer. It removed the optimisation that was accidentally hiding the race. The defect was identical in both; volatile changed only how often the window was open.

5. What volatile Is and Is Not

This is worth stating flatly because the belief is widespread and the keyword's name invites it.

volatile guarantees that each access in the source corresponds to an access in the generated code, and that accesses are not reordered with respect to other volatile accesses. It was designed for memory-mapped hardware registers, where reading twice is meaningful and the compiler must not cache the value — which is exactly the engineering context this curriculum is written for, and the one place volatile is right.

volatile does not provide:

• atomicity. ++vol is still a load, an add and a store, and another thread can interleave. §3 measured it.
• ordering with respect to non-volatile accesses, so it does not establish happens-before and cannot publish data.
• any protection from the processor's reordering. Chapter 26.8 §3 measures a store and a load being reordered by the hardware, which volatile does nothing about.

So for an MMIO register, volatile is correct and necessary. For sharing data between threads it is neither, and std::atomic is the type that provides both the atomicity and the ordering. In C++ the two concerns were deliberately separated, and volatile std::atomic<T> is the spelling for a register that is also shared.

A small note on the keyword's remaining C++ trajectory: ++vol on a volatile scalar is deprecated in C++20, which is the committee discouraging exactly the usage in §3. The compiler says so:

race.cpp:16:54: warning: increment of object of volatile-qualified type 'volatile long' is deprecated [-Wdeprecated-volatile]

6. A Negative Result: Tearing

A torn read — seeing half of one write and half of another — is permitted for non-atomic objects and is the usual illustration of why a race is worse than a lost update. I could not observe it here, and that is worth reporting rather than asserting.

One thread writing two fields of a struct, another reading the pair and comparing them:

B. a two-field struct written by one thread and read by another
   2000000 reads, 0 of them saw a != b  (0.00%)

  16-byte struct: 3000000 reads, 0 saw a != b (0.00%)

Zero torn reads in two million, and zero in three million with a 16-byte struct, repeated three times. On this core, an aligned 8-byte or 16-byte copy is performed by instructions that are single-copy-atomic in practice, so the reader never caught a half-updated pair.

The honest statement is therefore narrow: tearing is permitted by the standard, and I did not observe it for aligned 8- and 16-byte copies on an Apple M3. It is observable on other architectures, for unaligned data, and for larger objects — Chapter 26.7 §5 measures that std::atomic of a 32-byte type is not lock-free here, which is the same boundary from the other side.

This is Chapter 25.1 §7's lesson again: the undefined behaviour is real, the observable symptom is not guaranteed to appear, and the absence of a symptom is not evidence of anything.

7. What the Tool Says

The sanitizer does not care which optimisation happened to hide the window. Here it is on the sharpest possible case — a four-thread increment of a plain long which, at -O2, produced the correct total:

counter = 800000 of 800000  <-- CORRECT ANSWER
WARNING: ThreadSanitizer: data race (pid=…)
SUMMARY: ThreadSanitizer: data race tsanrace.cpp:5 in bump(int)

The first line is the un-instrumented run at -O2; the next two are the same program built with -fsanitize=thread. The program was right and the tool said it was broken, and the tool was correct.

TSan names the file, the line and the function, and it reports the race whether or not the run produced a wrong answer — because it reasons about the ordering of the accesses rather than the outcome. Chapter 25.6 §3's measurement is the sharpest demonstration of that: the same racy program was wrong in 5 of 10 runs at -O0 and 0 of 10 at -O1 and -O2, and TSan reported the race in both.

That is why TSan is not optional for threaded code. It is the only instrument in this module whose finding does not depend on the race having gone wrong during the run you watched.

Chapter 25.6 §6's costs apply: TSan is roughly an order of magnitude slower (measured 16.4×, range wide), cannot be combined with ASan, and exits 134 on a finding.

8. Removing the Race

Four mechanisms, in the order you should reach for them.

Do not share it. Chapter 26.1 §3 measured one shared counter taking a 3.9× speedup to 0.13×. Give each thread its own accumulator and combine at the end:

std::vector<long> partial(threads, 0);
// each thread writes only partial[i] -- distinct memory locations, no race
long total = std::accumulate(partial.begin(), partial.end(), 0L);

This removes clause (1) of §1's definition and is both correct and fast. It is the right answer far more often than people reach for it.

Make it immutable. Concurrent reads of data nobody writes are always safe — clause (2) removed. A const reference to data prepared before the threads start needs no synchronisation at all.

Order the accesses. A mutex is the general tool, and Chapter 26.4 is the chapter. A join() orders everything the thread did before everything after the join, which is why Chapter 26.2 §8's pattern is correct.

Make it atomic. For a single scalar — a counter, a flag, a pointer — std::atomic<T> removes clause (4) and costs 1.88 ns uncontended, measured in Chapter 26.7 §3. It is the right tool for exactly the case where the shared thing is one word; it does not help for an invariant spanning two variables, which needs a mutex.

The last distinction is the one people get wrong. Two atomics are not an atomic pair: a reader can see the first updated and the second not, which is §6's tearing at the level of your own invariant rather than the hardware's.

9. Where Races Hide in Engineering Code

Six shapes, all seen in real models:

A shared results container. Two threads push_back to one std::vector. This is not a lost update; it is a reallocation under another thread's feet, which is Chapter 25.4's invalidated iterator with two threads. One vector per thread, or a mutex.

A lazily initialised singleton. Two threads both find the pointer null and both construct. A function-local static is the fix — it is thread-safe initialisation guaranteed by the standard, and Chapter 24.1 §7 is the mechanism.

A "done" flag. bool done = false; written by a worker and read by a poller. Chapter 26.7 §4 measures why this needs to be atomic.

A cached value. if (!cached) { cached = compute(); } from two threads. Both compute, and one result is leaked or both are stored.

A counter of anything — transactions, errors, retries. The commonest instance, and the one §3 measured.

Logging. Two threads writing to one std::ostream is a race on the stream's internal state. Chapter 22.9's design, with one writer thread and a queue, exists for this.

10. The Table

11. What to Actually Do

• Learn the four-part definition and check each clause when you look at shared state.
• Prefer not sharing. Per-thread accumulators combined at the end remove the race and the contention together.
• Use std::atomic<T> for a single shared scalar, and a mutex for an invariant spanning more than one.
• Never use volatile for thread communication. Measured: it increased the failure rate by orders of magnitude and provided no safety.
• Do use volatile for memory-mapped registers, which is what it is for, and volatile std::atomic<T> when a register is also shared.
• Run TSan, because it is the only instrument whose result does not depend on the window having opened during your run.
• Do not infer correctness from repeated passing runs. The window is a property of the generated code.
• Treat a shared container as a race on its internals, not just on its elements.
• Use a function-local static for lazy initialisation; it is guaranteed thread-safe.
• Remember join() establishes ordering — that is why reading a result after a join is defined.

12. Mistakes Worth Making Once

Adding volatile to make shared state safe. It lost 60% of the updates it was supposed to protect.

Concluding from volatile breaking it that you introduced a bug. You revealed one.

Concluding from the plain version working that no synchronisation was needed. The window was one instruction wide.

A thousand passing runs as evidence. Chapter 25.6 §3: wrong 5 of 10 times at -O0, 0 of 10 at -O1.

Two atomics for one invariant. A reader sees one updated and not the other.

push_back from two threads. A reallocation under another thread's iterator.

Two threads writing adjacent bit-fields. One memory location, and a race.

Two threads writing to one std::ostream. A race on the stream's state, not just interleaved text.

Expecting to see tearing. Permitted, and not observable here for aligned 8- and 16-byte copies.

13. Interview Relevance

The frozen curriculum marks this chapter Critical, and the question is usually "what is a data race?". The four-part definition is the answer, and stating all four — same location, at least one write, no ordering, not atomic — is better than the usual one-line paraphrase because it immediately gives you four ways to remove one.

The essential follow-up is that a race is undefined behaviour, not a wrong answer, and the measurement here is the best possible support: four threads incrementing a plain shared long produced exactly the right total on every run, because the optimiser had collapsed the loop into a single add, while the same loop on a volatile long lost about 60% of its updates. Same defect, different window.

That also sets up the strongest thing you can say about volatile: it provides neither atomicity nor ordering, and measured here it made the symptom far more likely while leaving the defect untouched — so a developer who adds it and sees the program break has drawn the wrong conclusion from a real observation. It is for memory-mapped registers; std::atomic is for threads.

If testing comes up, the point from the callout is the one that lands: a passing concurrency test is evidence about the race window in that binary, and the window is a property of the generated code — so it moves with the optimisation level, the compiler version, and unrelated edits.

14. What You Should Be Able to Say Now

• The four parts of a data race, and the four corresponding ways to remove one.
• That distinct struct members are distinct memory locations but adjacent bit-fields are not.
• That a race is undefined behaviour, so the outcomes include a hoisted read, an eliminated branch, and the correct answer.
• That four threads on an unprotected long gave 800000 of 800000 every run, and the same loop on a volatile long gave a median 326,849, anywhere from 249,186 to 354,556.
• Why: the plain loop was collapsed to one load-add-store per thread (verified in the assembly), so the window was one instruction pair; volatile forbids the collapse and restores 200,000 windows.
• That volatile gives neither atomicity nor ordering, that it is for memory-mapped registers, and that ++vol is deprecated in C++20.
• That tearing was not observed here for aligned 8- and 16-byte copies — 0 in 2M and 0 in 3M reads — and that this is not evidence of safety.
• That TSan reports the race whether or not the run went wrong, because it reasons about ordering — and that it named race.cpp:4 in bump(int).
• That a passing concurrency test is evidence about a window, not a program.

15. Exercises

Exercise 1 — the three counters. Reproduce §3's table. Then run the plain version a thousand times and record how often it is wrong.

Exercise 2 — the mechanism. Dump the assembly for the plain and volatile loops and find the collapsed add. Then compile the plain version at -O0 and re-run the experiment; explain the new result.

Exercise 3 — widen the window. Add a clobber() barrier inside the plain loop so the collapse is forbidden without volatile, and confirm it now loses updates too. This isolates the optimisation from the keyword.

Exercise 4 — tearing. Try to observe a torn read: unaligned data, a 32-byte struct, and three fields written in sequence. Report what you can and cannot reproduce.

Exercise 5 — TSan. Run all three versions under TSan. Confirm it reports the two racy ones including the one that gives the right answer, and says nothing about the atomic one.

Exercise 6 — remove it four ways. Fix the counter by not sharing, by a mutex, by an atomic, and by making the data immutable. Measure all four and decide which you would ship.

16. Summary

• A data race has four parts: two threads, the same memory location, at least one write, no ordering, and a non-atomic object. Removing any one removes the race.
• It is undefined behaviour, not a wrong answer, so the permitted outcomes include a hoisted read, a deleted branch, and the right result.
• Measured, four threads and 200,000 increments each: the plain long produced the correct 800000 every run; the volatile long lost about 60% of its updates (median 326,849 of 800,000, range 249,186–354,556 over eleven runs); the atomic was correct.
• The assembly explains it. The plain loop was collapsed to a single load-add-store per thread, so its race window was one instruction pair. volatile forbids that collapse and restores 200,000 real read-modify-write pairs. volatile removed the optimisation that was accidentally hiding the race and provided no atomicity.
• volatile is for memory-mapped registers — no atomicity, no ordering for other data, no protection from hardware reordering — and ++vol is deprecated in C++20. std::atomic is the thread tool.
• Tearing was not observed here: 0 torn reads in 2,000,000 with an 8-byte struct and 0 in 3,000,000 with a 16-byte one. It is permitted by the standard; the absence of the symptom is not evidence of safety.
• TSan reports the race whether or not the run went wrong — it reasons about ordering, not outcome — which is why it caught the counter that was wrong 5 of 10 times at -O0 and 0 of 10 at -O1.
• So a passing concurrency test is evidence about the race window in that binary, and the window is a property of the generated code: it moves with the optimisation level, the compiler version, and unrelated edits.

Chapter 26.4 is the primitive that removes clause three of the definition, and the RAII guard that makes releasing it unforgettable.