Compiler Reordering, Hardware Reordering and Barriers

Part II · Module 17 · Chapter 17.6 · Compiler Reordering, Hardware Reordering and Barriers

1. Two Reorderers

When you write three memory accesses in a row, two independent agents may change the order in which they take effect.

The compiler may emit the instructions in a different order from the source, or merge them, or delete some. Chapter 17.1 measured four such transformations and Chapter 17.5 §3 measured a plain store being deleted and the source's ordering not surviving.

The processor and the memory system may let the instructions take effect in a different order from the one they were issued in — a store sitting in a store buffer while a later one completes, a load satisfied early, a write reaching one device before another that was issued first.

These are separate problems with separate mechanisms, and conflating them is why barrier code is so often wrong. volatile addresses the first one partially and the second one not at all — which is the clause this chapter's purpose ends on, and the thing Chapter 17.2 §5 meant by "a guarantee about the abstract machine only". The abstract machine has no store buffer.

2. The Three Regimes, Measured

The same three stores — ordinary, volatile, ordinary — under three regimes.

extern volatile uint32_t vreg;
extern uint32_t plain;

void none(void) {
    plain = 1u;
    vreg  = 2u;
    plain = 3u;
}

_none:
    str  w9, [x8]         <- vreg = 2, FIRST
    str  w9, [x8]         <- plain = 3
    ret

plain = 1u deleted, plain = 3u on the far side of the volatile store. Chapter 17.5 §3's measurement.

A compiler barrier. An inline assembly statement with no instructions and a "memory" clobber, which tells the compiler that memory may have changed arbitrarily:

void compiler_b(void) {
    plain = 1u;
    __asm__ __volatile__("" ::: "memory");
    vreg  = 2u;
    __asm__ __volatile__("" ::: "memory");
    plain = 3u;
}

_compiler_b:
    str  w9,  [x8]        <- plain = 1   (survives!)
    str  w10, [x9]        <- vreg  = 2
    str  w9,  [x8]        <- plain = 3
    ret

All three stores present, in source order. And the barrier itself emitted no instruction at all — there is nothing in that output corresponding to the two __asm__ statements, because the empty assembly template is empty.

That is the fact worth carrying away from this chapter: the compiler half of the problem costs zero instructions to fix.

A full barrier.

void full_b(void) {
    plain = 1u;
    atomic_thread_fence(memory_order_seq_cst);
    vreg  = 2u;
    atomic_thread_fence(memory_order_seq_cst);
    plain = 3u;
}

_full_b:
    str  w9,  [x8]        <- plain = 1
    dmb  ish                            <- a real instruction
    str  w10, [x9]        <- vreg  = 2
    dmb  ish
    str  w9,  [x8]        <- plain = 3
    ret

Same three stores in order, plus a dmb ish — a data memory barrier — at each fence. __sync_synchronize() produces byte-for-byte identical output.

3. What a Compiler Barrier Is

__asm__ __volatile__("" ::: "memory") has three parts and each matters:

• "" — the assembly template is empty. No instruction is generated.
• __volatile__ — the statement may not be deleted or moved, for the same reason a volatile access may not be.
• ::: "memory" — the clobber list. It tells the compiler that this statement may read or write any memory it can see.

The third is what does the work. Having been told memory may have changed arbitrarily, the compiler cannot carry a cached value across the statement, cannot move a store past it, and cannot delete a store before it on the grounds that a later one overwrites it. All three of Chapter 17.1 §2's assumptions are suspended at that point.

It is a GNU extension — Chapter 1.3's category — supported by GCC and clang. The portable spelling is atomic_signal_fence(memory_order_seq_cst) from <stdatomic.h>, which is specified to do exactly this: order with respect to signal handlers on the same thread, with no hardware barrier.

Use it when the other agent is on the same core and sees memory through the same path — most commonly an interrupt handler. The handler runs on your core; it sees your store buffer's effects in order; what you need is for the compiler not to have rearranged the stores, and that costs nothing.

4. What a Hardware Barrier Is, and the Honest Limit

A dmb exists because the processor and the memory system may complete accesses in an order other than the one in which they were issued. A store may sit in a store buffer while a later store to a different address completes; a load may be satisfied from a cache before an earlier load misses.

This cannot be demonstrated on this machine. Producing an observable reordering requires two agents racing — two cores, or a core and a DMA engine — and single-threaded C here cannot do it. Chapter 16.5 §4 and Chapter 17.5 §4 both declined to fabricate a race trace and this chapter declines for the same reason: an invented transcript would be worth nothing.

What can be stated with evidence:

The barrier instruction is real and is emitted — §2 measured dmb ish appearing at each fence and absent otherwise.

The architecture documents what may move. arm64 is weakly ordered: the architecture permits a wide range of reorderings and provides dmb, dsb and isb to constrain them. x86-64 is strongly ordered and permits far fewer, which is why code written and tested only on x86 often has missing barriers that appear the moment it is ported to ARM.

The memory type matters, and this is why the bug is rare. On arm64 the MMU assigns each region a memory type. Normal memory is freely reorderable and cacheable. Device memory — which is how a sane platform maps peripheral registers — has far stronger ordering rules, including that accesses to Device memory are not reordered with respect to each other.

That last point is the honest explanation of something that would otherwise look like folklore: most memory-mapped register code works without any explicit barriers, because the MMU configuration is doing the work. Two writes to two registers in the same Device region arrive in order because the region's memory type says so, not because volatile said so.

Where it stops being enough is exactly Chapter 17.5 §3's DMA case: a buffer in Normal memory and a start bit in Device memory are two different memory types, and nothing orders the buffer writes against the register write. That is where a barrier is not optional.

5. Which Mechanism for Which Problem

Read the last two rows together: they are why the same team can write a driver that needs no barriers and then a DMA path that fails once a day.

Two practical rules follow.

Write the reason, not the barrier. A bare dmb with no comment is unmaintainable — nobody can tell later whether it is required or cargo-culted, so nobody ever removes it and everybody adds more.

/* The DMA engine reads dma_buffer when it sees START. The buffer is Normal
   memory and DMA_CTRL is Device memory, so nothing orders them. Barrier
   required — see TRM §4.2.                                              */
atomic_thread_fence(memory_order_seq_cst);
DMA_CTRL = DMA_START;

Prefer the platform's own macros where they exist. Operating systems and vendor SDKs provide named barriers — __DMB(), dma_wmb(), mb() — that encode which barrier the platform actually needs. They are better than a raw fence because they carry the platform's knowledge, and they are better than nothing by a wide margin.

6. The Misconception: volatile Orders My Accesses

"My register accesses are volatile, so they happen in the order I wrote them. I do not need barriers."

Two separate errors, and Chapter 17.5 only covered the first.

It does not order plain accesses at all. Chapter 17.5 §3 measured a plain store deleted and the ordering lost across a volatile store. The DMA buffer in §4 is plain stores.

And even between two volatile accesses, it constrains only the compiler. This is the stronger claim and the less known one. Chapter 17.2's rule says accesses to volatile objects are part of the abstract machine's observable behaviour and are not reordered with respect to each other — and the abstract machine is a sequential model with no store buffer, no cache and no interconnect. The guarantee is that the compiler emits them in order. What the processor does with those instructions afterwards is outside the standard's vocabulary entirely.

In practice, two volatile accesses to the same Device-memory region do arrive in order — but §4 established that this is the MMU's memory type doing it, not the qualifier. Change the memory type, or cross between types, and the ordering is gone while the volatile is still there.

The accurate statement:

volatile keeps the compiler from reordering volatile accesses with each other. It orders nothing else, and it does not reach the hardware at all.

Which is exactly what this chapter's purpose says: volatile enforces neither.

7. Module 17 in One Page

The module was short by design and its job was precision. Every result was measured on Apple clang 21 / arm64 at -O2.

Three threads run through all six.

The guarantee is narrower than the reputation, and the narrowness is principled. Everything volatile does follows from one sentence — accesses are side effects, side effects are observable, observable behaviour is preserved. Everything it does not do follows from the same sentence. It is not a safety feature with gaps; it is a precise tool whose shape people misremember.

One sentence separates the guarantees from the gaps. volatile is about whether an access happens, not what happens between two accesses. Atomicity, exclusion and ordering are all questions about between.

Two of the module's findings could not be measured, and both said so. The read-modify-write window and hardware reordering both need a second agent this machine cannot provide. Chapters 16.5, 17.5 and 17.6 each stated the mechanism and declined to fabricate a trace — and §4's callout argues that for hardware ordering specifically, reasoning from the documents is not a fallback but the correct method, because every bench technique makes the symptom disappear.

8. What Comes Next

Modules 14 through 17 have been about one program talking to one device: the bytes it must produce, the bits inside them, the registers they travel through, and the qualifier that keeps the accesses alive. That is the whole of what a C program needs in order to drive hardware.

It is not the whole of what a C program does in a verification environment. Chapter 16.1's model of a peripheral — written in C, driven by test code, implementing a device's documented behaviour — was introduced as a teaching device, and it is also a real artefact: a reference model. The C in a verification flow is usually not a driver at all. It is a model that a SystemVerilog testbench calls into, a scoreboard that checks a design's output, a stimulus generator, a checker.

Module 18, C in SoC Verification, is where that begins, and it changes what the code is for rather than what the language does. Its own description is "the architecture before the syntax" — where C actually appears around a verification environment (18.1), what a reference model is and, as importantly, what it is not (18.2), reusing existing C libraries (18.3), firmware-driven tests (18.4), how C code gets into a simulation at all (18.5), and how to decide whether a given job belongs in SystemVerilog or in C (18.6).

Everything from Part II still applies and none of it is repeated: a reference model's integer widths are the design's widths (14.2), its byte order is the bus's (14.3), its field extraction is the register map's (15.4), and a model of a peripheral is the artefact Chapter 16.1 built in order to make this module's measurements runnable. What changes is that the C is no longer driving hardware — it is standing in for it, or checking it.

9. What You Should Be Able to Say Now

1. Two independent reorderers: the compiler, and the processor plus memory system. Separate problems, separate mechanisms.
2. volatile addresses the first partially and the second not at all — it is a statement about the abstract machine, which has no store buffer.
3. A compiler barrier costs zero instructions. Measured: __asm__ __volatile__("" ::: "memory") restored all three stores to source order and emitted nothing.
4. Its three parts: an empty template, __volatile__ so it cannot move, and the "memory" clobber that does the work. The portable spelling is atomic_signal_fence.
5. A full barrier emits a real instruction. Measured: dmb ish per fence, identical for atomic_thread_fence and __sync_synchronize.
6. Hardware reordering cannot be demonstrated here and no trace is fabricated. The evidence is the barrier instruction, the architecture reference, and the memory map.
7. arm64 is weakly ordered; x86-64 is strongly ordered — which is why x86-tested code acquires barrier bugs on ARM.
8. Device memory has stronger ordering than Normal memory. That is why most MMIO works without barriers, and why crossing between the two — a Normal-memory DMA buffer and a Device-memory start bit — is where it stops.
9. Every bench technique hides this bug: a printf acts as a compiler barrier, a debugger changes timing, -O0 removes the reordering. Reason from the documents.
10. Write the reason beside the barrier, and prefer the platform's named macros.

10. Exercises

1. Reproduce §2's three regimes and confirm all three outputs, including that the compiler barrier emits no instruction.
2. Replace the inline-assembly barrier with atomic_signal_fence(memory_order_seq_cst) and compare the generated code. Say which you would put in portable source.
3. Remove only the "memory" clobber from the inline-assembly barrier and recompile. What happens, and what does that tell you about which part does the work?
4. For each row of §5's table, write the one-sentence justification. Which two rows explain how one team can ship a correct driver and a broken DMA path?
5. Write §5's commented DMA barrier for a device of your choice, including what you would need from the TRM to justify it.
6. Explain why adding a printf to investigate an ordering bug can make it disappear, naming the mechanism.
7. A driver works on x86-64 and fails intermittently on arm64 with no source changes. Give two candidate explanations from this chapter and say what you would read first.

11. Summary

• Two reorderers: the compiler, and the processor and memory system. volatile restrains neither properly.
• A compiler barrier costs nothing — measured, zero instructions, three stores back in source order. __volatile__ plus a "memory" clobber; portably atomic_signal_fence.
• A full barrier emits dmb ish — atomic_thread_fence and __sync_synchronize identical.
• Hardware reordering is not demonstrated here and no trace is invented; the evidence is the instruction, the architecture, and the memory map.
• arm64 weak, x86-64 strong — the usual source of barrier bugs on porting.
• Device memory orders itself; Normal memory does not. That is why MMIO usually works without barriers and why crossing between types is where it fails.
• Every bench technique hides it — printf, debugger, -O0. Reason from the documents (16.8).
• Write the reason beside the barrier; prefer the platform's named macros.