Structure Layout as a Hardware Data Format

Part II · Module 14 · Chapter 14.5 · Structure Layout as a Hardware Data Format

1. The Temptation

A protocol specification says a header is eight bytes:

offset 0   version   u8
offset 1   flags     u8
offset 2   length    u16, big-endian
offset 4   address   u32, big-endian

C has a construct that appears to describe exactly this, and the thought that follows is immediate — declare the structure, point it at the received bytes, read the fields by name:

struct hdr {
    uint8_t  version;
    uint8_t  flags;
    uint16_t length;
    uint32_t address;
};

const struct hdr *h = (const struct hdr *)rx_buffer;   /* and we are done? */
if (h->version != 1) return -1;

It is clean, it is readable, and it is the single most common source of silent protocol bugs in C. This chapter is about the three separate questions that have to be answered yes before that cast is correct, and about the fact that Chapter 10.5 already answered the first one.

2. The Measurement: Declaration Order Decides Everything

Here are the same four fields, declared twice in orders that both look reasonable.

#include <stdio.h>
#include <stdint.h>
#include <stddef.h>

struct hdr_natural {        /* declared in the specification's order */
    uint8_t  version;
    uint8_t  flags;
    uint16_t length;
    uint32_t address;
};

struct hdr_misordered {     /* the same fields, grouped differently */
    uint8_t  version;
    uint16_t length;
    uint8_t  flags;
    uint32_t address;
};

int main(void) {
    printf("spec says 8 bytes on the wire\n\n");
    printf("hdr_natural    sizeof %zu  alignof %zu\n",
           sizeof(struct hdr_natural), _Alignof(struct hdr_natural));
    printf("   version %zu  flags %zu  length %zu  address %zu\n",
        offsetof(struct hdr_natural, version), offsetof(struct hdr_natural, flags),
        offsetof(struct hdr_natural, length),  offsetof(struct hdr_natural, address));

    printf("\nhdr_misordered sizeof %zu  alignof %zu\n",
           sizeof(struct hdr_misordered), _Alignof(struct hdr_misordered));
    printf("   version %zu  length %zu  flags %zu  address %zu\n",
        offsetof(struct hdr_misordered, version), offsetof(struct hdr_misordered, length),
        offsetof(struct hdr_misordered, flags),   offsetof(struct hdr_misordered, address));
    return 0;
}

spec says 8 bytes on the wire

hdr_natural    sizeof 8  alignof 4
   version 0  flags 1  length 2  address 4

hdr_misordered sizeof 12  alignof 4
   version 0  length 2  flags 4  address 8

hdr_natural matches the wire exactly: eight bytes, fields at 0, 1, 2 and 4.

hdr_misordered is twelve bytes, and only version is where the specification put it. length needs 2-byte alignment so a padding byte appears after version; flags lands at 4; address needs 4-byte alignment so three more padding bytes appear, putting it at 8. The structure is 50% larger than the format it is supposed to describe, and every field but the first reads the wrong bytes.

Nothing warned. Both declarations are perfectly ordinary C, and Chapter 10.5 established that the compiler is required to insert that padding — the alignment of each member has to be respected, so the only way to honour uint32_t address is to place it at a multiple of 4.

The middle row matching is worth dwelling on, because it is why the technique is tempting. It matched because the specification was designed by someone who put each field at a naturally aligned offset — which competent protocol designers do, precisely so that implementations in C and other languages can lay the record out without padding. When the format is well designed and the declaration order follows it, the overlay really does work.

The trouble is that nothing tells you which case you are in. hdr_natural and hdr_misordered compile identically quietly, and the only way to know which you have is to measure.

3. Packing: What It Fixes

Both clang and GCC provide an attribute that suppresses the padding.

struct hdr_packed {
    uint8_t  version;
    uint16_t length;
    uint8_t  flags;
    uint32_t address;
} __attribute__((packed));

sizeof 8  offsets 0 1 3 4

Eight bytes, and the fields sit immediately after one another with no gaps. Packing did exactly what it claims.

Two things should be said immediately.

It is a compiler extension, not ISO C. There is no standard way to specify a structure's layout; __attribute__((packed)) is a GNU extension that clang implements, and MSVC spells the same idea #pragma pack. Chapter 1.3 established the category and the discipline that goes with it: an extension is usable, and it must be recognised as one. What is worth noting here is that the compiler gives you no help identifying it — the declaration above is accepted silently even under -Wpedantic and -Weverything, so the flag whose job is to report uses of extensions does not report this one.

It does not make the structure portable. Packed layout is defined by each compiler, not by the language. The attribute is spelled differently on MSVC, and nothing requires two compilers to agree on how a packed structure containing unusual members is laid out. A packed structure in a header shared between toolchains is a compatibility assumption, not a guarantee.

4. Packing: What It Does Not Fix, and What It Breaks

It does not fix byte order. This is the error that survives longest, because packing fixes the visible problem and leaves an invisible one. hdr_packed puts length at offset 2 — correct — and then reads those two bytes in this machine's order. The specification said big-endian. Chapter 14.3 measured that memcpy of an object reproduces the host's order and that only shifts on the value produce the specified one. Packing changes where the bytes are, never how they are ordered within a field.

So a packed overlay on a big-endian protocol is wrong on a little-endian host in exactly the fields that are more than one byte wide, which is usually the interesting ones.

It introduces a hazard the unpacked version did not have. A member of a packed structure may be at an offset that does not satisfy its own alignment requirement — address at offset 4 is fine here, but in general a packed uint32_t can land at offset 1. The compiler handles reading it: it knows the offset, knows it may be underaligned, and emits whatever access sequence is safe. What it cannot handle is you taking its address and passing it elsewhere.

#include <stdint.h>
struct p { uint8_t a; uint32_t b; } __attribute__((packed));

uint32_t *addr_of(struct p *s) { return &s->b; }   /* underaligned pointer */
uint32_t  readval(struct p *s) { return s->b; }    /* fine: compiler knows */

$ cc -std=c17 -Wall -Wextra -c packaddr.c
packaddr.c:3:42: warning: taking address of packed member 'b' of class or
                 structure 'p' may result in an unaligned pointer value
                 [-Waddress-of-packed-member]
    3 | uint32_t *addr_of(struct p *s) { return &s->b; }
1 warning generated.

One warning, on the function that takes the address. None on the function that reads the value. That is the distinction exactly: the access is safe, the address is not. The returned uint32_t * claims 4-byte alignment that its target does not have, and Chapter 14.4 covered what happens next — the compiler is entitled to believe the pointer type.

Usefully, -Waddress-of-packed-member is on by default, so unlike -Wcast-align you already have it.

5. The Alternative: Read and Write Explicitly

The technique that has none of these problems appeared already in Chapter 14.3, and it is worth seeing applied to a whole header.

#include <stdint.h>
#include <string.h>
#include <stdbool.h>

/* The in-memory form. The compiler lays this out however it likes,
   because nothing outside the program ever sees it.                */
struct hdr {
    uint8_t  version;
    uint8_t  flags;
    uint16_t length;
    uint32_t address;
};

#define HDR_WIRE_SIZE 8u

static uint16_t get_be16(const unsigned char *p) {
    return (uint16_t)((uint32_t)p[0] << 8 | (uint32_t)p[1]);
}
static uint32_t get_be32(const unsigned char *p) {
    return (uint32_t)p[0] << 24 | (uint32_t)p[1] << 16
         | (uint32_t)p[2] <<  8 | (uint32_t)p[3];
}

bool hdr_parse(struct hdr *out, const unsigned char *buf, size_t len) {
    if (len < HDR_WIRE_SIZE) return false;      /* the length is checked once */
    out->version = buf[0];
    out->flags   = buf[1];
    out->length  = get_be16(buf + 2);
    out->address = get_be32(buf + 4);
    return true;
}

Count what this is immune to. The structure's padding is irrelevant, because the structure is never overlaid. The host's byte order is irrelevant, because every multi-byte field is assembled with shifts. Alignment is irrelevant, because only unsigned char is dereferenced. The offsets 0, 1, 2, 4 appear once each, in one function, next to the specification they came from. And the length is checked before any byte is read, which is Chapter 13.11's habit and Chapter 14.7's subject.

The cost is the function. The benefit is that all four of §4's failure modes are structurally absent rather than currently-not-happening — and Chapter 14.4 measured that the generated code for this style is the same single instruction the overlay would have produced.

6. The Misconception: "Packed Makes It a Wire Format"

"Add __attribute__((packed)) and the struct is the wire format."

It is the most efficient possible summary of a partial truth, which is why it spreads. §3 and §4 measured the whole of it:

Packing turns one yes into a yes, turns one into a no, and leaves the byte-order column untouched — and the byte-order column is the one that silently corrupts data rather than failing visibly.

The belief survives because of a specific and very common environment: a little-endian host talking to a little-endian device over an internal bus, in a project built with one compiler. In that environment packing really does make the overlay work, and it works for years. It stops working when the project acquires a big-endian peer, a second toolchain, or a protocol specified in network byte order — all three of which arrive without warning and without a compiler diagnostic.

7. The Judgement: When May You Overlay?

The honest answer is not never. It is a decision with three outcomes, and the first is reached more often than the purists admit.

Overlay is fine when all of these hold. The struct never leaves the build — it is a layout you invented for your own program's internal use, a shared-memory region between two processes built from the same source with the same compiler, or a scratch record in a file only you read. Here padding and byte order are consistent on both sides by construction, and overlaying is simply the natural way to use a structure. No extension, no packing, no parsing function. Most structures in most programs are in this category, including every structure in Part I.

Overlay with packing is a defensible platform decision when the format is externally specified and the byte order already matches the host and you are building with one known toolchain for one known target — the common case for a device driver talking to a peripheral on the same board. State it as a decision rather than letting it happen: pin the layout with assertions so a future change is a build failure rather than a field failure.

_Static_assert(sizeof(struct hdr_packed) == 8,            "header must be 8 bytes");
_Static_assert(offsetof(struct hdr_packed, address) == 4, "address must be at offset 4");

Chapter 13.3 §7 named this pattern — rely on implementation-defined behaviour deliberately: write it down, check it with _Static_assert, confine it, revisit it on a new target. Two lines, no runtime cost, and the next person to reorder a member gets a compile error instead of a corrupted packet.

Parse explicitly when the format crosses a boundary you do not control: a network protocol, a file read by other tools, a bus with a peer built by someone else, a testbench in another language, or any specification that fixes a byte order different from the host's. This is §5, it is portable, and it costs one function per record type.

8. What You Should Be Able to Say Now

1. Declaration order decides the layout. Measured: the same four fields give 8 bytes at offsets 0/1/2/4 in one order and 12 bytes at 0/2/4/8 in another. No diagnostic for either.
2. The compiler is required to pad so each member meets its alignment requirement — Chapter 10.5's rule, with a document on the other side of it now.
3. A well-designed format often matches an unpacked struct, because the designer aligned the fields on purpose. That is why the technique is tempting, and you cannot tell which case you have without measuring.
4. Packing fixes offsets only. Measured sizeof 8, offsets 0 1 3 4. It is a compiler extension, accepted silently even under -Weverything, and it is not portable across toolchains.
5. Packing does not fix byte order — the one failure that corrupts data silently rather than visibly.
6. Packing creates a new hazard: the member's address. Reading s->b is fine; &s->b yields an underaligned pointer and warns via -Waddress-of-packed-member, which is on by default.
7. Explicit parsing is immune to all four — padding, byte order, alignment, and toolchain — and costs one function per record.
8. The judgement has three outcomes: overlay freely inside one build; overlay with packing plus _Static_assert for a pinned platform; parse explicitly across any boundary you do not control.

9. Exercises

1. Run §2's program. Then add a uint64_t field at the end of hdr_natural and predict sizeof before re-running. Explain any tail padding from Chapter 10.5's rule.
2. Write the four-field header in every declaration order that gives exactly 8 bytes unpacked. How many are there, and what do they have in common?
3. Add the two _Static_assert lines from §7 to hdr_packed. Then reorder two members and record the build failure. Compare it with what would have happened without the assertions.
4. Write addr_of and readval from §4 and confirm which one warns. Then write a third function that passes s->b by value to a helper and say why it does not warn.
5. Take §5's hdr_parse and write the matching hdr_serialise. Verify a round trip, then verify the bytes against the specification's offsets by hand.
6. Decide, for three structures in code you have written, which of §7's three outcomes applies. At least one should land in the first category; say what makes it safe there.
7. A specification defines a 4-bit version and a 4-bit type sharing byte 0. Write the extraction with masks and shifts. Then write the bit-field version and say what you would need to know about the compiler to trust it.

10. Summary

• Declaration order decides layout. Measured: 8 bytes at 0/1/2/4 versus 12 bytes at 0/2/4/8 from the same four fields. No diagnostic either way.
• Padding is required, not optional — each member must meet its alignment (10.5).
• A well-designed format often matches an unpacked struct by the designer's intent; you still cannot tell without measuring.
• __attribute__((packed)) fixes offsets (measured sizeof 8, offsets 0 1 3 4), is a compiler extension accepted silently even under -Weverything, and is not portable.
• It does not fix byte order — the silent-corruption failure.
• It breaks member addresses: &s->b is underaligned and warns (-Waddress-of-packed-member, on by default); s->b is fine.
• Explicit parse is immune to padding, byte order, alignment and toolchain, for one function per record.
• Three outcomes: overlay inside one build · packed overlay plus _Static_assert on a pinned platform · explicit parse across any uncontrolled boundary.

Every structure in this chapter was laid over bytes the program already had — a buffer that arrived from somewhere, with an address the program was given. The next question is what that address actually is when the bytes are not a buffer at all, but a block of registers a hardware designer placed at a number written in a document.