Modeling Protocol Data: Fields, Packing and Byte Order

Part II · Module 27 · Chapter 27.9 · Modeling Protocol Data: Fields, Packing and Byte Order

1. The Specification and the Temptation

Here is a header of the kind every protocol has. Seven bytes, five fields:

Four plus four plus sixteen plus twenty-four plus eight is fifty-six bits — seven bytes on the wire, exactly.

And C++ has a construct that looks like the table:

struct Natural {
    std::uint8_t  dstSrc;
    std::uint16_t len;
    std::uint32_t seq;
    std::uint8_t  flags;
};

One member per field, in order, and the whole thing can be memcpy'd to a buffer. It is the obvious thing to write, it is what most people write, and §2 measures what it actually produces.

2. What the Compiler Actually Built

A. what the compiler chose
   wire format is 7 bytes
   sizeof(Natural)   = 12   alignof = 4
   sizeof(Bitfields) =  8   alignof = 4
   sizeof(Packed)    =  8   alignof = 1
   offsetof len/seq/flags in Natural = 2/4/8

Not one of the three is seven bytes.

The obvious struct is twelve, which is five bytes of padding on a seven-byte format — nearly 60% overhead, and worse, the fields are not where the wire puts them. offsetof says len is at 2, seq at 4 and flags at 8; the wire puts them at 1, 3 and 6. A memcpy of this struct onto a buffer produces a frame no receiver will parse.

And the packed version is still eight, which is the detail that deserves attention because it refutes the usual fix. #pragma pack(1) removed every byte of padding — alignof dropped from 4 to 1 — and the result is 1 + 2 + 4 + 1 = 8. The overshoot is not padding; it is that seq is 24 bits on the wire and there is no 24-bit integer type, so the member is std::uint32_t and carries a byte the wire does not have. No packing directive can fix a field width the language does not provide.

That is the general result, and it is why the rest of the chapter is about writing the layout out by hand:

3. The Bitfield Measurement

Bitfields look like the right tool for the 4-bit fields, and the specification says dst occupies the high nibble of byte 0. Declared in that order:

struct Bitfields {
    std::uint8_t  dst   : 4;      // the spec's HIGH nibble
    std::uint8_t  src   : 4;
    // ...
};

Set dst = 0xA and src = 0x5 and look at the first byte:

B. where the bitfields actually landed
   dst=0xA src=0x5 -> first byte = 0x5A  (dst in the LOW nibble)

The first byte is 0x5A, so dst landed in the low nibble and src in the high one — the reverse of the declaration's apparent meaning and the reverse of what the wire requires, which is 0xA5.

This is not a compiler bug. The order in which bitfields are allocated within a storage unit is implementation-defined, and this implementation allocates the first declared member to the least significant bits. That is a perfectly reasonable choice; it is simply not a choice you can rely on, and it is not the same across compilers or ABIs.

The consequences are worse than "get the order right once".

It is silent. 0x5A is a valid byte. Nothing warns, nothing faults, and the frame is wrong in a way that only a receiver or a careful hex dump reveals.

It is not portable. Fixing it by reversing the declaration makes the code correct here and wrong on an implementation that allocates the other way, which is the worst outcome — a working program whose correctness depends on an undocumented assumption.

And the standard permits still more freedom than this one measurement shows: whether a bitfield may straddle a storage-unit boundary, and what the underlying type of a bitfield means for allocation, are both implementation-defined too.

So the rule is blunt: bitfields are for your program's internal convenience and never for a wire format.

4. Byte Order

The third gap, and the most widely known:

C. byte order of a 16-bit field
   native  endianness   : little
   0x0102 in memory     : 02 01
   wire wants (big end) : 01 02

This machine is little-endian, so the len value 0x0102 sits in memory as 02 01 and the wire wants 01 02. Any route that copies the object representation — memcpy, a reinterpret_cast, a union — produces the bytes reversed.

std::endian from C++20 lets a program detect this, which is useful for an assertion and is not a solution: code that is correct only on little-endian hosts is code with a latent portability defect, and the fix costs nothing.

And std::byteswap in C++23 is the tidy primitive for the conversion, but the better habit is not to need it, because a serialiser that writes bytes one at a time has no byte order at all — it writes the byte the specification names, and the host's convention never enters.

5. The Only Reliable Technique

Write the layout by hand. One statement per wire byte, with explicit shifts and masks:

std::uint8_t wire[7];
wire[0] = (std::uint8_t)((dst << 4) | (src & 0x0F));   // dst HIGH, as specified
wire[1] = (std::uint8_t)(len >> 8);                     // big endian, by construction
wire[2] = (std::uint8_t)(len & 0xFF);
wire[3] = (std::uint8_t)(seq >> 16);                    // 24 bits, three bytes
wire[4] = (std::uint8_t)(seq >> 8);
wire[5] = (std::uint8_t)(seq);
wire[6] = flags;

D. the explicit serialiser -- no struct layout involved
   7 bytes: A5 01 02 0A BB CC 80

Exactly seven bytes, and every one of them checkable against the specification by eye. dst = 0xA and src = 0x5 give 0xA5; len = 0x0102 gives 01 02; seq = 0x0ABBCC gives 0A BB CC; flags = 0x80. I verified all seven field by field before publishing the transcript.

Five properties of this code are worth naming, because they are what the struct versions lack.

It cannot be wrong about padding, because there is no struct in the output path. It cannot be wrong about bitfield order, because the shift is written down. It cannot be wrong about endianness, because each byte is placed individually. It handles the 24-bit field naturally, because three statements do not care that there is no 24-bit type. And it is reviewable line by line against the specification, which is the property that matters most on a protocol with sixty fields.

The cost is verbosity, and it is worth being honest that on a large header this is a hundred lines of shifting. The mitigation is generation rather than cleverness — a table of field descriptors and a loop, or code generated from a machine-readable specification, which is Chapter 27.7 §6's argument applied to frames.

6. Decoding, and the Fields That Need Checking

The decoder is the mirror image, and it carries one extra responsibility: the wire may contain values the specification forbids.

Header decode(const std::uint8_t (&w)[7])
{
    Header h{};
    h.dst   = (std::uint8_t)(w[0] >> 4);
    h.src   = (std::uint8_t)(w[0] & 0x0F);
    h.len   = (std::uint16_t)((w[1] << 8) | w[2]);
    h.seq   = ((std::uint32_t)w[3] << 16) | ((std::uint32_t)w[4] << 8) | w[5];
    h.flags = w[6];
    return h;
}

Three things the decoder must do that the encoder need not.

Validate. A reserved bit set, a length longer than the frame, a seq above its 24-bit range — in verification these are findings rather than errors to be smoothed over. Chapter 27.2 §4's constructor is the right place if the decoded object is a transaction type, and the decoder should not quietly mask a value into range, because masking destroys the evidence that the DUT emitted something illegal.

Refuse a short buffer. A decoder taking (const std::uint8_t*, std::size_t) must check the size before reading seven bytes. The version above takes a reference to an array of exactly seven, which makes the check the compiler's job — a small use of the type system that removes a whole class of bug.

And never mutate its input. const throughout, so a decoder cannot normalise the captured bytes and destroy what the DUT actually sent.

7. Testing a Serialiser

Four tests, and the first is worth more than the rest together.

Round-trip. decode(encode(h)) == h for random headers. This catches almost every shift, mask and offset error in both directions at once, and it needs no knowledge of the correct bytes — which means it can be written before the first byte has been checked against the specification. With a defaulted operator== on Header, it is three lines.

Known vectors. Bytes from the specification's own example frames, or captured from a conforming implementation. Round-tripping is not enough on its own — an encoder and decoder that are both wrong in the same way round-trip perfectly, which is Chapter 27.4's shared-mistake failure wearing different clothes. The known vector is what pins the pair to the specification.

Boundaries. len at 0 and 0xFFFF, seq at 0 and 0xFFFFFF, every field at its maximum, and the field that is one bit wider than its type suggests. Chapter 21.2's width == 32 bug is the cautionary case.

And a hex dump in the failure message. When a frame comparison fails, the useful output is both byte sequences side by side with the differing offsets marked — Chapter 27.6 §4's per-bit histogram is the stream-level version of the same instinct.

8. The Table

9. What to Actually Do

• Never put a struct on the wire. No memcpy, no reinterpret_cast, no union between a struct and a buffer.
• Write encode and decode by hand, one statement per wire byte, with explicit shifts and masks.
• Keep two representations: an in-memory type chosen for your program, and the byte sequence, with the conversion in one place.
• Treat the wire format as the contract and the struct as an implementation detail. When they disagree, the struct changes.
• Never use bitfields for a wire layout. Allocation order is implementation-defined, and it measured as the reverse of the declaration here.
• Never rely on host byte order. Write each byte individually and the question disappears.
• Remember that #pragma pack cannot fix a field width the language lacks — a 24-bit field needs three statements, not a directive.
• Validate in the decoder and do not mask illegal values into range; in verification an illegal value is a finding.
• Take a fixed-size array reference, or check the length, so a short buffer cannot be read past.
• Round-trip test with random headers, and pin the pair with known vectors from the specification — round-tripping alone passes two matched errors.
• Print both byte sequences with differing offsets marked when a frame comparison fails.

10. Mistakes Worth Making Once

memcpying the obvious struct onto the wire. Twelve bytes for a seven-byte header, with the fields in the wrong places.

Reaching for #pragma pack(1) as the fix. Eight bytes, because a 24-bit field has no type.

Bitfields for the nibbles. 0x5A where the specification requires 0xA5, silently.

Reversing the bitfield declaration to fix it. Now correct here and wrong elsewhere, with nothing recording the assumption.

A reinterpret_cast<std::uint32_t*>(wire + 3). Undefined behaviour and misaligned.

Assuming the host's byte order. 0x0102 goes out as 02 01.

Masking an out-of-range decoded field into range. You deleted the evidence that the DUT sent something illegal.

A decoder taking a bare pointer with no length. It reads past a short frame.

Round-trip tests only. An encoder and decoder wrong in the same way pass every one.

Testing only middle values. The bug is at the field's maximum.

A failure message with no hex dump. You know the frames differ and not where.

11. Interview Relevance

The frozen curriculum marks this Critical and Expert. The question is often how do you represent a packet header in C++?, and the answer most people give — a packed struct — is the one this chapter measures as wrong three different ways.

Lead with the measurement, because it is concrete and surprising. For a seven-byte header, the obvious struct is 12 bytes with fields at the wrong offsets, a bitfield version is 8, and #pragma pack(1) is still 8 — because seq is 24 bits and there is no 24-bit integer type, so no packing directive can help. That last point is the one that usually ends the "just use pragma pack" discussion.

Then the bitfield result, which is the detail that marks real experience: with dst declared first, dst = 0xA and src = 0x5 produced a first byte of 0x5A, so the first declared field landed in the low nibble where the wire needs it high. That is implementation-defined allocation order, not a bug, and the important consequence is that reversing the declaration gives you code that is correct on one compiler and wrong on another with nothing recording the assumption.

Then the design conclusion, which is the actual answer to the question: keep two representations — an in-memory type for your program and the byte sequence for the wire — with hand-written encode/decode doing one statement per byte, so padding, bitfield order and endianness cannot enter. The wire is the contract; the struct is an implementation detail.

And the testing point is a good closer: round-trip tests catch nearly every shift and mask error, but they cannot catch an encoder and decoder that are wrong in the same way — so the pair must also be pinned to known vectors from the specification, which is the same independence argument as Chapter 27.4 in a smaller frame.

12. What You Should Be Able to Say Now

• That the wire format is 7 bytes and that sizeof gave 12, 8 and 8 for three struct formulations — none of them 7.
• That the obvious struct also puts the fields at the wrong offsets (2, 4 and 8 against the wire's 1, 3 and 6).
• That #pragma pack(1) cannot reach 7 bytes because a 24-bit field has no 24-bit type — the overshoot is not padding.
• That with dst declared first, the measured first byte was 0x5A, so the first bitfield landed in the low nibble while the wire needs 0xA5.
• That bitfield allocation order is implementation-defined, so reversing the declaration produces code whose correctness depends on an unrecorded assumption.
• That the host is little-endian and stores 0x0102 as 02 01, so any object-representation copy reverses it.
• That the hand-written serialiser produced exactly A5 01 02 0A BB CC 80, checkable field by field.
• Why it works: no struct in the output path, so padding, bitfield order and endianness cannot enter, and a 24-bit field is just three statements.
• That the design is two representations with conversion in between, and that the wire has authority while the struct is an implementation detail.
• That reinterpret_cast or memcpy between a struct and a buffer is the structural tell, and carries aliasing and misalignment defects of its own.
• That the decoder must validate without masking, refuse a short buffer, and never mutate its input.
• That round-tripping is necessary and not sufficient — two matched errors round-trip perfectly, so known vectors from the specification are required too.

13. Exercises

Exercise 1 — the three sizes. Declare the seven-byte header three ways and print sizeof, alignof and every offsetof. Confirm none is 7.

Exercise 2 — why packing cannot win. Work out from the field widths alone why the packed struct must be 8 bytes, before compiling it.

Exercise 3 — the nibble. Set dst = 0xA and src = 0x5 in a bitfield struct, memcpy one byte out, and print it. Then say what the wire required.

Exercise 4 — the byte order. Print the object representation of a 16-bit field and compare it with the specified wire order.

Exercise 5 — the serialiser. Write encode by hand and confirm it produces the seven bytes, checking each against the specification.

Exercise 6 — the round trip, and its limit. Write decode and a random round-trip test. Then introduce the same shift error into both and confirm the round-trip test still passes.

Exercise 7 — the known vector. Pin the pair with a frame from the specification, and confirm it catches the error Exercise 6's round trip missed.

Exercise 8 — the illegal value. Decode a frame whose seq exceeds 24 bits and make the decoder report it rather than masking it.

14. Summary

• A wire format is specified by someone else and a struct's layout is chosen by your compiler, so the two agree only by coincidence.
• Measured for a 7-byte header: sizeof was 12, 8 and 8 for the obvious struct, a bitfield version and #pragma pack(1). None was 7, and the obvious struct also places len, seq and flags at offsets 2, 4 and 8 where the wire puts them at 1, 3 and 6.
• Packing cannot reach 7 because seq is 24 bits and there is no 24-bit integer type. The overshoot is a missing type, not padding.
• The bitfield version put its first declared field in the low nibble — first byte 0x5A where the wire requires 0xA5 — because allocation order is implementation-defined. Reversing the declaration makes it correct here and wrong elsewhere.
• The host is little-endian, storing 0x0102 as 02 01 against the wire's 01 02, so any memcpy or cast of the object representation reverses it.
• The hand-written serialiser produced exactly A5 01 02 0A BB CC 80 — one statement per byte, no struct in the output path, every byte checkable against the specification.
• Keep two representations: an in-memory type for the program's convenience and the byte sequence for the contract, with encode/decode as the only bridge. The wire has authority; the struct is an implementation detail.
• reinterpret_cast or memcpy between a struct and a buffer is the tell that the two have been conflated, and it adds aliasing undefined behaviour and possible misalignment on top of the layout problem.
• The decoder validates without masking, refuses a short buffer by taking a fixed-size array reference, and never mutates its input — an illegal field value is a finding, and masking destroys the evidence.
• Round-trip tests are necessary and insufficient: an encoder and decoder wrong in the same way pass every one, so the pair must also be pinned to known vectors from the specification.

Chapter 27.10 steps back from modelling to the question the whole module has been circling: how to decide which half of a problem belongs in C++ at all.