Binary Files and Byte-Level Layout

Part I · Module 23 · Chapter 23.6 · Binary Files and Byte-Level Layout

1. Why Binary at All

Text is the right default. It is greppable, diffable, readable by a human at three in the morning, and portable without discussion. Chapter 23.2 and Chapter 23.8 are both about making text do this job well, and most engineering data should stay text.

Binary earns its place in three situations:

• Volume. Chapter 22.9 §3 measured per-cycle logging at 9 GB in text for a hundred million cycles. A 16-byte binary record for the same information is a fraction of that, and the difference decides whether the data can be kept at all.
• Exactness. Chapter 23.2 §5 measured that a double at the default precision does not round-trip; writing the eight bytes removes the question entirely.
• A format someone else defined. A waveform file, a memory image, a vendor's trace format. You do not get a choice.

What you give up is every property that made text safe, and the rest of the chapter is the inventory.

2. A Struct Does Not Occupy the Sum of Its Members

Four fields that an engineer would naturally declare in this order:

struct Sample {
    std::uint8_t  unit;
    std::uint32_t addr;
    std::uint16_t resp;
    double        us;
};

  members          : uint8 + uint32 + uint16 + double = 15 bytes of data
  sizeof(Sample)   : 24   alignof 8
  sizeof(Packed)   : 16   alignof 8   (same fields, large-to-small)
  sizeof(Tight)    : 15   alignof 1   (#pragma pack(1))
  Sample offsets   : unit 0  addr 4  resp 8  us 16
  Packed offsets   : us 0  addr 8  resp 12  unit 14
  -> reordering saved 8 bytes per record and wrote the SAME data

Fifteen bytes of data in twenty-four bytes of struct. The offsets show where it went: unit at 0 then three bytes of padding so addr can start at 4; resp at 8 then six bytes of padding so the double can start at 16. Nine bytes of padding, 37% of the record.

Reordering the same four fields largest-first gives sixteen bytes — us at 0, addr at 8, resp at 12, unit at 14, with one byte of tail padding to keep the eight-byte alignment:

struct Packed {
    double        us;
    std::uint32_t addr;
    std::uint16_t resp;
    std::uint8_t  unit;
};

Eight bytes per record saved, for identical data and no change in how you use it. At ten million records that is 80 MB. The rule is to declare fields in decreasing size order, which is a free optimisation whenever a struct is stored in bulk, and Chapter 10.4 covers the alignment rules it follows from.

#pragma pack(1) gets you to fifteen, and it is the wrong tool here. It is non-standard, it differs between compilers, and it makes every field potentially misaligned — which on some architectures is a fault and on this one is merely slower. If you need an exact byte layout, §6 is the honest way to get it.

3. The Padding Is Written, and It May Be Garbage

The padding is not conceptual. It is bytes in the record, and write sends them to the file. Here is the same record built twice — once value-initialised, once over simulated stack garbage — with identical field values:

Sample a{};                       // zero-initialised
a.unit = 1; a.addr = 0x4000A010; a.resp = 2; a.us = 1.25;

Sample b;                         // NOT initialised
std::memset(&b, 0xCD, sizeof b);  // simulate stack garbage
b.unit = 1; b.addr = 0x4000A010; b.resp = 2; b.us = 1.25;

  zero-init then set:   01000000 10A00040 02000000 00000000 00000000 0000F43F 
  garbage then set:     01CDCDCD 10A00040 0200CDCD CDCDCDCD 00000000 0000F43F 
  same field values, different BYTES -> a byte-compare of two records
  holding equal data reports them different: memcmp = -1

Every field is equal and the bytes differ. You can read the padding directly in the second line: 01CDCDCD is unit plus three bytes that were never assigned, and 0200CDCD CDCDCDCD is resp plus six more.

Three real consequences follow, and the third is the one that bites.

Two records with the same data are not byte-equal, so memcmp is not a comparison of your data and a byte-wise diff of two files reports differences that do not exist.

A hash of the record is a hash of the padding too, so the same data hashes differently run to run. A checksum over a file of such records is not reproducible.

The written padding is whatever was on the stack, which can include data from unrelated parts of the program. For a file that leaves the machine that is an information leak, and it is a well-known class of real vulnerability.

The fix is one character: always value-initialise a record you are going to write or hash. Sample s{}; zeroes the padding as well as the members. The default-initialised Sample s; does not.

4. The Round Trip That Works

With that in hand, the mechanical part is straightforward:

std::vector<Sample> out = build();

{
    std::ofstream o(path, std::ios::binary);
    o.write(reinterpret_cast<const char*>(out.data()),
            static_cast<std::streamsize>(out.size() * sizeof(Sample)));
}

std::vector<Sample> back(out.size());
std::ifstream i(path, std::ios::binary);
i.read(reinterpret_cast<char*>(back.data()),
       static_cast<std::streamsize>(back.size() * sizeof(Sample)));

  wrote 4 records of 24 bytes -> file is 96 bytes
  gcount() = 96, read ok
  byte-identical after the round trip: YES

Four details in that code are load-bearing.

std::ios::binary — §7 measures that it changes nothing on this platform and is still required.

read and write, not operator<< — the insertion operator formats. write moves bytes.

gcount() is the check, not the stream's bool. A short read succeeds for the bytes it got. If the file is truncated you get fewer bytes and a stream that may look fine; gcount() is the number that tells you, and comparing it against what you asked for is the only way to detect a partial record.

The reinterpret_cast is where the type system stops helping, which is §6's subject.

5. The Wrong Declaration, Which Reports Nothing

Now read the same 96-byte file into a struct with the same four fields, declared slightly differently — Tight, which is #pragma pack(1) and therefore 15 bytes:

  read 15 bytes into Tight (size 15, the writer used 24)
  first record should be unit=0 addr=0x40000000 resp=0 us=0.5
  Tight sees        unit=0 addr=0x00000000 resp=0 us=3.16202e-322
  -> no error, no exception, no short read. Just wrong numbers.

This is the section the chapter exists for. The read succeeded. gcount() returned the 15 bytes requested, because the file has plenty. The stream is good. And us is 3.16202e-322, a denormal produced by interpreting a few bytes of addr and padding as a double.

Notice what is not available to detect this. There is no length field, no magic number, no version, no type tag — the bytes are just bytes, and any 15 of them are a valid Tight. The writer and the reader disagreed about the record layout and nothing in the file could contradict either of them.

And notice how easily the disagreement arises in practice. Nobody writes Tight deliberately to read a Sample file. What happens is:

• a field is added to the struct, and an old file is read with the new declaration;
• a field's type changes from std::uint32_t to std::uint64_t;
• the fields are reordered for §2's eight-byte saving, and an old file is read;
• the reader is compiled with different packing or a different alignment setting;
• the reader is a Python struct.unpack with a format string written from the header file by hand.

Every one of those produces numbers rather than an error. A text format would have produced a parse failure, which is Chapter 23.7's whole argument about failing loudly.

6. The Type System Can Refuse the Unsafe Write

reinterpret_cast<const char*> will happily accept a type that must not be written this way:

struct HasString { std::uint32_t addr; std::string name; };

  is_trivially_copyable<Sample>    = 1
  is_trivially_copyable<Tight>     = 1
  is_trivially_copyable<HasString> = 0  <-- blitting this writes a POINTER
  sizeof(HasString) = 32, of which the std::string is 24 bytes of
  pointer-and-size, not characters.

Writing HasString as bytes writes a pointer to a file. Twenty-four of its thirty-two bytes are the std::string's internals — a pointer, a size, a capacity, or the small-string buffer — and none of them are the characters in any portable sense. Read it back in another process and the pointer refers to nothing; read it back in the same process after the original string is gone and it refers to freed memory. Chapter 20.9 §3 measured libc++'s small-object buffers, which is why a short string sometimes appears to survive this and a long one never does — the worst possible failure mode, because it works in testing.

The guard is one line, and it belongs in the writing function:

template <typename T>
void writeRecords(std::ostream& o, const std::vector<T>& v)
{
    static_assert(std::is_trivially_copyable_v<T>,
                  "a record written as raw bytes must be trivially copyable");
    o.write(reinterpret_cast<const char*>(v.data()),
            static_cast<std::streamsize>(v.size() * sizeof(T)));
}

Now HasString is a compile error at the call site rather than a pointer in a file. This is Chapter 18.10 §6's "check what you claim to support" applied to serialisation, and Chapter 21.8 §7 measured why a concept gives a better diagnostic than a static_assert if you have C++20:

template <typename T>
concept RawRecord = std::is_trivially_copyable_v<T>;

template <RawRecord T>
void writeRecords(std::ostream& o, const std::vector<T>& v);

Note what is_trivially_copyable does not promise. It says the bytes can be copied within one program; it says nothing about padding being initialised (§3), about the layout matching another compiler's (§5), or about byte order (§7). It is a necessary condition, not a sufficient one — which is exactly why it makes a good gate and a bad reassurance.

7. Byte Order, and What Is Actually on the Disk

  0x4000A010 in memory: 10A00040 
  this machine is LITTLE-endian
  after byteswap:       4000A010 
  a big-endian reader of our file would see 0x10A00040 where we wrote 0x4000A010
  -> a binary format needs a DECLARED byte order, converted on both sides
  wrote big-endian, read back and converted: 0x4000A010 (round-trip OK)

The value 0x4000A010 is stored as 10 A0 00 40 — least significant byte first. That is what goes in the file, and a big-endian reader interpreting those four bytes gets 0x10A00040.

The fix is to declare a byte order in the format and convert at both boundaries. C++20 and C++23 give you the tools:

#include <bit>

// C++20: which order is this machine?
constexpr bool little = (std::endian::native == std::endian::little);

// C++23: swap
const std::uint32_t onDisk = little ? std::byteswap(host) : host;

Big-endian — network byte order — is the conventional choice for an interchange format, and little-endian is defensible if every consumer is x86 and ARM and you have written that down. What is not defensible is not choosing, because then the format's byte order is a property of whichever machine happened to produce the file.

Two cautions. std::byteswap is C++23; before that you write it yourself or use the platform's htonl family. And byte order is per-scalar: swapping a struct's bytes wholesale is wrong, because you must swap each field within itself, which is another reason §8's field-by-field serialisation is the honest approach.

Floating point has a further wrinkle. IEEE 754 is near-universal in practice but is not guaranteed by the standard, and double has its own byte order on exotic platforms. For most engineering work, writing the eight bytes with a declared endianness is fine; if the format must be bulletproof, write the value as a decimal string at max_digits10 precision per Chapter 23.2 §5, or as a scaled integer.

8. A Format That Travels

Putting §5's insight, §6's gate and §7's byte order together:

struct Header {
    char          magic[4];     // "VMSM"
    std::uint16_t version;      // 1
    std::uint16_t recordBytes;  // sizeof the record as the writer sees it
    std::uint32_t recordCount;
};

void writeField(std::ostream& o, std::uint32_t v)
{
    const std::uint32_t be = (std::endian::native == std::endian::little)
                           ? std::byteswap(v) : v;
    o.write(reinterpret_cast<const char*>(&be), sizeof be);
}

bool checkHeader(const Header& h)
{
    return std::memcmp(h.magic, "VMSM", 4) == 0
        && h.version == 1
        && h.recordBytes == RECORD_BYTES;   // our own, declared constant
}

Four properties that the raw version did not have:

• A file of the wrong type is rejected by the magic number instead of producing numbers.
• A file of the wrong version is rejected by the version, which is §5's five drift scenarios turned into one comparison.
• A layout mismatch is rejected by recordBytes, and note that it is compared against a declared constant rather than sizeof(Sample) — if you compare it against sizeof, a reader whose struct changed will happily agree with itself.
• A truncated file is detectable from recordCount without depending on gcount().

The field-by-field write costs a line per field and buys the one thing the whole-struct write cannot have: the file's layout is specified by your code rather than by your compiler's alignment rules. It also makes the padding question disappear, since you never write padding at all.

For anything beyond a handful of record types this is where a serialisation library earns its place. The reason to have written it by hand once is that you then know what the library is doing for you, and which of its options correspond to §3, §5 and §7.

9. ios::binary, Which Does Nothing Here

  5 bytes incl. CR, LF, 0x1A and NUL
  written WITHOUT ios::binary -> 5 bytes on disk
  written WITH    ios::binary -> 5 bytes on disk
  identical contents: YES (no translation here)
  -> on Windows the first file would grow and 0x1A would end a text read.

Five bytes including a carriage return, a line feed, 0x1A and a NUL, written both ways, identical. On this platform the flag is a no-op, and it is still required.

On Windows, text mode translates \n to \r\n on output, so the first file would be six bytes while the program wrote five; and on input 0x1A historically terminates a text-mode read, so a binary file containing that byte is silently truncated at it. Both failures are silent and neither can be reproduced on macOS or Linux.

Chapter 23.4 §7 measures the same equality from the text side. The conclusion is identical and worth stating as a rule: pass std::ios::binary whenever the bytes matter, on every platform. It costs nothing where it does nothing, and it is the difference between working and silently corrupting where it does something.

10. The Table

11. What to Actually Do

• Prefer text unless volume, exactness, or somebody else's format forces binary. Chapter 23.2 and Chapter 23.8 make text do the job well.
• Declare fields in decreasing size order. Measured: eight bytes per record for free.
• Value-initialise every record you write or hash — Sample s{} — so the padding is zero rather than stack contents.
• Never memcmp or hash a padded struct and call it a data comparison. Compare fields, or serialise first.
• Gate every raw write on static_assert(std::is_trivially_copyable_v<T>), or a concept. It is the one mistake the compiler can catch for you.
• Check gcount() against the number of bytes you asked for. A short read succeeds.
• Put a magic number, a version, a declared record size and a record count in the header, and compare the record size against a declared constant rather than sizeof.
• Declare a byte order and convert on both sides. Big-endian for interchange; little-endian is fine if you write it down.
• Serialise field by field for any format that must outlive your build configuration. It also removes the padding question entirely.
• Always pass std::ios::binary. A no-op here, and the difference between working and truncating on Windows.

12. Mistakes Worth Making Once

write(&s, sizeof s) on a default-initialised struct. The padding is whatever was on the stack, and it goes in the file.

memcmp on two records with equal fields. Measured -1.

Hashing a padded struct. The hash changes run to run for identical data.

Blitting a struct with a std::string. You wrote a pointer. Short strings may appear to work, which is worse.

Reading with a struct that gained a field. No error; us = 3.16202e-322.

Trusting a successful read without gcount(). A truncated file gives a short read and a usable stream.

Comparing the header's recordBytes against sizeof(Record). A reader whose struct changed agrees with itself.

#pragma pack(1) as the fix for layout. Non-standard, compiler-specific, and it makes every field potentially misaligned.

Swapping a whole struct's bytes for endianness. Byte order is per-scalar.

Omitting ios::binary because it makes no difference. It makes no difference here.

13. Interview Relevance

The frozen curriculum marks this chapter Advanced and High, and the question is usually "what is wrong with write(reinterpret_cast<char*>(&s), sizeof s)?". A complete answer names four separate problems and does not stop at the first: the padding is written and may be uninitialised, so the file contains stack contents and equal data gives unequal bytes; the layout is the compiler's choice, so a reader with a different declaration gets numbers rather than an error; the byte order is the writer's machine's; and nothing stops you from doing it to a type containing a pointer.

The strong follow-up is "so how would you design the format?", and the answer is the self-describing header — magic, version, record size, count — plus field-by-field serialisation in a declared byte order. If you can explain why the checking has to be data rather than code — that no byte sequence is an invalid record, so the reader has nothing to reject — you have given the structural reason instead of a checklist.

The is_trivially_copyable gate is the detail that shows practice rather than theory, along with knowing that it is necessary and not sufficient.

14. What You Should Be Able to Say Now

• That 15 bytes of fields occupy 24 in declaration order and 16 reordered largest-first, and why that is a free optimisation for bulk storage.
• That the padding is written to the file and is uninitialised unless you value-initialise, so two records with identical fields can differ byte-wise — measured memcmp = -1.
• That a byte-wise compare or hash of a padded struct is not a comparison of your data.
• That reading with a different declaration produces wrong numbers and no error — us = 3.16202e-322 — and the five realistic ways that disagreement arises.
• Why no error is possible: every byte sequence is a valid record, so the reader has nothing to reject; the checking must be added as data.
• That gcount() is the short-read check, because a short read succeeds.
• That is_trivially_copyable is the gate the compiler can enforce, that HasString fails it, and that it is necessary but not sufficient.
• That this machine is little-endian, what 0x4000A010 looks like on disk, and that a format needs a declared byte order converted per scalar.
• That std::ios::binary is a no-op here and must still be passed.

15. Exercises

Exercise 1 — layout. Reproduce §2 for Sample, Packed and Tight, printing sizeof, alignof and every offsetof. Then add a std::uint64_t to each and predict the new sizes before measuring.

Exercise 2 — the padding. Reproduce §3's two hex dumps. Then write both records to a file, read them back, and compare them field by field and with memcmp; explain the two different answers.

Exercise 3 — the silent corruption. Write a file of Sample. Add a field to the struct, recompile the reader only, and read the old file. Record what you get. Then add §8's header and confirm the same mistake now produces a diagnostic.

Exercise 4 — the gate. Write the static_assert-guarded writeRecords and try to call it with HasString. Read the error. Then convert it to a concept and compare the diagnostics, as Chapter 21.8 §7 did.

Exercise 5 — byte order. Write 0x4000A010 little-endian and big-endian to two files, dump both, and write a reader that uses the header's declared order to produce the correct value from either.

16. Summary

• Binary is for volume, exactness, or a format you did not choose. Text keeps every safety property you are about to give up.
• A struct does not occupy the sum of its members: 15 bytes of data in 24, with 9 bytes of padding. Declaring fields largest-first gives 16 — eight bytes per record for identical data.
• The padding is written to the file and is uninitialised unless you value-initialise. Measured: two records with identical field values produced different bytes and memcmp = -1. So a byte compare is not a data compare, and a hash of the record is not a hash of the data.
• A wrong declaration produces wrong numbers and no error at all — us = 3.16202e-322, with a successful read and a good stream — and the realistic causes are an added field, a changed type, a reordering, different packing, or a hand-written reader in another language.
• No error is possible, structurally: every byte sequence is a valid record, so a reader has nothing to reject. The checking must be added as data — magic number, version, declared record size, record count — not as more careful code.
• static_assert(std::is_trivially_copyable_v<T>) is the one check the compiler can make. HasString fails it, and blitting it writes a pointer — which short-string optimisation may make appear to work.
• Check gcount(), because a short read succeeds.
• Byte order is yours to declare and convert per scalar. 0x4000A010 is 10 A0 00 40 here; a big-endian reader of that sees 0x10A00040.
• Always pass std::ios::binary — measured identical on this platform, and the difference between working and silent truncation on Windows.

Chapter 23.7 returns to text and to the hardest input of all: output that was written for a human to read, by a tool that will change its mind about the format.