Binary Files: Modes, Bytes and Layout Assumptions

Part I · Module 12 · Chapter 12.5 · Binary Files: Modes, Bytes and Layout Assumptions

1. The Two Functions

    size_t fread (void *ptr, size_t size, size_t count, FILE *stream);
    size_t fwrite(const void *ptr, size_t size, size_t count, FILE *stream);

Both take two size arguments, and the pair is not redundant. fread(buf, 1, n, f) asks for n bytes; fread(recs, sizeof recs[0], n, f) asks for n whole records. The difference matters:

asked for 3 records, fread returned 1   feof=1 ferror=0
asked for 64 bytes,  got 12             (one record is 12 bytes)

The by-items form returns complete items only. A file holding one and a half records returns 1, and the half record is silently left behind — so the by-items form cannot tell you the file was truncated mid-record. The by-bytes form can, because it reports the odd bytes.

Use fread(ptr, 1, n, f) when you need to know exactly how many bytes arrived, and the by-items form only when a partial item is genuinely impossible or genuinely uninteresting.

Both are convention B, so both need the Chapter 12.1 §3 follow-up:

    size_t got = fread(buf, 1, sizeof buf, f);
    if (got < sizeof buf) {
        if (feof(f))   { /* the file ended */ }
        if (ferror(f)) { /* a read error  */ }
    }

2. The Mode Flag

    FILE *f = fopen(path, "rb");        /* r, w, a, r+, w+, a+ each take a b */

Appending b to any mode from Chapter 12.4 §2 opens the stream in binary mode. Everything else about the mode is unchanged — "wb" still truncates at open, "ab" still appends.

What b changes is whether the library is permitted to translate bytes on the way in or out. Measured here, writing five bytes containing a carriage return and two newlines:

wrote 5 bytes each
  text mode "w"    -> file is 5 bytes
  binary mode "wb" -> file is 5 bytes

No difference on this platform. On POSIX systems, text and binary modes are the same thing, and that is why the flag is so often omitted.

On systems that represent a line ending as two characters — Windows most prominently — text mode translates: a \n you write becomes two bytes in the file, and two bytes in the file become one \n when read. For text that is exactly what you want and it is invisible. For binary data it is corruption: any byte that happens to equal \n is expanded on the way out, and any \r\n pair is contracted on the way in.

Always write b for binary data. It costs one character, it does nothing on the platform you are probably using, and it is the difference between a working program and a silently corrupted file on the platform you are not.

That asymmetry is also why the bug is so persistent: a program developed and tested on one platform passes every test, and the corruption appears only when someone builds it somewhere else. This chapter could not demonstrate the translation — it does not happen here — so what is published above is the measurement that it makes no difference on this platform, and the behaviour elsewhere is stated as the contract rather than as something observed.

3. The Obvious Approach, and What It Writes

Writing a structure straight to a file is the thing everyone tries first:

    struct record { char tag; int value; short flags; };

    struct record r = { 'A', 1, 2 };
    fwrite(&r, sizeof r, 1, f);

It works. On this machine the round trip is exact — tag=A value=1 flags=2. Now look at what is actually in the file.

sizeof(struct record) = 12  (members total 7)
offsets: tag=0 value=4 flags=8

the file is 12 bytes:  41 ee ee ee 01 00 00 00 02 00 ee ee

The object was painted with 0xEE before the members were set — Chapter 10.5 §7's technique — so that every byte in the dump is determinate and can honestly be shown. The ee bytes are padding, and there are five of them in a twelve-byte file.

Written field by field instead, the same record is nine bytes:

the file is 9 bytes:  41 00 00 00 01 00 00 00 02

4. The Four Assumptions

fwrite(&r, sizeof r, 1, f) is correct exactly when the program reading the file agrees with the program that wrote it about four separate things. Each has already been measured elsewhere in this curriculum.

1 — Padding. Chapter 10.5 established that a compiler inserts padding according to its own alignment rules, and §3 above shows those bytes going into the file. A different compiler, a different target, or a reordered declaration changes how many there are and where. And their contents are unspecified — Chapter 10.5 §7 measured two structures with equal members and different bytes, which is why a file written this way cannot be meaningfully compared or hashed.

2 — Type sizes. The file contains four bytes of int because int is four bytes here. Chapter 2.9 established that it need not be. A reader built where int is two or eight bytes reads the wrong number of bytes for every field after the first.

3 — Byte order. The value 1 appears as 01 00 00 00 — the least significant byte first. That is a property of this target, it is called byte order or endianness, and Chapter 14.3 is where it is explained. What matters here is only that it is a choice the machine made for you, not a property of the file format.

4 — Member order and the type itself. The layout is whatever the struct declaration happens to say today. Add a member, reorder two, change a short to an int, and every previously written file becomes unreadable — with no version marker and no error, because the bytes still parse into something.

fwrite of a structure produces a file whose format is "whatever this compiler did on this machine today". It is not a format; it is a side effect. It is fine for a file your own program writes and reads back on the same machine in the same run, and fragile in exactly four ways the moment anything else touches it.

Whether a structure may nonetheless be treated as a data format — and the circumstances in which the answer is yes — is a judgement that needs the alignment and byte-order material, and it belongs to Chapter 14.5. This chapter's job is that you can now list what the judgement is about.

5. The Alternative: Say What You Mean

Writing each field explicitly removes all four assumptions, because the code states what the file contains instead of inheriting it.

#include <stdio.h>
#include <stdint.h>

static int put_u32(FILE *f, uint32_t v)
{
    unsigned char b[4] = { (unsigned char)(v >> 24), (unsigned char)(v >> 16),
                           (unsigned char)(v >> 8),  (unsigned char)v };
    return fwrite(b, 1, 4, f) == 4;
}

static int get_u32(FILE *f, uint32_t *out)
{
    unsigned char b[4];
    if (fread(b, 1, 4, f) != 4) { return 0; }
    *out = ((uint32_t)b[0] << 24) | ((uint32_t)b[1] << 16) |
           ((uint32_t)b[2] << 8)  |  (uint32_t)b[3];
    return 1;
}

the file is 9 bytes: 41 00 00 00 01 00 00 00 02
read back: tag=A value=1 flags=2

Four properties, one per assumption:

No padding, because nothing but named fields is ever written.

Fixed widths, because uint32_t is four bytes everywhere it exists — Chapter 14.2 is about choosing these deliberately.

A chosen byte order. put_u32 writes the most significant byte first because it was told to, with shifts and masks that are ordinary arithmetic. The same code produces the same bytes on every machine, which is the whole point.

An explicit layout, so changing the structure does not silently change the file. Changing the format now requires changing put_u32's callers, which is a conversation rather than an accident.

And it is nine bytes against twelve. The explicit version is smaller as well as portable, because the padding was never information.

Two things any real format needs on top, named so you know they exist:

A version number, written first, so a reader can refuse a file it does not understand instead of misreading it.

A length or a count for anything variable, because Chapter 7.7's pointer-plus-length rule does not stop applying when the bytes are in a file.

6. Positioning Within a File

Binary files are often read out of order, which text files rarely are.

    int   fseek(FILE *f, long offset, int whence);   /* SEEK_SET / SEEK_CUR / SEEK_END */
    long  ftell(FILE *f);
    void  rewind(FILE *f);

fseek returns 0 on success and non-zero on failure — convention A, and it does fail, on streams that are not seekable such as a pipe or a terminal. ftell returns the current offset or -1L.

    if (fseek(f, (long)(index * sizeof(struct record)), SEEK_SET) != 0) {
        return -1;                              /* not seekable, or out of range */
    }

Three cautions.

long may be too narrow for a large file. Chapter 2.9's sizes apply, and files larger than two gigabytes are routine. The standard's answer is fgetpos/fsetpos, which use an opaque fpos_t instead of a number.

In text mode, an offset is not a byte count. Only values obtained from ftell are meaningful, which is one more reason to use b when the file is binary.

Switching between reading and writing on an update stream requires a positioning call in between — fseek, fsetpos or rewind — even if the position does not change. This is a real rule in the contract for "r+", "w+" and "a+", and omitting it is undefined behaviour rather than a no-op.

7. A Complete Example

#include <stdio.h>
#include <stdint.h>

#define FORMAT_VERSION 1

struct record { char tag; int value; short flags; };

static int put_u32(FILE *f, uint32_t v);        /* as in §5 */
static int get_u32(FILE *f, uint32_t *out);

static int write_file(const char *path, const struct record *recs, size_t n)
{
    FILE *f = fopen(path, "wb");                /* b, and w truncates */
    if (f == NULL) { return -1; }

    int rc = -1;
    if (fputc(FORMAT_VERSION, f) == EOF)  { goto done; }
    if (!put_u32(f, (uint32_t)n))         { goto done; }

    for (size_t i = 0; i < n; i++) {
        if (fputc((unsigned char)recs[i].tag, f) == EOF) { goto done; }
        if (!put_u32(f, (uint32_t)recs[i].value))        { goto done; }
        if (!put_u32(f, (uint32_t)recs[i].flags))        { goto done; }
    }
    rc = 0;

done:
    if (fclose(f) != 0) { rc = -1; }            /* 12.4 §4: the close is the write */
    return rc;
}

Five decisions, each from a measurement:

"wb" — binary, §2, and w truncates at open, Chapter 12.4 §2.

A version byte first, so a future reader can reject rather than misinterpret.

A count before the records, so the reader never has to guess where the data ends.

Every write checked — fputc returns EOF on failure, and a loop that ignores it writes a short file and reports success.

One cleanup label with a checked fclose — Chapter 9.8 §8's pattern and Chapter 12.4 §4's reason.

The matching reader checks the version first, reads the count, and then reads exactly that many records — refusing a file whose count disagrees with its length, which is the check that turns a corrupt file into an error message instead of a wrong answer.

8. What You Should Be Able to Say Now

1. fread/fwrite take a size and a count; the by-items form returns complete items only and cannot report a partial record.
2. Measured: asking for 3 records returned 1 with feof=1; asking in bytes returned 12.
3. Both are convention B — follow a short count with feof/ferror.
4. b permits no translation. Measured: no difference on this platform; on systems that translate line endings it is the difference between a correct file and a corrupted one.
5. Always write b for binary data.
6. Measured: fwrite of a 12-byte structure put five padding bytes into the file — 41 ee ee ee 01 00 00 00 02 00 ee ee.
7. Padding contents are unspecified, so such a file cannot meaningfully be compared or hashed — Chapter 10.5 §7.
8. The four assumptions: padding, type sizes, byte order (Chapter 14.3), and member order.
9. A struct dump is not a format; whether one may ever be treated as a wire format is Chapter 14.5's judgement.
10. Measured: the explicit field-by-field file was 9 bytes against 12 — smaller and portable.
11. A real format needs a version and a count or length.
12. fseek returns 0 on success, long may be too narrow, and an update stream needs a positioning call between reading and writing.

9. Exercises

Exercise 1 — choose the call. For each, write the fread call and say how you would detect a truncated file.

• read exactly 16 bytes of a header
• read up to 100 records of 24 bytes
• read the whole of a file whose size you do not know

Exercise 2 — count the padding. For each, give sizeof, the total of the members, and the number of padding bytes that fwrite would place in a file.

struct a { char c; double d; };
struct b { double d; char c; };
struct c { int x; int y; };
struct d { char a; short b; char c; };

Exercise 3 — name the assumption. A file written by fwrite(&rec, sizeof rec, 1, f) on one machine is misread on another. For each symptom, say which of §4's four assumptions was violated.

• every field after the first is wrong
• the numbers are byte-reversed
• the file is the right length and two fields are swapped
• two files with identical data compare unequal

Exercise 4 — write the format. Design an explicit byte format for struct { uint16_t id; uint32_t when; char name[8]; }, give its exact size, and say what you would put in the first two bytes of the file and why.

Exercise 5 — the mode flag. Explain why omitting b is harmless on one platform and destructive on another, which byte values are affected, and why the resulting bug is usually found by someone other than the author.

Exercise 6 — detect the truncation. A reader does fread(recs, sizeof recs[0], count, f) and gets back count - 1. List the two possible causes, the calls that distinguish them, and what the by-bytes form would have told you that this does not.

Exercise 7 — version the file. A format is shipped, then gains a field. Describe what happens to old readers with and without a version byte, and what the reader should do when it sees a version it does not know.

Exercise 8 — justify the size. The explicit format in §5 was nine bytes where the struct dump was twelve. Explain where the three bytes went and why the smaller file is also the more portable one.

10. Summary

• fread/fwrite take a size and a count; the by-items form returns complete items only and hides a partial record.
• Measured: 3 requested, 1 returned, feof=1; by bytes, 12.
• b permits no translation — measured as no difference here, and as corruption on platforms that translate. Always write it.
• Measured: a struct dump put five padding bytes in a twelve-byte file — 41 ee ee ee 01 00 00 00 02 00 ee ee.
• Four assumptions: padding · type sizes · byte order (14.3) · member order.
• A struct dump is a side effect, not a format. The wire-format judgement is 14.5's.
• Measured: the explicit format was 9 bytes against 12 — smaller and portable.
• Give a real format a version and a count.
• fseek can fail; long may be too narrow; an update stream needs a positioning call between read and write.

Every program in this module has been handed its input through a stream — stdin in Chapter 12.3, a path spelled out in the source here and in Chapter 12.4. Real programs are told what to work on by whoever runs them, and C's mechanism for that is a parameter list on main that this curriculum has mentioned exactly twice and never examined. It is a count and an array of strings, which sounds unremarkable until you notice that the count and the array disagree about how many elements there are, deliberately, and that the convention resolving the disagreement is one your program is expected to know rather than discover. The next chapter is about accepting arguments properly — the two forms main may have, what is guaranteed about each element, and the validation that stands between a command line and everything the rest of this module taught you to check.