What Kind of File Is This? Text, Binary and Line Endings

Part I · Module 4 · Chapter 4.7 · What Kind of File Is This?

1. The Name Is Not the Content

Chapter 2.5 made the point in passing and it is worth making properly: nothing enforces a file's extension.

A file called report.txt can contain a compiled program. A file called data can be perfectly ordinary text. A file with no extension at all can be anything. The .txt is a convention people follow for their own convenience, and no part of the system checks it.

On Linux this matters more than on systems where the extension decides which program opens a file. Here the extension is a hint, and a hint that can be wrong.

2. file — Ask What It Actually Is

$ file run.log
run.log: ASCII text

$ file /bin/ls
/bin/ls: ELF 64-bit LSB pie executable, x86_64, dynamically linked

$ file notes.tar.gz
notes.tar.gz: gzip compressed data, from Unix

$ file empty.txt
empty.txt: empty

file reads the beginning of the contents and reports what it recognises. It does not look at the name at all — which is exactly why it is trustworthy when the name is not.

Several at once, as usual:

$ file *
alu.sv:      ASCII text
logs.tar.gz: gzip compressed data, from Unix
notes.txt:   ASCII text
run.sh:      Bourne-Again shell script, ASCII text executable

That last line is doing real work: file has identified a shell script, not merely "text", because it read the first line and recognised the #! that Chapter 1.7 introduced.

The three answers you will see most:

That last row is Chapter 4.1's warning, now with a command behind it. Before printing an unfamiliar file, ask file. It costs nothing and prevents the garbled-terminal problem entirely.

3. Text Is Not One Thing

file saying "text" settles the safe-to-print question. It does not settle everything, because two text files can still disagree about something invisible.

Every line in a text file ends with a marker. Two conventions exist:

The names come from mechanical typewriters: carriage return moved the carriage back to the left margin, line feed rolled the paper up one line. A teleprinter needed both. Linux kept only the second and uses it to mean "new line".

So a file written on Windows and copied to Linux has an extra character at the end of every line — one your terminal does not display, because a carriage return tells the cursor to go to the start of the line, which it is already about to do.

The file is different. The screen is identical. That is the whole problem.

4. Seeing the Invisible Character

Two commands reveal it. First, file often says so directly:

$ file crlf.txt
crlf.txt: ASCII text, with CRLF line terminators

$ file lf.txt
lf.txt: ASCII text

with CRLF line terminators is the tell, and it is the fastest check there is.

Second, the option Chapter 4.1 introduced without explaining:

$ cat -A crlf.txt
alpha^M$
bravo^M$

$ cat -A lf.txt
alpha$
bravo$

Now the difference is visible:

alpha^M$
     │ └── $  marks the end of the line (the newline)
     └──── ^M marks a carriage return — the extra character

$ is the end of every line in both files. ^M appears only in the CRLF file, and it is the carriage return. On systems without -A, cat -ve is the usual equivalent — Chapter 1.6's rule about the local manual applies.

A third confirmation, if you want a number:

$ wc -c crlf.txt lf.txt
      14 crlf.txt
      12 lf.txt
      26 total

Two bytes larger for two lines — one extra character per line, exactly as described. Chapter 4.5's byte count, used as evidence.

5. The Error That Names a Character You Cannot See

Here is where it stops being trivia.

$ cat -A run.sh
#!/bin/bash^M$
echo hello^M$

$ ./run.sh
bash: ./run.sh: /bin/bash^M: bad interpreter: No such file or directory

Read that message carefully, because it is one of the most baffling errors a beginner meets.

Chapter 1.7 explained the #! line: it names the interpreter that should run the script. The system reads that line to find the interpreter — and on this file, the first line does not say /bin/bash. It says /bin/bash followed by a carriage return. So the system looks for a program whose name literally ends in that character, does not find one, and reports exactly that.

The message is completely accurate and looks like nonsense, because the offending character cannot be displayed. The ^M in the error is the system's attempt to show you something unprintable.

The tell is ^M appearing anywhere in an error message. When you see it, you have a line-ending problem, and the diagnosis is file or cat -A.

6. Fixing It

Once diagnosed, the repair is mechanical. The direct form uses tr, which deletes characters:

$ tr -d '\r' < run.sh > run_fixed.sh

$ cat -A run_fixed.sh
#!/bin/bash$
echo hello$

The carriage returns are gone and the ^M markers with them. Two notes on that line: \r is how the carriage return is written, and the < and > are redirection, which is Module 12. The command is shown so you can repair a file when you need to; the mechanism is not being taught here.

Writing to a new file rather than over the original is deliberate — Chapter 3.2 established that overwriting is silent and irreversible, and there is no reason to risk the original while you are still checking the fix worked.

Two other routes worth knowing:

Your editor can do it. Most editors show the line-ending convention somewhere in their status bar and will convert on request. If you are editing the file anyway, this is the least ceremony.

A dedicated command may exist. dos2unix does exactly this job and is installed on many systems. file first, then dos2unix run.sh if it is available.

The real fix is upstream. If files keep arriving this way, something in how they are produced or transferred is converting them. Fixing each file forever is worse than fixing the source once.

7. The Diff That Made No Sense

Chapter 4.6 ended on this and now it resolves.

$ diff old.txt new.txt
1,2c1,2
< alpha
< bravo
---
> alpha
> bravo

diff claims every line changed. The lines it shows are visibly identical. This looks like a broken tool.

It is not. One file has carriage returns and the other does not, so every line genuinely differs — by one invisible character each. diff is comparing bytes and reporting the truth; your screen is hiding the evidence.

Two ways forward:

$ file old.txt new.txt
old.txt: ASCII text, with CRLF line terminators
new.txt: ASCII text

Diagnosis in one command. Then either fix the endings per §6, or tell diff to look past them:

$ diff --strip-trailing-cr old.txt new.txt

Chapter 4.6's -w would also hide it, by ignoring whitespace generally. Prefer the specific option: -w hides other whitespace changes too, and some of those might be the thing you were looking for.

8. Encoding, Briefly

One more property of a text file, mentioned so the vocabulary is not a surprise later.

file distinguishes ASCII text from UTF-8 Unicode text. The difference is which characters the file can contain: ASCII covers unaccented English and basic punctuation; UTF-8 covers essentially everything, using more than one byte for the less common characters.

This is the reason Chapter 4.5's -c and -m can disagree — bytes and characters stop being the same thing the moment a file contains anything outside ASCII.

For Part I that is the whole of it. Modern systems default to UTF-8, text behaves as you expect, and encoding only becomes a subject when a file arrives from somewhere using an older convention. Knowing the word is enough.

9. Why This Matters in Engineering Work

Two situations, both common.

A script that will not run. A colleague sends a script, or you retrieve one from a shared location, and it refuses with bad interpreter or an error mentioning ^M. This is the single most common cause, and knowing it turns a confusing half-hour into a thirty-second fix. Chapter 1.7 gave you the other two reasons a script might not run — not found, and no execute permission — and this is the third.

Mixed-origin files in a shared project. On a team where people work on different operating systems, files drift into mixed conventions. The symptoms are diffuse and annoying: diffs that claim everything changed, tools reporting errors on lines that look fine, and comparisons that never match. file on the suspicious file is the first diagnostic, every time.

The habit that generalises: when a file behaves differently from how it looks, stop trusting your eyes and ask a command what is actually in it. file, cat -A and wc -c are three different ways of doing that, and between them they settle the question.

10. Practice

Question 1. Why can you not rely on a .txt extension to tell you a file is text? Which command answers the question properly?

Question 2. file data.out reports gzip compressed data. What should you not do next, and why?

Question 3. What are the two line-ending conventions, how many characters does each use, and which does Linux use?

Question 4. cat -A script.sh shows #!/bin/bash^M$ on the first line. Name both markers and say what each represents.

Question 5. A script fails with bad interpreter: /bin/bash^M. Explain precisely what the system looked for and why it did not find it.

Question 6. Two files look identical on screen; diff reports every line as changed. Give the diagnosis command and two different ways to proceed.

Question 7. wc -c reports 14 bytes for a two-line file that you expect to be 12. What does the difference tell you?

Question 8. List three ways a file with the wrong line endings could reach you without you ever using Windows.

Question 9. Why is it better to fix whatever is producing CRLF files than to convert each file as it arrives?

11. Remember

• A file's name tells you nothing about its contents. Extensions are a convention, not a rule.
• file reads the contents and reports what they are. Ask it before printing anything unfamiliar — that is Chapter 4.1's warning made actionable.
• Two line-ending conventions exist: LF, one character, used by Linux; and CRLF, two characters, used by Windows. A CRLF file has one extra invisible character per line.
• file usually says so directly — with CRLF line terminators — and cat -A shows it as ^M before the $.
• ^M in any error message means a line-ending problem. The classic is bad interpreter: /bin/bash^M, which is the system accurately reporting that no such program exists.
• A diff where every line changed, but the lines look identical, is this. Diagnose with file; fix the endings, or use --strip-trailing-cr.
• Repair with tr -d '\r' into a new file, with your editor, or with dos2unix — and fix the source if files keep arriving that way.
• ls -lh, then file, then head is the complete opening routine for an unknown file.
• When a file behaves differently from how it looks, ask a command what is in it. Your eyes cannot see every byte.

That completes Module 4. You can now read a short file, explore a long one, sample either end, watch one being written and know what its silences mean, measure a file as a number, compare two files and read the result in the notation the whole industry uses, and determine what a file actually contains when its name and its appearance are both unreliable. Module 5 turns from the contents of files to the rules about who may read, change and run them — the permission string that Chapter 2.5 taught you to read, and that you will now learn to reason about and change.