Archiving and Compressing: tar and gzip

Part I · Module 3 · Chapter 3.6 · Archiving and Compressing

1. Two Different Jobs

The confusion worth clearing first.

Bundling is putting many files into one file, preserving the directory structure and the names. The result is not smaller — it is roughly the sum of what went in. The tool is tar.

Compressing is making one file smaller by storing its contents more efficiently. It does not combine anything. The tool is gzip (and its relatives).

Two jobs, two tools, and the reason .tar.gz has two extensions:

logs.tar.gz
    │   └── then compressed with gzip
    └────── first bundled with tar

Read right to left as undoing, left to right as doing. A .tar.gz file was first bundled and then compressed, so unpacking it means decompressing and then unbundling.

2. Reading tar's Options

A typical command, of the kind usually copied without being read:

tar -czf logs.tar.gz logs/

Chapter 1.5 gives you everything needed to take it apart. -czf is three short options clustered into one word, and tar unpacks the letters itself:

So the whole line reads: create an archive, compress it with gzip, write it to logs.tar.gz, and put logs/ in it.

The letters you will use:

c, x and t are the three jobs, and exactly one of them appears in any command. f is in almost every command and §9 explains what happens when it is not. The rest are modifiers.

With that table, a command you have never seen becomes readable:

tar -xzvf results.tar.gz

Extract, gzip, verbose, and the next word is the file. No memorisation required.

3. Creating an Archive

$ pwd
/home/engineer/project/sim

$ ls logs
run1.log  run2.log  trace.txt

$ tar -czf logs.tar.gz logs/

$ ls -lh logs.tar.gz
-rw-r--r-- 1 engineer engineer 603 Mar 20 16:12 logs.tar.gz

One file now contains all three. Reading the listing with Chapter 2.5: an ordinary file, 603 bytes with -h making the size readable.

Note the operand order, which is the opposite of cp and catches people:

tar -czf  <archive to create>  <what to put in it>

The archive name comes first, because it is the value belonging to f. The things being archived are the operands after it. In cp the destination is last; here it is first. That is not an inconsistency to be annoyed by — it is f taking its value, exactly as Chapter 1.5 described for any option that takes one.

Add v to watch it work:

$ tar -czvf logs.tar.gz logs/
logs/
logs/run1.log
logs/run2.log
logs/trace.txt

Several sources go in one archive:

$ tar -czf backup.tar.gz logs/ scripts/ notes.txt

4. Always List Before You Extract

Before unpacking anything, look inside it. t lists without extracting:

$ tar -tzf logs.tar.gz
logs/
logs/run1.log
logs/run2.log
logs/trace.txt

Every entry begins with logs/, so extracting will create one directory called logs and put three files in it. Tidy, predictable, contained.

Now the reason this is a rule rather than a suggestion. An archive stores the paths it was given, and nothing requires them to be inside a single directory:

$ tar -tzf mystery.tar.gz
run1.log
run2.log
trace.txt
config.yaml
notes.txt

No common directory. Extracting this scatters five files directly into your working directory, mixed in with whatever is already there — and if any name matches something you already have, the existing file is overwritten. People call this a tar bomb, and it is unpleasant precisely because unpicking it means working out which of the files in your directory were not yours a minute ago.

Listing first costs one command and tells you which kind of archive you have.

$ tar -tvzf logs.tar.gz
drwxr-xr-x 0 engineer engineer 0 Mar 20 16:12 logs/
-rw-r--r-- 0 engineer engineer 9 Mar 20 16:12 logs/run1.log

Adding v to a listing gives Chapter 2.5's long-listing fields for each entry — type character, permissions, owner, size and timestamp. Useful when you want to know how big something will be before unpacking it.

5. Extracting

$ tar -xzf logs.tar.gz

$ ls
logs  logs.tar.gz

x extracts. The contents appear in the working directory — which is why Chapter 2.4's pwd reflex applies here as much as anywhere.

The safer form chooses the destination explicitly:

$ mkdir -p unpacked

$ tar -xzf logs.tar.gz -C unpacked

$ find unpacked
unpacked
unpacked/logs
unpacked/logs/run1.log
unpacked/logs/run2.log
unpacked/logs/trace.txt

-C means change to this directory first, so everything lands inside unpacked whatever the archive contains. Combined with mkdir -p from Chapter 3.1, that turns an unknown archive into a contained one: even a tar bomb can only scatter itself inside a directory you created for the purpose.

Note that -C is a single-dash multi-letter option that takes a value, which Chapter 1.5 flagged as one of the non-conforming grammars. tar is the example that chapter named.

Extract one file rather than all of them by naming it exactly as the listing showed it:

$ tar -xzf logs.tar.gz logs/run2.log

6. gzip on Its Own

gzip compresses a single file, and the surprise is what it does to the original:

$ ls -l notes.txt
-rw-r--r-- 1 engineer engineer 4096 Mar 20 16:20 notes.txt

$ gzip notes.txt

$ ls
notes.txt.gz

notes.txt is gone. gzip replaced it with the compressed version rather than producing a second file. That is standard behaviour and startles everyone once.

Reverse it:

$ gunzip notes.txt.gz

$ ls
notes.txt

To keep the original, ask:

$ gzip -k notes.txt

$ ls
notes.txt  notes.txt.gz

-k is keep. It is a relatively recent addition and not present on every system — Chapter 1.6's rule applies.

Two things gzip does not do, both worth stating because the misconception is common:

It does not bundle. Give it several files and you get several .gz files, not one archive.

It does not help much on everything. Text compresses dramatically — log files often to a tenth of their size. Already-compressed data barely moves. Compressing an archive of images saves almost nothing and costs the time.

7. Choosing Between Them

The short rule: more than one file, use tar. Exactly one file, gzip is enough.

8. Other Formats You Will Meet

Three you will see, and what to do with each:

.tar.bz2 and .tar.xz — the same tar bundle with a different compressor. Modern tar detects the compression itself, so tar -xf archive.tar.xz works without naming the letter. The z in the examples above is likewise optional on current versions; it is written here because it is explicit, and because older systems need it.

.zip — bundles and compresses in one step, and is what you receive from other operating systems. Different commands: unzip file.zip, and unzip -l file.zip to list, which is §4's habit under a different name.

.gz alone — one compressed file, per §6. No bundle inside it.

9. The Failures

Forgetting f. This is the classic.

$ tar -cz logs/

Without f, tar does not know where to write, so it writes the archive to its standard output — which is your screen. GNU tar refuses when that output is a terminal and tells you so; the BSD version used on macOS does not, and floods the terminal with binary. Either way the lesson is the same: f names the file, and without it the archive goes somewhere you did not intend. (Sending it somewhere useful on purpose is redirection, which is Module 12.)

Getting the clustered letters in the wrong order. This is Chapter 1.5's rule with real consequences:

$ tar -cfz logs.tar.gz logs/
tar: logs.tar.gz: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.

$ ls -l z
-rw-r--r-- 1 engineer engineer 10240 Mar 20 16:30 z

Read what happened. In -cfz, f is not last, so it took the next letter in the cluster as its value: the archive was named z. logs.tar.gz was then treated as a source to archive, and there is no such file, hence the error. You are left with a file called z and no archive.

Chapter 1.5 stated the rule — a clustered option that takes a value must come last — and this is the same mechanism, now producing a confusing error that names a file you never asked about. f last, always.

Extracting in the wrong directory. §5. pwd first, or use -C.

Extracting an archive you did not list. §4. One command avoids a cleanup.

Expecting tar alone to compress. tar -cf logs.tar logs/ produces a bundle that is not smaller. That is correct behaviour — §1.

10. Why This Matters in Engineering Work

One habit dominates, and it closes a loop opened in Chapter 3.3.

Archive before you clean. That chapter established that deleting is permanent and that disk pressure makes deleting routine. The two together are a trap: the directory you are about to clear is also the evidence of whatever went wrong in it.

$ pwd
/home/engineer/project/sim

$ tar -czf run_042_failed.tar.gz run_042/

$ tar -tzf run_042_failed.tar.gz | head -3
run_042/
run_042/compile.log
run_042/sim.log

$ rm -rf run_042

Four commands in a deliberate order: bundle it, confirm the bundle actually contains what you think, and only then delete. A compressed archive of text logs is typically a small fraction of the original, so keeping it costs very little and preserves the ability to answer questions about the failure later.

The second reason is transfer. Sending someone a thousand small files is slow and loses the directory structure; sending one archive is fast and arrives intact. On a remote machine this is often the only practical way to move a result set.

Part II treats retention, archiving and disk pressure as a subject of their own. The command is this one.

11. Practice

Question 1. Explain, in one sentence each, what tar does and what gzip does, and why .tar.gz has two extensions.

Question 2. Read tar -xzvf results.tar.gz letter by letter and say what each one contributes.

Question 3. Write the command that bundles and compresses sim/run_7/ into run_7.tar.gz.

Question 4. You have been sent data.tar.gz and do not know what is inside. Give the command you run first and say exactly what you are looking for in its output.

Question 5. The listing from Question 4 shows eight files with no common leading directory. What will plain extraction do, and give the two commands that contain it safely.

Question 6. tar -cfz backup.tar.gz logs/ reports that it cannot find backup.tar.gz and leaves behind a file called z. Explain precisely why, referring to the rule from Chapter 1.5.

Question 7. You run gzip report.log and then cannot find report.log. What happened, which command brings it back, and which option would have avoided the surprise?

Question 8. You need to free space in a results directory but may still have to investigate a failure. Write the three commands, in order, and say what the middle one is for.

Chapter 28.5 puts this to work on a failing simulation, and measures a --exclude that is silently ignored because of where it sits on the command line.

12. Remember

• Bundling and compressing are different jobs. tar combines many files into one; gzip makes one file smaller. .tar.gz is both, bundled first.
• tar's letters are ordinary clustered short options: c create, x extract, t list, f the archive name, z gzip, v verbose. Exactly one of c, x, t per command.
• f must be last in the cluster, because it takes the next word as its value. -cfz names the archive z and then fails confusingly.
• The archive name comes before the things being archived — the opposite of cp's destination-last order — because it is f's value.
• Always list before extracting. tar -tzf shows whether the contents are in one directory or about to scatter across yours.
• Extraction overwrites silently, like everything else in this module. Combine mkdir -p and -C to extract into a directory you chose.
• gzip replaces the original unless you pass -k, and compresses one file rather than bundling several.
• Archive before you clean. Bundle, verify the bundle, then delete — in that order.

That completes Module 3. You can now add to the tree, duplicate and relocate parts of it, remove things with a routine that makes the irreversible mistake structurally unlikely, write names that do not fight the shell, select groups of files with patterns and verify the selection before acting on it, and package a set of files into one object to keep or send. Module 4 stops changing the tree and starts reading what is inside the files — beginning with the ones too large to open.