Following a File While It Is Still Being Written

Part I · Module 4 · Chapter 4.4 · Following a Growing File

1. A File That Is Not Finished

$ tail -n 20 sim/run_042/sim.log
Compiling rtl/alu.sv
Compiling rtl/counter.sv
Elaborating top

That is where the run had got to when the command ran. Chapter 4.3 ended on exactly this limitation: tail prints and exits, so what you saw is already out of date if the job is still going.

Run it again a minute later and you get a different answer. Run it thirty times and you have invented a very inefficient way of watching.

$ tail -f sim/run_042/sim.log
Compiling rtl/alu.sv
Compiling rtl/counter.sv
Elaborating top

The last lines appear, and then the command does not exit. It sits there. As the running job writes more, the new lines appear on your screen as they are written.

-f is for follow. Press Ctrl-C to stop watching.

2. The Normal Use

$ tail -f build.log
Compiling module 41 of 120
Compiling module 42 of 120
Compiling module 43 of 120

New lines arrive every few seconds. You can see progress, you can see errors as they happen rather than afterwards, and you find out the moment something goes wrong instead of after it has wasted ten minutes.

Start further back with -n, which combines exactly as Chapter 1.5 describes:

$ tail -n 50 -f build.log

Fifty lines of context first, then live updates. This is usually what you want — the last few lines alone rarely explain what you are looking at.

Follow more than one file and tail labels each as it switches:

$ tail -f compile.log sim.log

==> compile.log <==
Compiling rtl/alu.sv

==> sim.log <==
Starting simulation

Useful when two parts of a job write separately, and the headers tell you which stream a line came from.

3. The First Silence: the Log Was Rotated

Here is the first of the two situations where following misleads you, and it is specific enough to recognise on sight.

Long-running systems do not let a log grow forever. Periodically the current log is renamed — run.log becomes run.log.1 — and a new, empty run.log is created to take its place. This is called log rotation, and it is automatic on most systems.

The problem is what -f is actually following.

The reason is Chapter 2.1's distinction between a name and the thing it names. tail -f opened a file and holds on to that file. Renaming it does not disturb that — Chapter 3.2 established that renaming only changes a directory entry, and the file itself is untouched. So tail -f keeps watching a file that nothing writes to any more, and reports, accurately and uselessly, that nothing is being added.

Your screen shows silence. The job is running fine. The new output is going somewhere you are not looking.

The fix is one character:

$ tail -F run.log

Uppercase -F follows the name. If the file at that name is replaced, it notices and reopens, usually telling you so:

tail: run.log: file truncated

That message is not an error — it is -F reporting that it has picked up the new file and carried on.

4. The Second Silence: Output Is Buffered

The second case is harder to recognise and causes more wasted time, because there is nothing to see and nothing is wrong.

A program writing to a file does not usually write every line the instant it is produced. It collects output in memory and writes it out in blocks — typically a few kilobytes at a time. This is called buffering and it exists because many small writes are far more expensive than one large one.

The consequence for you: the file does not contain the program's most recent output. Lines that have been produced may still be in the program's memory, not yet written. tail -f is showing you the file faithfully; the file is just behind.

What you see:

$ tail -f slow_job.log
Starting phase 2

...and then nothing, for a minute. Then suddenly forty lines appear at once.

Nothing hung. The program filled its buffer and flushed it.

How to recognise it: output arriving in bursts rather than steadily, with the bursts roughly equal in size, is buffering almost every time. A job that is genuinely stuck produces nothing at all, ever, with no bursts.

How to check properly — do not guess from the screen:

$ ls -l slow_job.log
-rw-r--r-- 1 engineer engineer 48192 Mar 22 11:04 slow_job.log

Run that twice, a minute apart. If the size is growing, work is happening. Chapter 2.5's size and timestamp fields answer directly what the silence cannot.

Module 7 teaches the process tools that tell you whether a program is consuming CPU, which is the other half of this question. Making a program flush its output sooner is possible and belongs to Module 12 and beyond; for now, the important thing is to stop reading silence as failure.

5. Following From Inside less

Chapter 4.2 mentioned this and it is worth putting side by side.

$ less run.log

Then press F to follow, Ctrl-C to stop following, and you are back in full navigation with the whole file available.

The second is better when you expect to react to what you see. With tail -f, noticing an error means stopping, then opening the file again to look around it. With less, Ctrl-C and you are already there with the error on screen.

6. Stopping, and What Stops With It

Ctrl-C

This ends tail, nothing else. Chapter 1.3 established that the terminal turns Ctrl-C into an interrupt for the foreground program, and the foreground program here is tail. The job writing the log is a different process entirely and carries on.

Two related points worth having:

Closing the terminal does stop the follow, and may stop more than that. Chapter 1.3 covered what closing a terminal does to things started from it; Module 7 covers how to start work that survives.

Following costs almost nothing. It is not consuming the job's resources or slowing it down. Leaving one open while you do something else is entirely reasonable.

7. Why This Matters in Engineering Work

This is the command engineers leave running on a second screen all day.

Watching a long job. A simulation or build that takes an hour produces output throughout. Following it means you see a failure at minute three instead of at minute sixty — and the time saved is the entire remaining runtime of a job that was never going to succeed.

Watching the right file. Jobs commonly write more than one log — one for compilation, one for the run itself. Following the wrong one shows a silence that means "this phase finished", not "nothing is happening". Chapter 4.3's head on each file, or the multi-file form in §2, resolves it.

Rotation is normal on shared machines. Long-lived services and busy systems rotate logs as a matter of routine, which is precisely where -f goes quiet and -F does not. This is the main reason to make -F the habit.

Part II treats log monitoring as a subject with tooling of its own. The manual skill here is what that tooling automates, and knowing what you would watch for by hand is what makes the automated version possible to write.

8. Practice

Question 1. What is the difference between tail -n 20 run.log and tail -n 20 -f run.log? Which one exits?

Question 2. You press Ctrl-C while following a build log. What stops, and what keeps going? Which chapter explains the distinction?

Question 3. tail -f run.log has shown nothing for five minutes. List the four possible explanations and say which single command distinguishes some of them.

Question 4. Explain, using the difference between a name and a file, why -f goes permanently silent after a rotation while -F does not.

Question 5. tail -F prints tail: run.log: file truncated. Is this an error? What has just happened?

Question 6. Output appears in bursts of about forty lines every minute or so. What is the most likely explanation, and why is it not a problem?

Question 7. You want to watch a log and be able to scroll back the moment an error appears. Give the two commands or keys, and say why this beats tail -f for that purpose.

Question 8. Which should you type by default, -f or -F, and what is the downside of the one you did not choose?

9. Remember

• tail -f prints new lines as they are written and does not exit. Ctrl-C stops the watching, not the job.
• tail -n 50 -f gives you context before the live view, which is almost always what you want.
• Silence has four possible meanings: between buffered bursts, rotated log, stuck job, or finished job. They look identical on screen.
• Log rotation renames the file. -f follows the file it opened and goes silent forever; -F follows the name and reopens. Make -F the default.
• Programs buffer their output, so the file can lag behind what the program has produced. Bursts of output are normal, not a fault.
• When a follow goes quiet, check the file, not the screen. ls -l twice, a minute apart, tells you whether anything is still being written.
• less then F follows too, and Ctrl-C returns you to full navigation — better when you expect to react to what you see.
• Following costs nothing and does not affect the job.

The next chapter stops reading content and starts measuring it — turning a file into a number, which is the simplest useful thing you can ask about one and the foundation of every summary you will later build.