Work That Outlives the Terminal: nohup, setsid and tmux

Part I · Module 7 · Chapter 7.8 · Work That Outlives the Terminal

Chapter 7.5 warned that backgrounding a job does not detach it from your terminal, and used that to explain why output still barges onto your screen. This chapter is about the much more expensive version of the same fact.

1. The Problem, Demonstrated

Three jobs, started in one terminal. One plain, two with nohup in front:

$ sleep 400 &
[1] 12465

$ nohup sleep 400 &
[2] 12469
appending output to nohup.out

$ nohup bash -c 'sleep 400; echo done' &
[3] 12472
appending output to nohup.out

$ ps -o pid,ppid,tty,stat,comm -p 12465,12469,12472,12473
  PID  PPID TTY      STAT COMM
12465 12464 ttys000  S    sleep
12469 12464 ttys000  S    sleep
12472 12464 ttys000  S    bash
12473 12472 ttys000  S    sleep

All four attached to ttys000, all children of the shell at 12464. Nothing yet distinguishes them.

Now close the terminal, and look again from somewhere else:

$ ps -o pid,ppid,tty,stat,command -p 12465,12469,12472,12473
  PID  PPID TTY      STAT COMMAND
12469     1 ??       S    sleep 400
12472     1 ??       S    bash -c sleep 400; echo done
12473 12472 ??       S    sleep 400

12465 is gone. The plain background job — the one started with nothing but & — did not survive the terminal closing.

The other three did, and two things about them changed:

• PPID is now 1. The shell that started them is gone, so PID 1 adopted them — exactly the orphan mechanism from Chapter 7.4.
• TTY is now ??. They no longer have a controlling terminal, which is the ? Chapter 7.2 taught you to read.

One word of difference on the command line, and one job out of two survived the night.

2. Why

When a terminal closes — the window is shut, an SSH connection drops, a laptop sleeps and the session times out — the kernel sends SIGHUP to the session's processes. The name is short for hang up, and it is a genuine survival from the era when the terminal was a device at the end of a telephone line and closing the session meant hanging up the handset.

SIGHUP's default action is to terminate. From Chapter 7.6, that means:

• a program that has not changed the default dies;
• a program that ignores SIGHUP does not.

That is the entire mechanism. The three tools in this chapter are three different ways of arranging the second outcome.

3. nohup

nohup runs a command with SIGHUP set to be ignored. The name is literally no hangup.

nohup ./long_job.sh &

Three things happen:

1. SIGHUP is ignored for that process and everything it starts, so the terminal closing has no effect on it.

2. Output is redirected to nohup.out in the current directory, if you have not redirected it yourself — because a detached job cannot write to a terminal that may not exist. nohup tells you it is doing this:

   Azvya Education Pvt. Ltd.VLSI Mentor

   Snippet

   $ nohup sleep 400 &
   [2] 12469
   appending output to nohup.out

   (The exact wording of that line differs slightly between systems; the behaviour does not.)

3. Input is disconnected. A nohup'd command cannot read from the terminal, which is the correct behaviour for something meant to run unattended — but means an interactive tool will fail immediately rather than hanging.

Always redirect explicitly

nohup.out is a poor default. It lands in whatever directory you happened to be in, it is the same name for every job so two jobs overwrite each other's context, and it tells you nothing about which run it belongs to.

nohup ./long_job.sh > run_$(date +%Y%m%d_%H%M).log 2>&1 &

When you redirect yourself, nohup leaves it alone and creates no nohup.out at all:

$ nohup bash -c 'echo hello from the detached job' > job.log 2>&1 &
[1] 14012

$ cat job.log
hello from the detached job

$ ls nohup.out
ls: nohup.out: No such file or directory

> file 2>&1 sends normal output and error output to the same file — Module 12 covers the notation; it is worth using now.

Keep the PID

Once the terminal is gone, so is the job table — Chapter 7.5 noted that jobs belong to one shell. The PID is what survives, so record it:

nohup ./long_job.sh > run.log 2>&1 &
echo $! > run.pid
echo "started as $(cat run.pid)"

Tomorrow, from any terminal:

kill -0 "$(cat run.pid)" && echo "still running"
tail -f run.log

That three-line pattern — redirect, record the PID, follow the log — is the whole of unattended job management, and it works on every Unix system without installing anything.

4. setsid

nohup makes a job ignore the hangup. setsid arranges for it never to be sent one, by starting the command in a brand-new session with no controlling terminal at all.

setsid ./long_job.sh > run.log 2>&1 &

The result looks the same in ps — PPID 1, TTY ? — and it is slightly more thorough: the process is not merely ignoring a signal it still receives, it is outside the session the signal is sent to.

In practice nohup is the one to reach for. It is older, more portable, and present on systems where setsid is not. (setsid is part of a Linux utilities package and is absent on macOS, for instance.) Use setsid when you want a process genuinely outside your session, and nohup the rest of the time.

5. tmux, and the Thing the Others Cannot Do

nohup and setsid both solve exactly one problem: the job keeps running. Neither lets you look at it again. The terminal it was attached to is gone, and nothing reattaches to a detached process — you get a log file and a PID, and that is all.

tmux solves a different problem. Instead of detaching the job from the terminal, it puts the job inside a terminal that does not go away.

A terminal multiplexer runs a long-lived terminal session on the machine itself. You attach to it to see it, detach to leave it running, and attach again later — from a different computer, a different network, days later. The programs inside never know anything happened; from their point of view the terminal was there the whole time.

The four commands that matter

tmux new -s build        # create a session named "build" and attach

You are now in a shell inside the session. Start whatever you like, interactively, watching the output as normal.

Ctrl-b  then  d          # detach — the session keeps running

Ctrl-b is tmux's prefix key: press and release it, then press the command key. d for detach. You are back in your ordinary shell, and everything in the session carries on.

tmux ls                  # list sessions
tmux attach -t build     # go back in

$ tmux ls
build: 1 windows (created Mon Oct  6 18:02:11 2026)

And when the work is finished, exit the shell inside the session normally, which ends it.

Why this is the one to learn

That last row is the one engineers feel most. A dropped SSH connection in the middle of an interactive tool loses the session; the same drop with tmux loses nothing, because the session is on the remote machine and you simply attach again.

6. Choosing

7. If You Forgot

A job is running in the foreground, you need to leave, and you did not plan ahead. Bash can rescue it, as long as the terminal is still open.

$ ./long_job.sh
^Z
[1]+  Stopped                 ./long_job.sh

$ bg
[1]+ ./long_job.sh &

$ disown -h %1

Three steps, each from a chapter you have already done:

1. Ctrl-Z suspends it (Chapter 7.5).
2. bg resumes it in the background.
3. disown -h %1 marks the job so that Bash does not send it SIGHUP when the shell exits.

disown with no -h removes the job from the job table entirely; -h keeps it listed but exempts it from the hangup. Either works for survival.

8. The Pattern Worth Keeping

# Start it, detached, with a named log and a recorded PID.
nohup ./nightly_regress.sh > "regress_$(date +%Y%m%d_%H%M).log" 2>&1 &
echo $! > regress.pid
echo "started $(cat regress.pid), logging to regress_$(date +%Y%m%d_%H%M).log"

# Later, from any terminal, on any day.
kill -0 "$(cat regress.pid)" 2>/dev/null && echo "running" || echo "finished"
tail -f regress_*.log

# And to stop it, using Chapter 7.7's escalation.
PID=$(cat regress.pid)
pkill -P "$PID"                 # the workers first
sleep 5
kill "$PID"                     # then the supervisor
sleep 5
kill -0 "$PID" 2>/dev/null && kill -9 "$PID"

Every line of that uses something from this module: nohup and redirection from this chapter, $! and kill -0 from 7.5, pkill -P and the escalation from 7.7, tail -f from Chapter 4.4.

Practice

1 — Reproduce the failure. In one terminal, start sleep 400 & and note the PID. Close that terminal. From another one, check whether the PID still exists. It will not.

2 — Reproduce the fix. Same thing with nohup sleep 400 &. Close the terminal, check again. It will still be there — and ps -o pid,ppid,tty will show PPID 1 and TTY ?.

3 — Watch nohup.out appear and then prevent it. Run nohup bash -c 'echo hello' & in an empty directory and find nohup.out. Then run it again with > my.log 2>&1 and confirm no nohup.out is created.

4 — Use the three-line pattern. Start a job with nohup, write $! to a file, and from a different terminal use that file to check whether it is alive and to follow its log.

5 — Learn the tmux loop. If tmux is available:

tmux new -s practice
# inside: run something that prints every second
#   while true; do date; sleep 1; done
# press Ctrl-b then d
tmux ls
tmux attach -t practice

Detach and attach three or four times until the prefix key is automatic.

6 — Prove tmux survives a disconnection. Over SSH, start something inside tmux, then close the SSH connection outright rather than detaching. Reconnect and tmux attach. The work is exactly where you left it.

7 — Rescue a job you forgot. Start a foreground job, Ctrl-Z, bg, disown -h %1. Then close the terminal and confirm from elsewhere that it survived.

8 — Find what you have already detached. List every process of yours whose parent is PID 1:

ps -u "$USER" -o pid,ppid,etime,cmd | awk '$2 == 1'

Check Your Understanding

1. You start a twelve-hour job with &, close your laptop, and find nothing running in the morning. What happened?

2. What is the difference between what & detaches and what nohup detaches?

3. Why does a nohup'd job write to nohup.out, and how do you avoid it?

4. A colleague says nohup makes a job immune to being killed. Correct them.

5. You need to run an interactive tool on a remote machine over an unreliable connection. Why is nohup the wrong choice?

6. You are already halfway through a long foreground job and need to leave. What do you do, and what do you lose?

9. Remember

• & does not survive logout. It detaches a job from the shell's waiting, not from the terminal.
• A closing terminal sends SIGHUP, whose default action is to terminate — which is why the plain background job dies.
• nohup makes a job ignore SIGHUP. It is everywhere, needs nothing installed, and is the portable answer.
• nohup redirects output to nohup.out unless you redirect it yourself. Always redirect yourself, to a named log.
• nohup disconnects input. Correct for unattended work; fatal for anything interactive.
• nohup does not make a job unkillable. It changes one signal and nothing else.
• Record $! to a file. The job table dies with the shell; the PID is what survives.
• setsid starts a job in a new session with no terminal, so the hangup is never sent. Slightly stronger, slightly less portable.
• tmux keeps a terminal alive on the machine itself, so you can detach, reconnect from anywhere, and find the session exactly as you left it.
• tmux new -s name, Ctrl-b d, tmux ls, tmux attach -t name — four commands, and the tool is yours.
• Start tmux before you start work, not after. A running job cannot be moved into it.
• disown -h %1 rescues a job you forgot to detach — but cannot rescue its output.
• A detached job shows PPID 1 and TTY ? in ps, which is how you recognise one later.

That completes Module 7, and with it the running machine. You can see what exists and who owns it, read whether the machine is genuinely in trouble, follow the parentage of any process, start and suspend and resume your own work, send any signal deliberately, end anything in the order that lets it finish properly, and leave a twelve-hour job running when you go home.

Module 8 starts making the shell compute — holding values in variables, turning a command's output into data, taking strings apart and counting. It is the step from using a shell to programming one.