Detaching a process from terminal - exec(), system(), setsid() and nohup

Linux processes are created by fork() and exec().
The very first process of POSIX systems is init and subsequent processes are derived from the init as parent.
These subsequent processes are child processes.
During forking the parent process copies itself - with all I/O, address space, stack, everything.
The only thing that is different is the process ID.
The parent and child will have 2 different process IDs.

The system() library function uses fork(2) to create a child process that executes the shell command specified in command using execl(3) as follows:
execl("/bin/sh", "sh", "-c", command, (char *) 0);
system() returns after the command has been completed.
system() executes a command specified in command by calling /bin/sh -c command, and returns after the command has been completed. During execution of the command, SIGCHLD will be blocked, and SIGINT and SIGQUIT will be ignored. 
system() calls are often made within programs to execute scripts.
Blocking on SIGCHLD means that the calling process will wait for child's termination.

exec() calls are different. It replaces the current process image with the exectued process. Which means if the exec() call returns it has failed.
e.g. // some fancy code
      //exec(...)
     // if it comes here, exec() call has failed.

2 more things to mention. Each process also has a process group ID (PGID) and a session ID (SID).
PGID is the ID of the process group leader and SID is the PID of the session leader.

Sessions and process groups are just ways to treat a number of related processes as a unit. All the members of a process group always belong to the same session, but a session may have multiple process groups.

Normally, a shell will be a session leader, and every pipeline executed by that shell will be a process group. This is to make it easy to kill the children of a shell when it exits.



Each line of the tree is a session.
The horizontal lines correspond to the same process group.
The session leaders are to the left.
The group leader is to the left of each horizontal line because a single session can have multiple process groups.

Now to the important stuff.

We want to detach our process from the terminal and let it run alone. That is we want a daemon process.
This essentially means that we need the process to run when we close the terminal that we launc the program from.
Or we can even use a program itself to launch another program and we want our child program to live after the parent program terminates.


Note: If a certain group process is terminated using SIGTERM, everyone in the group will receive this signal.

This also means ideally that our process should either be in a different process group where the session leader/process group leader is not the terminal/parent program
OR
better yet in a different session.

nohup

nohup and screen are tools that you can use to detach your process from the current terminal.
When you use nohup, all output is directed to the nohup file.
The important thing to understand is that nohup is a program and the parameter passed to it is the process it should run.
e.g. nohup ./tqxman &

If you don't have &, nohup will run as a foreground process and if you use ^C a SIGTERM will be sent and the process will close.
However if the terminal is just closed, the associated process will get the SIGHUP which will be ignored. Why?
nohup is capable of blocking the HUP signal.

This will ensure that the process will ultimately get attached to the init process, although initially ./tqxman would appear as belonging to the shell.

A quick look at nohup.c reveals this.

if (isatty(STDOUT_FILENO))
 dofile();
if (isatty(STDERR_FILENO) && dup2(STDOUT_FILENO, STDERR_FILENO) == -1)
 /* may have just closed stderr */
 err(EXIT_MISC, "%s", argv[0]);

(void)signal(SIGHUP, SIG_IGN);

execvp(*argv, argv);
exit_status = (errno == ENOENT) ? EXIT_NOTFOUND : EXIT_NOEXEC; 

err(exit_status, "%s", argv[0]);

First lets' look at the process tree when we launch the program tqxman via a shell. See that the tqxman is a child of the bash.

And what happens when the terminal is closed. See that tqxman's parent has died. The whole process group received a SIGHUP but nohupping tqxman ignored SIGHUP.
Now init process is the parent.

The output of the program will be redirected to nohup.out

Screen is a similar tool where you can keep the terminal open and retrieve it later.
Using nohup you could only take a look at program output but screen can allow you to do both input and output because the whole terminal can be retrieved even after you close the launching terminal.
Pretty cool....

OK. now what's setsid() ?

setsid()

The linux man page says, setsid() creates a new session if the calling process is not a process group leader. The calling process is the leader of the new session, the process group leader of the new process group, and has no controlling terminal. The process group ID and session ID of the calling process are set to the PID of the calling process. The calling process will be the only process in this new process group and in this new session.

This means that executing setsid ./tqxman will directly attach the process tqxman to init; in a new session.

What's the catch?
What if the calling process is actually the process group leader?
This will make setsid fail and return error.
The notes of setsid() are important.

A child created via fork(2) inherits its parent's session ID. The session ID is preserved across an execve(2).
A process group leader is a process with process group ID equal to its PID. In order to be sure that setsid() will succeed, fork(2) and _exit(2), and have the child do setsid().

Based on this a demonizing procedure for a C process can be written which works even if the calling process is the group leader.