NOTE: I'm not a sysvinit developer, so don't take this as being any kind of official response.  However, I was recently affected by a very similar issue, decided to read the init source code, and eventually solved it with some interesting tricks.  Thus, what I learned might be of use to somebody.

This is one of those issues that can be seen as either a bug or a feature request depending on your exact point of view.  Unmounting a live root filesystem is not normally possible on any Linux system, and that restriction is maintained by the kernel (not init, which has very little involvement at all).  As a side note, the current root filesystem and 'rootfs' are also technically two different things.  'rootfs' is what the kernel names the original root filesystem, which is either a ramfs or tmpfs, and sets up well before any storage devices are mounted.  The contents of the initramfs (if any) will be extracted to 'rootfs'.  You can't ever unmount that original root filesystem unless you want to forcefully cause a kernel panic or crash of some kind.  It's a (potentially flawed) design decision, just like the inability to kill PID 1.

Another part of this complex issue is that it's normally not necessary to actually unmount the root filesystem to get a clean shutdown.  For simple environments -- where there's no initramfs and no special conditions on the storage devices and their availability -- the kernel will do the key initialization and finalization work of loading and unloading the root filesystem.  Furthermore, the kernel seems to treat files that are open by PID 1 in a special manner  (or at least files open on virtual filesystems like devtmpfs and tmpfs).  Open handles such as these will not prevent a clean unmounting of the root filesystem in ordinary cases.  This means the vast majority of users never run into this issue at all.

Once we bring in special scenarios, such as having the main root filesystem stored on a loop device (for example, a loop-mounted file living on another filesystem), these conditions fall apart.  The kernel appears to stop the special behavior that it had in the trivial (or normal) case and it's not possible to cleanly unmount the loop-mounted root filesystem in an easy way.

There are two sources of open files that can keep the root filesystem "busy".  One is any shutdown programs or scripts that run during the system halting process.  We can eliminate any dependencies of these on the root filesystem by using a "shutdown" tmpfs and putting self-contained tools, scripts, and data into that.  Then we use pivot_root (not that simple, considering the bizarre semantics of the pivot_root system call) to switch the root filesystem and the shutdown tmpfs.  This solves the first problem, if implemented correctly (correct implementation is tricky for multiple reasons).

However, the second problem is that init, being PID 1, never dies, and thus any open file references it has will stay open unless and until it closes them.  Without the kernel giving special treatment to init, this will lock the old root filesystem even from inside the shutdown tmpfs.  (The kernel manages open files based on internal numeric descriptors, not paths.)  One would think this isn't too great an issue, though, because init actually uses very few files directly and normally closes all of them pretty much immediately after use.

Reading the source code, there appears to be one (maybe two) exceptions to this rule.  The more significant one is '/run/initctl', which used to be '/dev/initctl'.  This is named pipe that can be used to send certain control signals to init uses a particular binary (and vaguely documented) interface.  As far as I can tell, it's mainly used by separate instances of init itself (for example in the telinit form).  This named pipe is opened in check_init_fifo() and then does not appear to ever be closed during normal operation.  It will be closed and re-opened during some special operations, including in the re_exec() function.  So one way to release it is to cause re_exec() to fire, which can actually be done by writing a specially crafted file to the named pipe itself.  It's amusing and ironic that the problem can also be the solution in this case.

To make this method work, you need a usable executable /sbin/init in your shutdown tmpfs, you need the special binary file crafted according to the 'init_request' structure, and you must write the file to the old pipe after pivot_root has already succeeded and you are running a program or script in the new tmpfs root.  I've attached the little-endian version (should work in x86 and x86_64) of this crafted binary file, which I call 'reload-init.bin'.  If used correctly in the manner just described, it should accomplish the task of causing init to close the named pipe, activate an execle() system call and replace itself, thus completely freeing up any old open file descriptors.  If your /sbin/init in the shutdown tmpfs is actually another real init process (it shouldn't need to be, try using any non-terminating background program), then keep in mind you must create a dummy /etc/inittab to prevent it from doing anything.

There is one other file that init may have open sometimes, and that's /dev/console .  Usage of this is spread all over the source, and there's various places where it is opened and doesn't seem to be closed immediately.  However, by pivoting to the shutdown tmpfs and redirecting all standard file descriptors at the same time, all future references to /dev/console should be eliminated.  Any current references can then be dropped by making sure to mount devtmpfs in the shutdown environment before using the re-exec trick described in the previous paragraph.

So, that's one way to solve the issue, and it doesn't involve any changes to the source code.  A much easier way is just to setup a way to run 'init U' from inside the shutdown tmpfs, and thus have the program do the same work for you.  I don't like having the sysvinit init and dependencies in the tmpfs myself, so that approach is a non-starter for me.

Looking at the source once more, it looks like there might be a third and easier way to accomplish the release of the initctl pipe, and that's to send SIGUSR1.  For example, "kill -SIGUSR1 1" since init is PID 1.  This looks like it would work based on a shallow reading of the process_signals() function, but it might not in practice.  For example, the comment says that it is supposed to close and re-open the pipe file.  If the comment is correct, I'm not seeing where it is actually re-opened.

Nonetheless, I do think it's a bug that init holds any files open during the shutdown process.  Once we've passed the point where shutdown cannot be canceled anymore, it makes perfect sense to close the named pipe and /dev/console since we should be nearly 100% sure we won't need to accept any further commands or do any more I/O.  I'm not finding a particularly good place in the source to actually insert these close() calls, since it doesn't seem like there's any dedicated function for handling the shutdown process.  The closest matches for handlers of the pipe which also check or set the runlevel are fifo_new_level() and check_init_fifo().

As for the solution proposed by the Anonymous contributor back in Jun 2011, I would recommend against it.  Adding new features to solve what is basically a file handling bug is making the situation much more complicated than it needs to be.  Init already has enough ways to spawn root processes as it is, and adding another might open security flaws (whether through bugs or imperfections in the design).  Plus, it breaks forward compatibility due to changing the configuration format of inittab.  Likewise, it requires updating the documentation.  Too much work for what could be a much simpler fix.

Sorry for the verbosity, but hopefully the information is useful to someone.

(file #27982)