Bob's Guide to System Upgrades with Debian
There are many guides to upgrading Debian and Debian derived systems. This one is mine.
As with painting a house the quality of the result is mostly on the quality of the preparation for painting. Preparing a system for upgrade is the same. It is the preparation that most influences the quality of the result.
At this time I am working through an upgrade of Trisquel 9 to Triquel 10 and am using that as the working example for the documentation here.
Release Notes
The first thing to do is to read the release notes for the new system. If there are special handling instructions then those must be followed. Historically while most upgrades have been generic there have been upgrades that required stepping the Linux kernel and udev forward in a particular ordering. And in the changing of init systems there have been requirements about the ordering of particular package upgrades. Always read the release notes and pay attention to any needed special handling.
Install etckeeper and git
The etckeeper package is a utility to keep changes of /etc files in a version control system such as git. Using git is most popular but others can also be used. Then all changes to /etc can be seen in the version history. This means that one can "purge" packages without fear of the loss of any particular file. Since the changes will all be in version control. One can also rm any of those /etc files too. Since the changes will all be in version control. It's a great safety net.
The git version control system is most popular. However git wants to know something about the user making commits. In this case the user is the root user. Therefore let's set up the root user for git first before even installing either git or etckeeper. Write this file to the root .gitconfig file.
cat >/root/.gitconfig <<EOF
[user]
name = root
email = root@localhost
EOF
Alternatively there is now a git command to make git configuration modifications.
git config --global user.name root
git config --global user.email root@localhost
Since root is actually a pseudo-user I prefer to use generic
information for the generic user. This is the default information
that is logged in the commit log when no other overriding author
information is provided. Admins that wish to override this may use
the GIT_AUTHOR_NAME and GIT_AUTHOR_EMAIL and related env
variables. Since that is then set up ensure that git is installed and
then install etckeeper.
apt-get install git
apt-get install etckeeper
At that point upkeep is automated. One can forget that it is there for the most part. All commits are then automated. To examine the commit log use normal git commands.
cd /etc
git log
This is very useful. I strongly recommend it. It creates a safety net such that then one can do cleanup and other modifications without fear of losing track of those files and changes.
Clean Old/New conffiles
The dpkg package manager tracks files in /etc as "conffiles". These are tracked such that if dpkg believes the file has NOT been locally modified that it automatically upgrade the file on package upgrade. However if dpkg believes the file has been modified locally then it will present a question as to how to handle it. Does the admin want to keep the current file? If so then the new file will be installed with a ".dpkg-new" suffix. Does the admin want to upgrade to the new package maintainer's file? If so then the new file will be installed and the old file will be moved to a ".dpkg-old" suffix. This makes it easy in either case for the admin to locate and further take action about these files.
Additionally some packages in the maintainer scripts will use the name convention ".dpkg-dist" and add custom management. An example package that does this is the bind9 package. The use of ".dpkg-dist" is similar to ".dpkg-new" but outside the control of dpkg since it is custom handling added by the package maintainer.
Some people did not like the handling provided by dpkg. They wanted more features in the upgrade. Also some packages wanted to do their own handling of /etc files which by Debian Policy needed to have local modifications preserved. The "ucf" utility was created for this purpose. See the "man ucf" page for documentation. Enough to say that ucf is like dpkg but different. Here we only need to know that it uses ".ucf-new" and ".ucf-old" suffixes instead of ones using the ".dpkg-*" string. The cleanup of these is the same.
find /etc -name '*.dpkg-*'
find /etc -name '*.ucf-*'
...I usually type in the simple form above...
find /etc \( -name '*.dpkg-*' -o -name '*.ucf-*' \) -print
...but one can combine the find into one command...
That will print the list of these files that exist. Review the list. Handle them all. Which means that if features should be merged then merge them. Or if files should be removed then simply remove them.
Most of the ".dpkg-old" and ".ucf-old" files should simply be removed as part of the cleanup task. Those are older versions of the configuration files that were not used and are not being used. The upcoming upgrade will create more of these files. We want to clean the system of these files from previous upgrades so that we don't confuse what has happened a decade ago with something that has just now happened as part of the current upgrade.
find /etc \( -name '*.dpkg-old' -o -name '*.ucf-old' \) -print
If that list is okay then remove them.
find /etc \( -name '*.dpkg-old' -o -name '*.ucf-old' \) -delete
Look at new version of the file that are left.
find /etc \( -name '*.dpkg-*' -o -name '*.ucf-*' \) -print
New versions of these files need attention. These are new package maintainer's versions of the configuration files that upon upgrade the admin said to retain the old version of the file. Therefore the incoming new file was placed with a ".dpkg-new" or ".dpkg-dist" name. If one were installing a new pristine system then these are the new versions of the file that would have been installed. Since we want to leave the upgraded system looking as closely to a newly installed system as possible then we need to install these versions of these files. This usually means merging local changes into the new version of the file and then replacing the current file with the merged file.
Example. In a previous version of the sudo package the sudoers file
did not include the secure_path option. Then the binary executable
changed to require it and the package maintainer's version started
including it. At that upgrade admins were presented with the option
to keep their previous sudoers file or switch to the new sudoers file.
I daresay that many admins said to keep their existing file. And were
snagged by now missing the secure_path option! The result was that
when they ran sudo that they no longer had /usr/sbin:/sbin in PATH.
The fix for this was to merge in their local changes into the new
version of the sudoers configuration file.
That example is a good reason why nice packages with configuration
files provide a way to add local configuration in a different file
that is not tracked. A file specifically for the local admin to
modify. That way it will not need to be merged in any future package
upgrade. In this case sudo includes all files in /etc/sudoers.d/*
making that the best place for local configuration. I suggest using a
name with "local" in the name so that we can easily tell that it is a
local downstream configuration. /etc/sudoers.d/sudoers-local for
example. Then that file will never need merging. The nicer packages
all make this type of configuration available. Use it.
The /etc directory should be cleaned of all '.dpkg-' files before upgrading. And always checked for required merges after upgrading.
Clean "rc" Packages
Debian's dpkg package manager has an interesting and useful feature that removing a package leaves the /etc configuration files behind. This allows installing the package again and immediately having the same configuration files as before. This means that if testing two incompatible programs installing one may push out the other one and allows re-installing it again and immediately being back to the same state as before. Those /etc configuration files that are managed by the dpkg package manager have the special name "conffile". This is useful. But in order for the package manager to continue to manage those conffiles even after the package has been removed it means that removing the package does not actually remove EVERYTHING about the package. The package remains installed but in the removed-config state or "rc" state where the package has been removed but conffiles remain installed.
You can list packages on the system that are in this "rc" state.
dpkg -l | grep ^rc
Another way to list these packages using the grep-status command.
Install the dctrl-tools package to get this useful utility.
grep-status -sPackage -n -FStatus 'deinstall ok config-files'
Especially if a system has been operating for a long time and upgraded there may be quite a few packages in the rc state. These create obstacles for the upgrade process because in addition to the current packages and current package dependencies these older packages and older package dependencies also must all be considered when performing an upgrade.
These packages need to be purged in order to fully remove them from the system. Debian's dpkg differentiates the actions "remove" and "purge" for this. With dpkg "remove" will remove the non-conffile files, the files not in /etc, but will leave the conffiles in /etc on the system and will leave the package otherwise listed as installed in order to manage those conffiles. The "purge" action will remove those conffiles and remove the package from the listing. And of course that also opens up any packages in the Depends relation for removal too.
grep-status -sPackage -n -FStatus 'deinstall ok config-files'
Review the list of removed-config packages and if okay then purge them all using this command.
dpkg --purge $(grep-status -sPackage -n -FStatus 'deinstall ok config-files')
After these have been purged then other packages may now become candidates for automremoval. If a package was installed as a dependency of another package then it will be an automatically installed package. Automatically installed packages that no longer have a dependency keeping them installed are now candidates for autoremoval. And when we autoremove we want to use the --purge option so that packages are purged and we avoid creating more packages in the "rc" state and needing to clean those out again.
apt-get autoremove --purge
Example.
root@frontend1:~# apt-get autoremove --purge
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
fonts-dejavu* fonts-dejavu-extra* g++-5* gcc-6-base* imagemagick-6.q16*
libapache2-mod-php7.0* libassuan-dev* libconfig-file-perl* libgpg-error-dev*
libpython3.5-minimal* libpython3.5-stdlib* libstdc++-5-dev* libxtables11*
linux-image-unsigned-4.4.0-197-generic* linux-modules-4.4.0-197-generic*
lua-lpeg* php7.0* php7.0-cli* php7.0-common* php7.0-fpm* php7.0-gd*
php7.0-json* php7.0-mysql* php7.0-opcache* php7.0-readline* python-chardet*
python-debian* python-debianbts* python-httplib2* python-pkg-resources*
python-pycurl* python-pysimplesoap* python-reportbug* python-setuptools*
python-six* python3.5* python3.5-minimal*
0 upgraded, 0 newly installed, 37 to remove and 0 not upgraded.
After this operation, 156 MB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 72633 files and directories currently installed.)
Removing fonts-dejavu (2.37-1) ...
Removing fonts-dejavu-extra (2.37-1) ...
Removing g++-5 (5.5.0-12ubuntu1) ...
Removing gcc-6-base:amd64 (6.5.0-2ubuntu1~18.04) ...
Removing imagemagick-6.q16 (8:6.9.7.4+dfsg-16ubuntu6.13) ...
Removing php7.0 (7.0.33-0ubuntu0.16.04.16) ...
Removing libapache2-mod-php7.0 (7.0.33-0ubuntu0.16.04.16) ...
apache2_invoke php7.0 prerm: No action required
Removing libassuan-dev (2.5.1-2) ...
Removing libconfig-file-perl (1.50-3) ...
Removing libgpg-error-dev (1.27-6) ...
Removing python3.5 (3.5.2-2ubuntu0~16.04.13) ...
Removing libpython3.5-stdlib:amd64 (3.5.2-2ubuntu0~16.04.13) ...
Removing python3.5-minimal (3.5.2-2ubuntu0~16.04.13) ...
Unlinking and removing bytecode for runtime python3.5
Removing libpython3.5-minimal:amd64 (3.5.2-2ubuntu0~16.04.13) ...
Removing libstdc++-5-dev:amd64 (5.5.0-12ubuntu1) ...
Removing libxtables11:amd64 (1.6.0-2ubuntu3) ...
Removing linux-image-unsigned-4.4.0-197-generic (4.4.0-197.229+8.0trisquel3) ...
/etc/kernel-img.conf:5: W: ignoring unknown parameter do_boot_enable
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-4.4.0-197-generic
Removing linux-modules-4.4.0-197-generic (4.4.0-197.229+8.0trisquel3) ...
Removing lua-lpeg:amd64 (1.0.0-2ubuntu0.18.04.1) ...
Removing php7.0-fpm (7.0.33-0ubuntu0.16.04.16) ...
apache2_invoke php7.0-fpm prerm: No action required
Removing php7.0-cli (7.0.33-0ubuntu0.16.04.16) ...
Removing php7.0-mysql (7.0.33-0ubuntu0.16.04.16) ...
Removing php7.0-json (7.0.33-0ubuntu0.16.04.16) ...
Removing php7.0-gd (7.0.33-0ubuntu0.16.04.16) ...
Removing php7.0-opcache (7.0.33-0ubuntu0.16.04.16) ...
Removing php7.0-readline (7.0.33-0ubuntu0.16.04.16) ...
Removing python-reportbug (6.6.6ubuntu1) ...
Removing python-debian (0.1.32) ...
Removing python-chardet (3.0.4-1+ubuntu16.04.1+certbot+2) ...
Removing python-debianbts (2.7.2) ...
Removing python-pysimplesoap (1.16-2) ...
Removing python-httplib2 (0.9.2+dfsg-1ubuntu0.3) ...
Removing python-setuptools (39.0.1-2) ...
Removing python-pkg-resources (39.0.1-2) ...
Removing python-pycurl (7.43.0.1-0.2) ...
Removing python-six (1.11.0-2) ...
Removing php7.0-common (7.0.33-0ubuntu0.16.04.16) ...
Processing triggers for install-info (6.5.0.dfsg.1-2) ...
Processing triggers for libc-bin (2.27-3ubuntu1.6) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for fontconfig (2.12.6-0ubuntu2) ...
Processing triggers for mime-support (3.60ubuntu1) ...
(Reading database ... 69300 files and directories currently installed.)
Purging configuration files for php7.0-mysql (7.0.33-0ubuntu0.16.04.16) ...
Purging configuration files for php7.0-readline (7.0.33-0ubuntu0.16.04.16) ...
Purging configuration files for linux-modules-4.4.0-197-generic (4.4.0-197.229+8.0trisquel3) ...
dpkg: warning: while removing linux-modules-4.4.0-197-generic, directory '/lib/modules/4.4.0-197-generic' not empty so not removed
Purging configuration files for php7.0-opcache (7.0.33-0ubuntu0.16.04.16) ...
Purging configuration files for libapache2-mod-php7.0 (7.0.33-0ubuntu0.16.04.16) ...
apache2_invoke postrm: Purging state for php7.0
dpkg: warning: while removing libapache2-mod-php7.0, directory '/etc/php/7.0/apache2/conf.d' not empty so not removed
Purging configuration files for php7.0-gd (7.0.33-0ubuntu0.16.04.16) ...
Purging configuration files for php7.0-cli (7.0.33-0ubuntu0.16.04.16) ...
dpkg: warning: while removing php7.0-cli, directory '/etc/php/7.0/cli/conf.d' not empty so not removed
Purging configuration files for php7.0-fpm (7.0.33-0ubuntu0.16.04.16) ...
apache2_invoke php7.0-fpm postrm: No action required
dpkg: warning: while removing php7.0-fpm, directory '/etc/php/7.0/fpm/conf.d' not empty so not removed
Purging configuration files for python3.5-minimal (3.5.2-2ubuntu0~16.04.13) ...
Purging configuration files for php7.0-json (7.0.33-0ubuntu0.16.04.16) ...
Purging configuration files for linux-image-unsigned-4.4.0-197-generic (4.4.0-197.229+8.0trisquel3) ...
/etc/kernel-img.conf:5: W: ignoring unknown parameter do_boot_enable
Purging configuration files for php7.0-common (7.0.33-0ubuntu0.16.04.16) ...
dpkg: warning: while removing php7.0-common, directory '/etc/php/7.0' not empty so not removed
Purging configuration files for libpython3.5-minimal:amd64 (3.5.2-2ubuntu0~16.04.13) ...
Processing triggers for systemd (237-3ubuntu10.43) ...
Processing triggers for ureadahead (0.100.0-21) ...
In addition to autoremove I find using "deborphan" to be another good tool for more cleaning. Install deborphan. Run it. Use it to do more cleaning. The deborphan tool is an old tool. Its utility has mostly been replaced by autoremove. But it is still useful and still almost always finds more packages that are good candidates for purging.
deborphan
dpkg --purge $(deborphan)
Repeat the above sequence until there are no more packages listed from deborphan. After purging some things listed by deborphan it will then open up additional packages as candidates. Keep repeating until the list is reduced to zero packages being listed as orphans. In some cases you may need to preserve one of the listed packages if it is an orphan but just the same one that you want to preserve.
Clean Packages Without an Installation Candidate
Some packages have gone away over time. They just don't exist anymore. They will never be upgraded again. These create obstacles to the system since if they don't ever get upgraded the shared libraries that they use are also stuck. These packages need to be removed in order to allow the upgrade to occur without entanglements with ancient libraries and other dependencies.
Some packages have encoded version numbers. The emacs packages are examples of ones with an included version number such as emacs21 and so on from there. These will never get upgraded and their presence can cause problems. For example emacs shares some of the files in a common package to be shared across versions. But really old versions then cause failures due to the use of newer features. These packages need to be removed in order to allow the upgrade to occur without entanglements with ancient libraries and other dependencies.
The easiest way to list these is with apt-show-versions. Install
that utility and use it to list packages without an installation
source. Most packages will show as "uptodate" and so filtering those
out will show all of the rest of the packages that are interesting.
Specifically listing out packages without an install source are the
ones we want to know about.
apt-show-versions | grep -v -e uptodate
apt-show-versions | awk '/No available version in archive/{print$1}'
That will also list a few packages that we want to keep for a bit longer. Older Linux kernels that are installed for example. Those don't get upgraded but are useful to provide a safety net for rebooting back to if needed. Therefore some intelligence and filtering needs to be mentally applied to this list. Also any locally created packages will be listed if they have been installed as a package but do not have an associated repository. Review. Purge packages that these identify and that you agree should be cleaned up from the system.
Example.
root@frontend1:~# apt-show-versions | grep -v -e uptodate
emacs24-bin-common:amd64 24.5+1-6ubuntu1.1 installed: No available version in archive
emacs24-common:all 24.5+1-6ubuntu1.1 installed: No available version in archive
emacs24-common-non-dfsg:all 24.4+1-2+8.0trisquel2 installed: No available version in archive
emacs24-el:all 24.5+1-6ubuntu1.1 installed: No available version in archive
emacs24-nox:amd64 24.5+1-6ubuntu1.1 installed: No available version in archive
gcc-4.9-base:amd64 4.9.3-13ubuntu2 installed: No available version in archive
initscripts:amd64 2.88dsf-59.3ubuntu2 installed: No available version in archive
insserv:amd64 1.14.0-5ubuntu3 installed: No available version in archive
libcgmanager0:amd64 0.39-2ubuntu5 installed: No available version in archive
libreadline6:amd64 6.3-8ubuntu2 installed: No available version in archive
libreadline6-dev:amd64 6.3-8ubuntu2 installed: No available version in archive
libtasn1-6:amd64 4.13-2+8.0trisquel1 newer than version in archive
linux-image-unsigned-4.4.0-210-generic:amd64 4.4.0-210.242+8.0trisquel4 installed: No available version in archive
linux-modules-4.4.0-210-generic:amd64 4.4.0-210.242+8.0trisquel4 installed: No available version in archive
mountall:amd64 2.54ubuntu1 installed: No available version in archive
python-gpgme:amd64 0.3-1.1 installed: No available version in archive
sysv-rc:all 2.88dsf-59.3ubuntu2 installed: No available version in archive
In this case all of those packages are okay candidates for removal. If a new system were installed then none of those would be installed. But this makes a good example of a problem that needs manually handling. Because look at libtasn1-6! It is listed as "newer than version in archive". WAT?! How can that happen?
This is an odd Trisquel problem.
root@frontend1:~# apt-show-versions | grep libtasn1-6
libtasn1-6:amd64 4.13-2+8.0trisquel1 newer than version in archive
root@frontend1:~# apt-cache policy libtasn1-6
libtasn1-6:
Installed: 4.13-2+8.0trisquel1
Candidate: 4.13-2+8.0trisquel1
Version table:
*** 4.13-2+8.0trisquel1 100
100 /var/lib/dpkg/status
4.13-2 500
500 http://mirror.fsf.org/trisquel etiona/main amd64 Packages
It looks like it went forward and then backward in the upstream Trisquel repository? This is an unusual Trisquel specfic case. Let's manually walk it back to something that matches what is in the repository.
root@frontend1:~# apt-get install libtasn1-6=4.13-2
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be DOWNGRADED:
libtasn1-6
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.
Need to get 36.2 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://mirror.fsf.org/trisquel etiona/main amd64 libtasn1-6 amd64 4.13-2 [36.2 kB]
Fetched 36.2 kB in 0s (165 kB/s)
dpkg: warning: downgrading libtasn1-6:amd64 from 4.13-2+8.0trisquel1 to 4.13-2
(Reading database ... 60111 files and directories currently installed.)
Preparing to unpack .../libtasn1-6_4.13-2_amd64.deb ...
Unpacking libtasn1-6:amd64 (4.13-2) over (4.13-2+8.0trisquel1) ...
Setting up libtasn1-6:amd64 (4.13-2) ...
Processing triggers for libc-bin (2.27-3ubuntu1.6) ...
Now that problem is handled we return to the issue of removing things that are now no longer in the repository.
root@frontend1:~# apt-show-versions | awk -F: '!/uptodate/{print$1}'
emacs24-bin-common
emacs24-common
emacs24-common-non-dfsg
emacs24-el
emacs24-nox
gcc-4.9-base
initscripts
insserv
libcgmanager0
libreadline6
libreadline6-dev
linux-image-unsigned-4.4.0-210-generic
linux-modules-4.4.0-210-generic
mountall
python-gpgme
sysv-rc
root@frontend1:~# apt-get purge $(apt-show-versions | awk -F: '!/uptodate/{print$1}')
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
...
root@frontend1:~# apt-show-versions | awk -F: '!/uptodate/{print$1}'
root@frontend1:~#
And now it is clean for those types of files.
Packages with "obsolete conffiles"
This is one of the more confusing aspects of packages and managing packages. This is due to buggy maintenance of a package leaving behind "obsolete conffiles".
An obsolete file is one that was owned by a previous version of the package, removed from a later version of the package, but left behind when upgraded. These files are marked by dpkg as obsolete files in the package file listing. They are obsolete because if you were to install a pristine system with the new version of the package that conffile would not be installed. But when doing an upgrade if a previous version of the package contained the conffile and if the package does not handle the removal of it properly then this conffile is left behind as lint.
Here is some mailing list discussion for reference.
- http://lists.debian.org/debian-mentors/2011/07/msg00284.html
- http://lists.debian.org/debian-mentors/2013/05/msg00115.html
These commands will list out these files.
dpkg-query -W -f='${Conffiles}\n' | grep obsolete
dpkg-query -W -f='${Package} ${Conffiles}\n' | less +/obsolete
That latter one is most useful because it lists out both the package as well as the file that is marked as obsolete. Browse that list. Unfortunately as Paul noted in that message from 2013(!) there are still quite a few packges with these bugs in them. And here it is years and years later and we are still fighting with the exact same bugs and problems!
Browsing this list will show the files remaining behind that were included with a previous version of a package but are now no longer included. These fall into a couple of different cases that are useful to treat differently.
Moved /etc files. These files are harder to deal with because they were removed from one package and added to a different package. Simply removing them solves the problem of the obsolete conffile but creates a problem of a missing conffile for the other package that now includes it. Therefore one cannot simply remove it. Instead one must temporarily move it out of the way to a save location, then reinstall the package claiming it is obsolete, then move the file back into place.
I must talk about moved files first. Because otherwise we don't know if the file was moved and we remove it incorrectly. But in order for a file to be in both it will need to be listed as both obsolete for one package and not obsolete for another package.
apt-show-versions /etc/apt/apt.conf.d/20apt-show-versions e02489f6adb594c2609841ba130345fe
/etc/cron.daily/apt-show-versions ceccc6292ad7c2e762bd04f0bee3bf98
/etc/bash_completion.d/apt-show-versions db61f00b9796596527fb9510bacc681c obsolete
root@frontend1:~# dpkg -S /etc/bash_completion.d/apt-show-versions
apt-show-versions: /etc/bash_completion.d/apt-show-versions
That file only belongs to the apt-show-versions package. Therefore it is not moved. It has only been removed. This is the simpler case. Just remove it and reinstall the package.
root@frontend1:~# rm -f /etc/bash_completion.d/apt-show-versions
root@frontend1:~# apt-get install --reinstall apt-show-versions
If the file is owned by multiple packages however then we have to not remove it but instead move it to a saved location, reinstall the package marking it as obsolete, then move the file back to continue to be part of the other package.
Init files. With the migration to systemd (love it or hate it) many
package maintainers have summarily removed the /etc/init.d/* files
from the package without handling the removal properly. This leaves
those init scripts behind. This also includes init scripts from
"upstart" which is used in Ubuntu. If you are also drinking the
Kool-Aid of systemd then you will not be using any of these init
scripts again. In which case removing these files is always the
correct answer. A pristine installation would not include them.
However if you are counting on that init script then move it out of
the way, reinstall the package, and move the init script back. It is
something the local sysadmin would know.
Here is a simple example with an upstart init script.
xinetd
...
/etc/init/xinetd.conf 8b8bc68acf27ae09755d71b09d3ddab4 obsolete
Mitigation.
root@frontend1:~# rm -f /etc/init/xinetd.conf
root@frontend1:~# apt-get install --reinstall xinetd
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 108 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://mirror.fsf.org/trisquel etiona/main amd64 xinetd amd64 1:2.3.15.3-1 [108 kB]
Fetched 108 kB in 1s (106 kB/s)
[master 00aad4c] saving uncommitted changes in /etc prior to apt run
2 files changed, 54 deletions(-)
delete mode 100644 init/xinetd.conf
(Reading database ... 72637 files and directories currently installed.)
Preparing to unpack .../xinetd_1%3a2.3.15.3-1_amd64.deb ...
Unpacking xinetd (1:2.3.15.3-1) over (1:2.3.15.3-1) ...
Setting up xinetd (1:2.3.15.3-1) ...
Processing triggers for systemd (237-3ubuntu10.43) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Random /etc files. These are all unique one-off cases that need to be looked at individually. There are a lot of files and very likely they are not being used. But it is possible that you are using one of those files, perhaps even customized one of those files. If so then removing the file would remove part of your needed customization. Review the file. Understand if you need it. Remove it if it is not a file that you care about. Here is a simple example.
base-files
/etc/update-motd.d/50-motd-news 1008a5c52c15ce6f484da4ff9c36d294 obsolete
Mitigation.
root@frontend1:~# rm -f /etc/update-motd.d/50-motd-news
root@frontend1:~# apt-get install --reinstall base-files
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 58.4 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://mirror.fsf.org/trisquel etiona-updates/main amd64 base-files amd64 1:10.1ubuntu2.11+9.0trisquel3 [58.4 kB]
Fetched 58.4 kB in 0s (2,398 kB/s)
[master 8f9d49a] saving uncommitted changes in /etc prior to apt run
2 files changed, 147 deletions(-)
delete mode 100755 update-motd.d/50-motd-news
(Reading database ... 72636 files and directories currently installed.)
Preparing to unpack .../base-files_1%3a10.1ubuntu2.11+9.0trisquel3_amd64.deb ...
Unpacking base-files (1:10.1ubuntu2.11+9.0trisquel3) over (1:10.1ubuntu2.11+9.0trisquel3) ...
Setting up base-files (1:10.1ubuntu2.11+9.0trisquel3) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for install-info (6.5.0.dfsg.1-2) ...
Then look for another file. Remove it. Reinstall. Repeat until all of the obsolete conffiles have been worked through. This manual handling is rather tedious. There may be quite a few files. I know of no better way than to simply grind through it. Fortunately progress is made quickly and we will very soon finish the list and be done with this step.
Finding out how many of these files need to be fixed.
root@frontend1:~# dpkg-query -W -f='${Package} ${Conffiles}\n' | grep -c obsolete
26
Ugh! Well let's grind through those and get done with them then.
Transition and Dummy Packages
Look for packages named as transitional or dummy packages. These are cases where the names of the package has been changed. Sometimes packages have been reorganized.
dpkg -l | grep -e transition -e dummy
Review the list. Remove the transitional and dummy packages. But be aware that the grep might match on other packages that should not be removed. Also in some cases an OS release is made which is buggy and even though these packages are transitional packages other packages may depend upon them. In which case they cannot be removed without also removing the other packages that we want installed.
ii libncurses5-dev:amd64 6.1+20181013-2+deb10u2 amd64 transitional package for libncurses-dev
apt-get purge libncurses5-dev
Init Files Without LSB Headers
This part is of historical interest on systemd systems. On non-systemd systems however this is still an important step.
Look for files in /etc/init.d/ that are not part of any package but
may not have LSB headers. If not then the upgrade to insserv will
fail. Find those and enumerate them first.
dpkg -S $(find /etc/init.d -type f -perm /a+x) 2>&1 >/dev/null | awk '/not found/{print$2}'
Each of those files are probably some locally installed file that is not part of any package. Ensure that they contain LSB headers so that the "insserv" program can work with them without an error.
Completion of Cleaning
All of the above steps are important preparation for an upgrade. Whew! If you made it here then you have made it through a long preparation and cleaning process! Congratulations!
Upgrade!
This process basically follows as outlined in the release notes. Read them again. Then proceed!
However a generic upgrade process can be outlined and most of the time this is the process we use.
Release the Hold on Held Packages
List any held packages.
apt-mark showhold
That will list any packages that are explicitly held to prevent upgrades. Examine that list. Can they be released? At the least be aware of these. At the most unhold them and let them upgrade.
For an example the mercurial package is currently held due to incompatible changes in a security patch upgrade. The issues of the security issues did not apply to our case therefore not taking the upgrade was not a problem. But the mitigation for the issue was completely showstopping breaking. We still do not know what we are going to do about this long term.
apt-mark unhold foopackage
Check for /usr/sbin/policy-rc.d
This file controls starting of services by packages. Notably it does NOT control starting of packages at boot time by init services. It's a specific restriction targeting use of packages and handling of upgrades in a chroot container. But this file has been found accidentally existing in systems outside of containers. Since it does not control boot time init actions it was apparently not noticed when the system was provisioned and delivered.
ls -ld /usr/sbin/policy-rc.d
Remove Stale Preferences Configuration
Remove and/or update any stale preferences files that may have been installed and configured.
ls -l /etc/apt/preferences /etc/apt/preferences.d/*
Silence apt-listchanges
If the apt-listchanges package is installed then it is very noisy during a full system release upgrade. And what is one going to do? If one were installing a pristine system then we would get the new versions of the packages. This is regardless of whether apt-listchanges would warn about possible problems with packages. If it is installation time of a pristine system then these will get installed regardless. Therefore if apt-listchanges is installed it is better to purge it for the upgrade. If you want to install it back again after the upgrade then that at least avoids the problems it causes during the upgrade.
apt-get purge apt-listchanges
Free Apt Cache Space
If the system hasn't been getting cleaned regularly then there may be
gigabytes and gigabytes of old deb package files hanging around in
/var/cache/apt/archives directory. Since we are upgrading we will
be leaving those old versions behind never to be used again. Remove
them to free up space which can then be used for new packages.
apt-get clean
Non-Interactive Upgrades
The way I handle upgrades is a little different. I prefer to do as much of the upgrade non-interactively as possible. It's really deferring the interactive actions. Then interactively handling the parts that needed interaction after the non-interactive part of the upgrade has completed.
export DEBIAN_FRONTEND=noninteractive
These variables control the action of dpkg. It selects the non-interactive frontend. This is the part that asks the user whether they want to keep the old file or install the new file or show differences.
export DEBCONF_ADMIN_EMAIL=""
When noninteractive is selected dpkg will then decide to email the admin some information instead of presenting it interactively on the screen. Since I am upgrading many systems this would all be repeat information every time it was done. After I have seen the information once I don't need to see it every time after then.
export UCF_FORCE_CONFFNEW=1
The ucf program similar to dpkg manages conffiles and will ask if the new file should be used or the old file. This pre-selects using the new conffile. We will do the same for dpkg but dpkg does not use environment variables for this but command line options instead.
export UCF_FORCE_CONFFMISS=1
If the conffile has been removed do not treat this as a local modification to be preserved but install the package maintainer's version of the file the same as if it were a pristine installation. (Treating removal as a local modification was successfully argued by a pedantic Debian user. It wasn't always that way.)
Update /etc/apt/sources.list Files
Review the apt sources.list files. Sometimes configuration files will
be found in /etc/apt/sources.list.d that were not known about.
These most often contain 3rd party repositories.
ls -l /etc/apt/sources.list.d /etc/apt/sources.list
What to do about 3rd party repositories? It is best to never use 3rd party repositories. And if you do then it is best to never upgrade the system. Instead create a new system, install new stuff there, and then throw away the old system. Because 3rd parties most often do not handle upgrades well. They expect pristine installation only.
My advice if 3rd party software has been installed is to purge it
before the upgrade. Then if it is still required install it again
after the upgrade. That's safest. This was probably already found
during the apt-show-versions part of the cleaning. But maybe not.
Remove the 3rd party sources.list entries and then look at what is
left. Purge off those packages.
apt-show-versions | grep -v -e uptodate
If everything is ready for the upgrade then change the release name in the sources.list file to the new release. Update to download the new Packages index files.
sed --in-place 's/etiona/nabia/g' /etc/apt/sources.list
apt-get update
Download All Packages Before Upgrade
Download all of the packages first to the local machine to remove the dependency upon the network needing to be alive during the installation. If network-manager is in use it has a notorious reputation for dropping the network during the upgrade and not bringing it back online again. Unfortunately network-manager is now a default and pulled in by gnome during some upgrades. In any case it is much safer to download every package to the cache before starting the upgrade. The -d option downloads but does not install.
apt-get upgrade -d -q -y
apt-get upgrade --with-new-pkgs -d -q -y
apt-get dist-upgrade -d -q -y
Upgrade. This time with feeling!
All of the above has been the preparation. Now we are finally ready to perform the actual upgrade. This is the final place where it is critically important to have read the release notes and to be aware of any specific upgrade needs. Most often needed for udev and kernel related upgrades.
At this point it is possible to run Ubuntu's do-release-upgrade
command and perform an interactive upgrade. That is what the upstream
documentation will recommend at this point. If you want to do an
interactive upgrade then this is an okay thing to do. However I don't
do this myself because it will run for unpredictable length of time
grinding through the upgrade and then will stop at unpredictable times
to ask an interactive question. This cycle will repeat many times.
I prefer to upgrade as non-interactively as possible. Batchmode. Then react to things needing interactive action afterward. These are the commands. I will repeat the env variables again to reinforce their association though the rationale was explained above.
export DEBIAN_FRONTEND=noninteractive DEBCONF_ADMIN_EMAIL="" UCF_FORCE_CONFFNEW=1 UCF_FORCE_CONFFMISS=1
apt-get -o DPkg::Options::=--force-confnew -o DPkg::Options::=--force-confmiss upgrade -q -y < /dev/null
apt-get -o DPkg::Options::=--force-confnew -o DPkg::Options::=--force-confmiss upgrade --with-new-pkgs -q -y < /dev/null
apt-get -o DPkg::Options::=--force-confnew -o DPkg::Options::=--force-confmiss dist-upgrade -q
We pass in the dpkg options to force confnew so that if a conffile has been removed, which is almost certainly an accident, then it will be installed using the package maintainer's version. This is always what we want. (It's only the other way by default because of a pedantic user who successfully argued that removing a required conffile constituted a deliberate local admin change and must by Debian Policy be preserved. I disagree strongly.)
We pass in the dpkg options to select the new conffile to always replace the old conffile. The old conffile will be renamed to a name with ".dpkg-old" or ".ucf-old" suffix. After the upgrade we will look for those files and perform any merges needed.
The upgrade first uses "upgrade" which is a one-for-one package upgrade. No new package names are installed. No old package names are removed. But if packages can be upgraded in place then they are upgraded in place. This handles the majority of packages using an algorithm that cannot be in error.
Then use upgrade --with-new-pkgs which allows new package names to
be installed due to being pulled in by package dependencies. This
handles most of the rest of the upgrade topology. No packages are
allowed to be removed making this another safe upgrade algorithm.
Then use dist-upgrade to perform the final bits of the upgrade that
require removing packages. This is not a safe action and normally
requires careful review. But what's a person to do? We are most of
the way there. Therefore we can only do our best and barge forward.
I removed the -y yes option from the recipe above to require the admin
to review and confirm the action. But honestly in my own scripts I
keep the -y yes option there and barge forward.
When Things Don't Go Perfectly
Unfortunately sometimes there are errors. These are always package bugs. But they happen just the same. In the upgrade from Trisqual 9 to 10 I experienced an abort. Usually the best thing is to get as much of the collection configured as possible. Which simply means restarting the configuration step.
export DEBIAN_FRONTEND=noninteractive DEBCONF_ADMIN_EMAIL="" UCF_FORCE_CONFFNEW=1 UCF_FORCE_CONFFMISS=1
dpkg --force-confnew --force-confmiss --configure -a
If an error in a postinst script causes dpkg to abort the installation then this command will restart dpkg configuring all unconfigured packages, with the same options as before to select the package maintainer's version of conffiles and install missing conffiles. Very often the bug that aborted the upgrade won't be seen on a retry. This is often due to a missing dependencies where a package depends upon another package, does not state it, and the dependency does not happen to be ready when it is needed. In the next retry it might be available due to having unpacked and configured more packages before getting to it again. Seeing an error on cryptsetup I restarted and did not see that error again therefore it was one of those types.
Example. I have seen fail2ban have problems during the upgrade. It is possible that it will take "forever" to perform some command. I am sure that if there were visibility into what it is doing that it would make sense that it is doing some exponential time something. But that is not an uncommon package that will get stuck, will need to be interrupted, will need to be forcibly dragged kicking and screaming through the upgrade. Perhaps by removing it (not purging, keep the config files), completing the upgrade without it, then installing it again.
Also the /var/lib/fail2ban/fail2ban.sqlite3 file has been a problem.
It has become gigabytes in size with seemingly nothing reducing the
size of it. Purging fail2ban, removing that file, re-installing
fail2ban, verifying the new configuration, is sometimes needed at
major OS upgrade.
Post Upgrade Clean-Up
WARNING! Do not reboot immediately! There are a few items that need to be fixed up as soon as possible! Definitely required before rebooting.
Since we have instructed dpkg to install the new conffiles and save the locally modifed ones to ".dpkg-old" everywhere the first thing we need to do is to merge in changes. And specifically to the most critical services. Critical services such as ssh which we are undoubtedly using for the upgrade.
ls -l /etc/ssh/sshd_config*
-rw-r--r-- 1 root root 3207 Dec 29 2021 /etc/ssh/sshd_config
-rw-r--r-- 1 root root 3207 Dec 29 2021 /etc/ssh/sshd_config.dpkg-old
emacs /etc/ssh/sshd_config*
...merge local customizations from the old file to the new...
service ssh restart || systemctl restart ssh.service
Examine the old sshd_config.dpkg-old file. Merge in any local
changes from there to the new sshd_config file. Restart ssh. Verify
that login using it is still possible by logging in from a different
terminal. This is not as scary as I make it sound now. The sshd
defaults are back to something less scary. For a while it was
PermitRootLogin=no which might lock you out from logging in but now
the default is PermitRootLogin=prohibit-password which allows ssh
keys. That's standard operating procedure for most systems. So not
so critical now as it was for at least one release. In any case there
likely will be other configuration needed to be merged.
Examine the syslog for sshd messages. Over time this command has changed supported options. Here is an example where after an upgrade the old version of the file contained now deprecated options. The merge required removing these from the file.
Jun 24 17:06:10 frontend1 sshd[32160]: rexec line 16: Deprecated option UsePrivilegeSeparation
Jun 24 17:06:10 frontend1 sshd[32160]: rexec line 19: Deprecated option KeyRegenerationInterval
Jun 24 17:06:10 frontend1 sshd[32160]: rexec line 20: Deprecated option ServerKeyBits
Jun 24 17:06:10 frontend1 sshd[32160]: rexec line 31: Deprecated option RSAAuthentication
Jun 24 17:06:10 frontend1 sshd[32160]: rexec line 38: Deprecated option RhostsRSAAuthentication
A specific item that is needed for the FSF VMs for an upgrade from Trisquel 9 to Trisquel 10 is this package cryptsetup-initramfs which needs to be installed.
cryptsetup-initramfs
Originally the required files were in cryptsetup but that package is
now split and the new version does not "Depends" upon the
cryptsetup-initramfs package likely leaving the system upgraded
without it installed. The new package cryptsetup-initramfs is a
"Recommends" only and isn't guaranteed to get installed. It's needed
with the current VM setup (isn't needed on the older VMs) in order to
boot. If upgrading one must ensure that cryptsetup-initramfs gets
installed.
And less important is that for whatever reason the kernel symlinks did not track to the new kernel package. That's quite odd. I don't know why. These links.
lrwxrwxrwx 1 root root 34 Dec 7 2021 initrd.img -> boot/initrd.img-4.15.0-161-generic
lrwxrwxrwx 1 root root 31 Dec 7 2021 vmlinuz -> boot/vmlinuz-4.15.0-161-generic
They were still pointing to the old, in this case version 4, kernels. Ian ran this manually to fix them up.
linux-update-symlinks install 5.4.0-121-generic
/boot/vmlinuz-5.4.0-121-generic
lrwxrwxrwx 1 root root 33 Jul 8 18:34 initrd.img -> boot/initrd.img-5.4.0-121-generic
lrwxrwxrwx 1 root root 30 Jul 8 18:34 vmlinuz -> boot/vmlinuz-5.4.0-121-generic
I'll watch for the next kernel upgrade and verify that they update at that time or not. The current config file says this.
root@frontend1:~# cat /etc/kernel-img.conf
do_symlinks = yes
do_initrd = Yes
silent_modules=yes
clobber_modules=yes
link_in_boot = no
All of the Rest
find /etc \( -name '*.dpkg-*' -o -name '*.ucf-*' \) -print
Review and merge each of those files. Some of those services will be non-functional until that step is done.
The /etc/skel files are not upgraded automatically. Since we do not
customize /etc/profile and simply use the base-files copy directly
we can simply copy it into place. Review the other files there for
upgrades too.
diff -u /usr/share/base-files/profile /etc/profile
cp /usr/share/base-files/profile /etc/profile
File Bug Reports
Repeat all of the same pre-upgrade cleaning all over again. We have upgraded almost every package on the system. There will be new conffiles that need to be merged. There will be ".dpkg-" files to be handled. There will be packages no in the rc state. There will be new transition packages installed needing removal. Go back to the start of the cleaning and repeat all of the cleaning steps again.
Along the way we will find many packages that did not handle the upgrade correctly. They will have obsolete conffiles. They will have other problems. These are all upgrade bugs. File an appropriate bug report for the problems found.
The adequate package.
If you are a "go-getter" and want to help the project out then the "adequate" package is there to make finding these problems easier. This is something to install before going through other installation and upgrades. The adequate package will examine each install and upgrade and run checks against the package. It will complain, interactively requiring an OK to continue, about problems in the packages it finds. These might be obsolete conffiles, might be dangling symlinks, might be unresolved library symbols, or might be other things. These are all packaging bugs and in a perfect world a bug report would get filed for each of them and each of these bugs would get fixed. Because adequate is interactive this is not appropriate for fully automated installs and upgrades. But it is useful for interactive upgrades to find and notify of packaging problems.
Reboot
This is the final step. Hold your breath. Reboot. The system needs to boot in order to run the newly installed kernel. After rebooting run the system regression test suite. Any failures? Fix failures.
This first reboot is best done with one of the FSF admins around as a guardian angel. Because if it needs rescuing then it can only be done from the KVM side of things. That is not accessible to the mere mortals out in the field. Think about their time. Don't do this late on a Friday before a long weekend! Coordinate a good time during daylight hours in Boston when they can help if needed.
Savannah external test suite.
- https://git.savannah.nongnu.org/cgit/administration/savannah-tests.git
Congratulations!
If you have made it to here then you have completed the full upgrade process from one end to the other.