Klaus on Tilde Town

Safely updating to Kernel 6.6.9 in Devuan Ceres

Devuan Logo ripped in half

Happy new year, fediverse! It seems that almost on cue, following my first edition of the State of the Distro, my weapon of choice distro Devuan Ceres decided to break on me. Oh well, it happens. I had left on vacation for a few days and had not touched my computer, so as soon as I was back, I naturally wanted to update it to the cutting edge again, and my commands went to it almost by muscle memory.

This apt upgrade, however, did not go well. In particular, it seemed that Linux kernel 6.6.9 was choking the installation, to the point that apt wouldn't finish its job because it couldn't set the new kernel's package. I kept getting error messages that went kind of like this:

dpkg: error processing package linux-image-6.6.9-amd64 (--configure):
installed linux-image-6.6.9-amd64 package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of linux-image-amd64:
linux-image-amd64 depends on linux-image-6.6.9-amd64 (= 6.6.9-1); however:
Package linux-image-6.6.9-amd64 is not configured yet.
dpkg: error processing package linux-image-amd64 (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of linux-headers-6.6.9-amd64:
linux-headers-6.6.9-amd64 depends on linux-image-6.6.9-amd64 (= 6.6.9-1) | linux-image-6.6.9-amd64-unsigned (= 6.6.9-1); however:
Package linux-image-6.6.9-amd64 is not configured yet.
Package linux-image-6.6.9-amd64-unsigned is not installed.
...

That sounded strange at first. What would prevent the kernel from being set-up like this? Header dependencies?

I scratched my head for a while and eventually gave up. I headed straight to #devuan in Libera and asked around. First comment that came around was: is your installation usrmerged?

Say what?

Yup, usrmerged. Apparently, the Debian devs had been pushing for a move of most core utilities away from the root level of the filesystem into /usr/ instead, and for programs that queried things in /bin or /lib, they would leave a symbolic link pointing to the directory one level down instead. I wasn't aware that this was even going on, but it seems that from this kernel and on, it's now the rule. This explained why I got so many firmware errors, and some kernel modules didn't load either:

update-initramfs: Generating /boot/initrd.img-6.6.8-amd64
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8125b-2.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8125a-3.fw for module r8169
W: Possible missing firmware /lib/firmware/rtl_nic/rtl8107e-2.fw for module r8169
...

No wonder, huh. All that firmware is now under /usr/lib/...

So, OK, that's the problem. How do it fix it, then? Manually symlink everything from /lib to /usr/lib? While you could indeed do that (and I tried for a few easy binaries such as modprobe), I was informed that the Devuan repos include a convenient package from Debian itself designed to aid exactly on this very process. It's aptly named usrmerge, and once you install it, the post-install hook will attempt to do that merging (i.e. symlink everything from root to /usr) by themselves.

Sounds good, right? Except that when I installed that package, it also failed and I was left back to square 1. At that point I decided to take a closer look into the error messages log and figured out what was going on.

The post-transaction hook of usrmerge executes a Perl script placed in /usr/lib/usrmerge/convert-usrmerge whose function is to check and automate all of the moving of libraries and executables to the right final location, and adding a symlink in the original place. The one critical caveat, though: that script does not do merging, and will stop if the same file (not link) already exists in both /lib and /usr/lib. This is what happened to me, but the fix was simple, thankfully.

Taking a SHA256 hash for these duplicated files in both / and /usr locations revealed what I suspected: they were the same. Meaning that you could safely remove the ones under / and the script would fill the space in with a link! After that, the script broke again, in another duplicated file that again was the same in both locations. I repeated the previous steps for all of those duplicates and in the end, it was all clear!

# /usr/lib/usrmerge/convert-usrmerge
The system has been successfully converted.

Could that have been it? Absolutely. Case in point, I ran another apt upgrade just after it and it ran smoothly, kernel 6.6.9 installed and not a single error thrown. After a reboot, the new kernel loaded smoothly, all my drivers were loaded and my desktop was restored to normality. We can keep rolling again!

Conclusion

In summary, if you're stuck in Devuan Ceres unable to update to kernel 6.6.9, this is the workflow I used to solve it:

  1. Install usrmerge.
  2. Watch the installation fail (yes, it will fail. That's OK).
  3. Run /usr/lib/usrmerge/convert-usrmerge manually (as root) and read the error message to find out the problematic files that are duplicated between /lib and /usr/lib.
  4. Compare the two files with sha256sum or similar hash function.
  5. If they are the same, rm the duplicate from the /lib/ prefix (not /usr/lib).
  6. Re-run convert-usrmerge and watch it fail again, indicating another duplicated file found.
  7. Repeat steps 4 to 6 until everything is clean and you have no errors, just the message "The system has been successfully converted."
  8. Update your system with apt normally. If the installation of the kernel runs smoothly, reboot afterwards.

If you're "installing" Devuan Ceres from a stable release, you may also have to do these steps before upgrading it to ensure that your entire update goes well.

All things considered, there's a reason why the distribution is labelled as Unstable. These small things happen but thankfully we can work around them, and I am yet to suffer an irrecoverable error on running this OS. I guess that doing these small fixes - and learning from them - is one of the charms of using Linux ;)

Did kernel 6.6.9 break your Devuan install or in another distro? What did you do to solve it? Let me know on Mastodon

This post is number #50 of my #100DaysToOffload project. Finally, the 50% mark!

Last updated on 01/09/24