epposan
Posts: 9
Joined: Sun Aug 16, 2015 6:02 am
Location: Maastricht, Netherlands

upgrade of kernel and/or bootloader going wrong

Tue Aug 27, 2019 12:23 pm

I am trying to troubleshoot a problem with upgrading the kernel and/or bootloader on a Raspberry Pi 2 Model B

I remotely manage several of these older (headless) raspberry-pi's as OpenVPN servers for several years now. They still are up to their task.

However, the upgrade on ONE of these pi's is repeatedly problematic: it somehow gets misconfigured after almost every raspberrypi-kernel or -bootloader update.

Since the problem until now happened using Stretch, I was hoping that a fresh install of Buster on another sd card would fix it, but unfortunately, already the first (remote) upgrade after the fresh install failed similarly.

There were NO related errors/warnings during the upgrade, and /var/log/apt/term.log shows NO evidence of any problems

The upgrade was supposed to bring the kernel from 4.19.58 to 4.19.66, but after the necessary reboot, uname -a still returns 4.19.58 This preventing the openvpn server from starting up properly, and conseuently I have to go through several hoops to get access to the problem pi over ssl (the ssl server fortunately still starts up)

The problem is NOT related to this specific version of the kernel/bootloader: it happened several times before with previous versions as well, however only on ONE specific raspberry pi

Steps taken so far:

Checked /lib/modules: the (new) 4.19.66 subfolders are present

Compared md5sum of the following files/folders with a similar pi:

/boot/*
/boot/overlays/*
/lib/modules/*
/var/cache/apt/archives/raspberrypi-*
/var/cache/apt/archives/libraspberrypi-*


Ran debsums on libraspberrypi-(bin dev) and raspberrypi-(bootloader kernel sys-mods net-mods)

However, no differences show up.

The only difference found (apart from uname -a) is, when I run

sudo /opt/vc/bin/vcgencmd version

The problem pi returns:
Jul 9 2019 14:40:53
Copyright (c) 2012 Broadcom
version 6c3fe3f096a93de3b34252ad98cdccadeb534be2 (clean) (release) (start)

The working pi shows
Aug 15 2019 12:06:42
Copyright (c) 2012 Broadcom
version 0e6daa5106dd4164474616408e0dc24f997ffcf3 (clean) (release) (start)

Could it be a hardware problem? It happens on both on Stretch and Buster, and I have used another sd card for the Buster installation. However, as far as I know, all 'code' comes from the sd card and nothing is written to the board...

Any hints on how to proceed would be welcome.

epoch1970
Posts: 3655
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: upgrade of kernel and/or bootloader going wrong

Tue Aug 27, 2019 3:24 pm

Did you try switching PSUs?

There were multiple revisions of Pi2, and OTP bits might vary from one Pi to another.
No idea how that could affect the version of vcgencmd.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

epposan
Posts: 9
Joined: Sun Aug 16, 2015 6:02 am
Location: Maastricht, Netherlands

Re: upgrade of kernel and/or bootloader going wrong

Tue Aug 27, 2019 4:51 pm

The remote pi's all have their own small UPS that would shutdown the system gracefully if there would be PS or Mains issues.

This particular pi is located somehwere in France, more than 800 km from my place, so I took some extra measures to prevent issues caused by power glitches.

Nevertheless, I will follow your advise and replace the PS (and in the process the Pi and the UPS, to be able to troubleshoot the issue locally).

From what I have been able to determine up to this point is, that all files (that I know of) related to kernel and bootloader are correct for the 4.19.66 version, nevertheless the system shows 4.19.58 when running uname -a

I have been in this situation before, and have managed to get everything working again, usually by reinstalling the raspberrypi-(kernel bootloader) pkgs, however, this time I wanted to ask the community for some guidance.

I have a list of checks that I currently perform after any update of kernel and/or bootloader but before the first reboot thereafter, but the checklist still is missing something crucial, since I still keep running into this problem.

epoch1970
Posts: 3655
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: upgrade of kernel and/or bootloader going wrong

Tue Aug 27, 2019 5:02 pm

If uname differs it means the kernel.img file is different.
Yet you wrote above nothing in /boot/ differed from one machine to another?
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

User avatar
rpdom
Posts: 15177
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: upgrade of kernel and/or bootloader going wrong

Tue Aug 27, 2019 5:15 pm

Did you look at the checksums and sizes of the files in /boot before and after the upgrade?

Is the /boot partition that is mounted when the Pi is running the same one it boots from?

epposan
Posts: 9
Joined: Sun Aug 16, 2015 6:02 am
Location: Maastricht, Netherlands

Re: upgrade of kernel and/or bootloader going wrong

Tue Aug 27, 2019 6:08 pm

Ok. I must admit that the checksum statement is incorrect at this point in time.

I am sure that I checked the md5sums before the reboot (otherwise I would not have dared to reboot, as I cannot easily access it due to distance), but after the reboot I lost contact with the pi (because openvpn did not startup correctly)

if I run dpkg --list | grep raspberrypi on both pi's, they both report the same pkg versions:

ii libraspberrypi-bin 1.20190819-1
ii libraspberrypi-dev 1.20190819-1
ii libraspberrypi-doc 1.20190819-1
ii libraspberrypi0 1.20190819-1
ii raspberrypi-artwork 20150921
ii raspberrypi-bootloader 1.20190819-1
ii raspberrypi-kernel 1.20190819-1
ii raspberrypi-net-mods 1.3.0
ii raspberrypi-sys-mods 20190730
ii raspberrypi-ui-mods 1.20190717

However, when I now run debsums, both the kernel and bootloader pkgs show changed files, so I will need to reinstall the pkgs again.

Any idea what additional checks I could perform to make sure that the files after reboot will be the same as during the installation?

Possibly it has to do with the diversion to /usr/share/rpikernelhack/...

epposan
Posts: 9
Joined: Sun Aug 16, 2015 6:02 am
Location: Maastricht, Netherlands

Re: upgrade of kernel and/or bootloader going wrong

Tue Aug 27, 2019 6:58 pm

As far as the /boot partition is concerned: I did not make any changes to the system. but I will explicitly check before and after the next install.

All the Pi's that I manage are setup as similar as possible, but only one is showing this issue.

My problem is, that up to now, I have not found a way to predict that the upgrade will go wrong.

epposan
Posts: 9
Joined: Sun Aug 16, 2015 6:02 am
Location: Maastricht, Netherlands

Re: upgrade of kernel and/or bootloader going wrong

Tue Aug 27, 2019 7:48 pm

I have performed a re-install of raspberrypi-(kernel bootloader) and this time it worked.

I keep hoping that someone has a bright idea on: how to verify that after an apt-get upgrade of kernel and/or bootloader, but before the next reboot, that the system is setup properly to run the expected version of kernel and/or bootloader after the reboot.

Currently I check the following before the reboot:

debsums -c raspberrypi-kernel
debsums -c raspberrypi-bootloader
check dates and md5sums in /boot
check dates and md5sums in /boot/overlays
check dates and md5sums in /var/cache/apt/archives/raspberrypi-*
check for errors in /var/log/apt/term.log

Thanks for your help !

epoch1970
Posts: 3655
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: upgrade of kernel and/or bootloader going wrong

Tue Aug 27, 2019 7:58 pm

I don’t understand what is going on.
Especially I can’t understand why a package would fail to install yet there would be no error in apt logs.

You could use the brute force method, run the upgrade commands twice without trying to check anything...
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

epposan
Posts: 9
Joined: Sun Aug 16, 2015 6:02 am
Location: Maastricht, Netherlands

Re: upgrade of kernel and/or bootloader going wrong

Tue Aug 27, 2019 8:05 pm

Well, neither could I, that's why I ended up here to ask the community.

Next time, I will try your suggestion to install twice before rebooting.

epposan
Posts: 9
Joined: Sun Aug 16, 2015 6:02 am
Location: Maastricht, Netherlands

Re: upgrade of kernel and/or bootloader going wrong

Wed Aug 28, 2019 5:54 am

For the record:
I have posted the (adapted) question at Raspberrypi StackExchange
https://raspberrypi.stackexchange.com/q ... nal-reboot

User avatar
DougieLawson
Posts: 36098
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: upgrade of kernel and/or bootloader going wrong

Wed Aug 28, 2019 7:21 am

You are over thinking a simple, trivial and well tested upgrade.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

epposan
Posts: 9
Joined: Sun Aug 16, 2015 6:02 am
Location: Maastricht, Netherlands

Re: upgrade of kernel and/or bootloader going wrong

Wed Aug 28, 2019 7:45 am

I fully agree with you !

I maintain several local and remote Pi2 and Pi3 installations and have performed many kernel upgrades over time without any hick-ups

However one particular remote Pi2 has been giving kernel upgrade problems several times lately, and after each failing upgrade I try to come up with additional checks to hopefully find the real cause of the problem.

If the Pi would be local, I would not even bother to ask any questions and simply repeat/reinstall the upgrade (or even use rpi-update), however this Pi is at a remote location, where I need to get access through another pc using Teamviewer and/or VNC with access from the internet enabled (which I try to limit as much as possible)

Also, being fairly familiar with electronics and programming, I always have tried to find 'bugs' instead of working around them.

I hope you can bear with me ...

Ernst
Posts: 1231
Joined: Sat Feb 04, 2017 9:39 am
Location: Germany

Re: upgrade of kernel and/or bootloader going wrong

Wed Aug 28, 2019 9:02 am

epposan wrote:
Wed Aug 28, 2019 7:45 am
However one particular remote Pi2 has been giving kernel upgrade problems several times lately, and after each failing upgrade I try to come up with additional checks to hopefully find the real cause of the problem.
Have you changed the memory card ?
The road to insanity is paved with static ip addresses

epposan
Posts: 9
Joined: Sun Aug 16, 2015 6:02 am
Location: Maastricht, Netherlands

Re: upgrade of kernel and/or bootloader going wrong

Wed Aug 28, 2019 10:26 am

Yes, I have changed the sd card.

Return to “Troubleshooting”