Raspberry Pi Forums

Posted: **Tue Jan 29, 2019 1:35 am**

Hello,

12 September 2019: an updated, v1.3.0 release of the bootable image is now available, which adds RPi4 compatibility, uses a Raspbian Buster host OS, and runs under the official 64-bit kernel; please see here for further details.

as maintainer of the gentoo-on-rpi3-64bit image, I not infrequently receive email requests from users who:

don't want to give up on their familiar Raspbian desktop, repos, and tools;
don't want to have to build packages from source (during setup or maintenance);
would like to retain the ability to easily upgrade their Raspbian system going forward;
but still need to run a few (possibly GUI-based) 64-bit-only packages on their RPi3.

As my current Gentoo image is not such a great fit for those particular requirements (source-centric Portage pms, OpenRC init, Xfce4, etc. ^-^), over the last couple of weeks I've been putting together — as a proof-of-concept really — the sort of system I'd like to be able to point such users towards when they ask.
Note: the question here is not whether people really need the capability to run 64-bit software on their RPi3; perhaps many don't, but enough genuinely do (mongodb-3.2 users etc.) that I've chosen to run with it in what follows.

The system I ended up building looks as follows:

A 64-bit kernel (since under ARMv8a this is required to allow mixed-mode 64/32-bit userspace);
An (almost) vanilla 32-bit Raspbian Stretch with desktop as the userspace host OS (for familiarity);
A 64-bit (aarch64) Debian Stretch userspace guest OS booted inside a lightweight systemd-nspawn container (to provide an easy, safe and familiar 64-bit playground, without going all the way down the KVM wormhole ^-^).

The guest (64-bit Debian) OS is automatically started by the Raspbian host OS at boot time, but runs inside a fresh process namespace. Nevertheless, via a provided System Tools menu item (see below screenshot), users can easily open a 64-bit shell inside this container (to install new 64-bit packages, for example). Another menu item allows them to easily run 64-bit applications in the container, correctly set up so that they:

display on the Raspbian desktop;
can play audio; and
have access to the pi user's home directory.

Due to the containerization [1], such apps are however prevented from performing many harmful actions on the host (whilst still running in a fully-featured, booted OS instance with its own systemd, dbus etc.; this is not just a chroot). Package management in the guest uses the familiar apt-get, with the full Debian aarch64 repository available, and day-to-day Raspbian operation (with the exception of anything needing e.g. MMAL or OpenMAX IL) is pretty much per the stock image. WiFi, Bluetooth, I2C etc. all work, and 32-bit Raspbian apps can be installed and used just as on a regular RPi3 system.

Basically, you get 64-bit capability as and when you need it, without most of the headache ^-^

Here's a screenshot of the hybrid system in use, on an RPi3B+:

As of today, I've released this as a bootable image on GitHub, here.

You can burn the image (~925MiB compressed) to a microSD card (>=8GB), then boot your RPi3 from it directly (the root partition will be automatically resized to fill the card on first boot, per the standard Raspbian flow). Full instructions for download and use are provided on the project's GitHub page. 64-bit Firefox-ESR (as shown in the above screenshot) is pre-installed on the 64-bit guest (and may be launched from the Internet menu in Raspbian), as proof that full-scale 64-bit apps can usefully be deployed on this kind of hybrid system.

I've also posted a tutorial covering how to create such an image step-by-step, here.

hth, feedback welcome!

sakaki

Acknowledgement: The idea of this came from Crazyhead90's Raspbian-desktop-64bitkernel-pi3bplus image (64-bit kernel + 32-bit userland OS), and this post by jdonald (which suggests a chroot into a 64-bit guest from such a setup). I've just extended jdonald's idea to use the Linux containers technology leveraged by systemd-nspawn, and tied it all together in what I hope is a reasonably user-friendly way ^-^

[1] Shorthand for the trinity of namespacing, seccomp-bpf and capabilities, the common kernel facilities leveraged by "concessionaire" apps like systemd-nspawn, firejail and docker (of course, other facilities such as cgroups are also used).

Edit: fix minor typo

Posted: **Tue Jan 29, 2019 11:26 am**

This is great (as always) Sakaki, but I think there is probably an easier way to do this.

On Debian and Ubuntu you can use Multiarch. This certainly works with any recent Ubuntu on the pi. See https://wiki.ubuntu.com/ARM/RaspberryPi ... .2FAArch64 *

You should be able to do the same from armhf Debian. It should follow then that you could use armhf raspian packages and Debian arm64 packages together with a multiarch setup. The generic Debian kernels are bootable on the pi, you just need to change the fat partition so that it is mounted as /boot/firmware and use something like flash-kernel or raspi3-firmware to copy the kernel/initrd to where it is needed.

Just an idea, sorry about hijacking your thread!

*Note Ubuntu now builds official arm64 images for the pi

Posted: **Tue Jan 29, 2019 12:53 pm**

feelslikeautumn wrote: ↑
Tue Jan 29, 2019 11:26 am
This is great (as always) Sakaki, but I think there is probably an easier way to do this.

On Debian and Ubuntu you can use Multiarch. This certainly works with any recent Ubuntu on the pi. See https://wiki.ubuntu.com/ARM/RaspberryPi ... .2FAArch64 *

Yes, agreed, those users who don't mind moving away from Raspbian as their base OS, can certainly use an aarch64 distro that supports multiarch, if they wish. But then it isn't Raspbian. And of course, once you allow a distro move, then the requirement for 32-bit support also becomes, for most users to whom that would appeal, much less strong (unless you have some legacy 32-bit apps you don't want to lose, are working on an arm32 assembler project or whatever). At least that was the (maybe flawed ^-^) logic that led me to launch gentoo-on-rpi3-64bit as a 'pure' 64-bit system all those moons ago.

feelslikeautumn wrote: ↑
Tue Jan 29, 2019 11:26 am
You should be able to do the same from armhf Debian. It should follow then that you could use armhf raspian packages and Debian arm64 packages together with a multiarch setup. The generic Debian kernels are bootable on the pi, you just need to change the fat partition so that it is mounted as /boot/firmware and use something like flash-kernel or raspi3-firmware to copy the kernel/initrd to where it is needed.

This idea is sound, but converting Raspbian's existing packages to play nicely in a multilib environment is non-trivial. jdonald, out in front as always, has written a script to do this, here, which works by creating shadow copies of all affected debs. However in the absence of 'official' support upstream from RPF, maintenance of such a modified system (dealing with conflicts and such) is likely to be somewhat involved going forward, an unofficial mini-distro in itself.

So what to do? My conclusion in the end was that running a 'sidekick' 64-bit OS in a lightweight systemd-nspawn container, appropriately 'wired in' to the audio and Xserver of the host OS, would be the most natural / lightest touch way to go, which is what the bootable image provides. This way, apart from the kernel and a little orthogonal configuration, everything in the Raspbian side remains vanilla, so users should be able to keep updating their system using the official repos without problem (and the same is true for the guest OS, which is isolated at a package level from the host).

Although building such a system up from scratch (as detailed in my tutorial post here) is a little involved, those booting the raspbian-nspawn-64 image don't need to worry about any of that, since it all (should!) just work out of the box.

feelslikeautumn wrote: Just an idea, sorry about hijacking your thread!

No worries ^-^ If you get a chance to play with the image, I'd be very interested to hear what you think. It's not going to be my daily driver, but I think it addresses quite a few of the 'casual 64-bit' requests that keep popping up. And it'd be easy to create spins with different guest OSes (Arch or whatever) too (paging procount... ^-^)

Best, sakaki

Posted: **Tue Jan 29, 2019 1:36 pm**

More than happy to include them, if you would like to produce and host them

Posted: **Tue Jan 29, 2019 2:10 pm**

Thanks for the detailed reply. I hadn't realised multiarch had already been suggested. Must of missed that in the boring 64 bit threads (that I don't read anymore).

Looking at jdonald's scripts the problem seems to come from different mesa versions. Rather than hacking the control files, I wonder if recompiling the mesa package would fix things? But there comes a point when it is just easier to drop Raspbian....

Posted: **Tue Jan 29, 2019 6:02 pm**

Thanks sakaki! Any bench testing results to help the uneducated on reasons to try the 64 bit bootable image on GitHub?

Posted: **Tue Jan 29, 2019 7:40 pm**

energyi wrote: ↑
Tue Jan 29, 2019 6:02 pm
...Any bench testing results to help the uneducated on reasons to try the 64 bit bootable image on GitHub?

It took just 5 minutes to xzcat the compressed image to a 32G µSD on Raspbian, thanks to Sakaki's excellent instructions, so why not just try it for yourself?

Posted: **Tue Jan 29, 2019 8:09 pm**

energyi,

I'm wondering why the "uneducated" need more reason to try a thing than the fact that it is there, looks interesting, and might be useful?

Posted: **Wed Jan 30, 2019 12:58 am**

Well done Sakaki, brillant work on the OS and the wiki, again.
You don't happen to be visiting Cambridge by any chance?

Works booting from some tiny Lexar 32GB USB sticks.
Of course the first thing I did was install Lazarus/FPC

Old Debian version but it was a start.

Installing Ultibo via the install script requires the aarch64 3.0.2 FPC, which of course does not exist.
But scripts I can write and fix

Running 32 and 64 bit compilers at the same time?
Need get my head around that possibility now.

My benchmark at the moment is the WebGLsamples Aquarium, Gentoo64 still wins

This Debian is reporting 13.0.6 on glxinfo, I think Gentoo64 was up to 18.2.?
People forget Debian is meant to be the safe stable Linux OS, not the bleeding edge ones some of us use.

This is looking very good as a useful intermediate step of having 32 and 64bit Raspbian at the same time.
Sakaki calls herself a "unscary Gentoo girl"

I find Sakaki's skills very scary, I hope one day to understand how she does it

Any bench testing results to help the uneducated on reasons to try the 64 bit bootable image on GitHub?

Since the Pi's came out I had to learn a dozen plus computer languages, learning new ones has become fun.
I am now up to about half a dozen OS's, learning new ones are not quite fun yet, but no longer scary

The reason to do it, is to get educated, that is the whole point of Pi's

It has now come to the point of me even thinking about making my own langauge/compiler and perhaps fpga based cpu design.
Of course the more I learn the bigger the list get of things I want to do

Posted: **Wed Jan 30, 2019 6:53 am**

Great work! With this image, you get to keep all your familiar Raspbian 32-bit programs, while being able to run 64-bit Debian programs as well. This gives you the latest versions of Chromium and Firefox which aren't present in the Raspbian repositories. You get the efficiency of a 64-bit OS while sticking to a familiar, friendly OS.

Posted: **Wed Jan 30, 2019 7:20 pm**

energyi wrote: ↑
Tue Jan 29, 2019 6:02 pm
Thanks sakaki! Any bench testing results to help the uneducated on reasons to try the 64 bit bootable image on GitHub?

Well, tbh this particular image isn't so much targeted at performance improvement, as making the use of 64-bit packages possible in a 'vanilla' Raspbian environment for those who need this facility (for example, to enable the use of a modern variant of mongodb, PocketMine-MP etc., without having to change distro, even to one as self-evidently appealing as Gentoo ^-^).

That said, and subject to all the caveats that attach to the "real-world" validity of benchmarks generally, Roy Longbottom has posted (both here and on the Gentoo forums) extensive comparative 32 vs 64 bit performance analysis for the RPi3 B and B+. Of course, on the raspbian-nspawn-64 image you can download his tests and run the 32-bit ones in a standard console, and then the 64-bit ones in a 64-bit container console, so they should be easy to check for yourself ^-^. I reproduce one of his posts from the Gentoo forums below to get you started (he is comparing like-for-like 64-vs-32 bit benchmarks):

roylongbottom wrote: 64 Bit Benchmarks

Confirming Sakaki’s report, I have compiled and run the first set of 64 bit benchmarks. Full details and results are in the following, with benchmarks and source codes in the tar.gz file:

http://www.roylongbottom.org.uk/Raspber ... hmarks.htm
http://www.roylongbottom.org.uk/Rpi3-64 ... rks.tar.gz

The Classic Benchmarks are the first programs that set standards of performance for computers in the 1970s and 1980s. They are Whetstone, Dhrystone, Linpack and Livermore Loops. Improvements indicated relate to comparisons of gcc-6 64 bit versions and 32 bit compilations from gcc 4.8 via Raspbian.

Whetstone - This includes simple test loops that do not benefit from advanced instructions. There was a 40% improvement in overall performance. This was due to limited but dominant tests using such as COS and EXP functions.

Dhrystone - rated in VAX MIPS AKA DMIPS produced a 43% improvement, but this benchmark is susceptible to over optimisation.

Linpack - double and single precision versions (DP and SP), with results reported in MFLOPS. Speed improvements, over the 32 bit version, were around 1.9 times DP and 2.5 times SP. There is also a version that uses NEON intrinsic functions where, at 32 bits and 64 bits, are compiled as different varieties of vector instructions, with only a 10% improvement.

Livermore Loops - has 24 test kernels, where 64 bit performance increased between 1.02 and 2.88 times. The official average was 34% faster, at 279 MFLOPS. This is 21 times faster than the Cray 1 supercomputer, where this benchmark confirmed the original selection.

Memory tests - These measure cache and RAM speeds with results in MB/second. As could be expected, RAM speeds were generally quite similar for particular test functions.

MemSpeed - Nine tests measure speeds using floating point (FP) and integer calculations. Cache based improvements were 1.64 to 2.60 DPFP, 1.17 to 1.55 SPFP and 1.03 to 1.23 integer.

BusSpeed - this reads data via loops with 64 AND instructions, attempting to measure maximum data transfer speeds. It includes variable address increments to identify burst reading and to provide a means of estimating bus speeds. Main differences were on using L1 cache data, where average bursts speeds were 38% faster but reading all data was slower. This is surprising as the 64 bit disassembly indicate that far more registers were used, with fewer load instructions, and the same type of AND instructions.

NeonSpeed - All floating point data is single precision. The source code carries out the same calculations using normal arithmetic and more complicated NEON intrinsic functions, the latter being compiled as different types of vector instructions, with no real average 64 bit improvement. The normal SP calculations were slightly faster.

Remember, because of restrictions imposed by use of a 64-bit kernel, some things (like gpu-assisted video decoding) will be unavailable, so for for your particular use case, switching to a 64-bit-capable image may make things worse, not better.

hth, sakaki

Posted: **Wed Jan 30, 2019 7:29 pm**

procount wrote: ↑
Tue Jan 29, 2019 1:36 pm
More than happy to include them, if you would like to produce and host them

Happy to do so, at least for this Raspbian-32/Debian-64 variant!

If you don't mind though, please remind me what format you require for an externally hosted PINN image (as you've kindly been doing the conversion/hosting for Gentoo64 (née gentoo-on-rpi3-64bit) on your own infrastructure)?

best, sakaki

Posted: **Wed Jan 30, 2019 8:38 pm**

@sakaki -I'll drop you an email with details shortly.

Posted: **Thu Jan 31, 2019 7:01 pm**

sakaki, thanks for the bench test results from roylongbottom. I love this comparison for Livermore Loops: "This is 21 times faster than the Cray 1 supercomputer." And appreciate the note on restrictions.

Posted: **Fri Feb 01, 2019 2:19 am**

"This is 21 times faster than the Cray 1 supercomputer."

Yep, some of us can remember when the Cray 1 came out.
Now we all walk around with faster than Cray machines in our pockets.

https://www.raspberrypi.org/forums/view ... p?p=639030
I have those Byte mags in a box somewhere.

Weekend mission - try to break this by updating to 4.19 kernel

Posted: **Fri Feb 01, 2019 2:46 am**

Gavinmc42 wrote: ↑
Fri Feb 01, 2019 2:19 am
Weekend mission - try to break this by updating to 4.19 kernel

lol - weekend teaser: try doing so on an image with an up-to-date userland, and you might unlock something rather interesting... ^-^
S.

Posted: **Fri Feb 01, 2019 11:56 am**

and you might unlock something rather interesting

#1
#2
#3
Is that a hint from my administrator?

Too hard to break, might have to go back to normal Raspbian.
Now have another 64bit OS that just works

Wonder if those FPGA tools will install easier?
Copy the aarch64 RISC-V tools across?

I should be able to push it over the edge some how?
So far it looks like a minimal aarch64 with normal Raspbian aarch32.
Yep just another Sakaki magic trick, more wiki time.

Going to be interesting if I can get an aarch64 and aarch32 gcc compiler going.
Wow two gcc's 6.3.0 aarch64-linux-gnu , arm-linux-gnuabihf

Compile the RISC-V tools again at the same time, one 32 and one 64, that should break something

Need LLVM first, perhaps start with just the baremetal RISC-V compilers?

Posted: **Sat Feb 02, 2019 3:52 pm**

Based on feedback received, I'm going to add a service to ensure that if (regular) users are added to 32-bit Raspbian they are automatically reflected inside the 64-bit Debian container, and similarly that changes to the password for regular users in the host are automatically reflected inside the container. And I'm also going to make the whole of /home bind-mounted into the container. This should enable app and shell launching within the 64-bit container for all regular users, not just "pi".

When this is done and tested I'll re-release and also put up a version (hopefully) suitable for PINN (thanks for sending through the instructions for image conversion btw @procount).

Any other requests for changes to raspbian-nspawn-64, please post them here or email them to me (sakaki@deciban.com) by end-of-day Monday 4th Feb, when I will lock off a new version for testing. Release should then follow a week or so after that.

Thanks!

sakaki

Posted: **Sun Feb 03, 2019 9:08 am**

feelslikeautumn wrote: ↑
Tue Jan 29, 2019 11:26 am
You should be able to do the same from armhf Debian. It should follow then that you could use armhf raspian packages and Debian arm64 packages together with a multiarch setup. The generic Debian kernels are bootable on the pi, you just need to change the fat partition so that it is mounted as /boot/firmware and use something like flash-kernel or raspi3-firmware to copy the kernel/initrd to where it is needed.

From personal experience, adding Debian repositories to a Raspbian system and installing packages and upgrading from them can cause incompatibilities and can even break your Raspbian installation, giving you no other option but to reinstall Raspbian from scratch. And the generic kernel isn't even optimised for the Raspberry Pi 3 so you will face some issues while using the generic kernel on the Pi. Best bet is to use a chroot or container and an ARM64 kernel like sakaki's image does. Or, you could use a pure 64-bit system and then run Raspbian in a chroot.

Posted: **Sun Feb 03, 2019 11:02 pm**

Great work sakaki. I like how you take inspiration from various proofs of concept and carefully consider user feedback to come up with something awesome.

feelslikeautumn wrote: ↑
Tue Jan 29, 2019 2:10 pm
Looking at jdonald's scripts the problem seems to come from different mesa versions. Rather than hacking the control files, I wonder if recompiling the mesa package would fix things?

The Mesa patch version is just one minor detail. The main mechanism of that family of scripts is to hide the +rpi version suffixes and mismatching changelog files. code_exec and sakaki are correct that this does not result in a maintainable system. It's effectively only useful if a) you're setting up a multiarch demo system where in the future you never intend to upgrade or install new packages; or b) will get some 64-bit work done then later revert via the included undo.sh script.

sakaki wrote: ↑
Sat Feb 02, 2019 3:52 pm
Any other requests for changes to raspbian-nspawn-64, please post them here or email them to me (sakaki@deciban.com) by end-of-day Monday 4th Feb

1) First thing that comes to mind is that it would be nice to have a binfmt registered such that aarch64 programs executed from the default shell automatically spawn inside the container. This is inspired by some Wine+qemu guides for Raspbian that similarly try to make a more seamless experience launching guest applications.

2) Share /usr/share/applications between the host and the guest so that programs installed on the guest automatically appear in the PIXEL start menu.

3) bcm2835-unicam doesn't appear to be present in the kernel modules despite the version being 4.14.93+. Could it be included? This request is not too specific to this systemd-nspawn image. Rather, all 64-bit Pi systems could address one major shortcoming by making it possible to access the SoC camera.

Posted: **Mon Feb 04, 2019 1:54 am**

[quotebcm2835-unicam ][/quote]
Is that going to be in 4.19?

Sakakio is mostly a Gentoo girl, asking her to support another OS, cheeky

When 4.19 comes out it might be easier?
Probably only a few months away as RPT guys already have versions they and others are debugging.

Mind you if Sakaki knows how to do it she probably won't be able to resist.
And then she will document it all in her wiki

Posted: **Mon Feb 04, 2019 7:02 am**

sakaki has a GitHub repository where she provides a frequently updated 64-bit kernel for the Raspberry Pi 3. If you want to update your kernel, you should update it from that repository.

https://github.com/sakaki-/bcmrpi3-kernel

Posted: **Mon Feb 04, 2019 8:16 am**

code_exec, you might not have noticed but currently I probably the noisest in the Gentoo posts.
I'm well aware of Sakaki's wiki's

Even her B2/B3 and Gentoo forum posts, all very educational and well written, I'm learnng lots.

If I was not on the other side of the planet she might be worried I am stalking her posts and wiki's

Posted: **Mon Feb 04, 2019 12:34 pm**

jdonald wrote: ↑
Sun Feb 03, 2019 11:02 pm

sakaki wrote: ↑
Sat Feb 02, 2019 3:52 pm
Any other requests for changes to raspbian-nspawn-64, please post them here or email them to me (sakaki@deciban.com) by end-of-day Monday 4th Feb
1) First thing that comes to mind is that it would be nice to have a binfmt registered such that aarch64 programs executed from the default shell automatically spawn inside the container. This is inspired by some Wine+qemu guides for Raspbian that similarly try to make a more seamless experience launching guest applications.

2) Share /usr/share/applications between the host and the guest so that programs installed on the guest automatically appear in the PIXEL start menu.

3) bcm2835-unicam doesn't appear to be present in the kernel modules despite the version being 4.14.93+. Could it be included? This request is not too specific to this systemd-nspawn image. Rather, all 64-bit Pi systems could address one major shortcoming by making it possible to access the SoC camera.

Thanks for the feedback jdonald! To your points:

That's an interesting idea! But, how does the 32-bit shell user even get to the point of trying to invoke a 64-bit app in the first place (in order for the binfmt handler to then pick it up)? Would you propose they be navigating inside the /var/lib/machines/debian-stretch-64 folder? Or would you propose adding e.g. /var/lib/machines/debian-stretch-64/usr/bin etc. into the 32-bit PATH? Could you share some links to the Wine/QEMU guides you mentioned, so I can get a better idea? Thanks!
Applications in the 64-bit guest need a little manipulation to be launched via the host GUI. Specifically, the Exec entries in their .desktop files need to be prefixed by ds64-run or ds64-runner (an appropriate-environment systemd-run invocation, essentially). This example shows how firefox-esr (64-bit) is invoked from the host's "Internet" desktop menu on the current image. Also pixmaps used in the guest's .desktop files need to be path-visible on the host too. But, in principle something like this could be done (would be quite neat!) - perhaps via a systemd path unit watching the appropriate directories in the container's filesystem and automatically creating / removing 'shadow' .desktop files on the host. I'll have a look - maybe not for the next release though. (Of course, if the PATH were augmented as per point 1 then maybe this could all happen automagically via binfmt_misc, provided no absolute paths are in the guest's destkop files of course).
OK. I actually have two weekly 4.14 autobuild kernel projects for the RPi3. The first, bcmrpi3-kernel, is a vanilla bcmrpi3_defconfig build (so that won't change config unless upstream does), but the second (which is actually used in the raspbian-nspawn-64 image) is bcmrpi3-kernel-bis, and that contains a number of tweaks to bcmrpi3_defconfig (to enable KVM etc). No reason why the camera module couldn't be enabled in the latter (and that'll make it work for other consumers of this kernel too, such as gentoo-on-rpi3-64bit), so I'll look to do this for the next release. (I've also got a 4.19 autobuild, but that's currently a private repo; will be rolling it out once the USB issues get sorted on 64-bit).

Best,
sakaki

Posted: **Mon Feb 04, 2019 1:26 pm**

sakaki wrote: ↑
Mon Feb 04, 2019 12:34 pm
OK. I actually have two weekly 4.14 autobuild kernel projects for the RPi3. The first, bcmrpi3-kernel, is a vanilla bcmrpi3_defconfig build (so that won't change config unless upstream does), but the second (which is actually used in the raspbian-nspawn-64 image) is bcmrpi3-kernel-bis, and that contains a number of tweaks to bcmrpi3_defconfig (to enable KVM etc). No reason why the camera module couldn't be enabled in the latter (and that'll make it work for other consumers of this kernel too, such as gentoo-on-rpi3-64bit), so I'll look to do this for the next release. (I've also got a 4.19 autobuild, but that's currently a private repo; will be rolling it out once the USB issues get sorted on 64-bit).

AFAICT The USB issues will be present on ALL 64 bit kernels - it's not something that only affects 4.19. If someone would confirm that then it would be a useful data point.

I'm just creating the PRs to enable all the extra modules for camera drivers etc in bcmrpi3-defconfig.
Looking at your bcmrpi3-kernel-bis diffs, it is curious that CONFIG_IPVLAN is missing from bcmrpi3_defconfig when it's in the 32bit configs, so I'll tuck that one in as well.

Raspberry Pi Forums

Easily run 64-bit Debian Buster packages on a Raspbian system: new RPi4 / RPi3 image released (systemd-nspawn, LXDE)

Easily run 64-bit Debian Buster packages on a Raspbian system: new RPi4 / RPi3 image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)

Re: Easily run 64-bit Debian Stretch packages on a Raspbian system: new RPi3 demo image released (systemd-nspawn, LXDE)