MrQ
Posts: 6
Joined: Fri Sep 06, 2019 1:32 pm

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Sun Sep 08, 2019 4:51 pm

jdonald wrote:
Sat Sep 07, 2019 9:45 pm
MrQ wrote:
Fri Sep 06, 2019 2:39 pm
I don't cross compile. Just building directly on my RPI4.
That can work in theory. Raspbian doesn't provide a gcc-aarch64-linux-gnu package, so you would likely compile under a 64-bit userland (e.g. Gentoo) or chroot (see above).

And I think you should set KERNEL=kernel8 (not kernel7 or kernel7l).
MrQ wrote:
Fri Sep 06, 2019 6:17 pm
dom wrote:
Fri Sep 06, 2019 1:47 pm
CROSS_COMPILE=aarch64-linux-gnu- rather
BTW., there's no such command inside tools folder...
Sounds like you decided to cross-compile. Highly recommended if you might have to debug things and iterate. Get a toolchain via sudo apt install gcc-aarch64-linux-gnu on your Debian or Ubuntu x86_64 system. The toolchains under https://github.com/raspberrypi/tools are all rather old anyhow.
I managed to cross-compile on my pi4 by following this guide https://www.tal.org/tutorials/raspberry ... bit-kernel + PI4 specific commands from official manual https://www.raspberrypi.org/documentati ... uilding.md , just used the lastest binutils and gcc versions. Build was successful modules installed, dtbs and kernel copied to /boot .... but I get black screen after rainbow at boot. Do I need something more such as

Code: Select all

echo "arm_64bit=1" >> /boot/config.txt
?

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Mon Sep 09, 2019 11:33 am

mitchejj wrote:
Sat Sep 07, 2019 8:23 pm
Since we are talking about `rpi-update` and the 64-bit kernel... I suppose I should ask something I’ve yet to find an answer for with all my searching... how does one get the kernel headers for the kernel install via rpi-update? I would assume those people willing to something closer to bleeding edge might also be the type of person who might make use of the headers. So why not also install the headers?
The kernel headers are not wanted by most and are not trivial in size (around 8000 header files) so we are not going to include them by default with rpi-update. Stable kernels have kernel header files available from apt.

You might be interested in https://github.com/notro/rpi-source/wiki

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Mon Sep 09, 2019 11:37 am

MrQ wrote:
Sun Sep 08, 2019 4:51 pm
I managed to cross-compile on my pi4 by following this guide https://www.tal.org/tutorials/raspberry ... bit-kernel + PI4 specific commands from official manual https://www.raspberrypi.org/documentati ... uilding.md , just used the lastest binutils and gcc versions. Build was successful modules installed, dtbs and kernel copied to /boot .... but I get black screen after rainbow at boot. Do I need something more such as

Code: Select all

echo "arm_64bit=1" >> /boot/config.txt
?
Yes you need arm_64bit=1 if you have both 32-bit (kernel7l.img) and 64-bit (kernel8.img) kenels installed.
How did you install the kernel to boot? Which file did you copy from and what did you call it?

Kendek
Posts: 127
Joined: Thu Jul 25, 2019 4:39 pm
Location: Kaposvár, Hungary

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Mon Sep 09, 2019 12:18 pm

dom wrote:
Mon Sep 09, 2019 11:33 am
You might be interested in https://github.com/notro/rpi-source/wiki
The rpi-source script doesn't work with 64-bit kernels (Unrecognised CPU), only armv7l and armv6l.

MrQ
Posts: 6
Joined: Fri Sep 06, 2019 1:32 pm

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Mon Sep 09, 2019 12:42 pm

dom wrote:
Mon Sep 09, 2019 11:37 am
MrQ wrote:
Sun Sep 08, 2019 4:51 pm
I managed to cross-compile on my pi4 by following this guide https://www.tal.org/tutorials/raspberry ... bit-kernel + PI4 specific commands from official manual https://www.raspberrypi.org/documentati ... uilding.md , just used the lastest binutils and gcc versions. Build was successful modules installed, dtbs and kernel copied to /boot .... but I get black screen after rainbow at boot. Do I need something more such as

Code: Select all

echo "arm_64bit=1" >> /boot/config.txt
?
Yes you need arm_64bit=1 if you have both 32-bit (kernel7l.img) and 64-bit (kernel8.img) kenels installed.
How did you install the kernel to boot? Which file did you copy from and what did you call it?

Code: Select all

KERNEL=kernel7l
make bcm2711_defconfig
make -j4 Image modules dtbs
sudo make modules_install
sudo cp arch/arm64/boot/dts/broadcom/*.dtb /boot/
sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/overlays/
sudo cp arch/arm64/boot/dts/overlays/README /boot/overlays/
sudo cp arch/arm64/boot/Image /boot/$KERNEL.img
So I replaced kernel7l.img with new 64-bit kernel.

--------------------------------------------

Ok, I made it. Now 64 bit kernel boots without issues. All you have to do to make 64-bit kernel on 32-bit Raspbian is:

1. Follow this guide to compile binutils and gcc for aarch64 support https://www.tal.org/tutorials/raspberry ... bit-kernel
1a . Use lastest binutils and gcc (binutils-2.32.tar.xz and gcc-9.2.0.tar.xz in my case)

2. Backup /boot

3.Add thece CFLAGS to Makefile

Code: Select all

-Wno-error=int-to-pointer-cast -Wno-implicit-function-declaration -Wno-pointer-to-int-cast
4 Compile kernel

Code: Select all

KERNEL=kernel8
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-  bcm2711_defconfig
make -j4 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image modules dtbs
sudo make modules_install
sudo cp arch/arm64/boot/dts/broadcom/*.dtb /boot/
sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/overlays/
sudo cp arch/arm64/boot/dts/overlays/README /boot/overlays/
sudo cp arch/arm64/boot/Image /boot/$KERNEL.img
if you want to change kernel config use

Code: Select all

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig

5. Add to /boot/config.txt

Code: Select all

kernel=kernel8.img
arm_64bit=1
6. Reboot

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Mon Sep 09, 2019 3:25 pm

MrQ wrote:
Mon Sep 09, 2019 12:42 pm

Code: Select all

kernel=kernel8.img
arm_64bit=1
Shouldn't need the kernel=kernel8.img line. That is the default kernel name with arm_64bit=1.

MrQ
Posts: 6
Joined: Fri Sep 06, 2019 1:32 pm

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Mon Sep 09, 2019 7:50 pm

What's the correct way of installing custom headers and make them system-wide accessible? For 32-bit I used

Code: Select all

make-kpkg kernel_headers
then installed the deb file as usual, but this method doesn't work for cross-compiled kernel

arthurnorman
Posts: 1
Joined: Mon Sep 09, 2019 8:45 pm

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Mon Sep 09, 2019 9:34 pm

Two questions:
(1) Have I missed something or does the G2 GL (Fake KMS) option limit video resolution to HD such that 4K is not supported. This arises when installing the 64-bit kernel as per here and then following the January 29/Sakaki recipe for using systemd/nspawn to get a 64-bit userland to go with the kernel. Eg maybe the video stuff is closed enough that 4K is not in the 64bit world, or it may be that I have just not spotted explanations about how to activate it.

(2) With the 64-bit kernel active "uname -m" reports aarch64 even though one has a 32-but userland. I believe I have a case where this really muddles autoconf in libffi as well as my own configure.ac scripts that try to put built files somewhere related to the architecture in use. I am working to patch that for myself by making a /usr/local/bin/uname to take precedence over /bin/uname and return what I expected on a 32-bit platform, but does anybody have a better suggestion?

An rpi4 with the root filesytem on an SSD, a 64-bit kernel and then both 32 and 64-bit Buster user spaces via nspawn is looking good so far, but I need longer to assess reliability as and sort out my own confusions!

Thanks. Arthur

andrum99
Posts: 848
Joined: Fri Jul 20, 2012 2:41 pm

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Tue Sep 10, 2019 11:08 am

I'm getting the following messages in the log with the 64-bit kernel:

Code: Select all

Sep 10 04:53:25 livingroom kernel: [42196.434063] perf: interrupt took too long (2521 > 2500), lowering kernel.perf_event_max_sample_rate to 79250
Sep 10 06:20:46 livingroom kernel: [47437.778387] perf: interrupt took too long (3154 > 3151), lowering kernel.perf_event_max_sample_rate to 63250
Sep 10 06:25:53 livingroom kernel: [47744.652490] perf: interrupt took too long (3950 > 3942), lowering kernel.perf_event_max_sample_rate to 50500
Sep 10 06:27:45 livingroom kernel: [47856.451975] perf: interrupt took too long (4945 > 4937), lowering kernel.perf_event_max_sample_rate to 40250
Sep 10 06:44:17 livingroom kernel: [48849.052058] perf: interrupt took too long (6182 > 6181), lowering kernel.perf_event_max_sample_rate to 32250
Sep 10 07:59:01 livingroom kernel: [53332.825991] perf: interrupt took too long (7733 > 7727), lowering kernel.perf_event_max_sample_rate to 25750
There's nothing much going in at 04:53 in the morning, but from 06:20 onwards is when the daily backup runs and it takes several hours. The backup is using pigz and tar to create a tarball which is then sent to Google Drive, from the local USB 3.0 attached hard disk.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Tue Sep 10, 2019 12:06 pm

arthurnorman wrote:
Mon Sep 09, 2019 9:34 pm
(1) Have I missed something or does the G2 GL (Fake KMS) option limit video resolution to HD such that 4K is not supported. This arises when installing the 64-bit kernel as per here and then following the January 29/Sakaki recipe for using systemd/nspawn to get a 64-bit userland to go with the kernel. Eg maybe the video stuff is closed enough that 4K is not in the 64bit world, or it may be that I have just not spotted explanations about how to activate it.
4K is supported out of the box with fkms with or without 64-bit kernel.
You are limited to 4kp30 by default, but adding

Code: Select all

hdmi_enable_4kp60=1
to config.txt enables 4kp60 too.
But as this is unrelated to 64-bit kernel then discussion of this is best handled elsewhere.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6042
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Wed Sep 11, 2019 11:49 am

jdonald wrote: However, I'm skeptical as to outputting instructions specific to higher-end CPU cores not making a difference.
Having played around with a debian arm64 container a bit, I've got to eat crow.

Going from hardfp armv6 to Debian's armv7 binaries may not have been much of an improvement, but debian's arm64 binaries makes a much bigger difference than I would've expected - double-digit in some cases. Using debootstrap to install to tmpfs (to avoid IO bottlenecks) is 20% faster and xz is about 8% faster.

paul433
Posts: 19
Joined: Sun Dec 30, 2012 10:46 pm

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Wed Sep 11, 2019 1:57 pm

Is Onboard Audio working for others? Headphone Jack. USB based Audio works fine, I have not checked i2s yet or Audio out over HDMI. The BCM2835 device is showing as a valid Alsa device, but getting errors trying to open the device for playback.

I'm using a custom compiled kernel using the arm64 BMC2711 defconfig.

BTW: There are quite a few device omissions in the 64bit defconfig over the 32bit.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7431
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Wed Sep 11, 2019 4:08 pm

paul433 wrote:
Wed Sep 11, 2019 1:57 pm
BTW: There are quite a few device omissions in the 64bit defconfig over the 32bit.
I'd noticed the same. It should get fixed in the next few days.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mentaluproar
Posts: 13
Joined: Sun Mar 20, 2016 7:57 pm

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Thu Sep 12, 2019 12:27 am

In testing, I've found arduino is broken on 64 bit. avrdude throws "mmap error -1" in 64 bit, but works fine in 32 bit. I've also found Konsole will crash during apt installs for some things, but other consoles work fine.

BRX7
Posts: 26
Joined: Sat Aug 31, 2019 11:07 pm

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Thu Sep 12, 2019 12:31 pm

ShiftPlusOne wrote:
Wed Sep 11, 2019 11:49 am
jdonald wrote: However, I'm skeptical as to outputting instructions specific to higher-end CPU cores not making a difference.
Having played around with a debian arm64 container a bit, I've got to eat crow.

Going from hardfp armv6 to Debian's armv7 binaries may not have been much of an improvement, but debian's arm64 binaries makes a much bigger difference than I would've expected - double-digit in some cases. Using debootstrap to install to tmpfs (to avoid IO bottlenecks) is 20% faster and xz is about 8% faster.

does a 64bit raspbian buster image exist yet? using 64bit kernel in raspbian means its still using 32bit binaries of libraries and other binaries yes?
also can I use docker 64bit version within raspbian now with the new kernel? or can a arm 64bit container work like https://hub.docker.com/layers/ownyourbi ... e55c0a761b

User avatar
DougieLawson
Posts: 36322
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Thu Sep 12, 2019 12:44 pm

BRX7 wrote:
Thu Sep 12, 2019 12:31 pm
does a 64bit raspbian buster image exist yet? using 64bit kernel in raspbian means its still using 32bit binaries of libraries and other binaries yes?
The current experiment is 32-bit Raspbian userland code with a 64-bit kernel.

Sakaki-san's Gentoo64 has more 64-bit stuff than Raspbian.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6042
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Thu Sep 12, 2019 12:54 pm

BRX7 wrote: using 64bit kernel in raspbian means its still using 32bit binaries of libraries and other binaries yes?
Yes, but I was using a debian arm64 container within Raspbian.

Initially I was looking at what it would take to get multiarch working, but Raspbian and Debian don't mix well in that context.

Can't speak for docker specifically, but systemd-nspawn and plain chroot work fine.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Thu Sep 12, 2019 2:13 pm

The instructions here provide a pretty simple way to install and run 64-bit debian packages on raspbian.

User avatar
DougieLawson
Posts: 36322
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Thu Sep 12, 2019 2:34 pm

dom wrote:
Thu Sep 12, 2019 2:13 pm
The instructions here provide a pretty simple way to install and run 64-bit debian packages on raspbian.
Better option would be Sakaki-san's Raspbian 64 https://www.raspberrypi.org/forums/view ... 1#p1534981
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Thu Sep 12, 2019 2:54 pm

DougieLawson wrote:
Thu Sep 12, 2019 2:34 pm
Better option would be Sakaki-san's Raspbian 64 https://www.raspberrypi.org/forums/view ... 1#p1534981
Yes, that will get you the same rpi-update kernel described in this thread and has a lot of the configuration for 64-bit debian handled for you.
Recommended if you want to start with a fresh image. If you have an existing image you want to experiment with then the linked thread gives a simple (copy/paste a few commands and wait ten minutes) way of testing some 64-bit packages.

jdonald
Posts: 417
Joined: Fri Nov 03, 2017 4:36 pm

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Thu Sep 12, 2019 10:22 pm

dom's succinct guidelines for how to get a 64-bit user environment are perfect.

@ShiftPlusOne thanks for taking a look and running more benchmark experiments.
arthurnorman wrote:
Mon Sep 09, 2019 9:34 pm
With the 64-bit kernel active "uname -m" reports aarch64 even though one has a 32-but userland. I believe I have a case where this really muddles autoconf in libffi as well as my own configure.ac scripts that try to put built files somewhere related to the architecture in use. I am working to patch that for myself by making a /usr/local/bin/uname to take precedence over /bin/uname and return what I expected on a 32-bit platform
In the past I've also used a uname wrapper to deal with third-party software that runs into this problem.

If you're trying to make your own software robust against such configurations, here are two ways to query the userland architecture:

Code: Select all

dpkg --print-architecture
or

Code: Select all

cc -dumpmachine
dpkg has a shortcoming that it isn't portable to systems using yum or other package managers, while -dumpmachine can have other quirks like returning different answers from gcc vs clang.

For Ubuntu, I saw on Launchpad at one point they were building armhf packages on a system with an aarch64 kernel, so apparently they've solved this for most packages. Perhaps the debian/rules are often doing ./configure --host=$(gcc -dumpmachine)
BRX7 wrote:
Thu Sep 12, 2019 12:31 pm
also can I use docker 64bit version within raspbian now with the new kernel? or can a arm 64bit container work like https://hub.docker.com/layers/ownyourbi ... e55c0a761b
Docker alone doesn't work out of the box for jumping from 32-bit userland to 64-bit. I reported this issue at some point: https://forums.docker.com/t/unable-to-r ... cker/67187
I'm now seeing that was more than 3.5 years ago, so I suspect it's not trivial to fix. For some reason systemd-nspawn, LXC, and chroot just work while Docker doesn't.

The workaround I have for using aarch64 Docker images on Raspbian is to first set up an arm64 chroot with debootstrap then install and run docker-ce:arm64 in there.

I see the Docker Hub image you linked is using qemu-aarch64-static so it may not do what you expect. An example of a true aarch64 Docker image is arm64v8/debian.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6042
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Fri Sep 13, 2019 9:35 am

Wrapping commands in 'setarch linux32' will work around the uname issue.

User avatar
sakaki
Posts: 381
Joined: Sun Jul 16, 2017 1:11 pm

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Fri Sep 13, 2019 12:01 pm

Hello,

Slightly OT since the focus is on the Pi4, but: I have used this official 64-bit kernel (@ 4.19.69-v8+, with matching boot f/w, as pulled in by rpi-update) in my recently released raspbian-nspawn-64 image, so I've given it some testing on both an RPi4 and RPi3. Mostly, it seems to work well on both platforms; thanks for making it available!

One minor issue I encountered was that on the RPi3, while the path /dev/hwrng exists, there appears to be no functional driver for it, which causes the generator service rng-tools to fail. However, rng-tools starts up fine on the RPi4 (with the same kernel, same vanilla Raspbian 32-bit userland), and I've used the hwrng successfully on the RPi3 under Gentoo in 64-bit mode, so probably just a minor config or DT issue (CONFIG_HW_RANDOM_BCM2835 or such?)

Best, sakaki

mitchejj
Posts: 2
Joined: Fri Sep 06, 2019 5:20 pm

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Fri Sep 13, 2019 9:01 pm

dom wrote:
Mon Sep 09, 2019 11:33 am
The kernel headers are not wanted by most and are not trivial in size (around 8000 header files) so we are not going to include them by default with rpi-update. Stable kernels have kernel header files available from apt.

You might be interested in https://github.com/notro/rpi-source/wiki
rpi-source is exactly what I was hoping to find. As noted below it doesn’t support the 64-bit kernel :(

Now I’m heading down the rabbit hole to attempt to build a functional kernel in hopes have a better understanding of how the kernel builds and possible attempt to submit a patch to make rip-source work with the 64-bit kernel.

I think the last time I built my own kernel was circa Red Hat 6.0 w/ the 2.2 kernel.

graphicw
Posts: 87
Joined: Mon Sep 09, 2019 5:04 pm

Re: Pi4 64-bit raspbian kernel for testing - Focus on Pi4

Sat Sep 14, 2019 6:19 am

OpenGL performance decreases considerably with 64 bit kernel as is. Open Arena at 1080p was a slide show under the 64 bit whereas it performed at 20 fps and better with standard kernel. YouTube and such performed the same as standard kernel.

Return to “Advanced users”