bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: RPI3 QEMU

Wed Feb 07, 2018 2:01 pm

UPDATE: a good soul took my patch and picked up the fight with the qemu maintainers, so there's a new hope (sorry Lucas, I won't pay license fee for my words :-) ). Although he agrees with me that adding dummy device handlers is a total nonsense for a patch that only supposed to change the boot location and the cpu type, he did all the other modifications as they asked.

I hereby wish him good luck with successfully submitting the patch to the qemu mainline!

bzt

kete
Posts: 1
Joined: Fri Feb 09, 2018 6:07 pm

Re: RPI3 QEMU

Fri Feb 09, 2018 6:17 pm

Great work @bzt!

I work in the 64bit mode of RPi3 very much and I believe your project is very useful. I have these questions:

1. I applied your patch and built it for qemu 2.11. I should say that the latest git did not work because of another problem and 2.10 seems to lack some definitions like error: ‘MachineClass {aka struct MachineClass}’ has no member named ‘ignore_memory_transaction_failures’. However, when trying to load an image it seems that qemu hangs with around 90% cpu usage. I tested it with a 32bit kernel and it booted fine. Could you please give more insight on how to debug this? I suppose that we need a kernel that does not require a config.txt so I searched for one (mentioned in this thread), but it did not work either.

2. From the performance and convenience point of view, how does this patch compare with direct chrooting in arm64? Some programs cannot be compiled in that ways and they are a big pain. I wonder if everything will work using your solution or not.

bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: RPI3 QEMU

Tue Feb 13, 2018 2:29 am

Hi kete,

Thank you! But this is not a big thing.

1. It seems qemu guys think it's funny to mess around a lot with the MachineClass interface :-) Here are the patches for specific qemu versions I've tested with. At 2.10.50 it was the min_cpus field that caused trouble for me, not the 'ignore...' one. I've tested with all kernels in my tutorials, and others had reported them working as well. Frankly I've never even tried linux because I was aware of the missing peripheral emulation. This patch (which only adds booting support) is just the first step on a long way to run an AArch64 linux kernel under qemu.

2. This patch only changes the firmware address and the cpu type, does nothing more. As such it should not have any performance impact at all, unless there's already a bug in Cortex-A53 emu code somewhere. I suggest to try adding '-d int' to qemu arguments. I've noticed that qemu does not handle exceptions in exception handlers well (there's no such thing on AArch64 like triple fault on x86). If the problem for example is exception handler code not being mapped properly, that would result in an endless Instruction Fetch Abort loop. That could lead to 90% cpu usage, but this is just a guess. My other guess would be the aforementioned peripheral emulation, namely the missing System Timer. As it's MMIO register reads as a constant zero, a naive delay() loop could end up in an infinite loop easily. See my delay tutorial on how to workaround this.

Hopefully Pekka will manage to submit the patch into mainline soon. If the qemu maintainers would have accepted my boot patch, I would have been working on the peripherals support by now. But no, instead they convinced me that I don't want to do anything with them any more. C'la vie!

Bests,
bzt

bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: RPI3 QEMU

Fri Feb 16, 2018 7:22 pm

Good news Everyone! (TM) :-)

Thanks to Pekka, finally Peter has accepted 2 of the 3 patches required to boot 64 bit kernels in qemu on arm-next branch:
https://git.linaro.org/people/peter.may ... 00269d10eb

One more patch to go! Mainline raspi3 support in qemu may become reality sooner than I've hoped :-)

bzt

bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: RPI3 QEMU

Sat Feb 24, 2018 10:59 pm

And finally made it through: https://git.qemu.org/?p=qemu.git;a=comm ... b9040f3504

Raspberry Pi3 boot is in the mainline qemu! I'd like to say thanks to Pekka Enberg for making this possible!

Cheers,
bzt

bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: RPI3 QEMU

Mon Apr 30, 2018 11:19 pm

timanu90 wrote:
Tue Oct 17, 2017 11:01 am
Hi guys, seems that some people could run aarch64 rpi3 code in qemu. Anyone can tell me the version of qemu needed and what command you use to launch the session?
I can finally answer that question :-)
qemu 2.12 now has inital Raspberry PI 3 support! See change log.

The command to use it:

Code: Select all

qemu -M raspi3
Cheers
bzt

timanu90
Posts: 65
Joined: Sat Dec 24, 2016 11:54 am

Re: RPI3 QEMU

Fri May 04, 2018 12:27 pm

Hello bzt, very good news.

I have been busy, but I will try this soon.

Congratulations and thank you for your work.

Cheers
Tiago

gilius
Posts: 96
Joined: Sun Apr 08, 2018 1:12 pm

Re: RPI3 QEMU

Sat May 19, 2018 2:34 pm

Can somebody please give a tutorial on how to get anything to boot up at all with a display!?

qemu-system-aarch64 -M raspi3 -m 1G -kernel kernel8.img

I created a new disk image 100 MB FAT32 with some RPi3 boot files on it, but nothing displays - no rainbow screen - nothing.

adrien-lessard
Posts: 1
Joined: Mon Oct 29, 2018 7:39 pm

Re: RPI3 QEMU

Mon Oct 29, 2018 7:44 pm

I'm having the same issue as gilius here. I run

Code: Select all

qemu-system-aarch64 -M raspi3 -m 1G -kernel kernel8.img
and get a black window with nothing happening.

Could someone point out what I'm missing?

Note: qemu version is 3.0.0

bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: RPI3 QEMU

Wed Oct 31, 2018 11:57 pm

It's a bit more complicated than that, but I'll try to explain it for you in short.

When you emulate x86, then you emulate everything for the main processor (including the BIOS reading the boot sector from the provided disk image). For ARM (and specially with "-M raspi3"), this isn't the case.

On the Raspberry Pi board, ARM CPU is not the main processor, the VC GPU is. The files on the SD card (bootcode.bin, start.elf etc.) are compiled for the GPU and contain code to initialize the hardware (like the HDMI connection and the framebuffer with the rainbow splash). Start.elf parses config.txt and loads the kernel8.img from the SD card, finally enables the CPU by making it to execute that freshly loaded kernel image (which the GPU can't interpret btw).

Qemu on the other hand (being an ARM CPU emulator) does not emulate the VC GPU. Because the emulation starts at the ARM startup phase, bootcode.bin, start.elf and the others are NOT USED at all. Therefore there's no rainbow splash, neither is the kernel8.img loaded from the provided SD card image file (hence the need to pass the kernel directly with the "-kernel" argument). Although qemu does not emulate the VC, it does emulate the MailBox interface, so you can set screen resolution and get a framebuffer. You can draw on that buffer just like on a real hardware (but this time the framebuffer will not be read and displayed by the VC, but an X11 or SDL routine in qemu, unimportant detail from the ARM CPU's perspective).

What the "-M raspi3" does inside qemu is, that it tells qemu where to load the user provided kernel image in the emulated memory (to match the address that start.elf uses), and sets up some device emulation of the BCM2837 SoC, so that it seems to the emulated ARM CPU it's running on a Raspberry Pi board. Qemu is not a real, fully featured Raspberry Pi hardware emulator, just an ARM emulator.

Hopefully this makes now sense to you,
bzt

bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: RPI3 QEMU

Tue Feb 26, 2019 3:02 am

Hi,

For those who are interested, I've implemented the periodic ARM Control Local Timer in qemu.
https://github.com/bztsrc/qemu-local-timer

I've sent it to the qemu discussion list for a review.

If the qemu developers accept the patch, I'll continue with the BCM2835 System Timer. That's a bit more difficult to implement, as there's no class for it yet and it requires a dynamic memory value, but with that many bare metal projects would run under qemu (like @rst's Circle for one). Usually the reason of failure is the infinite delay() loop, caused by the register at 0x3F003004 (SYSTEMTIMER_CLO) not changing at all.

Cheers,
bzt

rst
Posts: 386
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany

Re: RPI3 QEMU

Tue Feb 26, 2019 7:18 am

bzt wrote:
Tue Feb 26, 2019 3:02 am
If the qemu developers accept the patch, I'll continue with the BCM2835 System Timer. That's a bit more difficult to implement, as there's no class for it yet and it requires a dynamic memory value, but with that many bare metal projects would run under qemu (like @rst's Circle for one). Usually the reason of failure is the infinite delay() loop, caused by the register at 0x3F003004 (SYSTEMTIMER_CLO) not changing at all.
It would be great to have the BCM2835 System Timer implemented, especially if it would be in the official QEMU.

FYI there is another issue with the USB CDC Ethernet device emulation. It drops packets with more than 63 bytes, which makes it difficult to use it with the Circle network driver. I applied a patch to my own fork of 0xabu at GitHub 's QEMU version for the Raspberry Pi 2, which is working well with Circle, but probably it would be difficult for me, to get this into official QEMU.
Last edited by rst on Tue Feb 26, 2019 2:36 pm, edited 1 time in total.

bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: RPI3 QEMU

Tue Feb 26, 2019 11:18 am

rst wrote:
Tue Feb 26, 2019 7:18 am
it would be difficult for me, to get this into official QEMU.
Tell me about it :-) Last time the "-M raspi3" patch only went through because Pekka helped. I hope the ARM Local Timer will pass as that's a very simple patch, but with the System Timer I'll definitely need Pekka's help again. I'm hoping he is willing to push through another patch of mine. But first I have to write it :-)

About your dev-network.c patch, if I were you, I would write to the qemu-devel at nongnu.org mailing list. I'm sure your patch qualifies as a trivial patch.

And I see in Circle, you have added USE_PHYSICAL_COUNTER in the CTimer class to workaround ARM_SYSTIMER_CLO not changing? Good work!

Cheers,
bzt

rst
Posts: 386
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany

Re: RPI3 QEMU

Tue Feb 26, 2019 12:14 pm

bzt wrote:
Tue Feb 26, 2019 11:18 am
I hope the ARM Local Timer will pass as that's a very simple patch, but with the System Timer I'll definitely need Pekka's help again. I'm hoping he is willing to push through another patch of mine. But first I have to write it :-)
In the repo of 0xabu, which I forked, the System Timer has already been implemented. I'm sure, that it works, because I'm using it from time to time with Circle. I think, this man did already a lot for RPi support in QEMU. I don't know, why this System Timer support is not part of the current QEMU 3.1.0, because 0xabu had it implemented and did add code to QEMU.
About your dev-network.c patch, if I were you, I would write to the qemu-devel at nongnu.org mailing list. I'm sure your patch qualifies as a trivial patch.
Thanks for this hint. Perhaps I will try this, when the System Timer works. At the moment the patch would not help me, because Circle is not working with the original QEMU and the patch cannot be tested. But I think, this has already been tried (search for "usb_net_handle_dataout") by flypie at GitHub and the patch was not accepted, because it is not contained in QEMU 3.1.0. :(

rst
Posts: 386
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany

Re: RPI3 QEMU

Tue Feb 26, 2019 2:52 pm

I have to apologize, because I mentioned the wrong author of the bcm2835_st driver for QEMU. The driver is here and the author is noticed in the file header. In the commit notice there is the suffix "et al".

Shouldn't talk about things, I do not know much about like QEMU. :oops: @bzt you knows for sure much better, how to handle this with QEMU, because, as far as I know, you have done a lot for it too. So I'm looking forward to your solution for the System Timer, based on this one above or a new one, what ever is better from your point of view.

bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: RPI3 QEMU

Wed Feb 27, 2019 1:14 am

rst wrote:
Tue Feb 26, 2019 2:52 pm
I have to apologize, because I mentioned the wrong author of the bcm2835_st driver for QEMU. The driver is here and the author is noticed in the file header. In the commit notice there is the suffix "et al".

Shouldn't talk about things, I do not know much about like QEMU. :oops: @bzt you knows for sure much better, how to handle this with QEMU, because, as far as I know, you have done a lot for it too. So I'm looking forward to your solution for the System Timer, based on this one above or a new one, what ever is better from your point of view.
Hey this is awesome, he wrote the driver exactly in a way I wanted to! And he has also added the power management interface! I see no point in reimplementing the wheel (or reinventing the timer, ehhh whatever :-) ). His solution looks pretty good, clean code, proper classes, using host friendly virtual clock etc. Do you know by any chance why on earth was his patch refused?

(just off the book, I had a feeling that M$ is - let's just say - not really motivated in a really good raspi emulator, and since every patch must be reviewed and accepted by them...well, you know. I suppose they are maybe afraid that their precious W10IoT could be reverse engineered in a vm or something, but that's really just a guess. Not that they could stop a hacker from running that spyware in a patched qemu anyway... :P )

Cheers,
bzt

ps.: The all winner comment is: "/* Ugly temporary hack to get Plan9 to boot... */", that alone made my day!!!

rst
Posts: 386
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany

Re: RPI3 QEMU

Wed Feb 27, 2019 7:00 am

bzt wrote:
Wed Feb 27, 2019 1:14 am
Hey this is awesome, he wrote the driver exactly in a way I wanted to! And he has also added the power management interface! I see no point in reimplementing the wheel (or reinventing the timer, ehhh whatever :-) ). His solution looks pretty good, clean code, proper classes, using host friendly virtual clock etc. Do you know by any chance why on earth was his patch refused?
It's good, if this driver helps you. :) No, I have no clue, but it seems to be not easy to get a patch accepted.
ps.: The all winner comment is: "/* Ugly temporary hack to get Plan9 to boot... */", that alone made my day!!!
;)

bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: RPI3 QEMU

Thu Feb 28, 2019 2:22 pm

Hi,

Good News, Everyone! (TM)

The ARM Local Timer is accepted for a review, and I got answer from Andrew. He said the only reason why those drivers weren't merged because he didn't had the time to bring them up with the modern qemu API. Yeah, lots of things changed, granted, but still I can do that easily. I also got Pekka on board, so this is really happening :-)

Keep tuned,
bzt

rst
Posts: 386
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany

Re: RPI3 QEMU

Thu Feb 28, 2019 3:44 pm

bzt wrote:
Thu Feb 28, 2019 2:22 pm
The ARM Local Timer is accepted for a review, and I got answer from Andrew. He said the only reason why those drivers weren't merged because he didn't had the time to bring them up with the modern qemu API. Yeah, lots of things changed, granted, but still I can do that easily. I also got Pekka on board, so this is really happening :-)
This is really good news! :) Thanks.

bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: RPI3 QEMU

Mon Mar 11, 2019 1:57 pm

Hi

This question goes to all, but mostly to @rst, @ultibo and @LdB. Assuming qemu could emulate the BCM System Timer, what other peripherals are needed to boot bare metal projects like Circle, Ultibo and SmartStart under qemu? (I know, the best would be all peripherals, but let's just stick to the bare minimum requirement for now. Small steps :-) ). I know that some of you already have a System Timer workaround, that's why I'm asking.

Once we have all the necessary devices to boot your projects in qemu without problems (hopefully that means just the System Timer), I'm planning to move on with other peripherals, but that's going to be a veeery long ride. I have the PMM as next in mind, which is certainly not essential, but would be nice if we could reset and power off the virtual machine just like the real one. There's a semihosting hack to turn off the vm, but currently no API to reset it (that I know of).

Cheers,
bzt

rst
Posts: 386
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany

Re: RPI3 QEMU

Mon Mar 11, 2019 3:56 pm

Hi bzt,

generally I'm happy with the current QEMU release now. It helped me very much to debug some AArch64 libgcc C++ exception handling problem this weekend. Because I needed a quick solution here, I have added the Generic Timer (Core Timer) support to Circle and the System Timer is not required to boot Circle any more on RPi 3.

I have one long term wish for QEMU, an emulation of the BCM283[567] USB controller in host mode. I know, this is not a small wish, but this would allow things like networking and it has already been done here. The comment in line 6 is not very inviting, but it works.

Thanks for asking!

rst

User avatar
Ultibo
Posts: 158
Joined: Wed Sep 30, 2015 10:29 am
Location: Australia
Contact: Website

Re: RPI3 QEMU

Tue Mar 12, 2019 10:48 pm

bzt wrote:
Mon Mar 11, 2019 1:57 pm
... Assuming qemu could emulate the BCM System Timer, what other peripherals are needed to boot bare metal projects like Circle, Ultibo...
We took the opposite approach, instead of waiting for QEMU to emulate a Raspberry Pi we added QEMU support to Ultibo. The advantage of this approach is that we get network, storage, keyboard, mouse, framebuffer etc without having to worry about incomplete or inaccurate emulations of the Pi specific devices.

At present we have drivers for most of the relevant hardware in the Versatile PB emulation and while that target is fairly old now it shares a lot of common devices with the later Versatile Express target that also provides multicore A9 and A15 so with a small amount of additional work we can have support for a multicore emulation. If you are focused on Aarch64 you might be interested to know that you can switch to a Cortex A53 by simply changing the command line CPU switch (even on the old Versatile PB).

Haven't looked at the state of the QEMU Raspi targets for a while but I suspect that once the standard devices are working correctly Ultibo should boot without changes.
Ultibo.org | Make something amazing
https://ultibo.org

Threads, multi-core, OpenGL, Camera, FAT, NTFS, TCP/IP, USB and more in 3MB with 2 second boot!

bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: RPI3 QEMU

Wed Mar 13, 2019 2:58 pm

Hi,

You are smart, I should have known that all of you came up with a solution already :-) Regardless to your answers, examining the tutorials and available sources online I've decided to go on with the BCM System Timer next, as many are still using it for microsec delay. It would be nice to boot RPIHaribote in qemu for one.

@ultibo: I've underestimated how sophisticated Ultibo is :-) I dig into the source now and I see you also have a Generic Timer workaround in place. Allow me to rephrase my question: is there any peripheral missing to boot Ultibo if you run it in qemu with "-M raspi3"?

Cheers,
bzt

User avatar
Ultibo
Posts: 158
Joined: Wed Sep 30, 2015 10:29 am
Location: Australia
Contact: Website

Re: RPI3 QEMU

Thu Mar 14, 2019 9:39 am

bzt wrote:
Wed Mar 13, 2019 2:58 pm
... I see you also have a Generic Timer workaround in place.
It isn't really a workaround, more of an option for those that need it.

We use the Generic Timers to drive the scheduler on each core (similar to what you were wanting to do here) but we also include an option to use the System Timer for the clock instead.
bzt wrote:
Wed Mar 13, 2019 2:58 pm
Allow me to rephrase my question: is there any peripheral missing to boot Ultibo if you run it in qemu with "-M raspi3"?
Let us do some quick tests and see, we'll let you know the results.
Ultibo.org | Make something amazing
https://ultibo.org

Threads, multi-core, OpenGL, Camera, FAT, NTFS, TCP/IP, USB and more in 3MB with 2 second boot!

bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: RPI3 QEMU

Thu Mar 21, 2019 2:01 am

Hi,

It took me about 2 hours to write and almost a month to get it merged, but the ARM Local Timer is finally in qemu mainline! :-)

Cheers,
bzt

Return to “Bare metal, Assembly language”