Morin
Posts: 2
Joined: Mon Aug 29, 2016 6:29 am

userland graphics driver (not firmware or kernel part)

Mon Aug 29, 2016 6:35 am

Hi all,

PREFACE: This is not meant to be a discussion whether the code can be considered open! Please don't turn it into one. I'm trying to get stuff done, not rant about it.

Also, I'm posting this here after the same question on Stackoverflow didn't yield any useful answers, just in case somebody stumbles over that question.

I'm trying to understand the userland part of the Raspberry Pi graphics driver code from https://github.com/raspberrypi/userland

My understanding so far is:
- a firmware blob runs in the GPU and offers an OpenGL-like interface which, on lower levels, is based on message (byte-array) passing on top of one of multiple 28-bit-word FIFOs called VCHIQ (the other VCHIQ queues are irrelevant for graphics)
- on the CPU part, OpenGL calls are turned into messages to the GPU. Access to the low-level facility (either the message queue or VCHIQ -- I haven't found that part yet in the code) requires a Linux kernel module, but no high-level logic happens in there.
- the GPU part is closed, but that's okay for my purposes. The (ARM) CPU part is, AFAIK, open

My ultimate goal is to get communication with the GPU working on bare metal (without Linux), but with the closed firmware blob intact. As a first goal, I want to understand how an OpenGL call is actually passed to the GPU.

However, I'm stuck at finding the actual code for this. The OpenGL calls use RPC_CALL* and in turn RPC_DO, which calls khronos_server_lock_func_table(). However, that function seems to be missing from the code, and to my surprise, I couldn't find anything useful about it on Google.

My questions:
- am I still on the ARM CPU side, or did I move to GPU land without noticing? If the latter is the case, where did I cross that line?
- Assuming I'm still on the CPU side -- where is the code for that function? Is it open at all, or do we actually have closed parts left around on the CPU side here? All sources on the web seem to indicate that the code for the CPU is 100% open.
- at which point does the implementation of the C OpenGL functions actually send a message to the GPU? I'm somewhat expecting a call to the kernel functionality that represents VCHIQ to be happening at some point, probably implemented as a device file.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24584
Joined: Sat Jul 30, 2011 7:41 pm

Re: userland graphics driver (not firmware or kernel part)

Tue Aug 30, 2016 9:07 am

All the ARM side code is open AIUI, so there functions should be there somewhere.

The userland libs do, as you say, eventually result in a VCHIQ call to the GPU, anything above that call should be available as source. I'd try debugging down the call to determine where all the functions are.

You might also be interested in the new userland driver, where everything is done on the ARM - the code pokes registers directly (the bit normally done by the GPU). Check out Eric Anholt's work on github.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

User avatar
Gavinmc42
Posts: 4204
Joined: Wed Aug 28, 2013 3:31 am

Re: userland graphics driver (not firmware or kernel part)

Tue Aug 30, 2016 11:42 am

Thanks for the clues, was not quite sure about Eric's code.

One of the things I want with Ultibo is access to the GPU for the camera interface.
Two second boot Pi cameras, baremetal robots with camera, etc.

VCHIQ would probably be best, don't want to write a jpeg/h264 engine :o
This may get reverse engineered one day but not by me.

OpenVG/GLES on Ultibo would be great too, makes for fast baremetal GUI's.
Have not been following Eric's progress closely but was there a rumour of Mesa/3D soon?
Any change with Wayland/Weston recently or is that going to be skipped?
Eric's porting Vulkan :P

Trying to stay up to date means camping out on github.
Hmm, just found out there is a raw/next/boot.... new bootload.bin - extra boots for Pi 1/2.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24584
Joined: Sat Jul 30, 2011 7:41 pm

Re: userland graphics driver (not firmware or kernel part)

Tue Aug 30, 2016 12:46 pm

Eric's stuff doesn't cover the camera, only 3D.

You'll need to use VCHIQ for any camera stuff. Good luck, you'll need if if you want to use the camera in baremetal! There is a hell of a lot of code required to make that work - MMAL for example, and that requires all sorts of high level OS features. It's not just a 'tell it to take a picture' type of thing.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

User avatar
Ultibo
Posts: 160
Joined: Wed Sep 30, 2015 10:29 am
Location: Australia
Contact: Website

Re: userland graphics driver (not firmware or kernel part)

Thu Sep 01, 2016 12:38 am

Morin wrote:My ultimate goal is to get communication with the GPU working on bare metal (without Linux), but with the closed firmware blob intact
There seems to be quite a few of us who have a similar goal, just to get bare metal communication with the firmware blob in the same way Linux does. It would be great if all of these efforts could be shared in some way.
jamesh wrote:There is a hell of a lot of code required to make that work - MMAL for example, and that requires all sorts of high level OS features
The issue is not a lack of high level OS features since we have all of the necessary things like scheduling, locks, interrupts, timers, tasks, wait queues etc etc etc.

The real issue is wading through the complexity of the userland libraries and the VCHIQ driver to work out how it all fits together and find the bits needed to get the communication working.
Ultibo.org | Make something amazing
https://ultibo.org

Threads, multi-core, OpenGL, Camera, FAT, NTFS, TCP/IP, USB and more in 3MB with 2 second boot!

User avatar
Gavinmc42
Posts: 4204
Joined: Wed Aug 28, 2013 3:31 am

Re: userland graphics driver (not firmware or kernel part)

Thu Sep 01, 2016 2:24 am

VCHIQ and JPEG/H264/MJPEG VC4 interface firmware is in the start.elf file?
So Eric's stuff just opens up the graphics display side?

Camera sort of working or at least VC4 is displaying dma buffer images?
http://anholt.livejournal.com/46702.html

So to understand the camera VCHIQ interface we need to look at vc, /raspistill/vid and picamera source?
More clues here?
viewtopic.php?f=71&t=156557&p=1021453&h ... Q#p1021453
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

Morin
Posts: 2
Joined: Mon Aug 29, 2016 6:29 am

Re: userland graphics driver (not firmware or kernel part)

Thu Sep 01, 2016 6:35 am

Ultibo wrote:
Morin wrote:My ultimate goal is to get communication with the GPU working on bare metal (without Linux), but with the closed firmware blob intact
There seems to be quite a few of us who have a similar goal, just to get bare metal communication with the firmware blob in the same way Linux does. It would be great if all of these efforts could be shared in some way.
Agreed.
jamesh wrote:The real issue is wading through the complexity of the userland libraries and the VCHIQ driver to work out how it all fits together and find the bits needed to get the communication working.
Exactly. I tried to start by digging down a specific OpenGL function call ("statically" though, by looking at the code, since I don't have a working Pi ATM). Then I stumpled upon the missing khronos_server_lock_func_table function.

As far as I understand, VCHIQ is a set of processor word granular queues (something like 6 bits queue ID, 26 bits payload, IIRC). The OpenGL message format is probably imposed as a higher level protocol. There is sample code out there that says that the GPU communication uses only one specific queue ID. But I couldn't find any specification of the higher-level message format (which is why I started digging the code in the first place. A wire protocol spec would be fine for me, too).

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24584
Joined: Sat Jul 30, 2011 7:41 pm

Re: userland graphics driver (not firmware or kernel part)

Thu Sep 01, 2016 9:00 am

Ultibo wrote:
Morin wrote:My ultimate goal is to get communication with the GPU working on bare metal (without Linux), but with the closed firmware blob intact
There seems to be quite a few of us who have a similar goal, just to get bare metal communication with the firmware blob in the same way Linux does. It would be great if all of these efforts could be shared in some way.
jamesh wrote:There is a hell of a lot of code required to make that work - MMAL for example, and that requires all sorts of high level OS features
The issue is not a lack of high level OS features since we have all of the necessary things like scheduling, locks, interrupts, timers, tasks, wait queues etc etc etc.

The real issue is wading through the complexity of the userland libraries and the VCHIQ driver to work out how it all fits together and find the bits needed to get the communication working.
The entire ARM side code uses a OS abstraction layer (VCOS) so map your baremetal OS functions on to that API and you will be able to get a lot of code compiling and working. You could probably build the whole of userland fairly easily once VCOS is sorted (it's been done before).

Just some history. Originally, the communications layer was VCHI (I did some work on it back in the day). It was complicated (ISO network layers IIRC), and a PITA to maintain, and a bit slow. Eben then wrote VCHIQ when he realised was a problem the VCHI stack was, loads less code, and quicker.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

Return to “Advanced users”