rasmus25
Posts: 13
Joined: Mon Feb 04, 2013 3:53 pm

[Solved with workaround] EGLImage to video_encode

Wed Jul 10, 2013 12:02 pm

We have all seen the excellent demo where a video is rendered on a teapot by passing an EGLImage as an output buffer to the egl_render component. I thought that using an EGLImage as an input to the video_encode component would be as simple as that. I was wrong. UseEGLImage() is not implemented in video_encode.

So here's the question: how much work would it be for the Foundation to implement passing EGLImage to video_encode? I would think that it would be less than 20 lines of code. However, somebody created an entire new component egl_render, instead of implementing UseEGLImage() in video_render, which makes me believe that the matter might not be as trivial as I thought.

Just one example of how using EGLImage would help a big part of the community, is colorspace conversion. Right now, if video_encode does not support your camera's colorspace (which it usually doesn't), colorspace conversion will severely slow down your framerates. Even if someone created a component that uses GPU for colorspace conversion, its output could still not be fed directly to video_encode's input without passing through CPU. I see lots of people complaining about lack of good colorspace conversion not only on Raspberry Pi, but on other ARM boards as well.

For a more detailed example, consider my use case. I have an USB camera that has no encoder on board. From the camera, I get a 1280x960 image in Bayer pattern. I then use Morgan McGuire's shaders from http://graphics.cs.williams.edu/papers/BayerJGT09/ to demosaic the image into a 1280x960 RGBA image. I have adapted these shaders to OpenGL ES, so they work on Raspberry Pi. Then I download the image to CPU and immediately re-upload to the GPU for video_encode, using glReadPixels and OMX_EmptyThisBuffer, respectively. This useless last step takes around 92 ms, which reduces the overall framerate to 10 FPS. And it's not even Full HD.

Shaders are a very powerful concept. Instead of colorspace conversion or demosaicing, people could add all kinds of special effects to videos before encoding and saving to disk. Combining different filters, various real time machine vision tasks could be implemented on the Pi etc. For this reason, I would ask that all OpenMAX graphics components should have EGLImage input and output. For selfish reasons ;) , I suggest that video_encode should be the next one getting this feature (after the already implemented egl_render, which is wonderful).
Last edited by rasmus25 on Fri Aug 09, 2013 11:58 am, edited 1 time in total.

rasmus25
Posts: 13
Joined: Mon Feb 04, 2013 3:53 pm

Re: EGLImage to video_encode

Wed Jul 10, 2013 3:03 pm

Just do demonstrate how simple a shader based colorspace conversion would be, look at the code listing 4.1 on page 27 of this student thesis. Just one matrix multiplication, accelerated by GPU.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23981
Joined: Sat Jul 30, 2011 7:41 pm

Re: EGLImage to video_encode

Wed Jul 10, 2013 4:05 pm

I discussed this with someone upstairs a while back, and there were some technical reasons (which i cannot remember) why its a PITA using the current Raspi firmware. IIRC the latest EGL and OpenGLES code from our top of tree is actually much newer than that on the Raspi (since we don't want to break the Raspi firmware, it's not that close to bleeding edge), and this newer has features which make this easier. Apparently. There may be other ways of doing it - I haven't investigated further than a quick chat - I wanted to put the output of the camera on to a texture on to the teapot.

I do not know what the schedule for Raspi firmware updates is.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

rasmus25
Posts: 13
Joined: Mon Feb 04, 2013 3:53 pm

Re: EGLImage to video_encode

Wed Jul 10, 2013 6:16 pm

Thanks for the quick reply! I can't blame you for not having implemented this feature, as it seems that all other single board computers suffer from the same problem. I have looked around and the graphics capabilities of Raspberry Pi appear to be more accessible than other boards out there, despite being quite locked down.

Having a method for pipelining several GPU tasks without the needless copy through CPU is a feature that I would pay several hundred dollars for, and Raspberry Pi seems to be closest to implementing this (proved by the existence of the egl_render component). More expensive boards could do it all in the CPU, but what a waste of money, space and power this would be!

Given how many improvements Raspberry Pi graphics have seen in the past year, I am confident that some day you will look into this problem once again and find a way to solve it. Keep up the good work!

rasmus25
Posts: 13
Joined: Mon Feb 04, 2013 3:53 pm

Re: EGLImage to video_encode

Fri Aug 09, 2013 11:52 am

Hi,

I found a workaround for my problem. I just found out that one can use mmap() to get the address of a framebuffer in the memory and then read out the data directly. This means that I could do away with these extra copies and run my application at the desired framerate. I could render to the framebuffer and pass the framebuffer to the video encoder (I have not tested this, but seems likely).

Here's a little test program that I patched together from code samples on the Internet.

Code: Select all

#include <unistd.h>
#include <stdio.h>
#include <fcntl.h>
#include <linux/fb.h>
#include <sys/mman.h>
#include <sys/ioctl.h>
#include <stdlib.h>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>

using namespace cv;

int main()
{
        int fbfd = 0;
        long int screensize = 656*416*2; // 656x416 R5G6B5
        char *fbp = 0;
        //Open the file for reading
        fbfd = open("/dev/fb0", O_RDWR);

        //Unblank the screen, otherwise it will be just black
        ioctl(fbfd, FBIOBLANK, FB_BLANK_UNBLANK);
        //Map the device to memory
        fbp = (char *)mmap(0, screensize, PROT_READ | PROT_WRITE, MAP_SHARED,
                fbfd, 0);
        if ((int)fbp == -1) { printf("Error: failed to map  framebuffer device to memory./n");
               exit(4);
        }
        printf("The framebuffer device was mapped to memory successfully./n");
        Mat out(416,656*2, CV_8U);
        memcpy(out.data, fbp, screensize);
        imwrite("out.png", out);
        munmap(fbp, screensize);
        close(fbfd);
        return 0;
}
Save it as main.cpp and compile with this command:

Code: Select all

g++ -o mmaptest -I/usr/include -lopencv_core -lopencv_highgui main.cpp
You need to have libopencv-dev installed for this to compile.

This program will read the framebuffer and save its contents into an image. The image will not look like the screen, as the framebuffer uses R5G6B5 colorspace as default and I did not bother with converting (if you just want a screengrab, use the fbgrab utility from fbcat package). The cool thing about this is that one can use framebuffer to convert between RGB and YUV (planar formats probably not supported). Just use fbset with the right parameters or change the framebuffer parameters with ioctls in the code.

Note: if you get a blank image using the code above, it means the screensaver is active.

Note 2: I moved on to another ARM board before I found out about mmap(), so I will not be developing this any further. But I am happy to answer any questions that I can and point people to resources on the Internet. I am still a fan of Raspberry Pi!

Ed_was_here
Posts: 4
Joined: Thu Oct 18, 2012 1:16 am

Re: EGLImage to video_encode (the workaround does not work)

Wed Aug 21, 2013 8:40 pm

Unfortunately, using data from the framebuffer or vc-mem will cause OMX_EmptyThisBuffer to hang. From the stack trace ilcs_pass_buffer calls ilcs_execute_function_ex which waits on a semaphore that never gets set. I'm not sure if using memory already in the gpu region as the source is prohibited or if it is a special case that isn't handled yet. Can someone more familiar with the hardware please clarify what might cause this?

The code I am using is based on hello_encode and modified to use a RGBA8888 source image from fb0 (vcmem from mmap) or local memory (the "dummy" array with a short test pattern of white dots).
The interesting part is near here: https://github.com/ed-was-here/hello_en ... ode.c#L316

sudo gdb --args ./hello_encode.bin dummy3.dat
run
ctrl+c
thread apply all backtrace
...
Thread 1 (Thread 0xb6ff9000 (LWP 3919)):
#0 0xb6f1b700 in sem_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
#1 0xb6f69b6c in ilcs_execute_function_ex () from /opt/vc/lib/libopenmaxil.so
#2 0xb6f6a61c in ilcs_pass_buffer () from /opt/vc/lib/libopenmaxil.so
#3 0x0000ac0c in video_encode_test (outputfilename=0xbefff938 "dummy3.dat") at encode.c:316
#4 0x0000afac in main (argc=2, argv=0xbefff804) at encode.c:383

All of the other threads are waiting in the same place before calling OMX_EmptyThisBuffer as when the program hangs.
Commenting out line 312 (https://github.com/ed-was-here/hello_en ... ode.c#L312) will encode a few white dots in the upper left hand corner of the screen as a test for using non-gpu memory as the source.

rasmus25
Posts: 13
Joined: Mon Feb 04, 2013 3:53 pm

Re: [Solved with workaround] EGLImage to video_encode

Wed Aug 21, 2013 8:53 pm

Is it possible that fb0 is not RGBA8888? You can check it with fbset and modify as needed. If fb0 was RGB565, for example, then reading width*height*4 from framebuffer would cause reading out of allowed region. Please test and let us know!

Ed_was_here
Posts: 4
Joined: Thu Oct 18, 2012 1:16 am

Re: EGLImage to video_encode

Wed Aug 21, 2013 9:16 pm

fb0 is normally rgb565 but it shouldn't matter for this since reading any data would be useful and the region being read is 640*360 and my frame buffer is 1920x1080, so it is well inside it. I tried it with fbset -depth 32 for a matching format to rule that out as well. I have the test to see if the ARM can read the data (line 313,314) as a sanity check. Also, I mmap'd beyond the size that will be copied to check that also.
Edit:
Interestingly, data up to 3988 bytes (0xF94) can be grabbed fine, but not more. It's a strange number, being 108 bytes short of a full page (4096 bytes). Mapping consecutive pages on the ARM (using the mmap MAP_FIXED flag) to consecutive pages of the vc-mem individually doesn't change the behavior. Also, inside the mapped region when starting from any offset up to 16 bytes before the end of the mapped region, up to 3988 bytes can be read.

Return to “OpenMAX”