doug4350
Posts: 9
Joined: Mon Dec 07, 2015 6:46 pm
Location: United States

OpenMAX - Capturing raw decoded H.264 frames

Sat Sep 22, 2018 7:25 pm

Hi all,

I've been struggling for a while on capturing the decoded frames from a streaming wifi camera. I've gone through numerous google searches and have found a lot of examples on using OpenMax and FFMPEG on the Pi, but none specific to my problem. I thought it would be pretty straight forward using the /opt/vc/src/hello_pi/hello_video example as a template to start from.

It compiles as is (as expected) and runs the test.h264 video file fine. So I retrofitted it to feed from my wifi camera instead of the test.h264 file and it streams & renders on the console. So far, so good. The h.264 frames from the camera feed into the input side of the video_decode component and the empty_buffer_done callback is called as expected.

To capture the raw frames, I thought it would be simple enough to either grab the output buffers from the video_decode component, which didn't work as expected. I modified the example to not create tunnels out of the video_decode component and enable the output port buffers on the video_decode. The closest I got it to work was on the empty_buffer_done callback, I would call ilclient_get_output_buffer(...) to get a video_decode output buffer, which would succeed, followed by OMX_FillThisBuffer(...) if the nFilledLen > 0. It was always zero though.

Thinking there might be some dependency on the clock and video_scheduler component, I also tried grabbing the output buffers from the video_scheduler component. So, I used the original tunnelling code and removed the render component, as well as enabling the video_scheduler output port buffers, but that failed. Here's the code snippet I used for this attempt:

Code: Select all

#define VIDEO_SCHEDULER_IN_10        10
#define VIDEO_SCHEDULER_OUT_11       11
#define CLOCK_OUT_80                 80
#define VIDEO_DECODE_IN_130         130
#define VIDEO_DECODE_OUT_131        131

#define DECODE_TUNNEL_0           (tunnel + 0)
#define SCHEDULER_TUNNEL_1        (tunnel + 1)
#define CLOCK_TUNNEL_1            (tunnel + 1)
#define CLOCK_TUNNEL_2            (tunnel + 2)
#define VIDEO_RENDER_TUNNEL_3     (tunnel + 3)

static int
video_decode_test2()
{

// (init code)
// ...
    if (ilclient_create_component(client, &video_decode, "video_decode", ILCLIENT_DISABLE_ALL_PORTS | ILCLIENT_ENABLE_INPUT_BUFFERS) != 0)
    {
        printf("video_decode_test2(): ERROR - Failed to create video_decode component.\n");
        status = -14;
    }
    list[0] = video_decode;

    // create video_render
    // if (status == 0 && ilclient_create_component(client, &video_render, "video_render", ILCLIENT_DISABLE_ALL_PORTS) != 0)
        // status = -14;
    // list[1] = video_render;

    // create clock.  Renamed to omx_clock because clashing with clock in time.h
    if (status == 0 && ilclient_create_component(client, &omx_clock, "clock", ILCLIENT_DISABLE_ALL_PORTS) != 0)
         status = -14;
    list[1] = omx_clock;

    memset(&cstate, 0, sizeof(cstate));
    cstate.nSize = sizeof(cstate);
    cstate.nVersion.nVersion = OMX_VERSION;
    cstate.eState = OMX_TIME_ClockStateWaitingForStartTime;
    cstate.nWaitMask = 1;
    if (omx_clock != NULL && OMX_SetParameter(ILC_GET_HANDLE(omx_clock), OMX_IndexConfigTimeClockState, &cstate) != OMX_ErrorNone)
        status = -13;

    // create video_scheduler
    if (status == 0 && ilclient_create_component(client, &video_scheduler, "video_scheduler", ILCLIENT_DISABLE_ALL_PORTS | ILCLIENT_ENABLE_OUTPUT_BUFFERS) != 0)
    {
        printf("video_decode_test2(): ERROR - Failed to create video_scheduler component.\n");
        status = -14;
    }
    list[2] = video_scheduler;

    set_tunnel(DECODE_TUNNEL_0,       video_decode,   VIDEO_DECODE_OUT_131, video_scheduler, VIDEO_SCHEDULER_IN_10);
    // set_tunnel(SCHEDULER_TUNNEL_1, video_scheduler, VIDEO_SCHEDULER_OUT_11,    video_render, VIDEO_RENDER_IN_90);
    set_tunnel(CLOCK_TUNNEL_1,           omx_clock,           CLOCK_OUT_80, video_scheduler, VIDEO_SCHEDULER_IN_CLOCK_12);

    // setup clock tunnel first
    if (status == 0 && ilclient_setup_tunnel(CLOCK_TUNNEL_1, 0, 0) != 0)
        status = -15;
    else if (ilclient_change_component_state(omx_clock, OMX_StateExecuting) != 0)
        printf("video_decode_test2(): ERROR - Failed to set clock state to Executing.\n");

    if (status == 0 && ilclient_change_component_state(video_decode, OMX_StateIdle) != 0)
        printf("video_decode_test2(): ERROR - Failed to set video_decode state to Executing.\n");
        
    ilclient_set_fill_buffer_done_callback(client, my_fill_buffer_done_callback, 0);
    ilclient_set_empty_buffer_done_callback(client, my_empty_buffer_done_callback, 0);
// ...
}

static int
handle_port_settings_changed2(int data_len)
{
    OMX_BUFFERHEADERTYPE *out;
    OMX_PARAM_PORTDEFINITIONTYPE portdef;
    int ret1 = -999, ret2 = -999;

    if ((data_len > 0 && (ret1 = ilclient_remove_event(video_decode, OMX_EventPortSettingsChanged, VIDEO_DECODE_OUT_131, 0, 0, 1)) == 0) ||
        (data_len == 0 && (ret2 = ilclient_wait_for_event(video_decode, OMX_EventPortSettingsChanged, VIDEO_DECODE_OUT_131, 0, 0, 1,
                                                  ILCLIENT_EVENT_ERROR | ILCLIENT_PARAMETER_CHANGED, 10000)) == 0))
        printf("handle_port_settings_changed2(): INFO - ***** REMOVED/WAITED for PORT SETTINGS CHANGED EVENT %d / %d data_len: .\n", ret1, ret2, data_len);
    else
        printf("handle_port_settings_changed2(): ERROR - ***** FAILED TO REMOVE/WAIT for PORT SETTINGS CHANGED EVENT %d / %d data_len: .\n", ret1, ret2, data_len);

    if (1)
    {
        if (ilclient_setup_tunnel(DECODE_TUNNEL_0, 0, 0) != 0)
        {
            printf("handle_port_settings_changed2(): ERROR - ***** FAILED TO SETUP DECODE_TUNNEL   *******.\n");
            status = -7;
            return -1;
        }

        ilclient_enable_port(video_scheduler, VIDEO_SCHEDULER_IN_10);
        printf("handle_port_settings_changed2(): INFO - ilclient_enable_port() COMPLETED for scheduler input port.\n");
        
        // This fails:
        if (ilclient_enable_port_buffers(video_scheduler, VIDEO_SCHEDULER_OUT_11, NULL, NULL, NULL) != 0)
            printf("handle_port_settings_changed2(): ERROR - ilclient_enable_port_buffers() failed for scheduler output.\n");
        else
            printf("handle_port_settings_changed2(): INFO - ilclient_enable_port_buffers() enabled for scheduler output.\n");

        if (ilclient_change_component_state(video_scheduler, OMX_StateExecuting) != 0)
            printf("handle_port_settings_changed2(): ERROR - video_scheduler failed to change state to executing.\n");
        else
            printf("handle_port_settings_changed2(): INFO - video_scheduler CHANGED state to executing.\n");

        if ((out = ilclient_get_output_buffer(video_scheduler, VIDEO_SCHEDULER_OUT_11, 0)) != NULL)
            printf("handle_port_settings_changed2(): ERROR - video_scheduler Failed to get output buffer.\n");
        else if (OMX_FillThisBuffer(ILC_GET_HANDLE(video_scheduler), out) != OMX_ErrorNone)
                printf("Failed to FillThisBuffer for VIDEO_SCHEDULER_OUT_11 output buffer.\n");
        else
            printf("handle_port_settings_changed2(): INFO - video_scheduler got output buffer and ready to be filled.\n");

        return 1;
}
        
Apparently, when using tunnelling, fill_buffer_done callback doesn't get called between tunnelled components.?.? At least it's not in my example.

As a side note, from another example, I have been able to get the image_encode component to tunnel from the scheduler and create a jpeg image for each frame, but that's not the purpose. I want to be able to do a couple of things on the raw video frames.

Hope my explanation is clear.
- Doug
"Nothing is impossible if ImPossible"

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6336
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: OpenMAX - Capturing raw decoded H.264 frames

Mon Sep 24, 2018 7:46 pm

If your source is producing data in realtime then there's no need for a scheduler and clock as the incoming data is your clock.

video_decode will consume data until it finds something that it can decode. H264 has header bytes that are needed. There are also a couple of mechanisms for framing the data, with start codes being the most common one for streamed data.

Can you save a sample of the webcam stream and post it to somewhere for analysis?
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

doug4350
Posts: 9
Joined: Mon Dec 07, 2015 6:46 pm
Location: United States

Re: OpenMAX - Capturing raw decoded H.264 frames

Tue Sep 25, 2018 7:51 pm

Thanks for the reply 6by9.
If your source is producing data in realtime then there's no need for a scheduler and clock as the incoming data is your clock.
This is what I thought, but it wasn't working and being on the up slope of the learning curve for OpenMAX & FFMPEG, figured it was worth a try.

Before getting you a streaming clip from my cam, let me try a few things first. I was able to get the test.h264 file to use port buffers from the output of the video_scheduler, basically by removing the video_render tunnel from the video_scheduler and enabling output port buffers on the scheduler. Now, the fill_buffer_done callback is being called as expected. I didn't try decoding the file after I got the video stream to go through video_render & figured I was in the clear.

Will keep you posted.
- Doug
"Nothing is impossible if ImPossible"

doug4350
Posts: 9
Joined: Mon Dec 07, 2015 6:46 pm
Location: United States

Re: OpenMAX - Capturing raw decoded H.264 frames

Fri Sep 28, 2018 4:08 am

An update, which will lead to another question if I can't figure it out...

I got the video_decode output port buffer to start showing signs of life. Through some tricks that kind of make sense now between feeding video_decode input buffer with video stream data until the OMX_EventPortSettingsChanged event happens after deciphering the H.264 SDP handshaking, and monitoring the size of the video_decode output buffer filled, the fill_buffer_done callback is being called several times for a bit. Confusing, I know...

I then had to call ilclient_get_output_buffer with non-blocking on video_decode and check if the size filled was zero until the fill_buffer_done callback was called. After that, I can call ilclient_get_output_buffer with blocking in the fill_buffer_done callback handler and continue handling filling the output buffer in the callback.

I could monitor the ping-pong of empty_buffer_done and fill_buffer_done callbacks being hit, then OMX_EmptyThisBuffer() pauses and fails trying to dump the video_decode input buffer into the decoder. Not sure why yet, but it behaved the same when I was grabbing output buffers from the scheduler when tunneling from the video_decode to the video_scheduler.

By using only the video_decode component, there is no longer any tunneling done. clock and scheduler aren't needed. Only port buffers on video_decode. Simpler, which I like, but not healthy to get it to work.

Progress, but not out of the woods yet.
- Doug
"Nothing is impossible if ImPossible"

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6336
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: OpenMAX - Capturing raw decoded H.264 frames

Fri Sep 28, 2018 1:43 pm

The simple answer is that IL is an absolute swine to get it to work, and does fail in many non-obvious ways. This is one of the main reasons that MMAL was written as a replacement.

IL was supposed to be the standard for multimedia integration on embedded systems, but it has fallen out of favour with almost all adopters. Even back in 2013/4 there were divisions within the working group over whether to try and fix the short-comings within the existing framework, or tear it up to make a framework that had a chance of working. AFAIK It's been pretty much abandoned since.

I know you've been bashing your head against IL. If you want to switch to MMAL then https://github.com/6by9/userland/blob/h ... eo/video.c should read and decode an H264 stream from file, delivering the buffers to your app. Do something with the buffers at line 425.
The pixel data is in buffer->data, length buffer->length, YUV420 planar, width/height are format_out->es->video.crop.width/height, stride is mmal_encoding_width_to_stride(format_out->encoding, format_out->es->video.width), U chroma plane starting at ((uint8*)buffer->data)[stride*format_out->es->video.height], and V chroma plane at U + stride/2*format_out->es->video.height/2.
Tarting up those examples and merging them into userland has been on my todo list for a while.

Please remember that the video codecs are NOT guaranteed to behave as one buffer in leads to one buffer out. (a) they happily handle unframed data, (b) when framed, header bytes don't produce any output, (c) the encoded frame doesn't necessarily fit into one input buffer.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

doug4350
Posts: 9
Joined: Mon Dec 07, 2015 6:46 pm
Location: United States

Re: OpenMAX - Capturing raw decoded H.264 frames

Fri Sep 28, 2018 2:47 pm

Thanks for the insight 6by9.

I've been avoiding switching to MMAL because it is locked in to Broadcom. With the lack of info on the BCM2837, I'm hesitant to do so and could switch to a different architecture if they stay tight lipped. I am impressed with the performance of the GPU and am ok switching as a work around for now and it isn't such a dragon wrestling match as OMX has been. I'd be interested to see if MMAL works on other Pi flavors such as Orange Pi. I purchased one of those a while back because it has a SATA port and a few extra goodies, but haven't tested any video streaming on it until I get something working.
Please remember that the video codecs are NOT guaranteed to behave as one buffer in leads to one buffer out. (a) they happily handle unframed data, (b) when framed, header bytes don't produce any output, (c) the encoded frame doesn't necessarily fit into one input buffer.
I remember reading something about that from another post and am checking the OMX_BUFFERFLAG_ENDOFFRAME and OMX_BUFFERFLAG_EOS flags on each call to the fill_buffer_done callback & handling appropriately. Will do something similar if I switch.

- Doug
"Nothing is impossible if ImPossible"

dickon
Posts: 233
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: OpenMAX - Capturing raw decoded H.264 frames

Fri Sep 28, 2018 3:18 pm

The OrangePi and whatnot are all knock-offs, mostly based on mobile phone chipsets which may have non-Android firmware available, but won't have their GPUs usefully available other than on Android, or on Android kernels with a non-Android userland (which is less useful than you'd think).

Anything with a Mali chipset -- and that's everything but the Pi ATM, AIUI -- is hamstrung in this way. Arm do not release Mali drivers and their support libraries for versions of the SoC that the SoC vendor has [not] purchased, and they're intimately tied to that specific SoC and version. Hence you'll find prehistoric 3.x kernels in builds for these things.

I'm with you on OMX IL, though. Ghastly as it is, it is at least crossish-platform.

-- edited to add the '[not]' missed the first time.
Last edited by dickon on Fri Sep 28, 2018 4:00 pm, edited 1 time in total.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6336
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: OpenMAX - Capturing raw decoded H.264 frames

Fri Sep 28, 2018 3:57 pm

dickon wrote:
Fri Sep 28, 2018 3:18 pm
I'm with you on OMX IL, though. Ghastly as it is, it is at least crossish-platform.
Vaguely cross-platform, although if you speak to some of the gst-omx developers you'll find out just how badly standardised it is as an API, with various quirks required .

We're looking at V4L2 M2M (memory to memory) as being the most sensible ways to support codecs in future, particularly doing zero copy with DRM/KMS via dmabufs. https://github.com/6by9/linux/tree/rpi- ... s-push-pt2 is my development branch implementing it. Other than code review it's pretty much ready to go.
- The OpenElec folk are trying to shift to it (and DRM) at the moment for as many platforms as possible.
- GStreamer already supports it as v4l2[h264|h263|mpeg4|mpeg2]dec and v4l2h264enc, including passing dmabufs.
- FFmpeg pretty much supports V4L2 M2M, and they're trying to merge dmabuf support at the moment.
- https://github.com/6by9/v4l2_m2m was my test app, although I haven't run it for a while. It was the fun of trying to develop a driver with no test app, so writing the test app at the same time, then not knowing whether it was driver or app at fault.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

doug4350
Posts: 9
Joined: Mon Dec 07, 2015 6:46 pm
Location: United States

Re: OpenMAX - Capturing raw decoded H.264 frames

Sat Sep 29, 2018 4:20 am

Gents, I believe I got it working. :o

Definitely some tricky timing to watch out for. I was on the right track checking that the fill_buffer_done callback on the decode_video output buffer was being invoked before calling OMX_EmptyThisBuffer() on the input port.

The rest of the story is not to fill the input buffer with the next encoded frame data until after fill_buffer_done has been called on the output. Waiting for the empty_buffer_done callback wasn't good enough. Probably something to do with zero-copy code stepping on memory and corrupting the data in the internal decode routines.

Anyway, it's been steadily decoding for an hour now without a hitch, but I don't know if it's *really* working until I can validate the frames by seeing if I can re-encode / render them and they look the same as what the camera is pointing at.

{edit} Yes, it is working woohoo! I saved the encoded stream to a file and can view it in VLC.

- Doug
"Nothing is impossible if ImPossible"

Return to “OpenMAX”