MarkDarcy
Posts: 21
Joined: Thu Sep 20, 2018 8:23 am

[SOLVED] OpenMAX: ilclient_setup_tunnel() returns -5

Tue Feb 11, 2020 12:59 pm

Hello all,

Hopefully somebody can help here. I am writing an application that's attempting to use two OpenMAX components in a tunnel to transcode JPEG stills/MJPEG frames to H264.

I have two separate test programs that successfully use the image_decode and video_decode components to decode my JPEG stills/MJPEG frames to raw YUV frames in format: OMX_COLOR_FormatYUV420PackedPlanar, so the output port to both components tells me. The linux program mplayer can play the decoded raw YUV frames fine.

I have a third test program which encodes H264 using video_encode. Up until now, I have taken raw YUV from cameras (either YUYV or UYVY), converted to YUV 420 packed planar in software and then fed to video_encode. Perfectly-formed Annex-B H264 has been flowing out of the video_encode component.

Three working test programs. However, when I try and tunnel the output of either video_decode/image_decode to the input of video_encode, ilclient_setup_tunnel() returns -5 (formats incompatible). Enabling logging with

Code: Select all

vcgencmd set_logging level=204
and dump the GPU with

Code: Select all

sudo vcdbg log msg
and the following errors appear in the log:

Code: Select all

729851.544: image_decode:30:ComponentTunnelRequest(321,setup)
729851.568: image_decode:30:CTR: Output port 321 is not supplier
729851.688: video_encode:31:GetParameter(7f000003)
729851.710: video_encode:31:ComponentTunnelRequest(200,setup)
729851.734: image_decode:30:GetParameter(2000001)
729851.753: video_encode:31:RIL:GetParameter(2000001)
729851.775: image_decode:30:SetParameter(7f00000c)
729851.792: image_decode:30:GetParameter(2000001)
729851.810: video_encode:31:RIL:SetParameter(2000001)
729851.828: video_encode:31:RIL: tunnel: could not set local portdef
729851.849: video_encode:31:ComponentTunnelRequest:ports not compatible (verify_format failed)
So does anybody know:

1) Why is port 321 not considered a supplier when it is the output port of the image_decode component?
2) How do I make it look like a supplier to video_encode?

Environment:

- Raspberry Pi 3 Model B Plus Rev 1.3
- Linux raspberry-1 4.14.79-v7+ #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l GNU/Linux
- OMX Version: 1.1.2 step 0

Pseudo-code:

Code: Select all

TUNNEL_T aomxTunnel[2];

ilclient_create_component(pomxImgDec, "image_decode", ILCLIENT_DISABLE_ALL_PORTS | ILCLIENT_ENABLE_INPUT_BUFFERS);
ilclient_create_component(pomxVidEnc, "video_encode", ILCLIENT_DISABLE_ALL_PORTS | ILCLIENT_ENABLE_OUTPUT_BUFFERS);

set_tunnel(&aomxTunnel[0], pomxImgDec, 321, pomxVidEnc, 200);
set_tunnel(&aomxTunnel[1], NULL,         0, NULL,         0);

OMX_GetParameter(ILC_GET_HANDLE(pomxImgDec), ...);
OMX_SetParameter(ILC_GET_HANDLE(pomxImgDec), ...);
OMX_GetParameter(ILC_GET_HANDLE(pomxVidEnc), ...);
OMX_SetParameter(ILC_GET_HANDLE(pomxVidEnc), ...);

ilclient_change_component_state(pomxImgDec, OMX_StateIdle)
ilclient_enable_port_buffers(pomxImgDec, 320, NULL, NULL, NULL);
ilclient_change_component_state(pomxImgDec, OMX_StateExecuting);

bPortChanged = false;

while (not end of input data)
{
    if ((pomxDecInBuf = ilclient_get_input_buffer(pomxImgDec, 320, true)) != NULL)
    {
        load next frame into buffer;
        OMX_EmptyThisBuffer(ILC_GET_HANDLE(pomxImgDec), pomxDecInBuf);
    }

    if (!bPortChanged)
    {
        if (!ilclient_remove_event(aomxTunnel[0].source, OMX_EventPortSettingsChanged, aomxTunnel[0].source_port, 0, 0, 1))
        {
            ilclient_change_component_state(pomxVidEnc, OMX_StateIdle);
            ilclient_enable_port_buffers(pomxVidEnc, 201, NULL, NULL, NULL);

            ilclient_setup_tunnel(&aomxTunnel[0], 0, 0);

            ...

            bPortChanged = true;
        }
    }

    get output buffer from 201 and process (code hasn't got here yet);
}
Thanks in advance,
Last edited by MarkDarcy on Tue Feb 18, 2020 5:56 pm, edited 1 time in total.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7889
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Tue Feb 11, 2020 2:29 pm

You generally don't need to care about who the buffer supplier is on a tunnel - it's magic within the core.

Your issue is "RIL: tunnel: could not set local portdef"
When setting up the tunnel the sink port gets the port format from the source port and tries applying it locally. Something in that process is saying they aren't compatible. If you can dump out and post the full OMX_PARAM_PORTDEFINITIONTYPE structure, then I can do an analysis to see what it doesn't like. The two ports should be compatible.

BTW video_encode natively supports the YUYV family of colour formats. It does have a requirement that nStride is aligned to a multiple of 32, but otherwise it should just take them.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

MarkDarcy
Posts: 21
Joined: Thu Sep 20, 2018 8:23 am

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Tue Feb 11, 2020 3:08 pm

Hello 6by9,

Thanks for your suggestions.

As this is a fixed-width pipeline, my code can set (and does set) the source and sink of the tunnel connector before image_decode is moved into Idle state. The sink (port 200 of video_encode) is configured using:

Code: Select all

objParamPort.format.video.nFrameWidth  = uWidth;
objParamPort.format.video.nFrameHeight = uHeight;
objParamPort.format.video.xFramerate   = (uint32_t) (dFrameRate * 65536.0);
objParamPort.format.video.nStride      = (uWidth  + 0x1F) & 0xFFFFFFE0;
objParamPort.format.video.nSliceHeight = (uHeight + 0x0F) & 0xFFFFFFF0;
objParamPort.format.video.eColorFormat = OMX_COLOR_FormatYUV420PackedPlanar;
As you suggested, I am already setting the stride to a multiple of 32. I am also setting the slice height to a single slice of a multiple of 16 which, I understand, is a pre-requisite of video_encode for certain input formats. My test program for H.264 decoding would not have worked (== "did not work") without these basic settings. I also tried executing a OMX_GetParameter on port 321 and re-applying exact structure unmodified to port 200 once the OMX_EventPortSettingsChanged event had been delivered (this was shown in one of your examples) and, while there was no error during the OMX_SetParameter, the tunnel still could not be set up.

One thing is curious, though. I don't know if this is related but I will let you know anyway. Please do not let it sidetrack you if it is not important.

The output port of image_decode seems to report two different image formats depending on the query method. If I query port 321 with OMX_IndexParamImagePortFormat, and receive a OMX_IMAGE_PARAM_PORTFORMATTYPE structure, the eColorFormat field returns OMX_COLOR_Format16bitRGB565. However, if I query the port with OMX_IndexParamPortDefinition and receive a OMX_PARAM_PORTDEFINITIONTYPE structure, the objParamPort.format.video.eColorFormat field returns OMX_COLOR_FormatYUV420PackedPlanar.

Why the discrepancy when querying the same format information through two different methods on the same port? It would appear that the information returned by one of these query methods is saying the image format for the output of the image decoder is matched to to the format of the video encoder, correct? Is this phenomenon related to my problem? Maybe not... but even so, I would appreciate knowing what is causing this discrepancy all the same, as it it may be related to the format mismatch I am seeing.

Thanks in advance,

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7889
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Tue Feb 11, 2020 3:30 pm

Please provide the port definition of [image|video]_decode's output port at the point that you try creating the tunnel.
As I said, the process of creating the tunnel will cause the sink to query the source for the port format, regardless of what you have set it to previously.

OMX_IndexParamImagePortFormat enumerates the supported formats by incrementing nIndex, as noted by the structure

Code: Select all

/**  
 * Port format parameter.  This structure is used to enumerate the various 
 * data input/output format supported by the port.
 * 
 * STRUCT MEMBERS:
 *  nSize              : Size of the structure in bytes
 *  nVersion           : OMX specification version information
 *  nPortIndex         : Indicates which port to set
 *  nIndex             : Indicates the enumeration index for the format from 
 *                       0x0 to N-1
 *  eCompressionFormat : Compression format used in this instance of the 
 *                       component. When OMX_IMAGE_CodingUnused is specified, 
 *                       eColorFormat is valid
 *  eColorFormat       : Decompressed format used by this component
 */
typedef struct OMX_IMAGE_PARAM_PORTFORMATTYPE {
    OMX_U32 nSize;
    OMX_VERSIONTYPE nVersion;
    OMX_U32 nPortIndex;
    OMX_U32 nIndex;
    OMX_IMAGE_CODINGTYPE eCompressionFormat;
    OMX_COLOR_FORMATTYPE eColorFormat;
} OMX_IMAGE_PARAM_PORTFORMATTYPE;
That list for image_decode will be (in order)

Code: Select all

OMX_COLOR_Format16bitRGB565
OMX_COLOR_FormatYUV420PackedPlanar
OMX_COLOR_FormatYUV422PackedPlanar
OMX_COLOR_Format32bitABGR8888
OMX_COLOR_Format32bitARGB8888
Presumably your structure will have left nIndex at 0, hence getting the first entry.
Note that not all formats will be available for all compressed input formats. The codecs write directly into the output buffers, therefore it depends on the codec.
Bizarrely setting this parameter does appear to configure the port definition. More joined up thinking in the IL spec :(

OMX_IndexParamPortDefinition sets/queries the port definition.
Note that image_decode has eDomain = OMX_PortDomainImage, therefore you should be looking at objParamPort.format.image.eColorFormat not .video. It matters as OMX_IMAGE_PORTDEFINITIONTYPE is missing the nBitrate and xFramerate fields, and therefore the offset for eCompressionFormat and eColorFormat changes.
Input ports for image and video domain components should accept the opposite domain, although they may well do the conversion to their native domain as part of the SetParameter.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

MarkDarcy
Posts: 21
Joined: Thu Sep 20, 2018 8:23 am

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Tue Feb 11, 2020 4:12 pm

Hello 6by9,

Thanks for your reply.

With regards to the behaviour of the OMX_IndexParamImagePortFormat message, understood. This explains why the format always comes out as OMX_COLOR_Format16bitRGB565 when I query it. It is because the enumeration index is always zero when I query it. I assumed that the OMX_GetParameter request on OMX_IndexParamImagePortFormat returned the current status, and was not used for enumerating current capabilities. My misunderstanding.

You asked for the status of the ports at the moment that ilclient_setup_tunnel() is called; I have provided the status below. The following code:

Code: Select all

memset(&objParamPort, 0, sizeof(objParamPort));
objParamPort.nSize             = sizeof(objParamPort);
objParamPort.nVersion.nVersion = OMX_VERSION;
objParamPort.nPortIndex        = 321;
OMX_GetParameter(ILC_GET_HANDLE(pomxImgDec), OMX_IndexParamPortDefinition, &objParamPort);

printf("main() - >> IMG domain [%i] size [%ux%u] stride [%u] slice [%u] comp [0x%08X] colour [0x%08X]\n",
       objParamPort.eDomain,
       objParamPort.format.image.nFrameWidth,
       objParamPort.format.image.nFrameHeight,
       objParamPort.format.image.nStride,
       objParamPort.format.image.nSliceHeight,
       objParamPort.format.image.eCompressionFormat,
       objParamPort.format.image.eColorFormat);

memset(&objParamPort, 0, sizeof(objParamPort));
objParamPort.nSize             = sizeof(objParamPort);
objParamPort.nVersion.nVersion = OMX_VERSION;
objParamPort.nPortIndex        = 200;
OMX_GetParameter(ILC_GET_HANDLE(pomxVidEnc), OMX_IndexParamPortDefinition, &objParamPort);

printf("main() - >> VID domain [%i] size [%ux%u] stride [%u] slice [%u] comp [0x%08X] colour [0x%08X]\n",
       objParamPort.eDomain,
       objParamPort.format.video.nFrameWidth,
       objParamPort.format.video.nFrameHeight,
       objParamPort.format.video.nStride,
       objParamPort.format.video.nSliceHeight,
       objParamPort.format.video.eCompressionFormat,
produces:

Code: Select all

20240990.125: main() - >> IMG domain [2] size [1280x720] stride [1280] slice [720] comp [0x00000000] colour [0x00000014]
20240990.320: main() - >> VID domain [1] size [1280x720] stride [1280] slice [720] comp [0x00000000] colour [0x00000014]
The domains appear to be correct for the settings I have dumped. Do any other fields come into play when matching port formats? They appear to be matched, and yet ilclient is whinging. Incidentally, in the original log message, what exactly is producing the "could not set local portdef" error? If I knew under which situations this error was emitted I could probably deduce the missing information on my part.

Thanks in advance,

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7889
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Tue Feb 11, 2020 4:32 pm

MarkDarcy wrote:
Tue Feb 11, 2020 4:12 pm

Code: Select all

20240990.125: main() - >> IMG domain [2] size [1280x720] stride [1280] slice [720] comp [0x00000000] colour [0x00000014]
20240990.320: main() - >> VID domain [1] size [1280x720] stride [1280] slice [720] comp [0x00000000] colour [0x00000014]
That looks reasonable, although you haven't dumped things like the buffer sizes which are also important. Stopping in gdb via a breakpoint, and using "p objParamPort" would be the quickest way to get it all.
MarkDarcy wrote:The domains appear to be correct for the settings I have dumped. Do any other fields come into play when matching port formats? They appear to be matched, and yet ilclient is whinging. Incidentally, in the original log message, what exactly is producing the "could not set local portdef" error? If I knew under which situations this error was emitted I could probably deduce the missing information on my part.
You're calling ilclient_setup_tunnel, that calls OMX_SetupTunnel.
That's passed across the IPC to the GPU to setup the tunnel. That calls a function that is part of video_encode to verify the format. Predominantly that calls GetParameter(OMX_IndexParamPortDefinition) on the source port, overwrites the port and direction, and then calls SetParameter(OMX_IndexParamPortDefinition) on the sink port. SetParameter(OMX_IndexParamPortDefinition) always sanity checks the port definition, and something in that sanity check is failing. There is no logging to report why it failed, just that it judged the format to be invalid.

As a simple test you could try calling amending the second part of your quoted test code to amend the port number, and call OMX_SetParameter(ILC_GET_HANDLE(pomxVidEnc), OMX_IndexParamPortDefinition, &objParamPort);
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

MarkDarcy
Posts: 21
Joined: Thu Sep 20, 2018 8:23 am

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Tue Feb 11, 2020 5:42 pm

Hello 6by9,

As indicated in a previous reply, I did try simply taking the output port definition of the decoder, changing the port number and re-applying it to the video encoder as you suggested. This policy succeeds (that is, I can apply the settings) if I do not move video_encode into Idle or allocate port buffers before I try to perform OMX_IndexParamPortDefinition. If I have already moved to Idle on video_encode and have allocated buffers, then attempting to modify the port definition for 200 (video_encode input) fails.

I dumped every field for both the source and sink on the tunnel connector as you suggested. The following code:

Code: Select all

memset(&objParamPort, 0, sizeof(objParamPort));
objParamPort.nSize             = sizeof(objParamPort);
objParamPort.nVersion.nVersion = OMX_VERSION;
objParamPort.nPortIndex        = 321;
OMX_GetParameter(ILC_GET_HANDLE(pomxImgDec), OMX_IndexParamPortDefinition, &objParamPort);

printf("main() - >> IMG dir [%i] bufcount-act [%u] bufcount-min [%u] buffersize [%u] enabled [%s] populated [%s]\n",
       objParamPort.eDir,
       objParamPort.nBufferCountActual,
       objParamPort.nBufferCountMin,
       objParamPort.nBufferSize,
       objParamPort.bEnabled ? "TRUE" : "FALSE",
       objParamPort.bPopulated ? "TRUE" : "FALSE");
printf("main() - >> IMG domain [%i] size [%ux%u] stride [%u] slice [%u] comp [0x%08X] colour [0x%08X]\n",
       objParamPort.eDomain,
       objParamPort.format.image.nFrameWidth,
       objParamPort.format.image.nFrameHeight,
       objParamPort.format.image.nStride,
       objParamPort.format.image.nSliceHeight,
       objParamPort.format.image.eCompressionFormat,
       objParamPort.format.image.eColorFormat);

memset(&objParamPort, 0, sizeof(objParamPort));
objParamPort.nSize             = sizeof(objParamPort);
objParamPort.nVersion.nVersion = OMX_VERSION;
objParamPort.nPortIndex        = 200;
OMX_GetParameter(ILC_GET_HANDLE(pomxVidEnc), OMX_IndexParamPortDefinition, &objParamPort);

printf("main() - >> VID dir [%i] bufcount-act [%u] bufcount-min [%u] buffersize [%u] enabled [%s] populated [%s]\n",
       objParamPort.eDir,
       objParamPort.nBufferCountActual,
       objParamPort.nBufferCountMin,
       objParamPort.nBufferSize,
       objParamPort.bEnabled ? "TRUE" : "FALSE",
       objParamPort.bPopulated ? "TRUE" : "FALSE");
printf("main() - >> VID domain [%i] size [%ux%u] stride [%u] slice [%u] comp [0x%08X] colour [0x%08X]\n",
       objParamPort.eDomain,
       objParamPort.format.video.nFrameWidth,
       objParamPort.format.video.nFrameHeight,
       objParamPort.format.video.nStride,
       objParamPort.format.video.nSliceHeight,
       objParamPort.format.video.eCompressionFormat,
       objParamPort.format.video.eColorFormat);
produces:

Code: Select all

24628929.671: main() - >> IMG dir [1] bufcount-act [1] bufcount-min [1] buffersize [1382400] enabled [FALSE] populated [FALSE]
24628929.704: main() - >> IMG domain [2] size [1280x720] stride [1280] slice [720] comp [0x00000000] colour [0x00000014]
24628929.893: main() - >> VID dir [0] bufcount-act [1] bufcount-min [1] buffersize [1382400] enabled [FALSE] populated [FALSE]
24628929.923: main() - >> VID domain [1] size [1280x720] stride [1280] slice [720] comp [0x00000000] colour [0x00000014]
If, in between the dump of port 321 and port 200 in the code above, I insert:

Code: Select all

objParamPort.nPortIndex        = 200;
OMX_SetParameter(ILC_GET_HANDLE(pomxVidEnc), OMX_IndexParamPortDefinition, &objParamPort);
then OMX_SetParameter returns 0x80001005 (OMX_ErrorBadParameter). Could this be because the domains are different? Or maybe, as I stated above, I have already moved to state Idle on video_encode so the settings cannot be updated? (in official code samples I have seen this style of coding before and the programs compile and appear to work). The thing is, I am setting up video_encode port 200 to the expected input format in the video domain before any components are moved to executing. So, if I do nothing, I would expect this format to already be present. The thing is, originally doing nothing is what produced the "format mismatch" error in the first place. What might be the missing link here?

Thanks in advance,

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7889
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Tue Feb 11, 2020 6:32 pm

AFAIK there are no hello_pi examples that use video_encode with tunnelled input. Hello_encode certainly doesn't.

Some components do allocations on moving from loaded to idle, even if the ports are disabled. I'd suggest you remain at loaded whilst the tunnel is setup, and let the call to ilclient_setup_tunnel call ilclient_enable_tunnel, which will move the sink component to idle anyway.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

MarkDarcy
Posts: 21
Joined: Thu Sep 20, 2018 8:23 am

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Wed Feb 12, 2020 9:45 am

Hello 6by9,

Apologies for the delayed response.

Thanks for all your suggestions. I have spent time trying different combinations of encoder initialisation, state changes, buffer enabling, but no matter what I try I get the same -5 error from ilclient_setup_tunnel().

As a sanity check, I wrote a third test program to use source (OMX.broadcom.source) to generate "swirly" frames and feed those to the encoder. I successfully read frames from the component, saved them in YUV420 Packed Planar format and played them with mplayer. However, when I attempted to tunnel the output of source to a video_encode that had exactly the same input format, the same -5 error popped out. With this latest attempt, as the source component has no input port, I assumed I was able to enable the tunnel directly after enabling the component, in the same way that the official hello_video example does with the clock component.

That now makes a total of three test programs that all work and three programs that can't tunnel into video_encode. However, I'm not the only one. This person was trying the same thing as me in 2013 and got the same error. He/she hasn't received a response in six years.

So, I appreciate you're busy, but would it at all be possible to confirm the following before I proceed any further with my investigation?

1) Have you (Raspberry Pi, Broadcom, etc.) ever got video_encode to be used as the sink end of a tunnel connector on the Pi using any interface? (ILClient, MMAL, some other interface)?
2) Does the design of video_encode component actually support tunnelling or has it just been crippled to return an error regardless?

With regard to my test programs, I have no qualms with releasing the source code to you personally (not posting here has an attachment) if you feel that taking a look at the code or building locally may help you to establish what is causing the "format mismatch" error.

Thanks in advance,

dickon
Posts: 702
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Wed Feb 12, 2020 9:52 am

MarkDarcy wrote:
Wed Feb 12, 2020 9:45 am
1) Have you (Raspberry Pi, Broadcom, etc.) ever got video_encode to be used as the sink end of a tunnel connector on the Pi using any interface? (ILClient, MMAL, some other interface)?
I have. See https://github.com/dickontoo/omxtx or https://github.com/dickontoo/omxmotion/ ... mxmotion.c. Not particularly pretty, but they were fun toys to write.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7889
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Wed Feb 12, 2020 11:23 am

Stackoverflow is not our support forum. Questions asked there may get assistance from those who participate there, but not from Pi Towers.
Technically I'm not paid for forum support, but it avoids the wheelie chair swordfighting.

Yes, video_encode previously supported tunnelling, as dickon's reported. Transcode was one of the normal test cases. video_encode has been updated relatively recently (about 6 months back), so it's possible there's a regression.

IL is an absolute pain of an API, so I'm not going to spend time generating a test case. If you want something specific investigated then it needs to be something I can compile and run quickly. Easiest is probably a private github repo and add me (6by9) as a collaborator. Any investigation is still as time allows.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7889
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Wed Feb 12, 2020 11:56 am

Total hack job, take hello_video, and replace video_scheduler with video_encode

Code: Select all

diff --git a/host_applications/linux/apps/hello_pi/hello_video/video.c b/host_applications/linux/apps/hello_pi/hello_video/video.c
index 2edf44c..25c722f 100755
--- a/host_applications/linux/apps/hello_pi/hello_video/video.c
+++ b/host_applications/linux/apps/hello_pi/hello_video/video.c
@@ -89,19 +89,19 @@ static int video_decode_test(char *filename)
       status = -13;
 
    // create video_scheduler
-   if(status == 0 && ilclient_create_component(client, &video_scheduler, "video_scheduler", ILCLIENT_DISABLE_ALL_PORTS) != 0)
+   if(status == 0 && ilclient_create_component(client, &video_scheduler, "video_encode", ILCLIENT_DISABLE_ALL_PORTS) != 0)
       status = -14;
    list[3] = video_scheduler;
 
-   set_tunnel(tunnel, video_decode, 131, video_scheduler, 10);
-   set_tunnel(tunnel+1, video_scheduler, 11, video_render, 90);
-   set_tunnel(tunnel+2, clock, 80, video_scheduler, 12);
+   set_tunnel(tunnel, video_decode, 131, video_scheduler, 200);
+   set_tunnel(tunnel+1, video_scheduler, 201, video_render, 90);
+//   set_tunnel(tunnel+2, clock, 80, video_scheduler, 12);
 
    // setup clock tunnel first
-   if(status == 0 && ilclient_setup_tunnel(tunnel+2, 0, 0) != 0)
-      status = -15;
-   else
-      ilclient_change_component_state(clock, OMX_StateExecuting);
+//   if(status == 0 && ilclient_setup_tunnel(tunnel+2, 0, 0) != 0)
+//      status = -15;
+//   else
+//      ilclient_change_component_state(clock, OMX_StateExecuting);
 
    if(status == 0)
       ilclient_change_component_state(video_decode, OMX_StateIdle);
The tunnel from video_decode to video_encode is established, but it then fails setting up the tunnel from video_encode to video_render (line 148) as video_render can't take encoded video.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7889
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Wed Feb 12, 2020 12:08 pm

For reference, the port format that it was quite happy with was

Code: Select all

nSize = 96 (0x60)
nVersion = {s = {nVersionMajor = 1; nVersionMinor = 1; nRevision = 2; nStep = 0}; nVersion = 131329 (0x20101)}
nPortIndex = 200 (0xc8)
eDir = 0 (OMX_DirInput)
nBufferCountActual = 1
nBufferCountMin = 1
nBufferSize = 3133440 (0x2fd000)
bEnabled = 0 (OMX_FALSE)
bPopulated = 0 (OMX_FALSE)
eDomain = 1 (OMX_PortDomainVideo)
format = 
	audio = {cMIMEType = 0; pNativeRender = 0; bFlagErrorConcealment = 1920; eEncoding = 1080}
	video = 
		cMIMEType = 0
		pNativeRender = 0
		nFrameWidth = 1920 (0x780)
		nFrameHeight = 1080 (0x438)
		nStride = 1920 (0x780)
		nSliceHeight = 1088 (0x440)
		nBitrate = 0
		xFramerate = 0
		bFlagErrorConcealment = 0 (OMX_FALSE)
		eCompressionFormat = 0 (OMX_VIDEO_CodingUnused)
		eColorFormat = 20 (OMX_COLOR_FormatYUV420PackedPlanar)
		pNativeWindow = 0
	image = {cMIMEType = 0; pNativeRender = 0; nFrameWidth = 1920 (0x780); nFrameHeight = 1080 (0x438); nStride = 1920 (0x780); nSliceHeight = 1088 (0x440); bFlagErrorConcealment = 0 (OMX_FALSE); eCompressionFormat = 0 (OMX_IMAGE_CodingUnused); eColorFormat = 0 (OMX_COLOR_FormatUnused); pNativeWindow = 0}
	other = {eFormat = 0 (OMX_OTHER_FormatTime)}
bBuffersContiguous = 0 (OMX_FALSE)
nBufferAlignment = 16 (0x10)
(Dumped from the VPU debugger. GDB is going to be different in formatting).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

MarkDarcy
Posts: 21
Joined: Thu Sep 20, 2018 8:23 am

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Wed Feb 12, 2020 12:12 pm

Hello 6by9, dickon

Many thanks for your replies.

>> dickon
Thanks very much for sharing the link to your example. I successfully built your program (omxtx) and ran it. It does what says on the tin. What I have noticed, though, is that you don't use the IL client library at all. I am using the client IL library provided as standard with the Pi so it may be that the standard IL client library contains a bug that you are circumventing. The only possibility right now (which I will try) is to junk the IL client library and see if it makes a difference.

>> 6by9
Understood. Regarding Stack Overflow, apologies if appeared I was having a dig. I wasn't; I was just trying to show that the problem is probably that difficult to solve (or inherently not solvable) that his question has not attracted a single hint at a reply from anybody in six years. However, thank you for your offer of assistance. If I ask for assistance you'll get a full compilation environment in a tar complete with test data.

Thanks for that quick hack. I took my local copy of hello_video.c, applied the hack and ran it. Unlike your environment, however, the call to ilclient_setup_tunnel() at line 145 hangs in my environment. The input file I used was a small 1280x720p raw H264 (Annex B) file that came out of, as it happens, dickon's program. Other files I have lying around, though, also cause the same result.

I am inclined to believe that the library I have built locally may contain a bug, or maybe it's the way I built it locally (structure packing, for example?) But then, nothing would work. Or, the firmware you are using is different from mine? (see top of this thread for my environment version).

Any thoughts?

Thanks in advance,

MarkDarcy
Posts: 21
Joined: Thu Sep 20, 2018 8:23 am

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Wed Feb 12, 2020 12:18 pm

>> 6by9
I just dumped the OMX_PARAM_PORTDEFINITIONTYPE structure size and it is 96 bytes, which matches the size displayed in your port dump. So I don't think structure packing is the problem.

MarkDarcy
Posts: 21
Joined: Thu Sep 20, 2018 8:23 am

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Wed Feb 12, 2020 12:26 pm

>> 6by9
If it is not too much of an inconvenience, would you mind sharing the binary of the hello_video hack you have created? Static link it to ensure there are no external dependencies. I could run it on my end and see if it proceeds as far as it did at your end. At minimum, we could be able to narrow it down to whether it is something "environmental" or not.

Thanks in advance,

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7889
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Wed Feb 12, 2020 12:30 pm

I'm running on a Pi4, admittedly with development firmware and kernel 5.4.8-v7l, but none of that should matter.
Tested with the 1080p30 and 720p versions of Big Buck Bunny and both fail at the "if(ilclient_setup_tunnel(tunnel+1, 0, 1000) != 0)" at line 148.

ilclient should build with the correct options by default. Do ensure you have the define OMX_SKIP64BIT set to avoid structure alignment issues (it should be in Makefile.include in the hello_pi directory).

IL is an absolute pig of an API. All the event completition callbacks make it difficult for the application developer, and there isn't a nice way of finding out which critical step was missed. That's a large part of the reason that Broadcom developed and switched to MMAL.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

dickon
Posts: 702
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Wed Feb 12, 2020 12:45 pm

MarkDarcy wrote:
Wed Feb 12, 2020 12:12 pm
Hello 6by9, dickon

Many thanks for your replies.

>> dickon
Thanks very much for sharing the link to your example. I successfully built your program (omxtx) and ran it. It does what says on the tin.
Yup. When it builds -- ie, when libav* haven't changed the API arbitrarily yet *again* for no obvious good reason -- it more or less just works. I wrote it in the summer / autumn of 2012 for the original Pi 1, and it's still working now on a Pi 4. It's broadly obsolete these days, but it's a reasonable is-my-Pi-firmware-broken test.
What I have noticed, though, is that you don't use the IL client library at all.
Yes, and that's deliberate. I wanted to learn a 'standard' API, rather than a Broadcom-specific shim, so did it with OMX IL. OMX IL is, as 6by9 says, a right royal PITA, and I wouldn't bother now: it's an obsolete API, with all sorts of really grim misfeatures, limitations, and downright stupidities. You did, however, ask two questions that omxtx (and omxmotion, an arguably more entertaining application) answers: has anyone got it working as a sink via *any interface* (yes, OMX IL), and has it been crippled (no).
I am using the client IL library provided as standard with the Pi so it may be that the standard IL client library contains a bug that you are circumventing. The only possibility right now (which I will try) is to junk the IL client library and see if it makes a difference.
Good luck.

MarkDarcy
Posts: 21
Joined: Thu Sep 20, 2018 8:23 am

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Wed Feb 12, 2020 1:48 pm

Hello 6by9,
I'm running on a Pi4, admittedly with development firmware and kernel 5.4.8-v7l, but none of that should matter.
Tested with the 1080p30 and 720p versions of Big Buck Bunny and both fail at the "if(ilclient_setup_tunnel(tunnel+1, 0, 1000) != 0)" at line 148.
Just tested with Buck Bunny 1080p and the hello_video program I built doesn't even run. The port changed notification (lines 136-138) which is meant to drive the tunnel setup code isn't even getting delivered. I am now seriously suspecting something environmental; either (1) firmware or (2) build environment.
ilclient should build with the correct options by default. Do ensure you have the define OMX_SKIP64BIT set to avoid structure alignment issues (it should be in Makefile.include in the hello_pi directory).
Yes, I have been defining OMX_SKIP64BIT from the start. I am sorry to be an imposition but it would really help if I could narrow down why it is that when we compile the same program (hello_video) we get two different results. Here are the potential ways that would yield some movement I think:

1) Could you please run your hello_video binary, statically linked, on a Pi 3 B+ (Rev. 1.3) that you may ;-) just have lying about.
2) Could you please post the build command line that you used to build hello_video. I could try it on my end.

Thanks in advance,

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7889
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Wed Feb 12, 2020 2:12 pm

I've thrown my binary at https://github.com/6by9/RPiTest/blob/ma ... _video.bin
ilclient should be statically linked, but it does request dynamically linked versions of bcmhost, vcos, and a few others.

It's just a straight clone of the userland tree.
For reasons I don't fully understand, they've linked directly to the version of ilclient that is stored at /opt/vc/src/hello_pi/libs/ilclient, instead of the version in the current workspace.

Code: Select all

cd /opt/vc/src/hello_pi/libs/
sudo make -C ilclient
sudo make -C revision
sudo make -C vgfont
cd ~
git clone https://github.com/raspberrypi/userland
cd ~/userland/host_applications/linux/apps/hello_pi
./rebuild.sh
will rebuild all the hello_pi apps, or cd hello_video and run make there for just hello_video. Use the resulting hello_video.bin file. I get it waiting at line 185 for the buffer flag that will never come as it didn't abort as the second tunnel wasn't created.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

MarkDarcy
Posts: 21
Joined: Thu Sep 20, 2018 8:23 am

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Thu Feb 13, 2020 7:31 am

Hello 6by9,

Thanks very much for sharing the binary and build procedure. I know it seems like we are being distracted by trivia here but the cause of many problems like we're seeing now usually lies in the detail.

Since we last spoke, I have:

1. Tried the binary you posted

Unfortunately, it hung in the same manner as the binary I had built locally (as the build procedure was assumed to be the same this was not unexpected). I have attached a ZIP file which contains the output of sudo vcdbg log msg for your binary and the source code to the hacked hello_video that I have been using. In the log, if you search for "tunnel" you will notice that the GPU never receives a tunnel setup request from your binary; only tunnel tear-down requests. Therefore, it can be concluded that your binary hangs in the same place as my binary, as it is setting up the tunnel.

My version of hello_video should be the same as yours except with a few printf()'s. It gets to point (4) and then hangs (line 139).

2. Pulled the latest tree from userland and rebuilt test programs

Having refreshed my local clone of userland, I rebuilt the ilclient library, relinked the test programs (and hacked hello_video) but they all exhibited the same broken behaviour.

3. Performed an apt full-upgrade and rebuilt test programs

It has been a while since I did a system update so I performed one, from release 9.6 to 9.11. It looked promising as there appeared to be firmware updates included. I then rebuilt ilclient from userland and relinked. Alas, all test programs still exhibited the same behaviour (e.g. hacked hello_video hung on line 139, ilclient_setup_tunnel()). The new system now looks like:

Code: Select all

$ cat /proc/cpuinfo
processor       : 0
model name      : ARMv7 Processor rev 4 (v7l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

<processors #1...#3 the same as above>

Hardware        : BCM2835
Revision        : a020d3
Serial          : 000000007#####44

$ cat /sys/firmware/devicetree/base/model
Raspberry Pi 3 Model B Plus Rev 1.3

$ uname -a
Linux raspberry-1 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 9.11 (stretch)
Release:        9.11
Codename:       stretch

$ grep "define  *OMX_VERSION_" userland/interface/vmcs_host/khronos/IL/OMX_Types.h
#define OMX_VERSION_MAJOR 1
#define OMX_VERSION_MINOR 1
#define OMX_VERSION_REVISION 2
#define OMX_VERSION_STEP 0
The situation we have now is that our build environments are the same but the binaries you build in your environment do not behave the same in my environment. In fact, the binaries you build fail in my environment in exactly the same way as the binaries I build locally. Our build environments are therefore probably not the cause.

When you ran the binary you shared with me, you said it was on a Pi 4. Did you manage to try it on a Pi 3 B/B+? Using the system information above, may I trespass upon your time a little longer and ask if you could do that test? The system image I am using is minimal. It is the Raspbian "Stretch Lite" image which I am running as a headless system. Hardly any applications are installed as it has no GUI; just enough SDKs to get stuff built.

Thanks in advance,
Attachments
tunnel_problem-001.zip
(17.33 KiB) Downloaded 2 times

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7889
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Thu Feb 13, 2020 10:54 am

Pi3. Just done a "sudo SKIP_KERNEL=1 rpi-update" to get the latest clean firmware

Code: Select all

pi@raspberrypi:~/RPiTest $ vcgencmd version
Feb 12 2020 12:39:27 
Copyright (c) 2012 Broadcom
version 53a54c770c493957d99bf49762dfabc4eee00e45 (clean) (release) (start_x)
pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.19.93-v7+ #1286 SMP Mon Jan 6 13:18:44 GMT 2020 armv7l GNU/Linux
If you're still on Stretch, then upgrade.

A standard hello_video plays back the clip fine. Run hello_video.bin from RPiTest repo and it stalls at

Code: Select all

(gdb) bt
#0  0x76ec5088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x76f0d68c <vcos_thread_main+12>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x76f0d68c <vcos_thread_main+12>, abstime=0x0) at sem_waitcommon.c:115
#2  0x76ec51f4 in __new_sem_wait_slow (sem=sem@entry=0x76f0d68c <vcos_thread_main+12>, abstime=0x0) at sem_waitcommon.c:282
#3  0x76ec528c in __new_sem_wait (sem=sem@entry=0x76f0d68c <vcos_thread_main+12>) at sem_wait.c:42
#4  0x76ef8d90 in vcos_semaphore_wait (sem=0x76f0d68c <vcos_thread_main+12>) at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_platform.h:254
#5  _vcos_thread_sem_wait () at /home/dom/projects/staging/userland/interface/vcos/pthreads/vcos_platform.h:646
#6  vcos_generic_event_flags_get (flags=0x33918, bitmask=32, op=5, suspend=4294967295, retrieved_bits=0x7effefd0)
    at /home/dom/projects/staging/userland/interface/vcos/generic/vcos_generic_event_flags.c:223
#7  0x00012210 in vcos_event_flags_get (flags=0x33918, requested_events=32, op=5, suspend=4294967295, retrieved_events=0x7effefd0)
    at /opt/vc/include/interface/vcos/generic/vcos_generic_event_flags.h:118
#8  0x00014f80 in ilclient_wait_for_event (comp=0x33900, event=OMX_EventBufferFlag, nData1=90, ignore1=0, nData2=1, ignore2=0, event_flag=32, suspend=-1) at ilclient.c:1155
#9  0x00012004 in video_decode_test (filename=0x7efff3e9 "../1.h264") at video.c:185
#10 0x0001213c in main (argc=2, argv=0x7efff234) at video.c:219
same as my Pi4.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

MarkDarcy
Posts: 21
Joined: Thu Sep 20, 2018 8:23 am

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Thu Feb 13, 2020 2:14 pm

Hello 6by9,

Thanks very much for confirming that behaviour on the Pi 3. I shall try your procedure and get back to you.

Thanks in advance,

MarkDarcy
Posts: 21
Joined: Thu Sep 20, 2018 8:23 am

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Fri Feb 14, 2020 4:51 pm

Hello 6by9,

Just a quick update. Much as I would have liked, I didn't get the opportunity to check this today. My apologies. I should have an opportunity to look at it over the weekend and then I can get back to you.

Regards,

MarkDarcy
Posts: 21
Joined: Thu Sep 20, 2018 8:23 am

Re: OpenMAX: ilclient_setup_tunnel() returns -5

Sun Feb 16, 2020 2:30 pm

Hello 6by9,

Apologies for the delay, and for the rather long post. Here is my report.

I've had a chance to retry the tests in the environment that you suggested. The environment now looks like:

Code: Select all

$ vcgencmd version
Feb 12 2020 12:38:08 
Copyright (c) 2012 Broadcom
version 53a54c770c493957d99bf49762dfabc4eee00e45 (clean) (release) (start)

$ uname -a
Linux raspberry-1 4.19.97-v7+ #1294 SMP Thu Jan 30 13:15:58 GMT 2020 armv7l GNU/Linux

$ cat /sys/firmware/devicetree/base/model
Raspberry Pi 3 Model B Plus Rev 1.3

$ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 10 (buster)
Release:        10
Codename:       buster
I tried the hacked hello_video and, in the same way as occurred in your tests, the tunnel setup on line 139 now succeeds. The program now hangs at line 185 in hello_video.c with a stack trace identical to the one you posted above. We have the same environment.

However, while it now appears that the call to ilclient_setup_tunnel() is succeeding in the hacked program, alas, I could not reproduce this success with my actual program. The only difference should be that the actual program applies settings to the ports of the components being tunnelled whereas the hacked test program does not. I did investigate further, restructuring the program, the API call order, port enable/disable timings, etc. but to no avail.

In the back of my mind, though, was the knowledge that dickon's program did manage it. So, I scrapped IL client and rewrote my application from scratch again using pure OMX. This time I successfully got the tunnel up and running.

Now the interesting part. The code that actually configured the component ports in both the non-working IL client version and the pure OMX version was identical; I simply copied and pasted the code verbatim. And yet, one works and the other doesn't. The only difference with the pure OMX version is that I get to control exactly when the ports are enabled/disabled and when buffers are allocated as opposed to having IL client do it for me. As a last ditch attempt with IL client, I brought back all the timing techniques from the pure OMX version (when ports are enabled/disabled, when buffers are allocated, etc.) and using IL client calls I tried to make it work. But still, the IL client-based program will not setup a tunnel for me. It either fails with "format incompatible" or hangs.

Here, then, is the current situation with regards to my use case (MJPEG -> H264 transcoding):

1) Tunnelling in pure OMX works

This is confirmed by both dickon's program and my own, completely differently designed program. I didn't derive my program from dickon's. Instead, it was intentionally a complete ground-up development as I wanted to confirm for myself exactly what the Pi's interpretation of the OMX specification was. However, both styles of OMX API usage yielded working programs.

2) Tunnelling in IL Client does not work

I have tried three different systems (Stretch Rev. 9.6, Rev. 9.11, Buster Rev. 10) and on all systems my IL-client-based programs have failed to work. However, my pure-OMX-based program does work on all these systems. It therefore looks highly likely that the IL client appears to be causing the problem. Furthermore, I have not seen a single article on the net of anybody claiming to have achieved my use case; only people stumped at not being able to achieve it, which leads me to believe that no-one has yet been able to do it.

So, while I have found a solution, it is in my view far from satisfactory, as it appears that the only solution to not being able to setup a tunnel in IL client for my use case is not to use IL client at all, but to use the lower-down OMX interface instead, ironically both opposite halves of the same specification.

I thank you very much for your efforts in helping me trying to debug my application but, at the end of the day, would you say this is a fair appraisal of the situation?

Thanks in advance,

Return to “Advanced users”