Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

video_decode and MJPEG in OMX

Mon Dec 14, 2015 10:05 am

Hi,

previously I successfully made a client application that was able to decode H.264 from an RTP stream using video_decode. Now I have a need to implement dynamic payload switching between MJPEG and H.264. I simply used the existing pipeline and switched the codec to OMX_VIDEO_CodingMJPEG.

The existing code feeds input buffers to video_decode and waits for the output port settings to change to setup the tunnel to video_render. However, even though the video_decode component processes the buffers without any errors no events are dispatched.

So I wanted to ask is waiting for port setting change a correct approach for MJPEG? And what is the expected bitstream format? I am reconstructing full JPEG frames from RTP - creating all the SOI, SOF, SOS headers and passing in the entropy coded scan data. I tried saving the data for a single frame to a .jpeg file an it seems fine. Is the video_decode component expecting full JPEG frames or some other format? Are there any additional options that must be set on video_decode for MJPEG to work?

Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Re: video_decode and MJPEG in OMX

Tue Dec 15, 2015 10:33 am

I tried adding an explicit EOI marker that is not present in RTP but still not luck.

As I read here - viewtopic.php?f=29&t=19334&start=275#p269992 - it is said that only SD resolutions are supported? Is this still true? So 1920x1080 mjpeg picture is not supported?

I tried playing a 320x240 mjpeg AVI file with omxplayer and that worked fine. So I assume some of the markers I am not generating may be needed - like APP0 etc.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7329
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: video_decode and MJPEG in OMX

Tue Dec 15, 2015 10:58 am

MJPEG is hardware accelerated, so Dom's original post there is badly phrased. It should take higher resolutions without issue.

What flags are you adding to the buffers? Have you already parsed the stream into frames and added OMX_BUFFERFLAG_ENDOFFRAME to the nFlags field?
H264 certainly does have parsing within the decoder to find frame boundaries in the absence of flags, but I don't know on MJPEG (I know the encoder siderather than decoder).

If you have the camera module, you can record to MJPEG (easiest via V4L2 as I don't think Raspivid has easily accessible options) and I know those streams decode OK. I haven't looked at what options are turned on/off in those streams.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Re: video_decode and MJPEG in OMX

Tue Dec 15, 2015 11:27 am

Hi 6by9,

nice to hear from you, you already helped a lot with H.264. I am setting the flags - on the first buffer I set the start time flag and on the last the end of frame flag plus I insert an EOI marker.

However I still see two things I can try out - adding the APP0 markers, and a second quantization table - ffmpeg for some reason ignores the RFC and sends only one. I do set the table 0 to be used for all components though, and all image viewers display the JPEG image correctly but maybe OMX is unhappy with that. Also as I said I have a working AVI file so in worst case I can try setting all the markers I see there to make the images equivalent. I just hoped maybe someone knows or there exists some kind of specification of what OMX expects.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7329
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: video_decode and MJPEG in OMX

Tue Dec 15, 2015 12:26 pm

Ruuzis wrote:nice to hear from you, you already helped a lot with H.264. I am setting the flags - on the first buffer I set the start time flag and on the last the end of frame flag plus I insert an EOI marker.

However I still see two things I can try out - adding the APP0 markers, and a second quantization table - ffmpeg for some reason ignores the RFC and sends only one. I do set the table 0 to be used for all components though, and all image viewers display the JPEG image correctly but maybe OMX is unhappy with that. Also as I said I have a working AVI file so in worst case I can try setting all the markers I see there to make the images equivalent. I just hoped maybe someone knows or there exists some kind of specification of what OMX expects.
Sorry, that's getting to details of the decoder I don't know and won't have a chance to look into until at least tonight.
You could see if there is anything useful in the additional logging.

Code: Select all

vcgencmd set_logging level=204
should turn on both codecs and IL level logging, and use "sudo vcdbg log msg" to view those logs.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Re: video_decode and MJPEG in OMX

Tue Dec 15, 2015 1:19 pm

Thanks for the info about additional logging - I'll definitely check that out.

Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Re: video_decode and MJPEG in OMX

Wed Dec 16, 2015 12:20 am

I tried enabling the 204 log level but did not find anything valuable. Then to eliminate the possibilities of errors in defragmentation of the RTP stream I simply tried to load a JPEG file in a single memory buffer, set OMX_BUFFERFLAG_STARTTIME | OMX_BUFFERFLAG_ENDOFFRAME | OMX_BUFFERFLAG_SYNCFRAME flags, but things did not get better. Then I compared the messages between OMX player on an MJPEG AVI file and my code.

My code (single preloaded 880x628 JPEG image):
3266689.344: video_decode:12:EmptyThisBuffer(1f5f5900,59717,50->130)
3266689.363: video_decode:12: queueBuffer (input : 59717)
3266689.376: video_decode:12:qb RIL:1
3266689.455: video_decode:12:RIL: setup image requirements (880x628(880x628)x15)
3276519.080: video_decode:12:EmptyThisBuffer(1f5f5880,59717,48->130)
3276519.099: video_decode:12: queueBuffer (input : 59717)
3276519.111: video_decode:12:qb RIL:2
3276519.297: video_decode:12:cb:consumed_input_buffer(1f5f5880,0b,0f) RIL:2
3280122.877: video_decode:12:EmptyThisBuffer(1f5f5880,59717,48->130)
3280122.894: video_decode:12: queueBuffer (input : 59717)
3280122.908: video_decode:12:qb RIL:2
3280123.076: video_decode:12:cb:consumed_input_buffer(1f5f5880,0b,0f) RIL:2
3282059.399: video_decode:12:EmptyThisBuffer(1f5f5880,59717,48->130)
3282059.418: video_decode:12: queueBuffer (input : 59717)
3282059.430: video_decode:12:qb RIL:2
3282059.599: video_decode:12:cb:consumed_input_buffer(1f5f5880,0b,0f) RIL:2
3306900.472: video_decode:12:EmptyThisBuffer(1f5f5880,59717,48->130)
3306900.489: video_decode:12: queueBuffer (input : 59717)
OMX (MJPEG AVI sample):
2138851.494: video_decode:63:EmptyThisBuffer(1f57daa0,24356,0->130)
2138851.513: video_decode:63: queueBuffer (input : 24356)
2138851.526: video_decode:63:qb RIL:1
2138851.607: video_decode:63:RIL: setup image requirements (320x240(320x240)x15)
2138852.715: video_decode:63:EmptyThisBuffer(1f6274a0,24390,0->130)
2138852.735: video_decode:63: queueBuffer (input : 24390)
2138852.749: video_decode:63:qb RIL:2
2138852.789: video_decode:63:RIL: extract from fifo: len:24356 flags:12 time:0
2138852.824: video_decode:63:RIL: decoding 24356 bytes, 0us, flags 12, (cb:0x1f5eab6c)
Apparently judging from the last two log lines OMX is getting further ahead already on the first frame. Any hints on why?

Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Re: video_decode and MJPEG in OMX

Thu Dec 17, 2015 10:04 am

Could this be related to extradata? Omxplayer uses ffmpeg and thus is able to pass in codec extradata extracted by ffmpeg, but I have no idea what is that in case of MJPEG - Huffmann tables, quantization tables or both? I'll try building omxplayer from sources and inspecting what exactly is it passing to the decoder.

One more interesting thing - omxplayer does not play a single jpeg file. Maybe it is not supposed to since that is not a video player, but still.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7329
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: video_decode and MJPEG in OMX

Thu Dec 17, 2015 12:01 pm

AFAIK there is no extra data for MJPEG - it really is just a stream of concatenated JPEG encoded frames. All tables etc should be contained within each frame.

I've not had a chance to run anything Pi related for a few days so haven't looked into this - decode was never really my area.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Re: video_decode and MJPEG in OMX

Sat Dec 19, 2015 2:45 pm

Ok I finally got to something interresting - after queuing 29 frames the decoder spit out an error that it needs full frames - I am feeding it a full frame in each buffer plus setting the OMX_BUFFERFLAG_ENDOFFRAME flag on each one (I assume this is approved by the 16->130 value in each EmptyThisBuffer invocation. Any hints what could be wrong? The frame itself is a normal JPEG file, all viewers are able to show it - I also did inspect the markers in it and found nothing wrong.
563529.760: OMX.broadcom.video_decode:init
563529.989: top:create_component video_decode
563530.372: video_decode:RIL:GetParameter(1f5f23c4)
563530.389: video_decode:RIL:GetParameter(1f5f23c4)
563530.404: video_decode:RIL:GetParameter(1f5f23c4)
563530.420: video_decode:RIL:GetParameter(1f5f23c4)
563530.451: video_decode:RIL:GetParameter(1f5f23c4)
563530.469: video_decode:RIL:GetParameter(1f5f23c4)
563530.484: video_decode:RIL:GetParameter(1f5f23c4)
563530.500: video_decode:RIL:GetParameter(1f5f23c4)
563530.515: video_decode:RIL:GetParameter(1f5f23c4)
563530.532: video_decode:RIL:GetParameter(1f5f23c4)
563530.547: video_decode:RIL:GetParameter(1f5f23c4)
563530.572: video_decode:RIL:SetParameter(7f0000a9)
563530.587: video_decode:1:SetCallbacks
563530.606: video_decode:1:SetParameter(7f000001)
563530.622: video_decode:1:GetParameter(7f000002)
563530.643: video_decode:1:SetParameter(7f000004)
563531.316: video_decode:1:SetCallbacks
583958.379: video_decode:1:GetParameter(2000001)
583958.396: video_decode:1:RIL:GetParameter(1f5f23c4)
583958.695: video_decode:1:SendCommand(2,130,1f568574) Disable 130
583958.715: video_decode:1:sending cmd 2 to 130
583958.746: video_decode:1:starting command 2
583958.762: video_decode:1:RIL:GetParameter(1f5f23c4)
583958.792: video_decode:1:processingCommand=0
583959.361: video_decode:1:GetParameter(2000001)
583959.376: video_decode:1:RIL:GetParameter(1f5f23c4)
583960.625: video_decode:1:SendCommand(2,131,1f568574) Disable 131
583960.645: video_decode:1:sending cmd 2 to 131
583960.674: video_decode:1:starting command 2
583960.691: video_decode:1:RIL:GetParameter(1f5f23c4)
583960.721: video_decode:1:processingCommand=0
583962.181: video_decode:1:GetState(1)
583962.612: video_decode:1:SendCommand(0,2,1f568574) State to idle
583962.641: video_decode:1:starting command 0
583962.659: video_decode:1:RIL:GetParameter(1f5f23c4)
583962.679: video_decode:1:RIL:GetParameter(1f5f23c4)
583962.698: video_decode:1:completed state transition to OMX_StateIdle
583962.722: video_decode:1:processingCommand=0
583964.046: video_decode:1:GetParameter(6000001)
583964.066: video_decode:1:RIL:GetParameter(1f5f23c4)
583964.491: video_decode:1:SetParameter(6000001)
583964.509: video_decode:1:RIL:SetParameter(6000001)
583964.901: video_decode:1:GetParameter(2000001)
583964.918: video_decode:1:RIL:GetParameter(1f5f23c4)
583967.075: video_decode:1:SetParameter(2000001)
583967.093: video_decode:1:RIL:SetParameter(2000001)
583967.108: video_decode:1:RIL:PortDef, port 130
583967.121: video_decode:1:RIL: set input format
583967.536: video_decode:1:SetParameter(7f000063)
583967.553: video_decode:1:RIL:SetParameter(7f000063)
583967.922: video_decode:1:SetParameter(7f000080)
583967.939: video_decode:1:RIL:SetParameter(7f000080)
583968.310: video_decode:1:GetParameter(7f0000d3)
583968.328: video_decode:1:RIL:GetParameter(1f5f23c4)
583968.804: video_decode:1:SetParameter(7f0000d3)
583968.822: video_decode:1:RIL:SetParameter(7f0000d3)
583968.839: video_decode:1:RIL: setting timestamp fifo mode 1
583969.255: video_decode:1:GetParameter(2000001)
583969.270: video_decode:1:RIL:GetParameter(1f5f23c4)
583969.836: video_decode:1:SendCommand(3,130,1f568574) Enable 130
583969.855: video_decode:1:sending cmd 3 to 130
583969.883: video_decode:1:starting command 3
583970.164: video_decode:1:RIL:GetParameter(1f5f23c4)
583970.183: video_decode:1:processingCommand=1
583970.865: video_decode:1:UseBuffer(130)
583971.509: video_decode:1:UseBuffer(130)
583972.187: video_decode:1:UseBuffer(130)
583972.817: video_decode:1:UseBuffer(130)
583973.444: video_decode:1:UseBuffer(130)
583974.089: video_decode:1:UseBuffer(130)
583974.752: video_decode:1:UseBuffer(130)
583975.362: video_decode:1:UseBuffer(130)
583975.988: video_decode:1:UseBuffer(130)
583976.604: video_decode:1:UseBuffer(130)
583977.336: video_decode:1:UseBuffer(130)
583977.939: video_decode:1:UseBuffer(130)
583978.559: video_decode:1:UseBuffer(130)
583979.157: video_decode:1:UseBuffer(130)
583979.966: video_decode:1:UseBuffer(130)
583980.594: video_decode:1:UseBuffer(130)
583981.225: video_decode:1:UseBuffer(130)
583981.838: video_decode:1:UseBuffer(130)
583982.469: video_decode:1:UseBuffer(130)
583983.062: video_decode:1:UseBuffer(130)
583983.096: video_decode:1:processingCommand=0
583984.445: video_decode:1:GetState(2)
583984.982: video_decode:1:SendCommand(0,3,1f568574) State to executing
583985.005: video_decode:1:starting command 0
583985.026: video_decode:1:completed state transition to OMX_StateExecuting
583985.050: video_decode:1:processingCommand=0
586738.961: video_decode:1:EmptyThisBuffer(1f5f6200,70086,18->130)
586738.979: video_decode:1: queueBuffer (input : 70086)
586738.991: video_decode:1:qb RIL:1
586739.073: video_decode:1:RIL: setup image requirements (1920x1080(1920x1080)x15)
587255.381: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
587255.397: video_decode:1: queueBuffer (input : 70086)
587255.410: video_decode:1:qb RIL:2
587255.604: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
595119.843: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
595119.862: video_decode:1: queueBuffer (input : 70086)
595119.875: video_decode:1:qb RIL:2
595119.991: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
596184.992: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
596185.010: video_decode:1: queueBuffer (input : 70086)
596185.021: video_decode:1:qb RIL:2
596185.219: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
596913.282: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
596913.300: video_decode:1: queueBuffer (input : 70086)
596913.314: video_decode:1:qb RIL:2
596913.429: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
597522.303: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
597522.320: video_decode:1: queueBuffer (input : 70086)
597522.332: video_decode:1:qb RIL:2
597522.525: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
597986.082: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
597986.101: video_decode:1: queueBuffer (input : 70086)
597986.115: video_decode:1:qb RIL:2
597986.231: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
603840.198: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
603840.216: video_decode:1: queueBuffer (input : 70086)
603840.227: video_decode:1:qb RIL:2
603840.423: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
604417.057: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
604417.076: video_decode:1: queueBuffer (input : 70086)
604417.090: video_decode:1:qb RIL:2
604417.196: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
604978.304: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
604978.320: video_decode:1: queueBuffer (input : 70086)
604978.333: video_decode:1:qb RIL:2
604978.524: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
605537.531: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
605537.549: video_decode:1: queueBuffer (input : 70086)
605537.563: video_decode:1:qb RIL:2
605537.678: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
606319.764: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
606319.782: video_decode:1: queueBuffer (input : 70086)
606319.794: video_decode:1:qb RIL:2
606320.002: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
606930.534: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
606930.552: video_decode:1: queueBuffer (input : 70086)
606930.566: video_decode:1:qb RIL:2
606930.679: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
607483.268: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
607483.285: video_decode:1: queueBuffer (input : 70086)
607483.298: video_decode:1:qb RIL:2
607483.488: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
607946.672: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
607946.691: video_decode:1: queueBuffer (input : 70086)
607946.704: video_decode:1:qb RIL:2
607946.819: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
608372.271: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
608372.289: video_decode:1: queueBuffer (input : 70086)
608372.300: video_decode:1:qb RIL:2
608372.491: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
608745.855: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
608745.873: video_decode:1: queueBuffer (input : 70086)
608745.886: video_decode:1:qb RIL:2
608745.992: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
615088.871: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
615088.887: video_decode:1: queueBuffer (input : 70086)
615088.900: video_decode:1:qb RIL:2
615089.091: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
615754.432: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
615754.451: video_decode:1: queueBuffer (input : 70086)
615754.464: video_decode:1:qb RIL:2
615754.579: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
616346.235: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
616346.253: video_decode:1: queueBuffer (input : 70086)
616346.265: video_decode:1:qb RIL:2
616346.455: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
616914.420: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
616914.438: video_decode:1: queueBuffer (input : 70086)
616914.452: video_decode:1:qb RIL:2
616914.572: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
617539.233: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
617539.250: video_decode:1: queueBuffer (input : 70086)
617539.262: video_decode:1:qb RIL:2
617539.455: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
618122.293: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
618122.313: video_decode:1: queueBuffer (input : 70086)
618122.326: video_decode:1:qb RIL:2
618122.441: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
618698.086: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
618698.104: video_decode:1: queueBuffer (input : 70086)
618698.115: video_decode:1:qb RIL:2
618698.304: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
619266.356: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
619266.374: video_decode:1: queueBuffer (input : 70086)
619266.388: video_decode:1:qb RIL:2
619266.503: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
625427.335: video_decode:1:EmptyThisBuffer(1f5f6180,70086,16->130)
625427.352: video_decode:1: queueBuffer (input : 70086)
625427.364: video_decode:1:qb RIL:2
625427.571: video_decode:1:cb:consumed_input_buffer(1f5f6180,0b,0f) RIL:2
626203.549: video_decode:1:EmptyThisBuffer(1f5f6100,70086,16->130)
626203.569: video_decode:1: queueBuffer (input : 70086)
626203.582: video_decode:1:qb RIL:2
626203.699: video_decode:1:cb:consumed_input_buffer(1f5f6100,0b,0f) RIL:2
626722.297: video_decode:1:EmptyThisBuffer(1f5f6100,70086,16->130)
626722.314: video_decode:1: queueBuffer (input : 70086)
626722.327: video_decode:1:qb RIL:2
626722.522: video_decode:1:cb:consumed_input_buffer(1f5f6100,0b,0f) RIL:2
627170.860: video_decode:1:EmptyThisBuffer(1f5f6100,70086,16->130)
627170.879: video_decode:1: queueBuffer (input : 70086)
627170.892: video_decode:1:qb RIL:2
627171.005: video_decode:1:cb:consumed_input_buffer(1f5f6100,0b,0f) RIL:2
627562.903: video_decode:1:EmptyThisBuffer(1f5f6100,70086,16->130)
627562.919: video_decode:1: queueBuffer (input : 70086)
627562.932: video_decode:1:qb RIL:2
627562.974: video_decode:1:RIL: extract from fifo: len:2032494 flags:80000000 time:0
627563.005: video_decode:1:RIL: decoding 2032494 bytes, 0us, flags 80000000, (cb:0x1f5f284c)
627563.016: mjpg_dec_frame: requires whole frames
627563.034: video_decode:1:RIL: dec_frame failed (-1073807358)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7329
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: video_decode and MJPEG in OMX

Sat Dec 19, 2015 10:58 pm

I suspect 2032494 bytes is the size of the input buffer. You've filled all of that, so it has no choice other to present it to the codec, which then complains that it hasn't been told it is a full frame, hence the error.
I don't have any answers as to why the data hasn't been presented to the codec before that poin.
One thing you could try is presenting an empty buffer with OMX_BUFFERFLAG_CODECCONFIG | OMX_BUFFERFLAG_ENDOFFRAME - it may just be a state thing, and it is waiting for that before starting decoding.

Hmm, just looked at the logging line for EmptyBuffer
printf("%s:%sThisBuffer(%p,%ld,%ld->%ld)", pPrivate->component_name,
eDir == OMX_DirInput ? "Empty" : "Fill", pBuffer,
pBuffer ? pBuffer->nFilledLen : 0,
pBuffer ? pBuffer->nOutputPortIndex : 0,
pBuffer ? pBuffer->nInputPortIndex : 0);
So the 16 you're seeing is in nOutputPortIndex , not in nFlags. Double check your code populating the buffer headers.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Re: video_decode and MJPEG in OMX

Mon Dec 21, 2015 11:47 pm

Hi,

thanks for pointing this out. Well if that is correct I am messing something up - but not sure what for now. It seems that what I am setting in flags goes to the input port - I am setting the right fields in my C++ code, I am never touching the output port field. What is weird the log shows the same for H.264 but it works and renders. I'll try to see if this can be something that has happened because of C++ struct packing or something but it is definitely weird.

Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Re: video_decode and MJPEG in OMX

Tue Dec 22, 2015 12:17 am

Ok for now I am totally puzzled - when I pass an empty buffer to OMX_UseBuffer the nSize field is set to 72, while compiling this:
#include <OMX_Core.h>
#include <stdio.h>

int main(int argc, char** argv)
{
OMX_BUFFERHEADERTYPE BufferHeader;
printf("sizeof(OMX_BUFFERHEADERTYPE) == %d", sizeof(BufferHeader));
return 0;
}
gives 80 for both C and C++. Judging from the definition it should be 72. So then Can I ask what compilation flags are used for the firmware side? Do they include any packing restrictions?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7329
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: video_decode and MJPEG in OMX

Tue Dec 22, 2015 12:32 pm

Ruuzis wrote:Ok for now I am totally puzzled - when I pass an empty buffer to OMX_UseBuffer the nSize field is set to 72, while compiling this:
#include <OMX_Core.h>
#include <stdio.h>

int main(int argc, char** argv)
{
OMX_BUFFERHEADERTYPE BufferHeader;
printf("sizeof(OMX_BUFFERHEADERTYPE) == %d", sizeof(BufferHeader));
return 0;
}
gives 80 for both C and C++. Judging from the definition it should be 72. So then Can I ask what compilation flags are used for the firmware side? Do they include any packing restrictions?
I'd agree that it should be 72. OMX structures almost always includes all packing fields required for alignment so as to avoid mismatches on platforms.

What are you compiling on? Bumping up to 80 sounds like OMX_U8* pBuffer and OMX_HANDLETYPE (typedef'ed to a void*) have become 64bit pointers. Potential fall out from cross compiling on a 64bit machine?
There are the hello_pi test apps which use OMX (eg https://github.com/raspberrypi/userland ... ello_video) - confirm compiler options with those.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Re: video_decode and MJPEG in OMX

Tue Dec 22, 2015 2:04 pm

As strange as it may sound I was compiling on a raspberry - even though with gcc 4.8 instead of the default gcc. When I get come I'll reconfirm the flags and see where this is coming from but it could solve all the weird stuff I've been seing. Even though - it is very weird that H.264 worker even with this. Thanks for the feedback.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5331
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: video_decode and MJPEG in OMX

Tue Dec 22, 2015 4:28 pm

OMX headers use an ifdef to indicate if platform is using 64-bit types. For historical reasons we define OMX_SKIP64BIT on both the firmware and user code.
See: https://github.com/raspberrypi/firmware ... include#L2

The absence of that define may make the firmware misinterpret your OMX structures.

Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Re: video_decode and MJPEG in OMX

Tue Dec 22, 2015 8:24 pm

Hi dom,

thanks for pointing this out - total failure from my side to notice this flag. If this turns out to be the cause of all this evil I apologize to you and 6by9 for not being careful enough and wasting your time:)

Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Re: video_decode and MJPEG in OMX

Tue Dec 22, 2015 10:33 pm

Yeah, that was it. Thank you both for your time - I would not have found this in a near future by myself:D

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7329
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: video_decode and MJPEG in OMX

Wed Dec 23, 2015 8:47 am

Glad it works. I
I suspect h264 works as there is a stream parser that will find the frame boundaries if they aren't provided, but the Mjpeg codec doesn't.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Re: video_decode and MJPEG in OMX

Wed Dec 23, 2015 10:58 pm

One more thing I wanted to ask since this is now working - if I am using a VNC style approach and rendering not entire JPEG frames but an initial full sized JPEG image that would correspond to a full frame and then smaller JPEGs that present the modified rectangles - what is my best bet to achieve the best performance? Using egl_render and the output from video_decode or image_decode (actually now I dont see a difference between these two for JPEG) to update the egl texture or is there some way I can still work with video_render? I don't see a way how I could tell video_render to present the buffer at a given offset and with a lower size - or can this be done with some cropping mechanism?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7329
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: video_decode and MJPEG in OMX

Thu Dec 24, 2015 10:36 am

video_render is just giving DispmanX elements to compose onto the screen. It is not writing to a cumulative framebuffer.
You could create a video_render component for each update, but you'll hit the limit of what can be composed on the fly (about 4 overlapping layers at any point on the screen if at 1080P).
The display location can be set using the cropping region - OMX_IndexConfigDisplayRegion and OMX_CONFIG_DISPLAYREGIONTYPE (https://github.com/raspberrypi/userland ... com.h#L250)

Sorry, I don't know enough about EGL and egl_render to say how efficient that would be either. I would suspect that it also kept all the images around as source material, and composed every frame from the sum of them all.

If you're only blitting small updates around, you'll probably find that the ARM is powerful enough to do it, particularly if you're on a Pi2 and can use Neon. I'm assuming you're using YUV formats, so libyuv has probably done most of the work for you, though you might need to hack it slightly. OMX is already providing you raw pixel buffers, so you're free to amend the contents in any way you like before passing to video_render.
Do remember that you'll need at least 2 full sized buffers so video_render can double buffer the display, so you may need to decode the reference frame twice. Don't try submitting the same buffer to a component before it has been returned to you!
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Re: video_decode and MJPEG in OMX

Sat Dec 26, 2015 9:05 pm

Hi 6by9,

thanks for the info. Yeah when thinking about this I also came to the conclusion that I could just decode the small fragment and simply memcpy it to the needed region in the buffer for video_render if that does not consume too much resources (in case of many wide lines) - I'll give that a shot and I'll look at OMX_CONFIG_DISPLAYREGIONTYPE.

Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Re: video_decode and MJPEG in OMX

Mon Dec 28, 2015 11:12 am

One more thing - if I use one video_decoder and set OMX_IndexConfigDisplayRegion before each new buffer is passed in would that result in a horrific slowdown? If so memcpy()'ing and providing full buffers would make most sense. If that still seems slow I will try glTexSubImage2D() on an egl surface to update the regions that are decoded.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5331
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: video_decode and MJPEG in OMX

Mon Dec 28, 2015 6:27 pm

Ruuzis wrote:One more thing - if I use one video_decoder and set OMX_IndexConfigDisplayRegion before each new buffer is passed in would that result in a horrific slowdown?
Not sure that would be useful. Only the active display region will be visible. If you set OMX_IndexConfigDisplayRegion to a quarter of the screen, three-quarters will be blank.
If so memcpy()'ing and providing full buffers would make most sense. If that still seems slow I will try glTexSubImage2D() on an egl surface to update the regions that are decoded.
I'd start with memcpy. It may be fine. memcpy is pretty well optimised for large blocks, so full width copies should be fast. Tall/narrow regions would be less efficient.

Return to “Graphics, sound and multimedia”