Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Behavior of MMAL_VIDEO_NALUNITFORMAT_FOURBYTEINTERLEAVELENGT

Tue Feb 07, 2017 12:13 am

I was expecting that MMAL_VIDEO_NALUNITFORMAT_FOURBYTEINTERLEAVELENGTH would work in a similar manner as in mp4 where the 4 byte interleaved size states the size of a complete NAL unit. Thus I expected to get a first buffer header with the 4 byte size and then that amount of payload. However, when trying to do this with the h.264 encoder on CM3 (and I assume that it works like that on VideoCore in general) I got slightly different results. I observed that I had values strangely close to the remaining nal size at the beginning of each new buffer header. So I enabled MMAL_VIDEO_NALUNITFORMAT_FOURBYTEINTERLEAVELENGTH and did a simple printf with the received buffers:

Code: Select all

printf("buffer l: %d, o: %d, f: %d, left in nal: %d\n", buffer->length, buffer->offset, buffer->flags, ntohl(*reinterpret_cast<unsigned int*>(buffer->data + buffer->offset)));
A fragment of the output:

Code: Select all

buffer l: 65536, o: 0, f: 8, left in nal: 231368
buffer l: 65536, o: 0, f: 8, left in nal: 165832
buffer l: 65536, o: 0, f: 8, left in nal: 100296
buffer l: 34764, o: 0, f: 12, left in nal: 34760
buffer l: 65536, o: 0, f: 8, left in nal: 231590
buffer l: 65536, o: 0, f: 8, left in nal: 166054
buffer l: 65536, o: 0, f: 8, left in nal: 100518
buffer l: 34986, o: 0, f: 12, left in nal: 34982
buffer l: 65536, o: 0, f: 8, left in nal: 232024
buffer l: 65536, o: 0, f: 8, left in nal: 166488
buffer l: 65536, o: 0, f: 8, left in nal: 100952
buffer l: 35420, o: 0, f: 12, left in nal: 35416
Lets look at the last one - so the first buffer has the size set to 232024 which is 3 * 65536 + 35420 - 4 (thus the first 4 bytes are not included, but all of the next interleaved sizes are).

According to this observation it seems to me that each buffer contains the remaining length of the NAL unit payload (including any further interleaved 4 byte sizes). Is my analysis correct? If so then it does not behave exactly as I expected to, but at least if this is how it works I can parse it accordingly.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Behavior of MMAL_VIDEO_NALUNITFORMAT_FOURBYTEINTERLEAVEL

Tue Feb 07, 2017 11:43 am

Can I confirm my understanding of your issue?

MMAL_VIDEO_NALUNITFORMAT_FOURBYTEINTERLEAVELENGTH (or OMX_IndexParamNalStreamFormat) gives you a four byte length, followed by that many bytes of the NAL. The length is not part of the NAL, therefore the 4 bytes taken by it are not included in the length.

Seeing as OMX_IndexParamNalStreamFormat has the comment against it
OMX_IndexParamNalStreamFormat: Control the NAL unit packaging. This is a Khronos extension.
I'm reading that it was an extension defined by the IL Working Group but never integrated into the full spec or official headers (it happened a lot!). I no longer have access to the Khronos working group docs as your company has to be a member to get access, but I'd expect that to define how this is meant to work, and the implementation will follow that.

Now are you saying that every buffer is getting a length inserted in the first 4 bytes? That would be incorrect as it should only replace the start code at the beginning of the NAL.
I'm struggling to replicate things, partly as to get a NAL of 231590 bytes you appear to be running at (231590*8*30) = 55.6 Mbit/s. How, why, what, huh!? The encoder is speced at level 4.0, so max 25Mbit/s. You can push that a little, but 55Mbit/s is almost at the limit of level 4.1/4.2.
What is your FULL raspivid line?

My test was to replace the first 4 bytes of the first buffer of a frame with a start code, and then analyse the resulting file and make sure it plays. Seeing as I can't get the thing to fragment I can't tell if there is a problem
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Behavior of MMAL_VIDEO_NALUNITFORMAT_FOURBYTEINTERLEAVEL

Tue Feb 07, 2017 11:54 am

Ah, kick it up to level 4.2 (but expect it to drop frames sometimes) and 40Mbit/s I can get it to fragments. The resulting file has corruption.
It's going to take some picking through the code to work out where the signalling is going wrong.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Re: Behavior of MMAL_VIDEO_NALUNITFORMAT_FOURBYTEINTERLEAVEL

Tue Feb 07, 2017 1:36 pm

Hi 6by9,

sorry for the late response.

Basically you already answered my main question. I am streaming the video over rtsp (so yeah, I am not using raspivid) and I had corruptions although the rtp packetization seemed fine. So I will decrease the level to 4.0.

As for the interleaved NAL lengths - the only thing why I ever switched to them was because I could not find any details on precise bitstream boundaries (and if you can maybe clarify them then I assume this could be very helpful for others in future).

What I disliked about the start codes was that I was not sure if a new NAL can begin in the middle of a buffer (ok,this can happen for sps and pps in the config buffer but that has an explicit flag so all good there). Because if that can happen then basically I need to run through each new buffer looking for the start code to know the end of the current NAL. If this can not happen and I will always get the first buffer with the start codes at the beggining and then 0 to n buffers with no new start codes where the last will have the end flag set everything becomes simple. In that case I do not need to look for start codes any more. The interleaved NAL length would have solved this probabilistic scenario since then I know the NAL length and can just read that many bytes and then parse the next length etc. etc., but with the current implementation that is not possible as well. I do not have the Khronos specification so you may as well be right. My assumption was based on the ISO MP4 specification where the nal length is inserted once before the sample data and tells the whole length of the sample and nothing is inserted until the sample payload ends.

So if you can state that there are no start codes in the middle of the buffers, I'm happy to move back to those and the NAL start codes are not an issue for me and I guess it is up to you then if and when you want to look at them. You've been very helpful in the past - if I can help with this anyhow when you get to it let me know. I can give the binaries/source of the rtsp server which produces this issue very easily.

Best Regards,
Rudolfs Bundulis

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Behavior of MMAL_VIDEO_NALUNITFORMAT_FOURBYTEINTERLEAVEL

Tue Feb 07, 2017 2:37 pm

Ruuzis wrote:Basically you already answered my main question. I am streaming the video over rtsp (so yeah, I am not using raspivid) and I had corruptions although the rtp packetization seemed fine. So I will decrease the level to 4.0.

As for the interleaved NAL lengths - the only thing why I ever switched to them was because I could not find any details on precise bitstream boundaries (and if you can maybe clarify them then I assume this could be very helpful for others in future).

What I disliked about the start codes was that I was not sure if a new NAL can begin in the middle of a buffer (ok,this can happen for sps and pps in the config buffer but that has an explicit flag so all good there). Because if that can happen then basically I need to run through each new buffer looking for the start code to know the end of the current NAL. If this can not happen and I will always get the first buffer with the start codes at the beggining and then 0 to n buffers with no new start codes where the last will have the end flag set everything becomes simple. In that case I do not need to look for start codes any more. The interleaved NAL length would have solved this probabilistic scenario since then I know the NAL length and can just read that many bytes and then parse the next length etc. etc., but with the current implementation that is not possible as well. I do not have the Khronos specification so you may as well be right. My assumption was based on the ISO MP4 specification where the nal length is inserted once before the sample data and tells the whole length of the sample and nothing is inserted until the sample payload ends.

So if you can state that there are no start codes in the middle of the buffers, I'm happy to move back to those and the NAL start codes are not an issue for me and I guess it is up to you then if and when you want to look at them. You've been very helpful in the past - if I can help with this anyhow when you get to it let me know. I can give the binaries/source of the rtsp server which produces this issue very easily.
To the best of my knowledge, there will be no start codes in the middle of buffers with the exception of the header bytes.
The exception might be if encoding each frame as multiple slices (MMAL_PARMETER_MB_ROWS_PER_SLICE). I seem to recall a thread about it in the past but haven't time to dig it up at the moment.

My reading of the code is that the FOURBYTEINTERLEAVED NAL format is broken if the buffer ever gets fragmented, and that is clearly a bug. It used to only be used in a mode that shared the ~2MB codec output FIFO with the downstream container writer (on the GPU) so never really got exercised when fragmenting.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Ruuzis
Posts: 49
Joined: Wed May 21, 2014 11:35 am

Re: Behavior of MMAL_VIDEO_NALUNITFORMAT_FOURBYTEINTERLEAVEL

Tue Feb 07, 2017 2:49 pm

Hi 6by9,

thank you very much for the reply, very useful information as always:) Ok, I'll fall back to start codes then. Hope anyone else who stumbles upon this finds this useful.

Return to “Graphics, sound and multimedia”