longo92
Posts: 45
Joined: Mon Sep 03, 2018 3:45 pm
Contact: Website Skype

MMAL Reziser input sets to opaque

Thu Oct 25, 2018 12:52 pm

Is possible to set the resizer input encoding to opaque?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7300
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: MMAL Reziser input sets to opaque

Thu Oct 25, 2018 1:10 pm

longo92 wrote:
Thu Oct 25, 2018 12:52 pm
Is possible to set the resizer input encoding to opaque?
If you're talking about "vc.ril.resize", then it should be.
"vc.ril.isp" certainly does, and is using hardware instead of (vector) software to do the resize and format conversion.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

longo92
Posts: 45
Joined: Mon Sep 03, 2018 3:45 pm
Contact: Website Skype

Re: MMAL Reziser input sets to opaque

Thu Oct 25, 2018 1:27 pm

I mean "vc.ril.resize". Also can i set the output to the opaque and tunnel to the mmal vidoe encoder? I'm using the firmware of 25 May 2018, it's safe to use or shall i update to eliminate possible bugs?
I have a simple application where a camera is tunneled to a splitter, the splitter tunneled to a resizer and the resizer tunneled to an encoder:
camera->splitter->resizer->encoder.
I want use opaque encoding and zero-copy for maximum performance.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7300
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: MMAL Reziser input sets to opaque

Thu Oct 25, 2018 1:46 pm

longo92 wrote:
Thu Oct 25, 2018 1:27 pm
I mean "vc.ril.resize". Also can i set the output to the opaque and tunnel to the mmal vidoe encoder? I'm using the firmware of 25 May 2018, it's safe to use or shall i update to eliminate possible bugs?
I have a simple application where a camera is tunneled to a splitter, the splitter tunneled to a resizer and the resizer tunneled to an encoder:
camera->splitter->resizer->encoder.
I want use opaque encoding and zero-copy for maximum performance.
Is another output of splitter used? Based on that pipe I fail to see why you need either splitter or resize - you may as well ask for your desired resolution out of the camera and feed it direct to video_encode.

Whilst vc.ril.resize does support opaque on the output there is no point in using it if connecting to video_encode. It doesn't pass all the information required by the encoder anyway, therefore the encoder has to do extra work before encoding.
If you use a more recent firmware (after mid-August) then video_encode uses the ISP hardware for the conversions it requires and is therefore more efficient.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

longo92
Posts: 45
Joined: Mon Sep 03, 2018 3:45 pm
Contact: Website Skype

Re: MMAL Reziser input sets to opaque

Thu Oct 25, 2018 1:52 pm

I use all the four output of the splitter. Each splitter output is linked to a resizer with different downscaled camera's resolution.
So there is no boost to have an opaque handler to input to the encoder.
Thanks again.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7300
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: MMAL Reziser input sets to opaque

Thu Oct 25, 2018 3:49 pm

longo92 wrote:
Thu Oct 25, 2018 1:52 pm
So there is no boost to have an opaque handler to input to the encoder.
Unless it is coming direct from the camera, then no for video_encode.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

longo92
Posts: 45
Joined: Mon Sep 03, 2018 3:45 pm
Contact: Website Skype

Re: MMAL Reziser input sets to opaque

Fri Oct 26, 2018 6:19 am

Another question: so if have to perform resize (not cropping) is always better to use the vc.ril.isp than the vc.ril.resize? Because isp uses hardware while resize is implemented in software (then it pass through the CPU), right?
When yuoi said that now the encoder uses isp for conversion, it means that can it can perform resizing inside (also format conversion)? So there is no need to tunnel isp->encoder?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7300
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: MMAL Reziser input sets to opaque

Fri Oct 26, 2018 10:49 am

longo92 wrote:
Fri Oct 26, 2018 6:19 am
Another question: so if have to perform resize (not cropping) is always better to use the vc.ril.isp than the vc.ril.resize? Because isp uses hardware while resize is implemented in software (then it pass through the CPU), right?
Everything has limitations.
The ISP can process one pixel per clock cycle, and runs at 250MHz, so theoretically can hit 250MPix/s.
A few constraints:
- It is limited at both ends of the pipe, so either one pixel in or one pixel out. So if downscaling then the input resolution will be the limit, whilst the output is the limit if upscaling. Other pipe stages stall for a cycle if necessary.
- The ISP processes in tiles, and each tile has a small amount of context required. That context has to be accounted for.
- It is almost impossible to keep the hardware fully occupied with things to process, so scheduling is an issue.
- Everything has to go to and from memory, and memory bandwidth is not infinite.
- The block is also used by the camera component, and now video_encode.

In contrast, vc.ril.resize uses the VPU (Vector Processing Unit), which is also used for almost all other processing within the VideoCore GPU.
It is a 16 way SIMD unit, so can process 16 bits of data per clock cycle. However resize is going to be about 100 instructions per pixel element (eg Y, U, V, or red, green, or blue), which would translate to a maximum of 40MPix/s for a single channel image if you had sole use of the VPU. Reality is that you don't have sole use of the VPU, and you don't deal in mono images, therefore the realistic maximum would be lower. Memory bandwidth is a restriction too.
longo92 wrote:When yuoi said that now the encoder uses isp for conversion, it means that can it can perform resizing inside (also format conversion)? So there is no need to tunnel isp->encoder?
I haven't added any option for resizing at the front end of video_encode, although in theory it would be possible.
I added it as the codec block requires two versions of the image in particular formats, and the ISP was designed to be able to write out both images at the same time. Doing it on the VPU resulted in too much contention when trying to exceed 1080p30, hence pulling in the ISP hardware to help out. There are a couple of conditions where it can't be used and it has to drop back to using the VPU, but generally it's a win, particularly as it extends the range of pixel formats that can be handled by the encoder.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

longo92
Posts: 45
Joined: Mon Sep 03, 2018 3:45 pm
Contact: Website Skype

Re: MMAL Reziser input sets to opaque

Fri Oct 26, 2018 11:49 am

Thanks for your really precious and precise answer. I try and the isp, on my case, for resizing is much faster.

Return to “Graphics programming”