User avatar
mihalop
Posts: 23
Joined: Thu Sep 03, 2015 7:04 am
Contact: Website

OpenMAX blending of video and alpha mask

Wed Apr 20, 2016 12:50 pm

I am trying to modify omxplayer to allow for a alpha/transparency mask for videos. In the optimal scenario, you would specify multiple " --transparent 'x1 y1 x2 y2' " options to allow for specifying transparent regions within the playback viewport defined by "--win".

I am digging into omxplayer code and OpenMAX IL spec. As far as I have understood up to now, the only way to achieve this is by preparing the mask in a buffer and using OMX_IndexConfigCommonColorBlend or OMX_IndexConfigCommonPlaneBlend to blend the the two components outputs.

Since I have zero experience with OpenMAX, is the above correct? Would this affect video playback performance/load? Has anyone done anything similar?
Yodeck - Digital Signage on the Raspberry Pi
Free Raspberry Pi 3 Player for all screens on Annual Plan

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5214
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: OpenMAX blending of video and alpha mask

Wed Apr 20, 2016 1:15 pm

OMX_IndexConfigCommonPlaneBlend and OMX_IndexConfigCommonColorBlend are not supported.
You can check the supported parameters for each component here: http://home.nouwen.name/RaspberryPi/doc ... omponents/

Do you really need transparency in the video?
The more usual scenario is to create a RBGA overlay on top of the video which has transparency where you want the video to show (e.g. as Kodi does for GUI overlay).

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6425
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: OpenMAX blending of video and alpha mask

Wed Apr 20, 2016 1:27 pm

Adding RGBA support to video_render is on my list of things to do, but it's a long list for the amount of spare time I have.
Not sure if that really helps you that much in this case anyway.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
dividuum
Posts: 112
Joined: Sun Jun 16, 2013 1:18 pm
Location: Germany
Contact: Website

Re: OpenMAX blending of video and alpha mask

Sat Apr 23, 2016 1:54 pm

One way to do this (for up to 720p videos) is to render the video into an opengl texture, then combine that texture with transparency mask from other texture using a shader. I've successfully done that with my software info-beamer. But 1080p isn't possible that way as the memory bandwidth (?) isn't enough for that.

Code: Select all

gl.setup(1920, 1080)

local video = resource.load_video "video.mp4"
local image = resource.load_image "image.png"

local shader = resource.create_shader[[
    uniform sampler2D Texture;
    uniform sampler2D Transparency;
    varying vec2 TexCoord;
    void main() {
        gl_FragColor = texture2D(Texture, TexCoord) * texture2D(Transparency, TexCoord).a;
    }
]]

function node.render()
    shader:use{
        Transparency = image;
    }
    video:draw(0, 0, WIDTH, HEIGHT)
end
Example output: http://i.imgur.com/INoSclW.jpg
info-beamer hosted - A user and programmer friendly digital signage platform for the Pi: https://info-beamer.com/hosted

User avatar
mihalop
Posts: 23
Joined: Thu Sep 03, 2015 7:04 am
Contact: Website

Re: OpenMAX blending of video and alpha mask

Mon Apr 25, 2016 8:25 pm

@dom:

Thanks for the link, thought that the Pi had the full OpenMAX spec supported. An overlay won't do, I am trying to show things rendered behind the video, on regions of the Linux framebuffer.

@6by9:

Indeed it won't help much, usually videos do not have an alpha channel and the transparent regions change in each playback.

@dividuum:

The end result would be exactly what I need. I had this solution in mind, using egl_render, but did not look into it since I thought it would cause hiccups on playback. I also use OpenGL for other animations and it would probably not take the load. And, also, 1080p videos are required. Have you tried that on a Pi 3? Perhaps it performs better.

Overall, I can't seem to find a feasible way to achieve this, so I will probably revert to rendering the X content in a buffer in arm space and copying it to a mmapped GPU texture. Using glTexSubImage2D is just too slow, can't get over 15-20 fps :-(
Yodeck - Digital Signage on the Raspberry Pi
Free Raspberry Pi 3 Player for all screens on Annual Plan

User avatar
dividuum
Posts: 112
Joined: Sun Jun 16, 2013 1:18 pm
Location: Germany
Contact: Website

Re: OpenMAX blending of video and alpha mask

Mon Apr 25, 2016 10:28 pm

mihalop wrote:The end result would be exactly what I need. I had this solution in mind, using egl_render, but did not look into it since I thought it would cause hiccups on playback. I also use OpenGL for other animations and it would probably not take the load. And, also, 1080p videos are required. Have you tried that on a Pi 3? Perhaps it performs better.
A bit, but combining two 1920x1080 textures (the mask image and the video rendered into a texture) using a shader is still too much when running at 60fps. It's fast enough for 30fps but the overall output feels jerky in that case.
mihalop wrote:Overall, I can't seem to find a feasible way to achieve this, so I will probably revert to rendering the X content in a buffer in arm space and copying it to a mmapped GPU texture. Using glTexSubImage2D is just too slow, can't get over 15-20 fps :-(
What's the best way of doing that? I remember investigating that a while ago but couldn't find a way to do that. Any hints welcome.
info-beamer hosted - A user and programmer friendly digital signage platform for the Pi: https://info-beamer.com/hosted

User avatar
mihalop
Posts: 23
Joined: Thu Sep 03, 2015 7:04 am
Contact: Website

Re: OpenMAX blending of video and alpha mask

Fri May 27, 2016 5:19 pm

I am using the mailbox interface to get lock on a GPU memory allocations. I also get a ARM-space pointer to it and have mmapped /dev/mem. I get the CPU to directly copy raster data into GPU space, and it works fine with DispmanX resources. But I need to use a GL texture. Just today I found out that textures are stored in Tile Format or Linear-tile Format.

@dom:

Saw in VideoCore IV docs that there is a Texture Datatype of RGBA32R, "Raster format 8-bit per channel red, green, blue, alpha", which is exactly what I need in order to avoid tiling/swizzling huge overhead. But as far as I can see, this is not exposed by the firmware. Why? Could this be added?
Yodeck - Digital Signage on the Raspberry Pi
Free Raspberry Pi 3 Player for all screens on Annual Plan

Return to “OpenMAX”