jamesh wrote: ↑Mon Jul 08, 2019 1:41 pm
Firefox are surely the best people to be sorting out HW acceleration, depending on how they do their codec handling, it should be fairly easy to leverage the Chromium changes (if you are a Firefox expert - we don't have one of those, or the time). Problem is that the devs of these browsers seem to ignore the fact that low power systems exist, and they have plenty of CPU and memory bandwidth to blast data around, which means the data paths for video can be quite inefficient.This is where all the Chromium work was aimed.
To give context:
- the base system is for a codec somewhere to deliver the raw YUV data as a buffer. The CPU then takes on the task of converting that to RGBA, resizing to fit the window, and blit it into the framebuffer. Performance for this sucks! It's even worse than it sounds as Chromium's sandboxing would often require the buffer to be copied along the way.
- the changes that we had implemented for Pi2 and 3 was to offload the resize and format conversion onto the hardware (which can do it very efficiently), leaving the CPU to blit (mempcy) it into the framebuffer. For fullscreen 1080p this is still quite a performance hit. It also passed a custom handle down parts of the pipeline, therefore avoiding some of the otherwise needless copies.
- the latest changes switch to using GL for the composition. In theory the codec can deliver the YUV frames into a memory buffer as YUV, the memory buffer can be exported to what is known as a dmabuf (sometimes DRM_PRIME), and that handle gets passed down the pipeline. The 3D system can then import the handle and render it nearly directly into the desktop scene.
At present the codec side is still done via MMAL, but that's mainly because Chromium's support for the V4L2 codec API is hidden/broken for X11 (It requires an abstraction layer called Ozone, and that blows up badly under X. They seem to mainly use it with ChromeOS).
V4L2 would be the obvious API to do all this cleanly and in a platform independent manner. Both ffmpeg and GStreamer implement it, although FFmpeg currently doesn't support exporting the dmabuf handle (the LibreELEC folk are looking at that at the moment). Firefox could take advantage of one of those helper libraries, or it can wrap the V4L2 stuff for itself. It's a case of whatever their developers think is worth the effort.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.