bbb
Posts: 55
Joined: Sat Jun 02, 2012 9:52 am

Suport for transform pixel format + Audio Encoders ?

Tue Jul 09, 2013 7:16 pm

Hi,

Background to these questions are: I am working using the pi + USB capture device / webcam as a IP camera. I've got h264 HW video encoding working more or less (issues are with generic Linux drivers and not Pi specific). Next steps are getting CPU usage down, and getting audio working as well.

So my questions I need help with please are :)

Is it possible via OpenMAX or MMAL to support the following:

1) Convert pixel formats uyvy422 to yuv420p or yuyv422 to yuv420p ? (uyvy422 and yuyv422 are common pixel formats for webcams and capture devices).
2) AAC or MP3 audio encoder ? Posted this thread in C++ forum with no answer :( http://www.raspberrypi.org/phpBB3/viewt ... 01#p378601
3) PCM sample rate conversion, e.g. 48.0 to 22.050 KHz?

Thanks.

bbb
Posts: 55
Joined: Sat Jun 02, 2012 9:52 am

Re: Suport for transform pixel format + Audio Encoders ?

Wed Jul 17, 2013 11:34 pm

bump,

I not looking for code samples etc ... just a yes / no that is supported ...

Don't want to burn time coding up OpenMAX C code to the specification (example http://home.nouwen.name/RaspberryPi/doc ... index.html) and find its not supported because of unlicensed codecs ...

rasmus25
Posts: 13
Joined: Mon Feb 04, 2013 3:53 pm

Re: Suport for transform pixel format + Audio Encoders ?

Thu Jul 18, 2013 2:51 pm

Like I mentioned in another thread (http://www.raspberrypi.org/phpBB3/viewt ... 70&t=49509), doing a colorspace conversion on the GPU is not very difficult using OpenGL ES 2.0 shaders. The hard part is doing it in real time and HD, as video encoder currently requires its input buffer to be in CPU-s memory space, so you cannot just do stuff on the GPU and then directly feed it into the encoder. If you are OK with 1-5 frames per second or low video resolution, then lowering CPU usage is doable. My benchmark showed around 685 megabits per second on the bus between CPU and GPU (not including overhead), as if there was one bit transfered on every CPU clock cycle.

I do not have experience with audo on Raspberry Pi

bbb
Posts: 55
Joined: Sat Jun 02, 2012 9:52 am

Re: Suport for transform pixel format + Audio Encoders ?

Mon Jul 22, 2013 3:50 pm

Thanks. I am not dealing with hd, more like qvga. Yea shame you cant pipe ogl output into openmax il. I likely just leave the pixel format conversion on the ARM. Other wise I end up with extra copies back and forth between ARM and video core: USB -> V4L -> ARM -> GL (VC) -> ARM -> video_encode (VC) -> ARM. Did you look at using the 'image_fx' OMX component for pixel convert or scaling? Interface spec does show it might work, I haven't tried it yet

rasmus25
Posts: 13
Joined: Mon Feb 04, 2013 3:53 pm

Re: Suport for transform pixel format + Audio Encoders ?

Tue Jul 23, 2013 10:12 am

Regarding pixel format conversion, you have the option of trading between CPU usage and memory bandwidth usage. Copying frames back and forth in the memory does not take much CPU, as DMA is used. So if you need the CPU power for something else, using OpenGL shaders for pixel format conversion might still be a good idea, if you don't need the memory bandwidth elsewhere. I am not sure if it will lower power usage, though.

In your particular case, we can calculate the memory bandwidth usage as follows:
1) yuv422 is 2 bytes or 16 bits per pixel, so QVGA frame is 320x240x16 = 1228800 bits
2) yuv420 is 1.5 bytes or 12 bits per pixel, so the frame is 320x240x12 = 921600 bits
3) USB -> V4L -> ARM -> GL (VC) -> ARM -> video_encode (VC) -> ARM is 3 copies of yuv422 and 2 copies of yuv420 and one copy of h.264 frame. The latter is so small that it can be left out of the calculations.
4) Memory bandwidth used for 1 frame is 3x1228800+2x921600 = 4300800 = 5.5 Mbits/s
5) Let's say you are doing 30 FPS. This gives us 30x5.5 = 166 Mbits/s
6) 166 Mbits/s is only 166Mbps/700Mbps x 100% = 23.7% of the memory bandwidth available, so you have plenty left for other stuff.

I have not looked into image_fx yet, thanks for the tip! What is this interface spec that you are talking about? Do you mean this: http://home.nouwen.name/RaspberryPi/doc ... ge_fx.html? That web page does not look very promising, as it says input and output are YUV only, using proprietary methods. I can imagine the following hack to make image_fx work for me, if it supports the needed color conversion:
a) Create an image file in RAM by writing a file to /run/shm/
b) Read input from camera into the file.
c) Use OMX.broadcom.image_read to read from the file
d) Link image_read to image_fx to video_encode

This is just an idea and I am still highly skeptical of image_fx. If only there was more documentation on the components...

bbb
Posts: 55
Joined: Sat Jun 02, 2012 9:52 am

Re: Suport for transform pixel format + Audio Encoders ?

Wed Jul 24, 2013 8:33 pm

rasmus25 wrote:Regarding pixel format conversion, you have the option of trading between CPU usage and memory bandwidth usage. Copying frames back and forth in the memory does not take much CPU, as DMA is used.
Your quite right. Only downside I can see is increasing the latency of the pipeline - but it does free up the CPU for other stuff. In the situation where the CPU load is high (>90%) then the extra latency might be negated.
rasmus25 wrote: I have not looked into image_fx yet, thanks for the tip! What is this interface spec that you are talking about? Do you mean this: http://home.nouwen.name/RaspberryPi/doc ... ge_fx.html

This is just an idea and I am still highly skeptical of image_fx. If only there was more documentation on the components...
I am skeptical as well and yes, pleasant way to put this is 'the documentation is slightly underwhelming' :)

Just trying it out 'image_fx' to convert from yuv422 to yuv420, disabling ports and enabling input / output buffers and seeing what happens with a raw capture from a web-cam. Take a day or two as I don't get much time to work on it :)

bbb
Posts: 55
Joined: Sat Jun 02, 2012 9:52 am

Re: Suport for transform pixel format + Audio Encoders ?

Thu Jul 25, 2013 9:59 pm

Following code was looking good, runs with any errors ...

Code: Select all

/*
Copyright (c) 2013 "Ben Farrell" "bbb"
All rights reserved.

Redistribution and use in source and binary forms are permitted
provided that the above copyright notice and this paragraph are
duplicated in all such forms and that any documentation,
advertising materials, and other materials related to such
distribution and use acknowledge that the software was developed
by Ben Farrell.
THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOS
*/

#include <stdio.h>
#include <bcm_host.h>
#include <ilclient.h>

#define WIDTH  320
#define HEIGHT 240

void omx_exit(const char *msg, OMX_ERRORTYPE r)
{
	fprintf(stderr, "%s error %d\n", msg, r);
	fflush(stderr);
	exit(2);
}

COMPONENT_T* omx_create_image_fx(ILCLIENT_T *client)
{
	COMPONENT_T *component;
	OMX_PARAM_PORTDEFINITIONTYPE def;
	OMX_ERRORTYPE r = 0;
	int in_port = 190;
	int out_port = 191;
	char * component_name = "image_fx";
	
	r = ilclient_create_component(client, &component, component_name,
				  ILCLIENT_DISABLE_ALL_PORTS |
				  ILCLIENT_ENABLE_INPUT_BUFFERS|
				  ILCLIENT_ENABLE_OUTPUT_BUFFERS);
				  
	if (r != 0) {
		fprintf(stderr,"ilclient_create_component(%s) failed with reason %d\n", component_name ,r);
		exit(1);
	}		  

	/*********************************************************************************************************/
	/* in_port */
	/*********************************************************************************************************/
	memset(&def, 0, sizeof(OMX_PARAM_PORTDEFINITIONTYPE));
	def.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
	def.nVersion.nVersion = OMX_VERSION;
	def.nPortIndex = in_port;
	
	if((r = OMX_GetParameter(ILC_GET_HANDLE(component), OMX_IndexParamPortDefinition, &def)) != OMX_ErrorNone) {
		omx_exit("omx_create_image_fx(): OMX_GetParameter() in_port", r);
	}
	
	def.nBufferSize = 0;			
	def.format.image.nFrameWidth = WIDTH;
	def.format.image.nFrameHeight = HEIGHT;
	def.format.image.nStride = WIDTH;
	def.format.image.nSliceHeight = HEIGHT;
	def.format.image.eColorFormat =  OMX_COLOR_FormatYUV422PackedPlanar;
	
	if((r = OMX_SetParameter(ILC_GET_HANDLE(component), OMX_IndexParamPortDefinition, &def)) != OMX_ErrorNone) {
		omx_exit("omx_create_image_fx(): OMX_SetParameter() in_port", r);
	}
	
	/*********************************************************************************************************/
	/* out_port */
	/*********************************************************************************************************/
	memset(&def, 0, sizeof(OMX_PARAM_PORTDEFINITIONTYPE));
	def.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
	def.nVersion.nVersion = OMX_VERSION;
	def.nPortIndex = out_port;
	
	if((r = OMX_GetParameter(ILC_GET_HANDLE(component), OMX_IndexParamPortDefinition, &def)) != OMX_ErrorNone) {
		omx_exit("omx_create_image_fx(): OMX_GetParameter() out_port", r);
	}
	
	def.nBufferSize = 0;	
	def.format.image.nFrameWidth = WIDTH;
	def.format.image.nFrameHeight = HEIGHT;
	def.format.image.nStride = WIDTH;
	def.format.image.nSliceHeight = HEIGHT;	
	def.format.image.eColorFormat = OMX_COLOR_FormatYUV420PackedPlanar;

	if((r = OMX_SetParameter(ILC_GET_HANDLE(component), OMX_IndexParamPortDefinition, &def)) != OMX_ErrorNone) {
		omx_exit("omx_create_image_fx(): OMX_SetParameter() out_port", r);
	}
	
	/*********************************************************************************************************/
	/* enable ports and set state */
	/*********************************************************************************************************/
	
	if (ilclient_change_component_state(component, OMX_StateIdle) == -1) {
		fprintf(stderr,"ilclient_change_component_state(OMX_StateIdle) failed");
	}
	
	if (ilclient_enable_port_buffers(component, in_port, NULL, NULL, NULL) != 0) {
		fprintf(stderr, "enabling port buffers for %d failed!\n", in_port);
		exit(1);
	}

	if (ilclient_enable_port_buffers(component, out_port, NULL, NULL, NULL) != 0) {
		fprintf(stderr, "enabling port buffers for %d failed!\n", out_port);
		exit(1);
	}

	if (ilclient_change_component_state(component, OMX_StateExecuting) == -1) {
		fprintf(stderr,"ilclient_change_component_state(OMX_StateIdle) failed");
	}

	return component;
}

void omx_write_to(COMPONENT_T *component, int in_port)
{
	OMX_BUFFERHEADERTYPE *buf;
	OMX_ERRORTYPE r = 0;
	FILE *in;
	
	buf = ilclient_get_input_buffer(component, in_port, 1);
	
	/* read raw image */
	in = fopen("image.raw", "rb");	
	buf->nFilledLen = fread( buf->pBuffer, 1, buf->nAllocLen, in);	
	fprintf(stderr, "read in file %d\n", buf->nFilledLen);
	fclose(in);

	if((r = OMX_EmptyThisBuffer(ILC_GET_HANDLE(component), buf)) != OMX_ErrorNone) {
		omx_exit("OMX_EmptyThisBuffer()", r);
	}		
}

void omx_read_from(COMPONENT_T *component, int out_port)
{
	OMX_BUFFERHEADERTYPE *buf;
	OMX_ERRORTYPE r = 0;	
	FILE *out;
		
	out = fopen("omxout.raw", "wb");
	
	buf = ilclient_get_output_buffer(component, out_port, 1);
	
    if((r = OMX_FillThisBuffer(ILC_GET_HANDLE(component), buf)) != OMX_ErrorNone) {
	    omx_exit("OMX_FillThisBuffer()", r);	    
	}
	
	if(buf->nFilledLen != 0){
	    fwrite(buf->pBuffer, 1, buf->nFilledLen, out);
		fflush(out);	    
		fclose(out);
	}
	else {
		fprintf(stderr, "After OMX_FillThisBuffer() buf->nFilledLen = 0\n");
		exit(1);
	}
}

int main(int argc, char *argv[])
{
	COMPONENT_T *list[2];
	ILCLIENT_T *client;
	int omx_chain_in_port = 190;
	int omx_chain_out_port = 191;
	
	bcm_host_init();
	
	memset(list, 0, sizeof(list));

	if ((client = ilclient_init()) == NULL) {
		return 1;
	}

	if (OMX_Init() != OMX_ErrorNone) {
		fprintf(stderr,"omx_init failed\n");
		ilclient_destroy(client);
		return 1;
	}
	
	list[0] = omx_create_image_fx(client);
	
	omx_write_to(list[0], omx_chain_in_port);
	omx_read_from(list[0], omx_chain_out_port);
	
	fprintf(stderr,".. shutdown .. \n");
	ilclient_disable_port_buffers(list[0], omx_chain_in_port, NULL, NULL, NULL);
	ilclient_disable_port_buffers(list[0], omx_chain_out_port, NULL, NULL, NULL);
	
	ilclient_state_transition(list, OMX_StateIdle);
	ilclient_state_transition(list, OMX_StateLoaded);
	ilclient_cleanup_components(list);
	
	return 0;
}

However:
  • pi@raspberrypi ~/mjpeg $ ls -l *.raw
    -rw-r--r-- 1 pi pi 153600 Jul 25 18:54 image.raw
    -rw-r--r-- 1 pi pi 153600 Jul 25 21:47 omxout.raw

    pi@raspberrypi ~/mjpeg $ diff image.raw omxout.raw
    pi@raspberrypi ~/mjpeg
So image_fx despite having different eColorFormat on the in port (190) and out port (191) definitions (OMX_COLOR_FormatYUV422PackedPlanar vs. OMX_COLOR_FormatYUV420PackedPlanar) is just passing the buffer through without any conversion :cry:

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23883
Joined: Sat Jul 30, 2011 7:41 pm

Re: Suport for transform pixel format + Audio Encoders ?

Fri Jul 26, 2013 7:45 am

image_fx doesn't do conversion, it just adds effects on top of the frame.

I think the resize component may do some conversion - you could try that.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

Return to “OpenMAX”