madCoder24
Posts: 18
Joined: Wed Feb 03, 2016 11:38 pm

Openmax Decode and Display

Sat Feb 13, 2016 11:06 pm

I didn't realize there was a forum for OpenMAX. Otherwise, I would have posted this question in this section. I didn't see a way to move my old post to this section. So it will be duplicated and I apologize for that as I understand it is bad etiquette.

I am currently having issues where my program just hangs. This program receives h264 data remotely using UDP. It receives data until it gets a complete frame. Next it checks if I have enough data to fill the entire buffer length or the allocated length. Initially, I thought that was my issue where I wasn't getting enough data in the buffer. However, that doesn't seem to be the issue. I have started off using the video_decode as a starting point. I am able to stream data to a file and then play the file using video_decode and that works just fine. So I know the sending and receiving is working, I just can't seem to get OpenMax to work correctly.

Any advice is appreciated.

Code: Select all

int video_decode_test()
{
   Socket s;
   H264_Decoder decoder;
   FILE *in;
   int status = 0;
   unsigned int data_len = 0;
   
   s.localFd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
   
   if (s.localFd < 0)
   {
      return -1;
   }
   
   s.fromLen = sizeof(s.si_other);

   memset((char *)&s.si_me, 0, sizeof(s.si_me));
   s.si_me.sin_family = AF_INET;
   s.si_me.sin_addr.s_addr = htonl(INADDR_ANY);
   s.si_me.sin_port = htons(5150);

   if (bind(s.localFd, (struct sockaddr *)&s.si_me, sizeof(s.si_me)) < 0)
   {
      return -1;
   }

   memset(decoder.list, 0, sizeof(decoder.list));
   memset(decoder.tunnel, 0, sizeof(decoder.tunnel));

   if ((decoder.client = ilclient_init()) == NULL)
   {
      fclose(in);
      return -3;
   }

   if (OMX_Init() != OMX_ErrorNone)
   {
      ilclient_destroy(decoder.client);
      fclose(in);
      return -4;
   }

      // create video_decode
   if (ilclient_create_component(decoder.client, &decoder.decode, "video_decode", (ILCLIENT_CREATE_FLAGS_T)(ILCLIENT_DISABLE_ALL_PORTS | ILCLIENT_ENABLE_INPUT_BUFFERS)) != 0)
      status = -14;
   decoder.list[0] = decoder.decode;

      // create video_render
   if (status == 0 && ilclient_create_component(decoder.client, &decoder.render, "video_render", ILCLIENT_DISABLE_ALL_PORTS) != 0)
      status = -14;
   decoder.list[1] = decoder.render;

      // create clock
   if (status == 0 && ilclient_create_component(decoder.client, &decoder.clock, "clock", ILCLIENT_DISABLE_ALL_PORTS) != 0)
      status = -14;
   decoder.list[2] = decoder.clock;

   memset(&decoder.cstate, 0, sizeof(decoder.cstate));
   decoder.cstate.nSize = sizeof(decoder.cstate);
   decoder.cstate.nVersion.nVersion = OMX_VERSION;
   decoder.cstate.eState = OMX_TIME_ClockStateWaitingForStartTime;
   decoder.cstate.nWaitMask = 1;
   if (decoder.clock != NULL && OMX_SetParameter(ILC_GET_HANDLE(decoder.clock), OMX_IndexConfigTimeClockState, &decoder.cstate) != OMX_ErrorNone)
      status = -13;

         // create video_scheduler
   if (status == 0 && ilclient_create_component(decoder.client, &decoder.scheduler, "video_scheduler", ILCLIENT_DISABLE_ALL_PORTS) != 0)
      status = -14;
   decoder.list[3] = decoder.scheduler;

   set_tunnel(decoder.tunnel, decoder.decode, 131, decoder.scheduler, 10);
   set_tunnel(decoder.tunnel + 1, decoder.scheduler, 11, decoder.render, 90);
   set_tunnel(decoder.tunnel + 2, decoder.clock, 80, decoder.scheduler, 12);

      // setup clock tunnel first
   if (status == 0 && ilclient_setup_tunnel(decoder.tunnel + 2, 0, 0) != 0)
      status = -15;
   else
      ilclient_change_component_state(decoder.clock, OMX_StateExecuting);

   if (status == 0)
      ilclient_change_component_state(decoder.decode, OMX_StateIdle);

   memset(&decoder.format, 0, sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE));
   decoder.format.nSize = sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE);
   decoder.format.nVersion.nVersion = OMX_VERSION;
   decoder.format.nPortIndex = 130;
   decoder.format.eCompressionFormat = OMX_VIDEO_CodingAVC;

   if (status == 0 &&
      OMX_SetParameter(ILC_GET_HANDLE(decoder.decode), OMX_IndexParamVideoPortFormat, &decoder.format) == OMX_ErrorNone &&
      ilclient_enable_port_buffers(decoder.decode, 130, NULL, NULL, NULL) == 0)
   {
      int currentFrame = 1;
      std::vector<uint8_t> buffer;
      
      while (true)
      {
         madProto proto;
         int result = recvfrom(s.localFd, (char *)&proto, sizeof(madProto), 0, &s.si_other, (socklen_t *)&s.fromLen);
      
         if (result < 0)
         {
            std::cout << "receive failed: " << errno << std::endl;
         }
         else
         {
            uint16_t nalLength = ntohs(proto.nal_length);
            std::copy(proto.payload, proto.payload + nalLength, std::back_inserter(buffer));
            std::cout << "received bytes: " << result << std::endl;
         
            if (ntohs(proto.frame_end) == 1)
            {
               if (currentFrame == 1)
                  decoder.first_packet = 1;
               else
               {
                  decoder.first_packet = 0;
                  currentFrame++;
               }
            
               std::cout << "we have a frame end" << std::endl;
               //uint8_t *data = buffer.data();
               //int size = buffer.size();
            
               videoDecode(&decoder, buffer);
            
               //fwrite(data, size, 1, f);
               //buffer.clear();
            }
         }
   }
   }

   fclose(in);

   omxCleanup(&decoder);
   return status;
}

int videoDecode(H264_Decoder *d, std::vector<uint8_t> &buffer)
{
   OMX_BUFFERHEADERTYPE *buf;

   printf("req buff!\n");
        /* Request a buffer to the decode component */
   buf = ilclient_get_input_buffer(d->decode, 130, 1);
   if (buf == NULL)
      return -1;
   printf("got buff!\n");
   
   if (buffer.size() < buf->nAllocLen)
      return 0;
   
   std::cout << "buffer size: " << buffer.size() << std::endl;
   std::cout << "buffer allocated length: " << buf->nAllocLen << std::endl;

      /* If len == 0 then we reached the end of stream (EOS) so let the decoder know */
   if (buffer.size() == 0) {
      buf->nFilledLen = 0;
      buf->nFlags = OMX_BUFFERFLAG_TIME_UNKNOWN | OMX_BUFFERFLAG_EOS;

      if (OMX_EmptyThisBuffer(ILC_GET_HANDLE(d->decode), buf) != OMX_ErrorNone)
         return -3;

               /* Wait for EOS from render */
      ilclient_wait_for_event(d->render, OMX_EventBufferFlag, 90, 0, OMX_BUFFERFLAG_EOS, 0, ILCLIENT_BUFFER_FLAG_EOS, 10000);

            /* Flush the renderer */
      ilclient_flush_tunnels(d->tunnel, 0);

            /* Disable the input port to the decoder component */
      ilclient_disable_port_buffers(d->decode, 130, NULL, NULL, NULL);

      return 1;
   }

   /* Feed in data to the decoder */
   memcpy(buf->pBuffer, buffer.data(), buf->nAllocLen);
   buf->nFilledLen = buf->nAllocLen;
   buf->nOffset = 0;
   
   buffer.clear();

   if (d->port_settings_changed == 0 &&
      (buffer.size() > 0 && ilclient_remove_event(d->decode, OMX_EventPortSettingsChanged, 131, 0, 0, 1) == 0)) {
      d->port_settings_changed = 1;
      printf("Port settings changed!\n");

            /* Setup tunnel 1 */
      if (ilclient_setup_tunnel(d->tunnel, 0, 0) != 0)
         return -4;

      ilclient_change_component_state(d->scheduler, OMX_StateExecuting);

      if (ilclient_setup_tunnel(d->tunnel + 1, 0, 1000) != 0)
         return -5;

      ilclient_change_component_state(d->render, OMX_StateExecuting);
   }

      /* Set the flags depending on first packet or not */
   if (d->first_packet) {
      buf->nFlags = OMX_BUFFERFLAG_STARTTIME;
      d->first_packet = 0;
   }
   else {
      buf->nFlags = OMX_BUFFERFLAG_TIME_UNKNOWN;
   }

   printf("empty!\n");
   if (OMX_EmptyThisBuffer(ILC_GET_HANDLE(d->decode), buf) != OMX_ErrorNone)
      return -6;
   printf("done empty!\n");
   return 0;
}

madCoder24
Posts: 18
Joined: Wed Feb 03, 2016 11:38 pm

Re: Openmax Decode and Display

Sun Mar 20, 2016 9:22 pm

I was finally able to get some working output. I reverted my code back to 'hello_video' completely and made modifications to it. Except now I am having issues where ilclient_get_input_buffer is blocking waiting for a buffer. Is there something I am missing that I just cannot see?

Any advice is appreciated.

Code: Select all

int video_decode_test()
{
	Socket s;
	
	s.localFd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
	
	if (s.localFd < 0)
	{
		return -1;
	}
	
	s.fromLen = sizeof(s.si_other);

	memset((char *)&s.si_me, 0, sizeof(s.si_me));
	s.si_me.sin_family = AF_INET;
	s.si_me.sin_addr.s_addr = htonl(INADDR_ANY);
	s.si_me.sin_port = htons(5150);

	if (bind(s.localFd, (struct sockaddr *)&s.si_me, sizeof(s.si_me)) < 0)
	{
		return -1;
	}

	OMX_VIDEO_PARAM_PORTFORMATTYPE format;
	OMX_TIME_CONFIG_CLOCKSTATETYPE cstate;
	COMPONENT_T *video_decode = NULL, *video_scheduler = NULL, *video_render = NULL, *clock = NULL;
	COMPONENT_T *list[5];
	TUNNEL_T tunnel[4];
	ILCLIENT_T *client;
	FILE *in;
	int status = 0;
	unsigned int data_len = 0;

	memset(list, 0, sizeof(list));
	memset(tunnel, 0, sizeof(tunnel));

	if((client = ilclient_init()) == NULL)
	{
		fclose(in);
		return -3;
	}

	if(OMX_Init() != OMX_ErrorNone)
	{
		ilclient_destroy(client);
		fclose(in);
		return -4;
	}

	   // create video_decode
	if(ilclient_create_component(client, &video_decode, "video_decode", (ILCLIENT_CREATE_FLAGS_T)(ILCLIENT_DISABLE_ALL_PORTS | ILCLIENT_ENABLE_INPUT_BUFFERS)) != 0)
		status = -14;
	list[0] = video_decode;

	   // create video_render
	if(status == 0 && ilclient_create_component(client, &video_render, "video_render", ILCLIENT_DISABLE_ALL_PORTS) != 0)
		status = -14;
	list[1] = video_render;

	   // create clock
	if(status == 0 && ilclient_create_component(client, &clock, "clock", ILCLIENT_DISABLE_ALL_PORTS) != 0)
		status = -14;
	list[2] = clock;

	memset(&cstate, 0, sizeof(cstate));
	cstate.nSize = sizeof(cstate);
	cstate.nVersion.nVersion = OMX_VERSION;
	cstate.eState = OMX_TIME_ClockStateWaitingForStartTime;
	cstate.nWaitMask = 1;
	if(clock != NULL && OMX_SetParameter(ILC_GET_HANDLE(clock), OMX_IndexConfigTimeClockState, &cstate) != OMX_ErrorNone)
		status = -13;

		   // create video_scheduler
	if(status == 0 && ilclient_create_component(client, &video_scheduler, "video_scheduler", ILCLIENT_DISABLE_ALL_PORTS) != 0)
		status = -14;
	list[3] = video_scheduler;

	set_tunnel(tunnel, video_decode, 131, video_scheduler, 10);
	set_tunnel(tunnel+1, video_scheduler, 11, video_render, 90);
	set_tunnel(tunnel+2, clock, 80, video_scheduler, 12);

	   // setup clock tunnel first
	if(status == 0 && ilclient_setup_tunnel(tunnel+2, 0, 0) != 0)
		status = -15;
	else
		ilclient_change_component_state(clock, OMX_StateExecuting);

	if(status == 0)
		ilclient_change_component_state(video_decode, OMX_StateIdle);

	memset(&format, 0, sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE));
	format.nSize = sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE);
	format.nVersion.nVersion = OMX_VERSION;
	format.nPortIndex = 130;
	format.eCompressionFormat = OMX_VIDEO_CodingAVC;

	if (status == 0 &&
	   OMX_SetParameter(ILC_GET_HANDLE(video_decode), OMX_IndexParamVideoPortFormat, &format) == OMX_ErrorNone &&
	   ilclient_enable_port_buffers(video_decode, 130, NULL, NULL, NULL) == 0)
	{
		OMX_BUFFERHEADERTYPE *buf;
		int port_settings_changed = 0;
		int first_packet = 1;

		ilclient_change_component_state(video_decode, OMX_StateExecuting);
		std::vector<uint8_t> buffer;
		
		while (true)
		{
			madProto proto;
			int result = recvfrom(s.localFd, (char *)&proto, sizeof(madProto), 0, &s.si_other, (socklen_t *)&s.fromLen);
			
			if (result < 0)
			{
				printf("receive failed! error: %d\n", errno);
			}
			else
			{
				uint16_t nal_length = ntohs(proto.nal_length);
				std::copy(proto.payload, proto.payload + nal_length, std::back_inserter(buffer));
				
				if (ntohs(proto.frame_end) == 1)
				{
					data_len = buffer.size();
					std::cout << "starting loop" << std::endl;
					
					while ((buf = ilclient_get_input_buffer(video_decode, 130, 1)) != NULL && data_len > 0)
					{
						std::cout << "starting total size: " << data_len << std::endl;
						
						// Since the frame entire frame is in the buffer
						// it could be bigger than what is allocated.
						// This adjusts the data_len and buffer appropriately.
						if (data_len > buf->nAllocLen)
						{
							memcpy(buf->pBuffer, buffer.data(), buf->nAllocLen);
							buf->nFilledLen = buf->nAllocLen;
							buffer.erase(buffer.begin(), buffer.begin() + buf->nAllocLen);
							data_len = buffer.size();
							std::cout << "total size: " << data_len << std::endl;
						}
						else if (buf->nAllocLen > data_len)
						{
							memcpy(buf->pBuffer, buffer.data(), data_len);
							buf->nFilledLen = data_len;
							buffer.erase(buffer.begin(), buffer.begin() + data_len);
							data_len = buffer.size();
							std::cout << "total size: " << data_len << std::endl;
						}
						
						if (port_settings_changed == 0 &&
						   ((data_len > 0 && ilclient_remove_event(video_decode, OMX_EventPortSettingsChanged, 131, 0, 0, 1) == 0) ||
						    (data_len == 0 && ilclient_wait_for_event(video_decode,
							OMX_EventPortSettingsChanged,
							131,
							0,
							0,
							1,
							ILCLIENT_EVENT_ERROR | ILCLIENT_PARAMETER_CHANGED,
							10) == 0)))
						{
							std::cout << "got a port settings change" << std::endl;
							port_settings_changed = 1;

							if (ilclient_setup_tunnel(tunnel, 0, 0) != 0)
							{
								status = -7;
								break;
							}

							ilclient_change_component_state(video_scheduler, OMX_StateExecuting);

																						            // now setup tunnel to video_render
							if (ilclient_setup_tunnel(tunnel + 1, 0, 1000) != 0)
							{
								status = -12;
								break;
							}

							ilclient_change_component_state(video_render, OMX_StateExecuting);
						}

						//buf->nFilledLen = data_len;
						//data_len = 0;

						//buf->nOffset = 0;
						if (first_packet)
						{
							std::cout << "first packet" << std::endl;
							buf->nFlags = OMX_BUFFERFLAG_STARTTIME;
							first_packet = 0;
						}
						else
						{
							std::cout << "unknown packet timestamp" << std::endl;
							buf->nFlags = OMX_BUFFERFLAG_TIME_UNKNOWN;
						}
						
						if (data_len == 0)
						{
							std::cout << "end of frame" << std::endl;
							buf->nFlags |= OMX_BUFFERFLAG_ENDOFFRAME;
						}

						if (OMX_EmptyThisBuffer(ILC_GET_HANDLE(video_decode), buf) != OMX_ErrorNone)
						{
							std::cout << "empty buffer failed" << std::endl;
							status = -6;
							break;
						}
					}
				}
			}
		}

		buf->nFilledLen = 0;
		buf->nFlags = OMX_BUFFERFLAG_TIME_UNKNOWN | OMX_BUFFERFLAG_EOS;

		if (OMX_EmptyThisBuffer(ILC_GET_HANDLE(video_decode), buf) != OMX_ErrorNone)
			status = -20;

			      // wait for EOS from render
		ilclient_wait_for_event(video_render,
			OMX_EventBufferFlag,
			90,
			0,
			OMX_BUFFERFLAG_EOS,
			0,
			ILCLIENT_BUFFER_FLAG_EOS,
			10000);

			      // need to flush the renderer to allow video_decode to disable its input port
		ilclient_flush_tunnels(tunnel, 0);
	}
	return status;
}

madCoder24
Posts: 18
Joined: Wed Feb 03, 2016 11:38 pm

Re: Openmax Decode and Display

Mon Mar 21, 2016 5:15 pm

I was finally able to solve this and it works... finally! All I did was move ilclient_get_input_buffer to just outside of the while loop. This somehow must have freed up the buffer(s) to allow more decoding. Hopefully, this helps anybody else facing the same kind of issues.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6191
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Openmax Decode and Display

Mon Mar 21, 2016 5:58 pm

Assuming you're referring to the loop

Code: Select all

while ((buf = ilclient_get_input_buffer(video_decode, 130, 1)) != NULL && data_len > 0)
Every time you hit that with data_len <= 0 you'll lose a buffer.
The terms get executed in order, so it will have pulled a buffer out of ilclient, then done nothing with it. Next time round the while(true) loop and buf will be overwritten. Bye bye buf.

Change it to

Code: Select all

while (data_len > 0 && (buf = ilclient_get_input_buffer(video_decode, 130, 1)) != NULL)
and I suspect it would keep going.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

madCoder24
Posts: 18
Joined: Wed Feb 03, 2016 11:38 pm

Re: Openmax Decode and Display

Fri Apr 29, 2016 2:38 pm

I appreciate your reply to my issue. I was also able to solve this by moving the 'ilclient_get_input_buffer' outside of the while loop. I will try your suggestion as well.

jxy
Posts: 1
Joined: Tue May 03, 2016 8:14 am

Re: Openmax Decode and Display

Tue May 03, 2016 8:22 am

I just want to get the data after decoding.How to get it?

madCoder24
Posts: 18
Joined: Wed Feb 03, 2016 11:38 pm

Re: Openmax Decode and Display

Tue May 03, 2016 12:38 pm

You probably want to open a separate topic asking that question, and also show some code. So that we can better debug your situation.

Return to “OpenMAX”