jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23647
Joined: Sat Jul 30, 2011 7:41 pm

Re: H 264 Motion Detection - Compressed Domain

Tue Apr 15, 2014 5:06 pm

It's pretty funky. It's actually possible to extract the motion vectors from a raw H264 stream on the ARM, but it's pretty processor intensive. What Gordon has done is put the results of the CME (coarse motion estimation) HW block as a separate set of 'side' data in the H264, making it much more easily accessible with no real ARM side effort. Just search for the flag saying what the data is and extract it. No real processing involved.

Pretty cool. Although I have a feeling it's just so he can video his golf switch to see where it needs work.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

User avatar
peepo
Posts: 305
Joined: Sun Oct 21, 2012 9:36 am

Re: H 264 Motion Detection - Compressed Domain

Tue Apr 15, 2014 8:15 pm

my experience OS X: Gimp and others report this is not a gif file, recent Safari and Firefox show first frame only, very small no zoom. Preview shows film strip and zoom, but no animation, so hard to identify what is going on...

best,
bring on the blog...

~:"

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1436
Joined: Sat Sep 10, 2011 11:43 am

Re: H 264 Motion Detection - Compressed Domain

Tue Apr 15, 2014 8:38 pm

That's strange considering I created it using GIMP... Plus I have no problems at all viewing it on OS X 10.9.2

Yes it's small, because there is one pixel per macroblock (so just 120x68 pixels).

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

steve_gulick
Posts: 31
Joined: Wed Jul 18, 2012 12:06 pm
Contact: Website

Re: H 264 Motion Detection - Compressed Domain

Tue Apr 15, 2014 11:11 pm

there's a great deal of info on the web regarding motion detection in the compressed domain.

e.g. google " motion vectors moving object detection extraction"

also..
http://iris.usc.edu/vision-notes/biblio ... -i764.html

this looks interesting....
http://sourceforge.net/projects/livevideo
https://sites.google.com/site/wsgyou/research/livevideo


Steve

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: H 264 Motion Detection - Compressed Domain

Wed Apr 16, 2014 11:29 am

This is cool.

gsh, hope you’ll allow me to re-upload that gif:

Image

Would’ve been funny if he’d drawn a sword or picked up a potion.

User avatar
peepo
Posts: 305
Joined: Sun Oct 21, 2012 9:36 am

Re: H 264 Motion Detection - Compressed Domain

Thu Apr 17, 2014 7:36 am

excellent!

links to details on h264, CME (coarse motion estimation) and HW block would be appreciated.
search engines, pah who'd have'em...

~:"

Nobodybeme
Posts: 6
Joined: Sat Jun 01, 2013 4:41 pm

Re: H 264 Motion Detection - Compressed Domain

Sat Apr 19, 2014 6:04 pm

Sweet! Looks great.

steve_gulick
Posts: 31
Joined: Wed Jul 18, 2012 12:06 pm
Contact: Website

Re: H 264 Motion Detection - Compressed Domain

Wed Apr 23, 2014 1:11 pm

Hi Gordon
Any chance of geting some of your early code to try soon?
That gif of your motion vector extraction looks great!!
Just what I need!
Thanks!
Steve

steve_gulick
Posts: 31
Joined: Wed Jul 18, 2012 12:06 pm
Contact: Website

Re: H 264 Motion Detection - Compressed Domain

Fri May 02, 2014 9:25 am

Like many on this forum, motion detection is one of my prime interests.
I also think that doing the motion detection in the compressed (.h264) domain
makes the most sense.
I had been struggling to attempt to extract the motion vectors on the arm side.
I was elated when I learned that Gordon had managed to do that on the GPU side! The gif of the person walking was just what I wanted. I abandoned my own puny attempts at that time in anticipation of using Gordon's code.
I hope it will be available soon!
Being able to access motion vectors using the GPU for motion detection is a feature I think is unique to the Raspberry Pi platform and should find many interested end users
Thank you

User avatar
peepo
Posts: 305
Joined: Sun Oct 21, 2012 9:36 am

Re: H 264 Motion Detection - Compressed Domain

Fri May 02, 2014 10:36 am

me too ~:"

I notice the -x option now produces:
Invalid command line option ((null))

which marks a change,
so perhaps unintended bug in push?

ethanol100
Posts: 585
Joined: Wed Oct 02, 2013 12:28 pm

Re: H 264 Motion Detection - Compressed Domain

Fri May 02, 2014 2:47 pm

peepo wrote:
...
I notice the -x option now produces:
Invalid command line option ((null))
...
only means the option is not known. If you try "-z" it will give the same message.
You need to wait until Gordon releases his changes to raspivid.

User avatar
peepo
Posts: 305
Joined: Sun Oct 21, 2012 9:36 am

Re: H 264 Motion Detection - Compressed Domain

Fri May 02, 2014 5:59 pm

well a few days ago the response was different...
so something in the codebase changed...

mahrk
Posts: 7
Joined: Wed Apr 23, 2014 3:14 pm

Re: H 264 Motion Detection - Compressed Domain

Mon Apr 17, 2017 7:22 pm

Hi, did anyone make any progress or code available for this? Thanks!

cmisip
Posts: 100
Joined: Tue Aug 25, 2015 12:38 am

Re: H 264 Motion Detection - Compressed Domain

Sat May 20, 2017 1:35 pm

Has there been any developments here regarding H264 Motion detection. FFmpeg 3.x has exposed the motion vectors for the software decoder. Is there something equivalent on the RPI hardware decoder side or are the motion vectors in compressed domain still only available on the hardware encode side only?

Thanks
Chris

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7288
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: H 264 Motion Detection - Compressed Domain

Tue May 23, 2017 2:32 pm

cmisip wrote:Has there been any developments here regarding H264 Motion detection. FFmpeg 3.x has exposed the motion vectors for the software decoder. Is there something equivalent on the RPI hardware decoder side or are the motion vectors in compressed domain still only available on the hardware encode side only?
No change on the Pi, and there's unlikely to be.

On the encode side it so happens that part of the motion estimation writes its results back to RAM. On decode it remains in the codec hardware block as the CABAC decoded bitstream is not required anywhere else.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

cmisip
Posts: 100
Joined: Tue Aug 25, 2015 12:38 am

Re: H 264 Motion Detection - Compressed Domain

Mon Jun 26, 2017 12:34 am

Is it possible to use the mmal hardware decoder and encoder simultaneously or can only one component be activated at a time?

Thanks,
Chris

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7288
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: H 264 Motion Detection - Compressed Domain

Mon Jun 26, 2017 7:11 am

cmisip wrote:Is it possible to use the mmal hardware decoder and encoder simultaneously or can only one component be activated at a time?
Both can be used simultaneously, and even multiple encodes and decodes.
The real-time limit is on the overall pixel rate. If you exceed what the hardware can do (around 1080p60 for decode, 1080p40 on encode, and encode requires some decode resource) then it will just start taking longer than the frame period to process the frame.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

cmisip
Posts: 100
Joined: Tue Aug 25, 2015 12:38 am

Re: H 264 Motion Detection - Compressed Domain

Mon Jun 26, 2017 10:35 am

Can you help me figure this out? I am getting:

Code: Select all

DECODER OUTPUT FORMATvc.ril.video_decode:out:0
 type: 3, fourcc: I420
 bitrate: 0, framed: 0
 extra data: 0, (nil)
 width: 704, height: 480, (0,0,704,480)
ENCODER OUTPUT FORMATvc.ril.video_encode:out:0(MP4V)
 type: 3, fourcc: MP4V
 bitrate: 0, framed: 0
 extra data: 0, (nil)
 width: 160, height: 480, (0,0,160,64)

mmal: mmal_vc_port_enable: failed to enable port vc.ril.video_encode:in:0(I420): ENOSPC
mmal: mmal_port_enable: failed to enable port vc.ril.video_encode:in:0(I420)(0xb756f0) (ENOSPC)
failed to enable encoder input port

I am trying to get the encoder component to work. The decoding part works. I modifed example2.c

Code: Select all

/*
Copyright (c) 2012, Broadcom Europe Ltd
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
    * Redistributions of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.
    * Redistributions in binary form must reproduce the above copyright
      notice, this list of conditions and the following disclaimer in the
      documentation and/or other materials provided with the distribution.
    * Neither the name of the copyright holder nor the
      names of its contributors may be used to endorse or promote products
      derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

#include "mmal.h"
#include "util/mmal_default_components.h"
#include "interface/vcos/vcos.h"
#include <stdio.h>
#include <libavcodec/avcodec.h>
#include <libavutil/motion_vector.h>
#include <libavutil/imgutils.h>
#include <libavformat/avformat.h>
#include "libswscale/swscale.h"



#define CHECK_STATUS(status, msg) if (status != MMAL_SUCCESS) { fprintf(stderr, msg"\n"); goto error; }

static uint8_t codec_header_bytes[512];
static unsigned int codec_header_bytes_size = sizeof(codec_header_bytes);

static FILE *source_file;

/* Macros abstracting the I/O, just to make the example code clearer */
#define SOURCE_OPEN(uri) \
   source_file = fopen(uri, "rb"); if (!source_file) goto error;
#define SOURCE_READ_CODEC_CONFIG_DATA(bytes, size) \
   size = fread(bytes, 1, size, source_file); rewind(source_file)
#define SOURCE_READ_DATA_INTO_BUFFER(a) \
   a->length = fread(a->data, 1, a->alloc_size - 128, source_file); \
   a->offset = 0; a->pts = a->dts = MMAL_TIME_UNKNOWN
#define SOURCE_CLOSE() \
   if (source_file) fclose(source_file)

/** Context for our application */
static struct CONTEXT_T {
   VCOS_SEMAPHORE_T semaphore;
   MMAL_QUEUE_T *queue;
} context, econtext;

/** Callback from the input port.
 * Buffer has been consumed and is available to be used again. */
static void input_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
   struct CONTEXT_T *ctx = (struct CONTEXT_T *)port->userdata;

   /* The decoder is done with the data, just recycle the buffer header into its pool */
   mmal_buffer_header_release(buffer);

   /* Kick the processing thread */
   vcos_semaphore_post(&ctx->semaphore);
}

/** Callback from the output port.
 * Buffer has been produced by the port and is available for processing. */
static void output_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
   struct CONTEXT_T *ctx = (struct CONTEXT_T *)port->userdata;
   /* Queue the decoded video frame */
   mmal_queue_put(ctx->queue, buffer);

   /* Kick the processing thread */
   vcos_semaphore_post(&ctx->semaphore);
}

static void einput_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
   struct CONTEXT_T *ctx = (struct CONTEXT_T *)port->userdata;

   /* The decoder is done with the data, just recycle the buffer header into its pool */
   mmal_buffer_header_release(buffer);

   /* Kick the processing thread */
   vcos_semaphore_post(&ctx->semaphore);
}

/** Callback from the output port.
 * Buffer has been produced by the port and is available for processing. */
static void eoutput_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
   struct CONTEXT_T *ctx = (struct CONTEXT_T *)port->userdata;
   /* Queue the decoded video frame */
   mmal_queue_put(ctx->queue, buffer);

   /* Kick the processing thread */
   vcos_semaphore_post(&ctx->semaphore);
}

static void SaveFrame(AVFrame *pFrame, int width, int height, int iFrame)
{
    FILE *pFile;
    char szFilename[32];
    int  y;

    // Open file
    sprintf(szFilename, "frame%d.ppm", iFrame);
    printf("Open File: frame%d.ppm\n", iFrame);
    pFile=fopen(szFilename, "wb");
    if(pFile==NULL)
        return;

    // Write header
    fprintf(pFile, "P6\n%d %d\n255\n", width, height);

    // Write pixel data
    for(y=0; y<height; y++)
        fwrite(pFrame->data[0]+y*pFrame->linesize[0], 1, width*3, pFile);
    
    // Close file
    fclose(pFile);
}


int main(int argc, char **argv)
{
   MMAL_STATUS_T status = MMAL_EINVAL;
   MMAL_COMPONENT_T *decoder = 0;
   MMAL_COMPONENT_T *encoder = 0;
   MMAL_POOL_T *pool_in = 0, *pool_out = 0;
   MMAL_POOL_T *epool_in = 0, *epool_out = 0;
   unsigned int count;

   if (argc < 2)
   {
      fprintf(stderr, "invalid arguments\n");
      return -1;
   }

   vcos_semaphore_create(&context.semaphore, "example", 1);

   SOURCE_OPEN(argv[1]);

   /* Create the decoder component.
    * This specific component exposes 2 ports (1 input and 1 output). Like most components
    * its expects the format of its input port to be set by the client in order for it to
    * know what kind of data it will be fed. */
   status = mmal_component_create(MMAL_COMPONENT_DEFAULT_VIDEO_DECODER, &decoder);
   CHECK_STATUS(status, "failed to create decoder");

   status = mmal_component_create(MMAL_COMPONENT_DEFAULT_VIDEO_ENCODER, &encoder);
   CHECK_STATUS(status, "failed to create encoder");

   /* Set format of video decoder input port */
   MMAL_ES_FORMAT_T *format_in = decoder->input[0]->format;
   format_in->type = MMAL_ES_TYPE_VIDEO;
   format_in->encoding = MMAL_ENCODING_H264;
   format_in->es->video.width = 704;
   format_in->es->video.height = 480;
   format_in->es->video.frame_rate.num = 30;
   format_in->es->video.frame_rate.den = 1;
   format_in->es->video.par.num = 1;
   format_in->es->video.par.den = 1;
   /* If the data is known to be framed then the following flag should be set:
    * format_in->flags |= MMAL_ES_FORMAT_FLAG_FRAMED; */


   SOURCE_READ_CODEC_CONFIG_DATA(codec_header_bytes, codec_header_bytes_size);
   status = mmal_format_extradata_alloc(format_in, codec_header_bytes_size);
   CHECK_STATUS(status, "failed to allocate extradata");
   format_in->extradata_size = codec_header_bytes_size;
   if (format_in->extradata_size)
      memcpy(format_in->extradata, codec_header_bytes, format_in->extradata_size);


   status = mmal_port_format_commit(decoder->input[0]);
   CHECK_STATUS(status, "failed to commit decoder input format");

   /* Display the output port format decoder */
   MMAL_ES_FORMAT_T *format_out = decoder->output[0]->format;
   fprintf(stderr, "DECODER OUTPUT FORMAT");
   fprintf(stderr, "%s\n", decoder->output[0]->name);
   fprintf(stderr, " type: %i, fourcc: %4.4s\n", format_out->type, (char *)&format_out->encoding);
   fprintf(stderr, " bitrate: %i, framed: %i\n", format_out->bitrate,
           !!(format_out->flags & MMAL_ES_FORMAT_FLAG_FRAMED));
   fprintf(stderr, " extra data: %i, %p\n", format_out->extradata_size, format_out->extradata);
   fprintf(stderr, " width: %i, height: %i, (%i,%i,%i,%i)\n",
           format_out->es->video.width, format_out->es->video.height,
           format_out->es->video.crop.x, format_out->es->video.crop.y,
           format_out->es->video.crop.width, format_out->es->video.crop.height);

   /* Set format of video encoder input port */
   MMAL_ES_FORMAT_T *eformat_in = encoder->input[0]->format;
   eformat_in->type = MMAL_ES_TYPE_VIDEO;
   eformat_in->encoding = MMAL_ENCODING_I420;
   eformat_in->es->video.width = 704;
   eformat_in->es->video.height = 480;
   eformat_in->es->video.frame_rate.num = 30;
   eformat_in->es->video.frame_rate.den = 1;
   eformat_in->es->video.par.num = 1;
   eformat_in->es->video.par.den = 1;
   /* If the data is known to be framed then the following flag should be set:
    * format_in->flags |= MMAL_ES_FORMAT_FLAG_FRAMED; */

   status = mmal_port_format_commit(encoder->input[0]);
   CHECK_STATUS(status, "failed to commit encoder input format");

   /* Set format of video encoder output port */
   MMAL_ES_FORMAT_T *eformat_out = encoder->output[0]->format;
   eformat_out->type = MMAL_ES_TYPE_VIDEO;
   eformat_out->encoding = MMAL_ENCODING_H264;
   eformat_out->es->video.width = 704;
   eformat_out->es->video.height = 480;
   eformat_out->es->video.frame_rate.num = 30;
   eformat_out->es->video.frame_rate.den = 1;
   eformat_out->es->video.par.num = 1;
   eformat_out->es->video.par.den = 1;

   status = mmal_port_format_commit(encoder->output[0]);
   CHECK_STATUS(status, "failed to commit encoder output format");

   /* Display the output port format encoder */
   //MMAL_ES_FORMAT_T *eformat_out = encoder->output[0]->format;
   fprintf(stderr, "ENCODER OUTPUT FORMAT");
   fprintf(stderr, "%s\n", encoder->output[0]->name);
   fprintf(stderr, " type: %i, fourcc: %4.4s\n", eformat_out->type, (char *)&eformat_out->encoding);
   fprintf(stderr, " bitrate: %i, framed: %i\n", eformat_out->bitrate,
           !!(eformat_out->flags & MMAL_ES_FORMAT_FLAG_FRAMED));
   fprintf(stderr, " extra data: %i, %p\n", eformat_out->extradata_size, eformat_out->extradata);
   fprintf(stderr, " width: %i, height: %i, (%i,%i,%i,%i)\n",
           eformat_out->es->video.width, eformat_out->es->video.height,
           eformat_out->es->video.crop.x, eformat_out->es->video.crop.y,
           eformat_out->es->video.crop.width, eformat_out->es->video.crop.height);

   getchar();
   

   
   

   /* The format of both ports is now set so we can get their buffer requirements and create
    * our buffer headers. We use the buffer pool API to create these. */
   decoder->input[0]->buffer_num = decoder->input[0]->buffer_num_min;
   decoder->input[0]->buffer_size = decoder->input[0]->buffer_size_min;
   decoder->output[0]->buffer_num = decoder->output[0]->buffer_num_min;
   decoder->output[0]->buffer_size = decoder->output[0]->buffer_size_min;
   pool_in = mmal_pool_create(decoder->input[0]->buffer_num,
                              decoder->input[0]->buffer_size);
   pool_out = mmal_pool_create(decoder->output[0]->buffer_num,
                               decoder->output[0]->buffer_size);

   encoder->input[0]->buffer_num = encoder->input[0]->buffer_num_min;
   encoder->input[0]->buffer_size = encoder->input[0]->buffer_size_min;
   encoder->output[0]->buffer_num = encoder->output[0]->buffer_num_min;
   encoder->output[0]->buffer_size = encoder->output[0]->buffer_size_min;
   epool_in = mmal_pool_create(encoder->input[0]->buffer_num,
                              encoder->input[0]->buffer_size);
   epool_out = mmal_pool_create(encoder->output[0]->buffer_num,
                               encoder->output[0]->buffer_size);

   /* Create a queue to store our decoded video frames. The callback we will get when
    * a frame has been decoded will put the frame into this queue. */
   context.queue = mmal_queue_create();
   econtext.queue = mmal_queue_create();

   /* Store a reference to our context in each port (will be used during callbacks) */
   decoder->input[0]->userdata = (void *)&context;
   decoder->output[0]->userdata = (void *)&context;
   encoder->input[0]->userdata = (void *)&econtext;
   encoder->output[0]->userdata = (void *)&econtext;

   /* Enable all the input port and the output port.
    * The callback specified here is the function which will be called when the buffer header
    * we sent to the component has been processed. */
   

   status = mmal_port_enable(encoder->input[0], einput_callback);
   CHECK_STATUS(status, "failed to enable encoder input port");
   status = mmal_port_enable(encoder->output[0], eoutput_callback);
   CHECK_STATUS(status, "failed to enable encoder output port");

   status = mmal_port_enable(decoder->input[0], input_callback);
   CHECK_STATUS(status, "failed to enable decoder input port");
   status = mmal_port_enable(decoder->output[0], output_callback);
   CHECK_STATUS(status, "failed to enable decoder output port");

   /* Component won't start processing data until it is enabled. */
   status = mmal_component_enable(decoder);
   CHECK_STATUS(status, "failed to enable component");

   status = mmal_component_enable(encoder);
   CHECK_STATUS(status, "failed to enable component");
   getchar();
   /* Start decoding */
   fprintf(stderr, "start decoding\n");
   int fcount=0;
   /* This is the main processing loop */
   for (count = 0; count < 500; count++)
   {
      MMAL_BUFFER_HEADER_T *buffer;

      /* Wait for buffer headers to be available on either of the decoder ports */
      vcos_semaphore_wait(&context.semaphore);

      /* Send data to decode to the input port of the video decoder */
      if ((buffer = mmal_queue_get(pool_in->queue)) != NULL)
      {
         SOURCE_READ_DATA_INTO_BUFFER(buffer);
         if (!buffer->length)
            break;

         fprintf(stderr, "sending %i bytes\n", (int)buffer->length);
         status = mmal_port_send_buffer(decoder->input[0], buffer);
         CHECK_STATUS(status, "failed to send buffer");
      }

      /* Get our decoded frames */
      
      while ((buffer = mmal_queue_get(context.queue)) != NULL)
      {
         /* We have a frame, do something with it (why not display it for instance?).
          * Once we're done with it, we release it. It will automatically go back
          * to its original pool so it can be reused for a new video frame.
          */
         fprintf(stderr, "decoded frame\n");
         fprintf(stderr, "receiving %i bytes\n", (int)buffer->length);
         // Initialize the AVFrame
         AVFrame* frame = av_frame_alloc();
         frame->width = 704;
         frame->height = 480;
         frame->format = AV_PIX_FMT_YUV420P;
         // Initialize frame->linesize
         avpicture_fill((AVPicture*)frame, buffer->data, frame->format, frame->width, frame->height);
         //avpicture_fill((AVPicture*)frame, NULL, frame->format, frame->width, frame->height);
         fcount++;
         //SaveFrame(frame, frame->width, frame->height, fcount);


         //char file_name[100];
         //sprintf(file_name, "frame%d.jpg", ct + 1);

         av_frame_free(&frame);
         mmal_buffer_header_release(buffer);
      }

      /* Send empty buffers to the output port of the decoder */
      while ((buffer = mmal_queue_get(pool_out->queue)) != NULL)
      {
         status = mmal_port_send_buffer(decoder->output[0], buffer);
         CHECK_STATUS(status, "failed to send buffer");
      }
   }

   /* Stop decoding */
   fprintf(stderr, "stop decoding\n");

   /* Stop everything. Not strictly necessary since mmal_component_destroy()
    * will do that anyway */
   mmal_port_disable(decoder->input[0]);
   mmal_port_disable(decoder->output[0]);
   mmal_component_disable(decoder);

 error:
   /* Cleanup everything */
   if (decoder)
      mmal_component_destroy(decoder);
   if (pool_in)
      mmal_pool_destroy(pool_in);
   if (pool_out)
      mmal_pool_destroy(pool_out);
   if (context.queue)
      mmal_queue_destroy(context.queue);

   SOURCE_CLOSE();
   vcos_semaphore_delete(&context.semaphore);
   return status == MMAL_SUCCESS ? 0 : -1;
}

I thought I was running out of video memory but I could just have set the wrong parameter in encoder input format. I don't know what the encoder accepts. I simply created the encoder component as a mirror of the decoder component.

Also, is the output format of the decoder AV_PIX_FMT_YUV420P in libavcodec?

Thanks,
Chris
Last edited by cmisip on Mon Jun 26, 2017 11:53 am, edited 2 times in total.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7288
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: H 264 Motion Detection - Compressed Domain

Mon Jun 26, 2017 11:32 am

It would have been better to open a new thread, but never mind.

Leaving width: 160, height: 64, (0,0,160,64) on the output of the encoder is telling the codec to try and crop the input down from 320x240 to 160x64. Not going to be too great.

On a similar track, 'MP4V' is MMAL_ENCODING_MP4V aka MPEG4. AFAIK Only H264 and MJPEG encode are enabled on the Pi, so using MMAL_ENCODING_H264 would be more sensible and probably explains the error.

The formats follow the normal definitions at http://fourcc.org/yuv.php, or you can look up OMX_COLOR_FormatYUV420PackedPlanar in the IL spec as that is the equivalent to I420.
AV_PIX_FMT_YUV420P based on https://ffmpeg.org/doxygen/2.7/pixfmt_8 ... 5b5d7f0cc2 saying it is "planar YUV 4:2:0, 12bpp, (1 Cr & Cb sample per 2x2 Y samples)" would follow as most likely the same format, although you may need to watch for any extra padding added to the images.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

cmisip
Posts: 100
Joined: Tue Aug 25, 2015 12:38 am

Re: H 264 Motion Detection - Compressed Domain

Mon Jun 26, 2017 11:52 am

I changed the output encoding format of the encoder to MMAL_ENCODING_H264 and the ENOSPC erro went away. But I can't seem to be able to change output size. Can you check the updated code in the previous post?
Here is the output

Code: Select all

DECODER OUTPUT FORMATvc.ril.video_decode:out:0
 type: 3, fourcc: I420
 bitrate: 0, framed: 0
 extra data: 0, (nil)
 width: 704, height: 480, (0,0,704,480)
ENCODER OUTPUT FORMATvc.ril.video_encode:out:0(H264)
 type: 3, fourcc: H264
 bitrate: 0, framed: 0
 extra data: 0, (nil)
 width: 160, height: 480, (0,0,160,64)
That's despite successful commit of encodr output format.

Thanks, Chris

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7288
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: H 264 Motion Detection - Compressed Domain

Mon Jun 26, 2017 12:50 pm

Does it encode correctly?
I can't see anything immediately obvious with your code setting the output format, but I haven't got the time to check it at the moment.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

cmisip
Posts: 100
Joined: Tue Aug 25, 2015 12:38 am

Re: H 264 Motion Detection - Compressed Domain

Thu Jun 29, 2017 3:47 am

I can't get it to work. I put together another test code where I am using libavcodec to decode the h264 stream and then I am hoping to pass along the AVFrame frame->buf data to the MMAL_BUFFER_HEADER_T buffer.

Without the mmal stuff, the program runs fine and writes 100 frames of ppm. If I try to integrate the mmal code, I get a segfault. I have made assumptions though so the implementation might be wrong. I assumed the format of frame->buf is useable as buffer->data. Earlier attempts to solve the problem yielded errors that stated something about no callback being defined. But the input and output callbacks have been defined.

Can you take a look and see where the problem might be?

I am trying to see if I can use the mmal encoder to extract motion vector information. The actually form of the program that I am trying to write is consistent with this latest effort. The only difference would be that I would be using the h264_mmal codec of libavcodec on the h264 decode of the rtsp stream as I know ffmpeg's implementation of mmal works without issue.

The input and output of the encoder as follows:

Code: Select all

INPUT FORMAT 
vc.ril.video_encode:in:0(I420)
 type: 3, fourcc: I420
 bitrate: 0, framed: 0
 extra data: 0, (nil)
 width: 704, height: 64, (0,0,160,64)
OUTPUT FORMAT 
vc.ril.video_encode:out:0(H264)
 type: 3, fourcc: H264
 bitrate: 0, framed: 0
 extra data: 0, (nil)
 width: 160, height: 480, (0,0,160,64)
As you can see, the widths and heights are messed up. Maybe the assignment of the callback functions are messed up also but I can't figure out why. As you pointed out, there is no obvious error on the mmal setup side.

Does the encoder perhaps accept only certain resolutions?
Do I need a specific GPU/MEM split, but I don't think that is the case since I have used ffmpeg's h264_omx without issue without needing to change GPU/MEM.


Thanks,
Chris

Code: Select all

#include "mmal.h"
#include "util/mmal_default_components.h"
#include "interface/vcos/vcos.h"
#include <stdio.h>
#include <libavcodec/avcodec.h>
#include <libavutil/motion_vector.h>
#include <libavutil/imgutils.h>
#include <libavformat/avformat.h>
#include "libswscale/swscale.h"




#include <libavutil/motion_vector.h>
#include <libavformat/avformat.h>

static AVFormatContext *fmt_ctx = NULL;
static AVCodecContext *video_dec_ctx = NULL;
static AVStream *video_stream = NULL;
static const char *src_filename = NULL;

static int video_stream_idx = -1;
static AVFrame *frame = NULL;
static int video_frame_count = 0;

static int decode_packet(  AVPacket *pkt)
{
 int framecomplete=0;
 int ret;
 while (!framecomplete) {
   if (av_read_frame(fmt_ctx, pkt) >= 0) {
      ret = avcodec_send_packet( video_dec_ctx, pkt );
      if ( ret < 0 ) {
        continue;
      }
      ret = avcodec_receive_frame( video_dec_ctx, frame );
      if ( ret < 0 ) {
        continue;
      }
      framecomplete = 1;
  }
}
  
    return 0;
}

static int open_codec_context(AVFormatContext *fmt_ctx, enum AVMediaType type)
{
    int ret;
    AVStream *st;
    AVCodecContext *dec_ctx = NULL;
    AVCodec *dec = NULL;
    AVDictionary *opts = NULL;

    ret = av_find_best_stream(fmt_ctx, type, -1, -1, &dec, 0);
    if (ret < 0) {
        fprintf(stderr, "Could not find %s stream in input file '%s'\n",
                av_get_media_type_string(type), src_filename);
        return ret;
    } else {
        int stream_idx = ret;
        st = fmt_ctx->streams[stream_idx];

        dec_ctx = avcodec_alloc_context3(dec);
        if (!dec_ctx) {
            fprintf(stderr, "Failed to allocate codec\n");
            return AVERROR(EINVAL);
        }

        ret = avcodec_parameters_to_context(dec_ctx, st->codecpar);
        if (ret < 0) {
            fprintf(stderr, "Failed to copy codec parameters to codec context\n");
            return ret;
        }

        /* Init the video decoder */
        av_dict_set(&opts, "flags2", "+export_mvs", 0);
        if ((ret = avcodec_open2(dec_ctx, dec, &opts)) < 0) {
            fprintf(stderr, "Failed to open %s codec\n",
                    av_get_media_type_string(type));
            return ret;
        }

        video_stream_idx = stream_idx;
        video_stream = fmt_ctx->streams[video_stream_idx];
        video_dec_ctx = dec_ctx;
    }

    return 0;
}


#define CHECK_STATUS(status, msg) if (status != MMAL_SUCCESS) { fprintf(stderr, msg"\n"); goto end; }

static uint8_t codec_header_bytes[512];
static unsigned int codec_header_bytes_size = sizeof(codec_header_bytes);

static FILE *source_file;

/* Macros abstracting the I/O, just to make the example code clearer */
#define SOURCE_OPEN(uri) \
   source_file = fopen(uri, "rb"); if (!source_file) goto end;
#define SOURCE_READ_CODEC_CONFIG_DATA(bytes, size) \
   size = fread(bytes, 1, size, source_file); rewind(source_file)
#define SOURCE_READ_DATA_INTO_BUFFER(a) \
   a->length = fread(a->data, 1, a->alloc_size - 128, source_file); \
   a->offset = 0; a->pts = a->dts = MMAL_TIME_UNKNOWN
#define SOURCE_CLOSE() \
   if (source_file) fclose(source_file)

/** Context for our application */
static struct CONTEXT_T {
   VCOS_SEMAPHORE_T semaphore;
   MMAL_QUEUE_T *queue;
} context;

/** Callback from the input port.
 * Buffer has been consumed and is available to be used again. */
static void input_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
   struct CONTEXT_T *ctx = (struct CONTEXT_T *)port->userdata;

   /* The decoder is done with the data, just recycle the buffer header into its pool */
   mmal_buffer_header_release(buffer);

   /* Kick the processing thread */
   vcos_semaphore_post(&ctx->semaphore);
}

/** Callback from the output port.
 * Buffer has been produced by the port and is available for processing. */
static void output_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
   struct CONTEXT_T *ctx = (struct CONTEXT_T *)port->userdata;
   //AVCodecContext *avctx = (AVCodecContext*)port->userdata;
   //MMALDecodeContext *ctx = avctx->priv_data;
   /* Queue the decoded video frame */
   mmal_queue_put(ctx->queue, buffer);

   /* Kick the processing thread */
   vcos_semaphore_post(&ctx->semaphore);
}

static void SaveFrame(AVFrame *pFrame, int width, int height, int iFrame)
{
    FILE *pFile;
    char szFilename[32];
    int  y;

    // Open file
    sprintf(szFilename, "frame%d.ppm", iFrame);
    printf("Open File: frame%d.ppm\n", iFrame);
    pFile=fopen(szFilename, "wb");
    if(pFile==NULL)
        return;

    //RGB
    // Write header 
    //fprintf(pFile, "P6\n%d %d\n255\n", width, height);

    // Write pixel data
    //for(y=0; y<height; y++)
    //    fwrite(pFrame->data[0]+y*pFrame->linesize[0], 1, width*3, pFile);
    
    //YUV420 
    fprintf(pFile, "P5\n%d %d\n255\n", width, height); //P5 is grayscale, need to convert to RGB24 to get color
    for (y = 0; y < height; y++)
    {
      fwrite(pFrame->data[0] + y*pFrame->linesize[0], 1, width, pFile);
    }

    for (y = 0; y < height / 2; y++)
    {
    fwrite(pFrame->data[1] + y*pFrame->linesize[1], 1, width / 2, pFile);
    fwrite(pFrame->data[2] + y*pFrame->linesize[2], 1, width / 2, pFile);
    }
   
    printf("Done writing");

    // Close file
    fclose(pFile);
}





int main(int argc, char **argv)
{
    int ret = 0;
    AVPacket pkt = { 0 };

    if (argc != 2) {
        fprintf(stderr, "Usage: %s <video>\n", argv[0]);
        exit(1);
    }
    src_filename = argv[1];

    av_register_all();

    avformat_network_init();

    if (avformat_open_input(&fmt_ctx, src_filename, NULL, NULL) < 0) {
        fprintf(stderr, "Could not open source file %s\n", src_filename);
        exit(1);
    }

    if (avformat_find_stream_info(fmt_ctx, NULL) < 0) {
        fprintf(stderr, "Could not find stream information\n");
        exit(1);
    }

    open_codec_context(fmt_ctx, AVMEDIA_TYPE_VIDEO);

    av_dump_format(fmt_ctx, 0, src_filename, 0);

    if (!video_stream) {
        fprintf(stderr, "Could not find video stream in the input, aborting\n");
        ret = 1;
        goto end;
    }

    frame = av_frame_alloc();
    if (!frame) {
        fprintf(stderr, "Could not allocate frame\n");
        ret = AVERROR(ENOMEM);
        goto end;
    }


    


 
//MMAL
   MMAL_STATUS_T status = MMAL_EINVAL;
   MMAL_COMPONENT_T *encoder = 0;
   MMAL_POOL_T *pool_in = 0, *pool_out = 0;
   unsigned int count;

   if (argc < 2)
   {
      fprintf(stderr, "invalid arguments\n");
      return -1;
   }

   vcos_semaphore_create(&context.semaphore, "example", 1);

   //SOURCE_OPEN(argv[1]);

   /* Create the decoder component.
    * This specific component exposes 2 ports (1 input and 1 output). Like most components
    * its expects the format of its input port to be set by the client in order for it to
    * know what kind of data it will be fed. */
   status = mmal_component_create(MMAL_COMPONENT_DEFAULT_VIDEO_ENCODER, &encoder);
   CHECK_STATUS(status, "failed to create encoder");

   /* Set format of video decoder input port */
   MMAL_ES_FORMAT_T *format_in = encoder->input[0]->format;
   format_in->type = MMAL_ES_TYPE_VIDEO;
   format_in->encoding = MMAL_ENCODING_I420;
   format_in->es->video.width = 704;
   format_in->es->video.height = 480;
   format_in->es->video.frame_rate.num = 30;
   format_in->es->video.frame_rate.den = 1;
   format_in->es->video.par.num = 1;
   format_in->es->video.par.den = 1;
   /* If the data is known to be framed then the following flag should be set:
    * format_in->flags |= MMAL_ES_FORMAT_FLAG_FRAMED; */

   /*SOURCE_READ_CODEC_CONFIG_DATA(codec_header_bytes, codec_header_bytes_size);
   status = mmal_format_extradata_alloc(format_in, codec_header_bytes_size);
   CHECK_STATUS(status, "failed to allocate extradata");
   format_in->extradata_size = codec_header_bytes_size;
   if (format_in->extradata_size)
      memcpy(format_in->extradata, codec_header_bytes, format_in->extradata_size);
 */

   status = mmal_port_format_commit(encoder->input[0]);
   CHECK_STATUS(status, "failed to commit input format");

   MMAL_ES_FORMAT_T *format_out = encoder->output[0]->format;
   format_out->type = MMAL_ES_TYPE_VIDEO;
   format_out->encoding = MMAL_ENCODING_H264;
   format_out->es->video.width = 704;
   format_out->es->video.height = 480;
   format_out->es->video.frame_rate.num = 30;
   format_out->es->video.frame_rate.den = 1;
   format_out->es->video.par.num = 1;
   format_out->es->video.par.den = 1;
 
   
   status = mmal_port_format_commit(encoder->output[0]);
   CHECK_STATUS(status, "failed to commit output format");

   /* Display the in port format */
   fprintf(stderr, "INPUT FORMAT \n");
   fprintf(stderr, "%s\n", encoder->input[0]->name);
   fprintf(stderr, " type: %i, fourcc: %4.4s\n", format_in->type, (char *)&format_in->encoding);
   fprintf(stderr, " bitrate: %i, framed: %i\n", format_in->bitrate,
           !!(format_in->flags & MMAL_ES_FORMAT_FLAG_FRAMED));
   fprintf(stderr, " extra data: %i, %p\n", format_in->extradata_size, format_in->extradata);
   fprintf(stderr, " width: %i, height: %i, (%i,%i,%i,%i)\n",
           format_in->es->video.width, format_in->es->video.height,
           format_in->es->video.crop.x, format_in->es->video.crop.y,
           format_in->es->video.crop.width, format_in->es->video.crop.height);

   /* Display the output port format */
   fprintf(stderr, "OUTPUT FORMAT \n");
   fprintf(stderr, "%s\n", encoder->output[0]->name);
   fprintf(stderr, " type: %i, fourcc: %4.4s\n", format_out->type, (char *)&format_out->encoding);
   fprintf(stderr, " bitrate: %i, framed: %i\n", format_out->bitrate,
           !!(format_out->flags & MMAL_ES_FORMAT_FLAG_FRAMED));
   fprintf(stderr, " extra data: %i, %p\n", format_out->extradata_size, format_out->extradata);
   fprintf(stderr, " width: %i, height: %i, (%i,%i,%i,%i)\n",
           format_out->es->video.width, format_out->es->video.height,
           format_out->es->video.crop.x, format_out->es->video.crop.y,
           format_out->es->video.crop.width, format_out->es->video.crop.height);

   getchar();

   /* The format of both ports is now set so we can get their buffer requirements and create
    * our buffer headers. We use the buffer pool API to create these. */
   encoder->input[0]->buffer_num = encoder->input[0]->buffer_num_min;
   encoder->input[0]->buffer_size = encoder->input[0]->buffer_size_min;
   encoder->output[0]->buffer_num = encoder->output[0]->buffer_num_min;
   encoder->output[0]->buffer_size = encoder->output[0]->buffer_size_min;
   pool_in = mmal_pool_create(encoder->input[0]->buffer_num,
                              encoder->input[0]->buffer_size);
   pool_out = mmal_pool_create(encoder->output[0]->buffer_num,
                               encoder->output[0]->buffer_size);

   /* Create a queue to store our decoded video frames. The callback we will get when
    * a frame has been decoded will put the frame into this queue. */
   context.queue = mmal_queue_create();

   /* Store a reference to our context in each port (will be used during callbacks) */
   encoder->input[0]->userdata = (void *)&context;
   encoder->output[0]->userdata = (void *)&context;

   /* Enable all the input port and the output port.
    * The callback specified here is the function which will be called when the buffer header
    * we sent to the component has been processed. */
   status = mmal_port_enable(encoder->input[0], input_callback);
   CHECK_STATUS(status, "failed to enable input port");
   status = mmal_port_enable(encoder->output[0], output_callback);
   CHECK_STATUS(status, "failed to enable output port");

   /* Component won't start processing data until it is enabled. */
   status = mmal_component_enable(encoder);
   CHECK_STATUS(status, "failed to enable component");

   /* Start decoding */
   fprintf(stderr, "start decoding\n");
   int fcount=0;


//ENDMMAL


    int bufsize=av_image_get_buffer_size(AV_PIX_FMT_YUV420P, 704, 480, 1);
    uint8_t * fbuffer=NULL;
    fbuffer = (uint8_t *) malloc(bufsize);

//LOOP

int video_frame=0;

//FOLLOWING works
/*while (video_frame < 100) {
     
        if (pkt.stream_index == video_stream_idx)
            video_frame++;
            memset(fbuffer,0,bufsize);
            ret = decode_packet(&pkt);
              //buffer now has data
            if (ret >=0)  
            ret=av_image_copy_to_buffer(fbuffer, bufsize, (const uint8_t **)frame->data, frame->linesize,
                                AV_PIX_FMT_YUV420P, frame->width, frame->height, 1);

            //TEST the fbuffer to see if it contains data
            // Initialize an AVFrame
            AVFrame* bframe = av_frame_alloc();
            bframe->width = 704;
            bframe->height = 480;
            bframe->format = AV_PIX_FMT_YUV420P;
            
            avpicture_fill((AVPicture*)bframe, fbuffer, bframe->format, bframe->width, bframe->height);
            SaveFrame(bframe, bframe->width, bframe->height, video_frame);  //writes 100 frames%d.ppm files so it works
            av_frame_unref(bframe);

            av_packet_unref(&pkt);
        if (ret < 0)
            break;

        av_frame_unref(frame);
}*/


//FOLLOWING DOES NOT WORK
while (video_frame < 100) {

      MMAL_BUFFER_HEADER_T *buffer;

      vcos_semaphore_wait(&context.semaphore);

      if ((buffer = mmal_queue_get(pool_in->queue)) != NULL)
      {
         //SOURCE_READ_DATA_INTO_BUFFER(buffer);  //REPLACED by the next lines         

         //BEGIN REPLACEMENT FOR SOURCE_READ
         memset(fbuffer,0,bufsize);
         if (pkt.stream_index == video_stream_idx)
 
            video_frame++;
            memset(fbuffer,0,bufsize);
            ret = decode_packet(&pkt);
            //frame->buf now has data
            if (ret >=0) //copy it to fbuffer  
               ret=av_image_copy_to_buffer(fbuffer, bufsize, (const uint8_t **)frame->data, frame->linesize,
                                AV_PIX_FMT_YUV420P, frame->width, frame->height, 1);
            SaveFrame(frame, frame->width, frame->height, video_frame);
            
            //copy fbuffer into buffer
            memcpy(buffer->data,fbuffer,bufsize);
            if (buffer->data[0]!=0)
               fprintf(stderr,"OK\n");
            buffer->length = bufsize;
            buffer->offset = 0; buffer->pts = buffer->dts = MMAL_TIME_UNKNOWN;
            
 
            av_packet_unref(&pkt);
        if (ret < 0)
            break;
        
        av_frame_unref(frame);
        //END REPLACEMENT FOR SOURCE_READ
  
         if (!buffer->length)
            break;

         fprintf(stderr, "sending %i bytes\n", (int)buffer->length);
         status = mmal_port_send_buffer(encoder->input[0], buffer);
         CHECK_STATUS(status, "failed to send buffer");
      }

      
      while ((buffer = mmal_queue_get(context.queue)) != NULL)
      {
         fprintf(stderr, "decoded frame\n");
         fprintf(stderr, "receiving %i bytes\n", (int)buffer->length);
         mmal_buffer_header_release(buffer);
      }

      while ((buffer = mmal_queue_get(pool_out->queue)) != NULL)
      {
         status = mmal_port_send_buffer(encoder->output[0], buffer);
         CHECK_STATUS(status, "failed to send buffer");
      } 
   } 



//ENDLOOP

    if (fbuffer)
      free(fbuffer);

    


    /* flush cached frames */
    //decode_packet(NULL);

end: 
    fprintf(stderr, "END");
    avcodec_free_context(&video_dec_ctx);
    avformat_close_input(&fmt_ctx);
    av_frame_free(&frame);

   /* Cleanup everything */
   if (encoder)
      mmal_component_destroy(encoder);
   if (pool_in)
      mmal_pool_destroy(pool_in);
   if (pool_out)
      mmal_pool_destroy(pool_out);
   if (context.queue)
      mmal_queue_destroy(context.queue);

   SOURCE_CLOSE();
   vcos_semaphore_delete(&context.semaphore);
   return status == MMAL_SUCCESS ? 0 : -1;

    return ret < 0;
}


6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7288
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: H 264 Motion Detection - Compressed Domain

Thu Jun 29, 2017 9:29 am

cmisip wrote:I am trying to see if I can use the mmal encoder to extract motion vector information.
Back up there. You're taking an H264 stream, decoding it, and then re-encoding it in an attempt to get the motion vectors? They're in the raw stream to start with!
Admittedly you need to do the CABAC/CAVLC decode step to get to them, but the decoder will be doing that anyway.
Google for "ffmpeg extract motion vectors" and I get a hit https://stackoverflow.com/questions/199 ... video-file. It looks like the tprintf lines he added have been added into the standard ffmpeg now, although I don't know the correct runes to extract the information.

And I've now noticed the section you're missing. You've set
format_in->es->video.width = 704;
format_in->es->video.height = 480;
but not set
format_in->es->video.crop.width = 704;
format_in->es->video.crop.height = 480;
width/height define the geometry/size of the overall buffer. Due to GPU restrictions width has to be a multiple of 32, and height a multiple of 16.
crop.width/height define the valid region within that buffer (there are also crop.x and crop.y, but generally they aren't supported).
You've updated the overall buffer dimensions, but not the valid region, hence why there are weird numbers in the port format.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

cmisip
Posts: 100
Joined: Tue Aug 25, 2015 12:38 am

Re: H 264 Motion Detection - Compressed Domain

Thu Jun 29, 2017 10:54 am

Eureka! It now works. That was the only change necessary. The handoff from ffmpeg frame->buf to mmal buffer->data worked without issues.

I wasn't aware that I needed to do that because I simply reworked the mmal decoding example ( only one I found ) into an encoding example. Can you suggest links that I can read regarding the rpi's mmal implementation.

Code: Select all

Input #0, rtsp, from 'rtsp://admin:admin@192.168.0.78/12':
  Metadata:
    title           : \12
  Duration: N/A, start: 33226.981633, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuv420p(progressive), 704x480, 120 tbr, 90k tbn, 180k tbc
    Stream #0:1: Audio: pcm_alaw, 8000 Hz, 1 channels, s16, 64 kb/s
INPUT FORMAT 
vc.ril.video_encode:in:0(I420)
 type: 3, fourcc: I420
 bitrate: 0, framed: 0
 extra data: 0, (nil)
 width: 704, height: 480, (0,0,704,480)
OUTPUT FORMAT 
vc.ril.video_encode:out:0(H264)
 type: 3, fourcc: H264
 bitrate: 0, framed: 0
 extra data: 0, (nil)
 width: 704, height: 480, (0,0,704,480)
You're taking an H264 stream, decoding it, and then re-encoding it in an attempt to get the motion vectors? They're in the raw stream to start with!
Admittedly you need to do the CABAC/CAVLC decode step to get to them, but the decoder will be doing that anyway.
Are you saying that I only need the mmal decoder to get the motion vectors?

I have working code that extracts motion vectors from h264 stream using libavcodec software decoder. However, there are no motion vectors when I use the ffmpeg hardware (h264_mmal) decoder. I read posts saying that the encoder needs to be used to get to the sideframe data and so that is why I needed to figure out how to send frames to the encoder. Going forward I am thinking I will send the encoder two frames with successive pts and dts and hopefully extract the sideframe data that way. Do you have another algorithm in mind?

I am trying to push the rpi's limits with zoneminder. I managed to add motion vector processing as an alternative to pixel processing for detecting motion but like I said, it only works with software h264 decode. On hardware h264 decode, I have to use pixel processing.

Thanks,
Chris

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7288
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: H 264 Motion Detection - Compressed Domain

Thu Jun 29, 2017 11:42 am

cmisip wrote:
You're taking an H264 stream, decoding it, and then re-encoding it in an attempt to get the motion vectors? They're in the raw stream to start with!
Admittedly you need to do the CABAC/CAVLC decode step to get to them, but the decoder will be doing that anyway.
Are you saying that I only need the mmal decoder to get the motion vectors?

I have working code that extracts motion vectors from h264 stream using libavcodec software decoder. However, there are no motion vectors when I use the ffmpeg hardware (h264_mmal) decoder. I read posts saying that the encoder needs to be used to get to the sideframe data and so that is why I needed to figure out how to send frames to the encoder. Going forward I am thinking I will send the encoder two frames with successive pts and dts and hopefully extract the sideframe data that way. Do you have another algorithm in mind?

I am trying to push the rpi's limits with zoneminder. I managed to add motion vector processing as an alternative to pixel processing for detecting motion but like I said, it only works with software h264 decode. On hardware h264 decode, I have to use pixel processing.
OK, I'd missed that you were using MMAL hardware decode through ffmpeg - you don't appear to have stated that until this last message, and my crystal ball often fails.
You are correct that the hardware can't spit out the motion vectors from the decode path.

Are you needing the decoded images, or are you solely using the motion vectors? If so then you should be able to shortcut the decode to only do the CABAC/CAVLC decode rather than the full decode. I'd hazard a guess that just using an ARM core that will be lighter weight and more efficient than a full decode and encode in the hardware. A Pi0/1 is likely to struggle, but a Pi 2 or 3 shouldn't bat much of an eyelid ... just checked and for a ballpark, on a Pi3 the CABAC decode on a 10Mbit/s stream is <10% of one CPU core.

Whether ffmpeg will do that as standard or not I don't know, but you've got all the code there so could roll your own that only does the first stage of decode.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”