gunny
Posts: 3
Joined: Thu Feb 20, 2014 4:42 pm

H 264 Motion Detection - Compressed Domain

Thu Feb 20, 2014 9:38 pm

Hello all,

I need some help from those more experienced in using the camera and and ffmpeg libraries (and maybe GPU) on the Raspberry Pi. First off, is anyone aware of an accelerated way to get the motion vectors from the GPU for a live h264 stream? If not, that is OK - it just would have been a fast (in terms of execution speed) way to solve my current problem.

The current problem is in using libavformat on the RPI. In my research, I am using motion_vector.c available from the following site:

http://www.princeton.edu/~jiasic/cos435/

This is a program that extracts motion vectors from an h264 stream. The utility hangs in av_open_input_file (or avformat_open_input if you update the utility) when trying to open either a raw h264 file or one that has been appropriately wrapped by MP4box. This utility runs fine on a 64-bit Ubuntu box. It appears to hang after a series of brk() calls according to strace. Has anyone experienced this difficulty?

For more background: I've been working on an application for the Raspberry PI + camera module to capture video of ski jumping for training and competition scoring purposes for our local hills. For both training and judging the requirement is that the camera capture at most two seconds of video of the desired event and save it to a server. The captures need to be motion triggered, preferably with a masking option. 30fps is fine for the moment, but when we get to competition judging on the big hills where skier velocity is ~ 100 kph, we're hoping 720p60 will be ready.

Ideally for our use, motion-mmal would use a faster motion detection method, and would have an option to save the salient clips of the raw h264 stream without ffmpeg processing. Unless I've missed it in the options, the pixel domain method used in motion is incapable of achieved the speed needed. To that end I've been looking at compressed domain motion detection. I've worked up a pretty simple/unsophisticated algorithm that extracts the motion vectors from the raw stream using the utility above, filters them a little to get a stronger hit on correlated vector movement, then creates an average of the square of the vector magnitudes. Macroblocks without MVs are assigned a relatively large pseudo-vector under the assumption that motion prompted the intra encoding. As a result, quiescent frames get a typical average magnitude^2 of under 1, and frames with movement range from 5 to over 100 - it should be easy to dial in whatever level of sensitivity needed. Speed-wise, unoptimized frame analysis code can process 60 720p frames in ~0.15 s on a normally clocked RPI. This does not include MV extraction from the stream (as I am obviously stuck at that point).

Ultimately, I hope to either incorporate this into motion-mmal, or maybe just hack up a version of raspivid with motion detection. Your help in getting to that point is much appreciated.

Matt

User avatar
peepo
Posts: 305
Joined: Sun Oct 21, 2012 9:36 am

Re: H 264 Motion Detection - Compressed Domain

Sat Feb 22, 2014 9:14 am

did you set up a github repository?

ie I would be interested to try your code...

in my limited experience the GPU is amazingly fast.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23708
Joined: Sat Jul 30, 2011 7:41 pm

Re: H 264 Motion Detection - Compressed Domain

Sat Feb 22, 2014 1:56 pm

Mentioned a while back, but the instantaneous bitrate of the video can be a half decent indicator of motion - since the bitrate needs to increase to cope with changes in the scene. Not sure what investigations people did - somewhere on the forums.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1440
Joined: Sat Sep 10, 2011 11:43 am

Re: H 264 Motion Detection - Compressed Domain

Sat Feb 22, 2014 3:04 pm

I'd suggest you trying to take that code and integrate it into Raspivid...

That way you can easily trigger start / stop of the capture within the data callback function... As long as the code is compilable into a library that you could call with the h264 frame it should be fairly simple.

Otherwise just download the H264 spec and hand decode the bitstream, it's not all that difficult...

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

gunny
Posts: 3
Joined: Thu Feb 20, 2014 4:42 pm

Re: H 264 Motion Detection - Compressed Domain

Mon Feb 24, 2014 3:38 pm

Thanks for the feedback.

I don't have anything in Github yet, but hope to soon. Right now, I am just using the aforementioned utility to extract motion vectors, a perl script to experiment with different filters, and small program that I used to benchmark the detection algorithm on the RPi.

Integration into Raspivid sounds like the quickest path to a working proof of concept. I'm working my way through the ITU spec and a few examples from the net, so hopefully the MV extraction code won't take me long to sort out.

I did consider the bitrate detection method, but we need to be able to mask detection areas. I can mask areas quite easily with the MV method, so long as 16x16 pixel granularity is OK. I can also filter based on the prevailing direction of the motion, which will be useful in our application.

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1440
Joined: Sat Sep 10, 2011 11:43 am

Re: H 264 Motion Detection - Compressed Domain

Mon Feb 24, 2014 3:57 pm

I've been discussing this with someone in the office and I'd like to be able to use the vectors to infer image segmentation and object tracking....

So if you create a library that allows me to access the vectors then I should be able to do something with it...

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

User avatar
peepo
Posts: 305
Joined: Sun Oct 21, 2012 9:36 am

Re: H 264 Motion Detection - Compressed Domain

Mon Feb 24, 2014 6:53 pm

gunny,

if you could post either pm, github, web or whatever it would be appreciated.

I like simple easy to understand code, before it gets built into a project....

the linked code was just right, but iirc it didn't build for me....

cheers

~:"

Nobodybeme
Posts: 6
Joined: Sat Jun 01, 2013 4:41 pm

Re: H 264 Motion Detection - Compressed Domain

Sat Mar 01, 2014 8:57 pm

I had a look at the ffmpeg/avcodec code as well - unfortunately it actually does decode the full h.264 bitstream to displayable pixels just to get the motion vectors I think. Also ffmpeg removed access to the motion vectors via that api in 2.x onwards - I think 1.2.x is the latest versions that have them. I can get just about realtime on the pi with 640x480 25fps h.264 baseline, with similar res h.264 high profile goes just about 50% realtime. That was with raspivid -> livehttp segments running in the background - thought that uses single digit cpu resources.

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1440
Joined: Sat Sep 10, 2011 11:43 am

Re: H 264 Motion Detection - Compressed Domain

Sat Mar 01, 2014 9:28 pm

I've been talking about this with the codec team at Broadcom, they think it would be possible, the only problem is they're flat out at the moment, so I might have to look into it!

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

gunny
Posts: 3
Joined: Thu Feb 20, 2014 4:42 pm

Re: H 264 Motion Detection - Compressed Domain

Sun Mar 02, 2014 11:27 pm

Others that I've talked to privately have confirmed that the motion_vector utility linked above does not build against current versions of the ffmpeg libraries. It (libavcodec) was really only useful for the initial research anyways, due to the fact that it fully decodes the picture, and apparently no longer gives easy access to the required MV data.

The library h264bitstream is intriguing.The latest release has a start on decoding slice data, but nothing for macroblocks yet. Unless Gordon comes up with a GPU side solution first (that'd be great!), I'm going to try building on h264bitstream for the MV extraction.

Nobodybeme
Posts: 6
Joined: Sat Jun 01, 2013 4:41 pm

Re: H 264 Motion Detection - Compressed Domain

Fri Mar 21, 2014 11:55 pm

I haven't done anything on this since the previous post. I'm not sure how performant the motion-mmal/v4l motion detection route is as I haven't been keeping up but it might be worth putting this ugly code here for someone to look at. I haven't looked at it in a while so I'm not 100% sure what state I left it in. I was thinking of ripping out unnecessary bits of the ffmpeg h264 decode path to speed it up but lost interest.

How are you getting along with the h264bitstream approach? I have a pretty wooly understanding of h264 but i seem to remember hearing that there's a CABAC decode step before you get access to motion vectors and that itself was reasonably resource intensive? (gprof of motion_watch shows a good bit of time in the ffmpeg cabac routines - but maybe I'm reading it's output wrong).

The code is wrapped in an inotify watcher - the idea being that you have the pi setup to stream video via livehttp - where it just writes transport stream chunks into a directory periodically. The code gets a notification for each new file and parses it. It's currently setup to read 2 frames out of every 6 - so 1/3 of the frames are inspected - it inspects 2 adjacent frames for motion between them. This worked for my raspivid --inline --profile high --nopreview -fps 25 -w 720 -h 405 -b 1048576 -t 0 -o - didn't quite keep up and it would drop a 5 second chunk every so often. It will work faster if the profile is baseline.

I was planning to put some work into triggering on the motion - currently its just summing the vectors and printing the output - I had planned to put some work into doing something more useful with the vectors but never got around to it. Idea was that it should decide if there was motion and if there was to copy the transport stream chunk elsewhere for archival/review - make a daily/hourly video or something. I can upload static binaries if that's useful too. I don't have a use case for this at the moment myself. might be interesting to see if it can be any more accurate than just triggering on bitrate as jamesh pointed out.

Build:

Code: Select all

gcc -O3 -std=c99  -o motion_watch motion_watch.c  -I/home/pi/ffmpeg/ffmpeg-1.2.x/target/include/ /home/pi/ffmpeg/ffmpeg-1.2.x/build/ffmpeg-1.2.1/libavfilter/libavfilter.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/ffmpeg-1.2.1/libswscale/libswscale.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/ffmpeg-1.2.1/libavdevice/libavdevice.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/ffmpeg-1.2.1/libavformat/libavformat.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/ffmpeg-1.2.1/libavcodec/libavcodec.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/ffmpeg-1.2.1/libavutil/libavutil.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/zlib-1.2.8/libz.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/libtheora-1.1.1/lib/.libs/libtheoraenc.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/libtheora-1.1.1/lib/.libs/libtheora.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/libtheora-1.1.1/lib/.libs/libtheoradec.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/yasm-1.2.0/libyasm.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/x264-snapshot-20140109-2245/libx264.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/libvpx-v1.1.0/libvpx_g.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/libvpx-v1.1.0/libvpx.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/faac-1.28/libfaac/.libs/libfaac.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/faac-1.28/common/mp4v2/libmp4v2.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/ffmpeg-1.2.1/libpostproc/libpostproc.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/ffmpeg-1.2.1/libavutil/libavutil.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/ffmpeg-1.2.1/libswresample/libswresample.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/bzip2-1.0.6/libbz2.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/lame-3.99.5/mpglib/.libs/libmpgdecoder.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/lame-3.99.5/libmp3lame/.libs/libmp3lame.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/xvidcore/build/generic/=build/libxvidcore.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/libpng-1.6.2/.libs/libpng16.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/libogg-1.3.1/src/.libs/libogg.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/libvorbis-1.3.3/lib/.libs/libvorbisenc.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/libvorbis-1.3.3/lib/.libs/libvorbis.a /home/pi/ffmpeg/ffmpeg-1.2.x/build/libvorbis-1.3.3/lib/.libs/libvorbisfile.a -lpthread -lbz2 -lm -lz
Running:
Just point it at a directory with new video files appearing as first argument and it gives output like.

Code: Select all

Watching:: /var/run/shm/
New video segment /var/run/shm/mystream-00076878.ts
Execution time = 4.000000
[1393726816] sum mv: 61619.221513, total # mv: 83571

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: H 264 Motion Detection - Compressed Domain

Sat Mar 22, 2014 1:07 pm

If you stream in Baseline profile, isn’t CABAC disabled then?

The the table under Profiles here: https://en.wikipedia.org/wiki/H264#Profiles

User avatar
GTR2Fan
Posts: 1601
Joined: Sun Feb 23, 2014 9:20 pm
Location: South East UK

Re: H 264 Motion Detection - Compressed Domain

Sat Mar 22, 2014 1:16 pm

towolf wrote:If you stream in Baseline profile, isn’t CABAC disabled then?
Unless it's being "illegally" forced on, yes. I believe you also lose B-frames. Baseline is horrendously inefficient in terms of picture quality vs bitrate compared to High.
Pi2B Mini-PC/Media Centre: ARM=1GHz (+3), Core=500MHz, v3d=500MHz, h264=333MHz, RAM=DDR2-1200 (+6/+4/+4+schmoo). Sandisk Ultra HC-I 32GB microSD card on '50=100' OCed slot (42MB/s read) running Raspbian/KODI16, Seagate 3.5" 1.5TB HDD mass storage.

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: H 264 Motion Detection - Compressed Domain

Sat Mar 22, 2014 4:00 pm

Still better than the mjpeg so many people here have a hard-on over.

User avatar
GTR2Fan
Posts: 1601
Joined: Sun Feb 23, 2014 9:20 pm
Location: South East UK

Re: H 264 Motion Detection - Compressed Domain

Sat Mar 22, 2014 4:05 pm

So true! :P
Pi2B Mini-PC/Media Centre: ARM=1GHz (+3), Core=500MHz, v3d=500MHz, h264=333MHz, RAM=DDR2-1200 (+6/+4/+4+schmoo). Sandisk Ultra HC-I 32GB microSD card on '50=100' OCed slot (42MB/s read) running Raspbian/KODI16, Seagate 3.5" 1.5TB HDD mass storage.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23708
Joined: Sat Jul 30, 2011 7:41 pm

Re: H 264 Motion Detection - Compressed Domain

Sat Mar 22, 2014 4:56 pm

towolf wrote:Still better than the mjpeg so many people here have a hard-on over.
As I understand it, more web browsers support MJPEG than H264.

Sadly; since H264 is a wondrous bit of work. And H265 is even better (no we don't support that)
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

Nobodybeme
Posts: 6
Joined: Sat Jun 01, 2013 4:41 pm

Re: H 264 Motion Detection - Compressed Domain

Sat Mar 22, 2014 5:37 pm

Had a try with baseline vs high profile ( raspivid --profile high -> raspivid --profile baseline ). Doesn't appear to make too much of a difference (4/5s -> 3s) - so h264bitstream (non-ffmpeg) parsing should be quite efficient. The timer resolution in the output is unfortunately only 1 second . The video segments are 5 seconds long.

Code: Select all

Baseline:

Seems stream 0 codec frame rate differs from container frame rate: 180000.00 (180000/1) -> 90000.00 (180000/2)
Input #0, mpegts, from '/tmp/mystream-00000007.ts':
  Duration: 00:00:07.13, start: 42178.302378, bitrate: 937 kb/s
  Program 1
    Stream #0.0[0x44]: Video: h264 (Baseline), yuv420p, 720x404, 90k tbr, 90k tbn, 180k tbc
At least one output file must be specified

pi@raspberrypi ~/motion_vector $ ./motion_watch /var/run/shm/
Watching:: /var/run/shm/
New video segment /var/run/shm/mystream-00000013.ts
Execution time = 3.000000
[1395508227] sum mv: 243289.380835, total # mv: 77132
New video segment /var/run/shm/mystream-00000014.ts
Execution time = 3.000000
[1395508234] sum mv: 200218.512869, total # mv: 77875
New video segment /var/run/shm/mystream-00000015.ts
Execution time = 3.000000
[1395508241] sum mv: 189522.803249, total # mv: 79098

High:

Seems stream 0 codec frame rate differs from container frame rate: 180000.00 (180000/1) -> 90000.00 (180000/2)
Input #0, mpegts, from '/tmp/mystream-00000011.ts':
  Duration: 00:00:07.13, start: 42335.868344, bitrate: 938 kb/s
  Program 1
    Stream #0.0[0x44]: Video: h264 (High), yuv420p, 720x404, 90k tbr, 90k tbn, 180k tbc
At least one output file must be specified

pi@raspberrypi ~/motion_vector $ ./motion_watch /var/run/shm/
Watching:: /var/run/shm/
New video segment /var/run/shm/mystream-00000020.ts
Execution time = 4.000000
[1395508406] sum mv: 76330.258367, total # mv: 81730
New video segment /var/run/shm/mystream-00000021.ts
Execution time = 4.000000
[1395508413] sum mv: 72297.966214, total # mv: 81527
New video segment /var/run/shm/mystream-00000022.ts
Execution time = 5.000000
[1395508420] sum mv: 77485.496729, total # mv: 81036
Re: popularity of mjpeg for browser viewing - might be a holdover from security camera viewers. Might be authentication difficulties with livehttp viewers. Might be latency. Livehttp h264 works quite widely: ios natively via link to .m3u8 ( apple tv too ), safari via <video>, other browsers via flash fallback https://github.com/mangui/HLSprovider/, there's play store android apps i haven't tried to patch android's fragmented hls support (search "hls").

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: H 264 Motion Detection - Compressed Domain

Sun Mar 23, 2014 3:34 pm

jamesh wrote:As I understand it, more web browsers support MJPEG than H264.

Sadly; since H264 is a wondrous bit of work. And H265 is even better (no we don't support that)

The problem at the moment is not so much that they don’t support H.264 (most do by now), the problem is that HTML5 video so far was for static, progressively downloaded files. You cannot do live video with that.

So they added HTML5 Media Source Extensions where you can feed little chunks of video to the <video> element using Javascript. This is a variant of Apple’s solution to live streaming over HTTP (called HLS).

Youtube started using MSE already, so they have more control over streaming and can change bitrates on the fly, all over HTTP.

You can see that here: http://www.youtube.com/html5

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1440
Joined: Sat Sep 10, 2011 11:43 am

Re: H 264 Motion Detection - Compressed Domain

Thu Apr 10, 2014 7:15 pm

I got the motion vectors out of the camera today and into Raspivid...

Currently I'm just writing them to a file, but they could be processed to understand the motion in the image simply enough...

Will see if I can get them checked in over the next few days

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

steve_gulick
Posts: 31
Joined: Wed Jul 18, 2012 12:06 pm
Contact: Website

Re: H 264 Motion Detection - Compressed Domain

Fri Apr 11, 2014 1:36 am

That's great, Gordon! Thanks - looking forward to trying it for motion detection
Steve

User avatar
peepo
Posts: 305
Joined: Sun Oct 21, 2012 9:36 am

Re: H 264 Motion Detection - Compressed Domain

Fri Apr 11, 2014 7:28 am

me too!

~:"

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1440
Joined: Sat Sep 10, 2011 11:43 am

Re: H 264 Motion Detection - Compressed Domain

Tue Apr 15, 2014 9:48 am

Just an example of the magnitude of the vectors...

https://www.dropbox.com/s/np7a3h4y1w3mi0y/vectors.gif

Shows you what is possible, this is 1080P30 frames which result in 1 integer per macroblock so 120x68x4 bytes per frame

The magic should be in rpi-update at the moment, but you need some code in raspivid to enable the inline vectors, I'll do a pull request later to sort it...

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

User avatar
peepo
Posts: 305
Joined: Sun Oct 21, 2012 9:36 am

Re: H 264 Motion Detection - Compressed Domain

Tue Apr 15, 2014 3:37 pm

Gordon,

could you provide a commentary or gloss?

the image is small and hard to decipher

many thanks

Jonathan

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1440
Joined: Sat Sep 10, 2011 11:43 am

Re: H 264 Motion Detection - Compressed Domain

Tue Apr 15, 2014 3:43 pm

I was going to do a blog post about it, probably take me a couple of days... Will make sure it's reproducible by then...

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

ethanol100
Posts: 585
Joined: Wed Oct 02, 2013 12:28 pm

Re: H 264 Motion Detection - Compressed Domain

Tue Apr 15, 2014 3:51 pm

That is cool.
Have you downloaded the gif and viewed it in a browser or image view software?
You clearly can the someone walking in the picture from left to right.
I am very interested in these motion vectors!

Return to “Camera board”