Convert video h264 to mp4

[6by9]

New firmware pushed to the same github location. Hopefully that should fix mode 6 and bring it in line with the others in terms of fps accuracy.

[ethanol100]

Yes now everything seems OK.

Newly created fps values:
Pi1: Pi 1 B+ normal camera

Code: Select all

#md fps_requested fps ms %diff
1 30 30.0219 33.309 +0.0730533%
2 15 15.0017 66.659 +0.0115013%
*3 1 0.996502 1003.51 -0.349772%
4 42 42.0253 23.7952 +0.0601962%
5 49 49.0057 20.4058 +0.0115813%
6 60 60.1135 16.6352 +0.189157%
7 90 90.1705 11.0901 +0.189458%

1 15 15.0042 66.6478 +0.028308%
2 5 5.00058 199.977 +0.0115013%
*3 1/6 0.166693 5999.05 +0.0158%
4 21 21.0024 47.6137 +0.0112313%
5 24 24.0111 41.6474 +0.0462614%
6 42 42.1116 23.7464 +0.265825%
7 60 60.1135 16.6352 +0.189157%

Pi2: Pi B+ noir camera

Code: Select all

#md fps_requested fps ms %diff
1 30 30.0183 33.313 +0.0610372%
2 15 14.9999 66.6669 -0.000349999%
*3 1 0.996383 1003.63 -0.361687%
4 42 42.0203 23.798 +0.0484234%
5 49 48.9997 20.4083 -0.000669996%
6 60 60.1063 16.6372 +0.177113%
7 90 90.1599 11.0914 +0.177715%
    
1 15 15.0025 66.6557 +0.0164527%
2 5 4.99998 200.001 -0.000499998%
*3 1/6 0.166673 5999.76 +0.0038%
4 21 20.9999 47.6193 -0.000529997%
5 24 24.0082 41.6524 +0.0342517%
6 42 42.1067 23.7492 +0.254004%
7 60 60.1063 16.6372 +0.177113%

(* For mode 3 I have requested -fps 0 and set the -ss to 1s and 6s.)

[6by9]

Yes now everything seems OK.

(* For mode 3 I have requested -fps 0 and set the -ss to 1s and 6s.)

I do want to double check mode 3. Setting exposure time with -ss is subtly different from selecting framerate, but I think your testing is valid. I just need to make sure that your two test points aren't the absolute extremes of the permissible values and are being slightly clipped - the 6s looks OK, but I'm surprised 1s gives more than 1s.

I'll try to get the changes pushed officially over the weekend, although it then depends on Dom to do an official firmware release.

[ethanol100]

I know setting -ss is wrong and therefore have skipped this mode until now. I think exposure time should be always a little less than the time between to frames. And for 1000ms exposure it takes a bit longer (3.5ms) to advance to the next frame. Which fps values are valid for mode 3? Including or excluding 1fps(1/6<fps<1)?

I can set -fps 1 and not set the -ss value, but then I get exactly the same values:
Pi1:

Code: Select all

3 1 0.996502 1003.51 1.549e-05 -0.349772%

But I can not set the frame rate to 1/6 without modifying raspivid.

And sorry, I only have these two pi's. I cannot provide more data. (I could swap the two cameras.)
Have no idea, how to test this further.

Edit: Got confused, if I just use -md3 -fps 1 the exposure time will be much less than 1s (i.e. 5.055ms) if the room is not dark and I do not see any reason why the requested frame rate should then be lower than 1fps.

[ethanol100]

Ok, after modifying raspivid to accept fractures for the frame rate some values for -md 3:

Code: Select all

#fps fps ms %diff
#pi1
1/1 0.996502 1003.51 -0.349772%
1/2 0.500173 1999.31 +0.0345119%
1/3 0.332145 3010.73 -0.356392%
1/4 0.250086 3998.63 +0.0342617%
1/5 0.199292 5017.76 -0.353943%
1/6 0.166106 6020.26 -0.33653%
#pI2
1/1 0.996383 1003.63 -0.361687%
1/2 0.500113 1999.55 +0.0225051%
1/3 0.332106 3011.09 -0.368305%
1/4 0.250056 3999.1 +0.0225051%
1/5 0.199268 5018.36 -0.365857%
1/6 0.166086 6020.98 -0.348448%

[joe4444]

I realize this thread is old, but I inherited an RPi that seems to have this audio/video sync problem, and I'm not sure what the best solution is. It's an RPi B+ with the official camera (rev 1.3) and the Cirrus Logic Audio card.

I'm using the python picamera module to record 1280x720 at 30 fps. I think I could switch to working with raspivid directly, but my predecessor wrote a script utilizing picamera, so for now I'm just building from that starting point.

For audio, arecord uses 16-bit Little Endian at the minimum frequency (8000 Hz).

Typical recordings will be 45-60 minutes, but in my testing I've only recorded for less than a minute, and synchronizing the audio and video with avconv never seems to work no matter how much I play with the -itsoffset option.

Should I upgrade my kernel and firmware? Is that all it would take to fix the video framerate bug? See my versions below.

Or do I need to update picamera? Or raspivid?

Code: Select all

$ uname -a
Linux raspberrypi 3.18.9+ #1 PREEMPT Thu Aug 13 22:34:38 UTC 2015 armv6l GNU/Linux

Code: Select all

# /opt/vc/bin/vcgencmd version
Jan 30 2015 18:25:11
Copyright (c) 2012 Broadcom
version d6e004c61a7a749897c482c860d0b2c28196437e (clean) (release)

Code: Select all

$ raspivid

raspivid Camera App v1.3.12

Code: Select all

>>> require('picamera')[0].version
'1.9'

Code: Select all

$ avconv
avconv version 9.14-6:9.14-1rpi1rpi1, Copyright (c) 2000-2014 the Libav developers

[joe4444]

I updated my RPi B+ to jessie, but even with the latest (I think) firmware I still have a major audio/video drift problem. The audio is more and more ahead of the video as the video progresses.

Code: Select all

$ uname -a
Linux raspberrypi 4.1.19+ #1 Fri Mar 11 15:05:42 CET 2016 armv6l GNU/Linux

Code: Select all

$ /opt/vc/bin/vcgencmd version
Mar  9 2016 18:15:17
Copyright (c) 2012 Broadcom
version 3a754304b032a5298ee7889b179c667bbc75dec5 (clean) (release)

Code: Select all

$ raspivid

raspivid Camera App v1.3.12

Code: Select all

>>> require('picamera')[0].version
'1.10'

...although, as noted below, I am trying things with raspivid directly now.

Code: Select all

$ avconv
avconv version 11.4-6:11.4-1~deb8u1+rpi1, Copyright (c) 2000-2014 the Libav developers

Here is my full bash script:

Code: Select all

#!/bin/bash

test_for_integer=$(echo "$1" | grep -P "^\d+$")
if [ "$test_for_integer" = "" ]; then
        echo "Must provide an integer for length of recording (in seconds). Try again."
        exit 1
fi

audio_length=$1
video_length=$((audio_length*1000))
timestamp=$(date +"%Y%m%d_%H%M%S")
audio_file="/tmp/audio_stream_${timestamp}"
video_file="/tmp/video_stream_${timestamp}"
merge_file="./merge_${timestamp}"

echo "Recording video and audio..."
raspivid -t ${video_length} -n -w 1280 -h 720 -fps 25 -o ${video_file}.h264 &
arecord -Dhw:sndrpiwsp -d ${audio_length} -c 2 -f s16_LE -r 8000 > ${audio_file}.wav

echo "Converting video to .mp4 format..."
MP4Box -add ${video_file}.h264 ${video_file}.mp4

echo "Merging video and audio files..."
avconv -y -fflags nobuffer -r 25 -i ${video_file}.mp4 -fflags nobuffer -analyzeduration 0 -i ${audio_file}.wav -map 0:0 -map 1:0 -filter:a aresample=async=1 -c:a copy -c:v copy ${merge_file}.mkv
mv "${video_file}.mkv" "${merge_file}.mkv"

Basically I'm trying the suggestions here (http://raspberrypi.stackexchange.com/qu ... microphone) with -r 25.375 (also tried -r 24 and -r 27) for -i ${video_file} and then -r 25 for -i ${audio_file} and and -r 25 for ${merge_file}.mkv with apparently no effect. Whether i specify 25.375 or 24 or 27, it seems like the audio/video drift is unchanged. In fact, the three output files for 25.375 or 24 or 27 appear to be identical.

At first I tried using the .h264 video as the input, but the output "video" is just a still frame with the audio, so I added the MP4Box intermediate step.

[inx]

Hi.

I ran into the same problem, with quite the same setup (rpi2, cirrus logic audiocard, picamera v1.3)
(Linux raspberrypi 3.18.9-v7+ #2 SMP PREEMPT Mon Mar 23 20:25:20 GMT 2015 armv7l GNU/Linux)

i did not look into your python script, but assume it is very similar to what i use.
https://github.com/i-n-g-o/ffch-phone

from using avconv i changed to mux audio and video with gstreamer, to produce an avi with uncompressed audio. it's much faster than using MP4Box. i also needed a delay to start video-recording (6 frames @ 25 fps), otherwise it would be async from the beginning.

i figured the drift is a bit too exact 1 frame per 10 seconds to be random.
i did not have time to investigate it, but it might be related to h264 compression? (just guessing)

how big is your drift?

i'm looking for a solution for this as well...

cheers
inx

[inx]

after posting i found this forum-thread:

viewtopic.php?f=43&t=89540&p=633838&hil ... io#p633838

in which this command is proposed:

Code: Select all

raspivid -t 0 -w 1280 -h 720 -ex fixedfps -fps 25 -b 6000000 -o - | ffmpeg -r 25.37 -i - -f alsa -ar 48000 -ac 1 -itsoffset 1 -i hw:1,0 -c:v copy -r 25 -vsync 1 -c:a aac -strict experimental output-mp4

there also is a link to a ffmpeg version with enabled alsa:

deivid » Wed Oct 29, 2014 8:39 pm
Link here (tar.gz)

it compensates the drift while recording. the values for '-r 25.37' and '-itsoffset 1' has to be figured out manually.
for me this is roughly:
-r 25.1
-itsoffset 5.36
this kind of works for me, but is more a workaround than a solution to the problem.

where does the wrong video-timing origin from?

[6by9]

You have two capture devices (audio and video) with independent clocks and encoding latencies. Something has to be done to pull them into sync.
Ideally one should be slaved off the other, but that is near impossible within Linux. Instead you need things to work off a common timestamp. Does the audio side provide timestamps to avoid drift between the sound card oscillator and Linux timestamp?

[inx]

excactly.
as it is pointed out in other posts, the problem is inaccurate timing in the video-stream. when muxing with audio, it drifts apart.
it seems this is because i use an older kernel and firmware. the reason for the older versions the cirrus audio-card used in the setup, which i could not manage to get working with newer kernel versions (kernel fails to build, project ran out of time, etc...)

i also tried to use gstreamer to record and video/audio together in sync. unfortunately when using the command below the audio-recording breaks off after less than a second:

Code: Select all

raspivid -t 999999 -w 1920 -h 1080 -fps 25 -hf -o - | gst-launch-1.0 -v fdsrc ! video/x-h264,width=1920,height=1080,framerate=25/1 ! h264parse ! queue ! mux. alsasrc device="hw:0,0" ! audioconvert ! 'audio/x-raw,rate=44100,channels=2' ! queue ! mux. avimux name=mux ! filesink location=test.avi

recording audio with a video test-source worked:

Code: Select all

gst-launch-1.0 videotestsrc num-buffers=250 ! 'video/x-raw,format=(string)I420,width=320,height=240,framerate=(fraction)25/1' ! queue ! mux. alsasrc device="hw:0,0" ! audioconvert ! 'audio/x-raw,rate=44100,channels=2' ! queue ! mux. avimux name=mux ! filesink location=test.avi

[6by9]

Why use raspivid (which loses timestamps) when v4l2 presents them? They should also then be off the correct same clock (assuming alsa does something sensible on timestamping to compensate for the audio clock drifting wrt to the system clock).

Don't forget to add the module parameter "gst_v4l2src_is_broken=1" when loading to avoid bad handling in some versions of gstreamer v4l2src component.

[inx]

yes, you are right. piping the frames into gstreamer like this is not optimal. it was a quick test, a last attempt to get it synchronized in one go, a couple of hours before i needed to deliver the hardware.
once i get the hardware back i can continue solving this.

what could be the reason the audio stopped being recorded after short?