Streaming audio and video

[neik]

I am trying to capture the video from the camera board and combine it with audio from a USB sound card. The problem is that the audio lags by over 2 seconds and the video is over 2 seconds behind that!

I am using ffmpeg as below. There are still major sync issues if I write the data to a file instead of streaming it.

Code: Select all

raspivid -o - -t 0 -n -w 1280 -h 720 -fps 25 | ffmpeg -report -probesize 32000 -thread_queue_size 1024 -y -f h264 -r 25 -framerate 25 -i - -thread_queue_size 512 -f alsa -ac 1 -channel_layout mono -i hw:1 -vcodec copy -acodec mp3 -f mpegts udp://44.0.0.95:1234

I can't attach the log as ".txt" and ".log" extensions are not allowed. So here it is:

Code: Select all

ffmpeg started on 2015-06-16 at 09:55:15
Report written to "ffmpeg-20150616-095515.log"
Command line:
ffmpeg -report -probesize 32000 -thread_queue_size 1024 -y -f h264 -r 25 -framerate 25 -i - -thread_queue_size 512 -f alsa -ac 1 -channel_layout mono -i hw:1 -vcodec copy -acodec mp3 -f mpegts udp://44.0.0.95:1234
ffmpeg version N-72834-gfd004e1 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.6 (Debian 4.6.3-14+rpi1)
  configuration: --enable-libmp3lame
  libavutil      54. 27.100 / 54. 27.100
  libavcodec     56. 41.100 / 56. 41.100
  libavformat    56. 36.100 / 56. 36.100
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 16.101 /  5. 16.101
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.100 /  1.  2.100
Splitting the commandline.
Reading option '-report' ... matched as option 'report' (generate a report) with argument '1'.
Reading option '-probesize' ... matched as AVOption 'probesize' with argument '32000'.
Reading option '-thread_queue_size' ... matched as option 'thread_queue_size' (set the maximum number of queued packets from the demuxer) with argument '1024'.
Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'.
Reading option '-f' ... matched as option 'f' (force format) with argument 'h264'.
Reading option '-r' ... matched as option 'r' (set frame rate (Hz value, fraction or abbreviation)) with argument '25'.
Reading option '-framerate' ... matched as AVOption 'framerate' with argument '25'.
Reading option '-i' ... matched as input file with argument '-'.
Reading option '-thread_queue_size' ... matched as option 'thread_queue_size' (set the maximum number of queued packets from the demuxer) with argument '512'.
Reading option '-f' ... matched as option 'f' (force format) with argument 'alsa'.
Reading option '-ac' ... matched as option 'ac' (set number of audio channels) with argument '1'.
Reading option '-channel_layout' ... matched as option 'channel_layout' (set channel layout) with argument 'mono'.
Reading option '-i' ... matched as input file with argument 'hw:1'.
Reading option '-vcodec' ... matched as option 'vcodec' (force video codec ('copy' to copy stream)) with argument 'copy'.
Reading option '-acodec' ... matched as option 'acodec' (force audio codec ('copy' to copy stream)) with argument 'mp3'.
Reading option '-f' ... matched as option 'f' (force format) with argument 'mpegts'.
Reading option 'udp://44.0.0.95:1234' ... matched as output file.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option report (generate a report) with argument 1.
Applying option y (overwrite output files) with argument 1.
Successfully parsed a group of options.
Parsing a group of options: input file -.
Applying option thread_queue_size (set the maximum number of queued packets from the demuxer) with argument 1024.
Applying option f (force format) with argument h264.
Applying option r (set frame rate (Hz value, fraction or abbreviation)) with argument 25.
Successfully parsed a group of options.
Opening an input file: -.
[h264 @ 0x21d93c0] Before avformat_find_stream_info() pos: 0 bytes read:4096 seeks:0
[h264 @ 0x21d93c0] Probe buffer size limit of 32000 bytes reached
[h264 @ 0x21d93c0] Stream #0: not enough frames to estimate rate; consider increasing probesize
[h264 @ 0x21d93c0] decoding for stream 0 failed
[h264 @ 0x21d93c0] After avformat_find_stream_info() pos: 33792 bytes read:49152 seeks:0 frames:3
Input #0, h264, from 'pipe:':
  Duration: N/A, bitrate: N/A
    Stream #0:0, 3, 1/1200000: Video: h264 (High), yuv420p, 1280x720, 25 fps, 25 tbr, 1200k tbn, 50 tbc
Successfully opened the file.
Parsing a group of options: input file hw:1.
Applying option thread_queue_size (set the maximum number of queued packets from the demuxer) with argument 512.
Applying option f (force format) with argument alsa.
Applying option ac (set number of audio channels) with argument 1.
Applying option channel_layout (set channel layout) with argument mono.
Successfully parsed a group of options.
Opening an input file: hw:1.
[alsa @ 0x21ffd00] All info found
Input #1, alsa, from 'hw:1':
  Duration: N/A, start: 1434444921.389533, bitrate: 768 kb/s
    Stream #1:0, 1, 1/1000000: Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s
Successfully opened the file.
Parsing a group of options: output file udp://44.0.0.95:1234.
Applying option vcodec (force video codec ('copy' to copy stream)) with argument copy.
Applying option acodec (force audio codec ('copy' to copy stream)) with argument mp3.
Applying option f (force format) with argument mpegts.
Successfully parsed a group of options.
Opening an output file: udp://44.0.0.95:1234.
Matched encoder 'libmp3lame' for codec 'mp3'.
Successfully opened the file.
detected 4 logical cores
[graph 0 input from stream 1:0 @ 0x21d1700] Setting 'time_base' to value '1/48000'
[graph 0 input from stream 1:0 @ 0x21d1700] Setting 'sample_rate' to value '48000'
[graph 0 input from stream 1:0 @ 0x21d1700] Setting 'sample_fmt' to value 's16'
[graph 0 input from stream 1:0 @ 0x21d1700] Setting 'channel_layout' to value '0x4'
[graph 0 input from stream 1:0 @ 0x21d1700] tb:1/48000 samplefmt:s16 samplerate:48000 chlayout:0x4
[audio format for output stream 0:1 @ 0x21ca470] Setting 'sample_fmts' to value 's32p|fltp|s16p'
[audio format for output stream 0:1 @ 0x21ca470] Setting 'sample_rates' to value '44100|48000|32000|22050|24000|16000|11025|12000|8000'
[audio format for output stream 0:1 @ 0x21ca470] Setting 'channel_layouts' to value '0x4|0x3'
[audio format for output stream 0:1 @ 0x21ca470] auto-inserting filter 'auto-inserted resampler 0' between the filter 'Parsed_anull_0' and the filter 'audio format for output stream 0:1'
[AVFilterGraph @ 0x21c9ea0] query_formats: 4 queried, 6 merged, 3 already done, 0 delayed
[auto-inserted resampler 0 @ 0x22436b0] ch:1 chl:mono fmt:s16 r:48000Hz -> ch:1 chl:mono fmt:s16p r:48000Hz
[mpegts @ 0x21eebf0] muxrate VBR, pcr every 2 pkts, sdt every 200, pat/pmt every 40 pkts
Output #0, mpegts, to 'udp://44.0.0.95:1234':
  Metadata:
    encoder         : Lavf56.36.100
    Stream #0:0, 0, 1/90000: Video: h264, yuv420p, 1280x720, q=2-31, 25 fps, 25 tbr, 90k tbn, 25 tbc
    Stream #0:1, 0, 1/90000: Audio: mp3 (libmp3lame), 48000 Hz, mono, s16p
    Metadata:
      encoder         : Lavc56.41.100 libmp3lame
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #1:0 -> #0:1 (pcm_s16le (native) -> mp3 (libmp3lame))
frame=   12 fps=0.0 q=-1.0 size=     523kB time=00:00:00.48 bitrate=8917.5kbits/s    
frame=   24 fps= 24 q=-1.0 size=    1043kB time=00:00:00.96 bitrate=8897.1kbits/s    
frame=   36 fps= 24 q=-1.0 size=    1511kB time=00:00:01.44 bitrate=8593.7kbits/s    
frame=   48 fps= 24 q=-1.0 size=    1980kB time=00:00:01.92 bitrate=8447.5kbits/s    
frame=   61 fps= 24 q=-1.0 size=    2463kB time=00:00:02.44 bitrate=8270.8kbits/s    
frame=   74 fps= 25 q=-1.0 size=    3007kB time=00:00:02.96 bitrate=8322.8kbits/s    
frame=   87 fps= 25 q=-1.0 size=    3475kB time=00:00:03.48 bitrate=8179.1kbits/s    
frame=   99 fps= 25 q=-1.0 size=    3980kB time=00:00:03.96 bitrate=8234.4kbits/s    
frame=  111 fps= 24 q=-1.0 size=    4498kB time=00:00:04.44 bitrate=8298.4kbits/s    
frame=  125 fps= 25 q=-1.0 size=    5160kB time=00:00:05.00 bitrate=8453.4kbits/s    
frame=  137 fps= 25 q=-1.0 size=    5652kB time=00:00:05.48 bitrate=8448.5kbits/s    
frame=  150 fps= 25 q=-1.0 size=    6174kB time=00:00:06.00 bitrate=8429.9kbits/s    
frame=  162 fps= 25 q=-1.0 size=    6638kB time=00:00:06.48 bitrate=8391.5kbits/s    
frame=  175 fps= 25 q=-1.0 size=    7141kB time=00:00:07.00 bitrate=8356.4kbits/s    
frame=  188 fps= 25 q=-1.0 size=    7705kB time=00:00:07.52 bitrate=8393.4kbits/s    
frame=  200 fps= 25 q=-1.0 size=    8161kB time=00:00:08.00 bitrate=8356.4kbits/s    
frame=  213 fps= 25 q=-1.0 size=    8662kB time=00:00:08.52 bitrate=8328.5kbits/s    
frame=  225 fps= 25 q=-1.0 size=    9125kB time=00:00:09.00 bitrate=8305.8kbits/s    
frame=  238 fps= 25 q=-1.0 size=    9615kB time=00:00:09.52 bitrate=8274.1kbits/s    
frame=  251 fps= 25 q=-1.0 size=   10178kB time=00:00:10.04 bitrate=8305.0kbits/s    
frame=  263 fps= 25 q=-1.0 size=   10647kB time=00:00:10.52 bitrate=8290.7kbits/s    
frame=  276 fps= 25 q=-1.0 size=   11220kB time=00:00:11.04 bitrate=8325.4kbits/s    
frame=  288 fps= 25 q=-1.0 size=   11756kB time=00:00:11.52 bitrate=8360.0kbits/s    
frame=  302 fps= 25 q=-1.0 size=   12420kB time=00:00:12.08 bitrate=8422.9kbits/s    
frame=  314 fps= 25 q=-1.0 size=   12907kB time=00:00:12.56 bitrate=8418.3kbits/s    
frame=  326 fps= 25 q=-1.0 size=   13428kB time=00:00:13.04 bitrate=8435.5kbits/s    
frame=  339 fps= 25 q=-1.0 size=   13892kB time=00:00:13.56 bitrate=8392.3kbits/s    
frame=  352 fps= 25 q=-1.0 size=   14414kB time=00:00:14.08 bitrate=8386.1kbits/s    
frame=  364 fps= 25 q=-1.0 size=   14964kB time=00:00:14.56 bitrate=8419.2kbits/s    
frame=  377 fps= 25 q=-1.0 size=   15481kB time=00:00:15.08 bitrate=8409.9kbits/s    
frame=  389 fps= 25 q=-1.0 size=   15959kB time=00:00:15.56 bitrate=8402.2kbits/s    
frame=  402 fps= 25 q=-1.0 size=   16496kB time=00:00:16.08 bitrate=8403.9kbits/s    
frame=  414 fps= 25 q=-1.0 size=   16987kB time=00:00:16.56 bitrate=8403.2kbits/s    
frame=  426 fps= 25 q=-1.0 size=   17582kB time=00:00:17.04 bitrate=8451.8kbits/s    
frame=  440 fps= 25 q=-1.0 size=   18088kB time=00:00:17.60 bitrate=8419.2kbits/s    
frame=  452 fps= 25 q=-1.0 size=   18549kB time=00:00:18.08 bitrate=8404.7kbits/s    
frame=  465 fps= 25 q=-1.0 size=   19047kB time=00:00:18.60 bitrate=8389.0kbits/s    
frame=  477 fps= 25 q=-1.0 size=   19495kB time=00:00:19.08 bitrate=8370.3kbits/s    
frame=  490 fps= 25 q=-1.0 size=   20063kB time=00:00:19.60 bitrate=8385.4kbits/s    
frame=  503 fps= 25 q=-1.0 size=   20566kB time=00:00:20.12 bitrate=8373.7kbits/s    
frame=  515 fps= 25 q=-1.0 size=   21034kB time=00:00:20.60 bitrate=8364.4kbits/s    
frame=  528 fps= 25 q=-1.0 size=   21536kB time=00:00:21.12 bitrate=8353.4kbits/s    
frame=  541 fps= 25 q=-1.0 size=   22039kB time=00:00:21.64 bitrate=8342.9kbits/s    
frame=  553 fps= 25 q=-1.0 size=   22568kB time=00:00:22.12 bitrate=8357.9kbits/s    
frame=  566 fps= 25 q=-1.0 size=   23071kB time=00:00:22.64 bitrate=8347.9kbits/s    
frame=  578 fps= 25 q=-1.0 size=   23534kB time=00:00:23.12 bitrate=8338.5kbits/s    
frame=  591 fps= 25 q=-1.0 size=   24034kB time=00:00:23.64 bitrate=8328.4kbits/s    
frame=  604 fps= 25 q=-1.0 size=   24614kB time=00:00:24.16 bitrate=8345.8kbits/s    
frame=  616 fps= 25 q=-1.0 size=   25112kB time=00:00:24.64 bitrate=8349.0kbits/s    
frame=  629 fps= 25 q=-1.0 size=   25691kB time=00:00:25.16 bitrate=8364.9kbits/s    
frame=  642 fps= 25 q=-1.0 size=   26271kB time=00:00:25.68 bitrate=8380.6kbits/s    
frame=  654 fps= 25 q=-1.0 size=   26825kB time=00:00:26.16 bitrate=8400.3kbits/s    
frame=  667 fps= 25 q=-1.0 size=   27483kB time=00:00:26.68 bitrate=8438.5kbits/s    
frame=  679 fps= 25 q=-1.0 size=   28040kB time=00:00:27.16 bitrate=8457.6kbits/s    
frame=  692 fps= 25 q=-1.0 size=   28510kB time=00:00:27.68 bitrate=8437.7kbits/s    
frame=  704 fps= 25 q=-1.0 size=   29061kB time=00:00:28.16 bitrate=8454.2kbits/s    
frame=  717 fps= 25 q=-1.0 size=   29675kB time=00:00:28.68 bitrate=8476.3kbits/s    
frame=  729 fps= 25 q=-1.0 size=   30207kB time=00:00:29.16 bitrate=8486.0kbits/s    
frame=  742 fps= 25 q=-1.0 size=   30750kB time=00:00:29.68 bitrate=8487.4kbits/s    
frame=  755 fps= 25 q=-1.0 size=   31316kB time=00:00:30.20 bitrate=8494.7kbits/s    
frame=  768 fps= 25 q=-1.0 size=   31908kB time=00:00:30.72 bitrate=8508.7kbits/s    
frame=  780 fps= 25 q=-1.0 size=   32431kB time=00:00:31.20 bitrate=8515.2kbits/s    
frame=  792 fps= 25 q=-1.0 size=   32954kB time=00:00:31.68 bitrate=8521.4kbits/s    
frame=  805 fps= 25 q=-1.0 size=   33463kB time=00:00:32.20 bitrate=8513.4kbits/s    
frame=  818 fps= 25 q=-1.0 size=   33966kB time=00:00:32.72 bitrate=8504.0kbits/s    
frame=  830 fps= 25 q=-1.0 size=   34476kB time=00:00:33.20 bitrate=8506.9kbits/s    
frame=  843 fps= 25 q=-1.0 size=   35083kB time=00:00:33.72 bitrate=8523.2kbits/s    
frame=  855 fps= 25 q=-1.0 size=   35522kB time=00:00:34.20 bitrate=8508.7kbits/s    
frame=  868 fps= 25 q=-1.0 size=   36012kB time=00:00:34.72 bitrate=8497.0kbits/s    
frame=  881 fps= 25 q=-1.0 size=   36581kB time=00:00:35.24 bitrate=8503.7kbits/s    
frame=  894 fps= 25 q=-1.0 size=   37109kB time=00:00:35.76 bitrate=8501.0kbits/s    
frame=  906 fps= 25 q=-1.0 size=   37637kB time=00:00:36.24 bitrate=8507.8kbits/s    
frame=  918 fps= 25 q=-1.0 size=   38174kB time=00:00:36.72 bitrate=8516.5kbits/s    
frame=  932 fps= 25 q=-1.0 size=   38819kB time=00:00:37.28 bitrate=8530.2kbits/s    
frame=  944 fps= 25 q=-1.0 size=   39395kB time=00:00:37.76 bitrate=8546.8kbits/s    
frame=  956 fps= 25 q=-1.0 size=   40028kB time=00:00:38.24 bitrate=8575.0kbits/s    
frame=  969 fps= 25 q=-1.0 size=   40582kB time=00:00:38.76 bitrate=8577.0kbits/s    
frame=  981 fps= 25 q=-1.0 size=   41052kB time=00:00:39.24 bitrate=8570.3kbits/s    
frame=  994 fps= 25 q=-1.0 size=   41559kB time=00:00:39.76 bitrate=8562.6kbits/s    
frame= 1007 fps= 25 q=-1.0 size=   42043kB time=00:00:40.28 bitrate=8550.5kbits/s    
frame= 1019 fps= 25 q=-1.0 size=   42525kB time=00:00:40.76 bitrate=8546.7kbits/s    
frame= 1032 fps= 25 q=-1.0 size=   43030kB time=00:00:41.28 bitrate=8539.3kbits/s    
frame= 1044 fps= 25 q=-1.0 size=   43489kB time=00:00:41.76 bitrate=8531.2kbits/s    
frame= 1058 fps= 25 q=-1.0 size=   44014kB time=00:00:42.32 bitrate=8519.8kbits/s    
frame= 1070 fps= 25 q=-1.0 size=   44449kB time=00:00:42.80 bitrate=8507.7kbits/s    
frame= 1082 fps= 25 q=-1.0 size=   44946kB time=00:00:43.28 bitrate=8507.3kbits/s    
frame= 1095 fps= 25 q=-1.0 size=   45433kB time=00:00:43.80 bitrate=8497.4kbits/s    
frame= 1107 fps= 25 q=-1.0 size=   45875kB time=00:00:44.28 bitrate=8487.0kbits/s    
frame= 1120 fps= 25 q=-1.0 size=   46374kB time=00:00:44.80 bitrate=8479.9kbits/s    
frame= 1133 fps= 25 q=-1.0 size=   46937kB time=00:00:45.32 bitrate=8484.4kbits/s    
frame= 1146 fps= 25 q=-1.0 size=   47622kB time=00:00:45.84 bitrate=8510.5kbits/s    
frame= 1158 fps= 25 q=-1.0 size=   48227kB time=00:00:46.32 bitrate=8529.2kbits/s    
frame= 1171 fps= 25 q=-1.0 size=   48815kB time=00:00:46.84 bitrate=8537.4kbits/s    
frame= 1184 fps= 25 q=-1.0 size=   49364kB time=00:00:47.36 bitrate=8538.6kbits/s    
frame= 1196 fps= 25 q=-1.0 size=   49867kB time=00:00:47.84 bitrate=8539.2kbits/s    
frame= 1209 fps= 25 q=-1.0 size=   50440kB time=00:00:48.36 bitrate=8544.4kbits/s    
frame= 1221 fps= 25 q=-1.0 size=   50899kB time=00:00:48.84 bitrate=8537.4kbits/s    
frame= 1234 fps= 25 q=-1.0 size=   51411kB time=00:00:49.36 bitrate=8532.4kbits/s    
frame= 1247 fps= 25 q=-1.0 size=   52025kB time=00:00:49.88 bitrate=8544.2kbits/s    
frame= 1259 fps= 25 q=-1.0 size=   52632kB time=00:00:50.36 bitrate=8561.6kbits/s    
frame= 1272 fps= 25 q=-1.0 size=   53295kB time=00:00:50.88 bitrate=8580.8kbits/s    
frame= 1284 fps= 25 q=-1.0 size=   53898kB time=00:00:51.36 bitrate=8596.8kbits/s    
frame= 1297 fps= 25 q=-1.0 size=   54539kB time=00:00:51.88 bitrate=8611.8kbits/s    
frame= 1310 fps= 25 q=-1.0 size=   55109kB time=00:00:52.40 bitrate=8615.5kbits/s    
frame= 1322 fps= 25 q=-1.0 size=   55692kB time=00:00:52.88 bitrate=8627.6kbits/s    
frame= 1335 fps= 25 q=-1.0 size=   56332kB time=00:00:53.40 bitrate=8641.8kbits/s    
frame= 1347 fps= 25 q=-1.0 size=   56910kB time=00:00:53.88 bitrate=8652.7kbits/s    
frame= 1360 fps= 25 q=-1.0 size=   57643kB time=00:00:54.40 bitrate=8680.3kbits/s    
frame= 1373 fps= 25 q=-1.0 size=   58272kB time=00:00:54.92 bitrate=8692.0kbits/s    
frame= 1385 fps= 25 q=-1.0 size=   58926kB time=00:00:55.40 bitrate=8713.5kbits/s    
frame= 1398 fps= 25 q=-1.0 size=   59559kB time=00:00:55.92 bitrate=8725.1kbits/s    
frame= 1410 fps= 25 q=-1.0 size=   60280kB time=00:00:56.40 bitrate=8755.1kbits/s    
frame= 1424 fps= 25 q=-1.0 size=   60941kB time=00:00:56.96 bitrate=8764.6kbits/s    
frame= 1436 fps= 25 q=-1.0 size=   61538kB time=00:00:57.44 bitrate=8776.5kbits/s    
frame= 1449 fps= 25 q=-1.0 size=   62222kB time=00:00:57.96 bitrate=8794.4kbits/s    
frame= 1461 fps= 25 q=-1.0 size=   62795kB time=00:00:58.44 bitrate=8802.4kbits/s    
frame= 1474 fps= 25 q=-1.0 size=   63414kB time=00:00:58.96 bitrate=8810.8kbits/s    
frame= 1487 fps= 25 q=-1.0 size=   64041kB time=00:00:59.48 bitrate=8820.2kbits/s    
frame= 1499 fps= 25 q=-1.0 size=   64609kB time=00:00:59.96 bitrate=8827.2kbits/s    
frame= 1511 fps= 25 q=-1.0 size=   65247kB time=00:01:00.44 bitrate=8843.5kbits/s    
frame= 1524 fps= 25 q=-1.0 size=   65893kB time=00:01:00.96 bitrate=8855.0kbits/s    
frame= 1537 fps= 25 q=-1.0 size=   66513kB time=00:01:01.48 bitrate=8862.6kbits/s    
frame= 1549 fps= 25 q=-1.0 size=   67085kB time=00:01:01.96 bitrate=8869.6kbits/s    
frame= 1562 fps= 25 q=-1.0 size=   67788kB time=00:01:02.48 bitrate=8888.0kbits/s    
frame= 1574 fps= 25 q=-1.0 size=   68438kB time=00:01:02.96 bitrate=8904.7kbits/s    
frame= 1587 fps= 25 q=-1.0 size=   69105kB time=00:01:03.48 bitrate=8917.8kbits/s    
frame= 1600 fps= 25 q=-1.0 size=   69682kB time=00:01:04.00 bitrate=8919.3kbits/s    
frame= 1612 fps= 25 q=-1.0 size=   70193kB time=00:01:04.48 bitrate=8917.8kbits/s    
frame= 1625 fps= 25 q=-1.0 size=   70806kB time=00:01:05.00 bitrate=8923.7kbits/s    
frame= 1638 fps= 25 q=-1.0 size=   71347kB time=00:01:05.52 bitrate=8920.5kbits/s    
frame= 1650 fps= 25 q=-1.0 size=   71835kB time=00:01:06.00 bitrate=8916.2kbits/s    
frame= 1662 fps= 25 q=-1.0 size=   72365kB time=00:01:06.48 bitrate=8916.8kbits/s    
frame= 1676 fps= 25 q=-1.0 size=   72921kB time=00:01:07.04 bitrate=8910.7kbits/s    
frame= 1688 fps= 25 q=-1.0 size=   73501kB time=00:01:07.52 bitrate=8917.7kbits/s    
frame= 1701 fps= 25 q=-1.0 size=   74101kB time=00:01:08.04 bitrate=8921.8kbits/s    
frame= 1713 fps= 25 q=-1.0 size=   74683kB time=00:01:08.52 bitrate=8928.8kbits/s    
frame= 1726 fps= 25 q=-1.0 size=   75252kB time=00:01:09.04 bitrate=8929.1kbits/s    
frame= 1738 fps= 25 q=-1.0 size=   75759kB time=00:01:09.52 bitrate=8927.2kbits/s    
frame= 1751 fps= 25 q=-1.0 size=   76361kB time=00:01:10.04 bitrate=8931.3kbits/s    
frame= 1763 fps= 25 q=-1.0 size=   76852kB time=00:01:10.52 bitrate=8927.6kbits/s    
frame= 1776 fps= 25 q=-1.0 size=   77378kB time=00:01:11.04 bitrate=8922.9kbits/s    
frame= 1788 fps= 25 q=-1.0 size=   77858kB time=00:01:11.52 bitrate=8918.0kbits/s    
frame= 1802 fps= 25 q=-1.0 size=   78508kB time=00:01:12.08 bitrate=8922.6kbits/s    
frame= 1814 fps= 25 q=-1.0 size=   79012kB time=00:01:12.56 bitrate=8920.4kbits/s    
frame= 1827 fps= 25 q=-1.0 size=   79541kB time=00:01:13.08 bitrate=8916.3kbits/s    
frame= 1839 fps= 25 q=-1.0 size=   80022kB time=00:01:13.56 bitrate=8911.6kbits/s    
frame= 1851 fps= 25 q=-1.0 size=   80521kB time=00:01:14.04 bitrate=8909.1kbits/s    
frame= 1864 fps= 25 q=-1.0 size=   81113kB time=00:01:14.56 bitrate=8911.9kbits/s    
frame= 1877 fps= 25 q=-1.0 size=   81635kB time=00:01:15.08 bitrate=8907.2kbits/s    
frame= 1889 fps= 25 q=-1.0 size=   82101kB time=00:01:15.56 bitrate=8901.1kbits/s    
frame= 1902 fps= 25 q=-1.0 size=   82629kB time=00:01:16.08 bitrate=8897.2kbits/s    
frame= 1914 fps= 25 q=-1.0 size=   83106kB time=00:01:16.56 bitrate=8892.4kbits/s    
frame= 1927 fps= 25 q=-1.0 size=   83777kB time=00:01:17.08 bitrate=8903.8kbits/s    
frame= 1940 fps= 25 q=-1.0 size=   84288kB time=00:01:17.60 bitrate=8898.0kbits/s    
frame= 1952 fps= 25 q=-1.0 size=   84801kB time=00:01:18.08 bitrate=8897.1kbits/s    
frame= 1965 fps= 25 q=-1.0 size=   85332kB time=00:01:18.60 bitrate=8893.7kbits/s    
frame= 1977 fps= 25 q=-1.0 size=   85843kB time=00:01:19.08 bitrate=8892.3kbits/s    
frame= 1990 fps= 25 q=-1.0 size=   86367kB time=00:01:19.60 bitrate=8888.5kbits/s    
frame= 2003 fps= 25 q=-1.0 size=   86871kB time=00:01:20.12 bitrate=8882.3kbits/s    
frame= 2015 fps= 25 q=-1.0 size=   87335kB time=00:01:20.60 bitrate=8876.5kbits/s    
frame= 2028 fps= 25 q=-1.0 size=   87909kB time=00:01:21.12 bitrate=8877.6kbits/s    
frame= 2040 fps= 25 q=-1.0 size=   88428kB time=00:01:21.60 bitrate=8877.4kbits/s    
frame= 2053 fps= 25 q=-1.0 size=   89073kB time=00:01:22.12 bitrate=8885.6kbits/s    
frame= 2066 fps= 25 q=-1.0 size=   89553kB time=00:01:22.64 bitrate=8877.3kbits/s    
frame= 2078 fps= 25 q=-1.0 size=   90024kB time=00:01:23.12 bitrate=8872.5kbits/s    
frame= 2091 fps= 25 q=-1.0 size=   90535kB time=00:01:23.64 bitrate=8867.3kbits/s    
frame= 2103 fps= 25 q=-1.0 size=   91085kB time=00:01:24.12 bitrate=8870.3kbits/s    
frame= 2116 fps= 25 q=-1.0 size=   91625kB time=00:01:24.64 bitrate=8868.0kbits/s    
frame= 2129 fps= 25 q=-1.0 size=   92167kB time=00:01:25.16 bitrate=8866.1kbits/s    
frame= 2142 fps= 25 q=-1.0 size=   92718kB time=00:01:25.68 bitrate=8864.9kbits/s    
frame= 2154 fps= 25 q=-1.0 size=   93219kB time=00:01:26.16 bitrate=8863.2kbits/s    
frame= 2166 fps= 25 q=-1.0 size=   93802kB time=00:01:26.64 bitrate=8868.9kbits/s    
frame= 2179 fps= 25 q=-1.0 size=   94275kB time=00:01:27.16 bitrate=8860.8kbits/s    
frame= 2192 fps= 25 q=-1.0 size=   94806kB time=00:01:27.68 bitrate=8857.8kbits/s    
frame= 2204 fps= 25 q=-1.0 size=   95268kB time=00:01:28.16 bitrate=8852.5kbits/s    
frame= 2217 fps= 25 q=-1.0 size=   95790kB time=00:01:28.68 bitrate=8848.8kbits/s    
frame= 2229 fps= 25 q=-1.0 size=   96343kB time=00:01:29.16 bitrate=8852.0kbits/s    
frame= 2242 fps= 25 q=-1.0 size=   96883kB time=00:01:29.68 bitrate=8850.0kbits/s    
frame= 2255 fps= 25 q=-1.0 size=   97404kB time=00:01:30.20 bitrate=8846.3kbits/s    
frame= 2267 fps= 25 q=-1.0 size=   97868kB time=00:01:30.68 bitrate=8841.4kbits/s    
frame= 2280 fps= 25 q=-1.0 size=   98415kB time=00:01:31.20 bitrate=8840.1kbits/s    
frame= 2292 fps= 25 q=-1.0 size=   98951kB time=00:01:31.68 bitrate=8841.7kbits/s    
frame= 2306 fps= 25 q=-1.0 size=   99512kB time=00:01:32.24 bitrate=8837.9kbits/s    
frame= 2318 fps= 25 q=-1.0 size=  100005kB time=00:01:32.72 bitrate=8835.6kbits/s    
frame= 2330 fps= 25 q=-1.0 size=  100501kB time=00:01:33.20 bitrate=8833.8kbits/s    
frame= 2343 fps= 25 q=-1.0 size=  101089kB time=00:01:33.72 bitrate=8836.1kbits/s    
frame= 2355 fps= 25 q=-1.0 size=  101597kB time=00:01:34.20 bitrate=8835.2kbits/s    
frame= 2368 fps= 25 q=-1.0 size=  102139kB time=00:01:34.72 bitrate=8833.7kbits/s    
frame= 2381 fps= 25 q=-1.0 size=  102659kB time=00:01:35.24 bitrate=8830.1kbits/s    
frame= 2393 fps= 25 q=-1.0 size=  103116kB time=00:01:35.72 bitrate=8825.0kbits/s    
frame= 2406 fps= 25 q=-1.0 size=  103672kB time=00:01:36.24 bitrate=8824.6kbits/s    
frame= 2418 fps= 25 q=-1.0 size=  104180kB time=00:01:36.72 bitrate=8823.8kbits/s    
frame= 2431 fps= 25 q=-1.0 size=  104721kB time=00:01:37.24 bitrate=8822.3kbits/s    
frame= 2444 fps= 25 q=-1.0 size=  105328kB time=00:01:37.76 bitrate=8826.1kbits/s    
frame= 2456 fps= 25 q=-1.0 size=  105892kB time=00:01:38.24 bitrate=8830.1kbits/s    
frame= 2469 fps= 25 q=-1.0 size=  106535kB time=00:01:38.76 bitrate=8837.0kbits/s    
frame= 2481 fps= 25 q=-1.0 size=  107052kB time=00:01:39.24 bitrate=8836.9kbits/s    
frame= 2494 fps= 25 q=-1.0 size=  107698kB time=00:01:39.76 bitrate=8843.8kbits/s    
frame= 2507 fps= 25 q=-1.0 size=  108303kB time=00:01:40.28 bitrate=8847.4kbits/s    
frame= 2520 fps= 25 q=-1.0 size=  108891kB time=00:01:40.80 bitrate=8849.5kbits/s    
frame= 2532 fps= 25 q=-1.0 size=  109461kB time=00:01:41.28 bitrate=8853.7kbits/s    
frame= 2544 fps= 25 q=-1.0 size=  109953kB time=00:01:41.76 bitrate=8851.5kbits/s    
frame= 2557 fps= 25 q=-1.0 size=  110502kB time=00:01:42.28 bitrate=8850.5kbits/s    
frame= 2570 fps= 25 q=-1.0 size=  111025kB time=00:01:42.80 bitrate=8847.5kbits/s    
frame= 2582 fps= 25 q=-1.0 size=  111556kB time=00:01:43.28 bitrate=8848.4kbits/s    
frame= 2595 fps= 25 q=-1.0 size=  112064kB time=00:01:43.80 bitrate=8844.2kbits/s    
frame= 2607 fps= 25 q=-1.0 size=  112562kB time=00:01:44.28 bitrate=8842.6kbits/s    
frame= 2620 fps= 25 q=-1.0 size=  113219kB time=00:01:44.80 bitrate=8850.1kbits/s    
frame= 2633 fps= 25 q=-1.0 size=  113733kB time=00:01:45.32 bitrate=8846.4kbits/s    
frame= 2646 fps= 25 q=-1.0 size=  114344kB time=00:01:45.84 bitrate=8850.2kbits/s    
frame= 2658 fps= 25 q=-1.0 size=  114875kB time=00:01:46.32 bitrate=8851.2kbits/s    
frame= 2671 fps= 25 q=-1.0 size=  115402kB time=00:01:46.84 bitrate=8848.5kbits/s    
frame= 2684 fps= 25 q=-1.0 size=  115953kB time=00:01:47.36 bitrate=8847.7kbits/s    
frame= 2696 fps= 25 q=-1.0 size=  116469kB time=00:01:47.84 bitrate=8847.5kbits/s    
frame= 2709 fps= 25 q=-1.0 size=  117056kB time=00:01:48.36 bitrate=8849.4kbits/s    
frame= 2721 fps= 25 q=-1.0 size=  117538kB time=00:01:48.84 bitrate=8846.7kbits/s    
frame= 2734 fps= 25 q=-1.0 size=  118040kB time=00:01:49.36 bitrate=8842.2kbits/s    
frame= 2747 fps= 25 q=-1.0 size=  118541kB time=00:01:49.88 bitrate=8837.7kbits/s    
frame= 2759 fps= 25 q=-1.0 size=  119000kB time=00:01:50.36 bitrate=8833.3kbits/s    
frame= 2771 fps= 25 q=-1.0 size=  119541kB time=00:01:50.84 bitrate=8835.1kbits/s    
frame= 2784 fps= 25 q=-1.0 size=  120028kB time=00:01:51.36 bitrate=8829.6kbits/s    
frame= 2797 fps= 25 q=-1.0 size=  120573kB time=00:01:51.88 bitrate=8828.5kbits/s    
frame= 2810 fps= 25 q=-1.0 size=  121101kB time=00:01:52.40 bitrate=8826.1kbits/s    
frame= 2822 fps= 25 q=-1.0 size=  121631kB time=00:01:52.88 bitrate=8827.1kbits/s    
frame= 2835 fps= 25 q=-1.0 size=  122186kB time=00:01:53.40 bitrate=8826.7kbits/s    
frame= 2847 fps= 25 q=-1.0 size=  122672kB time=00:01:53.88 bitrate=8824.5kbits/s    
frame= 2859 fps= 25 q=-1.0 size=  123136kB time=00:01:54.36 bitrate=8820.7kbits/s    
frame= 2872 fps= 25 q=-1.0 size=  123672kB time=00:01:54.88 bitrate=8819.0kbits/s    
frame= 2885 fps= 25 q=-1.0 size=  124259kB time=00:01:55.40 bitrate=8820.9kbits/s    
frame= 2898 fps= 25 q=-1.0 size=  124757kB time=00:01:55.92 bitrate=8816.5kbits/s    
frame= 2910 fps= 25 q=-1.0 size=  125218kB time=00:01:56.40 bitrate=8812.6kbits/s    
frame= 2923 fps= 25 q=-1.0 size=  125710kB time=00:01:56.92 bitrate=8807.9kbits/s    
frame= 2936 fps= 25 q=-1.0 size=  126203kB time=00:01:57.44 bitrate=8803.3kbits/s    
frame= 2948 fps= 25 q=-1.0 size=  126755kB time=00:01:57.92 bitrate=8805.8kbits/s    
frame= 2961 fps= 25 q=-1.0 size=  127283kB time=00:01:58.44 bitrate=8803.6kbits/s    
frame= 2973 fps= 25 q=-1.0 size=  127812kB time=00:01:58.92 bitrate=8804.6kbits/s    
frame= 2985 fps= 25 q=-1.0 size=  128380kB time=00:01:59.40 bitrate=8807.8kbits/s    
frame= 2999 fps= 25 q=-1.0 size=  128931kB time=00:01:59.96 bitrate=8804.6kbits/s    
frame= 3011 fps= 25 q=-1.0 size=  129475kB time=00:02:00.44 bitrate=8806.5kbits/s    
frame= 3024 fps= 25 q=-1.0 size=  130008kB time=00:02:00.96 bitrate=8804.8kbits/s    
frame= 3036 fps= 25 q=-1.0 size=  130526kB time=00:02:01.44 bitrate=8804.9kbits/s    
frame= 3049 fps= 25 q=-1.0 size=  131065kB time=00:02:01.96 bitrate=8803.6kbits/s    
frame= 3062 fps= 25 q=-1.0 size=  131654kB time=00:02:02.48 bitrate=8805.6kbits/s    
frame= 3074 fps= 25 q=-1.0 size=  132123kB time=00:02:02.96 bitrate=8802.5kbits/s    
frame= 3087 fps= 25 q=-1.0 size=  132630kB time=00:02:03.48 bitrate=8799.0kbits/s    
frame= 3099 fps= 25 q=-1.0 size=  133106kB time=00:02:03.96 bitrate=8796.4kbits/s    
frame= 3112 fps= 25 q=-1.0 size=  133610kB time=00:02:04.48 bitrate=8792.9kbits/s    
frame= 3125 fps= 25 q=-1.0 size=  134172kB time=00:02:05.00 bitrate=8793.1kbits/s    
frame= 3137 fps= 25 q=-1.0 size=  134617kB time=00:02:05.48 bitrate=8788.5kbits/s    
frame= 3150 fps= 25 q=-1.0 size=  135107kB time=00:02:06.00 bitrate=8784.1kbits/s    
frame= 3162 fps= 25 q=-1.0 size=  135579kB time=00:02:06.48 bitrate=8781.3kbits/s    
frame= 3175 fps= 25 q=-1.0 size=  136134kB time=00:02:07.00 bitrate=8781.2kbits/s    
frame= 3188 fps= 25 q=-1.0 size=  136743kB time=00:02:07.52 bitrate=8784.5kbits/s    
[libmp3lame @ 0x2253710] Trying to remove 47 more samples than there are in the queue
frame= 3188 fps= 25 q=-1.0 Lsize=  136789kB time=00:02:07.61 bitrate=8781.1kbits/s    
video:125507kB audio:997kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 8.130203%
Input file #0 (pipe:):
  Input stream #0:0 (video): 3188 packets read (128518751 bytes); 
  Total: 3188 packets (128518751 bytes) demuxed
Input file #1 (hw:1):
  Input stream #1:0 (audio): 103108 packets read (12251124 bytes); 103108 frames decoded (6125562 samples); 
  Total: 103108 packets (12251124 bytes) demuxed
Output file #0 (udp://44.0.0.95:1234):
  Output stream #0:0 (video): 3188 packets muxed (128518751 bytes); 
  Output stream #0:1 (audio): 5317 frames encoded (6125184 samples); 5318 packets muxed (1021056 bytes); 
  Total: 8506 packets (129539807 bytes) muxed
103108 frames successfully decoded, 0 decoding errors
[AVIOContext @ 0x21f07a0] Statistics: 0 seeks, 97026 writeouts
[AVIOContext @ 0x21d8e40] Statistics: 130988342 bytes read, 0 seeks
Received signal 2: terminating.

[6by9]

Test with audio and video separately to see what the latencies are then.
You don't say what your receiving software is - many PC based systems will deliberately have large input buffers to cope with network variability.

I suspect your A/V sync issues are due to raspivid already being spun up and producing frames before the audio side starts at all. You have provided a fixed frame rate of 25fps to both apps, so there will be a backlog of a few frames effectively stuck in the output FIFO that never get consumed. Don't ask me how to get around that using raspivid and piping the output around. You might be better off using the V4L2 driver and getting ffmpeg talking to that.

[neik]

Thanks for the useful comments. I've tried VLC and ffplay on the receiving end.

Video alone has a lag of about 2 seconds. I'll test audio and I'll try the v4l driver.

[6by9]

Actual encode latency has been discussed a few times and is down around the 100ms level. http://www.gertbot.com/toaste.html gives a streaming solution for video only with overall latency <250ms. Most of the extra latency is due to using off the shelf apps to do the conversions (eg VLC or ffmpeg).

There was a mod to raspistill a while back that added an fflush call after each file was saved as otherwise the Linux file functions buffer things up before actually passing the data on (https://github.com/raspberrypi/userland/issues/133). I don't think this will be a real issue in this case as the above also pipes the output of raspivid around and it doesn't suffer issues. It might be worth a quick check, but don't expect large gains.

[RpiName]

I am trying to capture the video from the camera board and combine it with audio from a USB sound card. The problem is that the audio lags by over 2 seconds and the video is over 2 seconds behind that!

You may want to try UV4L for perfectly sync'ed, low latency (~200ms), Audio/Video streaming to any WebRTC compliant peer (e.g. browsers). No configuration is required, just install and connect to the Streaming Server. Since the UV4L back-end is also a V4L2-compliant driver, you can still optionally use ffmpeg if really need to:

Homepage:
http://www.linux-projects.org

Installation:
http://www.linux-projects.org/modules/s ... e&artid=14

Examples:
http://www.linux-projects.org/modules/s ... e&artid=16

[neik]

My goal is to stream audio and video from the pi, from the command line. I will then capture it on another system and write it to a file.

I've tried the V4L2 driver but get an error:

Code: Select all

ffmpeg -report -f v4l2 -probesize 100000 -thread_queue_size 1024 -y -f h264 -r 25 -framerate 25 -i /dev/video1 -vcodec copy -f mpegts udp://44.0.0.95:1234

Code: Select all

[h264 @ 0x1bfb7b0] non-existing PPS 3 referenced
[h264 @ 0x1bfb7b0] missing picture in access unit with size 208
[h264 @ 0x1bfb7b0] slice type 32 too large at 1
[h264 @ 0x1bfb7b0] decode_slice_header error
[h264 @ 0x1bfb7b0] data partitioning is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it m            eans that your file has a feature which has not been implemented.
[h264 @ 0x1bfb7b0] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/incoming/ and contact the ffmpeg-devel mailing li            st. (ffmpeg-devel@ffmpeg.org)
[h264 @ 0x1bfb7b0] A non-intra slice in an IDR NAL unit.
[h264 @ 0x1bfb7b0] decode_slice_header error
[h264 @ 0x1bfb7b0] no frame!

I tried UV4L, but don't really understand it's architecture. I installed it OK, but could not find an example that matches my requirements.

[towolf]

I expect this would work very well with GStreamer, rpicamsrc, and alsasrc. rpicamsrc uses the MMAL RTC clock timestamps. Muxing this with audio from ALSA should work well.

[neik]

I expect this would work very well with GStreamer, rpicamsrc, and alsasrc. rpicamsrc uses the MMAL RTC clock timestamps. Muxing this with audio from ALSA should work well.

Can you provide an example command line?

[neik]

I expect this would work very well with GStreamer, rpicamsrc, and alsasrc. rpicamsrc uses the MMAL RTC clock timestamps. Muxing this with audio from ALSA should work well.

It seems that I am expected to compile the rpicamsrc plugin from source. Whilst I can do that, I feel that most people won't be doing that, and therefore I am doing something unusual.

Am I really the only one that wants to stream audio and video easily? I really expected this to be more of a plug and play experience. I don't mind if things are hard, but I feel I must be going down the wrong path here.

[gjimenez]

I had the same problem and I resolved it compiling this modified version of raspivid:
https://github.com/d3faultdotxbe/raspiv ... s/raspicam

Regards,

[neik]

Thanks for this. But I feel I am going down a hacky rabbithole here!

[towolf]

I expect this would work very well with GStreamer, rpicamsrc, and alsasrc. rpicamsrc uses the MMAL RTC clock timestamps. Muxing this with audio from ALSA should work well.

It seems that I am expected to compile the rpicamsrc plugin from source. Whilst I can do that, I feel that most people won't be doing that, and therefore I am doing something unusual.

I would ask the Raspbian developers why rpicamsrc is not found in the repository. It is very useful and developed specifically for the Raspberry Pi. It would only be one apt-get away.

Here’s one pipeline to mux audio and video into a file. For streaming this would have to be extended to replace the file muxing with netowrking code:

Code: Select all

gst-launch-1.0 alsasrc ! audioconvert ! avenc_aac ! aacparse ! qtmux name=mux ! filesink name=out.mp4 \
    rpicamsrc ! h264parse ! mux.

[towolf]

I would ask the Raspbian developers why rpicamsrc is not found in the repository. It is very useful and developed specifically for the Raspberry Pi. It would only be one apt-get away.

I mean, the project even has the appropriate Debian packaging scripts included: https://github.com/thaytan/gst-rpicamsr ... ter/debian

[kalegood]

Has anyone gotten the code from the above post to work?

Code: Select all

gst-launch-1.0 alsasrc ! audioconvert ! avenc_aac ! aacparse ! qtmux name=mux ! filesink name=out.mp4 \
    rpicamsrc ! h264parse ! mux.

I'm getting the following error:

Code: Select all

ERROR: Pipeling doesn't want to pause.
ERROR: from element /GstPipeline:pipeline0/GstFileSInk:out.mp4: No file name specified for writing.
Additional debug info:
gstfilesink.c(416): gst_file_sink_open_file (): /GstPipeline:pipeline0/GstFileSInk:out.mp4
Setting pipeline to NULL ...
Freeing pipeline ...

I'm new to gstreamer and that many pipes has me scratching my head.

[Lallala]

Is there a way to direct the preview of rpicamsrc into a window (eg Gtk) instead of using almost the full screen above all other windows?