jjl64
Posts: 7
Joined: Mon Dec 10, 2018 4:09 pm

Pi-to-Pi Low Latency One-To-Many Video Streaming

Mon Dec 10, 2018 5:47 pm

Hello,

I have been searching and experimenting for a few weeks how to set up what I had initially thought would be a simple application but have run into roadblocks at every turn so I'm now asking for others' assistance.

Here's what I wish to accomplish:

- I wish to stream video only from a Pi3B+ with Pi Camera to at least two, and possibly up to five, "client" viewers.

- Each of these viewers will be another Pi3B+ with HDMI TV screen.

- All of the computers will connect to the same wifi LAN.

- There is NO requirement to broadcast beyond the local wifi LAN, or upload to YouTube or anywhere else.

- There is NO requirement for recording or playback of the video stream.

- The lowest feasible latency is critically important.

The purpose of this application is to display the activity occurring onstage in a small theater to a handful of Pi's elsewhere in the building.

Simple, fast and cheap are the criteria for ultimate success.


What I have attempted thus far, with very mixed results:

- raspivid/ncat/hello_video: This works but the playback on the receiving Pi3B+ is so choppy as to be disconcerting to the people who watch it. Is there some tweak I should apply to correct that?

- raspivid/ncat/player (e.g. ffplay, mpv, vlc): No matter which player I run on the receiving Pi3B+, they either do not like raw H264 streams at all, or seem to introduce latency of ten seconds or more when decoding it. Perhaps I have not found the proper combination of codec, bit rate, frame rate, cache size, etc? This path seems simplest and most straightforward to me in theory, but in practice it is not working well and I strongly suspect that I just haven't stumbled upon the correct combination of parameters. Has anyone else found a combination that reduces latency to one second or less?

- raspivid/ncat/RPiViewer: This viewing app for OSX and Android works great over TCP provided all the receivers are iPhones or Android phones.

- uv4l: Its webRTC stream works great provided that you have exactly one viewer. If you want multiple viewers you need to buy a media server.

- uv4l: Its mjpeg and h264 streams have much too much latency to be useful.

- node server: I've tried one or two projects from github but they do not appear to handle multiple clients.


Theoretically speaking, it seems to me that this sort of application would be ideally suited to a local multicast group. Individual receivers may "tune in" to the stream as they please.

With respect to performance tuning of the client Pi, I have:

- compiled ffmpeg libraries from source on the Pi

- do not run any other applications on the Pi

- use a ramdisk for /tmp and /log (to avoid writing to the SD card, should some of these packages need to do so)


With respect to the wifi router, I considered that it might be a potential bottleneck but the webRTC and RPiViewer streams work perfectly, with no discernible latency, albeit for a single webRTC client or a pair of RPiViewer iPhones. Thus I conclude that my router is likely performing well enough.

RpiName
Posts: 712
Joined: Sat Jul 06, 2013 3:14 am

Re: Pi-to-Pi Low Latency One-To-Many Video Streaming

Mon Dec 10, 2018 8:30 pm

try uv4l + Janus for 1-to-many broadcasting

jjl64
Posts: 7
Joined: Mon Dec 10, 2018 4:09 pm

Re: Pi-to-Pi Low Latency One-To-Many Video Streaming

Tue Dec 11, 2018 6:57 pm

RpiName wrote:
Mon Dec 10, 2018 8:30 pm
try uv4l + Janus for 1-to-many broadcasting
For evaluation purposes and in the interests of time, I downloaded the UV4L/Janus videoconference bootable SD image from https://www.linux-projects.org/rpi-vide ... e-demo-os/ only to find that it appears to suffer from a firmware issue that prevents it from booting a 3B+, which is the only Pi I happen to have.

Ah, roadblocks!

Of course I reported the issue to them but in the meantime, might there be a more recent bootable image available somewhere else?

User avatar
mooblie
Posts: 128
Joined: Fri Oct 14, 2016 2:07 pm
Location: The Scottish Highlands

Re: Pi-to-Pi Low Latency One-To-Many Video Streaming

Tue Dec 11, 2018 7:07 pm

Have you investigated "Motion" surveillance software on a Pi? There are many tutorials - here's one:

https://www.instructables.com/id/How-to ... d-Stream-/

I used this to quickly setup a number of webcams, each serving webpages to anything on the network (including other Pis!). All worked rather well, and quite simple to setup. Webcams can also be read by any browser on the LAN, including people (actors) smartphones.

Simple installation via "apt-get install" is available ready-to-go - it just need configuring.

jjl64
Posts: 7
Joined: Mon Dec 10, 2018 4:09 pm

Re: Pi-to-Pi Low Latency One-To-Many Video Streaming

Wed Dec 12, 2018 1:51 am

mooblie wrote:
Tue Dec 11, 2018 7:07 pm
Have you investigated "Motion" surveillance software on a Pi? There are many tutorials - here's one:

https://www.instructables.com/id/How-to ... d-Stream-/

I used this to quickly setup a number of webcams, each serving webpages to anything on the network (including other Pis!). All worked rather well, and quite simple to setup. Webcams can also be read by any browser on the LAN, including people (actors) smartphones.

Simple installation via "apt-get install" is available ready-to-go - it just need configuring.
As a matter of fact, yes, I have.

I find that the abundance of tutorials, while they may have been valid at the time of their creation, are no longer valid today. Too much has changed in just a mere year.

The closest I have come is to get this tutorial sort of half-operating:

https://www.bouvet.no/bouvet-deler/utbr ... ry-pi-zero

in that the streaming video does not work, but still photos do. Hours or days of configuration twiddling may ensue. Who knows?


In the final analysis, it would be most helpful if someone were able to recommend a potential solution that actually works TODAY, as opposed to sometime last year, and actually solves the latency problem I have sought help with, because what I am learning here is that if and when I get this thing to work, I will never update the Pi afterwards. Ever!

jjl64
Posts: 7
Joined: Mon Dec 10, 2018 4:09 pm

Re: Pi-to-Pi Low Latency One-To-Many Video Streaming

Wed Dec 12, 2018 7:25 pm

jjl64 wrote:
Tue Dec 11, 2018 6:57 pm
RpiName wrote:
Mon Dec 10, 2018 8:30 pm
try uv4l + Janus for 1-to-many broadcasting
For evaluation purposes and in the interests of time, I downloaded the UV4L/Janus videoconference bootable SD image from https://www.linux-projects.org/rpi-vide ... e-demo-os/ only to find that it appears to suffer from a firmware issue that prevents it from booting a 3B+, which is the only Pi I happen to have.

Ah, roadblocks!

Of course I reported the issue to them but in the meantime, might there be a more recent bootable image available somewhere else?
For those who may come this way in the future, the bootable image referred to above contains Jessie, not Stretch, so it won't work on 3B+ or newer hardware. Their download page says the image was "updated February 22, 2018."

A less than helpful response from someone at the UV4L site essentially said, "tough luck."

noggin
Posts: 99
Joined: Sun Feb 21, 2016 1:55 pm

Re: Pi-to-Pi Low Latency One-To-Many Video Streaming

Fri Dec 14, 2018 9:57 am

I can't help with specific set-ups but I an offer some codec advice.

To run with the lowest latency possible you want to avoid anything with a long GOP (Group of Pictures). LongGOP compression is how H.264/MPEG2/VC-1 reduce their data rate or increase their picture quality - but in doing so they buffer a bunch of frames to detect redundancy within/between them. At the decoder a similar amount of buffering is required to reconstruct output frames. This is great for many applications, but for low latency encoding you don't want your encode/decode pipeline to buffer lots of frames.

LongGOP codecs are sometimes known as Inter-frame codecs. Codecs that encode every frame separately are known as Intra-frame codecs (as they only code within each frame) There ways of running codecs that support Inter-frame LongGOP compression (like MPEG2, H.264/AVC etc.) as Intra- codecs in what is known as I-frame only. I-frames are always coded 'stand alone' in LongGOP streams, so if you only run I-frames you should be running with the lowest latency possible. The Inter-frame LongGOP coding adds P (predictive), and possibly B (Bi-directional predictive) frames that increase quality for a given data rate, but at the expense of latency.

If you can find an MJPEG codec (MJPEG codes each frame separately) that is fast enough on the Pi, or an I-frame only H.264 encoder, that should give you minimal latency - but your bitrate will need to be pretty high. If your LAN is wired and only carrying this data that should still be doable. Very high-quality SD video runs at 50Mbs in I-Frame only MPEG2 (and 25Mbs gives good quality with DV25), and I suspect you'd get OK results with 50Mbs at 720p with H.264 I-frame only.

If you can cope with a tiny bit of latency then a 2-frame IP (or possibly IB) GOP may well let you reduce data rates a bit? I don't know how much flexibility you have with configuring the GOP structure of the internal H.264 encoder.

ALSO - some player solutions will want to buffer some incoming data before playing it to cope with network latency or throughput issues - which will add latency. If there are any configuration options in your player to reduce this - that might also help reduce latency.

(There are third party HDMI-over-IP solutions that use MJPEG with low-latency encoding that aren't hugely expensive - it may be worth looking at those - either fed from the output of a Pi with camera, or a cheap camcorder?)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7457
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Pi-to-Pi Low Latency One-To-Many Video Streaming

Fri Dec 14, 2018 12:34 pm

noggin wrote:
Fri Dec 14, 2018 9:57 am
I can't help with specific set-ups but I an offer some codec advice.
<snip>
Sorry, a fair amount of that is only partly true.

An H264 I frame is more efficiently coded than MJPEG. MJPEG really should be avoided unless you have a very good reason to use it (needing more than 1080P would be the only reason I can think of on the Pi).

H264 supports I(ntra-coded), P(redicted) and B(idirectional) frames.
With P frames the decoder will already have the reference frames required to decode the frame - it will have minimal effect on latency.
Yes, B-frames require all reference frames, which will normally include later frames in the stream and therefore increase latency.
The Pi encoder doesn't support B-frames, therefore it's a non-issue.

It is up to the decoder as to whether it fills the full DPB (Decoded Picture Buffer) that is allowed by the specified by the H264 level and profile, or does it only wait for the reference frames actually used to encode the frame.
On the Pi my recollection is that if you have provided the codec with framed data and correctly framed it, then it will check whether it has the necessary references and decode it as soon as it has the required frames.

Putting lots of B frames in a GOP (eg IBBBBBP) would increase latency as the decoder needs the P frame before it can decode any of the B's. Irrelevant on the Pi as it doesn't support B frames. The default GOP on the Pi is an I and 59 P's, which is an I frame every 2 seconds at the default 30fps. You can change that via raspivid's -g parameter.

To jjl64 (the OP), the main issue with raspivid, netcat, and hello_video is the buffering in the Linux kernel when you pipe data around. If you send directly to the network from raspivid (eg raspivid -o udp://224.1.2.3.4:1234), then adding the -fl flag means it calls fflush after every write to send the data over the network.
Piping the output of netcat to feed it into hello_video will have the same issue. There is a flag you can pass to switch to unbuffered operation (setvbuf with mode _IONBF), but IIRC there is an issue the setting will apply to the terminal after your application exits, therefore you want to put a signal handler in to restore the correct behaviour on quit. You still won't have framed data there, so there is likely to still be buffering within the codec unless you add additional bitstream parsing.
Worth also adding that multicast over Wifi can be rather hit and miss. If they're on the same Wifi network then generally it gets treated as broadcast (and that's fine). If you have a mix of Wifi and wired devices then the behaviour can depend on exactly how the AP and network switches behave.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jjl64
Posts: 7
Joined: Mon Dec 10, 2018 4:09 pm

Re: Pi-to-Pi Low Latency One-To-Many Video Streaming

Fri Dec 14, 2018 3:55 pm

6by9 wrote:
Fri Dec 14, 2018 12:34 pm
noggin wrote:
Fri Dec 14, 2018 9:57 am
I can't help with specific set-ups but I an offer some codec advice.
<snip>
Sorry, a fair amount of that is only partly true.
<snip>
Thank you very much for this needed background information and guidance. I will certainly attempt to put it to good use.

FWIW, and for the benefit of those who may also travel this road someday, I spent the better part of yesterday getting UV4L installed and configured just well enough to get its mjpeg and webrtc demo apps running. (They didn't just work "out of the box," but required a fair bit of toying with the config file.) The latency is very nearly acceptable for my purpose and will use your guidance going forward. At the moment I am observing alternating periods of very good playback and bouts of temporary stoppage/stuttering. When the stop/stutter occurs, the CPU usage on the receiving Pi drops from 45-50% down to 10-20%. This may be due to buffering or blocking I/O, or perhaps to the router, or some combination of all of that.

One of the things that surprised me in my earlier experiments was how well the RPiViewer app performed in conjunction with raspivid, netcat and TCP. Two simultaneous iPhone clients showed no perceptible latency whatsoever. (Too bad for me they're so small and expensive!) I find this outcome to be very interesting because it suggests that at least as far as the sending Pi is concerned, kernel stream buffering isn't introducing enough overhead or delay to be greatly perceptible. Looking at that app's source code, the receipt of the stream is a fairly simple and straightforward TCP socket implementation that feeds the data to what must surely be a highly optimized native app/video framework. But it makes me wonder, might there be an equivalent app/video framework for the Pi that I haven't yet heard of? JavaFX comes to mind initially as a conceptual equivalent, but I haven't seriously looked into it -- though I might, because ultimately I perceive my intended purpose to be so simple as to merit being kept on the local LAN entirely, without need for either local or remote Janus gateways and ICE/STUN servers and whatnot. Or, perhaps with a better choice of codec I might succeed in getting UV4L or ffplay to do the job. Time will tell.

noggin
Posts: 99
Joined: Sun Feb 21, 2016 1:55 pm

Re: Pi-to-Pi Low Latency One-To-Many Video Streaming

Sat Dec 15, 2018 4:56 pm

6by9 wrote:
Fri Dec 14, 2018 12:34 pm
noggin wrote:
Fri Dec 14, 2018 9:57 am
I can't help with specific set-ups but I an offer some codec advice.
<snip>
Sorry, a fair amount of that is only partly true.
Apologies.
An H264 I frame is more efficiently coded than MJPEG. MJPEG really should be avoided unless you have a very good reason to use it (needing more than 1080P would be the only reason I can think of on the Pi).

H264 supports I(ntra-coded), P(redicted) and B(idirectional) frames.
With P frames the decoder will already have the reference frames required to decode the frame - it will have minimal effect on latency.
Yes, B-frames require all reference frames, which will normally include later frames in the stream and therefore increase latency.
The Pi encoder doesn't support B-frames, therefore it's a non-issue.
Ah - so whilst a decoder may (will?) have to wait for the first I-frame to be received, once it has received it it can potentially output it ASAP, as it can P-frames (as they are based on previously received P or I frames only?)

Pilotnbr1
Posts: 10
Joined: Fri Aug 26, 2016 9:08 pm

Re: Pi-to-Pi Low Latency One-To-Many Video Streaming

Mon Mar 11, 2019 4:04 pm

Try OpenHD. Low latency one to many hd now with audio. Works on the latest pi. You will need specific WiFi adapters tho

drgeoff
Posts: 9901
Joined: Wed Jan 25, 2012 6:39 pm

Re: Pi-to-Pi Low Latency One-To-Many Video Streaming

Tue Mar 12, 2019 5:45 pm

The lowest latency and simplest system is analogue CCTV. The downside is the coax cables to the displays.

Mauronic
Posts: 3
Joined: Thu May 02, 2019 10:46 pm

Re: Pi-to-Pi Low Latency One-To-Many Video Streaming

Fri May 03, 2019 1:42 am

Any update on this?

Did you ever measure the latency? I am setting up a system that requires 120ms.

BTW - I experimented with the system that OpenHD is based on and it is an undocumented nightmare with no version control.

Return to “Graphics, sound and multimedia”