TheRocketSurgeon
Posts: 12
Joined: Wed Sep 11, 2019 9:56 am

woeful throughput on pi as raspbian router

Wed Sep 11, 2019 10:05 am

Howdy guys, I'm hoping there's an obvious response to this, so not going to go into too much detail...

I've a rpi4 with raspbian, and it seems to have crappy throughput, with just a simple MASQUERADE and ip_forward enabled. Running an internet speedtest from it or any other client hitting my ISP router directly, I'm clocking 130mbps, running one *through* it i get 10mbps.

I'm thinking there's something related to the NIC driver? I can see in a tcpdump that pulling something down from the net I get some huge frames, 16kb etc., however the pi cuts them up into 1450 byte packets instead of passing the whole thing through untouched? That's the only thing I can realistically see being a contributing factor..? I read somewhere that the driver for the 4 NIC doesn't support jumbo frames, and whilst this isn't about jumbo frames, it's still larger packets in other ways.... why wouldn't the pi just sling them back through? Or does this look likely to be irrelevant?

Running some iperf3 tests locally and remotely, when I push or pull directly to the pi, i get gig worthy LAN speeds, no problem there at all, but clearly the pi is one of the endpoints there, so has more control over the creation of the payload?

epoch1970
Posts: 3579
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: woeful throughput on pi as raspbian router

Wed Sep 11, 2019 11:53 am

If you want to test network performance of a computer, do that with 2 (or more) machines (or VMs) on a LAN (VLAN) of known and consistent QoS.
Testing via your ISP is meaningless.

You don't say between what and what you're routing, but I would suspect the interfaces setup.
I don't have a Pi 4 but according to this routing will give reasonably high throughput. A simple masquerade rule costs almost nothing.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

TheRocketSurgeon
Posts: 12
Joined: Wed Sep 11, 2019 9:56 am

Re: woeful throughput on pi as raspbian router

Wed Sep 11, 2019 12:08 pm

well no, it's not meaningless when that is the problem itself. every point within the LAN seems OK, but when traffic is routed via the pi, it's awful, it seems it's the pi doing a poor job of passing the routed traffic.

fruitoftheloom
Posts: 20495
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: woeful throughput on pi as raspbian router

Wed Sep 11, 2019 12:35 pm

TheRocketSurgeon wrote:
Wed Sep 11, 2019 12:08 pm
well no, it's not meaningless when that is the problem itself. every point within the LAN seems OK, but when traffic is routed via the pi, it's awful, it seems it's the pi doing a poor job of passing the routed traffic.

Try speedtest-cli the command line version, it is in the repositories ?

This forum has many posts stating Speedtest web page gives inconsistent results....
Retired disgracefully.....

TheRocketSurgeon
Posts: 12
Joined: Wed Sep 11, 2019 9:56 am

Re: woeful throughput on pi as raspbian router

Wed Sep 11, 2019 12:49 pm

Yes, I've been using the cli client this morning:

From the pi:

Retrieving speedtest.net configuration...
Testing from Virgin Media (x.x.x.x)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Unitron Systems & Development Ltd (Telford) [46.05 km]: 27.219 ms
Testing download speed................................................................................
Download: 125.82 Mbit/s
Testing upload speed......................................................................................................
Upload: 3.47 Mbit/s


From another lan client (ContainerOS):

Retrieving speedtest.net configuration...
Testing from Virgin Media (x.x.x.x)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Unitron Systems & Development Ltd (Telford) [46.05 km]: 26.694 ms
Testing download speed................................................................................
Download: 110.37 Mbit/s
Testing upload speed......................................................................................................
Upload: 20.76 Mbit/s


From the same client, after updating it's default GW to hit the pi:

Retrieving speedtest.net configuration...
Testing from Virgin Media (x.x.x.x)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Unitron Systems & Development Ltd (Telford) [46.05 km]: 27.789 ms
Testing download speed................................................................................
Download: 15.19 Mbit/s
Testing upload speed......................................................................................................
Upload: 20.09 Mbit/s

See that the Download absolutely hits the floor, even though from the pi itself it's also fine?

trejan
Posts: 526
Joined: Tue Jul 02, 2019 2:28 pm

Re: woeful throughput on pi as raspbian router

Wed Sep 11, 2019 5:24 pm

TheRocketSurgeon wrote:
Wed Sep 11, 2019 10:05 am
I can see in a tcpdump that pulling something down from the net I get some huge frames, 16kb etc.,
A 16KB frame from where? If it is coming from the Internet then you need to check it again as that isn't usually possible. Jumbo frames are usually restricted to your local network only as virtually all routers on the internet have a maximum MTU of 1500. Your speedtest-cli log shows you're on Virgin Media and their Superhub doesn't support jumbo frames.

If your other devices on your LAN do have the MTU cranked up past 1500 then something is probably breaking PMTU when going via the Pi. Are you blocking all ICMP messages in the firewall or something like that?
TheRocketSurgeon wrote:
Wed Sep 11, 2019 10:05 am
I read somewhere that the driver for the 4 NIC doesn't support jumbo frames, and whilst this isn't about jumbo frames, it's still larger packets in other ways.... why wouldn't the pi just sling them back through? Or does this look likely to be irrelevant?
The Ethernet MAC inside the Pi 4 SoC does support jumbo frames but the driver doesn't support anything > 1500. Anything over 1500 bytes is considered a jumbo frame BTW. An Ethernet controller will just drop the frame if it sees one that is too long. It doesn't send anything back.

TheRocketSurgeon
Posts: 12
Joined: Wed Sep 11, 2019 9:56 am

Re: woeful throughput on pi as raspbian router

Wed Sep 11, 2019 5:27 pm

Well here's the tcpdump screenshot showing 16kb https://imgur.com/a/fUjEAd3

If i'm wrong about that, then that's fine, probably for the better, I just can't see anything else to point at as to why the performance is so so shoddy.

I've not messed with jumbo frames anywhere at all, that's what the pi reports it's getting from it's NIC, so is the NIC offloading a lot of the work so the OS never sees the smaller packets? Again, if it's a red herring, I'm all for it if there's a real solution!

TheRocketSurgeon
Posts: 12
Joined: Wed Sep 11, 2019 9:56 am

Re: woeful throughput on pi as raspbian router

Thu Sep 12, 2019 12:52 pm

Really hoping someone can give me an insight on this one!

asavah
Posts: 362
Joined: Thu Aug 14, 2014 12:49 am

Re: woeful throughput on pi as raspbian router

Thu Sep 12, 2019 2:39 pm

Way too many unknowns.

Please state clearly:

What kind of internet connection you have?
How the networking is done?
How many interfaces you have?
Provide the output of 'ip addr sh' and 'ip ro sh' (mask your macs and last 2 "groups" of your public ip to avoid privacy issues)
Provide the output of 'ethtool eth0' and the same for any other nics you might have.
Provide the output of 'iptables-save'

I can only tell that my pi4 which acts as both 5Ghz access point and wireguard (vpn-like, encrypted) router, which mantains 2 tunnels, easily maxes out my 100/100 Mbit internet connection while encrypting traffic. Same goes for AP - I can get 100 Mbit from my phone.
I have quite advanced setup tho.

TheRocketSurgeon
Posts: 12
Joined: Wed Sep 11, 2019 9:56 am

Re: woeful throughput on pi as raspbian router

Thu Sep 12, 2019 2:48 pm

- Pi 4B
- I have a 200mbps advertised cable connection, ethernet connected router.
- Networking between main modes are all connected to a Netgear GS-110TP switch, reelvant ports all shown as 1000 Full on the switch and device.
- Pi is only using a single eth0 interface.

ip addr sh:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether dc:a6:32:xx:xx:xx brd ff:ff:ff:ff:ff:ff
inet 192.168.0.111/24 brd 192.168.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::dea6:32ff:fexx:xxxx/64 scope link
valid_lft forever preferred_lft forever
3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether dc:a6:32:xx:xx:xx brd ff:ff:ff:ff:ff:ff

ethtool eth0:
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes: 1000baseT/Full
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: gs
Wake-on: d
SecureOn password: 00:00:00:00:00:00
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

iptables-save:
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A POSTROUTING -s 192.168.0.0/24 -o eth0 -j MASQUERADE
COMMIT

I can switch the router into modem mode, and let the pi get the address directly, I doubt that will change anything in this scenario though? Could happily add a usb gig port for the internet side of the connection, but as i'm laughably far away from saturating the link, this feels pointless at present?

asavah
Posts: 362
Joined: Thu Aug 14, 2014 12:49 am

Re: woeful throughput on pi as raspbian router

Thu Sep 12, 2019 2:56 pm

That's not a router configuration at all.

You receive the packet on eth0, nat it and forward it on the same eth0 to your main router ... what's the point?
What exactly are you trying to achieve?

TheRocketSurgeon
Posts: 12
Joined: Wed Sep 11, 2019 9:56 am

Re: woeful throughput on pi as raspbian router

Thu Sep 12, 2019 3:05 pm

Ultimately to route most traffic via openvpn connections to NordVPN amongst other things. Some services will still need to route like this though long term, things that think we're trying to dodge Geo location controls etc. We are seeing similarly crappy performance when sending traffic up the openVPN tunnels, however that was complicating the issue so it's all turned off for now.

Oh... actually, I just tested with the vpn up and with a *UDP* VPN i AM getting about 100mbps. But with NO VPN I'm down at a lower value. That feels like it should point towards something?!

jerrm
Posts: 168
Joined: Wed May 02, 2018 7:35 pm

Re: woeful throughput on pi as raspbian router

Thu Sep 12, 2019 5:41 pm

TheRocketSurgeon wrote:
Thu Sep 12, 2019 3:05 pm
Oh... actually, I just tested with the vpn up and with a *UDP* VPN i AM getting about 100mbps. But with NO VPN I'm down at a lower value. That feels like it should point towards something?!
That's as about as good as you can get with OpenVPN on the Pi. In my tests on a point to point GbE connection the Pi CPU pegged at 112mpbs with iperf over OpenVPN with AES-128.

Wireguard can do MUCH better and should be able to saturate the link if NordVPN supports it.

TheRocketSurgeon
Posts: 12
Joined: Wed Sep 11, 2019 9:56 am

Re: woeful throughput on pi as raspbian router

Thu Sep 12, 2019 5:58 pm

No, you read it backwards. I get good performance WITH a UDP VPN, I get bad performances with NO VPN.

jerrm
Posts: 168
Joined: Wed May 02, 2018 7:35 pm

Re: woeful throughput on pi as raspbian router

Thu Sep 12, 2019 6:50 pm

TheRocketSurgeon wrote:
Thu Sep 12, 2019 5:58 pm
No, you read it backwards. I get good performance WITH a UDP VPN, I get bad performances with NO VPN.
I read it right, the point was that about 100mpbs is the most you will ever get out of OpenVPN on a Pi4 with most reasonable ciphers.

The differences between direct and vpn connections was not being addressed. No clue there. My first test would be with all local equipment using iperf an not speedtest.

TheRocketSurgeon
Posts: 12
Joined: Wed Sep 11, 2019 9:56 am

Re: woeful throughput on pi as raspbian router

Thu Sep 12, 2019 7:25 pm

Ok, no worries. 100mbps over a vpn is perfect for me, no concerns about that part at all .

iperf shows 700mbps to 900mbps in and out of the pi, loads of capacity there on the raw LAN.

TheRocketSurgeon
Posts: 12
Joined: Wed Sep 11, 2019 9:56 am

Re: woeful throughput on pi as raspbian router

Fri Sep 13, 2019 1:07 pm

So i've enabled the wifi interface, so routing in one nic and out the other. Now I can get about 50mbps, with the wifi strength presumabnly being the cap. So it's *something* to do with coming in and out of eth0?

epoch1970
Posts: 3579
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: woeful throughput on pi as raspbian router

Fri Sep 13, 2019 1:41 pm

This thread is not going anywhere.

Routing happens at the border of 2 or more networks.
As already said, where is your second network? Where is the interface to that network?

(If the second interface is related to openvpn, as already said the performance is "woeful" because openvpn is using a single core, not due to routing.)
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

TheRocketSurgeon
Posts: 12
Joined: Wed Sep 11, 2019 9:56 am

Re: woeful throughput on pi as raspbian router

Fri Sep 13, 2019 1:54 pm

It's going nowhere as no one appears to be actually reading what I'm saying. FOr better or worse, I'm an enterprise network engineer. I spend my days configuring load balancers and wading through tcpdumps. I know what a router is.

Whether it's a dumb idea in it's own right or not, there should be NO REASON that i can't route traffic in and out of the same interface with a MASQ rule. it should NOT reduce the throughput by 90%. But it is, and I want to find out why. It's just one element of a jigsaw, it's not the entire use case.

Again, this has nothing to do with OpenVPN.

Please please help me! For an enthusiast, hobbyist community, this should be curious and interesting, not dismissed because it's just bad design in someones view.

jerrm
Posts: 168
Joined: Wed May 02, 2018 7:35 pm

Re: woeful throughput on pi as raspbian router

Fri Sep 13, 2019 2:14 pm

Agree it should be possible without significant hit.

All I can do is suggest troubleshooting steps I would go through, realizing you may have already tried some or all:
  • Try it on a PC (is it a PI or Linux thing?)
  • Try with a virtual interface on the same subnet (looks more like conventional routing)
  • Try with a virtual interface on a different subnet(even more like conventional routing)
  • Try another USB adapter without using the onboard ethernet at all (is it a driver issue)
  • Try another USB adapter with the onboard ethernet (a conventional routing/gateway setup)

epoch1970
Posts: 3579
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: woeful throughput on pi as raspbian router

Fri Sep 13, 2019 8:58 pm

TheRocketSurgeon wrote:
Fri Sep 13, 2019 1:54 pm
... I'm an enterprise network engineer. ...
...this should ... not dismissed because it's just bad design in someones view.
Words have a meaning and the definition of routing is not a matter of opinion. As an engineer (...), you should know that.

If you have a single interface and a useless masquerading rule, it is simply not used.
Look at the number of packets the rule has processed after a run of iperf if you're not convinced.

You may edit the title of this thread.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

jerrm
Posts: 168
Joined: Wed May 02, 2018 7:35 pm

Re: woeful throughput on pi as raspbian router

Sat Sep 14, 2019 6:26 pm

epoch1970 wrote:
Fri Sep 13, 2019 8:58 pm
If you have a single interface and a useless masquerading rule, it is simply not used.
Look at the number of packets the rule has processed after a run of iperf if you're not convinced.
The rule does exactly what he wants. The other machines on the net have the pi set as their default gateway, the Pi then forwards their traffic to the border gateway, which happens to be on the same subnet.

Nonsensical for everyday usage, but the ultimate goal is the Pi as a VPN server, and there may be times you want data to flow if the VPN is down.

Not commenting on the overall design, but the no reason it should have the kind of performance hit he sees.

I'm happy to let the OP do the testing, but curious to see the results.

TheRocketSurgeon
Posts: 12
Joined: Wed Sep 11, 2019 9:56 am

Re: woeful throughput on pi as raspbian router

Mon Sep 16, 2019 8:25 am

jerrm wrote:
Sat Sep 14, 2019 6:26 pm
epoch1970 wrote:
Fri Sep 13, 2019 8:58 pm
If you have a single interface and a useless masquerading rule, it is simply not used.
Look at the number of packets the rule has processed after a run of iperf if you're not convinced.
The rule does exactly what he wants. The other machines on the net have the pi set as their default gateway, the Pi then forwards their traffic to the border gateway, which happens to be on the same subnet.

Nonsensical for everyday usage, but the ultimate goal is the Pi as a VPN server, and there may be times you want data to flow if the VPN is down.

Not commenting on the overall design, but the no reason it should have the kind of performance hit he sees.

I'm happy to let the OP do the testing, but curious to see the results.
I've picked up a 2nd nic now, but for various reasons not successfully been able to try it out (e.g the poxy ISP router won't actually change over into modem mode...) I'll hopefully get this tried out this evening though.

Ultimately though, a fix is a fix, and the root cause is presumably never going to come to light, however strange it is.

User avatar
mahjongg
Forum Moderator
Forum Moderator
Posts: 12130
Joined: Sun Mar 11, 2012 12:19 am
Location: South Holland, The Netherlands

Re: woeful throughput on pi as raspbian router

Mon Sep 16, 2019 11:31 am

Please don't make personal attacks, keep it friendly, posts ignoring this will be deleted, and repeats will lead to a ban.

Return to “Troubleshooting”