red5
Posts: 3
Joined: Tue Jan 17, 2017 12:14 am

OpenVPN performance degrades over time

Sat Feb 25, 2017 11:05 pm

I am having a strange and ongoing issue with using my Raspberry Pi3 as an openvpn gateway / router on Raspbian Jessie.

After using my Pi for 2 months or so the VPN connection starts to slow down unexplainably and rebooting does nothing to resolve it. Fortunately, the last time I rebuilt my VPN router from scratch and had it working I imaged the SD card so I can simply reflash the SD card and I'm up and running in minutes. Here is what happens.

I start with my clean installation and I get about 50mbps reliable performance from my VPN provider using the Rpi3 as the router / gateway. I have it running and connected to my VPN provider 24/7. I'll reboot via SSH once or twice a week.

Then, inexplicably, about 6-8 weeks later I will notice that I can't get much more than about 4-10mbps throughput.

I can login to my VPN provider on my Win10 desktop machine and get the normal 50mbps so I know it's not a performance issue with my ISP or my VPN provider. If I reboot the Rpi3 there is no effect and the throughput remains slow.

If I then pull the SD card and reflash it with my backup image and then reboot the Pi3 my performance instantly goes back up to the 50mbps I expect.

So what is happening overtime that causes the performance to degrade? Is there some sort of cache / buffer that is filling up that needs purging or cleanup? The Rpi3 is just standard Raspbian Jessie with nothing else installed or running on it.

This issue has been going for me for about 6 months. It always comes back and I can solve it easily by reflashing the SD card back to my known stable backup, but I'd love to know what is going wrong and if there isn't a simpler fix for this issue.

epoch1970
Posts: 3557
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: OpenVPN performance degrades over time

Sun Feb 26, 2017 1:26 am

OpenVPN has perfectly stable performance over time I think, although I'm not sure I manage to get much more than 2 months of uptime out of a Pi, for whatever reason. I never use default route redirection, and only kernel-level bridged setups, not routing. Ethernet, not wifi.

An hypothesis could be your SD fills up due to heavy logging from openvpn. If you use anything beyond "verb 3" it gets extremely verbose...
Or if you're launching scripts at connection time, perhaps the scripts don't die properly and you end up with zombies filling the system. Quite unlikely I suppose.
Another possibility is that your provider (ISP or VPN) does throttle the connexion because it doesn't like to see machines hog bandwidth. Perhaps you get migrated to a lower tier server after some amount of connexion time or when the client's MAC address gets flagged.
Or your router somehow is failing after a while.

That's all I can think of. I would advise you start by checking on available free space and memory condition on the Pi when it slows down.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

edo1
Posts: 136
Joined: Sun Jun 15, 2014 3:33 pm
Location: Russia

Re: OpenVPN performance degrades over time

Sun Feb 26, 2017 8:10 pm

Execute some commands on the "broken" system to check what is happening:

Code: Select all

df -h

Code: Select all

w

Code: Select all

ifconfig; sleep 10; ifconfig

Code: Select all

wget --output-document=/dev/null https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.10.tar.xz

Code: Select all

dmesg | tail -n 30
Publish the output here if you don't know how to interpret it.

red5
Posts: 3
Joined: Tue Jan 17, 2017 12:14 am

Re: OpenVPN performance degrades over time

Mon Apr 24, 2017 12:15 am

Well, this is still plaguing me and I just can't figure it out. In fact, it seems that things are even worse where I get about 7 days of really solid OpenVPN performance through the pi3 and then it falls through the floor to 2mbps. Powering off, rebooting, nothing sorts it out. Yank the SD card and reflash it from my clean image, reboot, connect to the exact same openvpn server and I'm back to 40mbps throughput. I'm confident that this is not a server issue, but something strange going on with the pi and openvpn since I've eliminated/accounted for external factors related to the my VPN provider and even my ISP.

It seems like there is some kind of buffer/cache or something in the pi's openvpn stack that needs to be refreshed/purged to bring performance back up to normal.

I've ran the commands suggested and there doesn't look to be anything obvious wrong to me.

Code: Select all

pi@RPi3:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.1G  3.7G  3.1G  55% /
devtmpfs        475M     0  475M   0% /dev
tmpfs           479M     0  479M   0% /dev/shm
tmpfs           479M  6.8M  472M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           479M     0  479M   0% /sys/fs/cgroup
/dev/mmcblk0p1   63M   21M   42M  33% /boot
tmpfs            96M     0   96M   0% /run/user/1000
pi@RPi3:~$ w
 12:04:54 up 9 min,  3 users,  load average: 0.01, 0.02, 0.00
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
pi       tty1                      11:55    9:47   0.35s  0.31s -bash
pi       pts/0    eskimo           11:56    8:04   0.31s  0.31s -bash
pi       pts/1    eskimo           12:04    6.00s  0.33s  0.02s w
The wget command shows slow throughput as well so I didn't include that result. Oh, I did dmesg and while I'm not sure there is nothing wrong, it looks ok as best I can tell.

Code: Select all

pi@RPi3:~$ dmesg | tail -n 30
[    5.687503] brcmfmac: power management disabled
[    6.013166] cfg80211: Regulatory domain changed to country: NZ
[    6.013191] cfg80211:  DFS Master region: FCC
[    6.013200] cfg80211:   (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp), (dfs_cac_time)
[    6.013215] cfg80211:   (2402000 KHz - 2482000 KHz @ 40000 KHz), (N/A, 3000 mBm), (N/A)
[    6.013231] cfg80211:   (5170000 KHz - 5250000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 1700 mBm), (N/A)
[    6.013246] cfg80211:   (5250000 KHz - 5330000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2400 mBm), (0 s)
[    6.013259] cfg80211:   (5490000 KHz - 5730000 KHz @ 160000 KHz), (N/A, 2400 mBm), (0 s)
[    6.013272] cfg80211:   (5735000 KHz - 5835000 KHz @ 80000 KHz), (N/A, 3000 mBm), (N/A)
[    6.214392] uart-pl011 3f201000.uart: no DMA platform data
[    6.516132] Adding 102396k swap on /var/swap.  Priority:-1 extents:3 across:208896k SSFS
[    7.010809] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[    7.011136] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    8.525516] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    8.526165] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xCDE1
[   10.841401] Bluetooth: Core ver 2.21
[   10.841453] NET: Registered protocol family 31
[   10.841458] Bluetooth: HCI device and connection manager initialized
[   10.841471] Bluetooth: HCI socket layer initialized
[   10.841480] Bluetooth: L2CAP socket layer initialized
[   10.841500] Bluetooth: SCO socket layer initialized
[   10.847604] Bluetooth: HCI UART driver ver 2.3
[   10.847617] Bluetooth: HCI UART protocol H4 registered
[   10.847622] Bluetooth: HCI UART protocol Three-wire (H5) registered
[   10.847726] Bluetooth: HCI UART protocol BCM registered
[   11.033707] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[   11.033718] Bluetooth: BNEP filters: protocol multicast
[   11.033733] Bluetooth: BNEP socket layer initialized
[   27.046176] tun: Universal TUN/TAP device driver, 1.6
[   27.046190] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
pi@RPi3:~$ 

pi_everalm
Posts: 33
Joined: Thu Apr 20, 2017 11:44 am

Re: OpenVPN performance degrades over time

Mon Apr 24, 2017 8:45 am

It is POSSIBLE the issue could be related to both log file writing and associated wear on the SD card itself.

Have you tried a new SD card as opposed to continually reflashing the same card..?

Assuming that log writing is an issue, one possible way around this would be create a RAM disk on boot and having the logs write to that then periodically have the logs written to SD.

There was a product called RAMLOG that could do this, not sure if it is still around or not

Another workaround would be to have the logs written to a connected USB thumb drive, eventually that would also wear but treat it as a replaceable consumable every year or so.

YMMV

abracadabricx
Posts: 6
Joined: Thu Apr 30, 2015 2:51 pm

Re: OpenVPN performance degrades over time

Sat Oct 21, 2017 3:17 pm

Hi, I was wondering if you ever found out what the cause is. I have the same problem but so far I have not found a cause or a resolution.

Besides your case I can't find much. I tried this: https://www.lowendtalk.com/discussion/4 ... cool-story, but that makes no difference for me.

Thanks

Pir8
Posts: 3
Joined: Tue Apr 12, 2016 5:52 pm

Re: OpenVPN performance degrades over time

Thu Aug 30, 2018 5:02 am

Hello,
Have anybody been able to figure this out ? Same problem here when using the raspberry as an openvpn gateway / router. On a freshly installed Raspbian OS I get speeds around 40Mbps but after a few days it goes down to 8-10Mbps.

THanks

Pir8

epoch1970
Posts: 3557
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: OpenVPN performance degrades over time

Thu Aug 30, 2018 10:57 am

Entropy starvation (not enough random numbers available in the platform to support encryption at full speed) is another reason why a tunnel could slow down.
In any case Pi or Raspbian have no specific ovpn performance issue AFAIK.

Troubleshooting a VPN issue is difficult to impossible unless have administrative access to both ends of the tunnel, that is the server and the client.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

MartinGoose
Posts: 19
Joined: Sun Jun 17, 2012 3:17 pm
Location: NW England

Re: OpenVPN performance degrades over time

Thu Aug 30, 2018 11:30 am

Pir8 wrote:
Thu Aug 30, 2018 5:02 am
...
On a freshly installed Raspbian OS I get speeds around 40Mbps but after a few days it goes down to 8-10Mbps.
Does a reboot bring the speed back?

Return to “Troubleshooting”