derekp
Posts: 6
Joined: Sat Jan 28, 2017 12:40 am

SSH speed limit?

Wed Aug 07, 2019 2:38 am

I've been experimenting with using a RPi 4 as a backup server, hoping to take advantage of the improved network and USB 3 speeds. The problem is I'm running into a bottleneck with SSH. I can only achieve about 40 MBPS, and top shows that ssh is taking 100% cpu. I used to be able to get around this on older systems with an older version of openssh, by switching to arcfour encryption, however that cipher has been obsoleted and removed.

I could resolve the speed issue by having ssh launch nc, then do the communications of the nc channel, but that is a bit less clean and harder to automate in some cases.

Is there any hardware encryption available on the RPi 4 SOC that ssh can use, that I'm not aware of, that can speed this up, or should I download an older version of ssh so I can use arcfour?


The second issue I have, is my backup software uses an SQLite database to store the backup metadata catalog (list of file names, file ownership/permissions, etc). This ends up being a bunch of small random I/O events on the attached USB drive, and as a result ends up being a major performance bottleneck. If I put the backup catalog DB on the SD card, performance is much improved, however I'd like to keep the DB and the actual backup data all on the external drive (right now I'm backing up the SQLite DB to the external USB drive after each backup run, but that is also an additional step). So the question -- does anyone have a recommendation for a relatively inexpensive USB 3 drive, that has better random I/O than the WD one I'm using? Maybe something that is a hybrid with a built in SSD write back cache? The other option I'm considering is hooking up a small SSD in addition to the HDD, to use with bcache configured in writeback mode.

Ultimately, I'm looking to add a section on using the RPi 4 as a backup server, to the documentation for Snebu, when I get around to putting out a new release. Currently working on bug squashing, documentation / setup improvements, and release packages.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23708
Joined: Sat Jul 30, 2011 7:41 pm

Re: SSH speed limit?

Wed Aug 07, 2019 8:38 am

What command are you using to do the copy?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

pica200
Posts: 138
Joined: Tue Aug 06, 2019 10:27 am

Re: SSH speed limit?

Wed Aug 07, 2019 9:18 am

There is no hardware acceleration. At least that's what Linux can tell ("cat /proc/cpuinfo"). With everything beeing encrypted these days hardware AES would have been nice because it reduces CPU load by a lot compared to software AES. I dunno by how much enabling the ARMv8 crypto extensions would have increased the SoC price but it can't be that much can it?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23708
Joined: Sat Jul 30, 2011 7:41 pm

Re: SSH speed limit?

Wed Aug 07, 2019 9:21 am

pica200 wrote:
Wed Aug 07, 2019 9:18 am
There is no hardware acceleration. At least that's what Linux can tell ("cat /proc/cpuinfo"). With everything beeing encrypted these days hardware AES would have been nice because it reduces CPU load by a lot compared to software AES. I dunno by how much enabling the ARMv8 crypto extensions would have increased the SoC price but it can't be that much can it?
Given 4 1500Mhz ARM CPU's, how much faster are these accelerators? I would guess not much. Not idea of the silicon real estate, but with these chips it's a constant battle to reduce die size - anything not important gets remove out.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

pica200
Posts: 138
Joined: Tue Aug 06, 2019 10:27 am

Re: SSH speed limit?

Wed Aug 07, 2019 9:58 am

Quite a lot faster. Found this by a quick search:
https://discourse.osmc.tv/t/support-for ... 4k/35358/5
Notice how AES got a big boost in performance. SHA1 not much difference but SHA256 benefits aswell. Would be nice if the Pi 5 or 6 has this enabled.

edit:
Of course it's not exactly comparable but gives you an idea what the acceleration can manage.

derekp
Posts: 6
Joined: Sat Jan 28, 2017 12:40 am

Re: SSH speed limit?

Wed Aug 07, 2019 3:17 pm

jamesh wrote:
Wed Aug 07, 2019 8:38 am
What command are you using to do the copy?
The command doesn't matter, as ssh is the bottleneck. As a quick example, which eliminates the network and disk drive, do the following:

dd if=/dev/zero bs=1K count=1M |ssh localhost 'dd of=/dev/null'

This will send 1 GB through the loopback connection, and ends up showing about 30 MB/sec, and ssh is pegging the CPU at 100%. So that means that even though the Gig E network port can get about 100 MB/sec, it won't be able to realize its full potential when used over ssh.

The actual backup process uses tar on the source machine, compresses with lzop, and sends it across the wire to the RPi using ssh to the backend backup process.

tpylkko
Posts: 382
Joined: Tue Oct 14, 2014 5:21 pm

Re: SSH speed limit?

Thu Aug 08, 2019 4:38 pm

Out of curiosity, what kind of numbers do you get without encryption?

Return to “General discussion”