castletonroad
Posts: 104
Joined: Sat Jul 25, 2015 11:23 pm

rsync to/from remote server - no password

Sun Oct 27, 2019 7:17 pm

Hi,

I use rsync to back-up root-access files from one server to another, using:

Code: Select all

sudo rsync -vazhPe ssh --rsync-path='sudo rsync' user@source:/path /path/
The 'user' on the remote and local machines is a member of the sudo group, so can execute rsync (and I have set 'user' in visudo to 'NOPASSWD' for rsync). I also have SSH passwordless login using SSH keygen enabled for 'user'.

However, the initial login to the remote server still prompts me for a password.

How can I securely feed a password to the command line above to avoid this.

Thanks very much.
Last edited by castletonroad on Sun Oct 27, 2019 11:19 pm, edited 1 time in total.
Raspberry Pi 4 Model B | Raspberry Pi 3 Model B | Raspberry Pi 2 Model B

User avatar
default_user8
Posts: 678
Joined: Mon Nov 18, 2013 3:11 am

Re: rsync to/from remote server - no password

Sun Oct 27, 2019 10:46 pm

Two heads are better than one, unless one's a goat head.

castletonroad
Posts: 104
Joined: Sat Jul 25, 2015 11:23 pm

Re: rsync to/from remote server - no password

Sun Oct 27, 2019 11:17 pm

Thanks for the prompt response.

Apologies - I should have added that I already have SSH Passwordless Login Using SSH Keygen enabled and functioning. (I'll edit my post to reflect this.)

If this is all that is required, why am I still being prompted for a password when I execute the command above..?
Raspberry Pi 4 Model B | Raspberry Pi 3 Model B | Raspberry Pi 2 Model B

User avatar
default_user8
Posts: 678
Joined: Mon Nov 18, 2013 3:11 am

Re: rsync to/from remote server - no password

Mon Oct 28, 2019 12:31 am

castletonroad wrote:
Sun Oct 27, 2019 7:17 pm
Hi,

I use rsync to back-up root-access files from one server to another, using:

Code: Select all

sudo rsync -vazhPe ssh --rsync-path='sudo rsync' user@source:/path /path/
The 'user' on the remote and local machines is a member of the sudo group, so can execute rsync (and I have set 'user' in visudo to 'NOPASSWD' for rsync). I also have SSH passwordless login using SSH keygen enabled for 'user'.

However, the initial login to the remote server still prompts me for a password.

How can I securely feed a password to the command line above to avoid this.

Thanks very much.
So can you ssh into the remote server at all without entering a password?
Two heads are better than one, unless one's a goat head.

castletonroad
Posts: 104
Joined: Sat Jul 25, 2015 11:23 pm

Re: rsync to/from remote server - no password

Mon Oct 28, 2019 12:54 am

Yes 'user' can ssh between the two machines without passwords.

I am logged-in as 'user' on the local (destination) machine. I am assuming that it is also 'user' who is running rsync on the (remote) source, not 'root'?
Raspberry Pi 4 Model B | Raspberry Pi 3 Model B | Raspberry Pi 2 Model B

User avatar
default_user8
Posts: 678
Joined: Mon Nov 18, 2013 3:11 am

Re: rsync to/from remote server - no password

Mon Oct 28, 2019 1:22 am

castletonroad wrote:
Mon Oct 28, 2019 12:54 am
Yes 'user' can ssh between the two machines without passwords.

I am logged-in as 'user' on the local (destination) machine. I am assuming that it is also 'user' who is running rsync on the (remote) source, not 'root'?
I'm not exactly sure, but it appears to me that you are trying to run rsync on the remote machine as sudo, which to me is why you are having to provide a password. I found this older post that seems to pertain to your issue.
https://serverfault.com/questions/13654 ... both-sides
Two heads are better than one, unless one's a goat head.

castletonroad
Posts: 104
Joined: Sat Jul 25, 2015 11:23 pm

Re: rsync to/from remote server - no password

Mon Oct 28, 2019 1:43 am

Hi, I do not think that’s the case.

The login is for accessing the remote machine (I get the host login prompt).

Once logged in, rsync runs without a sudo password request...
Last edited by castletonroad on Mon Oct 28, 2019 7:30 pm, edited 1 time in total.
Raspberry Pi 4 Model B | Raspberry Pi 3 Model B | Raspberry Pi 2 Model B

bls
Posts: 286
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA

Re: rsync to/from remote server - no password

Mon Oct 28, 2019 3:34 am

rsync can be run on the remote system as a daemon, and the access can be constrained via /etc/rsyncd.conf to a reasonable, although not super-fine-grained granularity. For instance, my rsyncd.conf is:

Code: Select all

use chroot = true
log format = %h %o %f %l %b
pid file = /var/run/rsyncd.pid

[albums]
    path = /f/music/albums
    hosts allow = 192.168.16.0/24
    use chroot = false
    uid = bls
    gid = users
    read only = true
    dont compress = *

[bls]
    path = /home/bls
    hosts allow = 192.168.16.3, 192.168.16.11
    use chroot = false
    uid = bls
    gid = users
    read only = false
    dont compress = *

This rsyncd.conf creates two rsyncd modules, each with their own configuration. In this case, the remote user is 'bls', but I restrict access to these modules to specific hosts.

They're pretty similar, but you should be able to get the idea. 'man rsyncd.conf' will give you additional information. I start the rsyncd server using systemd socket, so the rsync server only runs when there is a connection. Here's /etc/systemd/system/rsyncd.socket

Code: Select all

[Unit]
Description=Rsync Server Socket
Conflicts=rsyncd.service

[Socket]
ListenStream=873
Accept=yes

[Install]
WantedBy=sockets.target 
I also mentioned this in my post about Raspberry Pi Lite-Er: https://www.raspberrypi.org/forums/view ... &p=1515887

User avatar
rpdom
Posts: 15582
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: rsync to/from remote server - no password

Mon Oct 28, 2019 8:14 am

The reason you are getting the password response is because you are using sudo (at both ends).

I assume you have passwordless ssh set up for your normal (pi?) user.

When you run the rsync with sudo it runs as the root user, not pi, and it will try to connect to the remote system as root as well, so unless you have passwordless ssh set up for the root users it will ask for the password.

Having that set up for root can be a security risk: if someone gains access to one system as root, they will be able to ssh to the other system as root as well, but if your systems are secure they won't be able to get on at all. The advantage of running rsync as root is that all file permissions and ownerships are retained.

On servers I have worked on we did not have passwordless ssh for root, but we used the rsync daemon method described previously to give limited access instead. The rsync still ran as root, but was safer.
(In the end rsync wasn't the best method for what we were doing and I wrote a custom event triggered routine instead. rsync was better than the previous method that was used, but not really suited for trying to push updates on multiple servers to each other every few minutes. We often ended up with updates that were half complete.)

castletonroad
Posts: 104
Joined: Sat Jul 25, 2015 11:23 pm

Re: rsync to/from remote server - no password

Mon Oct 28, 2019 7:32 pm

Thank you, @rpdom and @bls.

Your explanations and instructions make sense.

I'll have a go at setting rsync up to run as a daemon, and will see how I get on with that.

Thanks again.
Raspberry Pi 4 Model B | Raspberry Pi 3 Model B | Raspberry Pi 2 Model B

castletonroad
Posts: 104
Joined: Sat Jul 25, 2015 11:23 pm

Re: rsync to/from remote server - no password

Mon Nov 04, 2019 7:18 pm

So, I have rsync running as a daemon on 'raspberrypi4' (source), with /etc/rsyncd.conf as follows:

Code: Select all

pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsync.log

[nextcloud]
path = /mnt/ssd1TB_B/nextcloud
comment = RSYNC FILES - nextcloud
hosts allow = raspberrypi3
uid = pi
gid = users
read only = no
timeout = 300
don't compress = *
I then run the following on raspberrypi3 (destination) to 'pull' from raspberrypi4:

Code: Select all

sudo rsync -vazhPe pi@raspberyypi4::nextcloud/mnt/ssd1TB_B/nextcloud /mnt/ssd1TB_C/ --delete
The command runs, but only outputs the incremental file list, and tells me what the total size is (340GB) - no file sync'ing takes place.

I must be missing something still, but I've no idea what... :(
Raspberry Pi 4 Model B | Raspberry Pi 3 Model B | Raspberry Pi 2 Model B

User avatar
scruss
Posts: 2620
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: rsync to/from remote server - no password

Mon Nov 04, 2019 8:38 pm

-e takes an argument (usually ssh) and you missed it off. The equivalent --rsh=COMMAND makes things clearer.

Can I suggest putting this in a script and using the long options (so instead of -vazhP, --verbose --archive --compress --human-readable --partial --progress) so you remember what each does? It helps prevent regressions from a working process to a not-working one.

Also, debug rsync commands by putting --dry-run as the very first option when you're testing, and only when the output looks good remove it? rsync's an amazing tool, but can really mess up a filesystem quickly: especially when using any of the --delete options as you are.
(It might have deleted all the unmatched files on the remote server instead of transferring anything, which appears to be what you asked it to do)
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

castletonroad
Posts: 104
Joined: Sat Jul 25, 2015 11:23 pm

Re: rsync to/from remote server - no password

Wed Nov 06, 2019 9:30 am

Thank you @scruss.

Yes, I forgot to remove the 'e' option (!). As per your suggestion, I have updated the command (not yet running from a script) and here is what I get:

Code: Select all

pi@raspberrypi3:/etc $ sudo rsync --verbose --archive --compress --human-readable --partial --progress pi@raspberrypi4::nextcloud /mnt/ssd1TB_C/nextcloud --delete --dry-run             receiving incremental file list
rsync: opendir "/data" (in nextcloud) failed: Permission denied (13)
IO error encountered -- skipping file deletion

sent 25 bytes  received 161 bytes  53.14 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1677) [generator=3.1.3]
pi@raspberrypi3:/etc $
I'm not sure that I have the rsync command written correctly, neither am I sure that my user permissions are set correctly...

I'd really welcome further help please (please!)...
Raspberry Pi 4 Model B | Raspberry Pi 3 Model B | Raspberry Pi 2 Model B

User avatar
scruss
Posts: 2620
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: rsync to/from remote server - no password

Wed Nov 06, 2019 1:45 pm

I've never knowingly used rsyncd, but do you need the double colon '::'? ssh/scp typically only uses one: user@host:path
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

bls
Posts: 286
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA

Re: rsync to/from remote server - no password

Wed Nov 06, 2019 2:46 pm

scruss wrote:
Wed Nov 06, 2019 1:45 pm
I've never knowingly used rsyncd, but do you need the double colon '::'? ssh/scp typically only uses one: user@host:path
From the rsync man page: "Contacting an rsync daemon directly happens when the source or destination path contains a double colon (::) separator after a host specification, OR when an rsync:// URL is specified "

Looks like the last rsync command that @castletonroad posted, with the long option names (great idea, BTW) is correct, but there appears to be a protection problem on the remote end. In the rsyncd.conf there is uid=pi,gid=users.

What is the uid/gid and protection of /mnt/ssd1TB_B/nextcloud?

castletonroad
Posts: 104
Joined: Sat Jul 25, 2015 11:23 pm

Re: rsync to/from remote server - no password

Wed Nov 06, 2019 7:25 pm

@bls

The remote folder (and ideally destination folder) are www-data:www-data.

The start of this thread was because of 'permissions' issues (using sudo); the idea of using rsync as a daemon was to get around this.

The problem statement is: use a script on a destination machine to sync files/folders that are restricted access from a (remote) source machine, using rsync.

So any ideas where I go next?

Thanks for everyone's continued support resolving this.
Raspberry Pi 4 Model B | Raspberry Pi 3 Model B | Raspberry Pi 2 Model B

swampdog
Posts: 272
Joined: Fri Dec 04, 2015 11:22 am

Re: rsync to/from remote server - no password

Fri Nov 08, 2019 6:36 pm

There's three potential hurdles.

(a) ensure the source "src" box can 'ssh' into the destination "dst" box without a password.
(b) unix file permissions.
(c) keeping it simple but with a mind to security.

Take user "foo" on "src" wants to copy root files to "dst"..

foo@src $ sudo id
^^^if that works then fine. Your first post indicated it does.

You essentially want..
foo@src $ sudo rsync --progress -auxv /wrk/ root@dst:/wrk/
^^^to keep dst:/wrk up to date with src:/wrk
Ignore my rsync options except to say I chose them because they are non-destructive to src:/wrk ("-c" for ((intensive)) checksum but man rsync).
^^^enable ssh root login on "dst" (/etc/ssh/sshd_config -> PermitRootLogin yes, restart sshd on dst. Possibly generate ssh key on root@dst (if not present already) then copy root@src:~/.ssh/id_rsa.pub (or whatever key) to dst and append to root@dst:~/.ssh/authorized_keys. Test..

root@src $ ssh root@dst
^^^should work without password.

That solves it for (a). Bear in mind it is a bad idea to allow passwordless root sshd login generally and the risk ramps up if either foo@src or root@dst is an internet facing machine. Having said that I do it myself inside my own network for the three hypervisors I run.

For (b) & (c) you might use a non root account for dst. It's a viable option when dst only needs to store an archive rather than access the files. You'd still be root on src but fling src:/wrk/ to dst:/wrk over ssh using (typically) tar such that bar@dst:/wrk ends up with a tarball, bar@dst:/wrk/src-wrk.tar.gz. Only requires passwordless login for bar@dst rather than root@dst.

There is also 'sshpass' which does what you ask (man sshpass). Very much frowned upon because it almost always gets abused by folk who don't care to think out (c) but under the right circumstances can be more secure (eg: adhoc jobs).

foo@src $ sshpass -p 'root@dst password' ssh root@dst
^^^noting that if 'ssh' asks any questions nothing will appear to happen (do 'ssh root@dst' manually - typically "Are you sure you want to continue connecting") and that with changing ip addresses the issue can randomly occur. You'll need extra options to 'ssh' to suppress that and I can't recall them. Last time I used 'sshpass' was to help a co-worker globally change a flag on a bunch of cisco devices .. many years ago. You'll note from the sshpass manpage even the author does not like his own tool. Perhaps he had to faff about with a load of cisco devices! :-)

(a) is best compromise.

As you mention www-data the real solution is to match the uid/gid on the two boxes when the infrastructure was first designed. If both 'httpd' match then you can rsync directly using (say) apache@src -> apache@dst or even NFS (v4 - older is firewall nightmare).

castletonroad
Posts: 104
Joined: Sat Jul 25, 2015 11:23 pm

Re: rsync to/from remote server - no password

Fri Nov 08, 2019 8:21 pm

@swampdog

Thanks very much for your detailed response.

I think my course of action is to issue the command on 'source', run the rsync daemon on 'destination', and change the permissions on the 'destination' folder.

Thanks very much for everyone's help, it's much appreciated.
Raspberry Pi 4 Model B | Raspberry Pi 3 Model B | Raspberry Pi 2 Model B

bls
Posts: 286
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA

Re: rsync to/from remote server - no password

Sat Nov 09, 2019 12:37 am

castletonroad wrote:
Wed Nov 06, 2019 7:25 pm
@bls

The remote folder (and ideally destination folder) are www-data:www-data.

The start of this thread was because of 'permissions' issues (using sudo); the idea of using rsync as a daemon was to get around this.

The problem statement is: use a script on a destination machine to sync files/folders that are restricted access from a (remote) source machine, using rsync.

So any ideas where I go next?

Thanks for everyone's continued support resolving this.
If the destination folder is owned by www-data:www-data, then you have two choices: 1) chown -R the remote folder to be pi:users, or change /etc/rsyncd.conf uid and gid lines to be uid=www-data and gid=www-data. They need to match, or the rsync daemon won't be able to write to the directory. Of course you change the protection on the folder tree to be 777, but that is usually undesirable.

castletonroad
Posts: 104
Joined: Sat Jul 25, 2015 11:23 pm

Re: rsync to/from remote server - no password [solved]

Mon Nov 11, 2019 9:23 pm

@bls

That fixed things for me.

Code: Select all

pi@raspberrypi3:~ $ sudo rsync --verbose --archive --compress --human-readable --partial --progress pi@raspberrypi4::nextcloud /mnt/ssdb1TB_C/nextcloud --delete
with /etc/rsyncd.conf:

Code: Select all

pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsync.log
use chroot = true
log format = %h %o %f %l %b

[nextcloud]
    path = /mnt/ssdb1TB_B/nextcloud
    comment = RSYNC FILES - nextcloud
    hosts allow = raspberrypi3
    uid = www-data
    gid = www-data
    read only = no
    timeout = 300
...syncs the contents of my remote source folder (/mnt/ssdb1TB_B/nextcloud on raspberrypi4) into my local destination folder (/mnt/ssdb1TB_B/nextcloud on raspberrypi3)

I tested this with the --dry-run option, but even that didn't stop me wiping out a critical folder on my destination!

All good now, and thanks everyone for your help.
Raspberry Pi 4 Model B | Raspberry Pi 3 Model B | Raspberry Pi 2 Model B

Return to “Troubleshooting”