Bosse_B
Posts: 836
Joined: Thu Jan 30, 2014 9:53 am

Need to script ftp transfers from RPi to public webserver

Sat Mar 09, 2019 6:03 pm

I need to script regular transfers of rather big binary files from the RPi to my webhosting server.
I was thinking of using sftp, but I have stumbled on the password entry because I don't want it to be part of the command line (will appear in history).
But it must be possible to run from cron or similar.
The webhosting company provides an sftp login to transfer files to the website's file system but I cannot create a certificate on the server.
So I need to use usr/pwd login.

What are my options?
Bo Berglund
Sweden

tpyo kingg
Posts: 639
Joined: Mon Apr 09, 2018 5:26 pm
Location: N. Finland

Re: Need to script ftp transfers from RPi to public webserver

Sat Mar 09, 2019 6:50 pm

Bosse_B wrote:
Sat Mar 09, 2019 6:03 pm
I was thinking of using sftp, but I have stumbled on the password entry because I don't want it to be part of the command line (will appear in history).
But it must be possible to run from cron or similar.
The webhosting company provides an sftp login to transfer files to the website's file system but I cannot create a certificate on the server.
Your best option would be to use keys for authentication. That will make it easy to script SFTP. It also has a batch mode which is available if you are using keys. Ed25519 keys are currently considered best. See "man sftp" and the options -i and -b.

Andyroo

Re: Need to script ftp transfers from RPi to public webserver

Sat Mar 09, 2019 6:55 pm

If your host will not allow keys you may be able to allow FTP from a fixed IP address. Not as safe but a work around :oops:

tpyo kingg
Posts: 639
Joined: Mon Apr 09, 2018 5:26 pm
Location: N. Finland

Re: Need to script ftp transfers from RPi to public webserver

Sat Mar 09, 2019 6:59 pm

Certificates and keys are different beasts. The technologies are similar but definitely not the same thing. But if you can't use either, then SFTP is still the way to go. There is an extension to TCL called Expect which would allow password-based logins over SFTP, but again, really, keys are the way to go.

It's 2019. FTP needs to go away and never be mentioned again.

d_older
Posts: 119
Joined: Mon Jun 25, 2012 5:04 pm
Location: East Yorkshire, UK

Re: Need to script ftp transfers from RPi to public webserver

Sat Mar 09, 2019 8:08 pm

Hi Bosse_B,

If you can't use authorized_keys and you're happy to have your sftp username (but not password) in the history buffer you could use lftp and .netrc

create a .netrc ( note the dot) in the user's home directory file with the following contents:-

Code: Select all

machine ftp.foo.bar login username password password
modified with your server, server-username and server-password.

Change the permissions on .netrc to allow rw only by the user, e.g.

Code: Select all

-rw------- 1 pi pi 50 Mar  9 18:14 .netrc
install lftp (sudo apt install lftp)
(Assuming that an sftp client is already installed, I think it is by default on Raspbian)
and then use the command (in a script)

Code: Select all

lftp -e "put filename.ext; bye" sftp://username@ftp.foo.bar
Hope this helps

Dave

Bosse_B
Posts: 836
Joined: Thu Jan 30, 2014 9:53 am

Re: Need to script ftp transfers from RPi to public webserver

Sat Mar 09, 2019 8:15 pm

tpyo kingg wrote:
Sat Mar 09, 2019 6:59 pm
There is an extension to TCL called Expect which would allow password-based logins over SFTP, but again, really, keys are the way to go.
Expect does not exist on my Linux machines...
which expect
returns nothing at all.
It's 2019. FTP needs to go away and never be mentioned again.
The company that hosts my websites directs me towards sftp to modify the site by sending new files to it.
I can't do much to change this. I use Filezilla for the manual work which is from Windows.
Now I want to automate some of this so I have thought of using a Linux machine and automate the stuff that way.
I have already scripted the creation and updates of the binary files (upwards of 150 Mbytes in size) in Linux, but the final step is the problem for me.
This is the sending of these files to the website directory where they are to reside.

Is it possible to use curl or sftp instead? Notice that the target directory is not browseable, i.e. it is outside of the Apache published web tree.

I have tried to use sftp manually in a terminal window but I see no feedback at all when I issue the command:
sftp user@host
I expected a request for password but nothing at all is shown, just an empty new line...
Bo Berglund
Sweden

Bosse_B
Posts: 836
Joined: Thu Jan 30, 2014 9:53 am

Re: Need to script ftp transfers from RPi to public webserver

Sat Mar 09, 2019 8:17 pm

d_older wrote:
Sat Mar 09, 2019 8:08 pm
If you can't use authorized_keys and you're happy to have your sftp username (but not password) in the history buffer you could use lftp and .netrc

create a .netrc ( note the dot) in the user's home directory file with the following contents:-

Code: Select all

machine ftp.foo.bar login username password password
modified with your server, server-username and server-password.

Change the permissions on .netrc to allow rw only by the user, e.g.

Code: Select all

-rw------- 1 pi pi 50 Mar  9 18:14 .netrc
install lftp (sudo apt install lftp)
(Assuming that an sftp client is already installed, I think it is by default on Raspbian)
and then use the command (in a script)

Code: Select all

lftp -e "put filename.ext; bye" sftp://username@ftp.foo.bar
Hope this helps

Dave
Thanks, I saw this only after posting my reply above...
I will try this suggestion. :)
Bo Berglund
Sweden

tpyo kingg
Posts: 639
Joined: Mon Apr 09, 2018 5:26 pm
Location: N. Finland

Re: Need to script ftp transfers from RPi to public webserver

Sat Mar 09, 2019 8:28 pm

SFTP and FTP are not the same thing. The quick overview is that SFTP is useful and can be secure and is easy to set up. FTP is the opposite. FileZilla supports SFTP and is a fine SFTP client. You have to choose it in the menus there. You can even use SSH keys with FileZilla. Your hosting provider must have some online instructions for connecting with SFTP, independent of SFTP client. If you don't mind, can you post a link to those instructions? I figure they may even require use of keys, since key-based authentication is established best practice for SSH these days.

Othewise, Expect is there in the repository and can be added easily:

Code: Select all

$ lsb_release -rd
Description:    Raspbian GNU/Linux 9.8 (stretch)
Release:        9.8
$ apt-cache policy expect
expect:
  Installerad: (ingen)
  Kandidat:    5.45-7+deb9u1
  Versionstabell:
     5.45-7+deb9u1 500
        500 http://raspbian.raspberrypi.org/raspbian stretch/main armhf Packages
Any SFTP client, even FileZilla, will be able to put files into any remote directory to which you have write access. But for scripting / automation, the default OpenSSH client "sftp" will be a good option.

Bosse_B
Posts: 836
Joined: Thu Jan 30, 2014 9:53 am

Re: Need to script ftp transfers from RPi to public webserver

Sat Mar 09, 2019 8:43 pm

Bosse_B wrote:
Sat Mar 09, 2019 8:17 pm
Thanks, I saw this only after posting my reply above...
I will try this suggestion. :)
I tried the suggestion after installing lftp and creating the .netrc file but fell flat:

Code: Select all

~$ lftp -e "pwd; bye" sftp:/xxxxxxx.com@ftpcluster.xxx.xx
lftp: sftp: Name or service not known

bosse@ubuntuserv:~$ which sftp
/usr/bin/sftp
For testing I wanted the command to just print the working directory but the result was this error...
As you can see sftp does exist...
Bo Berglund
Sweden

d_older
Posts: 119
Joined: Mon Jun 25, 2012 5:04 pm
Location: East Yorkshire, UK

Re: Need to script ftp transfers from RPi to public webserver

Sat Mar 09, 2019 9:09 pm

Hi Bosse_B

re.

Code: Select all

~$ lftp -e "pwd; bye" sftp:/xxxxxxx.com@ftpcluster.xxx.xx
lftp: sftp: Name or service not known
Does your username really end with .com?
You do need to use the word "put" and a filename
and you do need two slashes after "sftp:" e.g.

e.g.

Code: Select all

lftp -e "put filename.ext; bye" sftp://username@ftp.foo.bar

I've not tried a recursive upload- is that what you are trying with the "pwd"?

edit:
Sorry I missed the last bit of your post, using "ls" gave me a listing of the remote e.g.

Code: Select all

lftp -e "ls; bye" sftp://username@ftp.foo.bar



Dave

Bosse_B
Posts: 836
Joined: Thu Jan 30, 2014 9:53 am

Re: Need to script ftp transfers from RPi to public webserver

Sun Mar 10, 2019 7:50 am

OK, I now see the sftp:// error where I did only have sftp:/
But it did not work anyway when I tested it....
So I switched to curl instead and wrote a script that uses curl and plain ftp to do the job.
Then I locked down the script so it can't be read by anyone else than me because inside the script my ftp password is in clear text.
(It is like the apparently non-working .netrc file which holds the login for lftp.)
This works OK for the purpose of uploading the files with two cmd line arguments, file name and target dir on the server.

Code: Select all

COMMAND="curl -T $FILENAME ftp://$USER:$PASSWD@$SERVER/$BASEDIR/$DIRNAME/"
$COMMAND
The variables used are configuration statements earlier in the script or the two arguments on the command line (last two variables).
Bo Berglund
Sweden

huve
Posts: 1
Joined: Sun Mar 10, 2019 8:15 am

Re: Need to script ftp transfers from RPi to public webserver

Sun Mar 10, 2019 8:41 am

but there was a problem with ftp, which was overwriting the mp3 files everytime it ran a cronjob.

I have a company that provides background music to other companies. My method was leaving a computer in each one playing 24/7 or with other specific cronjobs, depending on the client. And the raspberry is a great way to do that instead a computer. The method i have right know that is working is a cronjob per folder. Each folder has a type of music. So i will be putting different music from time to time in the server and the cronjob will transfer those files once a week. It is set to transfer every mp3 file in that folder to the RPi. The thing is, it will transfer all the files there including the ones that were already there. If i put there, for example, 150 music files, it will take a long time transfering those, not to mention if it is done with all the folders since the RPi ARM is not that powerfull. The solution would be not overwriting the files already there, just the new ones. Then after some time another cronjob will delete all the files that have more than * days old.

I searched but it seems ftp doesnt have an option like this yet. So I found the wget command which allows to transfer without overwriting but i couldn't make it transfer multiple files. I have been trying to convert the script above with the wget command without success. Can someone with experience in this matter help out? it could be http also. Thanks in advance.[https://mypremiercreditcard.live/]premier credit card login[/https://mypremiercreditcard.live/]

tpyo kingg
Posts: 639
Joined: Mon Apr 09, 2018 5:26 pm
Location: N. Finland

Re: Need to script ftp transfers from RPi to public webserver

Sun Mar 10, 2019 9:20 am

I have to admit I'm surprised that FTP is even activated at the hosting provider in 2019. Just remember that anyone on the networks between you and your hosting provider, inclusive, can watch and record each transaction in full including password and user name.

I'd encourage trying SFTP some time so you can retire FTP for good. rsync is another option and, like SFTP, it goes over SSH and is encrypted properly. Both are quite easy to put into scripts for automated transfers.

Bosse_B
Posts: 836
Joined: Thu Jan 30, 2014 9:53 am

Re: Need to script ftp transfers from RPi to public webserver

Sun Mar 10, 2019 10:45 am

Note that I am using a commercial provider for my websites and I have no way to enable or configure the server side functionality.
I simply have to use whatever they say I should use and for file uploads to the website they guide towards using FTP.
I tried using FileZilla to connect explicitly to sftp://mysite but it was unable to connect, so I guess that SSH is not available...

Edit:
I have written the support and asked about sftp now...
EDIT2:
They responded that my account does not have access to sftp since I don't have SSH access...
But they suggest I can activate TLS transfer, which I don't know how to do and they refer to the ftp client docs. :(
Bo Berglund
Sweden

Andyroo

Re: Need to script ftp transfers from RPi to public webserver

Sun Mar 10, 2019 2:01 pm

TLS with FTP is available as an option In FileZilla.

It’s also known as FTPS - have a look for the lftp package as this allows scripting. Failing that I think curl can do it but I’ve not tried to be honest.

Too many options - maybe using a carrier pidgeon and USB stick saves you heartache?

Return to “General discussion”