txt3rob
Posts: 365
Joined: Sat Aug 11, 2012 3:45 pm
Contact: Website

Download a file in python

Wed Jun 26, 2013 2:19 pm

I've had a look around for a solution but i have an issue the address of the file i need is

Code: Select all

https://api.scraperwiki.com/api/1.0/datastore/sqlite?format=csv&name=trainsormskirk_4&query=select+*+from+`swdata`&apikey=
any suggestions on how to get this file? i think urlencode possibly?!
The Raspberry Pi Hell Guy - Random Ramblings to assist me and others.
http://raspberrypihell.blogspot.com
My Github - http:/www.github.com/random-robbie/
http://www.smspi.co.uk - send free uk sms via your raspberry pi from here

txt3rob
Posts: 365
Joined: Sat Aug 11, 2012 3:45 pm
Contact: Website

Re: Download a file in python

Wed Jun 26, 2013 2:51 pm

tried with curl and keep getting corrupt file :S
The Raspberry Pi Hell Guy - Random Ramblings to assist me and others.
http://raspberrypihell.blogspot.com
My Github - http:/www.github.com/random-robbie/
http://www.smspi.co.uk - send free uk sms via your raspberry pi from here

User avatar
rpdom
Posts: 15423
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Download a file in python

Wed Jun 26, 2013 3:41 pm

I just tried that link with wget on the command line. I had to escape the "`"s as "\`", otherwise it tries to run a command called swdata and put that in the url.

I think you are right that something like urlencode should prevent anything like that happening.

txt3rob
Posts: 365
Joined: Sat Aug 11, 2012 3:45 pm
Contact: Website

Re: Download a file in python

Thu Jun 27, 2013 8:45 am

i used a firefox addon last night for wget and curl to see if i could download as a browser as i thought it was getting corrupted last night and just does not want to work.

now my idea is try json to mysql but that seem's harder than anything!
The Raspberry Pi Hell Guy - Random Ramblings to assist me and others.
http://raspberrypihell.blogspot.com
My Github - http:/www.github.com/random-robbie/
http://www.smspi.co.uk - send free uk sms via your raspberry pi from here

Ogoshi
Posts: 45
Joined: Wed Dec 28, 2011 5:13 pm
Location: Southend-on-Sea, Essex, England

Re: Download a file in python

Fri Jun 28, 2013 12:42 pm

Have you considered mechanize? I use it to download a gpx file, which is esentially just a txt file, so a csv (looks like what you're trying to get) would be similar, just name the file in the open processes to <file name>.csv
I can't access all my code atm, but here is a snippet...

browser.open("http://<insert link here>")
gpx_file = browser.response().read()
open(<directory & file name to save to here>, 'w').close()
fo = open (<directory & file name to save to here>, "rw+")
fo.write(gpx_file)
fo.close()

Hope that helps a little

DrMag
Posts: 63
Joined: Sat Jun 01, 2013 7:59 pm

Re: Download a file in python

Sun Jun 30, 2013 3:42 pm

If you're using Python 2, it's easily done with urllib2:

Code: Select all

import urllib2
web_file = urllib2.urlopen('https:\\...')
out_file = open('/path/to/file', 'w')
out_file.write(web_file.read())
out_file.close()
This works if you're downloading a text file at least. It seemed to work for me with the address you provided above.

txt3rob
Posts: 365
Joined: Sat Aug 11, 2012 3:45 pm
Contact: Website

Re: Download a file in python

Mon Jul 01, 2013 8:05 am

sweet thank you very much.

i ended up using a bash script so i can get rid of that and use this instead.
The Raspberry Pi Hell Guy - Random Ramblings to assist me and others.
http://raspberrypihell.blogspot.com
My Github - http:/www.github.com/random-robbie/
http://www.smspi.co.uk - send free uk sms via your raspberry pi from here

Return to “Python”