User avatar
meltwater
Posts: 1015
Joined: Tue Oct 18, 2011 11:38 am

Re: Pi in the Sky - Cloud management?

Tue Nov 22, 2011 11:36 am

I was wondering if it would be possible to sync the system/user files to the cloud, then perhaps you could manage your R-pi's remotely.

One idea I would like to try is having one Pi at home and one at work (when available of course) which is kept synced/cloned (so essentially it would be like un-plugging it and taking it with me, without the remembering bit).

It would be then be very useful for schools, as they could reset or restore each class-room's Pi's for each lesson (perhaps even set-up different lessons ready to run). It could also be done so there is an option for the kids to get their own or borrow one for a project and sync with the server to clone what they've been working on.

Obviously similar system could be used if they are used in other places too.

I think in theory, since there are lots of free cloud based storage locations, it would be possible to get something like that working, even in a basic form. Obviously for schools etc they'd manage their own space etc.

Useful idea?
______________
http://www.themagpi.com/
A Magazine for Raspberry Pi Users
Read Online or Download for Free.

My new book: goo.gl/dmVtsc

Meltwater's Pi Hardware - pihardware.com

Like the MagPi? @TheMagP1 @TheMagPiTeam

obarthelemy
Posts: 1407
Joined: Tue Aug 09, 2011 10:53 pm

Re: Pi in the Sky - Cloud management?

Tue Nov 22, 2011 12:08 pm

Dropbox is not available for ARM linux. I don't know if there is something like it from another supplier.The great thing about Dropbox is that it is transparent, you don't *need* to remember to share your stuff, just save it to a shared folder, or, like I did in Windows, simply share your /home folder (or equivalent).
Old-school solution would be to set up an FTP server somewhere, with a cron job to backup your local files to it every 30 minutes and at shutdown.
For a school settings, it really depends on your goals. From shared storage to LAN booting, a wide array of choices.

Never forget that the cloud is not safe. Your data can be temporarily unavailable for a few minutes/hours/days, definitively lost or corrupted, or pirated. Backup and encrypt accordingly.

User avatar
meltwater
Posts: 1015
Joined: Tue Oct 18, 2011 11:38 am

Re: Pi in the Sky - Cloud management?

Tue Nov 22, 2011 12:38 pm

Yep, dropbox my Pi, that's about the idea of it. Although you could extend the whole thing to be a general way to manage them, once the mechanics are sorted out.

I was thinking it would have to be some extra software to interface with a particular service (either via api or some do ftp - or your own ftp depending on needs). I think the file syncing bit won't be too bad, it's the backup, restore/update bit I'm not too sure about. I don't have the skills but hope to learn.

Yes agree about cloud safety...put only what you don't mind sharing or encrypt it yourself (even if the site is never hacked, people get bored etc). Encryption wouldn't be too much of a problem, just adds another layer of difficulty to overcome in implementing. I think LAN booting will be good, but I wouldn't want to be reliant on connection to be able to boot and obviously not suited to all applications.
______________
http://www.themagpi.com/
A Magazine for Raspberry Pi Users
Read Online or Download for Free.

My new book: goo.gl/dmVtsc

Meltwater's Pi Hardware - pihardware.com

Like the MagPi? @TheMagP1 @TheMagPiTeam

greenqloud
Posts: 5
Joined: Mon Nov 21, 2011 12:41 pm
Contact: Website

Re: Pi in the Sky - Cloud management?

Thu Nov 24, 2011 11:15 pm

I would be very interested in seeing something like this happen, especially if using GreenQloud's Storage Qloud that has a web API ;)

User avatar
meltwater
Posts: 1015
Joined: Tue Oct 18, 2011 11:38 am

Re: Pi in the Sky - Cloud management?

Fri Nov 25, 2011 8:26 am

Well will see what I can manage, I think it will be useful, particularly the clone mode if possible. One issue I can possibly see is that the Linux images I've seen so far are quite large (4Gb+), so it'll be important to identify files for syncing, no doubt there are solutions we may be able make use of.

I think first, I will focus on solutions which handle just the home directory and learn from there (i.e. expand to installed software etc). It may be require a separate method for full image handling, but that could easily be handled just by file syncing the SD-Card separately until it can be automated.

Having a quick look, there appears to be a number of syncing/backup tools available so hopefully it will be a case of putting a full solution together using the parts already available.

Until there is a chance of having more than r-pi it's not really an issue.
______________
http://www.themagpi.com/
A Magazine for Raspberry Pi Users
Read Online or Download for Free.

My new book: goo.gl/dmVtsc

Meltwater's Pi Hardware - pihardware.com

Like the MagPi? @TheMagP1 @TheMagPiTeam

priceri
Posts: 1
Joined: Sun Aug 28, 2011 9:14 pm
Contact: Website

Re: Pi in the Sky - Cloud management?

Fri Nov 25, 2011 12:27 pm

Ubuntu provide free cloudspace (Ubuntu One) with options for purchasing additional services. I haven't looked in detail but there is an API :
https://one.ubuntu.com/developer/

.. so this could be a better option than dropbox.

ErvKosch
Posts: 82
Joined: Thu Sep 01, 2011 3:40 pm

Re: Pi in the Sky - Cloud management?

Fri Nov 25, 2011 2:26 pm

Didn't the latest Ubuntu versions drop all support for the Pi's processor?

kiaas
Posts: 18
Joined: Mon Nov 21, 2011 12:03 pm

Re: Pi in the Sky - Cloud management?

Fri Nov 25, 2011 2:31 pm

Quote from ErvKosch on November 25, 2011, 14:26
Didn't the latest Ubuntu versions drop all support for the Pi's processor?
yes, but Ubuntu One is not ubuntu. Ubuntu One is a partially-opensource backup solution similar to dropbox, that has a few other services available under the name. I believe all the client-side stuff is FOSS, so compiling it for ARMv6 shouldn't be too hard, as it also shouldn't be that big so any additional effort should be minimal, rather than making sure the whole OS works. Also note there's Ubuntu One clients for Ubuntu, Windows, and Android already.

User avatar
meltwater
Posts: 1015
Joined: Tue Oct 18, 2011 11:38 am

Re: Pi in the Sky - Cloud management?

Wed Dec 14, 2011 12:50 pm

Here are some additional ideas for this concept:

1. Fast/Easy backup of system/data to external storage or network (local or cloud).
Ideally, separation of user data and system data is possible (my understanding of linux is limited). UbuntuOne may well be a good solution for looking after user data without any additional work, or something similar to Dropbox syncing can be used so anything related to the use is automatically synced.

2. Management/Repository of base images.
Local repositories but perhaps even public ones where pre-build and ready to roll images can be easily downloaded and swapped out. Ideally you could extend and customise any particular image and continue to swap and change depending on your task (virtual alternative to having several SD cards which you switch around all the time). It would also allow easy try out of other peoples images and probably allow very easy group work and debugging of setups too.

3. Boot from network
This would be really nice if possible. Basically it would allow you to keep everything remotely stored, and the initial boot will check with the server what images are available and allow you to select what one you want to use. For instances where you can be sure of your network connection this would be very useful. As a fall back the same methods could be used to boot an image from a connected hard-drive too.
----------------------------------------
I guess most of the above probably would be possible by using a cut down sd image which acted as a bootstrap for other images, as well as having the ability to backup and restore them as local images.

Is user data and the system separate here or is it hard to filter out the user specific stuff?

My thinking here is that not all of us will be experts at setting systems up but some will be amazing at it (particularly if done by a number of people to build common images). Then other people can focus on putting the images to good use, extending these base images into specific uses.

Also (as mentioned before) if the data is not tied to the R-Pi then it would allow easy management within schools (you don't get 30 R-Pi's all of which might or might not have student data on which you need to backup/wipe/restore each time). Plus a number of users can work on the same virtual R-Pi system at the same time (teams could integrate changes into a core image).

Anyway, just ideas and dreams.
______________
http://www.themagpi.com/
A Magazine for Raspberry Pi Users
Read Online or Download for Free.

My new book: goo.gl/dmVtsc

Meltwater's Pi Hardware - pihardware.com

Like the MagPi? @TheMagP1 @TheMagPiTeam

User avatar
cnxsoft
Posts: 191
Joined: Sat Oct 15, 2011 2:33 pm
Location: Chiang Mai, Thailand
Contact: Website

Re: Pi in the Sky - Cloud management?

Wed Dec 14, 2011 3:17 pm

3.Boot from network:
It should be no problem actually if somebody can port a boot loader such as u-boot to the R-Pi.
Since u-boot can download the kernel and ramdisk and from a tftp at boot time.

However, the data would not be stored in those images, so another solution would be needed. It also depends on which software you use. If you use google docs then problem solved.

I've noticed dropbox daemon source code is available at https://www.dropbox.com/help/247 so it could probably be cross-compiled for ARM.

User avatar
cnxsoft
Posts: 191
Joined: Sat Oct 15, 2011 2:33 pm
Location: Chiang Mai, Thailand
Contact: Website

Re: Pi in the Sky - Cloud management?

Wed Dec 14, 2011 3:21 pm

OK.. So even if the source code is available, for some reasons dropbox can not work on ARM linux http://forums.dropbox.com/topi.....p?id=13031

User avatar
meltwater
Posts: 1015
Joined: Tue Oct 18, 2011 11:38 am

Re: Pi in the Sky - Cloud management?

Wed Dec 14, 2011 4:17 pm

Is there a way to use WebDAV since box.net supports it, that could be used instead.
Just suggested Dropbox more due to the way it works on windows, living syncing everything, even FTP would be enough for some setups.
______________
http://www.themagpi.com/
A Magazine for Raspberry Pi Users
Read Online or Download for Free.

My new book: goo.gl/dmVtsc

Meltwater's Pi Hardware - pihardware.com

Like the MagPi? @TheMagP1 @TheMagPiTeam

greenqloud
Posts: 5
Joined: Mon Nov 21, 2011 12:41 pm
Contact: Website

Re: Pi in the Sky - Cloud management?

Wed Dec 14, 2011 4:46 pm

Does ruby run yet on R-Pi?
If so then a cmdline tool like S3Sync would work with GreenQloud's QloudStorage. We would be happy to setup a playground account (Storage Qloud is an S3 clone so all the tools work with it).
http://support.greenqloud.com/.....torage-api

About DropBox:
DropBox is a sync tool like s3sync so you are always consuming local storage which isn't the best solution. Dropbox would work if Android could be run on R-Pi as would countless other apps.

Another cool DropBox like solution is OxygenCloud which has the cool feature of NOT copying the file to local storage unless you access it but they only run on mainstream OS's still.

So the first qualifying question really is sync or mounted storage? There are lots of webdav clients and servers (that mostly suck) and then there are "userland" network storage filesystems. Most of them are "virtualized" through Fuse.

If anyone could get Fuse built on R-Pi then that opens loads of options. Then again it's probably easier to just get NFS or Samba running...

cheers
Eiki

asb
Forum Moderator
Forum Moderator
Posts: 853
Joined: Fri Sep 16, 2011 7:16 pm
Contact: Website

Re: Pi in the Sky - Cloud management?

Wed Dec 14, 2011 4:48 pm

cnxsoft: That link is only a small part of Dropbox source code. They still have a proprietary, binary-only daemon to actually do the syncing.

greenqloud: Sure, there was a demo way back using kidsruby. Ruby runs great on ARM. I don't think I've made use of the fuse kernel module myself on the Raspberry Pi yet, but there's no reason it won't Just Work.

greenqloud
Posts: 5
Joined: Mon Nov 21, 2011 12:41 pm
Contact: Website

Re: Pi in the Sky - Cloud management?

Thu Dec 15, 2011 2:50 pm

Well if there is ruby then perhaps python can be used as well? If so there are many ways to do this e.g. s3fs (mount a bucket as drive), Boto (api from python), Brackup etc. etc... If only we had an R-Pi to do real world test... :p

-Eiki

obarthelemy
Posts: 1407
Joined: Tue Aug 09, 2011 10:53 pm

Re: Pi in the Sky - Cloud management?

Thu Dec 15, 2011 3:08 pm

Ubuntu One seem to be fully open source ? There are ports to *BSD.

User avatar
meltwater
Posts: 1015
Joined: Tue Oct 18, 2011 11:38 am

Re: Pi in the Sky - Cloud management?

Thu Dec 15, 2011 3:47 pm

Sweet sounds like it is do able (at least by talented people). I hope we can get something working when we get some units to use.

I'd say an option for local copy of data would be useful as well as just mounting a web drive. That way, you can take the device offline if required and sync it later on (as nice as living on the cloud sounds it is not always available).

Greenqloud sounds like a good place to work with if they are offering support, although I guess most users will want to use free storage solutions, if it can work with both, all the better (could have commercial use I suppose).

It sounds like there may be "something" useful in the ideas here and it'll be worth attempting to implement some of it. I think once we have some actual hardware, and a chance to play with the images etc it'll be worth setting up a wiki or project and see if we can get any of it working.

The project should be able to be divided into a number of smaller projects which when put together may well be able to perform the functions we want.
- User Data sync/web mount
- System Image backup/restore (to online/local network/local)
- Bootloader with boot from online/local network/local image
- Online image repository

Any other thoughts?
______________
http://www.themagpi.com/
A Magazine for Raspberry Pi Users
Read Online or Download for Free.

My new book: goo.gl/dmVtsc

Meltwater's Pi Hardware - pihardware.com

Like the MagPi? @TheMagP1 @TheMagPiTeam

User avatar
meltwater
Posts: 1015
Joined: Tue Oct 18, 2011 11:38 am

Re: Pi in the Sky - Cloud management?

Tue Dec 20, 2011 11:26 am

______________
http://www.themagpi.com/
A Magazine for Raspberry Pi Users
Read Online or Download for Free.

My new book: goo.gl/dmVtsc

Meltwater's Pi Hardware - pihardware.com

Like the MagPi? @TheMagP1 @TheMagPiTeam

greenqloud
Posts: 5
Joined: Mon Nov 21, 2011 12:41 pm
Contact: Website

Re: Pi in the Sky - Cloud management?

Tue Dec 20, 2011 11:37 am

Great to see a wiki for this topic. Talking about booting of the network (if even possible), my +1 would really go for cloud bootable Android distro with all the storage stuff built in. What do you think?

User avatar
meltwater
Posts: 1015
Joined: Tue Oct 18, 2011 11:38 am

Re: Pi in the Sky - Cloud management?

Tue Dec 20, 2011 12:17 pm

Android has been discussed before, the R-Pi may struggle with it, due the RAM/CPU but who knows it may appear in some form or another.

I suppose there could even be possibly for using the R-Pi to act as a thin-client to a server running a virtual machine, which could allow for some interesting uses.

As for booting off the network..I wouldn't have a clue how to do it myself, but I heard mention of it on here so hopefully someone can do it. I need to read up on how the process works anyway.

This is why I put it in the wiki, since there is no chance I can implement it (I have no linux skills, just a will to try) but each aspect seems like it is possible to do somehow. Some bits hopefully will be easy, others will need more work, but it could be very useful indeed.
______________
http://www.themagpi.com/
A Magazine for Raspberry Pi Users
Read Online or Download for Free.

My new book: goo.gl/dmVtsc

Meltwater's Pi Hardware - pihardware.com

Like the MagPi? @TheMagP1 @TheMagPiTeam

kingneutron
Posts: 17
Joined: Wed Dec 28, 2011 6:42 pm

Re: Pi in the Sky - Cloud management?

Thu Dec 29, 2011 1:08 am

--Sounds like gmailfs might be $useful to you, but not sure if it still works (or works on ARM...)

http://sr71.net/projects/gmailfs/

/ just a thought

Return to “Other projects”