self downloading an sd-card image from a server and writing it to new card

[mahjongg]

I think in the future there will be "clusters of R-pi users" who might not have access to a PC to download an SD-card image for creating writing the image to a bootable SD-card for the R-pi. The solution would be to have a program that would run totally in RAM, so that while running the SD-card can be removed and exchanged for a blank one, then the program should allow for the selection of an image from an "image store server", then (in chunks big enough to fit into the available RAM) download the selected image and burn it to the blank SD-card.

Obviously before overwriting the blank card  the software should check the card is indeed blank, and not the original card with the "download sd-card software" on it, and it should check if the card is big enough for the image before attempting to download it.

I heard some rumors claiming it was "inadvisable" to remove the card while running an OS, but that must be because normally the OS is still using the card, there should not be a hardware reason why you cannot replace the card if the OS is running completely from RAM.

Obviously there is a "chicken/egg" problem with this solution, but preferably a version of the program can be started from a normal Debian desktop, perhaps to download a "minimal RAM special distro" which has the only task to create more SD-cards. This program would then be the more optimal solution, as it can download and write bigger chunks of the image in one go, perhaps simultaneously downloading and writing to SD-card.

[jojopi]

Interesting.

Firstly the most obvious solution to the problem is to buy a USB card reader/writer.  Then the Pi can use relatively standard burn procedures like any other GNU+Linux box.  The system card might need to be a little bigger than the card being burned, in order to have room to store the compressed image file.  But for a one-off use you could stream with "wget -O- |dd".

It is extremely difficult to remove the disk while a standard OS is running.  You can not unmount the root filesystem without killing every process on the machine.  So your proposed program would essentially have to be an OS in its own right -- a kernel+initramfs started via kexec perhaps.  It would be a lot of work to get all the necessary functionality into a single standalone unit.

Also, I am not sure if kexec is actually possible on a Pi.  Without it you would have to rename the kernel files in the FAT fs and reboot into your program.  But then in case of any problem you would absolutely need a second working machine that could access SD cards.  So we are back to the USB card reader.

[dom]

The debian image has an "kernel_emergency.img" on it.

Set config.txt to contain "kernel=kernel_emergency.img" and reboot.

You are now in an initramfs / busybox kernel, which doesn't require the sdcard to be mounted.

It should support wget and dd, so can write a new sdcard.

[mahjongg]

jojopi said:


Interesting.

Firstly the most obvious solution to the problem is to buy a USB card reader/writer.


Yes, but I was hoping exactly that wouldn't be necessary, and you could do this with just software.

[rurwin]

You need at least a USB stick to store the image on. You might be able to compress it enough to fit onto a similar sized SD card along with Linux, but you cannot compress it enough to fit into 256MB of RAM along with Linux.

The alternative, as jojopi says, is to stream it off the net. That is possible but a waste of bandwidth, probably slower and certainly prone to breakage. It also depends on having a broadband link available without LEA firewalls.

So you need a USB stick and, if you have that, why not have a card-reader.

[mole125]

To be fair many more people have sticks lying around than card readers and spare cards.

This may actually be a feasible solution without even needing a second SD card.

If the default image does have a rescue kernel that runs totally in RAM and only uses files in the root partition, then with a usb stick containing an appropriate the new image, you could probably safely boot the rescue image, mount the usb stick, dd the image file onto the same card, optionally skipping the root partition image in the file and just overwriting the rest of the disk.

If it fails you should still have your current boot partition and would be able to retry it.

[mahjongg]

Using an USB stick to store the card image on is only half the solution, you still need to write that image to the SD-card.

IMHO requiring that R-PI users (who have no other computer than the R-PI) buy a card reader too so that they can do a basic task like this isn't a solution that seems desirable to me, especially as the R-PI does have all the hardware that is needed. Its quite possible that the user (kid) in question is poor enough that buying extra hardware for this task is a real problem.

I do not buy it that its impossible to do it with just software, especially because we were given pointers here on how to do it.

Wat is needed is:

A specific application that boots (leaves the normal linux desktop system) into a operating mode that runs completely in RAM, from which the only thing it does is to connect to a server on the web, present a list of images to the user to choose from, then streams the image (with error correction) and burns it into the SD-card. Seems feasible to me.

Alternatively, the user can first download the image to a separate partition on the SD-card, then checksum it, then give the image to an application that does the "sd-card first partition programming, while running from RAM" thing.

It would also be helpful to have a boot manager that can boot a newly downloaded image on a secondary partition, so you can test the image.

[Burngate]

There's a thread that I took off-topic to cover just this sort of thing -

http://www.raspberrypi.org/for.....ard-images

[hippy]

I think there are three scenarios to solve in practice -

1) A card which needs its rootfs partition restoring to a 'factory reset' state

2) A card which needs its boot partition restoring to a 'factory reset' state.

3) A card which needs both partitions restoring to 'factory reset' state.

All should be achievable with nothing more than an R-Pi and a ( same size or smaller ) second card already in 'factory reset' state, and the first should be achievable without a second card if the first has a backup partition upon it.

There seems to be plenty of discussions about such issues but little to suggest it can or will be catered for. While there undoubtedly will be technical hurdles to overcome it would be nice to hear that someone were planning to make it possible.

[jojopi]

hippy said:

1) A card which needs its rootfs partition restoring to a 'factory reset' state
2) A card which needs its boot partition restoring to a 'factory reset' state.

3) A card which needs both partitions restoring to 'factory reset' state.


These are all the same operation just with different partition numbers specified.

They are easier than the original problem because you do not need networking or any means of browsing to the correct image.  You can either use a usb card reader, or boot the standalone emergency initramfs as explained above.

However, copying a 2GB+ image using less than 192MB ram would require a lot of card swaps.  So you probably want to allow a usb flash drive to be used as intermediate storage.

[mahjongg]

jojopi said:

However, copying a 2GB+ image using less than 192MB ram would require a lot of card swaps.  So you probably want to allow a usb flash drive to be used as intermediate storage.
Well, a second sd-card is not what we want, the downloaded image should reside on the same card,  (in a different partition) or the image should be downloaded while writing to the (primary partition of the) card.

[boley]

I'm not saying this is not a worthwhile proposal, but for a number of reasons, using a SD card reader/writer seems more practical. USB - SD reader writers are very inexpensive, cheaper than the SD cards themselves. For example here is one for $4 and I didn't look very hard. IOGear Reader

It would seem that an island of several RPis would be able to share a USB - SD card reader. One more thought, I would be glad to sign on to donate a reader to one or  more someones who are located on metaphorical or real islands, who would agree provide this as a service, or to share the reader.  There might be others.

I really think an RPi SD card replicator is a good idea, regardless of whether needs an extra piece of hardware.

[mahjongg]

The cost may be small, but this potentially is something every raspberry pi owner may want to do at some time.

[rustyk123]

This is a good idea, but my thought is it is not the time for it, it may be necessary later, but I don't see a need now.  I believe that most of the people, if not all, that ordered a raspberry pi owns another computer at this point, this is merely an experiment for most of us.  So you can use your own computer to get started.  Also, there is 3 different flavors of the OS to choose from right now, it there will be more.

Now in the future, if this takes off, it may be necessary.  They other thing to think about is that there is the option of purchasing the SD card from them with the OS preloaded, yes it would cost more, but if you have no choice, you would do it.

Also, another thought, if they had the idea to include a basic OS pre-installed to download the full OS, wouldn't it make more since to increase the built-in storage just a little bit to include a full OS?  The computer is $35 dollars, you can deal with downloading the OS on another computer.

[danielverza]

No offense here, but I think this is completely unnecessary.

Considering the scenario you presented, if the person has absolutely no access to a computer (which I believe to be unlikely) then this person will probably also not have an SD card lying around and will have to buy one to use on the Pi. So, just buy one with the OS preloaded (which will be made available). There, problem solved !

[mole125]

Daniel Ferreira said:


No offense here, but I think this is completely unnecessary.

Considering the scenario you presented, if the person has absolutely no access to a computer (which I believe to be unlikely) then this person will probably also not have an SD card lying around and will have to buy one to use on the Pi. So, just buy one with the OS preloaded (which will be made available). There, problem solved !


What about if they have already brought 2 cards with the image on, now they want to upgrade to the latest OS release, it isn't reasonable to expect them to buy another.

Also shipping a brand new card half way around the world is likely to be far more expensive than buying a second hand card from the local market or being given it free on something like freecycle (many photographers will have 2gb cards sat at the back of drawers which they will happily give away as they are uselessly small)

[rustyk123]

mole125 said:


What about if they have already brought 2 cards with the image on, now they want to upgrade to the latest OS release, it isn't reasonable to expect them to buy another.


Well right there you have 2 already, so you get a card reader and update one then update the other problem solved.  I just checked the price of card readers, $3.99 from newegg.com, and less than that on ebay.

mole125 said:


Also shipping a brand new card half way around the world is likely to be far more expensive than buying a second hand card from the local market or being given it free on something like freecycle (many photographers will have 2gb cards sat at the back of drawers which they will happily give away as they are uselessly small)



My recommendation never buy a SD card second hand, they are so cheap new and they only last so long, they only way to make a used one worth the risk is free.  If you were given one by a photographer, they should have a card reader and if they were happy to give you the card free, they should be happy to let you barrow the card reader.

Besides all of this, I believe there would be some sort of updates that would update to the latest version, maybe not not to go every large update such as version 1 to 2, but going to every little update of version 1.  then you can spring a few dollars to get a new card for version 2 or invest $4 into a card reader and $5 into another card, it is not like the prices of hard drives, you are looking at less than $10.  Also, those prices included shipping, at least for USA.

[mahjongg]

I can image that programmers think this is too hard, with too little reward.

But if you want to download a new distro/game/video-player for your raspi, and you do not have a card-reader (at hand) then such a utility would be very helpful.

Imagine you are living in a small rural village, and you have to walk 30KM to the next town to buy a SD-card reader. And yes, such remote villages often do have a (wireless) internet connection, in case you are wondering! In even the most remote places they use GSM phones and such.

What about this solution, which might be easier to implement:

At boot time (before loading the kernel), copy the content of the small boot image needed to do this job to a "ram disk", then boot the ram disk, leaving the r-pi's sd-card unused.

As a side issue, (I realize we might have a chicken/egg situation here) I also propose that R-PI distro's use a "boot menu", (similar to those used on live_CD's) that make it possible to choose between booting the three possible memory models (GPU/CPU RAM sizes splits of 32/224 MB, 64/192MB and 128/128MB) maybe with three different (corresponding) distro's, one distro for "full desktops" without multimedia, one for distro's with light multimedia, and one distro (or stand alone application) for running games or other heavy multimedia applications (think XMBC) needing 128MB of GPU RAM. The same bootmenu could then also check if other partitions are present, for example one with this utility, and if present include it in the menu. That way you could partition off small section of the SD-card, fill it with this new (mini) image, then reboot and get the option to run the newly created utility image.

[Chris.Rowland]

It seems to me that there are a number of problems of the "Write to a new SD card" type and only some can easily be solved by having a tiny core that allows the SD card to be removed.

The use case I'm thinking of is cloning or backing up an existing SD card image to a new one. This needs both SD cards to be mounted so the data can be transferred easily and in one go.

Solutions involving an intermediate copy to a USB drive or by continually swapping two SD cards aren't really viable, especially when there's a simple and inexpensive hardware soluton.

[mahjongg]

Chris Rowland said:


Solutions involving an intermediate copy to a USB drive or by continually swapping two SD cards aren't really viable, especially when there's a simple and inexpensive hardware soluton.


These two cases would be unnecessary if the SD-card would be big enough to hold a few extra partitions, one for the utility, and one for the image file of the new SD-Card (partition).

As a full distro fits in just 2GB, a 4GB SD-card would be all that is needed.

[mahjongg]

Another reason that came to me why using the internal SD-Card slot for writing a new card image to is a good idea is that the R-PI only has two USB ports, and they can only power about a 100mA for both, so you will need a powered USB hub too, if you want to connect an USB card reader.