Page 1 of 2

raspberry pi as a SD-card copying machine

Posted: Wed Dec 16, 2015 7:33 am
by gsgs
is it possible, has it been done ?

you boot from the normal sd-card, it automatically starts
and copies the content of the SD-card in USB1 (in USB-adapter) to the SD-card in USB2
and the LED shows when it's done.
No keyboard,no monitor needed.

Copying SD-cards is a common problem

or are there other such copying machines already

Re: raspberry pi as a SD-card copying machine

Posted: Wed Dec 16, 2015 8:07 am
by fruitoftheloom
gsgs wrote:is it possible, has it been done ?

you boot from the normal sd-card, it automatically starts
and copies the content of the SD-card in USB1 (in USB-adapter) to the SD-card in USB2
and the LED shows when it's done.
No keyboard,no monitor needed.

Copying SD-cards is a common problem

or are there other such copying machines already
http://www.duplicator4all.com/SDMicro-S ... --c30.html

Re: raspberry pi as a SD-card copying machine

Posted: Wed Dec 16, 2015 11:02 am
by gsgs
these are very expensive as compared to a raspberry

http://www.duplicator4all.com/1-to-7-SD ... S07SD.html
copies (from memory?) to 7 SD-cards simultaneously , costs ~$700
33MB/sec
no computer included

Re: raspberry pi as a SD-card copying machine

Posted: Wed Dec 16, 2015 12:32 pm
by fruitoftheloom
gsgs wrote:these are very expensive as compared to a raspberry

http://www.duplicator4all.com/1-to-7-SD ... S07SD.html
copies (from memory?) to 7 SD-cards simultaneously , costs ~$700
33MB/sec
no computer included
The biggest issue is not all SD Cards are the same size, because they advertise as say 16GB does not mean they are identical byte size, so you have to factor that in to your RPi Copying Machine..May I ask how you propose to deal with size mis-match ?

Re: raspberry pi as a SD-card copying machine

Posted: Wed Dec 16, 2015 3:20 pm
by gsgs
I don't know yet...I just only now got it booting ...

first: only cards of same size
later maybe, auto-detect

do we have an utility that writes/reads one specified sector of the card ?

Re: raspberry pi as a SD-card copying machine

Posted: Wed Dec 16, 2015 4:55 pm
by stderr
fruitoftheloom wrote:
gsgs wrote:these are very expensive as compared to a raspberry

http://www.duplicator4all.com/1-to-7-SD ... S07SD.html
copies (from memory?) to 7 SD-cards simultaneously , costs ~$700
33MB/sec
no computer included
The biggest issue is not all SD Cards are the same size, because they advertise as say 16GB does not mean they are identical byte size, so you have to factor that in to your RPi Copying Machine..May I ask how you propose to deal with size mis-match ?
In the same way it is done now, copy an imagine you know is small enough to fit and then expand to fill remaining space. A fleet of these beasts networked together are probably already at work making those NOOB sdcards people like to get with their pi boards.

Re: raspberry pi as a SD-card copying machine

Posted: Wed Dec 16, 2015 5:05 pm
by gordon77
gsgs wrote:is it possible, has it been done ?

you boot from the normal sd-card, it automatically starts
and copies the content of the SD-card in USB1 (in USB-adapter) to the SD-card in USB2
and the LED shows when it's done.
No keyboard,no monitor needed.

Copying SD-cards is a common problem

or are there other such copying machines already
I was interested in this so I did a bit of research and testing.

I used a Pi2B

Insert your 1st SD card into a USB adaptor (the one to be copied FROM) and sudo fdisk -l
this should give you the card ID eg /dev/sda1

Insert your 2nd card into a 2nd USB adaptor (the one to be copied TO ) and sudo fdisk -l
this should give you the card ID eg /dev/sdb1

There maybe more than one partition eg sda1, sda2 etc

I then ran the command

sudo dd if=/dev/sda of=/dev/sdb bs=1M conv=noerror

and it worked !!

How you automate this is the next step (and work out how to deal with different size cards etc).

Others will know a lot more than me, I just googled.

Re: raspberry pi as a SD-card copying machine

Posted: Wed Dec 16, 2015 5:15 pm
by mikerr
Due to USB bottlenecks, I'd guess you can do 2 simultaneously before it slows down (assuming SD cards write at 10meg/sec)

Re: raspberry pi as a SD-card copying machine

Posted: Wed Dec 16, 2015 5:37 pm
by gordon77
mikerr wrote:Due to USB bottlenecks, I'd guess you can do 2 simultaneously before it slows down (assuming SD cards write at 10meg/sec)
I got the impression the OP was only looking to copy one at a time.

Re: raspberry pi as a SD-card copying machine

Posted: Wed Dec 16, 2015 5:40 pm
by gsgs
you can just reboot for the next copy ?!

===============================

wrt. automizing the process:
someone wrote at:

viewtopic.php?t=73585&p=528817

Code: Select all
#!/bin/bash
X :0 &
sleep 10
export DISPLAY=:0.0
<start your program>


--------------------------------

startx /usr/bin/twm

-----------------------

startx /path/to/your/program

------------------------

Thanks for the useful tips . I managed to auto login into terminal .
[how ?]
Also my program starts directly from terminal with the command .
"Startx /home/pi/mtx/default/run.sh"

Is there something similar to the old dos autoexec.bat ?

-----------------------------------

echo 'exec startx /path/to/my/program' > .profile

------------------------------------------------------

Please don't do that! The .profile is sourced by every login shell. If you add startx
to your .profile you will run startx every time you log in, including when you ssh
into the Raspberry Pi.
[what's ssh]

---------------------------------------------------
There are many reasons why you may log in to a Linux box more than once.
The OP is trying to run a single program under X11, if they need to log back
into there Raspberry Pi for any reason how are they going to do that with startx in their .profile?

I think if you are going to suggest putting startx in .profile, you should at least explain the
possibly undesirable side effect.
[just use another booting card ?]

-----------------------------------------------------

Re: raspberry pi as a SD-card copying machine

Posted: Wed Dec 16, 2015 9:01 pm
by stderr
mikerr wrote:Due to USB bottlenecks, I'd guess you can do 2 simultaneously before it slows down (assuming SD cards write at 10meg/sec)
You might not really care because your limiting factor might be that you don't want to sit there and replace cards with new ones very often. So you might want to write to ten different cards and come back in a few hours.

Re: raspberry pi as a SD-card copying machine

Posted: Wed Dec 16, 2015 9:07 pm
by gregeric
One Pi based solution: http://www.rockandscissor.com/projects/osid/

Hasn't mikronauts done something like this too? And I recall even the RPF in the early days proposing to use Pi's for this purpose...

Re: raspberry pi as a SD-card copying machine

Posted: Thu Dec 17, 2015 6:49 am
by gsgs
I hate it to admit, that I think I wont be able to install that in less than -say-
one hour. I'd prefer to buy a ready, bootable SD-card, that makes the
raspberry pi a copying machine, when booted.

(what's UI script, dd, Arch, Nginx, php-fpm,distro,)

Re: raspberry pi as a SD-card copying machine

Posted: Thu Dec 17, 2015 12:21 pm
by kusti8
gsgs wrote:I hate it to admit, that I think I wont be able to install that in less than -say-
one hour. I'd prefer to buy a ready, bootable SD-card, that makes the
raspberry pi a copying machine, when booted.

(what's UI script, dd, Arch, Nginx, php-fpm,distro,)
That's as good as you're going to get except for making your own script. It's a learning experience. Why not try it.

Re: raspberry pi as a SD-card copying machine

Posted: Thu Dec 17, 2015 5:37 pm
by gregeric
Agreed, take the plunge and try, you'll not learn unless you do. The mikronauts gang copier I was thinking about was for eeproms, not SDCard, unfortunately: http://www.mikronauts.com/2014/12/09/ma ... rogrammer/

Re: raspberry pi as a SD-card copying machine

Posted: Thu Dec 17, 2015 6:19 pm
by gordon77
I'm been having a go at writing a python program to do this.

I'm using the card byte sizes to check if the destination sd card is = or > than the source sd card and notice a brand new card of exactly the same model as the source, a card with raspbian etc installed, is smaller. Should it copy OK ?

Code: Select all

Disk /dev/sda: 29.7 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa6202af7

Device     Boot  Start      End  Sectors  Size Id Type
/dev/sda1         8192   122879   114688   56M  c W95 FAT32 (LBA)
/dev/sda2       122880 62333951 62211072 29.7G 83 Linux

Disk /dev/sdb: 28.8 GiB, 30908350464 bytes, 60367872 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start      End  Sectors  Size Id Type
/dev/sdb1        8192 60367871 60359680 28.8G  c W95 FAT32 (LBA)

Re: raspberry pi as a SD-card copying machine

Posted: Thu Dec 17, 2015 9:30 pm
by gregeric
If it's for example a freshly burned Raspbian image, before the file system has been expanded to fill the card, then there won't be any issues.

If the card's partitions do fill the available space, I think it may still be OK as long as it's empty space at the end. I stand a high chance of being corrected on that statement though.

Re: raspberry pi as a SD-card copying machine

Posted: Thu Dec 17, 2015 9:44 pm
by gordon77
Any idea how much bigger it could cope with?

With these examples, SanDisk 32GB, the difference is 31914983424 to 30908350464.

I need to allow for this in the program.

Thanks

Gordon

Re: raspberry pi as a SD-card copying machine

Posted: Thu Dec 17, 2015 9:47 pm
by mikronauts
Hi :)

I could make a gang 8x SD card copier fairly easily, however it would take approx. 1.5 hours per 8GB card (about 12 hours for eight cards), which is why I have not made one.

A simpler approach would be: (about 15-20 min per 8GB sd card)

- use an sd card for initial booting, move root fs to a USB hard drive (or flash drive), remount /boot onto the hard/flash drive
- copy the image to be cloned to the hard (or flash) drive
- keep dd'ing it to the Pi's sd slot

Simplest, already mentioned by OP above:

- get two USB flash card readers/writers
- dd from one to the other
- about 15-20 min like above
gregeric wrote:Agreed, take the plunge and try, you'll not learn unless you do. The mikronauts gang copier I was thinking about was for eeproms, not SDCard, unfortunately: http://www.mikronauts.com/2014/12/09/ma ... rogrammer/

Re: raspberry pi as a SD-card copying machine

Posted: Fri Dec 18, 2015 1:10 am
by HiroProtagonist
mikronauts wrote: - get two USB flash card readers/writers
- dd from one to the other
Using dd has advantages & disadvantages:
+ It will copy whatever's on the source disk, including partitioning, filesystem etc.
- It can't handle mismatched source & destination cleanly
- It will copy an entire card regardless of the amount of space actually used.

It shouldn't be too hard to devise a script that:

1) Detects partitioning on source & partitions dest to match [as long as dest is big enough]
2) Detects filesystems on source & creates same on dest
3) Copies contents of partitions to dest with rsync

Potentially a lot faster than dd in most cases.

Re: raspberry pi as a SD-card copying machine

Posted: Fri Dec 18, 2015 3:55 am
by rpdom
HiroProtagonist wrote:It shouldn't be too hard to devise a script that:

1) Detects partitioning on source & partitions dest to match [as long as dest is big enough]
2) Detects filesystems on source & creates same on dest
3) Copies contents of partitions to dest with rsync

Potentially a lot faster than dd in most cases.
That's the way I'd do it. I've had some ideas for making a simple dedicated SD card copier from a Pi using a text-based interface, which would allow for either copying from a card is a USB reader to other cards, or from an image file stored on a bootable card.

Actually, because I would make the whole thing run in a ramdisk, the boot SD card could be removed after booting and the master card inserted in the Pi's SD card slot. That should make things a bit faster and the USB would be used for writing only.

Re: raspberry pi as a SD-card copying machine

Posted: Fri Dec 18, 2015 5:55 am
by mikronauts
I think I tried rsync of a ~4GB filesystem once and it took longer than dd, likely due to all the directory updates needed by all those files.
HiroProtagonist wrote:
mikronauts wrote: - get two USB flash card readers/writers
- dd from one to the other
Using dd has advantages & disadvantages:
+ It will copy whatever's on the source disk, including partitioning, filesystem etc.
- It can't handle mismatched source & destination cleanly
- It will copy an entire card regardless of the amount of space actually used.

It shouldn't be too hard to devise a script that:

1) Detects partitioning on source & partitions dest to match [as long as dest is big enough]
2) Detects filesystems on source & creates same on dest
3) Copies contents of partitions to dest with rsync

Potentially a lot faster than dd in most cases.

Re: raspberry pi as a SD-card copying machine

Posted: Fri Dec 18, 2015 6:51 am
by rpdom
mikronauts wrote:I think I tried rsync of a ~4GB filesystem once and it took longer than dd, likely due to all the directory updates needed by all those files.
One thing that used to speed up copying an ext3 filesystem was mounting it as ext2 while doing the copy. That disabled the journalling, which isn't really useful during the initial load of a file system. You'd get quite a large increase in speed that way. Then, when the copy was complete you'd unmount it and activate the journal using tune2fs commands to convert it back to ext3. This may be possible with ext4 as well, but I haven't tried it.

Re: raspberry pi as a SD-card copying machine

Posted: Fri Dec 18, 2015 1:51 pm
by gsgs
I think time is not so important. Well, unless you use the pi for other things.

You have to insert the cards, start the pi and come back after some time to
collect the result. You could routinely just daily check the pi after breakfast or such.
Copying one card per day.
It needs few power.

As I do with loading Akkus - I rarely get the time when the Akkus are full,
I collect them the next day or such.

Re: raspberry pi as a SD-card copying machine

Posted: Fri Dec 18, 2015 3:45 pm
by mikronauts
That's a great idea!
rpdom wrote:
mikronauts wrote:I think I tried rsync of a ~4GB filesystem once and it took longer than dd, likely due to all the directory updates needed by all those files.
One thing that used to speed up copying an ext3 filesystem was mounting it as ext2 while doing the copy. That disabled the journalling, which isn't really useful during the initial load of a file system. You'd get quite a large increase in speed that way. Then, when the copy was complete you'd unmount it and activate the journal using tune2fs commands to convert it back to ext3. This may be possible with ext4 as well, but I haven't tried it.