gsgs
Posts: 114
Joined: Sun Dec 13, 2015 6:48 pm

raspberry pi as a SD-card copying machine

Wed Dec 16, 2015 7:33 am

is it possible, has it been done ?

you boot from the normal sd-card, it automatically starts
and copies the content of the SD-card in USB1 (in USB-adapter) to the SD-card in USB2
and the LED shows when it's done.
No keyboard,no monitor needed.

Copying SD-cards is a common problem

or are there other such copying machines already

fruitoftheloom
Posts: 20153
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: raspberry pi as a SD-card copying machine

Wed Dec 16, 2015 8:07 am

gsgs wrote:is it possible, has it been done ?

you boot from the normal sd-card, it automatically starts
and copies the content of the SD-card in USB1 (in USB-adapter) to the SD-card in USB2
and the LED shows when it's done.
No keyboard,no monitor needed.

Copying SD-cards is a common problem

or are there other such copying machines already
http://www.duplicator4all.com/SDMicro-S ... --c30.html
adieu

Asus CS10 Chromebit / HP Envy 4500 Wireless Printer / Raspberry Pi Model 2B v1.1 / RealVNC Software...

gsgs
Posts: 114
Joined: Sun Dec 13, 2015 6:48 pm

Re: raspberry pi as a SD-card copying machine

Wed Dec 16, 2015 11:02 am

these are very expensive as compared to a raspberry

http://www.duplicator4all.com/1-to-7-SD ... S07SD.html
copies (from memory?) to 7 SD-cards simultaneously , costs ~$700
33MB/sec
no computer included

fruitoftheloom
Posts: 20153
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: raspberry pi as a SD-card copying machine

Wed Dec 16, 2015 12:32 pm

gsgs wrote:these are very expensive as compared to a raspberry

http://www.duplicator4all.com/1-to-7-SD ... S07SD.html
copies (from memory?) to 7 SD-cards simultaneously , costs ~$700
33MB/sec
no computer included
The biggest issue is not all SD Cards are the same size, because they advertise as say 16GB does not mean they are identical byte size, so you have to factor that in to your RPi Copying Machine..May I ask how you propose to deal with size mis-match ?
adieu

Asus CS10 Chromebit / HP Envy 4500 Wireless Printer / Raspberry Pi Model 2B v1.1 / RealVNC Software...

gsgs
Posts: 114
Joined: Sun Dec 13, 2015 6:48 pm

Re: raspberry pi as a SD-card copying machine

Wed Dec 16, 2015 3:20 pm

I don't know yet...I just only now got it booting ...

first: only cards of same size
later maybe, auto-detect

do we have an utility that writes/reads one specified sector of the card ?

stderr
Posts: 2178
Joined: Sat Dec 01, 2012 11:29 pm

Re: raspberry pi as a SD-card copying machine

Wed Dec 16, 2015 4:55 pm

fruitoftheloom wrote:
gsgs wrote:these are very expensive as compared to a raspberry

http://www.duplicator4all.com/1-to-7-SD ... S07SD.html
copies (from memory?) to 7 SD-cards simultaneously , costs ~$700
33MB/sec
no computer included
The biggest issue is not all SD Cards are the same size, because they advertise as say 16GB does not mean they are identical byte size, so you have to factor that in to your RPi Copying Machine..May I ask how you propose to deal with size mis-match ?
In the same way it is done now, copy an imagine you know is small enough to fit and then expand to fill remaining space. A fleet of these beasts networked together are probably already at work making those NOOB sdcards people like to get with their pi boards.

gordon77
Posts: 4072
Joined: Sun Aug 05, 2012 3:12 pm

Re: raspberry pi as a SD-card copying machine

Wed Dec 16, 2015 5:05 pm

gsgs wrote:is it possible, has it been done ?

you boot from the normal sd-card, it automatically starts
and copies the content of the SD-card in USB1 (in USB-adapter) to the SD-card in USB2
and the LED shows when it's done.
No keyboard,no monitor needed.

Copying SD-cards is a common problem

or are there other such copying machines already
I was interested in this so I did a bit of research and testing.

I used a Pi2B

Insert your 1st SD card into a USB adaptor (the one to be copied FROM) and sudo fdisk -l
this should give you the card ID eg /dev/sda1

Insert your 2nd card into a 2nd USB adaptor (the one to be copied TO ) and sudo fdisk -l
this should give you the card ID eg /dev/sdb1

There maybe more than one partition eg sda1, sda2 etc

I then ran the command

sudo dd if=/dev/sda of=/dev/sdb bs=1M conv=noerror

and it worked !!

How you automate this is the next step (and work out how to deal with different size cards etc).

Others will know a lot more than me, I just googled.

mikerr
Posts: 2770
Joined: Thu Jan 12, 2012 12:46 pm
Location: UK
Contact: Website

Re: raspberry pi as a SD-card copying machine

Wed Dec 16, 2015 5:15 pm

Due to USB bottlenecks, I'd guess you can do 2 simultaneously before it slows down (assuming SD cards write at 10meg/sec)
Android app - Raspi Card Imager - download and image SD cards - No PC required !

gordon77
Posts: 4072
Joined: Sun Aug 05, 2012 3:12 pm

Re: raspberry pi as a SD-card copying machine

Wed Dec 16, 2015 5:37 pm

mikerr wrote:Due to USB bottlenecks, I'd guess you can do 2 simultaneously before it slows down (assuming SD cards write at 10meg/sec)
I got the impression the OP was only looking to copy one at a time.

gsgs
Posts: 114
Joined: Sun Dec 13, 2015 6:48 pm

Re: raspberry pi as a SD-card copying machine

Wed Dec 16, 2015 5:40 pm

you can just reboot for the next copy ?!

===============================

wrt. automizing the process:
someone wrote at:

viewtopic.php?t=73585&p=528817

Code: Select all
#!/bin/bash
X :0 &
sleep 10
export DISPLAY=:0.0
<start your program>


--------------------------------

startx /usr/bin/twm

-----------------------

startx /path/to/your/program

------------------------

Thanks for the useful tips . I managed to auto login into terminal .
[how ?]
Also my program starts directly from terminal with the command .
"Startx /home/pi/mtx/default/run.sh"

Is there something similar to the old dos autoexec.bat ?

-----------------------------------

echo 'exec startx /path/to/my/program' > .profile

------------------------------------------------------

Please don't do that! The .profile is sourced by every login shell. If you add startx
to your .profile you will run startx every time you log in, including when you ssh
into the Raspberry Pi.
[what's ssh]

---------------------------------------------------
There are many reasons why you may log in to a Linux box more than once.
The OP is trying to run a single program under X11, if they need to log back
into there Raspberry Pi for any reason how are they going to do that with startx in their .profile?

I think if you are going to suggest putting startx in .profile, you should at least explain the
possibly undesirable side effect.
[just use another booting card ?]

-----------------------------------------------------

stderr
Posts: 2178
Joined: Sat Dec 01, 2012 11:29 pm

Re: raspberry pi as a SD-card copying machine

Wed Dec 16, 2015 9:01 pm

mikerr wrote:Due to USB bottlenecks, I'd guess you can do 2 simultaneously before it slows down (assuming SD cards write at 10meg/sec)
You might not really care because your limiting factor might be that you don't want to sit there and replace cards with new ones very often. So you might want to write to ten different cards and come back in a few hours.

gregeric
Posts: 1509
Joined: Mon Nov 28, 2011 10:08 am

Re: raspberry pi as a SD-card copying machine

Wed Dec 16, 2015 9:07 pm

One Pi based solution: http://www.rockandscissor.com/projects/osid/

Hasn't mikronauts done something like this too? And I recall even the RPF in the early days proposing to use Pi's for this purpose...

gsgs
Posts: 114
Joined: Sun Dec 13, 2015 6:48 pm

Re: raspberry pi as a SD-card copying machine

Thu Dec 17, 2015 6:49 am

I hate it to admit, that I think I wont be able to install that in less than -say-
one hour. I'd prefer to buy a ready, bootable SD-card, that makes the
raspberry pi a copying machine, when booted.

(what's UI script, dd, Arch, Nginx, php-fpm,distro,)

User avatar
kusti8
Posts: 3439
Joined: Sat Dec 21, 2013 5:29 pm
Location: USA

Re: raspberry pi as a SD-card copying machine

Thu Dec 17, 2015 12:21 pm

gsgs wrote:I hate it to admit, that I think I wont be able to install that in less than -say-
one hour. I'd prefer to buy a ready, bootable SD-card, that makes the
raspberry pi a copying machine, when booted.

(what's UI script, dd, Arch, Nginx, php-fpm,distro,)
That's as good as you're going to get except for making your own script. It's a learning experience. Why not try it.
There are 10 types of people: those who understand binary and those who don't.

gregeric
Posts: 1509
Joined: Mon Nov 28, 2011 10:08 am

Re: raspberry pi as a SD-card copying machine

Thu Dec 17, 2015 5:37 pm

Agreed, take the plunge and try, you'll not learn unless you do. The mikronauts gang copier I was thinking about was for eeproms, not SDCard, unfortunately: http://www.mikronauts.com/2014/12/09/ma ... rogrammer/

gordon77
Posts: 4072
Joined: Sun Aug 05, 2012 3:12 pm

Re: raspberry pi as a SD-card copying machine

Thu Dec 17, 2015 6:19 pm

I'm been having a go at writing a python program to do this.

I'm using the card byte sizes to check if the destination sd card is = or > than the source sd card and notice a brand new card of exactly the same model as the source, a card with raspbian etc installed, is smaller. Should it copy OK ?

Code: Select all

Disk /dev/sda: 29.7 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa6202af7

Device     Boot  Start      End  Sectors  Size Id Type
/dev/sda1         8192   122879   114688   56M  c W95 FAT32 (LBA)
/dev/sda2       122880 62333951 62211072 29.7G 83 Linux

Disk /dev/sdb: 28.8 GiB, 30908350464 bytes, 60367872 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start      End  Sectors  Size Id Type
/dev/sdb1        8192 60367871 60359680 28.8G  c W95 FAT32 (LBA)

gregeric
Posts: 1509
Joined: Mon Nov 28, 2011 10:08 am

Re: raspberry pi as a SD-card copying machine

Thu Dec 17, 2015 9:30 pm

If it's for example a freshly burned Raspbian image, before the file system has been expanded to fill the card, then there won't be any issues.

If the card's partitions do fill the available space, I think it may still be OK as long as it's empty space at the end. I stand a high chance of being corrected on that statement though.

gordon77
Posts: 4072
Joined: Sun Aug 05, 2012 3:12 pm

Re: raspberry pi as a SD-card copying machine

Thu Dec 17, 2015 9:44 pm

Any idea how much bigger it could cope with?

With these examples, SanDisk 32GB, the difference is 31914983424 to 30908350464.

I need to allow for this in the program.

Thanks

Gordon

User avatar
mikronauts
Posts: 2709
Joined: Sat Jan 05, 2013 7:28 pm
Contact: Website

Re: raspberry pi as a SD-card copying machine

Thu Dec 17, 2015 9:47 pm

Hi :)

I could make a gang 8x SD card copier fairly easily, however it would take approx. 1.5 hours per 8GB card (about 12 hours for eight cards), which is why I have not made one.

A simpler approach would be: (about 15-20 min per 8GB sd card)

- use an sd card for initial booting, move root fs to a USB hard drive (or flash drive), remount /boot onto the hard/flash drive
- copy the image to be cloned to the hard (or flash) drive
- keep dd'ing it to the Pi's sd slot

Simplest, already mentioned by OP above:

- get two USB flash card readers/writers
- dd from one to the other
- about 15-20 min like above
gregeric wrote:Agreed, take the plunge and try, you'll not learn unless you do. The mikronauts gang copier I was thinking about was for eeproms, not SDCard, unfortunately: http://www.mikronauts.com/2014/12/09/ma ... rogrammer/
http://Mikronauts.com - home of EZasPi, RoboPi, Pi Rtc Dio and Pi Jumper @Mikronauts on Twitter
Advanced Robotics, I/O expansion and prototyping boards for the Raspberry Pi

HiroProtagonist
Posts: 212
Joined: Sat Jun 29, 2013 9:45 am

Re: raspberry pi as a SD-card copying machine

Fri Dec 18, 2015 1:10 am

mikronauts wrote: - get two USB flash card readers/writers
- dd from one to the other
Using dd has advantages & disadvantages:
+ It will copy whatever's on the source disk, including partitioning, filesystem etc.
- It can't handle mismatched source & destination cleanly
- It will copy an entire card regardless of the amount of space actually used.

It shouldn't be too hard to devise a script that:

1) Detects partitioning on source & partitions dest to match [as long as dest is big enough]
2) Detects filesystems on source & creates same on dest
3) Copies contents of partitions to dest with rsync

Potentially a lot faster than dd in most cases.

User avatar
rpdom
Posts: 14746
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: raspberry pi as a SD-card copying machine

Fri Dec 18, 2015 3:55 am

HiroProtagonist wrote:It shouldn't be too hard to devise a script that:

1) Detects partitioning on source & partitions dest to match [as long as dest is big enough]
2) Detects filesystems on source & creates same on dest
3) Copies contents of partitions to dest with rsync

Potentially a lot faster than dd in most cases.
That's the way I'd do it. I've had some ideas for making a simple dedicated SD card copier from a Pi using a text-based interface, which would allow for either copying from a card is a USB reader to other cards, or from an image file stored on a bootable card.

Actually, because I would make the whole thing run in a ramdisk, the boot SD card could be removed after booting and the master card inserted in the Pi's SD card slot. That should make things a bit faster and the USB would be used for writing only.

User avatar
mikronauts
Posts: 2709
Joined: Sat Jan 05, 2013 7:28 pm
Contact: Website

Re: raspberry pi as a SD-card copying machine

Fri Dec 18, 2015 5:55 am

I think I tried rsync of a ~4GB filesystem once and it took longer than dd, likely due to all the directory updates needed by all those files.
HiroProtagonist wrote:
mikronauts wrote: - get two USB flash card readers/writers
- dd from one to the other
Using dd has advantages & disadvantages:
+ It will copy whatever's on the source disk, including partitioning, filesystem etc.
- It can't handle mismatched source & destination cleanly
- It will copy an entire card regardless of the amount of space actually used.

It shouldn't be too hard to devise a script that:

1) Detects partitioning on source & partitions dest to match [as long as dest is big enough]
2) Detects filesystems on source & creates same on dest
3) Copies contents of partitions to dest with rsync

Potentially a lot faster than dd in most cases.
http://Mikronauts.com - home of EZasPi, RoboPi, Pi Rtc Dio and Pi Jumper @Mikronauts on Twitter
Advanced Robotics, I/O expansion and prototyping boards for the Raspberry Pi

User avatar
rpdom
Posts: 14746
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: raspberry pi as a SD-card copying machine

Fri Dec 18, 2015 6:51 am

mikronauts wrote:I think I tried rsync of a ~4GB filesystem once and it took longer than dd, likely due to all the directory updates needed by all those files.
One thing that used to speed up copying an ext3 filesystem was mounting it as ext2 while doing the copy. That disabled the journalling, which isn't really useful during the initial load of a file system. You'd get quite a large increase in speed that way. Then, when the copy was complete you'd unmount it and activate the journal using tune2fs commands to convert it back to ext3. This may be possible with ext4 as well, but I haven't tried it.

gsgs
Posts: 114
Joined: Sun Dec 13, 2015 6:48 pm

Re: raspberry pi as a SD-card copying machine

Fri Dec 18, 2015 1:51 pm

I think time is not so important. Well, unless you use the pi for other things.

You have to insert the cards, start the pi and come back after some time to
collect the result. You could routinely just daily check the pi after breakfast or such.
Copying one card per day.
It needs few power.

As I do with loading Akkus - I rarely get the time when the Akkus are full,
I collect them the next day or such.

User avatar
mikronauts
Posts: 2709
Joined: Sat Jan 05, 2013 7:28 pm
Contact: Website

Re: raspberry pi as a SD-card copying machine

Fri Dec 18, 2015 3:45 pm

That's a great idea!
rpdom wrote:
mikronauts wrote:I think I tried rsync of a ~4GB filesystem once and it took longer than dd, likely due to all the directory updates needed by all those files.
One thing that used to speed up copying an ext3 filesystem was mounting it as ext2 while doing the copy. That disabled the journalling, which isn't really useful during the initial load of a file system. You'd get quite a large increase in speed that way. Then, when the copy was complete you'd unmount it and activate the journal using tune2fs commands to convert it back to ext3. This may be possible with ext4 as well, but I haven't tried it.
http://Mikronauts.com - home of EZasPi, RoboPi, Pi Rtc Dio and Pi Jumper @Mikronauts on Twitter
Advanced Robotics, I/O expansion and prototyping boards for the Raspberry Pi

Return to “Other projects”