User avatar
ab1jx
Posts: 867
Joined: Thu Sep 26, 2013 1:54 pm
Location: Heath, MA USA
Contact: Website

piclone onto a hard drive?

Sat Jun 24, 2017 4:48 am

I think it almost works.. The FAT16 partition didn't get flagged as bootable so I did that with sfdisk. Then booting gets into some loop where it keeps discovering and disconnecting from my mouse. I let it sit for a few hours but it never got past that. I guess I should use dd but piclone seems preferable in cases where it works.

User avatar
DougieLawson
Posts: 35381
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: piclone onto a hard drive?

Sat Jun 24, 2017 7:01 am

That bootable flag is 100% irrelevant because there's no BIOS on a Raspberry. With the OTP USB boot bit set the hardware boot loader will enumerate every USB device then search for one that has bootcode.bin on it. That becomes the bootable device.

The easiest way to make any USB device bootable is to download 2017-06-21 Raspbian unzip it and dd that on to the raw (/dev/sda) device.
Note: Having anything remotely humorous in your signature is completely banned on this forum.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

User avatar
ab1jx
Posts: 867
Joined: Thu Sep 26, 2013 1:54 pm
Location: Heath, MA USA
Contact: Website

Re: piclone onto a hard drive?

Sat Jun 24, 2017 11:43 am

DougieLawson wrote:That bootable flag is 100% irrelevant because there's no BIOS on a Raspberry. With the OTP USB boot bit set the hardware boot loader will enumerate every USB device then search for one that has bootcode.bin on it. That becomes the bootable device.

The easiest way to make any USB device bootable is to download 2017-06-21 Raspbian unzip it and dd that on to the raw (/dev/sda) device.
No, it's not easy, that would take me at least a week. And I'm tired of this attitude that the user's files are expendable. That's been my main computer for 6 months or so. It has stuff I've written plus things built from sources that are outside the Raspbian debs. I've updated it once a month, its lineage is slightly different from the SD in this machine I'm running. I had cloned the SD on my main machine onto a 64 GB SD that was in my Zero a couple weeks before my main SD blew up. ffmpeg and cgminer are the first things that come to mind, you need to use a cgminer version that has drivers for your ASIC, not the one from Raspbian debs. There have been many forks of cgminer.

Anyway my bootcode.bin is dated 6/23/2017. I dded from the SD to the hard drive overnight, it works exactly the same as the piclone version. Differences are that the partition didn't expand to fill the hard drive, I assume I can do that with gparted. Also that I didn't know what blocksize (bs=?) to feed to dd so I didn't use anything, just let it run with its default.

I tried to snap a picture of it booting, but I'd need to get my tripod out. With the SD cards unplugged, it finds the mouse and keyboard, apparently doesn't like the mouse so it gets into a loop of detaching and re-attaching it. It scrolls the screen quickly until I guess it's doing something with /dev/random, then more slowly until the mouse bit. Using a Seagate laptop drive in a StarTech adapter/housing. Gparted is installed on my SD with about 2850 other packages.
gparted.gif
gparted.gif (14.44 KiB) Viewed 3646 times
Do I need to make a udev rule for my USB hard drive?

I can mount the hard drive and look in its /var/log/messages but nothing jumps out at me. Setting up a dmesg.boot on it might help, like at https://unix.stackexchange.com/question ... journalctl

User avatar
ab1jx
Posts: 867
Joined: Thu Sep 26, 2013 1:54 pm
Location: Heath, MA USA
Contact: Website

Re: piclone onto a hard drive?

Sat Jun 24, 2017 7:58 pm

Still at it. I ran rpi-update, rebooted, apt-get update and upgrade, rebooted. Then tried to piclone my current mounted and running sd to the hard drive. Same as the last 3 or 4 tries, it comes up whining about my mouse and looping. So I unplugged the mouse, after that it did nothing at all. I thought maybe it was confused by the extra USB device somehow, so I tried moving things around from the powered hub to the Pi, didn't seem to make much difference. This StarTech adapter is technically to USB3, but I have a USB2 hub plugged in, I don't see much difference whether it's direct or through the hub.

So then I looked in /boot/cmdline.txt amd it's setting root to /dev/mmcblk0p2 which no longer exists. Changed that to /dev/sda2. It still looks for something on /dev/mmcblk0p2, does what looks like a kernel panic, and comes up in single user mode. That second copy of the path is still in there somewhere.

cd /etc, grep -R /dev/mmcblk0p2 . and the only place it turns up is in fstab. Duh, of course. I'll try changing that too, it's not quick. I'll be back.

User avatar
ab1jx
Posts: 867
Joined: Thu Sep 26, 2013 1:54 pm
Location: Heath, MA USA
Contact: Website

Re: piclone onto a hard drive?

Sat Jun 24, 2017 8:19 pm

Hah, that was it! So to summarize, change the paths in /boot/cmdline.txt and /etc/fstab from pointing to the SD card to pointing at the hard drive.

Make these changes on the hard drive after you've cloned to it by mounting the partitions while you're still booted from the SD. Then shut down, pull out the SD, and reboot to try it.

Code: Select all

pi1# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       294G   37G  242G  14% /
devtmpfs        458M     0  458M   0% /dev
tmpfs           462M     0  462M   0% /dev/shm
tmpfs           462M  6.4M  456M   2% /run
tmpfs           5.0M  8.0K  5.0M   1% /run/lock
tmpfs           462M     0  462M   0% /sys/fs/cgroup
/dev/sda1        60M   22M   39M  36% /boot
tmpfs            93M     0   93M   0% /run/user/0
pi1# uname -a
Linux pi1 4.9.28-v7+ #998 SMP Mon May 15 16:55:39 BST 2017 armv7l GNU/Linux
pi1# 
My /etc/fstab now:

Code: Select all

proc            /proc           proc    defaults          0       0
/dev/sda1  /boot           vfat    defaults          0       2
/dev/sda2  /               ext4    defaults,noatime  0       1
My /boot/cmdline.txt now:

Code: Select all

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/sda2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
Now I can set up a real swap partition. Odd glitch, something seems to have happened to my Firefox cookies, I had to look up the password and log in here again. But I did it all with piclone and my text editor. My SD cards are sitting here on the table, I'm free of them. I'd much rather trust in Seagate than Sandisk.

User avatar
ab1jx
Posts: 867
Joined: Thu Sep 26, 2013 1:54 pm
Location: Heath, MA USA
Contact: Website

Re: piclone onto a hard drive?

Sun Jun 25, 2017 1:32 pm

It wasn't just a random glitch. Apparently Firefox cookies are stored at some path that includes the device name somehow. Web sites where I normally stay logged in, I'm now having to look up my password for all over again, the same as if I'd created a new Firefox profile. I do that a couple times a year because Firefox profiles get clogged up with junk causing it to run slower.

It's either the change in devices, or that I did apt-get upgrade and the current storage method isn't compatible with the old one. I'm now using:

Code: Select all

pi1# dpkg-query -l | grep firefox
ii  firefox-esr                            45.9.0esr-1~deb8u1                        armhf        Mozilla Firefox web browser - Extended Support Release (ESR)
Not a huge deal, just an unexpected nuisance. I have hundreds of passwords. This doesn't normally happen when deb upgrades change my Firefox version slightly though.

I asked Firefox support https://support.mozilla.org/questions/1165547

tvjon
Posts: 704
Joined: Mon Jan 07, 2013 9:11 am

Re: piclone onto a hard drive?

Sun Jun 25, 2017 4:30 pm

Hello Alan,

Can you elaborate on "piclone seems preferable in cases where it works."?
I use dd because it suits my pattern of working, is 100% reliable (as long as I'm not tired & manage to transpose source drive with target drive :-) & I know it's going to always take 45 minutes to backup a 32g µSD card.

Having looked at http://ab1jx.1apps.com/ham/index.html

I see why you're reticent about downloading big images!

As you're a radio amateur enthusiast, you've presumably looked at something like:

https://www.viasat.com/news/fastest-hom ... -mbps-plan ?

Is it just too expensive?

There seem to be a few providers. From that view from your place, it looks like a dish would be no problem?

It's sometimes cheaper, but less convenient, to have satellite download, but uplink via phone line.

User avatar
ab1jx
Posts: 867
Joined: Thu Sep 26, 2013 1:54 pm
Location: Heath, MA USA
Contact: Website

Re: piclone onto a hard drive?

Sun Jun 25, 2017 5:13 pm

Looking at http://www.exede.com/plan-results/liberty12/ I'd say yes, they're too expensive. I use a cell phone through Straight Talk which at about $50/month gives me 8 GB/month of fast mode internet and unlimited slow mode. Slow mode is about 7 kb/sec or about twice dialup speed. So mostly when I do apt-get update and upgrade it's in slow mode which is unmetered. We're expecting fiber optics some year but they've been talking about it for 8 years or more and the projected rates keep going up too. https://cyber.harvard.edu/publications/ ... case_study There are a few satellite companies around, maybe they're better. But once in a while I use my cell phone as a cell phone too and actually take it somewhere. I'd probably replace it with just a modem though like http://consumer.huawei.com/en/index.htm?type=broadband

Yes, dd is good except it also copies empty blocks. I'd have to look at the source of piclone but it does well at only copying files and the directory structure. rsync can do that too, but piclone if you start it from a terminal window you can see it umounting partitions that it might be overwriting. I've used it to clone a 128 GB SD onto a 64 GB one, no problem as long as the files fit. When I used dd to make a rescue backup of a failed SD onto a hard drive, I ended up with a 128 GB file. It copies everything. And I'm rarely sure what block size to tell it to use. Piclone seems more like BSDs backup and restore combination, which can also be used over a network.
as long as I'm not tired & manage to transpose source drive with target drive
If you're always doing about the same thing why don't you set up a script or alias for it, then you only have to remember what you called it? I've got about a dozen of them I've gotten very used to, some I've had around for years. Put them in /etc/bash.bashrc to have them work for all users.

Code: Select all

alias snap='sleep 10 ; xwd -root -out /tmp/snap.xwd'  # screenshot
alias mc='mc -ad'  # disable mouse, box-drawing in mc
alias gp='ping -c 1 www.google.com'  # to check network connectivity
alias ifc='ifconfig wlan0 && iwlist wlan0 scanning'
alias qj='qiv *.jpg'
alias qJ='qiv *.JPG'
alias qg='qiv *.gif'
alias qp='qiv *.png'
alias cls='clear'
alias bn "play -n synth brownnoise"
alias zulu='date -u'
alias xc "xclock -d -update 1"
I made several copies in experimenting mode, if I used dd to copy I always got a partition on the hard drive the same size as the source SD. When I used piclone I got a 60 MB /boot and 298 GB / which is about what I want. It didn't know that I also wanted a swap partition, so hopefully gparted can resize / to make some space.

User avatar
ab1jx
Posts: 867
Joined: Thu Sep 26, 2013 1:54 pm
Location: Heath, MA USA
Contact: Website

Re: piclone onto a hard drive?

Tue Jun 27, 2017 12:48 am

Done, 20 GB of swap, yeah. I was fairly often running out of swap and crashing with the standard amount. I know it's not fast, but since I can't add RAM I added swap. I just stuck my original SD card back in and booted, resized /dev/sda2 with gparted, then made a swap partition. Mounted /dev/sda2 and edited its /etc/fstab. Shut down, pulled out my SD card and booted back up from the hard drive. Too bad it's at the end of the drive because the head thrashing will make it slow but I don't see a way around that. I wasn't going to trust gparted to move data, maybe it works fine, I didn't want to find out.
swapon.gif
swapon.gif (13.86 KiB) Viewed 3467 times
Pretty much like a real computer, next thing is a USB DVD burner for photo storage. And under 2 watts? Sheesh.

User avatar
ab1jx
Posts: 867
Joined: Thu Sep 26, 2013 1:54 pm
Location: Heath, MA USA
Contact: Website

Re: piclone onto a hard drive?

Sat Jul 01, 2017 3:33 pm

Looks like piclone can work in the other direction too. As long as you don't have too much stuff on your hard drive so it won't fit on the SD you use, you can back up your hard drive to an SD. Right now df -h shows me I've only got 38 GB used, but I don't have any spare SDs to try it with.

If an SD card's lifetime depends on the number of write cycles, using one once or month or so for backup shouldn't wear it out very fast. If you think about getting into tape drives that size, an SD card is a lot cheaper. It would probably be best to have a few of them and always overwrite the oldest image. But you could start with just 1. And it doesn't need to be much bigger than what df -h shows the current size of /dev/root to be.

tvjon
Posts: 704
Joined: Mon Jan 07, 2013 9:11 am

Re: piclone onto a hard drive?

Sat Jul 01, 2017 9:44 pm

ab1jx wrote:...
If you're always doing about the same thing why don't you set up a script or alias for it, then you only have to remember what you called it? I've got about a dozen of them I've gotten very used to, some I've had around for years. Put them in /etc/bash.bashrc to have them work for all users.


..., next thing is a USB DVD burner for photo storage. And under 2 watts? Sheesh.

Yes, that's an idea.
When it's only a 1 line statement though, I quite like to use it directly to reinforce my memory of it :)


I use a Samsung SE-208BW which needs 20 watts. It has a 5volt 4 amps psu, & it won't work from less.

Maybe there exists a writer with a very long motor startup, to reduce power, but the laser too needs power...

Please let us know if you manage to find one using < 2 watts!

User avatar
ab1jx
Posts: 867
Joined: Thu Sep 26, 2013 1:54 pm
Location: Heath, MA USA
Contact: Website

Re: piclone onto a hard drive?

Sat Jul 01, 2017 11:39 pm

I had one years ago using rsynch to back up a directory on one machine to another, I have a mental block against keeping rsynch's syntax staight. I experiment and when I get it right I put it into a script.

I don't so much care about low power on a DVD burner since I'd only use it about once a month. I've had really good luck with Liteonit though, I think I've had about 3 and haven't worn one out (full-sized ones). I'd go for cheap (after next payday) like this https://www.walmart.com/ip/Lite-On-8x-E ... 3=&veh=sem

Actually I haven't looked that much because I don't have any spare money right now but all the ones I've looked at have been USB-powered, so not over about 7 watts. I've been watching for an excuse to buy one of these super hubs that can put out 70 watts, I've read good things about them. https://www.amazon.com/Superbpag-Portab ... Z0HXAAJVGE

ImageImage

User avatar
ab1jx
Posts: 867
Joined: Thu Sep 26, 2013 1:54 pm
Location: Heath, MA USA
Contact: Website

Re: piclone onto a hard drive?

Sat Jul 08, 2017 12:50 am

Well, no, using piclone to copy the hard drive back to the SD didn't work. I got the error message "could not create partition" in the GUI. Later I noticed it had sent "The location 584181759s is outside of the device /dev/sdb" to stderr. Where the s came from on the number I don't know. That's copied and pasted from the rxvt window.

It makes sense some things would be at offsets on the (bigger) hard drive which aren't available on the SD card. When I first started playing with Linux it was usually on a hard drive that had DOS/Windows installed. We used to run FIPS http://omniboot.org/txt/fips.txt which I think as well as splitting the existing partition also moved all the DOS files to the beginning of the DOS area so the end was uncluttered. It was certainly possible in speeddisk and other defragmenting programs. Maybe some option to fsck can do that, I didn't fiddle with it. You could have partitions that weren't bigger than your SD card except piclone wants to copy everything, there's no way to exclude anything.

Oh, wait dev/sdb is the hard drive. So my partition table has errors. Hmm. They were created using piclone in the first place. Maybe parted or gparted can fix it, I haven't looked into it.

[edit]
No, the hard drive is /dev/sda so this is what I'd expect. But also I've made a swap partition at the end of sda since I cloned onto it, is piclone trying to copy that? I don't see any way to prevent it.

Code: Select all

Disk /dev/sda: 38913 cylinders, 255 heads, 63 sectors/track
Units: cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1          0+      8-      8-     61440    e  W95 FAT16 (LBA)
/dev/sda2          8+  36363-  36356- 292025344   83  Linux
/dev/sda3      36363+  38913-   2550-  20480000   82  Linux swap / Solaris
/dev/sda4          0       -       0          0    0  Empty
But, I've got an uptime of 8 days, running fine.

User avatar
ab1jx
Posts: 867
Joined: Thu Sep 26, 2013 1:54 pm
Location: Heath, MA USA
Contact: Website

Re: piclone onto a hard drive?

Fri Jul 14, 2017 12:29 am

Doing an sfdisk -l on an SD card gets me this mess, I'm not sure what caused it originally. But piclone may have copied some of it.

Code: Select all

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/mmcblk0p1        128    2047    1920      61440    e  W95 FAT16 (LBA)
                start: (c,h,s) expected (128,0,1) found (0,130,3)
                end: (c,h,s) expected (1023,3,16) found (8,40,32)
/dev/mmcblk0p2       2048  3902143  3900096  124803072   83  Linux
                start: (c,h,s) expected (1023,3,16) found (8,40,33)
                end: (c,h,s) expected (1023,3,16) found (1023,254,63)
/dev/mmcblk0p3          0       -       0          0    0  Empty
/dev/mmcblk0p4          0       -       0          0    0  Empty
My original Jesse full that took me 10 days to download has a date of September 2016, I've been doing apt-get update and upgrade since. Written to an SD card with Win32 Disk Imager, then picloned to another SD card. Possibly cloned to a 64 GB then back to 128.

User avatar
ukscone
Forum Moderator
Forum Moderator
Posts: 4098
Joined: Fri Jul 29, 2011 2:51 pm
Contact: Website

Re: piclone onto a hard drive?

Fri Jul 14, 2017 1:29 am

bit late to the game but this is how I clone a running system

Code: Select all

sudo rsync -aAXv /* /mnt/partition_to_copy_to --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/var/tmp/*,/run/*,/mnt/*,/media/*,/lost+found, /boot}
I then copy the files from /boot to the partition i've setup on the new drive to be /boot and edit the /etc/fstab on the partition I copied to.

in theory differences in media sizes shouldn't matter

User avatar
ab1jx
Posts: 867
Joined: Thu Sep 26, 2013 1:54 pm
Location: Heath, MA USA
Contact: Website

Re: piclone onto a hard drive?

Fri Jul 14, 2017 11:13 am

I thought maybe it used rsync, or BSD's dump/restore. So I looked at the source at https://github.com/raspberrypi-ui/piclone

It's parted for the partitioning stuff, cp for the copying, the GUI's GTK using gtk_builder. And it's in C, not python. Yay!

It would be nice, when cloning an SD onto a hard drive, if it put a swap partition in. It could tell that's what it was being used for. The best thing about SD cards is that they're convenient to clone as a backup, keep different ones around with different operating systems. I suppose next I'd want to build multiple partitions for different operating systems, so merge in some Grub. Or just leave it alone.

User avatar
rpdom
Posts: 14472
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: piclone onto a hard drive?

Fri Jul 14, 2017 8:24 pm

ab1jx wrote:I thought maybe it used rsync, or BSD's dump/restore. So I looked at the source at https://github.com/raspberrypi-ui/piclone

It's parted for the partitioning stuff, cp for the copying, the GUI's GTK using gtk_builder. And it's in C, not python. Yay!

It would be nice, when cloning an SD onto a hard drive, if it put a swap partition in. It could tell that's what it was being used for. The best thing about SD cards is that they're convenient to clone as a backup, keep different ones around with different operating systems. I suppose next I'd want to build multiple partitions for different operating systems, so merge in some Grub. Or just leave it alone.
If you just want to copy your card contents to another card or hard disk with no changes, then piclone is fine (I would have used rsync though).

If you want to do anything non-standard like add a swap partition of change the partition sizes, then do it your self. You obviously know enough about Linux to be able to do that otherwise you wouldn't have asked.

User avatar
ab1jx
Posts: 867
Joined: Thu Sep 26, 2013 1:54 pm
Location: Heath, MA USA
Contact: Website

Re: piclone onto a hard drive?

Fri Jul 14, 2017 11:20 pm

I really didn't know whether it would work or not, but it did. I've done this sort of thing several times each under DOS/Windows and under OpenBSD. With both of those a particular file has to go at a certain sector on the disk because that's where something in the boot block or partition table points, not so with Linux evidently. Nothing in anything I'd read said it would work on a hard drive. It did with a couple edits to text files afterwards.

It's also good as a programming example. Gtk, threads, popen, parted are all used in ways that are easy to see how they're working, and it's well documented internally. Too many example programs try to demonstrate too much at once. Gtk in particular I've found difficult to learn. This could have been done in something I don't like, like Python calling Dialog, but C and Gtk are right up my alley. And no strange dependencies.

Return to “Troubleshooting”