RonR
Posts: 514
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Image File Utilities

Thu Aug 01, 2019 9:12 pm

Attached are several utilities I created and use to manipulate image files.


image-backup:

image-backup creates a backup of a running Raspbian system to a standard 'raw' image file that can be written to an SD card or a USB device card with Etcher, imageUSB, etc. It will also perform incremental updates to an existing backup image file.

Running image-backup with no parameters will create a full backup. To create the smallest possible image, specify an Image ROOT filesystem size of 0 to determine the minimum allowed size. If you plan to incrementally update the image file, specify a considerably larger size to allow for additional growth.

Running image-backup with a parameter of an existing image filename will incrementally update that image file.


Image-check:

image-check will check the integrity of a standard 'raw' image file. Usage is:

image-check imagefile [W95|Linux]

where W95 checks the BOOT partition and Linux checks the ROOT partition. If neither is specified, Linux is assumed.


image-compare:

image-compare compares a running Raspbian system to an existing standard 'raw' image file and displays the incremental changes that image-backup would perform if run. Usage is:

image-compare [imagefile]


image-mount:

image-mount mounts a standard 'raw' image file to allow it to be read or written as if it were a device. Usage is:

image-mount imagefile mountpoint [W95|Linux]

where W95 mounts the BOOT partition and Linux mounts the ROOT partition. If neither is specified, Linux is assumed.


image-set-ptuuid:

image-set-ptuuid sets the Partition Table UUID value of a standard 'raw' image file. Usage is:

image-set-ptuuid imagefile ptuuid

where ptuuid is 8 hex digits


image-shrink:

image-shrink shrinks a standard 'raw' image file to its smallest possible size (plus an optional additional amount of free space). Usage is:

image-shrink imagefile [Additional MB]

where Additional MB is an additional amount of free space to be added.
Attachments
image-utils.zip
(8.95 KiB) Downloaded 39 times
Last edited by RonR on Tue Oct 08, 2019 7:09 am, edited 2 times in total.

User avatar
rin67630
Posts: 532
Joined: Fri Mar 04, 2016 10:15 am

Re: Image File Utilities

Tue Aug 13, 2019 4:40 pm

Super! very useful applets!

Could you please tell me where to put the unzipped files? Just in /home/pi)?
Are they executable directly or must we first chmod them?

A detailed noob-proof description of what to do would really be useful.

Thank you!

RonR
Posts: 514
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Wed Aug 14, 2019 12:56 am

rin67630 wrote:
Tue Aug 13, 2019 4:40 pm
Could you please tell me where to put the unzipped files? Just in /home/pi)?
Are they executable directly or must we first chmod them?

They're not picky about where you place them (/home/pi would be fine). You must make them executable (chmod +x ...).
rin67630 wrote:
Tue Aug 13, 2019 4:40 pm
A detailed noob-proof description of what to do would really be useful.

I'll try to write up a few examples shortly.

itsmedoofer
Posts: 359
Joined: Wed Sep 25, 2013 8:43 am

Re: Image File Utilities

Wed Aug 14, 2019 11:19 am

Again, fantastic work, really appreciated !

User avatar
scruss
Posts: 2465
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: Image File Utilities

Sat Aug 24, 2019 4:57 pm

RonR wrote:
Wed Aug 14, 2019 12:56 am
They're not picky about where you place them (/home/pi would be fine). You must make them executable (chmod +x ...).
They could also go in ~/bin (that's /home/pi/bin). Raspbian automatically adds ~/bin to your path at login if the folder exists.

Code: Select all

mkdir ~/bin
is all people need to create it. The system will add it to your path the next time you login or reboot.

RonR — thanks very much for writing these. I see that some older links have expired and people can't seem to find them, so I created a github repo of your code as of 2019-08-01: RonR-RaspberryPi-image-utils. I hope that's okay with you.

I've also stored a copy of the attachment on the Internet Archive so it should stick around better.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

bls
Posts: 216
Joined: Mon Oct 22, 2018 11:25 pm

Re: Image File Utilities

Sat Aug 24, 2019 6:40 pm

image-mount is particularly sweet! Takes away all the hassle of mounting an SD card image. Thanks!

RonR
Posts: 514
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sat Aug 24, 2019 6:58 pm

bls wrote:
Sat Aug 24, 2019 6:40 pm
image-mount is particularly sweet! Takes away all the hassle of mounting an SD card image. Thanks!

Thanks for the feedback.

I use image-mount every time a new Raspbian image is released to inject an ssh and/or wpa_supplicant.conf file into the image file. Then I don't have to manually add them to the SD card or USB device each time I write one. It's also handy for extracting files from an image file created by image-backup.

bls
Posts: 216
Joined: Mon Oct 22, 2018 11:25 pm

Re: Image File Utilities

Sat Aug 24, 2019 7:13 pm

RonR wrote:
Sat Aug 24, 2019 6:58 pm
bls wrote:
Sat Aug 24, 2019 6:40 pm
image-mount is particularly sweet! Takes away all the hassle of mounting an SD card image. Thanks!

Thanks for the feedback.

I use image-mount every time a new Raspbian image is released to inject an ssh and/or wpa_supplicant.conf file into the image file. Then I don't have to manually add them to the SD card or USB device each time I write one. It's also handy for extracting files from an image file created by image-backup.
That's great! I never thought of that approach. Mine is slightly different. I have a script that adds all my stuff (scripts, config files, etc) to a newly-built SD card. I then boot it and install the packages I want. At that point I dd that beautiful SD card to an IMG file and use it for building new 'virgin' SD cards, as needed. image-mount lets me make certain classes of updates to my SD "golden master" without having to go through a complete rebuild.

User avatar
Botspot
Posts: 169
Joined: Thu Jan 17, 2019 9:47 pm
Location: Texas

Re: Image File Utilities

Fri Aug 30, 2019 2:50 pm

These are brilliant! I have used portions of image-mount in my bash script to let it mount and boot a Raspbian disk image using systemd-nspawn.

Bug report:
image-mount does not accept disk images located in a directory that contains the space character. (And yes, I tried using quotation marks.)

Code: Select all

pi@raspberrypi:~ $ sudo /home/pi/image-utils/image-mount "/home/pi/Web drives/DMA Google Drive/image files/2019-06-20-raspbian-buster-full.img" /media/pi/fs Linux

Mount /home/pi/Web drives/DMA Google Drive/image files/2019-06-20-raspbian-buster-full.img on /media/pi/fs (y/n)? y

/home/pi/image-utils/image-mount: line 66: drives/DMA * 512: division by 0 (error token is "DMA * 512")

Unable to create loop device
My doctor told me my brain is as useful as a Raspberry Pi. Is that a compliment?

User avatar
scruss
Posts: 2465
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: Image File Utilities

Fri Aug 30, 2019 10:22 pm

Yeah, it doesn't quite get the quoting right around starting round about line 60.

Of course, no-one I know would admit the creating or owning any file system with spaces in the name …
(joke, in case it was too obtuse.)
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

RonR
Posts: 514
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sat Aug 31, 2019 4:13 am

The problem was totally unrelated to quoting. The problem was due to using fdisk to display disk parameters and awk to parse them. The addition of spaces in path/file names caused awk to grab the wrong fields. The solution was to switch to sfdisk and sed.

An updated image-utils.zip file is now available in the first post of this topic which works properly with spaces in path/file names.

Thanks Botspot for reporting this.

Milliways
Posts: 451
Joined: Fri Apr 25, 2014 12:18 am
Location: Sydney, Australia

Re: Image File Utilities

Sun Sep 15, 2019 7:36 am

I had used image-mount: to view a downloaded image and tried to use image-backup to perform a backup.

The documentation is incredibly brief, there is no help and even examining the script didn't help much - it is devoid of any comments.

I have a mounted drive (in /mnt/PiHDD) which I wanted to use for the backup.

I have tried all the following (and more)

Code: Select all

sudo ./image-utils/image-backup /mnt/PiHDD/BusterBackup
/mnt/PiHDD/BusterBackup not found

sudo ./image-utils/image-backup /mnt/PiHDD/ BusterBackup
/mnt/PiHDD/ is a directory

cd /mnt/PiHDD
sudo /home/pi/image-utils/image-backup BusterBackup
BusterBackup does not begin with /mnt/ or /media/
How, on earth, do you perform a backup?

RonR
Posts: 514
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sun Sep 15, 2019 7:47 am

Milliways wrote:
Sun Sep 15, 2019 7:36 am
How, on earth, do you perform a backup?

"Running image-backup with no parameters will create a full backup."

Code: Select all

sudo image-backup

"Running image-backup with a parameter of an existing image filename will incrementally update that image file."

Code: Select all

sudo image-backup image_file_name

You must start with a full backup and then you can incrementally update it if you wish.

User avatar
jcyr
Posts: 334
Joined: Sun Apr 23, 2017 1:31 pm
Location: Atlanta

Re: Image File Utilities

Sun Sep 15, 2019 6:09 pm

How hard would it be to make this work for hybrid boot (/boot on SD, / on USB) with backup to CIFS share?
It's um...uh...well it's kinda like...and it's got a bit of...

RonR
Posts: 514
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sun Sep 15, 2019 6:15 pm

jcyr wrote:
Sun Sep 15, 2019 6:09 pm
How hard would it be to make this work for hybrid boot (/boot on SD, / on USB) with backup to CIFS share?

That already works.

User avatar
Botspot
Posts: 169
Joined: Thu Jan 17, 2019 9:47 pm
Location: Texas

Re: Image File Utilities

Sun Sep 15, 2019 7:38 pm

Thanks so much RonR for these powerful scripts!!

I am using image-mount in my vdesktop script, it allows me to mount a Stretch image on a Buster OS, then boot that as a VM in a VNC-style window!
Edit: shorten link
Last edited by Botspot on Mon Sep 16, 2019 1:30 pm, edited 2 times in total.
My doctor told me my brain is as useful as a Raspberry Pi. Is that a compliment?

Milliways
Posts: 451
Joined: Fri Apr 25, 2014 12:18 am
Location: Sydney, Australia

Re: Image File Utilities

Mon Sep 16, 2019 7:19 am

deleted

User avatar
Botspot
Posts: 169
Joined: Thu Jan 17, 2019 9:47 pm
Location: Texas

Re: Image File Utilities

Mon Sep 16, 2019 12:30 pm

Milliways wrote:
Mon Sep 16, 2019 7:19 am
deleted
Remember we want information...No information no help. Could you clarify "deleted"?
My doctor told me my brain is as useful as a Raspberry Pi. Is that a compliment?

Milliways
Posts: 451
Joined: Fri Apr 25, 2014 12:18 am
Location: Sydney, Australia

Re: Image File Utilities

Mon Sep 16, 2019 12:54 pm

Botspot wrote:
Mon Sep 16, 2019 12:30 pm
Milliways wrote:
Mon Sep 16, 2019 7:19 am
deleted
Remember we want information...No information no help. Could you clarify "deleted"?
Fix the Forum so that posts can be deleted.

User avatar
Botspot
Posts: 169
Joined: Thu Jan 17, 2019 9:47 pm
Location: Texas

Re: Image File Utilities

Mon Sep 16, 2019 1:36 pm

Milliways wrote:
Mon Sep 16, 2019 12:54 pm
Fix the Forum so that posts can be deleted.
Never done this personally, but I think you can flag your own post and tell the moderator you want it deleted.
My doctor told me my brain is as useful as a Raspberry Pi. Is that a compliment?

User avatar
jcyr
Posts: 334
Joined: Sun Apr 23, 2017 1:31 pm
Location: Atlanta

Re: Image File Utilities

Mon Sep 16, 2019 4:37 pm

Botspot wrote:
Mon Sep 16, 2019 1:36 pm
Milliways wrote:
Mon Sep 16, 2019 12:54 pm
Fix the Forum so that posts can be deleted.
Never done this personally, but I think you can flag your own post and tell the moderator you want it deleted.
Hmmm... I've never had any issues deleting my own posts! That's what the garbage can icon does...
It's um...uh...well it's kinda like...and it's got a bit of...

User avatar
rpdom
Posts: 15168
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Image File Utilities

Mon Sep 16, 2019 4:40 pm

jcyr wrote:
Mon Sep 16, 2019 4:37 pm
Botspot wrote:
Mon Sep 16, 2019 1:36 pm
Milliways wrote:
Mon Sep 16, 2019 12:54 pm
Fix the Forum so that posts can be deleted.
Never done this personally, but I think you can flag your own post and tell the moderator you want it deleted.
Hmmm... I've never had any issues deleting my own posts! That's what the garbage can icon does...
You can delete them until someone makes another reply following it.

Milliways
Posts: 451
Joined: Fri Apr 25, 2014 12:18 am
Location: Sydney, Australia

Re: Image File Utilities

Wed Sep 18, 2019 6:26 am

I have run the backup, but before copying the image to a SD Card decided to check it out.

I noticed that directories under /mnt were NOT copied. I have a number of directories which my fstab assumes exist, but they were missing.
This is not a major problem but is easily fixed.

I changed the rsync to copy the directories, but not contents with an exclude /mnt/*/*

Code: Select all

rsync -aDH --partial --numeric-ids --delete --force --exclude "${MNTPATH}" --exclude '/dev' --exclude '/media' --exclude '/mnt/*/*' --exclude '/proc' --exclude '/run' --exclude '/sys' \
This looks like a great script.
It makes more compact backups than dd and these compress to a much smaller .gz file because that omit a lot of transient rubbish.

Unfortunately the "documentation" is extremely brief. Even understanding the script is difficult as is totally devoid of comments.

I have commented my version and added a message to let you know it has started a backup update.

I will probably update my copy to facilitate making a fresh backup without the need to answer on-screen prompts.

RonR
Posts: 514
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Tue Oct 08, 2019 9:31 am

image-utils.zip (in the first post of this topic) has been updated.

Backups created with image-backup now auto expand the root filesysystem on the first boot. Manual resizing with raspi-config is no longer required regardless of whether the backup is restored to an SD card or a USB device.

itsmedoofer
Posts: 359
Joined: Wed Sep 25, 2013 8:43 am

Re: Image File Utilities

Tue Oct 08, 2019 10:16 am

Thanks Ron, appreciated!

Return to “Advanced users”