Image File Utilities

[RonR]

Attached are several utilities I created and use to manipulate image files.


image-backup:

image-backup creates a backup of a running Raspbian system to a standard 'raw' image file that can be written to an SD card or a USB device card with Etcher, imageUSB, etc. It will also perform incremental updates to an existing backup image file.

Running image-backup with no parameters will create a full backup. To create the smallest possible image, specify an Image ROOT filesystem size of 0 to determine the minimum allowed size. If you plan to incrementally update the image file, specify a considerably larger size to allow for additional growth.

Running image-backup with a parameter of an existing image filename will incrementally update that image file.


Image-check:

image-check will check the integrity of a standard 'raw' image file. Usage is:

image-check imagefile [W95|Linux]

where W95 checks the BOOT partition and Linux checks the ROOT partition. If neither is specified, Linux is assumed.


image-compare:

image-compare compares a running Raspbian system to an existing standard 'raw' image file and displays the incremental changes that image-backup would perform if run. Usage is:

image-compare [imagefile]


image-mount:

image-mount mounts a standard 'raw' image file to allow it to be read or written as if it were a device. Usage is:

image-mount imagefile mountpoint [W95|Linux]

where W95 mounts the BOOT partition and Linux mounts the ROOT partition. If neither is specified, Linux is assumed.


image-set-ptuuid:

image-set-ptuuid sets the Partition Table UUID value of a standard 'raw' image file. Usage is:

image-set-ptuuid imagefile ptuuid

where ptuuid is 8 hex digits


image-shrink:

image-shrink shrinks a standard 'raw' image file to its smallest possible size (plus an optional additional amount of free space). Usage is:

image-shrink imagefile [Additional MB]

where Additional MB is an additional amount of free space to be added.

[rin67630]

Super! very useful applets!

Could you please tell me where to put the unzipped files? Just in /home/pi)?
Are they executable directly or must we first chmod them?

A detailed noob-proof description of what to do would really be useful.

Thank you!

[RonR]

Could you please tell me where to put the unzipped files? Just in /home/pi)?
Are they executable directly or must we first chmod them?

They're not picky about where you place them (/home/pi would be fine). You must make them executable (chmod +x ...).

A detailed noob-proof description of what to do would really be useful.

I'll try to write up a few examples shortly.

[itsmedoofer]

Again, fantastic work, really appreciated !

[scruss]

They're not picky about where you place them (/home/pi would be fine). You must make them executable (chmod +x ...).

They could also go in ~/bin (that's /home/pi/bin). Raspbian automatically adds ~/bin to your path at login if the folder exists.

Code: Select all

mkdir ~/bin

is all people need to create it. The system will add it to your path the next time you login or reboot.

RonR — thanks very much for writing these. I see that some older links have expired and people can't seem to find them, so I created a github repo of your code as of 2019-08-01: RonR-RaspberryPi-image-utils. I hope that's okay with you.

I've also stored a copy of the attachment on the Internet Archive so it should stick around better.

[bls]

image-mount is particularly sweet! Takes away all the hassle of mounting an SD card image. Thanks!

[RonR]

image-mount is particularly sweet! Takes away all the hassle of mounting an SD card image. Thanks!

Thanks for the feedback.

I use image-mount every time a new Raspbian image is released to inject an ssh and/or wpa_supplicant.conf file into the image file. Then I don't have to manually add them to the SD card or USB device each time I write one. It's also handy for extracting files from an image file created by image-backup.

[bls]

image-mount is particularly sweet! Takes away all the hassle of mounting an SD card image. Thanks!

Thanks for the feedback.

I use image-mount every time a new Raspbian image is released to inject an ssh and/or wpa_supplicant.conf file into the image file. Then I don't have to manually add them to the SD card or USB device each time I write one. It's also handy for extracting files from an image file created by image-backup.

That's great! I never thought of that approach. Mine is slightly different. I have a script that adds all my stuff (scripts, config files, etc) to a newly-built SD card. I then boot it and install the packages I want. At that point I dd that beautiful SD card to an IMG file and use it for building new 'virgin' SD cards, as needed. image-mount lets me make certain classes of updates to my SD "golden master" without having to go through a complete rebuild.

[Botspot]

These are brilliant! I have used portions of image-mount in my bash script to let it mount and boot a Raspbian disk image using systemd-nspawn.

Bug report:
image-mount does not accept disk images located in a directory that contains the space character. (And yes, I tried using quotation marks.)

Code: Select all

pi@raspberrypi:~ $ sudo /home/pi/image-utils/image-mount "/home/pi/Web drives/DMA Google Drive/image files/2019-06-20-raspbian-buster-full.img" /media/pi/fs Linux

Mount /home/pi/Web drives/DMA Google Drive/image files/2019-06-20-raspbian-buster-full.img on /media/pi/fs (y/n)? y

/home/pi/image-utils/image-mount: line 66: drives/DMA * 512: division by 0 (error token is "DMA * 512")

Unable to create loop device

[scruss]

Yeah, it doesn't quite get the quoting right around starting round about line 60.

Of course, no-one I know would admit the creating or owning any file system with spaces in the name …
(joke, in case it was too obtuse.)

[RonR]

The problem was totally unrelated to quoting. The problem was due to using fdisk to display disk parameters and awk to parse them. The addition of spaces in path/file names caused awk to grab the wrong fields. The solution was to switch to sfdisk and sed.

An updated image-utils.zip file is now available in the first post of this topic which works properly with spaces in path/file names.

Thanks Botspot for reporting this.

[Milliways]

I had used image-mount: to view a downloaded image and tried to use image-backup to perform a backup.

The documentation is incredibly brief, there is no help and even examining the script didn't help much - it is devoid of any comments.

I have a mounted drive (in /mnt/PiHDD) which I wanted to use for the backup.

I have tried all the following (and more)

Code: Select all

sudo ./image-utils/image-backup /mnt/PiHDD/BusterBackup
/mnt/PiHDD/BusterBackup not found

sudo ./image-utils/image-backup /mnt/PiHDD/ BusterBackup
/mnt/PiHDD/ is a directory

cd /mnt/PiHDD
sudo /home/pi/image-utils/image-backup BusterBackup
BusterBackup does not begin with /mnt/ or /media/

How, on earth, do you perform a backup?

[RonR]

How, on earth, do you perform a backup?

"Running image-backup with no parameters will create a full backup."

Code: Select all

sudo image-backup

"Running image-backup with a parameter of an existing image filename will incrementally update that image file."

Code: Select all

sudo image-backup image_file_name

You must start with a full backup and then you can incrementally update it if you wish.

[jcyr]

How hard would it be to make this work for hybrid boot (/boot on SD, / on USB) with backup to CIFS share?

[RonR]

How hard would it be to make this work for hybrid boot (/boot on SD, / on USB) with backup to CIFS share?

That already works.

[Botspot]

Thanks so much RonR for these powerful scripts!!

I am using image-mount in my vdesktop script, it allows me to mount a Stretch image on a Buster OS, then boot that as a VM in a VNC-style window!
Edit: shorten link

[Milliways]

deleted

[Botspot]

deleted

Remember we want information...No information no help. Could you clarify "deleted"?

[Milliways]

deleted

Remember we want information...No information no help. Could you clarify "deleted"?

Fix the Forum so that posts can be deleted.

[Botspot]

Fix the Forum so that posts can be deleted.

Never done this personally, but I think you can flag your own post and tell the moderator you want it deleted.

[jcyr]

Fix the Forum so that posts can be deleted.

Never done this personally, but I think you can flag your own post and tell the moderator you want it deleted.

Hmmm... I've never had any issues deleting my own posts! That's what the garbage can icon does...

[rpdom]

Fix the Forum so that posts can be deleted.

Never done this personally, but I think you can flag your own post and tell the moderator you want it deleted.

Hmmm... I've never had any issues deleting my own posts! That's what the garbage can icon does...

You can delete them until someone makes another reply following it.

[Milliways]

I have run the backup, but before copying the image to a SD Card decided to check it out.

I noticed that directories under /mnt were NOT copied. I have a number of directories which my fstab assumes exist, but they were missing.
This is not a major problem but is easily fixed.

I changed the rsync to copy the directories, but not contents with an exclude /mnt/*/*

Code: Select all

rsync -aDH --partial --numeric-ids --delete --force --exclude "${MNTPATH}" --exclude '/dev' --exclude '/media' --exclude '/mnt/*/*' --exclude '/proc' --exclude '/run' --exclude '/sys' \

This looks like a great script.
It makes more compact backups than dd and these compress to a much smaller .gz file because that omit a lot of transient rubbish.

Unfortunately the "documentation" is extremely brief. Even understanding the script is difficult as is totally devoid of comments.

I have commented my version and added a message to let you know it has started a backup update.

I will probably update my copy to facilitate making a fresh backup without the need to answer on-screen prompts.

[RonR]

image-utils.zip (in the first post of this topic) has been updated.

Backups created with image-backup now auto expand the root filesysystem on the first boot. Manual resizing with raspi-config is no longer required regardless of whether the backup is restored to an SD card or a USB device.

[itsmedoofer]

Thanks Ron, appreciated!