wipe unused space on SDcard?

[Bosse_B]

I have goofed when installing a new Raspbian Wheezy....
I had to revert to Wheezy after trying Jessie on a brand new 32 GB SDcard.
So I took a used 32 GB Sdcard (had previously been used in my Android phone and was full of images and movies). Onto this I imaged the Raspbian Wheezy 2015-05-05 distribution and then did all of my usual setup things including a very lengthy process of installing Free-Pascal and Lazarus and tightvncserver.
Then I wanted to make a backup image of this working installation using Win32DiskImager.
Surprise! The img file turned out to be 30 GB in size and it would not compress either. I had expected something like 4 GB. In order to use the new card I then wrote the image onto the Jessie SDcard and continued from there.
Now I realize that the image contains the data from old movies and images from my phone in the unallocated area and therefore the image does not compress.
And I have written this crap to the new SDcard as well...

So is there a way to actually zero out all of the unused blocks on the SDcard or on the image file?
Best something running on the live file system in the Pi or else on the image file in Windows7.

I have googled and found a utility named zerofree, but that is apparently used in Linux on a file system that is mounted read-only, which is not what I can do here.

EDIT:
I found another page with a solution, but I don't know if it is the right way...
Anyway from his post I created this file using /dev/zero instead of /dev/urandom as input and /dev/root as filesystem:

Code: Select all

#!/bin/bash
# Source:
# http://www.linuxquestions.org/questions/linux-software-2/a-faster-way-to-wipe-free-space-701435/#post3514139
# Author: erikalm
#
# This script wipes the unused area of the disk by writing 
# a file full of zeros until disk space is used up, then
# it erases said file to return the sectors to the system
# Enter the filesystems to process in the FSLIST variable
# Separate each with a space.
#
FSLIST="/dev/root"
SCRATCH_FILE="DELETE_ME"

for FS in $FSLIST; do
	set +e +u;
	dd if=/dev/zero of="${FS}/${SCRATCH_FILE}";
	sync; sync;
	rm "${FS}/${SCRATCH_FILE}";
	sync; sync;
done

Could it be safely used and is it quick? It seems like other posts says that dd needs a bs parameter to increase the block size in order to speed up...

[Bosse_B]

I ended up creating my own shellscript file to run the wipe of the unused disk area:

Code: Select all

#!/bin/bash
echo "------- Start zeroing at $(date)-------"
dd if=/dev/zero of=/home/pi/delete_me
echo "------- Done  zeroing at $(date)-------"
sync
sync
echo "------- Delete dummy file -------"
rm -f /home/pi/delete_me
sync
sync
echo "-------Finish zeroing at $(date)-------"

Then when I ran this as su it processed my disk in 50 minutes (process speed 8.6 MB/s)
Now the image file created by Win32DiskImager is still 30 GB, but it compresses down to 2.8 GB using WinZip!
A lot better than wasting 30 GB of disk space!

[kraigo]

Thanks for that - I'll give it a try.

I'd love it if there was a daemon that ran in the background and zero'd out free space when the system was largely idle.

KO

[kraigo]

That worked brilliantly. Thanks a LOT!

KO

[broe23]

You do not want to write to the unused space on the card. Your partition should extend across the whole SD card. Continuing to wipe the empty space will shorten the life of the card.

[VincentiuCelMare]

I ended up creating my own shellscript file to run the wipe of the unused disk area:

Code: Select all

#!/bin/bash
echo "------- Start zeroing at $(date)-------"
dd if=/dev/zero of=/home/pi/delete_me
echo "------- Done  zeroing at $(date)-------"
sync
sync
echo "------- Delete dummy file -------"
rm -f /home/pi/delete_me
sync
sync
echo "-------Finish zeroing at $(date)-------"

Then when I ran this as su it processed my disk in 50 minutes (process speed 8.6 MB/s)
Now the image file created by Win32DiskImager is still 30 GB, but it compresses down to 2.8 GB using WinZip!
A lot better than wasting 30 GB of disk space!

Hello,

Congratulations with your script ! It's KISS like I love it. I use it for my Raspberry Pi backups (cq images).
It helped me to make the tarred version of the img file of the RPI SD DRASTICALLY smaller !

However some Q's:
After filling up the root dir, you give 2 sync commands.
Q's:
a) why 2 times ? just to make sure :p ?
but especially:
b) said sync command froze the script (and basically all my pi) and sent CPU usage to the moon (50 at a rare moment I could check on it on my basically frozen pi). I killed your script (which wasn't easy either on a basically frozen pi) after a while, even though it might have come to a good end if I waited longer ??? Idk, would it ?
After killing it I deleted the file your script created (like your script would have done after the sync commands) and that brought everything back to normal. After the file was deleted, giving the sync command worked perfectly and instantly.

Basically my Q is: Why executing the sync command after filling up the root dir to the notch ??? I'm not and expert on this, but doesn't the sync command write RAM to the root drive (which is completely full at the time you give the command) ??? So isn't it trying to do something impossible causing a freeze of the system ?

so:
c) wouldn't it be better deleting the file immediately after it's fully created, and performing a sync AFTER it's deleted ? So basically just deleting the 2 first sync commands from your script ?

or

d) AM I MISSING STH )) ?

Thanks in advance if you or someone else can/wants to take time to reply .

[VincentiuCelMare]

I ended up creating my own shellscript file to run the wipe of the unused disk area:

Code: Select all

#!/bin/bash
echo "------- Start zeroing at $(date)-------"
dd if=/dev/zero of=/home/pi/delete_me
echo "------- Done  zeroing at $(date)-------"
sync
sync
echo "------- Delete dummy file -------"
rm -f /home/pi/delete_me
sync
sync
echo "-------Finish zeroing at $(date)-------"

Then when I ran this as su it processed my disk in 50 minutes (process speed 8.6 MB/s)
Now the image file created by Win32DiskImager is still 30 GB, but it compresses down to 2.8 GB using WinZip!
A lot better than wasting 30 GB of disk space!

Hello,

Congratulations with your script ! It's KISS like I love it. I use it for my Raspberry Pi backups (cq images).
It helped me to make the tarred version of the img file of the RPI SD DRASTICALLY smaller !

However some Q's:
After filling up the root dir, you give 2 sync commands.
Q's:
a) why 2 times ? just to make sure :p ?
but especially:
b) said sync command froze the script (and basically all my pi) and sent CPU usage to the moon (50 at a rare moment I could check on it on my basically frozen pi). I killed your script (which wasn't easy either on a basically frozen pi) after a while, even though it might have come to a good end if I waited longer ??? Idk, would it ?
After killing it I deleted the file your script created (like your script would have done after the sync commands) and that brought everything back to normal. After the file was deleted, giving the sync command worked perfectly and instantly.

Basically my Q is: Why executing the sync command after filling up the root dir to the notch ??? I'm not and expert on this, but doesn't the sync command write RAM to the root drive (which is completely full at the time you give the command) ??? So isn't it trying to do something impossible causing a freeze of the system ?

so:
c) wouldn't it be better deleting the file immediately after it's fully created, and performing a sync AFTER it's deleted ? So basically just deleting the 2 first sync commands from your script ?

or

d) AM I MISSING STH )) ?

Thanks in advance if you or someone else can/wants to take time to reply .

Anyone ? Is the sync before deleting the file really necessary ?

[Kendek]

Anyone ? Is the sync before deleting the file really necessary ?

The sync command is required (but only once) to fully write the buffered data to the SD card.
Instead you can use the fstrim -av command to TRIM the unallocated space.

[rpdom]

Why executing the sync command after filling up the root dir to the notch ??? I'm not and expert on this, but doesn't the sync command write RAM to the root drive (which is completely full at the time you give the command) ??? So isn't it trying to do something impossible causing a freeze of the system ?

wouldn't it be better deleting the file immediately after it's fully created, and performing a sync AFTER it's deleted ? So basically just deleting the 2 first sync commands from your script ?

Yes, the sync is necessary. It will take some time to complete. If you don't do that before deleting the file, the file may not have been written to the card fully.

The writes are buffered and cached in RAM before writing to the card. The root drive may be shown as full, but some of the data in it hasn't been written at that time. The space on the card has been reserved for the file even though it is still sitting in the buffer.

Deleting the file before doing the sync would deallocate the used space - even though it hasn't been written to yet. The old contents could still be there!