User avatar
ksharindam
Posts: 143
Joined: Sat Jan 09, 2016 4:16 pm

why xz format not used for distribution of raspbian images

Sun Aug 25, 2019 4:23 pm

we know that xz format is more efficient compression method. but the raspbian images are distributed in zip compressed image.

If xz is used at extreme compression level (level 9), much disk space and bandwidth can be saved.
That means Raspbian buster desktop image without recommend softwares, will take around 700mb space instead of 1GB.
So , why zip archive is used instead of xz archive?

User avatar
topguy
Posts: 5967
Joined: Tue Oct 09, 2012 11:46 am
Location: Trondheim, Norway

Re: why xz format not used for distribution of raspbian images

Sun Aug 25, 2019 4:28 pm

Because very few people actually knows what xz is.

User avatar
ksharindam
Posts: 143
Joined: Sat Jan 09, 2016 4:16 pm

Re: why xz format not used for distribution of raspbian images

Sun Aug 25, 2019 4:35 pm

If someone knows how to write a disk image, he should know what xz is.

DirkS
Posts: 10018
Joined: Tue Jun 19, 2012 9:46 pm
Location: Essex, UK

Re: why xz format not used for distribution of raspbian images

Sun Aug 25, 2019 4:40 pm

ksharindam wrote:
Sun Aug 25, 2019 4:35 pm
If someone knows how to write a disk image, he should know what xz is.
The problem is on the user side. Too many users would have no idea what to do with an xz file. Better to use a format that is more common.

gordon77
Posts: 4311
Joined: Sun Aug 05, 2012 3:12 pm

Re: why xz format not used for distribution of raspbian images

Sun Aug 25, 2019 5:41 pm

It's probably only a small issue but if using a windows 10 pc to decompress it will work with a zip but you need 3rd party software eg 7zip for xz.

bls
Posts: 292
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA

Re: why xz format not used for distribution of raspbian images

Sun Aug 25, 2019 6:09 pm

There are a lot of tools (scripts) that have been built to manage the current zip format. Changing to xz would require them all to be changed, and create a lot of annoyance.

That said, it would be interesting to know whether such a change would really be worth the annoyance and aggravation.

Has anyone spent the cycles to test xz vs zip for this specific use?

User avatar
rpdom
Posts: 15606
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: why xz format not used for distribution of raspbian images

Sun Aug 25, 2019 6:20 pm

How many Windows users know what to do with an xz file?

Some form of unzip is commonly used with Windows (although the older versions can't handle the large file format).

The recommended Etcher software can handle a zip file seamlessly to the user. I'm not sure if it supports xz.

I download the most recent images and then recompress them with xz for local storage. I have a bash script on my Linux laptop (Raspberry Pi Desktop with Debian) that lists images for me to select, then decompresses them to an SD card or USB device and expands the filesystem to fit the media before disabling the "resize on first boot" thing and setting a few things like hostname, ssh, wpa_supplicant and certain ssh keys.

xz takes a little bit more work to decompress and a lot more CPU time to compress.

hippy
Posts: 6293
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: why xz format not used for distribution of raspbian images

Sun Aug 25, 2019 11:09 pm

I don't think it should be a case of .zip or .xz, but better to provide both.

There would only be a one-time overhead for building both, so it shouldn't be too onerous to do that. Then all users get to choose which they prefer, whichever suits them best.

If there is only going to be one format then it pragmatically has to be .zip

User avatar
ksharindam
Posts: 143
Joined: Sat Jan 09, 2016 4:16 pm

Re: why xz format not used for distribution of raspbian images

Mon Aug 26, 2019 5:31 am

hippy wrote:
Sun Aug 25, 2019 11:09 pm
I don't think it should be a case of .zip or .xz, but better to provide both.

There would only be a one-time overhead for building both, so it shouldn't be too onerous to do that. Then all users get to choose which they prefer, whichever suits them best.

If there is only going to be one format then it pragmatically has to be .zip
Yes , I also think that it is better to provide both. zip for those who dont know about xz, and xz for others.
For the first time users, NOOBS is already provided, that does not require another computer to write image.

User avatar
ksharindam
Posts: 143
Joined: Sat Jan 09, 2016 4:16 pm

Re: why xz format not used for distribution of raspbian images

Mon Aug 26, 2019 5:39 am

rpdom wrote:
Sun Aug 25, 2019 6:20 pm
xz takes a little bit more work to decompress and a lot more CPU time to compress.
time is not a factor, when it is compressed server side. A pi 2 takes around 5 hours to compress, and pi 4 will take around 1 hour to compress.
In the server, x86 computern will be used to compress, which will take much less time.

In case of decompression, the raspberry pi is faster enough to decompress at a speed higher than writing speed of SD card.

User avatar
rpdom
Posts: 15606
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: why xz format not used for distribution of raspbian images

Mon Aug 26, 2019 7:42 am

ksharindam wrote:
Mon Aug 26, 2019 5:39 am
rpdom wrote:
Sun Aug 25, 2019 6:20 pm
xz takes a little bit more work to decompress and a lot more CPU time to compress.
time is not a factor, when it is compressed server side. A pi 2 takes around 5 hours to compress, and pi 4 will take around 1 hour to compress.
In the server, x86 computern will be used to compress, which will take much less time.
What makes you think the Raspberry Pi people are using x86 based servers? (Hint: This forum runs on a load of 4Bs).

Also, time does come in to it when you want to release a new image in a hurry and you now have to generate eight compressed files in total and upload them before they can be announced.

User avatar
bensimmo
Posts: 4187
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: why xz format not used for distribution of raspbian images

Mon Aug 26, 2019 8:18 am

Is it really worth it, I've been with computers since the 80s, used and tinkered with Linux since the mid 90s and left for the easy world after that harsh time.i still don't really know what xz is.
Stick with Zip, it's easy for anyone to handle.

We have Windows PCs, MacOS, iOS, Android, Chromebooks and the multitude of Linux based OSs to use.

Keep it easy.

The torrent makes downloading easy, I understand it, others don't even know they are using it. It's quite seamless.

You could recompress, host a torrent for others.

But it's not up to me, as long as there is a simple drag and drop method for people with Etcher or similar, all is fine.

User avatar
bensimmo
Posts: 4187
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: why xz format not used for distribution of raspbian images

Mon Aug 26, 2019 8:20 am

rpdom, I thought the forum was on a midrange x86 server after the forum updates we had a while back.
I also though the rest of the site was on the Pi3s still other than a proof of concept at launch, that is until network booting works properly for mythic to use them.

User avatar
rpdom
Posts: 15606
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: why xz format not used for distribution of raspbian images

Mon Aug 26, 2019 9:05 am

bensimmo wrote:
Mon Aug 26, 2019 8:20 am
rpdom, I thought the forum was on a midrange x86 server after the forum updates we had a while back.
I also though the rest of the site was on the Pi3s still other than a proof of concept at launch, that is until network booting works properly for mythic to use them.
You could be right.

User avatar
bensimmo
Posts: 4187
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: why xz format not used for distribution of raspbian images

Mon Aug 26, 2019 5:25 pm

rpdom wrote:
Mon Aug 26, 2019 9:05 am
bensimmo wrote:
Mon Aug 26, 2019 8:20 am
rpdom, I thought the forum was on a midrange x86 server after the forum updates we had a while back.
I also though the rest of the site was on the Pi3s still other than a proof of concept at launch, that is until network booting works properly for mythic to use them.
You could be right.
Found the interesting website blog about the Pi4 at release/pi website.
https://blog.mythic-beasts.com/2019/06/ ... pberry-pi/

The forum server info is somewhere in the forum if anyone really wanted to know, but it has little to do with some file compression. :-)

I'm assuming the extra file compression is for people on low data usage connections.

Would a Lite image and then using apt to get the other parts use less 'internet' data?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24188
Joined: Sat Jul 30, 2011 7:41 pm

Re: why xz format not used for distribution of raspbian images

Mon Aug 26, 2019 7:29 pm

Just out of interest, anyone got numbers for the latest Raspbian as a zip and as xz?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

User avatar
DougieLawson
Posts: 36578
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: why xz format not used for distribution of raspbian images

Mon Aug 26, 2019 7:39 pm

jamesh wrote:
Mon Aug 26, 2019 7:29 pm
Just out of interest, anyone got numbers for the latest Raspbian as a zip and as xz?
It depends on the compression level used for xz.

https://catchchallenger.first-world.inf ... LZ4_vs_LZO (sorry that page doesn't consider plain old (crappy) zip)
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

User avatar
graysky
Posts: 93
Joined: Fri Apr 05, 2013 11:43 am
Location: /run/user/1000
Contact: Website

Re: why xz format not used for distribution of raspbian images

Mon Aug 26, 2019 7:50 pm

jamesh wrote:
Mon Aug 26, 2019 7:29 pm
Just out of interest, anyone got numbers for the latest Raspbian as a zip and as xz?
Meaning the compression differences? As an aside, if a format switch is considered, I would recommend zst which benefits from multi-threaded compression, and compression that will arrive at the same checksum regardless of threads. Compression difference vs xz can be all over the board (some better some worse). Here is a piece of a larger discussion about pros/cons.

FYI on sizes and times (worse on size in this case but better on time):

Code: Select all

-rw-r--r-- 1 facade users 263337912 Jul  9 20:21 2019-07-10-raspbian-buster-lite.img.xz
-rw-r--r-- 1 facade users 326830889 Aug 26 15:40 2019-07-10-raspbian-buster-lite.img.zst
-rw-r--r-- 1 facade users 426250971 Aug 26 15:37 2019-07-10-raspbian-buster-lite.zip

### zstd -c -T0 -q -18 - <2019-07-10-raspbian-buster-lite.img >2019-07-10-raspbian-buster-lite.img.zst took 1:33.21 min
### xz -9 2019-07-10-raspbian-buster-lite.img took 7:12.29 min

User avatar
Imperf3kt
Posts: 2986
Joined: Tue Jun 20, 2017 12:16 am
Location: Australia

Re: why xz format not used for distribution of raspbian images

Mon Aug 26, 2019 11:42 pm

I much prefer 7z myself.
In my own experience, it's popularity is second only to the proprietary rar format and it supports files >4GB across all versions of itself.
Most zipping tools can uncompress it and on Windows you can install a handy shell extension so it's accessible directly from the right click context menu.

I've never heard of xz
55:55:44:44:4C
52:4C:52:42:41

smoki
Posts: 3
Joined: Mon Aug 19, 2019 2:05 pm

Re: why xz format not used for distribution of raspbian images

Tue Aug 27, 2019 2:20 am

Even Linux kernel releases are XZ compressed default since 6+ years ago:

https://www.kernel.org/xz-by-default-and-json.html

But i guess people never heard of it as they prefer git checkout :lol:

Joke a side xz is pretty much standardized on Linux.

Now 'xz -9e' would likely produce best results, let alone slowness but it also eats memory to do that. If requirement is lowest common denominator, so reproducability and de/compression on Pi Zero without swaping, then better forget about these extreme levels of it :D

User avatar
ksharindam
Posts: 143
Joined: Sat Jan 09, 2016 4:16 pm

Re: why xz format not used for distribution of raspbian images

Tue Aug 27, 2019 4:14 am

jamesh wrote:
Mon Aug 26, 2019 7:29 pm
Just out of interest, anyone got numbers for the latest Raspbian as a zip and as xz?
Currently I am compressing The buster image , I will post the numbers when it will complete.

Milliways
Posts: 478
Joined: Fri Apr 25, 2014 12:18 am
Location: Sydney, Australia

Re: why xz format not used for distribution of raspbian images

Tue Aug 27, 2019 5:14 am

ksharindam wrote:
Sun Aug 25, 2019 4:23 pm
we know that xz format is more efficient compression method. but the raspbian images are distributed in zip compressed image.

If xz is used at extreme compression level (level 9), much disk space and bandwidth can be saved.
That means Raspbian buster desktop image without recommend softwares, will take around 700mb space instead of 1GB.
So , why zip archive is used instead of xz archive?
I download an image every couple of years to do a fresh install (OK 2 lite & Desktop) I don't care about a few hundred kB or a minute or so download time.

I use gz for backup images.

User avatar
DougieLawson
Posts: 36578
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: why xz format not used for distribution of raspbian images

Tue Aug 27, 2019 5:53 am

I unzipped plain Raspbian Buster

Code: Select all

-rw-r--r-- 1  117  126 1152275926 Jul 13 10:36 2019-07-10-raspbian-buster.zip
to

Code: Select all

-rw-r--r-- 1 root root 3779067904 Jul 10 01:27 2019-07-10-raspbian-buster.img
xz compressed that with -9 and the out of memory killer shot it dead.

xz compressed with -0 and got

Code: Select all

-rw-r--r-- 1 root root  994290944 Jul 10 01:27 2019-07-10-raspbian-buster.img.-0.xz
xz compressed with -6 (default value) and got

Code: Select all

-rw-r--r-- 1 root root  825458420 Jul 10 01:27 2019-07-10-raspbian-buster.img.xz
So we'd save some network bytes. I've not looked at how long it takes to unzip or uncompress as the files are on a NAS and I/O time isn't ideal.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

User avatar
ksharindam
Posts: 143
Joined: Sat Jan 09, 2016 4:16 pm

Re: why xz format not used for distribution of raspbian images

Tue Aug 27, 2019 5:57 am

jamesh wrote:
Mon Aug 26, 2019 7:29 pm
Just out of interest, anyone got numbers for the latest Raspbian as a zip and as xz?
Just compressed the Raspbian Desktop image.

Code: Select all

xz -9e 2019-07-10-raspbian-buster.img.tar
Size of...
Uncompressed image -> 3.5GB
zip compressed image -> 1.1 GB
tar.xz compressed image -> 732.2MB

User avatar
ksharindam
Posts: 143
Joined: Sat Jan 09, 2016 4:16 pm

Re: why xz format not used for distribution of raspbian images

Tue Aug 27, 2019 6:03 am

DougieLawson wrote:
Tue Aug 27, 2019 5:53 am
xz compressed that with -9 and the out of memory killer shot it dead.
How much RAM does your computer have? Is it a Raspberry Pi?
I have compressed the image using level 9 in a Pi4 2GB. Also compressed in a Pi2 two years ago.
never encountered out of memory error.

Return to “Raspbian”