Anyone tried ZFS on Pi 4 or planning to do so?

[JulianFP]

As in the title, has anyone tried or is close to trying ZFS on a Pi 4? I know it's resource hungry but at least in my case I want to have a go once my pi 4 4GB arrives and with Ubuntu looking to offer it even as a root file system option ZFS does look to be taking a significant step forward in its reach.

I use my Pi as a Kodi box and also as a very low traffic NAS to act as one of quite a number of backup target for the backups from my main PC where the total volume of write traffic is usually less than 100MB a day (a few incremental changes on spreadsheets and perhaps a few document scans) and the total backup archive size is currently about 0.6TB so for me it's the bit scrubbing and other data integrity stuff that is most important.

Right now my Pi3B+ runs my NAS just fine using a pair of 2TB USB drives formatted as ext4 and mirrored using mdadm but I do hanker after the extra data integrity features that zfs would give me since it's sole use is as a backup target. For my use case I think I would technically even be within ZFS memory resource guidelines which I have read are recommended to be 1GB RAM + an extra 1GB per 1TB of storage so, pessimistically assuming 1GB for OS and other stuff, that shouldn't be presenting problems at least in available RAM terms for a 2TB array which is what I have and, as mentioned above, my performance requirements are extremely modest.

[MarkJ62]

I run ZFS on a Debian x64 machine. The first issue will be that it requires 64 bit so that means aarch64 architecture. You should be fine with 2TB drives in a mirror using a 4GB Pi4. It only uses up to half the available memory when you start giving it a lot of work to do and it frees it again when its quiet. You won’t be able to turn on deduplication as that does require a lot of memory.

[BeauSlim]

Trying ZFS on the Pi4 is one of the reasons I grabbed the 4GB version. Mark is probably correct that it will require 64 bit.

There are still firmware issues (USB booting) and OS support (uasp bugs, 1GB RAM limitations, etc.) isn't there yet, so I'll probably wait a while before bothering.

EDIT: I gave a few 64-bit distros (gentoo, Ubuntu 64, Ubuntu Core18) a spin today. The gentoo build looks like it might be capable of running ZFS. It had no problem with my USB SATA devices and recognizes the full 4GB of the Pi4 (unlike Ubuntu 64).

A custom kernel (or at least a module) is required and I don't have time right now to learn gentoo's mid-90's style of build-from-source package admin, but if someone is interested, https://github.com/sakaki-/gentoo-on-rpi-64bit is probably the best place to start.

[colettace]

A few of us have been trying to compile it from scratch but we're running into issues with the makefiles and such. Here's the thread, scroll all the way down for the part about compiling on rpi4. Any help is appreciated!

[cyclic]

I have zfs installed on pi4 and testing with 2 ssd mirror for 2 weeks.
It seems to work. Have had 200GB plus read/write.
There is a problem with reboot or cold start whereby the ssd come up both "FAULTED".
However next cold start may come up OK and online.
Does anyone have USB3 problems intermittently on start or restart?

[BeauSlim]

cyclic, I'd like to hear more details about your setup. Are you running Raspbian? 32 or 64-bit? What USB enclosures are you using for your SSDs?

I found setting up ZFS to be relatively easy on the 64-bit Ubuntu Server images for Pi4 from https://github.com/TheRemote/Ubuntu-Ser ... l/releases and with a couple hints from https://github.com/TheRemote/Ubuntu-Ser ... /issues/12

Using ZFS as a backing store for docker works great, but I'm finding local write speeds to a simple ZFS volume on a Samsung 750 EVO in a JMicron 567-based USB3 enclosure to be a little slow at around 75 MB/s. Remote NFS writes to this same share drop to about 50MB/s. Recompiling the kernel to support UASP doesn't seem to make much difference to ZFS performance.

[Thinkcat]

To just run ZFS, I doubt 64 bits or enormous amount of memory is strictly necessary. I have a 10 year old Atom-based netbook that is i386 and had only 1 GB of memory when I installed lubuntu into it onto a ZFS root volume.

Atom laptops will get stuck while doing compiles and such unless one tunes a few kernel and ARC parameters before proceeding. One thing is the "vmalloc" kernel parameter. It should be maybe 512 MB in a 1 GB system. Another one is to set the maximum size of ARC clearly below that amount. To maybe 384 MB.

I haven't tried ZFS in a Pi yet. But I do plan to. Soon.

[Thinkcat]

I have compiled the 0.8.2 version from Github according to these instructions, but now I don't know how to create an initramfs. There are packages for 0.7.6 that I can see, but I'd rather not uninstall 0.8.2 and go to an older version. Any pointers?

Otherwise it works perfectly. I just followed manually what's in that script in my link.

[RandyOo]

I have compiled the 0.8.2 version from Github according to these instructions, but now I don't know how to create an initramfs. There are packages for 0.7.6 that I can see, but I'd rather not uninstall 0.8.2 and go to an older version. Any pointers?

Otherwise it works perfectly. I just followed manually what's in that script in my link.

@Thinkcat , your link is giving a 404 error, unfortunately.

I did manage to get zfs compiled and running based on the official build instructions , followed by a

Code: Select all

make install

, but then encountered this error when trying to execute the zfs command:

Code: Select all

zfs: error while loading shared libraries: libnvpair.so.1: cannot open shared object file: No such file or directory

.

The solution was found here:

Code: Select all

ln -s /usr/local/lib/nvpair.a /usr/lib/nvpair.a
ln -s /usr/local/lib/libnvpair.la /usr/lib/libnvpair.a
ln -s /usr/local/lib/libnvpair.so /usr/lib/libnvpair.so
ln -s /usr/local/lib/libnvpair.so.1 /usr/lib/libnvpair.so.1
ln -s /usr/local/lib/libnvpair.so.1.0.1 /usr/lib/libnvpair.so.1.0.1
ln -s /usr/local/lib/libuutil.a /usr/lib/libuutil.a
ln -s /usr/local/lib/libuutil.la /usr/lib/libuutil.la
ln -s /usr/local/lib/libuutil.so /usr/lib/libuutil.so
ln -s /usr/local/lib/libuutil.so.1 /usr/lib/libuutil.so.1
ln -s /usr/local/lib/libuutil.so.1.0.1 /usr/lib/libuutil.so.1.0.1
ln -s /usr/local/lib/libzfs.a /usr/lib/libzfs.a
ln -s /usr/local/lib/libzfs_core.a /usr/lib/libzfs_core.a
ln -s /usr/local/lib/libzfs_core.la /usr/lib/libzfs_core.la
ln -s /usr/local/lib/libzfs_core.so /usr/lib/libzfs_core.so
ln -s /usr/local/lib/libzfs_core.so.1 /usr/lib/libzfs_core.so.1
ln -s /usr/local/lib/libzfs_core.so.1.0.0 /usr/lib/libzfs_core.so.1.0.0
ln -s /usr/local/lib/libzfs.la /usr/lib/libzfs.la
ln -s /usr/local/lib/libzfs.so /usr/lib/libzfs.so
ln -s /usr/local/lib/libzfs.so.2 /usr/lib/libzfs.so.2
ln -s /usr/local/lib/libzfs.so.2.0.0 /usr/lib/libzfs.so.2.0.0

It does appear to be functional, thus far.

[Thinkcat]

@Thinkcat , your link is giving a 404 error, unfortunately.

Here now.

[RandyOo]

I just posted in the Pi4 64-bit kernel thread, but suppose it merits sharing here, too:
https://www.raspberrypi.org/forums/view ... 5#p1566275

TLDR: ZFS sorta works on the default kernel, but repeatedly triggered memory errors on a moderate load. With the 64-bit kernel, not only are the errors gone, but performance is much better, to boot!

@Thinkcat, thanks for the link. I haven't looked into ZFS on root, but that's not a big priority for me.

[andrum99]

I tried building ZFS on Raspbian and I get the following error:

Code: Select all

Making all in module
In file included from /home/pi/zfs/include/spl/sys/kstat.h:33,
                 from /home/pi/zfs/module/spl/spl-kstat.c:28:
/home/pi/zfs/module/spl/spl-kstat.c: In function ‘proc_kstat_open’:
/home/pi/zfs/include/spl/sys/proc.h:31:22: error: implicit declaration of function ‘PDE’ [-Werror=implicit-function-declaration]
 #define PDE_DATA(x) (PDE(x)->data)
                      ^~~
/home/pi/zfs/module/spl/spl-kstat.c:484:15: note: in expansion of macro ‘PDE_DATA’
  f->private = PDE_DATA(inode);
               ^~~~~~~~
/home/pi/zfs/include/spl/sys/proc.h:31:28: error: invalid type argument of ‘->’ (have ‘int’)
 #define PDE_DATA(x) (PDE(x)->data)
                            ^~
/home/pi/zfs/module/spl/spl-kstat.c:484:15: note: in expansion of macro ‘PDE_DATA’
  f->private = PDE_DATA(inode);
               ^~~~~~~~
cc1: some warnings being treated as errors
make[5]: *** [scripts/Makefile.build:303: /home/pi/zfs/module/spl/spl-kstat.o] Error 1
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [scripts/Makefile.build:544: /home/pi/zfs/module/spl] Error 2
make[4]: *** Waiting for unfinished jobs....

How do I fix this?