Seeking the illusive zero write

[lrhorer]

As well represented on the web, excessive writes to the SD card will shorten its life. I am creating an embedded application using the Raspberry Pi, and I definitely don't want a continuous stream of users complaining of dead units due to SD write failures. I believe I have undertaken the majority of measures to mitigate the issue, such as changing the log directories to tmpfs, etc. The next big data writer on the list is chromium-browser. The application is web based, so chromium is up 100% of the time, and it is doing lots of writes. Is there any way to limit or better yet eliminate writes from chromium? A native chromium solution would be great, although I could employ something like tmpfs and write to the volatile area(s) prior to invoking chromium. The main issue there is chromium uses nearly 100M of space for its files. Before embarking one something like this, I would want to trim the files as much as possible, and even then it would be much better if chromium didn't write to disk very much.

[JohnsUPS]

Have you considered a different storage medium such as a SSD? Size and/or power consumption may be an issue though...

[lrhorer]

Yes, but size and cost are the issues, plus even an SSD suffers from a limited number of writes, although much less so than an SD. There is only about 32mm of clearance between the USB ports and the side wall of the device. If someone makes an economical SSD that only sticks out about 32mm or so (like the tiny thumb drives), I would certainly consider it. I suppose I could also use a thumb drive and mount it to /home/pi/.config. That would eliminate the vast majority of writes (from chromium and LXDE) to the SD card. I don't think thumb drives are any more write-resilient than SD cards, are they? In fact, aren't they internally identical?

Power is not a significant issue, in terms of the current draw of an SSD, and the USB ports are unused.

[stuartiannaylor]

I created zram-conf that has various options for all of that.

As well as the normal zram swap in /etc/ztab you can assign directories to zram drives or create a zlog which is the same with some logrotate off zram to persistant directives.

Also has

Code: Select all

zram-config enable-ephemeral

that loads the whole root ro with zram upper

https://github.com/StuartIanNaylor/zram-config

With Chromium sometimes I just create a zlog and run `chromium-browser --user-data-dir=/var/log/chromium` just to save having a 2nd zdir
As it logs to that dir also if ` --enable-logging --v=1` is enabled

The default with zram-conf creates a demo zdir of the complete /home/pi dir.

If you are going to enable `zram-config enable-ephemeral` make sure any settings or changes are done before reboot as it is ephemeral on each boot `sudo zram-config disable-ephemeral` and reboot will get you back to normal.

If you use an overlayfs then its only the upper of the merge mount that is readwrite so the only files are ones written to or writing are in ram and fetched from lower by the CoW (copy on write) of the overlayfs.

Code: Select all

pi@raspberrypi:~ $ zramctl
NAME       ALGORITHM DISKSIZE  DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lz4           900M    4K   76B    4K       4 [SWAP]
/dev/zram1 lz4           300M 26.6M  4.5M    5M       4 /opt/zram/zram1
pi@raspberrypi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        30G  2.7G   26G  10% /
devtmpfs        460M     0  460M   0% /dev
tmpfs           464M  8.6M  456M   2% /dev/shm
tmpfs           464M   13M  452M   3% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
/dev/mmcblk0p1   43M   33M   11M  77% /boot
/dev/zram1      275M   22M  232M   9% /opt/zram/zram1
overlay1        275M   22M  232M   9% /home/pi
tmpfs            93M     0   93M   0% /run/user/1000

The complete /home/pi directory mount as the lower and upper only needs 5M with Chromium running as a zdir defined in /etc/ztab.

Code: Select all

pi@raspberrypi:~ $ zramctl
NAME       ALGORITHM DISKSIZE   DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lz4          1000M 138.5M 11.3M   12M       4 /rw
/dev/zram1 lz4           900M     4K   76B    4K       4 [SWAP]
pi@raspberrypi:~ $ df
Filesystem     1K-blocks    Used Available Use% Mounted on
devtmpfs          464676       0    464676   0% /dev
tmpfs              94948      64     94884   1% /mnt/run
/dev/mmcblk0p2  30570444 2806144  26472024  10% /ro
/dev/zram0        991512  130280    793648  15% /rw
overlayfs-root    991512  130280    793648  15% /
tmpfs             474724    8776    465948   2% /dev/shm
tmpfs             474724   18260    456464   4% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs             474724       0    474724   0% /sys/fs/cgroup
/dev/mmcblk0p1     43234   32912     10322  77% /boot
tmpfs              94944       0     94944   0% /run/user/1000

With sudo zram-config enable-ephemeral the whole system on start only occupies 12M of ram as with the compression of zram only writes are copied to upper.

[HawaiianPi]

Have you considered using a different OS?

Tiny Core Linux runs entirely from RAM, and there is a Pi version.

[Joel_Mckay]

If you use an overlayfs then its only the upper of the merge mount that is readwrite so the only files are ones written to or writing are in ram and fetched from lower by the CoW (copy on write) of the overlayfs.

The overlayfs method also allows one to track down which programs tend to slowly flood memory with logs (like systemd, apt status checks, and so on....). I noted widely timed small writes to the same inode can eat through your flash write limit quite quickly. We also looked at a hierarchical priority zram system using several allocations, and found it tended to be less stable over several days. Stretch probably has updated these packages if others are getting more reliable performance, and it may be worth another look.

In terms of the browser.... i seem to recall it has a read-only kiosk mode in addition to "--disable-sync-preferences"... you may want to limit the web file-cache size to under 300MB, system kernel io cache allocation to 2/3 of free ram, and set the system flush-to-disk priority as low as possible. This generally defers disk or swap writes, but also allows the io cache to swell into ram (or zram with a swap priority flag.) I would recommend looking at our kernel settings and config if you want to tweak your own read-only /home mount setup: https://sourceforge.net/projects/microm ... pberry-pi/

As HawaiianPi already stated, a purely ram disk approach is also possible. I am not sure how well Alpine is supported on the Pi, but it is also a minimalist system popular in several container recipes: https://www.alpinelinux.org/downloads/

At minimum, a SLC flash sdcard from digikey will extend your service life around 3-10 times longer than what MLC/TLC based cards offer. "High endurance" cards also tend to burn though their spare replacement sectors with the partial <512 byte small re-writes log files tend to make to the disk area. Normally the sync-to-disk defaults to 5 seconds, so some apps with sparsely-timed numerous smaller writes can sometimes wear the flash faster than expected.

Cheers, =)
J

[lrhorer]

Obviously, I need to dig more deeply into zram, however first I need to find out how chromium's footprint can be reduced. I have already made some inroads into the issue, reducing the footprint by over 17M through eliminating unnecessary files, such as the site blacklist (SafeBrowsing). Another 16M is created every time chromium runs in the two Broswer Metrics files, although I take it zram will compress these writes? They do compress quite nicely. Compressing with the default gz compression yields an image of only 75K. Most of the remaining 35M or so is in ~Default/Local Extension Settings (17M) and ~Default/Extensions (11M). How much of this can be blown away?

[lrhorer]

Have you considered using a different OS?

Tiny Core Linux runs entirely from RAM, and there is a Pi version.

The OS is not the problem. Other than LXDE, I have pretty much eliminated all writes to the drive by the OS by mounting /var/log, /tmp, and /var/tmp on tmpfs. LXDE does some writes that I think I can pretty much eliminate. The big offender is chromium-browser, which not only writes a lot, but has a lot of large configuration files.

[lrhorer]

If you are going to enable `zram-config enable-ephemeral` make sure any settings or changes are done before reboot as it is ephemeral on each boot `sudo zram-config disable-ephemeral` and reboot will get you back to normal.

If you use an overlayfs then its only the upper of the merge mount that is readwrite so the only files are ones written to or writing are in ram and fetched from lower by the CoW (copy on write) of the overlayfs.

I follow some, but definitely not all of that. Again, I will delve more deeply when I have had some sleep. One important thing of which I am not sure:

I want to be able to perform upgrades on the fly by updating certain files on the drive. Obviously, these changes cannot be volatile. Is there a way to make some changes permanent?

[stuartiannaylor]

If you are going to enable `zram-config enable-ephemeral` make sure any settings or changes are done before reboot as it is ephemeral on each boot `sudo zram-config disable-ephemeral` and reboot will get you back to normal.

If you use an overlayfs then its only the upper of the merge mount that is readwrite so the only files are ones written to or writing are in ram and fetched from lower by the CoW (copy on write) of the overlayfs.

I follow some, but definitely not all of that. Again, I will delve more deeply when I have had some sleep. One important thing of which I am not sure:

I want to be able to perform upgrades on the fly by updating certain files on the drive. Obviously, these changes cannot be volatile. Is there a way to make some changes permanent?

Problem is with the whole root ro that you need to umount the overlayfs as any further writes could be problematic as the merge down will break the logic between upper working directories and lower that result in the mount.
You prob could do it on a shutdown as the further writes are in volatile and its going to shutdown, but not sure of effect if any.
With the individual directories set in ztab they can be unmounted and merged on shutdown and all changes are merged down with no vagaries of effect.
OverlayFS took over Aufs as Aufs does have some performance problems, but think there is some kernel politics at play which is a shame as AuFs does have a range of utils that is totally lacking with OverlayFs.
Prob is the majority of distro's don't have the the Aufs kernel modules enabled or even included and OverlayFS is great but the lack of utils apart from the singular merge tool I use is not.
I am pretty sure an online (mounted) merge is more than possible but far beyond my ability or overall knowledge of the nuts and bolts of overlay FS but the mechanism of overlayfs is actually quite simple, deliberately by design and maybe why its been left light of accompanying utils and methods.

[lrhorer]

I still need to do some investigation into zram, but in the meantime, I have reduced the writes to the SD card tremendously. There are a few things that remain the highest write rate on the card:

1. Directory updates. In particular, even though the sub-directories are tmpfs mounts, the listing of several of the directories in the root are being updated because their contents are being updated. In particular, /run, /sys, and /tmp get updated a lot. I'm not sure how to prevent this while still allowing writes to the directory pointers (i.e. file create, file delete, and file move operations).

2. The file /var/lib/systemd/clock gets updated whenever the systemd-timesync daemon synchronizes the time. Is there some way to either stop this or else to have the daemon write to some other directory?

[stuartiannaylor]

tmpfs = ram just not compressed purely volatile memory not affected by power loss as not persistant.

Clock yeah do as I say just create a zram directory and either move the conf setting to point there or create the dir there.
I should really update zram-conf as thinking about it the lower RO directories could be an array or dorectories.

[jamesh]

Using a large SD card will increase lifetime as well, as any writes are spread over a larger area. Might be a cheap way of increasing lifetime.

[blackshard83]

using f2fs for the partition where chromium writes
mounting the partition with commit=x argument (x is the number of seconds the cache has to be flushed after, look for documentation for full explanation)
mount the partition with noatime argument, so there is the access time is not changed every time a file is accessed

[stuartiannaylor]

I noticed the other day that SSDs seem too of dropped ever further in price.
https://www.mymemory.co.uk/integral-120 ... 0mb-s.html

If you shop around then SSD + USB sata can be got for UK £20.

Not really sure why I even bother with SD still will not cure corruption but for some reason they seem to be far less fragile.