boelle
Posts: 257
Joined: Wed May 01, 2013 11:52 am

Re: Readonly Root with OverlyFS

Wed Nov 07, 2018 6:24 pm

ejolson wrote:
Wed Nov 07, 2018 5:52 pm
Did you include some sort of hysteresis so the Pi doesn't turn the panel on and off too frequently?
yes i did

boelle
Posts: 257
Joined: Wed May 01, 2013 11:52 am

Re: Readonly Root with OverlyFS

Wed Nov 07, 2018 6:27 pm

ejolson wrote:
Wed Nov 07, 2018 5:52 pm
Imagine trying to drive a car that was upside down and not knowing it.
if i did not know and the car drives fine i would not care much

ejolson
Posts: 4275
Joined: Tue Mar 18, 2014 11:47 am

Re: Readonly Root with OverlyFS

Wed Nov 07, 2018 7:21 pm

boelle wrote:
Wed Nov 07, 2018 6:27 pm
ejolson wrote:
Wed Nov 07, 2018 5:52 pm
Imagine trying to drive a car that was upside down and not knowing it.
if i did not know and the car drives fine i would not care much
But what if the car does not drive well with the wheels pointing upwards?

When things things don't work due to lack of understanding, it is fortunate if those mistakes don't lead to bodily harm, imprisonment or loss of life. That is why the Raspberry Pi projects described in this forum and the main website don't involve mains-level voltages among other things.

In the present case, not much harm can result (aside from the heater getting stuck in the on position) from a read-only root filesystem that doesn't work. Since most read-only approaches trend to exhaust available RAM over time (even if they don't rely on overlayfs), what do you think about adding a root crontab entry that proactively reboots the Pi every morning (or week) around breakfast time for reliability if this does not already happen? The watchdog installed by the Pimoroni script looks like a good idea in case something unexpected happens and the script itself seems reasonable as well.

jphl
Posts: 22
Joined: Tue Sep 05, 2017 2:53 pm

Re: Readonly Root with OverlyFS

Tue Dec 04, 2018 2:58 am

thanks for this i tried a somewhat similar script elsewhere that had weird effects like plymouth and the vt cursor being on despite having disabled them. hopefully when i try this tomorrow it will work better. my only guess on why the other did this is the use of initramfs so feeling optimistic. i do wonder if this script is hosted on github or something. just used to seeing projects there and would be interested in seeing what other projects you might be working on.

User avatar
SlowBro
Posts: 165
Joined: Sat Feb 18, 2017 1:30 am

Re: Readonly Root with OverlyFS

Tue Jan 22, 2019 7:19 pm

I too had the kernel panic, even though I downloaded it and installed it unmodified from your server directly to my Pi. Fixed with this:

Code: Select all

apt-get install dos2unix
dos2unix /sbin/overlayRoot.sh
I prefer your script to standard initfs methods as those do not easily work after a kernel upgrade.

kytkija
Posts: 13
Joined: Fri Feb 15, 2019 1:47 pm
Location: Finland

Re: Readonly Root with OverlyFS

Fri Feb 15, 2019 1:54 pm

Hello all!

I am totally new with raspbian and raspberry, been learning Python for a week now. And it already seems good, writing&reading to MySQL etc. :)

My app is just about to be ready. But it would be crucial for me to get the overlayFS working, as the raspbian will be connected to another machine and it will be powered down daily without halting it - the SD will get corrupted sooner or later.

There is lots of stuff about overlayFS in internet, but lots of them are old and it seems to me that it is important to have right scripts for right versions.

Could anyone confirm working method for overlayFS? I have bought Rpi 3B+ a week ago so I think it must be quite 'new version' or how do you call it. At the moment I have the desktopversion installed (downloaded week ago, so must be the most recent).

It is unclear to me is it possible to have overlayFS at all with desktop version? If it is, what would work with newest version?

If not, should I install raspbian lite - and does anyone have working tutorial for overlaying that?

Help would be more than appreciated, this forum has already given me a good start with my projects.

Best regards
Kytkija

emuola
Posts: 24
Joined: Wed Jun 19, 2013 5:50 pm

Re: Readonly Root with OverlyFS

Sat May 04, 2019 4:08 pm

Hi.

I managed to get RPI3B+ working with the Raspbian Desktop with this version of the script (I guess there are some fixes included?):

http://wiki.psuter.ch/doku.php?id=solve ... Az1orQco8A

Related to the script: Otherwise there are no problems regarding my "info-tv" build, except the dreaded fake-hwclock... As the fs is read-only, the time will drift and after a while the info-tv (standard Chromium) won't be able to access the url of my database (https) due to the system time being "too different". So, I'd need a way to update the system time and leave everything else read-only.

How could I make the fake-hwclock updateable and everything else read-only?

All the best,

-Olli

stuartiannaylor

Re: Readonly Root with OverlyFS

Sat May 04, 2019 11:40 pm

have a look at https://github.com/StuartIanNaylor/zram-config as have a system ephermal mode in that but you can also create zram dir that are the upper.

But the point of overlayfs is that only the lower is readonly but the upper is writable and presented through merge.
Have a look at the above and hack & copy as you see fit.
Also I think there are some problems with permissions with the current Pi Kernel and overlaysfs but give the above a try and will have a go myself.
You just need to comment out any zram swap, dir or log in /etc/ztab if you want just emphemeral only.

Code: Select all

sudo zram-config enable-ephemeral

emuola
Posts: 24
Joined: Wed Jun 19, 2013 5:50 pm

Re: Readonly Root with OverlyFS

Sun May 05, 2019 6:25 am

Hi stuartiannaylor and thanks for the reply.

Just to make sure I understand this: Does this solution replace the overlayFs solution completely? That's what I understood, but not 100% sure. The point with your solution would be to move everything else to zram (in /etc/ztab) and leave the fake-hwclock writable. Is this correct?

*edit*
Actually, I should make the whole root fs ephemeral:

sudo zram-config enable-ephemeral

But somehow exclude the fake-hwclock.data. Correct?

*edit*

Sorry, I'm not really familiar with the fs side of things regarding rpi.

All the best,

-Olli

stuartiannaylor

Re: Readonly Root with OverlyFS

Sun May 05, 2019 2:55 pm

emuola wrote:
Sun May 05, 2019 6:25 am
Hi stuartiannaylor and thanks for the reply.

Just to make sure I understand this: Does this solution replace the overlayFs solution completely? That's what I understood, but not 100% sure. The point with your solution would be to move everything else to zram (in /etc/ztab) and leave the fake-hwclock writable. Is this correct?

*edit*
Actually, I should make the whole root fs ephemeral:

sudo zram-config enable-ephemeral

But somehow exclude the fake-hwclock.data. Correct?

*edit*

Sorry, I'm not really familiar with the fs side of things regarding rpi.

All the best,

-Olli
OverlayFS allows you to merge various readonly file systems as a lower collection and a writeable as upper and these are presented as a merge mount.
I am using the same script which zram-config uses this lines from ro-root.sh https://github.com/StuartIanNaylor/zram ... ro-root.sh which we use the same script source

mount -t overlay -o lowerdir=/mnt/lower,upperdir=/mnt/rw/upper,workdir=/mnt/rw/work overlayfs-root /mnt/newroot

In mine upper is mount -t ext4 /dev/zram0 /mnt/rw so it uses compressed ram rather than tmpfs as a volatile r/w fs

rootDev=`awk '$2 == "/" {print $1}' /proc/mounts`
rootMountOpt=`awk '$2 == "/" {print $4}' /proc/mounts`
rootFsType=`awk '$2 == "/" {print $3}' /proc/mounts`
mount -t ${rootFsType} -o ${rootMountOpt},ro ${rootDev} /mnt/lower

Grabs the fs type options and device of the lower which because it is the lower its only readonly

That should end up as the merge mount of /mnt/newroot which is the combination of the 2 and because OverlayFS is a CoW (Copy on Write FS) the copy up to upper zram should happen only on write.

Your script isn't really readonly and as it is rw but only to volatile non persistent ram hence why I named it enable-ephemeral.
It should be able to write the hwclock updates to the upper but will just be lost on each reboot.

If not you can create another writeable zdir and make a script or edit hwclock to change the working dir.

I was just going to work with you and get it working whilst finding out what the permission problem was but with a break I spotted it straightaway. In my prog I am not copying the original perms like the rootMountOpt of the above which is something I am about to fix.

I will run up an image with zram-conf running empheral and see if I need to or can mangle hwclock to work. Can not even remember as actually thought it was fakehwclock but let me have a look.

stuartiannaylor

Re: Readonly Root with OverlyFS

Sun May 05, 2019 3:21 pm

So yeah installed zram-conf via git and a sudo sh install.sh
edited /etc/ztab and just commented out the zdir line and increased zswap to 300m 300%
Ran sudo zram-config enable-ephemeral and reboot

From the below you can see I have the / root mounted in an overlayfs which is using zram0 as upper.

Code: Select all

pi@raspberrypi:~ $ df
Filesystem     1K-blocks    Used Available Use% Mounted on
devtmpfs          464676       0    464676   0% /dev
tmpfs              94948      64     94884   1% /mnt/run
/dev/mmcblk0p2  30570444 2804860  26473308  10% /ro
/dev/zram0        991512  106148    817780  12% /rw
overlayfs-root    991512  106148    817780  12% /
tmpfs             474724       0    474724   0% /dev/shm
tmpfs             474724   12324    462400   3% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs             474724       0    474724   0% /sys/fs/cgroup
/dev/mmcblk0p1     43234   32913     10321  77% /boot
tmpfs              94944       0     94944   0% /run/user/1000
pi@raspberrypi:~ $ zramctl
NAME       ALGORITHM DISKSIZE   DATA  COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lz4          1000M 117.9M 395.7K  780K       4 /rw
/dev/zram1 lz4           900M     4K    76B    4K       4 [SWAP]
I just need to fix the zdir now as that is not bringing in the mount options of the old device and knew with a fresh look I could solve it :)
PS it is fake-hwclock and from the log with the above it works fine.

emuola
Posts: 24
Joined: Wed Jun 19, 2013 5:50 pm

Re: Readonly Root with OverlyFS

Sun May 05, 2019 5:27 pm

Hi stuartiannaylor and thanks for getting back on this :)

I actually got a bit forward just before you posted your replies. I did the following:

1. As the home directory of the user pi is in the ztab as default, I figured I'd make a new dir "fake" and move the fake-hwclock.data there with:

Code: Select all

sudo mv /etc/fake-hwclock.data /home/pi/fake/fake-hwclock.data
2. I also did a symlink with those two:

Code: Select all

sudo ln -s /home/pi/fake/fake-hwclock.data /etc/fake-hwclock.data
3. The file updates just fine, but is lost on boot -> time gets back to the date/time when the rootfs was rw the last time.

I gathered, you made some progress regarding the fake-hwclock, but I understood that saving the time between reboots (my infotv reboots every day), would require adding a dir into the ztab. I'm really not sure if the dir /home/pi/fake should be a writable zdir with the default ztab? Did I understand correctly? :)

Thanks one again for your help and effort here, couldn't do without it :)

All the best,

-Olli

stuartiannaylor

Re: Readonly Root with OverlyFS

Sun May 05, 2019 6:02 pm

emuola wrote:
Sun May 05, 2019 5:27 pm
Hi stuartiannaylor and thanks for getting back on this :)

I actually got a bit forward just before you posted your replies. I did the following:

1. As the home directory of the user pi is in the ztab as default, I figured I'd make a new dir "fake" and move the fake-hwclock.data there with:

Code: Select all

sudo mv /etc/fake-hwclock.data /home/pi/fake/fake-hwclock.data
2. I also did a symlink with those two:

Code: Select all

sudo ln -s /home/pi/fake/fake-hwclock.data /etc/fake-hwclock.data
3. The file updates just fine, but is lost on boot -> time gets back to the date/time when the rootfs was rw the last time.

I gathered, you made some progress regarding the fake-hwclock, but I understood that saving the time between reboots (my infotv reboots every day), would require adding a dir into the ztab. I'm really not sure if the dir /home/pi/fake should be a writable zdir with the default ztab? Did I understand correctly? :)

Thanks one again for your help and effort here, couldn't do without it :)

All the best,

-Olli
Yeah not sure why your script makes it necessary though as with the above the clock works fine with no need of modification.
Glad you are sorted.

emuola
Posts: 24
Joined: Wed Jun 19, 2013 5:50 pm

Re: Readonly Root with OverlyFS

Sun May 05, 2019 7:43 pm

Yeah not sure why your script makes it necessary though as with the above the clock works fine with no need of modification.
Glad you are sorted.
Sorry to bother you again, but I think I was bit unclear. The fake-hwclock.data (in /home/pi/fake/) gets updated ok, but it does not "survive a reboot", which is what I'd need.

How could I make the aforementioned file to survive a reboot?

All the best,

-Olli

stuartiannaylor

Re: Readonly Root with OverlyFS

Sun May 05, 2019 8:31 pm

I guess you need to turn off ro mode make your changes and then apply again.

Or sudo nano /etc/rc.local add your changes but the same will apply with the ro overlayfs as on each reboot all changes will be lost.


Or like I say use the empheral mode of zram-conf as the fkhdwareclock is not affected.

You could create a zdir with zram-config as on stop it merges down upper to lower so on reboot all changes are kept.

emuola
Posts: 24
Joined: Wed Jun 19, 2013 5:50 pm

Re: Readonly Root with OverlyFS

Mon May 06, 2019 4:36 am

Thanks again stuartiannaylor :)
Or like I say use the empheral mode of zram-conf as the fkhdwareclock is not affected.
That's how I have done it, but the fake-hwclock is "wrong" after the reboot. I'm really not sure if it should survive a reboot or not as you posted it is "not affected".

*edit*

I just realised, that apparently having the ztab and the directories there is an alternative to the (complete) emphemeral system. Or did I understand all wrong once again?

Emphemeral -> I cannot make any exclusions (everything is ro without the chance to make exceptions) and ztab directories -> there's still the possiblity to get fs corruption due to power loss etc?

Sorry, I'm not getting this :/

All the best,

-Olli

stuartiannaylor

Re: Readonly Root with OverlyFS

Mon May 06, 2019 6:34 am

emuola wrote:
Mon May 06, 2019 4:36 am
Thanks again stuartiannaylor :)
Or like I say use the empheral mode of zram-conf as the fkhdwareclock is not affected.
That's how I have done it, but the fake-hwclock is "wrong" after the reboot. I'm really not sure if it should survive a reboot or not as you posted it is "not affected".

*edit*

I just realised, that apparently having the ztab and the directories there is an alternative to the (complete) emphemeral system. Or did I understand all wrong once again?

Emphemeral -> I cannot make any exclusions (everything is ro without the chance to make exceptions) and ztab directories -> there's still the possiblity to get fs corruption due to power loss etc?

Sorry, I'm not getting this :/

All the best,

-Olli
You can run both if you wish but not much point. If you want a full system ro with upper writeable and presented in a overlayfs merge then the command sudo zram-config enable-ephemeral will do and and just comment out anything you don't want in ztab before reboot but you might want the zram swaps kept.
The clock is working fine on the desktop with me so it must be working. In syslog it seems fine May 6 02:59:27 raspberrypi fake-hwclock[212]: Mon 6 May 01:59:25 UTC 2019

Prob if you made a quick service that starts on multiboot target that does your symlink and stuff it prob will work but not sure what is causing the problem with your version or what the problem is.

The ephermal command is the script that will be installed to /usr/local/share/zram-config/ro-root.sh
edit these lines for memsize of write area
echo "lz4" > /sys/block/zram0/comp_algorithm
echo "1000M" > /sys/block/zram0/disksize
echo "333M" > /sys/block/zram0/mem_limit
lz4 usually doesn't dip below 300% compression and the effective disksize is always a bit of guesswork and I err low than high.
If the alg is changed to "deflate" prob will not dip below 400% but you could have highly compressed files that get minimal compression so zram hard control is done by actual mem_limit even if it does seem slightly back to front.

Code: Select all

pi@raspberrypi:~ $ cat /etc/ztab
# Stop the service sudo service zram-config stop edit /etc/ztab sudo nano /etc/ztab start the service sudo service zram-config start

# swap  alg     mem_limit       disk_size       swap_priority   page-cluster    swappiness
swap    lz4     300M            900M            75              0               90

# dir   alg     mem_limit       disk_size       target_dir              bind_dir
#dir    lz4     100M            300M            /home/pi                /pi.bind

# log   alg     mem_limit       disk_size       target_dir              bind_dir                oldlog_dir
#log    lz4     20M             60M             /var/log                /log.bind               /oldlog
I kept the default swap and commented out the pi home directory as going to run ephemeral
Did the git and install edited the above run sudo zram-config enable-ephemeral and clock is working fine after a reboot?
fakehwclock is never fine after a reboot as its fake and just updated via nettime
From the moment that service is down until restarted that time is static and just updated by a time server.
There isn't really any possibility of getting fs file corruption with ephemeral as all is ro and there are no writes to corrupt.
But there is no merge down on stop in ephemeral mode as each reboot is exactly as you set it in a kiosk mode.

If you want to do the same then you will have to molest the zram-conf script do a grep of /boot/cmdline.txt for "init=/bin/ro-root.sh" and call the merge routine and that will merge all volatile ram to the lower on shutdown if that is what you want.
Actually now I remember with some thought as you really should unmount before merge down and so you would lose root if you did a umount.
I have a feeling as long as its not continued to be used you could merge down live on a shutdown but you will have to test that.
That is why I left it ephemeral or create zdir of frequent writes.

Also I guess you could increase the commit value of ext4 to lessen the chances of write / powerloss
http://dustiesblog.blogspot.com/2012/12 ... ommit.html
But its still a matter of luck but higher means less writes but bigger bursts but chance wise it is less but not alleviated.
ztab dir is still ro but only on the directories you set and it will merge down changes on shutdown.
But you can not do the whole root / but you can do /var and /home or sub-directories which will be your likely culprits

If you want zero writes its ephemeral which steals system memory so using zram swap with high swapiness prob 100 of depending on gpu allocation maybe 500mb & 2gb disksize ? with swap 200-300mb with 800-1200mb disksize.

I would run normal and see if you have the timezone or something set right as like I say its no problem with me.
What does syslog say?

marklister
Posts: 2
Joined: Tue Jun 18, 2019 7:14 pm

Re: Readonly Root with OverlyFS

Tue Jun 18, 2019 7:27 pm

@psuter Thank you for sharing this. Works a treat for me, with some minor "improvements" :)

https://gist.github.com/marklister/c606 ... ebf2f26f8f

User avatar
SlowBro
Posts: 165
Joined: Sat Feb 18, 2017 1:30 am

Re: Readonly Root with OverlyFS

Tue Nov 19, 2019 9:39 pm

marklister wrote:
Tue Jun 18, 2019 7:27 pm
@psuter Thank you for sharing this. Works a treat for me, with some minor "improvements" :)

https://gist.github.com/marklister/c606 ... ebf2f26f8f
What did you change?

Zanstel
Posts: 31
Joined: Thu Jul 18, 2019 9:05 am

Re: Readonly Root with OverlyFS

Fri Nov 22, 2019 8:55 am

I used this script (well... my own version of a old version of this script) and you can find this usefull.

I used my raspberry mainly as a router so that helps a lot to allow me to power off directly without a proper shutdown.

But I like to update my distribution from time to time. There is other services and possible bugs, i'm worry specially about security bugs.
So, to update, of course you can edit config.txt, disable overlay, reboot, update, edit config.txt again and reboot.

BUT there is an alternative posibility that works most of the time.

# if [ ! -d /mnt/realroot ]; then mkdir /mnt/realroot; fi

# mount --bind /ro /mnt/realroot
# mount -o remount,rw /mnt/realroot
# mount --bind /boot /mnt/realroot/boot
# mount -t proc proc /mnt/realroot/proc
# chroot /mnt/realroot /bin/bash

And that's it. You are executing things on a rw version of the root, so you can apt update & upgrade normally
You can receive some warnings about unable to communicate with systemd, but I found that are normally non fatal.

After a reboot, all works OK.
With this method, I avoid one reboot and edit config.txt

User avatar
TimG
Posts: 297
Joined: Tue Apr 03, 2012 12:15 am
Location: Switzerland

Re: Readonly Root with OverlyFS

Fri Nov 22, 2019 9:35 pm

Zanstel wrote:
Fri Nov 22, 2019 8:55 am
[...]
And that's it. You are executing things on a rw version of the root, so you can apt update & upgrade normally
You can receive some warnings about unable to communicate with systemd, but I found that are normally non fatal.
This is great! I haven't tried it yet, but I certainly will soon.
Do you have any idea why systemd generates warnings after the chroot?
And after updating, is it possible to exit the chroot without rebooting?

Return to “Raspbian”