User avatar
rpdom
Posts: 14985
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: HOWTO: Move the filesystem to a USB stick

Sun Nov 03, 2013 7:23 pm

1. In the HOWTO there was the comment "Before you do that however, remember that we modified the /etc/fstab file on the SD card and then moved it to the USB stick.", so if you did it this way the fstab on the SD card would need to be put back to the default. If, however, you only changed it on the USB stick you don't need to change it on the SD card.

2. There used to be a "rule" (read as "guideline") to have swap as RAM size + a bit more. This is out dated now that most systems have plenty of RAM. Setting a large swapfile won't hurt though, and could help if you run memory hungry apps.

paulv
Posts: 558
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands

Re: HOWTO: Move the filesystem to a USB stick

Sun Nov 03, 2013 10:41 pm

Hi NaCaSi,

Thanks for the compliment and the suggestions.

Also rpdom, I just saw your input when I was editing this. Good inputs too, thank you!

Yes, you are both right about the fstab. If you did not change the fstab on the SD card, there is obviously no need to do anything there if you want to go back to using the SD card. All you need to do then is to change the /boot/cmdline.txt on the SD card.

After I read a lot more information about preventing issues with the SD card, or a memory stick, I now suggest that you don't use the SWAP file at all, if you can handle that. Swapping will cause many (small) read/write operations to the "disk", which is what we're trying to avoid regardless what medium we use as storage. The more we can execute in RAM, the better it is. In addition, I now put /tmp on tmpfs, which is in RAM. Just add the following line to your fstab:

Code: Select all

tmpfs   /tmp   tmpfs   defaults, noatime, mode=1777   0   0 
I have not yet decided myself on the best approach to move /var/log and /var/lock away from the SD card or stick. (Using aufs or unionfs) There are several forum discussions on several possible approaches and I don't know enough about Linux to know what to use. I need to study this more and try things out myself. A lot depends on what you are doing with your Pi however.

[edit]
I have made my decision on creating a -no-write- solution.
I use the RAMlog solution from /www.tremende.com/ramlog for all the log files, and I have made provisions in my code to move all application files of my thermostat program to /tmp, which is on tmpfs.

During the init of my application, it prepares the structure on /tmp :

Code: Select all

    # ----- setting up the environment
    #
    # copy all variable files to /tmp/therm, which has been mounted as a RAM disc in fstab
    #
    if RAM_DISK :
        if os.path.isdir(file_path):
            pass
        else:
            cmd = "mkdir "+file_path
            os_system_call(cmd)
        # either update or copy the files to the ram disk
        cmd = "rsync -qa "+safe_path+"/*.* "+file_path+"/."
        os_system_call(cmd)
        write_log("system", "set-up RAM disk and copied files there")
    #
And I make sure that on every possible exit of the application, including a Python error termination, the files are stored back to the SD card/USB stick. This is part of the function that gets called just before I reboot, terminate, or when there is a Python crash :

Code: Select all

    
...
# if we use the ram disk, save the files to the SD card/USB stick
    if RAM_DISK :
        write_log("system", "save the application files from the ram_disk")
        #
        # update the latest control and display parameters in the json structures
        # and update the server
        get_parms()
        check_parms() # this will also run update_web_control()
        update_web_display()
        #
        # copy all updated application files back to the SD card
        cmd = "rsync -qa "+file_path+"/*.* "+safe_path+"/."
        os_system_call(cmd)
    #
    # flush the file system cache
    cmd = "sync;sync;sync"
    os_system_call(cmd)
This function also saves the most critical log files (messages, dmesg, daemon.log and syslog) to the server so I can study them when I am remote and also before they get overwritten.

Using the following command (when using the SD card) will verify that there are no writes anymore :

Code: Select all

pi@raspberrypi ~ $ vmstat -p /dev/mmcblk0p1
mmcblk0p1     reads   read sectors  writes    requested writes
                  53        225          0          0
pi@raspberrypi ~ $
Executing tops will show the memory status of the system:

Code: Select all

top - 09:29:48 up 17:03,  1 user,  load average: 0.21, 0.22, 0.23
Tasks:  72 total,   1 running,  70 sleeping,   0 stopped,   1 zombie
%Cpu(s):  0.7 us,  1.3 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:    497504 total,   401720 used,    95784 free,    16444 buffers
KiB Swap:        0 total,        0 used,        0 free,   349236 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
 2152 root      20   0 37364 7668 3732 S   1.0  1.5   6:21.83 python
It looks like I have enough RAM to run this setup. I'll be watching this over the next week or so when the maximum log file size must have been reached (after compressing a week worth of old log files) Notice, that I also turned off SWAP.

I hope this helps...
Last edited by paulv on Sun Nov 10, 2013 3:37 pm, edited 1 time in total.

Lee Hambley
Posts: 10
Joined: Tue Jun 11, 2013 7:45 am

Re: HOWTO: Move the filesystem to a USB stick

Wed Nov 06, 2013 10:43 am

I'm just chiming in here to ask if you ever had any success with this? I'm keen to try something similar. I have a usecase which writes files to disk every 30s (and then deletes them 1-30s later, usually when the netowrk is up), I'm seeing a really frequent corruption of my SD cards, and wanted to move the scratch spaces of the filesystem away from the boot media.

I've some experience with unionsf and ram disks to do this (from an embedded linux project I worked on previously) - but I don't want to invest a tonne of time into this if someone has a bit of experience specifically with the Pi/SD/USB thing already.

User avatar
RaTTuS
Posts: 10409
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: HOWTO: Move the filesystem to a USB stick

Wed Nov 06, 2013 10:48 am

ther are several posts on how to move the rootfs to a USB HD I use 2 of my rpi's like that
but if you have temp files that are small then just use a tmpfs ramdisk ...
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

Lee Hambley
Posts: 10
Joined: Tue Jun 11, 2013 7:45 am

Re: HOWTO: Move the filesystem to a USB stick

Wed Nov 06, 2013 10:52 am

but if you have temp files that are small then just use a tmpfs ramdisk ...

Yeah, thought about that, it's mostly about /var/log and /var/run, which I guess I can fix in the fstab.

I'm surprised that it's getting corrupted so often, they're good quality SD cards, and afak I'm using ext4 (standard (modern) rasbian image.)

User avatar
RaTTuS
Posts: 10409
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: HOWTO: Move the filesystem to a USB stick

Wed Nov 06, 2013 11:26 am

http://www.raspberrypi.org/phpBB3/viewt ... 75#p448875
see that for GSH pointer to SD card curruption
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

paulv
Posts: 558
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands

Re: HOWTO: Move the filesystem to a USB stick

Wed Nov 06, 2013 5:16 pm

Lee,

In my application (a web based thermostat), I use json's to communicate with the web server (located on another continent), I record the temperatures in a daily log file every 30 seconds, and I log errors, warnings and system states in a another log file. Then I create graphs for the web server using RDD, so the RRD database is updated every 30 seconds and there are 4 graphs that are created about every 3 minutes. When there is a Python crash the trace back also goes into a file. These are all small write actions and some read actions, so I have put all these files in /tmp.

Before a shutdown, these files get copied to my "normal" structure on the SD card or USB stick.
When I boot, the reverse takes place.

This seems to work really well. Obviously, you just need to make very sure that the data always gets saved before the power goes off, or when there is a reboot, and that your program starts with the data in the correct place. I use a home-built UPS (discussed in another post) for my thermostat, so I have avoided unexpected power loss problems, and the program is in full control because it shuts down the power itself.

I have not done that yet, but it's simple to use cron to copy the files from the /tmp folder to the SD card/USB stick once per day or so, just to make sure. I myself have the application send the data to the server once per day.

I'm currently leaning towards using Aufs for the Debian log and lock files. I need to understand that better and play with that first.

Lee Hambley
Posts: 10
Joined: Tue Jun 11, 2013 7:45 am

Re: HOWTO: Move the filesystem to a USB stick

Wed Nov 06, 2013 5:40 pm

Hey Paul,

Thanks for the feedback, I guess the biggest thing for me is to stop using a 4Gb SD card, with my dev toolchain I'm so close to filling them up, I can't even rule out that I've filled the block storage.

I'll post back if I come up with a nice solution for aufs or overlayfs on the Pis, there's a script (German tutorial, and comments) which does a great job, but I haven't tried it on anything other than Ubuntu

- Lee

paulv
Posts: 558
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands

Re: HOWTO: Move the filesystem to a USB stick

Wed Nov 06, 2013 6:19 pm

Hi Lee,

Can you send me the link with the script?
I'm fluent in German, so maybe I can translate.

Paul

Lee Hambley
Posts: 10
Joined: Tue Jun 11, 2013 7:45 am

Re: HOWTO: Move the filesystem to a USB stick

Wed Nov 06, 2013 6:32 pm

http://wiki.ubuntuusers.de/Nur-Lesen_Root-Dateisystem

(helps to read German :-) - as I said, worked perfectly for me on *real* linux, never tried on a Pi with it's limited RAM/etc)

Viele Grüße aus Hamburg!

paulv
Posts: 558
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands

Re: HOWTO: Move the filesystem to a USB stick

Sun Nov 10, 2013 3:42 pm

I just posted my current solution to a "no-write" setup in an [edit] to a previous post (from Sun Nov 03, 2013 4:41 pm ) a little above this post, because it follows on other things I did.
Have a look if you're interested.

Lee Hambley
Posts: 10
Joined: Tue Jun 11, 2013 7:45 am

Re: HOWTO: Move the filesystem to a USB stick

Thu Nov 14, 2013 9:09 am

Hi Paul,

I just took the time to read through some of your scripting stuff, that's quite a neat solution, especially the trick about counting writes to the SD card with `vmstat ` (it could be so easy 'ehh!)

For me unfortunately that won't work, but I'm making two similar changes:

• I'll be using the aufs/overlayfs script linked above. (I'll probably re-publish it in English, too)
• I'll be moving to syslog-ng and writing the log lines directly to my main server using TCP+TLS in order to keep writes down, and avoid keeping logs.

My main problem is, now I *know* that my installation is powered down (rudely) every night when the stores where they are close for the night, and just throw the master switch and kill power to the whole shop. Which might have been nice to know in advance.

I might also experiment with partitioning the SD card, and mounting /var/log and /var/run (perhaps also /tmp) on a readwrite partition, whilst using overlayfs to protect the boot environment.

Lee Hambley
Posts: 10
Joined: Tue Jun 11, 2013 7:45 am

Re: HOWTO: Move the filesystem to a USB stick

Fri Nov 15, 2013 8:38 am

Hi Paul,

Another quick follow up. I was able to nail the perfect setup.

I exported a NSF share on my Mac:

Code: Select all

$ cat /etc/exports
/Users/leehambley/Projects/traxity/sensor-stack -maproot=root:wheel -network 192.168.178.0 -mask 255.255.255.0
Which is mounted on my RaspberryPi:

Code: Select all

# cat /etc/fstab
proc                                                            /proc       proc  defaults                                        0       0
/dev/mmcblk0p1                                                  /boot       vfat  ro,suid,dev,noexec,auto,nouser,async,fmask=0133 0       0
/dev/mmcblk0p2                                                  /           ext4  ro,suid,dev,exec,auto,nouser,async,noatime      0       0
192.168.178.167:/Users/leehambley/Projects/traxity/sensor-stack /mnt/data-1 nfs   defaults                                        0       0
Could be a little neater (symlinks to keep things sane, but I'm happy enough).

But the killer thing here, is that someone has released a version of Raspbian which is read only by default

http://nutcom.hu/?page_id=108 It's called "IPE – Industrial Perennial Environment" and it has two simple commands `ipe-rw` and `ipe-ro`, otherwise runs everything in a tempfs. They ensure you can't screw things up! Their use-case sounded exactly like yours.

I had to use some of the instructions from http://www.raspbian.org/RaspbianFAQ#How ... S_share.3F to get NFS, rpcbind and portmap support in Raspbian (perhaps this is specific to the stripped down IPE) - but either way, it was trivial.

Now my devices are on their way back out to the wild, so we'll just have to hope they stay up this time :)

paulv
Posts: 558
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands

Re: HOWTO: Move the filesystem to a USB stick

Fri Nov 15, 2013 2:43 pm

Good job!

I saw your reference to the "IPE – Industrial Perennial Environment". I've come across that earlier, but could not find any references of people using it at all. (just Googled it)
There is also no information available about what is actually different and how they get to this version of Raspbian. I'm not enough of an expert to experiment with it, so I hope others will pick this up.

In the meantime, I have found that my stock version of Raspbian is creating a lot of log files that I don't need, and I was getting to a critical level of free RAM within a week. Rather then tediously modifying the various log and logrotate routines, I created a simple shell script that get's executed by cron at 1am every day, and that removes all the extra log files (*.log.? and *.log.gz etc.) I jumped from about 30 free Meg to 300 Meg. I do keep the critical logs, and ship them daily to a server at midnight, just in case...

The extra free RAM meant that I could expand on my RRDTOOL database, and get more granularity for the longer periods.

joe_irvine
Posts: 1
Joined: Sat Nov 16, 2013 11:04 pm

Re: HOWTO: Move the filesystem to a USB stick

Sun Dec 08, 2013 2:38 pm

Great clear instructions that helped me, a relative novice, enormously :D

xhawk
Posts: 1
Joined: Tue Dec 17, 2013 4:39 pm

Re: HOWTO: Move the filesystem to a USB stick

Tue Dec 17, 2013 4:40 pm

Thank you man! You are the best :D

adrianidar
Posts: 20
Joined: Sun Dec 22, 2013 7:33 am
Location: London, UK

Re: HOWTO: Move the filesystem to a USB stick

Sun Dec 22, 2013 8:19 am

First of all, thank you for the instructions, very well written.

I'm intrigued though by some ideas circulated in subsequent posts: one recommends moving the swap to ram! This looks dodgy to me: swap was created exactly for the purpose of offloading to disk when ram gets in short supply!

I'll copy/paste from one of my posts where I ask some why questions:

Aren't USB sticks just like SD cards but with a different interface?
- What makes running from USB more resilient then? Or is it just a myth?
- Where is the best recipe for moving to USB? I've googled and seen a few. Seen one well written on this site: viewtopic.php?f=29&t=44177
- Would any USB stick do? Or is there a list with good and bad as for sd cards? Are we then going back to square one ;)?

Thanks!

User avatar
DougieLawson
Posts: 35793
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: HOWTO: Move the filesystem to a USB stick

Sun Dec 22, 2013 9:44 am

adrianidar wrote: - What makes running from USB more resilient then? Or is it just a myth?
It's probably a myth. It's the same memory technology just with a different packaging & interface.

Powered USB HDD is going to be most reliable.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

paulv
Posts: 558
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands

Re: HOWTO: Move the filesystem to a USB stick

Sun Dec 22, 2013 10:09 am

There are a significant number of experienced people that tend to disagree. SD cards were originally made for sequential storage of pictures and films and not for the random access demands of a multi-tasking OS. A significant amount of the earlier problems with the SD cards have been addressed in the newer OS releases for the Pi, but running the OS of a USB "stick" or a "real" USB connected hard drive or solid state solution adds more security, and sometimes even speed. In a teaching/hobby environment, the SD card is great, which was the original target audience for the Pi. For more serious work, the overall recommendation is to select better suited storage devices.

So, I guess the take-away is that a lot depends on what you do with the Pi.

User avatar
DougieLawson
Posts: 35793
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: HOWTO: Move the filesystem to a USB stick

Sun Dec 22, 2013 10:48 am

The SDCard is serial but so is a USB stick (or USB HDD for that matter). I can't see the difference.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

User avatar
rpdom
Posts: 14985
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: HOWTO: Move the filesystem to a USB stick

Sun Dec 22, 2013 11:51 am

DougieLawson wrote:The SDCard is serial but so is a USB stick (or USB HDD for that matter). I can't see the difference.
fcvo "serial". The SD card starts off as a serial device with a single data line, but once the drivers are loaded it can switch to using four data lines, effectively quadrupling throughput.

Having said this, the data rates of the two interfaces are different. The USB can run at up to 480Mbs, while the SD runs at up to 50MHz which gives 200Mbs. Both interfaces will have overheads at various levels, and the read/write speed of the actual flash/disk in the device will also be a factor.

I've found an old USB-IDE HDD to be a bit slower than a fast SD card, but more reliable and less prone to damage if the power fails.

sharix
Posts: 200
Joined: Thu Feb 16, 2012 11:29 am
Location: Slovenia

Re: HOWTO: Move the filesystem to a USB stick

Thu Jan 02, 2014 2:19 pm

Thanks for this wonderful guide.

jm3220
Posts: 6
Joined: Mon Jul 15, 2013 9:05 pm

Re: HOWTO: Move the filesystem to a USB stick

Sun Jan 05, 2014 3:33 pm

Paulv,

Thanks for such useful and clear instructions - it worked perfectly first time.

paulv
Posts: 558
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands

Re: HOWTO: Move the filesystem to a USB stick

Sun Jan 05, 2014 3:41 pm

Thank you Lee Hambley, joe_irvine, xhawk, adrianidar, jm3220, sharix and the many others!
Always nice to hear people appreciate your time & efforts.
Paul

zproc
Posts: 31
Joined: Mon Sep 03, 2012 12:05 pm

Re: HOWTO: Move the filesystem to a USB stick

Mon Jan 06, 2014 5:16 pm

Hello,
I followed the tutorial on the first post to move my sdcard filesystem on a 16gb flashdrive... the sdcard was 8gb but

df -h tells me

Code: Select all

Sys. fich.     Taille Util. Dispo Uti% Monté sur
rootfs           4,0G  3,7G   86M  98% /
/dev/root        4,0G  3,7G   86M  98% /
devtmpfs         108M     0  108M   0% /dev
tmpfs             24M  576K   23M   3% /run
tmpfs            5,0M     0  5,0M   0% /run/lock
tmpfs             47M   72K   47M   1% /run/shm
/dev/mmcblk0p1    56M   19M   38M  34% /boot
/dev/sdb1        466G  413G   53G  89% /media/mybook
The filesystem is 4gb and

fdisk -u /dev/sda then 'p' tells me this

Code: Select all

sudo fdisk -u /dev/sda

Command (m for help): p

Disk /dev/sda: 16.0 GB, 16008609792 bytes
255 heads, 63 sectors/track, 1946 cylinders, total 31266816 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000077d4

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048     8385929     4191941    6  FAT16
/dev/sda2   *     8386560    31266815    11440128    5  Extended
/dev/sda5         8388608    31266815    11439104   30  Unknown
I don't understand what happened. How can I fix this?

Return to “Advanced users”