nikhaldi
Posts: 3
Joined: Thu Jun 16, 2016 5:38 pm

Frequent filesystem corruption (Raspbian)

Thu Jun 16, 2016 8:28 pm

We have produced a few hundred devices based on the compute module, running Raspbian wheezy. We are seeing random filesystem corruption relatively often. We've seen it on at least 10 devices out of about 200.

The symptoms are:
  • A device appears to work fine
  • After powering it down and powering up again it suddenly fails to boot because the boot partition or the Linux partition can't be mounted. There doesn't seem to be a particular pattern across to the corruption across multiple devices.
  • Flashing the compute module (or in some cases running fsck) makes it operational again.
These are details on the partitions (these are the defaults from the official Raspbian wheeze image):

Code: Select all

$ mount
/dev/root on / type ext4 (rw,noatime,data=ordered)
devtmpfs on /dev type devtmpfs (rw,relatime,size=185772k,nr_inodes=46443,mode=755)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=38012k,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=76000k)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
/dev/mmcblk0p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
We are mounting the Linux partition read/write and in theory that can lead to FS corruption when cutting the power uncontrolled. But my understanding is with ext4 and data=ordered this should be very rare, certainly not as common as we're experiencing it.

Has anybody experienced similar issues or has any idea what we should look for to address this?

We've produced a batch of similar devices with the same compute module before and we have not seen the FS corruption on those devices. This leads me to think that there may be some interaction with other components or the way the device is put together that creates the problem. There is an Adafruit wifi module and a FONA gsm module hooked up to the compute module.

edo1
Posts: 136
Joined: Sun Jun 15, 2014 3:33 pm
Location: Russia

Re: Frequent filesystem corruption (Raspbian)

Fri Jun 17, 2016 11:05 pm

IMHO you problem is related to the GSM module.
Isn't your GSM module or antenna located to close to RPI (or another electronics)?
Is your power supply good enough?

If low cellular signal occurs then current consumption of GSM module could be high and EMI could be high too (especially during GPRS/EDGE data transmission and other 2G activity)

nikhaldi
Posts: 3
Joined: Thu Jun 16, 2016 5:38 pm

Re: Frequent filesystem corruption (Raspbian)

Mon Jun 20, 2016 7:40 pm

GSM module as the source of the problem is a possibility. Indeed, all devices where we've seen this do have a GSM module (we have others with a wifi module only). We are using 2G only with these devices. The GSM module and antenna are mounted relatively tight with the compute module.

We'll investigate power supply. Any ideas how we should go about investigating EMI issues?

rln
Posts: 174
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: Frequent filesystem corruption (Raspbian)

Tue Jun 21, 2016 8:26 am

Does the error occur only after a cold boot or does it happen after warm boots too?
Author of the robust Nard distro http://www.nard.se

nixy82
Posts: 42
Joined: Sat Sep 12, 2015 9:19 am

Re: Frequent filesystem corruption (Raspbian)

Tue Jun 21, 2016 10:16 am

+1 for under wattage PSU being used, seen that before causing flash corruption on a non-rpi device with GPRS modem.

nikhaldi
Posts: 3
Joined: Thu Jun 16, 2016 5:38 pm

Re: Frequent filesystem corruption (Raspbian)

Tue Jun 21, 2016 2:31 pm

rin, it seems to manifest only after cold boot. I'm actually still trying to reproduce it in the lab under either warm or cold boot, but data from the the field suggests it's at cold boot.

rln
Posts: 174
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: Frequent filesystem corruption (Raspbian)

Tue Jun 21, 2016 2:53 pm

nikhaldi wrote:it seems to manifest only after cold boot
Then I also suggest checking the power supply. If warm boot were affected I would suspect flash wear.
Author of the robust Nard distro http://www.nard.se

edo1
Posts: 136
Joined: Sun Jun 15, 2014 3:33 pm
Location: Russia

Re: Frequent filesystem corruption (Raspbian)

Wed Jun 22, 2016 1:36 pm

nikhaldi wrote: Any ideas how we should go about investigating EMI issues?
Try to create low signal condition.
Maybe big steel pot could help, I'm not sure ;)

If you use external antenna, try to remove it. In my experience detached/broken antenna could produce a lot of problems with electronics.

urosg
Posts: 13
Joined: Sat Aug 22, 2015 1:08 am

Re: Frequent filesystem corruption (Raspbian)

Sat Jun 25, 2016 8:51 pm

I also would suggest problems with powersupply, GSM modules can go over 2Amp in peak, don't know how much power does your supply have and what power regulators you use...

Return to “Compute Module”