swampdog
Posts: 230
Joined: Fri Dec 04, 2015 11:22 am

Pi 4, rootfs on SSD goes readonly

Sun Aug 04, 2019 10:10 pm

Hi Folks,

Just wanted to reach out to see if anyone else if having this issue. I've spent the day doing all manner of tests for little result.

Only errors to be had are on the console. My phone is a pig for not focusing on displays.
"ext4_find_entry:1442: inode N" (various values for N) which I think were preceded by "ext4_journal_check_start:61 Detected aborted journal" and am unsure if this was the first error onto the console in any case.

I tried to get some logs: nothing. I created LVM2 for /var,/tmp and gcc build directory, the idea obviously being able to get at /var/log/ on my linux box but I guess systemd had gone awol.

Initially I had the SSD mounted with "discard" option as it supports trim but took that out very early in the debug process (reinstalled OS from scratch). Initially my linux box could not fix the journal error so I downloaded and built e2fsprogs-1.45.3.tar.xz for it. Gave up on that after a while, went retro and prepped the SSD on my linux box. I even replaced the ext4 rootfs later with an ext3 one on the offchance it was an ext4 bug. Same.

I've just tested the SSD on my linux box. Mounted the same LVM2 volumes and built gcc (esata caddy). Fine. Repeat with the usb->sata adapter used on the Pi 4. Fine.

I'm able to get the Pi 4 to render rootfs readonly pretty quickly: It first did it while building gcc 7.4.0 but it takes a while. Repeat but while it's doing that have it 'dd' the two sdcard partitions onto the SSD. It won't get more than a couple of gig into /dev/mmcblk0p2 before rootfs goes readonly.

Tried the '"vl805_fw_0137a8.bin" firmware but that just made things very slow and rather than the above errors there was a lengthy kernel panic. Put the "vl805_fw_013701.bin" image back.

Begs the question: when the Pi 4 was using an LVM2 "gcc" folder for its sources, ditto for /tmp, why when it crapped out, did rootfs go readonly?

Would this bug be manifesting itself at a lower level than the filesystem?

Oh, also tried the SSD on the Pi 4 usb2 port as well as the usb3. No difference.

Hardware info available on request.

epoch1970
Posts: 3547
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Pi 4, rootfs on SSD goes readonly

Sun Aug 04, 2019 10:25 pm

If the filesystem is broken it tries to go read-only.
Why does it get corrupted? Could be the power supply, the adapter, or perhaps a bug in your non-mainstream filesystem, or in the SSD itself.

Maybe you want to "detune" the stack a little bit in order to find the culprit...
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

swampdog
Posts: 230
Joined: Fri Dec 04, 2015 11:22 am

Re: Pi 4, rootfs on SSD goes readonly

Mon Aug 05, 2019 12:16 am

Ah, looks like I was a bit unclear. Tired.

Originally, it was a standard configuration. Installed to an 8G sdcard. The next operation was simply to format the SSD (gpt,64G, ext4) then rsync /dev/mmcblk0p2 onto /dev/sda1 (the SSD), mounted as /mnt/tmp/ then adjust /boot/cmdline.txt and /mnt/tmp/etc/fstab to use the 'blkid' to place the UUID in both. Reboot and it was up and running. It's got a pimoroni fan shim so to test it I performed a quick build of gcc 8.3.0 whilst watching the temperature. Happy that it was coming in at <50 degrees I set it off on a full build of gcc 7.4.0 (all stages + tests) overnight. In the morning the filesystem was readonly.

Everything is new so obviously the problem could be anywhere. As the day wore on that simple /dev/sda1 partition mutated in an attempt to trap the issue, especially as it first looked like an ext4 bug. The power supply is the official one. I've now confirmed both the adapter and the SSD work on my linux box. The SSD has the latest firmware.

Here's some info I grabbed at various points..

Code: Select all

foo@pi18:/wrk $ lspci
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 10)
01:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01)

foo@pi18:~ $ lsusb
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 174c:55aa ASMedia Technology Inc. Name: ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
The SSD..

Code: Select all

foo@pi18:~ $ sudo hdparm -I /dev/sda
/dev/sda:

ATA device, with non-removable media
	Model Number:       CT1000MX500SSD1                         
	Serial Number:      1921E205464B        
	Firmware Revision:  M3CR023 
	Transport:          Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
	Used: unknown (minor revision code 0x006d) 
	Supported: 10 9 8 7 6 5 
	Likely used: 10
Configuration:
	Logical		max	current
	cylinders	16383	0
	heads		16	0
	sectors/track	63	0
	--
	LBA    user addressable sectors:   268435455
	LBA48  user addressable sectors:  1953525168
	Logical  Sector size:                   512 bytes
	Physical Sector size:                  4096 bytes
	Logical Sector-0 offset:                  0 bytes
	device size with M = 1024*1024:      953869 MBytes
	device size with M = 1000*1000:     1000204 MBytes (1000 GB)
	cache/buffer size  = unknown
	Form Factor: 2.5 inch
	Nominal Media Rotation Rate: Solid State Device
Capabilities:
	LBA, IORDY(can be disabled)
	Queue depth: 32
	Standby timer values: spec'd by Standard, with device specific minimum
	R/W multiple sector transfer: Max = 1	Current = 1
	Advanced power management level: 254
	DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6 
	     Cycle time: min=120ns recommended=120ns
	PIO: pio0 pio1 pio2 pio3 pio4 
	     Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
	Enabled	Supported:
	   *	SMART feature set
	    	Security Mode feature set
	   *	Power Management feature set
	   *	Write cache
	   *	Look-ahead
	   *	WRITE_BUFFER command
	   *	READ_BUFFER command
	   *	NOP cmd
	   *	DOWNLOAD_MICROCODE
	   *	Advanced Power Management feature set
	   *	48-bit Address feature set
	   *	Mandatory FLUSH_CACHE
	   *	FLUSH_CACHE_EXT
	   *	SMART error logging
	   *	SMART self-test
	   *	General Purpose Logging feature set
	   *	WRITE_{DMA|MULTIPLE}_FUA_EXT
	   *	64-bit World wide name
	   *	WRITE_UNCORRECTABLE_EXT command
	   *	{READ,WRITE}_DMA_EXT_GPL commands
	   *	Segmented DOWNLOAD_MICROCODE
	    	unknown 119[8]
	   *	Gen1 signaling speed (1.5Gb/s)
	   *	Gen2 signaling speed (3.0Gb/s)
	   *	Gen3 signaling speed (6.0Gb/s)
	   *	Native Command Queueing (NCQ)
	   *	Phy event counters
	   *	READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
	    	DMA Setup Auto-Activate optimization
	    	Device-initiated interface power management
	   *	Software settings preservation
	    	Device Sleep (DEVSLP)
	   *	SMART Command Transport (SCT) feature set
	   *	SCT Features Control (AC4)
	   *	SCT Data Tables (AC5)
	   *	SANITIZE_ANTIFREEZE_LOCK_EXT command
	   *	SANITIZE feature set
	   *	CRYPTO_SCRAMBLE_EXT command
	   *	BLOCK_ERASE_EXT command
	   *	reserved 69[3]
	   *	reserved 69[4]
	   *	reserved 69[7]
	   *	DOWNLOAD MICROCODE DMA command
	   *	WRITE BUFFER DMA command
	   *	READ BUFFER DMA command
	   *	Data Set Management TRIM supported (limit 8 blocks)
Security: 
	Master password revision code = 65534
		supported
	not	enabled
	not	locked
	not	frozen
	not	expired: security count
		supported: enhanced erase
	2min for SECURITY ERASE UNIT. 2min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 500a0751e205464b
	NAA		: 5
	IEEE OUI	: 00a075
	Unique ID	: 1e205464b
Device Sleep:
	DEVSLP Exit Timeout (DETO): 100 ms (drive)
	Minimum DEVSLP Assertion Time (MDAT): 10 ms (drive)
Checksum: correct

Code: Select all

foo@pi18:/wrk $ sudo lshw
pi18
    description: ARMv7 Processor rev 3 (v7l)
    product: Raspberry Pi 4 Model B Rev 1.1
    serial: 100000004b1859cf
    width: 32 bits
    capabilities: smp
  *-core
       description: Motherboard
       physical id: 0
     *-cpu:0
          description: CPU
          product: cpu
          physical id: 1
          bus info: cpu@0
          size: 1500MHz
          capacity: 1500MHz
          capabilities: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 cpufreq
     *-cpu:1
          description: CPU
          product: cpu
          physical id: 2
          bus info: cpu@1
          size: 1500MHz
          capacity: 1500MHz
          capabilities: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 cpufreq
     *-cpu:2
          description: CPU
          product: cpu
          physical id: 3
          bus info: cpu@2
          size: 1500MHz
          capacity: 1500MHz
          capabilities: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 cpufreq
     *-cpu:3
          description: CPU
          product: cpu
          physical id: 4
          bus info: cpu@3
          size: 1500MHz
          capacity: 1500MHz
          capabilities: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 cpufreq
     *-memory
          description: System memory
          physical id: 5
          size: 3906MiB
     *-pci
          description: PCI bridge
          product: Broadcom Limited
          vendor: Broadcom Limited
          physical id: 0
          bus info: pci@0000:00:00.0
          version: 10
          width: 32 bits
          clock: 33MHz
          capabilities: pci pm pciexpress normal_decode bus_master cap_list
          resources: memory:600000000-6000fffff
        *-usb
             description: USB controller
             product: VL805 USB 3.0 Host Controller
             vendor: VIA Technologies, Inc.
             physical id: 0
             bus info: pci@0000:01:00.0
             version: 01
             width: 64 bits
             clock: 33MHz
             capabilities: pm msi pciexpress xhci bus_master cap_list
             configuration: driver=xhci_hcd latency=0
             resources: irq:55 memory:600000000-600000fff
           *-usbhost:0
                product: xHCI Host Controller
                vendor: Linux 4.19.58-v7l+ xhci-hcd
                physical id: 0
                bus info: usb@1
                logical name: usb1
                version: 4.19
                capabilities: usb-2.00
                configuration: driver=hub slots=1 speed=480Mbit/s
              *-usb
                   description: USB hub
                   product: USB2.0 Hub
                   vendor: VIA Labs, Inc.
                   physical id: 1
                   bus info: usb@1:1
                   version: 4.20
                   capabilities: usb-2.10
                   configuration: driver=hub maxpower=100mA slots=4 speed=480Mbit/s
           *-usbhost:1
                product: xHCI Host Controller
                vendor: Linux 4.19.58-v7l+ xhci-hcd
                physical id: 1
                bus info: usb@2
                logical name: usb2
                version: 4.19
                capabilities: usb-3.00
                configuration: driver=hub slots=4 speed=5000Mbit/s
              *-usb
                   description: Mass storage device
                   product: ASM1153USB3.0TOSATA
                   vendor: ASM1153USB3.0TOSATA
                   physical id: 2
                   bus info: usb@2:2
                   logical name: scsi0
                   version: 1.00
                   serial: 123456789027
                   capabilities: usb-3.00 scsi
                   configuration: driver=uas speed=5000Mbit/s
                 *-disk
                      description: SCSI Disk
                      product: 2115
                      vendor: ASMT
                      physical id: 0.0.0
                      bus info: scsi@0:0.0.0
                      logical name: /dev/sda
                      version: 0
                      serial: 720987654321
                      size: 931GiB (1TB)
                      capabilities: gpt-1.00 partitioned partitioned:gpt
                      configuration: ansiversion=6 guid=a2c225ad-c71e-461d-b319-ad434e076c8b logicalsectorsize=512 sectorsize=4096
                    *-volume:0
                         description: EXT4 volume
                         vendor: Linux
                         physical id: 1
                         bus info: scsi@0:0.0.0,1
                         logical name: /dev/sda1
                         version: 1.0
                         serial: 871275b4-db6a-4938-a47e-a329e5a29998
                         size: 32GiB
                         capabilities: journaled extended_attributes large_files huge_files dir_nlink recover extents ext4 ext2 initialized
                         configuration: created=2019-08-04 15:26:59 filesystem=ext4 lastmountpoint=/ modified=2019-08-04 15:24:22 mounted=2019-08-04 16:25:01 name=primary state=clean
                    *-volume:1
                         description: LVM Physical Volume
                         vendor: Linux
                         physical id: 2
                         bus info: scsi@0:0.0.0,2
                         logical name: /dev/sda2
                         serial: TI6eQ1-19Wf-cOqF-UU2L-4oyX-6VjY-2eyJSm
                         size: 118GiB
                         capabilities: multi lvm2
                         configuration: name=primary
  *-usbhost
       product: DWC OTG Controller
       vendor: Linux 4.19.58-v7l+ dwc_otg_hcd
       physical id: 1
       bus info: usb@3
       logical name: usb3
       version: 4.19
       capabilities: usb-2.00
       configuration: driver=hub slots=1 speed=480Mbit/s
  *-network:0
       description: Ethernet interface
       physical id: 2
       logical name: eth0
       serial: dc:a6:32:07:64:2a
       size: 1Gbit/s
       capacity: 1Gbit/s
       capabilities: ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=bcmgenet driverversion=v2.0 duplex=full ip=192.168.1.13 link=yes multicast=yes port=MII speed=1Gbit/s
  *-network:1
       description: Wireless interface
       physical id: 3
       logical name: wlan0
       serial: dc:a6:32:07:64:2b
       capabilities: ethernet physical wireless
       configuration: broadcast=yes driver=brcmfmac driverversion=7.45.154 firmware=01-4fbe0b04 ip=192.168.1.56 multicast=yes wireless=IEEE 802.11

Code: Select all

foo@pi18:/wrk $ sudo ./vl805 
VL805 FW version: 00013701
At some point during the day..

Code: Select all

foo@pi18:/wrk $ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda            8:0    0 931.5G  0 disk 
├─sda1         8:1    0    32G  0 part /
└─sda2         8:2    0 118.9G  0 part 
  ├─pi18-src 254:0    0    32G  0 lvm  /home/foo/usr/src
  └─pi18-gcc 254:1    0     8G  0 lvm  /usr/local/GCC
mmcblk0      179:0    0   7.5G  0 disk 
├─mmcblk0p1  179:1    0   256M  0 part /boot
└─mmcblk0p2  179:2    0   7.2G  0 part

Code: Select all

foo@pi18:/wrk $ sudo smartctl -a /dev/sda
smartctl 6.6 2017-11-05 r4594 [armv7l-linux-4.19.58-v7l+] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     CT1000MX500SSD1
Serial Number:    1921E205464B
LU WWN Device Id: 5 00a075 1e205464b
Firmware Version: M3CR023
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Aug  4 17:58:55 2019 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(    0) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 (  30) minutes.
Conveyance self-test routine
recommended polling time: 	 (   2) minutes.
SCT capabilities: 	       (0x0031)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   000    Pre-fail  Always       -       0
  5 Reallocated_Sector_Ct   0x0032   100   100   010    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       11
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       21
171 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
172 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
173 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
174 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       20
180 Unused_Rsvd_Blk_Cnt_Tot 0x0033   000   000   000    Pre-fail  Always       -       39
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       2
184 End-to-End_Error        0x0032   100   100   000    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
194 Temperature_Celsius     0x0022   060   055   000    Old_age   Always       -       40 (Min/Max 0/45)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   100   100   000    Old_age   Always       -       12
202 Unknown_SSD_Attribute   0x0030   100   100   001    Old_age   Offline      -       0
206 Unknown_SSD_Attribute   0x000e   100   100   000    Old_age   Always       -       0
210 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
246 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       103607648
247 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       1676212
248 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       235816

SMART Error Log Version: 1
Warning: ATA error count 0 inconsistent with error log pointer 2

ATA Error Count: 0
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error -1 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  00 ec 00 00 00 00 00

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ec 00 00 00 00 00 00 00      00:00:00.000  IDENTIFY DEVICE
  ec 00 00 00 00 00 00 00      00:00:00.000  IDENTIFY DEVICE
  ec 00 00 00 00 00 00 00      00:00:00.000  IDENTIFY DEVICE
  ec 00 00 00 00 00 00 00      00:00:00.000  IDENTIFY DEVICE
  c8 00 00 00 00 00 00 00      00:00:00.000  READ DMA

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%        11         -
# 2  Short offline       Completed without error       00%        11         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
One thing I haven't been able to discover is how much power the SSD draws. Except for the fan there's only the SSD and its adapter (it's supposed to be headless). Anyone got a proper datasheet for the CT1000MX500SSD1 - no power specs on what I can find? Would have expected the adapter to have failed there on my linux box as well though plus I got one others on this site have said works.

https://www.linuxquestions.org/question ... 175646890/
https://www.crucial.com/wcsstore/Crucia ... all-en.pdf
https://www.crucial.com/usa/en/support-ssd-firmware

This a typical of it being fixed whilst attached to my linux box..

Code: Select all

foo@sdu ~ $ sudo su -
sdu ~ # parted /dev/sde p
Model: ATA CT1000MX500SSD1 (scsi)
Disk /dev/sde: 1000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  34.4GB  34.4GB  ext4         primary
 2      34.4GB  162GB   128GB                primary  lvm

sdu ~ # e2fsck -n -v /dev/sde1
e2fsck 1.42.13 (17-May-2015)
Warning: skipping journal recovery because doing a read-only filesystem check.
/dev/sde1: clean, 102628/2097152 files, 919738/8388608 blocks
sdu ~ # e2fsck -v /dev/sde1
e2fsck 1.42.13 (17-May-2015)
/dev/sde1: recovering journal
Setting free inodes count to 1994521 (was 1994524)
Setting free blocks count to 7468867 (was 7468870)
/dev/sde1: clean, 102631/2097152 files, 919741/8388608 blocks
I'll grab a powered usb/sata caddy tomorrow.

trejan
Posts: 515
Joined: Tue Jul 02, 2019 2:28 pm

Re: Pi 4, rootfs on SSD goes readonly

Mon Aug 05, 2019 12:32 am

You tried adding the quirks line mentioned on https://www.raspberrypi.org/forums/view ... 8#p1501426 ?

User avatar
HawaiianPi
Posts: 4530
Joined: Mon Apr 08, 2013 4:53 am
Location: Aloha, Oregon USA

Re: Pi 4, rootfs on SSD goes readonly

Mon Aug 05, 2019 3:07 am

I've been running the rootfs on a SATA-III SSD with a USB 3.0 adapter for just shy of a month now with no problems.
  • Pi 4B 2GB.
  • Older 5.1V/2.5A official (Pi3) micro USB power supply with a Volutz brand USB-C adapter.
  • Raspbian Buster with Desktop (2019-07-10).
  • SP Elite 16GB micro SD card (/boot).
  • SP A55 256GB SSD with Eluteng Blue adapter (/ rootfs).
I prepared my system differently than you did. I wrote the Raspbian image directly to the SSD (with Etcher) and then copied the contents of the "boot" partition to a FAT32 formatted micro SD card. No need to edit PARTUUID for the first boot when doing it that way, but I did remove the resize script launch from cmdline.txt (which is known to fail when /boot and / are on different devices).

After booting I edited /etc/fstab to mount the SD card as /boot (so that kernel and firmware updates are correctly applied), followed by a reboot and completing the startup configuration. I installed and used GParted to resize the rootfs partition to fill the unused space, and did a final edit of /etc/fstab to update the PARTUUID (which gets changed when GParted resizes the partition).

An alternate method is to configure everything the way you want it on the SD card first (including booting it and letting the resize script run), and then using the SD card copier to clone the card to the SSD (with New PARTUUID checked). After which I would shut down, reformat the micro SD card (FAT32) and copy the "boot" partition of the SSD to the card. You don't need to manually resize the partition that way, but you should still edit /etc/fstab to mount the SD card as /boot.

Recently I've been testing a Minecraft 1.14.4 SMP server on the 4B2 and I've done a few builds of Spigot without incident. Certainly not as complicated as building GCC, but I've played on the server for many hours without incident. ¯\_(ツ)_/¯

I know saying it works fine for me doesn't help you much, but at least you know it's not a problem everyone is having.

It could be something in the way you prepared your SSD.
It could be the USB-SATA adapter you are using (although mine is also using an ASMedia controller).
It could be the combination of SSD and USB adapter (Pi3 had some problem combinations, but may not be relevant to Pi4).
My mind is like a browser. 27 tabs are open, 9 aren't responding,
lots of pop-ups...and where is that annoying music coming from?

jerrm
Posts: 165
Joined: Wed May 02, 2018 7:35 pm

Re: Pi 4, rootfs on SSD goes readonly

Mon Aug 05, 2019 5:15 am

HawaiianPi wrote:
Mon Aug 05, 2019 3:07 am
It could be the USB-SATA adapter you are using (although mine is also using an ASMedia controller).not be relevant to Pi4).
Unfortunately that doesn't mean much. I've been testing controllers and have three asmedia controllers on my bench now. All with different chip revisions, but report the same vendor device ID. One is complete garbage under Linux, one needs quirks on the Pi, one runs in uas mode without quirks on the Pi out of the box.

My guess is his adapter needs quirks enabled, or is just Linux incompatible.

jerrm
Posts: 165
Joined: Wed May 02, 2018 7:35 pm

Re: Pi 4, rootfs on SSD goes readonly

Mon Aug 05, 2019 2:32 pm

Also meant to mention (was posting when I should have been asleep)...

I have seen ASMedia and JMicron adapters that are solid under PC Linux that have issues on the Pi.

If lsusb -t shows the UAS driver being used, try quirks to disable UAS (even if not needed on PC Linux).

Something is not right with USB3 on the Pi. Some adapters that work under PC Buster fail on the Pi.

swampdog
Posts: 230
Joined: Fri Dec 04, 2015 11:22 am

Re: Pi 4, rootfs on SSD goes readonly

Mon Aug 05, 2019 10:46 pm

Hi Folks,

Thanks all for the useful information. I dug out some powered usb/sata caddies and some old laptop hdd's. It is panning out to be the adapter+ssd combo. Not entirely certain yet: I spent so much time on it that I want to explore every avenue. I'll post proper results later.

Essentially, everything works in every combo(*) except adapter+ssd+Pi4, even with a 840mw laptap hdd plugged into that unpowered adapter so whilst I still cant find out how much juice the ssd draws it isn't going to be that much.

(*) I keep having a mental image of snatching defeat from the jaws of victory. First two caddies wouldn't work with the ssd but fine with hdd's. The hdd's are old though. Fortunately I had purchased a usb/esata caddy for my linux box which refused to work properly with esata. I was going to return it but lost the usb cable & kindof forgot about it. Touch wood .. the Pi4 has the ssd in it as I type and is busy building multiple gcc versions with the ocasional 'dd' off the sdcard->ssd and if it was going to fail it should have done by now. Assuming that works I'll revert to original config and blacklist the UAS via quirks then retest.

swampdog
Posts: 230
Joined: Fri Dec 04, 2015 11:22 am

Re: Pi 4, rootfs on SSD goes readonly

Thu Aug 08, 2019 7:24 pm

Hi Folks, (partial update)

I've now tried four unpowered adapters, none successful. The only joy so far is with a powered caddy. I've set off the Pi4 building gcc 6,7,8,9 (full with tests) so it'll be busy for a while. In the meantime I'm gathering stats. Is there anything I ought to add to this?

Code: Select all

#!/bin/bash

lsusb >pi18.lsusb
lsusb -t >pi18.lsusbT
sudo cat /sys/kernel/debug/usb/devices >pi18.sysKernelDebugUsb.devices
sudo smartctl -ax /dev/sda > pi18.smartctlAX
lsscsi >pi18.lsscsi
sudo parted /dev/sda p >pi18.parted
sudo hdparm -I /dev/sda >pi18.hdparm
cat /boot/cmdline.txt >pi18.cmdline.txt
cat /boot/config.txt >pi18.config.txt
dpkg -l >pi18.dpkg
sudo lsmod >pi18.lsmod
sudo pvs >pi18.pvs
sudo vgs >pi18.vgs
sudo lvs >pi18.lvs
mount >pi18.mount
df -hP >pi18.df
swapon -s >pi18.swapon
sudo cat /var/log/syslog >pi18.syslog
dmesg >pi18.dmesg
sudo cat /var/log/kern.log >pi18.kern.log
sudo cat /var/log/messages >pi18.messages
sudo journalctl -b >pi18.journalctlB
rsync -auv /wrk/HW/ foo@sdu:/wrk/HW/

Also, are this voltages reasonable?

Code: Select all

foo@pi18:/wrk/HW/152d:1561 $ sudo vcgencmd measure_temp && sudo vcgencmd measure_volts core && sudo vcgencmd measure_volts sdram_c && sudo vcgencmd measure_volts sdram_i && sudo vcgencmd measure_volts sdram_p && sudo vcgencmd get_throttled
temp=38.0'C
volt=0.8630V
volt=1.1000V
volt=1.1000V
volt=1.1000V
throttled=0x0

I'm purposely not identifying the unpowered adapters atm because at least one is confirmed to work. Oh, I ought to add firmware into the above (VL805 FW version: 00013701) and note that "uas" is the driver atm.

goodburner
Posts: 41
Joined: Sun Jun 16, 2019 3:20 am

Re: Pi 4, rootfs on SSD goes readonly

Fri Aug 09, 2019 3:58 am

Try disconnecting whatever else you have on the other USB 3 rail. My ssd would disconnect whenever I had something else on the USB 3.0 rail that was drawing a lot of power.

swampdog
Posts: 230
Joined: Fri Dec 04, 2015 11:22 am

Re: Pi 4, rootfs on SSD goes readonly

Sat Aug 17, 2019 10:59 pm

Hi Folks,

Unfortunately the stats I hoped to gather are now useless. Too much has changed. While I was away (longer than I thought) the gcc stress test failed because of a duff network connection downloading gmp-6.1.0 for gcc 7.4.0. I logged in remotely during the week and restarted it from scratch but (a) mucked up the command so everything was done on a single core "make -j1" (b) discovered when I returned the Pi4 fan was playing up so it may have throttled.

This is the unreliable esata caddy which has been rock solid for usb3 on Pi4..
[url]https://www.amazon.co.uk/gp/product/B07 ... &psc=1[url]
..it's a JMicron. Don't expect it to do esata on a linux box.

However, before anyone splashes out £36 on one, bear this in mind..

I have three other JMicron devices attached atm: identical usb3 caddies I bought some time ago (no purchase link, sorry) and one of them behaves differently to to the others. Except it doesn't..

Label these three caddies B0,B1,B2 and (all these are spinning rust disks) the Hitachi plain won't work in B0. The other two disks are Seagate. They work regardless of caddy. Put the Hitachi in B2 and they're all fine. The Hitachi is quite old (nearly 50,000 hours)(*). Dig out a reasonably new Toshiba and it won't even spin up.

I was getting gut feeling about this last week which is why I perhaps went on a bit of a rant in another thread.

(*) Hmm. Powered up in 1882.

More to this than Pi4.

User avatar
rpdom
Posts: 14985
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Pi 4, rootfs on SSD goes readonly

Sun Aug 18, 2019 9:11 am

There was a thread recently that had a link to a page that had some updated firmware for a JMicron USB3-SATA interface and an installer that works on a Pi 2 or above. I used it to upgrade my el-cheapo USB3.0-SATA adaptor and it stopped having the UAS errors that have been reported with some chipsets. Maybe it will fix your problem too?

I can't find that thread on the forums, but I did bookmark the page it linked to for the update program and firmware https://wiki.odroid.com/odroid-xu4/soft ... _fw_update

swampdog
Posts: 230
Joined: Fri Dec 04, 2015 11:22 am

Re: Pi 4, rootfs on SSD goes readonly

Sun Aug 25, 2019 5:42 pm

@rpdom,

That looks like an avenue worth pursuing. It'll have to wait a month though. I missed out on getting my second Pi4 as I wasn't here when the email arrived. I've backordered it this time. My "arc" backup server has failed so the current Pi4 has been pushed into service to do the job. I've fudged it for now with a 3 min cron job to touch a file on all three caddies to prevent them spinning down. Additionally, when I got back the Pi4 fan had failed. I've got a heat sink on order as it's the second time one of those tiny fans has failed. I'm making a note for the future to test what happens if a I just mount a PC fan over the top of the entire board.

Thanks for the link. :-)

User avatar
rpdom
Posts: 14985
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Pi 4, rootfs on SSD goes readonly

Sun Aug 25, 2019 6:36 pm

Having said that, I received another cheap JMicron USB3.0 SATA adaptor yesterday. That one was rubbish. Even after upgrading the firmware it gives IO errors on the HDD I tried it with. The HDD is fine with every other adaptor I've tried it with.

jerrm
Posts: 165
Joined: Wed May 02, 2018 7:35 pm

Re: Pi 4, rootfs on SSD goes readonly

Sun Aug 25, 2019 6:43 pm

rpdom wrote:
Sun Aug 25, 2019 6:36 pm
Having said that, I received another cheap JMicron USB3.0 SATA adaptor yesterday. That one was rubbish. Even after upgrading the firmware it gives IO errors on the HDD I tried it with. The HDD is fine with every other adaptor I've tried it with.
But those JMS578 based adapters have been reliable and bug free (without the hardkernel firmware) for us under PC Linux. We use the 578 adapters because they support TRIM and most other USB3 adapters don't. There is something off (or at least different) with the ARM or Pi USB3 support vs Intel.

On the Pi I can consistently break various JMS578s with the firmware update in uas mode. The update helps, but I don't consider it a reliable fix for the Pi.

User avatar
rpdom
Posts: 14985
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Pi 4, rootfs on SSD goes readonly

Sun Aug 25, 2019 6:50 pm

jerrm wrote:
Sun Aug 25, 2019 6:43 pm
rpdom wrote:
Sun Aug 25, 2019 6:36 pm
Having said that, I received another cheap JMicron USB3.0 SATA adaptor yesterday. That one was rubbish. Even after upgrading the firmware it gives IO errors on the HDD I tried it with. The HDD is fine with every other adaptor I've tried it with.
But those JMS578 based adapters have been reliable and bug free for us under PC Linux. We use the 578 adapters because they support TRIM and most other USB3 adapters don't. There is something off (or at least different) with the ARM or Pi USB3 support vs Intel.
The one I just received gave the same errors on my Intel Linux laptop.

jerrm
Posts: 165
Joined: Wed May 02, 2018 7:35 pm

Re: Pi 4, rootfs on SSD goes readonly

Sun Aug 25, 2019 7:14 pm

rpdom wrote:
Sun Aug 25, 2019 6:50 pm
The one I just received gave the same errors on my Intel Linux laptop.
Just the one problem adapter or 578 adapters in general?

User avatar
rpdom
Posts: 14985
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Pi 4, rootfs on SSD goes readonly

Sun Aug 25, 2019 7:35 pm

jerrm wrote:
Sun Aug 25, 2019 7:14 pm
rpdom wrote:
Sun Aug 25, 2019 6:50 pm
The one I just received gave the same errors on my Intel Linux laptop.
Just the one problem adapter or 578 adapters in general?
Just that one. All my others (and I have several) work fine on Pi and PC.

jerrm
Posts: 165
Joined: Wed May 02, 2018 7:35 pm

Re: Pi 4, rootfs on SSD goes readonly

Sun Aug 25, 2019 7:50 pm

rpdom wrote:
Sun Aug 25, 2019 7:35 pm
Just that one. All my others (and I have several) work fine on Pi and PC.

Try this on the Pi(s) with uas enabled - run a bonnie benchmark and see if /va/log/kern.log shows any bus resets.

Code: Select all

sudo apt-get install bonnie++ 
sudo bonnie++ -u root -d /mnt/path/to/ssd

swampdog
Posts: 230
Joined: Fri Dec 04, 2015 11:22 am

Re: Pi 4, rootfs on SSD goes readonly

Sat Sep 07, 2019 1:53 am

Hi Folks,

Another update.

Put this https://www.amazon.co.uk/gp/product/B07 ... UTF8&psc=1 into this https://www.amazon.co.uk/gp/product/B01 ... UTF8&psc=1 and no data corruption so far.

For cooling I tried this https://shop.pimoroni.com/products/alum ... 0673211475 however it does get a tad warm for my liking (and still throttled compiling gcc using all cores) so I added this https://www.amazon.co.uk/gp/product/B07 ... UTF8&psc=1 which handily can be powered from the Alxum caddy which kept the Pi4 temperature under 50C.

As this "laptop" turned into a "backup server" I'll note before I lose the info https://www.amazon.co.uk/gp/product/B07 ... UTF8&psc=1 seems reliable as a hub for the three spinning rust disks each of which is plugged into their own powered caddy via the Lenovo hub.

Return to “Troubleshooting”