FalcoGer
Posts: 37
Joined: Mon Jul 09, 2018 6:35 am

Remote Emergency Mode, can not connect

Thu Jul 19, 2018 11:17 am

I'm using a Raspberry Pi with Raspbian (kernel 4.14.34-v7) in a remote (very remote) setup to log some data.
I'm using an usb drive for booting and writing the data (m2 ssd adapter), no sd card is present.
The power was cut of dirtily without the pi shutting down properly.

after recreating the issue a few times here, I got this lovely screen (See attachment)
I assume it's a disk error due to the dirty shutdown.

The issue is I was unable to login via ssh to fix the problem.
After a reboot by cutting power and restarting (had to call them to cycle the power for us) it worked again. but I wish for a solution that allows me to fix things or to avoid this.
for example to try default mode again or to reboot after some time, or to start into something that allows ssh connections.
also it has a static ip and I connect via vpn that's handled by the remote router.

I think the problem also is the fact that you have to 'press enter to continue', which is a bit hard with the pi hanging in a locked basement and with no keyboard or screen attached.

Journal: https://drive.google.com/open?id=1hjuZT ... nYWBph0ioz

ps: I'd like to upload the journal here, but apparently you are only allowed to upload pictures, and the character limit for a post is 60 000.
Attachments
BootFail.jpg
BootFail.jpg (253.23 KiB) Viewed 175 times

FalcoGer
Posts: 37
Joined: Mon Jul 09, 2018 6:35 am

Re: Remote Emergency Mode, can not connect

Wed Jul 25, 2018 8:54 pm

Any advice?

ejolson
Posts: 2482
Joined: Tue Mar 18, 2014 11:47 am

Re: Remote Emergency Mode, can not connect

Thu Jul 26, 2018 5:06 am

FalcoGer wrote:
Wed Jul 25, 2018 8:54 pm
Any advice?
I've currently got a similar issue with a remote firewall: It didn't come back up after sudden power loss. I'll fix it in a month.

The problem with Raspbian, the Debian operating system on which it's based and most other Linux distributions is that they fail to configure networking if the root filesystem is damaged. In the data center people can log in remotely using IPMI and set things straight again, but with Raspberry Pi they can't.

One idea would be to create an initial RAM filesystem that sets up networking at the beginning with some sort of system monitor accessible through ssh that could be used to remotely log in and fix a damaged root filesystem.

Another idea would be to network boot the Pi and put the root filesystem on a server that can be easily accessed in case something goes wrong.

A final idea is to use a read-only root filesystem that is less likely to get damaged in the event of a sudden power loss in the first place.

I'm actually a little surprised that systemd doesn't include a built-in version of sshd that can be activated in case errors prevent a proper system boot. Does anyone know enough about systemd to confirm whether such a feature has been added or not?

Return to “Troubleshooting”