hulu60
Posts: 3
Joined: Sun Sep 01, 2019 8:35 pm

reset a frozen system remotely

Sun Sep 01, 2019 8:53 pm

Hi,
I have a remote raspberry pi board which normally is not accessible physically by any person. For worst case situations, I need to have a solution to recover the system from:

1- frozen OS: how to have a hard reset?
2- a power off system: how to turn on a fully off system? suppose we run "sudo poweroff". now how to turn it on remotely?

The solution currently I think about is to put a secondary small micro-controller board equipped with a battery as watch-dog processor to constantly check liveliness of raspberry (like checking a heartbeat signal on raspberry GPIOs). when detecting that raspberry is not live, apply a hard reset (how?).

Is there any better idea? or is there any special circuit to handle this situations?

drgeoff
Posts: 9804
Joined: Wed Jan 25, 2012 6:39 pm

Re: reset a frozen system remotely

Mon Sep 02, 2019 8:46 am

About the hard reset, do some research into the RUN connection on a RPi board..

User avatar
thagrol
Posts: 1836
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: reset a frozen system remotely

Mon Sep 02, 2019 10:38 am

drgeoff wrote:
Mon Sep 02, 2019 8:46 am
About the hard reset, do some research into the RUN connection on a RPi board..
Or the hardware watchdog.
This space unintentionally left blank.

Heater
Posts: 13299
Joined: Tue Jul 17, 2012 3:02 pm

Re: reset a frozen system remotely

Mon Sep 02, 2019 11:17 am

drgeoff,

I have been using the Pi since the beginning and Linux for far longer but I have no idea what you mean by the "the RUN connection on a RPi board". My research with google did not turn up anything interesting.

What on Earth are you talking about?

I suspect you may mean this reset line labeled RUN that appears at a hole on the Pi boards labeled "RUN". https://raspberrypi.stackexchange.com/q ... berry-pi-2

For remote systems like this I have used the on board watchdog facility to good effect. https://www.domoticz.com/wiki/Setting_u ... i_watchdog

A crucial thing to have for such a remote system is a root file system that cannot be corrupt on random power downs and resets. To this end ensure your root file system is mounted read-only. There are many discussions about this here on the forum.

Of course if your Pi is part of a system that includes GSM modems or whatever you may want to protect yourself against those getting stuck as well. In which case a little 8 bit micro as an external hardware watchdog that can power cycle the whole system is a good idea. Or other hardware solution.

I have also seen systems where the hardware logic itself would get jammed and a simple reset whilst powered up did not unjam them. Again that might call for a watchdog that can power cycle.

hulu60
Posts: 3
Joined: Sun Sep 01, 2019 8:35 pm

Re: reset a frozen system remotely

Mon Sep 02, 2019 12:40 pm

Thanks to all,
@Heater,
As I understand, in your solution, the watchdog software is installed on the same raspberry pi, not on a redundant one, right?. That is a cheap and useful solution without any cost for many cases, but anyway, if raspberry is totally frozen, or if it turn off completely, who will turn it on again? (for example if I accidentally run "sudo poweroff")

Do you agree that for critical products, there should be an external redundant hardware (as you said like 8-bit micro-controller) to take care of aliveness of raspberry? In that case, will sending a pulse on "RUN" pin power on a fully off system? or it only resets an already powered on system?

drgeoff
Posts: 9804
Joined: Wed Jan 25, 2012 6:39 pm

Re: reset a frozen system remotely

Mon Sep 02, 2019 12:53 pm

Heater wrote:
Mon Sep 02, 2019 11:17 am
drgeoff,

I have been using the Pi since the beginning and Linux for far longer but I have no idea what you mean by the "the RUN connection on a RPi board". My research with google did not turn up anything interesting.
@heater

Something wrong with your use of Google?

https://www.raspberrypi.org/forums/view ... hp?t=22690
https://www.raspberrypi.org/forums/view ... p?t=224024
https://www.raspberrypi.org/forums/view ... p?t=228605
https://www.raspberrypi.org/forums/view ... p?t=140077
https://www.raspberrypi.org/forums/view ... hp?t=82963
https://www.raspberrypi.org/forums/view ... p?t=218600

Andyroo
Posts: 4467
Joined: Sat Jun 16, 2018 12:49 am
Location: Lincs U.K.

Re: reset a frozen system remotely

Mon Sep 02, 2019 1:02 pm

Though rare, a software watchdog can hang so if the machine is that critical then look to a zero SD card Linux and an external watchdog board. There are many on Tindie and from the normal suppliers.

Personally I hate using computers for mission critical work but then again I think pigeon post beats the internet any day :lol:
Need Pi spray - these things are breeding in my house...

User avatar
rpdom
Posts: 15184
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: reset a frozen system remotely

Mon Sep 02, 2019 1:09 pm

Andyroo wrote:
Mon Sep 02, 2019 1:02 pm
Though rare, a software watchdog can hang so if the machine is that critical then look to a zero SD card Linux and an external watchdog board. There are many on Tindie and from the normal suppliers.
The Pi's Watchdog is hardware. The software resets the watchdog on a regular basis. If the watchdog is not reset in a certain time frame it will perform a hard reset of the Pi.

Andyroo
Posts: 4467
Joined: Sat Jun 16, 2018 12:49 am
Location: Lincs U.K.

Re: reset a frozen system remotely

Mon Sep 02, 2019 1:36 pm

Thought it needed the Linux software driver to activate?
Need Pi spray - these things are breeding in my house...

User avatar
rpdom
Posts: 15184
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: reset a frozen system remotely

Mon Sep 02, 2019 1:49 pm

Yes, you do need to enable it in the first place, but once enabled it will just keep running in hardware.

The only thing the Linux driver needs to do after that is keep poking it to say "I'm not dead yet". When the doggie fails to get that message in a reasonable time it will force a reset.

Aydan
Posts: 692
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: reset a frozen system remotely

Mon Sep 02, 2019 1:52 pm

Yes it does, but if you can't even get the kernel and some services loaded, then an external reset won't help you either.
The on chip watchdog is activated by the first access to it and cannot be stopped again. This makes it ideal to reset a hung system. It will do nothing for a system that does not boot.

Regards
Aydan

hulu60
Posts: 3
Joined: Sun Sep 01, 2019 8:35 pm

Re: reset a frozen system remotely

Mon Sep 02, 2019 2:04 pm

if it is a timer (hardware) that is great! and there is no need to have an external wathdog module to apply a hard reset.
(anyway, if the source of problem is not software, an external reset will not help either).

But what if I accidentally run "sudo poweroff" remotely ? :D can this timer save me too, or it will turn off too?

User avatar
RaTTuS
Posts: 10458
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: reset a frozen system remotely

Mon Sep 02, 2019 2:08 pm

hulu60 wrote:
Mon Sep 02, 2019 2:04 pm
...
But what if I accidentally run "sudo poweroff" remotely ? :D can this timer save me too, or it will turn off too?
no
that will switch it off
remove the power off command [or just dont use it ]
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

Heater
Posts: 13299
Joined: Tue Jul 17, 2012 3:02 pm

Re: reset a frozen system remotely

Mon Sep 02, 2019 2:30 pm

Aydan,

Well, if your boot media is so mangled it won't start up properly than no watchdog, hardware or software will help.

At some point, if the application is really critical, you will need a multiple redundant system. Say 3 Pi units all working together, if one fails the others can continue and still produce correct results. Except the solution to the Byzantine Generals problem (google it) shows that to tolerate one failed unit you actually need four to start with.

If you do go for a 8 bit micro or such as an external watchdog that can power cycle everything, be sure to enable it's own watch dog!

epoch1970
Posts: 3661
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: reset a frozen system remotely

Mon Sep 02, 2019 2:34 pm

An external watchdog can be useful:
- when you sudo poweroff in the wrong window :oops:
- when the system gets wedged during boot up and the built in watchdog is not activated yet (only useful assuming a custom boot loader that counts boot failures and can do something in case of repeated errors)
- when (if) the hardware watchdog for some reason does not reset the platform fully like a cold boot would

Iff the external watchdog is more reliable than the Pi of course.
All in all I think the application window for an external watchdog is pretty narrow.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

Return to “General discussion”