icepick
Posts: 9
Joined: Sun Jul 14, 2013 8:12 am

Dual redundancy raspberry PI

Sat Sep 07, 2019 3:59 am

Hi,
I use my raspberry PI for a home automation application (wrote it myself, not Home assitant etc).. been doing this for many years with great succes. A few weeks ago the SD card of the raspberry died (it's set up as read only but it still died). Wrote a new card and I was up and running again.
Since I have a couple of raspberries laying around I was wondering if there are any scripts out there that support a redundant system setup.. so if one fails, the other one would take over. One solution I can think of is that I use a remote controlled switch that would actually turn off the faulty one and turn on the working one (my raspberry is also monitored by a remote server so this would be an option). But maybe someone has a more efficient way?
Cheers!

ProDigit
Posts: 374
Joined: Tue Aug 30, 2011 1:24 am

Re: Dual redundancy raspberry PI

Sat Sep 07, 2019 9:28 am

I really would recommend Sandisk A1 or A2 microSD cards, they last longer, and run much faster in the Pi.
A second option, try to boot from USB instead (SSD).

The Pi doesn't really have a BIOS that can detect if an SD card went bad or not.
So once it is corrupted, the pi won't boot, or be able to load any kind of script.

User avatar
davidcoton
Posts: 4146
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK

Re: Dual redundancy raspberry PI

Sat Sep 07, 2019 10:26 am

ProDigit wrote:
Sat Sep 07, 2019 9:28 am
I really would recommend Sandisk A1 or A2 microSD cards, they last longer, and run much faster in the Pi.
This is nothing to do with redundancy.
But anyway; A1 yes, A2 no. The Pi does not have the ability to use the A2 spec.
Signature retired

jbudd
Posts: 990
Joined: Mon Dec 16, 2013 10:23 am

Re: Dual redundancy raspberry PI

Sat Sep 07, 2019 10:52 am

Did the Pi die or was it unresponsive but still connected to your network?

swampdog
Posts: 239
Joined: Fri Dec 04, 2015 11:22 am

Re: Dual redundancy raspberry PI

Sat Sep 07, 2019 11:24 am

I have an rpi zero which monitors a power clamp on our fuse box. It's failed for the third time. Kudos to the guy who wrote the code but I knew it would go belly up but have been constantly too lazy to do anything about it. The code writes to a sqlite database which wears out the cards. I should have bodged it such that it was writing to an nfs mountpoint on my nas. However, I always thought "next week" I'll modify the code to write to my maria/mysql database.

Unfortunately it can be tricky to tell if the device has died. Pinging it is not enough. It'll likely succeed even when it's dead. Fortunately you wrote the code so you can decide what happens.

That's not a cop-out. I used to work with ibm aix HA (high availability) clusters and they could get themselves into a real mess. There'd be the normal network plus a "heartbeat" connection. When the heatbeat fails, the current machine would be torn down and another machine bought up to take over - attempting to use the same disks.

Write your code such that your remote server has the data.

User avatar
rpdom
Posts: 15212
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Dual redundancy raspberry PI

Sat Sep 07, 2019 11:37 am

A heartbeat is one way that High Availability systems can work. You have one Live server and one Standby server. The data from the Live server is replicated to the Standby one. The Live one also sends a message to the Standby one every so many seconds with a datestamp to show that it is alive. If the Standby doesn't receive a signal it takes over the role of Live server.

That is an oversimplification, but it could be done with two Pi, even to the extent of changing IP address. I've worked on systems like this some time ago. They even had external hard drives that would switch over when a server went down.

Andyroo
Posts: 4502
Joined: Sat Jun 16, 2018 12:49 am
Location: Lincs U.K.

Re: Dual redundancy raspberry PI

Sat Sep 07, 2019 1:13 pm

One had an SQL system with three boxes in it.
Access was via one and it kept two replicas of the data on the other boxes. Fine till the first box died - then it was a short while till one of the other boxes took the role of master and the cluster fell back to a simple mirrored system in master-slave layout.

I would actually go for a master-slave system using MariaDB and throw a heartbeat either via SQL or MQTT with the slave having the ability to power the master down. A lot simpler than trying to get fancy as you are using the native database function to replicate the data.
Need Pi spray - these things are breeding in my house...

User avatar
rpdom
Posts: 15212
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Dual redundancy raspberry PI

Sat Sep 07, 2019 1:26 pm

Andyroo wrote:
Sat Sep 07, 2019 1:13 pm
I would actually go for a master-slave system using MariaDB and throw a heartbeat either via SQL or MQTT with the slave having the ability to power the master down. A lot simpler than trying to get fancy as you are using the native database function to replicate the data.
Another system I used recently had MariDB in multi-master mode. There were several masters and any one of them could be used at any time. The data replicated to the others quickly, so it didn't matter if any of them failed. Add in a HAProxy front end that forwarded traffic to which ever server was least busy and it was a pretty good system.

Andyroo
Posts: 4502
Joined: Sat Jun 16, 2018 12:49 am
Location: Lincs U.K.

Re: Dual redundancy raspberry PI

Sat Sep 07, 2019 2:05 pm

That proxy sounds neat - we used a round robin server system but it did mean that every now and then the system got hit by delays as a busy server got a pile of extra tasks in the queue - took a long time to isolate and we hit it by adding server VMs due to the cost of getting the dispatcher changed :lol: :o :lol:

The bits I did forget to mention for the OP where:

1) To move off SD and onto HDD or SSD - even a cheap disk and USB adapter should be acceptable performance. Do not mirror it though
2) Move the database into RAM and cut down the R/W access on the SD card (maybe a big problem if power goes off though - may not be though :?: )
3) Look at using an external monitoring service (I use https://uptimerobot.com) to monitor a web page and notify you if its not there - does not have to be fancy. I once did one that said 'I am alive' :lol:
4) Think about maintenance routines - you need to tell the secondary server to take over if the first is being updated / patched / rebooted
5) Think about how you come back - often the master/slave servers swap over when one is down.
6) Run the boxes on different UPS / ring circuits - nothing as embarrassing as having the kettle blow and take out the whole control system :oops:

My Python heartbeat code uses MQTT and is made up of:

Machine up and going down:

Code: Select all

# Tell the master system the machine has started up
def publish_heartbeat_start():
    global Computer

    log('Startup', 'Send MQTT message')
    msg = {
        "Message" : "Startup" ,
        "Device" : Computer ,
        "Date" : logdate()
    }
    mqtt_publish(MqttHeartbeat, msg)

# Tell the master system the machine has shut down
def publish_heartbeat_end():
    global Computer

    log('Ending', 'Send MQTT message')
    msg = {
        "Message" : "ShutDown" ,
        "Device" : Computer ,
        "Date" : logdate()
    }
    mqtt_publish(MqttHeartbeat, msg)
My actual heartbeat (depending on anxiety level :lol: ) is a simple counter:

Code: Select all

def publish_heartbeat(Counter):
    global Computer

    Counter +=1
    if Counter == 256:
        Counter = 1
    log('Heartbeat', 'Send MQTT message ' + str(Counter))
    msg = {
        "Message" : "Alive" ,
        "Device" : Computer ,
        "Counter" : Counter
    }
    mqtt_publish(MqttHeartbeat, msg)
    return Counter
To handle multiple tasks at once I use the threading module (still to play around with multiprocessing) and a queue to dispatch tasks. The heartbeat has its own task that is pushed to the queue by this code running in its own thread:

Code: Select all

# Push queue entry for heartbeat message sending
def push_heartbeat():
    global Computer
    global RunThread
    global Sleeping

    cmd = {
        "Device" : Computer ,
        "Command" : 'Heartbeat'
    }

    while RunThread:
        delay = int(1 * 60 * 2 / WarpSpeed)
        while (delay > 0) and RunThread:
            time.sleep(0.5)
            delay = delay - 1
        if not Sleeping:
            cmdq.put(cmd)
    log('Heartbeat', 'ended by RunThread')
Note for debugging I have a 'WarpSpeed' value in the config file that lets me speed up time on the computer :D Very handy for long running threads...

Sleeping and RunThread are globals that control the threads (crude but works) so I can sleep the machine (nothing is sent via MQTT) or cleanly end (RunThread goes false).

The other end then has to monitor the heartbeat queue for the next value - if its not in sequence (and no end message has been given) then it knows the box has rebooted for some reason or network connection has glitches etc. OK, I know it could fail by sending message 1, reboot, send message 1 but I get two starts in sequence and that is wrong.
Need Pi spray - these things are breeding in my house...

ProDigit
Posts: 374
Joined: Tue Aug 30, 2011 1:24 am

Re: Dual redundancy raspberry PI

Sun Sep 08, 2019 2:13 am

davidcoton wrote:
Sat Sep 07, 2019 10:26 am
ProDigit wrote:
Sat Sep 07, 2019 9:28 am
I really would recommend Sandisk A1 or A2 microSD cards, they last longer, and run much faster in the Pi.
This is nothing to do with redundancy.
But anyway; A1 yes, A2 no. The Pi does not have the ability to use the A2 spec.
The Pi Zero and 3B+ can use the A2, which in some cases is a bit faster than an A1.
Especially on the zero, the A2 is about 10% faster.
I believe the 4B will be able to use more of A2 than the previous versions.

User avatar
davidcoton
Posts: 4146
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK

Re: Dual redundancy raspberry PI

Sun Sep 08, 2019 11:00 am

ProDigit wrote:
Sun Sep 08, 2019 2:13 am
davidcoton wrote:
Sat Sep 07, 2019 10:26 am
ProDigit wrote:
Sat Sep 07, 2019 9:28 am
I really would recommend Sandisk A1 or A2 microSD cards, they last longer, and run much faster in the Pi.
This is nothing to do with redundancy.
But anyway; A1 yes, A2 no. The Pi does not have the ability to use the A2 spec.
The Pi Zero and 3B+ can use the A2, which in some cases is a bit faster than an A1.
Especially on the zero, the A2 is about 10% faster.
I believe the 4B will be able to use more of A2 than the previous versions.
Can you quote a source for that? Because it contradicts what others have posted.
Signature retired

User avatar
davidcoton
Posts: 4146
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK

Re: Dual redundancy raspberry PI

Sun Sep 08, 2019 11:01 am

ProDigit wrote:
Sun Sep 08, 2019 2:13 am
davidcoton wrote:
Sat Sep 07, 2019 10:26 am
ProDigit wrote:
Sat Sep 07, 2019 9:28 am
I really would recommend Sandisk A1 or A2 microSD cards, they last longer, and run much faster in the Pi.
This is nothing to do with redundancy.
But anyway; A1 yes, A2 no. The Pi does not have the ability to use the A2 spec.
The Pi Zero and 3B+ can use the A2, which in some cases is a bit faster than an A1.
Especially on the zero, the A2 is about 10% faster.
I believe the 4B will be able to use more of A2 than the previous versions.
Can you quote a source for that? Because it contradicts what others have posted.
Signature retired

ProDigit
Posts: 374
Joined: Tue Aug 30, 2011 1:24 am

Re: Dual redundancy raspberry PI

Tue Sep 10, 2019 3:34 pm

Well, I'd have to quote myself on this.

I own the 3B+ and zero. Done some fio testing, and the A2 came out ahead generally, but not by much.
The zero for some reason had better as performance than the 3B+ on the A2 card.
On the 3B+ it was pretty much a draw, with the A2 Boeing faster in some and the A1 being faster in others.

The Pi doesn't support an A2 USB card Reader from SanDisk, but the A2 card can be used in a regular card reader, as well as in the internal MicroSD card slot.

https://www.raspberrypi.org/forums/view ... 6&t=247256

Andyroo
Posts: 4502
Joined: Sat Jun 16, 2018 12:49 am
Location: Lincs U.K.

Re: Dual redundancy raspberry PI

Tue Sep 10, 2019 5:18 pm

I wonder if its the test set as the A2 was slower on the 3B but about equal on the 3B+

Not tried it one in a Zero though.

In my mind, they are not worth the extra (esp as my cameras do not handle the A2 cards as anything special).
Need Pi spray - these things are breeding in my house...

ProDigit
Posts: 374
Joined: Tue Aug 30, 2011 1:24 am

Re: Dual redundancy raspberry PI

Tue Sep 10, 2019 8:44 pm

True. However, the A2 is only a dollar or two more.
The benefit is, when you're using a laptop that accepts the A2 Sandisk cardreader.
It will dd image at ~70MB/s instead of ~7MB/s on the A1.

Return to “General discussion”