User avatar
davidcoton
Posts: 4035
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK

Re: Houseboat home automation / surveillance system

Tue Aug 29, 2017 10:14 pm

(Disclaimer: I haven't used Node-RED or MQTT)
What history do you want to save (rhetorical question)? Is it all in the MQTT messages? If so, it shouldn't be too hard to write a (Python?) modules to subscribe to all messages and copy them to a log. A bit more work, and they can be kept in a database, which should make selective recall easier.
Signature retired

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Tue Aug 29, 2017 10:46 pm

davidcoton wrote:
Tue Aug 29, 2017 10:14 pm
(Disclaimer: I haven't used Node-RED or MQTT)
What history do you want to save (rhetorical question)? Is it all in the MQTT messages? If so, it shouldn't be too hard to write a (Python?) modules to subscribe to all messages and copy them to a log. A bit more work, and they can be kept in a database, which should make selective recall easier.

Yep, that's pretty much what I had in mind; a something which subscribes to everything and pushes this into a database of some sort (append only is fine here). But. I then need to have some kind of job that runs periodically, and "thins out" the historical data, by creating aggregates of min/max/avg values - otherwise the amount of data will soon become unmanageable; I'm trying to be restrictive and don't have anything that refreshes more often than once a minute, but that's already quite a few dozens of parameters, and well, yeah, do the maths: 100 parameters x 60 times an hour x 24 hours a day x 365 days a year... that's 52,560,000* datapoints per year :o And while averaging numerical values is simple, many parameters are strings. What's the "average" of "LTE", "HSPA", "UMTS", "EDGE", and "GPRS"!? How do you "average" your position - that doesn't even seem to make sense? And then there's the even more challenging task of getting Node-RED to load this historical data, so it can be used in my flows, and explored via dashboards. My head hurts just thinking about it. I also want this data to be replicated over the VPN, not only for backup purposes, but also so that "external" dashboards can be hosted in the "cloud" where they won't tax the onboard systems (and perhaps selectively be made public). That bit should be fairly straightforward though.


*) Ok, so yeah, bit of an exaggeration there, since many values are only "published" when they change, but still; this thing generates a lot of data.

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Wed Aug 30, 2017 12:04 am

Quick note to anyone who's trying to follow this thread: the thread has paged and my (lengthy) post from earlier this evening can be found at the bottom of the first page. I just hate when that happens :roll:

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Wed Aug 30, 2017 10:38 am

Here's a screengrab of the flows which collect and publish all the data used on the dashboard in my earlier post.
Screenshot_2017-08-30_18-13-05.png
Screenshot_2017-08-30_18-13-05.png (73.39 KiB) Viewed 3684 times

And look what just arrived in the mail!
monarco.jpg
monarco.jpg (148.61 KiB) Viewed 3733 times

A beautifully made board, I must say. Nice touch with the adhesive legends. Cursed paying clients demanding my attention when I have such shiny toys to play with! Tonight, my darling, I will be plugging you in. :twisted:

Edit: Updated screengrab of flows, to include the time setting command.
Last edited by Lomax on Wed Aug 30, 2017 5:23 pm, edited 3 times in total.

stevend
Posts: 211
Joined: Fri Oct 11, 2013 12:28 pm

Re: Houseboat home automation / surveillance system

Wed Aug 30, 2017 12:35 pm

Getting MQTT data from Python (and, indeed, C) is very straightforward. You may need a number of ways to parse the payload, depending on what's being sent, but shouldn't be too bad.

I'd suggest that the "average" of "LTE", "HSPA", "UMTS", "EDGE", and "GPRS" is the percentage of time for which each is active over a defined period.

As for the average of position - since you're a houseboat, surely if it changes you have a problem? (Or maybe you could provide a local differential GPS capability?)

User avatar
davidcoton
Posts: 4035
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK

Re: Houseboat home automation / surveillance system

Wed Aug 30, 2017 3:20 pm

Lomax wrote:
Tue Aug 29, 2017 10:46 pm
Yep, that's pretty much what I had in mind; a something which subscribes to everything and pushes this into a database of some sort (append only is fine here). But. I then need to have some kind of job that runs periodically, and "thins out" the historical data, by creating aggregates of min/max/avg values - otherwise the amount of data will soon become unmanageable; I'm trying to be restrictive and don't have anything that refreshes more often than once a minute, but that's already quite a few dozens of parameters, and well, yeah, do the maths: 100 parameters x 60 times an hour x 24 hours a day x 365 days a year... that's 52,560,000* datapoints per year :o
Humanly speaking, lots! In computer terms, not so much. Say 50 bytes per datapoint, 2.5GB per year. 100 years of data on one quite small hard disk. (plus another two for backup). While you may not want to store that indefinitely, processing it to reduce the volume and get whatever views of the data you want is not critical to the initial deployment. (Obviously you don't want to pass it all over expensive data links -- but maybe you do want much of it (tracking especially) ashore anyway).

Oh, and while we're on storing time and locatiion, xkcd today seems relevant. :lol:
Signature retired

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Wed Aug 30, 2017 10:56 pm

davidcoton wrote:
Wed Aug 30, 2017 3:20 pm
In computer terms, not so much. Say 50 bytes per datapoint, 2.5GB per year. 100 years of data on one quite small hard disk. (plus another two for backup).

Very true, but in addition to storage requirements, data has other "weights"; transfer, processing and presentation.
davidcoton wrote:
Wed Aug 30, 2017 3:20 pm
While you may not want to store that indefinitely, processing it to reduce the volume and get whatever views of the data you want is not critical to the initial deployment. (Obviously you don't want to pass it all over expensive data links -- but maybe you do want much of it (tracking especially) ashore anyway).

Yeah, you're right. As long as it gets logged, and in a way which won't require too much work once I want to start using the data, I'm happy to deploy without data consolidation. Longer term, having previously used Munin fairly extensively, it would be interesting to explore the possibility of integrating Node-RED with RRDTool, which does this very well.
davidcoton wrote:
Wed Aug 30, 2017 3:20 pm
Oh, and while we're on storing time and locatiion, xkcd today seems relevant. :lol:

It's funny because it's true! :D

I didn't get round to plugging in the Monarco Hat tonight (sorry darling, got a bit of a headache), mainly because I got stuck dragging nodes around in Node-RED to make it all pretty, aspie style. I did however manage to move the last metrics, those coming from the OpenUPS, into Node-RED, courtesy of node-red-contrib-nut-ups. Consequently Domoticz has now been removed entirely from the system :shock: This freed up a bit of RAM, with the Pi now idling at ~25% utilisation, with a ridiculously low one minute load average of just 0.04. I am happy with this. Some screengrabs:
telemetry_flows.png
telemetry_flows.png (92.86 KiB) Viewed 3654 times
Parameter ingestion flows - note the addition of a pink "UPS" node
dashboard_flows.png
dashboard_flows.png (139.87 KiB) Viewed 3654 times
Dashboard flows - I worried that having so many MQTT subscribers would impact performace, but it doesn't - not even a little bit
dash_2017-08-30_23-53-45.png
dash_2017-08-30_23-53-45.png (108.36 KiB) Viewed 3654 times
What the main dashboard currently looks like

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Wed Aug 30, 2017 11:33 pm

stevend wrote:
Wed Aug 30, 2017 12:35 pm
Getting MQTT data from Python (and, indeed, C) is very straightforward. You may need a number of ways to parse the payload, depending on what's being sent, but shouldn't be too bad.

It's not how you store the data, but what you do with it that counts. ;) But yes, either some little Python thing, or one of the many DB connector nodes available for Node-RED, will probably be the way forward. C is not really my thing. Wish it was, just never had a strong enough reason to go there. Did some Atmel C for the ARV line of MCUs and enjoyed it, but then the Pi came and, well... And it did upset me a bit when Microchip gobbled up Atmel too, almost felt personal. Turned me off. I'm weird like that.
stevend wrote:
Wed Aug 30, 2017 12:35 pm
I'd suggest that the "average" of "LTE", "HSPA", "UMTS", "EDGE", and "GPRS" is the percentage of time for which each is active over a defined period.

Perhaps.
stevend wrote:
Wed Aug 30, 2017 12:35 pm
As for the average of position - since you're a houseboat, surely if it changes you have a problem? (Or maybe you could provide a local differential GPS capability?)

Errrm... noooooo... It's... a boat. Which means it can move. On water. In fact I would say that other than keeping you dry, that's kind of the point with its floating ability. Not moving your houseboat would be like living in a car and never drive it. Never understood the permanent mooring types; why not just buy a flat? Living in a houseboat is a massive compromise in so many areas; space, comfort, privacy, and dare I say: hygiene, just to mention a few. Why put up with all that if you're never going to enjoy its main benefit - being able to move around?

Heater
Posts: 13119
Joined: Tue Jul 17, 2012 3:02 pm

Re: Houseboat home automation / surveillance system

Wed Aug 30, 2017 11:56 pm

Hmmm....average position.

I guess if you travel in a straight line from A to B you average position might be C. Somewhere in the middle of the line joining A and B. Not very useful.

If you are traveling in an arc or complete circle your average position might be somewhere in the center. A place you have never actually been anywhere near !

In some frame of reference my average position over a day is at the center of the Earth!

Is "average position" even a meaningful statistic in these cases?

Might be meaningful if you are supposed to be stationary at some point but are randomly jiggling around it.

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Wed Aug 30, 2017 11:58 pm

Heater wrote:
Wed Aug 30, 2017 11:56 pm
Hmmm....average position.

[thoughts]

Love it! :) Yes, some kind of filter on the position might be good, but then I'm thinking I still want to log it continuously (every minute), since it's such an interesting, and relevant, metric. I already have a notification going out via SMS if she moves, and there's a lot of other stuff that can be done with position data - even if it's noisy.

Edit: Hell, I was disappointed to see how noisy and inaccurate the elevation data was from the RV50's GPS. I had hoped it could be used to log the boat being launched (big day), or going up on land for repairs (less so) - maybe even tidal fluctuations. Alas, the data was so poor I'm not even bothering to publish it as an MQTT topic.

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Thu Sep 07, 2017 8:01 pm

I know it's been a while since I posted an update - but that's not because "things" haven't been happening, in fact quite the opposite; between (paid) work and boaty things this has been a pretty crazy week. Here's a quick run-through of the latest developments:

I got the Monarco Hat last week, but it took a couple days before I got round to installing it. When I eventually did so, I immediately saw the value of the detachable connector, which makes it very easy to connect things up in the cramped enclosure. So far I have only connected the power input, the 1-Wire bus and the RS-485 bus. For the two latter I've been using Alpha Wire's 2461C, 0.2mm2 single twisted pair with foil shield and drain wire. This is a nice cable for a lot of things; not only 1-Wire and RS-485 but also audio and any other signalling where a shield may prove useful (basically, everything, considering the environment). It has tinned copper conductors, which is good for corrosion resistance, and the outer diameter is just 3.6mm, so it won't stuff up the cable conduits too much.

After quite a bit of fiddling with getting an RJ11 plug crimped onto the 0.2mm2 conductors, I was able to hook up my AAG/Dallas weather station, which provides air temperature, wind speed and wind direction data over 1-Wire. It worked straight away using the node-red-contrib-owfs Node-RED node. But while the thermometer reports the temperature as a simple float (in Celsius), the wind data returned needed quite a lot of processing to turn it into the actual numbers. First I split the incoming data based on which sensor it's coming from (yeah, I know, hard-coded device ids; bad bad bad, but hey):

Code: Select all

var device = msg.topic.split("/");
var msgs = [[],[],[]];

if(device[0] == "1D.005A01000000") {
    msg.topic = "owfs/wind/speed";
    msgs[0].push(msg);
} else if(device[0] == "20.E63307000000") {
    var tag = device[2].replace("volt\.","");
    var obj = {};
    obj[tag] = msg.payload;
    msg.topic = "owfs/wind/direction";
    msg.payload = obj;
    msgs[1].push(msg);
} else if(device[0] == "10.8B401B000800") {
    msg.topic = "owfs/temperature";
    msgs[2].push(msg);
}
return msgs;

The wind direction is sensed by a four channel ADC (A, B, C and D) connected to some reed switches, and it's configured so that each channel will see a "high", "medium" or "low" voltage depending on the orientation of the wind vane, resulting in a precision of 16 steps. I hacked together the following Node-RED function to turn this into a compass direction:

Code: Select all

var H = [3.5, 5];
var M = [1.5, 3.49];
var L = [0, 1.49];
var V = [msg.payload["A"], msg.payload["B"], msg.payload["C"], msg.payload["D"]];
var points = [
    [H,H,M,H],
    [H,M,M,H],
    [H,M,H,H],
    [M,M,H,H],
    [M,H,H,H],
    [M,H,H,L],
    [H,H,H,L],
    [H,H,L,L],
    [H,H,L,H],
    [H,L,L,H],
    [H,L,H,H],
    [L,L,H,H],
    [L,H,H,H],
    [L,H,H,M],
    [H,H,H,M],
    [H,H,M,M]
]
for(var i=0; i<points.length; i++) {
    if(V[0] > points[i][0][0] && V[0] < points[i][0][1] && 
       V[1] > points[i][1][0] && V[1] < points[i][1][1] && 
       V[2] > points[i][2][0] && V[2] < points[i][2][1] &&
       V[3] > points[i][3][0] && V[3] < points[i][3][1]) {
        msg.payload = i * 22.5;
        return msg;
    }
}

Not the prettiest code, particularly that "if" statement is rather messy (if you have a more elegant suggestion I'd be interested to know about it), but it works quite well. One quirk is that the "direction" of course is relative to the boat; at some point I might combine that with NMEA-0183 compass data to get the actual wind direction. Doesn't seem at all important right now though! The wind speed data also needs a function, to turn the pulse counts into m/s. I found the formula for this in a Java library originally written by AAG themselves, so hopefully it should be fairly accurate. The calculation needs a previous sample to compare with, so I store this in the node's "context":

Code: Select all

var previous = context.get("sample");

context.set("sample", [Date.now(), msg.payload]);

if(previous) {
    var time = Date.now();
    msg.payload = (((((msg.payload - previous[1]) * 1000.0) / (time - previous[0])) / 2.0) * 2.453).toFixed(1);
    return msg;
}

As you can probably see I'm quite fond of using arrays, even in cases where most people would opt for a JS object. I'm strange like that. All three of these parameters are then published as MQTT messages, to be read by the (new) "Environment" dashboard (more on that further down), and anyone/anything else that is interested.

Having got the 1-Wire weather station to work, I wanted to see how it compared to weather data from real weather stations in the area, so I signed up for an OpenWeatherMap API key and installed the openweathermap node from node-red-web-nodes. I changed the NMEA GPS data flow to include a function which sets two global variables to the current lat/long, and I use these when querying the OWM API (rounded to two decimal points, lest the black helicopters will know my exact address). This works very well, though I've run into issues with how to deal with the OWM forecast API data, which returns eight 3-hour segments per day, for the next five days - just wanting to capture the basics (min/max temp, wind speed, wind direction, weather description) would result in 8 x 5 x 5 = 200 messages every time the forecast is ingested - clearly excessive. That puts into question a fundamental design decision: to never publish more than one piece of data under any one MQTT topic. Consequently, as I'm currently unable to pick a path forward (JSON? arrays? averages?), I have left this out for the time being.

Here's is what the data ingestion flows currently look like:
ingestion_flows_170907.png
ingestion_flows_170907.png (140.93 KiB) Viewed 3456 times

Another piece of telemetry data I've been wanting to include is a 3D accelerometer. Being a boat, this is a very interesting measurement, as it will tell me something about the sea conditions, as well as the boat's trim & heel. I figure it might also be possible to use this to detect events such as someone stepping aboard, other boats bumping into her, or excessive "tugging" while moored. After a bit of research, I picked the ADXL345 from Analog Devices, specifically this breakout board from Adafruit. In order to be able to talk to it, I added the node-red-contrib-i2c node, which handles i2c communication (based on the node-I2C package). The ADXL345 is an interesting chip as it provides two interrupts, which can be hooked up to GPIO pins and used to trigger events in Node-RED. I don't want to be continuously sampling the accelerometer data and process it in code just to see if anything has "happened", so the "on-chip interrupts" seem like a great solution.

The first challenge was mounting it so that it has a rigid connection with the enclosure and (ideally) is oriented the right way up. There didn't seem to be any suitable place in the enclosure, but then I had a brainwave: why not attach it upside-down to a mount which is also upside down? So I made a small aluminium plate with two stand-offs tall enough to leave some room for the chips and glued it firmly to the enclosure with JB-Weld (my favourite 2-pack glue):
accelerometer.jpg
accelerometer.jpg (208.05 KiB) Viewed 3456 times

This worked very well, and I'm quite pleased with the result; the board is very rigidly attached, "right" way up and well out of the way, while being easily removable and in a location which it is easy to route the wires to. I added the third screw only to ensure a level base while the glue set; it doesn't have any other function. Again, the data coming from the accelerometer is not immediately usable, so we need another function to turn it into something understandable - this time the formula comes courtesy of a Python script for the ADXL345, originally released by Adafruit but there's a fork on GitHub which adds support for the interrupts as well. This saved me a lot of time and allowed me to come up with the following Node-RED function in minutes:

Code: Select all

var multiplier = 0.004;
var msgs = [{},{},{}];

bytes = msg.payload;

x = bytes[0] | (bytes[1] << 8);
if(x & (1 << 16 - 1)) {
    x = x - (1<<16);
}

y = bytes[2] | (bytes[3] << 8);
if(y & (1 << 16 - 1)) {
    y = y - (1<<16);
}

z = bytes[4] | (bytes[5] << 8);
if(z & (1 << 16 - 1)) {
    z = z - (1<<16);
}

x = x * multiplier;
y = y * multiplier;
z = z * multiplier;

msgs[0].topic = "mother/orientation/x";
msgs[0].payload = parseFloat(x.toFixed(4));
msgs[1].topic = "mother/orientation/y";
msgs[1].payload = parseFloat(y.toFixed(4));
msgs[2].topic = "mother/orientation/z";
msgs[2].payload = parseFloat(z.toFixed(4));

return [msgs];

Not only that, but by exectuing this function on load, I can tell the ADXL345 to enable interrupt #1 and set some parameters for it (big thanks to Sanjay Deshmukh!):

Code: Select all

var address = 0x53; 

var threshAct = [0x24, 0x05]; // sensitivity, x 62.5 for mg
var actInactCtl = [0x27, 0xF0]; // activity sens axis & mode
var actEnable = [0x2E, 0x10]; // enable activitiy interrupt
var intMap = [0x2F, 0x0]; // interrupt pin to use (1/2)
var powerCtl = [0x2D, 0x08]; // enable measurement

var sequence = [threshAct, actInactCtl, actEnable, intMap, powerCtl];

var msgs = [];

for(var i=0; i<sequence.length; i++) {
	msgs.push({
	    "address":parseInt(address), 
	    "command":parseInt(sequence[i][0]), 
	    "payload":parseInt(sequence[i][1])
	})
}

return [msgs];

Amazingly, it just works; if I tap the enclosure the interrupt gets triggered, pulling pin 13 high. This is picked up in Node-RED which then triggers another i2c call, resetting the interrupt, ready to detect the next tap. It's almost like magic. In case you're wondering why I convert the hex values to decimal integers, it's because the node-red-contrib-i2c node expects decimal values, for some unfathomable reason. I may have to fork and fix. As I mentioned earlier, all this new data is coming together in a new dashboard I call "Environment", which also includes the GPS data and camera still images. Very much work in progress, but this is what it currently looks like (GPS, 1-Wire weather data, OWM weather data, and still image(s) plus accelerometer data):
Screenshot_2017-09-07_20-41-11.png
Screenshot_2017-09-07_20-41-11.png (73.92 KiB) Viewed 3456 times

The full-scale events on the accelerometer graph is me turning the enclosure upside down, then laying it on one side, followed by the other side, followed by laying it on its back (I can't lay if front down as the screws holding the front battery plate are not currently installed). Seems pretty accurate; I get close to "1g" in whatever direction is "down". This is followed by me walking around heavily near the enclosure, though this signal is drowned by the previous extremes.

I think I deserve a beer now :P

Edit: Updated the ADXL345 initialisation function - it needed to set POWER_CTL (0x2D) to 0000 1000 (0x08) to enable the device after power up (the ADXL345 starts in standby mode). Took the opportunity to dry up the code while I was at it.
Last edited by Lomax on Sun Sep 10, 2017 9:22 pm, edited 10 times in total.

Heater
Posts: 13119
Joined: Tue Jul 17, 2012 3:02 pm

Re: Houseboat home automation / surveillance system

Thu Sep 07, 2017 8:18 pm

Indeed you do. Well done.

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Thu Sep 07, 2017 9:18 pm

Heater wrote:
Thu Sep 07, 2017 8:18 pm
Indeed you do. Well done.

Thank you! It tastes great :) While enjoying my beer, I'll just post a few more images which didn't make it in my update, since the forum only seems to allow three per post.

These are the flows used to talk (and listen) to the ADXL345 accelerometer:
accelerometer_flows.png
accelerometer_flows.png (35.19 KiB) Viewed 3426 times

Here is the reason why the weather data from the AAG 1-Wire station is such a poor match to the OWM data :D :
fake_weather.jpg
fake_weather.jpg (85.6 KiB) Viewed 3421 times

Finally, and I know I keep coming back to this, that cable entry system is really proving its worth - this would have been a nightmare without it:
cable_entry_populous.jpg
cable_entry_populous.jpg (163.96 KiB) Viewed 3426 times

User avatar
davidcoton
Posts: 4035
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK

Re: Houseboat home automation / surveillance system

Thu Sep 07, 2017 9:27 pm

And for your next trick -- the robot bar tender should automatically know when you have earned a beer, and have it served up when you walk into the bar area.
After all that coding, having to pour your own beer is just too much :lol:
Signature retired

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Thu Sep 07, 2017 9:34 pm

davidcoton wrote:
Thu Sep 07, 2017 9:27 pm
And for your next trick -- the robot bar tender should automatically know when you have earned a beer, and have it served up when you walk into the bar area.
After all that coding, having to pour your own beer is just too much :lol:

What a great idea :) Maybe I could have a solenoid valve connected up, listening to Travis (no, not the band) and rewarding me with a quantity of beer relative to the number of lines committed without any build failures... Actually, on second thoughts, that's a bit scary. Anyone seen Zero Theorem?

I actually visited "Cynthia's" a few times back in the day - that was an amazing place. :cry:

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Sun Sep 10, 2017 9:59 pm

Earlier in the week I finally had a reply from mini-box about the issues I'd been having with the OpenUPS - they suggested a firmware update, with their latest firmware (v1.9 July 2017). I have just performed this update, while leaving all settings at their default values, and early indications are the problem has gone away! I have just done an hour long discharge test on the battery, and when restoring external power it correctly identified that the battery needed to be charged, performed a bulk charge to 100% and then went into float charge. No sign of the strange and frustrating mode cycling seen earlier, even after a few shorter external power cut-outs. Result! That leaves only one of my three "show stopper" issues to resolve: getting the VPN to work.

I've also played a bit more with the accelerometer, and I've made a flow in Node-RED which samples the output once a second, but only passes through the messages to the chart if any axis has seen an acceleration > 0.03 G, while a side-chain function always allows one message per minute through. This gives me a fairly detailed rate when something is happening (one sample per second), without killing the graph with too much data, while the "idle" sample rate of one per minute allows me to still see slower changes in roll/trim:
Screenshot_2017-09-10_22-43-07.png
Screenshot_2017-09-10_22-43-07.png (25.18 KiB) Viewed 3296 times
This is done by splitting the sample into three message "parts", under different topics ("Gravity" node):

Code: Select all

var multiplier = 0.004;  //SCALE_MULTIPLIER
var axes = [["x",0],["y",0],["z",1]];
var msgs = [];

bytes = msg.payload;

x = bytes[0] | (bytes[1] << 8);
if(x & (1 << 16 - 1)) {
    x = x - (1<<16);
}

y = bytes[2] | (bytes[3] << 8);
if(y & (1 << 16 - 1)) {
    y = y - (1<<16);
}

z = bytes[4] | (bytes[5] << 8);
if(z & (1 << 16 - 1)) {
    z = z - (1<<16);
}

axes[0][1] = x * multiplier;
axes[1][1] = y * multiplier;
axes[2][1] = z * multiplier;

// z axis gravity fudge
axes[2][1] = axes[2][1] - 1;

for(var i=0; i<axes.length; i++) {
    msgs.push({
        "topic": "mother/orientation/" + axes[i][0],
        "parts": {
            "count":axes.length,
            "index": i
        },
        "payload": parseFloat(axes[i][1].toFixed(4))
    });
}

return [msgs];

Note the gravity "fudge" - I didn't like having Z always around 1 G with the other axes around 0 G as it gave the chart a very low vertical resolution. By placing the Z axis rest position at zero the chart is able to display much smaller movements. This function feeds both an RBE node, which only allows messages through for each axis should the value have changed by more than 0.03, while a second branch is fed to a function node which implements the following time limiting code ("Rate" node):

Code: Select all

// only allow one message per x ms through, drop the rest
var interval = 60000; 
var idx = msg.parts.index;
context.last = context.last || [0,0,0];

var now = Date.now();

if (now - context.last[idx] > interval) {
  context.last[idx] = now;
  return msg;
} else {
  return null;
}

Seems to work ok, but I would love to be able to sample the ADXL345 at a much higher rate (say 100 Hz) and do some averaging (+/-) on each axis to get it down to 1 Hz. As it is now, most samples are never seen, so very short events may be missed. If anyone has any suggestion how I might achieve this I would love to hear about it. Sample chart output, with me wiggling the enclosure around a bit, showing the dual sample rate scheme in action:
Screenshot_2017-09-10_22-57-21.png
Screenshot_2017-09-10_22-57-21.png (13.37 KiB) Viewed 3296 times

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Mon Sep 11, 2017 1:53 pm

I suddenly realised all this accelerometer stuff could be greatly simplified by utilising the interrupt functionality of the ADXL345 to control the sample rate. Triggering the interrupt now sets a global variable indicating that an accelerometer "event" has occurred, and by delaying the reset by five seconds, I can check for the existence of this event when determining the sample rate. This way the RBE node is no longer needed, and the whole messy logic with splitting messages and comparing each axis separately can be removed, with the sample rate control logic merged into the "Gravity" function. As a bonus, I've made the interrupt trigger the cameras, via a "link" node, so a sequence of photos will be taken to (hopefully) capture what it was that caused the "event". The flows now look like this:
Screenshot_2017-09-11_14-43-31.png
Screenshot_2017-09-11_14-43-31.png (30.75 KiB) Viewed 3248 times

And here is the (final-ish) code for the "Gravity" node:

Code: Select all

var time = Date.now();
var interval = 60000; 
context.last = context.last || time;

if(global.get("accelEvt") || time - context.last > interval) {
    context.last = time;
    var msgs = [];
	var bytes = msg.payload;
	var multiplier = 0.004; 
	var axes = [
		["x",bytes[0] | (bytes[1] << 8)],
		["y",bytes[2] | (bytes[3] << 8)],
		["z",bytes[4] | (bytes[5] << 8)]
	];

	for(var i=0; i<axes.length; i++) {
		if(axes[i][1] & (1 << 16 - 1)) {
			axes[i][1] = axes[i][1] - (1<<16);
		}
		axes[i][1] = axes[i][1] * multiplier;
		
		// z axis gravity fudge
		if(i == 2) { axes[i][1] = axes[i][1] - 1; }
		
		msgs.push({
			"topic": "mother/orientation/" + axes[i][0],
			"payload": parseFloat(axes[i][1].toFixed(4))
		});
	}
	return [msgs];
} else {
	return null;
}

Much better!

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Mon Sep 11, 2017 11:33 pm

I have been hunting high and low for good quality pin headers with square pins - not only for this project but basically any time I've needed to stack multiple hats. The problem with most "dual height", or "stacking" headers I've been able to find are that the pins are not solid square stock, but flimsy rectangular bits of metal, which are only good for soldering, if even that. A square cross-section of the pins is essential in order to make good contact with the next female header socket, yet they seem to be rarer than hen's teeth. Here's a photo which illustrates the difference:
good_pin_bad_pin.jpg
good_pin_bad_pin.jpg (149.77 KiB) Viewed 3222 times

I finally found a company here in the UK who have a vast catalogue of pin headers and board connectors, and I ordered a whole bunch of stuff from them, including 20x2 and 13x2 female stacking headers with square pins. The company is called Toby Electronics and I can highly recommend them for all your header needs - they have an amazing variety of hard to find stuff! I needed one 13x2 pin female stacking header for the IVPort, and a double row of double height male pins to carry the remaining 7x2 pins through to the Monarco Hat at the top of the stack. These all arrived today, so I proceeded to disassemble the system, and mock-up the position of the new boards. In addition to the IVPort, I also needed to find room for no less than four Petit Studio FPC-15 to HDMI adapter boards - this because I want to be able to upgrade the IVPort to a four port version in the future, should my tests prove successful. I don't want to have to disassemble the whole thing again to do this, so I decided to add the mounting points for adapter boards 3 & 4 while I was mounting the other two. The adapter boards will be mounted two high, with the top board inverted to match the orientation of the IVPort's connectors:
layer_cake_mockup.jpg
layer_cake_mockup.jpg (187.92 KiB) Viewed 3222 times


In order to make room for the adapter boards, I needed to move the Pi 1.5cm to the left, which will make access to the USB ports a bit of a squeeze, but I couldn't see any other option. The adapter boards must sit between the Pi and the fan, and must be mounted far enough from the top of the cabinet to allow room for the HDMI connectors, which are fairly chunky. They also need to be positioned so that it is possible to replace the MicroSD card without having to take everything apart. I marked and drilled the first four holes, but soon realised I'd made a mistake - looking at photos of the 4-port IVPort it was clear I needed to move adapter boards 3 & 4 further "up", or the FPC cable wouldn't be able to reach the connectors without some serious twisting. Consequently the base board now has no less than eight superfluous holes in the wrong locations :( Ah well.

Another, more serious, problem became apparent when I was about to solder the stacking header to the IVPort: the PCB sticks out too far either side of the socket, making it impossible to fit the double height 7x2 male header next to it. After inspecting the board carefully, I felt fairly certain that the guys at IVMech hadn't routed any traces that close to the edge, so I gave it a quick visit to the bench grinder and removed about 1.5mm of the PCB next to the header. It's possible of course that the board has internal layers which could not be seen, and that I've destroyed it by taking such a drastic action, but I just couldn't see any other option.
layer_cake_ivport_cut.jpg
layer_cake_ivport_cut.jpg (157 KiB) Viewed 3222 times
Last edited by Lomax on Tue Sep 12, 2017 12:19 am, edited 5 times in total.

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Mon Sep 11, 2017 11:44 pm

Yet another frustrating problem was having to desolder and remove the lower 7x2 pins so I could fit the double height pins which will carry those signals up to the Monarco Hat. It was a major pain, and I probably spent a good hour and a half removing the header, cleaning up the holes and fitting the double height pins - hopefully I've been able to do so without damaging anything; things did get fairly hot and sweary.
layer_cake_header.jpg
layer_cake_header.jpg (171.3 KiB) Viewed 3216 times

With the header in place I was finally able to plug in the IVPort and start to get some feeling for whether the stacking would work or not. I was relieved to see that I had got the measurements more or less correct and everything lines up quite nicely.
layer_cake_fitting.jpg
layer_cake_fitting.jpg (187.34 KiB) Viewed 3216 times
Finally, I was able to mount the first two adapter boards and screw everything down. Loctite was used when attaching the posts to the base board, which will hopefully allow future disassembly without loosening them. As nervous as I am to find out if the header transplant and the gouging out of the IVPort board has caused any damage, and as keenly as I look forward to trying the IVPort's camera switching capability, I am pretty exhausted by the whole ordeal, so reassembly and testing will have to wait until tomorrow evening. Here's the end result of today's tinkering:
layer_cake_baked.jpg
layer_cake_baked.jpg (175.35 KiB) Viewed 3216 times

stevend
Posts: 211
Joined: Fri Oct 11, 2013 12:28 pm

Re: Houseboat home automation / surveillance system

Tue Sep 12, 2017 12:42 pm

Lomax wrote:
Mon Sep 11, 2017 11:44 pm
Yet another frustrating problem was having to desolder and remove the lower 7x2 pins so I could fit the double height pins which will carry those signals up to the Monarco Hat.
I generally use 'wire wrap' double row sockets to just extend the pins in this type of situation. However you do then potentially hit your connector quality problem. And the less joints in a marine environment, the better (especially if you're in a 'salty' environment)

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Tue Sep 12, 2017 10:36 pm

stevend wrote:
Tue Sep 12, 2017 12:42 pm
I generally use 'wire wrap' double row sockets to just extend the pins in this type of situation. However you do then potentially hit your connector quality problem. And the less joints in a marine environment, the better (especially if you're in a 'salty' environment)

Yes, I did consider all sorts of options, including an extra tall loose 7x2 stacking header and breaking out the IVPort board with wires, but kept coming back to my original plan as the sturdiest and most compact solution. To be fair, conditions inside the cabinet should never be "marine" - it will be installed in my "home" after all and I wouldn't like it there much myself if it's too wet, cold and humid :) That said, it always feels good to do things "properly", and as you point out there's always salt in the air.

Finally had some time to re-assemble everything, and indications are: whohoo! At first I had some really odd behaviour which took me a long time to pin down; after plugging everything in - carefully double-checking every connection before re-applying power - I was getting frequent and random triggering on the accelerometer interrupt pin. I tried everything I could think of; checking every connection was correct and good, enabling the internal pull-down on the pin (which tbf it probably should be anyway), moving any current carrying wires well away from the accelerometer wires, before resigning to removing the IVPort and testing the sytem without it. This made the problem disappear, but I could not understand what it was about the IVPort that might cause this.

I checked and double checked the documentation; it makes no mention of my chosen interrupt pin (#13 / GPIO 27). Then I tried re-installing the IVPort with no cameras connected - no problem. I eventually tracked it down to the HDMI cable used for one of the cameras; as pointed out on the Petit Studio blog not all HDMI cables are the same, and the longer one of the two I used for testing turned out to be crippled. Just how this results in this random pin triggering is beyond me, but I've A/B tested both my cameras on the known "good" cable (an "Amazon Basics" no less) and it all works just fine, but as soon as "bad" cable gets involved it all goes wrong. Indeed connecting one of the cameras straight to the IVPort's FPC connector, in lieu of having a second "good" HDMI cable (and yes, I did try bridging the GND as suggested in the blog), I can switch between the two cameras from within Node-RED and perform a capture from each one!

This is exactly what I hoped would be possible, so I'm quite excited to finally see it in action :) I'm still working on figuring out the exact sequence required to switch cameras (the documentation is a bit rubbish - almost non-existent - but they do provide some Python examples which helped); it seems it requires both a pin state change on the chosen control pin(s) (#15 / GPIO 22, in my case) and writing to an i2c register (address 0x70, register 0x00, value 0x01/0x02 respectively). If I pull the pin high and and send 0x02 over i2c I get camera #2, with the pin low and 0x01 sent, I get camera #1. Beautiful. Here's a picture showing everything installed, from before I identified and removed the "bad" HDMI cable and bodged the connection to camera #1 with a long-ish FPC and some blu-tac... Note the additional, unconnected, Petit Studio FPC <> HDMI adapter boards; these will allow for a "plug-n-play" upgrade to a 4-port IVPort later. And yes; lots of erroneous drilling - I'm thinking of naming it "project Swiss cheeese" :D
all_fours.jpg
all_fours.jpg (205.27 KiB) Viewed 3120 times

To give you some idea of how busy the little Pi is now, here's the output from i2cdetect -y 1

Code: Select all

     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- 18 -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- 36 -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: -- -- -- 53 -- -- -- 57 -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- UU 
70: 70 -- -- -- -- -- -- --

Six devices just on the i2c bus (though I think the RTC shows up twice, as 6F and 57), plus SPI (for the Monarco Hat's I/O), a 1-Wire weather station, two RS-485 RTUs, PWM fan control, accelerometer interrupt signal - and that's just what's connected now; soon there will be a couple of I-Button readers (to switch between home/away status), a combined 1-Wire temperature/humidity/light/barometric pressure sensor (indoors), a couple of PIR sensors, bilge water level sensors, smoke/fire detectors... I think I'll run out of I/O long before I run out of ideas for what to monitor! :)

"Don't search for all the answers at once. A path is formed by laying one stone at a time"

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Thu Sep 14, 2017 12:39 pm

Still messing around with the IVPort and the dual camera set-up; it is working, but I'm having a few odd issues - maybe someone here can help? For one thing, it seems I need to run raspi-config and enable the camera interface after every cold boot, otherwise:

Code: Select all

# vcgencmd get_camera
supported=1 detected=0

Strangely, enabling the camera interface persists between "warm" reboots:

Code: Select all

# vcgencmd get_camera
supported=1 detected=1

Any ideas?

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Thu Sep 14, 2017 5:27 pm

Another thing I'm wondering about is what, if anything, can be done about collisions on the i2c bus; my comment about it being "busy" turned out to be prophetic as I'm occasionally getting communication errors due to units talking on top of each other... It's not a huge deal since they're fairly infrequent (once every 50-100 reads maybe), and I'll likely get the data on the next read, but I'd like to know why this is happening and what the strategy should be for avoiding it. I actually didn't know the i2c bus didn't have collision detection - you learn something new every day :)

Heater
Posts: 13119
Joined: Tue Jul 17, 2012 3:02 pm

Re: Houseboat home automation / surveillance system

Thu Sep 14, 2017 6:06 pm

As far as I understand it a slave I2C device should only respond if it sees it's address.

So how can collisions happen?

1) Two or more devices have the same address.

2) There are communications errors (noise) on the bus that cause more than one device to see the same address even if it was different when sent out.

Assuming you don't have two bus masters that is.

Lomax
Posts: 188
Joined: Wed May 20, 2015 9:43 pm

Re: Houseboat home automation / surveillance system

Tue Sep 19, 2017 9:14 pm

Heater wrote:
Thu Sep 14, 2017 6:06 pm
As far as I understand it a slave I2C device should only respond if it sees it's address.

That's my understanding as well, but I sometimes get a read error on the 1-Wire bus (which is connected via i2c) when I capture a still image with one of the cameras attached to the IVPort (which is controlled over i2c). Like I said; not a big deal, but it does make me curious.

A bigger problem is that I'm still seeing random triggering of the accelerator interrupt pin (it wasn't just a dodgy cable). Typically it goes like this: I set the accelerometer interrupt to trigger the capturing of an image (in Node-RED). Everything is calm and stable. I intenionally bump the enclosure to get the accelerometer to trigger its "activity" interrupt. The corresponding pin goes high and Node-RED triggers the image capture, and, after a five second delay, an interrupt clear (over i2c). But despite the enclosure being still, as soon as the interrupt has been cleared, it fires again, triggering another capture etc. This cycle repeats indefinetely. However, if I disable the image capture trigger the loop is broken. It seems something happens when either talking to the IVPort pre-capture, or when doing the actual capture (CSI) that makes the whole thing go crazy.
Screenshot_2017-09-19_21-41-37.png
Screenshot_2017-09-19_21-41-37.png (47.2 KiB) Viewed 2797 times

This is not a "show stopper" in any way, but I would like to find out what's causing it. Sadly my scope, along with most of the rest of the lab, is already in storage. I'm unable to investigate further until I've got things set up on board - which is not likely to happen until next year!

In other events, I've done battle with the OpenWeatherMap forecast API and think I have emerged victorious, albeit somewhat bruised. I wanted to publish each forecast metric as its own MQTT topic, but that would clearly be insane considering it returns 40 segments, each with 12 metrics of interest. In the end, I've resigned to pushing the whole thing as a single JSON object containing an array of "days":

Code: Select all

var topic = "owm/forecast";
var pl = {};
var prev = "";

function cap(string) {
    return string.charAt(0).toUpperCase() + string.slice(1);
}

for(var i=0; i<msg.payload.length; i++) {
    var date = new Date(msg.payload[i].dt_txt);
    var day = date.toLocaleString("en-GB", {weekday:"short"});
    var hour = ("0" + date.getHours()).substr(-2);
    var data = {
        "hour":hour,
        "pressure":msg.payload[i].main.pressure,
        "humidity":msg.payload[i].main.humidity,
        "rain":msg.payload[i].rain["3h"],
        "clouds":msg.payload[i].clouds["all"],
        "temp":Math.round(msg.payload[i].main.temp),
        "windspeed":Math.round(msg.payload[i].wind.speed),
        "winddirection":Math.round(msg.payload[i].wind.deg),
        "mintemp":Math.round(msg.payload[i].main.temp_min),
        "maxtemp":Math.round(msg.payload[i].main.temp_max),
        "description":cap(msg.payload[i].weather[0].description),
        "icon":msg.payload[i].weather[0].icon
    };
    if(day != prev){
        prev = day;
        pl[day] = [];
    }
    pl[day].push(data);
}
return {"topic":topic, "payload":pl};

This is subscribed to by the following template:

Code: Select all

<style>
.tabs {
    position: relative;   
    min-height: 250px;
    clear: both;
    margin: 25px 0;
}

.tab {
float: left;
}
    .tab label {
    background: #333; 
    padding: 10px; 
    position: relative;
    border: 1px solid #222;
    }
    .tab table {
    position: absolute;
    top: 28px;
    left: 0;
    width: 100%;
    background: #222;
    }
        .tab table td {
        text-align: right;
        font-size: .85em;
        }
            .tab table td span.hour {
            color: #777;
            }
        .tab table td:first-child {
        padding-left: 5px;
        text-align: left;
        }
        .tab table td:last-child {
        width: 50px;
        height: 26px;
        }

.tabs input[type=radio] {
display: none;   
}
    .tabs input[type=radio]:checked ~ label {
    background: #222;
    z-index: 2;
    }
        .tabs table {
        display: none;
        }
            .tabs input[type=radio]:checked ~ label ~ table {
            display: table;
            }
</style>

<div class="tabs">
    <div class="tab" ng-repeat="(day, data) in msg.payload">
        <input type="radio" id="{{day}}" name="day" ng-value="day" ng-checked="{{$index === 0}}" />
        <label for="{{day}}">{{day}}</label>
        <table>
		<tbody>
            <tr ng-repeat="segment in data">
                <td><span class="hour">{{segment.hour}}</span> {{segment.description}}</td>
                <td>{{segment.temp}}&deg;C</td>
                <td>{{segment.windspeed}}m/s</td>
                <td style="background: url(/icons/weather/{{segment.icon}}.png) no-repeat center;">&nbsp;</td>
            </tr>
        </tbody>
        </table>
    </div>
</div>

Resulting in this rather spiffy 5-day weather prognosis on the "environment" dashboard:
Screenshot_2017-09-19_22-02-54.png
Screenshot_2017-09-19_22-02-54.png (141.15 KiB) Viewed 2804 times

Return to “General discussion”