xXAzazelXx
Posts: 135
Joined: Tue Sep 18, 2012 8:32 am
Location: Australia
Contact: Website

Data Logging, Plotting Help

Tue Aug 13, 2019 11:40 pm

Hi Guys,

Apologies, I'm very new to all of this. Can someone please point me in the right direction of where to start with collecting and plotting data from sensors?

I have recently purchased Enviro-Plus https://shop.pimoroni.com/products/enviro-plus and ideally would like to plot the data from the sensors over time.

I was thinking of something like smokeping, prtg, mrtg equivalent but for this?

What do you guys use to collect and plot the date from your sensors outputs?

User avatar
Gavinmc42
Posts: 4020
Joined: Wed Aug 28, 2013 3:31 am

Re: Data Logging, Plotting Help

Wed Aug 14, 2019 1:30 am

I usually just collect the data into a csv file and once a day write it to SD.
Lots of way to then plot csv data.
Dygraphs.js is my webserver method.

Basic shell script is used, cron for the daily writes, watch for running sensor reading script.
Watch for grabbing the data, it calls a shell script that reads and converts it to csv and writes it to ram file.

Cron and watch have proven bullet proof running on PiCore Linux for years.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

xXAzazelXx
Posts: 135
Joined: Tue Sep 18, 2012 8:32 am
Location: Australia
Contact: Website

Re: Data Logging, Plotting Help

Wed Aug 14, 2019 10:24 am

Gavinmc42 wrote:
Wed Aug 14, 2019 1:30 am
I usually just collect the data into a csv file and once a day write it to SD.
Lots of way to then plot csv data.
Dygraphs.js is my webserver method.

Basic shell script is used, cron for the daily writes, watch for running sensor reading script.
Watch for grabbing the data, it calls a shell script that reads and converts it to csv and writes it to ram file.

Cron and watch have proven bullet proof running on PiCore Linux for years.
Thank you for for the advice :D

How are you collecting the data into csv? python? I have tried to edit the sample script into for loop but im not having much luck.

Code: Select all

#!/usr/bin/env python

import time
import csv
from bme280 import BME280
from datetime import datetime

try:
    from smbus2 import SMBus
except ImportError:
    from smbus import SMBus

dt = datetime.now()

full_date = dt.strftime("%Y-%m-%d")
full_time = dt.strftime("%H:%M:%S")
full_day = dt.strftime("%A")
year = dt.year
month = dt.month
date = dt.day
hour = dt.hour

bus = SMBus(1)
bme280 = BME280(i2c_dev=bus)



temperature = bme280.get_temperature()
pressure = bme280.get_pressure()
humidity = bme280.get_humidity()
csvWrite = full_date, full_time, full_day, year, month, date, hour, temperature, pressure, humidity
csvFile = open("/home/pi/enviroplot.csv", "a")

while True:
    with csvFile:
        writer = csv.writer(csvFile)
        writer.writerow(csvWrite)
        time.sleep(1)
        

User avatar
Gavinmc42
Posts: 4020
Joined: Wed Aug 28, 2013 3:31 am

Re: Data Logging, Plotting Help

Thu Aug 15, 2019 2:10 am

Awk and Sed are your Shell script friends.
In Linux everything is a file, Awk and Sed can write/edit files.
Horrible old stuff that comes with every Linux OS and just works.

If you use a webserver then no need for all that x11 desktop bloat stuff, just a headless Linux OS.
Easy fit all this on a 128MB card.
Busybox has a webserver or you can pick from a bunch of options.
I use shell script and Awk/Sed to rewrite the CGI/HTML files on the fly.

Looping Python code, no, no, no - never really got that to work.
Memory leaks? Don't know enough Linux/Python to fix it.
If I have to use Python I call it from script and make sure it exits.
Same with shell, no loops, let cron, watch do the timing.

Software loops use 100% cpu cycles, the "watch" method max out at 2% CPU usage on a PiB+.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

User avatar
OO-Dragon
Posts: 21
Joined: Sat Jun 02, 2018 3:09 pm
Contact: Website

Re: Data Logging, Plotting Help

Fri Aug 23, 2019 8:45 pm

This may be a little biased but I started a project last year called "Kootnet Sensors" to gather sensor data over long periods of time with Raspberry Pi's and it supports a bunch of sensors, including the EnviroPHAT and Enviro+.

It's decent on resources once it loads up, as I have it running on a Pi Zero W using about 3% CPU (Greatly dependant on how often you set it to record sensor data). It also saves all data into an SQLite3 Database for quick and portable data access. Graphing can be done right from the web interface or the "Control Center" software (Python3 GUI to interact with sensors).

Installation is easy, as you just run a single script and let it do all the work, you just enter in the username and password you want to use for accessing the Web Interface. It also creates shortcuts under the Accessories menu to launch the management functions. The recording and HTTPS management programs start right after install but require configuration of sensors (Tell it what sensors you have installed) before it will start to work with the sensors. It even enables everything the sensors needs, AKA you can load up a fresh install of Raspbian, run the install script, setup the installed sensors under the web management interface and everything just works.

Check the download page here http://kootenay-networks.com/?page_id=236 if you are interested.

The short version of installing it on the Raspberry Pi is as follows.
1. Install Raspbian on the Flask Card.
2. Physically install your sensor(s)
3. Connect to the internet and run the following in a terminal.

Code: Select all

wget http://kootenay-networks.com/utils/koot_net_sensors/Installers/raspbian/install_update_kootnet-sensors_http.sh && sudo bash install_update_kootnet-sensors_http.sh
If you do end up giving it a try, let me know! I haven't actually publicly posted this project yet and could use some feedback. It is released under the GPL, so feel free to look at the code on GitHub. You can find a link on the download page mentioned above.
OO-Dragon

Andyroo

Re: Data Logging, Plotting Help

Fri Aug 23, 2019 8:54 pm

One other option is to use io.adafruit to create a basic dashboard - it only stores a limited period of data on the free level but other solutions do exist.

A search of google for IoT dashboard will turn up lots.

Another option is to look at node-red to handle the data and dashboard needs.

amcdonley
Posts: 182
Joined: Mon Jan 26, 2015 5:56 pm
Location: Florida, USA

Local: Data Logging, Plotting

Sat Aug 24, 2019 4:28 pm

For simple logging sensor data to a file and then plotting it, all local:


1) Log Battery Voltage and Plot on linear scale by hour of day, and by time since first measurement
2) Log PiCamera Brightness Value and Plot on log scale by hour of day


It is also possible to tie sensor readings into the RPi-Monitor code which will serve plots to a browser.

You also can go the IoT route by following the Rosie The Red Robot blog - great stuff but involved.

xXAzazelXx
Posts: 135
Joined: Tue Sep 18, 2012 8:32 am
Location: Australia
Contact: Website

Re: Data Logging, Plotting Help

Tue Aug 27, 2019 9:37 am

Thank you all for your advice !! :) :)
I will have a look in to it all.
In the meantime i managed to write a python script to get all of the readings and put them in to a .csv file and use crontab to run every 5 minutes.
Not the best solution but I think its a good start :P

User avatar
bensimmo
Posts: 4187
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: Data Logging, Plotting Help

Tue Aug 27, 2019 9:58 am

If you want to learn some Python3 techniques have a look at the SenseHAT datalogging tutorials on the main projects website.
You then slip in the Pimoroni code as needed.
After that there are some graphing tutorials there too.

And then you can move on to IoT based setups, say sending it to some external source. Sending it via MQTT to a broker.

Have fun and the Pimoroni people will help out as well.
Also feed back any problems to them, the Enviro+ is quite new and the code may need feedback on how people use it.

kflmiami420
Posts: 62
Joined: Thu Mar 21, 2019 6:54 pm

Re: Data Logging, Plotting Help

Mon Sep 09, 2019 2:33 am

xXAzazelXx did you get your enviro+ board logging ? What did you finally decide to use to log your sensors ?

Return to “Automation, sensing and robotics”