ImagineZA
Posts: 19
Joined: Thu Oct 11, 2018 10:15 am

Python + JSON, how to create a good log file for diagnostics

Thu May 02, 2019 4:56 am

Hey Guys,

I have a piece of Python Code pulling data from an API and then saving it to a database in MySQL.

The Code works perfectly well enough, however I noticed that my Pi Disconnected from the WiFi and the code has failed since then. I have the python code running as a service, which has not stopped running, but the code is no longer pulling the API data since the disconnect

I'd like to add in some extra code to create a better log file than the one I have in the code to help diagnose AND to edit the code to restart itself if a disconnect happens again.

Below is my python code, up for scrutiny:

Code: Select all

#!/usr/bin/python3

#import modules
import cymysql
from time import sleep
from urllib.request import urlopen
import json
import datetime

#set MySQl Variables
host = "localhost"
user = "xxx"
password = "xxx"
schema = "xxx"

#connect to MySQL DB
db = cymysql.connect(host, user, password, schema)
curs = db.cursor()

#set api key for DarkSky API
apikey="xxx"
# Latitude & longitude
lati="-26.20227"
longi="28.04363"

# Add units=si to get it in sensible ISO units.
url="https://api.forecast.io/forecast/"+apikey+"/"+lati+","+longi+"?units=si"

#begin infinite loop
while True:

        #convert API reading to json and readable array 'weather'
        meteo=urlopen(url).read()
        meteo = meteo.decode('utf-8')
        weather = json.loads(meteo)

        #set variables for current weather
        cTemp = (weather['currently']['temperature'])
        cCond = (weather['currently']['summary'])
        cRain1 =  (weather['currently']['precipProbability'])
        cRain2 = cRain1*100
        cIcon = (weather['currently']['icon'])
        oaSum = (weather['daily']['summary'])

        #print variables - for testing purposes
        #print (cTemp)
        #print (cCond)
        #print (cRain2)
        #print (cIcon)
        #print (oaSum)

        #extract daily data from 'weather' array
        daily = (weather['daily']['data'])

        #create new arrays for daily variables
        listHigh = []
        listLow = []
        listCond = []
        listRain = []
        listIcon = []

        #set daily variables
        for i in daily:
                listHigh.append(i['temperatureHigh'])

        for i in range(0,len(listHigh)):
                high1 = listHigh[0]
                high2 = listHigh[1]
                high3 = listHigh[2]
                high4 = listHigh[3]
                high5 = listHigh[4]
                high6 = listHigh[5]
                high7 = listHigh[6]
                high8 = listHigh[7]

        for o in daily:
                listLow.append(o['temperatureLow'])

        for o in range(0,len(listLow)):
                low1 = listLow[0]
                low2 = listLow[1]
                low3 = listLow[2]
                low4 = listLow[3]
                low5 = listLow[4]
                low6 = listLow[5]
                low7 = listLow[6]
                low8 = listLow[7]

        for p in daily:
                listCond.append(p['summary'])

        for p in range(0,len(listCond)):
                cond1 = listCond[0]
                cond2 = listCond[1]
                cond3 = listCond[2]
                cond4 = listCond[3]
                cond5 = listCond[4]
                cond6 = listCond[5]
                cond7 = listCond[6]
                cond8 = listCond[7]

        for m in daily:
                listRain.append(m['precipProbability'])

        for m in range(0,len(listRain)):
                rain1 = listRain[0]
                rain2 = listRain[1]
                rain3 = listRain[2]
                rain4 = listRain[3]
                rain5 = listRain[4]
                rain6 = listRain[5]
                rain7 = listRain[6]
                rain8 = listRain[7]

        #convert rain chance to readable percentage
        prain1 = rain1*100
        prain2 = rain2*100
        prain3 = rain3*100
        prain4 = rain4*100
        prain5 = rain5*100
        prain6 = rain6*100
        prain7 = rain7*100
        prain8 = rain8*100

        for l in daily:
                listIcon.append(l['icon'])

        for l in range (0,len(listIcon)):
                icon1 = listIcon[0]
                icon2 = listIcon[1]
                icon3 = listIcon[2]
                icon4 = listIcon[3]
                icon5 = listIcon[4]
                icon6 = listIcon[5]
                icon7 = listIcon[6]
                icon8 = listIcon[7]

        #print daily variables - for testing purposes
        #print (high1)
        #print (low1)
        #print (cond1)
        #print (prain1)
        #print (icon1)
        #print (high2)
        #print (low2)
        #print (cond2)
        #print (prain2)
        #print (icon2)
        
        #update data in DataBase
        try:
                sql_update_query = """UPDATE weather SET current_temp = %s, cur$
                varis = (cTemp, cCond, cRain2, cIcon, high1, low1, cond1, prain$
                curs.execute(sql_update_query, varis)
                db.commit()
        except db.Error as error:
                print("Error: {}".format(error))
                db.rollback()
                
        #write date to log file
        with open ("/home/pi/CoRo/Projects/WeatherMan/weatherlog.txt", mode="w") as file:
                file.write('Last Data was pulled at: %s' %(datetime.datetime.now()))
        #set loop to sleep for 10 minutes and go again
        sleep(600)


I know the Database code has been snipped, it's a lot of variables being updated in the database, but I know that works 100%, if anybody wants me to add the full code, just ask.

If anyone has any tips/ideas to add in to assist with the disconnect issue and with making the log file better, please would you add them in?

Much appreciated

Aydan
Posts: 694
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: Python + JSON, how to create a good log file for diagnostics

Thu May 02, 2019 2:26 pm

Do you run this as a systemd service? If so, just add print() wherever (e.g. for each loop run).
You can read the logs via journalctl.
Your write to the log file should open the file with mode "a", not "w". Mode "w" will truncate the file each time.

Regards
Aydan

ghp
Posts: 1414
Joined: Wed Jun 12, 2013 12:41 pm
Location: Stuttgart Germany
Contact: Website

Re: Python + JSON, how to create a good log file for diagnostics

Fri May 03, 2019 4:14 am

there is a logging package in python
https://docs.python.org/3.7/howto/logging.html
which allows for file rotation, output for syslog, log levels, formatting and much more.

ImagineZA
Posts: 19
Joined: Thu Oct 11, 2018 10:15 am

Re: Python + JSON, how to create a good log file for diagnostics

Fri May 03, 2019 7:51 am

Aydan wrote: Do you run this as a systemd service? If so, just add print() wherever (e.g. for each loop run).
You can read the logs via journalctl.
Your write to the log file should open the file with mode "a", not "w". Mode "w" will truncate the file each time.

Regards
Aydan
So anywhere a print() appears in the code it would appear in journalctl? I don't need to adjust the systemd profile or anything?
ghp wrote: there is a logging package in python
https://docs.python.org/3.7/howto/logging.html
which allows for file rotation, output for syslog, log levels, formatting and much more.
For the File Rotation handler, if I set it to one backup, will it overwrite this backup when maximum size is reached?

Cheers,

Aydan
Posts: 694
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: Python + JSON, how to create a good log file for diagnostics

Fri May 03, 2019 10:01 am

ImagineZA wrote:
Fri May 03, 2019 7:51 am

So anywhere a print() appears in the code it would appear in journalctl? I don't need to adjust the systemd profile or anything?
Yes, any output to stdout goes into the systemd journal, which can be read with journalctl.
No need to set up anything in the systemd unit.
It will also end up in syslog by default.
If you want to log into a specific file see https://www.freedesktop.org/software/sy ... ardOutput=

ImagineZA
Posts: 19
Joined: Thu Oct 11, 2018 10:15 am

Re: Python + JSON, how to create a good log file for diagnostics

Fri May 03, 2019 10:10 am

Thanks Ayden,

Will use a combination of the built in Python Logging as well as the print for quick checks using journactl.

Cheers,

Return to “Python”