komarek
Posts: 146
Joined: Fri Mar 31, 2017 1:19 am

python script at startup

Fri Jun 30, 2017 10:00 am

hi folks
in my Rpi 3 i have a few python script running in background that are loaded a startup... basically these script handle my hous sensor system with IOT api's and things like that.

i load the scripts with /etc/rc.local
which is ending something like this:

Code: Select all

python3 /home/pi/Documents/PY/citofono.py &
python3 /home/pi/Documents/PY/fanspeed.py &
python3 /home/pi/Documents/PY/allarmecasa.py &
python3 /home/pi/Documents/PY/analitico.py &

exit 0
well they all wrok good except the last one (analitico.py) which is actually running as process but is not working at all.
so my solution (at every reboot) is not very comfortable... i have to kill the process and run it again from the terminal:
python3 /home/pi/Documents/PY/analitico.py &

and then it's working properly.

the problem is that if i close the terminal window the script stop working in background.
what do you think is the problem?
where can i find some useful logs?

User avatar
B.Goode
Posts: 8881
Joined: Mon Sep 01, 2014 4:03 pm
Location: UK

Re: python script at startup

Fri Jun 30, 2017 10:09 am

what do you think is the problem?
I think, although it is only a guess in the absence of any real information on which to base a reply, that there is probably something in

Code: Select all

 /home/pi/Documents/PY/analitico.py 
that makes the script dependent on interaction via a terminal.

Posting the script here might allow someone with appropriate knowledge to comment.

Running dmesg from a shell (command-line) prompt will show you recent system console output.

komarek
Posts: 146
Joined: Fri Mar 31, 2017 1:19 am

Re: python script at startup

Fri Jun 30, 2017 10:24 am

this is the script
but i dont think the problem is here. otherwise it won't work when i run it manually

Code: Select all

#!/usr/bin/python

from __future__ import print_function

import re
import argparse
import sys
import time

import paho.mqtt.publish as publish
import paho.mqtt.client as mqtt

from googleapiclient.errors import HttpError
from googleapiclient import sample_tools
from oauth2client.client import AccessTokenRefreshError

def get_first_profile_id(service):

	accounts = service.management().accounts().list().execute()
	if accounts.get('items'):
		firstAccountId = accounts.get('items')[0].get('id')
		webproperties = service.management().webproperties().list(accountId=firstAccountId).execute()

	if webproperties.get('items'):
		firstWebpropertyId = webproperties.get('items')[0].get('id')
		profiles = service.management().profiles().list(accountId=firstAccountId, webPropertyId=firstWebpropertyId).execute()
	if profiles.get('items'):
		return profiles.get('items')[0].get('id')
	return None


def hg_attivi(service, profile_id):
	return service.data().realtime().get(
	ids='ga:xxxxxx',
	metrics='rt:activeUsers').execute()

def tobado_attivi(service, profile_id):
	return service.data().realtime().get(
	ids='ga:xxxxxx',
	metrics='rt:activeUsers').execute()

def hg_oggi(service, profile_id):
        return service.data().ga().get(
        ids='ga:xxxxxx',
	start_date='today',
	end_date='today',
	metrics='ga:visits').execute()

def tobado_oggi(service, profile_id):
        return service.data().ga().get(
        ids='ga:xxxxxx',
        start_date='today',
        end_date='today',
        metrics='ga:visits').execute()

def print_results(results):
	# Print data table.
	if results.get('rows', []):
		for row in results.get('rows'):
			output = []
			for cell in row:
        			output.append('%30s' % cell)
			print(''.join(output))
	else:
		print('0')


def invia_analitici():
	try:
		service, flags = sample_tools.init(sys.argv, 'analytics', 'v3', __doc__, __file__, scope='https://www.googleapis.com/auth/analytics.readonly')
	except Exception as ecc:
		print(ecc)
	# Try to make a request to the API. Print the results or handle errors.
	try:
		first_profile_id = get_first_profile_id(service)
		if not first_profile_id:
			print('Could not find a valid profile for this user.')
		else:
			# HORROR GALORE
			results = hg_attivi(service, first_profile_id)			
			risultati = results.get("totalsForAllResults")
			risultati_pronti = str(risultati.get("rt:activeUsers"))
			publish.single("Analytics/HG/attivi", payload=risultati_pronti, qos=1, retain=False, hostname="localhost", port=1883, client_id="PI-Zero")		
			# HORROR GALORE OGGI
			results = hg_oggi(service, first_profile_id)
			risultati = results.get("totalsForAllResults")
			risultati_pronti = str(risultati.get("ga:visits"))
			publish.single("Analytics/HG/oggi", payload=risultati_pronti, qos=1, retain=False, hostname="localhost", port=1883, client_id="PI-Zero")
			# TOBADO
			results = tobado_attivi(service, first_profile_id)
			risultati = results.get("totalsForAllResults")
			risultati_pronti = str(risultati.get("rt:activeUsers"))
			publish.single("Analytics/TOBADO/attivi", payload=risultati_pronti, qos=1, retain=False, hostname="localhost", port=1883, client_id="PI-Zero") 
			# TOBADO OGGI
			results = tobado_oggi(service, first_profile_id)
			risultati = results.get("totalsForAllResults")
			risultati_pronti = str(risultati.get("ga:visits"))
			publish.single("Analytics/TOBADO/oggi", payload=risultati_pronti, qos=1, retain=False, hostname="localhost", port=1883, client_id="PI-Zero")

	except TypeError as error:
		# Handle errors in constructing a query.
		print(('There was an error in constructing your query : %s' % error))
	except HttpError as error:
		# Handle API errors.
		print(('Arg, there was an API error : %s : %s' % (error.resp.status, error._get_reason())))
	except AccessTokenRefreshError:
		# Handle Auth errors.
		print ('The credentials have been revoked or expired, please re-run ' 'the application to re-authorize')

while True:
	def on_connect(client, userdata, flags, rc):
		#print("Connected with result code "+str(rc))
		client.subscribe("Analitico")
	
	def on_message(client, userdata, msg):
		messaggio = str(msg.payload)
		if "richiediANALISI" in messaggio:
			try:
				#print("raccolgo i dati e te li invio")
				invia_analitici()
				#time.sleep(10)
			except Exception as ecc:
				print(ecc)
	try:
		client = mqtt.Client()
		client.on_connect = on_connect
		client.on_message = on_message

		client.connect("127.0.0.1",1883,60)
	
	except Exception as eccloop:
		print(ecc)
	
	else:
		client.loop_forever()

NotRequired
Posts: 195
Joined: Sat Apr 29, 2017 10:36 am
Location: Denmark

Re: python script at startup

Fri Jun 30, 2017 10:31 am

If "analitico.py" depends on "something" (eg. a sensor, network or alike) which is not available when rc.local is loaded or if the script fills up a std[out|err] pipe buffer, this would cause the script to enter a deadlock and this might be hard to debug. The first thing you should do is to modify the command in rc.local in order to redirect the output, like this:

python3 /home/pi/Documents/PY/analitico.py &> /dev/shm/analitico.log &

This will redirect any output from the script to a file named /dev/shm/analitico.log where you can review if anything sinister is going on. Please note that /dev/shm is volatile and stored in memory, therefore the log will vanish on reboot / shutdown.

Another tip: Do not use rc.local to start long running background services with dependencies. Use systemd and specify the dependencies instead - this will ensure that the script is invoked after the dependencies are ready to use :)
Please do not ask questions in private messages, they will not help others.

User avatar
B.Goode
Posts: 8881
Joined: Mon Sep 01, 2014 4:03 pm
Location: UK

Re: python script at startup

Fri Jun 30, 2017 10:42 am

I think your knowledge of Python probably exceeds mine, because I would not define functions within a while loop. But that is probably not directly related to this issue.

Is there anything in the script that might behave differently if run as root/superuser during boot, rather than as a 'normal' user via the shell prompt.

There are some print statements in the script, so try piping the standard output from the script to a log file which you can subsequently examine to see what happened.

Edit : I see the idea of looking at a log file was suggested independently while I was typing. Left to indicate that it is A Good Idea.

komarek
Posts: 146
Joined: Fri Mar 31, 2017 1:19 am

Re: python script at startup

Fri Jun 30, 2017 10:46 am

NotRequired wrote:If "analitico.py" depends on "something" (eg. a sensor, network or alike) which is not available when rc.local is loaded or if the script fills up a std[out|err] pipe buffer, this would cause the script to enter a deadlock and this might be hard to debug. The first thing you should do is to modify the command in rc.local in order to redirect the output, like this:

Another tip: Do not use rc.local to start long running background services with dependencies. Use systemd and specify the dependencies instead - this will ensure that the script is invoked after the dependencies are ready to use :)
ok thanks
but it's pretty strange because the other scripts loaded before that are quite similar and they give me no trouble at all...
about systemd you're absolutely right... my problem was that i'm lazy :)

User avatar
B.Goode
Posts: 8881
Joined: Mon Sep 01, 2014 4:03 pm
Location: UK

Re: python script at startup

Fri Jun 30, 2017 10:52 am

the other scripts loaded before that are quite similar and they give me no trouble at all...
Does the behaviour change if you change the startup order? Is it just that the latest script is 'Too much?'

komarek
Posts: 146
Joined: Fri Mar 31, 2017 1:19 am

Re: python script at startup

Fri Jun 30, 2017 11:15 am

B.Goode wrote:
the other scripts loaded before that are quite similar and they give me no trouble at all...
Does the behaviour change if you change the startup order? Is it just that the latest script is 'Too much?'
well i m not sure... but i think you're right
i tried once to change the order once and maybe the last one it was not working.
i need to test it again.
anyway now i m trying the "systemd" way
sounds better solution

komarek
Posts: 146
Joined: Fri Mar 31, 2017 1:19 am

Re: python script at startup

Fri Jun 30, 2017 11:21 am

NotRequired wrote: Another tip: Do not use rc.local to start long running background services with dependencies. Use systemd and specify the dependencies instead - this will ensure that the script is invoked after the dependencies are ready to use :)
well i tried with systemd now

this is my analitico.service file:

Code: Select all

[Unit]
Description=Servizio Gestione Google Analytics su MQTT
After=multi-user.target

[Service]
Type=idle
ExecStart=/usr/bin/python3 /home/pi/Documents/PY/analitico.py

[Install]
WantedBy=multi-user.target
the status says the service is active and running... but the script doesn't actually working.
if i call the MQTT command the script doesn't answer as it supposed to do.
and...
it's strange that if i set the line "ExecStart=/usr/bin/python3 /home/pi/Documents/PY/analitico.py" with the log file .... like this: ExecStart=/usr/bin/python3 /home/pi/Documents/PY/analitico.py > /dev/shm/analitico.log
it gives me error.

any ideas?

NotRequired
Posts: 195
Joined: Sat Apr 29, 2017 10:36 am
Location: Denmark

Re: python script at startup

Fri Jun 30, 2017 12:54 pm

komarek wrote:it's strange that if i set the line "ExecStart=/usr/bin/python3 /home/pi/Documents/PY/analitico.py" with the log file .... like this: ExecStart=/usr/bin/python3 /home/pi/Documents/PY/analitico.py > /dev/shm/analitico.log
it gives me error.

any ideas?
This is not strange, systemd commands does not run in a shell and the redirect operator ">" is handled by the shell (BASH). If you want to use ">" you should put the command line in a shell script and execute that instead. Also, since your py uses network you might want to change your service to include:

Wants=network.target
After=multi-user.target network.target
Please do not ask questions in private messages, they will not help others.

komarek
Posts: 146
Joined: Fri Mar 31, 2017 1:19 am

Re: python script at startup

Fri Jun 30, 2017 1:08 pm

NotRequired wrote: This is not strange, systemd commands does not run in a shell and the redirect operator ">" is handled by the shell (BASH). If you want to use ">" you should put the command line in a shell script and execute that instead. Also, since your py uses network you might want to change your service to include:

Wants=network.target
After=multi-user.target network.target
ok i did it
but still doesnt work

NotRequired
Posts: 195
Joined: Sat Apr 29, 2017 10:36 am
Location: Denmark

Re: python script at startup

Fri Jun 30, 2017 1:22 pm

Does the output from the py script give you any clue?
Please do not ask questions in private messages, they will not help others.

komarek
Posts: 146
Joined: Fri Mar 31, 2017 1:19 am

Re: python script at startup

Fri Jun 30, 2017 1:27 pm

NotRequired wrote:Does the output from the py script give you any clue?
no clue
the only thing i can say... that if i run the script manually: python3 /home/pi/Documents/analitico.py
it works like charm

NotRequired
Posts: 195
Joined: Sat Apr 29, 2017 10:36 am
Location: Denmark

Re: python script at startup

Fri Jun 30, 2017 2:04 pm

The only thing that would be different is the environment variables (accessed with os.environ[..]) which are different when run as root or outside shell. Have you tried to create a shell script (/home/pi/analitico.sh) with the content:

Code: Select all

#!/bin/bash
python3 /home/pi/Documents/PY/analitico.py &> /dev/shm/analitico.log
Remember to set executable bit: chmod u+x /home/pi/analitico.sh

Then create a systemd service for it (/etc/systemd/system/analitico.service):

Code: Select all

[Unit]
Description=Servizio Gestione Google Analytics su MQTT
After=multi-user.target
Wants=network.target network-online.target

[Service]
Type=idle
User=pi
ExecStart=/home/pi/analitico.sh

[Install]
WantedBy=multi-user.target
And then enable the service with "sudo systemctl enable analitico" and reboot? This should be exactely the same as launching from a command line as user pi..
Please do not ask questions in private messages, they will not help others.

komarek
Posts: 146
Joined: Fri Mar 31, 2017 1:19 am

Re: python script at startup

Fri Jun 30, 2017 2:32 pm

no way

Loaded: loaded (/lib/systemd/system/analitico.service; enabled)
Active: failed (Result: exit-code) since ven 2017-06-30 16:26:43 CEST; 5min ago
Process: 2018 ExecStart=/home/pi/Documents/SH/analitico.sh (code=exited, status=203/EXEC)
Main PID: 2018 (code=exited, status=203/EXEC)

NotRequired
Posts: 195
Joined: Sat Apr 29, 2017 10:36 am
Location: Denmark

Re: python script at startup

Fri Jun 30, 2017 2:40 pm

Did you remember to set executable bit for the *.sh?
Please do not ask questions in private messages, they will not help others.

komarek
Posts: 146
Joined: Fri Mar 31, 2017 1:19 am

Re: python script at startup

Fri Jun 30, 2017 2:48 pm

NotRequired wrote:Did you remember to set executable bit for the *.sh?
yes
and then i tried to chmod 644 also

same error

komarek
Posts: 146
Joined: Fri Mar 31, 2017 1:19 am

Re: python script at startup

Fri Jun 30, 2017 2:56 pm

NotRequired wrote:Did you remember to set executable bit for the *.sh?
now i've set the chmode to 777

and gave me denied permission for: /dev/shm/analitico.log

then i got rid of "/dev/shm/analitico.log"

and when i do sudo service analitico status i have this:

Code: Select all

   Loaded: loaded (/lib/systemd/system/analitico.service; enabled)
   Active: inactive (dead) since ven 2017-06-30 16:53:58 CEST; 3s ago
  Process: 15397 ExecStart=/home/pi/Documents/SH/analitico.sh (code=exited, status=0/SUCCESS)
 Main PID: 15397 (code=exited, status=0/SUCCESS)

what a mess :)

komarek
Posts: 146
Joined: Fri Mar 31, 2017 1:19 am

Re: python script at startup

Fri Jun 30, 2017 3:00 pm

obviously if i run /home/pi/Documents/SH/analitico.sh manually
it works well

komarek
Posts: 146
Joined: Fri Mar 31, 2017 1:19 am

Re: python script at startup

Fri Jun 30, 2017 3:02 pm

looks like something wrong in this file:

Code: Select all

[Unit]
Description=Servizio Gestione Google Analytics su MQTT
After=multi-user.target
Wants=network.target network-online.target

[Service]
Type=idle
User=pi
ExecStart=/home/pi/Documents/SH/analitico.sh

[Install]
WantedBy=multi-user.target


komarek
Posts: 146
Joined: Fri Mar 31, 2017 1:19 am

Re: python script at startup

Fri Jun 30, 2017 5:03 pm

i will never find a solution for this crap

NotRequired
Posts: 195
Joined: Sat Apr 29, 2017 10:36 am
Location: Denmark

Re: python script at startup

Fri Jun 30, 2017 5:20 pm

Please try to change your "analitico.sh" to contain:

Code: Select all

#!/bin/bash
LOG_FILE="/home/pi/analitico.log"
echo "Hello from shell" >> $LOG_FILE
python3 /home/pi/Documents/PY/analitico.py &>> $LOG_FILE
Do not change "&>>" to ">": "&" will redirect both stderr and stdout, ">>" will append.

Then try to modify "analitico.py" to:

Code: Select all

...
import paho.mqtt.client as mqtt
print "Hello from python"
from googleapiclient.errors import HttpError
...
And reboot with the analitico service enabled. When booted try to execute command "cat /home/pi/analitico.log" - what is the output?
Please do not ask questions in private messages, they will not help others.

Martin Frezman
Posts: 1020
Joined: Mon Oct 31, 2016 10:05 am

Re: python script at startup

Fri Jun 30, 2017 5:25 pm

Code: Select all

#!/bin/bash
LOG_FILE="/home/pi/analitico.log"
echo "Hello from shell" >> $LOG_FILE
python3 /home/pi/Documents/PY/analitico.py &>> $LOG_FILE
Note that a simpler way to do this is:

Code: Select all

#!/bin/bash
exec > /home/pi/analitico.log 2>&1
# Rest of script here - no more redirection needed.
If this post appears in the wrong forums category, my apologies.

komarek
Posts: 146
Joined: Fri Mar 31, 2017 1:19 am

Re: python script at startup

Fri Jun 30, 2017 5:30 pm

NotRequired wrote:Please try to change your "analitico.sh" to contain:


And reboot with the analitico service enabled. When booted try to execute command "cat /home/pi/analitico.log" - what is the output?
the log file just says:
hello from shell


and
the service status is:

Code: Select all

   Loaded: loaded (/lib/systemd/system/analitico.service; enabled)
   Active: active (running) since ven 2017-06-30 19:28:20 CEST; 9s ago
 Main PID: 1647 (analitico.sh)
   CGroup: /system.slice/analitico.service
           ├─1647 /bin/bash /home/pi/Documents/SH/analitico.sh
           └─1649 python3 /home/pi/Documents/PY/analitico.py

giu 30 19:28:20 pitre systemd[1]: Started Servizio Gestione Google Analytics su MQTT.

and
the script doesnt work

komarek
Posts: 146
Joined: Fri Mar 31, 2017 1:19 am

Re: python script at startup

Fri Jun 30, 2017 5:36 pm

everything works... the shell is been loaded but
when it cast the script something goes wrong... and there's no signs of where the issue is

Return to “General discussion”