User avatar
RDK
Posts: 254
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Error logs - pi stops, how do I find the cause

Tue Jul 03, 2018 1:15 pm

I've tried to search on this topic but have not found what I'm looking for.

I have a Pi v2 running Stretch. It is running a python program in a SCREEN session. This system has been running continuously for over a month without any issues. Yesterday and again today, I found the Pi non-responsive, cpu light red on continuously. Power cycling brings it back to life.

I would like to trace back the issue which has caused the system to shutdown. What are the best procedures to find the activities when it shutdown and the reasons for the shutdown?

Thanks....RDK

epoch1970
Posts: 3572
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Error logs - pi stops, how do I find the cause

Wed Jul 04, 2018 3:25 pm

Look in /var/log/syslog, backtrack to the previous boot, just before that you should see interesting stuff.
Beware of file rotation: there are older versions of syslog kept for a little while. If syslog is too recent, look into syslog.1, then in syslog.2.gz ...
Don't trust timestamps too much: the Pi boots at the last time it has saved the date (or at kernel build date), it's only after you see a line like "system time has changed" that you can trust the timestamps.
For example if the machine freezes and you restart it in the morning there is a good change syslog shows a freeze followed by an immediate boot.

If you want to know if the machine has died or not, syslog can generate '--MARK--' lines every so often, even when no logging is required.
I'm not sure you can customize the MARK facility but in the case of the Pi I would run a script that logs the value of "uptime" every so often. Uptime is a monotonic clock, it is only reset at boot.

HTH
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

User avatar
RDK
Posts: 254
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Error logs - pi stops, how do I find the cause

Wed Jul 04, 2018 4:57 pm

HTH….Thanks. I'm busy tomorrow, but will look into your ideas on Friday, hopefully the syslog's are not lost by then....RDK

User avatar
RDK
Posts: 254
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Error logs - pi stops, how do I find the cause

Sun Jul 15, 2018 3:23 pm

HTH….It took s bit longer for "Friday" to get here. In fact the Pi stopped again on the 5-th and I was busy with other things. I got to this project earlier today. Looking that SYSLOG, it might as well be written in Greek. I can see the last entry on the 5-th and the first for today, however the events leading up to the shutdown are not obvious to me.

The SYSLOG entries starting before the shutdown and up to where I restarted it today, can be found here https://drive.google.com/file/d/1AxgOso ... sp=sharing. Interestingly there seems to have been a restart initiated at Jul 5 10:11:41, but the previous entry in the log is Jul 5 10:17:01. Anyway there are several pages of meaningless (to me) output before it seems to have completed the reboot at Jul 5 10:11:55. That is the last entry before the line at Jul 15 11:30:35 for my reboot.

Can you shed any light on this issue? Are there other logs of interest to be viewed?.....RDK


PS....My Python program can initiate a "hard shutdown/restart" under certain circumstances. I suspect that may be why there was a restart at Jul 5 10:11:41...

epoch1970
Posts: 3572
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Error logs - pi stops, how do I find the cause

Mon Jul 16, 2018 11:09 am

I don't see a visible cause.
The time goes backwards because at reboot the OS sets the date to i. the date the OS was built on, or ii. the date it finds in the system's local clock (RTC), whichever most recent.
In the case of the Pi there is no RTC device so "fake-hwclock" saves the current date from time to time on the SD. This is the date (10:11) that gets picked by the OS. Briefly after boot, time is re-synced with time servers on the network and it shifts to 11:30.

"Something" happened between 10:17 and 11:30...
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

User avatar
RDK
Posts: 254
Joined: Wed Aug 13, 2014 10:19 am
Location: Wyoming and France

Re: Error logs - pi stops, how do I find the cause

Mon Jul 16, 2018 11:46 am

Thanks for the analysis. Is there another log I should be looking at?.....RDK

epoch1970
Posts: 3572
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Error logs - pi stops, how do I find the cause

Mon Jul 16, 2018 1:46 pm

Depends on your machine. If you suspect your python daemon, perhaps you can start with this one.
If you are using the watchdog, definitely consider it a suspect.

Otherwise, if nothing gets logged in syslog and syslog is in stock configuration, I see a few causes, in decreasing order of probability:
- True sudden death by loss of power. If your PSU/wiring/power source is dodgy, look into that
- Root filesystem gets dismounted, disk full or filesystem turning read-only without warning. A full/bad SD could give you that but the lack of indication of filesystem repair at reboot makes this hypothesis quite unlikely
- Death of the syslog daemon. AFAIK it should be restarted by systemd, and in normal circumstances it just doesn't go poof like that
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

Return to “Troubleshooting”