eviala
Posts: 9
Joined: Tue Sep 10, 2019 4:41 pm

Where can I hire a Raspberry Pi expert to solve random hang problems

Fri Sep 13, 2019 11:54 am

Hello,

Sorry if this is not the right place to post this. I am looking for an expert who can help me to solve random hang problems in Raspberry Pi Zero w and Raspberry Pi 3. I have a lot of experience in software development in other platforms but with Raspberry Pi I am experiencing system stability problems.

Is there anyone in the forum available or do you know where I can find an expert to hire?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23399
Joined: Sat Jul 30, 2011 7:41 pm

Re: Where can I hire a Raspberry Pi expert to solve random hang problems

Fri Sep 13, 2019 11:59 am

Best to ask questions here.

Do you have any details? Power supply, OS version, what HW is plugged in, any logs around the hang point, hard hand, soft hang (some things still working) ? As much detail as possible.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

eviala
Posts: 9
Joined: Tue Sep 10, 2019 4:41 pm

Re: Where can I hire a Raspberry Pi expert to solve random hang problems

Fri Sep 13, 2019 1:03 pm

Thanks for the answer.

The system consists of two parts:
- A Raspberry Pi Zero w that acts as node.
- A Raspberry Pi 3 that acts as Central Unit.

The Raspberry Pi 3 is powered with the standard power supply. The Raspberry Pi Zero w is plugged in a custom card with some electronic circuits and some low power relays and is powered (through the GPIO connector) with a +5V/3A switching power supply (if needed I can post the model). The power consumption of the card+RPi should be way lower than 3A.

Both Raspberrys are running a webserver and some Python scripts. Both webserver and Python scripts access a SQL database.

To make it simple I am going to focus on the Raspberry Pi Zero w problem. The OS is the last version of buster lite. In this case the problem is that the RPi resets randomly (we don't know why), and the problem is that after reset many times (but not always) the system hungs: neither the webserver nor the Python scripts are runnning (although they are started in the initialization according to the daemonlog) and the RPi cannot be accessed through SSH. The only way to recover the system is to switch off the RPi and switch it on again. If during normal operation we reboot the system from the command line (through SSH) the system operates normally. Apparently after a random reset the system detects a corruption problem in the SD card.

In order to try to understand why the RPi hungs after a faulty reset and operates normally after a reboot command, I have compared the daemon.log files information after the faulty reset and after a proper reboot.


The difference I have observed are:
- The logs are in a different order in most of the cases

- The following logs appear after the faulty reset and does not appear after the proper reboot :
Sep 10 11:55:45 rbpiz systemd-fsck[159]: 0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
Sep 10 11:55:45 rbpiz systemd-fsck[159]: Automatically removing dirty bit.
Sep 10 11:55:45 rbpiz systemd-fsck[159]: Performing changes.

- The following logs appear after the proper reboot and does not appear after the faulty reset:
* At the beggining of the booting process:
Sep 10 17:13:02 rbpiz systemd[1]: Started File System Check on Root Device.
Sep 10 17:13:02 rbpiz systemd[1]: Starting Remount Root and Kernel File Systems...
* At the end of the booting process
Sep 10 17:17:00 rbpiz systemd[1]: Created slice User Slice of UID 1001.
Sep 10 17:17:00 rbpiz systemd[1]: Starting User Runtime Directory /run/user/1001...
Sep 10 17:17:00 rbpiz systemd[1]: Started User Runtime Directory /run/user/1001.
Sep 10 17:17:00 rbpiz systemd[1]: Starting User Manager for UID 1001...
Sep 10 17:17:01 rbpiz systemd[1197]: Listening on GnuPG cryptographic agent and passphrase cache.
Sep 10 17:17:01 rbpiz systemd[1197]: Reached target Paths.
Sep 10 17:17:01 rbpiz systemd[1197]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Sep 10 17:17:01 rbpiz systemd[1197]: Reached target Timers.
Sep 10 17:17:01 rbpiz systemd[1197]: Listening on GnuPG network certificate management daemon.
Sep 10 17:17:01 rbpiz systemd[1197]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Sep 10 17:17:01 rbpiz systemd[1197]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Sep 10 17:17:01 rbpiz systemd[1197]: Reached target Sockets.
Sep 10 17:17:01 rbpiz systemd[1197]: Reached target Basic System.
Sep 10 17:17:01 rbpiz systemd[1]: Started User Manager for UID 1001.
Sep 10 17:17:01 rbpiz systemd[1197]: Reached target Default.
Sep 10 17:17:01 rbpiz systemd[1197]: Startup finished in 921ms.
Sep 10 17:17:01 rbpiz systemd[1]: Started Session c1 of user via.

The rest of the log lines are the same in both cases (but in a different order in each case).

I would be very grateful if anyone can explain me why the RPi Zero w gets hang after the faulty reset.

I also include the complete initialization sequence in the daemonlog files for both cases.

These are the log lines after the faulty reset

Sep 10 11:55:45 rbpiz systemd-modules-load[84]: Inserted module 'i2c_dev'
Sep 10 11:55:45 rbpiz fake-hwclock[76]: mar sep 10 09:17:02 UTC 2019
Sep 10 11:55:45 rbpiz systemd-fsck[93]: e2fsck 1.44.5 (15-Dec-2018)
Sep 10 11:55:45 rbpiz systemd-fsck[93]: rootfs: clean, 59691/915408 files, 576794/3821952 blocks
Sep 10 11:55:45 rbpiz systemd[1]: Started udev Coldplug all Devices.
Sep 10 11:55:45 rbpiz systemd[1]: Starting Helper to synchronize boot up for ifupdown...
Sep 10 11:55:45 rbpiz systemd[1]: Started Helper to synchronize boot up for ifupdown.
Sep 10 11:55:45 rbpiz systemd[1]: Started Remount Root and Kernel File Systems.
Sep 10 11:55:45 rbpiz systemd[1]: Starting Load/Save Random Seed...
Sep 10 11:55:45 rbpiz systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
Sep 10 11:55:45 rbpiz systemd[1]: Starting Flush Journal to Persistent Storage...
Sep 10 11:55:45 rbpiz systemd[1]: Starting Create System Users...
Sep 10 11:55:45 rbpiz systemd[1]: Started Set the console keyboard layout.
Sep 10 11:55:45 rbpiz systemd[1]: Started Load/Save Random Seed.
Sep 10 11:55:45 rbpiz systemd[1]: Started Create System Users.
Sep 10 11:55:45 rbpiz systemd[1]: Started Flush Journal to Persistent Storage.
Sep 10 11:55:45 rbpiz systemd[1]: Starting Create Static Device Nodes in /dev...
Sep 10 11:55:45 rbpiz systemd-tmpfiles[108]: [/usr/lib/tmpfiles.d/vsftpd.conf:1] Line references path below legacy directory /var/run/, updating /var/run/vsftpd/empty → /run/vsftpd/empty; please update the tmpfiles.d/ drop-in file accordingly.
Sep 10 11:55:45 rbpiz systemd[1]: Started Create Static Device Nodes in /dev.
Sep 10 11:55:45 rbpiz systemd[1]: Reached target Local File Systems (Pre).
Sep 10 11:55:45 rbpiz systemd[1]: Starting udev Kernel Device Manager...
Sep 10 11:55:45 rbpiz systemd[1]: Started udev Kernel Device Manager.
Sep 10 11:55:45 rbpiz systemd[1]: Found device /dev/ttyAMA0.
Sep 10 11:55:45 rbpiz systemd[1]: Found device /dev/disk/by-partuuid/564a05ee-01.
Sep 10 11:55:45 rbpiz systemd[1]: Starting File System Check on /dev/disk/by-partuuid/564a05ee-01...
Sep 10 11:55:45 rbpiz systemd-fsck[159]: fsck.fat 4.1 (2017-01-24)
Sep 10 11:55:45 rbpiz systemd-fsck[159]: 0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
Sep 10 11:55:45 rbpiz systemd-fsck[159]: Automatically removing dirty bit.
Sep 10 11:55:45 rbpiz systemd-fsck[159]: Performing changes.
Sep 10 11:55:45 rbpiz systemd-fsck[159]: /dev/mmcblk0p1: 219 files, 79949/516191 clusters
Sep 10 11:55:45 rbpiz systemd[1]: Started File System Check on /dev/disk/by-partuuid/564a05ee-01.
Sep 10 11:55:45 rbpiz systemd[1]: Mounting /boot...
Sep 10 11:55:45 rbpiz systemd[1]: Mounted /boot.
Sep 10 11:55:45 rbpiz systemd[1]: Reached target Local File Systems.
Sep 10 11:55:45 rbpiz systemd[1]: Starting Raise network interfaces...
Sep 10 11:55:45 rbpiz systemd[1]: Condition check resulted in Commit a transient machine-id on disk being skipped.
Sep 10 11:55:45 rbpiz systemd[1]: Starting Create Volatile Files and Directories...
Sep 10 11:55:45 rbpiz systemd[1]: Starting Preprocess NFS configuration...
Sep 10 11:55:45 rbpiz systemd[1]: Starting Set console font and keymap...
Sep 10 11:55:45 rbpiz systemd[1]: nfs-config.service: Succeeded.
Sep 10 11:55:45 rbpiz systemd[1]: Started Preprocess NFS configuration.
Sep 10 11:55:45 rbpiz systemd[1]: Condition check resulted in RPC security service for NFS client and server being skipped.
Sep 10 11:55:45 rbpiz systemd[1]: Condition check resulted in RPC security service for NFS server being skipped.
Sep 10 11:55:45 rbpiz systemd[1]: Reached target NFS client services.
Sep 10 11:55:45 rbpiz systemd[1]: Reached target Remote File Systems (Pre).
Sep 10 11:55:45 rbpiz systemd[1]: Reached target Remote File Systems.
Sep 10 11:55:45 rbpiz systemd[1]: Started Set console font and keymap.
Sep 10 11:55:45 rbpiz systemd-tmpfiles[173]: [/usr/lib/tmpfiles.d/vsftpd.conf:1] Line references path below legacy directory /var/run/, updating /var/run/vsftpd/empty → /run/vsftpd/empty; please update the tmpfiles.d/ drop-in file accordingly.
Sep 10 11:55:45 rbpiz systemd[1]: Started Create Volatile Files and Directories.
Sep 10 11:55:45 rbpiz systemd[1]: Starting Update UTMP about System Boot/Shutdown...
Sep 10 11:55:45 rbpiz systemd[1]: Starting Network Time Synchronization...
Sep 10 11:55:45 rbpiz systemd[1]: Started Update UTMP about System Boot/Shutdown.
Sep 10 11:55:45 rbpiz systemd[1]: Started Raise network interfaces.
Sep 10 11:55:45 rbpiz systemd[1]: Started Network Time Synchronization.
Sep 10 11:55:45 rbpiz systemd[1]: Reached target System Time Synchronized.
Sep 10 11:55:45 rbpiz systemd[1]: Reached target System Initialization.
Sep 10 11:55:45 rbpiz systemd[1]: Started Daily rotation of log files.
Sep 10 11:55:45 rbpiz systemd[1]: Started Daily apt download activities.
Sep 10 11:55:45 rbpiz systemd[1]: Started Daily apt upgrade and clean activities.
Sep 10 11:55:45 rbpiz systemd[1]: Started Daily Cleanup of Temporary Directories.
Sep 10 11:55:45 rbpiz systemd[1]: Listening on Avahi mDNS/DNS-SD Stack Activation Socket.
Sep 10 11:55:45 rbpiz systemd[1]: Listening on D-Bus System Message Bus Socket.
Sep 10 11:55:45 rbpiz systemd[1]: Started Daily man-db regeneration.
Sep 10 11:55:45 rbpiz systemd[1]: Reached target Timers.
Sep 10 11:55:45 rbpiz systemd[1]: Listening on triggerhappy.socket.
Sep 10 11:55:45 rbpiz systemd[1]: Reached target Sockets.
Sep 10 11:55:45 rbpiz systemd[1]: Reached target Basic System.
Sep 10 11:55:45 rbpiz systemd[1]: Starting System Logging Service...
Sep 10 11:55:45 rbpiz systemd[1]: Started D-Bus System Message Bus.
Sep 10 11:55:45 rbpiz systemd[1]: Starting rng-tools.service...
Sep 10 11:55:45 rbpiz systemd[1]: Starting dphys-swapfile - set up, mount/unmount, and delete a swap file...
Sep 10 11:55:45 rbpiz systemd[1]: Started Regular background program processing daemon.
Sep 10 11:55:45 rbpiz rng-tools[250]: Starting Hardware RNG entropy gatherer daemon: rngd.
Sep 10 11:55:45 rbpiz rngd[259]: rngd 2-unofficial-mt.14 starting up...
Sep 10 11:55:45 rbpiz systemd[1]: Starting WPA supplicant...
Sep 10 11:55:45 rbpiz rngd[259]: entropy feed to the kernel ready
Sep 10 11:55:45 rbpiz systemd[1]: Starting triggerhappy global hotkey daemon...
Sep 10 11:55:45 rbpiz systemd[1]: Condition check resulted in getty on tty2-tty6 if dbus and logind are not available being skipped.
Sep 10 11:55:46 rbpiz systemd[1]: Starting Avahi mDNS/DNS-SD Stack...
Sep 10 11:55:46 rbpiz thd[266]: Found socket passed from systemd
Sep 10 11:55:46 rbpiz systemd[1]: Starting LSB: Switch to ondemand cpu governor (unless shift key is pressed)...
Sep 10 11:55:46 rbpiz systemd[1]: Condition check resulted in Turn on SSH if /boot/ssh is present being skipped.
Sep 10 11:55:46 rbpiz systemd[1]: Starting Login Service...
Sep 10 11:55:46 rbpiz systemd[1]: Condition check resulted in Copy user wpa_supplicant.conf being skipped.
Sep 10 11:55:46 rbpiz avahi-daemon[268]: Found user 'avahi' (UID 108) and group 'avahi' (GID 113).
Sep 10 11:55:46 rbpiz avahi-daemon[268]: Successfully dropped root privileges.
Sep 10 11:55:46 rbpiz avahi-daemon[268]: avahi-daemon 0.7 starting up.
Sep 10 11:55:46 rbpiz systemd[1]: Starting dhcpcd on all interfaces...
Sep 10 11:55:46 rbpiz wpa_supplicant[264]: Successfully initialized wpa_supplicant
Sep 10 11:55:46 rbpiz avahi-daemon[268]: Successfully called chroot().
Sep 10 11:55:46 rbpiz avahi-daemon[268]: Successfully dropped remaining capabilities.
Sep 10 11:55:46 rbpiz avahi-daemon[268]: No service file found in /etc/avahi/services.
Sep 10 11:55:46 rbpiz avahi-daemon[268]: Network interface enumeration completed.
Sep 10 11:55:46 rbpiz systemd[1]: Started System Logging Service.
Sep 10 11:55:46 rbpiz avahi-daemon[268]: Server startup complete. Host name is rbpiz.local. Local service cookie is 3855448272.
Sep 10 11:55:46 rbpiz systemd[1]: Started triggerhappy global hotkey daemon.
Sep 10 11:55:46 rbpiz systemd[1]: Started rng-tools.service.
Sep 10 11:55:47 rbpiz dhcpcd[280]: dev: loaded udev
Sep 10 11:55:47 rbpiz dhcpcd[280]: no valid interfaces found
Sep 10 11:55:47 rbpiz dhcpcd[280]: no valid interfaces found
Sep 10 11:55:47 rbpiz dphys-swapfile[251]: want /var/swap=100MByte, checking existing: keeping it
Sep 10 11:55:47 rbpiz dbus-daemon[245]: [system] Successfully activated service 'org.freedesktop.systemd1'
Sep 10 11:55:48 rbpiz systemd[1]: Started Manage Sound Card State (restore and store).
Sep 10 11:55:48 rbpiz systemd-udevd[112]: Using default interface naming scheme 'v240'.
Sep 10 11:55:48 rbpiz alsactl[301]: alsactl 1.1.8 daemon started
Sep 10 11:55:48 rbpiz systemd[1]: Starting Save/Restore Sound Card State...
Sep 10 11:55:48 rbpiz systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
Sep 10 11:55:48 rbpiz raspi-config[271]: Checking if shift key is held down:Error opening '/dev/input/event*': No such file or directory
Sep 10 11:55:48 rbpiz systemd-udevd[115]: Using default interface naming scheme 'v240'.
Sep 10 11:55:48 rbpiz raspi-config[271]: No. Switching to ondemand scaling governor.
Sep 10 11:55:48 rbpiz systemd[1]: Started LSB: Switch to ondemand cpu governor (unless shift key is pressed).
Sep 10 11:55:48 rbpiz systemd[1]: Started Save/Restore Sound Card State.
Sep 10 11:55:48 rbpiz systemd[1]: Started dphys-swapfile - set up, mount/unmount, and delete a swap file.
Sep 10 11:55:48 rbpiz systemd[1]: Reached target Sound Card.
Sep 10 11:55:48 rbpiz systemd[1]: Starting Load/Save RF Kill Switch Status...
Sep 10 11:55:48 rbpiz systemd[1]: Started WPA supplicant.
Sep 10 11:55:48 rbpiz systemd[1]: Started Avahi mDNS/DNS-SD Stack.
Sep 10 11:55:48 rbpiz systemd[1]: Started Login Service.
Sep 10 11:55:49 rbpiz systemd[1]: Started Load/Save RF Kill Switch Status.
Sep 10 11:55:49 rbpiz dhcpcd[280]: wlan0: starting wpa_supplicant
Sep 10 11:55:49 rbpiz dhcpcd-run-hooks[330]: wlan0: starting wpa_supplicant
Sep 10 11:55:50 rbpiz dhcpcd[280]: wlan0: waiting for carrier
Sep 10 11:55:50 rbpiz dhcpcd[280]: wlan1: starting wpa_supplicant
Sep 10 11:55:50 rbpiz dhcpcd-run-hooks[348]: wlan1: starting wpa_supplicant
Sep 10 11:55:51 rbpiz dhcpcd[280]: wlan1: waiting for carrier
Sep 10 11:55:51 rbpiz dhcpcd[280]: wlan0: carrier acquired
Sep 10 11:55:51 rbpiz dhcpcd[280]: DUID 00:01:00:01:24:b7:ec:1f:b8:27:eb:dd:5e:58
Sep 10 11:55:51 rbpiz dhcpcd[280]: wlan0: IAID eb:f9:b0:31
Sep 10 11:55:51 rbpiz dhcpcd[280]: wlan0: adding address fe80::fdb9:aedb:e2aa:da3
Sep 10 11:55:51 rbpiz avahi-daemon[268]: Joining mDNS multicast group on interface wlan0.IPv6 with address fe80::fdb9:aedb:e2aa:da3.
Sep 10 11:55:51 rbpiz avahi-daemon[268]: New relevant interface wlan0.IPv6 for mDNS.
Sep 10 11:55:51 rbpiz avahi-daemon[268]: Registering new address record for fe80::fdb9:aedb:e2aa:da3 on wlan0.*.
Sep 10 11:55:51 rbpiz dhcpcd[280]: wlan0: probing address 10.10.10.1/24
Sep 10 11:55:51 rbpiz dhcpcd[280]: wlan0: carrier lost
Sep 10 11:55:51 rbpiz dhcpcd[280]: wlan0: deleting address fe80::fdb9:aedb:e2aa:da3
Sep 10 11:55:51 rbpiz avahi-daemon[268]: Withdrawing address record for fe80::fdb9:aedb:e2aa:da3 on wlan0.
Sep 10 11:55:51 rbpiz avahi-daemon[268]: Leaving mDNS multicast group on interface wlan0.IPv6 with address fe80::fdb9:aedb:e2aa:da3.
Sep 10 11:55:51 rbpiz avahi-daemon[268]: Interface wlan0.IPv6 no longer relevant for mDNS.
Sep 10 11:55:53 rbpiz dhcpcd[280]: wlan1: carrier acquired
Sep 10 11:55:53 rbpiz dhcpcd[280]: wlan1: IAID 38:ef:4f:48
Sep 10 11:55:53 rbpiz dhcpcd[280]: wlan1: adding address fe80::983b:3e8d:c84b:7262
Sep 10 11:55:53 rbpiz avahi-daemon[268]: Joining mDNS multicast group on interface wlan1.IPv6 with address fe80::983b:3e8d:c84b:7262.
Sep 10 11:55:53 rbpiz avahi-daemon[268]: New relevant interface wlan1.IPv6 for mDNS.
Sep 10 11:55:53 rbpiz avahi-daemon[268]: Registering new address record for fe80::983b:3e8d:c84b:7262 on wlan1.*.
Sep 10 11:55:53 rbpiz dhcpcd[280]: wlan1: probing address 192.168.2.111/24
Sep 10 11:55:53 rbpiz dhcpcd[280]: wlan1: soliciting an IPv6 router
Sep 10 11:55:54 rbpiz systemd[1]: systemd-rfkill.service: Succeeded.
Sep 10 11:55:57 rbpiz dhcpcd[280]: wlan1: using static address 192.168.2.111/24
Sep 10 11:55:57 rbpiz dhcpcd[280]: wlan1: adding route to 192.168.2.0/24
Sep 10 11:55:57 rbpiz dhcpcd[280]: wlan1: adding default route via 192.168.2.1
Sep 10 11:55:57 rbpiz avahi-daemon[268]: Joining mDNS multicast group on interface wlan1.IPv4 with address 192.168.2.111.
Sep 10 11:55:57 rbpiz avahi-daemon[268]: New relevant interface wlan1.IPv4 for mDNS.
Sep 10 11:55:57 rbpiz avahi-daemon[268]: Registering new address record for 192.168.2.111 on wlan1.IPv4.
Sep 10 11:55:58 rbpiz dhcpcd[280]: forked to background, child pid 416
Sep 10 11:55:58 rbpiz systemd[1]: Started dhcpcd on all interfaces.
Sep 10 11:55:58 rbpiz systemd[1]: Reached target Network.
Sep 10 11:55:58 rbpiz systemd[1]: Condition check resulted in fast remote file copy program daemon being skipped.
Sep 10 11:55:58 rbpiz systemd[1]: Starting OpenBSD Secure Shell server...
Sep 10 11:55:58 rbpiz systemd[1]: Starting Permit User Sessions...
Sep 10 11:55:58 rbpiz systemd[1]: Starting /etc/rc.local Compatibility...
Sep 10 11:55:58 rbpiz systemd[1]: Starting vsftpd FTP server...
Sep 10 11:55:58 rbpiz systemd[1]: Starting MariaDB 10.3.15 database server...
Sep 10 11:55:58 rbpiz systemd[1]: Started /etc/rc.local Compatibility.
Sep 10 11:55:58 rbpiz systemd[1]: Started vsftpd FTP server.
Sep 10 11:55:58 rbpiz systemd[1]: Started Permit User Sessions.
Sep 10 11:55:59 rbpiz systemd[1]: Started Serial Getty on ttyAMA0.
Sep 10 11:55:59 rbpiz systemd[1]: Started Getty on tty1.
Sep 10 11:55:59 rbpiz systemd[1]: Reached target Login Prompts.
Sep 10 11:55:59 rbpiz systemd[1]: Started OpenBSD Secure Shell server.
Sep 10 11:56:00 rbpiz mysqld[477]: 2019-09-10 11:56:00 0 [Note] /usr/sbin/mysqld (mysqld 10.3.15-MariaDB-1) starting as process 477 ...
Sep 10 11:56:02 rbpiz /etc/mysql/debian-start[512]: Upgrading MySQL tables if necessary.
Sep 10 11:56:02 rbpiz systemd[1]: Started MariaDB 10.3.15 database server.
Sep 10 11:56:02 rbpiz systemd[1]: Reached target Multi-User System.
Sep 10 11:56:02 rbpiz systemd[1]: Reached target Graphical Interface.
Sep 10 11:56:02 rbpiz systemd[1]: Starting Update UTMP about System Runlevel Changes...
Sep 10 11:56:02 rbpiz systemd[1]: Started VIACONFER WEB DEPOSITO NODEJS rev1.0.
Sep 10 11:56:02 rbpiz systemd[1]: Started VIACONFER DEPOSITO rev1.0.
Sep 10 11:56:02 rbpiz systemd[1]: systemd-update-utmp-runlevel.service: Succeeded.
Sep 10 11:56:02 rbpiz systemd[1]: Started Update UTMP about System Runlevel Changes.
Sep 10 11:56:02 rbpiz systemd[1]: Startup finished in 4.019s (kernel) + 39.757s (userspace) = 43.776s.
Sep 10 11:56:03 rbpiz /etc/mysql/debian-start[516]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Sep 10 11:56:03 rbpiz /etc/mysql/debian-start[516]: Looking for 'mysql' as: /usr/bin/mysql
Sep 10 11:56:03 rbpiz /etc/mysql/debian-start[516]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Sep 10 11:56:03 rbpiz /etc/mysql/debian-start[516]: This installation of MySQL is already upgraded to 10.3.15-MariaDB, use --force if you still need to run mysql_upgrade
Sep 10 11:56:03 rbpiz /etc/mysql/debian-start[528]: Checking for insecure root accounts.
Sep 10 11:56:03 rbpiz /etc/mysql/debian-start[532]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
Sep 10 11:56:06 rbpiz dhcpcd[416]: wlan1: no IPv6 Routers available
Sep 10 11:56:09 rbpiz systemd[1]: systemd-fsckd.service: Succeeded.


These are the log lines after a proper reboot:

Sep 10 17:13:02 rbpiz systemd-modules-load[70]: Inserted module 'i2c_dev'
Sep 10 17:13:02 rbpiz fake-hwclock[74]: mar sep 10 15:12:45 UTC 2019
Sep 10 17:13:02 rbpiz systemd[1]: Started File System Check on Root Device.
Sep 10 17:13:02 rbpiz systemd-fsck[92]: e2fsck 1.44.5 (15-Dec-2018)
Sep 10 17:13:02 rbpiz systemd-fsck[92]: rootfs: clean, 59686/915408 files, 573319/3821952 blocks
Sep 10 17:13:02 rbpiz systemd[1]: Starting Remount Root and Kernel File Systems...
Sep 10 17:13:02 rbpiz systemd[1]: Started Set the console keyboard layout.
Sep 10 17:13:02 rbpiz systemd[1]: Started Remount Root and Kernel File Systems.
Sep 10 17:13:02 rbpiz systemd[1]: Starting Create System Users...
Sep 10 17:13:02 rbpiz systemd[1]: Starting Load/Save Random Seed...
Sep 10 17:13:02 rbpiz systemd[1]: Starting Flush Journal to Persistent Storage...
Sep 10 17:13:02 rbpiz systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
Sep 10 17:13:02 rbpiz systemd[1]: Started udev Coldplug all Devices.
Sep 10 17:13:02 rbpiz systemd[1]: Starting Helper to synchronize boot up for ifupdown...
Sep 10 17:13:02 rbpiz systemd[1]: Started Helper to synchronize boot up for ifupdown.
Sep 10 17:13:02 rbpiz systemd[1]: Started Load/Save Random Seed.
Sep 10 17:13:02 rbpiz systemd[1]: Started Create System Users.
Sep 10 17:13:02 rbpiz systemd[1]: Starting Create Static Device Nodes in /dev...
Sep 10 17:13:02 rbpiz systemd[1]: Started Flush Journal to Persistent Storage.
Sep 10 17:13:02 rbpiz systemd-tmpfiles[108]: [/usr/lib/tmpfiles.d/vsftpd.conf:1] Line references path below legacy directory /var/run/, updating /var/run/vsftpd/empty → /run/vsftpd/empty; please update the tmpfiles.d/ drop-in file accordingly.
Sep 10 17:13:02 rbpiz systemd[1]: Started Create Static Device Nodes in /dev.
Sep 10 17:13:02 rbpiz systemd[1]: Reached target Local File Systems (Pre).
Sep 10 17:13:02 rbpiz systemd[1]: Starting udev Kernel Device Manager...
Sep 10 17:13:02 rbpiz systemd[1]: Started udev Kernel Device Manager.
Sep 10 17:13:02 rbpiz systemd[1]: Found device /dev/ttyAMA0.
Sep 10 17:13:02 rbpiz systemd[1]: Found device /dev/disk/by-partuuid/564a05ee-01.
Sep 10 17:13:02 rbpiz systemd[1]: Starting File System Check on /dev/disk/by-partuuid/564a05ee-01...
Sep 10 17:13:02 rbpiz systemd-fsck[161]: fsck.fat 4.1 (2017-01-24)
Sep 10 17:13:02 rbpiz systemd-fsck[161]: /dev/mmcblk0p1: 219 files, 79949/516191 clusters
Sep 10 17:13:02 rbpiz systemd[1]: Started File System Check on /dev/disk/by-partuuid/564a05ee-01.
Sep 10 17:13:02 rbpiz systemd[1]: Mounting /boot...
Sep 10 17:13:02 rbpiz systemd[1]: Mounted /boot.
Sep 10 17:13:02 rbpiz systemd[1]: Reached target Local File Systems.
Sep 10 17:13:02 rbpiz systemd[1]: Starting Set console font and keymap...
Sep 10 17:13:02 rbpiz systemd[1]: Starting Preprocess NFS configuration...
Sep 10 17:13:02 rbpiz systemd[1]: Starting Create Volatile Files and Directories...
Sep 10 17:13:02 rbpiz systemd[1]: Condition check resulted in Commit a transient machine-id on disk being skipped.
Sep 10 17:13:02 rbpiz systemd[1]: Starting Raise network interfaces...
Sep 10 17:13:02 rbpiz systemd[1]: Started Set console font and keymap.
Sep 10 17:13:02 rbpiz systemd[1]: nfs-config.service: Succeeded.
Sep 10 17:13:02 rbpiz systemd[1]: Started Preprocess NFS configuration.
Sep 10 17:13:02 rbpiz systemd[1]: Condition check resulted in RPC security service for NFS client and server being skipped.
Sep 10 17:13:02 rbpiz systemd[1]: Condition check resulted in RPC security service for NFS server being skipped.
Sep 10 17:13:02 rbpiz systemd[1]: Reached target NFS client services.
Sep 10 17:13:02 rbpiz systemd[1]: Reached target Remote File Systems (Pre).
Sep 10 17:13:03 rbpiz systemd[1]: Reached target Remote File Systems.
Sep 10 17:13:03 rbpiz systemd-tmpfiles[176]: [/usr/lib/tmpfiles.d/vsftpd.conf:1] Line references path below legacy directory /var/run/, updating /var/run/vsftpd/empty → /run/vsftpd/empty; please update the tmpfiles.d/ drop-in file accordingly.
Sep 10 17:13:03 rbpiz systemd[1]: Started Create Volatile Files and Directories.
Sep 10 17:13:03 rbpiz systemd[1]: Starting Update UTMP about System Boot/Shutdown...
Sep 10 17:13:03 rbpiz systemd[1]: Starting Network Time Synchronization...
Sep 10 17:13:03 rbpiz systemd[1]: Started Update UTMP about System Boot/Shutdown.
Sep 10 17:13:03 rbpiz systemd[1]: Started Raise network interfaces.
Sep 10 17:13:03 rbpiz systemd[1]: Started Network Time Synchronization.
Sep 10 17:13:03 rbpiz systemd[1]: Reached target System Initialization.
Sep 10 17:13:03 rbpiz systemd[1]: Listening on triggerhappy.socket.
Sep 10 17:13:03 rbpiz systemd[1]: Started Daily Cleanup of Temporary Directories.
Sep 10 17:13:03 rbpiz systemd[1]: Listening on Avahi mDNS/DNS-SD Stack Activation Socket.
Sep 10 17:13:03 rbpiz systemd[1]: Listening on D-Bus System Message Bus Socket.
Sep 10 17:13:03 rbpiz systemd[1]: Reached target Sockets.
Sep 10 17:13:03 rbpiz systemd[1]: Reached target Basic System.
Sep 10 17:13:03 rbpiz systemd[1]: Starting rng-tools.service...
Sep 10 17:13:03 rbpiz systemd[1]: Condition check resulted in Turn on SSH if /boot/ssh is present being skipped.
Sep 10 17:13:03 rbpiz systemd[1]: Starting LSB: Switch to ondemand cpu governor (unless shift key is pressed)...
Sep 10 17:13:03 rbpiz systemd[1]: Starting System Logging Service...
Sep 10 17:13:03 rbpiz rng-tools[245]: Starting Hardware RNG entropy gatherer daemon: rngd.
Sep 10 17:13:03 rbpiz rngd[249]: rngd 2-unofficial-mt.14 starting up...
Sep 10 17:13:03 rbpiz systemd[1]: Started Regular background program processing daemon.
Sep 10 17:13:03 rbpiz rngd[249]: entropy feed to the kernel ready
Sep 10 17:13:03 rbpiz systemd[1]: Starting Login Service...
Sep 10 17:13:03 rbpiz systemd[1]: Condition check resulted in Copy user wpa_supplicant.conf being skipped.
Sep 10 17:13:03 rbpiz systemd[1]: Starting dhcpcd on all interfaces...
Sep 10 17:13:02 rbpiz systemd[1]: Starting triggerhappy global hotkey daemon...
Sep 10 17:13:03 rbpiz systemd[1]: Starting dphys-swapfile - set up, mount/unmount, and delete a swap file...
Sep 10 17:13:03 rbpiz systemd[1]: Condition check resulted in getty on tty2-tty6 if dbus and logind are not available being skipped.
Sep 10 17:13:03 rbpiz thd[259]: Found socket passed from systemd
Sep 10 17:13:03 rbpiz dhcpcd[258]: dev: loaded udev
Sep 10 17:13:03 rbpiz systemd[1]: Started D-Bus System Message Bus.
Sep 10 17:13:03 rbpiz systemd[1]: Starting WPA supplicant...
Sep 10 17:13:03 rbpiz systemd[1]: Starting Avahi mDNS/DNS-SD Stack...
Sep 10 17:13:03 rbpiz dhcpcd[258]: no valid interfaces found
Sep 10 17:13:03 rbpiz dhcpcd[258]: no valid interfaces found
Sep 10 17:13:03 rbpiz systemd[1]: Reached target System Time Synchronized.
Sep 10 17:13:03 rbpiz systemd[1]: Started Daily apt download activities.
Sep 10 17:13:03 rbpiz systemd[1]: Started Daily apt upgrade and clean activities.
Sep 10 17:13:03 rbpiz systemd[1]: Started Daily man-db regeneration.
Sep 10 17:13:03 rbpiz systemd[1]: Started Daily rotation of log files.
Sep 10 17:13:04 rbpiz systemd[1]: Reached target Timers.
Sep 10 17:13:04 rbpiz avahi-daemon[275]: Found user 'avahi' (UID 108) and group 'avahi' (GID 113).
Sep 10 17:13:04 rbpiz systemd[1]: Started System Logging Service.
Sep 10 17:13:04 rbpiz avahi-daemon[275]: Successfully dropped root privileges.
Sep 10 17:13:04 rbpiz avahi-daemon[275]: avahi-daemon 0.7 starting up.
Sep 10 17:13:04 rbpiz systemd[1]: Started triggerhappy global hotkey daemon.
Sep 10 17:13:04 rbpiz systemd[1]: Started rng-tools.service.
Sep 10 17:13:04 rbpiz avahi-daemon[275]: Successfully called chroot().
Sep 10 17:13:04 rbpiz avahi-daemon[275]: Successfully dropped remaining capabilities.
Sep 10 17:13:04 rbpiz avahi-daemon[275]: No service file found in /etc/avahi/services.
Sep 10 17:13:04 rbpiz avahi-daemon[275]: Network interface enumeration completed.
Sep 10 17:13:04 rbpiz avahi-daemon[275]: Server startup complete. Host name is rbpiz.local. Local service cookie is 3733992928.
Sep 10 17:13:04 rbpiz wpa_supplicant[273]: Successfully initialized wpa_supplicant
Sep 10 17:13:04 rbpiz dphys-swapfile[264]: want /var/swap=100MByte, checking existing: keeping it
Sep 10 17:13:05 rbpiz systemd-udevd[110]: Using default interface naming scheme 'v240'.
Sep 10 17:13:05 rbpiz dbus-daemon[265]: [system] Successfully activated service 'org.freedesktop.systemd1'
Sep 10 17:13:05 rbpiz systemd[1]: Started Manage Sound Card State (restore and store).
Sep 10 17:13:05 rbpiz systemd-udevd[114]: Using default interface naming scheme 'v240'.
Sep 10 17:13:05 rbpiz alsactl[302]: alsactl 1.1.8 daemon started
Sep 10 17:13:05 rbpiz systemd[1]: Starting Save/Restore Sound Card State...
Sep 10 17:13:05 rbpiz systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
Sep 10 17:13:05 rbpiz raspi-config[246]: Checking if shift key is held down:Error opening '/dev/input/event*': No such file or directory
Sep 10 17:13:05 rbpiz raspi-config[246]: No. Switching to ondemand scaling governor.
Sep 10 17:13:05 rbpiz systemd[1]: Started LSB: Switch to ondemand cpu governor (unless shift key is pressed).
Sep 10 17:13:05 rbpiz systemd[1]: Started dphys-swapfile - set up, mount/unmount, and delete a swap file.
Sep 10 17:13:06 rbpiz systemd[1]: Started Save/Restore Sound Card State.
Sep 10 17:13:06 rbpiz systemd[1]: Reached target Sound Card.
Sep 10 17:13:06 rbpiz systemd[1]: Starting Load/Save RF Kill Switch Status...
Sep 10 17:13:06 rbpiz systemd[1]: Started Avahi mDNS/DNS-SD Stack.
Sep 10 17:13:06 rbpiz systemd[1]: Started WPA supplicant.
Sep 10 17:13:06 rbpiz systemd[1]: Started Login Service.
Sep 10 17:13:06 rbpiz systemd[1]: Started Load/Save RF Kill Switch Status.
Sep 10 17:13:07 rbpiz dhcpcd[258]: wlan0: starting wpa_supplicant
Sep 10 17:13:07 rbpiz dhcpcd-run-hooks[330]: wlan0: starting wpa_supplicant
Sep 10 17:13:07 rbpiz dhcpcd[258]: wlan0: waiting for carrier
Sep 10 17:13:07 rbpiz dhcpcd[258]: wlan1: starting wpa_supplicant
Sep 10 17:13:07 rbpiz dhcpcd-run-hooks[348]: wlan1: starting wpa_supplicant
Sep 10 17:13:08 rbpiz dhcpcd[258]: wlan1: waiting for carrier
Sep 10 17:13:08 rbpiz dhcpcd[258]: wlan0: carrier acquired
Sep 10 17:13:08 rbpiz dhcpcd[258]: DUID 00:01:00:01:24:b7:ec:1f:b8:27:eb:dd:5e:58
Sep 10 17:13:08 rbpiz dhcpcd[258]: wlan0: IAID eb:f9:b0:31
Sep 10 17:13:08 rbpiz dhcpcd[258]: wlan0: adding address fe80::fdb9:aedb:e2aa:da3
Sep 10 17:13:08 rbpiz avahi-daemon[275]: Joining mDNS multicast group on interface wlan0.IPv6 with address fe80::fdb9:aedb:e2aa:da3.
Sep 10 17:13:08 rbpiz avahi-daemon[275]: New relevant interface wlan0.IPv6 for mDNS.
Sep 10 17:13:08 rbpiz avahi-daemon[275]: Registering new address record for fe80::fdb9:aedb:e2aa:da3 on wlan0.*.
Sep 10 17:13:08 rbpiz dhcpcd[258]: wlan0: probing address 10.10.10.1/24
Sep 10 17:13:08 rbpiz dhcpcd[258]: wlan0: carrier lost
Sep 10 17:13:08 rbpiz dhcpcd[258]: wlan0: deleting address fe80::fdb9:aedb:e2aa:da3
Sep 10 17:13:08 rbpiz avahi-daemon[275]: Withdrawing address record for fe80::fdb9:aedb:e2aa:da3 on wlan0.
Sep 10 17:13:08 rbpiz avahi-daemon[275]: Leaving mDNS multicast group on interface wlan0.IPv6 with address fe80::fdb9:aedb:e2aa:da3.
Sep 10 17:13:08 rbpiz avahi-daemon[275]: Interface wlan0.IPv6 no longer relevant for mDNS.
Sep 10 17:13:10 rbpiz dhcpcd[258]: wlan1: carrier acquired
Sep 10 17:13:10 rbpiz dhcpcd[258]: wlan1: IAID 38:ef:4f:48
Sep 10 17:13:10 rbpiz dhcpcd[258]: wlan1: adding address fe80::983b:3e8d:c84b:7262
Sep 10 17:13:10 rbpiz avahi-daemon[275]: Joining mDNS multicast group on interface wlan1.IPv6 with address fe80::983b:3e8d:c84b:7262.
Sep 10 17:13:10 rbpiz avahi-daemon[275]: New relevant interface wlan1.IPv6 for mDNS.
Sep 10 17:13:10 rbpiz avahi-daemon[275]: Registering new address record for fe80::983b:3e8d:c84b:7262 on wlan1.*.
Sep 10 17:13:10 rbpiz dhcpcd[258]: wlan1: probing address 192.168.2.111/24
Sep 10 17:13:11 rbpiz dhcpcd[258]: wlan1: soliciting an IPv6 router
Sep 10 17:13:11 rbpiz systemd[1]: systemd-rfkill.service: Succeeded.
Sep 10 17:13:16 rbpiz dhcpcd[258]: wlan1: using static address 192.168.2.111/24
Sep 10 17:13:16 rbpiz dhcpcd[258]: wlan1: adding route to 192.168.2.0/24
Sep 10 17:13:16 rbpiz dhcpcd[258]: wlan1: adding default route via 192.168.2.1
Sep 10 17:13:16 rbpiz avahi-daemon[275]: Joining mDNS multicast group on interface wlan1.IPv4 with address 192.168.2.111.
Sep 10 17:13:16 rbpiz avahi-daemon[275]: New relevant interface wlan1.IPv4 for mDNS.
Sep 10 17:13:16 rbpiz avahi-daemon[275]: Registering new address record for 192.168.2.111 on wlan1.IPv4.
Sep 10 17:13:16 rbpiz dhcpcd[258]: forked to background, child pid 416
Sep 10 17:13:16 rbpiz systemd[1]: Started dhcpcd on all interfaces.
Sep 10 17:13:16 rbpiz systemd[1]: Reached target Network.
Sep 10 17:13:16 rbpiz systemd[1]: Condition check resulted in fast remote file copy program daemon being skipped.
Sep 10 17:13:16 rbpiz systemd[1]: Starting OpenBSD Secure Shell server...
Sep 10 17:13:16 rbpiz systemd[1]: Starting Permit User Sessions...
Sep 10 17:13:16 rbpiz systemd[1]: Starting /etc/rc.local Compatibility...
Sep 10 17:13:16 rbpiz systemd[1]: Starting vsftpd FTP server...
Sep 10 17:13:16 rbpiz systemd[1]: Starting MariaDB 10.3.15 database server...
Sep 10 17:13:16 rbpiz systemd[1]: Started /etc/rc.local Compatibility.
Sep 10 17:13:16 rbpiz systemd[1]: Started vsftpd FTP server.
Sep 10 17:13:17 rbpiz systemd[1]: Started Permit User Sessions.
Sep 10 17:13:17 rbpiz systemd[1]: Started Getty on tty1.
Sep 10 17:13:17 rbpiz systemd[1]: Started Serial Getty on ttyAMA0.
Sep 10 17:13:17 rbpiz systemd[1]: Reached target Login Prompts.
Sep 10 17:13:17 rbpiz systemd[1]: Started OpenBSD Secure Shell server.
Sep 10 17:13:18 rbpiz mysqld[477]: 2019-09-10 17:13:18 0 [Note] /usr/sbin/mysqld (mysqld 10.3.15-MariaDB-1) starting as process 477 ...
Sep 10 17:13:20 rbpiz /etc/mysql/debian-start[511]: Upgrading MySQL tables if necessary.
Sep 10 17:13:20 rbpiz systemd[1]: Started MariaDB 10.3.15 database server.
Sep 10 17:13:20 rbpiz systemd[1]: Reached target Multi-User System.
Sep 10 17:13:20 rbpiz systemd[1]: Started VIACONFER DEPOSITO rev1.0.
Sep 10 17:13:20 rbpiz systemd[1]: Reached target Graphical Interface.
Sep 10 17:13:20 rbpiz systemd[1]: Starting Update UTMP about System Runlevel Changes...
Sep 10 17:13:20 rbpiz systemd[1]: Started VIACONFER WEB DEPOSITO NODEJS rev1.0.
Sep 10 17:13:20 rbpiz systemd[1]: systemd-update-utmp-runlevel.service: Succeeded.
Sep 10 17:13:21 rbpiz systemd[1]: Started Update UTMP about System Runlevel Changes.
Sep 10 17:13:21 rbpiz systemd[1]: Startup finished in 3.330s (kernel) + 40.568s (userspace) = 43.898s.
Sep 10 17:13:21 rbpiz /etc/mysql/debian-start[515]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Sep 10 17:13:21 rbpiz /etc/mysql/debian-start[515]: Looking for 'mysql' as: /usr/bin/mysql
Sep 10 17:13:21 rbpiz /etc/mysql/debian-start[515]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Sep 10 17:13:21 rbpiz /etc/mysql/debian-start[515]: This installation of MySQL is already upgraded to 10.3.15-MariaDB, use --force if you still need to run mysql_upgrade
Sep 10 17:13:21 rbpiz /etc/mysql/debian-start[527]: Checking for insecure root accounts.
Sep 10 17:13:21 rbpiz /etc/mysql/debian-start[531]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
Sep 10 17:13:24 rbpiz dhcpcd[416]: wlan1: no IPv6 Routers available
Sep 10 17:13:27 rbpiz systemd[1]: systemd-fsckd.service: Succeeded.
Sep 10 17:17:00 rbpiz systemd[1]: Created slice User Slice of UID 1001.
Sep 10 17:17:00 rbpiz systemd[1]: Starting User Runtime Directory /run/user/1001...
Sep 10 17:17:00 rbpiz systemd[1]: Started User Runtime Directory /run/user/1001.
Sep 10 17:17:00 rbpiz systemd[1]: Starting User Manager for UID 1001...
Sep 10 17:17:01 rbpiz systemd[1197]: Listening on GnuPG cryptographic agent and passphrase cache.
Sep 10 17:17:01 rbpiz systemd[1197]: Reached target Paths.
Sep 10 17:17:01 rbpiz systemd[1197]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Sep 10 17:17:01 rbpiz systemd[1197]: Reached target Timers.
Sep 10 17:17:01 rbpiz systemd[1197]: Listening on GnuPG network certificate management daemon.
Sep 10 17:17:01 rbpiz systemd[1197]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Sep 10 17:17:01 rbpiz systemd[1197]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Sep 10 17:17:01 rbpiz systemd[1197]: Reached target Sockets.
Sep 10 17:17:01 rbpiz systemd[1197]: Reached target Basic System.
Sep 10 17:17:01 rbpiz systemd[1]: Started User Manager for UID 1001.
Sep 10 17:17:01 rbpiz systemd[1197]: Reached target Default.
Sep 10 17:17:01 rbpiz systemd[1197]: Startup finished in 921ms.
Sep 10 17:17:01 rbpiz systemd[1]: Started Session c1 of user via.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23399
Joined: Sat Jul 30, 2011 7:41 pm

Re: Where can I hire a Raspberry Pi expert to solve random hang problems

Fri Sep 13, 2019 2:31 pm

I would be suspicious first of all in the custom hardware you have attached to the Pi0, and perhaps the software that is driving it. Is there any way you can test without it?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

eviala
Posts: 9
Joined: Tue Sep 10, 2019 4:41 pm

Re: Where can I hire a Raspberry Pi expert to solve random hang problems

Fri Sep 13, 2019 2:52 pm

Thanks for your answer jamesh. Yes I can test this during the week-end and let you know the result. The problem is that the hang problem is random. The RPi may run during 4 days without problem and get hung the fifth day.

Why do you think it is a hardware problem? There are several Python scripts and a webserver running simultaneously and accessing to the SQL data base. And in the initialization sequence after the faulty reset (just before the system gets hung) I get the following lines in the daemonlog:

Sep 10 11:55:45 rbpiz systemd-fsck[159]: 0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
Sep 10 11:55:45 rbpiz systemd-fsck[159]: Automatically removing dirty bit.
Sep 10 11:55:45 rbpiz systemd-fsck[159]: Performing changes.

Couldn't it be a problem due to too much accesses to the SQL database?

And, in any case, why the RPi gets hung AFTER the reset?

jayben
Posts: 6
Joined: Mon Aug 19, 2019 9:56 pm

Re: Where can I hire a Raspberry Pi expert to solve random hang problems

Sun Sep 15, 2019 8:37 am

Intermittent problems like this are a nightmare; as Jamesh suggests, you can try to isolate some of the hardware or software components, to see if the problem goes away, but if it only occurs every few days, that'll be a slow business.

Personally I'd adopt a different approach: try and make the fault occur more frequently. Speed everything up, so your systems are working much harder; feed in lots of Web page requests using curl or wget (or a quick-and-dirty traffic generator you've written), and generally stress the system.

The scary thing is that you'll probably crash the system after the first 5 minutes of this punishment, but that is good; you've found a weakness that may or may not be related to your original problem. Chase that down, then repeat the exercise until you know that the system can handle everything thrown at it - failing gracefully under overload, but not crashing.

Then wind the traffic back down to normal levels, and relax...

eviala
Posts: 9
Joined: Tue Sep 10, 2019 4:41 pm

Re: Where can I hire a Raspberry Pi expert to solve random hang problems

Mon Sep 16, 2019 2:13 pm

Thank you jamesh and jayben for your advices. I am going to implement both of them and I will keep you informed.
Nevertheless, my main objective is to understand why the Raspberry gets block after a faulty reset.
Apparently, the way I can detect the faulty reset in because in the initialization sequence after the reset the following lines appear in the daemonlog.
Sep 10 11:55:45 rbpiz systemd-fsck[159]: 0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
Sep 10 11:55:45 rbpiz systemd-fsck[159]: Automatically removing dirty bit.
Sep 10 11:55:45 rbpiz systemd-fsck[159]: Performing change

According to what I have read in Internet this seems to indicate that there has been some kind of corruption. But theoretically the dirty bit is removed and the problem is solved. Anyhow the initialization sequence does not finish and the RPi gets block.

If then I power off and power on the RPi, the initialization is completed with no errors and the system starts properly. If I reset the board with a reboot command the RPi starts with no problem. So the RPi has not any problem that prevent it from start in normal conditions.
So my question is, what makes the RPi to get hung during initialization after a faulty reset? Is there any anything I can do to make the system to start properly after a faulty reset (changing the initialization sequence configuration for example )?

eviala
Posts: 9
Joined: Tue Sep 10, 2019 4:41 pm

Re: Where can I hire a Raspberry Pi expert to solve random hang problems

Thu Sep 19, 2019 2:37 pm

"I would be suspicious first of all in the custom hardware you have attached to the Pi0, and perhaps the software that is driving it. Is there any way you can test without it?"

I leave a RPi Zero disconnected from the hardware and it fails with the same problem (faulty reset with dirty bit and hang) than the RPi .

"Personally I'd adopt a different approach: try and make the fault occur more frequently. Speed everything up, so your systems are working much harder; feed in lots of Web page requests using curl or wget (or a quick-and-dirty traffic generator you've written), and generally stress the system."

I run a Google plug-in that produces a web page request every second we leave it during two days and we did not observe any different behavior. Some of the RPi Zero got hung randomly. The frequency of the hangs was similar to the one observed when no constant web page requests were run.

I am really lost with this problem. Could anyone please review the log text (daemon.log) I included in my first post and try to find any tip on the problem? If necessary, I can post more log information.
If none can help me to solve the problem through the forum, could anyone tell me where I can find an expert in Raspberry Pi I can hire to help me to solve the problem by direct remote debugging?

Thanks in advance

User avatar
davidcoton
Posts: 4039
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK

Re: Where can I hire a Raspberry Pi expert to solve random hang problems

Thu Sep 19, 2019 3:13 pm

eviala wrote:
Thu Sep 19, 2019 2:37 pm
If none can help me to solve the problem through the forum, could anyone tell me where I can find an expert in Raspberry Pi I can hire to help me to solve the problem by direct remote debugging?
There's a Forum section "Wanted" that has been used to recruit consultants. A handful of regulars (to that section, if not the rest of the Forum) usually reply. I can't speak for their actual expertise. Caveat Emptor.
Signature retired

ejolson
Posts: 3436
Joined: Tue Mar 18, 2014 11:47 am

Re: Where can I hire a Raspberry Pi expert to solve random hang problems

Thu Sep 19, 2019 3:46 pm

eviala wrote:
Thu Sep 19, 2019 2:37 pm
"I would be suspicious first of all in the custom hardware you have attached to the Pi0, and perhaps the software that is driving it. Is there any way you can test without it?"

I leave a RPi Zero disconnected from the hardware and it fails with the same problem (faulty reset with dirty bit and hang) than the RPi .
I think the reason to suspect a hardware problem is because you yourself said you know software but not so much hardware.

Memory and SD card corruption can be caused by low voltages, electrically noisy power supplies and overloaded GPIO. The fact that the Pi computers do not have ECC further increases the likelihood of memory errors. At any rate, if the filesystem has what appears to be a significant amount of corruption, the automatic repair algorithm that runs at boot time gets scared, quits and waits for user intervention. This may cause the failure to boot you are experiencing.

While the filesystem format repair algorithm can be set to automatically proceed even if things look grim, it would be better to track down what is causing the problem.

One way I have used to stabilize Pi and other computers in the past had been to lower the clock speed of the processor and memory. In your case I would edit config.txt in the boot partition and set the CPU frequency explicitly to 700. That's an easy change such that if it doesn't make a difference, you can just as easily put things back the way they were.

With power supplies there are two things to check: Whether the correct 5V potential actually appears at the correct places in the Pi Zero and how clean the 5V supply is. The potential can be checked with a cheap multimeter purchased from eBay, Amazon or your favorite discount electronics store. How clean the voltage source is can be checked with a scope or oscilloscope. As scopes are expensive, it's better to have such a device at work or through a friend. If you have a business relationship with the owner of a local electronics store--for example the store which sold you the power supply--you may be able to get them to check the power supply output. Note there are a number of 5V supplies designed to power things other than computers for which electrical noise can be a problem.

Good stress tests are important for identifying such problems. Such tests are difficult to create but can result in new patentable technology if one is lucky.
Last edited by ejolson on Thu Sep 19, 2019 3:57 pm, edited 4 times in total.

eviala
Posts: 9
Joined: Tue Sep 10, 2019 4:41 pm

Re: Where can I hire a Raspberry Pi expert to solve random hang problems

Thu Sep 19, 2019 3:53 pm

"There's a Forum section "Wanted" that has been used to recruit consultants. A handful of regulars (to that section, if not the rest of the Forum) usually reply. I can't speak for their actual expertise. Caveat Emptor."

Thank you for your anwer.

eviala
Posts: 9
Joined: Tue Sep 10, 2019 4:41 pm

Re: Where can I hire a Raspberry Pi expert to solve random hang problems

Thu Sep 19, 2019 4:26 pm

Thank you so much for your answer ejolson.


"I think the reason to suspect a hardware problem is because you yourself said you know software but not so much hardware."

I don't know if I said this but I am hardware expert and I have extensively done microcontrollers programming but it is my first time with Raspberry.


"Memory and SD card corruption can be caused by low voltages, electrically noisy power supplies and overloaded GPIO. The fact that the Pi computers do not have ECC further increases the likelihood of memory errors. At any rate, if the filesystem has what appears to be a significant amount of corruption, the automatic repair algorithm that runs at boot time gets scared, quits and waits for user intervention. This may cause the failure to boot you are experiencing"

What puzzles me is that the system gives the "dirty bit" error but in the same booting sequence it seems that the problem with the dirty bit is solved:
Sep 10 11:55:45 rbpiz systemd-fsck[159]: 0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
Sep 10 11:55:45 rbpiz systemd-fsck[159]: Automatically removing dirty bit.
Sep 10 11:55:45 rbpiz systemd-fsck[159]: Performing changes.
But then it get blocks. Besides if I swith on and off (without changing anything in the SD card) the sytem starts with no errors. So the SD card problem seems to be solved. The question is: if the problem of the dirty bit was corrected in the booting sequence, why the system gets blocked?


"While the filesystem format repair algorithm can be set to automatically proceed even if things look grim, it would be better to track down what is causing the problem."

Yes this is my next objective but now I really need to solve the problem with the RPi getting hung. I need to make the RPi to finish the booting sequence or at least to understand why the RPi does not finishes the booting sequence.


"One way I have used to stabilize Pi and other computers in the past had been to lower the clock speed of the processor and memory. In your case I would edit config.txt in the boot partition and set the CPU frequency explicitly to 700. That's an easy change sick that ifit doesn't make a difference, you can just as easily put things back the way they were."

I have read somewhere in internet that it is not possible to change this parameter in the RPi Zero w. I try to do it in Raspi-config and I got a message that overclocking is not posible and I could not find any place where I could change this parameter.


"With power supplies there are two things to check: Whether the correct 5V potential actually appears at the correct places in the Pi Zero and how clean the 5V supply is. The potential can be checked with a cheap multimeter purchased from eBay, Amazon or your favorite discount electronics store. How clean the voltage source can be checked with a scope or oscilloscope. As they are expensive, it's better to have such a device at work or through a friend. If you have a business relationship with the owner of a local electronics store--for example the store which sold you the power supply--you may be able to get them to check the output using a good scope. Note there are a number of 5V supplies designed to power things other than computers for which electrical noise is not a problem."

I have oscilloscopes. I have checked the +5V in one of the switching power supplies (connected to the RPi Zero w) and I have obtained a DC value of 5,1V and a maximum ripple of -/+100 mV. Anyhow the error also appears with RPi Zero w powered with the standard power supply. And all the power supplies are connected to a UPS.

"Good stress tests are important for identifying such problems. Such tests are difficult to create but can result in new patentable technology if one is lucky."

I have already tried to stress the web server with no significant differences with respect to the nomal operation.
I am going to try to stress the system on the side of the Python scripts that are running. But I need someone to guide me on how to debug the system, how to increase the log information (as what I am getting now form the logs is not very helpful).

Thanks again for yor help.

Return to “Advanced users”