mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

systemd high cpu usage

Thu Feb 19, 2015 8:58 pm

Hi, I installed webmin following the readme in this repository https://github.com/webmin/webmin.
After that I have the

Code: Select all

systemd
process that takes always at least the 29% of the cpu.
What happened?

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: systemd high cpu usage

Wed Jun 03, 2015 7:55 pm

no hints?

fruitoftheloom
Posts: 20685
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: systemd high cpu usage

Wed Jun 03, 2015 7:57 pm

mcgyver83 wrote:no hints?
Operating System :?:
Retired disgracefully.....

myoung008
Posts: 55
Joined: Mon Mar 18, 2013 9:56 pm

Re: systemd high cpu usage

Wed Jun 03, 2015 10:02 pm

My guess is that something is causing webmin to fail and systemd is repeatedly trying to restart it.

You've obviously strayed from Raspbian Wheezy, more info would be helpful.

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: systemd high cpu usage

Fri Jun 05, 2015 8:30 am

I'm on jessie.
I' stop webmin and efter installed other services, and the issue happens again.
No info in the logs, how can I detect which service causes the issue?

myoung008
Posts: 55
Joined: Mon Mar 18, 2013 9:56 pm

Re: systemd high cpu usage

Fri Jun 05, 2015 5:22 pm

Is it systemd itself that's taking the cpu time or systemd-*? (where * could be one of several different programs that fall under the systemd umbrella)

systemd doesn't use the normal logs. Are you using journalctl to view the journal? Systemd logs lots of data there. Anything going on should be there.

"systemctl" without any parameters will give you a list of the current status of all units. Look for those that are failing or consistently restarting.

Using top can you see a particular process starting and then dying quickly?

I'm not running Jesse but am running systemd on Gentoo and don't have any cpu consumption issues.

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: systemd high cpu usage

Mon Jun 08, 2015 8:34 pm

Using top I see that is systemd that is taking 37% of the cpu with only the raspberry just restarted.
If I run

Code: Select all

sudo journalctl
I have a lot of

Code: Select all

jun 08 22:15:27 raspberrypi sslh[466]: connection from localhost:55642 to localhost:https forwarded from localhost:55643 to localhost:https
a sometimes

Code: Select all

giu 08 22:15:01 raspberrypi systemd-journal[92]: Suppressed 182 messages from /system.slice/sslh.service
giu 08 22:35:05 raspberrypi systemd[1]: Unit dbus.service entered failed state.
giu 08 22:35:05 raspberrypi systemd[1]: systemd-logind.service has no holdoff time, scheduling restart.
giu 08 22:35:05 raspberrypi systemd[1]: Started System Logging Service.
giu 08 22:35:41 raspberrypi systemd[1]: ModemManager.service start operation timed out. Terminating.
giu 08 22:35:41 raspberrypi ModemManager[464]: <info>  Caught signal, shutting down...
giu 08 22:35:41 raspberrypi ModemManager[464]: <info>  ModemManager is shut down
giu 08 22:35:41 raspberrypi systemd[1]: Failed to start Modem Manager.
giu 08 22:35:41 raspberrypi systemd[1]: Unit ModemManager.service entered failed state.
giu 08 22:40:02 raspberrypi CRON[967]: pam_unix(cron:session): session opened for user pi by (uid=0)
giu 08 22:40:02 raspberrypi CRON[968]: (pi) CMD (hd-idle -a sda1 -i 5 >/dev/null 2>&1)
giu 08 22:40:02 raspberrypi CRON[967]: pam_unix(cron:session): session closed for user pi
giu 08 22:40:45 raspberrypi sshd[1014]: Accepted publickey for pi from 192.168.0.11 port 38635 ssh2: RSA 81:5c
giu 08 22:40:45 raspberrypi sshd[1014]: pam_unix(sshd:session): session opened for user pi by (uid=0)
giu 08 22:41:10 raspberrypi sshd[1014]: pam_systemd(sshd:session): Failed to create session: Connection timed 
giu 08 22:41:10 raspberrypi sshd[1014]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or dire
giu 08 22:41:10 raspberrypi sshd[1014]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or dire
I'm using sslh to drive traffic from incoming 443 call to local:22 for ssh ora local:443 for https.
Now I cannot execute

Code: Select all

sudo service xxxx stop
the system seems blocked and I tried to wait 10 minutes to stop the webmin service but nothing happened.
Also others services cannot be stopped.

myoung008
Posts: 55
Joined: Mon Mar 18, 2013 9:56 pm

Re: systemd high cpu usage

Mon Jun 08, 2015 10:26 pm

I've hacked up your last message to address the items a little more clearly...
I'm using sslh to drive traffic from incoming 443 call to local:22 for ssh ora local:443 for https.

Code: Select all

jun 08 22:15:27 raspberrypi sslh[466]: connection from localhost:55642 to localhost:https forwarded from localhost:55643 to localhost:https
It looks like you've set up a loop. You're forwarding localhost:443 in a roundabout way to localhost:443. It seems to me that what you really want to do is use sslh to forward to the webmin port @ localhost:10000 (IIRC). I've never used sslh, but that's the pattern I would use to proxy services through nginx.
Now I cannot execute

Code: Select all

sudo service xxxx stop
Debian Jesse may have a shim for service to remain compatible with old scripts/habits; but the syntax for starting/stopping services with systemd is:

Code: Select all

sudo systemctl stop xxxx

Code: Select all

giu 08 22:35:05 raspberrypi systemd[1]: Unit dbus.service entered failed state.
That means dbus has died. systemd accepts it's commands through dbus so you can't do much at this point. You may be able to restart dbus manually with:

Code: Select all

dbus-daemon --system

Code: Select all

giu 08 22:40:45 raspberrypi sshd[1014]: Accepted publickey for pi from 192.168.0.11 port 38635 ssh2: RSA 81:5c
giu 08 22:40:45 raspberrypi sshd[1014]: pam_unix(sshd:session): session opened for user pi by (uid=0)
giu 08 22:41:10 raspberrypi sshd[1014]: pam_systemd(sshd:session): Failed to create session: Connection timed 
the system seems blocked and I tried to wait 10 minutes to stop the webmin service but nothing happened.
Also others services cannot be stopped.
I think the pam_systemd failure and the other symptoms regarding starting/stopping services is probably related to dbus dying. dbus dying may be caused by resource exhaustion from the packets looping through sslh, in which case the manual dbus restart probably won't work. The correct configuration for sslh may take care of all of it though.

Hope this helps.

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: systemd high cpu usage

Tue Jun 09, 2015 8:37 pm

I'm using sslh since the beginning, 1 year ago...probably something I did trying to configure a usb wifi dongle as AccessPoint :(
I disabled sslh,let's see.

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: systemd high cpu usage

Mon Jun 15, 2015 12:17 pm

I have also problem installing packages from apt: it waits for ages...
How can I take all the output of

Code: Select all

sudo journalctl
and paste it there for support?

myoung008
Posts: 55
Joined: Mon Mar 18, 2013 9:56 pm

Re: systemd high cpu usage

Mon Jun 15, 2015 3:44 pm

Code: Select all

sudo journalctl -n100 > file.txt
The -n trims the output to the last X lines because the file is likely to be quite large otherwise. Depending on the size of the relevant lines you can post as an attachment or in a code block.

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: systemd high cpu usage

Tue Jun 16, 2015 8:18 am

Here the output of

Code: Select all

sudo journalctl -n10000 > file.txt
http://pastebin.com/0u2GQg0z

and this

Code: Select all

sudo systemctl list-units -n10000 > /tmp/units.txt
http://pastebin.com/GsS64KWV

myoung008
Posts: 55
Joined: Mon Mar 18, 2013 9:56 pm

Re: systemd high cpu usage

Tue Jun 16, 2015 7:38 pm

I just updated a fresh Raspbian Wheezy to Jessie and don't have any of the problems you are experiencing. Your journal points to dbus failure as the cause of most of your noticeable issues.

Code: Select all

giu 15 22:59:22 raspberrypi systemd[640]: Failed at step CGROUP spawning /usr/bin/dbus-daemon: No such file or directory
So it appears that dbus-daemon or one of it's dependencies is missing. You've run "dpkg --configure -a" (sudo logs in the journal as well). That should fix most installation problems if it completes with no errors. If there are errors you should try to correct those first.

Are you out of disk space? Maybe some filesystem error?

It took a little over 5GB to upgrade my Wheezy to Jessie. If you didn't resize your root partition before upgrade, or if you only have a 4GB card that could be the real underlying issue.

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: systemd high cpu usage

Tue Jun 16, 2015 8:40 pm

I run

Code: Select all

sudo dpkg --configure -a
but no output is returned.

I have space on the drives:

Code: Select all

pi@raspberrypi ~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7,3G  4,5G  2,5G  65% /
devtmpfs        182M     0  182M   0% /dev
tmpfs           186M     0  186M   0% /dev/shm
tmpfs           186M  4,8M  182M   3% /run
tmpfs           5,0M  4,0K  5,0M   1% /run/lock
tmpfs           186M     0  186M   0% /sys/fs/cgroup
/dev/sda1       444G  237G  186G  57% /media/screenplay
/dev/mmcblk0p1   56M   20M   37M  35% /boot

myoung008
Posts: 55
Joined: Mon Mar 18, 2013 9:56 pm

Re: systemd high cpu usage

Tue Jun 16, 2015 8:56 pm

I think your system is brobably broken beyond this, but try:

Code: Select all

sudo apt-get install --reinstall adduser libaudit1 libc6 libcap-ng0 libdbus-1-3 libexpat1 libselinux1 libsystemd0 lsb-base
That's dbus and all of it's direct dependencies. If it completes the reinstall correctly, reboot and see if things get better.

Otherwise, it may be time for a reinstall.

User avatar
rpdom
Posts: 15178
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: systemd high cpu usage

Wed Jun 17, 2015 10:28 am

myoung008 wrote:I think your system is brobably broken beyond this, but try:

Code: Select all

sudo apt-get install --reinstall adduser libaudit1 libc6 libcap-ng0 libdbus-1-3 libexpat1 libselinux1 libsystemd0 lsb-base
That's dbus and all of it's direct dependencies. If it completes the reinstall correctly, reboot and see if things get better.
I can't see dbus in that list, just libdbus. I'd add "dbus" on the end of that line, just to be sure.

Giuliano69
Posts: 9
Joined: Fri Jul 19, 2013 9:10 am

Re: systemd high cpu usage

Wed Jun 17, 2015 2:19 pm

Raspberry Pi 2 + Jessie
some situation occurring... found a workaround...


clean install of Weezy, and file-system expansion to 8 Gb
upgraded to Jessie

Problem:
Systemd taking 98% of cpu

solution
pkill 1

(systemd is the 1st process in top list of processes....)

After killing it, a new systemd emerge, but CPU usage drop to 0%

Hope it helps...

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: systemd high cpu usage

Wed Jun 17, 2015 4:02 pm

After myoung008 hints systemd now run fine!

Many Thanks!!!!

myoung008
Posts: 55
Joined: Mon Mar 18, 2013 9:56 pm

Re: systemd high cpu usage

Wed Jun 17, 2015 4:54 pm

How funny that I forgot dbus itself on the reinstall list! :!:

Anyway, I'm glad you got it all working.

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: systemd high cpu usage

Thu Jun 18, 2015 8:48 pm

Last thing: how can I check which services are running at startup and which I have installed but not autostarted?

myoung008
Posts: 55
Joined: Mon Mar 18, 2013 9:56 pm

Re: systemd high cpu usage

Thu Jun 18, 2015 8:56 pm

Code: Select all

systemctl list-unit-files
You may also want:

Code: Select all

systemctl enable xxxxx
systemctl disable xxxxx
systemd-analyze blame
systemd-analyze plot > bootchart.svg
After that last one, open bootchart.svg in a svg capable image viewer.

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: systemd high cpu usage

Thu Jun 18, 2015 10:20 pm

Oh gosh! Fantastic!!!!
Thank you.

With the graph I can check what is wasting time in startup!

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: systemd high cpu usage

Mon Jun 22, 2015 2:45 pm

Everything is fine now....except an annoying log message in

Code: Select all

sudo journalctl

giu 22 16:27:34 raspberrypi sslh[564]: connection from localhost:44275 to localhost:https forwarded from localhost:4428
giu 22 16:27:34 raspberrypi sslh[564]: connection from localhost:44274 to localhost:https forwarded from localhost:4428
giu 22 16:27:34 raspberrypi sslh[564]: connection from localhost:44282 to localhost:https forwarded from localhost:4429
giu 22 16:27:34 raspberrypi sslh[564]: connection from localhost:44264 to localhost:https forwarded from localhost:4428
As suggested seems to be relative to sslh. it has only these two config lines:

Code: Select all

DAEMON=/usr/sbin/sslh
DAEMON_OPTS="--user sslh --listen 0.0.0.0:443 --ssh 127.0.0.1:22 --ssl 127.0.0.1:443 --pidfile /var/run/sslh/sslh.pid"
How can I avoid writing so many lines in journal?
Where can I find where journal log is stored?

myoung008
Posts: 55
Joined: Mon Mar 18, 2013 9:56 pm

Re: systemd high cpu usage

Mon Jun 22, 2015 4:56 pm

Sslh is logging each new connection. I don't see an option for sslh to reduce logging. You may be able to filter it with journald. Two possible reasons for the excessive logs:

If you can NOT connect to your final https service correctly: You may still be looping your connections with your sslh config. When you listen on 0.0.0.0 you are probably binding to both 127.0.0.1 and the actual ip of the machine. When it forwards to 127.0.0.1:443, it is looping back through sslh instead of reaching your https service. Try changing the sslh --listen from 0.0.0.0:443 to the actual IP (keeping the :443), then make sure that the service providing https is bound only to 127.0.0.1.

If your https connection IS working then it is probably your browser requesting lots of files over multiple connections, or someone's automated bot hitting your server.

Journald config is in /etc/systemd/journald.conf. Default journal directory is /var/log/journal. That doesn't exist on my Jessie system so it's falling back to journal in memory. On my system that using up to 46MB of ram.

Hope this helps.

Return to “Troubleshooting”