e-raser
Posts: 71
Joined: Sat Apr 04, 2015 1:30 pm

Re: Raspberry Pi 3 B+ lockups

Fri Apr 06, 2018 1:43 pm

I also suffer from a very instable system (talking about having sporadic reboots, mostly initiated by watchdog, maybe one to four a day during the last week). I tried several config.txt settings:
  • (OKAY) with "arm_freq=1200" and "sdram_freq=450" --> system was running very stable - but also not very performant of course.
  • (not OK) with only "sdram_freq=450" --> system already had the first reboot (= instable)
I´m really wondering if it´s a hardware or software issue?

I spent something between 11 to 13 hours during the last week for investigating and fixing (e. g. almost 10 external fsck’s necessary putting the SD card to another pi) all the issues coming from this unstable system on the Pi 3 B+. I‘m therefore very frustrated (first time in 5 years of Pi experience) and I think about returning that piece of faulty Pi hardware and get another one. I want stability AND performance - and spend my time for more important things -.-

So the situation with even having a stable, but slower system (1.2 GHz core freq, 450 MHz sdram freq) is not satisfying. Might be the next Pi 3 B+ has the same issue, but as I read (here and on other places) that some users have two or more Pi 3 B+ and not all of them have theses issues, at least there´s a chance, right.
1x Nextcloud & Pi-hole & ... on Raspbian @ Pi (4 4 GB)
1x Kodi media center on LibreELEC @ Pi 3 B+

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5372
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Raspberry Pi 3 B+ lockups

Fri Apr 06, 2018 2:48 pm

e-raser wrote:
Fri Apr 06, 2018 1:43 pm
[*] (OKAY) with "arm_freq=1200" and "sdram_freq=450" --> system was running very stable - but also not very performant of course.
So I assume just adding arm_freq=1200 makes it stable?
If so, you could try removing arm_freq=1200 and adding "over_voltage=2" which may make it stable without affecting performance.

User avatar
PeterO
Posts: 5147
Joined: Sun Jul 22, 2012 4:14 pm

Re: Raspberry Pi 3 B+ lockups

Fri Apr 06, 2018 3:13 pm

When things have settled down, can someone update the information on this page https://www.raspberrypi.org/documentati ... locking.md to include the settings/values for the 3B+.

PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24162
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspberry Pi 3 B+ lockups

Fri Apr 06, 2018 8:28 pm

PeterO wrote:
Fri Apr 06, 2018 3:13 pm
When things have settled down, can someone update the information on this page https://www.raspberrypi.org/documentati ... locking.md to include the settings/values for the 3B+.

PeterO
I updated some numbers this afternoon....
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

horgh
Posts: 15
Joined: Wed Mar 28, 2018 12:10 am
Location: Vancouver, BC, Canada

Re: Raspberry Pi 3 B+ lockups

Sat Apr 07, 2018 1:10 am

I had switched to only arm_freq=1200 in config.txt (commented out sdram_freq=450). My Pi was mostly ok for about 2 days (I did have a process crash which was suspicious), but it just crashed again:

Code: Select all

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.655382] Internal error: Oops: 817 [#1] SMP ARM

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.698342] Process kswapd0 (pid: 44, stack limit = 0xbab64210)

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.700893] Stack: (0xbab65d20 to 0xbab66000)

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.703468] 5d20: bab65d4c bab65d30 80252c68 802520e0 859fdb5c bab5aa04 00000000 bab65e40

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.708716] 5d40: bab65d84 bab65d50 80252498 80252bf4 bab5aa00 00000000 bab65da8 bab65e40

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.713975] 5d60: 00000080 00000000 80c20060 00001ae5 80b8cd40 00000000 bab65d9c bab65d88

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.719260] 5d80: 802525b4 80252408 00000000 bab65e40 bab65dbc bab65da0 80252f6c 80252588

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.724709] 5da0: 00000000 bab65e40 00000211 000000cd bab65e7c bab65dc0 80235d78 80252f28

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.730491] 5dc0: 00000000 0000cb65 80c85544 00000001 bab65e1c 0000052f 80c85544 bab65f28

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.736578] 5de0: 00000000 00000001 bab65e7c bab64000 00000a84 00000000 00000077 000002a1

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.742905] 5e00: 000000cd 51eb851f 00001800 80c19814 00000000 00000000 00000000 014000c0

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.749559] 5e20: 00000000 00000000 00000056 00000000 00000000 bab65dc0 bab65e38 014000c0

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.756356] 5e40: 0000007e 00000080 00000000 00000000 80283e28 bab65f28 00000000 00000000

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.763313] 5e60: 00000000 bab65eac 000002a1 80c84e40 bab65ee4 bab65e80 80239bb8 80235ad4

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.769479] 5e80: 0001a39c 8023aad8 00000000 80d10edc 80c03ac0 bab65f20 0000028e 00000000

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.773012] 5ea0: 00000000 0001a39c 0001a39c 80c84e40 00000007 00000000 80235a40 00000000

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.776539] 5ec0: 00000000 00000001 00000000 80d1269c 80d10ee0 80c84e40 bab65f7c bab65ee8

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.780068] 5ee0: 8023ab28 80239854 00000001 80c200f8 80b8cd4c baa8da00 80c89128 80c03d68

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.783600] 5f00: 80160880 bab65ee8 00000003 bab65f34 80c854fc 80cbdc14 bab64000 0000028e

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.787129] 5f20: 00000000 00000000 00001800 014000c0 00000000 00000000 00000000 00000007

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.790657] 5f40: 00000000 00000007 000002a1 0000052f baa0951c baa09500 00000000 bab5aa00

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.794179] 5f60: 80c84e40 8023a808 baa0951c ba8efe40 bab65fac bab65f80 8013d880 8023a814

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.797709] 5f80: 80102dd0 bab5aa00 8013d744 00000000 00000000 00000000 00000000 00000000

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.801235] 5fa0: 00000000 bab65fb0 8010812c 8013d750 00000000 00000000 00000000 00000000

Message from syslogd@rpi5 at Apr  6 17:28:01 ...
 kernel:[188790.804761] 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
It sounds like this isn't unexpected as the suspicion is the sdram_freq change is the important one.

I'll try switching to just the sdram_freq=450 now.

horgh
Posts: 15
Joined: Wed Mar 28, 2018 12:10 am
Location: Vancouver, BC, Canada

Re: Raspberry Pi 3 B+ lockups

Sat Apr 07, 2018 1:20 am

I'm not sure if this is relevant, but I found these in the logs from yesterday:

Code: Select all

Apr  5 14:20:00 rpi5 kernel: [91109.253269] ------------[ cut here ]------------
Apr  5 14:20:00 rpi5 kernel: [91109.253304] WARNING: CPU: 2 PID: 44 at mm/workingset.c:466 shadow_lru_isolate+0x284/0x334
Apr  5 14:20:00 rpi5 kernel: [91109.253311] Modules linked in: ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter sg brcmfmac brcm
util cfg80211 rfkill snd_bcm2835(C) snd_pcm snd_timer snd uio_pdrv_genirq fixed uio ip_tables x_tables ipv6
Apr  5 14:20:00 rpi5 kernel: [91109.253429] CPU: 2 PID: 44 Comm: kswapd0 Tainted: G         C      4.14.31-v7+ #1104
Apr  5 14:20:00 rpi5 kernel: [91109.253434] Hardware name: BCM2835
Apr  5 14:20:00 rpi5 kernel: [91109.253464] [<8010fff8>] (unwind_backtrace) from [<8010c260>] (show_stack+0x20/0x24)
Apr  5 14:20:00 rpi5 kernel: [91109.253486] [<8010c260>] (show_stack) from [<80783ac4>] (dump_stack+0xd4/0x118)
Apr  5 14:20:00 rpi5 kernel: [91109.253505] [<80783ac4>] (dump_stack) from [<8011da74>] (__warn+0xf8/0x110)
Apr  5 14:20:00 rpi5 kernel: [91109.253519] [<8011da74>] (__warn) from [<8011db5c>] (warn_slowpath_null+0x30/0x38)
Apr  5 14:20:00 rpi5 kernel: [91109.253534] [<8011db5c>] (warn_slowpath_null) from [<80252e6c>] (shadow_lru_isolate+0x284/0x334)
Apr  5 14:20:00 rpi5 kernel: [91109.253549] [<80252e6c>] (shadow_lru_isolate) from [<80252498>] (__list_lru_walk_one+0x9c/0x180)
Apr  5 14:20:00 rpi5 kernel: [91109.253564] [<80252498>] (__list_lru_walk_one) from [<802525b4>] (list_lru_walk_one+0x38/0x40)
Apr  5 14:20:00 rpi5 kernel: [91109.253577] [<802525b4>] (list_lru_walk_one) from [<80252f6c>] (scan_shadow_nodes+0x50/0x68)
Apr  5 14:20:00 rpi5 kernel: [91109.253594] [<80252f6c>] (scan_shadow_nodes) from [<80235d78>] (shrink_slab+0x2b0/0x510)
Apr  5 14:20:00 rpi5 kernel: [91109.253610] [<80235d78>] (shrink_slab) from [<80239bb8>] (shrink_node+0x370/0x374)
Apr  5 14:20:00 rpi5 kernel: [91109.253626] [<80239bb8>] (shrink_node) from [<8023ab28>] (kswapd+0x320/0x824)
Apr  5 14:20:00 rpi5 kernel: [91109.253643] [<8023ab28>] (kswapd) from [<8013d880>] (kthread+0x13c/0x16c)
Apr  5 14:20:00 rpi5 kernel: [91109.253661] [<8013d880>] (kthread) from [<8010812c>] (ret_from_fork+0x14/0x28)
Apr  5 14:20:00 rpi5 kernel: [91109.253669] ---[ end trace c0e384bee6814d8c ]---
Apr  5 14:20:00 rpi5 kernel: [91109.253675] ------------[ cut here ]------------
Apr  5 14:20:00 rpi5 kernel: [91109.253687] WARNING: CPU: 2 PID: 44 at mm/workingset.c:458 shadow_lru_isolate+0x2e4/0x334
Apr  5 14:20:00 rpi5 kernel: [91109.253691] Modules linked in: ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter sg brcmfmac brcmutil cfg80211 rfkill snd_bcm2835(C) snd_pcm snd_timer snd uio_pdrv_genirq fixed uio ip_tables x_tables ipv6
Apr  5 14:20:00 rpi5 kernel: [91109.253789] CPU: 2 PID: 44 Comm: kswapd0 Tainted: G        WC      4.14.31-v7+ #1104
Apr  5 14:20:00 rpi5 kernel: [91109.253794] Hardware name: BCM2835
Apr  5 14:20:00 rpi5 kernel: [91109.253809] [<8010fff8>] (unwind_backtrace) from [<8010c260>] (show_stack+0x20/0x24)
Apr  5 14:20:00 rpi5 kernel: [91109.253824] [<8010c260>] (show_stack) from [<80783ac4>] (dump_stack+0xd4/0x118)
Apr  5 14:20:00 rpi5 kernel: [91109.253840] [<80783ac4>] (dump_stack) from [<8011da74>] (__warn+0xf8/0x110)
Apr  5 14:20:00 rpi5 kernel: [91109.253852] [<8011da74>] (__warn) from [<8011db5c>] (warn_slowpath_null+0x30/0x38)
Apr  5 14:20:00 rpi5 kernel: [91109.253866] [<8011db5c>] (warn_slowpath_null) from [<80252ecc>] (shadow_lru_isolate+0x2e4/0x334)
Apr  5 14:20:00 rpi5 kernel: [91109.253880] [<80252ecc>] (shadow_lru_isolate) from [<80252498>] (__list_lru_walk_one+0x9c/0x180)
Apr  5 14:20:00 rpi5 kernel: [91109.253894] [<80252498>] (__list_lru_walk_one) from [<802525b4>] (list_lru_walk_one+0x38/0x40)
Apr  5 14:20:00 rpi5 kernel: [91109.253907] [<802525b4>] (list_lru_walk_one) from [<80252f6c>] (scan_shadow_nodes+0x50/0x68)
Apr  5 14:20:00 rpi5 kernel: [91109.253922] [<80252f6c>] (scan_shadow_nodes) from [<80235d78>] (shrink_slab+0x2b0/0x510)
Apr  5 14:20:00 rpi5 kernel: [91109.253937] [<80235d78>] (shrink_slab) from [<80239bb8>] (shrink_node+0x370/0x374)
Apr  5 14:20:00 rpi5 kernel: [91109.253952] [<80239bb8>] (shrink_node) from [<8023ab28>] (kswapd+0x320/0x824)
Apr  5 14:20:00 rpi5 kernel: [91109.253967] [<8023ab28>] (kswapd) from [<8013d880>] (kthread+0x13c/0x16c)
Apr  5 14:20:00 rpi5 kernel: [91109.253983] [<8013d880>] (kthread) from [<8010812c>] (ret_from_fork+0x14/0x28)
Apr  5 14:20:00 rpi5 kernel: [91109.253990] ---[ end trace c0e384bee6814d8d ]---

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2489
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 10:19 am

It's relevant in that it suggests memory corruption, but I don't think we'll get any more useful information from it.

I'm more interested in your sdram_freq=450 test results. I have a feeling they will be more successful.

User avatar
PeterO
Posts: 5147
Joined: Sun Jul 22, 2012 4:14 pm

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 10:39 am

Mine's been stable running with 1200/450.
PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2489
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 10:43 am

Mine's been stable running with 1200/450.
I don't want you to risk any important data, but I'm interested to know if removing the arm_freq setting (or explicitly setting it to the default value of 1400), leaving sdram_freq at 450, makes your system unstable again.

NoSheds
Posts: 2
Joined: Wed Apr 04, 2018 11:27 am

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 11:28 am

Following on from my previous post, I'm still having problems. I have two backup systems backing up to a PI3b+ with a powered external USB disk. I have cloned the SD card to a different one. I have swapped from an unpowered USB disk to a powered one. I have tried two different PI3B+s. I have updated the Raspbian distribution using apt-get update (etc), and I have tried rpi-update. The BackupPC software fails with a "Broken Pipe" (I believe to be an rsync failure between client and server) wheras the SimpleBackup (sbackup) software generally results in a "hung" system with the "writing to the SD" card light on the PI permanently lit. Usually the PI is completely dead, but last time it happened it responded to pings, but not to ssh and the webserver it runs was unresponsive.

The machine is headless so I can't see any on-screen messages. There are no obvious error messages in the logs. As well as the software updates and RPI-Updates, I have tried dtparam=eee=off, arm_freq=1200 and sdram_freq=450 which don't seem to have fixed my problems.

Swapping back to my Pi 3B (not plus), and the backup systems work again.

Can anyone help? What should I do? Should I wait for a software fix, try tweaking again or just give up and return the B+ machines? :cry:

User avatar
RaTTuS
Posts: 10501
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
Contact: Twitter YouTube

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 11:35 am

NoSheds wrote:
Mon Apr 09, 2018 11:28 am
Following on from my previous post, I'm still having problems. I have two backup systems backing up to a PI3b+ with a powered external USB disk. I have cloned the SD card to a different one. I have swapped from an unpowered USB disk to a powered one. I have tried two different PI3B+s. I have updated the Raspbian distribution using apt-get update (etc), and I have tried rpi-update. The BackupPC software fails with a "Broken Pipe" (I believe to be an rsync failure between client and server) wheras the SimpleBackup (sbackup) software generally results in a "hung" system with the "writing to the SD" card light on the PI permanently lit. Usually the PI is completely dead, but last time it happened it responded to pings, but not to ssh and the webserver it runs was unresponsive.

The machine is headless so I can't see any on-screen messages. There are no obvious error messages in the logs. As well as the software updates and RPI-Updates, I have tried dtparam=eee=off, arm_freq=1200 and sdram_freq=450 which don't seem to have fixed my problems.

Swapping back to my Pi 3B (not plus), and the backup systems work again.

Can anyone help? What should I do? Should I wait for a software fix, try tweaking again or just give up and return the B+ machines? :cry:
personally I'd start a new thread
get a screen attached
start with a fresh install
sudo apt update && sudo apt-full-upgrade
don't use rpi-update
report lsusb -v
install whatever you need - show commands
make it crash and see what it says on the screen
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

User avatar
PeterO
Posts: 5147
Joined: Sun Jul 22, 2012 4:14 pm

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 12:11 pm

PhilE wrote:
Mon Apr 09, 2018 10:43 am
Mine's been stable running with 1200/450.
I don't want you to risk any important data, but I'm interested to know if removing the arm_freq setting (or explicitly setting it to the default value of 1400), leaving sdram_freq at 450, makes your system unstable again.
I'm currently keeping everything I'm working on replicated (manually) across several machines. I'll up the arm_freq to 1300 first and see what happens.
PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2489
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 1:17 pm

I would prefer you to go straight to 1400 - we will learn more sooner that way.

User avatar
PeterO
Posts: 5147
Joined: Sun Jul 22, 2012 4:14 pm

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 1:29 pm

PhilE wrote:
Mon Apr 09, 2018 1:17 pm
I would prefer you to go straight to 1400 - we will learn more sooner that way.
Are confident that it is the ram frequency that is the problem ?
PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2489
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 1:33 pm

It's hard to be confident because the failure rate is currently very low and we don't have any boards exhibiting this problem. Would you be happy to exchange your 3B+ for another so that we can run some extensive tests back at Pi Towers? PM me if you are.

e-raser
Posts: 71
Joined: Sat Apr 04, 2015 1:30 pm

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 2:29 pm

There‘s the same offer from James on the GitHub issue here (https://github.com/raspberrypi/linux/is ... -379260285), in my opinion we should use those chances to therefore help the experts to investigate, probably fix and in the end help us (affected) users.
1x Nextcloud & Pi-hole & ... on Raspbian @ Pi (4 4 GB)
1x Kodi media center on LibreELEC @ Pi 3 B+

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2489
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 2:50 pm

Yes - if anyone is seeing instability that is cured by reducing sdram_freq to 450 or arm_freq to 1200 then please contact me to arrange an exchange.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24162
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 2:55 pm

NoSheds wrote:
Mon Apr 09, 2018 11:28 am
Following on from my previous post, I'm still having problems. I have two backup systems backing up to a PI3b+ with a powered external USB disk. I have cloned the SD card to a different one. I have swapped from an unpowered USB disk to a powered one. I have tried two different PI3B+s. I have updated the Raspbian distribution using apt-get update (etc), and I have tried rpi-update. The BackupPC software fails with a "Broken Pipe" (I believe to be an rsync failure between client and server) wheras the SimpleBackup (sbackup) software generally results in a "hung" system with the "writing to the SD" card light on the PI permanently lit. Usually the PI is completely dead, but last time it happened it responded to pings, but not to ssh and the webserver it runs was unresponsive.

The machine is headless so I can't see any on-screen messages. There are no obvious error messages in the logs. As well as the software updates and RPI-Updates, I have tried dtparam=eee=off, arm_freq=1200 and sdram_freq=450 which don't seem to have fixed my problems.

Swapping back to my Pi 3B (not plus), and the backup systems work again.

Can anyone help? What should I do? Should I wait for a software fix, try tweaking again or just give up and return the B+ machines? :cry:
I suspect this is more a network issue than a CPU/speed issue. There have been a few things checked in in the last few days, might be worth trying rpi-update as that will pull the bug fixes in.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

horgh
Posts: 15
Joined: Wed Mar 28, 2018 12:10 am
Location: Vancouver, BC, Canada

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 2:58 pm

My Pi has been stable for over 2 days with sdram_freq=450 set and arm_freq left at the default:

Code: Select all

$ uname -a
Linux rpi5 4.14.31-v7+ #1104 SMP Thu Mar 29 16:52:18 BST 2018 armv7l GNU/Linux
$ uptime
 07:57:08 up 2 days, 13:44, 12 users,  load average: 0.16, 0.20, 0.17

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2489
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 3:58 pm

Thanks - you would be a good candidate for an exchange if you weren't so far away. Can we try adjusting some of the SDRAM parameters instead?

1. Start by installing memtester and capturing some of the current settings:

Code: Select all

pi@raspberrypi:~$ sudo apt-get install memtester
pi@raspberrypi:~$ vcgencmd get_config int | grep -E "(volt|sdram)"
2. Then put sdram_freq back to 500 and set over_voltage_sdram_p=4 and over_voltage_sdram_i=4, and reboot.
3. Run memtester to generate some heavy SDRAM traffic:

Code: Select all

pi@raspberrypi:~$ memtester 512M
If that is stable then you could back off either of the values and repeat, otherwise you could try pushing the over-voltages to 6.

e-raser
Posts: 71
Joined: Sat Apr 04, 2015 1:30 pm

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 4:06 pm

horgh wrote:
Mon Apr 09, 2018 2:58 pm
My Pi has been stable for over 2 days with sdram_freq=450 set and arm_freq left at the default:

Code: Select all

$ uname -a
Linux rpi5 4.14.31-v7+ #1104 SMP Thu Mar 29 16:52:18 BST 2018 armv7l GNU/Linux
$ uptime
 07:57:08 up 2 days, 13:44, 12 users,  load average: 0.16, 0.20, 0.17
Same for me for 2-3 days. Only one reboot after 30 minutes and two „freezes“ (system non-responsive, services like MySQL and PHP-FPM, NGINX etc. down for ~ one minute). So to be more specific: no reboots causing instability, but still not very reliable (like my Pi 2 B before: running for 180 days without one hickup or reboot).
1x Nextcloud & Pi-hole & ... on Raspbian @ Pi (4 4 GB)
1x Kodi media center on LibreELEC @ Pi 3 B+

evizw2
Posts: 7
Joined: Mon Apr 09, 2018 7:16 pm

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 7:49 pm

PhilE wrote:
Mon Apr 09, 2018 3:58 pm
...
If that is stable then you could back off either of the values and repeat, otherwise you could try pushing the over-voltages to 6.
I'm seeing similar issues on my 3B+. I'd leave it compiling something and come back to it a while later only to see it had frozen and was unresponsive to keyboard input/ping. I set arm_freq and sdram_freq to 1200/450 and in my limited testing it seemed to resolve the issue. After setting arm_freq and sdram_freq back to 500 I tried setting both of the over voltage parameters you mentioned to 4 and then running "memtest 512M". I got the following crash output on the serial console:
[ 575.609917] Unable to handle kernel paging request at virtual address febf7fc4
[ 575.617259] pgd = 88a34000
[ 575.620005] [febf7fc4] *pgd=00000000
[ 575.623641] Internal error: Oops: 5 [#1] SMP ARM
[ 575.628323] Modules linked in: rfcomm cmac bnep hci_uart btbcm serdev bluetooth ecdh_generic fuse snd_bcm2835(C) snd_pcm brcmfmac brcmutil snd_timer snd cfg80211 rfkill fixed uio_pdrv_genirq uio evdev i2c_dev ip_tables x_tables ipv6
[ 575.649290] CPU: 0 PID: 8375744 Comm: lxterminal Tainted: G C 4.14.32-v7+ #1107
[ 575.657848] Hardware name: BCM2835
[ 575.661298] task: 93d46900 task.stack: 888fe000
[ 575.665905] PC is at acct_collect+0x70/0x1ec
[ 575.670237] LR is at down_read+0x1c/0x60
[ 575.674218] pc : [<801aae64>] lr : [<8079ca84>] psr: a0000113
[ 575.680573] sp : 888ffdf0 ip : 75fe7000 fp : 888ffe14
[ 575.685870] r10: 0000000b r9 : 888fe000 r8 : 00000001
[ 575.691169] r7 : 00106001 r6 : 00000544 r5 : 0000000b r4 : 93fdd600
[ 575.697787] r3 : febf7fc0 r2 : 00004000 r1 : 01f3f000 r0 : 93fb0380
[ 575.704408] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[ 575.711644] Control: 10c5383d Table: 08a3406a DAC: 00000055
[ 575.717473] Process lxterminal (pid: 8375744, stack limit = 0x888fe210)
[ 575.724180] Stack: (0x888ffdf0 to 0x88900000)
[ 575.728599] fde0: 93d46900 0000000b 00000544 00106001
[ 575.736894] fe00: 00000001 888fe000 888ffe54 888ffe18 80122060 801aae00 888ffe18 888ffe18
[ 575.745190] fe20: 888ffedc 888fffb0 00000000 0000000b 93fdd600 888fe000 00106001 888ffedc
[ 575.753484] fe40: 888fe000 0000000b 888ffe74 888ffe58 8012267c 80121a54 888ffec8 00000000
[ 575.761783] fe60: b9066488 888fe000 888ffec4 888ffe78 8012da60 8012263c 888ffedc 80c02040
[ 575.770079] fe80: 418004fc 80c03d68 888ffec8 93fdd600 b90668c4 b90663c0 888fff4c 00000000
[ 575.778373] fea0: 888ffec8 888fffb0 00000000 00000000 888fe000 76315000 888fff8c 888ffec8
[ 575.786667] fec0: 8010b314 8012d700 888ffeec 888ffed8 804c2580 804c24a4 804c2544 0000000b
[ 575.794963] fee0: 00000000 00000001 6f777f7d 00000020 b8a93220 00000000 888ffea8 801236e0
[ 575.803259] ff00: 888fff40 80c046cc 20000113 ffffffff 888fff74 00000000 888fff3c 8010b810
[ 575.811556] ff20: 20000113 ffffffff 888fff74 00000000 888fffac 888fff40 8079f8c0 801eda18
[ 575.819853] ff40: 888fffb0 00000001 00000000 00000000 00000001 888fe010 00000000 888fffb0
[ 575.828150] ff60: 00000000 00000001 888fe010 00000000 888fffb0 00000000 888fe000 76315000
[ 575.836445] ff80: 888fffac 888fff90 8010b81c 8010b25c 7624fbe0 40000010 ffffffff 10c5383d
[ 575.844741] ffa0: 00000000 888fffb0 801080b4 8010b770 6f777f7d 762f7004 000f7134 7eb8de2c
[ 575.853037] ffc0: 6f777f7d 7b736f6d 00000004 00000001 762f7000 7eb8de94 76315000 7eb8dedc
[ 575.861333] ffe0: 0000001c 7eb8de18 761fff8c 7624fbe0 40000010 ffffffff 00000000 00000000
After setting them to 6 I got this:
[ 741.030792] Unable to handle kernel NULL pointer dereference at virtual address 00000034
[ 741.039019] pgd = 889d4000
[ 741.041768] [00000034] *pgd=08a3a835, *pte=00000000, *ppte=00000000
[ 741.048144] Internal error: Oops: 17 [#1] SMP ARM
[ 741.052917] Modules linked in: rfcomm cmac bnep fuse hci_uart btbcm serdev bluetooth ecdh_generic brcmfmac snd_bcm2835(C) brcmutil snd_pcm snd_timer cfg80211 rfkill snd fixed uio_pdrv_genirq uio evdev i2c_dev ip_tables x_tables ipv6
[ 741.073879] CPU: 0 PID: 1099 Comm: lxterminal Tainted: G C 4.14.32-v7+ #1107
[ 741.082172] Hardware name: BCM2835
[ 741.085621] task: b93fcb00 task.stack: 8888c000
[ 741.090228] PC is at bcm2836_chained_handle_irq+0x24/0x50
[ 741.095704] LR is at get_next_armctrl_hwirq+0x8c/0xac
[ 741.100827] pc : [<804c2568>] lr : [<804c2524>] psr: 00000193
[ 741.107180] sp : 8888dd38 ip : 00000039 fp : 8888dd4c
[ 741.112478] r10: 0000000b r9 : 8888c000 r8 : b9c03180
[ 741.117776] r7 : 00000001 r6 : 00000000 r5 : 00000000 r4 : 80c0469c
[ 741.124396] r3 : 00000000 r2 : 00000000 r1 : 00000001 r0 : 00000029
[ 741.131016] Flags: nzcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
[ 741.138341] Control: 10c5383d Table: 089d406a DAC: 00000055
[ 741.144170] Process lxterminal (pid: 1099, stack limit = 0x8888c210)
[ 741.150614] Stack: (0x8888dd38 to 0x8888e000)
[ 741.155032] dd20: 804c2544 80b8e92c
[ 741.163328] dd40: 8888dd5c 8888dd50 80174f10 804c2550 8888dd84 8888dd60 80175550 80174ee8
[ 741.171625] dd60: 8888dda0 80c046cc a0000113 ffffffff 8888ddd4 00000001 8888dd9c 8888dd88
[ 741.179921] dd80: 80101504 801754f0 80c04174 801aae68 8888de14 8888dda0 8079f8bc 80101468
[ 741.182957] Unable to handle kernel NULL pointer dereference at virtual address 00000004
[ 741.196420] dda0: 93d0c700 0240f000 765fc000 93d22480 93f83440 0000000b 00000544 00106001
[ 741.204716] ddc0: 00000001 8888c000 0000000b 8888de14 765ec000 8888ddf0 8079ca84 801aae68
[ 741.213011] dde0: a0000113 ffffffff 801aae40 7f000000 b93fcb00 0000000b 00000544 00106001
[ 741.221307] de00: 00000001 8888c000 8888de54 8888de18 80122060 801aae00 8888de18 8888de18
[ 741.229604] de20: 8888dedc 8888dfb0 00000000 0000000b 93f83440 8888c000 00106001 8888dedc
[ 741.237900] de40: 8888c000 0000000b 8888de74 8888de58 8012267c 80121a54 8888dec8 00000000
[ 741.246196] de60: b906ef08 8888c000 8888dec4 8888de78 8012da60 8012263c 8888dea4 80c02040
[ 741.254490] de80: 418004fc 80c03d68 8888dec8 93f83440 b906f344 b906ee40 8888defc 00000000
[ 741.262784] dea0: 8888dec8 8888dfb0 00000000 00000000 8888c000 00000001 8888df8c 8888dec8
[ 741.271082] dec0: 8010b314 8012d700 00000000 00020000 0002ec19 80c0963c 0000000f 0000000b
[ 741.030792] Unable to handle kernel NULL pointer dereference at virtual address 00000034
[ 741.039019] pgd = 889d4000
[ 741.041768] [00000034] *pgd=08a3a835, *pte=00000000, *ppte=00000000
[ 741.048144] Internal error: Oops: 17 [#1] SMP ARM
[ 741.052917] Modules linked in: rfcomm cmac bnep fuse hci_uart btbcm serdev bluetooth ecdh_generic brcmfmac snd_bcm2835(C) brcmutil snd_pcm snd_timer cfg80211 rfkill snd fixed uio_pdrv_genirq uio evdev i2c_dev ip_tables x_tables ipv6
[ 741.073879] CPU: 0 PID: 1099 Comm: lxterminal Tainted: G C 4.14.32-v7+ #1107
[ 741.082172] Hardware name: BCM2835
[ 741.085621] task: b93fcb00 task.stack: 8888c000
[ 741.090228] PC is at bcm2836_chained_handle_irq+0x24/0x50
[ 741.095704] LR is at get_next_armctrl_hwirq+0x8c/0xac
In testing with default voltages, arm_freq at the default and sdram_freq at 450 and running memtester it doesn't seem to crash. I've only been testing for a little while but it's already been running longer than it was when I got the two crashes with the output above.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24162
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 8:30 pm

e-raser wrote:
Mon Apr 09, 2018 4:06 pm
horgh wrote:
Mon Apr 09, 2018 2:58 pm
My Pi has been stable for over 2 days with sdram_freq=450 set and arm_freq left at the default:

Code: Select all

$ uname -a
Linux rpi5 4.14.31-v7+ #1104 SMP Thu Mar 29 16:52:18 BST 2018 armv7l GNU/Linux
$ uptime
 07:57:08 up 2 days, 13:44, 12 users,  load average: 0.16, 0.20, 0.17
Same for me for 2-3 days. Only one reboot after 30 minutes and two „freezes“ (system non-responsive, services like MySQL and PHP-FPM, NGINX etc. down for ~ one minute). So to be more specific: no reboots causing instability, but still not very reliable (like my Pi 2 B before: running for 180 days without one hickup or reboot).
Is it possible those freezes were to do with the network dropping out for 60/120 seconds causing the app the stall waiting for it to come back, rather than an actual CPU freeze?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2489
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 9:08 pm

In testing with default voltages, arm_freq at the default and sdram_freq at 450 and running memtester it doesn't seem to crash. I've only been testing for a little while but it's already been running longer than it was when I got the two crashes with the output above.
That's another useful data point. What does this command return on your 3B+?

Code: Select all

vcgencmd get_config int | grep -E "(volt|sdram)"

evizw2
Posts: 7
Joined: Mon Apr 09, 2018 7:16 pm

Re: Raspberry Pi 3 B+ lockups

Mon Apr 09, 2018 9:43 pm

This is what that command outputs with sdram_freq=450 in config.txt. If you're looking for the values without anything set I can also get that for you.

Code: Select all

over_voltage_avs=43750
over_voltage_avs_boost=0x249f0
sdram_freq=450

Return to “Troubleshooting”